Gaps

For the better part of six months, I would keep two browser tabs open on my phone and notebooks for nice.social and beta.nice.social. The first site ran v4 of the platform while the beta ran v5. This was sub-optimal, but allowed for a good deal of testing to take place with the newer software in a realistic setting. Earlier this week when a server update took down the v4 service, the decision was made to move everyone and everything over to the new platform because I felt that it was ready despite a handful of incomplete items. As was to be expected, there were a whole lot more gaps in the tool than I had anticipated.

A good amount of time has been dedicated to migrating data and resolving reported bugs over the last three days and it has brought back memories of many other migrations I've done over the years for personal projects, client projects, and with several employers. When things go smoothly, it means that something is most probably wrong. When things are hectic, it means that something's wrong but the people reporting the issues give a darn. Crazy as it might sound, I generally prefer any sort of migration that is going to involve people who give a darn.

Some of the problems reported include missing posts, broken avatars, missing functions, and site routing issues. When something is reported, I write it to an ever-growing list of tasks, making sure to set aside the time to resolve the matter. If the missing or broken item is actively affecting people, then it gets pushed up closer near the top. As of this writing the critical items have been resolved1, and a half-dozen other issues remain. The ones that will be tackled next include:

  • change the font on the Anri blogging theme to a better sans serif font
  • resolve some of the reported CSS issues on the Anri theme
  • enable messages via the OpsBar[2. The OpsBar is the name of the bar that runs along the top of a 10C site when signed in.
  • return a JSON response for an object with a canonical URL when the HTTP header requests a JSON response
  • enable follow/block lists on the social site
  • complete password-protection handling in the Anri theme

There are also close to 1800 blog posts that still need to be brought over, and the podcasts need additional work to ensure all of the meta data is imported and sent properly in the syndication feeds. If all goes according to plan, all of the core items will be resolved on Monday or early Tuesday and then the focus can shift from "Identify and Repair" to "Converse and Extend".

If there's one thing I can take away from this experience, it's that I should really look at having data migrated daily in an automated fashion during the development phase. This would ensure that migration scripts were complete, meaning the actual migration would be done at the full speed of he server.


  1. If they weren't resolved, I wouldn't be blogging.

More is More

There's a lot to be said for a minimalistic simplicity. There's even more to be said for something that just works. The Anri theme on the upcoming 10Cv5 platform is something that I would really like for people to use without thinking about the level of complexity that is operating under the covers. As with all my work, the code is human-readable for anyone who is interested in seeing why things do what they do, but only those interested should even have the thought cross their mind.

Graph Paper and Pencil

Over the last couple of months I've managed to fill an entire notebook with scribbles that describe how functions work, why certain decisions were made, why others were avoided, and which order work should be performed. The amount of effort that has gone into Anri over the last three weeks covers barely a quarter of what's been planned for completion in the next little bit, though every update is incredibly important.

Last week the RSS and JSON Feed mechanisms were published. Today the ability to upload files and edit posts directly from Anri has been released. The next set of updates will focus on conversation threads and the OpsBar that runs along the top of a site when people are signed in. None of these are easy, nor should they be. If the v5 version of 10C was going to be easy, then I would have created a static site generator.

All in all, I'm quite happy with what's been accomplished so far this month. There's just one more quick little update I'd like to complete and send live before moving my main site over to the new platform, and I might just be able to get it written and deployed before the end of the day.

Here's hoping that the people who are testing the new platform are just as happy with the recent updates as I am.

Am making slow but steady progress on this system. Really looking forward to seeing it deployed and put to the test. There are a number of fundamental design differences between v4 and v5, which means this update is going to be bigger than the v2 to v4 change .

The page still loads relatively fast, but I wonder if there's a way for me to make the caching even better. The problem that I need to overcome is site versioning. This essentially is a metric to say whether anything on a given site has changed, whether it's through an new/updated/deleted post, a future-dated post that is now live. A post with an expiration date that is now invisible. Etc.

For most sites, this version id can be generated in under 33ms. For my site, however, about 450ms is required. This delay is something I can actively feel when using the site …

With a much larger collection of posts in the database, I get to see some pretty interesting performance metrics. One of the items that I find interesting is optimising things based on the sheer volume of posts that I tend to generate … which dwarfs just about everyone else.

A Missing Piece

Something has been bothering me with the structure of the 10Centuries data model recently and I've been unable to identify exactly what the problem was until I was on the train home from work yesterday when, in a flash of insight, things became clear. The underlying problem stems from a core limitation in 10Centuries with regards to accounts and how they're used in the content publishing mechanism: every account is treated as an identity, and every identity is essentially an account. If a person wants to publish posts on one website as "Tom" and on another website as "Jerry", they can easily use the same account. The problem is that when someone looks at the blog post via the RESTful API, they'll see that Tom and Jerry are both the same account but with a different author name.

This is silly, and it's something I've also blindly dealt with over the years writing the occasional post under an alias on various sites all the while using this very account. To make the matter even more blindingly obvious, publishing a social post alongside a blog post or podcast would come from the origin account, not an alias. So what exactly was the point of allowing aliases in the first place?

But this line of thinking is how the missing piece fell into place …

Missing Piece.jpg

What 10Centuries could really use are "personas". A single account could have many personas, but a persona would belong to only one account. This would allow a person to have what appears to be multiple accounts on here that are all available through a single sign-in. But then comes the obvious question: why would anybody want to do this?

With the dwindling activity on 10C — across all functions, not just social — I've been looking at ways to potentially resolve the "dead timeline" problem that people can face when they look at one of the social apps or sites. A lot of people have micro.blog accounts, and that project appears to be "open" enough that it's possible to build the necessary interfaces to play nicely with the network. Making 10C an "easy" way for people to use their micro.blog account seems like a natural fit. People would see posts from that network in their timeline, and they'd also see locally-created posts as well. The system would be smart enough to know where to send the response, posts too long for a single 280-character object on micro.blog could be presented as a partial with a link back to the full message in a proper stream view on a person's personal 10C site. Of course, people who do not wish to use micro.blog wouldn't have to. They could continue to use the system however they choose and everything would continue to work as expected. As working with micro.blog would require the full adoption of microformats, 10C sites would also — finally — adapt to play nice in the IndieWeb space.

Why the multiple personas, though? Separation of identities, really. Not everyone may want their micro.blog account tied to their main 10C account, and not everyone would have just one account to link to. It makes sense to allow multiples for personal, professional, creative, and silly accounts to exist all within one account, while appearing as multiple personas with varying degrees of data visibility via the API.

This is more or less a logical progression so far … right? I certainly hope so.

But wait. IndieWeb is an alternative to the "corporate web" and its silos. While 10C is most certainly not a corporate entity, it is still very much a silo. So this means in order to make this idea ideologically complete, I need to share the source code for 10C with the world.

Okay. I can do that. Version 5 is still very much slated to be the 100% open version of 10C that anyone and everyone can download, install, use — and abuse — to their heart's content, ideally finding issues and contributing back to the community whenever possible. This just means that Version 5 needs to have files to share.

Another Ambitious Goal?

As I explained recently, the last little bit at the day job has been an absolute slog. I'm not excited to go to work and solve problems there anymore. I'm dreading the inbox with every glance at the little mail icon. I'm just not much into the corporate tools right now. What I'd like to do is focus a bit on making something that I can be proud of, even for a little while. A basic version of 10C v5 might just pull me out of the doldrums and get the brain firing on an extra cylinder or two.

But this means I need to put a bit more thought into the data model and how everything will be structured. With 10Cv4 I started playing with the idea of "channels" and how every object was a self-contained entity that resided in a channel that could be accessed via a site or a pipe. While this model did solve some interesting problems, there's still room for improvement. The same can be said about the account structures, which still use the terrible term user and will be changed as soon as is feasible. It's already been decided that accounts can have personas, and accounts can own channels and be granted permissions to them. But what else can be improved about accounts and the model? Are the ToDo and Notes entities as effective as they could be? How about the Photo entities? There is a lot to think about, but not all at once.

What I'm going to aim to do is release a proof-of-concept of a Version 5 implementation in the coming weeks. It will not be too ambitious at first, instead being a simple little tool that will contain an API, a web presentation layer, files, and the start of a self-hosted platform with a few ideas I've been toying around with for 10Cv4 that would probably make more sense to ship with a more modern API. If the project is something people see potential in, then maybe there will be a few people contributing to the project and building something much better than a single person could hope to accomplish. I won't hold my hopes out, though.

There's a lot that I still don't understand about microformats and the IndieWeb project, and there are things in both projects that break my brain as I try to parse the logic behind decisions without knowing the context. That said, it's one of the few ways forward for people who truly care about owning their data and for the 10Centuries platform that I've invested so much time and effort into building.

Let's see if this concept of personas is the missing piece that will make 10C a little more interesting.

Books I'm Reading (or About to Read)

Since leaving the classroom and returning to the world of software development, I've tried to spend at least an hour a day reading about the changes that have taken place in technology since 2010. In July I started to dedicate my in-bed-before-sleep time to this task, and it's resulted in a lot of books being read and a bunch of new skills being acquired or refined. While it's impossible for any one person to know everything about a given subject, it should not be impossible for one person to know a healthy amount about a number of different subjects. This has certainly been the case with me while I learn more about data modelling, database design, and data warehousing. In fact, looking back at the technical books I've read in the last six months, it's easy to see that the vast majority are all related to databases in one way or another, and the four I have dedicated for January, February, and March are all on SQL Server.

I think I may have a little bit of a database fetish.

SQL Server Books

Last week I finished Stacia Varga's Developing SQL Data Models exam reference and I'm currently going through Victor Isakov's Administering a SQL Database Infrastructure reference in preparation for an upcoming Microsoft certification exam. On deck is Jose Chinchilla's Implementing a SQL Data Warehouse as data warehousing is a topic that has recently piqued my interest. Randolph's SQL Server 2017 Administration Inside Out is expected to be released by the end of next month, and I'll likely set aside some additional time to ingest the wisdom contained in the book. There's just so much to learn and explore!

A few people have asked why I read so much. They want to know specifically why I read so many technical books. When I think about it, though, the answer is not so cut and dry. Sure, I'd like to learn more about these tools so that I can make better use of the technology, but this isn't the only reason. Buried deep in the curiosity is the desire to discover what I do not yet know. As Bart Simpson so eloquently said to Mrs. Lovejoy all those years ago, "what you don't know could fill a warehouse."

It's true. We generally do not know what we do not know, and it's because of this ignorance that incomplete or inefficient decisions can be made to solve problems that other, smarter people may have resolved years before. Not a day goes by where I don't learn something new about the tools I use, and I hope this does not change anytime soon.

There's a certain excitement that comes from reading about an interesting feature or function, then trying it out and thinking about how it might be used to solve a real problem elsewhere. Back in November, I said that 10C would switch from MySQL to SQL Server just because I wanted to gain some experience with the platform on Linux. The conversion was finished mid-December, and nobody has reported any issues with the service since the switch. It was, by all accounts, one of the smoothest migrations of 10C I've ever performed. Learning more about SQL Server will (hopefully) allow me to do even more with the database going forward. More than this, I'd like to better understand how complex business problems could be solved with better use of this powerful tool.

While 10C is a personal project that I take very seriously and put a lot of care into, there is simply not enough "hard" work for me to do with the database. Businesses, however, ask a lot of tough questions. Depending on the quality of those questions, businesses may ask the question again and again in the form of reports. Being able to build the SQL queries to quickly and accurately return the answers is certainly a worthy place to use the skills I'm working so hard to acquire.

I Need To Be Chris

Between 2002 and 2007, I worked at a medium-sized company in Canada that was best known for its calendars and other print materials. I started in the warehouse and, over the course of 3 years, moved into different roles that culminated in a position as a software developer and worked with a number of very smart people who taught me a lot about software development, and a lot about how to ask the right questions to find out what people want the software to do, rather than making the wrong assumptions and delivering something that isn't at all what they're looking for. The person I learned from the most, however, was a man named Chris1.

Chris had a rather wide range of knowledge on just about every technical subject, no matter how obscure the tools might have been. His knowledge on certain subjects would often run circles around others, even when it was their area of focus. And, while he most certainly did complain when he was called in to fix somebody else's problem, he tried to make education part of the solution. There really isn't any point being "the only person who knows X" in a company, because that doesn't benefit anybody in the long run2. The guy seemed to know everything he needed and then some, and was honest enough to say "I don't know" when he really didn't know right before investigating whatever needed to be learned so that he wouldn't answer the same question the same way later.

I learned a lot from him in the two years or so we worked together, and would be happy to work with him again if the opportunity arose.

The way Chris handled situations was often incredibly efficient, and it's something I really need to work on myself. The last few weeks at the day job have been incredibly stressful as I attempt to do four very different tasks simultaneously in order to deliver a project that should have started limited trials back in August. I've recently complained that I shouldn't be doing four very different tasks if bugs and enhancements are going to be resolved by arbitrary deadlines, but complaining about reality will rarely resolve the problems one faces.

I've been incredibly fortunate over the last two decades to have worked with a lot of very different technologies and worked in a lot of very different roles. This sort of make me a little like Chris, in that I can look at a problem from different angles, apply lots of experience to find a solution or — at the very least — know how to find a solution, and have the capacity to do it without necessarily asking for a great deal of help. What I need to learn is how to make common distractions from various groups into learning experiences rather than seeing them as work blockages. When people have questions about databases, I need to guide rather than brusquely answer. When people have questions about X over Y, or the alternatives to Z, I need to outline the gist and provide some basic links to sites with more in-depth answers. The people I work with are not fools. They genuinely want to do a good job and go home knowing they accomplished something, and this is the same goal I have at the end of every day. The question I have now, though, is how to do this without coming across as dismissive or as though I'm "mansplaining"3 something.

Having spent the better part of 8 years working in a classroom, you'd think this would be natural. That said, the teacher-student dynamic doesn't work with peers, nor do I want to have that dynamic with my colleagues. So how does one turn a work-stoppage into a learning opportunity while also meeting all of the arbitrary and constantly shifting deadlines that managers are all to happy to create?


  1. He had a last name, too, but I'll just use his first one here.

  2. Seriously. You don't want to be the person to receive a 3:00am phone call when things go bad … especially if it's with something that isn't technically your responsibility.

  3. I hate this pseudo-word like you wouldn't believe … but it seems to be part of the lexicon, now.

First Time's the Charm?

The last few months have gone by in a blur as both my personal and professional projects have been kicked into overdrive. On the personal side, 10Centuries has seen a remarkable amount of work in the last 90 days which has culminated in the release of version 4.1, an update that's almost as big as the move from v2 to v4 in January of this year. While the platform is not yet mature enough for me to say it's "done", it is coming along nicely with a minimal amount of frustration and just a handful of reported bugs. This gives me hope that the system may be ready for a larger amount of traffic in the near future as more features are completed. Professionally, though … I'm in trouble.

Back in March I was able to finally move out of the classroom and into a full-time development role with the employer. Since then, I've managed to build a lot of the foundation of a new LMS that will be used across the country. However, the project has also massively grown in scope over the same time period. What was originally going to be a relatively simple reporting tool has now become a much more critical reporting tool that can affect the company's finances … as well as a digital textbook.

Visual Studio - ASP and MVC

One of the things that I really like about the challenge of building a big, complicated piece of software for the employer is the fact that I'm creating a big, complicated piece of software. This project will test darn near every skill that I've learned over the last two decades and put them to good use. Every aspect of the code is being hand coded by me and me alone. A wonderful and terrifying prospect all at the same time. I would really much rather be working with another developer. Someone that I could bounce ideas off of. Someone who would disagree and challenge me to create something better.

This isn't to say that I'm working in isolation, though. Nothing could be further from the truth. There are a lot of people working on this project, and it's going to have an impact on the organization this year one way or another. It's my goal to make sure it's a positive one. The code must be solid. The UI must be exceedingly efficient and attractive. The features must be solid.

I'm nervous. With my personal projects, I can make mistakes. With this professional one, there are no second chances.

Complete Conversations

Earlier today I asked a question on App.Net that sparked a conversation that I think was misinterpreted as me bypassing even more of the system1 to allow people to view and interact with posts from people that have blocked them, or they themselves have blocked. While it would be a trivial matter to simply show all posts from all accounts regardless of whether blocks are in place to prevent such a thing or not, this is really not my intention. My question has more to do with completeness than anything else. The next few updates to Nice.Social will be attacking some of the more complex issues that people have been asking for, one of which is a different conversation view that will allow for a TreeView-style layout. Doing so will not be ridiculously difficult so long as conversations are complete entities. It’s when they are incomplete that the code needs to get tricky and begin making assumptions.

Allow me to illustrate.

conversation_a

In the image above, we can see a very basic type of conversation. @person_a starts it off, and @person_b replies. Eventually we see others join in the conversation and it begins to split and evolve into three distinct conversations. This is par for the course on a social network like App.Net, but there’s a problem. @person_d has blocked @person_c from seeing their posts, and @person_c wants to see the conversation view in order to better understand the flow of the conversation. What does @person_c see?

conversation_b

If @person_d blocked @person_c, the first and third conversations will be completely disconnected despite what happened in the second conversation. These two conversations may come from the same conversation thread but, for all intents and purposes, they’re completely different conversations … even if they’re the same subject with the same ultimate root.

What I posited was whether it would be a good idea to show @person_c that the second conversation did indeed happen, as they’d see it in their home timelines, anyway, and link it accordingly.

conversation_c

The difference here is that, rather than seeing every message, we would see that the second conversation includes blocked content. Again, this could be the result of @person_d blocking @person_c or vice versa. It really doesn’t matter. The conversation between @person_a and @person_b that stemmed from @person_d’s comment will appear just as they should in a second tier, and everything would be good to go. This allows for a complete view of a conversation as best as possible. As someone who has worked with databases for over 15 years, data completeness is incredibly important to me. Broken data — especially obscured, broken data — infuriates me to no end as my mind wonders if something is wrong with systems if the human interactions are "out of alignment".

Of course there are a few ways to handle the comment from @person_d. The post can be completely eliminated from the view and a disconnected conversation can be shown, leaving the reader to wonder how a branch replying to @person_d with no discernible parent record came to exist.

Another way to resolve this issue would be to eliminate anything that came after @person_d’s post. The posts from @person_a and @person_b will continue to appear in @person_c’s timeline, but they will not be able to see it in the conversation view as an entire entity.

The third, my preferred way, would be to show that a post (or string of posts) does indeed exist, but collapsed in such a way that it’s understood the material is best ignored. Of course, if a person *really* wanted to know what those messages said, a client could be configured to collect the information without an API key and display it. This, too, would be trivial for someone to add to a personal branch of Nice.Social. Heck, this could be extended to show that posts have been deleted from the conversation, too!

As I said above, when I think about doing a proper conversation view, I think about data completeness. Hiding missing chunks of data is the default answer by many, but strikes me as overly simplistic for the types of complex discussion that can arise on the App.Net platform.

Of course there is the problem of having our posts — posts that we own according to the contract of App.Net — shown to people whom we have revoked the privilege, and this is not something that I will blatantly disregard. There will be no stupid “opt out” function necessary to ensure our posts are not easily read by people we’ve blocked from view. The goal of the exercise was really more about answering the question of completeness and information display.

I’ve said this before and I’ll say it again here. No code has been written to allow a person to access or read posts that have been denied to them. This question was raised as a thought experiment with the hopes of collecting ideas from the people who actually use the network. You are free to examine the source code for Nice to confirm what I’ve said.