From Data to Information

For a lot of people it seems there is no difference between data and information. In reality, the two could not be more different. Data is primarily unstructured or semi-structured elements that exist without meaning or context. Information is structured data that has context and can be used to make decisions. One of the first questions I ask when I begin to design systems that will store data is what sort of information a person wants to get out of the tool later. The answer will fundamentally guide the sorts of data that is collected to ensure that the desired information can be returned and, more importantly, be expanded upon as time goes on. This second part is just as crucial as the first.

Earlier today I found myself reverse-engineering a system designed by a vendor in order to solve a rather serious business problem created by the very same system. The problem amounted to an API that was designed to present a great deal of data rather than actionable information. As a result, teachers in the classroom have found it incredibly difficult to deliver their lessons. What struck me as odd about the software is that an API is generally used to present data in a structure, thereby ensuring it is parsed as information. This particular tool, however, appeared to have a consistent structure but wound up being little more than a data dump of keys and values. It was up to the JavaScript that read the data to determine the context and convert the data into information. Unfortunately, the implementation resulted in a website that would crash older tablets or present just a partial subset of information, which put the onus of "filling in the gaps" onto the teacher who had neither the time nor the resources to do anything of the sort.

So, being the corporate fool, I quietly waded into the mess and started reverse-engineering this system in order to extract the data to populate a database of my own design, then structure the data in a logical manner for the business need, then present it to the teacher in a format they can use. Given that this is a system designed to show textbooks, the lack of structure and clarity in the vendor's system has me questioning whether they understood the actual problem the business needed to solve in the first place.

In the space of six hours, I managed to reverse out the entire system and copy the bulk of the data from the vendor's system into my own, then build a preliminary API structure to return to a browser. Tomorrow's task will be to take the information and turn it into a textbook with the same formatting and features, plus a bunch of other details that should have existed from the first day the system went live. Two days of work on my part and a brand new system can replace nearly two years of development from a high-priced vendor. This sort of turn-around for problem-solving solutions is probably why a lot of senior managers at the day job allow me to break rules from time to time.

While I can generally turn around and solve problems like this through sheer force of will1, how can others avoid making the mistake of leaving data bloated and without form?

It comes down to understanding what a person wants out of the system, that early question I ask before writing the first line of code.

For this example, the goal of the project was to have an API return enough information to dynamically construct a textbook. Leaving the front-end code out of this, what would an effective structure be for a textbook, or a group of textbooks? Let's break down what sort of data makes up the information that is a textbook.

At a minimum we would need:

  • title ⇢ the title of the book
  • chapters ⇢ the sections of the book, allowing for a table of contents to be built
  • pages ⇢ the pages associated with the book, and possibly a chapter object

There is a whole host of meta data that could be included, such as a cover image, authors, publisher, ISBN numbers, MSRP, inventory on hand, search keywords, access permissions, and the like. The sky is really the limit when it comes to metadata, but the receiving software needn't be overloaded with data it never reads. If an API is going to return structured data, most of it should be used. If a complete dataset is only sometimes required, then an API filter show allow an application to request a limited amount of data or the whole shebang. What's nice about going this route is that websites that call the API will not be returning large amounts of data to discard or uselessly store. The less data there is to transfer, the faster everything can operate.

The original API decided to include everything about a digital textbook, including elements that would never be read by the front-end code. Details relating to the source system with index keys and when the chapter or page was last edited in that tertiary system. Details outlining the amount of storage space remaining on the API server, which is of no value unless regularly uploading. Details that appeared to be just random numbers thrown into an array. Details that included the address and contact information for the publisher of the book … which was attached to every page object, resulting in 477 sets of duplicated publisher information for one common textbook. The entire package was 6.68 MB to download, which took an average of 4.1 seconds.

Not cool.

My solution, which is probably not the best solution, stripped a lot of this information out. I put the title, chapters, and pages into their own objects and ensured the basic metadata was in place to show ISBN numbers, and similar details. The entire package now weighs in at 682 KB and can be downloaded in under a quarter second. With some compression on the server, the JSON object can be reduced even further and expanded at the browser. The next step is to replicate the front end with less code and more functionality to aid the teacher in the classroom.

How did this happen?

The people who made the current system are not stupid. I've worked with them on a number of occasions in the past and know the main developers are doing the best they can within the bounds of the client-vendor relationship. One of the problems that I've seen time and again, though, is that people often fail to ask about the ultimate goal of any system. This one started out with a colleague saying "We need a digital textbook system" and then answering a hundred questions around the idea. Looking at the early notes from the project2. Not once did the question of "What does the teacher see?" get asked. Heck, from the meeting notes, that question wasn't asked until 7 months into the project! Well after the database and API were designed.

I'll admit that I tend to look at business problems from the point of view of the person who'll be stuck using the things I create rather than the managers authorizing my wage. This often means that I may not create something that leaders ask for and instead provide the solution their people want, which involves quickly turning data into information and getting the heck out of the way. Being an internal resource means I have a lot more flexibility and access to these people than a vendor might, which gives me an unfair advantage. Fortunately it's one that the right people have appreciated a few times in the past.

When it comes time to solve a business problem, one of the very first questions needs to be "what do you want out of the solution?" Everything else is just window dressing.


  1. Sheer force of will … and a quarter-century of experience writing software. I've made every mistake in the book, plus a bunch that have never been documented. It's important to remember past mistakes and their solutions so that future endeavours can be more successful from the start.

  2. Everything is recorded in JIRA … which is both a good and sad thing. Good because documentation is key. Sad because someone had to put all of this stuff into JIRA.

Misusing Time

Earlier today I was having a conversation with Sumudu about Evernote and the topic of where the data sits came up. A lot of SaaS platforms tend to store people's information on Amazon's Web Service platform or Microsoft's Azure alternative. Evernote is a little different in that the data we upload to them is stored on Google's Cloud Platform. Given my reticence at having Google know too much about me, the question of how comfortable I was with the knowledge that the scans of receipts and medical diagnoses that I've put into Evernote over the years is theoretically available to Google1. As one would expect, I am not all that keen on Google being the place where too much of my data is kept. But what is the alternative? I've tried multiple different note applications since leaving Evernote a few years back, and they're all a collection of compromises in comparison. What am I to do, then? Build my own solution?

A self-hosted Evernote alternative would be an interesting project to work on, and there is already prior work done for such a system that I could continue building on. One of the reasons I stopped working on my own Notes SaaS is because I wanted to focus on getting 10Cv5 ready and out the door. Another was that one of the big reasons I like Evernote is because of the seamless OCR feature that reads through PDFs and most image formats to find words and make them part of the note's metadata, making it possible to search for notes based on the text contained in an image. This is very cool and not at all easy to do with a personal project. There are software libraries out there that will do this, but many of them are either very expensive or just too resource-intensive to use on consumer-grade hardware. Rather than bang my head against the wall by building yet another note-taking app that requires people to compromise, it just makes sense to use Evernote platform … especially considering how I just picked up three Moleskin notebooks that work with the service.

Fifteen minutes later, while working on something for the day job, a solution to the OCR problem flashed through my head. I had devised a way to make OCR work, which would lead to the ability to search for words contained in images and PDFs, and to even handle handwriting to a certain degree. The difficult features that encouraged me to return to Evernote now had valid solutions. I quickly slapped together a proof of concept and, less than 20 minutes later, I had proved that the mechanism was sound. PDFs and high-resolution JPEG images were being "read" in both English and Japanese with minimal effort on my part. Lovely!

But is there a market for an Evernote alternative? What sort of features would I need to have right out of the gate to gain traction? Is this something that people would consider a subscription for — even if it's a one-time, lifetime subscription option — so that it might be possible to dedicate a proper amount of time and resources to the project?

There are a number of threads on various forums including Reddit where people ask about viable alternatives to the green elephant to no avail. A lot of people seem to want an alternative and lament the friction involved with the other note-taking applications.

An A5-sized notebook on the shelf next to me has a lot of handwritten notes and diagrams outlining the requirements of what a proper competitor would need in order to wrest people away from Evernote and the local-file options. There were even scribbles talking about how the server component could be made open source while saving some of the nicer features for the hosted version that I would make available. The ideas seemed reasonable.

Then I glanced at the clock and saw that an entire hour had been spent looking into something that, in reality, would likely be a gross misuse of time. If there was a strong market demand for an Evernote alternative, there would be plenty already. Fact of the matter is that there does not seem to be a large enough group of people (that I'm aware of) that would like yet another text editor on their devices. Rather than invest the time into something that would not help me towards the goal of self-employment by 2022, it would make much more sense to focus on the day job until the end of the shift, then spend some time with 10C and maybe fix a bug or two.

While the goal to be self-employed and provide useful tools to people is noble, it can't be done during regular working hours. A distraction like this may be alright for a couple of minutes, especially when a technical problem is solved, but any more than 5 and it quickly becomes a misuse of time.


  1. Evernote explains that, as a cloud provider, Google is subject to strict security and legal obligations which limit Google’s access to Evernote data. The data put into Evernote belongs to the uploader. Google will not process data for any purpose other than to fulfill contractual obligations such as delivery. Given the fallout that would occur should Google be found in breach of this, it's probably safe to assume that nobody will be doing anything stupid.

If Software Were Music ...

An odd thought crossed my mind the other day1. While listening to some of the better music to come out of the 80s and 90s, I wondered if there was any software from this time period that I'm still actively using. Given the speed at which technologies change and get rewritten, very little of what we see today is more than a couple of years old. Sure, some of the core components of Windows or Oracle might be a decade or two old, but these would be small components of larger projects, like a modern piece of music with a forever-repeating sample from James Brown.

Will any of the software tools that we use today continue to exist and be useful in 30 years?

Blurry Code

Being useful is important. Unless the planet is plunged into some sort of crisis that has wiped out all digitally-stored information everywhere, there are bound to be backups of software that is in use today sitting on an optical disc at the back of someone's closet. Crazy hypotheticals aside, I considered a semi-realistic one: of the software I use today, which ones could realistically continue to be useful until 2050 without any further updates?

Before continuing, I should state that I am fully entrenched in the world of Linux. While I do have a couple of iOS and Android-powered devices in the house, these sealed appliances with known operational lifespans do not count. I'm simply looking at the tools that I use on Linux.

Thinking through the question, I can think of just a handful of applications that are not part of the default installation of Ubuntu Linux that would still be useful in their current form in 2050:

  • Sublime Text, a pretty decent text editor
  • Typora, my favourite Markdown-friendly text editor
  • Gimp, the Gnu IMage Processing application
  • Glances, a command-line tool to see resource usage

Using the base installation of Ubuntu would mean that I could use the file manager, terminal, and a bunch of other built-in applications that make using the system easier. None of today's browsers would work very well with the web in 30 years, though. Grab a copy of Netscape Navigator 3.5 and try to open a site. Most of them will be an absolute mess. A lot of the other tools that I use would likely not work as expected, such as source control programs, API testing utilities, and database clients. A lot of these things would break because of new security protocols in place. Others might break for different reasons. Thinking back on all the support software I would use when deveoping over the years, none of the applications would work today … except maybe SQL Server Management Studio from around 2000, so long as it's connecting to a database that is also 20 years old2.

Given that we've been writing software for well over half a century, at what point will we start seeing applications — that are not on spacecraft — have operational lives stretching into decades? Will people use and enjoy older applications like a person might enjoy older music? I wonder ….


  1. Well, "odd thoughts" cross the mind all the time. This particular one seemed interesting, though.

  2. SQL Server Management Studio that shipped with SQL Server 2000 on a shiny silver CD — like I still have upstairs — would not connect to a SQL Server 2005 instance until later service packs were released. Even then, it's rare for an older SQL Server client to connect to too new a database engine.

Boutique Performance

Not a week goes by where a colleague doesn't complain to me about how slow or sluggish some system or piece of software is. More often than not it's the corporate-mandated tools that are being derided for their sub-optimal use of time and resources, but various websites are also starting to get mentioned more often. Sometimes people ask me how they could make these systems faster, whether it's a problem of not enough memory or CPU power, or why managers consistently choose the slowest software while demanding the fastest employees. There's no answer for the last one. The other two, however, share the same response: it depends.

There's no simple solution to improve the performance of software and any attempt to write about possible things to look for and check would be woefully incomplete1. Instead what I tend to do is nod in agreement and ask a couple of probing questions.

What are you trying to do?

This is always the first question, as most of my colleagues are generally trying to accomplish a relatively common task. My employer isn't trying to launch objects into orbit of far-away planets, cure cancer, or model climate change. We're an education company that has for decades subsisted on a combination of Excel sheets and sheer luck2.

How are you trying to solve this problem?

A lot of times when a person has a software problem, it's because they're doing something "the hard way" and can be taught an alternative method of performing the task3. So by better understanding how a person is approaching the problem, the myriad of options that might be available to solve a problem can be whittled down until there are just one or two good options to consider.

Is this a common task?

Common tasks should be programmatically solved. The role of a person is to be the brain and/or heart of an organisation. The role of a computer is to support that person so they can be as awesome as they want to be4.

Have you considered using ________?

This is the question that I generally try to get to if it's possible because I've found that a lot of the more common software tools that people use on a day-to-day basis are big, bloated, and don't always solve the problems a person might have. Some examples of this would be colleagues who have complained about how sluggish Evernote or OneNote has become after their 10,000th note. I can remember two instances where people did not want to use Word anymore because their computers would crash if both Outlook and Word were open at the same time5. Most people have probably had conversations like this at least once in the last year and it's a great opportunity to recommend tools from small and independent software developers who make a living by providing "boutique solutions".

I enjoy recommending tools like Sublime Text, Typora, Coda 2, Sequel Pro, and Mars Edit to people who need to scratch a specific itch6. It's even better when someone tells me they've bought a license for the software, meaning that the small developer — be they a studio or an independent — earned a little bit for their efforts. This is how software should be made.

There are a lot of reasons for why software might be written by a large team of people. Yet as the world becomes ever more complex, I find it's the smaller software shops that put out the better tools that can help us navigate this complexity with relative ease. Sublime and Typora have both saved me an incredible amount of time by being able to handle large files, or crazy-long line lengths, or just running with such a tiny memory footprint that the commercial memory hogs that run alongside these tools are not at all impacted. One of the many things that I hope to see in the future is a little bit of a return to software practices of old, where the goals are not just about completing the task at hand, but doing so with the most responsible use of resources possible. Applications that make genuine use of multiple CPU cores, reasonable amounts of memory, and simple UI language will always be in demand. So long as the people who make that software can get paid for doing so, there will always be a healthy number of creative problem solvers.


  1. When it comes to tracking down performance issues, sometimes an entire day (or more) needs to be invested to determine exactly what the problem is and what options exist going forward. A person can't simply blame a single component in the hardware or the software as applications are rarely "simple".

  2. This is a slight exaggeration, of course. There are a number of mission-critical systems that use SQL Server and Oracle databases, and our online infrastructure is staggeringly complex to support online lessons across the globe.

  3. The number of lives I've changed over the years after teaching someone how to use a pivot table or VLOOKUP or just Ctrl+D in Excel is by no means inconsequential.

  4. Some people don't want to be awesome at their job, and that's fine. There is still no reason for why someone wouldn't want a computer to do a repetitive task on their behalf (so long as it does not put them out of work).

  5. This turned out to be the result of a domain policy change pushed out by the IT department without anyone's knowledge. Yay, IT!

  6. Yes, I understand that most of these are for macOS. I talk to a lot of people who use Macs. I don't know many people personally who live in the Linux world like I do. Mind you, I will suggest switching to Ubuntu from time to time if someone is complaining about Windows or macOS.

We Shouldn't Be a Fan of Our Work

Last year, after almost a decade of circumventing rules at the day job to help people serve students better, I was moved out of the classroom and into a full-time development role to continue doing what I was doing as an instructor, but without all the cloak and dagger to make it happen. A lot of people were happy with the news, including myself. It meant that I could play a role in making something that colleagues all over the country might find value in, rather than something that just a handful of schools would use without really saying much to upper management about it. Over the last 15 months, though, I've come to dread going to work. I despise checking email. I want to be invisible on Skype all the time or, better yet, just shut the distraction down so that I can make it through the day without wanting to hurl a computer five stories to the pavement below1.

The problem is not with my colleagues. The problem is not with the endless complaints from people who storm into the little space where I do my work. Believe it or not, the problem is not even with the sound of silence from human silos within the organization who refuse to share their knowledge of the home-grown CMS my project must interface with. The problem boils down to a very fundamental issue that will never be resolved so long as I am working for someone else.

The issue is the result of an unsharable vision.

Steve Jobs and the First iPad

Way back in 2010, soon after Steve Jobs walked on stage and showed the world the iPad, I started thinking about how such a device could be used in education. By that time I had been teaching for almost three years and had the hubris to think that I could write software for a tablet that would make education easier for teachers, students, and all the support staff that are required to make a school function. Looking back at the early design sketches, I almost cringe at the naivety on display. The concepts I was dabbling with were far too similar to the way Microsoft approached tablet software in 1999.

Suffice it to say, the sketches went nowhere and I shelved the idea for a few years, revisiting the idea whenever I'd read an article about how tablets were being used in education.

Fast forward to 2013, I was asked to create a special kind of report for a new type of class that was being trialled in the area. Excel was a mainstay at the day job, and every report we gave to students or their sponsor came from this spreadsheet software. Me being me, I was one of the few people responsible for writing all of the reports in the region to ensure that every student and every sponsor would see a consistent message with consistent formatting and consistent quality. This new kind of report, though, needed something that Excel was not particularly good at without a complex series of macros. Instead, I used this opportunity to push through an idea that had been bouncing around in my head since the year before: build a data-collection website that is designed to be finger-friendly so that teachers simply tap-tap-tap their feedback and let the database do the heavy lifting.

Selling the idea was not easy, as people "just wanted an Excel report", but I used a long weekend to prototype the site and build some dummy reports. I presented it to the managers the following week, and they loved it.

This was shortly after my employer had rolled out iPads to all of the schools in a bid to make us seem more efficient and professional. Both counts failed and the project was bleeding money but, again, I had enough hubris to think that I could push through my own agenda while using company resources to solve company problems. Within six months the project had expanded to include several different types of reports, and people were generally happy with the system. A few times the project came close to being shut down when certain members of IT learned of the project2, but there was always just enough pushback from the local schools to keep the project alive.

In 2015, after a redesign of the iPad software teachers were supposed to use in class, a number of people gave up trying to use the tablets in the classroom. We still had to use them to record attendance, lesson goals, homework, and other details, but a large portion of the teaching staff gave up trying to use the tablets beyond the bare minimum. The problem was that the software was poorly designed for the job it was hired to do. The textbook application was nothing more than a frustrating PDF reader that stuttered and crashed every 15 minutes. The pedagogical tool was sluggish, hard to look at, and buried all of the student profile information, making it very difficult to learn more about students — or record updates — before walking into a classroom. Despite transitioning from paper to digital two years beforehand, people were pining for the day when we'd drop the iPads and go back to paper records. The older textbooks that made use of cassette tapes were easier to use and less embarrassing than the iPad software.

So I decided to do something about it.

Again, over a long weekend, I mocked up a new pedagogical system that would work on the tablets while making the system easier to use for teachers and staff. Information would be easier to find and filter. Textbooks would be searchable and come with custom lesson plans to help inexperienced or fatigued teachers. Reports — my specialty — would be built in to the pedagogical system meaning that teachers would spend less time writing them while students and sponsors received more data from them. In the space of four days the demo was ready and I started to show it around to people at the day job.

People loved it. Managers loved it. Even some of the students commented and said that it looked simpler. HQ, however, wouldn't hear of it. There were processes and procedures and hierarchies to obey, and I was bucking the system. They demanded it be shut down, even though there was zero student information in the system. I "conveniently forgot" to do so.

Then, in the fall of 2015, an interesting thing happened. The president of the company caught wind of these projects I was working on and asked to see them. He then asked why I "was being wasted". A week later I was approached with the opportunity to transfer to do software development in the IT department and, in March 2016, it became official. That 4-day design of the pedagogical replacement system is still being worked on and refined today, and people are generally happy with it … except when they aren't.

Back to the Problem

Earlier I said that my problem boils down to a very fundamental issue that will never be resolved so long as I am working for someone else, and this is completely accurate. I have been working quite hard on the problem of creating effective software for use in education for almost five years, the first four years of which was in near isolation where I was able to design and implement features and functions as I saw fit. When I would watch people interact with my software, I would find problems. These were often actions they would do that I never once considered, and I would go back and find a better way to support their goals while also ensuring mine were met. People would come at me with ideas or complaints, and I'd listen and find ways to make the system better for them, again ensuring that my goals for the system were not lost along the way. The way I looked at the tool was very simple: the UI is for the teachers, the printed reports are for the students, the database is for me.

By doing this I was able to create something that teachers actually liked to use. Students were happy. I had a nice database full of numbers from which to quickly answer questions from managers.

Since moving into a role with IT this has changed. People at HQ are accustomed to working with software that fights you every step of the way. Want to record someone's attendance? You'd best have 3 minutes to spare, because what used to be a circle or an X on a piece of paper needs to be infinitely more complex in the name of "security". Want to know what textbook your student will be using after they finish their current book? Go ask one of the school's support staff, because the teaching software will not let you know that information without a fight. This is the state of corporate software in the world, and the previous solutions for the iPad and schools all came from this group of people. My software with it's opposite approach to the same problems is completely alien to the way they think about the job. This isn't a criticism or a disparagement. It's a fact. They're looking at problems as A⇢B⇢C⇢…⇢Z, and I'm looking at problems as A⇢F⇢Z.

It's no wonder there is a great deal more confusion at head office than at the schools. It's no wonder that when members of the various departments in Tokyo report "bugs" in my software, it's because they're not accustomed to software understanding a person's job and performing a bunch of steps transparently on their behalf. From a big picture point of view, I completely understand this. In the heat of the moment when I'm reading that email or new issue on GitHub that has nothing to do with an actual bug and everything to do with making the software harder to use, however …

Flip that Table!

I'm too close to the project. I've invested a great deal of time and effort into making something that is designed to be used by people who really couldn't care less what the corporate interests are. That's why I invested so much time into making the UI for the people who would actually use the software rather than the people making snap decisions months after the initial decisions had already been made. This is why I call people people instead of using the same language as other people in the corporate structure. The whole thing has been designed to serve the people at the bottom of the totem pole. HQ wants things changed to serve their interests3, and I am growing tired of pushing back.

There are, of course, a lot of people that I've worked with over the last year at HQ who do understand the goals of this project and have gone to bat on my behalf more times than I can count. A lot of very smart people with very sharp insights have helped take a rough idea hammered out in 4 days through to the state it's in today. Many of them are just as frustrated with the various emails, non-issues, and Friday 5:30pm deployment cancellation calls as I am. But there's not much that can be done to change this. The vision of the project is simply too foreign at the moment for people, and the sole developer is too angry all the time to cast it in a positive light. I really need to take a step back …

… and another step …

… and one more.

Because it really doesn't make much sense to continue dreading going into work. There is a lot of good about going in, too. I like a lot of my colleagues. I like the ridiculous amount of freedom I have within the organization. I like seeing people use my software without realizing they're doing more in 30 seconds today than they did in 5 minutes last year. It's a great feeling! I just need to stop being so attached to this specific project.


  1. This would be especially bad, given that I'm using my computers at the day job.

  2. these are the same people I work with now

  3. 15 months into the project, mind you …

I Need To Be Chris

Between 2002 and 2007, I worked at a medium-sized company in Canada that was best known for its calendars and other print materials. I started in the warehouse and, over the course of 3 years, moved into different roles that culminated in a position as a software developer and worked with a number of very smart people who taught me a lot about software development, and a lot about how to ask the right questions to find out what people want the software to do, rather than making the wrong assumptions and delivering something that isn't at all what they're looking for. The person I learned from the most, however, was a man named Chris1.

Chris had a rather wide range of knowledge on just about every technical subject, no matter how obscure the tools might have been. His knowledge on certain subjects would often run circles around others, even when it was their area of focus. And, while he most certainly did complain when he was called in to fix somebody else's problem, he tried to make education part of the solution. There really isn't any point being "the only person who knows X" in a company, because that doesn't benefit anybody in the long run2. The guy seemed to know everything he needed and then some, and was honest enough to say "I don't know" when he really didn't know right before investigating whatever needed to be learned so that he wouldn't answer the same question the same way later.

I learned a lot from him in the two years or so we worked together, and would be happy to work with him again if the opportunity arose.

The way Chris handled situations was often incredibly efficient, and it's something I really need to work on myself. The last few weeks at the day job have been incredibly stressful as I attempt to do four very different tasks simultaneously in order to deliver a project that should have started limited trials back in August. I've recently complained that I shouldn't be doing four very different tasks if bugs and enhancements are going to be resolved by arbitrary deadlines, but complaining about reality will rarely resolve the problems one faces.

I've been incredibly fortunate over the last two decades to have worked with a lot of very different technologies and worked in a lot of very different roles. This sort of make me a little like Chris, in that I can look at a problem from different angles, apply lots of experience to find a solution or — at the very least — know how to find a solution, and have the capacity to do it without necessarily asking for a great deal of help. What I need to learn is how to make common distractions from various groups into learning experiences rather than seeing them as work blockages. When people have questions about databases, I need to guide rather than brusquely answer. When people have questions about X over Y, or the alternatives to Z, I need to outline the gist and provide some basic links to sites with more in-depth answers. The people I work with are not fools. They genuinely want to do a good job and go home knowing they accomplished something, and this is the same goal I have at the end of every day. The question I have now, though, is how to do this without coming across as dismissive or as though I'm "mansplaining"3 something.

Having spent the better part of 8 years working in a classroom, you'd think this would be natural. That said, the teacher-student dynamic doesn't work with peers, nor do I want to have that dynamic with my colleagues. So how does one turn a work-stoppage into a learning opportunity while also meeting all of the arbitrary and constantly shifting deadlines that managers are all to happy to create?


  1. He had a last name, too, but I'll just use his first one here.

  2. Seriously. You don't want to be the person to receive a 3:00am phone call when things go bad … especially if it's with something that isn't technically your responsibility.

  3. I hate this pseudo-word like you wouldn't believe … but it seems to be part of the lexicon, now.

Not Thinking Enough?

The human mind is an interesting piece of work. With it we're able to understand the world around us and make decisions that will hopefully affect our future in a positive manner while forever being stuck in the present. A good memory allows us to remember millions of cause and effect chains, enabling us to (hopefully) avoid making mistakes more than once. The vast majority of us will make decisions based on our experiences an unknowable number of times every day in every situation, from navigating the morning time crowds on our way to work to simply enjoying a cup of coffee. We're always thinking, often times on "auto-pilot" … but are we thinking enough?

Over the last few days I've been re-reading some of the books in my library from Jaron Lanier, Nicholas G. Carr, and Evgeny Morozov in a bid to re-examine some of the decisions I've made over the last year with regards to how I interact online as well as look at how I make software. All in all, I'm happy with the decisions I've made with regards to online interactions, but what about the latter?

Writing software is all about considering the results of one's instructions and what future possibilities are made available as a result of those decisions. A decade ago I started down the path of making an intelligent work scheduler at a printing company that went on to save the company millions in labour and materials as a result of the software learning about the employees and assigning them work based on each team's strengths. By using pattern recognition and quantifying people as though they were machines, I was able to solve a problem that was ultimately good for the company but bad for the employee. What started happening is that workers started getting bored soon after I left. There was less variation. Less opportunity to try new things. Less interest in the work.

The last I heard, this software was still being used by the company and has continued to "save money" by taking away people's ability to make mistakes and learn from them. Had I thought through the math a bit more, with a humanistic approach, people might actually enjoy their work more and have more opportunities to develop their skills. Fact of the matter is, I didn't think through the process far enough.

A few years ago I released a web service that measured account activity on App.net, a social platform that — depending on who you talk to — has historically had a spam problem. Discovery of new accounts to follow was not easy, and it was hard to determine who was actually using the service and who was simply cross-posting from Twitter but never interacting with anybody. NiceRank, my interaction-measuring tool, tried to solve this problem by giving every account a score between 0.1 and 4.995. The closer to 5 an account was, the more certain my math was that the account was "human powered".

Over the last two years the algorithms employed have proven to be incredibly efficient. With 86 filters and pattern-recognition cases in place, an account can be identified as human or machine within the space of 3 posts to the service with an error rate of lower than 0.13%. The system didn't care what language a person wrote in, either. It was completely agnostic! There was just one problem with NiceRank, though; a lot of people despised it.

No matter how many times I explained the system's purpose and how it wasn't used to measure popularity (which is a silly metric for any social platform with fewer than 10,000 accounts), it was generally distrusted. In the first six months of the algorithms' existence I had several dozen people demand to be removed from the index. Their requests were fair, and I complied within moments of their posts, but it showed that once again I had failed to think through the mechanism far enough. The consequence that people would see it as a network-specific version of Klout never once crossed my mind.

People-First Coding

Yesterday I was in Tokyo for a number of meetings with senior members of the company to talk about the new software I'm developing to solve some very real business problems. As it's being constructed from the ground up rather than building on what exists elsewhere, the sky is really the limit in terms of what sort of metrics can be captured and stored for later analysis. As a person who loves math, I've already seen a number of areas that could be optimized with some very fancy mechanisms that will save employees several minutes a day in paperwork which, when added up, can result in hours of saved time every month. Teachers will have the opportunity to spend more time looking at students instead of their tablets, and school staff will also benefit from less computer time. A win-win if I've ever seen one.

But what are the possible consequences of these algorithms? What sort of backlash might result in a year or two after a great deal of information has been indirectly collected from every single one of my colleagues? Right now I'm developing the system to save everybody's time and reduce frustration while also making errors harder to record … but where might this lead?

Evgeny Morozov has written extensively about how authoritarian governments are effectively using digital tools — mainly the Internet — to suppress free speech, hone their surveillance techniques, disseminate cutting-edge propaganda, and pacify their populations. Jaron Lanier has shared his insights as to how massive corporations (mainly Facebook and Google) are using populations as a means to provide endless sums of data in order to feed their own algorithms and ultimately benefit their customers who, oddly enough, are not the people who necessarily use these services. Could my employer at some point in the future require that the tools I create today be used in a similar fashion as a means to monitor my colleagues?

Do I have the responsibility, as the primary developer of these systems, to make any possible future surveillance of a group or individual as difficult as possible? When it comes to 10Centuries, I have gone out of my way to not monitor what people are doing or even how much of the service they're using. It's absolutely none of my business what people write or share on here. My business is making sure the service is available and responsive. This aside, as I've explained above, my own short history has shown on two occasions that my best intentions were not written as a human-first solution and ultimately resulted in a negative response. How can I not repeat the same mistake without intentionally hobbling any system I create?

Am I not thinking enough? Or am I thinking too much?

The Next, Big Mobile OS Won't Come From America

Andrew Cunningham recently wrote a pretty good opinion piece lamenting the lack of a "viable, third, mobile OS". He starts by talking about BlackBerry's focus on making Android devices, then moves on to talk about Microsoft's move to make their software and services available on both iOS and Android, which all but eliminates a lot of the compelling reasons a person might choose a phone running Windows Mobile — as it's called again — over one of the more common operating systems.

smartphones.jpg

All in all, it's a well thought out piece that captures the predicament that consumers who want neither Google nor Apple's software find themselves in. The majority of people who have smartphones are using software that was developed — for the most part — in California. Most of the non-Californian operating systems (BlackBerry OS, Windows Mobile, FirefoxOS, WebOS, and others) suffer from the same problem: when they were out, the tech press lamented the lack of apps and focused on asinine hiccups that were passed off as failures then, when the OS shrank or otherwise disappeared, the very same tech press lamented the loss of yet another mobile OS and further market consolidation.

Windows Mobile is still around, and alternatives like Ubuntu Touch are available for those who are willing to put in the effort of making the fledgling tool work on hardware that was designed to run some flavour of Android, but I don't see either of these growing to become the next iOS or Android. Even if Canonical invested millions in Ubuntu Touch and rolled out a full range of phones across North America, Europe, and Asia, they would not gain any sizeable foothold. The media would just lament the lack of apps and roll their eyes whine about how a phone as elegant and slick as the Meizu MX4 is not the "iPhone Killer" they're looking for.

The Meizu mx4 Ubuntu Edition

In order for any new mobile operating system to take away some of Google or Apple's market share in this space, it needs to be really big right out of the gate … and there's only one way that's going to happen: the OS is going to come from Asia.

Most of the consumer operating systems that are in use around the world today got their start in the United States, but this doesn't mean that a development team outside the US can't have just as much success, particularly if there are juicy incentives put in front of them. Samsung has their Android/iOS/WebOS mashup clone called Tizen but, with many South Koreans opting to buy something with an American-made OS, there's no way Tizen will gain the momentum it needs to build any sizeable audience. Japanese hardware manufacturers have all but given up trying to make an operating system for their phones with the demise of the "feature phone" market because software is simply not one of their strengths, and Google is practically giving away the work of their developers to get people on Android.

There is another market, though. One where a billion people can play a huge role on the software market of the future. China.

Over the next five years, I fully expect to see a viable, third mobile operating system come out of this country full of startups. The English-speaking world will complain about perceived security threats and "trusting the enemy" but, considering how 95% of all cell phones are manufactured in China already, what's the difference between hardware and software? A lot of things can be put into hardware without immediate detection. Heck, it's because of people's concern about Chinese software that I fully expect a company from there to invest heavily in making something at least as solid and reliable as iOS, if not moreso. Package a great operating system with a low-cost, decent phone, and the Chinese market will lap it up. Give it two years or so, and a huge market could emerge from right under our noses.

Of course, there are a lot of hypotheticals in this situation, but it's completely within the realm of possibility. There's very little chance a company from North America or Europe could create something today that would rival Android or iOS. China, however, has the massive market and national pride that one would need to effectively market an "iPhone Killer" … and I fully expect to see one before long.

First Post of 2016

So here it is, 2016. Hard to believe just how quickly the last year flew by. One moment it was March … and now this. A great deal has happened (and not happened) over the last year, while 2016 is shaping up to be a very interesting year. One of the many projects that I hope works out is the updated 10Centuries platform. The data migration from the old system to this new one will be a little tricky, but no an impossible challenge. One of the many things that I'll need to do is created a semi-automated system to allow people to migrate their data. That said, the themes will be severely limited at first.

As of this writing, there are just two accounts (and 8 websites) being powered by the v4 software. By this time next year, I hope it'll be a lot more.

Re-Imagining a Wiki

Six months ago one of my many managers came to me with a problem. We have literally thousands of useful information documents telling teachers and staff about our textbooks and other products, but no easy way to share them within the organisation. Information sheets, alternate lesson plans, internal memos, training manuals, handouts, grammar explanations, departmental reports, skills tests, and just about anything else that might answer a question could be housed in any one of four line management systems, most of which employees don't even know exist let alone have access to, in a dusty binder that most people have forgotten about, or archived in somebody's email application. For an organisation that excels at teaching people how to communicate, we're not very good at practising what we preach!

Something needed to be done, and it had to be done right the first time if the company is to make information more readily available to everyone. The last thing we nee is a fifth online management system that few people know about and fewer still use.

My first thought was a blog with a customised, magazine-styled theme. This would allow for rich pages to be created containing lots of useful information and links elsewhere. After some thought I scrapped the idea because it meant leaving the content creation in the hands of a few. I wanted to see something that would allow any employee regardless of position to add or edit the information contained in the system. What good is a repository of knowledge if nobody's allowed to add to it? Sure, everybody could be given a login and a complicated author-editor hierarchy could naturally form … but that seemed unnecessarily complex. No, we need something simple. We need a Wiki!

When Sir Tim Berners-Lee — before he was knighted — envisioned the Internet all those years ago, he saw a series of web pages that could be updated by anyone from anywhere at any time. The first web browser wasn't just a browser, but an editor, too! Other scientists were encouraged to add, edit, ad update articles so that the information would become better and more correct1 with every revision. The modern web is almost the opposite of this early concept, but I really admire Sir Berners-Lee's ultimate vision. A Wiki is essentially the epitome of this early version of the Internet, and an altruistic effort by a team of front-line and back-end employees who give a darn about the people they work with can amass a remarkable amount of data in an open and easily-accessible repository. There's just one problem …

Have you used Wiki software lately? Yes, it's quite capable, but it's also quite complex for many people. Adding to the challenge was the number of extra data fields and customisations I'd have to do to make the system friendlier for an educational company and then there was the problem of design! After some analysis I estimated I would need about 200 hours to customise the MediaWiki software to get it to where I needed it to be and, because my estimates are always off by a large margin, I added a zero to the end. Two thousand hours and I could certainly have a tool modified and designed to work well within the organisation while taking into account all the little things that we do internally when sorting and distributing data.

There were problems with this, though. Search wouldn't take into account some of the meta fields. More than this, the search code was slow and would need a lot of retooling to return data the way I wanted to see it. Then there was the externally-accessible API limitation. I'd be re-writing more than half the system to get what I wanted: a dead simple, online information system with a RESTful API that could return JSON data to an upcoming app and handle OAuth along with other nice things.

As you can see, not only do I overthink problems, I can talk myself into over-engineering solutions, too!

So what was the solution, you ask? A heavily-modified version of 10Centuries with a search mechanism that is insanely complex, but incredibly fast and reliable. It's multi-lingual right from the get-go and employees have one textbox and a button when they first load the site. Just as with any modern search engine, type in some words or phrases and hit enter. Let the computer do the hard work. Search results are sorted by relevance and documents of various types can be displayed with useful preview functions.

It just works.

Adding information is a click away, as is editing the content on screen. Changes are reflected immediately. Heck, the tool works so well that I may just fork it and make my won private Evernote. There's nothing preventing this from being an Evernote replacement save for the lack of OCR integration and apps on all platforms.

Oh … and ambition. I have no plans at this time to offer a replacement for Evernote.

All in all, I'm very happy with my custom Wiki. It hasn't rolled out across the company, so there are bound to be bugs and edge cases I never considered. That said, this is perhaps the smarted software I have ever written to be used in an educational organisation.


  1. More complete, to use an accounting term I really enjoy.