Misusing Time

Earlier today I was having a conversation with Sumudu about Evernote and the topic of where the data sits came up. A lot of SaaS platforms tend to store people's information on Amazon's Web Service platform or Microsoft's Azure alternative. Evernote is a little different in that the data we upload to them is stored on Google's Cloud Platform. Given my reticence at having Google know too much about me, the question of how comfortable I was with the knowledge that the scans of receipts and medical diagnoses that I've put into Evernote over the years is theoretically available to Google1. As one would expect, I am not all that keen on Google being the place where too much of my data is kept. But what is the alternative? I've tried multiple different note applications since leaving Evernote a few years back, and they're all a collection of compromises in comparison. What am I to do, then? Build my own solution?

A self-hosted Evernote alternative would be an interesting project to work on, and there is already prior work done for such a system that I could continue building on. One of the reasons I stopped working on my own Notes SaaS is because I wanted to focus on getting 10Cv5 ready and out the door. Another was that one of the big reasons I like Evernote is because of the seamless OCR feature that reads through PDFs and most image formats to find words and make them part of the note's metadata, making it possible to search for notes based on the text contained in an image. This is very cool and not at all easy to do with a personal project. There are software libraries out there that will do this, but many of them are either very expensive or just too resource-intensive to use on consumer-grade hardware. Rather than bang my head against the wall by building yet another note-taking app that requires people to compromise, it just makes sense to use Evernote platform … especially considering how I just picked up three Moleskin notebooks that work with the service.

Fifteen minutes later, while working on something for the day job, a solution to the OCR problem flashed through my head. I had devised a way to make OCR work, which would lead to the ability to search for words contained in images and PDFs, and to even handle handwriting to a certain degree. The difficult features that encouraged me to return to Evernote now had valid solutions. I quickly slapped together a proof of concept and, less than 20 minutes later, I had proved that the mechanism was sound. PDFs and high-resolution JPEG images were being "read" in both English and Japanese with minimal effort on my part. Lovely!

But is there a market for an Evernote alternative? What sort of features would I need to have right out of the gate to gain traction? Is this something that people would consider a subscription for — even if it's a one-time, lifetime subscription option — so that it might be possible to dedicate a proper amount of time and resources to the project?

There are a number of threads on various forums including Reddit where people ask about viable alternatives to the green elephant to no avail. A lot of people seem to want an alternative and lament the friction involved with the other note-taking applications.

An A5-sized notebook on the shelf next to me has a lot of handwritten notes and diagrams outlining the requirements of what a proper competitor would need in order to wrest people away from Evernote and the local-file options. There were even scribbles talking about how the server component could be made open source while saving some of the nicer features for the hosted version that I would make available. The ideas seemed reasonable.

Then I glanced at the clock and saw that an entire hour had been spent looking into something that, in reality, would likely be a gross misuse of time. If there was a strong market demand for an Evernote alternative, there would be plenty already. Fact of the matter is that there does not seem to be a large enough group of people (that I'm aware of) that would like yet another text editor on their devices. Rather than invest the time into something that would not help me towards the goal of self-employment by 2022, it would make much more sense to focus on the day job until the end of the shift, then spend some time with 10C and maybe fix a bug or two.

While the goal to be self-employed and provide useful tools to people is noble, it can't be done during regular working hours. A distraction like this may be alright for a couple of minutes, especially when a technical problem is solved, but any more than 5 and it quickly becomes a misuse of time.

  1. Evernote explains that, as a cloud provider, Google is subject to strict security and legal obligations which limit Google’s access to Evernote data. The data put into Evernote belongs to the uploader. Google will not process data for any purpose other than to fulfill contractual obligations such as delivery. Given the fallout that would occur should Google be found in breach of this, it's probably safe to assume that nobody will be doing anything stupid.

Getting Back Into Evernote

When I decided to put macOS back on the notebook a week or so back to remove some of the friction I was consistently rubbing up against when trying to work, I decided to also give Evernote another try. There was a time when the note application was my go-to resource. It was used extensively between 2011 and 2013 on the iPod Touches I had and, after getting an iPhone, the application became even more useful thanks to all the geo-tagging that could be done with the notes. Evernote was so much a part of my day-to-day life that the prototype version of 10Centuries was an Evernote-based blogging tool1. However, some time around 2015 the company started to change. The applications were slow, bloated, and too difficult to use on the phone. The desktop application did the job, but wasn't as good as previous versions as a result of a bunch of extra "chrome" that was added at the bottom of every note. By the autumn of that year, I uninstalled the software and let my premium account expire. The system just wasn't for me anymore.

The next couple of years weren't particularly great for digital note-taking. I tried OneNote and an array of alternatives that were all trying to be like Evernote, but none could solve the problems that I actually had. What I need from a note application is not at all revolutionary. In fact, Evernote did everything I needed and then some … but the applications were just hard to use. In 2017 I decided to give the service another try, installed the application on my phone, and almost instantly regretted it. My inbox was hit with a slew of spam from Evernote! They wanted to welcome me back. They wanted to offer a discount for a year of premium service. They wanted to let me know about new features. They wanted so much for me to centre my entire life around their service, which is not how notes work. Less than two hours after installing the application, I uninstalled it and added Evernote.com to the mail filter, sending everything automatically to trash.

Not having a decent digital note system is not an option anymore, though. An entire 150-page A5 book is filled with hand-written notes every five weeks for all the things I'm doing at the day job. Another one for 10C sees 15~20 pages written every week with ideas, bug analyses, data structures, and more. All of these things can remain in paper form and still be quickly referenced, but this still works out to over 1400 pages that I'm hand-writing every year just for development projects, not to mention client work and the various essays I've been working on that are in various states of completion.

Plain text files have been used, but don't easily support attachments or meta-data. Word processors like Microsoft's Word or LibreOffice's Write are overkill and do not have decent PDF OCR and indexing built-in. Try as I might, the best tool for the job since 2011 has been — in my mind — Evernote. So here I am with a trio of these …

Evernote's Squared Smart Notebook

This third time around has actually been quite positive. I'm not being inundated with spam. The applications on the tablet and desktop are actually pretty decent. The advertisements — for the moment — are minimal. I've even started scanning receipts and other documents into the service again, which is something I once did religiously in order to keep track of important things that needed to be quickly searchable later. For the most part, I'm enjoying the reduced friction that comes with using a tool that is wholly aligned with the things I need from a note management service.

My only hope is that the service continues to leave me alone while being dependable going forward. I'll get the premium account. I'll get the notebooks that make it easier to have hand-written notes and sketches get processed. I'll even learn to use Penultimate on the tablet with a stylus2. So long as the marketing engine doesn't get in the way of the service, it might just remain part of my digital toolbox for the foreseeable future.

  1. Quite literally. The only way to publish a post with Noteworthy is via Evernote.

  2. Much to Steve Jobs' chagrin, of course.

So Many Blog Posts ...

I still find it amazing that there are 2,644 blog posts on my personal site. If one includes all the other blogs that I have written for (on 10C and off), then there would be over 3000 posts written since October 2006. While this isn't anywhere near 1 post per day, it's a heck of a lot more than most other people I know.

Mind you, I probably would have written more posts by now if I still used WordPress. I created Noteworthy (and later 10C) because I was tired of dealing with WP issues. What resulted was me investing thousands of hours into a personal project that has taken time away from blogging 🙄

Of course, one of the bigger reasons I created Noteworthy was because I wanted to have a better way to write blog posts. Evernote was a pretty decent tool back in 2010~2012, and their iOS client rocked. So I used to write blog posts in Evernote on iOS (an iPod Touch) while out and about for the day job. When I got home, I'd copy from Evernote into WordPress, add some tags, then hit publish. Noteworthy cut that process out by syncing with the Evernote API and pulling in any new notes that were in a given notebook (or removing posts that were removed from the notebook).

Those were simpler times …

Abandoning Today for Yesterday

Several months ago when starting out on a new web application to solve some of the issues I've faced with Evernote, I made the decision that the tool would need to be able to work offline despite living solely in a browser. This would be my first attempt at making such a tool, and it looked like a great opportunity to use backbone.js and the backbone-localStorage.js extension. These two tools have received a great deal of positive press, and it's always good to learn how to use current web packages. Unfortunately, after several months of building the interface everyone will use, I've come to the conclusion that backbone is not the library I'm looking for.

FiloNote … and Post Dated

The problem that I've run into is one of abundance. There is just too much data being sent back to the application server when the web browser tries to synchronise itself. I could certainly fork backbone in order to pare down the data as much as possible … but this would take more time than it would to go with my standard REST API tools that have been used on 10Centuries and a number of other web projects consistently well over the last five years. Such a shame.

Of course, the argument will probably be made that I'm doing it wrong and should enable a setting that I am not aware of and could not find in the documentation. This may very well be the case, too. But time is money. A minimum viable product needs to be on the market in February1, and I barely have the system operational now.

With more time to dedicate to the project, I'm certain backbone could be made to work just the way I'd like. Unfortunately that will have to wait. Maybe with a little help from IndieGoGo investors, the time and expertise could be obtained …

  1. Further details as to why there is a February deadline will be revealed later

Evernote on iOS7 Needs Some Work

Apple today released the seventh version of their mobile operating system, iOS. All in all the upgrade went without a hitch, and I was gleefully swiping around the screens like a n00b less than 25 minutes after clicking the "Download" button. A lot of the applications that I use on a regular basis have been updated for the new design aesthetic of iOS7, and design is certainly the key focus a lot of developers have when it comes to putting out new versions. I couldn't be happier. That said,t here is one app that seems to have consistently gone from good to bad to worse over the course of three years: Evernote.

The Latest Evernote in iOS7

Evernote in iOS7 is not really usable when in portrait mode. This is going to cause some issues when I try to write ridiculously long blog posts. Heck, one of the reasons 10Centuries was made was so that I could write long blog posts in Evernote on my iDevice1 and have them appear automagically on my personal site without having to go through some other text editor.

I hope there will be a way to hide the formatting bar as well as the unnecessary "New Note" header in the near future. Unfortunately, looking at the amount of eye candy that's been consistently added to Evernote over the last few releases, I don't think this is going to happen.

Push Publishing?

Earlier today I discovered, much to my horror, that a bug in another piece of software that runs on my web servers has sapped close to 80% of the performance from every site. The glitch was one of those asinine coding issues that would have been caught in testing had I actually tested the software before deploying it. Unfortunately, when the code was written last Monday, I was juggling the full time job, lunch, and the hope to get a nice little side project launched. Suffice it to say the updated code did not solve the problem it was meant to address, and instead introduced some other problems that have affected every site I take care of. How embarrassing.

The problem has since been resolved but, while thinking about ways to mitigate this sort of problem from occurring again in the future, I started wondering how difficult it would be to offload all of the heavy lifting required by some of my sites to an auxiliary server. After some quick calculations and code considerations, it doesn't seem that difficult at all. I would need to update my Midori Web Presentation1 layer to have a Primary/Secondary2 switch, allowing for a great deal of code re-use without maintaining two separate packages, but the aggressive caching system that's in place would have no trouble accepting data computed elsewhere and presenting it.

Wonderful! Problem solved. All I need to do now is build it.

Why Not Offload It All?

Soon after working out what would need to be done to my code for this new Primary/Secondary model to become a reality, I started wondering whether this sort of system could also be implemented on Noteworthy. As it stands, Noteworthy will pull data from an Evernote account, store it in a MySQL database, then build the HTML that is presented when a person comes to visit the site exactly one time. The only time a post is re-created is when the site's design is updated, or when an update has been made to the post. There's really no reason at all to continually hammer the database when it's not required. As a result of this, Noteworthy-powered sites have directories of static content sitting on the server that is presented to visitors when they request a specific item.

If I take Evernote out of the equation, could Noteworthy be slimmed down to just a handful of files that exist only to facilitate theming and data collection? Both the short and long answers point to yes. The question, though, involves the writing mechanism.

Over the next few weeks I will be pulling all of my data off American servers3. This means that using Evernote to write content will no longer make sense. Instead, writing content to an actual file would be a logical and simple solution. But this raises some questions, too. How will files go from a desktop or notebook to the primary server before being sent off to the web server? Dropbox is out. Using a service like ownCloud looks interesting, but there are (currently) very few applications that work with it. In order for a system like this to really suit my needs, I need a to find a workflow that will allow me to write on either OS X or iOS and synchronise files between the two using my own servers.

Not having a replacement for Dropbox won't preclude the creation of a web tool that crunches local files and pushes off static assets to a web server where they're delivered to a waiting audience, but it sure would be nice.

What Would It Look Like?

This weekend I rebuilt my home network server which has freed up quite a bit of processing power. As a result of this extra power, it's now handling some of the heavier work that was once performed by my web servers. Using this server as part of my process seems only logical, and I'm sure many others would enjoy doing the same4. Here's how I see the process working.

  • write something in a word processor of some sort

  • put the finished file in a watched folder

  • the home server gets the file and processes it

  • completed files are uploaded, including any images, files, and other assets

  • the web server displays the new content

    Five steps in total. There are some trade-offs of not using a database, though. Search would be quite difficult, for one. That said, how many people actually search for content on a non-help website? In the last 6 months, the search field on this site has been used a grand total of 8,492 times. Most of which by scrapers.

    Doesn't seem worth it.

    I plan on building a preliminary version of this throughout the week, and will try to release the first version this month.

  • Postach.io

    A number of people have been asking for my opinion on Postach.io, a blogging tool that lets people blog directly from their Evernote accounts. The common question is whether I'm worried or not about this service as it does pretty much the same thing as Noteworthy and 10Centuries, but in a much nicer fashion. To be completely honest, I am not worried at all. The more solutions people have the better it is for everyone. More than this, though, is the fact that it would be stupid to worry about other people offering a similar service as we're not even in the same market.

    The <a target=Postach.io Website" />

    Postach.io and 10Centuries look a lot alike. Both services allow people to write posts in Evernote and have them get pushed out to a public facing website. Both services have options for themes, custom domains, code editing, and a host of other useful features. Both services put a priority on page load speeds and ease of use for everyone involved. Truth be told, I think the group at Postach.io has done an incredible job of putting this sort of site together. Having worked on Noteworthy since 2011, I can tell you that it's no small feat to handle all the different note structures Evernote might store and turn them into a readable website.

    I look forward to seeing what the team at Postach.io is able to accomplish in the coming months. They clearly have far more time to work on their baby than I have to work on mine. They also have a much easier sales pitch: write something in Evernote and see it online. They are certainly succeeding in their endeavour.

    A Fear of Success?

    Over the last few months I've been struggling with, what I call, the Demons of Self-Doubt. These apparitions are completely contained within our head, but often visit us whenever we try to accomplish good things. Phantom thoughts run through my head talking trash about everything that I'm trying to accomplish. Negative comments such as Who would ever trust you with their data for 1,000 years? and What makes you think you're better suited to host people's information than Google, Amazon, or Facebook? and You bash WordPress for being what it is, yet your software does less … much less. Perfectly valid questions and statements when it comes to the 10Centuries project, but it's not only this project that warrants visits from these chimeras1; they come whenever there is something new or different to be created. Why?

    I've been thinking about this a great deal over the last few weeks as the release date for 10Centuries inches ever closer. Over the next few weeks I'll be making the service available to (almost) anybody who would like to sign up and give me money, as well as submitting the service into the Evernote DevCup competition. I have every intention of raising awareness for the issues that the system tries to resolve, and seriously look forward to seeing what sort of reception the service gets from people. I've spoken with a number of clients, colleagues, and peers across the web and had a lot of positive feedback … and blank stares from people who are not really technologically adept2.

    The Competition

    There has been a lot of competition in the area of Evernote-connected blogging solutions over the last few months, with EverBlog.us and the newer postach.io being the two receiving the greatest amount of attention. WordPress has long had a plugin that will read from an Evernote RSS feed of your shared notebook and import the content, much like any generic content scraper. Then there's Noteworthy, which seems like the only self-hosted solution for people to run on their own servers. I don't have any exact numbers for how many people are running Noteworthy, as it's really none of my business, but the number isn't that high. I am incredibly interested to know what sort of reception EverBlog and Postach.io are receiving, though. Is there really a market for this? Is there nobody like me who wants to write and post things right from Evernote? I just don't know.

    Is this lack of information feeding my spectres?

    What Competition?

    One of the follow up questions that I immediately fire back at my demons is along the lines of Who cares if nobody uses 10Centuries or any of my tools. Having an eternal presence online is a serious issue for some people, and I want to offer a solution … even if it's not the one that's adopted by many. There are some tools and services available that will allow people to download all of their content to keep offline, and that's great. I'm not trying to solve that problem, though.

    The ultimate goal of 10Centuries, and Noteworthy in a round-about way, is to give people a little comfort in the knowledge that when it's our turn to meet our maker, the information that we put online for the world to read does not disappear into oblivion. More than this, 10Centuries will give people a single location from where they can search everything they've posted online.

    None of the other services are even coming close to offering this. None of them even sound like they'll be around for 10 years, let alone 10 centuries. EverBlog and Postach.io look like great solutions for people who want a quick way to put content online with a piece of software they already know and love. My tools may have a lot of similarities to the competition right now, but we're going after very different problems.

    So why are the demons even speaking to me? What purpose do they serve?

    I don't have the answer, but I wish they would shut up from time to time; they're the least productive figment of my imagination. My mother often said the old adage if you don't have anything nice to say, don't say anything at all. The spectres should take this to heart.

    It Finally Happened

    After many years of being mostly unscathed from hack attempts, Evernote has finally been hit hard enough that they were forced to reset everybody's passwords in an attempt to block unauthorised access to accounts. The issue has apparently been going on for a couple of days and the severity of the problem only came to light this morning1. Malicious users managed to get people's contact information and an encrypted version of the passwords used to access the system. Although we've been assured that the passwords were both salted and hashed2, Evernote wanted everybody to update their credentials immediately to ensure none of our data would be accessed. This was good thinking, though poorly implemented.

    Here's the initial email that I woke up to this morning in it's full, unadulterated form:

    My information had been updated within minutes of seeing this message, but that still meant that my account could have been hit within the six hours after the emails were sent or, worse, before Evernote even knew there was a problem.

    Are You Sure I'm Me?

    Resetting the password was a straightforward affair. I went to the Evernote site, hit the sign in button, entered my username and old password, then entered a new one. All in all the entire process was done in less than a minute, with the most complex part of the process being the creation of a new password with 1Password's generator3. There was just one question I was left with afterwards: how did Evernote know I was me?

    The attackers would have had my email address and login name. They would have had my password. What would stop them from signing in as me and downloading the several gigabytes of information I've uploaded over the last 16-odd months? Nothing.

    I hope that there was some sort of background check involving an IP address or two. I like that the Evernote team was smart enough to kill all of the private API keys, and I'm particularly impressed with Noteworthy as it noticed that the API key was no longer being accepted and stopped hitting the API server. But the lack of confirmation regarding my identity leaves me wondering whether some other people in the future might find their abandoned accounts had been compromised4. People who use the same password across multiple services will probably want to change more than just their Evernote password, too.

    With the string of hacks that have been reported in the news over the last few months, I hope this will force people to seriously look at their password habits. No web service is 100% safe, but with a little foresight we can ensure our entire web presence is not left wide open when a single service is cracked wide open.

    Markdown vs. Evernote

    Some people have asked me why I'm so stuck on Evernote when there are other note-taking options out there, with the most common alternative being plain text files formatted with Markdown. At one time I might have opted for the more geeky solution with hundreds of editing options, but I've decided to foolishly invest all of my note-taking activities in just one tool. Why?

    WYSIWYG Is Human-Friendly, and Mom-Approved

    For a decade and a half I was more than happy to store notes as plain text, Excel, and images organised in a series of directories. If I needed to find something I would hit F3 in Windows to pull up a search dialog, enter in the criteria, then wait for one or two minutes while the operating system read through tens of thousands of files to find some matches. This process became untenable after a few years and I started to keep all of my notes on paper with images, Word and Excel files, and other supporting files referenced by name in the hand-written notes. Unix-based systems are much faster when it comes to searching through thousands of files for matching strings but, as someone who was heavily invested in Microsoft tools for many years, I opted to not have a dual-system setup.

    More than this, though, Markdown just seemed like a huge pain in the ass to use.

    Dashes, asterisks, brackets, exponents, division, multiplication, addition, and subtraction … all to make something that should be easy look pretty? What a waste of time!

    That's what I used to think.

    Now when I look back at a number of my notes from a decade ago I'm wondering where the important bits of information are. Where's the emphasis? Where's the end of a comment and the start of a commentary? Sure, I still have many of the digital notes from that time with all of the same information, but the context has all changed. People change over time. Aside from food and coffee, what were my priorities in 20021

    ? Does this mean that a WYSIWYG is absolutely necessary to preserve pointers and other key elements when writing notes? Hardly … it just makes it easier.

    Why Evernote?

    One of the biggest reasons I prefer Evernote to Markdown is the ability to easily add extra media content to notes. Images, audio files, PDFs, and other items can be dropped in to Evernote with a simple motion and archived for all eternity. More than this, I can store extra bits of data in meta fields such as GPS coordinates, author information and, more importantly, tags. Tags have single-handedly changed the way I organize and search for information. Although we can add tags to individual note files in the form of metadata, it's not quite as simple or elegant as I'd like. Evernote gets almost everything right the first time.

    It's far from perfect, though.

    Some of the things that Markdown does well involves fonts. Evernote's font handling is highly suspect. You don't know when a font has been overridden with something else or when your notes contain a great deal of hidden CSS that can make an absolute mess of things across platforms. The more that's hidden in a note, the more difficult it is to ensure forward compatibility. In a perfect world, we would have the option to view the raw note and clean things up just the way we like … but this would make Evernote a bit too complicated for some people.

    At the end of the day it really comes down to personal preference. I prefer the ease of Evernote over the raw simplicity of Markdown today. Perhaps this will change one day in the future if there is a wonderful tool that will allow me to write in Markdown and synchronize those notes across all of my devices with an easy way of attaching tags, images, movies, and other items to a single file. Lots can change in a few years, after all.