Sugar Memory

Six weeks ago I started the "No, Thank You" diet where, essentially, I turn away all the yummy stuff that I so often want to eat and keep to a strict regiment of healthy meals and semi-decent exercise. Unfortunately the exercise bit has not been very decent, but I have been able to drastically reduce the amount of junk food and between-meal snacks that I would once enjoy. It's a good feeling to look at the clock around noon and realize that you're not particularly hungry, and it's an even better feeling to leave work at 7:00pm knowing that — aside from a homemade lunch — you haven't had anything at all while at the office, and dinner sounds like a wonderful reprieve from hunger. There's just one little problem that takes a while to overcome, and that's sugar memory.

Strawberry Shortcake Donut.jpg

Like billions of people all over the world, I love the taste of food. I love the feel of the food as my tongue savours every natural flavour and spice. And I especially like the chemical release that goes on in the brain while consuming meals. The downside of all this is that, when the subconscious wants to encourage a person to break their strict daily menu, it conjures up memories of wonderful food experiences. Donuts in particular have been on my mind, and a few times I've come very close to caving in. Rather than buy them, though, I'll walk past the Mr. Donut near my office and simply enjoy the smell. Not only does this save money, but it keeps the mind occupied as each scent is decoded to determine which donuts were baked most recently. It's not a "fun" activity, per se, but it beats the feeling of regret one has after taking the first bite of the forbidden food ... and the second bite ... and the third ....

I miss donuts. I miss cookies, too. Soft, chocolate chip cookies that are — unfortunately — so full of ingredients I can't pronounce that they're still soft five years past their expiration date. Ice cream, too. It's not often I have cold treats but, when I do, it's often vanilla ice cream. But that's taboo. All of the foods that take up residence in the sugar memory are taboo!

It's not at all fun.

Six weeks of dieting and what do I have to show for it? A drop of 4.3kg. That's nothing, really. I'm still 20kg away from my goal.

Hopefully the summer heat will break soon and I'll be able to head out for a quick power walk during lunch breaks, and have another in the evening as the sun sets. Summer in Asia is not particularly friendly to people who choose to be outside during daylight hours, and I'm not particularly impressed with how quickly I sweat through my clothes after 8:30am.

As for the dreams of sugary sweets and forbidden pleasures? I'm really hoping they disappear before too long. Imagining the taste and texture of a lovely donut can be terribly distracting while working.

Six Years of Puppiness

Six years ago today, Nozomi went from being an over-excited puppy without a name to an over-excited puppy with a name, home, and family. She has certainly left her indelible mark on the people and places around her, and I'd like to hope that it's a positive one for the most part. I know I can go on and on about this puppy¹ like an over-excited parent who thinks they're the luckiest person in the world, but so would anybody else had they been fortunate enough to bring this bundle of happiness home with them all those years ago.

Happy Nozomi on a Park Bench This Morning

Of all the souls I've encountered throughout my life, hers has been the most kindred.

  1. She'll always be a puppy to me. Even when she's 15 and grey.

Just Can't Shake It

The last few weeks of long hours have taken their toll, and I've been unable to write anything substantial as a result. If the current pace at work continues, I'll be on the way to burnout before the end of September.

Writer's Block

It wasn't too long ago when I would write three blog posts a day. One for this site, another for The Carbon Blog¹, and a third for a larger site under a pseudonym. This would often be done during the short breaks throughout the day, and there was always something to write about. Now I tend to be a little more choosy about what I write, and where it can be published. That said, when one is struggling to get ideas out in some readable fashion, there's nothing to be choosy about. Nothing gets published.

Hopefully this blah will dissipate as the summer heat gives way to typhoons and cooler temperatures. Despite the cognitive struggles, there is a lot of good going on. And sooner or later, that good will transfer into creative energy.

Maybe I Don't Want a Smart Phone Anymore

A few weeks ago I had the distinct impression that I was being watched while eating dinner in the living room. Not by a person, but by a machine. This was likely the result of an instinctual paranoia that surfaced from somewhere in the subconscious, but is nothing particularly new. Over the last few years I've become all too aware of the myriad of organizations — commercial and otherwise — that are watching all of us as we go about our day, but the feeling of being watched in my own house is still relatively new¹. I want to know where the feeling is coming from, and now it's a matter of elimination.

Offline Mode

Earlier today, while walking through a nearby park, I found myself observing all the people that were out and about. At 5:30pm on a Sunday the park is typically populated by people walking their dogs. Today was quite different, though, as the park was incredibly full with people under the age of 40. A good two-thirds of these people were standing still on a pathway or under a tree looking at their phone. Pokemon fever is at fever pitch and, until the copycat mobile game companies start putting out poor variations of this augmented reality pastime, residents close to the park will continue to head outside in their quest to catch as many pocket monsters as they can.

While watching these people I had flashbacks to my time working in Tokyo, writing software that would allow people to Tweet from their flip phones in an age when the iPhone was barely a year old and Samsung had yet to put out anything people wanted. The investors of the company wanted to know all sorts of information about the people using the applications. Their demographic, the number of Tweets, the pictures they shared, their social weight, and a bunch of other details that we most certainly could extract without anybody's knowledge. The scope of the intrusion was like a slap across the face, and I resisted it. In the end, my boss coded the spy functions and the investors were happy. This was in 2010, and the level of detail that companies can extract from people who use their software today makes anything that was going on six years ago look downright primitive by comparison.

These people in the park, faces lit from below, were sharing so much information with so many companies and likely had no idea it was happening or, as many young people are now wont to do, don't care that it's happening. "Privacy is dead. Get over it!" has been the status quo from many software developers and young people over the years as our lives become ever more intertwined with digital data collection and, rather than protest the obvious forsaking of Article 12 of the Universal Declaration of Human Rights², people are actively seeking new ways in which our habits, traits, preferences, and personalities can be used and abused by algorithms for the sake of recognition or immediate financial gain.

Cameras in the Park

There are three sets of cameras in this picture sending data in near realtime to Fujitsu Japan in Tokyo. Can you find them all? Here's the full-resolution version.

We cannot go outside without our movements being observed. We cannot use public transportation without being recorded by a minimum of 3 cameras. Our phones reveal our locations to companies such as Google and Yahoo! without our knowledge. Web advertisers try hard to put "fingerprints" in our computers so they can track every website we visit, how much time we spend at each site, and what we do there. I used to think my crazy uncles were insane in the 90s when they complained about how "the man" was always watching them, and I still think they were a little touched in the head given the level of technology we had at the time. What's happening today, however, is almost Orwellian.

I fully understand that when we're out in public, we should behave as though somebody is watching us or — at the very least — looking in our direction. We shouldn't spit, litter, cut in line, or any of the other little things we might feel compelled to do while out and about at any given time. I get this and the myriad of cameras that abound at places where people congregate do serve a small purpose when something bad happens and the local law enforcement needs to reconstruct a crime. I get this. But when I can't even sit at home without the distinct feeling that I'm being watched, something is wrong. Maybe that something is me. In fact, I'm willing to bet a dozen donuts that it's all in my head. But this doesn't preclude the fact that more and more of what we do, what we like, and where we are at any given time is being recorded and sent to any number of servers "in the cloud" without our explicit permission. And, in my case, the device that is best positioned to make all this information available to corporations and governments is almost always in my pocket: it's my phone.

Perhaps this is just excessive paranoia. Perhaps this is the onset of some sort of mental breakdown. Or maybe — just maybe — I don't want to have a smart phone anymore ...

  1. I do a lot to make it very difficult to track my computers online, but it's still within the realm of possibility if someone really wants to know what I'm doing. That said, this feeling is not coming from my Ubuntu-powered devices.
  2. Article 12 of the Universal Declaration of Human Rights reads: "No one shall be subjected to arbitrary interference with his privacy, family, home or correspondence, nor to attacks upon his honour and reputation. Everyone has the right to the protection of the law against such interference or attacks."

Good Intentions. Backwards Execution.

It seems that a lot of the freelance work I do is for people who work at a company whose IT departments get to dictate how employees do their jobs. During a recent email exchange, I was asked if it would be possible to set up WordPress on a company's local network in such a way that it was accessible inside the company, but not to the public Internet. I responded with my standard "Of course it's possible; it's just software!", and the client — a middle manager in a medium-sized business — was summarily surprised. She had been asking her IT department for years to have a WordPress installation inside the company, and had been summarily rejected without being given any reason beyond "it's not secure." If that's they're opinion, that's fine, but I find this common pattern a little concerning. At the end of the day, it shouldn't be the job of IT to tell employees of a company how to do their job, but to instead find ways to support the goals people are trying to achieve.

Something I Said to a Client In an Email

If WordPress is really so insecure that it cannot be used on a local Intranet, then how about a different blogging or CMS system? Why shut a person down — especially a manager — when they are asking for something that will make their job easier or more efficient? It boggles the mind.

I've had discussions with people who work in IT at various organizations around the planet, and the answer is almost always the same. "We are expected to keep the company's data safe, so we implement lots of rules on how systems can be used."

What a lazy approach to a serious concern. Yes, it's absolutely important that there are no data leaks or other problems that could affect the company's bottom line or put their customers at risk. This is IT's top priority. But this shouldn't come at the expense of getting work done. By simply asking a person "Why?" 5 times would be enough to better understand what the fundamental goal is, which may be sufficient to offer some other tool or mechanism. If something is truly impossible, then open communication should be used to ensure everybody understands why something is a bad idea. If a person asks for a list of all customers and their email addresses in an Excel file to store on a USB key to bring with them on a business trip, the answer should be a solid "no" with a clear explanation. Something like:

"I'd love to help you solve whatever problem you've got, but putting that much data on a USB key — even encrypted — opens the company up to too much risk. What would happen if the USB key was stolen or lost? I'm sorry. It's just not possible. Tell you what, though, why do you need this information? Maybe there's something else we can do to help you ..."

There are a lot of great people who work in IT that do try to help their colleagues solve problems. I work and have worked with many. That said, it seems a disproportionate number of surly tech goofballs — for lack of a better description — seem hellbent on making sure people suffer while they play the role of king in their imaginary fiefdoms. Technology opens the doors to a lot of wonderful things. Hopefully younger IT staff will replace the angry old guard who has stood in the way of progress for so long before it's too late.

Yet Another News Sabbatical

For about 16 months between 2014 and 2015, in a bit to have a brighter outlook on life, I stopped reading the news. The various RSS feeds I subscribed to were removed from the reader. The websites I would frequent were left unread. The sorts of accounts I would follow on social media drastically changed. An incredibly noticeable change in my attitude occurred within the first two weeks, and it was positive enough that I started to create more often than complain. Earlier this year, though, I started reading the news again in order to keep up with conversations people were having around me on topics that I knew almost nothing about ... and my outlook on the world began to slip downwards again.

If keeping up with the world means subjecting myself to an incredible amount of frustration and anger at the sheer lunacy of what make it to the front pages of news sites around the globe, why do it? Just because I know a horrible thing has happened does not necessarily mean that I can do something about it.


I cannot do anything about the various atrocities that take place in the name of various terrorist organizations. I cannot prevent an angry, xenophobic old man from running for president of the United States. I cannot do anything (of note) to help people lift themselves out of poverty. Ignoring these very real issues will not make them go away. Ignoring them will, however, bring a little peace of mind.

So, starting today, I'll be staying away from the various news sites that I typically read. I will stay away from the TV news that shows brief snippets of the insanity that permeates the 24-hour news cycle. And I'll use this extra time to ... relax. To simply disconnect from the glowing screens and be present in the real world ... or listen to a podcast ... or play with my dog. Words mean things, and it seems the tens of thousands of words that I consume are having an adverse effect. Clearly I'm not cut out to be on top of the news cycle anymore.

Lengthening Upgrade Cycles

Bryan Clark over at The Next Web seems to have a beef with the relative age of the current-selling MacBook Pro, saying that the machine hasn't been updated by Apple because the company has "forgotten what a computer is." This is an interesting hypothesis, but not one that I'm willing to entertain. Instead, I'll posit my own theory for why Apple isn't updating any of their computers on a regular 6 ~ 12 month cycle anymore: it's 2016.

Intel's Legendary 486 DX2/66

Twenty years ago when I first started to deep-dive into the world of technology, I would buy a copy of Windows Magazine from the Avondale near my high school and read it cover-to-cover and absorb the information like a sponge. Later, I'd go through the advertisements and watch as the processors described as "speed demons" just six months before were being replaced by faster, more capable CPUs at an ever faster rate. When Intel released the Pentium-branded chips, the MHz race was on and we quickly jumped from 200 to 450 in the space of a summer, then up to 800 before the next spring, and finally hitting the blazing fast 1.0GHz by the start of 2000. Anybody who had actively tried to keep up with the speed of technology at this time was investing close to $600 a month in upgrades ... and I was one of those suckers.

This was how technology (and marketing) seemed to go for well over a decade. The "l33t hotness" today would be considered obsolete by geeky peers less than 180 days later, and people would be openly mocked online as a result. This started to change for most people around 2008, though, as technology and software had mostly started to reach the "good enough" point where people were comfortable using a computer for longer than 18 months without complaints. Operating systems had become much more reliable by then, as well, meaning that wiping the hard drive and re-installing everything from scratch became a rare nuisance rather than a twice-annual happenstance. By 2012, a lot of non-gaming geeks also found that any hardware they had was good enough for at least four years. Batteries were lasting longer. Processors could keep up with 90% of the work we'd throw at them. Software crashed a lot less.

Now, there's a very real "problem" that arises when people don't have a problem with a product they own: the idea of replacing the item never enters their mind.

How often do people buy new appliances for their home? Washing machines, refrigerators, and microwaves are typically replaced once a decade if that. Many people are able to get fifteen years or more from these machines. As a result, manufacturers have a slower product refresh cycle. Does Maytag or Toshiba really need to put out a new front-load washer every 8 months in order to appear relevant? No. Having worked in the appliance industry for many years¹, I can tell you that most appliance manufacturers have refresh cycles similar to auto manufacturers. There are minor cosmetic changes every spring, maybe a new software update on the higher-end models, but the underlying hardware stays the same for a number of years.

While most manufacturers of cheap, plastic computers churn out an endless array of crap every 6 months, the companies that produce the better, more reliable hardware have slowed down. They choose to invest their time in designing better devices that they can then charge a premium for. Compare any "brand new" Toshiba, Sony, Samsung, Lenovo, HP, or Acer computer that sells for less than $1500 USD to a 440 day old MacBook Pro and tell me how they're better in any 3 measures. I guarantee you can't do it. I tried for 8 months to find a better notebook and failed miserably. As a result, I bought a MacBook Pro and soon replaced the default OS X installation with Ubuntu MATE 16.04 and haven't (often) looked back.

The current MacBook Pro line ships with the 5th Generation Intel Core processors. The 6th Generation is out with reduced power consumption and generally reduced clock rates, but does it make sense to update a whole line of professional-grade notebooks to get another few hours out of a notebook that already provides 12+ hours of life for most people? Heck, according to CPU Boss, going with a 6th Generation processor would be a step down in terms of processing speed. Given that OS X's performance problems are a direct result of the file system² and not the hardware, I'd say a MacBook Pro refresh would be a bad thing.

But what do I know? I'm just a person who uses his MacBook Pro running Ubuntu for 14+ hours a day, 7 days a week. The battery is being used for ten-to-twelve hours most of these days, and this machine is still the fastest non-server-grade computer I've ever used. Would I like a little more battery life? Sure. But I won't accept it at the cost of raw throughput. I won't pretend to know how Bryan uses his computer and what he might want a refreshed Pro line for, but as a person who spends his days programming for multiple platforms, running Virtual Machines, podcasting, and just about anything else aside from gaming, I can tell you that the most recent MacBook Pro line can likely go another 440 days before a refresh is honestly necessary.

  1. Many years ago, I might add ...
  2. Don't believe me? Put Ubuntu on a MacBook Pro and tell me it doesn't feel faster. Heck, put Windows on a MacBook Pro and compare it to an equivalently-spec'd Sony or Dell and tell me that it doesn't perform drastically better. Apple's hardware is incredibly capable. Their problems are all in the software.

How Long Is A Candle?

In September of 2015, nearly six months after scrapping Version 3 of 10Centuries, I started work on 10Cv4, the most complex and current release of my millennia-long pet project. In December I was encouraged to push even harder to get the system ready for podcasting and, after a few false starts, I managed to release a private version with a few people testing it for a month before opening the door to beta testers in January of this year. Since then, there have been hundreds of mini-releases, some big technical changes, a couple of hurdles, and a whole lot of UI development to make the system more approachable for people who have been trained to be more critical of software interfaces. There's still a lot to be done, including development of a multi-platform mobile application that will be available on Android and Apple devices, but the system is coming along nicely and is 80% of the way to becoming profitable.

Since March of this year, I've been working a lot of long hours in order to pull off what many people at work are calling "the miracle of the century." Creating and deploying a new piece of software from scratch to polish in less than six months to replace tools that have been in use and under development for years. A lot of the people I'm working with are incredibly motivated and work hard to accomplish all of their goals on time, and a few have — until recently — openly slowed the team down by remaining willfully ignorant of the project until it was too late. This is typical in most large organizations, and is compounded when working across borders and timezones with people who have never met each other in person. Yet, despite the hurdles, the local team has pulled together to accomplish some really amazing things.

Two months ago a client who I had done a lot of work for in 2013 and 2014 came at me with a new project request. It sounds interesting, and it could prove to be a huge hit in Japan given the rise of the "gig economy" in local communities. I've been offered the position of lead developer, which would likely transition to CTO should investment actually come through after the initial project is built and demoed in Tokyo this fall. At this point my role is strictly consulting, but I've been given the opportunity to hop on board for weekend and evening development sessions which would reduce the time between software releases by a wide margin.

In February 2017 a long-term project is set to begin. One that is going to be more complicated and more demanding than anything I have accomplished to date. Leading up to the start of this project I'm investing a lot of time in learning new skills and putting most of them to use almost immediately. This new project won't pay very well, and might never break even when examined from a strictly financial point of view, but it's one that I've been looking forward to starting for a very, very long time.

No, I haven't forgotten that I'm married and also have a puppy who desperately needs human attention at all times. My family takes priority to the vast majority of the projects I'm currently working on, though I do tend to ask for a few hours every weekend to turn some ideas into software and to read a book or two ... or do some podcasting ... or cook.

Burning the candle at both ends? Me? No. I chopped that candle up into smaller pieces and am burning many little ones!

Lit Candles Lighting the Way

A bad analogy is like a leaky screwdriver.

— Hugo¹

Sitting around doing nothing is very hard for me, and I actively avoid it at all costs. Idle hands are the devil's playthings and, while I don't particularly believe that the demigod synonymous with evil will make use of my body, I do see examples of people who are capable of so many things and have great ideas but zero ambition to actually accomplish their goals. People are free to use their time however they choose, of course. Who am I to judge? What I do know, however, is that I would not like to look back in seven days at what I did during the previous week only to shake my head.

So I work ...

... and work ...

... and work.

No, it's not particularly healthy, and this is a topic that I've touched on numerous occasions with blog posts lamenting approaching burnout, mental exhaustion, and generally feeling bad about myself. This is a lifestyle that I've chosen, but is it the right one?

Can't Nobody Hold Me Down. Not even me.

Over the last decade or so I've tried a number of times — to no avail — to slow down and relax. Occasionally this would involve disconnecting from the Internet where it seems that everybody is always doing something amazing, when really it's just three or four percent of everybody who's doing something amazing. When I see others achieve, I want to achieve, too. As childish as it may seem, the thrill of recognition is just too great to ignore. One cannot compete with everybody, though.

Other attempts to relax have involved me reducing the amount of time I sit in front of a glowing screen, opting instead to use more analog devices such as pens and paper, allowing for a less rigid, more organic creative process to take hold. This is generally a wonderful thing, but it's hard to then transfer that creativity from freeform paper to something that can be shared and appreciated by others.

Occasionally I've even taken the drastic measure of removing the software development tools from my computers so that I instead focus on writing, or communicating, or simply playing a game to unwind and relax. This tack rarely lasts long, though, and is pretty much impossible now that I'm being paid a full-time salary to write software during the day. The development tools are rather necessary at this point.

So why fight it? What purpose would preventing myself from going in six different directions at once actually serve aside from introducing distractive thoughts actually serve? Exploration of other pastimes? Spending time in places where I am physically present but cognitively elsewhere? Forcing myself to confront the demons of self-doubt that ceaselessly whisper into my subconscious ear? Wouldn't some measure of success and recognition address that last issue, thus validating the excessive efforts that are being put into a number of projects, both large and small?

And this is really the crux of the problem, isn't it? I don't feel I've accomplished anything worth being proud of and, when something does gain some modicum of acclamation, I shrug it off thinking that I don't deserve any praise because there are other five other people I could point to who are doing the same thing, only better.

10Centuries cannot hold a candle to WordPress or SquareSpace. The gig economy tool has long been the domain of CraigsList and AirBnB-like sites. The LMS I'm building at the day job? Well ... that's something that might garner a whole heap of recognition in the education sector, but only because there are 20 very dedicated people working on the project and all the existing commercial products are just awful. The big, long-term project that starts next year ... both excites and terrifies me. What if I'm not good enough?

People who have known me a long time will probably scoff at this endless cycle of self-flaggelation. Sure, the things I create may not be to the same standard as the offerings provided by others, but they do offer something unique — something that other tools either do not or cannot implement. This is what I'm told right before the ever-helpful "cheer up!" comment that comes across more as exasperation than encouragement ... though I won't fault them for trying.

So here I am, writing a blog post just minutes to midnight trying to sort out why I can't sit still or limit myself to just one or two projects instead of some ridiculous number. Believe it or not, posts like this do help me contextualize what's going on in my head, and nothing here is new to me. I know I need to wield better control over my time, and I know I shouldn't say "yes" to every project that lands on my desk. I can't take more than a day of doing "nothing", and that's just fine so long as I make sure the people around me are taken care of. I should get out more, too.

All of this I already know, but that desire to do something great demands the lion's share of the pie. What I really want to know is how to satisfy that seemingly insatiable need. Maybe when the mind is less concerned with trying to accomplish something, it will actually come to pass.

1. Hugo who, I don't know. The quote comes from an episode of The Ubuntu Podcast.


Back in 2006 when this blog was just a few days old, I wrote a post where I revealed my long-standing paranoia that I am being observed. Although nearly a decade has passed since that post first went online, I still battle with the keen awareness that at any given time my activities may be captured on any number of cameras — hidden or otherwise. Since Japan was awarded the 2020 Olympic games the feeling has only become more prevalent as companies like Fujitsu and NEC team up with pseudo security companies like Alsok and Secom to put a camera in every ceiling and stream that data back to a series of supercomputers that will use facial recognition to identify every person walking by in real time and notify the authorities should anyone on a wanted list make an appearance. While this sounds a little far fetched, it's been happening for at least the last two years at train stations across the nation. During that time the technology companies that support the pseudo security companies rolled out massive infrastructure upgrades to perform all of this observation with cold equations and hot processors. One place where I have often felt safe from observation, though, has been the inside of my home ... until today.


While eating dinner in the living room I had the distinct impression that my activities were being monitored. Not by my wife and not by my dog, but by a machine. Now, paranoia can make a person create enemies where none exists, but this particular feeling was especially strong. There was just too much wrong with the whole scenario. The TV was off, which is almost unheard of when my wife is home. She was using her computer, which is also almost unheard of after 9:00pm on a weekday. The neighbours were quiet. Dinner was one of my favourite meals. Most people would look at this and say "Oh, what's up with all the different stuff?", but I'm not most people. I see phantoms when many people don't, and I create phantoms when nervous energy has nowhere to go.

Despite my best efforts, I could not locate a camera in the living room. I also took a look at the network traffic to see if there was any chance a video stream might have been sent during the evening, but everything came up clean. There are no unrecognized wireless networks in the area, and the last one was two days ago when somebody's Android phone broadcasted its mobile hotspot SSID.

Yes. I log all of this stuff. I've battled paranoia for well over a decade and, like I've already said, I see phantoms where none exist.

All in all, this is probably just a case of nerves. The big project at work has been struck by delay after delay on upstream processes, and I'm pulling my hair out because the August 8th deadline is all but impossible to meet now. This is despite the fact that the project manager and I have both invested well over 150 hours into the project combined each and every week, and despite the fact that the front and back end that I have developed is 90% complete. This frustration and anxiety is likely manifesting itself in a manner that has me on edge looking for interlopers who will obstruct the project even more.

But why is it that of all my psychological deviations this energy decides to converge on paranoia? Why not depression? Why not insomnia? Why not addiction? Has my new diet made addiction and excessive sugar consumption all but impossible, pushing paranoia to the forefront of the demons I must battle?

Am I overthinking it?

I'm typing this post from my bedroom where I feel absolutely comfortable. There are no active listening or video devices in this room that I know of, and I don't have that weird feeling in the back of my mind telling me to be extra wary. Perhaps I should just call it a night and let the paranoia stay in the other room ...

Real-Time Search

This morning, while staring at the day-job workload waiting for me in the form of GitHub tickets, I started thinking about a problem on 10Centuries that I have long wanted to solve. It's a topic that has come up again and again over the years, and I think it's almost "solved". The problem of course, is search.

Search on a website is theoretically pretty simple. People enter some words in a field, those words are checked against the content in the database, and results are returned. In its simplest form, only the exact search term is sought. This means that if I were to ask for all posts containing the words bright and yellow, I would see only this result as there is an exact match for "bright yellow". But if I were to ask for yellow and bright, nothing would come back. This is clearly suboptimal, so it's better to have all of the words split apart, with results that include all posts with the words bright or yellow, ideally scoring the posts in such a way that the above-referenced post is at the top of the list. A lot of effective software uses this weighted search result method to return relevant results, but I wanted to do something different still.

I wanted people to see something instantaneously.

The New Search Box

One of the tricky parts of instantaneous results is dealing with network latency, server load, and all sorts of less-than-desirable problems that can make a theoretically semi-decent idea practically untenable. More than this is the general response times of the service. Most people can type several characters per second. Sending multiple calls to the API just for the illusion of supplying decent results in realtime seems silly. So I decided to go about solving the problem a little differently: a subset of every post is loaded into memory and called when requested.

This blog post is number 2,428 in publication order — so long as I haven't back-dated anything since this post went live — and the average size of each post is roughly 618 words. That's 1.5-million words. A crazy number one might say, but then I have been blogging for almost a decade. 150,000 words a year is nothing compared to the number of words that have been published on various social networks, forums, and IRC channels over the years. Loading all of these words into a browser would be absolute overkill so, instead, I am loading just a subset of the words that constitute a post. As people type their search query into the box, the browser scans through the data stored in memory, finds matches, scores them, and then updates the results. People with relatively recent hardware will see that the operations are pretty much smooth and responsive. People with hardware as old as this blog ... will unfortunately suffer some stuttering. People searching other 10C-powered sites will likely not notice a hiccup at all.

The browser is working with a subset of the posts, though. What's not included? The content.

For the moment, search will pull from titles, URLs, tags, and author names. Future updates will include the content of the pages and posts. Yet before it can happen, two things must first take place.

  1. I need to see that people are able to use the search in any browser on any platform. This is still in testing.
  2. I need to create a cached result for every post that contains just a single copy of every word in the article, excluding certain common words in various languages.

Once these two things are done, then I can build on the existing search tool in order to provide much better, more specific results.

In the meantime, people using the default blog theme on 10Centuries will see an "Archives" link in their navigation bar. Every post will be listed in reverse chronological order, and the search bar up top can be used to quickly find published items. If you don't see this link, it's because the cache for your site has not been refreshed. Simply write a new post (or update an existing one) to force the system to regenerate your website.

This isn't a perfect solution by any stretch of the imagination, but it solves a number of problems that I've been thinking about for quite some time, and it does it in the browser rather than taxing my own servers with Google-like search speeds. Hopefully this same search method will be employed in every theme going forward.

  1. 1
  2. 2
  3. ...
  4. 244