Nine Years

Nine years have passed since the Great Tōhoku Earthquake shook half of Japan and triggered a tsunami that wiped costal cities off the map while claiming the lives of thousands. In this time a great deal of rebuilding has taken place and people have moved on with their lives; some with far more success than others. Reiko, Nozomi, and I were quite fortunate that we were largely unaffected, though Nozomi did have some pretty serious eating problems until mid-way through the following year due to stress and anxiety. Every large truck driving by would result in our apartment shaking, which would freeze the puppy1 in her tracks. Getting her to eat was a trial of patience, though she did pull through. Moving back to the Tokai region of the country went a long way to helping all three of us calm down and relax.

Moving back is something that I am quite grateful for. Reiko was not happy very often while we lived in Kashiwa, as I was gone most of the day to work in central Tokyo while she was at home in a largely unfamiliar city. We moved to the Tokyo area because I had accepted a job at a tech startup. It paid quite well and there was so much that I learned while solving the problems of the day, but what value is this when your wife is unhappy? Sometimes I joke that the only reason we moved to the Tokyo area was because fate demanded we bring Nozomi into the family. Other times it doesn't seem like a joke at all.

The quake did leave some lasting marks, though. My mind continues to conjure up tremors that don't exist. Both Reiko and I are careful to keep our emergency preparedness packs ready and properly stocked with dry food, clothing, and various supplies. We also have several external batteries for our devices in the event we cannot charge them for extended periods. The car never has less than a quarter tank of gas, either. If something happens, we can leave the house and drive at least 100km to safety with just a moment's notice. This is something that I never would have done in Canada despite living through the Great Ice Storm of '98.

Yet, all in all, life has drastically improved in the nine years since. Reiko and Nozomi are both healthier. The boy has come along to provide a great deal of entropy to our lives. We no longer live with just a couple of months' worth of savings in the bank. We have a house in a safe neighbourhood that sits 120m above sea level and 30km from the nearest beach. The 3/11 Earthquake was a time of crisis, but we're in a better position as a result of getting through it. Again, I am grateful.

The gratitude I feel is not for the quake, of course, but for how Reiko and I responded, overcoming the challenges and accepting opportunities that were different from what we might have desired in order to regain some semblance of stability. When we had that platform by the middle of the following year, we worked towards the goals that have led us here to where we are today. It wasn't easy. It wasn't always positive. It was educational, though.

  1. She was actually a puppy back then.

Fewer Subscriptions

For a little over two years I've been using Setapp as a "developer's App Store", picking up applications that allow me to solve specific problems encountered both at the day job and in my personal projects. With it I have been able to try out a number of different tools to find the best one for the particular tasks at hand and, for $10 a month, it proved to be worth the subscription rate. That said, as my work has become more specific, there has been much less need to quickly find an application to solve the occasional "Problem X" as there was just a couple of years ago. For this reason, I pulled the plug on the subscription and instead bought the two applications that I use most often: TablePlus, which I am now using daily on account of all the PostgreSQL work, and ForkLift, which is used a couple of times a month to upload large sums of data to Amazon's S3 storage buckets.

This move is the continuation of a process that I seem to have subconsciously embarked upon, where subscriptions that no longer offer the value perceived in the past are cancelled. This month it's Setapp. Last month it was Netflix. The month before was Amazon Prime. None of these subscriptions are particularly expensive, but they do add up and it has me thinking about the business model of collecting relatively small fees for access to a walled garden of digital goods and services.

Over the last couple of years a lot of us have slowly accumulated a large number of subscriptions to all kinds of services. Sometimes this is to support the developer(s) of a project. Sometimes this is to access commercial materials. Sometimes this is for sheer convenience. These fees add up pretty quickly. Is every service worth the amount provided? It seems the only time I ask this question is after an email hits the inbox letting me know that I've been charged for another month or year of service. I'm all for supporting people in the quest to do good, though I sometimes wonder if there's a better way to use the money.

This reduction of subscriptions should be a good opportunity to examine the lot of them. By being a little more careful with what services remain active, it will be possible to invest a little more intelligently in independent projects as well as my own.

Still a Pretty Penny

Every day the morning news programs interrupt their soft journalism with semi-serious updates on anything and everything related to the coronavirus situation. There's always news of more patients, more quarantines, more market problems, and more challenges as people scramble to deal with an invisible enemy as best they can. One of the more common stories that these news programs like to focus on is how much trouble the tourism industry is in, as people choose to stay home rather than risk contracting the Chinese virus. Hotels are offering discount rates. Restaurants are offering free (non-alcoholic) drinks. Smaller companies are also offering discounts or special deals in an attempt to drum up enough revenue to float for a month or two while the global contagion runs its course. One group that is heavily dependent on people travelling long distances doesn't seem to have changed their prices, though: airlines.

Two years ago today I was flying from Tokyo to New York to attend a week of meetings at corporate HQ. Non-essential travel has been cancelled at the day job but, if I did have to buy a ticket to the other side of the planet, how much would it cost?

Tokyo (NRT) to New York (JFK)

Seems the price hasn't changed at all since the last time I went.

Five Things for March

This past week has shed a little bit of light on what it might feel like to be a pet, locked inside and unable to go outside or enjoy the freedoms so many of us take for granted. In a stunning turn of events, time has actually moved slower this week than at any point that I can recall in the last 40 months. This is most likely due to a drastic reduction in physical activity and the daily arguments with the boy around simple things like picking socks off the floor. That said, there have been some things of note over the past seven days that are worth a paragraph or two of time.

Star Trek: Picard Begins

Episodes six and seven have renewed my hope in the series after four and five went down poorly. Heck, I didn’t even finish episode five, choosing to end the show with ten minutes remaining because it was just … terribly predictable. Mind you, Star Trek has had ridiculously predictable stories since the late 80s, so it’s no surprise that every modern episode of the franchise follows similar patterns. By the end of episode 7, I pieced together what I see as being the outcome of the show. Only time will tell if the various hunches are correct.

Depression Returns

With the lack of exercise and fresh air, I’ve found myself sliding into the familiar cycle of self-loathing and equivocacy towards just about everything. A good amount of personal time has been dedicated to getting more of the 10C platform built out in an effort to get back into a productive cycle, but its not really helping dislodge the unyielding lack of motivation I feel nor the dark thoughts that crowd the mind’s eye. Hopefully there is a resolution to the Covid-19 situation in the near future so that I can get back into almost daily treks around the neighbourhood without the stresses and hassles that come with every exit from the house.

Intel (and AMD) Are to Blame

Over the last couple of weeks I’ve been working my way through a commercial Learning Management System in an effort to get it ready for use by the day job. The software was purchased through a vendor alongside a bunch of customizations and now the system has been handed over to me to coordinate and manage the future direction of the tool. One of the things that has become incredibly apparent from the source code of this very common learning system is that the developers have not had to think about resource consumption at any point in the last decade. The number of blatant inefficiencies boggles the mind and gives me more incentive every day to toss the expensive software into the garbage and write a couple of dozen python scripts that read from flat files to deliver the same experience with 1/1000th the processing power and 1/10000th the network usage. These numbers are probably grossly inaccurate but, given what I see in the source of the LMS, there is no way any developer contributed a line of code from a machine with less than 16GB RAM and a quad-core i7 (or equivalent) CPU. Schools running this system likely pay many times more for servers and bandwidth than they should, all because of inefficient and unnecessarily bloated code.

Then again, maybe I try too hard and focus on the wrong things. The number of people who say “Who cares? So long as it works!” is … disappointing.

Coffee Without Milk Isn’t Coffee

Thanks to China’s uncanny ability to export highly contagious viruses, Reiko has “forbidden” anyone in the house from going into crowded buildings, such as grocery and convenience stores. As a result, we’re ordering our food online and having the store deliver it to the house1. This means no quick runs for milk, which means using a powdered substitute or drinking the caffeinated beverage black. Of the two, a powdered substitute is preferable, but just barely. Coffee was meant to be had with a bit of milk. Anything else is just spoiling the beverage.

Polyphasic Breakdown

Back in December I started a polyphasic sleeping pattern that would see me get two (or three) power-naps during the day with a 4-hour block of rest at night. This allowed for 18 and 19-hour day lengths, making it possible to work on personal things, spend time with the family, and tend to the day job. This worked out relatively well until around the end of February when the old habit of falling asleep in the middle of the day when napping was not an option started up again. I’ve since has to re-adjust the body to seven hours of sleep rather than five, and this seems to be working a bit better. There’s still another year or so of the “special project” at the day job, which means a proper sleeping pattern is at least twelve months away. Hopefully the new schedule will survive that long.

There are a few other items I could have written here, but then this post would have become a “Nine Things” article, which would probably be a bit too long. One thing I can say, though, is that it’s good to hear my kid laugh at British comedy. This is something we both enjoy greatly.

  1. I know. I know. Believe me, I’ve argued the faults in the reasoning for hours to no avail.

Phantom Rumbles

Almost 9 years have passed since the Great Tohoku Earthquake that shook half the country's land and triggered a devastating tsunami that laid waste to entire coastal towns. Aftershocks were frequent and concerning, hitting the improbable number of 1,000 just thirty-three days after the event. People were wary of the ground beneath their feet and for good reason. Since 3/11 quake I have often tensed whenever the ground shook on account of a large truck going by or someone turning up a decent sound system nearby. Every time the floor moves, either in my mind or otherwise, I'll look for some indication that the ground really is in motion. This often means looking at a glass with liquid inside or a hanging ceiling light. If there is a real earthquake, these objects would show signs of its existence. Over the nine years since 2011, I've been right fewer than 5 times.

Last year I wrote about these phantom quakes and how I generally deal with them. Lately there's been a new twist to the imaginary seismic activity: phantom rumbles.

Before each earthquake is a low rumbling from the ground as the incredible energy contained in the earthquake's shockwaves ripple across the land. I've been hearing this sound once or twice every day for the last couple of years and, just like with the phantom quakes, I look for a glass or a ceiling light to verify the concern. Lately the sound has become a lot more frequent, though.

Every truck that drives by and some of the vans driven by neighbours all seem to carry the telltale sign of an impending quake. Try as I might to rationalise it, there's no getting around the fact that I'm clearly worried that there will be another 3/11-type event in this area. It's irrational and I know it to be such, but the deep-rooted concern that we'll have such an event cannot be ignored.

How long will this fear of large quakes last? The family and I were physically unscathed by the giant quakes. There's no logical reason for the concern to exist, yet it's very hard to get rid of.

Better Information

Since the news of COVID-191 started to circulate around Japan, people have wanted to know where specifically cases have been found. When there is a highly contagious strain of a foreign virus running rampant, the last thing a lot of people want to do is put themselves or their families at risk. Unfortunately, the municipal, prefectural, and federal governments in the country have tried to keep locations under wraps in an attempt to minimise the risk of a temporary economic disaster in any given area2, but the data is available if you know where to look. Fortunately a university student from Kyushu University invested a little bit of time, learned how to code up a quick website, and put the data online in a visual manner that is both easy to understand and appreciate.

001 - nationwide

The map is interactive and contains data from the ministry of health and welfare, breaking down the cases numerically and by municipality. Red dots are people who are currently infected, identified by order of the infection3. Green represents the number of cases where the infected patient recovered.

As one would expect, everybody is looking forward to seeing more green.

002 - nagoya area

Based on this chart, the nearest case to my home is about 13km away, not too far from the dealership where I bought my car. Given how many people in the neighbourhood work in and around the area with known cases, there is an obvious need to pay attention, refrain from licking sidewalks, and observe proper hygiene when using and returning from public spaces. That said, it's great that a university student with no background in programming was able to learn enough to build this website and share the data with people across the country. My wife and I are both indebted to him as it saves a great deal of time when sifting through the news for recent updates.

Hopefully some level of government takes the initiative to copy this website and provide people with easy-to-understand, reliable, and recent information. Sadly, governments are generally not known for being able to communicate effectively unless they're demanding something from citizens.

  1. Colloquially known as the coronavirus, though there are many coronaviruses. This XKCD explains the term's usage quite well.

  2. People are often too cautious about things in Japan. It can become very tiring very quickly. There are some advantages to the conservative approach, though.

  3. If the number is 350, that means the person identified is the 350th known case, not that there are 350 infected people in the area.

Everybody Should Have 32GB

A colleague of mine will have the good fortune of a RAM upgrade in her computer today, bringing the device up to a respectable 16GB in total. However, given the number of phones that are starting to ship with as much as 12GB right from the factory, it seems bizarre that the digital devices that we use at the day job continue to be hindered by a lack of relatively inexpensive upgrades. Given the complexity of the tasks we ask from our computers, everybody should have at least 32GB in their notebook or desktop machines1.

The reasoning behind this is pretty simple, but I'm going to over-simplify it: the more RAM a device has, the faster it is.

Sticks of RAM

Before using my current notebook, I was using one of two devices with 8GB of memory. While these machines had enough processing power to do a good bit of what I was asking, the lack of RAM made working with large sets of data rather time consuming. Often times I would find myself sending several gigabytes of data to the upstairs server, with its 32GB of RAM and really fast SSDs, in order to process the results without negatively impacting the performance of the main work notebook. The new machine with its 32GB of RAM has never run out of capacity regardless of how much I've thrown at it. More than this, memory is rarely flushed back to disk for later retrieval, making the system performant in every metric that counts. Is there a large Excel file to work with? Does an 80GB database need to be restored and processed? Will there be a series of videos to convert for use as a web-embedded resource? It really shouldn't matter. People deserve to have a machine that can keep up with them.

Generally the argument that comes up when I make statements like this is along the lines of "Well, not everyone is working with a lot of data. Some people just like to use Facebook and watch YouTube.". Fair enough. But why should anyone want to keep their machines limited if they're going to be used for any sort of work? Yes, there will be "wasted" memory sitting in machines if everyone has 32GB (or more), but it's better than the alternative.

  1. I say this fully understanding that a lot of modern motherboards simply cannot handle more than 16GB for various reasons. The fact remains that people should have ample RAM.

Thinking About "Smarter" Crawlers

This past week has seen a number of much-needed updates to 10C get released, one of which was a direct result of seeing an excessive number of requests coming from unabashed slurping bots. There is no denying that organisations that exist for the sole purpose of earning money from the efforts of others tend to follow each and every URL that is found on our websites, but is this really the best use of everyone's resources? Any crawler that wants to index each and every bit of public content that is hosted on 10C will need to read several million pages worth of text. Even accessing 5 URLs per second would mean that a crawling engine would need about 291 days to access each an every post. This is terribly inefficient for both the content-scraping scum and my infrastructure.

This got me thinking ….

As of this moment, every post on 10C works out to about 261 megabytes of text. Compressed it would be closer to 64MB. Updates are sequentially tracked and can be rolled back to any point at the drop of a hat. Would it not make more sense for crawling engines to have a URL that could generate a complete package of content on demand for them to download then, when the machines stop by in the future to look for updates, they can use the same URL but with a query string saying something akin to "give me all updates since last Tuesday at 3 o'clock" to download a much smaller package than the first. Doing this would be a win-win for everyone in that web servers could save their resources for actual visitors and companies that crawl every page online will not need to wait darn near a year to completely archive a platform as sprawling as this one.

Bandwidth consumption would go down. Page load times would drop. Server hosting requirements would decrease ever so slightly. All for the sake of an open mechanism that allows marketing companies to get what they want.

Then again, why would we want to make this easy for marketing organisations? No … I'm happy to have the recent updates that identify the bots and block them outright with a nice and clear message. The bandwidth usage has dropped. SQL operations per second have been halved. The server can actually relax a little bit now.

Darkness Across the Screens

Allergy season has started early this year, bringing a great deal of discomfort along with a little loss of vision. Fortunately there's some relief from the itchy, watery eyes with some over-the-counter medicines. The loss of vision, however, is new.

Late last week I noticed that it was almost impossible to use my phone in a dim room. Even at the lowest brightness, the screen was far too bright to be readable. This has happened a couple of times in my life when over-tired or staring at a CRT screen for too long, but the issue is so rare that I actually had to remember that my eyes would become sensitive to light after staring at a CRT monitor for a dozen hours or more. On weekends I seldom use the notebook for very long as Saturday and Sunday is "family time", and yesterday there weren't any real issues after putting in a full day's work at the day job. However, today the vision problem returned shortly after dinnertime.

One of the things that I've noticed about my eyesight is that things in the periphery are essentially just fuzzy blurs. While our peripheral vision is not designed to provide a high resolution depiction of our surroundings, it was generally decent enough up until the age of 35 to identify the brand and make of a car1. Now I'm hard-pressed to see there's a person next to me without paying a good deal of attention. That said, because the vast majority of my day is spent working with text, I can focus really well on characters that many people complain are far too small to be readable. To make these characters as legible as possible, I've generally stuck with dark text on a white background. This preference started back in the late 90s when I was working a great deal with Visual C++ and Visual Basic 5.0. Microsoft's development tools were all like this; black text with blue keywords and red errors on a white background, and I've generally set up every system I use — including how I write notes on paper with my three pens; black, blue, and red — to follow the same colour scheme2.

Tonight, though, I couldn't stand to look at the screens on my work desk after putting the boy to bed. It's not only that they were too bright; the white backgrounds everywhere were causing my eyes to water and blur. Could this just be exhaustion? Most likely. With all the pollen in the air and a distinct lack of the typical humidity associated with this part of the globe, my eyes are likely expressing a desire to be protected with a pair of closed eyelids. Unfortunately, there's still work to be done.

This is where "dark mode" can actually be useful.

A lot of people who use their computers for most of the day have sworn by dark mode for as long as I've been writing software. There are certainly a number of benefits to the inverted colour scheme, but I've always found it hard to read light text on a dark background, particularly if the characters are in any colour other than white or very light grey. That said, when there's work to be done, sometimes it's better to make do3 with what works rather than not work at all.

Sublime Text, the program I use for the vast majority of my coding work, was the first to get the dark treatment. That helped. Then it was Safari, followed by Outlook4, followed by Teams, then the desktop wallpaper, and finally the websites I use on a regular basis5.

My eyes responded well to this. Better than expected, I'd venture. They're not feeling nearly as tired as before. While I don't know how visible a dark theme will be during the daylight hours, I'll certainly make use of it during the nighttime.

Of course, I'll also make an appointment with my eye doctor when all this coronavirus kerfuffle comes to an end. This could be fatigue. It could be something else. I'd rather give up working with computers for the rest of my life than give up my vision6.

  1. One could probably argue that the key information was subconsciously collected with a rapid, involuntary glance that identified the shape and logo of the vehicle. I'm no expert on how our eyes work or how our brain translates photons into a visual expression of the universe we inhabit.

  2. The one exception to this rule has been the tools I use to write essays and blog posts, where the text is only black and the background is a light grey.

  3. I originally typed "make due", which is supposedly "a historical variant that is no longer accepted". If it's no longer accepted, then why the heck have I been using it my whole life? There's no way it was deprecated after I left college then fully expunged from the language at some point in the intervening years. Language doesn't work that way!

  4. Even after changing Outlook to "Dark Mode", there's another toggle you need to hit to make the email section flip from a white background to a black one. Who thought this was a good idea?

  5. My personal site has an auto-dark mode feature, so long as JavaScript is enabled.

  6. Vision is the one sense that I would never want to lose. Smell? Sound? Touch? Taste? Yeah … I can get by without them. It wouldn't be fun, but it wouldn't be impossible. No sight? That would make many of the things that I enjoy most in life all but impossible.

Old Databases

Earlier today I stumbled across an older post while looking for something else and noticed that there is no footnote defined for the 1 in the first sentence. At first I thought this was something that might not have come over during the data migration that converted posts from the v4 database to the very different v5 structure. However, after looking at the last backup from v4, I can see this problem dates back even further: to the time when v2 switched over to v4.

That was in January 2016.

The footnotes were eventually found in the last backup of the v2 database as rendered HTML, telling me that the source of the post was most likely Evernote. Sure enough, all the requisite metadata was there showing that the post did originate from Evernote back in 2012 and it's been sitting untouched ever since.

This blogging engine has seen quite a bit of change over the decade that it has existed, going from an Evernote-linked tool to something that could accept posts from multiple sources to something that could host podcasts to the something that currently exists. Each major revision has seen the database schema change quite a bit as past lessons guide future directions. So, as this was the first time in several years that I've looked at the 10Cv2 database, what does the 40 year old me think of the design?

Well … it wasn't very sophisticated. Heck, I'd say it was downright barbaric in its design, which leaned heavily on the Evernote API structure. The only way it could ever be performant for more than a few dozen sites would be to have the contents of every table kept in memory at all times, which is not recommended when running any version MySQL created since the Big Bang1. This lack of regard for previous efforts is not necessarily a bad thing, though. Working within the constraints of various shared hosting servers, then later VMs on Sakura and Amazon, I learned how to design databases that can run on very basic servers. The systems I make now, even if they're destined for powerful enterprise-grade server hardware, are designed to be as quick and nimble as possible while being flexible enough to allow for and endless amount of evolution going forward. Software projects are never finished, after all; they're abandoned. The early versions of 10C certainly prove the case.

I wonder what a future version of me would think of the v5 data schema.

Over the next few days I'll dedicate the time to restore these old footnotes so that posts from 2012 and earlier are properly structured. Hopefully there won't be too many items that need more than a straight update. The data structures have certainly evolved over the years, but the Markdown syntax that is used for Footnotes has remained incredibly consistent since I ripped it off from a WordPress plugin2 I had been using previously for the very same reason. With any luck, this will be the last time I have to go back to the v2 database to look for missing data.

  1. Yes. That Big Bang. Using the MEMORY engine is not recommended without a whole lot of failsafes put into place right from the onset and, even then, it's better to not use it.

  2. Many thanks to John Watson for creating FD Footnotes for WordPress. While the footnote implementation in 10C has evolved over the years to support various new requirements, the core functionality that went into Noteworthy, then later 10Cv2 and v4 was taken from John's plugin.