If You Solve Enough Problems ...

At some point, everything's going to go south on you. You're going to say "This is it. This is how I end." Now, you can either accept that, or you can get to work. You solve one problem, and then you solve the next problem, and the next, and if you solve enough problems, you get to go home.

— Mark Watney (The Martian)

This quote from the end of The Martian often drifts through the mind whenever I look under the covers of a "completed" task to find a mess of spaghetti and broken promises. Unlike the protagonist in this lovely work of fiction1, death does not wait for me at every turn. Instead I'm often looking at something that should have been done but isn't remotely close to being so.

This problem bit me today pretty hard when I went to access an old project of mine because a colleague wanted to see how something was done. The system was still up and running on the development server and I plugged in my credentials so that I could complete the task of creating a new account in a matter of seconds … except my password wasn't being accepted. So, because it was a SQL Server-based system, I fired up Azure Data Studio and tried to connect. No dice. The connection was made, but the pre-handshake was failing. So, being a Linux-based installation of SQL Server, I SSHed into the machine and confirmed everything was running … which it was. The credentials were still good, as I could connect through the command line, but it seemed that connections from external sources were not being accepted. Checking the firewall, I saw that there was nothing in place to prevent any other machine on my network from accessing this server.

So what was going on?

My checklist of operations for a failed SQL Server was brought up and I started going through each item line by line. The system was restarted. Any new updates were applied. A new account was created for testing purposes. The firewall was temporarily disabled. A full 30 items were run through to no avail. The machine simply refused to allow connections from outside the server. An hour on the Microsoft help forums and StackOverflow didn't result in any solutions but, after taking a short break for dinner, I remembered that I needed to do something after updating the system a few months back. SQL Server for Linux is officially supported only on Ubuntu Server 16.04 LTS, but I am using Ubuntu Server 18.04 LTS which has updated versions of some required libraries. Symbolic links need to be created to the older versions of two files in order for SQL Server to work.

cd /opt/mssql/lib
ln -s /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 libcrypto.so
ln -s /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 libssl.so

After creating the links I restarted SQL Server and tried to connect from the web server again to find that it worked like a charm. Azure Data Studio, too. The issue had to do with an incompatible SSL library being used. By forcing SQL Server to use the older version, everything worked as it is expected to.

There was no risk of death today. There was, however, a whole lot of problem solving.

  1. Yes, there are a lot of people who dislike Any Weir's book for some semi-valid reasons, but I enjoyed it, so leave me alone. Get off my lawn! Darn clouds ….