Effective Backups

Yesterday I wrote a bit about how people will likely continue to store a great deal of irreplaceable data on a single computer without ever once performing a backup of that important information. At the end of that post I said that today I'd talk a bit about how to perform backups. Here's how we can do it, and there are really only two options.

Option 1: Live a Life of Simplicity

Learn not to care about your data. Don't save any files. Use a film-based camera. Watch DVDs or Blu-Ray movies only, and listen to music only from the source CD, cassette tape, 8-track, or LP. Don't keep your old receipts. Don't keep copies of contracts you signed. If something is important enough, like a mortgage contract, you can be sure that somebody else will have a copy of it should you ever need to look at it again.

People who have nothing of value have nothing to lose.

Option 2: Be an Adult and Plan for Eventual Failure

Hard drives fail. Computers break down. Notebooks are stolen. Shit happens. So take precautions and be ready for when this happens. As with any machine designed and used by humans, the darn thing WILL break one day. There is no disputing this fact.

That out of the way, there are various ways we can perform backups. For people who don't want to splurge and set up a home server, follow these steps:

  • buy two more hard drives of the same capacity or greater than the one in your computer (or computers)
  • put these drives into an external enclosure (USB, FireWire, LightPeak/Thunderbolt … it really doesn't matter so long as your computer can use the external hard drives)
  • if you run Windows 7, tell Windows that's a backup hard drive and it will take care of the rest. If you run and older version of Windows, upgrade your system. Seriously. You don't need to buy a new computer. Don't believe what the 20-something dork at the computer shop tells you. Just buy the Windows 7 upgrade DVD and be done with it.
  • if you run OS X, tell Time Machine to use this as a backup location
  • if you run Linux, you should already know how to deal with backups and data loss prevention

Why two drives? Simple. Just as our desktop and notebook hard drives can fail, so can our backup hard drives. More importantly than that, though, is the offsite backup routine. Every weekend take the hard drive you've been using that week to somebody else's house1, or your office. Leave the drive there. Your other drive should be at that location. Bring that other one home and use it as the backup location for the next week. If your house does burn down, you will most likely not lose more than a week's worth of data. That can still be painful, but it's a lot less painful than losing 100% of everything.

Should you go the route of RAID on a desktop system? No2.

Should you go the route of storing everything "in the Cloud" and trusting companies you've probably never given a dime to with the long-term security of your data? No3.

Should you simply invest $600 in a new computer every 3 years rather than spend $200 for two hard drives plus enclosures? No. Don't tempt fate; she's a cruel mistress.

For 99% of the people reading this article, the steps I outlined above will be more than enough to protect the majority of your precious information. This is also a really good excuse for visiting the mother-in-law every weekend to pig out on free cookies and coffee. Trust me. When something really, really bad happens to your computer, you'll be glad that you followed the advice here.

Cheap Man's RAID With Ubuntu

It seems people are starting to look at putting some of their older hardware to use as simple network storage servers. Since posting a quick little screen shot a few days ago showing a 32-bit Ubuntu installation with a 7.5TB partition I've had two people ask me how they could do this with a netbook and a bunch of external drives. While I would usually not suggest that people do RAID over USB unless there's a dedicated box for the hard drives, there's nothing wrong with looking at this from a strictly academic standpoint.

What We'll Need:

  • one netbook (Atom processors welcome)
  • a bunch of same-sized USB hard disks (for this example, I will be using five 2TB Seagate hard drives)
  • a powered USB Hub (this depends on how many hard drives you want to connect to the netbook)
  • Ubuntu on a bootable USB stick
  • one of your favorite beverages
Ubuntu is incredibly easy to install, so I won't describe how to do it here. Instead, I'll start from the next step after plugging all the hard drives in to the system: configuring mdadm.

First, let's install the package. From the terminal, type: sudo apt-get install mdadm

This will install all of the required packages to use the software RAID tool, and will configure the system appropriately. Next, if you're like me, you'd probably prefer to use a file system like XFS for larger partitions spread across disks. I won't get into the details of why I would choose XFS over other file systems like Ext3, Ext4, or ZFS, but you can install it with a simple sudo apt-get install xfs and sudo apt-get install xfsprogs

Again, because this is strictly an academic exercise, this won't be necessary but it also won't hurt anything if we go this way.

Now comes the fun part. Let's create a definition with mdadm. First, identify the locations of your external hard drives. Ideally, the external disks would not be mounted, as that would make this process quite difficult. In my case, the five hard drives range from /dev/sdb to /dev/sdf. I would like to put these together in a RAID5 configuration, and I would like to use a 16K chunk size. With this in mind, I would type:

sudo mdadm —create /dev/md0 —chunk=16 —level=5 —raid-devices=5 /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf

Let's break this down.

—create /dev/mdo creates the array name (I typically use md0, md1, md2, etc.)

—chunk=16 this is optional, but sets the chunk size for the RAID array. Caution: bigger is not always better!

—level=5 signifies what kind of RAID array we would like (0, 1, 5, 6, etc.)

—raid-devices=5 states the number of drives (or partitions) that will be connected to the array. This is immediately followed by the locations of each drive (or partition) to be attached.

Ubuntu | mdadm started

You should now see something similar to the screen shot above, with no errors anywhere in sight. Let's format the sucker, now; sudo mkfs.xfs /dev/md0

Depending on the total size of the RAID array and the speed of the system, this could take a while. When it's all done, you'll see something like the screenshot on the right. So far so good? I bet you're not even half-way through that favorite beverage!

Next, let's confirm that everything is running as it should. If you've configured RAID5 or 6, you've probably noticed that the drives are still busy even though we're not using them yet. This is normal, as the system is creating a recovery plan should anything go wrong with one of the drives later. You can check on the status of the array by typing cat /proc/mdstat. You should see something like this:

Ubuntu | cat /proc/mdstat

Now we need to create the mdadm.conf file. This is done with a single line: sudo mdadm —detail —scan > /etc/mdadm/mdadm.conf

Next we need a mount point for the RAID array. This can be thought of as a very large directory. I like to use things like "ebs", "volA" and "volB" to keep things simple, but you can call it anything you'd like. Let's make that mount point now: mkdir /volA

Then we add a line to the end of the /etc/fstab file. This file controls how drives are auto-mounted after a reboot. Type: /dev/md0 /volA xfs errors=remount-ro 1 2

Please remember to change /dev/md0, /volA, and xfs to the array, mount point, and file system you chose earlier, otherwise none of this will work when you reboot. Now you should be able to mount the RAID array by typing: mount -a. This command will mount anything listed in the fstab file that isn't already mounted.

If you've made it this far, congratulations. You're all done. Just to make sure that everything is running properly, do a quick cat /proc/mdstat and df -h. You should see something like this:

Ubuntu | Final mdadm Check

Not too bad, if I say so myself. The RAID array is still being checked, and we can see the five 2TB drives have been assembled into a single 8TB partition just waiting to be crammed full of wonderful data! But remember; this is just for academic purposes. There are several advantages to this sort of situation, and several disadvantages. At the end of the day, though, it really comes down to what you need the machine to do.

Using SATA Drives With Windows Home Server in Virtualbox

Have you ever needed to get Windows Home Server up and running in a virtual machine with a bunch of SATA disks attached? Have you spent hours on end looking for a proper SATA driver that will work with VirtualBox and deliver all those wholesome volumes to the instance? Well you're in luck, because I have what you need right here.

Intel's Storage Manager 7.8 (zip)

This is the floppy image containing the SATA drivers and, once installed, you'll have no trouble seeing SATA-attached drives, as you can see from the screenshot below.

Windows Home Server on VirtualBox

I'll explain this screen shot with its terabytes of storage a little bit more in the coming posts …

Eliminating an EC2 Auto-Scaling Group

Delete KeySo the time has come to eliminate an Auto-Scaling Group running on your Amazon EC2 account, but you're getting the run-around from the API regarding the order of operations.  How the heck can we kill the instances without starting another, and then eliminate the triggers that will start new servers?

Luckily, it's not too difficult.

There are four commands that we need call with the Amazon Auto-Scaling API, and they must be done in a very particular order.  So here we go:

  1. as-update-auto-scaling-group {AS Group Name} —min-size 0
  2. as-set-desired-capacity {AS Group Name} —desired-capacity 0
  3. as-delete-auto-scaling-group {AS Group Name}
  4. as-delete-launch-config {AS Config Name}

First we change the Auto-Scaling Group to allow for zero instances to run, then set 0 as the desired capacity.  This will automatically terminate any instances that are running, which will give us the ability to delete the Auto-Scaling Group (which won't disappear so long as there are instances).  Once the Group is gone, we can delete the Auto-Scaling Launch Configuration.

One extra item that you might have to add is the region where your systems are running.  If the servers are in US-West, EU West, or Asia-Pacific, be sure to add the —region {Region Name} switch to the end of every API call.

Done and done.

MySQL Server & High Availability

MySQL LogoHere's a question I'm asked often: how can you set up MySQL to be High Availability in a cheap and effective way? In my experience, one of the best ways to accomplish this is with the use of a Master-Master Replication setup across two servers.  This way if one machine drops off the network or goes down, the other will take it's place without missing a beat.  Additionally, if the server returns, it will automatically catch up and work alongside the other live one.

There are several ways to accomplish this, but the following tutorial will show one of the simplest ways to accomplish this goal.

Disclaimer: As will any tutorial, I am not responsible for any actions you do on your own servers. If something breaks, feel free to leave a comment and I'll do what I can to help out. It's assumed you have MySQL Version 5.x on your Linux-based system and you're comfortable enough with vi and other basic elements of the system.

With that out of the way, let's set up some replication!  We're going to have two servers with the following details:

  • Server_A - 10.0.0.10
  • Server_B - 10.0.0.11

On Server_A, add the following lines to your /etc/my.cnf file:

server-id=10

log-bin

binlog-do-db={Database to Replicate}

binlog-ignore-db=mysql

Next, still on Server_A, we'll need to create a replication account in MySQL:

mysql> CREATE USER 'repl'@'10.0.0.11' IDENTIFIED BY 'slave';

mysql> GRANT ALL ON *.* TO 'repl'@'10.0.0.11' IDENTIFIED BY 'slave';

Restart MySQL on Server_A.

Next we'll need to edit the /etc/my.cnf file on Server_B. Add the following lines, changing it as you need:

server-id=11

master-host = 10.0.0.10

master-user = repl

master-password = slave

master-port = 3306

So far it's pretty easy, right? Notice that the server-id value is different in the two files. Make sure that this number is unique throughout your network.

Restart MySQL on Server_B and then enter the following commands into MySQL:

mysql> start slave;

mysql> show slave status \G;

You should see output similar to the following:

*************************** 1. row ***************************

Slave_IO_State: Waiting for master to send event

Master_Host: 10.0.0.10

Master_User: replication

Master_Port: 3306

Connect_Retry: 60

Master_Log_File: mysqld-bin.000008

Read_Master_Log_Pos: 98

Relay_Log_File: mysqld-relay-bin.000012

Relay_Log_Pos: 236

Relay_Master_Log_File: mysqld-bin.000008

Slave_IO_Running: Yes

Slave_SQL_Running: Yes

Replicate_Do_DB:

Replicate_Ignore_DB:

Replicate_Do_Table:

Replicate_Ignore_Table:

Replicate_Wild_Do_Table:

Replicate_Wild_Ignore_Table:

Last_Errno: 0

Last_Error:

Skip_Counter: 0

Exec_Master_Log_Pos: 98

Relay_Log_Space: 236

Until_Condition: None

Until_Log_File:

Until_Log_Pos: 0

Master_SSL_Allowed: No

Master_SSL_CA_File:

Master_SSL_CA_Path:

Master_SSL_Cert:

Master_SSL_Cipher:

Master_SSL_Key:

Seconds_Behind_Master: 0

1 row in set (0.00 sec)

Two really important lines are Slave_IO_Running and Slave_SQL_Running. Notice how they both say "yes". If yours doesn't, then the replication isn't set up properly.  Double-check your settings and try again.

If you've ever configured replication on MySQL server you're probably saying to yourself "How is this different from normal replication?"  It's a valid question, because we now have a working Master/Slave configuration.  This next part is where the magic comes in.

Going from Master/Slave to Master/Master

On Server_B, add the following lines to the /etc/my.cnf file:

log-bin

binlog-do-db={Database to Replicate}

Create a replication slave on Server_B for Server_A:

mysql> CREATE USER 'repl'@'10.0.0.10' IDENTIFIED BY 'slave';

mysql> GRANT ALL ON *.* TO 'repl'@'10.0.0.10' IDENTIFIED BY 'slave';

I've used the same username and password for the sake of simplicity, but you can set them as whatever you'd like.

On Server_A, add the following lines to the /etc/my.cnf file:

master-host = 10.0.0.11

master-user = repl

master-password = slave

master-port = 3306

Looks easy, right? Oddly enough, it really is just this simple. Restart the MySQL servers on Server_A and Server_B. Now it's time to finish the configuration.

Server_A:

mysql> start slave;

Server_B:

mysql> show master status;

Server_A:

mysql> show slave status \G;

*************************** 1. row ***************************

Slave_IO_State: Waiting for master to send event

Master_Host: 10.0.0.11

Master_User: replication

Master_Port: 3306

Connect_Retry: 60

Master_Log_File: mysqld-bin.000008

Read_Master_Log_Pos: 98

Relay_Log_File: mysqld-relay-bin.000012

Relay_Log_Pos: 236

Relay_Master_Log_File: mysqld-bin.000008

Slave_IO_Running: Yes

Slave_SQL_Running: Yes

Replicate_Do_DB:

Replicate_Ignore_DB:

Replicate_Do_Table:

Replicate_Ignore_Table:

Replicate_Wild_Do_Table:

Replicate_Wild_Ignore_Table:

Last_Errno: 0

Last_Error:

Skip_Counter: 0

Exec_Master_Log_Pos: 98

Relay_Log_Space: 236

Until_Condition: None

Until_Log_File:

Until_Log_Pos: 0

Master_SSL_Allowed: No

Master_SSL_CA_File:

Master_SSL_CA_Path:

Master_SSL_Cert:

Master_SSL_Cipher:

Master_SSL_Key:

Seconds_Behind_Master: 0

1 row in set (0.00 sec)

There we go! Check to make sure that Slave_IQ_Running and Slave_SQL_Running are both Yes (on both servers), and you're set to go. Now you can create your tables, populate them, and get some work done.

Master-Master ReplicationThe AUTO_INCREMENT Collision Conundrum

There is one little thing that might bite you with busy servers, and that's index collisions. If a both servers have data recorded to the same table(s) at exactly the same time, there is a chance that different data will exist for the same index value, causing a collision when the servers try to update each other.  There are two ways to get around this.

The first is to not use AUTO_INCREMENT for primary keys. Either use unique identifiers or have a single system parameter table ready to provide index numbers whenever they're required.  The other is to change the AUTO_INCREMENT behavior.  Personally, I prefer this second option because it's quite easy to do.

Once again, let's open up the my.cnf file on Server_A and Server_B and add the following lines:

Server_A

auto_increment_increment=5

auto_increment_offset=1

Server_B

auto_increment_increment=5

auto_increment_offset=2

The first line tells MySQL to auto-increment tables by 5 rather than 1, and the second is the start number.  In this example, Server_A would receive values 1, 6, 11, 16, and so on, while Server_B would receive 2, 7, 12, 17, and so on.  This is not a perfect solution, but it certainly beats some of the alternatives … especially if you're running servers that are being hammered with read/write requests.

Easy, right? Well … maybe after some practice. But that's all there is to it.  The nice thing about this setup is that it works really well with load balancing systems and can help resolve some serious bottlenecks if you have people doing heavy reporting on live data.  Is it the best solution? Maybe not. But if you're on a tight budget, this can save you quite a bit compared to some of the other options.

Solving Ubuntu's 100% CPU Problem

Ubuntu Server 9.04 Install CDUbuntu is a pretty decent operating system for home servers, and it's one that I've used on my 10.6 TB media server for years.  System requirements are typically light, and having a *nix-based server in the house means you have one less thing that needs rebooting (usually).  That said, there are a few little hiccups with the latest and greatest version of Ubuntu when using Remote Desktop.

Kenji was telling me how he recently installed 9.04 (Jaunty Jackelope) on an old computer in his home and how he was using it as a Torrent and MP3 server.  He really enjoyed the ease of installation and said he was thinking about replacing Vista Ultimate with the open-source alternative.  While I doubt he could live without all his Windows-specific software, it would certainly be interesting to see a non-geek change all of their home computers to use a Linux variant.  However, despite the many things that Kenji liked about Ubuntu, there was one thing that annoyed the heck out of him: his server's processor was always running at 100%.

Luckily, I've had this problem before, too!

Replacing vino-server With Something Else

The problem seems to be with the vino-server process when Remote Desktop is enabled.  Disabling Remote Desktop will almost instantly bring a pegged processor from 100% usage down to 1-2% in less than a heart-beat, and will give that computer a new lease on life.

Except now you can't connect remotely.

The great thing about software, though, is that there are so many alternatives that provide essentially the same functionality.  I've been using NoMachine's free NX Server software for a few months now, as well as their Windows and Linux clients.  At the end of the day, we can get everything up and running with three simple commands:

sudo dpkg -i nxclient_3.2.0-14_i386.deb
sudo dpkg -i nxnode_3.2.0-13_i386.deb
sudo dpkg -i nxserver_3.2.0-16_i386.deb

The server service starts up automatically after being installed, and will auto-start after every boot.  The Windows client is dead simple to install and, aside from the occasional visual weirdness when working over a poor wireless connection, it's been a great tool for those rare instances where I need to connect to the server.

Kenji's Smiling Again

Since installing NX, Kenji's been happier than a pig in poo.  Considering how he's #367087 — Remote Desktop" href="https://bugs.launchpad.net/ubuntu/+source/remotedesk/+bug/367087" target="_blank">not the only one reporting this issue, hopefully the Ubuntu team will find some way to resolve the problem.  Until then, there are several other remote desktop solutions out there we can use.

Have you had this problem with Ubuntu or another flavor of Linux? How did you solve it?

Hosting Multiple Blogs With a Single WordPress Installation

This post is going to be the first in a short series outlining how anyone can take measures to simplify WordPress installations on shared or private hosting solutions, improving security and reducing maintenance time.

Getting Many from OneIt's a common problem: We have dozens of WordPress installations on our web server(s), and updating each and every site when a plugin is updated or a new release of WordPress becomes available is a huge time sink.  WordPress MU looks nice but installation is pretty complicated for most people and it's outright forbidden on most shared hosting accounts.  So what can we do to simplify software management?

Why not run each WordPress site from the very same source?

Madness, you say?  I disagree.  As of this post, I am running 9 web sites with a single installation of WordPress, and there will be another 8 added (a.k.a. rebuilt from backups) over the next week.

Here's how you do it.

Multi-Site, Multi-Domain, or Both?

The first thing we need to identify is whether you'll be hosting multiple sites (sub1.example.com, sub2.example.com, etc.) , multiple domains (www.domain1.com, www.domain2.com, etc.), or both configurations.  The reason this is important is because it will play a role in how we set up WordPress' config.php file and where the installation files are put.  For the sake of this example, I'll show you how to handle both multi-site and multi-domain.

First, you'll need to install WordPress to some public-facing directory on your server.  For example, www.YourDomain.com/wp/.  From there, install WordPress normally.  We'll be making the super-easy changes in a few minutes.

Note: If your main site (www.YourDomain.com) will be using the very same installation of WordPress, be sure to copy the index.php file in /wp/ to your website, making sure that you're pointing to the WordPress installation folder.  Here's an example:

/** Loads the WordPress Environment and Template */
require('./wp/wp-blog-header.php');

Be sure not delete index.php from the /wp/ folder. The original file will be used for every other site.


We're almost done the hard stuff!  Now it's time for the sweet stuff.  But, before getting too far ahead of ourselves, we need to answer a question:

Single Database or Multi-Database?

The nice little trick I'm about to show you will allow you to put every blog into a single database, or into multiple databases.  It's really up to you how you proceed.  For the sake of this example, I'm going to show you how to handle multiple databases, including shared databases.

Add the following code just before the line that reads define ('DB_NAME', 'YourDatabaseName'); :

$host = $_SERVER['HTTP_HOST'];
$parts = explode('.',$host);
if ($parts[3] = "") {
$domain = "www." . $parts[0];
} else {
$domain = $parts1;
}switch ($domain) {
case "www.YourDomain":
$pre = "wp0_";
$db = "DatabaseOne";
break;
case "sub1.YourDomain":
$pre = "wp1_";
$db = "SharedDatabase";
break;
case "sub2.YourDomain":
$pre = "wp2_";
$db = "SharedDatabase";
break;
case "www.DomainTwo":
$pre = "wp_";
$db = "DatabaseTwo";
break;
case "sub1.DomainTwo":
$pre = "sub1_";
$db = "DatabaseTwo";
break;
}

From here, we need to  put the variable names into the declaration lines like so:

define('DB_NAME', $db);
$table_prefix = $pre;

We're almost done!  But, before going further, you have probably noticed that each one of these three databases would need to have the same username and password.  If you want to have different User/Pass credentials for different databases, you can add more variables to the case statement.

Now the magic …

Adding Domains and Sub-Domains

When adding a WordPress-powered domain or sub-domain, simply point the home address to your WordPress installation folder.  For example:

www.YourDomain.com -> /
sub1.YourDomain.com -> /wp/
sub2.YourDomain.com -> /wp/
www.DomainTwo.com -> /wp/
sub1.DomainTwo.com -> /wp/

You're probably noticing that each of these domains will be using the very same wp-content/uploads directory for media.  If this is a problem, it can be changed in WordPress to something more your liking.  In my case, I created yet another sub-domain called "uploads" and pointed it to /wp/uploads/.  From there, each domain has their own subdirectory.  The advantage here is that all uploads are in their own little corner of the server and can be easily backed up or restored in the future.

Done and Done?

Weighing Pros and ConsSo, that's all there is to it.  This is not incredibly difficult to implement and comes with advantages such as being able to update every blog's core files and plugins simultaneously.  That said, there are some plugins that will not work well under this configuration.

If your sites experience high traffic volumes and you want to use WP Super Cache, it will only work on one site.  This is because the cache directory is set in wp-content/cache.  Unfortunately, this means that if two pages were to use WP Super Cache, they would write and access conflicting files.  I have also noticed some /tmp accessing problems while using an older version of FireStats, but I'll be testing the more current releases to see if they're more friendly to this configuration.

Of course, the biggst disadvantage to this setup would come if you are sharing sites with friends or family.  The newer versions of WordPress allow people to install plugins directly from the admin panel, and this can be a problem if one of these plugins contains a hidden exploit.  Rather than damange just one blog, it could damage many.  However, with proper education, you can make sure that other people do not cause trouble for everyone else.

So there you have it.  A quick and simple way to share one WordPress installation across multiple sites.  In the next installment of this series, I'll show you how to take multiple WordPress databases and merge them into one big database without doing several Import/Export functions through the WordPress admin screens.


  1. "." . $parts[1