The Chillidog Software Blog

The ramblings of a developer

90 days of Backups on Ice

Every year we should try to improve and do a little better than the last. That is why, effective January 1, 2013, we will be improving our backup strategy to include 90 days worth of retention. Let's take a few minutes to review our backup strategy.

Server snapshots (daily & weekly)

Every day, a snapshot of the entire server is made. This provides us with a mechanism to fully recover the server in case something catastrophic happens. This backup is meant as a last resort and is not accessed or used unless really needed.

Incremental backups (every 4 hours)

Every 4 hours, all files that have been changed are backed up onsite. This means that any file that has been around for longer than 4 hours can be recovered and restored. It does not, however, include databases or mail messages.

Offsite backups to S3 & Glacier (daily)

Every night, a full account backup is performed for all users on Javabean Hosting. This backup includes all files, email, and databases in the user's account. The backup is then shipped off to Amazon S3 for safe keeping. This is where the changes are being made.

Current settings:

  • Offsite backup are currently kept on Amazon S3 for 30 days.

New settings:

  • Offsite backups are going to be kept on Amazon S3 for 72 hours.
  • Offsite backups stored on Amazon Glacier for 90 days.

What does this mean for you?

These changes mean that, to restore a backup older than 72 hours, it will take longer for us to recover. Unlike S3 where access is instant, the time it takes to recover files from Amazon Glacier is currently 3 - 5 hours. Once the files are retrieved, it may take us another 3 - 5 hours to fully restore your account.

Why the change?

Backups are essential and the more there are, the better. However, every time we have to restore files, we're always recovering from a version that is less than 24 hours old. As a result, we find that we don't need to retain immediate access to 30 days worth of backups. Instead, we believe we can maintain a balance of retention and availability by keeping recent backups in S3 and longer term backups in Glacier.