Incremental Backups

Jun 12th, 2010No Comments

Keeping copies of one’s system just in case something breaks has been a long-time bane of sysadmin’s lives, albeit as hardware has moved on things have become easier. The days of tape backups are gone for most of us and the problem is reduced to one of keeping mirrored disk images of our systems.

However, great as mirror copies are, more often than not it’s not the last copy you want to restore, in the event of a file corruption that’s gone un-noticed for a while, your backup may well contain the corrupted file and be useless to you. Once you decide to keep lots of copies of your system, storage becomes a problem, and once you decide to keep selective copies of your system, there’s always the chance that you’ll miss something you want.

Enter incremental backups

Although there are many software packages out there which will make a meal of your backups, they tend either to be expensive or not very good – certainly from the home user’s perspective, so here’s how we do it very simply and very reliably, for free.

First we need a list of files / directories we really don’t want to make copies of, let’s put this into /etc and call it backup.exclude;

[plain]
/proc
/sys
/tmp
/var/tmp
/var/log
/var/cache/apt/archives
[/plain]

Now we need a backup script, let’s call it incremental_backup.sh and put it in /usr/local/bin;

[bash]
#!/bin/bash
export RSYNC_PASSWORD=”mypassword”
export PATH=$PATH:/bin:/usr/bin:/usr/local/bin
HOST=`hostname`
DEST=”<backup server>”
SPEED=2048
BACK=`date “+%A_%d_%B_%Y_%H”`
EXCL=”/etc/backup.excludes”
OPTS=”–force –ignore-errors –delete-excluded –exclude-from=$EXCL –bwlimit=$SPEED –delete –backup –backup-dir=/$BACK -rvlpgoDtO”
rsync $OPTS / root@$DEST::$HOST/current
[/bash]

To activate this automatically at midnight every day, use crontab -e and add the following line;

[bash]

0 0 * * *   /usr/local/bin/incremental_backup.sh

[/bash]

Not all we need to do is make sure we have a backup server to copy our files to, start with a password file. Edit /etc/rsync/secrets and insert a line that says;

[bash]

root:mypassword

[/bash]

Then make sure that only the root user can read it with;

[bash]

chmod 700 /etc/rsync/secrets

[/bash]

Not move onto the final part, edit /etc/rsyncd.conf and create a file as follows;
[bash]
use chroot = yes
hosts allow = 0.0.0.0/0
secrets file = /etc/rsync/secrets
read only = no
uid = root
auth users = root
[<my hostname>]
path = <backup storage>
[/bash]

The bits you need to watch for are;

  • <my hostname> in rsyncd.conf needs to match up with the name of the machine you will be backing up, find it by typing hostname on the command line.
  • <backup storage> is where you want to store the copies of your files
  • You may want to change “mypassword” if security is an issue, this needs to change in the secrets file on the backup server, and in the incremental script on the machine you are backing up.
  • You may also want to limit hosts allow to the IP range of the machine(s) you want to back up

How does it work?

On the first pass the script will make a copy of everything on your machine not listed in the backup.exclude file, and store it on the server in the location specified, under a folder called current. If you ever need to restore the most recent backup of any file, this is the place to look, current should always be a complete and up-to-date snapshot of the system at the time of the last backup.

On the second (and subsequent passes) the script will make copies of any files that have changed since the last backup, storing them again in the current folder. The previous version of each file is however not overwritten but moved into an archive folder who’s name will be the date of the backup, so for example you end up with a folder that looks like this;

[plain]

# ls
current Saturday_05_June_2010_03
Thursday_10_June_2010_03 Friday_11_June_2010_03
Saturday_12_June_2010_03 Tuesday_08_June_2010_03

[/plain]

So you can find historical versions of any file simply by looking in the folder with the required date on it. This is of course one bit missing, these incremental folders will accumulate and eventually fill your disk, so you need to do some housekeeping. Use your crontab -e again and this time add;

[bash]

0 0 * * *  find <backup storage> -maxdepth 2 -name “*_2010_*” -ctime +10 -ls -exec rm -rfv {} \;

[/bash]

Which will ensure you will only keep the last 10 days worth of incremental backups.

Prerequisites

In order to make this work you will need to have the following packages installed on your systems;

  • ssh (openssh-client+openssh-server)
  • rsync (on the machines to badk up
  • rsyncd (ont he backup server)
avatar
About author:

All entries by

Leave a Reply

You must be logged in to post a comment.