Sunday, October 9, 2011

Offsite backups with rsync.net

If you don't do backups... well, you should.

An example of a good backup system could be:
  • External drive backups: your computer is backed up at an external drive;
  • CD/DVD backups: burn your backups to a CD or DVD;
  • Offsite backups: your backups are located at a different site, ie, another place other than your home or office.
A good idea is to have all three backup systems in place. External drivers can brake and DVDs are misplaced, so a offsite location for your backups are a good idea.

I started using rsync.net for my offsite backups. It's a paid service, but it gives you monthly payments for GB (minimum 7 GB) and ssh, ftp, sftp, webdav, https, and... rsync. Check the site for full features.

A great way to use rsync.net for your backups is using duplicity. Duplicity is an encrypted bandwidth-efficient backup that uses the rsync algorithm.

I will show you how I set up my system to create encrypted backups in a rsync.net server.

More info about this setup can be seen at rync.net own site.

Installing duplicity
$ sudo apt-get install duplicity

Generate an ssh key to connect to rsync.net
  • Generate the ssh key using the defaults (don't use a password when asked):
$ ssh-keygen -t rsa
  • Copy your key to rsync.net:
$ scp ~/.ssh/id_rsa.pub user@server.rsync.net:.ssh/authorized_keys
  • If you need to add more keys from different computers, generate the key and:
 $ cat ~/.ssh/id_rsa.pub | ssh user@server.rsync.net 'dd of=.ssh/authorized_keys oflag=append conv=notrunc'

Generate a gpg public and private key to encrypt your backups
  • Generate the gpg keys (get down your private key, example: 123AbcH123BB4321):
$ gpg --gen-key
  • You can check the keys in your system:
$ gpg --list-keys
  • Look in the output something like this. Look for the public key (in this example your public key is 1AAB123A)
pub   1239A/1AAB123A 2011-10-09 [expires: 2012-10-08]
uid                  Your Name (name) <mail@mail.com>
sub   54321/1234BBCC 2011-10-09 [expires: 2012-10-08]

Bash scripts to backup your files

I created two bash scripts, one that implements what I want to do with duplicity, and another one that uses the first one to backup my directories.

It uses the following duplicity commands (more at duplicity website):
  • Clean up and remove older backups
$ duplicity cleanup -v9 --encrypt-key="1AAB123A" scp://user@server.rsync.net/bck_dir
$ duplicity remove-older-than 30D -v9 --encrypt-key="1AAB123A" scp://user@server.rsync.net/bck_dir
  • Backup (full or inc)
$ duplicity full --encrypt-key="1AAB123A" /home/user/Documents scp://user@server.rsync.net/bck_dir
  • Verify the backup
$ duplicity verify --encrypt-key="1AAB123A" scp://user@server.rsync.net/bck_dir /home/user/Documents

First script: using duplicity to backup a given directorie

Notes about this script:
  • Has 4 parameters:
    • backup type: inc for an incremental backup, and full for a full backup
    • backup dir: origin for the backup
    • destination dir: rsync.net directorie
    • exclude filename: a filename to exclude
  • Careful: Has the private key hardcoded in the script (!)
  • It starts by cleaning up and remove older backups
  • Performs a full or incremental backup
  • Verify the backup
In the script you need to change the following to fit your needs:
  • The SERVER variable
  • The SCP_SERVER variable
  • The PUB_KEY variable
  • The PASSPHRASE variable (this is your private key)
  • In the remove-older-than command you may want to change the number of days until remove the backup
rsync.sh





Second script: use the previous script to backup your directories


Notes about this script:
  • It uses the notify-send command for desktop notifications
$ sudo apt-get install libnotify-bin
  • It's divided into groups that run in parallel
  • It only have one parameter: if it's a full backup or an inc one
In the script you need to change the following to fit your needs:
  • The BCK_MAIN_DIR variable (it's the directorie in rsync.net where your backups go; it will be created automatically)
  • In each group add as much func_run_duplicity functions as you want, changing the folder/file to backup and the name of the folder to keep that backup in rsync.net (this folder will be created automatically - example backups/BCK1)
  • You can add new groups
  • In Group 2 there is an example of a file to exclude from the backup
run_rsync.sh




Setting up the scripts and running them automatically

Create a folder for your tow scripts and a backup folder:
$ mkdir ~/rsync.net
$ mkdir ~/rsync.net/logs
I use cron to run a daily inc backup and weekly full backup (Sundays):
$ crontab -e
Add to cron:
00 21 * * * ~/rsync.net/run_rsync.sh inc
00 16 * * 0 ~/rsync.net/run_rsync.sh full

Restoring your backups

To restore your backups:
$ mkdir ~/restore
$ duplicity --encrypt-key="1AAB123A" scp://user@server.rsync.net/backups/BCK1 ~/restore

Please tell me about your setup, experiences and suggestions!