Offsite backups with rsync.net

If you don't do backups... well, you should.

An example of a good backup system could be:

• External drive backups: your computer is backed up at an external drive;
• CD/DVD backups: burn your backups to a CD or DVD;
• Offsite backups: your backups are located at a different site, ie, another place other than your home or office.

A good idea is to have all three backup systems in place. External drivers can brake and DVDs are misplaced, so a offsite location for your backups are a good idea.

I started using rsync.net for my offsite backups. It's a paid service, but it gives you monthly payments for GB (minimum 7 GB) and ssh, ftp, sftp, webdav, https, and... rsync. Check the site for full features.

A great way to use rsync.net for your backups is using duplicity. Duplicity is an encrypted bandwidth-efficient backup that uses the rsync algorithm.

I will show you how I set up my system to create encrypted backups in a rsync.net server.

More info about this setup can be seen at rync.net own site.

Installing duplicity

$ sudo apt-get install duplicity

Generate an ssh key to connect to rsync.net

• Generate the ssh key using the defaults (don't use a password when asked):

$ ssh-keygen -t rsa

• Copy your key to rsync.net:

$ scp ~/.ssh/id_rsa.pub user@server.rsync.net:.ssh/authorized_keys

• If you need to add more keys from different computers, generate the key and:

 $ cat ~/.ssh/id_rsa.pub | ssh user@server.rsync.net 'dd of=.ssh/authorized_keys oflag=append conv=notrunc'

Generate a gpg public and private key to encrypt your backups

• Generate the gpg keys (get down your private key, example: 123AbcH123BB4321):

$ gpg --gen-key

• You can check the keys in your system:

$ gpg --list-keys

• Look in the output something like this. Look for the public key (in this example your public key is 1AAB123A)

pub   1239A/1AAB123A 2011-10-09 [expires: 2012-10-08]
uid                  Your Name (name) <mail@mail.com>
sub   54321/1234BBCC 2011-10-09 [expires: 2012-10-08]

Bash scripts to backup your files

I created two bash scripts, one that implements what I want to do with duplicity, and another one that uses the first one to backup my directories.

It uses the following duplicity commands (more at duplicity website):

• Clean up and remove older backups

$ duplicity cleanup -v9 --encrypt-key="1AAB123A" scp://user@server.rsync.net/bck_dir

$ duplicity remove-older-than 30D -v9 --encrypt-key="1AAB123A" scp://user@server.rsync.net/bck_dir

• Backup (full or inc)

$ duplicity full --encrypt-key="1AAB123A" /home/user/Documents scp://user@server.rsync.net/bck_dir

• Verify the backup

$ duplicity verify --encrypt-key="1AAB123A" scp://user@server.rsync.net/bck_dir /home/user/Documents

First script: using duplicity to backup a given directorie

Notes about this script:

• Has 4 parameters:
• backup type: inc for an incremental backup, and full for a full backup
• backup dir: origin for the backup
• destination dir: rsync.net directorie
• exclude filename: a filename to exclude
• Careful: Has the private key hardcoded in the script (!)
• It starts by cleaning up and remove older backups
• Performs a full or incremental backup
• Verify the backup

In the script you need to change the following to fit your needs:

• The SERVER variable
• The SCP_SERVER variable
• The PUB_KEY variable
• The PASSPHRASE variable (this is your private key)
• In the remove-older-than command you may want to change the number of days until remove the backup

rsync.sh

#!/bin/bash ############################################################### # rsync.sh # # Parameters: # $1 - backup type ( full | inc ) # $2 - ori dir # $3 - backup dir # $4 - exclude filename (example: "file.zip") # # example: rsync.sh full /home/user/mydir backups/bckdir file.zip # ############################################################### BCK_TYPE=$1 ORI_DIR=$2 DEST_DIR=$3 if [[ -z $4 ]]; then EXCL="" else EXCL="--exclude **"$4 fi SERVER=user@server.rsync.net SCP_SERVER=scp://user@server.rsync.net PUB_KEY=1AAB123A # secret export PASSPHRASE=123AbcH123BB4321 RUNNING_SCRIPT=`basename "$0"` CUR_DIR=`echo "$0" | awk -F"$RUNNING_SCRIPT" '{ print $1 }'` LOG=${CUR_DIR}/logs/`date "+%Y%m%d"`.`echo "${DEST_DIR}" | awk -F'/' '{ print $2 }'`.${BCK_TYPE}.log # create backup dir if ssh ${SERVER} test -d ${DEST_DIR}; then echo "'"${DEST_DIR}"' already exists at '"${SERVER}"'" >> $LOG else ssh ${SERVER} mkdir ${DEST_DIR} fi # -- BACKUP -- echo "" >> $LOG echo "<<<<<<========================================INI====================================>>>>>>" >> $LOG echo "" >> $LOG echo "==============================> backing up '"${ORI_DIR}"' to '"${DEST_DIR}"'" >> $LOG echo ">" `date "+%Y-%m-%d %H:%M:%S"` >> $LOG # clean up echo "" >> $LOG echo "==============================> cleanup ('"${ORI_DIR}"')" >> $LOG duplicity cleanup -v9 --encrypt-key="${PUB_KEY}" ${SCP_SERVER}/${DEST_DIR} >> $LOG echo "" >> $LOG echo "==============================> remove-older-than 30D ('"${ORI_DIR}"')" >> $LOG duplicity remove-older-than 30D -v9 --encrypt-key="${PUB_KEY}" ${SCP_SERVER}/${DEST_DIR} >> $LOG # backup echo "" >> $LOG echo "==============================>" ${BCK_TYPE} "backup ('"${ORI_DIR}"')" >> $LOG duplicity ${BCK_TYPE} --encrypt-key="${PUB_KEY}" "${ORI_DIR}" ${EXCL} ${SCP_SERVER}/${DEST_DIR} >> $LOG # verify backup echo "" >> $LOG echo "==============================> verify ('"${ORI_DIR}"')" >> $LOG duplicity verify --encrypt-key="${PUB_KEY}" ${SCP_SERVER}/${DEST_DIR} "${ORI_DIR}" >> $LOG # rsync.net quota echo "" >> $LOG echo "==============================> rsync.net quota" >> $LOG ssh ${SERVER} quota >> $LOG echo "" >> $LOG echo "<<<<<<========================================END====================================>>>>>>" >> $LOG echo "" >> $LOG #eof



Second script: use the previous script to backup your directories


Notes about this script:

• It uses the notify-send command for desktop notifications

$ sudo apt-get install libnotify-bin

• It's divided into groups that run in parallel
• It only have one parameter: if it's a full backup or an inc one

In the script you need to change the following to fit your needs:

• The BCK_MAIN_DIR variable (it's the directorie in rsync.net where your backups go; it will be created automatically)
• In each group add as much func_run_duplicity functions as you want, changing the folder/file to backup and the name of the folder to keep that backup in rsync.net (this folder will be created automatically - example backups/BCK1)
• You can add new groups
• In Group 2 there is an example of a file to exclude from the backup

run_rsync.sh

#!/bin/bash ############################################################### # run_rsync.sh # # Parameters: # <type> full | inc # # example: run_rsync.sh full # ############################################################### BCK_TYPE=$1 BCK_MAIN_DIR=backups RUNNING_SCRIPT=`basename "$0"` CUR_DIR=`echo "$0" | awk -F"$RUNNING_SCRIPT" '{ print $1 }'` MAIN_LOG=${CUR_DIR}/logs/`date "+%Y%m%d"`.${BCK_TYPE}.log # ------------------------------------------------------------- # func_run_duplicity () # # function that runs duplicity # # Parameters: # $1 - group # $2 - ori dir # $3 - backup dir # $4 - exclude # ------------------------------------------------------------- func_run_duplicity() { GROUP=$1 ORI_DIR=$2 BCK_DIR=${BCK_MAIN_DIR}/$1$3 EXCL=$4 LOG=${CUR_DIR}/logs/`date "+%Y%m%d%H%M%S"`.`echo "${BCK_DIR}" | awk -F'/' '{ print $2 }'`.${BCK_TYPE}.log echo "Group" $GROUP "INIT:" ${ORI_DIR} `date "+%Y-%m-%d %H:%M:%S"` >> ${MAIN_LOG} $CUR_DIR/rsync.net.sh \ ${BCK_TYPE} \ ${ORI_DIR} \ ${BCK_DIR} \ ${EXCL} wait echo "Group" $GROUP "END:" ${ORI_DIR} `date "+%Y-%m-%d %H:%M:%S"` >> ${MAIN_LOG} } echo "<<<<<<========================================INI====================================>>>>>>" >> ${MAIN_LOG} notify-send "Running $BCK_TYPE rsyn.net duplicity sync" { echo "Group 1 INIT" `date "+%Y-%m-%d %H:%M:%S"` >> ${MAIN_LOG} func_run_duplicity 1 "/home/user/FolderA" "BCK1" func_run_duplicity 1 "/home/user/FolderB" "BCK2" echo "Group 1 END" `date "+%Y-%m-%d %H:%M:%S"` >> ${MAIN_LOG} } & { echo "Group 2 INIT" `date "+%Y-%m-%d %H:%M:%S"` >> ${MAIN_LOG} func_run_duplicity 2 "/home/user/FolderC" "BCK3" "File.zip" echo "Group 2 END" `date "+%Y-%m-%d %H:%M:%S"` >> ${MAIN_LOG} } & wait echo "<<<<<<========================================END====================================>>>>>>" >> ${MAIN_LOG} notify-send "$BCK_TYPE rsyn.net duplicity sync ended" #eof

Setting up the scripts and running them automatically

Create a folder for your tow scripts and a backup folder:

$ mkdir ~/rsync.net
$ mkdir ~/rsync.net/logs

I use cron to run a daily inc backup and weekly full backup (Sundays):

$ crontab -e

Add to cron:

00 21 * * * ~/rsync.net/run_rsync.sh inc
00 16 * * 0 ~/rsync.net/run_rsync.sh full

Restoring your backups

To restore your backups:

$ mkdir ~/restore
$ duplicity --encrypt-key="1AAB123A" scp://user@server.rsync.net/backups/BCK1 ~/restore

Please tell me about your setup, experiences and suggestions!