Backup Script

I have multiple local servers and machines I want to backup to one central location. The criteria are that it has to be encrypted, both when transferring and storing, automatic and transparent, and redundant. This method is based on Open Source projects.

Previous Routines

A few years ago I found a great way to backup my servers to one central backup server using rsync and ssh. The backup server was located 100 meters away from the location with the servers, connected via wireless lan.

Every server was set up to use keys to login to the backup server without a password, making the process transparent. The rsync process was done over encrypted ssh, making it secure when transferring.

This method worked great and every few problems cropped up, other than lack of free space on backup drives. Both ends (clients and server) had a simple bash script that initiated the mounting of the drives, setting up ssh, including/excludign files, and rsyncing them over. Crontab took care of the scheduling.

Backup Routine

The new routing will be based on the previous method but simplified and the disks will be encrypted using Truecrypt.

For redundancy I will be using two 250 GB external USB-powered 2.5-inch harddrives, mine are from made by Western Digital. Both drives are identical and allows me to take one for backup at home (1) while leaving the other at work (2) to backup the servers/desktops.

When I see fit, I can swap the one at home (1) with the one at work (2) (and vice versa) and rsync will take care of the discrepancies by doing a incremental update. The incremental update will find all the new and modified files since the last time it saw that drive (1) and only copy over those files, while at the same time I have a redundant backup of all the files on the other drive (2). The 'freshness' of the redundant files depends on how often I swap out the drives (probably once a week).

Another issue I had was to find a filesystem that allowed for Linux permissions to be stored while still be accessibly by Mac OS X and Windows. After trying to get ext2/ext3 and NTFS to work, I found that, weirdly, HFS+ would be the most compatible filesystem. It was the only fs to have proper read and write, large file, and attribute/permission support. I didn't want to store backups on tar-balls.

Tools

All the tools used are Open Source and distributed free.

rsync
Truecrypt
- truecrypt-install
OpenSSH
ssh-keyinstall
- netcat
Debian Linux (etch)

Many of these tools have dependencies, but these are the majo components. Command-line/terminal will be heavily used to manage, and run this routine.

Encrypted Drive

For the backup server I choose the one that would be up and running the most, and the best secured (no Internet ssh logins allowed, etc).

The external hard drive was automatically recognized and the modules were loaded without a problem by the vanilla 2.6.18-4 Debian kernel. The drive came with one partition that was FAT32 formatted.

If the drive is not formatted, use fdisk to add a primary partition. It doesn't matter which filesystem the partition is formatted as it will be reformatted during the Truecrypt setup.

elitus kernel: usb 1-2: new full speed USB device using uhci_hcd and address 5
elitus kernel: usb 1-2: configuration #1 chosen from 1 choice
elitus kernel: scsi3 : SCSI emulation for USB Mass Storage devices
elitus kernel: usb-storage: device found at 5
elitus kernel: usb-storage: waiting for device to settle before scanning
~
elitus kernel:   Vendor: WD        Model: 2500BEV External  Rev: 1.04
elitus kernel:   Type:   Direct-Access                      ANSI SCSI revision: 04
elitus kernel: SCSI device sdb: 488397168 512-byte hdwr sectors (250059 MB)
elitus kernel: sdb: Write Protect is off
elitus kernel: sdb: Mode Sense: 21 00 00 00
elitus kernel: sdb: assuming drive cache: write through
elitus kernel: SCSI device sdb: 488397168 512-byte hdwr sectors (250059 MB)
elitus kernel: sdb: Write Protect is off
elitus kernel: sdb: Mode Sense: 21 00 00 00
elitus kernel: sdb: assuming drive cache: write through
elitus kernel:  sdb: sdb1
elitus kernel: sd 3:0:0:0: Attached scsi disk sdb
elitus kernel: usb-storage: device scan complete

Unfortunately, this motherboard and chipset only support USB v1.1.

Truecrypt

As of writing, there are no package maintainer for Truecrypt and it has to be compiled from source. Fortunately, some one has put together a handy truescript-installer script package that automates major parts of the process. Most of the ground work for this part was done by jaalto over at debian-administration.org.

Prepare Kernel

Prepare by installing the kernel sources, headers and kbuild.

uname -a
apt-cache search 'linux-source|linux-kbuild|linux-headers' # Find the matching ones

KVER=$(uname -r | sed 's/-.*//')
apt-get install linux-source-$KVER linux-kbuild-$KVER linux-headers-$KVER

Create the required kernel CPU architecture files by entering and exiting the kernel configuration menu.

cd /usr/src
tar -jxf linux-source-$KVER*.bz2
cd /usr/src/linux-source-$KVER
cp /boot/config-$KVER .config
make menuconfig
# Look at the bottom, select exit, and Yes when prompted to save the settings

Build Truecrypt

Now, to create the truecrypt binary, docs, and mobules, downloading and executing the truecrypt-installer package.

Go to http://debian.cante.net/truecrypt-installer and download the installer package compatible with the available Truecrypt version, as of writing truecrypt-installer_20071024-1_i386.deb for Truecrypt 4.3a.

wget http://cante.net/~jaalto/tmp/debian/truecrypt-installer/truecrypt-installer_20071024-1_i386.deb
dpkg -i truecrypt-installer_20071024-1_i386.deb
# Dependencies could be a problem, I had to install the following packages
apt-get install bzr debhelper dpatch html2text python-celementtree python-central python-elementtree python-support

Execute truecrypt-download and if it can't find the truecrypt source code, download it manually.

truecrypt-download
# Or
cd  /usr/src
wget http://tldp.etf.bg.ac.yu/gentoo/distfiles/truecrypt-4.3a-source-code.tar.gz

Now begin the build process. If it stops mid-way, try to delete all the truecrypt* files and directories in /usr/src and start from the beginning. The script relies on online trunk files to be available, which could pose a problem in the future.

truecrypt-build

If the process successfully finishes, install the resulting packages.

dpkg -i /usr/src/truecrypt-{cli,doc,modules-linux,modules-modprobe}*.deb

The install script will add the truecrypt module to /etc/modules so that it loads on boot.

Encrypting Drive

There is only one main binary called truecrypt. The easiest way to setup a new drive is to follow the simple guide.

truecrypt --quick --create /dev/sdb1

Volume type:
 1) Normal
 2) Hidden
Select [1]: 1

WARNING: Data on device will be lost. Continue? [y/N]: y
Filesystem:
 1) FAT
 2) None
Select [1]: 1

Hash algorithm:
 1) RIPEMD-160
 2) SHA-1
 3) Whirlpool
Select [1]: 1

Encryption algorithm:
 1) AES
 2) Blowfish
 3) CAST5
 4) Serpent
 5) Triple DES
 6) Twofish
 7) AES-Twofish
 8) AES-Twofish-Serpent
 9) Serpent-AES
10) Serpent-Twofish-AES
11) Twofish-Serpent
Select [1]: 1

Enter password for new volume '/dev/sdb1':
Re-enter password:

Enter keyfile path [none]:

TrueCrypt will now collect random data.

Please type at least 320 randomly chosen characters and then press Enter:
4qGSW(&/CXU...

Done: 0.00 MB  Speed: 0.00 MB/s  Left: 45222:49:57

I choose not to use keyfiles as I'd experiences some problems with them earlier, might work later though. The quick option finishes the format at the end in under a minute. I don't want to hang around waiting for the drive to format when I'm going to reformat it later, but I still understand that truecrypt normally randomizes the sectors while formatting the drive making it less prone to obscurity attacks.

Formatting Drive

To make the drive cross-platform and compatible with as many file attributions as possible, I reformat the drive with HFS+ aka. the Mac OS filesystem.

I used my Mac Mini to format the drive but it's possible to format the drive using MacDrive for Windows too.

Download the latest Truecrypt for Mac OS X (starting v5.0) and install it the normal way.
Plug in the USB drive and click 'Ignore' when prompted with an unrecognized drive.
Open Truecrypt and click 'Select drive'
Find the USB drive and click Mount.
Enter the truecrypt password for parition and it will automatically mount on the desktop.
Open 'Disk Utility' from the Applications -> Utilities folder.
On the left, pick the mounted drive from the bottom of the list, and click Erase.
Select 'HFS Extended' and not the journaled one, click 'Erease'.
The drive is now ready to be used as a cross-platform backup drive.

Linux already has a good hfsplus-module that allows read and write. For Windows, MacDrive seems to be a good choice.

Backup Scripts

There are two kinds of scripts to set up, one for the clients (rsync-backup-client) and one for the central backup server (rsync-backup-server). Both scripts are pretty easy to interpret.

Client

The client script:

#!/bin/bash
# Originally by William Stearns - http://www.stearns.org/rsync-backup/
# Edited/redesigned by ivc - http://beta.ivancover.com/wiki/
#
# Add a cron job like this:
#0 7 * * * root /usr/home/ivc/bin/rsync-backup/rsync-backup-client / backupuser@192.168.1.1:/
#
unset PATH # suggestion from H. Milz: avoid accidental use of $PATH

# Settings
BACKUPDIR=/usr/home/ivc/bin/rsync-backup-client/

# File paths
RM=/bin/rm
ECHO=/bin/echo
GREP=/bin/grep
LOGGER=/usr/bin/logger
TAR=/bin/tar
RSYNC=/usr/bin/rsync
CAT=/bin/cat
HOSTNAME=/bin/hostname
DATE=/bin/date
CHMOD=/bin/chmod

# USAGE
if [ "$1" == "" ]; then
        $ECHO -e '\n  Usage: ./rsync-backup-client <src> <src> ... <user@backuphost>:/\n'
        exit 1;
fi

# DEBUG
debug () {
        $ECHO -e "$*" >/dev/stderr
        #$LOGGER -t rsync-backup-client "$*"
}

# REMOTE BACKUP
debug
debug ' '`$DATE +"%Y-%m-%d %H:%M:%S"`' Starting backup routine'

RSYNC_ARGS="-avR -e ssh --numeric-ids --delete --delete-after \
 --rsync-path=/usr/home/ivc/bin/rsync-backup-server/rsync-backup-server \
 --exclude-from=./exclude \
 --exclude /proc/ --exclude /sys/ --exclude /dev/ --exclude /mnt/ \
 $@"

debug "   Running rsync $RSYNC_ARGS"
debug

cd $BACKUPDIR
$RSYNC $RSYNC_ARGS

debug
debug ' '`$DATE +"%Y-%m-%d %H:%M:%S"`' Finished backup routine'

Server

The server script:

#!/bin/bash
# Originally by William Stearns - http://www.stearns.org/rsync-backup/
# Edited/redesigned by ivc - http://beta.ivancover.com/wiki/
#
# To set up password-less login:
# ./ssh-keyinstall -s spinus -u root -c 'export SSH_CLIENT SSH2_CLIENT \; /usr/bin/nice /usr/home/ivc/bin/rsync-backup-server/rsync-backup-server server1'
#

unset PATH # suggestion from H. Milz: avoid accidental use of $PATH

# Settings
BACKUPDIR=/mnt/backup
BACKUPDEV=/dev/sda1

# File paths
RM=/bin/rm
MKDIR=/bin/mkdir
CP=/bin/cp
ECHO=/bin/echo
LOGGER=/usr/bin/logger
CHROOT=/usr/sbin/chroot
HOST=/usr/bin/host
AWK=/usr/bin/awk
TR=/usr/bin/tr
DATE=/bin/date
MOUNT=/bin/mount
UMOUNT=/bin/umount
TRUECRYPT=/usr/bin/truecrypt
SUDO=/usr/bin/sudo
RSYNC_STATIC=/usr/home/ivc/bin/rsync-backup-server/rsync-static

# DEBUG
debug () {
        #$ECHO -e "$*"
        $LOGGER -t rsync-backup-server "$*"
}

debug
debug ' '`$DATE +"%Y-%m-%d %H:%M:%S"`' Starting backup server script'

# GET CLIENTNAME
if [ -n "$1" ] && [ "$1" != "--server" ]; then
        CLIENTNAME="$1"
        debug "  Using passed clientname: ${CLIENTNAME} (from /root/.ssh/authorized_keys2)"
elif [ -n "$SSH_CLIENT" ]; then
        CLIENTNAME=`$HOST ${SSH_CLIENT%% *} | $AWK '{print $5}' | $TR '.' ' ' | $AWK '{print $1}'`
        debug "  Using IP address from SSH1/OpenSSH shell environment - clientname: ${CLIENTNAME}"
elif [ -n "$SSH2_CLIENT" ]; then
        CLIENTNAME=`$HOST ${SSH2_CLIENT%% *} | $AWK '{print $5}' | $TR '.' ' ' | $AWK '{print $1}'`
        debug "  Using IP address from SSH2 shell environment - clientname: ${CLIENTNAME}"
else
        debug '  No passed clientname and null SSH_CLIENT and SSH2_CLIENT,'
        debug '  where do I store the backup?  Exiting'
        exit 1
fi

# GET COMMAND SENT FROM CLIENT
if [ -n "$SSH_ORIGINAL_COMMAND" ]; then
        debug "  Passed SSH command: $SSH_ORIGINAL_COMMAND"
elif [ -n "$SSH2_ORIGINAL_COMMAND" ]; then
        debug "  Passed SSH2 command: $SSH2_ORIGINAL_COMMAND"
else
        debug '  Not passed a command'
fi

# MOUNT
if [ ! -f "$BACKUPDIR/checker-file" ]; then
        debug "  Mouting backup device $BACKUPDEV to $BACKUPDIR via truecrypt"
        $TRUECRYPT -p <truecryptpassword> $BACKUPDEV $BACKUPDIR;
        #$MOUNT -t hfsplus $BACKUPDEV $BACKUPDIR;
        if (( $? )); then {
                debug '  Mounting device failed. Exiting'
                exit 1
        } fi;
fi

# SETTING UP FILES AND DIRECTORIES
if [ ! -f "$BACKUPDIR/checker-file" ]; then
        debug '  Making checker-file'
        $SUDO $ECHO "Yep, this is the backup drive" > $BACKUPDIR/checker-file;
fi

if [ ! -d "$BACKUPDIR/$CLIENTNAME/current/" ]; then
        debug "  Creating necessary $BACKUPDIR/$CLIENTNAME/current/ directories"
        $SUDO $MKDIR -p "$BACKUPDIR/$CLIENTNAME/current/"
fi

if [ ! -f "$BACKUPDIR/checker-file" ]; then
        debug '  Copying rsync binary for chroot jail'
        $SUDO $CP -p "$RSYNC_STATIC" "$BACKUPDIR/$CLIENTNAME/"
fi

debug '  Initilizing rsync-static server in chroot jail...'
# rsync parameters:
#        -l, --links                 copy symlinks as symlinks
#        -o, --owner                 preserve owner (root only)
#        -g, --group                 preserve group
#        -D, --devices               preserve devices (root only)
#        -t, --times                 preserve times
#        -p, --perms                 preserve permissions
#        -r, --recursive             recurse into directories
#        -R, --relative              use relative path names
#            --delete                delete files that don't exist on sender
#            --delete-after          receiver deletes after transfer, not before
#        -a is equivalent to -rlptgoD
$SUDO $CHROOT "$BACKUPDIR/$CLIENTNAME/" /rsync-static --server -logDtprR --delete --numeric-ids . /current/

debug "  Done doing backup from ${CLIENTNAME} to $BACKUPDIR/$CLIENTNAME/current"

# UNMOUNT
debug "  Unmounting backup device $BACKUPDEV via truecrypt"
#$UMOUNT $BACKUPDIR;
$TRUECRYPT -d

debug ' '`$DATE +"%Y-%m-%d %H:%M:%S"`' Finished backup server script'

Password-less Login

To automate the remote backup to a central server, login has to be password-less. A good way to do that is to set up ssh to login using keys.

Download the ssh-keyinstall script by William Stearns.
Make it executable, chmod +x ssh-keyinstall, and run ssh-keyinstall -s <192.168.1.1> -u <backupuser> -c 'export SSH_CLIENT SSH2_CLIENT \; /usr/bin/nice /usr/home/ivc/bin/rsync-backup-server/rsync-backup-server server1'
The script will ask you to provide the password for the specified user to transfer and execute commands on the remote server.
Once the public keys and settings are set up, you should be able to run the command at the end of the process without being prompted for a login password.
Next time a remote client logs in to the <backupuser> on <192.168.1.1>, only the command specified in ~/.ssh/authorized_keys2 on the server will be executed. The rsync-backup-server script takes only one argument, that is the name of the backup directory where rsync will put the files, e.g argument 'server1' will put files in /mnt/backup/server1/current/.

Backup Script

Contents

Previous Routines

Backup Routine

Tools

Encrypted Drive

Truecrypt

Prepare Kernel

Build Truecrypt

Encrypting Drive

Formatting Drive

Backup Scripts

Client

Server

Password-less Login

References

Navigation menu

Page actions

Page actions

Personal tools

Navigation

Search

Tools