Automated Backup System

[Spanky]

Hey guys,

I am interesting in setting up an automated backup facility. I haven't set anything like this up before but I do want to make sure my data is secure.

I basically want to be able to do the following:

- issue commands to Perforce to tell it to create its backup files
- zip/rar the selected backup files produced above
- zip/rar up two specified directories on my drive
- run another tool to generate checksums/recovery files for the rars
- upload all of these files into a new directory on my FTP server on the web.

I would love to be able to do this with a GUI as I hate writing scripts. I would also like to be able to tell it to run this every XX hours/days and to send an email to me if a certain step goes wrong.

I've tried FinalBuilder and it appears to support everything I wanted to do but it's $300 which I can't really afford. The trial worked out great but that's run out now.

I'm looking for any suggestions to tools/programs/scripts that people use to do their own backup procedures or things they know about.

Thanks
Shawn

[steven]

Check UltraBackup it does everything you want.

It is free, makes encrypted, compressed backup, pre- and post- scripts, automatic usb activation, mirror mode (very useful), network authentification, cron jobs, etc etc

Only drawback is works only under win2000 or higher and -perhaps- it is in french.

http://astase.com/produits/ultrabackup/2007/
http://astase.com/produits/ultrabackup/2007/captures/

There is even a server solution for small enterprise or soho:
http://astase.com/produits/ultrabackup/netstation/

[Klaim]

When i searched for a similar solution i heard a lot about Bacula but it seemed a bit too much for my needs so finally a wrot a some scripts. I guess i'll try that when i'll change my server.

[sinbad]

I wrote a bit about internet-based backup services recently on my blog here: http://www.stevestreeting.com/2008/02/1 ... p-service/

In the end I just used Amazon S3 direct and wrote some scripts, since it costs me mere pence every month that way, but I mentioned a bunch of other services there too. JungleDisk was a pretty good option I thought, if you wanted something with a GUI. I think Amazon S3 is a good option even if you have your own servers since you're charged on a very fine granularity and can scale as high as you want.

[Spanky]

Hey guys,

Thanks for the responses. I can't check them out too much right now as I'm currently back at my mothers place for the night. I'm moving out of university (finally) and into my first real apartment with my fiance. We've got to pack up 4 different places tomorrow and about 5 hours of driving. Woo haa).

Unfortunately, it looks like I can't use Amazon S3 as I'm in Canada. It seems like a decent option though. I'll have to check out UltraBackup. I've got a French speaking friend so I can get any translations I need.

As for the scripts. What language are you guys using and how did you hook them up to run? I'm using a windows box and I'll have an experimental one with Linux on it but I need it to run on Windows.

Also, do you have any type of error detection or notification? I think it'd be nice to have an email shot off if, for some reason, the procedure didn't go too smoothly. FinalBuilder had this which made my life a lot easier. I find I often check up on things to make sure it's happening all the time but I would just rather get a daily email or something about something failing or being done successfully.

Shawn

[steven]

UltraBackup. I've got a French speaking friend so I can get any translations I need.

They are planning a localization but it will take some time to enable it for any language.

As for the scripts. What language are you guys using and how did you hook them up to run?

I use simple batch scripts (.bat) but you can use anything you like even exe (hence c++).

Btw the same team made a script editor: PowerBatch
http://astase.com/produits/powerbatch/

Which includes an excellent batch tutorial (in french).
Seems I make a lot of publicity for them

Also, do you have any type of error detection or notification?

UltraBackup has email reporting.

[Spanky]

Awesome replies guys.

The link for UltraBackup ended up leading me to SyncBack which is almost exactly what I need. It's a GUI based tool that allows me to run a script before and after it backs up a folder. I can do all my management in those scripts and then use the program to send out the emails and what not on failure.

I was also looking at S3 from Amazon again and I was incorrect earlier. Apparently anyone can use it but the servers are only in the US and Europe. I thought that it said only people from there could use it.

Would anyone be willing to share their batch files? I'm particularly interested in the S3 ones if anyone *cough* Sinbad *cough* is willing to share Why re-invent the wheel when someone probably already has what I want eh? (That and I am kind of lazy...)

I've taken a look at JungleDisk and may be able to rig something up with that. I like what it does but I need to run something before to perform my actual backup through perforce. I may use task scheduler and just execute it an hour before JungleDisk runs. Not great but I don't change stuff that often (every few days).

Thanks
Shawn

[Game_Ender]

Would anyone be willing to share their batch files? I'm particularly interested in the S3 ones if anyone *cough* Sinbad *cough* is willing to share Why re-invent the wheel when someone probably already has what I want eh? (That and I am kind of lazy...)

Sinbad is a Mac user now .

[sinbad]

Ok, bear in mind my bash scripting is rudimentary, but this does the business for me.

s3archive.sh tars, bzips and gpg encrypts a folder, including doing incremental archives.

Code: Select all

#!/bin/bash

# Archives a given folder ready for uploading to S3
# Incremental backups available via backup level 1
# Encryption is encouraged via the -e option

S3BACKUPDIR=/home/steve/s3backups
SYNCDIR=$S3BACKUPDIR/sync
TMPDIR=$S3BACKUPDIR/tmp
SNARDIR=$S3BACKUPDIR/snar

if [[ $# == 0 ]]
then
	echo "Usage: "
	echo "  $0 [opts] <source> <backup level>"
	echo "Valid options:"
	echo "  -e <name>"
	echo "     Encrypt the resulting archive to GPG recipient"
	exit
fi

# Deal with optional named arguments
while getopts "e:" Opt
do
	case $Opt in
		e) GPGRECIPIENT=$OPTARG;;
	esac
done
shift $(($OPTIND - 1))


if [[ ! -e "$1" ]]
then
	echo "First parameter needs to be source folder"
	exit
fi

SOURCE=$1
#Replace all slashes with percent signs for name
NAMEBASE=${SOURCE//\//%}

case "$2" in
	"0")
		NAMEBASE=${NAMEBASE}_base
		INCREMENTAL_FILE=$SNARDIR/${NAMEBASE}.snar
		rm $INCREMENTAL_FILE
		;;
	"1")
		INCREMENTAL_FILE=$SNARDIR/${NAMEBASE}_inc.snar
		cp $SNARDIR/${NAMEBASE}_base.snar $INCREMENTAL_FILE
		NAMEBASE=${NAMEBASE}_inc
		;;
	*)
		echo "Backup level needs to be 0 or 1"
		exit;
		;;
esac

TARDEST=$TMPDIR/$NAMEBASE.tar.bz2
# Append parameter to name to differentiate incremental


# Create archive
rm -f $TARDEST
tar --listed-incremental=$INCREMENTAL_FILE -cvjf $TARDEST $SOURCE

# Encrypt?
if [[ "$GPGRECIPIENT" != "" ]]
then
	gpg -e -r $GPGRECIPIENT $TARDEST
	FINALFILE=$TARDEST.gpg
else
	echo "WARNING: Archive not encrypted!"
	FINALFILE=$TARDEST
fi

# Move to sync location
mv $FINALFILE $SYNCDIR/

I then use s3sync to upload these to S3. So a complete script in cron (for level 0 non-incremental) might be something like this:

Code: Select all

#!/bin/bash
./s3archive.sh -e your@gpg.identity /home/steve/somethingimportant 0
./s3archive.sh -e your@gpg.identity /home/steve/otherstuff 0

export AWS_ACCESS_KEY_ID=WHATEVERYOURSIS
export AWS_SECRET_ACCESS_KEY=YOURACCESSKEY

/usr/local/lib/s3sync/s3sync.rb --progress --ssl /home/steve/s3backups/sync/ com.stevestreeting.backups:

Pretty simple. I run a level 0 every month and a level 1 incremental every few days, that's enough for me because I already have other backups, this is just for my critical offsite stuff.

You can probably make these scripts better if you have some bash expertise, mine is a bit rubbish.