Server Backups on Amazon’s S3
March 3rd, 2009 by TheKidd
I was looking around on the web the other day for an easy way to backup data on a Linux server to Amazon S3. I ended up running into SoftwareBloat’s script and liked what they had done with it. One thing I needed, however, was to be able to clean up the backup bucket on S3. So here is the script i came up with from SoftwareBloats original. Hope it will help someone looking for the same!
Oh one note, after you issue the apt-get install s3cmd command to grab the program, you will also want to run s3cmd –configure which will ask for your S3 information. Important step before the below will work.
#!/bin/bash
## SoftwareBloat’s Script
## TheKidd had to play with it.
## Which means I screwed it up.
## Hehe, thx bloat! Awesome tutorial! Hope this might help.
# Dates to variables for the backup script
DATE_TAG=$(date +"%Y.%m.%d")
# to clean backups off s3 older than 28 days
# as i only want to keep 4 sets of backups
S3_DATE_TAG=$(date -d "-28 days" +"%Y.%m.%d")
# Admin email notification
EMAIL_ADMN="admin@domain.com"
EMAIL_SUBJ="S3 Backup Results"
EMAIL_FROM="s3backup@domain.com"
# Directories to backup to and from
SHELL_DIR="/shell"
WWW_DIR="/www"
HOME_DIR="/home"
ROOT_DIR="/root"
BACKUP_DIR="/backup/soho"
SQL_BK_NAME="athena.mysql.bk"
BUCKET_NAME="sohobackup"
# Intro for Admin Email
echo "This message is to inform you that your backup on $DATE_TAG just started running. Lets get started." > $BACKUP_DIR/logs/backup-$DATE_TAG.log 2>&1
# Clean up any files older than 7 days
# this allows me to catch the 7 daily mysql backups but delete anything created 8+ days ago.
echo "Cleaning folders before start…" >> $BACKUP_DIR/logs/backup-$DATE_TAG.log 2>&1
/usr/bin/find $BACKUP_DIR -maxdepth 0 -type f -mtime +7 -print | xargs rm -f
# backup key directories
echo "Backing up key directories…" >> $BACKUP_DIR/logs/backup-$DATE_TAG.log 2>&1
tar pczvf $BACKUP_DIR/shell-$DATE_TAG.tgz $SHELL_DIR
tar pczvf $BACKUP_DIR/www-$DATE_TAG.tgz $WWW_DIR
tar pczvf $BACKUP_DIR/home-$DATE_TAG.tgz $HOME_DIR
tar pczvf $BACKUP_DIR/root-$DATE_TAG.tgz $ROOT_DIR
# backup databases
# echo "Backing up Database…" >> $BACKUP_DIR/logs/backup-$DATE_TAG.log 2>&1
# mysqldump -u root –password=your_pw_here –all-databases | gzip -c > $BACKUP_DIR/$SQL_BK_NAME$DATE_TAG.gz
# **Don’t need the above as webmin provides daily MySql backups dumped into BACKUP_DIR**
# But everyone prefers their own way!
# Just to prevent errors, we attempt to create the bucket
/usr/bin/s3cmd -c /root/.s3cfg mb s3://$BUCKET_NAME
# Backup all the tgz’s with correct date stamp
echo "Uploading backup files…" >> $BACKUP_DIR/logs/backup-$DATE_TAG.log 2>&1
for f in $BACKUP_DIR/*$DATE_TAG.tgz
do
/usr/bin/s3cmd -c /root/.s3cfg put $f s3://$BUCKET_NAME/$(basename $f) >> $BACKUP_DIR/logs/backup-$DATE_TAG.log 2>&1
done
# now we kill the mysql daily backups made 28+ days ago
echo "Deleting old DB files on S3…" >> $BACKUP_DIR/logs/backup-$DATE_TAG.log 2>&1
for (( i=40; i>=28; i– ))
do
/usr/bin/s3cmd -c /root/.s3cfg del s3://$BUCKET_NAME/$SQL_BK_NAME`date -d "-$i days" +"%Y.%m.%d"`.tgz >> $BACKUP_DIR/logs/backup-$DATE_TAG.log 2>&1
done
# now we kill the key directory backups made 28+ days ago
echo "Deleting old key directories on S3…" >> $BACKUP_DIR/logs/backup-$DATE_TAG.log 2>&1
for f in shell www home root
do
/usr/bin/s3cmd -c /root/.s3cfg del s3://$BUCKET_NAME/$(basename $f)-$S3_DATE_TAG.tgz >> $BACKUP_DIR/logs/backup-$DATE_TAG.log 2>&1
done
# just to let the admin in on the backup
echo "All done! Below is a list of what was backed up."
echo "Additional info: $BACKUP_DIR/logs/backup-$DATE_TAG.log" $BACKUP_DIR/logs/backup-$DATE_TAG.log 2>&1
/usr/bin/s3cmd -c /root/.s3cfg ls s3://$BUCKET_NAME/ >> $BACKUP_DIR/logs/backup-$DATE_TAG.log 2>&1
echo "Sending Email! Have a good day!" >> $BACKUP_DIR/logs/backup-$DATE_TAG.log 2>&1
(echo "Subject: $EMAIL_SUBJ"; echo "From: $EMAIL_FROM"; echo "To: $EMAIL_ADMN";
echo "Reply-To: $EMAIL_FROM"; cat $BACKUP_DIR/logs/backup-$DATE_TAG.log;) | /usr/lib/sendmail -t
Great! I’m tickled to see this. Keep in touch!