FTP Backup Service/Feral Backup Service
In the absence of an easy to adopt and tune mechanism to capture those lumps of data considered important to people involved in developing prototype tools supporting research activity, we have created a basic system suited to linux and MacOS X environments.
The pattern used is very simple, creating snapshots of the data, and not getting involved in incremental/differential algorithms -everything (targeted) is backed up each time the cycle runs. Management of these snapshots is done elsewhere.
Snapshots contain the data and configuration at a point in time. So if something goes wrong while people are
fiddling with, um, developing a system, it can be recovered to a known point.
In the ideal world, the snapshots would be moved into HSM capable storage, where appropriate aging policies could be used to ensure the live file system is not too encumbered.
The bits and pieces
The following describes a specific implementation to illustrate the general idea.
The subject system is a LAMP -a platform made up of Linux, Apache, MySQL, and PHP. The server hosting the snapshots is a Windows file server configured to allow FTP upload.
As the root user, I created
(not to be confused with /var/backups -something looked after by the Ubuntu system I am using) and
data is the map -a folder of links pointing to the areas you want to snapshot, and a place to put files generated as part of the backup process
out is where the archived snapshot is held till it is uploaded to the FTP host.
backup.sh looks a bit like the following:
#!/bin/bash DATETIME=`date "+%Y%m%d"` THISHOST=`hostname` FTPHOST=ftpserver FTPPATH=/$THISHOST/ MYEMAIL=somebody@somewhere BACKUPDATA=/var/backup/data BACKUPDATABASES="somedb someotherdb" BACKUPOUT=/var/backup/out echo "$DATETIME" # is BACKUPOUT empty? FILECOUNT=`ls -1 $BACKUPOUT/ |grep $DATETIME |wc -l` if [ "$FILECOUNT" = 0 ]; then # let's do it #### MYSQL # delete the old sql backup(s) echo "Deleting the MySQL backup file" rm -f $BACKUPDATA/*.sql.z # create the new echo "Creating the new MySQL backup file" mysqldump -u backupuser --databases $BACKUPDATABASES |gzip >$BACKUPDATA/$DATETIME.sql.z #### Remember what we backed up? # delete the old file(s) echo "Deleting the old file and folder listing" rm -f $BACKUPDATA/*.txt # add a listing of the data folder echo "Creating the directory listing file" ls -l $BACKUPDATA >$BACKUPDATA/$DATETIME.txt #### TAR and ZIP # now tar it all up echo "Creating the new backup archive" tar chvzf $BACKUPOUT/$DATETIME.tgz $BACKUPDATA/ else # something odd here? echo "Today's backup is still in the folder" fi #### UPLOAD echo "Attempting to upload the backup to $FTPHOST" # do we have an archive? if [ -e "$BACKUPOUT/$DATETIME.tgz" ]; then ncftpput -u anonymous -p $MYEMAIL $FTPHOST $FTPPATH $BACKUPOUT/$DATETIME.tgz if [ $? -ne 0 ]; then echo "Upload did not work :-(" exit 1 else echo "Upload successful!" echo "Removing the archive!" rm -f $BACKUPOUT/$DATETIME.tgz fi else echo "No new archive to upload?" exit 1 fi echo "Finished!" exit 0
and is run by root. This is done by adding a call to the script in the crontab for the root user
sudo crontab -l ... ... 0 6 * * * /var/backup/backup.sh >/var/backup/backup.out 2>&1
The script requires a number of things to work:
- ncftpput (part of ncftp –sudo apt-get install ncftp)
Allows you to push (or pull) a file using a single line
- A backup user with appropriate access
I used something like GRANT SELECT, LOCK TABLES, SHOW VIEW, TRIGGER ON database.* TO ‘backupuser’@’localhost’;)
- An FTP server