Backup VPS To Amazon S3 | Automatic Script | Files & Mysql | S3cmd | Cron Job

This shell script creates an automatic backup of the server files and Mysql database and uploads it to an Amazon S3 bucket using S3cmd tool

Create an account with Amazon S3 and create a bucket

Amazon S3 charges a minimal amount for storage. The storage pricing for the USA region is as follows:

Storage Pricing (varies by region)

Region: US East (N. Virginia)

	Standard Storage	Standard – Infrequent Access Storage †	Glacier Storage
First 50 TB / month	$0.023 per GB	$0.0125 per GB	$0.004 per GB
Next 450 TB / month	$0.022 per GB	$0.0125 per GB	$0.004 per GB
Over 500 TB / month	$0.021 per GB	$0.0125 per GB	$0.004 per GB

So, for storing 50 GB of date under the “Standard – Infrequent Access Storage”, one is paying only $0.625 per month at the rate of $0.0125 per GB which is literally peanuts.

AWS Free Usage Tier

Amazon S3 also offers a AWS Free Usage Tier as per which upon sign-up, new AWS customers receive 5 GB of Amazon S3 standard storage, 20,000 Get Requests, 2,000 Put Requests, and 15GB of data transfer out each month for one year.

So, it is an offer too good to miss out on.

When you create the S3 Account, you will have to create the ‘Access key’ and the ‘Secret Key’. These are the credentials necessary for the VPS to get access to the S3 Bucket.

Create a bucket in a region closer to the VPS

I have a S3 Bucket in Singapore which worked very well when I had the VPS with Linode in Singapore. The transfer speeds were at nearly 20MB per second. However, when I moved from the Linode Singapore VPS to the Scaleway VPS in Paris, the speeds dropped to a level where it became unusable.

This is because the distance from Paris to Singapore is more than 10,000 kms.

When I created a bucket at Amazon’s S3 in London, the transfer speeds have improved dramatically. This is because the distance from Paris to London is about 500 kms.

So, choose the region for the S3 bucket depending on where the VPS is located.

Install the latest version of s3cmd tool

There is a fine tutorial here.

The salient steps are as follows:

sudo yum install unzip python-pip

wget https://github.com/s3tools/s3cmd/archive/master.zip

unzip master.zip

cd s3cmd-master/

sudo python setup.py install

sudo pip install python-dateutil

s3cmd --version

s3cmd --configure

The last command opens up a dialogue box for entering data relating to the ‘access key’ and the ‘secret key’ that is provided when the account is created with Amazon S3.

S3cmd will communicate with the S3 bucket if the credentials are okay.

Basic s3cmd commands for interacting with the S3 bucket

The following are the commands to remember

Make bucket

 s3cmd mb s3://name-of-bucket

Delete/ Remove bucket

 s3cmd rb s3:// name-of-bucket

List contents of bucket

  s3cmd ls s3:// name-of-bucket

List all object in all buckets

 s3cmd la

Put file into bucket

 s3cmd put name-of-file s3:// name-of-bucket

Get file from bucket

 s3cmd get s3:// name-of-bucket/name-of-file destination-folder

Delete file from bucket

 s3cmd del s3:// name-of-bucket /name-of-file

Delete file from bucket (alias for del)

      s3cmd rm s3:// name-of-bucket /name-of-file

Copy object

      s3cmd cp s3:// name-of-bucket-1/name-of-file-1 s3:// name-of-bucket-2/name-of-file-2

There are several other commands which are set out in the s3cmd site.

Script for creating a tar file of the directories and uploading it to the S3 Bucket

Backup script for files

Create a file called vps-files-backup-script.sh

 nano vps-files-backup-script.sh 

Paste the following contents in the file after making appropriate changes

#!/bin/sh

THEDATE=`date +%d-%m-%Y-%H%M`

# export files

tar -cpzf /path-to-backup-folder-in-VPS/SITE-BACKUP-${THEDATE}.tar.gz -C / /path-to-folder-in-VPS-which-has-to-be-backed-up/

# remove backups older than 1 days

find /path-to-backup-folder-in-VPS/SITE* -mtime +1 -exec rm {} \;

# sync to amazon

s3cmd sync /path-to-backup-folder-in-VPS/ s3://name-of-bucket/ --delete-after

This script does the following things:

(i) It creates a tar.gz file of the contents of the folder. The date is appended to the file to enable identification. ;

(ii) It deposits the tar.gz file in another backup folder of the VPS;

(iii) It deletes old tar.gz files from the backup folder;

(iv) it uploads the latest tar.gz file to the S3 Bucket;

(v) It synchronizes the backup folder with the S3 Bucket and deletes the old files in the bucket.

You can choose how often the script should run by creating a cron tab (discussed later)

Backup of MySql database

Create a file called vps-database-backup-script.sh

 nano vps-database-backup-script.sh 

Paste the following contents in the file after making appropriate changes

#!/bin/sh

THEDATE=`date +%d-%m-%Y-%H%M`

# export database

mysqldump --all-databases -uusername -ppassword | gzip > /path-to-backup-folder-in-VPS/DB-BACKUP-${THEDATE}.sql.gz

# remove backups older than 1 days

find /path-to-backup-folder-in-VPS/DB* -mtime +1 -exec rm {} \;

# sync to amazon

s3cmd sync /path-to-backup-folder-in-VPS/ s3://name-of-bucket/ --delete-removed

Make both scripts writeable

 chmod 0700 vps-files-backup-script.sh

chmod 0700 vps-database-backup-script.sh

Run the scripts to test whether everything is working okay

bash vps-files-backup-script.sh 

 bash vps-database-backup-script.sh 

Automate backup with Cron job via SSH

If everything works fine and the backup files are showing in the S3 bucket, then you can create a cron tab script which will automate the process

First decide on the frequency of the cron job.

This site provides a visual representation of the various entries in the cron tab

So an entry like

 0 2 * * *

means that that the specified event will happen at 2.00 am every day.

The entry

 0 2 * * 7

means that the event will happen at 2.00 am on Sunday.

So an entry to run the database backup script everyday at 2 am will read as follows:

 0 2 * * * /path-to-the-vps-database-backup-script.sh > /dev/null 2>&1

An entry to run the files database script on every Sunday at 3 am will read as follows:

 0 3 * * 7 /path-to-the-vps-files-backup-script.sh > /dev/null 2>&1

See the existing crontab entries with the command

 crontab -l 

Add new entries to the crontab with the command

 crontab -e  

Enter the aforesaid two entries and save the file by pressing ‘Ctrl + x’ and then ‘Y’.

The crontab can be edited with NANO with the command:

EDITOR=nano crontab -e

For some reason, the crontab entries run as per the UTC Clock even though the VPS itself is set to a different time zone.

So, 2 am UTC may mean some other time in your region. Choose a time when the traffic is minimal on the site.