Sync to S3 Storage Bucket

Synchronize Data to AWS S3 Storage from CLI

We have a folder /backups on your server that you want to syncronize to an AWS S3 Storage bucket. First thing is first, we'll want to install the aws command line tools. I'll use python virtual environments specific to functions or django webapps, etc. So first, create a virtualenv and source it (if it's successfully created using &&).

Create Virtual Environment and Source it

virtualenv /opt/python/envs/awsenv && source /opt/python/envs/awsenv/bin/activate

Install AWS CLI with pip

We will use pip to install the amazon cli tools. These are useful for more than s3 storage, but this is an example to just synchronize the folder on your server to AWS for backup purposes.

pip install awscli

Create AWS Config file and Set Permissions on it

Before creating this file, we need to ensure we've generated an Access Key ID/Secret Access Key pair. Logged into the AWS Web GUI, click your login name in the top right and select "My Security Credentials". From here, you can generate these, and copy and paste them into a config file located in ~/.aws/config. We also don't want this file to be world readable as anyone with access to it can access your AWS account which is an obvious security issue.

mkdir ~/.aws && vi ~/.aws/config

The contents of this should at minimal look something like this. AWS has pretty good documentation on other things that can go in here, so I won't get into all that right now.


Now, set the permissions on this folder/file, I use 700/600 and also mute it.

chmod 700 ~/.aws && chmod 600 ~/.aws/config && chattr +i ~/.aws && chattr +i ~/.aws/config

Lets now list the current contents of the bucket which you could have created from the webgui. 

aws s3 ls s3://mybucketname

And finally, we can sync our data to the buket. In this case, we will exclude the .git/* folder as it can at least double the amount of storage used if you're using git to manage changes and the integrity of binary files. (I've had my data tampered with/stolen by "trusted" parties in the past)...

aws s3 sync /path/to/sync/ s3://mybucketname --exclude ".git/*"
Current rating: 5