AWS CLI is a command line utility provided by the Amazon Web Services Team for managing the AWS infrastructure. It also allows us to create, and manage s3 buckets directly from our computers using command line interfaces. Before start syncing files, make sure you have installed awscli in your system, or use the following articles to install it
In this tutorial, you will learn about synchronizing files between the local file system and s3 buckets.
1. Sync files from Local => S3 Bucket
For example I want to sync my local directory /root/mydir/ to S3 bucket directory s3://tecadmin/mydir/ where tecadmin is bucket name. I have created some new files in /root/mydir/ and sync to s3 bucket using the following command.
aws s3 sync /root/mydir/ s3://tecadmin/mydir/upload: mydir/index.php to s3://tecadmin/mydir/index.php upload: mydir/readme.html -> s3://tecadmin/mydir/readme.html
Note: Do not forget to add a trailing slash (/) in the local directory path when specifying the s3 bucket with the full directory path.
To keep preserve file attributes like date/time etc use -p or –preserve parameter like below
aws s3 sync /root/mydir/
If we want to sync only newly created files on source use –skip-existing parameter. It will skip all files which already exist on the destination or it is modified on the source.
aws s3 sync /root/mydir/ --skip-existing s3://tecadmin/mydir/
If you want to delete all files from the s3 bucket which has been removed from the local use –delete-removed parameter.
aws s3 sync /root/mydir/ --delete-removed s3://tecadmin/mydir/
2. Sync files from S3 Bucket => Local
For this example, I am again using the same folder and bucket used above. To test this I have put some extra files in the s3 bucket (s3://tecadmin/mydir/) and executed the following command to sync all files to the local directory.
aws s3 sync s3://tecadmin/mydir/ /root/mydir/download: s3://tecadmin/mydir/logo.jpg to mydir/logo.jpg download: s3://tecadmin/mydir/user.php to mydir/user.php
We can also used –preserve, –skip-existing and –delete-removed parameters during syncing files from S3 bucket to Local directory as followings.
aws s3 sync s3://tecadmin/mydir/ --preserve /root/mydir/
aws s3 sync s3://tecadmin/mydir/ --skip-existing /root/mydir/
aws s3 sync s3://tecadmin/mydir/ --delete-removed /root/mydir/