How to Install s3cmd in Linux and Manage S3 Buckets

s3cmd is a command line utility used for creating s3 buckets, uploading, retrieving and managing data to Amazon s3 storage. This article will help you to how to use install s3cmd on CentOS, RHEL, OpenSUSE, Ubuntu, Debian & LinuxMint systems and manage s3 buckets via command line in easy steps. To install s3cmd on windows servers read article install s3cmd in Windows.

We can also mount s3 bucket as local drive in our system using S3FS with FUSE. To configure it read next article mount s3 bucket on Linux.

Advertisement

Install s3cmd on Linux

s3cmd is available in default package repositories for the Ubuntu, Debian, Fedora, CentOS, and RHEL Linux systems. You can install it using simply executing the following commands on your system.

On CentOS/RHEL and Fedora:

sudo dnf  install s3cmd 

On Ubuntu/Debian:

sudo apt-get install s3cmd

Install Latest s3cmd using Source

If you are not getting the latest version of s3cmd using package managers, You can install the last s3cmd version on your system using the source code. Visit this url or use below command to download latest version of s3cmd.

wget https://sourceforge.net/projects/s3tools/files/s3cmd/2.2.0/s3cmd-2.2.0.tar.gz
tar xzf s3cmd-2.2.0.tar.gz

Then install it using the below command with source files.

cd s3cmd-2.2.0 
sudo python setup.py install 

Configure S3cmd Environment

In order to configure s3cmd we would require Access Key and Secret Key of your S3 Amazon account. Get these security keys from aws securityCredentials page. If will prompt to login to your amazon account.

After getting key files, use the below command to configure s3cmd.

s3cmd --configure 

Output
Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3
Access Key: xxxxxxxxxxxxxxxxxxxxxx
Secret Key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: xxxxxxxxxx
Path to GPG program [/usr/bin/gpg]:

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP and can't be used if you're behind a proxy
Use HTTPS protocol [No]: Yes

New settings:
  Access Key: xxxxxxxxxxxxxxxxxxxxxx
  Secret Key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  Encryption password: xxxxxxxxxx
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: True
  HTTP Proxy server name:
  HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] Y
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Success. Encryption and decryption worked fine :-)

Save settings? [y/N] y
Configuration saved to '/root/.s3cfg'

Working with the s3cmd Command Line

Once the configuration is successfully completed. Now find below command details to how to manage s3 buckets using commands.

1. List All S3 Bucket

Use the following command to list all s3 buckets in your aws account.

s3cmd ls 

2. Creating New Bucket

To create a new bucket in Amazon s3 use the below command. It will create bucket named tecadmin in S3 account.

s3cmd mb s3://tecadmin 

Output
Bucket 's3://tecadmin/' created

3. Uploading file in Bucket

Below command will upload file file.txt to s3 bucket using s3cmd command.

s3cmd put file.txt s3://tecadmin/

Output
file.txt -> s3://tecadmin/file.txt  [1 of 1]
 190216 of 190216   100% in    0s  1668.35 kB/s  done

4. Uploading Directory in Bucket

If we need to upload entire directory use -r to upload it recursively as below.

s3cmd put -r backup s3://tecadmin/

Output
backup/file1.txt -> s3://tecadmin/backup/file1.txt  [1 of 2]
 9984 of 9984   100% in    0s    18.78 kB/s  done
backup/file2.txt -> s3://tecadmin/backup/file2.txt  [2 of 2]
 0 of 0     0% in    0s     0.00 B/s  done

Make sure you are not adding a trailing slash in the upload directory named backup (eg: backup/), else it will upload only content to the backup directory only.

s3cmd put -r backup/ s3://tecadmin/

Output
backup/file1.txt -> s3://tecadmin/file1.txt  [1 of 2]
 9984 of 9984   100% in    0s    21.78 kB/s  done
backup/file2.txt -> s3://tecadmin/file2.txt  [2 of 2]
 0 of 0     0% in    0s     0.00 B/s  done

5. List Data of S3 Bucket

List the objects of s3 bucket using ls switch with s3cmd.

s3cmd ls s3://tecadmin/

Output
                       DIR   s3://tecadmin/backup/
2013-09-03 10:58    190216   s3://tecadmin/file.txt

6. Download Files from Bucket

Sometimes if we need to download files from the s3 bucket, Use the following commands to download them.

s3cmd get s3://tecadmin/file.txt

Output
s3://tecadmin/file.txt -> ./file.txt  [1 of 1]
 4 of 4   100% in    0s    10.84 B/s  done

7. Remove Files from S3 Bucket

You can also remove a file or a directory from s3 bucket. Read the below examples to delete a file or a directory from an s3 bucket with s3cmd.

To Remove file from s3 bucket:

s3cmd del s3://tecadmin/file.txt

Output
File s3://tecadmin/file.txt deleted

Removing directory from s3 bucket:

s3cmd del s3://tecadmin/backup 

Output
File s3://tecadmin/backup deleted

8. Remove S3 Bucket

If we don’t need s3 bucket anymore, we can simply delete it using the following command. Before removing the bucket make sure it is empty.

s3cmd rb s3://tecadmin

Output
ERROR: S3 error: 409 (BucketNotEmpty): The bucket you tried to delete is not empty

Above command failed because of s3 bucket was not empty

To remove the bucket first remove all objects inside the bucket and then use the command again.

s3cmd rb s3://tecadmin

Output
Bucket 's3://tecadmin/' removed

Thanks for using this article. If you want to mount s3bucket in your system, you can article Mount S3bucket in Linux using s3fs. Also you can sync data between s3 bucket and local directory using s3cmd.