• Home
  • Ubuntu 20.04
    • Upgrade Ubuntu
    • Install Java
    • Install Node.js
    • Install Docker
    • Install LAMP Stack
  • Tutorials
    • AWS
    • Shell Scripting
    • Docker
    • Git
    • MongoDB
  • Funny Tools
  • FeedBack
  • Submit Article
  • About Us
TecAdmin
Menu
  • Home
  • Ubuntu 20.04
    • Upgrade Ubuntu
    • Install Java
    • Install Node.js
    • Install Docker
    • Install LAMP Stack
  • Tutorials
    • AWS
    • Shell Scripting
    • Docker
    • Git
    • MongoDB
  • Funny Tools
  • FeedBack
  • Submit Article
  • About Us

How to Install s3cmd in Linux and Manage Amazon s3 Buckets

Written by Rahul, Updated on March 27, 2018

s3cmd is a command line utility used for creating s3 buckets, uploading, retrieving and managing data to Amazon s3 storage. This article will help you to how to use install s3cmd on CentOS, RHEL, OpenSUSE, Ubuntu, Debian & LinuxMint systems and manage s3 buckets via command line in easy steps. To install s3cmd on windows servers read article install s3cmd in Windows.

We can also mount s3 bucket as local drive in our system using S3FS with FUSE. To configure it read next article mount s3 bucket on Linux.

amazone-s3-center-logo

Install s3cmd on Linux

s3cmd is available in default rpm repositories for CentOS, RHEL and Ubuntu systems, You can install it using simply executing following commands on your system.

On CentOS/RHEL:
yum install s3cmd
On Ubuntu/Debian:
sudo apt-get install s3cmd
On SUSE Linux Enterprise Server 11:
zypper addrepo http://s3tools.org/repo/SLE_11/s3tools.repo
zypper install s3cmd
Install Latest s3cmd using Source

If you are not getting the latest version of s3cmd using package managers, You can install last s3cmd version on your system using the source code. Visit this url or use below command to download latest version of s3cmd.

wget https://sourceforge.net/projects/s3tools/files/s3cmd/2.0.1/s3cmd-2.0.1.tar.gz
tar xzf s3cmd-2.0.1.tar.gz

Now install it using below command with source files.

cd s3cmd-2.0.1
sudo python setup.py install
Configure s3cmd Environment

In order to configure s3cmd we would require Access Key and Secret Key of your S3 Amazon account. Get these security keys from aws securityCredentials page. If will prompt to login to your amazon account.

After getting key files, use below command to configure s3cmd.

# s3cmd --configure
Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3
Access Key: xxxxxxxxxxxxxxxxxxxxxx
Secret Key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: xxxxxxxxxx
Path to GPG program [/usr/bin/gpg]:

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP and can't be used if you're behind a proxy
Use HTTPS protocol [No]: Yes

New settings:
  Access Key: xxxxxxxxxxxxxxxxxxxxxx
  Secret Key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  Encryption password: xxxxxxxxxx
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: True
  HTTP Proxy server name:
  HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] Y
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Success. Encryption and decryption worked fine :-)

Save settings? [y/N] y
Configuration saved to '/root/.s3cfg'
Uses of s3cmd Command Line

Once the configuration is successfully completed. Now find below command details to how to manage s3 buckets using commands.

1. List All S3 Bucket

Use the following command to list all s3 buckets in your aws account.

# s3cmd ls
2. Creating New Bucket

To create a new bucket in Amazon s3 use below command. It will create bucket named tecadmin in S3 account.

# s3cmd mb s3://tecadmin

Bucket 's3://tecadmin/' created
3. Uploading file in Bucket

Below command will upload file file.txt to s3 bucket using s3cmd command.

# s3cmd put file.txt s3://tecadmin/

file.txt -> s3://tecadmin/file.txt  [1 of 1]
 190216 of 190216   100% in    0s  1668.35 kB/s  done
4. Uploading Directory in Bucket

If we need to upload entire directory use -r to upload it recursively as below.

# s3cmd put -r backup s3://tecadmin/

backup/file1.txt -> s3://tecadmin/backup/file1.txt  [1 of 2]
 9984 of 9984   100% in    0s    18.78 kB/s  done
backup/file2.txt -> s3://tecadmin/backup/file2.txt  [2 of 2]
 0 of 0     0% in    0s     0.00 B/s  done

Make sure you are not adding trailing slash in upload directory named backup (eg: backup/), else it will upload only content to backup directory only.

# s3cmd put -r backup/ s3://tecadmin/

backup/file1.txt -> s3://tecadmin/file1.txt  [1 of 2]
 9984 of 9984   100% in    0s    21.78 kB/s  done
backup/file2.txt -> s3://tecadmin/file2.txt  [2 of 2]
 0 of 0     0% in    0s     0.00 B/s  done
5. List Data of S3 Bucket

List the objects of s3 bucket using ls switch with s3cmd.

# s3cmd ls s3://tecadmin/

                       DIR   s3://tecadmin/backup/
2013-09-03 10:58    190216   s3://tecadmin/file.txt
6. Download Files from Bucket

Sometimes if we need to download files from the s3 bucket, Use following commands to download it.

# s3cmd get s3://tecadmin/file.txt

s3://tecadmin/file.txt -> ./file.txt  [1 of 1]
 4 of 4   100% in    0s    10.84 B/s  done
7. Remove Data of S3 Bucket

To remove files are folder from s3 bucket use following commands.

 Removing file from s3 bucket 
# s3cmd del s3://tecadmin/file.txt

File s3://tecadmin/file.txt deleted

 Removing directory from s3 bucket 
# s3cmd del s3://tecadmin/backup

File s3://tecadmin/backup deleted
8. Remove S3 Bucket

If we don’t need s3 bucket anymore, we can simply delete it using the following command. Before removing bucket make sure its empty.

# s3cmd rb s3://tecadmin

ERROR: S3 error: 409 (BucketNotEmpty): The bucket you tried to delete is not empty

Above command failed because of s3 bucket was not empty

To remove bucket first remove all objects inside bucket and then use the command again.

# s3cmd rb s3://tecadmin

Bucket 's3://tecadmin/' removed

Thanks for using this article. If you want to mount s3bucket in your system, you can article Mount S3bucket in Linux using s3fs. Also you can sync data between s3 bucket and local directory using s3cmd.

Share it!
Share on Facebook
Share on Twitter
Share on LinkedIn
Share on Reddit
Share on Tumblr
Share on Whatsapp
Rahul
Rahul
Connect on Facebook Connect on Twitter

I, Rahul Kumar am the founder and chief editor of TecAdmin.net. I am a Red Hat Certified Engineer (RHCE) and working as an IT professional since 2009..

25 Comments

  1. Avatar Pankaj kumar Reply
    May 3, 2020 at 6:16 pm

    the issue in configuring s3cmd –configure. it says 400 error authentication failed but I’m giving the right access key.

  2. Avatar Janaki Reply
    June 26, 2018 at 7:28 am

    Hi Rahul,

    How to clear S3 cache using S3cmd tool?

  3. Avatar Gaurav Reply
    May 22, 2018 at 6:21 am

    Hi,
    I am unable to install it in RHEL 6.6 because this error has come at the time of installation. Please suggest.

    Processing dependencies for s3cmd==2.0.1
    Searching for python-magic
    Reading http://pypi.python.org/simple/python-magic/
    Couldn’t find index page for ‘python-magic’ (maybe misspelled?)
    Scanning index of all packages (this may take a while)
    Reading http://pypi.python.org/simple/
    No local packages or download links found for python-magic
    error: Could not find suitable distribution for Requirement.parse(‘python-magic’)

  4. Avatar mohamed ayas Reply
    January 20, 2018 at 1:00 pm

    what is port which s3cmd transfer file to s3 bucket. Port 80 or 443. How we can verify that ?

  5. Avatar Bhumika jain Reply
    December 6, 2017 at 1:19 pm

    Hi, can i transfer a file from my openstack set up to amazon s3. I mean swift-s3 sync up.

  6. Avatar Chen Reply
    May 12, 2017 at 6:11 pm

    How do you create multiple (configure) profiles?

    • Rahul Rahul K. Reply
      May 13, 2017 at 2:38 am

      Hi Chen,

      You can specify –config=FILE to create and use different config files during run.

  7. Avatar Paritosh Reply
    October 7, 2015 at 9:29 am

    Thanks for this artical. 🙂

  8. Avatar Ashish Reply
    July 9, 2015 at 12:17 pm

    How to make user specific permission on bucket folder’s.

  9. Avatar rishikesh Reply
    June 24, 2015 at 5:49 am

    Hi
    I really like your site and this post is very nice.
    Nice work and keep it up.

  10. Avatar Shweta Reply
    May 9, 2015 at 7:23 am

    I have question on your 2nd steps.

    2. Uploading file in Bucket

    <<
    I want to upload files from my another local folder to s3 bucket folder like below example from your code
    s3cmd put –acl-public folderpath/file.txt s3://tecadmin/folder1/

    • Rahul Rahul Reply
      May 10, 2015 at 7:59 am

      Hi Sweta,

      You can put file to s3 from any location of system. You just need to provide source file path correctly in command.

      Are you getting any error ?

      • Avatar Shweta Reply
        May 11, 2015 at 5:01 am

        Yes, I am getting error in terminal.

        Let me share snaps here – http://prntscr.com/73wgei

        Thanks for response!

        • Avatar Saad Reply
          June 8, 2015 at 7:26 pm

          If you are still getting the error, remove the “-” from your file name. Instead of “-aaa.jpg” make it “aaa.jpg”

  11. Avatar Anup Reply
    September 23, 2014 at 3:08 pm

    Very help full. Thanks for sharing with us.

    God bless you!!!

  12. Avatar Trichoplax Adhaerens Reply
    August 8, 2014 at 11:39 pm

    Hi Rahul,

    This procedure worked like a charm. Many thanks for taking the time to share your wisdom. It is much appreciated.

    Trichoplax

  13. Avatar Miguel Mota Reply
    April 6, 2014 at 10:55 pm

    Very helpful article!

  14. Avatar Eric Stauffer Reply
    February 27, 2014 at 8:05 pm

    Awesome tutorial, Rahul.

    I am working on a backup solution involving Amazon S3 and Glacier, and s3cmd is exactly what I was looking for. The instructions on the original website are a bit clunky, but yours laid it out in very easy to follow instructions. Do you have a guide for getting the sync feature to work? I plan on running it with cron, if possible.

    • Rahul Rahul Reply
      April 5, 2014 at 2:00 pm

      Hi Eric,

      Use below article for sync files with s3 bucket using s3cmd
      http://tecadmin.net/s3cmd-file-sync-with-s3bucket/

  15. Avatar Manish Shrivastava Reply
    February 3, 2014 at 10:19 am

    Great Help Rahul Bhai

  16. Avatar Azahar Reply
    December 31, 2013 at 10:26 am

    Excellent Tutorial

  17. Avatar Liju Reply
    November 11, 2013 at 4:41 am

    Bucket name will be unique for all users. So most of the names are already being used.

  18. Avatar Gabriel Reply
    October 29, 2013 at 7:46 pm

    Thanks, this helped but when I try to create a new bucket I get an error:
    ERROR: Bucket ‘WHATEVER already exists

    How can I fix this?

    • Rahul Rahul Reply
      October 30, 2013 at 5:48 am

      Hi Gabriel,

      First check if bucket “WHATEVER” already exists. If it doesn’t already exists, we recommend to use lowercase characters in bucket name like “whatever”.

      • Avatar Dale-Kurt Murray Reply
        April 23, 2014 at 4:13 pm

        Hi Gabriel,

        The reason you receive that error is a result of the S3 bucket name already exists. S3 bucket name must be unique (globally) on the services. So if you are trying to create a bucket called “whatever” and it exists, try creating “gabriel_whatever”

Leave a Reply Cancel reply

Popular Posts

  • How to View or List Cron Jobs in Linux
  • How to Install PHP 8 on Ubuntu 20.04
  • How to Set Up SSH Tunnel with PuTTY
  • How to Install MySQL 8.0 on Ubuntu 20.04
  • Issue with phpMyAdmin and PHP: Warning in ./libraries/sql.lib.php#613 count(): Parameter must be an array or an object that implements Countable”
  • How to Install Tor Browser on Ubuntu 20.04
  • How to Allow Remote Connections to MySQL
  • How To Install VNC Server on Debian 10
© 2013-2021 Tecadmin.net. All Rights Reserved | Terms  | Privacy Policy