Facebook Twitter Instagram
    TecAdmin
    • Home
    • FeedBack
    • Submit Article
    • About Us
    Facebook Twitter Instagram
    TecAdmin
    You are at:Home»AWS»How to Install s3cmd in Linux and Manage S3 Buckets

    How to Install s3cmd in Linux and Manage S3 Buckets

    By RahulSeptember 30, 20215 Mins Read

    s3cmd is a command line utility used for creating s3 buckets, uploading, retrieving and managing data to Amazon s3 storage. This article will help you to how to use install s3cmd on CentOS, RHEL, OpenSUSE, Ubuntu, Debian & LinuxMint systems and manage s3 buckets via command line in easy steps. To install s3cmd on windows servers read article install s3cmd in Windows.

    Advertisement

    We can also mount s3 bucket as local drive in our system using S3FS with FUSE. To configure it read next article mount s3 bucket on Linux.

    amazone-s3-center-logo

    Install s3cmd on Linux

    s3cmd is available in default package repositories for the Ubuntu, Debian, Fedora, CentOS, and RHEL Linux systems. You can install it using simply executing the following commands on your system.

    On CentOS/RHEL and Fedora:
    sudo dnf  install s3cmd 
    
    On Ubuntu/Debian:
    sudo apt-get install s3cmd
    
    Install Latest s3cmd using Source

    If you are not getting the latest version of s3cmd using package managers, You can install the last s3cmd version on your system using the source code. Visit this url or use below command to download latest version of s3cmd.

    wget https://sourceforge.net/projects/s3tools/files/s3cmd/2.2.0/s3cmd-2.2.0.tar.gz
    tar xzf s3cmd-2.2.0.tar.gz
    

    Then install it using the below command with source files.

    cd s3cmd-2.2.0 
    sudo python setup.py install 
    

    Configure S3cmd Environment

    In order to configure s3cmd we would require Access Key and Secret Key of your S3 Amazon account. Get these security keys from aws securityCredentials page. If will prompt to login to your amazon account.

    After getting key files, use the below command to configure s3cmd.

    s3cmd --configure 
    
    Output
    Enter new values or accept defaults in brackets with Enter. Refer to user manual for detailed description of all options. Access key and Secret key are your identifiers for Amazon S3 Access Key: xxxxxxxxxxxxxxxxxxxxxx Secret Key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Encryption password is used to protect your files from reading by unauthorized persons while in transfer to S3 Encryption password: xxxxxxxxxx Path to GPG program [/usr/bin/gpg]: When using secure HTTPS protocol all communication with Amazon S3 servers is protected from 3rd party eavesdropping. This method is slower than plain HTTP and can't be used if you're behind a proxy Use HTTPS protocol [No]: Yes New settings: Access Key: xxxxxxxxxxxxxxxxxxxxxx Secret Key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Encryption password: xxxxxxxxxx Path to GPG program: /usr/bin/gpg Use HTTPS protocol: True HTTP Proxy server name: HTTP Proxy server port: 0 Test access with supplied credentials? [Y/n] Y Please wait, attempting to list all buckets... Success. Your access key and secret key worked fine :-) Now verifying that encryption works... Success. Encryption and decryption worked fine :-) Save settings? [y/N] y Configuration saved to '/root/.s3cfg'

    Working with the s3cmd Command Line

    Once the configuration is successfully completed. Now find below command details to how to manage s3 buckets using commands.

    1. List All S3 Bucket

    Use the following command to list all s3 buckets in your aws account.

    s3cmd ls 
    

    2. Creating New Bucket

    To create a new bucket in Amazon s3 use the below command. It will create bucket named tecadmin in S3 account.

    s3cmd mb s3://tecadmin 
    
    Output
    Bucket 's3://tecadmin/' created
    3. Uploading file in Bucket

    Below command will upload file file.txt to s3 bucket using s3cmd command.

    s3cmd put file.txt s3://tecadmin/
    
    Output
    file.txt -> s3://tecadmin/file.txt [1 of 1] 190216 of 190216 100% in 0s 1668.35 kB/s done
    4. Uploading Directory in Bucket

    If we need to upload entire directory use -r to upload it recursively as below.

    s3cmd put -r backup s3://tecadmin/
    
    Output
    backup/file1.txt -> s3://tecadmin/backup/file1.txt [1 of 2] 9984 of 9984 100% in 0s 18.78 kB/s done backup/file2.txt -> s3://tecadmin/backup/file2.txt [2 of 2] 0 of 0 0% in 0s 0.00 B/s done

    Make sure you are not adding a trailing slash in the upload directory named backup (eg: backup/), else it will upload only content to the backup directory only.

    s3cmd put -r backup/ s3://tecadmin/
    
    Output
    backup/file1.txt -> s3://tecadmin/file1.txt [1 of 2] 9984 of 9984 100% in 0s 21.78 kB/s done backup/file2.txt -> s3://tecadmin/file2.txt [2 of 2] 0 of 0 0% in 0s 0.00 B/s done
    5. List Data of S3 Bucket

    List the objects of s3 bucket using ls switch with s3cmd.

    s3cmd ls s3://tecadmin/
    
    Output
    DIR s3://tecadmin/backup/ 2013-09-03 10:58 190216 s3://tecadmin/file.txt
    6. Download Files from Bucket

    Sometimes if we need to download files from the s3 bucket, Use the following commands to download them.

    s3cmd get s3://tecadmin/file.txt
    
    Output
    s3://tecadmin/file.txt -> ./file.txt [1 of 1] 4 of 4 100% in 0s 10.84 B/s done
    7. Remove Files from S3 Bucket

    You can also remove a file or a directory from s3 bucket. Read the below examples to delete a file or a directory from an s3 bucket with s3cmd.

    To Remove file from s3 bucket:

    s3cmd del s3://tecadmin/file.txt
    
    Output
    File s3://tecadmin/file.txt deleted

    Removing directory from s3 bucket:

    s3cmd del s3://tecadmin/backup 
    
    Output
    File s3://tecadmin/backup deleted
    8. Remove S3 Bucket

    If we don’t need s3 bucket anymore, we can simply delete it using the following command. Before removing the bucket make sure it is empty.

    s3cmd rb s3://tecadmin
    
    Output
    ERROR: S3 error: 409 (BucketNotEmpty): The bucket you tried to delete is not empty

    Above command failed because of s3 bucket was not empty

    To remove the bucket first remove all objects inside the bucket and then use the command again.

    s3cmd rb s3://tecadmin
    
    Output
    Bucket 's3://tecadmin/' removed

    Thanks for using this article. If you want to mount s3bucket in your system, you can article Mount S3bucket in Linux using s3fs. Also you can sync data between s3 bucket and local directory using s3cmd.

    Amazon s3 aws bucket s3 s3 s3 bucket s3cmd s3cmd sync
    Share. Facebook Twitter Pinterest LinkedIn Tumblr Email WhatsApp

    Related Posts

    S3FS: Mounting Amazon S3 Buckets on Ubuntu & Debian Systems

    Create AWS Cognito User with AWSCLI

    How to Create User Manually in AWS Cognito (CLI)

    How to Enable CORS in S3 Bucket

    How To Configure CORS in Amazon S3 Buckets

    View 25 Comments

    25 Comments

    1. Pankaj kumar on May 3, 2020 6:16 pm

      the issue in configuring s3cmd –configure. it says 400 error authentication failed but I’m giving the right access key.

      Reply
    2. Janaki on June 26, 2018 7:28 am

      Hi Rahul,

      How to clear S3 cache using S3cmd tool?

      Reply
    3. Gaurav on May 22, 2018 6:21 am

      Hi,
      I am unable to install it in RHEL 6.6 because this error has come at the time of installation. Please suggest.

      Processing dependencies for s3cmd==2.0.1
      Searching for python-magic
      Reading http://pypi.python.org/simple/python-magic/
      Couldn’t find index page for ‘python-magic’ (maybe misspelled?)
      Scanning index of all packages (this may take a while)
      Reading http://pypi.python.org/simple/
      No local packages or download links found for python-magic
      error: Could not find suitable distribution for Requirement.parse(‘python-magic’)

      Reply
    4. mohamed ayas on January 20, 2018 1:00 pm

      what is port which s3cmd transfer file to s3 bucket. Port 80 or 443. How we can verify that ?

      Reply
    5. Bhumika jain on December 6, 2017 1:19 pm

      Hi, can i transfer a file from my openstack set up to amazon s3. I mean swift-s3 sync up.

      Reply
    6. Chen on May 12, 2017 6:11 pm

      How do you create multiple (configure) profiles?

      Reply
      • Rahul K. on May 13, 2017 2:38 am

        Hi Chen,

        You can specify –config=FILE to create and use different config files during run.

        Reply
    7. Paritosh on October 7, 2015 9:29 am

      Thanks for this artical. 🙂

      Reply
    8. Ashish on July 9, 2015 12:17 pm

      How to make user specific permission on bucket folder’s.

      Reply
    9. rishikesh on June 24, 2015 5:49 am

      Hi
      I really like your site and this post is very nice.
      Nice work and keep it up.

      Reply
    10. Shweta on May 9, 2015 7:23 am

      I have question on your 2nd steps.

      2. Uploading file in Bucket

      <<
      I want to upload files from my another local folder to s3 bucket folder like below example from your code
      s3cmd put –acl-public folderpath/file.txt s3://tecadmin/folder1/

      Reply
      • Rahul on May 10, 2015 7:59 am

        Hi Sweta,

        You can put file to s3 from any location of system. You just need to provide source file path correctly in command.

        Are you getting any error ?

        Reply
        • Shweta on May 11, 2015 5:01 am

          Yes, I am getting error in terminal.

          Let me share snaps here – http://prntscr.com/73wgei

          Thanks for response!

          Reply
          • Saad on June 8, 2015 7:26 pm

            If you are still getting the error, remove the “-” from your file name. Instead of “-aaa.jpg” make it “aaa.jpg”

            Reply
    11. Anup on September 23, 2014 3:08 pm

      Very help full. Thanks for sharing with us.

      God bless you!!!

      Reply
    12. Trichoplax Adhaerens on August 8, 2014 11:39 pm

      Hi Rahul,

      This procedure worked like a charm. Many thanks for taking the time to share your wisdom. It is much appreciated.

      Trichoplax

      Reply
    13. Miguel Mota on April 6, 2014 10:55 pm

      Very helpful article!

      Reply
    14. Eric Stauffer on February 27, 2014 8:05 pm

      Awesome tutorial, Rahul.

      I am working on a backup solution involving Amazon S3 and Glacier, and s3cmd is exactly what I was looking for. The instructions on the original website are a bit clunky, but yours laid it out in very easy to follow instructions. Do you have a guide for getting the sync feature to work? I plan on running it with cron, if possible.

      Reply
      • Rahul on April 5, 2014 2:00 pm

        Hi Eric,

        Use below article for sync files with s3 bucket using s3cmd
        http://tecadmin.net/s3cmd-file-sync-with-s3bucket/

        Reply
    15. Manish Shrivastava on February 3, 2014 10:19 am

      Great Help Rahul Bhai

      Reply
    16. Azahar on December 31, 2013 10:26 am

      Excellent Tutorial

      Reply
    17. Liju on November 11, 2013 4:41 am

      Bucket name will be unique for all users. So most of the names are already being used.

      Reply
    18. Gabriel on October 29, 2013 7:46 pm

      Thanks, this helped but when I try to create a new bucket I get an error:
      ERROR: Bucket ‘WHATEVER already exists

      How can I fix this?

      Reply
      • Rahul on October 30, 2013 5:48 am

        Hi Gabriel,

        First check if bucket “WHATEVER” already exists. If it doesn’t already exists, we recommend to use lowercase characters in bucket name like “whatever”.

        Reply
        • Dale-Kurt Murray on April 23, 2014 4:13 pm

          Hi Gabriel,

          The reason you receive that error is a result of the S3 bucket name already exists. S3 bucket name must be unique (globally) on the services. So if you are trying to create a bucket called “whatever” and it exists, try creating “gabriel_whatever”

          Reply

    Leave A Reply Cancel Reply

    Advertisement
    Recent Posts
    • 20 Basic Linux Commands for the Beginners (Recommended)
    • tail Command in Linux with Examples
    • What is a Orphan Process in Unix/Linux
    • How To Display Warning Message to Unauthorized SSH Access
    • How to Set a Custom SSH Login Banner and MOTD
    Facebook Twitter Instagram Pinterest
    © 2023 Tecadmin.net. All Rights Reserved | Terms  | Privacy Policy

    Type above and press Enter to search. Press Esc to cancel.