S3FS is FUSE (File System in User Space) based solution to mount an Amazon S3 buckets, We can use system commands with this drive just like as another Hard Disk in system. On s3fs mounted files systems we can simply use cp, mv and ls the basic Unix commands similar to run on locally attached disks.
If you like to access S3 buckets without mounting on system, use s3cmd command line utility to manage s3 buckets. s3cmd is also provides faster speed for data upload and download rather than s3fs. To work with s3cmd use next articles to install s3cmd in Linux systems and Windows systems.
This article will help you to install S3FS and Fuse by compiling from source, and also help you to mount S3 bucket on your CentOS/RHEL and Ubuntu systems.
Step 1: Remove Existing Packages
First check if you have any existing s3fs or fuse package installed on your system. If installed it already remove it to avoid any file conflicts.
CentOS/RHEL Users: # yum remove fuse fuse-s3fs Ubuntu Users: $ sudo apt-get remove fuse
Step 2: Install Required Packages
After removing above packages. First we will install all dependencies for fuse and s3cmd. Install the required packages to system use following command.
CentOS/RHEL Users: # yum install gcc libstdc++-devel gcc-c++ curl-devel libxml2-devel openssl-devel mailcap Ubuntu Users: $ sudo apt-get install build-essential libcurl4-openssl-dev libxml2-dev mime-support
Step 3: Download and Compile Latest Fuse
Download and compile latest version of fuse source code. For this article we are using fuse version 2.9.3. Following set of command will compile fuse and add fuse module in kernel.
# cd /usr/src/ # wget http://downloads.sourceforge.net/project/fuse/fuse-2.X/2.9.3/fuse-2.9.3.tar.gz # tar xzf fuse-2.9.3.tar.gz # cd fuse-2.9.3 # ./configure --prefix=/usr/local # make && make install # export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig # ldconfig # modprobe fuse
Step 4: Download and Compile Latest S3FS
Download and compile latest version of s3fs source code. For this article we are using s3fs version 1.74. After downloading extract the archive and compile source code in system.
# cd /usr/src/ # wget https://s3fs.googlecode.com/files/s3fs-1.74.tar.gz # tar xzf s3fs-1.74.tar.gz # cd s3fs-1.74 # ./configure --prefix=/usr/local # make && make install
Step 5: Setup Access Key
Also In order to configure s3fs we would required Access Key and Secret Key of your S3 Amazon account. Get these security keys from Here.
# echo AWS_ACCESS_KEY_ID:AWS_SECRET_ACCESS_KEY > ~/.passwd-s3fs # chmod 600 ~/.passwd-s3fs
Note: Change AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY with your actual key values.
Step 6: Mount S3 Bucket
Finally mount your s3 bucket using following set of commands. For this example, we are using s3 bucket name as mydbbackup and mount point as /s3mnt.
# mkdir /tmp/cache # mkdir /s3mnt # chmod 777 /tmp/cache /s3mnt # s3fs -o use_cache=/tmp/cache mydbbackup /s3mnt
On Ubuntu steps 1-4 can be replaced with one command:
sudo apt-get install s3fs
Do this instead of steps 1-4 and continue from step 5. It works – tested.
Do NOT use fuse3* but stick to a fuse 2*, otherwise you will not be able to install s3fs.
I am getting below error.
fuse: warning: library too old, some operations may not not work
# rpm -qa | grep fuse
libconfuse-2.7-4.el6.x86_64
fuse-libs-2.8.3-5.el6.x86_64
#
though latest package is not available in Yum repo. please suggest
Hello, i followed your guide (on Ubuntu 14.04, Bitnami – EC2 ) , i am getting following error at “make”
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
collect2: error: ld returned 1 exit status
make[2]: *** [s3fs] Error 1
make[2]: Leaving directory `/tmp/s3fs-fuse-1.80/src’
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/tmp/s3fs-fuse-1.80′
make: *** [all] Error 2
use this if you get “bash: s3fs: command not found”
sudo /usr/local/bin/s3fs -o use_cache=/tmp/cache mydbbackup /s3mnt
Those of you having an error on CentOS: s3fs: error while loading shared libraries: libfuse.so.2: cannot open shared object file: No such file or directory
yum install fuse-libs
Great article! Worked a treat….
only issue was my library path required updating as when I ran s3fs it couldn’t find the fuse dependancies.
LD_LIBRARY_PATH=/usr/local/lib
export LD_LIBRARY_PATH
That did the trick 🙂
Hi..
i have mounted s3 sucessfully , but when i try to “cd ” to mounted dirs it says “operation not permitted”
cd: app_logs/: Operation not permitted
below is permission for dir
d———. 1 root root 1 May 6 2015 app_logs
below command used
s3fs -o use_cache=/tmp/cache s3bukcket /s3mnt
what permission i need to set for bucket or bucket folder ?
am i writing wrong command for mount?
Hi Rahul,
Can you please share the steps to mount Amazon S3 bucket on windows OS for both 2008 & 2012?
Thanks & Regards,
Mehul
Thanks! This was quite helpful.
You can take a look at the docker image which I built, with S3FS and S3 bucket mounting capabilities:
https://registry.hub.docker.com/u/ihealthtechnologies/s3-mount/
Hi all, I’m receiving the following error after trying to mount the bucker:
# s3fs -o user_cache=/tmp/cache lehar-backup /s3mnt
s3fs: /lib/libfuse.so.2: version `FUSE_2.8′ not found (required by s3fs)
I downloaded fuse 2.8 and compiled it per the instructions however if there is something I’m missing I’d love to know what that is. Let me know what information you might need. Thanks for any help you can provide.
~Dave
My disk is full after mount and not able to use the system. Ideally it should not take disk space.
How can i resolve the issue?
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda1 10321208 10297608 0 100% /
none 847476 0 847476 0% /dev/shm
s3fs 274877906944 0 274877906944 0% /s3mnt
I’m got error :
s3fs: HTTP: 403 Forbidden – it is likely that your credentials are invalid
are you have solved ??
I am getting error:
s3fs: unable to access MOUNTPOINT storingfiles1: No such file or directory
Hi Team,
Very good it working fine in my ubuntu system
So far so good but I get this error:
[[email protected] s3fs-1.74]# s3fs -o use_cache=/tmp/cache agarta /etc/httpd/imagestore/
bash: s3fs: command not found
any idea on how to resolve this?
Hi Rahul,
Thanks for the tutorial. All worked fine, except this messsage – Transport endpoint is not connected. Unmounted and mounted again using the link you shared in comments. I am using s3fs to connect to Google Cloud Storage.
Kindly share some pointers or might be issue with the permission level mentioned above.
Willian
do it:
# vim /etc/ld.so.conf
/usr/local/lib
include ld.so.conf.d/*.conf
# ldconfig
# s3fs -o use_cache=/tmp/cache ****.******.*** /s3mnt
s3fs: error while loading shared libraries: libfuse.so.2: cannot open shared object file: No such file or directory
I have a problem, can you helpme?
[[email protected]*********** s3fs-1.74]# s3fs -o use_cache=/tmp/cache ****.******.*** /s3mnt
s3fs: error while loading shared libraries: libfuse.so.2: cannot open shared object file: No such file or directory
hi
install fuse-libs and it should solve your issue… it did in my case…
sudo yum install fuse-libs
hope that helps
thanks
yes, you could add this in the tutorial ! it solved it in my case too. Thanks
thanks!
Followed this tutorial – doesn’t work:
[email protected]:/home/ubuntu# ls /s3mnt
ls: cannot access /s3mnt: Transport endpoint is not connected
[email protected]:/home/ubuntu#
Hi,
Try to umount and mount s3bucket again and wait for few time to sync. Also visit this link.
The problem was permissions on S3 bucket. Maybe you could add some recommendations on this subject? It would make your article more complete.
Thank you
Hi bo,
We would be thankful for your contributions and make the article more useful for users.
We will also check for the permission problems.
I would also add info on automatic mounting the s3 bucket. Something like editing /etc/fdtab to add
s3fs#bucketname mountpath fuse allow_other,_netdev,nosuid,nodev,url=https://s3.amazonaws.com 0 0
I have followed the tutorial and get to the state of “Transport endpoint is not connected”. Any updates on what permissions need to be set on S3 bucket?
and about fstab?
How can I put cache dir