How To Mount AWS S3 Bucket On Amazon EC2 Linux Using S3FS
Tags: s3, aws, amazonec2, ec2
Published: 2022-08-31 17:16:21

**In this blog, we will learn how to use S3 as a filesystem on EC2 Linux machine.**

Let's start!

1) Create an EC2 Linux (I have used Ubuntu in this demo) instance

Keep everything as default and add the below user data script to install awscli and s3fs utlity from advance section of wizard

```
sudo apt-get update -y
sudo apt-get install awscli -y
sudo apt-get install s3fs -y
```

2) Create an IAM user for s3fs


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/62lbeejalu3vc3afv9wi.png)

3) Give the user a unique name and enable programmatic access

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bhjh5fij2fbresx4410w.png)

Set permission --> create a new policy


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0xokovsrv1rl6np843c1.png)

Select the service as S3 and include below access levels

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pi28tw0vqfg0r7h2pirx.png)

Give the policy a unique name and click Create policy


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/69nev7ywzoytrwx6lp9z.png)

Once the policy is created, go back to the IAM tab and hit refresh so that newly created policy is included in the list
, filter by policy name and hit the enable checkbox to add the policy to our IAM user.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zjyznx5c6x2btyhm8v79.png)

Hit create user

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/37ioim2bm7m3u0m6ls3e.png)

Once the user is created, download the credentials. We are going to use it later.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2p63b4lqz87imcdf7j39.png)


4) Login to your Ec2 Instance


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qm3o1ozjtve6o7uqp8da.png)


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/q44vierl4fdg758ebvuk.png)

Go to your home directory and run below commands to create a new directory and to generate some sample files

```
mkdir /home/ubuntu/bucket; cd $HOME/bucket ;touch test1.txt test2.txt test3.txt

```


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7e68gu06tq2u7h997cqq.png)


Next step is to create an S3 bucket.
5) Go to S3 service and create a new bucket

give it a unique name and leave reast of the settings as default. 
Block public access to this bucket should be enabled by default


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/38j3rrx2ucarp3l9gz79.png)


Hit create bucket.

6) Once the bucket is created, go to the ssh session and configure our AWS credentails for authentication using the IAM account that we have created.

Use the command
``` 
aws configure

```
and provide the credential details that we have downloaded before


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0ifns2gi1u95nd1m7k96.png)

7) Now run the below command to sync local directory with the S3 bucket

```
aws s3 sync path_on_filesystem s3://bucketname

```
For example,

```
aws s3 sync /home/ubuntu/bucket s3://test-s3fs-101

```


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ixbibst94nu2spx5q3ge.png)

8) create the credential file for s3fs

s3fs supports the standard AWS credentials file stored in ${HOME}/.aws/credentials. Alternatively, s3fs supports a custom passwd file.
The default location for the s3fs password file can be created:
using a .passwd-s3fs file in the users home directory (i.e. ${HOME}/.passwd-s3fs)

file should have the below content:

```
$AWS_ACCESS_KEY_ID:$AWS_SECRET_KEY_ID

```

You can run the below command as well:
```
echo "AKIAQSCIQUH6XXYQMGDA:T5qM7rZmSaU3p/Y0xmuZyWv1/KUnT0Oc58sdCJ3t" > ${HOME}/.passwd-s3fs;
chmod 600 ${HOME}/.passwd-s3fs

```



![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/393l8s2ky0yradgehmr0.png)


9) Now you can run the command to mount S3 bucket as a filesystem

``` 
sudo s3fs bucketname path  -o passwd_file=$HOME/.passwd-s3fs,nonempty,rw,allow_other,mp_umask=002,uid=$UID,gid=$UID -o url=http://s3.aws-region.amazonaws.com
,endpoint=aws-region1,use_path_request_style
```

for example:

```
sudo s3fs s3fs-test-101 /home/ubuntu/bucket  -o passwd_file=$HOME/.passwd-s3fs,nonempty,rw,allow_other,mp_umask=002,uid=1000,gid=1000 -o url=http://s3.ca-central-1.amazonaws.com
,endpoint=ca-central-1,use_path_request_style
```

10) Once it is mounted successfully, you can verify by running the command 
```
mount|grep s3fs
```


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8ca2e4a4hzufcg8n4nrb.png)

11) Add the entry in fstab using the below command so that the changes become persistent after the server reboot as well:

```
bucketname directoryonfs fuse.s3fs _netdev,allow_other 0 0
```
For example:
```
s3fs-test-101 /home/ubuntu/bucket fuse.s3fs _netdev,allow_other 0 0
```

12) Now the moment of truth, go to your S3 bucket and hit refresh, you should see the files that were present in your file system


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ypssjdrnuh93ammql2xk.png)

13) Let's now verify whether it's getting synced properly after a object delete/addition

Go to your S3 bucket, and upload a new file

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dhq4axrye9pi99fjkvk7.png)

Go to your ssh session and do ls in the same directory


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9mnqukz36ruxx7xowlak.png)

**Eureka! The file that you just uploaded in your S3 bucket appears in your FileSystem.**

> Same way you can test the delete file operation. And it works both ways i.e if you perform any file operation on your filesystem, it will sync to your S3 bucket as well.

Feel free to checkout the below hands-on demo of what we have learned so far:

{% embed https://youtu.be/Qd85rhuFvqs %}


References:
https://github.com/s3fs-fuse/s3fs-fuse
https://aws.amazon.com/
https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html

#Limitations

**Generally S3 cannot offer the same performance or semantics as a local file system. More specifically:**

* random writes or appends to files require rewriting the entire object, optimized with multi-part upload copy
* metadata operations such as listing directories have poor performance due to network latency
* non-AWS providers may have eventual consistency so reads can temporarily yield stale data (AWS offers read-after-write consistency since Dec 2020)
* no atomic renames of files or directories
* no coordination between multiple clients mounting the same bucket
* no hard links
inotify detects only local modifications, not external ones by other clients or tools