Mount S3 Objects to Kubernetes Pods
Published: 2022-01-31 08:52:02

---
title: Mount S3 Objects to Kubernetes Pods
published: true
description: 
tags: 
cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d8lgeydiszs38byax12o.png
---


This post describes how to mount an S3 bucket to all the nodes in an EKS cluster and make it available to pods as a [hostPath volume](https://kubernetes.io/docs/concepts/storage/volumes/#hostpath). Yes, we're aware of the security implications of hostPath volumes,  but in this case it's less of an issue - because the actual access is granted to the S3 bucket (not the host filesystem) and access permissions are provided per serviceAccount.

## Goofys
We're using [goofys](https://github.com/kahing/goofys) as the mounting utility. It's a "high-performance, POSIX-ish Amazon S3 file system written in Go" based on [FUSE](https://en.wikipedia.org/wiki/Filesystem_in_Userspace) (file system in user space) technology.

## Daemonset
In order to provide the mount transparently we need to run a [daemonset](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) - so the mount is created on all nodes in the cluster.

## The Dockerfile and the Helm Chart
We've built our own goofys Docker image based on [Alpine Linux](https://alpinelinux.org/) and a Helm chart that installs the DaemonSet. 

The image is found on our Docker hub repo here: https://hub.docker.com/r/otomato/goofys

The Dockerfile and the Helm chart can be found here: https://github.com/otomato-gh/s3-mounter

## S3 Access per ServiceAccount

The Helm chart currently assumes that S3 access is provided by using an [IAM Role attached to a kubernetes serviceAccount](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html). We may add API access keys support in the future if needed.

## HowTo:
Here's how to set it all up:

### 1. OIDC Provider for EKS

Make sure you have an [IAM OIDC identity provider for your cluster](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html). If not - you can use the following commands (you'll need `eksctl` installed):

```bash
aws eks describe-cluster --name cluster_name --query "cluster.identity.oidc.issuer" --output text`
```

Example output:

`https://oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E`

List the IAM OIDC providers in your account. Replace `EXAMPLED539D4633E53DE1B716D3041E` with the value returned from the previous command.

```
aws iam list-open-id-connect-providers | grep EXAMPLED539D4633E53DE1B716D3041E
```
Example output

`"Arn": "arn:aws:iam::111122223333:oidc-provider/oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E"`

If output is returned from the previous command, then you already have a provider for your cluster. If no output is returned, then you must create an IAM OIDC provider with the following command. Replace `cluster_name` with your own value.

```bash
eksctl utils associate-iam-oidc-provider --cluster cluster_name --approve
```

### 2. Create a Managed Policy for Bucket Access

Create json file named `policy.json` with the appropriate policy definition. For example - the following code snippet creates a json file that allows full access to a bucket named `my-kubernetes-bucket`:
```bash
read -r -d '' MY_POLICY <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:*",
            ],
            "Resource": [
                "arn:aws:s3:::my-kubernetes-bucket"
            ]
        }
    ]
}
EOF
echo "${MY_POLICY}" > policy.json
```

Create the managed policy by running:
```bash
aws iam create-policy --policy-name kubernetes-s3-access --policy-document file://policy.json
```
Example output:
> {
>    "Policy": {
>        "PolicyName": "kubernetes-s3-access",
>        "PolicyId": "ANPAS3DOMWSIX73USJOHK",
>        "Arn": `"arn:aws:iam::04968064045764:policy/kubernetes-s3-access"`,

Note the policy ARN for the next step.

### 3. Create a Role for S3 Access

Set your AWS account ID to an environment variable with the following command:
```bash
ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text)
```
Set your OIDC identity provider to an environment variable with the following command. Replace the example values with your own values:
```bash
OIDC_PROVIDER=$(aws eks describe-cluster --name cluster-name --query "cluster.identity.oidc.issuer" --output text | sed -e "s/^https:\/\///")
```

Copy the following code block to your computer and replace the example values with your own values.
```bash
read -r -d '' TRUST_RELATIONSHIP <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${OIDC_PROVIDER}:sub": "system:serviceaccount:my-namespace:my-service-account"
        }
      }
    }
  ]
}
EOF
echo "${TRUST_RELATIONSHIP}" > trust.json
```
Run the modified code block from the previous step to create a file named `trust.json`.

Run the following AWS CLI command to create the role:
```
aws iam create-role --role-name eks-otomounter-role --assume-role-policy-document file://trust.json --description "Mount s3 bucket to EKS"
```
Run the following command to attach the IAM policy using the ARN created in the previous section to your role:
```
aws iam attach-role-policy --role-name eks-otomounter-role --policy-arn=IAM_POLICY_ARN
```

### 4. Finally - Install the S3 Mounter!

- Add the helm repo to your repo list:
```bash
helm repo add otomount https://otomato-gh.github.io/s3-mounter
```

- Inspect its arguments in `values.yaml`
```bash
helm show values otomount/s3-otomount
```
The values you want to set are in the end:
```yaml
bucketName: my-bucket
iamRoleARN: my-role
mountPath: /var/s3
hostPath: /mnt/s3data
```

- Install the chart by providing your own values:
```bash
helm upgrade --install s3-mounter otomount/s3-otomount   \ 
--namespace otomount --set bucketName=<your-bucket-name> \
--set iamRoleARN=<your-role-arn> --create-namespace
```

This will use the default hostPath for the mount - i.e `/mnt/s3data`

### 5. Use the mounted S3 bucket in your Deployments. 

Here's an example pod definition that provides its container the access to the mounted bucket:
 
```yaml
apiVersion: v1
kind: Pod
metadata:
  name: sleeper
spec:
  containers:
  - command:
    - sleep
    - infinity
    image: ubuntu
    name: ubuntu
    volumeMounts:
    - mountPath: /mydata:shared
      name: s3data
  volumes:
  - hostPath:
      path: /mnt/s3data
    name: s3data
```

Note the `:shared` - it's a [mount propagation](https://kubernetes.io/docs/concepts/storage/volumes/#mount-propagation) modifier in the `mountPath` field that allows this volume to be shared by multiple pods/containers on the same node.

And that's it! You can now access your bucket. If you've created the pod from our example - you can exec to verify:

```bash
kubectl exec sleeper -- ls /mydata
```

Note: running this on your cluster will cost you a few additional $ for S3 API calls that goofys performs to maintain the mount. So remember to monitor your cloud costs. But you should do that anyway, right?

Happy delivering!