AWS: Kubernetes  -  AWS Secrets Manager and Parameter Store integration
Tags: security, aws, kubernetes, tutorial
Published: 2023-08-19 14:15:59

---
title: AWS: Kubernetes  -  AWS Secrets Manager and Parameter Store integration
published: true
date: 2023-07-22 10:08:49 UTC
tags: security,aws,kubernetes,tutorial
canonical_url: https://rtfm.co.ua/en/aws-kubernetes-aws-secrets-manager-and-parameter-store-integration/
---

### AWS: Kubernetes  -  AWS Secrets Manager and Parameter Store integration

![](https://cdn-images-1.medium.com/max/1024/1*JZAYCCIy6Muaxtz2FKpu0w.png)

Storing access data in Kubernetes Secrets has an important drawback, because they are only available within the Kubernetes cluster itself.

To make them available to external services, we can use Hashicorp Vault and integrate it with Kubernetes using solutions such as [`vault-k8s`](https://github.com/hashicorp/vault-k8s)or use services from AWS - [Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html) or [Parameter Store](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html).

Integrating AWS Secrets Manager and Parameter Store into Kubernetes will give us the ability to create a new type of resource — `SecretProviderClass` which we can connect to Kubernetes Pods as files or environment variables.

For this, we will need [AWS Secrets and Configuration Provider (ASCP)](https://docs.aws.amazon.com/secretsmanager/latest/userguide/integrating_csi_driver.html) and [Kubernetes Secrets Store CSI Driver](https://secrets-store-csi-driver.sigs.k8s.io/getting-started/installation.html).

### AWS Secrets and Configuration Provider vs Hashicorp Vault

I haven’t used Vault for a long time, but regarding the question “What to use”, it’s a choice between the “setup, configure, and manage Hashicorp Vault yourself” (installation of the Helm chart and configuration of accesses) or “use a ready-made solution from AWS” (in fact, you only need to configure IAM roles).

Also, keep in mind that using AWS services (surprise!) is paid, so if you plan to have thousands of secrets, it’s probably better to use Vault.

In addition, Vault itself provides much more opportunities, for example, the generation of temporary tokens for services, plus, as far as I remember — Kubernetes Pods can receive parameters from Vault without the need to create Kubernetes Secrets, while when using AWS Secrets and Configuration Provider and Kubernetes Secrets Store, CSI Driver for connecting variables will create Kubernetes Secrets.

However, our project already uses Secrets Manager and Parameter Store, so I don’t see a point in Vault (yet), so let’s integrate our existing secrets into an AWS Elastic Kubernetes Service cluster.

### AWS Secrets Manager vs Parameter Store

You can read more about the difference between them here — [AWS — Difference between Secrets Manager and Parameter Store (Systems Manager)](https://medium.com/awesome-cloud/aws-difference-between-secrets-manager-and-parameter-store-systems-manager-f02686604eae), here just briefly.

Common features:

- both use AWS KMS to encrypt data
- both are Key/Value Store
- both support versioning

Key differences:

- **Cost** :
- Secrets Manager: Charges $0.40 for each secret and $0.05 for every 10,000 API requests
- Parameter Store: for the Standard, it does not take money for storage, with [higher throughput](https://docs.aws.amazon.com/systems-manager/latest/userguide/parameter-store-throughput.html) it costs $0.05 for every 10,000 API requests, with Advanced parameters — $0.05 for storage and $0.05 for every 10,000 API requests
- **Secrets rotation** :
- Secrets Manager: has a built-in rotation mechanism and integrates it with services (RDS, DocumentDB, etc)
- Parameter Store: You must implement the rotation yourself
- **Cross-account Access** :
- Secrets Manager: Supports
- Parameter Store: Not supported
- **Cross-Regions Replication** :
- Secrets Manager: Supports
- Parameter Store: Not supported
- **Data limits** :
- Secrets Manager: up to 10KB per secret
- Parameter Store: 4KB for each record (8KB for Advanced Parameters)
- **Quantity limits:**
- Secrets Manager: 500,000 per region and account
- Parameter Store: 10,000 per region and account

### Installing Secrets Store CSI Driver

So, for integration, we need two services — Secrets Store CSI Driver and AWS Secrets and Configuration Provider.

First, we’ll add the Secrets Store CSI Driver.

With its help, we will be able to connect secrets/parameters from AWS as files or variables to Kubernetes Pods.

Add a Helm chart and install it with the `syncSecret.enabled=true` parameter to create Kubernetes Secrets and synchronize them with AWS secrets during data rotation (see [Sync as Kubernetes Secret](https://secrets-store-csi-driver.sigs.k8s.io/topics/sync-as-kubernetes-secret.html) ):

```
$ helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
$ helm -n kube-system install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver — set syncSecret.enabled=true
```

Check Pods:

```
$ kubectl -n kube-system get pod | grep secret
csi-secrets-store-secrets-store-csi-driver-kzmcx 3/3 Running 0 31s
csi-secrets-store-secrets-store-csi-driver-t7bqc 3/3 Running 0 31s
```

### Installing AWS Secrets and Configuration Provider

Add a repository and install the Helm chart:

```
$ helm repo add aws-secrets-manager https://aws.github.io/secrets-store-csi-driver-provider-aws
$ helm install -n kube-system secrets-provider-aws aws-secrets-manager/secrets-store-csi-driver-provider-aws
```

Check Pods:

```
$ kubectl -n kube-system get pod | grep secret
csi-secrets-store-secrets-store-csi-driver-kzmcx 3/3 Running 0 9m
csi-secrets-store-secrets-store-csi-driver-t7bqc 3/3 Running 0 9m
secrets-provider-aws-secrets-store-csi-driver-provider-awskq5g8 1/1 Running 0 23s
secrets-provider-aws-secrets-store-csi-driver-provider-awsksq9d 1/1 Running 0 23s
```

And look at the [CSIDriver](https://kubernetes-csi.github.io/docs/csi-driver-object.html):

```
$ kubectl get csidriver
NAME ATTACHREQUIRED PODINFOONMOUNT STORAGECAPACITY TOKENREQUESTS REQUIRESREPUBLISH MODES AGE
ebs.csi.aws.com true false false <unset> false Persistent 46h
efs.csi.aws.com false false false <unset> false Persistent 4d
secrets-store.csi.k8s.io false true false <unset> false Ephemeral 10m
```

Next, we will configure IAM for IRSA.

### IAM Policy та IAM Role для ServiceAccount

In order for Kubernetes Pods to be able to access AWS SecretManager and Parameter Store, we will use IRSA — create a ServiceAacount that will use an IAM Role with an IAM Policy that will have permissions to call Secrets Manager and Parameter Store (see [AWS: EKS, OpenID Connect, and ServiceAccounts](https://rtfm.co.ua/en/aws-eks-openid-connect-and-serviceaccounts/)[)](https://rtfm-co-ua.translate.goog/aws-eks-openid-connect-ta-serviceaccounts/?_x_tr_sl=uk&_x_tr_tl=en&_x_tr_hl=en&_x_tr_pto=wapp).

Describe an IAM policy:

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "secretsmanager:DescribeSecret",
                "secretsmanager:GetSecretValue",
                "ssm:DescribeParameters",
                "ssm:GetParameter",
                "ssm:GetParameters",
                "ssm:GetParametersByPath"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}
```

Create it in the IAM:

```
$ aws iam create-policy --policy-name ascp-iam-policy --policy-document file://ascp-policy.json
{
“Policy”: {
“PolicyName”: “ascp-policy”,
“PolicyId”: “ANPAXFIUAIGSBPFEDKZZT”,
“Arn”: “arn:aws:iam::492***148:policy/ascp-iam-policy”,
…
```

Describe the Trust policy for the IAM Role:

```
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Principal": {
        "Federated": "arn:aws:iam::492 ***148:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/2DC*** 124"
      },
      "Condition": {
        "StringEquals": {
          "oidc.eks.us-east-1.amazonaws.com/id/2DC***124:aud": "sts.amazonaws.com",
          "oidc.eks.us-east-1.amazonaws.com/id/2DC***124:sub": "system:serviceaccount:default:ascp-test-serviceaccount"
        }
      }
    }
  ]
}
```

Create the Role itself with this trust policy:

```
$ aws iam create-role --role-name ascp-iam-role --assume-role-policy-document file://ascp-trust.json
{
“Role”: {
“Path”: “/”,
“RoleName”: “ascp-iam-role”,
“RoleId”: “AROAXFIUAIGSLDCB3L4AR”,
“Arn”: “arn:aws:iam::492***148:role/ascp-iam-role”,
```

…

Attach the `ascp-iam-policy` policy to that role:

```
$ aws iam attach-role-policy --role-name ascp-iam-role --policy-arn=arn:aws:iam::492***148:policy/ascp-iam-policy
```

Now we can create a SecretProviderClass and a Pod that will use it.

### Create a SecretProviderClass

Let’s add a SecretProviderClass, which will receive a string from the Secrets Manager and a string from the Parameter Store, and then we will connect them to a Kubernetes Pod.

Create a secret in Secrets Manager:

```
$ aws secretsmanager create-secret --name ascp-secret-test-string --secret-string "secretLine"
{
“ARN”: “arn:aws:secretsmanager:us-east-1:492***148:secret:ascp-secret-test-string-DNweNg”,
“Name”: “ascp-secret-test-string”,
“VersionId”: “9d4f490d-edcc-4ee0-b43d-5b4e25fa271b”
}
```

Create a record in the Parameter Store:

```
$ aws ssm put-parameter --name ascp-ssm-test-param --value "paramLine" --type "String"
{
“Version”: 1,
“Tier”: “Standard”
}
```

Next, describe a SecretProviderClass with two objects - in the `parameters.objects.objectName` set a name of the object in the Secrets Manager or Parameter Store and in the `objectType` set where we get this object from:

```
--- 
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata: 
  name: aspc-test-secret-class
spec:         
  provider: aws
  parameters:
    objects: |
        - objectName: "ascp-test-string"
          objectType: "secretsmanager"
        - objectName: "ascp-ssm-test-param"
          objectType: "ssmparameter"
```

Let’s go to the Pod.

#### Attaching a SecretProviderClass to a Pod as a file

Add a ServiceAccount with the IAM role we created earlier, and a Pod with this ServiceAccount:

```
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ascp-test-serviceaccount
  namespace:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::492***148:role/ascp-iam-role
---
apiVersion: v1
kind: Pod
metadata:
  name: ascp-test-pod
spec:
  containers:
    - name: my-aws-cli
      image: amazon/aws-cli:latest
      command: ['sleep', '36000']
      volumeMounts:
      - name: ascp-test-secret-volume
        mountPath: /mnt/ascp-secret
        readOnly: true
  restartPolicy: Never
  serviceAccountName: ascp-test-serviceaccount
  volumes:
  - name: ascp-test-secret-volume
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: aspc-test-secret-class
```

Deploy it:

```
$ kubectl apply -f ascp-test.yaml
serviceaccount/ascp-test-serviceaccount created
secretproviderclass.secrets-store.csi.x-k8s.io/aspc-test-secret-class created
pod/ascp-test-pod created
```

Check the Pod:

```
$ kk describe pod ascp-test-pod
…
Mounts:
/mnt/ascp-secret from ascp-test-secret-volume (ro)
…
Volumes:
…
ascp-test-secret-volume:
Type: CSI (a Container Storage Interface (CSI) volume source)
Driver: secrets-store.csi.k8s.io
FSType:
ReadOnly: true
VolumeAttributes: secretProviderClass=aspc-test-secret-class
…
```

And a content of the `/mnt/ascp-secret` directory:

```
$ kk exec -ti ascp-test-pod -- ls -l /mnt/ascp-secret
total 8
-rw-r — r — 1 root root 10 Jul 17 09:32 ascp-secret-test-string
-rw-r — r — 1 root root 9 Jul 17 09:32 ascp-ssm-test-param
```

And files there:

```
$ kk exec -ti ascp-test-pod -- cat /mnt/ascp-secret/ascp-secret-test-string
secretLine
$ kk exec -ti ascp-test-pod -- cat /mnt/ascp-secret/ascp-ssm-test-param
paramLine
```

#### Attaching a SecretProviderClass to a Pod as an environment variable

Mounting secrets as a file may be a good solution for some `.env` files, but what about environment variables? For example to pass a `DB_PASSWORD`.

To do so, we can add a `secretObjects` to the SecretProviderClass - then [Kubernetes Secrets Store CSI Driver](https://secrets-store-csi-driver.sigs.k8s.io/getting-started/installation.html) will create a Kubernetes Secret, which we can connect to the Pod:

```
---
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: aspc-test-secret-class
spec:
  provider: aws
  parameters:
    objects: |
        - objectName: "ascp-secret-test-string"
          objectType: "secretsmanager"
        - objectName: "ascp-ssm-test-param"
          objectType: "ssmparameter"
  secretObjects:
    - secretName: aspc-test-kube-secret
      type: Opaque
      data:
        - objectName: ascp-secret-test-string
          key: kube-secret-key
```

Here:

- `secretObjects.secretName`: a name of the Kubernetes Secret to be created
- `secretObjects.secretName.data.objectName`: must be the same as parameters.objects.objectName
- `secretObjects.secretName.data.key`: a key for the Kubernetes Secret - `data.kube-secret-key`

Deploy, and check Kubernetes Secret:

```
$ kk get secret aspc-test-kube-secret -o yaml
apiVersion: v1
data:
kube-secret-key: c2VjcmV0TGluZQ==
…
```

And a value of the `kube-secret-key`:

```
$ echo c2VjcmV0TGluZQ== | base64 -d
secretLine
```

Now, let’s connect it to the Pod — add `spec.containers.env` with the `valueFrom.secretKeyRef`:

```
---
apiVersion: v1
kind: Pod
metadata:
  name: ascp-test-pod
spec:
  containers:
    - name: my-aws-cli
      image: amazon/aws-cli:latest
      command: ['sleep', '36000']
      env:
      - name: SECRET
        valueFrom:
          secretKeyRef:
            name: aspc-test-kube-secret
            key: kube-secret-key
      volumeMounts:
      - name: ascp-test-secret-volume
        mountPath: /mnt/ascp-secret
        readOnly: true
  restartPolicy: Never
  serviceAccountName: ascp-test-serviceaccount
  volumes:
  - name: ascp-test-secret-volume
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: aspc-test-secret-class
```

Deploy, and check:

```
$ kk exec -ti ascp-test-pod -- printenv | grep SECRET
SECRET=secretLine
```

Or:

```
$ kk exec -ti ascp-test-pod -- bash
bash-4.2# echo $SECRET
secretLine
```

During this, we have to attach the `volumes` and `volumeMounts`, as it was done for mounting Secrets as a file.

#### Creating a SecretProviderClass from JSON

If a data in Secrets Manager and Parameter Store is stored in JSON, then we should use the [`jmesPath`](https://jmespath.org/) in our SecretProviderClass.

Let’s create another secret in Secrets Manager from JSON:

```
$ aws secretsmanager create-secret --name ascp-secret-test-json --secret-string '{"username":"admin", "password":"foobar"}'
{
“ARN”: “arn:aws:secretsmanager:us-east-1:492***148:secret:ascp-secret-test-json-iOtcBf”,
“Name”: “ascp-secret-test-json”,
“VersionId”: “32666608–6416–46cf-8b93-cf090eef1bc5”
}
```

Check it:

![](https://cdn-images-1.medium.com/max/524/0*1BXotGZV2ETpeBRg.png)

Update our SecretProviderClass:

```
---
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: aspc-test-secret-class
spec:
  provider: aws
  parameters:
    objects: |
        - objectName: "ascp-secret-test-string"
          objectType: "secretsmanager"
        - objectName: "ascp-ssm-test-param"
          objectType: "ssmparameter"
        - objectName: "ascp-secret-test-json"
          objectType: "secretsmanager"
          jmesPath:
              - path: "username"
                objectAlias: "ascp-test-username"
              - path: "password"
                objectAlias: "ascp-test-password"
  secretObjects:
    - secretName: aspc-test-kube-secret
      type: Opaque
      data:
        - objectName: ascp-secret-test-string
          key: kube-secret-key
    - secretName: aspc-test-kube-secret-json
      type: Opaque
      data:
        - objectName: ascp-test-username
          key: kube-secret-user
        - objectName: ascp-test-password
          key: kube-secret-pass
```

Here:

- in the `parameters.objects.objectName: "ascp-secret-test-json"` use `jmesPath`, which parses our secret and receives the values ​​of two fields - `username` and `password`, for which it creates two `objectAlias`- _ascp-test-username_ and _ascp-test-password_
- in the `secretObjects.secretName: aspc-test-kube-secret-json` we add a `data` with two `objectName`, in which we use `objectAlias` from the parameters

Update our Kubernetes Pod — add two `secretKeyRef` with the `kube-secret-user` and `kube-secret-user` keys from the `aspc-test-kube-secret-json` Secret:

```
---
apiVersion: v1
kind: Pod
metadata:
  name: ascp-test-pod
spec:
  containers:
    - name: my-aws-cli
      image: amazon/aws-cli:latest
      command: ['sleep', '36000']
      env:
      - name: SECRET
        valueFrom:
          secretKeyRef:
            name: aspc-test-kube-secret
            key: kube-secret-key
      - name: USER
        valueFrom:
          secretKeyRef:
            name: aspc-test-kube-secret-json
            key: kube-secret-user
      - name: PASS
        valueFrom:
          secretKeyRef:
            name: aspc-test-kube-secret-json
            key: kube-secret-pass
      volumeMounts:
      - name: ascp-test-secret-volume
        mountPath: /mnt/ascp-secret
        readOnly: true
  restartPolicy: Never
  serviceAccountName: ascp-test-serviceaccount
  volumes:
  - name: ascp-test-secret-volume
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: aspc-test-secret-class
```

Deploy and check Kubernetes Secrets:

```
$ kk get secret
NAME TYPE DATA AGE
aspc-test-kube-secret Opaque 1 2s
aspc-test-kube-secret-json Opaque 2 2s
```

And a value from the `aspc-test-kube-secret-json`:

```
$ kk get secret aspc-test-kube-secret-json -o yaml
apiVersion: v1
data:
kube-secret-pass: Zm9vYmFy
kube-secret-user: YWRtaW4=
…
```

And environment variables in the Pod:

```
$ kk exec -ti ascp-test-pod --printenv | grep 'SECRET\|USER\|PASS'
SECRET=secretLine
USER=admin
PASS=foobar
```

Done.

_Originally published at_ [_RTFM: Linux, DevOps, and system administration_](https://rtfm.co.ua/en/aws-kubernetes-aws-secrets-manager-and-parameter-store-integration/)_._

* * *