Level up your secrets management in Kubernetes using AWS Secret Manager and Helm
Tags: security, cloud, aws, tutorial
Published: 2022-12-26 09:04:11

Engineers' biggest struggle when writing Kubernetes resources is to keep all secrets secure. To be honest, the secrets k8s resource is not secured at all because base64 is not encryption!

Required skills: 

- Understanding basic concepts about Kubernetes
- Cloud providers resources knowledge
- Helm **v3** installed on a Kubernetes cluster.

## Understanding Helm

For this article and for Formance infrastructure we chose to use Helm as a templating system for our k8s resources. This software is commonly known as the package manager for Kubernetes but they also provide a good system for deploying many versions of the same Charts, like for production and staging environments.

### Initialize the application

First, we have to create a new application using the Helm CLI. You could find how to install it here: [https://helm.sh/docs/intro/quickstart/](https://helm.sh/docs/intro/quickstart/). Once all is installed correctly, you could run the command: 

```bash
helm create "<your_application_name>"
```

The directory Helm creates should look like this ⬇️

![Tree of the directory](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pot60aquqs1bviodsjo7.png)

In this article, we’re going to stay basic and remove some templates to keep it simple. We’re only keeping: 

- The deployment template
- The service template
- The service account template
- The helpers, Notes, and Tests

Finally, we will create a file named `secret.yaml` for the template of our SecretProvider object. 

If you’re not familiar with the Helm templating language, I recommend you take a look at the documentation: [https://helm.sh/docs/chart_template_guide/](https://helm.sh/docs/chart_template_guide/)

## The secret provider object

### Concept

#### What are CSI Drivers?

To perform what we want, we’re going to use a Kubernetes resource named SecretProviderClass from the API `secrets-store.csi.x-k8s.io/v1alpha1`.  The secret store is provided by the Kubernetes Container Storage Interface which here helps us to connect basic resources like volumes or secrets to cloud providers' services. 

The main problem of Secrets resources on Kubernetes is the lack of security and if I want to share some private values as database passwords to my application deployment, I need more than just a base64 encryption.

Here at Formance we use Amazon Web Services as cloud provider so I will use the service AWS Secret Manager (docs: [https://docs.aws.amazon.com/secretsmanager/?id=docs_gateway](https://docs.aws.amazon.com/secretsmanager/?id=docs_gateway)). When we have created our Relational Database Service with MariaDB or PostgreSQL or whatever, using InfraAsCode tools Terraform, we have automatically created a Secret with the database login information. The problem is to retrieve those values directly on my application container automatically. 

_Simplified overview of the workflow_
![Simplified overview of the workflow](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/uaowdoz6zu6vzohti99t.png)

#### Installing the Secret Store CSI driver

That’s here where Kubernetes CSI enters in action ! It will create an interface between our Secret object on AWS and our application deployment. First of all, we have to install the Secret Store CSI driver using helm.

```bash
helm repo add secrets-store-CSI-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver --namespace kube-system
```

You could find an alternative installation here: [https://secrets-store-csi-driver.sigs.k8s.io/getting-started/installation.html](https://secrets-store-csi-driver.sigs.k8s.io/getting-started/installation.html)

It will install things like the operator, and service account with the right RBAC permissions and most important, it will declare the Custom Resources Definition of the SecretProviderClass. Without this, the Kubernetes Scheduler will not understand the API version and the object kind. 

Secret store CSI driver also supports GCP, Azure, and Vault secrets providers.

### Deep dive into the secret template

The requirement is to make a template where we could inject some specific values like:

1. The name obviously
2. The secret provider (note that this example is only tested on AWS)
3. And the secrets to retrieve using the secret name or in AWS its Amazon Resource Name (ARN)

Keep in mind that this example is a simple template with multiple secrets but you personalize it as you wish. You can find various 

Let’s take a look at the template and its values:

![secret yaml](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ule95nfdlewdrs9octzt.png)

![extract of values.yaml](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9sfnn04spy7d5674sbdk.png)


Once you create all templates, all you have to do is to apply the command:

`helm install <application name> .`

And you're all done!

Image credits: [Midjourney](https://www.midjourney.com/home/)