kustomize: alternative to Helm for lightweight deployments with secrets
Tags: kubernetes, devops, microservices, tooling
Published: 2022-09-02 20:36:44

My journey started with the simple wish to move some of my applications from a docker environment to a kubernetes cluster I configured for some of my microservices.

## Table Of Contents
- [Why I looked for an alternative to Helm](#helm)
- [kustomize as an alternative](#kustomize)
- [The proposed deployment](#deployment)
- [Using system environment variables](#env)


Target applications for my thoughts were **small applications**, especially **Discord bots**. Those applications also need a couple of **secrets** like tokens and other credentials which might change depending on the deployment.

## Why I looked for an alternative to Helm <a name="helm"></a>

Since I was already used to utilising [Helm](https://helm.sh/) for quick and easy deployments, it seemed reasonable to also use it for a microservice like a Discord bot. For those who don't know Helm, it's a fully grown package manager for kubernetes with dozens of articles about it.

Being a - as mentioned "fully grown package manager" - also comes with the downside of shipping a lot of unnecessary bloat for very simple applications. To bring this into perspective, the `helm create` command allows us to generate a basic template - or - Helm chart. This very basic bootstrapped chart already comes with 10 files, spread over 3 directories (as shown below).

```
|-- Chart.yaml
|-- charts
|-- templates
|   |-- NOTES.txt
|   |-- _helpers.tpl
|   |-- deployment.yaml
|   |-- hpa.yaml
|   |-- ingress.yaml
|   |-- service.yaml
|   |-- serviceaccount.yaml
|   `-- tests
|       `-- test-connection.yaml
`-- values.yaml
```
I might be a bit of a purist, but **10 files** for the deployment of an application that doesn't even need a service sounds like a lot of bloat. Sure you can start cleaning the chart of unnecessary templates and keep the Helm benefits like versioning. This tho takes a considerable amount of time and doesn't really remove the clutter.

## kustomize as an alternative <a name="kustomize"></a>
[kustomize](https://kustomize.io/) is a built-in configuration management system that allows to _patch_ deployments if needed. It also ships a handy tool called `secretGenerator` that allows to **dynamically generate secrets** from a given source, something I'll heavily rely on in this example.

One sad thing about kustomize that I realised quite early: the documentation is still very poor and it lacks a lot of functionality that is already implemented. 

## The proposed deployment <a name="deployment"></a>

It's recommended to structure a kustomize deployment into two folders `base` for the default deployment and `overlays` for the patching deployments.

The simplest deployment that infuses our secrets can look like the following:

```
base
├── deployment.yaml
├── kustomization.yaml
└── secrets.env
```

The `deployment.yaml` contains the kubernetes manifest that we'll use for the deployment.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app
        image: ghcr.io/staubrein/myapp
        imagePullPolicy: Always
        env:
        - name: TOKEN
          valueFrom:
            secretKeyRef:
              name: my-app-secrets
              key: TOKEN
```
This manifest expects a secret `my-app-secret` with a key `TOKEN` and adds it to the container environment.

The content of this secret can be read from a casual `.env` file or in our case `secrets.env`.

```
TOKEN=XXXXXXXXXYXYYXYXYXYXYXYXYXYXXYXXXXXXXXYYY
```

To generate the secret from the file (and also system environment variables!) we first need to write the `kustomize.yaml`

```yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - deployment.yaml

secretGenerator:
- name: my-app-secrets
  env: secrets.env
```
This kustomize file takes the `deployment.yaml` as resource and combines it with generated secret from the `secretGenerator`.
It takes the values from the given `secrets.env` file into context and creates a secret with the given name `my-app-secrets`. It's important to import the file via the `env` key and **not** the `file` key, since `file` will turn the whole file into a secret while `env` makes all containing keys of the `secrets.env` accessible.

We can take a look at the finished manifests with `kubectl kustomize base`.

```yaml
apiVersion: v1
data:
  TOKEN: |
    T888as8d8dja8sdjjajd8ahjjdknkasndanddnasd8a8sda8sd
kind: Secret
metadata:
  name: my-app-secrets-fmk84tgkh5
type: Opaque
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: my-app
  name: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - env:
        - name: TOKEN
          valueFrom:
            secretKeyRef:
              key: TOKEN
              name: my-app-secrets-fmk84tgkh5
        image: ghcr.io/staubrein/myapp
        imagePullPolicy: Always
        name: my-app
```

The `secretGenerator` of kustomize dynamically generated a secret from the `secrets.env` file with base64 encoded values and those are accessible within the container.

### Using system environment variables <a name="env"></a>
The best part on this solution, what makes it also very interesting for **usage in pipelines** or environments that don't use variables from files, is that it also **works with system environment variables**.

By not assigning a value to the key in a environment file, it will automatically try to read out the value for this key from the system environment. Let's demonstrate this in a simple example and add another key to the `secrets.env`.
```
TOKEN=XXXXXXXXXYXYYXYXYXYXYXYXYXYXXYXXXXXXXXYYY
FOO
```
It is important, that there is **no equal sign** after the key, otherwise the value will be considered empty and not overwritten with a system environment variable.
Let's fill this variable locally with `export FOO=bar` and generate the secret with kustomize by running `kubectl kustomize base`.

```yaml
apiVersion: v1
data:
  FOO: YmFy
  TOKEN: WFhYWFhYWFhYWVhZWVhZWFlYWVhZWFlYWVhZWFhZWFhYWFhYWFhZWVk=
kind: Secret
metadata:
  name: dailyottr-secrets-h7b26g7c74
  namespace: my-app
type: Opaque
```
As seen in the output above, it set the variable `FOO` with the value we set in the system environment variable :)

The deployment can be triggered via `kubectl kustomize base | kubectl apply -f -`.

To fully show the potential of kustomize, especially the functionality of merging and overwriting different environments, I'll write another article soon that patches the `base` deployment.