Managing Kubernetes Secrets with the External Secrets Operator and Doppler
Published: 2022-10-26 05:41:00

---
title: Managing Kubernetes Secrets with the External Secrets Operator and Doppler
published: true
date: 2022-10-25 15:48:28 UTC
tags: 
canonical_url: https://blog.doppler.com/managing-kubernetes-secrets-with-the-external-secrets-operator-and-doppler
---

![Cover](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o5hzbzxxhk2psd0fqto6.jpg)

API keys, credentials, and other types of sensitive information are the primary way your application calls external services and interacts with the world outside its own runtime. Ensuring that your secrets are secure is one of the most important operational tasks that you need to address, but it's not just rigorous security that's important providing an ergonomic, audited, and maintainable flow for the _management_ of those secrets is crucial as well. If managing credentials is confusing or difficult, mistakes can undo all of the careful work that goes into keeping secrets secure.

Container orchestrators like Kubernetes offer helpful abstractions to address the need for sensitive values with native [Secrets](https://kubernetes.io/docs/concepts/configuration/secret/). However, a Kubernetes `Secret` is a somewhat rudimentary object it lacks encryption by default and normally exists as a plain key/value within `etcd`. By contrast, services like [Vault](https://www.vaultproject.io/) or [Doppler](https://www.doppler.com/) are intentionally designed to provide strong guarantees and well-defined access controls. Can we combine the operational power of Kubernetes with the assurance of a fully managed secret provider?

![Doppler Kubernetes External Secrets Operator](https://cdn.hashnode.com/res/hashnode/image/upload/v1666646955334/WHLpfqRIr.png)

Yes! The [External Secrets Operator](https://github.com/external-secrets/external-secrets) is a Kubernetes [operator](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) that bridges the gap between Kubernetes' native secret support and _external_ systems that provide a canonical source of truth for secret storage. It does this by leveraging [custom resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) that define how to retrieve external secrets in order to manage the lifecycle of `Secret` resources in your Kubernetes cluster. In this tutorial, we'll use [Doppler](https://www.doppler.com/) as the external secret provider to illustrate using external secrets in a real-world application.

## Outcomes

The end goal of this guide will be to leverage well-managed secrets in an application deployed in Kubernetes. To achieve this, we'll use:

- The Kubernetes [External Secrets Operator](https://github.com/external-secrets/external-secrets)
- [Doppler](https://www.doppler.com/) to store and manage external secrets
- [minikube](https://minikube.sigs.k8s.io/docs/) or [kind](https://kind.sigs.k8s.io/) for a local Kubernetes API

At the conclusion of this tutorial, you'll understand how to store and manage secrets in Doppler, use those same secrets as native _Kubernetes_ secrets, and ultimately use them in a running application. Let's begin!

## Prerequisites

This article assumes that you'll follow along with your own Kubernetes cluster. You may choose to operate in an existing cluster (potentially within a sandboxed namespace), but if you'd like to use a sandbox instead, we suggest using either [minikube](https://minikube.sigs.k8s.io/docs/) or [kind](https://kind.sigs.k8s.io/) to create a local Kubernetes installation which will provide a safe environment for testing. In order to provide a clean and pristine environment, this tutorial will use [minikube](https://minikube.sigs.k8s.io/docs/) as the Kubernetes target.

In addition to a functional Kubernetes cluster, ensure that the following command-line tools are installed:

- `helm`, which we'll use to install the External Secrets Operator Kubernetes resources. Use the helm [quickstart guide](https://helm.sh/docs/intro/quickstart/) to install helm.
- The Doppler command line utility `doppler` using the Doppler [CLI Guide](https://docs.doppler.com/docs/cli) documentation.
  - You will need to [register for a Doppler account](https://dashboard.doppler.com/register) to create projects and store external secrets.
- `kubectl` to interact with Kubernetes. `kubectl` is available for a wide range of operating systems, and the [Kubernetes documentation](https://kubernetes.io/docs/tasks/tools/) provides comprehensive installation guides for `kubectl` and other tools as well.
- If you choose to use a local Kubernetes sandbox, install [minikube](https://minikube.sigs.k8s.io/docs/) and follow the instructions to create a local instance of Kubernetes. This can be as simple as `minikube start`, but consult the documentation if you need additional assistance.
  - `minikube` bundles a version-compatible installation of `kubectl` if you choose to use `minikube` for this exercise. It's a handy time saver.

Once installed, confirm that your environment is ready to go:

`kubectl version` should confirm that `kubectl` is present and communicating with the Kubernetes API successfully. If you aren't using `minikube`, simply use the plain `kubectl` command for the remainder of the tutorial.

```
$ minikube kubectl -- version
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.0", GitCommit:"a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2", GitTreeState:"clean", BuildDate:"2022-08-23T17:44:59Z", GoVersion:"go1.19", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.0", GitCommit:"a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2", GitTreeState:"clean", BuildDate:"2022-08-23T17:38:15Z", GoVersion:"go1.19", Compiler:"gc", Platform:"linux/amd64"}
```

The `helm` and `doppler` commands should be available:

```
$ helm version
version.BuildInfo{Version:"v3.9.0", GitCommit:"7ceeda6c585217a19a1131663d8cd1f7d641b2a7", GitTreeState:"", GoVersion:"go1.17.13"}
```

```
$ doppler --version
v3.44.0
```

With the prerequisites installed, let's proceed with building and running our program.

## Sample Application

Before we start creating secrets, let's begin by illustrating the external secrets workflow with a simple example service. The following guide will assume the use of `minikube` to build and load an application container image, so if your Kubernetes environment is different, you may need to adapt the instructions to work with your specific container registry strategy.

First, perform some initial steps to bootstrap a python [Flask](https://flask.palletsprojects.com/en/2.2.x/) application:

```
$ mkdir -p ~/tmp/secret-sauce
$ cd ~/tmp/secret-sauce
$ git init
$ echo flask > requirements.txt
$ python3 -m venv venv
$ ./venv/bin/pip install -r requirements.txt
```

Create a file named `app.py` that contains the following simple web application:

```
from flask import Flask
from os import environ

app = Flask( __name__ )

sauce = environ.get('SECRET_SAUCE')

@app.route("/")
def index():
    if sauce:
        return f"The secret sauce is: {sauce}!"
    else:
        return "You'll never find my secret sauce."
```

Define a small `Dockerfile` that defines how to build a container image for our application:

```
FROM python:3

WORKDIR /usr/src/app

COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD ["python", "-m" , "flask", "run", "--host=0.0.0.0"]
```

You can run this application locally with:

```
$ ./venv/bin/flask run
```

Try accessing the running application at [http://localhost:5000](http://localhost:5000) to see a response. The root route (`/`) will render a static string when no secret is present in the environment but will print the secret when the indicated environment variable is defined. Don't print secrets in production! This application is just a demonstration of how to read the value in a real program.

We're ready to load this application into Kubernetes. Assuming that you're following along with `minikube`, proceed to build the application container in the Kubernetes node under the name "`app`":

```
$ minikube image build -t app .
...lots of output...
Successfully tagged app:latest
```

Create a new Kubernetes `Deployment` file named `deployment.yaml`:

```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: flask
  template:
    metadata:
      labels:
        app: flask
    spec:
      containers:
        - name: webapp
          image: app
          imagePullPolicy: Never
          ports:
            - containerPort: 5000
```

Take note of a few assumptions in this `Deployment`:

- `imagePullPolicy` has been set to `Never` because we're running a locally-built image. By default, Kubernetes would attempt to `pull` the `app` image, which doesn't exist. As previously mentioned, if you're following this tutorial in an environment other than `minikube`, you may need to push the container image to a registry and adjust these settings slightly.
- We've exposed port `:5000` which we can access later.

Load this `Deployment` into your running cluster:

```
$ kubectl apply -f deployment.yaml
deployment.apps/app created
```

Finally, forward the port in another terminal window in order to access the running application in a simple tunnel.

```
$ kubectl port-forward deployment/app 5000:5000
Forwarding from 127.0.0.1:5000 -> 5000
Forwarding from [::1]:5000 -> 5000
```

Send a request to your Kubernetes application to see it in action:

```
$ curl http://localhost:5000
You'll never find my secret sauce.
```

Fantastic! Note that we've received the response that indicates no secret value has been injected into the environment. How can we add a secret to the application and see the _secret sauce_?

## Doppler

By using [Doppler](https://www.doppler.com/), we can achieve a great deal of control over application secrets and manage them at each step of our application's lifecycle, from in our local development environment to within a Kubernetes workload.

If you haven't signed up with Doppler, you can [do so now](https://dashboard.doppler.com/register). Once you have an account, proceed to use the `login` command to set up your account locally:

```
$ doppler login
```

Setup Doppler for the demo application by defining a `doppler-template.yaml` file in the root of the application directory. This YAML file defines a template that we can import to create a new Doppler project easily from the command line:

```
projects:
  - name: secret-sauce
    description: Kubernetes demo app
    environments:
      - slug: dev
        name: Development
        configs:
          - slug: dev
      - slug: stg
        name: Staging
        configs:
          - slug: stg
      - slug: prd
        name: Production
        configs:
          - slug: prd
    secrets:
      dev:
        SECRET_SAUCE: tartar
      stg:
        SECRET_SAUCE: horseradish
      prd:
        SECRET_SAUCE: tzatziki
```

Enter the directory in your shell and run the following `doppler` command in order to bootstrap your Doppler project. This will create a new project called `secret-sauce`, set up a few different environments, and load an initial secret value for `SECRET_SAUCE`:

```
$ doppler import
```

You'll see output similar to the following:

```
┌──────────────┬──────────────┬─────────────────────┬──────────────────────────┐
│ ID           │ NAME         │ DESCRIPTION         │ CREATED AT               │
├──────────────┼──────────────┼─────────────────────┼──────────────────────────┤
│ secret-sauce │ secret-sauce │ Kubernetes demo app │ 2022-10-11T22:10:36.567Z │
└──────────────┴──────────────┴─────────────────────┴──────────────────────────┘
```

View the secrets for this project with `doppler secrets`:

```
$ doppler secrets
```

In addition to some default variables that begin with `DOPPLER_`, you'll also find the secret value we'd like to inject, `SECRET_SAUCE`:

```
┌─────────────────────┬──────────────┐
│ NAME                │ VALUE        │
├─────────────────────┼──────────────┤
│ DOPPLER_CONFIG      │ dev          │
│ DOPPLER_ENVIRONMENT │ dev          │
│ DOPPLER_PROJECT     │ secret-sauce │
│ SECRET_SAUCE        │ tartar       │
└─────────────────────┴──────────────┘   
```

Run a quick test to confirm that our application behaves as expected when a secret is present in its environment variables. To do this, we can use the `doppler run` command to easily inject the variable into our application.

```
$ doppler run -- ./venv/bin/flask run
```

Then issue a request to the listening port to see whether the response has changed:

```
$ curl http://localhost:5000
The secret sauce is: tartar!
```

Great! Let's continue on to install the External Secrets Operator.

## External Secrets

The [External Secrets Operator](https://github.com/external-secrets/external-secrets) provides the translation layer between Kubernetes' native secrets and external secrets. The operator leverages [custom resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) in order to model external secrets, which it then retrieves as necessary and translates into native Kubernetes secrets that your workload can easily consume.

The operator is provided as a Helm chart, so first add the upstream `external-secrets` Helm repository to access the chart:

```
$ helm repo add external-secrets https://charts.external-secrets.io
external-secrets has been added to your repositories
```

Next, install the chart into your Kubernetes cluster. The following command:

- Instructs `helm` to use the `external-secrets` chart,
- passes `-n` to install the chart into the `external-secrets` namespace,
- creates the namespace as necessary with `--create-namespace`, and
- configures the requisite CRDs as well (`--set installCRDs=true`)

```
$ helm install external-secrets \
     external-secrets/external-secrets \
     -n external-secrets \
     --create-namespace \
     --set installCRDs=true
```

You should see output similar to the following:

```
NAME: external-secrets
LAST DEPLOYED: Tue Oct 11 12:27:18 2022
NAMESPACE: external-secrets
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
external-secrets has been deployed successfully!

In order to begin using ExternalSecrets, you will need to set up a SecretStore
or ClusterSecretStore resource (for example, by creating a 'vault' SecretStore).

More information on the different types of SecretStores and how to configure them
can be found in our Github: https://github.com/external-secrets/external-secrets
```

With this operator installed, we're now ready to set up Doppler to serve as the source for external secrets.

## Integrating Doppler with Kubernetes

The first step is to create a [service token](//docs.doppler.com/docs/service-tokens) which serves as the authentication mechanism for the external secrets operator against Doppler. It should be stored inside of a generic Kubernetes secret that the operator will consume.

You can generate and store the token in a single step with some fancy footwork in the shell to avoid copying and pasting the token around. The following command creates a new Doppler token and then immediately loads it into Kubernetes:

```
$ kubectl create secret generic \
    doppler-token-auth-api \
    --from-literal dopplerToken=$(doppler configs tokens create --config prd doppler-auth-token --plain)
secret/doppler-token-auth-api created
```

Note that we're passing the `--config prd` flag to `doppler` in order to create a token scoped to the `prd` (production) configuration of our Doppler project. In our application directory, we defaulted to `dev`, which has its own secrets. With this technique, we're only interacting with development secrets locally while loading production secrets into our container runtime, which keeps the risk of exposure for production secrets minimal.

Next, create a `SecretStore` CRD that points the operator at your Doppler service token secret. This step sets up Doppler as a source for external secrets that we can call upon later.

```
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: doppler-auth-api
spec:
  provider:
    doppler:
      auth:
        secretRef:
          dopplerToken:
            name: doppler-token-auth-api
            key: dopplerToken
```

Create the `SecretStore` in Kubernetes:

```
$ kubectl apply -f secretstore.yaml
secretstore.external-secrets.io/doppler-auth-api created
```

We're ready to inject our secret sauce! A new `ExternalSecret` is the necessary resource that enables us to synchronize one of our _Doppler_ secrets to a related generic _Kubernetes_ secret. Create a new file called `externalsecret.yaml` with the following YAML:

```
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: secret-sauce
spec:
  secretStoreRef:
    kind: SecretStore
    name: doppler-auth-api

  target:
    name: secret-sauce

  data:
    - secretKey: SECRET_SAUCE
      remoteRef:
        key: SECRET_SAUCE
```

Load it into Kubernetes:

```
$ kubectl apply -f externalsecret.yaml
externalsecret.external-secrets.io/secret-sauce created
```

You can confirm that the secret was loaded by the operator by listing it with `kubectl`:

```
$ kubectl get secret secret-sauce
NAME TYPE DATA AGE
secret-sauce Opaque 1 3s
```

Success! Note that if you do not see the new secret, you can also inspect the External Secret Operator's logs to debug any issues:

```
$ kubectl -n external-secrets logs -lapp.kubernetes.io/name=external-secrets
```

Generic secrets can be seen in the Kubernetes dashboard as well. If you're using `minikube`, you can quickly install and view the Kubernetes dashboard with the following command:

```
$ minikube dashboard
```

Your browser will open to the dashboard landing page. You may use the left sidebar to navigate to the "Secrets" link to view your new secret:

![Kubernetes Dashboard](https://cdn.hashnode.com/res/hashnode/image/upload/v1666642558970/mVLa_og-O.png)

The only task left to do is to consume the secret from our application. Here's an updated `Deployment` that references the newly-created `Secret`. Note the new `env` key, which injects the variable `SECRET_SAUCE` drawn from a secret named `secret-sauce` under the key `SECRET_SAUCE`:

```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: flask
  template:
    metadata:
      labels:
        app: flask
    spec:
      containers:
        - name: webapp
          image: app
          imagePullPolicy: Never
          env:
            - name: SECRET_SAUCE
              valueFrom:
                secretKeyRef:
                  name: secret-sauce
                  key: SECRET_SAUCE
          ports:
            - containerPort: 5000
```

Apply the updated `Deployment`:

```
$ kubectl apply -f deployment.yaml
deployment.apps/app configured
```

The changed `Deployment` manifest will recreate any necessary pods, so invoke a new forwarded port once the pods have restarted to forward requests to the new containers:

```
$ kubectl port-forward deployment/app 5000:5000
```

Finally, Issue an HTTP request to the running container to see the injected Doppler secret in action:

```
$ curl http://localhost:5000
The secret sauce is: tzatziki!
```

Note that the content of this secret differs from the one we saw in our local development environment. The `--config prd` flag to the `doppler` command loaded a token with rights to the `prd` configuration in Doppler, which injects the correct secret for the configuration that the Doppler token has been scoped to.

Congratulations! You've successfully:

- Installed the External Secrets Operator into your Kubernetes cluster
- Managed an application secret with Doppler
- Connected Doppler with the External Secrets Operator
- Used a Doppler project secret within a Kubernetes Deployment

## What's Next?

There's even more you can do with the External Secrets Operator and Doppler, so if you'd like to learn more, dive into the [documentation](https://external-secrets.io/latest/provider/doppler/) to learn how to use features like JSON processing, filtering, and more.

If you're actively using Kubernetes secrets stored in `etcd` today, you should also [configure encryption at rest](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/) so that your secrets are secure no matter where they're stored. Whether you load them with `kubectl` or the External Secrets Operator, configuring encryption at rest is still an important step to address every step of your secret management lifecycle.