Tutorial: How to use Kubernetes Secrets for storing sensitive config data
Tags: kubernetes, tutorial, showdev, docker
Published: 2019-12-18 13:28:28

Hello and welcome 👋👋 We continue the "Kubernetes in a Nutshell" journey! In one of the previous blogs, we saw [how to configure Kubernetes apps using the `ConfigMap` object](https://dev.to/itnext/hands-on-guide-configure-your-kubernetes-apps-using-the-configmap-object-44gj). In this post, we will explore Kubernetes `Secrets` and how they can be used to store sensitive configuration data which needs to be handled securely e.g. database credentials, API keys etc.

As usual, this is going to be example driven and you will learn about:

- How to create `Secrets` (CLI, yaml etc.), and
- Various ways of using them in your apps (env variables, volumes, etc.)

The code (and `YAML`!) is available [on GitHub](https://github.com/abhirockzz/kubernetes-in-a-nutshell/blob/master/secrets)

> Happy to get your feedback via [Twitter](https://twitter.com/abhi_tweeter) or just drop a comment 🙏🏻

This blog has been divided into two logical sections:

1. Ways to *create* `Secrets`
2. Techniques to *use* `Secrets` in your applications

![](https://media.giphy.com/media/3orieOzVS1ROeHqpeo/giphy.gif)

## Pre-requisites:

To go through the examples in this post, all you need is a `minikube` cluster and `kubectl` CLI tool to access the cluster.

Install [`minikube`](https://kubernetes.io/docs/tasks/tools/install-minikube/) as a single-node Kubernetes cluster in a virtual machine on your computer. On a Mac, you can simply:

```
curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-darwin-amd64 \
  && chmod +x minikube

sudo mv minikube /usr/local/bin
```

Install [`kubectl`](https://kubernetes.io/docs/tasks/tools/install-kubectl/) to interact with the `minikube` cluster. On a Mac, you can:

```
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/darwin/amd64/kubectl
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl
```

## Creating Secrets

Let’s look at techniques using which you can create a `Secret`

### Using a manifest file

**Use the `data` section**

It’s possible to create a `Secret` along with the configuration data stored as key-value pairs in the `data` section of the definition.

```
apiVersion: v1
kind: Secret
metadata:
  name: service-apikey
data:
  apikey: Zm9vYmFy
```

The `Secret` contains key-value data representing the sensitive info, with `apikey` being the key and value is a `base64` encoded string

To create this `Secret` in Kubernetes:

```
kubectl apply -f https://raw.githubusercontent.com/abhirockzz/kubernetes-in-a-nutshell/master/secrets/secret-data.yaml
```
> To keep things simple, the YAML file is being referenced directly from the [GitHub repo](https://github.com/abhirockzz/kubernetes-in-a-nutshell), but you can also download the file to your local machine and use it in the same way.

To confirm that the `Secret` has been created:

```
kubectl get secret/service-apikey -o yaml
```

You will get a (YAML) response similar to:

```
apiVersion: v1
data:
  apikey: Zm9vYmFy
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"apikey":"Zm9vYmFy"},"kind":"Secret","metadata":{"annotations":{},"name":"service-apikey","namespace":"default"}}
  creationTimestamp: "2019-12-17T11:11:27Z"
  name: service-apikey
  namespace: default
  resourceVersion: "113009"
  selfLink: /api/v1/namespaces/default/secrets/service-apikey
  uid: 671b547c-3316-4916-b6dc-be2b551b974e
type: Opaque
```
> Fetching the `Secret` details using `kubectl get` does not disclose its contents

Notice that `apikey: Zm9vYmFy` was what we had provided in the YAML manifest. You can check the plain text form by decoding it:

```
echo 'Zm9vYmFy' | base64 --decode

//foobar
```

**Use the `stringData` section**

The  `data` attribute used in the above example is used to save `base64` encoded information. If you want to store plaintext data securely, you can use `stringData` section. Here is an example:

```
apiVersion: v1
kind: Secret
metadata:
  name: plaintext-secret
stringData:
  foo: bar
  mac: cheese
```

The values for `foo` and `mac` are being passed as plain text. Create this `Secret` and confirm:

```
kubectl apply -f https://raw.githubusercontent.com/abhirockzz/kubernetes-in-a-nutshell/master/secrets/secret-plaintext.yaml

kubectl get secret/plaintext-secret -o yaml
```

Here is the `data` portion of the YAML response. The actual data is stored in `base64` endcoded format

```
data:
  foo: YmFy
```

If you decode the data, you can confirm that it matches the original plain text input (`bar`) we had provided

```
echo 'YmFy' | base64 --decode

//bar
```

> Note that `data` section does *not* accept plain text attribute. Trying to do so will result in error similar to this: `illegal base64 data at input byte 8`

**File contents**

You can provide the contents of an entire file as input to the `stringData` section as well! Here is what this might look like:

```
apiVersion: v1
kind: Secret
metadata:
  name: secret-in-a-file
stringData:
  app-config.yaml: |-
    hello: world
    john: doe
```

Create this `Secret`:

```
kubectl apply -f https://raw.githubusercontent.com/abhirockzz/kubernetes-in-a-nutshell/master/secrets/secret-file.yaml
```

This resulting `Secret` will contain a key named `app-config.yaml` and its contents (value) will be the `base64` encoding of the provided data

As usual, you confirm this in Kubernetes as well as decode the contents

```
kubectl get secret/secret-in-a-file -o yaml
echo '<"data" content in yaml response> | base64 --decode
```

> Note: when you use this technique, your application is responsible for parsing out the data which represents the `Secret` configuration. In this case, it happens to be newline-separated key-value pairs, but it could be anything else

### Using `kubectl`

You can use the `kubectl create secret` command to create Secret objects

**Using `--from-literal`**

You can use plain text data to create `Secret` using the CLI (this will be stored in `base64` encoded format in Kubernetes)

```
kubectl create secret generic redis-credentials --from-literal=user=poweruser --from-literal=password='f0ob@r'
```

**Using `--from-file`**

```
kubectl create secret generic topsecret --from-file=api_keys.txt
```
This will create a Secret (`topsecret`) with

- a key with the same name of the file i.e. `api_keys.txt` in this case
- and, value as the contents of the file

**From files in a directory**

You can simply point to a directory and all the files within will be used to create the `Secret`

```
kubectl create secret generic topsecrets --from-file=/home/credentials/
```

You will end up with

- multiple keys which will the same as the individual file name
- the value will be the contents of the respective file

## Using Secrets

For Secrets to be useful, we need to ensure that they are available to our applications i.e `Pod`s. Let's explore the ways in which we can do this

### Environment variables

You can consume the `Secret` data as environment variables in a `Pod` (just like `ConfigMap`). Here is an example:

```
apiVersion: v1
kind: Pod
metadata:
  name: pod1
spec:
  containers:
    - name: nginx
      image: nginx
      env:
        - name: API_KEY
          valueFrom:
            secretKeyRef:
              name: service-apikey
              key: apikey
```

We use the key `apikey` from `Secret` `service-apikey` and make sure its value is available as an environment variable `API_KEY` inside the `Pod`.

Create the `Pod` (assuming you have the Secret created from the example before) and confirm

```
kubectl apply -f https://raw.githubusercontent.com/abhirockzz/kubernetes-in-a-nutshell/master/secrets/pod-secret-env.yaml
kubectl get pods -w
```

Wait for the `Pod` to transition to `Running` state. Then, confirm that the environment variable has been injected into the `Pod`

```
kubectl exec pod1 -- env | grep API_KEY
```

You should get this response - `API_KEY=foobar`

Instead of referring to individual entries in a `Secret`, you can use `envFrom` to conveniently use all entries as environment variables in a `Pod`. This is how you might use this:

```
apiVersion: v1
kind: Pod
metadata:
  name: pod2
spec:
  containers:
    - name: nginx
      image: nginx
      envFrom:
        - secretRef:
            name: plaintext-secret
```

We are referring to the `plaintext-secret` `Secret` using `envFrom.secretRef`. To create this `Pod`

```
kubectl apply -f https://raw.githubusercontent.com/abhirockzz/kubernetes-in-a-nutshell/master/secrets/pod-secret-envFrom.yaml
kubectl get pods -w
```

Wait for the `Pod` to transition to `Running` state and then confirm the presence of environment variables

```
kubectl exec pod2 -- env | grep foo
//foo=bar

kubectl exec pod2 -- env | grep mac
//mac=cheese
```

This confirms that both `foo` and `mac` were added as environment variables into `the` Pod along with their decoded values i.e. `bar` and `cheese` respectively

**Volumes**

You can mount `Secrets` as `Volume` within a `Pod`. For e.g.

```
apiVersion: v1
kind: Pod
metadata:
  name: pod3
spec:
  containers:
    - name: nginx
      image: nginx
      volumeMounts:
        - name: apikey-config-volume
          mountPath: /secret
          readOnly: true
  volumes:
    - name: apikey-config-volume
      secret:
        secretName: service-apikey
```

The `apikey-config-volume` volume refers to the `service-apikey` `Secret`. To create this `Pod`:

```
kubectl apply -f https://raw.githubusercontent.com/abhirockzz/kubernetes-in-a-nutshell/master/secrets/pod-secret-volume.yaml
kubectl get pods -w
```

Wait for the `Pod` to transition to `Running` state. Then execute the following command:

```
kubectl exec pod3 -- cat /secret/apikey
//foobar
```

This confirms that the key `apikey` in secret `service-apikey` was mounted as a file (with name `apikey`) in the `/secret` directory (as specific in Pod). The content of the file is nothing but the secret value i.e. `foobar` in this case

**Using `imagePullSecrets`**

There is a way to use `Secrets` such that your application Pod can use it to authenticate and pull Docker images from private Docker registries.

There are actually three types of `Secrets`

- `generic` - used to store key-value pairs, as we have seen in the examples so far
- `tls` - store public.private key pair info as `Secrets`
- `docker-registry` - credentials for authenticating to a Docker registry.

The way this technique is used is very simple:

- Use `docker-registry` `Secret` type to store private Docker registry credentials in Kubernetes
- And then, `imagePullSecrets` (in a Pod) to reference the `Secret` containing the Docker registry credentials

An example always helps:

```
apiVersion: v1
kind: Pod
metadata:
  name: pod4
spec:
  containers:
    - name: privateapp
      image: abhirockzz/test-private-repo:latest
      command: ["/bin/sh"]
      args: ["-c", "while true; do date; sleep 5;done"]
  imagePullSecrets:
    - name: docker-repo-secret
```

See how `imagePullSecrets.name` refers to a `Secret` called `docker-repo-secret`. Let's create it

> But, before that please ensure that you have a private Docker registry - I used [`DockerHub`](https://hub.docker.com/), but you can choose any other

Start by creating a `Secret` (with the name `docker-repo-secret`) which contain your Docker credentials using `kubectl create secret docker-registry` command

```
kubectl create secret docker-registry docker-repo-secret --docker-server=DOCKER_REG_SERVER --docker-username=DOCKER_REG_USERNAME --docker-password=DOCKER_REG_PASSWORD --docker-email=DOCKER_REG_EMAIL
```
For Docker Hub e.g.

```
kubectl create secret docker-registry docker-repo-secret --docker-server=https://index.docker.io/v1/ --docker-username=foobarbaz --docker-password=t0ps3cr3t --docker-email=foobarbaz@gmail.com

kubectl get secret/docker-repo-secret -o yaml
```
> `https://index.docker.io/v1/` is the Docker Hub registry server

To test things out, we will use a [`busybox`](https://hub.docker.com/_/busybox) image and `tag` it

```
docker pull busybox

docker tag busybox [DOCKER_REG]/[DOCKER_PRIVATE_REPO]:[IMAGE_TAG]
e.g. 
docker tag busybox abhirockzz/test-private-repo:latest
```

... and `push` it

```
docker push [DOCKER_REG]/[DOCKER_PRIVATE_REPO]:[IMAGE_TAG]
e.g. 
docker push abhirockzz/test-private-repo:latest
```

Once the private repo is ready, you can create the `Pod` which will pull the image from the private repo using the registry credentials supplied to it via the `Secret`

```
kubectl apply -f https://raw.githubusercontent.com/abhirockzz/kubernetes-in-a-nutshell/master/secrets/pod-secret-docker.yaml
kubectl get pods -w
```

Wait for the `Pod` to move to `Running` status. 

> If you see an `ErrImagePull` error, it indicates that there might be problem authenticating to the Docker registry. To get details use: `kubectl describe pod/pod4`

To confirm that the pod is working fine: `kubectl logs -f pod4`

Since the `busybox` image does not really do anything by itself, we execute: `while true; do date; sleep 5;done` (as provided in the `Pod` spec). As a result, you should see the logs (printed every 5 secs)

```
Tue Dec 17 14:17:34 UTC 2019
Tue Dec 17 14:17:39 UTC 2019
Tue Dec 17 14:17:44 UTC 2019
Tue Dec 17 14:18:49 UTC 2019
```

All good! What that means is that the Pod was able to pull down your image from a private Docker repo using the Docker credentials which was injected into the `Pod` using `imagePullSecrets` which itself referenced a `Secret`

## Good to know

Here is a (non-exhaustive) list of things which you should bear in mind when using `Secrets`:

- `Secret` has to be created before any `Pod` that wants to use it.
- `Secrets` are applicable within a `namespace` i.e. they can only be used by `Pods` in the same `namespace`
- The `Pod` will not start if there is a reference to a non existent key in a `Secret` (using `secretKeyRef`)
- 1MiB is the size limit for individual `Secrets`

That's it for this edition of the "Kubernetes in a Nutshell" series. Stay tuned for more!

If you are interested in learning Kubernetes and Containers using [Azure](https://azure.microsoft.com/services/kubernetes-service/?WT.mc_id=devto-blog-abhishgu), simply [create a **free** account](https://azure.microsoft.com/en-us/free/?WT.mc_id=devto-blog-abhishgu) and get going! A good starting point is to use the [quickstarts, tutorials and code samples](https://docs.microsoft.com/azure/aks/?WT.mc_id=devto-blog-abhishgu) in the documentation to familiarize yourself with the service. I also highly recommend checking out the [50 days Kubernetes Learning Path](https://azure.microsoft.com/resources/kubernetes-learning-path/?WT.mc_id=devto-blog-abhishgu). Advanced users might want to refer to [Kubernetes best practices](https://docs.microsoft.com/azure/aks/best-practices?WT.mc_id=devto-blog-abhishgu) or watch some of the [videos](https://azure.microsoft.com/resources/videos/index/?services=kubernetes-service&WT.mc_id=devto-blog-abhishgu) for demos, top features and technical sessions.

I really hope you enjoyed and learned something from this article 🙌 Please like and follow if you did!