How to manage Kubernetes secrets securely in Git
Tags: kubernetes, git
Published: 2020-10-21 14:02:23

Kubernetes has a declarative approach in managing resources. 

While it's a common practice to keep the Kubernetes manifest files in a Git repository, storing and managing Kubernetes secrets has always been a challenge. [Sealed Secrets](https://github.com/bitnami-labs/sealed-secrets) was developed to address this problem.

It's a Kubernetes operator which allows you to store secrets in Git. It uses asymmetric key encryption to encrypt the secrets, so that only the operator in the Kubernetes cluster can decrypt them. Therefore, the encrypted SealedSecrets are safe to store in a Git repository.

In this article, we will install the Sealed Secrets operator and demonstrate how to use it.

Sealed Secrets is composed of two parts:

-   The Sealed Secrets operator which runs in the Kubernetes cluster
-   The `kubeseal` client-side command line tool, which is used to encrypt the secrets and generates the SealedSecrets Kubernetes resources.

The original article is [available here](https://tomcode.com/blog/how-to-manage-kubernetes-secrets-securely-in-git).

#### Prerequisites

`kubectl` connected to a Kubernetes cluster

#### Install kubeseal

`kubeseal` is a command line tool to encrypt secrets and generate SealedSecrets. On macOS you can install `kubeseal` with Homebrew.

```
brew install kubeseal
```

You can find installation instructions for other platforms here: <https://github.com/bitnami-labs/sealed-secrets#installation>

#### Create a Kubernetes namespace for Sealed Secrets

Let's start by creating a namespace for the Sealed Secrets operator.

```
kubectl create namespace sealed-secrets
```

#### Deploy the Sealed Secret operator

Download the Kubernetes manifests for the Sealed Secret operator.

```
curl -L https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.12.4/controller.yaml > sealed-secrets.yaml
```

As we do not want the operator to run in the kube-system namespace, we use `sed` to change it to sealed-secrets.

```
sed -i -e 's/kube-system/sealed-secrets/g' sealed-secrets.yaml
```

Now we use `kubectl` to deploy the Sealed Secret operator.

```
kubectl apply -f sealed-secrets.yaml
service/sealed-secrets-controller created
rolebinding.rbac.authorization.k8s.io/sealed-secrets-service-proxier created
rolebinding.rbac.authorization.k8s.io/sealed-secrets-controller created
clusterrole.rbac.authorization.k8s.io/secrets-unsealer created
serviceaccount/sealed-secrets-controller created
customresourcedefinition.apiextensions.k8s.io/sealedsecrets.bitnami.com created
role.rbac.authorization.k8s.io/sealed-secrets-service-proxier created
role.rbac.authorization.k8s.io/sealed-secrets-key-admin created
clusterrolebinding.rbac.authorization.k8s.io/sealed-secrets-controller created
deployment.apps/sealed-secrets-controller created
```

When the Sealed Secret operator starts, it will generate a public/private RSA key pair if it does not exist yet. Afterwards it will print the public key which can be used for encryption / SealedSecret creation.

#### Verify deployment

Verify the deployment by checking the status Kubernetes pods in the sealed-secrets namespace.

```
kubectl get pods --namespace sealed-secrets

NAME                                        READY   STATUS    RESTARTS   AGE
sealed-secrets-controller-6f5785b7f-m9hmq   1/1     Running   0          20m
```

#### Create the demo Kubernetes namespace

```
kubectl create namespace demo

namespace/demo created
```

#### Create a Kubernetes secret

Use `kubectl` to create a local secret manifest file.

We need to pass the following options:

-   `namespace=demo` -- Specify the namespace where the secret should live
-   `from-literal` -- Specify the secret key=value pairs
-   `output=yaml` -- Specify the output format of kubectl
-   `dry-run=client` -- Only print the secret without actually creating it / sending it to the Kubernetes API server.
-   `> db-secret.yaml` -- This tells kubectl to store it's output in db-secret.yaml

```
kubectl create secret generic db-secret
--namespace=demo
--from-literal=username=admin
--from-literal=password=Y4nys7f11
--dry-run=client
--output=yaml > db-secret.yaml

cat db-secret.yaml

apiVersion: v1
data:
  password: WTRueXM3ZjEx
  username: YWRtaW4=
kind: Secret
metadata:
  creationTimestamp: null
  name: db-secret
  namespace: demo
```

As you see above, the normal Kubernetes resource contains the the `kev=value` pair, where the value is just base64 encoded and not encrypted.

#### Encrypt Kubernetes secret

Use `kubeseal` to encrypt the `db-secret`. The output is a SealedSecret resource which contains the encrypted secret.

We need to pass the following options:

-   `controller-namespace=sealed-secret` -- Specify the namespace where the Sealed Secret operator runs (default is kube-system)
-   `format=yaml` -- Specify the output format of kubeseal (default is json)
-   `< db-secret.yaml` -- This tells kubeseal to take the db-secret.yaml as input
-   `> sealed-db-secret.yaml` -- This tells kubeseal to store it's output in sealed-db-secret.yaml

```
kubeseal
--controller-namespace=sealed-secrets
--format=yaml < db-secret.yaml > sealed-db-secret.yaml

cat sealed-db-secret.yaml

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: db-secret
  namespace: demo
spec:
  encryptedData:
    password: AgDj/I4L89v9km3cuYyckAe1/sNcjH67fjhPAjZep37bQ+CCxPvQzEXJrQvI0fzzjF0cHU003JYCFUok4DFMi8nMw9YnXM//JiyBGvlEhmmQ/VI+HgruWcOEIuWw+NfR6GWVn0dLidnYOwnvJG4sAx3iPXcR2oyivj2v5yICx4CgAOS88tZF1AaSxYqhbAYfnD8IiNdfH6+gwYXwLp2pS1f/HLawtmnvAZLVb9JqP1Nidvj34a79xVbGZ7wz04W55uU/gNNZkP5QIvJ49DZkx4JqmKLISrXziPkgt/ZqB+346oXEM+iKoPOgn0sX8M6NcY8hd7SiT9RJ3fQ4TDGOfufn8fTXVtleSN3tOicMpbMhZ7XP/0x1bvN72yYbMPAbZxf+G6Qs6CwlG0ftU/FrD3W5e38ypg5E9Uk7GW0o2A74IOrpmUswDDeR1j+0TGIlOYcwAuQREy1aDcgvm+9Cn1KFp0Fd+q0DxnGf2V31a22DoE1Bxj+R4Jq5bLzI0ZBt/VYO65b1MnXetz3cy3T8gKTFzeflorPU5GBIwKs7AeDUgvXpefp/XfW70afBKhk1la6LstBUjVP6TPI7yKpVWq0FgR0MjeyZFp4eEezDoQJOX/siy5/weWv7sOfifpDp2lxSqPcTd+c1wa512tb6KSapu0vvb7JL9VTyNWdw6Th2ymD2RWtOV8dRXTLXkzd9Q34/8Y0OR4n/cSY=
    username: AgB+7E5iWzUELYgjxwwwnAk1SlR+ZwQV7OafBsZIPA47KgSlp8Xys7slJ2STIQ4nfmg/UTJimTN8mNvF3QK6zEfAeglN75r0HxMe8aWPnJ4OrKwuy8LUpEIKtdlPHLnyHNF2v5Nos/Gej1cshuA27EliqCwRLE7Z4FCxwiUIX0zBraWOJcXXzVylRK3WA2uGrBsH9P9k7+V1h/B66K4djCL47uOLaBQTXRlW45av1cSsDWLpq1em/6Og+GA5txxsNaCDADR2ybqpPg6RRoC1ayf7z+cYrFV84Wc9KNb3+cxHjuEEW/c3PT0xqyqZq7Iqr6254SVTv1bB0joD7wK7yEPUFJSc8M3GnXsG3Eq81QgpjeKHY1uZnT4B5/ztxSeczQoKksjvICUeysb09tZMRgF80A74uAv4e0gyRXBEgk7uPQg/JTqKFo77qntK5lzAGSVJQlumAq8FBeAKwwzOJrltbC8yMprQzapXBj8Vo43gEY1Avkc2Pov/bceppuah2Q9JFPCetpEtI+lruRGlZUR9Iw3T4QcunLXfgC73xWiPY6B/I+dFh2oDoiabo4gonGDLEhtAmtQU6MARWVBbs6AFGHQYRdvU23Xc5ISocG69mwFwevtwkOHQtx+VOcMN1Tmnj5htCb1CaK+rD2xfHjaWZ3VjorWuUCG5kXz5pjeRw95s+h8ru2emRqokj1jUf6xqc03QIg==
  template:
    metadata:
      creationTimestamp: null
      name: db-secret
      namespace: demo
```

You can now add this SealedSecret resource in Git. As you see it contains the secret, but the values are encrypted. Only the Sealed Secret operator in the cluster has the private key to decrypt the information.

#### Apply the SealedSecret resource

Use `kubectl` to apply the SealedSecret resource.

```
kubectl apply -f sealed-db-secret.yaml

sealedsecret.bitnami.com/db-secret created
```

Once you apply the SealedSecret, the Sealed Secret operator gets triggered, which is using its private key to decrypt the information and is generating the corresponding Kubernetes secret resource.

#### Verify deployment

Verify the deployment by checking the SealedSecret and Kubernetes secrets in the `demo` namespace.

```
kubectl get sealedsecrets,secrets -n demo

NAME                                 AGE
sealedsecret.bitnami.com/db-secret   55s
```

```
kubectl get secrets db-secret -n demo -o yaml

NAME                         TYPE                                  DATA   AGE
secret/db-secret             Opaque                                2      55s

apiVersion: v1
data:
  password: WTRueXM3ZjEx
  username: cHJvZHVzZXI=
kind: Secret
metadata:
  creationTimestamp: "2020-07-02T09:39:15Z"
  name: db-secret
  namespace: demo
  ownerReferences:
  - apiVersion: bitnami.com/v1alpha1
    controller: true
    kind: SealedSecret
    name: db-secret
    uid: 77d1231f-921c-4335-823f-643ec82fdb78
  resourceVersion: "55133137"
  selfLink: /api/v1/namespaces/demo/secrets/db-secret
  uid: 2df6c273-ca03-4523-b45d-f9849b39b92e
type: Opaque
```

#### Summary

Sealed Secret was designed to easily fit into automated workflows such as GitOps. Once a Kubernetes secret is converted into a SealedSecret, only the operator in the cluster can decrypt the original secret.

With SealedSecret, the creation and update of Kubernetes secrets does not require high privileges, which makes it easy and secure.