Store your Kubernetes Secrets in Git thanks to Kubeseal. Hello SealedSecret!
Tags: kubernetes, devops, security, git
Published: 2021-02-23 07:54:43

---
title: Store your Kubernetes Secrets in Git thanks to Kubeseal. Hello SealedSecret!
published: true
description: Encrypt your Kubernetes Secrets with Kubeseal for GitOps
tags: Kubernetes, DevOps, Security, Git
cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/aj0a86fwlpz7c2cykmnx.png
---

In the Kubernetes world when we want to handle sensitive data  the usage is to store it in a Secret in a Kubernetes cluster. Cool, but when we want to save our Secrets in our source code (Git) repositories, a security problem appears.
We will see in this article, a solution that can helps us ;-).

# Kubernetes Secrets

Secrets in Kubernetes are used to store sensitive data, like password, keys, certificates and token. Secrets are encoded in base64 and automatically decoded when they are attached and read by a Pod.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/u8jwcjs3ep89700522ha.png)

Cool, so what's the problem?

/!\ Be careful:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vav82m8ldui4n6p0gh30.png)

A secret in Kubernetes cluster is encoded in base64 but not encrypted!

Theses data are "only" encoded so if a user have access to your secrets, he can simply base64 decode to see your sensitive data (`kubectl get secret my-secret -o jsonpath="{.data.password}" | base64 -D` for Mac and `--decode` instead of `-D` for Linux platform).

And suddenly, since the secrets aren't encrypted, it can be unsecure to commit them to your Git repository :-(.

In the DevOps world, we can build, test, package and deploy our applications and infrastructure through a CI (Continuous Integration) & CD (Continuous Delivery) pipeline but we can also handle this through GitOps (a way of managing all our configurations and infrastructure through code in Git repositories).

So if we want to store in Git repositories our Kubernetes manifests, a problem appears. And as usual, several solutions exists , and today in this article we will see one of them: kubeseal.

# Encrypt/Seal your Secrets with Kubeseal

Secrets are cool but what about encrypted secrets with kubeseal?
The goal of [kubeseal](https://github.com/bitnami-labs/sealed-secrets), a bitnami tool, is to encrypt your Kubernetes Secret into a SealedSecret.
Thanks to that, it will be safe to store in public Git repository.
The SealedSecret can be decrypted *only* by the controller running in the target cluster and nobody else.

## Overview

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/aj0a86fwlpz7c2cykmnx.png)

As you can see in the schema, a *sealed-secret-controller* run in the Kubernetes cluster. He listens when a new SealedSecret object appears, unsealed it (thanks to known certificates) and create a Kubernetes secret in the same namespace as the SealedSecret.

Be careful, if you delete the SealedSecret in your cluster, the linked Secret will be deleted too.

# Usage - Step by step

After the theory, it's time to see all the steps.
Let's split the steps in two parts: server and client part.

## Server part

We will:
* deploy the sealed-secrets-controller in the cluster
* retrieve generated certificate keypair

### 1. Install SealedSecret CRD, server-side controller into *kube-system* namespace

```
$ kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.15.0/controller.yaml
```

NOTE: If you can't (or don't want) to use the kube-system namespace, please consider [this approach](https://github.com/bitnami-labs/sealed-secrets#kustomize).

NOTE: if you want to install it on a GKE cluster for which your user account doesn't have admin rights, please [read this guide](https://github.com/bitnami-labs/sealed-secrets/blob/master/docs/GKE.md).

Once you deploy the YAML Kubernetes manifest file, it will create the SealedSecret resource and install the controller into kube-system namespace, create a service account and necessary RBAC roles.

After a few moments, the controller will start, generate a key pair, and be ready for operation.

Resources created:
```
$ kubectl get all -n kube-system
NAME                                             READY   STATUS    RESTARTS   AGE
...           1/1     Running   63         2d
pod/sealed-secrets-controller-123456-abcdef   1/1     Running   2          4d

NAME                                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/sealed-secrets-controller   ClusterIP   xxx.xxx.xxx.xxx   <none>        8080/TCP   4d

NAME                                        DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/sealed-secrets-controller   1         1         1            1           4d

NAME                                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/sealed-secrets-controller-123abc456def   1         1         1       4d
```

If the controller Pod is not in Running state, check its logs:

```
$ kubectl logs sealed-secrets-controller-123456-abcdef -n kube-system
```

### 2. Retrieves the certificate keypair generated by the controller at start:

```
$ kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key
NAME                       TYPE                DATA   AGE
sealed-secrets-key<ID>     kubernetes.io/tls   2      4d
```

```
$ kubectl get secret sealed-secrets-key<ID> -o yaml -n kube-system

apiVersion: v1
data:
  tls.crt: <encoded_crt>
  tls.key: <encoded_key>
kind: Secret
metadata:
  creationTimestamp: "2021-02-20T20:37:11Z"
  generateName: sealed-secrets-key
  labels:
    sealedsecrets.bitnami.com/sealed-secrets-key: active
  name: sealed-secrets-key<ID>
  namespace: kube-system
  resourceVersion: "123456"
  selfLink: /api/v1/namespaces/kube-system/secrets/sealed-secrets-key<ID>
  uid: abc123-45def-67gh-89ij-687vf6
type: kubernetes.io/tls
```

Here you can see tls.crt and tls.key data that you can base64 decode and store in your Vault or in your secret management tool. 

## Client part

We will:
* install needed CLI tools
* create a Kubernetes Secret
* seal it into a SealedSecret
* convert JSON to YAML file and clean it
* deploy it in the Kubernetes cluster
* check if everything have been deployed & created

### 0. Pre-requisites

Install following tools:

* [Kubeseal](https://github.com/bitnami-labs/sealed-secrets/releases)

For MacOS you can install it via Brew:
```
$ brew install kubeseal
```

For others platform you can follow the [installation guide](https://github.com/bitnami-labs/sealed-secrets/releases).

* [kubectl-neat](https://github.com/itaysk/kubectl-neat) (via [Krew](https://krew.sigs.k8s.io/docs/user-guide/setup/install/))

```
$ kubectl krew install neat
```

### 1. Create Kubernetes secret

```
$ kubectl create secret generic my-token --from-literal=my_token='123456789abc123def456ghi789' --dry-run=client -o yaml -n my-namespace > my-token.yaml
```

The YAML file should be like this:

```
$ cat my-token.yaml

apiVersion: v1
data:
  my_token: <a_token>
kind: Secret
metadata:
  creationTimestamp: null
  name: my-token
  namespace: my-namespace
```

**Or** you can create it also with [Kustomize](https://kustomize.io/):

```
$ vi kustomization.yaml

namespace: my-namespace
secretGenerator:
- name: my-token
  literals:
  - my_token=<a_token>
generatorOptions:
  disableNameSuffixHash: true

$ kubectl kustomize . > my-token.yaml
```

### 2. Seal the secret (with the retrieved certificate)

```
$ kubeseal --cert tls.crt --format=yaml < my-token.yaml > mysealedtoken.yaml
```

The sealed YAML file should be like this:

```
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: my-token
  namespace: my-namespace
spec:
  encryptedData:
    my_token: <encrypted_token>
  template:
    metadata:
      creationTimestamp: null
      name: my-token
      namespace: my-namespace
```

### 3. Clean the YAML manifest file with "kubectl neat" command

```
$ kubectl neat -f mysealedtoken.yaml > mycleanedsealedtoken.yaml
```

The cleaned sealed secret YAML file should be like this:

```
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: my-token
  namespace: my-namespace
spec:
  encryptedData:
    my_token: <encrypted_token>
  template:
    metadata:
      creationTimestamp: null
      name: my-token
      namespace: my-namespace
    type: Opaque
```

### 4. Deploy SealedSecret in the cluster

```
$ kubectl apply -f mycleanedsealedtoken.yaml -n my-namespace
```

### 5. Check in your cluster

You can check/verify in your Kubernetes cluster if the SealedSecret and Secret have been correctly deployed:

```
$ kubectl get sealedsecret -n my-namespace
NAME           TYPE       DATA    AGE
my-token      Opaque       1      4d

$ kubectl get secret -n my-namespace
NAME          AGE
my-token    4d
```

Ok, my Secret have been successfully created by the controller! :-)

# Details

Be careful, the SealedSecret and Secret resources must have the same namespace and name. This is a feature to prevent other users on the same cluster from re-using your sealed secrets.

# Debug

As you know, some issues can appear, so in order to debug/troubleshoot the behavior of the controller, you can watch their logs: 

```
$ kubectl logs sealed-secrets-controller-<podID> -n kube-system
```

A possible issue is that you sealed the secret with another certificate than the ones the controller know ;-).

# Conclusion

Store your sensitive data in a Kubernetes Secret object is a common practice, but don't forget that a Secret is only encoded and not encrypted. So if you want to store them in a Git Repository (in GitHub or Gitlab repositories for example), you'll need to find a secure solution. Kubeseal is one of them and I hope this article helps you to see that it can be a solution that you can try in your side.