Securing Kubernetes Secrets with Conjur
Tags: kubernetes, security, conjur, cloudnative
Published: 2021-03-26 17:58:02

### Why to secure Kubernetes secrets?
Secrets management is one of the important aspects of securing your Kubernetes cluster. Out of the box, Kubernetes uses base 64 encoding for storing them, which is not enough. You have to implement a number of security best practices on top, to prevent possible security breaches. etcd encryption at rest, access control with RBAC, are a couple of examples of the same. Using secrets management solutions like CyberArk Conjur, not only secures them for Kubernetes, but also provides other benefits as we will see in the post.

### What is Conjur?
[CyberArk Conjur](https://www.conjur.org/) is a secrets manager. It helps you manage secrets in Kubernetes, as well as across applications, tools & clouds. It offers Role Based Access Control (RBAC) with an audit trail to easily track each stored secret. It implements encryption at rest with AES-256-GCM and in transit using mTLS. Additionally, you can manage the access for each secret & can also rotate the secrets automatically.

In this post, we will see how to install Conjur OSS on Kubernetes. We will go through a basic set of Conjur policies and will load them into Conjur. We’ll also see how to run an application in Kubernetes which uses secrets from Conjur by conforming to the defined policies.

## Pre-requisites

- **Familiarity with advanced YAML concepts.**
  
  You may be already familiar with the way Kubernetes spec files are written in YAML. Although you also need to understand a few more YAML concepts to understand & define Conjur policies, viz. **tags, anchors & aliases**. Conjur website has a [quick refresher](https://docs.conjur.org/Latest/en/Content/Operations/Policy/policy-syntax.htm) on this. Alternatively, you can go through the [full YAML documentation](https://yaml.org/spec).

- **A working Kubernetes cluster**

- **Docker installed locally**


## Setup

### How to Install Conjur?
The easiest way to Install Conjur on a Kubernetes cluster is by using the Helm chart. Let's first create a custom values file for the Helm chart.
```
$  DATA_KEY="$(docker run --rm cyberark/conjur data-key generate)"
$  HELM_RELEASE_NAME=conjur-oss
$  cat >values.yaml <<EOT
account:
  name: "default"
  create: true
authenticators: "authn-k8s/namespace,authn-k8s/deployment,authn-k8s/service_account,authn-k8s/demo,authn"
dataKey: $DATA_KEY
ssl:
  altNames:
  - $HELM_RELEASE_NAME
  - $HELM_RELEASE_NAME-ingress
  - $HELM_RELEASE_NAME.conjur.svc.cluster.local
  - $HELM_RELEASE_NAME-ingress.conjur.svc.cluster.local
service:
  external:
    enabled: false
replicaCount: 1
EOT
```
The `dataKey` is used for encrypting the secrets in the db. The `ssl.altNames` will be used for the SSL configuration of the Conjur service that the Helm chart will create. 

Install Conjur OSS on a Kubernetes cluster, with the following commands.
```
$  CONJUR_NAMESPACE=conjur
$  kubectl create namespace "$CONJUR_NAMESPACE"
$  VERSION=2.0.3
$  helm repo update
$  helm install \
   -n "$CONJUR_NAMESPACE" \
   -f values.yaml \
   "$HELM_RELEASE_NAME" \
   https://github.com/cyberark/conjur-oss-helm-chart/releases/download/v$VERSION/conjur-oss-$VERSION.tgz
```
The `VERSION` declared above is the Conjur Helm chart release version. As of writing this post, the latest Conjur OSS Helm chart version is `2.0.3`. Refer [Conjur Helm chart releases](https://github.com/cyberark/conjur-oss-helm-chart/releases) for the latest Conjur Helm chart available.

Once the helm chart is installed, it creates an admin user. You will need this key for the initial load of the Conjur policies, secrets, etc. You'll also need it in the "break-glass" scenarios. Hence you need to store it in a safe place. You can fetch the same using the commands below.

```
$ POD_NAME=$(kubectl get pods --namespace $CONJUR_NAMESPACE \
                     -l "app=$HELM_RELEASE_NAME,release=$HELM_RELEASE_NAME" \
                     -o jsonpath="{.items[0].metadata.name}")
$ kubectl exec --namespace $CONJUR_NAMESPACE \
          $POD_NAME \
          --container=$HELM_RELEASE_NAME \
          -- conjurctl role retrieve-key default:user:admin | tail -1
```

Verify the installation.
```
$ kubectl get po,svc -n $CONJUR_NAMESPACE
NAME                             READY   STATUS    RESTARTS   AGE
pod/conjur-oss-b888db5d5-vmfl5   1/2     Running   0          77s
pod/conjur-oss-postgres-0        1/1     Running   0          77s

NAME                          TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)         AGE
service/conjur-oss            NodePort    10.68.34.148   <none>        443:31022/TCP   79s
service/conjur-oss-postgres   ClusterIP   10.68.36.72    <none>        5432/TCP        79s
```

### Define Conjur Policies
Conjur policies help define objects in its database in a tree structure. Some examples of the objects defined in the policies are users, roles, secrets & applications. It also defines rules the for role based access control. While the Conjur documentation defines the [policy best practices](https://docs.conjur.org/Latest/en/Content/Operations/Policy/policy-best-practices.htm?TocPath=Fundamentals%7CPolicy%20Management%7C_____5), we will use one of the [Conjur demo repositories](https://github.com/conjurdemos/kubernetes-conjur-demo) to define policies. I've used policies in the demo repository as the base and have further simplified them to understand the basic concepts better. Download and review the [simplified policy files from my repository](https://github.com/samkulkarni20/conjur-k8s-integration/tree/main/policy). Note that all the policies need to have a `.yml` extension.
- `1_users.yml` file defines users and roles. It also grants role based access to a group of users as well as to individual users.
- `2_app_authn-def.yml` file defines applications & groups them into a layer for easier access management.
- `3_cluster-authn-svc-def.yml` file defines the authenticator service & the SSL certificates for the mTLS communication between Conjur & its clients. In this case, Conjur clients are applications running on Kubernetes. It also defines role based access to authenticate with the service.
- `4_app-identity-def.yml` file connects the authentication identities to application identities.
- `5_authn-any-policy-branch.yml` policy is defined to verify that hosts can authenticate with Conjur from anywhere in the policy branch to retrieve secrets for Kubernetes.
- `6_app-access.yml` defines secret variables for different applications that will use Conjur as its secrets manager. Note that the variables mentioned here are just secret variable names & not the values.

### Generate mTLS cert & key
Run the [create_mtls_certs.sh](https://github.com/samkulkarni20/conjur-k8s-integration/blob/main/create_mtls_certs.sh) shell script to create the mTLS cert & key. Make sure to update the `AUTHENTICATOR_ID` & `CONJUR_ACCOUNT` in the script with the values appropriate for the Conjur installation. `AUTHENTICATOR_ID` is the part that follows `conjur/authn-k8s/` in the `id` of the `3_cluster-authn-svc-def.yml` policy file.

### Load policies & secrets
We'll use `conjur-cli` to load policies & data in Conjur. Conjur has a container image pre-packaged with the Conjur cli. We will run the Conjur client as a pod on the cluster. The policies and certificates will get mounted on the same as configmap volumes. `conjur-cli` will load the policies & certificates to the Conjur server from these volumes. 

Have all the policy files under the `policy` directory and the mTLS cert-key pair in `mtls` directory to create configmaps out of it.

```
$ # Create configmap containing the mTLS cert & key
$ kubectl create configmap conjur-ca -n $CONJUR_NAMESPACE --from-file $(pwd)/mtls
$ # Create a configmap containing all the policy files
$ kubectl create configmap policies -n $CONJUR_NAMESPACE --from-file $(pwd)/policy
```

Run the Conjur client pod with the above configmaps mounted as volumes. Download the sample pod config from [here](https://github.com/samkulkarni20/conjur-k8s-integration/blob/main/deploy/conjur-client.yaml). Create the pod with downloaded config & exec into it to load values.

```
$ kubectl create -f conjur-client.yaml
$ kubectl exec -it -n $CONJUR_NAMESPACE conjur-client -- sh
```

Connect to the Conjur server & authenticate as an admin user
```
$ export CONJUR_URL=https://conjur-oss
$ export ACCOUNT=default
$ conjur init -u $CONJUR_URL -a $ACCOUNT
$ conjur authn login -u admin -p <admin_api_key_printed_by_helm_install>
```

We can start loading policies now. Loading the policy files `1_users.yml` and `2_app-authn-def.yml` in Conjur generates API keys for the users & hosts defined in it. The user API keys can be distributed to respective team members, allowing them to authenticate & interact with Conjur. We will use the [Kubernetes authenticator](https://docs.conjur.org/Latest/en/Content/Operations/Services/k8s_auth.htm) instead of host API keys to get the application authenticated with Conjur.

```
$ conjur policy load root policy/1_users.yml
$ conjur policy load root policy/2_app-authn-def.yml
$ conjur policy load root policy/3_cluster-authn-svc-def.yml
$ conjur policy load root policy/4_app-identity-def.yml
$ conjur policy load root policy/5_authn-any-policy-branch.yml
$ conjur policy load root policy/6_app-access.yml
```

Load mTLS certificate & key.

```
$ conjur variable values add conjur/authn-k8s/demo/ca/cert "$(cat conjur-ca/ca.cert)"
$ conjur variable values add conjur/authn-k8s/demo/ca/key "$(cat conjur-ca/ca.key)"
```

Load secret values.

```
$ conjur variable values add demo-app-vars/url "https://my.app.com"
$ conjur variable values add demo-app-vars/username "myuser"
$ conjur variable values add demo-app-vars/password "supersecret"
```

You don’t want the client pod to continue running in the cluster, especially because it’s currently logged in to the Conjur server. Hence either log out from the Conjur server with `conjur authn logout` or delete the `conjur-client` pod. Also, delete the configmaps mounted on the client pod.

```
$ kubectl delete -f conjur-client.yaml
$ kubectl delete cm conjur-ca policies
```

### Configure & run application
Conjur offers various [authenticators](https://docs.conjur.org/Latest/en/Content/Operations/Services/Authentication-new.htm?tocpath=Fundamentals%7CAuthentication%7C_____0) for users and hosts. Here we will use the [Kubernetes authenticator](https://docs.conjur.org/Latest/en/Content/Operations/Services/k8s_auth.htm?tocpath=Fundamentals%7CAuthentication%7C_____8) to get our application host authenticated with the Conjur server. Kubernetes authenticator uses Kubernetes APIs to authenticate resources like Pod, Deployment, etc. Refer [Conjur documentation](https://docs.conjur.org/Latest/en/Content/Integrations/Kubernetes_AppIdentity.htm?tocpath=Integrations%7COpenShift%252C%20Kubernetes%7C_____2#SupportedKubernetesresources) to see the full list of supported Kubernetes resources that can be defined as hosts. 

[Kubernetes Authenticator Client](https://docs.conjur.org/Latest/en/Content/Integrations/Kubernetes_deployAuthenticatorSidecar.htm) is one of the two options for using this authentication method. You can run it as `initContainer` or as `sidecar` for each application. Configure the Conjur url, account, login, etc as the env variables on the application and the authenticator container. Login is the full host id as defined in the `2_app-authn-def.yml` policy file. You also need to mount the SSL certs for the Conjur service. In this case, we have to use the SSL certificates generated by Helm chart during Conjur installation. If you have your own SSL certificates configured on the Conjur server, you can use the same. Note that the value of `CONJUR_AUTHN_URL` on the authenticator container is slightly different from the `CONJUR_APPLIANCE_URL` on the application container. `CONJUR_AUTHN_URL` has the authetication service id appended to the `CONJUR_APPLIANCE_URL`.

Authentication client authenticates itself to the configured Conjur server url & provides it's identity, i.e., the login id, pod name, namespace, etc. Conjur validates the information provided by the authenticator client against defined policies, as well as, Kubernetes & provides an access token. The access token is valid only for 8 mins. Client container saves the authentication token on an in-memory volume, which is mounted to both the containers viz. application & authenticator. 

[Summon](https://docs.conjur.org/Latest/en/Content/Tools/summon.html?tocpath=Developer%7C_____1) which is a separate open source utility from CyberArk Conjur, uses this token to fetch the values for secrets. Your application container needs to run `Summon` as its main process with path to a `secrets.yml` file that lists all the secret values that it needs to pull from Conjur. Summon runs the application executable as a sub-process & passes the secret values it fetched as env variables. See the command configured on the [Dockerfile](https://github.com/samkulkarni20/conjur-k8s-integration/blob/main/Dockerfile) of the application we're about to run.
```
$ tail -n1 Dockerfile
ENTRYPOINT ["summon", "--provider", "summon-conjur", "-f", "/etc/secrets.yml", "/bin/sh", "-c", "while true; do printenv | grep PASSWORD; sleep 5; done"]
```

The `!var` in the [secrets.yml](https://github.com/samkulkarni20/conjur-k8s-integration/blob/main/secrets.yml) file indicates that the value needs to be injected as env variable. It can be replaced with `!file:var` in case you want the value to be written to a file. In that case, the env variable name on the left side will contain the file's path where secret content is written. Observe the contents of our `secrets.yml` below.

```
$ cat secrets.yml
PASSWORD: !var demo-app-vars/password
```

Before running the application, let's first create our application namespace & copy the secret containing Conjur tls certs to our application namespace.
```
$ kubectl create namespace test
$ kubectl get secret conjur-oss-conjur-ssl-cert --namespace=conjur -oyaml |\
 sed 's/namespace: conjur/namespace: test/g' |\
 kubectl apply -f -
```

Use the example [application deployment file](https://github.com/samkulkarni20/conjur-k8s-integration/blob/main/deploy/busybox.yaml) to run the application. As you saw in the `Dockerfile`, my example application is a busybox container. It just prints the value of secret `demo-app-vars/password` from Conjur every 5 seconds. A typical application should never print secret values in logs; but we'll use it only to demonstrate that the value is available to the application from Conjur. Let's run the same & observe the logs.

```
$ kubectl create -f busybox.yaml
```

Check to see if the application has the secret value from Conjur available as an environment variable.

```
$ kubectl logs -f -lapp=busybox
pass: supersecret
pass: supersecret
pass: supersecret
pass: supersecret
```

An additional security feature you get by using Conjur & Summon is that the secret value is only available to the application & not to the entire container. This means, if an attacker were to get access inside the application container, they won't be able to access the secret values by listing the environment variables in the container.

```
$ kubectl exec -it -lapp=busybox -- sh
$ printenv | grep PASSWORD
$
$ # No output above
```
### Cleanup

Cleanup all the resources created in this post with the below commands:
```
$ kubectl delete -f busybox.yaml
$ helm delete $HELM_RELEASE_NAME -n $CONJUR_NAMESPACE
$ 
$ ## Delete client pod & configmap, if not already removed
$ kubectl delete -f conjur-client.yaml
$ kubectl delete cm -n $CONJUR_NAMESPACE conjur-ca policies
```

## Conclusion

In this post, we looked at what Conjur is, its uses & basic concepts. We also installed Conjur on a Kubernetes cluster and integrated it with a sample application running in Kubernetes. Hope this gives you a good start for using Conjur with Kubernetes.

*We’re always thrilled to connect to people working with cloud native technologies. For any queries or comments, you can reach out to us via [Twitter](https://twitter.com/infracloudio) and [LinkedIn](https://www.linkedin.com/company/infracloudio).*

## References
[conjur.org](https://conjur.org)  
[Conjur OSS Helm Chart](https://github.com/cyberark/conjur-oss-helm-chart/tree/master/conjur-oss)  
[Kubernetes Conjur Demo](https://github.com/conjurdemos/kubernetes-conjur-demo)  
[My GitHub repo with all the resources used in the post](https://github.com/samkulkarni20/conjur-k8s-integration)