Argo CD secrets management using Sops
Tags: kubernetes, devops, security, cicd
Published: 2020-05-04 14:56:19

At [Camptocamp](https://camptocamp.com), we've been using [Argo CD](https://argoproj.github.io/argo-cd/) for some time. It does a wonderful job of deploying Helm charts to Kubernetes clusters. But we lacked a way to manage our secrets so that we could safely commit them to our Git repositories.

# Helm Secrets plugin

We knew about [Helm Secrets](https://github.com/zendesk/helm-secrets), a Helm plugin which uses [Sops](https://github.com/mozilla/sops) under the hood to manage encrypted value files. We intended to use it with Argo CD but we faced several issues:

- To render an Helm chart's manifests, Argo CD issues a `helm template` command. To use Helm Secrets, it would have to execute `helm secrets template` instead. However, there is no way to edit the Helm command used by Argo CD. One has to configure a [custom plugin](https://argoproj.github.io/argo-cd/user-guide/config-management-plugins/). This has the drawback of loosing Helm specific features (like specifying value files or parameters).
- Helm Secrets works by decrypting secret value files to temporary files on disk before actually calling Helm. While not a big concern, it's not a security best practice. Secrets should never touch the disk when they are unencrypted.
- Helm Secrets seemed unmaintained at the time.


# Helm Sops

Since we only needed a small subset of Helm Secrets' functionalities, we decided to write a small Go binary which would transparently wrap Helm so that we could substitute the Helm binary with it in the Argo CD Docker image. Let's introduce [Helm Sops](https://github.com/camptocamp/helm-sops)!

{% github camptocamp/helm-sops %}

Helm Sops is loosely inspired by Helm Secrets but uses named pipes instead of temporary files to pass decrypted secret value files to Helm.

To use it:

1. deploy our custom Argo CD [Docker image](https://github.com/camptocamp/docker-argocd) integrating Helm Sops to your Kubernetes cluster,
2. install Sops on your workstation and create a [`.sops.yaml`](https://github.com/mozilla/sops#using-sops-yaml-conf-to-select-kms-pgp-for-new-files) file to specify the encryption method to use (GPG key, AWS KMS key, etc),
3. use Sops to create a file named `secrets.yaml` and put your Helm secret values inside,
4. `git commit`,
5. `git push`,
6. finally, visit the Argo CD web interface to deploy your Helm application \o/
7. (optional) install Helm Sops on your workstation to be able to use the Argo CD CLI with encrypted value files.

Full step by step instructions can be found [here](https://github.com/camptocamp/argocd-helm-sops-example).


# Security Considerations

While Sops uses battle tested cryptography, the way we use it with Argo CD makes it vulnerable to the [confused deputy problem](https://en.wikipedia.org/wiki/Confused_deputy_problem). Someone which can deploy applications using Argo CD and has access to an encrypted value file but cannot decrypt it, can deploy a new application accompanied by the encrypted value file and make Argo CD decrypt it for them.

This limitation can be problematic in multi-tenant environments. A way to solve it is to make use of [AEAD](https://en.wikipedia.org/wiki/Authenticated_encryption#Authenticated_encryption_with_associated_data). AWS KMS offers [encryption context](https://aws.amazon.com/fr/blogs/security/how-to-protect-the-integrity-of-your-encrypted-data-by-using-aws-key-management-service-and-encryptioncontext/) to this end.

Sops already supports AWS KMS encryption context but it relies on the one specified in the metadata of the encrypted file. We would need Argo CD to be able to overwrite this encryption context when decrypting secret value files so that it cannot be confused anymore. We'd be happy to work with Sops maintainers to add this possibility 😉