Managing app secrets in Kubernetes at Firmhouse
Tags: kubernetes, devops, azure, secrets
Published: 2019-07-09 09:47:38

---
title: Managing app secrets in Kubernetes at Firmhouse
published: true
tags: Kubernetes,DevOps,Azure,Secrets
canonical_url: https://www.michielsikkes.com/managing-and-deploying-app-secrets-at-firmhouse/
---

 ![Managing app secrets in Kubernetes at Firmhouse](https://www.michielsikkes.com/content/images/2019/07/shuto-araki-0Nlp0vqSgBY-unsplash.jpg)

At [Firmhouse](https://www.firmhouse.com) we're gradually migrating from a [Dokku](https://github.com/dokku/dokku)-based infrastructure onto a new [Kubernetes](https://kubernetes.io)-based infrastructure setup running on [DigitalOcean Kubernetes](https://www.digitalocean.com/products/kubernetes/). Out with manual work. In with automation!

In the new setup we heavily rely on [Terraform](https://terraform.io) for Infrastructure as Code collaboration and automation. One thing that we're adding to our Terraform repository is automated management of secrets and ENV vars for our app deployments.

In this article I'll show you how we're managing secrets and ENV vars via Terraform, [Azure Key Vault](https://azure.microsoft.com/services/key-vault/), and [Kubernetes Secrets](https://kubernetes.io/docs/concepts/configuration/secret/).

# App Deployment in Kubernetes

First of all, let's show you the thing that this is all about in the end: the application deployment. Our [Ruby on Rails](https://rubyonrails.org) applications are deployed using a [Kubernetes Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/). Here is a snippet of one of our applications from our Terraform repository:

{% gist https://gist.github.com/michiels/37fa5d85a6f9a5147d101f72d71d3290 %}

As you can see, we load the application's environment from a single Kubernetes Secret named _dispatch-env_.

We also put all Kubernetes objects for a given application in a specific [Kubernetes Namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/). Following this practice makes sure that by default we don't let secret stuff from one application deployment leak over to a different application.

# ENV vars via a single Kubernetes Secret

Simply loading the application's environment from a single Kubernetes secret makes it easy to manage the whole runtime environment via a single Terraform resource. Here is a redacted and shortened example of one of our apps:

{% gist https://gist.github.com/michiels/c165bc9994c1843bf072c307ebd02f9e %}

# Secret storage: Azure Key Vault

In our new infrastructure setup we store our secrets in an [Azure Key Vault](https://azure.microsoft.com/services/key-vault/). Azure Key Vault is comparable to [AWS Key Management Service](https://aws.amazon.com/kms/) or [HashiCorp Vault](https://www.vaultproject.io).

We used to not store secrets at all or have only certain manually generated secrets stored in our [1Password](https://1password.com) business account.

Because Terraform can talk to Azure Key Vault via [it's Azure provider](https://www.terraform.io/docs/providers/azurerm/index.html), we can now start managing secrets without ever touching them or making then visible to a human eye. This allows us to read a secret key from Azure Key Vault in our Terraform repository:

{% gist https://gist.github.com/michiels/f705a35226830d106c55e9fcc1943abc %}

# Setting up the application ENV variables with secrets from Azure Key Vault

So uptil this point we have three things prepared.

1. Applications are deployed using Kubernetes Deployments in their own namespace.
2. An application uses one single Kubernetes Secret to load the environment.
3. Azure Key Vault is set up and we can load secrets via Terraform resources.

Now let's bring it all together.

In the following redacted sample you'll see how we're pulling in SMTP login credentials and a database password to load those into the application deployment's ENV:

{% gist https://gist.github.com/michiels/2ced35ee5fe81aa9d9b0bcccda36725e %}

# Conclusion

## Why this is great

Such a setup is great for the following reasons:

- Anyone in the team can take a look at our Infrastructure as Code git repository to see what ENV vars are configured for a given application deployment.
- Secrets are safely stored in Azure Key Vault, including versioning, timestamps, access logs, and access policies for team members and applications.
- We don't have to manually log in to Dokku servers or into Heroku anymore to define the ENV variables for our apps.

## What can be improved

One thing that still bugs me is that the actual secret values are stored in a Kubernetes Secret. In practice, this is not worse than storing them in [Heroku](https://heroku.com) or Dokku. But it would even be greater to have a setup where the secret value is not visible to humans with access to the Kubernetes cluster.

[Jon Arild Tørresdal](https://www.michielsikkes.com/managing-and-deploying-app-secrets-at-firmhouse/Jon%20Arild%20T%C3%B8rresdal) has a solution in place at Sparebanken Vest. He has written great blog post about this setup. Check it out here: [Introducing Azure Key Vault to Kubernetes](https://mrdevops.io/introducing-azure-key-vault-to-kubernetes-931f82364354)