Using HashiCorp Vault to generate temporary credentials to Azure from GitHub Actions
Tags: vault, hashicorp, github, terraform
Published: 2023-03-10 13:16:20

## Introduction

The [HashiCorp Cloud Platform (HCP)](https://www.hashicorp.com/cloud) simplifies working with many of the great HashiCorp products that are available. One of these products is **Vault**. Vault is a platform for connecting your application secrets (or any kind of static or dynamic secrets for that matter) with the users and applications that require access to them. Secrets can be stored almost anywhere, and you can authenticate to Vault with a multitude of options.

The possibilities that Vault opens up are almost endless, but you have to start somewhere! In this post I will:

- Create a new Vault cluster on HCP using Terraform (another great tool from HashiCorp).
- Configure my Vault cluster with the [Azure secrets engine](https://developer.hashicorp.com/vault/docs/secrets/azure) and the [AppRole authentication method](https://developer.hashicorp.com/vault/docs/auth/approle).
- Create a GitHub Actions Workflow that connects to my Vault cluster, asks for temporary credentials to an Azure Service Principal, uses these credentials to authenticate the Azure CLI.

This post is not a gentle introduction to any of the technologies that I will use, but hopefully it will whet your appetite to learn more about everything!

## Prerequisites

If you want to follow along what I do in this post there are a number of prerequisites that must be in place. I assume you already have `git` installed, but apart from that you will also need to:

- Sign up for a HashiCorp Cloud Platform account [here](https://portal.cloud.HashiCorp.com/sign-up)
- Sign up for an Azure account [here](https://azure.microsoft.com/en-us/free/)
- Install the Azure CLI following [these](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli) instructions
- Install the GitHub CLI following [these](https://github.com/cli/cli#installation) instructions
- Install Vault following [these](https://developer.HashiCorp.com/vault/docs/install) instructions
- Install Terraform following [these](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli) instructions

## Create an Azure service principal for Vault

For Vault to be able to create and delete Azure service principals, as well as assigning roles to these service principals, it will require a service principal of its own with an appropriate Azure role assignment as well as Microsoft Graph API permissions.

There are three steps I need to perform:

1. Create the service principal.
1. Assign the service principal an Azure owner role on my Azure subscription.
1. Assign the service principal Microsoft Graph API permissions to allow it to create service principals on its own.

### Create a service principal and assigning Azure roles

I will manually create a service principal for Vault using the Azure CLI and give it the owner role on my subscription with a single command (i.e. step 1 and 2 from above):


![azure create service principal](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/k7jrhz3ed3r112ho901c.png)

The command creates a service principal named `hcp-vault` with the owner role on my subscription. The output includes details that I will provide to Vault when I configure the Azure secrets engine in a later step. For now I store the output somewhere safe for later.

### Assign Microsoft Graph API permissions

The service principal needs permission to create new service principals on its own. This means I have to give it an application permission (not a delegated permission) named `Application.ReadWrite.All`. I can do this with `az ad app permission add`:

![MS graph permission add](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6tnhlmwbfoalf4menz7e.png)

The strange looking GUID I provide for the `--api` flag corresponds to Microsoft Graph API, and the other GUID I provide for the `--api-permissions` flag corresponds to the `Application.ReadWrite.All` permission. Since this is an application permission (i.e. the application will act on its own without a signed-in user) this requires consent from an Azure AD administrator. Lucky for me I am my own Azure AD administrator, so I can consent using `az ad app permission admin-consent`:

![MS graph consent](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cibz5ghx3gj29pwre4iq.png)

Now I have an Azure service principal that Vault can use to do what I want it to do.

## Setting up a Vault cluster on HCP

I will use Terraform[^terraform] to set up my Vault cluster on HCP. For this purpose I will use the [HCP Terraform provider](https://registry.terraform.io/providers/hashicorp/hcp/latest/docs). The full Terraform configuration to set up a Vault development cluster on HCP looks like this:

![terraform configuration](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xoco4h1s59cry5ehiblb.png)

Let me explain some of the contents of this configuration:

- On line 10 I include an empty provider configuration block for the HCP provider. This is where I could supply the required credentials needed to authenticate to HCP, but I will set the required values as environment variables instead.
- On line 12 I create a `hcp_hvn` resource. This is a **HashiCorp Virtual Network** resource. Placing our cluster resources in a virtual network opens up the possibility to peer your own network together with it, to allow secure communication between your applications and the Vault cluster. I will not take that step in this post, however.
- Finally, on line 19 I create the cluster itself. As can be seen on line 23 I set `public_endpoint` to `true`, this is not recommended for production scenarios but I use it here to keep the level of this post reasonable. 

To be able to work with HCP from Terraform I must set up a HCP service principal to authenticate with. The steps to create a HCP service principle are as follows:

1. I sign in to HCP and go to my organization.
1. I click on `Access control (IAM)` in the left-hand menu.
1. I click on `Service principals` in the new left-hand menu that appeared.
1. I click on the `+ Create service principal` button.
1. I provide a name for my service principal and assign it the `Contributor` role.
1. I click on the `Save` button and I am presented with a client ID and a client secret.

With my HCP service principal in place I set the client ID and client secret as environment variables (`HCP_CLIENT_ID` and `HCP_CLIENT_SECRET`, respectively) in my terminal where I will run Terraform:

![terraform hcp credentials](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/koi374i38ee2t27fvh5d.png)

I now have my Terraform configuration and I have the required credentials to work with HCP from Terraform. If you are familiar with the Terraform core workflow you know that the first step is the initialization step with `terraform init`:

![terraform init](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2ky27v0lkx0o3bhsl4tg.png)

The initialization step installed the HCP provider in my working directory. I run `terraform plan` to generate a plan for what resources Terraform will create:

![terraform plan](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0arfmfl6m42jzzmhlcmt.png)

Terraform reports that two resources will be created, and that is the correct amount of resources that I expect. Since the plan looks good to me I go on and apply the plan using `terraform apply`:

![terraform apply](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tmiytgvlj68lnlo1ufqd.png)

It takes a while to create the cluster (**16 minutes and 6 seconds** in my case, plus **2 minutes and 26 seconds** for the network), so you can safely take a coffee break while you wait ☕️.

## Configuring the Vault cluster

Now it is time to configure my Vault cluster.

First of all, I need to set some environment variables with details that will allow me to communicate with my Vault cluster. To find what these environment variables are and what values they should have I follow these steps:

1. I sign in to HCP and click my way to my Vault cluster.
1. I click on **Generate token** under the **New admin token** header to generate an admin token for my cluster.
1. I click on **Access Vault** and then choose **Command-line (CLI)**.
1. I copy all of the commands that are shown and paste them into my terminal:

![vault environment variables](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/s4oyiueujnaxkbdkmsjq.png)

Now I can verify that I have a working connection to my cluster using `vault status`:

![vault status](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8cal7tkfw6upggg4pdf7.png)

We need to enable the [Azure secrets engine](https://developer.hashicorp.com/vault/docs/secrets/azure) for Vault to be able to work with Azure. I run the `vault secrets enable` command to enable the secrets engine:

![vault enable secrets engine](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wglqjyv6w1scpka8w19k.png)

Then I run a `vault write` command to write configuration for the Azure secrets engine and provide information for my Azure service principal that I created in the previous section:

![azure secrets engine](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/s7efb882fxfp6fzm55o8.png)

Next I need to specify what should happen when GitHub requests a temporary Azure service principal. To keep things simple I will let the temporary service principal become a contributor on my Azure subscription. To configure this I will use `vault write` to create a **role** named `github-role` in the Azure secrets engine:

![vault role](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/m8smfq4umbitzg378kp5.png)

I provided the list of Azure roles (with a single entry) inline, but I could also have stored it in a file. Note that I specified `ttl=5m` in the command, this tells Vault that the Azure service principal that is created should be deleted after 5 minutes, i.e. this is what makes the credentials (and the service principal itself) temporary.

I want GitHub to be able to communicate with my Vault cluster so GitHub will require a Vault token to do so. There are a few options on how to proceed. The method I will use is to activate the **AppRole** authentication method, and create an AppRole for GitHub with an appropriate policy attached. The policy I will use is this:

![vault policy](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hhxr5wxoj6kiea5xuo9k.png)

This policy says that GitHub will only be able to run `read` operations on the `azure/creds/github-role` path in my Vault cluster. I create the policy in Vault using `vault policy write`:

![vault policy write](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7civ90k4ov80forcvr37.png)

I enable the AppRole authentication method using the `vault auth enable` command:

![vault auth enable](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/llu0su3dt5bfyveyyesr.png)

I create an AppRole named `github` for GitHub with the Vault policy I created before:

![vault auth write](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z89e8yfz5dp6tkt55hdg.png)

The final step is to extract a role ID and a secret ID for the `github` AppRole, these values will be provided as secrets to my GitHub repository. First I need the role ID:

![approle details](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/12j3dz3rqkl1u713eczx.png)

Then I need the secret ID:

![approle details](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mc8gz45pz13nd441osoa.png)

Now I am done configuring Vault! I copy the role ID and the secret ID to a safe place, I will set them as repository secrets in GitHub later. The next step is to use Vault from GitHub! 

## Setting up a GitHub repository

All the prerequisites are taken care of, and now it is time for an imaginary developer team to set up a new GitHub repository and build a GitHub Actions workflow that will use Vault to obtain a temporary service principal.

For this imaginary scenario I will only use the temporary service principal to list all resources groups in my subscription and count how many they are.

I start in an almost empty local git repository[^git]. I create a new GitHub repository using the `gh` command line tool:

![create new repository](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4eqgup1duued07hjnhcr.png)

To provide the required secrets to GitHub I run the command `gh secret set` once each for the following keys:

- `VAULT_ROLE_ID` with the role ID I obtained from Vault.
- `VAULT_SECRET_ID` with the secret ID I obtained from Vault. 
- `TENANT_ID` with the value of my Azure AD tenant ID.
- `SUBSCRIPTION_ID` with the value of my Azure subscription ID.

I add a new GitHub Actions workflow to `.github/workflows/list-resource-groups.yaml` with the following content:

![github actions workflow](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/c04gsl0nbwjyypexpho2.png)

To communicate with Vault I have used the [Vault action](https://github.com/hashicorp/vault-action). This action provides many different ways of authenticating to Vault, but since I set up an AppRole I use the `approle` method. I provide the role ID and secret ID to vault for authentication, and I specify what secrets I want to extract from Vault in the `secrets` parameter. The syntax `azure/creds/github-role client_id | AZURE_CLIENT_ID` means "read the secret named `azure/creds/github-role` from Vault, take the `client_id` value from the output and make it available as an environment variable named `AZURE_CLIENT_ID`".

The rest of the workflow authenticates the Azure CLI using the temporary service principal credentials together with the subscription ID and tenant ID I stored as secrets before, and finally it lists and counts the number of resource groups that I have in my subscription.

Since I included a manual workflow trigger I can start the workflow using the `gh` command line tool:

![run github workflow](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/uielv18q7fa1mg2pp3rk.png)

Once the workflow has finished I can verify that the workflow worked from the result (looks like I have 6 resource groups):

![github workflow result](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ygo1zzfz17lvgt9cbm0x.png)

If I wait for five minutes then the service principal that Vault created will be removed. No valid credentials to Azure remains! Note that it takes a while (around 20-30 seconds) to create the service principal and associated secret, so this is not a method you would use for anything that require immediate credentials.

## Summary and outlook

This post contained a lot of technicalities! What have we actually achieved here? We have an embryo for a GitHub Actions workflow that obtains temporary credentials to perform actions in Azure. These credentials are short-lived, and there is no need to have long-lived service principals with secrets that have to be rotated.

One could argue that we have now moved the problem to the role ID and secret ID that we used to authenticate to Vault instead of the Azure service principal client secret that we would otherwise have to protect. That is true in some way, and if this was the only thing we used our Vault cluster for then this would indeed be a valid criticism (and also an expensive solution because HCP Vault is not free of charge). However, the idea is to use the Vault cluster for all your secrets management. What I demonstrated here was for Azure, but you might also require temporary credentials to a database, or to AWS, or for SSH-access to your build-agents. As with many great tools it is often overkill to use it for a small piece of your overall architecture, instead you should utilize it everywhere.

How can you make further improvements on this architecture? In a production scenario I would have set up my own VNET in Azure, and I would have peered this network with my HCP network. With that done I can turn off public access for my Vault cluster, and keep all traffic to my cluster inside my own network. This would also require me to use my own build agents in GitHub. I would also look in to different methods of authenticating GitHub to Vault, instead of using the AppRole method. If no other method seems to be suitable I would at least configure a TTL on the secret ID and have GitHub acquire a new one periodically.

There is much to learn, and my journey with HashiCorp Vault (and HCP Vault specifically) have only just begun. I am intrigued by the possibilities!

[^terraform]: This is not a course on how to use Terraform, but I do have such a course! Visit [mattias.engineer/terraform/](https://mattias.engineer/terraform/) to see more about it.
[^git]: I just ran `git init` in an empty directory, added a `README.md` file, and committed it with `git commit -am "Initial commit"`.