Handling an ACME challenge response with a temporary Azure Container Instance
Tags: azure, aci, acme, certbot
Published: 2020-11-27 13:18:12

---
title: Handling an ACME challenge response with a temporary Azure Container Instance
published: true
description: I wanted to have a temporary endpoint to handle a ACME HTTP-01 challenge response without having or messing up a server. This led me to a really nice use case for Azure Container Instances.
tags: #azure #aci #acme #certbot
cover_image: https://dev-to-uploads.s3.amazonaws.com/i/lcmfh4abyyi40btjyg6w.jpg
---

[ACME](https://en.wikipedia.org/wiki/Automated_Certificate_Management_Environment) brings a high level of convenience for issuing certificates for cloud services (global & local gateways, web services).

That works pretty fine in many variations, but what if

- I don't have an actual server where I can host some __certbot__ / __nginx__ magic to handle a HTTP-01 challenge - [see challenge types](https://letsencrypt.org/docs/challenge-types/)
- I do not want to spoil an infrastructure and pass down a request to the bowels of my business compute [see sample architecture](https://dev.to/kaiwalter/using-azure-private-links-and-private-dns-zones-with-globally-distributed-resources-4ce3)
- I'm not able to maintain DNS entries via automation for a DNS-01 challenge

and I want to automate anyway?

This post describes how to 

- build and bring up a container based on [certbot](https://certbot.eff.org/) & [nginx](https://www.nginx.com/) in [ACR - Azure Container Registry](https://docs.microsoft.com/en-us/azure/container-registry/) and [ACI - Azure Container Instances](https://docs.microsoft.com/en-us/azure/container-instances/container-instances-overview)
- wire up this container with an [Application Gateway](https://docs.microsoft.com/en-us/azure/application-gateway/overview) instance

### What is required

- an existing Application Gateway which currently routes traffic and which is bound (custom DNS) to the domain, for which the certificate is to be requested - having no handling for port 80 / HTTP so the route to the container instance can be injected
- or (as in my case) an existing [Front Door](https://docs.microsoft.com/en-us/azure/frontdoor/) from which the HTTP / port 80 traffic is routed to a temporary / dedicated Application Gateway

### What I use

- __Azure CLI 2.15.1__ to handle Azure resources
- __PowerShell Core 7.x__  to drive the process - because I like the primitives for handling JSON emitted by __Azure CLI__  - but this can be easily converted to __bash__ / __jq__
- a __Storage Account__ mounted to __ACI__ to capture __certbot__ output
- an __ACR__ to build and provide the Docker image
- a __Service Principal__ to allow __ACI__ to pull image from __ACR__ - I would have preferred __Managed Identity__ but there is currently [a limitation](https://docs.microsoft.com/en-us/azure/container-instances/container-instances-managed-identity#limitations)
- __nginx__ to handle incoming web traffic for HTTP-01 challenge
- __certbot__ to drive ACME process

## Structure

Script snippets provided in this post follow this structure

1. build up permanent components (ACR, Service Principal, create Storage, create Application Gateway HTTP route)
2. build up temporary components (renew Service Principal credentials, build image, create ACI, establish Application Gateway to ACI link)
3. execute certificate request
4. destroy temporary components (see 2. - but reverse)

assuming between steps 1 and 2, after a certificate has been initially created, some time passes until certificate is to be renewed.

Generally all components could be created and wired up temporarily - which would make it a even more secure and kind of stateless approach. However while building and trying I collected these topics to consider:

a) deploying a temporary dedicated __Application Gateway__ with all the settings can easily take 20-30 minutes
b) in my case - connecting this temporary __Application Gateway__ to __Front Door__ also can take another 20-30 minutes, until configuration really is effective
c) in one version I created __Service Principal__ temporarily; just because I ended up in a lot of iterations I splitted this process into a creation and renewal of credentials
d) __Storage__ also can be handled temporarily - just a matter of a few minutes of deployment time - but then with a GUID like name to guarantee uniqueness

----

## 0. common variables and initialization

I did not spend any energy in abstration of commonly used configuration information - hence I have these variables copied at the start of each of the scripts:

```PowerShell
$location = "{your-location}"
$resourceGroupName = "{your-resourcegroup}"

$appGwName = "myacmepoc" # name of the Application Gateway to create / manage
$containerGroupName = "myacmepoc" # name of the Azure Container Instance to be created
$publicIpName = "myacmepoc" # name of Public IP used for Application Gateway
$registryName = "myacmepoc" # name of Azure Container registry
$servicePrincipalName = "myacmepoc" # name of Service Principal to be used to pull from ACR to ACI
$storageAccountName = "myacmepoc" # name of Storage Account where certificates are captured (from ACI)
$storageAccountShareName = "certificates" # name of share on Storage Account to be mounted to ACI
$subscriptionId = "{your-subscription-id}"

az account set -s $subscriptionId

az configure -d location=$location group=$resourceGroupName
```

----

## 1. build up permanent components<a name="chapter-1"></a>

### Container Registry with a Service Principal

```PowerShell
az acr create -n $registryName --sku Basic

$registryResourceId = $(az acr show --name $registryName --query id --output tsv)
az ad sp create-for-rbac --name $servicePrincipalName --scopes $registryResourceId --role acrpull
```

### dedicated Application Gateway

> SKIP this section when you're working with an existing Application Gateway, which is already used for your backend exposure

Create a basic setup with HTTPS which allows modification of HTTP configuration without breaking the gateway.

> PFX path/password - in my case I already had an existing certificate which I could bind to the HTTPS configuration

```PowerShell
az network application-gateway create `
    --name $appGwName `
    --capacity 1 `
    --sku Standard_v2 `
    --http-settings-cookie-based-affinity Enabled `
    --public-ip-address $publicIpName

## - HTTPS configuration

az network application-gateway http-settings create `
    -n appGatewayBackendHttpsSettings `
    --gateway-name $appGwName `
    --port 443 --protocol Https --cookie-based-affinity Enabled

az network application-gateway ssl-cert create `
    -n httpsCert `
    --gateway-name $appGwName `
    --cert-file $(Read-Host "PFX path") `
    --cert-password $(Read-Host "PFX password")

az network application-gateway frontend-port create `
    -n appGatewayHttpsPort `
    --gateway-name $appGwName `
    --port 443

az network application-gateway http-listener create `
    -n appGatewayHttpsListener `
    --gateway-name $appGwName `
    --frontend-ip appGatewayFrontendIP `
    --frontend-port appGatewayHttpsPort `
    --ssl-cert httpsCert

az network application-gateway rule create `
    -n ruleHttps `
    --gateway-name $appGwName `
    --address-pool appGatewayBackendPool `
    --http-listener appGatewayHttpsListener `
    --http-settings appGatewayBackendHttpsSettings

## - HTTP configuration - clean-up

az network application-gateway rule delete `
    -n rule1 `
    --gateway-name $appGwName

az network application-gateway http-settings delete `
    -n appGatewayBackendHttpSettings `
    --gateway-name $appGwName
```

### add HTTP configuration to gateway

Create the HTTP configuration with a dummy backend - which will be set to the address of running __ACI__ container later. Add a probe so that health can be checked.

```PowerShell
## - HTTP configuration - build-up

az network application-gateway probe create `
    -n probeContainer `
    --gateway-name $appGwName `
    --path "/index.html"  --protocol Http `
    --host-name-from-http-settings true

az network application-gateway address-pool create `
    -n appGatewayBackendContainerPool `
    --gateway-name $appGwName `
    --servers 10.0.0.1

az network application-gateway http-settings create `
    -n appGatewayBackendHttpSettings `
    --gateway-name $appGwName `
    --probe probeContainer `
    --port 80 `
    --host-name-from-backend-pool true

az network application-gateway rule create `
    -n ruleContainer `
    --gateway-name $appGwName `
    --address-pool appGatewayBackendContainerPool `
    --http-listener appGatewayHttpListener `
    --http-settings appGatewayBackendHttpSettings
```

### (optional) link to Front Door

At this step I would link the exposed public IP on port 80 to __Front Door__.

----

## 2. build up temporary components<a name="chapter-2"></a>

### initialization

Initialize some values for the certificate generation process. My provider works with [External Account Binding](https://dev.digicert.com/services-api/api-keys/acme-external-account-binding/) - hence these additional credential information.

```PowerShell
$downloadFolder = "./certificates"
$domain = "webservice.mydomain.com"
$emailAddress = "my.email@mydomain.com"
$serverUrl = "https://{providers-ACME-directory-url}"
$eabkid = "{external-account-binding-key-identifier}"
$eabhmac = "{HMAC-key-for-external-account-binding}"
```

> `[Console]::ResetColor()` comes in handy in case console colors are messed up from a previous failed run

I like to specifically tag those kind of images just to be absolutely sure that the correct version is pulled and used.

```PowerShell
[Console]::ResetColor()

$registry = az acr show -n $registryName -o json | ConvertFrom-Json
$tag = Get-Date -AsUTC -Format yyMMdd_HHmmss
$image = "$($registry.loginServer)/certbot:$tag"

$storageAccountKey = $(az storage account keys list --account-name $storageAccountName --query "[0].value" --output tsv)
az storage share create --name $storageAccountShareName --account-name $storageAccountName
```

Renew credentials on __Service Principal__ so that I do not need to store those somewhere.

```PowerShell
$servicePrincipal = az ad sp credential reset --name $servicePrincipalName -o json | ConvertFrom-Json

$delay = 30
Write-Host "wait $delay seconds for service principal credential change"
Start-Sleep -Seconds $delay
```

### build and spin up container

Build container on __ACR__ (no local Docker required) and spin up in __ACI__. With that mount __Storage Account__ share to the container.

```PowerShell
az acr build -t $image -r $registryName .

az container create --name $containerGroupName `
    --image $image `
    --registry-login-server $registry.loginServer --registry-username $credentials.username --registry-password $credentials.passwords[0].value `
    --ip-address public `
    --azure-file-volume-account-name $storageAccountName `
    --azure-file-volume-account-key $storageAccountKey `
    --azure-file-volume-share-name $storageAccountShareName `
    --azure-file-volume-mount-path "/tmp/certificates" `
    --environment-variables DOMAIN=esb-dev.$dnsSuffix `
    --secure-environment-variables EMAIL=$EmailAddress SERVERURL=$ServerUrl EABKID=$EABKID EABHMACKEY=$EABHMACKEY
$containerGroup = az container show -n $containerGroupName -o json | ConvertFrom-Json
```

Sample configuration for __Docker__ and __nginx__ is based on this [post](https://geko.cloud/nginx-and-ssl-with-certbot-in-docker-alpine/).

#### Dockerfile

```Dockerfile
FROM nginx:alpine

RUN apk add python3 python3-dev py3-pip build-base libressl-dev musl-dev libffi-dev
RUN apk add gcc musl-dev openssl-dev cargo
RUN pip3 install pip --upgrade
RUN pip3 install certbot-nginx
RUN mkdir /etc/letsencrypt

COPY default.conf /etc/nginx/conf.d/default.conf
COPY challenge.sh /root/challenge.sh
RUN chmod u+x /root/challenge.sh
```

#### challenge.sh

> in an earlier version of this post, the sequence of commands was executed directly over `az container exec`. However when revisiting the scenario in summer 2021, this direct execution was not possible anymore - [see my StackOverflow post](https://stackoverflow.com/questions/68095848/azure-container-instance-command-execution-returns-no-such-file-or-directory).


```shell
##!/bin/sh
rm -rf /tmp/certificates/*
certbot register --email $EMAIL --server $SERVERURL --eab-kid $EABKID --eab-hmac-key $EABHMACKEY --agree-tos -n
certbot certonly --nginx --email $EMAIL --server $SERVERURL -d $DOMAIN
cp -avr /etc/letsencrypt /tmp/certificates
cp /var/log/letsencrypt/letsencrypt.log  /tmp/certificates/
```

#### default.conf

```
server {
    listen       80;
    server_name  _;

    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
    }

    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }
}
```

### modify Application Gateway and retrieve health

```PowerShell
az network application-gateway address-pool update `
    -n appGatewayBackendContainerPool `
    --gateway-name $appGwName `
    --set "backendAddresses[0].ipAddress=$($containerGroup.ipAddress.ip)"

$health = az network application-gateway show-backend-health -n $appGwName -o json | ConvertFrom-Json
$serverHealth = $health.backendAddressPools | ?{$_.backendAddressPool.id -match "appGatewayBackendContainerPool"} | %{$_.backendHttpSettingsCollection[0].servers}
$serverHealth
```

----

## 3. execute certificate request<a name="chapter-3"></a>

When __Application Gateway__ seems healthy, execute ACME server registration and certificate creation. Backup whole folder used during the process onto the __Storage Account__ share. These folder is then downloaded locally.

> In this sample, container group on __ACI__ is started with a public IP address. This is also possible with linking the container group to a virtual network with a private IP address. However the machine driving the process and submitting `az container exec` also needs to be linked to this virtual network then.

```PowerShell
if($serverHealth[0].health -eq "Healthy") {

    "=" * 80

    Read-Host "Hit enter to start certificate challenge"
    
    az container exec -n $containerGroupName --exec-command "/root/challenge.sh"
    
    "=" * 80
    
    Read-Host "Hit enter to download"
    
    if (Test-Path $downloadFolder -PathType Container) {
        Remove-Item $downloadFolder -Recurse -Force
    }
    New-Item $downloadFolder -ItemType Directory
    
    az storage file download-batch --account-name $storageAccountName --account-key $storageAccountKey -s $storageAccountShareName -d $downloadFolder
}
```

----

## 4. destroy temporary components

```PowerShell
Read-Host "Hit enter to clean up"

az container stop -n $containerGroupName

az container delete -n $containerGroupName --yes

az storage share delete --name $storageAccountShareName --account-name $storageAccountName
```

----

## variations

When working with a dedicated __Application Gateway__ it is possible to start & stop it at the beginning and the end of the process.

```PowerShell
az network application-gateway start --name $appGwName
```

```PowerShell
az network application-gateway stop --name $appGwName
```

## conclusion

I know, the depicted configuration might look very specific and some may wonder "Why all this hassle when there are services like Let's Encrypt?". However I hope that there is worthwhile information or some insights for you people out there. Please let me know what you think or where I could improve.