Deploy Azure DevOps Self-Hosted Build Agents on Kubernetes (AKS) and scale them using KEDA
Tags: kubernetes, azure, devops, terraform
Published: 2022-06-22 14:07:17

## Overview
 

KEDA is an event driven autoscaler for Kubernetes that allows you to scale containers based on events. 

It is a lightweight single purpose component that can be added to any Kubernetes cluster. Keda can also work alongside the Kubernetes horizontal autoscaler. 

The diagram below (taken from the Keda docs) shows how Keda integrates with the horizontal pod autoscalers, external events and the Kubernetes API:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/13ctfb8w8a65uyzaxkzo.png)
 
More information on Keda can be found in the official [docs](https://keda.sh/docs/2.7/concepts/#architecture)

In this tutorial will be going over how you can use deploy self-hosted build agents on to an Azure Kubernetes Cluster and scale them using KEDA ([scaled jobs](https://keda.sh/docs/2.3/concepts/scaling-jobs/)) based on the number of jobs in a build queue. 

As a side note, Azure Container Apps also supports KEDA and scaling using the pipelines trigger however KEDA kills containers half through a job and scaling jobs are not yet supported in Container Apps.

All of the code for this project can be found on my Github page [here](https://github.com/lakovmartin/azure/tree/main/kubernetes/keda-devops-agents)

## Tools Required

We will be using the following tools so make sure you have them installed on your local machine. 

- Helm (Kubernetes Package Manager) - Install [Guide](https://helm.sh/docs/intro/install/
)
- Azure CLI - Install [Guide](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli)
- Kubernetes Command Line - Install [Guide](https://docs.microsoft.com/en-us/cli/azure/aks#az-aks-install-cli)
- Terraform (Infrastructure as Code) - Install [Guide](https://learn.hashicorp.com/tutorials/terraform/install-cli) 
- Docker Desktop - Install [Guide](https://www.docker.com/products/docker-desktop/) 

## Deploy Azure resourcse
Before we get started we need to deploy the Azure components that will host our solution. These are the following:

- Azure Resource Group
- AKS Cluster
- Container Registry

The Kubernetes cluster will be basic with a single pool and single virtual machine. No advanced networking will be used for the purposes of proving the concept.

We will be deploying this through Terraform which is an infrastructure as code deployment tool. The Terraform deployment file can be found in my Git repository here.


**Step 1:** Fork the repository and create a local clone on your machine 

**Step 2:** Navigate to the [folder](https://github.com/lakovmartin/azure/tree/main/kubernetes/keda-devops-agents/iac) that contains the `main.tf` file and run: 

```terraform
Terraform Init 
```
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n6hu7rul4y0nnstr59df.png)

This will initialise Terraform and create the local state file.  

**Step 3:** Now we are going to run Terraform plan which will list the resources that will be deployed. It should be 4 in total as listed above (1 resource is a role assignment to link AKS to ACR).

```terraform
Terraform Plan 
```
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9lp98rdxveo43ymy4qcw.png)
 
**Step 4:** Once we are happy with what Terraform is going to deploy we can run the apply stage which will deploy the resources into Azure. 

```terraform
Terraform Apply 
```
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4z188ixhxnc0hqiaihlh.png)
 
You should now see the resources in the Azure portal:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d2g3gfu5kbyv3t2yl9mo.png)
 

## Build the docker images

Next we need to build the Docker image for the Azure DevOps self-hosted agents. Microsoft have documented this quite for Docker [here](https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/docker?view=azure-devops). I have modified the image slightly to include PowerShell and we are using run.sh rather than docker in the start.sh script as ContainerD is the new container runtime in AKS version 1.19 and higher. You can re-use my images for these next steps from my Github [repo](https://github.com/lakovmartin/azure/tree/main/kubernetes/keda-devops-agents).

**Step 1:** Start up docker Desktop

**Step 2:** Navigate to the repository you cloned earlier and navigate to the [folder](https://github.com/lakovmartin/azure/tree/main/kubernetes/keda-devops-agents/images) that contains the docker image.

**Step 3:** Let's build the image and tag it: 

```docker
Docker build -f <docker-image-path> -t <tag> 
```
You can test the image by running the container locally which will register it in DevOps. Just supply the environment variables.

```docker
docker run -e AZP_URL= -e AZP_TOKEN= -e AZP_AGENT_NAME= -e AZP_POOL= <image>
```

**Step 4:** Let's login to the Azure Container Registry.

```docker
Docker login <login-server> -u <username> -p <password>
```
You can get the above parameters from the container registry in the portal: 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/htvgtvbygptc4tea0rlv.png)

**Step 5:** Let's push the image to the Azure Container Registry

```docker
Docker push <imagename:tag>
```
The image should now be in the container registry which allow the containers running the agents to pull it. 

## Install KEDA on to the AKS cluster

KEDA runs in a container on the Kubernetes cluster and it's not built in so we need to install it. The KEDA pod handled all of the event driven scaling. We are going to use Helm to the install however you can apply the manifests [directly](https://keda.sh/docs/2.7/deploy/). 

Make sure you are authenticated to thee AKS cluster before running these next steps.

**Step 1:** Add the helm repo 

```helm
helm repo add kedacore https://kedacore.github.io/charts
```

**Step 2:** Update the helm repo

```helm
helm repo update
```

**Step 3:** Create a new namespace and install the KEDA helm chart

```helm
kubectl create namespace keda
helm install keda kedacore/keda --namespace keda
```
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cb1ljatgvx23yj5284v6.png)

You should now see the KEDA pods running in the keda namespace: 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9209wg7co14ywymamp5w.png)

We are now ready to start using KEDA for scaling our containers.   

## Deploying the agents

**Set up a new Agent Pool in Azure DevOps**

Before we apply the Kubernetes manifests we need to set up a new agent pool in DevOps.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gbpj2lf4mini9lymbscn.png)

**Step 1:** Set up a self hosted pool on the organisation level. Take a note of the pool id. This can be found in the URL when you select the pool:

https://dev.azure.com/organisation/_settings/agentpools?poolId=16&view=jobs

In this case it's 16.  

**Step 2:** You will also need to generate a PAT [token](https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops&tabs=Windows) with Agent Pools read & manage permissions.

**Step 3:** You will need to encode the token to Base64. You can do this either through Bash or through this [website](https://www.base64encode.org/). 

**Apply the manifests**

Because we are going to be using scaling jobs, we cannot specify idle agents. This creates a problem as you can't queue an Azure pipelines job on an empty agent pool so a workaround is to deploy a static agent and turn it offline. 

Apply the `deployment.yaml` from the cloned [repository](https://github.com/lakovmartin/azure/tree/main/kubernetes/keda-devops-agents/images) by running: 

```
kubectl apply -f ./deployment.yaml
```
Make sure you replace the variables in the YAML for the agent pool name, organisation URL, image and PAT token. See examples below: 

- image: = `<acr-name>.azurecr.io/<repository-name>:<tag>`
- AZP_URL = `https://dev.azure.com/<organisation-name>`
- AZP_TOKEN = Base64 encoded token that we created earlier.
- AZP_POOL = Name of the self-hosted agent pool in Azure DevOps

Once you have applied the manifest you should see the pod running in the default namespace:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jkslp7n16y2gs5pfanlt.png)

We can now see the agent running in DevOps:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/g1p4jggl4dxyqou24s1q.png)

Disable the agent and leave it running.

## Apply the KEDA Scale Job Manifest

The final step is to apply the scaled job object through our `keda-scaled-jobs.yaml` manifest. You will need to replace the values for the image, organisation, pool, token, and pool ID.

```shell
kubectl apply -f ./keda-scaled-jobs.yaml
```


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l1nuwb5jadxuyi9et1c4.png)
 
Now that we have applied the scale object, KEDA will be listening to the build queue every 10 seconds (can be customised). I have included a load testing [pipeline](https://github.com/lakovmartin/azure/blob/main/kubernetes/keda-devops-agents/test/load-test.yaml) for Azure DevOps that triggers 10 jobs which run some PowerShell loops. Let's run some jobs and see the scaling in action: 



Pods are spinning up:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4e2p6w6fg98htltsv19t.png)

Agents are coming online:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/f0f8ykgg3ha68muc92i9.png)

We have now configured our self-hosted agents to run in Docker using Kubernetes as the orchestrator and KEDA as the scaling engine!  

You can customise KEDA on how often it monitors the job queue, maximum number of replicas and more. 

## Limitations

I have discovered some limitations with Keda and scaling jobs which I'm working on resolving:

- If you run some jobs in DevOps and cancel them the containers keep running.
- KEDA does not remove any pods in a "completed" state. This can be resolved with a custom clean up shell script running as a cron on the cluster however it may be possible to do it with KEDA.
- Offline agents are not removed from the DevOps pool. I have a cron job running in Azure DevOps that cleans this up.

## Next Steps

All of the above steps can of course be automated through a simple build pipeline either through Github actions or Azure DevOps pipelines. 

I will be building all of this functionality into a Helm chart that will also support Github runners so keep an eye out!