Replacing self-hosted agents with ephemeral pipelines agents in Azure DevOps
Tags: azure, devops, tutorial, pipelines
Published: 2020-03-15 17:00:09

---
title: Replacing self-hosted agents with ephemeral pipelines agents in Azure DevOps
published: true
date: 2020-03-15 17:00:00 UTC
description: Sometimes we are in need of using self-hosted agents in our pipelines due to our Azure Resources not being exposed on the internet, and only accessible via a private network, because we are excluded from using Microsoft-hosted agents, and we need to maintain a pool of self-hosted agents. In this case, Ephemeral Pipelines Agents could be our solution to get rid of self-hosted agents (for deployment purposes).
tags: azure, devops, tutorial, pipelines
canonical_url: https://piraces.dev/posts/azure-ephemeral-pipelines-agents
cover_image: https://dev-to-uploads.s3.amazonaws.com/i/9al00ptqmijm4bqhed7w.png
---

![Cover image](https://piraces.dev/static/2a8a8400575881c6955625977dcc41fa/d587d/ephemeral_agents_schema.png)
<figcaption>Schema of sample architecture using ephemeral agents</figcaption>


If you have Azure Resources that aren’t exposed on the internet but only accessible via a private network, you can’t use [Microsoft-hosted agents](https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops) because they can’t connect to the private network. Therefore, we need to maintain a pool (or several pools) of [self-hosted agents](https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=azure-devops&tabs=browser#install), with the associated costs and effort to maintain that pool(s).

In this case it is where **Ephemeral pipelines agents come into action**.

# [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#overview)Overview

[Ephemeral pipelines agents](https://github.com/microsoft/azure-pipelines-ephemeral-agents) come to eliminate the need to use and maintain pools of self-hosted agents for deployment purposes.This type of agents are capable to deploy to private azure resources. The process is not very complex. Ephemeral pipelines agents run in an [Azure Container Instance (ACI)](https://azure.microsoft.com/en-us/services/container-instances/) with access to the private network where the other Azure resources are. The agents are created to run a pipeline job and then deleted to avoid extra costs and resources consumption.

This way, we can deploy to private Azure resources without having to expose them on the internet or having to maintain self-hosted agents on the same (or with access) virtual network (with their associated costs and cons).

The agent that runs in the ACI its only a basic Docker image with the necessary dependencies to run the agent and to be capable to deploy to our private resources. For example, a base Ubuntu image with the necessary dependencies and the [deploy agent](https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=azure-devops&tabs=browser) installed.

* * *

**TL;DR:** The purpose of [this task](https://marketplace.visualstudio.com/items?itemName=tiago-pascoal.EphemeralPipelinesAgents) is to create a short-lived Azure Pipelines Agent to run a deploy in a private virtual network so we can deploy to Azure Resources that are not internet accessible.

⚠️ **Important:** This approach (and task) is currently in preview and has [known issues and limitations](https://github.com/microsoft/azure-pipelines-ephemeral-agents#known-issueslimitations). Please take this into account before proceeding to use this process.

# [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#how-does-the-process-works)How does the process works

Only three steps/requirements:

1. One docker image that can run a deploy agent in a container (needed to provision agents).
2. Provision one ephemeral pipeline agent to run the deployment job. This process can be done using [this task](https://marketplace.visualstudio.com/items?itemName=tiago-pascoal.EphemeralPipelinesAgents). This task provisions, configures and registers an agent on an ACI using the docker image mentioned in the first step.
3. The container runs one pipeline job, and then it unregisters the agent and deletes the container (it self destructs). 

So considering we have setup a docker image that can run a deploy agent in a container, our pipeline will look like this:

![Example pipeline with ephemeral agents](https://piraces.dev/static/2b02e402758e0e9c7385d763649ccfbb/b5cea/release_sample.png)
<figcaption>Example pipeline with ephemeral agents</figcaption>


As you can see, two jobs are needed:

- The first one provisions the agent in one pool to run the deployment job.
- The second one (depending on the first being finished correctly), runs the deployment job. It runs on the agent that the first job has provisioned.

# [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#tutorial)Tutorial

In this post I will guide you through a simple tutorial on how to deploy assets on a container in an Azure Storage account in a private virtual network, using Ephemeral pipeline agents.

So let’s get started!

## [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#requirements)Requirements:

- A virtual network with a security group.
- A dedicated subnet in the virtual network to run the ephemeral agents.
- The agent must run **in the same Azure location** as the virtual network.
- All the created subnets must share the same security group.

## [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#overview-of-the-resources)Overview of the resources

For this tutorial I have created two resource groups:

- One for the virtual network and the security group.
- Another one for the Azure Container Registry (ACR) and the storage we will deploy to.

![Resource group for vnet and security group](https://piraces.dev/static/e70722777f22370d2bc198aacb9a8b51/29114/azure_sample_vnet.png)
<figcaption>Resource group for the VNet and security group</figcaption>


![Resource group for ACR and Storage account](https://piraces.dev/static/4d5a0b0f669ce04c1947ccc2a5dd0381/29114/azure_resource_group_sample.png)
<figcaption>Resource group for the ACR and Storage account</figcaption>


Also, the Azure Storage account should be in the private network, with a configuration similar to this one:

![Azure Storage Account VNet configuration](https://piraces.dev/static/5043b3fcbd0b06c6e70f1f9321590976/29114/vnet_storage_account.png)
<figcaption>Azure Storage Account VNet configuration</figcaption>


If you need it, the main repo of ephemeral agents has [sample scripts](https://github.com/microsoft/azure-pipelines-ephemeral-agents/tree/master/Samples) on how to deploy these resources.

## [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#pushing-the-base-image-for-ephemeral-agents)Pushing the base image for ephemeral agents

Once finished configuring our resources in Azure, we will need the base image to use with the ephemeral agents.In this case I have used the [Agent Images available in the GitHub repo](https://github.com/microsoft/azure-pipelines-ephemeral-agents/tree/master/AgentImages), specifically the Ubuntu one. We can use the pipeline in the GitHub repo to deploy our Agent image in the ACR (using an [ACR Service connection](https://docs.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=azure-devops&tabs=yaml#sep-docreg)).

After that, we should have one repository with one image in our ACR:

![ACR with Agent Image](https://piraces.dev/static/0dbbb8a4515bcfff00490c9bfc2eb7b7/29114/azure_acr_agent_image.png)
<figcaption>ACR with sample Agent Image based on Ubuntu</figcaption>



## [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#setting-up-the-pipeline-in-azure-devops)Setting up the pipeline in Azure DevOps

After creating the needed resources and pushing the Agent Image to the ACR, we are ready to create our pipeline and use ephemeral agents to deploy.

First of all, we will need to create an additional agent pool in our Project, and grant access to all pipelines.

### [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#permissions)Permissions:

For this process we will need to give specific permissions in order to allow the pipeline to register agents in the new pool, running jobs in these agents and unregister them.

In order to register the agent a token with sufficient permissions on the Azure DevOps Organization the agent is going to be registered is needed.

You can use two different types of tokens: a **Personal Access Token** or an **OAuth token (recommended)**.

- If using an Personal Access Token (PAT), it requires the _Agent Pools Read & Manage scope_ and that the user who owns the PAT has administration privileges on the agent pool you intend to register the agent.
- If using an OAuth token, the best approach is to use **[System.AccessToken](https://docs.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=azure-devops&tabs=yaml#systemaccesstoken)** which are short lived, dynamic and automatically managed by the system.

If using the OAuth token, you need to met the following conditions:

- The timeout for the job that creates the agent only expires after the deploy job in the agent is finished (because the token is used to register and unregister the agent in the pool).
- The account `Project Collection Build Service (YOUR_ORG)` needs to have administration permissions on the pool (granted at the organization level, not at team project level).

In this example, permissions on the pool (for the OAuth token approach) will look like the following:

![Pool permissions for OAuth token](https://piraces.dev/static/f4ee270cd5f5526ad0732206f549c80e/29114/pool_permissions.png)
<figcaption>Pool permissions for OAuth token</figcaption>



#### [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#additional-considerations)Additional considerations:

You must register an Azure Resource provider for the namespace `Microsoft.ContainerInstance` in order to create container instances for the agents. This can be done easily opening a Powershell instance in the Azure portal and executing the following command:

```
Register-AzureRmResourceProvider -ProviderNamespace 'Microsoft.ContainerInstance'
```

This will register the needed Azure Resource provider.

### [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#service-connections)Service connections:

We will need a Service connection using a Service principal, which has to have access to the VNet resource group and the ACR and Storage Account resource group. We can set it up in the project settings in Azure DevOps, granting permissions to one resource group and then going to the Azure Portal to grant permissions to the another one. Make sure to grant permissions to all pipelines.

Also, as mentioned above we will need an ACR Service Connection to our ACR to run the ephemeral agents in the ACI from the Agent Image in the ACR (granting permissions to all pipelines).

### [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#pipeline-definition)Pipeline definition:

In this part of the process two pipelines are created for my sample project, one for creating the base image for the agents in the ACR (which we have previously created), and another one for the main deploy process of our project.

![Two main pipelines defined](https://piraces.dev/static/f8e4d06f54566aa36a22eb55d66441a0/29114/two_pipelines.png)
<figcaption>Pipelines defined in our projects</figcaption>



The main deploy pipeline used in this tutorial is the same defined in the [GitHub repo sample](https://github.com/microsoft/azure-pipelines-ephemeral-agents/blob/master/Samples/storage-pipeline/Azure-pipelines.yml). As we have seen in the previous stages, we have to define two jobs: one to provision the agent and another one to perform the deploy job.

Configure the variables in the sample pipeline to reference your resources correctly and then try to run it:

![Pipeline preparing the agent in the first job](https://piraces.dev/static/14de6ef3b45893151e81670437ce31f5/29114/pipeline_running_prepare_agent.png)
<figcaption>Pipeline preparing the agent in the first job</figcaption>



If we have configured everything correctly, the pipeline will succeed:

![Final pipeline result](https://piraces.dev/static/d7e2f6b86afb845e8b6b58797181282b/29114/final_pipeline_result.png)
<figcaption>Final pipeline result</figcaption>


If we inspect our created agent pool, we will see the executed job but no agents registered in the pool (which is the main purpose of this process). This is because the agent has unregistered itself when the deployment job finished.

![Pool executed job](https://piraces.dev/static/a0dc1e6946e8b466d84776329838d9b2/29114/pool_results.png)
<figcaption>Pool executed job</figcaption>



### [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#final-result)Final result:

After the deploy pipeline execution, we can check how we successfully have deployed our assets to the Azure Storage container connected to the private virtual network:

![Deployed assets on Azure Storage container inside the VNet](https://piraces.dev/static/67e51f008539ac42eaccb8f520e47039/29114/azure_final_result_container.png)
<figcaption>Deployed assets on Azure Storage container inside the VNet</figcaption>



# [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#conclusion)Conclusion

We have seen how ephemeral pipelines agents works, how to replace self-hosted agents with ephemeral ones and the most important part: how to reduce maintenance costs of having our own self-hosted agents pool(s).

It is important to emphasize that this process its currently in preview, so maybe some things just do not work out of the box, but I personally think its an excellent approach to avoid having self-hosted agents in the mentioned situations.

**Happy deploy!** 🎉🎉

Sources:

1. [GitHub of Microsoft Azure Pipelines ephemeral agents. (2020, March 15)](https://github.com/microsoft/azure-pipelines-ephemeral-agents)
2. [Microsoft docs: Azure Pipelines agents. (2020, March 15)](https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=azure-devops&tabs=browser)
3. [Ephemeral Pipelines Agents extension. (2020, March 15)](https://marketplace.visualstudio.com/items?itemName=tiago-pascoal.EphemeralPipelinesAgents)