Virtual Network architecture 2 - Deployment pipelines
Published: 2022-08-29 14:38:00

# Summary

This article is Part.2 of virtual network architecture series. I will share the details of the entire pipeline in the sample template [api-management-vnet](https://github.com/koheikawata/api-management-vnet). 

- [Virtual Network architecture 1 - Do I need virtual network?](https://dev.to/koheikawata/virtual-network-architecture-1-do-i-need-virtual-network-1nhk)
- [Virtual Network architecture 2 - Deployment pipelines](https://dev.to/koheikawata/virtual-network-architecture-2-deployment-pipelines-39p5)
- [Virtual Network architecture 3 - Key Vault Private Endpoint](https://dev.to/koheikawata/virtual-network-architecture-3-key-vault-private-endpoint-43nj)
- [Virtual Network architecture 4 - SQL Database Private Endpoit](https://dev.to/koheikawata/virtual-network-architecture-4-sql-database-private-endpoit-49o2)
- [Virtual Network architecture 5 - App Service Private Endpoint](https://dev.to/koheikawata/virtual-network-architecture-5-app-service-private-endpoint-5eao)
- [Virtual Network architecture 6 - Service Bus Private Endpoint](https://dev.to/koheikawata/virtual-network-architecture-6-service-bus-private-endpoint-oa)
- [Virtual Network architecture 7 - Self-hosted agent](https://dev.to/koheikawata/virtual-network-architecture-7-self-hosted-agent-32j4)

# TOC
- [Overview](#overview)
- [Architecture](#architecture)
- [Pipelines](#pipelines)
  - [Two pipelines overview](#pipeline-overview)
  - [pipeline-base.yml](#pipeline-base)
  - [pipeline-vnet1.yml](#pipeline-vnet1)
  - [pipeline-vnet2.yml](#pipeline-vnet2)

# Overview <a name="overview"></a>

The sample template [api-management-vnet](https://github.com/koheikawata/api-management-vnet) includes two scenario - 1) Base architecture without virtual network, 2) Virtual network architecture. Both architectures use the same Azure services below.

**Used in both architectures**

- [Azure Pipelines](https://docs.microsoft.com/en-us/azure/devops/pipelines/get-started/what-is-azure-pipelines): Deploy infrastructure, build and deploy codes, build and deploy self-hosted agent, execute integration tests by sending REST API requests.
- [Azure API Management](https://docs.microsoft.com/en-us/azure/api-management/api-management-key-concepts): Platform to manage APIs behind which are running on Azure App Service in this template.
- [Azure App Service](https://docs.microsoft.com/en-us/azure/app-service/overview): Hosting service of web APIs.
- [Azure Functions](https://docs.microsoft.com/en-us/azure/azure-functions/functions-overview): Serverless hosting service that runs codes.
- [Azure SQL](https://docs.microsoft.com/en-us/azure/azure-sql/azure-sql-iaas-vs-paas-what-is-overview): SQL Server on the Azure cloud.
- [Azure Key Vault](https://docs.microsoft.com/en-us/azure/key-vault/general/overview): Secret management service that stores keys and certificates accessible from other Azure services.
- [Azure Service Bus](https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-messaging-overview): Message broker in the Azure cloud with message queue and pub-sub topics.

**Only in virtual network architecture**

- [Azure Application Gateway](https://docs.microsoft.com/en-us/azure/application-gateway/overview): A Web traffic load balancer that works as a gateway of virtual network architecture in this template.

# Architecture <a name="architecture"></a>

Both architectures have the same Web API, Functions, Database, and Service Bus topic. Azure Pipelines conducts the same integration tests that sends API requests to the Web API and Service Bus topic. There are two important differences in the virtual network architecture from the base architecture.

- Application Gateway resides in front of the API Management. API Management and App Service cannot be accessed from the internet but only through the Application Gateway. So, the API endpoint is different.

- Self-hosted agent is built in the first pipeline and deployed inside the virtual network subnet. The second pipeline runs on this self-hosted agent so it can access the private endpoints of App Service and Service Bus.

**Base architecture**

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ovbinoqu1gli5lp9ip83.png)

**Virtual network architecture**

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ilckahhhj7ue7qvomw9x.png)

# Pipelines <a name="pipelines"></a>

You can run the two pipelines by following [getting-started](https://github.com/koheikawata/api-management-vnet/blob/main/docs/Getting-started.md) documentation. In this article, I am going to explain the details of pipeline steps.

## Two pipelines overview <a name="pipeline-overview"></a>

Base architecture has only [pipeline-base.yml](https://github.com/koheikawata/api-management-vnet/blob/main/pipelines/pipeline-base.yml) but Virtual network architecture includes [pipeline-vnet1.yml](https://github.com/koheikawata/api-management-vnet/blob/main/pipelines/pipeline-vnet1.yml) and [pipeline-vnet2.yml](https://github.com/koheikawata/api-management-vnet/blob/main/pipelines/pipeline-vnet2.yml). The first pipeline of virtual network architecture deploys Azure resources without private endtpoints and builds a self-hosted agent, and the second pipeline runs on the self-hosted agent and deploys and sets up private points for resources. In the end of the second pipeline, it executes integration tests inside virtual network.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nzyy65sw2p87qncsp7nn.png)

## pipeline-base.yml <a name="pipeline-base"></a>

**Step 1.** In [keyvault-secret.yml](https://github.com/koheikawata/api-management-vnet/blob/main/pipelines/templates/keyvault-secret.yml), the pipeline creates Key Vault if it does not exist and pass secrets input from Azure Pipelines variables so you do not need to keep those secrets in the Azure Pipeline variables but after the second time the pipeline automatically detects pipeline variable and download secrets from existing Key Vault. This technique is described in [Azure Pipelines secret management with Key Vault](https://dev.to/koheikawata/azure-pipelines-secret-management-with-key-vault-3e01).

**Step 2.** [azure-resource.yml](https://github.com/koheikawata/api-management-vnet/blob/main/pipelines/templates/azure-resource.yml) deploys Azure resources with [main.bicep](https://github.com/koheikawata/api-management-vnet/blob/main/src/bicep/base/main.bicep).

  - You need to specify the secret variables handled in the previous Key Vault step. If you have more secret variables for improvement, you need to add variables here.

```
- job: AzureResourceDeployment
    variables:
    - template: ./variables.yml
    - name: SqlPass
      value: $[ stageDependencies.KeyVault.SQL_PASS.outputs['SQL_PASS.SQL_PASS'] ]
    - name: AadSecretClient
      value: $[ stageDependencies.KeyVault.AAD_SECRET_CLIENT.outputs['AAD_SECRET_CLIENT.AAD_SECRET_CLIENT'] ]
```

  - Azure API Management takes around one hour to deploy. In order to avoid a time out error, it specifies `timeout: 0`. However, I found errors sometimes after one hour. If you face a timeout error, you can just re-run the pipeline, and you will see everything goes smoothly after that.

```
- job: AzureResourceDeployment
    timeoutInMinutes: 0
```

**Step 3.** [sql.yml](https://github.com/koheikawata/api-management-vnet/blob/main/pipelines/templates/sql.yml) deploys SQL Database schema with `.dacpac` file built by [WeatherDB](https://github.com/koheikawata/api-management-vnet/tree/main/src/apps/WeatherDb) SQL project.

**Step 4.** [func.yml](https://github.com/koheikawata/api-management-vnet/blob/main/pipelines/templates/func.yml) deploys the .NET C# code of [ServicebusListener](https://github.com/koheikawata/api-management-vnet/tree/main/src/apps/ServicebusListener) to Azure Functions.

**Step 5.** [webapi.yml](https://github.com/koheikawata/api-management-vnet/blob/main/pipelines/templates/webapi.yml) deploys the .NET C# code of [WeatherAPI](https://github.com/koheikawata/api-management-vnet/tree/main/src/apps/WeatherAPI) project to Azure App Service. Furthermore, it defines the API in the API Management and generates `swagger.json` and deploys it to the API Management.

```
- task: DotNetCoreCLI@2
  displayName: dotnet new tool-manifest
  inputs:
    command: custom
    custom: new
    arguments: tool-manifest
    workingDirectory: ${{ parameters.apiProjectDirectory }}
- task: DotNetCoreCLI@2
  displayName: dotnet tool install
  inputs:
    command: custom
    custom: tool
    arguments: install Swashbuckle.AspNetCore.Cli --version $(SWASHBUCKLE_VERSION)
    workingDirectory: ${{ parameters.apiProjectDirectory }}
- task: DotNetCoreCLI@2
  displayName: dotnet swagger tofile
  inputs:
    command: custom
    custom: swagger
    arguments: tofile --output ${{ parameters.swaggerPath }} ${{ parameters.apiDllPath }} $(SWAGGER_VERSION)
    workingDirectory: ${{ parameters.apiProjectDirectory }}
- task: AzureCLI@2
  displayName: Deploy API to API Management
  inputs:
    azureSubscription: ${{ parameters.azureSvcName }}
    scriptType: bash
    scriptLocation: inlineScript
    inlineScript: |
      az apim api import -g $(RESOURCE_GROUP_NAME) \
        --service-name $(API_MANAGEMENT_NAME) \
        --api-id $(API_NAME) \
        --path /$(API_NAME) \
        --specification-format OpenApiJson \
        --specification-path ${{ parameters.swaggerPath }} \
        --service-url $(API_MANAGEMENT_SERVICE_URL)
```

**Step 6.** [integration-test-powershell.yml](https://github.com/koheikawata/api-management-vnet/blob/main/pipelines/templates/integration-test-powershell.yml) executes integration tests for deployed Web API and Functions.

  - With the powershell script below, it extracts OAuth2.0 token from Azure Active Directory so it can attach the token later when sending a REST request to the API Management. Because the Azure Pipelines agent sends a request with automation, it follows Client Credential Flow of OAuth2.0. This OAuth2.0 flow is explained in [Azure AD app configuration for Web API](https://dev.to/koheikawata/azure-ad-app-configuration-for-web-api-io2).

```
$clientSecret = Get-AzKeyVaultSecret -VaultName $env:KEYVAULT_NAME -Name $env:KVSECRET_NAME_AADCLIENT -AsPlainText
$authorizeUri = "https://login.microsoftonline.com/${{ parameters.aadTenantId }}/oauth2/v2.0/token"
$body = 'grant_type=client_credentials' + `
'&client_id=${{ parameters.aadAppidClient }}' + `
'&client_secret=' + $clientSecret + `
'&scope=api://${{ parameters.aadAppidBackend }}/.default'
$token = (Invoke-RestMethod -Method Post -Uri $authorizeUri -Body $body).access_token
```

  - The base architecture does not have Application Gateway and the endpoint is at the API Management. The route `/Weatherforecast` is to GET and POST the data to Azure SQL Databse and `/Weatherforecast/message` to send a message to Service Bus topic, which triggers Azure Functions and inserts that record in SQL Database.

```
parameters:
  url: https://$(API_MANAGEMENT_NAME).azure-api.net
```

```
$Uri = "${{ parameters.url }}/$env:API_NAME/Weatherforecast"
$UriMessage = "${{ parameters.url }}/$env:API_NAME/Weatherforecast/message"
```

## pipeline-vnet1.yml <a name="pipeline-vnet1"></a>

[pipeline-vnet1.yml](https://github.com/koheikawata/api-management-vnet/blob/main/pipelines/pipeline-vnet1.yml) includes steps below.

- Store secrets from Pipeline variables to Key Vault
- Generate SSL certificate for Application Gateway backend pool
- Deploy Azure services
- Build a self-hosted agent
- Deploy projects to App Service, Functions, and SQL Database
- Execute integration tests

After this pipeline is executed, some resources such as App Service, Key Vault, SQL Database, Functions, and Service Bus are not protected within the virtual network. Only Application Gateway and API Management are connected to and located in the virtual network. The self-hosted agent is built and deployed to Azure Container Instances and ready for usage.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xlxk7oevr6c9tit15v7v.png)

**Step 1.** [cert-appgateway.yml](https://github.com/koheikawata/api-management-vnet/blob/main/pipelines/templates/cert-appgateway.yml) generates and sets in Key Vault a SSL certificate for the backend pool of API Management. The Key Vault requires to grant the access right for Azure Pipeliens Microsoft hosted agent.

```
Set-AzKeyVaultAccessPolicy -VaultName $env:KEYVAULT_NAME -PermissionsToCertificates get,create -ObjectId ${{ parameters.aadObjectidSvc }}
```

**Step 2.** [keyvault-secret.yml](https://github.com/koheikawata/api-management-vnet/blob/main/pipelines/templates/keyvault-secret.yml) is the same template as used in the Base architecture. This time it adds `AZP_TOKEN` secret variable that is needed to deploy a self-hosted agent. How to get the personal access token is described in [Getting-started](https://github.com/koheikawata/api-management-vnet/blob/main/docs/Getting-started.md#personal-access-token-for-self-hosted-agent).

**Step 3.** [azure-resource.yml](https://github.com/koheikawata/api-management-vnet/blob/main/pipelines/templates/azure-resource.yml) deploys Azure resources with [main1.bicep](https://github.com/koheikawata/api-management-vnet/blob/main/src/bicep/vnet/main1.bicep). All the virtual networks and subnets are deployed this time with [VirtualNetwork.bicep](https://github.com/koheikawata/api-management-vnet/blob/main/src/bicep/vnet/modules/VirtualNetwork.bicep). All the virtual network IP address ranges are defined in [main.parameters.json](https://github.com/koheikawata/api-management-vnet/blob/main/src/bicep/vnet/main.parameters.json).

**Step 4.** [restart-appgateway.yml](https://github.com/koheikawata/api-management-vnet/blob/main/pipelines/templates/restart-appgateway.yml) restarts Application Gateway. I faced errors without restarting Application Gateway. According to the Microsoft documentation [Updates to the DNS entries of the backend pool](https://docs.microsoft.com/en-us/azure/application-gateway/application-gateway-backend-health-troubleshooting#updates-to-the-dns-entries-of-the-backend-pool), when you change DNS entries for the backend pool, Application Gateway must be restarted. In the Azure resource deployment with [PrivateDns1.bicep](https://github.com/koheikawata/api-management-vnet/blob/main/src/bicep/vnet/modules/PrivateDns1.bicep), it creates the DNS A record for the API Management. Then you need to restart the Application Gateway.

```
resource PrivateDnsAApim 'Microsoft.Network/privateDnsZones/A@2020-06-01' = {
  name: '${PrivateDnsApim.name}/${ApiManagementName}'
  properties: {
    ttl: 3600
    aRecords: [
      {
        ipv4Address: ApiManagementPrivateIPAddress
      }
    ]
  }
}
```

**Step 5.** [self-hosted-agent.yml](https://github.com/koheikawata/api-management-vnet/blob/main/pipelines/templates/self-hosted-agent.yml) builds and deploys a Linux based self-hosted agent.

- As of August 2022, you can run a Windows based self-hosted agent in the docker container but Windows based Azure Container Instance cannot be deployed on the virtual network, according to the Microsoft documentation [Deploy container instances into an Azure virtual network](https://docs.microsoft.com/en-us/azure/container-instances/container-instances-vnet). Windows based self-hosted agent is needed to deploy SQL Database projects, but for now the Windows self-hosted agent build part is commented out.

- The self-hosted agent container application is defined in [Dockerfile](https://github.com/koheikawata/api-management-vnet/blob/main/src/self-hosted-agent/linux/Dockerfile).

- After the container image is pushed into Azure Container Registry, the deployment to Azure Container Instance is implemented in [linux-agent.bicep](https://github.com/koheikawata/api-management-vnet/blob/main/src/bicep/self-hosted-agent/linux-agent.bicep).

**Step 6.** [sql.yml](https://github.com/koheikawata/api-management-vnet/blob/main/pipelines/templates/sql.yml) is the same as used in the Base pipeline and deploys SQL Database schema with `.dacpac` file built by [WeatherDB](https://github.com/koheikawata/api-management-vnet/tree/main/src/apps/WeatherDb) SQL project.

**Step 7.** [func.yml](https://github.com/koheikawata/api-management-vnet/blob/main/pipelines/templates/func.yml) is the same as used in the Base pipeline and deploys the .NET C# code of [ServicebusListener](https://github.com/koheikawata/api-management-vnet/tree/main/src/apps/ServicebusListener) to Azure Functions.

**Step 8.** [webapi.yml](https://github.com/koheikawata/api-management-vnet/blob/main/pipelines/templates/webapi.yml) is the same as used in the Base pipeline and deploys the .NET C# code of [WeatherAPI](https://github.com/koheikawata/api-management-vnet/tree/main/src/apps/WeatherAPI) project to Azure App Service and defines the API in the API Management and generates `swagger.json` and deploys it to the API Management.

**Step 9.** [integration-test-powershell.yml](https://github.com/koheikawata/api-management-vnet/blob/main/pipelines/templates/integration-test-powershell.yml) is the same as used in the Base pipeline and executes integration tests for deployed Web API and Functions.

## pipeline-vnet2.yml <a name="pipeline-vnet2"></a>

[pipeline-vnet2.yml](https://github.com/koheikawata/api-management-vnet/blob/main/pipelines/pipeline-vnet2.yml) includes steps below.

- Execute integration tests before setting private endpoints
- Deploy and set private endpoints
- Execute integration tests for private endpoints
- Deploy projects to App Service and Functions with private endpoints by the self-hosted agent
- Execute integration tests for code deployment through private endpoints.

This pipeline deploys and set private endpoints so all Azure services are protected by the virtual network. You cannot deploy new software to the App Service and Functions any more as they are not allowed to access through the Internet. The self-hosted agent running on Azure Container Instance is located inside the virtual network and capable of building and deploying the new software.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pv7fuvxrpgk2zu47oews.png)

You can see this pipeline use the self-hosted agent by specifying the pool like below. All steps below can be executed on a Linux based agent.

```
pool:
  name: Default
```

**Step 1.** [integration-test-bash.yml](https://github.com/koheikawata/api-management-vnet/blob/main/pipelines/templates/integration-test-bash.yml) executes integration tests before private endpoints are deployed and set up. The self-hosted agent is Linux based, and this integration test is written in bash scripts. Obtaining Azure Active Directory OAuth2.0 token by bash script is written in the way below.

```
clientSecret=$(az keyvault secret show --vault-name $(KEYVAULT_NAME) --name $(KVSECRET_NAME_AADCLIENT) --query value -o tsv)
authorizeUri="https://login.microsoftonline.com/${{ parameters.aadTenantId }}/oauth2/v2.0/token"
token=$(curl -X POST $authorizeUri -d "client_id=${{ parameters.aadAppidClient }}&grant_type=client_credentials&scope=api://${{ parameters.aadAppidBackend }}/.default&client_secret=$clientSecret" | jq ".access_token" | tr -d \")
```

**Step 2.** [private-endpoints.yml](https://github.com/koheikawata/api-management-vnet/blob/main/pipelines/templates/private-endpoints.yml) deploys Azure resources related to private endpoints with [main2.bicep](https://github.com/koheikawata/api-management-vnet/blob/main/src/bicep/vnet/main2.bicep). All the virtual network IP address ranges are defined in [main.parameters.json](https://github.com/koheikawata/api-management-vnet/blob/main/src/bicep/vnet/main.parameters.json).

**Step 3.** [restart-appgateway.yml](https://github.com/koheikawata/api-management-vnet/blob/main/pipelines/templates/restart-appgateway.yml) restarts Application Gateway. This should be executed for the same reason as the V-net pipeline 1 because private DNS A records for Key Vault, App Service, SQL Database, Funtions, and Service Bus are added.

**Step 4.** [integration-test-bash.yml](https://github.com/koheikawata/api-management-vnet/blob/main/pipelines/templates/integration-test-bash.yml) executes integration tests after private endpoints are set. This step can make sure all the configurations of private endpoints are correctly deployed.

**Step 5.** [func.yml](https://github.com/koheikawata/api-management-vnet/blob/main/pipelines/templates/func.yml) is the same as used in the Base pipeline but this time is executed on the self-hosted agent. Functions do not have accessible public IP any more, and it should be connected through the private IP inside the virtual network.

**Step 6.** [webapi.yml](https://github.com/koheikawata/api-management-vnet/blob/main/pipelines/templates/webapi.yml) is the same as used in the Base pipeline but this time is executed on the self-hosted agent. The App Service does not have accessible public IP any more, and it should be connected through the private IP inside the virtual network.

**Step 7.** [integration-test-bash.yml](https://github.com/koheikawata/api-management-vnet/blob/main/pipelines/templates/integration-test-bash.yml) executes integration tests to make sure the code deployment to App Service and Funtions are correctly done through the self-hosted agent.