Monitoring Azure Container Apps With Azure Managed Grafana
Tags: azure, containerapps, grafana, terraform
Published: 2022-09-21 19:54:57

---
title: Monitoring Azure Container Apps With Azure Managed Grafana
published: true
date: 2022-09-21 13:00:00 UTC
tags: #azure #containerapps #grafana #terraform
canonical_url: https://paulyu.dev/article/monitoring-azure-container-apps-with-azure-managed-grafana/
---

The Azure Monitor team has [announced the general availability of the Azure Managed Grafana (AMG) service](https://azure.microsoft.com/blog/elevate-your-visualizations-with-azure-managed-grafana-now-generally-available/). As part of the announcement, they also announced the availability of curated [Grafana dashboards](https://grafana.com/orgs/azuremonitorteam/dashboards) for various Azure services including [Azure Container Apps](https://grafana.com/grafana/dashboards/16592-azure-container-apps-container-app-view/) 🎉

[Grafana](https://grafana.com/) is very popular within the Cloud Native community and it seems natural to use it for Azure Container Apps (ACA) observability.

In this post, I will walk you through provisioning the ACA and AMG resources using the [Terraform AzAPI](https://registry.terraform.io/providers/Azure/azapi/latest/docs) provider and show you how easy it is to import the ACA dashboards into your AMG instance.

Let's go!

## Prerequisites

Before we begin, make sure you have the following:

- [Terraform CLI](https://www.terraform.io/downloads)
- [Azure CLI](https://aka.ms/azure-cli)
- [Azure Subscription](https://aka.ms/freeazure)

## Terraform project setup

In your favorite terminal window, create a project directory and drop into the folder.

```bash
mkdir aca-with-amg
cd aca-with-amg
```

> All instructions will be in Bash. Most of the commands *should* work in PowerShell. If you do not have a local Bash terminal, I'd recommend you try these commands in [Azure Cloud Shell (Bash)](https://aka.ms/cloudshell).

Create a new `main.tf` file. This is the entry point for Terraform.

```bash
touch main.tf
```

Open the `main.tf` file using your favorite text editor and add the following code to tell Terraform which [providers](https://www.terraform.io/language/providers) you will be using.

```bash
terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = ">=3.0.0"
    }

    azapi = {
      source  = "azure/azapi"
      version = ">=0.5.0"
    }
  }
}

provider "azurerm" {
  features {
    resource_group {
      prevent_deletion_if_contains_resources = false
    }

    key_vault {
      purge_soft_delete_on_destroy = false
    }
  }
}
```

Let's test to make sure everything is good so far. Run the following command in your terminal. Terraform will [initialize](https://www.terraform.io/cli/commands/init) the project and download [provider plugins](https://www.terraform.io/cli/plugins).

```bash
az login # log into azure
terraform init # initialize terraform directory
```

> You will see a note about initializing the [backend](https://www.terraform.io/language/settings/backends/configuration). This example will use a local backend, but you should use a remote backend like [Azure Blob Storage](https://docs.microsoft.com/azure/developer/terraform/store-state-in-azure-storage?tabs=azure-cli) or [Terraform Cloud](https://cloud.hashicorp.com/products/terraform) for production deployments

## Naming things

Naming things is hard! To make naming resources easy, let's use the [`random` provider](https://registry.terraform.io/providers/hashicorp/random/latest/docs) to help uniquely name resources. Add this to your `main.tf` file.

```bash
resource "random_pet" "aca" {
  length    = 2
  separator = ""
}

resource "random_integer" "aca" {
  min = 000
  max = 999
}

resource "random_string" "aca" {
  length  = 5
  lower   = true
  upper   = false
  numeric = true
  special = false

  keepers = {
    # Generate a new random_string on every run to avoid a conflict with the previous revision
    none = timestamp()
  }
}

locals {
  resource_name        = format("%s", random_pet.aca.id)
  resource_name_unique = format("%s%s", random_pet.aca.id, random_integer.aca.result)
  location             = "eastus"
}
```

> To keep things simple, I've hard-coded the `location` local variable to `eastus` change this to a different [region that supports both ACA and AMG](https://azure.microsoft.com/explore/global-infrastructure/products-by-region/?products=container-apps,managed-grafana) if necessary.

Now that we added a new provider, you'll have to run the `terraform init` command again.

## Sprinkle some `azurerm` on it

We'll start with the resources that are available in the [`azurerm` provider](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs).

### Azure Resource Group

Azure resources are deployed into a resource group. Add this to `main.tf`.

```bash
resource "azurerm_resource_group" "aca" {
  name     = "rg-${local.resource_name}"
  location = local.location
}
```

### Azure Log Analytics Workspace

ACA requires an ALA workspace. Add this to `main.tf`

```bash
resource "azurerm_log_analytics_workspace" "aca" {
  name                = "law-${local.resource_name_unique}"
  resource_group_name = azurerm_resource_group.aca.name
  location            = azurerm_resource_group.aca.location
  sku                 = "PerGB2018"
  retention_in_days   = 30
}
```

## What's a Terraform AzAPI provider?

At the time of this writing the ACA and AMG resources are not available in the `azurerm` provider, so we need to use the new [Terraform AzAPI provider](https://docs.microsoft.com/azure/developer/terraform/overview-azapi-provider). This provider calls into the [Azure ARM REST APIs](https://docs.microsoft.com/rest/api/resources/) which makes it possible to deploy and manage **any** [Azure ARM resource](https://docs.microsoft.com/azure/templates/) via Terraform.

[Azure Docs](https://docs.microsoft.com/azure/templates/) has documented all ARM resources with **Bicep**, **ARM template**, and now **Terraform** definitions.

In the example below for [Microsoft.App/managedEnvironment](https://docs.microsoft.com/azure/templates/microsoft.app/2022-03-01/managedenvironments?tabs=bicep&pivots=deployment-language-terraform), you will see a **Terraform** button. If you click the button, you will see the entire schema and object definitions for deploying this resource. Nice!

![Terraform AzAPI docs](https://assets.paulyu.dev/articles/2022/aca-monitoring-with-amg/azapi-docs.png)

### Azure Container App Environment

Let's add this to `main.tf` to create an ACA environment.

```bash
resource "azapi_resource" "env" {
  type      = "Microsoft.App/managedEnvironments@2022-03-01"
  name      = "env-${local.resource_name}"
  parent_id = azurerm_resource_group.aca.id
  location  = azurerm_resource_group.aca.location

  body = jsonencode({
    properties = {
      appLogsConfiguration = {
        destination = "log-analytics"
        logAnalyticsConfiguration = {
          customerId = azurerm_log_analytics_workspace.aca.workspace_id
          sharedKey  = azurerm_log_analytics_workspace.aca.primary_shared_key
        }
      }
    }
  })
}
```

### Azure Container App

With the environment in place, we can now define our first `helloworld` container. Add the following to the `main.tf`.

```bash
resource "azapi_resource" "helloworld" {
  type      = "Microsoft.App/containerApps@2022-03-01"
  name      = "helloworld"
  parent_id = azurerm_resource_group.aca.id
  location  = azurerm_resource_group.aca.location

  body = jsonencode({
    properties = {
      managedEnvironmentId = azapi_resource.env.id
      configuration = {
        ingress = {
          allowInsecure = false
          external      = true
          targetPort    = 80
          traffic = [
            {
              label          = "dev"
              latestRevision = true
              weight         = 100
            }
          ]
        }
      }
      template = {
        containers = [
          {
            name  = "helloworld"
            image = "mcr.microsoft.com/azuredocs/containerapps-helloworld:latest"
            resources = {
              cpu    = 0.5
              memory = "1.0Gi"
            }
          }
        ]
        revisionSuffix = random_string.aca.result
        scale = {
          minReplicas = 0
          maxReplicas = 30
          rules = [
            {
              name = "http-rule"
              http = {
                metadata = {
                  concurrentRequests = "100"
                }
              }
            }
          ]
        }
      }
    }
  })

  # this tells azapi to pull out properties and stuff into the output attribute for the object
  response_export_values = ["properties.configuration.ingress.fqdn"]
}
```

This configuration will deploy a container with external ingress and autoscaling enabled.

### Azure Managed Grafana

When deploying an AMG instance, you must have the **Owner** role assigned on your Azure subscription. This is so that we can assign the **Monitoring Reader** role on your AMG instance to read data from Azure Monitor across your subscription. Also, the **Grafana Admin** role should be assigned to your account so that you can manage and authenticate to the Grafana dashboard portal using Azure Active Directory.

To check your role assignments using Azure CLI, you can run the following command:

```bash
az role assignment list --assignee <YOUR_USER_NAME> --include-inherited
```

If you see a `"roleDefinitionName": "Owner"` over `"scope": "/"` or `"scope": "/subscriptions/<YOUR_SUBSCRIPTION_ID>"` then you are ready to proceed. If not, you may need to contact your administrator for assistance.

![Your role assignments](https://assets.paulyu.dev/articles/2022/aca-monitoring-with-amg/your-role-assignments.png)

Add the AMG configuration to `main.tf`.

```bash
resource "azapi_resource" "amg" {
  type      = "Microsoft.Dashboard/grafana@2022-08-01"
  name      = "amg-${local.resource_name}"
  parent_id = azurerm_resource_group.aca.id
  location  = azurerm_resource_group.aca.location

  identity {
    type = "SystemAssigned"
  }

  body = jsonencode({
    properties = {
      apiKey                            = "Enabled"
      autoGeneratedDomainNameLabelScope = "TenantReuse"
      deterministicOutboundIP           = "Enabled"
      publicNetworkAccess               = "Enabled"
      zoneRedundancy                    = "Disabled"
    }
    sku = {
      name = "Standard"
    }
  })

  # this tells azapi to pull out properties and stuff into the output attribute for the object
  response_export_values = ["identity.principalId"]
}
```

This will assign a [System-Assigned managed identity](https://docs.microsoft.com/azure/active-directory/managed-identities-azure-resources/overview) and export the `identity.principalId` property which is needed for role assignment.

#### Azure Role Assignments

Since we need to add Azure role assignments, we'll need to add the subscription and current login context as [data sources](https://www.terraform.io/language/data-sources). Add this to `main.tf`.

```bash
data "azurerm_subscription" "aca" {}
data "azurerm_client_config" "aca" {}
```

We can now add our role assignments to `main.tf`.

```bash
resource "azurerm_role_assignment" "amg_reader" {
  scope                = data.azurerm_subscription.aca.id
  role_definition_name = "Monitoring Reader"
  principal_id         = jsondecode(azapi_resource.amg.output).identity.principalId # the managed identity of the AMG resource
}

resource "azurerm_role_assignment" "amg_admin" {
  scope                = azapi_resource.amg.id
  role_definition_name = "Grafana Admin"
  principal_id         = data.azurerm_client_config.aca.object_id # your azure login context's object id
}
```

## Outputs for testing

Terraform can return deployment properties using [outputs](https://www.terraform.io/language/values/outputs). We'll format text to return `helloworld` container app's ingress URL. Add this to your `main.tf` file.

```bash
output "helloworld_ingress_url" {
  value = format("%s%s", "https://", jsondecode(azapi_resource.helloworld.output).properties.configuration.ingress.fqdn)
}
```

> Normally, the best practice is to put outputs in a `outputs.tf` file but for this demo, dropping everything into the `main.tf` file is fine for now.

## Deploy to Azure 🚀

Before we run the deployment using the [`terraform apply`](https://www.terraform.io/cli/commands/apply) command, we should perform [`terraform fmt`](https://www.terraform.io/cli/commands/fmt) to clean up our code according to [Terraform's style guide](https://www.terraform.io/language/syntax/style) and perform [`terraform validate`](https://www.terraform.io/cli/commands/validate) to ensure the code doesn't have errors. Run the following commands in your terminal.

```bash
terraform fmt # cleans up code spacing and alignment
terraform validate # validates code does not have errors
```

You should see `Success! The configuration is valid.`

Run the following command to deploy.

```bash
terraform apply
```

You should see an output from the Terraform plan with a prompt confirming your deployment.

> Normally you would run the [`terraform plan -out=tfplan`](https://www.terraform.io/cli/commands/plan) command followed by a `terraform apply tfplan` command as a best practice but since we are just demo'ing, this is fine for now.

## Testing the `helloworld` container app

You should see the `helloworld_ingress_url` output to your console. We can use this value to test the application with the following command.

```bash
curl -IL $(terraform output -raw helloworld_ingress_url)
```

If you have a load testing tool installed locally, you can use that to put some load on your application and generate some metrics.

I have [`hey`](https://github.com/rakyll/hey) installed locally so I'll use it to run a quick load test.

```bash
hey -c 10 -n 100 -m GET -z 3m $(terraform output -raw helloworld_ingress_url)
```

Congratulations, we've confirmed the app works 🥳

## Importing Grafana dashboards

We're nearly done. All that's left is to perform a few manual steps to import the Grafana dashboard for ACA.

In the Azure Portal, find your AMG service and click on the **Endpoint** URL to navigate to the Grafana dashboard portal.

![Azure Managed Grafana endpoint](https://assets.paulyu.dev/articles/2022/aca-monitoring-with-amg/amg-endpoint.png)

Expand the **Dashboards** menu item and click on the **+ Import** button.

![Import the dashboard](https://assets.paulyu.dev/articles/2022/aca-monitoring-with-amg/import-dashboard.png)

We can import using the dashboard URL or ID or load JSON. We'll use the dashboard ID to make things easy.

![Import via grafana.com](https://assets.paulyu.dev/articles/2022/aca-monitoring-with-amg/import-via-grafana.png)

If you browse to the [Azure / Container Apps / Container App View](https://grafana.com/grafana/dashboards/16592-azure-container-apps-container-app-view/) dashboard page, you will find information on how to import the dashboard. Copy the dashboard ID.

![Grafana.com dashboard ID for ACA](https://assets.paulyu.dev/articles/2022/aca-monitoring-with-amg/aca-dashboard-id.png)

Back in your Grafana portal, enter the ID (`16592`) into the **Import via grafana.com** text box then click the **Load** button.

Next, select **Azure Monitor** as the data source for the dashboard then click the **Import** button at the bottom.

![Set Azure Monitor as the data source for the dashboard](https://assets.paulyu.dev/articles/2022/aca-monitoring-with-amg/import-dashboard-datasource.png)

Browse to the dashboard and use the controls across the top of the page to select your container app. The dashboard is using Azure Monitor as a data source so you will have the option to view all container apps metrics tied to your ALA workspace.

![Select your ACA](https://assets.paulyu.dev/articles/2022/aca-monitoring-with-amg/select-aca-to-monitor.png)

Now we have a pretty dashboard with very useful metrics 😍 Based on our load test earlier, we can see the `helloworld` container app scaled up from `0` replicas to `22` and back down to `0` after a few minutes.

### Using Azure CLI to import Grafana dashboards

Alternatively, you can import dashboards into AMG with the following `az` command:

```bash
az grafana dashboard import -g <YOUR_RESOURCE_GROUP_NAME> -n <YOUR_GRAFANA_NAME> --definition 16592
```

> ⚠️ WARNING
>
> The `az grafana` command is currently in preview and functionality may change!

To take it a step further, we can automate the dashboard import process by adding this to your `main.tf` file.

```bash
resource "null_resource" "aca" {
  provisioner "local-exec" {
    command = <<-EOT
        az config set extension.use_dynamic_install=yes_without_prompt
        az grafana dashboard import -g ${azurerm_resource_group.aca.name} -n ${azapi_resource.amg.name} --definition 16592
    EOT
  }
}
```

> ⚠️ WARNING
>
> This `null_resource` block's `local-exec` command includes an `az config` command which sets your Azure CLI to install extensions without prompts. Add this **ONLY** if you are okay with this behavior.

## Summary

That was a lot of steps in getting the resources provisioned using Terraform and the AzAPI provider, but I hope you got a sense of how easy it is to add beautifully curated dashboards to monitor your Azure Container Apps.

We only imported the [Container App View](https://grafana.com/grafana/dashboards/16592-azure-container-apps-container-app-view/) dashboard and I'll leave you with the exercise of importing the [Container App Aggregate View](https://grafana.com/grafana/dashboards/16591-azure-container-apps-aggregate-view/) dashboard 😉

If you have any questions or feedback, please let me know in the comments below or reach out on Twitter [@pauldotyu](https://twitter.com/pauldotyu)

Cheers!

## Resources

- [Elevate your visualizations with Azure Managed Grafana](https://azure.microsoft.com//blog/elevate-your-visualizations-with-azure-managed-grafana-now-generally-available/)
- [What is Azure Managed Grafana?](https://docs.microsoft.com//azure/managed-grafana/overview)
- [Export and import Grafana dashboards](https://grafana.com/docs/grafana/v9.0/dashboards/export-import/)
- [Azure Container Apps overview](https://docs.microsoft.com//azure/container-apps/overview)
- [Azure Monitor data source](https://grafana.com/docs/grafana/latest/datasources/azuremonitor/)
- [azuremonitorteam Dashboards](https://grafana.com/orgs/azuremonitorteam/dashboards)
- [Overview of the Terraform AzAPI provider](https://docs.microsoft.com//azure/developer/terraform/overview-azapi-provider)
- [Entire code snippet for our example `main.tf` file](https://paulyu.dev/code/terraform-aca-with-amg-using-azapi-provider/)
- [Azure Open Source Labs: Container Apps with Terraform](https://github.com/Azure-Samples/azure-opensource-labs/tree/main/cloud-native/containerapps-terraform)