Does Workload Identity on AKS work across tenants?
Tags: azure, kubernetes, workloadidentity, cloudnative
Published: 2023-08-28 16:34:37

---
title: Does Workload Identity on AKS work across tenants?
published: true
date: 2023-08-25 13:00:00 UTC
tags: azure, kubernetes, workloadidentity, cloudnative
canonical_url: https://paulyu.dev/article/cross-tenant-workload-identity-on-aks/
cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/01kx2ijpi60qhtul4env.png
---

## Introduction

An interesting use case for Workload Identity came up recently. I was asked if a Pod in an AKS cluster that was deployed in one tenant can access Azure resources within another tenant.

I've configured Workload Identity on AKS many times, and I thought "in theory" it should "just work", but I never tested it across tenants. So I decided to give it a try.

**TL;DR: Yes, it does work.**

## What is Workload Identity?

If you are not familiar with [Azure AD Workload Identity](https://learn.microsoft.com/azure/active-directory/workload-identities/workload-identities-overview), it is a capability that allows an Azure identity to be assigned to an application workload which can then be used to authenticate and access services and resources protected by Azure AD

> I know Azure AD has been [renamed to Microsoft Entra ID](https://devblogs.microsoft.com/identity/aad-rebrand/) but I'll keep referring to it as Azure AD here 😉

In Kubernetes, this identity is assigned to a [ServiceAccount](https://kubernetes.io/docs/concepts/security/service-accounts/) and [Pods](https://kubernetes.io/docs/concepts/workloads/pods/) that use the ServiceAccount can access to Azure resources without having to manage credentials.

If you've heard of or worked with [Azure AD Pod Identity](https://learn.microsoft.com/azure/aks/use-azure-ad-pod-identity), this is the improved version and replacement for that. So, you should [migrate to Workload Identity](https://learn.microsoft.com/azure/aks/workload-identity-migrate-from-pod-identity) as soon as possible.


## How does Workload Identity work?

At a high-level, Azure AD federates with a Kubernetes cluster to establish trust, and Workload Identity uses [OpenID Connect](https://openid.net/connect/) (OIDC) to authenticate with Azure AD.

The Kubernetes cluster needs to be configured to use [ServiceAccount token volume projection](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#serviceaccount-token-volume-projection) which places a ServiceAccount token into a Pod. This token is sent to Azure AD where it is validated for authenticity and exchanged for an Azure AD bearer token. From there, the Pod uses the Azure AD token to access Azure resources.

How does Azure AD know where to validate this token, you ask? It validates the token against the OIDC endpoint that you tell it to when establishing the trust between Azure AD and your Kubernetes cluster.

**With OIDC, the tenant that the AKS cluster resides in doesn't really matter. As long as the OIDC endpoint of the Kubernetes cluster is accessible by Azure AD, the trust can be established.**

The token exchange is handled within your app using the [Azure Identity SDK](https://azure.github.io/azure-workload-identity/docs/topics/language-specific-examples/azure-identity-sdk.html#azure-identity-sdk) or the [Microsoft Authentication Library (MSAL)](https://azure.github.io/azure-workload-identity/docs/topics/language-specific-examples/msal.html). So there is a bit of application code that needs to be written to make this work. But the benefit is that you don't have to manage credentials for Azure resources 🔒

The diagram depicted [here](https://azure.github.io/azure-workload-identity/docs/concepts.html) gives a good visual representation of the process.
![Workload Identity](https://assets.paulyu.dev/articles/2023/cross-tenant-workload-identity/flow-diagram.png)


Cool thing about workload identity is that it works with any Kubernetes cluster; not just AKS, since workload identity leverages native capabilities of Kubernetes. It's just a little easier to configure on AKS since Azure [handles the configuration on the control plane](https://azure.github.io/azure-workload-identity/docs/installation/self-managed-clusters.html) for you 😎

Let's take a look at how you'd setup and use workload identity on AKS.

## Cross-tenant Workload Identity on AKS

I'll be demonstrating this using a simple Go app that reads Azure subscription resource quota information. The Go app will run in a Pod in an AKS cluster and will use Workload Identity to access Azure subscription info in another tenant.

Here is a diagram that illustrates the configuration process.

![Cross-tenant Workload Identity](https://assets.paulyu.dev/articles/2023/cross-tenant-workload-identity/cross-tenant-workload-identity-setup.png)

### Pre-requisites

In order do run through this demonstration, you will need the following:

- Two [Azure subscriptions](https://aka.ms/freeazure) each in a separate tenant (I'll call them **Tenant A** and **Tenant B** for simplicity)
- [Azure CLI](https://learn.microsoft.com/cli/azure/install-azure-cli)
- [Go](https://go.dev/doc/install)
- [ko: Easy Go Containers](https://ko.build/)
- [VS Code](https://code.visualstudio.com/)


### Configure Azure resources in Tenant A

In **Tenant A**, we will create an AKS cluster with Workload Identity enabled. This cluster will be used to deploy an app that will attempt to access Azure resources in **Tenant B**.

#### Create the AKS cluster with Workload Identity enabled

Login to your subscription in **Tenant A**.

```bash
az login 
```

Create the AKS cluster with Workload Identity and OIDC issuer features enabled.

```bash
az group create --name rg-wi-demo --location eastus
az aks create --name aks-wi-demo --resource-group rg-wi-demo --enable-oidc-issuer --enable-workload-identity
```

Azure AD will use the OIDC issuer endpoint to discover public signing keys and verify the authenticity of the service account token before exchanging it for an Azure AD token.

```bash
az aks show --name aks-wi-demo --resource-group rg-wi-demo --query "oidcIssuerProfile.issuerUrl" -o tsv
```

> ⚠️ **IMPORTANT** 
> This OIDC issuer URL is very important to the process so make a note of it.
 
That's all that is needed in **Tenant A** for now.

### Configure Azure resources in Tenant B

In **Tenant B**, we need to create the managed identity, assign it permissions to read subscription information, and establish the trust between the managed identity and the AKS cluster in **Tenant A**.

#### Create the managed identity and assign permissions

Login to your subscription in **Tenant B**.

```bash
az login
```

Create a user-assigned managed identity and retrieve the client ID. 

```bash
az group create --name rg-wi-demo --location eastus
az identity create --name mi-wi-demo --resource-group rg-wi-demo --query clientId -o tsv
```

> ⚠️ **IMPORTANT**
> The `clientId` of the managed identity will be needed later, so make a note of it too.

Next, retrieve the principal ID of the managed identity. 

```bash
principalId=$(az identity show -n mi-wi-demo -g rg-wi-demo --query principalId -o tsv)
```

Grant the newly created managed identity permissions to read subscription information.

```bash
subscriptionId=$(az account show --query "id" -o tsv)
az role assignment create --role Reader --assignee $principalId --scope /subscriptions/${subscriptionId}
```

### Establish trust between AKS cluster in Tenant A and managed identity in Tenant B

Now we need to create the federated identity credential to establish trust between the AKS cluster in **Tenant A** and Azure managed Identity in **Tenant B**. We will use the OIDC issuer URL from the AKS cluster and the name of the managed identity to link the two. 

The other bit of important information here is the value of `subject` in the federated credential. Here we are using a value of `system:serviceaccount:default:wi-demo-account`. This is the name of the Kubernetes ServiceAccount that we will create later in **Tenant A**. 

When authentication requests are made from our application pod, this value will be sent to Azure AD as the `subject` in the auth request, and Azure AD will determine eligibility if this value matches what we set when establishing trust. So it is important here that the Namespace and ServiceAccount names match what we will use later.

Place your OIDC issuer URL in the `oidcIssuerUrl` variable below and run the following command to create the federated credential (establishing trust).

```bash
oidcIssuerUrl=<YOUR_OIDC_ISSUER_URL>

az identity federated-credential create \
  --name fc-wi-demo \
  --identity-name mi-wi-demo \
  --resource-group rg-wi-demo \
  --issuer $oidcIssuerUrl \
  --subject system:serviceaccount:default:wi-demo-account
```

### Create the app to read subscription information

Now that we have all of the Azure resources configured, we can create a simple app that will attempt to read subscription information.

#### Initialize the Go project

```bash
# putting your go app in GOPATH is optional
cd $(go env GOPATH)

# create a new directory for the app
mkdir wi-demo
cd wi-demo

# initialize the go project
go mod init example.com/wi-demo

# install the Azure SDK for Go
go get github.com/Azure/azure-sdk-for-go/sdk/azidentity
go get github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/quota/armquota

# create the main.go file
touch main.go
```

#### Add code to the main.go file

Open the directory in VSCode, then open `main.go` and add the following code. 

Shout out to the [Azure SDK for Go developers](https://azure.github.io/azure-sdk/releases/latest/index.html#go) for putting together great docs with examples 🥳 I was able to lift code from [here](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/quota/armquota#pkg-functions) and [here](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/quota/armquota#UsagesClient) for this sample app.

```go
package main

import (
	"context"
	"log"
	"os"

	"github.com/Azure/azure-sdk-for-go/sdk/azidentity"
	"github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/quota/armquota"
)

func main() {
	subscriptionID := os.Getenv("AZURE_SUBSCRIPTION_ID")
	region := os.Getenv("AZURE_REGION")
	resourceName := os.Getenv("AZURE_RESOURCE_NAME")

	cred, err := azidentity.NewDefaultAzureCredential(nil)
	if err != nil {
		log.Fatalf("failed to obtain a credential: %v", err)
	}

	ctx := context.Background()

	clientFactory, err := armquota.NewClientFactory(cred, nil)
	if err != nil {
		log.Fatalf("failed to create client: %v", err)
	}

	clientResponse, err := clientFactory.NewClient().Get(ctx, resourceName, "subscriptions/"+subscriptionID+"/providers/Microsoft.Compute/locations/"+region, nil)
	if err != nil {
		log.Fatalf("failed to finish the request: %v", err)
	}

	clientJson, _ := clientResponse.CurrentQuotaLimitBase.MarshalJSON()
	log.Printf("limit: %s", clientJson)

	usagesClientResponse, err := clientFactory.NewUsagesClient().Get(ctx, resourceName, "subscriptions/"+subscriptionID+"/providers/Microsoft.Compute/locations/"+region, nil)
	if err != nil {
		log.Fatalf("failed to finish the request: %v", err)
	}

	usagesJson, _ := usagesClientResponse.CurrentUsagesBase.MarshalJSON()
	log.Printf("usage: %s", usagesJson)
}
```

The application will attempt to read the resource quota information using the `armquota` package and print the results to the console.

It will look for the following environment variables at runtime:

- `AZURE_SUBSCRIPTION_ID`: The subscription ID of the subscription in **Tenant B**
- `AZURE_REGION`: The Azure region where resources are deployed
- `AZURE_RESOURCE_NAME`: The name of the resource to read quota information for

If you are unsure of the values to use for `AZURE_RESOURCE_NAME` you can run the command `az vm list-usage --location eastus` to get a list of available resources.

#### Testing the app locally

Before we attempt to deploy the app to the AKS cluster, let's test it locally to make sure it works.

Log into any Azure tenant as we will just use Azure CLI credentials for local testing.

```bash
az login
```

Since our app will be reading subscription quota information using the [Azure Quota API](https://learn.microsoft.com/rest/api/reserved-vm-instances/quotaapi), we'll need to ensure the `Microsoft.Quota` provider is registered in the subscription.

First, check to see if the provider is registered.

```bash
az provider show --namespace Microsoft.Quota --query "registrationState"
```

If the registration state is `NotRegistered`, register the provider.

```bash
az provider register --namespace Microsoft.Quota
```

Wait until the registration state is `Registered` before continuing.

Once the provider is registered, we can run the app locally.

```bash
export AZURE_SUBSCRIPTION_ID=$(az account show --query "id" -o tsv)
export AZURE_REGION="eastus"
export AZURE_RESOURCE_NAME="cores"
go run main.go
```

If all went well, the app will run to completion and you should see output with JSON printed to the console.

### Deploy the app to AKS

Once we confirm the app works locally, it's time to push it to a container registry so that it can be pulled from within the AKS cluster.

#### Publish the container to a container registry

I like using [ko](https://ko.build/) to build and publish Go apps to a container registry. It's super easy to use and doesn't require Docker to be installed on your machine.

I also like to use [ttl.sh](https://ttl.sh) for quick container testing. It's a free service that allows you to push a container to a registry and have it available for a specified period of time by using tags (e.g., `1h` for 1 hour and `1m` for 1 minute). Being able to use ephemeral containers is perfect for quick testing and demos.

```bash
export KO_DOCKER_REPO=ttl.sh
ko build . --tags=1h
```

You should see something like this in the third line of the output. This is your image name:

```text
Publishing ttl.sh/wi-demo-09748ea200b8b1398c659943f01b0121:5m
```

> ⚠️ **IMPORTANT**
> `ko` will generate an image name for you with the tag you specify, so make a note of it as we will use it later.

#### Deploy the app to AKS

In **Tenant A**, log into the AKS cluster.

```bash
az aks get-credentials --name aks-wi-demo --resource-group rg-wi-demo
```

Using the `clientId` of the user-assigned managed identity, create a new ServiceAccount.

```bash
kubectl apply -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    azure.workload.identity/client-id: <YOUR_USER_ASSIGNED_MANAGED_IDENTITY_CLIENT_ID>
  name: wi-demo-account
  namespace: default
EOF
```

Notice the `azure.workload.identity/client-id` annotation on the ServiceAccount. This is what tells the AKS cluster which managed identity to use for Workload Identity.

Using the image name, and **Tenant B's** tenant and subscription IDs, create a new Pod.

```bash
kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: wi-demo-app
  namespace: default
  labels:
    azure.workload.identity/use: "true"
spec:
  serviceAccountName: wi-demo-account
  containers:
  - name: wi-demo-app
    image: <YOUR_IMAGE_NAME>
    env:
    - name: AZURE_TENANT_ID
      value: <TENANT_B_ID>
    - name: AZURE_SUBSCRIPTION_ID
      value: <TENANT_B_SUBSCRIPTION_ID>
    - name: AZURE_REGION
      value: eastus
    - name: AZURE_RESOURCE_NAME
      value: cores
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
EOF
```

Notice the `azure.workload.identity/use` label and the `serviceAccountName: wi-demo-account` in the Pod spec. This is what tells the AKS cluster to use Workload Identity for this Pod.

> ⚠️ **IMPORTANT**
> The `AZURE_TENANT_ID` environment variable is used by the Azure Identity SDK to authenticate with Azure AD. If this was not set, the SDK would attempt to authenticate with the tenant that the AKS cluster is in. Since we need the app to authenticate to **Tenant B**, we need to explicitly set this environment variable.

With the Pod running, we can check the logs to see if the app was able to read subscription quota information.

```bash
kubectl logs wi-demo-app
```

You should see output similar to this:

```text
2023/08/24 17:48:03 clientResponse: {"id":"/subscriptions/00000000-0000-0000-0000-000000000000/providers/Microsoft.Compute/locations/eastus/providers/Microsoft.Quota/quotas/cores","name":"cores","properties":{"isQuotaApplicable":true,"limit":{"limitObjectType":"LimitValue","limitType":"Independent","value":20},"name":{"localizedValue":"Total Regional vCPUs","value":"cores"},"properties":{},"unit":"Count"},"type":"Microsoft.Quota/Quotas"}
2023/08/24 17:48:04 usagesClientResponse: {"id":"/subscriptions/00000000-0000-0000-0000-000000000000/providers/Microsoft.Compute/locations/eastus/providers/Microsoft.Quota/usages/cores","name":"cores","properties":{"isQuotaApplicable":true,"name":{"localizedValue":"Total Regional vCPUs","value":"cores"},"properties":{},"unit":"Count","usages":{}},"type":"Microsoft.Quota/Usages"}
```

If you see output similar to this, then you have successfully used Workload Identity to access Azure resources in another tenant 🎉

## Conclusion

In this article, we explored using Workload Identity on AKS to access Azure resources in another tenant. I just assumed it would work, but it was nice to see it in action. The steps to make it cross tenant is not really anything special. The OIDC issuer URL is the key piece of information that is needed to establish trust between the AKS cluster and the managed identity regardless of which tenant they are in.

I'll admit the process to configure workload identity in general is a bit involved, but as long as you have a good understanding of the moving parts, it's not too bad. 

The Azure Workload Identity team also created a command line utility called [azwi](https://azure.github.io/azure-workload-identity/docs/topics/azwi.html) which may help automate some of the steps so be sure the check that out.

The sample app I demonstrated here is just one use case where you can use Workload Identity to read subscription information. This concept can be applied to any Azure resource that is protected by Azure AD. 

So, if you need to access Azure resources from a workload running in Kubernetes and have the flexibility to modify your application code (using Azure Identity SDK), I highly recommend you use Workload Identity.

If you have any feedback or suggestions, please feel free to reach out to me on [Twitter](https://twitter.com/pauldotyu) or [LinkedIn](https://www.linkedin.com/in/yupaul).

Peace ✌️

## Resources

- [What are workload identities?](https://learn.microsoft.com/azure/active-directory/workload-identities/workload-identities-overview)
- [What are managed identities for Azure resources?](https://learn.microsoft.com/azure/active-directory/managed-identities-azure-resources/overview)
- [Azure services that can use managed identities to access other services](https://learn.microsoft.com/azure/active-directory/managed-identities-azure-resources/managed-identities-status)
- [Use Azure AD workload identity with Azure Kubernetes Service (AKS)](https://learn.microsoft.com/azure/aks/workload-identity-overview?tabs=dotnet)
- [Tutorial: Use a workload identity with an application on Azure Kubernetes Service (AKS)](https://learn.microsoft.com/azure/aks/learn/tutorial-kubernetes-workload-identity)
- [Azure AD Workload Identity (GitHub)](https://azure.github.io/azure-workload-identity/docs/)