Unified Kubernetes Management Across Clouds with Azure Arc
Tags: kubernetes, azure, aws, cloud
Published: 2024-04-04 01:58:29

After relocating to South Korea, I began provisioning resources in the Korea Central and Korea South regions, moving away from the United Kingdom South. I quickly noticed that specific SKUs, and sometimes even entire resources or services, were unavailable in my new region. While this was manageable for my Proof of Concepts (POCs) and personal projects, such limitations could pose significant challenges for organizations operating in the region, particularly when compliance or security requirements necessitate keeping data within South Korea.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2rmzujay4zlez2n967vi.png)

These regional limitations, coupled with the varied performance metrics offered by different Cloud Service Providers (CSPs), the potential benefits of partnership programs (like discounts or free resource allocations), and the strategic advantage of minimizing vendor lock-in, are compelling more companies to adopt multi-cloud environments. However, this shift doesn’t come free and bring with it multiple challenges, such as the increased difficulty in monitoring and the need for engineers to possess a broader range of skills to integrate these diverse resources effectively.

In this article, I aim to explore the deployment of Kubernetes clusters on AWS and then use Azure Arc to manage these resources from a unified control plane along the other native Azure resources.

# What is Azure Arc?
Before we delve deeper into the demonstration, let’s clarify what Azure Arc is and how it functions. Azure Arc was introduced in April 2021, initially focusing on managing databases, with a particular emphasis on PostgreSQL servers. However, its support was quickly extended to include a wider range of resources, such as Virtual Machines, VMware vSphere, System Center VMM, and Kubernetes Clusters, both on-premises and from other Cloud Service Providers. Each resource type has its specific requirements for integration. For instance, to connect an external Kubernetes cluster to Azure Arc, the cluster must be running Kubernetes version 1.13 or later. Additionally, there must be connectivity to the cluster through ports 443 and 9418.

The core functionality of Azure Arc involves “projecting” your external resources into Azure. This is achieved by installing Azure Arc agents on these external resources. These agents facilitate communication with Azure, enabling management tasks such as patching, policy enforcement, and tagging across your infrastructure. Once connected, these resources appear in the Azure portal alongside your native Azure resources. Additionally, Azure Arc enhances monitoring capabilities by integrating with Azure Monitor, providing insights for all cloud-based resources registered under its umbrella.

From a financial point of view, this service is available at no additional cost. This is probably a strategic decision by Microsoft that aims to encourage customers to leverage its services over competitors’ or on-premise solutions, especially for those already utilizing other Microsoft services.

# Deploying Kubernetes Clusters on Different Clouds
After setting up a new Kubernetes cluster on AWS, the first step is to grant your user account with the necessary permissions. This is done by assigning the `AmazonEKSClusterAdminPolicy` access policy to your user.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/h4gkh1t19egt9z83kl8w.png)

Next, it’s time to create a Node Group. This Node Group will be linked to your EKS cluster and hosts the virtual machines, essentially the backbone where your application pods will reside and run.

```
$ aws eks list-nodegroups --cluster-name eks-training-dev-krc --region us-east-1
{
    "nodegroups": [
        "eks-training-dev-krc"
    ]
}

$ aws eks describe-cluster --name eks-training-dev-krc --region us-east-1
{
    "cluster": {
        "name": "eks-training-dev-krc",
        "arn": "arn:aws:eks:us-east-1:370515815379:cluster/eks-training-dev-krc",
        "createdAt": "2024-03-12T10:59:16.159000+09:00",
        "version": "1.29",
        "endpoint": "https://2281EF2EDCF25F5B4BC63704E5DE0468.gr7.us-east-1.eks.amazonaws.com",
        "roleArn": "arn:aws:iam::370515815379:role/EKSClusterRole",
        "resourcesVpcConfig": {
            "subnetIds": [
                "subnet-08b3a655cf510da0a",
                "subnet-09470b12b7dad0427",
                "subnet-01964a49f044d69a3"
            ],
            "securityGroupIds": [],
            "clusterSecurityGroupId": "sg-070ba0ce70c3c4ae9",
            "vpcId": "vpc-0b3737b03c3980dc1",
            "endpointPublicAccess": true,
            "endpointPrivateAccess": true,
            "publicAccessCidrs": [
                "0.0.0.0/0"
            ]
        }
        ...
    }
}
```

With your cluster and node groups in place, the next crucial step is generating a config file. This file will contains all the information required to access the Kubernetes cluster, such as cluster, user, server, and certificate.
```
$ aws eks update-kubeconfig --region us-east-1 --name eks-training-dev-krc
Added new context arn:aws:eks:us-east-1:xxxxxxxxxxxx:cluster/eks-training-dev-krc to C:\Users\Ivan\.kube\config
```
Moving to Azure’s part of the setup, you’ll need to enrich your Azure CLI with several extensions crucial for Kubernetes management via Azure Arc:
```
az extension add --name connectedk8s
az extension add --name k8s-extension
az extension add --name k8s-configuration
az extension add --name customlocation
```
Additionally, you must register the necessary Azure resource providers to support Azure Arc-enabled Kubernetes features:
```
az provider register --namespace Microsoft.Kubernetes
az provider register --namespace Microsoft.KubernetesConfiguration
az provider register --namespace Microsoft.ExtendedLocation
```
Now that we have provisioned out Kubernetes cluster on AWS, we can move forward and connect it to Azure via Azure Arc. The first step is to go to the Azure Arc service page, select Kubernetes cluster, and the click **Add a Kubernetes cluster with Azure Arc**.

**Note: the Create a Kubernetes cluster with Azure Arc is focused on the creation of Kubernetes clusters on-premises which is out of scope of this article.**

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9jjezpboqlqkc44vabt6.png)

Next, select the resource group where the subcription and resource is going to be added, and give it a name and region. The region in neccessary as Azure doesn’t know the region o the target Cloud Service provider where the resource is provisioned. You will also need to specify how to connect to the resource; this options are the same across all resoruce types and are the following:

- **Public endpoint**: The resource has direct access to the internet without any intermediary.
- **Proxy server**: The traffic to and from the Azure Arc agent to Azure services are routed through a proxy server. This method is common in enterprise environments where direct Internet access is restricted for security reasons.
- **Private endpoint**: The traffic is routed through the Azure backbone network rather than the public Internet.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9qhi8c05pcj4d5ooxbh7.png)

After that, you will be able to define some tags, but the magic happens in the scirpt section.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/isd7d03qk6j1n6x8klvk.png)

This script will need to be executed on the machine with the EKS clutter context and will take approximatively 10 minutes to run. First it will run a series of prerequisite checks, then it will provision several resources on Azure, and the start of the Azure Arc agent installation on your Kubernetes cluster.
```
$ az connectedk8s connect --name "eks-training-dev-krc" --resource-group "aks-training-dev-krc-infra" --location "koreacentral" --correlation-id "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" --tags "Datacenter=Demo City=Demo StateOrDistrict=Demo CountryOrRegion=Demo Scope=Private"
D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\cryptography/hazmat/backends/openssl/backend.py:27: UserWarning: You are using cryptography on a 32-bit Python on a 64-bit Windows Operating System. Cryptography will be significantly faster if you switch to using a 64-bit Python.
This operation might take a while...

The required pre-checks for onboarding have succeeded.
Azure resource provisioning has begun.
Azure resource provisioning has finished.
Starting to install Azure arc agents on the Kubernetes cluster.
{
  "agentPublicKeyCertificate": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
  "agentVersion": null,
  "connectivityStatus": "Connecting",
  "distribution": "eks",
  "id": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/aks-training-dev-krc-infra/providers/Microsoft.Kubernetes/connectedClusters/eks-training-dev-krc",
  "identity": {
    "principalId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    "tenantId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    "type": "SystemAssigned"
  },
  "infrastructure": "aws",
  "kubernetesVersion": null,
  "lastConnectivityTime": null,
  "location": "koreacentral",
  "managedIdentityCertificateExpirationTime": null,
  "name": "eks-training-dev-krc",
  "offering": null,
  "provisioningState": "Succeeded",
  "resourceGroup": "aks-training-dev-krc-infra",
  "systemData": {
    "createdAt": "2024-04-01T01:13:12.973246+00:00",
    "createdBy": "porta.ivan@outlook.com",
    "createdByType": "User",
    "lastModifiedAt": "2024-04-01T01:13:12.973246+00:00",
    "lastModifiedBy": "demo@outlook.com",
    "lastModifiedByType": "User"
  },
  "tags": {
    "Datacenter": "Demo City=Demo StateOrDistrict=Demo CountryOrRegion=Demo Scope=Private"
  },
  "totalCoreCount": null,
  "totalNodeCount": null,
  "type": "microsoft.kubernetes/connectedclusters"
}
```
When you inspect your Kubernetes cluster after running the script, you’ll notice several new resource types. These resources enable a range of functionalities from secure communication to Git repositories configuration for GitOps.
```
$ kubectl api-resources
NAME                              SHORTNAMES      APIVERSION                        NAMESPACED   KIND
...
connectedclusters                                 arc.azure.com/v1beta1             true         ConnectedCluster
...
arccertificates                                   clusterconfig.azure.com/v1beta1   true         ArcCertificate
azureclusteridentityrequests      azidentityreq   clusterconfig.azure.com/v1beta1   true         AzureClusterIdentityRequest
azureextensionidentities                          clusterconfig.azure.com/v1beta1   true         AzureExtensionIdentity
configsyncstatuses                                clusterconfig.azure.com/v1beta1   true         ConfigSyncStatus
customlocationsettings                            clusterconfig.azure.com/v1beta1   true         CustomLocationSettings
extensionconfigs                  ec              clusterconfig.azure.com/v1beta1   true         ExtensionConfig
extensionevents                                   clusterconfig.azure.com/v1beta1   true         ExtensionEvents
gitconfigs                                        clusterconfig.azure.com/v1beta1   true         GitConfig
healthstates                                      clusterconfig.azure.com/v1beta1   true         HealthState
...
```
You’ll also see multiple new pods within the azure-arc namespace. These pods use the resource types prefiously listed to perform specific activities like managing the cluster metadata and ensures it is synchronized with Azure Arc, collection of logs from across the cluster, and more.
```
$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                          READY   STATUS    RESTARTS   AGE
azure-arc     cluster-metadata-operator-7fd8878c5d-k95w4    2/2     Running   0          10m
azure-arc     clusterconnect-agent-6774cfff98-qdvlr         3/3     Running   0          10m
azure-arc     clusteridentityoperator-64d97957c9-p7qpt      2/2     Running   0          10m
azure-arc     config-agent-56445bc8-w2z46                   2/2     Running   0          10m
azure-arc     controller-manager-6996d658f4-5z862           2/2     Running   0          10m
azure-arc     extension-events-collector-6bb86644dc-wqg8m   2/2     Running   0          10m
azure-arc     extension-manager-6cf5c5965d-d4czf            3/3     Running   0          10m
azure-arc     flux-logs-agent-7b8b4d7fcb-wsw6n              1/1     Running   0          10m
azure-arc     kube-aad-proxy-697b664cfc-tnswg               2/2     Running   0          10m
azure-arc     logcollector-9b8f89d57-thhh4                  1/1     Running   0          10m
azure-arc     metrics-agent-68689d8c5d-6vx64                2/2     Running   0          10m
azure-arc     resource-sync-agent-d7dc9449b-th46d           2/2     Running   0          10m
...
```
Several of these pods will also internally communicate with each other and the Azure services via the following services.
```
$ kubectl get services --all-namespaces
NAMESPACE     NAME                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE
azure-arc     extension-events-collector   ClusterIP   10.100.108.124   <none>        8082/TCP                 14m
azure-arc     extension-manager-svc        ClusterIP   None             <none>        8081/TCP                 14m
azure-arc     flux-logs-agent              ClusterIP   10.100.243.81    <none>        80/TCP                   14m
azure-arc     kube-aad-proxy               ClusterIP   10.100.249.91    <none>        443/TCP,8080/TCP         14m
azure-arc     logcollector                 ClusterIP   10.100.216.251   <none>        24224/TCP                14m
...
```
The deployment also creates several Kubernetes secrets in the azure-arc namespace, which store sensitive information such as private keys, certificates, and tokens necessary for secure communication and operation within the Azure Arc environment:
```
$ kubectl get secrets --all-namespaces
NAMESPACE           NAME                                                                                      TYPE                 DATA   AGE
azure-arc-release   sh.helm.release.v1.azure-arc.v1                                                           helm.sh/release.v1   1      12m
azure-arc           azure-arc-connect-privatekey                                                              Opaque               1      12m
azure-arc           azure-identity-certificate                                                                Opaque               3      12m
azure-arc           connectproxy-agent-identity-request-token                                                 Opaque               1      12m
azure-arc           identity-request-2a051a512c1afcd426dd4090206c017a675c0f002bf329cc3165a7ba3abdcc97-token   Opaque               1      12m
azure-arc           kube-aad-proxy-certificate                                                                kubernetes.io/tls    2      12m
```
Non-confidential configuration data required for the operation of Azure Arc like identity requests, Azure-related settings, Fluent Bit are stored in the following Config-Maps.
```
$ kubectl get configmaps --all-namespaces
NAMESPACE           NAME                                                   DATA   AGE
azure-arc-release   kube-root-ca.crt                                       1      17m
azure-arc           azidentityreqleaderid-lock                             0      13m
azure-arc           azure-clusterconfig                                    59     13m
azure-arc           azure-fluentbit-collector-config                       5      13m
azure-arc           azure-fluentbit-config                                 5      13m
azure-arc           azure-telegraf-config                                  1      13m
azure-arc           extension-immutable-values                             0      13m
azure-arc           extensioncontrollerleaderid-lock                       0      13m
azure-arc           gitconfig-immutable-values                             0      13m
azure-arc           gitconfigleaderid-lock                                 0      13m
azure-arc           kube-root-ca.crt                                       1      13m
azure-arc           rsync-datastore                                        2      13m
azure-arc           rsync-datastore-1                                      1      13m
azure-arc           rsync-datastore-10                                     1      13m
azure-arc           rsync-datastore-11                                     1      13m
azure-arc           rsync-datastore-12                                     1      13m
azure-arc           rsync-datastore-13                                     1      13m
azure-arc           rsync-datastore-14                                     1      13m
azure-arc           rsync-datastore-15                                     1      13m
azure-arc           rsync-datastore-16                                     1      13m
azure-arc           rsync-datastore-17                                     1      13m
azure-arc           rsync-datastore-18                                     1      13m
azure-arc           rsync-datastore-19                                     1      13m
azure-arc           rsync-datastore-2                                      1      13m
azure-arc           rsync-datastore-20                                     1      13m
azure-arc           rsync-datastore-21                                     1      13m
azure-arc           rsync-datastore-22                                     1      13m
azure-arc           rsync-datastore-23                                     1      13m
azure-arc           rsync-datastore-24                                     1      13m
azure-arc           rsync-datastore-25                                     1      13m
azure-arc           rsync-datastore-26                                     1      13m
azure-arc           rsync-datastore-27                                     1      13m
azure-arc           rsync-datastore-28                                     1      13m
azure-arc           rsync-datastore-29                                     1      13m
azure-arc           rsync-datastore-3                                      1      13m
azure-arc           rsync-datastore-4                                      1      13m
azure-arc           rsync-datastore-5                                      1      13m
azure-arc           rsync-datastore-6                                      1      13m
azure-arc           rsync-datastore-7                                      1      13m
azure-arc           rsync-datastore-8                                      1      13m
azure-arc           rsync-datastore-9                                      1      13m
...
```
Finally, you can complete the Kubernetes cluster setup for Azure Arc.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cvcdsnic37hon8yv0i2m.png)

Before being able to inspect the EKS cluster resources from the azure portal, we will need to create a Service Account token within your EKS cluster and add it to the Azure portal.
```
$ kubectl create serviceaccount azure-user -n default
```
Next, you need to assign the necessary permissions to your new service account. In this case I liked the `azure-user` service account to the cluster-admin role, which grants it administrative access to the entire cluster.
```
$ kubectl create clusterrolebinding azure-user-binding --clusterrole cluster-admin --serviceaccount default:azure-user
clusterrolebinding.rbac.authorization.k8s.io/azure-user-binding created
```
Then, I’ll create a secret that holds the service account token.
```
$ kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: azure-user-secret
  annotations:
    kubernetes.io/service-account.name: azure-user
type: kubernetes.io/service-account-token
EOF
secret/azure-user-secret created
```
Finally, extract and decrypt the token from the secret you just created.
```
TOKEN=$(kubectl get secret azure-user-secret -o jsonpath='{$.data.token}' | base64 -d | sed 's/$/\n/g')
```
With the token in hand, you can complete the connection between Azure Arc and your Kubernetes cluster.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d4t5jh28n8w5bss16c5w.png)

After validating the token, you will be able to freely view and manage your cluster resources directly from the Azure portal.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xowago54ojvi1ph8yicp.png)

# References
-**Azure Arc**: [https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/](https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/)