Getting started with Windows Containers on Azure Kubernetes Service
Tags: azure, kubernetes, powershell, docker
Published: 2022-01-20 22:40:24

Windows support has finally arrived in Kubernetes and AKS. Learn how to migrate your workloads and what pitfalls to avoid in this short and sweet introduction to Windows Containers. 

[Windows support is relatively new](https://cloudblogs.microsoft.com/windowsserver/2020/04/28/windows-server-container-support-in-azure-kubernetes-service-is-now-generally-available/) to AKS and has been stable for about a year at the time of this article's writing. I've recently had to do a deep dive into the technology for a project, migrating legacy applications that still require the full .NET 4.8 Framework. While Kubernetes deployment and image building are quite similar to Linux-based containers, there are several key differences.


![A diagram of AKS.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/u15vvg71iod80byjet9f.png)

Kubernetes on Windows is a great way to make older .NET/IIS applications, Windows console apps and services more flexible, highly available and scalable using native Kubernetes and cloud technologies. Azure Kubernetes Service takes away the overhead of managing servers or VM images, and gives you a single pane of glass solution for both your Linux and Windows deployments.

#### Image Versions
Microsoft offers four different base images on [Docker Hub for Windows Server containers](https://docs.microsoft.com/en-us/virtualization/windowscontainers/manage-containers/container-base-images), which they also use in their collection of IIS and ASP.NET images:

- [windows/nanoserver](https://hub.docker.com/_/microsoft-windows-nanoserver) is ultra-light at a mere 112MB. It only runs .NET Core out of the box, but PowerShell Core can be installed during the image build.
- [windows/servercore](https://hub.docker.com/_/microsoft-windows-servercore) is 1.2GB in size and a good option for "lifting and shifting" Windows Server apps. It supports everything regular Windows Server 2019 Core offers.
- [windows/server](https://hub.docker.com/_/microsoft-windows-server) weighs in at a hefty 3.1GB, has full Windows API support, and allows you to use all Windows Server features. This image can only be pulled on Windows 11 and Windows Server 2022.
- [windows](https://hub.docker.com/_/microsoft-windows) is the largest image at 3.4GB. It has full Windows API support, but lacks native server features. This is Windows 10 under the hood. RDP connections are not possible out of the box with this image, so it cannot be used as an Azure Virtual Desktop.

_Note: These are compressed sizes, unpacked, the size may be up to 2–5 times larger._

Important to remember is that major version tags such as ltsc2019, 1909, and 20H2 are updated automatically [every second Tuesday of the month](https://docs.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/update-containers) at 00:00 UTC. For granular control over your Windows Update cycle, refer to KB or version-specific tags.

#### Memory allocation
Windows does not have an out-of-memory process killer as Linux does. Windows always treats all user-mode memory allocations as virtual, and pagefiles are mandatory. Windows nodes do not overcommit memory for processes running in containers. The net effect is that Windows won’t reach out of memory conditions the same way Linux does, and processes page to disk instead of being subject to out-of-memory (OOM) termination. If memory is over-provisioned and all physical memory is exhausted, then paging can slow down performance.

---

## Prerequisites
To follow along with the practical section, you’ll need a couple of tools:
- A Windows 10/11 PC/VM with Virtualization enabled
- [Docker Desktop, with Windows Containers enabled](https://github.com/rdvansloten/windows-containers-demo/blob/main/tools/docker-desktop.md)
- [An Azure account](https://azure.microsoft.com/en-us/free/)
- [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli)
- [kubectl](https://kubernetes.io/docs/tasks/tools/)
- [PowerShell](https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell?view=powershell-7.2)

Every resource used in this article is available in this git repository:
[rdvansloten/windows-containers-demo](https://github.com/rdvansloten/windows-containers-demo). You can run the scripts and Dockerfiles by git cloning it to your local machine/server:

```plaintext
git clone https://github.com/rdvansloten/windows-containers-demo.git
```

---

## Creating Azure resources

#### Quick version
If you don’t want to explore and create the resource setup step-by-step, there’s an all-inclusive, run-once PowerShell script to set up all required resources. `cd` into windows-containers-demo/powershell and execute deployment.ps1. Check out the brief [README.md](https://github.com/rdvansloten/windows-containers-demo/blob/main/powershell/README.md) in that folder for the script's arguments. A login window may pop up to verify your Azure credentials. 

Skip to **[Creating your Dockerfile](#creating-dockerfile)** after the deployment script is finished.

#### Long version
Let’s start off by logging into Azure via the CLI. To execute the scripts in this article, you’ll need to open PowerShell as an Administrator. This is required for interacting with the Docker daemon. macOS/Linux users may use the command `pwsh` before attempting to create these resources.


```posh
# Login to Azure
az login
# List and select the right Subscription
az account show
az account set --subscription "<YOUR SUBSCRIPTION ID/NAME HERE>"
```

Set the below variables in your terminal if you intend to run all the other snippets below. Replace the $AZUREUSER value with your Azure account’s email address.

```posh
# Generate a random number for uniqueness
$RANDOM = Get-Random -Minimum -100000 -Maximum 999999

$AZUREUSER      = "yourEmail@domain.com" # Your Azure account email
$NODEPOOLNAME   = "win"                  # Max 6 characters
$LOCATION       = "westeurope"           # The region you prefer
$RESOURCEGROUP  = "rg-aks"               # Name of your Resource Group
$CLUSTERNAME    = "myaks$($RANDOM)"      # Must be globally unique
```

After logging in, you start off by creating the basics; a Resource Group to hold your infrastructure and an Identity that AKS will use to talk to other services, such as the Virtual Network, Container Registry, and VM Scale Sets.

```posh
# Create Resource Group
az group create `
  --name $RESOURCEGROUP `
  --location $LOCATION
  
# Create Managed Identity
az identity create `
  --name $CLUSTERNAME-id `
  --resource-group $RESOURCEGROUP `
  --location $LOCATION
  
# Store Managed Identity ID in variable for AKS creation
$AKSIDENTITY = $(
  az identity show `
    --name $CLUSTERNAME-id `
    --resource-group $RESOURCEGROUP `
    --query id
)
```

#### Virtual Network
Let’s create a Virtual Network with enough IP addresses to accommodate a lot of Pods, Services, and Ingresses. Also, make sure that the AKS identity can create network resources, or your components inside Kubernetes will not be able to be provisioned.

```posh
# Create VNET and subnet
az network vnet create `
  --name $CLUSTERNAME-vnet `
  --resource-group $RESOURCEGROUP `
  --location $LOCATION `
  --address-prefixes 172.10.0.0/16 `
  --subnet-name kubernetes `
  --subnet-prefixes 172.10.128.0/17
  
# Store subnet ID in variable for AKS creation
$SUBNETID = $(
  az network vnet subnet list `
    --resource-group $RESOURCEGROUP `
    --vnet-name $CLUSTERNAME-vnet `
    --query "[0].id"
)

# Grant AKS identity VNET access for Azure CNI
az role assignment create `
  --assignee-object-id $(
    az identity show `
      --name $CLUSTERNAME-id `
      --resource-group $RESOURCEGROUP `
      --query principalId
    )`
  --scope $(
    az network vnet show `
      --resource-group $RESOURCEGROUP `
      --name $CLUSTERNAME-vnet `
      --query id
    )`
  --assignee-principal-type ServicePrincipal `
  --role "Network Contributor"
```

---

## Creating the AKS cluster
Now that the prerequisites have been deployed, you can finally build your AKS cluster.

#### Linux && Windows
Even though we want a Windows cluster, AKS still requires a Linux node for the control plane. Let’s set the single Linux node to one of the cheapest VM sizes, Standard_B2s (as it won’t be doing much work), and assign the Windows node pool a Standard_D2s_v4 node, sporting 2 vCPU cores and 7GB of RAM.

#### Networking
You’ll select the virtual network you’ve created in the previous step and define a few private ranges for AKS to use. Pods and Nodes will be assigned an address in the 172.10. range. Services and system resources, such as CoreDNS, will operate on a private 10. range. For testing purposes, let’s let Azure worry about public DNS and Ingress. The addon `http_application_routing` will provide these resources for you. You will only need to add an annotation to your application to get it published on the internet.

_Please note that the http application routing addon is not meant for production usage. Check out alternatives like [Traefik](https://github.com/traefik/traefik) for production workloads._

#### Azure CNI
In order for each Pod to have its own address, you will want to deploy Azure CNI rather than Kubenet for your networking layer. Some benefits of Azure CNI over Kubenet:

- No NAT, so the source of the packets is retained.
- Workloads can be scaled out to [Virtual Nodes](https://docs.microsoft.com/en-us/azure/aks/virtual-nodes).
- Virtual Networks can be managed separately/independently.
- Each Pod gets an IP address from the subnet, making debugging easier.
- Most importantly, **Windows node pools support Azure CNI only**. 

Azure CNI also forces you to plan out your Virtual Network beforehand, which helps enormously with network architecture and capacity planning.

```posh
# Assign AKS Kubelet Identity permissions on the AKS Resource Group
az role assignment create `
  --assignee-object-id $(
    az identity show `
      --name $CLUSTERNAME-id `
      --resource-group $RESOURCEGROUP `
      --query principalId
  )`
  --scope $(
    az group show `
      --name $(
        az group show `
          --resource-group $RESOURCEGROUP `
          --query name
      ) `
      --query id
    )`
  --assignee-principal-type ServicePrincipal `
  --role "Contributor"

# Create AKS cluster
az aks create `
  --resource-group $RESOURCEGROUP `
  --name $CLUSTERNAME `
  --location $LOCATION `
  --assign-identity "$AKSIDENTITY" `
  --assign-kubelet-identity "$AKSIDENTITY" `
  --node-vm-size Standard_B2s `
  --node-count 1 `
  --enable-aad `
  --enable-azure-rbac `
  --network-plugin azure `
  --vnet-subnet-id "$SUBNETID" `
  --docker-bridge-address 172.17.0.1/16 `
  --dns-service-ip 10.2.0.10 `
  --service-cidr 10.2.0.0/24 `
  --generate-ssh-keys `
  --enable-addons http_application_routing
  
# Add Windows node pool
az aks nodepool add `
  --name $NODEPOOLNAME `
  --resource-group $RESOURCEGROUP `
  --cluster-name $CLUSTERNAME `
  --node-vm-size Standard_D2s_v4 `
  --node-count 1 `
  --os-type Windows `
  --max-surge 33%
  
# Assign yourself Admin permissions on the AKS cluster
az role assignment create `
  --assignee $(
    az ad user show `
      --id $(
        az ad user list `
          --query "[?contains(@.otherMails,'$AZUREUSER')].userPrincipalName " -o tsv
      ) `
      --query userPrincipalName 
  ) `
  --scope $(
    az aks show `
      --name $CLUSTERNAME `
      --resource-group $RESOURCEGROUP `
      --query id
  )`
  --role "Azure Kubernetes Service RBAC Cluster Admin"

az role assignment create `
  --assignee $(
    az ad user show `
      --id $(
        az ad user list `
          --query "[?contains(@.userPrincipalName,'$AZUREUSER')].userPrincipalName " -o tsv
      ) `
      --query userPrincipalName 
  ) `
  --scope $(
    az aks show `
      --name $CLUSTERNAME `
      --resource-group $RESOURCEGROUP `
      --query id
  )`
  --role "Azure Kubernetes Service RBAC Cluster Admin"
  
# Assign AKS identity permissions to node pool Resource Group
az role assignment create `
  --assignee-object-id $(
    az identity show `
      --name $CLUSTERNAME-id `
      --resource-group $RESOURCEGROUP `
      --query principalId
  )`
  --scope $(
    az group show `
      --name $(
        az aks show `
          --name $CLUSTERNAME `
          --resource-group $RESOURCEGROUP `
          --query nodeResourceGroup
      ) `
      --query id
    )`
  --assignee-principal-type ServicePrincipal `
  --role "Contributor"
```

#### Permissions, RBAC
The AKS deployment script finishes off by assigning the AKS identity permissions to create resources in the special “MC_” Resource Group that AKS generates during creation. This is required for it to interactively spawn resources. Contributor is a very broad permission set and should not be used lightly in production scenarios.

Consult [Microsoft Docs: AKS Cluster Identity Permissions](https://docs.microsoft.com/en-us/azure/aks/concepts-identity) for an exhaustive list of permissions you may want to apply/deny when setting up AKS.

During the last script, you’ve also applied the Azure Kubernetes Service RBAC Cluster Admin role to your own Azure AD user. This is required because of the `--enable-aad` and `--enable-azure-rbac arguments` passed to `az aks create`. These arguments allow Azure AD/RBAC objects to be added to Kubernetes (Cluster)Roles.

---

## Deploying a Windows Server container image
If you don’t have your own container repository, let’s create one and give the AKS identity permissions to pull images from it. The [Azure Container Registry](https://azure.microsoft.com/en-us/services/container-registry/) is a private container repository that includes security scanning, image management, and authentication. It also has experimental support for Helm chart storage.

```posh
# Create Container Registry
az acr create `
  --resource-group $RESOURCEGROUP `
  --name "$($CLUSTERNAME)acr" `
  --sku Basic
  
# Add AKS to Container Registry
az role assignment create `
  --assignee-object-id $(
    az identity show `
      --name $CLUSTERNAME-id `
      --resource-group $RESOURCEGROUP `
      --query principalId
    )`
  --scope $(
    az acr show `
      --name "$($CLUSTERNAME)acr" `
      --query id
    )`
  --assignee-principal-type ServicePrincipal `
  --role "AcrPull"
```

---

## Creating your Dockerfile <a name="creating-dockerfile"></a>
With AKS up and running, you can now run Windows containers in the cloud. Start off by creating a Dockerfile and uploading the resulting image to an Azure Container Registry.

For this demo, I’ve created a simple PowerShell script for the container to loop. `cd` into the docker/powershell/ folder inside the cloned [windows-containers-demo](https://github.com/rdvansloten/windows-containers-demo) repo. 

Then, run `az acr build` inside that folder to upload it to your ACR:

```posh
az acr build `
  --image myapp:latest `
  --platform Windows `
  --registry "$($CLUSTERNAME)acr" .
```

This command combines several Docker and Azure commands to save time/effort. You can tag it however you want, and use the tag when deploying your app. For this example, I use `latest`.

Optionally, you may also test your newly uploaded image locally on your Windows machine using `docker run`:

```posh
# Uses the docker CLI with your Azure login
az acr login --name "$($CLUSTERNAME)acr"

# Run the container locally on port 5000
docker run `
  --publish 5000:5000 `
  --interactive `
  --detach "$($CLUSTERNAME)acr.azurecr.io/myapp:latest"
```

---

## Deploying your app
Now that your Windows nodes are up and running, and the image is uploaded to the Container Registry, you can deploy it to AKS using `kubectl`. For this demo’s purposes, I kept it simple with a the deploy.yaml file in the kubernetes/ folder you can feed into kubectl.

Before applying this file, `cd` into kubernetes/ and open deploy.yaml in a text editor like [Visual Studio Code](https://code.visualstudio.com/) or Notepad, and modify lines 20 and 68:

```yaml
host: myapp.<CLUSTER_DNS_ZONE>
image: <ACR_NAME>.azurecr.io/myapp:latest
```

Fill out the placeholder values with the resources you’ve created. For me, it looked something like this:

```yaml
host: myapp.a430e1209ded44679aab.westeurope.aksapp.io
image: myaks321123.azurecr.io/myapp:latest
```

The value for `image: <ACR_NAME>` can be found by running:

```posh
az acr list `
  --resource-group $RESOURCEGROUP `
  --query "[0].name" -o tsv
```

The value for `host: <CLUSTER_DNS_ZONE>` can be found by running:

```posh
az network dns zone list `
  --resource-group $(
    az group show `
      --name $(
        az aks show `
          --name $CLUSTERNAME `
          --resource-group $RESOURCEGROUP `
          --query nodeResourceGroup
      ) `
      --query name
  )`
--query "[0].name" -o tsv
```

`cd` into the kubernetes/ folder. Then, use the following snippet to log in to your AKS cluster, and deploy your image:

```posh
# Load AKS details into ~/.kube/config
az aks get-credentials `
  --name $CLUSTERNAME `
  --resource-group $RESOURCEGROUP
  
# Check if AKS is reachable and Windows node is up
kubectl get nodes

# Deploy yaml manifest to AKS
kubectl apply -f deploy.yaml

# Get the URL for your app
kubectl get ingress myapp -o=jsonpath='{.spec.rules[0].host}'
```

Due to the size of the servercore:ltsc2019 image, the first pull might take 5–10 minutes. Wait for a while and watch for logs or errors using `kubectl describe deployment myapp`. I recommend tracking the progress using a GUI app, such as [Lens Desktop](https://k8slens.dev/).

Open your browser and go to the URL (something.some-region.aksapp.io) presented after running the `kubectl get ingress` command above. You will now be able to view your first Windows application running on Kubernetes!

![Screenshot of a web browser.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/arlor78gu1ty55rgwy4d.png)

#### Caveats
In contrast to the the benefits of Kubernetes on Windows, there are a few drawbacks to using Windows Containers in AKS:

- Node pools are prohibitively more expensive than Linux node pools due to licensing. Bringing your own license with AHUB may alleviate some pain.
- Windows-based Pods have limited CSI Driver support.
- Windows versions of your DaemonSets are required. (monitoring, drivers)

Be mindful of the costs of running this setup. The above configuration will cost around $10/day, meaning you can only run it for 2–3 weeks on an Azure free account.

| Service                   | Size                  | $/day |
| ------------------------- | --------------------- | -----:|
| Azure Kubernetes Service  | Linux, Standard_B2s   | $1.15 |
| Azure Container Registry  | Basic, 25GB Bandwidth | $1.37 |
| Virtual Machine Scale Set | Windows, Standard_D2s | $7.13 |

Run the `az aks stop` command to stop AKS from draining your wallet, or delete the cluster in its entirety when you are done with it.

---

## Conclusion
With this, you are running your first basic Windows application in Kubernetes. I hope I’ve equipped you with some basic knowledge and tools to start tackling these issues. For a more production-ready environment, I would suggest looking into leveraging Terraform or ARM for infrastructure deployment, enterprise-grade Ingress Controllers, and setting your cluster, registry, and all its dependencies to private.

For any questions or inquiries, feel free to [reach out to me on LinkedIn](https://www.linkedin.com/in/rdvansloten/) or in the comments!

---

<sub>**Sources**</sub>
<sub>[kubernetes.io: Windows containers in Kubernetes](https://kubernetes.io/docs/setup/production-environment/windows/intro-windows-in-kubernetes/)</sub>
<sub>[Microsoft Windows Server Blog: Windows Server container support in AKS is now generally available](https://cloudblogs.microsoft.com/windowsserver/2020/04/28/windows-server-container-support-in-azure-kubernetes-service-is-now-generally-available/)</sub>
<sub>[Docker Hub: Windows base OS images](https://hub.docker.com/_/microsoft-windows-base-os-images)</sub>
<sub>[blog.sixeyed.com: Getting Started with Kubernetes on Windows](https://blog.sixeyed.com/getting-started-with-kubernetes-on-windows/)</sub>