Git going with GitOps on AKS: A Step-by-Step Guide using FluxCD AKS Extension
Tags: kubernetes, gitops, fluxcd, kustomize
Published: 2023-09-20 16:00:03

---
series: GitOps on AKS
title: Git going with GitOps on AKS: A Step-by-Step Guide using FluxCD AKS Extension
description: Learn how to harness the power of GitOps on Azure Kubernetes Service (AKS) with ease using the FluxCD AKS Extension. This comprehensive tutorial will walk you through creating an AKS cluster, enabling the FluxCD extension, preparing Kubernetes manifests with Kustomize, deploying applications, and efficiently managing changes, all while following best practices.
published: true
date: 2023-09-20 13:00:00 UTC
tags: kubernetes, gitops, fluxcd, kustomize
canonical_url: https://paulyu.dev/article/git-going-with-gitops-on-aks/
cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0q3oyjq3kzpydmfkkit6.png
---

In reading through [@StevenMurawski](https://dev.to/smurawski)'s blog post titled, [What Really is GitOps?](https://aka.ms/cloudnative/WhatReallyIsGitOps) we learned that GitOps is a way to do [Continuous Delivery](https://learn.microsoft.com/devops/deliver/what-is-continuous-delivery) of our applications on Kubernetes.

In this post, I will jump right into how you can "git" going with GitOps by enabling the [FluxCD AKS Extension](https://learn.microsoft.com/azure/azure-arc/kubernetes/conceptual-gitops-flux2) on your Azure Kubernetes Service (AKS) and using a tool called [Kustomize](https://kustomize.io/) to help with Kubernetes configuration management. 

We'll deploy my new favorite demo app, [AKS Store Demo](https://github.com/Azure-Samples/aks-store-demo) to our AKS cluster and then make some changes to the application and see how FluxCD handles them. 

## Prerequisites

Before you begin, you need to have the following:

- [Azure Subscription](https://azure.microsoft.com/get-started/)
- [GitHub Account](https://github.com/)
- [Azure CLI](https://docs.microsoft.com/cli/azure/install-azure-cli?view=azure-cli-latest)
- [GitHub CLI](https://github.com/cli/cli#installation)
- [kubectl](https://kubernetes.io/docs/tasks/tools/)
- [Kustomize](https://kubectl.docs.kubernetes.io/installation/kustomize/)
- [Flux CLI](https://fluxcd.io/flux/installation/)

Can't do GitOps without Git, so if you are completely new to Git, you should checkout [@StevenMurawski](https://dev.to/smurawski)'s blog post titled, [Just Enough Git for GitOps](https://aka.ms/cloudnative/JustEnoughGit) to have a basic understanding of it.

## Create an AKS cluster

Use the following Azure CLI commands to create a new resource group and AKS cluster.

```bash
RG_NAME=rg-gitops
AKS_NAME=aks-gitops
LOC_NAME=westus3

az group create -n $RG_NAME -l $LOC_NAME
az aks create -n $AKS_NAME -g $RG_NAME
```

When the cluster is created, run the following command to connect to the cluster.

```bash
az aks get-credentials -n $AKS_NAME -g $RG_NAME
```

## Install the FluxCD AKS extension

AKS extensions are Microsoft-maintained packages that can be installed using Azure Resource Manager to enable additional functionality on your AKS cluster. In a way, [AKS extensions](https://learn.microsoft.com/azure/aks/integrations#extensions) are like [AKS add-ons](https://learn.microsoft.com/azure/aks/integrations#add-ons). However, AKS extensions are offered via a different Azure resource provider and therefore offers the flexibility to be installed on a variety of Kubernetes clusters including [Azure Arc enabled Kubernetes](https://learn.microsoft.com/azure/azure-arc/kubernetes/overview) clusters. Take a look at this [article](https://azure.github.io/Cloud-Native/cnny-2023/aks-extensions-addons) by [@JorgeArteiro](https://twitter.com/JorgeArteiro) for more information on the differences between AKS extensions and AKS add-ons.

### What is FluxCD?

[FluxCD](https://fluxcd.io/) is a GitOps tool developed by [Weaveworks](https://www.weave.works/) that allows you to implement continuous and progressive delivery of your applications on Kubernetes. It is a [CNCF graduated project](https://www.cncf.io/announcements/2022/11/30/flux-graduates-from-cncf-incubator/) that offers a set of controllers to monitor Git repositories and reconciles the cluster's actual state with the desired state defined by manifests committed in the repo.

### Install Azure CLI extensions

Before we can install the FluxCD AKS extension using Azure CLI, you need to make sure that you have the proper extensions installed. 

Run the following command to install the Azure CLI extensions for Kubernetes.

```bash
# enable new features for aks
az extension add --name aks-preview

# enable aks extension installations
az extension add --name k8s-extension

# enable aks extension configuration
az extension add --name k8s-configuration
```

### Install FluxCD

With the Azure CLI extensions for Kubernetes installed, you can now install the FluxCD AKS extension using the following command.

```bash
az k8s-extension create \
  --cluster-name $AKS_NAME \
  --resource-group $RG_NAME \
  --cluster-type managedClusters \
  --extension-type microsoft.flux \
  --name aks-store-demo
```

> This command will install the FluxCD AKS extension on your AKS cluster. It is equivalent to running the `flux install` Flux CLI command.

The first parameters are self-explanatory... we are using Azure Resource Manager to install the extension so you need to pass in the AKS cluster name and resource group.

The parameters you need to be aware of here are `--cluster-type` and `--extension-type`. You need to specify `managedClusters` and `microsoft.flux` respectively which tells the Azure resource provider that you are installing the FluxCD extension on a managed AKS cluster. Getting these last parameters wrong will result in an error.

With the extension installed, you can run the following Flux CLI command to get the status of the installation.

```bash
flux check
```

Flux installs many new Custom Resource Definitions (CRDs) in the cluster. These CRDs are how you interact with FluxCD. You can run the following command to see all the new CRDs.

```bash
kubectl get crds | grep flux
```

Next, we need to connect FluxCD to a Git repository, but there's some prep work we need to do first.

## Prepare Kubernetes manifests for Kustomize

A successful GitOps implementation hinges on how well you structure your Git repository and how well you manage processes and workflows. GitOps tools are just that; they're tools, you need to do some work up front like determining file structure, branching strategy, branch protections, and workflows to ensure that you are setting yourself up for success.

### "Kustomizing" manifests

The first thing we need to do is decide on how we want to package and manage Kubernetes manifests for different environments. There's a tool called [Helm](https://helm.sh) which [@StevenMurawski](https://dev.to/smurawski) covered [here](https://aka.ms/cloudnative/GettingStartedWithHelm), and another tool called [Kustomize](https://kustomize.io) which I personally like to use. Both tools are great for packaging and managing Kubernetes manifests and also supported by FluxCD, but I prefer Kustomize because it is pretty easy to use and built into `kubectl`.

Kustomize is not a templating engine like Helm, it is more like a patching engine. It allows you to create a base set of Kubernetes manifests and then patch them with environment specific configurations. These environmental configurations are known as "overlays". So you can have a `dev` overlay, a `prod` overlay, a `staging` overlay, etc. and use them to patch the base manifests with environment specific configurations.

### Create a GitHub repository

Using the GitHub CLI, log into GitHub.

```bash
gh auth login
```

Next fork and clone the [aks-store-demo-manifests](https://github.com/pauldotyu/aks-store-demo-manifests) repository that I created for this tutorial.

```bash
gh repo fork https://github.com/pauldotyu/aks-store-demo-manifests.git --clone

# change into the repo directory
cd aks-store-demo-manifests
```

### Create a `base` overlay

All of our manifests currently sit a the root of the repo so we need to create a `base` directory so that we can create overlays for different environments.

Run the following command to create a `base` directory.

```bash
mkdir base
```

Copy all the manifests from the root of the repo into the `base` directory using the following command:

```bash
mv *.yaml base

# change into the base directory
cd base
```

We should now have a `base` directory with all the manifests in it. Next we need to create our initial `kusomtization.yaml` file.

Using the Kustomize CLI, run the following command to create the base `kustomization.yaml` file:

```bash
kustomize create --autodetect

# view the kustomization.yaml file
cat kustomization.yaml
```

Notice in the `kustomization.yaml` file that all of our manifest files were added as resources that Kustomize will patch.

### Create a `dev` overlay

Next, we need to create an overlay for our `dev` environment.

We need to navigate back to the root of the repo directory and create a `dev` overlay directory.

```bash
# navigate back to the root of the repo
cd ../
mkdir -p overlays/dev
cd overlays/dev
```

We want our dev deployment to deploy to a new namespace so we'll create a new manifest to create one. Run the following command to create a new manifest file.

```bash
kubectl create namespace store-dev --dry-run=client -o yaml > namespace.yaml
```

Generate the `kustomization.yaml` file for our `dev` overlay using the following command.

```bash
kustomize create --resources namespace.yaml,./../../base --namespace store-dev

# view the kustomization.yaml file
cat kustomization.yaml
```

In this `kustomization.yaml` file, we see that it will include our new `namespace.yaml` file and all manifests in the `base` directory. We also have a `namespace: store-dev` entry in the kustomization and this is what instructs Kustomize to patch and add the `store-dev` namespace to the base manifests.

Here is what the directory structure of the repo should look like:

```text
.
├── README.md
├── base
│   ├── kustomization.yaml
│   ├── makeline-service.yaml
│   ├── mongodb.yaml
│   ├── order-service.yaml
│   ├── product-service.yaml
│   ├── rabbitmq.yaml
│   ├── store-admin.yaml
│   ├── store-front.yaml
│   ├── virtual-customer.yaml
│   └── virtual-worker.yaml
└── overlays
    └── dev
        ├── kustomization.yaml
        └── namespace.yaml
```

This is good enough for now.

### Kustomize in action

Let's test the `dev` overlay by running the following command from the `dev` directory:

```bash
kustomize build
```

Notice how all the manifests are patched with the `store-dev` namespace and output to the console. This is how Kustomize works. It patches the base manifests with environment specific configurations and outputs the patched manifests to the console.

To deploy these patched manifests, you would run a command like this.

```bash
kustomize build | kubectl apply -f -
```

Kustomize is also built into `kubectl` so you can run the following command to apply the manifests to your cluster.

```bash
kubectl apply -k .
```

Hopefully you didn't run the commands above. If you did, no sweat 😅 you can delete this deployment with either of these commands.

```bash
# using kustomize
kustomize build | kubectl delete -f -

# or using kubectl
kubectl delete -k .
```

### Do the Git part of GitOps

Remember that Git stuff we talked about earlier? Well, let's put Git to use and commit and push our changes the GitHub repo.

Run the following commands to commit and push your changes to GitHub.

```bash
# make sure we are back at the root of the repo
cd ../../

# add all changes
git add .

# commit changes
git commit -m 'refactor: add kustomize base and dev overlay'

# push changes
git push
```

## Deploying Applications with FluxCD

With the repo prepped, we can now connect FluxCD to our GitHub repo. FluxCD will monitor the repo for changes and reconcile the cluster with the desired state defined in the repo.

Run the following command to get the GitHub HTTP URL for your repo.

```bash
GH_REPO_URL=$(gh repo view --json url | jq .url -r)
```

Using Azure CLI again, let's configure the FluxCD AKS extension to connect to our GitHub repo:

```bash
az k8s-configuration flux create \
  --cluster-name $AKS_NAME \
  --resource-group $RG_NAME \
  --cluster-type managedClusters \
  --name aks-store-demo \
  --url $GH_REPO_URL \
  --branch main \
  --kustomization name=dev path=./overlays/dev \
  --namespace flux-system
```

> This command is equivalent to running the `flux create source` and `flux create kustomization` Flux CLI commands.

Here, we are telling the FluxCD AKS extension to connect to our GitHub repo and monitor the `main` branch for changes. 

The Azure resource name is set to `aks-store-demo` and we pass in a kustomization name of `dev`. These two values will be used to create the `Kustomization` resources in the cluster.

We also tell the Flux resource where look for our manifests by passing it `path=./overlays/dev`. 

Lastly, we tell Flux to create new Flux `GitRepository` and `Kustomization` resources in the `flux-system` namespace. You can change this to whatever namespace you want. I used `flux-system` for simplicity. 

If you run the following Flux CLI commands you should see some resources created.

```bash
flux get source git
flux get kustomization
```

If all went well, you should see your pods coming online. Let's watch for them:

```bash
kubectl get pods -n store-dev -w
```

Once you see all the pods running, you can exit the watch by pressing `CTRL+C`.

Let's test the application by grabbing the public IP address of the `store` service:

```bash
kubectl get svc/store-front -n store-dev
```

Open a browser and navigate to the IP address. You should see the AKS Store Demo application running 🚀

![AKS Store Demo website](https://assets.paulyu.dev/articles/2023/git-going-with-gitops-on-aks/aks-store-demo.png)

Great, we have successfully deployed our application using GitOps! Now what? Well, let's make some changes to the application and see how FluxCD handles them.

## Making and managing changes

With the FluxCD AKS extension installed and connected to our GitHub repo, we can now make changes by simply editing the kubernetes manifests and committing/pushing the changes back to the remote repo. At this point, it's all about Git workflows and processes.

Let's make a small change to the dev overlay `kustomization.yaml` file. Let's say we want to change the name of the namespace from `store-dev` to just `dev`. 

Open the `overlays/dev/kustomization.yaml` file and change the `namespace` value from `store-dev` to `dev`. Optionally, you can run the following `sed` command to make the change.

```bash
# make sure you are in the root of the repo
sed -i -e 's/store-dev/dev/g' overlays/dev/kustomization.yaml
```

Compare the changes.

```bash
git diff overlays/dev/kustomization.yaml
```

Add the change, commit, and push to GitHub.

```bash
git add overlays/dev/kustomization.yaml
git commit -m 'refactor: change namespace to dev'
git push
```

I'm pushing directly to the `main` branch for simplicity, but you should be using a branch and pull request workflow.

Using the Flux CLI, you can force FluxCD to reconcile the cluster with the desired state defined in the repo.

```bash
flux reconcile kustomization aks-store-demo-dev --with-source
```

After a minute or two you should see the pods coming online in the new `dev` namespace. This is FluxCD reconciling the cluster with the desired state defined in the repo. 

You can check on the pods using the following command.

```bash
kubectl get pods -n dev

# type CTRL+C to exit watch
```

## Monitoring and Troubleshooting

Here's some tips when it comes to monitoring and troubleshooting Flux resources is to use the Flux CLI. You can use some of these basic commands to get information about Flux and its resources:

```bash
# check the status of the flux installation
flux check

# get info about the GitRepository resource
flux get source git aks-store-demo-dev -n flux-system

# get info about the Kustomization resource
flux get kustomization aks-store-demo-dev -n flux-system

# view event logs from the flux controllers
flux events

# view logs from the flux controllers
flux log

# view stats of the flux controllers
flux stats
```

If you are a little weary about an agent in your Kubernetes cluster doing all this work without your knowledge, don't worry, Flux also allows you to configure notifications for events. You can configure notifications for Slack, Discord, Microsoft Teams, and more. Check out the [Flux docs](https://fluxcd.io/docs/components/notification/) for more information.

## Conclusion

In this post, we learned how to enable the FluxCD AKS extension on Azure Kubernetes Service (AKS) and how to deploy applications using a GitOps approach. We did all this using the Azure CLI and the Flux CLI, but you can also use the Azure Portal to install the AKS extension in a "ClickOps" manner 😁

GitOps is a powerful way to do Continuous Delivery of your applications on Kubernetes. But remember, a successful GitOps implementation requires a bit of planning and prep work. You need to think about how you want to manage processes and workflows using a tool like Git. GitOps is not a "silver bullet", it is a way of doing things. It is a way of doing Continuous Delivery of your applications on Kubernetes.

Now, you might be thinking... "Why FluxCD and not ArgoCD?" Well, I will be doing a follow up post on [ArgoCD](https://argo-cd.readthedocs.io/en/stable/), so stay tuned for that.

If you have any feedback or suggestions, please feel free to reach out to me on [Twitter](https://twitter.com/pauldotyu) or [LinkedIn](https://www.linkedin.com/in/yupaul).

Peace ✌️

## References

- [Flux - the GitOps family of projects](https://fluxcd.io/)
- [GitOps Flux v2 configurations with AKS and Azure Arc-enabled Kubernetes](https://learn.microsoft.com/azure/azure-arc/kubernetes/conceptual-gitops-flux2)
- [Tutorial: Deploy applications using GitOps with Flux v2](https://learn.microsoft.com/azure/azure-arc/kubernetes/tutorial-use-gitops-flux2?tabs=azure-cli)
- [Cluster extensions](https://learn.microsoft.com/azure/azure-arc/kubernetes/conceptual-extensions)
- [Flux Graduates from the Cloud Native Computing Foundation Incubator](https://www.cncf.io/announcements/2022/11/30/flux-graduates-from-cncf-incubator/)
- [What is OpenGitOps?](https://opengitops.dev/)