Bootstrap your GitOps-enabled AKS cluster with Terraform: A code sample using the Flux v2 K8s Extension
Tags: tutorial, kubernetes, terraform, flux
Published: 2023-09-28 14:51:39

---
series: GitOps on AKS
title: Bootstrap your GitOps-enabled AKS cluster with Terraform: A code sample using the Flux v2 K8s Extension
description: Learn how to bootstrap GitOps onto your AKS cluster, the Azure way!
published: true
date: 2023-09-28 13:00:00 UTC
tags: tutorial, kubernetes, terraform, flux
canonical_url: https://paulyu.dev/article/bootstrap-gitops-with-aks-extension-using-terraform/
cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ztl3hso23hoc2mmctxt4.png
---

In my previous posts, we learned how to [get started with GitOps on AKS](https://aka.ms/cloudnative/GitGoingWithGitOps) using the K8s extension for AKS.

Then, we took a look at the Flux CLI and explored how it can be used to bootstrap your cluster and generate FluxCD manifests so that we can use GitOps to implement GitOps 🤯, and implemented Flux's [image update automation](https://aka.ms/cloudnative/ImageAutomationWithFluxCD) capability.

From there, we built on the concept of image update automation, and showed you how you can use Flagger to [automate canary deployments](https://aka.ms/cloudNative/ProgressiveDeliveryWithFlagger).

Much of this was done using the Flux CLI to generate Flux CRD manifests, but you might be thinking...

**_"All that Flux CLI stuff is cool, but I want to bootstrap my AKS cluster to use GitOps, using the AKS extension!"_**

In this post, I'll show you how to do just that... with my favorite IaC tool, Terraform!

## What's in the box?

To get started, head over to my [GitHub repo](https://aka.ms/cloudnative/bootstrap-flux-v2-on-aks-with-terraform) and fork/clone it. This is where I have all the Terraform code you need to deploy and bootstrap your AKS cluster using a single `terraform apply` command.

At a high-level, the Terraform code will do the following:

- Create a resource group
- Create an AKS cluster with the Istio service mesh add-on and external ingress gateway enabled
- Create a Kubernetes Secret in the `flux-system` namespace to store GitHub credentials for private Git repository access
- Install the Flux v2 AKS extension with the following components enabled:
  - `image-automation-controller`
  - `image-reflector-controller`
  - `notification-controller`
- Configure the Flux v2 AKS extension with the following Kustomizations:
  - The `dev-app` Kustomization deploys the `GitRepository` resource which points to the `flux-k8s-ext` branch in my [AKS Store Demo Manifests](https://github.com/pauldotyu/aks-store-demo-manifests/tree/flux-k8s-ext) repo, and `Kustomization` resource to deploy the [AKS Store Demo](https://github.com/Azure-Samples/aks-store-demo) application
  - The `dev-image` Kustomization deploys the `ImageRepository`, `ImagePolicy` and `ImageUpdateAutomation` resources to automate image updates for the `store-front` application; this Kustomization has a dependency on the `dev-app` Kustomization
  - The `dev-flagger` Kustomization deploys the `HelmRepository`, `HelmChart`, and `HelmRelease` to install Flagger along with the `OCIRepository` and `Kustomization` resource to install the Flagger Loadtester; this Kustomization also has a dependency on the `dev-app` Kustomization
  - The `dev-canary-store-front` Kustomization deploys the `Canary` resource that is used to automate progressive delivery of the `store-front` application; this Kustomization has a dependency on the `dev-flagger` Kustomization

As an added bonus, I've also included Azure Managed Grafana and Azure Managed Prometheus in case you wanted to explore the [monitoring capabilities Flux on AKS](https://learn.microsoft.com/azure/azure-arc/kubernetes/monitor-gitops-flux-2).

To deploy, simply follow the instructions found in the [README](https://aka.ms/cloudnative/bootstrap-flux-v2-on-aks-with-terraform) of the repo.

## Kustomizations, Kustomizations, Kustomizations...

There are some differences between the way Flux CLI bootstraps your GitOps cluster and the way you can bootstrap your AKS cluster using the AKS extension.

When Flux CLI bootstraps a cluster, it creates folders called `/clusters/dev` and `/clusters/dev/flux-system`. The `/clusters/dev/flux-system` folder contains manifests that are used to install Flux and store `GitRepository` source used for bootstrapping.

Here is what the directory structure looked like when we bootstrapped using Flux CLI.

```text
.
├── README.md
├── base
│   ├── kustomization.yaml
│   ├── makeline-service.yaml
│   ├── mongodb.yaml
│   ├── order-service.yaml
│   ├── product-service.yaml
│   ├── rabbitmq.yaml
│   ├── store-admin.yaml
│   ├── store-front.yaml
│   ├── virtual-customer.yaml
│   └── virtual-worker.yaml
├── clusters
│   └── dev
│       ├── aks-store-demo-kustomization.yaml
│       ├── aks-store-demo-source.yaml
│       ├── aks-store-demo-store-front-image-policy.yaml
│       ├── aks-store-demo-store-front-image-update.yaml
│       ├── aks-store-demo-store-front-image.yaml
│       ├── flagger-helmrelease.yaml
│       ├── flagger-loadtester-kustomization.yaml
│       ├── flagger-loadtester-source.yaml
│       ├── flagger-source.yaml
│       └── flux-system
│           ├── gotk-components.yaml
│           ├── gotk-sync.yaml
│           └── kustomization.yaml
└── overlays
    └── dev
        ├── kustomization.yaml
        └── namespace.yaml

7 directories, 25 files
```

The `/clusters/dev/flux-system/gotk-sync.yaml` file is the `Kustomization` resource that Flux uses to configure the cluster. It tells Flux to deploy all resources found in the `/clusters/dev` directory.

With the AKS extension, the `gotk-*` manifests are not saved or used from our repo, so we can discard them. However, in order to deploy the resources, we do need to make Flux aware that we want these resources deployed.

The files in the `/clusters/dev` directory needs to be split out and installed as separate Kustomizations.

Here is what the directory structure ends up looking like to bootstrap a cluster using the AKS extension.

```text
.
├── README.md
├── base
│   ├── kustomization.yaml
│   ├── makeline-service.yaml
│   ├── mongodb.yaml
│   ├── order-service.yaml
│   ├── product-service.yaml
│   ├── rabbitmq.yaml
│   ├── store-admin.yaml
│   ├── store-front.yaml
│   ├── virtual-customer.yaml
│   └── virtual-worker.yaml
├── clusters
│   └── dev
│       ├── flagger
│       │   ├── flagger-helmrelease.yaml
│       │   ├── flagger-loadtester-kustomization.yaml
│       │   ├── flagger-loadtester-source.yaml
│       │   ├── flagger-source.yaml
│       │   └── kustomization.yaml
│       └── image-update
│           ├── aks-store-demo-store-front-image-policy.yaml
│           ├── aks-store-demo-store-front-image-update.yaml
│           ├── aks-store-demo-store-front-image.yaml
│           └── kustomization.yaml
└── overlays
    └── dev
        ├── canary
        │   ├── kustomization.yaml
        │   └── store-front-canary.yaml
        ├── ingressgateway.yaml
        ├── kustomization.yaml
        └── namespace.yaml

9 directories, 25 files
```

As you can see, the `image-update` and `flagger` components have been split into their own Kustomization directories. I also created a `canary` Kustomization to deploy the `Canary` resource and moved it down to the `overlays/dev` directory since it is more specific to my app deployment.

By taking this directory structure approach, we can link the Kustomizations together using the `dependsOn` property and ensure things get installed and deployed in the right order.

So in my [`fluxcd.tf`](https://github.com/pauldotyu/awesome-aks/blob/9bd43dd94a2d2c4db23515e08ac1696adfd539d6/2023-09-28-bootstrap-flux-v2-on-aks/fluxcd.tf#L52) I am first deploying the `dev-app` Kustomization, then the `dev-image` and `dev-flagger` Kustomizations (which depends on `dev-app`), and finally the `dev-canary-store-front` Kustomization (which depends on `dev-flagger`).

When you run the Terraform, it will take roughly 15-20 minutes end-to-end, but at the end of it all, you'll have a fully functional AKS Store Demo app fully configured with image update automation and canary deployments 🥳

## Conclusion

Using the Flux CLI to generate Flux manifests is a great way to get started with GitOps on AKS. It gives you a way to understand exactly what CRDs need to be deployed and how the manifests are written. However, if you are looking for an Azure native (and supported way) of bootstrapping your cluster, you'll need to plan for a few more Kustomizations and nest them using Flux dependencies. Just a little more planning but not much more.

Now go and bootstrap your GitOps cluster, the Azure way! 🚀

## Continue the conversation

If you have any feedback or suggestions, please feel free to reach out to me on [Twitter](https://twitter.com/pauldotyu) or [LinkedIn](https://www.linkedin.com/in/yupaul).

You can also find me in the [Microsoft Open Source Discord](https://aka.ms/cloudnative/JoinOSSDiscord), so feel free to DM me or drop a note in the [_cloud-native_ channel](https://aka.ms/cloudnative/DiscordChannel) where my team hangs out!

- [Join the Microsoft Open Source Discord](https://aka.ms/cloudnative/JoinOSSDiscord)
- [Meet us in the Cloud Native Channel](https://aka.ms/cloudnative/DiscordChannel)

Peace ✌️