Automating Image Updates on AKS: A Step-by-Step Guide using open source FluxCD
Tags: kubernetes, gitops, fluxcd, kustomize
Published: 2023-09-22 20:41:32

---
series: GitOps on AKS
title: Automating Image Updates on AKS: A Step-by-Step Guide using open source FluxCD
description: Learn how to automate image updates in your Azure Kubernetes Service cluster using FluxCD and GitOps.
published: true
date: 2023-09-22 13:00:00 UTC
tags: kubernetes, gitops, fluxcd, kustomize
canonical_url: https://paulyu.dev/article/automating-image-updates-with-fluxcd-on-aks/
cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/s5vqdl6y2ohvd2iwh8vc.jpg
---

In my previous [post](https://aka.ms/cloudnative/GitGoingWithGitOps), we walked through the setup of FluxCD on AKS via AKS extensions. In this article, we'll go a bit deeper and take a look at how you can use FluxCD to automate image updates in your AKS cluster.

The goal here is to streamline the process of updating your application deployments in your cluster.

Here is our intended workflow:

1. Modify application code, then commit and push the change to the repo.
1. Create a new release in GitHub which kicks off a release workflow to build and push an updated container image to a GitHub Container Registry.
1. FluxCD detects the new image and updates the image tag in the cluster.
1. FluxCD rolls out the new image to the cluster.

We'll use same AKS store demo app we used in the previous post, but this time we'll go a bit faster.

If you need a refresher on how to deploy the application, you can refer to the [previous post](https://aka.ms/cloudnative/GitGoingWithGitOps).

## Prerequisites

Before you begin, you need to have the following:

- [Azure Subscription](https://azure.microsoft.com/get-started/)
- [GitHub Account](https://github.com/)
- [Azure CLI](https://docs.microsoft.com/cli/azure/install-azure-cli?view=azure-cli-latest)
- [GitHub CLI](https://github.com/cli/cli#installation)
- [kubectl](https://kubernetes.io/docs/tasks/tools/)
- [Flux CLI](https://fluxcd.io/flux/installation/)

## Create an AKS cluster

Use the following Azure CLI commands to create a new resource group and AKS cluster.

```bash
RG_NAME=rg-imageupdate
AKS_NAME=aks-imageupdate
LOC_NAME=westus3

az group create -n $RG_NAME -l $LOC_NAME
az aks create -n $AKS_NAME -g $RG_NAME
```

When the cluster is created, run the following command to connect to the cluster.

```bash
az aks get-credentials -n $AKS_NAME -g $RG_NAME
```

## Bootstrapping FluxCD

In my previous post, I used the AKS extension to install FluxCD. This time, I'll be using the Flux CLI to bootstrap FluxCD onto the AKS cluster. This process will enable us to have a bit more control over the installation and gives us the ability to save the Flux resources to our GitHub repo.

So we're going to implement GitOps even on top of our GitOps tooling 🤯

There's a few things we need to do before we can bootstrap the cluster. First, we need the [Flux CLI](https://fluxcd.io/flux/installation/). Once you have the CLI installed, run the following command to ensure your cluster is ready for bootstrapping.

```bash
flux check --pre
```

We also need a repo to land the Flux resources into. So let's fork and clone the [aks-store-demo-manifests](https://github.com/pauldotyu/aks-store-demo-manifests) repository. This with be our repo for both application deployment and our GitOps configurations.

We'll be doing a bunch of things within GitHub. I prefer using the [GitHub CLI](https://github.com/cli/cli#installation) over clicking around in the GitHub website.

Run the following commands to login to GitHub, clone the repo and prep it for our exercise.

```bash
# navigate to the home directory or wherever you want to clone the repo
cd ~

# login to GitHub
gh auth login --scopes repo,workflow

# fork and clone the repo to your local machine
gh repo fork https://github.com/pauldotyu/aks-store-demo-manifests.git --clone

# make sure you are in the aks-store-demo-manifests directory
cd aks-store-demo-manifests

# since we are in a forked repo, we need to set the default to be our fork
gh repo set-default

# merge the "kustomize" branch into "main"
git merge origin/kustomize

# push the change up to your fork
git push
```

We need to set some environment variables to for the Flux bootstrapping process. Run the following commands to set the environment variables.

```bash
# set the repo url
export GITHUB_REPO_URL=$(gh repo view --json url | jq .url -r)

# set your GitHub username
export GITHUB_USER=$(gh api user --jq .login)

# set your GitHub personal access token
export GITHUB_TOKEN=$(gh auth token)
```

Now we're ready to bootstrap FluxCD. Run the following command to bootstrap FluxCD. This command will install FluxCD with additional components to enable image automation. It will also generate the Flux resources and commit them to our Git repo in the `clusters/dev` directory.

```bash
flux bootstrap github create \
  --owner=$GITHUB_USER \
  --repository=aks-store-demo-manifests \
  --personal \
  --path=./clusters/dev \
  --branch=main \
  --reconcile \
  --network-policy \
  --components-extra=image-reflector-controller,image-automation-controller
```

After a few minutes, run the following commands and you should see the Flux resources in the repo.

```bash
git fetch
git rebase

# show the Flux resources
tree clusters/dev
```

In order for the `image-automation-controller` to write commits to our repo, we need to create a Kubernetes secret to store our GitHub credentials. Run the following command to create the secret.

```bash
flux create secret git aks-store-demo \
  --url=$GITHUB_REPO_URL \
  --username=$GITHUB_USER \
  --password=$GITHUB_TOKEN
```

> This command will create Kubernetes secrets in your cluster but you could also use Azure Key Vault with the Secret Store CSI driver to store your GitHub credentials.

We don't need the GitHub PAT token anymore, so run the following command to discard it.

```bash
unset GITHUB_TOKEN
```

Next we need to create a `GitRepository` resource. This resource will point to our fork of the [`aks-store-demo-manifests`](https://github.com/pauldotyu/aks-store-demo-manifests) repo where we have our app deployment and GitOps manifests stored.

Run the following command to create the configuration and export it to a YAML file which we'll commit to our repo.

```bash
flux create source git aks-store-demo \
  --url=$GITHUB_REPO_URL \
  --branch=main \
  --interval=1m \
  --secret-ref=aks-store-demo \
  --export > ./clusters/dev/aks-store-demo-source.yaml
```

We also need to specify the `Kustomization` resource to tell FluxCD where to find the app deployment manifests in our repo. Run the following command to create the configuration and export it to a YAML file which we'll also commit to our repo.

```bash
flux create kustomization aks-store-demo \
  --source=aks-store-demo \
  --path="./overlays/dev" \
  --prune=true \
  --wait=true \
  --interval=1m \
  --retry-interval=2m \
  --health-check-timeout=3m \
  --export > ./clusters/dev/aks-store-demo-kustomization.yaml
```

We have two new Flux resource manifests. Run the following command to commit and push the files to the repo so Flux can begin the reconciliation process.

```bash
git add -A
git commit -m "feat: add source and kustomization" 
git push
```

As soon as the code is pushed, Flux will do it's thing; reconcile. Run the following command to watch the `Kustomization` reconciliation process.

```bash
flux get kustomizations --watch
```

After a few minutes you should see something like this...

```text
aks-store-demo	main@sha1:95de7bde	False	True	Applied revision: main@sha1:95de7bde
```

Confirm the `GitRepository` and `Kustomization` resources have been created.

```bash
flux get sources git 
flux get kustomizations
```

Run the following command to ensure all the Pods are running.

```bash
kubectl get po -n dev
```

Once you see all the Pods are running, run the following command to grab the public IP of the `store-front` service.

```bash
kubectl get svc store-front -n dev
```

Using a web browser, navigate to the public IP and confirm the application is up and running.

## Setup the release workflow

The source code for the `aks-store-demo` application is located in a different repository than where our manifests are.

> To give you a bit of a warning, we'll be flipping back and forth between the `aks-store-demo` and `aks-store-demo-manifests` repository directories.
>
> Hint: you can use the `cd -` command to flip back and forth between the last two directories you were in.

The application code is in the [aks-store-demo](https://github.com/azure-samples/aks-store-demo) repository. Run the following commands to fork and clone the repo.

```bash
# navigate to the home directory or wherever you want to clone the repo
cd -

# fork and clone the repo to your local machine
gh repo fork https://github.com/azure-samples/aks-store-demo.git --clone

# make sure you are in the aks-store-demo directory
cd aks-store-demo

# since we are in a forked repo, we need to set the default to be our fork
gh repo set-default
```

Currently the `aks-store-demo` repository has Continuous Integration (CI) workflows to build and push container images using `GITHUB_SHA` as the image tag. We need to create a new release workflow that will build and push a new container image using [semantic versioning](https://semver.org/). Flux's image update automation policy which is used to determine the most recent image tag can use various methods including numerical tags, alphabetical tags, and semver tags. So in this case, as much as I'd like to use the `GITHUB_SHA`, it does not provide a conducive way to determine "latest".

Therefore, we need to tag our releases with semantic version numbers. GitHub offers a way to [create tagged releases](https://docs.github.com/repositories/releasing-projects-on-github/about-releases). Additionaly, GitHub Actions have the ability to [trigger workflows when a new release is published](https://docs.github.com/actions/using-workflows/events-that-trigger-workflows#release).

Run the following command to create a new workflow file.

```bash
# make sure you are in the aks-store-demo directory
touch .github/workflows/release-store-front.yaml
```

Open the `release-store-front.yaml` file using your favorite editor and paste the following code.

```yaml
name: release-store-front

on:
  release:
    types: [published]

permissions:
  contents: read
  packages: write

jobs:
  publish-container-image:
  
    runs-on: ubuntu-latest

    steps:
      - name: Set environment variables
        id: set-variables
        run: |
          echo "REPOSITORY=ghcr.io/$(echo ${{ github.repository }} | tr '[:upper:]' '[:lower:]')" >> "$GITHUB_OUTPUT"
          echo "IMAGE=store-front" >> "$GITHUB_OUTPUT"
          echo "VERSION=$(echo ${GITHUB_REF#refs/tags/})" >> "$GITHUB_OUTPUT"
          echo "CREATED=$(date -u +'%Y-%m-%dT%H:%M:%SZ')" >> "$GITHUB_OUTPUT"

      - name: Env variable output
        id: test-variables
        run: |
          echo ${{ steps.set-variables.outputs.REPOSITORY }}
          echo ${{ steps.set-variables.outputs.IMAGE }}
          echo ${{ steps.set-variables.outputs.VERSION }}
          echo ${{ steps.set-variables.outputs.CREATED }}

      - name: Checkout code
        uses: actions/checkout@v2

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2

      - name: Login to GitHub Container Registry
        uses: docker/login-action@v1
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ github.token }}

      - name: Build and push
        uses: docker/build-push-action@v2
        with:
          context: src/store-front
          file: src/store-front/Dockerfile
          push: true
          tags: |
            ${{ steps.set-variables.outputs.REPOSITORY }}/${{ steps.set-variables.outputs.IMAGE }}:latest
            ${{ steps.set-variables.outputs.REPOSITORY }}/${{ steps.set-variables.outputs.IMAGE }}:${{ steps.set-variables.outputs.VERSION }}
          labels: |
            org.opencontainers.image.source=${{ github.repositoryUrl }}
            org.opencontainers.image.created=${{ steps.set-variables.outputs.CREATED }}
            org.opencontainers.image.revision=${{ steps.set-variables.outputs.VERSION }}
```

This workflow will be triggered each time a new release is published and the `${GITHUB_REF#refs/tags/}` bit allows us to extract the release version to use as an image tag.

Save the file, then commit and push the changes to the repo.

```bash
git add .github/workflows/release-store-front.yaml
git commit -m "feat: add release workflow"
git push
```

## Update the application

Since we're in the app repo, let's make a change to it.

We'll make a very small edit to the `src/store-front/src/components/TopNav.vue` file and update the title of the application to include a version number.

Open the file and change the title on line 4 from `Azure Pet Supplies` to `Azure Pet Supplies v1.0.0`.

If you don't want to use an editor, you can run the following command to make the change using `sed` (I love and hate `sed` at the same time).

```bash
# make sure you are in the aks-store-demo directory
sed -i -e "s/Azure Pet Supplies/Azure Pet Supplies v1.0.0/g" src/store-front/src/components/TopNav.vue
```

Commit and push the changes to the repo.

```bash
git add src/store-front/src/components/TopNav.vue
git commit -m "feat: update title"
git push
```

Using GitHub CLI, create and publish a new release tagged as `1.0.0`. This will trigger the release workflow we just created.

```bash
gh release create 1.0.0 --generate-notes
```

Wait about 10 seconds then run the following command to watch the workflow run.

```bash
# you can select the release workflow from the list
gh run watch
```

With the `store-front` container image release workflow setup, let's configure image update automation in Flux.

## Configure image update automation

To setup FluxCD to listen for changes in the GitHub Container Registry, we need to create a new `ImageRepository` resource. You need to create an `ImageRepository` resource for each image you want to automate. In this case, we're only automating the `store-front` image.

> ⚠️ Flip back over to the `aks-store-demo-manifests` repo

Using the FluxCLI, run the following command to create the manifest for the `ImageRepository` resource.

```bash
flux create image repository store-front \
  --image=ghcr.io/$GITHUB_USER/aks-store-demo/store-front \
  --interval=1m \
  --export > ./clusters/dev/aks-store-demo-store-front-image.yaml
```

Run the following command to create an `ImagePolicy` resource to tell FluxCD how to determine the newest image tags. We'll use the `semver` filter to only allow image tags that are valid semantic versions and equal to or greater than `1.0.0`.

```bash
flux create image policy store-front \
  --image-ref=store-front \
  --select-semver='>=1.0.0' \
  --export > ./clusters/dev/aks-store-demo-store-front-image-policy.yaml
```

> There are other filters you can use as well. You can read more about them [here](https://fluxcd.io/flux/guides/image-update/#imagepolicy-examples).

Finally, run the following command to create an `ImageUpdateAutomation` resource which enables FluxCD to update images tags in our YAML manifests.

```bash
flux create image update store-front \
  --interval=1m \
  --git-repo-ref=aks-store-demo \
  --git-repo-path="./base" \
  --checkout-branch=main \
  --author-name=fluxcdbot \
  --author-email=fluxcdbot@users.noreply.github.com \
  --commit-template="{{range .Updated.Images}}{{println .}}{{end}}" \
  --export > ./clusters/dev/aks-store-demo-store-front-image-update.yaml
```

> If you are uncomfortable with FluxCD making changes directly to the checkout branch (in this case `main`), you can create a separate branch for FluxCD (using the `--push-branch` parameter) to specify where commits should be pushed to. This will then enable you to follow your normal Git workflows (e.g., create a pull request to merge the changes into the main branch).

Run the following commands to commit and push the new Flux resources to the repo.

```bash
git add -A
git commit -m "feat: add image automation"
git push
```

After a few minutes, run the following command to see the status of the image update automation resources.

```bash
flux get image repository store-front
flux get image policy store-front
flux get image update store-front
```

The image update automation is setup but it won't do anything until we tell it which Deployments to update.

## Update the manifest

We're going to sick with updating the `store-front` only. So we'll need to update the `store-front` deployment manifest to use the `ImagePolicy` resource we created earlier. This is done by **marking** the manifest with a comment.

Open the `base/store-front.yaml` file using your favorite editor.

On line 19, update the image to use your GitHub Container Registry. Then at the end of the line, add a comment to include the namespace and name of your `ImagePolicy` resource. This **marks** the deployment for Flux to update.

> The comment is super important because Flux will not implement the image tag policy if it doesn't have this comment.

The line should look something like this:

```yml
image: ghcr.io/<REPLACE_THIS_WITH_YOUR_GITHUB_USERNAME>/aks-store-demo/store-front:latest # {"$imagepolicy": "flux-system:store-front"}
```

> Setting the marker in the deployment manifest is fine for this demo, but ideally you'd want to set it in the kustomization manifest instead.

Commit and push the changes to the repo.

```bash
git add base/store-front.yaml
git commit -m "feat: add imagepolicy to store-front manifest"
git push
```

At this point, we've successfully setup image automation in our cluster. Time to test.

## Test the image automation

We're going to test the developer workflow we laid out at the beginning of this article.

### Make another change

> Flip back over to the `aks-store-demo` repo

Make another change to the `TopNav.vue` file. This time, change the title to `Azure Pet Supplies v2.0.0`.

```bash
# make sure you are in the aks-store-demo directory
sed -i -e "s/Azure Pet Supplies v1.0.0/Azure Pet Supplies v2.0.0/g" src/store-front/src/components/TopNav.vue
```

Commit and push the changes to the repo.

```bash
git add src/store-front/src/components/TopNav.vue
git commit -m "feat: update title again"
git push
```

Create a new `2.0.0` release. This will trigger the release workflow.

```bash
gh release create 2.0.0 --generate-notes

# wait about 5 seconds then run the following command
gh run watch
```

### Verify the image update

After a few minutes, you should see the new image tag in the `aks-store-demo` repo and the `store-front` deployment being updated in the cluster.

The reconcile interval for `ImagePolicy` was set to 1 minute. So after 1 minute or so, we should see that the update was successful.

```bash
flux get image policy store-front
```

Now check the `store-front` deployment to see the new image tag.

```bash
kubectl get deploy store-front -n dev -o yaml | grep image:
```

You should see that the image tag has been updated to `2.0.0`.

Sweet! We've successfully automated image updates in our cluster using FluxCD and GitOps 🚀

Get the public IP of the `store-front` service by running the following command.

```bash
kubectl get svc store-front -n dev
```

Using a web browser, navigate to the public IP address of the `store-front` service. You should see the application running with our new title that includes `v2.0.0` 🥳

## Conclusion

The power of GitOps is on full display here 🤖

In this article, we took a look at how you can use leverage an innovative feature of FluxCD to automate image updates in your AKS cluster. As an app developer, I can simply make my code changes and push it through a PR process. Once the PR is approved and merged and a new release is created, the image update automation kicks in and updates the image tag in the cluster for us without any manual intervention. This is a great way to streamline the deployment process and keep your cluster up to date with the latest images.

As an aside, I did all this open source FluxCD. This could also be done using the AKS extension for FluxCD but you do need to enable the `image-automation-controller` and `image-reflector-controller` components. You can find more information on how to do that [here](https://learn.microsoft.com/azure/azure-arc/kubernetes/tutorial-use-gitops-flux2?tabs=azure-cli#control-which-controllers-are-deployed-with-the-flux-cluster-extension)).

If you have any feedback or suggestions, please feel free to reach out to me on [Twitter](https://twitter.com/pauldotyu) or [LinkedIn](https://www.linkedin.com/in/yupaul).

Peace ✌️

## Resources

- [Git going with GitOps on AKS: A Step-by-Step Guide using FluxCD AKS Extension](https://aka.ms/cloudnative/GitGoingWithGitOps)
- [Automate Image Updates](https://fluxcd.io/flux/guides/image-update/)