Automatic Image Update to Git using Flux and GitHub Actions
Tags: gitops, flux, githubactions, kubernetes
Published: 2024-04-14 18:30:00



Have you ever had to manually update your container images, only to forget to do it or make a mistake? Automatic image updates can help you avoid these problems and ensure that your applications are always running the latest and most secure images. Manual image updates can be time-consuming and error-prone, especially if you have a large number of containerized applications. 

In this blog post, we'll explore the advantages of automatic image updates and explain how to implement them in your environment using [GitHub Actions](/blogs/github-actions-demystified/). Moreover, we'll cover the process of temporarily pausing these updates to ensure application stability during incidents or any unforeseen issues.

So, let’s deep dive into automatic image updates to Git using Flux and GitHub Actions!

## GitOps: A modern approach to software delivery

In the world of application management, GitOps fills a crucial need by simplifying and enhancing how we handle applications. But why do we need it? Managing applications can be complex and prone to errors. To address this, GitOps establishes Git as the central hub for handling configuration and infrastructure changes, allowing for streamlined, efficient, and reliable management. It speeds up deployment, ensures consistent and error-free setups, and reduces operational costs.

Moreover, it's important to note that GitOps isn't just about managing applications; it also intersects with the concept of auto-image updates. These updates involve automatically refreshing the underlying components of your application by regularly updating to the latest images pushed to the container registry, ensuring your application stays in sync with the most up-to-date software and dependencies.


GitOps streamlines software deployment by automating build, test, and deployment processes triggered by Git commits. This enhances reliability through a review and approval system, reducing errors and downtime. Collaboration is promoted through Git's features for change tracking and code reviews, with seamless integration of image updates. 

### Current pain points of dealing with image updates without GitOps

Here are several challenges you may face when dealing with image updates in the absence of GitOps:

- **Manual updates to manifest**: When a new image is released, the manifest must be manually updated to reference the new image tag. This can be error-prone and time-consuming, especially for large and complex applications.
- **Delayed deployments**: If the manifest is not updated promptly, the deployment of the new image will be delayed. This can impact the time to market for new features or bug fixes.
- **Inconsistent deployments**: If the manifest is not updated consistently across all environments, it can lead to inconsistent deployments. This can make it difficult to troubleshoot problems and ensure that applications are running as expected in all environments.
- **Manual build triggers**: When a new image is released, a new build must be manually triggered. This can be inefficient and time-consuming, especially for large and complex applications.


GitOps using auto-image updates can help to solve all of these pain points by providing a declarative and automated approach to application deployment and management. With GitOps, the manifest is automatically updated when a new image is released, and a new build is automatically triggered. This ensures that deployments are consistent and timely across all environments.

### Benefits of automatic image updates using GitOps

Here are several advantages of using GitOps for image updates:

- **Saves time and effort**: GitOps streamlines image updates by automating the process of updating images, tags, and manifests. This eliminates the need for manual intervention, reducing the risk of errors and ensuring a quick and reliable deployment of the latest application version across all environments.
- **Promotes consistency across environments**: Ensuring consistent application versions across various environments become seamless with automated image updates. This practice simplifies application management, reducing the risk of inconsistencies between environments. Consequently, maintenance and troubleshooting become more straightforward.
- **Fosters collaboration and transparency**: Encouraging teamwork and transparency, GitOps facilitates automatic image updates for development teams. Leveraging Git commits for all changes simplifies change tracking, code reviews, and maintains visibility in the software delivery process. This promotes efficient collaboration among developers and cultivates a culture of continuous improvement.
- **Improves application delivery**: Leveraging GitOps for automatic image updates speeds up application delivery and boosts reliability. Streamlining infrastructure and application management enables developers to focus more efficiently on delivering customer value.

## Approaches to automating image updates

In this section, we will explore two primary methods for automating image updates to keep your applications up-to-date and secure. 

1. Scripted approach
2. Automated approach using GitOps tools

### Scripted approach

A script can be configured to run on a schedule using a cron job or when certain events occur, such as when a new commit is pushed to the Git repository.

Here are a few pros and cons of the scripted approach:

**Pros of scripted approach**:  
- Flexible and customizable method
- Can be tailored to meet specific needs
- No need to rely on third-party tools

**Cons of scripted approach**:
- Complex and time-consuming to implement
- Requires expertise in scripting 
- Maintaining and troubleshooting the scripts can be challenging

### Using GitOps tools for automating image updates


In the previous section, we saw that scripting can be complex and time-consuming. That's where dedicated tools come in handy. In this section, we will explore how GitOps tools make automating image updates easier. We will specifically look at the two most popular tools, [Argo CD](https://argo-cd.readthedocs.io/en/stable/operator-manual/architecture/) and Flux, to understand how they help with automatic image updates.

#### Argo CD image updater

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/32m5pvlnv5yw0h5ancyj.png)
<center>(Image update process using Argo CD Image Updater)</center>

The Argo CD image updater is a tool that can help you update your Kubernetes workloads to the latest versions of their container images. It does this by setting appropriate application parameters for your [Argo CD](/argo-cd-consulting-support/) applications.

To use the image updater, you annotate your Argo CD application resources with a [list of images to be considered for update](https://argocd-image-updater.readthedocs.io/en/stable/basics/update/), along with a version constraint to restrict the maximum allowed new version for each image. The image updater will then periodically check for new versions of the images and update them if necessary.

The image updater can be used to update images in a variety of ways, including:
- **Automatic updates**: The image updater can be configured to automatically [update images to their latest allowed version on a schedule](https://argocd-image-updater.readthedocs.io/en/stable/basics/update-methods/). 
- **Manual updates**: The image updater can also be triggered manually to update images. 
- **Event-driven updates**: The image updater can also be configured to be triggered by events, [such as when a new image tag is pushed to a registry](https://argocd-image-updater.readthedocs.io/en/stable/basics/update-methods/). 

#### Image automation and reflector controllers in Flux


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/y5gfc6mcarb9fx2yejsj.png)
<center>(Image update process using Image Automation and Image Reflector Controller)</center>

Image automation in Flux can be used to automate the following tasks:

- **Updating image tags in Git**: Image automation can automatically update the image tags in your Git repository to the latest stable versions of your images. This can save you time and reduce the risk of errors.
- **Creating and managing image promotion pipelines**: Image automation can be used to create and manage image promotion pipelines. This allows you to automate the process of rolling out new image versions to production in a safe and controlled manner.
- **Implementing image policies**: Image automation can be used to implement image policies, such as requiring all images to be tagged with a specific version number or only using images from a specific registry.

**Image automation controllers**  

In Flux CD, the image automation controllers, which include the `Image Reflector Controller` and `Image Automation Controller`, are responsible for maintaining the synchronization between the image metadata in Kubernetes and the latest image metadata from the registry. The image reflector controller achieves this by regularly scanning the registry for changes to image metadata. When a change is detected, they update the Kubernetes resources that reference the affected images accordingly.

Moreover, the image automation controllers are capable of triggering image updates based on changes to the image metadata. For instance, you can configure automation to automatically update the image tags in your Kubernetes deployments to the latest stable versions of your images.

For more details about the implementation and design of the image automation controllers in Flux CD, refer to the official documentation on [image reflector and automation Controllers](https://fluxcd.io/flux/components/image/).

#### Pros and cons of using GitOps approach for automating image updates

Let us explore the pros and cons of [employing GitOps](/gitops-consulting/) for automating image updates, gaining insights into its efficiency and potential challenges.  

**Pros of GitOps approach**:
- **Reducing human error**: Both Argo CD and Flux CD offerings for image update aim to reduce the risk of human error by automating the process of updating container images. This helps in maintaining consistency and accuracy.
- **Workflow efficiency**: Automation in image updates improves workflow efficiency by saving time and allowing teams to focus on more critical tasks.
- **Automatic updates**: GitOps tools, including both Argo CD and Flux CD, provide a mechanism for automatic updates, which enhances the security, reliability, and efficiency of Kubernetes workloads.
- **Flexibility**: Both tools offer flexibility in how image updates can be handled, supporting automatic updates, manual updates, and event-driven updates, allowing users to choose the best approach for their specific requirements.

**Cons of GitOps approach**:
- **Complex setup**: Setting up and configuring GitOps tools for image automation, especially for users unfamiliar with the underlying technologies like Kubernetes or Flux, can be complex.
- **Troubleshooting complexity**: The complexity of interactions with various components (e.g., Kubernetes, registries, Git repositories) can make troubleshooting challenging if issues arise during the image update process.
- **Dependency on GitOps tool**: Both Argo CD and Flux CD are limited to updating container images for applications managed by their respective tools. This dependency might restrict users who prefer or have other GitOps tools in their workflows.
- **Image pull secrets location**: GitOps tools often require image pull secrets to be present in the same cluster where the tool is running, which might pose challenges for scenarios where secrets need to be fetched from other clusters.
- **Limited PR support**: Neither Argo CD nor Flux CD, as described, fully supports the creation of pull requests (PRs). External CI tools may be required for more comprehensive integration with version control systems.
- **Production readiness**: When considering GitOps tools for image automation in production, it's crucial to note that Flux CD is deemed "production-ready," while Argo CD's image updater lacks the same designation. This underscores the necessity of carefully evaluating the maturity and stability of GitOps tools to align with specific production environment requirements.

## How does image automation and image reflector controllers in Flux work

Let's understand how Flux employs two controllers, i.e. image automation controller and image reflector controller, to achieve image update automation.

These controllers, together, collaborate to update a Git repository whenever new container image tags become available. 

### Image reflector controller

It is composed of two custom resources: [Image Repository](https://fluxcd.io/flux/components/image/imagerepositories/) and [Image Policy](https://fluxcd.io/flux/components/image/imagepolicies/). Let us understand how they operate together.

- **Image repository**: This component scans the container image repository and retrieves image metadata, including tags and versions. You can create an `Image Repository` resource and configure the scanning interval.
- **Image policy**: This custom resource informs Flux which semantic versioning range to apply when filtering image tags.

### Image automation controller

The [image automation controller](https://fluxcd.io/flux/components/image/imageupdateautomations/) includes a custom resource called `Image Update Automation`.

`Image Update` resource clones the Git repository, updates YAML files based on the latest images from the image reflector controller, and commits changes to the specified Git repository. 
After cloning, the image automation controller identifies and updates the deployment YAML manifest, using comments to mark fields for updating. The automation process checks the image policy in the comment and updates the field value accordingly.

Once updated, the image update automation controller commits and pushes changes to the specified branch. The source controller pulls the updated manifest, and the [Kustomization controller](https://github.com/fluxcd/kustomize-controller) applies the changes. This is how the Flux image automation controller streamlines the process of automating image version updates.

## Automating container image updates with Flux image automation and GitHub Actions

We will explore a live demonstration of a Flux GitOps configuration that seamlessly manages a workflow from staging to production, showcasing the automatic deployment process when making changes to source code. Additionally, we'll highlight its integration with GitHub Actions to enhance the efficiency of the entire deployment pipeline.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/j6me7lsinhyz6cvz90ty.png)

In staging, developers make changes to application code, triggering a new build. The new image tag is then propagated to the Kubernetes manifest and finally deployed to the staging environment without requiring any approval in between.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/b5bcgqbnwthcfu3zq0qc.png)

In the case of production, a new PR is raised with an updated Kubernetes manifest, requiring approval for merge in the config repository. Only after approval is given does it get deployed to the production environment.

### Prerequisites

Ensure you have following things setup:

1. **Flux CLI** - which can be downloaded from the [official docs](https://fluxcd.io/docs/cmd/)
2. **Kubernetes cluster** -  for this demo, we will use minikube.
3. GitHub account - which has GitHub Actions enabled.

### Environment setup details

1. **Hello Env Flask application**: A simple application displaying the environment name and release number.
2. **Two environments**: Staging and production, housed in separate namespaces.
3. **Continuous deployment**: Source code changes trigger automated build and deployment to the staging environment.
4. **Release tagging**: Tagging a release initiates automatic build processes and creates a pull request for production deployment.


#### Git repositories

Let's take a look at two Git repositories we will be using.

1. **Application repository** - [https://github.com/infracloudio/flux-helloenv-app](https://github.com/infracloudio/flux-helloenv-app)  - code and Kustomization.
   1. **Kustomization for environment separation**: Utilizing Kustomize to segregate staging and production environments. 
   2. **Structured repository**: The repository combines source code and Kubernetes manifests for a unified demo setup. There are many possible ways to [structure your git repositories.](https://fluxcd.io/flux/guides/repository-structure/)
   3. **CI and automation**: Leveraging GitHub Actions for [continuous integration](/ci-cd-consulting/) and automation.
2. **Management Repository**-  [https://github.com/infracloudio/flux-gitops-helloenv](https://github.com/infracloudio/flux-gitops-helloenv) - GitOps manifests.

### Flux bootstrap configuration

Beginning with an empty cluster, our initial task is to [bootstrap Flux itself](https://fluxcd.io/flux/cmd/flux_bootstrap/). Flux serves as the foundation upon which we'll bootstrap all other components.

In the following sections, we'll take a detailed look at each file within the apps directory, one by one. 

```sh
├── apps
│   ├── git-repo.yaml
│   ├── image-auto-prod.yaml
│   ├── image-auto-staging.yaml
│   ├── image-policy-prod.yaml
│   ├── image-policy-staging.yaml
│   ├── image-repo.yaml
│   ├── kustomization-prod.yaml
│   └── kustomization-staging.yaml
```
 
#### git-repo.yaml

This instructs Flux on how to interact with the Git repository where your application's source code resides. It contains the following configuration:

```yaml
  ref:
    branch: main
  secretRef:
    name: ssh-credentials
  url: ssh://git@github.com/infracloudio/flux-helloenv-app
```

`ref:` Specifies the Git branch to watch for changes, in this case, main.  
`secretRef:` Refers to a Kubernetes Secret named ssh-credentials, which likely contains SSH keys for secure Git access.  
The `ssh-credentials` is a secret that needs to be created.  

`url:` Indicates the URL of the Git repository, Change this to `url: ssh://git@github.com/<your_github_username>/flux-helloenv-app` once you fork it.  

#### image-repo.yaml

This file is responsible for scanning the Docker image registry and fetching image tags based on the defined policy. Here's the configuration:  

```yaml
image: docker.io/shapai/helloenv
interval: 1m0s
```

`image:` Specifies the Docker image repository (for e.g. docker.io/shapai/helloenv) to scan for image tags. Change this to have your relevant Docker image repository, where the CI job will build and push the image. This same registry needs to be updated in your forked CI file, where the CI build will push images.  
`interval:` Sets the interval at which Flux will scan the image repository (every 1 minute in this case) and fetch image tags according to the defined policy.  
  
Next, we will go through the image policy files image-policy-staging.yaml and image-policy-prod.yaml  

#### image-policy-staging.yaml 

This file defines the image tagging policy for the staging environment. Here's the configuration:  

```yaml
  filterTags:
    extract: $ts
    pattern: ^main-[a-f0-9]+-(?P<ts>[0-9]+)
  imageRepositoryRef:
    name: helloenv
  policy:
    numerical:
      order: asc
```

`filterTags:` This section specifies how to filter image tags. It extracts the timestamp ($ts) from tags that match the specified pattern. Tags are filtered in ascending order based on this timestamp, ensuring that Flux fetches the latest built image for the staging environment.  

#### image-policy-prod.yaml 

This file defines the image tagging policy for the production environment:  

```yaml
  imageRepositoryRef:
    name: helloenv
  policy:
    semver:
      range: '>=1.0.0'
```

`imageRepositoryRef:` Refers to the image repository named helloenv.  
`policy:` Defines a Semantic Versioning (SemVer) policy that specifies a range for acceptable image tags (in this case, any version greater than or equal to 1.0.0).  
These image policies are crucial in ensuring that Flux deploys the correct images to the respective environments (staging and production) based on the defined criteria.  

#### kustomization-prod.yaml and kustomization-staging.yaml

These YAML files define Kustomization resources for managing Kubernetes resources in both staging and production environments within the flux-system namespace. They are configured to synchronize with a specified Git repository, allowing for automated deployment and management of Kubernetes resources.
 

Now, let's go through the **ImageUpdateAutomation** files.

#### helloenv-staging.yaml

This YAML file configures an ImageUpdateAutomation object to update the image tags in the `./demo/kustomize/staging` directory of the flux-helloenv-app Git repository. It will scan the repo every 1 minute (specified by the interval field).   

The `git` section specifies the branch to checkout and the commit message template. The sourceRef section specifies the Git repository containing the Kubernetes manifests to update.  

The `update` section specifies the path to the Kubernetes manifests to update and the strategy to use.

When this _ImageUpdateAutomation_ object is deployed, Flux will periodically check for new image updates. If it finds any new updates, it will update the image tags in the Kubernetes manifests and commit the changes to the Git repository.

```yaml
spec:
  git:
    checkout:
      ref:
        branch: main
    commit:
      author:
        email: fluxbot@users.noreply.github.com
        name: fluxbot
      messageTemplate: '{{range .Updated.Images}}{{println .}}{{end}}'
  interval: 1m0s
  sourceRef:
    kind: GitRepository
    name: flux-helloenv-app
  update:
    path: ./demo/kustomize/staging
    strategy: Setters
```

#### helloenv-prod.yaml  

This YAML file is very similar to the previous one but has an additional push configuration. This means that after updating the image tags in the Git repository, Flux will commit and push the changes to the flux-image-update branch. This is done specially for prod setup since we are not going to push directly to the main branch for prod manifests.  

This is useful in our case, as this will help us create a PR to the main branch from the flux-image-update branch, which will then go through manual approval. The process of creating PR is handled by the CI section in GitHub Actions.

```yaml
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
  name: helloenv-prod
  namespace: flux-system
spec:
  git:
    checkout:
      ref:
        branch: main
    commit:
      author:
        email: fluxbot@users.noreply.github.com
        name: fluxbot
      messageTemplate: '{{range .Updated.Images}}{{println .}}{{end}}'
    push:
      branch: flux-image-update
  interval: 1m0s
  sourceRef:
    kind: GitRepository
    name: flux-helloenv-app
  update:
    path: ./demo/kustomize/prod
    strategy: Setters
``` 

### Setting up Flux

Now that we've seen all the configurations, we can proceed to the bootstrap command.

Note the `--owner` and `--repository` switches here: we are explicitly looking for the `${GITHUB_USER}/flux-gitops-helloenv` repo. Make sure you fork both repos under your user and follow on.

```sh
flux bootstrap github \ 
  --components-extra=image-reflector-controller,image-automation-controller \
  --owner=$GITHUB_USER \
  --repository=flux-gitops-helloenv \
  --path=./clusters/my-cluster/ \
  --branch=main \
  --read-write-key \
  --personal --private=false
```

#### Understanding `flux bootstrap` command

Let's understand what the above command does.

To enable the Flux image automation feature, the extra components can be specified with the [`--components-extra` flag](https://fluxcd.io/flux/installation/configuration/optional-components/). We are enabling the image reflector and automation controllers.

`--path=./clusters/my-cluster/` specifies the path within the repository where the configuration files will be stored.  

`my-cluster` folder in the repository contains two files `infrastructure.yaml` and `apps.yaml`.  

- `infrastructure.yaml` defines a Kustomization resource called ingress-nginx. This Kustomization lives in the flux-system namespace, doesn't depend on anything, and has kustomize files at infrastructure/ingress-nginx.

- `apps.yaml`, which defines the `flux-helloenv-app` application itself. Within this file, you'll find references to the apps directory. This directory is a central location containing Kustomizations and configuration settings for setting up the `flux-helloenv-app` application in both the prod and staging environments. Additionally, it includes configurations for image automation for `helloenv` app in staging and prod env.

The process of bootstrapping everything may take some time. To monitor the progress and ensure everything is proceeding as expected, we can utilize the `--watch` switch.

```sh
flux get kustomizations --watch
```

Verify that all Flux pods are in running state by running get pods.
```sh
$ kubectl -n flux-system get pods

NAME                                           READY   STATUS    RESTARTS   AGE
image-automation-controller-6c4fb698d4-zrp78   1/1     Running   0          29s
image-reflector-controller-5dfa39212d-hnnvj    1/1     Running   0          29s
kustomize-controller-424f5ab2a2-u2hwb          1/1     Running   0          29s
source-controller-2wc41z892-axkr1              1/1     Running   0          29s
```

Since our flux-helloenv-app repository is public, application will get deployed as part of the bootstrap step. You can check both staging and prod environments with following commands.
```sh
kubectl rollout status -n helloenv-staging deployments
watch kubectl get pods -n helloenv-staging

kubectl rollout status -n helloenv-prod deployments
watch kubectl get pods -n helloenv-prod
```

#### Setup Git authentication for Flux

After bootstrapping Flux, we will grant it write access to our GitHub repositories. This will allow Flux to update image tags in manifests, create pull requests etc.

The [`flux create secret git`](https://fluxcd.io/flux/cmd/flux_create_secret_git/) command creates an SSH key pair for the specified host and puts it into a named Kubernetes secret in Flux's management namespace (by default flux-system). The command also outputs the public key, which should be added to the forked repo's "Deploy keys" in GitHub.


```sh
GITHUB_USER=<your_github_username>
flux create secret git ssh-credentials \
  --url=ssh://git@github.com/${GITHUB_USER}/flux-helloenv-app
```

If you need to retrieve the public key later, you can extract it from the secret as follows:

```sh
kubectl get secret ssh-credentials -n flux-system -ojson \
  | jq -r '.data."identity.pub"' | base64 -d
```

Use the public key as a Deploy key in your fork of the flux-helloenv-app repo. Browse to the following URL, replacing `<your_github_username>` with your GitHub username: `https://github.com/<your_github_username>/flux-helloenv-app/settings/keys`.   
The page will appear as follows.     

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wq6lhb5xeob63ylt0vzl.png)


Click "Add deploy key" and paste the key data (starts with `ssh-<alg>...` ) into the contents. The name is arbitrary, we use flux-helloenv-app-secret here.

### Accessing the application

The default image version can be checked by accessing our application using curl command. If you are using minikube, make sure you enable the ingress addon so the ingress functionality works.  

```sh
minikube addons enable ingress
```  

If you are running on the local cluster, you will have to add the minikube IP to your `/etc/hosts` file. The command `minikube ip` will give the IP of your cluster.  

For example, the entry in `/etc/hosts` file will look like this:

```sh
192.168.49.2 helloenv.prod.com helloenv.stage.com
```

You can check the ingress that is available for stage and prod.

```
$ kubectl get ingress --all-namespaces

NAMESPACE          NAME       CLASS    HOSTS                ADDRESS   PORTS   AGE
helloenv-prod      helloenv   <none>   helloenv.prod.com              80      9m22s
helloenv-staging   helloenv   <none>   helloenv.stage.com             80      9m21s
```

Now, you can access the application with following commands.

```sh
curl helloenv.prod.com

curl helloenv.stage.com
```

The result of these `curl` commands will show the current default image versions that the deployment is using.

### Releasing to stage 

Now, let's make some changes to the app and release it to the stage. We can make a minor change to the [app.py](https://github.com/infracloudio/flux-helloenv-app/blob/main/app/app.py) file in the application code repository `flux-helloenv-app`, by changing the message and pushing it to the main branch, of course, in real case it will be pushed to main through a PR.  

This change triggers the [CI](https://github.com/infracloudio/flux-helloenv-app/blob/main/.github/workflows/ci.yaml) job configured on the application code repository. This job contains the logic to check the tag and then create the image tag based on it.

```yaml
run: |
  if [[ ${{ github.event.ref }} =~ ^refs/tags/[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
    echo "IMAGE_ID=${{ github.ref_name }}" >> "$GITHUB_OUTPUT"
  else
    ts=$(date +%s)
    branch=${GITHUB_REF##*/}
    echo "IMAGE_ID=${branch}-${GITHUB_SHA::8}-${ts}" >> "$GITHUB_OUTPUT"
  fi
```

From this snippet, it can be seen that it checks for the condition for prod and stage, as well as how we set the tagging of images.  

So, since we are doing this for stage, it will tag the image as `${branch}-${GITHUB_SHA::8}-${ts}`.

A useful tag format is `<branch>-<sha1>-<timestamp>`.  

Including the branch information with an image makes it easier to trace the source code's branch and commit associated with that image. Additionally, having the branch information and [unix time](https://en.wikipedia.org/wiki/Unix_time) allows you to filter for images originating from a [specific branch when needed](https://fluxcd.io/flux/guides/sortable-image-tags/).  

Since this CI stage for Staging will push the image to the image repository, the image update and image policy will kick in and replace the image with the latest built image based on timestamp.  

#### Check status of stage environment

We can check the status of image automation and image policy for staging.

```sh
$ flux get image all

NAME                    	LAST SCAN           	SUSPENDED	READY	MESSAGE                       
imagerepository/helloenv	2023-10-17T12:19:26Z	False    	True 	successful scan: found 9 tags	

NAME                        	LATEST IMAGE                                       	READY	MESSAGE                                                                                
imagepolicy/helloenv-prod   	docker.io/shapai/helloenv:1.0.2                    	True 	Latest image tag for 'docker.io/shapai/helloenv' resolved to 1.0.2                    
imagepolicy/helloenv-staging	docker.io/shapai/helloenv:main-b470560d-1696668188	True 	Latest image tag for 'docker.io/shapai/helloenv' resolved to Build-b470560d-1696668188

NAME                                  	LAST RUN            	SUSPENDED	READY	MESSAGE                                                      
imageupdateautomation/helloenv-prod   	2023-10-17T12:18:41Z	False    	True 	no updates made                                             	
imageupdateautomation/helloenv-staging	2023-10-17T12:18:39Z	False    	True 	no updates made; last commit f798e64 at 2023-10-07T09:25:32Z	
```

Let's curl the stage domain and see the output:

```sh
$ curl helloenv.stage.com

This is staging environment with (Version: main-b470560d-1696668188)
```

The output will give you the latest version, which you can verify with commit id and timestamp with which the image was created.

### Releasing to production

Now, assuming we have tested the release in stage and got go-ahead, we are ready to release in production by tagging the release.

We will be releasing it in prod by tagging the release. 

```sh
git tag -a 1.0.2 -m "prod modified"
git push --tags
```

This will, in turn, trigger the CI and create an image tag with that Git tag that we pushed.
Once this is triggered and pushed, the image automation in Flux will commit and push the changes to the flux-image-update branch.   

This will then create a PR using workflow in [flux-helloenv-app repo](https://github.com/infracloudio/flux-helloenv-app/blob/main/.github/workflows/auto-pr.yaml). This PR needs a manual review and approval since it is a prod environment. Once this gets approved, it changes the tag with the latest tag in [prod kustomization](https://github.com/infracloudio/flux-helloenv-app/blob/main/demo/kustomize/prod/kustomization.yaml).


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yjd4xxh5thx4o0hmdv73.png)


curl to the prod DNS should give you the latest tag as the version. 

```sh
$ curl helloenv.prod.com

This is production environment (Version: 1.0.2)
```

## Incident Management

During an incident, you may want to halt Flux from updating images in your Git repository. You can accomplish this by suspending image automation either in-cluster or by editing the ImageUpdateAutomation manifest in Git.

### In-cluster suspension

You can suspend image automation directly in a cluster using the following command:

```sh
flux suspend image update helloenv-prod
```

Alternatively, you can suspend image automation by editing the ImageUpdateAutomation manifest in Git. Here's an example of the manifest:

```yaml
kind: ImageUpdateAutomation
metadata:
  name: helloenv-prod
  namespace: flux-system
spec:
  suspend: true
```

### Resuming automation

Once the incident is resolved, you can resume the automation using the following command:

```sh
flux resume image update helloenv-prod
```

### Pausing automation for a specific image

If you want to pause automation for a particular image only, you can suspend and resume image scanning for that specific image. For example:

```sh
flux suspend image repository helloenv
```

### Reverting image updates

Assuming you've configured Flux to update an application to its latest stable version and an incident occurs, you can instruct Flux to revert to a previous image version.  

#### Reverting via command

For instance, to revert from version 1.0.1 to 1.0.0, you can use the following command:

```sh
flux create image policy helloenv-prod --image-ref=helloenv --select-semver=1.0.0
```

#### Reverting via Git manifest

You can also make this change by editing the ImagePolicy manifest in Git. Here's an example of the manifest:

```yaml
kind: ImagePolicy
metadata:
  name: helloenv-prod
  namespace: flux-system
spec:
  policy:
    semver:
      range: 1.0.0
```

### Updating the image policy

When a new version, e.g., 1.0.2, becomes available, you can update the policy again to consider only versions greater than 1.0.1. This can be achieved using the following command:

```sh
flux create image policy helloenv-prod --image-ref=helloenv --select-semver=">1.0.1"
```

This change will prompt Flux to update the podinfo deployment manifest in Git and roll out the specified image version in-cluster.

## Conclusion

Flux's image automation and GitOps are powerful solutions for managing container image updates. By combining image automation and image reflector controllers, organizations can automate image version updates in their Git repositories. This not only simplifies the process but also ensures consistency and reliability in application deployment.

The practical example illustrated how GitOps with Flux can streamline the workflow from staging to production, providing a structured approach to managing deployments as code changes occur. This approach enhances efficiency and reliability in the development pipeline, making it a valuable asset in modern [DevOps practices](/blogs/successful-devops-assessment/).

Thank you for taking the time to read our post. We hope you found it both informative and engaging. We highly value your feedback and would love to hear your thoughts on this topic. Let's kickstart a meaningful conversation on [LinkedIn](https://in.linkedin.com/in/shashank-pai-405928a9) to exchange ideas and insights.

If you're seeking assistance in crafting a robust DevOps strategy or considering outsourcing your DevOps operations to seasoned experts, we invite you to discover why numerous startups and enterprises regard us as one of the [top-tier DevOps consulting and services companies](/devops-consulting-services/).

## References

1. [Flux Core Concepts](https://fluxcd.io/flux/concepts/)
2. [Flux Automate image updates to Git](https://fluxcd.io/flux/guides/image-update/)
3. [How to make sortable image tags to use with automation](https://fluxcd.io/flux/guides/sortable-image-tags/)
4. [GitOps with Flux CD - KodeKloud](https://kodekloud.com/courses/gitops-with-fluxcd/)