Refactoring GitOps repository to support both real-time and reconciliation window changes
Tags: flux, gitop, kubernetes
Published: 2023-01-13 09:36:49

> **Restructuring GitOps repository to be able to enable multiple reconciliation types. eg real-time and reconciliation window changes with the approach described in the [previous part](https://dev.to/mahrrah/how-to-enable-reconciliation-windows-using-flux-and-k8s-native-components-2d4i).**

For some scenarios allowing only updates to be applied during a reconciliation window is not enough.
There are cases when some application resources should be managed in real time, but others are still only allowed to change during a reconciliation window.
The example we use here is a `nginx` deployment to the cluster, which contains a `Deployment`, `Service`, and a `ConfigMap` manifest.
The `ConfigMap`, which defines the `nginx.conf` should me manageable in real time. However, the `Deployment` and the `Service` should only be changed with in a reconciliation window.

Hence, the problem statement changes slightly from the last part:

_We want to enable two ways of applying changes to a cluster using Flux:_

- _**Real-time changes:** Representing the default behavior of Flux when it comes to reconciling changes._
- _**Reconciliation windows changes:** Predefined time windows in which a change can be applied to the resource by Flux._

We can still use the core approach shown [here](https://dev.to/mahrrah/how-to-enable-reconciliation-windows-using-flux-and-k8s-native-components-2d4i) to solve our new problem. However, we need to make some adjustments to how we organize our GitOps repository, to enable real-time as well as reconciliation window changes.

Even though we are only demonstrating the restructuring of this GitOps repository on two reconciliation types. This approach can easily be extended for more types. Just note that, for each new type of reconciliation window, corresponding set of of CronJobs are needed to manage the new windows.

## Pre-requisits:

- **IMPORTANT:** If you haven't already read the [first part](https://dev.to/mahrrah/how-to-enable-reconciliation-windows-using-flux-and-k8s-native-components-2d4i), go back and do so, as we will use its approach on how to enable the reconciliation window in this blog.
- Intermediate knowledge of [Flux](https://fluxcd.io/flux/), [Kustomize](https://kustomize.io/) and [K8s](https://kubernetes.io/)

## Core Principles

Before we start restructuring the repository, it might be useful to understand why we have to do so in the first place.

As covered in the previous blog, to be able to control the reconciliation cycle differently for a group of resources, these resources need to be managed by an independent `Kustomization` resource.

Because of this the goal of the following sections are:
"Restructure the GitOps repository such that its resources can be managed by one of the N-`Kustomization` resources we will create.
Where N defines the number of schedules for applying changes."

As in this blog we are only interested in real-time and reconciliation window changes, N is equal to 2.

## Set up

### 1. Set up your applications or components

Let's start with the smallest unit of grouping we have in our GitOps repository: `apps`

Looking at the example in [this sample](https://github.com/MahrRah/flux-reconciliation-windows-sample/tree/main/Sample1), under `apps` we have an `nginx` folder, which contains the `Deployment`, a `Service`, and a `ConfigMap` manifest.

```text
apps
└── nginx
    ├── kustomization.yaml
    ├── deployment.yaml
    ├── service.yaml
    └── configmap.yaml
```

As mentioned, we want to now make sure we can change the `nginx` server configuration, defined in the `configmap.yaml` in real time, but infrastructure changes such as deployment and the service should only change between Monday 8 am to Thursday 5 pm.

To enable this, the first step is to make sure we can split resources that can be changed real-time from resources that can only change state during a reconciliation window from [`kustomizes`](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/) point of view.

> **Note:** If you are not familiar with how `kustomize` is used to manage resources check out the official doc from Kubernetes on this at [Overview of Kustomize](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/)

One of the ways we can achieve this is by splitting all the resources for each application we have defined under `apps/` (see [default GitOps folder structure for mono repos](https://fluxcd.io/flux/guides/repository-structure/#repository-structure)) into two versions. These versions' sole purpose is to package the resources to be either managed by the real-time or the reconciliation window `Kustomization` resource.

We can then split all manifest files into these two subfolders and add the respective suffixes to the subfolders:

- Real-time changes: `-rt`
- Reconciliation windows changes: `-rw`

Original structure:

```text
apps
└── nginx
    ├── kustomization.yaml
    ├── deployment.yaml
    ├── service.yaml
    └── configmap.yaml
```

Enabeling real time and reconciliation windows changes:

```text
apps
└── nginx
    ├── nginx-rt
    │   ├── kustomization.yaml
    │   └── configmap.yaml
    └── nginx-rw
        ├── kustomization.yaml
        ├── deployment.yaml
        └── service.yaml
```

The result of this splitting you can see in the sample repository [here](https://github.com/MahrRah/flux-reconciliation-windows-sample/tree/main/Sample2/apps/nginx)

### 2. Set up your clusters

The next step is to restructure the clusters directory. The goal is to make sure we can create two independents `Kustomization` resources. This means we need two entry points to point each of the `Kustomization` resources to.
For that we split the previous `apps` into two subfolders, `apps-rt`/`apps-rw`.
Where `./cluster/<cluster_name>/apps/apps-rt` will be the entry point for the real-time `Kustomization` resources and `./cluster/<cluster_name>/apps/apps-rw` for the reconciliation window controller.

Original structure:

```text
clusters/cluster-1
├── apps
│    └── nginx
└── infra
     └── reconciliation-windows
```

Enabeling real time and reconciliation windows changes:

```text
clusters/cluster-1
├── apps
│   ├── apps-rw
│   │   └── nginx
│   └── apps-rt
│       └── nginx
└── infra
      └── reconciliation-windows
```

Next, we need to add the `kustomization.yaml` and make sure they reference the right resources.

Let's first have a look at the the `kustomization.yaml` in `clusters/cluster-1/apps/app-rw` and `clusters/cluster-1/apps/app-rt` setup.
Both `app-rw` and `app-rt` will have a root `kustomization.yaml` which will point to all applications deployed onto the cluster. In our example, this is only the `nginx` app.

Folder structure:

```text
clusters/cluster-1
├── apps
│   ├── apps-rw
│   │   ├── kustomization.yaml
│   │   └── nginx
│   └── apps-rt
│       ├── kustomization.yaml
│       └── nginx
└── infra
```

The `kustomization.yaml` files:

```yaml
#clusters/cluster-1/apps/apps-rw/kustomization.yaml
resources:
  - ./nginx
```

```yaml
#clusters/cluster-1/apps/apps-rt/kustomization.yaml
resources:
  - ./nginx
```

Going one level deeper, both the `nginx` under `clusters/cluster-1/apps/app-rw` and `clusters/cluster-1/apps/app-rt` have a similar setup.
To not go over the same thing twice, we are going to only have a look at the `clusters/cluster-1/apps/app-rt`. To see the setup of the `app-rw` you can check the sample [here](https://github.com/MahrRah/flux-reconciliation-windows-sample/tree/main/Sample2/clusters/cluster-1/apps/apps-rw).

Folder structure:

```text
clusters/cluster-1
├── apps
│   ├── apps-rw
│   └── apps-rt
│       ├── kustomization.yaml
│       └── nginx
│           ├── namespace.yaml
│           └── kustomization.yaml
└── infra
```

The `kustomization.yaml` files:

```yaml
#clusters/cluster-1/apps/apps-rt/nginx/kustomization.yaml
resources:
  - ./../../../../../apps/nginx/nginx-rt
  - ./namespace.yaml
```

As shown above, the application resources referenced under `clusters/cluster-1/apps/apps-rt` are the resources we bundled up under `apps/nginx/nginx-rt` and should now only contain resources that can be changed in real-time.

And just like that you have separated all configurations to be managed by different `Kustomization` resources!

### Set up `Kustomization` resources.

Our GitOps repository is ready now, but how do we set up the `Kustomization` resources?
Let's first create a flux `Source` resources.

```sh
flux create source git source \
    --url="https://github.com/<github-handle>/flux-reconciliation -windows-sample" \
    --username=<username>\
    --password=<PAT> \
    --branch=main \
    --interval=1m \
    --git-implementation=libgit2 \
    --silent
```

Next, we now need two controllers for apps and one for infra.

```sh
flux create kustomization infra \
    --path="./clusters/cluster-1/infra" \
    --source=source\
    --prune=true \
    --interval=1m
```

```sh
flux create kustomization apps-rt \
    --depends-on=infra \
    --path="./clusters/cluster-1/apps/apps-rt" \
    --source=source\
    --prune=true \
    --interval=1m
```

```sh
flux create kustomization apps-rw \
    --depends-on= apps-rt \
    --path="./clusters/cluster-1/apps/apps-rw" \
    --source=source\
    --prune=true \
    --interval=1m
```

Not this should give you something like this.

```sh
user@cluster:~$ flux get kustomization
NAME    REVISION        SUSPENDED READY MESSAGE
infra   main/7cf3aaf  False     True  Applied revision: main/7cf3aaf
apps-rt main/7cf3aaf  False     True  Applied revision: main/7cf3aaf
apps-rw main/7cf3aaf  False     True  Applied revision: main/7cf3aaf
```

## Demo

Now that the cluster is set up, we can upgrade the `nginx` version and change the configuration `nginx.conf` to include the `nginx_status` endpoint and see how one is visible right away, while the other needs a reconciliation window to open.

#### 1. Initial state

Before we do any changes, we can check out the current state of the nginx deployment.
Get the public `ip` address of the machine you are running your cluster on and navigate to the `http://<ip>:8080/` we should see somehing like this.

> **Note:** if you are running it locally you can replace the `ip` with `localhost`

![`Nginx` landing page](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sfxyhpng1unsjfd6u6nn.jpg)

We can download the `nginx.conf` file by clicking on it and see what configuration is currently mounted into the `nginx` pod from the `ConfigMap`.

#### 2. Change state

The next step is to change the state of our application.
To change the state of the application we can change the image version number from `1.14.2` to the (currently) newest image `1.23.3` inside the `apps/nginx/nginx-rw/deployment.yaml`. And in the same commit, we can add the configuration shown below to the `nginx.conf` section in the `apps/nginx/nginx-rt/configmaps.yaml` file to include the new status endpoint.

```conf
location /nginx_status {
                stub_status;
                allow all;
            }
```

#### 3. See real-time changes

Now if we go back to the browser, refresh the page and re-download the file `nginx.conf`, we should see the new section we just added.

> **Note:** It might take up to 2 minutes in the worst case for the `Source` and then `Kustomization` resource to reconcile

#### 4. Wait for reconciliation window to open

If we now wait till the next reconciliation window opens, the pod should be restarted and we should be able to see the version either by checking the resource.

```bash
kubectl describe pod  <nginx-podname> -n nginx
```

Or if you don't want to access the machine directly you can go to a non-existing route in the browser eg http://<ip>:8080/settings/`. There you should see a standard `nginx` 404 page which contains the current deployed version at the bottom.

## Conclusions

Let's summarize what we did when it came to restructuring the repository.

1. We separated all application resources into two sub-versions. One for resources which can be changed in real-time and one for resources that can only be changed when a reconciliation window is open.

2. We split the `clusters` directory in such a way, so that we can create two independent `Kustomization` resources, which reference either one or the other application sub-version.

After this we could create the infra and the two apps `Kustomization` resource and start using the solution, as demonstrated.

So, at its core it boils down to separating the resource definition, in such a way that they are only managed by one of the `Kustomization` resources created. This can be done like it's shown above, or slightly differently to fit your needs.

But hopefully after this second part, you should be good to go on using these reconciliation windows and have the knowledge on how to tweak the setup to fit your use case :)