Taming FluxCD HelmReleases: The Kustomize Way approach
Tags: kubernetes, fluxcd, yaml, helm
Published: 2024-06-04 16:00:00

---
title: Taming FluxCD HelmReleases: The Kustomize Way approach  
published: true
description: HelmRelease looks like a good idea… but it has many problems. Let's see how to do better without HelmRelease for a better GitOps
tags: kubernetes, fluxcd, yaml, helm
cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/74sq5hjvn4w8fths8yb1.jpg
# published_at: 2024-06-04 18:00 +0200
---

[FluxCD](https://fluxcd.io/) is a powerful tool for managing deployments in Kubernetes using GitOps principles. While it offers a wide range of features, this post will explore scenarios where a simpler approach might be preferable, aligning with the #SimplerIsBetter philosophy.

**NOTE**: This article shares insights and perspectives gained through experience in various contexts and companies. Feel free to disagree, but with respect! 😉

## `HelmRelease`, what is this?

[`HelmRelease` is a custom resource provided by FluxCD](https://fluxcd.io/flux/use-cases/helm/), which provides users
a way to automatically install `helm` charts using FluxCD and its declarative system. 

For example, if you want to install the [PodInfo](https://github.com/stefanprodan/podinfo) app, you have to declare
the following manifest:

```yaml
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: podinfo
spec:
  chart:
    spec:
      chart: podinfo
      version: '6.5.*'
      sourceRef:
        kind: HelmRepository
        name: podinfo
      interval: 5m
  releaseName: podinfo
  values: # part dedicated to the all `values` the chart accept
    replicaCount: 2
```

**NOTE**: For the sake of simplicity, I kept only relevant attributes, but I can say FluxCD offers a rich API to cover most of use cases.

![Workflow](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lwizd0gvqtzctacvfjs5.png)

The simplified workflow can be described like this:
* The `HelmOperator` verifies and retrieves the Helm chart from a remote source like GitHub, GitLab, Artifactory or an OCI registry (recommended)
* It then renders the chart using configuration from the `HelmRelease` object, which can include values provided in different ways (inlined, using ConfigMap or Secret)
* Finally, the `HelmOperator` applies the generated `YAML` manifests to the Kubernetes API to install or upgrade the application.

This approach can work for simple deployments, but as an operator, I've encountered several design drawbacks that I'd like to discuss. Let's see these issues!

**NOTE** Like every GitOps solution, FluxCD requires a connection to a `GitRepository` too. To keep the diagram simple, I've not materialized this item. 

## GitOps, aka "only source of truth"

The GitOps philosophy is built on top of 3 key principles: 

* **Declarative System**: You define the desired state of your system (what you want) using declarative language ([4GL](https://en.wikipedia.org/wiki/Fourth-generation_programming_language)). This approach focuses on the "what" instead of the "how," making your configuration easier to understand and maintain.
* **System State Captured in a Git Repository**: The desired state of your system is stored in a Git repository. This provides a central location for managing your infrastructure configurations, enabling version control, collaboration, and easy rollbacks if needed.
* **Automatic Deployment system**: Any changes pushed to the Git repository trigger an automated deployment process. This automates the process of translating your desired state into actual changes within your system, reducing manual intervention and the risk of errors.

![3 pillars](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0ilcr9vf1y9jwia24au1.jpg)

The `HelmRelease` is a **Declarative** approach, where **automation** is managed by the controller. However, "**desired state of your system is stored in a Git repository**" and all it implies are not respected.

### External Chart Dependencies: A Potential Weak Point

A core principle of building resilient systems is ensuring the availability of all their components. When using `HelmRelease`, we introduce an external dependency, the location where the Helm chart resides.

![HelmRelease not able to fetch chart](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rr35t0rbcs9ilcb0mdr4.png)

If communication with this external location fails (due to server downtime, network issues, etc.), you might not be able to install or update your application. This introduces a potential single point of failure (SPOF) in your deployment process. Additionally, the desired state of your cluster is not solely defined by your `git` repository; it also depends on all the charts your system downloads.   

### Limited Visibility into Helm Chart Content 

Another purpose of this **System State Captured in a Git Repository** is its auditability. If you capture the complete state of a system in `git`, you can review it before an installation or upgrade. However, using `HelmRelease` introduces a layer of opacity.

While you declare the specific chart you want to use, you might not have a complete picture of what resources the chart will actually install in your cluster. It could potentially create various resources like `ClusterRoles`, `NetworkPolicies`, or `DaemonSets`. You can't say without… deploying it, running it locally or worth, reading the chart's source. 😞

{% twitter 1648619974934642689 %}

### Lack of Immutability in Helm Charts!

The problem we already have with container images applied the same way to `helm` charts. A chart produced and published at a specific date might be un-published or re-published with a different content. In those case, you expose your system to the two previous points again… 

![representation of immutability with zebras…](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wnafjo7x5jahkfhlycwb.jpg)

**NOTE** We can now store charts in OCI registries, with immutability feature. To leverage it, you have to complexify your system with `digest` pinning, because in security, we can't blindly trust a 3rd party 😇. 

### Chart customisation, another nightmare 👻

While Helm charts offer a convenient way to package deployments, maintaining them can be challenging due to their complexity. Kubernetes itself provides a wide range of configuration options, which can further complicate matters.

This complexity can lead to situations where the desired configuration isn't readily available within a chart.  For example, you might want to add an annotation to a workload or modify taints and tolerations, but the chart may not offer built-in ways to do so.

{% twitter 1622925402845970433 %}

To address this challenge, FluxCD introduced the concept of `postRenderers` (see [documentation](https://fluxcd.io/flux/components/helm/helmreleases/#post-renderers) within `HelmRelease` resources. This feature leverages the Kustomize API to customize deployments after the initial Helm chart rendering.

```yaml
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: podinfo
spec:
  releaseName: podinfo
  chart: { … }
  values: { … }
  postRenderers:
    - kustomize:
        patches:
          - target:
              version: v1
              kind: Deployment
              name: metrics-server
            patch: |
              - op: add
                path: /metadata/labels/environment
                value: production              
        images:
          - name: docker.io/bitnami/metrics-server
            newName: docker.io/bitnami/metrics-server
            newTag: 0.4.1-debian-10-r54
```

The configuration for `postRenderers` is separate from the Helm chart result. This can make it difficult to understand the complete picture of how the final deployment will be configured, potentially leading to hidden errors. 🤯

FluxCD doesn't provide robust mechanisms to analyze the resources created after the combined rendering and post-rendering steps. This can make troubleshooting issues arising from these modifications cumbersome. 😞

## Solution is simplicity 🚀!

We've discussed the challenges associated with relying solely on Helm charts within FluxCD deployments. These challenges can compromise the visibility, maintainability, and overall health of your GitOps workflow.

So, how can we achieve the ideal balance: a declarative system, automatic deployments, and a clear picture of your system state captured entirely within your Git repository?

![human printing document with an old machine made of wood](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7zogjy9jj9snirmvk8nb.jpg)

### Render the chart and store it into `git` for better auditability

The answer lies in a simple yet powerful sub-command – `helm template` (or `helmfile template` if you use [Helmfile](https://helmfile.readthedocs.io/)). This command allows you to locally render helm charts along with your desired values, generating the final deployment manifest files.

```shell
$ helm repo add podinfo https://stefanprodan.github.io/podinfo
"podinfo" has been added to your repositories
$ helm repo update podinfo
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "podinfo" chart repository
Update Complete. ⎈Happy Helming!⎈
$ echo "replicaCount: 2" > values.yaml
$ helm template podinfo/podinfo -f values.yaml --version 6.6.3 > podinfo.yaml
```

And that's it! It was not so complicated 😇. We have a file called `podinfo.yaml`, located in `/k8s/podinfo` of our `git` repository. 

This file is now yours, it can be read, analyzed and pushed to `kubernetes`. After this generation, your system is free from the external chart registry where the chart is located.

**NOTE** I recommend to **never** modify a file "generated" by another tool, because you will loose this modification during the next rendering. tldr; treat them as "read-only".

### How to deploy generated files with FluxCD?

Instead of using `HelmRelease`, we're going to use `Kustomization` resources provided by FluxCD. It is way simpler than `HelmRelease`, because it just deploys manifests located in a specific location.

```yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: podinfo
spec:
  sourceRef:
    kind: GitRepository
    name: our-gitops-repository
  path: "/k8s/podinfo" # our location in our GitOps repo
  prune: true
  timeout: 1m
```

Because we use the `GitRepository`, called `our-gitops-repository` and eventually used by `FluxCD` itself, there is no extra dependency in our system.

### Direct Customization with Kustomize

While `HelmRelease` offers `postRenderers` for some customizations, the `Kustomization` resource provides full access to the powerful Kustomize capabilities. 
This allows for more granular and flexible control over your manifests. Here's how to achieve a similar customization as the previous `postRenderer` example using a `kustomization.yaml` file:

```yaml
# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - podinfo.yaml # your generated file from the previous step

patches:
- target:
    version: v1
    kind: Deployment
    name: metrics-server
  patch: |
    - op: add
      path: /metadata/labels/environment
      value: production              

images:
- name: docker.io/bitnami/metrics-server
  newName: docker.io/bitnami/metrics-server
  newTag: 0.4.1-debian-10-r54
```

[kustomize](https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/) offers a wider range of functionalities compared to `postRenderers`. You can manipulate resources using features like `namePrefix`, `labels`, `replacements`, `components`… The list is too long to be detailed here 😇.

As a bonus point, you can run `kustomize build /k8s/podinfo/` and see the complete result of the generation before any interaction with FluxCD. 

### Enjoy reviews and audit with rich diff!

One of the significant advantages of managing your full Kubernetes state with GitOps is the ability to leverage Git's powerful version control capabilities for reviewing and auditing deployments.

From an operator perspective, there is nothing better than a clear and detailed diff views during tool upgrade:

![rancher/local-path-provisioner to v0.0.27 diff view](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tl6j9n7c5uusbhp8k7r9.png)

Obviously, your IDE will be your best friend to understand what happened, with clear context and details of changes:

![cert-manager modification history](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vlxqzs8koih0coeacccs.png)

**NOTE** Upgrade can be automated using tools like `renovate` or `dependabot`

## Conclusion

While Helm charts offer a convenient way to package deployments, maintaining them in a FluxCD workflow can introduce challenges related to **transparency**, **maintainability**, and **control**.

This article explored the limitations of `HelmReleases` and presented `helm template …` as a more powerful and flexible alternative, leveraging FluxCD `Kustomize` resource. Using Kustomize directly within your `git` repository, you gain greater control, visibility, and the benefits of Git version control for reviewing and auditing changes.

Ultimately, by adopting a Kustomize-based approach within your FluxCD workflows, you can achieve a more **declarative**, **transparent**, and **auditable** approach to managing your Kubernetes deployments.