Deploy with Kustomize, FluxCD and Remote Resources
Tags: kubernetes, devops, kustomize, fluxcd
Published: 2022-08-12 14:41:32

---
title: Deploy with Kustomize, FluxCD and Remote Resources
published: true
description: How to use Kustomize and remote resources in FluxCD 🚀
tags: kubernetes, DevOps, kustomize, fluxcd
cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/b4sw8oarlt74wa4yz4a5.jpg
# Use a ratio of 100:42 for best results.
---

## Introduction

As a Kubernetes user, I chose to use the default tooling available, named `Kustomize`, to manage my manifests and all modifications I need to do over them for each environment I want to deploy. For that, [`Kustomize`](https://kustomize.io/) is a powerful solution based on concepts of [inheritance](https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/resource/) (`resources`) and [composition](https://kubectl.docs.kubernetes.io/guides/config_management/components/) (`components`).

But one thing I miss the most is the ability to distribute manifests to other people, letting them inherit or compose them as they want. This is one subject where `helm` is ahead of `Kustomize` now.

![beautiful landscape](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gh3ffmhnziqp6509y2rp.jpg)

However, `Kustomize` provides a mechanism to use `remote` elements, for both `resources` and `components` using `git` under the hood. This solution is perfect, because we can use any `git` repository as a `yaml` registry for our `reousrces` and `components` 🔥.

But, the downside is `FluxCD` does not supports so well, for [a lot of good reasons](https://github.com/fluxcd/flux2/issues/1749#issuecomment-905410469) (especially when your repositories are private)… In this article, we will see how to bypass this limitation and use remote `resources` and `components`, the `FluxCD` way!

## Cluster & FluxCD configuration

In this article, I won't detail so much the cluster & flux configuration… because there is already plenty of documentation and blogposts about that!

For this article, I have set up a cluster using [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine) (aka GKE) using this [documentation](https://cloud.google.com/kubernetes-engine/docs/how-to/creating-a-zonal-cluster) and installed `FluxCD` connected to a GitLab repository with the following command (extracted from the [official documentation](https://fluxcd.io/legacy/flux/tutorials/get-started/)).

```shell
$ flux bootstrap gitlab \
  --owner=davinkevin.fr/articles/flux-and-kustomize-remote-base \
  --repository=clusters \
  --branch=gke \
  --path=fluxcd
```

**NOTE**: I don't like to use `default` branch, so I chose to create a branch for this cluster, so if I have another cluster, I just need another branch 😉. This is some kind of [GitLab Flow](https://docs.gitlab.com/ee/topics/gitlab_flow.html) for environment.

With this, I have a complete Kubernetes Cluster ready for GitOps, using [`clusters` repository](https://gitlab.com/davinkevin.fr/articles/flux-and-kustomize-remote-base/clusters) as source-of-truth. Every modification made in this repository will be automatically deployed by `FluxCD` 🤩.

## `bases` repository

I have set up another `git` repository dedicated to `resources` and `components`. It hosts what we usually call `base`, because it contains the common core of an application and also many `components` to enable some extra features (at infrastructure or application level).

This repository [is available in GitLab](https://gitlab.com/davinkevin.fr/articles/flux-and-kustomize-remote-base/bases) and is agnostic of my deployment environment. Again, I only host here the bare definition of my applications, we can compare this to an `helm` registry. Because it is managed in `git`, I can use branch & tags for management 🚀.

**NOTE**: Usually, this kind of `yaml` repository is only populated by CI from other projects automatically.

## PodInfo base publication

I have chosen the well known [`podinfo` application](https://github.com/stefanprodan/podinfo) as an example, and I have developed the `base` and some `components` associated to it.

```text
.
├── README.md
└── podinfo
    └── base
        ├── components
        │   ├── hpa
        │   │   ├── hpa.yaml
        │   │   └── kustomization.yaml
        │   ├── ingress
        │   │   ├── ingress.yaml
        │   │   └── kustomization.yaml
        │   └── redis
        │       ├── kustomization.yaml
        │       ├── redis.conf
        │       └── redis.yaml
        ├── kustomization.yaml
        └── podinfo.yaml
```

### Core PodInfo

This part is the [common part of all potential deployment](https://gitlab.com/davinkevin.fr/articles/flux-and-kustomize-remote-base/bases/-/blob/podinfo/podinfo/base/podinfo.yaml). It contains:
* The `podinfo` Kubernetes Service
* The `podinfo` Kubernetes Deployment

And nothing else… if you need to deploy any other part of the app, you have to use `components` 👇.

### Components

In this [`components`](https://gitlab.com/davinkevin.fr/articles/flux-and-kustomize-remote-base/bases/-/tree/podinfo/podinfo/base/components), we can find all **optional** part of our application. As an end-user, I need to declare those I want to use, because by default, they are not applied at all in the common core.

Here, we have:
* [`hpa`](https://gitlab.com/davinkevin.fr/articles/flux-and-kustomize-remote-base/bases/-/tree/podinfo/podinfo/base/components/hpa) with a default preconfigured [`Horizontal-Pod-Autoscaler`](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/) if you want the application to scale up and down automatically
* [`ingress`](https://gitlab.com/davinkevin.fr/articles/flux-and-kustomize-remote-base/bases/-/tree/podinfo/podinfo/base/components/ingress) definition to expose the application to the outside world ([see official documentation](https://kubernetes.io/docs/concepts/services-networking/ingress/))
* [`redis`](https://gitlab.com/davinkevin.fr/articles/flux-and-kustomize-remote-base/bases/-/tree/podinfo/podinfo/base/components/redis) component to enable cache on the PodInfo application with default configuration.

It's interesting to note components can be of any sort, `hpa` and `ingress` are focused on infrastructure when `redis` is a feature of the application. Without this `redis` component, the application is able to work, just without any caching system. This `redis` component is here to deploy a `redis` server (of course 😅) but also to enable caching at `podinfo` level too.

**NOTE**: If you want a better description of this `component` feature, I advise you to discover [this article from the official documentation](https://kubectl.docs.kubernetes.io/guides/config_management/components/)

So, we have a complete, environment agnostic `resource` and `components` committed to our [`base` repository](https://gitlab.com/davinkevin.fr/articles/flux-and-kustomize-remote-base/bases/-/tree/podinfo/). Now, It is time to use it!

## Publish PodInfo to `dev`

Back to our cluster, I want to deploy this `podinfo` application, but of course I want to apply some extra customizations on it, like:

* Define `components` I want to enable
* Define a domain name for the `ingress` definition
* Define the required `tls` configuration to the `ingress`
* Define an alternative registry for the image

So, to do that, I need to have [one (or multiple) `kustomization.yaml`](https://gitlab.com/davinkevin.fr/articles/flux-and-kustomize-remote-base/clusters/-/blob/gke/podinfo-dev/kustomization.yaml) to define this:

```yaml
# <clusters-repo>/podinfo-dev/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: podinfo-dev

images:
  - name: podinfo
    newName: ghcr.io/stefanprodan/podinfo

resources:
  - ../base
  - ingress

components:
  - ../base/components/redis
```

**NOTE**: The `ingress` modification is in [its own file](https://gitlab.com/davinkevin.fr/articles/flux-and-kustomize-remote-base/clusters/-/blob/gke/podinfo-dev/ingress/kustomization.yaml) to keep this one simple enough.

But, what is this `../base` folder? There is no `base` at all [in `cluster` repository](https://gitlab.com/davinkevin.fr/articles/flux-and-kustomize-remote-base/clusters). And you are right… we will use `FluxCD` feature available in flux to `include` the `base` repository at the right location during `reconciliation`. To do that, we need to define multiple flux resources.

### `podinfo-base` flux repository

First, we need a representation of our `flux.GitRepository` for `bases`, which is materialized in flux with:

```yaml
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: podinfo-base-dev
  namespace: flux-system
spec:
  interval: 5m
  ref:
    branch: podinfo
  url: ssh://git@gitlab.com/davinkevin.fr/articles/flux-and-kustomize-remote-base/bases
  secretRef:
    name: flux-system
```

**NOTE**: Here, I have chosen to define `spec.ref.branch: podinfo` to follow every commit made to this branch. It is, for development, a continuous deployment system 🚀.

### `podinfo` flux repository

Now, we need to create a `flux.GitRepository` for manifests in `clusters` repository **and** manifests from the `bases` repository created in the step before. To do that, we are going to use the [`include` feature](https://fluxcd.io/docs/components/source/gitrepositories/#include) from FluxCD:

```yaml
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: podinfo-dev
  namespace: flux-system
spec:
  interval: 5m
  include:
    - repository:
        name: podinfo-base-dev
      fromPath: podinfo/base
      toPath: base
  ref:
    branch: gke
  url: ssh://git@gitlab.com/davinkevin.fr/articles/flux-and-kustomize-remote-base/clusters
  secretRef:
    name: flux-system
```

The important part is the following:

```yaml
spec:
  include:
    - repository:
        name: podinfo-base-dev # <1>
      fromPath: podinfo/base # <2>
      toPath: base # <3>
```

This means the folder `podinfo/base` (**2**) previously defined in the `podinfo-base-dev` `flux.GitRepository` (**1**) will be injected in the path `base` (**3**) of the current `podinfo-dev` `flux.GitRepository` during FluxCD reconciliation… making all `resources` and `components` available.

### `podinfo-dev` flux Kustomization

This one is the standard `flux.Kustomization` you would use in any case. This one is just using the composite `flux.GitRepository` from the previous step as source (`spec.sourceRef`):

```yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: podinfo-dev
  namespace: flux-system
spec:
  suspend: false
  interval: 10m0s
  path: podinfo-dev # file path in the <clusters> repository  
  prune: true
  sourceRef:
    kind: GitRepository
    namespace: flux-system
    name: podinfo-dev
  validation: client
```

If we deploy all those files (you can put them in [the same one](https://gitlab.com/davinkevin.fr/articles/flux-and-kustomize-remote-base/clusters/-/blob/gke/fluxcd/podinfo-dev.yaml)), you will see your application deployed by FluxCD with your [provided custom configuration](https://gitlab.com/davinkevin.fr/articles/flux-and-kustomize-remote-base/clusters/-/tree/gke/podinfo-dev).

```shell
❯ kubectl get all -n podinfo-dev
NAME                           READY   STATUS    RESTARTS   AGE
pod/redis-869ff7c78b-jjbbk     1/1     Running   0          103m
pod/podinfo-6dbf56d6d7-ctxxr   1/1     Running   0          103m

NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
service/podinfo   ClusterIP   10.43.115.207   <none>        9898/TCP,9999/TCP   103m
service/redis     ClusterIP   10.43.192.84    <none>        6379/TCP            103m

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/redis     1/1     1            1           103m
deployment.apps/podinfo   1/1     1            1           103m

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/redis-869ff7c78b     1         1         1       103m
replicaset.apps/podinfo-6dbf56d6d7   1         1         1       103m
```

## Publish PodInfo to `prod`

Now, we want to use again the same `base` this time for a production environment.

Because we want something relatively stable, we chose to define `spec.ref.tag` to a specific tag in the `bases` repository (instead of the `podinfo` branch).

```yaml
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: podinfo-base-prod
  namespace: flux-system
spec:
  interval: 5m
  ref:
    tag: podinfo-1.0
  url: ssh://git@gitlab.com/davinkevin.fr/articles/flux-and-kustomize-remote-base/bases
  secretRef:
    name: flux-system
```

All others `flux.Kustomization` and `flux.GitRepository` are [the same](https://gitlab.com/davinkevin.fr/articles/flux-and-kustomize-remote-base/clusters/-/blob/gke/fluxcd/podinfo-prod.yaml).

Because we are in production, we also want to enable different modules, here the `HorizontalPodAutoscaler`. So we have a different [`kustomization.yaml`](https://gitlab.com/davinkevin.fr/articles/flux-and-kustomize-remote-base/clusters/-/blob/gke/podinfo-prod/kustomization.yaml). Thanks to the full control we have over the `kustomization.yaml` per environment, we can modify this only in production 😍.

Once everything is published and FluxCD reconciliation is OK, we have:

```shell
❯ kubectl get all -n podinfo-prod
NAME                           READY   STATUS    RESTARTS   AGE
pod/redis-869ff7c78b-lnll5     1/1     Running   0          113m
pod/podinfo-6dbf56d6d7-k8rjl   1/1     Running   0          113m
pod/podinfo-6dbf56d6d7-qldr8   1/1     Running   0          113m

NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
service/podinfo   ClusterIP   10.43.109.213   <none>        9898/TCP,9999/TCP   113m
service/redis     ClusterIP   10.43.252.54    <none>        6379/TCP            113m

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/redis     1/1     1            1           113m
deployment.apps/podinfo   2/2     2            2           113m

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/redis-869ff7c78b     1         1         1       113m
replicaset.apps/podinfo-6dbf56d6d7   2         2         2       113m

NAME                                          REFERENCE            TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/podinfo   Deployment/podinfo   4%/99%    2         4         2          113m
```

## Conclusion

It was tough, with many `yaml` but we have achieved our original goal. Being able to publish bare manifests in a central place and allow other team / people to consume them instantaneously with all features available in Kustomize.
We are then able to enjoy all the feature of a GitOps model with a dedicated (`yaml`) registry, like `helm` already have 😇.

![light after the article…](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8jpxcjoo57lqh0odlfbj.jpg)

There is some follow-up improvements to come, like OCI registries (announced in [FluxCD 0.32](https://fluxcd.io/blog/2022/08/july-2022-update/)), to replace `flux.GitRepository`. We still need [this issue](https://github.com/fluxcd/source-controller/issues/854) to be solved to be able to use it with all features provided by Kustomize.

In a more distant future, some other solution like [porch](https://kpt.dev/guides/porch-user-guide) could be a good candidate to simplify again our deployment strategy… but it is currently in `alpha` stage.

I hope you liked this article, you can find [all resources used in GitLab](https://gitlab.com/davinkevin.fr/articles/flux-and-kustomize-remote-base), and you can reproduce it in your own cluster! If you have any questions, don't hesitate to comment or ping me on twitter [@davinkevin](https://twitter.com/davinkevin)