Kustomize: managing Kubernetes manifests — an overview, and examples
Tags: kubernetes, todayilearned, tutorial, devops
Published: 2023-01-18 20:21:05

---
title: Kustomize: managing Kubernetes manifests — an overview, and examples
published: true
date: 2023-01-15 13:05:09 UTC
tags: kubernetes,todayilearned,tutorial,devops
canonical_url: https://rtfm.co.ua/en/kustomize-managing-kubernetes-manifests-an-overview-and-examples/
---

### Kustomize: managing Kubernetes manifests — an overview, and examples

![](https://cdn-images-1.medium.com/max/500/1*GUJpUDdpy9KTLXusf8RpDQ.png)

[Kustomize](https://kustomize.io/) is a configuration management tool for Kubernetes that allows you to use common sets of manifests that can be changed for each specific environment/cluster, and can be an alternative to (or complement) Helm templates.

The general concept of Kustomize is “where, what, and how”:

- “where” is a base manifest, for example deployment.yaml
- “what” — what exactly in the manifest need to change, for example, a number of pods (replicas) in a deployment
- “how” — Kustomize configuration files - ` kustomization.yaml`, describing how exactly to make a change

### Kustomize overview

As a simple example, let’s take a file `kustomization.yaml` with the following content:

```
resources:
- deployment.yaml
- service.yaml
namePrefix: dev-
namespace: development
commonLabels:
  environment: development
```

It describes that it needs to take the `resources` described in the files `deployment.yaml` and `service.yaml`, add the prefix _dev-_ (`namePrefix`) to the name of each created resource, deploy them to the `namespace` _development,_ and add labels `environment: development`.

See all options in the [Customize Feature List](https://translate.google.com/website?sl=ru&tl=en&hl=en&client=webapp&u=https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/%23kustomize-feature-list).

In addition, Kustomize is handy for creating configurations from common files but for different environments.

In this case, a directory overlays with its own `kustomization.yaml` file is used:

![](https://cdn-images-1.medium.com/max/776/0*zwT1C-PQ-cK-XOJU.png)

Since version 1.14, Kustomize is built into `kubectl`:

```
$ kubectl kustomize — help
Build a set of KRM resources using a ‘kustomization.yaml’ file. The DIR argument must be a path to a directory
containing ‘kustomization.yaml’, or a git repository URL with a path suffix specifying same with respect to the
repository root. If DIR is omitted, ‘.’ is assumed.
Examples:
Build the current working directory
kubectl kustomize
…
```

And it can be used with the apply command in order to first build (build) a required manifest, and immediately send it to the Kubernetes API:

```
$ kubectl apply — help
…
Apply resources from a directory containing kustomization.yaml — e.g. dir/kustomization.yaml
kubectl apply -k dir/
…
```

Since version 1.16, it is also available in `kubeadm`.

Besides `kubectl apply`, Kustomize can be used for:

- `kubectl get -k` - get resources from a Kubernetes cluster
- `kubectl describe -k` - resource description in a Kubernetes cluster
- `kubectl diff -k` - compare locally generated manifest with a resource in the cluster
- `kubectl delete -k` - remove a resource from a cluster

### Deploy with Kustomize

Create a test directory:

```
$ mkdir -p kustomize_example/base
cd kustomize_example/
```

Create two files in the directory base - in one we'll describe a Deployment, in the other - a Service:

```
$ vim -p base/deployment.yaml base/service.yaml
```

In the `deployment.yaml` describe launch of a Pod with the _nginxdemo_ container:

```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginxdemo
spec:
  selector:
    matchLabels:
      app: nginxdemo
  template:
    metadata:
      labels:
        app: nginxdemo
    spec:
      containers:
        - name: nginxdemo
          image: nginxdemos/hello
          ports:
          - name: http
            containerPort: 80
            protocol: TCP
```

And file `service.yaml` with a Service for this Deployment:

```
apiVersion: v1
kind: Service
metadata:
  name: nginxdemo
spec:
  selector:
    app: nginxdemo
  ports:
  - name: http
    port: 80
```

Next, in the same directory base, create a `kustomization.yaml`, where we will describe `resources` - files, from which Kustomize will assemble our future manifest for deployment:

```
resources:
  - deployment.yaml
  - service.yaml
```

Build the manifest:

```
$ kubectl kustomize base/
apiVersion: v1
kind: Service
metadata:
name: nginxdemo
spec:
ports:
- name: http
port: 80
selector:
app: nginxdemo
 — -
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginxdemo
spec:
selector:
matchLabels:
app: nginxdemo
template:
metadata:
labels:
app: nginxdemo
spec:
containers:
- image: nginxdemos/hello
name: nginxdemo
ports:
- containerPort: 80
name: http
protocol: TCP
```

Or via `kustomize` itself :

```
$ kustomize build base/
apiVersion: v1
kind: Service
metadata:
name: nginxdemo
spec:
…
```

Or build and immediately deploy:

```
$ kubectl apply -k base/
service/nginxdemo created
deployment.apps/nginxdemo created
```

Check:

```
$ kubectl get all -l app=nginxdemo
NAME READY STATUS RESTARTS AGE
pod/nginxdemo-7f8f587c74-kbczf 1/1 Running 0 26s
NAME DESIRED CURRENT READY AGE
replicaset.apps/nginxdemo-7f8f587c74 1 1 1 26s
```

Now let’s see how to set up this application for two environments — Dev and Prod.

### Kustomize Overlays

Create directories `overlays/dev` and `overlays/prod`:

```
$ mkdir -p overlays/{dev,prod}
```

We’ll get the following structure:

```
$ tree .
.
| — base
| | — deployment.yaml
| | — kustomization.yaml
| ` — service.yaml
` — overlays
| — dev
` — prod
```

In the directories dev and prod create separate `kustomization.yaml` files, in which specify bases:

```
bases:
- ../../base
```

If we’ll execute `kustomize build overlays/dev/` now, we will get a manifest similar to the one we created earlier.

### Kustomize features

#### namePrefix

To change the manifest, in the `kustomization.yaml` files for Dev and Prod add, for example, the `namePrefix`:

```
bases:
- ../../base

namePrefix: dev-
```

Check how it will look like now:

```
$ kustomize build overlays/dev/
apiVersion: v1
kind: Service
metadata:
name: dev-nginxdemo
spec:
ports:
- name: http
port: 80
selector:
app: nginxdemo
 — -
apiVersion: apps/v1
kind: Deployment
metadata:
name: dev-nginxdemo
…
```

The fields name now are prefixed with _dev-_.

#### patchesStrategicMerge

Next, let’s say we want to have 1 Pod on Dev, and 3 on Prod, i.e. change the `replicas` filed of the Deployment.

Let’s use the [`patchesStrategicMerge`](https://translate.google.com/website?sl=ru&tl=en&hl=en&client=webapp&u=https://github.com/kubernetes/community/blob/master/contributors/devel/sig-api-machinery/strategic-merge-patch.md).

Create a patch file  - ` overlays/dev/replicas.yaml`. The kind and name fileds of the resource to be patched must match the resource from base:

```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginxdemo
spec:
  replicas: 1
```

Similarly for Prod  -  file `overlays/prod/replicas.yaml`:

```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginxdemo
spec:
  replicas: 3
```

In the files `overlays/dev/kustomization.yaml` and `overlays/prod/kustomization.yaml` add the `patchesStrategicMerge`:

```
bases:
- ../../base

namePrefix: dev-

patchesStrategicMerge:
- replicas.yaml
```

Run it:

```
$ kustomize build overlays/dev/
apiVersion: v1
kind: Service
metadata:
name: dev-nginxdemo
spec:
ports:
- name: http
port: 80
selector:
app: nginxdemo
 — -
apiVersion: apps/v1
kind: Deployment
metadata:
name: dev-nginxdemo
spec:
replicas: 1
…
```

Deploy:

```
$ kubectl apply -k overlays/dev/
service/dev-nginxdemo created
deployment.apps/dev-nginxdemo created
kubectl apply -k overlays/prod/
service/prod-nginxdemo created
deployment.apps/prod-nginxdemo created
```

And check:

```
$ kubectl get all -l app=nginxdemo
NAME READY STATUS RESTARTS AGE
pod/dev-nginxdemo-7f8f587c74-vh2gn 1/1 Running 0 37s
pod/nginxdemo-7f8f587c74-kbczf 1/1 Running 0 104m
pod/prod-nginxdemo-7f8f587c74-dpc76 1/1 Running 0 33s
pod/prod-nginxdemo-7f8f587c74-f5j4f 1/1 Running 0 33s
pod/prod-nginxdemo-7f8f587c74-zqg8z 1/1 Running 0 33s

NAME DESIRED CURRENT READY AGE
replicaset.apps/dev-nginxdemo-7f8f587c74 1 1 1 37s
replicaset.apps/nginxdemo-7f8f587c74 1 1 1 104m
replicaset.apps/prod-nginxdemo-7f8f587c74 3 3 3 33s
```

#### configMapGenerator та secretGenerator

Kustomize can also generate new resources from templates.

Let’s take ConfgiMap for [Grafana Loki alerts](https://rtfm.co.ua/en/grafana-loki-alerti-z-ruler-ta-labels-z-logiv/) as an example.

Since the alerts are the same for both Dev and Prod, we can describe them with the `configMapGenerator` in the `base/kustomization.yaml`:

```
resources:
  - deployment.yaml
  - service.yaml

configMapGenerator:
- name: loki-ruler-alerts
  files:
  - loki-ruler-alerts.yaml
```

In the base directory, create the file itself `loki-ruler-alers.yaml` with the contents of ConfigMap:

```
groups:
  - name: systemd-alerts
    rules:
      - alert: Pod killed by OOM Killer
        expr: |
          sum(rate({job="systemd-journal"} |~ ".*OOM-killed.*" | regexp `pod=".*/(?P<pod>[a-zA-Z].*)".*` | pod!="" [15m])) by (pod, hostname) > 0.1
        for: 1s
        labels:
          severity: warning
        annotations:
          description: |-
            *OOM Killer detected in the WorkerNode's systemd-journal logs*
            WorkerNode: {{`{{ $labels.hostname }}`}}
```

Check it:

```
$ kustomize build base/
apiVersion: v1
data:
loki-ruler-alerts.yaml: |
groups:
- name: systemd-alerts
rules:
- alert: Pod killed by OOM Killer
expr: |
sum(rate({job=”systemd-journal”} |~ “.*OOM-killed.*” | regexp `pod=”.*/(?P<pod>[a-zA-Z].*)”.*` | pod!=”” [15m])) by (pod, hostname) > 0.1
for: 1s
labels:
severity: warning
annotations:
description: |-
*OOM Killer detected in the WorkerNode’s systemd-journal logs*
WorkerNode: {{`{{ $labels.hostname }}`}}
kind: ConfigMap
metadata:
name: loki-ruler-alerts-47678t7d89
 — -
apiVersion: v1
kind: Service
metadata:
name: nginxdemo
…
```

It is also possible to generate data from the command line.

For example, to add to the `base/kustomization.yaml` a new Kubernetes Secret, execute `kustomize edit add secret`:

```
$ cd base/
$ kustomize edit add secret nginx-password — from-literal=password=12345678
```

Check:

```
$ cat kustomization.yaml 
resources:
- deployment.yaml
- service.yaml

configMapGenerator:
- files:
  - loki-ruler-alerts.yaml
  name: loki-ruler-alerts
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
secretGenerator:
- literals:
  - password=12345678
  name: nginx-password
  type: Opaque
```

#### generatorOptions

If we apply the `base/kustomization.yaml`, then postfixes will be added to the names of ConfigMap and Secret:

```
$ kubectl apply -k base/
configmap/loki-ruler-alerts-47678t7d89 created
secret/nginx-password-72mh6dg77t created
service/nginxdemo unchanged
deployment.apps/nginxdemo unchanged
```

_47678t7d89_ and _72mh6dg77t_.

To change this behavior, add the [`generatorOptions`](https://translate.google.com/website?sl=ru&tl=en&hl=en&client=webapp&u=https://github.com/kubernetes-sigs/kustomize/blob/master/examples/generatorOptions.md%23generator-options) with the option `disableNameSuffixHash`:

```
resources:
- deployment.yaml
- service.yaml
  
configMapGenerator:
- files:
  - loki-ruler-alerts.yaml
  name: loki-ruler-alerts

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
secretGenerator:
- literals:
  - password=12345678
  name: nginx-password
  type: Opaque

generatorOptions:
  disableNameSuffixHash: true
```

Deploy it:

```
$ kubectl apply -k base/
configmap/loki-ruler-alerts created
secret/nginx-password created
service/nginxdemo unchanged
deployment.apps/nginxdemo unchanged
```

Now we have the names as we specified them in the template.

### Helm && Kustomize

And an example of how we can use [Helm](https://rtfm-co-ua.translate.goog/ru/helm-kubernetes-package-manager-obzor-nachalo-raboty/?_x_tr_sl=ru&_x_tr_tl=en&_x_tr_hl=en&_x_tr_pto=wapp) and Kustomize together.

For example, when you have a chart fork and you don’t want to change the data in it.

Create a helm chart directory:

```
$ mkdir -p kustomize-helm
```

Generate a chart in it:

```
$ helm create kustomize-helm
Creating kustomize-helm
```

Get the structure of a standard chart:

```
$ tree .
.
| — kustomize-helm
| | — Chart.yaml
| | — charts
| | — templates
| | | — NOTES.txt
| | | — _helpers.tpl
| | | — deployment.yaml
| | | — hpa.yaml
| | | — ingress.yaml
| | | — service.yaml
| | | — serviceaccount.yaml
| | ` — tests
| | ` — test-connection.yaml
| ` — values.yaml
` — templates
```

If we’ll execute `helm template kustomize-helm`, we will see the generated chart templates:

```
$ helm template kustomize-helm
 — -
Source: kustomize-helm/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: release-name-kustomize-helm
labels:
helm.sh/chart: kustomize-helm-0.1.0
app.kubernetes.io/name: kustomize-helm
app.kubernetes.io/instance: release-name
app.kubernetes.io/version: “1.16.0”
app.kubernetes.io/managed-by: Helm
 — -
Source: kustomize-helm/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
name: release-name-kustomize-helm
labels:
helm.sh/chart: kustomize-helm-0.1.0
app.kubernetes.io/name: kustomize-helm
app.kubernetes.io/instance: release-name
…
```

Now, in order to not change the chart, but to create a Secret, in the kustomize-helm directory create a file `kustomization.yaml`, in which we'll use resources with the file `helm-all.yaml`, that will be generated with helm template:

```
resources:
- helm-all.yaml

secretGenerator:
- literals:
  - password=12345678
  name: nginx-password
  type: Opaque
```

Run it:

```
$ cd kustomize-helm/
helm template . > helm-all.yaml && kustomize build .
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
app.kubernetes.io/instance: release-name
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: kustomize-helm
app.kubernetes.io/version: 1.16.0
helm.sh/chart: kustomize-helm-0.1.0
name: release-name-kustomize-helm
 — -
apiVersion: v1
data:
password: MTIzNDU2Nzg=
kind: Secret
metadata:
name: nginx-password-72mh6dg77t
type: Opaque
 — -
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/instance: release-name
…
```

Done.

_Originally published at_ [_RTFM: Linux, DevOps, and system administration_](https://rtfm.co.ua/en/kustomize-managing-kubernetes-manifests-an-overview-and-examples/)_._

* * *