yq : A command line tool that will help you handle your YAML resources better
Published: 2020-10-03 08:04:21


When you want to update the fields in your YAML files, the standard practice is to use a templating tool that can replace values dynamically. However, Kubernetes does not provide any templating mechanism of its own as the deployed manifests are supposed to be static YAML files.

Enter `yq`.

`yq` is a great command-line tool for templating YAML files, and it’s a lighter weight option to popular tools such as Helm and [Kustomize](https://kustomize.io/).




## What is `yq`?

`yq`  is a command-line tool designed to transform YAML. It is similar to `jq` which focuses on transforming JSON instead of YAML.



## What can it do?

`yq`  can take a YAML file as an input and:

1. Read values from the file.
2. Add new values.
3. Update Existing values.
4. Generate new YAML files.
5. Convert YAML into JSON.
6. Merge two or more YAML files.

## Installation

You can install `yq` on Mac OS with:

```bash
$ brew install yq
```
On Linux with:

```bash 
$ sudo add-apt-repository ppa:rmescandon/yq
sudo apt-get install yq
```
> In case you don't have the `add-apt-repository` command 
> installed, you can install it with `apt-get install software-
> properties-common`.


If you're on Windows, you can download the executable from [Github](https://github.com/mikefarah/yq/releases).




## Use case scenarios

### 1. Reading YAML values

Say you are working with the following Pod:

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  containers:
  - name: test-container
    image: k8s.gcr.io/busybox
    env:
    - name: DB_URL
      value: postgres://db_url:5432
```   

You could read the value for the environment variable `ENV` with the following command:

```bash
$ yq r pod.yaml "spec.containers[0].env[0].value"
postgres://db_url:5432

$ 
```

The command works as follows:
- `yq r` is the command to read a value from the YAML file.
- `pod.yaml` is the file path of the YAML that you want to read.
- `spec.containers[0].env[0].value` is the query path.



### 2. Changing YAML values


Let’s take the previous example here. Say you wish to deploy the app in a production environment and change the URL to the production database.


Here’s how you can do the same using just a single command:

```bash
$ yq w pod.yaml "spec.containers[0].env[0].value" "postgres://prod:5432"
apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  containers:
  - name: test-container
    image: k8s.gcr.io/busybox
    env:
    - name: DB_URL
      value: postgres://prod:5432

$
```

If you notice here, `yq`  printed the result on the standard output. If you prefer to edit the YAML in place, you should add the `-i` flag.

The advantage of using `yq`  over say `sed` (with bash) is that unlike `sed`, `yq`  understands the YAML format, and can navigate and mangle the structured markup.


> Note: `sed` treats files as strings and it doesn't mind if the 
> file isn't a valid YAML.




### 3. Merging YAML files

Let's assume that you want to inject an extra container to all the Pods submitted to the cluster.

But instead of using an [Admission Webhook](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#what-are-admission-webhooks), you decide to add an extra command in your deployment script.

You could save the YAML configuration for the extra container as a YAML file:

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: envoy-pod
spec:
  containers:
  - name: proxy-container
    image: envoyproxy/envoy:v1.12.2
    ports:
      - containerPort: 80
```

Assuming you have a Pod like this:

```yaml
$ apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  containers:
  - name: test-container
    image: k8s.gcr.io/busybox
    env:
    - name: DB_URL
      value: postgres://db_url:5432

$
```

You can execute the following command and merge the two YAMLs:

```bash
$ yq m --append pod.yaml envoy-pod.yaml
```

> Please notice the --append flag that is necessary to append 
> values to an array. You can find more details in [the official 
> documentation](https://mikefarah.github.io/yq/merge).

The output should have a proxy named Envoy as an additional container:

```yaml
$ apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  containers:
  - name: test-container
    image: k8s.gcr.io/busybox
    env:
    - name: DB_URL
      value: postgres://db_url:5432
  - name: proxy-container
    image: envoyproxy/envoy:v1.12.2
    ports:
      - containerPort: 80

```

> Please note that `yq` sorts the YAML fields in the output alphabetically, so the order of fields in your output could be different from the > above listing.


In other words, the two YAML files are merged into one.


### Limitations of using  `yq`?

While `yq`  works great with transforming YAML, it does have a few issues of its own:
1. The two YAML files are merged at the top level. For instance, you cannot add a chunk of YAML file under `.spec.containers[]`.

2. The order of the files matters. If you invert the order, `yq` keeps envoy-pod for the Pod's name in `metadata.name`.

3. You have to tell `yq` explicitly when to [append and overwrite values](https://mikefarah.github.io/yq/merge). Since those are flags that apply to the whole document, it's hard to get the granularity right.


Even with its limitations, `yq`  has a lot of potential for use, especially if you are working on smaller projects.


### Alternative(s)

If you wish to apply more complex transformations – Kustomize is a better option than `yq`. Kustomize uses a YAML file called `kustomization.yaml` to decide how to template the YAML – thus making all the changes traceable.


You can learn more about Kustomize and other interesting alternatives here:  [Templating YAML with real code](https://learnk8s.io/templating-yaml-with-code)