🦊 GitLab CI YAML Modifications: Tackling the Feedback Loop Problem
Tags: gitlab, devops, pipeline, cicd
Published: 2023-12-18 09:24:52

---
title: "\U0001F98A GitLab CI YAML Modifications: Tackling the Feedback Loop Problem"
tags:
  - gitlab
  - devops
  - pipeline
  - cicd
license: public-domain
cover_image: 'https://raw.githubusercontent.com/bcouetil/articles/main/images/gitlab/garage/garage-1.jpg'
published: true
id: 1564198
date: '2023-12-18T09:24:52Z'
---

<!-- humanoid orange fox as a mechanic repairing an orange car, lora:DieselPunkAI:.8 DieselPunkAI garage

Le problème principal quand on modifie de la CI c'est... le long délai avant d'avoir un retour, appelé boucle de feedback. Nous savons tous à quel point c'est pénible 🤯


Mais il est possible de grandement améliorer ça sur GitLab ! Dans cet article, nous parcourons quelques astuces pour accélérer cette boucle, et donc booster la productivité sur modification de pipeline: https://dev.to/zenika/gitlab-ci-yaml-modifications-tackling-the-feedback-loop-problem-4ib1


Et après plusieurs mois d'utilisation intensive de gitlab-ci-local, la tendance se confirme, le gain est loin d'être négligeable ! au moins 50% de productivité en plus 🤓


Nous présentons cet outil (et bien d'autres choses) dans nos formations GitLab by Zenika: https://training.zenika.com/fr-fr/search?q=gitlab


#gitlab #cicd #devops

-->

<!-- TOC start (generated with https://github.com/derlin/bitdowntoc) -->

- [Initial thoughts](#initial-thoughts)
- [1. Use a local docker image in early stages of job creation](#1-use-a-local-docker-image-in-early-stages-of-job-creation)
- [2. Use the pipeline editor for minor YAML changes](#2-use-the-pipeline-editor-for-minor-yaml-changes)
- [3. Use gitlab-ci-local to run pipelines locally](#3-use-gitlab-ci-local-to-run-pipelines-locally)
  - [Validate pipeline YAML](#validate-pipeline-yaml)
  - [Run a full pipeline](#run-a-full-pipeline)
  - [Run just a job](#run-just-a-job)
  - [Handle UI variables](#handle-ui-variables)
- [4. Declare test branches to simulate long-lived branches and tags](#4-declare-test-branches-to-simulate-long-lived-branches-and-tags)
- [Wrapping up](#wrapping-up)
- [Further reading](#further-reading)

<!-- TOC end -->

# Initial thoughts

As developers and CICD engineers, we are all too familiar with the time-consuming process of iterating on GitLab CI YAML modifications. It can be a frustrating cycle that involves multiple steps and setbacks:

- Make a change to the YAML file.
- Commit and push the changes.
- Verify that the CI YAML is valid and the graph is as expected.
- If there are any issues or errors, start the process all over again with a sense of despair.
- Wait for the entire pipeline to finish running on the current branch.
- If the pipeline fails or doesn't behave as desired, start over again with a heavy heart.
- Refactor commits to maintain a clean git graph and push with force.
- Merge multiple times to observe the behavior on long-lived branches and tags.
- If there are any issues with the merged code, start the process all over again with a sigh.

What we initially thought would only take a few minutes ends up consuming multiple days as we continuously go back and forth, all while trying to juggle other development tasks.

In this article, we aim to alleviate this pain by providing several tips that will make your life easier. Our goal is to transform the experience of tweaking your favorite pipelines from a tedious chore to an enjoyable task. So let's dive in and discover how you can streamline your GitLab CI YAML modifications!

# 1. Use a local docker image in early stages of job creation

When starting a new job with a new Docker image, it's natural to feel uncertain. Especially when using initially unknown public images, following the best practice [Start CI with versioned public CI docker images](https://dev.to/zenika/gitlab-ci-10-best-practices-to-avoid-widespread-anti-patterns-2mb5#1-start-ci-with-versioned-public-ci-docker-images). To accelerate the feedback loop and gain more confidence, we can try our commands directly in a local container.

Let's consider a NodeJS job as an example. The DockerHub image [alpine/k8s](https://hub.docker.com/r/alpine/k8s) seems to be the most suitable base image.

Before diving into the GitLab CI YAML, let's start an interactive container:

```shell
docker run -it --privileged=true node:20.5-alpine3.17 /bin/sh
Status: Downloaded newer image for node:20.5-alpine3.17
/#
```

You may wonder what shell GitLab is using in containers. In fact, it is a [discovery process starting with bash](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/1758#note_16419417), so best guess is to try `/bin/bash` and fallback to `/bin/sh`.

Now that you're inside the image, you can easily prepare and test your commands.

Although the base is `alpine`, let's assume we do not have that knowledge and want to use `envsubst`. If it's not available, we can try to install it with `apt-get update && apt-get install envsubst`:

```shell
/# envsubst --help
/bin/sh: envsubst: not found
/# apt-get update && apt-get install envsubst
/bin/sh: apt-get: not found
```

Since `envsubst` and `apt-get` are not available, it must be the `alpine` Linux package manager, `apk`:

```shell
/# apk --update add --no-cache envsubst
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/main/aarch64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/community/aarch64/APKINDEX.tar.gz
ERROR: unable to select packages:
  envsubst (no such package):
    required by: world[envsubst]
```

Ah! It seems that `envsubst` cannot be installed this way. A quick internet search reveals that the package is actually named `gettext`:

```shell
/apps# apk --update add --no-cache gettext
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/main/aarch64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/community/aarch64/APKINDEX.tar.gz
(1/9) Installing libgomp (12.2.1_git20220924-r4)
(8/9) Installing libxml2 (2.10.4-r0)
(9/9) Installing gettext (0.21.1-r1)
Executing busybox-1.35.0-r29.trigger
OK: 18 MiB in 26 packages
/#
/# envsubst --help
Usage: envsubst [OPTION] [SHELL-FORMAT]

Substitutes the values of environment variables.
```

By going through this discovery process within a few minutes, we saved ourselves from waiting for the pipeline to fail, which could have taken several hours for a job, especially positioned in last stages.

# 2. Use the pipeline editor for minor YAML changes

The GitLab CI/CD Pipeline Editor is a powerful web-based tool that provides a visual representation of your pipeline configuration. Accessible from the GitLab UI, the Pipeline Editor allows you to interactively modify your .gitlab-ci.yml file directly in the browser.

The Pipeline Editor offers several advantages. Before even commiting changes, it provides real-time feedback on the validity of your YAML syntax and helps prevent common errors. Additionally, it assists in exploring available keywords and includes inline documentation for GitLab CI/CD features. This visual approach can significantly streamline the process of making and validating changes to your pipeline configuration, reducing the need for multiple iterations.

The main limitation is that it can only edit the main `.gitlab-ci.yml` file.

![humanoid orange fox as a mechanic repairing an orange car, lora:DieselPunkAI:.8 DieselPunkAI garage](https://raw.githubusercontent.com/bcouetil/articles/main/images/gitlab/garage/garage-2.jpg)

# 3. Use gitlab-ci-local to run pipelines locally

Imagine being able to run your pipelines locally without the need to push, wait, or pollute the GitLab instance. Even better, what if you could achieve this without even interacting with the GitLab instance at all?

Initially, the `gitlab-runner exec` command was intended to fulfill this purpose. However, over time, the code has diverged too much from an actual pipeline running, rendering it capable of handling only 20% of the keywords. As a result, it is no longer usable. There is an official issue on this matter, but progress has been minimal.

Fortunately, the open-source community has made remarkable efforts to implement a local runner, resulting in several options:

{% embed https://github.com/firecow/gitlab-ci-local %}

{% embed https://github.com/mdubourg001/glci %}

{% embed https://gitlab.com/AdrianDC/gitlabci-local %}

{% embed https://gitlab.com/cunity/gitlab-emulator %}

Among these options, the one that has gained the most traction is [gitlab-ci-local](https://github.com/firecow/gitlab-ci-local):

![https://codetabs.com/github-stars/github-star-history.html](https://raw.githubusercontent.com/bcouetil/articles/main/images/gitlab/cicd/gcl-stars.jpg)

Here are some key features of gitlab-ci-local:

- The only prerequisite is having functional Docker commands.
- You can trigger the execution of the entire pipeline, a single job, or the fastest path to a job using the `needs:` chain.
- No need to push changes or establish a connection to the server. Committing is also not required.
- It supports 99% of the keywords used in GitLab CI pipelines and continuously closes the gap. This means that even complex real-life pipelines with artifacts and lightning-fast cache can be executed seamlessly.
- YAML from includes are seamlessly downloaded
- GitLab CI predefined variables are populated with valid values.
- Global, group, and project variables can be managed using files.
- It supports running unlimited jobs in parallel.

## Validate pipeline YAML

To validate your pipeline, simple try to display the jobs list:

```shell
$ gitlab-ci-local --list

parsing and downloads finished in 178 ms
name                                   description  stage              when        allow_failure  needs
📦✅-webapp-package-and-test                        📦 package ✅ test  on_success  false
📦✅-backend-package-and-test                       📦 package ✅ test  on_success  false
🐳🧪-webapp-build-push-dev                          🐳 build-push      on_success  false
🐳🧪-backend-build-push-dev                         🐳 build-push      on_success  false
🗑✨-delete-namespace-mr                            ☸ deploy           manual      true
☸🧪-webapp-deploy-dev                               ☸ deploy           on_success  false
☸🧪-backend-deploy-dev                              ☸ deploy           on_success  false
✔-e2e-tests                                         ✔ e2e tests       on_success  false
```

It helps test your [rules](https://docs.gitlab.com/ee/ci/yaml/#rules). For this, you can also force some predefined variables, to test alternative branching situations.

## Run a full pipeline

By default gitlab-ci-local will run the full pipeline for the current branch, or for the current merge request, depending of the predefined variables you overrode. It will trigger and run jobs in parallel, as a GitLab runner would do:

```shell
$ gitlab-ci-local
```

![gitlab-ci-local example](https://raw.githubusercontent.com/firecow/gitlab-ci-local/master/docs/images/example.png)

## Run just a job

We can also run a single job:

```shell
$ gitlab-ci-local 📦✅-webapp-package-and-test
```

Or a job and its dependencies before it (`needs:` chain and previous stages):

```shell
$ gitlab-ci-local 🐳🧪-webapp-build-push-dev --needs
📦✅-webapp-package-and-test           starting node:18.11-alpine (📦 package ✅ test)
📦✅-webapp-package-and-test           copied to docker volumes in 7.72 s
📦✅-webapp-package-and-test           mounting cache for path projects/webapp/node_modules
📦✅-webapp-package-and-test           mounting cache for path projects/webapp/.next/cache
📦✅-webapp-package-and-test           $ cd projects/$MODULE
📦✅-webapp-package-and-test           $ if [ ! -d node_modules/.bin ]; then yarn install; fi
[...]
```

## Handle UI variables

Global, group and project variables can be handled in `$HOME/.gitlab-ci-local/variables.yml`, or locally in each project.

Official example:

_$HOME/.gitlab-ci-local/variables.yml_

```yaml
project:
  gitlab.com/test-group/test-project.git:
    # Will be type Variable and only available if remote is exact match
    AUTHORIZATION_PASSWORD: djwqiod910321
  gitlab.com:project/test-group/test-project.git: # another syntax
    AUTHORIZATION_PASSWORD: djwqiod910321

group:
  gitlab.com/test-group/:
    # Will be type Variable and only available for remotes that include group named 'test-group'
    DOCKER_LOGIN_PASSWORD: dij3213n123n12in3

global:
  # Will be type File, because value is a file path
  KNOWN_HOSTS: "~/.ssh/known_hosts"
  DEPLOY_ENV_SPECIFIC:
    type: variable # Optional and defaults to variable
    values:
      "*production*": "Im production only value"
      "staging": "Im staging only value"
  FILE_CONTENT_IN_VALUES:
    type: file
    values:
      "*": |
        Im staging only value
        I'm great for certs n' stuff
```

![humanoid orange fox as a mechanic repairing an orange car, lora:DieselPunkAI:.8 DieselPunkAI garage](https://raw.githubusercontent.com/bcouetil/articles/main/images/gitlab/garage/garage-3.jpg)

# 4. Declare test branches to simulate long-lived branches and tags

Now that we have the tools to test our pipeline changes with confidence before pushing them, there is still one tricky aspect to consider: how will our pipeline behave on long-lived branches after the merge request is closed?

While we can use `gitlab-ci-local` locally to force certain variables, such as setting `$CI_COMMIT_BRANCH` to `main` in `$HOME/.gitlab-ci-local/variables.yml`, this approach is only applicable locally and does not serve as valid proof for people reviewing our merge request.

A satisfying solution is to always declare special branch/tag cases and include a corresponding test version. This means that our jobs will run on both `main` and `main-test`, on tags and `tags-test` branch, for example:

```yaml
deploy-module-staging:
  extends:
    - .staging
    - .deploy-module
  rules:
    - if: $CI_COMMIT_TAG || $CI_COMMIT_BRANCH == "tag-test"
```

By doing this, we can test all branches conveniently from our current merge request without actually merging anything. We can create these test branches from our latest commit in the GitLab UI. Once we are satisfied with the pipeline execution (or just the composition of pipeline jobs for sensitive tasks), we can stop the associated pipeline and delete the test branch. If we are not satisfied, we can simply add more commits to our merge request and recreate (or rebase) our test branch(es). When everything seems fine in all scenarios, we can confidently proceed with merging our merge request, knowing that it won't cause any unexpected issues.

# Wrapping up

By following these practices and leveraging the available tools, you can significantly reduce the time spent on iterating and testing GitLab CI pipelines. This will lead to a more efficient and enjoyable pipeline development process, ultimately improving your overall development workflow.

So go ahead and implement these tips in your GitLab CI pipeline development, and experience the benefits of faster iterations and more reliable pipelines. Happy coding!

Did this pieces of advice help you ? Do you have others on the same subject ? Please share in the comments below 🤓.

![humanoid orange fox as a mechanic repairing an orange car, lora:DieselPunkAI:.8 DieselPunkAI garage](https://raw.githubusercontent.com/bcouetil/articles/main/images/gitlab/garage/garage-4.jpg)

_Illustrations generated locally by Automatic1111 using ZavyComics model with DieselPunkAI LoRA_

# Further reading

{% embed https://dev.to/zenika/gitlab-a-python-script-displaying-latest-pipelines-in-groups-projects-5b5a %}

{% embed https://dev.to/zenika/gitlab-a-python-script-calculating-dora-metrics-258o %}

{% embed https://dev.to/zenika/gitlab-ci-the-majestic-single-server-runner-1b5b %}

{% embed https://dev.to/zenika/gitlab-ci-optimization-15-tips-for-faster-pipelines-55al %}

{% embed https://dev.to/zenika/gitlab-ci-10-best-practices-to-avoid-widespread-anti-patterns-2mb5 %}

{% embed https://dev.to/zenika/gitlab-pages-preview-the-no-compromise-hack-to-serve-per-branch-pages-5599 %}

{% embed https://dev.to/zenika/chatgpt-if-you-please-make-me-a-gitlab-jobs-attributes-sorter-3co3 %}

{% embed https://dev.to/zenika/gitlab-runners-topologies-pros-and-cons-2pb1 %}

_This article was enhanced with the assistance of an AI language model to ensure clarity and accuracy in the content, as English is not my native language._