Two ways to keep gitlab CI files maintainable
Tags: gitlab, betterprogramming, cleancode
Published: 2021-04-12 15:07:54

Once we had a gitlab CI file. It was short and sweet. One year later, it grew 350 lines long. 

There were these problems:

1. Too much content - too much scrolling, hard to visualize and work on.
2. Hard to disable some jobs temporarily - mostly for debugging infra or test environment or emergency deployment (let that never happen!).

Let's try to solve it with some features from gitlab.

### 1. Leverage templating

Gitlab CI supports using templates within the `.gitlab-ci.yml` file. 

Consider a sample `.gitlab-ci.yml` file as follows

```
stages:
   - setup
   - soft-qa //lint and unit tests
   - build
   - hard-qa //e2e's
   - deploy-storybook
   - pack
   - notify-devs-staging-can-be-deployed
   - deploy-staging
   - notify-devs-prod-can-be-deployed
   - deploy-production
   - suggest-release-notes

variables: 
    var1: '1'
    // .. and so on

conditions:
    only_master: 
       // configs
    branches:
       // configs
    // and more...

## cache related configs

## setup related configs

## jobs for each for the stages start
.
.
.
. // 200 lines later
.
## jobs end
```

I will still keep the stages in the `.gitlab-ci.yml` untouched so that I can get a preview of all the stages right at the start of the file.

Let's now split this into smaller templates. 

For templates, I will create a folder in the repo root called `ci-templates`. Now let's extract out one job from the main file and place it in a template in this folder. Note, all these files has to YAML, to be included into a `.gitlab-ci.yml` file.

```
/ci-templates/.soft-qa.yml

soft-qa:
  image: node:14.5
  stage: soft-qa
  <<: *npm_cache_pull
  allow_failure: false
  script:
    - yarn lint
    - yarn test:unit:ci
  artifacts:
    paths:
      - coverage/lcov.info
```

Time to use the template. Go to `.gitlab-ci.yml` and include this file like below. I will place it at the position of the replaced job.

```
include: '/ci-templates/.soft-qa.yml'
```
We are using this syntax - with relative path - as we are using the file from the same repo. You can keep the file in another repo on the same gitlab instance or even in a public remote repository and use it! Just look up the syntax and update accordingly, like below:

For another repo:
```
include:
  - project: 'my-space/my-another-project'
    file: '/templates/.build-template.yml'
```

For remote:
```
include:
  - remote: 'https://somewhere-else.com/example-project/-/raw/master/.build-template.yml'
```

In a similar way, extract out other jobs as well and remove them from the .gitlab ci file. I have abstracted the notify stages to a file called `.notifications.yml` and deployment related jobs to `.deploy.yml`, thus separating the concerns from one single file. Now, include becomes a list like below:

```
include: 
   - '/ci-templates/.soft-qa.yml'
   - '/ci-templates/.build.yml'
   - '/ci-templates/.hard-qa.yml'
   - '/ci-templates/.deploy-storybook.yml'
```
When pipeline starts, all included files are evaluated and deep merged into the .gitlab-ci file. 

### Catch, Catch, Catch

Things are getting interesting. I have certain conditions like only_tag, only_branches in these jobs. How would I provide them to these files without duplicating it in every files?

Enter `extends`. 

I will consolidate my conditions in a file, say, `.conditions.yml`.

```
// /ci-templates/.conditions.yml

.only_tag:
  only:
    - /^v\d+\.\d+\.\d+$/
  // and more...
```

To use it in a template, include it first in the .gitlab-ci file

```
include: 
   - '/ci-templates/.conditions.yml'
   - '/ci-templates/.soft-qa.yml'
   - '/ci-templates/.build.yml'
   - '/ci-templates/.hard-qa.yml'
   - '/ci-templates/.deploy-storybook.yml'
```
Then, in the template file, add extend. Extend is a way of extending the congfigurations to another file. YAML anchors can be used but only from the same file. So this is a way to use jobs from another template.

```
deploy_staging:
  extends: .only_tag
  <<: *deploy
```

Now, deploy_staging will be created only for tags. 

Finally, the .gitlab-ci file will look something like:

```
stages:
   - setup
   - soft-qa //lint and unit tests
   - build
   - hard-qa //e2e's
   - deploy-storybook
   - pack
   - notify-devs-staging-can-be-deployed
   - deploy-staging
   - notify-devs-prod-can-be-deployed
   - deploy-production
   - suggest-release-notes

variables: 
    var1: '1'
    // .. and so on

## cache related configs

## setup related configs

include: 
   - '/ci-templates/.conditions.yml'
   - '/ci-templates/.soft-qa.yml'
   - '/ci-templates/.build.yml'
   - '/ci-templates/.hard-qa.yml'
   - '/ci-templates/.deploy-storybook.yml'
   - '/ci-templates/.pack.yml'
   - '/ci-templates/.notifications.yml'
   - '/ci-templates/.deploy.yml'
   - '/ci-templates/.suggest-release-notes.yml'

```
Now let's go back to the initial problems and check whether they are fixed:

1. Too much content to scroll - now, we can view whole content in the IDE window - Fixed
2. Hard to disable jobs - now, we just have to comment the template in the include list -  Fixed

How will I know, if I commented out a required job? Like, by mistake, I commented the pack job. For deployment packed resources are necessary. Here, we will know that the deployment failed much later when the deployment runs. To solve this, there is a way to mark dependencies on previous jobs. That is explained in next step.

### 2. Use `needs` or `dependencies` as applicable

One way to track that you have the dependent artifacts are available before starting a job is by using `dependencies` or 
`needs` keyword.

By default, all artifacts from previous stages are passed to each job. However, you can use the dependencies keyword to define a limited list of jobs to fetch artifacts from. 

```
build_frontend:
   stage: build
   script: yarn build

deploy:
   stage: deployment
   dependencies: 
      - build_frontend
```

In this case, if the build_frontend is not available, while merging the templates, the pipeline will report error that the `build_frontend` is not available. Easy to understand.

Another utility is the `needs`. It is mainly used when you are running jobs out of order and still ensure that the dependency job is completed before starting a job. The difference of `needs` from `dependencies` is that, in `needs`, no artifacts from previous steps are downloaded by default. You need to use `artifacts: true` config on the job like below to download them:

```
deploy:
  needs:
   - job: build_1
     artifacts: true
   - job: build_2
     artifacts: false
```

This helped me reduce the complexity of the CI file. Hope it helps you too. 

Peace :v: