Creating templates for Gitlab CI Jobs
Tags: devops, beginners, architecture
Published: 2019-10-10 14:15:34

#### Writing Gitlab CI templates becomes repetitive when you have similar applications running the same jobs. If a change to a job is needed it will be most likely needed to do the same change in every repository. On top of this there is a vast amount of excess YAML code which is a headache to maintain. Gitlab CI has many built-in templating features that helps bypass these issues and in addition helps `automating` the process of setting up CI for various applications.

## Setting up a base CI template

As in most other coding languages you start projects with a base template. We'll create one which is very basic, and you can tailor it to your CI. The base template will be having the following variables:

- image
- .default_vars
- .default_delay
- stages
- services

We use `.default_vars` to store any default variables that ALL jobs will share. You can store `.default_delay` within these variables but most likely you'd prefer that all jobs don't have a delay. I use a delay to not set up a new AWS auto scaling instance for sequential jobs (as one jobs finish and the next start, it usually creates two instances, I delay the second job so it can reuse the same instance). The `.default_vars` contain a default tag for CI jobs.

We set up a default image, default stages and the default `docker-in-docker service` as most jobs use those variables.

I include a webhook CI template for default webhooks which runs with the CI function `after_script`. You can find the template in my blog post which goes into detail about [creating group webhooks for your CI](https://hodovi.cc/blog/creating-group-webhooks-gitlab-ci-without-a-paid-plan/). I reference all my includes by `project` as I find it the most clean solution but you have other options as [local, template, remote](https://docs.gitlab.com/ee/ci/yaml/index.html#include-examples).

`gitlab-ci-base.yml`

```yaml
image: docker:stable

services:
  - docker:dind

include:
  - project: 'honeylogic/gitlab-ci-templates'
    ref: master
    file: 'gitlab-ci-webhooks.yml'

stages:
  - build
  - test
  - deploy

# Default runner tag
.default_vars:
  tags:
    - shared_aws_spot_runner

# Used to not start a new instance
.default_delay:
  when: delayed
  start_in: 20s
```

Now you can include the default template and extend the default variables. Let's take a look at a default build stage. It'll extend the `.default_vars` but not the `.default_delay` as it is the first stage of the build. I use Google's Kaniko as it can use cache, build and push images with tags just by one command instead of docker build, docker push etc... We add all template commands in the `before_script` section as currently you cannot merge arrays and if you would like to append commands to the `script` you will overwrite the extended `script` completely. If your certain that you do not need the option to append commands you can add all commands to a `script` section.

`.gitlab-ci-kaniko-build.yml`

```yaml
build:
  extends:
    - .default_vars
  stage: build
  tags:
    - shared_aws_spot_runner
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  before_script:
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
    - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG --destination $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA --cache=true
```

We don't include the base templates for each stage e.g now in the build stage as you cannot have multiple of the same include variables, the YAML syntax will fail. The `.gitlab-ci` YAML file will include the base templates and the variables will be available for every stage. We have to add a empty script stage as the YAML syntax lint will fail otherwise and your jobs won't run(if anyone has a idea how to avoid this LMK).

```yaml
include:
  - project: 'honeylogic/gitlab-ci-templates'
    ref: master
    file: 'gitlab-ci-base.yml'
  - project: 'honeylogic/gitlab-ci-templates'
    ref: master
    file: 'gitlab-ci-kaniko-build.yml'

build:
  script:
    - ""
```

## Using include, extend and customizing the stage

Here is another example where I use the base template and extend it. My base deploy stage contains two stages `deploy` and `deploy_manual`. The `deploy_manual` stage uses YAML syntax to extend the `deploy stage`. As the `deploy` stage extends the `.default_delay` I have to overwrite it when deploying manually as a manual `when` can't have a `start_in` variable.

```yaml
deploy: &deploy
  extends:
    - .default_vars
    - .default_delay
    - .default_webhooks
  stage: deploy
  only:
    refs:
      - master
  image: adinhodovic/ansible-k8s-terraform
  variables:
    ANSIBLE_CONFIG: ./ansible.cfg  # https://github.com/ansible/ansible/issues/42388
  before_script:
    - echo "failed to deploy" > .job_status
    ........ many commands
    - ansible-playbook requirements.yml # Ansible requirements

deploy_manual:
  <<: *deploy
  only:
  when: manual # make the stage manual
  start_in: # overwrite the start_in
  stage: test # part of the test section
```

Now you can include the deploy template in your `.gitlab-ci` YAML file. We'll add the script section that makes this application's deployment unique which is a ansible playbook for this repository. We'll create a template called `.deployment_script_template` which defines the YAML variable `&deployment_script_definition`. Here we will simply add the repository specific deployment stages and reuse them in the `deploy` and `deploy_manual` stages. Now your deploy stage will have all the base commands in the `before_script`, the repository relevant commands in the `script` and the webhooks in the `after_script`. You can now use this repetition of code for any project you create.

```yaml
include:
  - project: 'honeylogic/gitlab-ci-templates'
    ref: master
    file: 'gitlab-ci-base.yml'
  - project: 'honeylogic/gitlab-ci-templates'
    ref: master
    file: 'gitlab-ci-kaniko-build.yml'
  - project: 'honeylogic/gitlab-ci-templates'
    ref: master
    file: 'gitlab-ci-k8s-ansible-deploy.yml'

build:
  script:
    - ""

test:
  script:
    - ""

.deployment_script_template: &deployment_script_definition
  script:
    - ansible-playbook hodovi_cc.yml -i environments/base -i environments/prod -e app_image=$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
    - echo "deployed" > .job_status

deploy:
  <<: *deployment_script_definition

deploy_manual:
  <<: *deployment_script_definition
```

## Conclusion

As CI jobs get more complex and an organization has more micro-services to run CI for, there becomes a need to set up reusable templates for all CI jobs. This minimizes maintenance and makes it easier to set up new pipelines for new projects. Gitlab CI offers great functionality that makes this easy to set up and keep your CI modular just as do with any other code you write.