GitLab CI: Creating your own pipeline template library 👷
Tags: gitlab, devops, pipeline, cicd
Published: 2020-05-19 13:00:42

---
title: GitLab CI: Creating your own pipeline template library 👷
published: true
description:
tags: gitlab, devops, pipeline, cicd
cover_image: https://dev-to-uploads.s3.amazonaws.com/i/ploe1y7bsvt84ewepg9n.jpg
canonical_url: https://medium.com/@anhtm/a-comprehensive-guide-to-creating-your-own-gitlab-ci-template-library-36a1a64f787a
---

As developers, we all know that it is a good idea to reuse code as much as possible. We all know the DRY mantra - *Don't Repeat Yourself*. Functions, classes or web components abstract away logic, parameterize data, allow code to be reusable, maintainable and extendable. 

With [GitLab CI/CD](https://docs.gitlab.com/ee/ci/), pipelines are defined in [YAML](https://yaml.org/). It's a *human-readable data- serialization language* ([Source](https://en.wikipedia.org/wiki/YAML)), so its focus is clear and concise data delivery rather than efficiency optimization. It's often not obvious how we can go about reusing code in YAML.

Today, we're going to learn how to create  **CI template library** - a library comprised of reusable job templates that can be shared, extended, and overridden by multiple projects 👌

But first, why should I create a CI template library?
--- 
If you're a developer working on a side project on GitLab, or if your team is relatively small and your projects are diverse in programming languages and build processes, then it's probably good enough to stick to a single `.gitlab-ci.yml` file.

On the other hand, as a company grows, it often has standardized testing, building, and deploying processes that are applied to most internal projects. A CI template library increases **time efficiency** in pipeline development and decreases **update and maintenance effort** across the ecosystem.

To build a template library, let's first dive into the basic component of it - the job template ⬇️

Job template
---

A `job` is a basic building block of a pipeline. It usually has a single purpose, executed in isolation and, most of the time, independent of other jobs. 

For example, let's say we have an awesome Node.js app called `awesome-node-app` and we need a job to install dependencies before building it. The `install` job would look something like this:

```yaml
# awesome-node-app/.gitlab-ci.yml

install:
  cache: 
    path:
      - node_modules/ # cache node_modules/ in subsequent pipeline
  script:
    # install dependencies in CI mode
    - npm ci 
```

Now, a **job template** is essentially a job, but it has the following extra properties:

- **Generic**: It is project-agnostic which means it does not contain any data that pertains to a specific project.
- **Importable**: It is easily imported and used directly in a project.
- **Customizable**: It can be extended or overridden.

If we examine the `install` job above once again, it seems like we could, and should, transform it into a template. We might want to reuse it in another JS application in the future! 

It's already *generic* enough, but not quite *importable* or *customizable*. Let's change that.

Making a template
---

### Create and include a template

To create a template, all we need to do is move the job to a new file, `install.yml`:

```yaml
# awesome-node-app/install.yml

install:
  script:
    - npm ci 
```

Then, we need to figure a way to "import" and use this job in `.gitlab-ci.yml` file. Luckily, GitLab has a pretty sweet keyword [`include`](https://docs.gitlab.com/ee/ci/yaml/#include) that allows us to do exactly that!

> 💡 `include` allows us to include and use content declared in an external `yml` or `yaml` file - either locally or remotely. 

We already created `install.yml` locally, so let's `include` it at the top of our `.gitlab-ci.yml`, like so:

```yaml
# awesome-node-app/.gitlab-ci.yml

include:
  - local: 'install.yml' # path to `install.yml`
```

The pipeline now has the job named `install` that does the same thing:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/lvzib5va711sv515xr8m.png)

Using local file here probably won't make sense since we might want to reuse the template in another project and we do *not* want to just copy and paste the job definition. Remember, keep it DRY. 

So let's go ahead and make a new template library that only stores templates! We can then refer to this library whenever we need to `install`.

Create a new repository `ci-templates` that's within the same group as `awesome-node-app`. Then, add `install.yml` at the root of the project. By now, you would have this:

```
ci-templates/
  | install.yml
```

Then, commit and push to  `master`. Cool! Now your library is up and accessible to other repositories.

Let's go back to `awesome-node-app/.gitlab-ci.yml`, and `include`  `install.yml` again, this time using [`include:file`](https://docs.gitlab.com/ee/ci/yaml/#includefile) directive:

```yaml
# .gitlab-ci.yml

include:
 # group name is your username if the project is under personal account
  - project: '<my-group>/ci-templates'  
    ref: 'master'
    file: 'install.yml'
```

> 💡We can modify `ref` to point to any other branch, commit SHA, or version tag of the file in Git history as we'd like. It's good practice to keep track of version history for your template file.

Great! We just basically told GitLab to "include this file `install.yml` from `ci-templates` repo on `master` branch into the pipeline". We now have `install` job imported from `ci-templates` to our `awesome-node-app` 💪

We can run the pipeline as-is - `install` is activated automatically without any further configuration. 

However, what if we do want to change or add configuration?

### Customize a template

Scenario: Right now, `install` only looks for dependencies declared in the `package.json` file at the root of a project. What if we have `another-awesome-node-app` that is a monorepo, and we want to run `install` multiple times in various locations? 

```
another-awesome-node-app/
  | project_one/
  |__ package.json
  | project_two/
  |__ package.json
```

We need to parameterize our `install` template to take in some sort of data that holds information about the **location** of the `package.json` file we're looking for. 

The most powerful way to parameterize a template is by using [**environment variables**](https://docs.gitlab.com/ee/ci/variables/README.html).

Environment variables come in two flavours:

- **Predefined environment variables**: Variables provided by GitLab out of the box and ready to use without any specification. 

> 💡They are references to branch names, merge request IDs, jobs info, and [much, much more](https://docs.gitlab.com/ee/ci/variables/predefined_variables.html).

Predefined environment variables are incredibly powerful. We can do things like conditionally skipping a job in a pipeline, allowing jobs to run on certain branches, leveraging custom variables, and so on. 

This topic deserves a separate article of its own, so if you're interested in knowing more about their use cases and real-world implementation, let me know in the comments below 💚
	
- **Custom environment variables**: Variables defined in `.gitlab-ci.yml` (you can also define them in GitLab UI and via the API).

> ⚠️ Make sure to avoid name collision with predefined variables when naming your variable.

Custom environment variables work in great harmony with job templates. The syntax is as follow:

```yaml
template:
  variables:
    # declare a key/value pair
    MY_VARIABLE: 'hello'
    ... # declare as many variables as you want 
  script:
    # call its value, this outputs "hello" in the runner
    - echo $MY_VARIABLE
```

Notice that the variable is declared within the job scope. This means that the variable is only accessible within the job and inaccessible from pipeline level.

Going back to our example, let's create a new custom `variable` named `INSTALL_DIRECTORY` and call it in our `install` script:

```yaml
# ci-templates/install.yml

install:
  variables:
    INSTALL_DIRECTORY: '.' # default to root directory
  cache:
    path:
      - $INSTALL_DIRECTORY/node_modules/
  script:
    # cd to the directory of package.json
    - cd $INSTALL_DIRECTORY
    # install dependencies in CI mode
    - npm ci 
```

One more thing before we move on, let's make the job **hidden** by default by changing the job name from `install` to `.install`. I'll explain how this works in just a bit. 

```yaml
# ci-templates/install.yml

.install:
  ... 
``` 

Cool! Now we're ready to use this template in  `another-awesome-node-app`. Let's include the template again:

```yaml
# another-awesome-node-app/.gitlab-ci.yml

include:
  - project: '<my-group>/ci-templates'
    ref: 'master'
    file: 'install.yml'
```

We've just included `.install`, but this time, it's [**hidden**](https://docs.gitlab.com/ee/ci/yaml/#hide-jobs), which means it's disabled by default. If you try running this pipeline in GitLab, it will not run simply because the pipeline is **empty** - there is no job!

So how do we use our template then?

Turns out, we can create a new job that [**`extends`**](https://docs.gitlab.com/ee/ci/yaml/#extends) our template to inherit its configuration. 

Let's create two jobs `install_project_one` and `install_project_two` that extend `.install`. After that, we also need to change the default value of `INSTALL_DIRECTORY` in each job to the expected path:

```yaml
# another-awesome-node-app/.gitlab-ci.yml

include:
  ...

install_project_one:
  extends: .install
  variables:
    INSTALL_DIRECTORY: 'project_one/'

install_project_two:
  extends: .install
  variables:
    INSTALL_DIRECTORY: 'project_two/'
```

Awesome! Now both `install_project_one` and `install_project_two` inherit the script from `.install`, but they find the `package.json` file in two different locations just like we wanted!

> ⚠️ Had we not specified `install` a **hidden** job earlier, we would have had one extra `install` job declared in our pipeline that runs in the root directory - where there is no `package.json`. This will fail the pipeline.

📝 To prevent side effects from including external jobs, it's good practice to declare all template jobs **hidden** and `extend` them when needed.

Going beyond templates
---

### Mixins 

We can also keep any other reusable snippets of configuration in the template library. I'd like to call them `mixins`.

Some mixin examples:
 
- Bash scripts
- Pipeline configuration partials

Here's one good example of what I meant by *pipeline configuration partial*:

```yaml
.auth_gitlab_registry:
  services:
    - docker:dind
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
  after_script:
    - docker logout $CI_REGISTRY
```

This mixin logs the user into GitLab Registry `before_script` and logs them out `after_script`. However, it does not have `script` declared, which means that it cannot be run as a `job`. 

> 💡`script` is required for job definition.

To use the mixin, all we need to do is `extend` it the same way we do with regular templates and, most importantly, define what's in `script` in the new job:

```yaml

include:
  ...

build:
  extends: .auth_gitlab_registry
  script:
    - docker build $MY_APP_IMAGE
    - docker push $MY_APP_IMAGE 
```

Summary
---

Here's what we've learned today:

- Properties of a job template: **generic**, **importable** and **customizable**
- `Include` and `extend` a template
- Use environment variables to parameterize templates
- Hide a job to prevent side effects
- Use mixins to further simplify the pipeline

I'm sure there are many other techniques in building a CI template not listed here. Please let me know in the comments what you think of my approach and any suggestions/recommendations for further optimization!

I hope you enjoyed this article 💚


*Cover photo by Pankaj Patel on Unsplash*