How to set up monorepo build in GitLab CI
Tags: gitlab, monorepo
Published: 2021-07-25 17:09:13

If you migrate your multirepo to a monorepo, or if your project is getting big enough to consider running only part of continuous integration (CI) - then it can make sense to run only those parts of CI that could have been affected by the change. This article will show how to achieve it on the GitLab platform, using a simple repository as an example.

# The approach
The project is split into folders. For distinguishing if a given part of the project was modified, we use [rules:changes](https://docs.gitlab.com/ee/ci/yaml/#ruleschanges) so every part of the project that we want to be able to run in separation from the rest should be placed in one folder.

My assumptions are as follow:
* you want to run only changed sections of CI in the merge requests (MR) - mainly to save CI resources. So we can avoid running whole 30 minutes of jobs for a change we know is not likely to affect them. This is especially important if we have code in the repo that is not depending on each other - for example, our main application in one place and some landing pages in other folders.
* after changes are merged to master/main, we want to build everything no matter if it was changed or not. In this way, our main branch is indeed continuously integrated, and we keep on checking on even less commonly changed parts of the project.

# Configuration
My project structure is simple:
```sh
$ git ls-files
.gitlab-ci.yml
README.md
backend/README.md
frontend/README.md
```

I have 2 folders, `backend` & `frontend`. Each would host files of a given part of our project. This approach scales for any number of sub-projects - we could have `company-website`, `slack-bot`, or whatnot inside.

## `.gitlab-ci.yml` step by step
The configuration starts with defining the stages:

```yml
stages:
  - build
  - test
  - deploy
```

This one is copied from GitLab's starting CI template. We can customize it with adding or removing stages. As we define `needs:`, there is no speed penalty for adding more stages - each job is executed as soon as its requirements are defined in `needs:` are met. For example, in my project, I ended up adding `pre-build` to run some preparation scripts before building docker images in my project.

```yml
variables:
  RULES_CHANGES_PATH: "**/*"
```

The default value for our changes configuration - by default, the job that extends our base config will be executed for any changes.

```yml
.base-rules:
  rules:
    ...
```

Our base config. I define it in a way that requires us to add it with `extends: .base-rule` - probably we could define those rules on the top level, but it's something a headache to configure everything in a way that works as expected in every case. I found it easier to have control over if the `.base-rules` are set or not.

```yml
.base-rules:
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: always
  ...
```
The `rules` are checked in order. Our 1 rule - if it's master/main, CI should always run the job.

```yml
    - if: '$CI_PIPELINE_SOURCE == "push"'
      when: never
```

Here, we avoid duplicated jobs for merge requests. Without, GitLab would create 1 pipeline for the branch and a "detached pipeline" for the MR. As the branch pipeline doesn't seem to support `changes:`, we disable branch one & delay starting CI until an MR is created.

```yml
    - if: $CI_COMMIT_TAG
      when: never
```

Similarly, we don't need CI running for a tag.

```yml
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
      changes:
        - $RULES_CHANGES_PATH
```

In MRs, we start jobs for when there are changes in the path defined in the `RULES_CHANGES_PATH` variable.

```yml
    - when: manual
      allow_failure: true
```

Otherwise, we all the job to be triggered manually.

### `.frontend` & `.backend`

Now, we define 2 more jobs to be extended from:
```yml
.backend:
  extends: .base-rules
  variables:
    RULES_CHANGES_PATH: "backend/**/*"

.frontend:
  extends: .base-rules
  variables:
    RULES_CHANGES_PATH: "frontend/**/*"
```

In this way, we avoid duplicating the same path definition in each job we define for one or the other part - a possible source of errors.

### Example jobs

On top of that all, we can define our jobs as:

```yml
backend-build:
  stage: build
  extends: .backend
  needs: []
  script:
    - echo "Compiling the backend code..."

frontend-build:
  stage: build
  extends: .frontend
  needs: []
  script:
    - echo "Compiling the frontend code..."
```

## Complete `.gitlab-ci.yml`

So, in the end, the complete config files are like this:
```yml
stages:
  - build
  - test
  - deploy

variables:
  RULES_CHANGES_PATH: "**/*"

.base-rules:
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: always
    - if: '$CI_PIPELINE_SOURCE == "push"'
      when: never
    - if: $CI_COMMIT_TAG
      when: never
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
      changes:
        - $RULES_CHANGES_PATH
    - when: manual
      allow_failure: true

.backend:
  extends: .base-rules
  variables:
    RULES_CHANGES_PATH: "backend/**/*"

.frontend:
  extends: .base-rules
  variables:
    RULES_CHANGES_PATH: "frontend/**/*"

backend-build:
  stage: build
  extends: .backend
  needs: []
  script:
    - echo "Compiling the backend code..."

frontend-build:
  stage: build
  extends: .frontend
  needs: []
  script:
    - echo "Compiling the frontend code..."

backend-test:
  stage: test
  extends: .backend
  needs: ["backend-build"]
  script:
    - echo "Testing the backend code..."

frontend-test:
  stage: test
  extends: .frontend
  needs: ["frontend-build"]
  script:
    - echo "Testing the frontend code..."

backend-deploy:
  stage: deploy
  extends: .backend
  needs: ["backend-test"]
  script:
    - echo "Deploying the backend code..."

frontend-deploy:
  stage: deploy
  extends: .frontend
  needs: ["frontend-test"]
  script:
    - echo "Deploying the frontend code..."
```

# Working CI
With a setup like this, you can have your backend CI run for backend MR:

![backend-mr.png](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ablggef3pg962o9kjy09.png)

frontend, for MR with frontend changes:

![frontend-mr.png](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/i9bcq7jppbqkabnlw5uu.png)

and each commit will trigger all CI jobs once it's merged to the main branch:

![merge-backend-2.png](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8gloi3h74am4p8hxdo2e.png)

# Refrences
You can find the repo I used to write this article [here](https://gitlab.com/marcin-wosinek/monorepo-ci-example/).

# Summary
In this article, we have seen how to set up partially split CI for branches in GitLab. Please leave a comment if you find this article helpful or have some questions about this approach.