An excellent way to deal with (lerna) monorepos in CircleCI: circletron
Tags: circleci, monorepo, circletron, lerna
Published: 2021-06-25 11:25:23

We started with a monorepo at my current company and have been using circle almost since the beginning. It was tough. It required a lot of boilerplate in our config and the necessity to get every developer to generate a circle API key and add it to git. It never felt that great but at least it worked. In April CircleCI released the [dynamic configuration API](https://circleci.com/docs/2.0/dynamic-config/) and this allowed us to refactor our monorepo support into something we think is pretty great. We've gone from tolerating Circle to enjoying it, and now we've released the code as an open source project and provided the functionality as an [orb](https://circleci.com/orbs/) so that any project can benefit.

The project is circletron, the code is hosted [on github](https://github.com/circletron/circletron) and the [orb is here](https://circleci.com/developer/orbs/orb/circletron/circletron). It currently supports lerna monorepos but we plan to support other monorepos in the future.

## Motivating example

Typically a circleci configuration exists in a single file within a repository at `.circleci/config.yml`. For a monorepo a very minimal example may look something like this:

```yaml
version: 2.1

jobs:
  validate-everything:
    steps:
      - checkout
      - run: npm run general-validation

  test-subpackage-a:
    steps:
      - checkout
      - run: cd packages/package-a && npm run test

  test-subpackage-b:
    steps:
      - checkout
      - run: cd packages/package-b && npm run test

  test-subpackage-c:
    steps:
      - checkout
      - run: cd packages/package-c && npm run test

  publish-subpackage-c:
    steps:
      - checkout
      - run: cd packages/package-c && npm run publish
  

workflows:
  validate-everything:
    jobs:
      - validate-everything

  subpackage-a:
    jobs:
      - test-subpackage-a

  subpackage-b:
    jobs:
      - test-subpackage-b

  subpackage-c:
    jobs:
      - test-subpackage-c
      - publish-subpackage-c
```

Obviously there are many problems with this, for a start all the CI is defined in one file at the root of the project. Worse, each of the jobs may take some time to complete and on every commit to the project, circleci will run every single job no matter whether there are any changes to the respective subpackages or not. It may not look so bad in this tiny example but the bigger the configuration gets, the worse it is to deal with.

## Using circletron to solve this

With `circletron` the `.circle/config.yml` is always the same:

```yaml
version: 2.1
setup: true
orbs:
  circletron: circletron/circletron@3.0.1

workflows:
  trigger-jobs:
    jobs:
      - circletron/trigger-jobs
```

The trigger job step will take many individual `circle.yml` distributed within the project and combine them into a single configuration which will be issued via the continuation API. It will modify the configuration to ensure that jobs that are not necessary are no longer run, in a way that is friendly to CI branch protection rules.

The single configuration file can now be split up across the monorepo, with one optional `circle.yml` in the root of the project and where the CI for each subpackage lives in the directory for that subpackage.

In this instance the `circle.yml` in the root of the project will host configuration not specific to any subpackage:
```yaml
version: 2.1

jobs:
  validate-everything:
    steps:
      - checkout
      - run: npm run general-validation

workflows:
  validate-everything:
    jobs:
      - validate-everything
```
The jobs here are run on every commit and it's also a good place to set `version`. It's also a good place to provide `commands` that can be used in subpackage specific `circle.yml` files.

Now lets look at `packages/package-a/circle.yml`:
```yaml
jobs:
  test-subpackage-a:
    steps:
      - checkout
      - run: cd packages/package-a && npm run test

workflows:
  subpackage-a:
    jobs:
      - test-subpackage-a
```

Everything related to this package all in one place. Even better, when the PR contains no changes within `packages/package-a` the jobs for this subpackage will be skipped. You will probably want to use branch protection rules to ensure that when the `test-subpackage-a` job fails the PR will be blocked so omitting this job entirely would not be ideal. Omitted jobs remain in `pending` state permanently, blocking the PR. circletron replaces unneeded jobs with a simple job using the `busybox:stable` docker image that echoes `"Job not required"` and issues a success error code.

## Dependencies and unconditional jobs

What if the code in `packages/subpackage-c` uses code from `packages/subpackage-a`? In this case the jobs within this package should run when the code in `subpackage-a` changes, even if the code in the subpackage itself doesn't change.

There may also be some instances where a job should run on every push.

The configuration at `packages/subpackage-c/circle.yml` shows how to add dependencies and create jobs which run unconditionally:
```yaml
dependencies:
  - subpackage-a

jobs:
  test-subpackage-c:
    steps:
      - checkout
      - run: cd packages/package-b && npm run test

  publish-subpackage-c:
    conditional: false
    steps:
      - checkout
      - run: cd packages/package-c && npm run publish

workflows:
  subpackage-c:
    jobs:
      - test-subpackage-c
      - publish-subpackage-a
```
In this instance `test-subpackage-c` will only be run when changes to `package-a` or `package-c` are detected and `publish-subpackage-c` will run on every push.

## Target branches

CircleCI does not pass the target branch to workflows or jobs so it becomes necessary to help circletron out. By default circletron considers the branches `main`, `master`, `develop` and any branch starting with `release/` as a target branch. The latest commit from the branch commit history which belongs to one of these branches is considered to be the branch-point. For pushes to one of the branches above, all of the jobs are run. This can be changed via the configuration file at `.circleci/circletron.yml`:
```yaml
targetBranches: ^(release/|main$|master$|develop$)
```
Any branch which matches this regex is considered a target branch, the above configuration shows the default.

circletron is available to use now.