Splitting .circleci/config.yml
Tags: circleci
Published: 2020-11-07 11:25:05

In this post I introduce how to split a huge `.circleci/config.yml`.

CircleCI doesn't support to split .circleci/config.yml, so we manage all workflows and jobs configuration into one file .circleci/config.yml.
If the repository is [Monorepo](https://en.wikipedia.org/wiki/Monorepo), the more the number of services increases, the more the size of .circleci/config.yml becomes large and it's hard to maintain .circleci/config.yml.
By splitting .circleci/config.yml per service, it makes easy to maintain .circleci/config.yml and we can configure split file's [CODEOWNERS](https://docs.github.com/en/free-pro-team@latest/github/creating-cloning-and-archiving-repositories/about-code-owners).

To split .circleci/config.yml, you have to generate .circleci/config.yml by merging split files and commit both split files and .circleci/config.yml.

## circleci config pack

We can merge split files with the command `circleci config pack`, but I introduce the other tool [circleci-config-merge](https://github.com/suzuki-shunsuke/circleci-config-merge).

CircleCI CLI is an official tool so it's reliable, but I feel the restriction of the file name and the directory structure is a little strict.
We have to manage all files on the same directory, and the file path is reflected to generated YAML content.

For the detail of `circleci config pack`, please see the official document.

https://circleci.com/docs/2.0/local-cli/#packing-a-config

If you can accept the restriction of `circleci config pack`, I recommend to use it because it is an official tool.
But if it is difficult to accept the restriction, maybe [circleci-config-merge](https://github.com/suzuki-shunsuke/circleci-config-merge) would help you.

## circleci-config-merge

circleci-config-merge is a CLI tool to generate .circleci/config.yml by merging split files.

The usage of circleci-config-merge is like the following.

```
$ circleci-config-merge merge <file1> [<file2> ...]
```

There is no restriction of file paths, and the format of split file is same as .circleci/config.yml.

For example, you can manage files on the same directory.

```
.circleci/
  config.yml # generated
  src/
    service1.yml # split config per service
    service2.yml
    ...
```

Or you can also manage files on each service directory.

```
service1/
  circleci/
    workflow.yml # you can split file freely
    jobs.yml
  ...
service2/
  circleci/
    config.yml
  ...
...
```

circleci-config-merge merges the list of workflow jobs.

For example,

```yaml
workflows:
  build:
    jobs:
    - foo
```

```yaml
workflows:
  build:
    jobs:
    - bar
```

The workflow `build`'s jobs are merged as the following.

```yaml
workflows:
  build:
    jobs: # sort by job name for comparison
    - bar
    - foo
```

## Test .circleci/config.yml in CI

If you split .circleci/config.yml, you should test in CI whether .circleci/config.yml is generated by merging split files.
circleci-config-merge doesn't provide such a feature, but you can implement the test with the other tool like [dyff](https://github.com/homeport/dyff).

I have created an example repository [suzuki-shunsuke/example-circleci-config-merge](https://github.com/suzuki-shunsuke/example-circleci-config-merge).
You can use this example as a reference to split .circleci/config.yml and setup CI.

## Use case

Lastly, I introduce a use case of circleci-config-merge.
Recently, I split a huge .circleci/config.yml which is over 6,000 lines to about 60 files.
It was hard to maintain the original .circleci/config.yml, but by splitting it became easy to maintain .circleci/config.yml.
If you are suffer from a huge .circleci/config.yml, let's split it!

## Conclusion

In this post I introduced how to split a huge .circleci/config.yml.
We can generate .circleci/config.yml by merging split files with [circleci-config-merge](https://github.com/suzuki-shunsuke/circleci-config-merge).
Please see the example [suzuki-shunsuke/example-circleci-config-merge](https://github.com/suzuki-shunsuke/example-circleci-config-merge) as a reference to split .circleci/config.yml and setup CI.