Meet cici-tools, a multi-tool for building GitLab CI/CD pipelines
Tags: cicd, gitlab, pipelines, yaml
Published: 2023-07-07 00:44:01

---
title: Meet cici-tools, a multi-tool for building GitLab CI/CD pipelines
published: true
date: 2023-07-03 00:00:00 UTC
tags: cicd, gitlab, pipelines, yaml
cover_image: https://assets.brettops.io/blog/meet-cici-tools/cover-1000x420.png
canonical_url: https://brettops.io/blog/meet-cici-tools/
---

I've been working on a new project called [`cici-tools`](https://gitlab.com/brettops/tools/cici-tools) (pronounced "see-see"). It provides a set of command line tools for working with GitLab CI/CD files, where each tool does something useful in its own right. The direction of the project has changed quite a bit in trying to understand what is most needed and what can be reasonably built, but it's gotten to a good enough place to start talking about it.

This project is still **experimental** and the documentation is a work in progress. I can't promise that it works very well at the moment and would forgive you for not wanting to try it, but for the enterprising among you, I would love your feedback and to know if you found it useful.

## Installation

`cici-tools` is available on [PyPI](https://pypi.org/project/cici-tools/), so you can install it with `pip`:

```bash
python3 -m pip install cici-tools
```

This will install the `cici` command into your local environment, which you can validate like so:

```bash
cici --version
```

```console
$ cici --version
cici 0.2.5
```

## Format CI files with `cici fmt`

The `cici fmt` tool mostly happened by accident while developing the `cici` tool. While it hasn't always been clear what I am building, it was always clear that it would modify GitLab CI files in some way.

In my early efforts, I wrote the tool to make as few changes as possible. Then I thought to create a new CI format that compiles back to GitLab CI. In the most recent iteration, `cici` now implements [GitLab CI's schema directly in Python](https://gitlab.com/brettops/tools/cici-tools/-/blob/main/cici/providers/gitlab/models.py).

This latest approach has been the most time-consuming so far, but it has meant that reading a file in and writing it back out corrects the formatting in the process. Hence, `cici fmt` was born.

`cici fmt` can be run with or without files as parameters, like so:

```bash
cici fmt
```

```console
$ cici fmt
.gitlab-ci.yml formatted
```

If no file is passed, it defaults to a file in the current directory named `.gitlab-ci.yml`.

When `cici fmt` is run, it will:

- Add quotes to strings where the syntax would be ambiguous otherwise,

- Reorder jobs in the file,

- Fix indents and line spacing,

- And some other random stuff.

Currently, it will also expand YAML anchors and `extends` keywords, because it shares a certain code path with `cici bundle`, even though it shouldn't. So it's not quite ready for prime time, but I'm working on it.

Once it's finished, it'll be pretty exciting to have a GitLab CI linter that doesn't require calling out to a GitLab instance.

## Pin `include` versions with `cici update`

There's a question I've pondered for a long time. How does one create shared pipelines, push updates to everyone quickly, and also track changes over time?

There are two obvious choices, with their own obvious problems:

- If everyone uses `main` / `latest` / what have you, everyone picks up changes immediately, and no one has any idea what versions are in use.

- If everyone pins to a specific version, they know exactly what versions are in use and will likely never upgrade them unless they absolutely have to.

`cici update` offers a third choice: developers can continuously track the latest pipeline changes using a version-pinning tool so that they always know what versions they have, but are also able to pick up updates automatically.

Here's an example CI file:

```yaml
# .gitlab-ci.yml
include:
  - project: brettops/pipelines/prettier
    file: include.yml
  - project: brettops/pipelines/python
    file:
      - lint.yml
      - setuptools.yml
      - twine.yml
```

Now call `cici update`:

```bash
cici update
```

```console
$ cici update
brettops/pipelines/prettier pinned to 0.1.0
brettops/pipelines/python is the latest at 0.5.0
```

If you check back into that CI file, you'll see pinned versions:

```yaml
# .gitlab-ci.yml
include:
  - project: brettops/pipelines/prettier
    ref: 0.1.0
    file: include.yml
  - project: brettops/pipelines/python
    ref: 0.5.0
    file:
      - lint.yml
      - setuptools.yml
      - twine.yml
```

That's it! That's all it does, but it helps me a lot.

Add the pre-commit hook to your project and `cici update` will run on every commit:

```yaml
# .pre-commit-config.yaml
repos:
  # other hooks ...

  - repo: https://gitlab.com/brettops/tools/cici-tools
    rev: "0.2.5"
    hooks:
      - id: update
```

`cici update` pulls the latest [GitLab Release](https://docs.gitlab.com/ee/user/project/releases/) for a pipeline project by date, so there are no versioning requirements for the upstream, but it does mean that the upstream needs to publish regular releases.

## Bundle CI files with `cici bundle`

The `cici bundle` command splits a large CI file into many CI "bundles", one for each job. These bundled CI files have everything each job needs to run, so that each job can be consumed à la carte by downstream projects. It currently expands `extends` keywords, YAML anchors, and global variable declarations, with `include` expansion planned.

Let's use the [`brettops/pipelines/python` pipeline](https://gitlab.com/brettops/pipelines/python) as an example. It provides a large number of jobs that all depend on one or two base jobs. I won't attempt to reproduce its [growing CI file](https://gitlab.com/brettops/pipelines/python/-/blob/main/.cici/.gitlab-ci.yml), but here are a few jobs:

```yaml
# .gitlab-ci.yml
# ...
python-black:
  extends: .python-base-small
  script:
    - $PYTHON -m pip install black
    - $PYTHON -m black --check --diff .

python-isort:
  extends: .python-base-small
  script:
    - $PYTHON -m pip install isort
    - $PYTHON -m isort --profile=black --check --diff .

python-mypy:
  extends: .python-base
  stage: test
  script:
    - *python-script-pip-install
    - $PYTHON -m pip install mypy
    - $PYTHON -m mypy "${PYTHON_PACKAGE}" --junit-xml report.xml
  artifacts:
    reports:
      junit: report.xml

python-pyroma:
  extends: .python-base-small
  script:
    - $PYTHON -m pip install pyroma
    - $PYTHON -m pyroma -n "$PYTHON_PYROMA_MINIMUM_RATING" .
  rules:
    - exists:
        - setup.py
# ...
```

If you'd like to use only some of the jobs here, but not all of them, you've got yourself a pickle. Splitting it into multiple CI files means that they'll need to depend on another file containing `.python-base` or `.python-base-small`. If you try to include more than one of these split files, GitLab will refuse, citing diamond inheritance. Ouch!

To overcome this, `cici bundle` will act as a compiler and build final versions that are independent from one another. Here's a bundled version of the `python-pyroma` job from above:

```yaml
# pyroma.yml
stages:
  - test
  - build
  - deploy

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "push" && $CI_OPEN_MERGE_REQUESTS
      when: never
    - when: always

variables:
  # ...

python-pyroma:
  stage: test
  image: "${CONTAINER_PROXY}python:${PYTHON_VERSION}-alpine"
  variables:
    GIT_DEPTH: "1"
    GIT_SUBMODULE_STRATEGY: "none"
    PIP_CONFIG_FILE: "$PYTHON_PIP_CONFIG_FILE"
    PYTHON: "/usr/local/bin/python3"
  before_script:
    - |-
      if [[-n "$PYTHON_PYPI_GITLAB_GROUP_ID"]] ; then
        export PYTHON_PYPI_DOWNLOAD_URL="https://${PYTHON_PYPI_USERNAME}:${PYTHON_PYPI_PASSWORD}@${CI_SERVER_HOST}/api/v4/groups/${PYTHON_PYPI_GITLAB_GROUP_ID}/-/packages/pypi/simple"
        echo "Pulling PyPI packages from GitLab group ID $PYTHON_PYPI_GITLAB_GROUP_ID"
      elif [[-n "$PYTHON_PYPI_GITLAB_PROJECT_ID"]] ; then
        export PYTHON_PYPI_DOWNLOAD_URL="https://${PYTHON_PYPI_USERNAME}:${PYTHON_PYPI_PASSWORD}@${CI_SERVER_HOST}/api/v4/projects/${PYTHON_PYPI_GITLAB_PROJECT_ID}/packages/pypi/simple"
        echo "Pulling PyPI packages from GitLab project ID $PYTHON_PYPI_GITLAB_PROJECT_ID"
      fi
    - |-
      if [[-n "$PYTHON_PYPI_DOWNLOAD_URL"]] ; then
      cat > "$PIP_CONFIG_FILE" <<EOF
      [global]
      index-url = ${PYTHON_PYPI_DOWNLOAD_URL}
      EOF
      fi
  script:
    - $PYTHON -m pip install pyroma
    - $PYTHON -m pyroma -n "$PYTHON_PYROMA_MINIMUM_RATING" .
  cache: {}
  rules:
    - exists:
        - setup.py
```

The above job is fully expanded and no longer depends on another CI file.

Adopting `cici bundle` on your own project isn't very complex. The first thing you'll need to do is move your existing shared CI file into a new `.cici/` directory as `.gitlab-ci.yml`:

```bash
mkdir -p .cici/
git mv include.yml .cici/.gitlab-ci.yml
```

The contents of your CI file can mostly stay the same, with the caveat that `cici` ignores hidden jobs (those that start with `.`), so those ones will not be bundled.

Ensure that every job in your CI file starts with the path of the project. For example, for `brettops/pipelines/ansible`, the jobs must start with `ansible-`. This restriction may be lifted in a future release.

It is also wise to prefix all global variables with the project path. So for `brettops/pipelines/ansible`, your variables should start with `ANSIBLE_` (though this is not currently enforced by the tool).

Now you can run `cici bundle`:

```bash
cici bundle
```

```console
$ cici bundle
pipeline name: python
bundle names: ['black', 'isort', 'mypy', 'pyroma', 'pytest', 'setuptools', 'twine', 'vulture']
created black.yml
created isort.yml
created mypy.yml
created pyroma.yml
created pytest.yml
created setuptools.yml
created twine.yml
created vulture.yml
```

As noted above, this creates new CI files. Add the pre-commit hook to your project, and your CI bundles will be rebuilt every time you try to commit:

```yaml
# .pre-commit-config.yaml
repos:
  # other hooks ...

  - repo: https://gitlab.com/brettops/tools/cici-tools
    rev: "0.2.5"
    hooks:
      - id: bundle
```

Now you can use as many or as few components of your reusable CI file as you like:

```yaml
# .gitlab-ci.yml
include:
  - project: brettops/pipelines/python
    file:
      - black.yml
      - isort.yml
      - mypy.yml
      - pyroma.yml
      - pytest.yml
      - setuptools.yml
      - twine.yml
      - vulture.yml
```

...which is awesome!

This tool is definitely still in development, so there is, again, no guarantee that it will work for any particular use case. Also, not all GitLab CI syntax is supported yet, but `cici` will loudly complain when it encounters syntax it doesn't recognize. Here is the [currently supported syntax](https://gitlab.com/brettops/tools/cici-tools/-/blob/main/docs/providers/gitlab.md).

I'm slowly working to adopt `cici-tools` across the BrettOps pipeline catalog, starting with the more complex ones that really, _really_ need this functionality. The old shared pipeline files are much harder to maintain, and while they will be kept around on a best-effort basis, it is highly recommended to transition over.

## Conclusion

These three tools make up the `cici` command currently, but there are more on the way. A lot of possibilities have opened up by having a GitLab CI [structurizer](https://brettops.io/blog/loading-config-files-python/#structurize-into-models) written in Python, which means I can now manipulate CI files all day long, in a type-safe, immutable way, using my favorite programming language.

Over the years, I've written a lot of scattered tools and scripts to perform automated edits and analyses to GitLab CI files, but I anticipate this effort will unify my approach a lot. Things that I expect to fall out of this effort include, but are not limited to:

- Automatic pinning of job image versions

- Extensions to GitLab CI, like sourcing job scripts from standalone bash scripts rather than only YAML files

- Bundle-time resolution of included CI pipelines to reduce blast radius of bad pipeline changes

- Converting GitLab CI/CD to / from other formats to some degree, both to provide an onramp onto GitLab from other CI systems, and to use GitLab CI syntax as a "write once, run everywhere" format

- Simulating a mostly complete GitLab CI pipeline locally

Who knows what will come next, but I'm excited to see where this tool goes. Drop me a line at <a href="mailto:contact@brettops.io">contact@brettops.io</a> to tell me what you think! Happy coding!