Safely Upgrading our Open Source Dependencies at Scale
Tags: devops, linters, opensource
Published: 2024-06-28 19:03:20

[Trunk Check](https://docs.trunk.io/check) installs and manages over a hundred linters, code formatters, and other code quality tools. **The underlying tools are open source projects that each run on their own release schedule and, like all software, sometimes have bugs.** Figuring out which of our 100+ [tools](https://docs.trunk.io/check/configuration/supported) are safe to upgrade for our customers is a really hard problem. **We have to validate every new version of every tool we support**, and their accompanying runtimes and upstream dependencies, before shipping it to our customers. With so many integrations, across all different languages, it’s important to ensure compatibility and continuity as new versions are released. _Even something as simple as changing the name of a cli option could break thousands of workflows._ Here's how we solved the problem of testing, validating, and automating our linter upgrades.

# Black Box Integration Testing
Within the our execution model, each linter can have a runtime, a download, a parser, and configuration for how to run. With so many linter idiosyncrasies, it isn't practical to assert on the behavior of each individual step or linter output; instead, we rely on black box testing.

Our aim is to guarantee Trunk output continuity. When a new linter version is released, it should have the same or a similar set of diagnostics as its predecessor. Fortunately, this problem lends itself well to [Jest snapshots](https://jestjs.io/docs/snapshot-testing). We can capture the JSON result of a linter’s `trunk check` run, or the contents of a formatted file, and use that as our source of truth.

## Versioned Linter Snapshots
When we first run tests on a linter, _we save a snapshot of its Trunk output_, named with the linter’s version. Subsequent test runs will use the _most recent matching snapshot_ for our assertions, allowing us to keep a historical record of linter output and maintain compatibility.

We detect the latest version of a linter by querying its runtime or by using [GitHub APIs](https://docs.github.com/en/rest/releases/releases?apiVersion=2022-11-28#get-the-latest-release). Using our snapshot setup, and the assumption that the latest release of a linter will pass the most recent snapshot (_in practice, true about 95% of the time for new releases_), we can run nightly tests on the latest versions of all our supported linters. If a linter fails, we can investigate and adapt as needed. If it passes, we can upload its version to our own _Release Version Service_, which provides the source of truth for linter upgrades.

![Linter Upgrade Validation Pipeline Diagram](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5crl2rv384ivxwxrnz1r.png)

# Example of Linters Breaking
With this system in place, let’s see what happens when a linter breaks compatibility. Trunk integrated with the linter `tidy` when it was released at `known_good_version` 1.0.0. So, we generated a snapshot named `tidy_v1.0.0.check.shot`:

```json
{“issues”: [{
      "linter": "tidy",
      "code": "missing-return",
      "file": “test_data/adder.py",
      "line": "8",
      "column": "12",
      "level": "LEVEL_HIGH",
      "message": "’add’ does not return a value"
    }
]}
```

We can see that running `tidy` with Trunk generates one file issue for `adder.py`. We expect new versions to return the same result - anything else would indicate there was a failure or regression worth investigating.

After we generated this snapshot, `tidy` then released versions 1.0.1, 1.1.0, and 1.1.1, all of which had an identical output for our test data, so each passed our tests. No need to generate any new snapshots. One week later, `tidy` released a new version 1.2.0, which renamed one of its CLI arguments. Suddenly, our test reports a failure:

![Slack Notification of Test Failure](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/y9vxqympeiv3zzt4c4u8.png)

```json
{"issues": [],
"failures": [{
  "report": `tidy exited with exit_code=2
    stdout: (none)
    stderr: |
      usage: tidy [options]
      tidy: error: Unrecognized option found: --verify`
}]}
```

Trunk doesn’t know how to run `tidy@1.2.0`. In response, we can add a new versioned subcommand to Trunk configuration, and it runs again!

```yaml
lint:
  definitions:
    - name: tidy
      …
      commands:
        - name: lint
          version: “>=1.2.0”
          output: sarif
          run: tidy --sarif --verify-target {target}
          success_codes: [0, 1]
        - name: lint
          version: “>=1.0.0”
          output: sarif
          run: tidy --sarif --verify {target}
          success_codes: [0, 1]
```

The output also changed slightly, so we can create another snapshot, and name it `tidy_v1.2.0.check.shot`. Going forward, new releases of tidy will use this snapshot, but we always have the option to go back in time and verify that older versions still work as intended, or we can add more test data if we need to. Snapshots are lightweight enough that we can store historical information as long as we need, and Trunk accommodates hermetically installing and running old and new linter versions.

Keeping your linters up to date should be painless. You shouldn’t have to worry about missing downloads, compatibility issues, or sudden flakiness. Trunk will only ever upgrade you to validated linter versions that have passed snapshot tests, sparing you the burden of any bleeding-edge headaches.

![Successful Linter Upgrade Screenshot](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/y428a34nnd8dmscri8lx.png)

# Paying Dividends in the Real World
With this system in place, we have already caught several bugs before they could hit user repos.

* [Semgrep](https://github.com/returntocorp/semgrep/issues/6904) released a binary that was broken on macOS. We identified this issue and helped push for a quick fix.
* [Ansible-lint](https://github.com/ansible/ansible-lint/issues/3277) introduced a bug that executed user playbooks. We blocked this version from installing on user machines until a fix was landed.
* [Kube-linter](https://github.com/stackrox/kube-linter/releases/tag/v0.6.1) changed their download URL schema. Within hours, we had landed a fix supporting this new URL.
* Trunk runs tools hermetically, and several tools (e.g. `nixpkgs-fmt`, `golangci-lint`, `gofmt`) deprecated support for older runtimes. Seeing this, we automatically upgraded the runtimes for users running these linters.
* Using older snapshots of these linters, we can block any potential regression before it happens, remaining backwards-compatible whenever possible.

## How to Contribute
Trunk’s [plugins repository](https://github.com/trunk-io/plugins) is open-source, and we are always welcoming new contributions for additional linter integrations and improvements, as well as new actions. If you want to try out Trunk for yourself, [it's just one shell command away](https://docs.trunk.io/check/usage).