Aggressive dependency caching in GitHub Actions
Tags: devops, webdev, node, python
Published: 2022-10-18 15:54:52

There are three things you can watch forever: fire burning, water falling, and how the build passes the stages in Pipeline after the next commit. To make the wait less tedious, it's best to take care of the CI setup from the beginning.

GitHub Actions has a cache that gets to the runner's virtual machine in seconds. In this article I'd like to share examples of how to set up aggressive dependency caching. Why did I call this approach "aggressive caching"? Because we will be caching not only the packages archives but also the state of the environment after installation.

For Node.js it will be the `node_modules` directory, and for Python it will be the virtualenv directory with installed dependencies.

## Node.js Example

Let's take the typical setup for dependency caching example mentioned in the documentation. If you don't need any exotics, you can use the standard `actions/setup-node` [action](https://github.com/actions/setup-node#caching-global-packages-data), specifying a package manager.
```yaml
steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
  with:
    node-version: 16
    cache: 'npm'
- run: npm ci
- run: npm test
```

This will save the `.npm` directory with the global package cache. Sounds great! Remember that if we have several workflow jobs requiring `npm ci` inside, this will also be time-consuming.

Let's imagine a pipelining with several jobs:
```
                         │
          Create         │                        Reuse
          Dependencies   │                        Dependencies
          Cache          │
                         │  ┌────────────────────┐
                ┌────────┼──►     Lint  Job      ├────────────────────┐
                │        │  └────────────────────┘                    │
                │        │                                            │
                │        │                                            │
┌───────────────┴────┐   │  ┌────────────────────┐                 ┌──▼─────────────────┐
│     Build Job      ├───┼──►     Test  Job      ├─────────────────►     Deploy Job     │
└───────────────┬────┘   │  └────────────────────┘                 └──▲─────────────────┘
                │        │                                            │
                │        │                                            │
                │        │  ┌────────────────────┐                    │
                └────────┼──►     E2E   Job      ├────────────────────┘
                         │  └────────────────────┘
                         │
                         │
                         │
                         │
```

Ideally, we want to install dependencies only in the first job and get a state with available dependencies in all subsequent jobs.
I'll show how to achieve it using a sample repo — [redux-react-realworld-example-app](https://github.com/drakulavich/ts-redux-react-realworld-example-app).

![Node.js CI workflow on GitHub Actions](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pip2fqk7w7so3zucq3v5.png)

The first (`build`) job might look like this:
```yaml
steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
  with:
    node-version-file: '.nvmrc' # (1)
    cache: 'npm'

- name: Cache NPM dependencies # (2)
  uses: actions/cache@v3
  id: cache-primes
  with:
    path: node_modules
    key: ${{ runner.os }}-node-${{ hashFiles('package-lock.json') }}

- name: Install dependencies # (3)
  if: steps.cache-primes.outputs.cache-hit != 'true'
  run: npm ci

- name: Build
  run: npm run build
```

Line #1 specifies the node version using `.nvmrc` file. That's the alternative way to specify the version and it helps follow the DRY - Don't Repeat Yourself principle.

In line #2 we use `actions/cache` to cache the `node_modules` directory. We use the hash from the `package-lock.json` file as the key.

In line #3 we only install dependencies if the cache is invalidated.

To automatically retrieve `node_modules` in subsequent jobs, you must declare `actions/cache` with **the same key**. For example the `test` job can be configured as:
```yaml
steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
  with:
    node-version-file: '.nvmrc'
    cache: 'npm'

- name: Cache NPM dependencies
  uses: actions/cache@v3
  with:
    path: node_modules
    key: ${{ runner.os }}-node-${{ hashFiles('package-lock.json') }} # (1)

- name: Tests
  run: npm run test # (2)
```

Line #1 specifies the cache key. The key must be the same as in the `build` job. After the `actions/cache` step we consider that the dependencies are installed and run the tests in line #2.

Check out the complete [workflow on GitHub](https://github.com/drakulavich/ts-redux-react-realworld-example-app/blob/main/.github/workflows/pipeline.yml).

## Python Example

Standard scenario from `actions/setup-python` [docs](https://github.com/actions/setup-python#caching-packages-dependencies):
```yaml
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
  with:
    python-version: '3.9'
    cache: 'pip' # caching pip dependencies
- run: pip install -r requirements.txt
```

This workflow will cache pip packages in `~/.cache/pip`, but the installation step will always be performed, as in the previous example with `npm ci`.

Let's see how we can optimize the installation of dependencies. I'll use the Django-based [education-backend](https://github.com/tough-dev-school/education-backend ) repo.

![Python CI workflow on GitHub Actions](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/arn91ubhnzb0vv4i2jht.png)

Let's dive into the `build` job:
```yaml
steps:
- uses: actions/checkout@v3

- uses: actions/setup-python@v4
  id: setup-python
  with:
    python-version-file: '.python-version'

- uses: actions/cache@v3
  with:
    path: venv
    key: ${{ runner.os }}-venv-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('**/*requirements.txt') }} # (1)

- name: Install dependencies # (2)
  if: steps.cache-primes.outputs.cache-hit != 'true'
  run: |
    python -m venv venv
    . venv/bin/activate
    pip install --upgrade pip pip-tools
    pip-sync requirements.txt dev-requirements.txt

- name: Run the linter
  run: |
    . venv/bin/activate # (3)
    cp src/app/.env.ci src/app/.env
    make lint
```

As you can see, we use the same idea for caching as for the Node.js project. There are some minor changes which are quite important. We need to specify the cache key for each python version involved in the workflow. Line #1 has `steps.setup-python.outputs.python-version` variable exactly for this purpose.

Dependencies installation from line #2 is tricky. For python we use a virtual environment created with the module `venv`. The environment directory `venv` will be cached. You can think about it as `node_modules` for node.

Line #3 has one more trick. After the cache is warmed up it's necessary to initialize virtualenv in the future steps. Otherwise, the python interpreter will not be able to detect the necessary libraries to import.

The simplified `test` job may look as follows:
```yaml
- uses: actions/checkout@v3

- uses: actions/setup-python@v4
  id: setup-python
  with:
    python-version-file: '.python-version'

- uses: actions/cache@v3
  with:
    path: venv
    key: ${{ runner.os }}-venv-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('**/*requirements.txt') }} # (1)

- name: Run the tests
  run: |
    . venv/bin/activate # (2)
    cp src/app/.env.ci src/app/.env
    make test
```

Ensure you use the same key for the caching step (line #1) and remember to activate the virtual environment before running the tests (line #2).

Check out the complete [workflow on GitHub](https://github.com/tough-dev-school/education-backend/blob/master/.github/workflows/ci.yml).

## Summary

We've practiced "aggressive caching" with Node.js and Python examples. As far as you have a significant number of dependencies the changes can speed up your GitHub workflow sensibly. I recommend trying to set up workflows for your project using the references I've mentioned:
* [Node.js CI Workflow](https://github.com/drakulavich/ts-redux-react-realworld-example-app/blob/main/.github/workflows/pipeline.yml)
* [Python CI Workflow](https://github.com/tough-dev-school/education-backend/blob/master/.github/workflows/ci.yml)

If you still have questions about caching in GitHub Actions, don't hesitate to ask in the comments. I'll try to help.
I would be grateful if you share your tips on how to speed up workflows on GitHub too.