GitLab CI: Cache and Artifacts explained by example
Tags: devops, node, cicd, gitlab
Published: 2021-08-04 10:13:46

Hi, DEV Community! I've been working in the software testing field for more than eight years. Apart from web services testing, I maintain CI/CD Pipelines in our team's GitLab.

Let's discuss the difference between GitLab cache and artifacts. I'll show how to configure the Pipeline for the Node.js app in a pragmatic way to achieve good performance and resource utilization.

There are three things you can watch forever: fire burning, water falling, and the build is passing after your next commit. Nobody wants to wait for the CI completion too much, it's better to set up all the tweaks to avoid long waiting between the commit the build status. Cache and artifacts to the rescue! They help reduce the time it takes to run a Pipeline drastically.

People are confused when they have to choose between cache and artifacts. GitLab has bright documentation, but [the Node.js app with cache example](https://docs.gitlab.com/ee/ci/caching/#cache-nodejs-dependencies) and the Pipeline [template for Node.js](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Nodejs.gitlab-ci.yml) contradict each other.

Let's see what the Pipeline in GitLab terms means. The [Pipeline](https://docs.gitlab.com/ee/ci/pipelines/pipeline_architectures.html) is a set of stages and each stage can have one or more jobs. Jobs work on a distributed farm of runners. When we start a Pipeline, a random runner with free resources executes the needed job. The GitLab-runner is the agent that can run jobs. For simplicity, let's consider Docker as an executor for all runners.

Each job starts with a clean slate and doesn't know the results of the previous one. If you don't use cache and artifacts, the runner will have to go to the internet or local registry and download the necessary packages when installing project dependencies.

### What is cache?
It's a set of files that a job can download before running and upload after execution. By default, the cache is stored in the same place where GitLab Runner is installed. If the distributed cache is configured, S3 works as storage.
![GitLab Cache](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/az3xa3hssyj8q12s6fiw.png)
Let's suppose you run a Pipeline for the first time with a local cache. The job will not find the cache but will upload one after the execution to runner01. The second job will execute on runner02, it won't find the cache on it either and will work without it. The result will be saved to runner02. Lint, the third job, will find the cache on runner01 and use it (pull). After execution, it will upload the cache back (push).

### What are artifacts?
Artifacts are files stored on the GitLab server after a job is executed. Subsequent jobs will download the artifact before script execution.
![GitLab artifacts](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pappaj15pbvty5z7k38q.png)
Build job creates a DEF artifact and saves it on the server. The second job, Test, downloads the artifact from the server before running the commands. The third job, Lint, similarly downloads the artifact from the server. 

To compare the artifact is created in the first job and is used in the following ones. The cache is created within each job.

Consider the CI template example for Node.js recommended by GitLab:
```
image: node:latest # (1)
 
# This folder is cached between builds
cache:
  paths:
    - node_modules/ # (2)
 
test_async:
  script:
    - npm install # (3)
    - node ./specs/start.js ./specs/async.spec.js
 
test_db:
  script:
    - npm install # (4)
    - node ./specs/start.js ./specs/db-postgres.spec.js
```

Line #1 specifies the docker image, which will be used in all jobs. The first problem is the `latest` tag. This tag ruins the reproducibility of the builds. It always points to the latest release of Node.js. If the GitLab runner caches docker images, the first run will download the image, and all subsequent runs will use the locally available image. So, even if a node is upgraded from version XX to YY, our Pipeline will know nothing about it. Therefore, I suggest specifying the version of the image. And not just the release branch (`node:14`), but the full version tag (`node:14.2.5`).

Line #2 is related to lines 3 and 4. The `node_modules` directory is specified for caching, the installation of packages (npm install) is performed for every job. The installation should be faster because packages are available inside `node_modules`. Since no key is specified for the cache, the word `default` will be used as a key. It means that the cache will be permanent, shared between all git branches.

Let me remind you, the main goal is to keep the pipeline _reproducible_. **The Pipeline launched today should work the same way in a year**.

NPM stores dependencies in two files — _package.json_ and _package-lock.json_. If you use _package.json_, the build is not reproducible. When you run `npm install` the package manager puts the last minor release for not strict dependencies. To fix the dependency tree, we use the _package-lock.json_ file. All versions of packages are strictly specified there.

But there is another problem, `npm install` rewrites package-lock.json, and this is not what we expect. Therefore, we use the special command `npm ci` which:
* removes the node_modules directory;
* installs packages from package-lock.json.

What shall we do if `node_modules` will be deleted every time? We can specify NPM cache using the environment variable `npm_config_cache`.

And the last thing, the config does not explicitly specify the stage where jobs are executed. By default, the job runs inside the test stage. It turns out that both jobs will run in parallel. Perfect! Let's add jobs stages and fix all the issues we found.

What we got after the first iteration:
```
image: node: 16.3.0 # (1)
 
stages:
  - test
 
variables:
  npm_config_cache: "$CI_PROJECT_DIR/.npm" (5)
 
# This folder is cached between builds
cache:
  key:
    files:
      - package-lock.json (6)
  paths:
    - .npm # (2)
 
test_async:
  stage: test
  script:
    - npm ci # (3)
    - node ./specs/start.js ./specs/async.spec.js
 
test_db:
  stage: test
  script:
    - npm ci # (4)
    - node ./specs/start.js ./specs/db-postgres.spec.js
```

We improved Pipeline and make it reproducible. There are two drawbacks left. First, the cache is shared. Every job will pull the cache and push the new version after executing the job. It's a good practice to update cache only once inside Pipeline. Second, every job installs the package dependencies and wastes time.

To fix the first problem we describe the cache management explicitly. Let's add a "hidden" job and enable only pull policy (download cache without updating):
```
# Define a hidden job to be used with extends
# Better than default to avoid activating cache for all jobs
.dependencies_cache:
  cache:
    key:
      files:
    	- package-lock.json
    paths:
      - .npm
    policy: pull
```

To connect the cache you need to inherit the job via `extends` keyword.
```
...
extends: .dependencies_cache
...
```

To fix the second issue we use artifacts. Let's create the job that archives package dependencies and passes the artifact with `node_modules` further. Subsequent jobs will run tests from the spot.
```
setup:
  stage: setup
  script:
    - npm ci
  extends: .dependencies_cache
  cache:
    policy: pull-push
  artifacts:
    expire_in: 1h
    paths:
      - node_modules
```

We install the npm dependencies and use the cache described in the hidden dependencies_cache job. Then we specify how to update the cache via a pull-push policy. A short lifetime (1 hour) helps to save space for the artifacts. There is no need to keep `node_modules` artifact for a long time on the GitLab server.

The full config after the changes:
```
image: node: 16.3.0 # (1)
 
stages:
  - setup
  - test
 
variables:
  npm_config_cache: "$CI_PROJECT_DIR/.npm" (5)
 
# Define a hidden job to be used with extends
# Better than default to avoid activating cache for all jobs
.dependencies_cache:
  cache:
    key:
      files:
        - package-lock.json
    paths:
      - .npm
    policy: pull
 
setup:
  stage: setup
  script:
    - npm ci
  extends: .dependencies_cache
  cache:
    policy: pull-push
  artifacts:
    expire_in: 1h
    paths:
      - node_modules
 
test_async:
  stage: test
  script:
    - node ./specs/start.js ./specs/async.spec.js
 
test_db:
  stage: test
  script:
    - node ./specs/start.js ./specs/db-postgres.spec.js
```

We learned what's the difference between cache and artifacts. We built a reproducible Pipeline that works predictably and uses resources efficiently. This article shows some common mistakes and how to avoid them when you are setting up CI in GitLab.
I wish you green builds and fast pipelines. Would appreciate your feedback in the comments!

## Links

* [Pipeline architecture](https://docs.gitlab.com/ee/ci/pipelines/pipeline_architectures.html);
* [Caching in GitLab CI/CD](https://docs.gitlab.com/ee/ci/caching/).