How to Publish an NPM Package from a GitLab CI/CD Pipeline
Tags: cicd, javascript, typescript, gitlab
Published: 2022-06-08 22:00:54

## TL;DR

Take a look at the `.gitlab-ci.yml` configuration file in my recent [safe-env-vars](https://gitlab.com/SaikoJosh/safe-env-vars) project. This shows how I configured GitLab’s built-in CI/CD pipelines to automatically build and publish the package to npm. This post will explain the steps required to set this up in detail.

## What’s in a pipeline?

Long before GitHub joined the party with GitHub Actions it was possible to run continuous integration and deployment (CI/CD) workloads on GitLab. For the best part of a decade now, we’ve been able to use the same tool that hosts our code repositories to run our pipelines. I was previously a big fan of [Circle CI](https://circleci.com/), especially when they introduced support for Docker containers, but I haven’t needed to use these guys in years.

But why do we care about CI/CD pipelines in the first place? Well, continuous integration (CI) and continuous deployment (CD) are essential for ensuring software quality and agility in distributed teams. They allow us to run automated quality checks, execute our test suites, scan for security vulnerabilities, publish and deploy our apps, and a myriad of other important tasks. They’re also great at saving time on small projects such as npm packages!

At a high level, CI/CD pipelines cover these areas:

1. **Building** — Installing dependencies, compiling your code or application, and anything else required to prepare your project for testing and deployment.
2. **Verifying** — Performing static analysis of your codebase, assets, and compiled artifacts to ensure they meet quality and security standards.
3. **Testing** — Executing your test suite to confirm everything works as expected and no bugs have crept in.
4. **Deploying** — Publishing your package or deploying your application to the cloud.
5. **Reporting** — Providing insight into build errors, development velocity, security vulnerabilities, and a lot more.

## What’s in a GitLab pipeline?

Recently I wrote a post about [environment variables](https://joshuacole.me/thoughts/youre-doing-environment-variables-all-wrong-b225358b-3890-5713-aa00-57debcd489a5) where I discussed an npm package I had published called [safe-env-vars](https://www.npmjs.com/package/safe-env-vars) — that package is automatically built and published using GitLab CI/CD (and it’s open source) so if you want to skip ahead why not take a browse of its [code repository](https://gitlab.com/SaikoJosh/safe-env-vars). The main point of interest for those wishing to skip ahead 🏃💨 is the `.gitlab-ci.yml` file which tells GitLab about the jobs we want to execute in the pipeline.

Before we can get into the nitty gritty there are a few terms GitLab uses that we ought to define:

- **Pipeline** — A single CI/CD execution consisting of several stages and jobs, typically triggered by a commit or a merge into a specific branch. You can have different pipelines for different purposes, for example, to verify changes in a merge request, or conduct a cloud deployment.
- **Stage** — A grouping of related jobs that must succeed before GitLab will proceed to the next stage in the pipeline. If there are no further stages, the pipeline will complete successfully.
- **Job** — A task you define that typically runs in its own idempotent container (e.g. `npm run build`, `npm publish`, etc). Multiple jobs within the same stage can be run in parallel as they are usually independent of each other.

GitLab pipelines are very powerful and can be configured in a lot of different ways. Some features are restricted to the paid and enterprise pricing tiers but all the features we’ll use below are available for free, as long as you have some pipeline minutes remaining! At the time of writing GitLab provides 400 free minutes per month for pipeline execution, which is more than ample for our needs here.

## Our goal

So what are we aiming to do here? When I created the [safe-env-vars](https://www.npmjs.com/package/safe-env-vars) package I had two goals:

1. Automatically publish my package to npm whenever I committed or merged into the `main` branch so I didn’t need to do this manually.
2. Run some code quality checks to catch any silly mistakes. I didn’t want to spend much time setting up these checks so I opted to limit them to code linting and unit testing.

Our pipeline will consist of three `stages` and four `jobs`:

| Stage Name | Job Name | Job Description |
| --- | --- | --- |
| Prepare | Install | Runs npm install to install key development dependencies such as TypeScript and Jest, and to build our package using the TS compiler. |
| Verify | Lint | Runs ESLint to verify our code meets the agreed linting rules. |
|  | Test | Runs Jest to verify all unit tests pass and nothing has broken. |
| Publish | Publish | Runs npm publish to publish our package to the npm registry. |

Our final pipeline will look something like this:
![Example pipeline](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xm6y99bb6t7ocumpx4rr.png)  

## Step by step guide

### Step 1: Create your package

Before you configure the pipeline you’ll need to have created a package with a suitable `package.json` file. Besides including the usual properties such as the dependecnies, author, and so on, you need to make sure the `“main”` property points to the JavaScript file which is the entrypoint to your package.

For **TypeScript** projects you’re going to need a few extra bits, but you can ignore these if you’re working with vanilla JavaScript (but why 😭):

- The `“types”` property should point to the entrypoint for exported type definitions.
- The `“scripts.build”` property should call the TS compiler, something like `"build": "npx tsc --outDir ./dist"`.
- The `“scripts.prepublishOnly”` property should look like `"prepublishOnly": "npm run build"` in order to automatically build your package before publishing with `npm publish`.

You might end up with a `package.json` file looking similar to this:

```json
{
	"name": "my-package",
	"version": "1.0.0",
	"main": "dist/src/main.js",
	"types": "index.d.ts",
	"scripts": {
		"build": "npx tsc --outDir ./dist",
		"lint": "npx eslint . --ext .ts,.tsx,.js,.jsx",
		"prepublishOnly": "npm run build",
		"test": "npx jest --watch",
		"test:cov": "npx jest --coverage --verbose --runInBand"
	},
  "dependencies": {...},
  "devDependencies": {...},
}
```

My personal preference is to have additional scripts defined in the package.json for linting (`lint`), testing (`test`), and testing with coverage (`test:cov`).

### Step 2: Add an npm ignore file

In the root of your code repository you’ll want to add a file called `.npmignore` and within it list all the paths you wish to exclude from your published package. This is important because there are some files you *don’t* need to include, and some you definitely *shouldn’t* include.

The syntax is very similar `.gitignore` and you might end up with something like the below. This ensures the `src` directory doesn’t get included (for TypeScript projects we only want our `dist` directory published). It also lists the various configuration files which aren’t important for people to actually use the package in their apps and just add unnecessary weight to a node_modules directory.

```
src/
.gitlab-ci.yml
.eslintignore
.gitignore
.eslintrc.js
.gitmodules
.prettierrc.js
jest.config.mjs
tsconfig.json
dist/**/*.spec.*
```

### Step 3: Add the CI/CD config file

In the root of your code repository create an empty `.gitlab-ci.yml` file with the following contents (the full reference for GitLab’s config file is available here: [https://docs.gitlab.com/ee/ci/yaml](https://docs.gitlab.com/ee/ci/yaml/)):

```yaml
image: node:16

variables:
  ARTIFACT_DOWNLOAD_ATTEMPTS: 2 # Number of retries for restoring saved artifacts.
  FORCE_COLOR: '1' # Whether to force colour output in compatible tooling.
  GET_SOURCES_ATTEMPTS: 2 # Number of retries for git commands.
  GIT_STRATEGY: fetch # How to pull submodules.
  GIT_SUBMODULE_STRATEGY: recursive # How to treat nested git submodules.
  RESTORE_CACHE_ATTEMPTS: 2 # Number of retries for restoring cached files.

stages:
  - prepare
  - verify
  - publish
```

The YAML above is in three parts:

1. The `image` string specifies the Docker image and tag to use (by default) for each of the jobs, in our case Node.js version 16.x. This can be overriden in each individual job later if needed. Advanced developers might want to create and publish their own Docker images with any additional dependencies their pipeline requires.
2. The `variables` dictionary lists the environment variables to make available to each job. These can also be customised for each job. I’ve started you here with some useful variables that customise the behaviour of the pipeline (see the comments beside each for what they do). For example, the `FORCE_COLOR` variable is used by many (but not all) tools to force them to output ANSI colour codes even when they can’t detect a TTY device.
3. Finally, the `stages` array lists the names of each of the three stages that will house our jobs. We will define each of the jobs next.

### Step 4: Add the “install” job

At the bottom of your config file add the following top-level key, `install`, which defines the job named “install”. The `install.stage` property defines which stage the job belongs to, in this case the “prepare” stage.

```yaml
install:
  stage: prepare
  interruptible: true
  retry: 1
  dependencies: []
  artifacts:
    name: 'deps_$CI_COMMIT_REF_SLUG'
    paths:
      - node_modules/
    when: on_success
    expire_in: 12h
  script:
    - npm config set -- '//registry.npmjs.org/:_authToken' "${NPM_AUTH_TOKEN}"
    - npm install --no-progress --no-audit --no-fund --verbose
  only:
    - merge_requests
    - develop
    - main
```

The full [keyword reference](https://docs.gitlab.com/ee/ci/yaml/) on GitLab’s website will explain all the various properties defined above, but the important ones to note are:

- **“artifacts”** — Tells GitLab which path(s) to hold on to between jobs, in this case, that’s the `node_modules` directory as we’ll be running `npm install` in this job only, and preserving the installed dependencies for use in subsequent jobs to save time.
- **“script”** — The lines of the shell script to execute for the job, in this case to set the NPM auth token that allows publishing to the registry. The `NPM_AUTH_TOKEN` environment variable should be set in the GitLab UI for your project or group and not comitted to your code repository! You can find the correct page in GitLab by clicking CI/CD in the left-hand menu of any project or group and going to the section titled “variables”.
- **“only”** — An array listing the events that can trigger the job to be included in the pipeline, in this case we execute the job for merge requests and for commits/merges to the `develop` and `main` branches. If none of the conditions match, the job will not be included in the pipeline.

### Step 5: Add the “lint” job

Add the next job, `lint`, at the bottom of the config file:

```yaml
# Lints the codebase.
lint:
  stage: verify
  interruptible: true
  retry: 1
  dependencies:
    - install
  script:
    - npm run lint
  only:
    - merge_requests
    - develop
    - main
```

Here, we have a similar set of properties defining what our job does, including `stage`, this time grouping our job in the `verify` stage, and `script` which simply runs the `npm run lint` script we defined in our package.json.

There is a new property to pay attention to, however. In order to execute our lint script ESLint and its associated dependencies need to be installed. Each CI/CD job is run in a new container and so doesn’t have any of the changes introduced in the preceeding `install` job, such as the `node_modules` directory.

In order to bring our installed `node_modules` into this `lint` job we need to specify the `dependencies` array and list our `install` job inside it. This will cause GitLab to pull in all the artifacts specified in the install job (i.e. `node_modules`) and put them back in the same places they were before. As far as npm will be concerned, the `node_modules` directory was always there!

### Step 6: Add the “test” job

Add the next job, `test`, at the bottom of the config file:

```yaml
# Runs the test suite.
test:
  stage: verify
  interruptible: true
  retry: 1
  dependencies:
    - install
  script:
    - npm run test:cov
  only:
    - merge_requests
    - develop
    - main
```

Once again, our job specifies the `install` job as a dependency so the `node_modules` directory is available again, and we specify a simple bash script that runs `npm run test:cov` - this simply executes our test runner, Jest, with coverage turned on. We also put this job under the `verify` stage and it will be run in parallel with the `lint` job to save time.

You might have noticed we specify a property called `interruptible: true` which simply tells GitLab it is allowed to kill our pipeline whilst this job is running if needed. This is useful when a new pipeline wants to start (i.e. something new has been committed to the same branch) that would otherwise have to wait for the current pipeline to finish, saving us both time and pipeline minutes!

### Step 7: Add the “publish” job

Add the next job, `test`, at the bottom of the config file:

```yaml
# Publishes the package to npm.
publish:
  stage: publish
  interruptible: false
  retry: 1
  dependencies:
    - install
  script:
    - npm config set -- '//registry.npmjs.org/:_authToken' "${NPM_AUTH_TOKEN}"
    - npm publish --verbose
  resource_group: 'deployment-$CI_COMMIT_REF_SLUG'
  rules:
    - if: '$CI_DEPLOY_FREEZE != null'
      when: never
    - if: '$CI_COMMIT_REF_NAME != "main" && $CI_COMMIT_REF_NAME != "develop"'
      when: never
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_COMMIT_REF_NAME == "develop"'
      when: never
    - when: on_success
```

Here we set the `interruptible` property to `false` because we don’t want GitLab to kill the pipeline halfway through a publish, leaving us in a situation where we can’t be sure what state our package is in… did that version get deployed or not? 🙀📦

We also specify `retry: 1` which tells GitLab to retry the job one additional time if it fails. This can be useful in cases where you’re relying on side effects for the job to succeed, for example `npm publish` makes HTTP requests to the npm registry which could fail due to transient network faults or a problem on npm’s side. In those cases, we want to automatically retry in case the problem resolves itself.

The other property of import here is `rules` where we list an array of objects. Each object defines a rule for when our publish job should run. We’re using this property instead of `only` here because we need to define much more complex behaviour than `only` will allow.

In the above code we have specified four rules and the first rule to match will be applied:

1. If the `CI_DEPLOY_FREEZE` environment variable is set, *never* execute the publish job.
2. If the branch we’re running on isn’t `main` or `develop`, *never* execute the publish job.
3. If we’re in the scope of a merge request, *never* execute the publish job.
4. Otherwise, and if the rest of the pipeline has been successful, then *execute* the publish job.

The syntax for `rules` is very powerful and allows us to specify a complex set of conditions for when the job should run (or not). There is far more to this than I can possibly detail here, so I encourage you to refer to the GitLab documentation: [https://docs.gitlab.com/ee/ci/yaml](https://docs.gitlab.com/ee/ci/yaml/).

### Step 8: Commit and push

Once your package and the GitLab CI/CD config file are ready, you can commit and push your changes up. If everything has gone to plan, you should see a pipeline appear in the GitLab UI for your repository under CI/CD in the left-hand menu. By clicking into the pipeline you can follow through the jobs as they execute, and drill down into the output logs for each individual job.

Below you can see an example of four pipelines that have been run. The first pipeline at the top of the list is the most recent and was run against the `main` branch. The second pipeline down was run on a merge request and does not include the `publish` job as the rules we configured exclude it from being included in merge request pipelines.

![Example pipelines](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wwo2lmvv0o0ofbahgsd5.png) 

Clicking into the first pipeline shows the stages and jobs that were executed. As this pipeline was run against the `main` branch our rules caused the `publish` job to be included and the package was subsequently (and automatically) published to npm. Happy days! 🥳

![Example pipeline stages and jobs](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/05uw0n9583oqlenp18xx.png) 

From here if we click into the `test` job we can see a snippet of the output emitted by Jest during the job. There is also some other log output which isn’t shown here for brevity.

![Example job output](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5gp6q6u2nvx6388grkos.png)
 
## Conclusion

In conclusion, it’s actually very simple to setup a powerful CI/CD pipeline in GitLab, especially if all you’re doing is publishing a package to the npm registry. Pipelines are a great way to verify the quality and robustness of your code, package, or application, and they are a fantastic timesaver when it comes to automatic publishing and deployment. To configure a pipeline in GitLab all you need is a `.gitlab-ci.yml` file in the root of your repository. You can find the keyword reference for that file on [GitLab’s website](https://docs.gitlab.com/ee/ci/yaml/).