AUTOMATING NPM PACKAGE PRERELEASES
Tags: githubhack23, devops, npm, opensource
Published: 2023-05-07 21:09:05

## What I built

We at the [SLIIT FOSS Community](https://sliitfoss.org) are a passionate and committed team dedicated to promoting the use and development of open source software. It has been quite some time since we have shifted focus on building NPM and Dart libraries for use by fellow developers. The subject of this post however will be the former where we'll be exploring the use of a plethora of tools and concepts to build a fully automated prerelease mechanism for these libraries.

### Category Submission:

DIY Deployments

### App Link

https://www.npmjs.com/~sliit.foss


### Screenshots

![Workflow runs](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0awt9tpe6ffgs8vjfv0x.jpg)

![Prerelease workflow](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jzwa1h4tbw1kxkgq93by.jpg)

![Release workflow](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/b52rb8d9wks7gcs0240r.jpg)

---

### Description

NPM Catalogue isn't exactly an app on it's own. It's just a development environment for our NPM libraries. The main idea behind this project was to have everything required for the development, testing and release of an NPM library from step 1 in one single place making it as easy as possible for developers to contribute to these libraries allowing them to focus on the actual development itself rather than the surrounding processes. Simply, it's one of the many ways we at the SLIIT FOSS Community are trying to make open source development easier for everyone and we hope that this project will be of use to many developers out there.

---

### Link to Source Code

Repository - https://github.com/sliit-foss/npm-catalogue

Full list of changes which were made under this development cycle - https://github.com/Akalanka47000/npm-catalogue/pull/3/files

`Note that this is just a fork and the actual workflow runs can be found in the parent repository`

---

### Permissive License

MIT

---

## Background

As of today, we have over a dozen packages which have been released publicly to the NPM registry which you can find [over here](https://www.npmjs.com/~sliit.foss) and till now we only released these as either major, minor or patch updates. However as of late, with more complex developments, we had identified the need for prereleases which would primarily help us with a smoother and safer way of getting things done. There were instances that a library was completed to a certain extent, was ready for internal testing but not just yet ready for a full release. 

At the time, we had already automated the existing release process with the use of a [GitHub Actions Workflow](https://github.com/sliit-foss/npm-catalogue/blob/7c0d8dd4513b9d9934f73fda79f87416ed1d2217/.github/workflows/release.yml) that would be run on a push to the main branch of the repository. This workflow took care preparing the project, executing the required scripts for versioning, releasing the packages to the NPM registry and finally syncing the newly published package versions back into the repository by execting a commit within the workflow itself.

All of our packages are housed within this single repository and it is bootstrapped with [Turborepo](https://turbo.build/repo) which was already making things quite easy for us. For example, we could run a single command to execute a script across all packages within the repository and in our case, we had a `release` script defined within all of our packages and a simple process of running `pnpm release` would take care of the rest. The release script was organized as follows:-

> Root package.json

```json
{
    "scripts": {
        ...
        "release": "turbo run release --concurrency=1",
        ...
    }
}
```


> Workspace package.json

```json
{
    "scripts": {
        ...
        "build": "node ../../scripts/esbuild.config.js",
        "bump-version": "pnpm build && npx automatic-versioning --name=@sliit-foss/<package-name> --no-commit --recursive",
        "release": "bash ../../scripts/release.sh",
        ...
    }
}
```

> release.sh

```bash
pnpm run bump-version && pnpm publish --access=public --tag=latest || true
```

Our versioning logic was quite simple, we decided which type of release it was based on the prefix of the commit which triggered the workflow run, for example:-

- `Fix: <commit message>` would trigger a patch release
- `Feat: <commit message>` would trigger a minor release
- `Feat!: <commit message>` would trigger a major release

This process was quite easy since we anyways follow [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) while checking in changes. We have [Husky](https://typicode.github.io/husky/#/) setup with [Commitlint](https://commitlint.js.org/#/) which ensures that this safe-zone is never crossed. The version bumping itself is handled by a library of our own making called [@sliit-foss/automatic-versioning](https://www.npmjs.com/package/@sliit-foss/automatic-versioning) which evaluates the commit history and
determines the next version based on the type of release.


While this process was working fine, it did have these following problems:-

1. What if there was a prerelease that needed to be done? This would mean that we would have to manually bump the version of the package, publish it to the NPM registry and then sync the version back into the repository. This was quite a tedious process and we wanted to automate this as well. Further to this, all releases were done from the main branch itself which meant that we had to be extra careful when merging pull requests to the main branch since we didn't want to accidentally release a package which wasn't ready for release.

2. There was no development branch or anything which matched it. All changes had to be held within the feature branches itself to avoid polluting the main branch. This was quite a hassle since we had to keep track of which feature branches were ready for release and which weren't.

3. The release process was quite slow. As you might have already seen, the release script at the root level was limited to a concurrency of 1 and the scripts have sequential dependencies on previous scripts. For example `bump-version` calls `pnpm build` within itself. This was primarily to ensure that the scripts were run in the order of `build ---> bump-version ---> release`. This was unnecessary since Turborepo already has more efficient ways to manage this.

</br>

Here is a diagram of the release process as it was before and after this development cycle.

</br>

![Release Process](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7prs6vxl3iiz8g6d14m9.png)

### Fun fact! 

- The prerelease tag `blizzard` was actually named after the [Witcher potion Blizzard](https://witcher.fandom.com/wiki/Blizzard) which is a potion that slows down time and increases perception. We thought it was quite fitting since prereleases are meant to be a slow down of the release process and a way to increase perception of the changes that are being made, a concept which is quite similar to [Canary Releases](https://martinfowler.com/bliki/CanaryRelease.html).

---

### How I built it

- First off, we had two approaches to solve this:-

* Quick way 

    - Add Prerelease commit prefix evaluation to `automatic-versioning` and increment the version accordingly. This would mean that we would be able to merge in prerelease changes to the main branch with the associated commit prefix and let the existing release workflow take care of the rest. But this obviously will be polluting the main branch and doesn't solve the problem of every release being published with the `latest` tag which can be quite dangerous.

* Complete way

    -  Extract the existing release workflow into a custom [Composite GitHub Action](https://docs.github.com/en/actions/creating-actions/creating-a-composite-action). This would make the exact same release steps reusable for both types of releases. The variables which in this case consists of only the release tag were provided as inputs into this action along with the needed repositry secrets.

    - Modify the existing release workflow to consume this new action and add in a second prerelease workflow which gets triggered on pushes to development branch. The final workflow files and the action looks like this:-

    </br>

    > .github/actions/release/action.yml

    ```yaml
    name: release
    description: Base package release action
    inputs:
        npm_token:
            description: "Token to authenticate with the npm registry"
            required: true
    runs:
        using: composite
        steps:
          - name: Setup Node
            uses: actions/setup-node@v3
            with:
                node-version: '16.x'
                registry-url: 'https://registry.npmjs.org'

          - name: Configure git
            run: |
                git config --global user.email "github-actions[bot]@users.noreply.github.com"
                git config --global user.name "github-actions[bot]"
            shell: bash

          - run: git fetch --prune --unshallow
            shell: bash

          - name: Install dependencies
            run: npm install -g pnpm@8 && pnpm install --production --ignore-scripts
            shell: bash

          - name: Create .npmrc
            run: echo "//registry.npmjs.org/:_authToken=${{ inputs.npm_token }}" > .npmrc
            shell: bash

          - run: echo "git-checks=false" >> .npmrc
            shell: bash

          - name: Sync submodules
            run: pnpm sync-submodules
            shell: bash

          - name: Populate prerequisities
            run: |
                echo "{\"release_tag\":\"$TAG\"}" > cache-control.json
                for dir in packages plugins; do
                cd "$dir" && for p in */; do
                    cp ../{.npmignore,LICENSE,cache-control.json} "$p"
                done && cd ..
                done
            shell: bash

          - name: Publish packages on NPM
            run: |
                pnpm --filter @sliit-foss/automatic-versioning build && npm i -g ./packages/automatic-versioning
                pnpm release
            shell: bash

          - name: Cleanup
            run: rm -rf cache-control.json && rm -rf p*/**/cache-control.json
            shell: bash

          - name: Update release info
            run: |
                git config pull.ff true
                git add . && git commit -m "CI: @sliit-foss/automatic-versioning - sync release" || true
                git pull --rebase && git push origin
            shell: bash
    ```

    </br>

    > .github/workflows/release.yml

    ```yaml
    name: CI Release
    on:
        push:
            branches:
                - main
        workflow_dispatch:

    jobs:
        release:
            runs-on: ubuntu-latest
            env:
                TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
                TURBO_TEAM: ${{ secrets.TURBO_TEAM }}
                TAG: latest
            steps:
                - uses: actions/checkout@v3
                - uses: ./.github/actions/release
                with:
                    npm_token: ${{ secrets.NPM_TOKEN }}
    ```

    </br>

    > .github/workflows/prerelease.yml

    ```yaml
    name: CI Prerelease
    on:
        push:
            branches:
                - development
        workflow_dispatch:

    jobs:
        release:
            runs-on: ubuntu-latest
            env:
                TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
                TURBO_TEAM: ${{ secrets.TURBO_TEAM }}
                TAG: blizzard
            steps:
                - uses: actions/checkout@v3
                - uses: ./.github/actions/release
                with:
                    npm_token: ${{ secrets.NPM_TOKEN }}
    ```

    </br>

    - This on its own was not enough, we had to add in a couple of new features to our library `automatic-versioning` to be able to support this process. The newly added features are as follows:-


        - The ability to specify a prerelease tag as a command line argument when invoking the script.
        
        - The ability to specify a prerelease branch as a command line argument when invoking the script which will essentially change the commit prefix evaluation as follows while in that branch:-

            - Feat! --> Premajor
            - Feat --> Preminor
            - Fix --> Prepatch

        - The ability to designate a list of prefixes as ignored prefixes which will not be considered when evaluating the commit prefix. This was useful as we needed to ignore the prerelease version sync commit from the development branch which is prefixed with `CI:` from being considered when evaluating the commit prefix and to avoid it being considered as an invalid value for version incrementing.

        - The ability to recongize the commit history for a specific workspace instead of the whole repository as a whole

        - Support for commit scopes. For example we now can add in commits with prefixes as follows -> Feat(automatic-versioning): commit scope support

    
    - Further in the midst of this release, we added in the following bugfix to `automatic-versioning`

        - Invalid versioning if the commit contains a url in the commit message such as a merge commit with a source branch url

</br>

- Problem two got automatically solved with the above. The availablity of a development branch with its own release cycle meant that we could now merge pull requests into the development branch and keep them there as long as we wanted to until they were ready to be merged into the main branch.

</br>

- Finally to address our third and final problem, we removed the concurrency limit from the turbo script and structured the pipeline in our turbo.json as follows:-

</br>

```json
{
    ...
    "pipeline": {
        "build": {
            "dependsOn": [],
            "outputs": ["dist/**"]
        },
        "bump-version": {
            "dependsOn": ["build"],
            "outputs": ["package.json"]
        },
        "release": {
            "dependsOn": ["bump-version"],
            "outputs": []
        }
    }
    ...
}
```

</br>

This indefinitely sped up the release process since the scripts were now run in parallel and the dependencies were handled by Turborepo itself. Further to this, we integrated [Vercel Remote Caching](https://vercel.com/docs/concepts/monorepos/remote-caching) in our CI pipeline which meant that all of the steps were now cached and the subsequent runs were much faster. This process turned out be quite simple since we were already using Vercel for a few of our other projects and the integration was quite seamless. [Jared Palmer](https://jaredpalmer.com/) has done a quite a good job of making this process as simple as possible as it just requires 2 environment variables to be set in the CI environment which we added as repository secrets referenced in our workflows. For more information, refer the [following docs](https://turbo.build/repo/docs/ci/github-actions).

</br>

![Repository Secrets](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hos4z56sb2rry8pgrnxq.jpg)

</br>

Overall these changes reduced the release time from 1-2 mins to 30-45 seconds which was a huge improvement.

---

There however was a catch, since we were using Vercel Remote Caching, we had to ensure that the cache was invalidated whenever a new release was made in the main branch. Say for example I'm adding a patch to a package with a version 1.0.0 and merging it with the development branch. This package will be released as 1.0.1-blizzard-0 during the prerelease workflow run of this branch. Now if I merge the same branch to the main branch, in the eyes of Turborepo, the inputs to the release script have not changed since the time it was run in the development branch which will cause Turbo to replay the cache from it thus skipping the actual release of the package to version 1.0.1. Fortunately, dealing this was quite simple, we just needed turbo to maintain two caches for the 2 release types which we did by adding a simple `cache-control.json` file to each of the project workspaces at the time of the CI run as in the action steps below where the tag is different for the two workflow runs:-

```yaml
    - name: Populate prerequisities
      run: |
        echo "{\"release_tag\":\"$TAG\"}" > cache-control.json
        for dir in packages plugins; do
          cd "$dir" && for p in */; do
            cp ../{.npmignore,LICENSE,cache-control.json} "$p"
          done && cd ..
        done
      shell: bash

    # Release step

    - name: Cleanup
      run: rm -rf cache-control.json && rm -rf p*/**/cache-control.json
      shell: bash
```

---

As an added bonus, we also updated our existing [testing workflow](https://github.com/sliit-foss/npm-catalogue/blob/ca0f5cd334fc1ab805d1ec926a578a226395fd48/.github/workflows/tests.yml) to incorporate linting along with unit tests. We could've just added in a separate workflow for `linting` or another job itself within the existing `test.yaml` file but that would have been a massive repetition of code. After all only the `run` command would have been different for both jobs which is why we decided to use the `strategy` feature of Github Actions to run the same job twice with different run commands as follows:-

> .github/workflows/lint-test.yml

```yaml
name: CI Code Quality + Tests
on:
  pull_request:
    branches:
      - main
      - development

jobs:
  scripts:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        command: ['lint', 'test']
    env:
      TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
      TURBO_TEAM: ${{ secrets.TURBO_TEAM }}
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: 16

      - name: Install dependencies
        run: npm install -g pnpm@8 && pnpm install --ignore-scripts
      
      - name: Run checks
        shell: bash
        run: |
          pnpm --filter @sliit-foss/eslint-config-internal build
          pnpm ${{ matrix.command }}
        env:
          GITHUB_ACCESS_TOKEN: ${{ secrets.ACCESS_TOKEN_GITHUB }}
          FIREBASE_CONFIG: ${{ secrets.FIREBASE_CONFIG }}
```

<br/>

![Code quality + tests workflow run](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wjavej9lksrmpwbucekw.jpg)

<br/>

---

### Additional Resources/Info

- Tools

    - [Turborepo](https://turbo.build/repo)
    - [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/)
    - [Husky](https://typicode.github.io/husky/#/)
    - [Commitlint](https://commitlint.js.org/#/)
    - [Automatic Versioning](https://www.npmjs.com/package/@sliit-foss/automatic-versioning)
    - [Remote Caching](https://vercel.com/docs/concepts/monorepos/remote-caching)

---

- Interesting Links

    - [Released Packages](https://www.npmjs.com/~sliit.foss)
    - [Automation in Action](https://github.com/sliit-foss/npm-catalogue/actions)