How we reduced our nodejs monorepo build time by 70%
Tags: javascript, opensource, webdev, programming
Published: 2022-01-10 16:13:26

At [Novu](https://github.com/novuhq/novu), we use a monorepo to manage our 24 libraries and apps. There are many debates over whether you should use a monorepo or a poly-repo. For us, visibility, code sharing, standardization, easier refactoring, and a few other reasons were the critical factors for choosing this approach for our open-source notification infrastructure project.

## TLDR;
We migrated from [yarn workspaces](https://classic.yarnpkg.com/lang/en/docs/workspaces/) & [lerna](https://github.com/lerna/lerna) to [PNPM](https://pnpm.io/) and [nx.dev](https://nx.dev)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vx11afhwbk4abnqpqbsm.png)

## The bigger, the slower
With all the advantages, there are a few drawbacks to using monorepos. We noticed a particular drawback when scaling the number of packages and amount of code in each one: The **time** it takes to bootstrap the project and then build any packages within. So a typical GitHub action for a service would run anywhere between 11 to 30 minutes. And that's for each time a PR was created or a code was pushed to remote. 

More than that, installing a package locally with `yarn install` could take around 5 minutes to install and build all the dependencies.

This amount of time spent bootstrapping and building reduced the developer experience and wasted collectively so much talented people's time. Being an open-source project with a growing number of contributors, this was unacceptable.

## Debugging the slowest tasks
Inspecting a typical 12 minutes GitHub action, it was clear that two specific steps took almost 70-80% of the overall time:
- **yarn install** - takes 5-6 minutes
- **yarn build:{package}** - could take from 3-6 minutes to build the selected package and its dependencies.

## Migrating from yarn workspaces to PNPM
PNPM is a fast, disk space-efficient package manager(as stated on their website), and from some of the [benchmarks](https://pnpm.io/benchmarks), there was a massive improvement in install time against `yarn workspaces`.

Moving from `yarn install` that took around 6 minutes, the migration to pnpm was effortless: Just adding a `pnpm-workspace.yaml` to the project's root and running `pnpm install`, that's all. The symlinks and dependencies for each package we're efficiently installed, in... wait for it... **just 1.5 minutes**! And that's without any cache at all! After PNPM caches the majority of the dependencies, it takes **less than 40 seconds** to build and install the dependencies from the cached store.

Reducing ~4 minutes from the bootstrap time for every CI run and locally for first-time contributors is a HUGE win. But wait, we can do even better.

## From Lerna to NX.dev
After seeing the [Turborepo demo](Turborepo Demo and Walkthrough (High-Performance Monorepos)) by vercel, I was intrigued by their distributed caching mechanism. With such a mechanism, we can reuse the already built packages by other maintainers and download the `dist` assets instead of rebuilding them each time. 

**turborepo** vs **nx.dev**?
After brief research, we decided to go with nx.dev for multiple reasons:
- **Maturity** - nx was in the market for a while now, and they have a pretty big community around them.
- **Performance** - Seeing some of the [benchmarks](https://github.com/vsavkin/large-monorepo) nx looks like a faster build system overall.

Our community member [nishit-g](https://github.com/nishit-g) took over the open GitHub [issue](https://github.com/novuhq/novu/issues/164) and quickly after we had a PR open, the results astonished us: **30 seconds** the building step! (Instead of the previous 3-6 minutes building a specific set of packages).

After implementing the nx.cloud for distributed caching, the entire 24 packages take less than **5 seconds** when fully cached building. But even without being fully cached due to the intelligent parallelism nx performs and builds our target package in less than **30 seconds**. 

## Summary
Reducing our build times from 12+ minutes to around 3 minutes significantly impacts the developer experience of our maintainers. It also reduces the feedback loop from creating a PR to running our test suite to merging the feature.

HUGE Kudos to [nishit-g](https://github.com/nishit-g) for migrating us from Lerna to NX. Check him out on his [Twitter](https://twitter.com/sh1nXan) as well!

<hr />

You can check the final configuration on our [GitHub repository](https://github.com/novuhq/novu).