Inside the pain of monorepos and hoisting
Tags: node, monorepo, javascript
Published: 2021-12-13 14:13:00

---
title: Inside the pain of monorepos and hoisting
published: true
date: 2021-12-10 23:01:51 UTC
tags: node,monorepo,javascript
canonical_url: https://www.jonathancreamer.com/inside-the-pain-of-monorepos-and-hoisting/
---

 ![Inside the pain of monorepos and hoisting](https://www.jonathancreamer.com/content/images/2021/12/cave.JPG)

Working in a monorepo comes with a long list of pros, and a few cons. One of the most painful cons, when it comes to working in a specifically JavaScript based monorepo, is the pain that comes from hoisting.

## What's this hoisting thing?

![Inside the pain of monorepos and hoisting](https://www.jonathancreamer.com/content/images/2021/12/image-5.png)

As we all know, node\_modules is a deep dark place with lots and lots of stuff. That problem is even more massive in a large monorepo.

Let's take a step back for a sec and take a look at what happens when you `require` something.

### node module resolution

If you take a read through [the docs](https://nodejs.org/api/modules.html#all-together), you'll find this...

```
require(X) from module at path Y
1. If X is a core module,
   a. return the core module
   b. STOP
2. If X begins with '/'
   a. set Y to be the filesystem root
3. If X begins with './' or '/' or '../'
   a. LOAD_AS_FILE(Y + X)
   b. LOAD_AS_DIRECTORY(Y + X)
   c. THROW "not found"
4. If X begins with '#'
   a. LOAD_PACKAGE_IMPORTS(X, dirname(Y))
5. LOAD_PACKAGE_SELF(X, dirname(Y))
6. LOAD_NODE_MODULES(X, dirname(Y))
7. THROW "not found"
```

What this is basically saying is...

If you require `X`, see if it exists in node, things like `fs`, `child_process`, etc.

If you start `require('/X');` you're at the file system root.

If you `require('./X');` then see if `.X` is file, then a directory.

_...and this is the interesting one..._

If you do `require('X');` and it's not a node thing, then keep traversing the file system looking in every `node_modules` along the way.

### Hoisting

Package managers such as `yarn` and `npm` implemented a hoisting algorithm as a part of their different `workspaces` implementations.

What this hoisting does is, it scans your `package.json` files across your workspaces and figures out what the most common versions of dependencies are.

If you have 50 packages, and 47 of them are using react 16.9.0, but 3 are using 16.13.0, it'll "hoist" the common version of react, 16.19.0 to the top level `./node_modules` directory. That way you don't have to have 50 different versions of react in your project, the `import React` or `require('react')` calls will simply pull from the root `node_modules/react` one in the case of 16.9.0, or `./packages/*/node_modules/react` for the 3 cases of 16.13.0.

Voila, space saved.

However, the plot thickens...

### Monorepo structure

So, _most_ monorepo structures have a root `package.json`, and a `packages` folder.

```
./packages/foo/
./packages/foo/node_modules
./packages/bar/
./packages/bar/node_modules
./package.json
./node_modules
```

Let's say we're working with the `./packages/foo/` and it does an `import React from 'react';`.  
  
Following the node module resolution from above, it'll eventually look in the `./packages/foo/node_modules` directory. If it's _not_ there, it'll look at `../packages/node_modules/`, which definitely should not be there, and then it'll look at the `./node_modules` directory and see if it's there.

Ok, so that seems fine, where can we go wrong here?

Well, a couple of different bad things can happen here.

Let's say the `foo/package.json` has `"react": "^16.9.0"`.

Then, let's say over in `bar`, they forgot to add react in the package.json, but in `bar/src/index.ts` someone does an `import React...`, and it also happens that `bar` used a feature that only exists in the `16.13.0` version of React.

What's gonna happen? Welp, because of the fact that `bar` doesn't have a `react` in its `package.json`, the node module resolution algorithm will kick in and look up to the `../../node_modules/` directory and grab react from there.

But OH NOES, that code won't work at runtime because the code written in `bar` needs that fancy new feature in React 16.13! This is a problem that is also nicely summed up in the Rush docs as a "[Phantom Dependency](https://rushjs.io/pages/advanced/phantom_deps/)" and a "[Doppleganger](https://rushjs.io/pages/advanced/npm_doppelgangers/)".

### Real world ouchies

Here's a real world example of how this came into play in a recent release pipeline failure.

In the monorepo supported by my team, the most common version of `jest-environment-jsdom` that was hoisted to the root `node_modules` folder in the repo was `24.9.0`.

A pull request when in which added `jest-environment-jsdom` 26.10.0 to one of the packages in the `./packages` folder. Well, what happened was, there were a couple of other places using that same version across the repo so, `yarn` in its attempt to de-duplicate by hoisting decided to switch the hoisted version to `26.10.0`!

Here's where things got bad.

Let's say the package which added `jest-environment-jsdom` was called, `cool-button`. The pull request for adding the new dependency will get pushed to CI, and the CI server does a check for all the places where `cool-button` is used, and start running tests, builds, etc on the dependencies in the repo to make sure that the changes `cool-button` doesn't break any of it's downstream dependencies.

Ok, PR is green, all looks great!

Well, let's say there's another package called `cool-color-picker`. And `cool-color-picker` had some tests which was more or less like...

```
it('should pull from memory storage when localStorage isnt there', () => {
  expect(storageUtilThing.get('item')).toBeTruthy();
})
```

Well, in this crazy insane case... `cool-color-picker` _was_ relying on the hoisted `jest-environment-jsdom` version, 24.9.0. Well, that particular version used jsdom `11.11.0`. In that particular version of `jsdom` there was NO local storage support in the `jsdom` environment, so the test would pass.

However! In the new version of 26.10.0 of `jest-environment-jsdom`, the version of `jsdom` gets bumped to `16.x.x` and it just so happens every so ironically that version `11.12.0` of `jsdom` implemented localStorage. See [Changelog](https://github.com/jsdom/jsdom/blob/master/Changelog.md).

![Inside the pain of monorepos and hoisting](https://www.jonathancreamer.com/content/images/2021/12/image-6.png)

All the sudden, the test is now failing because the test had previously assumed that there was no localStorage support in jsdom without some kind of 3rd party mocking like `jest-localstorage-mock`.

What's worse here is, because `cool-color-picker` isn't in the dependency graph of `cool-button`, it never got tested as a part of the pull request.

Therefore, the change landed and attempted to go through the release pipeline.

THANKFULLY, although also painfully, the release pipeline currently does a "build the world" strategy regardless of what changed.

It's here where the release pipeline which always builds all the packages that the failure happened and broke releases for a few hours until we figured out what happened which involved LOTS of code spelunking.

![Inside the pain of monorepos and hoisting](https://www.jonathancreamer.com/content/images/2021/12/cave-dive.gif)

## So what next?

Well, in an ideal world we'll soon be switching to a "strict" package manager like the one we're currently building called [midgard-yarn-strict](https://www.npmjs.com/package/midgard-yarn-strict), which will eventually migrate much of it's feature set into the implementation of [Isolated Mode](https://github.com/npm/rfcs/pull/436) in NPM. Yarn v2 or later can also solve this problem with plug and play, or `pnpm` as well, but we're currently focusing on the former of the solutions with NPM and midgard yarn strict.

A `strict` package manager such as these options will do a couple of things.

First of all, the packages are forced to correctly declare their dependencies in the `package.json` so as to eliminate the phantom dependency / doppleganger problem.

Secondly, the root `node_modules` folder can be moved to a different location on disk, and pointed to with symlinks.

```
./packages/foo/
./packages/foo/node_modules/react -> node_modules/.store/react-16.9.0
./packages/bar/
./packages/bar/node_modules/react -> node_modules/.store/react-16.13.0
./package.json

# This breaks the node module resoution
./node_modules/.store
./node_modules/.store/react-16.13.0
./node_modules/.store/react-16.9.0
```

By moving the `node_modules` folder somewhere else, and simply symlinking all of the package's dependencies to the new location, the hoisting algorithm breaks.

So, in the previous case where the `bar` package had forgotten to declare react in its package, `bar` can no longer rely on hoisting!

Other things you can do to help are implementing a couple of solutions like [DepCheck](https://www.npmjs.com/package/depcheck) to make based on scanning your code that your dependencies are all correctly declared. There's also a tool called [TypeSync](https://github.com/jeffijoe/typesync) which does something similar for the `@types` packages to make sure they're present. The hoisting problem with`@types` packages presents a whole different set of complexities because their usage isn't in code necessarily, but only in the TypeScript setup.

## Conclusion

Hoisting helped a lot in solving _some_ problems for _many_ repos out there. However, the moment your monorepo starts to scale a bit, you'll inevitably run into these problems.