Why aren't Node.js package managers interoperable?
Tags: node, dependencymanagement
Published: 2022-10-07 21:13:25

This post was inspired by [this Twitter conversation](https://mobile.twitter.com/mykeels/status/1574871809887027203).

The tweet asserts:

> I should be able to use pnpm while my teammates use yarn or npm, without issues.

This is a real concern. I've worked on some Yarn projects where I sometimes forget and accidentally run npm. Although these package managers \(I'll call them PMs for short\) _are_ interoperable to some degree, there are important differences between these tools, so you should know what you're getting into. Even in seemingly simple projects, results can vary. Here are [two](https://github.com/alex3165/react-mapbox-gl/issues/578) [reports](https://github.com/doczjs/docz/issues/276) of things working with one PM, but not another. Moreso, running `install` once is very different from continuous concurrent use, where you're adding and removing packages and more.

Let's talk about the key things that make these package managers different, and then we'll see the extent to which they're interoperable. We'll also see how things are getting better, thanks to the work of some smart folks.

For this article, we'll pretend there are only four package managers: npm \(v7\+\), pnpm \(v6\+\), Yarn v1 \(aka Classic\), and Yarn v2\+ \(aka Berry, aka Modern\).

# Key differences

## Features

This is the most obvious difference, but also the least significant if your priority is portability. Each package manager has a unique set of features \(or the same feature, expressed in different ways\). For instance:

* Yarn 2\+ introduced \(optionally\) [Plug'n'Play](https://yarnpkg.com/features/pnp), which gets rid of `node_modules` altogether, and expects you to run files with `yarn node` instead of just `node`.
* Yarn 2\+ and pnpm support patching dependencies with your custom code. For npm and Yarn 1, you can use [a library](https://www.npmjs.com/package/patch-package), but remember to somehow ignore it when using other PMs.
* pnpm lets you write [custom functions](https://pnpm.io/next/pnpmfile) to hook into the install process.
* Yarn's `resolutions` is `overrides` in npm and `pnpm.overrides` in pnpm. You'd need to have all three in your `package.json`, if you want to make this portable.
* All four support workspaces \(aka monorepos\), but to different levels: [Yarn 1](https://classic.yarnpkg.com/lang/en/docs/workspaces/) and [npm](https://docs.npmjs.com/cli/v8/using-npm/workspaces) have a basic set of features, while [Yarn 2\+](https://yarnpkg.com/features/workspaces) and [pnpm](https://pnpm.io/workspaces) are more robust and compatible, although configured differently. Using the wrong package manager will almost certainly give you wrong results.

But maybe you don't care about these extra features. Your project isn't big or complex, and your dependencies are few \(in JavaScript? I doubt that, but okay.😆\) and manageable, so you're willing to stick to the minimum supported feature set. You mostly just want to be able to add and remove packages. But you still have to watch out for some behaviour differences.

## Behaviours and configuration

Being different tools, they have different philosophies and defaults.

* **Configuration**: npm and pnpm use `.npmrc` files for configuration. Yarn 1 merges `.npmrc` and `.yarnrc`. Yarn 2\+ uses only `.yarnrc.yml`.
* **Installing dependencies**: `npm install` will check for newer versions of your dependencies, while Yarn's will not. Yarn's `install` is closer to npm's `ci` \(and, in fact, Yarn doesn't have a `ci` command\).
* **Upgrading dependencies**: Yarn 1's `yarn upgrade`[only upgrades direct dependencies](https://github.com/yarnpkg/yarn/issues/4986) of the current workspace. Yarn 2's `up` ignores the version ranges in your `package.json` and upgrades for all workspaces. npm's and pnpm's`update` respect your version ranges and upgrade indirect dependencies as well.
* **Adding dependencies**: I couldn't find any docs on this, but I noticed that `yarn add package@version` sometimes adds `package` to your `package.json` at that exact version, while `npm install` sets the constraint to `^version`.
* **Dependency resolution**: While the resolution algorithms \(going from a list of packages in `package.json` to a fully resolved dependency tree\) are converging, they aren't always the same across packages. The npm blog has [an example](https://blog.npmjs.org/post/621733939456933888/npm-v7-series-why-keep-package-lockjson.html#:~:text=in%20source%20control.-,Nesting%20and%20Deduplication,-Furthermore%2C%20there%20is) of this.

Finally, you have to contend with the lockfiles and output.

## Lockfiles and output

It's probably well\-known that these PMs use different lockfiles with different formats: JSON \(`package-lock.json`\), YAML \(`pnpm-lock.yaml`\), and uh, YAML\-like \(`yarn.lock`\). But more important is the information contained within these lockfiles, which isn't always equivalent.

Here's a small experiment. We'll use a dependency tree that looks like this:

```sh
# Our app (the root) and its dependencies
root -> (blerg@^1.2.5, bar@^1.2.3, baz@^1.2.3)

# Available packages and their dependencies
blerg@1.2.5 -> ()
bar@1.2.4 -> (blerg@^1.3.4, baz@2.x, asdf@*)
baz@1.2.3 -> (quux@3.x, bar@^1.0)
baz@2.x -> (quux@3.x)
asdf@2.3.4 -> ()
quux@3.x -> (bar@*)
```

Now we'll try installing this with each of the PMs.

Yarn's lockfile is quite easy on the eyes. Here's a snippet:

```php
"@shalvah/asdf@*":
  version "2.3.4"
  resolved "https://registry.yarnpkg.com/@shalvah/asdf/-/asdf-2.3.4.tgz#f37231e5f142b994a5226227d4279aff041bebd0"
  integrity sha512-pHBtQbOWS+aW+B3AKMT/25pFAum06BAv8OGo4I3nL1ezL/Dqm6z0nZyDqD76t9MkGngJ/HBpH+IyVqD3D6RfqA==

"@shalvah/bar@*", "@shalvah/bar@^1.2.3":
  version "1.2.4"
  resolved "https://registry.yarnpkg.com/@shalvah/bar/-/bar-1.2.4.tgz#efa1ab4960f79edfcb8d029c3d0d295de88258bc"
  integrity sha512-1NjVgWLKhFpkIPU6TihPQ22N5GQ/U3sGsIyzbj1tbgPpyQeZ8tM59rGuHYtVArg/LYBorPxN84CS6lz9qdqR0w==
  dependencies:
    "@shalvah/asdf" "*"
    "@shalvah/baz" "2.x"
    "@shalvah/blerg" "^1.3.4"

"@shalvah/baz@2.x":
  version "2.0.2"
  resolved "https://registry.yarnpkg.com/@shalvah/baz/-/baz-2.0.2.tgz#f5f092f10088eab2522ccf4df2e64bebdcb152bc"
  integrity sha512-xz7oyoS2Jk1oYPtlHfItIb7gXTNQ8S2v2RVGLqyMXOo24hpL9+Ee5uFIl1o1NcC9wRY2G8fmRlXrJizFRFwxrQ==
  dependencies:
    "@shalvah/quux" "3.x"
```

The Yarn lockfile describes the "logical" dependency tree Yarn has resolved from your package.json. It's a map of entries, where each key is one or more version specifiers Yarn encountered in your dependencies, and the value contains details of the package they resolve to. In this example, this lockfile declares that whenever `@shalvah/asdf` is requested in the version `*`, it should be resolved to version `2.3.4`, fetched from that URL and verified with that integrity hash. The v2 lockfile adds a few more details, but is substantially the same.

pnpm's lockfile, like Yarn's, stores only the resolutions, but in a slightly different way, as it's not limited by hoisting constraints.

![](https://cdn.swish.ink/caf2243d-94c4-49d1-9204-11e4c63a6b5f/media/pnpm_lock "pnpm_lock")

npm's lockfile \(as of v7\) contains not just the resolutions, but the "physical" dependency tree \(ie the exact layout of your `node_modules`\). This makes the lockfile bigger, but also self\-contained: npm can recreate the entire `node_modules` using only the lockfile, while Yarn's lockfile [can produce different results on disk](https://classic.yarnpkg.com/blog/2017/05/31/determinism/) if different Yarn versions are used.

![](https://cdn.swish.ink/caf2243d-94c4-49d1-9204-11e4c63a6b5f/media/package_lock "package_lock")

Note: the important part here is the `packages` node. The `dependencies` node only exists for backwards compatibility with the older lockfile version.

Speaking of `node_modules`, not every PM uses them in the same way \(or at all\)\! Yarn 1 and npm generate a "flat" node\_modules \(all your dependencies' dependencies are hoisted to the top\-level as much as possible, in order to reduce duplication\). In most cases, the two tools produce the same output by default \(but not always\).

![](https://cdn.swish.ink/caf2243d-94c4-49d1-9204-11e4c63a6b5f/media/node_modules "node_modules")

pnpm installs packages globally and then creates a \(non\-hoisted\) node\_modules that consists entirely of symlinks to these packages.

![](https://cdn.swish.ink/caf2243d-94c4-49d1-9204-11e4c63a6b5f/media/pnpm_node_modules.png "pnpm_node_modules.png")

Yarn 2\+'s PnP linker \(although not the only option\) does not use a `node_modules` folder at all, but instead a different folder, `.yarn/cache`, where each package is stored as a single ZIP file, and encourages you to commit them. For completeness, this is what the folder looks like in this case:

![](https://cdn.swish.ink/caf2243d-94c4-49d1-9204-11e4c63a6b5f/media/yarn_cache "yarn_cache")

# Efforts in interoperability

But all hope isn't lost. The various tools have made some effort to be compatible with some others, to some degree:

## Lockfiles

* Using npm on a project with a `yarn.lock`[will update the `yarn.lock` file alongside the `package-lock.json`](https://blog.npmjs.org/post/621733939456933888/npm-v7-series-why-keep-package-lockjson.html): "In npm v7, if a yarn.lock file exists, npm will use the metadata it contains\[...\]. If packages are added or removed, then the yarn.lock file will be updated."
    I was pleasantly surprised to find out that this worked \(minus [a few bugs](https://github.com/npm/cli/issues/5126)\). Running `npm remove x` removed it from `package.json`, `package-lock.json`, and `yarn.lock`. Even more impressive, removing/adding packages by manually editing `package.json`, then running `npm i` also updated the Yarn lockfile. Well done, npm team.
* [`pnpm import`](https://pnpm.io/cli/import) will import your dependencies as\-is from a Yarn or npm lockfile, but this is a one\-time operation; the other lockfile won't be updated in tandem.
* Yarn v1 also has a one\-time [`import`](https://classic.yarnpkg.com/en/docs/cli/import). It's probably not technically feasible for Yarn to update the `package-lock.json`, considering that `package-lock.json` also stores the physical node\_modules tree, which Yarn doesn't do.
* Yarn 2\+ isn't directly compatible with other lockfiles, and `import` doesn't exist here.

## Linkers

* Yarn 2 supports various ["linkers"](https://yarnpkg.com/configuration/yarnrc#nodeLinker). While the default is PnP, the traditional \(npm\-style, flat\) `node_modules` is also supported. And from v3.1, so is `pnpm` \(symlinked, non\-flat `node_modules`\).
* npm also [plans to](https://github.com/npm/rfcs/blob/main/accepted/0042-isolated-mode.md) support pnpm\-style node\_modules
* pnpm [supports](https://pnpm.io/blog/2020/10/17/node-modules-configuration-options-with-pnpm) Yarn's PnP and npm\-style flat `node_modules`

That said, I wouldn't recommend relying on these compatibility features. One PM's implementation of another's feature might have minor differences in order to fit in with the PM's design and paradigms. For example, pnpm's flat node\_modules version still uses global packages and symlinks, so it's not likely to work well with actual npm output.

## Corepack

Since version 16.9, Node.js has included [Corepack](https://nodejs.org/api/corepack.html). \(You can also install Corepack manually on earlier versions.\) Corepack lets you use any of these PMs without explicitly installing them. In fact, my experiments with pnpm while writing this post were via Corepack \(by using `corepack pnpm`\). I don't have pnpm installed on my machine.

```sh
➜ pnpm -v
pnpm: The term 'pnpm' is not recognized as a 
name of a cmdlet, function, script file, or executable program.
Check the spelling of the name, or if a path was included, 
verify that the path is correct and try again.

➜ corepack pnpm -v
7.5.2
```

There's a [guide to it](https://dev.to/cloudx/corepack-the-node-js-manager-of-package-managers-44dd), but I'll be honest, I haven't wrapped my head around that yet. I just ran these commands, and it worked.

# So what can you do?

The best approach is probably to require one package manager for your project. But you can do some things to make it less of a pain for other contributors.

## `packageManager`

With Yarn \(v2\+ _and_ v1\), you can run [`set version`](https://yarnpkg.com/cli/set/version) to lock the project to a specific Yarn version. It will download the Yarn release and include it in the repo, as well as add the new [`packageManager` field](https://nodejs.org/api/packages.html#packagemanager) in `package.json`, set to `"yarn@<specified-version>"`.

[`packageManager`](https://nodejs.org/api/packages.html#packagemanager) is a new field for `package.json`, intended to define and enforce what package manager and version is used in a project. Corepack respects this; when I try to run `corepack pnpm` in this case, I get:

```sh
➜ corepack pnpm
Usage Error: This project is configured to use yarn

$ pnpm ...
```

Changing the version \(but keeping it as Yarn\) seemed to make Corepack try to auto=download that version.

Ironically, in my tests, Yarn itself did not seem to respect this field. When I changed the `packageManager` to other versions of Yarn, or even to npm, Yarn happily continued to work. No other package managers support it at the time of writing, so right now, it's only useful if you're running all your commands through Corepack.

## `engines`

A workaround is to use the older `engines` field in `package.json`. All four PMs support this field as a way to define the version of the PM the project needs, but once again, it's a mixed bag. They don't check for the existence of other PMs, just that the version you put in the `engines` field matches the installed version. However, we can leverage this by using a nonexistent version as the requirement for the PMs we don't want. For example, with this `package.json`:

```json
{
  "engines": {
    "node": ">= 16.9",
    "npm": "999999999.0",
    "pnpm": "999999999.0",
    "yarn": ">= 1.22"
  }
}
```

pnpm and Yarn will raise an error. Unfortunately, npm will only warn you, so you need to set `engine-strict=true` in the `.npmrc` as well. Here's what the error messages look like:

```sh
➜ npm i --engine-strict
npm ERR! code EBADENGINE
npm ERR! engine Unsupported engine
npm ERR! engine Not compatible with your version of node/npm: undefined
npm ERR! notsup Not compatible with your version of node/npm: undefined
npm ERR! notsup Required: {"npm":"9999999999.0"}
npm ERR! notsup Actual:   {"npm":"8.14.0","node":"v16.14.0"}

➜ yarn
yarn install v1.22.19
warning package-lock.json found. Your project contains lock files generated by tools other than Yarn. It is advised not to mix pack
age managers in order to avoid resolution inconsistencies caused by unsynchronized lock files. To clear this warning, remove package-lock.json.
[1/5] Validating package.json...
error foo@: The engine "yarn" is incompatible with this module. Expected version "9999999999.0". Got "1.22.19"
error Found incompatible module.

➜ corepack pnpm install
 ERR_PNPM_UNSUPPORTED_ENGINE  Unsupported environment (bad pnpm and/or Node.js version)

Your pnpm version is incompatible with "C:\Users\shalvah\Projects\Temp\fake-packages".

Expected version: 999999999.0
Got: 7.5.2

This is happening because the package's manifest has an engines.pnpm field specified.
To fix this issue, install the required pnpm version globally.

To install the latest version of pnpm, run "pnpm i -g pnpm".
To check your pnpm version, run "pnpm -v".
```

## Corepack

With Corepack, you wouldn't have to tell folks to install a new package manager. You could prefix all the PM commands in your docs with `corepack `, so users can get productive right away. It still has some rough edges, but combining either of the above with Corepack should be a good starting point.

## Additional tooling

If this is a big enough problem for your team, you can even throw in some extra tools to help:

* [preferred\-pm](https://www.npmjs.com/package/preferred-pm) and [which\-pm](https://www.npmjs.com/package/which-pm) can be used to detect a project's package manager \(although I don't 100% agree with the detection rule order\).
* VS Code's `npm.packageManager` setting can be handy if your team uses VS Code, but it won't affect normal terminal commands.