How does pnpm work
Tags: javascript, npm, frontend, programming
Published: 2023-03-31 15:40:03

As one of the very popular package managers, pnpm is mainly characterized by fast speed and saving disk space. I will introduce how pnpm works to help you understand the principle of pnpm.

## Introduction
The meaning of pnpm is performant npm. From the benchmarks in pnpm website, we can see in many scenarios pnpm has good performance advantages compared with npm/yarn/yarn_pnp.

![benchmarks](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bpvikrh2dvdpw2emkcfy.png)



## Directory structure of node_modules

### Nested structure
In earlier versions of npm@2, corresponding to Node.js 4.x and previous version, node_modules is a nested structure while installing.

A simple case [here](https://github.com/lvqq/blog-samples/tree/master/pnpm/01-npm%402), both demo-foo and demo-baz depend on demo-bar. When demo-foo and demo-baz are installed in the same repository, we can get the following node_modules structure:

```
node_modules
└─ demo-foo
   ├─ index.js
   ├─ package.json
   └─ node_modules
      └─ demo-bar
         ├─ index.js
         └─ package.json
└─ demo-baz
   ├─ index.js
   ├─ package.json
   └─ node_modules
      └─ demo-bar
         ├─ index.js
         └─ package.json
```

Although the directory structure is relatively clear at this time, each dependent package will have its own node_modules directory, and the same dependency has not been reused. For the above example, the same dependency demo-bar has been installed twice.

Another problem is the [Maximum Path Limitation](https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry) of windows. In some complex cases when the project has a deep dependency level, the dependent path often exceeds the length limit.

### Flat structure
In order to solve the above problems, yarn proposed a flat structure design: flattening all dependencies in node_modules. And the implementation of the later npm v3 version is similar, so use yarn or npm@3+ to install the above example, you will get the following flat directory structure:
```
node_modules
└─ demo-bar
   ├─ index.js
   └─ package.json
└─ demo-baz
   ├─ index.js
   └─ package.json
└─ demo-foo
   ├─ index.js
   └─ package.json
```

In addition, for different versions of the same dependency in this way, only one of them will be hoisted, and the remaining versions will still be nested in the corresponding packages. 

For [instance](https://github.com/lvqq/blog-samples/tree/master/pnpm/03-npm-doppelgangers), if we upgrade the above demo-bar to v1.0.1 (its dependency demo-foo is also v1.0.1), you will get the following structure, which version will be hosited to the top depends on the order of installation:
```
node_modules
└─ demo-bar
   ├─ index.js
   └─ package.json
└─ demo-baz
   ├─ index.js
   ├─ package.json
   └─ node_modules
      └─ demo-bar
         ├─ index.js
         └─ package.json
└─ demo-foo
   ├─ index.js
   ├─ package.json
   └─ node_modules
      └─ demo-bar
         ├─ index.js
         └─ package.json
```

### Problems with flat structures
The flat solution is not perfect, and accompanies with some new problems:

#### Phantom dependencies
Phantom dependencies means the dependencies that are not declared in package.json but can be directly used in your project. This issue is caused by the flat structure, and the dependencies of the dependencies will also be hoisted to the top level of node_modules, so that you can reference it directly in the project. And if some day this sub-dependency is no longer a dependency of the reference package, there will be problems with the references in the project.

For example, in the project containing demo-foo and demo-baz as the dependencies, demo-bar also appears in node_modules as a dependent dependency:
```
node_modules
└─ demo-bar
   ├─ index.js
   └─ package.json
└─ demo-baz
   ├─ index.js
   └─ package.json
└─ demo-foo
   ├─ index.js
   └─ package.json
```

#### Npm doppelgangers
NPM doppelgangers refer to different versions of the same dependency, due to the hoist mechanism, only one version will be hoisted, and other versions may be installed repeatedly. The same [example](https://github.com/lvqq/blog-samples/tree/master/pnpm/03-npm-doppelgangers) as above, when we upgrade demo-bar to v1.0.1, the 1.0.0 version that demo-baz and demo-foo depends on will be repeatedly installed in a nested way:
```
node_modules
└─ demo-bar // v1.0.1
   ├─ index.js
   └─ package.json
└─ demo-baz
   ├─ index.js
   ├─ package.json
   └─ node_modules
      └─ demo-bar // v1.0.0
         ├─ index.js
         └─ package.json
└─ demo-foo
   ├─ index.js
   ├─ package.json
   └─ node_modules
      └─ demo-bar // v1.0.0
         ├─ index.js
         └─ package.json
```

## How to solve
First of all, pnpm installs the dependencies to the global store, and then use symbolic link and hard link to organize directory structure. It links the global dependencies into the project, and links the direct dependencies into top level of node_modules directory. All dependencies are flattened under the `node_modules/.pnpm` directory, which realizes the global dependencies shared of all projects, and solves the problem of phantom dependencies and NPM doppelgangers.

### Symbolic link and hard link
Link is the way of file sharing in the operating system, where the one is symbolic link, also known as soft link, and the onther one is hard link. From the point of view of use, there is no difference between them, both supporting reading and writing, and if it's an excutable file, it can also be excuted directly. The main difference is the core principle:
![Symbolic link and hard link](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jn78bdgkpj82kdmil95s.png)


#### Hard link
- Hard links don't create a new index node, the source file and the hard link point to the same index node
- Hard links don't support directories, only file level, and don't support cross disk partitions
- The file is not actually deleted until the source file and all hard links are deleted

#### Symbolic link
- The symbolic link stores the path of the source file, pointing to the source file, similar to the shortcut of Windows
- The symbolic link supports directories and files, which are different files from source files. It has different inode values and different file types, so the symbolic link can be accessed across partitions
- After deleting the source file, the symbolic link still exists, but the source file cannot be accessed through it

#### How to create links through command
```bash
# symbolic ink
ln -s myfile mysymlink

# hard link
ln myfile myhardlink
```

### How does pnpm work
At first, pnpm will installs the dependencies into `<home dir>/.pnpm-store` in the current partition. The current store location can be obtained by the following command：
```bash
pnpm store path
```

And then hard link the required packages from `node_modules/.pnpm` into the store path. Finally, symbolically link the top-level dependencies and dependent dependencies in node_modules to `node_modules/ .pnpm`, an [example](https://github.com/lvqq/blog-samples/tree/master/pnpm/04-pnpm) that depends on demo-foo@ 1.0.1 and demo-baz@ 1.0.0, the node_modules structure is:
```
node_modules
└─ .pnpm
   └─ demo-bar@1.0.0
      └─ node_modules
         └─ demo-bar -> <store>/demo-bar
   └─ demo-bar@1.0.1
      └─ node_modules
         └─ demo-bar -> <store>/demo-bar
   └─ demo-baz@1.0.0
      └─ node_modules
         ├─ demo-bar -> ../../demo-bar@1.0.0/node_modules/demo-bar
         └─ demo-baz -> <store>/demo-baz
   └─ demo-foo@1.0.1
      └─ node_modules
         ├─ demo-bar -> ../../demo-bar@1.0.1/node_modules/demo-bar
         └─ demo-foo -> <store>/demo-foo
└─ demo-baz -> ./pnpm/demo-baz@1.0.0/node_modules/demo-baz
└─ demo-foo -> ./pnpm/demo-baz@1.0.1/node_modules/demo-foo
```

Here is a screenshot of the pnpm website to help you better understand how symbolic link and hard link are organized in the project structure:

![Directory structure](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/va6n4hnc6pjxc85t3lez.png)


For the actual usage of the link in pnpm, the following is the relevant [source code](https://github.com/pnpm/pnpm/blob/main/fs/indexed-pkg-importer/src/index.ts#L19~L43):
```typescript
function createImportPackage (packageImportMethod?: 'auto' | 'hardlink' | 'copy' | 'clone' | 'clone-or-copy') {
  // this works in the following way:
  // - hardlink: hardlink the packages, no fallback
  // - clone: clone the packages, no fallback
  // - auto: try to clone or hardlink the packages, if it fails, fallback to copy
  // - copy: copy the packages, do not try to link them first
  switch (packageImportMethod ?? 'auto') {
  case 'clone':
    packageImportMethodLogger.debug({ method: 'clone' })
    return clonePkg
  case 'hardlink':
    packageImportMethodLogger.debug({ method: 'hardlink' })
    return hardlinkPkg.bind(null, linkOrCopy)
  case 'auto': {
    return createAutoImporter()
  }
  case 'clone-or-copy':
    return createCloneOrCopyImporter()
  case 'copy':
    packageImportMethodLogger.debug({ method: 'copy' })
    return copyPkg
  default:
    throw new Error(`Unknown package import method ${packageImportMethod as string}`)
  }
}
```

## Other abilities
At present pnpm can be installed and used out of Node.js runtime. And pnpm can also manage Node.js version through pnpm env, similar to nvm. For a full feature comparison with npm/yarn see [feature-comparison](https://pnpm.io/feature-comparison)

## Limitations

- Due to compatibility issues with symbolic links in certain scenarios, pnpm cannot currently be used on applications deployed on Electron and Lambda, as outlined in: [discussion](https://github.com/nodejs/node/discussions/37509)


By adding `node-linker=hoisted` to .npmrc, a flat node_modules directory without symbolic links can be created, which is similar to the directory structure created via npm/yarn.

- As the same store is shared globally, modifying the contents within node_modules will directly affect the corresponding content in the global store, which will also have an impact on other projects.

For the above issue, the most recommended approach is to use clone([copy-on-write](https://en.wikipedia.org/wiki/Copy-on-write)). By default, multiple references point to the same file, and only when the user needs to modify it, a copy is made so that it will not affect other references that are reading the content of the source file.

However, not all operating systems support this. By default, pnpm attempts to use clone. If it is not supported, it falls back to using hard link. You can also manually set the package reference method by specifying [package-import-method](https://pnpm.io/npmrc#package-import-method) in your .npmrc configuration file.

- For other limitations, see: [https://pnpm.io/limitations](https://pnpm.io/limitations)

## Other tools

- bun: https://github.com/oven-sh/bun
    - bun is a JS runtime written in Zig. It also offers a package management tool, but it may encounter some compatibility issues.


-   Volt：https://github.com/dimensionhq/volt
    - volt is a Node.js package manager written in Rust, which is known for its extremely fast performance. It is currently in beta, but it has not been updated for several months.

-   tnpm: https://github.com/cnpm/npminstall
    - tnpm implements a virtual mapped directory by utilizing the features of Filesystem in Userspace([FUSE](https://en.wikipedia.org/wiki/Filesystem_in_Userspace)) and [OverlayFS](https://en.wikipedia.org/wiki/OverlayFS).


## Reference
- [Flat node_modules is not the only way](https://pnpm.io/blog/2020/05/27/flat-node-modules-is-not-the-only-way)
- [Symlinked node_modules structure](https://pnpm.io/symlinked-node-modules-structure)
- [In-depth of tnpm rapid mode - how we managed to be 10 second faster than pnpm](https://dev.to/atian25/in-depth-of-tnpm-rapid-mode-how-could-we-fast-10s-than-pnpm-3bpp)
- [Complete code samples](https://github.com/lvqq/blog-samples/tree/master/pnpm)