🧶YarnLocking🔓 your dependencies
Tags: yarn, webdev, javascript
Published: 2022-10-09 04:13:02

---
title: 🧶YarnLocking🔓 your dependencies
published: true
description: Yarn.lock has its own purpose and it's own problems. Almost a supply chain disruption. It's time to reveal hidden benefits.
tags: yarn, webdev, showdev, javascript
cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hddy5uppkp8rngwtmtl1.png
---

Hey mate 👋. Have you seen a `yarn.lock` file? It can be `package-lock.json`, `pnpm.lock` or `npm-shrinkwrap`, it does not matter that much. What matters - this file have a purpose.

> package-lock.json is automatically generated for any operations where npm modifies either the node_modules tree, or package.json. It describes the exact tree that was generated, such that subsequent installs are able to generate identical trees, regardless of intermediate dependency updates.
 – [A manifestation of the manifest](https://docs.npmjs.com/cli/v8/configuring-npm/package-lock-json)


In short - lock files represents and store information about dependencies other than your direct ones, and particular version of such indirect dependency to use.

{% embed https://twitter.com/devongovett/status/1573750424581963776?s=20&t=p0ext4u87XA8AOT52imAvg %}


There is no time to explain everything in the details, let me show you the problem. 

## Chapter 1 - understanding the problem
### Lets create an issue
Many of us used lock files for years and used them without any troubles. Everything seems good, but there is one nuance we need to discover. A little feature we are going to reveal. Let's run an experiment. 
_I will use `yarn` to demonstration purposes_.

> The experiment itself is for _demonstration purposes_ only. Just blindly follow the steps or magic might not happen

- __Step 1__. Create empty repository. 
 - `yarn init` and "install" it - `yarn`
 - you should get an empty `yarn.lock` file and empty `node_modules` folder
- __Step 2__. Let's add first dependency. Let it be `tslib@2.0.3`
 - `yarn add tslib@2.0.3`
 - that would create a record in `yarn.lock` like

```yaml
tslib@2.0.3:
  version "2.0.3"
  resolved "https://something-something/tslib-2.0.3.tgz"
  integrity sha512-long-and-ugly==
```
 
- __Step 3__. Let's massage our environment and __manually__ amend `package.json` and `yarn.lock` to use `tslib:^2.0.3` 
 - in `package.json`: `"tslib": "^2.0.3"`
 - in `yarn.lock`: `tslib@^2.0.3`
 - then run `yarn`
 - ⚠️ it is important to have the following record in `yarn.lock`, which means "tslib@2.0.3 used for tslib:^2.0.3" 
      - 🤔 the same will happen if 2.0.3 is the latest version available. Well, it's not the latest _now_, but was in the past
      - 💡 please __remember__ this moment as a `direct edit` of the lock file
```yaml
tslib@^2.0.3:
  version "2.0.3"
```
- __Step 4__. Add another __dependency depending__ on `tslib`
  - `yarn add focus-lock` (v0.11.2). This package depends `tslib:^2.0.3`😉. Perfect match.

> let's verify - our `yarn.lock` should contain __only two dependencies__ - `focus-lock` and `tslib`
-----

So __that's it__. We have created a problem 😎. You don't see it? Yep, the real issue is that nobody sees this problem. But it's already here.

The problem is - we have two dependencies "perfectly" matching each other.

### Is it a problem?

😉 Let's perform one more experiment.

- __Step 1__: remove `tslib`
  - `yarn remove tslib`
- __Step 2__: check your `yarn.lock`, and it will be __unchanged__, as tslib is still used by `focus-lock`
- __Step 3__: shrug 🤷‍♂️
- __Step 4__: remove `yarn.lock`
  - run `rm ./yarn.lock`
- __Step 5__: reinstall packages
  - run `yarn`

Now check your lock file 😉. It will contain the same `tslib:^2.0.3`, but this time it will be __resolved to 2.4.0__ (the last version available by the time of writing this article)
```yaml
tslib@^2.0.3:
  version "2.4.0"
```

You might not get the joke, let me rephrase it
> 😳 Existing dependencies and the order of their addition affects the shape of the end artefact 😬
-----

- In the first experiment focus-lock _reused_ preexisting `tslib`
- In the second one there will no such constrain and it used the "freshest" one.

Read it another way - any _new_ dependency you will try reuse what you already have. Any dependency you update will try to reuse what you already have. Your's legacy, your's past, your's yesterday affects your tomorrow.

> As an example - nextjs installed today and __the same version__ of nextjs installed tomorrow can have at least different version of `caniuse`, as it released on a daily schedule.

But that is just a half of the problem. The real one - you can still have _unexpectedly obsolete_ packages and hear me out - `focus-lock` by itself is using and is being tested with `tslib 2.3.1`, declaring a version below as a dependency because "it does not need any newer". Or needs, but have no idea about it, because cannot test itself agains other versions of own dependencies.

> Yep, this is exactly how I broke someone one's project - [import error: '__spreadArray' is not exported from 'tslib'](https://github.com/theKashey/react-remove-scroll/pull/70)

### Another example maybe?
The example above was a little synthetic. Ok, VERY synthetic.  Frankly speaking it took a while for me to find a way of reproducing such behaviour in a short and sound way, as _usually_ everything was working well. Without `direct edit` hack, the one I've asked you to remember, issue will be not created.
But what would be different?

Let's redo everything from scratch, but skip the hack. Depending on particular actions and events it can result in
```yaml
tslib@^2.0.3, tslib@^2.4.0:
  version "2.4.0"
```
or __TWO__ versions of `tslib`, which can be _collapsed_ into one via __deduplication__ and there is [another story about](https://medium.com/@bnaya/yarn-deduplicate-the-hero-we-need-f4497a362128) it.

Our problem is a part of duplication one. About the same version fragmentation and bloating node_modules with, but more often without a reason. 

> You might have already A LOT of duplications in your lock file - try to dedupe them if you never tried and tell me the results. At least check how many `tslib`s you've got.

> > PS: deduplication is available for any package manager, as a built-in or as a separate package. Just google.

{% embed https://twitter.com/devongovett/status/1569409158163865601 %}

### A real life example?
🤔 probably the last example was also now quite sound. Let's create something better. Something that has bitten me personally multiple times.

Let's talk about a real stuff, about consuming _intertwined_ packages.

> 1️⃣ `cool-widget` uses `cool-form` uses `cool-button` (and other packages) 
> 2️⃣ `cool-widget` uses `cool-button` as well 😎

![Modal Form](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/b215puko4y4l0qin5okp.png)

Again - this case is not applicable to simple projects. This case is not very applicable to monorepos. 
This case requires a presence of some _flexibility_ and a non-zero _distance_ between package. It needs a lag, it needs a gap, it needs a latency.
This case needs one repo consuming another to have a __gap__ in the _npm logistic layer_ in between
---

Let's create a _supply chain disruption_ ⬇️

- 1️⃣ Imagine you have a `Button`. Just a `button` exported as a npm package - `cool-button`. Every UIKit has such primitive.
- 2️⃣ and then you use that `cool-button` in many other   packages. For example in `cool-form` __and__ `cool-widget`
- 3️⃣ and `cool-widget` uses `cool-form`
  - It is a big organism consuming other molecules(form) __and atoms__ (button).
- ...
- ➡️ And then you introduce a new major version of your very `cool-button` 🙀.


The tricky part here is how "major version update" _propagates_ through packages:
- `cool-button` should have get a major version bump due to public interface change
- `cool-form` might got a patch update, as nothing changes for it's public interface. Button is implementation details and is not affecting look-n-feel of a form
- `cool-widget` actually does not _have_ to do anything....
or haven't updated all required dependencies _yet_.

----

Our `cool-widget` might be patch-updated, as it does not change public interface, just internals, and 
- it may use a new version of a `cool-button`, because some renovate-bot (or a button's developer) updated it in all consumers.
- depend on a `cool-form`, which also was just updated to use a new button, but... 😉 there is no reason for `cool-widget` to update dependency on `cool-form` as nothing actually was changed. It's the same `cool-form`.

```
In other words - dependency `cool-form` will be kept as is
```

What 😕? How so? Why not updated? 

Let's stop here and check if cool-widget have to react and incorporate every single move of every single piece. Any reason?

🤷‍♂️ Why someone should be bothered to bump every `dependency` in package.json every time every `dependency `, a little spare part of a bigger whole, updates? It could be ten times a day for ten packages, and it sounds like a lot of useless work! Especially some work is required to perform update, run test, verify the result, deploy the result. Oh

Ok, ok. Let's imagine you are fine with it, You want it. So a `button` update will cause:
- 1️⃣ a cascade update of ALL packages depending on it, 
- 2️⃣ and then of all packages depending on just changed 
- 3️⃣ and then of all packages depending on just changed
- 4️⃣ and then of all packages depending on just changed
- 5️⃣ should I stop here?

It actually can be a loop - _sometimes_ packages might depends on previous versions of themselves. Like `babel` uses `babel` to compile self.
So 😕 I really hope you don't want that. You probably do update in batches, or not really doing it at all, and that's ok - if stuff does work, just don't touch it.

----

Long story short - `cool-widget` have got a new `cool-button@v2` directly updated, but still have the old `cool-button@v1` from the _kept-as-is_ `cool-form`

So now we have two buttons, with different major versions, and this cannot be _deduplicated_.
----

![Ball of yarn](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fvlfttexer7iijqs6hjs.jpg)

> 😠: _"HEY! Wait a minute. I've got you and I am promising to be a good lad and keep my deep dependencies up to date!"_
> 😿: _"Unfortunately this is not about you, this is __about packages created by someone else__, about the packages you don't control"_
> 😕: _"I dont control?"_
> 😾: _"You don't"_

You have got a problem with __stale intermediate dependency__. A dependency of your dependency which you don't directly control. And a dependency [of your dependency of your dependency]. And a dependency [of your dependency of your dependency of your dependency]. And ....

And that is __the problem we were chasing__ all this time.


> First time this issue was encountered back in 2017 – [Yark: How to upgrade indirect dependencies?](https://github.com/yarnpkg/yarn/issues/4986)


## Chapter 2 - Solving the problem

It's an easy job to point a finger on a problem and tell everyone - _"People! Here is the problem"_. 
It is completely another job to propose a solution.

### Going back and upgrading

Lets make another step back and look on what you do control - the direct dependencies:
- you can declare dependencies you need and their versions
  - as their versions might represent a wide range - the particular versions to use will be stored in your `lock file`
  - if by any reason you get a newer version of already existing dep - it will be automatically _raised_ to the highest one. If not automatically, then after _dedupe_.
  - 😅 so by installing a new dep you can change dependencies of your existing? 
     - This is why changes in `yarn.lock` might need a review
- you can manage and especially update those deps.
  - use `yarn up`, or `yarn upgrade-interactive`
  - or use a little bit more advanced [npm-check-updates](https://github.com/raineorshine/npm-check-updates)
  - in both cases only `package.lock` gonna be changed

This is how you can control __direct__ dependencies. The ones explicitly declared in your `package lock`. You have no control over indirect dependencies totally _derived_ from requirements of the _real ones_.

Package managers are trying to preserve as much as possible in order to reduce the impact of the change and amount efforts users might need to perform in order to manage consequences. Known as __Principle of least action__.

Package manager's job is to keep things. Our goal is to _loosen some bolts...


Please welcome...
# Yarn-unlock-file

[yarn-unlock-file](https://github.com/theKashey/yarn-unlock-file) is a new library, capable of editing your lock file and addressing issues created by indirect dependencies 

The simplest use case is nothing more than
```bash
npx yarn-unlock-file all
# and don't forget to
yarn
```
That is it - the command will:
- 🔐 keep any of dependencies declared by you
- 🔓 "unlock" all indirect ones
  - "unlock" means "delete" old records from `yarn.lock` letting the newer, might be more correct versions to be installed

### What you are dependencies?

There is __one thing we forgot__ – the purpose of package managers. What is `yarn`, what is `npm` and what they do...

PackageManagers are primarily managing `logistics` - they provide transport layer between `npm repository` and your `project`.
Then PackageManagers are in charge to recreate the same environment, the same combination of packages across different machines to make our live more _predictable_.

However, if you use a library, you probably _rely_ on it to be tested and behave well, but this is possible __only__ if the library was _assembled_ in your projects in exactly same way it was built and tested in the place of the origin - it's own space.

> So it's important to _let free_ artificial boundaries preventing the similar environments

Everything has limits, and a perfect tool capable to capture and _freeze_ your decisions in time might need a hand.

No tool should change your decisions and change package versions picked by you, but it might help you manage decisions you haven't made consciously - concrete versions for your indirect dependencies.

Try commands like
- `npx yarn-unlock-file dev` to update only devDependencies, or
- `npx yarn-unlock-file matching @material-ui/*` to update dependencies of `material-ui` __but not__ MUI itself

## Safety first
There is a reasons why we are not just deleting `yarn.lock` - the result can be unpredictable. Or it can be "too much" - you will get a broken build or a broken application if your tests are not great in detecting anomalies.

- One should bumping your dependencies regularly, but it's not always safe and can go unpredictable in large projects.
  - One might update versions in the same range, and "unlocking" is a good way to do that.
- One might still be cautions about updating direct dependencies, or even dependencies of dependencies. But not many layers is there?
  - `npx yarn-unlock-file levels all` will tell you how much.     
      - I haven't seen more than 8. 
      - I haven't seen less than 3.
 - 😎 updating dependencies of dependencies of dependencies should be safe
     - `npx yarn-unlock-file all --min-level 3` 

#### Dry run
Well, why one should blindly trust some tool to delete only what you want. Trust, but verify. Use dry run mode.
`npx yarn-unlock-file all --min-level 3 --dry-run`
![memoize-one unlock](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fbees0trv1zphzl3qu9y.png)

The picture above is a good indication of how many layers can be hidden inside a simple solution, [memoize-one](https://github.com/alexreardon/memoize-one) in this case.

One might ask the question - why level 8 dependency `is-arraysh` is here, and what causes it?
- lets ask: `yarn why is-arrayish`
> nyc#test-exclude#read-pkg-up#read-pkg#load-json-file#parse-json#error-ex#is-arrayish
- let's update only this branch: `npx yarn-unlock-file matching nyc --dry-run` 

![remove nyc](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/41wp9uwvfak6qes2ehlr.png)

1️⃣ look 2️⃣ pick `min-level` you are comfortable with 3️⃣ wipe


#### Note about Npm-check-Updates
[ncu has a "doctor" mode](https://github.com/raineorshine/npm-check-updates#doctor-mode) to update dependency, run tests, and only then update another dependency.
While this works just great - it actually can _tangle_ your lock file beyond imagination because packages you install today affects the shape of package you install tomorrow 😉

Having more control over this process is quite beneficial.

------

- For years I was giving an advice of _"just find something in your lock file and delete"_.

- For years I was looking into `yarn.lock` updates to correct result of deduplication sometimes by selecting a few hundred of lines to _delete and regenerate_.
 
🤷‍♂️ enough is enough. It's time to understand that deterministic builds is one problem, but _supply chain disruption_ is different. 

-----

Authors of packages and tools you use _expect and test_ their creations  using some know (to them) set of other packages and tools, which are in turn rely on some other set of libraries. Any dependency specified within an accepted "range", and most of them are, is a subject for version mismatch. Or an _expectation_ that the end consumers will use a new patch version of a tiny library with something very important fixed.

There are dependencies you know, the ones you want to use.
Everything else has to be unlocked on a weekly basic.

You can use Renovate, Dependabot or `npm-check-updates`
{% embed https://twitter.com/sitnikcode/status/1569310121880350722 %}


But it's not enough
https://github.com/theKashey/yarn-unlock-file

> PS: Did you know that `dependabot` can update dependencies in your lock.file? Like [here it bumps terser](https://github.com/theKashey/react-focus-lock/pull/223) changing only lock file, because it's an indirect dependency.

# So what?
Just try it
```bash
npx yarn-unlock-file dev
yarn
```
And then we will see 😈

💡: this is a first, partially experimental solution, currently capable to handle only yarn(v1 and v3).