Identify and fix bundle npm duplicate dependencies
Tags: webpack, npm, webperf, frontend
Published: 2024-04-28 15:59:58

Modern web applications are becoming more complex and rely on an increasing number of dependencies. The larger the dependency tree is, the more likely it is to encounter version conflicts, bundle duplicate dependencies, ship unnecessary code, and negatively impact user performance.

On the front-end side, modern web applications include a median of **98** direct and transitive dependencies in the [bundle](#fn-1), a growth of **108%** in the last three years.

---

## Why the bundle contains duplicate dependencies

When multiple other dependencies and/or the application depend on a package, **npm** resolves the dependency tree, and if a common version does not satisfy all the semver queries, it installs multiple instances of the same package. By installing multiple instances of the same dependency, **npm** meets the dependent package.json and avoids breaking existing functionality.

If the duplicate package is a front-end dependency, it is bundled multiple times. This increases the size of the bundle, negatively impacting the application's web performance and, in some cases, causing JavaScript runtime errors (ex: [Duplicate React](https://legacy.reactjs.org/warnings/invalid-hook-call-warning.html#duplicate-react)).

Resources:
- [About semantic versioning](https://docs.npmjs.com/about-semantic-versioning)
- [semver](https://semver.npmjs.com)

---

## Example

Here's an example of how the bundle can contain duplicate dependencies. In an application that depends on `@ant-design/icons@4.7.0` (resolved to `v4.7.0`) and `antd@4.24.14`, **npm** installs two different instances of `@ant-design/icons` because `antd@4.24.14` depends on `@ant-design/icons@^4.7.0` (resolved to `v4.8.1`):

```json
{
  "name": "example-app",
  // ... package information
  "dependencies": {
    "@ant-design": "4.7.0",
    "antd": "4.24.14"
  }
}
```

<br />

```shell
npm list @ant-design/icons

example-fix-npm-dependencies@1.0.0 repo
├── @ant-design/icons@4.7.0
└─┬ antd@4.24.14
  └── @ant-design/icons@4.8.1
```

As a result, the bundle includes two versions of the same dependency:

![Bundle analysis - duplicate packages](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ruq3xxge67h93vg9hgwn.png)

## How to identify duplicate dependencies

**RelativeCI** analyzes the bundle stats modules data, extracts the list of bundled dependencies, and identifies duplicate dependency instances. It presents the information as custom insights, metrics, and a list that you can filter and sort to discover the regressions quickly.

### Duplicate packages insight

The **duplicate packages insight** shows the status and result of the duplicate packages analysis, allowing you to easily identify when there are regressions (new duplicate instances) or improvements (removed duplicates). The insight information is available on the bundle analysis report summary, job list, and all the integration summaries.


Bundle analysis - duplicate packages insight
![Bundle analysis - duplicate packages insight](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/my3yxce0zyd87c45hl6m.png)
<br />

Bundle analysis list - duplicate packages insight
![Bundle analysis list - duplicate packages insight](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o149rkc9xqgwv4s244z3.png)
<br />

GitHub Pull Request Comment - Duplicate packages insight
![GitHub Pull Request Comment - Duplicate packages insight](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2ua4j6iqkm5oi7jjmjqw.png)

### Duplicate packages metric

The number of duplicate packages is shown as a distinct [metric](https://relative-ci.com/documentation/metrics-and-data/#duplicate-packages) on the report summary, the job list, and all the integration summaries.

Bundle analysis summary - duplicate packages metric
![Bundle analysis summary - duplicate packages metric](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/r42b28pef4dqcleib6na.png)
<br />

Bundle analysis list - duplicate packages metric
![Bundle analysis list - duplicate packages metric](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dzz74p8t3mrpxyc6oded.png)


You can also use the metric as a condition on the [GitHub Commit Status Review](https://relative-ci.com/documentation/setup/configure/integrations/github-commit-status-review/#configure) or the [Slack notification](https://relative-ci.com/documentation/setup/configure/integrations/slack-notification/#send-slack-notifications-based-on-rules).

Bundle analysis review flow
![Bundle analysis review flow](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/am8h1rcbmoyla3153fqg.png)

### Duplicate packages data

To see the list of duplicate packages, go to the **Packages** section of the bundle analysis report and filter the results by the **Duplicate** filter. This will give you an overview of all the duplicate dependencies and their impact on the bundle size.

Bundle analysis - packages
![Bundle analysis - packages](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lb4dsmnrhu8prfoc2rs3.png)

## How to fix npm duplicate dependencies

The appropriate strategies for resolving **npm** duplicate dependencies may vary depending on your setup, dependencies, and versions. However, the following methods can help you eliminate duplicate packages, reduce bundle size, and enhance your application's web performance.

### Run npm dedupe

[npm dedupe](https://docs.npmjs.com/cli/commands/npm-dedupe) can help quickly fix any existing issues in the `package-lock.json` file by flattening the dependency tree and regenerating the `package-lock.json`.

For example, the app dependency depends on `@ant-design/icons@^4.6.0`, and the correct semver resolution should be `4.8.1`, though the `package-lock.json` resolves to `4.6.0`:

```shell
npm explain @ant-design/icons

@ant-design/icons@4.6.0
node_modules/@ant-design/icons
  @ant-design/icons@"^4.6.0" from the root project

@ant-design/icons@4.8.1
node_modules/antd/node_modules/@ant-design/icons
  @ant-design/icons@"^4.7.0" from antd@4.24.14
  node_modules/antd
    antd@"^4.18.6" from the root projects
```

Running `npm dedupe` will correct the `package-lock.json` and remove the duplicate dependency:

```shell
npm dedupe
npm list @ant-design/icons
example-fix-npm-dependencies@1.0.0 repo
├── @ant-design/icons@4.8.1
└─┬ antd@4.24.14
  └── @ant-design/icons@4.8.1 deduped
```

<br />

Bundle analysis - duplicate packages - fixed using npm dedupe
![Bundle analysis - duplicate packages - fixed using npm dedupe](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/inuloc82lmp6xp55i6zx.png)

- [View example GitHub pull request](https://github.com/relative-ci/example-fix-npm-duplicates/pull/4)
- [View RelativeCI bundle analysis](https://app.relative-ci.com/projects/pAhM7X9PvW10mTVYY4s2/jobs/16-30vHJN2LYGTs7aubsMEf)

### Upgrade/downgrade to a common version

Another potential solution is to update and/or downgrade the dependency to a common version that the dependents can share.
In the example above, we can update the application version to `v4.8.1`, the same as the version resolved for antd. **npm** will resolve both dependencies to v4.8.1.


Updating a package version, especially a major version, can break existing functionality. Follow the change log of the dependency for changes, and make sure to test after the update.

<br />

```shell
npm install --save @ant-design/icons@^4.8.1
example-fix-npm-dependencies@1.0.0 repo
├── @ant-design/icons@4.8.1
└─┬ antd@4.24.14
  └── @ant-design/icons@4.8.1 deduped
```

- [View example GitHub pull request](https://github.com/relative-ci/example-fix-npm-duplicates/pull/2)
- [View RelativeCI bundle analysis](https://app.relative-ci.com/projects/pAhM7X9PvW10mTVYY4s2/jobs/12-caftkDBfWyJKm4q6tQyd)


### Use npm overrides

The [npm overrides configuration](https://docs.npmjs.com/cli/v8/configuring-npm/package-json?v=true#overrides) allows you to explicitly set package versions and override the npm resolution for any dependency, direct or transitive. In our example, we can override the resolved version of `@ant-design/icons` to `4.7.0`.

Again, updating a package version, especially a major version, can break existing functionality. Remember to follow the change log of the dependency for changes, and ensure you test after updating.

<br />

```json
{
  "name": "app",
  /* ... */
  "dependencies": {
    "@ant-design/icons": "^4.6.0",
    "antd": "^4.18.6",
    /* ... */
  },
  "overrides": {
    "antd": {
      "@ant-design/icons": "4.7.0"
    }
  }
}
```

<br />


```shell
npm list @ant-design/icons

example-fix-npm-dependencies@1.0.0
├── @ant-design/icons@4.7.0
└─┬ antd@4.24.14
  └── @ant-design/icons@4.7.0 deduped
```

- [View example GitHub pull request](https://github.com/relative-ci/example-fix-npm-duplicates/pull/3)
- [View RelativeCI bundle analysis](https://app.relative-ci.com/projects/pAhM7X9PvW10mTVYY4s2/jobs/17-BOq3doE2qSSdW0UzsKQk)

### Use webpack configuration

Webpack allows dependency imports to be overridden at bundle time using the `resolve.alias` configuration option. 

Don't forget that updating a package version, especially a major version, can break existing functionality. Always follow the change log of the dependency for changes and test after the update.

To point all the app imports of `@ant-design/icons` to the same version, we can add an entry to `resolve.alias`:

```js
module.exports = {
  // ... webpack config
  resolve: {
    alias: {
      '@ant-design/icons': path.join(
        __dirname,
        'node_modules/@ant-design/icons',
      ),
    }
  }
};
```

- [View example GitHub pull request](https://github.com/relative-ci/example-fix-npm-duplicates/pull/5)
- [View RelativeCI bundle analysis](https://app.relative-ci.com/projects/pAhM7X9PvW10mTVYY4s2/jobs/18-WQzTEirA9xvau2bsKDzj)

----

<br />
<a href="#fn-1"><span id="fn-1">[1]</span></a>: Data based on open-source projects analyzed by RelativeCI
<br />
<br />

| | 2020 | 2021 | 2022 | 2023 |
|---|--:|--:|--:|--:|
| `Median package count` | `47` | `52` | `95`| `98` |
| `Mean package count` | `46` | `76` | `130`| `138` |