Tips from open-source: Use “Set” to remove duplicates from an array.
Tags: javascript, nextjs, opensource, node
Published: 2024-04-11 20:58:55

This tip is picked from [Next.js source](https://github.com/vercel/next.js/blob/canary/packages/next/src/export/index.ts#L473). In this article, you will learn how to remove duplicates from an array using Set in Javascript.

![Set example](https://media.licdn.com/dms/image/D4E12AQHrHmbtbX5fgA/article-inline_image-shrink_1500_2232/0/1712868150694?e=1718236800&v=beta&t=XfpZ7JZfvuIikh4P0mzSaaGbR7UqiilDRcLIztO9-ic)

From this comment above in the picture, I noticed you could remove duplicates from an array using Set.

You can execute the code below as an example to see this in action:

> [_Learn the best practices used in open source_](https://tthroo.com/)

```
// Suppose exportPathMap is an object containing paths
const exportPathMap = {
  "/home": true,
  "/about": true,
  "/contact": true,
  "/home": true // Duplicate path
};

// Sample normalizePagePath function
function normalizePagePath(path) {
  return path.toLowerCase(); // Convert path to lowercase
}

// Sample denormalizePagePath function
function denormalizePagePath(path) {
  return path.toUpperCase(); // Convert path to uppercase
}

// execute the code
const exportPaths = \[
  ...new Set(
    Object.keys(exportPathMap).map((path) =>
      denormalizePagePath(normalizePagePath(path))
    )
  ),
\];

console.log(exportPaths); // Output: \["/HOME", "/ABOUT", "/CONTACT"\]
```

[denormalizePagePath](https://github.com/vercel/next.js/blob/canary/packages/next/src/shared/lib/page-path/denormalize-page-path.ts#L12) and [normalizePagePath](https://github.com/vercel/next.js/blob/canary/packages/next/src/shared/lib/page-path/normalize-page-path.ts#L14) are functions from Next.js source code. I have picked a simpler functions to demonstrate this example.

It is simple:

1.  Define a Set with an array that can contain duplicates
2.  Spread the Set into a new array as shown above.

`denormalizePagePath(normalizePagePath(path))`

I asked myself “What? why would you normalize and then denormalize?” These functions code is provided below. You first normalize to ensure paths meet certain requirements, in this case, appending /index to the path and then denormalize it by removing /index. This is to ensure consistency in the page paths. Take away here is, if you are unsure about certain variable values, normalize them aka set a standard to make it consistent and then denormalize.

normalize-page-path.ts
----------------------

```
// source: https://github.com/vercel/next.js/blob/canary/packages/next/src/shared/lib/page-path/normalize-page-path.ts#L14
import { ensureLeadingSlash } from './ensure-leading-slash'
import { isDynamicRoute } from '../router/utils'
import { NormalizeError } from '../utils'

/\*\*
 \* Takes a page and transforms it into its file counterpart ensuring that the
 \* output is normalized. Note this function is not idempotent because a page
 \* \`/index\` can be referencing \`/index/index.js\` and \`/index/index\` could be
 \* referencing \`/index/index/index.js\`. Examples:
 \*  - \`/\` -> \`/index\`
 \*  - \`/index/foo\` -> \`/index/index/foo\`
 \*  - \`/index\` -> \`/index/index\`
 \*/
export function normalizePagePath(page: string): string {
  const normalized =
    /^\\/index(\\/|$)/.test(page) && !isDynamicRoute(page)
      ? \`/index${page}\`
      : page === '/'
      ? '/index'
      : ensureLeadingSlash(page)

  if (process.env.NEXT\_RUNTIME !== 'edge') {
    const { posix } = require('path')
    const resolvedPage = posix.normalize(normalized)
    if (resolvedPage !== normalized) {
      throw new NormalizeError(
        \`Requested and resolved page mismatch: ${normalized} ${resolvedPage}\`
      )
    }
  }

  return normalized
}
```

denormalize-page-path:
----------------------

```
import { isDynamicRoute } from '../router/utils'
import { normalizePathSep } from './normalize-path-sep'

/\*\*
 \* Performs the opposite transformation of \`normalizePagePath\`. Note that
 \* this function is not idempotent either in cases where there are multiple
 \* leading \`/index\` for the page. Examples:
 \*  - \`/index\` -> \`/\`
 \*  - \`/index/foo\` -> \`/foo\`
 \*  - \`/index/index\` -> \`/index\`
 \*/
export function denormalizePagePath(page: string) {
  let \_page = normalizePathSep(page)
  return \_page.startsWith('/index/') && !isDynamicRoute(\_page)
    ? \_page.slice(6)
    : \_page !== '/index'
    ? \_page
    : '/'
}
```

Conclusion:
-----------

I changed my article title to “Tips from open-source” from “Lessons from open-source” as these are short concised tips extracted from opensource code to improve your coding abilities.

You can use Set to remove duplicates from an array and then convert this Set back to an array using spread operator. I also saw function calling another function as its parameter — denormalizePagePath(normalizePagePath(path)).

I have written functions using this style before to improve readability. Condensed functions => less lines of spaghetti code to deal with => improved readability.