Migrate a 60k LOC TypeScript (NodeJS) repo to ESM and testing become 4x faster (1/2)
Tags: javascript, typescript, node, webdev
Published: 2022-12-25 16:57:52

## Intro

After the third try, we successfully migrated all existing Node.js code from CJS to native ESM, and the CI time of unit testing was significantly reduced.

**Pull requests** [refactor: use ESM](https://github.com/logto-io/logto/pull/2491) [test: use native ESM](https://github.com/logto-io/logto/pull/2621)

Before we start, I'd like to demonstrate the status quo ante for a better context. You may have different choices on the repo setting or toolchain, but the core steps and concepts should be the same:

- A TypeScript monorepo that includes both frontend and backend projects / packages.
- Total TypeScript code ~60k LOC (including frontend).
- Use `import` in TypeScript.
- Use PNPM for workspace management.
- Use `tsc` to compile Node.js, and Parcel to bundle frontend projects.
- Use Jest + `ts-jest` for unit testing.
- Use package `module-alias` for internal path aliases.

BTW, our project [Logto](https://github.com/logto-io/logto) is an open-source solution for auth.

---

## Why ESM?

When we noticed more and more NPM packages are "ESM-only", and we closed tons of PR because of it (except Parcel / Jest ones):

![PR closed due to ESM-only](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/518mwe9gszu3yc5ectju.png)

Disregard the war of ESM v.s. CJS, we found ESM does have several advantages:

**No another-language-like code transpilation**

Especially for TypeScript: Comparing the original code to the compiled version, ESM is much easier to read, edit, and debug.

Given a simple TypeScript snippet:

```ts
import path from 'path';

const replaceFile = (filePath: string, filename: string) =>
  path.resolve(path.dirname(filePath), filename);
```

Results after `tsc`:

```ts
// CJS
"use strict";
var __importDefault = (this && this.__importDefault) || function (mod) {
    return (mod && mod.__esModule) ? mod : { "default": mod };
};
Object.defineProperty(exports, "__esModule", { value: true });
const path_1 = __importDefault(require("path"));
const replaceFile = (filePath, filename) => path_1.default.resolve(path_1.default.dirname(filePath), filename);

// ESM
import path from 'path';
const replaceFile = (filePath, filename) => path.resolve(path.dirname(filePath), filename);
```

**Top-level await**

This is one of our favorites. Finally no need for wrapping top-level async expressions into a function and using `void` to execute it.

```ts
// CJS
(async () => {
  await doSomething();
})();

// ESM
await doSomething();
```

**Compatibility**

- While ESM can easily load CJS modules, it'll be harder for CJS to load ESM. The primary reason is CJS `require()` is synchronous, while ESM `import` is asynchronous. You must use `await import()` in CJS which is painful w/o top-level await.
- Plus, CJS is Node-only, which means a universal package needs to compile another version for browser users. (We know there are transpilers, but, huh)

**Stick with the standard**

ESM is [the module standard](https://tc39.es/ecma262/#sec-modules) of JavaScript. TypeScript also uses ESM syntax by default.

**Immutable**

This is more like a double-edged sword. ESM can significantly improve module security by its immutable design (see [this article](https://webreflection.medium.com/cjs-vs-esm-5f8b90a4511a)), but it also brings a little inconvenience for test mocking.

**Dramatic testing time reduction**

Yes, we're talking about Jest. While Jest is sticking with CJS and only has experimental ESM support, we've been using `ts-jest` for a while, but obviously, it's a challenging task even for an M1 Pro MacBook. The fan spins with a noticeable sound, and the core temperature keeps going up while running unit tests.

After migrating all unit tests to ESM as well, my MacBook became silent and pro again! Here's the CI time comparison (with default machine spec. in GitHub Actions):

![Time comparison](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nkp2q8iia2cqylygd1es.png)

Execution time are not that stable in GitHub Actions, on average it shows 3x - 4x faster.

## Code migration

For official docs, you may find [Modules: ECMAScript modules](https://nodejs.org/api/esm.html) and [ECMAScript Modules in Node.js](https://www.typescriptlang.org/docs/handbook/esm-node.html) helpful.

### Basic configuration

Let's start with `tsconfig.json`. Two key points:

- Set `compilerOptions.moduleResolution` to `nodenext` in order to tell TSC to use the "cutting-edge" Node.js module resolution strategy.
- Set `compilerOptions.module` to `esnext` to ensure the output also keeps ESM.

For Node.js, add `"type": "module"` to your `package.json` to let it treats the package as ESM.

### Path aliases

We were mapping `@/` to `./src/` using `module-alias` alias but it doesn't work in ESM. The good news is Node.js provides a native support called [Subpath imports](https://nodejs.org/api/packages.html#subpath-imports) by defining the `imports` field in `package.json`:

```js
{
  "imports": {
    "#src/*": "./build/*" // Point to the build directory, not source
  }
}
```

Note `imports` can only starts with `#` and must have a name. So we use `#src/` to replace the original `@/`. Also update your `tsconfig.json` accordingly:

```json
{
  "compilerOptions": {
    "paths": {
      "#src/*": ["src/*"]
    }
  }
}
```

We also need to replace all `@/` with `#src/` in source code.

### File extension

At this point, both TSC and Node.js start to work in ESM mode, but most likely some "Cannot find module..." errors float. Because the existing convention of importing is minimalistic and elegant:

```ts
import x from './foo';

// It can be
import x from './foo.js';
// Or
import x from './foo/index.js';
```

And the `.js` extension can be replaced with `.jsx`, `.ts`, and `.tsx`, etc.

However, this becomes unacceptable in native ESM. You must explicitly write the full path with the extension, e.g. `import x from './foo/index.js';`.

So how it should be in TypeScript? Our first idea is changing the file extension to `.ts`, which turns the path to `'./foo/index.ts'`, since that's the file we can find in the source directory, right?

Unfortunately, the TypeScript team has the principles like "TS is the superset of JS" and "TS doesn't rewrite paths". (You can see [#13422](https://github.com/microsoft/TypeScript/issues/13422) [#16577](https://github.com/microsoft/TypeScript/issues/16577) [#42151](https://github.com/microsoft/TypeScript/issues/42151) since 2017) So `.ts` doesn't work here, and it led to the result: use `.js`. :-)

It doe works, and I think I'm not qualified to judge the solution. So let's move to the actions we took to add extensions:

> Since most packages in `node_modules` are not affected by this (at least for the main entry), we can omit them during the process.

1. Replace all `from '\.'` (RegExp) with `from './index'`.
2. Replace all `from '\./(.*)'` (RegExp) with `from './$1.js'`.
3. If you have path alias, use the similar technique in step 2 to add extensions to them.
4. Try to compile the project. It may show some errors for those paths with omitted `/index`, e.g. a `./foo` which actually points to `./foo/index.js`. They are updated to wrong paths like `./foo.js` in step 2.
5. Try to compile again and it should show no error this time.

### Misc.

It's exciting to see the output files are almost the same as the input, but when you run the project, Node.js may complain about some special variables, for example:

```bash
ReferenceError: __dirname is not defined in ES module scope
```

Don't give up, we're almost there! Read the Node.js official doc [Differences between ES modules and CommonJS](https://nodejs.org/api/esm.html#differences-between-es-modules-and-commonjs) to handle them, then you're good to go.

## Recap

We successfully migrated our Node.js packages from CJS to ESM. Now the entry-point file is runnable after `tsc`.

However, some issues still remain in unit tests:

- As of today (12/26/22), Jest only has experimental support for ESM.
- ESM is immutable, thus `jest.mock()` will not work and `jest.spyOn()` also doesn't work on first-level variables (`export const ...`). This also applies to other test libraries like Sinon.
- You may find some libraries for mocking ESM, but almost all of them are creating "a new copy" of the original module, which means if you want to import module A that depends on module B, you must import A AFTER B is mocked to get it to work.

The solution is the key to boosting the CI time. Since this article already pulled out a lot of things to digest, we'll cover them in the next chapter:

[Migrate a 60k LOC TypeScript (NodeJS) repo to ESM and testing become 4x faster (2/2)](https://dev.to/gaosun/migrate-a-60k-loc-typescript-nodejs-repo-to-esm-and-testing-become-4x-faster-22-4a4k)

You can find our `ts-with-node-esm` repo for the key result of this series:

https://github.com/logto-io/ts-with-node-esm

Thank you for reading, feel free to comment if you have any questions!

---

This series is based on our experience with [Logto](https://github.com/logto-io/logto), an open-source solution for auth.