Migrate a 60k LOC TypeScript (NodeJS) repo to ESM and testing become 4x faster (2/2)
Tags: javascript, webdev, typescript, node
Published: 2022-12-26 09:43:12

In [the previous article](https://dev.to/gaosun/migrate-a-60k-loc-typescript-nodejs-repo-to-esm-and-testing-become-4x-faster-12-5f82), we discussed the advantages of using ESM in Node.js and showed the migration steps for a TypeScript project.

But our unit testing was still in CJS (with `ts-jest`), which is the last piece of our migration and slowed down my M1 Pro MacBook and CI time. In this article, we'll explain how to use native ESM in Jest and provide a minimum ESM repo to cover the key results of this series.

![Time comparison](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nkp2q8iia2cqylygd1es.png)

BTW, our project [Logto](https://github.com/logto-io/logto) is an open-source solution for auth.

Let's get started!

---

## The disharmony

If you are using Jest, you must have seen this classic error:

```bash
SyntaxError: Cannot use import statement outside a module
```

This usually happens when your Jest runs in CJS mode but meets an ESM module. Searching the message in Google, there are two popular genres (or a combination) to solve this:

1. Introduce a transpiler like `babel`.
2. For an existing transpiler, fine-tune a complex RegExp to let it work for specific ESM dependencies, e.g.:

```js
// jest.config.js
{
  transformIgnorePatterns: ['node_modules/(?!(.*(nanoid|jose|whatever-esm))/)'],
}
```

Both of them are OK and just OK. Since our code base is already in ESM, it looks redundant to go back to CJS again.

## Challenges

However, everything unusual has reasons. For Jest, they are:

1. As of today (12/26/22), Jest only has experimental support for ESM.
2. ESM is immutable, thus jest.mock() will not work and jest.spyOn() also doesn't work on first-level variables (export const ...). This also applies to other test libraries like Sinon.
3. You may find some libraries for mocking ESM, but almost all of them are creating "a new copy" of the original module, which means if you want to import module A that depends on module B, you must import A AFTER B is mocked to get it to work.

If you are good with 1, just like us, then 2 and 3 will be good, too.

## Enable ESM in Jest

Jest has [an official doc](https://jestjs.io/docs/ecmascript-modules) of ESM as reference. But our situation is slightly different: we are using TypeScript with ESM.

### Configuration

Let's temporarily forget `ts-jest` or `ts-node` or `babel` and return to the nature. How about use `tsc` for transpilation and directly run the JavaScript?

We set up a dedicated `tsconfig.test.json` for compiling with   tests:

```js
{
  // Extends the base config
  "extends": "./tsconfig",
  // Loose some configs for testing
  "compilerOptions": {
    "isolatedModules": false,
    "allowJs": true,
  },
  // Bring back test files we excluded before
  "exclude": []
}
```

And add some `package.json` scripts:

```json
{
  "scripts": {
    "build:test": "tsc -p tsconfig.test.json --sourcemap",
    "test:only": "NODE_OPTIONS=--experimental-vm-modules jest",
    "test": "pnpm build:test && pnpm test:only",
    "test:coverage": "pnpm run test --coverage"
  }
}
```

- `build:test` is to build all TypeScript files to JS with sourcemap, thus we can have the error stack points to the source code along with the coverage report.
- `test:only` is to run tests without build, just for convenience.
- `test` combines build and run.
- `test:coverage` appends `--coverage` to generate coverage report. Note some package manager may require an additional `--` to pass arguments to the inner command.

For Jest config, remove all presets and special RegExp for supporting ESM or TypeScript, since there's only one key config:

```js
/** @type {import('jest').Config} */
const config = {
  roots: ['./build'], // Point to the build directory
};
```

To simplify, we changed `jest.config.ts` to `jest.config.js` and added `@type` annotation for type reference in VSCode.

### Mocking ESM

At this point, execute `pnpm run test` should be able to run tests without module system error. If it doesn't, refer to the [Jest official doc](https://jestjs.io/docs/ecmascript-modules) to ensure config has been replaced correctly.

If you have used the `jest` namespace functions, like `jest.mock()` or `jest.spyOn()`, Jest now complains `jest is not defined`.

Two solutions:

1. Put a `const { jest } = import.meta;` before all `jest.*` calls.
2. Install `@jest/globals` and use `import { jest } from '@jest/globals';`.

We chose 1.

But module mock is still not working, which leads to expectation errors in some test suites. Because the API is slightly different under ESM:

```ts
jest.unstable_mockModule('node:child_process', () => ({
  execSync: jest.fn(),
}));
```

Although it has been marked as "unstable", our tests have been running stably for a while without exception.

> You may find `unstable_mockModule()` is not typed for `import.meta.jest`, so we added the signature manually by module augmentation.

### The order matters

By adopting `unstable_mockModule()`, you can see some tests are fixed, but some are still not. Don't go crazy because we are just one step away.

Because of its immutable nature, ESM CANNOT be edited. This means `unstable_mockModule()` is creating a new copy of that module instead of update it in place.

Say you have a module `bar.ts` which imports another module `foo.ts` (same for JS):

```ts
// foo.ts
export const value = 1;

// bar.ts
import { value } from './foo.js';
export const add = () => value + 1;
```

In `bar.test.ts`:

```ts
import { add } from './bar.js';

const { jest } = import.meta;
jest.unstable_mockModule('./foo.js', () => ({
  value: 2,
}));

describe('bar', () => {
  it('should have value 3', () => {
    expect(add()).toEqual(3); // Error, still 2
  });
});
```

Because `bar` loads the original version of `foo`. Remember ESM has top-level await, so just change to:

```ts
const { jest } = import.meta;
jest.unstable_mockModule('./foo.js', () => ({
  value: 2,
}));

const { add } = await import('./bar.js');

describe('bar', () => {
  it('should have value 3', () => {
    expect(add()).toEqual(3); // OK
  });
});
```

Not hard, right? Actually, the new code makes more sense to us because we find the code becomes predictable. In CJS, `jest.mock` will be automatically hoisted (that's why we drop `var` and use `let`).

You'll also find the following error also disappears:

```bash
The module factory of `jest.mock()` is not allowed to reference any out-of-scope variables.
```

### Partial mocking

Due to the same reason, `jest.spyOn()` doesn't work in top-level variables or functions. E.g.:

```ts
// foo.ts
export const run = () => true;

// test.ts
import * as functions from './foo.js';
jest.spyOn(functions, 'run'); // Doesn't work
```

The essence of `.spyOn()` a top-level member is *partial mocking*, i.e., keep everything else the same, but spy on a specific member. So this can be archived by:

```ts
const actual = await import('./foo.js');

jest.unstable_mockModule(moduleName, () => ({
  ...actual,
  run: jest.fn(actual.run),
}));

const { run } = await import ('./foo.js'); // `run` has type `jest.Mock`
```

Looks a little bit verbose, so we made a helper function:

```ts
const mockEsmWithActual = async <T>(
  ...[moduleName, factory]: Parameters<typeof jest.unstable_mockModule<T>>
): Promise<T> => {
  const actual = await import(moduleName);

  jest.unstable_mockModule(moduleName, () => ({
    ...actual,
    ...factory(),
  }));

  return import(moduleName);
};
```

**Caveat** The `moduleName` may be problematic if BOTH conditions are met:

- The helper function is not located in the same directory as the caller file.
- The caller is trying to mock a relative path module, e.g. `mockEsmWithActual('./foo.js')`. Path alias will be fine.

We doubt the reason is importing process is running in a Promise, which leaves the original context. To use relative paths w/o worries, see [this file](https://github.com/logto-io/ts-with-node-esm/blob/master/packages/esm-helper/src/index.ts#L62).

## Closing note

That's it! Our Node.js code is all in native ESM now, and the dev experience has become much better.

You can find our `ts-with-node-esm` repo for the key result of this series:

https://github.com/logto-io/ts-with-node-esm

Thank you for reading, feel free to comment if you have any questions!

---

This series is based on our experience with [Logto](https://github.com/logto-io/logto), an open-source solution for auth.