Transitioning from CommonJS to ESM
Tags: commonjs, esm
Published: 2024-03-15 02:45:59

> This is a translated article from the original post: https://tech.kakao.com/2023/10/19/commonjs-esm-migration/

In this article, we would like to explain why we switched from the old module system CommonJS to ESM and how all this switch happened in the process of upgrading the version of the library used by the service in operation.

* cjs: CommonJS
* ESM: EcmaScript Modules

## 1. Why we transitioned to ESM
The FE Platform team uses **Pharus**, a service that periodically analyzes the performance of web services using Google's Lighthouse.

**Lighthouse** is a static performance analytics tool created by Google that accesses web services in Chrome to collect performance information and generate reports. It is a library that is constantly updated, and versions 10 and 11 were released this year. Version 10 includes major changes such as performance score changes, Typecript support, and transition to ESM.

Pharus, on the other hand, uses Lighthouse to measure the performance of a web service every six hours and makes it easier for developers to keep track of changes in performance. It also supports the authentication processing needed to access a web service to measure performance, as well as setting up user agents and headers.

Pharus is constantly following Lighthouse's version updates to support performance measurements in the latest version of Chrome, and our goal was to apply the new version of Lighthouse (v.10) in Pharus.

Previously, Pharus was loading Lighthouse v.9 in the CommonJS environment. Because the module system switched from CommonJS to ESM, the `require()` function of CommonJS does not allow the module in the ESM system to be loaded. In response, Lighthouse v.10 provides two ways 
for CommonJS environment compatibility.

1. `await import(‘lighthouse’)`
2. `require(‘lighthouse/core/index.cjs’)`

However, if we were to apply both methods, many changes had to be made, and also there were restrictions on using some features, so we switched to ESM so that Pharus (which is based on Lighthouse v.10) could load the modules.

### (1) Importing through `dynamic import`
Although option no. 1, `await import(‘lighthouse’)`, was not selected, we'll explain the process we took.

Pharus is written in Typescript and compiled into module syntax that can be interpreted in a Common JS environment by setting `tsconfig.json` as follows.
```json
// tsconfig.json
{
  "compilerOption": {
    "module": "CommonJS"
  }
}
```

Then, we changed the code to get Lighthouse using `await import()` as below.
```js
// import lighthouse from ‘lighthouse’;
const lighthouse = await import('lighthouse');
```

However, an error occurred that the `require() could not get the ES Module`.
```
Error [ERR_REQUIRE_ESM]: require() of ES Module ... not supported.
```

This error occurs because when module property is set to "CommonJS" in tsconfig, `await import()` is converted to "require()". Therefore, we changed to "Node16" again as below.
```json
// tsconfig.json
{
  "compilerOption": {
    "module": "Node16"
  }
}
```

If we try this again, we will get an error saying that the await syntax should be used within the async function.

```
const lighthouse = await import('lighthouse');
                   ^^^^^

SyntaxError: await is only valid in async functions and the top level bodies of modules
```

Although the `await import()` allows you to import aa ESM Lighthouse module in a CommonJS environment, the await syntax can only be used within the async function, so more code need to be added to import the module. In the end, the `await import()` was discarded.

### (2) Importing as `cjs` file
Option no.2, `require(‘lighthouse/core/index.cjs’)`, was not selected due to some functional constraints. Here's what happened.

Lighthouse still provides `index.cjs` files for existing CommonJS users. `index.cjs` imports some modules from index.js, which is the entry point for the Lighthouse, into `await import()` and then exports them using `module.exports`.

Because Pharus defines and uses a new measurement environment by expanding the measurement elements in Lighthouse, we needed to import and use Gatherer and Audit classes. However, since `index.cjs` does not export these classes, importing index.cjs alone could not solve the problem.

As such, we decided to switch Pharus to ESM because the functionality of the Lighthouse was not available in the CommonJS environment.

## 2. Differences between CommonJS and ESM
Common JS and ESM have different syntax for exporting and importing modules. For now, they are not compatible with each other, so you should use the appropriate syntax.

Now, let's get to the point where Node.js decides which module system to use to parse files, and see the difference in syntax between CommonJS and ESM.

### Determining the module type
This is the way Node.js decides how to parse JavaScript files. Each order means priority, and the order determines the module system if the conditions are correct. More details are available in the [Determining Module System](https://nodejs.org/api/packages.html#determining-module-system) document.

1. Check the file extension. The `.cjs` extension is CommonJS, whereas `.mjs` is for ESM.
2. Check the 'type' field in the closest upper `package.json`. If its value is 'module', then it's ESM, whereas if it's 'commonJS' or empty, then it's CommonJS.
3. Check Node.js `--input-type` flag. `--input-type=module` is ESM, while `--input-type=commonjs` means it's a CommonJS.

### Exporting the modules
**CommonJS**
- `module.exports`
```js
module.exports = 'module';
```
```js
module.exports = {
  someNumber: 123,
  someMethod: () => {}
}
```
- `exports`: When you export multiple values from a single file, you can pass them to the properties of the `exports` rather than as objects in `module.exports`. `exports` and `module.exports` have the same reference, so if a different value is assigned to `exports` the reference to module.exports disappears and cannot be exported.
```js
exports.someNumber = 123;
exports.someMethod = () => {};

// although this will not be evaluated, it will still be exported
if (false) {
  exports.falseModule = 'false';
}


// modifying exports will prevent the module to be exported
(function (e) {
  e.aliasModule = 'alias';
})(exports)

// not exported
exports = 'abc';

// not exported
exports = {
  something: 123
}
```

**ESM**
- `export default`: You can export one particular module value to the default export.
```js
export default {
  something: 123
}
```
- `export`: You can export values to the specified name.
```js
export const namedVar = 'module';
```
- `export from`: Imports a module and exports it right away.
```js
export otherModule from './otherModule.js';
```

### Importing modules
**CommonJS**
- `require()`: We use the `require()` function when importing modules in CommonJS, which cannot import ESM files (ERR_REQUIRYRE_ESM error occurs when trying to import ESM files). Also, you do not need to create a file extension.
```js
const module = require('./moduleFile');
```
- `import()`: Imports ESM modules asynchronously in CommonJS. You must specify a file extension.
```js
import('./moduleFile.js').then(module => {
  ...
});
```
**ESM**
- `import`: Use `import` statements to import ES modules. Dynamic values are not evaluated, and therefore not available, because the module is imported in the parsing phase. Both CommonJS and ESM modules can be imported with this syntax and you must specify a file extension.
```js
import { functionName } from './moduleFile.js';
// imports all the named exports 
import * as allProperty from './moduleObject.js';
// Don't
import {AorB_Module} from condition ? './A_module.js' : './B_module.js';
```
- `import()`: Dynamically imports modules.
```js
const module = await import('./moduleFile.js');

const aOrb_Module = await import(condition ? './A_module.js' : './B_module.js');
```

## Using the CommonJS Modules in an ESM Environment
Still a lot of libraries in Pharus use CommonJS. Now let's look at how to import CommonJS modules when you switch to ESM. First, you can import CommonJS modules as import statements or import expressions in ESM, unlike CommonJS' require().
```js
// Modules exported as module.exports are exported in default properties.
import { default as cjsModule } from 'cjs';

// For convenience, you can import it without using { default as cjs} the way ESM uses it.
import cjsSugar from 'cjs';

// The cjsModule and cjsSugar have the same value.
// <module.exports>

import * as module from 'cjs';
const m = await import('cjs');

// The module and m have the same value.
// [Module] { default: <module.exports> }
// module.default has the same value as cjsModule.
```
CommonJS may also use `exports` to export multiple modules by giving a specific name in a file. This is because `cjs-module-lexer` is integrated into Node.js, which allows us to import modules of CommonJS individually in an ESM environment. This method can work independently of the evaluation of the value because it imports modules from the parsing stage.

Because the syntax is different, import/export statements are not available in CommonJS and require/module.exports is not available in ESM. We need to export and import each module using the appropriate syntax.

## 3. Set to ESM environment
Let's set the service environment to the ESM system.

### Module type configuration
To transform the environment of the service into an ESM system, change the extension of all files to `mjs` or change the type property from `package.json` to `"module"` as shown below.
```json
// package.json
{
  "type": "module"
}
```
### TypeScript configuration
When written in Typescript, the module syntax gets converted  into a specific one. It supports the `module` property to determine which module system to compile the module `import/export` syntax into. And by default it's "CommonJS". Set compilation options so that when Typescript gets compiled, it can be built into ESM's import and export statements.

> It gets compiled into ESM syntax if we use ES2015, ES6, ES2020, ES2022, ESNext values.

```json
// tsconfig.json
{
  "compileOption": {
    "module": "ES2020"
  }
}
```

## 4. Learn how ESM works under the hood
We experienced two issues in transitioning Pharus from a Common JS environment to an ESM. But before we have a look at the solutions to the issues, let's first learn how ESM works. More information is available [here](https://hacks.mozilla.org/2018/03/es-modules-a-cartoon-deep-dive/).

ESM operates in three stages: **configuration, instantiation, and evaluation**. After getting module resources(files), parsing, setting the memory address for each variable or function, and finally executing the code and populating the values.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cb8j9hic1xif14yvizcx.png)
source: _https://hacks.mozilla.org/2018/03/es-modules-a-cartoon-deep-dive/_

Here's what happens in detail:

- Configuration: Gets module resources and performs parsing.
- Instantiation: Allocates and shares the memory address of the variable that each module imports and exports.
- Evaluation: Fills a value into memory while executing the code.

Unlike CommonJS, which performs each step synchronously while traversing all modules, ESM performs each step **asynchronously**, allowing one module to use the `await` syntax at the module **top level**, such as the async function, and consequently **improving performance for the entire file load.**

Let's dig into each one them in more detail.

### (1) Configuration

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yxane26nnfzd0xiqktet.png)
Source: _https://hacks.mozilla.org/2018/03/es-modules-a-cartoon-deep-dive/_

1. Check where to import the file containing the module.
After parsing, the **Loader checks the module designator in the import statement and determines the file path according to the module resolution rules (initially checks the entry point file).
2. Imports the file.
Before importing them, it checks the module map to prevent duplicate files from being imported. Then import process is initialized. Different loaders can be used depending on the platform. For example, **browsers** import files through **HTTP** communication based on HTML specifications, while **Node.js** imports them using the **file system**. Node.js provides the ability to customize the Loader.

3. Parse the file into a module record.
The file itself cannot be read by the browser or Node.js. Therefore, it gets parsed into a module record that contains an **Abstract Syntax Tree (AST)**.
When parsing is complete, the process is repeated until all linked files are imported.

The tree-structure module record is created once the repetition is over.

### (2) Instantiation
The detailed process of ESM's instantiation process is as follows.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/i7zzego3bafo7mzg3wmr.png)
Source: _https://hacks.mozilla.org/2018/03/es-modules-a-cartoon-deep-dive/_

The Javascript engine configures the module environment records per module record. The module environment records manage the variables in module records and allocate and track the memory of each variable. Depth-first search(`DFS`) is applied to the module records tree to assign memory addresses to module records from modules that only export without dependencies on other modules.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7h3qvdiy80jo3rvbic2n.png)
Source: https://hacks.mozilla.org/2018/03/es-modules-a-cartoon-deep-dive/

At this time, the module that is importing will share what memory address the imported variable has.

> CommonJS replicates each module value, but ESM shares the memory address


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6ft2t62g0yz895287m44.png)
Source: https://hacks.mozilla.org/2018/03/es-modules-a-cartoon-deep-dive/

Since ESM shares the memory, we cannot assign a value to an imported module on an imported file. Otherwise we might end up having side effects.

At the end of the instantiation process, the memory locations for all variables or functions that have been imported/exported get associated.

### (3) Instantiation
To fill the actual memory with values, the JS engine executes the code from the top. To avoid possible side effects, the module gets evaluated only once. For example, if a module calls a server, the results may vary depending on the number of evaluations. Through the evaluation process, a module map is constructed so that each module has one module record, allowing each module to be evaluated only once.

To be continued in part II...