Repository: node
Issue: Proposal: Have the ESM loader handle all entry points
Labels: module,performance,esm

In current Node, the ESM loader handles the application entry point when it is an ES module _or_ if any of the following flags are passed: `--import`, `--experimental-default-type`, `--experimental-detect-module`, `--experimental-loader`. This means that the ESM loader handles CommonJS entry points when certain flags are passed. (The ESM loader supports ES modules, CommonJS modules, WebAssembly modules and JSON files.) A function [`shouldUseESMLoader` in `run_main.js`](https://github.com/nodejs/node/blob/0fb512344fe97b508adf1332c3e82d33937c4669/lib/internal/modules/run_main.js#L52-L87) determines whether the ESM loader should handle the main entry point.

If we were to remove this function and simply _always_ have the ESM loader handle all entry points, there would be a few gains:

- There would be only one code path for starting the application entry point, rather than two, simplifying the codebase and making it more maintainable.
- The simpler the codebase can become, the more likely it will be that we can find performance improvements, such as by moving parts of the flow into C++.

It would also mean the end of monkey-patching the CommonJS loader, which has never been officially supported and I think is ready to be officially retired now that the [module customization hooks](https://nodejs.org/api/module.html#customization-hooks) exist and provide fully equivalent functionality, including for customizing `require` calls. They are Release Candidate and I expect them to become stable in the next few months. This potential change of having the ESM loader handle all entries would happen after that, at the next major version of Node. It _shouldn’t_ be a semver-major change, but monkey-patching is common enough that I’d expect some breakage and therefore we should treat it as semver-major.

If you take the [`run_main.js` `if (useESMLoader) {` line](https://github.com/nodejs/node/blob/0fb512344fe97b508adf1332c3e82d33937c4669/lib/internal/modules/run_main.js#L130) and change it to `if (true) {` and run `make test`, as of today there are 154 failing tests. For the most part, these are tests that just need updating because the assertions in the tests are more specific than they need to be for what they’re validating. They test things like the number of promises created since Node started up, which varies when the ESM loader is engaged because the loader itself creates a few; or they need `--allow-fs` to include allowing reading some `package.json` files that the ESM loader loads that the CommonJS loader didn’t; or they are validating too precise of a stack trace, that changes when the ESM loader is used; and so on. The largest group of failing tests is for async hooks, and I’ve discussed this with @Qard (see https://github.com/nodejs/node/pull/44323#issuecomment-1775434155) and we think that those tests should simply be updated, once the major refactor of `AsyncLocalStorage` lands.

In the meantime, I think we should start making smaller PRs to start updating batches of tests that fail when `shouldUseESMLoader` is always true, so that these tests pass both today and in “always using the ESM loader” mode. Once we can get that 154 number down to zero, then we can do the cutover, ideally before Node 22 to be released as part of 22.0.0. This issue can serve as the tracking issue for the effort.

<details>
<summary>These are the tests (or APIs) that need updating:</summary>
<ul>
<li><code>./node --experimental-permission --allow-fs-read=* --allow-child-process test/parallel/test-permission-fs-wildcard.js</code></li>
<li><code>./node --experimental-permission --allow-fs-read=* --allow-fs-write=* --allow-child-process test/parallel/test-permission-fs-read.js</code></li>
<li><code>./node --experimental-permission --allow-fs-read=* --allow-fs-write=* --allow-child-process test/parallel/test-permission-fs-symlink-target-write.js</code></li>
<li><code>./node --experimental-permission --allow-fs-read=* --allow-fs-write=* --allow-child-process test/parallel/test-permission-fs-symlink.js</code></li>
<li><code>./node --experimental-permission --allow-fs-read=* --allow-fs-write=* --allow-child-process test/parallel/test-permission-fs-traversal-path.js</code></li>
<li><code>./node --experimental-permission --allow-fs-read=* test/parallel/test-permission-experimental.js</code></li>
<li><code>./node --expose-gc test/async-hooks/test-pipewrap.js</code></li>
<li><code>./node --expose-gc test/async-hooks/test-querywrap.js</code></li>
<li><code>./node --expose-gc test/parallel/test-heapdump-async-hooks-init-promise.js</code></li>
<li><code>./node --expose-internals test/async-hooks/test-emit-after-on-destroyed.js</code></li>
<li><code>./node --expose-internals test/async-hooks/test-emit-before-on-destroyed.js</code></li>
<li><code>./node --expose-internals test/async-hooks/test-emit-init.js</code></li>
<li><code>./node --expose-internals test/async-hooks/test-http-agent-handle-reuse-parallel.js</code></li>
<li><code>./node --expose-internals test/async-hooks/test-http-agent-handle-reuse-serial.js</code></li>
<li><code>./node --expose-internals test/async-hooks/test-improper-order.js</code></li>
<li><code>./node --expose-internals test/async-hooks/test-zlib.zlib-binding.deflate.js</code></li>
<li><code>./node --expose-internals test/message/internal_assert_fail.js</code></li>
<li><code>./node --expose-internals test/message/internal_assert.js</code></li>
<li><code>./node --expose-internals test/parallel/test-bootstrap-modules.js</code></li>
<li><code>./node --expose-internals test/parallel/test-cluster-dgram-bind-fd.js</code></li>
<li><code>./node --expose-internals test/parallel/test-util-promisify.js</code></li>
<li><code>./node --inspect=0 test/known_issues/test-inspector-cluster-port-clash.js</code></li>
<li><code>./node --no-warnings --expose-internals test/parallel/test-whatwg-webstreams-adapters-to-writablestream.js</code></li>
<li><code>./node --pending-deprecation test/parallel/test-module-parent-deprecation.js</code></li>
<li><code>./node --test-udp-no-try-send test/async-hooks/test-udpsendwrap.js</code></li>
<li><code>./node test/addons/callback-scope/test-async-hooks.js</code></li>
<li><code>./node test/addons/make-callback-recurse/test.js</code></li>
<li><code>./node test/addons/report-api/test.js</code></li>
<li><code>./node test/async-hooks/test-async-await.js</code></li>
<li><code>./node test/async-hooks/test-crypto-pbkdf2.js</code></li>
<li><code>./node test/async-hooks/test-crypto-randomBytes.js</code></li>
<li><code>./node test/async-hooks/test-disable-in-init.js</code></li>
<li><code>./node test/async-hooks/test-embedder.api.async-resource-no-type.js</code></li>
<li><code>./node test/async-hooks/test-embedder.api.async-resource.js</code></li>
<li><code>./node test/async-hooks/test-enable-disable.js</code></li>
<li><code>./node test/async-hooks/test-enable-in-init.js</code></li>
<li><code>./node test/async-hooks/test-filehandle-no-reuse.js</code></li>
<li><code>./node test/async-hooks/test-fseventwrap.js</code></li>
<li><code>./node test/async-hooks/test-fsreqcallback-access.js</code></li>
<li><code>./node test/async-hooks/test-fsreqcallback-readFile.js</code></li>
<li><code>./node test/async-hooks/test-getaddrinforeqwrap.js</code></li>
<li><code>./node test/async-hooks/test-getnameinforeqwrap.js</code></li>
<li><code>./node test/async-hooks/test-graph.fsreq-readFile.js</code></li>
<li><code>./node test/async-hooks/test-graph.http.js</code></li>
<li><code>./node test/async-hooks/test-graph.intervals.js</code></li>
<li><code>./node test/async-hooks/test-graph.pipe.js</code></li>
<li><code>./node test/async-hooks/test-graph.pipeconnect.js</code></li>
<li><code>./node test/async-hooks/test-graph.shutdown.js</code></li>
<li><code>./node test/async-hooks/test-graph.signal.js</code></li>
<li><code>./node test/async-hooks/test-graph.statwatcher.js</code></li>
<li><code>./node test/async-hooks/test-graph.tcp.js</code></li>
<li><code>./node test/async-hooks/test-graph.timeouts.js</code></li>
<li><code>./node test/async-hooks/test-graph.tls-write-12.js</code></li>
<li><code>./node test/async-hooks/test-graph.tls-write.js</code></li>
<li><code>./node test/async-hooks/test-httpparser.request.js</code></li>
<li><code>./node test/async-hooks/test-httpparser.response.js</code></li>
<li><code>./node test/async-hooks/test-immediate.js</code></li>
<li><code>./node test/async-hooks/test-nexttick-default-trigger.js</code></li>
<li><code>./node test/async-hooks/test-pipeconnectwrap.js</code></li>
<li><code>./node test/async-hooks/test-promise.chain-promise-before-init-hooks.js</code></li>
<li><code>./node test/async-hooks/test-promise.js</code></li>
<li><code>./node test/async-hooks/test-promise.promise-before-init-hooks.js</code></li>
<li><code>./node test/async-hooks/test-queue-microtask.js</code></li>
<li><code>./node test/async-hooks/test-shutdownwrap.js</code></li>
<li><code>./node test/async-hooks/test-signalwrap.js</code></li>
<li><code>./node test/async-hooks/test-statwatcher.js</code></li>
<li><code>./node test/async-hooks/test-tcpwrap.js</code></li>
<li><code>./node test/async-hooks/test-timers.setInterval.js</code></li>
<li><code>./node test/async-hooks/test-timers.setTimeout.js</code></li>
<li><code>./node test/async-hooks/test-tlswrap.js</code></li>
<li><code>./node test/async-hooks/test-ttywrap.readstream.js</code></li>
<li><code>./node test/async-hooks/test-udpwrap.js</code></li>
<li><code>./node test/async-hooks/test-unhandled-exception-valid-ids.js</code></li>
<li><code>./node test/async-hooks/test-unhandled-rejection-context.js</code></li>
<li><code>./node test/async-hooks/test-writewrap.js</code></li>
<li><code>./node test/message/assert_throws_stack.js</code></li>
<li><code>./node test/message/util_inspect_error.js</code></li>
<li><code>./node test/message/util-inspect-error-cause.js</code></li>
<li><code>./node test/node-api/test_callback_scope/test-async-hooks.js</code></li>
<li><code>./node test/node-api/test_make_callback_recurse/test.js</code></li>
<li><code>./node test/node-api/test_policy/test_policy.js</code></li>
<li><code>./node test/parallel/test-async-hooks-correctly-switch-promise-hook.js</code></li>
<li><code>./node test/parallel/test-async-hooks-disable-during-promise.js</code></li>
<li><code>./node test/parallel/test-async-hooks-enable-recursive.js</code></li>
<li><code>./node test/parallel/test-async-hooks-promise-triggerid.js</code></li>
<li><code>./node test/parallel/test-async-hooks-promise.js</code></li>
<li><code>./node test/parallel/test-async-hooks-top-level-clearimmediate.js</code></li>
<li><code>./node test/parallel/test-async-local-storage-exit-does-not-leak.js</code></li>
<li><code>./node test/parallel/test-async-wrap-promise-after-enabled.js</code></li>
<li><code>./node test/parallel/test-child-process-fork-closed-channel-segfault.js</code></li>
<li><code>./node test/parallel/test-cli-permission-deny-fs.js</code></li>
<li><code>./node test/parallel/test-cluster-advanced-serialization.js</code></li>
<li><code>./node test/parallel/test-cluster-dgram-2.js</code></li>
<li><code>./node test/parallel/test-cluster-fork-windowsHide.js</code></li>
<li><code>./node test/parallel/test-cluster-send-deadlock.js</code></li>
<li><code>./node test/parallel/test-cluster-send-socket-to-worker-http-server.js</code></li>
<li><code>./node test/parallel/test-cluster-worker-disconnect-on-error.js</code></li>
<li><code>./node test/parallel/test-cluster-worker-events.js</code></li>
<li><code>./node test/parallel/test-cluster-worker-forced-exit.js</code></li>
<li><code>./node test/parallel/test-cluster-worker-init.js</code></li>
<li><code>./node test/parallel/test-cluster-worker-no-exit.js</code></li>
<li><code>./node test/parallel/test-debugger-break.js</code></li>
<li><code>./node test/parallel/test-debugger-breakpoint-exists.js</code></li>
<li><code>./node test/parallel/test-debugger-clear-breakpoints.js</code></li>
<li><code>./node test/parallel/test-debugger-exceptions.js</code></li>
<li><code>./node test/parallel/test-debugger-exec-scope.mjs</code></li>
<li><code>./node test/parallel/test-debugger-extract-function-name.mjs</code></li>
<li><code>./node test/parallel/test-debugger-heap-profiler.js</code></li>
<li><code>./node test/parallel/test-debugger-help.mjs</code></li>
<li><code>./node test/parallel/test-debugger-list.js</code></li>
<li><code>./node test/parallel/test-debugger-object-type-remote-object.js</code></li>
<li><code>./node test/parallel/test-debugger-profile-command.js</code></li>
<li><code>./node test/parallel/test-debugger-profile.js</code></li>
<li><code>./node test/parallel/test-debugger-random-port-with-inspect-port.js</code></li>
<li><code>./node test/parallel/test-debugger-random-port.js</code></li>
<li><code>./node test/parallel/test-debugger-run-after-quit-restart.js</code></li>
<li><code>./node test/parallel/test-debugger-sb-before-load.js</code></li>
<li><code>./node test/parallel/test-debugger-scripts.js</code></li>
<li><code>./node test/parallel/test-debugger-set-context-line-number.mjs</code></li>
<li><code>./node test/parallel/test-debugger-use-strict.js</code></li>
<li><code>./node test/parallel/test-debugger-watch-validation.js</code></li>
<li><code>./node test/parallel/test-debugger-watchers.mjs</code></li>
<li><code>./node test/parallel/test-diagnostics-channel-process.js</code></li>
<li><code>./node test/parallel/test-error-reporting.js</code></li>
<li><code>./node test/parallel/test-inspector-debug-brk-flag.js</code></li>
<li><code>./node test/parallel/test-inspector-exception.js</code></li>
<li><code>./node test/parallel/test-inspector.js</code></li>
<li><code>./node test/parallel/test-module-main-preserve-symlinks-fail.js</code></li>
<li><code>./node test/parallel/test-node-output-console.mjs</code></li>
<li><code>./node test/parallel/test-node-output-errors.mjs</code></li>
<li><code>./node test/parallel/test-node-output-sourcemaps.mjs</code></li>
<li><code>./node test/parallel/test-node-output-vm.mjs</code></li>
<li><code>./node test/parallel/test-performance-nodetiming.js</code></li>
<li><code>./node test/parallel/test-process-external-stdio-close-spawn.js</code></li>
<li><code>./node test/parallel/test-process-external-stdio-close.js</code></li>
<li><code>./node test/parallel/test-process-uncaught-exception-monitor.js</code></li>
<li><code>./node test/parallel/test-promise-hook-create-hook.js</code></li>
<li><code>./node test/parallel/test-promise-hook-exceptions.js</code></li>
<li><code>./node test/parallel/test-promise-hook-on-after.js</code></li>
<li><code>./node test/parallel/test-promise-hook-on-resolve.js</code></li>
<li><code>./node test/parallel/test-runner-output.mjs</code></li>
<li><code>./node test/parallel/test-sync-io-option.js</code></li>
<li><code>./node test/parallel/test-v8-coverage.js</code></li>
<li><code>./node test/parallel/test-worker-debug.js</code></li>
<li><code>./node test/parallel/test-worker-load-file-with-extension-other-than-js.js</code></li>
<li><code>./node test/parallel/test-worker-message-port-inspect-during-init-hook.js</code></li>
<li><code>./node test/sequential/test-debugger-custom-port.js</code></li>
<li><code>./node test/sequential/test-debugger-launch.mjs</code></li>
<li><code>./node test/sequential/test-module-loading.js</code></li>
<li><code>./node test/sequential/test-perf-hooks.js</code></li>
<li><code>./node test/sequential/test-watch-mode.mjs</code></li>
<li><code>python test/pseudo-tty/../../tools/pseudo-tty.py ./node test/pseudo-tty/console_colors.js</code></li>
<li><code>python test/pseudo-tty/../../tools/pseudo-tty.py ./node test/pseudo-tty/test-fatal-error.js</code></li>
<li><code>python test/pseudo-tty/../../tools/pseudo-tty.py ./node test/pseudo-tty/test-trace-sigint.js</code></li>
</ul>
</details>

@nodejs/loaders @nodejs/tsc @nodejs/performance @nodejs/async_hooks