How and why do we bundle zx?
Tags: zx, node, javascript, tooling
Published: 2024-05-06 08:12:21

When [zx](https://github.com/google/zx) first appeared, it was a [tiny esm script](https://github.com/google/zx/commit/ded16dd382b393a54a7aa7900f41ac306e4f27b8) that just proposed a new idea for how `child_process.spawn` API could be enhanced with [string template literals](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_literals). 

```ts
const result = await $`call smth --with any ${args}`
```
As we now know, this was exactly what JS folks had been looking for for many years :-). The main goal was to improve the developer experience, so we focused on finding the best combination of utils to solve everyday problems, expand the area of application.

## Short history
The project went through 8 major releases, but in the context of the assembly issues, there are only 3 significant ones.

### Packed as CommonJS
Despite the fact that the future belongs to ESM, many users have to remain in the notation of their legacy codebase (without any irony). So we have added compatibility with the `require` API for them.

https://github.com/google/zx/commit/303feea3cd0d95e8d733ee72b96b3316ea5827aa

### Dropped CJS support
On the other hand, there were libraries on which zx depend: and only the latest _modern_ major versions received patches, including vulnerabilities fixes. In fact we inherited CVE issues transitively, which was completely unacceptable. And we switched back to ESM.

https://github.com/google/zx/commit/5578d67ebae2049d430bf6489784f6a2aa47b0b1

### Migrated to TypeScript
As the code base grew, it became harder for us to ensure consistency of internal contracts. Adding typing was the solution, but it also made us dependent on transpilation and we could no longer control the package artifact in detail.

https://github.com/google/zx/commit/8c2aea89fe7c65767d29c52bfb48f6a46af01864

## Problems

### Size
Filling the zx with new functionality, we continued to expand the dependency tree, which required their own dependencies, etc. Installation time increased significantly. And at some point it became impossible to ignore: [zx#712](https://github.com/google/zx/issues/712), [zx#669](https://github.com/google/zx/issues/669)

| 1.15.2 | 2.1.0 | 3.1.0 | 4.3.0 | 5.3.0 | 6.2.5 | 7.2.3 |
|---|---|---|---|---|---|---|
| 1.38 MB | 3.01 MB | 3.01 MB | 3.89 MB | 10.5 MB | 10.7 MB | 16.3 MB |

### Compatibility
When we define [node engine](https://github.com/google/zx/commit/fadc6a31517f22afc332fd33ac47a40f2bda5f78) requirement we limit not users, but ourselves in what platform API we can use. But this in no way prevents our dependencies from [pursuing their own policies](https://github.com/sindresorhus/globby/blob/main/package.json#L20) which affects zx.

### Stability & Security
Reproducible setups are important. Especially for utilities of our type. We can try to pin down versions, but if there is a caret (^, ~) or an asterisk (*) at the secondary level, this introduces node_modules variability. Lockfiles can mitigate the problem, but not for npx/yarn dlx execution case.

## Bundle
At first we wanted to just [get rid of all](https://github.com/google/zx/issues/550) the helper utilities. Keep only the kernel, but this would mean a loss of backward compatibility. We needed some efficient code processing instead with recomposition and tree-shaking. We needed a bundler. But which one? Our testing approach [relies on targets](https://github.com/google/zx/blob/main/test/core.test.js#L21), not sources. We rebuilt the project frequently, speed was critical requirement. In essence, we chose a solution from a couple of among all available alternatives: [esbuild](https://esbuild.github.io/) and [parcel](https://parceljs.org/). Esbuild won. **Specifically in our case**, it proved to be more productive and customizable.

### js
_Customizable_ means that we found a way to adapt the tool for our tasks. And a lot had to be improved, including writing our own plugins:

1. Esbuild provides [code-splitting](https://esbuild.github.io/api/#splitting) with dynamic imports to reuse common code chunks. But not is static form. The [manual chunks API](https://github.com/evanw/esbuild/issues/207) is missing.<br/>[esbuild-plugin-entry-chunks](https://github.com/antongolub/misc/tree/master/packages/esbuild/plugin-entry-chunks)

2. By default, esbuild injects helpers into each cjs module. It's fine, when you have just a few, but definitely not when there are many.<br/>[esbuild-plugin-extract-helpers](https://github.com/antongolub/misc/tree/master/packages/esbuild/plugin-extract-helpers)

3. We also wanted to bring back cjs support, but at the same time avoid duplicating code in two versions of the bundles.<br/>[esbuild-plugin-hybrid-export](https://github.com/antongolub/misc/tree/master/packages/esbuild/plugin-hybrid-export)

4. Esbuild and its plugins mostly focus on sources processing, but sometimes additional modification is also necessary for dependencies or bundles:

  * Polyfill injects: [esbuild#2840](https://github.com/evanw/esbuild/issues/2840), [esbuild#3517](https://github.com/evanw/esbuild/issues/3517), [esbuild#3099](https://github.com/evanw/esbuild/issues/3099)
  * Custom patches: [esbuild#3360](https://github.com/evanw/esbuild/issues/3360)
  * Dynamic banners: [esbuild#3291](https://github.com/evanw/esbuild/issues/3291)
  * Importable helpers: [esbuild#1230](https://github.com/evanw/esbuild/issues/1230).

[esbuild-plugin-transform-hook](https://github.com/antongolub/misc/tree/master/packages/esbuild/plugin-transform-hook)

### d.ts
While we were fighting against the modules, we forgot one small detail - their built-in typings. Esbuild can't do this at all yet. Unbelievable, but the tsc, native TS compiler, also does not provide a [typings concat feature](https://github.com/microsoft/TypeScript/issues/4433). Got around this problem: we've introduced a [utility to combine typings](https://github.com/antongolub/misc/tree/master/packages/dep/tsc-dts-fix) of zx own code, and applied some _monkey_ patches for external libdefs squashed via [dts-bundle-generator](https://github.com/timocov/dts-bundle-generator).

## Results
There is no silver bullet, all tools and approaches have limitations. Summary:
* Install size reduced: **16Mb → 1Mb**
* Reproducible npx invocations
* Hybrid package: ESM and CJS
* Wide range of supported Node.js versions 12...22
* Easy to audit: complete code in one place

Perhaps our efforts are not commensurate with the profit. But we gained invaluable experience. Now we understand the entire zx code in detail down to each char.

[zx build setup](https://github.com/google/zx/tree/main/scripts)

{% embed https://github.com/google/zx %}