Repository: TypeScript
Issue: A minimal custom transformer plugin proposal
Labels: Suggestion,In Discussion,Fix Available,Rescheduled

In the TypeScript compiler, we have the concept of a "transformer". A transformer is a function which "transforms" one AST to another. TypeScript ships many such transforms, including all of the various transformation steps needed to downlevel newer syntax to older syntax, to convert from `import`/`export` to `require`/`module.exports`, and so on. Even the declaration emitter is a transformer, stripping away function bodies, filing in types, etc.

Since TypeScript 2.3, TypeScript has had the following API:

<!-- dprint-ignore-start -->

```ts
interface Program {
    emit(
        targetSourceFile?: SourceFile,
        writeFile?: WriteFileCallback,
        cancellationToken?: CancellationToken,
        emitOnlyDtsFiles?: boolean,
        customTransformers?: CustomTransformers, // <--- 👀
    ): EmitResult;
}

interface CustomTransformers {
    before?: (TransformerFactory<SourceFile> | CustomTransformerFactory)[];
    after?: (TransformerFactory<SourceFile> | CustomTransformerFactory)[];
    afterDeclarations?: (TransformerFactory<Bundle | SourceFile> | CustomTransformerFactory)[];
}
```

<!-- dprint-ignore-end -->

This API allows users to provide a set of "custom transformers" at `emit` time. These transformers run alongside TypeScript's own transformers, either before or after, depending on where they are placed in `CustomTransformers`.

However, while this API exists and does work, a major gotcha is that custom transformers can only be provided to the _API_. There is no way to use custom transformers if you want to use `tsc` alone.

This has been a long-standing issue; [#14419](https://github.com/microsoft/TypeScript/issues/14419) is the fourth most upvoted issue on the repo.

# What do people use custom transformers for?

This list can go on and on, but transformers are commonly used these days for:

- Generating serialization / deserialization / runtime type checks
  - `typia`,`ts-runtime-checks`, which top the [benchmark charts](https://moltar.github.io/typescript-runtime-type-benchmarks/), but there are many more
- Powering internationalization
  - `formatjs`
- Generating database code
  - `ts-graphql-plugin`
- Transforming paths (please don't do this one, though 😅)

I also know of many ideas which _want_ to be implemented via plugins, but are not willing to do so without official support, including other runtime type checks, tracing code, and so on.

# How do users currently use custom transformers?

Given the current constraints of custom transformers, there are two ways that downstream users have been able to use them:

1. Build TypeScript code via the API, either directly or by using some system which uses the API.
   - `webpack`, `rollup`'s TS plugins, `nx`, all provide configuration which allows users to specify a set of custom transformers.
1. Patch TypeScript.
   - `ttypescript`, `ts-patch` wholly replace TypeScript in a user's workflow.

The first one is obviously fine. The latter, eek!

# What changed?

In TypeScript 5.0, we shipped the biggest architectural / packaging change in TypeScript's history; conversion of the entire project to modules, including the bundling of our code. The relevant side-effects are twofold:

1. The structure of `tsc.js` (and other outputs) changed massively. This broke anyone who was patching the contents of the TypeScript package.
1. The API is no longer runtime patchable; we use `esbuild` to bundle our code and the objects it creates are not "configurable", correctly emulating how "real" ES modules would behave. Any runtime patches were already extremely fragile, only functioning due to the structure of our old `namespace` emit.

In response to this, I did [an analysis](https://gist.github.com/jakebailey/3e73c7aab4da2044a95121467aa3b4e9) of as many TS "patchers" as I could find, and found that there are actually very few reasons to patch TypeScript at all:

- `ttypescript`/`ts-patch`, which seem to be almost exclusively used to enable the use of custom transformers.
- Language service plugins, mainly injecting their own module resolution and custom source file types (`vue`/`volar`, `css`, etc).
- Build systems like `heft`.
- Yarn PnP.
- Those who want to tree shake out our parser (`prettier`).

For 5.0, I was able to fix many of these projects (or at least guide / warn maintainers ahead of time), but many of these patchers have no viable alternative.

We have been seriously considering ways that we can approach the problems that remain in hopes that we can eliminate the need for people to patch TypeScript. This includes (no promises):

- Custom transformers.
- Custom module resolution.
- A future API that actually allows a consumer to tree shake out _just_ a parser (or similar).

This issue intends to address the first bullet.

# A conservative, targeted proposal

In short, my proposal is to add to TypeScript the ability to define "custom transformer plugins". This new functionality is _targeted and minimal_. Its only intent is to satisfy those who just want to be able to add custom transformers.

Many details still have yet to be expanded on, but in short:

- These plugins are placed into the `plugins` array of `compilerOptions`, with a discriminator like `"type": "transformer"`.
  - This is the approach taken by `ttypescript`/`ts-patch`, and offers up a future base for additional `tsc` plugins (e.g. file watching plugins, module resolution plugins).
  - The choice in discriminator (`"type": "transformer"`, for now) is intended to avoid conflicting with existing plugins defined by `ttypescript` or `ts-patch`.
- Plugins would take a form similar to language service plugin, receiving the `ts` object, returning a factory that can be used to create a `CustomTransformers` object. The factory would receive the `Program`, which is required for any type-using plugins.
  - The `ts` object must be a part of the API for similar reasons to the language service plugins; the APIs available within the core compiler are very, very limited, and the bundle is _not_ the same as `typescript.js`.
  - The plugin object entry is passed to the plugin for extra configuration, like LS plugins.
- A new `tsclibrary.d.ts` file is added, describing the limited API within `tsc`. This is similar to `tsserverlibrary.d.ts`.
- When invoking `tsc`, you must pass `--allowPlugins` to opt into executing plugins. When using the API or tsserver, plugins are enabled by default.
  - No sandbox is present otherwise. If you currently use a patching-based method to use plugins, you're already taking matters into your own hand. If you currently use `webpack` or `rollup`, your config files are executable anyway.
  - `--allowPlugins` is also a part of the watch plugin PR.
- If multiple plugins are used, their returned `CustomTransformers` are merged.

There are almost assuredly other things that people want to be able to do with custom transformers or other plugins, however, I believe that we can serve the vast majority of those who want custom transformers with this API, and I do not want to let perfect be the enemy of the good.

In the future, we can expand `plugins` to do more, allowing the customization of other aspects of TypeScript, or simply add on to what custom transformer plugins can do. [Prior research](https://gist.github.com/rbuckton/090b2b5d27079dec5639e79c072a289e) has shown that there are more interested parties, which I believe that we can eventually support.

# An example

With the above proposal, a `tsconfig.json` would look like this:

```jsonc
{
    "compilerOptions": {
        "plugins": [
            { "type": "transformer", "path": "@jakebailey/cool-transformer" }
        ]
    }
}
```

The transformer would look something like:

```ts
import type * as ts from "typescript/tsclibrary";

const factory: CustomTransformersModuleFactory = ({ typescript }) => {
    return {
        create: ({ program }) => {
            return {
                before: [(context) => (file) => {/* ... */}],
            };
        },
    };
};

export = factory;
```

Of course, the details may change.

# Unsolved problems

- It's very unfortunate to have to define _yet another_ random `d.ts` API used for one thing.
  - Can we do better?
  - What if the executables were ESM?
- Like LS plugins, tsc plugins would need to be CJS.
  - Can we instead make use of `import`?
  - How do we do `async`? (This may already be "solved" in a prior WIP for plugins)
- Should we do any sort of performance metric counting for plugins?
  - Like LS plugins, can we just say "no warranty" when plugins are involved?
- How much of a problem is it going to be to need to use the `typescript` namespace passed into the plugin?
  - What happens to existing transformers which mistakenly import `typescript.js`'s API directly, even though that may not have always worked?
  - Will plugins unknowingly use the `typescript.js` API and rely on it, assuming users won't use `tsc`?
  - Should we try and attempt converting our binaries/libraries to ESM first?
- Is the proposed API too factory-y?
- Should we even do this?

# Please give feedback!

If you are currently using transformers, creating transformers, or otherwise, I would appreciate your feedback.