Repository: TypeScript
Issue: Support for pre-compilation & post-compilation build scripts in tsconfig.json
Labels: Suggestion,Needs Proposal

# Suggestion

## 🔍 Search Terms

`preprocessor`, `postprocessor`, `shell script`, `external script`, `precompilation`, `postcompilation`, `precompile`, `postcompile`, `worker`, `WebWorker`, `web worker`

Obviously the first term matched a lot of issues, but those were more in terms of preprocessor directives, not toolchain hooks.

## ✅ Viability Checklist

My suggestion meets these guidelines:

* [x] This wouldn't be a breaking change in existing TypeScript/JavaScript code
* [x] This wouldn't change the runtime behavior of existing JavaScript code
* [x] This could be implemented without emitting different JS based on the types of the expressions
* [x] This isn't a runtime feature (e.g. library functionality, non-ECMAScript syntax with JavaScript output, new syntax sugar for JS, etc.)
* [x] This feature would agree with the rest of [TypeScript's Design Goals](https://github.com/Microsoft/TypeScript/wiki/TypeScript-Design-Goals).
    - Possible conflict on non-goal 4, as it can be worked around (somewhat clumsily) in part by use of external tooling.
    - However, workarounds cannot overcome significant limitations for `--watch` builds without this feature.

## ⭐ Suggestion

New, optional configuration options in tsconfig.json:

```
  "compilerOptions": {
    "prebuild": "<external, terminal command>",
    "postbuild": "<external, terminal command>",
  }
```

Example: 

```
  "compilerOptions": {
    "prebuild": "./write_time-of-build.ts.sh"
  }
```

or

```
  "compilerOptions": {
    "postbuild": "./wrap-worker-code.sh"
  }
```

The first toy example would use a shell script to write `time-of-build.ts`, a file defining the exact UTC time that the build command was run.  This could then be used as part of the build itself, allowing the built product to include its contents and output this as part of an about "page"/"element".

The second example... well, that's the primary motivation for this proposal.

## 📃 Motivating Example

Now for a meaty case:  bundled WebWorker code.

**tl;dr**:  multi-level composite projects exist that can benefit from incremental builds, `--watch`, etc.  Due to library limitations, one level must be built first, and a little bit of `tsconfig.json`-triggered automatic external tooling would make all the difference for linking the two tiers together while still compatible with `tsc --watch` and `tsc -b`.

Now, for the long-form explanation:

We have a group of projects collectively targeting the standard DOM, and we've been in the process of converting to use of a composite-project configuration scheme.  (For further context, our codebase still uses namespaces, and we bundle-compile the main top-level products with `outFile`.)  The WebWorker's code is, conceptually, a mid-level product - it utilizes some lower-level projects and is most directly used by a 'wrapper' project designed to provide a DOM-friendly API for communication with the worker, handling all of the needed `postMessage` and `onMessage` patterns internally.

The issue we ran into:  how do we bundle WebWorker-space code with DOM-space code?  We took the 'standard' approach, so far as we know:

**Steps for WebWorker bundling**
1. You embed the WebWorker code in a wrapping function that preserves the JS (read:  TS build product).
2. Embed the wrapping function into the project that will construct the worker.
3. `<wrapped function>.toString()` allows you to retrieve the original JS "source" (read:  TS build product) at run-time.
4. Trim off the wrapper components.
5. Construct a blob from the now-unwrapped source.
6. Pass that constructed (text) blob as the source file for a WebWorker.

For a minimum-repro example of this approach, refer to [this example code](https://github.com/keymanapp/keyman/tree/2d0fed11d37aec94875054413a335d94fdefc69d/common/predictive-text/testing/two-stage-embedded-webworker).  It uses `tsc -p` rather than `-b`, but that won't affect its validity as a reference.  A little effort can convert it right over; it was written pre-3.7.  (It's what we used to validate the approach before adopting it on a larger scale.)

As far as raw JS goes, the approach works swimmingly.  That said, both our DOM projects and the WebWorker projects are TS, so it'd be nice if there were a way forward for this in `tsc`.  However, as far as compilation goes, `tsc` will throw an absolute fit trying to compile the two together directly because the two scenarios require _very_ different library inclusions.  We recognize that this is a very natural and likely _very_ necessary limitation, so it's reasonable to require some sort of workaround.  (For reference, before converting to composite builds, we always just included the lower-level build products - raw JS - and used a _lot_ of shell script tooling to manage everything.)  Hence, separate projects.

WebWorker project:

```
  "compilerOptions": {
    "lib": ["webworker", "es6"]
  }
```

The worker-interfacing project:

```
  "compilerOptions": {
    "lib": ["es6", "dom"]
  }
```

If we could reasonably author all of the WebWorker's code within a single file, maybe the wrapping aspect wouldn't be a big issue... but there's a lot of code, and so it's a multi-file project.  (Multiple classes, etc.  Monolithic files aren't the best development practice.)  As a result, there's no way we can directly build to a single function that wraps all worker related code via `tsc`.  (See steps 1 and 3 above.)  That said, `outFile` does allow us to build a single output file that we can externally wrap.  That gives us one important tool to proceed, and it already exists.

So, next we have a dedicated shell script (say, `build.sh`) that runs `tsc` (with `outFile`) and then creates a new file with a wrapped version.  The problem... is that for the higher-level projects, we now have a `tsconfig` `"include"` file that's a _build output from a shell script_.  We can't `"reference"` it in consuming `tsconfig.json` files - we've got that `"lib"` conflict to avoid, which would block step 2 above.  And we want to leverage composite-project TypeScript builds.

In case anyone came here suffering a similar problem, our current workaround:  as the worker's build output is in JS, we needed to rename it as a TS file and add `@ts-ignore` at the top of the wrapping file - otherwise, we'd get problems with a lack of internal type decoration, which would also block step 2 above.  After all, composite projects require declaration emit, and you can't emit well from wrapped raw JS.  The exact error we got before using `@ts-ignore`:  `error TS9005: Declaration emit for this file requires using private name 'xxx'. An explicit type annotation may unblock declaration emit.`  (One suggestion we received -`"jsCheck": false` - didn't prevent it.)  The wrapping file can then be added as an `"include"`, not a `"reference"`.

**Why it matters**

This leads to the main issue and the reason for this proposal:  because those projects have to ~reference~ include the build product, not the WebWorker's project itself, we're required to run its build-wrapping shell-script each time a change is made in that project.  `--watch` can't utilize this.  After all... `tsc`, by itself, doesn't know to check for any changes to the worker's project, as the _project_ is not _referenced_.  All higher level projects will be ~referencing~ including a build product of the worker project, not referencing its original source.  As a result, the top-level project can't adequately `watch` for code changes throughout and auto-update.  (I'm also curious about implications for incremental builds, though I suppose they could just hash the wrapped output file and note change, or lack thereof, that way?)

We'd really like it if there were a place in `tsconfig.json` where we could stick a call to that project's `build.sh` (which builds the wrapped worker) so that the top-level projects can be fully kept up to date from their respective top-level `tsc -b` composite-build calls.  One simple call to a simple shell script (via the proposed `tsconfig.json` `compilerOptions.postBuild` option) would be enough to auto-wrap the worker code for consumption by the next project in line, which could then include it.  --watch could be configured to watch for changes in that project's files, and that project could, in post-build, run the wrapping script automatically.  Consuming projects then automatically receive the updated build product via include, and _everything_ updates appropriately.

I will admit that an alternative approach would be to simply compile a dummy version of the wrapping function into the build product, then after the main `tsc` build, do a search-and-replace on it (via unique, identifying string) and dynamically insert the worker code in then.  That said, this approach still suffers the same issue - now we need a post-build script at the top level, rather than a pre-build script at the mid-level.  (And it's less "encapsulated", as far as the projects go.  We prefer wrapping the worker code in the project that directly consumes it if possible.)  Even then, I suppose there is some leeway - I don't see why a post-build on the mid-level couldn't be run as part of the top-level's post-build step as well, should a future implementation choose to go that far.

## 💻 Use Cases

See:  motivating example with WebWorkers.

Other cases:
- We currently use a similar approach to build a `version.inc.ts` file in order to embed variations on our product's current version (major, minor, patch, major.minor.patch, major.minor.patch-releaseTier, etc) into our projects at build time.
    - We find this useful for cross-language, cross-platform semantic versioning.  There are also C++, Obj-C, Java, and Swift projects all assigned the same version information at the same time... and they can't rely on package.json version information like a JS/TS project can.  (Especially for Mac projects, with their dependence on .plist files.)
        - This, in turn, is very helpful to us for preventing version confusion during customer support interactions.
    - Obviously, this part isn't relevant for the `--watch` aspects of this proposal, but it does benefit from the same approach.
        - A single `pre-build` to emit the version info from the common source into a TS file could then be compiled into the project and included in that level and in any higher-level projects.
- The hypothetical "time of build" case.

We've documented our main workaround above in terms of WebWorkers, but it also works pretty well for the `version.inc.ts` case; while there's no need to "wrap" raw JS output in TS, we still need to call a script to write TS - the dynamically-generated version `.ts` file - for inclusion in the projects that use it.  (So, a pre-`tsc` `write_version_info.sh` step.)  Obviously, this requires the main driver of the build to be a shell script that first runs the proposed "precompile" step, then runs the main `tsc` build.

Can _that_ be achieved by external tools?  Yes - relying on shell scripts that use `tsc` builds is enough for main builds of the project.

**However**, you lose the ability for **`--watch`** to auto-build updates when needed, which _particularly_ matters for the WebWorker scenario.  Consider how this ties into coverage for `--watch`-based unit testing solutions -- integration-style tests will not receive code updates.  We do have a few defined integration-style tests for our projects - tests to ensure that an outer project is able to communicate with and receive expected data from the eventual WebWorker.

### 💭 Other notes

Obviously, nomenclature and the exact manner of solution may be changed to best fit the vision of the TS team; this is simply the best way I know to explain the problem in a relatively straightforward and clear manner.  In writing this up, I realize I may be wanting composite builds to be... "a little more than they are", so to speak... but they do appear to be the best tool for the job right now, despite their limitations in this matter.

The core essence of our issue probably arises from the fact that we want to distribute a DOM project that depends on a WebWorker project in a single output-file bundle.  `tsc` can't do hybrid-`lib` builds, so we're forced to stage the build into 'layers'.  However, this layering blocks the ability of `--watch` to cope and properly watch anything at a 'lower level' of the build, so a full project `--watch` is currently impossible.

It may or may not also affect aspects of incremental building?  But that part's just a guess on my end.