Repository: TypeScript
Issue: TypeScript watcher treats present files as missing after hitting FSEvents watcher limit
Labels: Needs Investigation

# Bug Report

### Search Terms

<!--
  What search terms did you use when trying to find an existing bug report?
  List them here so people in the future can find this one more easily.
-->

- EMFILE: too many open files, watch
- fsevents
- FSEventStreamStart

### 🕗 Version & Regression Information

<!-- When did you start seeing this bug occur?

"Bugs" that have existed in TS for a long time are very likely to be FAQs; refer to
  https://github.com/Microsoft/TypeScript/wiki/FAQ#common-bugs-that-arent-bugs

If possible, please try testing the nightly version of TS to see if it's already been fixed.
For npm: `typescript@next`
This is also the 'Nightly' version in the playground: http://www.typescriptlang.org/play/?ts=Nightly

Note: The TypeScript Playground can be used to try older versions of TypeScript.

Please keep and fill in the line that best applies:
-->
TypeScript version: 4.5.4
macOS version: 11.6.2
node version: v17.4.0

### 💻 Code

<!-- Please post the relevant code sample here as well-->
This issue is macOS specific.

```sh
git clone git@github.com:gluxon/typescript-emfile-repro
npm install
npm run repro
```

The repository above contains 2050 project references. Each of these projects have only an empty `index.ts` file to keep the repro simple.

The solution style tsconfig at the root is set to use fsevents.

```jsonc
{
  "watchOptions": {
    "watchFile": "useFsEventsOnParentDirectory",
    "watchDirectory": "useFsEvents"
  },
  "references": [
    // ...
  ]
}
```

The 2050 project references may seem unreasonable, but in a large monorepo this becomes more realistic when factoring in `node_modules` and nested watchers. (See below.) The real world repository this issue appeared on has around 2 million lines of TypeScript and ~800 packages declaring project references on each other.

### 🙁 Actual behavior

<!-- What happened, and why it was wrong -->

When running the repro linked above, `tsc` eventually switches all existing watchers to a `MissingFileSystemEntryWatcher`.

```
[11:48:15 AM] Found 0 errors. Watching for file changes.

DirectoryWatcher:: Added:: WatchInfo: /Volumes/git/typescript-emfile-repro 1 {"watchFile":5,"watchDirectory":0} Wild card directory
Elapsed:: 0.4360110005363822ms DirectoryWatcher:: Added:: WatchInfo: /Volumes/git/typescript-emfile-repro 1 {"watchFile":5,"watchDirectory":0} Wild card directory
sysLog:: /Volumes/git/typescript-emfile-repro:: Changing watcher to MissingFileSystemEntryWatcher
sysLog:: /Volumes/git/typescript-emfile-repro/packages/0:: Changing watcher to MissingFileSystemEntryWatcher
sysLog:: /Volumes/git/typescript-emfile-repro/packages/0:: Changing watcher to MissingFileSystemEntryWatcher
```

This behavior comes from the `watchPresentFileSystemEntry` function here.

https://github.com/microsoft/TypeScript/blob/5e09e86612f775785137b74482ba77764e22246e/src/compiler/sys.ts#L1633-L1641

Updating the `presentWatcher.on("error")` listener to print the error shows:

```ts
presentWatcher.on("error", (err) => {
  console.error(err);
  invokeCallbackAndUpdateWatcher(watchMissingFileSystemEntry);
});
```

```
Error: EMFILE: too many open files, watch
    at FSEvent.FSWatcher._handle.onchange (internal/fs/watchers.js:168:28)
Emitted 'error' event on FSWatcher instance at:
    at FSEvent.FSWatcher._handle.onchange (internal/fs/watchers.js:174:12) {
  errno: -24,
  syscall: 'watch',
  code: 'EMFILE',
  filename: null
}
```

Once `EMFILE` errors start emitting here, TypeScript no longer responds to any `.ts` file edits in in the repository since `watchMissingFileSystemEntry` only responds to `FileWatcherEventKind.Created`, and these files already exist.

https://github.com/microsoft/TypeScript/blob/5e09e86612f775785137b74482ba77764e22246e/src/compiler/sys.ts#L1687

From testing on my MacBook Pro (15-inch, 2018) on macOS 11.6.2, the number of `fs.watch()` calls that triggers `EMFILE` seems to be around ~4099.

### 🙂 Expected behavior

<!-- What you expected to happen instead, and why -->
I think it makes sense to update the error event listener to no longer assume all [`fs.FSWatcher`](https://nodejs.org/api/fs.html#class-fsfswatcher) errors are due to a missing file/directory, but I'm actually not sure what TypeScript should in response to a `EMFILE` after that. Ideally TypeScript falls back to using the polling watcher like it does for Linux.

https://github.com/microsoft/TypeScript/blob/5e09e86612f775785137b74482ba77764e22246e/src/compiler/sys.ts#L1644-L1651

However in my tests, once a Node.js program hits `EMFILE` on an [`fs.FSWatcher`](https://nodejs.org/api/fs.html#class-fsfswatcher) instance, all `FSWatcher` instances start throwing on `EMFILE` too. Because of the Node.js APIs here, I think TypeScript has to revert fully to polling mechanisms.

Hoping for ideas from the TypeScript team. Thanks in advance. 🙂

---

### Additional Research

#### Notes from libuv

Adding some additional research into Node.js and libuv.

I was surprised that `ulimit -n unlimited` had no effect on the repro. It turns out the `EMFILE` error here is from `libuv`, not a syscall.

https://github.com/libuv/libuv/blob/47e0c5c575e92a25e0da10fc25b2732942c929f3/src/unix/fsevents.c#L380-L384
```js
if (!pFSEventStreamStart(ref)) {
  pFSEventStreamInvalidate(ref);
  pFSEventStreamRelease(ref);
  return UV_EMFILE;
}
```

In Node.js land that error gets emitted here:

https://github.com/nodejs/node/blob/ec1364b6ae8f3be9054af5c56c7bf7c2d1b7c735/lib/internal/fs/watchers.js#L204-L210

```js
const error = uvException({
  errno: status,
  syscall: 'watch',
  path: filename
});
error.filename = filename;
this.emit('error', error);
```

Looking at [`FSEventStreamStart` documentation from Apple](https://developer.apple.com/documentation/coreservices/1448000-fseventstreamstart):

> Return Value
>
> True if it succeeds, otherwise False if it fails. It ought to always succeed, but in the event it does not then your code should fall back to performing recursive scans of the directories of interest as appropriate.

#### fs.watch Logging

Tangential to this issue, it may be possible for TypeScript to optimize its `fs.watch` calls. In the case below, the `src` is set to be recursively watched, but TypeScript ends up adding a few more watchers under it.

```js
fs.watch("/Volumes/example/packages/foo", { persistent: true, recursive: false });
fs.watch("/Volumes/example/packages/foo/src", { persistent: true, recursive: true });
fs.watch("/Volumes/example/packages/foo/typings", { persistent: true, recursive: true });
fs.watch("/Volumes/example/packages/foo/src", { persistent: true, recursive: false });
fs.watch("/Volumes/example/packages/foo/src/__tests__", { persistent: true, recursive: false });
fs.watch("/Volumes/example/packages/foo/src/bar", { persistent: true, recursive: false });
fs.watch("/Volumes/example/packages/foo/src/bar/__tests__", { persistent: true, recursive: false });
fs.watch("/Volumes/example/packages/foo/typings", { persistent: true, recursive: false });
```

From debugging, it does appear to be the number of `fs.watch` calls made that triggers the `EMFILE` error, rather than the number of files in a directory or its storage size. For example, watching the root of a large monorepo package does not cause `EMFILE` in the repro:

```js
fs.watch("/Volumes/git/typescript-emfile-repro", { persistent: true, recursive: true })
```