Repository: go
Issue: x/tools/gopls: experiment with an option/capability for server-side file watching
Labels: gopls,Tools

Philosophically, file watching should be done by the client. In addition to the arguments [here](https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#workspace_didChangeWatchedFiles), client-side file watching means that the client is responsible for all state transitions on the server, which means the client owns the session lifecycle. This seems like a good idea.

Except that it doesn't work well. Not all clients support file watching, and those that do have spotty and inconsistent support. Sometimes, a file watching pattern that is performant for one client is prohibitively expensive on another, so much so that this is the only place in gopls where we have been forced to [specialize behavior](https://cs.opensource.google/go/x/tools/+/master:gopls/internal/cache/snapshot.go;l=967;drc=3c293ad67a98a86d273bf69c6a742f04a6a367a3) based on the client's user agent.

Recently, the cross platform [fsnotify](https://github.com/fsnotify/fsnotify) package [gained support](https://github.com/fsnotify/fsnotify/commit/a618f076c8cfc422e6d1f6f1f881af9152aa7c82) for recursive directory watching (on all platforms except macOS; thanks to @stapelberg for pointing this out). We've also heard from our friends maintaining the Dart LSP that server-side watching has worked well for them.

We should experiment with server-side file watching. In theory, it shouldn't be that hard to wire in:
- Add a new [internal gopls setting](https://cs.opensource.google/go/x/tools/+/master:gopls/internal/settings/settings.go;l=393;drc=01018ba9edc26f327f5c78d72e7596b6bf23c480) to control the experiment.
- Hook into [`server.updateWatchedDirectories`](https://cs.opensource.google/go/x/tools/+/master:gopls/internal/server/general.go;l=375;drc=9795facf6c2784362d043f791599d7ee49cc6cab) to manage glob patterns. I think it's sufficient to check server.Options inside updateWatchedDirectories, and perform state transitions based on the file watching setting in the previous bullet. (register/unregister client side watching as necessary, and manage the lifecycle of the server-side watcher).
- Call server.didModifyFiles on file notifications. See [server.DidChangeWatchedFiles](https://cs.opensource.google/go/x/tools/+/master:gopls/internal/server/text_synchronization.go;l=176;drc=d940b335673153bd68f2d82860c96d4472ab6832). In theory, we could even call `DidChangeWatchedFiles` on change events, but that is a bit inaccurate as file watcher events are not LSP events.

Once that is done, we should perform a bake off with VS Code's client-side watching. For example: perform a branch switch, and see whether server-side watching or client-side watching gets it right. Also check the CPU cost of file watching.

One possibility is that neither server-side nor client-side watching gets it right. This could be due to gopls bugs around large numbers of simultaneous state changes, or it could be due to bugs in file watching. With server-side watching, we actually have an opportunity to get it right. For example, we could add heuristics by which we validate state with additional polling, thereby enforcing eventual correctness.

CC @adonovan