Repository: neovim
Issue: Use language daemons for language providers
Labels: needs:design

It is not unknown that `:python` and other languages startup is being slow now due to having to check and start external programs. E.g. with `init.vim` as simple as

```VimL
" let g:powerline_old_vim = 1
set laststatus=2
python import vim
python from powerline.vim import setup as powerline_setup
python powerline_setup(gvars=globals())
python del powerline_setup

set guicursor=
```

(using `editors-refactor` branch of powerline, powerline/powerline#1507) Vim takes 54 ms to source that, Neovim takes 325 ms, all with system FS caches up, worse without. The problem is that each Neovim has to start Python by itself.

I have a suggestion thus: instead of having architecture with which one Neovim has one Python which it directly communicates with add a “language daemon”. The idea is that “language daemon” starts once and creates a socket in a set location used to communicate with Neovim, when Neovim needs `:python` it starts a daemon if it was not already started (decreased performance, but only at startup, and “starting a daemon” may be a per-user systemd service), then on `:python` it just tells daemon that Neovim client with ID X (Neovim socket, but generally just any unique ID) needs to run `:python` command Y. If such ID is unknown to a language server then it spawns a new Python child process and relay requested command there, if it is known it either relays requested command or gives some details regarding the fate of the child (better error reporting if Python crashed).

Additionally if relaying does not work well in term of performance it should be possible to use language server to communicate direct interaction, then in “channel is broken” cases it should also be possible to get some details from the language server.

The main ideas here are

1. `fork()` is much cheaper then `fork()+exec()`.
2. Need not run checks regarding the availability and version of `neovim` packages more then once.

Main problems:

1. To update `neovim` package one will need to restart daemon.
2. Handling virtualenvs is going to be complicated.
3. Actually Neovim can’t just “request daemon to run command Y”, need to transfer environment variables at first command.

Both problems may be solved by another complication: make daemon a daemon manager: daemon manager is communicating with Neovim like daemon in the suggestion, but instead it spawns daemons  (key used to determine whether new daemon is needed: periodically checked `neovim` package version plus significant environment variables like where virtualenv is defined, or path to Python executable (`$PATH`, but only if it really changes executable file location), or Python package location (`$PYTHONPATH`); insignificant may just be changed on language process startup) using `fork()+exec()` (i.e. `subprocess`), daemons themselves only `fork()` to provide language processes.

Note that while my suggestion may bring down startup times significantly and provide better error reporting, this complicates code a lot and there are different solutions possible for error reporting in case of crashed language process.