Repository: neovim
Issue: simplify remote plugins, massively
Labels: enhancement,api,plugin,remote-plugin

# Problem

The "remote plugin" model currently is too complex, and 99% of the complexity provides very little benefit.

- Lifecycle is complicated, unnecessarily hard to test, troubleshoot, and explain to plugin authors.
- _Every_ "remote plugin host" must implement 1000s of lines of code to solve the _same problem_ in _every API client_. This is a pointless waste of time.
    - Example: _all of the code below_ is just for the "remote plugin" implementation in:
        - `node-client`:
            - [neovim/src/plugin/](https://github.com/neovim/node-client/tree/e03e409a5e61713259148cb5f83e1b85420a9343/packages/neovim/src/plugin)
            - [decorators/src/plugin/](https://github.com/neovim/node-client/tree/e03e409a5e61713259148cb5f83e1b85420a9343/packages/decorators/src/plugin)
            - [neovim/src/host/](https://github.com/neovim/node-client/tree/e03e409a5e61713259148cb5f83e1b85420a9343/packages/neovim/src/host)
        - `go-client`:
            - [nvim/plugin/](https://github.com/neovim/go-client/tree/2163acbdee29ae71daea9e154219de5bb3ef9dc7/nvim/plugin)
    - The benefits of the extra code are
        1. "multi-tenancy" (one node process for all node rplugins).
        2. "decorators" can be used in the remote module to define Nvim commands/autocmds.
- Too many moving parts, too many concepts.
    - why is a "plugin host" needed?
    - why can't I just call `vim.rplugin('node', 'path/to/index.js')`, then call functions defined in the remote process?
    - why do I need "decorators" (`@pynvim.plugin`) ? why can't I just use Lua to define commands/events in `plugin/foo.lua`?
        - `@pynvim.command('MyCommand', …)` vs Lua `vim.api.nvim_create_user_command('MyCommand', function(...) vim.rpcrequest('MyCommand', ...))`
- `:UpdateRemotePlugins` and the "manifest" are extra state that must be resolved, refreshed, and healthchecked.

# Solution

- Reposition "remote plugins" as "remote modules". For example, a Node.js "remote plugin" is just a Node.js module that (1) imports ["neovim"](https://github.com/neovim/node-client/), (2) can be started as a Nvim RPC client, and (3) handles requests from Nvim.
    - Users no longer need to globally install `neovim-node-host`.
    - Client defines RPC method handlers via `setHandler()`.
    - Client calls `setup()` (placeholder name) which attaches and calls `nvim_set_client_info()` with the `methods` defined by `setHandler()`.
- Eliminate the "host" / "client" distinction. The client is the host (it can handle requests from Nvim).
- Drop "multi tenancy". Each plugin starts its own client ("host"). Plugins do not share a "host".
- Drop host-specific "sugar" for registering Nvim-side commands/autocmds (aka "bindings"). Plugins define commands/autocmds in Lua, like any other Nvim plugin. The commands call methods in the remote module.

# Implementation

<details>
<summary>Implementation details</summary>

From `doc/remote_plugin.txt`:

> Plugin hosts ... take care of most boilerplate involved in defining commands, autocmds, and functions implemented over |RPC| connections. Hosts are loaded only when one of their registered plugins require it.

We can address the above use-cases as follows:

1. Remote-plugins are just plain Lua plugins (`plugin/foo.lua`) that start API clients and call functions on the client.
    - Eliminates all the unnecessary complexity of trying to find `foo.js`/`foo.py`/… plugin "mains".
2. Make it easy from _Lua_ to start _any_ API client and treat it as a "remote host".
    - One (API client) process per plugin.
        - **Future:** Consider "sharing" client processes in the future. Out of scope for now.
    - Eliminates all the redundant, per-platform (js/py/…) impls that implement a "remote host".
    - Drop the concept of a "plugin host". We only need _plain old API clients_, not a "plugin host".
    - Nvim Lua stdlib will provide (once—_not_ in every API client!) an ergonomic interface to:
        1. start a "known" API client
        2. allow the caller to specify the path to the remote-plugin "module" or "main.
        3. define commands/autocmds/functions that call remote functions.
            - "Decorators" are not need for this! If it's painful to do this in Lua, _fix that once, in Nvim_ instead of "fixing" it N times in every API client!
    - Examples (compare to `:help remote-plugin-example`):
      - [go-client example](https://github.com/neovim/go-client/pull/172/files)
      - pynvim example:
        "Remote" Python code:
        ```python
        import pynvim

        def main():
            pynvim.setHandler('command_handler', command_handler)
            pynvim.setHandler('autocmd_handler', autocmd_handler)
            pynvim.setHandler('function_handler', function_handler)
            # (placeholder name, TBD) 
            # Attaches and calls `nvim_set_client_info()` with the `methods` defined by `setHandler()`.
            pynvim.setup()
        main()
        ```
        "Local" Lua code:
        ```lua
        local rplugin = vim.rplugin('python', 'path/to/init.py')

        vim.api.nvim_create_user_command('Cmd',
          function(args)
            vim.rpcrequest(rplugin.chan_id, 'command_handler', args.args, args.range)
          end,
          { nargs = '*' })
        vim.api.nvim_create_autocmd({'BufEnter'}, {
          pattern = {"*.c", "*.h"},
          callback = function(ev)
            vim.rpcrequest(rplugin.chan_id, 'autocmd_handler', ev.file)
          end
        })
        function my_func(...)
          vim.rpcrequest(rplugin.chan_id, 'function_handler', {...})
        end
        ```
3. With the above design...
    - ELIMINATES:
        - a fuckton of redundant [documentation](https://github.com/neovim/node-client?tab=readme-ov-file#remote-plugin-api) explaining crap like decorators, the "sync" flag, Remote plugin manifest, `remote#host#RegisterPlugin`, "bootstrapping" details.
        - a fuckton of code: [neovim/src/plugin](https://github.com/neovim/node-client/tree/e03e409a5e61713259148cb5f83e1b85420a9343/packages/neovim/src/plugin) [decorators/src/plugin](https://github.com/neovim/node-client/tree/e03e409a5e61713259148cb5f83e1b85420a9343/packages/decorators/src/plugin) [neovim/src/host](https://github.com/neovim/node-client/tree/e03e409a5e61713259148cb5f83e1b85420a9343/packages/neovim/src/host)
        - `rplugin/node/` runtimepath directory
        - "registration", `:UpdateRemotePlugins`, rplugin "manifest"
        - `remote#host#PluginsForHost`
        - `NvimPlugin.registerFunction`, `NvimPlugin.registerAutocmd`, `NvimPlugin.registerCommand`
    - STILL NEEDED:
        - instead of `NvimPlugin`, the remote plugin uses the exact same `NvimClient` type that is returned by `attach()`.
          A remote plugin is just a library that operates on the same old `NvimClient` that any other API client operates on.
        - `provider#Poll()`
        - `detect()`
        - `require()`
    - GAINS:
        - The "bootstrapping" is now extremely obvioius and the plugin implementor fully controls when to call `vim.rplugin()` from Lua.
        - `vim.rplugin()` loads the plugin "main" and returns a channel:
          1. find the platform interpreter (`node`, `python`, …)
          2. start the interpreter with a stdio RPC channel, targeting the plugin "main" file.
              - The plugin "main" file is expected to `attach()` to stdio, and use the resulting `NvimClient` to serve requests.
          3. calls `provider#Poll()` until success.
          4. returns a channel id.
        - NO sugar for creating commands/functions/autocmds that connect to the remote plugin.
          If creating commands/functions/autocmds is cumbersome we should fix that IN GENERAL, not only for remote plugins.
          ```
          vim.api.nvim_create_autocmd({'BufEnter'}, {
            pattern = {"*.c", "*.h"},
            callback = function(ev)
              vim.rpcrequest(rplugin.chan_id, 'autocmd_handler', ev.file)
            end
          })
          ```

</details>

## FAQ

- How does a plugin avoid loading the interpreter (slow) on startup?
    - To "share" an API client we could add an optional `ns` (namespace) parameter to `vim.rplugin()`, then it could be called on-demand and re-uses the existing channel if found.
        - This doesn't need to be solved right now, can just use `jobstart()`.


## Related

- The above proposal has [similarities to nvim-yarp](https://github.com/neovim/neovim/issues/5532#issuecomment-336474810) cc @roxma
    - But unlike nvim-yarp, we can go further and eliminate the concept of "registering" anything.


# Work plan

- [ ] Add a util function that migrates legacy plugins using their ["specs"](https://github.com/neovim/neovim/blob/d9af8c2431b7c6395abece8104be6a47d6f61f39/runtime/autoload/remote/host.vim#L132).
- [ ] Deprecate `:UpdateRemotePlugins`.
    - point to `:help remote-plugin-migrate`
    - ask user to report legacy plugins that haven't migrated yet: https://github.com/neovim/neovim/issues/29270
- [ ] Update the core API clients (implement `addHandler()`; `setup()` in the client automatically calls `nvim_set_client_info()` with the `methods` defined by `addHandler()`; let `jobstart()` invoke the module directly):
    - go-client https://github.com/neovim/go-client/issues/167
    - node-client https://github.com/neovim/node-client/pull/344
    - pynvim https://github.com/neovim/pynvim/issues/567
    - ruby client https://github.com/neovim/neovim-ruby/issues/108
- [ ] Update handling of `g:node_host_prog`, so it can point to `node`
    - Transitional phase: `neovim-node-host` will continue to be accepted; the path to `node` will be derived by inspecting the shebang in `neovim-node-host`.
- [ ] Update handling of `g:ruby_host_prog`, so it can point to `ruby`
    - Transitional phase: `neovim-ruby-host` will continue to be accepted; the path to `ruby` will be derived by inspecting the shebang in `neovim-ruby-host`.
- [ ] Update `:checkhealth`.
- [ ] Update `:help remote-plugin`.
- [ ] (Nvim 0.12) Remove old rplugin-related code
- [ ] (Nvim 0.12) Require `g:node_host_prog` to point to `node`, remove support for `neovim-node-host`.
- [ ] (Nvim 0.12) Require `g:ruby_host_prog` to point to `ruby`, remove support for `neovim-ruby-host`.