Repository: neovim
Issue: Lua: structured concurrency, Promises, task pipelines
Labels: enhancement,lua,architecture,async

related: https://github.com/neovim/neovim/issues/11312

## Problem

There's no builtin abstraction for:

- Representing a task.
    - Example: `vim.system()` returns its own ad-hoc "task" that can be "awaited" via `:wait()`.
- Orchestrating "pipelines" (quasi monads?) of work ("tasks") and handling errors.
    - Example: shell-like task chains: `vim.fs.files():filter(..):map(function(f) vim.fs.delete(f) end)`
    - See also `vim.iter`.

## Expected behavior

- "Task" abstraction:
    - Maximally leveraging Lua coroutines + libuv. Only add concepts ("task", "promise") if absolutely needed.
    - Coroutines (or tasks that wrap coroutines) can be nested. ([ref](https://gregorias.github.io/posts/using-coroutines-in-neovim-lua/#addendum-whats-wrong-with-plenary-async))
    - Util to create an awaitable task from "normal" functions (cf. "promisify"?).
        - Don't want to call `await` everywhere, that is a horrible consequence of the JS async/await model. Instead consider Go's `go` for opt-in concurrency.
        - Can e.g. `vim.system()` be "promisified" without its knowledge? Or could it handle differently when it detects that it's in a coroutine? So `vim.system()` would be synchronous normally, but `vim.async(vim.system, ...):wait()` would be asynchronous.
            - See `cb_to_co()` from [this article](https://gregorias.github.io/posts/using-coroutines-in-neovim-lua/#callbackcoroutine-conversion).
    - Document (or generalize) ["coroutine to callback"](https://gregorias.github.io/posts/using-coroutines-in-neovim-lua/#coroutine-to-callback-conversion).
- Structured concurrency:
    - `await_all`, `await_any` (pseudo-names). See JS [Promise.all()](https://developer.mozilla.org/de/docs/Web/JavaScript/Reference/Global_Objects/Promise/all).
    - Tasks can be _canceled_.
    - Results (and failures) can be _aggregated_. (Can't do this with `jobwait()`!)
    - Failures/errors can be handled (possibly canceling the rest of the task tree).

## Implementations

- https://github.com/lewis6991/async.nvim
- https://github.com/nvim-neotest/nvim-nio "enables async I/O through higher level abstractions and utilities around coroutines"
- https://github.com/kevinhwang91/promise-async
- https://github.com/ms-jpq/lua-async-await

## Related

- https://github.com/sourcegraph/conc
- https://github.com/bitfield/script (example of "pipeline" interface)
- https://gregorias.github.io/posts/using-coroutines-in-neovim-lua/

### notes from [reddit discussion](https://old.reddit.com/r/neovim/comments/1g8lutp/coroutine_tutorial_for_neovim_lua/lt0rp9m/)

> Briefly speaking, without cancellability, a structured concurrency API can and should be based on fire-and-forget coroutine functions (fafcf). Anything else would likely either be reinventing the wheel or running into a non-composable mess that is currently Plenary’s async.
> 
> With pure fafcfs, you can have most of your requirements: ability to chain, start concurrent computations, wait for completion of arbitrary “fafcfs.” It’d be quite a expressive and elegant system.
> 
> As you notice, what’s missing is cancellability. That would require creating a special protocol that fafcfs need to conform to.
>
> After briefly looking at [async.nvim](https://github.com/lewis6991/async.nvim), it looks like it’s trying to build concurrency by stepping through chunks, which is kind of what Plenary’s async is trying to do.
> 
> Ideally, I’d like to try to achieve cancellability without resorting to reimplementing an event loop and a scheduler in Lua. Perhaps fafcfs with a special protocol for indicating that it shouldn’t proceed would be enough.
> 
> I believe that Neovim/Lua can take design hints from Python. In this regard, Lua fafcfs are [Python coroutines](https://docs.python.org/3/library/asyncio-task.html#awaitables). They can be nested freely. If you want to a cancellable computation, you need to use [Tasks](https://docs.python.org/3/library/asyncio-task.html#creating-tasks), which builds upon coroutines to provide a richer interface.