Omnipytent 1.3.0: Async Tasks and Selection UIs
Tags: vim, workflow, automation
Published: 2018-11-15 23:58:58

---
title: Omnipytent 1.3.0: Async Tasks and Selection UIs
published: true
description: Explaining Omnipytent 1.3.0's new feature - async tasks
tags: vim,workflow,automation
---

If you don't know what [Omnipytent](https://github.com/idanarye/vim-omnipytent) is, read this first: {% link idanarye/omnipytent-5g5l %} 

Version 1.3.0 of [Omnipytent](https://github.com/idanarye/vim-omnipytent) introduces a new concept - async tasks. In this post I'll try to explain what are async tasks and what are they good for.

# The problem: single-threaded, event loop based UI

(you can skip this section if you already understand why we need async tasks)

Vim has a single-threaded architecture. When you perform an operation (by typing a key, running a run-mode command, using `:autocmd` etc.) that operation takes over the thread, and nothing else can get updated. The operation may receive input from the user (e.g. with `input()` or `getchar()`) or update the TUI (e.g. with `:echo` or by running a shell command with `:!`) but Vim's event loop itself is stuck until the operation is finished. You can't run other commands, which means that there is nothing to update the TUI, but even the jobs and terminals can't read anything from their streams while that operation is running.

Your everyday Vim operations are quick and "atomic" enough for this to not be a problem. Vim can't do anything else when you type `w` twe needo jump to the next word, but this happens so fast that it doesn't matter, and you don't need Vim do do anything else during. If you are running shell commands, on the other end, they can take quite long and you'd have to patiently wait for them to finish. Luckily, Neovim and Vim8 have jobs and terminals, so you can just launch it and it runs in the background.

But what if you need the "result" of the command you ran in the terminal? [fzf](https://github.com/junegunn/fzf), for example, is running `fzf` in a terminal, and use the result to open the chosen file. This can't be done in a single Vim function invocation - it needs in one function to start the terminal, then yield executing back to the event loop, and once `fzf` exist run another function to deal with the result. If it was done in the same function that launched the terminal, Vim would be unable to update the terminal buffer and the user wouldn't be able to use it.

And it's not just shell commands - sometimes you just need to user to use Vim's UI itself. A good example is [fugitive.vim](https://github.com/tpope/vim-fugitive)'s `:Gcommit` command. It opens a new buffer for the user to type the commit message in, and when they save and close that buffer the plugin creates a new Git commit with that commit message. Until the function called by `:Gcommit` finishes it hogs the event loop and the user can't write the commit message nor can they save and close the window, so `:Gcommit` must terminate before that and another function must be called when the window is close to finish the process.

Plugins like fugitive or the one bundled with fzf are registering callbacks to implement that behavior. It's not that hard to do, but it is quite cumbersome (even more than in JS!) and requires some familiarity with Vimscript and Vim's architecture. Omnipytent tasks are supposed to be simple and streamlines, so this callback registration is too much - we need something simpler. And that's where async tasks come in.

# The idea: generator based async tasks

(you can skip this section if you don't care how they work, and go directly to the next sections for examples on how to use them)

A generator function in Python is a function that has at least one `yield` expression. If you want to learn more about it [read the Python docs](https://docs.python.org/3/reference/expressions.html#yield-expressions) - for our purpose it's enough to mention that the `yield` yields both a value and the execution itself - so if a callee `yield`s, the execution goes back to the caller which can chose, at a later point, to resume the callee from where it `yield`ed.

This was used for async IO before the `async` and `await` keywords were introduced in Python 3.5, and since Omnipytent wants to support older versions it uses `yield` for its async tasks.

An async task looks something like this:

```python
@task
def my_async_task(ctx):
    do_something()
    result = yield ASYNC_COMMAND()
    followup(result)
```

When the task `yield`s, `ASYNC_COMMAND()` does three things:
* It prepares of the command (e.g. open a window or start a terminal)
* It registers itself in Omnipytent.
* It registers a callback/`:autocmd` in Vim to resume itself once done.

After that - `:OP my_async_task` terminates and control goes back to Vim's event loop. But `my_async_task` itself is not terminated yet - at some point it will be resumed and the task will continue and do the followup.

And of course - you can `yield` another async command later in the task.

Omnipytent comes bundled in with some useful async commands. If you need to create your own - refer to `:help omnipytent-creating-AsyncCommand`.

# `INPUT_BUFFER` - basic async user input

For the example, I'll use the same example project from the first post - [Spring's example pet clinic web application](https://github.com/spring-projects/spring-petclinic). Lets say we want a task for adding animal owners. The API is simple - a POST request with the details - but how will we prompt the user (which is us) to enter the fields?

Up until now, we'd have to use `input()` to allow the user to enter that data. With async tasks, we can do better:

```python
import requests, yaml

OWNER_FIELDS = ('firstName', 'lastName', 'address', 'city', 'telephone')


@task
def add_owner(ctx):
    empty_form = '\n'.join('%s: ' % field for field in OWNER_FIELDS)
    filled_form = yield INPUT_BUFFER(text=empty_form, filetype='yaml')
    parsed_form = yaml.load('\n'.join(filled_form))
    requests.post('http://localhost:8080/owners/new', data=parsed_form)
```

The important line is the second line of the task function - the one where we do the `yield`. `INPUT_BUFFER` is one of the async commands bundled with Omnipytent, and it opens a new window with a buffer for the user to edit. We set the original text - a YAMLish form with blank fields for the user to fill - and also set the file type to YAML to get some nice coloring. Then we `yield` this command object - and the execution control returns to Vim's event loop. Now the user can fill the form, and when they close it Omnipytent kicks back in and the task resumes - with `filled_form` set to the lines of the buffer the user filled.

This is how it looks in action:

![input buffer example](https://thepracticaldev.s3.amazonaws.com/i/v45lk1uespbz8g9dg1a2.gif)

# `CHOOSE` - the power of selection UIs

The primary motivation behind async tasks was to support selection UIs. These are usually fuzzy matchers like fzf or Unite, but I use the term _Selection UIs_ because this mechanism is not limited to tools that allow fuzzy search - any tool that provides a TUI for selecting from a list can fit.

Tests can accept arguments, but it's not always convenient to type the arguments. When the arguments are long, or complex, or hard to remember, and when there is an easy way to programmatically generate the list of possibilities, it is far more convenient for the user to filter and pick what they want with a selection UI.

For my next trick I'll need to add a missing method to the example project:

```java
// src/main/java/org/springframework/samples/petclinic/owner/OwnerController.java

    @GetMapping("/owners.json")
    public @ResponseBody List<Map<String, Object>> showResourcesVetList() {
        return this.owners.findByLastName("").stream().map(owner -> {
            Map<String, Object> entry = new HashMap<>();
            entry.put("id", owner.getId());
            entry.put("firstName", owner.getFirstName());
            entry.put("lastName", owner.getLastName());
            entry.put("address", owner.getAddress());
            entry.put("city", owner.getCity());
            entry.put("telephone", owner.getTelephone());
            return entry;
        }).collect(Collectors.toList());
    }
```

All it does is add a new path - `/owners.json` - that generates a JSON of all the owners. Nothing fancy, but the orig example project only supported getting them as an HTML page, which is harder to parse (unless you use a regex)

We want to write an Omnipytent task that reads that list with a GET request, lets the user pick one of the owners, edit it, and update it with a POST request. To do the selection, we are going to use another async command - `CHOOSE`:

```python
import json


@task
def edit_owner(ctx):
    entries = json.loads(requests.get('http://localhost:8080/owners.json').text)
    entry = yield CHOOSE(
        entries,
        fmt='{firstName} {lastName}'.format_map,
        preview=lambda entry: yaml.dump(entry, default_flow_style=False))

    owner_id = entry.pop('id')
    orig_form = yaml.dump(entry, default_flow_style=False)
    edited_form = yield INPUT_BUFFER(text=orig_form, filetype='yaml')
    parsed_form = yaml.load('\n'.join(edited_form))

    edit_url = 'http://localhost:8080/owners/%s/edit' % owner_id
    requests.post(edit_url, data=parsed_form)
```

`CHOOSE` runs whatever selection UI installed in your Vim. It first checks for [fzf](https://github.com/junegunn/fzf), then [Denite](https://github.com/Shougo/denite.nvim), [Unite](https://github.com/Shougo/unite.vim), [CtrlP](https://github.com/ctrlpvim/ctrlp.vim), and finally - if none of the above is available, it uses an `inputlist()` based selection UI. Or - if you have other preferences - you can set the selection UI with `g:omnipytent_selectionUI`.

Other than the list of options, we pass two more arguments to `CHOOSE`:

* `fmt`: The options are `dict`s but the selection UIs pick from lines. This argument is a function that formats each option into a line.
* `preview`: fzf, Denite and Unite support a preview of the items, and this argument is a function for rendering that preview.

After we yield `CHOOSE`, we get back the picked option and use it to display an `INPUT_BUFFER` - which I already explained earlier.

Let's see it at work:

![selection UI example](https://thepracticaldev.s3.amazonaws.com/i/0qgrv68jaryf4h7f8wvw.gif)

# Multi selection and generator options tasks - an actually motivational example

The previous examples were a nice way to show the power of async tasks, but they are not really something you'd write Omnipytent tasks for. If you want to actually add and edit owners you'd use the web application, and if you need a quick way to run these while developing to test your code, editing the text buffer each and every time is a bit cumbersome.

So how about something you actually want to create a task for? How about... running tests?

Running tests is a great use case for `CHOOSE`. When I work on a piece of code I often want to run a test that checks it. I don't want to run **all** the tests, because it'll take too long and will generate too much output. I want to run just this specific test and see just its output - which will reflect the changes I did to the code.

Test names are often long - they need to encode the name of the class/module that contains the test, the name of the test function itself, and sometimes the parametrization. Writing the full name each time is cumbersome. Writing that name inside the test makes it easier to run, but now we have to edit the tasks file when we want another test - and figure the name of that test. Not that hard, but can now do better - we now have `CHOOSE`!

Decent project management tools usually have a command for listing all the tests, but this is Maven so we'll have to parse the files ourselves:

```python
import re


@task
def run_tests(ctx):
    pattern = re.compile(r'@Test\n\s*public void (\w+)\(')
    test_names = []
    for path in local.path('src/test').walk(lambda p: p.basename.endswith('.java')):
        for match in pattern.finditer(path.read('utf8')):
            test_names.append('#'.join((path.stem, match.group(1))))

    chosen_tests = yield CHOOSE(test_names, multi=True)

    cmd = local['mvn']['test']
    cmd = cmd['-Dtest=' + ','.join(chosen_tests), 'test']
    cmd & TERMINAL_PANEL
```

Note that there is a new argument to `CHOOSE`: `multi=True`. As you may have guessed from the name, it allows the user to select multiple options. Only fzf, Denite and Unite support this, but even with CtrlP and `inputlist()` it'll still return a list to keep some uniformity in the task.

We then join the chosen tests, and voila!

![multi selection example](https://thepracticaldev.s3.amazonaws.com/i/wo28my7fd82ddtgp4fsw.gif)

But... we still need to pick the test we want to run each time. Picking it with fzf is definitely better than typing it, but since we usually want to run the same test(s) many times when we work on the same area of the code, it could be nice if Omnipytent could remember our last choice.

Well - it can. Omnipytent already had `@task.options` that remembers the user's choice, but you could only pick one option and the option keys had to be hard-coded as local variables. Omnipytent 1.3.0 solves both these problems:

1. `@task.options` is now based on `CHOOSE` - so it can use more elaborate selection UIs. A new variant - `@task.options_multi` - allows you to pick multiple options. If you are using CtrlP or `inputlist()` and still want multiple choices you'll have to pass them as arguments. Or just upgrade to fzf/Denite/Unite.
2. If the task function is a generator, instead of using the local variables as options it uses the `yield`ed values as options.

This means we can split our `run_tests` into two tasks: `pick_tests` and `run_tests`. `pick_tests` will always prompt us to choose the tests, but `run_tests` will remember our last choice:

```python
@task.options_multi
def pick_tests(ctx):
    ctx.key(str)
    pattern = re.compile(r'@Test\n\s*public void (\w+)\(')
    for path in local.path('src/test').walk(lambda p: p.basename.endswith('.java')):
        for match in pattern.finditer(path.read('utf8')):
            yield '#'.join((path.stem, match.group(1)))

@task(pick_tests)
def run_tests(ctx):
    cmd = local['mvn']['test']
    cmd = cmd['-Dtest=' + ','.join(ctx.dep.pick_tests), 'test']
    cmd & TERMINAL_PANEL
```

Note the first line of `pick_tests`: `ctx.key(str)`. Because the `yield`ed options can be objects, we need a string keys of them, and `ctx.key` sets the function for picking these keys. The key must be deterministic, because these keys will be used to cache the choice. There is also `ctx.preview` for setting a preview function, but we don't need one here.

And here is how it works:

![multi_options example](https://thepracticaldev.s3.amazonaws.com/i/4mwtx6kzlaffrors1a22.gif)

# Conclusion

Omnipytent's goal was to allow micro-automation of simple project tasks. Async tasks allow you to add better UI to that automation, farther enhancing the power at your fingertips.