Moonicipal Explained
Tags: neovim, workflow, automation
Published: 2023-01-11 20:32:58

---
title: Moonicipal Explained
published: true
description: Introducing and explaining the Moonicipal plugin for Neovim
tags: neovim,workflow,automation
# cover_image: https://direct_url_to_image.jpg
# Use a ratio of 100:42 for best results.
# published_at: 2023-01-08 14:12 +0000
---

I've recently published a new task runner plugin for Neovim - [Moonicipal](https://github.com/idanarye/nvim-moonicipal). Several years ago I've wrote a similar post about Moonicipal's predecessor - Omnipytent. I've opened that post with [an explanation why such a plugin is needed](https://dev.to/idanarye/omnipytent-5g5l#the-problem-running-commands). The same reasoning apply for Moonicipal, so I'll not repeat them here.

## Why a new plugin?

When Omnipytent was written in 2016, the Neovim ecosystem looked very different that how it looks today. Plugins were not written in Lua - they were written either in Vimscript or in some external language by using Neovim's RPC API, and many plugins still aimed to support both Vim and Neovim. Additionally, Python 2 was still alive and kicking, with many plugins written in it, and Vim had some issues supporting both versions of Python in the same instance.

So I decided to write Omnipytent using Python, in a way that'll support both Python 2 and Python 3. This means that the tasks themselves can be written in either Python 2 or Python 3. Since this was before Lua plugins, most plugin used Vim commands and/or keybinds as their interface, and these only required passing strings - which were pretty easy and straightforward to build in Python (though no quoting conventions was a bit of a pain)

Nowadays, most Neovim plugins are in Lua, and their interface is Lua functions that often accept tables and functions. These are a bit harder to generate from Python - especially the back-and-forth function calls - and it's starting to seem easier to just write everything in Lua.

Another problem was asynchronous tasks. Having to support Python 2 meant I was limited to Python 2's version of async (only using `yield` - I couldn't even use `yield from`!), but that was manageable. A bigger problem was GC - having to maintain a references to the Python generators so that Vim code can resume them, and having to clean that reference manually after usage. With Lua coroutines this is simply not an issue - Lua functions can just be stored as Vim functions.

So, even though Lua is less expressive than Python, its superior interoperability with Neovim and Neovim's plugin ecosystem makes the switch worthwhile.

## Using Moonicipal

For the sake of demonstration, I'm going to use the same project I've used in [my post about Omnipytent](https://dev.to/idanarye/omnipytent-5g5l) - the Java Sprint example project [PetClinic](https://github.com/spring-projects/spring-petclinic).

After cloning the repository and `cd`ing into it, we fire up Neovim and:

{% embed https://asciinema.org/a/550481 %}

So what just happened here?

I started by writing the command `:MCedit test` to edit the Moonicipal task named "test". This opened a Lua file named `.idanarye.moonicipal.lua`.

{% details Why that name? %}
Because I've configured Moonicipal as:

```lua
require'moonicipal'.setup {
    file_prefix = '.idanarye',
}
```

But even if I left `file_prefix` out of the `setup`, Moonicipal would just take it from my `$USER` environment variable.

The purpose of the file prefix is to make the tasks file personal. Sharing task files causes all sorts of problems:

* You need to be careful about what you change. You can't, for example, just add a parameter to a command in a task - that will cause problems to other people who use that task.
* You can't just use Neovim plugins and locally installed Lua modules - that will break the tasks file for other developers who don't have them. You'll have to add a dependency management (or at least list the dependencies), which is kind of an overkill for a task runner...
* You won't be able to put personal information in the tasks file. I'm not talking about passwords or tokens - take extra care with those! - but things like paths on your local machine or URLs to servers assigned to you. Other developers will have their own paths and their own URLs, and if you share a tasks file you'll need to put a mechanism to avoid sharing these.
* A shared task file means you are writing code that other people will use, which generally requires higher quality code. Not that high quality code is a bad thing - but it does cost time and effort, and Moonicipal is supposed to allow quick and dirty rapidly changing tasks.

It's better not to commit the tasks file at all, but having a different filename for each developer also helps in ensuring they remain separate.

{% enddetails %}

`:MCedit test` created a scaffold for the tasks file and for the task:

```lua
local moonicipal = require'moonicipal'
local T = moonicipal.tasks_file()

function T:test()
    
end
```

A task file is simple. It requires the module and uses its `tasks_file()` function to create a tasks registrar `T`. We can then add our tasks as functions on that registrar - like `T:test()`, which was automatically added because I've added it as an argument to the `:MCedit` command.

Then I write the body of the task:

```lua
vim.cmd'!mvn test' 
```

`vim.cmd` is a Neovim Lua API for running Vim commands. The parameter passed to it is `!mvn test` - which means it'll run the `:!mvn test` Vim command, which is just running `mvn test` in the non-interactive shell.

What if we want interactive shell? This requires some more commands, but it's still basic usage of Neovim's Lua scripting interface:

{% embed https://asciinema.org/a/550483 %}

Or, if you're more comfortable with Vim commands, we can just use Lua's multiline quotes:

{% embed https://asciinema.org/a/550484 %}

This simple example already demonstrates an advantage of Moonicipal over CLI task runners - we can control, from the test, on how Neovim will run the command. We can run it in the non-interactive shell, open an interactive shell buffer, use a terminal management plugin - whatever we want! If we can code it in Lua, we can put it in a task.

## Getting input

Running all the tests is good for CI, but during development we often want to run a single test that focuses on the changes we are currently making. This takes less time to run and less effort to look at the results.

Maven lets us run a single tests class by setting the `test` property:

```lua
vim.cmd.terminal('mvn test -Dtest=ClinicServiceTests')
```

The Moonicipal tests file is built to be easy to change, so always editing the task when you want to run a different test is not that bad - but we can also make our task receive input:

{% embed https://asciinema.org/a/550487 %}

`moonicipal.input` is a function for receiving input. By default it functions like Vim's builtin input.

_So why can't we just use `vim.fn.input()`?_

Because Neovim has `vim.ui.input()`, which can be [overridden by plugins to offer nicer input UI](https://github.com/stevearc/dressing.nvim#screenshots).

_Okay, wise guy, why can't we just use `vim.ui.input()` then?_

`vim.ui.input()` works with a callback. You pass a function to it, and when the user enters their text the function gets called with that text. If you need more input after receiving the first input, you need another callback in the `vim.ui.input()` call that's nested inside the callback of the first `vim.ui.input()` call. This pattern is called "Callback Hell", and considered not very pleasant.

To avoid this callback hell, many programming languages introduce some form of asynchronous execution. [Lua does it with coroutines](https://www.lua.org/pil/9.1.html), and Moonicipal runs all its tasks inside coroutines. `moonicipal.input()` can only run inside a coroutine, and instead of using a callback, it resumes the coroutine once the input was entered.

Entering the test class name each and every time seems like a step backwards though. `moonicipal.input()` is not really that useful. But just like Neovim has `vim.ui.input()` and `vim.ui.select()` - Moonicipal also has `moonicipal.select`. We can collect all possible test files inside our tasks, and then just select the test file to run from the list:

{% embed https://asciinema.org/a/550500 %}

Note that just like `moonicipal.input()`, `moonicipal.select()` uses `vim.ui`. I get my nice interactive list from [fzf-lua](https://github.com/ibhagwan/fzf-lua#neovim-api), but there are several other plugins which provide their own UIs.

Another new thing we see here is `moonicipal.abort()`. If I cancel my selection, I don't want the task to open a new window and then fail formatting a command to run in it and print a traceback while keeping the new window open. Instead I use `or moonicipal.abort()` so that if `moonicipal.select()` returns `nil`, it'll just abort the task without much fuss.

## Caching choices

Even if we choose it from a list, it is still a waste to have to choose the test every single time we want to run it - especially during development, when we want to keep running the same test over and over. Luckily, Moonicipal has some caching facilities that let us cache our choice:

{% embed https://asciinema.org/a/550522 %}

Caching in Moonicipal is task-bound, so we need to create a new task. They are simple enough to write by hand, but here I used `:MCedit choose_test` to scaffold a new task. The task's `self` has some caching facilities - one of them is `self:cached_choice()`, which lets us cache a selection from a list. Only the choice is cached - the list itself is generated from scratch every time we use the cache - so we must set a key for Moonicipal to recognize the choice by. In this case we use strings, so a simple `tostring` suffices.

Then we call the cached choice object as a function with all the cache choices. We already had a loop for generating them, so I just reused that.

Finally, we use the `select()` method to run the actual selection - either by asking the user or by using the cache.

Back in the original `test` task, we can access the cached choice by calling its task as a method on the registrar object - `T:choose_test()`.

When I run `:MC test`, it prompts me to choose a test class - just like before. But when I run it a second time - it remembered my choice of test to run!

Eventually, though, we may want to change our choice. To do that, all we have to do is run the `choose_test` task directly with `:MC choose_test`. This will let us choose again, even if there already is a cached choice. Of course, it won't actually run the test until we use `:MC test` again.

## Cached buffers and Channelot terminals

Let's leave the tests, and look at another aspect of development cycle - trying queries against a live server. For this, one of Moonicipal's supplemental plugins - [Channelot](https://github.com/idanarye/nvim-channelot).

{% embed https://asciinema.org/a/550547 %}

Here we also use a new caching facility - `cached_buf_in_tab`. This method accepts a function, and expects that function to finish in a different buffer. After the function runs, `cached_buf_in_tab` will automatically jump back to the window where it started (if its still open in the current tab)

The caching part is that if the buffer from a previous run is open in the current tab - the function will not run, and instead `cached_buf_in_tab` will return the value from its previous run. This makes it ideal for processes like interactive shells that other tasks can use.

As for the plugin I mentioned, Channelot - I hope its API is clear enough:

```lua
local j = channelot.terminal_job('bash')
j:writeln('mvn spring-boot:run &')
```

This creates a `bash` job, and writes a text line to its STDIN to run the server in a background process. This seems a bit weird - why not simply use `vim.cmd.terminal('mvn sprint-boot:run')`?

Because we can use the same `bash` instance to run queries against the server:

{% embed https://asciinema.org/a/550553 %}

By calling `T:launch()` - just like `T:choose_test()` from before - we get the returned value from that task, which is the job handler `j` returned from the `launch` job. We can than use that handler to send commands the the running `bash` process, and see the results live in the terminal.

Also, you may have noticed that instead of `:MC launch` and `:MC start` here I've used just `:MC` without arguments, which prompted me to select the task from a list. `:MC <task-name>` is useful for keybinding, but if you have a good selection UI `:MC` is more ergonomic for running tasks you don't have keybindings for.

## Data cells and BuffLS

One of the most powerful types of input we can cache is a _data cell_ - a buffer that can be edited with Neovim's full editing capabilities, including plugins and language servers. The buffered is stored in memory even when closed (as long as it's not deleted with something like `:bdelete`), and 

(To save precious screen space (I'm recording these at 24 rows on 80 columns - much smaller than typical modern terminals) I'll abandon the `bash` terminal split and just use `print(vim.fn.system(...))` instead (`vim.cmd'!...'` has problem with line breaks). We'll lose syntax highlighting for the JSON, but I can live with that)

{% embed https://asciinema.org/a/550752 %}

`cached_data_cell`'s argument is a table. The `default` parameter should be obvious - default content for the buffer, the first time we run the task to edit it. `buf` is a function (or a string with Vim commands) for preparing the buffer. It is called every time the task is invoked to edit the data cell.

Note that there is no need to create the split - `cached_data_cell` will do it automatically. This is because `buf` is called every time the task is invoked, and we want to reuse the split if it is open. You can change how the split is opened with the `win` parameter, but that's rarely necessary.

Merely setting the filetype to `jq` gives us syntax highlighting and everything else we have configured for editing `jq` query files. But we can do more with another one of Moonicipal's supplemental plugins - [BuffLS](https://github.com/idanarye/nvim-buffls):

{% embed https://asciinema.org/a/550789 %}

`require'buffls.TsLs'` is BuffLS' Treesitter based language server. Its `for_buffer` method creates an instance, attaches it to a buffer (current by default - or you can pass a buffer number), and sets the Treesitter syntax based on the buffer's filetype. In this case - `jq` (so you need to have that syntax installed for it to work - easiest way is to install [nvim-treesitter](https://github.com/nvim-treesitter/nvim-treesitter) and run `:TSInstall jq`)

This gives us a BuffLS language server object we can use to configure BuffLS **for that specific buffer**. Which means we can use it to add very specialized LSP functionalities.

The simplest one is a code action. Not that LSP code actions are that simple in general - but BuffLS uses null-ls behind the scenes, and because null-ls runs inside the Neovim instance its code actions are simply Lua functions. We can add our code actions with `ls:add_action` - all it needs is a caption for the action and a function.

Actions that set the buffer content to some template are the simplest - all they have to do is use `moonicipal.set_buf_content` (which is a wrapper for `nvim_buf_set_lines` with a simpler signature). But of course they can also read the content, configure things in the buffer (or in Neovim in general - but don't get crazy please), or anything else you can do in a Lua function (which is basically everything). Additionally, they run in a Lua coroutine - which means they can use things like `moonicipal.select()` to get input from the user.

But... we didn't really need the `jq` Treesitter syntax for that, did we?

Let's add something that does use Treesitter - completion!

{% embed https://asciinema.org/a/550956 %}

By using a Treesitter query, we can detect that we are inside the string part of `.firstName == ""`, and in that case offer appropriate completions.

Treesitter queries can be a bit too complex for most Moonicipal usecases. Luckily, these usecases are often easy to wrap. One common usecase is `bash` - authoring commands with the specific flags our project uses.

BuffLS supports this with `buffls.ForBash`:

{% embed https://asciinema.org/a/551009 %}

## Conclusion

These tasks are all just Lua functions that you need to write yourself, but Moonicipal provides scaffolding for these tasks, a nice entry point for loading and running them in coroutines, and ways to collect input and cache it. Because of that, the task writing becomes accessible enough to allow automating even small things you would otherwise have done manually because adding them to your `init.lua` or creating a plugin for them would have been too much of a hassle.