Using Vim As Your Shell Command-Line Scratch
Tags: shell, vim
Published: 2022-04-20 15:02:38

Using (n)vim as your command-line scratch helps you to efficiently fix, create, and run ad-hoc commands. 

If you are not familiar with vim, I suggest you start playing with the `vimtutor` first. I’ll try my best to explain each keys that I have pressed. In this blog, I’m using [Neovim](https://github.com/neovim/neovim), but you can still follow along even if you’re using [Vim](https://github.com/vim/vim). I have aliased my `nvim` to `vim` with `alias vim=nvim`. Please follow along with the demonstrations, so you can get used to this workflow.

## Fixing Ad-Hoc Commands

Have you experienced running a command and realizing that you made a mistake after executing it? You missed some syntax or forgot adding a `sudo`, and much more?

### Commands that requires privilege

You are copying a file to another directory, or installing a package, but you forgot that it requires a `sudo` privilege.

{% embed https://youtu.be/q6ZV4yQMye4 %}

```
I - insert mode before the first character
Escape or Ctrl + C - to go to the normal mode
ZZ or :wq- to save and exit
```

Note that in vim, keys are case-sensitive, this means that the lowercase `i` is different to the capital `I`.

As you have noticed, we were able to edit our previously executed command with fix command (`fc`), it opened the vim editor for us with the command as our content. If we exit from the editor, it’ll run the command that we have edited.

For this case, it is better to use sudo-bang-bang (`sudo !!`) if you need to add `sudo` before the previous command.

{% embed https://youtu.be/eq6CV0_d9jg %}

`!!` - will use your previously executed command

If we view the man page of `fc`. It checks for the `FCEDIT` and `EDITOR` environment variables. If both of them are unassigned, it’ll use `vi`. 

```
-e ename
...
If ename is not given, the value of the FCEDIT variable is used,
and the value of EDITOR if FCEDIT is not set.
If neither variable is set, vi is used.
When editing is complete, the edited commands are echoed and executed.
```

I have exported my `EDITOR` variable in my config file to use neovim.

```bash
export EDITOR=nvim
```

You can assign `nano`, or `code` here to use VS Code as your `fc` editor, but I do not recommend it. You’ll see why it’s better to use vim later on.

{% embed https://youtu.be/K7wjiU9dd8o %}

### Commands with typo

Another use case is if you made a typo in the middle of the command, or missed any quotation marks. 

**Arrow Keys Spam**

{% embed https://youtu.be/lIAmBhPlZSg %}

When we try to fix our command, we will spam the thing out of the left arrow key to navigate to the middle of the command to fix it. This workflow is so slow, and even annoying to deal with.

**`fc` to the rescue!**

{% embed https://youtu.be/XAw9RP_sQo0 %}

Invoke the `fc` command to fix the previous command that you have executed. This will open a vim editor which we can easily perform search and replace commands. This would be more useful if we have really long commands and arguments.

`:s/ooo/o` - perform a substitute `ooo` to `o`

`:s/hir/here` - perform a substitute `hir` to `here`

For more information about the vim substitute command, please check the help file with `:h :s`.

### Commands with shell expansion

Let’s say you want to say hello to the current `USER`.

{% embed https://youtu.be/JHGCmF9nNmo %}

Hmmmmm 🤔, our `USER` is not printing, but it has value as we have checked it. We have found out that we were using single quotes (`'`), and this means that [shell expansion](https://www.gnu.org/software/bash/manual/html_node/Shell-Expansions.html) never happened, what we need is to surround them with double quotes (`"`).

We can still use `:s/'/"` here, but we can also leverage the power of vim with plugins!

{% embed https://youtu.be/qk2lxHbRIh0 %}

I have navigated to the text to replace the surrounding quotes, then used [tpope/vim-surround](https://github.com/tpope/vim-surround) plugin.

```
cs'" - it means change surrounding single quote (') to double quote (").
                c      s                         '                   "
```

If you want to delete a quote, you can use `ds'`, or `ds"` for double quotes.

### Commands that you want to edit without executing

We have learned that we can use `fc` to edit or fix our previous command. What if you noticed a typo, and you do not want to execute the command? In that case, what we can do is press `ctrl + a` to move the cursor before the first character in your command line and type pound (`#`), this results into making the command as a comment. 

{% embed https://youtu.be/yFt5w1qS-gg %}

If the command with pound (`#`) in front is executed, it won’t execute the command itself, but instead it’ll result into being a shell comment. The good thing with this is you can use `fc` to open it in your editor and edit it. You can do this because the commented-out command was already added to your command history, and `fc` will edit the previously executed command. Just remove the `#` on your edit.

{% embed https://youtu.be/xGgQLjYbTmc %}

### Commands with long parameters

If we want to run a postgres container with a command from their documentation, we will run the following command:

```bash
docker run -d --name some-postgres -e POSTGRES_PASSWORD=mysecretpassword -e PGDATA=/var/lib/postgresql/data/pgdata -v /custom/mount:/var/lib/postgresql/data postgres
```

After we have pasted this in our terminal, it’s hard for us to update the parameter variables. We will use so many arrow key presses just to navigate to the environment variable values and edit it. We can comment this for now and run the `fc` command. By doing this, we can leverage the vim editor to manipulate the text.

{% embed https://youtu.be/XbZ10YsqU4I %}

`x` - delete the `#` symbol

`:s/some-/my-` - substitute the container name

```
/secretciwmyverysecret
  - '/secret' search and navigate to the first 'secret' word match
  - 'ciw' delete the inner word, and go into insert mode
  - 'myverysecret' literal word to replace the password value
```

Add new lines before the `-e` and `-v` 

```
:s/ -[ev]/\r\0/g
  - ':s/' substitute command
  - ' -[ev]' search for text that has ' -e' or ' -v'
  - '/\r' add a carriage return
  - '\0' substitute with the first captured group
  - '/g' substitute globally, not just the first match
```

With this substitute, we were able to format our command, but there is a problem, in shell with multiple lines, we need to suffix every line with a backslash (`\`). We can achieve that with performing a `normal` command.

```
:%norm A \
  - ':%norm' perform a normal command for the whole buffer
  - 'A' go to insert mode to the end of the line
  - ' \' add literal space and backslash
```

The `normal` command helps you to perform it on multiple lines as if you’re actually typing it directly on the buffer, to learn more about it, check the help file for `normal` using `:h norm` (`norm` command is just a shorter version of `normal` command).

We were now able to update the values of a command with multiple long parameters, and also properly formatted it. If you noticed that my line numbers are not linearly increasing, I’m using a relative numbers, you can learn more about it with `:h relativenumber`. I highly recommend setting `set rnu` config together with `set nu` so you can easily jump to arbitrary lines relative to your current line.

### Commands that you are currently fixing and want to abort it

If you have edited your previous command and you do not want to execute it. If you use `ZQ` or `:wq!` to exit vim without saving, it’ll still execute your command.

{% embed https://youtu.be/v2XK8w-ke7k %}

What happened? It turns out that when the editor exits with `0`, it’ll execute your command, and using `:wq!` gracefully exits vim.

```bash
echo $?
0
```

How to solve this? You need to use `:cq` to exit vim with error code. You can add a key bind for this, or you can just delete the content of your vim buffer with `dd` and exit.

{% embed https://youtu.be/gr0FeLG3gTQ %}

## Vim Buffers and Shell

Fixing your previous command is cool, but what if you want to perform a command operation to the output of your previous command, and update its content? You can use Vim to achieve that.

If we perform an operation and pipe it to our Vim buffer, it’ll use the stdin as your buffer content.

```bash
echo hello | nvim -
# look ma, no dash
echo hello | nvim
```

There are times that you need to use dash (`-`) to pipe the standard output to standard input of the next command. In nvim, the dash is not required.

### Sort and remove duplicate names

We have a file with a list of names, but there are duplicates. We want to remove the duplicates, and sort them.

```bash
cat names.txt
John Doe
John Doe
David William
Joseph Thomas
David William
Joseph Thomas
Joseph Thomas
```

We can pipe the output of the `cat names.txt`, or just directly open it with vim.

```bash
vim names.txt
```

Then we can now perform a command operation to update the content.

```bash
:%!sort -u
David William
John Doe
Joseph Thomas
```

Anything after the `!` character is the shell command.

{% embed https://youtu.be/EMRnCyAWByI %}

Of course, we do not need to open it in vim to remove the duplicates, it is to simply demonstrate that you can perform commands on the current buffer and replace it with the output of the shell command.

### Sort and remove duplicate names of some part of the buffer

We can also specify some area that we want to perform the operation using the visual mode of vim.

```bash
cat names2.txt
John Doe
John Doe
David William
Joseph Thomas
David William
Joseph Thomas
Joseph Thomas
# please do not modify the following lines
Christopher Daniel
Mark Anthony
Christopher Daniel
Paul Steven
Mark Anthony
Paul Steven
```

{% embed https://youtu.be/9xSjN_y_jVM %}

First, we need to select the range with the visual line (`V`), and run the `:!sort -u` command.

Note that the `'<,'>` characters are automatically added by vim, this represents the selected line range.

### Formatting and filtering JSON content

```bash
cat characters.json
[ {
"id": 1,
          "name": "Tomoko Kuroki",
    "age": 15,
"height": "147 cm"
  }, {
    "id": 2,
          "name": "Marin Kitagawa",
    "age": 15,
"height": "164 cm" },
  {
          "id": 3,
"name": "Kaguya Shinomiya", "age": 17,
                "height": null
  } ]
```

This is a JSON file with a content that is not properly formatted. We also want to remove some noise, we only want the `id`, and the `name` properties of the characters. We can do this by using the [jq](https://stedolan.github.io/jq/) command. We can also use any vim plugins like prettier for different file types.

{% embed https://youtu.be/g21jGOxmV10 %}

We have formatted it with `:%!jq .`

```bash
:%!jq .
[
  {
    "id": 1,
    "name": "Tomoko Kuroki",
    "age": 15,
    "height": "147 cm"
  },
  {
    "id": 2,
    "name": "Marin Kitagawa",
    "age": 15,
    "height": "164 cm"
  },
  {
    "id": 3,
    "name": "Kaguya Shinomiya",
    "age": 17,
    "height": null
  }
]
```

Filtering the properties that we only want with `:%!jq '[.[] | {id, name}]'`. This is really helpful to analyze some JSON data, and want to play with the properties.

{% embed https://youtu.be/mOY9fB_unYY %}

```bash
:%!jq '[.[] | {id, name}]'
[
  {
    "id": 1,
    "name": "Tomoko Kuroki"
  },
  {
    "id": 2,
    "name": "Marin Kitagawa"
  },
  {
    "id": 3,
    "name": "Kaguya Shinomiya"
  }
]
```

Pressing `u` to undo the changes to the buffer. You can learn more about `jq` from their [manual](https://stedolan.github.io/jq/manual/).

We were able to easily parse the JSON without leaving our editor!

## Searching file contents

You can also pipe the output of your grep to vim, so you can easily navigate to the matched files. If the cursor is under a file path, and pressing `gf` will open another buffer with the file and the cursor is on the line number that was created in our `grep` result.

{% embed https://youtu.be/8LzGbPxIzgU %}

```
grep -Hnri 'search keyword here' . | nvim
H - print file name
n - add line number
r - recursive search for each directory
i - ignore case
```

Now that we have the matched searches in our buffer with their associated file names and line numbers, we can use `gf` here to navigate to the file under the cursor. This can help you preview and analyze the files, you can get back to the previous buffer with `ctrl + 6`. See `:h buffers` for more information.

## Writing to Shell

Can we do it the other way around? Using the content of your buffer to execute them in your shell?

One common misconception when you run `vim <file>`, is that your’e editing the actual file, but in fact, you are actually opening a buffer of that file, and not editing the file itself.

This is different to the other editors where they will immediately create the file for you.

{% embed https://youtu.be/LNefH_a6JbQ %}

When you save it with `:w`, you are just overwriting the file with your current buffer, and if the file does not exist, it’ll create it. This is equivalent to `:w %` or `:w yourfile.txt`

{% embed https://youtu.be/3LnOlBfes0Q %}

With this in mind, what if you write the buffer to your shell? Well...you guessed it right (pun not intended 😉), if you write the buffer to your shell, you can execute commands from your Vim buffer!

I am using `zsh` here, but any shell should work.

If we write the buffer to a shell, it’ll invoke the command to a sub-shell.

```bash
:w !zsh
:w !$SHELL
```

```bash
cat file-that-contains-shell-script.txt

echo 'i have echo'
echo $(pwd)
touch newfile.txt
```

{% embed https://youtu.be/xaPSGL1wrPU %}

We can also pick which line to execute.

```bash
:.w !zsh
```

Notice the dot(`.`) before `w`, this means “write the current line of the cursor to the shell”.

```bash
cat sample-line.txt
echo hello1
echo hello2
echo hello3
echo hello4
echo hello5
```

{% embed https://youtu.be/5-04YkJgjB8 %}

### Shell Level

If we print out the `SHLVL` environment variable, we are currently at level `1`. Invoking this command inside vim will result into the level `2`. This means that it is executing it to the sub-shell.

{% embed https://youtu.be/131E9H4shvQ %}

### Bindings Configuration

Typing `:w !zsh` or `:w !$SHELL` again and again is too much, I recommend adding your own bindings similar to what I have

```bash
" write to shell
let mapleader=" "
nnoremap <silent> <leader>xs :.w !$SHELL<CR>
xnoremap <silent> <leader>xs :.w !$SHELL<CR>
nnoremap <silent> <leader>xS :w !$SHELL<CR>
```

You can put this into your `.vimrc` file or the `init.vim` for neovim. Then you can just press the leader key which is set to space and `xs`.

{% embed https://youtu.be/XF2hPDsck40 %}

### File Analysis

From our previously found files in the “Searching file contents” section, we decided that we want to delete those files. We need to filter out the file paths first, and there are multiple ways to achieve this. We can use `cut` here, but for now, let’s use `awk` with a colon separator.

```bash
:%!awk -F: '{print $1}'
./subfolder0/somefile.txt
./subfolder0/anotherfile.txt
./subfolder1/alpha.txt
./subfolder3/beta.txt
./subfolder4/magic.txt
./sample-line.txt
./sample-line.txt
./sample-line.txt
./sample-line.txt
./sample-line.txt
```

{% embed https://youtu.be/e59b94-250A %}

There are duplicate files, let’s perform a unique sort and prepare to delete with `rm`.

```bash
:%!sort -u
./sample-line.txt
./subfolder0/anotherfile.txt
./subfolder0/somefile.txt
./subfolder1/alpha.txt
./subfolder3/beta.txt
./subfolder4/magic.txt
```

{% embed https://youtu.be/olP7tBWj8PY %}

Now that we have added the `rm` command on each line, and writing them to your shell will remove the files.

## Shell Intellisense

We can improve our workflow by adding intellisense using the [Language Server Protocol (LSP)](https://microsoft.github.io/language-server-protocol/). LSP is a protocol that can help provide some auto completion, quick documentation, go to definition, and many other features. You can use any LSP provider, at the time of this writing, I’m using [coc.nvim](https://github.com/neoclide/coc.nvim) for my LSP.

Setting up your own LSP and their configuration is out of scope for this blog, you can check their documentation for it.

In order for our LSP to work, there are multiple ways to enable it, but the easiest one I can think of is to set the file type of the buffer that we are currently working on. You can check the current file type with `set filetype?`, and assign a file type using `set filetype=sh`. Now that we have an `sh` filetype, our LSP now knows that we are editing a shell buffer, which will provide us the language features of a shell.

### Command Auto-Completion and Documentation

{% embed https://youtu.be/JPvUm4VYbDk %}

### Path Auto-Completion

{% embed https://youtu.be/0WVa-ERmRe4 %}

This is just the tip of LSP, and you can check the language server of what features it can offer for you to help in your shell scratch workflow.

## Other Use Cases

In this section, I will share some of my workflow that I use day to day.

### Workflows in Git

### Long Branch Names

Whenever we create new features, or fixing a bug. It’s better to create new branch from the `main` or any base branch. We usually create branches using the ticket name `feat/ZERO-1234`, but browsing these branches does not provide any context as to what is the purpose of that branch, unless you memorized the ticket number and their description.

We can add more details to the branch name, but the problem with this branch names is that the checkouts, rebase, and renames is a bit of pain to type, unless you have auto complete. You can pipe the branch output to vim, and perform the checkouts.

```bash
feat/ZERO-123-create-some-cool-feature-here-that-no-one-usees
fix/ZERO-456-the-bug-that-will-consume-your-days
```

{% embed https://youtu.be/I9tLevfBEJU %}

The aliases that you have configured in your Git will work here like `git config --global alias.co checkout`.

### Deleting multiple branches

Deleting multiple branches with `grep` is a bit of a hassle, unless you run the `grep` first to see the list of the branches. What we can do is pipe the output to vim, and perform any searches.

{% embed https://youtu.be/-ki6so7KKCY %}

By doing this, you have a faster feedback of what branches would be deleted, you can even easily remove any branch that was part of the initial `grep`.

### Reading Command Outputs

APIs mostly use JSON as their payload. We can easily create them using [jo](https://github.com/jpmens/jo). We can read the command output and put it to your current buffer. For example, we want to create a JSON object with a lower case uuid value for its `id` property, and a simple name.

```bash
:read !jo id=$(uuidgen | tr A-Z a-z) name='zaerald'
```

Anything after the `!` will run a shell command.

{% embed https://youtu.be/MQSmpn-hp1M %}

## Conclusion

You can now efficiently edit previously executed commands using `fc`. Using vim with plugins and LSP can help improve your workflow. Sky is the limit! Using vim as your command-line scratch is much fun to use. Just play with it, and don’t forget to have fun!

You can check my dotfiles in [github.com/zaerald/dotfiles](https://github.com/zaerald/dotfiles). Please note that my dotfiles continues to evolve, and there’s a higher chance that it is different now compared at the time of this writing.

This blog post is originally posted [here](https://blog.zaerald.com/using-vim-as-your-shell-command-line-scratch).