Combining awk and vim to get real power
Tags: awk, gawk, vim, neovim
Published: 2019-04-03 15:47:49

---
title: Combining awk and vim to get real power
published: true
description: In this article I will show you how magic can be awk plus vim
tags: awk, gawk, vim, neovim
---

# Foreword
After the initial post of this little, "how to", [@Sundeep](https://dev.to/learnbyexample) [commented](https://dev.to/learnbyexample/comment/9p6n):

> why not use awk itself to insert the snippet? since you are using gawk, you can also use -i  inplace if you want inplace editing

> I don't remember exactly, but I think you can use silent to avoid having to use Enter manually, check these: ...

I will explain the scenario, my old solution and then the gawk solution 

# The Scenario
I have about 500 markdown files on my personal vimwiki, for years I have been making notes to help myself and others through Linux/vim commands. Recently I started using [pandoc](https://pandoc.org/) to convert my markdown files to html and LaTeX. 


## using vim + gawk

I have two mappings on my `~/.config/init.vim` that makes it easy to convert my current documment to html and pdf. During the process of convertion pandoc warns about the lack of the *title*, and this brought me here.

```vim
if !exists('*PandocMappings')
    function! PandocMappings()
        " NOTE: <F16> equals to Shift-F4
        noremap <F4> :! pandoc '%:p' -o /tmp/'%:p:t'.html --template=easy_template.html --toc && firefox /tmp/'%:p:t'.html &<CR><CR>
        noremap <F16> :! pandoc '%:p' -o /tmp/'%:p:t'.pdf --pdf-engine=pdflatex --toc && evince /tmp/'%:p:t'.pdf &<CR><CR>
    endfun
endif

augroup my_markdown
    autocmd!
    autocmd FileType markdown call PandocMappings()
augroup END
```

My markdown files lack metadata, something like this:

```
---
file: the file name
lang: pt-BR
author: Sergio Araujo
date: abr 03, 2019
created: abr 03, 2019
abstrac: This aims...
---
```

After adding manually the metadata in about eight of them I realized that I had better tools and the knowledge (including web search) to solve my problem. The metadata should start at first line.

# Finding out the files with no metadata
Using awk, more specificly gnu awk "gwak" printing the list of files who have the metadata is simple

    gawk '(FNR==1 && $1 ~ /^---/) {print FILENAME}' *.md

    FNR==1  ...................... the first line equals to the first register
    && ........................... it allows us to test a second condition
    ~  ........................... match regular expression
    /^---/ ....................... the regular expression itself

The command above showed me the files I already had inserted the metadata, inverting the *regex* is very simple, just use exclamation. So the list of files that did not had the metatada can be printed with: 

    gawk '(FNR==1 && $1 !~ /^---/) {print FILENAME}' *.md

# Combining the awk result with vim
Using the shell *[Command Substitution](https://unix.stackexchange.com/questions/440088/what-is-command-substitution-in-a-shell)* we can deliver to vim the list we wanted.

    vim $(gawk '(FNR==1 && $1 !~ /^---/) {print FILENAME}' *.md)

# The vim/neovim argdo command
When you open vim with something like:

    vim *.md

All the opened files are in the arglist, so we can use something like:

    :argdo command

# A UltiSnippet snippet to insert the header

```
snippet headermd "markdown header" w
---
file: `!p snip.rv = snip.fn`
lang: pt-BR
author: `!v g:snips_author`
date: `!v strftime("%b %d, %Y")`
created: `!v strftime("%b %d, %Y")`
---
endsnippet
```

# the vim function that can be called by `argdo` to inserts the snippet

```
fun! InsertHeaderMd()
    0pu_
    exe "normal! iheadermd\<C-r>=UltiSnips#ExpandSnippet()\<CR>"
endfun
```

    0pu_  ............................ On the line zero insert black hole register (blank line)
    exe normal! ...................... normal command
    iheadermd ........................ enters insert mode and types headermd
    \<C-r>= .......................... expression register
    UltiSnips#ExpandSnippet()\<CR> ... the UltiSnippet trigger to call the snippet

# Finishing up everything
We do not even have put the function (just the snippet) on the configuration files, you can copy the function to the clipboard and type this on vim:

    :@+

Now with all files opened you can do:

    :silent argdo call InsertHeaderMd() 
    :silent argdo update

Thanks to [@Sundeep](https://dev.to/learnbyexample) I added the *silent* option which allowed me run the command without hiting Enter a bunch of times. And he also suggested awk as a solution

# Using a vim macro with argdo

*Opening the files without metadata*

    vim $(gawk '(FNR==1 && $1 !~ /^---/) {print FILENAME}' *.md)

*Creating a vim macro*

    :let @a=''
    :let @a="ggO---\<Return>title: \<C-r>=expand('%:t')\<CR>\<Return>author: Sergio Araújo\<Return>abstract:\<Return>---\<Esc>"

*Explanation*

    let @a='' ............................ delete any marcro 'a'
    ggO .................................. got to the first line and open a line above
    ---\<Return> ......................... insert three dashes and a new line
    \<C-r>=  ............................. starts a expression register
    expand('%:t') ........................ expands the filename "t = tail" 
    \<Esc> ............................... goes back to the normal mode

*Running the argdo command plus a macro*

    :set hidden ............................ allows to go to the next file without saving

    :silent argdo execute "normal! @a" | update 

    :silent ................................. does not show any messages


# The gawk solution 
This solution came from a @Sundeep [comment](https://dev.to/learnbyexample/comment/9p6n)

      var=$(date "+%b %m, %Y")
      awk -i inplace -v date="$var" 'BEGINFILE {print "--\nfile: "FILENAME"\ndate: "date"\nabstract: \n---"}; (FNR==1 && $1 !~ /^---/) {print}' *.md 


Job done!

## Explaining the awk long command...

    -i inplace ..................... makes possible edit files directly in awk
    -v date="$var" ................. captures the content of var=$(date "+%b %m, %Y")
    BEGINFILE ...................... insets some strings "between quotes"
    FILENAME ....................... gawk variable that returns the file name
    FNR==1  ........................ the first line equals to the first register
    && ............................. it allows us to test a second condition
    ~  ............................. match regular expression
    /^---/ ......................... the regular expression itself

The next step will be using a variable using shell *Here Documents*. If you guys can help me with this I would thank you a lot!

# Could it be done with *sed*?
I know that sed can also insert a header in a bunch of files, but I should have write a shell script to get each file name or use `-V` sed option. The solution I figured out was good for my problem, and it also inspired me enough to share this ideas with you. Feel free to comment and suggest alternatives or improvements. (Please fix my typing, I am not a native English speaker).


# References:
+ https://stackoverflow.com/questions/32481877/what-is-nr-fnr-in-awk
+ https://unix.stackexchange.com/questions/440088/what-is-command-substitution-in-a-shell
+ https://www.cyberciti.biz/faq/how-to-print-filename-with-awk-on-linux-unix/