Command Line Control: awk
Tags: commandlinecontrol, shell
Published: 2019-06-16 17:09:56

---
title: Command Line Control: awk
published: true
tags: command line control,shell
canonical_url: https://thomashartmann.dev/blog/command-line-control-awk
cover_image: https://thomashartmann.dev/images/awk.svg
---
Unless you spend a whole lot of time working primarily on the command line, <code>awk</code> is one of those commands you’ve likely come across a few times, but never really learned how to use properly. Similar to [my recent adventures with <code>xargs</code>](https://thomashartmann.dev/blog/xargs-and-the-unruly-tags/), though, I recently came across a little use case where I could benefit from using it, so I decided to sit down and look into it.</p>
<p>Even more so than with <code>xargs</code>, this is a <em>really</em> powerful tool, but let’s focus on the basics and see what we can make of it.</p>
<h1 id="what-is-awk">What is <code>awk</code>?</h1>
<p><code>awk</code> is a command that processes text using programs written in the AWK language.</p>
<p>To understand what it can do, understanding the language it uses is probably a good place to start. Let’s use the words of Alfred Aho—one of the creators of the <em>AWK</em> language—from a <a href="https://web.archive.org/web/20080808234125/http://www.computerworld.com.au/index.php/id%3B1726534212%3Bpp%3B2">2008 <em>Computerworld</em> interview</a> to get a rough grasp of what it is:</p>
<blockquote>
<p>AWK is a language for processing files of text. A file is treated as a sequence of records, and by default each line is a record. Each line is broken up into a sequence of fields, so we can think of the first word in a line as the first field, the second word as the second field, and so on. An AWK program is of a sequence of pattern-action statements. AWK reads the input a line at a time. A line is scanned for each pattern in the program, and for each pattern that matches, the associated action is executed.</p>
</blockquote>
<p>That may be a bit dense, so let’s take it apart:</p>
<dl>
<dt>“AWK is a language for processing [..] text”</dt>
<dd>AWK is a <em>domain-specific language</em> (DSL) focused on text processing. The <code>awk</code> command expects that its first argument is a script or a string in this language.
</dd>
<dt>“AWK reads the input a line at a time”</dt>
<dd>AWK is line-oriented and works through the input line by line. (It’s actually <em>record</em>-oriented, but the default separator is a newline character, so this is the same thing by default.)
</dd>
<dt>“Each line is broken into a sequence of fields”</dt>
<dd>Each word in a line maps to a field. These fields are accessed with the <code>$</code> operator, e.g. <code>$1</code> for the first word, <code>$2</code> for the second, and so on. <code>$0</code> is the whole line. By default, fields are delimited by whitespace (which is why I’ve called them words) but this can be customized.
</dd>
<dt>“An AWK program is of a sequence of pattern-action statements”</dt>
<dd>
<p>This means it’s a sequence of predicates with actions. If a predicate evaluates to true, perform the specified action. If no predicate is specified, it will always evaluate to true, and if no action is specified, it will default to printing the whole line.</p>
<p><em>Huh?</em> Yeah, this is still a bit confusing, but maybe some examples will make it clearer:</p>
<dl>
<dt>Patterns</dt>
<dd>
<p>This is a predicate to check each line against. It usually takes the form of a regex enclosed in forward slashes. <code>/foo/</code>, <code>/b[ar]?/</code>, <code>/^baz/</code>, <code>/(fizz|buzz)$/</code> are all examples. Most of the regex skills you have will be applicable here (character sets, character classes, alternations, etc.).</p>
<p>You can also match specific fields against a regex. Only want to match lines where the second field contains ‘cheese’? <code>$2 ~ /cheese/</code></p>
<p>The pattern can also consist of functions and comparisons; so if you wish to act only on lines that aren’t empty: <code>length &gt; 0</code></p>
<p>If no pattern is given, every line will match.</p>
</dd>
<dt>Actions</dt>
<dd>
<p>These are commands telling <code>awk</code> what to execute. They are enclosed in curly braces (<code>{}</code>). This is where you might instruct <code>awk</code> to print a certain field of the string—<code>print $3</code>, for instance—or increment a counter if you’re counting words: <code>word_count += NF</code> (yeah, I’ll get to what <code>NF</code> means in a bit).</p>
<p>If no action is given, <code>awk</code> will print the matching line.</p>
</dd>
</dl>
</dd>
</dl>
<h1 id="basic-auth-awk">Basic <del>auth</del> <code>awk</code>
</h1>
<p>That’s a quick overview of how the language is structured. Before we start playing with it, let’s explore some of the features.</p>
<h2 id="built-in-variables">Built-in variables</h2>
<p><code>awk</code> has a number built-in variables, and while I won’t cover all of them, these are the ones that I’ve found the most useful:</p>
<dl>
<dt><code>NR</code></dt>
<dd>
<p>Gives you the line number of the current match. Could be used for adding line numbers to an input file: <code>awk '{print NR, $0}'</code>. Or maybe looking for lines that contain ‘ice cream’ is more your speed: <code>awk '/ice cream/ {print NR}'</code>.</p>
</dd>
<dt><code>NF</code></dt>
<dd>
<p>This is the number of fields in a line. Useful if you’re looking for the last field, either for finding out how many fields are in a line or for seeing if it contains a pattern: <code>awk '$NF ~ /out/ {print NR}</code></p>
</dd>
<dt><code>FS</code></dt>
<dd>
<p>The field separator value. This is what <code>awk</code> will use to split each line into fields. By default this is whitespace. If you have a file full of comma-separated values and want to split each line on commas instead of whitespace: <code>BEGIN {FS=","}</code></p>
</dd>
<dt><code>RS</code></dt>
<dd>
<p>This is the line (‘record’) equivalent of the field separator. The default is <code>\n</code>. Say you want to print your <code>PATH</code> over multiple lines: <code>echo $PATH | awk 'BEGIN {RS=":"} {print}'</code></p>
</dd>
</dl>
<h2 id="begin-and-end">
<code>BEGIN</code> and <code>END</code>
</h2>
<p><code>awk</code> lets you supply commands to be executed at the start and end of a script by using <code>BEGIN</code> and <code>END</code>. While they may seem like it, they’re not really special at all. Instead, think of them as being patterns that evaluate to true before <em>anything</em> is evaluated and after <em>everything</em> is evaluated, respectively.</p>
<p><code>BEGIN</code> could be used to set the field separator or initialize a variable.</p>
<p><code>END</code> is useful for printing out results accrued through the life of the program, such as a word count. If we bring back our word counting example: <code>awk '{word_count += NF} END {print word_count}'</code></p>
<h2 id="functions-and-conditionals">Functions and conditionals</h2>
<p>Like most languages, AWK has a number of built-in functions (such as the <code>print</code> and <code>length</code> functions we saw earlier) and also lets you define your own functions if you so please. This is probably overkill for most trivial operations but could come in handy in certain cases.</p>
<p>And AWK has conditionals too! While you can use the <code>if else</code> construct within actions, I’d like to highlight that you can do conditional statements based on the supplied pattern. I.e. <code>awk '/foo/ {print "FOO"} {print "bar"}'</code> will print ‘FOO’ for lines that match <code>/foo/</code> and ‘bar’ for lines that don’t.</p>


# Applications

Now that we've got some idea how it works, what can we use it for? Let's
look at some sample applications for it:

## Sorting lines in a file by length

Here's a fun little application: Let's take a file, count the number of
characters in each line, and then sort it based on the number of
characters:

``` bash
awk '{print length, $0}' <file> | sort -n
```

And if you want to exclude empty lines, try this:

``` bash
awk 'length > 0 {print length, $0}' <file> | sort -n
```

## Friendly `PATH`

This is the same one that was listed above, and is useful if you're
looking for certain entries that can easily get lost when the path is on
one line:

``` bash
# bash, zsh
echo $PATH | awk 'BEGIN {RS=":"} {print}'

# fish
echo $PATH | awk 'BEGIN {RS=" "} {print}'
```

## Counting words

Another example that was used previously. Count the words in a file or
any input stream:

``` bash
awk '{word_count += NF} END {print word_count}'
```

## Parsing git logs

Maybe your team has agreed that all commit messages should start with
`#<task_no>` if it relates to a task and `#--` if it doesn't. To find
all commits that relate to a specific task—say `#42`—we could do this:

``` bash
git log --oneline | awk '/#42/ {print $1}'
```

Or how about finding the ratio of commits that belong to a task versus
those that don't?

``` bash
git log --oneline --no-decorate | \
awk '$2 ~ /#[0-9]+/ {task += 1} {notask +=1} \
  END {printf "Tasks: %d\nNo-tasks: %d\nTask to no-task ratio: %f", \
  task, notask, task / notask }'
```

(Yeah, the `printf` function works pretty much the same as in C—so much
so that I haven't looked it up yet\!)

## Killing processes

This is actually what triggered me to look into `awk` and what
eventually led to this post being brought into existence.

There are a number of ways to kill applications from the command line,
and for a while, my default fallback has been the classic `ps aux | grep
<app>` combo.

While this lists out all the relevant processes and their process IDs
(*PIDs*), you then have to manually copy the PID and put it into the
`kill` command to shut it down. This is annoying at best, and gets even
worse if the process has spawned children that we want to take down as
well.

How do we deal with this? Well:

``` bash
ps aux | awk '/<app>/ {print $2}' | xargs kill
```

1.  `ps` lists all processes on the system

2.  `awk` then goes over every line that contains the application name,
    extracts the second whitespace-delimited word—which in this case is
    the PID—and prints that.

3.  For the last stretch, we use `xargs` to feed the PIDs into the
    `kill` command, thus killing all the processes.

This works just fine, though it comes with the caveat that it'll also
try and kill the `awk` process (because the application name is part of
the command, so it gets listed by `ps`), but that's only a minor
annoyance, and I'll leave fixing that as an exercise for the reader.

Now, I'm sure you can make `killall` do something quite similar to this,
but I've found this way to be more effective (by which I mean: *closer
to what I expect*).

# Closing up

*Phew*. We've learned a lot today, and this post grew much longer than
what I had originally imagined—about four or five times longer—but it's
been quite the journey and I'm glad you were there with me, partner. I
hope you've gleaned some new insight too and that you'll find some
application for this in your day-to-day.

See you next time\!