Hands-on AWK
Tags: programming, tutorial, productivity, beginners
Published: 2023-06-19 21:20:01

> This article was authored by **Mehvish Poshni**, a member of Educative's technical content team. 

For many of us, our first exposure to a programming language is a general-purpose programming language like C, Python, or Java. AWK, on the other hand, was designed with a very targeted goal of being able to process text-based data without having to write several lines of code. That's not to say that AWK is limited to performing this function alone—far from it. However, effectiveness of AWK is largely due to the control it offers over writing these quick one-liner command-line programs, as well as short scripts to serve an immediate need. Imagine having the prowess to manipulate system logs, configuration files and spreadsheet data from the command line in just a few keystrokes. Another reason why it's worthwhile learning AWK is because it comes pre-installed as the utility `awk` on Unix-like operating systems, and its inclusion into the Unix ecosystem makes it very convenient to use.

The letters A, W, and K in AWK stand for the last names of the individuals (Alfred Aho, Caspar Weinberger, and Brian Kernighan) who designed the programming language in the late 1970s.

![AWK](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5sj7b9utui9svcsfg5m3.png)

> **Note:** This blog assumes a passing familiarity using a command-line shell (`cat`, `echo`, pipe, and redirection), some prior programming exposure (concepts like comparison and logical operators, expressions, conditionals, and loops).

## Input structure for an AWK program

An AWK program takes input either in the form of one or more text files, or as the standard input stream coming from the shell environment in which `awk` executes. The default behavior is that each line in the input stream is considered one **record**, and each record has **fields** (text) separated by one or more whitespace characters (spaces or tabs). This default behavior can be overridden easily. 

The coding environments included in this blog make use of an input text file. For convenience, we show the file here in a tabular format:

![AWK input](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vostoz812axwc6faaomt.png)

**Code format:**(`inputfile`)

```
Colton Dominguez 28 Feb-20-2021 33 Marketing No
Megan Porter 29 Dec-03-2021 81 Engineering Yes
Candace Walsh 25 Apr-14-2023 43 Sales No
Grady Clements 40 Feb-15-2023 36 Sales Yes
Macaulay Roy 33 Jul-11-2022 63 Engineering Yes
Abraham Strickland 31 Aug-25-2022 93 Marketing Yes
Joelle Higgins 42 Sep-23-2022 89 Engineering Yes
```

> **Note:** The input file may not necessarily have the same number of fields on each line.

## An unusual workflow

The manner in which an AWK program runs is unusual because when it's run, the code is _repeatedly_ executed for each record in the input — behind the scenes. 

![AWK](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dl0ri5gk6equ2iqtnx1f.png)


Whenever a record is read, there are special built-in variables `$1`, `$2`, `$3` (and so on) that can be used for accessing the values in the first, second, third (and so on) fields of that record. The entire record can also be retrieved all at once using the built-in variable `$0`.

## An AWK program

A basic AWK program consists of one or more pattern-action pairs in the following general form. 

```shell 
pattern { action }
```

- The `pattern` is an expression that evaluates to a value that's regarded as _true_ or _false_.  

   > **Note:** AWK does not have a boolean data type, but `0` and the empty string `""` are regarded as false, and all other values as true.

- The `action` consists of one or more statements. In case there are multiple statements within an `action`, they may be separated by either a semicolon character (`;`) or a newline. 

When running an AWK program, each `pattern` is tested against every record of the input stream, one by one. 
Whenever the `pattern` evaluates to _true_, the corresponding `action` is executed.

![AWK programming](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0e0lpwlgacmm6swy9qhr.png)


The entire program is enclosed within single quotes, and can be run from the command line using the `awk` utility:

```shell
awk 'pattern { action }' inputfile
```
Here, `inputfile` is the input to the program. More than one file can also be passed as input.

```shell
awk 'pattern { action }' file1 file2
```

Since we are running the program using the `awk` utility in the shell, output can be stored in a file using the redirection operator `>`.

```shell
awk 'pattern { action }' inputfile > outputfile
```

In the same vein, the input can also be taken using the pipe operator.

```shell
cat inputfile | awk 'pattern { action }'
```

### Examples

1- In the following one-liner, we print the records for which the age (in the third column) is less than *30*.

```shell
awk '$3 < 30 { print $0 }' inputfile
```

**Output:**

```
Colton Dominguez 28 Feb-20-2021 33 Marketing No
Megan Porter 29 Dec-03-2021 81 Engineering Yes
Candace Walsh 25 Apr-14-2023 43 Sales No
```

2- See how none of the records get printed when `0` or the empty string `""` is used as a pattern.

```shell
cat inputfile | awk '0 { print $0 }' 

awk '"" { print $0 }' inputfile
```

**Output:** "Success" 

3- The pattern in the following snippet is a non-empty string (from the second column in `inputfile`) which is considered true. So the action is executed for all records. 
  
Observe, also, how we can concatenate different strings by placing them side by side.

```shell
awk '$2 { print $2 ", " $1 }' inputfile > outputfile

cat outputfile # To display the contents of the outputfile
```

**Output:**

```
Dominguez, Colton
Porter, Megan
Walsh, Candace
Clements, Grady
Roy, Macaulay
Strickland, Abraham
Higgins, Joelle
```

### Patterns and actions are optional

It isn't necessary to specify both `pattern` and `{ action }`. Just one of them suffices:

* When `pattern` is not specified, the `action` is performed for _all_ the records. 

```shell
awk '{ print $1 "." $2 "@educative.io" }' inputfile
```

**Output:**

```
Colton.Dominguez@educative.io
Megan.Porter@educative.io
Candace.Walsh@educative.io
Grady.Clements@educative.io
Macaulay.Roy@educative.io
Abraham.Strickland@educative.io
Joelle.Higgins@educative.io
```

* When `{ action }` is not specified, the default action is to print all the _matched_ records. 

```shell
awk '$6 == "Marketing" || $7 == "No"' inputfile
```

**Output:**

```
Colton Dominguez 28 Feb-20-2021 33 Marketing No
Candace Walsh 25 Apr-14-2023 43 Sales No
Abraham Strickland 31 Aug-25-2022 93 Marketing Yes
```

### The `BEGIN` and `END` patterns
There are other ways to specify a pattern than creating expressions using numbers, strings, arithmetic or logical operators.

* The pattern `BEGIN` is matched with the beginning of the input file. So, its associated action is executed in the beginning before any other record is read. It makes sense to use it for tasks like initializing variables.
* The pattern `END` matches the end of the file and is executed once at the end of the input file. 

Think about how to add the scores listed in the fifth column of the input file.

```shell
awk 'BEGIN { sum = 0 } 
{ sum += $5 } 
END { print "Sum of scores is " sum }' inputfile
```

**Output:**

```
Sum of scores is 438
```

## Regular expressions as patterns
**Regular expressions (regex)** are symbolic ways to represent a pattern, and specify what the matching text should look like. 

> The syntax of regular expressions used in AWK is known as the **Extended Regular Expression (ERE)**.

This syntax is also used by many other languages and unix-based utilities. So, it's super useful to know.

### Using a regex

![Regex](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/723h749dgkweo8vld7m3.png)

In AWK, when specifying a regex as a pattern, we can include it between two forward slashes. The simplest form of a regex is as a plain sequence of characters. For example, the pattern `/Feb/` matches all records containing the text `Feb`.

```shell
awk '/Feb/' inputfile
```

**Output:**

```
Colton Dominguez 28 Feb-20-2021 33 Marketing No
Grady Clements 40 Feb-15-2023 36 Sales Yes
```

Instead of searching the entire record for a match against a regex, we can use the operator `~` to check if a regular expression matches a smaller portion of the given text. Similarly, the operator `!~` is useful for checking if there is no match.

> **Usage:** The regular expression must appear on the right of `~` or `!~`, and the text being searched must go on the left.

```shell
awk '$6 ~ /Sa/' inputfile # Sa present in the 2nd last column
echo " "
awk '$(6+1) !~ /Y/' inputfile #  Absence of Y in the last column 
```

**Output:**

```
Candace Walsh 25 Apr-14-2023 43 Sales No
Grady Clements 40 Feb-15-2023 36 Sales Yes
 
Colton Dominguez 28 Feb-20-2021 33 Marketing No
Candace Walsh 25 Apr-14-2023 43 Sales No
```

### Regex metacharacters

A regular expression may include some special characters called **metacharacters**, so called because they are not matched with a text in a literal sense. Instead they are interpreted as a rule for matching text. Here are some examples:


* The metacharacters `[` and `]` match one of (possibly) many characters that appear enclosed within the brackets. For example, `[AbC]` means a single character: either `A`, `b`, or `C`. Expressions like these are called **character classes**.  
* A range of characters can also be represented as character classes. For example: 
    - `[0-9]` means a single numeric character from $0$ to $9$. 
    - `[a-zA-Z]` means a single alphabetical character in upper or lower case.
* The metacharacters `[^ ]` specify a single character other than the ones appearing after the symbol `^` inside the character class.  For example, `[^bcd]` means any character other than `b`, `c`, or `d`.

```shell
# score column contains a number in the 20-25 range 
awk '$5 ~ /[20-25]/ { print $1 " scored in the 20 to 25 range" }' inputfile
echo " "
# 2022 or 2023 not present in the 4th column
awk  '$4 ~ /202[^23]/ { print "Joining year of " $1 " is neither 2022 nor 2023" }' inputfile 
```

**Output:**

```
Megan scored in the 20 to 25 range
 
Joining year of Colton is neither 2022 nor 2023
Joining year of Megan is neither 2022 nor 2023
```

The metacharacters `$` and `^` (outside a character class) take on meaning relative to some other character `X` in the following way:
   * `^X` means lines that start with `X`.
   * `X$` means lines that end with `X`.

```shell
awk '/^J/' inputfile # Lines that start with J
echo " "
awk '/o$/' inputfile #Lines that end with o
```

**Output:**

```
Joelle Higgins 42 Sep-23-2022 89 Engineering Yes
 
Colton Dominguez 28 Feb-20-2021 33 Marketing No
Candace Walsh 25 Apr-14-2023 43 Sales No
```

* The metacharacter `.` means any single character.
* The metacharacter `|` means characters specified by the regex on its left or its right. For example, `ab|[cd]` matches either `ab`, `c` or `d`.
* The metacharacters `()` are used for grouping characters. For example, `^M` versus `^(Me)` mean two different things (lines beginning with `M` versus lines beginning with `Me`). 

> **Note:** The GNU implementation of AWK, known as **GAWK**, also supports additional features including the use of metacharacters `()` for _capturing_ portions of matched text for later use.

```shell
awk '/(M..a)/' inputfile # Matches substrings of Megan and Macaulay
echo " "
awk '/(D|P)o/' inputfile # Matches substrings of Dominguez and Porter
```

**Output:**

```
Megan Porter 29 Dec-03-2021 81 Engineering Yes
Macaulay Roy 33 Jul-11-2022 63 Engineering Yes
 
Colton Dominguez 28 Feb-20-2021 33 Marketing No
Megan Porter 29 Dec-03-2021 81 Engineering Yes
```

The metacharacters `*`, `+`, `?`, `{m,n}` are called **quantifiers**. They also take on meaning relative to their preceding character, say `X`:
- The expression `X*` means zero or more occurrences of `X`.
- The expression `X+` means one or more occurrences of `X`.
- The expression `X?` means zero or one occurrence of `X`.
- The expression `X{n,m}` means at least `n` and at most `m` occurrences of X. (This is not supported below.) 

```shell
awk '/i[g]+/' inputfile # Colton Mscaulsy Joelle
echo " "
awk '/oe?l/' inputfile #  Joelle and Colton
```

**Output:**

```
Joelle Higgins 42 Sep-23-2022 89 Engineering Yes
 
Colton Dominguez 28 Feb-20-2021 33 Marketing No
Joelle Higgins 42 Sep-23-2022 89 Engineering Yes
```

> **Note:** To match a metacharacter literally, we  need to use the escape character `\`. For example `/\*/` to match the character `*`.

## Data structure: Associative array

An **associative array** is the only data structure supported by AWK. It essentially consists of index and value pairs, where the index can be used for retrieving the corresponding value.

![Array](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/faq0x9b8jyutfboili9g.png)


An associative array is created simply through an assignment statement that maps a value to an index. The syntax looks like this: 

```
arr["ind"] = "val"
```
We can also add more elements to an array using assignment statements like the one above.

```shell
awk '{ arr[$1] = $3 } 
END{ for (i in arr ) 
        { 
            print i " " arr[i]
        } 
    }' inputfile
```

**Output:**

```
Grady 40
Macaulay 33
Megan 29
Colton 28
Joelle 42
Candace 25
Abraham 31
```

Notice how we loop over the array `arr` using a `for(i in arr)` style loop. In each round, `i` is set to the index of an element in `arr` (and not an element in `arr`). 


AWK also supports a C-style `for` loop (see exact syntax below), but it isn't suitable for traversing over an associative array because the keys of an associative array may not fall in the required range of numbers. 

```sh
for(i = 1; i < 10; i++) 
```
Here's another example where the number of individuals in each team count is computed.

```shell
awk '!arr[$6] { arr[$6] = 0 }
{ arr[$6] += 1 }
END {
        for (i in arr)
        {
           print i " : " arr[i]
        }
    }' inputfile
```

**Output:**

```
Marketing : 2
Sales : 2
Engineering : 3
```

## Built-in variables and functions

Other than `$0`, and `$1`, `$2`, `$3` etc., there are other built-in variables that are easy to remember and easy to use. Some of these are shown in the following table:

![AWK](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xph2t4vgrix8b4a36bhv.png)

AWK supports many predefined mathematical functions (like `log`, `sqrt`, `exp`, `sin`) as well as functions for working with strings (such as `substr`, `length`, `toupper` ). 

Let's see a few more examples before we call it a day.

## Example 1: Overriding default values

We can use any character as a field separator in the output by changing the default value of the variable `OFS`. The default values for `OFS` can be overridden as shown below.

Also note how, in the following example, we print the record number for each row using the variable `NR` (for number of records).

```shell
awk '{ print NR, $1, $5 }' OFS=, inputfile
```

**Output:**

```
1,Colton,33
2,Megan,81
3,Candace,43
4,Grady,36
5,Macaulay,63
6,Abraham,93
7,Joelle,89
```

## Example 2: Accessing fields using rvalues

If a variable `varname` is assigned an integer *k*, then the syntax `$varname` can be used for accessing the fields in the *k^{th}* column. 

For example, since `NF` stores the number of fields in the current record, we can access the last field in that row using the syntax `$NF`.   

```shell
awk '{ print NR, $(NF-1), $NF }' inputfile 
```

**Output:**

```
1 Marketing No
2 Engineering Yes
3 Sales No
4 Sales Yes
5 Engineering Yes
6 Marketing Yes
7 Engineering Yes
```

## Example 3: Formatting output
The C style `printf` is used for showing the output formatted in a tabular form. The argument `%-20s` sets the width of the padded string at *20&* characters and aligns it to the right.

```shell
awk 'BEGIN { printf "%-20s | %-5s\n", "Full Name", "Score" } 
{ printf "%-20s | %-5d\n",  $1 " " $2, $3 }' inputfile
```

**Output:**

```
Full Name            | Score
Colton Dominguez     | 28   
Megan Porter         | 29   
Candace Walsh        | 25   
Grady Clements       | 40   
Macaulay Roy         | 33   
Abraham Strickland   | 31   
Joelle Higgins       | 42   
```

The next two examples use built-in functions.

## Example 4: Splitting a string

The built-in function `split(str, arr, ch)` is used, which splits the string `str` around the character `ch` and stores the resulting substrings in the array `arr`. We use this function below to extract the month and year from each individual's joining date.

```shell
awk '{ split($4, arr, "-"); 
printf "%-10s | %-10s\n",  $1 , arr[1] " " arr[3] }' inputfile 
```

**Output:**

```
Colton     | Feb 2021  
Megan      | Dec 2021  
Candace    | Apr 2023  
Grady      | Feb 2023  
Macaulay   | Jul 2022  
Abraham    | Aug 2022  
Joelle     | Sep 2022  
```

### Example 5: Find and replace

The function `gsub(regex,subst,str)` looks for all matches made by the regular expression `regex` in the string `str`, and replaces it by string `subst`. The `g` in `gsub` is for "global". There's also a related function `sub` (for replacing a single occurrence).

```shell
awk '{ gsub(/[0-9]+/,"X",$0); print }' inputfile 
```

**Output:**

```
Colton Dominguez X Feb-X-X X Marketing No
Megan Porter X Dec-X-X X Engineering Yes
Candace Walsh X Apr-X-X X Sales No
Grady Clements X Feb-X-X X Sales Yes
Macaulay Roy X Jul-X-X X Engineering Yes
Abraham Strickland X Aug-X-X X Marketing Yes
Joelle Higgins X Sep-X-X X Engineering Yes
```

## Example 6: Bigger programs

In AWK programs, we can use many constructs similar to the ones available in other languages like `if`, `else`, `while`, `switch`, and more. One can also define a function in an AWK program, and then call it from within the scope of an action.

```shell
awk 'BEGIN { max = -1; name = "" } 
{
    if (max < $5)
    { 
        max = $5
        name = $1 
    }
}  
END { print name ": " getMaxScore() } 

function getMaxScore() { return max }' inputfile
```

**Output:**

Abraham: 93

## A final word

It's worth noting that AWK is a Turing complete language, which means that it can be utilized for implementing any algorithm. That being said, AWK is primarily useful for tasks like data filtration and manipulation. 

This blog is far from being a complete tutorial, but we hope that it is effective in removing any entry level barriers for a faster and a happier learning experience.

* [**The Complete Guide to Bash Programming**](https://www.educative.io/courses/guide-to-bash-programming?utm_campaign=&utm_source=medium&utm_medium=text&utm_content=&utm_term=&eid=5082902844932096)

* [**Learn Data Science with Bash Shell**](https://www.educative.io/courses/learn-data-science-with-bash-shell?utm_campaign=&utm_source=medium&utm_medium=text&utm_content=&utm_term=&eid=5082902844932096)

*Happy learning!*