Processing Text with Linux Shell - Part 2 
Tags: bash, linux, shell, textprocessing
Published: 2018-07-29 19:39:20

---
title: Processing Text with Linux Shell - Part 2 
published: true
description: grep is very useful when we want to search a piece of text in a file or in all the files within a directory. 
tags: bash, linux, shell, text-processing
---

# `grep` - searching the haystack made easy

When you need to search a piece of text from a very large source, `grep` command is the answer. `grep` accepts strings, regular expressions and it can produce output in various formats.


The most basic syntax to search a piece of text in a file is:

```shell
grep "pattern" file_name
```

If we need to search multiple files for a pattern, we can do the following:

```shell
grep "pattern" file1 file2 file3 ...
```


`grep` can also be used with standard input.

```shell
echo "linux is awesome" | grep linux
```

If no file or directory is specified, `grep` reads the `STDIN` for input.



### # Highlight the matched text


One major feature of grep is it's ability to highlight the matched string in its output. There are 3 color options: `auto`, `always` and `never`. To use the color options, we can pass an additional `--color` flag to instruct which option to use.

```shell
grep "pattern" file_name --color=always
```

As we can already guess, with `--color=always` option it will always color the output matched string. With `--color=auto`, `grep` displays color in the output only if the output is not piped to a command or redirected to any file. The last option, `--color=never` will turn off the coloring.

If you want to turn on coloring for every `grep` operation, but feeling lazy to type `--color=always` with each command (which would be me), you could add `GREP_OPTIONS` to your environment.

```shell
export GREP_OPTIONS='--color=always'
```
This will turn on coloring for all `grep` commands.


### # Limit what to output

Normally, `grep` only outputs the line that contains the matched pattern. But there are circumstances where we might want to see the contents of the entire file with the matched pattern highlighted. We can do the following for this:

```shell
grep --color 'pattern\|' file_name
```

This will output all the lines containing the pattern as well as the lines that have an end, i.e the complete contents of the file with only the matched part colored. Of course, we can pass multile patterns to be matched.

```shell
grep --color 'pattern_one\|pattern_two\|' file_name
```



Notice the `\` before `|` . This is because `grep` only interprets some of the special characters. However we can also write commands without `\` .

```shell
grep -E --color 'pattern_one|pattern_two|' file_name
```

The `-E` option instructs `grep` to use full set of regular expressions. However, there is a better way of writing such patterns containing regular expressions . We can use `egrep`, which is an extended version of `grep` that supports extended regular expressions out of the box. Therefore, we can rewrite the above commands like this.

```shell
egrep --color 'pattern_one|pattern_two|' file_name
```



But what if we neither need the complete line that contains the matching text nor the complete contents, rather just the matching content? 

For example, if we are searching for all the email addresses in a file, we do not need the line that contains the email, rather we just need all the emails that appears in the file.

Well, we can use the `-o` option, which stands for *only matching*, that limits the output to only the matched text.

```shell
egrep -o "\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,6}\b" file_name
```

Neat, isn't it? 

What if we need to see all the lines *except* the ones where the given pattern appears?

```shell
grep -v "pattern" file_name
```

The `-v` option inverts the results and thus the above command will output all the contents except the lines containing the matched pattern.


### # Count the appearances

If we need to get a count of how many times a pattern appears in a file, we can use the `-c` flag. From the above email pattern example, if we need to see how many email addresses are there in a file, we could do the following.

```shell
egrep -c "\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,6}\b" file_name
```

But wait, there is a gotcha. 

The `-c` counts only the number of lines that contains the pattern, not the actual number of matched pattern. So if there are multiple matches in a single line, it will count only once!

We can overcome this with a little tweak. 

Remember how we printed only the matching portions with the `-o` flag above. If pipe `wc -l` with it, 

```shell
egrep -o "\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,6}\b" file_name | wc -l
```
it will give us the exact number of times a particular pattern appears in a file.


### # Extract additional information

If we want to print the line numbers where the searched pattern appears, we can use the `-n` flag.

```shell
grep -n --color "pattern" file_name
```

The `-n` flag will also print the name of the file, in case we are searching in multiple files.

If we want to see the offset where the pattern starts, we can use the `-b` flag.

```shell
echo "linux is the answer" | grep -b -o "the"
```

This is print `9:the` as the searched word *"the"* starts at the 9th position of the line. Note that that `-b` flag is always used with the `-o` flag.



### # Search within directories

To see which files contain a particular pattern , we can use the `-l` flag. 

```shell
grep -l "pattern" file1 file2 file3 ...
```

This will output only the file names contains the given pattern. If we want to see the name of the files that does not contain the files, we can use `-L` flag (note the case-sensitive flags). This basically inverts results we get with `-l` flag.



Well, so far we have only seen how to search in a single file or a set of file which are specified in the command. However, we can also search all the files in a complete directory and it's subdirectories recursively.

```shell
grep -r "pattern" .
```

Here `.` specifies the current directory. The `-r` flag implies that the search should be done recursively in all subdirectories. This search is case-sensitive. 

If we want to search all occurances of the pattern, we can append `-i` flag.

```shell
grep -ir "pattern" .
```

This will output all matching patterns irrespective of the case.


### # The pattern file

If we have too many patterns, strings etc that we want to search, piping them with `|` is very tedious. In such scenarios, we can use a pattern file, that contains all the strings/patterns we want to search, *each in a new line*. Then we can use that pattern file in the `grep` command.

```shell
grep -ir . -f pattern_file
```

This will print all the matching strings from the pattern file.

ヽ(´▽`)/
 

(You can find Part 1 of this series [here](https://dev.to/shamilchoudhury/processing-text-with-linux-shell---part-1-4ajd) )