Multiline fixed string search and replace with cli tools
Tags: tutorial, perl, ripgrep, sed
Published: 2020-11-30 12:58:57

---
title: "Multiline fixed string search and replace with cli tools"
published: true
description: "Using cli tools for fixed string search and replace processing across multiple lines"
canonical_url: https://learnbyexample.github.io/multiline-search-and-replace/
tags: tutorial,perl,ripgrep,sed
cover_image: https://dev-to-uploads.s3.amazonaws.com/i/gxfbuzzowvl7oxc2whn7.png
---

Credit: *Cover image generated using [carbon](https://carbon.now.sh/)*

This post shows how you can use `ripgrep`, `perl` and `sd` to perform multiline fixed string search and replace operations from the command line. Solution with `GNU sed` is also discussed, along with its limitations.

## Fixed string matching

The below sample input file will be used in the examples in this post.

```
$ cat ip.txt
This is a multiline
sample input with lots
of special characters
like . () * [] $ {}
^ + ? \ and ' and so on.
This post shows how
you can do fixed
-string multiline
search with cli tools.
```

### ripgrep

[ripgrep](https://github.com/BurntSushi/ripgrep) supports `-U` option to allow multiline matching. Use `-F` option to turn off regexp matching, i.e. treat the search string literally. In `bash` shell (and likely most other shells), you can press enter key to insert literal newline character for quoted values. When you do so, the next line starts with `>` and a space character. This isn't shown in the examples below to make it easier to copy-paste the commands.

```
$ rg -UF 'like . () * [] $ {}
^ + ? \ and' ip.txt
4:like . () * [] $ {}
5:^ + ? \ and ' and so on.

# -l option shows only filename instead of all the matching lines
$ rg -lUF 'like . () * [] $ {}
^ + ? \ and' ip.txt
ip.txt
```

You'll have an issue if your search string itself contains single quote characters. Avoid using double quotes as a workaround, as that has its own set of special characters. You can work around by concatenating multiple strings next to each other, along with escaped single quote characters as needed.

```
# -N option disables line number prefix
$ rg -NUF 'like . () * [] $ {}
^ + ? \ and '\'' and' ip.txt
like . () * [] $ {}
^ + ? \ and ' and so on.
```

If your search string starts with `-` character, you'll have to use `--` before the search argument.

```
$ rg -NUF -- '-string multiline
search' ip.txt
-string multiline
search with cli tools.
```

### perl

You can use `-0777` option with `perl` to slurp the entire input as a single string. Another advantage with `perl` is that you can use files to pass the search and replace strings. Thus, you don't have to worry about any character that may clash with shell metacharacters. See my [Perl one-liners cookbook](https://learnbyexample.github.io/learn_perl_oneliners/) if you are not familiar with using `perl` from the command line.

```
$ cat search_1.txt
like . () * [] $ {}
^ + ? \ and ' and so on.

# display filename if the given search string matches
$ perl -0777 -nE '!$#ARGV ? $s=$_ :
                  /\Q$s/ && say $ARGV' search_1.txt ip.txt
ip.txt
```

However, you'll have to make sure the file doesn't end with a newline if you are providing partial lines for searching, or take care of it within the `perl` script.

```
$ cat search_2.txt
-string multiline
search

# no output because there's a newline at the end of search_2.txt file
$ perl -0777 -nE '!$#ARGV ? $s=$_ :
                  /\Q$s/ && say $ARGV' search_2.txt ip.txt

# this will remove newline at the end of file before assigning to $s
$ perl -0777 -nE '!$#ARGV ? $s=s/\n\z//r :
                  /\Q$s/ && say $ARGV' search_2.txt ip.txt
ip.txt
```

By default, `ripgrep` gives entire matching lines. To get rest of the line with `perl`, you'll have to explicitly add a pattern around the search string.

```
# $& variable has the entire matching portion
$ perl -0777 -nE '!$#ARGV ? $s=s/\n\z//r :
                  /\Q$s/ && say $&' search_2.txt ip.txt
-string multiline
search

# use 'say $& while /.*\Q$s\E.*/g' if there are multiple matches
$ perl -0777 -nE '!$#ARGV ? $s=s/\n\z//r :
                  /.*\Q$s\E.*/ && say $&' search_2.txt ip.txt
-string multiline
search with cli tools.
```

## Fixed string substitution

### ripgrep

`ripgrep` also supports replacing matched string with something else using the `-r` option. By default, you'll see only matched lines in the output. Use `--passthru` option to display all the input lines, even if they do not match the given search string. See [my blog post](https://learnbyexample.github.io/substitution-with-ripgrep/) for more details about the `-r` option and various ways you can use it for substitution requirements.

```
$ rg --passthru -NUF 'like . () * [] $ {}
^ + ? \ and' -r '====
----
====' ip.txt
This is a multiline
sample input with lots
of special characters
====
----
==== ' and so on.
This post shows how
you can do fixed
-string multiline
search with cli tools.
```

Apart from having to workaround single quote, you'll have to use `$$` instead of `$` as it is used for backreferences in the replacement section.

```
$ echo 'sample input' | rg --passthru -F 'in' -r '$a'
sample put
$ echo 'sample input' | rg --passthru -F 'in' -r '$$a'
sample $aput
```

### perl

With `perl`, you can use files for both search and replace strings. And, you can easily choose to replace first or all occurrences, unlike `ripgrep` where it always replaces all the matches.

```
$ cat replace.txt
---------------------
$& = $1 + $2 / 3 \ 4
=====================

$ perl -0777 -ne '$#ARGV==1 ? $s=$_ : $#ARGV==0 ? $r=$_ :
                  print s/\Q$s/$r/gr' search_1.txt replace.txt ip.txt
This is a multiline
sample input with lots
of special characters
---------------------
$& = $1 + $2 / 3 \ 4
=====================
This post shows how
you can do fixed
-string multiline
search with cli tools.
```

As seen before, you'll have to remove newline from search string for partial line matching.

```
# use $r=s/\n\z//r to avoid trailing newline from replace.txt
$ perl -0777 -ne '$#ARGV==1 ? $s=s/\n\z//r : $#ARGV==0 ? $r=$_ :
                  print s/\Q$s/$r/gr' search_2.txt replace.txt ip.txt
This is a multiline
sample input with lots
of special characters
like . () * [] $ {}
^ + ? \ and ' and so on.
This post shows how
you can do fixed
---------------------
$& = $1 + $2 / 3 \ 4
=====================
 with cli tools.
```

### sd

[sd](https://github.com/chmln/sd) supports fixed string and Rust regexp based substitution. Unlike `ripgrep`, `-s` option for fixed string will apply to both search and replacement sections. `sd` does in-place editing for file inputs by default, you can use `-p` to preview results on the terminal. Multiline matching is automatically performed by default.

```
$ echo 'sample input' | sd -s 'in' '$a'
sample $aput

$ sd -ps 'like . () * [] $ {}
^ + ? \ and' '====
----
====' ip.txt
This is a multiline
sample input with lots
of special characters
====
----
==== ' and so on.
This post shows how
you can do fixed
-string multiline
search with cli tools.
```

## Saving file contents to a variable

Trailing newlines and ASCII NUL characters will be lost if you wish to save contents of a file as `bash` variables using `var=$(< filename)` command. See [stackoverflow: pitfalls of reading file into shell variable](https://stackoverflow.com/questions/7427262/how-to-read-a-file-into-a-variable-in-shell/22607352#22607352) for details.

```
$ printf '\na\0b\n123\n\n\n\n\n\n\n\n' > t1
$ a=$(< t1)

# NUL character is lost after the assignment
# all the trailing newlines are lost as well
$ printf '%b' "$a" | cat -A
$
ab$
123
```

### ripgrep

If your search string doesn't have multiple trailing newlines or ASCII NUL characters, then you can save file contents to variables and then pass them to `ripgrep`. Single trailing newline will not normally cause an issue for searching operations as `ripgrep` will append a newline while displaying results anyway. If you want to make sure input file also contains the trailing newline, then you can manually concatenate a newline character to the search string.

```
$ s=$(< search_1.txt)
# use "$s"$'\n' if you want to match trailing newline as well
$ rg -NUF "$s" ip.txt
like . () * [] $ {}
^ + ? \ and ' and so on.

# use -- if the search string starts with - character
$ s=$(< search_2.txt)
$ rg -NUF -- "$s" ip.txt
-string multiline
search with cli tools.
```

For substitution operations, you'll have to preprocess the replacement file to replace `$` with `$$`.

```
$ s=$(< search_1.txt)
$ r=$(sed 's/\$/$$/g' replace.txt)

# here, removal of trailing newline doesn't cause an issue,
# as it evens out between search and replace strings
$ rg --passthru -NUF "$s" -r "$r" ip.txt
This is a multiline
sample input with lots
of special characters
---------------------
$& = $1 + $2 / 3 \ 4
=====================
This post shows how
you can do fixed
-string multiline
search with cli tools.
```

Here, partial line has to be matched. So, `$()` assignment works well for the search string. If the trailing newline of the replacement string isn't needed, then `$()` assignment again is good enough. Otherwise, you can modify the replacement string as `-r "$r"$'\n'`

```
$ s=$(< search_2.txt)
$ r=$(sed 's/\$/$$/g' replace.txt)

$ rg --passthru -NUF -r "$r" -- "$s" ip.txt
This is a multiline
sample input with lots
of special characters
like . () * [] $ {}
^ + ? \ and ' and so on.
This post shows how
you can do fixed
---------------------
$& = $1 + $2 / 3 \ 4
===================== with cli tools.
```

### sd

As mentioned before, `-s` option for `sd` applies to both search and replacement sections. So, the usage is lot simpler compared to `ripgrep`.

```
# -- is needed here because replace.txt starts with - character
$ sd -ps -- "$(< search_1.txt)" "$(< replace.txt)" ip.txt
This is a multiline
sample input with lots
of special characters
---------------------
$& = $1 + $2 / 3 \ 4
=====================
This post shows how
you can do fixed
-string multiline
search with cli tools.
```

### GNU sed

To follow a similar approach with `GNU sed`, you'll have to preprocess the strings to escape metacharacters. Assuming input doesn't have ASCII NUL characters, you can use `-z` option to slurp entire input as a single string.

Here's an example for multiline search.

```
# escape all BRE metacharacters
# replace literal newlines with \n
$ s=$(sed -z 's#[[^$*.\/]#\\&#g; s/\n/\\n/g' search_1.txt)

# since newlines are replaced with \n,
# trailing newlines will be preserved here
$ echo "$s"
like \. () \* \[] \$ {}\n\^ + ? \\ and ' and so on\.\n

# display filename if input matches the given multiline search string
# tr is used to change NUL character after filename to newline
$ sed -nz '/'"$s"'/F' ip.txt | tr '\0' '\n'
ip.txt
```

And here's an example for multiline substitution.

```
# last newline is removed here to allow partial line matching
$ s=$(sed -z 's#[[^$*.\/]#\\&#g; s/\n$//; s/\n/\\n/g' search_2.txt)

# escape all replacement section metacharacters
# and prefix \ character to literal newlines, except the last line
$ r=$(sed 's:[\\/&]:\\&:g; $!s/$/\\/' replace.txt)
$ echo "$r"
---------------------\
$\& = $1 + $2 \/ 3 \\ 4\
=====================

# if you need trailing newline from replace.txt,
# use sed -z 's/'"$s"'/'"$r"'\n/g'
$ sed -z 's/'"$s"'/'"$r"'/g' ip.txt
This is a multiline
sample input with lots
of special characters
like . () * [] $ {}
^ + ? \ and ' and so on.
This post shows how
you can do fixed
---------------------
$& = $1 + $2 / 3 \ 4
===================== with cli tools.
```