Quick intro to Bash I/O redirections, *here documents* and *here strings*
Published: 2021-12-27 09:20:44

*Note: this article was originally published on my personal blog. Check it out [here](https://bastien-antoine.fr).*

If you have done a bit of scripting in Bash, you may know that you can redirect the input and output of the commands to other commands or files. This article introduces the basic of I/O redirections and explains some of the syntax that may seems cryptic at first.



- [`stdin`, `stdout` and `stderr`](stdin-stdout-and-stderr)
- [Controlling the program input and output](controlling-the-program-input-and-output)
	- [Controlling the input](controlling-the-input)
		- [*Here documents* and *here strings*](here-documents-and-here-strings)
	- [Controlling the output](controlling-the-output)
		- [Controlling `stdout`](controlling-stdout)
		- [Controlling `stderr`](controlling-stderr)
		- [A note on the pipe operator](a-note-on-the-pipe-operator)



# `stdin`, `stdout` and `stderr`

In Bash every command or script has three files[^1] opened when running:

1. `stdin` (*STanDart INput*): the input of the data fed to the program
2. `stdout` (*STanDart OUTput*): the location where the program will output anything it needs to output
3. `stderr` (*STanDart ERRor*): the location where the program will output any error occured during the run

Each of them get assigned to a specific *file descriptor* (ie. a number assigned to the file used by the OS to handle it): O for `stdin`, 1 for `stdout` and 2 for `stderr`. These numbers allow you to control and use them in your program.

# Controlling the program input and output

## Controlling the input

Depending on the programm that you want to run, there might be different ways to control on their input is provided: some are built to accept files, some will read only from `stdin` and others may accept both.

When it comes to directly providing files to the program, I'll let you read their manpage to know how to do so.

When your program is launched directly in a terminal, its `stdin` is attached to your keyboard, providind you with a way to input the data.

If you want to use something else as the input, there are a few ways:

- Using the `<` operator: this operator allows to use the content of a file as the input data stream.

```shell
$ cat >input-file <<EOF
Line 1
Line 2
Line 3
Line 4
Line 5
Line 6
Line 7
Line 8
Line 9
Line 10
Line 11
EOF
$ grep "^Line 1" <input-file
Line 1
Line 10
Line 11
```

(See section [below about *here docs*](#here-documents-and-here-strings) to understand the syntax of the first command.)

You can also use the `0<` operator, which is equivalent to `<`. The 0 here is a reference to the file descriptor 0 attached to `stdin`, but because a program only has one `stdout`, it can be abreviated from `0<` to `<`.

- Using the pipe `|` operator: the pipe operator is a special operator that combines input and output redirection to allow chaining commands. See [below](#a-note-on-the-pipe-operator) for more info.

- Using *here documents* and *here strings*

### *Here documents* and *here strings*

*Here documents* (or *heredoc*) are a special form of input redirection allowing to input a multiline string to a program. The syntax is the following:

```shell
COMMAND <<DELIMITOR
...
...
DELIMITOR
```

`DELIMITOR` is used to control when to start and stop the input. It can be any string without spaces, and should then be choosed wiselly so that it won't apear in the input:

```shell
$ cat <<ThisIsADelimitor
Hello
There
ThisIsADelimitor
Hello
There
$ cat <<1
Hello
There
1
Hello
There
```

*Heredoc* allows for parameter expansion as well:

```shell
$ export FIRSTNAME='Obi-Wan'
$ export LASTNAME='Kenobi'
$ cat <<EOF
Hello $FIRSTNAME $LASTNAME
EOF
Hello Obi-Wan Kenobi
```

If you have played a bit with the *heredocs*, you may have noticed that tabs included in the input were kept:

```shell
$ cat <<EOF
        hello
there
EOF
    hello
there
```

Fortunately you can strip them by prefixing the delimitor with a dash:

```shell
$ cat <<-EOF
        hello
there
EOF
hello
there
```

This will strip all leading tabs, allowing the input to be more readable visually. Note though that this strips only leading tabs, not spaces, so be careful, especially if you copy/paste the example above.

#### *Here strings* 

*Here strings* are a more consise version of the *here documents*. They only allow a single line input, and thus don't need the use of a string delimitor. Like the *heredocs*, it allows for parameter expansion:

```shell
$ cat <<EOF > file
This is
a
multiline
file
EOF
$ export TITLE="Title of the file"
$ export FILE="file"
$ cat - $FILE <<<$TITLE > $FILE.new
$ cat $FILE.new
Title of the file
This is
a
multiline
file
```

## Controlling the output

Like for the input, there are a few ways to redirect the output of a program, but unlike the first, a program has two type of possible outputs, `stdout` and `stderr`, so there are ways to control which output gets redirected.

When the program is launched directly in a terminal, its `stdout` and `stderr` are attached to the terminal, so everything the program outputs gets printed on your screen.

### Controlling `stdout`

- Using the `>` operator: redirects `stdout` to the provided filename, creates it if needed, otherwise overwrite it.

```shell
$ ls -la > list-files.txt
```

- Using the `>>` operator: same as `>`, but appends to the end of the file if it already exists

```shell
$ ls -lat | head -n3 >> most-recents-files.txt
```

### Controlling `stderr`

- Using the `2>` operator: same as `>` for `stdout`, but for `stderr`

```shell
$ export ERRORFILE=script.errors
$ bad_command1 2>$ERRORFILE
$ cat $ERRORFILE
zsh: command not found: bad_command1
```

- Using the `2>>` operator: same as `>>` for `stdout`, but for `stderr`

```shell
$ export ERRORFILE=script.errors
$ bad_command1 2>>$ERRORFILE
$ cat $ERRORFILE
zsh: command not found: bad_command1
```

- `2>&1`: redirects `stderr` to `stdout`

- `&> filename`: redirects both `stdout` and `stderr` to `filename`

### A note on the pipe operator

The pipe operator `|` is special operator that allows to chain multiple commands by binding the `stdout` of one to the `stdin` of the following:

```shell
$ ls -la | wc -l
15
```

The example above would be roughly the same as the following script:

```bash
#!/bin/bash
filename=temp
ls -la >$filename
exec 3<$filename
wc -l <&3
3<&-
rm -rf $filename
```

# Sources 

- [*Advanced Bash-Scripting Guide: Chapter 20. I/O Redirection*](https://tldp.org/LDP/abs/html/io-redirection.html#IOREDIRREF)
- [*Advanced Bash-Scripting Guide: Chapter 19. Here Documents*](https://tldp.org/LDP/abs/html/here-docs.html)
- [*Advanced Bash-Scripting Guide: Chapter 19.1. Here Strings*](https://tldp.org/LDP/abs/html/x17837.html)

[^1]: While they may use regular files as input or output these are actually more like data streams, but by convention the UNIX and Linux world treat them like files.