How the Unix Shell sees the world (Expansion, Quoting)
Tags: bash, linux, computerscience, beginners
Published: 2019-12-19 16:17:09

#TL;DR
It was me using Unix terminal every single day, typing hundred lines of terminal command, but sometimes, I encounter some unexpected behaviors at the moment I press Enter button after a command. It's pretty frustrated at some points, it turns out that the way the Shell sees the world is different than mine. I read some books (at least some chapters) about how the Shell acts upon a command, I find out it's really interesting so that I share with you today. Hope this helpful.

Because this is pretty straightforward. With only **echo** command, we can examine several interesting features, the "magic" of the Shell

#The Expansion

Every time we press enter after a command, it get transformed by the Shell under the hood before get carried out. The process that makes it happen is call **expansion**. The expansion is that when you type in something, it get expanded to another thing before the shell acts upon it. For example: The * means a lot to the Shell. To illustrate this, I will use the **echo** command, as you recall that **echo** is a simple Shell built-in function that directs our typing to standard output
```bash
phuong@Arch ~$ echo Hello world
Hello world
```

So the **\*** wildcard means: Match any characters in file-name. The fact  that the * get transformed into other characters before being carried out by the shell so that the echo command never see the *

```bash
phuong@Arch ~/kitchen$ echo *
bash  device  devops  gateway  go  hook  iuh  js  linux  ml  python
```
### Filename expansion

In fact, this behavior carried out by a mechanism, which is popularly called *path name expansion*. As we know, file names begin with period character (ex: .git) is hidden. The path name extension respect this rule. An expansion does not reveal hidden files.
```bash
phuong@Arch ~$ echo D*
Desktop Documents Downloads
```
But we can show all hidden files by following command:
```bash
phuong@Arch ~$ echo .*
. .. .3T .ansible .anydesk .bash_history .bash_logout .bash_profile .bashrc .bashrc.backup .boto .cache .config .dbshell .designer .docker
```

### Arithmetic expansion
We are also able to use our terminal command as a calculator thanks to arithmetic expansion, but this it only works with simple integers (We need tools like bc,... for float numbers). In spite of its limitation, it might useful in some scenarios, it also supports many popular operators. Let's say: *, -, **, %, /, +,...
Arithmetic expansion follows below pattern:

**$((expression))**

```bash
phuong@Arch ~$ echo This month has $((4*7)) days
This month has 28 days
```
As an alternative to double parentheses, we can also use single parentheses to nest sub expansion. Such as:
```bash
phuong@Arch ~$ echo This year has $((((4*7)*12) + $((3 * 9)))) days
This year has 363 days
```

###Tilde expansion
When we using *~* as the beginning of the string in terminal, it expands to home directory of named user.
Let's say, I am using an account on the cloud where there are many users there and I don't known where their home directory are, I just know usernames of them. That is where it come into play

``` bash
phuong@devcloud1:~$ echo ~wingzero
/home/wingzero
phuong@devcloud1:~$ echo ~phuong
/home/phuong
```
###Brace expansion
The brace expression itself may contain either a comma-separated list of strings or a range of integers or single characters.

For the comma-separated example:
```bash
phuong@Arch ~/kitchen/ips$ echo A{A,B}
AA AB
```
Similarly, the range of single characters
```bash
phuong@Arch ~$ echo {A..Z}
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
```
From my point of view, this kind of expansion is one of the most interesting expansion mechanisms as it is pretty useful in specified situations. Suppose that I want to create each folder for each network IP in the subnet range. Pretending that **192.168.43.1 -> 192.168.43.13**. So  I confess the way that I would do 4 months a go is that: **mkdir ...**. But this approach is prone to typo errors. So that is where *the brace expansion* comes and saves.
```bash
phuong@Arch ~/kitchen/ips$ mkdir 192.168.43.{1..13}
phuong@Arch ~/kitchen/ips$ ls
192.168.43.1   192.168.43.12  192.168.43.3  192.168.43.6  192.168.43.9
192.168.43.10  192.168.43.13  192.168.43.4  192.168.43.7
192.168.43.11  192.168.43.2   192.168.43.5  192.168.43.8

```
It is true as you may guest that the brace expansion can also be nested.
```bash
phuong@Arch ~/kitchen/ips$ echo A{a{1,2},b{9,10}}
Aa1 Aa2 Ab9 Ab10
```
###Command substitution
Beside above expansion mechanisms, the *command substitution* maybe the one that you might feel familiar with because its widely used in scripting. The *command substitution* allows us to use the output of a command as an expansion. For instance:

```bash
phuong@Arch ~/kitchen$ ls -l $(which go)
lrwxrwxrwx 1 root root 18 Nov  2 17:25 /usr/bin/go -> /usr/lib/go/bin/go
```
Here is another example

```bash
phuong@Arch ~$ echo Hello $(w=world; echo $w)
Hello world
```

###Parameter substitution
This kind of substitution is intensively used in scripting than directly in command line. For simple example, you want to print USER variable, you just add dollar symbol($) at the beginning.
```bash
phuong@Arch ~$ echo $USER
phuong
```
Note that if you mistyping the name of a variable, the expansion will still take place, but it results into empty string.

For the sake of simplicity, I will not go into detail. Because this powerful mechanism deserve another post for it. The scope of this post is to point out some popular expansion mechanisms and how to control them.

#Quoting

So we have seen how powerful the expansion in the Shell is, that is why it could lead to some unexpected behaviors unless we know how to control it.
Now we come to interesting part.

Consider two samples below:
Example 1:
```bash
phuong@Arch ~$ echo Hello I am a          string
Hello I am a string
```
Example 2:
```bash
phuong@Arch ~$ echo It costs me $200
It costs me 00
```
Feel familiar ah?
![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/7etomm2e1wz9s0a6us46.gif)

In the first example, the *word splitting* ditches extra white spaces from *the echo command*'s list of arguments. In the second example, the *$2* is expanded by parameter expansion mechanism, that is on purpose, the problem is $2 variable is undefined so the Shell suppress unexpected expansion which finally leads to how it behaves. Let's figure out how to fix this.

###Double Quotes
If you put special characters into *double quotes* will lost their special meanings to the Shell, but there are 3 exceptions for that, namely *dollar sign, backslash and back tick*.

This is pretty useful in specified scenario. To show you what I mean, imagine that one of your friends send you a homework file named "*homework part1.doc*". Well, that might drive you crazy.

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/j4nhsnpvepdbfjqxm553.png)

Which leads you to below situation

```bash
phuong@Arch ~/Downloads$ cat homework part1.doc 
cat: homework: No such file or directory
cat: part1.doc: No such file or directory
```

The double quotes comes to save you.

```bash
phuong@Arch ~/Downloads$ mv "homework part1.doc" homework_part1.doc 
phuong@Arch ~/Downloads$ cat homework_part1.doc 
This is my stupid homework part, now is your turn
```
Once double quotes are added , our command contains a command followed by a single argument. That is how we preserve our extra white spaces, because *world splitting* mentioned above only take effect for multiple parameters. With everything putted in double quotes, it behave like only one parameter following the echo command

```bash
phuong@Arch ~/Downloads$ echo Hello I am a          string
Hello I am a string
phuong@Arch ~/Downloads$ echo "Hello I am a          string"
Hello I am a          string
```

###Single quote
If you simply need to suppress all special characters, just use single quote
```bash
phuong@Arch ~/Downloads$ echo 'cat ~/Download/.doc {a,b} $(echo bar) $((3+2)) $USER'
cat ~/Download/.doc {a,b} $(echo bar) $((3+2)) $USER
```

###Escaping characters
Sometimes we want to quote only a single character, to do this, just preceding that character with a *backslash* \\
```bash
phuong@Arch ~/Downloads$ echo "You own me $30"
You own me 0
phuong@Arch ~/Downloads$ echo "You own me \$30"
You own me $30
```
It's common to selectively prevent the expansion of a filename.


#Conclusion
To summarize, the expansion mechanisms in Unix Shell is great and powerful. If we know how to control it, I believe those features surely enhance your productivity. If you have any question, feel free to drop a comment below, that would be nice to hear from you.

Thanks for your time.

#References
- The Linux Command Line - William E. Shotts JR.