rsync - 10 examples in 11 days (Day 03)
Tags: linux, rsync, bash, tutorial
Published: 2020-07-18 18:39:29

#Day 03

> 10 examples in 11 days

## Exclude and Include

Now it's time to explain the exclude and include option. This options are great if you want to get control of what needs to be sync.

With the **--exclude** option, you can exclude a file, a directory and even a list of files/directories listed inside a file. Let's check out the the **rsync** syntax with the **--exclude** option.

I will create a file called _exclude-me.txt_ inside _~/Users_ folder

```
iamgroot@laptop:~$ touch ~/Users/exclude-me.txt && echo "Hello World" >> ~/Users/exclude-me.txt
```

Now Let's check the size and files inside _~/Users_ and _~/Sync/Users_ folders

```
iamgroot@laptop:~$ stat -c "%s %n" ~/Users/* && stat -c "%s %n" ~/Sync/Users/*
12 /home/iamgroot/Users/exclude-me.txt
151 /home/iamgroot/Users/user01.json
57 /home/iamgroot/Users/user02.json
69 /home/iamgroot/Users/user03.json
43 /home/iamgroot/Users/user.config
151 /home/iamgroot/Sync/Users/user01.json
57 /home/iamgroot/Sync/Users/user02.json
69 /home/iamgroot/Sync/Users/user03.json
88 /home/iamgroot/Sync/Users/user.config
4096 /home/iamgroot/Sync/Users/Users
```

To exclude a file, we need to wrap the file name in quotes after the **--exclude** option. This way, **rsync** leaves the file in source and doesn't include it inside destination.

```
iamgroot@laptop:~$ rsync -vhr --exclude 'exclude-me.txt' ~/Users ~/Sync/
sending incremental file list

sent 146 bytes  received 17 bytes  326.00 bytes/sec
total size is 320  speedup is 1.96
```

Let's check if the file _exclude-me.txt_ was copied from source to destination

```
iamgroot@laptop:~$ stat -c "%s %n" ~/Users/* && stat -c "%s %n" ~/Sync/Users/*
12 /home/iamgroot/Users/exclude-me.txt
151 /home/iamgroot/Users/user01.json
57 /home/iamgroot/Users/user02.json
69 /home/iamgroot/Users/user03.json
43 /home/iamgroot/Users/user.config
151 /home/iamgroot/Sync/Users/user01.json
57 /home/iamgroot/Sync/Users/user02.json
69 /home/iamgroot/Sync/Users/user03.json
88 /home/iamgroot/Sync/Users/user.config
4096 /home/iamgroot/Sync/Users/Users
```

It wasn't included. Like I said, this option comes very handy. It also works with directories. Let's create a folder and include multiple files in it

```
iamgroot@laptop:~$ mkdir ~/Users/exclude-dir && touch ~/Users/exclude-dir/e{1..5}.txt && ls -a ~/Users/exclude-dir
.  ..  e1.txt  e2.txt  e3.txt  e4.txt  e5.txt
```

Now we have a folder called _exclude-dir_ with 5 files calles e1.txt, e2.txt, etc

Let's sync them excluding this folder

```
iamgroot@laptop:~$ rsync -vhr --exclude 'exclude-dir' ~/Users ~/Sync
sending incremental file list
Users/exclude-me.txt
Users/user.config
Users/user01.json
Users/user02.json
Users/user03.json

sent 731 bytes  received 112 bytes  1.69K bytes/sec
total size is 332  speedup is 0.39
jorge@code-lab:~$ stat -c "%s %n" ~/Users/* && stat -c "%s %n" ~/Sync/Users/*
4096 /home/jorge/Users/exclude-dir
12 /home/jorge/Users/exclude-me.txt
151 /home/jorge/Users/user01.json
57 /home/jorge/Users/user02.json
69 /home/jorge/Users/user03.json
43 /home/jorge/Users/user.config
12 /home/jorge/Sync/Users/exclude-me.txt
151 /home/jorge/Sync/Users/user01.json
57 /home/jorge/Sync/Users/user02.json
69 /home/jorge/Sync/Users/user03.json
43 /home/jorge/Sync/Users/user.config
```

We excluded the _exclude-dir_ folder but it copied the _exclude-me.txt_ file that we previously did not wanted to include at destination. How can we tell **rsync** to exclude this file as well?, easy, just tell **rsync** you want to eclude it as well

I will remove the file from destination and repeat the rsync command and check what was sync into destination

```
iamgroot@laptop:~$ rm ~/Sync/Users/exclude.me-txt
iamgroot@laptop:~$ rsync -vhr --exclude 'exclude-dir' --exclude 'exclude-me.txt' ~/Users ~/Sync
sending incremental file list

sent 150 bytes  received 17 bytes  334.00 bytes/sec
total size is 320  speedup is 1.92
iamgroot@laptop:~$ stat -c "%n" ~/Sync/Users/*
/home/iamgroot/Sync/Users/user01.json
/home/iamgroot/Sync/Users/user02.json
/home/iamgroot/Sync/Users/user03.json
/home/iamgroot/Sync/Users/user.config
```

Great, we achieve what we wanted.

If you really think about it, using mutiple **--exclude** options is not very clean and specially if you are working with multiple files/folders. There is a simple way to use **--exclude** in the same **rsync** line and its using curly braces

```
iamgroot@laptop:~$ rsync -vhr --exclude={'exclude-dir','exclude-me.txt'} ~/Users ~/Sync
sending incremental file list

sent 150 bytes  received 17 bytes  334.00 bytes/sec
total size is 320  speedup is 1.92
```

> Note: You have to watch out and not to include whitespaces between the files/folders isnide the curly braces, if you include a whitespace it will sync all files/folders

**GOOD** `rsync -vhr --exclude={'exclude-dir','exclude-me.txt'} ~/Users ~/Sync`

**NOT GOOD** `rsync -vhr --exclude={'exclude-dir', 'exclude-me.txt'} ~/Users ~/Sync`

Again, if we have multiple, by multiple a mean hundreds or even more, using this technique is not so handy.

The good news is that **rsync** have a way to do this easily hehe (well done **rsync**!!!).

Today, we are very used to use **git** and just like **git** has **.gitignore** with **rsync** is pretty much the same thing. We can create a file and list all the files we want to exclude. To achieve this, we have the **--exclude-from** option.

I will create a file called _rsync-ignore.txt_ and inside of it I will list all files that we want to exclude, notice that I will include the _rsync-ignore.txt_ file as well

```
iamgroot@laptop:~$ touch ~/Users/rsync-ignore.txt && echo -e "exclude-dir\nexclude-me.txt\nrsync-ignore.txt" >> ~/Users/rsync-ignore.txt && cat ~/Users/rsync-ignore.txt
exclude-dir
exclude-me.txt
rsync-ignore.txt
```

```
iamgroot@laptop:~$ rsync -vha --exclude-from='/home/iamgroot/Users/rsync-ignore.txt' ~/Users ~/Sync
sending incremental file list
Users/

sent 165 bytes  received 20 bytes  370.00 bytes/sec
total size is 320  speedup is 1.73
```

> notice I included the full path of the **--exclude-from**, using (~/) will not work

```
iamgroot@laptop:~$ ls -h ~/Sync/Users
user01.json  user02.json  user03.json  user.config
```

As you can see, everything worked just as expected. Now I will point out some other options to be used with the **--exclude** option.

## What about the Include option?

The include option behaves pretty much the same as **--exclude**, is just the opposite. It's useful when you mix them along the same line. Let's see an example

```
iamgroot@laptop:~$ rsync -avhm --include '*.json' --exclude '*.txt' ~/Users ~/Sync
building file list ... done
Users/
Users/user.config
Users/user01.json
Users/user02.json
Users/user03.json

sent 695 bytes  received 95 bytes  1.58K bytes/sec
total size is 320  speedup is 0.41
```

What this did was:

- included all .json files
- excluded all the .txt files
- using the -m option it didn't copy the exclude-dir folder since it was empty because inside there were only .txt files and got empty

Below are some more options to be used.

---

### Exclude Files using the wildcard (\*)

This option let's you specify a pattern you want to exclude. To achieve this, you need to use the wilcard (\*), it can also be used to exclude specific filetypes (it's all about how you use the wildcard. Let's see an example:

```
iamgroot@laptop:~$ rsync -avh --exclude '*.json' ~/Users ~/Sync
```

The above example will exclude all **.json** files. The * can be used at the beggining, at the end, in the middle, etc. i.e. `rsync -avh --exclude '*ignore*' ~/Users ~/Sync` will ignore the *rsync-ignore.txt\* file

### Exclude Files based in File Size

This option let's you specify between the maximum or minimum size of the files you want to exclude.

```
iamgroot@laptop:~$ rsync -avh --min-size=1K --max-size=50K ~/Users ~/Sync
```

The above example will exclude any file bigger than 1K and smaller than 50K

Here is what the man page says about it

> **--max-size=SIZE**
>
> This tells rsync to avoid transferring any file that is larger than the specified SIZE. The SIZE value can be suffixed with a string to indicate a size multiplier, and may be a fractional value
> (e.g. "--max-size=1.5m").
>
> This option is a transfer rule, not an exclude, so it doesn’t affect the data that goes into the file-lists, and thus it doesn’t affect deletions. It just limits the files that the receiver requests to be transferred.
>
> The suffixes are as follows: "K" (or "KiB") is a kibibyte (1024), "M" (or "MiB") is a mebibyte (1024*1024), and "G" (or "GiB") is a gibibyte (1024*1024\*1024). If you want the multiplier to be 1000 instead of 1024, use "KB", "MB", or "GB". (Note: lower-case is also accepted for all values.) Finally, if the suffix ends in either "+1" or "-1", the value will be offset by one byte in the indicated direction.
>
> Examples: --max-size=1.5mb-1 is 1499999 bytes, and --max-size=2g+1 is 2147483649 bytes.
>
> Note that rsync versions prior to 3.1.0 did not allow --max-size=0.
>
> **--min-size=SIZE**
>
> This tells rsync to avoid transferring any file that is smaller than the specified SIZE, which can help in not transferring small, junk files. See the --max-size option for a description of SIZE and other information.
>
> Note that rsync versions prior to 3.1.0 did not allow --min-size=0.

---

Ok, that'll be for today's example, thanks for reading!!!

Follow, ❤ or 🦄