managing disk space in bash
Tags: linux, bash
Published: 2024-04-29 15:23:42

a week ago, i ran this db insert-heavy script on a server and forgot to turn off mysql‘s binlog feature first. the result, of course, is that i filled up the disk in about three minutes and brought the whole server down. not great for a tuesday. fortunately, finding and fixing the problem was straightforward. the downtime was only a couple of minutes.

in this post we’re going to go over inspecting our disk space; figuring out how much we have left and finding out what we spent all those blocks on. we’ll look at three basic tools:

`df` for inspecting disk space
`du` for getting directory sizes
`find` for finding files to delete

these all come pre-packaged with your linux or linux-like operating system, so put that `apt` back in your pocket.

##using df to find out how much space you actually have

`df` stands for “disk free” and, as the name implies, it tells you how much free space you have on your disk (plus some other stuff). let’s look:
```bash
$ df
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/root       24199616 22983688   1199544  96% /
tmpfs            1010020        0   1010020   0% /dev/shm
tmpfs             404012      916    403096   1% /run
tmpfs               5120        0      5120   0% /run/lock
/dev/xvda15       106858     6165    100693   6% /boot/efi
/dev/xvdf      102626232 68248896  29118072  71% /var/backups
/dev/xvdg       61611820  5569548  52880160  10% /var/www/html
tmpfs             202004        4    202000   1% /run/user/1000
```

this is useful data. kinda. it shows all the partitions we have and how big they are and how much is available, and it even gives us a ‘percent used’ column. but those numbers are all in blocks. if we want something more human-readable, we can apply the `-h` switch which stands for, conveniently, ‘human readable’:
```
$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        24G   22G  1.2G  96% /
tmpfs           987M     0  987M   0% /dev/shm
tmpfs           395M  916K  394M   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/xvda15     105M  6.1M   99M   6% /boot/efi
/dev/xvdf        98G   66G   28G  71% /var/backups
/dev/xvdg        59G  5.4G   51G  10% /var/www/html
tmpfs           198M  4.0K  198M   1% /run/user/1000
```
now we can easily see that our root partition disk is 22 gigs and we have 1.2 gigs available. much better.

###filtering df output

of course there’s still the issue of all those `tmpfs` entries. we don’t really care about them. fortunately, we can limit `df`‘s output to just `ext4` partitions with the `--type=<filesystem>` argument:

```bash
$ df -h --type=ext4
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        24G   22G  1.2G  96% /
/dev/xvdf        98G   66G   28G  71% /var/backups
/dev/xvdg        59G  5.4G   51G  10% /var/www/html
```

if we’re not sure what file systems we have mounted, we can check with `df -T`

```bash
$ df -T | awk '{print $2}' | sort | uniq
Type
ext4
tmpfs
vfat
```

here we got `df` to output only the filetype of each partition. we then used a little `awk` to strip away the clutter and piped the results through `uniq` to give us a nice list of all the file systems we have.

or maybe we want to get the results for every partition *except* `tmpfs`. we can do that with `--exclude-type=<filesystem>`

```bash
$ df -h --exclude-type=tmpfs --exclude-type=vfat
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        24G   22G  1.2G  96% /
/dev/xvdf        98G   66G   28G  71% /var/backups
/dev/xvdg        59G  5.4G   51G  10% /var/www/html
```

we can pass as many `--exclude-type` arguments as we want. here, for instance, we’re excluding both `tmpfs` and `vfat`.

if we want to get data about only one partition, we can pass a path to a file or directory as an argument and `df` will only report on the partition where that file or directory resides. this is great stuff if, like me, you can’t be bothered to figure out what partition the directory you can’t write to lives on.

```bash
$ df -h /path/to/dir/or/file
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvdg        59G  5.4G   51G  10% /path/to
```

###what we really care about is percentage use

if we’re writing some sort of home-rolled disk-monitoring script, probably the thing we’re most worried about is the percentage used. anything north of 90 and maybe it’s time to freak out.

we can tell `df` to just give us the percentage used by passing the `--output` argument with `pcent` like so:

```bash
$ df -h --output=pcent /var/www/html
Use%
 10%
```

and if we want that as a nice integer that we can use in an `if` statement, we can apply a little `awk` and `sed`.

```bash
$ df -h --output=pcent /var/www/html | awk END'{print $0}' | sed -e 's/%\| //g'
10
```

the `--output` argument takes a bunch of possible values, the most-useful ones being:

    **`avail`** the amount of available space
    **`used`** the amount of used space
    **`source`** the name of the partition, ie. /dev/sdb
    **`target`** the mount point of the partition, ie. /var/www/html

maybe we also want a nice digest of our disk space usage that we can put in our `motd` output, so we see it every time we log in. for that we can combine some `df` arguments we’ve learned and use `tail` to strip off the headers.

```bash
$ df --output=target --output=pcent --type=ext4 | tail -n +2
```

throw that in our `motd` script and we’ll be greeted with

```bash
/              79%
/var/www/html  65%
```

##inspecting directories with du

knowing that our disk is full is great (well, not “great”, per se), but if we’re going to fix it, we’re going to need to know where all that disk space is going. how big is our `/var/log/` dir? how much cruft do we have stuffed in `/tmp/`? that sort of thing.

the `du` command shows us the disk usage of any directory and the files in it. that’s why it’s called `du`; it’s about ‘disk usage’.

```bash
$ du /path/to/directory/
```

running `du` without any arguments outputs an unholy mess: the size of every single file in every single directory in the target path. if you have a lot of files, this can take a coffee-making length of time. not super useful or efficient.

we can limit that output with the `-csh` switches:

```bash
$ du -csh /path/to/dir/
433M    /path/to/dir/
433M    total
```

let’s look at those switches:

**`-c`** output only the total disk usage of the directory.
**`-s`** output only the ‘summary’ for the target directory, not the size of every single thing in it.
**`-h`** show the results in a ‘human readable’ format. we want to see gigs and megs, not blocks. 

as an alternative to `-h`, we can use `-k` to get our output in kilobytes without the unit displayed. this is great stuff if we’re writing a script that needs a number we can do math with.

if we want our `du -csh` call to just output the total number and unit without any of the explanatory cruft, we can apply a little `awk` to clean it up.

```bash
$ du -csh /path/to/dir/ | awk 'END{print $1}'
433M
```

of course, that’s a hassle to remember (and type!) every time we want to know how much space our mp3 collection is taking up. if we’re using the bash shell, we can pack that line into a nice function:

```bash
$ dtotal() { du -csh $1 | awk 'END{print $1}'; }
```

and use it like so:

```bash
dtotal /path/to/dir
433M
```

##finding disk hogs with find

finding the directories that are eating up our disk space only gets us so far. we’re going to want to find specific files that are candidates for deletion. we can do that with `find`. 

usually, we use the find command to search for files by name, but we can also use the `--size` argument to search for files that are large:

```bash
$ find /path/to/directory/ -size +1G
/path/to/directory/very-large-tarball.tar
```

here, we’re searching for every file in a directory that has a size greater than one gig. if we want to search by megs instead, we could use something like `-size +500M`, and if, for some reason, we’re looking for smaller files, we can use a minus sign where the plus sign goes to get files smaller than the stated size.

###filtering find by file age

if our disk space problem is a new thing, we’re going to want to find large files that were created or written to recently. we’ve all known devs who’ve done a `print_r()` of a large object to a file in a loop and let it run for weeks. we can find that file by narrowing our find down to files that have been modified in the last day.

```bash
$ find /path/to/directory/ -size +1G -mtime -1
/path/to/directory/very-large-and-recent-tarball.tar
```

the `-mtime` argument here means ‘modified time’ and the unit of `-1` is days. by passing `-mtime -1` to find we’re saying we want files larger than one gig that were modified less than one day ago.

of course we can use any number of days we wish, or pass something like `-mtime +7` to find files that were modified more than seven days ago.

great for ferreting out those old backup files we totally forgot about.

> 🔎  this post was originally written in the [grant horwood technical blog](https://gbh.fruitbat.io/2024/04/29/managing-disk-space-in-bash/)