bash: splitting tarballs the ‘easy’ way
Tags: linux, bash
Published: 2024-05-22 14:48:13

there are times when we `tar` and `gzip` a directory and the final tarball is just too damn big. maybe it doesn’t fit on any of those old thumbdrives we got at the 7-eleven, or maybe we’re trying to [upload it to s3](https://dev.to/gbhorwood/uploading-to-s3-with-bash-2jmn) and aws is complaining it’s too large.

let’s take a look at a quick and dirty way to split our tarballs into a bunch of smaller files of a set size.

we’ll go over creating the segmented tarballs first and then cover how to put them back together into something usable.

first, we can create our tarball parts like so:

```bash
$ tar czpf - /path/to/directory | split -d -b 10M - tarballname.tgz-part
```

this little pipeline has two components: a call to `tar` to pack up our directory and a call to `split` to slice it all up into ten meg chunks.

## the `tar` part
we’re calling `tar` here with the switches `czpf`. let’s look at what those actually mean:

* **`-c`** this stands for ‘create’, as opposed to ‘extract’.
* **`-z`** the ‘z’ is for ‘zip’. in this case, gzip
* **`-p`** this is for ‘preserve permissions’
* **`-f`** this means ‘file name’. if we don’t use this, tar will chose a filename for us, and we won’t like it

the next interesting bit is the lone dash as the first path argument. normally, when using `tar` with `-f` we provide an input directory and an output file name as arguments. in this example, we don’t want to do that: `split` will be creating our name for us. by providing that single `-` character, we’re telling `tar` to take all of our output and send it to `STDOUT` instead of to a file. this allows us to use our pipe operator.

generally speaking, when we see that single dash, we are using `STDIN` if reading or `STDOUT` if writing, instead of a file.

add all this together, and we have a `tar` call that is tarring and zipping our directory and dumping the output into the pipe operator for split to pick up.

## the `split` part
the `split` command here is reading it’s input from `STDIN` through that pipe. there’s that single dash argument, again, telling `split` that we’re using the piped-in contents instead of a file.

the `-b 10M` argument here is where we set the maximum size of our parts, in this case ten megs. although `-b` stands for ‘bytes’, we can use more convenient measurement units here; `M` for megabytes and `G` for gigs.

lastly, we have the `-d` switch to tell `split` to add some numbers to the end of our output file names. the ‘d’ stands for ‘digits’. if we don’t use this switch, `split` will suffix our files with increments like ‘aa’, ‘ab’, and so on. hideous stuff. never send a letter to do a number’s job.

the `tarballname.tgz-part` argument is just the prefix for the name of all our tarballs parts. we can call it whatever we want.

after running this pipeline on a suitably large directory, we’ll get a list of files that looks something like:

```bash
$ ls -1
thename.tgz-part00
thename.tgz-part01
thename.tgz-part02
thename.tgz-part03
```

great stuff.

## putting it all back together
of course, splitting up a tarball is no good if we can’t re-assemble and extract it. let’s do that.

```bash
$ cat thename.tgz-part* | tar xzpf -
```

most people use `cat` as a quick way to dump a file to screen but, fun fact, ‘cat’ stands for ‘concatenate’. it can take a list of files and stick them all together in order.

here, we’re `cat`ting all the parts of our tarball together and dumping the output through a pipe.

on the other side of the pipe, we’re calling `tar` with `xzpf` to extract our archive. note, again, that single dash that tells tar to accept input from `STDIN` and not a file.

when we’re done, our directory is on disk, untarred. victory.

> 🔎  this post was originally written in the [grant horwood technical blog](https://gbh.fruitbat.io/2024/05/21/bash-splitting-tarballs-the-easy-way/)