Packing and unpacking bytes
Tags: data, encoding
Published: 2023-07-22 00:04:09

**Goal**: A brief exploration of what it means to "pack" and "unpack" bytes.

## Inspiration

I've come across Ruby's [`Array#pack`](https://rubydoc.info/stdlib/core/1.9.3/Array:pack) and [`String#unpack`](https://rubydoc.info/stdlib/core/1.9.3/String:unpack) methods, but never had the time to dive into them. While researching another article, I came across [this question](https://codereview.stackexchange.com/questions/3569/pack-and-unpack-bytes-to-strings) and decided to stop to explore it.

## Exploration 1: Packing into two bytes

I can't define "packing", but I've gathered that it's a term for representing a series of bytes as a string. And depending on how you do it, you can even do this in fewer bytes than the original. Unpacking is the reverse: recovering the original information.

Trying an example based on the Stack Overflow question. I have a bunch of bytes, ie values between 0 \(`00000000`\) and 255 \(`11111111`\). Supposing I take two at random, maybe 126 and 2.

```js
let [a, b] = [126, 2]

console.log(a.toString(2).padStart(8, '0'))  // 01111110
console.log(b.toString(2).padStart(8, '0'))  // 00000010
```

I could represent them in a string by using the JS escape hexadecimal sequence:

```js
console.log(a.toString(16).padStart(2, '0'))  // 7e
console.log(b.toString(16).padStart(2, '0'))  // 02

console.log('\x7E\x02') // "~"
```

However, this isn't what I want, as this string has two characters. JavaScript strings are UTF\-16 \[[_note 1_](#note-1)\], so this string has 4 bytes, which is _more_ than the original.

```js
Buffer.from('\x7E\x02', 'utf16le').byteLength
// 4
```

This string has two characters of two bytes each: `00 7e` and `00 02`. I want to pack the bytes so the string has only one character, `7e 02`. Here's how:

```js
let char = String.fromCharCode((a << 8) | b)
console.log(char); // "縂"
Buffer.from(char, 'utf16le').byteLength // 2
```

This is a bit of [bit arithmetic](https://code.tutsplus.com/understanding-bitwise-operators--active-11301a) \(haha\).

* `a << 8` means "shift the bits in `a` left 8 times"
* shifting 126 \(`01111110`\) left 8 times gives us `01111110 00000000`
* `| b` is a bitwise `OR` operation
* `01111110 00000000` ORed with 2 \(`00000010`\) gives `01111110 00000010`, which is what I want \(`7E 02`\)

So there it is. I started with two bytes, and was able to fit them into a 2\-byte character \[[_note 2_](#note-2)\]. How about unpacking? Some more bitwise magic.

```js
let bytes = char.charCodeAt(0)
let byteA = bytes >> 8 // Shift the bits to the right 8 times to get the first byte
let byteB = bytes & 0xFF // Bitwise AND the bits with 11111111 to keep only the second byte
// Alternative:
// byteB = bytes ^ (byteA << 8)
console.log(byteA, byteB) // 126, 2
```

Cool, cool.

I also found out you can do this packing natively with [the `TextDecoder` API](https://developer.mozilla.org/en-US/docs/Web/API/TextDecoder)\! \[[_note 3_](#note-3)\]

```js
let byteArray = new Uint8Array([a, b])
let packedStr = new TextDecoder('utf-16be').decode(byteArray)
console.log(packedStr) // "縂"
```

However, unpacking with `TextEncoder` gives wrong results for this use case, since it only supports UTF\-8:

```js
let unpackedArray = new TextEncoder.encode(packedStr)
console.log(unpackedArray) // Uint8Array [231, 184, 130]
```

## Exploration 2: packing into one byte

Speaking of UTF\-8, it's time to try that. But I'm changing some things:

1. I won't use JS here, since its strings are UTF\-16. I probably can use it, but I don't want that headache. Plus, I love any excuse to work with Ruby.
2. All the bytes I'll pack are in the range **0 to 15**. I've intentionally made it smaller so that I can pack two bytes into one UTF\-8 character \(one byte\). I'll use 13 and 2 as my test bytes.

Packing in Ruby is pretty similar:

```rb
a, b = 13, 2

puts a.to_s(2).rjust(8, '0') # 00001101
puts b.to_s(2).rjust(8, '0') # 00000010

# hex
puts a.to_s(16).rjust(2, '0') # 0d
puts b.to_s(16).rjust(2, '0') # 02

char = ((a << 4) | b).chr # Shift by 4 bits, not 8, since I'm now packing in one byte
puts char # => "\xD2"
puts char.length # => 1
puts char.bytes.length # => 1

bytes = char[0].ord
byteA = bytes >> 4
byteB = bytes & 0x0F # AND with 0F, not FF, since I'm splitting up one byte
puts byteA, byteB # 13, 2
```

The output string here is a single byte "\\xD2"...which is simply the original `0D` and `02` bytes packed together 😀 Unfortunately, it's not a valid printable character, so printing it shows `�`, but it's there.

As mentioned earlier, Ruby has inbuilt `pack` and `unpack` methods, but they can only map byte to byte, so i couldn't use them for this example.

```rb
packed = [a, b].pack('c*') # => "\r\x02"
packed.unpack('c*') # => [13, 2]
```

But they work with the original UTF\-16 example:

```rb
a, b = 126, 2
packed = [a, b].pack('c*') # => "~\x02"
packed.unpack('c*') # => [126, 2]
```

It may not look like that, but the packed version here \("~\\x02"\) is _exactly_ the same as my manually packed JavaScript version. It contains the exact two bytes, `7E 02`. The difference is the encoding; in Ruby, this string is UTF\-8, so it's rendered differently. But I can change the encoding and see for myself\!

```rb
packed.force_encoding 'utf-16be' # => "\u7E02" 
packed.length # => 1
packed.bytes.length # => 2
```

## Possible uses of packing

Why would you want to pack, though? I'm thinking, perhaps in a constrained environment like gaming over the Internet. If there is a limited number of possible buttons a player can press \(say 12\), instead of transmitting each button press as one byte, I could:

* wait for a few milliseconds, to gather the next few keypresses and send in a batch
* pack these keypresses into a byte. 12 possible buttons can fit in 4 bits \(2^4 = 16\), so two keypresses can go in one byte \(8 bits\).

In this, packing serves as a form of compression, to send less data over the network and improve the gaming experience \(less data to download, so responses can be faster\).

I also found [this question](https://stackoverflow.com/questions/24684570/packing-unpacking-uuids-in-ruby), from a user who wanted to send a UUID as binary data. This is a valid use, since UUIDs are often rendered as strings, but they're actually a sequence of 16 bytes. Sending them as a string would take 36 bytes, so packing is useful here. You could also do this for other "binary\-but\-look\-like\-strings" data, like SHA\-512 hashes for instance.

Let me know if you can think of any other uses.

## Notes

[1.]() The ECMAScript spec says:

> When a String contains actual textual data, each element is considered to be a single UTF\-16 code unit.

So JS strings are UTF\-16. However, many modern Web APIs, like `Blob` and `TextEncoder`, and even older Node.js ones like `Buffer` assume \(or accept only\) UTF\-8. My guess is that they expect the string to be from the outside world \(reading a file, an API response, etc\), in which case, it's most likely UTF\-8.

[2.]()The only reliable way I found to get the byte length of a native JS string \(UTF\-16\) is `Buffer.from(string, 'utf16le').byteLength`. Commonly suggested ways I found include `TextEncoder` and `Blob`, but they always assume UTF\-8.

[3.]()For this to work as expected, I had to specify UTF\-16 Big Endian \(`utf-16be`\) as the encoding. UTF\-16 because I want 2\-bytes per character, and big\-endian because I want the **big** digits at the **end**, like I did in the custom packer.