JSON compression in the browser, with gzip and the Compression Streams API.
Published: 2023-03-19 23:40:14

[... just not yet in Firefox](https://caniuse.com/mdn-api_compressionstream_compressionstream_gzip)

## Introduction
When building web apps, we sometimes find ourselves with a need to persist large JSON objects. Either directly to the browser using [Web Storage](https://developer.mozilla.org/en-US/docs/Web/API/Web_Storage_API), or externally through the [File](https://developer.mozilla.org/en-US/docs/Web/API/File) or [Fetch](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API) APIs.

This problem lead me to ask the question: **Is there a better solution to storing raw or encoded JSON? Can we achieve some level of compression?**

As it turns out, the answer is __Yes__ and it's surprisingly simple with the [Compression Streams API](https://developer.mozilla.org/en-US/docs/Web/API/Compression_Streams_API).

This example was created for the specific needs of [Web Storage](https://developer.mozilla.org/en-US/docs/Web/API/Web_Storage_API), [File](https://developer.mozilla.org/en-US/docs/Web/API/File) and [Solid Pod](https://solidproject.org/) persistence, for a scaling representation of JSON application state.

## Background 
To set the scene a little, the project I am working on doesn't communicate with a server. Application data is structured transactionally in a [Merkle Tree](https://en.wikipedia.org/wiki/Merkle_tree) (git, blockchain). The tree structure naturally grows with each interaction with the application and along with it, its memory and storage footprint.

To keep an eye on this growth, I built a small indicator in the control bar to visualise the size. I found as the JSON object grew, the data started to reach an extent I wasn't completely comfortable with, so I decided to take a look and see if I could do better.


![943kb raw JSON](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/inpxwl2pbdeaptq74zru.png)

This brought me to the [Compression Streams API](https://developer.mozilla.org/en-US/docs/Web/API/Compression_Streams_API). A browser API that enables [`gzip`](https://en.wikipedia.org/wiki/Gzip) compression on a [`ReadableStream`](https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream), directly in the browser.

__TL;DR__ After enabling `gzip` compression on my JSON data, I saw a size reduction of 98%, from `943kb` to `10kb`.

![10kb compressed JSON](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sgywxw3z2b08zhow0kzy.png)

This wasn't a scientific test and the data didn't scale how it would organically in the app, but it serves as a fair benchmark that I will get the results I want from this API.

## Solution

As the [Compression Streams API](https://developer.mozilla.org/en-US/docs/Web/API/Compression_Streams_API) is part of the [Streams API](https://developer.mozilla.org/en-US/docs/Web/API/Streams_API), we first need to transform our JSON data object to a [ReadableStream](https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream).

A [Blob](https://developer.mozilla.org/en-US/docs/Web/API/Blob) can be constructed from text, and JSON can be serialized to text, so to start we can create a [Blob](https://developer.mozilla.org/en-US/docs/Web/API/Blob) from our JSON and get our stream from there.

```typescript
// Convert JSON to Stream
const stream = new Blob([JSON.stringify(data)], {
    type: 'application/json',
}).stream();
```

Now we have our [`ReadableStream`](https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream), we can use the [`ReadableStream.pipeThrough`](https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream/pipeThrough) method to pipe our data through the `gzip` [`CompressionStream`](https://developer.mozilla.org/en-US/docs/Web/API/CompressionStream) transform.

```typescript
// gzip stream
const compressedReadableStream = stream.pipeThrough(
    new CompressionStream("gzip")
);
```

Then to get this compressed stream back to our main code, we can create a new [Response](https://developer.mozilla.org/en-US/docs/Web/API/Response) and handle the data like any other we would receive from the [Fetch API](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API).

```typescript
// create Response
const compressedResponse = 
  await new Response(compressedReadableStream);
```

So what can we do with this data now it's compressed?

__*Well, quite a lot actually.*__

We could base64 encode it to a string and then store it in [`localStorage`](https://developer.mozilla.org/en-US/docs/Web/API/Window/localStorage).

```typescript
// Get response Blob
const blob = await compressedResponsed.blob();
// Get the ArrayBuffer
const buffer = await blob.arrayBuffer();

// convert ArrayBuffer to base64 encoded string
const compressedBase64 = btoa(
  String.fromCharCode(
    ...new Uint8Array(buffer)
  )
);

// Set in localStorage
localStorage.setItem('compressedData', compressedBase64);
```

We could save it as a [File](https://developer.mozilla.org/en-US/docs/Web/API/File/File).


```typescript
// Get response Blob
const blob = await compressedResponse.blob();

// Create a programmatic download link
const elem = window.document.createElement("a");
elem.href = window.URL.createObjectURL(blob);
elem.download = '';
document.body.appendChild(elem);
elem.click();
document.body.removeChild(elem);
```

Or we could take more traditional approach, sending through HTTPS and the [Fetch API](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API).

```typescript
// Get response Blob
const blob = await compressedResponse.blob();

await fetch(url, {
  method: 'POST',
  body: blob,
  headers: {
    'Content-Encoding': 'gzip',
  }
});
```

And I'm sure there are many other techniques that can be used with our newly compressed JSON object. 

__*Pretty cool, right? But how do we get our data back into a readable form in the browser?*__

To decompress, we first want to get our compressed data back into a [`ReadableStream`](https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream).

Either directly from our [Blob](https://developer.mozilla.org/en-US/docs/Web/API/Blob) (or file/fetch [Response](https://developer.mozilla.org/en-US/docs/Web/API/Response))
```typescript
const stream = compressedResponse.blob().stream();
```
Or decode back from base64.

```typescript
// base64 encoding to Blob
const stream = new Blob([b64decode(compressedBase64)], {
  type: "application/json",
}).stream();
```

Edit: The b64 decode function I'm using looks like this.
```typescript
export function b64decode(str: string): ArrayBuffer {
  const binary_string = window.atob(str);
  const len = binary_string.length;
  const bytes = new Uint8Array(new ArrayBuffer(len));
  for (let i = 0; i < len; i++) {
    bytes[i] = binary_string.charCodeAt(i);
  }
  return bytes;
}
```

Then again, this time with our compressed  [`ReadableStream`](https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream), we go back through the [`ReadableStream.pipeThrough`](https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream/pipeThrough) method, piping our data through the `gzip` [`DecompressionStream`](https://developer.mozilla.org/en-US/docs/Web/API/DecompressionStream).

```typescript
const compressedReadableStream = stream.pipeThrough(
  new DecompressionStream("gzip")
);
```
We then go back into a [Response](https://developer.mozilla.org/en-US/docs/Web/API/Response), take the [Blob](https://developer.mozilla.org/en-US/docs/Web/API/Blob) and call the [`Blob.text()`](https://developer.mozilla.org/en-US/docs/Web/API/Blob/text) method to get the content back in string format.

```typescript
  const resp = await new Response(compressedReadableStream);
  const blob = await resp.blob();
```

All to do now is parse this string back to JSON and we have our decompressed object, back in Javascript and ready to use.

```typescript
const data = JSON.parse(await blob.text());
```

## Demo and Results

I've put together a small [CodeSandbox](https://codesandbox.io/s/relaxed-chaplygin-mc6238) that takes a few online JSON sources and runs them through the `gzip` [`CompressionStream`](https://developer.mozilla.org/en-US/docs/Web/API/CompressionStream) transform described in the solution above.

{% embed https://codesandbox.io/embed/relaxed-chaplygin-mc6238 %}

_Oh, and yes. ❤️ VueJS._

## Conclusion

Firstly, I had a blast writing this piece and exploring the [Compression Streams API](https://developer.mozilla.org/en-US/docs/Web/API/Compression_Streams_API). It's simple to use and feels like a really nice addition to the Javascript ecosystem.

In terms of how useful the API itself is, it has __solved a real problem__ for me in being able to reduce my persistent data footprint.

I have _already_ built this into a small feature in my app, but I do plan to integrate it deeper and bake it into the core functionality soon. Which should be another interesting problem to solve as the app has integrated client-side encryption using [Age](https://github.com/FiloSottile/age) ([rage](https://github.com/str4d/rage) ([rage-wasm](https://github.com/kanru/rage-wasm))). But that's for another day...

It's a little disappointing that this API isn't fully supported yet, at only 72.53% support coverage, according to [CanIUse](https://caniuse.com/mdn-api_compressionstream_compressionstream_gzip). But I still think it's very worthwhile. We can implement this to fail gracefully where the [Compression Streams API](https://developer.mozilla.org/en-US/docs/Web/API/Compression_Streams_API) is not supported and forgo the performance boost, so that's not a deal breaker for me.

I'm not sure how often the [Compression Streams API](https://developer.mozilla.org/en-US/docs/Web/API/Compression_Streams_API) will be a tool I reach for, but I'm glad to have learnt it and I'm sure this won't be the last time I use it, or at the very least consider it.