Reading and Parsing Large Files in JavaScript the Right Way
Tags: programming, typescript, performance
Published: 2023-05-29 21:39:56

In the previous article in the series, we had a look at Generator Functions that let you generate values in a way that can be consumed by a `for...of` loop. We used this to create a sequence of natural numbers without having to allocate the entire number set in memory, lazily producing each value when the consumer requests one.

In the same vein, [Streams][streams] allow for incremental reading of a large chunk of data (or files!). Instead of forcing us to allocate memory for the data, the decoded content and any subsequent results, it allows us to process data as a series of steps on individual slices.

## A Gentle Introduction to Streams

Streams come in three variations:

- [`ReadableStream`][readable]: a stream that can be read from,
- [`WritableStream`][writable]: a stream that can be written to,
- [`TransformStream`][transform]: a stream that can take the output of a readable or transform stream and output a transformed result. It is both readable and writable.

The one that will be useful most of the time is `ReadableStream`. This is also the stream type that is provided by the [`File`][file] and [`Response`][response] objects as an alternative to the more common [`text()`][response#text] and [`json()`][response#json] methods that returns a decoded string asynchronously.

As an example, we'll build an application that will can count the number of lines of text for a file dropped into the web page. First, let's look at the core of the problem — how do we read a file using streams?

### Consuming a `ReadableStream`

Let's start with a function that takes a [`File`][file] object and reads it from start to end, counting the occurence of line feeds (empty lines also count as a line) and returns the final count:

```ts
const countLines = async (file: File): Promise<number> => {
  let count = 1;
  const stream = file.stream().pipeThrough(new TextDecoderStream());
  const reader = stream.getReader();

  try {
    while (true) {
      const { done, value } = await reader.read();
      if (done) break;
      const lineBreaks = value.match(/\n/g)?.length ?? 0;
      count += lineBreaks;
    }
  } finally {
    reader.releaseLock();
  }

  return count;
};
```

The first thing to note is that files are 8-bit binary streams (`Uint8Array`) which must first be decoded into UTF-16 strings that JavaScript for string representation. To do this, we pipe the file's `ReadableStream<Uint8Array>` through a transform stream called `TextDecoderStream` which converts the byte stream into a stream of strings that we can consume using the reader (an instance of [`ReadableStreamDefaultReader`][defaultreader]).

A reader provides the `read()` method which returns a `Promise` that resolves to the next chunk in the stream. We can use an infinite loop to read the next chunk and check if the stream has been exhausted by testing the `done` property on the resolved value. If `true`, we simply break the loop.

Now that we have a function that can count lines in files, let's build a quick web application that we can demonstrate its functionality. I've decided to use [Vite][vite], [Preact][preact] and [`@krofdrakula/drop`][@krofdrakula/drop] to build the web application shell that uses the above function to display line counts for any file (or multiple files) dropped into the page:

{% embed https://stackblitz.com/edit/vitejs-vite-ndca3e?embed=1&file=src%2Fcount_lines.ts&view=preview %}

Since the files are read locally and not sent over the wire, you can drop any size files (even hundreds of MB!) and the browser will handle it just fine. It doesn't even have to be text, but the line count will not really make much sense in that case.

### Creating Your Own `ReadableStream`

At this point you may be wondering how one can one create a `ReadableStream` in the first place. A `ReadableStream` is created using the [`ReadableStream` constructor][readablestream#constructor] which takes a configuration object with the following optional methods:

- `pull(controller)`: whenever the stream is read, this method is called by the reader in order to produce the next chunk of data. The `controller` object is used to send values to the consumer of the stream.
- `start(controller)`: whenever the stream is constructed, this method is called and can optionally send values via the `controller` object
- `cancel(reason?)`: called when the stream's `cancel()` method is called with an optional reason. Used to clean up any open handles to resources or similar.

Let's take the example of natural numbers from the previous article and write a `ReadableStream` that emits all natural numbers up to the given maximum:

```ts
const createNaturalNumberStream = (max: number = Infinity) => {
  let i = 1;
  return new ReadableStream<number>({
    pull(controller) {
      if (i <= max) {
        controller.enqueue(i);
        i++;
      } else {
        controller.close();
      }
    },
  });
};
```

Reading this stream will keep emitting numbers in sequence up until, and including, `max`.

### Transforming Streams Using `TransformStream`

As previously mentioned, [`TextDecoderStream`][textdecoder] takes a stream of bytes and converts them to a stream of UTF-16 strings suitable for JavaScript. Streams do not have to provide data in byte arrays or strings, however; they can emit any kind of JavaScript value.

Constructing a `TransformStream` is achieved similarly to `ReadableStream` but [has a different set of optional methods][transformstream#constructor]:

- `transform(chunk, controller)`: called whenever a chunk is read from the transform stream. Use `controller` to emit values to the reader.
- `start(controller)`: called when the transform stream is constructed.
- `flush(controller)`: called when the source has been consumed and is about to be closed.

To demonstrate the API, let's create a mapping stream that will take a value from the input stream and apply a mapping function on the emitted value. The only method we need to implement is the `transform` function that takes a chunk and enqueues the mapped value:

```ts
const createMapStream = <I, O>(
  mapper: (value: I) => O
): TransformStream<I, O> =>
  new TransformStream({
    transform(value, controller) {
      controller.enqueue(mapper(value));
    },
  });
```

Using this transformer is simply a matter of piping any compatible readable stream through it:

```ts
const evenNumberStream = createNaturalNumberStream().pipeThrough(
  createMapStream((n) => n * 2)
);
```

## Other Streams

- [`CompressionStream`][compressionstream]: compresses the content of a `Uint8Array` stream into a new `Uint8Array` stream
- [`DecompressionStream`][decompressionstream]: does the opposite of the above
- [`FileSystemWritableFileStream`][filesystemwritable]: allows writing to an open file system file handle
- [`WebSocketStream`][websocketstream]: an open standard proposal to allow writing to a web socket as a `WritableStream`

## Converting `ReadableStream`s into Asynchronous Iterables

Admittedly, streams are not as straightforward to consume as a `for...of` loop, but there is good and bad news.

The good news is that the `ReadableStream` specification implements `Symbol.asyncIterator` that allows you to read a `ReadableStream` using a `for await...of` loop, e.g.:

```ts
for await (const number of evenNumberStream) {
  console.log(number);
}
```

The bad news is that it [currently isn't supported in all JavaScript engines][caniuse#streamiterable]. However, what we _can_ do is leverage what we learned in the previous article regarding generator functions and create a utility function that takes a stream and creates an `AsyncGenerator` yielding values from the input stream:

```ts
const streamToIterable = async function* <T>(
  stream: ReadableStream<T>
): AsyncGenerator<T> {
  const reader = stream.getReader();
  try {
    while (true) {
      const { done, value } = await reader.read();
      if (done) return;
      yield value;
    }
  } finally {
    reader.releaseLock();
  }
};
```

This makes it possible to consume any stream as an async iterable:

```ts
for await (const number of streamToIterable(evenNumberStream))
  console.log(number);
```

## Applied Streamology

![](https://media.giphy.com/media/z6ccg9ZZzWT2E/giphy.gif)

In the next installment, we'll have a look at applying our knowledge of streams and generators to read large files in the browser while keeping memory consumption low and the UI responsive while data is being processed.

[streams]: https://developer.mozilla.org/en-US/docs/Web/API/Streams_API
[writable]: https://developer.mozilla.org/en-US/docs/Web/API/WritableStream
[readable]: https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream
[transform]: https://developer.mozilla.org/en-US/docs/Web/API/TransformStream
[file]: https://developer.mozilla.org/en-US/docs/Web/API/File
[response]: https://developer.mozilla.org/en-US/docs/Web/API/Response
[response#text]: https://developer.mozilla.org/en-US/docs/Web/API/Response/text
[response#json]: https://developer.mozilla.org/en-US/docs/Web/API/Response/json
[defaultreader]: https://developer.mozilla.org/en-US/docs/Web/API/ReadableStreamDefaultReader
[vite]: https://vitejs.dev/
[preact]: https://preactjs.com
[stackblitz]: https://stackblitz.com
[dataset]: https://catalog.data.gov/dataset/popular-baby-names "Popular baby names dataset"
[@krofdrakula/drop]: https://dev.to/krofdrakula/want-to-be-able-to-drag-files-into-a-web-app-got-ya-covered-jh3
[textdecoder]: https://developer.mozilla.org/en-US/docs/Web/API/TextDecoderStream
[readablestream#constructor]: https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream/ReadableStream
[transformstream#constructor]: https://developer.mozilla.org/en-US/docs/Web/API/TransformStream/TransformStream
[readablestream#asynciter]: https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream#async_iteration
[caniuse#streamiterable]: https://caniuse.com/mdn-api_readablestream_--asynciterator
[compressionstream]: https://developer.mozilla.org/en-US/docs/Web/API/CompressionStream
[decompressionstream]: https://developer.mozilla.org/en-US/docs/Web/API/DecompressionStream
[filesystemwritable]: https://developer.mozilla.org/en-US/docs/Web/API/FileSystemWritableFileStream
[websocketstream]: https://developer.chrome.com/articles/websocketstream/