How to stream big JSON files with low-memory footprint in nodeJS
Tags: node
Published: 2023-11-08 21:37:29

---
title: How to stream big JSON files with low-memory footprint in nodeJS
published: true
date: 2023-08-17 00:00:00 UTC
tags: ["nodejs"]
canonical_url: https://blog.lepape.me/how-to-stream-big-json-files-with-low-memory-footprint-in-node-js
---

For production-grade applications, actions like reading a big file in memory is not ideal as they might cause OOM crash (Out Of Memory), and require a lot of resources for sometimes a really short duration. It would be a waste of resources and money to have a big server to only handle rare cases, and run at 5% of the capacity the rest of the time.

![Example OOM crash](https://lepape.me/assets/static/big-json-resources-v2.fddb1dc.f5c23ca968c0048a8244865319c71483.png)

__Legend:__ Example of a container with average memory consumption between 50 mb to 150 mb, when processing files it spikes to 4GB of RAM, leading to OOM crash. 

Most of the time, especially with applications running on Kubernetes (like the one from the screenshot), a best practice is to stream data instead of loading it in memory. Here is how you can do it using Stream Transformers, a less-known Node.js feature.

## Streaming big objects with nodejs

### Generate data

Generate dummy data with this one-liner to paste in a terminal, to simulate a big JSON file.

```bash
node << EOF
require('node:fs').writeFile('dummy.json', JSON.stringify(
  new Array(process.argv[2] || 50).fill({ id: 0, name: 'dummy' })
), _ => console.log('done'))
EOF
```

## JSON stream NPM Module

To help with processing JSON, we can use a tiny library like `stream-json`.

As written on the GitHub README:

> _stream-json is a micro-library of node.js stream components with minimal dependencies for creating custom data processors oriented on processing huge JSON files while requiring a minimal memory footprint. It can parse JSON files far exceeding available memory. Even individual primitive data items (keys, strings, and numbers) can be streamed piece-wise._

Install the json stream module:

```bash
npm install stream-json
```

## Code example

```js
const { pipeline, Transform } = require("node:stream");
const fs = require("node:fs");

const { streamArray } = require("stream-json/streamers/streamArray");
const Batch = require("stream-json/utils/Batch");
const { parser } = require("stream-json");

const logMemory = () =>
  console.log(
    `Memory usage: ${
      Math.round((process.memoryUsage().heapUsed / 1024 / 1024) * 100) / 100
    } MB`
  );

const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));

const customTransformer = new Transform({
  objectMode: true, // to accept arrays
  transform: async (chunk, encoding, cb) => {
    logMemory();
    console.log("chunk length: ", chunk.length);
    await sleep(500);
    cb();
  },
});

pipeline(
  // read from file
  fs.createReadStream("dummy.json"),
  // parse JSON
  parser(),
  // takes an array of objects and produces a stream of its components
  streamArray(),
  // batches items into arrays
  new Batch({ batchSize: 2 }),
  customTransformer,
  (err) => {
    if (err) {
      console.error("Pipeline failed", err);
    } else {
      console.log("Pipeline succeeded");
    }
  }
);
```

and it should give an output similar to:

```bash
➜  node-stream-demo node index.js
Memory usage: 4.6 MB
chunk length:  2
Memory usage: 4.64 MB
chunk length:  2
[...]
chunk length:  2
Memory usage: 4.32 MB
chunk length:  2
Memory usage: 4.33 MB
chunk length:  2
Memory usage: 4.33 MB
chunk length:  2
Pipeline succeeded
```

The 2 interesting parts are in the `pipeline()` function:

- `new Batch({ batchSize: 2 })` grouping data 2 by 2, very convenient to process data in batches.

- `customTransformer` to do async operations on the batches, like inserting data in a database by X.

Hope the example helps and provides ideas on how to use Node Stream Transformers!