How to handle binary data in Node.js?
Tags: node, javascript
Published: 2021-12-04 19:56:52

---
title: How to handle binary data in Node.js?
published: true
date: 2021-11-30 00:00:00 UTC
tags: node, javascript
canonical_url: https://www.mariokandut.com/how-to-handle-binary-data-in-node-js/
---

Handling binary data in server-side programming is an essential feature, and a must-have for every developer dealing with Node.js. In Node.js binary data is handled with the `Buffer` constructor. Let's have a look at the anatomy of a Buffer instance.

## [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#the-buffer-instance-in-nodejs)The Buffer instance in Node.js

The Buffer constructor is a `global`, hence, no import is needed to use it.

Type `node -p "Buffer"` in your terminal and have a look at the output.

```
[Function: Buffer] {
  poolSize: 8192,
  from: [Function: from],
  of: [Function: of],
  alloc: [Function: alloc],
  allocUnsafe: [Function: allocUnsafe],
  allocUnsafeSlow: [Function: allocUnsafeSlow],
  isBuffer: [Function: isBuffer],
  compare: [Function: compare],
  isEncoding: [Function: isEncoding],
  concat: [Function: concat],
  byteLength: [Function: byteLength],
  [Symbol(kIsEncodingSymbol)]: [Function: isEncoding]
}
```

The `Buffer` constructor was introduced into Node.js, when Javascript did not have a native binary type. Javascript evolved and different views of a buffer got added to the language, like an `ArrayBuffer` or other typed arrays.

For instance, an `ArrayBuffer` instance can be accessed by a `Float64array`, where each set of 8 bytes is interpreted as a 64-bit floating-point number. Have a look at the MDN article [Javascript Typed Arrays](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Typed_arrays). When these new data structures where added, the Buffer constructor internals were refactored on top of the `Uint8array` typed array. This means a buffer object is both, **an instance of a Buffer** and **an instance of Uint8array**.

Let's open the REPL and double-check this.

```
# enter REPL
node

## Allocates a new Buffer of size bytes.
const buffer = Buffer.alloc(10)

buffer instanceof Buffer
## returns true

buffer instanceof Uint8Array
## returns true
```

**Important:** The method `Buffer.prototpye.slice` overrides the `Uint8Array.prototype.slice` method. The Uint8Array method will take a copy of a buffer, the Buffer method will return a buffer instance that references the binary data.

### [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#allocating-buffers)Allocating Buffers

Usually a constructor is called with the `new` keyword, with the `Buffer` constructor this is deprecated. The correct and safe way to allocate a buffer of a certain amount of bytes is to use `Buffer.allocate`, like:

```
const buffer = Buffer.allocate(10);
```

The `Buffer.alloc` function produces a zero-filled buffer by default. Let's use the dynamic evaluation to see the output directly.

```
node -p "Buffer.alloc(10)"
## the output should be <Buffer 00 00 00 00 00 00 00 00 00 00>
```

When a buffer is printed to the terminal, the ellipsis in `<Buffer ...>` are **hexadecimal numbers**. For example a single byte buffer with a decimal value of 100, is 1100100 in binary and 64 in hexadecimal. Hence, the output would be `<Buffer 64>`

There is also an **unsafe way** to allocate buffers.

```
const buffer = Buffer.allocUnsafe(10);
```

Any times a buffer is created, it's allocated (or assigned) from unallocated (or unassigned) memory. Unassigned memory is only unlinked, it is never wiped. This implies, that unless the buffer is overwritten, (zero-filled), it can contain fragments of previously deleted data. This poses a **security risk**. The method `allocUnsafe` is used only for advanced use cases, like performance optimization. If you have to create a buffer, only use the safe method `Buffer.alloc`.

### [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#converting-strings-to-buffers)Converting Strings to Buffers

The String primitive in JavaScript is a frequently used data structure.

A buffer can also be created from a string by using `Buffer.from`. The string characters are converted to by values.

```
const buffer = Buffer.from('Hello World');
```

Let's dynamically evaluate this.

```
node -p "Buffer.from('Hello World')"
```

The output is `<Buffer 48 65 6c 6c 6f 20 57 6f 72 6c 64>`.

In order to convert a string to a binary representation, an encoding has to be defined. The default encoding for `Buffer.from` is **UTF8**. The UTF8 encoding may have up to four bytes per character, so string length will not always match the converted buffer size. Especially, when dealing with emojis.

```
node -p "'🔥'.length"
## will return 2

node -p "Buffer.from('🔥').length"
## will return 4
```

When the first argument passed to `Buffer.from` is a string, a second argument can be passed to set the encoding. Two types of encodings are available in this context: character encodings and binary-to-text encodings. UTF8 is one character encoding, UTF16LE is another one. Different encodings result in different buffer sizes.

### [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#converting-buffers-to-strings)Converting Buffers to Strings

To convert a buffer to a string, call the `toString` method on a `Buffer` instance. Let's try it out, use the **REPL** or create a file and run it with node.

```
const buffer = Buffer.from('hello world');
console.log(buffer); // prints <Buffer 68 65 6c 6c 6f 20 77 6f 72 6c 64>
console.log(buffer.toString()); // prints 'hello world'
```

The `toString` method also accepts an encoding argument.

```
const buffer = Buffer.from('mario');
console.log(buffer); // prints <Buffer 6d 61 72 69 6f>
console.log(buffer.toString('hex')); // prints '6d6172696f'
```

### [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#json-serializing-and-deserializing-buffers)JSON Serializing and Deserializing Buffers

A very common format for serialization is **JSON** , especially when working with JavaScript-based applications. When `JSON.stringify` encounters any object, it attempts to call the `toJSON` method on that object, if it exists.`Buffer` instances have a `toJSON` method, which returns a plain JavaScript object.

```
node -p "Buffer.from('hello world').toJSON()"
```

Calling `toJSON` on the above Buffer instance returns the following JSON.

```
{
  "type": "Buffer",
  "data": [104, 101, 108, 108, 111, 32, 119, 111, 114, 108, 100]
}
```

Buffer instances are represented in JSON by an object, that has a type property with a string value of `Buffer` and a data property with an array of numbers, representing the value of each byte.

## [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#tldr)TL;DR

- If you have to create a buffer, only use the safe method `Buffer.alloc`.
- The `Buffer.alloc` function produces a zero-filled buffer by default.
- The `Buffer` constructor does not need the `new` keyword.
- There is an unsafe way to allocate buffer `allocUnsafe()`, this poses a **security risk** , though there are some advanced use cases.

Thanks for reading and if you have any **questions** , use the **comment** function or **send me a message** **[@mariokandut](https://www.twitter.com/mariokandut)**.

If you want to know more about **[Node](https://www.mariokandut.com/tags/node)**, have a look at these [Node Tutorials](https://www.mariokandut.com/tags/node).

**References (and Big thanks):**

[JSNAD - Using Buffers](https://training.linuxfoundation.org/training/nodejs-application-development-lfw211/)[Node.js - Buffer](https://nodejs.org/docs/latest-v14.x/api/buffer.html),[MDN - Javascript Typed Arrays](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Typed_arrays)