How to connect streams with pipe?
Tags: node, javascript
Published: 2021-08-11 11:00:50

---
title: How to connect streams with pipe?
published: true
date: 2021-08-08 00:00:00 UTC
tags: node, javascript
canonical_url: https://www.mariokandut.com/how-to-connect-streams-with-pipe-in-node-js/
---

Streams are a built-in feature in Node.js and represent asynchronous flow of data. Streams are also a way to handle reading and/or writing files. A Node.js stream can help process large files, larger than the free memory of your computer, since it processes the data in small chunks.

## [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#streams-in-nodejs)Streams in Node.js

This is the second article of a series about streams in Node.js. It explains what pipe does in Node.js, and how to connect streams using pipe.

**Streams in Node.js**

- [What is a Stream in Node.js?](https://www.mariokandut.com/what-is-a-node-js-stream/)
- Connect streams with the pipe method (this article)
- Handle stream errors (planned)
- Connect streams with the pipeline method (planned)

### [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#connect-streams-with-pipe)Connect streams with pipe

The recommended way to consume streams is the `pipe` and `pipeline` methods, which consume streams and handle the underlying events for you. To connect streams together and start the flow of data, the `pipe` method is available on `readable` streams. It is also possible to listen to stream events, but it is not recommended for consuming data. The main goal of `pipe` is to limit the buffering of data so that sources and destinations will not overwhelm the available memory.

The `pipe` method uses, under the hood, the events emitted by streams and abstracts away the need to handle these events. The only exception is, that the **handling of error events** is not included in the abstraction and has to be done separate. Unhandled stream errors can crash your application.

The `pipe` method is available on streams, which implement a `Readable` interface. Check out the article [What is a Stream in Node.js?](https://www.mariokandut.com/what-is-a-node-js-stream/) for the different types of streams.

The streams **Readable** , **Duplex** , **Transform** and **PassThrough** implement a `Readable` interface. The method accepts a destination to pipe data to. The destination stream must implement a `Writable` interface. The streams **Writable** , **Duplex** , **Transform** and **PassThrough** implement a `Writable` interface.

Let's look at an example. Node has a globally available readable stream `process.stdin` (stdin stands for standard in), and a writable stream `process.stdout` (stdout stands for standard out).

Create a file (or use the REPL).

```
touch stream-it.js
```

Add the following code into it.

```
process.stdin.pipe(process.stdout);
```

Then run it in the CLI `node stream-it.js` and type `Banana` and hit the enter key. You will see that `Banana` is echoed back to you.

Let me explain what is happening. The `process.stdin` is the readable source of data, and the `process.stdout` is the writable destination. When you input a text, the text is piped from `stdin` to `stdout`, creating the echo. Calling `pipe` will return the destination stream.

With the `pipe` method it is possible to chain multiple streams together. The requirement for this is that the destination stream is both readable and writable, like `Duplex`, `Transform` and `PassThrough`.

```
const { PassThrough } = require('stream');

const passThrough = new PassThrough();

process.stdin.pipe(passThrough).pipe(process.stdout);
```

### [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#using-the-fs-module-to-create-streams-from-files)Using the fs module to create streams from files

Implementing streaming interfaces and consuming streams have quite a few differences. Creating streams is not as common as consuming streams, but there are some instances where creating your own stream is useful. The most common use cases is streaming data from and to a file using the `fs` module.

The `fs` module is able to create read and writeable streams using the helper methods `fs.createReadStream` and `fs.createWriteStream`. The method `createWriteStream` takes a file path as the first argument, and then optional config arguments.

Let's dive into code and create a simple stream that writes text from `stdin` to a file called `output.txt`.

Create a file.

```
touch stream-to-file.js
```

Add code.

```
const fs = require('fs');

const outputStream = fs.createWriteStream('output.txt');

process.stdin.pipe(outputStream);
```

Run the code in the CLI with `node stream-to-file.js` and type `Hello Stream` and hit the enter key. Then log the `output.txt` to the console with `cat output.txt` or open the file in a text editor. You will see that `Hello Stream` was written to the file. In this example, we have replaced the `stdout` with the variable `outputStream` which holds the stream created with `fs.createWriteStream`.

Since there is some data now in the `output.txt` file, let's invert this and create a readable stream with piping the data from `output.txt`.

Create a file.

```
touch stream-out.js
```

Add code.

```
const fs = require('fs');

const inputFileStream = fs.createReadStream('output.txt');

inputFileStream.pipe(process.stdout);
```

Run the file with `node stream-out.js` and you will see the text from the `output.txt` file written in the terminal.

When creating a writeable stream from a file, the file will be overwritten by default. This behaviour can be changed with adding a flag when creating the stream. Read more about [file system flags](https://nodejs.org/api/fs.html#fs_file_system_flags). So we can pass `{flags: 'a'}` for _Open file for appending. The file is created if it does not exist._

```
const fs = require('fs');

const outputStream = fs.createWriteStream('output.txt', {
  flags: 'a',
});

process.stdin.pipe(outputStream);
```

This will append data to the file if it exists already, or otherwise create the file.

## [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#tldr)TL;DR

- The recommended way to consume streams is the `pipe` and `pipeline` method.
- The main goal of `pipe` is to limit the buffering of data so memory will not be overloaded.
- The `pipe` method is available on streams, which implement a `Readable` interface.
- With the help of `pipe` streams can be chained.
- The `fs` module can create readable and writeable streams.

Thanks for reading and if you have any **questions** , use the **comment** function or **send me a message** **[@mariokandut](https://www.twitter.com/mariokandut)**.

If you want to know more about **[Node](https://www.mariokandut.com/tags/node)**, have a look at these [Node Tutorials](https://www.mariokandut.com/tags/node).

**References (and Big thanks):**

[HeyNode](https://heynode.com/tutorial/what-stream/),[Node.js - Streams](https://nodejs.org/api/stream.html#stream_stream),[MDN - Streams](https://developer.mozilla.org/en-US/docs/Web/API/Streams_API)[Node.js - fs](https://nodejs.org/api/fs.html#fs_file_system_flags)