ETL: Extract Data with Node.js
Tags: node, javascript, express, datascience
Published: 2021-07-28 12:38:34

---
title: ETL: Extract Data with Node.js
published: true
date: 2021-04-03 00:00:00 UTC
tags: node, javascript, express, datascience
canonical_url: https://www.mariokandut.com/extract-data-etl-pipeline-node-js/
---

ETL is a process of extracting, transforming, and loading data from one or multiple sources into a destination. Have a look at the article [ETL pipeline explained](https://www.mariokandut.com/what-is-the-node-js-etl-pipeline/) for a general overview on ETL pipelines.

This is the first article of a series about ETL pipelines.

- Extract (this article)
- Transform
- Load

## [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#extract-data-in-etl-pipeline)Extract Data in ETL pipeline

The first step in an ETL pipeline is to **Extract** the data, which we are going to **Transform** and **Load** in future steps. In the extract phase the decision is, which data sources to extract from and how exactly (API, authorization, DB, ...).

In the example, the [jsonplaceholder.typicode.com](https://jsonplaceholder.typicode.com) will be used. It is a REST API, has a lot of options and is free for testing and prototyping. We are going to create two functions. One to extract the data, and the other one to orchestrate the different stages of the ETL pipeline.

In a real world example the data source would likely be a database, but for this example the placeholder API is fine. Since the same approach has to be followed, only the interface (directly connecting and querying or through middleware) to the database would be different.

Important in the **Extract** phase is:

- Select the data source
- Decide what data to extract
- How to Query or retrieve the data (what method that is available, depends on the data source)

Let's start with creating the basics and then go through the steps:

- 
  1. Query the data source

Create or add a project folder.

```
mkdir node-etl
```

Initialize project with `npm init -y` to be able to install node packages.

```
cd node-etl
npm init -y
```

Install `node-fetch`.

```
npm install node-fetch
```

### [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#1-query-the-data-source)1. Query the data source

Create an `extract.js` file, for handling querying the data.

```
touch extract.js
```

If you want to learn more about making API requests, check out [How to make an API request in Node](https://www.mariokandut.com//how-to-make-an-API-request-in-Node-javascript/).

Add test code. We are going to query the photos for a specific album.

```
const fetch = require('node-fetch');

const URL = 'https://jsonplaceholder.typicode.com/albums';

async function getPhotos(albumId) {
  return await fetch(`${URL}/${albumId}/photos`).then(response =>
    response.json(),
  );
}

module.exports = { getPhotos };
```

The `getPhotos()` async function will take an `albumId` as an argument, will make a `GET` request to the fake API and return an array where each object has the following interface:

```
interface Photo {
  albumId: number;
  id: number;
  title: string;
  url: string;
  thumbnailUrl: string;
}
```

For the example, we will skip the validation of the input argument for albumId, which should not exceed a 100, or an empty array will be returned. We are exporting the function `getPhotos` so we can import it in the `index.js`, which will orchestrate the ETL pipeline. Have a look at the [How to organize Node](https://www.mariokandut.com/how-to-organize-code-into-modules-in-node-javascript/) article for maintainable Node.js code.

### [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#2-setup-the-pipeline)2. Setup the pipeline

Create an `index.js` file, which will be the entry point of the application and used to orchestrate the ETL pipeline.

```
touch index.js
```

We are going to create an `orchestrateEtlPipeline()` function, which will coordinate each step in the ETL pipeline: extracting data, transforming it and loading it into its destination.

```
const { getPhotos } = require('./extract');

const orchestrateEtlPipeline = async () => {
  try {
    const photosAlbum = await getPhotos(1);
    console.log(photosAlbum);

    // TODO - TRANSFORM

    // TODO - LOAD
  } catch (error) {
    console.error(error);
  }
};

orchestrateEtlPipeline();
```

The `orchestrateEtlPipeline()` function uses `async/await` again in a `try/catch` block to handle any errors or promise rejections.

### [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#3-wait-for-multiple-requests)3. Wait for multiple requests

In the example code, we are only making one request, but what if we want to make more requests to the same data source or extract data from multiple sources. With the method `Promise.all` we can wait for all extracted data before proceeding. However, this method fires all requests at once, which can overwhelm some sources (think of multiple intensive requests to the DB). Read more about [Multiple Promises in Node.js](https://www.mariokandut.com/how-to-wait-for-multiple-promises-in-node-javascript/).

An example for requesting photos from multiple albums, would look something like this:

```
const { getPhotos } = require('./extract');

const orchestrateEtlPipeline = async () => {
  try {
    const allPhotos = Promise.all([
      getPhotos(1),
      getPhotos(2),
      getPhotos(3),
    ]);
    const [photos1, photos2, photos3] = await allPhotos;

    console.log(photos1[0], photos2[0], photos3[0]); // to log the first photo object of all three albums

    // TODO - TRANSFORM

    // TODO - LOAD
  } catch (error) {
    console.error(error);
  }
};

orchestrateEtlPipeline();
```

Once the data is extracted, the next step in the ETL pipeline is to transform it.

## [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#tldr)TL;DR

- First step in an ETL pipeline is to **extract** the data from the data source.
- Important in this phase is to Select the data source, to decide what data to extract and how to query or retrieve the data.

Thanks for reading and if you have any **questions** , use the **comment** function or **send me a message** **[@mariokandut](https://www.twitter.com/mariokandut)**.

If you want to know more about **[Node](https://www.mariokandut.com/tags/node)**, have a look at these [Node Tutorials](https://www.mariokandut.com/tags/node).

**References (and Big thanks):**

[HeyNode](https://heynode.com/tutorial/extract-data-api-requests/),[MDN async/await](https://developer.mozilla.org/en-US/docs/Learn/JavaScript/Asynchronous/Async_await)