Node.js - Streams and Promises
Tags: javascript, programming, node, tutorial
Published: 2022-04-03 13:53:56

I have been working on a project that requires reading large `.csv` files from the local file system and then working with the data.  Node.js has some great tools for working with this, namely streams, event emitters, the [`readline`](https://nodejs.org/docs/latest-v16.x/api/readline.html) native modules.  However, all of the example code/tutorials fell into one of three categories:
- print the data to the console (not useful)
- write the data to a file
- push the incoming data to an outside array
- use an external library

I started with using the external library [csv-parser](https://www.npmjs.com/package/csv-parser).  However, since it is basically a wrapper around the base Node.js technologies I listed above I has the same problems working with my data that I will list below.  I eventually uninstalled it and wrote my own light-weight version.

## Background
<blockquote cite="https://nodejs.org/docs/latest-v16.x/api/readline.html">
  The `readline` module provides an interface for reading data from a Readable stream...one line at a time.

  from [Node.js Documentaion](https://nodejs.org/docs/latest-v16.x/api/readline.html)
</blockquote>
<blockquote cite="https://nodejs.org/docs/latest-v16.x/api/stream.html">
  All streams are instances of EventEmitter.

  from [Node.js Documentaion](https://nodejs.org/docs/latest-v16.x/api/stream.html)
</blockquote>

Basically working with streams means listening for events with your data.  And since the `.on` method of an `EventEmitter` expects a callback, everything you want to do next needs to happen in that callback.  The `readline` module gives you the `line` event to listen for.

## Solution #1
At first I tried the "push the incoming data to an outside array" approach.
```javascript
const incomingData = [];

rl.on('line', data => [
  incomingData.push(data);
])
  .on('close', () => {
    // do something with incomingData
  });
```
This solution does actually work if you are only reading one file.  Unfortunately, I need to loop through a directory of files and read each one, and **then** do something with the data.  I tired all sorts of things with counters and what not, but kept running into race conditions with the loops and what needed to happen next.  So not really a solution for me.

## Solution #2
This solution actually came from a member of my local code mentoring [meetup](https://www.meetup.com/Austin-Code-Mentorship/). This solution uses Promises.

First, I created a JavaScript `class` for my various `.csv` needs.
```javascript
const fs = require('fs');
const readline = require('readline');
const path = require('path');

class CSVHelpers {
  constructor () {
    super();
  }

  /**
   * @param  {string} filePath
   * @return {promise} Array of row objects. Key: header, value: field value
   */
  read (filePath) {
    return new Promise ((resolve, reject) => {
      try {
        const reader = this._createReadStream(filePath);
        let rows = [];
        let headers = null;

        reader.on('line', row => {
          if (headers === null) {
            headers = row.split(',');
          } else {
            const rowArray = row.split(',');
            const rowObject = {};
            rowArray.forEach((item, index) => {
              rowObject[headers[index]] = item;
            });

            rows.push(rowObject);
          }
        })
          .on('close', () => {
            resolve({
              rows,
              file: filePath
            });
          });
      } catch (error) {
        reject(error);
      }
    });
  }

  /**
   * @param  {type} filePath
   * @return {type} Readline event emitter
   */
  _createReadStream (filePath) {
    const fd = fs.openSync(path.resolve(filePath));
    const fileStream = fs.createReadStream(path.resolve(filePath), {fd});
    return readline.createInterface({
      input: fileStream
    });
  }
}

module.exports = CSVHelpers;
```
Then in my code:
```javascript
const csv = new CSVHelpers();
const dataFiles = fs.readdirSync(<pathToDirectory);

const filePromises = dataFiles.map(file => {
  return csv.read(<pathToFile>);
});

Promise.all(filePromises)
  .then(values => {
    // do something with the values.
  });
```
This `Promise` approach means I don't need to trying to next loops or callbacks.

## Conclusion
I do not know if this is the best solution, but it works for my use case, and solves the race conditions I was having.  If you have better ways to solve the problem, please let me know.