Node.js Tips — Streams, Scraping, and Promisifying Functions
Tags: node, webdev, programming, javascript
Published: 2021-01-26 03:03:41

**Check out my books on Amazon at https://www.amazon.com/John-Au-Yeung/e/B08FT5NT62**

**Subscribe to my email list now at http://jauyeung.net/subscribe/**


Like any kind of apps, there are difficult issues to solve when we write Node apps.

In this article, we’ll look at some solutions to common problems when writing Node apps.

### Converting a Buffer into a ReadableStream in Node.js

We can convert a buffer to a readable stream with the `ReadableStreamBuffer` constructor.

For instance, we can write:

```
const { ReadableStreamBuffer } = require('stream-buffers');
const readableStreamBuffer = new ReadableStreamBuffer({
  frequency: 10,
  chunkSize: 2048
});

readableStreamBuffer.put(aBuffer);
```

where `aBuffer` is the buffer we want to convert.

`frequency` is the frequency in which the chunks are pumped out.

The size can be changed in the constructor with the `chunkSize` property.

### Scrape Web Pages in Real-Time with Node.js

We can use the cheerio library to scrape web pages in real-time.

To install it, we run:

```
npm install cheerio
```

Then we can use it by writing:

```
const cheerio = require('cheerio');
const $ = cheerio.load('<h1 class="title">Hello world</h1>');

$('h1.title').text('Hello James');
$('h1').addClass('welcome');

$.html();
```

We can get HTML’s content and manipulate it as we do with jQuery.

We can combine this with an HTTP client like Axios to get the HTML.

Then we can use Cheerio to parse its content.

For instance, we can write:

```
const axios = require('axios');
const cheerio = require('cheerio');

axios.get('https://example.com')
.then(({ data }) => {
  const $ = cheerio.load(data);
  const text = $('h1').text();
  console.log(text);
})
```

We make a GET request with Axios to a website.

Then we use cheerio to parse the data with `cheerio.load` .

Then we get the content of `h1` with the `text` method.

Any selector can be used to get data.

### Generate an MD5 file Hash in JavaScript

We can generate an MD5 hash with the crypto-js package.

To install it, we can run:

```
npm install crypto-js
```

Then we can write:

```
import MD5 from "crypto-js/md5";
const md5Hash = MD5("hello world");
```

to generate the hash.

### Server-Side Browser Detection with Node.js

We can get the `user-agent` header to get the user agent string from the request.

To parse the string, we can use the ua-parser-js package.

To install it, we run:

```
npm install ua-parser-js
```

In our Express app, we can create our own middleware to check the user agent:

```
const UAParser = require('ua-parser-js');

const checkBrowser = (req, res, next) => {
  const parser = new UAParser();
  const ua = req.headers['user-agent'];
  const browserName = parser.setUA(ua).getBrowser().name;
  const fullBrowserVersion = parser.setUA(ua).getBrowser().version;

  console.log(browserName);
  console.log(fullBrowserVersion);
  next();
}

app.all(/*/, checkBrowser);
```

We get the `user-agent` header with `req.headers[‘user-agent’]` .

Then we use the `UAParser` constructor to parse the user agent string.

We call `getBrowser` to get the browser data.

`name` has the browser name and `version` has the version.

### Best Way to Store Database Config in an Express App

We can store the config in a configuration file.

For instance, we can write:

```
const fs = require('fs');
const configPath = './config.json';
const configFile = fs.readFileSync(configPath, 'utf-8')
const parsedConfig = JSON.parse(configFile);
exports.storageConfig = parsedConfig;
```

We called `readFileSync` to read `config.json` .

Then we parse the data from the JSON string.

And we export the config in the last line.

### Promisify Node’s child_process.exec and child_process.execFile Functions

We can convert the `child_process` `exec` and `execFile` methods with Bluebird.

For instance, we can write:

```
const util = require('util');
const exec = util.promisify(require('child_process').exec);

const listFiles = async () => {
  try {
    const { stdout, stderr } = await exec('ls');
    console.log('stdout:', stdout);
    console.log('stderr:', stderr);
  } catch (e) {
    console.error(e);
  }
}

listFiles();
```

We used the `util` library’s `promisify` method to convert the `exec` method to a promise.

Then we can call the promisified `exec` method with the `ls` command.

And then we get the full output with `stdout` and `stderr` .

`stdout` has the results. `stderr` has the error output.

### Conclusion

We can convert a buffer into a readable stream.

We can scrape web pages with an HTTP client and cheerio.

crypto-js has the MD5 method to create an MD5 hash.

Methods from the `child_process` can be converted to promises.

We can parse the user agent string on the server-side and parse the browser version data.