Composing the results of nested asynchronous calls in Javascript using async.waterfall and ES6 Promises
Tags: node, es6, asynchronous, javascript
Published: 2017-07-12 19:21:22

---
title: Composing the results of nested asynchronous calls in Javascript using async.waterfall and ES6 Promises
published: true
description: 
tags: node, es6, asynchronous, javascript
canonical_url: http://sites.bu.edu/perryd/2017/07/11/composing-the-results-of-nested-asynchronous-calls-in-javascript
---
<i>I posted this recently [on my blog](http://sites.bu.edu/perryd) and thought I'd share it here. I taught a course this summer in MEAN application development and observed that many of my students had a tough time working in an asynchronous environment (most had not done any Javascript coding). This article was my attempt to both help my students with some patterns and to help solidify the topic in my own head. I welcome constructive criticism, especially if I'm off track!
</i>

<h2>Grokking asynchronicity</h2>
One ofÂ the toughest things to get your head around in Javascript is how to handle nested asynchronous calls, especially when aÂ function depends on the result of a preceding one. I see this quite a bit in my software engineering course where teams are required to synthesize new information from two distinct third-party data sources. The pattern is to look something up that returns a collection, and then look something else up for each ofÂ the members in the collection.

There are multiple approaches to this problem, but most rely on Javascript Promises, either from a third-party library in ES5 and earlier or ES6's built-in Promises. In this article we'll use both by using ES6 native Promises and the async.js package. This isn't a discussion of ES6 Promises per se, but it might help you get to the point of understanding them.

The file discussed here is intended to be mounted on a route in a Node/Express application:
<pre style="padding-left: 30px;"><span>//From app.js
const </span><span>rp </span>= <span>require</span>(<span>'./routes/request-promises'</span>)
<span>app</span>.<span>use</span>(<span>'/rp'</span>, <span>rp</span>)</pre>
[Unfortunately the Wordpress implementation used on the BU blog does a horrific job displaying code, so the code snippets below are images for the bulk of the discussion. The complete file is available at the bottom of the post.]

The route hereÂ usesÂ two APIs:
<em>https://weathers.co</em> for current weather in 3 cities, and
<em>https://api.github.com/search/repositories?q=&lt;term&gt;</em> to search GitHub repos forÂ the 'hottest' city.

In ourÂ example we'll grab weather for 3 cities, pick the hottest one,Â then hit GitHub with a search for repos that have that city's name. It's a bit contrivedÂ but we just want to demonstrate using the results of one API call to populate a second when all of them are asynch.

There are two packages (apart from Express) used:

![packages](https://thepracticaldev.s3.amazonaws.com/i/pkf9zy7eoz6g4oklmhim.png)


The first is a reduced-size packaging of the standard <em>request-promise</em> package; we don't need all of the features of the larger library. <em>request-promise</em> itself is a wrapper that adds Promises to the standard <em>request</em> package used to make network calls, for example to an HTTP endpoint.

There's only one route in this module, a GET on the top-level URL. In the snippet above, this router is mounted at '/rp', and so the complete URL path is http://localhost:3000/rp (assuming the server is set to listen to port 3000).

![route](https://thepracticaldev.s3.amazonaws.com/i/ap7psiptu2a4bed5c4f0.png)

On line 50 the <em>async.waterfall</em> method is used to create a chain of functions that will execute in turn. Each function returns a Promise, and so the next function in line waits until the preceding function's Promise is resolved. We need this because each function provides data that the following function will use in some way. That data is providedÂ to each function in turn along with a callback, which is the next function in the waterfall.

<em>async.waterfall</em> takes two parameters in this implementation: An array of functions to call in order, and a final function (<em>renderTable </em>on line 51)Â to execute last. A small side note: The final function can be anonymous, but when debugging a big application it can be tough to figure out exactly which anonymous function out of potentially hundreds is causing a problem. Naming those functions helps immensely when debugging!
<h3>getCityTemperatures</h3>
The first function in the waterfall, <em>getCityTemperatures</em>, calls the weathers.co API for each city in a hardcoded array. This is theÂ first function in the waterfall and so it receives only one param, the callback (cb).

![getCityTemperatures](https://thepracticaldev.s3.amazonaws.com/i/ueptjzehhnq9schcqkr8.png)

This is the most interesting function of the three because it has to do an API call forÂ each city in the <em>cities</em> array, and they all are asynchronous calls. We can't return from the functionÂ until all of the city weather has been collected.

In each API call, once the current temperature is known it is plugged into the city's objectÂ in the <em>cities</em> array. (Note that weathers.co doesn't always return the actual current temperature.Â Often it is a cached value.)Â The technique here is to create a Promise that encompasses the API calls on line 83. This Promise represents the entire function, and it won't be resolved until all of the city weather has been collected.

A secondÂ Promise is set up in the local function <em>getWeather</em> on line 91,Â and that's the one that handlesÂ each individual city's API call. <em>request.get()</em> itself returns a Promise on line 92 (because we are using the request-promise-lite package), and so we make the <em>request.get()</em> and followÂ it with a then() which will run when the API call returns. The resolve() at line 96Â gets us out of this inner Promise and on to the next one.

Note that we don't yet execute <em>getWeather()</em>,Â we're just <em>defining</em> it here. Line 109 uses the <em>Array.map</em> method to return an array that has three functions in it. Each function has one of the city objects passed into it, and each returns a Promise (as defined on line 91). That <em>cityPromises</em> array looks like
<pre>[getWeather({name: 'Miami', temperature: null}),Â <span>getWeather({name: 'Atlanta', temperature: null}),Â </span><span>getWeather({name: 'Boston', temperature: null})]</span></pre>
after line 109 completes.

We <em>still</em> haven't executed anything. The mapping in line 109 sets us up to use the <em>Promise.all( )</em> method from the ES6 native Promise library in line 121, which only resolves when all of the Promises in its input array have resolved. It's a bit like the async.waterfall( ) call from line 50, but in this case order doesn't matter. If any of the Promises reject, the entire Promise.all( ) structure rejects at that moment, even if there are outstanding unresolved Promises.

Once all of the Promises have resolved (meaning that each city's weather has been recorded), the then( ) in line 122 executes, and calls the passed-in callback with the now-complete array of cities and temperatures. The <em>null</em> value handed to the callback is interpreted as an error object.

Of interest here is that Promise.all( ) is, practically speaking, running the requests in parallel.
<h3>getHottestCity</h3>
The next function in the chain, getHottestCity, simply finds the largest temperature in the array of cities and returns that city's object to the next function in the waterfall. This is synchronous code and so doesn't require a Promise.

![getHottestCity](https://thepracticaldev.s3.amazonaws.com/i/vw2wlcc9hu28ezwwtgup.png)

Line 140 creates a slice with just the temperatures from the <em>cities</em> array and calls <em>Math.max( )</em> to determine the largest value. Line 143 then looks for the hottest temperature in the original cities array and returns the corresponding object. Both of these lines use the ES6 way of defining functions with fat-arrow notation (=&gt;), and line 140 uses the new spread operator.

Once the 'hot' city has been determined it is passed to the next function in the waterfall in line 148. Here again, the <em>null</em> is in the place of an error object.
<h3>findGithubRepos</h3>
The next-to-last function in the waterfall searches Github for projects that include the name of the 'hot' city discovered earlier. The User-Agent header in line 167 is required by the Github search API -- you can change it to any string that you like.

![findGithubRepos](https://thepracticaldev.s3.amazonaws.com/i/v068wbtybka21tm6vkl3.png)

The API call in line 165 uses <em>request-promise</em> (from line 29) and so returns a Promise. Once the Promise is resolved, the <em>then( )</em> function builds an array of objects from some of the data returned.

In line 180 the completed array of items is passed to the next function in the waterfall along with the original 'hot' city object.
<h3>The final function</h3>
The last function to execute simply takes the results of our waterfall and renders a Pug page on line 57.

![waterfall](https://thepracticaldev.s3.amazonaws.com/i/njnbfzc6wk00bny8vu5f.png)

Here's a <a href="https://gist.github.com/pdonham/8d2c7e0ccd482150a6877b5075794b23">link to the complete file</a>. If you have questions or corrections, leave a comment!