Three Ways to Retrieve JSON from the Web using Node.js
Tags: javascript, node, json, career
Published: 2019-08-16 21:57:44

---
title: Three Ways to Retrieve JSON from the Web using Node.js
published: true
description: looking at retrieving json with node-fetch package, https module, and request package
tags: javascript, node, json, career
series: ["Junior JavaScript Jobhunting: Quick Tips for Technicals and Takehomes"]
cover_image: https://thepracticaldev.s3.amazonaws.com/i/4olljtrz2tqd4y10y573.png
---

*Cover image credit: [**Hunter x Hunter** manga](https://en.wikipedia.org/wiki/Hunter_%C3%97_Hunter) by Yoshihiro Togashi, meme-ified by yours truly. <3*

In a recent technical challenge, I was asked to build a small Node.js app that first needed to retrieve some JSON from the web. Since I'm still relatively new to Node.js, I did not realize that Node.js **does not natively include the [fetch() API](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API)**, which I was used to using in my front-end JavaScript. 

*(What I didn't realize is that fetch() is actually a method from the [`Window` interface](https://developer.mozilla.org/en-US/docs/Web/API/Window)--and not having a front-end, there was no window! Awkward thing to realize at the beginning of a technical interview...)*

After getting through the challenge, I spent this past weekend refactoring the code and experimenting with a few different ways to retrieve JSON from the web using Node.js. **Spoiler:** there's an [http](https://nodejs.org/api/https.html)/[https](https://nodejs.org/api/https.html) module in Node.js, but also some cool packages that [mimic fetch()](https://www.npmjs.com/package/node-fetch), or that [simplify request syntax](https://www.npmjs.com/package/request) even further!

But before we get into that, I want to introduce a tool that I foolishly **did not** use during my technical challenge: [JSONView](https://jsonview.com/), a super-handy web extension for Chrome and Firefox that pretty-prints JSON in your browser.

# [JSONView](https://jsonview.com/), a super-handy web extension for Chrome and Firefox that pretty-prints JSON in your browser

One issue I had during my code challenge was (due to my own error) not working on my own computer, which includes this extension. Compare the following:

![raw json from reddit](https://thepracticaldev.s3.amazonaws.com/i/idoe9wtu7rlpahi3r740.png)
*Unformatted JSON from [https://www.reddit.com/r/popular.json](https://www.reddit.com/r/popular.json)*

versus

![formatted json from reddit using JSONView](https://thepracticaldev.s3.amazonaws.com/i/eunlo6kn684w6zyl26r6.png)
*Same JSON from [https://www.reddit.com/r/popular.json](https://www.reddit.com/r/popular.json), pretty-printed with JSONView*

On top of that, hovering your cursor over a particular field will show the path to access it:

![highlighted view in JSONView showing selected json property's path](https://thepracticaldev.s3.amazonaws.com/i/3ws6eijnvpznwuuzqxua.png)
*Cursor hovering over the "ups" field, with the path shown in the bottom-left corner*

Having this handy will make parsing and accessing the data you need way faster and easier.



# Replicating fetch() with ['node-fetch'](https://www.npmjs.com/package/node-fetch) package

The node-fetch package does pretty much what you expect: provide you with the fetch() syntax in Node.js. To install, run `npm install node-fetch`, and set up your code like this:

```javascript
const fetch = require('node-fetch');

let url = "https://www.reddit.com/r/popular.json";

let settings = { method: "Get" };

fetch(url, settings)
	.then(res => res.json())
	.then((json) => {
		// do something with JSON
	});
```

Here, we've started by importing the package via `require()`, and created a `settings` variable to define our http method as a Get request. From there, we use `fetch(url, settings)` just like we would on the front-end. As usual, we can parse the response `res` as JSON, and then do whatever we need to with it.

**Note:** from some VERY RUDIMENTARY benchmark testing, it *appears* that node-fetch is the fastest of the three options covered in this article. Here are the times clocked by each (however, this DOES include running the rest of the code from the challenge, not just the fetch/https/request itself):

```javascript
fetch: 0.689 seconds
https: 2.827 seconds
request: 3.65 seconds
```

I'd love for someone else to do a little more testing and verify/disprove this! Feel free to comment below if you're that person. ;)


# Using the [http](https://nodejs.org/api/https.html)/[https](https://nodejs.org/api/https.html)  modules provided by Node.js

Node.js comes with a pair of http/https modules, and in this case, the https module provides a built-in method for Get requests. Here's the code we'll be looking at:

```javascript
const https = require('https');

let url = "https://www.reddit.com/r/popular.json";

https.get(url,(res) => {
	let body = "";
	
	res.on("data", (chunk) => {
		body += chunk;
	});

	res.on("end", () => {
		try {
			let json = JSON.parse(body);
			// do something with JSON
		} catch (error) {
			console.error(error.message);
		};
	});

}).on("error", (error) => {
	console.error(error.message);
});
```

There's a bit more going on here! First, we import the https module with `require()`.  We can then call `https.get(url, (res) => {} )` to initiate a Get request. Then, inside the body of the callback, we start by creating an empty string `body` that we'll add our the text of our response (again called `res`)  to. 

From there, we have a few examples of the `.on` syntax, which will listen for a few different events--namely, `"data"`, `"end"`, and `"error"`.

When the response encounters `"data"`, we add each chunk as text to our body variable. Once we hit the `"end"` of the response, we use the 	`try / catch` syntax to try to parse our body's text as JSON, and return an error if it can't. Lastly, we chain another `.on` call to catch `"error"` for our initial `https.get()` request.

I find this syntax to be pretty clunky and verbose, although I do like the explicit error handling that is **required** by `https.get()`. However, this module is slower than the node-fetch package--see the benchmark results above.

# Simplifying syntax with ['request'](https://www.npmjs.com/package/request) package

The third strategy I used was the request package, which aims to simplify the (often verbose) syntax of Node.js's http requests. Since this is an external package, start by installing it with `npm install request`. 

Here's the code we'll be looking at:

```javascript
const request = require('request');

let url = "https://www.reddit.com/r/popular.json";

let options = {json: true};

  

request(url, options, (error, res, body) => {
	if (error) {
		return  console.log(error)
	};

	if (!error && res.statusCode == 200) {
		// do something with JSON, using the 'body' variable
	};
});
```

Wow, that's really readable! Let's break it down. As with the other examples, we import the package with `require()`, and set our url variable. The request package also has a nifty `options` feature, where you can specify a lot of things--but here, in setting `{ json: true }`, we tell the request to automatically parse the response's body as JSON if there's no error (and we get a 200 status code back). So, to access the JSON we want, just use the `body` variable!

This readability comes at the price of speed, however. Per the benchmark results above, this is the slowest option, most likely because so much is happening under-the-hood. However, the readability is top-notch, and configuring other http requests are just as simple as this Get request example! 

# Conclusion

This particular technical challenge was a great opportunity to dive into Node.js's http requests! Now, you should feel armed with a variety of tools to meet different situations.

As I said above, I'd love to have another person do some testing/benchmarking and verify or disprove the speed test results that I got! Since testing is still relatively new to me, I'd very much like to see how others approach benchmarking these methods. Thanks for reading, and feel free to comment below!