Scraping the web with Node.js
Tags: node, javascript, opensource, scraping
Published: 2020-01-30 03:19:29

---
title: Scraping the web with Node.js
published: true
date: 2020-01-29 09:49:00 UTC
tags: Node.js, javascript, open source, scraping
cover_image: https://blog.transitivebullsh.it/content/images/2019/09/pipes.jpg
canonical_url: https://blog.transitivebullsh.it/scraping-the-web-with-node-js/
---

_An in-depth guide to building a minimal, robust web scraper for extracting structured data on the internets._

Node.js provides a perfect, dynamic environment to quickly experiment and work with data from the web.

While there are more and more visual scraping products these days ([import.io](https://www.import.io/), [Spider](https://tryspider.com), [Scrapinghub](https://scrapinghub.com/), [Apify](https://apify.com/), [Crawly](http://crawly.diffbot.com), ……), there will always be a need for the simplicity and flexibility of writing one-off scrapers manually.

This post is intended as a tutorial for writing these types of data extraction scripts in Node.js, including some subtle best practices that I’ve learned from writing dozens of these types of crawlers over the years.

In particular, we’ll be walking through how to create a scraper for GitHub’s [list](https://github.com/trending) of trending repositories. If you want to follow along with the code, **check out the repo** [**scrape-github-trending**](https://github.com/transitive-bullshit/scrape-github-trending).

### Building Blocks

One of the best features of Node.js is the extremely comprehensive community of open source modules it has to offer. For this type of task, we’ll be leaning heavily on two modules, [**got**](https://github.com/sindresorhus/got) to robustly download raw HTML, and [**cheerio**](https://github.com/cheeriojs/cheerio) which provides a jQuery-inspired API for parsing and traversing those pages.

Cheerio is really great for quick & dirty web scraping where you just want to operate against raw HTML. If you’re dealing with more advanced scenarios where you want your crawler to mimic a real user as close as possible or navigate client-side scripting, you’ll likely want to use [**Puppeteer**](https://github.com/GoogleChrome/puppeteer).

Unlike cheerio, puppeteer is a wrapper for automating headless chrome instances, which is really useful for working with modern JS-powered SPAs. Since you’re working with Chrome itself, it also has best-in-class support for parsing / rendering / scripting conformance. Headless Chrome is still relatively new, but it will likely phase out older approaches such as [PhantomJS](http://phantomjs.org/) in the years to come.

As far as [**got**](https://github.com/sindresorhus/got) goes, there are dozens of HTTP fetching libraries available on NPM, with some of the more popular alternatives being [superagent](https://github.com/visionmedia/superagent), [axios](https://github.com/axios/axios), [unfetch](https://github.com/developit/unfetch) (isomorphic === usable from Node.js or browser), and finally [request](https://yarnpkg.com/en/package/request) / [request-promise-native](https://github.com/request/request-promise-native) (most popular library by far though the maintainers have officially deprecated any future development).

### Getting Started

Alright, for this tutorial we’ll be writing a scraper for GitHub’s list of [trending repositories](https://github.com/trending).

The first thing I do when writing a scraper is to open the target page in Chrome and take a look at the how the desired data is structured in dev tools.

![Scraping the web with Node.js](https://cdn-images-1.medium.com/max/2400/1*IpxFkpKHX7j69pyPVVwanw.png)<figcaption>Chrome Devtools is very handy for scraping 💯Also, I know I know I need to charge my laptop…. 😂</figcaption>

Switching back and forth between the `Console` and `Elements` tabs, you can use the `$$(‘.repo-list li’)` selector in the console to select all of the trending repos.

What you’re looking for in creating these [CSS selectors](https://adam-marsden.co.uk/css-cheat-sheet) is to keep them as simple as possible while also making them as focused as possible. By looking through the `Elements` tab and selecting the elements you’re interested in, you’ll usually come up with some potential selectors that may work. The next step is to try them out in the `Console` tab using the `$$()` syntax to make sure you’re only selecting the elements you intended to select. One rule of thumb here is to try and avoid using aspects of the HTML’s structure or classes that may change more often in refactors or code rewrites.

### Let’s write a scraper!

Now that we have a good idea of some CSS selectors that will target our desired data, let’s convert them to a Node.js script:

{% gist https://gist.github.com/transitive-bullshit/b02d91d276ddd2ff8497ff9b076fd323.js %}

Note that we’re using [async / await syntax](https://zeit.co/blog/async-and-await) here to handle downloading the external web page asynchronously in a way that looks synchronous.

- Line 12: we download the remote page and extract it’s text `body` (HTML).
- Line 14: we load that HTML into cheerio so that it’s easy to traverse and manipulate.
- Line 15: we select all the repository `li` elements using our previous CSS selector and map over them.
- Lines 16–32: we extract the relevant portions of each trending repo into a plain JSON object.
- Line 33: here we’re filtering out any repos that failed to parse correctly or threw an error. These will be `undefined` in the array and `[].filter(Boolean)` is a shorthand syntax for filtering any non-truthy values.

At this point, we’ve succeeded in scraping a single web page and extracting some relevant data. Here’s some example JSON output at this point:

{% gist https://gist.github.com/transitive-bullshit/60b7a776b263691c6947506fd10089f9.js %}

### Crawling Deeper

Now that we’ve explored how to scrape a single page, the next logical step is to branch out and crawl multiple pages. You could even get fancy and crawl links recursively from this point on, but for now we’ll just focus on crawling one level down in this data, that is the repository URLs themselves.

We’ll follow a very similar approach to how we scraped the original trending list. First, load up an example GitHub repository in Chrome and look through some of the most useful metadata that GitHub exposes and how you might target those elements via CSS selectors.

Once you have a good handle on what data you want to extract and have some working selectors in the `Console`, it’s time to write a Node.js function to download and parse a single GitHub repository.

{% gist https://gist.github.com/transitive-bullshit/2d74b84f3a959ab4eb3e3c6f5f231d17.js %}

The only real difference here from our first scraping example is that we’re using some different `cheerio` utility methods like [$.find()](https://github.com/cheeriojs/cheerio#findselector) and also doing some additional string parsing to coerce the data to our needs.

At this point, we’re able to extract a lot of the most useful metadata about each repo individually, but we need a way of robustly mapping over all the repos we want to process. For this, we’re going to use the excellent [**p-map**](https://github.com/sindresorhus/p-map) module. Most of the time you want to set a practical limit on parallelism whether it be throttling network bandwidth or compute resources. This is where [p-map](https://github.com/sindresorhus/p-map) really shines. I use it 99% of the time as a drop-in replacement for `Promise.all(…)`, which doesn’t support limiting parallelism.

{% gist https://gist.github.com/transitive-bullshit/23a870b0a35735fd43206c1de4aad23f.js %}

Here, we’re mapping over each repository with a maximum concurrency of 3 requests at a time. This helps significantly in making your crawler more robust against random network and server issues.

If you want to add one more level of robustness here, I would recommend wrapping your sub-scraping async functions in [p-retry](https://github.com/sindresorhus/p-retry) and [p-timeout](https://github.com/sindresorhus/p-timeout). This is what [got](https://nicedoc.io/sindresorhus/got) is actually doing under the hood to ensure more robust HTTP requests.

### All together now

Here is the full executable Node.js code. You can also find the full reproducible project at [scrape-github-trending](https://github.com/transitive-bullshit/scrape-github-trending).

{% gist https://gist.github.com/transitive-bullshit/8bc72ad41cb2f8c6234e46a85c6917f3.js %}

And an example of the corresponding JSON output:

{% gist https://gist.github.com/transitive-bullshit/60af3b44c1bfd83678cf7cc4b8e8c3f5.js %}

### Conclusion

I have used this exact pattern dozens of times for one-off scraping tasks in Node.js. It’s simple, robust, and really easy to customize to practically any targeted crawling / scraping scenarios.

It’s worth mentioning that [scrape-it](https://github.com/IonicaBizau/scrape-it) also looks like a very well engineered library that is essentially doing everything under the hood in this article.

If your crawling use case requires a more distributed workflow or more complicated client-side parsing, I would highly recommend checking out [Puppeteer](https://github.com/GoogleChrome/puppeteer), which is a game changing library from Google for automating headless Chrome. You may also want to check out the related crawling resources listed in [awesome-puppeteer](https://github.com/transitive-bullshit/awesome-puppeteer) such as [headless-chrome-crawler](https://github.com/yujiosaka/headless-chrome-crawler) which provides a distributed crawling solution built on top of Puppeteer.

In my experience, however, 95% of the time a simple one-file script like the one in this article tends to do the job just fine. And imho, [KISS](https://en.wikipedia.org/wiki/KISS_principle) is the single most important rule in software engineering.

Thanks for your time && I wish you luck on your future scraping adventures!