Anonymous Web Scraping with Node.js, Tor, Puppeteer and cheerio
Tags: node, tor, javascript
Published: 2020-01-25 20:47:27

---
title: Anonymous Web Scraping with Node.js, Tor, Puppeteer and cheerio
published: true
description: Ever wondered how to scrape both dynamic and static web pages without worrying if someone ban your IP? This article is all about anonymous web scrapping.
canonical_url: https://medium.com/p/3b36ec6a45dc
tags: node, tor, javascript
cover_image: https://cdn-images-1.medium.com/max/1200/1*3Zq1RSCl88XCEUbEYMMCAg.jpeg
---
Web Scraping is the technique of extracting data from websites. The term is used typically for automated data extraction. Today, I am going to show you how to crawl websites anonymously. The reason why you want to hide your identity is due to the fact that many web servers apply rules to websites which ban IPs after a certain amount of continuous requests. We are going to use [Puppeteer](https://github.com/puppeteer/puppeteer) for accessing web pages, [cheerio](https://github.com/cheeriojs/cheerio) for HTML parsing, and [Tor](https://www.torproject.org/) to run each request from a different IP address.

> While the legal aspects of Web Scraping vary, with many grey zones, 
> remember to always respect the [Terms of Service](https://en.wikipedia.org/wiki/Terms_of_service) of each web page you 
> scrape. [Ben Bernard](https://twitter.com/mbenbernard) has wrote a [nice article](https://benbernardblog.com/web-scraping-and-crawling-are-perfectly-legal-right/) about those legal issues.

## Setting Up Tor

First things first, we have to install our Tor client by using the following command.

```bash
sudo apt-get install tor
```

## Configure Tor

Next, we are going to configure our Tor client. The default Tor configuration uses a [SOCKS](https://en.wikipedia.org/wiki/SOCKS) port to provide us with one circuit to a single exit node (i.e. one IP address). This is handy for everyday use, like browsing, but for our specific scenario we need multiple IP addresses, so that we can switch between them while scraping.

To do this, we'll simply open additional ports to listen for [SOCKS](https://en.wikipedia.org/wiki/SOCKS) connections. This is done by adding multiple `SocksPort` options to main configuration file under `/etc/tor`.

Open `/etc/tor/torrc` file with your preferred editor and add the next lines in the end of the file.

{% gist https://gist.github.com/GeorgeGkas/fac4c98d1c1d53e220db2f41d198fb9e %} 

There a couple of things to notice here:

- The value of each `SocksPort` is a number, the port that Tor will listen for connections from SOCKS-speaking applications, like browsers.
- Because `SocksPort` value is a port to be open, the port must not already be used by another process.
- The initial port starts with value `9050`. This is the default SOCKS of the Tor client.
- **We bypass value `9051`**. This port is used by Tor to allow external applications who are connected to this port to control Tor process.
- As a simple convention, to open more ports, we increment each value after `9051` by one.

Restart the tor client to apply the new changes.

```bash
sudo /etc/init.d/tor restart
```


## Create a new Node project

Create a new directory for your project, I’ll call it `superWebScraping`.

```bash
mkdir superWebScraping
```


Navigate to `superWebScraping` and initialize an empty Node project.

```bash
cd superWebScraping && npm init -y
```

Install the required dependencies.

```bash
npm i --save puppeteer cheerio
```


## Browse with Puppeteer

[Puppeteer](https://github.com/puppeteer/puppeteer) is a [headless browser](https://developers.google.com/web/updates/2017/04/headless-chrome) that uses [DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/) to communicate with [Chrome](https://www.google.com/chrome/) or [Chromium](https://www.chromium.org/). The reason why we don’t use a request library, like [tor-request](https://www.npmjs.com/package/tor-request), is due to the fact that request libraries cannot process SPA websites that load their content dynamically.

Create an `index.js` file and add the below script. The statements are documented inline.

{% gist https://gist.github.com/GeorgeGkas/a59fc4544b3260215eb307253901dd9f %}

Run the script with

```bash
node index.js
```

You should see the Chromium browser navigating to [https://api.ipify.org](https://api.ipify.org) like the following screenshot

![](https://miro.medium.com/max/1365/1*YTPeS730jihtRUQJ28ThZw.png)

There is a reason why I chose the specific web page in my example. [https://api.ipify.org](https://api.ipify.org) is able to send us our public IP address. This is the IP you are browsing the web, without using Tor.

Change the above code by adding the following key in `puppeteer.launch` statement:

{% gist https://gist.github.com/GeorgeGkas/b770e74c691785a658ae3105c6ffe12c %}

We provide the `--proxy-server` argument to our browser. The value of this argument tells the browser to use a socks5 proxy in our local machine on top of port `9050`. The value of the port is one of the values we provided earlier in `torrc` file.

Now run again the script.

```bash
node index.js
```


This time you should see a different IP address. This is the IP that comes with Tor circuit.

![](https://miro.medium.com/max/1365/1*VudGr6lBoAlP4Djc_VX9Pw.png)

Mine is `144.217.7.33`, but you might have a different one. Note that if you run the script again on the same port `9050` you'll get the same IP address as before.

![](https://miro.medium.com/max/1365/1*VudGr6lBoAlP4Djc_VX9Pw.png)

This is the reason why we opened many ports in our Tor configuration. Try to use another port, like `9051`. The IP will not be the same.

## Scrape content with Cheerio

Now that we have a nice way to get our pages, it's time to scrape them. We are going to use the [cheerio](https://github.com/cheeriojs/cheerio) library. Cheerio is an HTML parser designed to use the same API as [jQuery](https://jquery.com/). Our task is to scrape the last 5 post titles of [Hacker News](https://news.ycombinator.com/).

Let's navigate to [Hacker News](https://news.ycombinator.com/).

![](https://miro.medium.com/max/1366/1*Cbpk6VOBeRabUQaLMXLhjQ.png)

We want to scrape the first 5 titles ("HAKMEM (1972), "Larry Roberts has died", etc). Inspecting the title of an article using my browser's DevTools I can see that each article is being wrapped with an HTML link element which has `storylink` class.

![](https://miro.medium.com/max/1365/1*CB4LvugprTfW5QOV6iOXiQ.png)

The procedure we are going to follow can be denoted by the bellow list:

- Launch a browser instance in headless mode with Tor proxy
- Create a new page
- Navigate to [https://news.ycombinator.com/](https://news.ycombinator.com/)
- Get page HTML content
- Load the HTML content in Cheerio
- Create an array to save the article titles.
- Access all the elements that have `storylink` class
- Get only the first 5 such elements, using Cherrio's [slice()](https://cheerio.js.org/#slice-start-end-) method.
- Iterate through those 5 elements using Cherrio's [each()](https://cheerio.js.org/#each-functionindex-element-) method.
- Append each article title in the array.

{% gist https://gist.github.com/GeorgeGkas/dda50ea74e238dbba75694c216e3cfe0 %}

And here is the output of the above code.

![](https://miro.medium.com/max/1366/1*K2U6x5KVJD8bxZyTH-89eA.png)

## Continuous scraping with different IP

The last thing to consider is to take advantage of all the SOCKS ports we defined in torrc file. This is quite easy. We’ll define an array with each item to be a different port number. Then we’ll rename `main()` to `scrape()` and we will define a new `main()` function that will call `scrape()` each time with a different port.

Here is the complete code.

{% gist https://gist.github.com/GeorgeGkas/28dc6484ae5effbbe592a3c432ee6f7c %}

## Conclusion

This article was intended to provide you with a gentle introduction to Web Scraping. Thank you for taking time to read it. 🙏