Web scraping Yelp data using Node JS and Puppeteer
Published: 2021-02-13 13:10:23

In this article, we will learn how to quickly scrape Yelp data using Puppeteer.

Puppeteer uses the Chromium browser behind the scenes to render HTML and Javascript, and so is very useful if getting the content that is loaded by javascript/AJAX functions.

For this, you will need to install Puppeteer inside a directory where you will write the scripts to scrape the data. For example, make a directory like this.
That will take a moment to install Puppeteer and Chromium.

Once done, let's start with a script like this.
Even though it looks like a lot, It just loads up the puppeteer browser, creates a new page, and loads the URL we want and waits for the full of the HTML to be loaded.

The evaluate function now gets into the page's content and allows you to query it with Puppeteer's query functions and CSS selectors.

The second line where the launch happens instructs Puppeteer to load in in the headless mode, so you dont see the browser, but it's there behind the scenes. The —user-agent string imitates a Chrome browser on a Mac so you dont get blocked.

Save this file as get_yelp.js, and if you run it, it should not return any errors.
Now let's see if we can scrape some data.

Open Chrome and navigate to the Yelp Sanfransico Restaurants page.
We are going to scrape the restaurant names, their address, phone number, rating, and the number of reviews. Let's open the inspect tool to see what we are up against.

You can see with some tinkering around that each post is encapsulated in a tag with the class name that starts with the container. We will use that as enough of a pattern to get the data we want.
We can use the CSS selector with an operator *= to instruct the engine to match anything with the word container in its class name
Since everything is inside this one class, we are going to use a forEach loop to get the data inside them and get all the individual pieces separately.

So the code will look like this.
You can see how the reviewCount is the partial class name that gets us the review count
For the address and phone number we run the secondaryAtrributes class and get all the subsequent

Tags inside like so.
The star rating is tricky as innerText won't fetch it. It's only supplied as an attribute, which is good enough for us. This trickery gets us the star rating data.
We put this in a try-catch. Because some might not have a summary and might raise an error and break the code.

If you run this, it should print all the restaurant details like so.
If you want to use this in production and want to scale to thousands of links, then you will find that you will get IP blocked quickly by Yelp. In this scenario, using a rotating proxy service to rotate IPs is almost a must.

Otherwise, you tend to get IP blocked a lot by automatic location, usage, and bot detection algorithms.

Our rotating proxy server Proxies API provides a simple API that can solve all IP Blocking problems instantly.

With millions of high speed rotating proxies located all over the world
With our automatic IP rotation
With our automatic User-Agent-String rotation (which simulates requests from different, valid web browsers and web browser versions)
With our automatic CAPTCHA solving technology
Hundreds of our customers have successfully solved the headache of IP blocks with a simple API.

A simple API can access the whole thing like below in any programming language.

You dont even have to take the pain of loading Puppeteer as we render Javascript behind the scenes, and you can just get the data and parse it any language like Node, Puppeteer or PHP or using any framework like Scrapy or Nutch. In all these cases, you can just call the URL with render support like so.