Web scraping Yelp Filters with Nodejs
Tags: webscraping, node, serpapi
Published: 2022-12-15 11:13:12

<h2 id='what'>What will be scraped</h2>

![what](https://user-images.githubusercontent.com/64033139/204550611-889e402b-7eb8-4533-af19-d71448eb612e.png)

<h2 id='full_code'>Full code</h2>

If you don't need an explanation, have a look at [the full code example in the online IDE](https://replit.com/@MikhailZub/Scrape-Yelp-Filters-with-NodeJS-SerpApi#withPuppeteer.js)

```javascript
const puppeteer = require("puppeteer-extra");
const StealthPlugin = require("puppeteer-extra-plugin-stealth");

puppeteer.use(StealthPlugin());

const serchQuery = "pizza"; //Parameter defines the query you want to search
const location = "Seattle, WA"; //Parameter defines from where you want the search to originate

const searchParams = {
  query: encodeURI(serchQuery),
  location: encodeURI(location),
};

const URL = `https://www.yelp.com/search?find_desc=${searchParams.query}&find_loc=${searchParams.location}`;

async function getFiltersFromPage(page) {
  const priceAndDistance = await page.evaluate(() => {
    return Array.from(document.querySelectorAll("aside[aria-labelledby='search-vertical-filter-panel-label'] > div > div")).reduce((result, el) => {
      if (!el.querySelector(":scope > div > div:nth-child(2)")) {
        return {
          ...result,
          price: Array.from(el.querySelectorAll(":scope > div > div:nth-child(1) button")).map((el) => {
            const text = el.querySelector("span").textContent;
            return {
              text,
              value: `RestaurantsPriceRange2.${text.length}`,
            };
          }),
        };
      } else {
        const filterTitle = el.querySelector(":scope > div > div:nth-child(1) p").textContent;
        if (filterTitle === "Distance") {
          return {
            ...result,
            distance: Array.from(el.querySelectorAll(":scope > div > div:nth-child(2) label")).map((el) => ({
              text: el.querySelector("span").textContent,
              value: el.querySelector("input").value,
            })),
          };
        } else return result;
      }
    }, {});
  });
  const filters = { ...priceAndDistance };
  const seeAllButtons = await page.$$("aside[aria-labelledby='search-vertical-filter-panel-label'] > div > div a");
  for (button of seeAllButtons) {
    await button.click();
    await page.waitForTimeout(2000);
    const filterTitle = await page.evaluate(() =>
      document.querySelector("#modal-portal-container div[aria-modal] div[role='presentation'] h4").textContent.split(" ")[1].toLowerCase()
    );
    filters[`${filterTitle}`] = await page.evaluate(() => {
      return Array.from(document.querySelectorAll("#modal-portal-container div[aria-modal] div[role='presentation'] li")).map((el) => ({
        text: el.querySelector("span").textContent,
        value: el.querySelector("input").value,
      }));
    });
    await page.click("#modal-portal-container div[aria-modal] div[role='presentation'] button[aria-label='Close']");
    await page.waitForTimeout(2000);
  }
  return filters;
}

async function getFilters() {
  const browser = await puppeteer.launch({
    headless: false, // if you want to see what the browser is doing, you need to change this option to "false"
    args: ["--no-sandbox", "--disable-setuid-sandbox"],
  });

  const page = await browser.newPage();
  page.setViewport({ width: 1600, height: 800 });
  await page.setDefaultNavigationTimeout(60000);
  await page.goto(URL);

  const filters = await getFiltersFromPage(page);

  await browser.close();

  return filters;
}

getFilters().then((result) => console.dir(result, { depth: null }));
```

<h2 id='preparation'>Preparation</h2>

First, we need to create a Node.js\* project and add [`npm`](https://www.npmjs.com/) packages [`puppeteer`](https://www.npmjs.com/package/puppeteer), [`puppeteer-extra`](https://www.npmjs.com/package/puppeteer-extra) and [`puppeteer-extra-plugin-stealth`](https://www.npmjs.com/package/puppeteer-extra-plugin-stealth) to control Chromium (or Chrome, or Firefox, but now we work only with Chromium which is used by default) over the [DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/) in [headless](https://developers.google.com/web/updates/2017/04/headless-chrome) or non-headless mode.

To do this, in the directory with our project, open the command line and enter:

```bash
$ npm init -y
```

And then:

```bash
$ npm i puppeteer puppeteer-extra puppeteer-extra-plugin-stealth
```

\*<span style="font-size: 15px;">If you don't have Node.js installed, you can [download it from nodejs.org](https://nodejs.org/en/) and follow the installation [documentation](https://nodejs.dev/learn/introduction-to-nodejs).</span>

📌Note: also, you can use `puppeteer` without any extensions, but I strongly recommended use it with `puppeteer-extra` with `puppeteer-extra-plugin-stealth` to prevent website detection that you are using headless Chromium or that you are using [web driver](https://www.w3.org/TR/webdriver/). You can check it on [Chrome headless tests website](https://intoli.com/blog/not-possible-to-block-chrome-headless/chrome-headless-test.html). The screenshot below shows you a difference.

![stealth](https://user-images.githubusercontent.com/64033139/173014238-eb8450d7-616c-42ae-8b2f-24eeb5fd5916.png)

<h2 id='process'>Process</h2>

We need to extract data from HTML elements. The process of getting the right CSS selectors is fairly easy via [SelectorGadget Chrome extension](https://selectorgadget.com/) which able us to grab CSS selectors by clicking on the desired element in the browser. However, it is not always working perfectly, especially when the website is heavily used by JavaScript.

We have a dedicated [Web Scraping with CSS Selectors](https://serpapi.com/blog/web-scraping-with-css-selectors-using-python/#css_gadget) blog post at SerpApi if you want to know a little bit more about them.

The Gif below illustrates the approach of selecting different parts of the results using SelectorGadget.

![how](https://user-images.githubusercontent.com/64033139/204551365-a92cbc97-4999-46b9-ac9b-809f66f1afc5.gif)

<h3 id='code_explanation'>Code explanation</h3>

Declare [`puppeteer`](https://www.npmjs.com/package/puppeteer-extra) to control Chromium browser from `puppeteer-extra` library and [`StealthPlugin`](https://www.npmjs.com/package/puppeteer-extra-plugin-stealth) to prevent website detection that you are using [web driver](https://www.w3.org/TR/webdriver/) from `puppeteer-extra-plugin-stealth` library:

```javascript
const puppeteer = require("puppeteer-extra");
const StealthPlugin = require("puppeteer-extra-plugin-stealth");
```

Next, we "say" to `puppeteer` use `StealthPlugin`, write what we want to search (`serchQuery` constant), search location, search URL and make search parameters with [`encodeURI`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/encodeURI) method:

```javascript
puppeteer.use(StealthPlugin());

const serchQuery = "pizza"; //Parameter defines the query you want to search
const location = "Seattle, WA"; //Parameter defines from where you want the search to originate

const searchParams = {
  query: encodeURI(serchQuery),
  location: encodeURI(location),
};

const URL = `https://www.yelp.com/search?find_desc=${searchParams.query}&find_loc=${searchParams.location}`;
```

Next, we write a function to get filters from the page:

```javascript
async function getFiltersFromPage(page) {
  ...
}
```

Then, we get price and distance filters info from the page context (using [`evaluate()`](https://pptr.dev/api/puppeteer.page.evaluate) method) and save it in the `priceAndDistance` object:

```javascript
const priceAndDistance = await page.evaluate(() => {
    ...
});
```

Next, we need to make and return a new array ([`Array.from()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/from) method) from all `"ul > li > div"` selectors ([`querySelectorAll()`](https://developer.mozilla.org/en-US/docs/Web/API/Document/querySelectorAll)) and using [`reduce`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/reduce) method make an object from an array:

```javascript
return Array.from(document.querySelectorAll("aside[aria-labelledby='search-vertical-filter-panel-label'] > div > div")).reduce((result, el) => {
    ...
}, {});
```

In the `reduce` method we need to check if `":scope > div > div:nth-child(2)"` selector is not present (using [`querySelector()`](https://developer.mozilla.org/en-US/docs/Web/API/Document/querySelector) method) we return `price` filters (using [`querySelectorAll()`](https://developer.mozilla.org/en-US/docs/Web/API/Document/querySelectorAll) method and [`textContent`](https://developer.mozilla.org/en-US/docs/Web/API/Node/textContent) property).

Otherwise (`else` statement), we get the filter category title, and return only "Distance" filters, because other filters are hidden and show only a few of them and we get it all later:

```javascript
if (!el.querySelector(":scope > div > div:nth-child(2)")) {
  return {
    ...result,
    price: Array.from(el.querySelectorAll(":scope > div > div:nth-child(1) button")).map((el) => {
      const text = el.querySelector("span").textContent;
      return {
        text,
        value: `RestaurantsPriceRange2.${text.length}`,
      };
    }),
  };
} else {
  const filterTitle = el.querySelector(":scope > div > div:nth-child(1) p").textContent;
  if (filterTitle === "Distance") {
    return {
      ...result,
      distance: Array.from(el.querySelectorAll(":scope > div > div:nth-child(2) label")).map((el) => ({
        text: el.querySelector("span").textContent,
        value: el.querySelector("input").value,
      })),
    };
  } else return result;
}
```

Next, we write `priceAndDistance` in the `filters` constant (using [`spread syntax`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Spread_syntax)) and get "See all" buttons from other filter categories with [`$$()`](https://pptr.dev/api/puppeteer.page.__) method:

```javascript
const filters = { ...priceAndDistance };
const seeAllButtons = await page.$$("aside[aria-labelledby='search-vertical-filter-panel-label'] > div > div a");
```

Next, we need to iterate over `seeAllButtons` ([`for...of loop`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for...of)), click each of them ([`element.click()`](https://pptr.dev/api/puppeteer.elementhandle.click) method), wait 2 seconds (using [`waitForTimeout`](https://pptr.dev/api/puppeteer.page.waitfortimeout) method), get filter category title and add filters from the page with this title to `filters` object. Then we click on "Close" button ([`page.click()`](https://pptr.dev/api/puppeteer.page.click) method), wait 2 seconds and repeat the loop with other categories.

To get data from the page we use next methods:

- [`querySelectorAll()`](https://developer.mozilla.org/en-US/docs/Web/API/Document/querySelectorAll);
- [`querySelector()`](https://developer.mozilla.org/en-US/docs/Web/API/Document/querySelector);
- [`textContent`](https://developer.mozilla.org/en-US/docs/Web/API/Node/textContent);
- [`value`](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input#value);
- [`split()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/split);
- [`toLowerCase()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/toLowerCase);

```javascript
for (button of seeAllButtons) {
  await button.click();
  await page.waitForTimeout(2000);
  const filterTitle = await page.evaluate(() =>
    document.querySelector("#modal-portal-container div[aria-modal] div[role='presentation'] h4").textContent.split(" ")[1].toLowerCase()
  );
  filters[`${filterTitle}`] = await page.evaluate(() => {
    return Array.from(document.querySelectorAll("#modal-portal-container div[aria-modal] div[role='presentation'] li")).map((el) => ({
      text: el.querySelector("span").textContent,
      value: el.querySelector("input").value,
    }));
  });
  await page.click("#modal-portal-container div[aria-modal] div[role='presentation'] button[aria-label='Close']");
  await page.waitForTimeout(2000);
}
```

Next, write a function to control the browser, and get information:

```javascript
async function getOrganicResults() {
  ...
}
```

In this function first we need to define `browser` using `puppeteer.launch({options})` method with current `options`, such as `headless: true` and `args: ["--no-sandbox", "--disable-setuid-sandbox"]`.

These options mean that we use [headless](https://developers.google.com/web/updates/2017/04/headless-chrome) mode and array with [arguments](https://peter.sh/experiments/chromium-command-line-switches/) which we use to allow the launch of the browser process in the online IDE. And then we open a new `page` and set page viewport resolution ([`setViewport()`](https://pptr.dev/api/puppeteer.page.setviewport) method) to show filters panel:

```javascript
const browser = await puppeteer.launch({
  headless: true, // if you want to see what the browser is doing, you need to change this option to "false"
  args: ["--no-sandbox", "--disable-setuid-sandbox"],
});

const page = await browser.newPage();
page.setViewport({ width: 1600, height: 800 });
```

Next, we change default ([30 sec](https://github.com/puppeteer/puppeteer/blob/2a0eefb99f0ae00dacc9e768a253308c0d18a4c3/src/common/TimeoutSettings.ts#L17)) time for waiting for selectors to 60000 ms (1 min) for slow internet connection with [`.setDefaultNavigationTimeout()`](https://pptr.dev/api/puppeteer.page.setdefaultnavigationtimeout) method, go to `URL` with [`.goto()`](https://pptr.dev/api/puppeteer.page.goto) method:

```javascript
await page.setDefaultNavigationTimeout(60000);
await page.goto(URL);
```

And finally, we get filters from the page, close the browser, and return the received data:

```javascript
const filters = await getFiltersFromPage(page);

await browser.close();

return filters;
```

Now we can launch our parser:

```bash
$ node YOUR_FILE_NAME # YOUR_FILE_NAME is the name of your .js file
```

<h2 id='output'>Output</h2>

```json
{
   "price":[
      {
         "text":"$",
         "value":"RestaurantsPriceRange2.1"
      },
      {
         "text":"$$",
         "value":"RestaurantsPriceRange2.2"
      },
        ... and other items
   ],
   "distance":[
      {
         "text":"Bird's-eye View",
         "value":"g:-122.43782043457031,47.55614031294337,-122.23320007324219,47.69497434186282"
      },
      {
         "text":"Driving (5 mi.)",
         "value":"g:-122.38666534423828,47.590651847264034,-122.28435516357422,47.6600691664467"
      },
        ... and other items
   ],
   "categories":[
      {
         "text":"Restaurants",
         "value":"restaurants"
      },
      {
         "text":"Pizza",
         "value":"pizza"
      },
      ... and other items
   ],
   "features":[
      {
         "text":"Reservations",
         "value":"OnlineReservations"
      },
      {
         "text":"Waitlist",
         "value":"OnlineWaitlistReservation"
      },
     ... and other items
   ],
   "neighborhoods":[
      {
         "text":"Admiral",
         "value":"WA:Seattle::Admiral"
      },
      {
         "text":"Alki",
         "value":"WA:Seattle::Alki"
      },
    ... and other items
   ]
}
```

<h2 id='howToUse'>How to apply filters</h2>

You can apply filters those was scraped to the Yelp search using the following URL and change `searchParams` constant in the DIY solution section in our [Web scraping Yelp Organic Results with Nodejs](https://serpapi.com/blog/web-scraping-yelp-organic-results-with-nodejs/#full_code) and [Web scraping Yelp Ads Results with Nodejs](https://serpapi.com/blog/web-scraping-yelp-ads-results-with-nodejs/#full_code) blog posts:

```javascript
const serchQuery = "pizza"; //Parameter defines the query you want to search
const location = "Seattle, WA"; //Parameter defines from where you want the search to originate
const priceAndFeaturesFilter "RestaurantsPriceRange2.1,OnlineReservations"; // for price and features filters
const categoryFilter "restaurants"; // for category filters
const locationFilter "g:-122.43782043457031,47.55614031294337,-122.23320007324219,47.69497434186282"; // for neighborhoods or distance filters (distance and neighborhoods filters can't be used together)


const searchParams = {
  query: encodeURI(serchQuery),
  location: encodeURI(location),
  priceAndFeaturesFilter: encodeURI(priceAndFeaturesFilter),
  categoryFilter: encodeURI(categoryFilter),
  locationFilter: encodeURI(locationFilter),
};

const URL = `https://www.yelp.com/search?find_desc=${searchParams.query}&find_loc=${searchParams.location}&attrs=${searchParams.priceAndFeaturesFilter}&cflt=${searchParams.categoryFilter}&l=${searchParams.locationFilter}`;
```

<h2 id='serp_api'>Using<a href="https://serpapi.com/yelp-filters"> Yelp Filters API </a>from SerpApi</h2>

This section is to show the comparison between the DIY solution and our solution.

The biggest difference is that you don't need to create the parser from scratch and maintain it.

There's also a chance that the request might be blocked at some point from Google, we handle it on our backend so there's no need to figure out how to do it yourself or figure out which CAPTCHA, proxy provider to use.

First, we need to install [`google-search-results-nodejs`](https://www.npmjs.com/package/google-search-results-nodejs):

```bash
npm i google-search-results-nodejs
```

Here's the [full code example](https://replit.com/@MikhailZub/Scrape-Yelp-Filters-with-NodeJS-SerpApi#withSerpApi.js), if you don't need an explanation:

```javascript
const SerpApi = require("google-search-results-nodejs");
const search = new SerpApi.GoogleSearch(process.env.API_KEY); //your API key from serpapi.com

const params = {
  engine: "yelp", // search engine
  device: "desktop", //Parameter defines the device to use to get the results. It can be set to "desktop" (default), "tablet", or "mobile"
  find_loc: "Seattle, WA", //Parameter defines from where you want the search to originate.
  find_desc: "pizza", // Parameter defines the query you want to search
};

const getJson = () => {
  return new Promise((resolve) => {
    search.json(params, resolve);
  });
};

const getResults = async () => {
  const json = await getJson();
  return json.filters;
};

getResults().then((result) => console.dir(result, { depth: null }));
```

<h3 id='serp_api_code_explanation'>Code explanation</h3>

First, we need to declare `SerpApi` from [`google-search-results-nodejs`](https://www.npmjs.com/package/google-search-results-nodejs) library and define new `search` instance with your API key from [SerpApi](https://serpapi.com/manage-api-key):

```javascript
const SerpApi = require("google-search-results-nodejs");
const search = new SerpApi.GoogleSearch(API_KEY);
```

Next, we write the necessary parameters for making a request:

```javascript
const params = {
  engine: "yelp", // search engine
  device: "desktop", //Parameter defines the device to use to get the results. It can be set to "desktop" (default), "tablet", or "mobile"
  find_loc: "Seattle, WA", //Parameter defines from where you want the search to originate.
  find_desc: "pizza", // Parameter defines the query you want to search
};
```

Next, we wrap the search method from the SerpApi library in a promise to further work with the search results:

```javascript
const getJson = () => {
  return new Promise((resolve) => {
    search.json(params, resolve);
  });
};
```

And finally, we declare the function `getResult` that gets data from the page and return it:

```javascript
const getResults = async () => {
  ...
};
```

In this function we get `json` with reuslts and return `filters` from received `json`:

```javascript
const json = await getJson();
return json.filters;
```

After, we run the `getResults` function and print all the received information in the console with the [`console.dir`](https://nodejs.org/api/console.html#consoledirobj-options) method, which allows you to use an object with the necessary parameters to change default output options:

```javascript
getResults().then((result) => console.dir(result, { depth: null }));
```

<h2 id='serp_api_output'>Output</h2>

```json
{
   "neighborhoods":{
      "value":"p:WA:Seattle::",
      "list":[
         {
            "text":"Waterfront",
            "value":"Waterfront"
         },
         {
            "text":"Fremont",
            "value":"Fremont"
         },
        ... and other items
      ]
   },
   "distance":[
      {
         "text":"Bird's-eye View",
         "value":"g:-122.43782043457031,47.55614031294337,-122.23320007324219,47.69497434186282"
      },
      {
         "text":"Driving (5 mi.)",
         "value":"g:-122.38666534423828,47.590651847264034,-122.28435516357422,47.6600691664467"
      },
        ... and other items
   ],
   "price":[
      {
         "text":"$",
         "value":"RestaurantsPriceRange2.1"
      },
      {
         "text":"$$",
         "value":"RestaurantsPriceRange2.2"
      },
        ... and other items
   ],
   "category":[
      {
         "text":"Cheesesteaks",
         "value":"cheesesteaks"
      },
      {
         "text":"Middle Eastern",
         "value":"mideastern"
      },
      ... and other items
   ],
   "features":[
      {
         "text":"Waiter Service",
         "value":"RestaurantsTableService"
      },
      {
         "text":"Open to All",
         "value":"BusinessOpenToAll"
      },
      ... and other items
   ]
}
```

<h2 id='howToUseSerpApi'>How to apply filters</h2>

You can apply filters those was scraped to the Yelp search by changing `params` constant in the SerpApi solution section in our [Web scraping Yelp Organic Results with Nodejs](https://serpapi.com/blog/web-scraping-yelp-organic-results-with-nodejs/#serp_api) and [Web scraping Yelp Ads Results with Nodejs](https://serpapi.com/blog/web-scraping-yelp-ads-results-with-nodejs/#serp_api) blog posts:

```javascript
const params = {
  engine: "yelp", // search engine
  device: "desktop", //Parameter defines the device to use to get the results. It can be set to "desktop" (default), "tablet", or "mobile"
  find_loc: "Seattle, WA", //Parameter defines from where you want the search to originate.
  find_desc: "pizza", // Parameter defines the query you want to search
  cflt: "restaurants", // for category filters
  attrs: "RestaurantsPriceRange2.1,OnlineReservations", // for price and features filters
  l: "g:-122.43782043457031,47.55614031294337,-122.23320007324219,47.69497434186282", // for neighborhoods or distance filters (distance and neighborhoods filters can't be used together)
};
```

<h2 id='links'>Links</h2>

- [Code in the online IDE](https://replit.com/@MikhailZub/Scrape-Yelp-Filters-with-NodeJS-SerpApi#index.js)
- [Yelp Filters API](https://serpapi.com/yelp-filters)

If you want other functionality added to this blog post or if you want to see some projects made with SerpApi, [write me a message](mailto:miha01012019@gmail.com).

---

<p style="text-align: center;">Join us on <a href="https://twitter.com/serp_api">Twitter</a> | <a href="https://www.youtube.com/channel/UCUgIHlYBOD3yA3yDIRhg_mg">YouTube</a></p>
<p style="text-align: center;">Add a  <a href="https://github.com/serpapi/public-roadmap/issues">Feature Request</a>💫 or a <a href="https://github.com/serpapi/public-roadmap/issues">Bug</a>🐞</p>