Web scraping Google Play App Reviews with Nodejs
Tags: webscraping, node, serpapi
Published: 2022-11-08 17:43:30

<h2 id='what'>What will be scraped</h2>

![what](https://user-images.githubusercontent.com/64033139/197349503-dc4a6cb6-2187-431d-bacf-5cd3dbc4fe9a.png)

📌Note: You can use official the [Google Play Developer API](https://developers.google.com/android-publisher) which has a default limit of [200,000 requests per day](https://developers.google.com/android-publisher/quotas#:~:text=The%20Google%20Play%20Developer%20API,of%20the%20Google%20Cloud%20Console.) for [retrieving the list of reviews and individual reviews](https://developers.google.com/android-publisher/reply-to-reviews#retrieving_reviews).

Also, you can use a complete third-party Google Play Store App scraping solution [google-play-scraper](https://github.com/facundoolano/google-play-scraper). Third-party solutions are usually used to break the quota limit.

This blog post is meant to give an idea and step-by-step examples of how to scrape Google Play Store App Reviews using [Puppeteer](https://www.npmjs.com/package/puppeteer) to create something on your own.

<h2 id='full_code'>Full code</h2>

If you don't need an explanation, have a look at [the full code example in the online IDE](https://replit.com/@MikhailZub/Scrape-Google-Play-App-Review-with-NodeJS-SerpApi#withPuppeteer.js)

```javascript
const puppeteer = require("puppeteer-extra");
const StealthPlugin = require("puppeteer-extra-plugin-stealth");

puppeteer.use(StealthPlugin());

const reviewsLimit = 100; // hardcoded limit for demonstration purpose

const searchParams = {
  id: "com.discord", // Parameter defines the ID of a product you want to get the results for
  hl: "en", // Parameter defines the language to use for the Google search
  gl: "us", // parameter defines the country to use for the Google search
};

const URL = `https://play.google.com/store/apps/details?id=${searchParams.id}&hl=${searchParams.hl}&gl=${searchParams.gl}`;

async function scrollPage(page, clickElement, scrollContainer) {
  let lastHeight = await page.evaluate(`document.querySelector("${scrollContainer}").scrollHeight`);
  while (true) {
    await page.click(clickElement);
    await page.waitForTimeout(500);
    await page.keyboard.press("End");
    await page.waitForTimeout(2000);
    let newHeight = await page.evaluate(`document.querySelector("${scrollContainer}").scrollHeight`);
    const reviews = await page.$$(".RHo1pe");
    if (newHeight === lastHeight || reviews.length > reviewsLimit) {
      break;
    }
    lastHeight = newHeight;
  }
}

async function getReviewsFromPage(page) {
  return await page.evaluate(() => ({
    reviews: Array.from(document.querySelectorAll(".RHo1pe")).map((el) => ({
      title: el.querySelector(".X5PpBb")?.textContent.trim(),
      avatar: el.querySelector(".gSGphe > img")?.getAttribute("srcset")?.slice(0, -3),
      rating: parseInt(el.querySelector(".Jx4nYe > div")?.getAttribute("aria-label")?.slice(6)),
      snippet: el.querySelector(".h3YV2d")?.textContent.trim(),
      likes: parseInt(el.querySelector(".AJTPZc")?.textContent.trim()) || "No likes",
      date: el.querySelector(".bp9Aid")?.textContent.trim(),
      response: {
        title: el.querySelector(".ocpBU .I6j64d")?.textContent.trim(),
        snippet: el.querySelector(".ocpBU .ras4vb")?.textContent.trim(),
        date: el.querySelector(".ocpBU .I9Jtec")?.textContent.trim(),
      },
    })),
  }));
}

async function getAppReviews() {
  const browser = await puppeteer.launch({
    headless: true, // if you want to see what the browser is doing, you need to change this option to "false"
    args: ["--no-sandbox", "--disable-setuid-sandbox"],
  });

  const page = await browser.newPage();

  await page.setDefaultNavigationTimeout(60000);
  await page.goto(URL);

  await page.waitForSelector(".qZmL0");

  const moreReviewButton = await page.$("c-wiz[jsrenderer='C7s1K'] .VMq4uf button");

  if (moreReviewButton) {
    await page.click("c-wiz[jsrenderer='C7s1K'] .VMq4uf button");
    await page.waitForSelector(".RHo1pe .h3YV2d");
    await scrollPage(page, ".RHo1pe .h3YV2d", ".odk6He");
  }
  const reviews = await getReviewsFromPage(page);

  await browser.close();

  return reviews;
}

getAppReviews().then((result) => console.dir(result, { depth: null }));
```

<h2 id='preparation'>Preparation</h2>

First, we need to create a Node.js\* project and add [`npm`](https://www.npmjs.com/) packages [`puppeteer`](https://www.npmjs.com/package/puppeteer), [`puppeteer-extra`](https://www.npmjs.com/package/puppeteer-extra) and [`puppeteer-extra-plugin-stealth`](https://www.npmjs.com/package/puppeteer-extra-plugin-stealth) to control Chromium (or Chrome, or Firefox, but now we work only with Chromium which is used by default) over the [DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/) in [headless](https://developers.google.com/web/updates/2017/04/headless-chrome) or non-headless mode.

To do this, in the directory with our project, open the command line and enter:

```bash
$ npm init -y
```

And then:

```bash
$ npm i puppeteer puppeteer-extra puppeteer-extra-plugin-stealth
```

\*<span style="font-size: 15px;">If you don't have Node.js installed, you can [download it from nodejs.org](https://nodejs.org/en/) and follow the installation [documentation](https://nodejs.dev/learn/introduction-to-nodejs).</span>

📌Note: also, you can use `puppeteer` without any extensions, but I strongly recommended use it with `puppeteer-extra` with `puppeteer-extra-plugin-stealth` to prevent website detection that you are using headless Chromium or that you are using [web driver](https://www.w3.org/TR/webdriver/). You can check it on [Chrome headless tests website](https://intoli.com/blog/not-possible-to-block-chrome-headless/chrome-headless-test.html). The screenshot below shows you a difference.

![stealth](https://user-images.githubusercontent.com/64033139/173014238-eb8450d7-616c-42ae-8b2f-24eeb5fd5916.png)

<h2 id='process'>Process</h2>

First of all, we need to scroll through all games listings until there are no more listings loading which is the difficult part described below.

The next step is to extract data from HTML elements after scrolling is finished. The process of getting the right CSS selectors is fairly easy via [SelectorGadget Chrome extension](https://selectorgadget.com/) which able us to grab CSS selectors by clicking on the desired element in the browser. However, it is not always working perfectly, especially when the website is heavily used by JavaScript.

We have a dedicated [Web Scraping with CSS Selectors](https://serpapi.com/blog/web-scraping-with-css-selectors-using-python/#css_gadget) blog post at SerpApi if you want to know a little bit more about them.

The Gif below illustrates the approach of selecting different parts of the results using SelectorGadget.

![how](https://user-images.githubusercontent.com/64033139/197349667-4862fd1f-68f9-4cf4-b6d7-5d6fd85fa7a3.gif)

<h3 id='code_explanation'>Code explanation</h3>

Declare [`puppeteer`](https://www.npmjs.com/package/puppeteer-extra) to control Chromium browser from `puppeteer-extra` library and [`StealthPlugin`](https://www.npmjs.com/package/puppeteer-extra-plugin-stealth) to prevent website detection that you are using [web driver](https://www.w3.org/TR/webdriver/) from `puppeteer-extra-plugin-stealth` library:

```javascript
const puppeteer = require("puppeteer-extra");
const StealthPlugin = require("puppeteer-extra-plugin-stealth");
```

Next, we "say" to `puppeteer` use `StealthPlugin`, write the necessary request parameters, search URL and set how many reviews we want to receive (`reviewsLimit` constant):

```javascript
puppeteer.use(StealthPlugin());

const reviewsLimit = 100; // hardcoded limit for demonstration purpose

const searchParams = {
  id: "com.discord", // Parameter defines the ID of a product you want to get the results for
  hl: "en", // Parameter defines the language to use for the Google search
  gl: "us", // parameter defines the country to use for the Google search
};

const URL = `https://play.google.com/store/apps/details?id=${searchParams.id}&hl=${searchParams.hl}&gl=${searchParams.gl}`;
```

Next, we write a function to scroll the page to load all reviews:

```javascript
async function scrollPage(page, clickElement, scrollContainer) {
  ...
}
```

In this function, first, we need to get `scrollContainer` height (using [`evaluate()`](https://pptr.dev/api/puppeteer.page.evaluate) method).

Then we use `while` loop in which we click ([`click()`](https://pptr.dev/api/puppeteer.page.click) method) on the review element to stay in focus, wait 0.5 seconds (using [`waitForTimeout`](https://pptr.dev/api/puppeteer.page.waitfortimeout) method), press "End" button to scroll to the last review element, wait 2 seconds and get a new `scrollContainer` height.

Next, we check if `newHeight` is equal to `lastHeight` or if the number of received reviews is more than `reviewsLimit` we stop the loop. Otherwise, we define `newHeight` value to `lastHeight` variable and repeat again until the page was not scrolled down to the end:

```javascript
let lastHeight = await page.evaluate(`document.querySelector("${scrollContainer}").scrollHeight`);
while (true) {
  await page.click(clickElement);
  await page.waitForTimeout(500);
  await page.keyboard.press("End");
  await page.waitForTimeout(2000);
  let newHeight = await page.evaluate(`document.querySelector("${scrollContainer}").scrollHeight`);
  const reviews = await page.$$(".RHo1pe");
  if (newHeight === lastHeight || reviews.length > reviewsLimit) {
    break;
  }
  lastHeight = newHeight;
}
```

Next, we write a function to get reviews data from the page:

```javascript
async function getReviewsFromPage(page) {
  ...
}
```

In this function, we get information from the page context and save it in the returned object. Next, we need to get all HTML elements with `".RHo1pe"` selector ([`querySelectorAll()`](https://developer.mozilla.org/en-US/docs/Web/API/Document/querySelectorAll) method).

Then we use [`map()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/map) method to iterate an array that built with [`Array.from()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/from) method:

```javascript
return await page.evaluate(() => ({
    reviews: Array.from(document.querySelectorAll(".RHo1pe")).map((el) => ({
      ...
    })),
}));
```

And finally, we need to get all the data using the following methods:

- [`slice()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/slice);
- [`querySelector()`](https://developer.mozilla.org/en-US/docs/Web/API/Document/querySelector);
- [`getAttribute()`](https://developer.mozilla.org/en-US/docs/Web/API/Element/getAttribute);
- [`textContent`](https://developer.mozilla.org/en-US/docs/Web/API/Node/textContent);
- [`trim()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/trim).

```javascript
title: el.querySelector(".X5PpBb")?.textContent.trim(),
avatar: el.querySelector(".gSGphe > img")?.getAttribute("srcset")?.slice(0, -3),
rating: parseInt(el.querySelector(".Jx4nYe > div")?.getAttribute("aria-label")?.slice(6)),
snippet: el.querySelector(".h3YV2d")?.textContent.trim(),
likes: parseInt(el.querySelector(".AJTPZc")?.textContent.trim()) || "No likes",
date: el.querySelector(".bp9Aid")?.textContent.trim(),
response: {
    title: el.querySelector(".ocpBU .I6j64d")?.textContent.trim(),
    snippet: el.querySelector(".ocpBU .ras4vb")?.textContent.trim(),
    date: el.querySelector(".ocpBU .I9Jtec")?.textContent.trim(),
},
```

Next, write a function to control the browser, and get information:

```javascript
async function getAppReviews() {
  ...
}
```

In this function first we need to define `browser` using `puppeteer.launch({options})` method with current `options`, such as `headless: true` and `args: ["--no-sandbox", "--disable-setuid-sandbox"]`.

These options mean that we use [headless](https://developers.google.com/web/updates/2017/04/headless-chrome) mode and array with [arguments](https://peter.sh/experiments/chromium-command-line-switches/) which we use to allow the launch of the browser process in the online IDE. And then we open a new `page`:

```javascript
const browser = await puppeteer.launch({
  headless: true, // if you want to see what the browser is doing, you need to change this option to "false"
  args: ["--no-sandbox", "--disable-setuid-sandbox"],
});

const page = await browser.newPage();
```

Next, we change default ([30 sec](https://github.com/puppeteer/puppeteer/blob/2a0eefb99f0ae00dacc9e768a253308c0d18a4c3/src/common/TimeoutSettings.ts#L17)) time for waiting for selectors to 60000 ms (1 min) for slow internet connection with [`.setDefaultNavigationTimeout()`](https://pptr.dev/api/puppeteer.page.setdefaultnavigationtimeout) method, go to `URL` with [`.goto()`](https://pptr.dev/api/puppeteer.page.goto) method and use [`.waitForSelector()`](https://pptr.dev/api/puppeteer.page.waitforselector) method to wait until the selector is load:

```javascript
await page.setDefaultNavigationTimeout(60000);
await page.goto(URL);
await page.waitForSelector(".qZmL0");
```

And finally, we check if "show all reviews" button is present on the page (using [`$()`](https://pptr.dev/api/puppeteer.page._) method), we click it and wait until the page was scrolled, save reviews data from the page in the `reviews` constant, close the browser, and return the received data:

```javascript
const moreReviewButton = await page.$("c-wiz[jsrenderer='C7s1K'] .VMq4uf button");

if (moreReviewButton) {
  await page.click("c-wiz[jsrenderer='C7s1K'] .VMq4uf button");
  await page.waitForSelector(".RHo1pe .h3YV2d");
  await scrollPage(page, ".RHo1pe .h3YV2d", ".odk6He");
}
const reviews = await getReviewsFromPage(page);

await browser.close();

return reviews;
```

Now we can launch our parser:

```bash
$ node YOUR_FILE_NAME # YOUR_FILE_NAME is the name of your .js file
```

<h2 id='output'>Output</h2>

```json
{
   "reviews":[
      {
         "title":"Faera Rathion",
         "avatar":"https://play-lh.googleusercontent.com/a-/ACNPEu_jb8bwx7nBMUAm6ogXkSy2udBVV7GYnygiESuv=s64-rw",
         "rating":1,
         "snippet":"I would've given this 5 stars a few months ago, being a long time user, but these recent updates have made the app extremely frustrating to use. I get randomly put into channels when I open the app, they scroll me back sometimes hundreds of messages, it's impossible to see all the channels in some Discords, doesn't clear notifications without having to try to fully scroll through a channel I was mentioned in to the point of having to refresh it multiple times and many more consistent issues.",
         "likes":2,
         "date":"October 19, 2022",
         "response":{
            "title":"Discord Inc.",
            "snippet":"We're sorry for the inconvenience. We hear you and our teams are actively working on rolling out fixes daily. If you continue to experience issues, please make sure your app is on the latest updated version. Also, your feedback greatly affects what we focus on so please let us know if you continue to have issues at dis.gd/contact.",
            "date":"October 19, 2022"
         }
      },
      {
         "title":"Avoxx Nepps",
         "avatar":"https://play-lh.googleusercontent.com/a-/ACNPEu_WAW8BQ6SiTqR2gFzjxXpjSjFiAEx3E3cMKGQ1w5o=s64-rw",
         "rating":2,
         "snippet":"The new update has made it borderline unusable. It is extremely glitchy and a lot of times doesn't even work properly. Can't even join a voice call without it leaving and rejoining by itself or muting me for unknown reason. The new video system absolutely sucks. All of the minor inconveniences the previous version had is nothing compared to this update which looks like it was thrown together by a team of teenagers in Middle School in a month for a school project.",
         "likes":"No likes",
         "date":"October 20, 2022",
         "response":{
            "title":"Discord Inc.",
            "snippet":"We'd like to know more about the issues you've encountered after the recent update. Could you please submit a support ticket so we can look into the issue?: dis.gd/contact If you have any suggestions about what should be changed or improved, please share them on our Feedback page here: dis.gd/feedback",
            "date":"October 21, 2022"
         }
      },
      ...and other reviews
   ]
}
```

<h2 id='serp_api'>Using<a href="https://serpapi.com/google-play-product-api"> Google Play Product API </a>from SerpApi</h2>

This section is to show the comparison between the DIY solution and our solution.

The biggest difference is that you don't need to create the parser from scratch and maintain it.

There's also a chance that the request might be blocked at some point from Google, we handle it on our backend so there's no need to figure out how to do it yourself or figure out which CAPTCHA, proxy provider to use.

First, we need to install [`google-search-results-nodejs`](https://www.npmjs.com/package/google-search-results-nodejs):

```bash
npm i google-search-results-nodejs
```

Here's the [full code example](https://replit.com/@MikhailZub/Scrape-Google-Play-App-Review-with-NodeJS-SerpApi#withSerpApi.js), if you don't need an explanation:

```javascript
const SerpApi = require("google-search-results-nodejs");
const search = new SerpApi.GoogleSearch(process.env.API_KEY); //your API key from serpapi.com

const reviewsLimit = 100; // hardcoded limit for demonstration purpose

const params = {
  engine: "google_play_product", // search engine
  gl: "us", // parameter defines the country to use for the Google search
  hl: "en", // parameter defines the language to use for the Google search
  store: "apps", // parameter defines the type of Google Play store
  product_id: "com.discord", // Parameter defines the ID of a product you want to get the results for.
  all_reviews: "true", // Parameter is used for retriving all reviews of a product
};

const getJson = () => {
  return new Promise((resolve) => {
    search.json(params, resolve);
  });
};

const getResults = async () => {
  const allReviews = [];
  while (true) {
    const json = await getJson();
    if (json.reviews) {
      allReviews.push(...json.reviews);
    } else break;
    if (json.serpapi_pagination?.next_page_token) {
      params.next_page_token = json.serpapi_pagination?.next_page_token;
    } else break;
    if (allReviews.length > reviewsLimit) break;
  }
  return allReviews;
};

getResults().then((result) => console.dir(result, { depth: null }));
```

<h3 id='serp_api_code_explanation'>Code explanation</h3>

First, we need to declare `SerpApi` from [`google-search-results-nodejs`](https://www.npmjs.com/package/google-search-results-nodejs) library and define new `search` instance with your API key from [SerpApi](https://serpapi.com/manage-api-key):

```javascript
const SerpApi = require("google-search-results-nodejs");
const search = new SerpApi.GoogleSearch(API_KEY);
```

Next, we write how many reviews we want to receive (`reviewsLimit` constant) and the necessary parameters for making a request:

```javascript
const reviewsLimit = 100; // hardcoded limit for demonstration purpose

const params = {
  engine: "google_play_product", // search engine
  gl: "us", // parameter defines the country to use for the Google search
  hl: "en", // parameter defines the language to use for the Google search
  store: "apps", // parameter defines the type of Google Play store
  product_id: "com.discord", // Parameter defines the ID of a product you want to get the results for.
  all_reviews: "true", // Parameter is used for retriving all reviews of a product
};
```

Next, we wrap the search method from the SerpApi library in a promise to further work with the search results:

```javascript
const getJson = () => {
  return new Promise((resolve) => {
    search.json(params, resolve);
  });
};
```

And finally, we declare the function `getResult` that gets data from the page and return it:

```javascript
const getResults = async () => {
  ...
};
```

In this function first, we declare an array `allReviews` with results data:

```javascript
const allReviews = [];
```

Next, we need to use `while` loop. In this loop we get `json` with results, check if `reviews` are present on the page, push ([`push()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/push) method) them to `allReviews` array (using [`spread syntax`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Spread_syntax)), set `next_page_token` to `params` object, and repeat the loop until results aren't present on the page or number of the received reviews more than `reviewsLimit`:

```javascript
while (true) {
  const json = await getJson();
  if (json.reviews) {
    allReviews.push(...json.reviews);
  } else break;
  if (json.serpapi_pagination?.next_page_token) {
    params.next_page_token = json.serpapi_pagination?.next_page_token;
  } else break;
  if (allReviews.length > reviewsLimit) break;
}
return allReviews;
```

After, we run the `getResults` function and print all the received information in the console with the [`console.dir`](https://nodejs.org/api/console.html#consoledirobj-options) method, which allows you to use an object with the necessary parameters to change default output options:

```javascript
getResults().then((result) => console.dir(result, { depth: null }));
```

<h2 id='serp_api_output'>Output</h2>

```json
[
   {
      "title":"Johnathan Kamuda",
      "avatar":"https://play-lh.googleusercontent.com/a-/ACNPEu9QaKcoysS5G21Q5DQxs5nm2pg07GfJa-M_ezvOWfU",
      "rating":5,
      "snippet":"Been using Discord for many, many years. They are always making it better. It's become so much more robust and feature filled since I first started using it. And it's platform to pay for extras is great. You don't NEED to, but it's nice to have that kind of service a available if we wanted some perks. I think some of the options could be laid out better. Personal example - changing individuals volume in a call, not an intuitive option to find at first. Things like that fixed, would be perfect.",
      "likes":29,
      "date":"October 19, 2022"
   },
   {
      "title":"Lark Reid",
      "avatar":"https://play-lh.googleusercontent.com/a-/ACNPEu-RDynxDvoH-8_jnUj48AbZvXYrrafsLP3WT0fyTA",
      "rating":1,
      "snippet":"Ever since the new update me and other people that I know have completely lost the ability to upload more than one image/video at a time. It freezes on 70-100% when uploading multiple at a time. Now the audio on videos that I upload turn into static. I played the videos on my phone to make sure they weren't corrupted, and they are just fine. Sometimes when I open the app it gets stuck connecting and I have to restart it. Please fix your app asap. It's just not my phone that is effected.",
      "likes":84,
      "date":"October 21, 2022",
      "response":{
         "title":"Discord Inc.",
         "snippet":"We're sorry for the inconvenience. We hear you and our teams are actively working on rolling out fixes daily. If you continue to experience issues, please make sure your app is on the latest updated version. Also, your feedback greatly affects what we focus on so please let us know if you continue to have issues at dis.gd/contact.",
         "date":"October 21, 2022"
      }
   },
    ... and other reviews
]
```

<h2 id='links'>Links</h2>

- [Code in the online IDE](https://replit.com/@MikhailZub/Scrape-Google-Play-App-Review-with-NodeJS-SerpApi#index.js)
- [Google Play Product API](https://serpapi.com/google-play-product-api)

If you want to see some projects made with SerpApi, [write me a message](mailto:miha01012019@gmail.com).

---

<p style="text-align: center;">Join us on <a href="https://twitter.com/serp_api">Twitter</a> | <a href="https://www.youtube.com/channel/UCUgIHlYBOD3yA3yDIRhg_mg">YouTube</a></p>
<p style="text-align: center;">Add a  <a href="https://github.com/serpapi/public-roadmap/issues">Feature Request</a>💫 or a <a href="https://github.com/serpapi/public-roadmap/issues">Bug</a>🐞</p>