Web Scraping All Google Play App Reviews in Python
Tags: python, programming, tutorial, webscraping
Published: 2022-09-26 11:36:12

- <a href="#what_will_be_scraped">What will be scraped</a>
- <a href="#prerequisites">Prerequisites</a>
- <a href="#full_code">Full Code</a>
    - <a href="#code_explanation">Code Explanation</a>
- <a href="#serp_api">Using Google Play Product Reviews API</a>
- <a href="#links">Links</a>
- <a href="#outro">Outro</a>


<h2 id="what_will_be_scraped">What will be scraped</h2>

![image](https://user-images.githubusercontent.com/78694043/192235235-cca423af-8e84-448c-ba2d-39e1801d7efd.png)

<h2 id="prerequisites">Prerequisites</h2>

**Basic knowledge scraping with CSS selectors**

CSS selectors declare which part of the markup a style applies to thus allowing to extract data from matching tags and attributes.

If you haven't scraped with CSS selectors, there's a dedicated blog post of mine about [how to use CSS selectors when web-scraping](https://serpapi.com/blog/web-scraping-with-css-selectors-using-python/) that covers what it is, pros and cons, and why they're matter from a web-scraping perspective.

**Separate virtual environment**

In short, it's a thing that creates an independent set of installed libraries including different Python versions that can coexist with each other at the same system thus prevention libraries or Python version conflicts.

If you didn't work with a virtual environment before, have a look at the dedicated [Python virtual environments tutorial using Virtualenv and Poetry](https://serpapi.com/blog/python-virtual-environments-using-virtualenv-and-poetry/) blog post of mine to get familiar.

📌Note: this is not a strict requirement for this blog post.

**Install libraries**:

```bash
pip install playwright parsel
```

You also need to install [chromium](https://chromedriver.chromium.org/downloads) for `playwright` to work and operate the browser:

```bash
playwright install chromium
```

After that, if you're on Linux, you might need to install additional things (`playwright` will prompt you in the terminal in case something is missing):

```bash
sudo apt-get install -y libnss3 libnspr4 libatk1.0-0 libatk-bridge2.0-0 libcups2 libdrm2 libxkbcommon0 libxcomposite1 libxdamage1 libxfixes3 libxrandr2 libgbm1 libatspi2.0-0 libwayland-client0
```

**Reduce the chance of being blocked**

There's a chance that a request might be blocked. Have a look at [how to reduce the chance of being blocked while web-scraping](https://serpapi.com/blog/how-to-reduce-chance-of-being-blocked-while-web/), there are eleven methods to bypass blocks from most websites and some of them will be covered in this blog post.


<h2 id="full_code">Full Code</h2>

```python
import time, json, re
from parsel import Selector
from playwright.sync_api import sync_playwright


def run(playwright):
    page = playwright.chromium.launch(headless=True).new_page()
    page.goto("https://play.google.com/store/apps/details?id=com.collectorz.javamobile.android.books&hl=en_GB&gl=US")

    user_comments = []

    # if "See all reviews" button present
    if page.query_selector('.Jwxk6d .u4ICaf button'):
        print("the button is present.")

        print("clicking on the button.")
        page.query_selector('.Jwxk6d .u4ICaf button').click(force=True)

        print("waiting a few sec to load comments.")
        time.sleep(4)
        
        last_height = page.evaluate('() => document.querySelector(".fysCi").scrollTop')  # 2200

        while True:
            print("scrolling..")
            page.keyboard.press("End")
            time.sleep(3)

            new_height = page.evaluate('() => document.querySelector(".fysCi").scrollTop')

            if new_height == last_height:
                break
            else:
                last_height = new_height

    selector = Selector(text=page.content())
    page.close()

    print("done scrolling. Exctracting comments...")
    for index, comment in enumerate(selector.css(".RHo1pe"), start=1):

        comment_likes = comment.css(".AJTPZc::text").get()   

        user_comments.append({
            "position": index,
            "user_name": comment.css(".X5PpBb::text").get(),
            "user_avatar": comment.css(".gSGphe img::attr(srcset)").get().replace(" 2x", ""),
            "user_comment": comment.css(".h3YV2d::text").get(),
            "comment_likes": comment_likes.split("people")[0].strip() if comment_likes else None,
            "app_rating": re.search(r"\d+", comment.css(".iXRFPc::attr(aria-label)").get()).group(),
            "comment_date": comment.css(".bp9Aid::text").get(),
            "developer_comment": {
                "dev_title": comment.css(".I6j64d::text").get(),
                "dev_comment": comment.css(".ras4vb div::text").get(),
                "dev_comment_date": comment.css(".I9Jtec::text").get()
            }
        })

    print(json.dumps(user_comments, indent=2, ensure_ascii=False))


with sync_playwright() as playwright:
    run(playwright)
```

<h2 id="code_explanation">Code Explanation</h2>

Import libraries:

```python
import time, json
from playwright.sync_api import sync_playwright
```

- `time` to set a `sleep()` intervals between each scroll.
- `json` just for pretty printing.
- `sync_playwright` for synchronous API. [`playwright` have asynchronous API](https://playwright.dev/python/docs/intro#usage) as well using `asyncio` module.

Declare a function: 

```python
def run(playwright):
    # further code..
```

Initialize `playwright`, connect to `chromium`, `launch()` a browser `new_page()` and `goto()` a given URL: 

```python
page = playwright.chromium.launch(headless=False).new_page()
page.goto("https://play.google.com/store/apps/details?id=com.collectorz.javamobile.android.books&hl=en_GB&gl=US")

user_comments = [] # temporary list for all extracted data
```

- [`playwright.chromium`is a connection to the Chromium](https://playwright.dev/python/docs/api/class-playwright#playwright-chromium) browser instance.
- [`launch()` will launch the browser](https://playwright.dev/python/docs/api/class-browsertype#browser-type-launch), and `headless` argument will run it in headless mode. Default is `True`.  
- [`new_page()` creates a new page in a new browser context](https://playwright.dev/python/docs/api/class-browser#browser-new-page).
- [`page.goto("URL")`](https://playwright.dev/python/docs/api/class-page#page-goto) will make a request to provided website.

Next, we need to check if the button responsible for showing all reviews is present and click on it if present:

```python
if page.query_selector('.Jwxk6d .u4ICaf button'):
    print("the button is present.")

    print("clicking on the button.")
    page.query_selector('.Jwxk6d .u4ICaf button').click(force=True)

    print("waiting a few sec to load comments.")
    time.sleep(4)
```
- [`query_selector`](https://playwright.dev/python/docs/api/class-page#page-query-selector) is function that accepts CSS selectors to be searched.
- [`click`](https://playwright.dev/python/docs/api/class-page#page-click) is to click on the button and `force=True` will bypass any auto-waits and click immidiately.

Scroll to the bottom of the comments window:

```python
last_height = page.evaluate('() => document.querySelector(".fysCi").scrollTop')  # 2200

while True:
    print("scrolling..")
    page.keyboard.press("End")
    time.sleep(3)

    new_height = page.evaluate('() => document.querySelector(".fysCi").scrollTop')

    if new_height == last_height:
        break
    else:
        last_height = new_height
```

- [`page.evaluate()`](https://playwright.dev/python/docs/api/class-worker#worker-evaluate) will run a JavaScript code in the browser context that will measurement of the height of the `.fysCi` selector. [`scrollTop`](https://developer.mozilla.org/en-US/docs/Web/API/Element/scrollTop) gets the number of pixels scrolled from a given element, in this case CSS selector.
- `time.sleep(3)` will stop code execution for 3 seconds to load more comments.
- Then it will measure a `new_height` after the scroll running the same measurement JavaScript code.
- Finally, it will check `if new_height == last_height`, and if so, exit the `while` loop by using `break`. 
- `else` set the `last_height` to `new_height` and run the iteration (scroll) again.


After that, pass scrolled HTML content to `parsel`, `close` the browser:

```python
selector = Selector(text=page.content())
page.close()
```

Iterate over all results after the `while` loop is done:

```python
for index, comment in enumerate(selector.css(".RHo1pe"), start=1):

    comment_likes = comment.css(".AJTPZc::text").get()   

    user_comments.append({
        "position": index,
        "user_name": comment.css(".X5PpBb::text").get(),
        "user_avatar": comment.css(".gSGphe img::attr(srcset)").get().replace(" 2x", ""),
        "user_comment": comment.css(".h3YV2d::text").get(),
        "comment_likes": comment_likes.split("people")[0].strip() if comment_likes else None,
        "app_rating": re.search(r"\d+", comment.css(".iXRFPc::attr(aria-label)").get()).group(),
        "comment_date": comment.css(".bp9Aid::text").get(),
        "developer_comment": {
            "dev_title": comment.css(".I6j64d::text").get(),
            "dev_comment": comment.css(".ras4vb div::text").get(),
            "dev_comment_date": comment.css(".I9Jtec::text").get()
        }
    })
```

Print the data:

```python
print(json.dumps(user_comments, indent=2, ensure_ascii=False))
```

Run your code using [context manager](https://realpython.com/python-with-statement/):

```python
with sync_playwright() as playwright:
    run(playwright)
```

### Output

```json
[
  {
    "position": 1,
    "user_name": "JazzTripp",
    "user_avatar": "https://play-lh.googleusercontent.com/a-/ACNPEu8THUUDL3yzcd0bHSDRR4OegOWLmfbFi70On0HbRg",
    "user_comment": "This app takes a bit if getting used to at first, but the catalogue is extensive, and most bar codes and isbn numbers can be used to autofill a good chuck of a collection. I personally use this app for manga, and while its only correct about 70% of the time, its still easy to update and change as you see fit. The 'add to core' option makes me feel like im actually helping out the app, so i add data whenever i can. Keep up the good work guys!",
    "comment_likes": "20",
    "app_rating": "5",
    "comment_date": "May 06, 2022",
    "developer_comment": null
  }, ... other results
  {
    "position": 875,
    "user_name": "Originalbigguy",
    "user_avatar": "https://play-lh.googleusercontent.com/a/ALm5wu3dYTOHvlG8SUqgyTbRnjv9I49JtxgySY-RwTJU=s64-rw-mo",
    "user_comment": "Not free",
    "comment_likes": null,
    "app_rating": "1",
    "comment_date": "9 April 2021",
    "developer_comment": {
      "dev_title": "Collectorz.com",
      "dev_comment": "The app is never advertised as free anywhere. The app information clearly states this is a paid subscription app.\n",
      "dev_comment_date": "10 April 2021"
    }
  }
]
```


<h2 id="serp_api">Using <a href="https://serpapi.com/google-play-product-reviews">Google Play Product Reviews API</a></h2>


As we support extracting reviews data from Google Play App, this section is to show the comparison between the DIY solution and our solution.

The biggest difference is that you don't need to use browser automation to scrape results, create the parser from scratch and maintain it.

Keep in mind that there's also a chance that the request might be blocked at some point from Google (or CAPTCHA), we handle it on our backend.


Installing [`google-search-results` from PyPi](https://pypi.org/project/google-search-results/):

```bash
pip install google-search-results
```

```python
from serpapi import GoogleSearch
from urllib.parse import (parse_qsl, urlsplit)

params = {
  "api_key": "...",                                        # your serpapi api key
  "engine": "google_play_product",                         # serpapi parsing engine
  "store": "apps",                                         # app results
  "gl": "us",                                              # country of the search
  "hl": "en",                                              # language of the search
  "product_id": "com.collectorz.javamobile.android.books"  # app id
}

search = GoogleSearch(params)                              # where data extraction happens on the backend

reviews = []

while True:
    results = search.get_dict()                            # JSON -> Python dict

    for review in results["reviews"]:
        reviews.append({
            "title": review.get("title"),
            "avatar": review.get("avatar"),
            "rating": review.get("rating"),
            "likes": review.get("likes"),
            "date": review.get("date"),
            "snippet": review.get("snippet"),
            "response": review.get("response")
        })

    # pagination
    if "next" in results.get("serpapi_pagination", {}):
        search.params_dict.update(dict(parse_qsl(urlsplit(results.get("serpapi_pagination", {}).get("next")).query)))
    else:
        break
        
print(json.dumps(reviews, indent=2, ensure_ascii=False))
```

### Output:

```json
[
  {
    "title": "JazzTripp",
    "avatar": "https://play-lh.googleusercontent.com/a-/ACNPEu8THUUDL3yzcd0bHSDRR4OegOWLmfbFi70On0HbRg",
    "rating": 5.0,
    "likes": 20,
    "date": "May 06, 2022",
    "snippet": "This app takes a bit if getting used to at first, but the catalogue is extensive, and most bar codes and isbn numbers can be used to autofill a good chuck of a collection. I personally use this app for manga, and while its only correct about 70% of the time, its still easy to update and change as you see fit. The 'add to core' option makes me feel like im actually helping out the app, so i add data whenever i can. Keep up the good work guys!",
    "response": null
  }, ... other reviews
  {
    "title": "Originalbigguy",
    "avatar": "https://play-lh.googleusercontent.com/a/ALm5wu3dYTOHvlG8SUqgyTbRnjv9I49JtxgySY-RwTJU=mo",
    "rating": 1.0,
    "likes": 0,
    "date": "April 09, 2021",
    "snippet": "Not free",
    "response": {
      "title": "Collectorz.com",
      "snippet": "The app is never advertised as free anywhere. The app information clearly states this is a paid subscription app.",
      "date": "April 10, 2021"
    }
  }
]
```


<h2 id="links">Links</h2>

- [GitHub Repository](https://github.com/dimitryzub/scrape-google-play-store-app)

<p style="text-align: center;">Join us on <a href="https://www.reddit.com/r/SerpApi/">Reddit</a> | <a href="https://twitter.com/serp_api">Twitter</a> | <a href="https://www.youtube.com/channel/UCUgIHlYBOD3yA3yDIRhg_mg">YouTube</a></p>