Scrape Google Product Page with Python
Tags: webscraping, tutorial, python, programming
Published: 2022-12-09 12:38:23

- <a href="#what_will_be_scraped">What will be scraped</a>
- <a href="#fullcode">Full Code</a>
- <a href="#preparation">Preparation</a>
- <a href="#code_explanation">Code Explanation</a>
- <a href="#google_product_page_api">Using Google Product Page API from SerpApi</a>
- <a href="#links">Links</a>


<h2 id="what_will_be_scraped">What will be scraped</h2>

![wwbs-google-shopping-product-page](https://user-images.githubusercontent.com/81998012/201950347-beb99825-4adc-45b5-ae49-3ebffa69f6f4.png)


<h2 id="fullcode">Full Code</h2>

If you don't need explanation, have a look at [full code example in the online IDE](https://replit.com/@chukhraiartur/blog-google-shopping-product-page#parcel_solution.py).

```python
import requests, json
from parsel import Selector


def get_product_page_results(url, params, headers):
	html = requests.get(url, params=params, headers=headers)
	selector = Selector(html.text)
	
	title = selector.css('.sh-t__title::text').get()
	prices = [price.css('::text').get() for price in selector.css('.MLYgAb .g9WBQb')]
	low_price = selector.css('.KaGvqb .qYlANb::text').get()
	high_price = selector.css('.xyYTQb .qYlANb::text').get()
	shown_price = selector.css('.FYiaub').xpath('normalize-space()').get()
	reviews = int(selector.css('.YVQvvd .HiT7Id span::text').get()[1:-1].replace(',', ''))
	rating = float(selector.css('.uYNZm::text').get())
	extensions = [extension.css('::text').get() for extension in selector.css('.OA4wid')]
	description = selector.css('.sh-ds__trunc-txt::text').get()
	media = [image.css('::attr(src)').get() for image in selector.css('.sh-div__image')]
	highlights = [highlight.css('::text').get() for highlight in selector.css('.KgL16d span')]
	
	data = {
		'title': title,
		'prices': prices,
		'typical_prices': {
			'low': low_price,
			'high': high_price,
			'shown_price': shown_price
		},
		'reviews': reviews,
		'rating': rating,
		'extensions': extensions,
		'description': description,
		'media': media,
		'highlights': highlights
	}
	
	return data


def main():
	# https://docs.python-requests.org/en/master/user/quickstart/#passing-parameters-in-urls
	params = {
		'product_id': '16230039729797264158',	# product id
		'hl': 'en',     						# language
		'gl': 'us'	     						# country of the search, US -> USA
	}

	# https://docs.python-requests.org/en/master/user/quickstart/#custom-headers
	headers = {
	    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36'
	}

	URL = f'https://www.google.com/shopping/product/{params["product_id"]}?hl={params["hl"]}&gl={params["gl"]}'
	
	product_page_results = get_product_page_results(URL, params, headers)
	
	print(json.dumps(product_page_results, indent=2, ensure_ascii=False))


if __name__ == "__main__":
    main()
```


<h2 id="preparation">Preparation</h2>

**Install libraries**:

```lang-none
pip install requests parsel
```

**Reduce the chance of being blocked**

Make sure you're using [request headers](https://docs.python-requests.org/en/master/user/quickstart/#custom-headers) [`user-agent`](https://developer.mozilla.org/en-US/docs/Glossary/User_agent) to act as a "real" user visit. Because default `requests` `user-agent` is [`python-requests`](https://github.com/psf/requests/blob/589c4547338b592b1fb77c65663d8aa6fbb7e38b/requests/utils.py#L808-L814) and websites understand that it's most likely a script that sends a request. [Check what's your `user-agent`](https://www.whatismybrowser.com/detect/what-is-my-user-agent/).

There's a [how to reduce the chance of being blocked while web scraping blog post](https://serpapi.com/blog/how-to-reduce-chance-of-being-blocked-while-web/) that can get you familiar with basic and more advanced approaches.


<h2 id="code_explanation">Code Explanation</h2>

Import libraries:

```python
import requests, json
from parsel import Selector
```

| Library  | Purpose   |
|----------|-----------|
| [`requests`](https://requests.readthedocs.io/en/latest/user/quickstart/)    | to make a request to the website.                                  |
| [`json`](https://docs.python.org/3/library/json.html) | to convert extracted data to a JSON object. |
| [`Selector`](https://parsel.readthedocs.io/en/latest/) | XML/HTML parser that have full [XPath](https://en.wikipedia.org/wiki/XPath) and CSS selectors support. |

At the beginning of the `main()` function, parameters and headers are defined for generating the `URL`. If you want to pass other parameters or headers to the URL, you can do so using the `params` and `headers` dictionaries: 

```python
def main():
	# https://docs.python-requests.org/en/master/user/quickstart/#passing-parameters-in-urls
	params = {
		'product_id': '16230039729797264158',	# product id
		'hl': 'en',     						# language
		'gl': 'us'	     						# country of the search, US -> USA
	}

	# https://docs.python-requests.org/en/master/user/quickstart/#custom-headers
	headers = {
	    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36'
	}

	URL = f'https://www.google.com/shopping/product/{params["product_id"]}?hl={params["hl"]}&gl={params["gl"]}'
```

Next, the `URL`, `params` and `headers` is passed to the `get_product_page_results(URL, params, headers)` function to get all data. The `product_page_results` dictionary holds the retrieved data that this function returns. At the end of the function, the data is printed out in JSON format:

```python
product_page_results = get_product_page_results(URL, params, headers)

print(json.dumps(product_page_results, indent=2, ensure_ascii=False))
```

This code uses the generally accepted rule of using the [`__name__ == "__main__"`](https://docs.python.org/3/library/__main__.html#) construct: 

```python
if __name__ == "__main__":
    main()
```

This check will only be performed if the user has run this file. If the user imports this file into another, then the check will not work. You can watch the video [Python Tutorial: if __name__ == '__main__'](https://www.youtube.com/watch?v=sugvnHA7ElY&t=1s) for more details.

Let's take a look at the `get_product_page_results(url, params, headers)` function mentioned earlier.

This function takes `url`, `params` and `headers` parameters to create a request. Now we need to parse the HTML from the [`Parsel`](https://parsel.readthedocs.io/en/latest/) package, into which we pass the `HTML` structure that was received after the request. This is necessary for successful data extraction:

```python
def get_product_page_results(url, params, headers):
	html = requests.get(url, params=params, headers=headers)
	selector = Selector(html.text)
```

Data like `title`, `low_price`, `high_price` and `description` are pretty easy to retrieve. You need to find the selector and get the value:

```python
title = selector.css('.sh-t__title::text').get()
low_price = selector.css('.KaGvqb .qYlANb::text').get()
high_price = selector.css('.xyYTQb .qYlANb::text').get()
description = selector.css('.sh-ds__trunc-txt::text').get()
```

|Code|Explanation|
|----|-----------|
|[`css()`](https://parsel.readthedocs.io/en/latest/usage.html#using-selectors)| to access elements by the passed selector.|
|[`::text` or `::attr(<attribute>)`](https://github.com/scrapy/parsel/blob/90397dcd0b2c1cbb91e44f65c50f9e11628ba028/parsel/csstranslator.py#L48-L51)| to extract textual or attribute data from the node. |
|[`get()`](https://parsel.readthedocs.io/en/latest/usage.html#usage)| to actually extract the textual data.|

Extracting `show_price` differs from the previous ones in that you need to extract the text not only from this selector, but also from those nested in it:

```python
shown_price = selector.css('.FYiaub').xpath('normalize-space()').get()
```

Data such as `reviews` and `rating` must be converted to the numeric data type. I want to draw your attention to the fact that `reviews` is retrieved in this format: `(63,413)`. To convert to a number, you need to remove the brackets and the comma:

```python
reviews = int(selector.css('.YVQvvd .HiT7Id span::text').get()[1:-1].replace(',', ''))
rating = float(selector.css('.uYNZm::text').get())
```

The `prices`, `extensions`, `media` and `highlights` lists contain multiple elements in their selector, so they are extracted using [list comprehensions](https://docs.python.org/3/tutorial/datastructures.html#list-comprehensions):

```python
prices = [price.css('::text').get() for price in selector.css('.MLYgAb .g9WBQb')]
extensions = [extension.css('::text').get() for extension in selector.css('.OA4wid')]
media = [image.css('::attr(src)').get() for image in selector.css('.sh-div__image')]
highlights = [highlight.css('::text').get() for highlight in selector.css('.KgL16d span')]
```

After extracting all the data, the `data` dictionary is formed:

```python
data = {
    'title': title,
    'prices': prices,
    'typical_prices': {
        'low': low_price,
        'high': high_price,
        'shown_price': shown_price
    },
    'reviews': reviews,
    'rating': rating,
    'extensions': extensions,
    'description': description,
    'media': media,
    'highlights': highlights
}
```

At the end of the function, the `data` dictionary is returned.

```python
return data
```

Output:

```json
{
  "title": "Sony PlayStation 5 - Standard",
  "prices": [
    "$499.00",
    "$700.00",
    "$729.00"
  ],
  "typical_prices": {
    "low": "$499.00",
    "high": "$719.75",
    "shown_price": "$499.00 at EvQ"
  },
  "reviews": 63413,
  "rating": 4.7,
  "extensions": [
    "Blu-ray Compatible",
    "4K Capable",
    "Backward Compatible",
    "Standard Edition",
    "With Motion Control",
    "Bluetooth",
    "Wi-Fi"
  ],
  "description": "Experience lightning-fast loading with an ultra-high-speed SSD, deeper immersion with support for haptic feedback, adaptive triggers and 3D audio, and a next generation of incredible PlayStation games.",
  "media": [
    "https://encrypted-tbn3.gstatic.com/shopping?q=tbn:ANd9GcSbKnqqdMH6hYKh8mzk9kje2m3KI-bRktHWihZ_LYAHQF0BNIXyfzjjusW0XMVpuUk13pFiHLVztP7Rk7GDgxBUnC6hFY84sQ&usqp=CAY",
    "https://encrypted-tbn1.gstatic.com/shopping?q=tbn:ANd9GcTa0aWvl4ZCffiyfM3sBvdYLk1K8SkMIo6ZkmN3ASkW7GPgVmB_XMOFCBgmW-AMOspQ9KFLJjKN9uPZbj0ScCVOizsmX8Fegg&usqp=CAY",
    "https://encrypted-tbn2.gstatic.com/shopping?q=tbn:ANd9GcRJRtfshsdgf4JJGzS-QzvYXjzOy4NKV-y_0yQn-W6n109ziyqOzTvDcX-YXNmr3rPu4cHKpo7OVV2fkDzodE7LK6Pxh63l&usqp=CAY",
    "https://encrypted-tbn3.gstatic.com/shopping?q=tbn:ANd9GcS6lCLgdUU42DbmP2Y8o5MPMHF_j1LFpMvdBHNTPLIfBOn8bnpC-xPBYl14wDMiPK7lQ1YL_BEeOm5vqVmfJpBLnOomYoXy&usqp=CAY"
  ],
  "highlights": [
    "Integrated I/O: Marvel at incredible graphics and experience new PS5 features.",
    "Ultra-high speed SSD: Maximize your play sessions with near-instant load times for installed PS5 games.",
    "HDR technology: With an HDR TV, supported PS5 games display an unbelievably vibrant and lifelike range of colors.",
    "8K output: PS5 consoles support an 8K output, so you can play games on your 4320p resolution display.",
    "4K TV gaming: Play your favorite PS5 games on your stunning 4K TV. Up to 120 fps with 120Hz output"
  ]
}
```


<h2 id="google_product_page_api">Using Google Product Page API from SerpApi</h2>

This section is to show the comparison between the DIY solution and our solution.

The main difference is that it's a quicker approach. [Google Product Page API](https://serpapi.com/product-page) will bypass blocks from search engines and you don't have to create the parser from scratch and maintain it.

First, we need to install [`google-search-results`](https://pypi.org/project/google-search-results/):

```lang-none
pip install google-search-results
```

Import the necessary libraries for work:

```python
from serpapi import GoogleSearch
import os, json
```

Next, we write a search query and the necessary parameters for making a request:

```python
params = {
    # https://docs.python.org/3/library/os.html#os.getenv
    'api_key': os.getenv('API_KEY'),        # your serpapi api
    'engine': 'google_product',             # SerpApi search engine	
    'product_id': '16230039729797264158',   # product id
    'hl': 'en',                             # language
    'gl': 'us'                              # country of the search, US -> USA
}
```

We then create a `search` object where the data is retrieved from the SerpApi backend. In the `results` dictionary we get data from JSON:

```python
search = GoogleSearch(params)   # where data extraction happens on the SerpApi backend
results = search.get_dict()     # JSON -> Python dict
```

The data is retrieved quite simply, we just need to turn to the `'product_results'` key.

```python
product_results = results['product_results']
```

Example code to integrate:

```python
from serpapi import GoogleSearch
import os, json

params = {
    # https://docs.python.org/3/library/os.html#os.getenv
    'api_key': os.getenv('API_KEY'),        # your serpapi api
    'engine': 'google_product',             # SerpApi search engine	
    'product_id': '16230039729797264158',   # product id
    'hl': 'en',                             # language
    'gl': 'us'                              # country of the search, US -> USA
}

    
search = GoogleSearch(params)               # where data extraction happens on the SerpApi backend
results = search.get_dict()                 # JSON -> Python dict

product_results = results['product_results']

print(json.dumps(product_results, indent=2, ensure_ascii=False))
```

Output:

```json
{
  "product_id": 16230039729797264158,
  "title": "Sony PlayStation 5 - Standard",
  "prices": [
    "$499.99",
    "$499.00",
    "$700.00"
  ],
  "conditions": [
    "New",
    "New",
    "New"
  ],
  "typical_prices": {
    "low": "$499.00",
    "high": "$719.75",
    "shown_price": "$499.99 at Gamestop"
  },
  "reviews": 63413,
  "rating": 4.7,
  "extensions": [
    "Blu-ray Compatible",
    "4K Capable",
    "Backward Compatible",
    "Standard Edition",
    "With Motion Control",
    "Bluetooth",
    "Wi-Fi"
  ],
  "description": "Experience lightning-fast loading with an ultra-high-speed SSD, deeper immersion with support for haptic feedback, adaptive triggers and 3D audio, and a next generation of incredible PlayStation games.",
  "media": [
    {
      "type": "image",
      "link": "https://encrypted-tbn3.gstatic.com/shopping?q=tbn:ANd9GcRoN7Gg6r9ZxPZGkfTEbukowBuBvalGRrJG44Dwnw8_PAmLUNjt&usqp=CAY"
    },
    {
      "type": "image",
      "link": "https://encrypted-tbn1.gstatic.com/shopping?q=tbn:ANd9GcQOuj8omxssTbuSixiKmldKmSOCllkb1jLSqYHbThqgR3l78gjS&usqp=CAY"
    },
    {
      "type": "image",
      "link": "https://encrypted-tbn2.gstatic.com/shopping?q=tbn:ANd9GcQmw7DOYYmm5nQSQoEhAaE78a5IyNW3tHoCE1VRI2cxTHn9QGg&usqp=CAY"
    },
    {
      "type": "image",
      "link": "https://encrypted-tbn3.gstatic.com/shopping?q=tbn:ANd9GcRsejj3qFlCeXGkvHMG7yGdM6gR_AbzoT_fWZUcYrhS3QKxpHI&usqp=CAY"
    }
  ],
  "highlights": [
    "Integrated I/O: Marvel at incredible graphics and experience new PS5 features.",
    "Ultra-high speed SSD: Maximize your play sessions with near-instant load times for installed PS5 games.",
    "HDR technology: With an HDR TV, supported PS5 games display an unbelievably vibrant and lifelike range of colors.",
    "8K output: PS5 consoles support an 8K output, so you can play games on your 4320p resolution display.",
    "4K TV gaming: Play your favorite PS5 games on your stunning 4K TV. Up to 120 fps with 120Hz output"
  ]
}
```

<h2 id="links">Links</h2>

- [Code in the online IDE](https://replit.com/@chukhraiartur/blog-google-shopping-product-page#parcel_solution.py)
- [Google Product Page API](https://serpapi.com/product-page)


<p style="text-align: center;">Join us on <a href="https://twitter.com/serp_api">Twitter</a> | <a href="https://www.youtube.com/channel/UCUgIHlYBOD3yA3yDIRhg_mg">YouTube</a></p>

<p style="text-align: center;">Add a  <a href="https://github.com/serpapi/public-roadmap/issues">Feature Request</a>💫 or a <a href="https://github.com/serpapi/public-roadmap/issues">Bug</a>🐞</p>