The Easiest Way to Scrape Amazon Products and Reviews
Tags: webscraping, tutorial
Published: 2021-10-21 14:07:22

## Introduction   




**Amazon** is a vast Internet-based enterprise that sells all kinds of goods either directly or as the middleman between other retailers and **Amazon's** millions of customers.


You can scrape their data to run different types of tasks such as:


- competitor analysis
- improving your products and value proposition
- identifying market trends and what influences them
- price monitoring   



Luckily, **Amazon** is a website that is pretty easy to scrape if you have the right tools.

For this purpose, we will use [Page2API](https://www.page2api.com/) - a powerful and delightful API that makes web scraping easy and fun.


In this case, we will scrape 2 types of amazon pages:

- Search results page
- Product reviews page   



## Prerequisites



To perform this simple task, you will need the following things:


-  [A Page2API account](https://www.page2api.com/users/sign_up/)    

The free trial offers a credit that covers up to **1000** web pages to scrape, and it takes under **10** seconds to create the account if you sign up with Google.

- A product or a category of products that we are about to scrape.   

In our case, we will search for **'luminox watches'**
and then scrape the reviews for a random product.   


## Scraping all products from the search page   



First what we need is to type **'luminox watches'** into the search input from amazon's search page.


This will change the browser URL to something similar to:
  
```
https://www.amazon.com/s?k=luminox+watches
``` 

The URL is the first parameter we need to perform the scraping.

The page that you see must look like the following one:

![Amazon search page](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zd7xmt3pr7mhfk05x6d9.png)


If you inspect the page HTML, you will find out that a single result is wrapped into a div that looks like the following:

![Amazon search result](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/uo6nvtst0jteke0j3emm.png)


The HTML for a single result element will look like this:

![Search result HTML breakdown](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vs8plri6ju4udowqnn2f.png)



The last part is the pagination handling.

In our case, we must click on the Next → button while the list item's class will be active:

![Next page enabled](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3ykdu2ci404hgktsgpid.png)


And stop our scraping request when the **Next →** button became disabled.

In our case, a new class (**a-disabled**) is assigned to the list element where the button is located:

![Next page disabled](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8l12sibqpu2y2rux9jwr.png)


Now it's time to prepare the request that will scrape all products that the search page returned.   
   


**Setting the api_key as an environment variable**
```bash
export API_KEY=YOUR_PAGE2API_KEY
```     
  

**Running the scraping request with cURL**
```json
    curl -v -XPOST -H "Content-type: application/json" -d '{
      "api_key": "'"$API_KEY"'",
      "url": "https://www.amazon.com/s?k=luminox+watches",
      "real_browser": true,
      "merge_loops": true,
      "scenario": [
        {
          "loop" : [
            { "wait_for": ".a-pagination li.a-last" },
            { "execute": "parse" },
            { "execute_js": "document.querySelector(\".a-pagination li.a-last a\").click()" }
          ],
          "stop_condition": "document.querySelector(\".a-last.a-disabled\") !== null"
        }
      ],
      "parse": {
        "watches": [
          {
            "_parent": "[data-component-type='s-search-result']",
            "title": "h2 >> text",
            "link": ".a-link-normal >> href",
            "price": ".a-price-whole >> text",
            "stars": ".a-icon-alt >> text"
          }
        ]
      }
    }' 'https://www.page2api.com/api/v1/scrape' | python -mjson.tool
  ```

**The result**
  ```json
  {
    "result": {
      "watches": [
        {
          "title": "Men's Luminox Leatherback Sea Turtle 44mm Watch",
          "link": "https://www.amazon.com/Luminox-Leatherback-Turtle-Giant-Black/dp/B07CVFWXMR/ref=sr_1_2?dchild=1&keywords=luminox+watches&qid=1634327863&sr=8-2",
          "price": "$223.47",
          "stars": "4.5 out of 5 stars"
        },
        {
          "title": "The Original Navy Seal Mens Watch Black Display (XS.3001.F/Navy Seal Series): 200 Meter Water Resistant + Light Weight Case + Constant Night Visibility",
          "link": "https://www.amazon.com/Luminox-Wrist-Watch-Navy-Original/dp/B07NYXV77C/ref=sr_1_3?dchild=1&keywords=luminox+watches&qid=1634327863&sr=8-3",
          "price": "$254.12",
          "stars": "4.3 out of 5 stars"
        },
        {
          "title": "Leatherback SEA Turtle Giant - 0323",
          "link": "https://www.amazon.com/Luminox-Leatherback-SEA-Turtle-Giant/dp/B07PBC31N8/ref=sr_1_4?dchild=1&keywords=luminox+watches&qid=1634327863&sr=8-4",
          "price": "$179.00",
          "stars": "4.3 out of 5 stars"
        },
        ...
      ]
    },
    ...
  }
```
  

## Scraping product reviews   


First what we need is to click on the **See all reviews** link from the Product page.

This will change the browser URL to something similar to:

```
https://www.amazon.com/product-reviews/B072FNJLBC
```

The URL is the first parameter we need to perform the reviews scraping.

The HTML from a single review will look like this:

![Amazon review HTML breakdown](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7ho664sj4k0mymv09oq7.png)


Luckily, the pagination handling is similar to the one described above, so we will use the same flow.

Now it's time to prepare the request that will scrape all reviews.


**Setting the api_key as an environment variable (if needed)**
```bash
export API_KEY=YOUR_PAGE2API_KEY
```

**Running the scraping request with cURL**
  ```json
    curl -v -XPOST -H "Content-type: application/json" -d '{
      "api_key": "'"$API_KEY"'",
      "url": "https://www.amazon.com/product-reviews/B072FNJLBC",
      "real_browser": true,
      "merge_loops": true,
      "scenario": [
        {
          "loop" : [
            { "wait_for": ".a-pagination li.a-last" },
            { "execute": "parse" },
            { "execute_js": "document.querySelector(\".a-pagination li.a-last a\").click()" }
          ],
          "stop_condition": "document.querySelector(\".a-last.a-disabled\") !== null"
        }
      ],
      "parse": {
        "reviews": [
          {
            "_parent": "[data-hook='review']",
            "title": ".review-title >> text",
            "author": ".a-profile-name >> text",
            "stars": ".review-rating >> text",
            "content": ".review-text >> text"
          }
        ]
      }
    }' 'https://www.page2api.com/api/v1/scrape' | python -mjson.tool
  ```


**The result**
```json 
  {
    "result": {
      "reviews": [
          {
            "title": "Great watch & easy to read in low light conditions",
            "author": "Paul E. Papas",
            "stars": "5.0 out of 5 stars",
            "content": "I'm a 60+ year old equestrian and outdoorsman. I was looking for a watch that could take the shock of firearm discharge ..."
          },
          {
            "title": "Not Water Resistant, impossible to get amazon help",
            "author": "Benjamin H. Curry",
            "stars": "2.0 out of 5 stars",
            "content": "This watch has a 2 year warranty from date of purchase however after not even on full year my adult son went swimming with it ..."
          },
        ...
      ]
    },
    ...
  }
```
  

## Conclusion


That's pretty much of it!   
As mentioned, **Amazon** is a website that is pretty easy to scrape if you have the right tools.
This is the place where  [Page2API](https://www.page2api.com/)  shines, making web scraping super easy and fun.   

The original article can be found here:      
[https://www.page2api.com/blog/scrape-amazon-product-data](https://www.page2api.com/blog/scrape-amazon-product-data/)