Scraping Amazon Product Reviews
Published: 2024-01-11 18:30:59

To read more articles like this, [visit my blog](https://www.mdfaisal.com/blog)


In this age of data, web scraping is the unsung hero. It’s a way to extract information from the vast web. The knowledge of web-scraping comes in handy in many scenarios.

Today, we will **learn web scraping from scratch**. We will get familiar with the concepts and the challenges.

Here is what we will do step by step.

 1. Set up a Python project following all the **best practices**.

 2. Understand how to think like a scraper and use the tools.

 3. How to do **authentication**.

 4. How to **extract** data from a web page.

 5. How to **export** the data.

At the end of the article, we will have a Python project to scrape any Amazon product’s page and get the review data.

Our target URL is the following.

[**ENHANCE Gaming Headphone Stand Headset Holder with 4 Port USB Hub, Customizable LED Lighting…**](https://www.amazon.com/ENHANCE-Headphone-Customizable-Lighting-Flexible/dp/B07DR59JLP/)

### Pre Requisites

To follow this article, you must be familiar with Python.

But if you are not, don’t worry. You still can follow. The concepts are the same, and similar tools exist in all languages.

Let’s begin!

## 1. Create a New Project Directory

First, create a new directory for the project.

```
mkdir scrape-web-data
    
cd scrape-web-data
```

## 2. Initialize a Virtual Environment

A virtual environment is a way to isolate the dependencies of a project.

It helps us to resolve the conflicts in various versions of the dependencies.

Let’s create one.

```
python3 -m venv venv
    
# On Windows: python -m venv venv
```

### Activate the virtual environment:

Creating a virtual environment is not enough; you must activate it before using it.

Let’s do that.

```
source venv/bin/activate
    
# On Windows: venv\Scripts\activate
```

If you inspect the terminal, you will notice a `(venv)` prefix on the second line.

![Virtual environment indi](https://cdn-images-1.medium.com/max/3080/1*QsEeOgsQG3eFGvupDt7Jrw.png)

It means our virtual environment is activated!

If you want to exit from the virtual environment, Just type

```
deactivate
```

![Exit virtual environment](https://cdn-images-1.medium.com/max/2820/1*I08N_3K-2enTupd_p8W6Gw.png)

That’s it!

## 3. Install Required Packages

As a start, we will use a single library

### Requests

This is a library to make HTTP requests. It has a simple interface, and calling an external API is as simple as

```
import requests
    
response = requests.get("URL")
```

To install these libraries, run the following

```
pip3 install requests 
```

### Optionally

You can add these dependencies to a permanent `requirements.txt` file. So, when you share the code with others, they will know exactly what to install.

```
pip3 freeze > requirements.txt
```

If you run the following command, it will create a `requirements.txt` file inside the project directory.

If you share the project with someone else, they must run it.

```
pip3 install -r requirements.txt
```

This makes it easy to share dependencies between teams.

## Enough chit-chat! Let’s code

Now comes the fun part. Let’s create our scraper file.

```
touch scraper.py
```

And open it in your favorite editor.

## First Things First

The first step to being a web scraper is understanding where the data comes from.

You see the following when you visit a web page like this [product page](https://www.amazon.com/ENHANCE-Headphone-Customizable-Lighting-Flexible/dp/B07DR59JLP/).

![Amazon product page](https://cdn-images-1.medium.com/max/3808/1*h-306pwTXvGABPAl1-_clA.png)

Now, this is designed for humans (Like you!). You can capture the information through your eyes.

However, our machines don’t have eyes, so they only see some HTML code.

You can see the underlying code

* By hitting the **F12** key on your keyboard. or,

* Right-click and **inspect**

You will see something like this when you do that,

![HTML code for the page.](https://cdn-images-1.medium.com/max/3546/1*d-fYSxgKesxsyu9RN9L3Ew.png)

Now, If you want the product's title, there is a title for that. Just right-click on the title and inspect.

It will take you right to the exact thing you are looking for.

![Product Title](https://cdn-images-1.medium.com/max/3546/1*iZ3HdcDah3mHgV1zh736gA.png)

Nice!

Notice that the product title has some distinct properties.

* This is an `<span>` element.

* It has an id `productTitle`

* It has a class `ofa-size-large product-title-word-break`

It is essential to understand the different characteristics of the HTML elements.

Later, you will see how we utilize these properties to grab the desired data.

## Game Plan!

Now that we have a good idea about what we are dealing with let’s see how we can get the data.

Let’s assume we want to get the product title of any Amazon product link.

 1. The first step is to get the **HTML** content

 2. The second step is to extract the data.

### Step 1: Get the HTML content

When you visit any website, the browser makes an HTTP request, and the server sends back the HTML code.

We will do the same for our product page. And you will see how easy it is to get the HTML data using the `requests` library.

```js
import requests

product_link = "https://www.amazon.com/ENHANCE-Headphone-Customizable-Lighting-Flexible/dp/B07DR59JLP/"

response = requests.get(product_link)

print(response.text)
```

Here, we are hitting the URL and printing the response.

Below is a small part of what we get back.

![HTML response](https://cdn-images-1.medium.com/max/2054/1*KW1EsZbPK8FmW2Ieu0fntw.png)

We got back the HTML code. So far, so good.

### But wait, We have a problem!

Look closer and see the `title` tag.

```
<title>Sorry! Something went wrong!</title>
```

If you analyze the returned HTML response, you will also find

```
<!--
To discuss automated access to Amazon data
please contact api-services-support@amazon.com.
.....
-->
```
Well, we got a 200 response, but Amazon detected that we were calling the API using a bot and not as a human.

It’s a very common practice by organizations to prevent automated access to data, and as a web scraper, you have to learn how to work around them.

### The solution — Headers

In this case, the problem is that Amazon treats us as a bot.

This problem can be easily solved by passing some extra headers with our request.

But where to get that?

Let’s head back to the browser and open the network tab.

![Browsers Network tab](https://cdn-images-1.medium.com/max/3720/1*pSMF-0ExVIpm15F-KQyBDw.png)

Refresh the page and click on the first request. And Then open the `Headers` tab.

Under the Headers, you will find the `User Agent` header that is being passed to the backend.

This way, Amazon identifies where the request is coming from.

We will do the same and update our code.

```
import requests
    
product_link = "https://www.amazon.com/ENHANCE-Headphone-Customizable-Lighting-Flexible/dp/B07DR59JLP/"
user_agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36"
    
human_headers = {
  'user-agent': user_agent,
}
response = requests.get(product_link, headers=human_headers)
    
print(response.text)
```

We pass some custom headers in the above code to identify ourselves as human.

```
user_agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36"
    
human_headers = {
  'user-agent': user_agent,
}
```

And then passing the headers using the `requests` library.

```
response = requests.get(product_link, headers=human_headers)
```

Now run the code,

```
python3 scraper.py
```

And you will get the complete HTML!

Nice. So we can get product page data. This is currently in the HTML format, but we can deal with that later.

## Take it to the next level.

The page you just scraped was public. So we didn’t have to handle any authorization or authentication.

But that’s not always the case.

What if you want some data that is only available after logging in?

### Understanding Authentication

Different companies have different ways of handling authentication.

Amazon uses email and password authentication. So, if you want to log in, you have to provide the `email and the password field`.

But wait, How do you do that? So far, we only used the `requests` library that can handle HTTP calls, but how can we deal with user inputs?

### Enter Automated Browsers

Automated browsers are a way to mimic a user's interaction without any human interacting directly with it.

It can work like an assistant.

That means we can make it behave like a normal browser and simulate user behavior.

The most popular tool for that is `selenium` .

## Let’s install the dependencies.

First, install the required dependencies.

```
pip3 install selenium undetected_chromedriver 
```

Here,

* **Selenium:** This is our headless browser and

* **Undetected Chromedriver**: This is a library that will help us to avoid bot detection.

### Login Process

As mentioned earlier, we will simulate a human login behavior. So, let’s analyze the Login process and gather information.

1 . First go to your Amazon account and click **Sign in** at the top right corner.

![Sign in](https://cdn-images-1.medium.com/max/2412/1*Y7BcFb_QiXGu2otVG6zkNg.png)

2. Go to the Sign-in page and copy the URL.

3. Grab the **Email** element and copy the id.

![Email field](https://cdn-images-1.medium.com/max/2874/1*ggmFBwaxdgvEhmLjFSUVUg.png)

4. Grab the **Continue** button element and copy the id

![Continue button](https://cdn-images-1.medium.com/max/2874/1*4ekrY9LegblkEfNtfoRyLg.png)

5. Click the continue button and go to the next page. do the same for the **Password** field and the **Sign-In** button

![Password field](https://cdn-images-1.medium.com/max/2874/1*9GvvzHZ0rNMknIciCkumow.png)

So now we have all the data we need to sign in. Let’s code it!

You will do the following.

 1. Gather all necessary details (We already did that above)

 2. Create a browser instance

 3. Navigate between the pages and fill out the proper credentials

 4. Get the result back!

First, create a new file named. login.py

```
touch login.py
```

We will create a browser instance, grab the correct elements, fill out the proper information, and try to sign in.

Below is the required code.

```js
import undetected_chromedriver as uc
import time
from selenium.webdriver.common.by import By
from selenium import webdriver
    
# Create a web browser instance
chromeOptions = uc.ChromeOptions()
chromeOptions.headless = False
driver = webdriver.Chrome(options=chromeOptions)

product_link = "https://www.amazon.com/ENHANCE-Headphone-Customizable-Lighting-Flexible/dp/B07DR59JLP/"

login_url = "YOUR_LOGIN_URL"
user_email = "YOUR_EMAIL"
user_password = "YOUR_PASSWORD"

email_input_id = "ap_email"
password_input_id = "ap_password"

continue_button_id = "continue"
sign_in_button_id = "signInSubmit"

# visit the login url
driver.get(login_url)
time.sleep(5)

# get the email input and fill up
email = driver.find_element(By.ID, email_input_id)
email.send_keys(user_email)

# get the continue button and click
driver.find_element(By.ID, continue_button_id).click()

# wait for navigation
time.sleep(5)

# get the password and fill up
password = driver.find_element(By.ID, password_input_id)
password.send_keys(user_password)

# get the sign in button and click
driver.find_element(By.ID, sign_in_button_id).click()

time.sleep(10)

# visit product url
driver.get(product_link)

# Extract the full HTML of the page and store it in the 'page_html' variable
page_html = driver.page_source

print(page_html)

# Close the WebDriver instance
driver.close()
```

Let’s run the code.

```
python3 login.py
```

You will see a browser pop up, simulate the whole behavior in front of your eyes, and return the HTML page content.

```
THE HTML OUTPUT IS TOO LONG TO SHOW HERE
```

However, sometimes, you will get caught in the bot detection and see the following page.

![Captcha page](https://cdn-images-1.medium.com/max/6880/1*pNPYTEJtN5i-ZofHs75A_w.png)

Bypassing the captcha is not as straightforward as we have been doing so far. Because we can’t just get an ID and type whatever we like.

There are some paid tools to

## Extract the data

So Now we know how to get the HTML content of a page.

Now, we need to extract the required data from the HTML content.

To do that, we will use `beautiful soup` library.

### Beautiful Soup

This is a library to parse HTML data quickly.

To install,

```
pip3 install beautifulsoup4
```

Then, using it is as simple as

```
import BeautifulSoup

soup = BeautifulSoup("SOME_HTML_CODE", 'html.parser')
```

### Extract product review data

We need to analyze the HTML to find the proper identifiers to find the product review data.

![Review information](https://cdn-images-1.medium.com/max/2710/1*JF-xZl2GWSQcUnf39Y8hMg.png)

Here each review

* The title is inside a `span` and have a class of `a-profile-name`

* The review is inside a span under a div with a class named `review-title-content`

* The date is inside a class named `review-date`

Let’s use these pieces of information to get our desired values.

```
# here page_html is whatever we received from the previous steps
soup = BeautifulSoup(page_html, 'html.parser')

# get all span elements with the desired class
user_names = soup.find_all('span', class_="a-profile-name")

# loop over each 
for user_name in user_names:
    print(user_name.text)
```

It will give you the following output

![User names](https://cdn-images-1.medium.com/max/2000/1*Tzjc839ORI3XiqnJw2cxKQ.png)

Nice! Now we know how to get the user names. We will do the same for the `review` and title; the following is our final code.

```
# parse the data
soup = BeautifulSoup(page_html, 'html.parser')

# grab desired elements
review_users = soup.find_all('span', class_="a-profile-name")
all_review_divs = soup.find_all('div', class_='review-text-content')
review_dates = soup.find_all('span', class_="review-date")

# total reviw count
num_reviews = len(all_review_divs)
reviews = []

for i in range(0, num_reviews):
    reviews.append([review_users[i].text, all_review_divs[i].find('span').text,review_dates[i].text])

print(reviews)
```

Now, you will get the following output

![Reviews in array format](https://cdn-images-1.medium.com/max/3744/1*SMLSAo2bzNH7bMR08uGQ5g.png)

So, as you can see, we have all the review data available to us.

## Export the Data

Now, as we already have the data. Let’s export it.

Exporting the data to CSV format is easy using the CSV package. Let’s first install it

```
pip3 install csv
```

Then, we will use the package to iterate over the `reviews` array and write the content to the file.

```
import csv

# scraping logic...

# csv file name
csv_file_path = "reviews.csv"

# Write the data to the CSV file
with open(csv_file_path, 'w', newline='') as csv_file:
    csv_writer = csv.writer(csv_file)

    # Write each row from the data array
    for row in reviews:
        csv_writer.writerow(row)

print(f"CSV file '{csv_file_path}' has been created.")
```

You will see

![csv file generation](https://cdn-images-1.medium.com/max/2000/1*owcoI4O8ESt26Sj7JV61Lw.png)

You will also see a reviews.csv file generated in the same directory. After opening with any editor, you can view the result.

![csv output](https://cdn-images-1.medium.com/max/2000/1*tqqvkBGDXr_KsbQINcSNOQ.png)

This is great. We have successfully scraped the product page and exported the data.

Thank you for reading so far! I hope you have a wonderful day!


**Have something to say? Get in touch with me via [LinkedIn](https://www.linkedin.com/in/56faisal/) or [Personal Website](https://www.mdfaisal.com/)**