Web Scraping Multiple Pages using Python & BeautifulSoup
Tags: python, tutorial, beginners, webscraping
Published: 2022-01-27 06:26:59

## Web Scraping
Web Scraping is used to examine websites for Unstructured Data and store it Structurally for our use. Web Scraping helps us iterate over several web pages and extract the required information, It is then stored in a format suitable for the user.

In this project, we will learn **'How to use Python and Beautiful Soup to scrape Mobile Phone Names and Prices of Brand Mi on a Flipkart E-commerce Website'**.

[Click here for a code](https://github.com/sudhanshumukherjeexx/WebScrapingFlipkart)

### Inspecting the Element
The objective of this project is to scrape Flipkart's Website and extract the 'Name' and 'Prices' of Mi Mobile Phones.
Get the URL that will be needed to make a request.

The URL which we will be using: _https://www.flipkart.com/mobiles/mi~brand/pr?sid=tyy%2C4io&otracker=nmenu_sub_Electronics_0_Mi&page=_

Now, It is very necessary to identify the right class of the elements stored which we need to scrape. To inspect elements follow these steps.
Step 1 - Visit the URL
Step 2 - Right on the website and select _inspect_ or press _Ctrl_ + _shift_ + _I_ together.
Step 3 - Hover on the name of the phone and click it. Select the class from the window appearing on the right.
Step 4 - Apply the same process for price.
Step 5 - Copy this class somewhere, we will need it later in our code.
![Blue Highlighted Text on the right determines class of phone name](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8dx3hc43023h3lshx75c.PNG)

### Import Libraries, modules, and Packages
Let's import the required libraries and packages first. To begin with, I will be using Beautiful Soup for parsing HTML documents which can later be used to extract data from HTML. Then, We will import 'requests' which allows us to interact with the web, It contains many useful features and methods to make HTTP requests.
Lastly, we will need pandas to store our scraped data in an organized way. 

`from bs4 import BeautifulSoup
import requests
import pandas as pd`

### Write the program

- Create an empty list for Phone Name and Phone Price.
`phone_name = []
phone_price = []`

- Declare a variable 'page_num' which will help us take input from users about the number of pages they want to scrape.
`page_num = int(input("Enter number of pages:"))`

- Now we will use _for Loop_ to iterate over multiple pages looking for relevant information and operations to implement.
- Then, Let's store our _URL_ in the variable _url_ and make _requests_ to build a connection. Don't forget to add(+) _str(i)_. This will help our program to iterate over multiple pages while using for loop.
- After, making a request, let's use Beautiful Soup for parsing HTML and store all the data in a variable called _content_.

![Code](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7xqd6pf3y6grllv77asa.png)

- Declare a variable called _name_ to find and store all _div_ and the relevant _class_(which we copied using inspect element) of phone name.
- Declare a variable called _price_ to find and store all _div_ and the relevant _class_(which we copied using inspect element) of phone price.

![Code](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rik626wcrm6be8ty1325.png)

- Let's append the data extracted in the empty list which we have created at the start.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n6q7ldg0rjnsemopwncs.png)

- Now, let's create a DataFrame to store our data structurally and export DataFrame into _CSV_ file.

![Code](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2j7v0i2u9ouebfitnge4.png)

This is our Aha moment folks. We have successfully created our web scraper using python and Beautiful Soup. To get a better understanding of this, Use a website that you personally like and try scraping it.