Scrape E-Commerce Site using BeautifulSoup and store Data in csv file.
Tags: python, scrape, beautifulsoup, websitescrap
Published: 2022-08-21 10:12:00

**Scrape Flipkart website using [beautifulsoup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) and store data in csv file.**

IDE Used: [PyCharm](https://www.jetbrains.com/pycharm/)

First install all packages one by one

```
pip install bs4
pip install requests
pip install pandas
```

Import all modules

```
import requests
from bs4 import BeautifulSoup
import pandas as pd
```

`requests` to send HTTP request.
`BeautifulSoup` extract all content from web page (title, etc.)
`pandas` to make csv file.

## Step 1: Loading web page with 'request'

`request` module allows to send HTTP requests. The HTTP request returns a Response object with all the response data (content, encoding, etc.)

```
url = "https://www.flipkart.com/search?q=laptop&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off"

req = requests.get(url)
```

## Step 2: Extracting title with BeautifulSoup

BeautifulSoup provides lot of simple methods for navigating, searching, and modifying a DOM tree.

Extract the title
```
page = BeautifulSoup(req.content, "html.parser")
print(page.prettify())

title = page.head
print(title.text)
```

`page.prittify()` print all the content of page into a nicely format.

![prittify](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ofgt0xwrtedgglbe14us.png)

`title.text` print title of the page.

![title](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2tf7p6eabh7hump1om8g.jpg)

print without calling `.text` it will print full markup.


![title](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9n3b195ki7nx4a76ucw1.jpg)

```
all_products = []
```

Add all extract data in this `all_products` list. Print this list at the end.

## Step 3: Start scrapping

```
products = page.findAll("div", {"class": "_3pLy-c row"})
```

`findAll` is used for returning all the matches after scanning the entire document. 

`findAll(tag, attributes, recursive, text, limit, keywords)`

In our case find all `div` tags with `class` name "_3pLy-c row"

`.select()` it returns a list of elements. The `select()` method to locate all elements of a particular `CSS` class. It find elements by Attr., ID, Class, etc.

`.strip()` method use to remove any extra newlines/whitespaces.

## Step 4: Extract Name and Price of products

```
for product in products:
    lname = product.select("div > div._4rR01T")[0].text.strip()
    print(lname)
    price = product.select("div > div._30jeq3 ")[0].text.strip()
    print(lprice)

    all_products.append([lname, lprice])

print("Record Inserted Successfully...")
```

`all_products.append` append `Name` and `Price` in `all_products` list.


![all_products](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8hrl6zscdui7jk8ox7t1.jpg)

```
print(all_products)
```

![img](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nffcykrt62frinw8kjty.jpg)

## Step 5: Add Data in CSV file.

```
col = ["Name", "Price"]
```
First create list for column name.

```
data = pd.DataFrame(all_products, columns=col)
print(data)
```
`DataFrame` It is a 2 dimensional data structure, like a 2 dimensional array, or a table with rows and columns.

![csv](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ihyqky6he256vee3231l.jpg)

```
data.to_csv('extract_data.csv', index=False)
```
`to_csv` used to export pandas `DataFrame` to csv file.


![img](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wk83lr3lfbdecx7tzzc9.jpg)

### Code

```
import requests
from bs4 import BeautifulSoup
import pandas as pd

url = 'https://www.flipkart.com/search?q=laptop&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off'
req = requests.get(url)

page = BeautifulSoup(req.content, "html.parser")
print(page.prettify())

title = page.head
print(title.text)

all_products = []
col = ['Name', 'Price']

products = page.findAll("div", {"class": "_3pLy-c row"})
print(products)

for product in products:
    lname = product.select("div > div._4rR01T")[0].text.strip()
    print(lname)
    price = product.select("div > div._30jeq3 ")[0].text.strip()
    print(lprice)

    all_products.append([lname, lprice])

print("Record Inserted Successfully...")

data = pd.DataFrame(all_products, columns=col)
print(data)
data.to_csv('laptop_products.csv', index=False)
```

[GitHub](https://github.com/SaishJ/Flipkart-website-scrape)

**Thank You 😊**