Simple Web scraping project using python and Beautiful soup
Tags: beginners, tutorial, python, datascience
Published: 2022-01-24 06:53:04


![Web scraping a shopping site](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ev5hmr5bfke1kre7v14i.png)
  
 
##Introduction

Web scraping is an automated method used to extract large amounts of data from websites. The data on the websites are unstructured. Web scraping helps collect these unstructured data and store it in a structured form.

In this project I will show you how to scrape data from a Kenyan website called Jumia https://www.jumia.co.ke/. The data we gather can be used for price comparison.

**Website Inspection**

The aim of this project is to scrape all products, their prices and rating. So first, we need to inspect the website, this is done by:

1.Visiting this site https://www.jumia.co.ke/all-products/

2.Right clicking and selecting **inspect** or clicking **ctrl+shift+i** to inspect the website.
![Inspect](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fa7gdo723ii1wkkpzi2t.png)
3.Move the cursor around till a product is selected.Then search for the div tag that has the name, price and rating of the product.
![Web scraping](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hkh5ifau6o06cyy382bg.png)

**Write the code**
We start by importing the necessary libraries
``` python
from bs4 import BeautifulSoup
import requests
```
The requests library will make a GET request to a web server, which will download the HTML contents of a given web page for us.
``` python
jumia = requests.get('https://www.jumia.co.ke/all-products/')
```
**Parsing a page using BeautifulSoup**
```python
soup = BeautifulSoup(jumia.content , 'html.parser')
products = jsoup.find_all('div' , class_ = 'info')
```
Use the find_all method, which will find all the instances of the div tag that has a class called 'info' on the page.

We now extract the name, price and rating.If you want to find the first instance of a tag, you can use the find method, which will return a single BeautifulSoup object:
```python
Name = product.find('h3' , class_="name").text.replace('\n', '')
Price = product.find('div' , class_= "prc").text.replace('\n', '')
Rating = product.find('div', class_='stars _s').text.replace('\n', '')
```
replace() is an inbuilt function in the Python programming language that returns a copy of the string where all occurrences of a substring are replaced with another substring. 

We can now loop over all products on the page to extract the name, price and rating.
```python
for product in products:
      Name = product.find('h3' , class_="name").text.replace('\n', '')
      Price = product.find('div' , class_= "prc").text.replace('\n', '')
      Rating = product.find('div', class_='stars _s').text.replace('\n', '')
      
      info = [ Name, Price,Rating]
      print(info)
```
Note that we are storing all these in a list called info.

**Loop over all pages**
We have only scraped data from the first page. The site has 50 pages and when you click on the second page you notice that the url changes. So to get the new url we do this:
```python
url = "https://www.jumia.co.ke/all-products/" + "?page=" +str(page)+"#catalog-listing"
```
That is a simple string concatination. The code to loop through all the pages is:
```python
for page in range(1,51):
  url = "https://www.jumia.co.ke/all-products/" + "?page=" +str(page)+"#catalog-listing"
  furl = requests.get(url)
  jsoup = BeautifulSoup(furl.content , 'html.parser')
  products = jsoup.find_all('div' , class_ = 'info')

  for product in products:
      Name = product.find('h3' , class_="name").text.replace('\n', '')
      Price = product.find('div' , class_= "prc").text.replace('\n', '')
      try:
        Rating = product.find('div', class_='stars _s').text.replace('\n', '')
      except:
        Rating = 'None'
      
      info = [ Name, Price,Rating]
      print(info)
```
range() function goes up to but doesn't include the last number. The website has 50 pages this range is up to 51.
Since some of the products have no ratings, we put it between try catch clause and print None in that instance.

**Saving to csv** 
```python
df = pd.DataFrame({'Product Name':Name,'Price':Price,'Rating':Ratings}) 
df.to_csv('products.csv', index=False, encoding='utf-8')
```

**The whole code**
```python
from bs4 import BeautifulSoup
import requests

for page in range(1,51):
  url = "https://www.jumia.co.ke/all-products/" + "?page=" +str(page)+"#catalog-listing"
  furl = requests.get(url)
  jsoup = BeautifulSoup(furl.content , 'html.parser')
  products = jsoup.find_all('div' , class_ = 'info')

  for product in products:
      Name = product.find('h3' , class_="name").text.replace('\n', '')
      Price = product.find('div' , class_= "prc").text.replace('\n', '')
      try:
        Rating = product.find('div', class_='stars _s').text.replace('\n', '')
      except:
        Rating = 'None'
      
      info = [ Name, Price,Rating]
      print(info)
```
##Conclusion

This is a simple web scraping beginners project into data analytics. All the best in your journey.