#Day23 - How to Scrape Websites using Requests and Beautiful Soup Part1
Tags: 100daysofcode, python, programming, codenewbie
Published: 2021-04-13 03:49:58

Today, we will be discussing web scraping using Python. We will be using the library Beautiful Soup and Requests to scrape [Worldometer's Covid Tracker](https://www.worldometers.info/coronavirus/). Before we get started with scraping the website, let's install the required packages.

### Pre-requisites
Some Familiarity with HTML and CSS Selectors

### Install Packages
```python
pip install beautifulsoup4, requests
```
### Creating an instance of Beautiful Soup
Beautiful Soup is an HTML parser, i.e if it is given a string with a bunch of HTML code, it can help us extract specific content we need, for example
- tables
- lists
- links
- text inside a particular section
etc

How do we get a string with a bunch of HTML code? We use the requests library. If we use the requests library to make a request to a webpage, it will return the HTML content of the webpage as a string

```python
import requests
from bs4 import BeautifulSoup

html = requests.get("https://www.worldometers.info/coronavirus/").text
print(html)
print(type(html))
```
The output will be the HTML for the home page of Worldometer. The type of the variable 'html' is str.

Now, we will create an instance of BeautifulSoup and pass the HTML string as an argument.
```python
soup = BeautifulSoup(html, features= 'html.parser')
```

### Some Useful Functions
#### Prettify
```python
print(soup.prettify())
```
This prints the HTML string in a pretty format with indentations.

#### First instance of a tag
We will use the find function. Let's try to find the first instance of an 'div' tag on the worldometer page
```python
print(soup.find("div"))
print("-x-x-x-x-x-x-x-")
print(soup.find("div").get_text())
```
The first print statement will print the first instance of the div tag and all the other HTML elements inside. If we only want the text inside the div block, we can use the get_text() function.

#### Get all instances of a tag
```python
div_elements = soup.find_all("div")
for element in div_elements:
  print(element)
  print("-x-x-x-x-x-x-x")
  print(element.get_text())
  break
```
This returns an iterable consisting of all the div elements on the page. We can simply loop over the iterable and use the get_text() function to get the text content inside each div. Notice how the output of the above code and code in the previous section are the same. This is because the first element in the iterable is the same div block as the one returned by find().

#### Specifying CSS class and ID
The find() and find_all() function can take in the following arguements
- **class_** = The name of the class
- **id** = The name of the id

Let's try to get the number of Corona Virus Cases

![Screen Shot 2021-04-12 at 11.24.25 PM.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1618284300515/JvIPXqfC-G.png)

As you can see, the number is inside a div block with a class name "maincounter-number". The following piece of code would return the number
```python
print( soup.find('div',class_='maincounter-number').get_text() )
```
We specify that we are looking for a div and pass the class name "maincounter-number" as an argument. Again, we use the get_text() method to get the text content, i.e the number.

Now let's try to get the text inside the navbar using it's id.

![Screen Shot 2021-04-12 at 11.29.16 PM.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1618284583155/0mrpomvH6Y.png)

It has id = "navbar-main".
```python
print( soup.find('div',id = "navbar-main").get_text() )
```
As you can see it's pretty similar to when we were trying to get the number of coronavirus cases. The only difference we are setting the value for **id** instead of **class_**.

#### CSS Selectors
We can combine elements and class names or ids as well. The select() function is similar to the find_all() function. Let's look at some CSS Selectors. Below is the syntaxt
```python
soup.find('css_selectors').get_text()
```

- h1 - finds all h1 tags
- div p -  all the blocks which have a <p> inside a <div>
- div.maincounter-number - finds all div blocks with a class name "maincounter-number"
- div#navbar-main - finds the div block with id "navbar-main"
- body p a — finds all a tags inside of a p tag inside of a body tag.

Let's try to get the number of coronavirus cases and the text inside the navbar using the select() function. 
```python
print( soup.select('div.maincounter-number')[0].get_text())

print( soup.select('div#navbar-main')[0].get_text())
```
The select() functions returns an iterable, therefore we will either need to iterate over it or use an index to get the text inside the element.

In the next part, we will try to scrape the table and discuss some more usecases using Beautiful Soup