Getting Started with Web Scraping with Beautiful Soup
Tags: python, webdev, productivity
Published: 2020-10-13 20:06:22

**Check out my books on Amazon at https://www.amazon.com/John-Au-Yeung/e/B08FT5NT62**

**Subscribe to my email list now at http://jauyeung.net/subscribe/**


We can get data from web pages with Beautiful Soup.

It lets us parse the DOM and extract the data we want.

In this article, we’ll look at how to scrape HTML documents with Beautiful Soup.

### Getting Started

We get started by running:

```
pip install beautifulsoup
```

Then we can write:

```python
from bs4 import BeautifulSoup
html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>

<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
print(soup.prettify())
```

to add an HTML string and parse it with the `BeautifulSoup` class.

Then we can print the parsed document in the last line.

### Get Links and Text

We can get the links from the HTML string with the `find_all` method:

```python
from bs4 import BeautifulSoup
html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>

<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
for link in soup.find_all('a'):
    print(link.get('href'))
```

We just pass in the selector for the elements we wan to get.

Also, we can get all the text from the page with `get_text()`:

```python
from bs4 import BeautifulSoup
html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>

<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
print(soup.get_text())
```

### Parse an External Document

We can parse an external document by opening it with `open` :

```python
from bs4 import BeautifulSoup

with open("index.html") as fp:
    soup = BeautifulSoup(fp, 'html.parser')
    print(soup.prettify())
```

### Kinds of Objects

We can get a few kinds of objects with Beautiful Soup.

They include `Tag` , `NavigableString` , `BeautifulSoup` , and `Comment` .

### Tag

A `Tag` corresponds to an XML or HTML tag in the original docuemnt.

For example, we can write:

```python
from bs4 import BeautifulSoup

soup = BeautifulSoup('<b class="boldest">Extremely bold</b>', 'html.parser')
tag = soup.b
print(type(tag))
```

to get the `b` tag from the HTML string.

Then we get:

```
<class 'bs4.element.Tag'>
```

printed from the last line.

### Name

We can get the name of the tag:

```python
from bs4 import BeautifulSoup

soup = BeautifulSoup('<b class="boldest">Extremely bold</b>', 'html.parser')
tag = soup.b
print(tag.name)
```

Then we see `b` printed.

### Attributes

We can get attributes from the returned dictionary:

```python
from bs4 import BeautifulSoup

tag = BeautifulSoup('<b id="boldest">bold</b>', 'html.parser').b
print(tag['id'])
```

We get the `b` element.

Then we get the `id` value from the returned dictionary.

### Conclusion

We can get parse HTML and XML and get various elements, text, and attributes easily with Beautiful Soup.