From Web to Word using Python
Tags: python, beautifulsoup, pandoc
Published: 2020-11-24 12:31:15

Python is an excellent choice when it comes to automating repetitive tasks. In this article we will look at how we can leverage the power of Python to scrape web content and convert it to a well formatted Word document.

Continue reading to learn how to:

* Scrape content using `BeautifulSoup`
* Convert scraped data to Word format using `Pandoc`


# Defining the problem

Let's suppose that we have a website with a catalogue of books organised into categories. We want to have a version of the catalogue in a Word document. The are essentially two elements to this problem: (1) How do we collect the data? and (2) How do we create a Word document from this data?

The examples in this tutorial we will use the following dummy website created for scraping practice: http://books.toscrape.com/

![Screenshot of http://books.toscrape.com/ website](https://dev-to-uploads.s3.amazonaws.com/i/z8hiotb9m1nui77hruny.JPG)

The following steps summarise what is required to solve this problem:

1. Collect data from index page
2. Scrape a list of links to all film category pages
3. Visit each category page and scrape a list of films in that category
4. Visit each film in each category and scrape relevant data for that film
5. Format collected data into a markdown string
6. Store collected data as a Word document


# Collect data from index page

The first step is to get the data from the index page:

```python
from urllib.request import urlopen
from bs4 import BeautifulSoup


def collect_website_data(url):
    """
    Takes url string parameter, returns BeautifulSoup object with website content
    """
    index_page = urlopen(url) # HTTP Response
    scrape_data = BeautifulSoup(index_page, "html.parser") # BeatifulSoup Object
    return scrape_data


if __name__ == "__main__":
    BASE_URL = "http://books.toscrape.com/"
    page_data = collect_website_data(BASE_URL)

```

Let's understand what is going on in the script above. We define a function called `collect_website_data`. This function takes a string as a parameter. We expect this to be the url of the page we want to scrape.

Here we are using the `urlopen` method from the `urllib.request` library.

The http response is then converted into a `BeautifulSoup` object and it is now ready for scraping.

In the `main` section of the script we are calling the `collect_website_data` function and provide the `BASE_URL` as the parameter.


# Getting a list of categories

Now that we have a web scraping object to work on we need figure out how to get a list of all the category pages.

For that purpose we create another function

```python
...

def get_list_of_category_urls(bs_object, base_url):
    """
    Takes a BeautifulSoup object as parameter, returns a list of urls
    """
    category_a_elements = bs_object.find("div", {"class": "side_categories"}).li.ul.findAll("a")
    category_links = []

    for item in category_a_elements:
        href = base_url + item.get("href")
        category_links.append(href)

    return category_links


if __name__ == "__main__":
    ...
    links_to_categories = get_list_of_category_urls(page_data, BASE_URL)
```

The `get_list_of_category_urls` takes two parameters: a BeautifulSoup object and a string representing the base url.

Analysis of the category sidebar section reveals that the `<div class="side_categories">` holds the sidebar menu.

![Screenshot of category sidebar](https://dev-to-uploads.s3.amazonaws.com/i/pj6wot071p2tm3usjzeq.JPG)

However, we need to drill down to the child `<ul>` element and find all `<a>` elements there. Using BeautifulSoup this can be achieved like so:

```python
category_a_elements = bs_object.find("div", {"class": "side_categories"}).li.ul.findAll("a")
```

As this is not a BeautifulSoup tutorial per se. I refer you to the [Beautiful documentation](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) for additional information and examples.

Next, we are then using a `for` loop to go through every element in the list to pick out the `href` property. Because all the `hrefs` are relative in our example we need to prefix the `BASE_URL` to each to build a list of absolute links.

```python
for item in category_a_elements:
        href = base_url + item.get("href")
        category_links.append(href)
```

With each loop, a `href` item is added to the `category_links = []` list. After cycling through all the categories we return a list of strings which are full urls to category pages.

We store the list in `links_to_categories` variable.


# Scrape data from each category page

With the list of category urls ready, we can now proceed to write a scraping script that will visit each page and retrieve the information that we want.

Let's start from something easy like getting the category title from each category page:

```python
...

def get_category_titles_from_each_page(list_of_urls):
    """
    Takes a list of urls, returns category titles from each visited page
    """
    titles = []

    for url in list_of_urls:
        category_page = urlopen(url)
        scrape_data = BeautifulSoup(
            category_page, "html.parser")  # BeatifulSoup Object
        title = scrape_data.h1.text
        titles.append(title)
    
    return(titles)


if __name__ == "__main__":
    ...
    titles = get_category_titles_from_each_page(links_to_categories)

``` 

Using the list we have in `links_to_categories` as input we run the `get_category_titles_from_each_page` function.

The function uses an empty list called `titles`. We are then using a `for` loop to repeat the process of requesting a web page and converting it to a BeautifulSoup object and then retrieving the text of `<h1>` tag from it. With each run we are appending the `titles` list. The completed list is what the function returns.


# Apply Markdown formatting

The next step is to convert the list of records into a Markdown string.

Why Markdown? It allows us to apply simple string manipulation techniques on raw text to turn it into Markdown. Markdown format can then be easily converted to a number of output formats. 

For the purposes of this exercise we need to know the following Markdown principles:

```
# One '#' marks a h1 header

## Two '#'s mark a h2 header

This will be a paragraph

This will another paragraph
```

The `apply_markdown_formatting` function prefixes a `#` followed by a `space` and suffixes with two new line `\n\n` characters.

```python
...

def apply_markdown_formatting(list_of_records):
    """
    Takes an iterable, returns markdown formatted string
    """
    markdown_string = ""

    for item in list_of_records:
        line = "#" + " " + item + "\n\n"
        markdown_string += line

    return markdown_string


if __name__ == "__main__":
    ...
    markdown = apply_markdown_formatting(titles)

```

If we run `print(markdown_string)` we should get something like this:

```markdown
# Travel

# Mystery

# Historical Fiction

# Sequential Art
...
```


# Convert to Word

It is now time to check if we can successfully convert the Markdown string into a Word document.

For this you will need to [install pandoc universal document converter](https://pandoc.org/). Pandoc is available on Windows, macOS as well as Linux. Pandoc supports a large number of document markup formats and is not limited to Markdown and Word.

In the script itself we will be using the `pypandoc` library which provides a thin wrapper for pandoc. You might also wish to install an additional filter called `pandoc-docx-pagebreakpy` which will allow you to insert page breaks into Word output. Both packages are available on pypi and can be installed via pip.

```python
...
import pypandoc
...

def convert_markdown_to_docx(markdown_string):
    """
    Takes a markdown string, converts it to docx
    """
    filters = ['pandoc-docx-pagebreakpy']
    output = pypandoc.convert_text(
        markdown_string, 'docx', format='md', outputfile="output.docx", filters=filters)
    pass


if __name__ == "__main__":
    ...
    convert_markdown_to_docx(markdown)
```

Voilà! We've created a Word document from scraping data! The `output.docx` file should contain a list of scraped film categories formatted as `Heading 1`.

This provides the bare-bones functionality for the web-to-docx script but let's spice it up a bit.


# A more complex example

Consider the following function `get_info_for_each_film` which combines the functionality of `get_category_titles_from_each_page` and `apply_markdown_formatting`. We will reuse the `collect_website_data` and `get_list_of_category_urls` as well as the `convert_markdown_to_docx` functions:

```python
import progressbar
import pypandoc
from urllib.request import urlopen
from bs4 import BeautifulSoup


def collect_website_data(url):
    ...
    return scrape_data


def get_list_of_category_urls(bs_object, base_url):
    ...
    return category_links


def get_info_for_each_film(list_of_urls, base_url):
    """
    Takes a list of urls, returns markdown formatted string
    """
    markdown_string = ""

    print('Retrieving film data for each category:')
    with progressbar.ProgressBar(max_value=len(list_of_urls)) as bar:
        for counter, url in enumerate(list_of_urls):
            category_page = urlopen(url)
            scrape_data = BeautifulSoup(
                category_page, "html.parser")
            category = scrape_data.h1.text
            category_md = "#" + " " + category + "\n\n"
            markdown_string += category_md
            links_to_films = scrape_data.find_all("h3")
            links_to_films = [base_url + "catalogue/" +
                              i.a.get("href")[9:] for i in links_to_films]
            for film_link in links_to_films:
                film_page = urlopen(film_link)
                scrape_data = BeautifulSoup(
                    film_page, "html.parser")
                film_title = scrape_data.h1.text
                film_title_md = "##" + " " + film_title + "\n"
                markdown_string += film_title_md
                try:
                    description = scrape_data.find(
                        "div", {"id": "product_description"}).next_sibling.next_sibling.text
                    description_md = description + "\n\n"
                    markdown_string += description_md
                except AttributeError as e:
                    markdown_string += '\n\n'
            markdown_string += '\\newpage'
            bar.update(counter)
    return markdown_string


def convert_markdown_to_docx(markdown_string):
    ...
    pass


if __name__ == "__main__":
    BASE_URL = "http://books.toscrape.com/"
    page_data = collect_website_data(BASE_URL)
    links_to_categories = get_list_of_category_urls(page_data, BASE_URL)
    film_data = get_info_for_each_film(links_to_categories, BASE_URL)
    convert_markdown_to_docx(film_data)

```

The above function will do the necessary web scraping and string formatting to create a valid Markdown string in one go.

Because the script take a while to run, I have also added a progressbar using the `progressbar2` library. The progress bar uses the length of the list containing category urls to set the maximum range for progress increments. The `enumerate` method adds a counter to the `for` loop that we can use to increment the bar's progress.

For the script to be more useful we can drill down and scrape data from each film on each category. The aim will be to create a document that will follow this pattern:

```
[Heading 1] -- Category
	[Heading 2] -- Title
		[Paragraph] -- Film description
```

Each category will become a `Heading 1` with each film title becoming a `Heading 2`. We will also add a `Paragraph` with the scraped film description.

In order to do that we need to list the urls for films on each category page. Analysis of the underlying html reveals that links to films are contained in `h3` elements on category pages. The first step is then to retrieve all `h3` elements. To get the urls for film pages I am using list comprehension which perhaps sacrifices some of the code readability for conciseness:

```python
links_to_films = scrape_data.find_all("h3")
links_to_films = [base_url + "catalogue/" + i.a.get("href")[9:] for i in links_to_films]
```

Let's go through what's going on here: For each item in `links_to_films`, get the `href` property from the `a` tag, trim characters from the beginning of the string (those refer to relative url characters in the link), prepend the `base_url` variable as well as "catalogue/". Together this builds a list of film page urls i.e.:

```python
[
...
"http://books.toscrape.com/catalogue/emma_17/index.html",
"http://books.toscrape.com/catalogue/alice-in-wonderland-alices-adventures-in-wonderland-1_5/index.html"
...
]
```

A nested `for` loop allows us to drill down to each film page and scrape data from it. Some of the BeautifulSoup trickery such as the doubled `next_sibling` in this example is explained in [BeautifulSoup documentation](https://www.crummy.com/software/BeautifulSoup/bs4/doc/#contents-and-children):

```python
description = scrape_data.find(
                        "div", {"id": "product_description"}).next_sibling.next_sibling.text
```

In short, a single `next_sibling` only retrieves the white space between the elements. The second `next_sibling` correctly retrieves the `p` element that holds the film description.

I have added a `Try/Except` block around the `description` scraper to handle situations where films have no descriptions.

When using Pandoc to export to a Word document, the `\\newpage` tag will require you to install the `pandoc-docx-pagebreakpy`.

You are welcome to check out the full script on Github:

{% github pavstermeister/web-to-doc %}


# Real-life application of this solution 

The inspiration to write this article came from a problem I had to solve recently. In my day job I am an IT Business Analyst at a large university. One of the administrative offices asked for help in order to provide an improved way of generating a Word document containing an export of postgraduate course data available on the courses website.

The process to generate the file would involve a member of staff going through some 200 course pages meticulously copying and pasting course information into a Word file. That Word file would serve as the source for the printed version of the College Postgraduate Prospectus.

After some trial and error the methodology outlined above worked and I ended up with a Word document structured more or less like this:

```
[Heading 1] -- Faculty
	[Heading 2] -- School
		[Heading 3] -- Course title
			[Paragraph] -- Course details
			[Paragraph] -- Course description
			[Paragraph] -- Course requirements
```

The final output files was well received by the office in question and I got some nice feedback too 😄:

> As Clarke once put it: "Any sufficiently advanced technology is indistinguishable from magic.", and this is magic, it will be of huge assistance for the development of the prospectus.

It's great to see how a little bit of programming can make someone's life much easier.

**Before you go... If you enjoyed this article, go ahead and click the Follow button. Consider [following me on Twitter](https://twitter.com/pavstermeister). See you soon :)**