The one where we build a web scrapper and a slackbot - Part 1
Tags: selenium, beautifulsoup4, automation, python
Published: 2020-02-23 09:37:55

---
title: The one where we build a web scrapper and a slackbot - Part 1
published: true
description: In this article, I write about automation and building a web scraper using beautifulSoup4 and selenium.
tags: #selenium, #beautifulSoup4, #automation, #python
cover_image: https://dev-to-uploads.s3.amazonaws.com/i/eazo53nl8u39zz9cqwdl.jpg
series: The one where we build a web scrapper and a slackbot.
---

# The problem

As software engineers, part of what we do revolves around making seemingly easy things a little bit easier. who would imagine doing these three things would be a chore?

- Visit [Brainyquote](https://www.brainyquote.com/topics/excellence-quotes)
- Find and copy a random quote about excellence from the site.
- Post the quote to a slack channel.

It seems simple enough to do but if done every day for a year becomes boring and tedious.

Python is a scripting language built for things like this! With python, we could automate this whole process and not have to do the same thing every day.


# The Product

Well, that's exactly what we would be doing 🎉. In this two-part series, we would be building a slackbot that periodically sends a random quote about excellence to a specified slack channel. Some of our MVP features would include 

- Scraping tool: This would be responsible for getting a whole lot of quotes and saving them to a JSON file for future use
- A Slack bot:  That would be responsible for periodically(maybe every morning?) sending one random quote to a slack channel. This part of the project would require us to write some simple code for posing the message to a Slack channel at intervals.  


## Prerequisites

- A python environment and some basic knowledge of Python. That's it


Part 1: The scrapping tool

First off we need to get some groundwork done by creating a basic project setup, a virtual environment and installing some packages

```bash
- cd newly_created_folder
- mkdir scrapping-tool
- cd scrapping-tool
- touch __init__.py main.py scroll.py selenium_driver.py
```

At this point, we're good to go but I strongly recommend you create a virtual environment for this project. If you have  `virtualenv` installed on your PC all you have to do is run the following commands 

```bash
- virtualenv --python=python3 venv
- source venv/bin/activate
```

If you don't or have questions around what a virtualenv is... you may want to [Read this] (https://realpython.com/python-virtual-environments-a-primer/)

Next, install the following 3rd party packages 
- [BeautifulSoup] (https://pypi.org/project/beautifulsoup4/) to help us scrape any website for data
- [selenium] (https://pypi.org/project/selenium/) to automate browser interactions while doing so and [lxml] (https://pypi.org/project/lxml/) to interface with BeautifulSoup and parse data to LXML.

run the following  command on your terminal

```python
pip3 install BeautifulSoup selenium lxml
```

Finally, download `chrome driver` by following basic instructions [here] (https://www.kenst.com/2015/03/installing-chromedriver-on-mac-osx/). This would enable us to run a headless version of chrome when using selenium for automation. If you're on a mac you can simply run 

```bash
brew cask install chromedriver
```

> Setup sometimes may endure for a night but code comes in the morning... UNKNOWN

Let's write some code!

In the `scrapping-tool` folder you created, locate the `selenium_driver.py` file and paste the following code in 

```python
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument('--incognito')
options.add_argument('--headless')
driver = webdriver.Chrome("/usr/local/bin/chromedriver", chrome_options=options)

```

This piece of code imports `webdriver` from selenium and adds some configuration options for webdriver like incognito, headless mode, etc, finally we make use of the chromedriver we installed earlier by pointing to the path where it was downloaded to. we save this in a `driver` variable for future use.

> By adding the `__init__.py` file in our folder we told python to consider every file in that folder a package. This means functions, variables, etc are exposed by default from any location in our app 😎.

Part of the hassles that come with automation in web browsers comes up when human interaction is needed. For example, the website we are trying to scrape has some functionalities you would notice once you open the site. 
- On the first visit to the website, you would have to click and accept the privacy policy
- After that, we see the page with all those quotes we would like to get, but then this page implements an infinite scroll. 

We won't be doing much automation if we were to help our browser click that button or help the browser scroll when it gets to the bottom of the page. These problems bring us to our next step `scroll.py`.

> The key to scrapping a website properly lies in your ability to hit inspect and find that class or id with which you can access that element

In the file `scroll.py` Copy and paste the code below.

```python
import time


def scroll(driver, timeout):
    scroll_pause_time = timeout

    # wait for terms modal to popup and then click
    driver.implicitly_wait(timeout)
    privacy_button = driver.find_elements_by_css_selector(".qc-cmp-buttons > button:nth-child(2)")
    privacy_button[0].click()
    time.sleep(2)

    # Get scroll height
    last_height = driver.execute_script("return document.body.scrollHeight")

    while True:
        # Scroll down to bottom
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

        # Wait to load page
        time.sleep(scroll_pause_time)

        # Calculate new scroll height and compare with last scroll height
        new_height = driver.execute_script("return document.body.scrollHeight")
        if new_height == last_height:
            # If heights are the same it will exit the function
            break
        last_height = new_height

```

A few things to note

- We create a `scroll` function which takes two parameters, `driver`(our page source) and `timeout`(wait time). 
- We make use of some methods available on the driver object like `find_elements_by_css_selector` this would help us locate elements. like in our case locate the privacy button and where to start our infinite scrolling.
- We also make use of `execute_script` method which takes our browsers window object as a parameter to enable us to scroll the website, determine page height, etc 
- Notice the `while` loop? This loop checks our browser and calculate's new scroll height by comparing our current height with the last scroll height. if both heights are the same we break the loop meaning we are at the end of the page.


Bringing it all together, we build the scrapper itself.

In `main.py` still within the `scrapping-tool` folder, add the following code

```python 
import re
import json
from bs4 import BeautifulSoup
from selenium_driver import driver # here we import the driver we configured earlier
from scroll import scroll # the scroll method


def get_quotes(url):
    try:
        # implicitly_wait tells the driver to wait before throwing an exception
        driver.implicitly_wait(30)
        # driver.get(url) opens the page
        driver.get(url)
        # This starts the scrolling by passing the driver and a timeout
        scroll(driver, 5)
        # Once scroll returns bs4 parsers the page_source
        soup = BeautifulSoup(driver.page_source, "lxml")
        # Them we close the driver as soup_a is storing the page source
        driver.close()

        # Empty array to store the links
        quotes = []

        regex_quotes = re.compile('^b-qt')
        regex_authors = re.compile('^bq-aut')

        quotes_list = soup.find_all('a', attrs={'class': regex_quotes})
        authors_list = soup.find_all('a', attrs={'class': regex_authors})

        quotes = []
        zipped_quotes = list(zip(quotes_list, authors_list))
        for i, x in enumerate(zipped_quotes):
            quote = x[0]
            author = x[1]
            quotes.append({
                "id":  f"id-{i}",
                "quote": quote.get_text(),
                "author": author.get_text(),
                "author-link": author.get('href')
            })

        with open("quotes.json", 'w') as json_file:
            json.dump(quotes, json_file)
    except Exception as e:
        print(e, '>>>>>>>>>>>>>>>Exception>>>>>>>>>>>>>>')


get_quotes('https://www.brainyquote.com/topics/excellence-quotes')

```

What do we have here?
- We import the `BeautifulSoup4` library, some inbuilt python packages like `re(regular expression )` and json.
- We also import the functions packages we created earlier like scroll and driver.
- We create a `get_quotes` function that takes in a URL as a parameter.
- With this, we tell our browser to wait a *Lil* before throwing an error(sometimes network issues may slow things down).
- We called the scroll function to do its thing.
- And once that is done we pass `driver.page_source` to BeautifulSoup4. printing `driver.page_source` at this point would show a bunch of HTML tags
-We call close to stop browser interactions, we have all we need now

The goal is to scrape a quote, its author and a link to get all of that author's quotes. at this point, we have all of that data albeit in a format we cannot work with yet(HTML tags) also notice from the code that we are extracting data for the author separately and the same for quotes. How do we link each quote to its author? we also need to create a python dictionary containing all those pieces of information, give them unique id's and also form the author's links. Python `zip` function to the rescue, to put it simply this function takes two lists and generates a series of tuples containing elements from each list. We also made use of `enumerate` function this means we can unpack index and data from the tuples returned from the zip function. With that, we unpack and loop over the returned tuple, create a python dictionary containing the data we want and append that to the quotes array. We also called a BeautifulSoup4 method  `get_text()` on the author and quote to enable us to return actual texts from our HTML tags. we also called `get('href')` which returns any property of a tag we specify, in our case href, this is how we get the link to the author's quotes. Finally, we save the contents of our `quotes` list to a json file by creating a `quotes.json` file and dumping our data into it by calling `json.dump`.
 

To run the scrapper 
```python
python scrapping-tool/main.py
```

> To see all this in action, you can comment out this piece of code ` options.add_argument('--headless')` in the file `selenium_driver.py`.

Yo! That’s it for now. Feel free to leave a comment, feedback or opinions in the comments. In part two of this article, we will go through creating a slackbot that would display these scrapped quotes on a slack channel. That would also mean we configure a flask project that would enable us to run a server and implement a scheduler!

> To view the full code for this article [click here] (https://github.com/vic3king/excel/tree/part-one)