The one where we build a web scrapper and a slackbot - Part 2
Tags: slackbot, apscheduler, slackapi, python
Published: 2020-02-23 14:55:57

---
title: The one where we build a web scrapper and a slackbot - Part 2
published: true
description: In this article I walk through my process of building a slackbot and sending messages to a channel at intervals
tags: #slackbot, #apscheduler, #slackapi, #python
cover_image: https://dev-to-uploads.s3.amazonaws.com/i/487fpur2i6kmx0jo78rz.jpeg
series: The one where we build a web scrapper and a slackbot.
---

# The problem

In part one of this two-part series we set out on a quest to automate these tasks

- Visit [Brainyquote](https://www.brainyquote.com/topics/excellence-quotes)
- Find and copy a random quote about excellence from the site.
- Post the quote to a slack channel.

we were able to build a web scraper that enabled us to scrape a website for quotes and save them to a JSON file. In this second part of the article series, we would be bringing it all together by building a slackbot that would periodically post one of the quotes we scrapped to a slack channel!

If you missed part one you can find it [here](https://dev.to/vic3king/the-one-where-we-build-a-web-scrapper-and-a-slackbot-part-1-3blk)


# The Product

- Slackbot: Here we would visit slack [apps page](https://api.slack.com/apps), and create a bot associated with a slack channel.
- Slackapi backend:  Making use of slacks open API we would write some python functions that when called posts the message to a channel.
- Apscheduler: Finally we would write another function that would help us call those API functions at intervals.

## Prerequisites

- Building up from part one, all you'll need at this point is a slack workspace, you could easily create one for free [here](https://slack.com/intl/en-ng/help/articles/206845317-Create-a-Slack-workspace) or just build the bot into any workspace by creating a channel where you can play around with.

Part 2: The slackbot

To get started visit [slack](https://api.slack.com/apps?new_granular_bot_app=1) a modal should pop up like this ![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/9dmux14kv2btl77411gw.png)

- Type in your app name
- Select a development workspace(slack workspace where the bot would be installed)
- Click `create App` and you'll be navigated to this page ![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/pytjrewrijuf261azopk.png)

- Now we need to set a `scope` for our bot 
> Scopes give your bot permission to do things like sending a message in your development workspace.

- Navigate to `OAuth & Permissions` on the sidebar to add scopes to your app
- Scroll down to the `Bot Token Scopes` section and click `Add an OAuth Scope`.
- Select or search for `chat:write` scope to grant your bot the permission to post messages in channels it's a member of. you should see a success banner after this ✅
- Next, we have to install the bot to our workspace, by scrolling to the top of the `OAuth & Permissions` pages and clicking the green `Install App to Workspace` button.
- After the above step, you should be navigated to the below page. All you have to do is click the `allow` button.
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/mx2uu9jgtep7hay9i144.png)
- Finally 🎉, copy your bot token, you'll need it to communicate with slack API.

Next steps... We need to write our backend code to interface with slack. we will be doing this in the same folder where we created our `web scrapper` you can find that code if you don't have it by clicking [here] (https://github.com/vic3king/excel/tree/part-one)

```bash
- cd clonedOrExistingProject
- mkdir bot
- cd scrapping-tool
- touch __init__.py builder.py post_message.py
```

Create and activate a virtual environment if you don't have one already

```bash
- virtualenv --python=python3 venv
- source venv/bin/activate
```

In the `builder.py` file add the following code

```python
def construct_payload(HEADER_BLOCK, DIVIDER_BLOCK, MESSAGE_BLOCK):

    return {
        "channel": "your-channel-id-goes-here",
        "username": "name-of-your-app",
        "icon_emoji": ":robot_face:",
        "text": ":wave: Your daily dose of motivation has arrived",
        "blocks": [
            HEADER_BLOCK,
            DIVIDER_BLOCK,
            MESSAGE_BLOCK
        ],
    }

```

Slack has a tool called the block builder kit, it's basically used to generate markdowns that can be used to style the appearance of messages on slack. Try it out [here](https://api.slack.com/tools/block-kit-builder).

For us, we created a function that accepts various blocks from the block builder kit and in the end, we want to have something looking like this![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/oz0okqlic242zechqb1e.png)

From the code, we passed in 
- Channel: The id of a Slack channel where we want to invite our bot(you can copy it from your slack channel, make sure to invite the bot to this channel)
- Username: which would be the name of the app you created earlier
- Text: would be what appears on the slack notification popup.
- Blocks: contains the various parts of the message, the header, a divider line, and the actual message.

Let's start by creating our random quote generator function, this file will read our previously scrapped json quotes and return any random one when the function is called. from the root of your project folder, run the command

```bash
- mkdir utils
- touch __init__.py random_quote.py
```

In the random_quote file, add the following code

```python
import random
import json


def random_quotes_generator():
    with open('quotes.json', 'r') as f:
        quotes_dict = json.load(f)


        quote = random.choice(quotes_dict)
        return quote

```

This file creates a function `random_quotes_generator` reads our quotes from the json file and with python's random module returns one random quote when called.


Next, we create our post message function. Install these packages

```python
pip3 install slack dotenv
```

`slack` to interface with slack API and `dotenv` to load in our bot secret token.

Navigate to the `post_message.py` file and post the following code.

```python
import os
from slack import WebClient
from dotenv import load_dotenv
from bot.builder import construct_payload
from utils.random_quote  import random_quotes_generator

load_dotenv()

SLACK_BOT_TOKEN = os.getenv("SLACK_BOT_TOKEN")

# Initialize a Web API client
slack_web_client = WebClient(token=SLACK_BOT_TOKEN)


def post_quote_to_channel():
    quote = random_quotes_generator()
    author = quote['author']
    link = f"https://www.brainyquote.com{quote['author-link']}"


    # Get the onboarding message payload
    message = construct_payload({
        "type": "section",
        "text": {
                "type": "mrkdwn",
                "text": "*Your daily dose of motivation* :blush:"
        }
    },
        {
        "type": "divider"
    },
        {
        "type": "section",
        "text": {
            "type": "mrkdwn",
            "text": f"The test of the artist does not lie in the will with which he goes to work, but in the excellence of the work he produces... *By <{link}|{author}>*"
        }
    })
    # Post the onboarding message in Slack
    response = slack_web_client.chat_postMessage(**message)

```

So what are we doing here... 

We imported a couple of 3rd party packages, our `message builder` function and our `random_quotes_generator`.

We also load in our `SLACK_BOT_TOKEN`, create a `.env` file use the key `SLACK_BOT_TOKEN` and set it to your bot token.


```bash
SLACK_BOT_TOKEN=token
```

- We initialized slack WebClient and created the `post_quote_to_channel` inside which we call our random generator function and save it to a variable `quote`.
- We construct a link to each author's page.
- Next, we construct our message by calling the `construct_payload` function exported from the message builder file and passing in all required parameters.
- Finally, we call a function `chat_postMessage` from slack WebClient and pass in our constructed message.

You can test that this works now by calling the function on your terminal running this command

```pyhton
python3 post_message.py
```

The final step in this process would be to create our scheduler to periodically call the `post_quote_to_channel`. to do this we need to install some packages, create a flask server and our scheduler function.

On the root of your project folder create a new file `app.py` and run the following commands

```python
pip3 install APScheduler Flask
```

```bash
- cd bot
- mkdir jobs
- touch __init__.py scheduler.py
```
In `bot/jobs` folder locate the `scheduler.py` file and paste in the following code

```python
from apscheduler.schedulers.background import BackgroundScheduler

# jobs
from bot.post_message import post_quote_to_channel

scheduler = BackgroundScheduler()

scheduler.add_job(
    post_quote_to_channel,
    'interval',
    seconds=10
)
```

Here we import `BackgroundScheduler` from the scheduler package and initialize it.

- We also import the `post_quote_to_channel` function from the file `post_message` in the bot folder.
- With that, we call `add_job` function from the scheduler, pass the `post_quote_to_channel` function, the string `interval` and time interval in `seconds`.
to read more about app scheduler click [here] (https://apscheduler.readthedocs.io/en/stable/userguide.html)

with that, we have our scheduler function ready.

# Bringing it all together

In the `app.py` file we created earlier, post the following code

```python
import os
import time
from flask import Flask
from bot.jobs.scheduler import scheduler

# Initialize a Flask app to host the events adapter
app = Flask(__name__)

if __name__ == "__main__":
    scheduler.start()
    print('Press Ctrl+{0} to exit'.format('Break' if os.name == 'nt' else 'C'))

    try:
        # This is here to simulate application activity (which keeps the main thread alive).
        while True:
            time.sleep(5)
    except (KeyboardInterrupt, SystemExit):
        # Not strictly necessary if daemonic mode is enabled but should be done if possible
        scheduler.shutdown()

```

Here we import our scheduler, create a flask server and call our scheduler function. we also use a nifty trick from StackOverflow to keep our main thread alive.

Run the program

```python
python app.py
```

If we've done everything right we would start receiving quotes in our slack channel every 10 seconds.

> You can tweet the time interval to suit your needs. e.g minutes=69 || hours=24

That's it guys, through these series we've been able to cover a few micro-skills 

- building a slackbot
- scraping a website
- scheduling tasks

These skills can be applied separately or brought together to build a cool product.

I'll like to hear your feedback and question, feel free to leave a comment.

> To view the full code [click here] (https://github.com/vic3king/excel)