Web Scraper with Python (Beautiful Soup) & Deployment of it into Heroku [Part2]
Tags: webscraping, python, heroku
Published: 2022-10-10 14:08:04

After writing the [code portion](https://dev.to/mirzokhid/a-web-scraper-with-python-beautiful-soup-deployment-of-it-into-heroku-part1-22kb) of my project and testing it, I pushed it into the [Heroku](https://www.heroku.com/home) server. Since running the program regularly manually might get tedious over time I scheduled it (a.k.a cron job) so it runs automatically at a given time (every day in my case). Turns out Heroku does not allow unverified users ([here is](https://devcenter.heroku.com/articles/account-verification#when-is-verification-required?c=&utm_campaign=freedynolimits&utm_medium=telex&utm_source=nurture&utm_content=devcenter&utm_term=when-verify) how to verify your account) to use [add-ons](https://devcenter.heroku.com/articles/add-ons?c=&utm_campaign=freedynolimits&utm_medium=telex&utm_source=nurture&utm_content=devcenter&utm_term=add-ons) so I scheduled it manually with the [python schedule](https://schedule.readthedocs.io/en/stable/) module. Later on, after I verified my account with a credit card I was able to use the [Heroku Scheduler](https://devcenter.heroku.com/articles/scheduler). In this post we will go through both of the ways. However, first we have to connect PostgreSQL to your database in Python.
<br>
**Connect Python to Postgresql**
[Connecting in Python](https://devcenter.heroku.com/articles/connecting-heroku-postgres#connecting-in-python) describes connecting to the database in the Heroku server with PostgreSQL. First you should install psycopg2 package
`pip install psycopg2-binary`
then connect to DATABASE_URL with this package
```
import os
import psycopg2

DATABASE_URL = os.environ['DATABASE_URL']

conn = psycopg2.connect(DATABASE_URL, sslmode='require')
```
<br>
**Scheduling with Python Schedule**
Python schedule module, as the name suggests, runs Python functions (or any other callable) periodically using a friendly syntax.

We install it with the command:
`$ pip install schedule`

Import schedule and time module:

```
import schedule
import time
```

Define a function:
```
def function_name():
    # ToDo

schedule.every(10).minutes.do(function_name)
schedule.every().hour.do(function_name)
schedule.every().day.at("10:30").do(function_name)
schedule.every().monday.do(function_name)
schedule.every().wednesday.at("13:15").do(function_name)
schedule.every().minute.at(":17").do(function_name)

while True:
    schedule.run_pending()
    time.sleep(1)
```
Source: https://schedule.readthedocs.io/en/stable/
https://www.youtube.com/watch?v=qquCAgwvL8Q
<br>
**Pushing the code into the Heroku Server**
Heroku is a quite popular cloud platform. On the [Getting Started on Heroku with Python](https://devcenter.heroku.com/articles/getting-started-with-python?singlepage=true) you will see in detail how to install Heroku CLI onto your machine and push your project into the server using Git.
Keep in mind that, unlike the tutorial above, [Procfile](https://devcenter.heroku.com/articles/procfile), we must use worker process type!

![Procfile](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/di802e2rkc5uh821ojmx.png)

<br>
**Scheduling with Heroku Scheduler**
For a free dyno Heroku gives you 550 hours per month ([read more about dynos](https://www.heroku.com/dynos)) plus 450 hours if you verify your account. 
Pushing your code into Heroku with Python Schedule might use a lot of free dyno hours. 
![heroku ps](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zaf1ke76z3b2zoq80zcl.png)
This is why we will take advantage of the [Heroku Scheduler](https://devcenter.heroku.com/articles/scheduler)

Go to the "Recources" section of your app
![Recources](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/322z6x2ocz0obb09zy2r.png)
Find Heroku Scheduler and add it
![Search Heroku Scheduler](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tm30al2n73muq9at5b38.png)
Click on Heroku Scheduler add-on
![Click into Heroku](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cgj2sbzcxsze2nfk2ddr.png)
Create a job for a suitable time period and Save it
![Create a job](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2xsg28scfz975pmgqwjp.png)

At the end you might check your work with 
`heroku logs --tail`


<br>
**Disclaimer!**
Starting November 28th, 2022, free Heroku Dynos, free Heroku Postgres, and free Heroku Data for Redis will no longer be available.
More information
https://blog.heroku.com/next-chapter