Price Tracker Application with Django | Track Discounts
Tags: python, django, tutorial, webdev
Published: 2019-10-26 19:01:24

By <a href="https://reversepython.net/">Reverse Python</a>
<h3>Hi DEVs!</h3>

<p>Youtube Channel with video tutorials - <a href="https://www.youtube.com/channel/UCEwYRHE5ZAOMfUaIpCQvjEQ">Reverse Python Youtube</a></p>

<p>In this tutorial we are going to build <b> price tracker application</b> which will notify us for discounts.</p>

<p>Creating complex projects is the key to learn fast! If you are following me for a while, you already know I like complex stuff, so we are going to use <b>RabbitMQ</b> + <b>Celery</b> + <b>BeautifulSoup</b> + <b>Django</b> for creating this app.</p>

<p>Alright! Let's Start!</p>

<p>Celery is the best choice for doing background task processing in the Python/Django ecosystem. It has a simple and clear API, and it integrates beautifully with Django. So, we are using Celery to handle the time-consuming tasks by passing them to queue to be executed in the background and always keep the server ready to respond to new requests.</p>

<p>Celery requires a solution to send and receive messages; usually this comes in the form of a separate service called a message broker. We will be configuring celery to use the RabbitMQ messaging system, as it provides robust, stable performance and interacts well with celery.</p>

<p>We can install RabbitMQ through Ubuntu’s repositories by following command:</p>

```
sudo apt-get install rabbitmq-server
```
<p>Then enable and start the RabbitMQ service:</p>

```
sudo systemctl enable rabbitmq-server
sudo systemctl start rabbitmq-server
```

<a href="https://www.rabbitmq.com/install-homebrew.html">Install RabbitMQ on Mac</a>

<p>Well, create a new Django project named <b>pricetracker</b> and app named <b>tracker</b></p>

<p>Install following dependencies:</p>

```
pip3 install beautifulsoup4 httplib2 Celery
```

<p>Once installation completed, add the CELERY_BROKER_URL configuration to the <b>settings.py</b> file: </p>

```python
CELERY_BROKER_URL = 'amqp://localhost'
```

<p>Then, create <b>celery.py</b> inside your project.</p>

<img src="https://thepracticaldev.s3.amazonaws.com/i/307eagil3qtve1hhr1qa.png" alt="celery" />

<b>celery.py</b>
```python
import os
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'pricetracker.settings')

app = Celery('pricetracker')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()
```
<p>We are setting the default Django settings module for the 'celery' program and loading task modules from all registered Django app configs.</p>
<p>Now inside your <b>__init__.py</b> import the celery:</p>

```python
from .celery import app as celery_app

__all__ = ['celery_app']
```
<p>This will make sure our Celery app loaded every time Django starts.</p>

<p>Now, Let's create our model</p>
<b>models.py</b>

```python
from django.db import models

class Item(models.Model):
    title = models.CharField(max_length=200)
    url = models.CharField(max_length=600)
    requested_price = models.IntegerField(default=0)
    last_price = models.IntegerField(null=True, blank=True)
    discount_price = models.CharField(max_length=100, null=True, blank=True)
    date = models.DateTimeField(auto_now_add=True)

    def __str__(self):
        return self.title
```

<p>We are going to crawl <a href="https://www.ebay.com/">eBay</a>. User will enter URL of specific item and requested price.</p>

<p>So, let's create a form for that:</p>
<b>forms.py</b>

```python
from django import forms

class AddNewItemForm(forms.Form):
    url = forms.CharField(max_length=600)
    requested_price = forms.IntegerField()
```
<p>We will use beautifulsoup to crawl price and title of item in given URL. After data crawled we have to convert price to float and create new object in database.</p>
<b>views.py</b>

```python
from urllib.request import urlopen, Request
from bs4 import BeautifulSoup

def crawl_data(url):
    # User Agent is to prevent 403 Forbidden Error
    req = Request(url, headers={'User-Agent': 'Mozilla/5.0'})
    html = urlopen(req).read()
    bs = BeautifulSoup(html, 'html.parser')
    
    title = bs.find('h1', id="itemTitle").get_text().replace("Details about", "")
    price = bs.find('span', id="prcIsum").get_text()
    clean_price = float(price.strip().replace("US", "").replace("$", ""))
    return {'title': title, 'last_price':clean_price }
```
<p> <i>strip()</i> removes spaces at the beginning and at the end of the string and <i>replace()</i> method replaces a specified phrase with another specified phrase. So, we used these methods to get clean price and title of item.</p>


<p>Once data crawled successfully, it is time to create new object in database. We will use this function in form submission to crawl title and price of new item.</p>

<b>views.py</b>

```python
from django.shortcuts import render, get_object_or_404,HttpResponseRedirect 
from .models import Item
from .forms import AddNewItemForm

def tracker_view(request):
    items = Item.objects.order_by('-id')
    form = AddNewItemForm(request.POST)
    if request.method == 'POST':
        if form.is_valid():
            url = form.cleaned_data.get('url')
            requested_price = form.cleaned_data.get('requested_price')
            # crawling the data 
            crawled_data = crawl_data(url)
            Item.objects.create(
            url = url,
            title = crawled_data['title'],
            requested_price=requested_price,
            last_price=crawled_data['last_price'],
            discount_price='No Discount Yet',
            )
            return HttpResponseRedirect('')
        else:
            form = AddNewItemForm()
    context = {
        'items':items,
        'form':form,
    }
    return render(request, 'tracker.html', context)
```
<p>Great! Now, we need to crawl the data for all objects continuously to be aware of discounts. If we do this without celery the server connection will timeout which means that a server is taking too long to reply to a data request and our application will crash.</p>

<p>Create tasks.py in your app and let's handle it with celery tasks. </p>
<b>tasks.py</b>
```python
import time
from celery import shared_task
from .models import Vehicle
from tracker.views import crawl_data

@shared_task
# do something heavy
def track_for_discount():
    items = Item.objects.all()
    for item in items:
        # crawl item url
        data = crawl_data(item.url)
        # check for discount
        if data['last_price'] < item.requested_price:
            print(f'Discount for {data["title"]}')
            # update discount field to notify user
            item_discount = Item.objects.get(id=item.id)
            item_discount.discount_price = f'DISCOUNT! The price is {data["last_price"]}'
            item_discount.save()      
while True:
    track_for_discount()
    time.sleep(15)  
```
<p>@shared_task will create the independent instance of the task for each app, making task reusable. This makes the @shared_task decorator useful for libraries and reusable apps, since they will not have access to the app of the user.</p>
<p>We are simply crawling data every 15 seconds and comparing last price with requested price. If last price is smaller than requested price then we are updating the discount price field.</p>

<p>What if price will increase again?</p>

```python
@shared_task
def track_for_not_discount():
    items = Item.objects.all()
    for item in items:
        data = crawl_data(item.url)
        if data["last_price"] > item.requested_price:
            print(f'Discount finished for {data["title"]}')
            item_discount_finished = Item.objects.get(id=item.id)
            item_discount_finished.discount_price = 'No Discount Yet'
            item_discount_finished.save()
```

<p>Great! Now, it will possible to track discounts properly. You can add one more function which will detect closer prices and notify user about it. For instance, if item price is 100$ and requested price is 97$. But let's keep it simple for now.</p>
<p> Finally, we can create our template.</p>
<b>tracker.html</b>
```html
{% extends 'base.html' %}

{% block content %}
<form method="POST">
    {% csrf_token %}
    {{form.as_p}}
    <button class="btn btn-primary" type="submit">Send</button>
</form>
<table class="table">
    <thead>
      <tr>
        <th scope="col">Title</th>
        <th scope="col">Requested Price</th>
        <th scope="col">Last Price</th>
        <th scope="col">Discount Price</th>
        <th scope="col">Date Created</th>
      </tr>
    </thead>
    <tbody>
    {% for item in items %}
      <tr>
        <td>{{item.title}}</td>
        <td>{{item.requested_price}}</td>
        <td>{{item.last_price}}</td>
        <td>{{item.discount_price}}</td>
        <td>{{item.date}}</td>
      </tr>
    {% endfor %}  
    </tbody>
  </table>
  {% endblock %}  
```
<p>Well, you can improve the project by adding email functionality so Django will send email about discounts. Take a look <a href="https://simpleisbetterthancomplex.com/tutorial/2016/06/13/how-to-send-email.html">How to Send Email in a Django App</a></p>

<p>Feel free to contribute the project :) </p>
<p>You can clone this project from my GitHub repository below</p>
{% github raszidzie/Price-Tracker-Application %}

<h3>Mission Accomplished!</h3>
<p>That's it! Make sure you are following me on social media and see you in next post soon DEVs!</p>

<a href="https://reversepython.net/">Reverse Python</a>
<a href="https://www.instagram.com/coderasha/?hl=en">Instagram</a>
<a href="https://twitter.com/coderasha?lang=en">Twitter</a>