Building the classifier - Part I (Live tweet sentiment analysis)
Tags: python, machinelearning
Published: 2020-08-14 11:18:18

In this series, I will show how to create a sentiment analysis app and perform analysis on any hashtag. The series is divided into 3 parts
- Building the classifier
- Building the backend
- Building the frontend

In the final post, we'll bring everything together to make the app. We'll use tools like `nltk`, `docker`, `streamlit`, `fastAPI` and the link to the code will be provided.

### Final product
![sentwitter](https://j.gifs.com/4Q01k7.gif)

### Building the classifier

We'll be using a pre-trained model that I trained using `open-source` code. This is not a SOTA(State-of-the-Art) model, but for our task, this should be fine. 

Let's begin by making a project directory.
```bash
mkdir sentwitter && cd sentwitter
mkdir backend
mkdir frontend
```
Download the trained model to `backend/models` directory
```bash
wget https://raw.githubusercontent.com/amalshaji/sentwitter/master/backend/models/sentiment_model.pickle

mv sentiment_model.pickle /backend/models
```

#### Install required libraries
```bash
python3 -m pip install nltk
python3 -m nltk.downloader punkt
python3 -m nltk.downloader wordnet
python3 -m nltk.downloader stopwords
python3 -m nltk.downloader averaged_perceptron_tagger
```

#### Let's write a function to pre-process the input(tweet)

```python
# backend/classify.py
import re, string
from nltk.tag import pos_tag
from nltk import WordNetLemmatizer
from nltk.tokenize import word_tokenize


def remove_noise(tweet_tokens, stop_words=()):

    cleaned_tokens = []

    for token, tag in pos_tag(tweet_tokens):
        token = re.sub(
            "http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+#]|[!*\(\),]|"
            "(?:%[0-9a-fA-F][0-9a-fA-F]))+",
            "",
            token,
        )
        token = re.sub("(@[A-Za-z0-9_]+)", "", token)
        # remove all the links and special characters

        if tag.startswith("NN"):
            pos = "n"
        elif tag.startswith("VB"):
            pos = "v"
        else:
            pos = "a"

        lemmatizer = WordNetLemmatizer()
        token = lemmatizer.lemmatize(token, pos)
        # pos tagging

        if (
            len(token) > 0
            and token not in string.punctuation
            and token.lower() not in stop_words
        ):
            cleaned_tokens.append(token.lower())
        # remove stopwords and punctuations

    return cleaned_tokens

```

#### Write a helper function to load the model.

```python
# backend/utils.py

import pickle

def load_model():
    f = open("./models/sentiment_model.pickle", "rb")
    classifier = pickle.load(f)
    f.close()

    return classifier
```

#### Test out classifier
```python
# backend/test.py

from utils import load_model
from classify import remove_noise
from nltk.tokenize import word_tokenize

model = load_model()

while True:
    _input = input("Enter a sample sentence: ")
    custom_tokens = remove_noise(word_tokenize(_input))
    result = model.classify(dict([token, True] for token in custom_tokens))
    print(f"{_input}: {result}")
```

#### Output
```bash
❯ python .\test.py
Enter a sample sentence: I am awesome
I am awesome: Positive
Enter a sample sentence: I hate you
I hate you: Negative
Enter a sample sentence: I have a gun
I have a gun: Positive
Enter a sample sentence: I like your hair
I like your hair: Negative
```

This isn't the best model😂, gun labeled as `Positive` and hair sentence as `Negative`. Feel free to build your own model or try with the heavy ones using transformers( [huggingface](https://huggingface.co/models)). 

In the next post, we'll be building the backend to serve predictions through an API.

### References
- The `open-source` code used to train the model was used a long time ago for a project, I can't find the source. So if you do, or you are the author, please comment the link.
- Article Cover by  [MorningBrew](https://unsplash.com/@morningbrew) 
- Series Cover by [Ravi Sharma](https://unsplash.com/@ravinepz)