Sentiment Analysis with Finetuned Models
Tags: ai, machinelearning, datascience, programming
Published: 2023-06-05 21:50:42

<h3>[tta_listen_btn]

What is Sentiment Analysis?</h3>
<a class="attrlink" href="https://www.youtube.com/watch?v=i4D5DZ5ZG-0" target="_blank" rel="noopener noreferrer nofollow">Sentiment analysis</a> is a natural language processing technique to determine the emotional tone behind a series of words, phrases or sentences.

In recent times, sentiment analysis has gained popularity in various fields, such as customer service, brand monitoring, and social media analysis.

With pre-trained language models, such as BERT (Bidirectional Encoder Representations from Transformers), it has become easier to perform sentiment analysis tasks.

This article will explore how to fine-tune a pre-trained BERT model for sentiment analysis using Hugging Face and push it to the Hugging Face model hub.

All the code for this project is available here: <a class="attrlink" href="https://github.com/ikoghoemmanuell/Sentiment-Analysis-with-Finetuned-Models/tree/main" target="_blank" rel="noopener noreferrer nofollow">GitHub</a>
<h3>Why Hugging Face?</h3>
Hugging Face is a platform that provides a comprehensive set of tools and resources for natural language processing (NLP) and machine learning tasks.

It offers a user-friendly interface and a wide range of pre-trained models, datasets, and libraries that can be utilized by data analysts, developers, and researchers.

Hugging Face offers a vast collection of pre-trained models trained on large datasets and designed to perform specific NLP tasks such as text classification, sentiment analysis, named entity recognition, and machine translation.

These models provide a starting point for your analysis and save you time and effort in training models from scratch.

For this project, I recommend you take <a class="attrlink" href="https://huggingface.co/learn/nlp-course/chapter1/1" target="_blank" rel="noopener noreferrer nofollow">this course</a> to learn all about natural language processing (NLP) using libraries from the Hugging Face ecosystem.

Please, <a class="attrlink" href="https://huggingface.co/" target="_blank" rel="noopener noreferrer nofollow">go to the website and sign</a> <a class="attrlink" href="https://huggingface.co/" target="_blank" rel="noopener noreferrer nofollow">in</a> to access all the features of the platform.

<a class="attrlink" href="https://huggingface.co/tasks/text-classification" target="_blank" rel="noopener noreferrer nofollow">Read more about Text classification with Hugging Face</a>
<h3>Using GPU Runtime on Google Colab</h3>
Before we start with the code, it's important to understand why using <a class="attrlink" href="https://www.youtube.com/watch?v=ovpW1Ikd7pY" target="_blank" rel="noopener noreferrer nofollow">GPU runtime on Google Colab</a> is beneficial.

GPU stands for Graphical Processing Unit, a powerful hardware designed for handling complex graphics and computations.

The Hugging face models are Deep Learning based, so will need a lot of computational GPU power to train them.

Please use <a class="attrlink" href="https://colab.research.google.com/" target="_blank" rel="noopener noreferrer nofollow">Colab</a> to do it, or another GPU cloud provider, or a local machine having NVIDIA GPU.

In our project, we utilized the GPU runtime on Google Colab to speed up the training process.

To access a GPU on Google Colab, we only need to select the GPU runtime environment when creating a new notebook.

This allows us to fully utilise the GPU's capabilities and complete our training tasks much faster.
<h3><img src="https://cdn-images-1.medium.com/max/800/1*1NJACD6Geh69ttzA0F09rQ.gif" align="left" /></h3>
changing runtime to GPU
<h3>Setup</h3>
Now that we understand the importance of GPU let’s dive into the code. We begin by installing the transformers library, a Python-based library developed by Hugging Face.

This library provides a set of pre-trained models and tools for fine-tuning them. We’ll also install other requirements too.
<pre><code class="language-python">!pip install transformers
!pip install datasets
!pip install --upgrade accelerate
!pip install sentencepiece</code></pre>
Next, we import the necessary libraries and load the dataset. In this project, we will use the dataset from this <a class="attrlink" href="https://zindi.africa/competitions/to-vaccinate-or-not-to-vaccinate" target="_blank" rel="noopener noreferrer nofollow">Zindi Challenge</a> , which can be downloaded <a class="attrlink" href="https://github.com/ikoghoemmanuell/Sentiment-Analysis-with-Finetuned-Models/tree/main/data" target="_blank" rel="noopener noreferrer nofollow">here.</a>
<pre><code class="language-python">import huggingface_hub # Importing the huggingface_hub library for model sharing and versioning
import numpy as np
import pandas as pd
from datasets import load_dataset
from sklearn.model_selection import train_test_split
from datasets import DatasetDict, Dataset

from transformers import AutoModelForSequenceClassification
from transformers import TFAutoModelForSequenceClassification
from transformers import AutoTokenizer, AutoConfig
from transformers import TrainingArguments, Trainer, DataCollatorWithPadding

# Load the dataset from a GitHub link
url = "https://raw.githubusercontent.com/ikoghoemmanuell/Sentiment-Analysis-with-Finetuned-Models/main/data/Train.csv"
df = pd.read_csv(url)

# A way to eliminate rows containing NaN values
df = df[~df.isna().any(axis=1)]</code></pre>
After loading the dataset and deleting NaN values, we create the training and validation sets by splitting the preprocessed data.

We also created a PyTorch dataset. PyTorch datasets provide a standard format that is more efficient and convenient to use for our machine-learning process.

By following this dataset format, we can ensure consistency in our data handling and seamlessly integrate with other PyTorch functionalities.
<pre><code class="language-python"># Split the train data =&gt; {train, eval}
train, eval = train_test_split(df, test_size=0.2, random_state=42, stratify=df['label'])

# Create a pytorch dataset 

# Create a train and eval datasets using the specified columns from the DataFrame
train_dataset = Dataset.from_pandas(train[['tweet_id', 'safe_text', 'label', 'agreement']])
eval_dataset = Dataset.from_pandas(eval[['tweet_id', 'safe_text', 'label', 'agreement']])
# Combine the train and eval datasets into a DatasetDict
dataset = DatasetDict({'train': train_dataset, 'eval': eval_dataset})
# Remove the '__index_level_0__' column from the dataset
dataset = dataset.remove_columns('__index_level_0__')</code></pre>
<h3>Preprocessing</h3>
Next, we clean the text data and tokenize it. Machine learning models can only understand numbers.

Tokenization is necessary for creating numerical representations of text, commonly known as word embeddings.

Word embeddings are dense vector representations that capture the semantic meaning and relationships between words.

These representations enable machines to understand the contextual information and similarities between words, facilitating more advanced NLP tasks.

For preprocessing, we will use two commonly used functions for text preprocessing and label transformation.
<h3><img src="https://paper-attachments.dropboxusercontent.com/s_C7B474B4CDEABA224DBDA4E76947FB3A94E397758B92445353AA73FE6110AC64_1685884719473_image.png" align="left" /></h3>
<ul>
 	<li>The <strong>preprocess</strong> function modifies text by replacing usernames and links with placeholders.

The <strong>transform_labels</strong> function converts labels from a dictionary format to a numerical representation.</li>
</ul>
<h3><img src="https://paper-attachments.dropboxusercontent.com/s_C7B474B4CDEABA224DBDA4E76947FB3A94E397758B92445353AA73FE6110AC64_1685885968605_image.png" align="left" /></h3>
<h3>Tokenization</h3>
<pre><code class="language-python">checkpoint = "cardiffnlp/twitter-xlm-roberta-base-sentiment"
# define the tokenizer
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
def tokenize_data(example):
    return tokenizer(example['safe_text'], padding='max_length')
# Change the tweets to tokens that the models can exploit
dataset = dataset.map(tokenize_data, batched=True)
# Transform labels and remove the useless columns
remove_columns = ['tweet_id', 'label', 'safe_text', 'agreement']
dataset = dataset.map(transform_labels, remove_columns=remove_columns)</code></pre>
<ol>
 	<li>We define the checkpoint variable, which holds the name or identifier of the pre-trained model we want to use. In this case, it's the "cardiffnlp/twitter-xlm-roberta-base-sentiment" model.</li>
</ol>
<ol start="2">
 	<li><strong>tokenizer = AutoTokenizer.from_pretrained(checkpoint)</strong>: We create a tokenizer object using the <strong>AutoTokenizer</strong> class from the <strong>transformers</strong> library. The tokenizer is responsible for converting text data into numerical tokens that the model can understand.</li>
</ol>
<ol start="3">
 	<li><strong>def tokenize_data(example)</strong>: We define a function called <strong>tokenize_data</strong> that takes an example from the dataset as input. This function uses the tokenizer to tokenize the text in the example, applying padding to ensure all inputs have the same length.</li>
</ol>
<ol start="4">
 	<li><strong>dataset = </strong><a href="http://dataset.map" target="_blank" rel="noopener noreferrer nofollow"><strong>dataset.map</strong></a><strong>(tokenize_data, batched=True)</strong>: We apply the <strong>tokenize_data</strong> function to the entire dataset using the <strong>map</strong> method. This transforms the text data in the 'safe_text' column into tokenized representations, effectively preparing the data for model consumption. The <strong>batched=True</strong> parameter indicates that the mapping operation should be applied in batches for efficiency.</li>
</ol>
<ol start="5">
 	<li><strong>remove_columns = ['tweet_id', 'label', 'safe_text', 'agreement']</strong>: We create a list called <strong>remove_columns</strong> that contains the names of the columns we want to remove from the dataset.</li>
</ol>
<ol start="6">
 	<li><strong>dataset = </strong><a href="http://dataset.map" target="_blank" rel="noopener noreferrer nofollow"><strong>dataset.map</strong></a><strong>(transform_labels, remove_columns=remove_columns)</strong>: We apply another transformation to the dataset using the <strong>map</strong> method. This time, we use the <strong>transform_labels</strong> function to transform the labels in the dataset, mapping them to numerical values. Additionally, we remove the columns specified in the <strong>remove_columns</strong> list, effectively discarding them from the dataset.</li>
</ol>
By tokenizing the text data and transforming the labels while removing unnecessary columns, we preprocess the dataset to prepare it for training or evaluation with the sentiment analysis model.
<h3>Training</h3>
Now that we have our preprocessed data, we can fine-tune the pre-trained model for sentiment analysis. We will first specify our training parameters.
<pre><code class="language-python"># Configure the trianing parameters like `num_train_epochs`: 
# the number of time the model will repeat the training loop over the dataset
training_args = TrainingArguments("test_trainer", 
                                  num_train_epochs=10, 
                                  load_best_model_at_end=True, 
                                  save_strategy='epoch',
                                  evaluation_strategy='epoch',
                                  logging_strategy='epoch',
                                  logging_steps=100,
                                  per_device_train_batch_size=16,
                                  )</code></pre>
We set the hyperparameters for training the model, such as the number of epochs, batch size, and learning rate.

We’ll load a pre-trained model, shuffle the data, and then define the evaluation metric. In this case, we are using rmse.
<pre><code class="language-python"># Loading a pretrain model while specifying the number of labels in our dataset for fine-tuning
model = AutoModelForSequenceClassification.from_pretrained(checkpoint, num_labels=3)

train_dataset = dataset['train'].shuffle(seed=24) 
eval_dataset = dataset['eval'].shuffle(seed=24) 

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    return {"rmse": mean_squared_error(labels, predictions, squared=False)}</code></pre>
We can easily train and evaluate our model using the provided training and evaluation datasets by initialising the Trainer object with the parameters below.

The <strong>Trainer</strong> class handles the training loop, optimization, logging, and evaluation, making it easier for us to focus on model development and analysis.
<pre><code class="language-python">trainer = Trainer(
    model,
    training_args, 
    train_dataset=train_dataset, 
    eval_dataset=eval_dataset,
    tokenizer=tokenizer,
    compute_metrics=compute_metrics,
)</code></pre>
Finally, we can train our model using.

<code>trainer.train()</code>

And launch the final evaluation using.

<code>trainer.evaluate() </code>

Find below a simple example, with just 10 epochs of fine-tuning.

<a class="attrlink" href="https://nbviewer.org/github/ikoghoemmanuell/Sentiment-Analysis-with-Finetuned-Models/blob/main/dev/Notebooks/finetuning_twitter_xlm_roberta.ipynb" target="_blank" rel="noopener noreferrer nofollow"><strong>Notebook on nbviewer</strong></a>

Read more about the fine-tuning concept <a class="attrlink" href="https://deeplizard.com/learn/video/5T-iXNNiwIs#:~:text=Fine%2Dtuning%20is%20a%20way,perform%20a%20second%20similar%20task." target="_blank" rel="noopener noreferrer nofollow">here</a>.
<h3>Next Steps</h3>
Wondering where to go from here? The next step would be to deploy your model using <a class="attrlink" href="https://huggingface.co/spaces/ikoghoemmanuell/Sentiment-analysis-WebApp" target="_blank" rel="noopener noreferrer nofollow">Streamlit</a> or <a class="attrlink" href="https://huggingface.co/spaces/ikoghoemmanuell/sentiment-Classifier-App-Gradio" target="_blank" rel="noopener noreferrer nofollow">Gradio</a>, for example.

This would be a web application that your users can interact with in order to make predictions. Here are screenshots of two web apps built with the model we just finetuned.
<h3><img src="https://paper-attachments.dropboxusercontent.com/s_C7B474B4CDEABA224DBDA4E76947FB3A94E397758B92445353AA73FE6110AC64_1685895364626_ezgif.com-optimize.gif" align="left" /><img src="https://paper-attachments.dropboxusercontent.com/s_C7B474B4CDEABA224DBDA4E76947FB3A94E397758B92445353AA73FE6110AC64_1685895327786_ezgif.com-crop+1.gif" align="left" /></h3>
<h3>Conclusion</h3>
In conclusion, we have fine-tuned a pre-trained model for sentiment analysis on a dataset using the Hugging Face library. The model achieved a rmse score of 0.7 on the validation set after 10 training epochs.

If you discover this, publish thrilling, discover extra thrilling posts like this on <a class="attrlink" href="http://blog.learnhub.africa/" target="_blank" rel="noopener noreferrer nofollow">Learnhub Blog</a>; we write a lot of tech-related topics from <a class="attrlink" href="https://blog.learnhub.africa/category/cloud-computing/" target="_blank" rel="noopener noreferrer nofollow">Cloud computing</a> to <a class="attrlink" href="https://blog.learnhub.africa/category/frontend/" target="_blank" rel="noopener noreferrer nofollow">Frontend Dev</a>, <a class="attrlink" href="https://blog.learnhub.africa/category/security/" target="_blank" rel="noopener noreferrer nofollow">Cybersecurity</a>, <a class="attrlink" href="https://blog.learnhub.africa/category/data-science/" target="_blank" rel="noopener noreferrer nofollow">AI</a> and <a class="attrlink" href="https://blog.learnhub.africa/category/blockchain/" target="_blank" rel="noopener noreferrer nofollow">Blockchain</a>. Take a look at <a class="attrlink" href="https://blog.learnhub.africa/2023/02/16/how-to-build-offline-web-applications/" target="_blank" rel="noopener noreferrer nofollow">How to Build Offline Web Applications. </a>

This article was written by <a class="ay aj az ba bb am bc w an ao ap aq ar wl" href="https://medium.com/@emmanuel.ikogho?source=post_page-----3456c488aaf9--------------------------------" rel="noopener follow">Emmanuel Ikogho</a>; Emmanuel is a Data Scientist currently training with Azubi Africa to advance his Data Career.
<h3>Resources</h3>
<ul>
 	<li>A q<a class="attrlink" href="https://www.youtube.com/watch?v=CMrHM8a3hqw" target="_blank" rel="noopener noreferrer nofollow">uick intro to NLP</a></li>
 	<li><a class="attrlink" href="https://www.youtube.com/watch?v=QEaBAZQCtwE" target="_blank" rel="noopener noreferrer nofollow">Getting Started With Hugging Face in 15 Minutes</a></li>
 	<li><a class="attrlink" href="https://www.youtube.com/watch?v=5T-iXNNiwIs" target="_blank" rel="noopener noreferrer nofollow">Fine-tuning a Neural Network explained</a></li>
 	<li><a class="attrlink" href="https://www.youtube.com/watch?v=zcW2HouIIQg" target="_blank" rel="noopener noreferrer nofollow">Fine-Tuning-DistilBert - Hugging Face Transformer for Poem Sentiment Prediction | NLP</a></li>
 	<li><a class="attrlink" href="https://www.youtube.com/playlist?list=PLM8wYQRetTxCCURc1zaoxo9pTsoov3ipY" target="_blank" rel="noopener noreferrer nofollow">Introduction to NLP: Playlist</a></li>
</ul>