Embeddings at BLP -2023 Task 2: Optimizing Fine-Tuned Transformers with Cost-Sensitive Learning for Multiclass Sentiment Analysis (2023)

Abstract: In this study, we address the task of Sentiment Analysis for Bangla Social Media Posts, introduced in first Workshop on Bangla Language Processing (CITATION). Our research encountered two significant challenges in the context of sentiment analysis. The first challenge involved extensive training times and memory constraints when we chose to employ oversampling techniques for addressing class imbalance in an attempt to enhance model performance. Conversely, when opting for undersampling, the training time was optimal, but this approach resulted in poor model performance. These challenges highlight the complex trade-offs involved in selecting sampling methods to address class imbalances in sentiment analysis tasks. We tackle these challenges through cost-sensitive approaches aimed at enhancing model performance. In our initial submission during the evaluation phase, we ranked 9th out of 30 participants with an F1-micro score of 0.7088 . Subsequently, through additional experimentation, we managed to elevate our F1-micro score to 0.7186 by leveraging the BanglaBERT-Large model in combination with the Self-adjusting Dice loss function. Our experiments highlight the effect in performance of the models achieved by modifying the loss function. Our experimental data and source code can be found here.

## introduction sentiment analysis is an important task in natural language processing that involves automatic detection of expressed opinions within text. the proliferation of online social media interactions has led to a surge in textual content, necessitating strategies to address associated challenges. while sentiment analysis for high-resource languages like english has made significant progress, low-resource languages like bangla are still lagging behind. low resource languages have intricate sentence structures and grammatical rules, making the development of systems resource-intensive. achieving optimal model performance requires substantial annotated data, leading to longer processing times as data volume increases. additionally, when performing multiclass sentiment analysis, there is a common challenge related to class imbalance, which can lead to models exhibiting bias towards particular classes. previous studies have concentrated on improving the quantity of training data instances, although this approach can extend the duration of model training. numerous studies have been undertaken to advance the development of linguistic resources for the bangla language. @xcite introduced sentnob dataset for multiclass sentiment analysis task. @xcite summarized the sentiment analysis task for three indian language , namely bangla, hindi and tamil. they showed the results for shared task on binary sentiment analysis and introduced the sail dataset. @xcite introduced the bengfasttext dataset which was able to capture semantics of bangla words. they experimented their corpus with traditional ml algorithms and also utilized mconv-lstm network to tackle the binary sentiment analysis task. (tripto and ali, 2018) introduced bangla language corpus from bangla youtube comments. @xcite focused on aspect based sentiment analysis and introduced the research community with absa cricket and restaurant datasets. but all of this datasets had class imbalances in their classes. @xcite and @xcite compiled all the previously mentioned datasets and benchmarked their results with different traditional and transformer based models. the ongoing challenge lies in the escalating fine-tuning time due to the increasing data volume. this study seeks to enhance fine-tuned transformer model efficiency by employing cost-sensitive learning to tackle class imbalance problem. our contributions can be summarized as follows: ## system overview recent developments in nlp have seen the emergence of pre-trained transformer models, based on the transformer architecture proposed by @xcite . these models consistently achieve state-of-the-art performance across a wide range of nlp tasks. in our study, we initially fine-tuned multiple pre-trained transformer models using the default cross-entropy loss as our baseline approach. subsequently, we aimed to enhance model performance through cost-sensitive learning, which effectively addresses class imbalances and mitigates biases towards the majority classes. ## experiments and results we explored various model and custom loss function combinations as described in section 4. in this section, we outline the evaluation for the shared task competition, with the f1-micro score as the key performance metric. our model assessments were conducted on the test set, and, as outlined in section 6, we noted improved model performance without text preprocessing as mentioned in appendix 6.2. in our initial experimentation with transformer models, we fine-tuned each model using the default cross-entropy loss function. among the models in our baseline study, banglabert-large stood out, achieving the highest f1-micro score of 0.7101. subsequently, we investigated the impact of costsensitive loss functions on model performance. we implemented focal loss, self-adjusting dice loss, and f1-micro loss. notably, for two models, banglabert-large and mbert, these alternative loss functions led to significant improvements compared to the baseline approach. for banglabert-large, self-adjusting dice loss produced the best result, with an f1-micro score of 0.7186, surpassing all other transformer models used in our research. for mbert, focal loss resulted in improved performance, achieving an f1-micro score of 0.6606. other loss functions for these two models also outperformed the baseline, as shown in the however, for banglagpt2, incorporating costsensitive loss functions did not enhance model performance; the baseline approach yielded the highest f1-micro score at 0.6788. regarding the indicbert model, self-adjusting dice loss improved performance compared to the baseline crossentropy loss, achieving an f1 score of 0.6263. however, focal loss and f1-micro loss did not yield performance improvements for this model. ## ablation study in the scope of our study, we conducted a sequence of experiments to understand key factors affecting our model's performance. ## error analysis ## conclusion this research paper primarily emphasizes the enhancement of transformer-based models' performance through the application of cost-sensitive learning techniques, aimed at alleviating issues related to class imbalance and overfitting. among various combinations of transformers and loss functions explored, the banglabert model utilizing the self-adjusting dice loss exhibited the highest f1 score of 0.7186 on the test dataset. although the combination of cost-sensitive techniques with transformer models led to notable enhancements in performance, it's important to highlight that the model's effectiveness still falls short, especially when it comes to the 'neutral' class.