Semantics Squad at BLP -2023 Task 2: Sentiment Analysis of B angla Text with Fine Tuned Transformer Based Models (2023)

Abstract: Sentiment analysis (SA) is a crucial task in natural language processing, especially in contexts with a variety of linguistic features, like Bangla. We participated in BLP-2023 Shared Task 2 on SA of Bangla text. We investigated the performance of six transformer-based models for SA in Bangla on the shared task dataset. We fine-tuned these models and conducted a comprehensive performance evaluation. We ranked 20th on the leaderboard of the shared task with a blind submission that used BanglaBERT Small. BanglaBERT outperformed other models with 71.33% accuracy, and the closest model was BanglaBERT Large, with an accuracy of 70.90%. BanglaBERT consistently outperformed others, demonstrating the benefits of models developed using sizable datasets in Bangla.

## introduction social networking sites' widespread use in the digital age has produced an unheard-of influx of usergenerated content. these sites act as gathering places where people can publicly express their opinions and feelings. it has become popular to identify and measure the emotional tone in textual data through sentiment analysis (sa), a key component of natural language processing (nlp). while sa has been extensively studied for resource-rich languages like english, it is still largely unexplored for many low-resource languages like bangla. understanding public opinion is crucial for making well-informed decisions in democratic countries. developing efficient sa tools for the bangla language has not been possible due to the lack of sa resources, such as datasets and evaluation benchmarks. this study is devoted to sa and focuses specifically on bangla being the 7 th most spoken language globally @xcite , and its use on social media sites, particularly facebook, x, and youtube, has increased significantly. while much research has been conducted in sa, most attempts have been based on traditional machine learning (ml). traditional ml techniques have drawbacks in feature engineering, representation learning, scalability, and handling sequential data. they perform best when working with structured data that has clearly defined features. in contrast, deep learning (dl) models like transformers have excelled at a variety of tasks, especially when dealing with unstructured data like natural language text. despite the enormous amount of data generated on social media platforms, not many bangla benchmark datasets are available. this study addresses this gap by concentrating on the sa of bangla text in the context of social media. we employ multiple state-of-the-art pretrained transformer models: multilingual bert (bidirectional encoder representations from transformers) @xcite , xlm-roberta @xcite , banglabert @xcite , banglishbert @xcite , fine-tuned for sa in bangla. we use the dataset provided in the shared task 2 of blp-2023 @xcite of bangla text for sa in order to thoroughly assess the efficacy of these models. we measure and report the accuracy, precision, recall, and f1-score as important performance metrics used for sa evaluation. according to the performance matrices, monolingual models such as banglabert banglabert large outperform other transformer-based models. we secured the 20th position with the submission of banglabert small of micro f1 score of 67.42%. the shared task followed a blind submission process, meaning the last submission was considered the final submission. later, we reran the experiment with models including multilingual bert, xlm-roberta base, banglish-bert, banglabert small, banglabert and banglabert large. the best-performing system on the leaderboard achieved a micro f1 score of 73.1%, while banglabert in our work achieved a micro f1 score of 70.62%, which is in close proximity to the best system's performance and significantly exceeds the baseline. banglabert large also achieved an f1 score of 70.34%, and closely approaches the performance of the best system. other models in our study did not achieve the same performance level as banglabert and banglabert large. ## related work there have been many attempts to address nlp tasks with traditional machine learning (ml). however, it has limitations related to feature engineering, representation learning, scalability, and handling unstructured data. in contrast, transformerbased models can capture contextual information, rely on pre-trained representations, and can be applied to various languages and domains. hence, we focus on nlp tasks that were addressed with deep learning (dl) and language models (lm). several attempts have been made by the researchers to develop resources for sa @xcite @xcite , one of the most comprehensive and rigorous overviews of bangla nlp tasks was conducted by @xcite and @xcite . they provided a comparative analysis of bangla nlp tasks using both classical machine learning algorithms and transformerbased pre-trained models. their study demonstrates that transformer-based pre-trained models outperform traditional machine learning algorithms. in order to address the lack of high-quality bangla sa datasets, @xcite developed a dataset that focuses on attitudes toward the conflict between russia and ukraine. they fine-tuned various transformer-based models and achieved the best performance with 86% accuracy and 82% f-1 score using banglabert. @xcite introduced two manually tagged sa datasets and a dl model called bertbsa. ## experimental methodology this section outlines our experimental methodology. we begin with an overview of the dataset, followed by a discussion of our pre-processing procedures, and conclude by presenting detailed descriptions of the models employed in our study. data: we used the dataset that was offered in the blp shared task 2. the dataset employed for this shared task is a combination of bangla text data from two distinct sources: mubase @xcite and @xcite . sentnob is a compilation of public comments extracted from various social media platforms, spanning 13 domains such as politics, education, and agriculture, and manually annotated. the level of agreement among annotators for this dataset is moderate, with an agreement score of 0.53. on the other hand, the mubase dataset comprises a comprehensive col- ## results analysis and discussion to determine which models were most effective and could be applied to real-life sa problems, we fine-tuned and applied the m-bert, xlm-roberta base, banglishbert, banglabert small, banglabert and banglabert large models. in particular, banglabert consistently outperformed the other models in terms of various performance metrics. however, we were unable to extract insights into why banglabert exhibited superior performance compared to m-bert and xlm-roberta models. it is possible that banglabert's training on a substantial bangla dataset provided a slight advantage over the other multi-lingual models. the superior performance of banglabert indicates that models specifically trained on a sizable bangla dataset have a natural advantage when identifying subtle sentiment nuances in bangla text. this may be attributed to the fine-tuning process used by banglabert, which allowed it to better comprehend the nuances of bangla language and sentiment expression. however, despite being intended to be multi-lingual models, m-bert and xlm-roberta may not have fully adapted to the nuances of the bangla language, which resulted in their comparatively poorer performance. although banglabert outperformed the other models, our study could not pinpoint the precise causes of this performance disparity. for example, despite being larger and having three times more parameters than banglabert, banglabert large could not perform as expected. the observed behavior may be attributed to several potential factors within the context of the data provided for shared task 2 of blp-2023. one likely contributor could be the inadequacy of the data structure for the models to perform optimally. another possibility is that the pre-processing steps applied to the data may not have been sufficient to enable the models to achieve their expected levels of performance. additionally, the choice of hyper-parameters for the models, including the fine-tuning process, might not have been optimal, potentially impacting their overall performance. ## conclusion and future work this study conducted a comprehensive evaluation of fine-tuned transformer-based models for sentiment analysis (sa) in bangla text. the importance of models specifically trained on large bangla datasets for sa tasks is highlighted by banglabert's consistent and superior performance across a variety of performance metrics. the advantage that banglabert showed over the multi-lingual models, m-bert and xlm-roberta, suggests that a deeper comprehension of the bangla language and sentiment expression is crucial for obtaining accurate sa results. the precise linguistic and contextual factors contributing to banglabert's superior sa abilities need to be further investigated. in our future research endeavors, we aim to delve deeper into why transfer-based multi-lingual models struggled to compete with banglabert, further enhancing our understanding of their performance disparities.