Z -Index at BLP -2023 Task 2: A Comparative Study on Sentiment Analysis (2023)

Abstract: In this study, we report our participation in Task 2 of the BLP-2023 shared task. The main objective of this task is to determine the sentiment (Positive, Neutral, or Negative) of a given text. We first removed the URLs, hashtags, and other noises and then applied traditional and pretrained language models. We submitted multiple systems in the leaderboard and BanglaBERT with tokenized data provided thebest result and we ranked 5th position in the competition with an F1-micro score of 71.64. Our study also reports that the importance of tokenization is lessening in the realm of pretrained language models. In further experiments, our evaluation shows that BanglaBERT outperforms, and predicting the neutral class is still challenging for all the models.

## introduction sentiment analysis is one of the most modern and sophisticated natural language processing (nlp) applications. it is used for analyzing how people feel about the words they write in publicly accessible spaces like social media in the form of posts or comments. social networking sites and other ways to use digital technology are commonly used to post a lot of information about feelings, ideas, and actions. access to such a great amount of data provides the researchers the advantage to analyze the contents in order to help make decisions to process and understand the sentiment of a product and system, or views on social, international, cultural, and political agendas @xcite . the majority of current research is limited to resource-rich languages due to the availability of resources. the interest in low-resource languages is growing over time in sentiment analysis @xcite @xcite . unlike other languages, a limited number † the authors contributed equally to this work of study has been done to develop resources for bangla sentiment analysis @xcite @xcite @xcite . from the perspective of modeling, there have been studied both classical (i.e., svm, rf, naive bayes) and deep learning (i.e., cnn, lstm) models. pretrained language models (i.e., bert, xlm-roberta, distilbert) have also been studied in recent years @xcite for sentiment classification. due to the availability of public data and inadequate information on annotation agreements @xcite , it is challenging for the researchers to focus on this area. this shared task provides a dataset by combining perfect and moderate agreement to shed light on sentiment analysis. in this study, we participated in the sentiment analysis shared task at blp-2023 and worked on a multiclass dataset where the class labels are positive, negative, and neutral. we utilize both classical and transformer-based pretrained language models. for the classical model, we choose support vector machine (svm) and random forest (rf). we fine-tuned bert multilingual, banglabert base, and banglabert large pretrained language models to train and evaluate models. our findings from the study conclude as: ## literature review researchers are increasingly interested in investigating sentiment analysis utilizing social media data as a result of the rise of social media. the development of sentiment analysis began in the early 2000s @xcite . early research includes rule-based and classical methodologies whereas recent studies include deep learning-based and pretrained language models. researchers have been trying to develop resources over time and as a result, manual and semi-supervised approaches @xcite @xcite @xcite have been adopted in developing sentiment classification datasets. chowdhury and chowdhury (2014 ) used a semi-supervised technique to annotate data and train classical models. the study by @xcite constructs a dataset using manual annotations done by the annotators and presents 15,000 data in 13 domains. @xcite in their work, used the absa dataset consisting of human-annotated user comments on cricket and customer reviews of restaurants where svm offered the maximum precision rate for both datasets. @xcite developed a sentiment classification system utilizing svm and naive bayes for textual movie reviews in bangla and provided comparative results. additionally, naive bayes with rules has been studied by @xcite for bangla facebook statuses sentiment classification. @xcite worked on 10,000 post-processed text samples in both bangla and romanization of bangla and by experimenting with lstm, the authors achieved the maximum accuracy score of 55%. @xcite conducted comparison experiments using various datasets that existed in the literature to understand model performances, training difficulties, and consequences for real-world deployment. in this study, deep learningbased models outperform traditional models. furthermore, @xcite used the most sophisticated techniques currently available to compare datasets and conclude that xlm-roberta exhibits the best performance over other deep learning approaches. classifying the tweets of positive, negative, and neutral polarity was the major goal of @xcite . various well-known supervised classification methods have been studied in this study. tripto and ali (2018) used lstm for identifying sentiment and emotions in bangla writings achieving an accuracy of 65.97 and 54.24 for three and five classes respectively. chowdhury et al. (2019) providing a method for conducting sentiment analysis on bangla-language movie reviews that can automatically analyze viewer responses to a certain film or television program was the main work and the authors used social media websites' publicly accessible comments and posts serving as the source of the dataset that was manually compiled and labeled for this experiment. focusing on the largest publicly available dataset mubase @xcite consolidated from social media data consisting of 33,605 tweets and facebook comments about bangla news and carried out experiments that went beyond traditional approaches and smaller transformer-based models. the authors focused on the efficiency of sophisticated algorithms in zero-and few-shot conditions, including flan-t5, gpt-4, and bloomz. the findings show that while llms are an interesting study area, smaller variations of precise pre-trained models perform better. in the context of sentiment analysis @xcite provides a commonsensebased neurosymbolic framework that seeks to address these problems. they evaluated senticnet 7 and concluded that of all 20 lexica, senticnet 7 was the most effective. @xcite worked with the manually produced and labeled datasets that were obtained from social media. the accuracy achievement at the end of 140 epoch with the best performance using the nadam optimizer. we utilized the dataset provided by the organizers of the blp-2023 for task 2: sentiment analysis @xcite . the goal is to identify the sentiment contained within a text. the dataset is consolidated from two distinct sources, i) mubase @xcite and ii) sent-nob @xcite consisting of social media tweets, posts, and comments. in this dataset, there are three columns, id refers to sentence id, text refers to input text, and label containing positive, neutral, and negative tags. in ## results and discussion the official overall ranking and results determined by the lab organizers are presented in f1 score of 73.10. our system also performed better than both the majority and random baseline with a large margin of 21.87 and 38.08 respectively. the detailed results of all the performed experiments are presented in in in our study, we found that monolingual pretrained language provides superior performance compared with the multilingual pretrained language model. we achieved an f1 score of 66.81 with respect to the positive class using bertmultilingual while the banglabert-base model has an f1 score of 75.59 with respect to the positive class which demonstrates the superior performance of the monolingual pretrained language model. we also observed that the base model outperforms the large model. the large model has more trainable parameters than the base model and the amount of data is not sufficient to train and overfit the large model. ## conclusion in this study, we run comparative experiments and analysis on the bangla sentiment dataset provided by the task organizers of blp-2023. we presented a detailed comparison of the fine-tuned models along with traditional models. comparing the traditional model, we found that svm outperforms rf with a margin of 1.47%. banglabert outperforms all the models we used in our study. our study also reveals that tokenization has little to no control over performance during the use of pretrained language models. in the submission of task 2 on the sentiment analysis dataset, we ranked 5 th position among all the participants. to extend this work, we will employ large language models (llms) and gpt-based models for comparative and in-depth sentiment analysis.