E mpty M ind at BLP -2023 Task 2: Sentiment Analysis of B angla Social Media Posts using Transformer-Based Models (2023)

Abstract: With the popularity of social media platforms, people are sharing their individual thoughts by posting, commenting, and messaging with their friends, which generates a significant amount of digital text data every day. Conducting sentiment analysis of social media content is a vibrant research domain within the realm of Natural Language Processing (NLP), and it has practical, real-world uses. Numerous prior studies have focused on sentiment analysis for languages that have abundant linguistic resources, such as English. However, limited prior research works have been done for automatic sentiment analysis in low-resource languages like Bangla. In this research work, we are going to finetune different transformer-based models for Bangla sentiment analysis. To train and evaluate the model, we have utilized a dataset provided in a shared task organized by the BLP Workshop co-located with EMNLP-2023. Moreover, we have conducted a comparative study among different machine learning models, deep learning models, and transformer-based models for Bangla sentiment analysis. Our findings show that the BanglaBERT (Large) model has achieved the best result with a micro F1-Score of 0.7109 and secured 7th position in the shared task 2 leaderboard of the BLP Workshop in EMNLP 2023.

## introduction nowadays, social media platforms produce a large amount of text data by posting, commenting, and messaging. finding the sentiment of social media data is an active research area among practitioners due to its numerous practical applications. however, conducting sentiment analysis on social media data is a difficult task due to the natural variation of writing patterns among users. a significant amount of effort has been devoted to analyzing sentiment in social media data for resourced enriched languages like english @xcite . however, we have found a limited number of relevant studies focused on sentiment analysis in the bangla language due to the lack of a standardized annotated dataset of bangla text sourced from social media platforms @xcite . the main objective of this research work is to analyze sentiment on bangla social media posts. moreover, we conduct a comparative analysis among different ml, dl, and transformer-based models for bangla sentiment analysis. to train and evaluate different models, we have utilized a dataset provided in a shared task named sentiment analysis of bangla social media posts organized by the first workshop on bangla language processing co-located with emnlp-2023 @xcite . various ml models and dl models have been deployed for bangla sentiment analysis. we have utilized three popular transformer-based model architectures named bangla bert base, banglabert, and banglabert (large) for the sentiment analysis model. among the machine learning models, svm utilizing tf-idf yields the best performance, achieving a micro f1-score of 0.57. in the realm of deep learning, the bilstm + cnn model with word2vec attains the highest micro f1-score at 0.61. the transformer-based banglabert (large) models @xcite outperform the rest, achieving an impressive micro f1-score of 0.7109. the implementation of our research work has been shared in the following github repository - https://github.com/ml-emptymind/ blp-task2 . ## related work we divide all the previous works related to sentiment analysis into three different categories: ml approaches, dl approaches, and transformer-based approaches. machine learning techniques like svm, multinomial naive bayes, knn, logistic regression, decision trees, and random forests are used for sentiment analysis. among these, svm and multinomial naive bayes classifiers @xcite have demonstrated the best performance, with svm achieving the highest accuracy scores. the dataset is subsequently transformed using a tf-idf vectorizer, and svm is used as the classifier for data classification @xcite . numerous deep-learning techniques are employed for sentiment analysis as well. rnn with lstm model is used @xcite for sentiment analysis to classify and categorize the sentiments of social media posts about cricket as positive, negative, or neutral. in order to analyze sentiment or opinion in bangla, the attention mechanism is suggested @xcite in the study. it examines the difficulties with sentiment analysis and evaluation, particularly in the bangla language. alongside a deep learning model that utilizes multilingual bert and transfer learning, the research incorporates datasets for two-class and threeclass sentiment analysis that have been manually annotated in bangla, as mentioned in @xcite . this model surpasses the current state-ofthe-art algorithms in terms of accuracy, attaining a 71% accuracy rate for two-class sentiment classification and a 60% accuracy rate for three-class sentiment classification. the approach is also used to examine the tone of reader comments in an online daily newspaper, demonstrating that while comments on religious articles tend to be more positive than those on political or sports news, the former are more numerous for those topics. the objectives of the study include finetuning the transformer-based models for bangla sentiment analysis and providing a comparison analysis with the baseline models using ml and dl approaches. ## dataset during our research work, we have capitalized the dataset provided using the shared task 2 (sentiment analysis of bangla social media posts) organized by the blp workshop @ emnlp 2023 @xcite . the dataset used for this shared task consists of multiplatform bangla sentiment (mubase) @xcite and sentnob @xcite . the mubase dataset is a cross-platform collection of facebook and twitter posts that has been manually annotated with sentiment polarity. the sentnob dataset comprises user comments sourced from social media platforms in response to news articles and videos. the dataset covers several fields, such as politics, education, and agriculture. the provided dataset has three sentiment categories: positive, negative, and neutral. this dataset has train, dev, and test split containing 35266, 3934, and 6707 texts respectively. in the provided dataset contains urls, emojis, and other symbols which are removed in the preprocessing step. ## methodology in this section, we outline the methodology of our research. we establish baseline models by employing both ml and dl techniques. subsequently, we enhance performance by incorporating a transformer-based model. figure 1 shows an overview of our methodology. ## results and analysis in this section, we provide the outcomes obtained from our experimentation. ## conclusion in this research work, we have explored various transformer-based models for analyzing sentiment in the bangla language. to train and evaluate different models, we have employed a dataset made available through the blp workshop in conjunction with emnlp-2023. additionally, we have conducted a comprehensive comparison among different ml, dl, and transformer-based approaches for bangla sentiment analysis. we have found that the banglabert (large) model has outperformed the others, achieving the highest micro f1-score of 0.7109. in the future, we intend to investigate various architectures and employ ensemble methods to enhance model performance. additionally, we will apply different techniques to address issues arising from the use of an imbalanced dataset.