nlp BD patriots at BLP -2023 Task 2: A Transfer Learning Approach towards B angla Sentiment Analysis (2023)

Abstract: In this paper, we discuss the entry of nlpBDpatriots to some sophisticated approaches for classifying Bangla Sentiment Analysis. This is a shared task of the first workshop on Bangla Language Processing (BLP) organized under EMNLP. The main objective of this task is to identify the sentiment polarity of social media content. There are 30 groups of NLP enthusiasts who participate in this shared task and our best-performing approach for the task is transfer learning with data augmentation. Our group ranked 12^{th} position in this competition with this methodology securing a micro F1 score of 0.71.

## introduction nlp has become a major domain of modern computational research, offering a lot of applications from machine translation to chatbots. however, much of this research has been concentrated on english and other high-resource languages like french, german, and spanish. bangla, despite being the seventh most spoken language in the world with approximately 273 million speakers @xcite , has not received similar attention from the nlp community. this gulf is not just an academic oversight; it has realworld implications. bangla is a language of significant cultural heritage and economic activity. the development of nlp technologies for bangla is both a scientific necessity and a practical imperative. the limited availability of bangla nlp resources has led to a reliance on traditional machine learning techniques like svms and naive bayes classifiers for classification tasks such as sentiment analysis. the advent of deep learning models has opened new avenues. models like bert (devlin *these three authors contributed equally to this work. warning: this paper contains examples that are offensive in nature. @xcite have shown promising results in languages other than english and has been recently trained to support bangla @xcite . sentiment analysis is increasingly becoming a vital tool for understanding public opinion and people's behavior @xcite . it has found applications in various sectors, including finance, where it helps investors to leverage social media data for better investment decisions @xcite . in the context of bangla, the utility of sentiment analysis extends beyond mere academic interest. it can serve as a powerful tool for businesses to gauge customer satisfaction, for policymakers to understand public sentiment, and even for social scientists studying behavioral trends. in this paper, we evaluate several models and implement transfer learning for the shared task on sentiment analysis of bangla social media posts organized at the first workshop on bangla language processing (blp) @xcite . moreover, an ensemble model consisting of three transformer-based models generates a superior performance over the other approaches. rahman et al. (2020) presented an approach using the transformer-based models bert and electra with transfer learning. the models were fine-tuned on three bangla datasets. similarly, rahman et al. (2020) explored characterlevel deep learning models for bangla text classification, testing convolutional neural networks (cnn) and long short-term memory (lstm) models. on the other hand, for opinion mining, haque et al. (2019) analyzed bangla and phonetic bangla restaurant reviews using machine learning on a dataset of 1500 reviews. svm achieved the highest accuracy of 75.58%, outperforming prior models. advancements of sentiment analysis in bangla islam et al. (2020) presented two new bangla sentiment analysis datasets which achieved state-of-theart results with multi-lingual bert (71% accuracy for 2-class, 60% for 3-class), and notes sentiment differences in newspaper comments. tuhin et al. (2019) proposed two bangla sentiment analysis methods: naive bayes and a topical approach, aiming at six emotions, which achieved over 90% accuracy for sentence-level emotion classification, outperforming naive bayes. similarly, al kaiser et al. ( 2021 ) discussed research focused on sentiment analysis and hate speech detection in bangla language facebook comments; compiling a dataset of over 11,000 comments, categorized by polarity (positive, negative, neutral) and various sentiment types, including gender-based hate speech. furthermore, there are researches conducted on sentiment analysis in the field of online bangla reviews. for example, @xcite detected depression in bangla social media using sentiment analysis. they preprocessed a small dataset and employed machine learning classifiers, but faced limitations due to the dataset's size and basic classifiers. akter et al. ( 2021 ) used machine learning for bangla e-commerce review sentiment analysis, with knn achieving 96.25% accuracy, outperforming other classifiers. this highlighted machine learning's potential in analyzing bangla ecommerce reviews. whereas, @xcite introduced a bangla movie review sentiment analysis system using 800 annotated social media reviews. @xcite introduced a significant dataset of 33,605 manually annotated bangla social media posts and examined how different language models perform in zero-and few-shot learning situations. thus, the research of sentiment analysis is continuously growing, and it's helping us better understand sentiment in bangla online content. ## dataset the dataset provided for the shared task @xcite , consists of a training set, a development set, and a blind test set. for each set, the texts have been annotated using three labels -'positive', 'neutral', or 'negative' @xcite . the label distribution for each set is provided in ## experiments we conduct a wide range of experiments with several models and data augmentation strategies. our experiments include statistical models, transformer-based models; data augmentation strategies like back-translation, multilinguality and also prompting proprietary llms. ## results and analysis at the start of the share task competition, 3 baseline micro f1 scores are provided by the organizers. for random selection the provided baseline is 0.34, for majority selection 0.50, and n-gram 0.55. the results of different models are given in amongst the statistical machine learning models, we use logistic regression and support vector machine. for logistic regression, we achieve a micro f1 score of 0.45 and for the support vector machine, the f1 is 0.55. banglabert, muril and xlm-r where we get the best f1 score of 0.70 by xlm-r. a few shot learning procedure is used by using gpt3.5 turbo. we give a few instances of each label as prompt and got 0.51 f1 which is significantly lower than our other attempted approaches except logistic regression. it is because gpt3.5 is still not efficient enough for any downstream classification problem in bangla like this shared task. moreover, we augment the data of bangla youtube sentiment and emotion dataset by @xcite . the dataset has highly positive, positive labels which we consider as positive and negative, highly negative labels which we consider negative. we keep the neutral label unchanged. this is how we get three labels out of five labels and merge it with our train data. following this procedure, we finally achieve micro f1 score of 0.71 which we this shared task's leader board. ## error analysis the classification report provides a comprehensive understanding of our model's performance across the three classes. the overall accuracy of the model is 0.71. the 'positive' class has the highest f1score of 0.78, driven by a precision of 0.75 and a recall of 0.80. the 'neutral' class, on the other hand, shows a relatively weaker performance with an f1-score of 0.42, a result of its lower precision and recall, 0.51 and 0.37 respectively. the 'negative' class offers a competitive performance with an f1-score of 0.74, a precision of 0.72, and a recall of 0.76. on a macro level, the average values indicate a precision of 0.66, recall of 0.64, and an f1-score of 0.65. when weighted by support, the averages show a slightly better picture with precision at 0.69, recall identical to the overall accuracy at 0.71, and an f1-score of 0.70. further dissecting the errors by text length offers more insights. texts with lengths in the range of 50 to 100 characters contribute the most to the dataset, constituting 43.73% of the samples, and have an f1-score of 0.74. the second largest group, texts ranging from 20 to 50 characters, contribute 26.64% to the dataset with a slightly better f1score of 0.70. it is also worth noting that the performance drastically reduces for texts with lengths between 500 and 1000 characters, yielding the lowest f1-score of 0.39, albeit they only make up 0.73% of the samples. few misclassified examples are given in figure 4 . figure 4: few examples of misclassified labels ## conclusion in this shared task, we use statistical machine learning models, transformer-based models, a few shot prompting, some customization with transformerbased models with transfer learning, data augmentation, and an ensemble-based approach. the transfer learning and data augmentation procedure is reported as the most successful approach in terms of a micro f1 score of 0.71. but additional experiments by doing an ensemble over three transformer-based models provide a 0.72 f1 score. overall, this paper can be treated as a holistic experimental outcome for this shared task.