Ushoshi2023 at BLP -2023 Task 2: A Comparison of Traditional to Advanced Linguistic Models to Analyze Sentiment in B angla Texts (2023)

Abstract: This article describes our analytical approach designed for BLP Workshop-2023 Task-2: in Sentiment Analysis. During actual task submission, we used DistilBERT. However, we later applied rigorous hyperparameter tuning and pre-processing, improving the result to 68% accuracy and a 68% F1 micro score with vanilla LSTM. Traditional machine learning models were applied to compare the result where 75% accuracy was achieved with traditional SVM. Our contributions are a) data augmentation using the oversampling method to remove data imbalance and b) attention masking for data encoding with masked language modeling to capture representations of language semantics effectively, by further demonstrating it with explainable AI. Originally, our system scored 0.26 micro-F1 in the competition and ranked 30th among the participants for a basic DistilBERT model, which we later improved to 0.68 and 0.65 with LSTM and XLM-RoBERTa-base models, respectively.

## introduction sentiment analysis and opinion-mining techniques determine a text's sentiment or emotional polarity and then analyze it @xcite . throughout diverse fields, such as marketing, customer feedback analysis, and social media monitoring, sentiment analysis has gained significant attention in recent years. while sentiment analysis has been extensively studied in languages like english, there is a growing interest in applying this technique to other languages, including bangla. analyzing sentiment in bangla text presents unique challenges due to its complex grammar, script, and nuances. this article aims to explore sentiment analysis in the bangla language with an example dataset provided for the blp workshop competition for task 2 using sequential data analysis models, such as lstm and large language models, along with traditional models. this multi-class classification task determines whether the sentiment expressed in the text is positive, negative, or neutral. even though lstm provides the highest performance among the deep learning models, xlm-roberta-base @xcite uses masked language modeling (mlm) to handle multilingual and cross-lingual tasks, making it a powerful tool for understanding and generating text in multiple languages. mlm is a pre-training objective used in models like xlm-roberta-base. using mlm, a fraction of input tokens are replaced with unique @xcite tokens, and the model is trained to predict the original tokens from the context provided by the surrounding tokens. mlm is a self-supervised learning task where a model learns to understand the statistical properties of the language by making predictions. we provide the competition results on the github foot_0 which was implemented with distillbert. the final implementation with the higher accuracy and comparative analysis on different models is available in the github foot_1 . our rigorous experiments on a dataset and with various models have resulted in the following observations in addition to designing the system. moreover, oversampling with boosting combats the data skew for all the models. ## system overview: experiment and setup this section describes our data preprocessing steps for traditional machine learning models, vanilla deep learning models, and transformers. next, we discuss the training and hyperparameter tuning of each model group. ## evaluations and discussion on results in the original competition, we generated the results using a basic distillbert model without any preprocessing and fine-tuning. distillbert can process maximum 10k data -even batchwise processing and averaging the output scores couldn't give a good result. we improved with a rigorous comparative analysis with vanilla deep learning, transformer-based llms, and traditional machine learning models that can handle large datasets. svm achieved the highest accuracy and f1-score of 75.02% and 74.85% ( xlm-roberta-base better predicts actual positive labels (figure 2 ). however, it reports higher false negative (fn) values for negative classes and more false positive (fp) values for positive classes. in contrast, banglabert reports fewer fp and fn values for each class but fails to predict tp with about 103 data points deviation. therefore, xlm-roberta-base and banglabert performed well on the test dataset. fp and fn for each class in lstm made minimal impact on accuracy because their values show a slightly equal distribution. though lstm generates better accuracy than the transformer model, transformers produce more correct instances for negative and other classes. in figure 4 , the learning curve backs up the finding of the unstable nature of the lstm model, showing how it is underfitting. we saw the similar pattern for traditional ml models such as svm. therefore, models like lstm and svm may not generalize to another dataset with new figure 4: learning curve for underfit lstm we used shap (lundberg and lee, 2017)a state-of-the-art explainable ai (xai) tool, to interpret the classification results of the xlm-roberta-base transformer model's output, in our case "accuracy" (figure 3 ). this shap plot combines the significance of the features with their impacts. the y-axis lists the features from top to bottom or most important to least important order. the labels on the y-axis represent the most influential word features for xlm-roberta-base and their associated indexing in the word vector. the x-axis shows the shapely values from 0 to 1. blue, green, and pink spectrum are representations of shapley values for "positive," "negative," and "neutral" classes. not only the length of the spectrum but also the color has significance. for example, the "পু িলশ" feature correlates less than 20% with the model output accuracy. however, this word influences a post's identification as only negative (green color). another good example is the "ভাই" feature, the most influential feature in the predictions with xlm-roberta-base. the shapely value for blue (positive) is 70%, whereas for pink (neutral) and green (negative) is 20%. that means having a "ভাই" word in a post mostly co-related to a positive post, which is also intuitively correct since it is a respectful salutation. the shapely values of the features are more positively correlated with the positive class (labeled with blue) since blue spectrums are larger than the others. the neutral class (labeled with pink) has the lowest correlation with the model output. this result also aligns with the confusion matrix (figure 3 ), where prediction accuracy for positive classes is higher with xlm-roberta. therefore, the positive class operated on a higher accuracy scale with a higher correlation of approximately 70% with the most influential feature (feature 1754). the plot also shows that the impact of the "negative class" is very low-it does not frequently appear as the positive or neutral classes. ## limitations and conclusion in summary, we compared multiple ml approaches to discuss the multi-class sentiment analysis. we analyzed and compared results based on preprocessing techniques, rigorous output analysis, and xai. our analysis shows that the xlm-roberta-base generates a stable model even with lower accuracy regarding confusion matrix, evaluation metrics, and xai than lstm and traditional models. the first challenge we faced is that vector assembler on huge data made the dimensions of the feature very large and computationally expensive, difficult to address with low computing resources. secondly, the highly imbalanced dataset has only 20% "neutral" labels, which skewed the prediction against this class and caused some models to underfit. developing mlm-based masked models with oversampled datasets improved the quality of the classification tasks for xlm-roberta. it understands contextual relationships between words better and effectively predicts missing or masked words within a sentence. our future work will focus on mitigating the challenge of bangla sentiment analysis for lacking high-quality datasets, generalizable tools, comprehensive sentiment lexicons, and standardized evaluation metrics.