Aambela at BLP -2023 Task 2: Enhancing B angla BERT Performance for B angla Sentiment Analysis Task with In Task Pretraining and Adversarial Weight Perturbation (2023)

Abstract: This paper introduces the top-performing approachof “Aambela” for the BLP-2023 Task2: “Sentiment Analysis of Bangla Social MediaPosts”. The objective of the task was tocreate systems capable of automatically detectingsentiment in Bangla text from diverse socialmedia posts. My approach comprised finetuninga Bangla Language Model with threedistinct classification heads. To enhance performance,we employed two robust text classificationtechniques. To arrive at a final prediction,we employed a mode-based ensemble approachof various predictions from different models,which ultimately resulted in the 1st place in thecompetition.

## introduction in recent years, natural language processing (nlp) has advanced significantly, highlighting the importance of sentiment analysis. this application provides insights into public opinion and social media trends. in the context of bangla text, sentiment analysis is crucial, aiding businesses in interpreting customer feedback, assisting policymakers in understanding public sentiment, and boosting media engagement. concerning the importance of sentiment analysis, the organziers of blp-shared task 1 @xcite provide one of the largest manually annotated datasets for sentiment analysis which encompasses sentiment across multiple platforms. the proposed sentiment analysis approach involves fine-tuning the bangla language model, such as banglabert @xcite , and utilizing three distinct classification heads to enhance model performance. to address overfitting and ensure robust generalization, strategies like cross-validation and adversarial perturbation techniques are employed. task-specific pretraining of banglabert on both the train and train+validation datasets is explored, yielding performance improvements. different classification heads in various techniques focus on distinct aspects of sentiment classification reasoning. to capture these diverse perspectives, a mode-based ensemble technique is applied. the ensemble predictions prove to be the best-performing model in the experiments, securing the top position on the leaderboard. ## result and analysis different models and experiments were done during the development phase which are reproted in appendix b. the experiment set up and hyper parameters details are described in appendix a. another experiments for model choice encompassed machine learning models (svm, random-forest, xgboost) using tf-idf feature extraction, deep learning models (lstm, lstm+attention), and multilingual transformer models (mbert, mdeberta, xlmroberta base), with mdeberta showing superior performance. additionally, two bangla language models were considered, with the csebuetnlp-banglabert model emerging as the top performer. these experiments employ 5-fold cross-validation. the inclusion of awp (section 3.2) enhances both cross-validation scores and generalization to validation and test datasets by approximately 1-2%. however, incorporating the validation data into training yields a slightly lower test set performance despite boosting the cv score. itpt (section 3.1) on training data significantly enhances performance across all classification heads. conversely, including the validation data during itpt yields mixed results, with slight improvements in some heads and minor reductions in others. a final prediction is made by ensembling all classification heads from different techniques. the ensemble technique employed is a mode-based ensemble, aggregating predictions from all models across techniques and selecting the mode as the final prediction. this approach achieved an accuracy of 73.10% (micro f1) and 68.74% (macro f1) on the test set, placing it at the top of the leaderboard. though the model has a highest score in the leaderboard, it has some limiations and scope for improvements which are describe in limiation 5 section and appendix c. ## conclusion in this work,we have experimented with fine-tuning banglabert in different aspects using different classification heads. the result showed that it gives a better score. adverserial training and cross validation made the model more robust. in task pretraining helped the model to further investigate different classes of sentiment analysis. our finding is that using adverserial training and in task pretraining we can improve our model further and build up a better model.