L ow R esource at BLP -2023 Task 2: Leveraging B angla B ert for Low Resource Sentiment Analysis of B angla Language (2023)

Abstract: This paper describes the system of the LowResource Team for Task 2 of BLP-2023, which involves conducting sentiment analysis on a dataset composed of public posts and comments from diverse social media platforms. Our primary aim was to utilize BanglaBert, a BERT model pre-trained on a large Bangla corpus, using various strategies including fine-tuning, dropping random tokens, and using several external datasets. Our final model is an ensemble of the three best BanglaBert variations. Our system achieved overall 3rd in the Test Set among 30 participating teams with a score of 0.718. Additionally, we discuss the promising systems that didn’t perform well namely task-adaptive pertaining and paraphrasing using BanglaT5. Our training codes are publicly available at https://github.com/Aunabil4602/bnlp-workshop-task2-2023

## introduction in the field of natural language processing, sentiment analysis has earned significant attention as a research area dedicated to the analysis of textual content. a considerable body of research on sentiment analysis in bangla has been conducted. some of these works (e.g. @xcite , @xcite ) are based on introducing new datasets. in parallel, other works(e.g. @xcite , @xcite ) are done on novel approaches. in spite of these numerous works, different opportunities still exist to improve the analysis of sentiments. in this paper, we describe our system for task 2 of the bangla language processing workshop @ @xcite . we employ various systems based on banglabert and banglabert-large @xcite . our experimental systems include fine-tuning, increasing the generalization based on dropping random tokens, using open-source external data during pre-training, and other methods described in section 4. utilization of random token drop and external datasets has benefited our systems by improving micro-f1 scores around 0.006 to 0.01. our best model, an ensembled model from three top models based on the development test-set score, has scored a micro-f1 score of 0.7179, standing overall 3rd among 30 participants. many of the related works are primarily focused on novel datasets covering diverse domains. islam et al. (2022) have developed a dataset comprised of various public comments from social media platforms. rahman and dey (2018) have created their datasets based on cricket and restaurant reviews. most recently, (kabir et al., 2023) have published a dataset entirely comprised of book reviews from online bookshops. existing approaches to sentiment analysis on bangla language primarily rely on machine learning and deep learning techniques. for example, arafin mahtab et al. (2018) have used support vector machine(svm) for sentiment analysis on public opinions on cricket. recurrent neural network based models are also highly used. (e.g. hassan et al. (2016)). irtiza tripto and eunus ali (2018) have explored a variety of approaches including lstm, svm, and naive bayes. moreover, convolutional neural network based models are also used for sentiment analysis on bangla(e.g. alam et al. (2017) ). in recent years, large language models(llm), trained on huge corpus, have become popular for their capability to understand the language and can easily fine-tuned for any task like sentiment analysis. llms based on the bangla language(e.g. banglabert @xcite , shaha-jbert @xcite , banglat5 (bhattacharjee et al., 2023)) are also available, which opens opportunities to work on various tasks for bangla. ## task description this is a multi-class classification task where the objective is to detect the sentiment of the given text into 3 different classes: positive, negative, and neutral. the score will be calculated using the micro-f1. the task consists of two phases: a development phase followed by a test phase. the final standing is based on the score of the test set provided during the test phase. ## system description here, we discuss several systems that we have experimented with for the task including the preprocessing of the dataset. ## experimental setup we have used models and trainer from huggingface foot_1 (pytorch version). we employ mixed precision training @xcite that enables faster training and consumes low gpu memory. moreover, we built a code such that the results are reproducible. all of the experiments are done using a single v100 gpu in google colaboratory foot_2 . we do hyper-parameters search on learning rate, batch size, dropout ratios, and total epochs. we start the search with the parameter settings as suggested @xcite . our best training parameters of fine-tuning and tapt are shown in the ## results to begin, we discuss the systems that have scored well on the development-test's score. the top individual model is banglabert-large with a random token drop that has scored 0.733, and even without any enhancement, it can score 0.723. the next best single model is banglabert with random token drop(rtd) and 2-stage fine-tuning that has scored 0.729. we have submitted the ensembled model as our best model in the test phase and has scored 0.718. moreover, we have submitted the 3 individual best models. our scores on the test set are shown in there are some systems that didn't achieve favorable performance from the beginning of our experiments. firstly, tapt didn't improve our results but rather declined the score by 0.039 with respect to simple fine-tuning as shown in paraphrasing to create additional data using banglat5 also didn't work well. its score is shown in other than banglabert, we try the xlm-roberta-large, a multi-lingual model, which is used by several task winners (e.g. @xcite ). however, it has scored low on the development-test set even with all enhancements. its score is also shown in our research has been mostly based on finetuning. as future work, we would like to explore using common data augmentation techniques @xcite for the given data. besides, there are several multilingual pre-trained models that include the bangla language are need to be explored along with sophisticated methods and may even achieve better results. ## conclusion in this paper, we stated our systems based on banglabert and banglabert-large for that sentiment analysis task. we used simple techniques like, 2-stage fine-tuning, using external datasets, and dropping random tokens. our system scored 3rd overall in the task. we also discussed some potential systems that didn't demonstrate satisfactory performance. more importantly, we have discussed the score inconsistency of our best model between development-test set and test set as our limitation. finally, we discussed directing some future research like applying tapt and dapt on banglabert and trying more data augmentations or sophisticated methods.