Sentiment Analysis For B engali Using Transformer Based Models (2021)

Abstract: Sentiment analysis is one of the key Natural Language Processing (NLP) tasks that has been attempted by researchers extensively for resource-rich languages like English. But for low resource languages like Bengali very few attempts have been made due to various reasons including lack of corpora to train machine learning models or lack of gold standard datasets for evaluation. However, with the emergence of transformer models pre-trained in several languages, researchers are showing interest to investigate the applicability of these models in several NLP tasks, especially for low resource languages. In this paper, we investigate the usefulness of two pre-trained transformers models namely multilingual BERT and XLM-RoBERTa (with fine-tuning) for sentiment analysis for the Bengali Language. We use three datasets for the Bengali language for evaluation and produce state-of-the-art performance, even reaching a maximum of 95% accuracy for a two-class sentiment classification task. We believe, this work can serve as a good benchmark as far as sentiment analysis for the Bengali language is concerned.

## introduction in this era of the world wide web, sharing of information, knowledge, opinion, etc. has been increased by a huge margin since the last decade. internet users are coming forward to review stuff like books, movies, videos, e-commerce products, etc., and are sharing their experiences which in turn help the next in line users to get feedback upfront. this genre of texts brings in the essence of sentiment analysis task which helps in polarity classification, i.e. determining whether a given text expresses positive, negative, or neutral sentiment. sentiment analysis of texts can be useful for different applications, like detecting cyberbullying @xcite , hate speech @xcite , e-commerce recommendation system @xcite , etc. there has been a substantial amount of work done by the researchers to tackle sentiment analysis for resourcerich languages like english @xcite , but for low resource languages, such attempts are scarce @xcite . in recent times, for low resource languages like hindi, telegu, bengali, assamese, manipuri, indonesian, etc. @xcite and even for english-hindi, english-bengali code-mixed languages @xcite , researchers have come up with a solution for sentiment analysis tasks. in another work, r et al. (2012) performed cross-lingual sentiment analysis task where the opinion polarity of a text in a language is predicted using classifier trained in another language. the authors report results on two widely spoken indian languages, hindi and marathi. @xcite used an lstm-rnn based approach to determine the sentiment of hindi tweets and also compared their approach with cnn, machine learning, and lexicon based approaches. @xcite uses hindi senti-wordnet (hswn) proposed by @xcite as a lexicon generating tool for hindi text. so hindi being a major indian language has been explored whereas more insights are still needed in bengali. in one of the very recent works, islam et al. (2020) prepare a two-class and a three-class sentiment analysis dataset in bengali and report performances of multilingual bert @xcite which is impressive. moving forward in a similar direction, in this paper we apply two pretrained transformers models namely multilingual bert and xlm-roberta @xcite after fine-tuning and conducting the analysis. in addition to the datasets proposed by islam et al. (2020), we use two other datasets proposed by @xcite and @xcite for our study. we observe that, by applying fine-tuned multilingual bert and xlm-roberta (for convenience we will refer xlm-roberta as xlm-r in our paper), we achieve an accuracy of 63%-94% and 68%-95%, while evaluating against these three target datasets leading to state-of-the-art performances. to the best of our knowledge, this is the first attempt at such a comprehensive study for the bengali language where pre-trained transformer models' applicability (with fine-tuning) has been investigated for sentiment analysis tasks and evaluated against three datasets. all the codes and datasets are made publicly available foot_0 . ## dataset for this study, we use three datasets. the details of these datasets are described below. prothom alo: this is the first dataset foot_1 used in this study which is a publicly available dataset created from user comments on 10 popular news topics from an online bengali news portal, prothom alo foot_2 . this dataset is introduced by islam et al. (2020), for convenience, we refer to this dataset as 'prothom alo'. the authors scrape user comments from news threads and clean to obtain a total of 17,852 user comments. each of the comments is tagged by bengali domain experts into one of the following three classes: positive, negative, and neutral. the authors prepare a variant of this dataset as well which has only two classes by removing the neutral class entries. this step results in a dataset for two-class classification with 13,120 entries. youtube-b: this is a collection of reviews manually annotated from youtube bengali drama foot_3 consisting of 8500 positive reviews and 3307 negative reviews and is introduced by @xcite . this dataset is a two-class dataset having only positive and negative as labels. we refer to this dataset as 'youtube-b' for the rest of the paper. 'b' stands for bengali language. book-b: this is the third dataset introduced by @xcite . it is a collection of bengali book reviews collected from web resources such as blogs, facebook, and e-commerce sites. this dataset is also a two-class dataset (having positive and negative classes) with 2000 entries of book reviews. we refer to this dataset as 'book-b' for the rest of the paper. the details of these three datasets are shown in ## proposed approach xlm indicates a cross-lingual language model. xlm-roberta (xlm-r) is a pre-trained multilingual model that is considered to be superior over multilingual bert when evaluated against various nlp tasks. one probable reason could be that xlm-r is trained using a much bigger corpus. xlm-r is also trained in approximately 100 languages. similarly incase of xlm-r, in our approach we use xlm-roberta-large checkpont of pretrained xlm-r model. we opt for the idea of fine-tuning bert and xlm-r models especially for low resource language like bengali, as fine- ## experimental setup for the series of experiments performed, we first adopt the model from the work by islam et al. (2020) and use it as a baseline model. this benchmark model consists of a multilingual bert (bertbase-multilingual-cased) pre-trained on multiple languages. three different deep neural network layers: gru, lstm, cnn are used as an extra layer on top of bert separately to produce three separate architectures. we use their code repository and train the baseline models using the same set of hyper-parameters and attempt to replicate the results. next, we replace bert with xlm-r in the same architecture. as xlm-r models we use xlm-roberta-large. for this set of exper-samples model #(classes) target prediction প্রথম আেলার এই িডপারেমন্ট খু ব কঁ াচা িরেপােটর িদক িদেয় । this department of prothom alo is very raw in terms of reporting. ## results and discussion note that, to have a fair comparison with the most recent baseline models proposed by @xcite , we report validation accuracy following their performance measures and we see finetuned xlm-r outperforms this baseline by a significant margin for the 'prothom alo' dataset. in addition, we also report test accuracy in the last column of we further investigate a few predicted samples from different datasets to check for the cases that were predicted wrongly by fine-tuned bert or xlm-r . few correctly predicted and wrongly predicted samples are presented in ## conclusion in this paper, we conduct an experimental study showing the applicability of multilingual bert and xlm-r (with fine-tuning) for the bengali sentiment analysis task. we use three datasets to evaluate the models and obtain promising performances for all three datasets. the immediate future step would be investigating the erroneously classified cases and trying to find the reason behind such errors and mitigate it. broadly, we plan to investigate sentiment analysis for other low-resource languages like tamil, oriya, gujrati, etc and attempt to propose variants of transformer based models.