S ent N o B : A Dataset for Analysing Sentiment on Noisy B angla Texts (2021)

Abstract: In this paper, we propose an annotated sentiment analysis dataset made of informally written Bangla texts. This dataset comprises public comments on news and videos collected from social media covering 13 different domains, including politics, education, and agriculture. These comments are labeled with one of the polarity labels, namely positive, negative, and neutral. One significant characteristic of the dataset is that each of the comments is noisy in terms of the mix of dialects and grammatical incorrectness. Our experiments to develop a benchmark classification system show that hand-crafted lexical features provide superior performance than neural network and pretrained language models. We have made the dataset and accompanying models presented in this paper publicly available at https://git.io/JuuNB .

## introduction sentiment analysis is one of the classic problems in computational linguistics, and it has shown a massive impact on different real-life applications. the capability to quantify sentiment polarity of english texts has enabled the creation of solutions for a diverse set of problems like understanding the possible movement of stock markets, public sentiment towards any event or product, and understanding client satisfaction for customer support. a major reason behind such a success is the amount of collaborative efforts invested in the research and development of the creation of public resources like sentiment140 @xcite , sentiwordnet @xcite , imdb review corpus @xcite , stanford sentiment treebank @xcite , ts-lex @xcite , and semeval twitter sentiment analysis corpus @xcite . positive [b] অ অ অ অ সাধারন । আিম েকান িদনই পারেবা না । িহংসা হেচ্ছ [e] great. i will never be able to do it. feeling jealous. neutral [b] িপছেন দু জন মু িতর্ দারা করায় লাগেছ [e] two people placed idols behind them. negative [b] ভাই আপনার ক ােমরা েমনেক িদেলন্না একাই সব সাবার করেলন, হা হা হা [e] bro, you didn't share with your cameraman and ate the whole thing, ha ha ha. hindi @xcite foot_0 with 268m speakers. bangla is the native language of bangladesh and some regions of india, such as west bengal. while technology is dramatically improving the lives of people from these densely populated and economically burgeoning regions, it is a timely need of building technologies that can understand the language, enhancing the overall impact on social welfare and businesses. existing datasets for sentiment analysis for a low-resource language like bangla suffer from three major limitations: 1) none to slight inter annotator agreement score questioning the annotation reliability (e.g., 0.11 in ashik et al., 2019 and 0.18 in islam et al., 2020), 2) lack of cross-domain generalization capability due to large domain @xcite @xcite , and 3) lack of public availability for further research @xcite @xcite @xcite . in this paper, we aim at creating a domainrepresentative sentiment polarity classification dataset by collecting public opinions on various topics. during the data collection and annotation process, we invest efforts to improve the quality of the dataset using data curation techniques. on one hand, it includes the steps for duplicate removal, while on the other hand we increase the vocabulary size by incorporating instances that will help to increase the unique word percentage. our contributions can be summarized as follows: ## development of sentnob data collection we defined the following objectives before creating the dataset as we believe these objectives will enhance the generalization capability of sentnob: 1) samples should represent many different domains to encourage domainindependent solutions. 2) samples should contribute to making the dataset less repetitive. we start by collecting public comments on articles on the most popular 13 topics from prothom alo foot_1 , the most circulated newspaper in bangladesh foot_2 . then we collect comments from a set of youtube videos on similar topics. out of ≈ 31k collected comments, we keep the comments that are written in only bangla alphabets. to reduce repetitiveness and noise, we remove duplicates and exclude instances shorter than three or longer than 50 words tokens. additionally, we aim at increasing the vocabulary size by ## methodology in this section, we describe the methods we investigate to develop a benchmark model for classifying sentiment polarity on sentnob. we start by training linear svm @xcite models with traditional hand-engineered linguistic features. then, we experiment with recurrent neural network models and pre-trained transformer based language models due to their recent success on a wide variety of nlp tasks. ## experimental setup we implement our experimental framework using pytorch @xcite , transformers @xcite , and scikit-learn @xcite . we evaluate our methods using micro averaged f1. as the baseline systems, we compare our results with the majority, random, and weighted random baselines. to reduce noise, we replace the numerical tokens with a cc token and normalize english and bangla sentence stoppers. due to the class imbalance, we perform per-topic stratified split to create training (80%), development (10%), and test (10%) sets. while we evaluate all the individual features using the same hyper-parameter setting, we tune the svm regularizer c foot_3 of the model on the validation set performance for the best performing feature combination. for training the bilstm model with mini batches, we left pad the instances and perform hyper-parameter tuning on learning rate, batch size, dropout rate, number of lstm cells and layers. for fine-tuning mbert, we only tune the learning rate and batch size. ## conclusion in this paper, we present sentnob, a dataset for analysing sentiment in noisy bangla texts collected from the comments section of bangla news and videos from 13 different domains. sentnob contains ≈ 15k instances labeled with positive, negative, or neutral class label. we found that lexical feature combinations demonstrate stronger classification performance compared to neural models. as the future work, we will focus on different preprocessing techniques and more investigation with pre-trained language models.