Emotion Classification in a Resource Constrained Language Using Transformer-based Approach (2021)

Abstract: Although research on emotion classification has significantly progressed in high-resource languages, it is still infancy for resource-constrained languages like Bengali. However, unavailability of necessary language processing tools and deficiency of benchmark corpora makes the emotion classification task in Bengali more challenging and complicated. This work proposes a transformer-based technique to classify the Bengali text into one of the six basic emotions: anger, fear, disgust, sadness, joy, and surprise. A Bengali emotion corpus consists of 6243 texts is developed for the classification task. Experimentation carried out using various machine learning (LR, RF, MNB, SVM), deep neural networks (CNN, BiLSTM, CNN+BiLSTM) and transformer (Bangla-BERT, m-BERT, XLM-R) based approaches. Experimental outcomes indicate that XLM-R outdoes all other techniques by achieving the highest weighted f_1-score of 69.73% on the test data.

## introduction classification of emotion in the text signifies the task of automatically attributing an emotion category to a textual document selected from a set of predetermined categories. with the growing number of users in virtual platforms generating online contents steadily as a fast-paced, interpreting emotion or sentiment in online content is vital for consumers, enterprises, business leaders, and other parties concerned. ekman @xcite defined six basic emotions: happiness, fear, anger, sadness, surprise, and disgust based on facial features. these primary type of emotions can also be extracted from the text expression @xcite . the availability of vast amounts of online data and the advancement of computational processes have accelerated the development of emotion classification research in highresource languages such as english, arabic, chinese, and french (plaza del @xcite . however, there is no notable progress in low resource languages such as bengali, tamil and turkey. the proliferation of the internet and digital technology usage produces enormous textual data in the bengali language. the analysis of these massive amounts of data to extract underlying emotions is a challenging research issue in the realm of bengali language processing (blp). the complexity arises due to various limitations, such as the lack of blp tools, scarcity of benchmark corpus, complicated language structure, and limited resources. by considering the constraints of emotion classification in the bengali language, this work aims to contribute to the following: ## related work substantial research activities have been carried out on emotion analysis in high-resource languages like english, arabic, and chinese @xcite . a multi-label with multi-target emotion detection of arabic tweets accomplished using decision trees, random forest, and knn, where random forest provided the highest f 1 -score of 82.6% (alzu'bi et al., 2019). lai et al. (2020) proposed a graph convolution network architecture for emotion classification from chinese microblogs and their proposed system achieved an f-measure of 82.32%. recently, few works employed transformer-based model (i.e., bert) analyse emotion in texts. (huang et al., 2019; al-omari et al., 2020) used a pretrained bert for embedding purpose on top of lstm/bilstm to get an improved f 1 -score of 76.66% and 74.78% respectively. although emotion analysis on limited resource languages like bengali is in the preliminary stage, few studies have already been conducted on emotion analysis using ml and dnn methods. irtiza tripto and eunus ali (2018) proposed an lstm based approach to classify multi-label emotions from bengali, and english sentences. this system considered only youtube comments and achieved 59.23% accuracy. another work on emotion classification in bengali text carried out by azmin and dhar (2019) concerning three emotional labels (i.e., happy, sadness and anger). they used multinomial naïve bayes, which outperformed other algorithms with an accuracy of 78.6%. pal and karn (2020) developed a logistic regression-based technique to classify four emotions (joy, anger, sorrow, suspense) in bengali text and achieved 73% accuracy. das and bandyopadhyay (2009) conducted a study to identify emotions in bengali blog texts. their scheme attained 56.45% accuracy using the conditional random field. recent work used svm to classify six raw emotions on 1200 bengali texts which obtained 73% accuracy (ruposh and hoque, 2019). ## bemoc: bengali emotion corpus due to the standard corpus unavailability, we developed a corpus (hereafter called 'bemoc') to classify emotion in bengali text. the development procedure is adopted from the guidelines stated in (dash and ramamoorthy, 2019). ## methodology figure 2 shows an abstract view of the used strategies. various feature extraction techniques such as tf-idf, word2vec, and fast-text are used to train ml and dnn models. moreover, we also investigate the bengali text's emotion classification performance using transformer-based models.all the models are trained and tuned on the identical dataset. 4.1 feature extraction ml and dnn algorithms are unable to learn from raw texts. therefore, feature extraction is required to train the classifier models. tf-idf: term frequency-inverse document frequency (tf-idf) is a statistical measure that determines the importance of a word to a document in a collection of documents. uni-gram and bi-gram features are extracted from the most frequent 20000 words of the corpus. word2vec: it utilizes neural networks to find semantic similarity of the context of the words in a corpus @xcite . we trained word2vec on skip-gram with the window size of 7, minimum word count to 4, and the embedding dimension of 100. fasttext: this technique uses subword information to find the semantic relationships @xcite . we trained fast-text on skip-gram with character n-grams of length 5, windows size of 5, and embedding dimension of 100. for both word2vec and fasttext, there are pre-trained vectors available for the bengali language trained on generalized bengali wiki dump data @xcite . we observed that the deep learning models perform well on vectors trained with our developed bemoc rather than the pre-trained vectors. ## results and analysis this section presents a comprehensive performance analysis of various ml, dnn, and transformer-based models to classify bengali texts emotion. the superiority of the models is determined based on the weighted f 1 -score. however, the precision (p r), recall (re) and accuracy (acc) metrics also considered. among ml approaches, lr achieved the highest (60.75%) f 1 -score than rf (52.78%), mnb (48.67%) and svm (59.54%). lr also performed well in p r, re and acc than other ml models. in dnn, bilstm with fast-text outperformed other approaches concerning all the evaluation parameters. it achieved f 1 -score of 56.94%. however, bilstm (fast-text) achieved about 4% lower f 1 -score than the best ml method (i.e., lr). after employing transformer-based models, it observed a significant increase in all scores. among transformer-based models, bangla-bert achieved the lowest of 61.91% f 1 -score. however, this model outperformed the best ml and dnn approaches (56.94% for bilstm (fasttext) and 60.75% for lr). meanwhile, m-bert shows almost 3% improved f 1 -score (64.39%) than bangla-bert (61.91%). xlm-r model shows an immense improvement of about 6% compared to bangla-bert and 5% compared to m-bert, respectively. it achieved a f 1 -score of 69.73% that is the highest among all models.