Towards Emotion Recognition in H indi- E nglish Code-Mixed Data: A Transformer Based Approach (2021)

Abstract: In the last few years, emotion detection in social-media text has become a popular problem due to its wide ranging application in better understanding the consumers, in psychology, in aiding human interaction with computers, designing smart systems etc. Because of the availability of huge amounts of data from social-media, which is regularly used for expressing sentiments and opinions, this problem has garnered great attention. In this paper, we present a Hinglish dataset labelled for emotion detection. We highlight a deep learning based approach for detecting emotions using bilingual word embeddings derived from FastText and Word2Vec approaches in Hindi-English code mixed tweets. We experiment with various deep learning models, including CNNs, LSTMs, Bi-directional LSTMs (with and without attention), along with transformers like BERT, RoBERTa, and ALBERT. The transformer based BERT model outperforms all current state-of-the-art models giving the best performance with an accuracy of 71.43%.

## introduction with the growth of social networking sites like facebook and twitter, humans have started communicating online much more than ever before. this leads to the generation of huge amounts of textual data which introduces interesting challenges in the domain of nlp. automatic detection of various linguistic expressions like irony, hate, sarcasm, aggression etc. is being widely explored. another problem that has drawn keen interest of nlp researchers is detecting emotions of a human via the texts they have produced. in order to aid humancomputer interaction, determining the emotions via texts becomes significant @xcite . there are multiple ways of detecting emotions, including but not limited to speech @xcite , facial expressions recognition @xcite and text-based approaches. text-based emotion detection is based on the assumption that when a person is happy, they would use positive words. likewise, when they are angry, frustrated or upset, negative emotions will be depicted by a certain kind of words carrying negative connotation. contrary to popular belief, emotions are not only significant in human creativity, but they also play an instrumental part in making rational decisions. with the rise of artificial intelligence and increased focus on human-computer interaction, smart machines that will communicate naturally and intelligently with humans, need to recognize their emotions effectively. affective computing has emerged as an exciting field with recent focus on emotion detection @xcite . most of earlier work has been carried out on a mono-lingual dataset due to easy availability of a large corpus of annotated data @xcite . however, in multilingual cultures, use of multiple languages while exchanging information on social media is quite common. studies show that as many as 314.9 million people in india are bilingual foot_0 . this leads to the issue of code-mixing and code-switching especially while communicating on social media platforms like twitter, facebook and reddit @xcite . code-mixing occurs when lexicons and grammatical features of multiple languages are used in the same sentence @xcite @xcite . the major issue in dealing with code-mixed problems is the absence of sufficiently annotated datasets @xcite . in this paper, we present our findings on one of the most challenging problems in the domain of natural language processing, 'emotion detection'. while a lot of work has been carried out for the english language @xcite , the domain of hindi-english code-mixed texts remains relatively new and not much explored. we present an annotated hindi-english code-mixed dataset of 150k tweets for addressing this issue and for enabling future researchers to contribute to this domain. our aim in this paper is to compare multiple deep learning models including cnns, lstms, bi-directional lstms (with and without attention) with the aid of bilingual self-trained word embeddings on a code-mixed dataset, along with transformer based models like bert, roberta, and albert. the paper is organized as follows -section 2 details about the background and related work in this domain. section 3 enumerates the methodology we used to perform the experiments including data annotation, pre-processing, embeddings and models used. section 4 lists down the experimental settings to replicate the work done. section 5 contains details of the results obtained and section 6 consists of conclusions drawn from the results. ## related work with the huge growth of micro-blogging platforms like facebook and twitter, there has been an increased interest in detecting sentiments and emotions in large text corpus @xcite . in initial work aimed at emotion detection in textual data, experiments have been carried out with text-based emotion classification in fairy tales for kids on the lines of basic emotions @xcite . in another related work @xcite , the authors work on real-world knowledge bases highlighting human's natural reactions towards various situations, aimed at identifying emotions at the sentence-level. with the increase of non-native english speakers on social media, sentiment analysis on regional languages and code-mixed data has gained momentum. a pivotal work of sentiment analysis in hindi corpus was done, where the authors were successful in extracting sentiment lexons from hindiword-net and were able to achieve an accuracy of 87% in the domain of movie @xcite . in a detailed analysis of data of english-hindi bilingual users on facebook, it was shown that 17.2% of all posts, which accounted for around one-fourth of the words in their dataset, revealed some form of code-mixing @xcite . sub-word level lstm architecture for performing sentiment analysis was introduced on hindi-english code-mixed dataset @xcite . experiments were conducted with supervised learning (svm) on a hindi-english code-mixed corpus for emotion detection @xcite . ## proposed methodology this section describes the series of steps that constitute the methodology proposed, including detailed descriptions of dataset creation, annotation, preprocessing, embeddings, and the deep learning models. ## experimental settings a split of ten percent was made on the total training dataset and the model was trained for a total of 20 epochs. at each epoch, we saved the model checkpoints, and particularly used that checkpoint which was saved before the model begins to overfit to calculate the metrics on the ten percent test dataset split. different hyper parameters are involved for the task of training embeddings as well as the models. after working with several optimizers, loss functions and activation functions, the adam optimizer with categorical cross entropy loss function produced the best results for all stated deep learning models. we used relu activation function for all the layers except the output layer, which has sigmoid activation function. we evaluated the performance of cnn models with different values for kernel sizes, activation functions, number of kernels, dropouts, strides and optimizers. we use pre-trained models like bert-baseuncased, roberta-base, and albert-base-v2, to fine tune the transformer based models on our dataset. ## results using all features, @xcite show that the baseline model i.e. the svm classifier with rbf kernel, presented an accuracy of 58.2%, when dealing with the same emotion labels as we deal with in this paper. in the domain of emotion detection in hindi-english code-mixed data, as far as we know, we are the first to compare transformer based models and word representations. in the transformer based bert model clearly outperforms both cnn and rnn based models, majorly because of its profound efficiency and its ability to process the input out of order. the major obstacles in the task of detecting emotions in hindi-english code-mixed data are handling the linguistic complexities associated with code-mixed data and absence of clean data. thus, we require even more class-specific cleaner data, in order to reduce the effect of noise, which comes from spelling mistakes, stemming words and the presence of multiple contexts. ## conclusion as recent years have seen the rise in usage of social media for open expression of stance and opinions, sentiment analysis and opinion mining have gained attention as problems and become primary areas of research. in this paper, we present an openly available class-balanced dataset of hindi-english codemixed data, consisting of tweets belonging to 6 types of emotions, which are happiness, sadness, anger, surprise, fear and disgust. we contrast the performance of two types of word representations, both trained on relevant scraped tweets from scratch. we develop two different types of embeddings, one which is trained on solely hinglish tweets, the other which is trained on a mix of hinglish and english tweets, and present the performance in both cases. also, we present deep learning based models including cnns, rnns, and transformers, where bert performs the best among all. as future scope, the problem can be solved to obtain even better results by carrying out a comparison of muse aligned vectors, pre-aligned fast-text word embeddings and language specific transformer based word embeddings.