Annotated Corpus of Tweets in E nglish from Various Domains for Emotion Detection (2020)

Abstract: Emotion recognition is a very well-attended problem in Natural Language Processing (NLP). Most of the existing works on emotion recognition focus on the general domain and in some cases to specific domains like fairy tales, blogs, weather, Twitter etc. But emotion analysis systems in the domains of security, social issues, technology, politics, sports, etc. are very rare. In this paper, we create a benchmark setup for emotion recognition in these specialised domains. First, we construct a corpus of 18,921 tweets in English annotated with Paul Ekman’s six basic emotions (Anger, Disgust, Fear, Happiness, Sadness, Surprise) and a non-emotive class Others. Thereafter, we propose a deep neural framework to perform emotion recognition in an end-to-end setting. We build various models based on Convolutional Neural Network (CNN), Bi-directional Long Short Term Memory (Bi-LSTM), Bi-directional Gated Recurrent Unit (Bi-GRU). We propose a Hierarchical Attention-based deep neural network for Emotion Detection (HAtED). We also develop multiple systems by considering different sets of emotion classes for each system and report the detailed comparative analysis of the results. Experiments show the hierarchical attention-based model achieves best results among the considered baselines with accuracy of 69%.

## introduction online social media provides a platform for people to share their perspectives on various issues with their close ones or in the public forum. twitter is a very popular and heavily used platform among the most social media users. the usa and india are the 1 st and 3 rd leading countries based on number of twitter users as of july 2020 1 . twitter data serves as a rich source for text analysis tasks as this type of data is not very long, yet very rich in emotion content. this type of communication is free from barriers of age, race, culture, gender, etc. in recent times, understanding people's opinion and sentiment have been the need of the hour to meet various purposes, such as real-time trending, better customer service, winning elections, etc. particularly in indian context, we often hear stories how a single social-media post has changed the life of an individual or an organization in a positive 2 or negative way. sentiment analysis or opinion mining deals with the automatic identification and extraction of the underlying subjective information from text. it is often synonymously described as 'polarity detection' which is concerned with classifying an instance of data as 'positive', 'negative' or 'neutral'. contrary to sentiment analysis, the emotional analysis relies on a more fine-grained analysis of the subjectivity information. it deals with the deeper analysis of human emotions and sensitivities. emotion analysis goes a step further into a person's motives and impulses. it gives valuable and exact insights that are easily transformed into actions. it is usually based on a wide spectrum of moods rather than a couple of static categories. inside positive sentiment, it detects specific emotions like happiness, pride, satisfaction, thankfulness or excitement, depending on how it is configured. similarly, negative sentiment may span a variety of emotions like anger, sadness, fear, hopelessness, blame, etc. usually, sentiment or emotion analysis works better on subjective texts (texts having emotions or feelings) than objective ones (statements or facts). in this work, we focus on building a corpus of emotion-annotated tweets from the relevant topics of interest of recent times (i.e. social issues, technology, cyber-security etc.) that are quite a buzz among today's online social platform users. most of the available corpora for emotion are from the domains limited to blogs, weather, elections, fairy tales and some more. there are some available corpora in the general domain too, but they mostly cover tweets from politics, world news, sports, etc. at the present, hardly any corpora cover diverse domains. we consider paul ekman's @xcite basic emotions (anger, disgust, fear, happiness, sadness, surprise) for the emotion labelling task. a non-emotive label others is also introduced for tweets which do not fall within the scope of ekman's basic emotions @xcite . for effective usage of the dataset, we develop multiple single-task models for emotion classification. we develop cnn, bi-gru and bi-lstm based deep learning models for the emotion classification task. we propose a bi-gru based framework with hierarchical attention @xcite mechanism to extract important information from each sentence in a tweet effectively. we also develop a couple of models on a sub-set of emotion classes (4 classes and 6 classes) and perform a comparative analysis with the developed systems with 7 classes. the proposed hierarchical attention system for 7 classes attains superior results than the considered baselines with an overall test accuracy of 69% for the emotion classification task. the rest of the paper is organized as follows. in section 2, we discuss some of the existing work and corpora concerning emotion analysis. various aspects of resource creation, challenges and analysis are discussed in section 3. next, we discuss the methodologies we implement in section 4. in section 5, we discuss the implementation details, results and qualitative error analysis. finally, we conclude in section 6 and acknowledge the funding agency for this work in section 7. ## background in the past decades, several annotated corpora have been created for emotion recognition from texts. various annotation schemes were introduced to serve the specific purpose for which the corpus is created. @xcite collected questionnaires answered by people with different cultural backgrounds to form the international survey on emotion antecedents and reactions (isear) dataset. people reported on their emotional events. the dataset contains a total of 7,665 sentences from reports by approximately 3,000 respondents. sentences are annotated with single labels, chosen from the set of following labels: joy, fear, anger, sadness, disgust, shame, and guilt. the affective text task @xcite in se-meval 2007 was proposed to focus on the emotion classification of news headlines extracted from news web sites. given a set of predefined six emotion labels (paul ekman's basic emotions @xcite ), classify the titles with the appropriate emotion label and/or with a valence indication (positive/negative). @xcite published a dataset of blog content consisting of 5,205 sentences from 173 blogs. each instance is annotated with an emotion label from ekman's basic emotions @xcite and also with an intensity score for that emotion. @xcite researched the text-based emotion prediction problem in the literature domain. the author provided an annotated corpus of 15,302 sentences from 176 stories annotated from among the following seven emotion classes (angry, disgusted, fearful, happy, sad, positive surprise, and negative surprise). crowdflower's dataset, the emotion in text foot_2 , is a noisy single-labelled crowd-sourced annotated corpus of tweets. it primarily follows plutchik's 8 basic emotions @xcite in addition to another 3 emotions (love, confusion and no emotion). the electoral-tweets dataset, published by @xcite , targets the domain of elections (2012 us presidential election). it consists of over 100,000 crowdsourced responses to two detailed online questionnaires (the questions targeted emotions, purpose, and style in electoral tweets). @xcite published the emotion-stimulus dataset to predict the cause of emotion in the text. the dataset consists of 820 sentences which are annotated both with emotions (one label per sentence) and their causes, and 1,549 sentences which are marked only with their emotion. ekman's basic emotions @xcite with an added class shame have been used for the annotation. the hashtag emotion corpus, also known as twitter emotion corpus (tec), was published by @xcite , and consists of 21,051 tweets. this resource was created to understand if emotion-word hashtags can successfully be used as emotion labels. ekman's basic emotions @xcite have been considered for the annotation process. tweets were scraped that contained hashtags in the form #emotion corresponding to ekman's @xcite 6 basic emotions (like #anger, #disgust). dailydialogs is a dataset of dialogs published by @xcite spanning over a variety of topics and better structured than any social media data. the ssec corpus @xcite is an annotation of the semeval 2016 twitter stance and sentiment corpus @xcite ) with plutchik's emotion labels @xcite . the authors studied the relation between emotion annotation and the other annotation layers like stance and sentiment. the emoint dataset published by (mohammad and bravo-marquez, 2017) for evaluation of the wasaa-2017 shared task of emotion intensity (emoint) contains 7,097 tweets annotated with a pair of emotion tag and intensity score of the corresponding emotion. the annotation was done via crowdsourcing with primarily (but not limited to) one among the following 4 emotions anger, joy, sadness, and fear and their respective intensity score ranging between 0 to 1. the affect in tweets dataset of the semeval 2018 task 1 @xcite was introduced to determine the intensity of both emotion and sentiment as well as multi-label emotion classification of tweets. recent works have shown the effectiveness of multi-task systems by learning several correlated tasks simultaneously @xcite @xcite . ## corpus creation in this section we briefly describe the data creation process starting from the collection, pre-processing, annotation and inter-annotator reliability. ## methodologies we develop various deep learning-based multi-task models for automatic detection of emotion and its intensity. as base learning techniques, we use convolution neural network (cnn) @xcite , long short term memory (lstm) network @xcite and gated recurrent unit (gru) network @xcite . we build three separate multi-task models (cnn based, bi-gru based and bi-lstm based) on top of pre-trained word embedding (glove 7 @xcite ). the embedding layer is initialized with the pre-trained weights and is learned during the training in accordance with our dataset. we employ word attention @xcite mechanism to focus on the informative words in a document (tweet) and obtain an aggregated representation(document vector) which is passed through two fully-connected layers (100 neurons in each 7 http://nlp.stanford.edu/data/ wordvecs/glove.840b.300d.zip layer) and an output layer (with 7 neurons, one for each class) with softmax activation. we use the categorical cross-entropy as the loss function. ## conclusion in this work, we have proposed a benchmark deep learning setup for emotion detection. the