S eq2 E mo: A Sequence to Multi-Label Emotion Classification Model (2021)

Abstract: Multi-label emotion classification is an important task in NLP and is essential to many applications. In this work, we propose a sequence-to-emotion (Seq2Emo) approach, which implicitly models emotion correlations in a bi-directional decoder. Experiments on SemEval’18 and GoEmotions datasets show that our approach outperforms state-of-the-art methods (without using external data). In particular, Seq2Emo outperforms the binary relevance (BR) and classifier chain (CC) approaches in a fair setting.

## introduction emotion classification from text @xcite plays an important role in affective computing research, and is essential to human-like interactive systems, such as emotional chatbots @xcite @xcite . early work treats this task as multi-class classification @xcite , where each data instance (e.g., a sentence) is assumed to be labeled with one and only one emotion. more recently, researchers relax such an assumption and treat emotion analysis as multilabel classification (mlc, @xcite . in this case, each data instance may have one or multiple emotion labels. this is a more appropriate setting for emotion analysis, because an utterance may exhibit multiple emotions (e.g., "angry" and "sad", "surprise" and "joy"). the binary relevance approach (br, @xcite is widely applied to multilabel emotion classification. br predicts a binary indicator for each emotion individually, assuming that the emotions are independent given the input sentence. however, evidence in psychotherapy suggests strong correlation among different emotions @xcite . for example, "hate" may co-occur more often with "disgust" than "joy." an alternative approach to multi-label emotion classification is the classifier chain (cc, @xcite . cc predicts the label(s) of an input in an autoregressive manner, for example, by a sequenceto-sequence (seq2seq) model @xcite . however, seq2seq models are known to have the problem of exposure bias @xcite , i.e., an error at early steps may affect future predictions. in this work, we propose a sequence-to-emotion (seq2emo) approach, where we consider emotion correlations implicitly. similar to cc, we also build a seq2seq-like model, but predict a binary indicator of an emotion at each decoding step of seq2seq. we do not feed predicted emotions back to the decoder; thus, our model does not suffer from the exposure bias problem. compared with br, our seq2emo model implicitly considers the correlation of emotions in the hidden states of the decoder, and with an attention mechanism, our seq2emo is able to focus on different words in the input sentence that are relevant to the current emotion. we evaluate our model for multi-label emotion classification on semeval'18 @xcite and goemotions @xcite benchmark datasets. experiments show that seq2emo achieves state-of-the-art results on both datasets (without using external data). in particular, seq2emo outperforms both br and cc in a fair, controlled comparison. ## related work emotion classification is an activate research area in nlp. it classifies text instances into a set of emotion categories, e.g., angry, sad, happy, and surprise. well-accepted emotion categorizations include the six basic emotions in @xcite and the eight primary emotions in plutchik's wheel of emotions @xcite . early work uses manually constructed emotion lexicons for the emotion classification task @xcite @xcite . such lexicon resources include wordnet-affect @xcite , emosenticnet @xcite , and the nrc emotion intensity lexicon @xcite . distant supervision @xcite has been applied to emotion classification, as researchers find existing labeled datasets are small for training an emotion classifier. for example, @xcite finds that social media users often use hashtags to express emotions, and thus certain hashtags can be directly regarded as the noisy label of an utterance. likewise, @xcite use emojis as noisy labels for emotion classification. such distant supervision can also be applied to pretrain emotionspecific embeddings and language models @xcite . in addition, @xcite apply multi-task learning to combine polarity sentiment analysis and multi-label emotion classification with dual attention. different from the above studies that use extra emotional resources, our work focuses on modeling the correlations among emotions. this improves multi-label emotion classification without using additional data. a similar paper to ours is the sequence generation model (sgm, @xcite . sgm accomplishes multi-label classification by an autoregressive seq2seq model, and is an adaptation of classifier chains @xcite in the neural network regime. our paper models emotion correlation implicitly by decoder hidden states and does not suffer from the drawbacks of autoregressive models. ## methodology consider a multi-label emotion classification problem. suppose we have k predefined candidate emotions, and an utterance or a sentence x can be assigned with one or more emotions. we represent the target labels as @xmath0, • • • , y k ) ∈ {0, 1} k with y @xmath1. our seq2emo is a seq2seq-like framework, shown as figure 1 . it encodes x with an lstm, and iteratively performs binary classifications over y i with another lstm as the decoder. encoder. we use a two-layer bi-directional lstm to encoder an utterance. specifically, we use both token-level and contextual pretrained embeddings to represent a word in the sentence. formally, let a sentence be @xmath2, • • • , x m ). we first encode each word x i with glove embeddings @xcite , denoted by glove(x i ). we further use the elmo contextual embeddings @xcite , which processing the entire sentence x by a pretrained lstm. the corresponding hidden state is used as the embedding representation of a word x i in its context. this is denoted by elmo(x) i . we use a two-layer bi-directional lstm on the above two embeddings. the forward lstm, for example, has the form here, we use bilstm for simplicity, following @xcite and @xcite . other pretrained models, such as the tranformerbased bert @xcite , may also be adopted. this, however, falls out of the scope of our paper, as we mainly focus on multi-label emotion classification. empirical results on the goemotions dataset shows that, by properly addressing multi-label classification, our model outperforms a transformer-based model ( decoder. in seq2emo, an lstm-based decoder is used to make sequential predictions on every candidate emotion. suppose a predefined order of emotions is given, e.g., "angry," "joy," and "sad." the decoder will perform a binary classification over these emotions in sequence. the order, in fact, does not affect our model much, as it is the same for all training samples and can be easily learned. in addition, we feed a learnable emotion embedding as input at each step of the decoder. this enhances the decoder by explicitly indicating which emotion is being predicted at a step. different from a traditional seq2seq decoder, we do not feed previous predictions back as input, so as to avoid exposure bias. this also allows seq2emo to use a bi-directional lstm as the decoder, which implicitly model the correlation among different emotions. without loss of generality, we explain the forward direction of the decoder lstm, denoted by lstm -→ d . the hidden state at step j is given by where e j is the embedding for the jth emotion, and h-→ d j-1 is calculated by the attention mechanism in @xcite . here, the attention mechanism dynamically aligns source words when predicting the specific target emotion at a decoding step. let α → j,i be the attention probability of the jth decoder step over the ith encoder step, computed by likewise, we compute h← -d j for the backward decoder lstm. they are further concatenated for predicting the emotion in question: ## experimental setup datasets. we conduct experiments on two multilabeled emotion datasets: semeval'18 (affect in tweets: task e-c, @xcite and goemotions @xcite . compared with goemotions, semeval'18 has fewer emotion categories, and is smaller in size. both datasets come with standard train-dev-test splits. appendix a shows the statistics of these datasets. ## conclusion in this work, we propose seq2emo for multi-label emotion classification. our approach implicitly models the relationship of different emotions in its bi-directional decoder, and is shown to be better than an individual binary relevance (br) classifier. our model does not suffer from the exposure bias problem and also outperforms the classifier chain (cc). in general, we achieve state-of-the-art performance for multi-emotion classification on the semeval'18 and goemotions datasets (without using additional emotion labels).