Contextualized Emotion Recognition in Conversation as Sequence Tagging (2020)

Abstract: Emotion recognition in conversation (ERC) is an important topic for developing empathetic machines in a variety of areas including social opinion mining, health-care and so on. In this paper, we propose a method to model ERC task as sequence tagging where a Conditional Random Field (CRF) layer is leveraged to learn the emotional consistency in the conversation. We employ LSTM-based encoders that capture self and inter-speaker dependency of interlocutors to generate contextualized utterance representations which are fed into the CRF layer. For capturing long-range global context, we use a multi-layer Transformer encoder to enhance the LSTM-based encoder. Experiments show that our method benefits from modeling the emotional consistency and outperforms the current state-of-the-art methods on multiple emotion classification datasets.

## introduction with the prevalence of conversation-based service, emotion recognition in conversation (erc) has been attracting attention recently @xcite @xcite . due to great potential in many scenarios such as recommendation system, customer service feedback and health-care, researchers keep focusing on empowering machine to understand emotions in conversation with emotional dynamics, which is a work with challenges lying in several aspects such as modeling the emotion inertia for each speaker and the influence of the interaction between speakers on emotional dynamics @xcite . recent works on erc rely on recurrent neural networks (rnns) to compute context-dependent representations of utterances @xcite . due to a carefully designed cell, rnns like long short-term memory figure 1 : emotional consistency on iemocap @xcite . in a conversation, similar emotions tend to appear adjacently while dissimilar emotions seldom appear in the neighborhood. we call this phenomenon emotional consistency. for example, if the emotion of current utterance is happy, the tag of next utterance tends to be happy, excited or neutral rather than sad, angry or frustrated. this pattern also applies to other emotions. nevertheless, including lstm and gru, rnns are limited in their capability to process tasks involving very long sequences in practice @xcite . for mitigating this issue, the transformer architecture @xcite and graph convolution networks (gcns) @xcite have been introduced to erc for propagating contextual information among distant utterances and yielded state-of-the-art performance @xcite . these approaches leverage contextualized utter-ance features to predict emotion tags, but they ignore the inherent relation between emotion tags. we observe, that the phenomenon of emotional consistency exists widely in conversations, that is, similar emotions are much more likely to appear adjacently than dissimilar emotions, as shown in figure 1 . we surmise modeling the emotional consistency is helpful to find a more reasonable distribution of emotion tags and thus further improves the performance of emotion classification. in this work, we propose a method to address emotion classification as sequence tagging. for a given conversation, instead of predicting the distribution of emotion tags independently, we consider relations between nearby emotion tags and choose the globally best tag sequence for the entire conversation at once. hence, we employ a crf @xcite to take into account the dependency between emotion tags in neighborhoods. contextualized utterance representations fed into the crf layer are computed by lstm-based context encoders. by the aid of individual context encoder, our model tracks the self dependency which depicts emotional inertia of individual speakers. the inter-speaker dependency reflecting the influence of other speakers on a certain speaker is understood by the global context encoder. we use a multi-layer transformer encoder to enhance the global context encoder so that our model can take advantage of long-range contextual information when computing contextualized utterance representations. in summary, our contributions are as follows: the remainder of this paper is organized as follows. section 2 discusses related works. section 3 describes our sequence labeling architecture. section 4 presents the experimental setting. section 5 reports extensive experimental results and makes a detailed analysis. we conclude this paper in section 6. transformer: transformer has achieved great success in various nlp tasks due to its rich representation and high computation efficiency. selfattention mechanisms endow transformer with the capability of capturing longer-range dependencies than rnns. recent works such as bert @xcite and gpt @xcite use transformer encoder and decoder respectively to learn representations on large-scale datasets. these representations are transferred to down-stream tasks such as named entity recognition (ner) and question answering and achieves state-of-the-art results. @xcite introduces the notion of recurrence to address context fragmentation limitations of transformer. @xcite explores transformer with additional lstm layers to better capture the sequential context while retaining the high computation efficiency. sequence tagging: sequence tagging has drawn research attention for a few decades. it includes a bunch of nlp tasks such as part of speech tagging (pos), chunking and ner. the most common statistical models for sequence tagging includes hidden markov model (hmm), maximum entropy markov model (memm) and crf @xcite . these traditional sequence tagging methods rely heavily on hand-crafted features. in the past few years, convolutional neural networks (cnns) and rnns are introduced to tackle sequence tagging problems and achieves competitive performance against traditional methods @xcite . @xcite has pointed out that the combination of bidirectional lstm and crf can efficiently use both past and future input features as well as past and future tags information. hence, bilstm-crf model produces state-of-the-art results on many sequence tagging tasks. ## cesta: contextualized emotion sequence tagging existing works @xcite @xcite define the erc task as the prediction of emotion tags of constituent utterances. however, emotional consistency which is an important characteristic of the conversation is not taken into consideration. cesta differs from those methods in that it treats erc as a task of sequence tagging of which performance is generally improved by choosing the globally best set of tags for the entire sequence at once. to this end, cesta employs a crf to take advantage of past and future tags to predict the current tag. for the tth utterance in a conversation, the textual feature u t is extracted by a single-layer cnn and fed into the global and individual context encoders which learn inter-speaker and self dependency respectively. moreover, the global context encoder is enhanced by a number of transformer blocks to propagate long-range contextual information effectively. the concatenation of the global context encoding g t and individual context encoding s t is considered as a matrix of scores and fed into the final crf layer. the overall architecture is shown in figure 2 . figure 2 : overview of cesta. the transformerenhanced global context encoder takes the textual feature u t of the tth utterance in a conversation as input and produces encoding g t . also, u t is fed into the individual context encoder to update states for the corresponding speaker of which index is @xmath0. a crf layer is applied over the concatenation of each g t and s t to obtain the final prediction for each utterance in the conversation. ## conclusion we have introduced a new method, cesta, to model erc task as sequence tagging. based on the contextualized utterance representations, it leverages past and future emotion tags to jointly decode the best tag sequence for the entire conversation at once. we conduct numerous experiments on three benchmark datasets. through ablation studies, we have confirmed modeling the emotional consistency via crf and enhancing the context encoder via the transformer are beneficial to our model. experimental results show that cesta leads to a further performance improvement against strong baselines and achieves new state-of-the-art results. future works will focus on the representation of emotional consistency for each interlocutor in the conversation. we also plan to incorporate multimodal information into cesta and evaluate it on more natural conversation datasets. since cesta needs to use emotion information of the whole dialogue, we will study its performance on the online dialogue system which has no access to the information of future emotions.