Summarize before Aggregate: A Global-to-local Heterogeneous Graph Inference Network for Conversational Emotion Recognition (2020)

Abstract: Conversational Emotion Recognition (CER) is a crucial task in Natural Language Processing (NLP) with wide applications. Prior works in CER generally focus on modeling emotion influences solely with utterance-level features, with little attention paid on phrase-level semantic connection between utterances. Phrases carry sentiments when they are referred to emotional events under certain topics, providing a global semantic connection between utterances throughout the entire conversation. In this work, we propose a two-stage Summarization and Aggregation Graph Inference Network (SumAggGIN), which seamlessly integrates inference for topic-related emotional phrases and local dependency reasoning over neighbouring utterances in a global-to-local fashion. Topic-related emotional phrases, which constitutes the global topic-related emotional connections, are recognized by our proposed heterogeneous Summarization Graph. Local dependencies, which captures short-term emotional effects between neighbouring utterances, are further injected via an Aggregation Graph to distinguish the subtle differences between utterances containing emotional phrases. The two steps of graph inference are tightly-coupled for a comprehensively understanding of emotional fluctuation. Experimental results on three CER benchmark datasets verify the effectiveness of our proposed model, which outperforms the state-of-the-art approaches.

## introduction conversational emotion recognition (cer) has attracted increasing interests for its promising applications in intelligent interactive systems with diverse functionalities, including medical-care systems and online recommendation systems @xcite @xcite . as shown in figure 1 , conversations in cer datasets are segmented into multiple utterances based on breaths or pauses of the speaker, and each utterance is associated with an emotion label. existing works with deep learning approaches generally capture emotional features solely from an utterance-level perspective by modeling interactions between utterances via recurrent neural network (rnn) structures @xcite @xcite or graph structures @xcite . however, phrase-level semantic connection between utterances is still underexplored, creating a substantial barrier for comprehensively understanding of the source of emotional fluctuation. an example of a topic-related emotional phrase is shown in figure 1 , where the noun phrase "fifty dollars" is not an emotional expression at first, but under the topic of compensation, it has been associated with anger and frustration by referring to the unsatisfactory compensation. therefore, it is crucial to recognize phrase-level emotional patterns globally across different utterances. furthermore, topic-related emotional phrases generally scatter in different utterances throughout the conversation, and the emotions they convey depend on local context. to accurately distinguish emotions behind these "fifty dollars" phrases, the model is required to be comprehensively aware of both the topic about compensation and the subtle differences in attitude from the responses of the male speaker. by seamlessly ## methodology before diving into the details of our proposed model, we begin by introducing the basic mathematical notations and terminologies for the task of cer. the goal of cer is to infer the emotion label (happy, sad, neutral, angry, excited, and frustrated) for each utterance in a conversation. given a cer dataset d, an example of the dataset is denoted as {u i , s i , y i } n @xmath0, where @xmath1, u 2 , ..., u n } represents the n utterances of the conversation with each utterance u i containing l i words. assuming there are m participants, the speaker corresponds to the i-th utterance u i is represented as s @xmath2, ..., m -1}. y @xmath3, ..., n -1} indicates the emotion label for utterance u i . an overview of our proposed sumagggin model is shown in figure 2 , which consists of an encoding module, a summarization graph, an aggregation graph and a classification module. cnn cnn cnn cnn 2. summarization graph 3. aggregation graph reasoning utterance-level features phrase-level features ## conclusion in this work, we present a two-stage summarization and aggregation graph inference network (sumagggin) for the cer task. combining global topic-related emotional interactions from the heterogeneous summarization graph and short-term emotional effects from the aggregation graph, our model is capable of comprehensively capturing emotional influences in a global-to-local fashion. experimental results show that our proposed sumagggin outperforms the state-of-the-art approaches on three cer benchmark datasets. in the future, we plan on applying our two-stage graph inference framework to other multi-turn dialogue tasks.