Past, Present, and Future: Conversational Emotion Recognition through Structural Modeling of Psychological Knowledge (2021)

Abstract: Conversational Emotion Recognition (CER) is a task to predict the emotion of an utterance in the context of a conversation. Although modeling the conversational context and interactions between speakers has been studied broadly, it is important to consider the speaker’s psychological state, which controls the action and intention of the speaker. The state-of-the-art method introduces CommonSense Knowledge (CSK) to model psychological states in a sequential way (forwards and backwards). However, it ignores the structural psychological interactions between utterances. In this paper, we propose a pSychological-Knowledge-Aware Interaction Graph (SKAIG). In the locally connected graph, the targeted utterance will be enhanced with the information of action inferred from the past context and intention implied by the future context. The utterance is self-connected to consider the present effect from itself. Furthermore, we utilize CSK to enrich edges with knowledge representations and process the SKAIG with a graph transformer. Our method achieves state-of-the-art and competitive performance on four popular CER datasets.

## introduction as one of the most ubiquitous ways of communicating, conversations contain rich information and emotional expressions of the participants. with the explosive growth of conversational data on the internet, it is of great importance to employ machines to automatically identify the emotions expressed by speakers in the conversation. therefore, in recent years, conversational emotion recognition (cer) receives increasing attention from the researchers @xcite @xcite . unlike traditional emotion recognition, cer needs to model not only the semantic information of an utterance, but also the conversational * zheng lin is the corresponding author. the utterance #1 provides the action of speaker b for #2, and #4 provides the intention. both give positive and rational hints for #2 to predict the positive emotion excited. the descriptions of action, intention are generated by comet @xcite . although these works yield significant performance, the modeling of conversational context and speakers does not consider psychological states of speakers. the psychological state will control the speaker's action and intention along the conversation, which can help predict the emotion more reasonably. as the original conversation provides no extra information about psychological states, to guide a model to realize psychological states, com-monsense knowledge (csk) can be introduced. from the perspective of csk proposed by @xcite , which is a kind of widely-used socialized csk @xcite , action means what the speaker wants to do in the next step, which can be triggered by speaker him/herself or other speakers. intention means what the speaker wanted to do before this step, which can only be inferred by speaker him/herself. therefore, for a targeted utterance, the action can be inferred from its past context, the intention from its future context. as illustrated in fig. 1 , the targeted utterance #2 can be positively enhanced by the action inferred from #1 of speaker a and the intention from #4. cosmic @xcite introduces this kind of csk into cer to model the speaker's psychological state, and then utilizes bidirectional grus to model these states in every time step. however, cosmic ignores the structural psychological influences from contextual utterances to the targeted utterance (i.e. an utterance can directly and explicitly pass psychological messages to other utterances over several time steps, which is more than just sequential and implicit modeling of psychological states over utterances). in addition, modeling all psychological states both forwards and backwards does not conform with the nature of the csk @xcite mentioned above (e.g. intention cannot be inferred forwards and should be only inferred backwards as illustrated in fig. 1 ). to alleviate these issues, we propose a psychological-knowledge-aware interaction graph (skaig). utterances, which are locally connected, act as the nodes in the graph. there are four relations considered in skaig: xwant, owant, xintent, xeffect. for a targeted utterance, xwant, owant model the action indicated by utterances in the past context with the same speaker (x) and other speakers (o) respectively. conversely, xintent models the intention inferred by utterances in the future context. and xeffect is the self-connected relation to model the influence from the present utterance itself. by taking the three sources: past, present, and future into consideration, we believe the graph can more structurally and rationally enhance context modeling. furthermore, these relations will be assigned to edges between utterances accordingly. therefore, edges take the role to model the psychological interactions between utterances. to realize this, we enrich edges with their corresponding knowledge representations. these representations are produced by commonsense transformer comet @xcite which takes utterances and relations as inputs. as edges in skaig possess knowledge representations that require to be considered, we therefore utilize the graph transformer @xcite for message passing. we then use the final outputs for classification. to evaluate our method, we conduct experiments on four datasets: iemocap, dailydialog, emorynlp, and meld. our method achieves stateof-the-art performance on the first three datasets, and competitive performance on meld. further experiments also demonstrate the efficacy of our proposed method. ## methodology in this section, we first formalize the cer task, and then elaborate on our proposed model. the framework of the model (illustrated in fig. 2 ) consists of three parts: utterance-level encoder, conversationlevel encoder, and emotion classifier. ## related work conversational emotion recognition is a hot-spot task in recent years. unlike traditional emotion recognition, cer involves conversational context. @xcite ; @xcite utilize memory network to model such context. to consider the speaker and listener in the conversation, @xcite propose dialoguernn, which utilizes grus to update speakers' states and the global line of the conversation. dialoguegcn @xcite models two relations between speakers: self and inter-speaker dependencies, and utilizes graph networks @xcite to model the graph constructed by these relations. @xcite propose a graph attention to extract information from external knowledge base and utilize transformer @xcite to model conversations. furthermore, with the spreading of pretrained models, new works are based on these highperformance and large-scale models. @xcite propose a relation-aware position encoding based on dialoguegcn and utilize bert @xcite to encode utterances. @xcite utilize bert and propose a speaker relation prediction task to augment cer. @xcite utilize xlnet @xcite and model the whole conversation in one step. by introducing commonsense knowledge to cer, @xcite propose comsic which is based on di-aloguernn equipped with roberta @xcite to model the speakers' internal, external, intent states. ## conclusion in this paper, we study conversational emotion recognition. the sota method ignores the psychological interactions between utterances over several time steps and does not conform with the nature of psychological states. we therefore propose a psychological-knowledge-aware interaction graph (skaig). the graph contains four relations to model psychological states of speakers. enhanced by commonsense knowledge and the deployment of the graph transformer, our method yields sota or competitive performance on benchmark datasets.