D etective NN : Imitating Human Emotional Reasoning with a Recall-Detect-Predict Framework for Emotion Recognition in Conversations (2024)

Abstract: Emotion Recognition in conversations (ERC) involves an internal cognitive process that interprets emotional cues by using a collection of past emotional experiences. However, many existing methods struggle to decipher emotional cues in dialogues since they are insufficient in understanding the rich historical emotional context. In this work, we introduce an innovative Detective Network (DetectiveNN), a novel model that is grounded in the cognitive theory of emotion and utilizes a “recall-detect-predict” framework to imitate human emotional reasoning. This process begins by ‘recalling’ past interactions of a specific speaker to collect emotional cues. It then ‘detects’ relevant emotional patterns by interpreting these cues in the context of the ongoing conversation. Finally, it ‘predicts’ the speaker’s current emotional state. Tested on three benchmark datasets, our approach significantly outperforms existing methods. This highlights the advantages of incorporating cognitive factors into deep learning for ERC, enhancing task efficacy and prediction accuracy.

## introduction in recent years, recognizing emotions in dialogues has gained increasing attention in the field of natural language processing (nlp). this is driven by its vast potential for application in areas like humancomputer interaction and empathetic dialogue systems @xcite @xcite . in the realm of conversational emotion recognition, interpreting emotional cues embedded in conversational context is crucial @xcite . emotional cues in conversations are subtle patterns or indicators that hint at the underlying emotions of a speaker. these cues often act as triggers for the emotions expressed in current utterances @xcite . erc seeks to detect and interpret these emotional clues within the flow of conversation, aiming for a nuanced understanding of the emotional context. traditional erc approaches typically adopt a 'recall-then-predict' strategy @xcite , modeling both speaker-level and dialogue-level contexts to predict emotional states in conversations. dialoguegcn @xcite models interactions between speakers using graph networks to capture emotional cues throughout the conversation. dialogxl @xcite introduces a dialog-aware self-attention mechanism within a transformer structure to capture emotional cues, including intra-and inter-speaker dependencies. c-lstm @xcite leverages an lstm-based approach to encode the global context, whereas dialoguernn @xcite employs grus to track both speaker state and global state for each conversation. cosmic @xcite leverages external commonsense knowledge to enhance the model's ability to detect rich emotional cues. additionally, dialogue-crn @xcite employs a multi-turn reasoning module that extracts and combines emotional clues from the dialogue. however, these methods lack a distinct process for interpreting extracted emotional cues before classifying the emotion. emotion recognition can be understood as the process of deciphering emotional cues to comprehend the cognitive context, aligning with the cognitive theory of constructed emotion @xcite @xcite . this theory suggests that emotions are formed from an individual's cognitive context, shaped by their thoughts, memories, and social interactions @xcite . inspired by this theory, we approach erc tasks as an internal cognitive process that deciphers each participant's emotional cues based on their past emotional experiences in a dialogue. this process involves identifying and organizing emotional cues, synthesizing them into a coherent emotional narrative, and subsequently examining this narrative throughout the conversational context to validate the cues. we propose a novel detective network (detectivenn) with a 'recall-detect-predict' strategy for enhanced erc accuracy. the detectivenn model features a detection phase that deciphers emotional cues throughout the conversation context, connecting these cues to decode the evolution of a speaker's emotional responses. this phase reveals patterns in a speaker's emotional flows, akin to a detective piecing together clues to map an individual's emotional states. detectivenn begins with a recall phase, where we utilize gated recurrent units (grus), a type of sequence-based model, for their efficiency in handling sequential data and their ability to capture long-term dependencies. this is crucial for retaining diverse contextual information from the speakers' emotional experiences and interactions. this approach is inspired by the pioneering work of hu @xcite and yang @xcite , who demonstrated the efficacy of sequence-based models in learning diverse contextual information. in the detection phase, we employ a transformerlike architecture to iteratively analyze and decode emotional cues drawn from the extensive emotional experiences of a specific speaker. this phase is divided into two key operations: an examination process and a conscious detection process. during the examination phase, we employ transformer encoder blocks with multi-head attention (mha) layers. mha layers enable the model to simultaneously focus on various parts of the input sequence by assigning disparate levels of attention or importance. transformer encoder blocks integrate cues to achieve a deep understanding of the speaker's emotional context. the conscious detection process employs a cross-attention mechanism, probing the speaker's constructed emotional narrative and integrating the dynamic interplay between emotional cues and the speaker's historical interactions. this method uncovers patterns that decode the speaker's emotional journey, offering insights into the evolution of emotional states over time. following the insights gained from the detection phase, an emotion classifier predicts the emotion label of each utterance. by incorporating the 'recall-detect-predict' framework, detectivenn effectively mirrors the cognitive reasoning process humans use to understand emotional states. we hy-pothesize that integrating cognitive reasoning into deep learning models significantly enhances their capability to analyze and interpret emotions in each dialogue segment. to assess the efficacy of our proposed model, extensive experiments were conducted on three widely accepted benchmark datasets: iemocap, emorynlp and dailydialog. the experimental results demonstrate that our model significantly outperforms existing methods, primarily attributed to the application of a cognitive approach in deciphering emotional cues. the primary contributions of our research are as follows: ## related work the erc field has advanced significantly, emphasizing the extraction and integration of emotional cues from conversations. this progress can be grouped into three major methodologies: sequencebased models, pre-trained language model-based models, and graph-based models. sequence-based models: dialoguernn @xcite uses grus to track emotional states by integrating speaker identity, context, and emotions from neighboring utterances. dialogue-crn @xcite combines cognitive theories of emotion with lstm networks for iterative extraction and integration of emotional cues. bc-lstm @xcite employs bidirectional lstms to capture the influence of preceding and following utterances. cmn @xcite models past utterances using grus in a multimodal approach. emotionic (yingjian et al., 2023) inte-2 9171 grates attention and recurrence with immha, di-agru, and skipcrf for comprehensive emotion detection. cosmic @xcite incorporates commonsense knowledge with grus to model various conversational states. sacl-lstm @xcite uses contrastive learning to compare emotional representations from both original and adversarial data, allowing the model to better generalize in recognizing emotions across different conversational contexts, while df-erc @xcite uses contrastive learning to separate modality and utterance features. eacl @xcite utilizes label encodings as anchors and develops an auxiliary loss to better distinguish similar emotions. pre-trained language model-based models: di-alogxl @xcite uses xlnet @xcite and dialogue-level self-attention to handle multi-party conversation dynamics. emoberta @xcite employs roberta @xcite to predict speaker emotions by learning both speaker-level and dialogue-level context. graph-based models: dialoguegcn @xcite uses a graph convolutional neural network to model conversational context by representing utterances as nodes and their dependencies as edges. @xcite and @xcite employ graph convolutional networks with attention mechanisms to capture context-sensitive and speaker-sensitive dependencies. ## visualization & analysis we present heatmap visualizations to interpret the model's emotion predictions using a randomly selected conversation sample from the iemocap test dataset. the first heatmap (see figure 2 ) illustrates features extracted from the transformer encoder's final layer during the detection phase, while the second heatmap (see figure 3 ) shows activations from the first layer of the mlp. in both visualizations, the x-axis represents the dimensions of the feature vector, each value corresponding to the average feature activation for a specific emotion class. the y-axis denotes different emotion classes. the second heatmap (see figure 3 ) shows more intense activations for the frustrated class compared to the angry class, suggesting that the model integrates contextual information to decode the complex interplay between emotions, events, and the unfolding narrative within a sentence. to quantify these activations, we calculate a cosine similarity matrix based on the average activations from the output of the first layer of the mlp for each emotion class across the entire test dataset (see in this paper, we introduce detectivenn, a novel framework for emotion recognition in conversation. this framework utilizes an innovative recalldetect-predict structure to interpret emotions in conversations. initially, detectivenn identifies key emotional cues within the dialogue. subsequently, it conducts a thorough analysis of these cues to accurately predict the emotional state. rigorously evaluated across three benchmark datasets, detectivenn has demonstrated its superiority over existing models, revealing the profound impact of integrating cognitive reasoning into deep learning architectures. this cognitive factor plays an important role not only in enhancing the model's efficiency and accuracy in prediction but also in advancing erc methodologies. ## limitations detectivenn improves emotion prediction accuracy for long-term dialogue turns but struggles with short-term turns due to its reliance on extended interaction context. this context depen-dence limits its ability to detect emotional cues in brief exchanges. additionally, lacking information on speakers' personality traits hinders detec-tivenn's performance in capturing complex emotional dynamics, such as sarcasm and humor, which are prevalent in datasets like emorynlp from the "friends" tv series. integrating personality trait knowledge is essential for accurately predicting nuanced emotions in conversations.