Multiple Knowledge-Enhanced Interactive Graph Network for Multimodal Conversational Emotion Recognition (2024)

Abstract: Multimodal Emotion Recognition in Conversations (ERC) aims to identify emotions in conversational videos. Current efforts focus on modeling both context-sensitive and speaker-sensitive dependencies and multimodal fusion. Despite the progress, models in Multimodal ERC (MERC) still struggle due to a lack of CommonSense Knowledge (CSK). In contrast, models in textual ERC typically employ CSK to enhance emotion inference. However, in multimodal scenarios, relying solely on textual CSK while neglecting visual CSK may hinder the understanding of visual emotional cues. To address this, we introduce a novel approach called Multiple Knowledge Enhanced Interactive Graph Network (MKE-IGN) to integrate multiple knowledge, such as textual and visual CSK, into the edge representations, thereby facilitating the modeling of relations between utterances and different types of CSK. Furthermore, considering that irrelevant CSK might be retained as noise, MKE-IGN adaptively selects this CSK guided by the mood-congruent effect and refines it based on contexts. Experimental results show that MKE-IGN outperforms state-of-the-art methods on two popular datasets.

## introduction emotion recognition in conversations (erc) is challenging due to its dynamic and spontaneous nature, as individuals express various emotions @xcite . recently, erc has garnered substantial interest because of its valuable applications in recommendation systems @xcite and dialogue generation @xcite @xcite . traditional erc paradigms primarily focus on the textual modality and typically require modeling both context-sensitive and speaker-sensitive dependencies @xcite @xcite prompt: please describe the event that occurred in this picture event people's intents at present: display the furniture that ross and rachel are carrying. events that can happen after: notice the sofa being moved. visual csk xeffect of rachel: cries xreact of rachel: sad xwant of rachel: to go home … owant of ross: to thank rachel oeffect of ross: they are left alone oreact of ross: sad visual comet comet rachel: okay, y'know what? there is no more left, left! (anger) textual csk figure 1 : an example highlights the limitation of relying solely on textual if-then csk for comprehending visual emotional cues in multimodal scenarios. @xcite . unfortunately, these methods struggle to replicate human-like understanding due to challenges in interpreting emotions expressed through common sense knowledge (csk) @xcite . recent research has addressed this limitation by modeling knowledge-sensitive dependencies into erc models (zhong and di @xcite @xcite using knowledge bases such as conceptnet @xcite , senticnet @xcite , and if-then csk generated by the pre-trained commonsense transformer (comet) @xcite . comet is trained on atomic @xcite , a collection of everyday inferential if-then commonsense knowledge organized through textual descriptions. this body of knowledge has significantly advanced the field of erc. despite the progress, textual cues prove insufficient for understanding deep emotions @xcite . multimodal erc (merc), incorporating audio and visual cues alongside the text, is gaining increasing research attention @xcite . existing merc methods focus on aggregation-based fusion through concatenation @xcite @xcite , attention network @xcite , and heterogeneous graphbased fusion @xcite @xcite . however, these methods often rely on future utterances to predict the current one's emotion, which is impractical in real-world scenarios. more importantly, integration of csk in merc has been overlooked. relying solely on textual if-then csk can impede understanding of visual emotional cues, as it fails to incorporate visually contextualized information. for instance, visual if-then csk can infer from an image that individuals moving a sofa will likely handle it with care later, a nuance that textual if-then csk misses (see fig. 1 ). moreover, incorporating diverse types of ifthen csk inevitably introduces irrelevant csk as noise @xcite @xcite in merc. yet, research on the selection of multiple csk still remains unexplored. to address the above issues, we propose an innovative solution, the multiple knowledge-enhanced interactive graph network (mke-ign), which employs a directed graph structure, eliminating the introduction of future utterances. it integrates multiple csk (visual and textual if-then csk) from visualcomet @xcite and comet as the edge representations to enhance utterance understanding. this compensates for the deficiency in textual csk regarding the comprehension of visual emotional clues. especially, compared to incorporating knowledge into node representations, the edge representations in mke-ign enable the modeling of relations between utterances and various types of if-then csk through diverse edge types. furthermore, to reduce noise from irrelevant csk, mke-ign leverages lassonet @xcite to adaptively select the most emotionally relevant csk from the generated candidate set, guided by the mood-congruent effect. considering that ifthen csk is often regarded as a fixed, gradient-less representation, limiting its adaptability across various conversational contexts, we refine it based on contexts during training. our main contributions can be summarized as follows: in addition, the role of participants in erc is also important to the speaker's emotional state @xcite . to model the speaker-sensitive dependency, researchers have a greater emphasis on speaker-specific models @xcite , graph-based models @xcite , and so on. for example, @xcite utilized three grus to track global context, speaker state, and emotional state in conversations. @xcite introduced a speaker-aware network to extract emotional cues by simulating cognitive conversational dynamics. @xcite and @xcite employed a graph-based model to model self-and inter-speaker dependencies. knowledge-sensitive models: although the above works have achieved respectable performance, they are not able to work like a human because of the lack of commonsense knowledge @xcite . recent research has addressed this limitation by modeling knowledge-sensitive dependencies into erc models @xcite @xcite @xcite using knowledge bases such as conceptnet @xcite , senticnet @xcite , and if-then csk generated by pre-trained commonsense transformers (comet) @xcite , which has significantly advanced the field of erc. for instance, @xcite proposed a graph-based model to model the knowledge-sensitive dependencies by incorporating concepts retrieved from con-ceptnet @xcite . @xcite introduced if-then csk into the conversation model through customized architectures, enhancing its emotional reasoning capabilities. ## methodology in this section, we provide a detailed introduction to each component of the proposed mke-ign, as depicted in fig. 2 . ## conclusion this paper introduces a novel mke-ign for the merc task, employing a directed graph structure to prevent future utterances from influencing the current one. more importantly, mke-ign integrates multiple csk as edge representations for the modeling of relations between utterances and different types of csk, improving the understanding of utterances in multimodal scenarios. furthermore, considering that irrelevant csk might be retained as noise, mke-ign adaptively selects this csk guided by the mood-congruent effect and refines it based on contexts. extensive evaluations and an ablation study prove the superiority of mke-ign and the significant impact of its components.