T el ME : Teacher-leading Multimodal Fusion Network for Emotion Recognition in Conversation (2024)

Abstract: Emotion Recognition in Conversation (ERC) plays a crucial role in enabling dialogue sys- tems to effectively respond to user requests. The emotions in a conversation can be identi- fied by the representations from various modal- ities, such as audio, visual, and text. How- ever, due to the weak contribution of non-verbal modalities to recognize emotions, multimodal ERC has always been considered a challenging task. In this paper, we propose Teacher-leading Multimodal fusion network for ERC (TelME). TelME incorporates cross-modal knowledge distillation to transfer information from a lan- guage model acting as the teacher to the non- verbal students, thereby optimizing the efficacy of the weak modalities. We then combine multi- modal features using a shifting fusion approach in which student networks support the teacher. TelME achieves state-of-the-art performance in MELD, a multi-speaker conversation dataset for ERC. Finally, we demonstrate the effec- tiveness of our components through additional experiments.

## introduction emotion recognition holds paramount importance, enhancing the engagement of conversations by providing appropriate responses to the emotions of users in dialogue systems @xcite . the application of emotion recognition spans various domains, including chatbots, healthcare systems, and recommendation systems, demonstrating its versatility and potential to enhance a wide range of applications @xcite . emotion recognition in conversation (erc) aims to identify emotions expressed by participants at each turn within a conversation. the dynamic emotions in a conversation can be detected through multiple modalities such as textual utterances, facial expressions, and acoustic signals @xcite figure 1 : examples of multimodal erc. even the same "okay" answer varies depending on the conversation situation and captures emotions in various modalities. 2022; @xcite @xcite . figure 1 illustrates an example of a multimodal erc. much research on erc has mainly focused on context modeling from text modality, disregarding the rich representations that can be obtained from audio and visual modalities. text-based erc methods have demonstrated that contextual information derived from text data is a powerful resource for emotion recognition @xcite @xcite . however, non-verbal cues such as facial expressions and tone of voice, which are not covered by text-based methods, provide important information that needs to be explored in the field of erc. multimodal approaches demonstrate the possibility of integrating features from three modalities to improve the robustness of erc systems @xcite @xcite . nevertheless, these frameworks frequently ignore the varying degrees of impact the individual modalities have on emotion recognition and instead treat them as homogeneous components. this implies a promising opportunity to improve the erc system by differentiating the level of contribution made by each modality. in this paper, we propose teacher-leading multimodal fusion network for the erc task (telme) that strengthens and fuses multimodal information by accentuating the powerful modality while bolstering the weak modalities. knowledge distillation (kd) can be extended to transfer knowledge across modalities, where a powerful modality can play the role of a teacher to share knowledge with a weak modality @xcite . while figure 2 shows the robustness of text in erc tasks compared to the other two modalities, the other modalities present valuable information nonetheless. thus, telme enhances the representations of the two weak modalities through kd utilizing the text encoder as the teacher. our approach aims to mitigate heterogeneity between modalities while allowing students to learn the preferences of the teacher. telme then incorporates attentionbased modality shifting fusion, where the student networks strengthened by the teacher at the distillation stage assist the robust teacher encoder in reverse, providing details that may not be present in the text. specifically, our fusion method creates displacement vectors from non-verbal modalities, which are used to shift the emotion embeddings of the teacher. we conduct experiments on two widely used benchmark datasets and compare our proposed method with existing erc methods. our results show that telme performs well on both datasets and particularly excels in multi-party conversations, achieving state-of-the-art performance. the ablation study also demonstrates the effectiveness of our knowledge distillation strategy and its interaction with our fusion method. our contributions can be summarized as follows: ## conclusion this paper proposes teacher-leading multimodal fusion network for erc (telme), a novel multimodal erc framework. telme incorporates a cross-modal distillation that transfers the knowledge of text encoders trained in linguistic contexts to enhance the effectiveness of non-verbal modalities. moreover, we employ the fusion method that shifts the features of the teacher model by referring to non-verbal information. we show through experiments on two benchmarks that our approach is practical in erc. telme delivers robust performance in both datasets and especially achieves state-of-the-art results in the meld dataset, which consists of multi-party conversational scenarios. we believe that this research presents a new direction that can incorporate multimodal information for erc.