Multimodal Emotion Recognition in Conversations: A Survey of Methods, Trends, Challenges and Prospects (2025)

Abstract: While text-based emotion recognition methods have achieved notable success, real-world dialogue systems often demand a more nuanced emotional understanding than any single modality can offer. Multimodal Emotion Recognition in Conversations (MERC) has thus emerged as a crucial direction for enhancing the naturalness and emotional understanding of human-computer interaction. Its goal is to accurately recognize emotions by integrating information from various modalities such as text, speech, and visual signals. This survey offers a systematic overview of MERC, including its motivations, core tasks, representative methods, and evaluation strategies. We further examine recent trends, highlight key challenges, and outline future directions. As interest in emotionally intelligent systems grows, this survey provides timely guidance for advancing MERC research.

## introduction emotion recognition in conversations (erc) @xcite ) is an increasingly important task in natural language processing (nlp), focusing on identifying the emotional state associated with each utterance in a dialogue. unlike conventional emotion classification on isolated sentences, erc requires understanding the interplay between utterances and tracking speakerspecific context across the conversation @xcite . its relevance has grown due to its potential in various real-world applications, including social media monitoring @xcite , intelligent healthcare services @xcite , and the design of emotionally aware dialogue agents @xcite . however, human emotions are typically conveyed through multiple modalities, including auditory (e.g., speech), visual (e.g., facial expressions, gestures), and linguistic (e.g., the semantic content conveyed by transcribed text). as a result, recent research (e.g., @xcite @xcite has increasingly focused on multimodal settings in dialogue, a direction we refer to as multimodal emotion recognition in conversations (merc). researchers aim to identify the emotional state of a given utterance by integrating contextual information from different modalities, which often includes subtle personal emotional states such as happiness, anger, and hatred @xcite , thereby improving the effectiveness of emotion recognition in dialogues. figure 1 illustrates an example of erc with textual, acoustic, and visual inputs. multimodal emotion recognition (mer) itself has gained increasing attention due to the challenges of integrating diverse modalities, prompting research in both non-conversational and conversational settings. existing surveys such as @xcite and aruna @xcite focus on non-conversational mer but overlook key aspects like interlocutor modeling and context. @xcite reviews both unimodal and multimodal conversational mer, yet primarily centers on fea-ture fusion, offering limited insight into core challenges such as cross-modal alignment, reasoning, modality missingness, and conflicts. despite growing interest, the merc task remains underexplored. existing surveys @xcite also lag behind recent advances, particularly the rise of multimodal large language models (mllms). to bridge this gap, we present a timely and comprehensive review of merc. we first introduce the task definition and our survey methodology ( §2), benchmark datasets and evaluation methods ( §3), followed by a review of preprocessing techniques ( §4); then categorize recent methods ( §5) and outline key challenges and prospects ( §6). in summary, the specific contributions of this survey are threefold: we identify key open issues in the merc domain and put forward several potential future research directions, aiming to guide ongoing and future investigations by researchers and practitioners in merc. in this section, we present the task settings of merc and outline the methodology employed to compile the content of this survey, which details the strategy and selection criteria used to curate the final content for this survey paper. modalities. in the context of merc, we define a modality as a distinct source or channel of information that conveys emotional or communicative signals. these modalities correspond to observable representations derived from different sensory inputs and are typically processed in separate feature spaces. the most common modalities include: task definition. given a dialogue @xmath0, u 2 , . . . , u n } consisting of n utterances spoken by multiple speakers, the goal of the merc task is to predict an emotion label e @xmath1. each utterance is associated with three modalities: textual (u t i ), acoustic (u a i ), and visual (u v i ), which provide complementary information for emotion recognition. the multimodal representation of an utterance is denoted as: ## datasets and evaluation in this section, we describe the evaluation datasets and the evaluation metrics used for the merc task, focusing on multimodal resources across multiple languages. for more detailed information about the single benchmarks, see appendix §a. as shown in ## feature processing preprocessing dataset features is essential for effectively extracting meaningful information. we summarize feature preprocessing methods used in prior merc research and analyze the typical preprocessing pipeline, which is often tailored to conversational settings. specifically, we distinguish between two key components: feature extraction and context modeling. ## methodology this section discusses state-of-the-art approaches to the merc tasks. we summarize them from three perspectives: graph-based methods ( §5.1), fusionbased methods ( §5.2), and generation-based methods ( §5.3). an overview of the methods and subcategories with representative examples is presented in figure 2 . it is worth noticing that some methods in merc inherently involve multiple components (e.g., fusion modules within graph-based or generationbased frameworks). our taxonomy is not intended to be strictly disjoint, but rather to organize the literature based on the core modeling paradigm or innovation focus of each method. the categorization of methods is thus as follows: ## challenges and prospects based on current trends and developments in the merc task, in this section, we outline several existing challenges and open questions, highlighting opportunities for future improvements. we structure our discussion along a logical progression: starting with foundational limitations in data collection and fair compliance, we then examine challenges in multimodal modeling and conclude with considerations for real-world deployment. this trajectory reflects how upstream issues in data and modeling propagate downstream, ultimately shaping the robustness, inclusivity, and applicability of merc systems. ## conclusion merc seeks to understand emotions by integrating various modalities in to dialogue of linguistic, acoustic, visual signals, and beyond. while recent progress has introduced diverse modeling strategies, significant challenges remain in data scarcity, modality alignment, and generalization across languages and cultures. this survey provides a structured review of the merc landscape, compares representative approaches, and highlights key open research problems. we hope it serves as a practical reference to support the future development of robust and inclusive emotion recognition systems.