A Survey on Multi-modal Intent Recognition: Recent Advances and New Frontiers (2025)

Abstract: Multi-modal intent recognition (MIR) requires integrating non-verbal cues from real-world contexts to enhance human intention understanding, which has attracted substantial research attention in recent years. Despite promising advancements, a comprehensive survey summarizing recent advances and new frontiers remains absent. To this end, we present a thorough and unified review of MIR, covering different aspects including (1) Extensive survey: we take the first step to present a thorough survey of this research field covering textual, visual (image/video), and acoustic signals. (2) Unified taxonomy: we provide a unified framework including evaluation protocol and advanced methods to summarize the current progress in MIR. (3) Emerging frontiers: We discuss some future directions such as multi-task, multi-domain, and multi-lingual MIR, and give our thoughts respectively. (4) Abundant resources: we collect abundant open-source resources, including relevant papers, data corpora, and leaderboards. We hope this survey can shed light on future research in MIR.

## introduction intent recognition (ir) foot_0 has achieved remarkable success in unimodal settings, particularly in textual @xcite and visual domains @xcite . however, traditional unimodal approaches are inherently limited in capturing the complexity of real-world communication, where intent is often conveyed through a combination of verbal and non-verbal signals. this limitation arises because human communication is inherently multi-modal, relying not only on explicit textual content but also on prosodic variations, facial expressions, and body gestures. to this end, multi-modal intent recognition (mir) has emerged as a key research direction for enhancing intent understanding by systematically integrating diverse modalities, as illustrated in figure 1 . leveraging textual semantics, acoustic features (e.g., tone and prosody), and visual cues (e.g., gestures and facial expressions), mir facilitates a more holistic interpretation of human intent @xcite @xcite , with broad implications for applications such as human-computer interaction @xcite . despite rapid advancements, there is still a lack of a comprehensive survey that summarizes recent advances and new frontiers. to bridge this gap, we present the first survey on mir, reviewing over 60 cutting-edge studies published between 2019 and 2024. in a nutshell, our contributions can be summarized as follows: ❶ extensive survey: we categorize existing studies based on their modality combinations, encompassing textual-visual, textual-acoustic, and textual-visual-acoustic intent recognition. ❷ unified taxonomy: we provide a systematic review of existing progress from evaluation protocol and advanced methods perspectives, establishing three leaderboards under unified metrics. ❸ emerging frontiers: we highlight key challenges recent advances of mir textual-visual ir ( §3) evaluation protocol ( §3.1) benchmark datasets e.g., mdid @xcite , behance intent discovery @xcite , multimet @xcite , msairs @xcite , mcic @xcite , intentqa @xcite evaluation metrics e.g., accuracy, area under curve, precision, recall, (weighted) f-score advanced methods ( §3.2) plm-based methods e.g., @xcite , @xcite , @xcite , @xcite , vgt @xcite , mmsair @xcite , hqga @xcite llm-based methods e.g., blindgpt @xcite , cavir @xcite , sevila @xcite , llovi @xcite , gcg @xcite , vamos @xcite textual-acoustic ir ( §4) evaluation protocol ( §4.1) benchmark datasets e.g., fluent speech commands (fsc) @xcite , slurp @xcite , minds-14 @xcite evaluation metrics e.g., accuracy, slu-f1 advanced methods ( §4.2) pipeline methods e.g., phoneme-bert @xcite , spokencse @xcite , ml-lmcl @xcite , mclf @xcite , pcad @xcite end-to-end methods e.g., mtl-slt @xcite , speech-brain @xcite , espnet-slu @xcite matl @xcite , hubert slu @xcite , cif-pt @xcite , cti @xcite llm-based methods e.g., chatgpt, speechgpt @xcite ) benchmark datasets e.g., mintrec @xcite , meld-da @xcite , emoint-md (singh et al., 2022), mintrec2.0 @xcite , mc-eiu @xcite evaluation metrics e.g., accuracy, (weighted) precision, recall, (weighted) (average) f1-score advanced methods ( §5.2) msa-inspired fusion e.g., mult @xcite , mag-bert @xcite , misa @xcite knowledge-driven modeling e.g., cagc @xcite , teco @xcite , mintood @xcite information-disentangled strategy e.g., lvamoe @xcite , duodn @xcite , inmu-net @xcite semantic-enhanced alignment e.g., tcl-map @xcite , sdif-da @xcite figure ## background and preliminary this section first outlines the background of mir, and then provides an overview of mir. background. mir enhances intent understanding by integrating multiple modalities, distinguishing it from unimodal ir. however, research on mir remains in its early stages due to: ❶ dataset: while numerous multi-modal language datasets have been introduced for tasks such as sentiment analysis and emotion recognition, highquality datasets specifically annotated for mir remain scarce @xcite . ❷ methodology: ir is inherently more abstract than tasks involving explicit emotional expression @xcite , making effective multi-modal fusion considerably more complex. preliminary. given a multi-modal input that may comprise any combination of textual (t), visual (v), 2 https://github.com/zhihong-zhu/mir-survey and acoustic (a) modalities, mir aims to determine the most appropriate intent label(s) of the input as: ## textual-visual intent recognition text on social media or e-commerce platforms is often accompanied by visual signals (i.e., images or videos), which are common ways for users to express their intentions. in the following, we detail the collected textual-visual ir benchmarks and their corresponding metrics ( §3.1), as shown in modality evaluation dataset name source #intent v t a metric additional remarks mdid instagram 7 ✓ ✓ ✗ acc, annotated manually via consensus (kruk et al., 2019) emnlp auc multimet twitter, facebook (zhang et al., 2021) acl and (ye et al., 2019) 4 ✓ ✓ ✗ acc annotates metaphor authorial intent behance intent discovery behance livestreams 2 ✓ ✓ ✗ p, r, f manually annotated via crowdsourcing; (maharana et al., 2022) naacl each sample contains a transcribed phrase mcic 30,716 multi-modal dialogues with (yuan et al., 2022) nlpcc jd.com 212 ✓ ✓ ✗ acc images and ocr texts (85% images contain text) msairs wechat, tiktok and qq 20 ✓ ✓ ✗ acc, human annotation combined with gpt-4v review (shi et al., 2024) arxiv wf1 intentqa next-qa annotated via amazon mechanical turk (amt) (li et al., 2023a) iccv (xiao et al., 2021) -✓ ✓ ✗ acc with contrastive samples (same action, different intents) slurp home assistant 18×46 ✗ ✓ ✓ acc, contains 72k audio recordings (58 hours); (bastianelli et al., 2020) emnlp slu-f1 supports both pipeline (asr+nlu) and end-to-end slu approaches fluent speech commands (fsc) contains 30,043 audio utterances (19 hours); (lugosch et al., 2019) interspeech crowdsourcing 31 ✗ ✓ ✓ acc designed for end-to-end slu minds-14 crowdsourcing 14 ✗ ✓ ✓ acc for the e-banking domain across 14 languages; (gerz et al., 2021) emnlp includes spoken data and asr transcriptions mintrec acc, first tri-modal intent dataset; (zhang et al., 2022) mm tv series superstore 20 ✓ ✓ ✓ p, r, f1 includes automatic speaker annotation emotyda meld (poria et al., 2019) 11 ✓ ✓ ✓ acc, emotion-aware multi-modal dialogue act (da) (saha et al., 2020) acl iemocap (busso et al., 2008) p, r, f1 classification dataset; joint learning of das and emotions emoint-md movies acc, (singh et al., 2022) taslp (drama, action, fantasy, etc.) 15 ✓ ✓ ✓ f1 32k dialogues annotated with 15 empathetic intents mintrec2.0 tv series superstore, 30 ✓ ✓ ✓ acc, p, r tri-modal dataset with 15,040 samples (zhang et al., 2024b) iclr the big bang theory, f1, wp, wf1 (9,304 in-scope, 5,736 out-of-scope); and friends supports multi-turn, multi-party conversations mc-eiu weighted emotion and intent joint understanding dataset; (liu et al., 2024b) arxiv tv series 9 ✓ ✓ ✓ average f covers two languages (english and mandarin) ## textual-acoustic intent recognition acoustic signals in voice-based platforms (e.g., voice assistants or spoken dialogue systems) often serve as the primary modality for intent expression, with textual content derived through automatic speech recognition (asr) to complement paralinguistic information. below, we introduce the curated textual-acoustic ir benchmark datasets and their associated evaluation metrics ( §4.1), as summarized in ## related work intent recognition (ir) is one of the foundational tasks in natural language understanding (nlu), with early surveys dating back to @xcite , which reviewed automatic query intent detection. recently, @xcite focused on user intent in multimedia search, primarily involving visual intent in images and videos. on the other hand, @xcite summarized the methods and applications of textual intent detection. more recently, ir is typically surveyed in conjunction with slot filling @xcite @xcite @xcite , as they are highly relevant in dialogue systems @xcite . however, there has yet to be a comprehensive survey on ir covering multiple modalities, which motivates this first work. as shown in ## conclusion in this paper, we present the first comprehensive survey on the mir task, which begins by systematically summarizing existing works that cover various modality combinations. additionally, we compile and review currently available datasets and metrics while organizing three leaderboards to benchmark performance. furthermore, we highlight emerging trends in this research field, provid-ing insights into future directions. we hope this first survey with a website serves as a valuable resource to advance research in mir.