Dual-oriented Disentangled Network with Counterfactual Intervention for Multimodal Intent Detection (2024)

Abstract: Multimodal intent detection is designed to leverage diverse modalities for a comprehensive understanding of user intentions in real-world scenarios, thus playing a critical role in modern task-oriented dialogue systems. Existing methods have made great progress in modal alignment and fusion, however, two vital limitations are neglected: (I) close entanglement of multimodal semantics with modal structures; (II) insufficient learning of the causal effects of semantic and modality-specific information on the final predictions under the end-to-end training fashion. To alleviate the above limitations, we introduce the Dual-oriented Disentangled Network with Counterfactual Intervention (DuoDN). DuoDN addresses key limitations in current systems by effectively disentangling and utilizing modality-specific and multimodal semantic information. The model consists of a Dual-oriented Disentangled Encoder that decouples semantics-oriented and modality-oriented representations, alongside a Counterfactual Intervention Module that applies causal inference to understand causal effects by injecting confounders. Experiments on three benchmark datasets demonstrate DuoDN’s superiority over existing methods, with extensive analysis validating its advantages.

## introduction with the rise of intelligent technology, taskoriented dialogue systems are rapidly advancing and proving their potential in practical applications like health consulting, financial services, and home automation. a crucial component of these systems is natural language understanding (nlu), which captures the semantics of user queries to enhance conversational interactions @xcite . after converting spoken words to text via speech recognition, the system needs to understand the user's intent to respond accurately. recent years have seen significant advancements in text-based intent detection @xcite @xcite . however, as modern dialogue * corresponding author. recently, research has focused on bi-modal intent detection using textual guidance @xcite @xcite , which is a step forward but still falls short for real-world natural language understanding. to address this, @xcite introduced mintrec, the first tri-modal intent detection benchmark, incorporating text, video, and audio. building on this, @xcite proposed the shallow-to-deep interaction framework with data augmentation (sdif-da), which aligns multimodal features for better fusion and uses chat-gpt to generate additional utterances, tackling data scarcity. @xcite introduced a tokenlevel contrastive learning method with modalityaware prompting (tcl-map), creating an optimal multimodal semantic space to refine the text modality. however, these approaches either overlook the rich, complex semantic information from different modalities or neglect the causal effects on final predictions from semantic and modality-specific information. addressing the first issue, simply fusing multimodal information is challenging due to the persistent gaps between heterogeneous modalities, which are often complex and interdependent. these interdependencies are frequently overlooked, resulting in inadequate task modeling. regarding the second issue, optimizing the model through end-to-end learning often leads to suboptimal outcomes, weakening the focus on semantic and modality-specific information. in summary, existing methods face two major limitations in complex scenarios: (i) multimodal semantic information is deeply intertwined with modality-specific structures; (ii) the causal effects of semantic and modality-specific information on final predictions are not properly learned in end-to-end training. visual text audio visual-oriented text-oriented audio-oriented text-visual semantics-oriented text-audio semantics-oriented model direct fusion to address the first limitation, we employ disentangled representation learning. drawing inspiration from advances in domain adaptation @xcite , we aim to learn two distinct types of representations for each modality. the process begins with encoders that encode the tri-modal inputs. then, a dual-oriented disentangled encoder decouples the multimodal representations into semantics-oriented and modalityoriented factors. on one hand, the semanticsoriented representations aim to reduce modality gaps. despite originating from different sources, multimodal information often shares common motives. the semantic-oriented encoder captures these commonalities as aligned projections in a shared subspace. to enhance semantic consistency, we apply semantic-level contrastive learning on the disentangled semantics-oriented features, bringing similar semantic features from different modalities closer together. on the other hand, multimodal signals embody both commonalities and distinctive characteristics. each modality's unique traits can be useful for predicting the speakers' intentions. therefore, the modality-oriented representations complement the shared semantic features and promote comprehensive fusion learning. to address the second limitation, we use causal inference to analyze the pure causal effects of semantic-and modality-specific information on final predictions. causal inference, widely applied in video understanding @xcite and audio-visual video question answering @xcite , reveals implicit causal relationships among variables and enhances model generalization. end-to-end training often lacks directed supervision towards significant fea-tures, leading to suboptimal outputs. inspired by causal inference, we incorporate a counterfactual intervention module as additional supervision. this module highlights the semantics-and modalityoriented representations by injecting confounders, enabling the model to understand causal effects. we introduce the indirect effect by maximizing the difference between the original output and a counterfactual output, altered only by changes in the semantic-or modality-oriented representations. this helps the model perceive their functions. both the original and intervened outputs are used to optimize supervision, resulting in a comprehensive understanding of causal effects and improving the model's performance and robustness. our duodn has been shown to outperform previous state-of-the-art methods on three benchmark datasets: mintrec @xcite , mintrec 2.0 @xcite , and meld-da @xcite , demonstrating its effectiveness. specifically, it exhibits an excellent understanding of semantics, achieving superior performance on mintrec 2.0, both with and without out-of-scope samples. additionally, ablation analysis confirms that our proposed methods of disentangled representation learning and counterfactual intervention are effective, contributing to advancements in related research. to summarize, our major contributions are as follows: (i) we propose a dual-oriented disentangled network with counterfactual intervention for multimodal intent detection, effectively disentangling and utilizing modality-and semantic-specific information. (ii) to the best of our knowledge, we are the first to introduce causal inference to determine the causal effects of semantic-and modality-specific information on the final predictions in multimodal intent detection. (iii) extensive experiments on three benchmarks show that duodn significantly outperforms previous methods. further analysis verifies the advantages of our model. ## conclusion in this paper, we propose a novel dual-oriented disentangled network with counterfactual intervention (duodn) for multimodal intent detection. through the dual-oriented disentanglement, duodn creates multimodal semantic spaces and optimizes the semantics-oriented representations with semantic-level contrastive learning. additionally, we introduce a counterfactual intervention module as an additional supervision to highlight the semantics-oriented and modality-oriented rep-resentation, leading to a better modeling of the task. extensive experiments on three benchmark datasets demonstrate duodn's effectiveness.