T ri SP rompt: A Hierarchical Soft Prompt Model for Multimodal Rumor Detection with Incomplete Modalities (2025)

Abstract: The widespread presence of incomplete modalities in multimodal data poses a significant challenge to achieving accurate rumor detection. Existing multimodal rumor detection methods primarily focus on learning joint modality representations from complete multimodal training data, rendering them ineffective in addressing the common occurrence of missing modalities in real-world scenarios. In this paper, we propose a hierarchical soft prompt model TriSPrompt, which integrates three types of prompts, i.e., modality-aware (MA) prompt, modality-missing (MM) prompt, and mutual-views (MV) prompt, to effectively detect rumors in incomplete multimodal data. The MA prompt captures both heterogeneous information from specific modalities and homogeneous features from available data, aiding in modality recovery. The MM prompt models missing states in incomplete data, enhancing the model’s adaptability to missing information. The MV prompt learns relationships between subjective (i.e., text and image) and objective (i.e., comments) perspectives, effectively detecting rumors. Extensive experiments on three real-world benchmarks demonstrate that TriSPrompt achieves an accuracy gain of over 13% compared to state-of-the-art methods. The codes and datasets are available at https: //anonymous.4open.science/r/code-3E88.

## introduction social media has emerged as a primary platform for the rapid dissemination of information, where multimodal communication-integrating various forms of media-has surpassed traditional unimodal formats to become the dominant mode @xcite . compared to unimodal content, multimodal information conveys richer and more nuanced messages @xcite , enhancing engagement but also introducing greater complexity, which poses significant challenges for effective rumor detection @xcite . earlier research on rumor detection predominantly centered on text-based (unimodal) analysis @xcite , leveraging features such as writing style and dissemination patterns to identify rumors. however, as multimodal content has become the dominant form of communication, the focus has shifted towards multimodal rumor detection. this approach integrates information from different modalities, such as text and images, to enhance detection accuracy. for instance, some studies employ feature fusion techniques to combine text and image data, aiming to improve performance. yet, simple fusion methods often fall short of capturing the intricate relationships between modalities. to overcome this limitation, @xcite introduced a cross-attention mechanism to more effectively integrate features from each modality. building on this, @xcite utilized graph neural networks to incorporate con-textual information from comments. @xcite proposed a fine-grained fusion approach that directly operates on the original modality information using graph-based methods, representing a significant advancement in multimodal rumor detection research. however, existing multimodal rumor detection methods often fail to address two critical practical challenges. firstly, incomplete modalities (ch1). in realworld scenarios, rumors spread rapidly, often outpacing the ability to collect complete multimodal data. for instance, during the covid-19 pandemic in 2020, a deluge of false information and rumors circulated rapidly, leading to inevitable modality loss during data collection due to factors such as network instability, device limitations, or other uncontrollable circumstances. despite this reality, most existing methods assume the availability of complete modalities. while clkd-imrd @xcite partially addresses missing modalities using a distillation model, its reliance on complete training data significantly limits its generalizability. when faced with more common scenarios where both training and testing data are incomplete, as illustrated in figure 1 , the performance of such methods degrades considerably. moreover, existing multimodal large language models (mllms), excelling in understanding multimodal content, struggle with rumor detection due to their inherent assumption that input information is "reality" and lack a mechanism for verifying content accuracy. secondly, the nature of rumors (ch2). in a post comprising text, images, and comments, each modality plays a distinct role. text and images typically convey the subjective perspective of the publisher, while comments often reflect the objective viewpoints of reviewers or readers. existing methods, however, tend to process and fuse all three modalities indiscriminately, overlooking the nuanced relationships between subjective and objective perspectives. this lack of differentiation not only prevents the model from effectively distinguishing between the two types of information but also introduces noise through direct fusion, which hinders judgment and degrades performance. to address these challenges, we propose a novel hierarchical soft prompt model, trisprompt, consisting of three types of soft prompts: modalityaware (ma) prompt, modality-missing (mm) prompt, and mutual-views (mv) prompt, that effectively detects rumors in incomplete multimodal data. specifically, for addressing ch1, in cases of missing modalities, the ma prompt learns both heterogeneous information from specific modalities and homogeneous features from available modality data, enabling the recovery of missing modalities. moreover, the mm prompt integrates the missing states in incomplete multimodal data, enabling the model to better understand and adapt to the nature of modality missing information. for addressing ch2, the mv prompt reveals the potential relationships between the subjective perspectives (i.e., text and image) provided by the publisher and the objective perspectives (i.e., comment) contributed by other reviewers in the post, to effectively detect rumors. the main contributions of this paper can be summarized as follows: to the best of our knowledge, this is the first approach to tackle the general modalitymissing problem with incomplete data in both training and testing phases in the field of multimodal rumor detection.