Leveraging Contrastive Learning and Knowledge Distillation for Incomplete Modality Rumor Detection (2023)

Abstract: Rumors spread rapidly through online social microblogs at a relatively low cost, causing substantial economic losses and negative consequences in our daily lives. Existing rumor detection models often neglect the underlying semantic coherence between text and image components in multimodal posts, as well as the challenges posed by incomplete modalities in single modal posts, such as missing text or images. This paper presents CLKD-IMRD, a novel framework for Incomplete Modality Rumor Detection. CLKD-IMRD employs Contrastive Learning and Knowledge Distillation to capture the semantic consistency between text and image pairs, while also enhancing model generalization to incomplete modalities within individual posts. Extensive experimental results demonstrate that our CLKD-IMRD outperforms state-of-the-art methods on two English and two Chinese benchmark datasets for rumor detection in social media.

## introduction social media platforms like twitter and weibo allow people to contribute vast amounts of content to the internet. however, this content is often rife with rumors, which can lead to significant societal problems. for instance, in the first three months of 2020, nearly 6,000 people were hospitalized due to coronavirus misinformation 2 , while covid-19 vaccine misinformation and disinformation are estimated to cost 50 to 300 million each day 3 . humans are generally susceptible to false information or rumors and may inadvertently spread them @xcite . suffering from the low coverage and long delay of rumor detection manually, automatic rumor detection models are essential. previous text modality-based rumor detection models focused on exploring propagation @xcite , user information @xcite , and writing styles @xcite . furthermore, as reported in @xcite , microblogs with pictures have been found to have 11 times more access than those without pictures, which highlights the importance of multimodal content in rumor detection. specifically, some studies have investigated the fusion of different modalities, such as images and text, by directly concatenating their representations @xcite . nevertheless, directly concatenating two different modalities (i.e., textual and visual modalities) may not capture their deep semantic interactions. to address this issue, co-attention-driven rumor detection models aim to extract the alignment between the two modalities @xcite . however, current multimodal rumor detection models typically overlook the problem of incomplete modalities, such as the lack of images or text in a given post, making them less effective in handling such cases. to address the above issues, we propose a novel approach, contrastive learning and knowledge distillation for incomplete modality rumor detection (clkd-imrd). in fact, supervised contrast learning effectively pulls together representations of the same class while excluding representations from different classes, and knowledge distillation can effectively handle the incomplete modality cases when debunking rumors. more specifically, we first construct a teacher model that consists of multimodal feature extraction, multimodal feature fusion, and contrastive learning. then, we adopt knowledge distillation to construct a student model that can handle incomplete modalities (i.e., lack of images or texts). experimental results and visualizations demonstrate that our clkd-imrd outperforms state-of-the-art methods on both english and chinese datasets in rumor detection on social media. this paper makes three major contributions. ## conclusion in this paper, we propose a rumor detection framework that combines supervised contrastive learning and knowledge distillation. our framework leverages hierarchical co-attention to enhance the representation of textual (source text and comments) and visual modalities, enabling them to complement each other effectively. the utilization of contrastive learning has proven to be successful in debunking rumors. additionally, knowledge distillation has demonstrated its efficacy in handling incomplete modalities for rumor detection. moving forward, our future work aims to integrate graph structures, such as social graphs, into our proposed framework for further improvement. • one co-attention: here, only the first coattention is employed. the enhanced textual-visual representation is then concatenated with the comment representation. the enhanced textual and visual representations are concatenated with the comment representations. from