Multimodal Misinformation Detection by Learning from Synthetic Data with Multimodal LLM s (2024)

Abstract: Detecting multimodal misinformation, especially in the form of image-text pairs, is crucial. Obtaining large-scale, high-quality real-world fact-checking datasets for training detectors is costly, leading researchers to use synthetic datasets generated by AI technologies. However, the generalizability of detectors trained on synthetic data to real-world scenarios remains unclear due to the distribution gap. To address this, we propose learning from synthetic data for detecting real-world multimodal misinformation through two model-agnostic data selection methods that match synthetic and real-world data distributions. Experiments show that our method enhances the performance of a small MLLM (13B) on real-world fact-checking datasets, enabling it to even surpass GPT-4V.

## introduction multimodal misinformation, which appears more credible and spreads faster than text-only misinformation, has become a significant concern. about one-third of verification claims include multimodal data, with the primary modality being image-text pairs @xcite . this underscores the importance of multimodal misinformation detection (mmd), which involves determining the veracity of such image-text pairs. these pairs can be created by pairing a textual claim with an out-of-context image, or by manipulating the content of the image, the text, or both. an illustrative example is depicted in figure 1 , where an image shows elon musk holding a flag bearing the statement "trump won, democrats cheated," a false claim debunked by fact-checkers 1 . the success of training a mmd model highly depends on the availability of large-scale, highquality datasets, especially in the era of multimodal 1 https://www.snopes.com/fact-check/ elon-musk-trump-won-flag/ large language models (mllms) that require instruction tuning for downstream tasks. however, acquiring such real-world fact-checking datasets presents significant challenges as labor-intensive annotation has to be done by fact-checkers. given this data scarcity, a cost-effective solution is to utilize the advancements in generative ai technology to generate synthetic multimodal misinformation based on a vast repository of readily accessible real news @xcite @xcite . for instance, the synthetic dataset newsclipings @xcite ) contains more than 1 million instances, whereas the popular real-world dataset mediaeval @xcite , annotated by fact-checkers, only has around 10,000 instances with just 514 unique images. studies have proposed different multimodal misinformation detectors trained on synthetic datasets and have achieved reasonable performance @xcite @xcite . however, it is unclear how well detectors trained on large synthetic datasets can generalize to realworld fact-checking data, as there remain salient gaps between them (visualization example is shown in appendix c). since synthetic and realworld data are out-of-distribution (ood) relative to each other, the distributional gap between the two types of datasets can lead to significant discrep-ancies in detection performance. detectors directly trained on synthetic datasets may yield limited results or even fail in real-world applications. to address this gap, we propose to select valuable synthetic data instances based on similarity metrics that are oblivious to the downstream detection tasks, and utilize it to improve the generalization capability of detectors to real-world fact-checking data. we first compile a large-scale synthetic dataset by amalgamating three distinct synthetic datasets, ensuring coverage of diverse multimodal misinformation categories. given this synthetic dataset as training set and a very small number of unlabeled real-world instances as the validation set, our goal is to effectively select a small subset of relevant and valuable synthetic data to enhance the models' capacity for detecting realworld multimodal misinformation. we approach this using two model-agnostic data selection methods: 1) semantic similarity-based selection, which prioritizes synthetic instances with the highest similarity scores to the validation set; and 2) distributional similarity-based selection using gradient information derived from the optimal transport (ot) problem @xcite to increase the density of data around the desirable region by choosing data points in the synthetic dataset that are close to the target real-world distribution. these methods evaluate the relevance of synthetic data points without requiring model retraining, ensuring efficiency and scalability. moreover, the small selected subset of synthetic training instances makes the fine-tuning of mllms feasible in computing resource-constrained scenarios. our main contributions are four-fold foot_0 : ## problem formulation we consider a large dataset d s consisting of synthetic instances that cover common categories of multimodal misinformation, and a target dataset d t which contains high-quality, annotated, and diverse real-world fact-checking data, where from d s to fine-tune a pre-trained multimodal model m, such that it generalizes well to the realworld test dataset d t . to reflect real-world constraints related to practical computational resource limitations, we also restrict the size of the selected subset being small, i.e. with a few randomly sampled instances from d t is given to facilitate data selection. we assume the unlabeled validation set is sampled i.i.d from the target distribution as they originate from the same source. it is expected that mllms trained on the selected synthetic instances based on d u could generalize well on unseen test instances. ## methodology data selection aims to find a set of training instances that improve model performance @xcite . some approaches use data valuation and attribution methods such as influence function @xcite @xcite , leave-one-out @xcite , and shapley value @xcite . these approaches perturb one data point from the training set to get a new subset, with or without permutations, to trace the change of the model validation performance, and quantify the effect of that point. however, such process requires re-training the model on each subset that excludes each perturbed data point to be evaluated, which will cause expensive computational costs @xcite . additionally, it requires labeled validation sets to trace performance changes, which differs from our setting. recently, distributional distance metrics such as wasserstein distance and kl divergence, have emerged as effective proxies for conducting data selections by providing an upper bound on performance change @xcite @xcite @xcite . as model-agnostic approaches, they are more desirable in the context of llms. there have been some successful applications for either pre-training or pre-fine-tuning @xcite @xcite . for example, dsir @xcite proposes importance resampling on the hashed n-gram features of text data, and measure the distributional similarity based on kl-reduction. gio @xcite uses iterative gradient methods to prune training samples by minimizing kl-divergence. got-d @xcite enjoys the geometric property of optimal transport to select data that nudges the pre-training distribution closer to the target distribution. however, these methods are not directly applicable to multimodal data due to the difficulties in quantifying the probability density of multimodal misinformation data, typically necessitating tens of thousands of data points. additionally, applying these textual data selection methods in the context of mmd task is incorrect. the factuality of a piece of information is determined by the combined information from text and image in mmd. relying solely on a single modality fails to capture cases like out-of-context misuse, where the standalone textual claim might be accurate while the image-text pair together conveys false information. several studies in multimodal data selection primarily focus on sampling large corpora, ranging from millions to billions of samples, from vast volumes of noisy, web-curated datasets @xcite @xcite @xcite . these methods involve non-trivial data-dependent filtering strategies to select high-quality data for pre-training, such as clipscore @xcite that ranks each data point based on the cosine similarity scores between its clip image and text embeddings. different from these works, our goal is to select a small number of useful synthetic instances for finetuning, with the aim of aligning distributions be-tween synthetic and real-world datasets for mmd. another line of work focuses on selecting features or modalities to boost model performance @xcite @xcite @xcite , which is beyond the scope of this paper. we employ two model-agnostic data selection approaches, i.e., semantic similarity-based selection and distributional similarity based selection, which provide straightforward, efficient, and scalable estimates of synthetic data relevance, facilitating the tuning of mllms for detecting real-world multimodal misinformation. feature extraction. we adopt a mixed-modal encoder, consisting of a text encoder and an image encoder, to extract multimodal features for data selection. it is a simple extension of the off-theshelf clip model @xcite , which has demonstrated good performance in multimodal retrieval tasks @xcite . specifically, given an image-claim pair x i , we obtain the visual features and textual features via the image encoder and text encoder. this pair's multimodal features e i are then computed by averaging these two modalities, with the l 2 norm scaled to 1. ## conclusion and future work we propose leveraging synthetic data to address the mmd task in the scenarios of real-world data scarcity, in which relevant and valuable instances can be selected from a large-scale synthetic dataset for fine-tuning mllms. we advocate for utilizing two model-agnostic similarity-based data selection approaches for this task. results demonstrate that even using a very small number of selected synthetic training instances can significantly boost mllms' detection performance on real-world factchecking data, enabling a small mllm, llava-13b, to outperform gpt-4v. in future research, we plan to determine the optimal number of selected instances for different data selection methods for maximizing the effectiveness of useful training data. additionally, we plan to explore more design solutions to obtain better multimodal features for further improving misinformation detection performance.