M ulti R eflect: Multimodal Self-Reflective RAG -based Automated Fact-Checking (2025)

Abstract: In this work, we introduce MultiReflect, a novel multimodal self-reflective Retrieval Augmented Generation (RAG)-based automated fact-checking pipeline. MultiReflect is designed to address the challenges of rapidly outdated information, limitations in human query capabilities, and expert knowledge barriers in fact-checking. Our proposed pipeline leverages the latest advancements in Large Language Models (LLMs) and Retrieval Augmented Generation (RAG) to enhance fact verification across text and images. Specifically, by integrating multimodal data processing with RAG-based evidence reflection, our system improves the accuracy of fact-checking by utilizing internet-sourced verification. We evaluate our results on the VERITE benchmarks and using several multimodal LLMs, outperforming baselines in binary classification.

## introduction information plurality, particularly on the internet, presents both opportunities and challenges in identifying accurate and up-to-date information. given the increasing reliance on online platforms for news consumption, learning, and interaction @xcite , developing effective mechanisms to distinguish between truthful and false information has become more critical. however, the increase of coordinated misinformation movements by spam bots, and other forms of informational chaos have significantly complicated this process. therefore, more advanced and systematic approaches are required to evaluate and verify (fact-check) the credibility of information sources. with the emergence of large language models (llms), which can understand and learn from billions of texts, automated fact-checking has grown in popularity as an alternative to traditional manual methods @xcite . while llms are stateof-the-art tools for various language understanding and reasoning tasks, they still face several limitations, such as hallucinations, overconfidence, and bias @xcite . to address these issues, several studies have employed retrieval augmented generation (rag) techniques, which allow the model to check based on externally verified information @xcite . language, however, is only part of the challenge when it comes to fact-checking information on the internet. online information is presented in various forms, including text, images, video, and sound. as a result, fact-checking also requires the retrieval and reasoning of information across multiple modalities @xcite . more recently, several state-of-the-art models, such as gpt-4v (openai, 2023), gpt-4o (openai, 2024), deepseek-vl2 @xcite and claude 3 @xcite , have made reason across multimodal data possible. these rapid advancements highlight the need for multimodal fact-checking, which has grown with the increased prevalence of complex information that spans various data types: text, image, video, and audio. systems like cosmos @xcite , twitter-comms @xcite , exmulf @xcite , chartbert @xcite , red-dot and @xcite have made significant progress in tackling the challenges posed by multimodal data. however, despite their successes, these systems have not fully taken advantage of rag, a crucial component for dynamic and context-aware evidence retrieval in the multimodal setting. to address this gap, we introduce multireflect, illustrated in figure 1 , which integrates multimodal fact-checking (image + text) with a self-reflective rag framework. our system is designed to dynamically retrieve, evaluate, and rank supporting no verification phase 1 phase 2 phase 2 phase 6 top text evidence ivanka, don jr. & eric in russia-address: 8 kiyevskaya ul., moscow i was recently sent these photos of ivanka, don jr. & eric trump in moscow, russia.they're definitely from a few years back, we guess sometime in the 2006-2009 range.and it's definitely in russia.it looks like they're doing business in russia, looking at property. take a look for yourself below. on the left is elena baronoff -who worked for trump in florida & is apparently a former russian spy-gil dezer who ran trump properties in the middle & we have no idea who it is on the right. also just fyi: in the cover photo they are standing right next to this address: 8 kiyevskaya ul., moscow. therefore, the caption claiming that these individuals are members of the united states congress is out of context. ## data for our experiments, we utilize the verite dataset, a multimodal fack-checking benchmark dataset @xcite . the dataset contains 892 different image-text pairs with the labels "true" (302), "miscaptioned" (302), and "out-of-context" (288). the dataset incorporates a wide range of real-world data while specifically excluding "asymmetric multimodal misinformation" (asymmetric-mm), which refers to scenarios where one form of modality significantly amplifies misinformation while others have minimal impact. also, the data implements "modality balancing," ensuring that each image and caption are represented twice in the dataset: once within truthful contexts and once within misleading pairs. ## proposed method: multireflect in this section, we introduce our proposed sixphase pipeline: (1) consistency checking, (2) evidence checking, (3) retrieval, (4) evidence filtering, (5) evidence ranking, and (6) verification. ## experimental results in the following section, we introduce our (1) baselines and experimental results (2). ## ablation study we conduct an ablation study of our bestperforming model, gpt-4v, on the benchmark to evaluate the role of evidence within the multireflect pipeline. this analysis focuses on two key questions: (1) is any evidence necessary for effective verification? (2) does the ranking of evidence contribute meaningfully to performance? to address the first question, we evaluate the system's performance when no evidence is provided during the verification stage. for the second, we provide all available evidence without applying any ranking. the results demonstrate that rag-enhanced retrieval and ranking both play a critical role in strengthening multimodal reasoning. ## related works in this section, we introduce several related methods and papers to our work. we additionally highlight the main feature differences between the methods in automated fact-checking. fact-checking methods have significantly evolved with advancements in artificial intelligence, particularly through the development of llms and automated systems. early systems such as those introduced by @xcite and @xcite relied on static knowledge bases for evidence retrieval for fact-checking and correction. however, these systems lacked the ability to update their knowledge bases dynamically, which is critical in the fastpaced information era. recent efforts have seen the integration of retrieval augmented generation (rag) techniques to enhance the reliability and accuracy of factchecking systems. for instance, models such as murag @xcite and self-rag @xcite have utilized not only static knowledge bases but also the internet to retrieve current and relevant information. these models enhance the fact-checking process by employing rag for dynamic evidence retrieval, allowing for a more accurate verification of facts by evaluating various aspects of information quality. this approach significantly surpasses earlier models that relied only on static databases or lacked evidence ranking mechanisms @xcite . multimodal fact-checking. the need for multimodal fact-checking has grown with the increased prevalence of complex information that spans various data types: text, image, video, and audio. systems like cosmos @xcite , twitter-comms @xcite , exmulf @xcite , chartbert @xcite , red-dot and @xcite have made significant progress in tackling the challenges posed by multimodal data. however, prior works rely heavily on training, limiting usability in lowresource settings, and often focus only on intramodal relationships, overlooking nuanced crossmodal relationships. to the best of our knowledge, our multireflect approach is the first to integrate evidence retrieval, multimodality, verification, evidence ranking, and rag into a single fact-checking pipeline. ## conclusions we introduce multireflect, a novel multi-modal rag-based fact-checking pipeline. the novelty of the pipeline lies in its new evidence ranking and reflection scheme over multimodal posts. we validate the efficiency of the pipeline using a specialized multimodal fact-checking benchmark dataset verite. our results show that multirelflect underperforms in the multiclass setting but outperforms other baselines in the binary class scenario. future works could improve this pipeline by focusing on how to better identify in which modality the error exists. additionally, incorporating modalityspecific retrieval strategies could help disentangle complex cross-modal contradictions.