RAGAR , Your Falsehood Radar: RAG -Augmented Reasoning for Political Fact-Checking using Multimodal Large Language Models (2024)

Abstract: The escalating challenge of misinformation, particularly in political discourse, requires advanced fact-checking solutions; this is even clearer in the more complex scenario of multimodal claims. We tackle this issue using a multimodal large language model in conjunction with retrieval-augmented generation (RAG), and introduce two novel reasoning techniques: Chain of RAG (CoRAG) and Tree of RAG (ToRAG). They fact-check multimodal claims by extracting both textual and image content, retrieving external information, and reasoning subsequent questions to be answered based on prior evidence. We achieve a weighted F1-score of 0.85, surpassing a baseline reasoning technique by 0.14 points. Human evaluation confirms that the vast majority of our generated fact-check explanations contain all information from gold standard data.

## introduction in the age of digital information, rapid dissemination of news, both genuine and fabricated, has become a defining feature of public discourse. the phenomenon of fake news -which more precisely denotes misinformation, disinformation, or a combination of both @xcite -is particularly prevalent on social media: false information spreads six times faster than the truth on platforms like twitter @xcite . this trend poses a critical challenge to the democratic process since it makes voters increasingly prone to making decisions based on incorrect information. the matter is further aggravated by visual information, which provides yet another widespread and consequential source of fake news. for instance, fake news stories that include images spread further than those containing only text @xcite . a potential solution to these issues is provided by automated fact-checking systems. they have bene-figure 1 : an overview of the fact-checking pipeline contrasting the baseline sub-question generation approach from the chain of rag and tree of rag approach followed by veracity prediction and explanation. addressing this gap, we introduce ragar -rag-augmented reasoning techniques, which we apply to multimodal fact-checking in the political domain (see figure 1 for a high-level overview). we rely on a multimodal llm to verbalize the textual and visual elements of a claim, and use rag responses to motivate successive steps in determining veracity. the system is underpinned by elaborate reasoning strategies instantiated in two distinct approaches: chain of rag (corag) and tree of rag @xcite . we evaluate them using a multimodal fact-checking dataset as well as human annotation of generated explanations. our contributions are as follows. ## multimodal fact-checking pipeline our fact-checking pipeline comprises four parts: (i) multimodal claim generation, which analyzes both the textual claim and associated image to formulate a new claim incorporating both; (ii) multimodal evidence retrieval, which extracts evidence from the web for a question posed by the llm; (iii) llm-based and rag-augmented reasoning for fact-checking, our reasoning approach to factcheck a claim; and (iv) veracity prediction and explanation. the pipeline is shown in figure 2 . ## evaluation and results we now present two evaluations employed across the set of 300 multimodal claims. in section 5.1, we analyze system performance based on the correctness of veracity predictions. in section 5.2, we zoom into explanation generation by conducting a human annotation study to compare the generated and gold explanations. ## conclusion this paper introduces and tests two new methods for political fact-checking using large language models (llms): chain of rag (corag) and tree of rag (torag). these methods tackle misinformation in political discussions, focusing on multimodal claims, and show notable improvements over traditional fact-checking approaches that use sub-question generation with llms. corag uses a step-by-step questioning strategy for thorough claim examination, while torag extends upon this by following a branching strategy with evidence elimination thereby enhancing veracity prediction. we evaluate these methods in two ways. in terms of correctness of generated veracity label, we see an increase of 0.06-0.14 f1 points when using the ragar framework with standard, cot vp , and cove veracity prediction prompts compared to the baseline subq+cot vp . for explanation generation, the quality of ragar-generated explanations was consistently rated higher than the baseline method. our study shows that rag-augmented reasoning (ragar) techniques are effective in multimodal political fact-checking, improving both the accuracy of veracity predictions and the quality of detailed fact-check explanations.