Face the Facts! Evaluating RAG -based Pipelines for Professional Fact-Checking (2025)

Abstract: Natural Language Processing and Generation systems have recently shown the potential to complement and streamline the costly and time-consuming job of professional fact-checkers. In this work, we lift several constraints of current state-of-the-art pipelines for automated fact-checking based on the Retrieval-Augmented Generation (RAG) paradigm. Our goal is to benchmark, following professional fact-checking practices, RAG-based methods for the generation of verdicts - i.e., short texts discussing the veracity of a claim - evaluating them on stylistically complex claims and heterogeneous, yet reliable, knowledge bases. Our findings show a complex landscape, where, for example, LLM-based retrievers outperform other retrieval techniques, though they still struggle with heterogeneous knowledge bases; larger models excel in verdict faithfulness, while smaller models provide better context adherence, with human evaluations favouring zero-shot and one-shot approaches for informativeness, and fine-tuned models for emotional alignment.

## introduction despite the efforts to validate the accuracy of online content, professional fact-checkers are increasingly struggling to keep up with the rapid spread of misinformation @xcite @xcite . therefore, natural language processing (nlp) has been proposed as a viable solution to partially automate the costly process of verifying misleading claims online @xcite . within this context, the task of verdict production, i.e., explaining why a claim is true or false, stands as one of the most challenging @xcite . framing verdict production as a summarization task over fact-checking articles is a suitable solution due to the possibility of generating highly readable verdicts even for non-expert users @xcite @xcite . despite their promising results, summarization-based approaches suffer from two main limitations: (i) they rely on the assumption that a fact-checking article always exists for a given claim; and (ii), they further assume that claims are already paired with a fact-checking article, which is typical in fact-checking websites but not on social media platforms, where most of the misinformation spreads @xcite . grounding textual generation on retrieved evidence, an approach named retrieval-augmented generation (rag), has been shown to be effective for knowledge-intensive tasks like fact verification @xcite @xcite ; it also allows addressing the limitations of previous summarization approaches, e.g., the assumption that the claims are already paired with gold fact-checking articles. moreover, ragbased approaches have proven useful in reducing potential factual inconsistencies, often referred to as "hallucinations" @xcite , during text generation @xcite , making them attractive for fact-checking tasks. thus, researchers have increasingly adopted rag to enhance the accuracy of the generated verdicts @xcite . current studies depend on fact-checking websites, resulting in verdicts characterized by formal and dry language. this style contrasts sharply with the language used on social media platforms (smps), which is typically more complex and includes noise such as personal commentary, or emotional content that surrounds the core fact. such mismatch might pose serious issues when countering misinformation online @xcite . we challenge these common assumptions on verdict generation, by testing rag-based pipelines across scenarios that progressively approximate professional fact-checking practices. to this end, we present a thorough evaluation of verdict production across several dimensions of a rag pipeline, testing key configurations for each of them (see figure 1 ): we show that llm-based retrievers consistently outperform other methods, though they face challenges with silver knowledge bases. dense retrievers manage stylistic variations of the claim effectively but fall short compared to llms, whereas sparse retrievers exhibit high sensitivity to noise present in emotional and smp claims. hybrid approaches and query pre-processing improve per-formance. turning to generation, larger models excel in faithfulness and alignment with gold verdicts, while smaller ones are more consistent in context adherence. fine-tuning boosts similarity of the generated verdict to the gold but reduces contextual accuracy, with human evaluations favouring verdicts generated under zero/one-shot strategies for informativeness and fine-tuning for emotional alignment. foot_0 ## related work early approaches to verdict generation leveraged either attention modules to highlight salient tokens from the evidence text @xcite @xcite @xcite , or horn rules to reason upon structured knowledge bases @xcite . however, both lack readability, being hard to interpret by common users @xcite . to overcome this issue, researchers started casting verdict production as a summarization task over fact-checking articles, either through extractive @xcite , abstractive @xcite , or hybrid @xcite summarization pipelines. more recently, @xcite introduced a reinforcement learning-based framework for generating counter-misinformation responses to social media content. ad-hoc data collection strategies for verdict generation rely either on synthetic data generation, like e-fever @xcite , or on journalistic sources, such as liarplus @xcite , pubhealth @xcite , ru22fact @xcite , liar++, and full-fact @xcite . @xcite proposed euroverdict, a multilingual, manually curated dataset for verdict generation. for more realistic, smp-style claims, @xcite extended fullfact with vermouth, incorporating emotional claims and verdicts grounded in trustworthy fact-checking articles. while the latest approaches provide readable verdicts, they may lack faithfulness due to language models generating factual inaccuracies. additionally, summarization approaches assume that trustworthy evidence is always available for the claim under inspection. therefore, rag-based approaches have been employed to guide the generation of reliable verdicts upon trustworthy evidence previously retrieved from a knowledge base (kb). to this end, @xcite proposed justilm, a few-shot rag-based approach for the generation of verdicts for real-world claims, by leveraging both fact-checking articles and auxiliary evidence during model training. @xcite developed an end-to-end rag-based system to perform verdict prediction and production in a multimodal setting jointly. nevertheless, both studies concentrate on journalistic data and writing styles, without considering the communication style employed on smp. building on the work of zeng and gao (2024), we present an extensive evaluation of rag pipelines for verdict generation, testing various combinations of retrieval and generation strategies. progressing toward increasingly realistic scenarios, with respect to the workflow adopted by professional fact-checkers, we address the challenge of handling claims and sources that vary in style and complexity, aiming to closely mimic the kinds of claims that everyday users might encounter on social media platforms. specifically, we assess the impact of claim writing style on retriever performance, highlighting where differences arise as the style shifts toward that used by smp users. finally, we explore the extreme scenario where no factchecking article exists, relying solely on reliable supporting evidence. ## experimental design in this section, we provide details on the experimental design: from the datasets used, through the retrieval methods adopted, to the configurations of the llms employed for verdict generation. ## retrieval experiments retrieval from gold kb we tested the retrievers on fullfact and vermouth test sets with an increasing number of retrieved documents (@xmath0, . . . , 10). for each claim used as a query, we considered as relevant documents the fact-checking article, or its chunks, linked to the claim. for space reasons, results on the emotional datasets will be presented in aggregated form, referred to as the 'emotional' set. experiments were carried out integrating into the llamaindex @xcite ) framework either rank-bm25 (brown, 2020) or huggingface's models @xcite ; retrieval performance was assessed with ranx @xcite . in chunk retrieval, the llm-reranker excelled, achieving a hit_rate@10 that always included a gold chunk and reaching an average map@10 of 70%. sparse retrievers showed low map scores (∼40% for neutral claims), particularly for smp and emotional claims (<30%), while dense and hybrid approaches followed trends similar to article retrieval. thus, the low map scores indicate that sparse retrievers must retrieve more chunks from the knowledge base to select the relevant content. however, this comes at the cost of also retrieving more non-relevant content, which could potentially compromise the subsequent generation phase. overall, the llm-retriever consistently outperforms other approaches. notably, it remains stable even when exposed to different input claim styles, 9 more details in appendix d.2. with minimal degradation when noise is introduced. a paired t-test confirms that the performance gains of llm-retrieval are statistically significant compared to other methods, with the exception of the hybrid retriever over hit_rate@1. still, it maintains a slightly higher mean score (0.934 vs. 0.925) and lower variance (0.061 vs. 0.069). dense retrievers perform worse but show robustness to stylistic variations. in contrast, sparse retrievers are significantly affected by data noise, resulting in performance drops across all three datasets: we find that the fact extraction module we included yields consistent performance improvements, and particularly helps when using sparse and dense retrievers. retrieval from silver kb we tested the optimal retriever methodologies from the previous experiments, specifically the llm-retriever and the hybrid retriever. as outlined in section 3.2, the silver kb consists of reliable sources that have been extracted from the initial fact-checking articles. the evidence consisted of 9983 chunks, each corresponding to a fact-checking article considered a gold standard during evaluation. the results ( prepending a fact extraction module generally improves retrieval results: even robust retrievers can benefit from preprocessing when dealing with heterogeneous kb (that do not contain gold factchecking articles) and complex (e.g., emotional) claims. ## generation experiments for verdict generation, the claim and its corresponding retrieved evidence were combined into a prompt fed to the five different llms (section 3.3). for evidence retrieval, we employed the bestperforming retriever (i.e., llm-retriever). we tested the five llms under three setups: zero-shot, one-shot, and fine-tuned. 10 for fine-tuning, we employed the best-performing model in both the zeroshot and one-shot configurations, llama-2-13b. when using the gold_kb art , we included the top-1 article (i.e., the most relevant) in the prompt, a choice justified by the llm retriever's remarkable hit_rate@1 results. conversely, with retrieval from gold_kb chunks , we fed the model with 10 retrieved chunks, based on its map@10 performance (see to sum up, generations with neutral claims outperformed those with emotional data in both zeroshot and one-shot experiments but produced more accurate results when paired with emotional data. in terms of faithfulness to context and similarity to the gold, larger models generally performed better. however, smaller models exhibited superior contextual adherence. ## conclusions the fight against misinformation has become increasingly demanding, amplified by the vast reach of popular social media. with this work, we provided a comprehensive overview of how ragbased approaches can effectively be employed for the automatic generation of verdicts when addressing realistic fact-checking scenarios. we thus present an extensive experimental analysis that explores different approaches at each stage of these pipelines, accounting for diverse computational constraints relevant to researchers, practitioners, and fact-checking organizations. our results show that llm-based retrievers consistently demonstrate superior adaptability and performance in retrieval tasks, outperforming other methods. however, they face challenges when dealing with heterogeneous knowledge bases, where performance declines. hybrid retrieval approaches offer a cost-effective alternative, while dense retrievers remain robust to stylistic variations. notably, incorporating fact extraction modules enhances retrieval effectiveness across setups; thus, query preprocessing is particularly beneficial for addressing more complex claims. although claim 17 further details are provided in appendix f. complexity reduces generation accuracy, larger models achieve greater faithfulness to context and alignment with gold-standard verdicts. human evaluations also favour zero-shot and one-shot approaches, particularly for their informativeness.