C laim C heck: Automatic Fact-Checking of Textual Claims using Web Evidence (2025)

Abstract: We introduce ClaimCheck, an efficient fact-checking system that verifies textual claims using smaller, open-source large language models. ClaimCheck integrates two fact-checking strategies, claim-matching and novel claim processing. Claim-matching uses related fact-checks from trusted organizations to fact-check a claim. Novel claim processing breaks down fact-checking into manageable subtasks—generating targeted questions, retrieving Web evidence, extracting answers, and synthesizing verdicts. Evaluation on the AVeriTeC benchmark demonstrates 62.6% verdict prediction accuracy, with claim-matching providing a 2.8% improvement. ClaimCheck approaches the performance of state-of-the-art systems while requiring significantly fewer computational resources, demonstrating the effectiveness of using small language models for fact-checking tasks. Furthermore, our code is publicly available to help make automated fact-checking more accessible.

## introduction the rapid proliferation of digital content has transformed how information is disseminated and consumed, but it has also amplified the spread of misinformation. in an era where public discourse is increasingly influenced by online narratives, combating the impact of false claims has become a critical societal challenge. the world economic forum's 2024 global risks report 1 underscores this urgency, identifying misinformation as a top threat to democratic stability, rivaling even climate crises and geopolitical conflicts. as the scale of misinformation grows, so does the necessity for robust, scalable solutions capable of fact-checking claims efficiently @xcite . automated fact-checking systems have emerged as a promising technological response to this challenge @xcite . these systems enhance the efficiency of traditional fact-checking processes by leveraging advancements in machine learning, natural language processing, knowledge bases, and databases @xcite . in this work, we focus on leveraging large language models (llms) for fact-checking due to their strong language understanding and reasoning capabilities, as well as their ability to process diverse sources of evidence. llms facilitate techniques such as retrieval-augmented generation (rag) and structured reasoning, which enhance fact-checking capabilities @xcite . however, building an effective llm-based fact-checking system entails overcoming numerous challenges, such as contextual ambiguity, temporal sensitivity of evidence, and incomplete or misleading information @xcite @xcite . one of the drawbacks of the current state-of-theart systems is that most use very large llms which can be monetarily prohibitive @xcite . smaller llms require much less computational resources and are more accessible to users at the cost of some loss in task-specific performance and reasoning capabilities @xcite . we focus this work on utilizing smaller llms for the automatic fact-checking pipeline, because this can present more accessible system for the common public. although the limitations of smaller llms impede systems using smaller llms from achieving the performance of systems using larger llms, we aim to demonstrate that a open-source, less computationally intensive system can be a useful fact-checking system. this paper presents claimcheck, an automatic fact-checking system for textual claims. claim-check first uses a matching process to determine if a claim has already been fact-checked. for claims not previously fact-checked, i.e., the given claim is novel, the system decomposes the fact-checking task into targeted subtasks: generating specific questions required to fact-check the claim, retrieving real-time evidence via web search, processing the evidence to extract answers for each question, and synthesizing these answers to predict a verdict. this design simplifies the subtasks for our system, an important consideration for using smaller llms as presenting them with focused tasks helps to prevent them from being overwhelmed by complex instructions and data. furthermore, our choice of using web search for evidence retrieval avoids requiring the users to store massive knowledge sources locally for evidence retrieval. averitec @xcite is a popular real-world claim benchmark dataset consisting of only textual claims. our experimental evaluation on averitec indicates that the proposed approach achieves a verdict prediction accuracy of 62.6%. when compared to the current highest accuracy on averitec-75.2% @xcite -claimcheck demonstrates that small, open-source llms can be leveraged in a more computationally efficient and scalable manner while approaching the performance of state-of-the-art systems. to summarize, our work makes the following key contributions to automated fact-checking: ## background to fact-check textual claims, the workflow of most llm-based systems @xcite @xcite @xcite @xcite contains four steps, as follows. 1) question generation: the system generates questions to identify the core aspects of the claim. this step ensures that the fact-checking process is focused and systematic. 2) evidence retrieval: the system retrieves supporting or refuting evidence from trusted knowledge sources, such as wikipedia. this step is critical for grounding the fact-checking process with verifiable information. 3) question answering: the system processes the retrieved evidence to generate precise answers to the questions generated in step 1. this step involves analyzing the evidence and extracting relevant information to address the claim. 4) verdict prediction: the system synthesizes the evidence to predict a verdict (e.g., true or false). the final step determines the overall thurthfulness of the claim. fact-checking systems such as claim-buster @xcite delineate fact-checking strategies, including claim-matching and novel claim processing, which are evidence collection and processing methods to provide a verdict on the truthfulness of the claim. recent studies @xcite @xcite have refined this process into llm-specific tasks. these systems aim to support the functions of traditional fact-checking organizations in addressing misinformation by enhancing efficiency. successful textual claim fact-checking requires world and common knowledge, along with some reasoning ability @xcite . llms have shown to be one of the best tools for these tasks @xcite . advancements in automated fact-checking have been significantly influenced by the integration of large language models (llms) and retrievalaugmented generation (rag) pipelines. for example, @xcite introduced a framework for llm-based systems that incorporates an internal mechanism to determine the most suitable llm for verifying a specific claim. ragar @xcite improves fact-checking by leveraging multi-modal inputs and iterative reasoning. evidence retrieval methods are important for the credibility and accuracy of automatic fact-checking systems, and are one of the most challenging first, the claim is passed into the claim-matching component. if successful, the fact-check article and summarized evidence from the article will be used as evidence. if claim-matching fails, the system proceeds to do novel claim verification, which generates question-answer pairs and summaries of highly relevant evidence to use as evidence. using previously gathered evidence, an llm generates a verdict and justification for the claim. factllama (hin cheung and lam, 2023) combines pre-trained llama models with external evidence retrieval to validate claims, while peng et al. (2023) enhances accuracy by integrating external knowledge and providing iterative feedback. additionally, singal et al. (2024) tackles misinformation in rag pipelines by re-ranking retrieved documents based on their credibility scores. ullrich et al. (2023); drchal et al. (2023) demonstrate the advantage of using whole documents as evidence to answer questions instead of individual sentences or paragraphs. for verdict prediction, many fact-checking pipelines use llm-based verification @xcite . finetuning llms improves the verdict accuracy @xcite , and helps avoid inbuilt biases towards certain verdict categories @xcite . ## methodology we use two strategies to gather evidence for factchecking claims: claim-matching and novel claim processing (figure 1 ). we first attempt to search for a relevant fact-check for the given claim as we can use them as evidence for the verdict prediction. if a claim has not been previously factchecked, it is considered novel. for novel claims, we break down the fact-checking process into these key steps: claim reformulation, question generation, query generation, online evidence retrieval, question answering and evidence curation. using the evidence collected from either claim-matching or novel claim processing, an llm is used to predict the veracity of the claim along with a justification. for novel claim processing, we only use online search so that our system is applicable to real-world uses. ## conclusion claimcheck demonstrates the viability of web evidence retrieval for automatic fact-checking systems using smaller language models. it achieved 0.626 accuracy on the averitec benchmark dataset. our approach establishes essential procedures for end-to-end fact-checking systems without relying on resource-intensive larger models. by developing an llm-size agnostic process, we ensure that claimcheck can benefit from future llm advancements while maintaining independence from specific model designs. we additionally show that claim matching can be a useful evidence retrieval approach to fact-checking non-novel claims. the use of small, open-source llms enhances reproducibility and accessibility. our online search mechanism efficiently leverages external search algorithms to retrieve only the most relevant evidence, significantly reducing computational resource demands compared to retrieving and analyzing information from knowledge bases. however, challenges with evidence quality necessitated our multi-question approach and content curation task. future work could explore iterative systems rather than single-pass frameworks, incorporate multimedia analysis capabilities for social media claims, and investigate targeted fine-tuning approaches that balance performance improvements with system independence.