Hybrid Fact-Checking that Integrates Knowledge Graphs, Large Language Models, and Search-Based Retrieval Agents Improves Interpretable Claim Verification (2025)

Abstract: Large language models (LLMs) excel in generating fluent utterances but can lack reliable grounding in verified information. At the same time, knowledge-graph-based fact-checkers deliver precise and interpretable evidence, yet suffer from limited coverage or latency. By integrating LLMs with knowledge graphs and real-time search agents, we introduce a hybrid fact-checking approach that leverages the individual strengths of each component. Our system comprises three autonomous steps: 1) a Knowledge Graph (KG) Retrieval for rapid one‐hop lookups in DBpedia, 2) an LM-based classification guided by a task-specific labeling prompt, producing outputs with internal rule-based logic, and 3) a Web Search Agent invoked only when KG coverage is insufficient. Our pipeline achieves an F1 score of 0.93 on the FEVER benchmark on the Supported/Refuted split without task‐specific fine‐tuning. To address Not enough information cases, we conduct a targeted reannotation study showing that our approach frequently uncovers valid evidence for claims originally labeled as Not Enough Information (NEI), as confirmed by both expert annotators and LLM reviewers. With this paper, we present a modular, open-source fact-checking pipeline with fallback strategies and generalization across datasets.

## introduction llms have advanced knowledge-intensive nlp tasks, but can generate ungrounded or hallucinated content, which undermines their reliability for automated fact checking @xcite . knowledge-graph (kg)-based systems can provide explicit and transparent evidence through structured triples, but remain restricted due to their limited coverage and slower response times in open-domain scenarios (jiang et al., 2020; kim * these authors contributed equally. @xcite . recent work, such as generate-on-graph @xcite , treats llms as agents that generate missing kg triples, highlighting the potential of hybrid agent-kg reasoning frameworks. this paper asks how a modular hybrid system can make fact-checking more reliable, and shows how a real-time pipeline improves both coverage and interpretability. we propose a realtime, agent-based pipeline (figure 1 ) that integrates three autonomous steps: 1) a kg retrieval for rapid one-hop lookups in dbpedia @xcite ; 2) language models to classify claims with a task-specific classification prompt using labels such as supported, refuted, or not enough information (nei) @xcite ; and 3) a web search agent invoked only when nei is returned, rewriting the claim for on-demand retrieval @xcite . 1 while our system does not perform multi-hop reasoning, it remains modular across evidence types (structured kg evidence, unstructured web evidence), using retrieval to compensate for kg's single-hop limitations. this kg-first, web-adaptive strategy leverages the explainability of structured data while preserving open-domain coverage. we evaluated our approach on the fever benchmark @xcite , its adversarial extension fever 2.0 @xcite , and, that is, the factkg dataset @xcite , achieving up to 0.93 f1 on fever and competitive results across all three without task-specific tuning. a focused not enough information reannotation study shows that our pipeline can uncover valid evidence for claims labeled as unverifiable, a finding corroborated by both expert human annotators and llm reviewers. ## related work recent work in automated fact verification has focused on integrating structured knowledge sources, retrieval components, and llms to improve factual consistency and evidence grounding @xcite @xcite . a growing number of systems have been combining neural models with kgs @xcite @xcite or using web-based retrieval to expand coverage @xcite . kg-based methods often rely on symbolic triples of the form (subject, predicate, object) as evidence. prior studies have explored how to align natural language claims with kg facts using embedding models @xcite , graph-based reasoning @xcite , semantic matching between claims and triples @xcite , and llms @xcite . while kgs offer structured and interpretable evidence, they can be limited by coverage and connectivity, particularly for claims requiring multi-hop or commonsense reasoning @xcite . in contrast, web-based fact-checking systems retrieve textual evidence from open-domain sources. oe-fact @xcite , for instance, used llms to process retrieved snippets and generate decisions. retrieval-augmented generation (rag) @xcite has also been applied to fact verification tasks by conditioning generation on retrieved content. however, reliance on web-based, unstructured evidence raises concerns around evidence quality and verifiability. there is a growing interest in agent-based and modular architectures for fact verification. the fire system @xcite employs an iterative retrieval and verification process, where the model dynamically decides whether to retrieve more evidence or make a decision. such approaches reflect a broader trend toward separating evidence retrieval from claim evaluation, often across different evidence sources or reasoning stages @xcite . finally, several studies have pointed out limitations with benchmark labels, particularly in the nei category @xcite . prior work has shown that some nei claims can be verified with external evidence @xcite , highlighting the role of human judgment in evaluating evidence sufficiency and the need for annotation guidelines that reflect real-world complexity. to expand on this prior work, we developed a modular pipeline that combines structured kg evidence with an agent fallback retrieval and includes an interpretable classification component. ## methodology given a natural language claim c, our goal is to predict a label @xmath0, refuted, nei}, along with a small set of textual or structured evidence e * that justifies the decision. our system follows a two-stage architecture: a kg-first classification stage, followed by a fallback retrieval and reasoning stage using open-domain web evidence. the system does not require task-specific training and operates in a zero-shot inference mode. an overview of the pipeline is shown in figure 1 . ## implementation our system is built in a modular way so that it can be accessed through a simple rest interface @xcite . the modularity makes it easy to test different components or replace models. we experiment with two evidence classifiers: gpt-4o mini (llm): for each evidence item e i , we construct a json prompt containing the claim c and the list {e i } (triples or snippets). the model returns {"label": s|r|n, "reason": r}, where r is a single sentence that cites evidence. during development, we tested various llm prompt variants to maximize classification accuracy and robustness before settling on the final versions reported in our results. the final prompts can be found in the appendix b. deberta-v3-mnli: we cast fact verification as natural-language inference. every pair ⟨c, @xcite outputs logits (ℓ e , ℓ n , ℓ c ) for {entailment, neutral, contradic-tion}. we apply softmax and pick the label with the highest probability p max . afterwards, we map them back to the fever labels. datasets: for our main experiments, we use the fever dataset, which labels claims as supported, refuted, or not enough information. to ensure fair comparison across experiments and with other papers and avoid ambiguity, we randomly sample 1,000 fever claims, explicitly removing all neilabeled instances. ## results and discussion baselines: following the claim-only setting in prior work, zero-shot llms without retrieval can resolve a portion of fever claims but remain ungrounded. the best baseline here (zero-shot 4o-mini) results in an f 1 0.801, while zero-shot 4.1-nano leads to f 1 0.734. although these models are competitive, the absence of explicit evidence limits the verifiability of their reasoning. separate stages: single-source variants show opposing error profiles. kg-only with an llm results in high precision (0.944) but lower recall (0.734), reflecting reliable yet sparse coverage. in contrast, web-only configurations are more balanced (e.g., llm web-only: prec. 0.912, rec. 0.908), suggesting broader coverage at the cost of increased noise. ## conclusion and future work we present a real-time fact-checking pipeline that combines the strengths of kgs and web retrieval to address the limitations of existing llm-based and kg-based systems. our kg-first, web-adaptive approach delivers both high precision and broad coverage, achieving strong empirical results across fever and other standard benchmarks without task-specific fine-tuning. it offers competitive accuracy with stronger reliability and interpretability than purely web-based or neural setups. in addition, our nei re-annotation study shows that in over 70% of cases, the system retrieves meaningful evidence for claims originally labeled not enough information. however, subjectivity in human judgments remains a challenge. overall, our work demonstrates the value of integrating structured and unstructured evidence for robust, interpretable open-domain fact verification. for future work, we plan to enhance support for multi-hop evidence, improve the detection of truly unverifiable claims, explore alternative classifiers, and extend our approach to additional knowledge sources and datasets.