F a S t F act: Faster, Stronger Long-Form Factuality Evaluations in LLM s (2025)

Abstract: Evaluating the factuality of long-form generations from Large Language Models (LLMs) remains challenging due to accuracy issues and costly human assessment. Prior evaluation pipelines attempt this by decomposing text into claims, searching for evidence, and verifying claims, but suffer from critical drawbacks: (1) inefficiency due to complex pipeline components unsuitable for long LLM outputs, and (2) ineffectiveness stemming from inaccurate claim sets and insufficient evidence collection of one-line SERP snippets. To address these limitations, we adapt the existing decompose-then-verify evaluation framework and propose **FaStFact**, a fast and strong evaluation pipeline that achieves the highest alignment with human evaluation and efficiency among existing baselines. FaStFact first employs chunk-level claim extraction integrated with confidence-based pre-verification, significantly reducing the cost of web searching and inference calling while ensuring reliability. For searching and verification, it gathers document-level evidence from crawled website pages for retrieval during verification, addressing the evidence insufficiency problem in previous pipelines. Extensive experiments based on an aggregated and manually annotated benchmark demonstrate the reliability of FaStFact in both efficiently and effectively evaluating the factuality of long-form LLM generations. We submit the paper with code and benchmark, and will make them publicly available to facilitate research.

## introduction large language models (llms) have demonstrated remarkable progress on downstream tasks @xcite @xcite . despite these advancements, a significant challenge remains in their ability to reliably answer in-depth factuality questions. a particular concern is the tendency for llms to * corresponding authors. in this paper, we first analyze prior factuality evaluation methods and find that existing pipelines suffer from significant drawbacks in both efficiency and effectiveness, substantially hindering their applicability. from a systematic analysis, inefficiency stems from the sentence-level decomposition and separate decontextualization during claim extraction, which subsequently harms the later verification steps. the high processing time and inference costs significantly limit their practical use, particularly for evaluating the increasingly lengthy outputs of modern llms. in terms of ineffectiveness, metrics using hierarchical sentence-level decomposition yield inaccurate or incomplete claim sets. they mistakenly include unverifiable or redundant claims while missing essential information. these fundamental failures primarily result from inherent pipeline design constraints in the extractors, undermining subsequent verification and degrading overall evaluation quality. furthermore, existing methods often struggle with insufficient evidence retrieval, frequently relying on short, inadequate snippets that prevent conclusive verification even when ample online information exists. to address the identified limitations of both inefficiency and ineffectiveness, we then propose fastfact for evaluating the factuality of longform responses. fastfact begins with claim extraction and pre-verification, where an llm extracts atomic claims from the response. to combat the inefficiency of prior complex methods, this step utilizes dynamic chunking for longer context and accelerated processing, and incorporates a confidence-based pre-verification leveraging the llm's internal knowledge, accepting definite judgments only if confidence is high, thus reducing the need for external verification on straightforward claims. to sufficiently fact-check the claims that are pre-verified with high uncertainty, fastfact proceeds to collect document-level evidence via web-scraping. this design directly addresses the issue of insufficient evidence retrieval by fetching entire web page contents from search results to create a comprehensive knowledge base. to reliably assess fastfact, we further aggregate and manually annotate a benchmark fastfact-bench that consists of 400 pairs of long-form qa, and investigate fastfact's alignment with human judgment in long-form factuality evaluation, followed by a horizontal experiment comparing existing evaluation tools in parallel. fastfact shows superiority in processing time, token cost, and absolute variance from ground-truth scores, rendering it a more efficient, effective, and reliable tool for the factuality evaluation of long-form generation. ## related work llms exhibit factual errors, particularly in openended qa responses, leading to the development of factuality evaluation benchmarks. while early efforts utilized short-form benchmarks like sim-pleqa @xcite , @xcite , and halluqa @xcite to assess individual factoids based on answer-matching correctness, evaluating the factuality of complex long-form generations presents greater challenges. the prevailing approach for long-form factuality evaluation is the decompose-then-verify framework @xcite @xcite @xcite @xcite . systems like factscore @xcite pioneered this framework for biographies using a closeddomain corpus, while safe @xcite , expertqa @xcite , and factcheck-bench @xcite extended evidence gathering to open-domain search. veriscore @xcite refined claim extraction and context engineering, and factool @xcite broadened evaluation domains to include reasoning, coding, and scientific literature by incorporating diverse evidence sources like python executors and google scholar. additional related work on factuality benchmarks is in appendix b. ## analysis of existing long-form factuality evaluation systems this section analyzes current factuality evaluation pipelines for long-form content within the decompose-then-verify framework that follows a static agent workflow: (1) claim extraction: decompose long-form content into atomic claims; (2) evidence retrieval: retrieve evidence pertinent to each claim; (3) claim verification: verify the factuality of each claim based on the gathered evidence; and (4) overall scoring: combine factuality of each claim into an overall factuality score. we examine their limitations in terms of efficiency, effectiveness, and reliability compared to human judgment. ## fastfact to address the limitations of existing pipelines, we propose fastfact, a factuality evaluation framework of novel pipeline and metric design for efficient assessment of long-form generation. ## efficiency analysis to systematically compare the efficiency of fast-fact with existing evaluation baselines, we offer a complexity analysis by summarizing the workflow of each pipeline. ## fastfact-bench following discussions in section 3, despite numerous benchmarks targeting the evaluation of llm long-form factuality, existing evaluations are scattered in their reported score metrics, benchmark choices, and task domains, lacking a holistic parallel baseline comparison. many existing factuality benchmarks also lack ground-truth data for the subprocesses, such as claim extraction and verification @xcite , making it impossible to compare the reliability of factuality evaluation scores among pipelines. therefore, we propose fastfact-bench aggregated over five representative benchmarks on llm factuality in long-form generation and manually annotated them on both claim extraction and verification, enabling a comprehensive and robust evaluation across existing evaluation pipelines including fastfact. ## conclusion this paper discusses the challenges in long-form factuality evaluation, and introduces fastfact, a novel pipeline for a more robust and scalable solution. it tackles issues including complex claim decontextualization, inauthentic claim sets, and insufficient evidence retrieval. fastfact incorporates several key features: efficient and flexible chunk-based claim extraction, confidence-based pre-verification, and comprehensive documentlevel evidence retrieval. future studies could benefit from using fastfact for fast factuality evaluation of long-form generations in sota models.