F act L ens: Benchmarking Fine-Grained Fact Verification (2025)

Abstract: Large Language Models (LLMs) have shown impressive capability in language generation and understanding, but their tendency to hallucinate and produce factually incorrect information remains a key limitation. To verify LLM-generated contents and claims from other sources, traditional verification approaches often rely on holistic models that assign a single factuality label to complex claims, potentially obscuring nuanced errors. In this paper, we advocate for a shift towards fine-grained verification, where complex claims are broken down into smaller sub-claims for individual verification, allowing for more precise identification of inaccuracies, improved transparency, and reduced ambiguity in evidence retrieval. However, generating sub-claims poses challenges, such as maintaining context and ensuring semantic equivalence with respect to the original claim. We introduce **FactLens**, a benchmark for evaluating fine-grained fact verification, with metrics and automated evaluators of sub-claim quality. The benchmark data is manually curated to ensure high-quality ground truth. Our results show alignment between automated FactLens evaluators and human judgments, and we discuss the impact of sub-claim characteristics on the overall verification performance.

## introduction large language models (llms) have proven to be powerful tools, demonstrating impressive capabilities in language generation and understanding @xcite . however, a well-known limitation of llms is their tendency to hallucinate, generating information that is factually incorrect or unsupported by evidence @xcite . as llms become more widespread, especially in applications where factual accuracy is crucial, there has been increasing research on methods to verify the factuality of llm-generated content as well as claims from other sources. previous works on building fact-checking benchmarks focus on generating claims with a ground truth label, and in some cases provide the evidence/context to verify the claim @xcite . claims are generated using human annotators @xcite , synthetic processes @xcite , or considering llm outputs on question-answering tasks @xcite . to increase the complexity of the fact-checking process, the claims are generated from source data of multiple domains & modalities, such as wikipedia text and/or tables @xcite @xcite , web pages @xcite , knowledge graphs @xcite , online posts/chats @xcite , and qa tasks from various domains such as statistics, finance, legal, etc @xcite . these works also provide baseline fact-checking pipelines, which typically involves two main stages: (1) the retrieval of relevant evidence using search apis and multimodal data-lakes @xcite and (2) the verification of claims based on that evidence using nli-based, llm-based and fine-tuned fact-verification models @xcite . some works also explore delegating these steps entirely to an llm-based policy framework @xcite . despite this structured pipeline, most existing methods rely on a holistic verification model, where complex claims are assigned a single factuality label, often obscuring the nuanced nature of the errors or inaccuracies in the claims. in this work, we echo the sentiments of @xcite ; @xcite ; @xcite ; @xcite ; @xcite for a shift towards fine-grained verification of complex claims, where claims are decom-posed into smaller, more manageable sub-claims that can be individually verified. we additionally emphasise on the need to provide evaluation metrics to benchmark such fine-grained verification, and enrich existing benchmarks with fine-grained verification labels. as shown in figure 1 , the benefits of finegrained verification are substantial. by breaking down a complex claim into its constituent subclaims, verification is more precise, allowing for pinpointing exact locations of factual inaccuracies. additionally, this approach enables more transparent rationalizations and explanations, as each subclaim can be linked directly to its corresponding evidence or lack thereof. fine-grained decomposition also narrows the scope of evidence retrieval, making the subsequent verification process more focused and less prone to ambiguity. achieving fine-grained verification, however, presents its own challenges. decomposing a raw, complex claim into smaller sub-claims is not simply a matter of splitting it into sentences. poorly constructed sub-claims can introduce a variety of issues: they may lose the context necessary for proper verification, lack atomicity, or misrepresent the original information by either omitting key details or introducing new fabricated ones. ensuring the quality and verifiability of these sub-claims is, therefore, critical for the overall success of the verification process. to address these challenges, we introduce factlens, a benchmark designed specifically for fine-grained fact verification. factlens provides a novel suite of metrics for evaluating the quality of sub-claim generation and incorporates automated evaluators that combine llm-based assessments with statistical metrics. the dataset has been manually curated to ensure high-quality sub-claims. through empirical evaluation, we demonstrate that our sub-claim evaluators align closely with human judgments. moreover, our end-to-end evaluation shows that these fine-grained scores correlate strongly with improved downstream verification performance. we also present the results of stateof-the-art models on sub-claim generation, revealing the challenges inherent in this task and the need for further research in this area. ## evaluating sub-claims with factlens at the core of fine-grained verification is the decomposition of complex claims into smaller, more the ioc is responsible for organizing the summer olympics. the ioc is responsible for organizing the paralympic games. the ioc was founded in 1894. the ioc is based in lausanne, switzerland. specific sub-claims when necessary. the accuracy of the overall verification process depends heavily on the quality of these sub-claims; errors (e.g. oversimplification, omission of important details, or incorrect contextualization) in their formulation can lead to flawed verification outcomes. to detect potential issues early on in the sub-claim generation process, we propose a set of metrics to quantitatively assess sub-claim quality across several dimensions. ## conclusion in this paper, we introduce the benchmark factlens to evaluate fine-grained claim verification, enriching existing benchmarks. we also identify important metrics for assessing the quality of fine-grained sub-claims and propose an automated evaluator to provide early signals of decomposition failures and evaluate claim decomposition approaches.