V eri F ast S core: Speeding up long-form factuality evaluation (2025)

Abstract: Metrics like FactScore and VeriScore that evaluate long-form factuality operate by decomposing an input response into atomic claims and then individually verifying each claim. While effective and interpretable, these methods incur numerous LLM calls and can take upwards of 100s to evaluate a single response, limiting their practicality in large-scale evaluation and training scenarios. To address this, we propose VeriFastScore, which leverages synthetic data to fine-tune Llama3.1 8B for simultaneously extracting and verifying all verifiable claims within a given text based on evidence from Google Search. We show that this task cannot be solved via few-shot prompting with closed LLMs due to its complexity: the model receives \sim\mathbf{4K} tokens of evidence on average and needs to concurrently decompose claims, judge their verifiability, and verify them against noisy evidence. However, our fine-tuned VeriFastScore model demonstrates strong correlation with the original VeriScore pipeline at both the example level ( r=\mathbf{0.80} ) and system level ( r=\mathbf{0.94} ) while achieving an overall speedup of \mathbf{6.6}\times ( 9.9 \times excluding evidence retrieval) over VeriScore. To facilitate future factuality research, we publicly release our VeriFastScore model and synthetic datasets.

## introduction modern methods to evaluate the factuality of long-form generation, such as factscore (min et al., 2023), safe (wei et al., 2024), and veriscore (song et al., 2024), rely on a pipelined approach that uses llms to decompose an input text into "atomic" claims and then verify each claim against retrieved evidence documents. while this pipeline results in interpretable outputs that correlate strongly with human annotations of factuality, it is also slow (liu et al., 2025), requiring many calls to large models to evaluate a text, which limits its usability in both evaluation and training (e.g., as a reward model in rlhf). in this paper, we propose a significantly faster alternative: a single-pass factuality evaluator that simultaneously performs claim decomposition and verification over a long-form input. our method, verifastscore, is trained to jointly extract all verifiable claims from a response and assess their factuality against retrieved evidence. to train ver-ifastscore, we collate sentence-level evidence with claim-level outputs produced by veriscore, a high-quality general-domain factuality evaluation metric, into a synthetic dataset used to fine-tune llama3.1 8b instruct (grattafiori et al., 2024). each training example contains a response to be evaluated as well as a corresponding evidence context that consists of web search results retrieved by querying individual sentences in the response. verifastscore is then asked to output a list of verifiable claims from the response, each labeled as supported or unsupported based on the evidence (see figure 1 ). this task is non-trivial: the model must internally decontextualize the response, identify atomic claims, resolve coreferences, and assess support against an evidence context spanning up to 4k tokens of potentially noisy information. standard few-shot prompting of powerful closed models like gpt-4o struggles with this complexity, achieving a pearson correlation of only 0.33 with veriscore despite high api costs. despite these challenges, verifastscore matches the quality of veriscore closely while offering substantial improvements in both cost and runtime efficiency: it achieves a pearson correlation of 0.80 (pearson r, @xmath0.001) with veriscore's scores, a 9.9× modeling speedup and a 6.64× overall speedup in wall-clock time, reducing both modeling and retrieval latency while maintaining the metric's interpretability. human evaluation corroborates the high quality of outputs from verifastscore. we release veri-fastscore and our synthetic training dataset to support efficient, interpretable factuality evaluation and facilitate its use in alignment settings. ## building a faster factuality metric in this section, we first describe the limitations of existing factuality metrics like veriscore, whose separate decomposition, retrieval, and verification stages require several calls to llms to complete for a single response. these limitations motivate our design and development of verifastscore, which combines all steps in the existing factuality evaluation pipeline into a single model call. to enable such a fast evaluation, we generate synthetic data from veriscore and use it to fine-tune a language model on response-level decomposition and verification. ## results in this section, we describe the test set performance and computational cost of the trained verifastscore model and compare it with veriscore. @xcite gpt 4o mini @xcite claude 3.7 @xcite o3 mini @xcite gemini2 flash @xcite mistral 7b @xcite mixtral 8×7b @xcite llama3 8b @xcite qwen 7b distill @xcite qwen 1.5b @xcite olmo 7b @xcite ## runtime analysis to more precisely quantify the efficiency difference, we implemented a parallelized version of veriscore and compared it directly with veri-fastscore. we distributed sentence-level claim decomposition and claim-level verification across 8 gpus on a h100 node. ## qualitative analysis to better understand verifastscore's behavior, the first author conducted a small-scale human evaluation of 401 (response, evidence, claim, label) tuples, drawn from 21 test instances in the tulu3 personas dataset. the evaluation focused on three core aspects of verifastscore's output: claim extraction, claim verifiability, and verification accuracy. claim grounding and hallucination: all 401 extracted claims were grounded in the original model response, with no instances of hallucinated or fabricated claims. this is a crucial property for a factuality metric, particularly in downstream use cases like hallucination detection and rlhf training where spurious claim hallucination could mislead supervision. extraction coverage and omissions: veri-fastscore was generally effective at identifying claims, especially in responses rich with fac-tual content. however, it showed reduced reliability in responses that interweave factual and non-factual content, or contain few factual claims overall. claim omission was observed in about 6 out of the 21 evaluated responses, though 4 of these cases involved fewer than four missing claims. notably, in 3 of the omission cases, veri-fastscore extracted claims only from the final few sentences of the response, missing earlier factual content-suggesting a positional bias or early truncation. unverifiable and ill-formed claims: about 42 out of the 401 extracted claims were judged to be unverifiable due to missing key details (e.g., named entities, dates) in the original model response. in rare cases, verifastscore also extracted claims from inherently unverifiable sentences, such as advice or vague generalizations. ill-formed claims, those missing critical contextual elements, were rare (around 4 claims out of the total 401 claims), but they did alter the meaning of the original sentence in notable ways. verification accuracy: verifastscore was largely accurate in labeling claims against the provided evidence, with an estimated verification error rate of 91.5%. among the 34 incorrect verification labels, 25 of them were false positives: instances where verifastscore incorrectly marked a claim as supported. these errors often arose when partial matches or topical overlaps misled the model, particularly in cases where evidence spanned multiple unrelated contexts or referenced similar entities in different time periods or locations. evidence quality and noise: finally, we observed that the retrieved evidence was often noisy, especially in cases where the original response contained few or vague factual claims. this is likely due to the sentence-level retrieval strategy used by verifastscore, which may issue underspecified queries and produce loosely related results. these limitations in evidence quality can contribute both to omission in extraction and errors in verification. ## related work factuality evaluation via decomposition and verification factscore computes factual precision over atomic claims (min et al., 2023), while safe uses gpt-4 for claim-wise verification (wei et al., 2024). veriscore extracts only verifiable claims and uses google search to verify them (song et al., 2024). other systems like factool (chern et al., 2023), rarr (gao et al., 2023), and cove (elazar et al., 2021) operate similarly albeit with document-level or retrieval-augmented reasoning. decomposition quality and claim granularity. end-to-end and lightweight approaches. minicheck avoids decomposition entirely, using a small model to classify sentence-level factuality (tang et al., 2024). llm-oasis introduces a synthetic benchmark for scalable factuality evaluation (scirè et al., 2025), while factcheck-gpt trains an end-to-end verifier integrating retrieval and judgment (wang et al., 2024). ## conclusion we present verifastscore, a model that unifies claim decomposition and verification into a single model pass. trained on synthetic data derived from veriscore, our method processes long-form outputs with consolidated evidence and produces finegrained factuality judgments. despite the complexity of the task, veri-fastscore closely matches the accuracy of veriscore (pearson @xmath1.80), while achieving a 6.6× speedup in evaluation time. it also outperforms strong few-shot prompting baselines such as gpt-4o, and provides interpretable, claim-level outputs suitable for evaluation and alignment. we release the model and data to support scalable, efficient factuality assessment and to facilitate further research on long-form factuality and reward modeling.