Verify with Caution: The Pitfalls of Relying on Imperfect Factuality Metrics (2025)

Abstract: Improvements in large language models have led to increasing optimism that they can serve as reliable evaluators of natural language generation outputs. In this paper, we challenge this optimism in regards to factuality evaluation.We re-evaluate five state-of-the-art factuality metrics on a collection of 11 datasets for summarization, retrieval-augmented generation, and question answering.We find that these evaluators are inconsistent with each other and often misestimate the factual accuracy of NLG systems, both of which can lead to a variety of pitfalls.We further show that these metrics exhibit biases against highly paraphrased outputs and outputs that draw upon faraway parts of the source documents.We urge users of factuality metrics to proceed with caution and manually validate the reliability of these metrics in their domain of interest.

## introduction building automated evaluation metrics that match human judgment is difficult, ongoing research @xcite . past work has highlighted the flaws of automated evaluators in several nlp research domains, particularly machine translation @xcite . nonetheless, automated evaluation metrics are perennially appealing because they allow nlg system designers to bypass slower and costlier human evaluation. most recently, llm-based metrics have led to optimism that nlg evaluation can be automated @xcite . in particular, there is a growing demand for automated attribution evaluators, as llms are increasingly used for tasks in which factual reliability is crucial, such as summarization, retrievalaugmented generation, and open-ended chat @xcite . 1 details of our code and data are available at github.com/ameyagodbole/verify-with-caution in this work, we investigate automated metrics for evaluating "attribution to identified sources" (autoais; @xcite , i.e., judging whether a source document fully supports a claim. we perform a comprehensive re-evaluation of 5 state-of-the-art autoais evaluators (2 proprietary and 3 open-source) on the llm-aggrefact benchmark @xcite , a collection of 11 datasets of claim-document pairs that are annotated for attributability. we find several reasons to be cautious when using autoais evaluators. first, state-of-the-art (sota) autoais evaluators with comparable leaderboard scores have large differences in predictions. sota evaluators have low agreement on an instance level ( §3.1); error analysis based on different evaluators may yield different conclusions. evaluators can achieve comparable balanced accuracy by trading off true positive and true negative rates in different ways on different datasets ( §3.2); evaluators cannot be relied on without verification on new datasets. second, evaluators also often give poor estimates of system-level error rate: autoais metrics on some datasets overestimate and on others underestimate how frequently unattributable claims are generated by nlg systems ( §3.3). this can lead to misestimation of the scope for improvement on generation tasks ( §3.4); new system design ideas (such as new lms, new decoding algorithms, etc) may be incorrectly cast aside based on imperfect automated metrics. we highlight that current benchmarks may be insufficient to test whether evaluators can correctly rank nlg systems ( §3.5). we identify two biases in the current sota au-toais metrics. in many domains, autoais metrics struggle to detect unattributable claims with a high surface-level similarity with the document ( §4.1). we also show that the common practice of chunking long source documents to fit the evaluator's context window inherently limits their performance because certain claims may become unverifiable ( §4.2). both these properties-paraphrasing without directly copying and synthesizing information from different parts of a long input document-are desirable in an nlg system and may be penalized if not appropriately addressed by evaluators. in § 5, we compare approaches to reduce the bias/discrepancy between the labeled and predicted (estimated) system error rates. threshold tuning to minimize the absolute bias on a calibration set is a consistent method for achieving low absolute bias. for autoais evaluators that do not have a tunable threshold, posthoc adjustment of the estimated error rate @xcite can reduce the absolute estimation bias (with certain caveats). finally, in §6, we discuss the impact of these findings on downstream users of the autoais metrics, such as dataset developers and researchers studying how to improve the factuality of nlg systems. since metrics do not yet transfer consistently to new datasets, we urge users of these metrics to first perform human validation of metric predictions in new data domains and on new systems. we advise developers of new autoais metrics to report a breakdown of metric behavior on the different error types across different bias axes of the evaluation data and with an evaluation of system-level error quantification. ## re-evaluating factuality metrics we begin by comparing the predictions of factuality metrics at the example and system levels. we highlight how inconsistencies in the metrics affect their downstream use for facilitating benchmarks and comparing nlg systems. ## analysis of metric biases we identify two concerning biases that may affect evaluator predictions: (1) dependence on surfacelevel matches and (2) constraints due to contextwindow limitations. these biases may cause the evaluators to penalize desirable system outputs. ## metric adjustment as discussed in § 3.3, the autoais evaluators have a high bias in estimating the true error rate of nlg systems. we experiment with methods to reduce this bias and make the metrics more reliable in downstream applications. we assume a scenario where some human-labeled claim document pairs are available for calibration foot_8 . for these experiments, we use the predictions and scores assigned by the bespoke-7b evaluator on examples from the ragtruth datasets. we study three methods for reducing quantification bias (details in appendix a.7): (1) post-hoc adjustment @xcite that changes the predicted error rate based on the known tpr and fpr of the evaluator, (2) threshold tuning to minimize the absolute bias, and (3) threshold tuning to maximize bacc. the "adjusted counts" approach is appealing to use if the autoais evaluator does not provide a scalar score and directly returns a label. the method shows an inconsistent reduction in the absolute bias. in particular, we find that using the bestperforming system (gpt-3.5-turbo for ragtruth datasets) based on labeled error rate leads to poor estimation of the tpr and fpr of the evaluator. this is due to the low prevalence of the label '0' in examples of the best system. simply adjusting counts based on these bad estimates leads to high bias on the remaining systems (see