A Tale of Evaluating Factual Consistency: Case Study on Long Document Summarization Evaluation (2025)

Abstract: Ensuring factual consistency in summarization remains a challenge, especially for long-document evaluation. While automated, reference-free evaluation models are essential given the impracticality of large-scale human assessment for lengthy texts, challenges persist in evaluating different systems on how to handle different summary granularities and evolving model generations. In this work, we conduct a systematic study on diverse factual-consistency evaluation systems across four long-document datasets, encompassing summaries generated by models from non-LLMs to proprietary LLMs. Our analysis reveals that fine-grained continuous scores can provide more reliable assessments of different evaluation systems’ capabilities than binary classification. We also examine the relationship between sentence-level and summary-level model performance, highlighting its dependency on dataset characteristics. Moreover, our study reveals that advanced systems can achieve higher recall in error detection for older summaries, yet struggle with false positives and fine-grained error detection. Our analysis and case studies provide further insights into designing robust factuality evaluation systems, which are becoming increasingly in demand as generative models advance rapidly.

## introduction despite the recent progress for summarization models in producing fluent summaries, they still encounter challenges in producing summaries that are factually consistent with the source context @xcite @xcite @xcite . while human evaluation remains the best practice to judge the factual consistency of automatically generated summaries, it becomes increasingly challenging when long sequences of generated texts (close to or over 100 words) and inputs (over thousands of words) need to be evaluated @xcite . thus, building reference-free evaluation metrics becomes much more appealing. yet, testing and developing reliable evaluation systems foot_0 faces challenges, especially for long document summarization. different from short news/dialogue summarization benchmarks-such as summac @xcite , aggrefact @xcite and di-alsumm @xcite ) -which provide summary-level binary labels (1/0) indicating the factual consistency of brief (1-3 sentence) summaries, long-document evaluation benchmarks @xcite @xcite typically feature summaries of three or more sentences and are over 100 words in length. this discrepancy raises the question about how to produce the summary-level predictions for long document summaries. while some works @xcite opt for a binary classification setup, others aggregate human-annotated sentence-or clause-level scores to produce a continuous measure of the overall factual consistency. while different evaluation systems are proposed to optimize for individual datasets, clarifying the impact of label setups is crucial to provide fair comparisons of different approaches. moreover, inspired by the study by @xcite , several long-document evaluation datasets collect sentence-level fine-grained error types. although recent work @xcite started incorporating error prediction at the sentence level, their approaches relied on llm's zero-shot capability, and there lacks a systematic evaluation of how different systems perform in de-tecting the fine-grained error types. meanwhile, although summarization models have quickly evolved from pre-trained models such as bart @xcite and pegasus @xcite to large language models (llms) @xcite @xcite , the development of benchmarking datasets has lagged behind (i.e., the human-annotations of bart-generated summaries @xcite still serve as one primary reference to assess the performance of varied strong llm-based evaluators @xcite ). as found by @xcite , proprietary llm-generated summaries have a lower rate of intrinsic errors compared to those of non-llmgenerated summaries. before attempting to excel on any given benchmark, it is vital to understand the strengths and limitations of different evaluation systems across model-generated summary types. to overcome the limitations mentioned above, we conduct a systematic study over four different long-document summarization evaluation datasets that cover different source text domains, with summaries generated by models ranging from non-llms to proprietary llms. we empirically evaluate different factual consistency evaluation systems over the combined datasets, motivated by multiple research questions. exploring the optimal utilization of summary-level annotation in evaluating long summaries (rq1), we find that fine-grained, continuous scoring methods are more effective than binary classification (which is heavily relied on in short document summarization evaluations) ( §5.1.1). we also examine the transferability of models' sentence-level prediction capabilities to summary-level evaluations (rq2), noting a high correlation between model efficiency in detecting sentence-level inconsistencies and the performance on summary-level tasks, supposing utilization of suitable datasets and metrics ( §5.1.2). additionally, we assess the strengths and limitations of different evaluation systems in capturing fine-grained error types (rq3), observing that more advanced systems can significantly improve the recall in detecting errors, yet still face challenges in minimizing false-positives ( §5.2). lastly, our analyses on how factual-inconsistency errors evolve as summarization systems are updated ( §3.2 and §6) and the systematic comparison of system performances across time (rq4, §5.3) underscores the need for continuous efforts in co-designing benchmarks and iteratively refining evaluation systems to enhance factual error detection. lastly, we conduct both quantitative and qualitative analyses of model predictions, which reveal key takeaways about the strengths and limitations of different approaches, as well as actionable insights for guiding the development of future methods ( §6). foot_1 2 related work long document evaluation benchmarks research on automatic factual consistency evaluation metrics and resources for long document summarization is limited. @xcite released annotated model-generated summaries assessing factual consistency at the sentence and summary levels for govreport @xcite and arxiv @xcite and @xcite introduced human-evaluation guidelines for long document summaries and released their crowdsourced datasets. furthermore, @xcite , @xcite , @xcite , and @xcite introduced multiple benchmarks on long documents and covered diverse domains. however, most prior studies proposed and validated factual consistency evaluation methods using their own released corpora, which often lacked standardized reference labels (some used binary summary labels while others prefer continuous scores). we aggregate multiple long document evaluation datasets that span multiple domains, unify the sentence-level and summary-level prediction tasks, and conduct a comprehensive study on the evaluation tasks. evaluation systems to tackle the challenges of long documents, numerous approaches leveraging either natural language inference (nli) models @xcite @xcite or question-answering-based models @xcite were explored. with the rise of more capable llms, evaluating summaries on either sentence or summary level through zero-shot prompting became increasingly popular in the community @xcite @xcite . our study benchmarks different evaluation systems and systematically evaluates the strengths and limitations of those systems, providing discussions on the future directions in developing more robust factual consistency evaluation systems. ## experimental setup with the varied research questions introduced, our main focus is to perform a systematic study on existing factual consistency evaluation systems under the long document evaluation setting. below, we describe the systems we experiment with to produce sentence-level labels ( §4.1), as well as the evaluation metrics suitable for sentence-level and summary-level evaluations ( §4.2). ## analysis we conducted both quantitative and qualitative analyses of model predictions across different systems, leading to the following insights and takeaways. takeaway 1: the main types of factual consistency errors have shifted as summarizer models continue to improve, from surface-level and grammatical errors to those requiring a deeper understanding of context and inferential reasoning. ## discussion and conclusion for long document summary evaluation, the connection between summary-level and sentence-level evaluation has largely been overlooked. most prior work directly adopted approaches from the shorttext domain or proposed a new benchmark and demonstrated the superior performance of their proposed approach. instead, we examine more fundamental challenges of selecting suitable evaluation granularity and metrics. without such reflection, researchers risk optimizing their methods toward objectives that are misaligned. for instance, many novel systems tested on aggrefact @xcite show minimal differences in scores. however, @xcite point out that this dataset contains at least 6% mislabeled content (flipped binary labels), casting doubt on the reliability and usefulness of the improved benchmarking results for informing future studies. our work serves as a foundational study to clarify existing challenges and provide guidance for future research. to sum up, we conducted a comprehensive evaluation of factual-consistency detection systems across multiple long-document datasets, analyzing their performance at different granularity levels. our findings highlight the benefits of fine-grained continuous scoring over binary classification and reveal the transferability of sentence-level predictions to summary-level evaluation. our analysis reveals that while advanced llm-based systems exhibit higher recall in detecting errors compared to older benchmarks, they struggle with false positives and the fine-grained detection of errors. our work further provides thorough analyses and useful insights into the current state of evaluation systems, calling for continuous benchmark development and system refinement to enhance factual consistency assessment in long document summarization.