Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors (2023)

Abstract: The propensity of abstractive summarization models to make factual errors has been studied extensively, including design of metrics to detect factual errors and annotation of errors in current systems’ outputs. However, the ever-evolving nature of summarization systems, metrics, and annotated benchmarks makes factuality evaluation a moving target, and drawing clear comparisons among metrics has become increasingly difficult. In this work, we aggregate factuality error annotations from nine existing datasets and stratify them according to the underlying summarization model. We compare performance of state-of-the-art factuality metrics, including recent ChatGPT-based metrics, on this stratified benchmark and show that their performance varies significantly across different types of summarization models. Critically, our analysis shows that much of the recent improvement in the factuality detection space has been on summaries from older (pre-Transformer) models instead of more relevant recent summarization models. We further perform a finer-grained analysis per error-type and find similar performance variance across error types for different factuality metrics. Our results show that no one metric is superior in all settings or for all error types, and we provide recommendations for best practices given these insights.

## introduction although abstractive summarization systems @xcite @xcite have improved dramatically in recent years, these models still often include factual errors in generated summaries @xcite . a number of metrics have emerged to detect factuality errors, including methods based on sentence entailment @xcite , finer-grained entail-ment @xcite , question generation and answering @xcite @xcite , and discrimination of synthetically-constructed error instances @xcite . despite recent analyses @xcite , reliably comparing these metrics remains difficult. in this paper, we provide a new benchmark that allows for finer-grained comparison between different factuality systems. we aggregate 9 existing annotated factuality datasets to create our benchmark aggrefact. we stratify it according to the underlying summarization model, categorized into ftsota, exformer and old based on their development timeline (see section 2). first, we ask: do factuality metrics perform equally well at identifying errors from state-of-the-art summarization models and from earlier models? for nine recent factuality metrics, including recent chatgpt-based metrics, we show that metric performance varies substantially between different categories of summarization models. most importantly, we found that the standard way of reporting improvements on category-agnostic benchmarks can be misleading, as most of these gains are on the old or exformer subset of the data which are less important to detect. on summaries generated by ftsota models, we found that there is no single metric that is superior in evaluating summaries from both the cnn/dm @xcite and xsum @xcite datasets. to better understand their behavior, we next analyze what error types are different factuality metrics capable of identifying (section 4). to do this, we leverage datasets from our benchmark that have fine-grained error annotations and unify these into a single taxonomy. we find that the error type distribution changes over time and even differs between annotations of the same summarization models across factuality datasets. analysis of the factuality metrics shows that metrics claim- ## comparison of factuality metrics first, we evaluate several sota factual consistency metrics on our benchmark, namely dae @xcite , questeval @xcite , summac-zs, summac-conv @xcite and qafacteval @xcite . 2 we also benchmark recent chatgptbased evaluation metics from @xcite and @xcite . chatgpt-zs and chatgpt-cot @xcite prompt llms to directly output a binary factuality decision. on the other hand, chatgpt-da and chatgpt-star @xcite ask llms to score the factuality of generated summaries on a scale of 0-100 and 1-5 respectively. more details about these metrics, including exact prompts are included in appendix b. ## finer-grained error analysis having established differences among factuality metrics across underlying summarization models, we now explore differences in metrics according to factuality error types. to do this, we need a way to unify error types across datasets in our benchmark and map them into a shared taxonomy. ## recommendations evaluate factuality models on modern summarization systems we have seen that ftsota yields significantly different results than ex-former or old. because of the prevalence of these systems, we believe that any new work should prefer evaluating on these sota summaries. particularly for factuality metrics that are either based on latest llms or on pre-trained models, evaluating on modern summarization systems is needed to see if these metrics are actually improving from the current state-of-the-art or merely patching errors in outdated systems that have already been fixed by other advances. annotate factual consistency errors from summaries generated by llms recent work @xcite shows that llms like gpt-3 are capable of generating summaries that are preferred over ftsota summaries by human annotators. furthermore, they show that existing factuality metrics cannot reliably detect errors in summaries from gpt-3 models as these latter summaries differ substantially from existing benchmarks and training sets. we encourage future work to annotate errors from llm-generated summaries and evaluate new factual consistency metrics on this set as well in addition to the ftsota set. as such, we believe that future work should construct "living" benchmarks for factuality evaluation that are consistently updated as more powerful summarization systems are introduced. choose the right metric for the job we note that there is no one clear winner among the metrics evaluated here (section 3). depending on the downstream application, different methods may be more or less appropriate, as our analysis shows. moreover, none of current factuality metrics can identify certain error types across datasets equally well. as qg/qa and nli models get better, we expect all of these methods to improve further. alternatively, although recent chatgpt-based metrics @xcite do not perform well on modern summarization systems, they can be a starting point for leveraging llms to perform factual consistency evaluation. use more consistent error types with our taxonomy, we have mapped error types annotated in previous work. it is relatively easier and more accurate to map errors from xsumfaith, goyal'21, and cliff to our unified error types as they have annotation granularity finer than sentence-level. we encourage future work to follow this taxonomy where possible and leverage definitions in prior work to make cross-dataset comparisons possible. here also, we encourage future work to prioritize annotation and evaluation of sota summaries. ## conclusion in this work, we analyzed several factuality metrics across a large meta-benchmark assembled from existing datasets. we find state-of-the-art fine-tuned summarization models still present challenges for detecting factual errors, and the performance of error detectors is often overestimated due to the reliance on older datasets. furthermore, we unify existing datasets into a common taxonomy and use this to highlight differences between datasets and summarization models, as well as the complexity of unifying concepts in this problem space.