Truth or Error? Towards systematic analysis of factual errors in abstractive summaries (2020)

Abstract: This paper presents a typology of errors produced by automatic summarization systems. The typology was created by manually analyzing the output of four recent neural summarization systems. Our work is motivated by the growing awareness of the need for better summary evaluation methods that go beyond conventional overlap-based metrics. Our typology is structured into two dimensions. First, the Mapping Dimension describes surface-level errors and provides insight into word-sequence transformation issues. Second, the Meaning Dimension describes issues related to interpretation and provides insight into breakdowns in truth, i.e., factual faithfulness to the original text. Comparative analysis revealed that two neural summarization systems leveraging pre-trained models have an advantage in decreasing grammaticality errors, but not necessarily factual errors. We also discuss the importance of ensuring that summary length and abstractiveness do not interfere with evaluating summary quality.

## introduction we are currently witnessing a sharp increase of research interest in neural abstractive text summarization. however, we have also seen growing concern that truth, as represented in the original document, becomes lost or twisted during the summarization process. the issue was raised recently by @xcite , who point out that widely used automatic metrics, which rely mostly on word overlap, fail to reflect factual faithfulness of a summary to the original text. until now, work on summarization has not provided systematic analysis of factual faithfulness. instead, the trend has been for papers to provide a few examples or general descriptions of frequent errors. an example is @xcite , who state that "[c]ommon mistakes are using wrong subjects or objects in a proposi-tion [...], confusing numbers, reporting hypothetical facts as factual @xcite or attributing quotes to the wrong person.", but stop short of providing a more rigorous analysis. recent work that breaks the trend is @xcite , who propose an evaluation framework for faithfulness in abstractive summarization. the summaries used to develop the framework are annotated with different types of faithfulness errors. however, the annotation scheme does not incorporate linguistic concepts, e.g., does not differentiate between semantic and pragmatic faithfulness. the aim of our research is to go beyond existing characterizations and provide a comprehensive typology that can be used to understand errors that neural abstractive summarization systems produce, and how they affect the factual faithfulness of summaries. the contribution of this paper is an error typology that was created by analyzing the output of four abstractive summarization systems. the systems vary in their use of pre-training, their model architecture and in the integration of extractive tasks during training. we carry out a comparative analysis that demonstrates the ability of the typology to uncover interesting differences between systems that are not revealed by conventional overlap-based metrics in current use. this paper represents the main results of @xcite , which contains additional examples and analysis. further, annotations used for our analysis and more detailed statistics are publicly available foot_0 to support future research on faithfulness errors. ## related work over the years there have been several methods to evaluate summarization methods @xcite , each with their own strengths and challenges. in this section, we first cover the rouge score, which is the main target of the criticism of overlap-based summarization metrics, such as from @xcite mentioned in section 1. we then provide a discussion on the relatively limited amount of work that has dealt with factual errors in summaries. finally, we introduce the automatic summarization systems that we use in our study. ## building the typology in this section, we describe our methods and present the typology that we created. ## comparison of summarizers in this section, we carry out a comparative analysis of the four summarization systems using the error typology. this analysis highlights the usefulness of the typology for achieving insight into the nature of summary errors. we made a random selection of 170 articles and one annotator annotated all four summaries for each article using the typology. these were combined with the previously annotated set of 30 articles. this yielded a total of 800 summaries with roughly 2600 annotated sentences. sentence annotations were additionally aggregated to summary level: a summary is labeled as malformed if it contains at least one malformed sentence, but no misleading sentence. if it contains at least one misleading sentence, it is labeled as misleading. ## conclusion and outlook in this section, we tie together the main contributions and insights of this paper, and discuss the avenues that it opens for future work.