Re-evaluating Evaluation in Text Summarization (2020)

Abstract: Automated evaluation metrics as a stand-in for manual evaluation are an essential part of the development of text-generation tasks such as text summarization. However, while the field has progressed, our standard metrics have not – for nearly 20 years ROUGE has been the standard evaluation in most summarization papers. In this paper, we make an attempt to re-evaluate the evaluation method for text summarization: assessing the reliability of automatic metrics using top-scoring system outputs, both abstractive and extractive, on recently popular datasets for both system-level and summary-level evaluation settings. We find that conclusions about evaluation metrics on older datasets do not necessarily hold on modern datasets and systems. We release a dataset of human judgments that are collected from 25 top-scoring neural summarization systems (14 abstractive and 11 extractive).

## introduction in text summarization, manual evaluation, as exemplified by the pyramid method @xcite , is the gold-standard in evaluation. however, due to time required and relatively high cost of annotation, the great majority of research papers on summarization use exclusively automatic evaluation metrics, such as rouge @xcite , js-2 @xcite , s3 @xcite , bertscore @xcite , mover-score @xcite etc. among these metrics, rouge is by far the most popular, and there is relatively little discussion of how rouge may deviate from human judgment and the potential for this deviation to change conclusions drawn regarding relative merit of baseline and proposed methods. to characterize the relative goodness of evaluation metrics, it is necessary to perform metaevaluation @xcite , where a dataset annotated with human judgments (e.g. tac foot_0 2008 @xcite ) is used to test the degree to which automatic metrics correlate therewith. however, the classic tac meta-evaluation datasets are now 6-12 years old foot_1 and it is not clear whether conclusions found there will hold with modern systems and summarization tasks. two earlier works exemplify this disconnect: (1) @xcite observed that the human-annotated summaries in the tac dataset are mostly of lower quality than those produced by modern systems and that various automated evaluation metrics strongly disagree in the higher-scoring range in which current systems now operate. (2) @xcite observed that the correlation between rouge and human judgments in the tac dataset decreases when looking at the best systems only, even for systems from eight years ago, which are far from today's state-of-the-art. constrained by few existing human judgment datasets, it remains unknown how existing metrics behave on current top-scoring summarization systems. in this paper, we ask the question: does the rapid progress of model development in summarization models require us to re-evaluate the evaluation process used for text summarization? to this end, we create and release a large benchmark for meta-evaluating summarization metrics including: ## preliminaries in this section we describe the datasets, systems, metrics, and meta evaluation methods used below. ## related work this work is connected to the following threads of topics in text summarization. human judgment collection despite many approaches to the acquisition of human judgment @xcite @xcite , pyramid @xcite has been a mainstream method to meta-evaluate various automatic metrics. specifically, pyramid provides a robust technique for evaluating content selection by exhaustively obtaining a set of semantic content units (scus) from a set of references, and then scoring system summaries on how many scus can be inferred from them. recently, @xcite proposed a lightweight and crowdsourceable version of the original pyramid, and demonstrated it on the duc 2005 @xcite and 2006 @xcite multi-document summarization datasets. in this paper, our human evaluation methodology is based on the pyramid @xcite and litepyramids @xcite techniques. chaganty et al. (2018) also obtain human evaluations on system summaries on the cnndm dataset, but with a focus on language quality of summaries. in comparison, our work is focused on evaluating content selection. our work also covers more systems than their study (11 extractive + 14 abstractive vs. 4 abstractive). ## implications and future directions our work not only diagnoses the limitations of current metrics but also highlights the importance of upgrading the existing meta-evaluation testbed, keeping it up-to-date with the rapid development of systems and datasets. in closing, we highlight some potential future directions: (1) the choice of metrics depends not only on different tasks (e.g, summarization, translation) but also on different datasets (e.g., tac, cnndm) and application scenarios (e.g, system-level, summary-level). future works on meta-evaluation should investigate the effect of these settings on the performance of metrics.