S umm E val: Re-evaluating Summarization Evaluation (2021)

Abstract: The scarcity of comprehensive up-to-date studies on evaluation metrics for text summarization and the lack of consensus regarding evaluation protocols continue to inhibit progress. We address the existing shortcomings of summarization evaluation methods along five dimensions: 1) we re-evaluate 14 automatic evaluation metrics in a comprehensive and consistent fashion using neural summarization model outputs along with expert and crowd-sourced human annotations; 2) we consistently benchmark 23 recent summarization models using the aforementioned automatic evaluation metrics; 3) we assemble the largest collection of summaries generated by models trained on the CNN/DailyMail news dataset and share it in a unified format; 4) we implement and share a toolkit that provides an extensible and unified API for evaluating summarization models across a broad range of automatic metrics; and 5) we assemble and share the largest and most diverse, in terms of model types, collection of human judgments of model-generated summaries on the CNN/Daily Mail dataset annotated by both expert judges and crowd-source workers. We hope that this work will help promote a more complete evaluation protocol for text summarization as well as advance research in developing evaluation metrics that better correlate with human judgments.

## evaluation metrics and summarization models we briefly introduce metrics included in our evaluation toolkit as well as the summarization models for which outputs were collected at the time of releasing this manuscript. ## resources we now describe the resources collected and released together with this manuscript. ## model re-evaluation we now turn to an analysis of model scores across human evaluations and automatic metrics. the evaluated models were released between 2017 and 2019, represent different approaches to summarization: abstractive, extractive, and hybrid, and their architectures reflect the trends in summarization research. although in many cases we obtained multiple variants of the same model, in the study we focus on the versions with the highest rouge-l scores. in the source documents nor relevant for the summaries, the annotators interpreted it as hallucinations and assigned lower consistency and relevance scores. additionally, many reference summaries in the cnn/dailymail dataset were constructed by naively concatenating bullet-point summaries into contiguous sequences. such processing steps negatively affected the coherence of examples. similar trends in human studies of reference summaries were reported by @xcite . examples of noisy reference summaries are shown in presented results provide a comprehensive perspective on the current state of the field and highlight directions for future modeling work. ## conclusions we introduced summeval, a set of resources for summarization model and evaluation research that include: a collection of summaries generated by recent summarization models on the cnn/dailymail dataset, an extensible and unified toolkit for summarization model evaluation, and a diverse collection of human annotations of model outputs collected from the crowd-source and expert annotators. using the accumulated resources we re-evaluated a broad selection of current models and evaluation metrics in a consistent and comprehensive manner. we hope that this work will prove to be a valuable resource for future research on text summarization evaluation and models. we also encourage the research community to join our efforts by contributing model outputs and extending the evaluation toolkit with new metrics.