Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2023)

Abstract: Human evaluation is the foundation upon which the evaluation of both summarization systems and automatic metrics rests. However, existing human evaluation studies for summarization either exhibit a low inter-annotator agreement or have insufficient scale, and an in-depth analysis of human evaluation is lacking. Therefore, we address the shortcomings of existing summarization evaluation along the following axes: (1) We propose a modified summarization salience protocol, Atomic Content Units (ACUs), which is based on fine-grained semantic units and allows for a high inter-annotator agreement. (2) We curate the Robust Summarization Evaluation (RoSE) benchmark, a large human evaluation dataset consisting of 22,000 summary-level annotations over 28 top-performing systems on three datasets. (3) We conduct a comparative study of four human evaluation protocols, underscoring potential confounding factors in evaluation setups. (4) We evaluate 50 automatic metrics and their variants using the collected human annotations across evaluation protocols and demonstrate how our benchmark leads to more statistically stable and significant results. The metrics we benchmarked include recent methods based on large language models (LLMs), GPTScore and G-Eval. Furthermore, our findings have important implications for evaluating LLMs, as we show that LLMs adjusted by human feedback (e.g., GPT-3.5) may overfit unconstrained human evaluation, which is affected by the annotators’ prior, input-agnostic preferences, calling for more robust, targeted evaluation methods.

## introduction human evaluation plays an essential role in both assessing the rapid development of summarization systems in recent years @xcite @xcite @xcite and in assessing the ability of automatic metrics to evaluate such systems as a proxy * equal contribution for manual evaluation @xcite @xcite . however, while human evaluation is regarded as the gold standard for evaluating both summarization systems and automatic metrics, as suggested by @xcite an evaluation study does not become "gold" automatically without proper practices. for example, achieving a high inter-annotator agreement among annotators can be difficult @xcite , and there can be a near-zero correlation between the annotations of crowd-workers and expert annotators @xcite . also, a human evaluation study without a large enough sample size can fail to find statistically significant results due to insufficient statistical power @xcite . therefore, we believe it is important to ensure that human evaluation can indeed serve as a solid foundation for evaluating summarization systems and automatic metrics. for this, we propose using a robust human evaluation protocol for evaluating the salience of summaries that is more objective by dissecting the summaries into finegrained content units and defining the annotation task based on those units. specifically, we introduce the atomic content unit (acu) protocol for summary salience evaluation ( §3), which is modified from the pyramid @xcite and @xcite protocols. we demonstrate that with the acu protocol, a high inter-annotator agreement can be established among crowd-workers, which leads to more stable system evaluation results and better reproducibility. we then collect, through both in-house annotation and crowdsourcing, rose, a large human evaluation benchmark of human-annotated summaries with the acu evaluation protocol on recent state-of-the-art summarization systems, which yields higher statistical power ( §4). to support evaluation across datasets and domains, our benchmark consists of test sets over three summarization datasets, cnn/dailymail (cnndm) (nalla- we summarize our key findings in tab. 1. our contributions are the following: (1) we propose the acu protocol for high-agreement human evaluation of summary salience. (2) we curate the rose benchmark, consisting of 22000 summary-level annotations and requiring over 150 hours of in-house annotation, across three summarization datasets, which can lay a solid foundation for training and evaluating automatic metrics. 1 (3) we compare four human evaluation protocols for summarization and show how they can lead to drastically different model preferences. (4) we evaluate automatic metrics across different human evaluation protocols and call for human evaluation to be conducted with a clear evaluation target aligned with the evaluated systems or metrics, such that task-specific qualities can be evaluated without the impact of general, input-agnostic preferences of annotators. we note that the implications of our findings can become even more critical with the progress of llms trained with human preference feedback @xcite and call for a more rigorous human evaluation of llm performance. ## related work human evaluation benchmarks human annotations are essential to the analysis of summarization research progress. thus, recent efforts have focused on aggregating model outputs and annotating them according to specific quality dimensions @xcite @xcite @xcite . the most relevant work to ours is @xcite , which annotates summaries according to semantic content units, motivated by the pyramid @xcite and @xcite protocols. however, this benchmark only covers a single dataset (cnndm) without a focus on similarly-performing state-of-the-art systems, which may skew metric analysis @xcite and not fully reflect realistic scenarios @xcite . in contrast, our benchmark consists only of outputs from recently-introduced models over three datasets. summarization meta-evaluation with a human evaluation dataset, there exist many directions of meta-evaluation, or re-evaluation of the current state of evaluation, such as metric performance analyses, understanding model strengths, and human evaluation protocol comparisons. within metric meta-analysis, several studies have focused on the analysis of rouge @xcite , and its variations @xcite , across domains such as news @xcite , meeting summarization @xcite , and scientific articles @xcite . other studies analyze a broader set of metrics @xcite @xcite @xcite , including those specific to factual consistency evaluation @xcite @xcite @xcite @xcite . regarding re-evaluating model performance, a recent line of work has focused on evaluating zeroshot large language models @xcite @xcite , noting their high performance compared to smaller models. as for the further understanding of human evaluation, prior work has compared approaches to human evaluation @xcite , studied annotation protocols for quality dimensions such as linguistic quality @xcite and factual consistency @xcite , and noted the effects of human annotation inconsistencies on system rankings @xcite . the unreliability and cost of human evaluation in certain settings have been emphasized @xcite , with some work noting that thousands of costly data points may need to be collected in order to draw statistically significant conclusions @xcite . our meta-analysis focuses on this latter aspect, and we further analyze potential confounding factors in evaluation such as length and protocol design, with respect to both small and large zero-shot language models. ## atomic content units for summarization evaluation we now describe our atomic content unit (acu) annotation protocol for reference-based summary salience evaluation, including the procedure of writing acus based on reference summaries and matching the written acus with system outputs. ## rose benchmark analysis we first analyze the robustness of our collected annotations and a case study on the system outputs. ## evaluating annotation protocols apart from acu annotations, we collect human annotations with three different protocols to better understand their characteristics. specifically, two reference-free protocols are investigated: prior protocol evaluates the annotators' preferences of summaries without the input document, while ref-free protocol evaluates if summaries cover the salient information of the input document. we also consider one reference-based protocol, ref-based, which evaluates the content similarity between the generated and reference summaries. appendix d.1 provides detailed instructions for each protocol. ## conclusion and implications we introduce rose, a benchmark whose underlying protocol and scale allow for more robust summarization evaluation across three datasets. with our benchmark, we re-evaluate the current state of human evaluation and its implications for both summarization system and automatic metric development, and we suggest the following: human judgments for summary quality can be diverse and affected by various factors such as summary lengths, and reference summaries are not al-ways favored. therefore, existing summarization datasets (e.g. cnndm) should only be used for the appropriate tasks. for example, they can be used to define a summarization task with specific requirements (e.g. maximum summary lengths), and be important for studying reference-based metrics.