Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics (2022)

Abstract: How reliably an automatic summarization evaluation metric replicates human judgments of summary quality is quantified by system-level correlations. We identify two ways in which the definition of the system-level correlation is inconsistent with how metrics are used to evaluate systems in practice and propose changes to rectify this disconnect. First, we calculate the system score for an automatic metric using the full test set instead of the subset of summaries judged by humans, which is currently standard practice. We demonstrate how this small change leads to more precise estimates of system-level correlations. Second, we propose to calculate correlations only on pairs of systems that are separated by small differences in automatic scores which are commonly observed in practice. This allows us to demonstrate that our best estimate of the correlation of ROUGE to human judgments is near 0 in realistic scenarios. The results from the analyses point to the need to collect more high-quality human judgments and to improve automatic metrics when differences in system scores are small.

## introduction automatic evaluation metrics are the most common method that researchers use to quickly and cheaply approximate how humans would rate the quality of a summarization system @xcite @xcite @xcite . the quality of a metric -how similarly it replicates human judgments of systems -is quantified by calculating the correlation between the metric's scores and human judgments on a set of systems, known as the system-level correlation @xcite . accurately estimating system-level correlations is critically important. summarization researchers use automatic metrics during system development to make decisions about which ideas work and which do not, and systems from different research groups are ranked by automatic metrics to define which system is the "state-of-the-art." if we do not have precise estimates of metric quality, it is not clear how much trust the community should put in such evaluation methodologies. at present, there are disconnects between how automatic metrics are evaluated and how they are used to evaluate systems. first, the metrics' scores which are used in practice are not the ones which are evaluated in system-level correlations; researchers compare systems based on metric scores calculated on the entire test set but calculate scores for system-level correlations when evaluating metrics on a much smaller subset of judged summaries. second, metrics are evaluated in a setting that is much easier than how they are actually used. metric correlations are calculated using systems that vary greatly in quality, whereas researchers compare new systems to recent work, which are likely to be very close in quality. discriminating between two systems of similar quality is much harder than doing so between low and high quality systems. in this work, we re-examine how system-level correlations are calculated and propose two independent changes to make the evaluation of metrics better aligned to how they are actually used to evaluate systems. first, we propose to modify the system-level correlation definition to use the entire test set to calculate the system scores for automatic metrics instead of only the subset of summaries judged by humans ( §3). with this change, the scores which are used to compare systems are directly evaluated, and we further demonstrate how the precision of our estimate of system-level correlations improves as a result. calculating system scores over a larger number of instances reduces the variance of the scores, which results in confidence intervals @xcite for the correlations that are 16-51% more narrow on average ( §3.2). second, we redefine a high quality metric to be one for which a small difference in score reliably indicates a difference in quality ( §4). then, instead of calculating the correlation with all available system pairs, we only evaluate with pairs of systems whose automatic metric scores differ by some threshold. this allows us to show, for example, that a rouge-1 score difference of less than 0.5 between systems has almost no correlation to how humans would rank the same two systems according to our best estimates ( §4.2). for two other metrics, bertscore @xcite and qaeval @xcite , we show their correlations calculated on system pairs of similar quality are much worse than under the standard correlation definition. these results cast doubt on how reliable automatic evaluation metrics are for measuring summarization system quality in realistic scenarios. our analyses point to the need to collect more high-quality human judgments of summaries in order to have more accurate estimates of metric correlations as well as the need to improve the ability of automatic metrics to discriminate between similarly performing systems. ## background automatic evaluation metrics are most commonly used to argue that one summarization system is better than another, typically by showing that the value of a metric improves with the "better" system. how similarly automatic metrics replicate human judgments of system quality is quantified by system-level correlations as follows. the summaries from n systems on m jud input documents are judged by humans z and scored with an automatic metric x . then, the systemlevel correlation between x and z is calculated as figure 1 : the system-level correlation is calculated between the average x and z scores on a set of summarization systems. x j i and z j i are the scores for the summary produced by system i (represented by rows) on input document j (represented by columns). in this work, we use kendall's τ (the "b" variant foot_1 ) as the correlation function because we are most concerned with a metric's ability to correctly determine whether one system is better than another since that is how metrics are used in practice. kendall's τ is computed based on the number of system pairs out of n 2 which are ranked the same by x and z. it is defined as because the computation of r sys involves randomness -its value depends on which m jud input documents (and even which n systems) were used -it is only an approximation of the true correlation between x and z. as such, @xcite proposed various methods for calculating confidence intervals for r sys . for instance, their boot-inputs method uses bootstrapping to repeatedly resample the m jud input documents used to calculate r sys , thereby calculating a confidence interval for the true r sys value for x and z. ## evaluating with all available instances although the above definition of the system-level correlation has been used by recent meta-evaluation studies of metrics @xcite @xcite , there is a disconnect between how the automatic metrics are evaluated and how they are used in practice. researchers who develop summarization systems evaluate those systems with automatic metrics on all m test test instances, not just the subset of m jud instances which were judged by humans. evaluating a system on a larger number of summaries may end up changing the system's score, which could potentially alter the overall ranking of a set of systems. therefore, the rankings that are used by practitioners to determine system quality are not the ones which are being evaluated in the standard definition of system-level correlation. 3 systems 87.0 87.5 88.0 88.5 89.0 89.5 bertscore ## evaluating with realistic system pairs next, we argue that the set of systems used to evaluate metrics is not reflective of how metrics are used in practice and propose a new system-level correlation variant to address this problem. ## related work the methodology behind meta-evaluating summarization evaluation metrics was established during the duc/tac shared tasks @xcite . in addition to competitions for developing high-quality summarization systems, there were also shared tasks for creating automatic metrics that correlated well with human judgments. the benchmark datasets created during duc/tac were small in size by today's standards because they were manually collected multi-document summarization datasets, which are hard to create at scale. as such, all of the model-generated summaries on the full test set were judged (so m @xmath0; §3), unlike for current datasets which are too large to fully judge. recently, there has been growing interest in revisiting the meta-evaluation of automatic evaluation metrics for summarization, in part due to the large differences between currently popular summarization datasets and those used in duc/tac. we view our work as continuing this direction of research, described next. @xcite argues that current evaluation metrics do not work as well when they are used to evaluate high-performing systems compared to those which were evaluated in duc/tac. both @xcite and @xcite re-evaluated how well existing evaluation metrics work on the popular cnn/dailymail dataset @xcite by collecting judgments of summary quality using recent state-ofthe-art systems. these datasets were used in our analyses. while the goal of these works was to identify which metrics correlated best with human judgments, our goal is to point out the ways in which the current methodology of meta-evaluating metrics is inconsistent with how they are used. then, the work of @xcite proposed statistical methods for estimating and comparing correlation values. in contrast to our work, they provide statistical tools for analyzing correlations, whereas we propose new definitions of correlations. finally, @xcite provided a theoretical analysis of the bias and variance of automatic and human evaluations of machine translations and summaries. among their conclusions, they argue for evaluating metrics with pairwise accuracy (kendall's τ ) and that it may be prohibitively expensive to collect enough human judgments to distinguish between two systems with very similar quality. our work further argues that metrics should be evaluated with a variant of kendall's τ calculated using realistic system pairs ( §4). unfortunately, their results suggest that collecting enough human judgments to accurately measure how well automatic metrics perform in this setting may be very difficult. related studies to ours have examined how the choice of which systems to include in metric evaluations impacts the correlation values. both @xcite and @xcite identify that metrics perform worse when scoring only the top-k systems in machine translation and summarization, respectively, and examine the use of pairwise comparisons for metric evaluation. further, @xcite demonstrate that outlier systems have an out-sized influence on the correlation values and recommend removing them from the metric evaluations. in contrast, our work proposes to change the evaluation methodology for metrics so that it more closely resembles how they are used in practice. this results in evaluating only on system pairs which are realistically compared by researchers, that is, those separated by small margins in automatic metric scores. we believe that this is a more principled approach to how to select which system pairs to evaluate on compared to previous work. ## conclusion in this work, we proposed two independent changes to how the system-level correlation of metrics is calculated to better align with how they are used to evaluate systems. our analyses showed that these modifications led to lower-variance estimates of correlations and that commonly reported improvements in metric scores may not reliably predict how humans would judge system quality. the results from the analyses point to the need for future data collection efforts of high-quality human judgments and improving automatic evaluation metrics when differences in system performance are small.