A Statistical Analysis of Summarization Evaluation Metrics Using Resampling Methods (2021)

Abstract: The quality of a summarization evaluation metric is quantified by calculating the correlation between its scores and human annotations across a large number of summaries. Currently, it is unclear how precise these correlation estimates are, nor whether differences between two metrics’ correlations reflect a true difference or if it is due to mere chance. In this work, we address these two problems by proposing methods for calculating confidence intervals and running hypothesis tests for correlations using two resampling methods, bootstrapping and permutation. After evaluating which of the proposed methods is most appropriate for summarization through two simulation experiments, we analyze the results of applying these methods to several different automatic evaluation metrics across three sets of human annotations. We find that the confidence intervals are rather wide, demonstrating high uncertainty in the reliability of automatic metrics. Further, although many metrics fail to show statistical improvements over ROUGE, two recent works, QAEval and BERTScore, do so in some evaluation settings.1

## introduction accurately estimating the quality of a summary is critical for understanding whether one summarization model produces better summaries than another. because manually annotating summary quality is costly and time consuming, researchers have developed automatic metrics that approximate human judgments @xcite @xcite @xcite . currently, automatic metrics themselves are evaluated by calculating the correlations between their scores and human-annotated quality scores. the value of a metric's correlation represents how similar its scores are to humans', and one metric is said to be a better approximation of human judgments than another if its correlation is higher. however, there is no standard practice in summarization for calculating confidence intervals (cis) for the correlation values or running hypothesis tests on the difference between two metrics' correlations. this leaves the community in doubt about how effective automatic metrics really are at replicating human judgments as well as whether the difference between two metrics' correlations is truly reflective of one metric being better than the other or if it is an artifact of random chance. in this work, we propose methods for calculating cis and running hypothesis tests for summarization metrics. after demonstrating the usefulness of our methods through a pair of simulation experiments, we then analyze the results of applying the statistical analyses to a set of summarization metrics and three datasets. the methods we propose are based on the resampling techniques of bootstrapping @xcite and permutation @xcite . resampling techniques are advantageous because, unlike parametric methods, they do not make assumptions which are invalid in the case of summarization ( §3.1; §4.1). bootstrapping and permutation techniques use a subroutine that samples a new dataset from the original set of observations. since the correlation of an evaluation metric to human judgments is a function of matrices of values (namely the metric's scores and human annotations for multiple systems across multiple input texts; §2), this subroutine must sample new matrices in order to generate a new instance, in contrast to standard applications of bootstrapping and permutation that sample vectors of numbers. to that end, we propose three different bootstrapping ( §3.2) and permutation ( §4.2) techniques for resampling matrices, each of which makes different assumptions about whether the systems or inputs are constant or variable in the calculation. in order to evaluate which resampling methods are most appropriate for summarization, we perform two simulations. the first demonstrates that the bootstrapping resampling technique which assumes both the systems and inputs are variable produces cis that generalize best to held-out data ( §5.1). the second shows that the permutation test which makes the same assumption has more statistical power than the equivalent bootstrapping method and williams' test @xcite , a parametric hypothesis test that is popular in machine translation ( §5.2). finally, we analyze the results of estimating cis and applying hypothesis testing to a set of summarization metrics using annotations on english single-and multi-document datasets @xcite @xcite . we find that the cis for the metrics' correlations are all rather wide, indicating that the summarization community has relatively low certainty in how similarly automatic metrics rank summaries with respect to humans ( §6.1). additionally, the hypothesis tests reveal that qaeval @xcite and bertscore @xcite emerge as the best metrics in several of the experimental settings, whereas no other metric consistently achieves statistically better performance than rouge ( §6.2; @xcite . although we focus on summarization, the techniques we propose can be applied to evaluate automatic evaluation metrics in other text generation tasks, such as machine translation or structure-to-text. the contributions of this work include (1) a proposal of methods for calculating cis and running hypothesis tests for summarization metrics, (2) simulation experiments that provide evidence for which methods are most appropriate for summarization, and (3) an analysis of the results of the statistical analyses applied to various summarization metrics on three datasets. ## preliminaries: evaluating metrics summarization evaluation metrics are typically used to either argue that a summarization system generates better summaries than another or that an individual summary is better than another for the same input. how similarly an automatic metric does these two tasks with respect to humans is quantified as follows. let x be an evaluation metric that is used to approximate some ground-truth metric z. for example, x could be rouge and z could be a human-annotated summary quality score. the similarity of x and z is evaluated by calculating two different correlation terms on a set of summaries. first, the summaries from summarization systems @xmath0, . . . , s n } on input document(s) @xmath1, . . . , d m } are scored using x and z. we refer to these scores as matrices x, @xmath2. then, the correlation between x and z is calculated at one of the following levels: ## correlation confidence intervals although the strength of the relationship between x and z on one dataset is quantified by the correlation levels r sys and r sum , each r is only a point 2 for clarity, we will refer to r sum and r sys as correlation levels and pearson, spearman, and kendall as correlation coefficients. ## significance testing although cis express the strength of the correlation between two metrics, they do not directly express whether one metric x correlates to another z better than y does due to their shared dependence on z. this statistical analysis is performed by hypothesis testing. the specific onetailed hypothesis test we are interested in is: one method for hypothesis testing the difference between two correlations with a dependent vari-able that is used frequently to compare machine translation metrics is williams' test @xcite . it uses the pairwise correlations between x, y , and z to calculate a t-statistic and a corresponding p-value. 6 williams' test is frequently used to compare machine translation metrics' performances at the system-level @xcite . however, the test faces the same issues as the fisher transformation: it assumes the input variables are normally distributed @xcite , and it is not clear whether the test should be applied at the summary-level. ## simulation experiments we run two sets of simulation experiments in order to determine which ci ( §5.1) and hypothesis test ( §5.2) methods are most appropriate for summarization metrics. the datasets used in the simulations are the multi-document summarization dataset tac'08 @xcite and two subsets of the single-document summarization cnn/dm dataset @xcite annotated by @xcite and @xcite . these datasets have @xmath3, respectively. the summaries were assigned overall responsiveness, relevance, or lightweight pyramid @xcite scores, respectively, by human annotators. the scores of the automatic metrics are correlated to these human annotations. ## summarization analysis we run two experiments that calculate cis ( §6.1) and run hypothesis tests ( §6.2) for many different summarization metrics on the tac'08 and cnn/ dm datasets ( §5). each experiment also includes an analysis which discusses the implications of the results for the summarization community. the metrics used for experimentation are the following: autosummeng @xcite , bertscore @xcite , bewt-e @xcite , meteor (denkowski and lavie, 2014), memog @xcite , moverscore @xcite ,npower @xcite , qaeval @xcite , rouge @xcite , and s 3 @xcite . we use the metrics' implementations in the sacrerouge library @xcite . ## limitations the large widths of the cis in §6.1 and the lack of some statistically significant differences between metrics in §6.2 are directly tied to the size of the datasets that were used in our analyses. however, to the best of our knowledge, the datasets we used are some of the largest available with annotations of summary quality. therefore, the results presented here are our best efforts at accurately measuring the metrics' performances with the data available. if we had access to larger datasets with more summaries labeled across more systems, we suspect that the scores of the human annotators and automatic metrics would stabilize to the point where the ci widths would narrow and it would be easier to find significant differences between metrics. although it is desirable to have larger datasets, collecting them is difficult because obtaining human annotations of summary quality is expensive and prone to noise. some studies report having difficulty obtaining high-quality judgments from crowdworkers @xcite , whereas others have been successful using the crowdsourced lightweight pyramid score @xcite , which was used in @xcite . then, it is unclear how well our experiments' conclusions will generalize to other datasets with different properties, such as documents coming from different domains or different length summaries. the experiments in @xcite show that metric performance depends on which dataset you use to evaluate, whether it be tac or cnn/dm, which is supported by our results. however, our experiments also show variability in performance within the same dataset when using different quality annotations (see the differences in results between @xcite ). clearly, more research needs to be done to understand how much of these changes in performance is due to differences in the properties of the input documents and summaries versus how the summaries were annotated. as an alternative to comparing metrics' correlations, @xcite argue for comparison based on the number of system pairs in which both human judgments and metrics agree on statistically significant differences between the systems, a metric also used in the tac shared-task for summarization metrics @xcite . this can be viewed similarly to kendall's τ in which only statistically significant differences between systems are counted as concordant. however, the differences in discriminative power across metrics was not statistically tested itself. more broadly in evaluating summarization systems, @xcite argue for comparing the performance of summarization models via paired t-tests or wilcoxon signed-rank tests @xcite . they demonstrate these tests have more power than the equivalent unpaired test when used to separate human and model summarizers. ## conclusion in this work, we proposed several different methods for estimating cis and hypothesis testing for summarization evaluation metrics using re-sampling methods. our simulation experiments demonstrate that assuming variability in both the systems and input documents leads to the best generalization for cis and that permutation-based hypothesis testing has the highest statistical power. experiments on several different evaluation metrics across three datasets demonstrate high uncertainty in how well metrics correlate to human judgments and that qaeval and bertscore do achieve higher correlations than rouge in some settings.