  The quality of a summarization evaluation metric is quantified by calculating
the correlation between its scores and human annotations across a large number
of summaries. Currently, it is not clear how precise these correlation
estimates are, nor whether differences between two metrics' correlations
reflects a true difference or if it is due to random chance. In this work, we
address these two problems by proposing methods for calculating confidence
intervals and running hypothesis tests for correlations using two resampling
methods, bootstrapping and permutation. After evaluating which of the proposed
methods is most appropriate for summarization through two simulation
experiments, we analyze the results of applying these methods to several
different automatic evaluation metrics across three sets of human annotations.
We find that the confidence intervals are rather wide, demonstrating high
uncertainty in how reliable automatic metrics truly are. Further, although many
metrics fail to show statistical improvements over ROUGE, two recent works,
QAEval and BERTScore, do in some evaluation settings.
