Assessing Inter-metric Correlation for Multi-document Summarization Evaluation (2022)

Abstract: Recent advances in automatic text summarization have contemporaneously been accompanied by a great deal of new metrics of automatic evaluation. This in turn has inspired recent research to re-assess these evaluation metrics to see how well they correlate with each other as well as with human evaluation, mostly focusing on single-document summarization (SDS) tasks. Although many of these metrics are typically also used for evaluating multi-document summarization (MDS) tasks, so far, little attention has been paid to studying them under such a distinct scenario. To address this gap, we present a systematic analysis of the inter-metric correlations for MDS tasks, while comparing and contrasting the results with SDS models. Using datasets from a wide range of domains (news, peer reviews, tweets, dialogues), we thus study a unified set of metrics under both the task setups. Our empirical analysis suggests that while most reference-based metrics show fairly similar trends across both multi- and single-document summarization, there is a notable lack of correlation between reference-free metrics in multi-document summarization tasks.

## introduction summarization systems, which aim to preserve salient information from the source text in a more concise form, are being applied to an increasingly diverse range of domains, such as summarizing news articles, messenger-style text conversations, tweets, and so on @xcite @xcite . evaluating the performance of these systems is still challenging, and since human evaluation is expensive to obtain, automatic evaluation metrics continue to provide an effective way of evaluating summary quality. since no single metric can comprehensively measure every aspect of a summary, it is becoming increasingly common to report system performance in terms of multiple metrics @xcite . as such, it becomes desirable to find a small set of metrics that each reflect different aspects of system performance without redundantly repeating information. conversely, if a metric is highly correlated with another metric but outperforms it when compared with human evaluation, then that performance difference is more significant @xcite @xcite . however, in order to do this, one must first understand how these different metrics correlate with each other. previous work has focused on studying these metrics under the single-document summarization (sds) setup, especially news @xcite . however, it is well known that news summarization datasets contain a strong sentence position bias where the most salient information tends to be at the beginning of the article @xcite , which has been shown to have a strong impact on the behavior and performance of some summarization systems @xcite , but does not hold in other domains @xcite . evaluation metrics have also been reevaluated in the context of scientific articles @xcite , and more recently, dialogues @xcite , both using single documents as input. in contrast to sds, multi-document summarization (mds) is the task of generating a summary from several related documents @xcite @xcite . understanding how these metrics estimate mds tasks, however, remains unexplored. this is notable because many reference-free metrics in particular rely on the source document to evaluate the summary, and when the source consists of stylistically diverse multiple documents, we postulate that it makes the task especially challenging for reference-free metrics. it is unclear whether the automatic evaluation metrics will correlate with each other in the same way in mds as they do in sds tasks. to address these gaps, we present a systematic study on assessing the inter-metric correlations between evaluation metrics for multi-document summarization. our findings suggest a striking lack of correlation between the reference-free metrics under the mds paradigm. our contributions include the following: (1) we conduct a comprehensive set of experiments for multi-document summarization using several summarization models and datasets from different domains and evaluate them over a unified set of 16 metrics; (2) we contextualize our results by drawing comparable insights under the single-document summarization paradigm. (3) lastly, we summarize our key takeaways and discuss some potential implications of our findings. ## related work conventionally, automatic metrics for evaluating summarization systems are mostly reference-based which require human-written reference summaries against which system-summaries can be compared @xcite . however, since human annotation remains expensive to obtain, automatic evaluation metrics that rely on the source document(s) rather than human-generated reference summaries are becoming increasingly popular @xcite . in parallel to this, researchers have re-assessed how effective these different types of evaluation metrics are, with almost all prior work focused on the single-document framework. @xcite find that rouge is not effective at evaluating the performance of summarization systems in the domain of scientific articles. more recently, @xcite collect human pyramid-score evaluations @xcite ) of sets of 100 summaries generated from 25 top-scoring summarization systems on the cnn/dailymail dataset @xcite . they then assess how well 8 different automatic evaluation metrics correlate with the human annotations using the william's test @xcite , and they also see how well these metrics perform on the shared tasks from the text analysis conferences (tac). their analysis finds that most of the metrics fail to generalize well to all the datasets they tested, and that different metrics perform well on different datasets: moverscore @xcite is found to correlate type dataset domain #docs/input mds multi-news news ∼2.75 peersum v2 peer reviews ∼7.75 tsix tweets ∼35.7 sds cnn/dm news 1 samsum dialogues 1 we also collect system summaries and evaluate them with automatic metrics in our work, except we focus on the correlation between metrics, rather than comparing with human evaluation which is infamously difficult @xcite . while prior work has focused on sds, our analysis considers both mds and sds frameworks, a first such study to our knowledge, across datasets from four different domains. ## results and analysis in this section, we discuss the results of two main experiments where we investigate the inter-metric correlations across two types of summarization (multi-document and single-document) over four different domains (peer reviews, tweets, news, and dialogues). in each experiment we calculate the pearson's r correlations between metrics and report statistically significant values (@xmath0.05). ## conclusions and future work we conduct an in-depth assessment of the correlations between numerous evaluation metrics, including those that use reference summaries and those that do not, in the context of multi-document summarization tasks. as a further investigation, we also evaluate single-document summarization datasets on the same set of metrics. our results indicate that evaluation metrics behave noticeably differently when studied under mds and sds paradigms, which makes metrics for mds an interesting avenue of research to be explored further. moreover, measuring how these metrics correlate with different dimensions of human evaluation on mds might be beneficial.