Metrics also Disagree in the Low Scoring Range: Revisiting Summarization Evaluation Metrics (2020)

Abstract: In text summarization, evaluating the efficacy of automatic metrics without human judgments has become recently popular. One exemplar work (Peyrard, 2019) concludes that automatic metrics strongly disagree when ranking high-scoring summaries. In this paper, we revisit their experiments and find that their observations stem from the fact that metrics disagree in ranking summaries from any narrow scoring range. We hypothesize that this may be because summaries are similar to each other in a narrow scoring range and are thus, difficult to rank. Apart from the width of the scoring range of summaries, we analyze three other properties that impact inter-metric agreement - Ease of Summarization, Abstractiveness, and Coverage.

## introduction automatic metrics play a significant role in summarization evaluation, profoundly affecting the direction of system optimization. due to its importance, evaluating the quality of evaluation metrics, also known as meta-evaluation has been a crucial step. generally, there are two meta-evaluation strategies: (i) assessing how well each metric correlates with human judgments @xcite @xcite @xcite , which requires procuring manual annotations that are expensive and time-consuming, and (ii) measuring the correlation between different metrics @xcite , which is a human judgment-free method. in this work, we focus on the latter and ask two research questions: rq1: how do automated metrics correlate when ranking summaries in different scoring ranges (low, average, and high)? we revisit the experiments of @xcite which concludes that automated metrics strongly disagree for ranking high-scoring summaries. 2 we find that the scoring range has little effect on the correlation of metrics. it is rather the width of the scoring range which affects inter-metric correlation. specifically, we observe that metrics agree in ranking summaries from the full scoring range but disagree in ranking summaries from low, average, and high scoring ranges when taken separately. rq2: which other factors affect the correlations of metrics? in addition to the width of the scoring range, we analyze three properties of a reference summary on inter-metric correlation -ease of summarization, abstractiveness and coverage. overall we find that for highly extractive document-reference summary pairs, inter-metric correlation is high whereas metrics disagree when ranking summaries of abstractive document-reference summary pairs. we summarize our contributions as follows: (1) we extend the analysis of @xcite and find that not only do metrics disagree in the high scoring range, they also disagree in the low and medium scoring range. (2) we perform our analysis on the popular cnn/dailymail dataset using traditional lexical matching metrics like rouge as well as recently popular semantic matching metrics like bertscore and moverscore. (3) apart from the width of the scoring range, we analyze three linguistic properties of reference summaries which affect inter-metric correlations. ## summary generation to simulate the full scoring range of summaries that are possible for a document, we follow @xcite and use a genetic algorithm @xcite to generate extractive summaries. we optimize for 5 metrics -rouge-1, rouge-2, rouge-l, bertscore, and moverscore, generating 100 summaries per metric for each of the nearly 11k documents in the cnndm test set resulting in 500 summaries per document. after de-duplication, we are left with nearly 419 summaries per document on average. for the tac dataset, we randomly sample 500 summaries for each document from the nearly 2000 output summaries provided by @xcite .