A Critical Look at Meta-evaluating Summarisation Evaluation Metrics (2024)

Abstract: Effective summarisation evaluation metrics enable researchers and practitioners to compare different summarisation systems efficiently. Estimating the effectiveness of an automatic evaluation metric, termed meta-evaluation, is a critically important research question. In this position paper, we review recent meta-evaluation practices for summarisation evaluation metrics and find that (1) evaluation metrics are primarily meta-evaluated on datasets consisting of examples from news summarisation datasets, and (2) there has been a noticeable shift in research focus towards evaluating the faithfulness of generated summaries. We argue that the time is ripe to build more diverse benchmarks that enable the development of more robust evaluation metrics and analyze the generalization ability of existing evaluation metrics. In addition, we call for research focusing on user-centric quality dimensions that consider the generated summary’s communicative goal and the role of summarisation in the workflow.

## introduction the evaluation of natural language processing systems is crucial to ensure their effectiveness and reliability in real-world applications. it helps compare systems, validate whether the designed properties work as intended, understand the strengths and weaknesses of the underlying model, and often guide iterative improvements @xcite . although human evaluation, especially for natural language generation systems, is considered the most reliable evaluation method @xcite @xcite , automatic evaluation metrics are more widely used due to their cost-effectiveness, ease of use, repeatability, and speed @xcite . in addition to assessing the performance of summarisation systems, automatic summarisation evaluation metrics are also used for other purposes during summarisation system development, such as filtering noisy datasets to improve the quality of training data @xcite , ranking sampled candidates to output the best summary @xcite , and, integrating with reinforcement learning framework as a reward function @xcite . a critically important question is how effective these automatic summarisation evaluation metrics are. in other words, do the evaluation results obtained using these automatic metrics reflect the genuine quality of the summaries and summarisation systems under examination? for example, @xcite conclude that existing automatic metrics cannot reliably evaluate summaries generated using instruct-tuned gpt-3 model @xcite , because they find that gtp-3 summaries receive much lower scores than state-of-theart fine-tuned models @xcite on automatic metrics while outperforming them on human evaluation using a/b testing. meta-evaluating summarisation evaluation metrics, especially building resources that enable assessing the automatic metrics, has become urgent and attracted significant research interest @xcite @xcite @xcite . however, these resources were built and used in various ways, leading to inconsistent and confusing conclusions about the usefulness of these metrics. in this position paper, we take a critical look at the practices of meta-evaluating summarisation evaluation metrics. our paper is organised as follows: we first review recent meta-evaluation practices for summarisation evaluation metrics (section 2); then, in section 3, we discuss research trends and gaps around four critical decisions that must be made when we assess the automatic metrics, namely, choosing data to annotate, defining quality dimensions, collecting human judgements, and comparing automatic metrics against human judgements. finally, we provide recommendations in section 5. ## preliminaries the task of summarisation aims to generate a summary ŷ given a source text x, where ŷ encapsulates the key information in x. the summarisation evaluation metric typically takes the generated summary ŷ, optionally the source x or a (few) reference summary y, as input and produces a numeric value, which is a proxy of the overall quality or a particular dimension of quality, of ŷ. ## discussion we identify there are four critical decisions that must be made when we assess the automatic metric: (1) what source texts and summaries to use; (2) what quality dimensions to consider; (3) how to col-lect human judgements; and, (4) how to compare the automatic metric against human judgements. in this section, we discuss research trends and gaps around these four aspects. ## related work similar to summarisation, other natural language generation tasks, such as machine translation (mt), dialogue, and data-to-text generation, also have a long history of employing automatic evaluation metrics, such as bleu @xcite and meteor @xcite , to assess the quality of machine-generated texts. assessing the effectiveness and reliability of these automatic metrics is also an active research area, and regular shared tasks (e.g., wmt metrics shared task foot_1 ) are organised to encourage researchers to explore the strengths and weaknesses of automatic metrics. unfortunately, similar efforts to meta-evaluate summarisation evaluation metrics were unsustained, partially due to the complexity of the summarisation task itself. as observed by @xcite , although there are obvious parallels between summarisation and machine translation (mt), methodologies applied to meta-evaluate mt metrics have not been well explored in summarisation. with the advancement of large-scale generative models, evaluating text generated by llms and meta-evaluate corresponding evaluation metrics have also attracted significant interests @xcite @xcite @xcite . these studies usually concern similar quality dimensions as those in the summarisation field, and have a similar desire to find cost-effective ways to collect human judgements. ## conclusions and recommendations in this position paper, we critically examine the practices of meta-evaluating summarisation evaluation metrics in the literature. we identify several avenues in the field that can be further improved, regarding: choosing data to annotate, defining quality dimensions, collecting human judgements, and comparing automatic metrics against human judgements. for practitioners aiming to assess the effectiveness of automatic metrics for their particular use case, we suggest starting by considering the role a summarisation system plays in the real-world workflow. this includes identifying the readers of the generated summaries, understanding what information they seek, and what decisions they might make after reading the summary. once we have a clear picture of this, we can create documentsummary pairs that meet requirements and focus on the quality dimensions that end-users value most. human judgements can be collected from real endusers regarding both their perceived quality of the summary and the effect of these summaries on the actual downstream tasks they perform. finally, automatic evaluations can be assessed depending on the purposes of the evaluation, such as determining which summarisation system is better (system-level correlation is informative), choosing the best summary from multiple candidates (summary-level correlation/ranking), and detecting problematic summaries (binary classification). for researchers who aim to develop metaevaluation resources and novel evaluation metrics, we believe it is time to build more diverse benchmarks using data sampled from different domains and considering various summarisation constraints. that is to say, the generality of evaluation metrics should be tested to mitigate the risk of overestimating the effectiveness of automatic metrics across domains and applications @xcite . secondly, we believe there is an urgent need to standardise human evaluation practices to ensure reproducible human judgements over time and, more importantly, to make the collected re-sources extensible to new summarisation models. we recommend some best (basic) practices: (1) being aware of previous work and reusing previous resources (taxonomy, guideline, interface, etc.) whenever possible @xcite ; (2) adopting quality controls, such as training annotators to make sure they understand the annotation task, filtering out unqualified annotators and their annotations, etc @xcite ; (3) documenting the creation process (e.g., preprocessing, annotating) and recommended uses @xcite . finally, we argue that claims on the usefulness of evaluation metrics should be made based on comprehensive and reliable assessment under various usage scenarios, such as detecting summaries with significant errors, distinguishing summary systems of closing quality, or identifying more fine-grained issues in the generated summary.