Investigating Metric Diversity for Evaluating Long Document Summarisation (2022)

Abstract: Long document summarisation, a challenging summarisation scenario, is the focus of the recently proposed LongSumm shared task. One of the limitations of this shared task has been its use of a single family of metrics for evaluation (the ROUGE metrics). In contrast, other fields, like text generation, employ multiple metrics. We replicated the LongSumm evaluation using multiple test set samples (vs. the single test set of the official shared task) and investigated how different metrics might complement each other in this evaluation framework. We show that under this more rigorous evaluation, (1) some of the key learnings from Longsumm 2020 and 2021 still hold, but the relative ranking of systems changes, and (2) the use of additional metrics reveals additional high-quality summaries missed by ROUGE, and (3) we show that SPICE is a candidate metric for summarisation evaluation for LongSumm.

## introduction text summarisation is an increasingly sought-after capability that is required by corporations and governments for productivity gains. for such usecases, long documents with complex structures are often used as the input data. however, work on summarizing long documents into detailed summaries has not dominated the summarisation research field. there have been some exceptions to this, for example, work on government reports @xcite and pubmed literature @xcite . in contrast, most of the text summarisation work focuses on shorter documents or generating shorter summaries (for example, wikipedia data @xcite , scientific articles @xcite , and news summarisation @xcite ) ). figure 1 : an example of a long document abstractive summary from the longsumm data set, presented using summvis @xcite . to bridge this gap, the shared task of summarizing long scientific articles (longsumm) was proposed, where the system should produce a detailed and informative technical summary of a source article. this shared task was introduced in the 2020 scholarly document processing workshop @xcite . the shared task includes an extractive and abstractive version of the problem. the former is based on the talksumm dataset @xcite , an alignment of presentation transcripts to the publication. the latter is captured using a data set of technical blogs and publications @xcite . the abstractive data set is interesting in that summaries must provide both high-level and low-level details. an example is provided in figure 1 , where the summary is a blog "walkthrough" of the main points of a paper (presented using the summvis tool @xcite , showing colored alignments of content to the source material). the 2020/2021 longsumm shared tasks resulted in a couple of key learnings for abstractive summarisation: (1) that there was no clear difference in performance between extractive and abstractive methods; and (2) approaches that focus on the representation of long documents, such as the bigbird @xcite and pegasus @xcite combination outperformed simpler abstractive methods like bart @xcite . one potential weakness of the longsumm shared tasks is that they were limited to the rouge family of metrics @xcite , including recall of unigrams (rouge-1), bigrams (rouge-2), and longest common subsequences (rouge-lcs). in contrast, current trends in natural language generation (nlg), for example, the e2e evaluation @xcite , and image caption generation (icg), for example, the ms coco evaluation @xcite , employ multiple metrics. there are also issues with the application of rouge to new data sets. for example, rouge has been shown to be problematic when used on text types other than news, like microblogs @xcite , meeting summaries, @xcite and online review text @xcite . 2 given that it is not clear that rouge is necessarily the best metric for this new domain, we take the approach that diversity of metrics is key. we thus employ the metrics from nlg e2e shared task and ms coco evaluation scripts. we also add some of the new metrics from these fields, such as spice @xcite , a metric considering semantic graphs that has been demonstrated to improve image captioning evaluation, and bertscore @xcite , a metric that utilizes bert contextual embeddings to better capture lexical and structural semantics and which is increasingly used in evaluating text summarisation. 3 these metrics can be seen as covering a range of linguistic phenomena. we provide more detail on the metrics in section 4. to consider the role of the different metrics for the longsumm evaluation, we use a spectrum of different system approaches, including oracle methods, baselines, and state-of-the-art approaches. in addition, where the original longsumm evaluation uses a single test set, we repeat our experiments 2 note: the rouge metric was originally designed for the duc 2001 data set of news articles at a time when extractive summarisation methods were the dominant method. for more information about duc 2001, visit https://duc.nist. gov/pubs.html#2001 our contributions are as follows. (i) we retest key outcomes from the earlier shared tasks, e.g, (i) abstractive and extractive methods perform similarly on the longsumm abstractive data set, and (ii) the relative performance of tested algorithms. (ii) we show that the informativeness of rouge might be affected by stopword matching. (iii) we show that spice agrees somewhat with rouge and bertscore, offering a complementary view on summarisation quality. the remainder of the paper is structured as follows. in section 2, we outline related work. section 3 describes the different summarisation methods and baseline approaches. we outline our experimental procedure in section 4. in section 5, we describe our experimental results that address the research questions above. section 6 presents qualitative analysis and future work. we present concluding remarks in section 7. ## related work in this section, we outline some of the highlights in which the nlp community has critically examined evaluation methodology. we provide more details on shared task data, leading approaches, and metrics examined in subsequent sections. we note that the field of machine translation has been a source of inspiration for other nlp fields. indeed, the rouge metric is itself inspired by the bleu metric from translation research. this field has shown that reliance on intrinsic metrics and reference summaries is problematic. for example, the bleu metric may not correlate with human judgments @xcite . indeed, in recent years, machine translation has turned to the research topic of quality estimation (qe) (specia and astudillo, 2018), the task of estimating run-time translation quality without ground truth data. our work has some superficial similarities to qe methodology, in examining summary rankings and high and low-quality quartiles. however, our analysis differs from the core focus of qe, as we investigate the utility of multiple metrics. within the nlg community, bleu has been used as an evaluation metric even though it is problematic. for example, it has been shown not to correlate with human judgments (for example, @xcite and @xcite ). the use of these metrics is further called into doubt when we see that n-gram matching metrics like bleu are also not suitable for evaluating text simplification @xcite , a closely related task to text summarisation. this has led to the research in new metrics (for example, gleu @xcite and bleurt @xcite ). in this work, we follow the nlg and icg best practice, which is to use a combination of metrics, knowing that each individual metric may have its failings. there have been some recent works on evaluating summarisation metrics @xcite , which highlights the limitation of current metrics and the need for upgrading evaluation protocols. we note that other metrics exist to overcome some of the limitations of rouge @xcite , such as needing to account for multiple judgments of content saliency as in the pyramid method @xcite . a linear ensemble of diverse metrics has also been shown to be able to outperform metrics in isolation @xcite . the nlg community has tended to report human quality assessments, for example, collecting judgments for quality and naturalness @xcite . in this respect, our work is again complementary in that we use summvis @xcite to inspect the quality of the system summaries. ## conclusions we present a detailed evaluation of multiple text summarisation metrics for long document summarisation. utilising a oracle, baseline and state-of-theart systems, we show that a diverse suite of metrics can capture work in a complementary fashion, so that an evaluation framework is not subject to the limitations of a single metric. in a rigorous analysis over 10 repeated trials, we show that performance of the tested approaches is roughly the same as published results. however, while some findings from the longsumm shared task can be replicated, we find the ranking of methods in our experiments differs from prior results. when we examine the top and bottom quartiles of summarisation performance, we show that rouge and bertscore are often in agreement. further diversity in evaluation may be obtained using the metrics commonly used natural language generation and image captioning. in particular, we present preliminary results that show that the spice metric, which considers graph comparisons of semantic information, also agrees with the rouge and bertscore metrics. we see that spice can identify other situations in which summarisation systems are performing well, complementing the insights gained from rouge and bertscore.