Revisiting Automatic Evaluation of Extractive Summarization Task: Can We Do Better than ROUGE ? (2022)

Abstract: It has been the norm for a long time to evaluate automated summarization tasks using the popular ROUGE metric. Although several studies in the past have highlighted the limitations of ROUGE, researchers have struggled to reach a consensus on a better alternative until today. One major limitation of the traditional ROUGE metric is the lack of semantic understanding (relies on direct overlap of n-grams). In this paper, we exclusively focus on the extractive summarization task and propose a semantic-aware nCG (normalized cumulative gain)-based evaluation metric (called Sem-nCG) for evaluating this task. One fundamental contribution of the paper is that it demonstrates how we can generate more reliable semantic-aware ground truths for evaluating extractive summarization tasks without any additional human intervention. To the best of our knowledge, this work is the first of its kind. We have conducted extensive experiments with this new metric using the widely used CNN/DailyMail dataset. Experimental results show that the new Sem-nCG metric is indeed semantic-aware, shows higher correlation with human judgement (more reliable) and yields a large number of disagreements with the original ROUGE metric (suggesting that ROUGE often leads to inaccurate conclusions also verified by humans).

## introduction text summarization is a difficult nlp task and an automatic evaluation of this task is even more challenging. however, automatic evaluation is vital for large-scale experiments as it acts as a replacement for time consuming and pricey human evaluation. as such, the reliability and robustness of automatic evaluation is very crucial. the most commonly used metric for evaluating text summarization is rouge @xcite . although rouge has been criticized for considering direct lexical overlap and thus not being semanticaware, the majority of summarization models' assessments today are still based on rouge scores. in this paper, we revisit the popular rouge metric exclusively in the context of evaluating extractive summarization task, a task where phrases and sentences from the original text are extracted to create a summary. as such, if the human-written summary includes more novel words than the original document, rouge will provide a poor score to extractive summaries due to a lack of semantic awareness. another limitation of the rouge metric in the context of extractive summarization is the following: while the extractive summarization task is generally framed as a sentence ranking problem, the rouge metric was not originally proposed for evaluating the quality of a ranker. indeed, the heavily used technique behind extractive summarization is to rank sentences from the original document according to how well they reflect the overall description and then create a summary by concatenating the top-ranked sentences. thus, the "right" evaluation metric for the extractive summarization task should also consider the quality of the sentence ranker. again think about a human-written summary which is highly abstractive in nature. a good ranker that ranks the most informative sentences at the top may still suffer from low rouge scores due to fewer direct lexical overlaps between the system summary and human-written summary. to address these limitations, we propose an alternative gain-based evaluation metric in this paper (called sem-ncg) for evaluating extractive summarization tasks, which is both 1) semantic-aware and 2) rewards a system-generated summary based on some groundtruth ranking of sentences from original document. ncg (normalized cumulative gain) is a widely used metric for evaluating the performance of ranking systems, especially when conducting multi-level relevance judgements @xcite . although ncg evaluation is not entirely new @xcite @xcite , one fundamental contribution of this paper is that it demonstrates how we can automatically generate a reliable semantic-aware groundtruth ranking of sentences within a source document, which essentially enables automatic sem-ncg based evaluation without any additional human intervention. to the best of our knowledge, this work is the first of its kind. to be more specific, given an original document and a human-written summary for evaluation purposes, we used several state-of-the-art sentence embedding techniques (including infersent, sentence transformer, elmo, google universal sentence encoding and their ensemble) to prepare groundtruth ranking of sentences from original document by computing semantic similarity between each individual sentence of original document and entire human written summary. finally, this groundtruth ranking is compared against model-inferred ranking to compute sem-ncg score, where a higher number means a better extractive summary. we have conducted extensive experiments with this new metric using the cnn/dailymail dataset and 6 state-of-the-art extractive summarization models (bert base , mobilebert, distilbert, roberta, xlnet, gpt-2). experimental results show that the new sem-ncg metric is: 1) semanticaware, 2) shows higher correlation with human judgement (more reliable), and 3) yields a large number of disagreements with the original rouge metric (suggesting rouge often leads to inaccurate conclusions). when cross-examined by humans, we found sem-ncg to be more accurate (62% of the time) than rouge on average where the two metrics disagreed on the relative performance of a pair of extractive summarization models. thus, in response to the question of whether we can do better than rouge for evaluating extractive summarization tasks, the answer appears to be "yes". ## related work evaluation of the text summarization task is challenging and has been studied vastly in the past. @xcite proposed the relative utility (ru) metric, which evaluates extractive summarization as a ranking task (similar to our formulation), but has not gained much popularity, because their approach requires manual labor to rank each sentence of a document, and it is not practical to manually annotate such large data-sets. rouge @xcite is perhaps the most popular metric used today for the evaluation of the automated summarization techniques, mainly because it is a simple and automatic process. however, rouge has been criticized a lot for primarily relying on lexical overlap @xcite of n-grams. later, @xcite suggested using a broad domain-independent paraphrase table derived from a bilingual parallel corpus to enable paraphrase matching for summary evaluation. @xcite showed that rouge suffers from poor performance in cases of terminology variation and paraphrasing. as of today, around 192 variants of rouge have been proposed @xcite including rouge with word embedding @xcite and synonym @xcite , graph-based lexical measurement @xcite , vanilla rouge @xcite and highlight-based rouge @xcite . however, none of the variants of rouge considers the ranking quality (core technique of extractive summarization); let alone providing an automatic way to do it, which is the primary goal of our work. researchers have also proposed metrics alternative to rouge: factoids-based (atomic information units for sentence meaning) @xcite and pyramid-based @xcite approaches are two of them. multiple different reference summaries are a must for both approaches, where the pyramid-based approach requires additional manual labor to construct the pyramid. since the pyramid must be built by hand and gives imprecise scores, this technique failed to gain much attraction. many enhancements have been made to the pyramid-based approach: precise automated system for calculating pyramid ratings @xcite , pyramid evaluation via automated information extraction @xcite , lightweight sampling-based version that is crowdsourcable @xcite and facet-aware evaluation @xcite for better assessment of knowledge coverage in extractive summarization. still, the pyramid-based approach necessitates significant additional manual labour making it less appealing for large-scale evaluation. researchers also attempted to develop methods for evaluating reference-free model summaries @xcite . distance measures between the system summary and reference summary based on word embeddings have also been proposed @xcite . moreover, model based evaluation for text generation (also adopted for text summarization) has also been a recent trend @xcite @xcite . yet, none of these metrics explicitly assess the quality of ranking performed by an extractive summarization method. here, k is the cutoff position (e.g., @xmath0, g@k and cg@k are the gain and cumulative gain, respectively, at the k-th position in the list. ncg@k is cg@k divided by the maximum achievable cg@k, also called ideal cg (icg@k), which is computed from the ideal ranking of the documents with respect to the query. the ideal ranking places the document(s) with the highest gain on the very top, followed by the documents with the next level of gain, etc. mathematically: the main motivation for introducing the sem-ncg metric is to ensure a fair evaluation of the extractive summarization task where the metric is both semantic-aware as well as captures the ranking quality of the extractive summarizer. indeed, for extractive summarization, sentences in the original document are ranked based on how well they reflect the overall description, and thus, evaluating it with a rank-aware metric like sem-ncg is more equitable . but, how can we develop a sem-ncg metric for the extractive summarization task that was originally designed for information retrieval systems? what would be the query in this case? what would be the definition of a document? how do we define the gains? how can we compute the groundtruth ideal ranking? all of these are important questions we need to answer before one can use sem-ncg evaluation for extractive summarization tasks. problem formulation: we formulate extractive text summarization as a ranking problem, where the output is a ranked-list of sentences based on how well they convey the overall content of the original document. let us assume that, input is a document groundtruth gains: it is indeed a philosophical question to ask what should be the definition of gains in case of the extractive summarization. in this work, we define gain as the following: definition 4.1 given document d and a sentence s from d, gain of s with respect to d is proportional to the degree of how well s captures the overall semantic meaning of document d. one way to measure this capturing power is to ask human judges. however, human judgment in this case is problematic for multiple reasons as follows: 1) human evaluation is time-consuming and expensive, 2) some human raters have the tendency to give higher ratings than deserved, this is known as the leniency problem, which results in higher variance @xcite , 3) natural language descriptions are noisy and ambiguous, which makes manual ordering of sentences by annotators even harder resulting in low inter-rater agreement. this is why we opted for an automated way to create the groundtruth gains without involving humans, as demonstrated by algorithm 1. automatic gain computation: how can we automatically infer groundtruth gains in order to automate sem-ncg@k computation? fortunately, in most summarization benchmark datasets, one or more reference summaries written by humans are also provided along with the original documents. we leverage these human-written reference summaries to automatically infer groundtruth gains. the exact process is presented in algorithm 1, where we utilize the semantic similarity between ## discussions and conclusion in this paper, we revisited the problem of automatic evaluation for the extractive summarization task, exclusively focusing on the popular rou ge metric. we first argued that any summary evaluation should be more semantic-aware and demonstrated that rou ge fails to capture semantics through comprehensive experiments. indeed, rou ge score drops (5-7%) even only for small percentages (20%) of synonym perturbation, and thus is not optimal for evaluating any summarization task. next, we argued that a "good" metric for evaluating extractive summarization task should assess its core ranking quality, which rou ge does not. to address this issue, we proposed a new metric called sem-ncg which is both semantic-aware and considers ranking quality. more importantly, sem-ncg provides an automated way to compare a set of top-ranked model-extracted sentences (the systemextracted summary) against an ideal ranking of sentences, where the ideal ranking is automatically inferred by computing gains based on some humanwritten summary. this saves us from tedious process of manual annotation of each sentence within the original document, thus making it practically suitable for large scale automated evaluation. the correctness of the sem-ncg metric depends largely on the reliability of the groundtruth gains computed by algorithm 1. therefore, to verify the quality of the groundtruth gains, we conducted extensive quantitative evaluations which confirm that the sem-ncg metric is stable across multiple sentence embedding techniques (very robust) [section 6.2]. through additional experiments, we have demonstrated the following as well: 1) sem-ncg correlates better with humans [section 7.2]; 2) sem-ncg often disagrees with rouge for pairwise comparison of summarization methods [section 6.4]; 3) in the cases of such disagreements, further verification from human judges confirmed that sem-ncg is more reliable than the rouge metric [section 7.1]; and 4) sem-ncg is a superior metric when evaluating summaries against a more abstract (low overlap) reference [section 7.2]. to conclude, we recommend the following practice: we recognize that our proposed sem-ncg metric overlooks redundancy when computing groundtruth gains; thus, our immediate future goal is to design a redundancy-aware sem-ncg, as well as expand sem-ncg for multi-references and multidocument summarization settings. article paul merson has restarted his row with andros townsend after the tottenham midfielder was brought on with only seven minutes remaining in his team's 0-0 draw with burnley on sunday. 'just been watching the game, did you miss the coach? #rubberdub #7minutes,' merson put on twitter. merson initially angered townsend for writing in his sky sports column that 'if andros townsend can get in (the england team) then it opens it up to anybody.' paul merson had another dig at andros townsend after his appearance for tottenham against burnley townsend was brought on in the 83rd minute for tottenham as they drew 0-0 against burnley andros townsend scores england's equaliser in their 1-1 friendly draw with italy in turin on tuesday night the former arsenal man was proven wrong when townsend hit a stunning equaliser for england against italy and he duly admitted his mistake. 'it's not as though i was watching hoping he wouldn't score for england, i'm genuinely pleased for him and fair play to him -it was a great goal,' merson said. 'it's just a matter of opinion, and my opinion was that he got pulled off after half an hour at manchester united in front of roy hodgson, so he shouldn't have been in the squad. 'when i'm wrong, i hold my hands up. i don't have a problem with doing that -i'll always be the first to admit when i'm wrong.' townsend hit back at merson on twitter after scoring for england against italy sky sports pundit merson (centre) criticised townsend's call-up to the england squad last week townsend hit back at merson after netting for england in turin on wednesday, saying 'not bad for a player that should be 'nowhere near the squad' ay @paulmerse?' any bad feeling between the pair seemed to have passed but merson was unable to resist having another dig at townsend after tottenham drew at turf moor. lor mor hor an athlete was brought in to save the game during an event against a rival team. although many disagreed with this decision as players have been known to get in trouble from time to time. andros townsend an 83rd minute sub in tottenham's draw with burnley. he was unable to find a winner as the game ended without a goal. townsend had clashed with paul merson last week over england call-up. paul merson and andros townsend have been in about for a while now, merson felt that townsend did not deserve a place in the english team. townsend scored for england with a crucial goal to which merson apologized and acknowledge the performance of townsend in that game and wished him well on his performance. the back and forth between the two men has continued regardless but it appears that now their bad feelings have subsided despite some light jest between the two.