Understanding the Extent to which Content Quality Metrics Measure the Information Quality of Summaries (2021)

Abstract: Reference-based metrics such as ROUGE or BERTScore evaluate the content quality of a summary by comparing the summary to a reference. Ideally, this comparison should measure the summary’s information quality by calculating how much information the summaries have in common. In this work, we analyze the token alignments used by ROUGE and BERTScore to compare summaries and argue that their scores largely cannot be interpreted as measuring information overlap. Rather, they are better estimates of the extent to which the summaries discuss the same topics. Further, we provide evidence that this result holds true for many other summarization evaluation metrics. The consequence of this result is that the most frequently used summarization evaluation metrics do not align with the community’s research goal, to generate summaries with high-quality information. However, we conclude by demonstrating that a recently proposed metric, QAEval, which scores summaries using question-answering, appears to better capture information quality than current evaluations, highlighting a direction for future research.

## introduction the development of a reliable metric that can automatically evaluate the content of a summary has been an active area of research for nearly two decades. over the years, many different metrics have been proposed @xcite @xcite @xcite @xcite . the majority of these of approaches score a candidate summary by measuring its similarity to a reference summary. the most popular metric, rouge @xcite , compares summaries based 1 our code is available at https://github.com/ cogcomp/content-analysis-experiments . figure 1 : both candidate summaries are similar to the reference, but along different dimensions: candidate 1 contains some of the same information, whereas candidate 2's information is different, but it at least discusses the correct topic. the goal of this work is to understand if summarization evaluation metrics' scores should be interpreted as measures of information overlap or, less desirably, topic similarity. on their lexical overlap, whereas more recent methods, such as bertscore @xcite , do so based on the similarity of the summary tokens' contextualized word embeddings. ideally, metrics that evaluate the content of a summary should measure the quality of the information in the summary. however, it is not clear whether metrics such as rouge and bertscore evaluate summaries based on how much information they have in common with the reference or some less desirable dimension of similarity, such as whether the two summaries discuss the same topics (see fig. 1 ). in this work, we demonstrate that rouge and bertscore largely do not measure how much information two summaries have in common. further, we provide evidence that suggests this result holds true for many other evaluation metrics as well. together, this allows us to conclude that the most frequently used metrics in summarization fail to evaluate the quality of information in a summary. our analysis casts rouge and bertscore into a unified framework in which the similarity of two summaries is calculated based on an alignment be-tween the summaries' tokens ( §3). this alignmentbased view of the metrics enables performing two different analyses of how well they measure information overlap. the first analysis demonstrates that only a small proportion of the metrics' token alignments are between phrases which contain identical information according to domain experts ( §4). the second reveals that token alignments which represent common information are vastly outnumbered by those which represent the summaries discussing the same topic ( §5). overall, both analyses support the conclusion that rouge and bertscore largely do not measure information overlap. then, we expand our analysis to consider if 10 other evaluation metrics successfully measure information quality or not ( §6). by demonstrating that nearly all of the metrics correlate much more strongly to rouge than to the gold-standard method of manually comparing two summaries' information @xcite , we argue the metrics are likely to measure information overlap no better than rouge does. however, one recently proposed metric that evaluates summaries using question-answering (qa), qaeval @xcite , correlates equally well to rouge and gold-standard annotations of information overlap. by viewing qaeval as inducing an alignment between two summaries and reasoning about its behavior, we demonstrate evidence that it measures information overlap much more strongly than either rouge or bertscore does, supporting that qa-based metrics are a promising direction for future research ( §7). while the summarization community has been aware, informally, of the shortcomings of the current evaluation metrics, this study provides experimental evidence beyond correlations to support these intuitions. the contributions of this work include (1) an analysis which reveals that rouge and bertscore largely do not measure the information overlap between two summaries, (2) evidence that many other evaluation metrics likely suffer from the problem, and (3) preliminary results which show that qaeval does measure information quality better than rouge and bertscore. ## understanding evaluation metrics reference-based evaluation metrics assume that human-written reference summaries have goldstandard content and score a candidate summary based on its similarity to the reference. an ideal evaluation metric that measures the content quality of a summary should score the quality of its information. for reference-based metrics, this means that the comparison between the two summaries should measure how much information they have in common. in this work, our definition of information is equivalent to what can be expressed through predicate-argument relations in text. information can be expressed in other ways, such as entailment, but we focus on predicates and arguments since they are more easily directly evaluated. metrics such as rouge and bertscore calculate the similarity of two summaries either by how much lexical overlap they have or how similar the summaries' contextual word embeddings are (discussed in more detail in §3). although we understand how their scores are calculated, it is not clear how the scores should be interpreted: are they representative of how much information the two summaries have in common, or do they describe how similar the summaries are on some other less desirable dimension, such as whether they discuss the same topics? our goal is to answer this question. knowing the answer is critically important. the goal of summarization is to produce summaries which contain the "correct" information (among other desiderata). automatic metrics are the most frequent method that researchers use to argue that one summarization model generates better summaries than another. if our evaluation metrics are not aligned with our research goals -or if we do not understand what they measure at all -then we do not know whether we are making progress as a community. ## a common framework the focus of our analysis will be primarily on two evaluation metrics, rouge and bertscore. although on the surface these two metrics appear to compare two summaries very differently, here we demonstrate how they can both be viewed as calculating a score based on a weighted alignment between the summaries' tokens. this common framework enables us to reason about how to interpret their scores. let @xmath0, r 2 , . . . , r m and @xmath1, s 2 , . . . , s n be the tokens of the reference and candidate summaries. rouge-1 counts the number of unigrams that are in common between the two summaries: foot_0 bertscore calculates a similarity score between two pieces of text based on the pairwise cosine similarities of their tokens' bert embeddings. let b ij be the similarity score between the embeddings for r i and s j . to calculate recall, bertscore first aligns every reference token to its most-similar summary token (eq. 2). then, the sum of the corresponding similarities is normalized by the number of reference tokens to get the recall score @xcite . a similar procedure is followed to calculate precision, but instead, every summary token is aligned to its most-similar reference token, and the sum of the similarities is normalized by the number of summary tokens. when multiple references are available, the precision and recall scores are defined to be the maximum respective values across references. because the token alignments are selected via the max operation, bertscore's alignment is unique, unlike for rouge. by formulating rouge and bertscore in a framework based on token alignments, we can reason about their behaviors by examining the tokens they align in two different analyses, as described next. ## scu-based analysis the first analysis compares the two metrics' token alignments to annotations derived from the pyramid method @xcite . the pyramid method is a technique to manually evaluate the content of a candidate summary by comparing it to a set of reference summaries. the method uses a domain-expert annotator to exhaustively identify atomic units of meaning in the summaries, known as summary content units (scus), and mark their occurrences in the reference and candidate summaries. two phrases marked with the same scu are considered to express the same information. since the pyramid method annotation is exhaustive, we can assume that any two phrases in the reference and candidate summaries that are not marked with the same scu do not have the same meaning. these annotated phrases can be used to reason about rouge and bertscore: if a large proportion of their token alignments is between phrases that express the same information, then their scores can potentially be interpreted as representing the summaries' information overlap. otherwise, it is evidence that they do not compare summaries based on information. for this analysis, we use the summaries and pyramid annotations from the tac 2008 and 2009 english multi-document summarization datasets (dang and @xcite . tac 2008 has 48 document clusters and 58 system summaries, and tac 2009 has 44 clusters and summaries from 55 systems. all clusters have around 10 documents each and 4 reference summaries, and every summary has been been annotated with scus. for each of the system summaries, we calculate the proportion of the total alignment weight 0% 10% 20% 30% 40% 50% that can be explained by matches between identical scus, as defined in eqs. 4 and 5: a @xmath2, j, w) : (i, j, w) ∈ a, ## category-based analysis the second analysis of rouge and bertscore focuses on grouping token alignments into categories ( §5.1), then using those categories to reason about how much of the metrics' scores is explained by information or topic matches ( §5.2). ## other evaluation metrics the analyses thus far have exploited the structure of rouge and bertscore to reason about the extent to which they measure information overlap between two summaries. although it is desirable to ask the same question about other evaluation metrics, the metrics may not directly fit into this analysis framework or it would require significant effort to repeat this analysis for each one. instead, we indirectly reason about how much information overlap other metrics measure through their correlations to rouge and the pyramid score. 3 although there are versions of rouge that remove stopwords, including them is significantly more common, and therefore we analyze the more popular rouge variant. 4 the numbers in first, we assume that the pyramid score is the gold-standard for measuring the information overlap between summaries. this is a relatively safe assumption because the pyramid method is annotated by domain experts, and a candidate's pyramid score is based solely on how much information it has in common with a reference. there is no credit given to a candidate for discussing the right topics but with the incorrect information. then, the correlations of the other metrics to both rouge and the pyramid score are calculated and compared. if the correlation to rouge is much higher than the correlation to the pyramid score, then it is more likely that the metric suffers from the same issues that rouge does than it is to directly measure information overlap. notably, most of the metrics' pearson correlations to rouge are much higher than to the pyramid score by around 0.2 points, suggesting these metrics do not measure information overlap well. further, their correlations to the pyramid score are roughly the same as rouge's, around 0.6. this means that these metrics correlate to a direct measure of information overlap as well as one would expect a metric which measures information overlap at the level of rouge to correlate. although the results of this experiment are not direct evidence that many of the other evaluation metrics do a poor job at measuring information overlap, they do strongly suggest it. one metric, however, does appear to behave differently from the others. the difference between the correlations for qaeval-f 1 is only 0.02, the smallest among the metrics. in the next section, we run a preliminary analysis of qaeval to understand if this recently proposed metric behaves different than rouge and bertscore. ## examining qaeval among the metrics analyzed in §6, qaeval stands out not only because it is nearly as similar to rouge as it is to the pyramid score, but also because its approach, evaluating summaries using qa, is rather different than the other methods, which largely rely on measuring textual overlap. in this section, we briefly examine qaeval and demonstrate evidence that it addresses some of the shortcomings of rouge and bertscore. qaeval is a reference-based evaluation metric proposed by @xcite that uses qa to evaluate a summary. the metric automatically generates a wh-question for every noun phrase in the reference summary, then predicts an answer span in the candidate summary for each question if the qa model predicts an answer exists. the score of the metric is the proportion of those questions which are answered correctly, as evaluated by the squad f 1 string similarity @xcite . the mapping between the noun phrase in the reference summary and the predicted span in the candidate summary for the corresponding question facilitates viewing qaeval as inducing an alignment between the two summaries. the final metric score is the total weight of the alignment, where the weight of an edge is the f 1 score between the two spans, normalized by the number of questions. since qaeval can be viewed as a weighted alignment, we are able to repeat the analysis from §4 for qaeval and measure what proportion of its score can be explained by matches between scus. 5 the distribution of the proportion of qaeval-f 1 that can be explained by scu matches on tac 2008 is shown in fig. 6 . the average proportion, 42%, is higher than rouge (25%) and bertscore (15%), indicating qaeval captures information similarity better than the other two metrics. the table in fig. 6 summarizes this distribution for qaeval, rouge, and bertscore (fig. 3 ), demonstrating that 46% of summaries have at least 50% of their qaeval score explained by scu matches, whereas this is true for less than 4% and 0% of rouge and bertscore. overall, this experiment provides preliminary evidence that qaeval is indeed measuring information quality more than either rouge or bertscore, and suggests that qa-based evaluation metrics are an exciting direction for future research. ## discussion responsiveness correlations many of the automatic metrics analyzed in this work have demonstrated very high system-level correlations to ground-truth summary responsiveness judgments (pearson's @xmath3.8; dang and @xcite , so the results that indicate they do not measure information overlap are somewhat contradictory. since the metrics appear to compare summaries based on the topics they discuss, it is likely that only comparing summary topics is a very strong baseline for these benchmark datasets. indeed, we find in limitations there are some limitations to our analysis. first, the results are specific to the datasets and summarization models that were used. however, tac'08 and '09 are the benchmark datasets for evaluating content quality and have been widely used to measure the performance of different metrics. further, because the results from §5.2 are consistent across two rather different datasets (tac and cnn/dailymail), we believe these results are likely to hold for other datasets. then, the predicate-argument based information categories from §5.2 do not capture all of the information from a summary. a phrase like "the turkish journalist" expresses the nationality of the journalist, but this information would not be represented by the tuples included in our analysis. however, we do not believe the addition of more tuples that express information outside of predicate-argument relationships would significantly change the experimental results. ## related work most of the work that reasons about how to interpret the scores of evaluation metrics does so indirectly through correlations to human judgments @xcite . however, a high correlation is not conclusive evidence about what a metric measures since it is possible for the metric to directly measure some other aspect of a summary, which is in turn correlated with the ground-truth judgments (see §8). our work can be viewed as more direct evidence about what rouge and bertscore measure. recent work by @xcite argues that many of the same evaluation metrics covered in this work do not successfully measure the faithfulness of a summary based on low correlations to groundtruth judgments. the results from our experiments offer an explanation for why this is the case: the metrics do not compare summaries based on their information, therefore they cannot determine if a summary is factually consistent with its input. metrics which do attempt to directly measure information overlap between summaries are based on the gold-standard comparison technique, the pyramid method @xcite . although it relies heavily on annotations by experts, there have been attempts to crowdsource @xcite or automate all or parts of the pyramid method @xcite @xcite including pyreval @xcite , which we analyzed in §6. these metrics have been met with less success than the text overlap-based ones covered by this work, potentially because measuring information overlap is more difficult than comparing summaries by their topics, and topic-based evaluations strongly correlate to responsiveness judgments (see §8). ## conclusion in this work, we argued that rouge, bertscore, and many other proposed metrics for evaluating the content quality of summaries largely do not compare summaries based on their information overlap. the implications of this result are that the summarization community does not have a reliable metric that aligns with its research goal, to generate summaries with high-quality information. however, we demonstrate that qaeval does appear to measure information overlap better than text overlap-based metrics, highlighting qa-based evaluations as a promising direction for future research.