Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary (2021)

Abstract: A desirable property of a reference-based evaluation metric that measures the content quality of a summary is that it should estimate how much information that summary has in common with a reference. Traditional text overlap based metrics such as ROUGE fail to achieve this because they are limited to matching tokens, either lexically or via embeddings. In this work, we propose a metric to evaluate the content quality of a summary using question-answering (QA). QA-based methods directly measure a summary’s information overlap with a reference, making them fundamentally different than text overlap metrics. We demonstrate the experimental benefits of QA-based metrics through an analysis of our proposed metric, QAEval. QAEval outperforms current state-of-the-art metrics on most evaluations using benchmark datasets, while being competitive on others due to limitations of state-of-the-art models. Through a careful analysis of each component of QAEval, we identify its performance bottlenecks and estimate that its potential upper-bound performance surpasses all other automatic metrics, approaching that of the gold-standard Pyramid Method.1

## introduction evaluating the content quality of a summary is a fundamental task of text summarization. as such, it has received the attention of researchers for the past two decades @xcite @xcite @xcite . the most popular approaches are reference-based metrics, which treat a human-written reference summary as the gold standard and score a candidate summary based on how similar its content is to the reference. it is desirable to have reference-based evaluation metrics that calculate this similarity score based on how much information the two summaries have in common. the vast majority of previous automatic evaluation metrics compare two summaries based on matching their tokens, either through some lexical @xcite @xcite or embedding-based similarity @xcite . although they capture a valuable quality signal, these methods match tokens that do not express the same information and instead end up comparing the similarity of two summaries based on the topics they discuss @xcite . in this work, we propose a metric to evaluate the content quality of a summary using questionanswering (qa). metrics within a qa evaluation framework represent the information of a reference summary using qa pairs, then estimate how much of this information is contained in a candidate summary by calculating the proportion of questions it can answer. because the questions can only be answered if the candidate summary contains the corresponding information, qa-based metrics directly measure the information overlap, providing a summary quality signal that is not effectively captured by text overlap based metrics. we build upon previous work in this direction @xcite and propose and analyze a more general qa-based metric, which we call qaeval ( §3.1). we experimentally show the benefit of qaeval, both with current state-of-the-art methods and by estimating its potential upper-bound performance. we show that with current question-generation and question-answering models, qaeval achieves state-of-the-art correlations to human judgments on benchmark datasets when used to evaluate summarization systems (by averaging scores over dozens of summaries), outperforming all other automatic metrics and equalling the gold-standard pyramid method @xcite . when used to rank individual summaries, the metric is equal to or better than other metrics on summaries that are very similar to the ground-truth and is competitive on others due to shortcomings of current state-of-the-art models ( §7). through a careful analysis of each component of qaeval ( §5- §7), we identified 2 performance bottlenecks: (1) the qa model and (2) the task of verifying if the predicted answer is correct ( §7), whose noise likely explains the lower summarylevel performance in some scenarios. based on a manually annotated set of 2.9k qa pairs, we show that with human-level qa and answer verification performance, the summary-level upper-bound correlations of qaeval are better than all other automatic metrics and approach the gold-standard pyramid method. in combination with state-ofthe-art correlation results, this strongly indicates that qa-based evaluation metrics are a promising direction for future research. the contributions of this work include (1) a proposal of qaeval, a more general qa-based metric for evaluating the content of summaries, (2) experimental evidence that demonstrates qaeval's state-of-the-art performance on benchmark datasets, (3) an analysis that identifies the qa model and answer verification as the performance bottlenecks, and (4) an estimate that qaeval's upperbound summary-level performance in scenarios in which it currently lags behind is high, approaching that of the gold-standard manual evaluation metric, the pyramid method. because these text overlap metrics do nothing to specifically measure how much information is common between two summaries, their scores are polluted by spurious matches between tokens that do not express the same information. in contrast, qa-based evaluation metrics do directly compare summaries based on their information. the gold-standard for manually comparing two summaries' information overlap is the pyramid method @xcite . it uses a domain-expert to identify spans of text between the candidate and reference summaries that express the same information, known as summary content units (scus). because the pyramid method's final score is calculated exclusively on the number of common scus, it is a purely information-based evaluation. while there have been efforts to crowd source the pyramid method @xcite , fully automatic approximations peak @xcite and pyreval @xcite have also been proposed, with pyreval reporting the best performance. pyreval identifies and matches scus by decomposing sentences into clauses, then calculating the similarity of the clauses based on their phrase embeddings. this style of metric has been met with less success than text overlap metrics. several recent works also use qa to evaluate summaries. @xcite use qa as part of a human evaluation to measure how much important document information was maintained by the summary. feqa @xcite and qags @xcite automate evaluating the faithfulness of a summary. faithfulness and content quality are related, yet distinct, concepts. quality is a measure of whether the summary contains the correct information, whereas faithfulness measures whether the information is consistent with the input, regardless of its importance. feqa and qags compare summaries to the input documents, whereas we compare summaries to references. because the datasets used in our experiments are extractive summaries or have relatively high faithfulness ratings @xcite , we assume faithfulness is not an issue for simplicity. then, the most closely related work to ours is @xcite , who also use qa to evaluate the content of summaries via their metric apes. they create fill-in-the-blank questions by removing named entities from the reference summary and use a reading comprehension model to predict which entity was removed using the candidate summary. there are several differences between their work and ours. our proposed metric qaeval is more general than apes because qaeval asks and answers questions about noun phrases, whereas apes is restricted to named entities. apes may fail to accurately score summaries that do not have a sufficient number of named entities. then, our evaluation of qaeval is more comprehensive: the experiments in @xcite were limited to evaluating apes on 8 input instances from tac'11, 2 whereas our experiments are run on 92 instances from benchmark content quality datasets tac'08 and '09 as well as 100 instances from the cnn/dailymail dataset @xcite . since our evaluation is more comprehensive and we demonstrate our metric has a high upper-bound performance, we believe it is a more convincing argument that qa-based metrics are a promising direction of future research. further, we perform an extensive evaluation on the individual components of the metric. we compare our metric's performance to apes' in §8 and §9. ## qa-based evaluation the standard line of research for evaluating the content quality of a summary is based on comparing the text of a candidate summary to a reference summary. metrics that follow this approach include rouge, basic elements @xcite , autosummeng @xcite , meteor (denkowski and lavie, 2014), bertscore (zhang et al., 2020), moverscore @xcite , and many more. it is desirable to evaluate a summary based on the quality of the summary's information. for reference-based metrics, this means measuring the overlap in information between the candidate and reference summary. however, there is evidence that suggests text overlap metrics do not successfully accomplish this @xcite . they match tokens which do not express the same information and end up comparing the similarity of two summaries based on the topics they discuss. we argue that a much better method of comparing the information content of two summaries is through qa. in an ideal qa-based evaluation framework, all of the reference summary's information is represented by a set of qa pairs, and the candidate summary's recall of this information is measured by answering the questions against the candidate. the questions should only be answerable if the information necessary to answer them is present in the candidate. therefore, this approach is fundamentally different from text overlap methods because it explicitly measures how much of the reference's information is contained in the candidate. while we cannot yet achieve this ideal qabased metric (our qa-based representations may be incomplete, our qa models are imperfect, etc.), we next propose a specific instantiation of this framework that represents our best effort at reaching this goal with today's state-of-the-art models. ## experimental methodology we briefly review the experimental methodology that is used to evaluate metrics. evaluation metrics are used to estimate some property of a summary that is difficult to directly measure, such as the quality of its content. in order to estimate how well the metric approximates the desired property (i.e., evaluate the evaluation metric), a set of summaries that have been annotated by human judges for that property is scored by the metric, and then the correlation between the two sets of scores is calculated. the summaries are typically the outputs from multiple summarization models for the same set of inputs. there are two standard ways to calculate correlations in the summarization literature: summarylevel and system-level. assume x j i and y j i are two scores of metrics x and y for the summary output by system @xmath0, . . . , n} on input @xmath1, . . . , m}. the summary-level correlation is calculated between the scores for each summary for the same input, then averaged across inputs: it measures how similarly x and y rank summarization systems. in this work, we examine how well evaluation metrics estimate the content quality of a summary using three english summarization datasets: the benchmark tac'08 and '09 datasets @xcite as well as the subset of the cnn/dm dataset @xcite that was annotated by @xcite . the tac datasets consist of 48/44 multidocument summarization instances, each with 4 reference summaries written by human annotators. domain-expert judges rated the summaries output by 58/55 extractive models for each input on a scale of 1 to 5 based on how well they respond to an information need included in the task description. each summary is also assigned a pyramid score @xcite using a pyramid constructed from the 4 reference summaries. our experiments on tac calculate the correlations of the metrics to the responsiveness score for the 58/55 model summarizers and 48/44 instances. the annotations provided by fabbri et al. ( 2020 ) on the single-document summarization cnn/dm dataset score the outputs of 16 models across 100 instances. the models are a mixture of extractive and abstractive approaches, and each instance has 1 reference summary. @xcite collected relevance scores from 3 expert annotators that captures if the summary contains important content from the input document. our experiments report the correlation between the metrics' scores and the expert relevance judgments. ## answer selection in order for a qa-based evaluation metric to be successful, the qa pairs it uses to probe the candidate summary must represent a significant proportion of the reference summary's information. therefore, in this section, we aim to understand how much information the qa pairs in qaeval do represent and whether that may limit the metric's performance. we explore three different answer selection strategies which pick phrases that are (1) namedentities, (2) noun phrase chunks, (3) or maximally sized noun phrases. the maximally-sized noun phrases in a sentence are identified by traversing the dependency tree down from the root until a noun is reached, then selecting the entire subtree for that noun. example answers selected by each strategy are presented in figure 1 . since there is no well-established method of measuring how much semantic content is represented by a set of qa pairs, we instead compare the content covered by the qa pairs to that of another semantic representation, the pyramid method scus (see §2 for details). this approach allows us to compare answer selection strategies to a common point of reference as well as understand what types of information are represented by each formalism. in order to compare the content covered by qa pairs and scus, each qa pair is manually mapped to an scu based on whether the information that is being probed by the qa pair is included in the scu description. for instance, in figure 1 , qa 1 and qa 2 map to scu 4 because they target what was attacked, which is included in the scu description, whereas qa 3 would not because the scu does not describe the location of the attacks. this mapping allows us to calculate the proportion of qa pairs that map to some scu, called the qa precision, and the proportion of scus that are mapped to by some qa pair, called the scu coverage. to ensure that the generated questions are of high-quality, one of the authors manually wrote questions for every answer selected by each strategy for 20 reference summaries across 10 input document sets from tac'08, totaling 801 questions. the same author further mapped every qa pair to scus. the results (averaged over reference summary) are presented in the most significant result we find is that the np chunks strategy covers 91% of the semantic strategy avg #qas qa precision scu coverage ner 11.7 83% 57% np chunks 28.8 79% 91% max. nps 17.3 82% 77% comparing qa pairs and scus upon comparing the information that is represented by one formalism and not the other, there are some key differences. the qa pairs miss information represented by nominal and adjectival modifiers because that information is contained within the answer noun phrase. for instance, for sentence [a turkish novelist] was arrested, the question asks about who was arrested, and not about the nationality of the novelist, which the scus do include. in contrast, the scus often miss specific details and generalize over information that the qa pairs do not. for instance, in figure 1 , although the scus do represent that the church attacks happened, it does not include information about their location, whereas this information is targeted by the qas pairs. ## question generation an ideal question generation model should generate questions that are high enough quality that they do not impact the overall performance of the metric. in this section, we compare questions generated by the learned model to expert-written questions, both empirically and extrinsically through downstream correlations to human judgments. ## question answering and verification the task of the qa model and answer verification step are to determine whether a question is answerable against a summary, predict an answer if it is, then compare the prediction to the ground-truth answer to determine if it is correct. in this section, we evaluate the performance of both components on the summarization data, first by calculating the qa performance ( §7.1) and then by estimating the downstream correlation of qaeval if both components had human-level performance ( §7.2). ## overall metric analysis after analyzing each component of qaeval, we now turn to calculate the metric's correlations in addition to the qaeval correlations, we also report those of several baselines and stateof-the-art metrics, including the pyramid score, several variants of rouge, pyreval, and mover-score (which reports better correlations than bertscore), and apes. see §2 for descriptions of these metrics. results in bold are the highest among the automatic metrics. those underlined are statistically indistinguishable from the highest under a single-tailed permutation test for correlations with α = 0.05 @xcite . tac'08 and '09 the correlations for tac are presented in however, the system-level results are quite surprising. the qaeval metrics achieve stateof-the-art system-level performance on nearly every correlation coefficient across both datasets, reaching correlations comparable to the pyramid method itself. for instance, on tac'08, qaeval-em has a kendall's τ of 0.76 compared to 0.70 for the pyramid method and 0.67 for the next-highest automatic metric, rouge-2. this pattern largely holds for tac'09, with the exception of pearson's r due to an outlier. 4 it is unexpected that qaeval should achieve both state-of-the-art system-level results and lower summary-level results simultaneously and that the system-level results are even better than the pyramid method's. we believe the discrepancy between the summary-and system-level results can be explained by the number of questions that is used by each evaluation. qaeval estimates the quality of an individual summary using around 110 questions. in contrast, the system-level scores are based over 5,000 qa pairs across 48 or 44 fabbri et al. (2020) metric system-level summary-level r ρ τ r ρ τ rouge-1 .61 .62 .50 .28 .26 .20 rouge-2 .64 .60 .43 .23 .19 .14 rouge-l .61 .48 .32 .21 .18 .14 rouge-su4 .62 .56 .38 .23 .19 .15 moverscore .56 .54 .42 .28 .24 .18 apes .68 .73 .58 .10 .09 .07 qaeval-em .80 .91 .77 .23 .23 .19 qaeval-f 1 .82 .91 .77 .30 .29 .22 cnn/dm the results on the cnn/dm dataset are shown in we hypothesize that the improved performance on cnn/dm compared to tac is due to the qa model's quality on this dataset. in §7.1, we demonstrated that the qa performance did drop on cnn/dm with respect to the model's results on the squad data, however that performance decrease was not nearly as large on cnn/dm as on tac. since the qa model and answer verification are the performance bottlenecks and both suffer less on cnn/dm, the qaeval metrics achieve strong correlations. this result is evidence to support that qaeval is a very effective metric for evaluating current state-of-the-art systems on today's popular summarization datasets. comparison to apes across all three datasets, qaeval achieves higher or comparable correlations than the other qa-based metric, apes, at both the summary-and system-levels. we suspect this is due to at least two reasons. first, their reading comprehension model likely has lower performance than the qa model used in qaeval. the qaeval pretrained model leverages recent state-of-the-art models that use contextual word embeddings, which the model of @xcite does not use. second, apes targets named entities in the summaries, which we demonstrated does not probe for as much information as using all noun phrases ( §5). if the summaries do not contain a sufficient number of entities, apes may fail to accurately score it. overall since the performance of qaeval using em and f 1 is roughly equal at the system-level, but f 1 is clearly better at the summary-level, we recommend that future work that evaluates with qaeval use the f 1 variant. overall, since evaluation metrics are most commonly used in the summarization community to rank summarization systems, these experimental results suggest that qaeval is one of the most effective evaluation metrics to date. ## apes experiments to further compare qaeval to apes, we reproduce some of the experiments reported by @xcite and compare the results of the two metrics. ## conclusion in this work, we proposed a qa-based evaluation metric called qaeval. we demonstrated that qaeval already achieves state-of-the-art systemlevel correlations, and we estimate its upper-bound summary-level performance on multi-document summaries is quite high. through a careful analysis of each component of qaeval, we identified that the performance bottlenecks are both the qa model and verifying whether or not the qa model's predicted answer is correct. we believe that these results are strong evidence that qabased evaluation metrics are a promising direction for future research on summarization evaluation.