Asking and Answering Questions to Evaluate the Factual Consistency of Summaries (2020)

Abstract: Practical applications of abstractive summarization models are limited by frequent factual inconsistencies with respect to their input. Existing automatic evaluation metrics for summarization are largely insensitive to such errors. We propose QAGS (pronounced “kags”), an automatic evaluation protocol that is designed to identify factual inconsistencies in a generated summary. QAGS is based on the intuition that if we ask questions about a summary and its source, we will receive similar answers if the summary is factually consistent with the source. To evaluate QAGS, we collect human judgments of factual consistency on model-generated summaries for the CNN/DailyMail (Hermann et al., 2015) and XSUM (Narayan et al., 2018) summarization datasets. QAGS has substantially higher correlations with these judgments than other automatic evaluation metrics. Also, QAGS offers a natural form of interpretability: The answers and questions generated while computing QAGS indicate which tokens of a summary are inconsistent and why. We believe QAGS is a promising tool in automatically generating usable and factually consistent text. Code for QAGS will be available at https://github.com/W4ngatang/qags .

## introduction automatic summarization aims to produce summaries that are succinct, coherent, relevant, andcrucially -factually correct. recent progress in conditional text generation has led to models that can generate fluent, topical summaries @xcite . however, model-generated summaries frequently contain factual inconsistencies, limiting their applicability @xcite . the problem of factual inconsistency is due in part to the lack of automatic evaluation metrics that can detect such errors. standard metrics for 1 pronounced "kags". we argue that evaluation metrics that are able to capture subtle semantic errors are required to build better models. in this work, we introduce a general framework for evaluating conditional text generation that is designed to detect factual inconsistencies in generated text with respect to some input. our framework consists of three steps: (1) given a generated text, a question generation (qg) model generates a set of questions about the text. this approach leverages recent progress in qa and qg to ask and answer human readable, ontopic questions @xcite . it only assumes access to a question answering dataset to train the qg and qa models, and is applicable to any modality where a qa model is available, e.g. text, images, or knowledge graphs. we use this framework to develop qags (question answering and generation for summarization), a metric for evaluating the factual consistency of abstractive document summaries. compared to commonly used automatic metrics such as rouge @xcite , qags shows dramatically higher correlations with human judgements of factuality, for example achieving a pearson correlation coefficient of 54.52 on the cnn/dailymail summarization task, compared to 17.72 for rouge-2. qags also achieves new state-of-the-art results on evaluating the factuality of summaries, outper-forming recently proposed nli models for this task @xcite . finally, we analyse the robustness of qags through an ablation study. qags shows robustness to the quality of the underlying qg and qa models, the domain of the models, and the number of questions asked. even under the worst ablation settings, qags still has stronger correlation with human judgments than other automatic metrics. overall, we contribute the following: (1) we introduce qags, an automatic model-based evaluation metric for measuring the factual consistency of model-generated text. (2) we collect a new set of human judgments of factual consistency of model-generated summaries for two summarization datasets. we demonstrate that qags correlates with these judgments significantly better than other automatic metrics. (3) we show via ablations that qags is robust to a number of factors including underlying model quality and domain mismatch. ( 4 ) we analyze the questions and answers produced in computing qags to illustrate which parts of summaries are inconsistent. (5) we will release models and code to compute qags. ## background: automatically evaluating machine generated text standard approaches to evaluating generated text are primarily based on counting n-gram overlap. these methods assume access to one or more reference texts, and score a generated summary based on the precision and recall of all reference n-grams in the generated summary. we briefly describe the most common metrics in this family, and refer readers to @xcite for further discussion. rouge @xcite was developed specifically for evaluating automatic summarization, and its variants are the de facto standard for such. the most common variant is rouge-n (typically @xmath0, 2}), which computes the f1 score for all reference n-grams in the generated summary. rouge-l, another commonly used variant, is the length of the longest common subsequence (possibly nonconsecutive) between a summary and references. bleu @xcite is closely related to rouge but was developed for machine translation. bleu computes the precision of the reference ngrams in the generated summary. meteor @xcite extends bleu by using an alignment between the generated text and a reference, as well as using stemming and synonym replacement for more flexible n-gram matching. we identify two key deficiencies when using these n-gram based evaluation metrics to detect factual inconsistencies in generated text. first, these metrics require one or more reference texts to compare against. obtaining references can be expensive and challenging, and as such many text generation datasets contain only a single reference. this problem is exacerbated with highentropy generation tasks, such as summarization or dialogue, where there is a very large number of acceptable outputs. in these settings, comparing against a single reference is woefully inadequate. second, given a reference to compare against, n-gram based approach weigh all portions of the text equally, even when only a small fraction of the n-grams carry most of the semantic content. factual inconsistencies caused by minor changes may be drowned out by otherwise high n-gram overlap, making these metrics insensitive to these errors. for example, the sentences "i am writing my paper in vancouver." and "i am not writing my paper in vancouver." share nearly all unigrams and bigrams despite having the opposite meaning. ## a framework for automatically evaluating factual consistency we introduce a framework for automatically detecting factual inconsistencies in generated text while also addressing the deficiencies of current approaches. let x and y be sequences of tokens coming from a vocabulary v where x is a source text and y is a summary of x. we define p(q|y ) as a distribution over all possible questions q given summary y , and p(a|q, x) and p(a|q, y ) as distributions over all possible answers a to a particular question q given either the source x or the summary y . we constrain the questions q and answers a to also be sequences of tokens from v . then the factual consistency of the summary y is ## qags using this framework requires specifying the question distribution p(q|y ), the answer distributions p(a|q, * ), and the answer similarity function d. we apply this framework to summarization to de-velop qags and describe our instantiations of these components. question generation to instantiate p(q|y ), we draw on recent work on automatic question generation (qg), which models this distribution using neural seq2seq models @xcite . we over-sample questions, and then filter out low quality questions as follows. first, we train and generate from answerconditional qg models. during training, the model receives both the answer and the source article, and is trained to maximize the likelihood of the paired question. at test time, given a summary y , we determine candidate answers. we condition on these answers and the summary to generate questions. next, we filter out low-quality questions using a number of heuristics, such as duplicates and questions less than three tokens long. we also found it especially useful to run the qa model (see next section) on all of the candidate questions, and filter out questions for which the qa model predicted no answer or a different answer than expected. ## qualitative analysis interpreting qags the questions and answers produced in computing qags are directly interpretable, and highlight errors in summaries. we present examples of articles, summaries, and the qags questions and answers in on the first example ( the second example ( error analysis the interpretability of qags allows for error analysis on the metric. we manually annotate 400 triplets of generated questions, article answers, and summary answers that are produced in computing qags on the xsum summaries, and label them by the quality of the generated questions, predicted answers, and answer similarity scores. among the generated questions, 8.75% are nonsensical, while 3.00% are well-formed but unanswerable using the generated summary they were conditioned upon. these figures indicate that the vast majority of questions are understandable and on-topic. we frequently observe multiple questions with slightly different wordings, which is likely due to the low number of answer candidates in xsum summaries (which are one sentence long) and due to beam search. 8.25% of questions are well-formed but unanswerable using the source, which is usually due to a hallucinated fact in the summary that the qg model turns into a question. among predicted answers, 1.75% of questions are potentially answerable using the summary, but are incorrectly answered. this percentage increases to 32.50% for the article, which indicates that the transfer ability of the qa model is lacking. in a small number of cases, we found that while a question had a single answer in the summary, it could have multiple answers in the article. finally, for 8.00% of the examples, the question is answered correctly using both the article and summary, but the answers have high lexical variation such that f1 score fails to detect their similarity. while this happens in a relatively small number of cases, exploring similarity metrics other than n-gram based approaches could be useful. limitations we emphasize that qags and our overall framework are specifically designed to detect factual inconsistencies in generated summaries relative to the source article. qags does not measure other desirable properties of generated text, the choices of qg and qa models in qags are particular to abstractive summarization and may require adaptation to be used for other conditional text generation tasks. for example, we expect that extractive summarization models may obtain nearly perfect qags scores because facts and statements are directly copied from the source article. ## related work automatic summarization and its evaluation are long-standing lines of work in nlp, dating at least as far back as the document understanding conferences @xcite . the primary evaluation metric then and now is rouge @xcite , though much work has demonstrated the limited ability of rouge and its relatives to evaluate summaries @xcite @xcite . other metrics have focused on specific aspects of summarization quality, including content selection @xcite , relevance prediction @xcite , and many more. the idea of evaluating summaries by their ability to answer a set of questions is also long-standing @xcite . like our work, @xcite and @xcite extend this line of work by incorporating neural network modules. we diverge from these works in two important ways. first, both works use cloze-style questions, which are generated by masking entities in either the source document or the reference summary. we instead generate the questions with a model, allowing a much greater range of questions. second, we produce questions conditioned on the generated summary, rather than the reference summary or source article. producing questions from the generated summary is more appropriate for verifying the accuracy of the text, whereas using the reference or source measures content selection. there has been a recent resurgence of work leveraging nlu models for evaluating the factuality of generated text. @xcite use information extraction models to measure factual overlap, but facts are restricted to pre-defined schemas. @xcite investigate the use of nli models to evaluate the factual correctness of cnn/dm summaries, and conclude that current nli models are too brittle to be reliably used in this manner. @xcite train a nli-based factchecking model by building a dataset of factual inconsistencies based on noise heuristics. our qa approach allows a finer-grained analysis, because nli operates on complete sentences, whereas qags can ask many different questions about the same sentence. ## conclusion we introduce a framework for automatically detecting factual inconsistencies in conditionally generated texts and use this framework to develop qags, a metric for measuring inconsistencies in abstractive summarization. qags correlates with human judgments of factuality significantly better than standard automatic evaluation metrics for summarization, and outperforms related nli-based approaches to factual consistency checking. qags is naturally interpretable: the questions and answers produced in computing qags indicate which tokens in a generated summary are inconsistent and why. the framework we present is general, and extending it to other conditional text generation tasks such as image captioning or machine translation is a promising directions. inspecting the generated questions and answers, we identify the transfer ability of qa models and the rigidity of f1 score as a measure of answer similarity as two key performance bottlenecks. we expect improvements in either would straightforwardly improve the quality of qags evaluation. additionally, incorporating a content selection mechanism to focus the generated questions on salient facts is a promising direction. overall, we believe qags demonstrates the potential of this framework to quantify and incentivize factually consistent text generation.