FEQA : A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization (2020)

Abstract: Neural abstractive summarization models are prone to generate content inconsistent with the source document, i.e. unfaithful. Existing automatic metrics do not capture such mistakes effectively. We tackle the problem of evaluating faithfulness of a generated summary given its source document. We first collected human annotations of faithfulness for outputs from numerous models on two datasets. We find that current models exhibit a trade-off between abstractiveness and faithfulness: outputs with less word overlap with the source document are more likely to be unfaithful. Next, we propose an automatic question answering (QA) based metric for faithfulness, FEQA, which leverages recent advances in reading comprehension. Given question-answer pairs generated from the summary, a QA model extracts answers from the document; non-matched answers indicate unfaithful information in the summary. Among metrics based on word overlap, embedding similarity, and learned language understanding models, our QA-based metric has significantly higher correlation with human faithfulness scores, especially on highly abstractive summaries.

## introduction abstractive summarization models must aggregate salient content from the source document(s) and remain faithful, i.e. being factually consistent with information in the source documents. neural abstractive models are effective at identifying salient content and producing fluent summaries @xcite @xcite . however, the generated summary may not always contain faithful information, which is vital for real-world applications. source. the world's oldest person has died a few weeks after celebrating her 117th birthday. born on march 5, 1898, the greatgrandmother had lived through two world wars, the invention of the television and the first successful powered aeroplane flight by the wright brothers... output sentence. the world 's oldest person has died on march 5, 1898. in this paper, we address the problem of evaluating faithfulness of generated summaries given their source documents. our key insight is that current models are limited by a trade-off between abstractiveness and faithfulness (section 2). on a wide range of systems and two datasets with varying levels of abstractiveness (cnn/dm and xsum), we show that the number of unfaithful sentences (annotated by humans) increases as the summary becomes more abstractive (i.e. less overlap with the source document). next, we investigate a diverse set of existing automatic evaluation metrics such as rouge, bertscore @xcite , and learned entailment models. we find that their correlations with human scores of faithfulness drop significantly on highly abstractive summaries, where deeper text understanding beyond surface similarity is needed. recently, question answering (qa) based automatic metrics have been proposed for evaluating content selection in summarization @xcite @xcite . specifically, cloze-style qa is used to evaluate whether important information in the source is recovered from the summary. inspired by prior work, we use automatically generated qa pairs to represent information in the summary and validate it against the source. concretely, we generate a set of "groundtruth" qa pairs from the summary, using a learned model that converts a declarative sentence and an answer span to a question (section 3). then, off-the-shelf reading comprehension models are evaluated on this set by extracting answer spans from the source documents. high accuracy means that the summary and the source document tend to produce the same answers, thus they are factually consistent with respect to the questions. compared to prior approaches using cloze tests, our question generation approach enables evaluation with a broader range of qa models and answer types (e.g. extractive and generative), thus maximally taking advantage of progress in qa. among automatic metrics based on n-gram overlap, word embeddings, and language understanding models (relation extraction and entailment), feqa has significantly higher correlation with human scores of faithfulness and is the only metric that correlates with human scores on highly abstractive summaries from xsum. ## the abstractiveness-faithfulness tradeoff while extractive summarizers are largely faithful (since they copy sentences from the source document), current abstractive models struggle to produce faithful summaries without copying. similar to @xcite , we observe that factual errors occur more frequently as models generate more abstractive summary sentences, i.e. less overlap with the source document. in this section, we analyze generated summaries along two dimensions: abstractiveness and faithfulness. specifically, we aim to answer the following questions: (1) how to quantify abstractiveness of a summary? (2) is abstractiveness encouraged more by the data or the model? (3) how does being abstractive affect faithfulness? ## feqa: faithfulness evaluation with question answering our analysis above shows that the number of unfaithful sentences increases significantly as more abstractive summaries are generated. thus the key challenge to faithfulness evaluation is to verify highly abstractive sentences against the source document, where surface similarity match- ## experiments we aim to understand to what extent the proposed qa-based metric and existing metrics capture faithfulness of a summary. given pairs of documents and summary sentences without reference summaries, we measure correlations between human-annotated faithfulness scores (section 2.3) and scores computed using each metric described below. ## related work problems in current neural generation models. since the beginning of neural text generation, problems with repetition and generic responses have received lots of attention @xcite @xcite . recently, more work has focused on semantic errors in model outputs, such as adequacy in machine translation @xcite , faithfulness in summarization @xcite , and consistency in dialogue @xcite . our analysis on the abstractiveness-faithfulness tradeoff reveals additional limitation of current models, and suggests that we need new inductive bias on how to summarize beyond copying. qa as a proxy. question answering is a broad format that subsumes many tasks @xcite . to the best of our knowledge, @xcite first use qa as an extrinsic evaluation for summarization: a good summary should answer key questions a reader might have about an article. later, qa is incorporated in human evaluation where one person writes questions and another person answers them based on the summary @xcite . the closest to our work are recent efforts in automating this protocol, including rule-based approaches @xcite and cloze-test qa @xcite . our work is the first to apply automated question generation. while we focus on faithfulness, our qabased metric is applicable to semantic comparison between any two pieces of text. automated evaluation for nlg. automated nlg evaluation is challenging as it often requires deep understanding of the text. although metrics based on word overlap with the reference text are commonly used, it is widely known that they do not correlate well with human judgments @xcite . recently, more work has focused on model-based evaluation using discriminators @xcite , entailment models @xcite , information extraction @xcite , and question answering @xcite . ## conclusion we investigate the faithfulness problem in neural abstractive summarization and propose a qabased metric for evaluating summary faithfulness. we show that current models suffer from an inherent trade-off between abstractiveness and faithfulness. they are good at copying important source content, but tend to concatenate unrelated spans and hallucinate details when generating more abstractive sentences. a new inductive bias or additional supervision is needed for learning reliable models. while our qa-based metric correlates better with human judgment and is useful for model development, it is limited by the quality of the qa model. the final evaluation should still rely on human annotation or human-in-the-loop methods @xcite .