Factual Error Correction for Abstractive Summarization Models (2020)

Abstract: Neural abstractive summarization systems have achieved promising progress, thanks to the availability of large-scale datasets and models pre-trained with self-supervised methods. However, ensuring the factual consistency of the generated summaries for abstractive summarization systems is a challenge. We propose a post-editing corrector module to address this issue by identifying and correcting factual errors in generated summaries. The neural corrector model is pre-trained on artificial examples that are created by applying a series of heuristic transformations on reference summaries. These transformations are inspired by the error analysis of state-of-the-art summarization model outputs. Experimental results show that our model is able to correct factual errors in summaries generated by other neural summarization models and outperforms previous models on factual consistency evaluation on the CNN/DailyMail dataset. We also find that transferring from artificial error correction to downstream settings is still very challenging.

## introduction self-supervised methods have achieved success in a wide range of nlp tasks, and automatic summarization is no exception @xcite @xcite @xcite . these state-ofthe-art abstractive summarization models typically finetune pre-trained transformer-based models on a summarization dataset @xcite . despite significant improvements over previous methods in terms of automatic evaluation scores such as rouge @xcite , ensuring factual consistency of the generated summary with respect to the source remains challenging. for example, cao source: jerusalem (cnn)the flame of remembrance burns in jerusalem, and a song of memory haunts valerie braham as it never has before. this year, israel's memorial day commemoration is for bereaved family members such as braham. "now i truly understand everyone who has lost a loved one," braham said. (...) original: france's memorial day commemoration is for bereaved family members as braham. (inconsistent) after correction: israel's memorial day commemoration is for bereaved family members as braham. (consistent) different approaches have been proposed to detect or ensure the factual consistency of generated summaries, including using fact extraction or applying attention on fact triples @xcite @xcite , applying natural language inference or question answering models for consistency checking @xcite @xcite and training the model on artificial datasets @xcite . most of these approaches either require a high-quality fact extraction model or they only focus on factual consistency evaluation. improving factuality correction by editing inconsistent parts in generated summaries is a direction that has not been explored much. in this work, we propose a model to improve the factual consistency of system summaries with post-editing correction ( the empirical results based on automatic and human evaluations indicate that our model not only corrects factual errors in summaries, it is also a better and more reliable factuality evaluation model than factcc @xcite , a recent proposed factuality evaluation method. in a downstream setting where we apply the corrector to the output of an abstractive summarizer, we find that our corrector is able to accurately correct errors in the generated summaries. however, the overall recall on correcting factual errors in real system summaries remains low, suggesting the errors introduced by heuristics have a different distribution than errors made by abstractive summarization systems. ## background and related work previous work on factual consistency in abstractive summarization can be divided into two categories: abstractive summarization models tailored towards factual consistency @xcite @xcite , and evaluation models for factual consistency in abstractive summarization @xcite @xcite . @xcite proposed a dual attention module in an abstractive summarizer that attends to both the source document and to relation triples extracted from the document. @xcite propose to improve their abstractive summarization model by optimizing fact scores defined in radiology reports with reinforcement learning methods. @xcite jointly train their model's encoder on summarization and nli tasks. @xcite train an abstractive summarization system with the auxiliary tasks of question and entailment generation and show that their generated summaries are less likely to produce extraneous facts. @xcite show that neural abstractive summarizers often assign higher posterior likelihood to perturbed contrastive summaries that are inconsistent with the source text than to human-written gold-standard ones. concurrently to our work, @xcite recently proposed a fact-aware summarization model that uses a knowledge graph. they use a pre-trained corrector module to modify generated summaries. concurrent to our work, @xcite proposes factual correction models that leverages knowledge learned from question answering models via span selection. their models employ single or multimasking strategies to either iteratively or autoregressively replace entities. in terms of evaluating abstractive summarization models for factual consistency, @xcite proposed a metric to check factual consistency by checking the overlapped fact triples between a source document and generated text on wikidata. @xcite shows that factual error detection is a difficult task on its own and adapting entailment models for factual error detection do not offer the desired performance. @xcite finetune a bert model on heuristically-created data with six types of rulebased text transformations for factual consistency checking. @xcite propose a framework for measuring inconsistencies in abstractive summarization by answering questions based on both generated summaries and documents. ## proposed approach in this section, we describe our procedure of introducing artificial errors in the datasets for training and propose our end-to-end error corrector model. ## conclusions in this paper, we proposed a novel approach to correct inconsistent content in summaries generated by abstractive summarization models. we train an end-to-end correction model with artificial examples created by corrupting reference summaries. our model achieved promising performance on our artificial test set and outperformed previous models on the manually annotated test set by wide margins. our human evaluation indicates that our model is able to correct some factually inconsistent summaries generated by abstractive summa-rization model. however, low recall on the inconsistent summaries and false positive samples remain as challenges.