Multi-Fact Correction in Abstractive Text Summarization (2020)

Abstract: Pre-trained neural abstractive summarization systems have dominated extractive strategies on news summarization performance, at least in terms of ROUGE. However, system-generated abstractive summaries often face the pitfall of factual inconsistency: generating incorrect facts with respect to the source text. To address this challenge, we propose Span-Fact, a suite of two factual correction models that leverages knowledge learned from question answering models to make corrections in system-generated summaries via span selection. Our models employ single or multi-masking strategies to either iteratively or auto-regressively replace entities in order to ensure semantic consistency w.r.t. the source text, while retaining the syntactic structure of summaries generated by abstractive summarization models. Experiments show that our models significantly boost the factual consistency of system-generated summaries without sacrificing summary quality in terms of both automatic metrics and human evaluation.

## introduction informative text summarization aims to shorten a long piece of text while preserving its main message. existing systems can be divided into two main types: extractive and abstractive. extractive strategies directly copy text snippets from the source to form summaries, while abstractive strategies generate summaries containing novel sentences not found in the source. despite the fact that extractive strategies are simpler and less expensive, and can generate summaries that are more grammatically and semantically correct, abstractive strategies are becoming increasingly popular thanks to its flexibility, coherency and vocabulary diversity @xcite . * *most of this work was done when the first author was an intern at microsoft. cnndm source (cnn) about a quarter of a million australian homes and businesses have no power after a "once in a decade" storm battered sydney and nearby areas. about 4,500 people have been isolated by flood waters as "the roads are cut off and we won't be able to reach them for a few days,"... bottom-up summary a quarter of a million australian homes and businesses have no power after a decade. corrected by spanfact about a quarter of a million australian homes and businesses have no power after a "once in a decade" storm. gigaword source all the 12 victims including 8 killed and 4 injured have been identified as senior high school students of the second senior high school of ruzhou city, central china's henan province, local police said friday. pointer-generator summary 12 killed, 4 injured in central china school shooting. corrected by spanfact 8 killed, 4 injured in central china school shooting. xsum source st clare's catholic primary school in birmingham has met with equality leaders at the city council to discuss a complaint from the pupil's family. the council is supporting the school to ensure its policies are appropriate... bertabs summary a muslim school has been accused of breaching the equality act by refusing to wear headscarves. corrected by spanfact a catholic school has been accused of breaching the equality act by refusing to wear headscarves. recently, with the advent of transformer-based models @xcite pre-trained using self-supervised objectives on large text corpora @xcite @xcite , abstractive summarization models are surpassing extractive ones on automatic evaluation metrics such as rouge @xcite . however, several studies @xcite @xcite @xcite observe that despite high rouge scores, system-generated abstractive summaries are often factually inconsistent with respect to the source text. factual inconsistency is a well-known problem for conditional text generation, which requires models to generate readable text that is faithful to the input document. consequently, sequence-to-sequence generation models need to learn to balance signals between the source for faithfulness and the learned language modeling prior for fluency @xcite . the dual objectives render abstractive summarization models highly prone to hallucinating content that is factually inconsistent with the source documents @xcite . prior work has pushed the frontier of guaranteeing factual consistency in abstractive summarization systems. most focus on proposing evaluation metrics that are specific to factual consistency, as multiple human evaluations have shown that rouge or bertscore @xcite correlates poorly with faithfulness @xcite . these evaluation models range from using fact triples @xcite , textual entailment predictions @xcite , adversarially pre-trained classifiers @xcite , to question answering (qa) systems @xcite . it is worth noting that qa-based evaluation metrics show surprisingly high correlations with human judgment on factuality @xcite , indicating that qa models are robust in capturing facts that can benefit summarization tasks. on the other hand, some work focuses on model design to incorporate factual triples @xcite or textual entailment @xcite to boost factual consistency in generated summaries. such models are efficient in boosting factual scores, but often at the expense of significantly lowering rouge scores of the generated summaries. this happens because the models struggle between generating pivotal content while retaining true facts, often with an eventual propensity to sacrificing informativeness for the sake of correctness of the summary. in addition, these models inherit the backbone of generative models that suffer from hallucination despite the regularization from complex knowledge graphs or text entailment signals. in this work, we propose spanfact, a suite of two neural-based factual correctors that improve summary factual correctness without sacrificing informativeness. to ensure the retention of semantic meaning in the original documents while keeping the syntactic structures generated by advanced summarization models, we focus on factual edits on entities only, a major source of hallucinated errors in abstractive summarization systems in practice @xcite . the proposed model is inspired by the observation that fact-checking qa model is a reliable medium in assessing whether an entity should be included in a summary as a fact @xcite . to our knowledge, we are the first to adapt qa knowledge to enhance abstractive summarization. compared to sequential generation models that incorporate complex knowledge graph and nli mechanisms to boost factuality, our approach is lightweight and can be readily applied to any system-generated summaries without retraining the model. empirical results on multiple summarization datasets show that the proposed approach significantly improves summarization quality over multiple factuality measures without sacrificing rouge scores. our contributions are summarized as follows. (i) we propose spanfact, a new factual correction framework that focuses on correcting erroneous facts in generated summaries, generalizable to any summarization system. (ii) we propose two methods to solve multi-fact correction problem with single or multi-span selection in an iterative or auto-regressive manner, respectively. (iii) experimental results on multiple summarization benchmarks demonstrate that our approach can significantly improve multiple factuality measurements without a huge drop on rouge scores. ## related work the general neural-based encoder-decoder structure for abstractive summarization is first proposed by @xcite . later work improves this structure with better encoders, such as lstms @xcite and grus @xcite , that are able to capture longrange dependencies, as well as with reinforcement learning methods that directly optimize summarization evaluation scores @xcite . one drawback of the earlier neural-based summarization models is the inability to produce out-ofvocabulary words, as the model can only generate whole words based on a fixed vocabulary. @xcite proposes a pointer-generator framework that can copy words directly from the source through a pointer network @xcite , in addition to the traditional sequence-to-sequence generation model. abstractive summarization starts to shine with the advent of self-supervised algorithms, which allow deeper and more complicated neural networks such as transformers @xcite to learn diverse language priors from large-scale corpora. models such as bert @xcite , gpt @xcite and bart @xcite have achieved new state-of-the-art performances on abstractive summarization @xcite @xcite @xcite . these models often finetune pre-trained transformers with supervised summarization datasets that contain pairs of source and summary. however, encoder-decoder architectures widely used in abstractive summarization systems are inherently difficult to control and prone to hallucination @xcite @xcite , and often leads to factual inconsistency: the system-generated summary is fluent but unfaithful to the source @xcite . studies have shown that 8% to 30% system-generated abstractive summaries have factual errors @xcite that cannot be discovered by rouge scores. recent studies have proposed new methods to ensure factual consistency in summarization. @xcite ; @xcite pro-pose rnn-based and transformer-based decoders that attend to both source and extracted knowledge triples, respectively. li et al. (2018) our fact correction models are inherently different from these models, as we focus on postcorrecting summaries generated by any model. our models are trained with the objective of predicting masked entities identified for fact correction (figure 1 ), and learn to fill in the entity masks of any system-generated summaries with single or multi-span selection mechanism (figure 2 ). the most similar work to ours is proposed concurrently by @xcite , where they fine-tune a bart @xcite ) model on distant supervision examples and use it as a post-editing model for factual error correction. ## multi-fact correction models in this section, we describe two models proposed for factual error correction: (i) qa-span fact correction model, and (ii) auto-regressive fact correction model. as both methods rely on span selection with different masking and prediction strategies, we call them spanfact collectively. ## experiment in this section, we present our results on using spanfact for multiple summarization datasets. ## conclusion we present spanfact, a suite of two factual correction models that use span selection mechanisms to replace one or multiple entity masks at a time. spanfact can be used for fact correction on any abstractive summaries. empirical results show that our models improve the factuality of summaries generated by state-of-the-art abstractive summarization systems without a huge drop on rouge scores. for future work, we plan to apply our method for other type of spans, such as noun phrases, verbs, and clauses. the reviewers for their valuable comments and special thanks to yuwei fang and other members of the microsoft dynamics 365 ai research team for the feedback and suggestions.