Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling (2022)

Abstract: Abstractive summarization models often generate inconsistent summaries containing factual errors or hallucinated content. Recent works focus on correcting factual errors in generated summaries via post-editing. Such correction models are trained using adversarial non-factual summaries constructed using heuristic rules for injecting errors. However, generating non-factual summaries using heuristics often does not generalize well to actual model errors. In this work, we propose to generate hard, representative synthetic examples of non-factual summaries through infilling language models. With this data, we train a more robust fact-correction model to post-edit the summaries to improve factual consistency. Through quantitative and qualitative experiments on two popular summarization datasets— CNN/DM and XSum—we show that our approach vastly outperforms prior methods in correcting erroneous summaries. Our model—FactEdit—improves factuality scores by over ~11 points on CNN/DM and over ~31 points on XSum on average across multiple summarization models, producing more factual summaries while maintaining competitive summarization quality.

## introduction while modern summarization models generate highly fluent summaries that appear realistic @xcite , these models are prone to generating non-factual and sometimes entirely fabricated content @xcite @xcite . with the increasing adoption of language generation tools in user-facing products, such unreliability poses severe risks, including the spread of misinformation, panic and other potentially harmful effects @xcite . since it is difficult to control for factuality at training or inference time @xcite figure 1 : model generated summaries often produce content which is factually inconsistent w.r.t. to the source. factedit rewrites these summaries by maintaining the abstractiveness but correcting factual errors. @xcite , a popular approach to fix the factual inconsistencies is via post-editing generated summaries @xcite . this allows summarization models to focus on fluency and content-relevance while improving factual consistency. however, there is no suitable data for training post-editing models to directly "translate" an incorrect summary to a correct one. prior work constructed synthetic training data by introducing simple heuristic errors like replacing entities or numbers in reference summaries @xcite , but it is not clear whether such synthetic errors have sufficient coverage and accurately represent the types and distribution of actual errors made by language models. further, with increasing language generation capabilities, models make more complex factual errors involving discourse structures and paraphrasing which cannot be easily captured with heuristics @xcite . the goal of our work is to develop post-editing models vaccine for [mask] is unlikely to be ready this year. covid-19 coronavirus ebola the virus vaccine for ebola is unlikely to be ready this year. vaccine for the virus is unlikely to be ready this year. world leaders met ban ki moon for un meeting in 2020[sep] pandemic response … [sep] un sec. gen. antonio gutteres met the leaders world leaders met antonio gutteres for un meeting in 2020 input: masked ref summary (s masked ) [sep] source context (ctx) input: incorrect summary sent (sg') [sep] summary context (g') [sep] relevant source passages output: corrected summary sent (g) infilling language model (mi) seq-to-seq model (mc) infilling-based generation of adversarial summaries train decode output: beam search candidates factedit: factual error correction incorrect summary candidates (r') training data training data figure 2 : architecture framework for factedit. using masked versions of existing reference summaries, we use an infilling language model to produce alternative candidates for the mask position. we construct factually incorrect summaries by replacing the mask with the lower ranked candidates. finally, we train a sequence-to-sequence model for fact correction using the synthetically constructed data. we propose factedit-a novel approach to post-editing text, to control for content factuality in generated summaries. rather than manually defining a list of heuristic errors, it incorporates a new algorithm to generate adversarial (non-factual) examples using infilling language models @xcite . we use lower ranked beam-search candidates from the language model as a source for potentially factually-incorrect summary facts, thereby producing a set of plausible, likely, and fluent, incorrect synthetic summaries for a particular correct reference summary. in this way, we leverage the capabilities of large language models to produce multiple candidates of alternative, erroneous summaries. these examples, along with factually correct references, are then used to train a sequence-to-sequence fact-correction model that aims at generating a factually consistent version of the candidate summary ( §2). we evaluate factedit on two datasets -cnn/dailymail @xcite and xsum @xcite and across nine summarization models with the frank benchmark @xcite for evaluating various categories of factual errors in generated summaries ( §3). the two summarization datasets represent varied distributions of factual errors in models trained on them and hence constitute a good test bed to evaluate the generalizability of our model. we show that factedit substantially improves factuality scores across two metrics -ent-dae @xcite and factcc @xcite . on the ent-dae metric, factedit improves results by ∼11 points (cnn/dm) and ∼31 points (xsum), and on the factcc metric we show improvements of ∼6 points (cnn/dm) and ∼24 (xsum) points on average across models ( §4). further, our analysis shows that factedit effectively corrects diverse error categories without the need for special heuristics or annotations ( §5). an important application of factedit is to audit summarization systems and facilitate their reliability. ## model assume a summarization model trained to process a document d and generate a coherent and fluent summary 2 g ′ which has been shown to often misrepresent facts from the document. factedit is a fact correction model m c which takes the generated summary g ′ and document d, identifies factual errors and generates a rewritten summary g by correcting them (as outlined in figure 2 ). we present an adversarial data generation approach which leverages the power of pre-trained language models to produce fluent and complex factually incorrect summaries. we train an infilling language model m i using documents from summarization training data and use the model to introduce diverse factual errors in sentences from them ( §2.1). using the trained model, we introduce factual errors in reference summaries of the training data r producing an incorrect summary r ′ resulting in a synthetic dataset {r ′ , r, d} train of erroneous summaries mapped to their corrected versions (pink section in figure 2 ). we train a sequence-to-sequence model m c for factual error correction using the generated synthetic data ( §2.2). finally, we use the trained correction model to rewrite model generated summaries g ′ producing a corrected version g ( §2.3 -green section in figure 2 ). ## related work factuality evaluation standard n-gram based metrics do not correlate well with human judgements of factuality and are unsuitable for evaluating factuality @xcite . several automated metrics were proposed to detect factual errors in generated summaries. they primarily fall in two paradigms-entailment based and qa based metrics. goodrich et al. (2019); kryscinski et al. (2020); maynez et al. (2020); goyal and durrett (2021) model factuality as an entailment verifying whether the summary is entailed by the source. lee et al. (2022b) use similar masked infilling to generate training data for such metrics. qa models can be used to answer questions about the document, separately using the article and the output summary as context and compare the answers to score the factuality of summaries @xcite . to evaluate these metrics, recent work collec human judgements for factuality @xcite @xcite . additionally, @xcite also obtain annotations on factual error categories, which we use for our evaluations. this paper con-siders the problem of improving factuality, not measuring it. while this is a different task, it is related: e.g., measuring the number of corrections made by factedit might be useful as a factuality measure.