Improving Factual Consistency in Summarization with Compression-Based Post-Editing (2022)

Abstract: State-of-the-art summarization models still struggle to be factually consistent with the input text. A model-agnostic way to address this problem is post-editing the generated summaries. However, existing approaches typically fail to remove entity errors if a suitable input entity replacement is not available or may insert erroneous content. In our work, we focus on removing extrinsic entity errors, or entities not in the source, to improve consistency while retaining the summary’s essential information and form. We propose to use sentence-compression data to train the post-editing model to take a summary with extrinsic entity errors marked with special tokens and output a compressed, well-formed summary with those errors removed. We show that this model improves factual consistency while maintaining ROUGE, improving entity precision by up to 30% on XSum, and that this model can be applied on top of another post-editor, improving entity precision by up to a total of 38%. We perform an extensive comparison of post-editing approaches that demonstrate trade-offs between factual consistency, informativeness, and grammaticality, and we analyze settings where post-editors show the largest improvements.

## introduction text summarization aims to compress a long document(s) into a short and fluent form that preserves salient information. state-of-the-art models, however, are often not factually consistent with the input they are conditioned on @xcite . while recent modeling techniques have been proposed to improve factual consistency @xcite @xcite , a model-agnostic approach is to post-edit the summaries. prior work in post-editing for factual consistency has focused on swapping inconsistent entities with those in the input @xcite , including by reranking the entity-replaced source: the oil price collapse sent global markets reeling throughout 2015. ... brent crude oil was up 3% at $37.60 per barrel for the day but down 35% over the year. ## conclusion in this work, we propose a sentence-compressionbased post-editing model to improve factual consistency in summarization while maintaining the informativeness and grammaticality of the resulting summaries. we show that this model can be used in tandem with a post-editor that performs extensive rewriting for further improvement, especially in pretrained models and datasets with a high proportion of entity errors. in future work, we plan to build on these models by studying the role of dataset artifacts in error correction and addressing unfixable summaries.