Improving the Faithfulness of Abstractive Summarization via Entity Coverage Control (2022)

Abstract: Abstractive summarization systems leveraging pre-training language models have achieved superior results on benchmark datasets. However, such models have been shown to be more prone to hallucinate facts that are unfaithful to the input context. In this paper, we propose a method to remedy entity-level extrinsic hallucinations with Entity Coverage Control (ECC). We first compute entity coverage precision and prepend the corresponding control code for each training example, which implicitly guides the model to recognize faithfulness contents in the training phase. We further extend our method via intermediate fine-tuning on large but noisy data extracted from Wikipedia to unlock zero-shot summarization. We show that the proposed method leads to more faithful and salient abstractive summarization in supervised fine-tuning and zero-shot settings according to our experimental results on three benchmark datasets XSum, Pubmed, and SAMSum of very different domains and styles.

## introduction abstractive summarization aims to generate a compact and fluent summary that preserves the most salient content of the source document. recent advances in pre-trained language models @xcite @xcite have led to improvements in the quality of generated summaries. however, one prominent limitation of existing abstractive summarization systems is the lack of faithfulness of generated outputs. faithful summaries should only contain content that can be derived from the source document instead of hallucinated or fabricated statements. summary hallucination could be categorized by the information source as intrinsic and extrinsic hallucinations. @xcite ; @xcite showed that about 30% of the summaries generated by seq2seq models suffer from the hallucination phenomenon at source: when the experiments are eventually run, the results will be streamed live on youtube. alongside prof hawking, the judging panel consists of [...] summary: stephen hawking joined the judging panel of a science competition on the internet education site gumtree. faithfulness and factuality in abstractive summarization has received growing attention from the nlp community @xcite @xcite . recent works have attempted to address the hallucination problem at the entity level by reducing hallucinated entities during generation. @xcite proposed a post-processing method, which replaces the hallucinated entities in the generated outputs with the same type entities in the source document. however, it introduces additional errors to the summary and increases the intrinsic hallucination. @xcite proposed to address entity hallucination by filtering the training data and multi-task learning with summary-worthy named-entities classification. however, the method sacrifices part of the training data and decreases the quality of the summary. to address the above issues, we propose to solve entity hallucination by guiding the model learning process with entity control code (ecc) @xcite @xcite . we utilize the entity coverage precision between the training document and its reference summary as faithfulness guidance and prepend it to the corresponding document in the training phase. then, we prepend faithful control code during inference and reduce hallucinated entities effectively without decreasing the fluency and salience of generated summaries according to our experimental results. in addition, we extend control code to a wikipediabased intermediate fine-tuning model, which generates faithful and salient summaries across domains in the zero-shot setting. we validate our methods on three benchmark datasets across different domains, and experimental results demonstrate the effectiveness of our methods. ## analysis and discussion does our model generate fewer entities to be safe? one obvious way to get higher entity coverage precision is to avoid generating entities or generating extra non-sense named entities from the source document. we show the distribution of the number of entities in the generated summaries by our model and bart in fig 2 . we see that the two distributions are very similar and have almost the same mean number of entities. as a result, we argue that our method doesn't under-generate nor over-generate entities from the source document, and we don't need to separately control the entity compression rate. ## related work the faithfulness and factuality in abstractive summarization has received growing attention by the summarization community recently @xcite @xcite . @xcite categorized hallucinations by the information source as intrinsic and extrinsic hallucinations. researchers have turned to textual entailment @xcite , question answering (qa) @xcite , natural language inference (nli) @xcite and entity level precision @xcite for automatic faithfulness evaluation. to improve the faithfulness of generated summaries. @xcite proposes a fact-aware summarization model with open information extraction and dependency parse technologies. @xcite uses graph attention to integrate factual relations into the summary generation process. recent works also focus on addressing entity-level hallucination problems. @xcite proposes a post-processing method to correct hallucinated entities and @xcite addresses entity hallucination by filtering the training data and multi-task learning. one concurrent work @xcite incorporates entity chain content planning to guide faithful summary generation. the transformer-based seq2seq architecture @xcite currently dominates the stateof-the-art performance in many nlp tasks @xcite @xcite . we use bart @xcite as a backbone for abstractive summarization in this work, but our method is generally appliable for all seq2seq models. our work is also related to controllable abstractive summarization. @xcite controls the summary length by extending a convolutional sequence to sequence model. @xcite introduces a keyword guided framework for entity-centric, length-controllable summarization and question-guided summarization. @xcite proposes to control the summary generation with a list of desired named entities. recently, @xcite proposes to use language models to generate pseudo labels to control the generation of dialogue summarization. our work uses control code to improve summary generation faithfulness and cross-domain generalizability. ## conclusion in this paper, we propose ecc to address extrinsic hallucination in abstractive summarization in both supervised and zero-shot settings. our extensive experiments demonstrate that the proposed method effectively reduces entity hallucination without hurting the quality of the generated summaries.