F act PEGASUS : Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization (2022)

Abstract: We present FactPEGASUS, an abstractive summarization model that addresses the problem of factuality during pre-training and fine-tuning: (1) We augment the sentence selection strategy of PEGASUS’s (Zhang et al., 2019) pre-training objective to create pseudo-summaries that are both important and factual; (2) We introduce three complementary components for fine-tuning. The corrector removes hallucinations present in the reference summary, the contrastor uses contrastive learning to better differentiate nonfactual summaries from factual ones, and the connector bridges the gap between the pre-training and fine-tuning for better transfer of knowledge. Experiments on three downstream tasks demonstrate that FactPEGASUS substantially improves factuality evaluated by multiple automatic metrics and humans. Our thorough analysis suggests that FactPEGASUS is more factual than using the original pre-training objective in zero-shot and few-shot settings, retains factual behavior more robustly than strong baselines, and does not rely entirely on becoming more extractive to improve factuality.

## introduction abstractive summarization aims at generating short summaries that capture the essentials of a long document. research in this challenging task has made significant progress with the help of large pre-trained models @xcite @xcite . however, current models suffer from the crucial problem of hallucinations @xcite , where a summary contains facts or entities not present in the original document. such unfaithful generation raises the question of whether the models can be trustworthy and used safely for real-world applications. to tackle this problem, many approaches propose postprocessing models @xcite @xcite , but such methods are often constrained by external resources to train additional correction or selection models. an alternative line of works focuses on learning factuality directly during fine-tuning by filtering nonfactual training data @xcite or, most recently, incorporating contrastive learning @xcite to encourage generating faithful summaries. in this work, we propose factpegasus, a model that addresses the problem of hallucinations for abstractive summarization holistically, by incorporating factuality into the whole training pipeline: we tackle the lack of factuality objective in pre-training and the presence of hallucinations in the downstream dataset during finetuning. current pre-training objectives focus on improving the quality of the generated output in the downstream tasks but often overlook the factuality aspect. thus, we explore incorporating factuality into the pre-training objective of pegasus @xcite (a state-of-the-art abstractive summarization model). the original objective, gap sentence generation (gsg), transforms any text into a pseudo-summarization dataset by selecting important sentences using rouge @xcite as output summaries. we explore strategies for combining rouge and the factuality metric factcc @xcite as the selection criteria, so that the model learns to generate sentences that cover the most important information of the input document as well as remain faithful to it. next, we propose three complementary modules that further address factuality problems during fine-tuning: (1) corrector that removes hallucinations existing in reference summaries, allowing training on the full training set without learning unfaithful behaviors; (2) contrastor that encourages the model to better differentiate factual summaries from nonfactual ones by paying attention to the document using contrastive learning; (3) connec-tor, a special mask-token fine-tuning technique enabled by the gsg-style objective, that simulates the pre-training task during fine-tuning by inserting the mask token into the input document so that the pre-trained model can adapt its knowledge of generating factual summaries directly to the downstream tasks. the connector, corrector, and contrastor address the input, output, and training objective of the downstream task, respectively, and the combination of the components reduces potential confounding problems that cannot be addressed by a single module. we show that the full model improves three factuality metrics, the token and sentence error of dep entail @xcite and factcc, on the downstream datasets of xsum @xcite , wikihow @xcite , and gigaword @xcite . most notably, factpegasus outperforms existing factualityaware summarization models by more than 40% and 34% on xsum for token error and factcc, respectively. ablation studies show the usefulness of each of our fine-tuning components as well as the additive gain of combining our complementary modules, and human evaluation confirms that factpegasus generates significantly more factual summaries over strong baselines. finally, we perform a detailed analysis of fact-pegasus, which points to several important observations regarding learning and maintaining factuality: (1) zero-shot setting demonstrates the utility of our factuality-aware pre-training objective, as our model outperforms pegasus (which uses the original objective) on all three factuality metrics when evaluated directly on the downstream task without any supervised training data. few-shot experiment indicates that even a small number of nonfactual examples can have a strong negative impact on factuality and can nullify much of the gain from factuality pre-training, highlighting the importance of ensuring factuality during fine-tuning. (2) factuality dynamics @xcite further shows that factpegasus exhibits a lesser degree of factuality degradation than what is observed for bart-base. (3) factuality vs abstractiveness tradeoff curve reveals that factpegasus effectively improves factuality by not simply relying on the increase in extractiveness. to summarize, our contributions are as follows: transformer-based models have achieved state-ofthe-art performance for abstractive summarization @xcite @xcite . many such pre-trained models study the effect of useful pre-training objectives, often in the form of masking certain parts of the input. bart @xcite randomly masks spans of tokens in the text as input and asks the model to reconstruct the original text. our work builds on the success of pegasus's (zhang et al., 2020) pre-training objective that closely resembles the downstream summarization task. their objective selects sentences that best represent the document as the output summary, and masks out the selected sentences in the original text as the input document. we explore various sentence selection strategies to encourage the model to generate summaries that cover the most important information of the document and also remain faithful to it. ## factpegasus we describe our training procedure consisting of pre-training with a factuality-aware objective (section 3.1) and fine-tuning with three complementary modules for improving factuality (section 3.2). ## experimental setup we describe our experimental setup, and refer to appendix a for more details. ## conclusion in this work, we proposed factpegasus, a model for abstractive summarization consisting of factuality-aware pre-training and modules for ensuring factuality during fine-tuning. we demonstrated the effectiveness of our model at improving factuality on three downstream abstractive summarization datasets, confirmed by our human evaluation. our analysis showed that our proposed factuality-aware pre-training objective is effective at capturing knowledge of factuality compared to the original objective and that our fine-tuning modules reduce the factuality degradation observed with current models. we finally showed that improvement in factuality is not solely explained by the increase of extractiveness.