Improving Factuality of Abstractive Summarization without Sacrificing Summary Quality (2023)

Abstract: Improving factual consistency of abstractive summarization has been a widely studied topic. However, most of the prior works on training factuality-aware models have ignored the negative effect it has on summary quality. We propose {pasted macro ‘MODEL’}name (i.e. Effective Factual Summarization), a candidate summary generation and ranking technique to improve summary factuality without sacrificing quality. We show that using a contrastive learning framework with our refined candidate summaries leads to significant gains on both factuality and similarity-based metrics. Specifically, we propose a ranking strategy in which we effectively combine two metrics, thereby preventing any conflict during training. Models trained using our approach show up to 6 points of absolute improvement over the base model with respect to FactCC on XSUM and 11 points on CNN/DM, without negatively affecting either similarity-based metrics or absractiveness.

## introduction although recent methods have made significant improvements in abstractive summarization @xcite @xcite , they do still lack a very critical component -factual consistency. recent works @xcite @xcite have shown that a majority of the model-generated summaries are unfaithful and suffer from a wide range of hallucination @xcite . making summarization models factually consistent is critical for its trustworthiness in real-world applications. recent studies have made several attempts to improve factuality of abstractive summarization by either modifying the maximum likelihood estimation (mle) training objective @xcite figure 1 : overview of our approach. for a given article, we generate a number of summaries that can be either factual (blue) or non-factual (yellow). grey summaries are filtered out. we select a balanced set using rouge and then finally train the model to rank them based on the factuality score. @xcite , directly optimizing factuality metrics using reinforcement learning @xcite or improving the quality of the training data @xcite . however, most of these works have reported a negative relationship between factual consistency and summary quality foot_0 . for example, @xcite improve factuality at a cost of a 6-point drop in rouge-l, @xcite also observe a 2-point drop in rouge-l. prior approaches have also optimized factuality at the cost of abstractiveness @xcite . this leads to a critical question: can we improve the factuality of summarization without the cost on the summary quality? to this end, we propose efactsum (i.e. effective factual summarization): a candidate summary generation and ranking technique for contrastive summarization training (fig. 1 ) that not only achieves significant gains in factuality of abstractive summarization but also improves the sum-mary quality. unlike prior works which often sacrifice summary quality for improving faithfulness, we take an alternative approach to improve both faithfulness and summary quality. we make use of the fine-tuning strategy by @xcite and make key modifications to the ranking process. as depicted in fig. 1 we start with generating a number of candidate summaries using existing fine-tuned models. using these summaries, we select a subset by effectively combining two evaluation metrics of the two different criteria ( §2), thus avoiding optimizing one at the cost of the other. this technique helps obtain gains over methods that simply optimize one metric ( §3.4). the promising results by efactsum on xsum and cnn/dm have shown consistent improvements in both aspects over strong baselines, demonstrating effectively enhanced summarization factuality without sacrificing the quality. ## approach given a document (d), the task of summarization seeks to generate its summary (s) that satisfies some conditions like factuality, coherence, etc. the standard fine-tuning process involved the use of maximum likelihood estimation (mle). inspired by @xcite , in addition to the cross-entropy loss, we incorporate a contrastive loss that encourages models to provide a higher probability mass to the more factual summaries. formally, for every training document d and a ranked list of the most probable candidate summaries [s 1 , s 2 , . . . s n ], the model learns to rank the summaries according to the factuality score. to achieve this, we make use of the following loss: ## experiments we state the experimental setup in §3.1 and report the results in §3.2, followed by an abstractiveness analysis in §3.3. in §3.4, we analyze the importance of the various components in our approach. ## related work factual consistency in abstractive summarization has garnered much attention recently @xcite . existing works have explored improving factual consistency during fine-tuning, inference, and pre-training stages, respectively. for factual fine-tuning, works have applied contrastive learning @xcite , reinforcement learning (gunasekara et al., 2021) or knowledge integration @xcite to teach the model identify summaries of high factual consistency while @xcite modify the pretraining process to introduce factuality-awareness. several works have also improved summary factuality through postprocessing in inference, such as correcting errors and re-ranking by factual scores @xcite @xcite @xcite . our work differs from the aforementioned works as we improve both factuality and summary quality, unlike other methods, which often sacrifice one for the other. ## conclusion we present efactsum (effective factual summarization), a candidate summary generation and ranking technique for contrastive summarization training, which helps make models more faithful without adversely affecting summary quality. results show that this simple, yet effective method can achieve consistent gains on both factuality and similarity-based metrics without negatively affecting the degree of abstractiveness. we hope that our findings will encourage future research on factuality-consistent summarization to focus more on the tradeoffs between summary quality and factuality. training details for training we use the adam optimizer with linear learning rate scheduling for the model training. tab. 5 and tab. 6 contain the best set of hyper-parameters for training pegasus on xsum and bart on cnn/dm. these hyperparameters were obtained after an extensive grid search. we perform validation after every 1600 steps and save the best model using the validation cross-entropy loss.