Improving Factuality of Abstractive Summarization via Contrastive Reward Learning (2023)

Abstract: Modern abstractive summarization models often generate summaries that contain hallucinated or contradictory information. In this paper, we propose a simple but effective contrastive learning framework that incorporates recent developments in reward learning and factuality metrics. Empirical studies demonstrate that the proposed framework enables summarization models to learn from feedback of factuality metrics using contrastive reward learning, leading to more factual summaries by human evaluations. This suggests that further advances in learning and evaluation algorithms can feed directly into providing more factual summaries. Code and human evaluation results will be publicly available at \url{https://github.com/EthanC111/factuality_summarization}.

## introduction one major challenge in current abstractive summarization models is how to generate more factual summaries that are consistent with the source text @xcite . various approaches have been proposed to address this challenge, including augmenting the model input @xcite , performing post-processing @xcite , and modifying the learning algorithms @xcite . in particular, learning-based methods possess the advantage of not requiring modification to the existing model architecture or the addition of new modules. in the meantime, with the growing interest in aligning learning objectives with evaluation criteria of interest, utilizing feedback of automatic evaluation metrics @xcite or human preferences @xcite as rewards for fine-tuning abstractive summarization models has gained substantial attention. these methods learn to optimize rewards using techniques such as reinforcementlearning (rl) @xcite , minimum risk training (mrt) @xcite ; wieting language model (e.g. bart) source document candidate summaries step 1: generate candidate summaries step 2: rank candidate summaries step 3: train model with contrastive learning reward model (e.g. factuality metrics) summarization model embedding space candidate summaries training et al., 2019), and contrastive reward learning (crl) (liu and liu, 2021; liu et al., 2022). given the benefits of learning-based methods in improving factuality of abstractive summarization, and recent advancements in factuality metrics for detecting factual inconsistencies in generated summaries, it is of interest to apply reward learning to enforce models to learn from feedback of factuality metrics to improve the factuality of abstractive summarization models. we aim to investigate the following questions in this paper -q1: can contrastive reward learning effectively utilize existing factuality metrics to improve the factuality of abstractive summarization models? q2: can the improvement in factuality be reflected in human evaluation studies? in this paper, we propose a contrastive reward learning framework that enables abstractive summarization models to directly learn from feedback of factuality metrics in a sample-efficient manner. in contrast to other contrastive learning frameworks @xcite , our proposed framework does not rely on the complex construction of negative samples. instead, similar to @xcite , all candidate summaries used for contrastive learning are generated from pretrained sequence-to-sequence models @xcite using diverse beam search @xcite . additionally, our framework also incorporates the use of quality metrics to provide more fine-grained information on the ranking (positive / negative) of candidate summaries. specifically, we investigate learning from the rewards of two factuality metrics: bartscore @xcite and dae @xcite . through automatic and human evaluation studies, we demonstrate that our framework enables summarization models to generate significantly more factual summaries. ## conclusion in this work, we present a simple contrastive reward learning framework that enforces abstractive summarization models to learn from feedback of existing factuality metrics. empirical studies demonstrate the effectiveness of this approach, showing that abstractive summarization models that learn from factuality metric feedback through contrastive reward learning can generate more factual summaries without sacrificing coherence or relevance. this suggests that further advancements in the reward learning paradigm and factuality metrics can facilitate the development of more factually consistent abstractive summarization models.