Model-based Preference Optimization in Abstractive Summarization without Human Feedback (2024)

Abstract: In abstractive summarization, the challenge of producing concise and accurate summaries arises from the vast amount of information contained in the source document. Consequently, although Large Language Models (LLMs) can generate fluent text, they often introduce inaccuracies by hallucinating content not found in the original source. While supervised fine-tuning methods that maximize likelihood contribute to this issue, they do not consistently enhance the faithfulness of the summaries. Preference-based optimization methods, such as Direct Preference Optimization (DPO), can further refine the model to align with human preferences. However, these methods still heavily depend on costly human feedback. In this work, we introduce a novel and straightforward approach called Model-based Preference Optimization (MPO) to fine-tune LLMs for improved summarization abilities without any human feedback. By leveraging the model’s inherent summarization capabilities, we create a preference dataset that is fully generated by the model using different decoding strategies. Our experiments on standard summarization datasets and various metrics demonstrate that our proposed MPO significantly enhances the quality of generated summaries without relying on human feedback. The code is publicly available at https://github.com/cjaep/MPO .

## introduction large language models (llms) have demonstrated remarkable capabilities in generating fluent and plausible text @xcite @xcite . however, despite these advancements, llms often produce summaries that, while plausible, contain incorrect or contradictory information-a phenomenon known as hallucination @xcite figure 1 : summarized results via automated metrics. our method mpo, which uses the model-generated summaries for preference optimization, proves to be more effective than ppo and dpo, both of which use human preference datasets for optimization. the results are from using the gpt-j on the tl;dr dataset. 2020). the fundamental reason for this issue is that llms are primarily trained to predict the most likely next token based on maximum likelihood, which is the most common objective for pre-training language models @xcite . in principle, reinforcement learning based objectives can circumvent these failures by choosing an appropriate reward function @xcite . recently, reinforcement learning from human feedback (rlhf) has focused on aligning language models with human preferences, thereby effectively enhancing the models' summarization abilities @xcite @xcite @xcite . while rlhf and other preference-based optimization methods @xcite effectively fine-tune models to align with human preferences, human feedback is not always reliable. for example, even though the quality of text summaries depends on various factors, @xcite demonstrated that human preferences often overlook factuality and consistency, which are crucial in avoiding hallucination. this implies that a summary judged as favorable by humans is not necessarily free from hallucination. in other words, preference optimization with human feedback does not guarantee improved faithfulness. moreover, the use of human preference faces challenges related to the collection of human-annotated data. although rlhf does not require massive amounts of data to enhance performance, sourcing high-quality human preference data remains an expensive process @xcite . to address these challenges, prior works have aimed to conduct preference optimization without relying on human preferences @xcite @xcite . such methods often require external metrics or complex filtering processes to establish preference pairs. for instance, @xcite utilized lexical overlap (rouge) to assess salience and an entailment score to evaluate factual consistency. similarly, @xcite employed factscore @xcite to gauge reward signals between generated summaries. however, as stated by goodhart's law-'when a measure becomes a target, it ceases to be a good measure'-relying excessively on these imperfect metrics carries the risk of overfitting to the metrics alone @xcite . in response, we propose model-based preference optimization (mpo), a novel and straightforward approach that leverages the model's inherent summarization capabilities without relying on any human feedback or external metrics. this method generates faithful summaries by aligning preferences between responses generated using different decoding strategies. in particular, we utilize (1) a deterministic decoding strategy (e.g., beam search decoding) to generate chosen samples and (2) a stochastic decoding strategy (e.g., temperature sampling) to generate rejected samples. therefore, our approach does not require any external knowledge or metrics to construct preference pairs. previous studies have shown that deterministic decoding strategies tend to generate outputs that are less surprising and closely aligned with the source text, while stochastic decoding introduces randomness, making it more prone to hallucinations @xcite @xcite . specifically, @xcite presented empirical evidence indicating that beam search yields the most faithful summaries, while the randomness introduced by sampling reduces faithfulness. based on these findings, we align our model's preference toward summaries generated via beam search rather than those randomly sampled. as illustrated in figure 1 , our approach outperforms models trained with standard supervised fine-tuning (sft) or those optimized with human preferences (e.g., ppo, dpo) in terms of faithfulness and relevance to the source text. our main contribution is model-based preference optimization (mpo), a simple and straightforward approach for fine-tuning language models to improve abstractive summarization without relying on any human feedback or external metrics. our experimental results demonstrate that mpo outperforms models optimized with human preferences, offering superior overall performance and greater generalizability across diverse language models and datasets. ## proposed method in this section, we detail our process for encouraging faithfulness in abstractive summarization. we follow the typical pipelines of preference optimization @xcite @xcite . however, by leveraging the differences between deterministic and stochastic decoding strategies, our pipeline does not require any external knowledge (e.g., evaluation metrics) or human feedback. this pipeline is depicted in figure 2 . ## related work in the realm of auto-regressive language models, there are two primary approaches aimed to enhance the model's summarization capabilities: adjusting the learning algorithm or refining the decoding strategy @xcite . the former involves updating the model's parameters through a learning objective, while the latter entails improving the decoding algorithm during generation while maintaining the existing pre-trained parameters frozen. in this paper, we will review two ap-proaches in abstractive summarization aimed at alleviating hallucination. ## conclusion this study introduces model-based preference optimization (mpo), a novel approach to improve the faithfulness and quality of abstractive summaries generated by large language models (llms). unlike traditional methods that rely heavily on costly human feedback, mpo leverages the model's inherent summarization capabilities to create a preference dataset using different decoding strategies. our extensive experiments demonstrate that mpo significantly enhances the summarization performance, providing an efficient and scalable solution to address the challenges of hallucination in llmgenerated summaries.