Self-supervised Preference Optimization: Enhance Your Language Model with Preference Degree Awareness (2024)

Abstract: Recently, there has been significant interest in replacing the reward model in Reinforcement Learning with Human Feedback (RLHF) methods for Large Language Models (LLMs), such as Direct Preference Optimization (DPO) and its variants. These approaches commonly use a binary cross-entropy mechanism on pairwise samples, i.e., minimizing and maximizing the loss based on preferred or dis-preferred responses, respectively. However, while this training strategy omits the reward model, it also overlooks the varying preference degrees within different responses. We hypothesize that this is a key factor hindering LLMs from sufficiently understanding human preferences. To address this problem, we propose a novel Self-supervised Preference Optimization (SPO) framework, which constructs a self-supervised preference degree loss combined with the alignment loss, thereby helping LLMs improve their ability to understand the degree of preference. Extensive experiments are conducted on two widely used datasets of different tasks. The results demonstrate that SPO can be seamlessly integrated with existing preference optimization methods and significantly boost their performance to achieve state-of-the-art performance. We also conduct detailed analyses to offer comprehensive insights into SPO, which verifies its effectiveness. The code is available at https://github.com/lijian16/SPO.

## introduction the alignment of large language models (llms) with human preferences is paramount, as it ensures that the outputs of llms are congruent with human values and ethical standards @xcite @xcite . through meticulous tuning and ongoing learning of human preferences, llms can more accurately meet user needs while avoiding the generation of harmful or biased content @xcite . effective preference alignment not only enhances the applicability and safety of llms but also constitutes a critical step towards the responsible utilization of artificial intelligence. to achieve human preference alignment of llms, a variety of methods have been developed. one prominent approach is reinforcement learning from human feedback (rlhf) @xcite , such as proximal policy optimization (ppo) @xcite , reinforce @xcite and their variants @xcite , which involve training reward models to optimize for objectives that are iteratively refined based on human feedback. however, these methods introduce increased complexity into the training process, involving training multiple models and sampling from the llm in the loop of training @xcite . to streamline this process, recent works have proposed alternative solutions to reinforcement learning @xcite @xcite . dpo @xcite and its variants @xcite @xcite @xcite @xcite directly leverage pairwise responses to imbue the model with preference knowledge without a reward function. these methods achieve preference alignment by minimizing or maximizing the loss between each token in the language model's output and the tokens that are either preferred or not preferred. however, this training strategy overlooks a crucial aspect of a reward model: its ability to differentiate between varying degrees of human preferences in responses. we hypothesize that this is a key factor that prevents llms from fully understanding human preferences in those rlhf methods without a reward model. to address this issue, we propose a novel selfsupervised preference optimization (spo) scheme to help llms learn the degree of human preference and align llms with human preferences, simultaneously. the proposed method is illustrated in figure 1 . specifically, we design a novel auxiliary self-supervised task that selectively removes key content in llm outputs to generate responses with varying degrees of preference. during the training process, we employ a keyword extractor @xcite on the outputs of llms to extract key content. by removing different amounts of the content, we construct responses with different degrees of preference. these responses are then fed into a self-supervised module for classification and the loss is integrated into the primary preference alignment loss (based on existing alignment methods) to jointly optimize llms. we observe that the key content within the llms' outputs is closely associated with preference information, as described in section 4. by gradually removing the content, we can effectively construct varying degrees of preferences. on the other hand, this method allows for the generation of multiple responses from a single output of llms, obviating the need for additional data collection and annotation efforts. we conduct comprehensive experiments on two widely used datasets of different tasks, i.e., antropic hh @xcite and tl;dr summarization @xcite . the results demonstrate that our proposed spo can significantly enhance the performance of various existing alignment methods and achieve state-of-the-art results. additionally, we conduct detailed analyses of multiple aspects and modules of our proposed spo to provide comprehensive insights and verify its effectiveness. the contributions of this work can be summa-rized as follows: ## method in this section, we initially examine the pipeline of methods alternative to rlhf, with a primary focus on pairwise approaches that do not incorporate a reward model. subsequently, we present the selfsupervised preference optimization (spo), aimed at assisting llms in learning preference degrees at a fine-grained level. ## conclusion in this work, we first identify a gap in alternative methods to rlhf, which overlooks the learning of preference degrees. to this end, we introduce a novel self-supervised preference optimization framework that integrates fine-grained human preference information into large language models (llms), thereby enhancing the understanding of human preferences. this approach does not require additional manual annotation and inference overhead. the proposed spo can extract key content from the prediction of llms and selectively remove the content to construct responses with varying preference intensity. subsequently, these responses are classified by the self-supervised modules and their losses are integrated with the alignment loss to jointly optimize llms. extensive experiments and analyses fully demonstrate the effectiveness of our spo.