SPO : Self Preference Optimization with Self Regularization (2025)

Abstract: Direct Preference Optimization (DPO) is a widely used offline preference optimization algorithm that enhances the simplicity and training stability of reinforcement learning through reward function reparameterization from PPO. Recently, SimPO (Simple Preference Optimization) and CPO (Contrastive Preference Optimization) have proposed reference-free preference optimization methods to simplify DPO’s training process. We observe that these reference-free methods exhibit higher training efficiency but are prone to overoptimization, leading to performance degradation. To address these issues, we propose Self Preference Optimization (SPO). SPO employs the SiLU function to replace the conventional logsigmoid loss function. The SiLU function attains its minimum at a finite value, preventing the model from excessively amplifying the chosen-rejected sample probability ratio and thereby mitigating overoptimization problem. We theoretically demonstrate that the SPO loss is an upper bound of the DPO loss, implying that optimizing the SPO objective implicitly optimizes the DPO objective. We evaluate SPO’s effectiveness across multiple benchmarks including AlpacaEval 2 and MT-Bench. Experimental results show that SPO achieves a 7% improvement over SimPO in length-controlled win rate on AlpacaEval 2, while demonstrating superior performance on MT-Bench.

## introduction benefiting from large-scale pre-training @xcite and instruction fine-tuning on high-quality data @xcite , large language models (llms) have demonstrated exceptional capabilities in generating human-like responses. one of the key steps in building stateof-the-art llms is preference optimization, which aligns pre-trained llms with human preferences using human assessment data, making the model more helpful, truthful, and harmless. figure 1 : the primary difference among most offline preference optimization methods lies in the choice of the loss function, with logsigmoid (softplus) being a popular option. however, logsigmoid approaches its minimum at negative infinity, which may lead to overoptimization. to mitigate this issue, spo replaces logsigmoid with silu, which has a finite minimum value and possesses regularization properties. recently, researchers have been exploring simpler offline algorithms, with direct preference optimization (dpo) @xcite being a representative approach. dpo reparameterizes the reward function in ppo @xcite , enabling direct policy learning from preference data without the need for an explicit reward model. due to its simplicity and stability, dpo has been widely adopted in practical applications. dpo requires a reference policy to ensure that the aligned model does not deviate excessively from the reference model. in contrast, approaches like simpo @xcite , cpo @xcite and orpo @xcite propose methods that do not rely on a reference model. current methods typically implement sft and preference optimization sequentially. however, this often leads to catastrophic forgetting and complicates the training process. orpo integrates both dpo( ; ref) = simpo( ) = spo( ) = [ log ( log (yw|x) ref(yw|x) log (yl|x) ref(yl|x) in preference learning, especially in dpo and related methods, over-optimization refers to the phenomenon where the policy drifts too far from the reference model. it happens when maximizing the proxy reward (such as the likelihood gap or reward model score) leads to decreased overall performance, consistency, or diversity. after writing out the dpo loss in its standard form, we arrive at the equivalent expression: it is immediate that this term is minimized when: if the probability of the rejected sample is pushed too close to zero, the ratio becomes infinitely large regardless of the actual probability of the preferred sample. in this case, the model may exploit weaknesses in the reward signal by focusing solely on minimizing the probability of the negative sample while ignoring the probability of the positive sample, leading it to deviate from the preferred output and the reference policy. figure 3 illustrates an over-optimization issue where both positive and negative sample probabilities decrease simultaneously from simpo's training curves. both dpo and simpo optimize the probability difference between positive and negative samples but their objectives allow a scenario where both positive and negative samples probabilities decrease, with the negative sample probability decreasing more significantly. in an extreme case, if the negative sample probability reaches zero, the optimization objective becomes entirely unrelated to the positive samples. this creates a reward hacking issue @xcite , where the model can minimize the loss simply by only reducing the probability of negative samples and completely ignore the positive samples. ideally, only the negative sample probability should decrease while the positive sample probability remains stable. practically, this may be difficult to achieve because there is a certain degree of correlation between positive and negative samples. therefore, how to design preference optimization algorithms to achieve both high performance and high efficiency remains an open challenge. we believe that the choice of the loss function is key to addressing this issue. as shown in the figure 1 , a recent study @xcite has pointed out that different methods mainly differ in their loss functions, and proposed gpo (generalized preference optimization), which parameterizes preference optimization losses through a family of convex functions f , and views dpo, @xcite , and slic @xcite as special cases. we found that methods like dpo typically use logsigmoid as the loss function. however, due to the properties of the logsigmoid function, it reaches its minimum when the probability ratio between positive and negative samples approaches infinity. this causes the model to excessively amplify the probability gap between positive and negative samples. on the other hand, if relu is used as the loss function, the gradient becomes zero when the positive-to-negative sample probability ratio exceeds a certain threshold, which may lead to underfitting. in this paper, we investigate the role and impact of loss functions in pairwise preference datasets for preference optimization models and propose a simple and effective preference optimization method-self-preference optimization (spo), as shown in figure 2 , which utilize silu as loss function in the training process. compared to dpo, our method does not require a reference model, making it resource-efficient. additionally, unlike other reference-free methods like orpo and simpo, we avoid the performance degradation caused by simplifying the preference optimization process through loss function regularization. after training the spo model and comparison models with ultrafeedback @xcite , we evaluate them on instruction-following benchmarks, including alpacaeval 2 @xcite and mt-bench @xcite . under the gemma @xcite model setup, spo outperforms dpo by up to 9.6% in length-controlled win rate on alpacaeval 2 compared to @xcite , and surpasses simpo by 7.3%. we perform ablation studies on the key design aspects of spo. since preference optimization methods are sensitive to hyperparameters, we conducted a hyperparameter search based on the suggestions in the simpo paper to achieve optimal performance. the ablation experiments show that the spo loss function outperforms other loss functions. furthermore, referring to methods like cpo and rpo that use sft loss for regularization, we compare the effects of different sft weights. the results indicate that a small sft weight slightly improves the performance on mt-bench while decreasing performance on alpacaeval 2, suggesting that the regularization effect of the spo loss function itself is strong, and therefore using only the spo loss is also viable. our contributions can be summarized as follows: preference optimization objectives. in addition to dpo @xcite , many other methods have been proposed. some approach explores simplified preference optimization objectives that do not rely on a reference model, such as orpo @xcite and simpo @xcite . orpo merges the sft and preference optimization stages and optimizes the odds ratio of positive and negative samples. simpo uses length normalization techniques to improve the performance of the model. rpo @xcite regularizes the optimization process using sft loss. although this method has been applied in previous research, rpo theoretically demonstrates its effectiveness. gpo @xcite points out that the main differences between preference optimization methods lie in the design of the loss function and proposes a unified perspective to generalize them. in this study, we compare spo with a series of offline optimization algorithms, including dpo, @xcite , slic @xcite , and simpo, and find that spo outperforms these methods in various experimental settings. ## spo: self preference optimization in this section, we first introduce the background of dpo. then, we derive the reference-free spo objective by replacing the actual reference model with an ideal reference model and regularizing it using the silu function. finally, we incorporate additional length normalization and sft regularization to obtain the final spo objective. the pseudocode of our method is as follows (algorithm 1). ## experiments in this section, our goal is to understand the performance of spo compared to other preference optimization methods under different experimental settings. we outline our experimental settings (section 4.1) and present and analyze the main results (section 4.2). then, we show the results of ablation studies to compare the impact of several key designs on spo (section 4.3). ## conclusion in this paper, we propose self-preference optimization (spo), which is a new reference-free offline preference optimization method. spo mitigates the overoptimization issues in dpo and other reference-free methods through loss function regularization. the key design of spo lies in replacing the logsigmoid loss function with silu. we theoretically prove that the spo loss function is an upper bound of the dpo loss function, meaning that optimizing the spo objective implicitly optimizes the dpo objective. experimental results show that, across models of various sizes, spo is preferred over other preference optimization methods in evaluations on al-pacaeval 2 and mt-bench. furthermore, as the base model size increases, spo's win rate against dpo also improves, further demonstrating its stability and generality. extensive ablation studies indicate that loss function regularization of spo is crucial and validate its effectiveness.