SGDPO : Self-Guided Direct Preference Optimization for Language Model Alignment (2025)

Abstract: Direct Preference Optimization (DPO) is broadly utilized for aligning Large Language Models (LLMs) with human values because of its flexibility. Despite its effectiveness, it has been observed that the capability of DPO to generate human-preferred response is limited and the results of DPO are far from resilient. To address these limitations, in this paper we propose a novel Self-Guided Direct Preference Optimization algorithm, i.e., SGDPO, which incorporates a pilot term to steer the gradient flow during the optimization process, allowing for fine-grained control over the updates of chosen and rejected rewards. We provide a detailed theoretical analysis of our proposed method and elucidate its operational mechanism. Furthermore, we conduct comprehensive experiments on various models and benchmarks. The extensive experimental results demonstrate the consistency between the empirical results and our theoretical analysis and confirm the effectiveness of our proposed approach (up to 9.19% higher score).

## introduction large language models (llms) pretrained with next-token prediction have experienced rapid advancements @xcite @xcite . this progress underscores the necessity to align llm outputs with human values and preferences while safeguarding societal values from harm. reinforcement learning from human feedback (rlhf) has emerged as a critical method for achieving this alignment and has become an essential component within the llm training pipeline @xcite @xcite . traditional rlhf typically involves three key steps: supervised fine-tuning (sft), reward learn- † equal contribution * corresponding author: jiliuwork@gmail.com ing, and reinforcement learning (rl) optimization. because the rl optimization step relies heavily on the reward model, it is essential to train a high-quality reward model. however, this necessity adds complexity to the rlhf training process, making it intricate @xcite . to tackle this issue, direct preference optimization (dpo) removes the need for reward training by reparameterizing the reward model @xcite . specifically, it maps reward functions to optimal policies by employing the bradley-terry model (bradley and @xcite , thereby transforming preference feedback from online reward models into offline implicit modeling. as a result, dpo simplifies the posttraining process. while it has been widely adopted for its flexibility with similar performance levels compared to classic rlhf methods, e.g., ppo @xcite , chatglm-rlhf @xcite , the limitations of dpo are observed in a bunch of investigation, which lead to suboptimal alignment performance in llm training. these limitations include high computational costs @xcite @xcite , verbosity @xcite @xcite , and overfitting @xcite @xcite . in addition, dpo may still incur inferior capability of llms in producing responses that resonate with human preferences @xcite . while llms trained with dpo tend to avoid generating responses humans dislike, they struggle to generate responses that humans prefer. furthermore, the efficacy of dpo is inconsistent while being sensitive to the effectiveness of supervised fine-tune (sft) @xcite . for instance, llms with improper and ineffective settings may lead to poor dpo performance. as illustrated in figures 1 (a ), (b), (c), and (d), the training reward curves of dpo on various base models using the same preference dataset exhibit extremely high diversity. recently, some theoretical works @xcite reveal the reasons behind the limitations of dpo. first, the standard dpo loss can lead to a reduction in the likelihood of preferred examples generated by llms @xcite , especially when the hamming distance between preferred and dispreferred responses is low. second, the limitations of dpo may be attributed to undesired distinct update patterns in gradient flow between chosen and rejected rewards @xcite . when the optimization process enters an undesired region, the gradient flow of dpo tends to generate an unbalanced update to different variables or incurs difficulties in escaping saddle points, leading to inferior optimization performance. in this paper, we propose a novel self-guided direct preference optimization algorithm, i.e., sgdpo, to address the aforementioned limitations. we introduce a pilot term into the objective function of sgdpo. this pilot term can be adjusted to steer gradient updates towards different regions, resulting in diverse gradient update patterns and consequently leading to distinct optimization processes. in this case, sgdpo can enhance the alignment capability of llms to generate responses preferred by humans, while contributing to the stabilization and resilience of the llm training process, as well. in addition, we carry out a detailed theoretical analysis to illustrate the robustness and resilience of sgdpo. furthermore, we conduct extensive experiments across various models and benchmarks to demonstrate the superb performance of sgdpo. the major contributions are summarized as follows: ## related work rlhf has been proven effective in aligning llms with human values and has seen widespread adoption across various applications, e.g., summarization @xcite , safety alignment @xcite , instruction following @xcite , and translation @xcite . nevertheless, rlhf requires a complex training pipeline, which has spurred the proposal of dpo @xcite to simplify the llm training pipeline. since the introduction of dpo, a variety of extensions have been proposed to either address its limitations or provide theoretical interpretations. these include new preference optimization techniques @xcite @xcite and analytical studies @xcite . for instance, simpo @xcite reduces computational overhead by adopting a reference-free training strategy, while simper @xcite introduces an inverse perplexity objective to lower the complexity and fine-tuning time of large language models (llms). although simper results in a smaller decrease in chosen likelihoods compared to simpo, it still exhibits a declining trend in chosen likelihoods, indicating limited flexibility. in contrast, our method introduces a mechanism that allows for adjustable control over both chosen and rejected likelihoods, thereby offering greater adaptability. similarly, sampo and ld-dpo @xcite aim to reduce the verbosity often introduced by alignment algorithms due to prior biases in preference data, ultimately improving alignment performance. tdpo @xcite enhances alignment and diversity through a token-level optimization approach. @xcite mitigates overfitting by introducing a regularization term that pulls the solution toward a reference policy. cal-dpo @xcite , on the other hand, improves performance by incorporating absolute reward values instead of relying solely on relative ones-similar in spirit to ipo's regularization goal. however, in many practical scenarios, exact absolute reward values may not be available, requiring approximations that can lead to suboptimal outcomes. our method avoids this issue entirely, as it does not depend on absolute reward values and instead provides additional flexibility through tunable parameters for both chosen and rejected likelihoods, as well as their ratios. unintentional unalignment @xcite investigates how similar embeddings from preference data can lead to unintended misalignment. the authors introduce a metric called centered hidden embedding similarity (ches) to improve training sample selection. while this approach is promising for dataset curation, our method operates at the optimization level rather than the data level, making it more robust and independent of dataset modifications. additionally, nca @xcite leverages noise contrastive estimation (nce) to achieve robust alignment, and bco @xcite proposes training a binary classifier where the logit serves as a reward signal, also yielding robust results. despite these advances, none of the existing methods effectively tackle both the issue of reduced updates on preferred examples and the challenge of unbalanced updates with difficulties in escaping saddle points simultaneously. our approach addresses both concerns, offering a more comprehensive and flexible solution to preference-based alignment. ## experimental evaluation in this section, we compare sgdpo with 6 stateof-the-art performance optimization algorithms, exploiting 4 model configurations and 8 tasks. we first present the experimental setup. then, we illustrate the experimental results. finally, we show the ablation study. ## conclusions in this paper, we present a novel self-guided direct preference optimization algorithm, i.e., sgdpo, for aligning llms with human preferences. sgdpo incorporates a pilot term in the objective function in order to guide the gradient updates of the rewards during training. we provide a detailed theoretical explanation of sgdpo. furthermore, extensive experimental results across various model settings and benchmarks demonstrate the significant advantages (up to 9.19% higher score) of sgdpo.