Rethinking DPO : The Role of Rejected Responses in Preference Misalignment (2025)

Abstract: Direct Preference Optimization (DPO) is a simple and efficient framework that has attracted substantial attention. However, it often struggles to meet its primary objectives—increasing the generation probability of chosen responses while reducing that of rejected responses—due to the dominant influence of rejected responses on the loss function. This imbalance leads to suboptimal performance in promoting preferred responses. In this work, we systematically analyze the limitations of DPO and existing algorithms designed to achieve the objectives stated above. To address these limitations, we propose Bounded-DPO (BDPO), a novel method that bounds the influence of rejected responses while maintaining the original optimization structure of DPO. Through theoretical analysis and empirical evaluations, we demonstrate that BDPO achieves a balanced optimization of the chosen and rejected responses, outperforming existing algorithms.

## introduction aligning large language models (llms) with human values using human preference data has become essential in natural language processing (nlp). direct preference optimization (dpo; @xcite ) has gained significant attention for efficiently optimizing preference comparisons directly from pairwise feedback, without relying on explicit reward models or reinforcement learning, offering a computationally cheaper alternative to conventional rlhf methods @xcite @xcite . despite its simplicity, dpo exhibits two key limitations. first, it tends to increase the probability of generating out-of-distribution (ood) actions, a drawback attributed to the absence of sampling during training @xcite . second, several studies @xcite @xcite have highlighted that dpo struggles to sufficiently increase the probability of chosen actions. to address this, dpo+nll was proposed, augmenting dpo with a negative log-likelihood (nll) loss-commonly used in supervised learning-to explicitly improve the likelihood of chosen responses. another alternative, dpo-positive (dpop), introduces a penalty when the probability of a chosen response falls below that assigned by the initial model @xcite . this study identifies two core objectives of dpo: (1) increasing the generation probability of chosen responses while decreasing that of rejected responses, and (2) ensuring the model does not deviate too far from the reference model. the analysis reveals that, in theory, the dpo loss can achieve an optimal solution by simply lowering the probability of generating rejected responses, without necessarily increasing the probability of generating chosen responses. empirical results corroborate this finding, showing that dpo fails to adequately fulfill objective (1). this work also examines alternative approaches, such as dpo+nll and dpop, and discusses their respective limitations in detail. we propose bounded-dpo (bdpo), a novel algorithm designed to address the two primary objectives of dpo more effectively. bdpo preserves the overall structure of the original dpo loss, but replaces the rejected response distribution of the updated model with a mixture distribution that incorporates the reference model. this modification bounds the influence of rejected responses on the loss function, ensuring adherence to objective (2) while effectively achieving objective (1). following the introduction of bdpo, we present a theoretical analysis showing that the bdpo loss guarantees a lower bound on the probability of generating the chosen response (section 4.1). we conduct both toy examples and real-world model experiments to compare the behavior of bdpo with existing algorithms (section 4.2). experimental results demonstrate that bdpo achieves superior performance in preference optimization (section 5). additionally, we conducted an ablation study to gain deeper insights into bdpo's performance and provided a detailed analysis of the findings. ## related works direct preference optimization @xcite ) simplifies the rlhf framework by eliminating the need for reward modeling, instead directly training models on human preference data. due to its efficiency and simplicity, dpo has been widely adopted in various nlp applications @xcite @xcite . however, despite its practical advantages, dpo often yields models that underperform compared to those trained with conventional rlhf methods @xcite @xcite . recent studies have identified several notable limitations of dpo, including: dpo+nll to prevent dpo from reducing the probability of chosen responses, a simultaneous supervised learning step on chosen responses has been proposed @xcite @xcite . this method, referred to as dpo+nll in this paper, adds an explicit negative log-likelihood (nll) loss term to the original dpo objective to directly increase the likelihood of chosen responses. however, we show that it introduces sensitivity to hyperparameters, often causing the training process to be dominated by the nll term. ## analysis of existing objectives for preference optimization in this section, we revisit the objectives of dpo and review prior findings showing that it can deviate from its intended goals. to deepen the understanding of dpo's behavior, we provide a detailed analysis of its underlying mechanisms. we also examine the limitations of two representative extensions-dpo+nll and dpop-that have been proposed to mitigate these issues. the dpo objective is defined as: the dpo loss captures two key objectives: these characteristics mirror those of the original rlhf framework. the first reflects the use of reward models trained to favor chosen responses over rejected ones, with policies optimized to maximize these rewards. the second aligns with behavior regularization, which constrains the learned policy to remain close to the reference model and mitigates over-optimization on uncertain rewards. ## our algorithm in this section, we introduce bounded-dpo (bdpo), our proposed method designed to satisfy the two primary objectives of dpo. we provide its theoretical foundation and show empirically that bdpo achieves the intended behavior while overcoming the limitations of existing methods. ## conclusions in this paper, we identified the two core objectives of dpo and showed through theoretical and empirical analysis that it fails to fully achieve them. we also examined alternative algorithms, such as dpo+nll and dpop, which aim to address dpo's limitations, and highlighted their respective shortcomings through both theoretical and experimental analysis. to overcome these issues, we proposed bounded-dpo (bdpo), a method specifically designed to fulfill dpo's objectives without inheriting the weaknesses of previous approaches. theoretically, bdpo offers an ideal optimal solution, guarantees a lower bound on the chosen response, and reduces sensitivity to the rejected response. empirically, bdpo consistently outperforms other methods across various datasets, model scales, and evaluation benchmarks. furthermore, our ablation study provides additional insights into how bdpo operates and the role of its key hyperparameter. limitations due to gpu constraints, we were unable to conduct experiments across a broader range of models and algorithms. similar to prior work, our theoretical analysis focuses on the single-pair setting-one preference pair per prompt. while bdpo can be extended both theoretically and empirically to the multi-pair response setting, we leave this for future work. our evaluation focused on instruction-following and mathematical reasoning benchmarks, but we were not able to include a wider range of evaluation tasks. additionally, although bdpo is applicable to variants such as online or iterative dpo, these directions are beyond the scope of this study.