Robust Preference Optimization via Dynamic Target Margins (2025)

Abstract: The alignment of Large Language Models (LLMs) is crucial for ensuring their safety and reliability in practical applications. Direct Preference Optimization (DPO) has emerged as an efficient method that directly optimizes models using preference pairs, significantly reducing resource demands. However, the effectiveness of DPO heavily depends on the data quality, which is frequently compromised by noise. In this work, we propose \gamma -PO, a dynamic target margin preference optimization algorithm that adjust reward margins at the pairwise level. By introducing instance-specific margin calibration, \gamma -PO strategically prioritizes high-confidence pairs (those demonstrating higher reward margins) while suppressing potential noise from ambiguous pairs. Moreover, \gamma -PO is a plug-and-play method, compatible with variants of DPO that rely on reward margin between preference pairs. Across benchmarks such as AlpacaEval2 and Arena-Hard, \gamma -PO achieves an average 4.4% improvement over other baselines, setting new benchmarks for state-of-the-art performance. Additionally, \gamma -PO requires minimal code changes and has a negligible impact on training efficiency, making it a robust solution for enhancing LLMs alignment. Our codes are available at https://github.com/sunjie279/gammaPO.

## introduction the alignment of large language models (llms) with human values remains critical for their safe deployment @xcite . while reinforcement learning from human feedback (rlhf) @xcite pioneered model alignment through reward modeling and proximal policy optimization (ppo) @xcite , its computational intensity and training instability @xcite motivate simpler alternatives. direct preference optimization (dpo) (rafailov et al., * corresponding authors. 𝒚 𝒘 : use your weekends to go outdoors for hiking, cycling and other sports to get closer to nature. (@xmath0.9) 𝒚 𝒍 : do some simple stretching exercises during work breaks to relax your body. (@xmath1.8) ambiguous pair v.s. unambiguous pair prompt: how to relieve work stress? 𝒚 𝒘 : you can talk to your friends or family and share your worries and stress. (@xmath2.8) 𝒚 𝒍 : drink more alcohol. when you are drunk, you don't have to think about anything and the stress will naturally disappear. (@xmath3.1) figure 1: comparison of ambiguous and unambiguous sample pairs. ambiguous pairs exhibit narrow reward margins, indicating low confidence in model predictions, whereas unambiguous pairs demonstrate wide reward margins, reflecting high prediction confidence. existing robust alignment approaches typically adopt two distinct paradigms: (1) applying soft labels through label smoothing techniques to reduce overconfidence in potentially noisy annotations @xcite , and (2) implementing rule-based data filtering mechanisms to eliminate ambiguous preference pairs @xcite . both categories face critical limitations: • one-size-fits-all objective: current methods @xcite @xcite employ uniform loss functions that treat all preference pairs equally, disregarding varying confidence scores across samples. for modeling preference pair ambiguity, the bradley-terry model @xcite provides a probabilistic foundation through p (y w ≻ y l ) = σ(r wr l ), where smaller reward margin indicates higher uncertainty. building on this insight, recent work simpo @xcite introduces a fixed target margin γ 0 to reshape the reward landscape via σ(r wr lγ 0 ). while effective in clean data scenarios, this rigid margin fails to adapt to the spectrum of uncertainty inherent in real-world preference pairs. this limitation motivates our key research question: can adaptive target margin adjustment based on reward margin enhance alignment robustness? we propose γ-po, a plug-and-play method that dynamically adjusts target margins γ i for each preference pair. our approach employs confidenceaware margin scaling: for unambiguous pairs with a significant difference in rewards, we increase the target margins to strengthen gradient signals during training. conversely, for ambiguous pairs with small reward margins, we reduce the target margins to mitigate the influence of uncertain label. this dual mechanism provides two complementary benefits: (1) algorithm-agnostic design compatible with any reward margin-based method (e.g., dpo, simpo), and (2) theoretical connections to dynamic label smoothing @xcite . we derive that γ-po inherently implements dynamic label smoothing through its dynamic target margins. specifically, smaller reward margins induce larger ε (i.e., smoother labels), effectively smoothing labels for ambiguous pairs. conversely, larger margins reduce ε (i.e., sharper labels), sharpening supervision for unambiguous samples. experiments across four base models (llama-3-8b-instruct, mistral-7b-instruct, gemma-2-9b-instruct, and qwen-2.5-7b-instruct) show γ-po achieves 4.4% average gains on alpacaeval2 @xcite , arena-hard @xcite , and mt-bench @xcite , with minimal computational overhead. additionally, our plugand-play approach requires only a few lines of code and has a negligible impact on training efficiency, yet significantly enhances performance. ## method in this section, we first present the methodology for deriving the adaptive target margin. next, we present our proposed method, γ-po, along with its two variants, γ-dpo and γ-simpo. finally, we conduct a comprehensive comparative analysis with rdpo @xcite , highlighting its superior robustness and alignment capabilities. ## experiments this section starts with an overview of the experimental setup, including various base models, training datasets, evaluation methods, and baselines. we then present the experimental results and visualize the dynamic target reward margin along with the hyperparameter analysis. ## conclusion in this work, we introduce a plug-and-play approach called γ-po, which is applicable to dpo and its variants. the key idea behind γ-po is personalizing the target margin based on the reward margin, increasing them for high-confidence pairs to prioritize learning from these instances while reducing them for low-reward margin pairs to mitigate the impact of uncertain preference, which improves the handling of ambiguous preference data and reduces the risk of noisy data distorting the alignment policy. furthermore, γ-po offers a simple and efficient solution with minimal impact on training efficiency, making it a valuable tool for improving alignment in practical applications. extensive experiments demonstrate that the γ-po enhances the alignment performance, achieving robust results on alpacaeval2 and arena-hard. our findings highlight the potential of dynamic margin strategies as a promising direction for improving the alignment of llms, particularly in the presence of noisy data. there is still substantial opportunity for deeper exploration of this approach to enhance model alignment in real-world settings.