Adaptive Preference Optimization with Uncertainty-aware Utility Anchor (2025)

Abstract: Offline preference optimization methods are efficient for large language models (LLMs) alignment. Direct Preference optimization (DPO)-like learning, one of the most popular approaches, stands out for its efficiency in reward modeling. However, these methods typically follow the convention to use Bradley-Terry (BT) reward modeling that faces several critical assumptions, including the requirement for pairwise training data, model distribution shifting, human rationality assumption, etc. To address these limitations, we propose a general framework for offline preference optimization methods, Adaptive Preference Optimization with Utility Anchor (UAPO), which introduces an anchoring function to estimate the uncertainties brought from preference data annotation. Our method enables training even in scenarios where the data is unpaired, significantly enhancing data utilization efficiency. Moreover, the anchor design makes UAPO more robust in the training process. Experimental results demonstrate that UAPO achieves competitive outcomes without the strict dependency on data pairing, paving the way for more flexible and effective preference optimization methods.

## introduction aligning large language models (llms) with human values and preferences is a crucial step in llm training. this alignment ensures that llms can better follow human instructions, becoming more helpful, harmless, and honest. however, ensuring that these models align with human preferences and ethical standards remains a significant hurdle. previous work has made strides in this direction by employing techniques. reinforcement learning from human feedback (rlhf) @xcite is a widely adopted method in the alignment domain, yielding significant improvements *corresponding authors in llm performance. however, the high computational cost and time consumption associated with rlhf present substantial challenges, limiting its broader application. to address these challenges, offline preference optimization methods, such as dpo @xcite , have been developed as a more manageable offline training process. this shift makes the training procedure more controllable and resource-efficient. despite the inspiring progress and superior benchmark performance made, current methodologies predominantly adhere to the bradley-terry (bt; @xcite ) model or its generalized form, the plackett-luce model @xcite , for reward modeling, a rank-based method widely applied to predict rational preference distribution given pairwise data. in this work, we challenge this convention in section 2.2 by thoroughly analysis of its discrepancies with real-world scenarios. from the data perspective, the bt model's reliance on pairwise comparisons imposes a significant constraint on the collection of preference data, such as identifying winning and losing responses from a superior model or a pre-trained reward function. this process is both labor-intensive and data-inefficient. from the optimization perspective, over-optimization in preference learning, as highlighted by goodhart's law @xcite , can lead to distribution shifts and reward hacking when applied to out-of-distribution samples. this may also lead to a disparity between training and testing reward functions. from the cognitive perspective, the bt model presupposes that human annotators are fully rational and maximize utility values, which has been challenged by behavioral economics @xcite ) that humans are typically risk-averse. these limitations are also observed in synthetic data @xcite generated by modern language models. in this work, we introduce adaptive preference optimization with uncertainty-aware utility an-chor (uapo), a cognitively inspired preference optimization framework designed to address the above limitations across data, learning, and cognitive dimensions. specifically, drawing inspiration from the anchoring effect identified in behavioral economics by @xcite , uapo introduces the concept of a utility anchor to better model human preferences in a more flexible and robust manner to solve the uncertainty in the data labeling process. this integration allows for more efficient processing of unpaired data and yields a more precise estimation of human preferences. by decoupling the learning objective into unpaired forms, the utility anchor enhances robustness to various hyperparameter settings, eliminating the necessity for meticulous customization or tuning across different models and datasets. we show that uapo is flexible and can easily be integrated to enhance previous offline reward learning methods, such as dpo @xcite and simpo @xcite . additionally, we provide theoretical analysis from the perspective of uncertainty penalty, a strategy commonly used in pessimistic rl that use uncertainty rewards @xcite to address the overoptimization problem in offline reward learning. in section 3, we run extensive experiments to demonstate the effectiveness of our uapo and its variant simuapo. we highlight the main advantages of uapo as follows. our results highlight the potential of uapo as a versatile and robust solution for preference optimization, paving the way for its application in real-world scenarios where data and computational resources are limited. ## uncertainty-aware utility anchor in this section, we start by briefly discussing prior offline preference learning methods ( §2.1). then we thoroughly analyze critical assumptions of reward modeling in previous methods that do not hold in real-world settings ( §2.2). to address the issues, we introduce a cognitive compelling utility anchor and devise our formulation of uapo in §2.3. we further provide theoretical justification in §2.4 from the perspective of pessimistic rl. ## conclusion in this paper, we propose a general framework, uapo, for offline preference optimization, which introduces a novel component, the uncertaintyaware utility anchor, to enhance learning stability and effectiveness. we validate uapo on both dpo and simpo, demonstrating that it achieves strong and consistent performance across three benchmarks. unlike traditional approaches that rely solely on pairwise preference data, uapo can incorporate multiple pair data, leading to significant improvement. these findings highlight the versatility and robustness of uapo, making it a promising approach for real-world preference optimization tasks, especially in settings with limited data or computational resources.