AM o PO : Adaptive Multi-objective Preference Optimization without Reward Models and Reference Models (2025)

Abstract: Existing multi-objective preference alignment methods for large language models (LLMs) face limitations: (1) the inability to effectively balance various preference dimensions, and (2) reliance on auxiliary reward/reference models introduces computational complexity. To address these challenges, we propose Adaptive Multi-objective Preference Optimization (AMoPO), a novel framework that achieves dynamic balance across preference dimensions. By introducing the multi-objective optimization paradigm to use the dimension-aware generation metrics as implicit rewards, AMoPO aligns LLMs with diverse preferences without additional reward models or reference models. We introduce an adaptive weight assignment mechanism that models the generation space as a Gaussian distribution, allowing dynamic prioritization of preference dimensions. Empirical results demonstrate that AMoPO outperforms state-of-the-art baselines by 28.5%, and the experiments on 7B, 14B, and 32B models reveal the scaling ability of AMoPO. Moreover, additional analysis of multiple dimensions verifies its adaptability and effectiveness. These findings validate AMoPO’s capability to achieve dimension-aware preference alignment, highlighting its superiority. Our codes and datasets are available at https://github.com/Javkonline/AMoPO.

## introduction recent breakthroughs in large language models (llms) @xcite @xcite have intensified the need for alignment with multidimensional human preferences, which is a critical requirement for deploying safe and controllable ai systems. while preference optimization methods like rlhf @xcite and dpo series @xcite @xcite have advanced single-objective alignment, fundamental limitations persist when addressing the multidimensional nature of real-world human preferences @xcite @xcite . figure 1 shows that different llms favor different criteria but struggle to balance these dimensions, highlighting the need for more comprehensive multi-objective alignment methods. existing multi-objective alignment methods can be categorized into two main branches. on the one hand, instruction-controlled methods @xcite @xcite insert control tokens into the prompt for each preference. however, these methods are constrained by predefined prompts and lack adaptability in aligning with various dimensions. on the other hand, multi-model integration methods involve training multiple reward models, or even language models, for each preference dimension. for example, methods like morlhf @xcite and reward soups @xcite require training multiple reward models, while modpo @xcite integrates multiple preference objectives into the margin calculation of the reward model. hm3 @xcite uses model merging to combine pre-trained models across different dimensions. obviously, these methods are time-consuming and costly. to alleviate these issues, we aim to reduce the reliance on multiple models and enhance adaptability to multiple dimensions by proposing adaptive multi-objective preference optimization (amopo). firstly, we propose a multi-objective optimization framework by designing the multidimensional alignment model, enabling it to focus more precisely on each dimension. by integrating generation metrics @xcite of language models as implicit rewards, our framework allows llms to inherently understand and align with the meaning associated with different dimensions. furthermore, we develop an adaptive weight assignment mechanism that automatically determines the importance of each dimension based on the modeled output space. the contributions of this paper can be summarized as follows: ## preliminary bradley-terry (bt) reward model. given an llm π θ parameterized by θ, query x, and response y, classical preference alignment methods can be formulated as: in measuring human preferences, one of the most commonly used approximations is the bradley-terry (bt) reward model @xcite @xcite . assumed r * is the latent ground-truth reward function, the bt model defines the preference distribution π * as: ## methodology this section begins with the process of data preparation. we then present our amopo by exploring the multi-objective derivation and the adaptive weight assignment mechanism. the complete pipeline of our method is illustrated in figure 2 . ## conclusions in this paper, we propose an adaptive multiobjective preference optimization (amopo) framework to effectively address the limitations of existing preference alignment methods in capturing the complex, multi-dimensional nature of human preferences. by introducing a new mobt paradigm and using generation metrics as implicit rewards across multiple dimensions, amopo achieves precise alignment with diverse preferences without the need for additional reward models or reference models, ensuring it remains lightweight and effective. additionally, we incorporate the adaptive weight assignment mechanism, which models the llm generation space as a gaussian distribution and allows for dynamic determination of the importance of each preference dimension, thereby enhancing adaptability and efficiency. empirical results demonstrate that amopo outperforms state-of-the-art baselines by 28.5% across multiple benchmarks, showcasing its superior performance. furthermore, detailed analysis across multiple dimensions confirms its adaptability and effectiveness.