Self-Improvement Towards P areto Optimality: Mitigating Preference Conflicts in Multi-Objective Alignment (2025)

Abstract: Multi-Objective Alignment (MOA) aims to align LLMs’ responses with multiple human preference objectives, with Direct Preference Optimization (DPO) emerging as a prominent approach. However, we find that DPO-based MOA approaches suffer from widespread preference conflicts in the data, where different objectives favor different responses. This results in conflicting optimization directions, hindering the optimization on the Pareto Front. To address this, we propose to construct Pareto-optimal responses to resolve preference conflicts. To efficiently obtain and utilize such responses, we propose a self-improving DPO framework that enables LLMs to self-generate and select Pareto-optimal responses for self-supervised preference alignment. Extensive experiments on two datasets demonstrate the superior Pareto Front achieved by our framework compared to various baselines

## introduction aligning large language models (llms) with human preferences @xcite has evolved from single-objective to multi-objective, aiming to comprehensively capture the inherent heterogeneity of human preferences. multi-objective alignment (moa) @xcite @xcite has jointly considered multiple human preference objectives, such as safety, helpfulness, factuality, and diversity, to optimize the llm. the optimization outcome of moa is a set of llms optimized under various preference weights across these objectives, forming a (close-to) pareto front. existing moa approaches can be broadly classified into two categories by their optimization strategies. reinforcement learning (rl)-based approaches @xcite learn a proxy reward model for each objective and then update the llm using rl, targeting at a weighted combination of the proxy rewards. direct preference optimization (dpo)-based approaches @xcite follow a distinct paradigm, where dpo optimization targets are derived for each objective and jointly aggregated under the preference weight. since dpo-based methods offer advantages in cost and stability over rl-based approaches @xcite , it has been a promising direction to study moa via dpo. however, after comprehensively investigated dpo-based moa (cf. section 2), we observe that these approaches are prone to be impacted by the widespread preference conflicts in the training data, which hinders the achievement of superior pareto front. given question and a pair of responses, different objectives often favor different responses, resulting in preference conflicts among these objectives. these preference conflicts create contra-dictory optimization targets for different objectives under the aggregation of dpo-based approaches, potentially disrupting the alignment toward each objective and hindering the achievement of superior pareto front (see analysis in section 2). given the high prevalence of conflicting preferences in existing datasets (cf. to solve the issue of preference conflicts, we propose to construct pareto-optimal responses (cf. figure 1 ). given a pair of responses with conflicting preferences, denoted as a and b, where a is better than b in objective 1 but worse in objective 2 we propose to construct a pareto-optimal response c, which surpasses a and b on both objectives. we think learning the preference between c and a for objective 1 and c and b for objective 2 not only incurs no preference conflicts, but also guides the llm toward generating responses that perform well on both objectives, thus achieving a superior pareto front. to obtain pareto-optimal responses, it is not advisable to manually write due to the large amount of preference conflict instances. therefore, we consider using automatic approach to obtain pareto-optimal responses from the llm itself. to this end, we propose a novel self-improvement dpo framework towards pareto optimality (sipo), which guides the llm to self-generate and select pareto-optimal responses, thereby mitigating preference conflicts and enhancing the pareto front. after initial alignment, sipo samples high-quality responses with a self-refinement strategy, which are then evaluated and filtered for pareto-optimality over original responses. finally, the pareto-optimal responses are paired with original responses for non-conflicting dpo-based preference fine-tuning. sipo can be easily incorporated with existing dpo-based moa approaches. experimental results on helpsteer @xcite and beavertails @xcite demonstrate significant improvement over baseline methods. our contributions are three-fold: ## preliminary experiments background the alignment objectives are denoted as a set of n ground-truth reward functions, r * (x, y) = [r * 1 (x, y), ..., r * n (x, y)] ⊺ . the goal of moa is to align the llm based on a set of preference weights @xmath0. each preference weight vector w @xmath1, ..., w m n ] ⊺ satisfies the constraint n @xmath2, which balances these objectives. aligning the llm to a given preference weight entails maximize the weighted reward w ⊺ r * (x, y). the resulting set of aligned llms form a (close-to) pareto front. the alignment is typically achieved using a multi-objective preference dataset, @xmath3, ..., d n }, where d @xmath4, y w , y l )} represents the preference dataset for objective i. here, x is the input, while y w and y l denote the preferred and dispreferred responses, respectively. frequently, the inputs and responses remain the same across all preference datasets in d, with only the preference labels differing across objectives, as this format simplifies the annotation process for human annotators. thus we can reformulate the dataset as @xmath5, y -1 , y 1 , p 1 , ..., p n )}, p @xmath6, 1} as the label of the preferred response for objective i. ## method in this section, we introduce our sipo framework (cf. figure 5 ), which leverages self-generated pareto-optimal responses to mitigate the impact of preference conflicts. we will introduce the definition of pareto-optimal responses ( § 3.1), and detail the sipo framework design ( § 3.2, § 3.3). ## experiments experimental setup we conduct experiments on two widely-used moa datasets. beavertails @xcite ) contains ai safety-related questions, aiming for harmless and helpful llm responses. we utilize the beavertails-10k subset and split the training and validation data as 9:1, and utilize an additional split from the beavertails-30k dataset as the test data. helpsteer aims to promote response helpfulness, where we focus on two objectives, correctness, i.e., factuality precision and relevance, and verbosity, i.e., response length and level of detail. since helpsteer is not formulated as our definition of d, we manually transform the dataset to follow the definition. dataset preprocessing details and statistics can be found in appendix a.1. for backbone llms, we mainly utilize supervised fine-tuned llama-2-7b @xcite , denoted as llama-2-7b-sft. we fine-tune all the responses in the training dataset to obtain a llama-2-7b-sft for each dataset. we also conduct experiments on alpaca-7b @xcite to show the applicability of sipo on different llms (see more details in appendix a.3). ## conclusion this paper addressed the negative impact of preference conflicts on achieving a superior pareto front in dpo-based moa. through extensive analysis and experiments, we revealed the impact of preference conflicts on pareto front optimization. to mitigate this issue, we proposed sipo, a framework that automatically generates and leverages paretooptimal responses to resolve preference conflicts, which outperformed baseline methods in achieving a superior pareto front. in the future, we plan to extend our experiments to more objectives and additional dpo-based methods. we will also explore how to improve the efficiency of obtaining pareto-optimal responses to reduce the cost.