Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment (2024)

Abstract: Alignment in artificial intelligence pursues the consistency between model responses and human preferences as well as values. In practice, the multifaceted nature of human preferences inadvertently introduces what is known as the ”alignment tax”–a compromise where enhancements in alignment within one objective (e.g., harmlessness) can diminish performance in others (e.g., helpfulness). However, existing alignment techniques are mostly unidirectional, leading to suboptimal trade-offs and poor flexibility over various objectives. To navigate this challenge, we argue the prominence of grounding LLMs with evident preferences. We introduce controllable preference optimization (CPO), which explicitly specifies preference scores for different objectives, thereby guiding the model to generate responses that meet the requirements. Our experimental analysis reveals that the aligned models can provide responses that match various preferences among the ”3H” (helpfulness, honesty, harmlessness) desiderata. Furthermore, by introducing diverse data and alignment goals, we surpass baseline methods in aligning with single objectives, hence mitigating the impact of the alignment tax and achieving improvements in multi-objective alignment.

## introduction large language models (llms) trained on massive corpora have become surprisingly capable ai assistants of human beings @xcite . to develop these powerful models behaving in accord with human expectation, we need to align their behaviors with a broad spectrum of human preferences and values, which are extremely diverse and inclusive. in principle, previous research * equal contribution. † corresponding authors. 1 our data is open-sourced at https://huggingface.co/ datasets/openbmb/ultrasafety . and our code can be found at https://github.com/openbmb/cpo . we alleviate trade-offs through controlling specific objectives based on user preferences. for example, h 1 corresponds to helpfulness, and h 2 corresponds to honesty. when only h 1 is provided, the optimization direction is confined to a plane. when both h 1 and h 2 are provided, the optimization direction is confined to a straight line. while the "3h" principle sets a foundational guideline, its application reveals a complex interplay, sometimes even controversial with each other. for example, a highly helpful assistant should not decline to answer any user questions even dangerous ones, which violates the harmlessness principle. as a result, improving one alignment objective may come at the expense of a performance decrease of other objectives @xcite , as shown in figure 1 . this trade-off in multiobjective optimization is known as the "alignment tax" @xcite . addressing such performance trade-offs requires innovative approaches. however, most prior alignment techniques are unidirectional @xcite @xcite , meaning that they predominantly focus on optimizing toward a scalar reward signal rather than integrating multi-objective human preferences. to empirically balance these goals, existing work heuristically mixes either alignment data @xcite or reward models @xcite , but fails to address the core challenge-alleviating the tension across alignment objectives in principle. in this work, we recognize controllability as the key to multi-objective alignment. we argue that you can't please all of the people all of the time. for instance, users may prioritize utility for problemsolving questions and concerning moralities for controversial questions. thus, as shown in figure 1 , our approach deviates from the conventional strategy of maximizing preference scores across all objectives. instead, we formulate the controllable multi-objective alignment problem to introduce explicit preference conditions to guide llms towards desirable behaviors while striving to optimize preference scores for other objectives as much as feasible. by narrowing the focus to fewer maximizing objectives, we can alleviate the inherent conflicts among different alignment objectives. specifically, we propose a controllable preference optimization (cpo) algorithm, which consists of two stages: (1) controllable preference supervised fine-tuning (cpsft) which augments the input with explicit preference conditions through preference tokens (such @xmath0:5> @xmath1:1>) and learns to generate responses following the given preference conditions; and (2) controllable direct preference optimization (cdpo) which directly compares the human preference of given responses under the value conditions with a conditional multi-preference value, and then increasing the probability of the better one as well as decreasing the other. we study our proposed cpo algorithm using two typical alignment datasets hh-rlhf @xcite and ultrafeedback @xcite . for jailbreak safety @xcite , we additionally create a preference dataset ultrasafety. experimental results based on the mistral-7b @xcite open-source llm show that (1) cpo can achieve good controllability in a single objective while maintaining the alignment performance compared with dpo; (2) cpo surpasses the original sft, ppo, dpo and curry- @xcite on all three objectives including helpfulness, honesty, and harmlessness, via explicitly grounding the preference conditions. this demonstrates its ability to mitigate the conflict issue related to multi-objective alignment in dpo to some extent, thus achieving improvement. ## method in this section, we propose a controllable preference optimization (cpo) algorithm, of which the main idea is to determine the optimization direction through preference tokens, transforming multi-objective optimization problems into conditional multi-objective optimization problems (section 2.1). specifically, cpo encompasses two stages: controllable preference supervised finetuning (section 2.2.1) and controllable direct preference optimization (section 2.2.2). ## experiments in this section, we evaluate the "3h" metrics (helpfulness, honesty, and harmlessness) in two aspects: controllability on different aspects (section 3.2) and multi-objective alignment evaluation (section 3.3). ## conclusion this paper manages to alleviate the performance trade-off problem in llm alignment. from the view of controllability, we find explicit conditioning is essential for this trade-off. to this end, we propose a novel alignment technique, controllable preference optimization (cpo), containing both supervised fine-tuning as well as preference learning. in evaluation, we validate the excellent flexibility and performance of cpo in aligning with helpfulness, honesty, and harmlessness.