Arithmetic Control of LLM s for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards (2024)

Abstract: Fine-grained control over large language models (LLMs) remains a significant challenge, hindering their adaptability to diverse user needs. While Reinforcement Learning from Human Feedback (RLHF) shows promise in aligning LLMs, its reliance on scalar rewards often limits its ability to capture diverse user preferences in real-world applications. To address this limitation, we introduce the Directional Preference Alignment (DPA) framework. Unlike the scalar-reward RLHF, DPA incorporates multi-objective reward modeling to represent diverse preference profiles. Additionally, DPA models user preferences as directions (i.e., unit vectors) in the reward space to achieve user-dependent preference control. Our method involves training a multi-objective reward model and then fine-tuning the LLM with a preference-conditioned variant of Rejection Sampling Finetuning (RSF), an RLHF method adopted by Llama 2. This method enjoys a better performance trade-off across various reward objectives. In comparison with the scalar-reward RLHF, DPA offers users intuitive control over LLM generation: they can arithmetically specify their desired trade-offs (e.g., more helpfulness with less verbosity). We also validate the effectiveness of DPA with real-world alignment experiments on Mistral-7B. Our method provides straightforward arithmetic control over the trade-off between helpfulness and verbosity while maintaining competitive performance with strong baselines such as Direct Preference Optimization (DPO).

## introduction large language models (llms) @xcite have demonstrated remarkable capabilities across various domains and tasks, such as mathematical reasoning @xcite and medical question answering @xcite @xcite . however, for an assistant to be truly useful, it must align with human preferences, such as being helpful, honest, harmless, and managing verbosity. reinforcement learning from human feedback (rlhf) @xcite @xcite @xcite , is the leading approach to adapt llms towards these complex, often implicitly-defined goals. typically, the most popular rlhf framework @xcite @xcite first constructs a scalar reward model to represent the difficult-to-specify goal of being preferred by human and then use this reward model to provide signals for the subsequent reward optimization stage. its success spans various practical applications, including recommendation systems @xcite , image generation @xcite @xcite , robotics @xcite , and most notably, aligning llms with human values and preferences, such as chatgpt @xcite , claude @xcite , llama 2 @xcite and gemini @xcite . while recent advancements in rlhf are noteworthy, a fundamental challenge persists due to problem misspecification. this means that a single reward function may not sufficiently capture complex human values. for example, a generative model aligned by rlhf for helpfulness tends to produce verbose responses as shown in figure 1 (left) @xcite , even though many users prefer answers that are both helpful and concise. assuming scalar-objective reward implies a total order over preferences, which is hard to satisfy when the preference is aggregated across a diverse set of human groups @xcite , because humans typically have a set of intricate or even contradictory targets @xcite . in real-world applications, the 1 8642 in the case of @xcite , it is capable of generating helpful responses, but these tend to be excessively verbose. conversely, with our dpa (right), it allows for arithmetic control of llms to meet various user preferences. for instance, setting the directional preference (unit vector) to v " ⟨0.8, ´0.6⟩ leads to less verbose responses from our aligned llm. scalar-reward rlhf tends to align the llms toward an "average-user" preference, which cannot capture the complicated nature of human preferences and can be unfair for the under-represented groups @xcite . for example, consider user-1, 2, 3, and responses a, b, c in fig. 2 (left). user-1 and 3 prefer response b over c (b ă c), while user-2 prefers c over b (c ă b). this could occur as response c is more verbose than b, while user-2 prefers concise answers. when these diverse preferences are aggregated across human groups, the typical reward models with scalar rewards tend to learn the "average-user" preference (which is b ă c in this case), overlooking the individual preference of user-2, as shown in figure 2 (middle). this is also known as the "condorcet paradox" in the theory of social choice @xcite . in general, human opinions and expertise can vary significantly @xcite @xcite . meanwhile, the importance of these targets may also change over time, depending on the users and their expectations. to address the limitations of the existing scalar reward model, previous works suggest the use of multi-objective rewards that characterize human preferences from different aspects (e.g., helpfulness, verbosity, harmlessness) @xcite . one common way is to take the human feedback as a multi-dimensional reward vector and each dimension models one objective @xcite . then, one may apply a linear combination to transform the multi-objective rewards into a scalar for llm alignment @xcite . however, this approach still cannot handle the userdependent needs from a diverse user population and can be unfair for minority groups. one may further adopt a user-dependent linear combination to multi-objective rewards for aligning a model for each user preference @xcite . however, this approach is quite inferenceunfriendly because we have to switch between different models in response to the different user preferences. finally, in social choice theory, a gamebased formulation was studied under the name maximal lotteries @xcite , as well as the subsequent works in rlhf @xcite @xcite , to handle the diversity of user preferences. we remark that their framework is fundamentally different from the multi-objective rewards and cannot offer a user-dependent preference control in the inference stage, either. refer to section 2.3 for a more detailed discussion with existing methods. in recognition of the aforementioned limitations, we propose a novel and practical alignment approach, directional preference alignment (dpa), to enhance the adaptability and controllability of 2 8643 responses a b c ≺ ≺ labelling a b c prompt reward model a b c ≺ ≺ explain the moon landing ## directional preference alignment in a typical rlhf pipeline @xcite @xcite , we first construct a reward model based on a labeled preference dataset (e.g., preference a ă b ă c annotated by a labeler) and then use the reward model to provide supervision for the subsequent reward optimization stage. in this section, we first present the problem setup, where we additionally consider multi-objective rewards and user preferences in the framework. then, we present our algorithm, the directional preference alignment, to handle the problem of preference-aware alignment. notation. we denote the prompt space and the response space as x and y, respectively. s k " tv p r k : }v} 2 " 1u is the unit sphere under the ∥¨∥ 2 norm. we use π θ to denote the policy (generative) llm whose parameter is θ. ## empirical results we conduct experiments on mistral-7b @xcite , focusing on two reward objectives: helpfulness and verbosity. our proposed dpa achieves arithmetic control of llm generations for different helpfulness-verbosity preferences while demonstrating an excellent balance between the two objectives. verbosity bias. recently, the verbosity bias in llms and humans, meaning that llms and humans sometimes prefer more verbose answers even though they are of similar qualities, has attracted considerable attention @xcite . it has been exploited or even "hacked" by the rlhf-aligned models. for instance, @xcite demonstrated that 77% of chatgpt answers are verbose, while @xcite found that the average output length increases to 2.5 times as the dpo iterates. preliminary experiments have been conducted in response to this bias, such as those by @xcite , which explicitly consider verbosity as a response feature. benchmark creators like alpacaeval @xcite and mt-bench @xcite have observed verbosity bias in their llm judges (typically gpt-4), and alpacaeval-2.0 has adjusted to account for output length foot_0 . ## limitations a primary constraint of our dpa framework is its reliance on a robust multi-objective reward model. the efficacy of dpa is intrinsically linked to the precision and discriminative capability of this reward model. should the reward model not adequately capture the subtleties of specific preferences or exhibit bias in its reward distribution, the dpa might inadvertently exacerbate these shortcomings throughout the fine-tuning process. furthermore, if the reward model fails to recognize harmful content, it could lead the aligned model to produce such content during inference. ## conclusion in this paper, we introduce directional preference alignment (dpa) to incorporate multidimensional user preferences. dpa addresses the limitation of conventional scalar reward models by alleviating conflicting user preferences through a highdimensional preference vector in a multidimensional space. we demonstrate that dpa efficiently explores the pareto front in the multidimensional reward space, revealing a more effective trade-off between helpfulness and verbosity on mistral-7b compared to existing strong baselines such as dpo.