Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization (2024)

Abstract: A single language model, even when aligned with labelers through reinforcement learning from human feedback (RLHF), may not suit all human preferences. Recent approaches therefore prefer customization, gathering multi-dimensional feedback, and creating distinct reward models for each dimension.Different language models are then optimized for various preferences using multi-objective RLHF (MORLHF) with varying reward weights.However, RL fine-tuning is unstable and resource-heavy, especially with diverse and usually conflicting objectives.In this paper, we present Multi-Objective Direct Preference Optimization (MODPO), an RL-free extension of Direct Preference Optimization (DPO) for multiple alignment objectives.Essentially, MODPO folds language modeling directly into reward modeling, training language models as implicit collective reward models that combine all objectives with specific weights. MODPO theoretically yields the same optimal solutions as MORLHF but is practically more stable and efficient.Empirical results in safety alignment and long-form question answering show that MODPO matches or outperforms existing methods, producing a Pareto front of language models catering to diverse preferences with three times less computational resources compared to MORLHF.Code is available at https://github.com/ZHZisZZ/modpo.

## introduction modern transformer-based language models (lms), pre-trained on internet-scale corpus and refined with human feedback, typically align well with a specific group. the primary alignment method, reinforcement learning from human feedback (rlhf), uses a single reward model to represent average labeler preferences, steering a language model to maximize this reward model for desired outcomes @xcite @xcite . while early successes in language model alignment assumed homogeneous human preferences @xcite , actual human preferences vary widely and are hard to satisfy with a single language model @xcite . therefore, recent efforts focus on a multipolicy strategy @xcite , training a collection of candidate language models so that "different models can be deployed and used by groups that endorse different values" @xcite . this customization involves dividing human feedback into multiple fine-grained dimensions and creating distinct reward models for each, such as helpfulness, harmlessness, or honesty @xcite @xcite . different language models are fine-tuned for different preferences using multi-objective rlhf (morlhf) with varying reward weights. iterating over the spectrum of reward weights produces a pareto front of language models, selectable during inference to satisfy customized preferences @xcite . most morlhf pipelines use linear scalarization @xcite to combine multiple reward functions into one, allowing reuse of the standard rlhf pipeline. however, rlhf is complex, unstable, and inefficient. these issues are exacerbated in morlhf due to usually conflicting objectives and the need to train multiple language models to meet diverse needs @xcite . in this paper, we introduce multi-objective direct preference optimization (modpo), an rlfree method extending direct preference optimization (dpo) @xcite for multiple alignment objectives with minimal overheads. our approach folds language modeling early into reward modeling, training different language models to implicitly represent different collective reward models that combine all objectives with specific weightings. while theoretically guaranteed to produce the same optimal solutions as morlhf, modpo is practically more stable and computa- ## background we review two main methodologies for aligning language models with human preferences: homogeneous preference alignment and multi-objective preference alignment. ## multi-objective direct preference optimization (modpo) to address the diversity of human preferences and the complexity of optimizing multiple objectives with rl, we introduce multi-objective direct preference optimization (modpo), a stable and efficient extension of dpo that optimizes eq. 6 exactly without rl. the key insight is that instead of first training parametrized reward models and then using post-hoc linear scalarization to obtain different collective reward models for rl fine-tuning, we can train a series of parametrized collective reward models to directly predict the results of linear scalarization under different w. by parametrizing these collective reward models with language models @xcite , we can directly obtain an empirical front of language models {π θw | @xmath0. assumption. modpo assumes that d contain at least one preference dataset d k . this assumption does not restrict the method's applicability for two reasons: (1) preference feedback is efficient and common @xcite . (2) in the absence of preference data, a random preference dataset can fulfill this requirement, introducing a dummy objective that does not influence the trained language model (see appendix a.3 for further discussions). ## conclusion we have introduced multi-objective direct preference optimization (modpo), an rl-free method that extends direct preference optimization (dpo) for multiple alignment objectives. modpo is a theoretically equivalent but practically more efficient alternative to multi-objective rlhf (morlhf). it optimizes language models using a simple crossentropy loss and demonstrates good empirical results across multiple tasks. while we demonstrate modpo's advantages by evaluating language model fronts, if the target preference is known, we can also treat the preference vector as a tunable hyperparameter to customize a single language model easily. this flexibility makes modpo an effective and accessible way to produce customized language models for diverse preferences.