Unlocking Decoding-time Controllability: Gradient-Free Multi-Objective Alignment with Contrastive Prompts (2025)

Abstract: The task of multi-objective alignment aims at balancing and controlling the different alignment objectives, e.g., helpfulness, harmlessness and honesty) of large language models to meet the personalized requirements of different users. However, previous methods tend to train multiple models to deal with various user preferences, with the number of trained models growing linearly with the number of alignment objectives and the number of different preferences. Meanwhile, existing methods are generally poor in extensibility and require significant re-training for each new alignment objective considered. Considering the limitation of previous approaches, we propose MCA, which constructs an expert prompt and an adversarial prompt for each objective to contrast at the decoding time and balances the objectives through combining the contrast. Our approach is verified to be superior to previous methods in obtaining a well-distributed Pareto front among different alignment objectives.

## introduction aligning large language models (llms) trained on vast web corpora @xcite @xcite with human preferences is an important step to mitigate the production of unsafe @xcite , hallucinated @xcite and biased @xcite contents. with the recent development of preference learning techniques like ppo @xcite , dpo @xcite and other variants @xcite @xcite , there has been progress toward building an open-domain ai assistant that could follow user preferences. however, human preferences are not a fixed standard but vary significantly from person to person. † this work was done during an internship at ucsd. * equal senior authorship. for instance, a ph.d. student inquiring about an academic problem probably expects a factual and informative reply; a five-year-old asking for a virtual playmate would put emphasis on safety and humor. however, it is rather difficult to obtain an ai assistant excelling at all alignment dimensions foot_0 since different alignment dimensions might intrinsically interfere with each other @xcite @xcite . for example, as illustrated in figure 1 , we measure the correlation between the helpfulness and harmlessness of phi-2 @xcite generated response. we find that the performance on the two alignment objectives is negatively correlated, with the spearman's correlation coefficient being ρ = -0.51 for hh-rlhf @xcite and ρ = -0.61 for saferlhf @xcite (@xmath0.01). the negative correlation indicates a potential trade-off between helpfulness and harmlessness. consequently, controllability in multi-objective alignment is vital to satisfy the diverse preferences of different users with a single language model and the task of multi-objective alignment is drawing heated attention @xcite . to control the trade-off between multiple objectives to serve different users, as an initial attempt, @xcite tune a language model for each preference, which is time-consuming and costly. to avoid tuning a language model for all potential preferences, there are generally two concurrent lines of work. on one hand, aggregation-based methods @xcite tune a series of specialized models for each alignment dimension and meet with various preferences through model merging or model ensemble, reducing the numbers of tuned models to the alignment dimensions considered. moreover, instruction-based method @xcite @xcite insert control tokens into the prompt, resulting in a single controllable aligned model. but as a cost, their methods are poor in extensibility since their prompt format is pre-defined on existing alignment objectives and cannot extend to a new alignment objective. therefore, we attempt to reduce the number of trained models further and propose a gradient-free controllable alignment approach that requires no additional model training. getting inspiration from contrastive decoding @xcite , in this study we propose mca. for each alignment dimension, we perform response augmentation with an llm to obtain responses with different rewards. the responses with maximum or minimum reward then serve as demonstrations to induce an expert prompt and an adversarial prompt, which are used for promoting and suppressing the corresponding alignment dimension, respectively. the predictions in the logit space induced by the two prompts then constitute a contrast for the language model. by manipulating the weight of the contrast, users can control the language model at their own preference and incorporate any new required alignment objectives at decoding time if necessary. overall, our contribution can be summarized as: n is the number of alignment objectives considered and m is the number of preferences (i.e., a set of weight coefficients for different alignment objectives). despite the preference data being collected from crowd workers with diverse backgrounds, previous alignment techniques mostly fit on the "average" preference of the crowd while overlooking the personalized preference @xcite . furthermore, @xcite theoretically proves the impossibility of alignment with a single reward in rlhf, which is too restrictive to reflect the opinion and preference of some minority groups @xcite , leading to a biased language model. ## method in this section, we present a new multi-objective alignment framework to manipulate the trade-off between conflicting alignment dimensions. we start with the problem formulation for multiobjective alignment in section 3.1. next, we elaborate on our simple two-step framework in which we first construct an expert prompt and an adversarial prompt for each alignment objective (section 3.2), and then employ the constructed prompt pair for contrastive decoding (section 3.3). by contrasting and combining the next-token probability induced by different prompts at inference time, we attain better flexibility and controllability over different alignment dimensions with no parameter updates. ## analysis apart from verifying the efficacy of our approach, to have a better understanding of its working mechanism, we further conduct the following experimental analysis: ## conclusion in this work, we focus on the multi-objective alignment problem and propose a new gradient-free approach as a possible solution. by contrasting and combining the logits at decoding time, mca is verified to extend the original frontier between different alignment objectives by an obvious margin on various backbones and datasets. meanwhile, we observed that the relationship between objectives can change under different circumstances, and we plan to work on the complex interrelation between multiple alignment objectives in the future.