M id PO : Dual Preference Optimization for Safety and Helpfulness in Large Language Models via a Mixture of Experts Framework (2025)

Abstract: As large language models (LLMs) are increasingly applied across various domains, enhancing safety while maintaining the helpfulness of LLMs has become a critical challenge. Recent studies solve this problem through safety-constrained online preference optimization or safety-constrained offline preference optimization. However, the safety-constrained online methods often suffer from excessive safety, which might reduce helpfulness, while the safety-constrained offline methods perform poorly in adaptively balancing safety and helpfulness. To address these limitations, we propose MidPO, a Mixture of Experts (MoE) framework for safety-helpfulness dual Preference Optimization. Firstly, MidPO devises single-preference enhanced direct preference optimization approach to transform the base model into two independent experts, termed safety and helpfulness experts, and fine-tunes the two independent experts for optimal safety or helpfulness performance. Secondly, to achieve an effective balance between safety and helpfulness, MidPO incorporates the two experts into the MoE framework and designs a dynamic routing mechanism to allocate contributions from each expert adaptively. We conduct quantitative and qualitative experiments on three popular datasets to demonstrate the proposed MidPO significantly outperforms state-of-the-art approaches in both safety and helpfulness. Code is available at https: //github.com/OutdoorManofML/MidPO.

## introduction large language models (llms) have achieved remarkable achievements across various natural language processing (nlp) tasks @xcite @xcite . through further fine-tuning of llms based on human preferences, also known as model alignment, their generative capabilities can be greatly enhanced in terms of helpfulness, such as coding ability @xcite and stylistic coherence @xcite . however, leveraging such powerful generative capabilities, llms can also be guided to produce harmful content. it has been proven that responses generated by llms have issues such as misinformation @xcite , offensive language @xcite and personal data leakage @xcite . as llms continue to gain widespread applications, the safety risks they incur are also growing. therefore, improving safety while preserving the helpfulness of llms has become crucial. to achieve the objective of safety-helpfulness dual preference optimization, existing studies can be divided into two mainstream methods based on the sampling strategy employed during fine-tuning: safety-constrained online alignment and safetyconstrained offline alignment @xcite . the safety-constrained online alignment first trains two independent reward models separately for safety and helpfulness, and then employs multi-objective reinforcement learning with human feedback (rlhf) to achieve dual preference optimization @xcite @xcite . however, since the reward model is not a perfect proxy for human preferences, excessively optimizing its value may degrade the llm's true performance, which is in accordance with goodhart's law @xcite . as shown in figure 1 -q1 (a), the llm exhibits immoderate safety behaviors, leading to over-refuse of users' safe request, which reduces the llm's helpfulness @xcite . in contrast, the safety-constrained offline alignment approaches typically introduce a safety margin controlled by a fixed hyper-parameter, which is equivalent to adding a safety preference loss compared to conduct direct preference optimization (dpo) on helpfulness @xcite @xcite . however, as shown in figure 1 -q2 (b), when faced with unsafe inputs, safetyconstrained offline alignment excessively exposes the helpfulness and generates harmful response, resulting in poor performance for balancing safety and helpfulness. to tackle the above challenges, we propose the midpo framework for dual preference optimization of safety and helpfulness, in which two core components are devised, respectively single preference enhanced experts and the dynamic routing mechanism. first, we propose a single-preference enhanced dpo method (spe-dpo). by applying spe-dpo with lora fine-tuning, the base model can be transformed into two safety and helpfulness enhanced experts. second, we design a dynamic routing mechanism that integrates the lora weights of both experts into the moe framework and adopts an adaptive router to allocate the experts' contributions. this dynamic routing mechanism is significantly effective in achieving a tradeoff between safety and helpfulness. extensive experiments demonstrate that midpo can effectively improve the safety and helpfulness of llms simultaneously. our main contributions are summarized as follows: ## midpo framework: dual preference optimization for safety-helpfulness the overview of our proposed midpo framework is shown in figure 2 , which mainly includes two core components: single-preference enhanced experts and the dynamic routing mechanism. we illustrate the detailed designs in the following paragraphs. ## experiment we answer the following three research questions to manifest the contributions as claimed in the introduction section: we provide the case study to demonstrate that midpo surpasses existing safety-constrained alignment methods (section 4.4). in addition, we implement human evaluation in appendix d and conduct efficiency analysis in appendix e. ## related work safety alignment. for the safety-constrained online alignment, @xcite ) considered harmfulness as the cost and utilized the lagrangianbased method for preference optimization. @xcite proposed to train separate reward models, and utilized a piecewise function to combine the preferences. @xcite introduced the rule-based rewards and employed the learned network to integrate preferences. as for safety-constrained offline alignment, (zhou et al., ## conclusion in this paper, we introduce midpo, a moe framework designed for safety-helpfulness preference optimization. specifically, midpo fine-tunes two single-preference enhanced experts via spe-dpo to separately improve safety and helpfulness. to combine these two preferences, midpo integrates both experts into the moe framework and utilizing a dynamic routing mechanism for to balance safety and helpfulness. experimental results on three widely used datasets show that midpo significantly outperforms existing methods in both safety and helpfulness.