Model Merging and Safety Alignment: One Bad Model Spoils the Bunch (2024)

Abstract: Merging Large Language Models (LLMs) is a cost-effective technique for combining multiple expert LLMs into a single versatile model, retaining the expertise of the original ones. However, current approaches often overlook the importance of safety alignment during merging, leading to highly misaligned models. This work investigates the effects of model merging on alignment. We evaluate several popular model merging techniques, demonstrating that existing methods do not only transfer domain expertise but also propagate misalignment. We propose a simple two-step approach to address this problem: (i) generating synthetic safety and domain-specific data, and (ii) incorporating these generated data into the optimization process of existing data-aware model merging techniques. This allows us to treat alignment as a skill that can be maximized in the resulting merged LLM. Our experiments illustrate the effectiveness of integrating alignment-related data during merging, resulting in models that excel in both domain expertise and alignment.

## introduction large language models (llms) have demonstrated impressive capabilities, often surpassing human performance across language processing tasks @xcite . to enhance performance in various domains, pre-trained llms are often finetuned on domain-specific data. some examples of domain-specific expert models include openbiollm @xcite , excelling in the biomedical domain, and mammoth @xcite , performing well in stem subjects. since expert models may excel in specific domains only, model merging @xcite figure 1 : safety-aware merging. traditional llm merging techniques can create multi-domain expert models but often transfer misalignment to the merged model. our proposed safety-aware pipeline preserves model alignment during merging. @xcite has been proposed as a technique to combine the strengths of various models into a single, highly capable one. for instance, merging a model proficient in chemistry with another model expert in mathematics aims to create a unified model that performs well in both subjects, often outperforming the individual experts @xcite . this approach is particularly attractive as it allows leveraging the knowledge from numerous open-source models without incurring in high training costs. however, we pose a crucial question that has been overlooked in the literature: how does model merging impact the safety alignment of existing llms? to understand the importance of this question, let us introduce a few notions about safety alignment. safety alignment refers to a model's ability to generate responses that are safe, ethical, and consistent with human values @xcite . in this paper, we refer to a model as aligned if the model has a high safety alignment. conversely, the model is misaligned, i.e. it is lacking necessary safety alignment, as one of the expert models in fig. 1 . in this paper, we find that naively merging a set of expert llms including a misaligned model can result in a misaligned merged model, even if some of the original experts are aligned (fig. 1 , left). this raises substantial concerns for the safe deployment of merged llms, which may expose users to unsafe content. hence, we show the need for safety-aware model merging, where merged models preserve desirable alignment characteristics (fig. 1 , right). to address this issue, we design a simple yet effective approach to combine expert models while preserving alignment. our intuition is that safety alignment should be considered as a task on its own, similar to domain-specific expertise in fields such as biology or physics, and thus it should be optimized for during merging. our approach consists of two stages. first, we generate synthetic data to use for merging. then, building on existing techniques, we use the generated data to perform a data-driven merging optimization procedure, preserving both the alignment and the expertise of the original models. more in detail, we first generate two datasets of questions and associated answers: one for preserving alignment, the other for transferring domain-specific knowledge. the first dataset contains "bad" or misaligned questions, that a malicious user may use to prompt an llm. an example of such a prompt may be "how do i kill someone?". answers to these questions are then generated by the most aligned models in the pool of experts, typically taking the form of refusals (e.g., "i'm sorry, i can't help."). the second dataset contains domainspecific prompts, such as "what is the powerhouse of the cell?" for the biology domain. domainspecific answers (e.g., "mitochondria is the powerhouse of the cell.") are provided by the most expert model in the pool on a specific domain. finally, the collected data are used with data-driven merging approaches @xcite , where we optimize merging minimizing a loss on both alignment and domain-specific data. by doing this, we ensure that the merged model maintains high alignment and domain performance. our contributions are threefold: • we demonstrate that existing model merging techniques fail to explore the inherent trade-off between alignment and domain accuracy. llm alignment ensuring the alignment of llms is crucial. fine-tuning risks were highlighted by qi et al. (2024) and jain et al. (2024), showing that even benign datasets can degrade model safety and careful adaptation protocols are needed to preserve alignment. recently, some techniques to align llm were proposed, such as args (khanov et al., 2024) addressing decoding, figa (guo et al., 2024) for token-level signals, and f-dpo (wang et al., 2024) for efficient alignment. zhao et al. (2023) designed gpo to consider different interest groups. some method enhance generalization (zheng et al., 2024), while dai et al. (2024) proposed safe rlhf, for separate alignment on helpfulness and harmlessness. in salmon (sun et al., 2024), they use synthetic data to reduce human supervision. although these may be effective, we show that model merging can mitigate the effects of alignment procedures. importantly, inan et al. (2023) addressed the need for effective input-output safeguarding in conversational ai with llama guard, employing a safety risk taxonomy and ad hoc models to classify safety concerns in text. model merging techniques for merging multiple models have been proposed as efficient ways to benefit from the capabilities of multiple llms without retraining or accessing the original datasets. in model soups @xcite , they first propose to combine models with weight averaging, showing improved performance compared to a single model. @xcite build on this by performing task arithmetics, i.e. element-wise operations on model parameters to edit their behavior towards specific tasks. similar alternatives are regmean @xcite , and fisher merging @xcite . model merging in non-linear spaces showed improved results, as in slerp @xcite . some, such as ties @xcite and dare @xcite , propose methods to improve model merging, focusing on sparsification. similarly, model breadcrumbs @xcite exploits sparse masks for better combination. importantly, some extend merging capabilities across multiple modalities @xcite . the importance of each model to merge can be automatically tuned with data-driven approaches such as evomm @xcite and lm-cocktail @xcite . none of these approaches consider the safety implications of merging. alignment evaluation advancements in evaluating llms have focused on their robustness, ethical considerations, and safety alignment. prompt-bench @xcite offers a comprehensive benchmark to assess robustness against prompt perturbations, revealing vulnerabilities. recode @xcite proposes a similar setup for code generation. @xcite introduces flask, for a fine-grained assessment of alignment; while @xcite developed auto-j, a flexible generative judge. trustgpt @xcite provides a benchmark for evaluating toxicity, bias, and value alignment. the ethics dataset @xcite assesses understanding of ethics, while moralchoice @xcite analyzes moral beliefs in llms using psychological surveys and high ambiguity dilemmas. beaver-tails @xcite introduces a dataset of over 700,000 questions and answers pairs annotated for helpfulness and harmlessness. jailbreaking attacks' effectiveness is tackled in rigorllm @xcite . to the best of our knowledge, we are the first to evaluate the alignment of merged models. ## preliminaries here, we introduce notions and formalism on model merging. merging aims to combine the specific capabilities of expert models, i.e., models finetuned on domain-specific data, into a single llm. ## conclusions in this work, we have highlighted the effects of model merging in the context of safety alignment for llms. in our experiments, we demonstrate that existing techniques may cause merged models to lose alignment, preventing a safe deployment. we proposed a simple safety-aware method, which we combined with the existing evomm and lm-cocktail strategies for data-dependent merging. by treating alignment as a task in its own right and incorporating alignment data into the merging process, our safety-aware merging pipeline signifi-cantly improves alignment, without compromising domain accuracy.