Safeguard Fine-Tuned LLM s Through Pre- and Post-Tuning Model Merging (2025)

Abstract: Fine-tuning large language models (LLMs) for downstream tasks often leads to catastrophic forgetting, notably degrading the safety of originally aligned models. While some existing methods attempt to restore safety by incorporating additional safety data, the quality of such data typically falls short of that used in the original alignment process. Moreover, these high-quality safety datasets are generally inaccessible, making it difficult to fully recover the model’s original safety. We ask: How can we preserve safety while improving downstream task performance without additional safety data? We show that simply merging the weights of pre- and post-fine-tuned models effectively mitigates safety degradation while enhancing performance. Experiments across different downstream tasks and models validate the method’s practicality and effectiveness.

## introduction the rapid advancement and increasing accessibility of large language models (llms) necessitate a critical focus on aligning these technologies with human values, cultural norms, and trustworthiness @xcite . to address these challenges, researchers and developers have introduced safety techniques such as preference tuning @xcite @xcite , aimed at preventing llms from generating harmful or inappropriate content. many applications now leverage safety-aligned models as foundation models-referred to as aligned models in this paper-to further customize for downstream tasks via supervised fine-tuning (sft) @xcite . however, recent studies @xcite @xcite highlight a critical challenge: fine-tuning aligned models can degrade their safety, even when using benign datasets. to address this issue, mainstream approaches often incorporate additional safety data during fine-tuning ( @xcite . however, since the original safety data used to align llms are rarely available, surrogate data are typically generated by other llms-raising concerns about quality, and the potential for alignment drift. in this paper, we demonstrate a simple yet effective method for improving downstream task performance while mitigating safety degradation. as illustrated in figure 1 , our approach consists of two steps: (1) fine-tune the aligned model on the downstream task, and (2) merge the aligned model with the fine-tuned model. we evaluate this strategy across various models and downstream tasks. experimental results show that this method consistently enhances downstream task performance while substantially preserving model safety, offering a simple and robust solution for fine-tuning safety-aligned llms. our key contributions are: ## methodology our method comprises just two stages: (1) finetuning the aligned model on a target downstream task, and (2) merging the original aligned model with the fine-tuned model by interpolating their weights. despite its simplicity, this merging strategy effectively mitigates the degradation in safety commonly observed following fine-tuning, while preserving performance on the target downstream task, without requiring additional data. step 1: supervised fine-tuning of the large language model we fine-tune the aligned model with parameters θ base on a given task t, resulting in a task-specific model θ t . for each task t, given an instruction x t and its corresponding response y t , we minimize the negative loglikelihood: step 2: merging the fine-tuned model with the aligned model after fine-tuning, we merge the parameters of the aligned model (θ base ) with those of the fine-tuned model (θ t ) via linear interpolation: here, θ merged denotes the parameters of the merged model, and λ ∈ [0, 1] controls the relative contribution of the fine-tuned model. eq. 2 is the formulation for the native linear merging method; other advanced merging methods can also be applied. ## experimental setups downstream tasks we conduct experiments on four downstream tasks: reasoning, medical assistance, code generation, and tool usage proficiency. reasoning is enhanced using chain-of-thought data from the flan v2 dataset @xcite and evaluated on the big bench hard (bbh) dataset @xcite . medical assistance uses patient-doctor dialogues from the chatdoctor dataset @xcite . code generation is trained on the magicoder dataset @xcite and evaluated using the humaneval benchmark @xcite . tool usage proficiency leverages the openfunctions dataset @xcite to improve api call generation. for medical assistance and tool usage proficiency, response similarity to reference answers is measured using bertscore foot_0 @xcite . see appendix a for additional details on the downstream tasks. ## conclusion we present a simple yet effective method to address the safety degradation that often occurs when adapting llms to downstream tasks, without requiring additional safety data or auxiliary models. the method also preserves capabilities such as instruction-following, making it a practical and scalable solution for adapting llms to new tasks.