Sens-Merging: Sensitivity-Guided Parameter Balancing for Merging Large Language Models (2025)

Abstract: Recent advances in large language models have led to numerous task-specialized fine-tuned variants, creating a need for efficient model merging techniques that preserve specialized capabilities while avoiding costly retraining. While existing task vector-based merging methods show promise, they typically apply uniform coefficients across all parameters, overlooking varying parameter importance both within and across tasks. We present Sens-Merging, a sensitivity-guided coefficient adjustment method that enhances existing model merging techniques by operating at both task-specific and cross-task levels. Our method analyzes parameter sensitivity within individual tasks and evaluates cross-task transferability to determine optimal merging coefficients. Extensive experiments on Mistral 7B and LLaMA2 7B/13B models demonstrate that Sens-Merging significantly improves performance across general knowledge, mathematical reasoning, and code generation tasks. Notably, when combined with existing merging techniques, our method enables merged models to outperform specialized fine-tuned models, particularly in code generation tasks. Our findings reveal important trade-offs between task-specific and cross-task scalings, providing insights for future model merging strategies.

## introduction the rapid advancement of large language models (llms) has significantly enhanced performance across a diverse range of tasks @xcite . as these models continue to be fine-tuned for specialized domains, the necessity to merge these specialized models into a unified framework becomes increasingly critical @xcite . while multi-task learning has been proposed as a solution, † corresponding author. it incurs substantial training costs and requires simultaneous access to data and labels for all tasks @xcite . recently, researchers have developed parameter-level model merging methods that not only comply with data privacy regulations but also improve efficiency by eliminating the need for retraining @xcite . in the context of model merging, task vectors @xcite have emerged as a powerful component for representing task-specific capabilities. these vectors, defined as the differences between parameter values before and after fine-tuning, enable effective integration of specialized knowledge from different models. while task vector-based merging methods @xcite have shown promising results, their reliance on uniform coefficients for each task and parameter limits their potential effectiveness. this uniformity implies that every task and every parameter is treated with equal importance during the merging process. consequently, it overlooks the fact that parameters within each layer demonstrate varying levels of importance for specific tasks, and parameters from different tasks contribute distinctly during the merging process. to address these challenges, we propose sens-merging, a sensitivity-guided merging coefficient adjustment method that functions as a plug-andplay enhancement to existing task vector-based merging techniques. our method operates at two levels: within individual tasks and across different tasks, allowing for fine-grained control over parameter importance. within each task-specific model, we perform parameter sensitivity analysis to highlight critical layers that significantly impact performance. concurrently, across different tasks, we conduct task sensitivity analysis to prioritize models that enhance the performance of others. by combining these two factors, we derive the final merging coefficients, which are then applied to merge the corresponding layers. figure 1 highlights how sens-merging enhances existing task-vector techniques like task arithmetic @xcite and dare @xcite . notably, when combined with dare, sens-merging enables merged models to outperform specialized fine-tuned models, particularly in code generation. to empirically demonstrate the effectiveness of sens-merging, we conduct extensive experiments by combining it with existing model merging approaches. we merged three widely adopted finetuned models-specializing in general knowledge (chat), mathematical reasoning (math), and code generation (code)-derived from the llama2-7b/13b and mistral 7b families. the integration of our sens-merging not only improves baseline merging performance but enables merged models to surpass individual fine-tuned models. notably, when merging code model with math and chat models using sens-merging, it achieves superior performance on coding tasks compared to codespecific fine-tuning alone. these results indicate that model merging can effectively address the challenges of training a single model for complex tasks by integrating the specialized capabilities of multiple fine-tuned models. to sum up, our contributions include: (1) we propose a novel model merging coefficient determination method based on both task-specific and cross-task sensitivity analysis. (2) through comprehensive evaluations, we validate that our proposed method enhances model merging performance across various domains. (3) we empirically demonstrate that different task-specific models contribute unequally to model merging, and parameter importance varies across different layers within each model. (4) we validate that each scaling ap-proach presents distinct trade-offs: task-specific scaling excels in specialized domains like mathematics but offers limited general benefits, while cross-task scaling achieves broader performance gains at the cost of peak task-specialized performance. ## related work modeling merging @xcite , as a complementary approach to training-based methods, has the capability to integrate multiple task-specialized models into a unified one @xcite @xcite , to improve model performance on individual tasks by merging checkpoints without requiring additional training @xcite , and to alleviates the issue of catastrophic forgetting @xcite . according to whether the based models are in same architecture, the model merging methods can be divided into heterogeneous model merging and homogeneous model merging. ## methodology our sens-merging method combines two levels of sensitivity analysis: layer-wise analysis within individual models and cross-task analysis across different models to achieve a balanced parameter competition. for layer-wise analysis, we compute sensitivity scores using gradient information from calibration datasets. for cross-task analysis, we evaluate model alignment through logits comparison. these two components determine the final merging coefficients used to merge corresponding layers into a unified model, as shown in figure 2 . ## conclusion we introduce sens-merging, a novel method that determines model merging coefficients by analyzing parameter sensitivity both within specific tasks and across multiple tasks. through extensive evaluation on mistral 7b and llama-2 7b/13b model families, we demonstrate that sens-merging enhances model merging performance across multiple domains, consistently outperforming both existing merging techniques and individually fine-tuned models. this improvement is particularly pronounced in code generation tasks, where merged models achieve superior results compared to specialized fine-tuning.