STAR : Spectral Truncation and Rescale for Model Merging (2025)

Abstract: Model merging is an efficient way of obtaining a multi-task model from several pretrained models without further fine-tuning, and it has gained attention in various domains, including natural language processing (NLP). Despite the efficiency, a key challenge in model merging is the seemingly inevitable decrease in task performance as the number of models increases. In this paper, we propose **S**pectral **T**runcation **A**nd **R**escale (STAR) that aims at mitigating “merging conflicts” by truncating small components in the respective spectral spaces, which is followed by an automatic parameter rescaling scheme to retain the nuclear norm of the original matrix. STAR requires no additional inference on original training data and is robust to hyperparamater choice. We demonstrate the effectiveness of STAR through extensive model merging cases on diverse NLP tasks. Specifically, STAR works robustly across varying model sizes, and can outperform baselines by 4.2% when merging 12 models on Flan-T5. Our code is publicly available at https://github.com/IBM/STAR.

## introduction with the popularity of pretrained models on large neural networks, the same architecture is often deployed to fine-tune individual natural language processing (nlp) tasks. a natural question then arises about whether it is possible to merge these samearchitecture fine-tuned models into one multi-task model. for example, researchers are interested in understanding if we can empower a fine-tuned conversational large language model (llm) with reasoning capabilities by merging with an llm specializing in solving math problems. specifically, @xcite has formally defined a task vector as θ ftθ pre , where θ pre and θ ft denote the vectorized parameters of the pre-trained model and the fine-tuned model, respectively. thus, task vectors figure 1 : the averaged normalized performance of flan-t5-base merged models by ties @xcite , metagpt @xcite , tall-masks @xcite , and star (this paper). mark the updates made to the pretrained model's weights when fine-tuned on specific tasks. then, model merging essentially studies ways of fusing different task vectors that are trained separately and merging them with the pretrained model. however, as the number of fine-tuned models increases, the multi-task performance of their merged model also decreases drastically. fig. 1 shows the averaged normalized performance (y-axis) v.s. the number of models merged (x-axis). furthermore, we point out that when the number of models exceeds a certain threshold, the multi-task performance of the merged model could be even worse than that of the original pretrained model, diminishing the fundamental goal of model merging. for example, ties @xcite , metagpt @xcite , and tall-masks @xcite merged models drop below 0.82 when we merge 6, 5, and 7 fine-tuned models, respectively, in fig. 1 . the complexity of existing model merging methods varies largely depending on whether they require fine-tuning or inference on training data @xcite . in this paper, we study the "data-free" setting when we are not authorized to change the fine-tuning protocol nor do we have access to the training data. in this work, we propose to use spectral decomposition (e.g. singular value decomposition, svd) to remove noisy components on model merging. we will also motivate the potential gain of our spectral space merging scheme by comparing the upper bounds of the task conflicts. a rescaling step is then followed to restore the original nuclear norm. we give the overview of the proposed method in fig. 2 . our proposed merging scheme, spectral truncation and rescale (star), is effective and efficient as it requires no additional inference on original training data and is not sensitive to hyperparameters. our extensive experimental results show that star is superior across various model size settings and can effectively merge up to 20 models while achieving positive performance gains, compared to the pretrained model before merging. ## methodology sec. 3.1 provides the rationale behind performing truncations in the spectral space. sec. 3.2 defines the rescaling step for restoring the nuclear norm. sec. 3.3 gives the complete star algorithm. ## conclusion in this paper, we propose spectral truncation and rescale (star) for model merging by removing noisy components via spectral decomposition and restoring the original nuclear norm through rescaling. star requires no additional inference and is robust to different hyperparameter choices and language models. star provides a principaled way of automatic rank determination and is intuitively complimentary to other merging methods.