L o RE -Merging: Exploring Low-Rank Estimation For Large Language Model Merging (2025)

Abstract: While most current approaches rely on further training techniques, such as fine-tuning or reinforcement learning, to enhance model capacities, model merging stands out for its ability of improving models without requiring any additional training. In this paper, we propose a unified framework for model merging based on low-rank estimation of task vectors without the need for access to the base model, named LoRE-Merging. Our approach is motivated by the observation that task vectors from fine-tuned models frequently exhibit a limited number of dominant singular values, making low-rank estimations less prone to interference. We implement the method by formulating the merging problem as an optimization problem. Extensive empirical experiments demonstrate the effectiveness of our framework in mitigating interference and preserving task-specific information, thereby advancing the state-of-the-art performance in model merging techniques.

## introduction large language models (llms) have become ubiquitous in numerous real-world applications @xcite . the utilization of llms typically involves fine-tuning them for specific tasks, a process that often yields superior performance compared to general-purpose llms. a rapidly emerging technique in this domain is model merging @xcite @xcite , which aims to create a single multi-task model by combining the weights of multiple task-specific models. this approach facilitates the construction of multi-task models by integrating knowledge from fine-tuned (ft) models without requiring additional training. building on recent studies @xcite @xcite , task vectorbased merging approaches have demonstrated significant effectiveness, where task vectors are de-fined as the parameter differences between finetuned models and the base llm. achieving optimal results in model merging often requires minimizing interference between task vectors associated with different tasks. to address this, existing approaches utilize modified task vectors instead of the original ones. for instance, @xcite applied random dropping with probability p to obtain a sparse representation of task vectors, while @xcite retained only the top-k elements of each task vector based on magnitude, setting the remaining elements to zero. these strategies aim to produce sparse estimations of task vectors, a common technique for mitigating interference. nevertheless, task vector-based model merging approaches remain constrained by two fundamental limitations. first, the computation of task vectors necessitates access to the base model parameters and demonstrates heightened sensitivity to parametric variations @xcite . as fine-tuning progress goes deeper, substantial parametric divergence emerges between the original base model and its fine-tuned counterpart, thereby greatly hindering them merging effectiveness @xcite . second, empirical evidence from @xcite reveals that conflicting task vectors interactions could appear even when employing sparse estimation techniques. on the other hand, the sparsification process risks inadvertently eliminating essential task-specific features, thereby compromising the efficacy of the resultant merged model. these inherent constraints of sparse approximation methodologies underscore the necessity for developing alternative frameworks to estimate higherfidelity low-rank task vector representations. to this end, we first empirically validate that task vectors exhibit a small number of dominant singular values, with the remaining singular values being significantly smaller in magnitude, as shown in figure 1 . additionally, the dimension of the intersection of the images of two matrices is bounded by the minimum of their ranks. therefore, we propose lore-merging, a unified framework for model merging based on low-rank estimation of task vectors, which eliminates the need for access to the base model. specifically, given a set of ft models, we formulate the merging problem as an optimization problem whose goal is to simultaneously identify an approximate base model integrated with a set of low-rank task vectors. together, these vectors collectively approximate the behavior of the ft models. by leveraging low-rank estimations, task vectors become inherently less susceptible to interference, effectively addressing a fundamental challenge in model merging. we conduct extensive experiments on optimization modeling problems and math word problems to confirm the effectiveness of our method. ## related work merging fine-tuned models has been shown to offer several benefits, such as improving performance on a single target task @xcite @xcite , enhancing outof-domain generalization @xcite @xcite , creating multi-task models from different tasks @xcite @xcite , supporting continual learning @xcite , and addressing other challenges @xcite . besides the full parameters merging, @xcite al. (2024); jang et al. (2024) discussed the model merging for lora parameters. among these methods, task-vector-based merging approaches play an important role. task arithmetic @xcite first introduced the concept of task vectors and shows that simple arithmetic operations can be performed to obtain the merged models. building on this idea, methods like dare @xcite and ties-merging @xcite adopt pruning-then-scaling techniques to merge task vectors, based on the assumption that not all parameters equally contribute to the final performance. however, these methods based on sparsity estimation consistently suffer from the interference among task vectors and require access to the base model, thus limiting their overall effectiveness. ## experiments baselines & settings we compare lore-merging with following popular merging methods. average merging @xcite : this method computes the element-wise mean of all the individual models. dare @xcite : this approach randomly drops task-specific vectors and rescales the remaining vectors back to the base model. we set the hyperparameter for the random probability to 0.5. ties-merging @xcite : in this method, task-specific vectors are randomly dropped, and only the parameters aligned with the final agreed-upon sign are merged. for ties-merging, we set the top-k value to 20%, and the hyperparameter λ is fixed at 1. for lore-merging, the rank r is determined dynamically. for a given task vector δ ∈ r m×n , we set the rank @xmath0.2 × min{m, n} to get a low-rank estimation. for the lore-merging method, there's no predefined base model, so we construct θ 0 from the given list of models. for other merging methods applied to @xcite wizardmath & llama-2-13b-code merging, we consistently designate the first model as the base model and the second as the target model. for other checkpoint merging scenarios, where no inherent reason dictates a specific base model, we randomly select one checkpoint to serve as the base model. evaluation we first evaluate lore-merging on diverse benchmarks, including gsm8k @xcite , math @xcite (math word problem), mmlu @xcite , glue @xcite (commonsense reasoning) and mbpp @xcite code )). additionally, we also evaluate on the advanced task, i.e. mathematical optimization modeling problems @xcite @xcite . this task aims to generate solvable mathematical models given an optimization problem in natural language. as the lack of public models on this task, we first fine-tuned qwen-2.5-coder-7b-instruct model @xcite with the dataset provided by @xcite and merge checkpoints in the training process. the evaluations are conducted on mamo dataset @xcite which in-cludes two subsets easylp and complexlp, and nl4opt dataset @xcite . ## conclusion in this paper, we propose a unified framework for merging models based on low-rank estimation, named lore-merging. we achieve it by formulating the merging problem as an optimization problem. extensive experiments demonstrate the efficacy and efficiency of our proposed methods.