Dynamic Fisher-weighted Model Merging via B ayesian Optimization (2025)

Abstract: The fine-tuning of pre-trained language models has resulted in the widespread availability of task-specific models. Model merging offers an efficient way to create multi-task models by combining these fine-tuned models at the parameter level, without the need for training data or joint training on multiple datasets. Existing merging approaches typically involve scaling the parameters model-wise or integrating parameter importance parameter-wise. Both approaches exhibit their own weaknesses, leading to a notable performance gap compared to multi-task fine-tuning. In this paper, we unify these seemingly distinct strategies into a more general merging framework, and introduce Dynamic Fisher-weighted Merging (DF-Merge). Specifically, candidate models are associated with a set of coefficients that linearly scale their fine-tuned parameters. Bayesian optimization is applied to dynamically adjust these coefficients, aiming to maximize overall performance on validation sets. Each iteration of this process integrates parameter importance based on the Fisher information conditioned by the coefficients. Experimental results show that DF-Merge outperforms strong baselines across models of different sizes and a variety of tasks. Our analysis shows that the effectiveness of DF-Merge arises from the unified view of merging and that near-optimal performance is achievable in a few iterations, even with minimal validation data.

## introduction modern transformer-based pre-trained language models (plms) @xcite @xcite have driven a paradigm shift towards fine-tuning plms for specific tasks, achieving state-of-the-art performance across various applications. the general-purpose representations learned through pretraining have significantly enhanced numerous downstream tasks, leading to the widespread development of fine-tuned expert models @xcite . for example, over a million models have been uploaded to the hugging face repository @xcite , with many publicly available for research study foot_0 . most off-the-shelf models are fine-tuned independently for individual tasks, which limits their performance outside of their specialized domains. ideally, models should be capable of handling multiple tasks relevant to a particular use case. although multi-task learning (mtl) @xcite offers a straightforward solution, it must require simultaneous access to the labeled datasets of all tasks and training over those datasets. this challenge is pronounced given the increasing difficulty of fine-tuning plms with their ever-growing sizes. more importantly, since the original training data for each model is often proprietary, it is time-consuming or even infeasible for users to label a large amount of data for mtl. model merging offers a cost-effective alternative for building multi-task models by combining off-the-shelf models in the parameter space without additional training. for example, this can be done by simply weight-averaging the task vectors (i.e. fine-tuned part of the parameters from the pretrained model) @xcite . the success of model merging is supported by recent findings that the local minima, optimized from pre-trained parameters, are linearly connected in a flat basin of the loss landscape with no barriers in between @xcite . as a result, linearly interpolating between fine-tuned models potentially produces a well-behaved model with multi-task capabilities. despite the advantages of model merging, current methods still lag behind the performance of multi-task fine-tuned models. this shortfall can be attributed to the fact that existing approaches improve only specific aspects of model merging. these model merging methods can be divided into two groups: (1) scaling the task vectors model-wise @xcite @xcite , and (2) accounting for parameter importance parameter-wise (matena and @xcite @xcite . we present a general merging framework that these two seemingly distinct approaches can be unified into. building on this framework, we introduce dynamic fisherweighted merging (df-merge) which leverages the strengths of both strategies, as illustrated in figure 1 . in essence, df-merge uses bayesian optimization to adjust the scaling coefficients in order to maximize the overall performance, with each iteration targeting a low-loss basin informed by (approximated) fisher information. experimental results show that df-merge significantly outperforms competitive baselines across plms of different sizes on a variety of tasks. ablation study confirms that the components of df-merge collectively contributes to the performance, validating the advantage of the general merging framework. additionally, our analysis demonstrates that df-merge can achieve near-optimal performance within just a few iterations using minimal validation data. our contributions are summarized as follows: task arithmetic (ta). @xcite coined a concept of task vector which represents the direction in the parameter space that enhances a pre-trained model's performance on the task. in particular, the task vector τ i for the i-th task is specified by the fine-tuned part of parameters from the pre-trained model, expressed as τ @xmath0. task vectors can be combined by arithmetic operations to steer the pre-trained model's behavior on various tasks. this concept has been extended to model-wise model merging @xcite @xcite , in which multiple task vectors are added to the pre-trained parameters fisher information. understanding the landscape of the loss function allows for better alignment of different models' parameters. the local curvature of a loss function ℓ(θ) at the point θ is captured by its second-order derivatives ∇ 2 ℓ(θ), denoted as the hessian matrix h θ ∈ r d×d . then the expectation of hessian over the data distribution p θ (x, y) describes how sensitive the loss function is to the parameters in the data distribution modeled by θ, where highly sensitive parameters potentially imply greater importance. assuming the model is fine-tuned using negative log-likelihood loss ℓ(θ) =log p(y|x, θ), the expectation of h θ can be efficiently computed by the fisher information (fi): as estimating the expectation over the input distribution x ∼ q(x) is intractable, f θ is approximated with the empirical fisher information fθ note that the expectation over y is not calculated over the true labels, but rather measured on the predictive distribution y ∼ p θ (y|x) parameterized by θ. in practice, y ∼ p θ (y|x) can either be modeled exactly or through sampling depending on the size of label space @xcite . fisher merging from geometric perspective. @xcite studied a geometric analysis of fisher merging @xcite , by representing fisher merging (without some approximations) as building upon this insight, we consider the following geometric objective g(θ) and show that fisher merging is a natural result of minimizing it: as g(θ) is convex, setting its gradient to zero leads to the closed-form solution: in practice, f θ i is replaced by its diagonal approximation to reduce computational complexity @xcite , which can be seen as assuming independence between the parameters (i.e., q @xmath1. ## dynamic fisher-weighted merging performance drop after merging models fine-tuned on different tasks occurs due to parameter interference, in which each task vector may represent a loss-increasing direction for the other tasks. our pilot study (figure 2 ) shows that linearly interpolating between a pre-trained model and a fine-tuned model reveals numerous alternative local minima, motivating us to search for a set of coefficients such that applying fisher merging on the models interpolated by the coefficients minimizes parameter interference. we apply bayesian optimization for an efficient search of optimal coefficients. an overview of our proposed method, df-merge, is illustrated in figure 3 . in the following paragraphs, we present an unified view of model merging, then proceed to the details of df-merge, divided into merge function and coefficient optimization. an unified view of model merging. we show that fisher merging (parameter-wise) and task arithmetic (model-wise) both falls under the restricted cases of a more generic form of model merging. this generalized perspective offers a natural way to link both approaches. in particular, we propose a general function of model merging drawing from this insight, we propose the merge function of df-merge via linking the benefits of the two: coefficient optimization. we employ bayesian optimization to determine the coefficients {λ i } m @xmath2.8 that maximize average accuracy on the held-out validation sets. unlike gradient descent, bayesian optimization is well-suited for optimizing non-differentiable metrics like accuracy, precision or recall, which directly aligns with the goal of model merging. in addition, bayesian optimization finds near-optimal coefficients within a few iterations, making it far more scalable than grid search as the number of models increases. we utilize gaussian process to maximize the black box function f b (λ) (λ := [λ 1 , ..., λ m ] ∈ r m ) that returns a scalar metric (i.e., average accuracy) given the merging coefficients λ. specifically, the gaussian process prior is placed over the initial random observations on t points @xcite : subsequently, the next point λ t+1 to sample is determined by the acquisition functions, and we consider expected improvement (ei) @xcite and upper confidence bound (ucb) @xcite in our experiments. ei chooses λ t+1 such that it maximizes the expected value of improvement than the current best value f * b (t) over its posterior distribution: ## conclusion in this work, we propose an unified merging framework and introduce dynamic fisher-weighted merging (df-merge). this approach assigns scaling coefficients to fine-tuned model parameters and dynamically adjusts them using bayesian optimization, with the goal of maximizing validation performance. through this process, df-merge tries to efficiently identify low-loss basins using fisher information. experimental results demonstrate that df-merge consistently outperforms strong baselines across models of different sizes on diverse tasks. the method proves effective in achieving near-optimal performance in just a few iterations, even with minimal validation data, highlighting its potential as a powerful tool for multitask model merging.