Merging Models with Fisher-Weighted Averaging
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Averaging the parameters of models that have the same architecture and initialization can provide a means of combining their respective capabilities.
In this paper,we take the perspective that this “merging” operation can be seen as choosing parameters that approximately maximize the joint likelihood of the posteriors of themodels’ parameters.
Computing a simple average of the models’ parameters therefore corresponds to making an isotropic Gaussian approximation to their posteriors.
We develop an alternative merging procedure based on the Laplace approximationwhere we approximate each model’s posterior as a Gaussian distribution whoseprecision matrix corresponds to its Fisher information.
We ﬁrst show that our“Fisher merging” technique provides a performance boost in settings where simpleparameter averaging is currently used – speciﬁcally, robust ﬁne-tuning and modelensembling.
Then, we compare merging to standard gradient-based transfer learning and demonstrate that merging enables a fundamentally different method fortransferring capabilities across models.
Speciﬁcally, we show that Fisher mergingis competitive with gradient-based transfer learning approaches (while being signiﬁcantly cheaper) in intermediate-task training and domain-adaptive pre-training.
We also show that our merging procedure makes it possible to combine models inpreviously unexplored ways.
We release our code to facilitate future research intomethods for merging models.
Introduction
How should we transfer knowledge and capabilities across trained models?
One popular approachis transfer learning [44], which ﬁne-tunes a pre-trained model on a target task through additionalgradient-based training.
The preparatory step of pre-training the model on a data-rich task ideallyinstills useful “knowledge” into the network’s parameters which allows the model to learn morerapidly and effectively when ﬁne-tuned on a downstream task of interest.
Transfer learning hastherefore become a particularly important and omnipresent tool across many ﬁelds, including naturallanguage processing [57, 13, 9, 52, 53, 46] and computer vision [43, 24, 68].
Recently, it has beenshown that training on an “intermediate” task between pre-training and ﬁne-tuning can further boostperformance through additional transfer of capabilities from the intermediate task [47, 60, 51, 48].
Alternatively, continued self-supervised training on unlabeled domain-specialized data can serve as aform of domain adaptation [19].
All of the aforementioned transfer learning methods transfer knowledge by using a trained networkto initialize another network followed by iterative gradient descent.
While demonstrably powerful,several drawbacks arise from this: First, improvements to ancestor models cannot be passed down todescendants; instead, we must restart the whole process from the improved ancestor model, throwingaway our previous work.
For example, if we ﬁne-tune a pre-trained model on a downstream task,but then the pre-trained model is improved through additional training, we must re-ﬁne-tune thenew model on our downstream task if we want to confer beneﬁts from this additional pre-training
[Figure]
Furthermore, if we gain access to a checkpoint that has been ﬁne-tuned on a useful intermediate task,we must again throw away our previous work and ﬁne-tune from the intermediate task checkpoint.
Existing methods for transfer learning also have the disadvantage of only being able to transferinformation from a single model.
While it may be possible to train on multiple intermediate taskssequentially, one quickly either runs into a combinatorial explosion of saved checkpoints or facesthe issue of “catastrophic forgetting” in continual learning [28].
In addition to slowing downexperimentation by preventing reuse of work, these drawbacks impose limitations on the types oftransfer that can occur.
A less common way of transferring capabilities across models is to simply average their parameters.
This procedure, which we call “merging”, is generally only feasible when the models being averagedshare a common architecture and initialization.
Merging is the core component of the FedAvgalgorithm used in Federated Learning [39], where updates to a shared model computed by individualworkers that are training on different datasets are combined by simpling averaging the updates.
Recently, Wortsman et al.
[66] demonstrated that merging can be used to improve robustness todomain shift in ﬁne-tuned models by averaging the parameters of the original pre-trained modelwith the ﬁne-tuned parameters.
Merging is also a common way of performing ensembling [49, 67],where the parameters of individual models trained on the same dataset are averaged to create a singleperformant model.
In this work, we view model merging as approximately maximizing the joint likelihood of the models’posterior distribution over parameters.
Since gradient-based maximum likelihood training onlyprovides a point estimate of the posterior, some approximation of the posterior distribution is required.
When an isotropic Gaussian distribution is used to approximate the posterior (with identity precisionmatrix and mean set to the model’s parameter values), we show that maximizing the joint likelihoodacross models is equivalent to simply averaging their parameters.
We therefore refer to mergingmodels by averaging parameters as isotropic merging .
The view of merging as maximizing the jointlikelihood of model posteriors suggests that using a better estimate of the posterior distribution mayyield improved merging results.
This leads us to introduce Fisher merging , which leverages the
Laplace approximation by using the diagonal of each model’s Fisher information as the precisionmatrix for that model’s posterior.
Empirically, we demonstrate that merging models with Fisher merging outperforms isotropic mergingin a variety of settings.
We ﬁrst focus on the existing applications of model ensembling [49, 67] andimproving ﬁne-tuned model robustness [66].
Then, we demonstrate for the ﬁrst time that merging isa viable alternative to traditional gradient-based transfer learning.
Speciﬁcally, we compare mergingto intermediate-task transfer learning [47, 51] and domain-adaptive pre-training [19], ﬁnding thatmerging can achieve comparable performance at signiﬁcantly lower cost.
Additionally, we showthat merging can provide an additional boost to models created via traditional intermediate-tasktraining.
This provides a concrete example of transfer that is fast and easy with merging but onerousor impossible to do with existing methods.
Diagrams of the merging patterns we consider in thiswork are shown in ﬁg. 1.
The rest of our paper is structured as follows:
In section 2, we provide necessary background anddetail our Fisher merging procedure.
Section 3 provides experimental results on model ensembling,robust ﬁne-tuning, intermediate-task training, and domain adaptation.
We explore related works insection 4 and provide conclusions and thoughts on future work in section 5.
Weighted Parameter Averaging for Model Merging
Our focus is on procedures for model merging , i.e. averaging the parameters of models that sharean architecture and initialization.
In this section, we ﬁrst frame the common practice of averagingtogether model parameters as approximately maximizing the joint likelihood of model posteriors.
Speciﬁcally, we show that parameter averaging corresponds to using an isotropic Gaussian as theapproximate posterior for each model.
We then introduce Fisher merging , which uses the model’sdiagonal Fisher information matrix as the precision matrix of the Gaussian approximate posterior.
Fisher merging can be implemented by setting each merged parameter value to a weighted average ofthe corresponding parameter values from the original models, with the weighting for each parameterdetermined by its Fisher information.
In addition, we add model-level weightings as additionalhyperparameters to set the relative importance of each model.
2.1 Isotropic merging
Consider the problem setting where we have M trained neural networks with parameters ✓ 1 , . .
. , ✓ Mand our goal is to create a single neural network with parameters ✓ that, loosely speaking, inheritsthe capabilities of the M trained neural networks.
Assume that all of these neural networks share acommon architecture and had the same set of initial parameter values before being trained.
Mergingattacks this problem by ﬁnding the parameters ✓ that maximize the joint likelihood of the posteriordistributions of the M models.
Unfortunately, typical neural network training procedures do notprovide access to a posterior distribution, which necessitates approximation.
If the posterior ofeach model is approximated via an isotropic Gaussian with mean set to the model’s parameters,the optimization problem can be written as ✓ ⇤ =
argmax ✓
P i log p ( ✓ | ✓ i , I ) where p ( ✓ | ✓
i , I ) is theprobability distribution of the aforementioned approximate isotropic Gaussian posterior distributionused for model
i and I is the identity matrix.
This optimization problem has a closed-form solutiongiven by ✓ ⇤ = 1 M
P i ✓ i , i.e. an average of the model parameters.
Such an averaging procedure hasbeen used in past work aiming to combine model capabilities, e.g. in federated learning [39], modelensembling [49, 67], and robust ﬁne-tuning [66].
2.2 Per-model weights
In this work, we additionally introduce model-speciﬁc scalar hyperparameters
i , i 2 { 1 , . . .
, M }into the model merging framework described above.
Speciﬁcally, we change the optimization problemto ✓ ⇤ = argmax ✓
P i  i log p ( ✓ | ✓ i , I ) where  i  0 ,
P i  i = 1 .
In the case of isotropic merging, thischanges the solution to ✓ ⇤ =
P i  i ✓ i , These hyperparameters provide control over the importanceassigned to each of the models that are being merged.
For example, when using merging to performensembling we might expect each model to be equally important and therefore set  i = 1 /M for all i .
On the other hand, when mimicking the setup of intermediate-task training where the capabilities ofa “donor” model are used to improve performance of a recipient model, we might weigh the recipientmodel more highly.
Wortsman et al.
[66] introduce a similar hyperparameter ↵ when averaging theparameters of two models and report results for varying values of ↵ .
2.3 Laplace Approximation
Framing merging as approximate maximization of the joint posterior likelihood reveals that simpleparameter averaging is implicitly using an isotropic Gaussian posterior approximation.
Such anapproximation may be overly simplistic and lead to degraded performance.
To explore improvedmerging procedures, we consider improved methods for creating an approximate posterior from apoint estimate.
Speciﬁcally, we use the Laplace approximation to the posterior, which corresponds toa second-order Taylor expansion of the log density around a mode [36, 10].
This leads to a Gaussianapproximation N ( ✓ , H  1 ) of the posterior, where H is the Hessian matrix and ✓ are the model’strained parameter values.
More precisely, we assume that the parameter values ✓ of a trained neural network are a local maximum of the posterior.
It can then be shown that the precision matrix of the
Laplace approximation is given by the Fisher information matrix of the network at ✓ .
The Fisher information matrix F ✓ [16, 3] of a neural network p ✓ ( y | x ) trained to predict an output yfrom input data x is a | ✓ | ⇥ | ✓ | positive semideﬁnite matrix given by the formula
[Equation]
It can be shown that the Fisher information matrix coincides with the Hessian H at modes of thedistribution [45], explaining its use in the Laplace approximation.
The Fisher information matrix F ✓can also be used to relate changes in the model parameters to changes in the model output by notingthat E
x [ D KL ( p ✓ ( y | x ) || p ✓ +  ( y | x ))] ⇡ 12  T F ✓  as  ! 0 , where D KL denotes the KL-divergence 135 [45].
As the full Fisher matrix takes O ( | ✓ | 2 ) memory to store, it quickly becomes impractical for all butthe smallest models.
We are thus forced to use an approximation to the full Fisher in practice.
In thispaper, we follow the common practice of using the diagonal of the Fisher matrix [28].
While othermethods (e.g. [1]) exist for estimating the Fisher, we leave their exploration for future work.
In ourexperiments, we estimated the diagonal of the Fisher matrix via
[Equation]
where x 1 , . .
, x N are drawn i.i.d .
from the dataset that was used to train the model.
The expectationover y can be estimated via sampling from p ✓ ( y | x i ) or computed exactly when the number ofclasses is small.
We note that computing the Fisher requires N per-example gradients, which can bestraightforwardly computed for neural networks using backpropagation.
This makes computing thediagonal
Fisher have roughly the same computational cost as training on N examples.
2.4 Fisher Merging
Having noted that the Laplace approximation provides a tractable way to obtain a better approximationto the posterior, we now use it to create an improved merging procedure that we call Fisher merging .
Letting F 1 , . .
. , F M correspond to the diagonal approximate Fisher matrices, we construct p ( ✓ | ✓ i , F i )as a Gaussian-distributed posterior over the parameters of the merged model with mean ✓ i andprecision F i .
To obtain the merged model, we ﬁnd a single set of parameters that is given a highprobability under all posteriors.
Formally, we have
[Equation]
which has the closed-form solution
[Equation]
where j = 1 , . . .
, | ✓ | .
Intuitively, we can think of Fisher merging as computing a weighted averageof the parameter values in each model where the weighting is done according to each parameter’s Fisher information.
Since the Fisher information is a local property of a single parameter value,
Fisher merging might be less performant when applied to models whose parameters are far apart inparameter space.
We therefore limit our focus to models that were trained from the same initialization.
Numerical Issues.
Note that (4) can run into numerical issues when the Fisher is close to zero acrossall models for a given parameter.
In practice, we choose a privileged “target model” in all of ourexperiments and “default” to the parameter’s value in the target model in these cases.
An alternativewould be to take an average weighted only by the merging coefﬁcients (i.e., pretend the Fisher is thesame across all models).
In practice, the choice of a “default” value for these parameters had littleimpact on performance (likely because a small Fisher value implies that changing the parameter has aminute effect on the model’s outputs and is therefore relatively unimportant to the model’s behavior).
Unmergeable Parameters.
In many cases, we have some parameters from each model that donot appear in all of the models we are merging.
For example, this includes having task-speciﬁc classiﬁcation heads on top of a common body architecture.
We handle this by only applying themerging procedure (3) to the shared body parameters and keeping the task-speciﬁc heads unchanged.
Although this may lead to a distribution shift in the classiﬁcation head inputs, we found it to workwell in practice for the datasets and tasks we consider.
Experiments
Our ﬁrst experimental goal is to validate that our use of an improved estimate of the posterior yieldsimproved merging performance.
To test this hypothesis, we apply Fisher merging to two settingswhere isotropic merging has already proven successful:
Model ensembling [49, 67] and robustﬁne-tuning [66].
Then, we demonstrate that Fisher merging provides a cheap and effective alternativeto traditional transfer learning pipelines by validating its performance in intermediate-task transferlearning [47, 51] and domain-adaptive pre-training [19].
Finally, we demonstrate that merging opensup new paths of transferring capabilities across models by demonstrating a boost in performancewhen merging an intermediate task-trained model with different donor models.
3.1 Ensembling
An existing application of isotropic merging is for ensembling , i.e. combining models trained on thesame dataset to obtain better predictions.
Ensembling is most commonly performed by averaging thepredictions of the individual models.
This form of ensembling requires computing the output of all
M models in the ensemble, thereby increasing the computational cost by a factor of M comparedto computing the output for a single model.
A cheaper alternative is to average the parameters ofthe models themselves.
This approach is diagrammed in ﬁg. 1, left.
Such an approach is used in theclassical method of Polyak averaging [49], where parameter values from the ﬁnal M iterations oftraining are averaged.
More recently, Wortsman et al.
[67] introduced the “Model Soup” approachwhere ﬁne-tuned models with different hyperparameter settings are averaged to improve performance.
To the best of our knowledge, all parameter-averaging ensemble methods have used isotropic merging,i.e. an unweighted average.
To test whether Fisher merging provides a boost over isotropic merging when averaging parametersfor ensembling, we consider ensembling ﬁne-tuned checkpoints derived from the same pre-trainedmodel.
Speciﬁcally, we consider the BERT-Base model [13] ﬁne-tuned on the RTE [8], MRPC [14],and SST-2 [59] datasets.
For each dataset, we use ﬁve ﬁne-tuned checkpoints downloaded from the
Hugging Face model hub.
These checkpoints were ﬁne-tuned with a variety of hyperparametersettings that were not chosen by us, so our experimental setting most closely matches the “Model
Soup” approach [67].
A list of the checkpoints used is available in appendix A. Since we do notanticipate that any member of the ensemble should be given a larger weight, we set  i = 1 / 5 for allmodels.
Our results are shown in ﬁg. 2.
We report validation set scores for Fisher merging, isotropicmerging, and prediction ensembling (speciﬁcally, averaging the output probabilties of all models).
Fisher merging signiﬁcantly outperforms isotropic merging in all cases and attains comparableperformance to prediction ensembling.
Notably, performing inference after merging is M ⇥ cheaperthan prediction ensembling, suggesting that merging can provide a cheaper alternative to standardensembling procedures.
3.2 Robust Fine-Tuning
Recently, Wortsman et al.
[66] found that while ﬁne-tuning a pre-trained vision model tends toimprove performance on the downstream task, it also tends to decreases accuracy on the original pretraining task.
They therefore propose a “robust ﬁne-tuning” procedure called WiSE-FT that computesa weighted average of the original pre-trained parameters and the ﬁne-tuned parameters.
Differentweighting values produce different trade-offs between pre-training and ﬁne-tuning task performance.
In some cases, robust ﬁne-tuning can even improve performance on the original pre-training taskwithout sacriﬁcing performance on the downstream ﬁne-tuning task relative to traditional ﬁne-tuning.
This procedure implicitly uses isotropic merging and therefore provides another natural testbed /models for determining whether Fisher merging provides a boost in performance.
A schematic of robustﬁne-tuning is shown in ﬁg. 1, center top.
We use the codebase and experimental setup of Wortsman et al.
[66] exactly, simply replacingisotropic merging with Fisher merging.
For full details of this setup, we refer to Wortsman et al.
As a short summary, we apply WiSE-FT to the ImageNet [11, 58] pre-trained ViT-B/16 model [15]on ﬁve out-of-domain (OOD) datasets: ImageNet-A [21], ImageNet-R [20], ImageNet Sketch [62],
ImageNet V2 [56], and ObjectNet [4].
Following Wortsman et al.
[66], we measure IID (ImageNet)and OOD performance when averaging together the original pre-trained model parameters andparameters from models ﬁne-tuned on each of the OOD datasets, varying  1 (the averaging weightfor the pre-trained model, called ↵ by Wortsman et al.
[66]) from 0 to 1 in 0 .
1 -step increments (with 2 = 1   1 correspondingly decreasing from 1 to 0 ).
To determine whether Fisher merging confersa boost in performance, we compare parameter averaging using either isotropic or Fisher merging.
We plot the IID (ImageNet) accuracy against the average accuracy on the ﬁve OOD datasets forvarying values of  1 in ﬁg. 3, with plots for individual OOD datasets in ﬁg. 7 (appendix).
Fishermerging produces a signiﬁcantly better trade-off between IID and OOD accuracy.
In particular, Fishermerging seems to general improve IID accuracy compared to isotropic merging.
For example, for thevalue of  1 producing the best average OOD accuracy, Fisher merging produces about 1% higher IIDaccuracy than isotropic merging.
3.3 Intermediate-task training
Having established that Fisher merging produces better results than isotropic merging in settingswhere merging has been attempted before, we now explore the use of merging as an alternative toa gradient-based transfer learning procedure.
Speciﬁcally, we explore intermediate-task training[47, 51], where a model is ﬁne-tuned on an intermediate “donor” task before being trained on thetarget task of interest.
To the best of our knowledge, no prior work has considered parameter averagingas a way of performing intermediate-task transfer learning.
For the most part, intermediate-tasktraining has mainly been considered in the NLP domain; as such, we limit our experiments to the
BERT [13] and RoBERTa [33] pre-trained language models.
To enable comparison to past work, wemostly explored merging pairs of models but we are interested in exploring merging more than twomodels in future work.
As in section 3.1, we made use of ﬁne-tuned BERT and RoBERTa checkpointsfrom the Hugging Face repository [65].
Following previous work [47, 51], we ﬁrst ran experiments using BERT-base on the GLUE benchmark[61].
The GLUE benchmark consists of the sentence acceptability task CoLA [64], the sentimentdetection task SST-2 [59], the paraphrase detection tasks MRPC and QQP [14, 23], the sentencesimilarity task STS-B [7], and the natural language inference (NLI) tasks MNLI, QNLI, RTE, and
WNLI [6, 54, 8, 31].
All of the GLUE tasks are classiﬁcation tasks except for STS-B, which is a regression task with a score ranging from 0 to 5.
To simplify computation of the Fisher, we turn
STS-B into a classiﬁcation task by partitioning the continuous label into 25 equally-sized buckets[53].
Following common practice, we do not run experiments on WNLI due to the tricks required toget a good score [12, 29].
See Wang et al.
[61] for more details on these tasks and their associatedmetrics.
We detail how we obtained ﬁne-tuned checkpoints on these tasks in appendix C. Wecomputed a diagonal Fisher approximation for each checkpoint using up to 4096 examples from thecorresponding train set.
Since it is not clear a priori what weighting coefﬁcients  i to use in thissetting, we chose  i by a grid search with 50 points, using the score on the ﬁrst 2048 validationexamples as the selection metric.
We compare Fisher merging to isotropic merging as well as astandard gradient-based intermediate-task ﬁne-tuning baseline [47].
A diagram of intermediate-taskmerging is shown in ﬁg. 1, center bottom.
In initial experiments (reported in tables A1 to A3), we performed intermediate-task training forpossible pair of datasets from the GLUE benchmark.
Congruent with past work [47, 51, 60], wefound that intermediate-task training provided the most notable performance boost when the RTEdataset was the target.
We therefore focus on RTE results in the main text.
Figure 4 shows the resultsof intermediate-task training of BERT-base with RTE as the target task and the other GLUE datasetsas donor tasks, using Fisher merging, isotropic merging, or standard gradient-based training.
Notably,performing gradient-based intermediate-task training hurts on some datasets, whereas merging alwayshelps.
Fisher merging gets comparable or better performance than isotropic merging with the largestgap observed when using MNLI as the intermediate task.
On the other hand, merging performs worsethan standard gradient-based training when using MNLI as the donor task.
Exploring new paths for transfer Given this performance gap, we were interested to see whethermerging could provide an additional boost on top of gradient-based intermediate-task training.
Wetherefore performed Fisher merging on a BERT-base model that was ﬁrst ﬁne-tuned on MNLI andthen ﬁne-tuned on RTE.
A diagram of this setup is shown in ﬁg. 1, right.
This procedure does nothave a direct analog in traditional gradient-based, and as we will show later, performing multi-stagegradient-based intermediate-task training generally harms results.
We consider Fisher merging the intermediate-task trained RTE model with all GLUE tasks and showthe results in ﬁg. 5.
Fisher merging provides a boost over gradient-based intermediate-task trainingfor all tasks.
Interestingly, a boost is still conferred when merging with an MNLI-trained model,suggesting that merging provides a complementary path for transferring capabilities across models.
Scaling to RoBERTa-large Seeing that merging can provide a boost on top of intermediate-tasktraining, we explored whether this boost could still be obtained for a stronger model than BERTbase.
We therefore applied the same procedure to a RoBERTa-large RTE model that had beenﬁne-tuned from an MNLI intermediate checkpoint.
Our donor models were the original RoBERTa- large checkpoint (i.e., not ﬁne-tuned on MNLI) ﬁne-tuned on MRPC, RTE, STS-B, and SST-2.
Weadditionally ran a sequential gradient-based ﬁne-tuning baseline where we started with the MNLIcheckpoint, ﬁne-tuned on the donor task, and then ﬁne-tuned on the target task.
The results are shown in ﬁg. 6.
We ﬁnd merging provides a boost in performance even on the moreperformant RoBERTa model.
The largest boost of 2.2 points came from Fisher merging with another
RTE checkpoint, which is reminiscent of using merging for ensembling.
Notably, including anadditional intermediate task in gradient-based training signiﬁcantly harmed performance comparedto performing intermediate-task training on MNLI alone.
We hypothesize this is related to thephenomena of catastrophic forgetting [17], where the model’s capabilities on MNLI are forgotten asit is trained on the next intermediate task.
Nevertheless, this illustrates model merging’s ability tosidestep the issue of catastrophic forgetting and enable exploration of novel transfer strategies.
Costs We had previously noted that our merging procedure could potentially be substantially moreefﬁcient than standard gradient-based ﬁne-tuning.
To measure this claim concretely, we computed the
FLOPs required for ﬁne-tuning and merging an RTE checkpoint based on the heuristics described in
Kaplan et al.
Fine-tuning BERT-base on RTE for 10 epochs would require about 5.5e14 FLOPs.
Our merging procedures require computing the merged checkpoint (eq. (4)) and then evaluating iton the validation set with Fisher merging also requiring the estimation of the Fisher matrix (eq.
(2))beforehand.
These steps require about 4.0e8, 2.0e12, and 9.1e13 FLOPs respectively, resulting ina roughly 6 ⇥ lower total cost compared to ﬁne-tuning for Fisher merging and 275 ⇥ lower cost forisotropic merging.
We note that the Fisher matrix only needs to be computed once and can be reusedfor subsequent merges, which amortizes the most expensive step in Fisher merging.
To explore methods for further reducing costs, we experimented with using fewer examples to estimatethe Fisher.
Speciﬁcally, we experimented with intermediate-task Fisher merging of BERT-base with
MNLI as the donor task and RTE as the target task.
The results are shown in table A4.
While usingthe full training set to estimate the Fisher produced the best performance (73.4%), using only 256examples to estimate the Fisher only produced a mild degradation in accuracy (72.7%) and stilloutperformed the isotropic merging baseline.
This suggests that computing the Fisher over fewerexamples could further reduce computational costs without sacriﬁcing a great deal of accuracy.
3.4 Domain Adaptation
We now turn our attention to the “domain-adaptive pre-training” (DAPT) approach for domainadaptation advocated by Gururangan et al.
[19], which is methodologically similar to intermediatetask training.
DAPT consists of additional pre-training of an original general-purpose pre-trainedcheckpoint on domain-speciﬁc unlabeled data.
We explore the beneﬁts of merging in an experimentalsetup similar to Gururangan et al.
We focus on the biomedical (B IO M ED ) and computer science(CS) domains because they correspond to the classiﬁcation tasks that saw the largest gains fromdomain-adaptive pre-training in [19].
Namely, we experimented with the C HEM P ROT [30] relationclassiﬁcation task on the B IO M ED domain.
On the CS domain, we used the citation intent task of
ACL-ARC [25] and the relation classiﬁcation task of S CI ERC [35].
Following Gururangan et al.[19], we report macro-
F 1 for ACL-ARC and S CI ERC, and we report micro- F 1 for C HEM P ROT .
Weused RoBERTa-base [33] as our baseline model.
Appendix D includes full details of the pre-training,ﬁne-tuning, and merging procedures used.
We present our results in table 1.
Merging provided the largest boost on A CL -A RC , and outperformedtraditional ﬁne-tuning in this setting.
We only observed a minor improvement in performance on
C HEM P ROT and S CI ERC.
We note that our boosts from gradient-based ﬁne-tuning were smallerthan reported in [19], which was likely because we were only able to train on public data and weapplied domain-adaptive pre-training for fewer steps.
However, our results are consistent in the sensethat A CL -A RC received the largest boost and C HEM P ROT received the smallest boost.
Related Work
Like our work, elastic weight consolidation (EWC) [28] uses the Laplace approximation to theposterior over model parameters to create a regularizer to prevent catastrophic forgetting in thecontext of continual learning.
While their framework supports the use of posteriors from multiplemodels as well, they restrict such models to be previous checkpoints of a continually trained model.
[Table]
EWC keeps the model from losing previously acquired knowledge while merging provides a meansof directly adding new knowledge to a model.
Some other existing procedures such as distillation [22] and ensembling [42] can also be thought ofas combining or transferring knowledge between neural networks.
However, those methods representknowledge solely through the output of models.
The knowledge contained within the parameters of anetwork will necessarily be greater than the knowledge contained in its output [2].
Hence, methodsthat directly combine model parameters such as merging have the potential to be more powerfulthan those methods.
Furthermore, our merging procedure has an efﬁcient and closed-form solution(eq.
(4)) while distillation requires iterative gradient descent-based training.
Isotropic checkpoint averaging is used by federated learning [39] and Polyak averaging [49].
However,the checkpoints merged by those methods can be thought of coming from the same training run ofsingle model.
We believe we are the ﬁrst to demonstrate cross-task transfer coming from checkpointaveraging and to explore it in the context of transfer learning.
However, adapting ideas from federatedlearning such as [32, 63] could provide a fruitful avenue for future model merging research.
Natural gradient descent refers to an optimization procedure that uses KL-divergence of modelpredictions as a distance measure rather than the Euclidean distance in parameter space employed byregular gradient descent [3].
It does this by performing gradient descent on a Riemannian manifoldwith the Fisher information matrix as its metric [45].
In practice, this amounts to using the Fisher as apreconditioner during gradient descent.
Some work on natural gradient descent may prove relevant formodel merging such as using Kronecker-factorized Fisher matrices as an alternative to the diagonalapproximation employed in this paper [37, 18, 38].
More broadly, in the ﬁeld of information geometrythe Fisher information matrix plays the role of a metric on a Riemannian manifold [40].
This has ledto explorations of model averaging using tools from information geometry, e.g. [5, 41, 50].
Conclusion
In this paper, we introduced Fisher merging , a way to combine the capabilities of different models bycomputing a weighted average of their parameters.
Fisher merging is motivated by a novel perspectiveof parameter averaging as maximizing the joint likelihood of model posteriors.
Through extensiveexperiments, we demonstrated that using the Fisher information as a weight on the contributionof each parameter outperforms using an unweighted average.
Furthermore, we showed that Fishermerging attains comparable and sometimes better performance than traditional gradient-based transferlearning methods at signiﬁcantly lower costs.
Our experiments also demonstrated various mergingstrategies that would be onerous with traditional gradient-based training, which opens up new avenuesfor transferring capabilities across models.
In future work, we plan to investigate different methodsfor approximating the Fisher information and model posteriors as well as more esoteric combinationsof models.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] Settings where merging under-
performs traditional gradient-based training are covered in section 3 and also discuss caveats of our method in section 2.4.
(c) Did you discuss any potential negative societal impacts of your work?
[No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] We state thatmerging assumes a shared architecture and initialization at various points in the paper
and explain why in section 2.4.
(b) Did you include complete proofs of all theoretical results?
[N/A]
No theoretical
advances were sufﬁciently complex to warrant proof.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] Code is
provided in the supplementary.
All datasets are public and are downloadable with our
code.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See appendix C and appendix D
(c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
[Yes] See tables A1 to A3.
(d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[No]
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No] All datasets are widely-used (hundreds
or thousands of citations) public datasets.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]We include code in the supplemental.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[Table] A
[Table]
If you used crowdsourcing or conducted research with human subjects...
Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
you
participant
to Institutional Review Board approvals,
applicable
you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation
N