Original: D ATALESS K NOWLEDGE
Revised: D ATALESS K NOWLEDGE

Original: F USION
Revised: F USION

Original: BY M ERGING W EIGHTS OF L ANGUAGE M ODELS
Revised: BY M ERGING W EIGHTS OF L ANGUAGE M ODELS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Fine-tuning pre-trained language models has become the prevalent paradigm for building downstream NLP models.
Revised: Fine-tuning pre-trained language models has become the prevalent paradigm for building downstream NLP models.

Original: Oftentimes fine-tuned models are readily available but their training data is not, due to data privacy or intellectual property concerns.
Revised: Oftentimes fine-tuned models are readily available but their training data is not, due to data privacy or intellectual property concerns.

Original: This creates a barrier to fusing knowledge across individual models to yield a better single model.
Revised: This creates a barrier to fusing knowledge across individual models to yield a better single model.

Original: In this paper, we study the problem of merging individual models built on different training data sets to obtain a single model that performs well both across all data set domains and can generalize on out-ofdomain data.
Revised: In this paper, we study the problem of merging individual models built on different training data sets to obtain a single model that performs well both across all data set domains and can generalize on out-ofdomain data.

Original: We propose a dataless knowledge fusion method that merges models in their parameter space, guided by weights that minimize prediction differences between the merged model and the individual models.
Revised: We propose a dataless knowledge fusion method that merges models in their parameter space, guided by weights that minimize prediction differences between the merged model and the individual models.

Original: Over a battery of evaluation settings, we show that the proposed method significantly outperforms baselines such as Fisher-weighted averaging or model ensembling.
Revised: Over a battery of evaluation settings, we show that the proposed method significantly outperforms baselines such as Fisher-weighted averaging or model ensembling.

Original: Further, we find that our method is a promising alternative to multi-task learning that can preserve or sometimes improve over the individual models without access to the training data.
Revised: Further, we find that our method is a promising alternative to multi-task learning that can preserve or sometimes improve over the individual models without access to the training data.

Original: Finally, model merging is more efficient than training a multi-task model, thus making it applicable to a wider set of scenarios.
Revised: Finally, model merging is more efficient than training a multi-task model, thus making it applicable to a wider set of scenarios.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The dominant paradigm for solving NLP tasks ranging from classification to sequence tagging involves fine-tuning a pretrained language model (PLM) using task-specific labeled data (Devlin et al., 2019;
Revised: The dominant paradigm for solving NLP tasks ranging from classification to sequence tagging involves fine-tuning a pretrained language model (PLM) using task-specific labeled data (Devlin et al., 2019;

Original: He et al., 2021).
Revised: He et al., 2021).

Original: This results in specialized models that are explicitly trained to run inference over a single domain and task.
Revised: This results in specialized models that are explicitly trained to run inference over a single domain and task.

Original: Multi-task learning has shown that leveraging information across domains or tasks can be beneficial if the data sets, data set size and algorithms are well selected (Phang et al., 2018; Pruksachatkun et al., 2020; Poth et al., 2021;
Revised: Multi-task learning has shown that leveraging information across domains or tasks can be beneficial if the data sets, data set size and algorithms are well selected (Phang et al., 2018; Pruksachatkun et al., 2020; Poth et al., 2021;

Original: Weller et al., 2022).
Revised: Weller et al., 2022).

Original: Combining knowledge of multiple data sets in a single model can lead to better overall performance on in-domain data (Poth et al., 2021), can better generalize on out-of-domain data (Wang et al., 2020b) and results in a model that is more practical and parameter efficient than maintaining specialized models.
Revised: Combining knowledge of multiple data sets in a single model can lead to better overall performance on in-domain data (Poth et al., 2021), can better generalize on out-of-domain data (Wang et al., 2020b) and results in a model that is more practical and parameter efficient than maintaining specialized models.

Original: However, the multi-task learning setup suffers from two practical limitations.
Revised: However, the multi-task learning setup suffers from two practical limitations.

Original: First, the training process requires access to the original labeled data, which may not be realistic as annotated data may be private to the agent fine-tuning the model which can happen in order to ensure data or annotation privacy or to guard intellectual property to annotations.
Revised: First, the training process requires access to the original labeled data, which may not be realistic as annotated data may be private to the agent fine-tuning the model which can happen in order to ensure data or annotation privacy or to guard intellectual property to annotations.

Original: Second, because a significant amount of data or task combinations are not beneficial to performance (Poth et al., 2021), building a single model requires training on all data set combinations to identify the optimal one, which can be prohibitive especially if there are many available source data sets or models.
Revised: Second, because a significant amount of data or task combinations are not beneficial to performance (Poth et al., 2021), building a single model requires training on all data set combinations to identify the optimal one, which can be prohibitive especially if there are many available source data sets or models.

Original: Model merging is defined as combining multiple models into a single one in parameter space without access to data (Matena & Raffel, 2021).
Revised: Model merging is defined as combining multiple models into a single one in parameter space without access to data (Matena & Raffel, 2021).

Original: This technique provides an alternative to building a single model while satisfying data privacy constraints.
Revised: This technique provides an alternative to building a single model while satisfying data privacy constraints.

Original: Weight merging algorithms usually also have a closed-form solution, making them very efficient as no retraining is necessary, thus enabling usage even when a large number of data sets or model combinations are available.
Revised: Weight merging algorithms usually also have a closed-form solution, making them very efficient as no retraining is necessary, thus enabling usage even when a large number of data sets or model combinations are available.

Original: Merging can be considered as an alternative to model ensembling (Opitz & Maclin, 1999; Rokach, 2010), where the outputs of individual models are combined to produce the final prediction.
Revised: Merging can be considered as an alternative to model ensembling (Opitz & Maclin, 1999; Rokach, 2010), where the outputs of individual models are combined to produce the final prediction.

Original: Model merging algorithms are a key step in federated learning (McMahan et al., 2017; Lin et al., 2022), where multiple agents train their own model using private data and share only model updates with other models.
Revised: Model merging algorithms are a key step in federated learning (McMahan et al., 2017; Lin et al., 2022), where multiple agents train their own model using private data and share only model updates with other models.

Original: However, in federated learning, model merging happens in multiple rounds of updates, after which the merged model is broadcast to all agents before the next round of training with private data.
Revised: However, in federated learning, model merging happens in multiple rounds of updates, after which the merged model is broadcast to all agents before the next round of training with private data.

Original: [Figure]
Revised: [Figure]

Original: This dataless model merging is thus an extreme case of federated learning, where a single round of synchronization is admissible.
Revised: This dataless model merging is thus an extreme case of federated learning, where a single round of synchronization is admissible.

Original: Figure 1 provides an overview of the various related setups.
Revised: Figure 1 provides an overview of the various related setups.

Original: We thus aim to use model merging to build a single model that can be used for inference on multiple domains or tasks and can generalize to new domains, in line with (Wang et al., 2020b).
Revised: We thus aim to use model merging to build a single model that can be used for inference on multiple domains or tasks and can generalize to new domains, in line with (Wang et al., 2020b).

Original: In contrast, simple averaging of weights for model merging was used by existing works such as Wortsman et al.
Revised: In contrast, simple averaging of weights for model merging was used by existing works such as Wortsman et al.

Original: (2022) to improve the performance of a specific model, where weight averaging was done over models fine-tuned using the same data set with different hyperparameters.
Revised: (2022) to improve the performance of a specific model, where weight averaging was done over models fine-tuned using the same data set with different hyperparameters.

Original: Separately, Matena & Raffel (2021) focus on improving performance over a single target task by leveraging models trained on other donor tasks by merging models using Fisher-weighted averaging.
Revised: Separately, Matena & Raffel (2021) focus on improving performance over a single target task by leveraging models trained on other donor tasks by merging models using Fisher-weighted averaging.

Original: This paper focuses on merging fine-tuned models that originate fromthe same pre-trained language model.
Revised: This paper focuses on merging fine-tuned models that originate from pre-trained language models with the same architecture and pretrained weights .

Original: We introduce a novel model merging method named Regression Mean (RegMean), which is computationally efficient and extendable to merging any number of models.
Revised: We introduce a novel model merging method named Regression Mean (RegMean), which is computationally efficient and extendable to merging any number of models.

Original: The method is inspired by the optimal solution for linear models that minimizes ℓ 2 distance between merged and individual models and has a closed form solution.
Revised: The method is inspired by the optimal solution for linear models that minimizes ℓ 2 distance between merged and individual models and has a closed form solution.

Original: We evaluate model merging algorithms in setups that range in complexity and type of fused knowledge.
Revised: We evaluate model merging algorithms in setups that range in complexity and type of fused knowledge.

Original: The experimental results across multiple model types (e.g. RoBERTa, T5, DeBERTa) show that our proposed method consistently and significantly outperforms other model merging and ensembling baselines and achieves higher generalization performance than the best individual models on out-of-domain data sets across several data collections.
Revised: The experimental results across multiple model types (e.g. RoBERTa, T5, DeBERTa) show that our proposed method consistently and significantly outperforms other model merging and ensembling baselines and achieves higher generalization performance than the best individual models on out-of-domain data sets across several data collections.

Original: Our contributions are three-fold: (1) A novel model merging algorithm
Revised: Our contributions are three-fold: (1) A novel model merging algorithm

Original: (Regression Mean); (2) an evaluation protocol for model merging algorithms that tests both in-domain and out-of-domain generalization ability; (3) analysis of computation and parameter efficiency across setups.
Revised: (Regression Mean); (2) an evaluation protocol for model merging algorithms that tests both in-domain and out-of-domain generalization ability; (3) analysis of computation and parameter efficiency across setups.

Original:  D ATALESS M ODEL M ERGING FOR K NOWLEDGE F USION
Revised:  D ATALESS M ODEL M ERGING FOR K NOWLEDGE F USION

Original: We consider the problem formulation that there are two main roles in the framework: (1) the agents ( e.g. , individuals or organizations) that train and release models; (2) the developers who aim to build a single model by fusing knowledge of multiple available models.
Revised: We consider the problem formulation that there are two main roles in the framework: (1) the agents ( e.g. , individuals or organizations) that train and release models; (2) the developers who aim to build a single model by fusing knowledge of multiple available models.

Original: Each agent i ∈ { 1 ..N } fine-tunes a language model (LM) f i of pre-trained weights θ LM over their private labeled dataset
Revised: Each agent i ∈ { 1 ..N } fine-tunes a language model (LM) f i of pre-trained weights θ LM over their private labeled dataset

Original: D i =
Revised: D i =

Original: ⟨ X i , Y i ⟩ to obtain fine-tuned model weights θ
Revised: ⟨ X i , Y i ⟩ to obtain fine-tuned model weights θ

Original: i , where X i ∈
Revised: i , where X i ∈

Original: R N i , ∗ are inputs, Y i ∈
Revised: R N i , ∗ are inputs, Y i ∈

Original: R N i , ∗ are labels and N i is the number of annotated examples.
Revised: R N i , ∗ are labels and N i is the number of annotated examples.

Original: The agents keep the labeled data set D i private.
Revised: The agents keep the labeled data set D i private.

Original: In addition to the fine-tune model weights f i ( · ; θ i ) , the agents can also optionally disseminate certain statistics S i , as long as these do not leak information about the labeled data set D i .
Revised: In addition to the fine-tune model weights f i ( · ; θ i ) , the agents can also optionally disseminate certain statistics S i , as long as these do not leak information about the labeled data set D i .

Original: In turn, the developers use the fine-tuned model weights f i ( · ; θ i ) and statistics S i as inputs to a merging function g .
Revised: In turn, the developers use the fine-tuned model weights f i ( · ; θ i ) and statistics S i as inputs to a merging function g .

Original: The merging function is applied to a subset of fine tuned models K ⊆ { 1 ..N } (of size K = |K| ) to obtain parameters θ M K of a merged model f M K , where θ M K = g
Revised: The merging function is applied to a subset of fine tuned models K ⊆ { 1 ..N } (of size K = |K| ) to obtain parameters θ M K of a merged model f M K , where θ M K = g

Original: [Equation]
Revised: ( θ K , S K ) .

Original: In general, we expect the function g to be computationally efficient and to produce θ M K with a closed-form formulation.
Revised: In general, we expect the function g to be computationally efficient and to produce θ M K with a closed-form formulation.

Original:  R EGRESSION M EAN FOR M ODEL M ERGING
Revised:  R EGRESSION M EAN FOR M ODEL M ERGING

Original: The key role in the model merging setup is played by the merging function g .
Revised: The key role in the model merging setup is played by the merging function g .

Original: We start with briefly introducing existing techniques for model merging, followed by the basic intuition for our proposedmethod, which we then extend to transformer-based language models. The underlying assumption
Revised: We start with briefly introducing existing techniques for model merging, followed by the basic intuition for our proposed

Original: Linear
Revised: Linear

Original: Linear Linear
Revised: Linear Linear

Original: Attention
Revised: Attention

Original: Linear Linear
Revised: Linear Linear

Original: Linear Linear
Revised: Linear Linear

Original: Attention
Revised: Attention

Original: Linear 𝑌 = 𝑊 !"
Revised: Linear 𝑌 = 𝑊 !"

Original: 𝑋 !
Revised: 𝑋 !

Original: ⨯⨯⨯
Revised: ⨯⨯⨯

Original: 𝑌 =
Revised: 𝑌 =

Original: 𝑊 #" 𝑋 # ⨯ ⨯⨯ 𝑋 !"
Revised: 𝑊 #" 𝑋 # ⨯ ⨯⨯ 𝑋 !"

Original: 𝑋 !
Revised: 𝑋 !

Original: 𝑋 #" 𝑋 # ⨯ ⨯⨯ ⨯ ⨯⨯
Revised: 𝑋 #" 𝑋 # ⨯ ⨯⨯ ⨯ ⨯⨯

Original: Metrics: ℓ # Optimal 𝑊 $ ? + 𝑋 !"
Revised: Metrics: ℓ # Optimal 𝑊 $ ? + 𝑋 !"

Original: 𝑋 !
Revised: 𝑋 !

Original: 𝑋 #" 𝑋 # + -𝑊 !
Revised: 𝑋 #" 𝑋 # + -𝑊 !

Original: + / +
Revised: + / +

Original: Simple
Revised: Simple

Original: Fisher
Revised: Fisher

Original: RegMean 𝑋 !
Revised: RegMean 𝑋 !

Original: 𝑋 # 𝑊 # 𝑊 ! 𝑊 # 𝑊 !
Revised: 𝑋 # 𝑊 # 𝑊 ! 𝑊 # 𝑊 !

Original: 𝑊 # 𝐹 !
Revised: 𝑊 # 𝐹 !

Original: 𝐹 # ⊙ ⊙  is that the model architecture for all models f i is the same, allowing for element-wise operations if needed and resulting in a merged model f M K of the same architecture and size as any individual model.
Revised: 𝐹 # ⊙ ⊙ method, which we then extend to transformer-based language models. The underlying assumption is that the model architecture for all models f i is the same, allowing for element-wise operations if needed and resulting in a merged model f M K of the same architecture and size as any individual model.

Original: We also assume models are fine-tuned from the same pretrained LM checkpoint.
Revised: We also assume models are fine-tuned from the same pretrained LM checkpoint.

Original: The study of methods that relax this constraint are outside the scope of this paper and are left for future work.
Revised: The study of methods that relax this constraint are outside the scope of this paper and are left for future work.

Original: 3. P RELIMINARIES
Revised: 3. P RELIMINARIES

Original: Simple Averaging (Simple) computes the merged weights as the element-wise arithmetic mean of the weights of all other models: θ M K = 1 /K (cid:80) i ∈K i
Revised: Simple Averaging (Simple) computes the merged weights as the element-wise arithmetic mean of the weights of all other models: θ M K = 1 /K (cid:80) i ∈K i

Original: θ i .
Revised: θ i .

Original: This technique was proved to be effective when merging model weights that are already similar or in a similar space, such as checkpoints generated after each epoch in a training process (Wortsman et al., 2022).
Revised: This technique was proved to be effective when merging model weights that are already similar or in a similar space, such as checkpoints generated after each epoch in a training process (Wortsman et al., 2022).

Original: We expect simple averaging to under-perform when model weights live in a different space and are substantially different to each other, such as when merging models trained with different data or when performing merging for models fine-tuned after the entire training process, as opposed to synchronizing models after rounds as in the federated learning setup.
Revised: We expect simple averaging to under-perform when model weights live in a different space and are substantially different to each other, such as when merging models trained with different data or when performing merging for models fine-tuned after the entire training process, as opposed to synchronizing models after rounds as in the federated learning setup.

Original: Fisher-Weighted Averaging (Fisher) aims to address the limitation of simple averaging of weights with potentially different importance.
Revised: Fisher-Weighted Averaging (Fisher) aims to address the limitation of simple averaging of weights with potentially different importance.

Original: The method relies on computing per-weight importance F i for each individual model i , and reweighting the weights with this importance factor during merging as follows: θ 
Revised: The method relies on computing per-weight importance F i for each individual model i , and reweighting the weights with this importance factor during merging as follows: θ M K = (cid:80)

Original: [Equation]
Revised: [Equation]

Original: F i .
Revised: F i .

Original: Here, F i is the diagonal of the Fisher Information matrix, where F i = E x ∼ D
Revised: Here, F i is the diagonal of the Fisher Information matrix, where F i = E x ∼ D

Original: i E y ∼ p
Revised: i E y ∼ p

Original: θ ( y | x ) ∇ θ
Revised: θ ( y | x ) ∇ θ

Original: i (log p θ i ( y | x i )) 2 .
Revised: i (log p θ i ( y | x i )) 2 .

Original: Intuitively, F i measures averaged gradient norm of parameters w.r.t.
Revised: Intuitively, F i measures averaged gradient norm of parameters w.r.t.

Original: log likelihood of each label, where parameters with high average norms are considered important.
Revised: log likelihood of each label, where parameters with high average norms are considered important.

Original: 3. M ERGING L INEAR M ODELS
Revised: 3. M ERGING L INEAR M ODELS

Original: Next, we recast the problem of model merging as a straightforward optimization problem.
Revised: Next, we recast the problem of model merging as a straightforward optimization problem.

Original: We start by inferring the optimal solution of merging two linear regression models trained on different data distributions and analyze its relationship to Simpleand Fisher-weighted averaging.
Revised: We start by inferring the optimal solution of merging two linear regression models trained on different data distributions and analyze its relationship to Simple averaging.

Original: Consider two linear models f 1 ( x )
Revised: Consider two linear models f 1 ( x )

Original: =
Revised: =

Original: W T 1 x and f 2 ( x ) =
Revised: W T 1 x and f 2 ( x ) =

Original: W T 2 x , where x ∈ R m , and W 1 , W 2 ∈ R m × n , that are trained on two different annotated datasets ⟨ X 1 , y 1 ⟩ , ⟨ X 2 , y 2 ⟩ , where X 1 ∈ R N 1 × m and X 2 ∈ R N 2 × m .
Revised: W T 2 x , where x ∈ R m , and W 1 , W 2 ∈ R m × n , that are trained on two different annotated datasets ⟨ X 1 , y 1 ⟩ , ⟨ X 2 , y 2 ⟩ , where X 1 ∈ R N 1 × m and X 2 ∈ R N 2 × m .

Original: Each row in X i corresponds to a training example.
Revised: Each row in X i corresponds to a training example.

Original: Our goal is to obtain a single merged model f M ( x )
Revised: Our goal is to obtain a single merged model f M ( x )

Original: =
Revised: =

Original: W TM x with outputs similar to f 1 on X 1 and f 2 on X 2 .
Revised: W TM x with outputs similar to f 1 on X 1 and f 2 on X 2 .

Original: With ℓ 2 distance as the metric, the optimization problem can be formulated as:
Revised: With ℓ 2 distance as the metric, the optimization problem can be formulated as:

Original: [Equation]
Revised: [Equation]

Original: Eq. 1 describes a linear regression problem, where the inputs are [ X 1 ; X 2 ] (row concatenation of Xand X 2 ) and the targets are [ W T 1 X 1 ; W T 2 X 2 ] , which has a closed form solution W M =
Revised: Eq. 1 describes a linear regression problem, where the inputs are [ X 1 ; X 2 ] (row concatenation of Xand X 2 ) and the targets are [ W T 1 X 1 ; W T 2 X 2 ] , which has a closed form solution W M =

Original: ( X T 1 X 1 +
Revised: ( X T 1 X 1 +

Original: X T 2 X 2 ) − 1 ( X T 1 X 1 W 1 + X T 2 X 2 W 2 ) .
Revised: X T 2 X 2 ) − 1 ( X T 1 X 1 W 1 + X T 2 X 2 W 2 ) .

Original: The algorithm extends to merging K models W i , i ∈ K with little modifications to the optimization problem in Eq.
Revised: The algorithm extends to merging K models W i , i ∈ K with little modifications to the optimization problem in Eq.

Original: [Equation]
Revised: [Equation]

Original: We refer to Eq.
Revised: We refer to Eq.

Original: 2 as Regression Mean (RegMean).
Revised: 2 as Regression Mean (RegMean).

Original: To summarize, to merge a linear model f i with other models, wefirst pre-compute the Gram matrices of training data 
Revised: To summarize, to merge a linear model f i with other models, we pre-compute the inner product matrices of training data X T

Original: X Ti X i ; we do not recompute 
Revised: i X i ; we do not recompute X

Original: Gram matrices when merging with different models.
Revised: Ti X i when merging with different models.

Original: The merger retrieves the weights and Gram matrices of inputs of individual models and compute the weights as in Eq. 2.
Revised: The merger retrieves the weights and inner product matrices of inputs of individual models and compute the weights as in Eq. 2.

Original: Relationship to Simple Averaging .

Original: We note that when all X Ti X i (or all X i ) are the same, Eq.transforms into simple averaging, i.e. , W M = 1 /K (cid:80) i ∈K i

Original: W i .

Original: Interpretation .
Revised: Interpretation .

Original: RegMean can be also interpreted as reweighting and linearly combing rows in weight matrices, where the diagonal items of X Ti X i mainly reweight the rows, while non-diagonal items linearly combine them.
Revised: RegMean can be also interpreted as reweighting and linearly combing rows in weight matrices, where the diagonal items of X Ti X i mainly reweight the rows, while non-diagonal items linearly combine them.

Original: In an extreme case when X Ti X i is diagonal, RegMean simply reweights the rows in X i by the importance of neurons (in contrast to “importance of parameters” in Fisher-weighted averaging).
Revised: In an extreme case when X Ti X i is diagonal, RegMean simply reweights the rows in W i by the importance of neurons. Besides, when all X Ti X i (or all X i ) are the same, Eq. 2 transforms into simple averaging, i.e. , W M = 1 /K (cid:80) i ∈K i W i .

Original: 3. R EG M EAN FOR T RANSFORMER L ANGUAGE M ODELS
Revised: 3. R EG M EAN FOR T RANSFORMER L ANGUAGE M ODELS

Original: Transformer models consist of feed forward layers and attention heads where linear layers are important components.
Revised: Transformer models consist of feed forward layers and attention heads where linear layers are important components.

Original: For all linear layers, we independently apply RegMean.
Revised: For all linear layers, we independently apply RegMean.

Original: We recordGram matrices X ( j ) T
Revised: We record X ( j ) T

Original: i
Revised: i

Original: X ( j ) i of each linear layer f ( j ) , where X ( j ) i is the input features of the linear layer.
Revised: X ( j ) i of each linear layer f ( j ) , where X ( j ) i is the input features of the linear layer.

Original: The other types of weights, such as embeddings and bias terms, that represent a small portion of the overall parameter set are merged using simple averaging.
Revised: The other types of weights, such as embeddings and bias terms, that represent a small portion of the overall parameter set are merged using simple averaging.

Original: Reducing Non-Diagonal Items of Gram Matrices.
Revised: Reducing Non-Diagonal Items of Inner Product Matrices.

Original: We empirically find that directly applying 
Revised: We empirically find that directly applying Eq.

Original: Eq. 2 for merging yields degenerated models in case of some pre-trained LM architectures.
Revised: 2 for merging yields degenerated models in case of some pre-trained LM architectures.

Original: We therefore decrease the non-diagonal items of the Gram matrices by multiplying them with a scalar α (set as 0.9 most of the times).
Revised: We therefore decrease the non-diagonal items of the inner product matrices by multiplying them with a scalar α (set as 0.9 most of the times).

Original: This also corresponds to adding a regularization term in the optimization objective in Eq.
Revised: This also corresponds to adding a regularization term in the optimization objective in Eq.

Original: 1 that penalizes the Euclidean distance between the merged weights W M and individual model weights W 1 ..K .
Revised: 1 that penalizes the Euclidean distance between the merged weights W M and individual model weights W 1 ..K .

Original: We include a formal derivation and proof in Appendix A.
Revised: We include a formal derivation and proof in Appendix A.

Original: We illustrate RegMean in Figure 2 and summarize the complete RegMean method in Algorithm 1.
Revised: We illustrate RegMean in Figure 2 and summarize the complete RegMean method in Algorithm 1.

Original: [Table]
Revised: [Table]

Original:  i for all linear layers 1 ≤ j ≤ J and models 1 ≤ i ≤ K , Scaling factor of non-diagonal items α Result:
Revised: ) i = X ( j ) T i X ( j ) i for all linear layers 1 ≤ j ≤ J and models 1 ≤ i ≤ K , Scaling factor of non-diagonal items α Result:

Original: Merged model f M for j in 1 , 2 , ..., J do
Revised: Merged model f M for j in 1 , 2 , ..., J do

Original: W ( j ) 1 , W ( j ) 2 ...,
Revised: W ( j ) 1 , W ( j ) 2 ...,

Original: W ( j ) K ← getLinearWeights ( f 1 ..K , j ) ;
Revised: W ( j ) K ← getLinearWeights ( f 1 ..K , j ) ;

Original: Reduce non-diagonal items of Gram matrices G ( j ) i as ˜ G ( j ) i ← αG ( j ) i + (1 − α ) diag ( G ( j ) i ) ;
Revised: Reduce non-diagonal items of inner product matrices G ( j ) i as ˜ G ( j ) i ← αG ( j ) i + (1 − α ) diag ( G ( j ) i ) ;

Original: W ( j ) M ← ( (cid:80) i ∈K i
Revised: W ( j ) M ← ( (cid:80) i ∈K i

Original: ˜ G
Revised: ˜ G

Original: ( j ) i ) − 1 (cid:80) i ∈K i
Revised: ( j ) i ) − 1 (cid:80) i ∈K i

Original: ( ˜ G ( j ) i W ( j ) i ) and set the weight as W ( j ) M in f M end
Revised: ( ˜ G ( j ) i W ( j ) i ) and set the weight as W ( j ) M in f M end

Original: Average weights as W M = 1 K (cid:80) i ∈K i
Revised: Average weights as W M = 1 K (cid:80) i ∈K i

Original: W i for weights other than linear layer weights in f M
Revised: W i for weights other than linear layer weights in f M

Original: 3. P ROPERTIES OF R EG M EAN
Revised: 3. P ROPERTIES OF R EG M EAN

Original: Computational Efficiency.
Revised: Computational Efficiency.

Original: Gram matrices of all linear layer inputs can be computed within one single forward pass over training data after individual models are trained.
Revised: Inner product matrices of all linear layer inputs can be computed within one single forward pass over training data after individual models are trained.

Original: It is more efficient than computing Fisher Information matrices, which requires an additional backward pass to compute gradients.
Revised: It is more efficient than computing Fisher Information matrices, which requires an additional backward pass to compute gradients.

Original: In practice, we compute Gram matrices with at most 1 k training batches.
Revised: In practice, we compute inner product matrices with at most 1 , 000 training batches.

Original: Memory Overhead .
Revised: Memory Overhead .

Original: The memory overhead of Gram matrices is (cid:80)
Revised: The memory overhead of inner product matrices is (cid:80)

Original: Jj =1 d 2 j , where J is the number of linear layers in the model and d j is the input dimension of linear layers.
Revised: Jj =1 d 2 j , where J is the number of linear layers in the model and d j is the input dimension of linear layers.

Original: For transformer models, this overhead is comparable to the number of parameters and Fisher Information matrices (which has the same overhead as the number of parameters).
Revised: For transformer models, this overhead is comparable to the number of parameters and Fisher Information matrices.

Original: Data Privacy .
Revised: Data Privacy .

Original: It should be noted that RegMean never requires training data X i when merging; instead, it only requires low-dimensional Gram matrices.
Revised: It should be noted that RegMean never requires training data X i when merging; instead, it only requires low-dimensional inner product matrices.

Original: The agents that train the models can share theGram matricesalong with the models without sharing the private training data and their labels.
Revised: The agents that release the models can share the matrices without sharing the private training data and their labels.

Original:  E XPERIMENTAL S ETUP 4. E VALUATION S ETTINGS
Revised:  E XPERIMENTAL S ETUP 4. E VALUATION S ETTINGS

Original: We expect two major benefits of merging models for the developer.
Revised: We expect two major benefits of merging models for the developer.

Original: First, by combing knowledge of individual models f 1 ..N (or a subset K of them, f K ) trained on D 1 ..
Revised: First, by combing knowledge of individual models f 1 ..N (or a subset K of them, f K ) trained on D 1 ..

Original: N , we expect the resulting merged model f M to achieve competitive test performance across all datasets D 1 ..N .
Revised: N , we expect the resulting merged model f M to achieve competitive test performance across all datasets D 1 ..N .

Original: This model is useful for example when the test distribution is a mixture of D 1 ..N .
Revised: This model is useful for example when the test distribution is a mixture of D 1 ..N .

Original: In addition, a single model has the additional advantage of being able to run inference across multiple domains when the user of the model provides data from one of the domains, but is not aware of the domain label (Wang et al., 2020b).
Revised: In addition, a single model has the additional advantage of being able to run inference across multiple domains when the user of the model provides data from one of the domains, but is not aware of the domain label (Wang et al., 2020b).

Original: In our case, D 1 ..N can represent different non-i.i.d.
Revised: In our case, D 1 ..N can represent different non-i.i.d.

Original: partitions of the same dataset, different domains for the same task or different tasks altogether.
Revised: partitions of the same dataset, different domains for the same task or different tasks altogether.

Original: Second, we expect the merged model to achieve higher out-of-domain (OOD) generalization ability.
Revised: Second, we expect the merged model to achieve higher out-of-domain (OOD) generalization ability.

Original: Formally, we evaluate the performance of the merged model f M over the out-of-domain test sets
Revised: Formally, we evaluate the performance of the merged model f M over the out-of-domain test sets

Original: D
Revised: D

Original: o 1 ..
Revised: o 1 ..

Original: N o where the data distributions are different from any of D 1 ..N .
Revised: N o where the data distributions are different from any of D 1 ..N .

Original: Datasets .
Revised: Datasets .

Original: We use the GLUE datasets (Wang et al., 2018) for studying merging models trained for non-i.i.d.
Revised: We use the GLUE datasets (Wang et al., 2018) for studying merging models trained for non-i.i.d.

Original: partitions and merging models trained for different tasks.
Revised: partitions and merging models trained for different tasks.

Original: We use emotion classification and named entity recognition (NER) as base tasks for studying merging models trained on different domains of the same task.
Revised: We use emotion classification and named entity recognition (NER) as base tasks for studying merging models trained on different domains of the same task.

Original: For emotion classification, we use the collection of preprocessed datasets from (Oberl¨ander & Klinger, 2018).
Revised: For emotion classification, we use the collection of preprocessed datasets from (Oberl¨ander & Klinger, 2018).

Original: We choose 5 high-resource datasets for training individual models and 5 low-resources datasets for evaluation of out-of-domain generalization ability.
Revised: We choose 5 high-resource datasets for training individual models and 5 low-resources datasets for evaluation of out-of-domain generalization ability.

Original: For
Revised: For

Original: NER tasks, we use 6 domains in OntoNotes (Hovy et al., 2006) for training individual models, and use CoNLL (Sang & De Meulder, 2003) and Twitter NER (Rijhwani & Preotiuc-Pietro, 2020) to measure out-of-domain generalization performance.
Revised: NER tasks, we use 6 domains in OntoNotes (Hovy et al., 2006) for training individual models, and use CoNLL (Sang & De Meulder, 2003) and Twitter NER (Rijhwani & Preotiuc-Pietro, 2020) to measure out-of-domain generalization performance.

Original: We include details of datasets in Apppendix B.
Revised: We include details of datasets in Apppendix B.

Original: Metrics .
Revised: Metrics .

Original: In the case of merging models trained on non-i.i.d.
Revised: In the case of merging models trained on non-i.i.d.

Original: partitions of the same dataset, we evaluate the merged models over a single test set with a joint distribution of all partitions.
Revised: partitions of the same dataset, we evaluate the merged models over a single test set with a joint distribution of all partitions.

Original: For merging models trained on different domains or tasks, we measure the performance over all single domains or tasks incorporated into merging and take their macro-average.
Revised: For merging models trained on different domains or tasks, we measure the performance over all single domains or tasks incorporated into merging and take their macro-average.

Original: For out-of-domain evaluation, we similarly take macro-average over the performance over the out-of-domain test sets.
Revised: For out-of-domain evaluation, we similarly take macro-average over the performance over the out-of-domain test sets.

Original: 4. C OMPARED M ETHODS
Revised: 4. C OMPARED M ETHODS

Original: Model Merging.
Revised: Model Merging.

Original: For model merging algorithms, we compare the performance of RegMean with the previously introduced methods of simple averaging ( Simple ) (Wortsman et al., 2022) and Fisherweighted averaging ( Fisher ) (Matena & Raffel, 2021).
Revised: For model merging algorithms, we compare the performance of RegMean with the previously introduced methods of simple averaging ( Simple ) (Wortsman et al., 2022) and Fisherweighted averaging ( Fisher ) (Matena & Raffel, 2021).

Original: Model Ensembling.
Revised: Model Ensembling.

Original: Model ensembling represents an alternative to model merging when access to the original data is not available.
Revised: Model ensembling represents an alternative to model merging when access to the original data is not available.

Original: We thus build an ensemble model ( Ensemble ) by obtaining all logits from the individual model predictions and  doing an argmax.
Revised: We thus build an ensemble model ( Ensemble ) by obtaining all logits from the individual model predictions and averaging them before doing an argmax.

Original: Individual Models.
Revised: Individual Models.

Original: To provide context into the benefits of merging, we report the performance of individual models involved in merging.
Revised: To provide context into the benefits of merging, we report the performance of individual models involved in merging.

Original: We thus report: (1) the average performance of all individual models ( Avg. f 1 ..N ); (2) the performance of the best single individual model ( Best. f 1 ..N ), as determined by using the validation set; (3) the performance of the individual models corresponding to the training data set for each test set ( Domain-Specific ).
Revised: We thus report: (1) the average performance of all individual models ( Avg. f 1 ..N ); (2) the performance of the best single individual model ( Best. f 1 ..N ), as determined by using the validation set; (3) the performance of the individual models corresponding to the training data set for each test set ( Domain-Specific ).

Original: Multi-task Learning (MTL).
Revised: Multi-task Learning (MTL).

Original: We also consider MTL which trains a single model over the joint training data sets D 1 ..N .
Revised: We also consider MTL which trains a single model over the joint training data sets D 1 ..N .

Original: We note that the multi-task method should represent an upper-bound for model merging, as multi-task learning has access to the original labeled data which it can leverage to train a better model when compared to dataless approaches such as model merging.
Revised: We note that the multi-task method should represent an upper-bound for model merging, as multi-task learning has access to the original labeled data which it can leverage to train a better model when compared to dataless approaches such as model merging.

Original: Depending on the data sets, the task can be the same ( e.g. , emotion prediction) or different ( e.g. , GLUE tasks).
Revised: Depending on the data sets, the task can be the same ( e.g. , emotion prediction) or different ( e.g. , GLUE tasks).

Original: 4. E XPERIMENT D ETAILS
Revised: 4. E XPERIMENT D ETAILS

Original: Pre-trained Models .
Revised: Pre-trained Models .

Original: We initialize all models
Revised: We initialize all models

Original: f
Revised: f

Original: i using the same architecture and by using the same pre-trained model weights θ LM .
Revised: i using the same architecture and by using the same pre-trained model weights θ LM .

Original: We experiment with multiple pre-trained models as starting points for merging.
Revised: We experiment with multiple pre-trained models as starting points for merging.

Original: We experiment with both encoder-only models including the classic RoBERTabase (Liu et al., 2019) and state-of-the-art models like DeBERTa-large-v3 (He et al., 2021) and with  sequence  tasks We finetune models a batch size of 16 for on GLUE emotion classification details
Revised: We experiment with both encoder-only models including the classic RoBERTabase (Liu et al., 2019) and state-of-the-art models like DeBERTa-large-v3 (He et al., 2021) and with encoder-decoder models represented by T5-base-v1.1 (Raffel et al., 2020). We note that T5-basev1.1 is not applicable to sequence labelling tasks represented by our NER experiments.

Original:  B.
Revised: Further training details are in Appendix B.

Original: Model Initialization.
Revised: Model Initialization.

Original: It has been shown that model merging is more successful when individual models share the same weight initialization (McMahan et al., 2017).
Revised: It has been shown that model merging is more successful when individual models share the same weight initialization (McMahan et al., 2017).

Original: However, fine-tuning may also introduce new parameters: for Masked Language Models such as RoBERTa, it is common to replace the pre-trained masked LM head with a randomly initialized task-specific classification head before fine-tuning.

Original: In contrast, for Encoder-Decoder models such as T5, the common practice is to convert examples to a sequence-to-sequence format and directly fine-tune the pre-trained LM head for prediction; therefore, no new parameters are introduced in fine-tuning.

Original: In this paper, we focus on merging fine-tuned language models of the same architectures and initialized from the same pretrained model weights θ LM before fine-tuning.
Revised: In this paper, we focus on merging fine-tuned language models of the same architectures and initialized from the same pretrained model weights θ LM before fine-tuning.

Original: For new classification heads, we present the results of both shared initialization ( Same Head Init, SH ) and different initialization ( Diff Head Init,  DH ), as our proposed method is amenable to both.
Revised: For new classification heads, we present the results of both shared initialization ( Same Head Init, SH ) and different initialization ( Diff Head Init, [Figure] DH ), as our proposed method is amenable to both.

Revised: This does not apply to T5 where we fine-tune the pretrained LM head for prediction.

Original: Hyperparameters.
Revised: Hyperparameters.

Original: The only hyperparameter for model merging is the non-diagonal multiplier α which is set to 0 .
Revised: The only hyperparameter for model merging is the non-diagonal multiplier α which is set to 0 .

Original: 9 for all experiments, with the exception of merging multiple T5-base models, where it is 0 .
Revised: 9 for all experiments, with the exception of merging multiple T5-base models, where it is 0 .

Original: Sensitivity analysis for α is presented in Section 5.3.
Revised: Sensitivity analysis for α is presented in Section 5.3.

Original:  R ESULTS
Revised:  R ESULTS

Original: The main goal of our experiments is to benchmark the performance of different dataless model merging methods and compare these with individual model performance before merging.
Revised: The main goal of our experiments is to benchmark the performance of different dataless model merging methods and compare these with individual model performance before merging.

Original: In addition, we aim to situate these methods in context of other methods which represent upper bounds due to having access to more information ( i.e. data for fine-tuning) than model merging.
Revised: In addition, we aim to situate these methods in context of other methods which represent upper bounds due to having access to more information ( i.e. data for fine-tuning) than model merging.

Original: Our experiments examine knowledge fusion from two perspectives: (1) in-domain performance over test data sets similar to those over which individual models are trained, and (2) out-of-domain generalization performance over data sets from held-out domains or tasks.
Revised: Our experiments examine knowledge fusion from two perspectives: (1) in-domain performance over test data sets similar to those over which individual models are trained, and (2) out-of-domain generalization performance over data sets from held-out domains or tasks.

Original: We study performance dynamics in a range of scenarios ranging in difficulty.
Revised: We study performance dynamics in a range of scenarios ranging in difficulty.

Original: First, we study a simple scenario where merging is performed on models are trained on non-i.i.d.
Revised: First, we study a simple scenario where merging is performed on models are trained on non-i.i.d.

Original: partitions of the same data set.
Revised: partitions of the same data set.

Original: Next, we study merging of models trained on different domains of the same task and lastly merging models trained on different tasks.
Revised: Next, we study merging of models trained on different domains of the same task and lastly merging models trained on different tasks.

Original: 5. M ODEL M ERGING FOR F USING I N -D OMAIN K NOWLEDGE
Revised: 5. M ODEL M ERGING FOR F USING I N -D OMAIN K NOWLEDGE

Original: [Table]
Revised: [Table]

Original: Merging Models Trained on Non-i.i.d.
Revised: Merging Models Trained on Non-i.i.d.

Original: Partitions.
Revised: Partitions.

Original: We start with a setup in which we merge models trained on non-i.i.d.
Revised: We start with a setup in which we merge models trained on non-i.i.d.

Original: partitions of the same data set, which is simulated using synthetic data splits over the 8 tasks in the GLUE benchmark.
Revised: partitions of the same data set, which is simulated using synthetic data splits over the 8 tasks in the GLUE benchmark.

Original: For each task, we split training data into two partitions with 1,000 training examples with different label distributions.
Revised: For each task, we split training data into two partitions with 1,000 training examples with different label distributions (details in Appendix B).

Original: We then finetune 8 pairs of individual models over the two partitions and merge each pair of the models.
Revised: We then fine-tune 8 pairs of individual models over the two partitions and merge each pair of the models.

Original: The merged models are evaluated on the official validation sets ( i.e. with a joint distribution of both partitions).
Revised: The merged models are evaluated on the official validation sets ( i.e. with a joint distribution of both partitions).

Original: In Table 1, we find that model merging consistently improves over average performance of individual models across the 8 tasks.
Revised: In Table 1, we find that model merging consistently improves over average performance of individual models across the 8 tasks.

Original: This verifies that weight merging allows combining knowledge from individual models and can lead to a more powerful single model.
Revised: This verifies that weight merging allows combining knowledge from individual models and can lead to a more powerful single model.

Original: We further note that RegMean outperforms simple averaging and is similar in performance to Fisher-weighted averaging.
Revised: We further note that RegMean outperforms simple averaging and is similar in performance to Fisher-weighted averaging.

Original: This is a proof-of-concept that model merging and RegMean work in a simple scenario.
Revised: This is a proof-of-concept that model merging and RegMean work in a simple scenario.

Original: Merging Models Trained on Different Domains.
Revised: Merging Models Trained on Different Domains.

Original: We next shift to a more challenging setup where individual models are trained on data from different domains of the same task.
Revised: We next shift to a more challenging setup where individual models are trained on data from different domains of the same task.

Original: Pairwise Merging.
Revised: Pairwise Merging.

Original: We start by merging pairs of models trained on different domains.
Revised: We start by merging pairs of models trained on different domains.

Original: For emotion classification and NER, we have 10 ( C 25 ) and 15 ( C 26 ) combinations of domain-specific models respectively.
Revised: For emotion classification and NER, we have 10 ( C 2 5 ) and 15 ( C 2 6 ) combinations of domain-specific mod- [Table] [Table]

Original: The boxplots in Fig.
Revised: The boxplots in Fig.

Original: 3 summarize the relative performance drop compared to domain-specific models as
Revised: 3 summarize the relative performance drop compared to domain-specific models as

Original: [Equation]
Revised: [Equation]

Original: where M ( f, D ) denotes the metric score obtained by evaluating f on the test set of D .
Revised: where M ( f, D ) denotes the metric score obtained by evaluating f on the test set of D .

Original: The performance drop is reasonable as the merged model can run inference on both domains; when the test set is a mixture of all domains, the merged model usually outperforms single individual models, as we will see in the next paragraph.
Revised: The performance drop is reasonable as the merged model can run inference on both domains; when the test set is a mixture of all domains, the merged model usually outperforms single individual models, as we will see in the next paragraph.

Original: We see clear differences between model merging algorithms, where RegMean performs the best.
Revised: We see clear differences between model merging algorithms, where RegMean performs the best.

Original: On RoBERTa-base and DeBERTa-large, RegMean reduces performance drop on Emotion from 55% to 12% and 85% to 15% compared to simple average.[Figure] [Figure] [Table]
Revised: On RoBERTa-base and DeBERTa-large, RegMean reduces performance drop on Emotion from 55% to 12% and 85% to 15% compared to simple average.

Original: [Figure]

Original: [Table]

Original: Merging All Domain-Specific Models.
Revised: Merging All Domain-Specific Models.

Original: We further experiment in a setup of merging all 5 or 6 domain-specific models on Emotion Classification and NER.
Revised: We further experiment in a setup of merging all 5 or 6 domain-specific models on Emotion Classification and NER.

Original: Table 2 summarizes the results.
Revised: Table 2 summarizes the results.

Original: [Figure]
Revised: [Figure]

Original: Results show that merging all models is a challenging setup.
Revised: Results show that merging all models is a challenging setup.

Original: The large differences between the average and the best performance of individual models (Avg. f 1 .. N and
Revised: The large differences between the average and the best performance of individual models (Avg. f 1 .. N and

Original: Best f 1 .. N ) indicate the performance of individual models have a high variance.
Revised: Best f 1 .. N ) indicate the performance of individual models have a high variance.

Original: As a result, model ensembling suffers from poor individual models: the improvements are mostly marginal compared to Best f 1 .. N , while on
Revised: As a result, model ensembling suffers from poor individual models: the improvements are mostly marginal compared to Best f 1 .. N , while on

Original: DeBERTa-large on Emotion, the performance is actually lower.
Revised: DeBERTa-large on Emotion, the performance is actually lower.

Original: In contrast, MTL improves performance significantly over Best f 1 .. N and achieves performance similar to or better than domain-specific models, which implies a single model is capable of encoding knowledge of all domains in our experimental setup.
Revised: In contrast, MTL improves performance significantly over Best f 1 .. N and achieves performance similar to or better than domain-specific models, which implies a single model is capable of encoding knowledge of all domains in our experimental setup.

Original: We then compare three different merging algorithms.
Revised: We then compare three different merging algorithms.

Original: RegMean achieves the best in-domain performance on both Emotion and NER tasks, except for DeBERTa-large on Emotion, where Fisher performs slightly better.
Revised: RegMean achieves the best in-domain performance on both Emotion and NER tasks, except for DeBERTa-large on Emotion, where Fisher performs slightly better.

Original: Simple averaging performs poorly (except for
Revised: Simple averaging performs poorly (except for

Original: T5), especially on RoBERTa-base and DeBERTa-large in the emotion tasks.
Revised: T5), especially on RoBERTa-base and DeBERTa-large in the emotion tasks.

Original: We note that Fisher clearly under-performs RegMean in our previous pairwise merging experiments; Fisher-weighted averaging may actually produce a merged model that is very similar to one of the individual model.
Revised: We note that Fisher clearly under-performs RegMean in our previous pairwise merging experiments; Fisher-weighted averaging may actually produce a merged model that is very similar to one of the individual model.

Original: RegMean also outperforms ensembling in all but one of the five scenarios.
Revised: RegMean also outperforms ensembling in all but one of the five scenarios.

Original: RegMean also clearly outperforms Best f 1 .. N on RoBERTa and T5-base on Emotion, which makes model merging with RegMean useful for performance purposes, in addition to the practical convenience of deploying and maintaining a single model for multiple domains.
Revised: RegMean also clearly outperforms Best f 1 .. N on RoBERTa and T5-base on Emotion, which makes model merging with RegMean useful for performance purposes, in addition to the practical convenience of deploying and maintaining a single model for multiple domains.

Original: Merging Models Trained on Different Tasks.
Revised: Merging Models Trained on Different Tasks.

Original: We also experiment with merging models trained on different tasks using DistilBERT-base and RoBERTa-base.
Revised: We also experiment with merging models trained on different tasks using DistilBERT-base and RoBERTa-base.

Original: We train individual models with full training data of 8 GLUE tasks.
Revised: We train individual models with full training data of 8 GLUE tasks.

Original: We do not merge task-specific classification heads as these can have different dimensions depending on the task and output space.
Revised: We do not merge task-specific classification heads as these can have different dimensions depending on the task and output space.

Original: We summarize the results in Figure 4.
Revised: We summarize the results in Figure 4.

Original: [Table]

Original: We again see a similar pattern when comparing model merging techniques with RegMean clearly improving over Simple averaging and Fisher-weighted averaging.
Revised: We again see a similar pattern when comparing model merging techniques with RegMean clearly improving over Simple averaging and Fisher-weighted averaging.

Original: 5. M ODEL M ERGING FOR O UT - OF -D OMAIN G ENERALIZATION
Revised: 5. M ODEL M ERGING FOR O UT - OF -D OMAIN G ENERALIZATION

Revised: [Table]

Revised: [Figure]

Original: Out-of-Domain Generalization when Merging all Domain-Specific Models.
Revised: Out-of-Domain Generalization when Merging all Domain-Specific Models.

Original: Table 3 summarizes OOD generalization performance when merging all domain-specific models.
Revised: Table 3 summarizes OOD generalization performance when merging all domainspecific models.

Original: We see a similar pattern in OOD generalization performance where RegMean in general performs the best across all model merging algorithms.
Revised: We see a similar pattern in OOD generalization performance where RegMean in general performs the best across all model merging algorithms.

Original: The performance is lower than Fisher only on RoBERTa-base and DeBERTa-large with different head initialization.
Revised: The performance is lower than Fisher only on RoBERTa-base and DeBERTa-large with different head initialization.

Original: We also see that RegMean outperforms model ensembling in most cases, which is comparable in the amount of information it can use.
Revised: We also see that RegMean outperforms model ensembling in most cases, which is comparable in the amount of information it can use.

Original: Further, on the emotion classification data sets, it is notable that RegMean achieves higher OOD performance than the best f 1 .. N on T5-base.
Revised: Further, on the emotion classification data sets, it is notable that RegMean achieves higher OOD performance than the best f 1 .. N on T5-base.

Original: We also found that knowledge fusion itself can negatively impact performance when there are poor individual models.
Revised: We also found that knowledge fusion itself can negatively impact performance when there are poor individual models.

Original: On NER, all merging algorithms and even MTL does not achieve better OOD performance on CoNLL and Twitter than picking the Best f 1 .. N , as previously indicated in (Wang et al., 2020b).
Revised: On NER, all merging algorithms and even MTL does not achieve better OOD performance on CoNLL and Twitter than picking the Best f 1 .. N , as previously indicated in (Wang et al., 2020b).

Original: Incrementally Merging a Subset of Models .
Revised: Incrementally Merging a Subset of Models .

Original: In a scenario where OOD performance of each individual model is known ( e.g. , when the validation sets of the OOD data sets are provided), we can mitigate the impact of having poor individual models by merging only a subset K ⊆ { 1 .. N } of models.
Revised: In a scenario where OOD performance of each individual model is known ( e.g. , when the validation sets of the OOD data sets are provided), we can mitigate the impact of having poor individual models by merging only a subset K ⊆ { 1 .. N } of models.

Original: We apply a similar technique as Wortsman et al.
Revised: We apply a similar technique as Wortsman et al.

Original: (2022); Ram ´ e et al.
Revised: ; Ram´e et al.

Original: (2022) which greedily identifies new individual models to merge.
Revised: (2022) which greedily identifies new individual models to merge.

Original: We use their OOD performance on the validation sets to incrementally add models and plot the results in Figure 6.
Revised: We use their OOD performance on the validation sets to incrementally add models and plot the results in Figure 6.

Original: In general, merging only a subset of models is better than merging all models, e.g. , on RoBERTa-base with the same head initialization,
Revised: In general, merging only a subset of models is better than merging all models, e.g. , on RoBERTa-base with the same head initialization,

Original: RegMean outperforms Best f 1 .. N by merging only two models.
Revised: RegMean outperforms Best f 1 .. N by merging only two models.

Original: 5. D ISCUSSION
Revised: 5. D ISCUSSION

Original: [Figure]

Original: Pre-trained Model Impact in Merging.
Revised: Pre-trained Model Impact in Merging.

Original: Our results show that the underlying pre-trained model is an important factor that affects the performance of merged models with performance on RoBERTa-base, T5-base, and DeBERTa-large differing significantly.
Revised: Our results show that the underlying pre-trained model is an important factor that affects the performance of merged models with performance on RoBERTabase, T5-base, and DeBERTa-large differing significantly.

Original: Overall, merging T5-base models is successful even with simple averaging, while DeBERTa-large is hard to merge, which hints to an interaction between merge-ability and pre-training objective.
Revised: Overall, merging T5-base models is successful even with simple averaging, while DeBERTa-large is hard to merge, which hints to an interaction between merge-ability and pre-training objective.

Original: We believe a more comprehensive study of such factors is an interesting direction of future work.
Revised: We believe a more comprehensive study of such factors is an interesting direction of future work.

Original: Impact of Scaling Non-Diagonal Values in Gram Matrices .
Revised: Impact of Scaling Non-Diagonal Values in Inner Product Matrices .

Original: We noticed when α = 1 . 

Revised: We noticed when α = 1 .

Original: ( i.e. , no scaling), RegMean yields degenerated performance on T5-base and DeBERTa when merging two models, while slightly decreasing α to 0.9 eliminates the issue.
Revised: ( i.e. , no scaling), RegMean yields degenerated performance on T5-base and DeBERTa when merging two models, while slightly decreasing α to 0.9 eliminates the issue.

Original: In the other extreme case when α = 0 , the Gram matrices become diagonal and RegMean simply reweigh rows of weight matrices, making the method similar to Simple Average.
Revised: In the other extreme case when α = 0 , the inner product matrices become diagonal and RegMean simply reweigh rows of weight matrices, making the method similar to Simple Average.

Original: We plot the pairwise merging performance of RegMean with 0 ≤ α ≤ 1 in Figure. 5a for T5-base and DeBERTa-large, as well as the performance of merging multiple T5 models in 5b.
Revised: We plot the pairwise merging performance of RegMean with 0 ≤ α ≤ 1 in Figure 5a for T5-base and DeBERTa-large, as well as the performance of merging multiple T5 models in 5b.

Original: We observe[Figure] that the performance of RegMean is mostly stable between α = 0 . 1 and 0.9, but suddenly drops at α = 1 . 0 .
Revised: We observe that the performance of RegMean is mostly stable between α = 0 . 1 and 0.9, but suddenly drops at α = 1 . 0 .

Original: When merging multiple T5-base models, both in-domain and OOD performs reaches maximum at α = 0 .
Revised: When merging multiple T5-base models, both in-domain and OOD performs reaches maximum at α = 0 .

Original: 1 and slowly drops with an increase in α , whereas OOD performance suffers a slightly larger drop. 
Revised: 1 and slowly drops with an increase in α , whereas OOD performance suffers a slightly larger drop. [Figure] Limitations.

Revised: We note that the requirement of inner product matrices in RegMean (and Fisher Information in Fisher-weighted averaging) can be a limitation.

Original:   R ELATED W ORK
Revised: To merge existing models released online without these statistics, a few training examples (see Appendix C for the sensitivity to the number of training examples) are needed to compute them. Besides, there is a risk that inner product matrices may reveal information about training data. Quantitatively measuring information leakage in these statistics should be a good direction of research in the area of privacy.  R ELATED W ORK

Original: Model Merging and Weight Averaging.
Revised: Model Merging and Weight Averaging.

Original: Past research studied model merging for different end goals.
Revised: Past research studied model merging for different end goals.

Original: Izmailov et al.
Revised: Izmailov et al.

Original: ; Gupta et al.
Revised: ; Gupta et al.

Original: ; Wortsman et al.
Revised: ; Wortsman et al.

Original: (2022) aim to improve model performance by averaging weights across different checkpoints or different runs.
Revised: (2022) aim to improve model performance by averaging weights across different checkpoints or different runs.

Original: Cha et al.
Revised: Cha et al.

Original: Arpit et al.
Revised: Arpit et al.

Original: ; Ram´e et al.
Revised: (2021); Ram ´ e et al.

Original: (2022) study domain-generalization by averaging weights of models trained over the same datasets with different configurations.
Revised: (2022) study domain-generalization by averaging weights of models trained over the same datasets with different configurations.

Original: Matena & Raffel (2021) study merging using Fisher-weighted averaging with the aim of improving performance on a single target task by leveraging other ‘donor’ tasks.
Revised: Matena & Raffel (2021) study merging using Fisher-weighted averaging with the aim of improving performance on a single target task by leveraging other ‘donor’ tasks.

Original: Weight averaging was also used by Li et al.
Revised: Weight averaging was also used by Li et al.

Original: for building language models with multi-domain capabilities by pre-training experts over specialized domains.
Revised: for building language models with multi-domain capabilities by pre-training experts over specialized domains.

Original: New experts added to the framework are initialized using weight averaging from the existing experts.
Revised: New experts added to the framework are initialized using weight averaging from the existing experts.

Original: Wang et al.
Revised: Wang et al.

Original: (2022) use weight averaging to fuse knowledge learned when training multiple adapters with the aim of obtaining better few-shot capabilities and increased model robustness.
Revised: (2022) use weight averaging to fuse knowledge learned when training multiple adapters with the aim of obtaining better few-shot capabilities and increased model robustness.

Original: Merging updates of private models is a crucial intermediate step in federated learning (McMahan et al., 2017; Li et al., 2019).
Revised: Merging updates of private models is a crucial intermediate step in federated learning (McMahan et al., 2017; Li et al., 2019).

Original: However, key in federated learning algorithms is that the joint model is iteratively updated in multiple rounds, which is not allowed for model merging.
Revised: However, key in federated learning algorithms is that the joint model is iteratively updated in multiple rounds, which is not allowed for model merging.

Original: The success of simple arithmetic mean for model merging has been explained from the perspective of loss landscapes and linear mode connectivity (Frankle et al., 2020; Neyshabur et al., 2020; Draxler et al., 2018;
Revised: The success of simple arithmetic mean for model merging has been explained from the perspective of loss landscapes and linear mode connectivity (Frankle et al., 2020; Neyshabur et al., 2020; Draxler et al., 2018;

Original: Ainsworth et al., 2022).
Revised: Ainsworth et al., 2022).

Original: Further, improved merging algorithms aim to find matching or permutations between the weights of different models (Singh & Jaggi, 2020; Nguyen et al., 2021;
Revised: Further, improved merging algorithms aim to find matching or permutations between the weights of different models (Singh & Jaggi, 2020; Nguyen et al., 2021;

Original: Ainsworth et al., 2022; Wang et al., 2020a).
Revised: Ainsworth et al., 2022; Wang et al., 2020a).

Original: Knowledge Fusing via Distillation.
Revised: Knowledge Fusing via Distillation.

Original: Recent work has used the knowledge distillation framework to fuse the capabilities of multiple teacher models by distilling them into a smaller student model at fine-tuning or pre-training stage (Khanuja et al., 2021), albeit requiring full access to data for distillation.
Revised: Recent work has used the knowledge distillation framework to fuse the capabilities of multiple teacher models by distilling them into a smaller student model at fine-tuning or pre-training stage (Khanuja et al., 2021), albeit requiring full access to data for distillation.

Original: Dataless distillation, although for computer vision architectures and not using Transformerbased approaches, was attempted in (Lopes et al., 2017; Nayak et al., 2019).
Revised: Dataless distillation, although for computer vision architectures and not using Transformerbased approaches, was attempted in (Lopes et al., 2017; Nayak et al., 2019).

Original: These have the additional disadvantage of not having a closed form solution and are thus not computationally efficient.
Revised: These have the additional disadvantage of not having a closed form solution and are thus not computationally efficient.

Original:  C ONCLUSIONS AND F UTURE W ORK
Revised:  C ONCLUSIONS AND F UTURE W ORK

Original: This paper studied the problem of fusing knowledge of multiple fine-tuned language models by model merging without access to training data.
Revised: This paper studied the problem of fusing knowledge of multiple fine-tuned language models by model merging without access to training data.

Original: We proposed a new method inspired by linear models named Regression Mean (RegMean).
Revised: We proposed a new method inspired by linear models named Regression Mean (RegMean).

Original: We introduced a series of experimental setups in which we demonstrated that our method outperforms other alternatives to dataless merging or ensembling.
Revised: We introduced a series of experimental setups in which we demonstrated that our method outperforms other alternatives to dataless merging or ensembling.

Original: Further, in non-i.i.d.
Revised: Further, in non-i.i.d.

Original: and out-of-domain experiments, we showed that model merging can outperform individually trained models.
Revised: and out-of-domain experiments, we showed that model merging can outperform individually trained models.

Original: Merged models are also very practical, especially when compared to hosting multiple models, as the merging algorithm is very efficient, adds a minimal number of additional parameters and has a similar inference speed to any individual model.
Revised: Merged models are also very practical, especially when compared to hosting multiple models, as the merging algorithm is very efficient, adds a minimal number of additional parameters and has a similar inference speed to any individual model.

Original: The implications of model merging are wide ranging from efficient intermediary-task selection to improve performance to combining models trained with private data in a federated learning setup.
Revised: The implications of model merging are wide ranging from efficient intermediary-task selection to improve performance to combining models trained with private data in a federated learning setup.

Original: Future work can focus on merging models with different initialization or architectures, merging models sequentially at scale or merging pre-trained models before the fine-tuning stage.
Revised: Future work can focus on merging models with different initialization or architectures, merging models sequentially at scale or merging pre-trained models before the fine-tuning stage.

Original: A D ERIVATION OF THE C OMPLETE F ORMULATION OF R EG M EAN
Revised: A D ERIVATION OF THE C OMPLETE F ORMULATION OF R EG M EAN

Original: Consider merging of K linear models.
Revised: Consider merging of K linear models.

Original: We have the optimization problem formulation,
Revised: We have the optimization problem formulation,

Original: [Equation]
Revised: [Equation]

Original: where for all i , W, W i ∈ R m × n , X i ∈
Revised: where for all i , W, W i ∈ R m × n , X i ∈

Original: R N i × m , and Λ i
Revised: R N i × m , and Λ i

Original: = diag
Revised: = diag

Original: ( λ i 1 , λ i 2 , ..., λ iK ) ⪰ 0 .
Revised: ( λ i 1 , λ i 2 , ..., λ iK ) ⪰ 0 .

Original: The second term is a regularization term that encourages W to be close to W i , where λ ij is the regularization strength for j -th row of W i .
Revised: The second term is a regularization term that encourages W to be close to W i , where λ ij is the regularization strength for j -th row of W i .

Original: Here, λ ij can be set as any non-negative values.
Revised: Here, λ ij can be set as any non-negative values.

Original: The optimal solution for this problem is,
Revised: The optimal solution for this problem is,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We compute the gradient of the objective function (noted as L ) w.r.t the merged weight W .
Revised: We compute the gradient of the objective function (noted as L ) w.r.t the merged weight W .

Original: [Equation]
Revised: [Equation]

Original: We see L is convex w.r.t.
Revised: We see L is convex w.r.t.

Original: W .
Revised: W .

Original: Therefore, we may find minizer of L by letting ∂L∂W = 0 .
Revised: Therefore, we may find minizer of L by letting ∂L∂W = 0 .

Original: [Equation]
Revised: [Equation]

Original: Usually, in linear regression, the regularization strength Λ i is manually specified as a constant value.
Revised: Usually, in linear regression, the regularization strength Λ i is manually specified as a constant value.

Original: However, in our case, the scale of X Ti X i may differ a lot across models, layers, or datasets.
Revised: However, in our case, the scale of X Ti X i may differ a lot across models, layers, or datasets.

Original: Therefore, we let Λ i to scale with X Ti X i , and set Λ i = γ diag ( X Ti X i ) , where γ is a fixed scalar, so that,
Revised: Therefore, we let Λ i to scale with X Ti X i , and set Λ i = γ diag ( X Ti X i ) , where γ is a fixed scalar, so that,

Original: [Equation]
Revised: [Equation]

Original: This formulation is equivalent to increasing the scale of diagonal items of Gram matrices
Revised: This formulation is equivalent to increasing the scale of diagonal items of inner product matrices

Original: X Ti X i .
Revised: X Ti X i .

Original: Decreasing all non-diagonal items of Gram matrices by multiplying α = 11+ γ has the same effect, as we have done in Sec. 3.3.
Revised: Decreasing all non-diagonal items of inner product matrices by multiplying α = 11+ γ has the same effect, as we have done in Sec. 3.3.

Original: [Equation]
Revised: [Equation]

Original: B D ETAILS FOR D ATASETS , P REPROCESSING , M ETRICS , AND T RAINING
Revised: B D ETAILS FOR D ATASETS , P REPROCESSING , M ETRICS , AND T RAINING

Original: GLUE.
Revised: GLUE.

Original: For GLUE (Wang et al., 2018) experiments, we use CoLA (Warstadt et al., 2019), SST2 (Socher et al., 2013), MRPC (Dolan & Brockett, 2005), STS-B (Cer et al., 2017), MNLI (Williams et al., 2018),QNLI (Rajpurkar et al., 2016), QQP, and RTE (Giampiccolo et al., 2007) datasets the
Revised: For GLUE (Wang et al., 2018) experiments, we use CoLA (Warstadt et al., 2019), SST2 (Socher et al., 2013), MRPC (Dolan & Brockett, 2005), STS-B (Cer et al., 2017), MNLI (Williams et al., 2018),QNLI (Rajpurkar et al., 2016), QQP, and RTE (Giampiccolo et al., 2007) datasets the

Original: GLUE task collections.
Revised: GLUE task collections.

Original: We run evaluation on the official development sets because test labels are hidden.
Revised: We run evaluation on the official development sets because test labels are hidden.

Original: We compute Matthews Correlation for CoLA, Pearson Correlation for STS-B, and accuracy for all other tasks. 
Revised: We compute Matthews Correlation for CoLA, Pearson Correlation for STS-B, and accuracy for all other tasks. [Table]

Original: To study merging models trained on non-i.i.d. partitions, we construct two partitions for each of the GLUE tasks.
Revised: To study merging models trained on non- partitions, we construct two partitions for each of the GLUE tasks.

Original: We first randomly sample a “key class” from the task and draw 80% of data of the class from the training set and put them into one partition.
Revised: We first randomly sample a “key class” from the task and draw 80% of data of the class from the training set and put them into one partition.

Original: The rest of the data constitute the other partition.
Revised: The rest of the data constitute the other partition.

Original: We uniformly draw examples that do not belong to the “key class” from one partition to the other so that two partitions have the same number of examples.
Revised: We uniformly draw examples that do not belong to the “key class” from one partition to the other so that two partitions have the same number of examples.

Original: We uniformly sub-sample each partition so that each partition has 1,000 training examples.
Revised: We uniformly sub-sample each partition so that each partition has 1,000 training examples.

Original: Emotion.
Revised: Emotion.

Original: For emotion classification, we use the preprocessed datasets by Oberl¨ander & Klinger (2018).
Revised: For emotion classification, we use the preprocessed datasets by Oberl ¨ ander & Klinger (2018).

Original: We use DailyDialogs (Li et al., 2017), CrowdFlower, TEC (Mohammad, 2012), 
Revised: We use DailyDialogs (Li et al., 2017), CrowdFlower, TEC (Mohammad, 2012), Tales-Emotion (Alm et al., 2005), and

Original: TalesEmotion (Alm et al., 2005), and ISEAR (Scherer & Wallbott, 1994) for training domain-specific models.
Revised: ISEAR (Scherer & Wallbott, 1994) for training domain-specific models.

Original: We use Emoint (Mohammad & Bravo-Marquez, 2017), SSEC (Schuff et al., 2017), ElectoralTweets (Mohammad et al., 2015), GroundedEmotions (Liu et al., 2017), and AffectiveText (Strapparava & Mihalcea, 2007) as held-out datasets for evaluating out-of-domain generalization.
Revised: We use Emoint (Mohammad & Bravo-Marquez, 2017), SSEC (Schuff et al., 2017), ElectoralTweets (Mohammad et al., 2015), GroundedEmotions (Liu et al., 2017), and AffectiveText (Strapparava & Mihalcea, 2007) as held-out datasets for evaluating out-of-domain generalization.

Original: All the selected datasets have the classes anger, disgust, fear, joy, sadness, surprise in their label space, while some of them have more classes ( e.g. guilt).
Revised: All the selected datasets have the classes anger, disgust, fear, joy, sadness, surprise in their label space, while some of them have more classes ( e.g. guilt).

Original: For in-domain performance of each dataset, we compute Macro-F1 of all classes that present in the dataset.
Revised: For in-domain performance of each dataset, we compute Macro-F1 of all classes that present in the dataset.

Original: For out-of-domain performance, we only compute Macro-F1 over anger, disgust, fear, joy, sadness, surprise .
Revised: For out-of-domain performance, we only compute Macro-F1 over anger, disgust, fear, joy, sadness, surprise .

Original: In some of the datasets, inputs may be associated with multiple emotion labels.
Revised: In some of the datasets, inputs may be associated with multiple emotion labels.

Original: We therefore formulate the emotion classification task as a multi-label classification task for all datasets. 
Revised: We therefore formulate the emotion classification task as a multi-label classification task for all datasets. [Table]

Original: On RoBERTa and DeBERTa, we create a binary classification head for each class.
Revised: On RoBERTa and DeBERTa, we create a binary classification head for each class.

Original: We exclude the classification heads that are not learned in the training process when merging the weights of classification heads – e.g. if one dataset has the class “guilt” but the other does not, the weights of the classification head for “guilt” of the other model will not be used for merging.
Revised: We exclude the classification heads that are not learned in the training process when merging the weights of classification heads – e.g. if one dataset has the class “guilt” but the other does not, the weights of the classification head for “guilt” of the other model will not be used for merging.

Original: For T5, we reformulate the task into a sequence-to-sequence format with the template: does the sentence express { class name } ?
Revised: For T5, we reformulate the task into a sequence-tosequence format with the template: does the sentence express { class name } ?

Original: { sentence } .
Revised: { sentence } .

Original: with possible outputs yes or no .
Revised: with possible outputs yes or no .

Original: Such an example will be created for each class that present in the dataset.
Revised: Such an example will be created for each class that present in the dataset.

Original: During evaluation, we treat the exact match yes as the the prediction of the positive label, and otherwise treat as prediction of the negative label.
Revised: During evaluation, we treat the exact match yes as the the prediction of the positive label, and otherwise treat as prediction of the negative label.

Original: NER.
Revised: NER.

Original: We use 6 domains (newswire, broadcast news, broadcast conversation, magazine, telephone conversation and web data) in OntoNotes (Hovy et al., 2006) for training 6 domain-specific individual models.
Revised: We use 6 domains (newswire, broadcast news, broadcast conversation, magazine, telephone conversation and web data) in OntoNotes (Hovy et al., 2006) for training 6 domain-specific individual models.

Original: For testing out-of-domain generalization, we use CoNLL Sang & De Meulder (2003) and a Twitter NER data set Rijhwani & Preotiuc-Pietro (2020). 
Revised: For testing out-of-domain generalization, we use CoNLL Sang & De Meulder (2003) and a Twitter NER data set Rijhwani & Preotiuc-Pietro (2020). [Table]

Original: Implementation.
Revised: Implementation.

Original: We use huggingface’s transformer library (Wolf et al., 2019) to download pretrained LM checkpoints and fine-tune the models.
Revised: We use huggingface’s transformer library (Wolf et al., 2019) to download pretrained LM checkpoints and fine-tune the models.

Original: We specifically note that we use the forward function hook feature in PyTorch (Paszke et al., 2019) to obtain the inputs of all linear layers in order to compute Gram matrices.
Revised: We specifically note that we use the forward function hook feature in PyTorch (Paszke et al., 2019) to obtain the inputs of all linear layers in order to compute inner product matrices.

Original: It makes the code implementation of RegMean agnostic to the model architecture. 
Revised: It makes the code implementation of RegMean agnostic to the model architecture. [Table]

Original: Training Details.
Revised: Training Details.

Original: We fine- tune DistilBERT-base, RoBERTa-base, and DeBERTa-large with an initial learning rate 1e-5, and fine-tune T5-base with an initial learning rate 1e-4.
Revised: We fine-tune DistilBERTbase, RoBERTa-base, and DeBERTa-large with an initial learning rate 1e-5, and fine-tune T5base with an initial learning rate 1e-4.

Original: We use AdamW optimizer throughout the experiments.
Revised: We use AdamW optimizer throughout the experiments.

Original: The learning rate gradually warms up in the first 6% of training steps and linearly decay to 0.
Revised: The learning rate gradually warms up in the first 6% of training steps and linearly decay to 0.

Original: We train models with a batch size of 16 and for 10 epochs on GLUE, 30 epochs on emotion classification and 20 epochs on NER.
Revised: We train models with a batch size of 16 and forepochs on GLUE, 30 epochs on emotion classification and 20 epochs on NER.

Original: We evaluate the performance of the model after each epoch and resume the best performing checkpoint at the end of training.
Revised: We evaluate the performance of the model after each epoch and resume the best performing checkpoint at the end of training.

Revised: C S ENSITIVITY A NALYSIS Number of examples for computing inner

Revised: product matrices.

Revised: In our main experiments, we use N = 1 , 000 examples for computing inner product matrices.

Revised: We present additional analysis about the effect of N and summarize results in

Revised: Table 6.

Revised: In general, performance improves as we increase N , but the performance soon saturates around N = 100 .

Revised: [Table]

Revised: Alternative methods for regularization.

Revised: As we mentioned in Sec. 3.3 and Appendix A, we reduce non-diagonal items of inner product matrices by a fixed scale α , which has a regularization effect of encouraging merged weights to be closer to individual model weights.

Revised: Here we present analysis of an alternative regularization method, which adds a fixed scalar β to diagonal items instead of relatively scaling them.

Revised: We experiment with emotion classification on

Revised: T5 where regularization seems to be most necessary.

Revised: We merge each pair of models onemotion classification datasets and report the average performance over all pairs (a setting similar to Figure. 3) in Table 8.

Revised: We see relative scaling achieves clearly better performance than adding a constant to diagonals.

Revised: As we mentioned in Appendix A, this may be caused by differences in the scale of inputs in different layers, models, and datasets, which makes it difficult to find a single additive regularizer.

Revised: [Figure]

Revised: Layer

Revised: Layer

Revised: Layer

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: Choice of models to merge and its effect on OOD performance.

Revised: Table 7 summarizes OOD performance when merging each pair of RoBETa-base emotion classification models with same head initialization with RegMean.

Revised: We see the OOD performance is clearly dependent on the models chosen for merging.

Revised: Merging TEC and ISEAR models, which correspond to two individual models that achieve best OOD performance, produces a model that achieves best OOD performance.

Revised: D P ERMUTATION M ATCHING A LGORITHMS FOR M ERGING L ANGUAGE M ODELS

Revised: Several existing works (Singh & Jaggi, 2020; Ainsworth et al., 2022) propose algorithms to match weight permutations in two models before merging, as models with similar outputs may involve distinct permutations in their weights.

Revised: However, experiments in these works do not cover transformers

Revised: LMs.

Revised: In this section, we present an analysis to address two research questions about permutation matching algorithms in the setup of merging language models fine-tuned from shared pretrained weights: (1) does the issue of weight permutation exist in this setup?

Revised: (2) do existing permutation matching algorithms improve the performance of model merging?

Revised: We experiment with merging two RoBERTa-base models fine-tuned on emotion classification datasets.

Revised: We visualize results on merging models trained on Tales-Emotion and ISEAR in Figuresand 8.

Revised: Weight-Based Matching.

Revised: We apply weight-based matching in OTFusion (Singh & Jaggi, 2020).

Revised: To find permutations between weight matrices W A and W B in the same layer of two different models, the algorithm computes a ground metrics matrix M ∈ R n × n , where n is the dimension of the output.

Revised: Each element M ij ∈ M measures ℓ 2 distance between a pair of weight vectors W : ,iA and W : ,jB .

Revised: [Figure]

Revised: Layer

Revised: Layer

Revised: Layer

Revised: Layer

Revised: Layer

Revised: [Figure]

Revised: Layer

Revised: Layer

Revised: Layer

Revised: Layer

Revised: Layer

Revised: [Figure]

Revised: Layer

Revised: [Figure]

Revised: Layer

Revised: Layer

Revised: Layer

Revised: Layer

Revised: Layer

Revised: [Figure]

Revised: Layer

Revised: Layer

Revised: Layer

Revised: Layer

Revised: Layer

Revised: [Figure]

Revised: Assuming no permutations in weights, we should expect the diagonal items of M (distance of weight vectors in the corresponding positions) to be much smaller than non-diagonal items.

Revised: Otherwise, we may obtain non-trivial permutations by solving an optimal transport problem with M .

Revised: In Figure 7, we visualize the matrix M on the two-layer MLP after each transformer block, which is the only place where linear layers are stacked without residual connections in transformers, making weight permutations most likely to happen.

Revised: However, in Figure 7, we see a clear picture that the diagonal items of M are significantly smaller than non-diagonals.

Revised: The results imply there is no permutations in weights.

Revised: In this case, the permutation matrix we obtain by solving optimal transport is a trivial identity matrix.

Revised: We conjecture that sharing the same pretrained LM weight initialization contributes to stability in training, resulting in no permutations in weights.

Revised: The residual connections in transforms may further prevent weights in other modules from getting permuted.

Revised: Activation-Based Matching.

Revised: We apply activation-based matching in Git Re-Basin (Ainsworth et al., 2022).

Revised: The algorithms relies on a similarity matrix C ∈ R n × n that measures pairwise similarity of activations over N training examples in a certain layer.

Revised: More formally, C is computed as

Revised: Z TA Z B , where Z A , Z B ∈ R N × n are activations at a given layer in the models f A and f B .

Revised: The algorithm solves a linear assignment problem with C to obtain permutations in activations.

Revised: Similarity, if there is no permutation, we expect the diagonal items of C to be large.

Revised: We visualize the matrix C in Figure 8.

Revised: We see a different picture from weight-based matching that C is far from being diagonal.

Revised: This allows activation-based matching algorithms to produce non-trivial permutation matrices.

Revised: However, as we apply these permutations, we obtain performance that is far below simple average without matching.

Revised: We conjecture that in our setup permutations of activations could not faithfully represent permutations in weights.

Revised: Though we just present empirical findings in this paper, we consider figuring out the reasons for such discrepancy as an interesting future work.