Paper: AdaMerging: Adaptive Model Merging for Multi-Task Learning (ICLR-2024)

Reviewer comment (Novelty):
The novelty remains a question for me.

Author response:
## 1. About the question of "The novelty of this paper":

(1) The research direction of this paper is an important direction, which is to merge models trained independently on multiple tasks into one to perform MTL. It can reduce the effort of collecting original training data and avoid the cost of model retraining. However, there is still a significant performance gap between SOTA model merging schemes and traditional MTL, and we observe that the model merging coefficient has a very large impact on performance. In this paper, we first propose AdaMerging to automatically learn Task-wise or Layer-wise merging coefficients. Further, we verify through extensive experiments that unsupervised entropy minimization can be used as a proxy objective for supervised cross-entropy loss to optimize model merging coefficients. Our approach, although seemingly simple, is highly effective. For example, compared with two SOTA task vector-based methods, Task Arithmetic (ICLR'2023) and Ties-Merging (NeurIPS'2023), our AdaMerging and AdaMerging++ bring up to 11% and 8.7% performance improvement, respectively. Even if only 0.1% of the test data is available, our method can bring about a 5% performance improvement. Our method also exhibits stronger generalization and robustness.

| Method | Avg ACC (\%) | | Method | Avg ACC (\%) | 
| :------ | :------: |   :------: |   :------ | :------: | 
| Task Arithmetic (0%)  | 69.1 |    |  Ties-Merging (0%)  |  72.4 | 
| Our AdaMerging (0.1%)   | 74.0 |  |  Our AdaMerging++ (0.1%) | 77.9  | 
| Our AdaMerging (1%)   | 77.5 |  |  Our AdaMerging++ (1%) | 80.6  | 
| Our AdaMerging (5%)   | 80.1 |  |  Our AdaMerging++ (5%) |   81.0 | 
| Our AdaMerging (100%) | 80.1 |  |  Our AdaMerging++ (100%) |  81.1  | 

(2) The tasks and data of our experiments strictly follow the settings of Task Arithmetic[1] and Ties-Merging[2]. We have added similarity calculations for task vectors in the modified version (as shown in Figure 6 in the appendix). We have observed that a small number of task vectors have a certain degree of correlation (such as SVHN and MNIST, which are both digit recognition tasks), and most task vectors have low correlation. That is, these task vectors are almost orthogonal (this is consistent with Task Arithmetic). We did not observe a direct correlation between the 'similarity of the task vectors' and the 'merging coefficients'. We further added task vector merging experiments with different correlation degrees, as shown in Figure 7 in the appendix. We observed that our AdaMerging is effective under various correlation degrees.

(3) The suggestion that "weights learned from task vectors can be used to guide MTL training" may not be suitable for the setting of this paper. Model merging represents only the trained model for each task, without the original training data. Therefore, model merging technology is unsuitable for scenarios where a weighted MTL can be trained directly from scratch. Model merging is mainly used for foundation models or large language models, which are very expensive to train from scratch using raw data.

[1] Ilharco, Gabriel, et al. "Editing models with task arithmetic." ICLR2023.

[2] Yadav, Prateek, et al. "Resolving Interference When Merging Models." NIPS2023.

Reviewer comment (Experiments):
The practical value of this method is not well supported.

Author response:
## 4. About the question of "The limited application of this kind of work":

At this stage, model merging has a certain performance gap with traditional MTL, but it is a potential and popular research direction [1-8]. The traditional MTL method requires the original data of all tasks to be collected together in advance, and then iteratively updates the MTL model, which has the risk of data privacy leakage. Especially in the era of large models, this method of training MTL models from scratch is inefficient. Model merging does not require original task training data, but only requires models trained independently for each task to complete MTL. Reviewer eRin also noted that "developing a single versatile model from diverse off-the-shelf fine-tuned models is important to LLM (or Foundation model) communities".

In addition, regarding the issue of relying on the same pre-trained model. In the era of large models, the backbone of most research work is highly homogeneous and is often based on the same pretrain model. Ties-Merging [1] points out that "these benefits have resulted in the release of *thousands* of fine-tuned checkpoints derived from popular PTMs such as ViT for vision and T5 for language." In addition, learning from models [2] rather than raw data is a critical research direction.

[1] Yadav, Prateek, et al. "Resolving Interference When Merging Models." NIPS 2023.

[2] Zheng, Hongling, et al. "Learn From Model Beyond Fine-Tuning: A Survey." arXiv preprint arXiv:2310.08184 (2023).

[3] Ilharco, Gabriel, et al. "Editing models with task arithmetic." ICLR2023.

[4] Zhang, Jinghan, et al. "Composing parameter-efficient modules with arithmetic operations." arXiv preprint arXiv:2306.14870 (2023).

[5] Li, Weishi, et al. "Deep model fusion: A survey." arXiv preprint arXiv:2309.15698 (2023).

[6] Wang, Song, et al. "Knowledge Editing for Large Language Models: A Survey." arXiv preprint arXiv:2310.16218 (2023).

[7] Chronopoulou, Alexandra, et al. "Language and Task Arithmetic with Parameter-Efficient Layers for Zero-Shot Summarization." arXiv preprint arXiv:2311.09344 (2023).

[8] Ortiz-Jimenez, Guillermo, Alessandro Favero, and Pascal Frossard. "Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models." arXiv preprint arXiv:2305.12827 (2023).

Reviewer comment (Novelty):
I'm not sure the paper has sufficient novelty to be published in the top-tier conference since the proposed method only goes one step further from Task Arithmetic [1] and TIES-MERGING [2] by incorporating trainable weights for task vectors. The concept seems thin to support an entire paper, with only one page (page 6) dedicated to the novel part. Authors should consider diving deeper into this direction. For example, exploring the underlying reasons for the weight relationships between different tasks and their potential correlation with task relationships could enhance the paper's depth. Additionally, the learned weights could be utilized to guide the training of multi-task models, as seen in Auto-lambda [3].

Author response:
## 1. About the question of "The novelty of this paper":

(1) The research direction of this paper is an important direction, which is to merge models trained independently on multiple tasks into one to perform MTL. It can reduce the effort of collecting original training data and avoid the cost of model retraining. However, there is still a significant performance gap between SOTA model merging schemes and traditional MTL, and we observe that the model merging coefficient has a very large impact on performance. In this paper, we first propose AdaMerging to automatically learn Task-wise or Layer-wise merging coefficients. Further, we verify through extensive experiments that unsupervised entropy minimization can be used as a proxy objective for supervised cross-entropy loss to optimize model merging coefficients. Our approach, although seemingly simple, is highly effective. For example, compared with two SOTA task vector-based methods, Task Arithmetic (ICLR'2023) and Ties-Merging (NeurIPS'2023), our AdaMerging and AdaMerging++ bring up to 11% and 8.7% performance improvement, respectively. Even if only 0.1% of the test data is available, our method can bring about a 5% performance improvement. Our method also exhibits stronger generalization and robustness.

| Method | Avg ACC (\%) | | Method | Avg ACC (\%) | 
| :------ | :------: |   :------: |   :------ | :------: | 
| Task Arithmetic (0%)  | 69.1 |    |  Ties-Merging (0%)  |  72.4 | 
| Our AdaMerging (0.1%)   | 74.0 |  |  Our AdaMerging++ (0.1%) | 77.9  | 
| Our AdaMerging (1%)   | 77.5 |  |  Our AdaMerging++ (1%) | 80.6  | 
| Our AdaMerging (5%)   | 80.1 |  |  Our AdaMerging++ (5%) |   81.0 | 
| Our AdaMerging (100%) | 80.1 |  |  Our AdaMerging++ (100%) |  81.1  | 

(2) The tasks and data of our experiments strictly follow the settings of Task Arithmetic[1] and Ties-Merging[2]. We have added similarity calculations for task vectors in the modified version (as shown in Figure 6 in the appendix). We have observed that a small number of task vectors have a certain degree of correlation (such as SVHN and MNIST, which are both digit recognition tasks), and most task vectors have low correlation. That is, these task vectors are almost orthogonal (this is consistent with Task Arithmetic). We did not observe a direct correlation between the 'similarity of the task vectors' and the 'merging coefficients'. We further added task vector merging experiments with different correlation degrees, as shown in Figure 7 in the appendix. We observed that our AdaMerging is effective under various correlation degrees.

(3) The suggestion that "weights learned from task vectors can be used to guide MTL training" may not be suitable for the setting of this paper. Model merging represents only the trained model for each task, without the original training data. Therefore, model merging technology is unsuitable for scenarios where a weighted MTL can be trained directly from scratch. Model merging is mainly used for foundation models or large language models, which are very expensive to train from scratch using raw data.

[1] Ilharco, Gabriel, et al. "Editing models with task arithmetic." ICLR2023.

[2] Yadav, Prateek, et al. "Resolving Interference When Merging Models." NIPS2023.

Reviewer comment (Novelty):
Is it really necessary to conduct experiments to show the relationship between Shannon entropy and cross entropy? Actually from the information theory, the two concepts are almost the same thing or we can say cross entropy is derived from Shannon entropy. It's kind of trivial or even unnecessary to do experiments in Figure 3. Besides, the usage of Shannon entropy to train the adaptive weights also limits the method to only be used for classification tasks.

Author response:
## 2. About the question of "The relationship between Shannon entropy and cross entropy":

Indeed, there is a close relationship between Shannon entropy and cross-entropy in information theory. However, they still have obvious differences in merging coefficients training. Entropy minimization does not require labels, while cross-entropy loss requires labeled data. Experimental confirmation of this relationship can provide readers with a more intuitive understanding that entropy minimization can serve as an effective proxy for cross-entropy loss when merging models. In the modified version (Figure 10 in the appendix), our method further increases the correlation between entropy minimization and cross-entropy loss minimization across different training stages.

In addition, although entropy minimization can only be used for classification tasks, classification tasks are very common in various applications in machine learning, and they cover many practical problems, including but not limited to image recognition, natural language processing, bioinformatics, and other fields.

Reviewer comment (Experiments):
It's better to show the performance of the pre-trained model on each task as well in Tables 1 and 2.

Author response:
## 3. About the question of "The performance of the pre-trained model":

As shown in the below tables, we added the pre-trained model to predict the performance of each task. It can be observed that there is a big gap in performance between pre-trained and Task Arithmetic. Our AdaMerging significantly outperforms them.

| Method (merging ViT-B/32)  | SUN397 | Cars | RESISC45 | EuroSAT | SVHN | GTSRB | MNIST | DTD | Avg | 
| :------ | :------: |  :------: |:------: |  :------: |:------: |  :------: |:------: |  :------: | :------: |  
| Pre-trained   | 62.3 | 59.7 | 60.7  | 45.5  | 31.4  | 32.6  | 48.5  | 43.8  | 48.0  |
| Task Arithmetic   |  55.2 | 54.9  | 66.7  | 78.9 |  80.2 |  69.7 |  97.3 |  50.4  | 69.1 | 
| AdaMerging(Ours) | 64.5 | 68.1 | 79.2 | 93.8 | 87.0 | 91.9 | 97.5 | 59.1 | 80.1| 

| Method (merging ViT-L/14)  | SUN397 | Cars | RESISC45 | EuroSAT | SVHN | GTSRB | MNIST | DTD | Avg | 
| :------ | :------: |  :------: |:------: |  :------: |:------: |  :------: |:------: |  :------: | :------: |  
| Pre-trained   | 66.8  | 77.7  | 71.0  |  59.9 | 58.4  | 50.5  | 76.3  | 55.3  | 64.5  |
| Task Arithmetic   | 73.9 | 82.1 | 86.6 | 94.1 | 87.9 | 86.7 | 98.9 | 65.6 | 84.5|
| AdaMerging(Ours) | 79.0 | 90.3 | 90.8 | 96.2 | 93.4 | 98.0 | 99.0 | 79.9 | 90.8|

Reviewer comment (Experiments):
Limited application of this kind of work. From Tables 1 and 2, we can clearly see that traditional MTL or we say all-shared MTL can achieve a very high accuracy, not to say SOTA MTL methods like AdaShare [4] and AutoMTL[5]. In practice, machine learning engineers might prefer these alternatives due to their superior performance. Besides, for the model merging direction, it's weird to assume that although we may not be able to get the train data for each task, we can still get the pre-trained weights of the model. Most importantly, those task-specific models even need to be trained from the same pre-trained weights.

Author response:
## 4. About the question of "The limited application of this kind of work":

At this stage, model merging has a certain performance gap with traditional MTL, but it is a potential and popular research direction [1-8]. The traditional MTL method requires the original data of all tasks to be collected together in advance, and then iteratively updates the MTL model, which has the risk of data privacy leakage. Especially in the era of large models, this method of training MTL models from scratch is inefficient. Model merging does not require original task training data, but only requires models trained independently for each task to complete MTL. Reviewer eRin also noted that "developing a single versatile model from diverse off-the-shelf fine-tuned models is important to LLM (or Foundation model) communities".

In addition, regarding the issue of relying on the same pre-trained model. In the era of large models, the backbone of most research work is highly homogeneous and is often based on the same pretrain model. Ties-Merging [1] points out that "these benefits have resulted in the release of *thousands* of fine-tuned checkpoints derived from popular PTMs such as ViT for vision and T5 for language." In addition, learning from models [2] rather than raw data is a critical research direction.

[1] Yadav, Prateek, et al. "Resolving Interference When Merging Models." NIPS 2023.

[2] Zheng, Hongling, et al. "Learn From Model Beyond Fine-Tuning: A Survey." arXiv preprint arXiv:2310.08184 (2023).

[3] Ilharco, Gabriel, et al. "Editing models with task arithmetic." ICLR2023.

[4] Zhang, Jinghan, et al. "Composing parameter-efficient modules with arithmetic operations." arXiv preprint arXiv:2306.14870 (2023).

[5] Li, Weishi, et al. "Deep model fusion: A survey." arXiv preprint arXiv:2309.15698 (2023).

[6] Wang, Song, et al. "Knowledge Editing for Large Language Models: A Survey." arXiv preprint arXiv:2310.16218 (2023).

[7] Chronopoulou, Alexandra, et al. "Language and Task Arithmetic with Parameter-Efficient Layers for Zero-Shot Summarization." arXiv preprint arXiv:2311.09344 (2023).

[8] Ortiz-Jimenez, Guillermo, Alessandro Favero, and Pascal Frossard. "Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models." arXiv preprint arXiv:2305.12827 (2023).

Reviewer comment (Experiments):
Though we don’t need to train model again via original training data, we still need to access a certain amount of testing data for testing-time-entropy minimization. How the (minimum) amount of testing data can affect the quality of merging weights ($\lambda$ in the paper) is encouraged to study and present in the paper.

Author response:
Indeed, using all test data for model pooling coefficient training may not be feasible in some scenarios. Based on your suggestions, we experiment with the performance of AdaMerging when different numbers (such as 0.1%, 1%, 5%, 100%) of test sets are available, as shown in the table below. We observe that even when only 0.1% of the test data is available, AdaMerging and AdaMerging++ have about 5% performance improvement compared to Task Arithmetic and Ties-Merging. When 5% of the test data is available, AdaMerging and AdaMerging++ can almost reach the performance when all test data is available. This significantly improves the applicability of our work.
| Method | Avg ACC (\%) | | Method | Avg ACC (\%) | 
| :------ | :------: |   :------: |   :------ | :------: |  
| Task Arithmetic (0%)  | 69.1 |    |  Ties-Merging (0%)  |  72.4 | 
| AdaMerging (0.1%)   | 74.0 |  |  AdaMerging++ (0.1%) | 77.9  | 
| AdaMerging (1%)   | 77.5 |  |  AdaMerging++ (1%) | 80.6  | 
| AdaMerging (5%)   | 80.1 |  |  AdaMerging++ (5%) |   81.0 | 
| AdaMerging (100%) | 80.1 |  |  AdaMerging++ (100%) |  81.1  |

Reviewer comment (Theory):
I didn't see any restriction or regularization on $\lambda$, especially in the optimization objective in Sec. 3.2.2. Does $\lambda$ always need to be $\sum_{i=k}^K \lambda_i = 1$?

Author response:
We do not require $\sum_i \lambda_i =1$, as shown in the visualization of the coefficients in the paper and the merging coefficients learned by Task-wise AdaMerging in the table below.
| Task Weight | SUN397 | Cars | RESISC45 | EuroSAT | SVHN | GTSRB | MNIST | DTD |
| :------ | :------: |  :------: |:------: |  :------: |:------: |  :------: |:------: |  :------: |  
| AdaMerging  | 0.2202 | 0.1413 | 0.2826 | 0.3284 | 0.2841 | 0.4003 | 0.1978 | 0.1692 | 
| AdaMerging++| 0.3171 | 0.1698 | 0.4235 | 0.5198 | 0.4386 | 0.5803 | 0.2452 | 0.2885 |

Reviewer comment (Experiments):
The task relationships across 8 tasks included in the paper can be mentioned. I am curious about the performance changes when we have unrelated tasks and high-correlated tasks.

Author response:
We have added similarity calculations for task vectors in the modified version (as shown in Figure 6 in the Appendix). As shown in the table, we have observed that a small number of task vectors have a certain degree of correlation (such as SVHN and MNIST, which are both digit recognition tasks). Most of the task vectors have low correlation, which means that these task vectors are almost orthogonal (this is consistent with Task Arithmetic). In addition, in the modified version, we further added task vector merging experiments with different degrees of correlation (such as merging SVHN with MNIST, GTSRB, SUN397 and EuroSAT, respectively). As shown in Figure 7, we observe that our AdaMerging is effective when performing model merging at various correlation levels.

|  | SUN397 | Cars | RESISC45 | EuroSAT | SVHN | GTSRB | MNIST | DTD |
| :------ | :------: |  :------: |:------: |  :------: |:------: |  :------: |:------: |  :------: |  
| **Cars**     | 1.00  | 0.03  | 0.04  | 0.03  | 0.02  | 0.03  | 0.03 | 0.04 | 
| **Cars**     | 0.03  | 1.00  | 0.03  | 0.03  | 0.02  | 0.03  | 0.03 | 0.03 |
| **RESISC45** | 0.04  | 0.03  | 1.00  | 0.07  | 0.03  | 0.04  | 0.03 | 0.04 |
| **EuroSAT**  | 0.03  | 0.03  | 0.07  | 1.00  | 0.05  | 0.04  | 0.04 | 0.04 |
| **SVHN**     | 0.02  | 0.02  | 0.03  | 0.05  | 1.00  | 0.08  | 0.15 | 0.03 |
| **GTSRB**    | 0.03  | 0.03  | 0.04  | 0.04  | 0.08  | 1.00  | 0.07 | 0.04 |
| **MNIST**    | 0.03  | 0.03  | 0.03  | 0.04  | 0.15  | 0.07  | 1.00 | 0.03 |
| **DTD**      | 0.04  | 0.03  | 0.04  | 0.04  | 0.03  | 0.04  | 0.03 | 1.00 |

Reviewer comment (Evaluation):
Does the $\lambda$ correlate to the performance gain in Table 1 and Table 2? Is there any relationship between weights and performance gain/loss?

Author response:
Yes, $\lambda$ significantly impacts the accuracy of model mergin. For example, in Figure 1, different $\lambda$ are manually set. An inappropriate $\lambda$ will cause the performance of model merging to be unacceptably low. In Table 1 and Table 2, our AdaMerging is adaptively learned $\lambda$, and the coefficients learned by Layer-wise AdaMerging and AdaMerging++ are shown in Figure 5 of the paper. The coefficients learned by Task-wise AdaMerging and AdaMerging++ are shown in the following table.
| Task Weight | SUN397 | Cars | RESISC45 | EuroSAT | SVHN | GTSRB | MNIST | DTD |
| :------ | :------: |  :------: |:------: |  :------: |:------: |  :------: |:------: |  :------: |  
| AdaMerging  | 0.2202 | 0.1413 | 0.2826 | 0.3284 | 0.2841 | 0.4003 | 0.1978 | 0.1692 | 
| AdaMerging++| 0.3171 | 0.1698 | 0.4235 | 0.5198 | 0.4386 | 0.5803 | 0.2452 | 0.2885 |

Reviewer comment (Evaluation):
The conclusion of **Section 3.2.2** seems a bit strong to me from the conducted experiment: the analysis shows that the entropy and loss of a trained MTL model are nicely correlated, but it does not necessarily mean that they yield equally good directions during training. Doing the same analysis at different timesteps during the MTL model training could show whether and how this correlation holds during training.

Author response:
Based on your suggestions, we have added a correlation analysis of entropy and loss at different training stages. As shown in the table below, the Spearman correlation coefficient of entropy and loss is always high, indicating that **Entropy and Loss are positively correlated at different stages** , and entropy minimization can be used as an effective proxy for cross-entropy loss to optimize the merging coefficient of AdaMerging. Experimental results of Spearman correlation coefficients on eight datasets are added to the modified version of Figure 10.

| Iterations | Spearman correlation coefficient |  
| :------: | :------: |  
| 0     | 0.87 |   
| 100   | 0.89 |  
| 200   | 0.91 |  
| 300   | 0.91 |   
| 400   | 0.92 | 
| 500   | 0.92 |

Reviewer comment (Experiments):
**Discrepancy in supervision**: If I understood correctly, the single-task and MTL baselines use standard supervised training schemes; task merging baselines (e.g., task arithmetic) are data-free methods and only require the task vectors; finally, `AdaMerging` learns task arithmetic weights on unlabelled test samples ($B_k$ on page 6). While these are unsupervised, seeing test samples seems unfair compared to other baselines, especially the task arithmetic ones; it also may not be realistic to have access to a whole (unlabelled) test dataset at once (as opposed to, e.g., a few-shot setting).

Author response:
Yes, access to the full test dataset may not be practical. Based on your suggestions, we added how the performance of our method changes when different amounts (0.1%, 1%, 5%, 100%) of the test dataset can be accessed. As shown in the table below, we observe that our method also has an improvement of about 5% when only 0.1% of the test data is accessible. When 5% of the test data is accessible, our method achieves comparable accuracy to when the whole test data is available.
| Method | Avg ACC (\%) | | Method | Avg ACC (\%) | 
| :------ | :------: |   :------: |   :------ | :------: |  
| Task Arithmetic (0%)  | 69.1 |    |  TIES-Merging (0%)  |  72.4 | 
| AdaMerging (0.1%)   | 74.0 |  |  AdaMerging++ (0.1%) | 77.9  | 
| AdaMerging (1%)   | 77.5 |  |  AdaMerging++ (1%) | 80.6  | 
| AdaMerging (5%)   | 80.1 |  |  AdaMerging++ (5%) |   81.0 | 
| AdaMerging (100%) | 80.1 |  |  AdaMerging++ (100%) |  81.1  |

Reviewer comment (Experiments):
The experimental evaluation only considers **ViT-based backbones on small/medium computer vision benchmarks**. This is a bit different from the introduction, which focuses on readily available pretrained foundation models for which the original training data may be unknown. Furthermore, this also raises the question of MTL baselines: there is a very large literature on automatically weighing tasks losses/gradients for computer vision tasks, which may be stronger baselines than `traditional MTL`, and increase the performance gap even more (e.g., GradNorm, PCGrad, GradDrop, etc.).

Author response:
In the experiment, we strictly followed the ViT architecture used by Task Arithmetic and Ties-Merging. Task Arithmetic, Ties-Merging and our AdaMerging did not require the training data of the original task, and only needed the provided trained models to construct the task vectors. Each task vector can be understood as the ability to complete a specific task. Furthermore, these task vectors are merged into the pre-trained model through merging coefficients to achieve MTL. Of course, our approach is model-agnostic and can be used with foundation models of any architecture.

In addition, our method is compared fairly with baselines that merge models without original training data. As for more advanced MTL baselines such as GradNorm, PCGrad, GradDrop, etc., they are not the main baselines of the model merging community at this stage. Because there is still a gap between the current model merging and the traditional MTL baseline, these advanced MTL baselines need to be further considered only when this gap is further narrowed or even surpassed in the future. Of course, model merging technology has many advantages compared to traditional MTL methods. First, it does not require raw data for each task, significantly reducing the need for raw data collection and protecting data privacy. In addition, model merging is very efficient (especially for foundation models) compared with traditional MTL to train models from scratch, and it can usually be completed in minutes to tens of minutes.

Reviewer comment (Theory):
In the motivation, the authors need to explain the intuitive motivation for entropy minimization as a proxy objective for loss.

Author response:
The intuitive motivation for entropy minimization is to reduce model uncertainty. The lower the Shannon entropy, the more concentrated the probability distribution is and the more confident the model's predictions are. By minimizing entropy, we want the model to produce a more deterministic output when faced with a given input, resulting in a more robust and accurate model.

Reviewer comment (Evaluation):
In the experimental analysis, the author needed to explain why AdaMerging has better generalization and robustness.

Author response:
Our AdaMerging allows us to adapt to unlabeled test data of unseen tasks or unlabeled corrupted data in an unsupervised way when training model merging coefficients, thereby obtaining model merging coefficients that are most suitable for these data. Therefore, the AdaMerging merged model better bridges the gap between training and testing data distribution shifts. However, existing methods ignore this point. Therefore, our method will have better generalization and robustness compared with existing task vector-based model merging methods.

Reviewer comment (Reproducibility):
Is AdaMering in Tables 2, 3, and 4 a Task-wise or a Layer-wise version?

Author response:
AdaMerging in Tables 2, 3, and 4 are all Layer-wise versions. As we introduced in the experimental settings in Section 4.1, all versions not specifically emphasized are Layer-wise.

Reviewer comment (Evaluation):
Why is the model merging performance closer to traditional MTL in a larger architecture?

Author response:
On the one hand, larger architectures usually have more parameters and more powerful representation learning capabilities, and larger architectures can better learn shared feature representations. These shared underlying features benefit knowledge transfer between different tasks, so the merged model performs better and is closer to traditional MTL. On the other hand, larger architectures have more layers, and our Layer-wise AdaMerging can have more learnable merging coefficients. For example, under ViT-B/32 and ViT-L/14, the number of coefficients for Layer-wise AdaMerging is 1,248 and 2,408, respectively. As a result, we can balance the task vectors more flexibly, leading to better performance.