Paper: Parameter-Efficient Multi-Task Model Fusion with Partial Linearization (ICLR-2024)

Reviewer comment (Experiments):
Can the proposed methods demonstrate effectiveness on SOTA models?

Author response:
**Effectiveness on SOTA models.** We are confident that our partial linearization method can be applied to SOTA models to enhance their multi-task learning capabilities. The positive results we observed in CLIP and Flan-T5 models, which are themselves highly competitive, give us a strong indication that our method would be beneficial when applied to other advanced models. We will take your question as a valuable suggestion for future work and aim to include such results in subsequent publications. 

To demonstrate this, we conducted a small-scale segmentation experiment. We utilized the SAM model ('sam_vit_b_01ec64.pth') to test the segmentation tasks with both LoRA and L-LoRA on the Pascal VOC 2012 and NYUD v2 datasets. We set the batch size to 4 and the LoRA r hyperparameter to 32. *Due to time constraints and resource limitations, the models were still far from convergence at this point, but the results were sufficient to demonstrate the effectiveness of L-LoRA (The code for this experiment will be put on GitHub together with a clean version of the original code).* We report the performance of the pre-trained models, the fine-tuned models, and the fused models obtained through weight averaging on the validation sets of Pascal VOC and NYUD v2 in terms of mIoU in the table below.

Table: reported mIoU of LoRA and L-LoRA.

| Model                             | fine-tuning method | VOC 2012 | NYUD v2 | Mean mIoU |
|-----------------------------------|--------------------|----------|---------|-----------|
| Pretrained Model                  |                    | 0.65     | 0.75    | 0.70      |
| fine-tuned (far from converge)    | LoRA               | **4.51**     | **2.42**    | **3.47**      |
|                                   | L-LoRA             | 4.09     | 1.93    | 3.01      |
| two-model fusion (simple average) | LoRA               | 1.92(43%)     | 1.15(47%)    | 1.53      |
|                                   | L-LoRA             | **4.05(99%)**     | **1.51(78%)**    | **2.78**      |

Notably, the two-model fusion using simple averaging yields different results for each method. The LoRA fusion only retains 43% and 47% of the pretrained model's performance on VOC 2012 and NYUD v2, respectively. In contrast, the L-LoRA fusion maintains a remarkable 99% and 78% of the fine-tuned LoRA's performance on the respective datasets, resulting in a mean mIoU of 2.78.

Reviewer comment (Theory):
The definition of the task vector of LoRA in this paper seems to be unreasonable.

Author response:
**W2 & Q3: The definition of the task vector of LoRA in this paper seems to be unreasonable. 1. In [2], the task vector of full fine-tuning is defined as $\nu_i=\theta_i - \theta_0$, which means the parameter change of $\theta_0$. Thus, why the task vector of LoRA in this paper is defined as the change of LoRA parameters $\phi_i-\phi_0$ rather than the change of $\theta_0$ as in [3], i.e., $A_i B_i$, where $\phi_i=[A_i, B_i]$ is the LoRA parameters.**

**Task vector definition.** Our definition of the task vector as $\phi_i - \phi_0$ (the trainable parameter space, LoRA parameter space) rather than the change of $\theta$ (the original parameter space) as done in [3, 4] is a deliberate and calculated decision that aligns with the nature of our fine-tuning approach.

1. When partially linearizing a LoRA model, we concentrate on the dynamics of the trainable parameters and compute the Jacobian-verctor product for all trainable parameters, specifically the LoRA parameters denoted by $\phi$. The original model parameters, denoted by  $\theta_0$, are treated as constants during fine-tuning—akin to model buffers that provide a stable foundation upon which the adaptable components of the model, the LoRA parameters, can exert their influence.
2. From a mathematical standpoint, we perform a first-order Taylor expansion on the trainable parameters, which does not involve the parameter merging operations of the LoRA module.

   $$f^{\text{lin}}\_{\theta_0}(x;\phi)=f\_{\theta_0}(x; \phi_0) + \nabla_\phi f\_{\theta_0}(x;\phi_0)^\top (\phi -\phi_0)$$

   It's important to note that this process of Taylor expansion is applied exclusively to the parameters that are designated as trainable—those that we actively adjust during the fine-tuning phase.
3. Although our experiments have solely utilized LoRA fine-tuning, such a definition is helpful for generalize our method to other parameter-efficient fine-tuning techniques, such as adapter-tuning.

[2] Editing Models with Task Arithmetic. ICLR, 2023.  
[3] Effective and Parameter-Efficient Reusing Fine-Tuned Models. arXiv:2310.01886.  
[4] Composing Parameter-Efficient Modules with Arithmetic Operations. http://arxiv.org/abs/2306.14870.

Reviewer comment (Evaluation):
The L-LoRA has a large performance drop on single-task fine-tuning (Figure 7) and a slight increase on the merged case (Table 1, especially in the NLP domain) compared to LoRA. Thus, it is unclear what are the advantages of L-LoRA compared to LoRA.

Author response:
**W3 & Q6: The L-LoRA has a large performance drop on single-task fine-tuning (Figure 7) and a slight increase on the merged case (Table 1, especially in the NLP domain) compared to LoRA. Thus, it is unclear what are the advantages of L-LoRA compared to LoRA. Why use the proposed L-LoRA rather than the existing LoRA?**

**Advantages of L-LoRA over LoRA.** Despite the performance drop in single-task fine-tuning observed in Figure 7, L-LoRA demonstrates improved performance in the multi-task fusion setting. The increase in performance in the merged case, especially in the NLP domain, suggests that L-LoRA may offer advantages in scenarios where multi-task performance is prioritized.

The multi-task domain is particularly challenging due to the need to balance and optimize across various objectives. The performance gains achieved by L-LoRA in this setting suggest that our method can effectively navigate the trade-offs between tasks, finding what could be described as a Pareto optimal point—a solution where no task's performance can be improved without compromising another's. The advantages of L-LoRA become evident when considering the broader context of its application. The advantages of L-LoRA become evident when considering the broader context of its application. While single-task performance is important, the ability to excel in multi-task environments is increasingly becoming a benchmark for success in advanced machine learning systems. Our method contributes to this goal by enhancing multi-task model fusion capabilities, which is a significant step forward in developing more capable and efficient AI systems.

By the way, based on the small-scale model fusion experiments we conducted in response to Q5, we observe that superior prompt templates can significantly narrow the performance gap between L-LoRA and LoRA. But searching for better prompt templates is not the focus of our research in this paper.

Reviewer comment (Novelty):
The novelty of this paper is limited. [1] studies the linearized full fine-tuning while this paper simply replaces full fine-tuning with LoRA. Although the authors emphasize the proposed method is more efficient than [1], it is obvious since LoRA is more efficient than fully fine-tuning.

Author response:
Thank you for your thoughtful review and constructive feedback on our paper. As per your suggestions, our paper has undergone significant improvement!

**Q1: Novelty: If my understanding is correct, this paper uses the method in [1] in LoRA fine-tuning instead of the full fine-tuning in [1]. Are there any technical challenges when adapting the method in [1] to LoRA?**

We selectively identify and apply linearization to specific parameters, enhancing efficiency and enabling theoretical analysis. Besides, adopting partial linearization resulting in significantly improved speed and reduced GPU memory usage than full linearization.

1. **Identify linearization parts**. In LoRA, only a sparse, low-rank structure of the weight matrices is updated. We need to identify which aspects of the model’s parameters will benefit from linearization while full fine-tuning methods have more freedom as all parameters are fine-tuned. In our experiments, LoRA adapt to the attention weights ($W=W_0+AB$). Our linearization did not extend to the entire attention weights $W$ but was specifically applied to the LoRA parameters $\phi=[A,B]$. This not only makes linearization more efficient but also facilitates theoretical analysis. We have the following:
   $$\phi(\Delta t)-\phi(0) \propto \mathbb{E}\_{(x_{\tau_i},y_{\tau_i})\sim D\_{\tau_i}} [
\boldsymbol{K}(x, x_{\tau_i}; \phi(0))
\nabla_f \mathcal{L}\_{\text{CE}}(f\_{\theta_0}(x\_{\tau_i}; \phi(0)), y\_{\tau_i})].$$
	This formula underscores the direct correspondence between the changes in LoRA weights $[A,B]$ of a linearized model and the gradient of the loss. It hints at the underlying mechanisms of weight disentanglement, revealing that the adjustment in LoRA parameters is essentially directed by the aggregated gradient information from the task-specific data. This understanding is crucial as it provides insight into how a model's weights evolve in response to the learning process.

1. **Computation overhead**. Implementing linearization across the entire model (linearized full fine-tuning) significantly increases memory usage and inference cost. Through partially linearization, we have significantly reduced the overhead associated with linearization. The training and inference speed of partial linearization is 2-3 times faster than that of full linearization, with GPU memory usage only being one-third to one-half of the latter.

Table 1: Summary of training and inference cost.

| **Scenario**                                          | **Training (Adam)** |      | **Inference**           |      |
| ------------------------------------------------------ | ------------------- | ---- | ------------------------ | ---- |
|                                                        | **Time**            | **VRAM** | **Time**           | **VRAM** |
| *CLIP-ViT-B-32 (batch size=64)*                        |                     |      |                          |      |
| full fine-tuning   | 8.25 it/s           | $\approx 4.2$GB | 13.28 it/s           | $\approx 0.9$GB |
| full linearization | 4.94 it/s           | $\approx 6.3$GB | 8.52 it/s            | $\approx 2.0$GB |
| LoRA (r=16)        | 8.43 it/s           | $\approx 2.6$GB | 13.01 it/s           | $\approx 0.9$GB |
| L-LoRA (r=16)      | 7.55 it/s           | $\approx 2.7$GB | 10.65 it/s           | $\approx 1.0$GB |
| *CLIP-ViT-B-16 (batch size=64)*                        |                     |      |                          |      |
| full fine-tuning                                       | 2.97 it/s           | $\approx 12.4$GB | 8.92 it/s            | $\approx 1.6$GB |
| full linearization                                     | 1.49 it/s           | $\approx 20.1$GB | 2.50 it/s            | $\approx 3.7$GB |
| LoRA (r=16)                                            | 3.83 it/s           | $\approx 9.1$GB  | 8.60 it/s            | $\approx 1.6$GB |
| L-LoRA (r=16)                                          | 3.24 it/s           | $\approx 9.2$GB  | 7.23 it/s            | $\approx 1.8$GB |
| *CLIP-ViT-L-14 (batch size=16)*                        |                     |      |                          |      |
| full fine-tuning                                       | 2.55 it/s           | $\approx 13.9$GB | 8.23 it/s            | $\approx 1.8$GB |
| full linearization                                     | 1.32 it/s           | $\approx 21.8$GB | 2.26 it/s            | $\approx 4.8$GB |
| LoRA (r=16)                                            | 3.49 it/s           | $\approx 9.1$GB  | 7.99 it/s            | $\approx 1.8$GB |
| L-LoRA (r=16)                                          | 2.99 it/s           | $\approx 9.3$GB  | 6.55 it/s            | $\approx 1.9$GB |

Reviewer comment (Theory):
In [2], the task vector of full fine-tuning is defined as $\nu_i=\theta_i-\theta_0$, which means the parameter change of $\theta_0$. Thus, why the task vector of LoRA in this paper is defined as the change of LoRA parameters $\phi_i-\phi_0$ rather than the change of $\theta_0$ as in [3], i.e., $A_iB_i$, where $\phi_i=[A_i, B_i]$ is the LoRA parameters.

Author response:
**W2 & Q3: The definition of the task vector of LoRA in this paper seems to be unreasonable. 1. In [2], the task vector of full fine-tuning is defined as $\nu_i=\theta_i - \theta_0$, which means the parameter change of $\theta_0$. Thus, why the task vector of LoRA in this paper is defined as the change of LoRA parameters $\phi_i-\phi_0$ rather than the change of $\theta_0$ as in [3], i.e., $A_i B_i$, where $\phi_i=[A_i, B_i]$ is the LoRA parameters.**

**Task vector definition.** Our definition of the task vector as $\phi_i - \phi_0$ (the trainable parameter space, LoRA parameter space) rather than the change of $\theta$ (the original parameter space) as done in [3, 4] is a deliberate and calculated decision that aligns with the nature of our fine-tuning approach.

1. When partially linearizing a LoRA model, we concentrate on the dynamics of the trainable parameters and compute the Jacobian-verctor product for all trainable parameters, specifically the LoRA parameters denoted by $\phi$. The original model parameters, denoted by  $\theta_0$, are treated as constants during fine-tuning—akin to model buffers that provide a stable foundation upon which the adaptable components of the model, the LoRA parameters, can exert their influence.
2. From a mathematical standpoint, we perform a first-order Taylor expansion on the trainable parameters, which does not involve the parameter merging operations of the LoRA module.

   $$f^{\text{lin}}\_{\theta_0}(x;\phi)=f\_{\theta_0}(x; \phi_0) + \nabla_\phi f\_{\theta_0}(x;\phi_0)^\top (\phi -\phi_0)$$

   It's important to note that this process of Taylor expansion is applied exclusively to the parameters that are designated as trainable—those that we actively adjust during the fine-tuning phase.
3. Although our experiments have solely utilized LoRA fine-tuning, such a definition is helpful for generalize our method to other parameter-efficient fine-tuning techniques, such as adapter-tuning.

[2] Editing Models with Task Arithmetic. ICLR, 2023.  
[3] Effective and Parameter-Efficient Reusing Fine-Tuned Models. arXiv:2310.01886.  
[4] Composing Parameter-Efficient Modules with Arithmetic Operations. http://arxiv.org/abs/2306.14870.

Reviewer comment (Evaluation):
Table 1 only shows the average normalized scores over multiple datasets, so how about the performance of each dataset? Could you provide it in the Appendix?

Author response:
**Q4: Table 1 only shows the average normalized scores over multiple datasets, so how about the performance of each dataset? Could you provide it in the Appendix?**

**Performance on Each dataset.** We agree that a detailed breakdown of performance across individual datasets would be beneficial. We will include these results in the Appendix to provide a more comprehensive view of our method's performance.

Table: normalized scores of merged CLIP-ViT-B/16 across individual datasets.

|     Method      | Fine-tuning Mode |   Cars   |   DTD    | EuroSAT  |  GTSRB   | RESISC45 |  SUN397  |   SVHN   |   Mean   |
| :-------------: | :--------------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: |
| Simple Average  | full fine-tuning |   0.77   |   0.62   | **0.76** |   0.49   |   0.71   |   0.87   | **0.56** | **0.68** |
|                 |       LoRA       |   0.81   |   0.69   |   0.39   |   0.40   |   0.62   |   0.93   |   0.16   |   0.57   |
|                 |      L-LoRA      | **0**.87 | **0.86** |   0.41   | **0.58** | **0.74** | **0.97** |   0.19   |   0.66   |
| Task Arithmetic | full fine-tuning |   0.80   |   0.63   | **0.87** |   0.73   |   0.78   |   0.86   | **0.85** |   0.79   |
|                 |       LoRA       |   0.86   |   0.72   |   0.51   |   0.50   |   0.73   | **0.91** |   0.48   |   0.67   |
|                 |      L-LoRA      | **0.95** | **0.91** |   0.65   | **0.74** | **0.86** |   0.96   |   0.78   | **0.84** |
|  Ties-Merging   | full fine-tuning |   0.78   |   0.65   | **0.92** |   0.69   | **0.83** |   0.88   | **0.93** | **0.81** |
|                 |       LoRA       |   0.81   |   0.74   |   0.54   |   0.50   |   0.71   |   0.92   |   0.40   |   0.66   |
|                 |      L-LoRA      | **0.89** | **0.91** |   0.60   |   0.69   |   0.82   | **0.97** |   0.51   |   0.77   |
|     LoraHub     | full fine-tuning |   0.93   | **0.90** |   0.32   |   0.30   |   0.47   |   0.77   |   0.13   |   0.55   |
|                 |       LoRA       |   0.83   |   0.68   |   0.38   |   0.76   |   0.70   |   0.86   |   0.64   |   0.69   |
|                 |      L-LoRA      | **0.94** |   0.88   | **0.73** | **0.85** | **0.85** | **0.91** | **0.80** | **0.85** |

Reviewer comment (Evaluation):
It seems the proposed L-LoRA method does not perform well on the NLP domain, according to the results in Table 1 and the similarity heatmap in Figure 8.

Author response:
**Q5: It seems the proposed L-LoRA method does not perform well on the NLP domain, according to the results in Table 1 and the similarity heatmap in Figure 8.**

**Performance in the NLP Domain.** The observed underperformance in the NLP domain could be due to our initial choice of prompt templates not being optimal for fine-tuning the language models—a slight oversight in our experimental design. Nevertheless, we are confidence that this does not skew the comparison of multi-task model fusion performance between various methods, given that the same prompt template was employed throughout the experiments. 

To enhance the model's output, we should provide more explicit cues within the input. For instance, as suggested in Appendix E.1.6, a more instructive prompt would be "{sentence}. Is this sentence 'positive' or 'negative'?" This type of prompt would clearly signal to the language model that the response should be either 'positive' or 'negative', potentially resulting in increased accuracy.

To demonstrate this, we carried out a series of small-scale multi-task model fusion experiments focusing on three downstream tasks: CoLA, MNLI, and RTE. For all fine-tuning processes, we utilized the Adam optimizer, standardizing the batch size to 16 and setting the number of fine-tuning steps to 2000 for all models. The learning rate was configured to 1e-5 for full fine-tuning, while for both LoRA and L-LoRA, we increased the learning rate to 4e-5. Table 1 presents a comparison between the prompt templates originally used in our study and an improved version of these templates. Table 2 details the individual performance of fine-tuned models employing different prompt templates. In Table 3, we assess the 'exact-match' accuracy on validation datasets through a simple model averaging technique for three-model fusion. Our findings indicate that a better and more predictable prompt yields superior results, with L-LoRA outperforming LoRA in terms of both absolute scores and normalized scores.

Table 1: Comparison of the templates used in our paper and a better version of prompt templates.

| Task | Prompt Templates        | input text                                                                                                                                                                      | target text                                                                 |
| ---- | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- |
| CoLA | Templates in Appendix E | "cola sentence: {sentence}"                                                                                                                                                     | "unacceptable" if label=0 else "acceptable"                                 |
|      | Better Prompt Templates | "Indicate if the following sentence is grammatically correct or not:   \"{sentence}\". **Answere 'acceptable' or 'unacceptable'.**"                                             | "unacceptable" if label=0 else "acceptable"                                 |
| MNLI | Templates in Appendix E | "mnli hypothesis: {hypothesis} premise: {premise}"                                                                                                                              | "entailment", "neutral", "contradiction" if   label is 0, 1,2, respectively |
|      | Better Prompt Templates | "Does the premise: '{premise}' logically imply, contradict, or is   neutral to the hypothesis: '{hypothesis}'? **Answere with 'entailment',   'contradiction', or 'neutral'.**" | "entailment", "neutral", "contradiction" if   label is 0, 1,2, respectively |
| RTE  | Templates in Appendix E | "rte sentence1: {sentence1} sentence2: {sentence2}"                                                                                                                             | "entailment" if label=0 else "not_entailment"                               |
|      | Better Prompt Templates | "Does the text: '{sentence1}' entail that '{sentence2}' is true? **Provide   'yes' or 'no'.**"                                                                                  | "yes" if label=0, else "no"                                                 |

Reviewer comment (Presentation):
The caption of Figure 3(c): not linearized?

Author response:
**Minor concern 1: caption of figure 3(c)**

This is a typo, thanks for pointing it out and we will correct it. "(c) Linearized..." should be "(d) Linearized..."

Reviewer comment (Presentation):
A key result of the paper is the result from Appendix A which allows the authors to hypothesize that partial linearization of a subset of module parameters (here LoRA) can improve weight disentanglement.

Author response:
**Key result placement.** We acknowledge your suggestion regarding the placement of the key result from Appendix A. We will provide a more detailed explanation in section 4.

**Connection between linearization and weight disentanglement.** We agree that the derivation is central to our hypothesis and warrants more visibility. We will clarify this explanation in the manuscript to make the connection more explicit.

The derivation in Appendix A shows that the change of the linearized model output is determined by the gradient of the loss. The connection to central hypothesis is that this property of linearized models enables a more predictable and controlled update during fine-tuning. By fine-tuning in the linearized space, updates of trainable parameters are perpendicular to the loss contours, this orthogonality promotes that the updates contribute to the separation of the weight representations corresponding to distinct tasks, a concept known as weight disentanglement.

Mathematically, the change of linearized model output for a input sample $x$ around the initialization $\phi(0)$ can be written as:

$$
f^{\text{lin}}_{\theta_0}(x; \theta(\Delta t))-f\_{\theta_0}(x;\theta(0))
=\nabla\_{\theta_0}f(x;\theta_0)^\top (\theta(\Delta t)-\theta(0))
=-\eta \mathbb{E}\_{(x\_{\tau_i},y\_{\tau_i})\sim D\_{\tau_i}} [
\boldsymbol{K}(x, x\_{\tau_i}; \theta(0))
\nabla_f \mathcal{L}\_{\text{CE}}(f\_{\theta_0}(x\_{\tau_i}; \theta(0)), y\_{\tau_i})]
$$

Thus we have the following formula that captures the relationship between task-specific gradient of the loss and weight disentanglement for linearized model

$$\theta(\Delta t)-\theta(0) \propto \mathbb{E}\_{(x\_{\tau_i},y\_{\tau_i})\sim D\_{\tau_i}} 
[\boldsymbol{K}(x, x\_{\tau_i}; \theta(0))
\nabla_f \mathcal{L}\_{\text{CE}}(f\_{\theta_0}(x_{\tau_i}; \theta(0)), y_{\tau_i})]$$

Reviewer comment (Presentation):
This result from Appendix A needs to be in the main body of the text and properly explained. Without it, it is difficult to understand exactly why the authors claim this.

Author response:
**Key result placement.** We acknowledge your suggestion regarding the placement of the key result from Appendix A. We will provide a more detailed explanation in section 4.

Reviewer comment (Theory):
Having checked the derivation in Appendix A, the authors show that the model output of a linearised model is only determined by the gradient of the loss in the non-linearised model on task $t_i$. How exactly then does this allow the authors to make the central hypothesis, which guides the presented method?

Author response:
**Connection between linearization and weight disentanglement.** We agree that the derivation is central to our hypothesis and warrants more visibility. We will clarify this explanation in the manuscript to make the connection more explicit.

The derivation in Appendix A shows that the change of the linearized model output is determined by the gradient of the loss. The connection to central hypothesis is that this property of linearized models enables a more predictable and controlled update during fine-tuning. By fine-tuning in the linearized space, updates of trainable parameters are perpendicular to the loss contours, this orthogonality promotes that the updates contribute to the separation of the weight representations corresponding to distinct tasks, a concept known as weight disentanglement.

Mathematically, the change of linearized model output for a input sample $x$ around the initialization $\phi(0)$ can be written as:

$$
f^{\text{lin}}_{\theta_0}(x; \theta(\Delta t))-f\_{\theta_0}(x;\theta(0))
=\nabla\_{\theta_0}f(x;\theta_0)^\top (\theta(\Delta t)-\theta(0))
=-\eta \mathbb{E}\_{(x\_{\tau_i},y\_{\tau_i})\sim D\_{\tau_i}} [
\boldsymbol{K}(x, x\_{\tau_i}; \theta(0))
\nabla_f \mathcal{L}\_{\text{CE}}(f\_{\theta_0}(x\_{\tau_i}; \theta(0)), y\_{\tau_i})]
$$

Thus we have the following formula that captures the relationship between task-specific gradient of the loss and weight disentanglement for linearized model

$$\theta(\Delta t)-\theta(0) \propto \mathbb{E}\_{(x\_{\tau_i},y\_{\tau_i})\sim D\_{\tau_i}} 
[\boldsymbol{K}(x, x\_{\tau_i}; \theta(0))
\nabla_f \mathcal{L}\_{\text{CE}}(f\_{\theta_0}(x_{\tau_i}; \theta(0)), y_{\tau_i})]$$

Reviewer comment (Evaluation):
The results on the NLP task (Flan T-5-Base) need to be better explained. The method (L-LoRA) not only performs worse than full fine-tuning (so does LoRA fine-tuning) but also worse on average than LoRA. Why is this? Is the presented model only applicable to vision tasks?

Author response:
**Performance on NLP Tasks.** Our selection of prompts was rather naive, following the approach used by (Raffel et al. in 2020), which was a minor oversight in our experimental design. Nonetheless, we are confident that this does not compromise the comparative analysis of multi-task model fusion performance across different methods, as the same prompt template was consistently applied in all experiments. We find that with fine-tuning of the prompt selection and other task-specific adjustments, L-LoRA can be made effective in the NLP domain. 

To demonstrate this, we carried out a series of small-scale multi-task model fusion experiments focusing on three downstream tasks: CoLA, MNLI, and RTE. For all fine-tuning processes, we utilized the Adam optimizer, standardizing the batch size to 16 and setting the number of fine-tuning steps to 2000 for all models. The learning rate was configured to 1e-5 for full fine-tuning, while for both LoRA and L-LoRA, we increased the learning rate to 4e-5. Table 1 presents a comparison between the prompt templates originally used in our study and an improved version of these templates. Table 2 details the individual performance of fine-tuned models employing different prompt templates. In Table 3, we assess the 'exact-match' accuracy on validation datasets through a simple model averaging technique for three-model fusion. Our findings indicate that a better and more predictable prompt yields superior results, with L-LoRA outperforming LoRA in terms of both absolute scores and normalized scores.

Table 1: Comparison of the templates used in our paper and a better version of prompt templates.

| Task | Prompt Templates | input text | target text | 
| ---- | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- |
| CoLA | Templates in Appendix E | "cola sentence: {sentence}"                                                                                                                                                     | "unacceptable" if label=0 else "acceptable" |
| | Better Prompt Templates | "Indicate if the following sentence is grammatically correct or not:   \"{sentence}\". **Answere 'acceptable' or 'unacceptable'.**" | "unacceptable" if label=0 else "acceptable" |
| MNLI | Templates in Appendix E | "mnli hypothesis: {hypothesis} premise: {premise}"                                                                                                                              | "entailment", "neutral", "contradiction" if   label is 0, 1,2, respectively |
| | Better Prompt Templates | "Does the premise: '{premise}' logically imply, contradict, or is   neutral to the hypothesis: '{hypothesis}'? **Answere with 'entailment',   'contradiction', or 'neutral'.**" | "entailment", "neutral", "contradiction" if   label is 0, 1,2, respectively |
| RTE  | Templates in Appendix E | "rte sentence1: {sentence1} sentence2: {sentence2}"                                                                                                                             | "entailment" if label=0 else "not_entailment" |
| | Better Prompt Templates | "Does the text: '{sentence1}' entail that '{sentence2}' is true? **Provide   'yes' or 'no'.**"                                                                                  | "yes" if label=0, else "no" |

Reviewer comment (Theory):
Section 4.2 introduces parameter scaling laws for weight disentanglement. However, what does this have to do with the method or key results? The scaling laws suggest over-parameterisation is necessary for weight disentanglement. I struggle to see the connection between this and the need to partially linearize a model for fine-tuning.

Author response:
**Parameter Scaling Law.** The parameter scaling law is an empirical phenomenon suggesting that models with a larger number of trainable parameters tend to exhibit better performance in multi-task model fusion. Consequently, a logical deduction would be that parameter-efficient fine-tuning could potentially diminish the performance of multi-task model fusion. Indeed, our experimental results corroborate this (as illustrated in Figure 5). 

However, our method stands out in that the performance of L-LoRA's multi-task model fusion remains on par with that of full fine-tuning, despite its parameter efficiency. This indicates that partial linearization, with a very limited increase in inference cost overhead, effectively bridges the gap typically observed between parameter efficiency and multi-task fusion performance.

Reviewer comment (Theory):
A remark in the manuscript is made at the bottom of page 8 that "...higher cosine similarity...implies greater redundancy and overlap...This results in more destructive task interference with naïve merging". I don't follow this as it seems to be opposite to most work in multi-task training. In that setting, naïve training would favour similar tasks as it would not require methods to mitigate task interference (see GradNorm method). Why is this then opposite in the context of this paper?

Author response:
**cos similarity and task interference.** You've raised an insightful point. When we discuss the impact of high cosine similarity on model performance, it's important to distinguish between the contexts of multi-task training and model fusion. *In the former, tasks with similar characteristics often benefit from the ability to share representations, as this commonality can facilitate learning by leveraging the same features or patterns.*

However, in the context of model fusion, which is the focus of our paper, the dynamics differ from simultaneous multi-task training. *Here, we are dealing with models that have been independently trained on separate tasks and are subsequently being merged.* In such a post-hoc fusion, a high cosine similarity between task weights is indicative of overlap in the information encoded by the models, thus be problematic. While this might suggest a common underlying structure, it also raises the risk that merging these similar weights could amplify shared errors or redundancies, rather than complementing each model's strengths. It can signal redundancy, where multiple models encode similar information without adding new insights, or even conflict, where the shared information is not entirely compatible across tasks.

In summary, while shared representations are advantageous in simultaneous multi-task learning, our work suggests that for the fusion of independently trained models, distinctiveness in task representations—achieved through orthogonality—is key to preserving the integrity and performance of the fused model. This distinction is crucial for understanding the different implications of cosine similarity in multi-task training versus model fusion contexts.

Reviewer comment (Experiments):
In vision domain, only the high-level vision task like image classification tasks evaluated, the mid-level and low level task are missing, for example, semantic segmentation.

Author response:
**Extending evaluation on mid-level and low-level task (Evaluating on image segmentation models).** We acknowledge the limitation you pointed out regarding the scope of our evaluations in the vision domain. Our decision to focus on high-level vision tasks like image classification was driven by the desire to establish a strong foundational understanding of multi-task model fusion in a well-studied context, and we agree that extending the evaluation to lower-level vision tasks would provide a more comprehensive understanding.

Evaluating our method on mid-level and low-level task, such as image segmentation, is indeed possible. This is due to the task-agnostic nature of our approach, which does not impose specific requirements on the loss function. The only prerequisite for our method is that the tuned parameter-efficient modules must be capable of computing the Jacobian-vector product, thereby enabling it to be efficiently linearized. 

To demonstrate this, we conducted a small-scale experiment. We utilized the SAM model ('sam_vit_b_01ec64.pth') to test the segmentation tasks with both LoRA and L-LoRA on the Pascal VOC 2012 and NYUD v2 datasets. We set the batch size to 4 and the LoRA r hyperparameter to 32. *Due to time constraints and resource limitations, the models were still far from convergence at this point, but the results were sufficient to demonstrate the effectiveness of L-LoRA (The code for this experiment will be put on GitHub together with a clean version of the original code).* We report the performance of the pre-trained models, the fine-tuned models, and the fused models obtained through weight averaging on the validation sets of Pascal VOC and NYUD v2 in terms of mIoU in the table below.

Table: reported mIoU of LoRA and L-LoRA.

| Model                             | fine-tuning method | VOC 2012 | NYUD v2 | Mean mIoU |
|-----------------------------------|--------------------|----------|---------|-----------|
| Pretrained Model                  |                    | 0.65     | 0.75    | 0.70      |
| fine-tuned (far from converge)    | LoRA               | **4.51**     | **2.42**    | **3.47**      |
|                                   | L-LoRA             | 4.09     | 1.93    | 3.01      |
| two-model fusion (simple average) | LoRA               | 1.92(43%)     | 1.15(47%)    | 1.53      |
|                                   | L-LoRA             | **4.05(99%)**     | **1.51(78%)**    | **2.78**      |

Notably, the two-model fusion using simple averaging yields different results for each method. The LoRA fusion only retains 43% and 47% of the pretrained model's performance on VOC 2012 and NYUD v2, respectively. In contrast, the L-LoRA fusion maintains a remarkable 99% and 78% of the fine-tuned LoRA's performance on the respective datasets, resulting in a mean mIoU of 2.78.

Reviewer comment (Presentation):
From table 1, seems the L-LoRA method are outperforming full-finetuning under some model fusion settings, do we have some possible illustrations?

Author response:
**Results from Table 1.** Indeed, as observed in Table 1, our L-LoRA (Linearized Low-Rank Adaptation) appears to outperform full fine-tuning in terms of normalized score under certain fusion settings. 

On one hand, the partially linearization likely leads to better weight disentanglement*, meaning that the task-specific knowledge encoded in the weights is more separable. 

On the other hand, we should consider *the data requirement of full fine-tuning versus parameter-efficient fine-tuning. As for fine-tuning, the amount of data in downstream tasks is sometimes not enough. However, full fine-tuning generally demands extensive data to optimize the large number of parameters effectively. In scenarios where abundant data is available, full fine-tuning is expected to excel as it can leverage the comprehensive dataset to refine a large amount of model's weights for each specific task. However, in situations with limited data, full fine-tuning may not perform optimally due to overfitting concerns or insufficient information for the model.

L-LoRA, on the other hand, with its focus on updating only a subset of the model's parameters, is less dependent on large amounts of data. This characteristic could explain why L-LoRA sometimes outperforms full fine-tuning in multi-task fusion settings, especially when data availability of downstream dataset is a limiting factor.

Reviewer comment (Experiments):
Is it possible to evaluate L-LoRA on image segmentation foundational models like SAM?

Author response:
**Extending evaluation on mid-level and low-level task (Evaluating on image segmentation models).** We acknowledge the limitation you pointed out regarding the scope of our evaluations in the vision domain. Our decision to focus on high-level vision tasks like image classification was driven by the desire to establish a strong foundational understanding of multi-task model fusion in a well-studied context, and we agree that extending the evaluation to lower-level vision tasks would provide a more comprehensive understanding.

Evaluating our method on mid-level and low-level task, such as image segmentation, is indeed possible. This is due to the task-agnostic nature of our approach, which does not impose specific requirements on the loss function. The only prerequisite for our method is that the tuned parameter-efficient modules must be capable of computing the Jacobian-vector product, thereby enabling it to be efficiently linearized. 

To demonstrate this, we conducted a small-scale experiment. We utilized the SAM model ('sam_vit_b_01ec64.pth') to test the segmentation tasks with both LoRA and L-LoRA on the Pascal VOC 2012 and NYUD v2 datasets. We set the batch size to 4 and the LoRA r hyperparameter to 32. *Due to time constraints and resource limitations, the models were still far from convergence at this point, but the results were sufficient to demonstrate the effectiveness of L-LoRA (The code for this experiment will be put on GitHub together with a clean version of the original code).* We report the performance of the pre-trained models, the fine-tuned models, and the fused models obtained through weight averaging on the validation sets of Pascal VOC and NYUD v2 in terms of mIoU in the table below.

Table: reported mIoU of LoRA and L-LoRA.

| Model                             | fine-tuning method | VOC 2012 | NYUD v2 | Mean mIoU |
|-----------------------------------|--------------------|----------|---------|-----------|
| Pretrained Model                  |                    | 0.65     | 0.75    | 0.70      |
| fine-tuned (far from converge)    | LoRA               | **4.51**     | **2.42**    | **3.47**      |
|                                   | L-LoRA             | 4.09     | 1.93    | 3.01      |
| two-model fusion (simple average) | LoRA               | 1.92(43%)     | 1.15(47%)    | 1.53      |
|                                   | L-LoRA             | **4.05(99%)**     | **1.51(78%)**    | **2.78**      |

Notably, the two-model fusion using simple averaging yields different results for each method. The LoRA fusion only retains 43% and 47% of the pretrained model's performance on VOC 2012 and NYUD v2, respectively. In contrast, the L-LoRA fusion maintains a remarkable 99% and 78% of the fine-tuned LoRA's performance on the respective datasets, resulting in a mean mIoU of 2.78.