Paper: Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank Matrices (ICLR-2024)

Reviewer comment (Theory):
[Major] **Delta-LoRA seems to be fundamentally equivalent to LoRA.** I am very unclear about the fundamental difference between updating the pretrained weights using Delta-LoRA and LoRA. Basically, **both of them are performing low-rank updates**. Updating the pretrained weights does not seem to have any benefits, and if we treat W+AB as a whole $\hat{W}$, then, what you are doing can be viewed as LoRA but applied with a different learning rate. Please correct me if I am wrong.

Author response:
We would like to clarify some points and help you understand our Delta-LoRA better. We want to point out that Delta-LoRA is fundamentally different from LoRA in the following three aspects:

 - Given $\boldsymbol{W + AB}$, $\boldsymbol{W}$ is fixed in LoRA, while $\boldsymbol{W}$ will be updated in our Delta-LoRA. This is **the largest difference between LoRA and Delta-LoRA**. This modification can enhance the representative ability of the model.
 - $\text{Rank}(\Delta \boldsymbol{W_{Delta-LoRA}}) = \text{Rank}(\boldsymbol{W}^{(T)} - \boldsymbol{W}^{(0)} + \boldsymbol{AB})>\text{Rank}(\Delta \boldsymbol{W}_{LoRA}) = \text{Rank}(\boldsymbol{AB})$. The rank of the learned incremental weight matrix in our Delta-LoRA is larger than that in the original LoRA.
 - The gradient flow is different between LoRA and Delta-LoRA. We can know that $\frac{\partial \mathcal{L}}{\partial A} = B^{\top}\cdot\frac{\partial \mathcal{L}}{\partial W}$ and $\frac{\partial \mathcal{L}}{\partial B} = \frac{\partial \mathcal{L}}{\partial W}\cdot A^{\top}$ according to the chain rule.

Here is the **back-propagation process of LoRA**: $\frac{\partial \mathcal{L}}{\partial A^{(t+1)}}= \frac{\partial \mathcal{L}}{\partial W^{(t+1)}}\cdot (B^{(t+1)})^{\top}=\frac{\partial \mathcal{L}}{\partial W^{(0)}+(A^{(t)}+\Delta A^{(t)})(B^{(t)} + \Delta B^{(t)})}\cdot (B^{(t)}+\Delta (B^{(t)}))^{\top}$
$\frac{\partial \mathcal{L}}{\partial B^{(t+1)}} = (A^{(t+1)})^{\top} \cdot \frac{\partial \mathcal{L}}{\partial W^{(t+1)}}=(A^{(t)}+\Delta (A^{(t)}))^{\top} \cdot \frac{\partial \mathcal{L}}{\partial W^{(0)}+(A^{(t)}+\Delta A^{(t)})(B^{(t)} + \Delta B^{(t)})}$ 
Here is the **back-propagation process of Delta-LoRA**:
$\frac{\partial \mathcal{L}}{\partial A^{(t+1)}} = \frac{\partial \mathcal{L}}{\partial W^{(t+1)}}\cdot (B^{(t+1)})^{\top}=\frac{\partial \mathcal{L}}{\partial ((W^{(t)}+ \lambda \Delta A^{(t)}B^{(t)}) + (A^{(t)}+\Delta A^{(t)})(B^{(t)}+\Delta B^{(t)})) }\cdot (B^{(t)}+\Delta (B^{(t)}))^{\top}$ 
$\frac{\partial \mathcal{L}}{\partial B^{(t+1)}} = (A^{(t+1)})^{\top} \cdot \frac{\partial \mathcal{L}}{\partial W^{(t+1)}} = (A^{(t)}+\Delta (A^{(t)}))^{\top} \cdot \frac{\partial \mathcal{L}}{\partial ((W^{(t)}+ \lambda\Delta A^{(t)}B^{(t)}) + (A^{(t)} + \Delta A^{(t)})(B^{(t)} + \Delta B^{(t)})  )}$.

Reviewer comment (Experiments):
[Major] Neither DyLoRA nor AdaLoRA outperforms LoRA as they claimed in their paper. I noticed that the authors chose different settings with the AdaLoRA, this might be the reason for the bad performance. However, the authors do not provide any explanation on why they chose this different setting. This makes the outperformance of Delta-LoRA reported in the tables less convincing.

Author response:
***For a fair comparison, we use the same training setups as LoRA. The AdaLoRA and DyLoRA chose different training settings as LoRA. In the following, we want to clarify the differences among them.*** 

First, we want to point out that

1. DyLoRA aims to improve the performance of lower rank when inference rather than the rank they selected at the training stage. That means the performance they claimed in their paper was based on a higher rank, which can yield a much more memory overhead. For a fair comparison, we report their results based on the low-rank setting. 

Second, AdaLoRA uses a different setting from LoRA, the differences are summarized as:

1. The AdaLoRA fine-tuned each layer of the pretrained matrices in Transformer. This operation will cause extra **activation memory** consumption (i.e. **19.79 GB extra GPU memory** for LLaMA-7B with batch size of 1024 and 1024 max length implemented by Pytorch). Therefore, we do not choose the similar training setups with AdaLoRA. Instead, we used the same training setups with LoRA.
2. AdaLoRA has 7 extra hyper parameters (i.e. target_rank, reg_orth_coef, init_warmup, final_warmup, mask_interval, beta1, beta2). We cannot search the best hyper-parameters for the different datasets and models.

Reviewer comment (Experiments):
[Major] Why is the learning rate fixed? In my intuition, the only difference between the Delta-LoRA seems to be the resulting learning rate as I explained in the first bullet point. Therefore, it is unclear whether the reported performance of Delta-LoRA happen to outperform LoRA in this learning rate.

Author response:
To make fair comparison, we use the same learning rate as reported in the original paper and code(e.g. 2e-4 on E2E Challenge dataset). Besides, we apply linear learning rate scheduler at the training stage, which is same as the setting of LoRA. We could use better hyper-parameters to further improve the results of our Delta-LoRA with standard errors. 


| MNLI| SST-2 | MRPC | CoLA | QNLI | QQP | RTE | STS-B | AVG|
|----------|----------|----------|----------|----------|----------|----------|----------|----------|
| 87.62 $\pm$ 0.21 | 95.29 $\pm$ 0.23 | 90.60 $\pm$ 0.14 | 64.64 $\pm$ 0.86 | 93.09 $\pm$ 0.15 | 91.01 $\pm$ 0.06 | 87.00 $\pm$ 0.36 | 91.61 $\pm$ 0.04 | 87.60 |

Reviewer comment (Evaluation):
[Medium] The improvements look not significant.

Author response:
We have decent improvement on most NLG datasets (e.g. 1.24 on E2E, 0.91 on WebNLG, 10.4\% on LLaMA-7B with Alpaca dataset), and some NLU datasets (e.g. 0.65 on CoLA). More importantly, we use the same learning rate and same model architecture with LoRA, we do not search the best hyper-parameter and show the real performance of our Delta-LoRA in our paper. The Delta-LoRA has consistent performance improvement over 4 NLG datasets and 8 NLG datasets, which is a general method and can be applied to multiple domains.

Reviewer comment (Reproducibility):
[Medium] The code is not publically available thus I cannot verify the reproducibility of the experiment results.

Author response:
We will release our models and all codes after the decision of this conference.

Reviewer comment (Evaluation):
The only major drawback seems to be that the differences in empirical results do not seem statistically significant (and do not have any errors provided). Given the incremental increase, it is not clear whether the performance of the algorithm stems from real gains or is noise from insufficient tuning.

Author response:
We would like to emphasize the effectiveness of the proposed Delta-LoRA from the following points:


- **We kept the similar training settings as LoRA**, such as learning rate, weight decay, rank number, training epochs and random seed. We do not use a parameter search and select the random seed (we kept all experiments with random seed=0). Therefore, the improvement that claimed in our paper is convincing. 

- **We achieved significant improvement compared with AdaLoRA and DyLoRA.** We used the same training setting to evaluate the two methods. We observed that our method could have pronounced performance gain.

- **We witnessed consistent improvement across multiple datasets.** According to our experiment, we achieved improvement across 8 NLU datasets and 4 NLG datasets, which proves that our method works well across different tasks.

Reviewer comment (Theory):
The whole idea is pretty much a heuristic which works very well in practice (but the same can be said of LoRA).

Author response:
Thanks for your comment. Our initial motivation was indeed heuristic, while we gave a mathematical formulation and presented our analysis of its rationality. We supplied a more theoretical proof to further demonstrate the difference between Delta-LoRA and LoRA in Sec. A.4 of the revised version.

Reviewer comment (Presentation):
The overt emphasis on the fact that $g_W = g_{AB}$ seems unnecessary (specifically Section 4.1) since the results stems from fairly straightforward linear algebra and calculus. It might be better to reduce that section, and allocate space to more impactful discussion/experiments.

Author response:
Thanks for your kind advice. We appropriately reduce this statement in our revised paper. We modified some content in our revised paper, and marked these descriptions with blue color.

Reviewer comment (Evaluation):
I am very divided on accepting this paper simply due to the fact that all algorithms seem very close to each other in empirical results, the differences being marginal in most cases. However, due to the fact that this method does provide positive gains across the board, I vote to accept the paper.

Author response:
We would like to emphasize the effectiveness of the proposed Delta-LoRA from the following points:


- **We kept the similar training settings as LoRA**, such as learning rate, weight decay, rank number, training epochs and random seed. We do not use a parameter search and select the random seed (we kept all experiments with random seed=0). Therefore, the improvement that claimed in our paper is convincing. 

- **We achieved significant improvement compared with AdaLoRA and DyLoRA.** We used the same training setting to evaluate the two methods. We observed that our method could have pronounced performance gain.

- **We witnessed consistent improvement across multiple datasets.** According to our experiment, we achieved improvement across 8 NLU datasets and 4 NLG datasets, which proves that our method works well across different tasks.

Reviewer comment (Evaluation):
Can you provide standard errors on some of your empirical results?

Author response:
We provide the standard errors on GLUE benchmark in the following table. We used better learning rate and start steps here which can be found in the Section A.5 of our revised paper.

| MNLI| SST-2 | MRPC | CoLA | QNLI | QQP | RTE | STS-B | AVG|
|----------|----------|----------|----------|----------|----------|----------|----------|----------|
| 87.62 $\pm$ 0.21 | 95.29 $\pm$ 0.23 | 90.60 $\pm$ 0.14 | 64.64 $\pm$ 0.86 | 93.09 $\pm$ 0.15 | 91.01 $\pm$ 0.06 | 87.00 $\pm$ 0.36 | 91.61 $\pm$ 0.04 | 87.60|

A better choice of learning rate and start steps can further improve the performance of Delta-LoRA.

Reviewer comment (Experiments):
The main weakness of the paper is the lack of experiments on large models where the memory efficiency of Delta-LoRA would be most beneficial. The models tested in the paper are relatively small, which makes the memory-saving advantage less meaningful.

Author response:
Thanks a lot for your suggestions! We have evaluated the performance of LLaMA-7B model on Alpaca dataset, in the revised manuscript, we have clarified the experimental results. We find that, using evaluation of GPT-4, Delta-LoRA improves  LoRA by around 10\%, which proves the effectiveness of our Delta-LoRA.

Reviewer comment (Experiments):
There is a performance loss compared to fine-tuning, which is expected. However, the paper should have reported the performance of full finetuning with stateless optimizers (such as SGD) and showed that Delta-LoRA outperforms this baseline.

Author response:
The AdamW is widely used in the training of transformers, but SGD is not so popular. In NLP and multimodal tasks, most people choose AdamW instead of SGD, since it has a stable performance and converges fast. As the reasons described before, we chose the AdamW optimizer to conduct experiments to make fair comparison. We run the experiments with SGD optimizer on CoLA dataset with rank=8, $\alpha$=16, learning rate=0.01 and momentum=0.9. LoRA achieves 63.67 while Delta-LoRA has 64.32. Besides, we observed that the training process occurred crash after 57 epochs for the LoRA's training, which emphasizes the instablity of the SGD optimizer for current NLP tasks.

Reviewer comment (Experiments):
Could you provide more evidence to support the claim that Delta-LoRA can nearly match the performance of full finetuning with "large" models?

Author response:
We have tested the effectiveness of Delta-LoRA with LLaMA-7B on Alpaca dataset. We found that the memory cost of Delta-LoRA is basically same as the LoRA, while our method outperforms the LoRA for around 10\% under the evaluation of LLMs.

Reviewer comment (Experiments):
How does Delta-LoRA's performance compare to full finetuning with stateless optimizers? (Note -- the memory footprint is actually slightly lower with full finetuning with stateless optimizers)

Author response:
The AdamW is widely used in the training of transformers, but SGD is not so popular. In NLP and multimodal tasks, most people choose AdamW instead of SGD, since it has a stable performance and converges fast. As the reasons described before, we chose the AdamW optimizer to conduct experiments to make fair comparison. We run the experiments with SGD optimizer on CoLA dataset with rank=8, $\alpha$=16, learning rate=0.01 and momentum=0.9. LoRA achieves 63.67 while Delta-LoRA has 64.32. Besides, we observed that the training process occurred crash after 57 epochs for the LoRA's training, which emphasizes the instablity of the SGD optimizer for current NLP tasks.