Paper: The Expressive Power of Low-Rank Adaptation (ICLR-2024)

Reviewer comment (Writing):
A notation table would help understand all the notations, since the paper is mostly about theoretical proof.

Author response:
Thanks for the suggestion. Due to the page limit, We have summarized a list of the notations in Appendix A.

Reviewer comment (Theory):
In section 2, why a L-layer (instead of one-layer) linear model is considered, which is still a linear model?

Author response:
Correct. In our paper, we employ *deep linear models* (L-layer linear model) as a preliminary model, which serves as a foundation for extending our results to nonlinear models (i.e., FNN and TFN). Studying this toy model is a common technique in theoretical deep learning research, which offers valuable insights into deep nonlinear models, and has been employed in many notable studies, including those by `Saxe et al., 2014`, `Kawaguchi, 2016`, `Lu & Kawaguchi et al., 2017`, `Hardt & Ma`, and `Laurent & Brecht, 2018`.

Reviewer comment (Theory):
While it is okay to not have them in this paper, I think it would be interesting to study other effects of LoRA theoretically.

Author response:
We agree with the reviewer that it is an interesting research topic to explore the generalization and optimization problems of LoRA theoretically. While an in-depth analysis of these properties is beyond the scope of this paper, as the reviewer acknowledges, we have included additional simulation results and discussion based on your comments that provide initial insight into these issues. `Please find our responses in the general comment to all reviewers and AC, specifically under "(Major Update #2) Illuminating the Optimization and Generalization Aspects of LoRA"`.

Reviewer comment (Theory):
What can we say about how fast LoRA can converge even if the target model can eventually be found by LoRA exactly?

Author response:
We agree with the reviewer that it is an interesting research topic to explore the generalization and optimization problems of LoRA theoretically. While an in-depth analysis of these properties is beyond the scope of this paper, as the reviewer acknowledges, we have included additional simulation results and discussion based on your comments that provide initial insight into these issues. `Please find our responses in the general comment to all reviewers and AC, specifically under "(Major Update #2) Illuminating the Optimization and Generalization Aspects of LoRA"`.

Reviewer comment (Evaluation):
From Figure 1, I can see that LoRA of FNN performs on par with gradient update, whereas LoRA of TFNs significantly outperform gradient updates. Could the author explain this performance difference?

Author response:
In Fig. 1, we plot MSE, while in Fig. 5, we plot log(MSE). Sorry for the confusion, and we will make this clear in the revision. We have added Fig. 4 (https://hackmd.io/_uploads/Skgt22SV6.png) to report performances of FNN in log(MSE), and we observe that gradient update underperforms our construction on both FNN and TFN.

Reviewer comment (Evaluation):
It is impressive that LoRA with rank=1 can match the performance of gradient update in Figure 3. Does this mean the gradient update does not actually learn well?

Author response:
The reviewer is correct that gradient update might fail to find the optimal LoRA parameters in the TFN cases. This underperformance of gradient update method, especially compared to our LoRA adapter construction, suggests that LoRA's current effectiveness might still have room for further improvement. A more sophisticated implementation could potentially yield even better results, indicating a promising direction for future exploration and development in this area.