Paper: ROSA: Random Orthogonal Subspace Adaptation (ICLR-2024)

Reviewer comment (Novelty):
People are increasingly interested in LoRA than other PEFT methods because LoRA stores and loads a small number of task-specific parameters during inference. ROSA reintroduces this challenge, making it less practical than LoRA for inference.

Author response:
We are aware this benefit of storing few parameters on disk when fine-tuning a baseline model for many downstream tasks is absent in ROSA. However, we think that there are many settings in which users would like to fine-tune a model for a single downstream task only. In such settings, ROSA shines as it provides the same benefits as LoRA while maintaining the same expressivity as full fine-tuning. We have added a few sentences to Section 4.6 to try emphasize the settings in which ROSA excels.

Reviewer comment (Experiments):
The paper did not use real-world datasets to verify the expressive ability of ROSA. Considering that there are many real-world datasets suitable for regression experiments, there is no need to use synthetic data as in Section 4.1.

Author response:
The intent of the synthetic experiments is to illustrate how ROSA bypasses the low rank limitation of LoRA in a simple controlled environment where we know exactly what the target rank is — with a real-world dataset, we would not know precisely what the target rank of the adapter is. We have also made an adjustment to the figure, adding the loss curve for full fine-tuning for better comparison. We have also added a few sentences to Section 4.1 to clarify the intent of this figure.

Reviewer comment (Experiments):
Is it possible to perform some extra steps before merging to enforce orthogonality?

Author response:
We are not entirely sure by what is meant by performing some extra steps before merging to enforce orthogonality. If you are asking if it is possible to include experiments where we actually enforce orthogonality before performing the merge operation, then this could indeed be another possible direction for performing the merge operation. However, this may reduce model performance as it would amount to merging a projection of the trained PEFT parameters.

Reviewer comment (Experiments):
Since the entire model parameters are updated, it becomes crucial to consider a comparison with the full fine-tuning training strategy. If the convergence speed of the full-finetuning is not improved or comparable, the primary advantage of ROSA only relies on low memory usage for finetuning, which can also be achieved by other means like model sharding for faster training with a number of GPUs.

Author response:
**Response to weakness 1**- Using model sharding for fine-tuning will split the total memory used to train a model across multiple GPUs. In cases where users have access to many GPUs to fine-tune their models for a single task, then there is no need to use any PEFT methods. However, in resources limited settings, users would like to fine-tune models using a limited number of GPUs with a collective memory that is less than the memory required for full fine-tuning of a model. In these settings PEFT models are needed as the only alternative would be layer-wise training. However, layer-wise training incurs significant latency overhead due to constant data movement onto and from the GPUs. We have added a discussion on this point in section 4.6

Reviewer comment (Evaluation):
The paper compares the training time of one epoch of fine-tuning in Table 1, where the Epoch time of full finetuning, as reported, is ~157 seconds. When comparing with the proposed method, the Epoch time + Factorize Time also results in a similar time of ~153 seconds + 4 seconds, since at every epoch, a ROSA factorization step is performed, the overall finetuning time of the proposed method and the finetuning of the entire model is similar. If the convergence speed of the proposed method is similar (or lower) to finetuning, the primary motivation of PEFT approaches (less training time) turns out to be missing. I am not sure if I am missing something; however, if the entire model is updated (not keeping the source knowledge intact in terms of pretrained model parameters as done in other adapter-based approaches), the proposed method results in the same model performance as finetuning with no advantage of PEFT approaches. Moreover, as highlighted by the authors, this also limits the usage of the proposed technique for multi-task/domain adaptation.

Author response:
**Response to weakness 2**-  
The primary advantage of PEFT methods is reducing the memory required to fine-tune a model on a downstream task, as opposed to speeding up training. The motivation is to allow users with potentially a single GPU to have the ability to fine-tune a large scale model in a memory constrained setting. This can be achieved by both LoRA and ROSA as they both provide a means of fine-tuning large models using significantly less memory. Moreover, other PEFT methods do not always lead to a speedup on GPUs (as shown in Table 1,  LoRA and Full fine-tuning all have comparable inference time). However, we realize the motivation may have been a little under-emphasized so we have reworded some sentences throughout the paper to try emphasize the motivation of ROSA (see text in blue in the introduction and at the beginning of the methodology section).

Reviewer comment (Novelty):
The proposed scheme seems like an Alt-Opt optimization scheme, where the model is updated on a few of the selected low-rank subspace directions, keeping other subspace directions fixed for a single update and later iterating over different directions to update the entire model’s parameter space. More detailed discussion on similar lines would help understand the proposed method better.

Author response:
**Response to question 1**-  
We are not very clear on this question. Do you mean that we should better explain in the paper that in ROSA the model is updated on a few of the selected low-rank subspace directions, while keeping other subspace directions fixed for a single update and later iterating over different directions?

Reviewer comment (Evaluation):
A detailed analysis of the convergence rate with various PEFT methods, along with the full finetuning, would be required to obtain a transparent picture of the proposed PEFT technique. It would be interesting to observe if the proposed scheme helps converge with less variability and low sensitivity.

Author response:
**Response to question 2**-  
We agree with you on this point. For better transparency we have added the training curve of CoLA to the paper in the appendix, and refered to it in the experimental section. Currently we do not observe a major differences between the convergence of ROSA, LoRA and full fine-tuning on the CoLA dataset. We plan to add more of the GLUE dataset training curves to the paper in the same manner.

Reviewer comment (Experiments):
In Figure 2, it would be good to make a comparison with full-finetuning, highlighting the comparison between the rate of convergence.

Author response:
**Response to question 3**- Per your suggestion we have added a comparison with full fine-tuning in Figure 2

Reviewer comment (Novelty):
The ROSA method has been published in ICML23 [1]. The main idea is similar. This paper provides a more solid theoretical analysis and compares ROSA with LoRA on a few NLU and NLG tasks. However, improvements do not appear to be significant.

Author response:
The ROSA method was presented as a poster in a non-archival workshop very recently (less than 4 months from the submission deadline of ICLR). As mentioned in the guidelines, authors are not required to compare their work to other works not published in peer-reviewed conference proceedings or journals, as well as to contemporaneous work (less than 4 months before the ICLR deadline).

Reviewer comment (Experiments):
More experiments are needed to verify the effectiveness of the proposed method. The task selections in the experiment section are unclear; for example, the QQP task is missing in GLUE tasks. Ablation studies are performed on CoLA and MRPC, both of which use small training sets and whose results often fluctuate significantly.

Author response:
1. We have added the QQP results for LoRA and ROSA in the updated PDF file for rank=2 (rank=8 results are pending). Moreover, to improve our ablation study - we are are currently running experiments on the STS-B dataset that will be completed by the end of the author-reviewer discussion period.  
2. Unfortunately, we do not have access to industry scale computing resources, which prevents us from running experiments on very large language models. Note that the experiment results provided in the paper already required 100 GPU/Days of compute time. We also believe that the results in the paper already deliver strong evidence to ROSA’s efficiency.  
3. Investigating the efficacy of our method in other domains such as computer vision is indeed an interesting direction. However, this goes beyond the scope of this paper and is deferred to future work.

Reviewer comment (Experiments):
PEFT methods are typically used for large models, so it is necessary to check the performance of the proposed method on large models.

Author response:
Indeed, this is the main limitation of ROSA compared with LoRA. However, we believe there are many situations where there is a need to fine-tune a model efficiently, for a single downstream task, in memory limited settings. ROSA excels in such settings as it retains the low-memory-usage property of LoRA while achieving significantly higher performance (on par with full fine-tuning) on the downstream task.

Reviewer comment (Experiments):
LoRA is particularly useful for image generation tasks such as SD models, so it would be beneficial to include more experiments in computer vision tasks.

Author response:
1. We have added the QQP results for LoRA and ROSA in the updated PDF file for rank=2 (rank=8 results are pending). Moreover, to improve our ablation study - we are are currently running experiments on the STS-B dataset that will be completed by the end of the author-reviewer discussion period.  
2. Unfortunately, we do not have access to industry scale computing resources, which prevents us from running experiments on very large language models. Note that the experiment results provided in the paper already required 100 GPU/Days of compute time. We also believe that the results in the paper already deliver strong evidence to ROSA’s efficiency.  
3. Investigating the efficacy of our method in other domains such as computer vision is indeed an interesting direction. However, this goes beyond the scope of this paper and is deferred to future work.

Reviewer comment (Theory):
Lack of clarity regarding limitation 2 of LoRA. The authors claim that "Second, initializing the adapter AB to zero can be thought of as learning new representations from scratch separately from the pre-trained ones (φ(x) = Wx + ABx := φpre-trained(x) + φtrainable(x)), rather than leveraging the pre-trained features the model already has to initialize the adapter." From the reviewer's perspective, this design element is a benefit rather than a limitation, as it allows fine-tuning to start from the pre-trained checkpoint, safeguarding its integrity. In ROSA, users can also start training directly from the pre-trained checkpoint without initiating new parameters. Both methods are trained using pre-trained checkpoints; therefore, the reviewer would not consider this a LoRA limitation and seeks the authors’ thoughts.

Author response:
Indeed, starting fine-tuning from the pre-trained checkpoint is a beneficial feature of both ROSA and LoRA. However, our ablation study shows that initializing LoRA with SVD (cf row “ROSA (SVD Init)” in Table 4) leads to better performance on the CoLA dataset. This supports our intuition that leveraging the pre-trained features to initialize the adapter is beneficial, compared to initializing the adapter to zero and learning new features from scratch. We added a sentence clarifying this observation in the ablation study.

Reviewer comment (Experiments):
Question about the synthetic data experiments. LoRA is a method for fine-tuning, which assumes that fine-tuning tasks or domain adaptations can be achieved with low-rank tuning on a well-pre-trained base model. The reviewer is confused about the synthetic data experiment; although it can show that LoRA may not be optimized well in some cases, this is not the typical application scenario for methods like LoRA.

Author response:
Indeed, LoRA assumes that the downstream fine-tuning task or domain adaptations can be achieved with a low-rank tuning. In contrast ROSA does not make this assumption and in Theorem 1 we show that ROSA can fine-tune a model to the same solutions as full fine-tuning. The synthetic experiment mainly serves as an illustration to demonstrate that the theoretical claims made for the ROSA method are indeed realized in practice (for non-linear models trained by gradient descent, in contrast to the linear regression model considered in Theorem 1). We realize we had not included the loss curve for full fine-tuning. We have included it to better communicate the intent of Figure 2. (black curve labelled FT in Figure 2).We have also added additional remarks in  Section 4.1 to better communicate this point.