Paper: VeRA: Vector-based Random Matrix Adaptation (ICLR-2024)

Reviewer comment (Evaluation):
In Table 2 for the RTE dataset the LoRA algorithm achieves significantly higher results on the RoBERTa-base model compared to the the other algorithms. Is this a mistake?

Author response:
Results reported for LoRA are taken from the paper that introduced the method [1]. Our reproduced results are _close_ to this value, which may indicate that it's not a mistake.

Reviewer comment (Evaluation):
Why are the PEFT algorithms achieving higher accuracy compared FT on the RTE dataset? Is this just overfitting?

Author response:
Indeed, overfitting may be the reason of lower performance of FT compared to PEFT methods. It may also explain higher performance of ELoRA on certain tasks, as it comes with highly reduced number of trainable parameters, potentially acting as regularisation.

Reviewer comment (Experiments):
Performance can be hurted using this method, even the evaluation is only on a limited set of GLUE benchmarks.

Author response:
The primary objective of our method is to significantly reduce the number of trainable parameters, and therefore storage requirements, with *comparable* performance to other methods. As detailed in Figure 3, our analysis acknowledges the parameter-performance trade-off inherent in both LoRA and ELoRA methods. While in theory, ELoRA _could_ perform worse than LoRA, in practice we see that with a sufficient number of trainable parameters, ELoRA's performance aligns with that of LoRA, despite our method using much less parameters. 

In addition, we have also increased the types of evaluations by comparing our method on E2E generation  (see answer below) and on instruction following tasks - additionally to the results reported in the initial version of our paper, we are introducing evaluation with MT-Bench, a well-established instruction-following benchmark. Similarly to the initial experiment, we evaluate model finetuned on the *alpaca-cleaned* dataset, this time taking the full training set (40k+ samples), instead of 10k subset. Results are shown in Table 1. 


**Table 1 Instruction tuning of more models and evaluated on MT-Bench**

| Model        | Method | Parameters | Score |
|--------------|--------:|------------:|-------|
| Llama 13B    |       | -          | 2.61  |
||||
| Llama 7B     | LoRA   | 159.9M     | 5.03  |
| Llama 7B     | ELoRA  | 1.6M       | 4.77  |
||||
| Llama 13B    | LoRA   | 250.3M     | 5.31  |
| Llama 13B    | ELoRA  | 2.4M       | 5.22  |
||||
| Llama2 7B    | LoRA   | 159.9M     | 5.19  |
| Llama2 7B    | ELoRA  | 1.6M       | 5.08  |
||||
| Llama2 13B   | LoRA   | 250.3M     | 5.77  |
| Llama2 13B   | ELoRA  | 2.4M       | 5.93  |

From this table, we make the following observations:
 - Base models do not perform well compared to instruction tuned versions (scores of 2.6 vs 5.3)
 - ELoRA generally closely matches LoRA's performance. In particular for the larger model sizes, such as Llama 13B the scores are very similar (5.22 vs 5.31) and for Llama2 13B ELoRA even outperforms LoRA (5.93 vs 5.77).

Reviewer comment (Experiments):
The method should be evaluated on generative tasks as well, including translation and question answering. Classification task and summarization task are known to be more resilient to the lose of model capacity or trainable parameters.

Author response:
We have conducted additional experiments on a generative E2E task with GPT2, medium and large variants, finetuned with LoRA and ELoRA methods, adhering to the setup described in the original LoRA paper [1]. For LoRA we use hyperparameters reported in [1], and for ELoRA we select a rank equal to 1024 and tune the learning rate. Results after 5 epochs are reported in the following table:

| Method               | Parameters | BLEU  | NIST   | METEOR | ROUGE_L | CIDEr  |
|---------------------|------------|-------|--------|--------|---------|--------|
| LoRA (GPT2-M)        | 0.35M      | 68.92 | 8.69 | 46.42  | 71.34   | **2.51** |
| ELoRA (GPT2-M)    | **0.10M**     | **70.08** | **8.81** | **46.55**  | **71.45**   | 2.50 |
|||||
| LoRA (GPT2-L)         | 0.77M      | 70.14 | 8.84 | 46.68  | **71.85**   | 2.52 |
| ELoRA (GPT2-L)     | **0.17M**     | **70.28** | **8.85** | **46.89**  | 71.63   | **2.54** |

As we can see in the table, our method **performs on-par or better on E2E-NLG, despite using only 22%-28% of the number of trainable parameters of LoRA**. We thank the reviewer for suggesting this additional experiment and will add these results to the paper.

Reviewer comment (Evaluation):
The paper is not well written. Figure 3 and Figure 4 are each on a single but different task. Why not reporting a mean score on GLUE?

Author response:
While Figures 3 and 4 are independent and address different aspects, we agree that consistency could enhance the clarity of our presentation.   Consequently, we will update Figure 4 to showcase data from the RTE task, making it consistent with Figure 3.

In terms of reporting on the GLUE score, our methodology is in line with that of the LoRA paper [1], which opts to report median values. This approach enables a straightforward comparison between our results and those reported in the LoRA paper and is more robust to outliers.

Reviewer comment (Evaluation):
The paper lacks explanation on why Table 3, with fewer tunable parameters, ELoRA has better instruction finetuning scores than LoRA. It is a bit of counterintuitive.

Author response:
One possible explanation is that ELoRA is less prone to overfitting and may achieve better results on particular tasks, especially in low-data regime. This is similar to the existing findings of PEFT methods often achieving better results than full finetuning on some tasks.

Reviewer comment (Theory):
Are there any theoretical proof that random matrices with scaling vectors would work better than LoRA?

Author response:
Please see our response in the "Theoretical framework/expressivity of ELoRA" part of the [common reply](https://openreview.net/forum?id=NjNfLdxr3A&noteId=xpnpKQguRK).

Reviewer comment (Experiments):
Consider evaluating the method on Generative tasks?

Author response:
We have conducted additional experiments on a generative E2E task with GPT2, medium and large variants, finetuned with LoRA and ELoRA methods, adhering to the setup described in the original LoRA paper [1]. For LoRA we use hyperparameters reported in [1], and for ELoRA we select a rank equal to 1024 and tune the learning rate. Results after 5 epochs are reported in the following table:

| Method               | Parameters | BLEU  | NIST   | METEOR | ROUGE_L | CIDEr  |
|---------------------|------------|-------|--------|--------|---------|--------|
| LoRA (GPT2-M)        | 0.35M      | 68.92 | 8.69 | 46.42  | 71.34   | **2.51** |
| ELoRA (GPT2-M)    | **0.10M**     | **70.08** | **8.81** | **46.55**  | **71.45**   | 2.50 |
|||||
| LoRA (GPT2-L)         | 0.77M      | 70.14 | 8.84 | 46.68  | **71.85**   | 2.52 |
| ELoRA (GPT2-L)     | **0.17M**     | **70.28** | **8.85** | **46.89**  | 71.63   | **2.54** |

As we can see in the table, our method **performs on-par or better on E2E-NLG, despite using only 22%-28% of the number of trainable parameters of LoRA**. We thank the reviewer for suggesting this additional experiment and will add these results to the paper.

Reviewer comment (Experiments):
Have you tried extending this method to finetuning large image models? What about large multimodal models?

Author response:
Thank you for your positive feedback and your suggestions to further improve the paper! Most importantly, we have now conducted experiments using Vision Transformers and shown benefits of ELoRA in terms of GPU memory. Please see below for our responses to your questions.

>Have you tried extending this method to finetuning large image models? 

Please see our response in the "Experiments on vision models" part of the [common reply](https://openreview.net/forum?id=NjNfLdxr3A&noteId=xpnpKQguRK).

Reviewer comment (Novelty):
The idea of using random matrices for parameter-efficient finetuning has been proposed previously in the context of text-to-image generation [1]. Although the ideas of the submission and that paper are quite different (also, the paper is quite recent), in my opinion, it would be great to mention LiDB (Lightweight DreamBooth) in the related work section to give the reader additional context.

Author response:
Thank you for pointing this out. We acknowledge the paper's relevance and will include it in our updated related work section.

Reviewer comment (Experiments):
My primary concern with respect to the experiments is that the instruction tuning evaluation is performed mostly through GPT-4 queries on a very small dataset. In my opinion, this approach suffers both from the lack of reproducibility (as we know, the model behind the GPT-4 API might change in the future) and a narrow coverage of possible contexts that might result in high variance of the metric. I think that using more established benchmarks (such as MMLU or BIG-Bench) would give a better picture of how instruction-finetuned models perform after PEFT.

Author response:
While using GPT-4 as-a-judge is well-established (e.g. [1,2]), we have now run **additional experiments**: We have evaluated our instruction-tuned models under the well-known multi-turn evaluation, MT-Bench [1], which offers a more robust evaluation compared to the previous method and is one of the key metrics for comparing instruction-tuned methods.
Similarly to the initial experiment, we evaluate model finetuned on the *alpaca-cleaned* dataset, this time taking the full training set (40k+ samples), instead of 10k subset. As shown in Table 1 below, we see that our model closely matches the performance of LoRA-based finetuning here too, despite using much less parameters.

Reviewer comment (Experiments):
Similarly, I think that broader evaluation of instruction tuning should include a larger set of models than just LLaMA-2. Personally, I would move Table 4 and Table 5 to the appendix (they take a lot of space and are not directly related to the topic of the paper) and replace that with additional experiments, ideally for models with higher parameter counts.

Author response:
We have further increased the **model diversity**: Our additional experiments include two variants of LLaMA-1 and LLaMA-2, specifically the 7B and 13B models. This demonstrates ELoRA's applicability across different model sizes.

In addition, we now **compare against the baseline model**: We've included results for the base LLaMA 13B model without instruction tuning as a baseline [3].




**Table 1 Instruction tuning of more models and evaluated on MT-Bench**

| Model        | Method | Parameters | Score |
|--------------|--------:|------------:|-------|
| Llama 13B    |       | -          | 2.61  |
||||
| Llama 7B     | LoRA   | 159.9M     | 5.03  |
| Llama 7B     | ELoRA | 1.6M       | 4.77  |
||||
| Llama 13B    | LoRA   | 250.3M     | 5.31  |
| Llama 13B    | ELoRA  | 2.4M       | 5.22  |
||||
| Llama2 7B    | LoRA   | 159.9M     | 5.19  |
| Llama2 7B    | ELoRA  | 1.6M       | 5.08  |
||||
| Llama2 13B   | LoRA   | 250.3M     | 5.77  |
| Llama2 13B   | ELoRA  | 2.4M       | 5.93  |

From this table, we make the following observations:
 - Base models do not perform well compared to instruction tuned versions (scores of 2.6 vs 5.3)
 - ELoRA generally closely matches LoRA's performance. In particular for the larger model sizes, such as Llama 13B the scores are very similar (5.22 vs 5.31) and for Llama-v2 13B ELoRA even outperforms LoRA (5.93 vs 5.77).

Thank you for suggesting this. We will add these results to the paper.

Reviewer comment (Experiments):
Technically, it should be possible to have different random matrices in different layers (the cost of storing one random seed per layer should be negligible), which might increase the capacity of the model even further. I wonder if you have explored this idea in preliminary experiments? It is quite surprising to me that a single random structure is sufficient for PEFT across all layers in a Transformer model — perhaps there is something we can infer from that.

Author response:
We have conducted additional experiments to assess the impact of using unique versus shared random matrices in different layers of a Transformer model. As shown in Table 2, for a rank of 256, our results indicate a slight improvement in performance when using unique random pairs. A reason for this might be that this alleviates the effect of non-optimally initialised shared matrices. Yet, given how small the difference is, this suggests that even this additional slack is not necessary when it comes to finetuning, echoing our findings of comparing LoRA to our method where less parameters did not translate into worse performance.

Besides this, opting for unique matrices can lead to increased training times (when re-generating the matrices from seed) or, if these matrices are stored in memory, there will be a minor increase in memory usage. We will add this analysis to the paper.



**Table 2 Comparison of shared vs unique matrices**
Our experiments were carried out on four GLUE tasks (the fastest ones to compute): STS-B, CoLA, RTE, and MRPC. We employed two setups to compare shared and unique matrices, repeating each experiment five times to calculate the mean results and standard deviation. 
| Task | Shared Matrices (Mean ± SD) | Unique Matrices (Mean ± SD) |
|------|-----------------------------|-----------------------------|
| STSB | 91.5 ± 0.6                  | 91.5 ± 0.2                  |
| CoLA | 67.7 ± 0.8                  | 68.3 ± 1.8                  |
| RTE  | 84.6 ± 1.5                  | 84.6 ± 0.8                  |
| MRPC | 90.0 ± 0.9                  | 90.7 ± 0.3                  |

Reviewer comment (Experiments):
I think that the work would benefit from a bit more analysis behind the reasons of why random projections work that well in the context of parameter-efficient finetuning. At the very least, it might be interesting to compare the structure of the learned weights for LoRA and ELoRA trained with a single rank: intuitively, the latter should approximate the former, but is that true in practice?

Author response:
To address your query, we compared the weights trained with LoRA and ELoRA at a single rank of 64 across all *query* layers. For each method and adapted layer, we constructed a _weight difference_. In LoRA's case, this involved the multiplication of two low-rank matrices, while for ELoRA, it also included multiplication by scaling vectors. We then calculated the cosine similarity of these flattened weights. Additionally, we compared the similarity between trained LoRA weights and randomly initialized matrices as a baseline: We find that similarities of ELoRA to LoRA are on average 2e-3  while LoRA to random matrices is -8e-5 (low numbers are due to the high dimensionality).


Our findings reveal a notable increase in similarity between the trained weights, particularly in the latter layers. This observation aligns with our earlier findings (referenced as Figure 4 in our paper) that the highest adaptation occurs in these layers. These results support the notion that ELoRA can approximate the weights trained with LoRA. Please also see our response about the theory and expressive power of ELoRA in the [common reply](https://openreview.net/forum?id=NjNfLdxr3A&noteId=xpnpKQguRK).