Paper: BiLoRA: A Bi-level Optimization Framework for Low-rank Adapters (ICLR-2024)

Reviewer comment (Experiments):
My main concern is that there is no evidence to support the claim the basis of this paper, that AdaLoRA overfits because it trains both singular values and vectors on the same full training set. While the authors claim this to motivate the need for bi-level optimization, it is unclear that the basis of their work is an actual limitation of existing methods. Without any evidence on the alleged limitations of AdaLoRA, it is unclear what advantages the bi-level optimization approach brings.

Author response:
>Weakness 1: The basis of this paper.

The basis of BiLoRA can be concluded in 4 aspects. 
* Limitations of existing fine-tuning methods.
* Inspirations and advantages of bi-level optimization.
* Accelerating the fine-tuning process.
* Opportunities for deeper understanding of PEFT process.

First, existing fine-tuning methods can suffer from overfitting. The large number of parameters in pre-trained models may make the full fine-tuning process more prone to overfitting[1]. Existing PEFT methods do not well solve this problem. The generalization capability of a fine-tuned model on the evaluation dataset can reflect the range of overfitting. We plot the training loss and evaluation loss of LoRA, AdaLoRA and BiLoRA to illustrate the generalization capability. We conduct experiments on RoBERTa-base on CoLA dataset and clip the first 2k steps. Figure 2(a) and Figure 2(b) show that for both LoRA and AdaLoRA, the evaluation loss starts to increase at or before 1k step while the training loss still decreases. The gap between the 2 losses is getting huge. This demonstrates the limitations of existing methods that they still suffer from overfitting issues. Figure 2(c) shows that both inner training loss and outer training loss are close to the evaluation loss. The evaluation loss is still decreasing after 2k steps and is obviously lower than that of LoRA and AdaLoRA. These results convincingly demonstrate the limitations of existing methods and the effectiveness of BiLoRA in addressing overfitting.

Second, we gain inspiration from DARTS (Differentiable Architecture Search) which utilizes a bi-level optimization for searching network architectures. DARTS uses training dataset and validation dataset separately for training operations and their corresponding proportions in 2 levels. DARTS conducts experiments on comparison between bi-level optimization and the normal training method(single-level). The latter performs much worse and they concludes that the latter “would cause α (analogous to hyperparameters) to overfit the training data, leading to poor generalization”[2]. We gain this inspiration of preventing overfitting and innovatively explore the possibilities of applying bi-level optimization to PEFT domain for addressing the existing limitations of overfitting. 

Third, one important goal of PEFT is to accelerate the fine-tuning process. The bi-level optimization framework effectively accommodates larger learning rates by iteratively optimizing between the two levels without affecting the training stability. Through BiLoRA, we significantly reduce the training steps and training time needed for convergence.

Fourth, we expect BiLoRA to encourage deeper understanding of low-rank fine-tuning processes and more design potentials. We offer more flexible and direct interaction with singular values and singular matrices at separate levels and design different parameterization of singular values. More efficient methods and deeper insights can be inspired by this bi-level optimization framework.

Reviewer comment (Experiments):
Although the main methodological difference with AdaLoRA is the bi-level optimization approah, the paper doesn't really experiment with the associated hyper-parameters. How is the number of iterations for singular value/vector updates found? Does this result in large performance/training-time differences? If the results are robust to the choice of $T_1$ and $T_2$, why not alternate between the two, i.e. $T_1 = T_2 = 1$? How does the dataset partition (choice of data on which singular values and singular vectors are trained on) affect the performance and training times?

Author response:
>Weakness 2: The influence of the dataset partition.

**Data Partition:**
The Dataset Partition, together with learning rate can help keep the balance of inner/outer optimization, which can contribute greatly to preventing the model from overfitting. Lower level has more trainable parameters, so it’s natural to use more data for training singular vectors, while using the left for training singular values. We further experiment on DeBERTa-v3-base on CoLA and SST2 datasets to show the influence of different dataset partitions. We change the inner level dataset partition from 0.6 to 1.0 with 0.1 interval.
| Data Partition | CoLA  | SST2  |
|----------------|-------|-------|
| 0.6            | 68.01 | 94.84 |
| 0.7            | 70.16 | 95.87 |
| 0.8            | **70.52** | 96.02 |
| 0.9            | 70.39 | **96.44** |
| 1.0            | 67.94 | 94.51 |
		                       
Results show that too small partitions(<=0.6) or too large partitions(1.0, only train singular matrix) can harm the overall performance.When the inner partition is too small, singular vectors are not well trained and when the inner partition is 1.0, singular vectors are not trained, which can cause a great performance drop. These results can also show that the two levels are both necessary in preventing overfitting and enhancing performances.
In the paper, we don’t ever change the partition of the data and keep it 8:2.

[1] Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers.

[2] Liu H, Simonyan K, Yang Y. Darts: Differentiable architecture search.

>Weakness 2 addition: The influence of $T_1/T_2$.

**Number of iterations:**
In the paper, we didn’t change the iteration numbers throughout our experiments and kept T1=T2=1. Actually there can be a set of hyperparameters that can all achieve good results because they can all keep good balance of inner/outer level and achieve 3 perspectives. 1) performing well on inner dataset; 2) performing well on outer dataset; 3) not overfitting on either subset and generalizing well on test dataset.
We further experiment on different iteration numbers as follows. We experiment on DeBERTa-v3-base on CoLA dataset. For larger T1 and T2, one global step contains more training steps so we reduce the total global steps accordingly. All other hyperparameters are kept the same.

| T1/T2 | 1     | 3     |
|-------|-------|-------|
| 1     | 70.52 | 70.40 |
| 2     | 69.96 | 70.14 |
| 5     | 70.01 | 69.19 |

The total optimization steps for inner level and outer level are close for different T1/T2. Typically, a single inner optimization step is faster than a single outer optimization step due to the calculation of hypergradients of outer level. So using a larger T1 is also an efficient choice.

We don’t exactly tune the iteration numbers for 2 reasons. 

* T1=T2=1 is an empirical choice in existing bi-level optimization tasks. The main effect of iteration numbers can be the balance of inner and outer level. Intuitively and practically, T1=T2=1 means BiLoRA frequently optimizes between the 2 levels, preventing the model from overfitting on either subset and effectively addresses overfitting. 

* Simplicity. We expect BiLoRA to be powerful and effective, yet easy to use. The number of hyperparameters for BiLoRA is kept nearly the same as LoRA, which is much less than AdaLoRA.

Reviewer comment (Experiments):
Ablation study on orthogonality regularization for the singular vectors illustrate that the performance is largely unaffected by the coefficient of this regularization, i.e. penalty for singular vectors violating orthonormality has minimal effect. Is this because the resulting pseudo-singular vectors turn out to be nearly orthogonal? It would be nice to see if the optimal solutions found are nearly orthogonal by plotting their angles & magnitudes.

Author response:
>Question 1: The orthogonality of optimal solutions.

We suspect the reason for “the performance is largely unaffected by the coefficient of this regularization” can be from these 3 aspects.
* The value distribution of $P^{T}P$ and $QQ^{T}$ in optimal solutions with regularization coefficient $\lambda=0.0$ is close to Identity Matrix. Despite magnitudes, singular vectors are largely orthogonal. When regularization coefficient $\lambda >= 0.1$, singular matrices are almost orthogonal.
* This “natural orthogonality" without regularization can be kind of due to the Normal Initialization we use for initializing all the singular vector matrices.
* Optimal solutions with different regularization coefficients including $\lambda=0.0$ can all contribute greatly to prevent overfitting.

First, we sample one of the singular vector matrices from the optimal solutions with reg coefficient=0.0 and 0.1. Since we use rank=8, the dimension of the resulting matrix is 8×8. We plot $P^{T}P$ in both AdaLoRA and BiLoRA. We sincerely refer you to the plots in Figure 6 (a)(b). The plots show that the value distribution of $P^{T}P$ from optimal solutions with reg coefficient=0.0  is very close to the Identity Matrix. When the reg coefficient > 0.0, the pseudo-singular vectors turn out to be almost orthogonal. These results can explain the robustness of BiLoRA across different regularization coefficients.

Secondly, this “natural orthogonality" can be kind of due to the Normal Initialization. We further plot the sampled matrix just after initialization. We sincerely refer you to the plots in Figure 6 (c). We use “nn.init.normal_()” for initializing all the singular vector matrices.

Third, optimal solutions with different reg coefficients can all contribute greatly to prevent overfitting. We plot the loss curves with reg coefficient=0.0 and 0.1. We sincerely refer you to the curve in Figure 7 and Figure 2.

Results demonstrate that with different regularization coefficients, BiLoRA can effectively prevent overfitting.

Reviewer comment (Theory):
I fail to understand why “Learning Λ by minimizing a single dataset’s training loss can easily render these contributions and parameter amounts tailored to this dataset”. In machine learning, aren’t we always learn parameters by minimizing a single dataset’s training loss?

Author response:
>Weakness 1 and 3: The limitation of minimizing a single dataset’s training loss and the motivation of BiLoRA.

The motivation of BiLoRA can be concluded in 4 aspects. 
* Limitations of existing fine-tuning methods.
* Inspirations and advantages of bi-level optimization.
* Accelerating the fine-tuning process.
* Opportunities for deeper understanding of PEFT process.

First, existing fine-tuning methods can suffer from overfitting. The large number of parameters in pre-trained models may make the full fine-tuning process more prone to overfitting[1]. Existing PEFT methods do not well solve this problem. The generalization capability of a fine-tuned model on the evaluation dataset can reflect the range of overfitting. We plot the training loss and evaluation loss of LoRA, AdaLoRA and BiLoRA to illustrate the generalization capability. We conduct experiments on RoBERTa-base on CoLA dataset and clip the first 2k steps. Figure 2(a) and Figure 2(b) show that for both LoRA and AdaLoRA, the evaluation loss starts to increase at or before 1k step while the training loss still decreases. The gap between the 2 losses is getting huge. This demonstrates the limitations of existing methods that they still suffer from overfitting issues. Figure 2(c) shows that both inner training loss and outer training loss are close to the evaluation loss. The evaluation loss is still decreasing after 2k steps and is obviously lower than that of LoRA and AdaLoRA. These results convincingly demonstrate the limitations of existing methods and the effectiveness of BiLoRA in addressing overfitting.

Second, we gain inspiration from DARTS (Differentiable Architecture Search) which utilizes a bi-level optimization for searching network architectures. DARTS uses training dataset and validation dataset separately for training operations and their corresponding proportions in 2 levels. DARTS conducts experiments on comparison between bi-level optimization and the normal training method(single-level). The latter performs much worse and they concludes that the latter “would cause α (analogous to hyperparameters) to overfit the training data, leading to poor generalization”[2]. We gain this inspiration of preventing overfitting and innovatively explore the possibilities of applying bi-level optimization to PEFT domain for addressing the existing limitations of overfitting. Extensive experiments demonstrate the effectiveness of BiLoRA.

Third, one important goal of PEFT is to accelerate the fine-tuning process. The bi-level optimization framework effectively accommodates larger learning rates by iteratively optimizing between the two levels without affecting the training stability. Through BiLoRA, we significantly reduce the training steps and training time needed for convergence, thus making PEFT more efficient.

Fourth, we expect BiLoRA to encourage deeper understanding of low-rank fine-tuning processes and more design potentials. We offer more flexible and direct interaction with singular values and singular matrices at separate levels and design different parameterization of singular values. More efficient methods and deeper insights can be inspired by this bi-level optimization framework.

Reviewer comment (Writing):
It is mentioned in many places that existing methods train "on a single training dataset". "single training dataset" can be confusing here. The proposed BiLoRA even though uses different subsets, it is also still trained on the same single dataset.

Author response:
>Weakness 2: The clarification of "single training dataset".

When we mention that existing methods train "on a single training dataset", we mean training on a single training dataset in a single training process. Compared with this common practice in machine learning, we innovatively leverage the characteristics of bi-level optimization, optimizing low-rank adapters iteratively between upper and lower levels, to address the overfitting issues that arise in existing fine-tuning methods on a single dataset during a single training process. BiLoRA aligns with intuition: by iteratively optimizing between two levels, the model is less prone to overfitting on each of the two sub-datasets and the whole “single dataset”, resulting in enhanced generalization capability and better performance. Extensive experiments on multiple models and datasets strongly support this observation.

Reviewer comment (Evaluation):
Can you compare the distribution of the singular values learned in BiLoRA vs AdaLoRA? This can be helpful to understand BiLoRA more.

Author response:
>Weakness 4: Comparison between the distribution of the singular values learned in BiLoRA and AdaLoRA.

We plot the distribution of singular values learned both in BiLoRA and AdaLoRA. We use the singular values of RoBERTa-base on the CoLA dataset. AdaLoRA implements low-rank adapters to all linear layers including query, key, value, intermediate, layer.output and attention.output while LoRA and BiLoRA only implements low-rank adapters to query and key. We use abs for all the singular values of AdaLoRA. We sincerely refer you to the plots in Figure 5. 

There are some similar features. 1)  A lot of singular values are nearly zero. AdaLoRA gradually prunes ranks to make some of the singular values zero. BiLoRA utilizes bi-level optimization to automatically learn this distribution. 2) There are some important ranks with large singular values.

More results can be analyzed in future work in the distribution of singular values of optimal solutions. We expect BiLoRA can provide flexible interaction with singular values and help us better understand the process of low-rank fine-tuning.

Reviewer comment (Novelty):
It seems that the empirical model performance and training speed are mostly similar as LoRA.

Author response:
**For performance:**

For NLU tasks, our method, BiLoRA, has demonstrated significant performance improvements over baselines on multiple datasets and models. As shown in Table 1 in the paper, using BiLoRA to fine-tune the RoBERTa model with different-sizes on the eight datasets of GLUE surpassed other baselines on almost all datasets. The average performance is significantly higher than other baselines, with 0.7, 0.8, 0.9 margin respectively.. On the DeBERTa-v3 model, AdaLoRA outperformed LoRA by 0.7 on average across the eight datasets. Our method, surpassing LoRA by 1.3, achieved a remarkably superior performance over LoRA and other baselines.
To further demonstrate the performance improvement of BiLoRA, we further conduct experiments on GPT-2 on 2 more NLG datasets, DART and WebNLG.

| BLEU           | WebNLG | DART |
|----------------|--------|------|
| Full Finetune  | 46.5   | 46.2 |
| Adapter L      | 50.2   | 42.4 |
| FTtop2         | 54.9   | 41.0 |
| Prefix         | 36.0   | 46.4 |
| LoRA           | 55.3   | 47.1 |
| BiLoRA         | **56.1**   | **49.0** |

To the best of our knowledge, the results of fine-tuning RoBERTa, DeBERTa and GPT-2 on these NLU and NLG tasks of BiLoRA are the best among all the existing fine-tuning methods. Results on different models and on both NLU and NLG datasets show that our method, BiLoRA outperforms other baselines by a large margin, demonstrating the effectiveness of our method.

**For speed:**

Firstly, we provide the total training steps needed for convergence for LoRA and BiLoRA in the following tablet. We use the results of  RoBERTa-base on MNLI and SST2 datasets. 
| Method/steps   | MNLI       | SST2       |
|----------------|------------|------------|
| Ro(LoRA)       | 184k       | 63k        |
| Ro(BiLoRA)     | 2*15k(1/6×)  | 2*3k(1/11×)  |

Secondly, we provide the total training time training steps needed for convergence for AdaLoRA and BiLoRA in the following tablet. We use the results of  DeBERTa-v3-base on MNLI and SST2 datasets. The time is measured in min.
| Method/Time   | MNLI            | SST2           |
|---------------|-----------------|----------------|
| Dv3(AdaLoRA)     | 753.54          | 240.57         |
| Dv3(BiLoRA)   | 446.21(1/1.7×)  | 56.71(1/4.2×)  |
|               |                 |                |
| **Method/Steps**  | **MNLI**            | **SST2**           |
| Dv3(AdaLoRA)     | 85.9k           | 50.5k          |
| Dv3(BiLoRA)   | 2*15k(3×)       | 2*3k(8×)       |

Results show that BiLoRA uses 6 times/11 times less steps than LoRA for convergence and  that BiLoRA uses 3 times/8 times less steps than AdaLoRA for convergence.
Results demonstrate that BiLoRA can converge much faster than LoRA, AdaLoRA and take much less time for training than baselines. BiLoRA is especially time-efficient on larger datasets such as MNLI, which can reduce the total time of LoRA from 3191 min to 1407 min.

Reviewer comment (Theory):
No theoretical insights are provided for understanding LoRA with bi-level optimization. Specifically, it doesn't seem convincing to replace LoRA with BiLoRA.

Author response:
We’d like to offer some empirical insights and notable advantages of BiLoRA in 2 aspects.
* Limitations of existing fine-tuning methods.
* Inspirations and advantages of bi-level optimization.

First, existing fine-tuning methods can suffer from overfitting. The large number of parameters in pre-trained models may make the full fine-tuning process more prone to overfitting[1]. Existing PEFT methods do not well solve this problem. The generalization capability of a fine-tuned model on the evaluation dataset can reflect the range of overfitting. We plot the training loss and evaluation loss of LoRA, AdaLoRA and BiLoRA to illustrate the generalization capability. We conduct experiments on RoBERTa-base on CoLA dataset and clip the first 2k steps. Figure 2(a) and Figure 2(b) show that for both LoRA and AdaLoRA, the evaluation loss starts to increase at or before 1k step while the training loss still decreases. The gap between the 2 losses is getting huge. This demonstrates the limitations of existing methods that they still suffer from overfitting issues. Figure 2(c) shows that both inner training loss and outer training loss are close to the evaluation loss. The evaluation loss is still decreasing after 2k steps and is obviously lower than that of LoRA and AdaLoRA. These results convincingly demonstrate the limitations of existing methods and the effectiveness of BiLoRA in addressing overfitting.

Second, we gain inspiration from DARTS (Differentiable Architecture Search) which utilizes a bi-level optimization for searching network architectures. DARTS uses training dataset and validation dataset separately for training operations and their corresponding proportions in 2 levels. DARTS conducts experiments on comparison between bi-level optimization and the normal training method(single-level). The latter performs much worse and they concludes that the latter “would cause α (analogous to hyperparameters) to overfit the training data, leading to poor generalization”[2]. We gain this inspiration of preventing overfitting and innovatively explore the possibilities of applying bi-level optimization to PEFT domain for addressing the existing limitations of overfitting. Extensive experiments demonstrate the effectiveness of BiLoRA.

Reviewer comment (Experiments):
How do you decide two datasets D1 and D2?

Author response:
The Dataset Partition, together with learning rate can help keep the balance of inner/outer optimization, which can contribute greatly to preventing the model from overfitting. Lower level has more trainable parameters, so it’s natural to use more data for training singular vectors, while using the left for training singular values. We further experiment on DeBERTa-v3-base on CoLA and SST2 datasets to show the influence of different dataset partitions. We change the inner level dataset partition from 0.6 to 1.0 with 0.1 interval.
| Data Partition | CoLA  | SST2  |
|----------------|-------|-------|
| 0.6            | 68.01 | 94.84 |
| 0.7            | 70.16 | 95.87 |
| 0.8            | **70.52** | 96.02 |
| 0.9            | 70.39 | **96.44** |
| 1.0            | 67.94 | 94.51 |
		                       
Results show that too small partitions(<=0.6) or too large partitions(1.0, only train singular matrix) can harm the overall performance.When the inner partition is too small, singular vectors are not well trained and when the inner partition is 1.0, singular vectors are not trained at all, which can cause a great performance drop. These results can also show that the bi-level optimization is efficient and the two levels are both necessary in preventing overfitting and enhancing performances.
In the paper, we don’t ever change the partition of the data and keep it 8:2. Tuning the partition may further improve the overall performance.

Reviewer comment (Experiments):
The approach is compared against AdaLoRA only in one of the experiments since it is the most relevant baseline for this paper so I'd expected a more thorough comparison against AdaLoRA.

Author response:
In the paper, we compare BiLoRA with AdaLoRA on DeBERTa-v3. To have a more thorough comparison with AdaLoRA, we further conduct experiments on RoBERTa-base on 4 NLU tasks and GPT-2 on 2 NLG tasks. The experiment results can be seen as below, higher is better for all the scores.
First, we compare AdaLoRA with BiLoRA on 4 NLU datasets.
| Method   | MNLI | CoLA | QNLI | QQP | Avg. |
|----------|------|------|------|-----|------|
| AdaLoRA  | 87.2 | 63.4 | 92.8 | 91.0 | 83.6 |
| BiLoRA   | **87.9** | **64.8** | **93.3** | **91.4** | **84.4** |

Second, we compare AdaLoRA on 2 NLG datasets, WebNLG and DART. Results of other methods are from LoRA as a score reference.

| BLEU           | WebNLG | DART |
|----------------|--------|------|
| Full Finetune  | 46.5   | 46.2 |
| Adapter L      | 50.2   | 42.4 |
| FTtop2         | 54.9   | 41.0 |
| Prefix         | 36.0   | 46.4 |
| LoRA           | 55.3   | 47.1 |
| AdaLoRA        | 55.5   | 47.6 |
| BiLoRA         | **56.1**   | **49.0** |

Results on different models and on both NLU and NLG datasets show that our method, BiLoRA outperforms LoRA, AdaLoRA and other baselines by a large margin, demonstrating the effectiveness of our method.

[1] Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers.

[2] Liu H, Simonyan K, Yang Y. Darts: Differentiable architecture search.

Reviewer comment (Reproducibility):
Does BiLoRA also apply iterative pruning of singular values (similar to AdaLoRA)? (the paper seems to talk only about the optimization of pseudo-SVD matrices).

Author response:
In the paper, BiLoRA doesn’t prune singular values. In fact, the pruning process can be applied easily to BiLoRA with only several code lines since singular values are separately in the upper level and can be interacted with directly. 
We expect BiLoRA to be powerful and effective, yet easy to use. The number of hyperparameters for BiLoRA is kept nearly the same as LoRA, which is much less than AdaLoRA. For such simplicity, we don’t apply pruning which will introduce some extra complexities. 
After all, BiLoRA is highly compatible with existing low-rank adaptation methods and we are looking forward to combining other LoRA methods with BiLoRA to achieve higher efficiency and better performance.

Reviewer comment (Experiments):
Table 7 compares the total training time between LoRA and BiLoRA and shows BiLoRA is faster due to faster convergence, what are the total training steps needed for convergence for these two methods? what are the per-update cost differences between these two? Also, can we do the cost comparison with AdaLoRA?

Author response:
Firstly, we provide the total training steps needed for convergence for LoRA and BiLoRA and the per-update cost for the two methods in the following tablet. We use the results of  RoBERTa-base on MNLI and SST2 datasets. The per-update cost is measured in min/k.
| Method/steps   | MNLI       | SST2       |
|----------------|------------|------------|
| Ro(LoRA)       | 184k       | 63k        |
| Ro(BiLoRA)     | 2*15k(1/6×)  | 2*3k(1/11×)  |
|                |            |            |
| **Method/per-update** | **MNLI**     | **SST2**       |
| Ro(LoRA)       | 17.34      | 17.40      |
| Ro(BiLoRA)     | 46.90(2.7×)| 43.33(2.49×)|

Results show that BiLoRA uses 6 times/11 times less steps than LoRA for convergence. The per-step cost for BiLoRA is roughly 2.5 times as LoRA, since BiLoRA needs to iteratively optimize between the two levels and the calculation of outer hypergradients can cost more than simple gradient calculation. Results demonstrate that BiLoRA can converge much faster than LoRA and takes much less time for training than LoRA.

Secondly, we provide the total training steps needed for convergence for AdaLoRA and BiLoRA and the per-update cost for the two methods in the following tablet. We use the results of  DeBERTa-v3-base on MNLI and SST2 datasets. According to AdaLoRA paper, it can be seen that each training epoch of AdaLoRA is longer than LoRA, thus intuitively BiLoRA is also faster than AdaLoRA.   The per-update cost is measured in min/k and time is measured in min.

| Method/Time   | MNLI            | SST2           |
|---------------|-----------------|----------------|
| Dv3(AdaLoRA)     | 753.54          | 240.57         |
| Dv3(BiLoRA)   | 446.21(1/1.7×)  | 56.71(1/4.2×)  |
|               |                 |                |
| **Method/Steps**  | **MNLI**            | **SST2**           |
| Dv3(AdaLoRA)     | 85.9k           | 50.5k          |
| Dv3(BiLoRA)   | 2*15k(1/3×)       | 2*3k(1/8×)       |
|               |                 |                |
| **Method/per-update** | **MNLI**         | **SST2**           |
| Dv3(AdaLoRA)     | 8.77            | 4.76           |
| Dv3(BiLoRA)   | 14.87(1.7×)     | 9.45(2.0×)     |

Results show that BiLoRA uses 3 times/8 times less steps than AdaLoRA for convergence. The per-step cost for BiLoRA is roughly 1.7/2.0 times as AdaLoRA. Results demonstrate that BiLoRA can converge much faster than LoRA, AdaLoRA and take much less time for training than baselines.

Reviewer comment (Experiments):
In the approximately binary parameterization of singular values, does removing the regularization help i.e. just sigmoid over the real values (since this would be similar to real value baseline but with only positive values)?

Author response:
It’s a really good suggestion of just using sigmoid over the real values without regularization. We further conduct experiments on RoBERTa-base on MNLI, SST2 and MRPC datasets.

| Method/Acc     | MNLI | SST2 | MRPC |
|----------------|------|------|------|
| Softmax        | **87.9** | **95.1** | 91.7 |
| Reg_Binary     | 87.6 | 94.8 | 91.4 |
| Binary         | 87.8 | 94.7 | **92.1** |

Results show that using sigmoid over the real values without regularization can achieve equal or better performance than with regularization and achieve better performance than Softmax on MRPC dataset.
The design purpose of Binary with regularization is to automatically assign either a high or low importance to each singular vector pair with the corresponding singular value as zero or one. Since singular values can be seen as importance scores of different ranks, the ranks with zero singular values can be pruned. The whole method can serve as an automatic rank selection mechanism. In the future, we will further explore the potentials of this design and improve the performance.