A DAPTIVE B UDGET A LLOCATION FOR P ARAMETERE FFICIENT F INE -T UNING
A BSTRACT
Fine-tuning large pre-trained language models on downstream tasks has become an important paradigm in NLP.
However, common practice fine-tunes all of the parameters in a pre-trained model, which becomes prohibitive when a large number of downstream tasks are present.
Therefore, many fine-tuning methods are proposed to learn incremental updates of pre-trained weights in a parameter efficient way, e.g., low-rank increments.
These methods often evenly distribute the budget of incremental updates across all pre-trained weight matrices, and overlook the varying importance of different weight parameters.
As a consequence, the finetuning performance is suboptimal.
To bridge this gap, we propose AdaLoRA, which adaptively allocates the parameter budget among weight matrices according to their importance score.
In particular, AdaLoRA parameterizes the incremental updates in the form of singular value decomposition.
Such a novel approach allows us to effectively prune the singular values of unimportant updates, which is essentially to reduce their parameter budget but circumvent intensive exact
SVD computations.
We conduct extensive experiments with several pre-trained models on natural language processing, question answering, and natural language generation to validate the effectiveness of AdaLoRA.
Results demonstrate that
AdaLoRA manifests notable improvement over baselines, especially in the low budget settings.
Our code is publicly available at https://github.com/
QingruZhang/AdaLoRA .
I NTRODUCTION
Pre-trained language models (PLMs) have manifested superior performance in various natural language processing tasks (Devlin et al., 2019; Liu et al., 2019;
He et al., 2021b; Radford et al., 2019; Brown et al., 2020).
The most common way to adapt pre-trained models to down-stream tasks is to fine-tune all the parameters (full fine-tuning, Qiu et al.
(2020); Raffel et al.
However, pre-trained models typically incurs large memory footprint.
For example, BERT model (Devlin et al., 2019) consists up to 300 million parameters; T5 (Raffel et al., 2020) comprises up to 11 billion parameters and GPT-3 (Brown et al., 2020) contains up to 175 billion parameters.
When building a NLP system upon these pre-trained models, we usually handle multiple tasks that arrive simultaneously (Radford et al., 2019).
Given a large number of down-stream tasks, full fine-tuning requires that each task maintains a separated copy of large models.
The resulting memory consumption is prohibitively expensive.
To address this issue, researchers have proposed two main lines of research to reduce the fine-tuning parameters, while maintaining or even improving the performance of PLMs.
Specifically, one line of research focuses on adding small neural modules to PLMs and fine-tune only these modules for each task – the base model is kept frozen and shared across tasks.
In this way, only a small number of task-specific parameters are introduced and updated, greatly enhancing the practicality of large models.
For example, adapter tuning (Houlsby et al., 2019; Rebuffi et al., 2017; Pfeiffer et al., 2020;
W q W k
W v W o W f 1 W f 2 88 .88 .
M N L
I
M a t c
h e d
A cc 88.
(a) Selected weight matrix 1,2, 4,5, 7,8, 10,11,
M N L
I
M a t c
h e d
A cc 77.
[Figure]
He et al., 2022) inserts small neural modules called adapters between the layers of the base model.
Prefix tuning (Li & Liang, 2021) and prompt tuning (Lester et al., 2021) attach additional trainable prefix tokens to the input or hidden layers of the base model.
These methods have shown to achieve comparable performance to full fine-tuning, while only updating less than 1% of the original model parameters, significantly releasing the memory consumption.
Another line of research proposes to model the incremental update of the pre-trained weights in a parameter-efficient way, without modifying the model architecture (Zaken et al., 2021;
Guo et al., 2020;
Hu et al., 2022).
Given a pre-trained weight matrix 1 W (0) , for example, diff pruning (Guo et al., 2020) models its incremental update ∆ as a sparse matrix.
Diff pruning initializes ∆ as the same dimension as W (0) and then prunes ∆ element-wise based on the magnitude of the entries.
As such, diff pruning can increase the parameter efficiency substantially by adaptively retaining important updates and pruning unimportant ones.
Nonetheless, diff pruning has several limitations.
First, it relies on low-level implementation to speed up the computation of unstructured sparse matrices, which is not well supported by existing deep learning frameworks.
Therefore, we have to store ∆ as a dense matrix during training.
Second, it needs to update every entry of ∆ with their gradients and then prune them.
This results in similar computational cost as full fine-tuning (Guo et al., 2020).
To overcome these drawbacks, Hu et al.
(2022) propose a method named LoRA, which parameterizes ∆ as a low-rank matrix by the product of two much smaller matrices:
[Equation]
where W (0) ,
∆ ∈ R d 1 × d 2 , A ∈ R r × d 2 and B ∈ R d 1 × r with r ≪ { d 1 , d 2 } .
During fine-tuning, only
A and B are updated.
The rank r is chosen to be much smaller than the dimension of W (e.g., r =when d 1 = d 2 = 1024 ).
With less than 0 .
5% additional trainable parameters, the training overhead can be reduced up to 70% , compared to full fine-tuning.
However, LoRA achieves comparable or even better performance than full fine-tuning (Hu et al., 2022).
Meanwhile, the product of two samll matrices is more friendly to implement and deploy than unstructured sparse matrices in diff pruning.
LoRA still has limitations as it prespecifies the rank r of each incremental matrix ∆ identical.
This ignores the fact that the importance of weight matrices varies significantly across modules and layers when fine-tuning pre-trained models.
To illustrate this point, we present an concrete example in
Figure 1.
We compare the performance of LoRA when fine-tuning specific modules or layers with the same number of trainable parameters.
Figure 1a shows that fine-tuning feed-forward networks (FFN) achieves better performance than self-attention modules.
In addition, Figure 1b demonstrates that weight matrices in top layers are more important than those in bottom layers.
Adding more trainable parameters to the critical weight matrices can lead to better model performance.
In contrast, adding more parameters to those less important weight matrices yields very marginal gains or even hurt model performance.
Given the parameter budget, i.e., the number of total trainable parameters, we always prefer to allocate more parameters to those important modules.
Distributing the budget evenly to all weight matrices/layers, like LoRA and other methods (e.g., adapter and prefix tuning), often gives suboptimal performance.
To this end, a natural question is:
How can we allocate the parameter budget adaptively according to importance of modules to improve the performance of parameter-efficient fine-tuning?
To answer this question, we propose a new method – AdaLoRA (Adaptive Low-Rank Adaptation), which dynamically allocates the parameter budget among weight matrices during LoRA-alike finetuning.
Specifically, AdaLoRA adjusts the rank of incremental matrices to control their budget.
Critical incremental matrices are assigned with high rank such that they can capture more fine-grained and task-specific information.
Less importance ones are pruned to have lower rank to prevent overfitting and save the computational budget.
There are some methods to control the rank of matrices in the existing literature of matrix approximation (Cai et al., 2010; Koltchinskii et al., 2011; Toh & Yun, 2010).
Most of them directly compute singular value decomposition (SVD) of a matrix and then truncate the smallest singular values.
Such an operation can manipulate the rank explicitly and, more importantly, minimize the difference between the resulting matrix and the original matrix.
However, for fine-tuning large models, it becomes prohibitively expensive to iteratively apply SVD for a large number of high-dimensional weight matrices.
Therefore, instead of computing SVD exactly, we parameterize ∆ as ∆ = P Λ Q to mimic SVD.
The diagonal matrix Λ contains singular values while the orthogonal matrices P and Q represent left/right singular vectors of ∆ .
To regularize the orthogonality of P and Q , an additional penalty is added to training loss.
Such a parameterization avoids the intensive computations of SVD.
Besides, another advantage is that we only need to drop the unimportant singular values while the singular vectors are maintained.
This preserves the possibility of future recovery and stabilizes the training.
See a detailed comparison to LoRA in Section 3.
Based on our SVD parameterization, AdaLoRA dynamically adjusts the rank of ∆ =
PV Q by importance scoring .
Specifically, we divide the incremental matrix P Λ Q into triplets, where each triplet G i contains the i -th singular value and the corresponding singular vectors.
To quantify the importance of triplets, we propose a novel importance metric, which takes account of the contribution of every entry in G i to the model performance (Sanh et al., 2020; Liang et al., 2021; Zhang et al., 2022).
Triplets with low importance scores are granted low priority and hence the singular values are zeroed out.
Triplets with high importance are retained for fine-tuning.
Moreover, we also propose a global budget scheduler to facilitate the training.
In particular, we start from an initial parameter budget, which is slightly higher than the final budget, and then gradually reduce it until matching the target.
Such a scheduler can improve the training stability and model performance.
Please see
Section 3 for a detailed description of our importance metric and budget scheduler.
We conduct extensive experiments on a wide range of tasks and models to demonstrate the effectiveness of AdaLoRA.
Specifically, we evaluate the performance using DeBERTaV3-base (He et al., 2021a) on natural language understanding (GLUE, Wang et al.
and
question answering (SQuADv1, Rajpurkar et al.
(2016) and SQuADv2, Rajpurkar et al.
datasets.
We also apply our methods to BART-large (Lewis et al., 2019) and evaluate the performance on natural language generation (XSum, Narayan et al.
and CNN/DailyMail, Hermann et al.
(2015)) tasks.
We show AdaLoRA consistently outperforms the baseline, especially under low budget settings.
For example, with less than 0 .
1% trainable parameters of full fine-tuning, AdaLoRA achieves a 1.2% F improvement on the SQuAD2.0 dataset compared with state-of-the-art approaches.
B ACKGROUND
Transformer-based Models.
A typical transformer model consists of L stacked blocks, where each block contains two submodules: a multi-head attention (MHA) and a fully connected FFN.
Given the input sequence X ∈ R n × d , MHA performs the attention function in parallel h heads:
[Equation]
where W o ∈ R
d × d is an output projection and W q i , W k
i , W v i ∈
R
d × d h are query, key and value projections of head i .
d h is typically set to d/h .
The other important module is a FFN which consists of two linear transformations with a ReLU activation in between: FFN ( X ) =
ReLU ( XW f 1 + b 1 ) W f 2 + b 2 , where W f 1 ∈ R d × d m and W f 2 ∈ R
d m × d .
Finally, a residual connection is used followed by a layer normalization (Ba et al., 2016).
Low Rank Adaptation.
LoRA (Hu et al., 2022) models the incremental update of the pre-trained weights by the product of two small matrices.
For h = W (0) x , the modified forward pass is:
[Equation]
where W (0) ,
∆ ∈ R d 1 × d 2 , A ∈ R r × d 2 and B ∈ R d 1 × r with r ≪ { d 1 , d 2 } .
A typically adopts a random Gaussion initialization while B is initialized with zero to have ∆ = 0 at the beginning of training.
We further denote A i ∗ as the i -th row of A , B ∗ i as the i -th column of B , and G i =
{ A i ∗ , B ∗ i } as the i -th doublet.
Hu et al.
(2022) only apply LoRA to query and value projections (i.e, W q and
W v ) in the MHAs.
He et al.
(2022) extend it to weight matrices of FFNs (i.e, W f 1 and W f 2 ), leading to the performance improvement .
Meanwhile, they propose a unified view of various efficient tuning methods including adapter tuning, prefix tuning and LoRA.
A DA L O RA M ETHOD
Our method contains two important components: (i) SVD-based adaptation, which formulates the incremental matrices in the form of singular value decomposition; (ii) Importance-aware rank allocation, which prunes redundant singular values based on our newly-designed importance metric.
3. SVD-B ASED A DAPTATION
As mentioned in Section 1, we propose to parameterize the incremental updates of the pre-trained weight matrices in the form of singular value decomposition:
[Equation]
where P ∈ R d 1 × r and Q ∈ R
r × d 2 represent the left/right singular vectors of ∆ and the diagonal matrix Λ ∈ R r × r contains the singular values { λ i } 1 ≤ i ≤ r with r ≪ min( d 1 , d 2 ) .
We further denote
G i =
{ P ∗ i , λ i , Q i ∗ } as the triplet containing the i -th singular value and vectors.
In practice, since Λ is diagonal, we only need to save it as a vector in R r .
Λ is initialized with zero while P and Q adopt a random Gaussian initialization to ensure ∆ = 0 at the beginning of training.
To enforce the orthogonality of P and Q , i.e., P ⊤ P =
QQ ⊤
=
I , we utilize the following regularizer 2 :
[Equation]
In our method, Λ is iteratively pruned to adjust the rank after each gradient decent step.
As mentioned in Section 1, one can directly compute SVD for every ∆ to manipulate singular values.
The computational complexity, however, is O (min( d 1 , d 2 ) d 1 d 2 ) .
It becomes extremely expensive to iteratively apply SVD for a large number of high-dimensional incremental matrices.
In contrast, our parameterization avoids intensive SVD computation, greatly releasing the computational overhead.
We remark that one can also apply structured pruning to LoRA to control the rank (i.e., prune BA doublet-wise in (1)), whereas it has the following disadvantages.
First, when a doublet is measured as unimportant, we have to prune all of its elements.
It makes scarcely possible to reactivate the pruned doublets as their entries are all zeroed out and not trained.
In contrast, AdaLoRA only masks out the singular values based on (3) while the singular vectors are always maintained.
It preserves the potential of future recovery for the triplets dropped by mistake.
Second, A and B of LoRA are not orthogonal, meaning the doublets can be dependent with each other.
Discarding the doublets can incur larger variation from the original matrix than truncating the smallest singular values.
Therefore, the incremental matrices are often altered dramatically after each step of rank allocation, which causes training instability and even hurts generalization.
To demonstrate this point, we present an ablation study in Section 4.4, which compares AdaLoRA with structured pruning for LoRA.
3. I MPORTANCE - AWARE R ANK A LLOCATION
We apply the SVD-based adaptation (3) to every weight matrix including W q , W k , W v , W f 1 and
W f 2 of each transformer layer.
In order to control the budget, we iteratively prune singular values in correspondence to their importance score during the training.
For clear reference, we use k to index the incremental matrix, i.e., ∆ k
=
P
k
Λ
k Q k for k = 1 , . .
. , n , where n is the number of adapted weight matrices.
We denote the i -th triplet of ∆ k as G k,i = { P k, ∗ i , λ k,i , Q k,i ∗ } and its importance score as S k,i .
We further denote the parameter sets P
= { P k } nk =1 , E = { Λ k } nk =1 ,
Q = { Q k } nk =1 and training cost as C ( P , E , Q ) .
With the regularization (4), the training objective is given by
L ( P , E , Q ) =
C ( P , E , Q ) + γ (cid:80) n k =1 R
( P k , Q k ) , where γ > 0 is the regularization coefficient.
At the t -th step, we first take a stochastic gradient step to update P ( t ) k , Λ ( t ) k and Q ( t ) k for k = 1 , . . . , n .
Specifically, for Λ ( t ) k
[Equation]
where η > 0 is learning rate.
Then, given importance score S ( t ) k , the singular values are pruned following [Equation] S
[Equation]
is in the top- b (
[Equation]
of S ( [Equation] ) , otherwise, (6) where S ( t ) = { S ( t ) k,i } 1 ≤ k ≤ n, 1 ≤ i ≤ r contains the importance score of all triplets.
Here b ( t ) is the budget of remaining singular values at the t -th step, which we explain more in Section 3.3.
In this way, we leave more budget to the incremental matrices of higher priority by pruning the singular values of less important ones.
In the sequel, we introduce several options to design the importance score.
Magnitude of singular values is the most straightforward way to quantify the importance of every triplet, i.e., S k,i = | λ k,i | .
In this way, only the least significant singular values are discarded.
It minimizes the deviation from the original matrix and further stabilizes the training.
Many existing methods use this criterion to control the rank of matrix (Cai et al., 2010; Koltchinskii et al., 2011;
Toh & Yun, 2010).
However, we remark that such a simple metric cannot properly quantify the contribution of parameters to model performance.
Sensitivity-based importance is another option for importance scoring, which quantifies the sensitivity of parameters to the training loss (Molchanov et al., 2019; Sanh et al., 2020; Liang et al., 2021;
Zhang et al., 2022).
The prior work, however, leverages the sensitivity to quantify the importance of single entries and applies it for unstructured pruning that prunes weights element-wise.
When it turns to our case, we have to design a new metric as the triplets are discarded group-wise.
Every entry’s sensitivity ought to be considered and properly combined to quantify the overall contribution of the triplet to model performance.
Therefore, we propose a newly-designed importance metric in account of both the singular value and vectors in triplet G k,i :
[Equation]
where we calculate the mean importance of P k, ∗ i and Q k,i ∗ such that S k,i does not scale with the number of parameters in G k,i .
Here s ( · ) is a specific importance function for single entries.
We can adopt the sensitivity for s ( · ) , which is defined as the magnitude of the gradient-weight product:
[Equation]
where w ij is any trainable parameter.
(8) essentially approximates the change in loss when a parameter is zeroed out.
If the removal of a parameter has a large influence, then the model is sensitive to it and we should retain it (Molchanov et al., 2019; Liang et al., 2021; Zhang et al., 2022).
However, Zhang et al.
(2022) point out that the sensitivity in (8) is not yet a reliable importance indicator.
Such a score is estimated on the sampled mini batch.
The stochastic sampling and complicated training dynamics incur high variability and large uncertainty for estimating the sensitivity with (8).
Therefore, Zhang et al.
(2022) propose to resolve this issue by sensitivity smoothing and uncertainty quantification:
[Equation] [Equation]
10) where 0 < β 1 , β 2 < 1 .
I ( t ) is the smoothed sensitivity by exponential moving average and U ( t ) is the uncertainty term quantified by the local variation between I ( t ) and I ( t ) .
Then they define the importance as the product between I ( t ) and U ( t ) , which can be another option for s ( · ) :
[Equation]
We present a detailed ablation study in Section 4.4 to compare the performance of different importance metrics.
We find the proposed metric (7) based on the sensitivity variant (11) generally performs best.
We summarize the detailed algorithm in Algorithm 1 presented in Appendix A.
3. G LOBAL B UDGET S CHEDULER
As mentioned in Section 1, adjusting the rank is naturally to control the parameter budget in the context of low-rank adaptation.
Hence we define the budget b ( t ) as the total rank of all incremental matrices, i.e., the number of total singular values.
Recall that the budget allocation is iteratively conducted during the fine-tuning.
To facilitate the training, we propose a global budget scheduler.
Specifically, we start from an initial budget b (0) that is slightly higher than the target budget b ( T ) (e.g., 1.5 times of b ( T ) ).
We set the initial rank of each incremental matrix as r = b (0) /n .
We warm up the training for t i steps, and then follow a cubic schedule to decrease the budget b ( t ) until it reaches b ( T ) .
Finally, we fix the resulting budget distribution and fine-tune the model for t f steps.
The exact equation for the budget schedule is presented in Appendix B.
This allows AdaLoRA to explore the parameter space first and then focus on the most important weights later.
E XPERIMENTS
We implement AdaLoRA for fine-tuning DeBERTaV3-base (He et al., 2021a) and BART-large (Lewis et al., 2019).
We evaluate the effectiveness of the proposed algorithm on natural language understanding (GLUE, Wang et al.
(2019)), question answering (SQuADv1, Rajpurkar et al.
(2016) and SQuADv2, Rajpurkar et al.
(2018)), and natural language generation (XSum, Narayan et al.
(2018) and CNN/DailyMail Hermann et al.
All the gains have passed significant tests with p < 0 .
Implementation Details.
We use PyTorch (Paszke et al., 2019) to implement all the algorithms.
Our implementation is based on the publicly available Huggingface Transformers 3 (Wolf et al., 2019) code-base.
All the experiments are conducted on NVIDIA V100 GPUs.
LoRA scales ∆ x by α/r where α is a constant in r .
As a result, the magnitude of output can be consistent given different r .
It reduces the efforts of retuning learning rate when varying r .
Typically α is set as 16 or 32 and never tuned (Hu et al., 2022; Yang & Hu, 2020).
Following LoRA, we add the same scaling for (3) and fix α as LoRA.
Besides, in Algorithm 1, we prune singular values every ∆ T steps (e.g., ∆ T = 100 ) such that the pruned triplets can still get updated within these intervals and possibly reactivated in future iterations.
Baselines.
We compare AdaLoRA with the following methods:
• Full fine-tuning is the most common approach for adaptation.
During fine-tuning, the model is initialized with pre-trained weights and biases, and all model parameters undergo gradient updates.
• Bitfit (Zaken et al., 2021) is an effective parameter-efficient fine-tuning method.
The method only fine-tunes bias vectors in the pre-trained model.
• Adapter tuning (Houlsby et al., 2019; Pfeiffer et al., 2020) inserts two-layer adapters between transformer blocks.
We compare with two types of adapter.
Houlsby adapter as proposed in Houlsby et al.
(2019) is inserted between the self-attention module and the FFN module followed by a subsequent residual connection.
Recently, Pfeiffer et al.
(2020) propose a more efficient design with adapters only applied after FFN modules and LayerNorm modules (Ba et al., 2016), which we call Pfeiffer adapter .
The number of trainable parameters is determined by the number of layers, the hidden dimension of adapters and the dimension of their inputs.
• LoRA
(Hu et al., 2022) is a state-of-the-art method for parameter-efficient fine-tuning.
The method parameterizes incremental updates by two small matrices and only fine-tune them.
The number of trainable parameter is controlled by the rank r and the number of adapted weight matrices n .
Hu et al.
(2022) apply LoRA to query and value projections only.
In empirical, we find that applying LoRA to all weight matrices, i.e., W q , W k , W v , W f 1 and W f 2 , can further improve its performance (Please see Appendix H).
Hence, we compare with this generalized LoRA to maximize its performance.
We use publicly available implementation 4 to run all the baselines.
Please refer to Hu et al.
and reference therein for details.
[Table]
4. N ATURAL L ANGUAGE U NDERSTANDING
Models and Datasets.
We evaluate the fine-tuning performance of DeBERTaV3-base (He et al., 2021a) using the proposed algorithm.
We conduct experiments on the General Language Understanding Evaluation (GLUE, Wang et al. 2019) benchmark.
The benchmark includes two single-sentence classification tasks, three similarity and paraphrase tasks and four natural language inference tasks.
Dataset details are summarized in Appendix D.
Implementation Details.
DeBERTaV3-base consists of 183 millions parameters.
We compare
AdaLoRA with the baselines under different budget levels, for example, given the total trainable parameters as 0.3/0.6/1.2 million.
In order to match the parameter budget, we select the hidden dimensions of adapters from { 8 , 16 , 32 , 64 } , set the rank r of LoRA as { 2 , 4 , 8 } , and choose the final budget b ( T ) of AdaLoRA from { 144 , 288 , 576 } .
Then we set b (0) as 1.5 times of b ( T ) for
AdaLoRA and select the regularization coefficient γ from { 0 . 1 , 0 . 3 , 0 . 5 } .
We set the exponential moving average parameters β 1 and β 2 as their default value 0 . 85 .
We select the learning rate from { 5 × 10 − 5 , 8 × 10 − 5 , 1 × 10 − 4 , 2 × 10 − 4 } .
More details are presented in Appendix E.
Main results.
We compare AdaLoRA with the baseline methods under different budget settings.
Table 1 shows experimental results on the GLUE development set.
We see that AdaLoRA achieves better or on par performance compared with existing approaches on all datasets under all budget levels.
For example, when the parameter budget is 0.3M, AdaLoRA achieves 87.36% accuracy on
RTE, which is 1.8% higher than the best-performing baseline.
Besides, AdaLoRA with extreme low budget can often perform better than the baselines with higher budget.
For example, AdaLoRA achieve 70.04% Mcc.
score on CoLA with 0.3M fine-tuning parameters, which is higher than all baseline methods with lager budget (e.g., 0.6M and 1.2M).
4. Q UESTION A NSWERING
Models and Datasets.
We evaluate performance of the proposed algorithm on two question answering (QA) datasets: SQuAD v1.1 (Rajpurkar et al., 2016) and SQuADv2.0 (Rajpurkar et al., 2018), where we use AdaLoRA to fine-tune DeBERTaV3-base.
These tasks are treated as a sequence labeling problem, where we predict the probability of each token being the start and end of the answer span.
Dataset details can be found in Appendix F.
Implementation Details.
We compare AdaLoRA with the baseline methods under different parameter budgets.
That is we have the number of trainable parameters as 0 .
65% of total pre-trained parameters.
To match the budget requirements, we select the hidden dimensions of adapters from { 4 , 8 , 16 , 32 , 64 } , set the rank r of LoRA as { 1 , 2 , 4 , 8 } and choose the final total rank b ( T ) of AdaLoRA from { 72 , 144 , 288 , 576 } .
We set the batch size as 16 .
We use AdamW (Loshchilov & Hutter, 2019) as the optimizer and we set the learning rate as 1 × 10 − 3 for AdaLoRA.
Please refer to Appendix F for more details.
Main Results.
Table 2 summarizes experimental results when we fine-tune DeBERTaV3-base under 4 different budget settings: 0.08%, 0.16%, 0.32% and 0.65% of total pre-trained parameters.
From the
[Table]
result, we see that AdaLoRA consistently outperforms existing approaches under all the budget levels in term of two evaluation metrics: exact match (EM) and F1.
Notice that the performance of Houlsby adapter and Pfeiffer adapter are notably decreased when we reduce the parameter budget.
In contrast, our method shows the consistent performance under different budget levels.
For example, AdaLoRA achieves 88.7% F1 on SQuADv2.0 with the smallest budget 0.08%.
It is close to its performance under the high budget and it is also 1.2% higher than the best-performing baseline.
4. N ATURAL L ANGUAGE G ENERATION
[Table]
Models and Datasets.
To provide a comparison with the state-of-the-art in natural language generation (NLG) tasks, we apply AdaLoRA to fine-tune a BART-large model (Lewis et al., 2019).
We evaluate model performance on two datasets: XSum (Narayan et al., 2018) and CNN/DailyMail (Hermann et al., 2015).
Implementation Details.
Similarly as DeBERTav3-base [Table] we apply low-rank/SVD-based adaptation to every weight matrix of both encoder and decoder layers.
We report ROUGE 1/2/L scores (R-1/2/L,
Lin (2004)).
We set the training epochs as 15.
For XSum, we set the beam length as 8 and batch size as 64.
For CNN/DailyMail, we set the beam length as 4 and batch size as 32.
Please see Appendix G for the detailed configuration.
Main Results.
Experimental results are summarized in Table 3, where we compare the fine-tuning performance under four budget levels: the number of trainable parameters is 0.13%, 0.26%, 1.10% and 2.20% of total pre-trained parameters.
We see that AdaLoRA achieves better or on par performance compared with the baseline on both datasets (XSum and CNN/DailyMail) under all the budget levels.
For example, AdaLoRA achieves 21.13 R-2 score when budget level is 1.10%, compared with 19. for LoRA.
4. A NALYSIS
Different budget levels.
Figure 2 illustrates experimental results of fine-tuning DeBERTaV3-base under different budget levels.
We see that on all the three datasets (MNLI-m, SQuADv2.0 and XSum),
AdaLoRA achieves consistent performance improvement under all the budget levels compared with the baseline.
The performance gain is more significant when increasing the budget for the XSum
[Figure]
task, suggesting a high budget can help NLG tasks.
Note that on the MNLI and SQuADv2.0 datasets, the performance of AdaLoRA under low budget levels ( ≤ 1% ) can match the results of high budget settings.
For example, AdaLoRA achieves 88 .
78% F1 on SQuADv2.0 when the budget is 0 .
It is close to the performance (88.89% F1) of the highest budget ( 4 . 65% ) with a more significant gain over the baseline.
Comparison to low-rank parameterization.
As mentioned in Section 3.1, one can alternatively prune LoRA doublet-wise to conduct the rank allocation.
In this case, the doublets are zeroed out entirely, raising the barrier to reactivate them.
It can cause training instability and hurt the generalization when some crucial doublets are pruned by mistake.
In Table 4, we compare AdaLoRA with pruning LoRA on three datasets (SST-2, RTE, and CoLA) to illustrate this point.
We apply the same importance score, budget scheduler and training setups as Section 4.1 for pruning LoRA.
We can see that AdaLoRA outperforms pruning LoRA on all the datasets under all the budget levels
[Table]
Variants of the importance score.
Recall that in AdaLoRA, the importance score is defined by the sensitivity and uncertainty of every entry in the triplet (7).
In Table 4, we examine two variants of the importance score: (i) changing s ( · ) in (7) to sensitivity-only; (ii) directly defining S i as | λ i | .
From the results
[Table]
we can see that the proposed importance score generally performs best.
The other two variants can degenerate the model performance up to 0 .
The resulting budget distribution.
Figure 3 in Appendix C shows the resulting rank of each incremental matrix of DeBERTaV3-base fine-tuned with AdaLoRA.
We find that AdaLoRA always prefers to allocating more budget to FFNs and top layers.
Such behavior aligns with our empirical conclusions presented in Figure 1 that weight matrices of FFN moduels and top layers are more important for model performance.
Hence, it validates that our proposed importance metric can guide
AdaLoRA to focus on crucial modules.
Meanwhile, the rank distribution generated by AdaLoRA is consistent across different budget levels, tasks and models.
It means the number of remaining parameters is linearly scaled with b ( T ) and hence we can tune b ( T ) to control the remaining parameters.
C ONCLUSION
We propose a parameter-efficient fine-tuning method – AdaLoRA that adaptively allocates the parameter budget according to importance scoring.
In AdaLoRA, we parameterize the incremental updates of weight matrices in the form of singular value decomposition.
Then, we dynamically allocate the parameter budget among incremental matrices by manipulating the singular values based on a new importance measurement.
Such an a pproach effectively improves the model performance and parameter efficiency.
We conduct extensive experiments on natural language processing, question answering and natural language generation tasks.
Results show that AdaLoRA outperforms existing approaches.
A T HE D ETAILED A LGORITHM Algorithm
do Compute S k = P ( t )
k − η ∇ P k L ( P , E , Q ) and Q ( t +1) k
= Q ( t )
k − η ∇ Q k L ( P , E , Q ) ; 8:
Λ ( t +1) k
= T (Λ k − η ∇ Λ for 10: Output:
As mentioned in Section 3.3, we propose a global budget scheduler to gradually decrease the budget b ( t ) following a cubic schedule.
The detailed equation is given as follows:
[Equation]
[Equation]
[Equation]
C T HE B UDGET D ISTRIBUTION
12 Layer
W f W f W o
W v W k W q
T h
e ﬁ n a l r a nk
Figure 3 shows the resulting rank of each incremenal matrix when fine-tuning DeBERTaV3-base on MNLI with AdaLoRA.
We can see that AdaLoRA allocates more parameter budget to weight matrices of 8, 9, 10, and 11 layers, especially for W f 1 and W v .
Such behavior aligns with our conclusions presented in Figure 1.
D GLUE D ATASET S TATISTICS
We present the dataset statistics of GLUE (Wang et al., 2019) in the following table.
[Table]
E N ATURAL L ANGUAGE U NDERSTANDING E. B UDGET C ONFIGURATION
For each budget level [Table] we tune the final budget b ( T ) for AdaLoRA, the rank r for LoRA, the hidden dimension d for two adapters to match the budget requirements.
[Table]
Alternatively, we can also set the final average rank ¯ r ( T ) =
b ( T ) /n for AdaLoRA to control the budget, which is set as 2, 4, and 8 given the final budget as 144, [Table] and 576 respectively.
Then we select the initial rank r from { 4 , 6 , 12 } for the final average rank { 2 , 4 , 8 } respectively.
E. T RAINING D ETAILS
We tune the learning rate from { 8 × 10 − 5 , 5 × 10 − 5 , 3 × 10 − 5 , 1 × 10 − 4 , 3 × 10 − 4 , 5 × 10 − 4 , 8 × 10 − 4 , 1 × 10 − 3 } and pick the best learning rate for every method.
For each dataset, the batch size is set as identical for every method.
[Table]
F Q UESTION A NSWERING F. B UDGET C ONFIGURATION
Given the budget, we control the trainable parameters for each method as the following table.
[Table]
F. T RAINING D ETAILS
We set the batch size as 16.
We select the learning rate from { 8 × 10 − 5 , 5 × 10 − 5 , 3 × 10 − 5 , 1 × 10 − 4 ,× 10 − 4 , 5 × 10 − 4 , 8 × 10 − 4 , 1 × 10 − 3 } and pick the best-performing learning rate for every method.
The configuration of AdaLoRA is listed in the following table.
[Table]
F. D ATASET
The statistics of question answering datasets are summarized in Table [Table]
[Table]
G N ATURAL L ANGUAGE G ENERATION G. B UDGET C ONFIGURATION
Given the budget
[Table]
we control the trainable parameters for each method as the following table.
[Table]
G. T RAINING D ETAILS
We set the batch size as 16.
We select the learning rate from { 8 × 10 − 5 , 5 × 10 − 5 , 3 × 10 − 5 , 1 × 10 − 4 ,× 10 − 4 , 5 × 10 − 4 , 8 × 10 − 4 , 1 × 10 − 3 } and pick the best-performing learning rate for every method.
The configuration of AdaLoRA is listed in the following table.
[Table]
H A BLATION S TUDY FOR L O RA
As mentioned in Section
[Table]
we find that the performance of LoRA can be further improved when applying it to every weight matrix, compared to fine-tuning W q and W v only (Hu et al., 2022).
This observation aligns with the empirical results of He et al.
In Table [Table] we follow the same training configuration as Section 4.1 and present an ablation study to illustrate this point. [Table]
I O RTHOGONAL R EGULARIZATION
[Figure]
To verify the effectiveness of (4), we plot ∥ P ⊤ P − I ∥ 2 F and ∥ QQ ⊤ − I ∥ 2 F to show whether P and Q are regularized to be orthogonal.
We fine-tune a DeBERTaV3-base model on SST-2 with AdaLoRA and follow the same training configuration as Section 4.1.
We set γ as 0.1 and plot the two terms along the training horizon.
From Figure 4, we can see that two regularization terms can be optimized to a very small value (e.g., 0.001) at the beginning of training.
Therefore, both P and Q can be enforced to be orthogonal quickly during the initial warm-up of AdaLoRA.
It ensures that the triplets are not dependent with each other.
J T HE ROLE OF TWO COMPONENTS
We remark that both two components of our method - SVD adaptation and adaptive budget allocation, play vital roles for the performance gain.
To demonstrate it, we compare AdaLoRA with the following variants: (i) SVD-LoRA: fine-tuning only with the proposed SVD-based adaptation in (3) and (4); (ii) LoRA regu : LoRA with orthogonal regularization (4) on A and B ; (iii)
AdaLoRA γ = 0 :
AdaLoRA without orthogonal regularization (4).
Table 14 present the results when fine-tuning
DeBERTaVe-base on SST-2 and MNLI.
We can see that fine-tuning only with SVD adaptation shows an improvement over LoRA but cannot match the performance of AdaLoRA.
Meanwhile, without
SVD orthogonal regularization, the performance of AdaLoRA can degenerate.
These results validate that both components contribute to the model performance.
[Table]
K C OMPARISON OF T RAINING C OST
We compare the training cost between AdaLoRA and LoRA in the following table.
We use two methods to fine-tune DeBERTaV3-base on a single NVIDIA V100 GPU.
We do training only and set hyperparameters, e.g., batch size and training epochs, the same as in Section 4.
[Table]
Table 15 shows that MARVEL incurs 11% additional training time on MNLI and 16% on SQuADv under different budgets.
The memory footprint of two methods are quite close.
Such results demonstrate that MARVEL does not incur significant training overheads.
The reason behind is that we only evaluate the importance score for small incremental matrices P Λ Q .
Their total number of parameters is usually less than 1% of pre-trained weights.
Therefore, it does not lead to significant computational cost to update the importance scores of these well-structured small matrices, compared to forward-backward pass of full model
[Table]