D y L o RA : Parameter-Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation (2023)

Abstract: With the ever-growing size of pretrained models (PMs), fine-tuning them has become more expensive and resource-hungry. As a remedy, low-rank adapters (LoRA) keep the main pretrained weights of the model frozen and just introduce some learnable truncated SVD modules (so-called LoRA blocks) to the model. While LoRA blocks are parameter-efficient, they suffer from two major problems: first, the size of these blocks is fixed and cannot be modified after training (for example, if we need to change the rank of LoRA blocks, then we need to re-train them from scratch); second, optimizing their rank requires an exhaustive search and effort. In this work, we introduce a dynamic low-rank adaptation (DyLoRA) technique to address these two problems together. Our DyLoRA method trains LoRA blocks for a range of ranks instead of a single rank by sorting the representation learned by the adapter module at different ranks during training. We evaluate our solution on different natural language understanding (GLUE benchmark) and language generation tasks (E2E, DART and WebNLG) using different pretrained models such as RoBERTa and GPT with different sizes. Our results show that we can train dynamic search-free models with DyLoRA at least 4 to 7 times (depending to the task) faster than LoRA without significantly compromising performance. Moreover, our models can perform consistently well on a much larger range of ranks compared to LoRA.

## introduction pre-training/fine-tuning has become a popular paradigm for solving many tasks in natural language processing (nlp) @xcite @xcite and computer vision @xcite @xcite @xcite ). there are many different model compression techniques in the literature for transformer-based models such as matrix factorization @xcite , pruning @xcite , quantization @xcite , and knowledge distillation @xcite @xcite @xcite . there are also different types of pet techniques in the literature such as low-rank adapters @xcite @xcite , and prompt-based techniques @xcite . although model compression solutions are wellestablished in recent years in the literature, applying them to large language models can be very costly, because compression techniques usually need to train (or fine-tune) the original large model. a case in point is knowledge distillation which relies on fine-tuning a large teacher model or even pre-training the student model as suggested in @xcite . moreover, using compression techniques usually leads to degrading the model performance. pets can be alternatives to the compression methods, especially when we would like to use the full capacity of the large pretrained models with light training efforts (such as the language-modelas-a-service scenario @xcite ). among pet techniques, low-rank adapters have received much attention because, in contrast to prompttuning techniques, low-rank adapters do not add to the sequence length, get trained faster, and perform better @xcite . even though there are several low-rank adaptation techniques in the literature, such as adapter @xcite , compacter @xcite , and lora (hu et al., 2021b); they all suffer from two major common problems: first, it is not clear how to select the size of their rank (while their performance is very sensitive to this rank selection); second, their training is static which means that if a low-rank model is trained based on a particular rank size, it will not work well in other rank values (i.e. for any other rank value we need to train a separate model). this paper proposes a dynamic low-rank adapter technique (dylora) to address these two problems. without loss of generality, we focus on lora(hu et al., 2021a) and train lora blocks for a range of ranks instead of a single rank by sorting out the representation learned at different ranks during training. while our model is more flexible, it can outperform lora in a much wider range of ranks without adding to the training time. moreover, our technique does not need extra training for searching across ranks. we summarize our contributions in the following: ## related work this section reviews low-rank adaptation techniques for parameter-efficient tuning and potential existing solutions to make these techniques dynamic and search-free. it has been shown in @xcite ) that for classification tasks such as natural language understanding (nlu), plms have a low intrinsic dimension. this observation motivates the use of low-rank adapters for parameter-efficient tuning. there are several low-rank adapters in the literature such as lora @xcite , adapter @xcite , compacter @xcite , and parallel adapter (pa) @xcite . lora is a low-rank up-projection/down-projection transformation without any non-linearity applied in parallel to key and value attention matrices. the main benefit of lora is that the adapter module, after training, can be integrated into the original weight matrices of the model, which in turn can lead to a very efficient inference time. adapters also have a low-rank up-projection/downprojection transformation with an intermediate nonlinearity. the adapter module is applied in series with the feed-forward network (ffn). having the adaptor module in-line with other blocks in the model can increase the inference time of the model. pa is a faster version of the adapter, which can be applied in parallel with the ffn block. the compactor is a more memory-efficient version of the adapter, which deploys the sum of kronecker products to reconstruct each up-projection and downprojection matrices. all these low-rank adapters suffer from two major issues: first, finding the best rank requires heavy exhaustive training and search; second, the tuned adapter module works well only with a particular rank. while there have been some efforts in the literature towards dynamic networks such as dyn-abert @xcite and gradmax @xcite , to the best of our knowledge, this problem for factorized networks and low-rank adapters is still open. drone @xcite propose a technique for data-aware low-rank model compression however their approach is not search-free, and also, it is not dynamic. dynabert introduces a two-stage method to train width and depth-wise dynamic networks. however, dynabert requires a fine-tuned teacher model on the task to train its sub-networks which makes it unsuitable for pet techniques. gradmax is a technique that gradually adds to the neurons of a network without touching the already trained neurons. but it is unclear how gradmax can be deployed to alleviate the rank-search problem in low-rank adapters. @xcite propose a structured pruning technique called factorized low-rank pruning (flop). flop decomposes weight matrices of a network into the sum of rank-1 components, which are regularized during training to gain sparsity. it is worth mentioning that flop aims at compressing the main model, and even if it can be used for finding a good rank in the lower-rank representation of full-weight matrices, the final low-rank model will not be dynamic (i.e. it is trained well only for one rank and not a range of ranks, same as lora.). in this paper, we propose a new methodology for training lowrank modules for multiple ranks simultaneously rather than training a single-rank adapter at a time (without changing the training budget). inspired by the idea of nested dropout @xcite , we pursue ordering the representations of the bottleneck at the low-rank adapter modules with a new recipe. to the best of our knowledge, it is the first time that the concept of ordering representations has been deployed in training plms. ## our method: dylora in this section, we introduce our solution to get dynamic low-rank adapters that can be trained and deployed well on a range of ranks instead of a single particular rank (with a fixed training budget). this flexibility can free us from searching for the best ranks by training the model multiple times. without loss of generality, we explain our solution on top of lora as one of the prominent low-rank adapter techniques in the literature. in each lora module, we have an up-projection (w @xmath0. let's assume that we would like to train the lora module to operate in the range of @xmath1, r max ] where r min and r max can be treated as new hyper-parameters. to make the lora module work in a range of ranks instead of a single rank, we need to ensure that increasing or decreasing the rank will not significantly hamper the model's performance. one way to implement such behavior would be by sorting the information content of different ranks in the training process of lora modules. in this regard, at each training step, we sample b ∼ p b (.), @xmath2, r min + 1, ..., r max } form a pre-defined categorical distribution (which has a support in range[r min , r max ]) and truncate w dw and w up matrices accordingly. then, the forward pass of this truncated lora module during training will be calculated as following: for simplicity, let's assume that we have only one lora module in the network (the one which is described in eq. 7). let's first consider the regular static loss function (l s ) of the network f (x; w dw , w up ) with w dw and w up tunable parameters for n given input-output pairs (x, y) = bear in mind that, our loss function has a major difference from the nested dropout loss, which makes it more efficient. the nested dropout loss is in the form of rmax @xmath3, y; w dw↓b , w up↓b ) which requires to sum the loss over the entire possible range of ranks and it is computationally expensive. to overcome this computational restriction, we replace it by optimizing the model parameters for each target rank individually at each time step. we show that this scheme quite works well. the other difference with nested dropout is that in the parameter update phase, we add a new mode (so-called frozen) as a hyper-parameter to our training. this new mode suggests to only update the b th corresponding row and column sampled in the truncation phase (i.e. a single row or column will be updated at a time to prevent the learning parameters from being forgotten at previous time steps.). with a minor performance cost, this approach can improve the efficiency of our algorithm even further. ## experiments in this section, we describe the experiments used to evaluate our dylora model on both natural language understanding (nlu) and natural language algorithm 1 dylora -training require: ## conclusion in this paper, we presented our solution dylora to address two problems in low-rank adapters regarding rank selection and making them dynamic. we showed that dylora can select the rank without requiring multiple re-training and is able to make lora dynamic at inference time. as a result, we can avoid the process of searching for the most optimal ranks for many real-life scenarios. it has been demonstrated that dylora performance is comparable with lora, yet we can support a wider range of ranks without adding additional time and effort.