PRIL o RA : Pruned and Rank-Increasing Low-Rank Adaptation (2024)

Abstract: With the proliferation of large pre-trained language models (PLMs), fine-tuning all model parameters becomes increasingly inefficient, particularly when dealing with numerous downstream tasks that entail substantial training and storage costs. Several approaches aimed at achieving parameter-efficient fine-tuning (PEFT) have been proposed. Among them, Low-Rank Adaptation (LoRA) stands out as an archetypal method, incorporating trainable rank decomposition matrices into each target module. Nevertheless, LoRA does not consider the varying importance of each layer. To address these challenges, we introduce PRILoRA, which linearly allocates a different rank for each layer, in an increasing manner, and performs pruning throughout the training process, considering both the temporary magnitude of weights and the accumulated statistics of the input to any given layer. We validate the effectiveness of PRILoRA through extensive experiments on eight GLUE benchmarks, setting a new state of the art.

## introduction the current paradigm for natural language processing tasks is to exploit pre-trained models, which were trained using large amounts of data and expensive resources, and fine-tune them to various downstream tasks @xcite @xcite @xcite . such fine-tuning was traditionally conducted by gradient update of all parameters of the model @xcite @xcite . with the ever increasing size of models, such as llama 7b-65b @xcite , palm 540b @xcite , and others, trained with resources consisting of hundreds of gpus in parallel, which are available only to some institutions and corporations, full finetuning can become prohibitive, lengthy, and with high carbon footprint @xcite . additionally, fully fine-tuning this way requires storing all parameters of the fine-tuned model for every downstream task. to tackle the aforementioned challenges, a few research directions for parameter-efficient fine-tuning (peft) were proposed. these directions aim to maintain or even improve the accuracy of a full fine-tuning approach, while training only a small fraction of the parameters. one approach is to add small modules to the base model, which is kept frozen throughout the training process. such adapter tuning techniques @xcite @xcite add modules between the layers. the implication, due to increased model depth, is longer training time and higher latency during inference. alternatively, prompt and prefix tuning @xcite attach trainable tokens to the beginning of layers in the model, thus potentially reducing its effective maximal token length. lora @xcite fine-tunes linear layers by viewing each layer as a matrix of weights w 0 , freezing it, and adding to it a small rank matrix, with the same shape as the original weight matrix, that is obtained as a product of two low-rank matrices a and b. the low-rank r is chosen to be much smaller than the input dimension to the layer, thereby significantly reducing the number of trainable parameters. during lora training, only the two low-rank matrices are updated, which are usually 0.01% to 1.00% of the original parameter count, depending on the low-rank of the two matrices. in addition to being efficient and often exceeding the performance of full fine-tuning @xcite , this method has the advantage of being able to be merged back to the original matrix during inference, without increasing latency. lora has been used in various downstream tasks successfully @xcite @xcite one limitation of lora is that the low-rank r is an arbitrarily set parameter, and in the original lora it is set to be fixed across layers and weights. efforts were made to address the issue of the fixed rank of lora. adalora @xcite starts from an initial parameter budget, which is slightly higher than the final budget, and then gradually reduces it until matching the target by removing weights based on svd. in this work, we encourage the usage of linearly increasing the rank from one layer to the next while concurrently adhering to the same budget of parameters. as we show, this strategy provides a distribution of the learned parameters that is better than a uniform placement, or even the learned alternatives. a second contribution is obtained by pruning matrix a. this is done by considering both the elements of a and an exponential moving average over the layer's input. although we prune, in most cases, half of the elements of a, the main metric we seek to improve by pruning is the overall accuracy obtained after pruning. we conduct extensive experiments over eight different general language understanding evaluation @xcite benchmarks, and present evidence that the proposed method outperforms lora and its recent variants, that both the linear distribution of ranks and the specific pruning approach are beneficial, and that the method does not require more gpu memory or training time than the conventional lora, unlike recent extensions of lora. ## related work in recent years, parameter efficient fine-tuning (peft) has garnered increasing interest among researchers as a means to reduce both the expenses associated with fine-tuning and storing large-scale pre-trained models and the time required for training. various approaches have emerged, each exhibiting distinct characteristics pertaining to memory utilization, storage requirements, and computational overhead during inference. these approaches can be classified into two primary categories, namely, selective and additive peft methods, based on whether the original model parameters undergo fine-tuning during the training phase. selective methods involve the selection and modification of a model based on its original pa-rameters. an early instance of this concept was observed in the fine-tuning of only a subset of the top layers of a network, as demonstrated by @xcite , and by more recent work @xcite . in more recent developments, various approaches have been proposed, each targeting specific layers or internal modules of the model. for instance, the bitfit method @xcite updates only the bias parameters, resulting in a substantial reduction in the number of trainable parameters, but at the cost of suboptimal performance. other methods use a scoring function when selecting trainable parameters @xcite @xcite , while others select top parameters based on a fisher information calculation @xcite . additive methods represent an alternative to full-parameter fine-tuning by introducing additional trainable parameters into the backbone network. adapters are a type of trainable component initially applied in the context of multi-domain image categorization by @xcite , that were subsequently integrated into transformer networks, specifically in the attention and feedforward layers @xcite . prefix-tuning and prompt-tuning (li and liang, 2021; @xcite involve the addition of trainable parameters preceding the sequence of hidden states across all layers. lst (ladder side-tuning) @xcite operates by short-cutting hidden states from the original network into a compact trainable side network, eliminating the need for backpropagating gradients through the backbone network. lora @xcite emulates the adjustment of the weight matrix in the model through the multiplication of two low-rank matrices. notably, the trained parameters resulting from this process can be incorporated seamlessly into the original network during the inference phase without incurring additional computational overhead. recently, hybrid approaches have emerged, combining the selective and additive methods and presenting a unified framework @xcite @xcite . other methods are based on the hypothesis that parameter redundancy exists in peft modules, therefore pruning the trainable parameters to achieve superior finetuning performance @xcite . network pruning methods @xcite @xcite @xcite reduce the size of the network by removing or shrinking matrices from the network, which effectively is equivalent to setting them to zero. such methods require further full re-training, or other computationally intensive iterations. magnitude pruning @xcite removes individual parameter weights when the magnitude is below a certain threshold. the threshold is determined either based on the relative magnitude to other weights in the same parameter or layer @xcite , or for the whole network @xcite . ffn(x) = relu(xw 1 + b 1 )w 2 + b 2 , where w 1 ∈ r d×dm and w 2 ∈ r dm×d are parameters of the block. lastly, a residual connection is applied and a layer normalization @xcite . adapters. @xcite the adapter technique injects a module between the transformer layers, such that the input is down-projected to a lower-dimensional space using w @xmath0, followed by non-linearity σ, and up-projected using w @xmath1, combined with a residual connection: low rank adaptation. lora @xcite freezes the pre-trained model weights and injects two trainable rank decomposition matrices into each layer of the transformer architecture, greatly reducing the number of trainable parameters for fine-tuning tasks. for a linear layer @xmath2, the lora-modified forward function is: ## experiments we apply prilora to debertav3-base @xcite (184 million parameters), and evaluate the method on eight natural language understanding benchmarks included in the general language understanding evaluation -glue @xcite . summary of the glue benchmarks can be found in ## discussion moving from one task to another requires an adaptation of both the input and the output domain. while the input domain of large language models may be comprehensive enough to support new downstream tasks, the generation of the output is very much context-and-task-dependent. therefore, it should not come as a surprise that fine-tuning requires more adaptation of the top lay-ers, which are closer to the output, than of the earlier, input-processing, layers. however, if one is to change only the top layers, as we showed in the ablation study, there would not be enough co-adaptation of the earlier layers to enable the top layers to produce the required output. it seems, therefore, that the gradual increase in the allocated resources, which we apply, is a reasonable strategy. ## conclusions in this paper, we introduced prilora, a novel, yet simple and parameter-efficient method for improving low-rank adaptation during fine-tuning. our extensive experiments encompass eight glue benchmarks across multiple seeds, illustrating the effectiveness of prilora. notably, we achieve superior performance compared to state-of-the-art metrics while maintaining the same number of trainable parameters, reducing the non-zero parameters by a quarter on most benchmarks, and adhering to the same memory constraints and running time per epoch.