A uto L o RA : Automatically Tuning Matrix Ranks in Low-Rank Adaptation Based on Meta Learning (2024)

Abstract: Large-scale pretraining followed by task-specific finetuning has achieved great success in various NLP tasks. Since finetuning all parameters of large pretrained models poses substantial computational and memory challenges, several efficient finetuning methods have been developed. Among them, low-rank adaptation (LoRA), which finetunes low-rank incremental update matrices on top of frozen pretrained weights, has proven particularly effective. Nonetheless, LoRA’s uniform rank assignment across all layers, along with its reliance on an exhaustive search to find the best rank, leads to high computation costs and suboptimal finetuning performance. To address these limitations, we introduce AutoLoRA, a meta learning based framework for automatically identifying the optimal rank of each LoRA layer. AutoLoRA associates each rank-1 matrix in a low-rank update matrix with a selection variable, which determines whether the rank-1 matrix should be discarded. A meta learning based method is developed to learn these selection variables. The optimal rank is determined by thresholding the values of these variables. Our comprehensive experiments on natural language understanding, generation, and sequence labeling demonstrate the effectiveness of AutoLoRA. The code is publicly available at https://github.com/ruz048/AutoLoRA

## introduction large language models (llms) @xcite have demonstrated stateof-the-art performance across a variety of nlp tasks, spanning from natural language understanding (nlu) @xcite to natural language generation (nlg) @xcite , a trajectory highlighted by the success of models like @xcite . their success largely *equal contribution. to address this challenge, many efficient finetuning methods @xcite have been developed. for instance, the adapters method @xcite inserts lightweight layers (called adapters) into pretrained networks. during finetuning, only these adapters are updated while the pretrained layers are kept frozen. one limitation of this method is that the adapters incur additional computation overhead during inference. another approach, prefix tuning @xcite , introduces trainable prefix parameters which are prepended to the input sequence while making the pretrained model parameters frozen. nevertheless, determining the optimal length of the prefix can be tricky. a prefix that is too short cannot capture enough information, while an overlong prefix may largely reduce the maximum length of the input sequence. to address these limitations, lora @xcite proposes to add low-rank incremental update matrices to pretrained weight matrices. during finetuning, only the incremental matrices are trained while the pretrained ones are frozen. the low-rank parameterization significantly reduces the number of finetuning parameters. while achieving parameter-efficient finetuning without increasing inference costs, lora has two limitations. first, the update matrices at different layers share the same rank, without considering the varying properties across layers. different layers in a pretrained model have varying importance to a downstream task and should be adapted differently, which requires the number of trainable parameters to be layer-specific. em- optimize update matrices low-rank matrix 𝑈 low-rank matrix 𝑉 selection variables optimize selection variables low-rank matrix 𝑈 low-rank matrix 𝑉 selection variables iterate until convergence meta learning of selection variables thresholding selection variables retrain update matrix with optimal rank determine matrix rank in the meta learning process, autolora learns selection variables with two iterative steps. firstly, the weights in the update matrices are optimized on the training dataset. secondly, the selection variables are updated on the validation dataset. these two steps are iterated until convergence is achieved. upon acquiring the optimal values of the selection variables, autolora determines the optimal matrix ranks by thresholding these values. subsequently, the ranks of update matrices in the lora layers are set to the learned optimal ranks and retrained on the combination of training and validation data. to address the aforementioned limitations of lora, we introduce the autolora framework to automatically determine the optimal rank for each lora layer. in autolora, we first decompose an update matrix into the product of two low-rank matrices (with rank k), in alignment with the lora methodology. this product can be expressed as the summation of k rank-1 matrices. for each rank-1 matrix, we assign a continuous trainable selection variable α ∈ [0, 1] indicating the matrix's relative importance in the summation. after learning, if α is close to zero, the corresponding rank-1 matrix is removed from the summation. these selection variables effectively control the rank of an update matrix. learning α directly on a training dataset together with the update matrices can result in overfitting, and the network learned in this way lacks generalization ability. to mitigate this problem, we formulate the search process of α as a meta learning @xcite problem. first, we finetune the weights in the rank-1 matrices on a training dataset. second, we optimize the α values by mini-mizing the loss on a validation dataset. these two steps iterate until convergence. subsequently, we derive the optimal rank of each lora layer by thresholding the learned α values. once the optimal rank is identified for each layer, the weights in the low-rank update matrices are retrained on the combination of training and validation data. an overview of our proposed method is illustrated in figure 1 . the major contributions of this paper are summarized as follows. ## preliminaries in lora @xcite , a weight matrix w @xmath0, where w l is the weight matrix at layer l in a pretrained model and ∆ l is an incremental update matrix. ∆ l is parameterized as the product of two low-rank matrices: equivalently, ∆ l can be written as the summation of k l rank-1 matrices: ## discussion in this section, we discuss the potential issue of overfitting associated with autolora. in contrast to adalora @xcite , our approach demonstrates enhanced resilience against overfitting by learning rank-1 matrices and selection variables across two separate sub-datasets, thereby reducing the likelihood of overfitting to any single dataset. as shown in ## conclusions and future work in this paper, we introduce autolora, a meta learning based framework designed to automatically search for the optimal ranks for lora layers. our method associates each rank-1 matrix in lora updates with a selection variable and formulates the rank-tuning problem as optimizing the selection variables via meta learning. thresholding is applied to derive discrete rank values from continuous selection variables and retraining is performed to bridge the gap incurred by thresholding. comprehensive experiments show the efficacy of autolora across various nlp tasks. similar to the lora method, the lora layers in autolora are manually specified, which may be suboptimal. as a future work, we will investigate how to automatically select lora layers, by developing a meta learning framework similar to that in eq.( 5 ).