AFL o RA : Adaptive Freezing of Low Rank Adaptation in Parameter Efficient Fine-Tuning of Large Models (2024)

Abstract: We present a novel Parameter-Efficient Fine-Tuning (PEFT) method, dubbed as \textit{Adaptive Freezing of Low-Rank Adaptation} (AFLoRA). Specifically, for each pre-trained frozen weight tensor, we add a parallel path of trainable low-rank matrices, namely a down-projection and an up-projection matrix, each of which is followed by a feature transformation vector. Based on a novel freezing score , we incrementally freeze these projection matrices during fine-tuning to reduce the computation and alleviate over-fitting. Our experimental results demonstrate that we can achieve state-of-the-art performance with an average improvement of up to 0.85% as evaluated on the GLUE benchmark while yielding up to 9.5\times fewer average trainable parameters. While compared in terms of runtime, AFLoRA can yield up to 1.86\times improvement as opposed to similar PEFT alternatives. Besides the practical utility of our approach, we provide insights on the trainability requirements of LoRA paths at different modules and the freezing schedule for the different projection matrices.

## introduction pre-trained language models such as bert @xcite , gpt-3 @xcite , and llama2 @xcite have demonstrated commendable performance on various natural language processing (nlp) tasks @xcite . however, their zero-shot performance on many downstream tasks often falls short of expectations. one possible solution is full fine-tuning (fft) of the model on the downstream dataset. however, the large model parameter size makes this process prohibitively costly. to address this challenge, various parameterefficient fine-tuning (peft) methods including low rank adaptation (lora) @xcite , adapter tuning @xcite , and prompt tuning @xcite are proposed. these methods add parameters to the trained model for fine-tuning, bypassing the need to adjust the weights of the pre-trained model. in particular, lora @xcite and its variants @xcite add a trainable low-rank path consisting of down-projection and up-projection matrices to the model, inspired by @xcite which showed that such low-rank paths can effectively approximate the trained weight tensors. elora @xcite extends lora by adding trainable feature transformation vectors to the output of each project matrix. they showed that sota accuracy can be achieved with the projection matrices frozen after random initialization while keeping the two feature transformation vectors trainable. this approach significantly reduces the number of trainable parameters. however, compared to lora, elora incurs higher computation costs due to the higher rank needed for the frozen projection matrices. fig. 1 illustrates lora and elora, contrasting them to our proposed method aflora. ## motivational case study to understand the high-rank requirement for the frozen projection matrices in elora, we conduct two sets of fine-tuning on sst-2 and mrpc, with elora having rank (r) of 1024 and 4, respectively. as we can see in fig. 2 , the model with @xmath0, yields poorer performance, highlighting the need for high rank for the frozen tensors. this high rank causes elora to potentially be flops inefficient. ## aflora: methodology module structure. inspired by the framework proposed by @xcite , we design the lora module to encompass four components, namely, the down-projection linear layer (lora a ), the up-projection linear layer (lora b ), and two feature transform vectors (s d , and s b ) placed before and after lora b . however, unlike @xcite , we keep both the projection matrices (lora a and lora b ) and vectors trainable at the beginning and keep the rank very low. the module processes a given input x through these components to produce an output y . the complete operation for a layer l can be described as follows: here, a l and b l are the trainable lora tensors of lora l a and lora l b , respectively. λ d and λ b are the vectors of s d , and s b , respectively. w l 0 represents the frozen pre-trained weights. we use kaiming uniform initialization for a l and b l , and follow @xcite to initialize the vectors. adaptive freezing. in pruning literature @xcite @xcite @xcite , sensitivity is gauged to reflect weight variability, necessitating consideration of both the weights' magnitudes and their gradients. small weight values suggest minimal impact, while minor gradient values indicate stability. taking inspiration from this idea, here we introduce the concept of a "freezing score". however, unlike pruning where both magnitude and gradient play a critical role in identifying insignificant weight, we leverage only gradient as a proxy to compute the freezing score. this is because, we assume large magnitude weights with negligible change has the same priority to be frozen as that for small magnitude weights. this score quantifies the degree to which weights vary throughout the training process. consequently, when the expected changes to the weights become negligible, we may consider them to be frozen, thereby saving computational resources and energy. the following equation describes the freezing score evaluation steps for a low-rank tensor a l . a l ) and uncertainly tensor (u (t) a l ), as shown in eq. 2 and 3, respectively. we then evaluate the freezing score s to apply thresholding on the lora freezing scores, we use the cubic schedule as @xcite . in specific, we keep the projection matrices trainable for the initial t i training steps, and then progressively freeze them by calculating the freezing fraction r(t) as shown in eq. 5. finally, all the projection matrices freeze beyond tt f steps. note, at step t, for a computed freezing fraction k, we freeze the lowest k% projection matrices. ## experiments models & datasets. we use the peft framework of @xcite and evaluate the fine-tuning performance of debertav3-base @xcite to fine-tune on our framework on the general language understanding evaluation (glue) benchmark @xcite . the details of the hyperparameter settings for each dataset are listed in appendix a.2. ## ablations and discussions we conducted our ablation studies on six glue benchmark datasets, omitting qqp and mnli, the two most computationally demanding datasets. ## conclusions in this paper, we presented aflora, adaptive freezing of lora adapters that allow near-optimal trainability of the lora projection matrices and freezes them driven by a "freezing score" after certain fine-tuning steps. compared to lora, aflora can reduce the trainable parameters by up to 9.5× while yielding 0.85% average improved performance as evaluated on the glue benchmark.