L o RAN : Improved Low-Rank Adaptation by a Non-Linear Transformation (2024)

Abstract: In this paper, we study parameter-efficient fine-tuning methods for large pre-trained models. Specifically, we improve LoRA approaches to alleviate the performance loss from the constrained adapter by introducing a non-linear transformation (call it LoRAN). For a better adaptation, we also design a new non-linear function to appropriately fit the accumulated weight updates. We test our method in multiple advanced large language models. Experimental results show that our LoRAN significantly outperforms a strong baseline on SAMSum and 20 Newsgroups tasks. Moreover, when a lower rank is applied, our approach even yields a 1.95-point improvement in the classification task.

## introduction recently, large language models (llms) have shown great improvements on a wide range of nlp tasks. methods of this kind make it possible to learn universal representations from large corpora and adapt pre-trained models to downstream tasks through fine-tuning @xcite . early fine-tuning methods optimize models within the entire parameter space @xcite , whereas parameter-efficient fine-tuning (peft) have successfully trained downstream models with fewer parameters @xcite @xcite . as an instance of the latter, the low-rank adaptation (lora) introduces lightweight trainable matrices to fit the accumulated weight updates while most llm parameters are frozen. this leads to an efficient training process with a smaller memory footprint than full fine-tuning. like recent adapter-based methods, lora adds a small number of extra parameters to adapt models to downstream data. under the low-intrinsic-rank hypothesis @xcite , the multiplication of two low-dimension matrices is introduced to fit every weight update matrix during fine-tuning (see figure 1(a) ), and then the downstream model can be easily obtained by merging back all the multiplications with the frozen parameters. this low-rank decomposition dramatically reduces the number of trainable parameters, even making it possible to equip multiple portable lora models for different tasks onto one shared foundation. however, the effectiveness of lora heavily relies on the match between the adapter capacity and the downstream task. if the adapters are too small, there will be a significant gap between adapter outputs and optimal weight updates, leading to a performance loss in task transfer. previous efforts on this issue have focused on budgeting adapter parameters @xcite and upgrading matrix decomposition @xcite . these methods are able to narrow the gap by sufficient parameters and effective computations, but cannot thoroughly plug the gap to the same fitting precision with full finetuning efficiently due to low-rank multiplications. in this work, we enhance the adapter capacity without additional parameters, enabling the adapter to fit weight updates more delicately. we present an improvement of lora, introducing a non-linear transformation after matrix decomposition (call it low-rank adaptation with non-linear transformation, or loran for short). it leads to an adapter that even matches the full-rank model capacity (see figure 1(b) ). in addition to the new framework, we also develop a novel non-linear function for lo-ran, that ensures a reasonable practice based on the matrix features of weight updates. our loran is simple to implement and straightforwardly applicable to various lora variants. we experiment with it in flan-t5, falcon, and llama 2 systems. experimental results on samsum and 20 newsgroups tasks show significant improvement to the baseline. when the rank is limited lower, our system even yields a more considerable improvement of +1.95 point accuracy in the 20 newsgroups task. more interestingly, in the classification task, loran presents greater sensitivity to distinguishing minor differences between similar classes. this corresponds to our motivation for addressing the rough weight update fitting in low-rank adapters and indicates a promising line of research on applying loran to complicated tasks, such as extremely low-resource fine-tuning. ## the method in this work, we use lora for description foot_0 . taking a downstream model as an example, numerous fully connected layers are equipped. a fine-tuned weight matrix w d i can be defined as w d @xmath0. ## related work peft has been proposed to improve computing and memory efficiency for applying llms to downstream tasks @xcite @xcite . one classic method, lora, uses adapters to save a substantial number of trainable parameters, making it popular to produce taskspecific models on resource-limited devices @xcite . however, its matrix decomposition restricts weight updates to the low-rank space, constraining the model's expressiveness. a strand of addressing this issue is to budget adapter parameters, adding suitably sized adapters at critical neural positions @xcite , and the other typical line is to upgrade the decomposition way for higher performance @xcite . nevertheless, both approaches either fail to thoroughly bridge the gap between the adapter capacity and weight updates, or require additional parameters. no discussion exists on a method that can achieve low-rank training comparable to full fine-tuning under the same number of parameters. ## conclusions we have presented an improved low-rank adaptation with non-linear transformation for a more delicate weight update fitting in fine-tuning. meanwhile, for more reliable progress, a brand-new nonlinear function is proposed. experiments on sam-sum and 20 newsgroups tasks both show significant improvements over the baseline. when a lower rank is used, it even achieves a 1.95-point improvement in the classification task.