R es L o RA : Identity Residual Mapping in Low-Rank Adaption (2024)

Abstract: As one of the most popular parameter-efficient fine-tuning (PEFT) methods, low-rank adaptation (LoRA) is commonly applied to fine-tune large language models (LLMs). However, updating the weights of LoRA blocks effectively and expeditiously is challenging due to the long calculation path in the original model. To address this, we propose ResLoRA, an improved framework of LoRA. By adding residual paths during training and using merging approaches to eliminate these extra paths during inference, our method can achieve better results in fewer training steps without any extra trainable parameters or inference cost compared to LoRA. The experiments on NLG, NLU, and text-to-image tasks demonstrate the effectiveness of our method. To the best of our knowledge, ResLoRA is the first work that combines the residual path with LoRA. The code of our method is available at [this url](https://github.com/microsoft/LMOps/tree/main/reslora).

## introduction in recent years, large language models (llms) @xcite with hundreds of billions of parameters have shown remarkable performance on various tasks. fine-tuning llms on specific datasets typically leads to better performance than merely giving instructions in the prompt during inference @xcite . however, the cost of it is often prohibitive due to the large number of parameters involved. to address this problem, various parameterefficient fine-tuning (peft) methods have been proposed. peft methods freeze all parameters in the original model, and only tune a few parameters in the newly added modules. among them, one of the most popular peft methods is lora @xcite , which stands for low-rank adaptation. lora uses two matrices parallel to the original figure 1 : an illustration of reslora method on accuracy for svamp @xcite . reslora achieves a 2.5x faster convergence speed and improves performance by 14.3%. the basic lora method still has some limitations. previous studies mainly focused on either dynamically adjusting the rank of lora modules in different layers of the model @xcite , or using fewer trainable parameters to achieve a similar effect as the original lora method @xcite . however, they overlooked a potential problem: a long backward path hinders the updating of parameters in lora blocks. as a prominent method, resnet @xcite has proven to be widely efficient, and is also used in transformer models @xcite , between different encoder and decoder blocks. parallel to linears in these blocks, lora blocks can also benefit from the original shortcut design. however, unlike linears, lora blocks are more fine-grained. one lora block only corre-sponds to one linear, so the original shortcut is not perfectly suitable for lora blocks. let's use encoders of transformer as an example. if we add lora blocks in query, key, or value linears, the previous gradient must go through the output linear and softmax function when calculating backward, which may cause gradient vanishing or explosion. in this paper, we present reslora, a new framework that merges the shortcut of resnet into lora blocks. to validate the efficiency of different residual structures, we propose three residual structures and conduct experiments on different models and tasks. however, the shortcut cannot be directly merged into the original network due to its nonplain structure, which undermines the advantage of lora. considering this, we discuss different merging approaches to convert reslora to the original lora blocks, in order to ensure that it can still be merged into the original modules during inference. after merging, our method introduces neither extra parameters nor computational complexity. compared to lora, reslora does not use any additional parameters, but achieves 1% to 20% improvement in performance during inference, and lower and faster convergence of loss during training. moreover, our method can be easily applied to not only the basic lora method, but also the other variants of lora. finally, we evaluate the robustness of the results on different experiments. to the best of our knowledge, reslora is the first work that combines the residual path with the lora method. the effectiveness of our method is illustrated in figure 1 . overall, our contribution can be summarized as follows: low-rank training method (lora) recent studies on lora aim to achieve lower cost and better performance. some researchers explore more flexible and appropriate ranks, such as dylora @xcite , relora @xcite , loha @xcite and lokr @xcite . adalora @xcite design a method to dynamically allocate the rank of lora blocks in different layers based on their importance, which can reduce the unimportant rank of lora blocks. other works focus on the combination of lora and other approaches, such as adamix @xcite and qlora @xcite . besides, lorahub @xcite and lo-ramoe(anonymous, 2024) focus on how to merge multiple lora blocks that are fine-tuned on different tasks respectively. despite this, no one has focused on the potential barrier of gradient propagation in lora. residual network(resnet) he et al. (2016a) and previous works(srivastava et al., 2015) first introduced the residual network. this work solves the gradient vanishing or explosion and improves numerical stability during gradient updating. considering that the extra shortcut path requires extra computational cost, some works @xcite attempt to remove extra paths in the inference stage. inspired by them, we first extend the main idea of resnet to lora to achieve a faster and more stable training stage, and then design merging approaches to preserve the plain structure of lora, so as to maintain the advantages of both lora and resnet at the same time. ## method in this section, we introduce our framework, which mainly consists of two parts: (1) reslora blocks, which add various residual paths in lora blocks, mainly used in training stage; (2) merging approaches, which remove the residual paths to convert reslora blocks to lora blocks, mainly used in inference stage. ## experiments 4.1 experimental setup to evaluate the effectiveness of reslora, we conduct experiments on a wide range of models and tasks, including natural language generation (nlg), natural language understanding (nlu), and image generation. we present the details of the tasks in the following subsections respectively. we compare our method with lora(hu et al., 2022), adalora @xcite , loha @xcite and lokr @xcite , which we detailedly describe in section a. all details of experiments can be found in section c. ## conclusion we develop a new improved framework reslora for low-rank adaptation (lora). reslora adds residual paths during training, and uses merging approaches to remove these paths during inference. without any extra trainable parameters, reslora can achieve better results in fewer training steps compared to the original lora and other baseline methods. we conduct experiments with three types of reslora structures on nlg, nlu, and textto-image tasks, and the results of almost all tasks verify the effectiveness of our method.