F an L o RA : Fantastic L o RA s and Where to Find Them in Large Language Model Fine-tuning (2024)

Abstract: Full-parameter fine-tuning is computationally prohibitive for large language models (LLMs), making parameter-efficient fine-tuning (PEFT) methods like low-rank adaptation (LoRA) increasingly popular. However, LoRA and its existing variants introduce significant latency in multi-tenant settings, hindering their applications in the industry. To address this issue, we propose the Fantastic LoRA (FanLoRA) framework, which consists of four steps: (a) adding LoRA modules to all the Transformer linear weights and fine-tuning on a large-scale instruction tuning dataset. (b) The importance of each module is then assessed using a novel importance scoring method. (c) only the most critical modules per layer are retained, resulting in the FanLoRA setting. (d) The FanLoRA setting is applied to fine-tune various downstream tasks. Our extensive experiments demonstrate that: (a) FanLoRA outperforms existing PEFT baselines across a wide collection of tasks with comparable tunable parameters. (b) FanLoRA significantly reduces the inference latency of LoRA, making it valuable for further broadening the applications of LLMs in the industry.

## introduction in the era of large language models (llms), parameter-efficient fine-tuning (peft) @xcite has raised much attention in the research field since in peft, the tunable parameters are often less than 1% of the llms and the hardware requirements for fine-tuning are significantly decreased. among many peft methods, the reparameterization-based method, lowrank adaptation (lora) @xcite , is considered one of the most effective methods for llms @xcite @xcite . although lora and its more recent variants @xcite ; hu et al., 2023) are effective and can bring stable downstream performance, they cause inference inefficiency under the multi-tenant setting @xcite , where one llm backbone has to serve multiple users/tasks with the help of multiple sets of lora parameters and the lora parameters can not be merged to the llm backbone. lora has to add low-rank modules to multiple linear weights of the transformer layer, introducing significant additional latency in every token generation step. thus, to promote efficiency in industrial usage, it is of vital importance to investigate the following research question: rq1. for a given llm backbone, can we find a lora setting that adds as few fantastic lora modules as possible to ensure efficiency, and is this setting universally transferable to different industrial tasks? to address the above rq1, we now propose the fantastic lora (fanlora) framework (figure 2 ), which makes lora more suitable for industrial applications. first, we add lora modules with an equal rank to each transformer weight (full lora setting) and fine-tune them on a general-purpose large-scale instruction tuning dataset (d train ). second, after fine-tuning, we calculate the importance of each lora module via ab-score, a novel importance scoring method we propose. third, we keep at most two lora modules for each transformer layer based on the importance scores and obtain the fanlora setting. in the fourth step, the lora modules under the fanlora setting are randomly initialized and fine-tuned on downstream tasks. figure 1 presents the distribution of selected lora modules in the fanlora setting when the llm backbone is qwen1.5 7b foot_0 or llm-assist 7b, a proprietary llm developed by an industrial participant foot_1 . we conduct extensive experiments on various open benchmark and proprietary tasks, including question answering, content generation, math reasoning, and general llm evaluation, demonstrating that our fanlora setting can be widely applied to different downstream tasks. our method can consistently outperform strong peft baselines with comparable tunable parameter budgets, especially the recent lora variants. through our experiments and analysis, we can obtain the following takeaways: (a) fanlora demonstrates that one can effectively fine-tune the llms by adding a small number of lora modules. (b) the fanlora setting performs well on a wide range of downstream tasks for a given llm backbone, demonstrating its broad transferability. (c) our fanlora method has significantly lower latency than the previous lora variants, showing potential for wide industrial applications. our contributions are summarized as follows: ## related works lora @xcite is proven to be an effective peft method when applied to both relatively small pretrained backbones and large language models @xcite . recently, many lora variants have been proposed. adalora @xcite expresses the lowrank matrix multiplication of lora in the form of singular value decomposition (svd), and it identifies the most critical ranks by a sensitivity-based importance score. sora @xcite prunes abundant lora ranks by imposing a l 0 norm and optimizing with proximal gradient descent. sa-lora @xcite prunes the lora ranks via the lagrange multiplier method. despite its tractability and effectiveness, lora still has room for improvement in both downstream task performances and efficiency under the multi-tenant setting @xcite . despite these recent efforts, the above lora variants still add lora modules to almost all the weight matrices in the transformer backbone, which results in significant latency under the multitenant setting and poses difficulties for industrial usage. our work complements the existing literature by addressing lora's efficiency issue. ## experiments in this section, we conduct experiments to evaluate our fanlora method. ## conclusion in this work, we introduced the fantastic lora (fanlora) framework to enhance the efficiency of parameter-efficient fine-tuning (peft) for large language models (llms) in industrial applications. by using a novel ab-score method to identify the most critical lora modules, fanlora effectively reduces latency overhead while maintaining high performance across diverse downstream tasks. our extensive experiments demonstrate that fanlora outperforms existing peft baselines with comparable tunable parameters, proving its versatility and efficiency in multi-tenant settings where an llm backbone has to serve multiple users/tasks via different sets of lora parameters.