M i L o RA : Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning (2024)

Abstract: Low-rank adaptation (LoRA) and its mixture-of-experts (MOE) variants are highly effective parameter-efficient fine-tuning (PEFT) methods. However, they introduce significant latency in multi-tenant settings due to the LoRA modules and MOE routers added to multiple linear modules in the Transformer layer. To address this issue, we propose Mixture of Low-Rank Adaptation (MiLoRA), a novel and efficient LoRA variant. MiLoRA differs from previous MOE-style LoRA methods by considering each LoRA module as an expert and employing a prompt-aware routing mechanism. This mechanism calculates expert routing results once before generating the first new token and reuses these results for subsequent tokens, reducing latency. Extensive experiments and analysis on commonsense reasoning tasks, math reasoning tasks, and widely used LLM evaluation benchmarks demonstrate that MiLoRA consistently outperforms strong PEFT baselines with comparable tunable parameter budgets. Additionally, MiLoRA significantly reduces latency in multi-tenant settings compared to previous LoRA-based methods.

## introduction large language models (llms) have been achieving state-of-the-art (sota) results not only in various natural language processing tasks @xcite but also in numerous challenging evaluation tasks @xcite , such as question answering, reasoning, math, safety, and instruction following. although llms are evolving into general task solvers, fine-tuning remains essential for efficient llm inference and for controlling the style of the generated content @xcite . full-parameter fine-tuning of such large models is impractical due to the significant gpu memory and computational resources required. consequently, parameter-efficient fine-tuning (peft) @xcite has garnered considerable attention in the research community, as it typically involves tuning less than 1% of the llms' parameters, thereby substantially reducing computational costs. among many peft methods, the reparameterization-based method low-rank adaptation (lora) @xcite is considered one of the most effective methods for llms @xcite @xcite . although lora is effective and can bring stable performance with the original setting in @xcite , it still brings inconvenience under the multi-tenant setting @xcite : it has to add lora modules to multiple weights of the transformer layer and introducing significant additional latency in every generation steps under the multi-tenant setting. recently, the mixture-of-experts (moe) style lora methods @xcite @xcite @xcite have surged, further pushing the performance ceilings of lora fine-tuning. however, they introduce the calculation of moe routers, further increasing inference latency. thus, it is essential to develop a novel variant of the lora method that introduces minimum latency during generation and still can perform competitively in downstream tasks. in this work, we propose a novel peft method called mixture of low-rank adaptation (milora). our milora method differs from the previous literature on moe-style lora methods in the following two aspects. first, in milora, an entire lora module is considered a lora expert, and the lora router is responsible for determining which lora expert to activate. second, we propose the prompt-aware routing mechanism instead of calculating the expert routing results for @xcite . there are seven linear modules and seven positions to add lora modules. right: upon receiving an input prompt, the lora router before each transformer layer will take the input prompt's hidden states as input features and go through a pooler, an activation function, and the moe router network to determine which lora module is activated (or used) (e.g., lora u in the figure). this routing decision is repeatedly used when generating subsequent tokens. we conduct extensive experiments and analysis on various challenging tasks, including five commonsense reasoning tasks, two math reasoning tasks, and three widely used llm evaluation benchmarks. our method can consistently outperform strong peft baselines with comparable tunable parameter budgets, especially the recent lora variants. in addition, our milora method has significantly lower latency under the multi-tenant setting @xcite than the previous lora-based methods with comparable tunable parameters. our contributions are summarized as follows: ## related works since lora is the most popular peft method in the era of large language models, many works are devoted to improving upon lora. adalora @xcite looks into the parameter allocation of lora modules. vera @xcite investigate whether one could freeze the randomly initialized lora matrices and only learn a set of scaling vectors. recently, a series of works has been looking into combining mixture-of-experts (moe) @xcite and lora. llava-mole @xcite effectively routes tokens to domain-specific lora experts, mitigating data conflicts and achieving consistent performance gains over the original lora method. moelora @xcite proves that fine-tuning lora modules with a moe router en-ables the llms to perform well in a multi-task learning setting. moral @xcite addresses the challenge of adapting llms to new domains/tasks and enabling them to be efficient lifelong learners using the moe techniques. lo-ramoe @xcite integrates loras using a router network to alleviate world knowledge forgetting after instruction tuning. mocle @xcite proposes a moe architecture to activate taskcustomized model parameters based on instruction clusters. although performing well in fine-tuning, these methods introduce high additional latency since (a) these methods do not reduce the number of lora modules in the transformer backbone. (b) the routers and lora modules must be called when generating each new token. our milora method addresses this efficiency issue by (a) only calling the lora routers when encoding the input prompt and before generating the first new token. (b) only activate one lora module per transformer layer. ## methods in this section, we first introduce the foundational concepts of lora and moes and then elaborate on the architectural design of milora. ## experiments in this section, we conduct a series of experiments and analysis to evaluate our milora method. ## conclusion this work presents the mixture of lora (milora) method, a novel method for the parameter-efficient fine-tuning of large language models. different from previous literature on moe style lora methods, milora: (a) activates lora experts at the transformer layer level, determining which transformer module's lora is activated. (b) the decision to activate which lora expert is conditioned on the input prompt. (c) for a given prompt, the lora routers are called only once. the subsequent token generation steps reuse the routers' decisions. in order to improve our framework's downstream performance, we propose to learn different activation functions during fine-tuning for lora routers of different depths. our method is convenient to implement and off-the-shelf. experiments on various tasks demonstrate that our milora method outperforms the baseline methods while being efficient in inference.