L ora R etriever: Input-Aware L o RA Retrieval and Composition for Mixed Tasks in the Wild (2024)

Abstract: Low-Rank Adaptation (LoRA) provides an effective yet efficient solution for fine-tuning large language models (LLMs). The modular and plug-and-play nature of LoRA enables the integration of diverse domain-specific LoRAs to enhance the capabilities of LLMs. Previous research on exploiting multiple LoRAs either focuses on specific isolated downstream tasks or fixes the selection of LoRAs during training. However, in real-world scenarios, LLMs receive diverse prompts covering different tasks, and the pool of candidate LoRAs is often dynamically updated. To bridge this gap, we propose LoraRetriever, a retrieve-then-compose framework that adaptively retrieves and composes multiple LoRAs according to the input prompts. LoraRetriever contains three main components: firstly, identifying and retrieving LoRAs relevant to the given input; secondly, formulating strategies for effectively integrating the retrieved LoRAs; and thirdly, developing efficient batch inference to accommodate heterogeneous requests. Experimental results indicate that LoraRetriever consistently outperforms the baselines, highlighting its practical effectiveness and versatility. Our code is available at https://github.com/StyxXuan/LoraRetriever.

## introduction recently, large language models (llms) such as chatgpt @xcite and llama @xcite have shown notable successes in a range of fields @xcite @xcite . nevertheless, due to the prohibitively high computation costs for finetuning llms on specific domains, there is a growing shift towards parameter-efficient fine-tuning (peft) @xcite @xcite , which only updates a small fraction of the model's parameters or integrates new * corresponding authors. trainable parameters that augment the model's capabilities. within this sphere, low-rank adaptation (lora) @xcite stands out for its remarkable effectiveness, modularity, and plug-and-play capabilities. as ai communities like huggingface and modelscope witness an influx of lora parameters tailored for diverse tasks and domains, there is an increasing emphasis on employing various lora experts to provide a comprehensive service @xcite . recent research has explored the integration of the mixture of expert (moe; @xcite ; @xcite ) with loras @xcite @xcite @xcite . however, these methods lock in the selection of lo-ras while training, lacking the ability to dynamically update and scale in scenarios where the lora pool may consistently expand. lorahub @xcite and adaptersoup @xcite explore composing loras for specific downstream tasks. however, these two methods offer a one-size-fits-all solution for downstream tasks, overlooking the heterogeneous nature of diverse real-world requests. to bridge these gaps, our paper explores the "mixed-task scenario" as exemplified by platforms like chatgpt @xcite and gemini @xcite , wherein the nature of user requests encompasses diverse prompts covering different tasks. while llms present a unified solution for a broad spectrum of tasks, their performance can still falter in certain specialized areas @xcite @xcite . this is where the integration of loras becomes crucial. as shown in fig. 1 , our vision encompasses a multi-lora serving framework capable of dynamic enhancement, continuously improving its functionality as new lora modules are added and updated. through the plug-and-play capabilities of lora, the framework can provide personalized services for heterogeneous downstream requests. in this paper, we introduce loraretriever, a retrieve-then-compose framework designed to exploit the plug-and-play nature of lora in mixedtask scenarios. our framework consists of three key components: (1) input-aware lora retrieval: the first step of our framework is aligning user inputs with the corresponding loras through sentence embeddings and is further refined by an instruction fine-tuning @xcite for effective lora retrieval. through the retriever, we achieve a more flexible lora routing mechanism, whose training stage is disentangled from the training and inference of the llm. to assess the performance of loraretriever, we established a mixed-task evaluation benchmark comprising 48 loras spanning a variety of natural language understanding and generation tasks. the experimental results underline the effectiveness of the proposed methods in serving both in-domain and out-of-domain downstream requests. furthermore, the retrieval routing method exhibits a robust generalization capability: although the retriever is trained on just 40% of the tasks, it effectively retrieves the corresponding loras for unseen tasks. adapter merging. in addition to model ensembling through the moe, there is an increasing focus on aggregating adapters from different domains through the method of adapter merging. adapter-soup @xcite aggregates different adapters in the parameter space, allowing large language models to adapt to new domains without additional training. lorahub @xcite employs random sampling of lora parameters from various domains and tasks, followed by black-box optimization to learn the weights of different lora parameters without involving model gradient backpropagation. these methods offer a one-size-fits-all solution for downstream tasks, which cannot be applied in the mixed-task scenario for providing personalized service. ## preliminaries this section begins with a concise introduction to the low-rank adaptation, followed by a detailed formalization of the mixed-task scenario. ## loraretriever framework in this section, we describe the loraretriever framework as shown in fig. 2 for serving multi-loras in mixed-task scenarios. this framework contains three major components: the input-aware lora retrieval module ( §4.1), the lora composition module ( §4.2), and the batch inference strategy ( §4.3). ## experiments this section outlines the evaluation framework for assessing different approaches in mixed-task scenarios. furthermore, a comprehensive analysis of the proposed loraretriever framework is presented.