Mixture-of- L o RA s: An Efficient Multitask Tuning Method for Large Language Models (2024)

Abstract: Instruction Tuning has the potential to stimulate or enhance specific capabilities of large language models (LLMs). However, achieving the right balance of data is crucial to prevent catastrophic forgetting and interference between tasks. To address these limitations and enhance training flexibility, we propose the Mixture-of-LoRAs (MoA) architecture which is a novel and parameter-efficient tuning method designed for multi-task learning with LLMs. In this paper, we start by individually training multiple domain-specific LoRA modules using corresponding supervised corpus data. These LoRA modules can be aligned with the expert design principles observed in Mixture-of-Experts (MoE). Subsequently, we combine the multiple LoRAs using an explicit routing strategy and introduce domain labels to facilitate multi-task learning, which help prevent interference between tasks and ultimately enhances the performance of each individual task. Furthermore, each LoRA model can be iteratively adapted to a new domain, allowing for quick domain-specific adaptation. Experiments on diverse tasks demonstrate superior and robust performance, which can further promote the wide application of domain-specific LLMs.

## introduction large language models (llms) have played a pivotal role in expediting the advancement of natural language processing (nlp), offering a versatile and task-agnostic foundation that underpins an extensive array of applications. the intrinsic diversity found in domain-specific data poses a substantial challenge in training a general-purpose base llms. consequently, there has been a surge in the adoption of domain-specific llms to tackle intricate problems within specialized domains, such as sql-palm @xcite ), * corresponding author bloomberggpt @xcite , chatlaw @xcite , pdfgpt @xcite . in realworld application scenarios, the demand often arises for a multitude of customized capabilities. llms with multiple customized capabilities can efficiently address a diverse range of user problems, and each specific functional module can be optimized individually. domain specification techniques are key to make large language models disruptive in various applications @xcite . to learn sufficient domain knowledge and not lose basic capability, adapter-based fine-tuning methods (e.g., adapters @xcite , lora @xcite ) introduce a limited number of domain-specific parameters to retain domainrelated knowledge and do not need to fine-tuning all parameters of the pre-trained model, which can effectively reduce the training cost of llms. in order to obtain multiple customized capabilities, the simplest and efficient fine-tuning approach is to directly mix data from multiple domains together and only add one lora module for instruction fine-tuning. the second is some two-stage approaches. we first train multiple domain lora modules individually, and then introduce a domain classifier to select appropriate lora model. in addition, @xcite ; @xcite add the adapterfusion layer and element-wise lora composition to implicitly fuse parameter knowledge of multiple task adapters. the moe @xcite introduces multiple experts to process different types of input data. @xcite is to consider the outputs of ensemble llms comprehensively. however, these approaches have two main issues. first, in the practical application scenario, there are often multiple destructive domain tasks with heterogeneous and imbalance training data, as well as limited computing resources. therefore, the implicit parameter fusion methods exist mutual disturbance, which results in degraded model performance on in-domain tasks, as illustrated in figure 1 . we often focus more on the domainspecific expertise of a domain-specific llm than its generalization performance. in addition, the moe mechanism needs to be trained from scratch based on a new model structure and a large amount of training corpus. the ensemble llms require sufficient computing resources to deploy multiple independent llms simultaneously. in this paper, we address these limitations and propose an end-to-end parameter-efficient tuning method designed for multi-task learning on llms, dubbed moa. first, we design a routing mechanism within decoder-only model architecture to automatically select lora experts, which can be applied to current mainstream llms, and can simultaneously deploy lora modules of multiple tasks with limited computing resources (using the same llm). meanwhile, our comprehensive model can still achieve excellent performance on different types of tasks. additionally, to improve the efficiency of training and inference, we implement a parallel processing strategy of different domain samples within a batch during the training process, and a lora expert selection approach in the inference time. our approach leverages the power of different expert models and the base llm, and the complementarity of knowledge in different domains. in summary, our contributions are as follows: ## methodology the proposed mixture-of-loras (moa) can be observed in figure 2 . given the variations in data scales and training complexities, we commence by separately training n lora modules {m 1 , ..., m n } for n distinct task types {t 1 , ..., t n }. it is worth noting that the task types mentioned here and the domain-specific data or tasks referred to in the paper have the same meaning, all representing scenario tasks that require certain expertise to solve. initially, we obtain the optimal lora module parameters on each scenario task data. these modules demonstrate commendable performance within their respective domains. subsequently, a routing mechanism is employed to integrate the n lora modules under a shared llm. specifically, multiple lora modules are simultaneously incorporated alongside each transformer layer. preceding each transformer layer, a routing mechanism is inserted to guide the selection of distinct lora experts. ## conclusions we introduce moa architecture, which provide an efficient multi-task fine-tuning method for llm, addressing interference among tasks and training instabilities. each lora model can be iterated individually to quickly adapt to new domains. meanwhile, moa uses routing strategy to flexibly select the appropriate lora expert to solve the problem. it can arbitrarily combine multiple domain-specific lora modules to implement a llm with multiple specific capabilities. future work may focus on how to flexibility add or remove lora modules with unsupervised learning, optimize the current routing algorithm, or reduce the scale of training data in domain specialization of llms.