M o DULA : Mixture of Domain-Specific and Universal L o RA for Multi-Task Learning (2024)

Abstract: The growing demand for larger-scale models in the development of Large Language Models (LLMs) poses challenges for efficient training within limited computational resources. Traditional fine-tuning methods often exhibit instability in multi-task learning and rely heavily on extensive training resources. Here, we propose MoDULA (Mixture of Domain-Specific and Universal LoRA), a novel Parameter Efficient Fine-Tuning (PEFT) Mixture-of-Expert (MoE) paradigm for improved fine-tuning and parameter efficiency in multi-task learning. The paradigm effectively improves the multi-task capability of the model by training universal experts, domain-specific experts, and routers separately. MoDULA-Res is a new method within the MoDULA paradigm, which maintains the model’s general capability by connecting universal and task-specific experts through residual connections. The experimental results demonstrate that the overall performance of the MoDULA-Flan and MoDULA-Res methods surpasses that of existing fine-tuning methods on various LLMs. Notably, MoDULA-Res achieves more significant performance improvements in multiple tasks while reducing training costs by over 80% without losing general capability. Moreover, MoDULA displays flexible pluggability, allowing for the efficient addition of new tasks without retraining existing experts from scratch. This progressive training paradigm circumvents data balancing issues, enhancing training efficiency and model stability. Overall, MoDULA provides a scalable, cost-effective solution for fine-tuning LLMs with enhanced parameter efficiency and generalization capability.

## introduction recent advancements in open-source large language models (llms), such as llama (tou-* equal contribution. to address the challenges, we propose a threestage training paradigm called modula, where different domain-specific experts can be trained separately. moreover, we introduce a more advanced method modula-res (mixture of domainspecific and universal lora with residual connection), which incorporates a residual structure to make the training more stable, as seen in figure 1(c ). unlike molora, which employs multiple identical lora adapters as experts, our paradigm incorporates a universal expert alongside multiple domain-specific experts. the universal expert learns task-agnostic representations, while each domain-specific expert operates as a bias adapter, focusing on domain-specific knowledge. intuitively, arranging these adapters in parallel and allocating weights to each adapter via a router constitutes the modula-flan (mixture of domain-specific and universal lora with flan routing) method as seen in figure 1(b ). however, this method may potentially compromise universal capabilities. to address this, modula-res introduces a refined method that enables domainspecific experts to receive input from the output of the universal expert. this design ensures a coherent flow of information and facilitates the optimal integration of both universal and domain-specific expert functionalities through a residual connection. by dynamically adjusting the contributions of domain-specific experts, modula-res adapts to individual tasks while preserving broad generalization capabilities. this flexibility allows the model to leverage its general competencies for task understanding and summarization when encountering new tasks, thereby achieving a more balanced and effective adaptation in multi-task scenarios. during model training, our modula employs a three-stage optimization process, with detailed illustrations displayed in figure 2 : 1) initially, only the universal expert is trained to adapt to general tasks quickly; 2) subsequently, each domainspecific expert is trained individually, focusing on its corresponding task; 3) finally, the parameters of all experts are frozen, and only router is trained to learn the optimal combination strategy for different tasks. this progressive training paradigm allows our methods to avoid retraining from scratch, distinguishing it from molora, which trains only a new expert for a new specific task and retraining the router. this paradigm significantly reduces computational costs, mitigates data balancing challenges, and enhances the model's pluggability. to evaluate the effectiveness of our proposed methods, we conduct extensive experiments on a diverse set of open-source llms, including llama-2 @xcite , qwen @xcite , and yi @xcite , across various tasks. the results consistently demonstrate that mod-ula exhibits a significant performance, achieving 4.5% improvements compared to molora. by introducing residual connections, modula-res achieves even greater improvements without compromising the general capabilities. additionally, our approach showcases superior adaptability to new tasks, outperforming molora in finance and e-commerce domain with less training data and parameters, highlighting the enhanced task pluggability of our approach, making it an efficient and general solution for multi-task learning in llms. ## conclusion in this paper, we introduce modula, a novel multi-stage training peft moe paradigm that enhances efficiency and domain-specific adaptation for llms. by integrating universal and domainspecific experts through a three-stage training methodology, modula optimizes both generalization and specialized performance. experiments on various open-source llms, such as llama-2, qwen, and yi, demonstrate that modula outperforms existing methods, achieving over 80% reduction in training costs and a 5% performance improvement. these results highlight modula's potential as a scalable and efficient solution for fine-tuning llms, paving the way for future advancements in nlp.