Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks (2024)

Abstract: Large language models (LLMs) have demonstrated considerable proficiency in general natural language processing (NLP) tasks. Instruction tuning, a successful paradigm, enhances the ability of LLMs to follow natural language instructions and exhibit robust generalization across general tasks. However, these models often encounter performance limitations across multiple tasks due to constrained model capacity. Expanding this capacity during the instruction tuning phase poses significant challenges. To address this issue, we introduce parameter-efficient sparsity crafting (PESC), which crafts dense models into sparse models using the mixture-of-experts (MoE) architecture. PESC integrates adapters into the MoE layers of sparse models, differentiating experts without altering the individual weights within these layers. This method significantly reduces computational costs and GPU memory requirements, facilitating model capacity expansion through a minimal parameter increase when guaranteeing the quality of approximation in function space compared to original sparse upcycling. Our empirical evaluation demonstrates the effectiveness of the PESC method. Using PESC during instruction tuning, our best sparse model outperforms other sparse and dense models and exhibits superior general capabilities compared to GPT-3.5.Our code is available at https://github.com/wuhy68/Parameter-Efficient-MoE.

## introduction recent advancements in nlp have been significantly propelled by the advent of llms such as gpt @xcite , llama @xcite , mistral @xcite , etc. the increasing scale of llms has established them as the experts for nlp tasks due to their exceptional ability to identify complex linguistic patterns @xcite . a prominent method for training llms is instruction tuning @xcite . this approach utilizes large-scale, well-formatted instruction data, enabling llms to refine their pre-trained representations to comply with human instructions @xcite @xcite . such instruction-tuned llms exhibit remarkable generalization capabilities in nlp tasks @xcite . this generalization requires training on a broad range of instruction-following tasks from multiple domains such as math, code, biology, etc @xcite . however, the inherent complexity of these tasks can hinder model fine-tuning @xcite . specifically, models of certain sizes may struggle to optimize losses from conflicting tasks, resulting in subpar performance for general tasks. the scaling law @xcite suggests that increasing the model's scale is crucial for better performance. expanding the model's capacity can also improve instruction tuning effectiveness for general tasks @xcite . nonetheless, most llms are pre-trained dense models designed based on transformer architecture, which limits scalability during instruction tuning. @xcite presented a method for upcycling dense models into sparse activated moe models, which boast greater capacity @xcite @xcite . notably, @xcite suggested that moe models respond more effectively to instruction tuning compared to dense models. consequently, converting dense models into moe models during instruction tuning has the potential to achieve great performance on general tasks. this conversion involves initializing each expert in the moe models as a copy of the feedforward neural network (ffn) layers @xcite . given the parameter scale of current llms, training such giant models requires updating the weights of experts in the moe layer, which is constrained by gpu memory resources and computational costs. to mitigate these challenges, we introduce parameter-efficient sparsity crafting (pesc), an approach that effectively expands model capacity while synergizing with parameter-efficient finetuning (peft) techniques @xcite . pesc involves inserting adapters @xcite into the moe layers of sparse models, allowing differentiation between experts without altering each expert's weights in the moe layers when guaranteeing the quality of the approximation in function space compared to original sparse upcycling @xcite . considering that the more sophisticated construction can improve the approximation @xcite , we also apply the qlora @xcite technique to update other weights in the sparse models. as shown in figure 1 , our camelidae-8×34b-pro, instruction fine-tuned utilizing pesc, achieved the best performance among various open-source sparse models and dense models. our contributions are described as follows: this approach involves tuning only the parameters added by the adapters. an adapter consists of two matrices, w @xmath0, coupled with a non-linear function σ(•). here, d 1 and d 2 denote the feature dimensions in the pretrained models and the adapter's hidden dimension, respectively, with d 2 < d 1 typically. given a feature @xmath1, the output of the adapter module is expressed as: mixture-of-experts. as depicted in figure 2 , an moe layer comprises n experts, {e i } n @xmath2, and a router r. the output y for an input x in the moe layer is computed as: for clarity, let us define f i (θ i ) as the objective function for the i-th expert in the moe layer, where θ i represents the parameters for e i . θ i is initialized from θ o , which are the parameters of the ffn layer f from the original dense model. the essence of the sparsity crafting training regimen lies in the optimization of f i (θ i ). the goal is to derive θ + i , the optimized parameters for each expert. this is formally expressed as: after the instruction tuning process utilizing the sparsity crafting technique, the optimized parameter sets {θ + i } n @xmath3. ## conclusion in this paper, we introduce parameter-efficient sparsity crafting (pesc) which upcycles dense models into sparse models utilizing the moe architecture. pesc incorporates adapters @xcite within the moe layers of sparse models, enabling the differentiation of experts without modifying the individual weights of each expert, and guarantees the quality of the approximation compared to traditional sparsity upcycling @xcite in function space (section 2.2). this technique significantly reduces computational costs and gpu memory requirements compared to sparse upcycling. it facilitates the expansion of model capacity with a minimal parameter increase due to the integration of adapters. we apply the pesc method to instruction tuning across various general tasks, resulting in notable performance enhancements on various benchmarks (section 3). additionally, we develop sparse models, camelidae, using the pesc approach and achieve superior performance across various open-source sparse models and demonstrate superior general capabilities compared to gpt-3.5.