Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models (2024)

Abstract: Parameter-efficient fine-tuning ( PEFT ) is crucial for customizing Large Language Models (LLMs) with constrained resource. Although there have been various PEFT methods for dense-architecture LLMs, PEFT for sparse-architecture LLMs is still underexplored. In this work, we study the PEFT method for LLMs with the Mixture-of-Experts (MoE) architecture and the contents of this work are mainly threefold: (1) We investigate the dispersion degree of the activated experts in customized tasks, and found that the routing distribution for specific task tend to be highly concentrated, while the distribution of activated experts varies significantly across different tasks. (2) We propose the expert-specialized fine-tuning method, which tunes the experts most relevant to downstream tasks while freezing the other experts; experimental results demonstrate that our method not only improves the tuning efficiency, but also matches or even surpasses the performance of full-parameter fine-tuning. (3) We further analyze the impact of the MoE architecture on expert-specialized fine-tuning. We find that MoE models with finer-grained experts are more advantageous in selecting the combination of experts that are most relevant to downstream tasks, thereby enhancing the both the training efficiency and effectiveness.

## introduction as the parameter scale of large language models (llms) continues to increase @xcite @xcite , parameter-efficient fine-tuning (peft) methods @xcite are becoming increasingly important in adapting pre-trained llms to downstream customization tasks. however, existing works on peft like low-rank adaptation (lora) and p-* work done during internship at deepseek. tuning @xcite have primarily focused on dense-architecture llms, with research on sparse-architecture llms still being markedly insufficient. in this work, we focus on exploring peft techniques within the mixture-of-experts (moe) llms @xcite , as introduced in §3.1. unlike dense models where all tasks are handled by the same parameters, in the moe architecture, different tasks are processed by distinct activated experts @xcite . observations indicate that task specialization in expert systems is the key to the moe llm performance @xcite . we further illustrate such specialization in §3.2 that experts activated by the same task's data are concentrated, while those for different tasks vary significantly, suggesting moe models use specialized expert combinations to handle different tasks. motivated by this, we propose expert-specialized fine-tuning (esft), as illustrated in §3.3. esft only tunes the experts with the highest affinity to the task, while freezing the parameters of other experts and modules. the primary advantages of esft lie in two aspects: (1) maintaining expert specialization: esft prevents the decrement of specialization in full-parameter fine-tuning, where experts not adept at the task also update their parameters. experimental results in §5.1 show that esft can achieve aligned or even superior performance in downstream tasks compared to full-parameter finetuning, and better maintains performance in general tasks. (2) saving computation resources: esft only trains the parameters of the selected experts, which effectively reduces the storage of up to 90% and training time up to 30% compared to full-parameter fine-tuning, as shown in §5.2. besides, we delve deeper into the working mechanism of the esft method. we analyze the expert selection process in §6.1 and demonstrate how esft leverages specialized experts effectively, as selecting 5-15% experts can achieve promising performance in different tasks. we investigate the efficiency of esft under different computational constraints in §6.2, showcasing its ability to leverage training resources efficiently compared to other peft methods like lora. our studies in §6.3 analyze the effects of shared and non-shared parameters in the model on specialized and general performance, pointing out the priority to selectively train non-shared parameters in esft. through ablation studies in §6.4, we highlight the importance of our expert relevance scores and the fine-grained expert segmentation architecture. ## analysis in this section, we investigate the expert selection process of esft in §6.1, and demonstrate the performance of esft and lora under different computational constraints in §6.2. we analyze the effects of training shared and non-shared parameters in §6.3, and conduct ablation studies in §6.4 to verify the importance of our expert relevance scores and model structure of fine-grained experts. ## conclusion in this work, we study parameter-efficient finetuning methods for sparse large language models with the mixture of experts (moe) architecture. we first observe that tasks from different domains are handled by distinct combinations of experts. we then propose selecting the most relevant experts for downstream tasks using two metrics: average gate score and token selection ratio. experimental results show that our method significantly reduces training costs while matching or surpassing full parameter fine-tuning results. further analysis confirms that our method enhances the specialization of the expert system within the moe architecture.