H yper M o E : Towards Better Mixture of Experts via Transferring Among Experts (2024)

Abstract: The Mixture of Experts (MoE) for language models has been proven effective in augmenting the capacity of models by dynamically routing each input token to a specific subset of experts for processing. Despite the success, most existing methods face a challenge for balance between sparsity and the availability of expert knowledge: enhancing performance through increased use of expert knowledge often results in diminishing sparsity during expert selection. To mitigate this contradiction, we propose HyperMoE, a novel MoE framework built upon Hypernetworks. This framework integrates the computational processes of MoE with the concept of knowledge transferring in multi-task learning. Specific modules generated based on the information of unselected experts serve as supplementary information, which allows the knowledge of experts not selected to be used while maintaining selection sparsity. Our comprehensive empirical evaluations across multiple datasets and backbones establish that HyperMoE significantly outperforms existing MoE methods under identical conditions concerning the number of experts. Our code is publicly available at https://github.com/Bumble666/Hyper_MoE

## introduction the accelerated advancement of large language models has culminated in their widespread application across various domains, including healthcare, education, and social interactions @xcite @xcite . the remarkable capabilities of these models are attributed to the enhancements in their scale. nevertheless, the scaling of dense models is often hampered by significant computational demands, posing a challenge to developing the natural language processing (nlp) community. in response, * corresponding authors. sparse activation models have emerged as a solution @xcite , activating only a subset of parameters for different inputs, thus mitigating computational costs. one of the most representative methods is the mixture of experts @xcite ), which routers different inputs to specific groups of experts, thereby enlarging the model's capacity without increasing computational burdens. the key to effectively reducing computational costs lies in the sparsity of expert selection, with the number of experts selected for each token being kept at a lower level. in practical applications or experiments, existing works @xcite @xcite ) usually select only one or two experts per input. however, increasing the number of selected experts per token can enhance the availability of expert knowledge and improve the performance of downstream tasks @xcite @xcite . this scenario positions moe model in a predicament akin to a zero-sum game: a choice between increasing the number of available experts to improve performance or preserving a lower level of available experts to ensure sparsity, as depicted in figure 1 . to mitigate this contradiction, one solution would be to use the knowledge of other experts to assist the sparsely selected experts. this is similar to multi-task learning, which transfers knowledge among related tasks. some works @xcite @xcite suggest using hypernetworks @xcite to generate task-specific knowledge to enhance positive transfer between tasks. inspired by this, we aim to increase the availability of expert knowledge by transferring the knowledge of unselected experts while sparsely selecting experts. in this paper, we propose hypermoe, a novel moe framework built upon hypernetworks, which captures the information from every expert by leveraging expert-shared hypernetwork while achieving positive expert transfer by generating conditioned modules individually. we refer to the information as cross-expert information. specifically, a hypermoe consists of hyperexperts, which are generated based on the information of unselected experts and serve as supplementary information for selected experts while maintaining sparsity. we further improve upon this by introducing the concept of cross-layer hypernetworks: a hypernetwork is shared among all transformer layers, which enables information flow among moes in different layers. this brings additional efficiency in terms of parameters and computational costs: despite the additional computation, our method only experienced a decrease foot_0 of approximately 15% in training speed and 10% in inference speed compared to the standard moe. we evaluate hypermoe on 20 representative nlp datasets across diverse tasks: sequence classification, extractive question answering, summariza-tion, and text generation. extensive experimental results show that hypermoe outperforms baselines, including switch transformer @xcite with moe architecture. this demonstrates the effectiveness of our method in transferring knowledge to experts, which increases the utilization of expert knowledge while keeping the number of experts selected at a low level. to summarise, our core contributions are: ## method overview. taking inspiration from knowledge transferring between different tasks in multi-task learning, we propose hypermoe. the key idea of hypermoe is to enhance the availability of knowledge for the current input through positive knowledge transfer between experts. through the condition input we designed, the relevant knowledge within the cross-expert information captured by the hypernetwork is encoded into hyperexpert, serving as supplementary information for the currently selected experts. in this work, we introduce conditional expert, in which we use shared hypernetworks to generate the expert weights based on the information of the unselected experts. the hypernetworks capture information across experts and transfer relevant knowledge to the selected experts by conditional generation. ## conclusion in this work, we introduce hypermoe, a novel mixture of experts (moe) architecture. inspired by the concept of knowledge transfer in multi-task learning, we propose a method to facilitate knowledge transfer between experts through conditional generation. our method enhances expert knowledge availability while maintaining expert selection's sparsity. we show the effectiveness of our approach across a wide range of nlp tasks. experimental results demonstrate that our method exhibits excellent performance compared to the conventional moe. furthermore, our analysis shows that without any measures, there could be negative knowledge transfer across experts when transferring knowledge to specific experts. our approach mitigates this issue by capturing the contextual information of experts. we explore the feasibility of knowledge transfer between experts in moe, providing a new perspective for future improvements in moe architectures.