XM o E : Sparse Models with Fine-grained and Adaptive Expert Selection (2024)

Abstract: Sparse models, including sparse Mixture-of-Experts (MoE) models, have emerged as an effective approach for scaling Transformer models. However, they often suffer from computational inefficiency since a significant number of parameters are unnecessarily involved in computations by multiplying values by zero or low activation values. To address this issue, we present XMoE, a novel MoE designed to enhance both the efficacy and efficiency of sparse MoE models. XMoE leverages small experts and a threshold-based router to enable tokens to selectively engage only essential parameters. Our extensive experiments on language modeling and machine translation tasks demonstrate that enhances model performance and can decrease the computation load at MoE layers by over 50% without sacrificing performance. Furthermore, we present the versatility of by applying it to dense models, enabling sparse computation during inference. We provide a comprehensive analysis and make our code available at https://anonymous.4open.science/r/XMoE .

## introduction recently, remarkable advancements in large language models have been achieved through scaling up their sizes @xcite @xcite . however, this progress has come with a significant increase in training costs, posing a challenge to further scaling. to address this issue, sparse models, such as sparse mixture-of-experts (moe) models, have emerged as an alternative approach. these models allow for scaling the model size without a corresponding increase in computational cost @xcite @xcite @xcite . the key to this ability lies in sparsely activated moe layers. these layers comprise multiple sub-networks, or "experts", typically implemented as feed-forward networks (ffns) @xcite . unlike traditional models that utilize all parameters for each input token, moe models selectively activate a subset of experts. this approach effectively decouples computational costs from model size, paving the way for more efficient scaling. while moe models are effective for scaling, this paper argues that moe models exacerbate the issue of computational inefficiencies. initially identified in the dense model t5 @xcite , this issue is characterized by a significant portion of computations in the ffn layer being wasted on multiplying values by zero @xcite token 1 top-1 router token 2 expert 1 expert 2 adaptive router expert 1 expert 2 expert 3 expert 4 token 1 token 2 token 1 token 2 token 1 token 2 token 2 a) vanilla moe b) xmoe et al., 2023b). as illustrated in figure 1 , the computational inefficiency issue is also prevalent in sparse models and even worsens as the number of experts increases. this observation suggests that only a small portion of the parameters in an expert is useful for the input, while others are unnecessarily involved in the computation. consequently, selecting one expert for each input can already lead to a significant waste of computation. in order to alleviate this problem, a more fine-grained and adaptive strategy for parameter selection is necessary. figure 2 shows the overview of a novel moe design, named xmoe, which allows tokens to select fewer parameters to improve the efficiency without hindering model performance. to achieve this, xmoe proposes to exploit small experts and a threshold-based router. first, considering that expert is the smallest unit of parameter selection in moe models, utilizing small experts is the prerequisite for the more fine-grained selection. it allows models to choose the useful parameters precisely without activating the redundant parameters. in order to ensure the effectiveness, a novel adaptive router is further exploited by xmoe. different from the widely-used top-k router that dispatches each input to a fixed number of experts, this adaptive router allows tokens to self-determine the number of required experts based on a pre-defined threshold. intuitively, an easily processed token can be routed to a single small expert, while a critical token may require multiple experts @xcite ). an adaptive router allows models to leverage the difference of the input complexity to dynamically allocate computational resources. this not only enhances model efficiency but also yields potential quality improvements when computational resources are constrained. in conjunction with the aforementioned design, xmoe aims to enhance the efficiency of moe mod-els, with a focus on improving quality with a fixed computational budget or reducing computational costs without compromising performance. extensive experiments in language modeling and machine translation demonstrate performance gains over existing moe methods. xmoe also enables a reduction in floating point operations (flops) by over 50% with minimal impact on performance. additionally, our investigation extends to training dense models using xmoe to facilitate sparse computation during inference. this approach not only matches the performance of dense counterparts but also facilitates a substantial reduction in flops. ## conclusion this paper proposes a novel moe design, xmoe, with the primary objective of improving the efficacy and efficiency of sparse moe models. by employing small experts and a threshold-based router, xmoe demonstrates performance enhancements while significantly reducing flops, leveraging the inherent sparsity of the model. our research sheds light on the utilization of sparsity to improve model quality. as for future directions, we aim to further harness the advantages of sparse computation, fo-cusing on enhancements from both hardware and algorithmic perspectives.