D eep S eek M o E : Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models (2024)

Abstract: In the era of large language models, Mixture-of-Experts (MoE) is a promising architecture for managing computational costs when scaling up model parameters. However, conventional MoE architectures like GShard, which activate the top- K out of N experts, face challenges in ensuring expert specialization, i.e. each expert acquires non-overlapping and focused knowledge. In response, we propose the DeepSeekMoE architecture towards ultimate expert specialization. It involves two principal strategies: (1) finely segmenting the experts into mN ones and activating mK from them, allowing for a more flexible combination of activated experts; (2) isolating K_s experts as shared ones, aiming at capturing common knowledge and mitigating redundancy in routed experts. Starting from a modest scale with 2B parameters, we demonstrate that DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5 \times expert parameters and computation. In addition, DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with the same number of total parameters, which sets the upper bound of MoE models. Subsequently, we scale up DeepSeekMoE to 16B parameters and show that it achieves comparable performance with DeepSeek 7B and LLaMA2 7B, with only about 40% of computations.

## introduction recent research and practices have empirically demonstrated that, with sufficient training data available, scaling language models with increased parameters and computational budgets can yield remarkably stronger models @xcite @xcite @xcite . however, the endeavor to scale models to an extremely large scale is also associated with exceedingly high computational costs. considering the substantial costs, the mixture-of-experts (moe) architecture @xcite @xcite has emerged as a popular solution, which enables parameter scaling while concurrently keeping modest computational costs. despite the promising potential of moe architectures, existing moe architectures like gshard @xcite potentially suffer from issues of knowledge hybridity and knowledge redundancy: in response to the aforementioned issues, we introduce deepseekmoe, an innovative moe architecture specifically designed towards ultimate expert specialization. our architecture involves two principal strategies: (1) fine-grained expert segmentation: while maintaining the number of parameters constant, we segment the experts into a finer granularity by splitting the ffn intermediate hidden dimension. correspondingly, keeping a constant computational cost, we also activate more fine-grained experts to enable a more flexible and adaptable combination of activated experts. finegrained expert segmentation allows diverse knowledge to be decomposed more finely and be learned more precisely into different experts, where each expert will retain a higher level of specialization. in addition, the increased flexibility in combining activated experts also contributes to more accurate knowledge acquisition. (2) shared expert isolation: we isolate certain experts to serve as shared experts that are always activated, aiming at capturing and consolidating common knowledge across varying contexts. through compressing common knowledge into these shared experts, redundancy among other routed experts will be mitigated. this can enhance the parameter efficiency and ensure that each routed expert remains specialized by focusing on distinctive aspects. these architectural innovations in deepseekmoe offer opportunities to train a parameter-efficient moe language model where each expert is highly specialized. starting from a modest scale with 2b parameters, we validate the advantages of the deepseekmoe architecture. empirical results on 12 diverse benchmarks indicate that deepseekmoe 2b surpasses gshard 2b @xcite by a substantial margin, and even matches gshard 2.9b, a larger moe model with 1.5× expert parameters and computation. remarkably, we find that deepseekmoe 2b nearly approaches the performance of its dense counterpart with an equivalent number of parameters, which sets the strict upper bound of moe language models. we also conduct elaborate ablation studies and specialization analysis, and the studies validate the effectiveness of our main strategies, and provide evidence supporting that deepseek-moe can achieve higher expert specialization. subsequently, we scale up the model parameters to 16b and train deepseekmoe 16b on a largescale corpus with 2t tokens. evaluation results reveal that with only about 40% of computations, it achieves comparable performance with deepseek 7b @xcite and llama2 7b @xcite , two strong 7b dense models. our contributions are summarized as follows: (1) we introduce deepseekmoe, an innovative moe architecture aiming at achieving ultimate expert specialization. (2) we conduct extensive experiments to empirically validate the effectiveness of deepseekmoe and reveal its high level of expert specialization. (3) we scale up deepseekmoe to train a 16b moe model which shows strong performance. (4) we will release the code and model checkpoint of deepseekmoe 16b to the public. ## preliminaries we first introduce a generic moe architecture for transformer language models. a standard trans-former language model is constructed by stacking l layers of standard transformer blocks, where each block can be represented as follows: a typical practice to construct an moe language model usually substitutes feed-forward networks (ffns) in a transformer with moe layers at specified intervals @xcite @xcite . an moe layer is composed of multiple experts, where each expert is structurally identical to a standard ffn. then, each token will be assigned to a few experts. if the l-th ffn is substituted with an moe layer, its computation can be expressed as: ## deepseekmoe architecture on top of the generic moe architecture, deepseek-moe introduces two principal strategies, finegrained expert segmentation and shared expert isolation, as illustrated in figure 1 . both strategies aim at elevating the level of expert specialization. ## scaling up to deepseekmoe 16b with the deepseekmoe architecture, we further scale up our moe model to a larger scale with 16b total parameters and train it on 2t tokens. ## related work the mixture of experts (moe) technique is first proposed by @xcite ; @xcite to deal with different samples with independent expert modules. @xcite introduce moe into language model training and build a large-scale lstm-based @xcite moe models. as transformer become the most popular architecture for nlp, many attempts extend ffns in a transformer as moe layers to build moe language models. gshard @xcite and switch transformer @xcite are pioneers which employ learnable top-2 or top-1 routing strategies to scale the moe language models to an extremely large scale. hash layer @xcite and stablemoe @xcite use fixed routing strategies for more stable routing and training. @xcite propose an expert-choice routing strategy, where each token can be assigned to different numbers of experts. @xcite focus on the issues of training instability and fine-tuning difficulty in moe models, and propose st-moe to overcome these challenges. @xcite investigate parameterefficient moe architectures via sharing informa-tion among experts. @xcite attempt to improve the expert specialization on toy data. in addition to research on moe architectures and training strategies, recent years have also witnessed the emergence of numerous largescale language or multimodal models @xcite @xcite based on existing moe architectures. by and large, most of the previous moe models are based on conventional top-1 or top-2 routing strategies, leaving large room for improving expert specialization. in response, we design the deepseekmoe architecture to improve the expert specialization. ## conclusion in this paper, we introduce the deepseekmoe architecture for moe language models, with the objective of achieving ultimate expert specialization. through fine-grained expert segmentation and shared expert isolation, deepseekmoe achieves significantly higher expert specialization and performance compared with prevailing moe architectures. starting with a modest scale of 2b parameters, we validate the advantages of deepseekmoe, demonstrating its capability to approach the upper bound performance for moe models. further-more, we provide empirical evidence to show that deepseekmoe has a higher level of expert specialization than gshard. scaling up to a larger scale of 16b total parameters, we train deepseekmoe 16b on 2t tokens and demonstrate its outstanding performance comparable with deepseek 7b and llama2 7b, with only about 40% of computations. for research purposes, we will release the model checkpoint of deepseekmoe 16b to the public, which can be deployed on a single gpu with 40gb of memory. we aspire for this work to provide valuable insights for both academia and industry, and contribute to the accelerated advancement of large language models.