Bag of Tricks for Sparse Mixture-of-Experts: A Benchmark Across Reasoning, Efficiency, and Safety (2025)

Abstract: Mixture-of-Experts (MoE) has emerged as a promising approach for scaling large language models efficiently. However, how to design a desired MoE architecture given performance, efficiency, or safety goals remains absent. Existing benchmarks often focus on isolated aspects (e.g., reasoning, efficiency, safety), and there is a lack of consensus on optimal design choices, such as the number and size of experts, the type of routers, and the regularization during pre-training, or strategies like freezing, learning rate adjustments, and limiting expert collaboration during fine-tuning, with prior works often yielding conflicting conclusions. Motivated by this research gap, we introduce MoEBench, the first comprehensive assessment of MoE designs across the three dimensions of reasoning ability, efficiency, and safety. Our benchmark systematically evaluates optimal architectural choices during both pre-training and fine-tuning phases. We evaluate two popular MoE backbones across four dimensions of design choices on over eight metrics. Our empirical findings uncover hidden underlying correlations among MoE design choices. Specifically, we observe that (1) token-level routing and z-loss regularization improve reasoning performance; (2) shared experts enhance training stability but reduce specialization; and (3) collaboration-constrained routing and freezing strategies significantly influence load balance, specialization, and safety alignment. Furthermore, we propose three “sweet point” combinations of optimal strategies tailored to different scenarios. We hope this study provides actionable insights for building more robust, efficient, and secure MoE models. Code, checkpoints, and raw data will be released upon acceptance of the paper.

## introduction the pursuit of enhanced capabilities in large language models (llms) has demonstrated a strong correlation between model scale and performance @xcite @xcite . however, naively increasing model size leads to prohibitive computational costs for training and substantial inference latency @xcite . to mitigate these challenges, various techniques have been investigated, including quantization @xcite , lowprecision training @xcite , and sparse computing @xcite @xcite . among these diverse approaches, mixture of experts (moe) has emerged as a particularly compelling paradigm, adopted by numerous state-of-the-art (sota) models @xcite @xcite . the fundamental principle of moe architectures involves selectively activating only a sparse subset of the model's parameters for each input token, thereby decoupling the total parameter count from the per-token computational load. moe first demonstrated significant success in language modeling @xcite and machine translation @xcite , and its use has since been extended to computer vision @xcite and multi-modal applications @xcite . a primary driver of moe's success lies in its capacity to expand model size cost-effectively while maintaining favorable scaling properties @xcite . moreover, the inherent sparse activation mechanism within moe models offers potential avenues for enhanced model interpretability @xcite . despite the remarkable success of moe models, their intricate design introduces significant complexities. the choice of expert architecture and routing mechanisms, for instance, profoundly impacts the model's reasoning capabilities @xcite . furthermore, considerations such as inter-expert load balancing and collaborative behavior necessitate the introduction of auxiliary regularization techniques to left: evaluation of various moe designs during pretraining and fine-tuning across reasoning ability, safety, and efficiency dimensions using over six metrics, including analysis of their intercorrelations. right: "sweet point" fine-tuning techniques demonstrate comprehensive improvements over the baseline in performance, safety, and efficiency. none denotes full-parameter fine-tuning; none->router denotes full-parameter fine-tuning for the initial training phase, then fine-tuning with a frozen router. maintain efficiency and performance @xcite @xcite . compared to their dense counterparts, moe models offer additional avenues for optimization, such as expert pruning @xcite and enhancing inter-expert communication protocols @xcite , which can further reduce inference latency but also add layers to the design complexity. however, existing research on moe models predominantly focuses on singular evaluation dimensions, such as reasoning ability, often neglecting a holistic assessment across multiple critical aspects including efficiency and safety @xcite . critically, prevalent metrics for evaluating load balancing are often inadequate for assessing the efficacy of many contemporary moe optimization strategies utilizing expert activation or collaboration @xcite . this yields the primary question to be explored: motivated by these limitations and the research question, we introduce moe-bench, a comprehensive benchmark designed to rigorously evaluate moe architectures across multiple performance dimensions. as illustrated in figure 1 , moe-bench quantifies and assesses a suite of metrics spanning three key domains: (1) we introduce novel concepts of expert capacity and specialization to gauge the importance and potential redundancy of individual experts. (2) we incorporate metrics for evaluating expert collaboration, offering insights for the development of advanced optimization techniques. (3) we conduct a thorough assessment of model-generated content for hallucinations and safety. furthermore, moe-bench systematically explores the vast moe design space, scrutinizing a range of moe-specific hyperparameters, including expert granularity, load balancing schemes, and routing strategies, to identify optimal design configurations. our contributions are summarized as follows: ## bag of tricks for improving moe efficiency 4.1 evaluation metrics to systematically evaluate the efficiency of moe models, we adopt four key metrics, each reflecting a distinct aspect of expert utilization: load balance. we measure the uniformity of expert selection using the entropy of the routing distribution h(p) = -n @xmath0, where p i denotes the probability of selecting expert i among n experts. a higher entropy indicates a more balanced load across experts, which is desirable for efficient resource utilization @xcite . ## bag of tricks for improving moe safety due to space constraints, please refer to appendix a for the discussion on safety. ## conclusion in this work, we present moe-bench, the first comprehensive benchmark for evaluating moe architectures across reasoning, efficiency, and safety. our findings reveal key trade-offs and highlight @xmath1, balancing resource efficiency and model performance. these results emphasize the importance of co-designing pretraining and fine-tuning to build moe models that are not only powerful but also safe and efficient. moe-bench offers a solid foundation for advancing responsible moe development. we acknowledge that our work does not propose a new algorithmic method, but rather centers on benchmarking. our primary contribution lies in designing and executing a systematic and reproducible evaluation that unifies the critical axes of reasoning, efficiency, and safety. by consolidating these dimensions into a unified framework, accompanied by an open codebase, we surface hidden trade-offs and provide a robust foundation for future moe research. we believe this fills an important gap in the literature and enables more informed and targeted development of moe architectures.