Efficiently Editing Mixture-of-Experts Models with Compressed Experts (2025)

Abstract: Mixture-of-Experts (MoE) models have become a key approach for scaling large language models efficiently by activating only a subset of experts during training and inference. Typically, the number of activated experts presents a trade-off: fewer experts reduce computational costs, while more experts improve performance. Recent studies reveal that not all activated experts contribute equally to model performance, with some providing minimal utility, particularly when finetuning pretrained MoE models for specialized downstream tasks. The co-existence of significant and redundant parameters in experts provides us an opportunity to reduce the number of activated experts while maintaining model performance. In this work, we propose the concept of compressed experts, lightweight modules that serve as compact representations of full experts. Our approach preserves the most important experts while replacing other auxiliary activated experts with compressed experts. The reduction of active parameters significantly lowers inference costs while achieving comparable performance. Extensive experiments on models including Phi-MoE and OLMoE demonstrate that compressed experts recover over 90% of full expert performance across various tasks while reducing more than 30% active parameters and saving 20% in inference costs. This approach enables efficient deployment of MoE models in resource-constrained settings and facilitates scaling to larger models with manageable overhead.

## introduction mixture-of-experts (moe) models have emerged as an effective approach to scale up the sizes of large language models (llm) with minimal computational overhead @xcite @xcite . in * work done during an internship at microsoft. transformer-based moe models, the feed-forward networks (ffns) are replaced by moe layers, each containing multiple experts. for a given input, a routing network routes it only to a selected subset of relevant experts, ensuring that only a fraction of the network is activated during each forward pass. this sparse activation significantly reduces computational costs compared to dense models while maintaining high model capacity. in practice, moe models often require multiple activated experts to achieve desirable performance. for instance, mixtral @xcite and @xcite activate two experts, qwen-moe @xcite activates four and olmoe @xcite activates eight. the efficiency of moe models largely depends on the number of activated experts. specifically, the inference cost of moe models increases linearly with this number (as shown in figure 1 ). fewer activated experts reduce training and deployment costs as a smaller portion of the network is used. conversely, activating more experts increases utilization of model capacity, often leading to superior performance. thus, the selection of the number of activated experts poses a fundamental trade-off between efficiency and performance. recently, several studies highlight the potential redundancy in activated experts. for instance, @xcite demonstrates that increasing the number of activated experts yields diminishing returns in performance gains. additionally, @xcite shows that not all tasks necessitate full utilization of all the top-k experts. these findings suggest that only a subset of the activated experts, which we term main experts, contribute significantly to model utility, while others act as auxiliary experts with limited impact on performance. despite their lower contribution, passing through auxiliary experts incurs the same computational cost as main experts. this redundancy is particularly pronounced when finetuning a pretrained moe model on specific downstream tasks, where expert utility varies depending on task difficulty @xcite . this inefficiency motivates our approach: compressing auxiliary experts to reduce computational overhead while maintaining model performance. to achieve this, we outline three design principles: i) efficiency: the module should be computationally efficient, ensuring minimal overhead for training and inference. ii) expressiveness: it should retain the capabilities of the auxiliary experts to minimize performance loss. iii) flexibility: the module should adapt to various expert combinations, as the top-k experts may differ across tokens within the same sequence. following these principles, we propose compressed experts, which are embedding vectors serving as compact representations of auxiliary experts. in moe layers, we introduce one compressed expert per full expert, maintaining a one-to-one correspondence. these compressed experts are ex-tremely lightweight, with a dimensionality matching the hidden states and consisting of less than 0.05% foot_0 of the parameters of a full expert. during forward passes, the compressed experts corresponding to auxiliary experts are aggregated via a weighted sum and incorporated into the hidden states through element-wise multiplication. instead of activating both main and auxiliary experts, the augmented hidden states are processed only through main experts, significantly reducing computation while preserving contributions of auxiliary experts. we illustrate this approach in fig. 2 , with a top-2 moe layer containing one main expert and one auxiliary expert. in this example, the second expert is replaced by a compressed expert, reducing the number of forward passes from two to one. through extensive experiments on popular moe models including phi-moe (abdin et al., 2024) and olmoe @xcite , we demonstrate that compressed experts recover over 90% of full-expert performance across diverse tasks while reducing more than 30% of active parameters and cutting inference cost by 20%. by bridging the gap between performance and efficiency, compressed experts enable scalable and cost-effective deployment of moe models, making them more practical for resource-constrained environments. ## experiments to comprehensively evaluate the effectiveness of compressed experts, we integrate them into moe models during the supervised finetuning (sft) stage. we focus on the common scenario where a model is pretrained with a fixed number of ac- in section 3.1, we detail the experimental setup, including model configurations, datasets and training specifics. in section 3.2, we first present the performance of compressed experts on evaluation benchmarks, then show their the inference cost savings, demonstrating that compressed experts achieves an effective balance between performance and efficiency. finally, in section 3.3, we provide a detailed analysis on the task-specific performance of compressed experts. ## conclusions we introduce compressed expert, a lightweight and efficient approach to reducing the number of activated experts in moe models while maintaining strong performance. by replacing auxiliary experts with compact expert representations, our method significantly reduces computational overhead while preserving model capacity. extensive experiments on phi-moe and olmoe across various tasks demonstrate that compressed experts reduce active parameters by over 30% and cut inference costs by 20% while retaining over 90% of the performance of full-expert configurations. beyond reducing inference costs, our findings suggest broader implications for scaling moe models efficiently. compressed experts offer a promising direction for optimizing sparse activation in moe architectures.