Dropping Experts, Recombining Neurons: Retraining-Free Pruning for Sparse Mixture-of-Experts LLM s (2025)

Abstract: Sparse Mixture-of-Experts (SMoE) architectures are widely used in large language models (LLMs) due to their computational efficiency. However, though only a few experts are activated for each token, SMoE still requires loading all expert parameters, leading to high memory usage and challenges in deployment. Previous work has tried to reduce the overhead by pruning and merging experts, but primarily focused on expert-level operations, leaving neuron-level structure underexplored. We propose **DERN** (**D**ropping **E**xperts, **R**ecombining **N**eurons), a task-agnostic and retraining-free framework for expert pruning and reconstruction. We observe that experts are often misaligned and contain semantic conflicts at the neuron level, which poses challenges for direct merging. To solve this, DERN works in three steps: it first prunes redundant experts using router statistics; then it decomposes them into neuron-level expert segments, assigning each segment to its most compatible retained expert; and finally, it merges segments within each retained expert to build a compact representation. Experiments on Mixtral, Qwen, and DeepSeek SMoE models show that DERN improves performance by more than 5% on commonsense reasoning and MMLU benchmarks under 50% expert sparsity, without extra training. It also greatly reduces the number of experts and memory usage, making SMoE LLMs easier to deploy in practice.

## introduction large language models (llms) have become the foundation models of modern nlp @xcite , with their application rapidly expanding into diverse domains such as multimodal learning @xcite , scientific research @xcite , and advanced reasoning @xcite . ## proposed method our dern framework performs expert pruning and merge through a three-stage process, as illus-trated in fig. 2 . first, we identify and retain the topk experts based on routing activations (sec. 3.1). next, pruned experts are decomposed into neuronlevel segments and reassigned to retained experts (sec. 3.2). finally, each expert merges its segments through clustering to reconstruct a compact structure (sec. 3.3). the overall procedure is summarized in alg. 1. ## conclusion we propose dern, a task-agnostic and retrainingfree framework for compressing smoe llms. by treating experts as modular assemblies of functional segments, dern performs decomposition and reconstruction based on neuron-level similarity. this enables compact expert representations that preserve model performance while reducing memory and latency. more broadly, dern highlights a core insight: structural reorganization at the sub-expert level offers a robust and adaptable path to efficiency in smoe.