Automated Fine-Grained Mixture-of-Experts Quantization (2025)

Abstract: The Mixture of Experts (MoE) architecture enables efficient model scaling through conditional computation, where only subset of parameters are activated per input. However, this distributed architecture poses unprecedented challenges for model compression, as conventional quantization methods optimized for dense networks prove inadequate. This paper introduces a specialized quantization framework for MoE architectures, motivated by our discovery that weight matrices across expert networks exhibit distinctive channel-wise outlier distributions, necessitating a more nuanced compression approach. Through theoretical analysis incorporating Fisher Information matrices and condition number characteristics, we establish a fundamental relationship between layer functionality and quantization sensitivity, demonstrating that down-projection layers inherently demand higher precision compared to up-projection layers. Leveraging these insights, we develop an automated channel-wise quantization framework that dynamically determines optimal bit-width allocations while maintaining minimal computational overhead through efficient statistical approximations. When evaluated on the Mixtral-8x7b-v0.1 architecture, our methodology demonstrates a 3.96% improvement over existing state-of-the-art approaches across natural language understanding benchmarks, while achieving superior compression ratios.

## introduction in recent years, large language models (llms) @xcite @xcite have demonstrated unprecedented advancements in natural language processing. however, this remarkable progress has been accompanied by an exponential increase in * corresponding author: rrji@xmu.edu.cn computational resource requirements. the mixture of experts (moe) architecture @xcite @xcite has emerged as a transformative paradigm in large language models, introducing efficient parameter scaling through selective activation of context-relevant parameter subsets. through strategic partitioning of the parameter space into specialized expert networks coupled with dynamic routing mechanisms, moe models achieve substantial performance gains while maintaining modest training costs and introducing only marginal computational overhead during inference @xcite @xcite . this innovative approach enables the processing of increasingly sophisticated tasks while optimizing resource utilization, marking a significant milestone in scaling language model capabilities @xcite . the exponential growth of language models has heightened the critical importance of model compression. various compression techniques, including quantization @xcite @xcite @xcite , sparsification @xcite , and knowledge distillation @xcite , have demonstrated remarkable efficacy in reducing model footprint while preserving performance. among these approaches, quantization has emerged as particularly promising, with recent advances successfully addressing the challenging outlier problem in large dense models through sophisticated weight compensation and affine transformation techniques. however, these quantization methodologies, originally designed for dense architectures, encounter unique challenges when applied to moe-llm architectures. the fundamental distinction lies in the dynamic routing mechanism @xcite , where input tokens are intelligently directed to specific experts based on semantic content. this sophisticated routing mechanism, combined with sparse computational patterns, renders conventional quantization techniques inadequate without compromising model performance @xcite . moreover, the inherent heterogeneity of expert structures indicates that individual experts within the moe framework possess distinct parameters and specialized functionalities. unlike traditional homogeneous neural networks, this architectural diversity suggests that each expert requires a tailored compression strategy, introducing additional complexity to the quantization process. the distinctive characteristics of moe architectures present unprecedented quantization challenges, particularly in managing outliers effectively across heterogeneous expert networks. recent research has begun addressing these challenges in moe compression. @xcite introduced a unified compression framework integrating expert slimming with quantization, marking an initial step toward comprehensive moe compression. additionally, @xcite investigated variable quantization bit-width allocation based on importance metrics across different structural granularities. however, these approaches predominantly focus on static structural metrics, without fully addressing the unique characteristics of moe architectures. the field currently lacks a specialized quantization solution engineered specifically for the distinctive nature of moe models, particularly in effectively managing heterogeneous expert structures. in response to the unprecedented challenges inherent in quantizing moe models @xcite )), we present a novel automated mixed-precision quantization framework specifically designed for moe-based large lan-guage models. our approach is fundamentally motivated by a critical empirical observation: weight matrices within moe architectures exhibit distinctive channel-wise outlier patterns, with substantial heterogeneity across different channels within individual experts. as illustrated in figure 1 and 2, this phenomenon is consistently observed across diverse moe architectures, with particularly pronounced patterns in down-projection layers. these findings fundamentally challenge conventional layer-wise quantization paradigms and underscore the necessity for more granular quantization strategies. through comprehensive theoretical analysis, we establish that various components within moe architectures demonstrate differential susceptibility to quantization effects. specifically, we prove that down-projection layers exhibit inherently greater sensitivity to quantization perturbations compared to their up-projection counterparts, as evidenced by their elevated condition numbers and fisher information metrics. this theoretical foundation provides principled guidance for optimal bit allocation across the model's architectural components. to reconcile these theoretical insights with practical deployment constraints, we introduce computationally efficient statistical approximation techniques that facilitate automated bit allocation without introducing additional training complexity. our methodology maintains rigorous theoretical guarantees while achieving the computational efficiency necessary for large-scale model deployment. experimental results demonstrate significant improvements over existing approaches, achieving a 3.96% performance gain on the mixtral-8x7b model under extreme low-bit configurations. the primary contributions of this work are: ## preliminary mixture of experts. the mixture-of-experts (moe) architecture represents a significant advancement in transformer models @xcite , superseding the conventional feed-forward network (ffn) sublayer with a more sophisticated dynamic structure. an moe layer incorporates n parallel ffn modules, designated as experts e 1 , e 2 , . . . , e n , alongside a router network @xcite that facilitates dynamic input allocation through a specialized gating mechanism. for an input vector @xmath0, the moe layer generates output @xmath1: here, s denotes the quantization step size, and clamp(•) represents a function that constrains values within a specified range. the quantization error, denoted as ϵ b (w ) = w -q b (w ), exhibits specific statistical properties. ## conclusion this paper presents an automated fine-grained quantization framework for moe models. our key theoretical contribution establishes the relationship between fisher information, condition numbers, and quantization sensitivity, providing principled guidance for bit allocation across different model components. through extensive empirical analysis, we demonstrate that moe weight matrices exhibit distinct channel-wise outlier patterns, necessitating fine-grained quantization approaches. our proposed framework successfully bridges theoretical insights with practical efficiency considerations through statistical approximations, enabling automated bit allocation without additional training overhead. limitations while our work demonstrates promising results in moe quantization, several limitations warrant discussion. first, our experimental validation, although thorough on mixtral-8x7b, deepseek-v2 and qwen-1.5-moe, is constrained to a limited set of moe architectures. despite the block-wise quantization approach being computationally feasible on a single gpu, resource constraints prevented us from extending our evaluation to emerging models such as deepseek-v3, potentially limiting our understanding of the framework's generalizability across different moe architectures. additionally, while our work focuses specifically on quantizationbased compression, it does not explore potential synergies with other compression techniques such as pruning, knowledge distillation, or structured sparsification. future work could investigate the integration of our fine-grained quantization framework with these complementary compression methods, potentially yielding more comprehensive and efficient moe compression solutions.