M o E -I ^2 : Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition (2024)

Abstract: The emergence of Mixture of Experts (MoE) LLMs has significantly advanced the development of language models. Compared to traditional LLMs, MoE LLMs outperform traditional LLMs by achieving higher performance with considerably fewer activated parameters. Despite this efficiency, their enormous parameter size still leads to high deployment costs. In this paper, we introduce a two-stage compression method tailored for MoE to reduce the model size and decrease the computational cost. First, in the inter-expert pruning stage, we analyze the importance of each layer and propose the Layer-wise Genetic Search and Block-wise KT-Reception Field with the non-uniform pruning ratio to prune the individual expert. Second, in the intra-expert decomposition stage, we apply the low-rank decomposition to further compress the parameters within the remaining experts. Extensive experiments on Qwen1.5-MoE-A2.7B, Deepseek-V2-Lite, and Mixtral-8 \times 7B, demonstrate that our proposed methods can both reduce the model size and enhance inference efficiency while maintaining performance in various zero-shot tasks.

## introduction large language models (llms) have recently demonstrated remarkable language understanding and generation proficiency, excelling in complex tasks @xcite @xcite . however, deploying these models presents substantial challenges due to their significant storage and computational demands. to overcome these issues, the mixtureof-experts (moe) llm has been proposed @xcite , which activates only a subset of its * equal contribution. parameters during training and inference. for instance, with a smaller model size, the mixtral-8×7b model with a total of 47b parameters surpasses the performance of dense transformer models like llama-2-70b @xcite . additionally, qwen1.5-moe-a2.7b @xcite demonstrates highly competitive performance compared to other 7b models, and the recently introduced deepseekv2 moe @xcite achieves performance levels comparable to gpt-4, demonstrating the powerful capabilities of moe models. moe models have garnered significant attention recently due to their ability to dynamically select subsets of parameters for each input, enabling efficient handling of diverse tasks. despite their potential, a notable challenge with moe models is that they are still burdened by substantial parameter size and computation cost. for example, mixtral-8×7b @xcite ) not only has 47b parameters but also activates 13b parameters during inference. while this architecture allows for scalability and flexibility, it also introduces complexities and huge memory in deployment and inference, particularly when considering resource constraints and efficiency. consequently, decreasing and maintaining these large-scale models remains a critical area of research. model compression techniques, such as pruning, knowledge distillation, and quantization, have been utilized to slim the model size. @xcite proposed to reduce the parameter count of moe models by expert pruning, but it does not reduce the parameters during inference efficiently. @xcite merges several experts into one and applies the low-rank decomposition to further reduce the model size. although this approach achieves a good compression ratio and performance, it requires calibration and fine-tuning for each downstream task individually, which is not suitable for large-scale llms, and time costs are very high. several works @xcite @xcite focus on unstructured sparsity to decrease the parameters of models while maintaining high performance. however, unstructured pruning struggles to achieve practical acceleration, decrease inference, and save storage without a specific design for hardware and libraries. to solve these problems, we start by analyzing parameter redundancy in the moe model from multiple levels. first, since identifying redundant experts using brute-force search @xcite is infeasible in practice, it is necessary to design efficient methods to reduce the time complexity. second, we aim to compress as many experts as possible while ensuring that the model maintains its zero-shot performance, rather than being limited to handling a single down-stream task @xcite . finally, our method can adapt to any moe model, particularly those with a large number of experts and diverse structures, and automatically identifies a suitable compression strategy for each type of moe model without the need for manual settings. in this paper, we propose a novel end-to-end framework for moe models, moe-i 2 , for the taskagnostic compression of the moe models. to our knowledge, moe-i 2 is the first end-to-end framework designed task-agnostic for structured compression of moe llms. our contributions are summarized as follows: ## method in this section, we introduce the details of our proposed framework, moe-i 2 , which consists of three stages: inter-expert pruning stage (sec. 3.1), intra-expert decomposition stage (sec. 3.2), and finetuning stage (sec. 3.3). the overall pipeline is shown in figure 1 .