Structured Pruning for Large Language Models Using Coupled Components Elimination and Minor Fine-tuning (2024)

Abstract: Large language models (LLMs) have demonstrated powerful capabilities in natural language processing, yet their vast number of parameters poses challenges for deployment and inference efficiency. Structured model pruning emerges as a viable approach to reduce model size and accelerate inference, without requiring specialized operators and libraries for deployment. However, structured pruning often severely weakens the model’s capability.Despite repetitive fine-tuning can restore the capability to a certain extent, it impairs LLMs’ utility as versatile problem solvers.To address this issue, we propose a novel structured pruning algorithm tailored for LLMs. It derives the importance of different components, namely rows and columns in parameter matrices, based on intermediate data dependencies. Then it removes coupled components across different layers simultaneously and preserves dependency relationships within remaining parameters, avoiding significant performance degradation. The pruned model requires only few epochs of fine-tuning to restore its performance, ensuring the model’s ability to generalize.Empirical evaluations on LLaMA, Vicuna, and ChatGLM3 demonstrate our algorithm’s efficacy, yielding 20% parameter reduction while retaining at least 94.4% of original performance metrics.

## introduction large language models (llms) have demonstrated powerful capabilities in solving a variety of general problems @xcite , particularly in language understanding and generating. however, the large number of parameters @xcite @xcite in llms poses significant challenges for deployment and inference efficiency. structured pruning @xcite @xcite has been proved to be a viable approach to compress deep neural networks. it removes entire structural components of the neural network, without requiring specialized operators and libraries for executing the pruned model, so that it is convenient for deployment and acceleration. despite structured pruning algorithms have long been investigated @xcite @xcite , they face new challenges when tackling llms. existing state-of-the-art pruning algorithms follow an iterative scheme @xcite @xcite for specific tasks. this scheme conducts iterative evaluating, pruning and finetuning on a large model for a single task, achieving low performance degradation. however, due to the repetitive fine-tuning on a single task, the pruned model has much less generalization ability on other tasks. this is a particularly serious issue for llms, since they are expected to be general-purpose models solving extensive problems. simply extending the fine-tuning on more corpus and tasks to reserve the generalization ability is still challenging @xcite , because llms require huge volume of training corpus. in this study, we propose a novel structured pruning algorithm tailored for llms. in contrast to existing iterative pruning works, our algorithm first conducts iterative evaluating and pruning, until the desired sparsity level is achieved. after completing all the iterations of evaluating and pruning, it then conducts one stage of fine-tuning, which involves few epochs of training on a small dataset. the intuition of our algorithm is to limit the finetuning operations as few as possible, so that the pruned model will not import too much bias towards specific tasks. to ensure that the remaining parameters are consistently important and do not need repetitive finetuning to restore performance, we need to precisely evaluate the importance of structured components, namely rows and columns in parameter matrices. more concretely, our algorithm derives the importance and uncertainty of different components based on intermediate data dependencies, as shown in figure (1). according to the transformer-based model architecture, we can identify the coupled components that have data dependency on pruned components. these coupled components across different layers can be removed simultaneously, and the dependency relationships within remaining parameters can be still preserved, avoiding significant performance degradation. moreover, we employ lora @xcite fine-tuning to restore model performance, and use lora gradients @xcite instead of full-scale fine-tuning gradients to reduce the computational overhead during pruning. the model pruned by our algorithm preserves the original architecture with smaller parameter matrices, thus it is compatible to any other transformer-specific optimization techniques, e.g, flashattention @xcite . we have validated our algorithm on llama @xcite , vicuna @xcite , and chatglm3 @xcite , achieving about 20% parameter reduction while retaining at least 94.4% of original performance metrics. contribution. in this paper, (i) we propose a new structured pruning algorithm for llms that uses minimal fine-tuning to recover model performance. the algorithm effectively reduces the number of parameters while maintaining model generalization. (ii) we propose a novel evaluation method that evaluates the impact of structured pruning on an llm by evaluating coupled components instead of individual weights. (iii) we conduct our algo-rithm on representative llms, including llama, vicuna, and chatglm3. by reducing the parameter count by 20%, we maintain at least 94.4% of the model's performance while reducing macs by 20%. ## method our pruning consists of three steps. (i) partitioning the model into kernels and features, and grouping the coupled components formed by kernels. (ii) iteratively evaluating and pruning coupled components and features until the desired sparsity level is achieved. (iii) after all evaluating and pruning finish, a fine-tuning stage is conducted to restore the model performance. ## conclusion in this paper, we propose a structured pruning algorithm for llms. our algorithm categorizes parameters into kernels and features based on their relationships between parameter matrices and word vectors in computations. we evaluated these components considering their coupling relationships and the computational characteristics of transformer architecture. experimental evaluations on llama, vicuna, and chatglm3 models demonstrated that our algorithm achieves compression to 20% of the original size with minor performance degradation. our algorithm preserves the model structure, facilitating integration with other optimization techniques and practical deployment.