B lock P runer: Fine-grained Pruning for Large Language Models (2025)

Abstract: With the rapid growth in the size and complexity of large language models (LLMs), the costs associated with their training and inference have escalated significantly. Research indicates that certain layers in LLMs harbor substantial redundancy, and pruning these layers has minimal impact on the overall performance. While various layer pruning methods have been developed based on this insight, they generally overlook the finer-grained redundancies within the layers themselves. In this paper, we delve deeper into the architecture of LLMs and demonstrate that finer-grained pruning can be achieved by targeting redundancies in multi-head attention (MHA) and multi-layer perceptron (MLP) blocks. We propose a novel, training-free structured pruning approach called BlockPruner. Unlike existing layer pruning methods, BlockPruner segments each Transformer layer into MHA and MLP blocks. It then assesses the importance of these blocks using perplexity measures and applies a heuristic search for iterative pruning. We applied BlockPruner to LLMs of various sizes and architectures and validated its performance across a wide range of downstream tasks. Experimental results show that BlockPruner achieves more granular and effective pruning compared to state-of-the-art baselines.

## introduction large language models (llms) @xcite have demonstrated outstanding performance across a diverse array of natural language processing tasks. however, their growing size and complexity have led to substantial computational demands and increased memory usage, creating obstacles for deployment in resourceconstrained environments. model compression techniques @xcite @xcite have emerged as a promising solution figure 1 : block influence (bi) scores @xcite for the llama2-7b model @xcite computed at both layer and block levels, where blocks/layers with lower bi scores indicate less importance. the model has 32 transformer layers, each containing one mha and one mlp block, totaling 64 blocks. blocklevel bi scores are generally lower than layer-level scores, indicating finer-grained redundancies. to address the challenges of deploying large, computationally intensive models. these techniques aim to transform large models into more compact versions that require less storage and execute with lower latency, while minimizing performance degradation. model compression methods typically involve knowledge distillation @xcite , quantization @xcite , and pruning (van der @xcite . in this study, we primarily focus on pruning, a technique that can be combined with these other methods to achieve more effective and efficient compression. recent research on layer redundancy has shown that llms contain a substantial number of redundant layers @xcite @xcite . removing these layers does not severely impact the model's performance. to quantify this redundancy, researchers have investigated various similarity-based measurement methods and developed corresponding pruning strategies, includ-ing layer merging @xcite and layer removal @xcite . these methods not only maintain the original width of the model architecture and avoid introducing additional structures, but also demonstrate superior performance. furthermore, @xcite posited that this observed redundancy may be intrinsically linked to the residual structure @xcite inherent in the transformer architecture. building on this intuition and recognizing that transformer layers can be further subdivided into smaller residual blocks, namely multi-head attention (mha) and multi-layer perceptron (mlp) foot_0 , we hypothesize that fine-grained block redundancies could exist within llms. consequently, we conducted a preliminary experiment to assess the significance of blocks at varying granularities. specifically, we sampled 32 instances from the alpaca dataset @xcite and employed the block influence (bi) metric @xcite to evaluate blocks at layer and block levels, as depicted in figure 1 . the results reveal that block-level bi scores are generally lower than layer-level bi scores, indicating that fine-grained redundancies at the block level are more significant within the model. building on these findings, we argue that finergrained pruning can be effectively implemented in llms. therefore, we introduce blockpruner, a novel, training-free structured pruning approach. unlike existing methods that focus on entire layers, blockpruner segments each transformer layer into mha and mlp blocks. it then evaluates the importance of these blocks using perplexity measures and applies a heuristic search for iterative pruning. to validate the effectiveness of our method, we applied blockpruner to six llms of varying sizes and architectures, and evaluated their performance using five representative benchmarks. our experimental results demonstrate that blockpruner provides more granular and effective pruning compared to state-of-the-art baselines. additionally, we performed a series of analytical experiments to investigate the impact of block type, block importance metrics, and data on pruning effectiveness. our findings confirm that llms contain substantial redundancies at the block level compared to the layer level, demonstrating that fine-grained pruning is more effective and appropriate than layer-based approaches for compressing these models. ## related work pruning is a well-established technique to compress and accelerate neural networks by removing superfluous weights or structures within models. pruning methods can be broadly categorized into unstructured pruning and structured pruning. unstructured pruning. unstructured pruning targets individual weights, eliminating redundant connections in neural networks by setting the corresponding weights to zero. for instance, sparsegpt (frantar and alistarh, 2023) formulates pruning as a layer-wise sparse regression problem, approximately solving it via a sequence of efficient hessian updates and weight reconstructions. wanda @xcite computes the importance score of each weight based on the product of the magnitude of each weight and the norm of the corresponding input activation, identifying and removing weights with lower importance scores. owl @xcite identifies the correlation between pruning efficacy and the retention ratio of outliers, assigning different sparsity ratios to each layer based on the observed outlier ratio. ria @xcite introduces a metric that considers both weight and activation information, utilizing a permutation strategy for the input channels of weight matrices to enhance pruning performance. besa @xcite adopts a layer-wise pruning strategy, independently pruning each transformer layer to minimize the reconstruction error between the outputs of pruned and dense transformer layers, which avoids accumulating errors across layers. ## methodology the proposed fine-grained block pruning method (blockpruner) is depicted in figure 3 . it begins by decomposing each transformer layer into two minimal residual blocks ( §3.1). we then evaluate the importance of each block by leveraging perplexity for our iterative block pruning framework ( §3.2). finally, we iteratively prune the block with the lowest importance ( §3.3). ## analyses 5.1 ablation study to assess the influence of various key operations within the proposed pruning algorithm on its performance, we undertake a thorough ablation study across six models. in particular, we first remove all blocks with the lowest importance scores at once, without the iterative search procedure. then, we substitute the fine-grained block pruning with a coarser-grained layer pruning approach. the results of these experiments are shown in the experimental findings highlight that solely relying on the perplexity metric without incorporating a search component can result in subpar pruning results and even performance deterioration. this phenomenon may stem from the intrinsic nature of perplexity, which, unlike other importance metrics focusing solely on local block influence, is inherently influenced by the interaction among multiple blocks due to its derivation from the model's ## conclusion in this work, we introduce blockpruner, a novel structured pruning approach for efficiently pruning llms. blockpruner decomposes transformer layers into two minimal residual blocks and leverages a block importance metric based on perplexity in conjunction with an iterative pruning search algorithm, where the two components work together to progressively eliminate redundant blocks. extensive experiments across various models show that our method outperforms other baselines in post-pruning performance. our findings uncover fine-grained block redundancy in llms, highlighting significant differences in redundancy levels across different block types. we hope our work contributes to a deeper understanding of the importance of different blocks within llms.