PIP : Perturbation-based Iterative Pruning for Large Language Models (2025)

Abstract: The rapid increase in the parameter counts of Large Language Models (LLMs), which often reach into the billions or even trillions, presents significant challenges for their practical deployment, particularly in resource-constrained environments. To address this issue, we propose PIP (Perturbation-based Iterative Pruning), a novel double-view structured pruning method to optimize LLMs, which combines information from two different views: the unperturbed view and the perturbed view. With the calculation of gradient differences, PIP iteratively prunes those that struggle to distinguish between these two views. Our experiments show that PIP reduces the parameter count by approximately 20% while retaining over 85% of the original model’s accuracy across varied benchmarks. In some cases, the performance of the pruned model is within 5% of the unpruned version, demonstrating PIP’s ability to preserve key aspects of model effectiveness. Moreover, PIP consistently outperforms existing state-of-the-art (SOTA) structured pruning methods, establishing it as a leading technique for optimizing LLMs in constrained environments.

## introduction large language models (llms) @xcite based on the transformer architecture @xcite have demonstrated impressive capabilities across a wide range of tasks, but their capabilities come at the expense of massive parameter counts and high computational requirements @xcite . for illustration, the llama3-405b model @xcite , with about 405 billion parameters, demands at least 810 gb of memory with 11 a100 gpus when using half-precision (fp16) format. therefore, an issue presents itself that warrants further exploration: can we produce a smaller, general-purpose, * corresponding author: xujj@suda.edu.cn and competitive llm by leveraging existing pretrained llms, while using much less compute than training one from scratch? @xcite to this end, researchers have been exploring strategies like pruning @xcite @xcite , quantization @xcite , knowledge distillation @xcite , and low-rank factorization @xcite . among them, pruning methods have gained considerable attention due to their potential to significantly reduce model size while preserving performance. yet, traditional pruning strategies typically assess importance through isolated metrics such as weight magnitudes or input-output similarity @xcite . while intuitive, these single-view approaches suffer from fundamental flaws: they overlook the capacity to preserve semantic robustness under adversarial or natural input variations, which is essential for ensuring reliable deployment of models in real-world applications. to overcome the limitation, we propose a novel double-view approach that evaluates layer importance based on their awareness of text perturbations. for each input, we generate two complementary perspectives: an original sample and its perturbed counterpart-crafted via character-level edits that preserve syntax but maximally distort meaning (section 3.1). by contrasting parameter gradients between these views through first-order taylor approximation, we identify layers exhibiting weak semantic discrimination. beyond the initial double-view comparison, our approach employs an iterative gradient reassessment strategy to further refine the pruning process. after pruning the least sensitive layers identified in each cycle, we proceed to recompute gradient differences on the updated architecture. this dynamic process, which is akin to curriculum learning, progressively focuses on layers that are critical for semantic stability. by doing so, it ensures a thorough and comprehensive importance evaluation through successive approximations. our contributions are summarized as follows: ## related work pruning techniques for llms can generally be classified into two categories: unstructured pruning and structured pruning. unstructured pruning sparsifies weight matrices by setting individual elements to zero, which often requires specialized hardware support. notable works in this area include @xcite . structured pruning, on the other hand, focuses on eliminating predefined units within the model, which makes it more compatible with hardware constraints. the concept of structured pruning for llms is introduced by @xcite , which proposes parameterizing each weight matrix via low-rank factorization and actually adaptively removing rank-1 components during training. this pioneering work has truly led to the development of several other methods, such as @xcite and @xcite . these methods, however, are primarily designed for compression within specific domains or tasks, falling under the category of taskspecific compression. while effective for their intended applications, they often limit the versatility of llms as general task solvers. in contrast, @xcite introduces a genetic pruning framework called llm-pruner, which aims to maintain task-agnostic capabilities while minimizing reliance on the original training dataset. following the pipeline proposed by @xcite , llm-pruner consists of three stages: discovery, estimation, and recovery. it selectively removes non-critical coupled structures based on dependency analysis @xcite , preserving the core functionality of the model. however, its potential integration with lora @xcite presents several challenges in achieving an optimal balance between efficiency and performance. inspired by the approach of llm-pruner, the research community has proposed several methods for structured pruning in general tasks. these methods can be broadly categorized into two main types: width pruning and depth pruning @xcite . width pruning focuses on compressing the weight matrix by reducing its hidden dimension, while depth pruning targets the pruning of layers or blocks within the model. for example, shearedl-lama @xcite implements structured pruning through a combination of targeted pruning and dynamic batch loading. targeted pruning removes specific layers of the model in an end-toend fashion to achieve a predefined compression ratio. dynamic batch loading adjusts the composition of training data batches based on the varying losses from different domains. although this method achieves competitive performance, it suffers from the same retraining challenges as llm-pruner @xcite . to avoid retraining, which can be resourceintensive and time-consuming, @xcite proposes shortgpt, a method based on layer importance. it introduces a novel importance metric called block influence, which quantifies the importance of each layer by calculating the similarity between the inputs and outputs of each layer. layers with low importance scores are then removed. similarly, @xcite proposes shortened llama, a block-importance-based method that removes blocks based on a block-level importance metric. another related work, sleb @xcite evaluates the importance of transformer blocks using the similarity between inputs and outputs, and removes the blocks with low importance scores. while these methods are straightforward to understand and implement, they fail to provide strong empirical results and lack rigorous theoretical support. moreover, these single-view approaches are inherently limited as they neglect the necessity to maintain semantic robustness under adversarial or natural input variations, which is essential for reliable deployment. in summary, while existing pruning methods offer trade-offs in terms of model efficiency and performance, they often either require retraining or lack solid guarantees in theory, limiting their applicability to real-world scenarios. ## conclusion and future work in this paper, we propose pip, a novel perturbationbased iterative structured pruning method that unifies unperturbed and perturbed model views. it demonstrates theoretical rigor and sota performance across multiple benchmarks. future work could explore adaptive perturbation mechanisms, like dynamic scaling or task-specific perturbations, to enhance the precision of pruning. additionally, we aim to collaborate with industry partners to deploy pip in real-world applications, such as edge computing systems, to validate its practicality and address potential issues.