Wanda++: Pruning Large Language Models via Regional Gradients (2025)

Abstract: Large Language Models (LLMs) pruning seeks to remove unimportant weights for inference speedup with minimal accuracy impact. However, existing methods often suffer from accuracy degradation without full-model sparsity-aware fine-tuning. This paper presents Wanda++, a novel pruning framework that outperforms the state-of-the-art methods by utilizing decoder-block-level regional gradients. Specifically, Wanda++ improves the pruning score with regional gradients for the first time and proposes an efficient regional optimization method to minimize pruning-induced output discrepancies between the dense and sparse decoder output. Notably, Wanda++ improves perplexity by up to 32% over Wanda in the language modeling task and generalizes effectively to downstream tasks. Moreover, despite updating weights with regional optimization, Wanda++ remains orthogonal to sparsity-aware fine-tuning, further reducing perplexity with LoRA in great extend. Our approach is lightweight, pruning a 7B LLaMA model in under 10 minutes on a single H100 GPU.

## introduction the growing size of large language models (llms) improves performance @xcite at the cost of memory consumption and inference latency. for example, loading the weights of llama-2 70b requires 140 gb of gpu memory in fp16 format. hosting a such model even with a batch size of 1 and 512 input tokens needs at least four a100-40gb gpus, with the time to first token (ttft) exceeding 100 milliseconds @xcite . to address these challenges, various model compression approaches, including weight decomposition @xcite @xcite , quantization @xcite , and pruning @xcite @xcite , have been explored. similar to llm quantization, many recent llm pruning methods have shifted from intraining approaches @xcite to post-training methods, as seen in sparsegpt @xcite and wanda @xcite . however, unlike post-training quantization like awq @xcite , which compresses model weights almost losslessly by 4× (from 16-bit to 4-bit), these pruning methods have yet to achieve comparable levels of performance. to mitigate the performance degradation, gblm and pruner-zero @xcite propose improved pruning criteria that enhance the layer-wise wanda score by incorporating gradient information obtained through full-model backpropagation (bp). these studies highlight the importance of gradient information, demonstrating that it provides valuable insights even though wanda assumes that the gradients of a fully trained network are small and contribute little when higher-order terms are considered. meanwhile, other approaches focus on recovering model performance through full-model sparsity-aware tuning @xcite or distillation @xcite . however, all these methods, which heavily depend on full-model loss, suffer from impractical memory requirements and excessive pruning time due to the high computational cost of full-model backpropagation. this raises the question: is there a way to effectively involve gradient information while still in a lightweight manner? in this paper, we propose wanda++ pruning framework to leverage gradients at the "decoderblock" level, termed regional gradients, which shows significant improvement compared to wanda @xcite without sacrificing the pruning efficiency. compared with traditional full model bp, the regional gradient can be obtained by only loading and computing gradients per single decoder block, which largely reduce the gpu hours required. compare with previous gradient-free method like wanda, the regional gradient can improve the pruning performance by accessing the gradient information. wanda++ prune each decoder block by iteratively prune the model based on a regional gradient score (rgs) and slightly update the weights with our proposed regional optimizer (ro). to effectively obtain the regional gradient in the rgs score, we compute the regional gradient through a backward process on a regional loss, which is defined as the ℓ 2 norm of each decoder's output hidden states. to involve the regional gradient into the pruning score, we follow the design of gblm score @xcite . note that the regional gradient is only computed once during the iterative pruning process of each decoder block and reused for all ro iterations to further reduce the computation. as a result, the regional gradient may become less accurate after weights being updated during the ro process. the design of gblm score provide an effective way to reflect changes by fetch the input per in-block layer and blend it into the rgs along with regional gradients. for the ro in wanda++, we construct a simple loss function between the outputs of the dense and pruned decoder blocks to update the model weights within each block. the concept of regional optimization traces back to previous local optimization approaches in convolutional neural networks (cnns), which aimed to optimize models by focusing on individual convolutional layers @xcite . differently, ro in wanda++ extends this idea for mitigating pruning-induced loss on a small portion of calibration data at each llm decoderblock, instead of optimizing the cross-entropy loss for classification tasks. our proposed framework achieves up to a 32% reduction in wikitext perplexity at 2:4 sparsity, compared to the wanda method under the same experimental setup, as shown in figure . 1. our contributions can be summarized as follows: ## prior solutions existing solvers for post-training llm pruning often start with magnitude pruning @xcite , which removes individual weights based on their magnitudes. this approach has been widely adopted as a baseline method in both vision @xcite and language processing @xcite . magnitude pruning is built on the assumption that neurons with larger weight elements contribute higher gradients when input features have similar magnitudes. however, this assumption does not always hold true for llms, where input features can differ significantly in scale, as observed in @xcite . to address this challenge, sun et al. propose the wanda method @xcite , which introduces a pruning criterion that multiplies the weight magnitude by the input activation on an element-wise basis. specifically, given a linear layer with a weight matrix @xmath0, sequence length is l, the wanda pruning score for the weight element connect input j and output i is defined as: ## the wanda++ framework in this section, we use the wanda method to highlight the drawbacks of the linearity assumption in previous layer-wise post-training pruning approaches. we then introduce our wanda++ framework, which efficiently reduces pruning degradation with a two-stage process for each decoder. this involves layer-wise pruning within blocks based on a new pruning criterion called the regional gradient score (rgs) and regional optimization (ro), as shown in figure 2 . wanda++ is efficient as it operates at each decoder block where its ro requires only 5 iterations. ## experiment we evaluate the proposed pruning method based on four criteria. the first is perplexity in nexttoken prediction, which is the main objective of llm pre-training. a lower perplexity is often a strong indicator of superior performance in language understanding tasks. the second criterion is the performance on zero-shot/few-shot nlp tasks, such as text classification, question answering, and text generation. this is to ensure that the gradients involved in regional optimization do not lead to overfitting for downstream tasks. the third criterion is pruning time and memory consumption. finally, we examine the actual latency reduction under both fp16 and fp8 quantization settings for various batch sizes and input/output lengths. ## extending wanda++ to structured pruning while our main focus is on unstructured and semistructured pruning, we further evaluate the applicability of our method to structured pruning to inspire the future work. specifically, we conduct experiments on a naive row-wise structured pruning (sp) for each weight matrices in the model, where the pruning score for each row is the average score of all paramters in the row. prior work has shown that naively adapt wanda methods to structured pruning (wanda-sp) tend to degrade sharply under structured settings when applied naively @xcite . nonetheless, we adapt our approach (wanda++-sp) to this setting and compare it against a baseline wanda-sp method across varying pruning ratios. as shown in ## conclusion in this paper, we proposed wanda++, a lightweight post-training llm pruning method that leverages regional gradients to effectively mitigate pruninginduced performance degradation. wanda++ defines a region as each decoder block. the method includes a pruner based on regional gradient scores (rgs) and a regional optimizer (ro) for inblock, sparsity-aware calibration. by utilizing regional gradients, it outperforms wanda on various llama models. wanda++ is efficient, especially compared to full gradient fine-tuning methods, pruning 7b llms in 10 minutes, as it operates only within each decoder block. we also note that the regional optimization component of wanda++ is orthogonal to most existing llm pruning approaches, as well as to works exploring dense-to-moe transformations @xcite . incorporating regional optimization from wanda++ provides a strong initialization for the foundation model, which can be further combined with downstream optimization techniques such as lora fine-tuning @xcite , direct preference optimization (dpo) @xcite , or group relative policy optimization (grpo) @xcite ). an exciting direction for future work is to explore the potential of using wanda++ as a general tool to enhance model performance following architectural adaptations.