S wift P rune: Hessian-Free Weight Pruning for Large Language Models (2025)

Abstract: Post-training pruning, as one of the key techniques for compressing large language models (LLMs), plays a vital role in lightweight model deployment and model sparsity. However, current mainstream pruning methods dependent on the Hessian matrix face significant limitations in both pruning speed and practical effectiveness due to the computationally intensive nature of second-order derivative calculations. This paper presents SwiftPrune, a novel Hessian-free weight pruning method that achieves hardware-efficient model compression through two key innovations: 1) SwiftPrune eliminates the need for computationally intensive Hessian matrix calculations by introducing a contribution-based weight metric, which evaluates the importance of weights without relying on second-order derivatives. 2) we employ the Exponentially Weighted Moving Average (EWMA) technique to bypass weight sorting, enabling the selection of weights that contribute most to LLM accuracy and further reducing time complexity. Our approach is extended to support structured sparsity pruning, facilitating efficient execution on modern hardware accelerators. We validate the SwiftPrune on three LLMs (namely LLaMA2, LLaMA3, and Pythia), demonstrating that it significantly enhances compression performance. The experimental findings reveal that SwiftPrune completes the pruning process within seconds, achieving an average speedup of 12.29x (up to 56.02x) over existing SOTA approaches.

## introduction in recent years, the capabilities of llms have experienced explosive growth. however, this advancement comes at the cost of exponential expansion in model scale, resulting in significant financial and energy expenditures @xcite . consequently, there has been growing effort to mitigate these costs through model compression. (fran-* *corresponding author @xcite @xcite @xcite . among these, pruning has emerged as one of the most widely adopted techniques, with its fundamental principle involving the elimination of redundant parameters by selectively zeroing out network weights. contemporary pruning methods for large language models primarily eliminate retraining requirements through hessian-based loss analysis @xcite @xcite @xcite . while mathematically elegant, these methods face persistent implementation challenges due to slow pruning speeds. specifically, computing second-order derivatives across all network weights creates a hessian matrix whose dimensionality scales quadratically with parameter count, leading to intractable computational complexity. this limitation becomes critical in emerging real-time pruning scenarios such as training sparse models from scratch @xcite , finding the optimal sparsity @xcite and other scenarios requiring frequent pruning operations @xcite @xcite . with existing methods requiring hundreds of seconds per pruning iteration (see furthermore, the emergence of advanced gpu architectures underscores the demand for structured hardware-aware pruning methods that achieve genuine acceleration while maintaining computational efficiency @xcite @xcite , thereby highlighting the importance of pruning approaches compatible with structured sparse formats. in this study, we propose swiftprune, a novel pruning method designed to circumvent the high computational complexity associated with hessian matrix and its inverse calculations by developing an alternative algorithm. first, our observations indicate that identifying weights with minimal loss contribution depends more on their relative importance than on absolute values. to leverage this, swiftprune replaces hessian matrix computations by constructing a numerically preserved sequence as contribution-oriented weight metrics, derived through a series of loss values. secondly, we introduce the exponentially weighted moving average (ewma) method, borrowed from the transmission control protocol (tcp), to replace traditional sorting methods, further reducing computational complexity. moreover, we extend this approach to support structured sparsity pruning. we conduct comprehensive evaluations of swiftprune across three prominent open-source llms families: pythia @xcite , llama2 @xcite , and llama3 @xcite . compared to previous state-of-the-art methods for large language model pruning @xcite , our swiftprune framework achieves the pruning process within seconds, delivering an average 12.29× speedup (with peak acceleration reaching 56.02×) while maintaining comparable accuracy retention across standard benchmarks. experimental results demonstrate that swiftprune can finish pruning tasks more rapidly without requiring any retraining or weight updates, thereby addressing application scenarios that necessitate frequent pruning. ## background post-training pruning has emerged as a prevalent model compression technique, originating from quantization research @xcite @xcite and later extended to llms pruning @xcite @xcite . in neural network optimization, the primary mechanism for minimizing the target loss function involves iterative adjustment of network weights through first-order gradient computation. however, post-training pruning methods operate under a distinct paradigm: these approaches are typically applied to models that have already converged to a local (or potentially global) minimum through standard training procedures. in such optimized states, the first-order derivatives of weights with respect to the loss function asymptotically approach zero(i.e, ∂e ∂∆w ≈ 0 in equation 1). this mathematical condition fundamentally shifts the optimization focus to second-order sensitivity analysis. to formalize this concept, we employ a taylor expansion of the loss function e around the trained network parameters. the expansion reveals: where higher-order terms become nonnegligible precisely when first-order derivatives vanish, necessitating explicit consideration of second-order derivatives for effective post-training pruning. from this, we can infer that if a weight has a significant second-order derivative with respect to the target function, it indicates that the convergence of the weight is not yet stable. the impact of the weight change δw i on the loss function is reflected by the second-order derivative ∂ 2 e ∂w 2 i ∆w i . unfortunately, to compute the second-order derivatives of weights, we need to construct the hessian matrix to characterize the differences in outputs obtained from the compressed model and the original model under the same input, we select @xmath0, where w i is the i th weight, and d row is the dimension of a row in a module's weights. since weights from different rows act on the same input, resulting in elements in the same column in the output, we assume that for any linear layer, weights across different rows are independent of each other. specifically, in linear layers, every row in w never multiplies with another row, so there are no cross terms in loss functions, meaning they can be optimized independently. in this scenario, @xmath1. building upon this theoretical foundation, this work focuses on developing a novel compression approach that bypasses the high computational cost of the hessian matrix and its inverse while closely approximating its accuracy-preserving performance, achieving a balance between runtime efficiency and precision, and enabling scalability to very large models. ## conclusion in this paper, we propose swiftprune, a hardware-friendly approach for pruning of llms. the core innovation of our study is the development of a novel hessian-free llms pruning method, which significantly reduces time complexity from o(n 3 ) to o(n) compared to mainstream algorithms. this theoretical breakthrough ensures that our method consistently outperforms existing approaches in terms of both computational efficiency and scalability. built on a rigorous mathematical foundation, swiftprune demonstrates exceptional effectiveness and relevance, particularly as the scale of future llms continues to expand. by significantly reducing computational resource demands and energy consumption.