Pruning Foundation Models for High Accuracy without Retraining (2024)

Abstract: Despite the superior performance, it is challenging to deploy large language models (LLMs) due to their massive parameters and computations. While pruning is a promising technique to reduce model size and accelerate the inference, the traditional pruning techniques can hardly be applied for LLMs as they need to finetune the model on the full dataset with multiple epochs consuming massive data and hardware resources. To deal with this problem, post-training pruning methods are proposed to prune LLMs in one-shot without retraining. However, their accuracy after pruning may suffer from certain performance degradation due to the lack of retraining with massive data. To address this issue, in this paper, we first formulate the post-training problem for layer-wise LLM compression to simultaneously prune multiple weights in LLMs. Next, we provide an optimal solution for this problem and design our post-training pruning algorithm for both unstructured and semi-structured sparsity. Our extensive experiments demonstrate the superior performance of the proposed methods in comparison to SOTA baselines across various LLM families including transformer-based LLMs and Mamba-based LLMs.

## introduction foundation models or large language models (llms) have achieved remarkable performance on a variety of tasks. however, it is challenging to deploy llms in practical applications due to their massive parameters and computations. to facilitate llm deployment in practice, various model compression techniques targeting llms including pruning @xcite and quantization @xcite @xcite have been proposed to reduce memory and computation costs. the traditional pruning techniques, which finetune or retrain models @xcite on full datasets for many epochs (i.e., pruning-aware training), are too expensive for llms in terms of data and gpu resources. thus, post-training pruning based on well-pre-trained models with reduced resource requirements represents a more reasonable approach for llms. notably, sparsegpt @xcite is the representative post-training pruning work with outstanding performance. it reduces memory cost by sequentially loading transformer blocks, one at a time, instead of loading the whole model. moreover, it reduces the data cost by using only a small amount of calibration data, eliminating the retraining process on massive data. besides the optimization based sparsegpt, there are some other heuristic posttraining pruning methods such as @xcite , achieving accuracy close to sparsegpt. however, the performance of sparsegpt is still sub-optimal as it relies on the solution of single removal problem (srp) @xcite to address the pruning of multiple weights, which is essentially a multiple removal problem (mrp). in particular, the srp provides the optimal solution to prune one weight at a time and modify all other weights to compensate the pruned single weight and minimize the loss. however, as the optimal solution is unaware of all previous pruned weights and requires to modify all other weights including previous pruned ones (making them unpruned again), it is at odds with the mrp for multiple pruned weights. thus, the optimal solution in srp can not be directly applied to solve mrp. to successfully incorporate the srp solution, a series of approximation methods are adopted in sparsegpt, at the cost of certain performance degradation, as detailed in sec. 2.3. different from the srp-based sparsegpt, we directly formulate the mrp for layer-wise llm pruning to simultaneously prune multiple weights in llms. next, we derive the optimal solution for the mrp problem with detailed analysis for its advantages. based on the optimal solution, we design our post-training pruning algorithms for both unstructured and semi-structured sparsity. in our comprehensive evaluation, we demonstrate our superior performance in terms of perplexity and zero-shot accuracy compared with sota baselines for both transformer-based and mamba-based llms (such as our 4.278 perplexity on wikitext2 v.s. 5.698 from sparsegpt for llama2-70b under 2:4 sparsity). our contributions are summarized as follows: ## methodology we first derive our optimal solution for the mrp and then discuss the algorithm design. ## experimental results alistarh, 2022, 2023). we conduct experiments on one nvidia a100 gpu. similar to @xcite , we do not incorporate any finetuning. our experiments can be finished on one single gpu within a few hours. for example, for a llama2-7b model, our pruning method can be finished on one single gpu within 4 hours, which is still not very long. our method can achieve a better perplexity and accuracy especially for large sparsity, which can hardly or never be achieved by the baselines. compared with other typical pruning-aware training methods which require to fine-tune/re-train the pruned model with massive data, our method is very efficient with a few gpu hours cost on a single gpu, since we do not require finetuning. for example, the llama pro method @xcite ) needs over 2800 gpu hours to finetune a small part of the llama2-7b model on multiple gpus. ## related work post-training pruning. it is challenging to apply the traditional pruning methods for llms, since it requires to retrain or finetune the model on the full dataset for many epochs with massive data and computation costs @xcite @xcite @xcite . to address this problem, post-training pruning for llms are explored to prune the model with a small amount of calibration data, requiring much less resources compared with retraining. the posttraining idea is originally proposed in quantization @xcite @xcite for transformers and llms, and then successfully applied for pruning @xcite @xcite @xcite . post-training compression usually investigates the compression for a single layer of the llm instead of the whole model for simplicity. the memory cost is reduced significantly as the memory only loads one block at a time @xcite . post-training solvers. adaprune @xcite uses weight magnitudes to determine the pruning mask, and then uses an optimizer such as sgd to update unpruned weights and improve the performance based on a small amount of calibration data. its performance is sub-optimal due to the limited number of finetuning data. to further improve the performance, the optimizationbased post-training methods are proposed such as obc @xcite and sparsegpt @xcite . based on the srp and its solution (singh and alistarh, 2020), obc introduces a greedy solver to remove one single weight at a time, and then reconstructs the remaining weights following closed-form solutions in each iteration. sparsegpt further improves the solution of obc and applies to the largest available open-source llm models, achieving 60% unstructured sparsity with sota performance in perplexity. besides optimization-based methods, the heuristic wanda @xcite suffers from significant accuracy loss when the sparsity is large. ## conclusion we first formulate the mrp in llms to prune multiple weights simultaneously. then we derive the optimal solution. based on the optimal solution, we propose accurate post-training pruning algorithms for unstructured and semi-structured sparsity. our comprehensive experiments demonstrate that our method is more accurate than sota baselines under the same configurations.