Rethinking Pruning Large Language Models: Benefits and Pitfalls of Reconstruction Error Minimization (2024)

Abstract: This work suggests fundamentally rethinking the current practice of pruning large language models (LLMs). The way it is done is by divide and conquer: split the model into submodels, sequentially prune them, and reconstruct predictions of the dense counterparts on small calibration data one at a time; the final model is obtained simply by putting the resulting sparse submodels together. While this approach enables pruning under memory constraints, it generates high reconstruction errors. In this work, we first present an array of reconstruction techniques that can significantly reduce this error by more than 90%. Unwittingly, however, we discover that minimizing reconstruction error is not always ideal and can overfit the given calibration data, resulting in rather increased language perplexity and poor performance at downstream tasks. We find out that a strategy of self-generating calibration data can mitigate this trade-off between reconstruction and generalization, suggesting new directions in the presence of both benefits and pitfalls of reconstruction for pruning LLMs.

## reconstruction techniques this section explains three optimization schemes we use to reduce reconstruction errors in this work. ## further exploration we have seen that reconstruction techniques are useful but they can lead to undesirable overfitting. here we explore potential ways to alleviate this risk. in particular, we identify that the calibration data is highly limited in two aspects: it is too little (compared to optimization variables) foot_0 and does not represent the training data (as it is arbitrarily given); the former is related to the general representationgeneralization complexity trade-off, and the latter is about whether the reconstruction can mimic the behavior of the original model. to this end, we reflect on the fact that what we are dealing with is a generative (language) model, meaning that we can create calibration data that is potentially much bigger in size and closer to the original distribution. we find that this selfgeneration technique has recently been proposed in other contexts @xcite @xcite , and thus, follow the process therein to produce high-quality text data. using that, we perform reconstruction again, and the results are reported in figure 4 . we observe that making use of more self-generated calibration data (without unfairly violating the given setting) reduces both test error and perplexity, mitigating overfitting quite effectively. ## conclusion in this work, we take a close look at the current practice of minimizing reconstruction errors for pruning llms. we first find that with various reconstruction techniques, one can reduce the error quite significantly and improve quality of pruning results on both language perplexity and zero-shot accuracy. nevertheless, it turns out that decreasing error as it is now is not always desirable since it may cause overfitting calibration data. we present initial results that this issue can be potentially mitigated by self-generating calibration data. there are many remaining possibilities, and we believe our findings suggest opportunities for future work.