Is C4 Dataset Optimal for Pruning? An Investigation of Calibration Data for LLM Pruning (2024)

Abstract: Network pruning has emerged as a potential solution to make LLMs cheaper to deploy. However, existing LLM pruning approachesuniversally rely on the C4 dataset as the calibration data for calculating pruning scores, leaving its optimality unexplored. In this study, we evaluate the choice of calibration data on LLM pruning, across a wide range of datasets that are most commonly used in LLM training and evaluation, including four pertaining datasets as well as three categories of downstream tasks encompassing nine datasets. Each downstream dataset is prompted with In-Context Learning (ICL) and Chain-of-Thought (CoT), respectively. Besides the already intriguingobservation that the choice of calibration data significantly impacts the performance of pruned LLMs, our results also uncover several subtle and often unexpected findings, summarized as follows: (1) C4 is not the optimal choice for LLM pruning, even among commonly used pre-training datasets; (2) arithmetic datasets—when used as calibration data—performs on par or even better than pre-training datasets; (3) pruning with downstream datasets does not necessarily help the corresponding downstream task, compared to pre-training data; (4) ICL is widely beneficial to all data categories, whereas CoT is only useful on certain tasks. Our findings shed light on the importance of carefully selecting calibration data for LLM pruning and pave the way for more efficient deployment of these powerfulmodels in real-world applications. We release our code at: https://github.com/abx393/llm-pruning-calibration-data.

## introduction in the 2020s, the landscape of ai has transitioned into a new era, propelled forward by the advancements made in large language models (llms) @xcite @xcite . the astonishing language capacities of llms have significantly shaped the solutions to various real-life tasks such as natural language understanding @xcite ), text generation @xcite , vision tasks @xcite , coding @xcite , and math @xcite . however, the enormous size of these powerful llms poses a significant challenge for deployment in many real-world applications. for instance, deploying a 7b llm requires around 10gb of main memory (dram) even after adopting int8 quantization, which unfortunately exceeds the memory capacity of most commodity edge devices. network pruning, as one of the most wellestablished approaches in model compression, demonstrated the possibility of removing around 50% of the parameters @xcite @xcite , or even more @xcite with minimal performance degradation. interestingly, while consistently producing robust performance in small-scale deep neural networks @xcite @xcite , magnitude pruning @xcite seems to lose importance in the context of llm pruning. all state-of-the-art llm pruning approaches unanimously choose to use a small set of data (known as calibration data) from the c4 training dataset @xcite to calculate their pruning scores @xcite @xcite . using c4 as the calibration data for pruning makes sense if the models are pre-trained on it to preserve better the desired distribution learned during pre-training. however, not all large language models are pre-trained with the c4 dataset, raising the question of whether the c4 is the optimal choice for the calibration data for llm pruning. in addition, it is well-known that llms are very sensitive to how the input is structured and provided to them @xcite . as a result, it is unclear how the input format of calibration data would affect llm pruning. to answer these questions, in this work, we conduct a comprehensive study to investigate the effect of calibration data on llm pruning across a broad range of evaluation tasks, along two dimensions of interest: varying types of datasets and different data input formats. specifically, we investigate the following possible alternatives of calibration data for llm pruning, as illustrated in figure 1: • pre-training data: apart from the c4 dataset, several other datasets are widely used for pretraining llms. we examine three of the most representative datasets: pile @xcite , oscar @xcite , and redpajama (together computer, 2023). to investigate the impact of these datasets, we prune llms using various calibration datasets and evaluate the resulting sparse models across nine downstream tasks. our key and encouraging finding is that, while c4 consistently produces robust sparse models, it is not the best calibration dataset for pruning. in addition, our study unveils several more subtle and unexpected findings, which can be summarized as follows: ## results in this section, we report the results of our experiments. our primary goal is to explore how performance fluctuates when using various calibration data across different formats. we analyze overall performance trends across these differing setups. ## further analysis can we do better by including more steps in cot? in our previous construction of the calibration data, we selected question-answer pairs with no restriction on the number of steps in cot in the answer. this inspires a follow-up question: does the number of steps of cot rationale in the calibration data affect the sparse llm's performance? we investigated this by constructing calibration data by concatenating multiple question-answer pairs, where each answer rationale contains exactly x steps. since 1-step or 2-step cot data was scarce, we performed this for @xmath0, 4, 5} as seen in does more q-a pairs in icl calibration data lead to a better sparse model? to investigate this, we evaluated the pruning performance when calibration data contains 5, 10, 15, 20, and 25 q-a pairs, filling the rest of the context window with padding tokens. our default icl calibration data fills the context window with q-a pairs until it reaches length 2048 tokens, which in practice can be anywhere from 25 to 30 q-a pairs. we compare the pruning performance of all of these calibration data formats in how does input length affect the pruning performance? in our main experiments, the calibration data for pruning consisted of 128 sequences, each 2048 tokens in length. it is crucial to investigate whether this specific token length is necessary for effective pruning. to address this question, we used the c4 dataset for calibration and systematically varied the calibration data lengths between 256, 512, 1024, and 2048 tokens. we then evaluated the perplexity of llama 2-chat 7b pruned to 50% unstructured sparsity using wanda. as detailed in ## conclusion this study critically examines the widely held belief that the c4 dataset is the optimal calibra-tion choice for pruning llms. through an extensive evaluation encompassing a variety of calibration data types-both pre-training and downstream datasets, our findings reveal that c4 does not hold universal superiority. specifically, our analysis demonstrates that the pretraining dataset pile consistently outperforms c4, while alternative downstream datasets, particularly those involving arithmetic reasoning tasks, yield comparable pruning outcomes. furthermore, our investigation into various downstream task formats has uncovered that in-context learning (icl) offers significant benefits across all data categories. in-context learning w/ chain-of-thought (icl w/ cot) calibration is particularly effective in enhancing performance in arithmetic reasoning tasks. our study advocates for a more nuanced selection and curation of calibration data, which could lead to more efficient and effective llm pruning strategies, ultimately facilitating the deployment of more robust models in practical settings.