Paper: The Cost of Scaling Down Large Language Models: Reducing Model Size Affects Memory before In-context Learning (ICLR-2024)

Reviewer comment (Writing):
Relatedly, there is very little discussion regarding down-scaling vs. pruning. For general readers it would be helpful to understand what are the difference between the two, and is one a specific version of the other?

Author response:
We can view dense down-scaling as a form of structured pruning. While SparseGPT prunes at the granularity of individual weight, we can view dense-downscaling as pruning at the granularity of entire neurons and entire layers. Some important differences between dense-scaling and pruning include:
- The order of weights removal and training – viewing dense downscaling as structured pruning, weights removal happens before training. However, for pruning, weight removal happens after training (and is followed by repair).
- Granularity of weights removal – dense downscaling removes entire neurons & layers, whereas pruning removes individual weight.

Reviewer comment (Evaluation):
For learning tasks evaluation, why only consider task with scalar values as labels? I understand this needs to be something that model can generalize through the examples, but if we focus on language capability of the model, I would expect that a natural language task is used instead.

Author response:
Please refer to our global response, additional task evaluation section. We provide additional results on the LAMBADA benchmark, a natural language based ICL task. 
Our focus on synthetic ICL tasks (thus scalar/synthetic labels) is driven by our goal to disentangle fact recall from ICL tasks. Many existing natural language based ICL tasks run the risk of relying on the model's parametric knowledge. For example, one of the ICL examples in GPT3 paper is language translation, which clearly heavily depends on pre-trained knowledge of source and target language. We already know that such pre-trained knowledge is impaired in our QA experiments. Synthetic tasks, due to their synthetic nature, are largely free of such issues.

Reviewer comment (Evaluation):
For ICL results, it seems the performance drops significantly (not gradually) from 70% above, do you have intuition why?

Author response:
We conjecture that ICL learning ability depends on a small set of weights that act as the in-context gradient descent module [1]. At 70% and above, SparseGPT may eventually decide to remove this small set of weights, causing sudden drop of accuracy.

Reviewer comment (Experiments):
As the paper only use some particular pruning methods, do you have any opinion on whether the same findings will hold for other pruning methods? It would be good to have a short discussion on the difference between them.

Author response:
We believe our findings will generalize to other pruning methods. This is because our findings not only apply to pruning (SparseGPT, Wanda), but also to dense down-scaling. The consistency of our results between sparse and dense down-scaling provides evidence that our observation is fundamentally about model size reduction in general, measured in the number of parameters.

Reviewer comment (Writing):
Section 5 is hard to follow without examples, there are multiple notations without explanation, e.g. K (page 7), x, D=4, N=32, etc.

Author response:
We will make sure to improve our writing and let you know once we finish updating the paper!

Reviewer comment (Reproducibility):
Would be good to release the version of datasets that are used for benchmarking.

Author response:
Yes we will release the datasets we use!

Reviewer comment (Experiments):
The major weakness of the paper is the effective testing of LLM parametric and ICL knowledge. In particular, it is unclear how the authors verify that the in-context evidence contradicts a fact present in training data. Although LLaMA was trained on publicly available data, it is not simple to verify that the answers in the Q&A datasets align or misalign with the massive dataset used to train LLaMA, especially since the dataset itself was never released, making it necessary to evaluate all public datasets in their entirety for verification. It is not clear how verifying that "answers do not exist in the pre-training corpus" is possible.

Author response:
We agree with your point and took actions to address your concern. It is indeed nearly impossible to verify that counter-factual answers do not exist in pre-training corpus. We amended the phrasing of our experiment design to avoid making absolute statement about dataset contamination. Furthermore, we quantify the extent to which they exist in the pre-training corpus, by running the counterfactual QA evaluation using dense models without the counterfactual context – we count the model’s answer as correct if and only if it matches the counterfactual answer. If the training dataset of LLaMA is contaminated with the counterfactual question/answers, we should expect to see reasonably high accuracy under this setup. Instead, we observe near-zero accuracy for both LLaMA-13 and LLaMA-33B model (0.1% for both LLaMA-13B and LLaMA-33B).

Reviewer comment (Evaluation):
The exact measurement of accuracy used in the paper is potentially incorrect and too conservative for recently released instruction-tuned LLMs like LLaMA and OPT. The paper reports the percentage of answers that exactly match ground truth using greedy decoding. However, greedy decoding is known to be suboptimal for such LLMs, and exact match may not be fair due to the models' tendency to produce verbose outputs. If the correct answer is preceded by additional text, this metric will incorrectly mark it as wrong. While greedy decoding can be justified for reproducibility across sparsity levels, the limitations of the exact match metric are a significant source of bias and should be addressed.

Author response:
To begin with, both LLaMA[4] and Pythia[5] models (which we study) use greedy decoding and exact match during evaluation. So our experimental design is standard as is. Furthermore, we do not use instruction-tuned models, so verbosity should not be a concern either. Nevertheless, we collected extra results to address your concern about decoding and wordiness; and show that our conclusion holds up to additional scrutiny. We re-run key subset of our experiments using a more advanced sampling scheme and more tolerant matching criteria according to your proposal. Specifically, we repeat our LLaMA-13B and LLaMA-33B experiments on TriviaQA benchmark with and without context. We use beam search with beam size of 3 (some paper in the past, e.g., OPT[6], GPT3[7] used beam search during evaluation). To get an answer from the model, we generate 32 tokens from the model, and check whether the answer appears anywhere within the generated tokens. The average number of tokens for TriviaQA answer is 4.7, so a model can generate extra tokens irrelevant to the answer whilst still correctly answering the question. Here’re our results:
- Our conclusion still holds in this setting: using our current experimental design, averaging over two LLaMA models, one can prune 30% and 40% weights on Closebook and Openbook TriviaQA task, respectively. Using beam search, one can prune 30% and 50% weights on Closebook and Openbook TriviaQA tasks, respectively. The ability to retrieve answers from provided context remains more resilient to pruning than the ability to recall information learnt during pre-training. 
- Additionally, we observe a noticeable accuracy boost likely due to better decoding strategy using beam search (about 10-20% accuracy improvement). This is expected, as beam search with beam size of 3 costs ~3x the compute of greedy decoding.
- Please see full pruning curves and comparison with greedy decoding in Appendix L of our newly updated paper draft.

Reviewer comment (Evaluation):
The claim that "down-scaling neural networks affects more than just top-line metrics or task accuracy" and that pruning can introduce biases or disproportionate effects on certain subsets of the data is too strong. Significant work has shown pruning can be effective for vision models, so the certainty implied by the statement should be moderated.

Author response:
We agree. Our use of the word “can” intends to convey the uncertainty. If you think this is insufficient, we are happy to be more precise about the precondition of the said phenomenon – “Pruning to very high sparsity, for example, can introduce biases…”

Reviewer comment (Novelty):
The claim that scaling down model size alone has little impact on tasks demanding information processing in an LLM’s context is not novel. Previous works such as Chen et al. (2023), Dohan et al. (2022), and the Wanda paper already explore how parameter size and pruning affect ICL performance, and relevant prior work should be acknowledged.

Author response:
Please also refer to our global response for a discussion of our contribution within the context of existing pruning work.

FrugalGPT[8] (Chen et al.,), SparseGPT[1] and Wanda[2] indeed all investigated the task performance-size/cost trade-off of LLMs. However, when a new task emerges outside of those examined by these prior studies, how can a practitioner know if it is a good idea to route it to a smaller model? Our work provides insight in this scenario: practitioners can assess how much the said task relies on parametric knowledge versus context processing capability. The more a task relies on context processing, the more likely such a task can be routed to a smaller model.

Reviewer comment (Novelty):
The stated contribution that "we focus on scaling model size alone" is not valid as it falls within the scope of studies on "joint scaling of both pre-training corpus size and model size." Furthermore, Kaplan et al. (2020b) does not study pruning but examines model size, training, and resulting performance. The differences between these paradigms need to be clearly stated.

Author response:
We use this statement to highlight a difference in methodological choice – we want to be upfront about the more focused and thus narrower scope of our study. To be clear, we use this statement to disambiguate, not to present a contribution.

Reviewer comment (Novelty):
The claim that the proposed evaluation of pruning’s effect on fact recall and ICL begins to fill a research gap is misleading, as fact recall (zero-shot Q&A) and the effect of pruning on various tasks, including ICL, have been addressed in previous works like SparseGPT, Wanda, and LLM-Pruner. Contributions should be revised and contextualized.

Author response:
We emphasize that fact recall is not equivalent to zero-shot closebook question answering (correct us if we’re wrong, by zero-shot QA we believe you mean zero-shot **closebook** QA). The model may rely on the ability to recall parametric knowledge even in openbook QA settings. Hence we argue that no single task exclusively test the model’s ability to recall information from pretraining versus its ability to process information in context. Hence a primary contribution of ours is to disentangle the assessment of both abilities to the best of our abilities. Please refer to our global response for the discussion of our contribution in the context of prior pruning work.

Nevertheless, we are happy to better acknowledge the work done by prior pruning research, and revised our phrasing to the following: … approach to assessing pruned models. Prior work does a commendable job to assess model size and task accuracy trade-off. Our work continues to expand our toolkits for empirical assessment, proposing to evaluate pruning’s effect on fact recall and ICL.

Reviewer comment (Novelty):
The statement that "as a simple point of comparison, we measure the effect of downscaling on perplexity" should note that this has been previously considered in both SparseGPT and Wanda papers.

Author response:
Agree. Revised to credit this to prior pruning papers.

Reviewer comment (Writing):
The claim that "we focus on foundational capabilities of LLMs – fact recall and ICL" misrepresents fact recall as a foundational capability rather than as zero-shot question answering, which is a specific task. This claim should be adjusted accordingly.

Author response:
We emphasize that fact recall is not equivalent to zero-shot closebook question answering (correct us if we’re wrong, by zero-shot QA we believe you mean zero-shot **closebook** QA). The model may rely on the ability to recall parametric knowledge even in openbook QA settings. Hence we argue that no single task exclusively test the model’s ability to recall information from pretraining versus its ability to process information in context. Hence a primary contribution of ours is to disentangle the assessment of both abilities to the best of our abilities. Please refer to our global response for the discussion of our contribution in the context of prior pruning work.

Nevertheless, we are happy to better acknowledge the work done by prior pruning research, and revised our phrasing to the following: … approach to assessing pruned models. Prior work does a commendable job to assess model size and task accuracy trade-off. Our work continues to expand our toolkits for empirical assessment, proposing to evaluate pruning’s effect on fact recall and ICL.

Reviewer comment (Experiments):
Why did the authors not consider the GINC dataset for ICL, from Xie et al.’s "An Explanation of In-context Learning as Implicit Bayesian Inference?"

Author response:
GINC dataset requires pre-training on the synthetically generated pre-training dataset for the evaluation task to make sense. The task is purely synthetic, a model pre-trained on natural language corpus cannot not solve it. Since our focus is on evaluating pre-trained LLMs, this task does not appear to suit our needs. Please see the eval dataset sample here [9].

Reviewer comment (Reproducibility):
In "From the OPT family, we evaluate the two largest models that fit in our hardware setup", the hardware setup should be stated.

Author response:
Our hardware setup is available in appendix J. All our evaluations are performed on TPU v3-8; each core of TPU v3 has 16GB HBM memory. We used batch size 1 and additional tricks such as CPU-offloading to fit larger models inside TPUs.

Reviewer comment (Writing):
In Table 1, it is unclear what is meant by "Context Type"; please define this term.

Author response:
It means what is the type of supporting evidence we insert into the model’s context to help the model solve the task. Empty means no evidence provided, and is used in closebook QA evaluation. Factual means we provide a paragraph of supporting evidence containing the factual answer to the question, and is used in openbook QA evaluation. Synthetic means we provide a paragraph of evidence containing the counterfactual answer to the question, and is used in counterfactual QA evaluation. Exemplary means we provide in-context examples to guide models to learn parametric functions in-context.

Reviewer comment (Experiments):
The study focuses on evaluating two pruning algorithms that are unstructured pruning, evaluation on structured pruning methods are expected.

Author response:
Thank you for your helpful feedback!

> The study focuses on evaluating two pruning algorithms that are unstructured pruning, evaluation on structured pruning methods are expected.
> How the structured pruning methods, e.g., LLM-Pruner, performs on these two LLM capabilities?

We are working on providing some structured pruning results. We will get back to you once results become available.

Reviewer comment (Experiments):
How the structured pruning methods, e.g., LLM-Pruner, performs on these two LLM capabilities?

Author response:
We're writing to follow up on structured pruning results:

Please refer to our Appendix P. Structured Pruning Results, our conclusion remains valid with structured pruning algorithms LLM-pruner. Structured pruning results demonstrate that in-context learning of parametric functions is more robust to pruning compared to Openbook QA. Openbook QA itself shows more robustness to pruning than Closebook QA. This is consistent with the conclusion from our main paper.