Paper: Scaling Laws for Sparsely-Connected Foundation Models (ICLR-2024)

Reviewer comment (Experiments):
Experiments are only tested on two examples. Though due to limited resources, more examples may be challenging, two examples may not be very convincing.

Author response:
Our main experiments feature 112 training runs for ViT models and 48 training runs for T5 models, plus several runs for n:m sparsity, pruning pretrained models, ablation studies and two larger extrapolation experiments. Many of these take days to execute on multiple accelerators (one extrapolation experiment for example takes 2 days on 128 devices).

Additionally, towards addressing this concern, we **have conducted additional experiments for autoregressive decoder-only models**, showing that our scaling laws also hold in this context. We present those new results in Appendix G of the most recent revision or, for convenience, at this anonymized link: https://github.com/iclr24-1372/iclr24-1372/blob/master/decoder-only.pdf.

Finally, we would like to note that most scaling works focus only on a single domain and model family, while we study both vision and text models (and now also encoder-decoder and decoder-only variants). This highlights the generality of our sparsity scaling laws.

Reviewer comment (Theory):
The scaling law has too many degrees of freedom (7 free parameters) that could be hard to generalize to new dataset without sufficient information.

Author response:
Standard dense scaling laws usually have at least 5 parameters for two inputs N (model size) and D (training data). Hence, we think allowing 2 more parameters to model the impact of an additional third input S (sparsity) is a reasonable increase. The 7 free parameters are fitted over 112 ViT and 48 T5 training runs, respectively, hence the #samples to #parameters ratio is ~16 and ~7, which limits overfitting capability. At the same time, we still observe good fits and extrapolation performance.

We note that the coefficients of a scaling law are usually fit on a per model family and dataset/task basis, only the overall functional form of the law is supposed to generalize across settings, which we demonstrate by studying both vision and language models.

Reviewer comment (Writing):
Procedure to identify $S_{opt}$ in 2.2 Optimal Sparsity is confusing. Could you explain this procedure with more details?

Author response:
The high level idea is as follows: 
1. We first reparametrize our $L(S, N, D)$ scaling law in terms of $L(S, N, C)$ by replacing $D$ with the corresponding amount of training data that, for a model with parameter count $N$ and sparsity $S$, leads to overall compute $C$. 
2. Next, we would like to minimize $L(S, N, C)$ with respect to $S$, in order to determine the best sparsity value. This is done in standard fashion by setting the derivative of $L$ with respect to $S$ to 0. 
3. Our expression given for the contour lines is found by rearranging terms on both sides. Similarly, $S_\text{opt}$ is determined by solving for $S$ with standard algebraic manipulations. We provide those algebraic details in Appendix D.

Reviewer comment (Novelty):
My main concern lies in the contribution of section 2 on fair evaluation, which argued that a sparse network should be compared with a dense model with the same number of parameters and compute. A very similar argument is made in [1] (See “training budget” in section 3), which argued that the compute budget for comparing sparse and dense models should be the same. It would be good to clarify the differences to [1] in the paper.

Author response:
The main purpose of this paragraph was to highlight that this resource normalization is critical when dealing with foundation models that exhibit strong scaling behavior. This is not as important on standard smaller-scale pruning benchmarks (e.g., CIFAR/ImageNet or BERT/GLUE) and is thus often neglected. Nevertheless, as pointed out by the reviewer, training budget normalization has indeed been considered by some prior works as well, in different contexts. We now explicitly state this in Section 2 of the newest revision, referencing Liu et al. (2019) [1], as well as Jin et al. (2022).

Apart from the fact that those works also highlight similar considerations regarding fair evaluation of sparse models, they are quite different in scope. Specifically, neither of them develops scaling laws or studies foundation models.

Reviewer comment (Experiments):
The evaluation on the encoder-decoder architecture T5 seems limited. The paper do not evaluate the more popular autoregressive decoder-only architecture but the original scaling laws is built on such autoregressive language models.

Author response:
We initially selected the T5 encoder-decoder architecture to demonstrate that our sparsity scaling law is also applicable to slightly more complex Transformer variants. However, we agree that a standard autoregressive decoder-only setup would be a useful addition and **have conducted an additional scaling sweep for autoregressive decoder-only Transformers.**

Specifically, we train several autoregressive decoder-only Transformers for next-token-prediction on the C4 dataset, using similar size/sparsity/data settings as in our T5 experiments. The results are presented in Appendix G of the most recent revision or, for convenience, at this anonymized link: https://github.com/iclr24-1372/iclr24-1372/blob/master/decoder-only.pdf.

Overall, we observe all the key aspects of our scaling law also in this setting (sparsity acts as a constant multiplier on size scaling, is mostly independent of the amount of training data, and yields diminishing returns at higher values), which consequently leads to a good coefficient fit as well. This suggests that standard autoregressive decoder-only models exhibit the same general sparsity scaling behavior as ViTs and encoder-decoder models, further confirming our results.

Reviewer comment (Experiments):
I am curious as to whether lottery ticket hypothesis would hold as to foundation models, as it is the most impactful pruning paper in recent years. Since this work is closely related, I am wondering if the authors have explored the LTH setting for sparse training. If not, it would be good to add a discussion on LTH [2].

Author response:
We did not explore lottery tickets specifically; this is primarily because first training a dense model only to select a mask and then rewind any optimization progress (by resetting the weights to their initial values) is not a particularly efficient pruning strategy. This is especially the case when fairly accounting for the overall training costs, which was a key goal of our study. The inefficiency of lottery tickets has been shown in [1], which finds that rewinding the weights performs worse than learning rate restarts. Additionally, [2, 3] provide explanations on why lottery tickets fail at scale.

* [1] https://arxiv.org/abs/2003.02389
* [2] https://arxiv.org/abs/2010.03533
* [3] https://arxiv.org/abs/2210.03044

Reviewer comment (Experiments):
Can the proposed scaling laws be useful in the extreme sparsity regime, e.g. 95% or even 99%? Or is it applicable only for the sparsity level within a range?

Author response:
We did initially also perform a few experiments for 93.75% sparsity (16x compression); however, as also suggested by, e.g. Figure 2, improvements were diminishing and producing those models quite expensive (we have to train 16x larger models to eventually reach a certain target nonzero count). Thus, we decided to focus our computational budget on the most interesting and practical range. In principle, our scaling laws should also extend to higher sparsities (though, one should be careful with extreme extrapolation for any kind of scaling law due to potential problems like training instabilities for very unusual configurations).