Scaling Laws Under the Microscope: Predicting Transformer Performance from Small Scale Experiments (2022)

Abstract: Neural scaling laws define a predictable relationship between a model’s parameter count and its performance after training in the form of a power law. However, most research to date has not explicitly investigated whether scaling laws can be used to accelerate model development. In this work, we perform such an empirical investigation across a wide range of language understanding tasks, starting from models with as few as 10K parameters, and evaluate downstream performance across 9 language understanding tasks.We find that scaling laws emerge at finetuning time in some NLP tasks, and that they can also be exploited for debugging convergence when training large models. Moreover, for tasks where scaling laws exist, they can be used to predict the performance of larger models, which enables effective model selection. However, revealing scaling lawsrequires careful hyperparameter tuning and multiple runs for the purpose of uncertainty estimation, which incurs additional overhead, partially offsetting the computational benefits.

## introduction transformer-based language models (lms) @xcite @xcite @xcite which are at the foundation of modern nlp systems, have been recently shown to exhibit scaling laws @xcite @xcite , that is, the test loss of lms obeys a predictable power law with respect to the number of model parameters, dataset size, and computation budget. this finding ignited substantial research that demonstrated scaling laws in a wide range of areas including computer vision @xcite @xcite @xcite , acoustic models @xcite , and board games @xcite , among others. on top of being a fascinating phenomenon on its own, scaling laws can potentially be harnessed for more efficient development of models. specifically, if scaling laws hold, one can perform modeling decisions at multiple small scales, and extrapolate to infer which model will perform best at a larger scale. while starting small is an established technique @xcite , scaling laws can provide a more principled framework for this methodology. moreover, scaling laws can potentially accelerate the development cycle, and reduce carbon footprint caused by training large neural models @xcite . however, for this idea to materialize, several questions must be addressed, which have not been fully considered by past literature. first, do scaling laws consistently occur at finetuning time across a wide range of language understanding tasks? second, do they manifest reliably at the small-scale regime? and third, what is the predictive power of scaling laws when used to predict the behavior of large models for the purpose of model selection? while recent work touched upon these questions @xcite @xcite , most work focused on post-hoc analysis of highlyparameterized models, and looked at performance on aggregate benchmarks, such as glue @xcite and superglue @xcite , without analyzing performance at the level of individual dataset. in this work, we perform a thorough empirical evaluation of scaling laws from the perspective of nlp researchers with resource constraints, addressing the aforementioned questions. we analyze scaling laws at finetuning time across 9 different tasks with models ranging from merely 10k parameters up to roughly 100m (e.g. bert-base @xcite ). moreover, we move away from posthoc analysis and evaluate whether scaling laws can be used to predict the performance of larger models. as a case study, we test whether scaling laws can be used for model selection, by comparing the performance of two lms with two different pretraining objectives: masked language modeling (mlm) @xcite and pointwise mutual information (pmi) @xcite . our experiments reveal several findings. overall, our empirical findings paint a nuanced picture of the potential of scaling laws as a tool for model design. on one hand, we observe scaling laws at finetuning time for some nlp tasks, and show that they can be used to predict the performance of a model that is 10x larger. on the other hand, this does not happen consistently on all tasks, and revealing these scaling laws requires careful control over hyperparameters and convergence conditions, incurring additional overhead that might counteract the computational benefits. ## method we describe our experimental setup in §2.1, and then our procedure for evaluating goodness-of-fit and the predictive power of scaling laws in §2.2. ## pretraining we pretrain all models from scratch on the datasets @xcite wikipedia dump, see experimental details in app. a.1. we use early stopping to declare convergence, but note that determining convergence is non-trivial, as we discuss in §3.1. fig. 2 shows the evaluation loss of each configuration as a function of the parameter count for both mlm and pmi. as can be seen from the clean linear relationships (in log-log scale), both aspect ratios and both objectives present a power law, with r 2 exceeding 0.99 in all cases. this is consistent with the language modeling results of @xcite , and shows that scaling laws exist at pretraining time for the mlm and pmi objectives. past work @xcite ) examined scaling laws of different architectures (e.g., transformer @xcite , reformer @xcite , performer @xcite ). they showed that while different architectures affect scaling laws, architectural hyperparameters, such as aspect ratio (ar), make little difference as long as they are within a reasonable range. interestingly, in both mlm and pmi the slope of ar 64 is slightly better than the slope of ar 32, even when taking into account the slope's 95% confidence intervals. the intersection of the two ar lines illustrates the potential of using scaling laws for model selection: choosing the ar based on small scale experiments would lead to choosing ar 32, while the performance of models with ar 64 seems comparable and perhaps better in the larger scale. however, the confidence intervals of the fit, depicted as sleeves in fig. 2 , intersect at the larger scales, meaning we cannot predict a performance difference with confidence. we note that our largest mlm model, which uses ar 64, 12 layers, and has 85m trainable parameters performs slightly worse than predicted by the power law, which might hint that it is undertrained. we leave verifying this and training larger models with different ars to future work. ## discussion and future work in this work, we show that scaling laws can be used as an effective tool for model selection and as a diagnostic tool to test convergence of large-scale models. our practical takeaways are: ## conclusion this work is motivated by a practical question: can scaling laws provide a principled method for developing models at very small scale and extrapolating to larger ones? we perform a thorough empirical analysis on the emergence of scaling laws on a wide range of language understanding tasks. we find that scaling laws emerge for some tasks, potentially as a function of the proximity between the downstream task and the pretraining objective, but revealing them incurs the overhead of hyperparameter tuning across multiple scales. our results show that scaling laws are beneficial for debugging model convergence when training large models, and to predict model behavior when they emerge at small scale.