A Proposal for Scaling the Scaling Laws (2024)

Abstract: Scaling laws are predictable relations between the performance of AI systems and various scalable design choices such as model or dataset size. In order to keep predictions interpretable, scaling analysis has traditionally relied on heavy summarisation of both the system design and its performance. We argue this summarisation and aggregation is a major source of predictive inaccuracy and lack of generalisation. With a synthetic example we show how scaling analysis needs to be _instance-based_ to accurately model realistic benchmark behaviour, highlighting the need for richer evaluation datasets and more complex inferential tools, for which we outline an actionable proposal.

## introduction analysing how ai systems scale -how their performance is affected by various design choices such as parameter count or dataset size -has become a fruitful empirical tool: it informs the design of new generations of (scaled-up) systems @xcite , uncovers architectural limitations @xcite , and generally helps both industry and policy in planning for what the near future of ai might look like. for example, the concept of scaling laws @xcite deals with capturing predictable patterns in the relation between scale and performance into simple mathematical relations, from which data driven extrapolations and predictions about nextgeneration performance can then be made. despite the usefulness of scaling analysis, there are also several issues. a primary concern is generalisation. scaling laws need to be tailored (i.e. fitted) to different domains, architectures, and often even to each set of model hyperparameters independently. there is no universal 'scaling law' @xcite . insights that generalise across tasks and metrics are rare. a second notable issue is predictive accuracy. for example, modelling breakpoints -changes in the behavioural trend -has proven difficult, partly because of the limited expressivity of the functional forms @xcite , but also because new capabilities seemingly emerge out of the blue at certain scales @xcite foot_0 . we argue that oversummarisation is a significant contributing factor to these issues. firstly, the dimensions of scale and size capture only a small part of technological innovation, and are a rudimentary summary of the attributes that define and differentiate ai systems overall. current methods typically consider only one or two scalable design choices. this is the oversummarisation of systems. secondly, the empirical aggregate performance metrics that act as the unit of analysis are, by construction, summary statistics. by not looking at the actual features of the task instances -like a researcher might -performance is treated as an abstract number, devoid of information that could explain differences. the detection of patterns underlying the relation between task features, system features, and performance is off the table from the start. for example, the aggregate metrics cannot capture any difference in scaling behaviour between subsets of the benchmark. this is the oversummarisation of task performance. while this heavy summarisation is sensible in the light of interpretability or data scarcity, it comes at a cost of generalisation and predictive power. with major nlp evaluation efforts like big-bench @xcite and helm @xcite producing huge quantities of instance-level evaluation results across a plethora of different ai systems, it is time to capitalise on the available data, and much like we scale ai itself, to also scale the inferential tools we use in our analysis of ai. the plot shows ten synthetic ai systems, whose synthetic evaluation scores are designed to be dependent on an abstract feature of the system. for example, system 2 has feature-value 20 (e.g. number of parameters), and has a mean score of about 0.2. the violin plots, with the quantiles marked, represent the distribution of scores of the respective systems. the red line is a power law fitted to the mean scores, while the blue line represents the aggregated predictions of a simple multi-layer perceptron (mlp) that predicts instance level scores. both are trained/fitted on the smallest seven systems only. the last three systems then act as a test for the performance predictor. ## synthetic example to illustrate the challenges outlined earlier and to lay the foundation for our proposed methodology, we present a synthetic scenario where the scaling behaviour cannot be modelled from aggregate measures. the setup is as follows: we hypothesise ten ai systems, each of which scales up over the same (abstract) system feature, e.g. number of model parameters. we also devise a simple synthetic dataset consisting of 1000 instances divided into two subpopulations. the instances of the dataset synthetically have only one feature: a one-hot coded vector indicating which of the two different subgroups of the benchmark the instance belongs to. to bring this to life, consider the task of sentiment classification of english text, whose domain would naturally contain a blend of english varieties, e.g. 'standard english', acting as subpopulation 1, and african-american vernacular english (aave), acting as subpopulation 2. in this scenario, a onehot vector indicating the subpopulation would not be provided explicitly, but actual features of the english variants would allow identifying the texts as belonging to different populations. we now generate synthetic evaluation results, where we design the scores to be dependent on the scalable system feature. we simply let the mean score increase as the system feature scales. we also make this relation between scale and score differ between the two subpopulations, e.g. the sentiment of avee might be harder to classify than that of standard english, for example due to lower representation in training data. the scores are in the range [0, 1], representing e.g. the probability assigned to the correct class. figure 1 illustrates the example. observing only the mean scores, a conventional scaling analysis could sensibly only make a linear extrapolation (in red). on the other hand, an instance-based approach could discern the distinct subpopulations, noting that performance must saturate in the first group while increasing more gradually in the second. to exemplify this, we train a simple neural network foot_1 on the set of synthetic evaluation records foot_2 to predict instance-level scores, that can correctly extrapolate to larger systems (blue curve). evaluation records system features (id, #params, #tokens) instance features (same as the systems gets) model outputs (optional) score (number) gpt4, 1.8t, 2t what movie do these emojis represent? 🚗⌚ 1 gpt4, 1.8t, 2t translate "can i have the bill please?" into italian. 0.7 bard, 350g, 1.5t what movie do these emojis represent? 🚗⌚ 0 ... ... ... while the example is obviously exaggerated and idealistic, benchmark subgroups are not uncommon @xcite . in a high-dimensional problem space like nlp, the subgroups are however not as crisp as in our example, and identifying them is far from straightforward; this complexity is precisely why we need more sophisticated statistical methods beyond simple aggregate measures. in general, it is hard to isolate a single capability in benchmark @xcite , if that even makes sense for novel kinds of intelligence like llms. in reality, there will be a mixture of (meta-)features of both system and instance that influence the scores in complicated ways. example instance features that the literature has shown to be impactful are input length or grammatical complexity @xcite ; clever hans phenomena and general confounding @xcite ; mislabelling @xcite , label disagreement @xcite , or task ambiguity @xcite ; or general dependence on other skills, e.g. for dealing with numeric values @xcite , negation @xcite , or social understanding @xcite . while these phenomena are also tested for individually, they are nonetheless confounding factors in most benchmarks. they influence scaling behaviour in currently unknown ways and require us to actually relate scores to instance features, instead of treating performance as an abstract number. ## proposal our proposed approach emphasises the integration of detailed evaluation data. it involves following three-step process: ## related work scaling laws in deep learning research focus on empirical relationships between performance metrics and design choices such as architecture, model size, or dataset size. initially driven by findings that test loss scales with training data size in a power-law fashion @xcite , research has diversified to analyse a range of tasks and architectures @xcite @xcite and to theorise scaling exponents @xcite @xcite . however, recent work highlights the non-universal applicability of these laws, particularly in predicting downstream task performance @xcite @xcite , which is further complicated by the nuances of transfer learning @xcite . in general, we find a critical gap in current methods: the over-reliance on aggregated data and limited system characteristics. approaches that deal with oversummarization of systems are proposed by @xcite and @xcite , which learn or meta-learn from multiple system features and therefore generalise better across systems and tasks, but still work at the aggregate performance level. instance-level score prediction is closely related to the notion of predictive uncertainty and calibration in probabilistic systems. including for llms, it revolves around the idea that these systems can signal their own confidence by assigning probabilities to potential outcomes, much as we expect from evaluative models. predictive uncertainty is the focus of intense research @xcite @xcite , but conclusions are often contradictory or context dependent. the fields of anomaly detection and confidence estimation (e.g. @xcite are closely related as well. as described by @xcite , these investigations typically assume requirements that make them differ from the pure 'performance prediction' perspective adopted in our approach, e.g. by not being anticipative and requiring access to model outputs or internals, both of which are not available in the context of scaling laws. the performance prediction idea also extends and is influenced by other research areas, such as item response theory @xcite , which predicts success based on system ability and task difficulty, and techniques such as surrogate evaluation @xcite and datamodels @xcite , which examine model behaviour in relation to training data. in addition, methods for detailed error analysis @xcite contribute to the understanding of model performance by identifying incorrect predictions and highlighting strengths and weaknesses. ## conclusion acknowledging the challenges of scaling analysis, our proposal aims to mitigate them by leveraging a richer dataset and more powerful inferential tools, i.e. "scaling the scaling laws". the approach unlocks various new applications and aspires to enhance predictive accuracy and generalisation, ultimately aiming for a single assessor model doing inference about scaling behaviour for all tasks and systems with sufficient evaluation data available. we invite the research community to contribute to this endeavour by harnessing instance-level evaluations and amplifying the collective progress in understanding ai performance.