Scaling Laws Are Unreliable for Downstream Tasks: A Reality Check (2025)

Abstract: Downstream scaling laws aim to predict task performance at larger scales from the model’s performance at smaller scales. Whether such prediction should be possible is unclear: some works discover clear linear scaling trends after simple transformations of the performance metric, whereas others point out fundamental challenges to downstream scaling laws, such as emergence and inverse scaling. In this work, we conduct a meta-analysis of existing data on downstream scaling laws, and we find that predictable scaling only occurs in a minority of cases: 39% of the time. Moreover, seemingly benign changes to the experimental setting can completely change the scaling behavior. Our analysis underscores the need to understand the conditions under which scaling laws succeed. To accurately model the relationship between pretraining loss and task performance, we must embrace the cases in which scaling behavior deviates from linear trends.

## introduction scaling laws for pretraining establish that the loss improves reliably when increasing the size of the model, training data, or compute @xcite @xcite . however, better pretraining loss does not always translate to better downstream performance @xcite . this gap can be caused by a variety of issues; among the best known is emergence, or the fact that on some tasks models below a certain scale show no trend or near-chance performance @xcite . in addition, model performance can increase then decrease, in what is called inverse scaling @xcite . despite these challenges, several works also find downstream performance is roughly linear in the pretraining loss, possibly after a transformation @xcite @xcite . 2025 ), we find that only 18 tasks-or 39%demonstrate smooth, predictable improvement (figure 5 ). the other 28 tasks are shown in figures 6 through 10, where we group them into different degenerate scaling behaviors: inverse, nonmonotonic, noisy, trendless, and breakthrough scaling. see figure 2 for examples. thus, the extent to which downstream scaling laws work is unclear. how can they follow linear forms when we know of many tasks that exhibit emergence or inverse scaling? are these just edge cases, or are they common, with no established explanation? here, we aim to clarify this confusion. we explore and consider three core factors affecting downstream scaling laws: 1) the data used for pretraining and validation, 2) the downstream task, and 3) the experimental setup. realistic changes to each of these factors can change the relationship to downstream performance, to the point that the scaling law's functional form might no longer hold. the behavior can even change qualitatively-with a decrease in trend under one set of conditions flipping to an increase under another. such qualitative changes can not be removed by a new functional form. they pose fundamental obstacles to studying large scale models with small scale proxies. figure 2 : a taxonomy of different scaling behaviors. predictable scaling fits closely to a linear functional form after, for example, exponentiating the cross-entropy loss. however, depending on the downstream task, models do not always improve with scale (inverse, nonmonotonic, and trendless), or the improvement might be highly noisy. the improvement might also follow a functional form that is difficult to extrapolate like a sigmoid (breakthrough). in particular, we find that: our analysis suggests that scaling laws are context-specific. as such, we cannot assume downstream scaling will always be strictly linear. rather, we need to better understand the failure modes of existing scaling laws and develop a holistic (and perhaps more complex) model of how foundation models improve on downstream tasks. ## background downstream scaling laws try to extrapolate the performance of large-scale models from smallscale proxies. when successful, these scaling laws enable cost-effective experiments at the small scale that transfer to the large. scaling laws for pretraining are well established @xcite @xcite ; however, the ultimate goal of language models is to perform well on downstream tasks. as such, downstream scaling laws are also of great interest. unlike pretraining, there is little consensus on how to approach downstream scaling laws. early efforts tried to predict downstream performance directly from parameters, data, or compute @xcite @xcite , but downstream performance often showed a noisy relationship to these quantities @xcite . other efforts sought surrogates for scale, like latent capabilities @xcite or task-specific losses @xcite , in the hope of mapping from compute to surrogate, and from surrogate to downstream performance. many works have found downstream scaling laws are more stable when stated in terms of pretraining loss, the most widely used surrogate for scale @xcite @xcite @xcite . specifically, if two models differ in their number of parameters or pretraining tokens but still attain the same pretraining loss, then they tend to achieve the same downstream performance @xcite @xcite . some authors generalize this principle, viewing pretraining loss as a general way to compare models-not just with different scales but different training recipes @xcite . however, while pretraining loss correlates with downstream performance to a surprising degree, this relationship is far from absolute @xcite . in the best case, downstream task performance is roughly linear in some monotonic transformation of validation loss (x) and downstream metric (y): for instance, gadre et al. ( 2025 ) relate the error rate to the validation loss via: @xmath0. the functions, f and g, must depend on the metric; in some cases, a linear relationship alone could be enough @xcite . such global structure enables extrapolation-the ultimate goal of downstream scaling laws; however, predictable structure might not exist. phenomena such as emergence, inverse, and u-shaped scaling preclude this kind of global structure @xcite @xcite . they destroy it by creating structural breaks, or points where the function describing one part of the scaling curve does not describe another @xcite . without global structure, it is impossible to extrapolate from small to large. certain tasks, like multiple choice questions, are more susceptible to emergence @xcite . sometimes emergence can be mitigated by choosing a different downstream metric @xcite , but in other cases breakthrough improvements remain "stubbornly emergent" @xcite . @xcite found scaling laws generally difficult to reproduce and sensitive to the functional form, training setup, data collection, and fitting algorithm, while @xcite found that comparing models at a smaller scale-without extrapolation-performed as well or better than scaling laws in determining the best data mix. 3 scaling laws are specific to the data downstream scaling laws depend on several factors, including the pretraining data, validation data, and downstream task. if you vary the pretraining data, then you must fix a validation corpus to compare the loss across models. once it is fixed, you might hope to focus on the validation loss alone and simplify your research. it is unclear how to choose the correct validation data but, what is more, its choice can entirely reverse which pretraining setup appears superior for a downstream task if we do not consider the full context. the pretraining data, validation data, and downstream task all interact in forming the scaling law-you can not consider one without the other two. in particular, you can not compare data mixes by their validation losses or downstream scaling laws based upon them. to evince these claims, we reexamine the results from @xcite , who pretrained models over different corpora. we observe how pretraining data, validation data, and task all interact in forming the scaling law (figure 3 ). the colors correspond to pretraining corpora: c4 @xcite and redpajama @xcite ; the columns correspond to validation data: c4 @xcite ) and paloma's 100 program-ming languages @xcite ; and the rows correspond to tasks: coqa @xcite and hellaswag @xcite . if you ignore the other factors, then choice of validation data can completely change the scaling trend: exaggerating differences. for hellaswag with c4 as the validation corpus (top left), pretraining on either c4 or redpajama produces the same scaling law. however, when using 100 programming languages (100 pls, top right), the scaling laws for c4 and redpajama no longer superimposepretraining on c4 appears to achieve much better performance even for a worse validation loss. ## irregular scaling is common phenomena like emergence and inverse scaling suggest that linear scaling laws do not capture all task scaling behavior. however, it is also unclear how prevalent these phenomena are in practice. thus, we re-examine scaling behavior on the 46 tasks tested by @xcite , classifying them into six categories visualized in figure 2 . we find that linear scaling actually occurs in a minority of cases in their setting: 39% of the time (figure 1 ). for some experimental setups, non-linear scaling is actually the norm. are the experimental choices of @xcite abnormal? on the contrary, all three of pretraining corpora, validation datasets, and downstream tasks are from well-known sources in the literature. downstream tasks are comprised of established evaluations like boolq, hellaswag, and big-bench @xcite @xcite . irregular scaling occurs within popular tasks and is easy to find. ## scaling behavior is not always robust finally, we show that conclusions about scaling laws may not generalize across settings: setups with the same validation data and downstream tasks may observe entirely different scaling trends. to show this, we take the 10 overlapping tasks between @xcite and @xcite . these authors consider several of the same pretraining corpora and downstream tasks; however, their implementation details differ (see appendix a). for example, gadre et al. ( 2025 ) use fewer answer choices for commonsense qa. thus, we might expect to see quantitative differences in some of their scaling laws, but what is more surprising is that we also see qualitative changes in their scaling behavior. to enable comparison, we evaluate all models from gadre et al. (2025) and @xcite on the same validation corpus (c4). for magnusson et al. (2025) , which does not use c4, we evaluate 200 released checkpoints, which vary in their pretraining setup: foot_0 1. parameters: {20m, 60m, 150m, 300m, 1b}. 2. pretraining corpus: {dolma, c4, dclm-baseline, refinedweb (falcon), fineweb-pro} @xcite @xcite @xcite . 3. training steps: from 5,000 to 40,000, in intervals of 5,000. out of the 10 overlapping downstream tasks, figure 4 shows three that produce different scaling trends between setups. in both setups, mmlu shows a positive trend; however, the noisiness and shape of that trend differs greatly. at an even greater extreme, commonsenseqa's scaling behavior qualitatively changes: while common-senseqa shows nonmonotonic scaling in gadre et al.'s (2025) results, it exhibits a clean scaling law with @xcite . on the other hand, in @xcite results, boolq's scaling law appears trendless; however, this lack of trend comes from the models spanning too small a validation loss range. since gadre et al.'s (2025) models cover a wider range of validation losses, the trend more clearly emerges. thus, scaling behavior can fluctuate, even between controlled studies for the same task with the same pretraining corpora. ## discussion given the cost of training modern foundation models, scaling laws have become an invaluable tool for making informed modeling decisions. scaling laws enable us to extrapolate results where compute costs would otherwise make extensive experimentation infeasible. however, extrapolations are only worthwhile when their assumptions are faithful to the data. as practitioners of scaling laws, we must realize that predictable scaling laws often exist, but one cannot assume that they hold for all contexts. even if scaling is stable on the same task for the same validation data, other aspects of the experimental setup might change the scaling behavior ( §5). to some extent, scaling laws are investigatorspecific @xcite , and so each investigator must verify the scaling law's presence with visualizations and regression diagnostics @xcite . for researchers, downstream scaling laws offer many fascinating new directions. empirically, we need better ways to stabilize scaling laws and detect when irregular scaling might occur. we must understand the factors affecting scaling laws, and what parts of the experimental setup must remain constant for linear scaling laws to hold. theoretically, we need a model for why predictable scaling occurs @xcite , and a core goal of such a theory should be explaining exactly the cases in which it does not. ## conclusion in this work, we surveyed where downstream scaling laws break down. depending on the pretraining corpus, validation corpus, or downstream task, scaling laws can change. better perplexity does not always translate to better downstream performance ( §3); perplexity is not all you need. even when holding pretraining and validation data the same, more often than not a predictable scaling law does not exist at all ( §4). irregular behaviors like nonmonotonic, trendless, or breakthrough scaling are all common; one must establish predictable scaling for the given task before relying on it. finally, seeing predictable scaling in one experimental setup does not guarantee it for another ( §5). until we better understand why predictable scaling arises and its sufficient conditions, investigators must verify scaling laws in their own settings.