Compute Optimal Scaling of Skills: Knowledge vs Reasoning (2025)

Abstract: Scaling laws are a critical component of the LLM development pipeline, most famously as a way to forecast training decisions such as ‘compute-optimally’ trading-off parameter count and dataset size, alongside a more recent growing list of other crucial decisions. In this work, we ask whether compute-optimal scaling behaviour can be skill-dependent. In particular, we examine knowledge and reasoning-based skills such as knowledge-based QA and code generation, and we answer this question in the affirmative: scaling laws are skill-dependent. Next, to understand whether skill-dependent scaling is an artefact of the pretraining datamix, we conduct an extensive ablation of different datamixes and find that, also when correcting for datamix differences, knowledge and code exhibit fundamental differences in scaling behaviour. We conclude with an analysis of how our findings relate to standard compute-optimal scaling using a validation set, and find that a misspecified validation set can impact compute-optimal parameter count by nearly 50%, depending on its skill composition.

## introduction used both to forecast performance for early pretraining decisions as well as decide on the optimal trade-off between parameters and pretraining dataset size given a particular compute budget, scaling laws @xcite have played an important role in the development of large language models (llms). famously, with a series of experiments with models with different data/parameter @xcite showed that previous llms were all erring on the side of too many parameters, causing a shift in the amount of training tokens to train llms. more recently, @xcite used scaling laws not only to determine the optimal parameter count given their available compute budget, but also to forecast the impact of data selection decisions on evaluation scores. in these works, the compute optima (cos), describing the optimal parameter count and number of training tokens, are selected based on aggregate performance estimators (apes), in the form of negative log-likelihood (nll) on a validation set not part of the pretraining corpus. little is known, however, about whether the cos of individual skills such as mathematical reasoning, question answering (qa), or coding, align with these ape cos. while some studies use scaling laws to predict how downstream task performance improves with scale (e.g. @xcite , none of these studies cover whether cos themselves may be skill dependent. is it possible that some skills are more data-hungry, whereas others benefit more from extra parameters? if so, how should that impact model training and training data selection? in this paper, with an extensive set of experiments across 9 different compute scales and 2 skills as measured with 19 datasets across two different splits, we study exactly that. specifically, we focus on the three research questions: r1. are cos skill dependent? first, we consider how the isoflop curves and corresponding cos for code foot_0 -and knowledge-based skills compare to the ape cos, given a canonical datamix. across the board, we find pronounced differences between the cos for these different skills: where knowledge qa tasks are capacity-hungry, code tasks instead prefer data. r2. is this an artefact of the pretraining datamix, or are code and knowledge skills fundamentally different? using only the smallest compute scale, we next investigate whether these difference are a consequence of the proportion of skill-relevant data in the pretraining datamix of our experiments or that there is in fact a difference in how data-or capacity-hungry the different skills are. we find that both are true: changing the proportion of skill-relevant data shifts the co for that skill, but cos for skills differ even with comparable proportions of skill specific data. r3. how does the existence of skill-dependent cos impact llm training? lastly, we focus on the more practical question of how these findings should impact model pretraining. first, again using only the smallest compute scale, we investigate if it is possible to align the cos for the two skills under consideration. we find that it is, but that whether this should be considered advantageous for overall model training depends highly on the choice of validation set. we then investigate how much the estimated optimal parameter count depends on the choice of validation set. we find that, on our smallest scale experiments (6 × 10 18 ), the optimal parameter count for the same skill varies by almost 50% across datamixes, and by more than 30% across validation sets. at larger scale, the difference is smaller, but it still exceeds 10% for the three largest compute scales. these experiments show that choosing a validation set that adequately represents what the final model should capture is critical to finding the right parameter counts. outline with our work, we contribute to an empirical understanding of how different skills behave across model scales, and explore the concrete implications for co training of llms. in the remainder of this paper, we first provide the necessary background and definitions to understand our work ( § 2), and we describe the data we use for our experiments ( § 3). next, in § 4, we describe our experiments and their results. lastly, we discuss related works about scaling and data selection ( § 5), and we conclude in § 6. ## background when computing scaling laws, prior work typically focused on modelling the loss on a validation set, which is usually a general text corpus drawn from the same distribution as the pretraining dataset. in this setup, the loss is effectively a weighted combination of all what the model should learn during training. in contrast, we consider skill-dependent scaling laws. in this section, we discuss the relevant background and notions of scaling laws, cos, and skills underpinning our analysis. in § 5, we describe related efforts in more detail, and in appendix c, we provide formal definitions of the concepts that this section introduces. ## data in our experiments, we focus on two overarching skills: knowledge qa and code. in this section, we describe which datasets we use to measure models' abilities on those skills and describe how we infer their corresponding cos. ## experiments now, we present the main results for our experiments, starting from our finding that cos are skill specific for our canonical datamix ( § 4.1), followed by a deeper exploration into the impact of the datamix on these conclusions ( § 4.2), and finishing with a small investigation of how these results may impact model pretraining decisions ( § 4.3). ## related work scaling laws and compute optimality the first large-scale empirical study of statistical scaling laws of neural networks was done by @xcite , who studied scaling laws for a variety of figure 6 : fluctuation in optimal parameter count as function of validation set. in (a), we show the cos of the validation sets that we evaluate, showing that at a high ratio of knowledge qa to code, two of the three validation sets have cos with substantially higher parameter counts. in (b), we see this information in terms of a relative increase in parameter count compared to the stack, and see that the increase can be up to roughly 50%. in (c), we show the co scaling behaviour across compute scales between validation sets. in (d), we see the relative parameter count increases over the stack, and see that at high compute scales, the increase is smaller, yet it still exceeds 10%. ## conclusion in this work, motivated by the impact that scaling laws have had on the landscape of training llms, we examine the existence of skill-dependent scaling laws. specifically, we investigate how the optimal trade-off between capacity (in the form of parameter count) and training data varies across skills. focusing on two different skills -code generation and knowledge-based qa -we find pronounced differences in how data-or capacity-hungry they are: where knowledge qa is capacity-hungry, code instead prefers data. these differences hold across tasks and splits, and are true even if the proportion of skill-dependent data in the pretraining datamix is factored out. furthermore, as the proportion of skill-dependent data grows, the capacity-hunger of knowledge-based skills grows more rapidly than that of code, indicating that models are able to compress the latter better than the former. in the second part of our experiments, we focus on the impact of these findings to llm training. we find that both the use of validation set and the specific datamix used to do scaling experiments have a substantial impact on the estimated compute optimal points. in our 6 × 10 18 flop experiments, we find that varying the ratio of code and knowledge data can result in a difference of 30% or more in the estimated optimal parameter count for that budget for a single validation set, and almost 50% depending on the validation set. similarly, the use of validation set can result in a mismatch of over 30% on the smallest scale, and over 10% on the largest three scales (up to 3 × 10 21 ). as such, our experiments show that choosing a validation set that adequately represents a mix of what the final model should capture is critical to finding the right cos as well as that doing data ablations after selecting cos may result in suboptimal parameter counts for the specific datamix. our analysis also opens intriguing directions for future study, such as understanding the relationship between compression and skills through the lens of cos, as well as understanding the flatness or curvature of isoflop curves computed on the nll compared to accuracy-based metrics. crucially, we note that the bulk of our analysis can be easily extended to additional skills using open source scaling suites, as it only requires running evaluations on pretrained models.