Training Compute-Optimal Large Language Models
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget.
We ﬁnd that current largelanguage models are signiﬁcantly undertrained, a consequence of the recent focuson scaling language models whilst keeping the amount of training data constant.
By training over 400 language models ranging from 70 million to over 16 billionparameters on 5 to 500 billion tokens, we ﬁnd that for compute-optimal training,the model size and the number of training tokens should be scaled equally: forevery doubling of model size the number of training tokens should also be doubled.
We test this hypothesis by training a predicted compute-optimal model, Chinchilla , that uses the same compute budget as Gopher but with 70B parameters and4 ⇥ more more data.
Chinchilla uniformly and signiﬁcantly outperforms Gopher(280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) ona large range of downstream evaluation tasks.
This also means that Chinchillauses substantially less compute for ﬁne-tuning and inference, greatly facilitatingdownstream usage.
As a highlight, Chinchilla reaches a state-of-the-art averageaccuracy of 67.5% on the MMLU benchmark, greater than a 7% improvement over
Gopher .
Introduction
A series of Large Language Models (LLMs) have recently been introduced [7, 31, 39, 50, 53], with the 19 largest dense language models now having over 500 billion parameters.
These large autoregressivetransformers [54] have demonstrated impressive performance on many tasks using a variety of 21 evaluation protocols: zero-shot generalization, few-shot training and as a basis for ﬁne-tuning.
Thecompute and energy cost for training large language models is substantial [39, 53] and rises with 23 increasing model size.
In practice, the allocated training compute budget is often known in advance:practitioners have access to a certain number of accelerators for a given period of time.
Since it 25 is typically only feasible to train these large models once, accurately estimating the best modelhyperparameters for a given compute budget is critical [52].
Kaplan et al.
[24] showed that there is a power law relationship between the number of parameters 28 in an autoregressive language model (LM) and its performance (measured in evaluation perplexity).
One notable conclusion in [24] is that large models should not be trained to their lowest possible loss 30 to be compute optimal; they argue that model size should grow faster than the size of the trainingset for a given increase of computational budget.
As a result, the ﬁeld has been training larger 32 and larger models while keeping the size of the training set to approximately 300 billion tokens,expecting performance improvements (Table 1).
While we ﬁnd that there is effectively a trade-off 34 between model size and training set size, we estimate that large models should be trained for manymore training tokens than recommended by [24]. Speciﬁcally, given a 10 ⇥ increase computational predict substantially smaller and therefore currently
In Figure , we with the predicted plotted against the
Chinchilla outperforms Gopher and the other large models (see Section
[Table]
budget, Instead, we ﬁnd that model size and the number of training tokens should be scaled in equal 37 proportions.
In this work, we revisit the question: Given a ﬁxed FLOPs budget, 1 how should one trade-off model 39 size and the number of training tokens?
To answer this question, we model the ﬁnal pre-trainingloss 2 L ( N, D ) as a function of the number of model parameters N , and the number of trainingtokens, D .
Since the computational budget C is a deterministic function FLOPs ( N, D ) of the numberof seen training tokens and model parameters, we are interested in minimizing L under the constraint 43 FLOPs ( N, D ) = C :
[Equation]
The functions N opt ( C ) , and D opt ( C ) describe the optimal allocation of a computational budget reduced model size reduces inference cost considerably and greatly facilitates downstream uses on 54 smaller hardware.
The energy cost of a large language model is amortized through its usage forinference and ﬁne-tuning.
The beneﬁts of a more optimally trained smaller model, therefore, extend 56 beyond the immediate beneﬁts of its improved performance.
Related Work
Large language models.
A variety of large language models have been introduced in the last few 59 years.
These include both dense transformer models [7, 31, 50, 39, 53] and mixture-of-expert (MoE)models [12, 13, 61].
The largest dense transformers have passed 500 billion parameters [50, 9].
The 61 drive to train larger and larger models is clear—so far increasing the size of language models has beenresponsible for improving the state-of-the-art in many language modelling tasks.
Nonetheless, large 63 language models face several challenges, including their overwhelming computational requirements(the cost of training and inference increase with model size) [39, 53] and the need for acquiring more 65 high-quality training data.
In fact, in this work we ﬁnd that larger, high quality datasets will play a keyrole in any further scaling of language models.
Concurrent to our work, a 540 billion parameter model 67 trained on 768 billion tokens was released– PaLM [9].
While this model outperforms Chinchilla , ituses approximately 5 ⇥ the compute and is nearly 8 ⇥ larger, making it more difﬁcult to use.
Modelling the scaling behavior.
Understanding the scaling behaviour of language models and their 70 transfer properties has been important in the development of recent large models [24, 19].
[24] ﬁrstshowed a predictable relationship between model size and loss over many orders of magnitude.
The 72 authors investigate the question of choosing the optimal model size to train for a given compute budget.
Similar to us, they address this question by training various models.
Our work differs from [24] in 74 several important ways.
First, the authors use a ﬁxed number of training tokens and learning rateschedule for all models; this prevents them from modelling the impact of these hyperparameters on 76 the loss.
In contrast, we ﬁnd that setting the learning rate schedule to approximately match the numberof training tokens results in the best ﬁnal loss regardless of model size—see Figure A1.
For a ﬁxed 78 learning rate cosine schedule to 130B tokens, the intermediate loss estimates (for D 0 << 130 B) aretherefore overestimates of the loss of a model trained with a schedule length matching D 0 .
Using these 80 intermediate losses results in underestimating the effectiveness of training models on less data than130B tokens, and eventually contributes to the conclusion that model size should increase faster than 82 training data size as compute budget increases.
In contrast, our analysis predicts that both quantitiesshould scale at roughly the same rate.
Secondly, we include models with up to 16B parameters, as we 84 observe that there is slight curvature in the FLOP-loss frontier (see Appendix E)—in fact, the majorityof the models used in our analysis have more than 500 million parameters, in contrast the majority 86 of runs in [24] are signiﬁcantly smaller—many being less than 100M parameters.
Clark et al.
[10]speciﬁcally looked in to the scaling properties of Mixture of Expert language models, showing that 88 the scaling with number of experts diminishes as the model size increases—their approach models theloss as a function of two variables: the model size and the number of experts.
However, the analysis 90 is done with a ﬁxed number of tokens, potentially underestimating the improvements of branching.
Estimating hyperparameters for large models.
The model size and the number of training tokens 92 are not the only two parameters to chose when selecting a language model and a procedure to trainit.
Other important factors include learning rate, learning rate schedule, batch size, optimiser, and 94 width-to-depth ratio.
In this work, we focus on model size and the number of training steps, andwe rely on existing work and provided experimental heuristics to determine the other necessary 96 hyperparameters.
Yang et al.
[58] investigates how to choose a variety of these parameters for trainingan autoregressive transformer, including the learning rate and batch size.
McCandlish et al.
[34] ﬁnds 98 only a weak dependence between optimal batch size and model size.
Shallue et al.
[48], Zhang et al.[60] suggest that using larger batch-sizes than those we use is possible.
Levine et al.
[29] investigates 100 the optimal depth-to-width ratio for a variety of standard model sizes.
We use slightly less deepmodels than proposed as this translates to better wall-clock performance on our hardware.
Improved model architectures.
Recently, various promising alternatives to traditional dense 103 transformers have been proposed.
For example, through the use of conditional computation large
MoE models like the 1.7 trillion parameter Switch transformer [13], the 1.2 Trillion parameter GLaM 105 model [12], and others [1, 61] are able to provide a large effective model size despite using relatively [Equation] fewer training and inference FLOPs.
However, for very large models the computational beneﬁts of 107 routed models seems to diminish [10].
An orthogonal approach to improving language models isto augment transformers with explicit retrieval mechanisms, as done by [5, 16, 30].
This approach 109 effectively increases the number of data tokens seen during training (by a factor of ⇠ 10 in [5]).
This 110 suggests that the performance of language models may be more dependant on the size of the trainingdata than previously thought.
Estimating the optimal parameter/training tokens allocation
We present three different approaches to answer the question driving our research: Given a ﬁxed 114 FLOPs budget, how should one trade-off model size and the number of training tokens?
In all threecases we start by training a range of models varying both model size and the number of training 116 tokens and use the resulting training curves to ﬁt an empirical estimator of how they should scale.
Weassume a power-law relationship between compute and model size as done in [10, 24], though future 118 work may want to include potential curvature in this relationship for large model sizes.
The resultingpredictions are similar for all three methods and suggest that parameter count and number of training 120 tokens should be increased equally with more compute —with proportions reported in Table 2.
Thisis in clear contrast to previous work on this topic and warrants further investigation.
3.1 Approach 1: Fix model sizes and vary number of training tokens
In our ﬁrst approach we vary the number of training steps for a ﬁxed family of models (ranging 124 from 70M to over 10B parameters), training each model for 4 different number of training sequences.
From these runs, we are able to directly extract an estimate of the minimum loss achieved for a given 126 number of training FLOPs.
Training details for this approach can be found in Appendix D.
For each parameter count N we train 4 different models: each uses a different horizon (measured 128 in number of training tokens) over which we decay the learning rate by a factor of⇥ ; the range 129 of horizons varies by a factor of 16 ⇥ for each parameter count.
We smooth and interpolate eachtraining loss curve.
From this, we obtain a continuous mapping from FLOP count to training loss for 131 each run.
We then determine which run achieves the lowest loss for each FLOP count.
Using theseinterpolants, we obtain a mapping from FLOP count C to the most efﬁcient choice of model size N optand number of training tokens D opt such that FLOPs ( N opt , D opt ) =
C .
We apply this mapping onto 134 logarithmically spaced values of C and obtain many empirical triplets ( C i , N opt,i , D opt,i ) i .
Finally,we ﬁt power laws to these empirical data, estimating a and b such that N opt / C a and D opt / C b .
136 We ﬁnd that a = 0 .
50 and b = 0 .
50 —as summarized in Table 2.
We perform a simple experiment forearly validating our analysis: given a budget of 10 21 FLOPs, we compare the performance of traininga model with a size recommended by our analysis to training a model with a size suggested by the 139 analysis of [24]—using the model size we predict has a clear advantage (Section D.4).
3.2 Approach 2: IsoFLOP proﬁles
In our second approach we vary the model size for a ﬁxed set of 9 different training FLOP counts 142 (ranging from⇥ 10 18 to 3 ⇥ 10 21 FLOPs), and consider the ﬁnal training loss for each point.
This 143 differs from the ﬁrst approach that considers points ( N i , D i , L i ) i along the entire training runs; thedata points are here scarcer but more representative of the performance of a fully trained model.
For each FLOP budget, we plot the ﬁnal loss (after smoothing) against the parameter count in Figure 3 146 (left).
In all cases, we ensure that we have trained a diverse enough set of model sizes to see a clearminimum in the loss.
We ﬁt a parabola to each IsoFLOPs curve to directly estimate at what model 148 size the minimum loss is achieved (Figure 3 (left)).
As with the previous approach, we then ﬁt apower law between FLOPs and loss-optimal model size and number of training tokens, shown in 150 Figure 3 (center, right).
Again, we ﬁt exponents of the form N opt / C a and D opt / C b and we ﬁnd 151 that a = 0 . 49 and b = 0 .
51 —as summarized in Table 2.
3.3 Approach 3: Fitting a parametric loss function
Lastly, we model all ﬁnal losses from experiments in Approach 1 & 2 as a parametric function of 154 model parameter count and the number of seen tokens.
Following a classical risk decomposition (see
Section D.2)
, we propose the following functional form
[Equation]
The ﬁrst term captures the loss for an ideal generative process on the data distribution, and should 157 correspond to the entropy of natural text.
The second term captures the fact that a perfectly trainedtransformer with N parameters underperforms the ideal generative process.
The ﬁnal term captures 159 the fact that the transformer is not trained to convergence, as we only make a ﬁnite number ofoptimisation steps, on a sample of the dataset distribution.
Model ﬁtting.
To estimate ( A, B, E, ↵ ,  ) , we minimize the Huber loss [20] between the predicted 162 and observed log loss using the L-BFGS algorithm [37]:
[Equation]
We account for possible local minima by selecting the best ﬁt from a grid of initialisations.
The 164 Huber loss (  = 10  3 ) is robust to outliers, which we ﬁnd important for good predictive performanceover held-out data points.
Section D.2 details the ﬁtting procedure and the loss decomposition.
Efﬁcient frontier.
We approximate the functions N opt and D opt by minimizing the parametricloss ˆ L under the constraint FLOPs ( N, D ) ⇡ 6 ND [24].
The resulting N opt and D opt balance 168 the two terms in Equation (3) that depend on model size and data.
By construction, they have a
Approach
Coeff.
a where N opt / C a
Coeff
[Equation]
b where D opt / C b
1. Minimum over training curves
[Table]
2. IsoFLOP proﬁles
[Table]
3. Parametric modelling of the loss
[Table]
Kaplan et al.
[Table] [Table]
power-law form.
We show contours of the ﬁtted function ˆ L in Figure 4 (left), and the closed-formefﬁcient computational frontier in blue.
From this approach, we ﬁnd that a = 0 .
46 and b = 0 .
54 —assummarized in Table 2.
3.4 Optimal model scaling
We ﬁnd that the three approaches, despite using different ﬁtting methodologies and different trained 174 models, yield comparable predictions for the optimal scaling in parameters and tokens with FLOPs(shown in Table 2).
All three approaches suggest that as compute budget increases, model size and 176 the amount of training data should be increased in approximately equal proportions.
The ﬁrst andsecond approaches yield very similar predictions for optimal model sizes, as shown in Figure 1 and 178 Figure A3.
The third approach predicts even smaller models being optimal at larger compute budgets.
We note that the observed points ( L, N, D ) for low training FLOPs ( C  1 e 21 ) have larger residualsk L
ˆ L ( N, D ) k 2 2 than points with higher computational budgets.
The ﬁtted model places increased 181 weight on the points with more FLOPs—automatically considering the low-computational budgetpoints as outliers due to the Huber loss.
As a consequence of the empirically observed negative 183 curvature in the frontier C !
N opt (see Appendix E), this results in predicting a lower N opt than the 184 two other approaches.
In Table A3 we show the estimated number of FLOPs and tokens that would ensure that a model of a 186 given size lies on the compute-optimal frontier.
Our ﬁndings suggests that the current generation oflarge language models are considerably over-sized, given their respective compute budgets, as shown in Figure 1.
Furthermore, the amount of training data that is projected to be needed is far beyond 189 what is currently used to train large models, and underscores the importance of dataset collectionin addition to engineering improvements that allow for model scale.
While there is signiﬁcant 191 uncertainty extrapolating out many orders of magnitude, our analysis clearly suggests that given thetraining compute budget for many current LLMs, smaller models should have been trained on more 193 tokens to achieve the most performant model.
In Appendix C, we reproduce the IsoFLOP analysis ontwo additional datasets: C4 [41] and GitHub code [39].
In both cases we reach the similar conclusion 195 that model size and number of training tokens should be scaled in equal proportions.
Chinchilla
Based on our analysis in Section 3, the optimal model size for the Gopher compute budget is 198 somewhere between 40 and 70 billion parameters.
We test this hypothesis by training a model on thelarger end of this range—70B parameters—for 1.4T tokens, due to both dataset and computational 200 efﬁciency considerations.
In this section we compare this model, which we call Chinchilla , to Gopherand other LLMs.
Both Chinchilla and Gopher have been trained for the same number of FLOPs 202 but differ in the size of the model and the number of training tokens.
While pre-training a largelanguage model has a considerable compute cost, downstream ﬁne-tuning and inference also make 204 up substantial compute usage [39].
Due to being⇥ smaller than Gopher , both the memory footprint 205 and inference cost of Chinchilla are also smaller.
4.1 Model and training details
The full set of hyperparameters used to train Chinchilla are given in Table 3.
Chinchilla uses the same 208 model architecture and training setup as Gopher with the exception of the differences listed below.
• We train Chinchilla on MassiveText (the same dataset as Gopher ) but use a slightly differentsubset distribution (Table A1) to account for the increased number of training tokens.
• We use AdamW [33] for Chinchilla rather than Adam [25] as this improves the languagemodelling loss and the downstream task performance after ﬁnetuning.
• We train Chinchilla with a slightly modiﬁed SentencePiece [26] tokenizer that does notapply NFKC normalisation.
The vocabulary is very similar– 94.15% of tokens are the sameas those used for training Gopher .
We ﬁnd that this particularly helps with the representationof mathematics and chemistry, for example.•
Whilst the forward and backward pass are computed in bfloat16 , we store a float32copy of the weights in the distributed optimiser state [43].
See Lessons Learned from [39]for additional details.
In Appendix G we show the impact of the various optimiser related changes between Chinchilla 221 and Gopher .
All models in this analysis have been trained on TPUv3/TPUv
[23] with JAX [6] and
Haiku [18].
We include a Chinchilla model card [36] in Table A
[Table]
[Table] [Table] [Table]
[Table]
We perform an extensive evaluation of Chinchilla , comparing against various large language models.
We evaluate on a large subset of the tasks presented in [39], shown in Table A6.
As the focus of thiswork is on optimal model scaling, we included a large representative subset, and introduce a few new 227 evaluations to allow for better comparison to other existing large models.
The evaluation details forall tasks are the same as described in [39].
Language modelling.
Chinchilla signiﬁcantly outperforms Gopher on all evaluation subsets of The 230 Pile [14], as shown in Figure A8.
Compared to Jurassic-1 (178B) [31], Chinchilla is more performanton all but two subsets– dm_mathematics and ubuntu_irc – see Table A7 for a raw bits-per-byte 232 comparison.
On Wikitext103 [35], Chinchilla reaches 7.16 perplexity compared to 7.75 for Gopher .
MMLU.
The Massive Multitask Language Understanding (MMLU) benchmark [17] consists of a 234 range of exam-like questions on academic subjects.
In Table A8, we report Chinchilla ’s average 5shot performance on MMLU (the full breakdown of results is shown in Table A9).
On this benchmark, 236 Chinchilla signiﬁcantly outperforms Gopher despite being much smaller, with an average accuracyof 67.6% (improving upon Gopher by 7.6%).
Remarkably, Chinchilla even outperforms the expert 238 forecast for June 2023 of 63.4% accuracy (see Table A8)
Furthermore, Chinchilla achievesgreater than 90% accuracy on 4 different individual tasks– high_school_gov_and_politics,international_law, sociology , and us_foreign_policy .
To our knowledge, no other model 241 has achieved greater than 90% accuracy on a subset.
In Figure A9, we show a comparison to Gopherbroken down by task.
Reading comprehension.
On the ﬁnal word prediction dataset LAMBADA [38], Chinchilla achieves 244 77.4% accuracy, compared to 74.5% accuracy from Gopher and 76.6% from MT-NLG 530B (see
Table 4).
On RACE-h and RACE-m [28], Chinchilla greatly outperforms Gopher , improving accuracy 246 by more than 10% in both cases—see Table 4.
BIG-bench.
We analysed Chinchilla on the same set of BIG-bench tasks [3] reported in [39].
Similar 248 to what we observed in MMLU, Chinchilla outperforms Gopher on the vast majority of tasks (see
Figure A10).
We ﬁnd that Chinchilla improves the average performance by 10.7%, reaching an [Table] of 65. [Table]
% versus 54.
[Table]
[Table] Full [Table] results can be found in Table
[Table]
Common sense.
We evaluate Chinchilla on various common sense benchmarks: PIQA [4], SIQA 252 [47], Winogrande [46], HellaSwag [59], and BoolQ [11].
We ﬁnd that Chinchilla outperforms both
Gopher and GPT-3 on all tasks and outperforms MT-NLG 530B on all but one task—see Table 5.
On TruthfulQA [32], Chinchilla reaches 43.6%, 58.5%, and 66.7% accuracy with 0-shot, 5-shot, 255 and 10-shot respectively.
In comparison, Gopher achieved only 29.5% 0-shot and 43.7% 10-shotaccuracy.
In stark contrast with the ﬁndings of [32], the large improvements (14.1% in 0-shot 257 accuracy) achieved by Chinchilla suggest that better modelling of the pre-training data alone can leadto substantial improvements on this benchmark.
Closed-book question answering.
Results on closed-book question answering benchmarks are 260 reported in Table A11.
On the Natural Questions dataset [27], Chinchilla achieves new closed-book
SOTA accuracies: 31.5% 5-shot and 35.5% 64-shot, compared to 21% and 28% respectively, for 262 Gopher .
On TriviaQA [22] we show results for both the ﬁltered (previously used in retrieval and openbook work) and unﬁltered set (previously used in large language model evaluations).
In both cases, 264 Chinchilla substantially out performs Gopher .
On the ﬁltered version, Chinchilla lags behind theopen book SOTA [21] by 7.9%.
On the unﬁltered set, Chinchilla outperforms GPT-3 (see Table A11).
Discussion & Conclusion
The trend so far in large language model training has been to increase the model size, often without 268 increasing the number of training tokens.
The largest dense transformer, MT-NLG 530B, is nowover 3 ⇥ larger than GPT-3’s 170 billion parameters from just two years ago.
However, this model, 270 as well as the majority of existing large models, have all been trained for a comparable numberof tokens—around 300 billion.
While the desire to train these mega-models has led to substantial 272 engineering innovation, we hypothesize that the race to train larger and larger models is resulting inmodels that are substantially underperforming compared to what could be achieved with the same 274 compute budget.
We propose three predictive approaches towards optimally setting model size and training duration, 276 based on the outcome of over 400 training runs.
All three approaches predict that Gopher issubstantially over-sized and estimate that for the same compute budget a smaller model trained on 278 more data will perform better.
We directly test this hypothesis by training Chinchilla , a 70B parametermodel, and show that it outperforms Gopher and even larger models on nearly every measured 280 evaluation task.
Whilst our method allows us to make predictions on how to scale large models when given additional 282 compute, there are several limitations.
Due to the cost of training large models, we only have twocomparable training runs at large scale ( Chinchilla and Gopher ), and we do not have additional 284 tests at intermediate scales.
Furthermore, we assume that the efﬁcient computational frontier canbe described by a power-law relationship between the compute budget, model size, and number of 286 training tokens.
However, we observe some concavity in log ( N opt ) at high compute budgets (see
Appendix E).
This suggests that we may still be overestimating the optimal size of large models.
Finally, the training runs for our analysis have all been trained on less than an epoch of data; futurework may consider the multiple epoch regime.
Despite these limitations, the comparison of Chinchilla 290 to Gopher validates our performance predictions, that have thus enabled training a better (and morelightweight) model at the same compute budget.
Though there has been signiﬁcant recent work allowing larger and larger models to be trained, our 293 analysis suggests an increased focus on dataset scaling is needed.
Speculatively, we expect thatscaling to larger and larger datasets is only beneﬁcial when the data is high-quality.
This calls for 295 responsibly collecting larger datasets with a high focus on dataset quality.
Larger datasets will requireextra care to ensure train-test set overlap is properly accounted for, both in the language modelling 297 loss but also with downstream tasks.
Finally, training for trillions of tokens introduces many ethicaland privacy concerns.
Large datasets scraped from the web will contain toxic language, biases, and 299 private information.
With even larger datasets being used, the quantity (if not the frequency) of suchinformation increases, which makes dataset introspection all the more important.
Chinchilla does 301 suffer from bias and toxicity but interestingly it seems less affected than Gopher (see Appendix I)..
Better understanding how performance of large language models and toxicity interact is an important 303 future research question.
While we have applied our methodology towards the training of auto-regressive language models, we 305 expect that there is a similar trade-off between model size and the amount of data in other modalities.
As training large models is very expensive, choosing the optimal model size and training steps 307 beforehand is essential.
The methods we propose are easy to reproduce in new settings.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on 534 how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing 536 the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.
Please do not modify the questions and only use the provided macros for your answers.
Note that the 540 Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes] The claims in the abstract describe the work clearly.
(b) Did you describe the limitations of your work?
[Yes] We address limitations of ourwork.(c)
Did you discuss any potential negative societal impacts of work
discussion model card
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A] Results arenot theoretical.
(b) Did you include complete proofs of all theoretical results?
[N/A] Results are nottheoretical/no proofs. [Table]
If you ran experiments...
[Table]
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
[Table]
perimental results (either in the supplemental material or as a URL)?
[No] The codeand the data are proprietary.
However, for the scaling methodology we provide clearinstructions on how to reproduce the results.
[Table]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] We provided all training details and hyperparameters.
[Table]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
[Table]
ments multiple times)?
[No] Due to the cost of training large models, we do not havemultiple runs.
However, the clear & predictable trends suggest the noise is very small.
[Table] (d) Did you include the total amount of compute and the type of resources used (e.g.,type of GPUs, internal cluster, or cloud provider)?
[Yes] All experiments are ran onTPUv3/
[-TPUv
-]TPUv4 and this is stated in the text.
[Table] If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a)
If your work uses existing assets, did you cite the creators?
[Yes] We cite the preexistingwork for all data/models that we use. [Table]
(b) Did you mention the license of the assets?
[Yes] We use the same data as in Rae et al. [Table] [39] which uses a proprietary dataset.
We also show results with an open source dataset–
[Table]
C4 Raffel et al.
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
[Table]
We do not introduce new assets.
[Table] (d) Did you discuss whether and how consent was obtained from people whose data you’re [Table]
using/curating?
[Yes]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁableinformation or offensive content?
[Yes] We include a model card which includes this [Table]
information.
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
There was no human subjects or crowd sourcing in this work.
[Table] (b) Did you describe any potential participant risks, with links to Institutional Review [Table]
Board (IRB) approvals, if applicable?
[N/A]
There was no human subjects or crowd
[Table]
sourcing in this work.
[Table] (c) Did you include the estimated hourly wage paid to participants and the total amount [Table]
spent on participant compensation?
[N/A]
There was no human subjects or crowd
[Table]
sourcing in this work.
[Table]