Training Trajectories of Language Models Across Scales (2023)

Abstract: Scaling up language models has led to unprecedented performance gains, but little is understood about how the training dynamics change as models get larger. How do language models of different sizes learn during pre-training? Why do larger language models demonstrate more desirable behaviors? In this paper, we analyze the intermediate training checkpoints of differently sized OPT models (Zhang et al., 2022)—from 125M to 175B parameters—on next-token prediction, sequence-level generation and downstream tasks. We find that 1) at a given perplexity and independent of model sizes, a similar subset of training tokens see the most significant reduction in loss, with the rest stagnating or showing double-descent behavior (Nakkiran et al., 2020); 2) early in training, all models learn to reduce the perplexity of grammatical sequences that contain hallucinations, with small models halting at this suboptimal distribution and larger ones eventually learning to assign these sequences lower probabilities; and 3) perplexity is a strong predictor of in-context learning performance on 74 multiple-choice tasks from BIG-Bench, and this holds independent of the model size. Together, these results show that perplexity is more predictive of model behaviors than model size or training computation.

## introduction scaling up language models has been shown to improve language modeling perplexity @xcite as well as zero-or few-shot end task accuracies @xcite @xcite . however, relatively little is understood about why or how this happens. how do the training dynamics differ as models get larger? what do language models of different sizes learn 1 code is publicly available at https://github.com/ xiamengzhou/training_trajectory_analysis . we attempt to make progress to answer these questions by studying the training trajectories of differently-sized opt models @xcite through analyzing their intermediate checkpoints. in contrast to prior work, which studies the trajectories of small models with up to 300m parameters @xcite @xcite or focuses on the language modeling objective alone @xcite @xcite , we are the first to comprehensively study the training trajectories of large-scale autoregressive language models with up to 175b parameters across a wide range of settings. repeatedly across training and different model scales, we analyze three aspects of model performance: (i) next-token prediction on subsets of tokens (ii) sequence-level generation and (iii) downstream task performance. we use perplexity, which is closely tied to language model evaluation, as the major metric throughout the study. for next-token prediction ( §3), we study the trajectory by categorizing each token's prediction as stagnated, upward or downward according to its perplexity trend as training progresses. we find each category comprising a significant number of tokens: while a significant number of tokens' perplexity stagnate, a subset of tokens with an increasing perplexity in smaller models exhibit a doubledescent trend @xcite where perplexity increases and then decreases in larger models. these behaviors primarily emerge at a similar validation perplexity across model scales. for sequence-level generation ( §4), we study the distribution shift at a document level (50-500 tokens) by decoding sequences that small/large models favor more than the other. human texts present expected scaling patterns in that they are best modeled by larger (or longer trained) models. however, to our surprise, large models are better at modeling 10 18 10 19 10 20 10 21 10 22 10 23 flops 10 20 30 40 50 figure 1 : validation perplexity (ppl) of opt models against training flops. our work suggests that models with comparable perplexity levels during training exhibit similar predictions, regardless of their scales. our work presents a comprehensive study of training trajectories of language models trained with similar procedures, e.g., opt. we conclude that language models learn the same phenomena in the same order across different model sizes. the overall model perplexity is a composite measure of which language phenomena have been learned. ## experimental settings models. unless otherwise indicated, all of our experiments use opt @xcite , a collection of open-source autoregressive language models. opt models serve as a good fit for this study due to their controlled pre-training procedures across all model sizes. in particular, all the models share the same tokenization and are trained on the same training data, covering a total of 300b tokens (180b unique). note that different-sized models differ in batch sizes and total number of steps. 3 we collect intermediate checkpoints from the authors and perform evaluations of these checkpoints across six different sizes: 125m, 1.3b, 6.7b, 13b, 30b, and 175b. validation perplexity. throughout this paper, we use validation perplexity (valid ppl) to refer to the autoregressive language modeling perplexity measured on the entire validation set. we use the original opt validation set, a held-out subset of the training corpus that covers a wide range of domains, such as books, news, and subtitles. we plot the trajectory of validation perplexity in figure 1 , which follows a similar power-law pattern observed in previous scaling work @xcite . methodology. we aim to understand how models of different sizes behave throughout training as a function of computing (flops) 4 and validation perplexity. throughout the paper, we use different measurements to characterize model behavior and plot them against these two metrics. ## next-token prediction autoregressive language models are trained to predict the next token given a context. figure 1 shows that validation perplexity, aggregated over all positions, gradually declines as training progresses. however, it is not clear if all token instances evolve similarly to the aggregated measurement. in this section, we study the trajectory of next-token predictions, dividing them into three categories-stagnated, upward trend, or downward trend-to understand how language models gradually learn new language phenomena. ## conclusion to summarize, our study demonstrates that validation perplexity is a reliable indicator of the behavior of opt models, regardless of their sizes. larger models, with increased computational power and capacity, exhibit behavior similar to that of smaller models while also unlocking new phenomena and capabilities as validation perplexity decreases further. however, there are certain exceptional cases where models behave differently, sometimes even in opposite directions, such as in the perplexity of texts generated by contrasting two models. this suggests that the underlying model distributions are not entirely identical at the same perplexity level. the availability of a larger number of opensourced model checkpoints, such as those provided by @xcite , offers opportunities for interpreting language model behaviors through the analysis of training trajectories. the techniques we propose can be extended to analyze language models trained using different resources and methodologies. additionally, we leave open questions for future research, such as further exploring the phenomenon of double-descent more in-depth.