Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stability (2024)

Abstract: How do language models learn to make predictions during pre-training? To study this, we extract learning curves from five autoregressive English language model pre-training runs, for 1M unseen tokens in context. We observe that the language models generate short repetitive phrases before learning to generate longer and more coherent text. We also find that individual tokens often exhibit sudden increases or decreases in loss that are surprisingly consistent across pre-training runs. To better understand these fluctuations, we quantify the final surprisal, within-run variability, age of acquisition, forgettability, and cross-run variability of learning curves for individual tokens in context. More frequent tokens reach lower final surprisals, exhibit less variability within and across pre-training runs, are learned earlier, and are less likely to be “forgotten” during pre-training. Higher n-gram probabilities further accentuate these effects. Independent of the target token, shorter and more frequent contexts correlate with marginally more stable and quickly acquired predictions. Based on our results, we argue for the existence of sequential learning dependencies between different model capabilities, and we characterize language model learning as early n-gram learning before gradual refinement of tail n-gram predictions.

## introduction language models have received unprecedented attention in recent years due to impressive performance on natural language tasks (e.g., @xcite @xcite . however, these models are initialized as random word (token) generators, and it remains unclear how the models achieve complex linguistic abilities during pre-training. previous work has investigated when syntactic, semantic, and reasoning abilities emerge @xcite , quantified ages of acquisition for tokens averaged over contexts @xcite , and extracted learning curves for individual examples @xcite . however, features that influence individual learning curves have yet to be identified (e.g., n-gram probabilities and context lengths). given any token in context, it is largely unknown when or how stably that token would be learned. from a scientific perspective, understanding when examples are learned by language models can provide insights into possible mechanisms for language acquisition. regardless of their similarity to human language processing, language models are exemplars of how learning from language statistics alone (i.e., ''distributional'' learning) can lead to complex linguistic abilities @xcite @xcite . notably, despite smoothly decreasing corpus-level loss and independent and identically distributed (i.i.d.) data throughout pre-training, individual text examples exhibit learning curves with sudden decreases and increases in loss @xcite . this highlights the importance of examining individual example learning curves for pre-training dynamics research; aggregate curves often do not capture the fluctuations exhibited by individual examples. our work seeks to characterize these fine-grained convergence patterns in terms of simpler distributional statistics. from a practical perspective, understanding language model learning curves can inform the pre-training and deployment of language models. learning curve results might allow nlp practitioners to determine how much pre-training is necessary for different capabilities and what behaviors will remain stable after additional pre-training (e.g., ''continual learning'' on more recent data; @xcite . learning curve results can also help identify scenarios in which to expect high levels of variability among fully-trained models, or even develop better pre-training curricula. for example, better curricula might maximize the presence of tractable features that a language model can learn at different pre-training steps. thus, our work seeks to quantify convergence patterns for individual tokens in context during language model pre-training. we focus on learning curve convergence, including learning speed, forgetting, and stability. rather than evaluate model performance on downstream tasks throughout pre-training, we study individual tokens in context (c.f. @xcite . specifically, we run five english language model pre-training runs, and we extract learning curves for 1m unseen tokens in context. we quantify the final surprisal, variability within and across pre-training runs, age of acquisition, and forgettability of each example. we report general learning curve patterns, and we assess the impact of token frequencies, n-gram probabilities, context lengths and likelihoods, and part-of-speech tags on the speed and stability of language model learning. based on our results, we argue that there exist sequential dependencies between when language models acquire different capabilities ( §7). we then characterize language model learning as early n-gram learning, before gradual refinement of low probability n-gram predictions based on longer context and more nuanced linguistic capabilities. finally, we discuss implications of our work for informed language model deployment. ## related work previous work has studied the pre-training dynamics of language models @xcite . @xcite and @xcite find that language models learn linguistic generalizations in similar stages regardless of model architecture, initialization, and data-shuffling. in masked language models, syntactic rules are learned early, but world knowledge and reasoning are learned later and less stably @xcite . @xcite find that copy mechanisms (''induction heads'' for in-context learning) appear at an inflection point during pre-training. these results establish when a variety of abilities emerge in language models. our work studies more finegrained learning trajectories by evaluating individual tokens in context. indeed, previous work has studied how individual tokens are learned during pre-training. for example, word learning is highly dependent on word frequency @xcite . larger models memorize more examples during pre-training without overfitting @xcite , but the time step that a model sees an example does not affect memorization @xcite . most similar to our work, @xcite collect learning curves for individual tokens in context, finding that some examples exhibit a ''double-descent'' trend where they first increase then decrease in surprisal. all of the studies above collect language model learning curves during pre-training, either for individual examples or targeted benchmark performance. here, we introduce metrics to characterize such curves, we identify general learning patterns, and we isolate text features that are predictive of learning speed and stability. ## language model learning curves we extract learning curves for 1m unseen tokens in context from five english language model pre-training runs. similar learning curves are computed in @xcite ; we extend their work by defining metrics to characterize such learning curves ( §5), and we identify text features that predict each metric ( §6). in this way, we aim to demonstrate the connection between simple distributional statistics (e.g., n-gram probabilities) and language model learning. foot_0 ## overall learning patterns before considering fine-grained learning patterns for individual surprisal curves, we observe several overall trends during language model pre-training. many of these trends echo results from previous work (e.g., n-gram learning in @xcite or intuitive results known by language model pre-training practitioners (e.g., the slow development of the ability to generate long coherent text), but these trends establish basic intuitions about how language models progress throughout pre-training. early in pre-training, models generate short repetitive phrases. sample outputs from different model checkpoints are shown in step training model output tokens 0 0 ''this is469 gush liqueur defense trophies jakarta sale berlin deservingexception validate jalapeno...'' 100 3.3m ''this is, ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, the the the the,,,,,,,......'' 1k 33m ''this is a few of the first of the same of the world's the most of the first of the the same of the first of the world.'' 10k 330m ''this is a great way to make a difference in your life.'' 100k 3.3b ''this is a very important part of the process of getting your business off the ground.'' 1m 33b ''this is a great opportunity to own a beautiful home in the desirable area of north vancouver.'' models later generate longer and more coherent text. by step 10k, the models generally produce coherent sentence completions, but they still contain repetitive phrases (10.8% of completions with a three-word phrase repeated at least three times). by step 100k, the repetition rate drops to 6.0%, and completions appear more specific to the context. by step 1m, the repetition rate is 4.7%, and the models can produce coherent multi-sentence completions. still, due to our relatively small model size (124m parameters, the size of the original gpt model; @xcite , we do not expect our models to exhibit text generation capabilities at the level of larger language models. ## characterizing learning curves we then consider fine-grained analyses of learning curves for individual tokens in context. we introduce five metrics to characterize language model learning curves, each motivated by previous work. ## discussion in the previous sections, we report general patterns during language model pre-training ( §4), define ways to characterize learning curves ( §5), and isolate specific features that predict the speed and stability of learning for individual tokens in context ( §6). our results contribute to ongoing work studying language model pre-training dynamics, with implications for robust model deployment. sequential learning. previous work has demonstrated that language models exhibit finegrained learning patterns that are not captured by the corpus-level loss curve (related work in §2). in particular, sudden increases and decreases in example loss @xcite may be somewhat surprising given that the pre-training text is i.i.d. for all pre-training steps. by demonstrating that many of these sudden changes are consistent regardless of random initialization and data shuffling ( §5.2), our work indicates that some instances of sudden learning and ''forgetting'' are not due to random chance or the specific examples observed in a given step. 9 rather, they reflect some change in model processing that consistently occurs partially into pre-training (roughly step @xmath0. because such a sudden change cannot be attributed to the specific examples observed (robust to random shuffling) or any change in the pre-training distribution at time t (the data is always i.i.d.), the primary remaining explanation is that the models' sudden ''learning'' at step @xmath1. at step 0. framed from a potentially more interesting perspective, some types of language model ''learning'' appear to be dependent on previous learning and the linguistic abilities already present in the model. this aligns with previous work showing that language models acquire linguistic abilities in a systematic order during pre-training @xcite , although not necessarily due to sequential dependencies. for example, @xcite show that despite similar acquisition orders across models, different syntactic abilities are learned in parallel; performance for most individual abilities increases from the onset of pre-training. our work provides evidence that there exist other capabilities or types of generalizations (e.g., non-syntactic abilities or even more fine-grained syntactic sub-abilities) that can only be learned after others, or at least only once the model reaches some particular state. isolating these sequential dependencies is an exciting direction for future work. n-gram learning and refinement. as a further step towards understanding fine-grained learning patterns in language models, our work investigates whether simple statistical regularities can explain learning patterns such as the sudden loss changes discussed above. we demonstrate that learning curves are more stable and converge faster for frequent tokens, n-gram probable tokens, and frequent contexts ( §6.2). high probability n-grams in particular are less likely to be ''forgotten'', suggesting that evolving model generalizations throughout pre-training have larger effects on low-probability n-grams. combined with findings that language models roughly follow n-gram learning early in pre-training and only later produce longform coherent text ( §4; chang and bergen 2022b), language model learning might be characterized as early n-gram learning, then gradual refinement of the tail n-gram probabilities based on longer contexts and more nuanced linguistic capabilities (e.g., world knowledge and reasoning; @xcite . ## conclusion in this work, we identify learning patterns during language model pre-training, including concrete features that predict when and how stably individual examples are acquired. we assess the impact of n-gram probabilities, context lengths and likelihoods, and part-of-speech tags on the speed and stability of language model learning. we propose a high-level characterization of language model learning based on simple distributional statistics, and we discuss implications for deploying robust language models in practice.