The Grammar-Learning Trajectories of Neural Language Models (2022)

Abstract: The learning trajectories of linguistic phenomena in humans provide insight into linguistic representation, beyond what can be gleaned from inspecting the behavior of an adult speaker. To apply a similar approach to analyze neural language models (NLM), it is first necessary to establish that different models are similar enough in the generalizations they make. In this paper, we show that NLMs with different initialization, architecture, and training data acquire linguistic phenomena in a similar order, despite their different end performance. These findings suggest that there is some mutual inductive bias that underlies these models’ learning of linguistic phenomena. Taking inspiration from psycholinguistics, we argue that studying this inductive bias is an opportunity to study the linguistic representation implicit in NLMs.Leveraging these findings, we compare the relative performance on different phenomena at varying learning stages with simpler reference models. Results suggest that NLMs exhibit consistent “developmental” stages. Moreover, we find the learning trajectory to be approximately one-dimensional: given an NLM with a certain overall performance, it is possible to predict what linguistic generalizations it has already acquired. Initial analysis of these stages presents phenomena clusters (notably morphological ones), whose performance progresses in unison, suggesting a potential link between the generalizations behind them.

## introduction children present remarkable consistency in their patterns of language acquisition. they often acquire linguistic phenomena in a similar order @xcite , and make similar generalizations and over-generalizations @xcite . this consistency provides an important starting point for linguistic study. for example, arguments in favor of single or dual system accounts of morphological representation are often backed by computational models of children learning trajectories (e.g., @xcite @xcite . in this paper, we embrace this program for the study of computational language models, investigating learning trajectories. 1 the representations that language models (lm) acquire have been studied extensively, including studying their learning dynamics to improve training (see §6). however, very little work aimed at drawing connections between the training dynamics and the learned representations. in this work we adopt a behavioral approach, thus revealing that nlms share learning trajectories and generalize in similar ways during training. this implies that studying trajectories of nlms is worthwhile, in the sense that results on one architecture or size are expected to be reproducible by others. these findings call for a characterization of these trajectories, a new and promising territory for research. we take first steps to explore these directions, emphasizing their potential benefit to a better future understanding of what models learn. specifically, we train nlms on next-word prediction, but evaluate and compare them by tracking their performance on grammar learning in english, using the blimp dataset (see 2.1). blimp is a dataset that consists of 67k minimal pairs, where each pair includes a grammatically correct and a grammatically erroneous sentence. nlms are tested for their ability to assign higher probability to the correct one. see example in we begin ( §3) by establishing that nlms learn grammatical phenomena in a consistent order. we evaluate nlms at different time points along their training, showing that the performance on linguis-challenge correct erroneous animate subject galileo had talked to bell. this car had talked to bell. ## the learning order of nlms in this section, we examine various aspects of nlms, generally showing that their learning trajectories are similar. we evaluate network similarity by adopting a behavioral approach. accordingly, networks are viewed as functions, whose latent features manifest themselves only by their influence on the network's behavior. latent features are the unobserved causes of the measured behavior. consequently, parameters, activation patterns and representations can be completely different among similar models. this is unlike the approaches employed by @xcite ; @xcite ; @xcite , which analyze internal representations directly. to formalize the above notion, let l t denote a checkpoint, the language model l at time t. let pv(l t ) denote its performance vector -the accuracy obtained by l on each blimp challenge p: pv(l t ) = [acc(l t , p)] @xmath0 ## phases of learning having established that different nlms learn in a consistent order, we investigate the emerging learning trajectory by comparing it with simpler reference models. our goal is to identify distinct learning phases that characterize nlm's training. setup. we compare gpt2 tiny to fully trained lms (same as §3.3), as well as to a variety of metrics. for each metric m we compute the average score over each example for each of the 67 sets e p @xmath1. the results are replicated with gpt2 small and transformerxl and lead to similar conclusions (see app. §e). sentence-level metrics. first, we consider two sentence-level metrics: sentence length (in tokens) and syntactic depth. assuming a sentence parse tree, the depth is the longest path from a word to the root. sentence length is often considered to be a source of challenge for infants @xcite and networks @xcite , regardless of the sentence's complexity. syntactic depth (yngve, 1960) is a measure used to assess how cognitively complex a sentence is. we leave the question of which measure of linguistic complexity @xcite correlates best with the trajectory exhibited by nlms to future work. our results (fig. 5 ) show that neither sentencelevel metric (length and syntactic depth) can predict well what is difficult for the model. this is not surprising, as both measures only capture sentence complexity at a general level, and are not directly related to the linguistic phenomenon that is being tested. we do see that the syntactic depth starts off as a worse predictor of the nlm performance and ends as a better one. we provide a different perspective on this initial learning phase, before and after that switch, later in this section. next, we compare the performance vector with task difficulty for humans, as reported in the original blimp paper. we observe that correlation is fairly high after a sufficient number of steps. in fact, the network becomes more similar to humans as it improves: at the beginning, the network relies on different features than humans, but with time more of the hurdles are shared. however, correlation saturates at a mid-range correlation of under 0.5. this suggests that the network (partially) relies on features that are not used by human annotators. these may be valid generalizations not tested by blimp, or erroneous ones that are still beneficial to reduce the score on the task it was trained on (cf. @xcite . we revisit this issue in §5. comparison with limited context and locality. our methodology opens the door to examine other potential biases of lms. we now do so, starting with context and locality. we consider models that take into account different scopes of context: unigram, and 2-5 gram lms that can exploit the order of preceding words. we argue that the correlation between nlms and n-gram lms may indicate that features based on limited context are also employed by nlms. surprisingly, the unigram model, which doesn't use context, perfectly classifies 7 phenomena, achieves 98.1% accuracy on 1, and completely fails (0% accuracy) on 8. this suggests that high accuracy on some syntactic and semantic challenges (as defined by blimp) can be achieved by simple heuristics. note, however, that the nlms we test are not trained towards any specific phenomena and are not fine-tuned in any way. hence, nlms can only attain heuristics or biases (generalization errors) which are beneficial in general, not ones specific to our test challenges. while nlms initially present a strong correlation with the unigram model, this correlation quickly drops (see fig. 5 ). from the outset, gpt2 tiny succeeds on 6 of the 8 phenomena that are classified well by unigrams, and 4 of the 8 that the unigram model utterly fails on. interestingly, for 3 of the other phenomena on which the unigram failed, gpt2 tiny initially achieves 0% accuracy (chance level is 50%), but its accuracy does climb during training (e.g., see app. §a). we conclude that, as expected, the nlm acquires a bias towards predicting frequent words early in training, but that this bias is weighed in against other (contextual) considerations later on in training. comparing different scopes of context, our results (fig. 6 and app. §e) show that throughout training, the network presents high correlation with n-gram models. from a certain point onward, the network becomes more similar to the bi-gram model than to the other n-gram lms. we also note that similarity peaks early on, but with time the correlation decreases. this may suggest that initially, the nlms acquire grammatical behavior that resembles a markov model, or even a bi-gram model. only later does the network rely more on global features. this is in line with our earlier findings, which show an increasing correlation with syntactic depth as compared to sentence length. at the very beginning, nlms often generate one word repetitions (e.g., "the" @xcite . this seems to be at odds with our finding that grammar learning already begins at this early stage. however, while frequency may dictate the most probable predictions, comparing two options that differ only slightly may prove to depend more on context, as our results indicate. limited context and word order. by comparing nlms to n-grams, we examined the effect of context within a fixed window size. now we ex-amine the effect of word order, within a window and in general. to this end, we create two ablated gpt2 tiny models. bow is agnostic to the order between preceding tokens, while window-5 is similar but relies only on 5 tokens (details in §2). our results suggest that initially, the identity of the preceding words is more important than their order. both bow and window-5 better correlate with our nlm than the n-gram models. later on, this trend reverses and the n-grams, that do exploit word order, become better correlated. furthermore, the correlation with window-5 is significantly smaller than with bow at later stages of learning, suggesting that the network gradually learns to rely on more context (cf. @xcite . ## classifying the learning trajectories to understand the latent features learned by nlms, we categorize linguistic phenomena through the lens of their learning trajectories. we ask whether linguistically similar phenomena are learned in a similar fashion, and whether what is learned similarly is defined by linguistic terms. we inspect linguistic categories by comparing the learning trajectories of their phenomena. in the morphology field, we find that they display similar gradual curves, ultimately reaching high performance (median accuracy 0.85, see fig. 7a ). this may indicate that some latent features learned are morphological, and affect performance on almost all 'morphology' phenomena. syntax-semantics phenomena also present unique behavior: their scores plateau near chance performance (see fig 7b ), suggesting that the learned features are insufficient to correctly represent phenomena in this field. the other fields, "semantics" and "syntax" (figs 7c , 7d ), do not present prototypical learning curves, suggesting that they are too broad to correspond to a single learning pattern. this, in turn, may suggest that they do not all correspond to a well-defined set of latent features. next, we follow the reverse direction and cluster the learning curves of gpt2 tiny . we use spectral clustering with 10 clusters and sklearn default parameters, by projecting the learning curves into a normalized laplacian and applying k-means. intuitively, learning curves with similar values along the principal directions, are clustered together. other clustering methods show similar results. the clusters (fig. 8 and app. §h) reflect several (a) (b) (c) (d) figure 8: some phenomena are learned, others (c) deteriorate, implying the network (that learns language modelling, not phenomena) learns orthogonal features. learning curves of gpt2 tiny on blimp challenges, obtained by spectral clustering and colored by fields. we manually inspect a few phenomena with strong initial performance that then deteriorates. we find that some of these challenges are solvable by a simple rule, easily learnable by an n-gram model. for example, in "principle a case 1", always preferring subjective pronouns (e.g., "she" or "he") over reflexive ones (e.g., "himself", "herself") is sufficient to obtain a perfect score, and preferring "not ever" over "probably/fortunately ever" solves "sentential negation npi licensor present". the fact that nlm performance deteriorates, fits our finding that nascent nlms resemble an n-gram model. ## related work characterizing what networks learn is a longstanding challenge. recently, studies suggested methods to analyze trained models such as probing @xcite , analyzing attention heads @xcite and neurons (finding also correlations across epochs; @xcite and assessing the extent to which lms represent syntax @xcite . other works compare outputs, like us, to assess network generalizations @xcite , look for systematic biases @xcite or evaluate characteristics of outputs @xcite . @xcite fine-tuned bert and tested generalizations on the adversarial dataset hans @xcite , finding models to make inconsistent generalizations. their results differ from ours, but so is their setup, which in-volves fine-tuning for inference. characterizing the features learned by networks according to the order in which examples and phenomena are learned is a relatively new topic. recently, @xcite ; @xcite ; @xcite showed that classifiers learn to label examples in the same order. while their focus was on computer vision, it provided motivation for this work. other studies use learning dynamics as a tool, rather than a topic of study. they choose training examples @xcite , categorize examples @xcite or characterize the loss-space @xcite . little research on nlm learning dynamics and generalization types was previously conducted. perhaps the closest to this work is @xcite , which compared lstm representations with 3 types of linguistic tagger outputs, finding that correlation is low and that later in training, more context is used. the latter is reminiscent of our findings in §4. in parallel work, @xcite probe models during training. they show that, early in training, information required for linguistic classifications is found somewhere in the layers of the model. our work supports their findings by showing that grammar learning experiments conducted with one model are likely to replicate on another. our methodology differs from theirs in requiring the information the model learnt to manifest itself in behavior rather than to be extractable with a dedicated classifier. studying the trajectories of language learning is a mostly untapped area in nlp, but is a longestablished field of research in linguistics and psychology. such lines of research study topics such as acquisition of phonemes @xcite , morphology @xcite , complex constructions @xcite and innate learning abilities @xcite . considerable computational work was also done on constructing models that present similar learning trajectories to those of infants @xcite @xcite . our work suggests that the generalizations nlms make are coupled with the bottom-line performance. this gives a new angle and opens avenues of research when combined with previous work about bottom-line performance. for exam-ple, the bottom-line performance of small models could predict the performance of larger models @xcite . in such cases, the type of generalizations made might also be predicted from the smaller models. our work is also closely related to fields such as curriculum learning @xcite , self-paced learning @xcite , hard data mining @xcite , and active learning @xcite @xcite . in these fields, the order in which data should be presented to the learner is investigated. on the other hand, in our work, we study the order of the data in which the learner is learning -which may shed some light on the advancement of such fields. ## summary and conclusions we showed that nlms learn english grammatical phenomena in a consistent order, and subsequently investigated the emerging trajectory. our findings suggest that nlms present consistent and informative trends. this finding suggests a path for studying nlms' acquired behavior through their learning dynamics, as a useful complementary perspective to the study of final representations. future work will consider the impact of additional factors, architectures and learning phases that appear only later in training. we hope that this work will increase the affinity between the knowledge and methodologies employed in developmental studies, and those used for studying nlms. our goal is to obtain a better understanding of what makes linguistic generalization complex or simple to learn, for both humans and nlms.