Word Acquisition in Neural Language Models (2022)

Abstract: We investigate how neural language models acquire individual words during training, extracting learning curves and ages of acquisition for over 600 words on the MacArthur-Bates Communicative Development Inventory (Fenson et al., 2007). Drawing on studies of word acquisition in children, we evaluate multiple predictors for words’ ages of acquisition in LSTMs, BERT, and GPT-2. We find that the effects of concreteness, word length, and lexical class are pointedly different in children and language models, reinforcing the importance of interaction and sensorimotor experience in child language acquisition. Language models rely far more on word frequency than children, but, like children, they exhibit slower learning of words in longer utterances. Interestingly, models follow consistent patterns during training for both unidirectional and bidirectional models, and for both LSTM and Transformer architectures. Models predict based on unigram token frequencies early in training, before transitioning loosely to bigram probabilities, eventually converging on more nuanced predictions. These results shed light on the role of distributional learning mechanisms in children, while also providing insights for more human-like language acquisition in language models.

## introduction language modeling, predicting words from context, has grown increasingly popular as a pretraining task in nlp in recent years; neural language models such as bert @xcite , elmo @xcite , and gpt @xcite have produced state-of-the-art performance on a wide range of nlp tasks. there is now a substantial amount of work assessing the linguistic information encoded by language models @xcite ; in particular, behavioral approaches from psycholinguistics and cognitive science have been applied to study language model predictions @xcite . from a cognitive perspective, language models are of theoretical interest as distributional models of language, agents that learn exclusively from statistics over language @xcite . however, previous psycholinguistic studies of language models have nearly always focused on fully-trained models, precluding comparisons to the wealth of literature on human language acquisition. there are limited exceptions. @xcite famously studied past tense verb form learning in phoneme-level neural networks during training, a study which was replicated in more modern character-level recurrent neural networks @xcite . however, these studies focused only on sub-word features. there remains a lack of research on language acquisition in contemporary language models, which encode higher level features such as syntax and semantics. as an initial step towards bridging the gap between language acquisition and language modeling, we present an empirical study of word acquisition during training in contemporary language models, including lstms, gpt-2, and bert. we consider how variables such as word frequency, concreteness, and lexical class contribute to words' ages of acquisition in language models. each of our selected variables has effects on words' ages of acquisition in children; our language model results allow us to identify the extent to which each effect in children can or cannot be attributed in principle to distributional learning mechanisms. finally, to better understand how computational models acquire language, we identify consistent patterns in language model training across architectures. our results suggest that language models may acquire traditional distributional statistics such as unigram and bigram probabilities in a systematic way. understanding how language models acquire language can lead to better architectures and task designs for future models, while also providing insights into distributional learning mechanisms in people. our work draws on methodologies from word acquisition studies in children and psycholinguistic evaluations of language models. in this section, we briefly outline both lines of research. ## method we trained unidirectional and bidirectional language models with lstm and transformer architectures. we quantified each language model's age of acquisition for each word in the cdi @xcite . similar to word acquisition studies in children, we identified predictors for words' ages of acquisition in language models. foot_0 ## results significant predictors of age of acquisition are shown in ## discussion we found that language models are highly sensitive to basic statistics such as frequency and bigram probabilities during training. their acquisition of words is also sensitive to features such as sentence length and (for unidirectional models) lexical class. importantly, the language models exhibited notable differences with children in the effects of lexical class, word lengths, and concreteness, highlighting the importance of social, cognitive, and sensorimotor experience in child language development. ## conclusion in this work, we identified factors that predict words' ages of acquisition in contemporary language models. we found contrasting effects of lexical class, word length, and concreteness in children and language models, and we observed much larger effects of frequency in the models than in children. furthermore, we identified ways that language models aquire unigram and bigram statistics early in training. this work paves the way for future research integrating language acquisition and natural language understanding. figure 6 : evaluation loss during training for all four language models. note that perplexity is equal to exp(loss). finally, we evaluated each of our models for word acquisition at 208 checkpoint steps during training, sampling more heavily from earlier steps. we evaluated checkpoints at the following steps: