A Distributional Perspective on Word Learning in Neural Language Models (2025)

Abstract: Language models (LMs) are increasingly being studied as models of human language learners.Due to the nascency of the field, it is not well-established whether LMs exhibit similar learning dynamics to humans, and there are few direct comparisons between learning trajectories in humans and models.Word learning trajectories for children are relatively well-documented, and recent work has tried to extend these investigations to language models.However, there are no widely agreed-upon metrics for word learning in language models.We take a distributional approach to this problem, defining lexical knowledge in terms of properties of the learned distribution for a target word.We argue that distributional signatures studied in prior work fail to capture key distributional information. Thus, we propose an array of signatures that improve on earlier approaches by capturing knowledge of both where the target word can and cannot occur as well as gradient preferences about the word’s appropriateness.We obtain learning trajectories for a selection of small language models we train from scratch, study the relationship between different distributional signatures, compare how well they align with human word learning trajectories and interpretable lexical features, and address basic methodological questions about estimating these distributional signatures.Our metrics largely capture complementary information, suggesting that it is important not to rely on a single metric.However, across all metrics, language models’ learning trajectories fail to correlate with those of children.

## introduction there is a long tradition of characterizing words in terms of their statistical properties @xcite . the distributional hypothesis @xcite , which characterizes knowledge of a word in terms of "the company it keeps" @xcite , has proven surprisingly prescient. indeed, such a characterization is the idea behind static word representations @xcite @xcite @xcite estimated from data, as well as modern (large) language models @xcite . while such a distributional approach to training language models (lms) is now well-established, only recently has distributional information been explored as a tool for evaluating lexical knowledge in lms. over the last few years, there has been a growing interest in studying word learning in language models @xcite @xcite @xcite @xcite . most of these studies are part of a larger research program to use lms to inform the study of human language acquisition by serving as convenient, controllable, and effective models of human development @xcite @xcite . from this perspective, it is desirable to have lms with human-like learning trajectories, as they can better serve as generalizable models of human learners. word learning has a potentially important role in the success of this research program because it is one of the best proving grounds for comparing the learning trajectories of humans and lms head to head. while some studies (e.g., @xcite have tracked syntax learning in lms using benchmarks like blimp @xcite , corresponding data for children is more limited in scope @xcite . there is also child data on phonological learning @xcite which can be explored further as audio-based lms improve. fortuitously, word learning trajectories in text-based lms can be easily compared against a wealth of child data in multiple languages thanks to the massive efforts of caregivers and scholars who report and curate child word learning data in the wordbank database @xcite . unfortunately, the caregiver reporting approach @xcite used in wordbank is not immediately applicable to lms, and there is no consensus on how to benchmark word learning in lms. @xcite explored word learning through different methods, including comparing lms' word similarity scores to humans' @xcite @xcite , classifying lexical entailment relations @xcite , predicting semantic features @xcite and using minimal pairs to measure lm preferences for appropriate word usage @xcite . other articles rely on visual stimuli to ground evaluations for multimodal models @xcite @xcite . notably, @xcite and @xcite take a distributional approach, characterizing lexical knowledge in terms of the lm's surprisal, an information-theoretic quantity that has been widely studied in psycholinguistics @xcite . in this study, we take inspiration from @xcite approach to tracking the model's distributional knowledge about a particular word throughout training. we formalize their approach and improve on it in several respects. while chang and bergen only consider the surprisal under an lm of a word in a context where the target word is appropriate and (implicitly) rely on a trivial approximation of the ground truth distribution in evaluating the quality of lexical knowledge, in contrast, we propose a family of distributional signatures that allow for the consideration of the lm's learned distribution in both appropriate and inappropriate contexts. we also introduce distributional signatures that are truly intrinsic to the model itself, as well as strongly reference signatures that compare the learned distribution to a non-trivial ground truth, which we approximate using a large pretrained lm. in our experiments, we train language models from scratch on three datasets resembling the input to children to varying degrees. we record the distributional signatures for a set of common words throughout training, and following chang and bergen, we apply a threshold to the measured learning trajectories to obtain an age-of-acquisition (aoa) for each word. we then conduct analyses to answer the following questions: we find that the learning trajectories for different distributional signatures are indeed different from each other, suggesting that earlier approaches failed to capture some aspects of word learning. while many signatures, like @xcite , give trajectories that are highly correlated with simple features like lexical frequency, other signatures are harder to predict and therefore may capture more nontrivial information. however, we find that learning trajectories for some distributional signatures fail to converge, making aoas difficult to infer. finally, no signature yields aoa scores that are strongly correlated with children's aoa, supporting the conclusion that with current methods, lms' learning patterns are poorly aligned with humans' and underscoring a limitation of current lms as models of human development. we, therefore, call for future work to evaluate and improve the human-likeness of lms' learning trajectories using the distributional signatures we propose. ## preliminaries let σ be an alphabet, a finite, non-empty set of characters, e.g., unicode symbols. 1 a string is a finite sequence of characters drawn from an alphabet σ. the set σ * , the kleene closure of σ, is the set of all strings with characters drawn from σ-including the empty string ε. we consider two distinguished types of strings. first, we define a word foot_1 as a character string @xmath0, which is believed to operate as a lexical item. second, we refer to an arbitrary character string that precedes a word as a context. we denote a context as @xmath1. a language model p is a probability distribution over σ * . a language model's prefix probability is defined as the following sum throughout the paper, we will primarily be interested in a specific ratio of p's prefix probabilities, which we use to define the probability of a word in a context foot_2 as follows we are also interested in the surprisal of a word in a context, denotedlog -→ p (w | c). in the remainder of this paper, we will distinguish three lms: p, the underlying distribution assumed to have generated the observed strings; q, a parameterized model whose parameters we estimate; and r, a pre-trained reference lm, potentially larger and trained on more data. a standard method of constructing a language model p that approximates q is maximum-likelihood estimation. suppose we observe a bag of n samples y (n) n @xmath2, then we choose a model q that minimizes the following crossentropy: -n @xmath3. ## defining lexical knowledge our goal is to evaluate word learning in lms by following the trajectory of a distributional signature for each target word throughout lm training. however, both in terms of trajectory extraction and signature design, there are many design choices. in this section, we explore and discuss the implications of a range of choice points in defining the distributional signature that is tracked during training. in §4, we discuss how to extract a trajectory from a timestamped sequence of distributional signatures. ## analyzing trajectories given our goal of studying the word acquisition process in lms, we aim to study the trajectory of a signature σ for various words throughout the training of the target lm. however, an entire trajectory may contain too much information for some analyses. in this section, we consider a family of statistics that can be extracted from the trajectory and review the main choice points in doing so. determining aoa by thresholding while many statistics are possible, we focus on age of acquisition (aoa), which is a single number that should be interpreted as the point at which learning has advanced to a satisfactory degree. for human learners, @xcite define aoa as the age at which 50% of children are such that their caregivers report them as understanding the word. @xcite apply this thresholding approach to lms. given a trajectory, they define the aoa to be the first time step at which the signature reaches a threshold defined as τ % of the way between some initial value representing the beginning of learning and some final value representing the endpoint of learning. 5 unfor-tunately, thresholding in this way is only suitable when σ exhibits (roughly) monotonic change over time. while this is true of some signatures we consider, we find empirically that σ + , σ @xmath4, σ i+ , and σ i-are exceptions. thus, we adopt a different approach to extracting aoas based on the notion of a cauchy sequence. intuitively, we say that the target word is learned at the point in the trajectory where the value of the signature becomes close to its neighboring points in the trajectory. our approach is defined formally in app. d. for the sake of uniformity, we apply this approach to all signatures and leave an exploration of thresholding approaches for suitable signatures to future work. smoothing the trajectory empirical trajectories may be noisy due to estimation errors or local instabilities during training @xcite . thus, we consider several techniques for smoothing the trajectory. one approach to smoothing is parametric curve fitting; @xcite and @xcite employ such an approach and assume trajectories follow the form of a sigmoid curve. however, parametric curve fitting requires the modeler to assume the functional form of the curve. if the functional form of the curve is unknown, one can instead smooth the curve using a non-parametric method, e.g., a moving average or a generalized additive model @xcite , as done by @xcite . for simplicity, we opt for a moving average to smooth the trajectories in this paper. ## examining lm learning trajectories before quantitatively comparing lm and child word learning trajectories in §7, we conduct several analyses focusing solely on lm trajectories. ## human vs. lm learning trajectories we now examine the similarities and differences between word learning in lms and humans. ## discussion and conclusion our main objective was to explore the space of distributional signatures of words more comprehensively, with an application to understanding word learning in lms. we showed that the distributional test adopted by @xcite and portelance et al. ( 2023 ) can be viewed as an estimator of a more general distributional signature. this insight also enabled us to define a broader family of signatures that follow a clear typology. however, the question remains: which of these evaluations should be the focus of researchers interested in studying word learning in lms? one of our key findings in §6.3 is that many of these signatures are complementary. this is true with respect to children's aoas as well as in comparison to each other. arguably, considering both positive and negative contexts provides a more complete picture of the lm's distributional knowledge, and comparing the lm's distribution against an llm allows the signature to better reflect the gradient of the ground truth distribution, which is not directly observable. nonetheless, each signature we propose has a clear interpretation and may be useful for specific applications, though extracting usable aoa scores is not always feasible. in §7 we found that we could not predict children's aoas well from any of our proposed distributional signatures. this result might seem somewhat surprising in light of @xcite finding that lm surprisal improves predictions of children's aoas. however, we note that that work uses σ + at the end of training as a predictor of aoa, rather than the aoa of the model as determined by a specific distributional signature. our results do further corroborate @xcite conclusions on this question, and significantly expand them to a wider variety of signatures. additionally, our findings contribute to a growing body of work finding specific differences between the language learning patterns of humans and lms in other domains (e.g., @xcite . on the other hand, @xcite show that multimodal lms can exhibit more human-like learning trajectories and also introduce a novel training objective that further improves human-likeness. future work should apply our distributional tests to these and other potentially more human-like training procedures. besides learning in a world grounded in sensory experience, children also learn through interaction both with the physical world and with other agents @xcite . moreover, unlike lms, children have constraints on production, going through one-word and two-word utterance phases @xcite . these factors no doubt influence the kinds of words children use early in development and may account for the precedence of concrete words. there are only a few examples of training regimes for lms inspired by interaction @xcite @xcite . furthermore, the reliance on stochastic gradient descent and cross-entropy loss likely skew learning trajectories in lms in ways that are not entirely human-like. there are many opportunities to explore more human-like lm training, and we expect that word learning will be an important evaluation of human likeness as these are explored. having better mapped out the space of distributional signatures of lexical knowledge, our work paves the way for comparing trajectories of language models and humans. our findings provide strong empirical support that there are large differences between how human and artificial language learners develop throughout learning and draw attention to the fact that there is significant work to be done to explore pre-training methods and datasets that result in more developmentally plausible language models.