Evaluating Neural Language Models as Cognitive Models of Language Acquisition (2023)

Abstract: The success of neural language models (LMs) on many technological tasks has brought about their potential relevance as scientific theories of language despite some clear differences between LM training and child language acquisition. In this paper we argue that some of the most prominent benchmarks for evaluating the syntactic capacities of LMs may not be sufficiently rigorous. In particular, we show that the template-based benchmarks lack the structural diversity commonly found in the theoretical and psychological studies of language. When trained on small-scale data modeling child language acquisition, the LMs can be readily matched by simple baseline models. We advocate for the use of the readily available, carefully curated datasets that have been evaluated for gradient acceptability by large pools of native speakers and are designed to probe the structural basis of grammar specifically. On one such dataset, the LI-Adger dataset, LMs evaluate sentences in a way inconsistent with human language users. We conclude with suggestions for better connecting LMs with the empirical study of child language acquisition.

## introduction the growth of neural language models (lms) for nlp over the past decade has been followed by a growth in research on the potential of these models to provide insights into the cognitive aspects of human language acquisition, representation, and processing @xcite . good, even human-like, performance on nlp tasks does not necessarily imply that lms solve these in humanlike ways, so computational linguists have designed a wide variety of experimental paradigms to probe specific properties of the models' linguistic knowledge @xcite @xcite @xcite @xcite @xcite et al., 2021) these range from ways of classifying or extracting structures from internal representations (e.g., hewitt and manning, 2019; tenney @xcite @xcite , to building tasks inspired by psycholinguistic processing studies and classic acceptability rating task that theoretical linguists use to infer grammatical knowledge (e.g., @xcite @xcite . of these approaches, acceptability rating may be the most popular. large acceptability rating data sets focusing on syntax, semantics, and morphology, such as blimp @xcite , syntaxgym @xcite , and cola @xcite lend themselves to benchmarking, and these sit alongside myriad smaller scale studies focused on specific lingusitic phenomena (e.g., @xcite @xcite . results have been impressive for the most part. it appears, from the logic of these studies, that many state-of-the-art neural models are capable of inducing human-like grammatical knowledge on unannotated data -like children during language acquisition. ## knowledge of language and its acquisition one of the goals of linguistic theory is to characterize the properties that distinguish grammatical from ungrammatical sentences in a language. the empirical study of grammaticality, however, mainly relies on native speakers' acceptability judgments, which interact with other cognitive and perceptual systems and generally produce gradient results. for example, longer and more complex sentences, even when fully grammatical, are rated as less acceptable than shorter and simpler sentences. nevertheless, large-scale investigations have established the structural basis of a categorical grammar @xcite . for example, syntactic constraints that prohibit certain transformational processes are shown to have a "super-additive" effect that go beyond acceptability rating due to sentence length and other non-structural factors. furthermore, acceptability judgments collected at scale are highly consistent with the data reported in the theoretical literature typically gathered informally with few consultants @xcite @xcite . the structural basis of language and its uniformity across the linguistic community can be better appreciated from the perspective of child language acquisition. recent years have seen renewed interest in individual differences across child learners @xcite , especially with respect to vocabulary acquisition @xcite . it is at least possible that children differ in their cognitive abilities for language and learning, but it is empirically obvious that they differ in their experience with language. longitudinal records of child language development have made it possible to track both children's vocabulary growth, and the development of the structural aspects of their grammar. in the providence corpus @xcite , for example, six children were recorded at regular intervals from age 1 to 3. on average, fewer than 20% of the first 100 words are shared between any two children. the overlap merely rises to about 40% for the first 1,000, which is the upper limit of a three-year old's vocabulary size @xcite . yet these children's grammars are highly uniform even at this stage. major syntactic categories, word order and argument structure, and the core morphological rules are firmly established before age three @xcite on the basis of at most around 10m words per year @xcite and a vocabulary size of only a few hundred types @xcite , and all children produce similar grammatical errors during this time. recent decades have also seen a convergence between the psychological and formal study of language development and the quantitative study of language variation in early childhood. the sociolinguist, bill labov, remarks that "the end result is a high degree of uniformity in both the categorical and variable aspects of language production, where individual variation is reduced below the level of linguistic significance" @xcite . the acquisition of vocabulary and grammar provide clues for investigating the capacities of lms. vocabulary learning is a matter of rote learning. this includes not just the arbitrary pairing of sounds and meanings, but also morphological processes (e.g., irregularity) and syntactic structures (e.g., sub-categorization, collocations, etc.). there is no escape from experience: more data results in better learning. but, the structural aspects of the grammar are different. they require form generalizations over the vocabularies. the distinction between rote learning and structural learning (words vs. rules) is not well reflected by existing lm benchmarks including those discussed in this paper. in practice, these benchmarks are a mixture of tests for both vocabulary learning and grammar learning. moreover, they are stochastically generated by templates: as such, a large number of test sentences are immediately available, but they lack the structural diversity that has proven revealing in the theoretical study of grammar. furthermore, the sentences are sometimes highly unnatural and semantically/pragmatically uncontrolled, which is precisely the confounding factor that linguists seek to neutralize when attempting to uncover the structural basis of language. @xcite are aware that their templates generate unnatural sentences, presenting the blimp sentence 'sam ran around some glaciers.' as an example. we found similar issues in zorro, such as 'the lie on the foot is flat .,' the first sentence in zorro's across_prepositional_phrase paradigm (lie is a noun). the blimp authors state that this is not a problem because it affects both sentences in a pair, but how can we rule out unintended interactions between the grammatical phenomenon under evaluation and the semantic implausibility? @xcite find that this semantic implausibility may affect judgments of sentence wellformedness, even in the forced choice (fc) task used to collect the human baselines in blimp. indeed, there are already a large amount of carefully curated linguistic materials that are not only structurally diverse but also have minimized lexical and semantic confounds. furthermore, these datasets (e.g., the adger/li dataset; section 4) have been evaluated for acceptability at an individual level by a large pool of native speaker subjects and show very high convergence rates across individuals. they will be especially informative if we are to explore the structural knowledge of lms. minimal pairs need to be carefully constructed to control for length and lexical frequencies. blimp aims to accomplish this with automatic generation from templates, but as we discuss, it often yields sentences with low structural diversity and implausible semantics. the benchmark corpus includes data sets for 12 linguistic phenomena, including anaphor agreement, argument structure, binding, control/raising, and others listed in the appendix. these are further divided into 67 paradigms, each containing 1000 sentences pairs, which are meant to test variants of the phenomena, for example the phenomenon determiner-noun agr. contains 6 paradigms for adjacent agreement, agreement with irregular nouns, and agreement with adjectives intervening. blimp has become a standard nlp benchmark for this task and will be used as part of the test data for the upcoming babylm challenge. zorro @xcite explicitly aim to evaluate the relationship between lms and the acquisition of grammar. they introduce baby-berta_ao-childes "an acquisition-friendly version of roberta," trained on english childdirected/produced speech (cds) approximating the total input of a typical english-learning sixyear-old. they train variants on only cds from ao-childes @xcite , a pre-processed version of english childes @xcite , as well as variants on larger datasets from other sources. because babyberta_ao-childes (henceforth babyberta) was trained on much less text than typical large transformer models are, its vocabulary is much smaller. to mitigate the impact of out-ofvocabulary (oov) items on their tests, the authors introduce a new grammaticality test suite, zorro, foot_2 in the style of blimp. sentence pairs are generated for one paradigm each for 11 of blimp's 12 phenomena, along with two additional phenomena. however, we show that the zorro sentences are not only lexically simpler as intended, but their templates are also far less complex and even less varied than the sentences in the corresponding blimp phenomena. full lists of paradigms for each data set can be found in the appendix, and the full data sets themselves are made available by the benchmarks' original authors. ## an alternative: the li-adger dataset the li-adger dataset is a comprehensive collection of 519 sentence types, 300 collected by @xcite from linguistic inquiry (li) 2001-2010, foot_4 a major theoretical journal in linguistics, and 219 collected by sprouse and almeida (2012) from @xcite core syntax textbook. 7 each sentence type includes eight hand-constructed, semantically plausible sentences, assembled into 150 pairwise (li) and 105 multi-condition (adger) phenomena where each minimal pair is lexically matched. we provide an example of each in the li-adger dataset improves upon the prior two datasets in three key ways. firstly, unlike blimp and zorro, the li-adger sentences are controlled for semantic implausibility, which has been shown to be a strong confounding factor when eliciting human judgments @xcite . second, the 255 total pairwise and multi-condition phenomena achieve much more diverse coverage of syntactic phenomena than the 67 paradigms in blimp, and the 23 paradigms in zorro. third, the human judgments were collected using the magnitude estimation (me) task (and likert scale (ls) in the case of the li sentences) in addition to forced-choice (fc) as in the blimp human baselines. we believe this to be a crucial advantage because the fc task treats sentence acceptability as functionally categorical: a sentence is only acceptable or not relative to its minimal pair counterpart, whereas tasks such as me allow us to make comparisons within and across minimal pairs, thereby treating sentence acceptability as a gradient measure. with this dataset, we conduct the following two tests. first, in line with vázquez martínez (2021), we sort the li-adger dataset into 2391 unique minimal pairs. we then collect pseudo log-likelihood scores for each sentence from several models evaluated by @xcite , and score them using the same criteria as blimp and zorro. as a baseline for the models, we include log-likelihood and syntactic log-odds ratio (slor; @xcite scores by a trigram model trained on the british national corpus (bnc; 100m words) by @xcite . we include the results of this test in figure 1 . we observe that all models are further from the human baseline as compared to those in blimp (no human baselines were reported for zorro). but more importantly, we observe that the trigram model scored using slor performs on par with the baby-berta models and approaches the performance of roberta @xcite trained on 10m words. if we were to adopt the "positive results from model learners are more meaningful than negative results" argument, then the trigram model is as suitable a model of language acquisition as babyberta is. raw accuracy notwithstanding, we proceed to conduct a novel test of judgment variability on our collection of lms. we take advantage of the structure of the li-adger dataset in the following way: there are 519 sentence types, and for each type there are eight sentences that retain the same syntactic structure but vary lexical items at the locus of the syntactic structure tested (e.g., the head of a verb phrase from which extraction takes place). these datasets thus allow us to contrast the consistency of human judgment across and within construction types against that of the lms. we z-score the lm judgments to make them comparable to the human judgments. then, for each set of eight sentences, we take the mean and standard deviation of all the judgments for humans and each lm. we find that the models are much more variable in their judgments: the human judgments, on average, vary by 0.288 standard deviation (std. dev.) units within a given set of sentences. on the other hand, the lm that least varies is baby-berta wiki, varying by 0.451 std. dev. units on average. the rest of the models nearly double the variability of the human judgments, ranging from 0.518 for roberta-10m-1 to 0.554 for bert-large-cased. variability appears to increase rather than decrease as training size and performance increase. surprisingly, the trigram model, when scored using log probabilities, is the closest in variability to the human judgments at 0.331 std. dev. units, but also the furthest when scored using slor with a variability of 0.599. once again we find that a positive result on one test or another is not enough to draw positive conclusions. for further illustration, we correlate the means and standard deviations of 512 sentence types across each lm and humans and plot the results in figure 2 . both in terms of mean and standard deviations, we observe generally high correlations between the various neural lms, but much lower correlations between the lms and humans. this suggests that whatever the lms are doing, good or bad, does not appear to be human. interestingly, the babyberta lms show very high correlations with the naive trigram log-likelihood scores and very low with trigram slor scores, raising further suspicions that these small acquisition-inspired lms behave like a very non-human-like model. ## discussion it is widely recognized that children acquire language in ways that appear quite different from lm training. there is a growing realization that the cognitive relevance of lms can only be established in a comparable setting. bringing down training size requirements stands not to not only improve the applicability of such models to the study of language acquisition but also to efficient nlp on low-resource languages. however, in this paper, we observed several weaknesses in blimp and zorro, two minimal pair benchmarks for evaluating the linguistic knowledge of neural language models. we believe that it is worth critically revisiting the underlying assumption that positive results on such benchmarks are a demonstration of human-like representations or human-like language acquisition, especially if an evaluation can be solved in unintended ways, or if it does not reflect an adequately broad range of linguistic structures. it is unlikely that a behavioral probe, such as these large binarized benchmarks, can fully capture the complexity of linguistic knowledge. to this end, we made a case for also evaluating with curated benchmarking datasets: the gradient acceptability judgments from human subjects makes these effective probes for the structural basis of grammar. together with a range of tests, from carefully constructed tests of grammaticality to probes correlating the internal state of lms with their predictions need to complement theoretical, psycholinguistic, and neurolinguistic studies before a meaningful cognitive claim about the nature of neural language models can be made. we end with some broader discussion about language acquisition and the cognitive interpretation of computational models. while it is now widely recognized that children learn language with only a fraction of the data needed for large lm training, merely reducing the amount of training data alonesuch as the 100m word threshold in the babylm challenge -still falls short of the requirement for an adequate model of language acquisition. while it is true that a native speaker's knowledge of language can be established on the basis of approximately 100 million words, child language research makes clear that not all aspects of linguistic knowledge are learned at the same time. some, such as inflectional morphology, case marking, word order, and major transformations are acquired very early in all languages studied so far (e.g., @xcite at an order of magnitude fewer words of input, while others are learned rather late: these include derivational morphology @xcite , passivization @xcite , control and cleft structures @xcite and the dative constructions @xcite in the case of english, but these may emerge much earlier in other languages. this suggests that successful learning in the limit (e.g., 100m word) is not sufficient. for example, while a neural model of english past tense @xcite eventually learns the "add -ed" rule, it does so with over 3,000 verb lemmas. by contrast, children learn that rule before or around 3 @xcite , when their vocabulary only contains around 300 or so verbs @xcite . to serve as cognitive models of language, it is thus important to compare the training trajectory of lms as a function of the training data volume against the developmental benchmarks of specific linguistic phenomena which have been amply documented in the empirical literature on child language.