B aby BERT a: Learning More Grammar With Small-Scale Child-Directed Language (2021)

Abstract: Transformer-based language models have taken the NLP world by storm. However, their potential for addressing important questions in language acquisition research has been largely ignored. In this work, we examined the grammatical knowledge of RoBERTa (Liu et al., 2019) when trained on a 5M word corpus of language acquisition data to simulate the input available to children between the ages 1 and 6. Using the behavioral probing paradigm, we found that a smaller version of RoBERTa-base that never predicts unmasked tokens, which we term BabyBERTa, acquires grammatical knowledge comparable to that of pre-trained RoBERTa-base - and does so with approximately 15X fewer parameters and 6,000X fewer words. We discuss implications for building more efficient models and the learnability of grammar from input available to children. Lastly, to support research on this front, we release our novel grammar test suite that is compatible with the small vocabulary of child-directed input.

## introduction the rise of transformer-based language models (tlms) powered by multi-head self-attention @xcite has been accompanied by impressive performance gains in virtually all areas of nlp. when pre-trained on massive datasets on the order of billions of words, models like gpt-2 @xcite , @xcite , and roberta @xcite achieve impressive scores on benchmarks of natural language understanding tasks @xcite . although tlms were developed for applications in language technology, their successes raise fundamental questions for acquisition research, including unsupervised grammar induction. to address these questions, however, we require models that are trained on developmentally plausible input, matched in quantity and quality to the input that children are exposed to. cognitive scientists have long debated to what extent grammatical knowledge -the categories and structures that determine how words are ordered and inflected in a given language -can be learned from exposure to language input alone @xcite @xcite , or requires built-in linguistic knowledge or inductive biases @xcite . demonstrations of strong grammatical knowledge acquired by tlms trained on raw language data @xcite make tlms a promising new tool for developmental psychologists interested in this debate. however, with rare exceptions, most tlms have beenand were designed to be -trained on many orders of magnitude more data than is available to children. by the time middle-class english-speaking children have acquired near adult-like grammatical knowledge (by about age 6 @xcite , though improvements can be detected long after), they have been exposed to no more than 10-50m words @xcite -at least 600 times fewer words than roberta. current tlms thus have an enormous data advantage over children, which limits the cognitive conclusions that can be drawn from evaluating off-the-shelf models. to remedy this, we present an acquisitionfriendly version of roberta, babyberta, that is trained on input which is both qualitatively and quantitatively comparable to that of the average english-speaking 6-year-old. while our primary aim is to develop a useful resource for developmental psychologists, we also discuss implications for developing more efficient models that will be of broad interest to the nlp community. ## discussion transformer based language models (tlms) extract linguistic generalizations from raw, unanno-tated language data. their impressive scores on grammatical benchmarks @xcite ) can be used to estimate lower bounds for how much linguistic knowledge -in principle -can be acquired based on word-string input alone, and what kinds of architectures make this learning possible. however, the tlms used in such studies were not trained on language matched in quantity and quality to the input children receive. when trained on massive datasets, tlms operate in a very different regime, in which it becomes possible to memorize a large number of individual observations, and/or support abstraction with orders of magnitude more data than are available to children. therefore, claims about the grammatical proficiency of existing off-the-shelf tlms cannot inform questions about children's grammatical development. at minimum, claims about what children might learn without the aid of built-in linguistic knowledge should be based on models trained on developmentally plausible datasets. towards this end, we developed babyberta, a variation of robertabase with 15x fewer parameters. when trained only on 5m words of child-directed input, baby-berta achieved an overall accuracy on our grammar test suite competitive with roberta-base. performance, however, was far from perfect, and babyberta was at chance in evaluating the grammaticality of sentence pairs that contrast negativepolarity-item (npi) licensing, gender agreement, ellipsis, superlative quantifiers, and island effects involving adjuncts. more research is needed using a larger set of grammatical phenomena, and childdirected input in languages other than english, before making strong conclusions about learnability. ## conclusion while child language acquisition research and nlp have largely developed independently, we think that tlms present a promising opportunity for inter-disciplinary researchers to gain new insight into fundamental questions about the learnability of grammar. by using tlms to study what is learnable and not learnable given language available to children, work in this area has the potential to shed new light on debates concerning the innate structure necessary for language acquisition. to support research on this front, we proposed and released babyberta, a tlm trained and optimized on developmentally plausible language data. wei xu, chris callison-burch, and courtney napoles. 2015. problems in current text simplification research: new data can help. transactions of the association for computational linguistics, 3:283-297. karolina zaczynska, nils feldhus, robert schwarzenberg, aleksandra gabryszak, and sebastian möller. 2020. evaluating german transformer language models with syntactic agreement tests. in proceedings of the 5th swiss text analytics conference and the 16th conference on natural language processing, swisstext/konvens 2020, zurich, switzerland, june 23-25, 2020, volume abs/2007.03765, zurich, switzerland. ceur workshop proceedings. yian zhang, alex warstadt, haau-sing li, and samuel r bowman. 2021. when do you need billions of words of pretraining data? in "proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers)", page 1112-1125.