Is Child-Directed Speech Effective Training Data for Language Models? (2024)

Abstract: While high-performing language models are typically trained on hundreds of billions of words, human children become fluent language users with a much smaller amount of data. What are the features of the data they receive, and how do these features support language modeling objectives? To investigate this question, we train GPT-2 and RoBERTa models on 29M words of English child-directed speech and a new matched, synthetic dataset (TinyDialogues), comparing to OpenSubtitles, Wikipedia, and a heterogeneous blend of datasets from the BabyLM challenge. We evaluate the syntactic and semantic knowledge of these models using developmentally-inspired evaluations. Through pretraining experiments, we test whether the global developmental ordering or the local discourse ordering of children’s training data supports high performance relative to other datasets. The local properties of the data affect model results, but surprisingly, global properties do not. Further, child language input is not uniquely valuable for training language models. These findings support the hypothesis that, rather than proceeding from better data, the child’s learning algorithm is substantially more data-efficient than current language modeling techniques.

## introduction transformer-based language models (lm) show very strong performance on a wide variety of downstream tasks, but typically only after pretraining on hundreds of billions to trillions of words @xcite . in contrast, human learners use language fluently after far less training data -in the 10s to 100s of millions of words. this "data gap" @xcite of several orders of magnitude poses a substantial challenge for machine learning. is the source of human children's efficient learning a function of their data or their learning algorithms? while children receive rich multi-modal input from their exploration of the world, here we focus on their language input, which has been a major focus of study in developmental psychology @xcite . one hypothesis is that the language data that children receive is a uniquely rich learning signal -conversational interaction with their caregivers -that is curricularized optimally to support learning @xcite @xcite . indeed, interventions to increase the quality of caregiver language do produce improvements in children's language learning @xcite , and interventions to simplify model training data also result in stronger performance @xcite . language model pretraining experiments provide a targeted method for investigating dataset quality @xcite : we can manipulate the training data available to models to create "controlled rearing" experiments. we take advantage of this method to investigate the properties of childdirected speech for learning the syntactic and semantic structure of language. we use gpt-2 and roberta as our simulated learners, and pretrain on natural and synthetic child-language data. for each of these, we conduct two experiments. first, we investigate whether the natural curricularization of children's input -from simpler utterances to more complex conversations -affects language model learning. second, we test whether the local discourse coherence structure of dialogue results in better learning. finally, we compare to learning on other data sources: opensubtitles (more general conversation data), wikipedia, and a more heterogeneous blend of data from various sources. we find that the curricularization of child language does not provide a uniquely valuable signal for language models, supporting the hypothesis that other aspects of children's learning (not simply the data) -perhaps interactions with their training data -are responsible for their efficiency relative to language models. on the other hand, the source, composition, and local properties of the training data have measurable effects on model performance. ## related work the efficiency of children's learning has been an important focal point for recent nlp efforts @xcite . last year's babylm challenge held the training data for models constant, while encouraging entrants to investigate alternative learning architectures @xcite . smaller models of this type must be evaluated using more appropriate targeted benchmarks, including evaluations of semantic @xcite and grammatical abilities @xcite . these evaluations have even been used to benchmark performance based on data from a single child @xcite . there have also been multimodal investigations of children's learning, particularly in the form of developmental egocentric video data, e.g. @xcite ; @xcite . the method of "controlled rearing" (manipulating data while holding the model constant) for language models (frank, 2023b) has a long history in cognitive science, e.g. @xcite , but has recently become prominent for testing learnability claims @xcite @xcite . often, models trained on naturallyoccurring corpora are contrasted with counterfactual corpora constructed via targeted experimental manipulations -for example, shuffling sentence ordering @xcite or removing particular constructions @xcite . curricularization of training data is widely investigated in machine learning @xcite , with the guiding idea being that an appropriate ordering of training examples can lead to a smoother path to the desired objective. children's development is argued to create a curriculum to facilitate their learning @xcite , and starting small is hypothesized to be efficient for language learning @xcite . in one study from the visual domain, @xcite trained self-supervised models on data from infants and found that a developmental ordering leads to stronger eventual performance compared with a reversed ordering. our study tests this hypothesis in the language domain. language curricularization has been investigated as part of the babylm challenge @xcite . our goal is not to assess curriculum learning more generally but to measure the extent to which the specific developmental curriculum that children are exposed to is helpful. to do this, we focus on the developmental data available to children, i.e. child-directed speech. on the other hand, babylm contributors relied on the use of proxies rather than age-related information, including ranking sentences by surprisal @xcite and lexical frequency @xcite . ## results and analysis major results of our experiments can be found in tables 1 to 5 . statistical significances can be found in tables 11 to 15 childes and wikipedia. td performs best on ws, tying with babylm. this suggests that a diverse mixture of data sources, or more varied conversational data, may be more effective for training smaller autoregressive models on limited data. additionally, synthetic conversation data appears more effective than natural data for training such models at a smaller scale. from childes continues to perform the worst on both zorro and ws, and synthetic conversation data proves more effective than natural data for smallscale lm training. childes is heavily skewed towards younger ages (see figure 1 in appendix c), whereas td is more uniform across ages with more sophisticated conversations intended to simulate speech to older children. as such, it contains a higher fraction of more grammatical utterances. while collecting td, we ensured that it was diverse in conversation type, participants, and content, likely resulting in a more comprehensive coverage of the distribution of potential conversations. this may lead to more effective learning of syntax and semantics, and similar logic likely applies to opensubtitles. further, high-quality synthetic data -in contrast to naturalistic data, which contains disfluencies and occasional garbled tokens due to transcription issues -may simply be better suited for training lms, especially when data is limited. while wikipedia is complex and diverse, it may lack conversational elements crucial for small-scale grammar learning, such as back-and-forth interaction and pragmatic cues found in dialogue. its expository style also limits exposure to informal speech patterns, which could be important for improving syntactic understanding at smaller scales. as seen in we omit roberta global ordering experiment results here as our repeated buckets training approach did not work well; the models seemed unable to converge properly. their behavior was close to random, barely achieving above chance on zorro and ws, and we do not interpret them here. results can be found in from tables 4 and 5 , we see that local ordering affects model performance. disrupting discourse coherence negatively affects zorro and ws for childes, despite zorro focusing on singlesentence evaluations. the effect, especially on ws, is more pronounced for childes than td, likely due to childes' shorter average utterances (≈ 4 words vs. 13). hence, reordering childes utterances likely has a greater effect on the model's ability to learn semantics across a larger set of short utterances. surprisingly, random utterance order has little to no effect on td performance, suggesting that td, and possibly synthetic data, may be more robust to local coherence disruptions. ## conclusion & future work why do children need much less data than language models to achieve fluency? in experiments with gpt-2 and roberta on childes, opensubtitles, wikipedia, babylm, and our synthetic tiny-dialogues dataset, we found that synthetic childdirected data outperformed natural child-directed data. in general, more diverse datasets (e.g. general conversation data or a mixture of different data sources) may result in better learning than homogeneous child-directed data. interestingly, global developmental curricularization had little impact, whereas local discourse coherence mattered, especially for natural child-directed conversation data. in sum, it seems that the curricularization of child language does not provide a uniquely valuable signal for language models. however, the source, composition, and local properties of the training data affect model learning. we hope that future work builds on our work here to expand upon the available evaluation benchmarks and data mixtures for comparison between models and children.