Language acquisition: do children and language models follow similar learning stages? (2023)

Abstract: During language acquisition, children follow a typical sequence of learning stages, whereby they first learn to categorize phonemes before they develop their lexicon and eventually master increasingly complex syntactic structures. However, the computational principles that lead to this learning trajectory remain largely unknown. To investigate this, we here compare the learning trajectories of deep language models to those of human children. Specifically, we test whether, during its training, GPT-2 exhibits stages of language acquisition comparable to those observed in children aged between 18 months and 6 years. For this, we train 48 GPT-2 models from scratch and evaluate their syntactic and semantic abilities at each training step, using 96 probes curated from the BLiMP, Zorro and BIG-Bench benchmarks. We then compare these evaluations with the behavior of 54 children during language production. Our analyses reveal three main findings. First, similarly to children, the language models tend to learn linguistic skills in a systematic order. Second, this learning scheme is parallel: the language tasks that are learned last improve from the very first training steps. Third, some – but not all – learning stages are shared between children and these language models. Overall, these results shed new light on the principles of language acquisition, and highlight important divergences in how humans and modern algorithms learn to process natural language.

## introduction language acquisition is marked by a series of successive stages @xcite @xcite . within their first year of existence, humans infants successively acquire prosody contours @xcite , phonetic categories @xcite @xcite and frequent words @xcite . ## results we aim to compare the learning trajectories of deep language models to those observed in 54 children aged between 18 months and 6 years. for this, we trained variants of gpt-2 models @xcite from 48 different random seeds with the wikitext103 dataset @xcite and evaluated each model on 96 linguistic probes every 100 steps. at the end of this training, 64 probes (66%) were achieved above chance level (50% accuracy) by all models. in comparison, a pretrained version of gpt-2 large @xcite provided by hugging face foot_0 , and trained on a much larger dataset foot_1 , achieves above-chance performance on 93 of the 96 probes. ## discussion the stages followed by children to acquire language has been the topic of intense research @xcite @xcite . while this learning trajectory is becoming clearer for sublexical representations @xcite , the acquisition of higher-level syntactic and semantic processing remains largely unclear. here, we approach this long-lasting question through the lens of a deep language architecture, gpt-2 @xcite , to test whether this model follows a learning trajectory similar to children.