Investigating Critical Period Effects in Language Acquisition through Neural Language Models (2025)

Abstract: Humans appear to have a critical period (CP) for language acquisition: Second language (L2) acquisition becomes harder after early childhood, and ceasing exposure to a first language (L1) after this period (but not before) typically does not lead to substantial loss of L1 proficiency. It is unknown whether these CP effects result from innately determined brain maturation or as a stabilization of neural connections naturally induced by experience. In this study, we use language models (LMs) to test the extent to which these phenomena are peculiar to humans, or shared by a broader class of language learners. We vary the age of exposure by training LMs on language pairs in various experimental conditions, and find that LMs, which lack any direct analog to innate maturational stages, do not show CP effects when the age of exposure of L2 is delayed. Our results contradict the claim that CP effects are an inevitable result of statistical learning, and they are consistent with an innate mechanism for CP effects. We show that we can reverse-engineer the CP by introducing a regularizer partway through training to simulate a maturational decrease in plasticity. All in all, our results suggest that L1 learning on its own may not be enough to induce a CP, and additional engineering is necessary to make language models more cognitively plausible.

## introduction the tension between nature and nurture is central to questions surrounding how humans acquire language. the critical period (cp) for language acquisition is no exception. around the onset of adolescence, humans exhibit a loss in ability to acquire a second language through immersion and a tendency not to forget their first language under deprivation @xcite @xcite . scholars of human development have long debated whether these phenomena are predetermined by innately encoded developmental changes in the maturing brain @xcite @xcite , or natural consequences of increased experience that any typical statistical learner would be subject to @xcite @xcite . until recently, it was difficult to differentiate these two hypotheses; as we could only observe one kind of statistical learner (i.e., humans), we could not identify which properties of its learning process were responsible for its behavior. recent improvement in neural language modeling upends this state of affairs @xcite . language models (lms) can learn to simulate many native-like grammatical judgments @xcite @xcite -long regarded as one of the main behavioral measures of native speaker knowledge @xcite )and, like humans, they acquire this knowledge from unstructured input without the need for negative evidence. however, their learning algorithm and structure differ from those of humans in a number of ways. lms can thus provide additional information about which phenomena are likely to be typical of general language learners, and which are peculiar to humans. in this work, we use language models to study the cp for language acquisition, focusing on second language acquisition and first language attrition. our experiments test for cp effects 1 in lms by training them from scratch on bilingual data, varying only the age of exposure to l 2 . we test whether, like humans, lms learn l 2 more easily when exposed to it simultaneously with l 1 from the beginning of training, rather than when exposed to it only after learning l 1 . similarly, we test whether they fail to forget l 1 after extensive training on it. experimentally, we find that lms are unlike humans in both respects. 2 thus, our results contradict the view that cp effects are an expected consequence of statistical learning, and they are consistent with (but only provide weak evidence for) the view that the cp in humans is a biologically programmed developmental stage. the benefits of studying the cp in neural models, however, go beyond just discerning the two hypotheses above. if lms do differ from humans, it may be useful to attempt to reverse-engineer those learning properties exhibited by humans @xcite . furthermore, minimizing differences between lms and humans is a necessary step in enabling their use more broadly as models of human language acquisition @xcite . thus, we also attempt to reverseengineer a cp by simulating a loss of neural plasticity using elastic weight consolidation (ewc; @xcite , a bayesian regularizer used in machine learning to mitigate catastrophic forgetting. our experiments show that both of the cp effects emerge in tandem when the model's plasticity is explicitly controlled in this way. our findings demonstrate the utility of lms as tools for theories about human language acquisition, and they suggest a path forward to making lms more developmentally plausible models of human language acquisition. ## background: the critical period the proposition that there is a critical period for language learning has long been prominent in language acquisition research @xcite . discussions around the cp, however, typically cluster a number of related observations; these must be teased apart in order to be properly understood @xcite . the critical period can be divided into 3 main phenomena, namely: a cp for l 1 acquisition, a cp for l 2 acquisition, and a cp for l 1 attrition. we focus on the latter two here. 3 2.1 critical period for l 2 acquisition cp effects for l 2 acquisition consist of greater difficulty in learning a second language and worse learning outcomes as the age of exposure increases. as humans vary greatly in the beginning of l 2 exposure, this is perhaps the most well-known of the cp phenomena. the effects of age of exposure on phonetics and phonology (i.e., one's accent) are part of folk knowledge and were a key piece of evidence in the first works to propose a neurological mechanism for these @xcite . numerous studies also show that age of exposure correlates with worse l 2 performance on morphological and syntactic acceptability judgment tasks @xcite . while the exact nature and reliability of these effects has been questioned at times @xcite , the existence of age-of-exposure effects is generally accepted. we refer the reader to several thorough reviews of the relevant evidence @xcite @xcite . in the realm of computational learners, no prior work has tested this cp in a controlled manner. in a more general form, however, l 2 acquisition has been studied in depth @xcite . most related to our work, @xcite trained a number of language models on an l 1 and then fine-tuned these models on both l 1 and l 2 ; they find that, unlike humans, this two-step training improves lms' l 2 performance. this already suggests that lms may not show cp effects for l 2 learning. however, they do make l 2 learning relatively easier by fine-tuning their models on l 1 and l 2 simultaneously within the same bidirectional transformer context, which we contend is not cognitively plausible. beyond @xcite study, weak evidence against the existence of a cp for l 2 in neural networks is suggested by a large body of work on transfer learning which fine-tunes pretrained neural networks to perform new tasks (e.g., @xcite @xcite . these studies indicate that a neural model can achieve superior performance on a fine-tuned task compared to training on it from scratch. however, these studies do not manipulate age-of-exposure while controlling for the total amount of input, and their main focus is on tasks other than language modeling itself, such as classification. ## research questions and methodology to study the claims above, we now put forward two research questions which we investigate with the help of language models. rq 1. can we find evidence of a critical period for l 2 learning in language models? rq 2. can we find evidence of a critical period for l 1 attrition in language models? these are the main questions that we want to investigate, and on which we focus our experiments. we analyze them by first training lms in various multilingual setups-while altering the ages (epochs) at which l 1 and l 2 are acquiredand then evaluating our models on l 1 and l 2 . importantly, we do not make any modifications to the lms' architecture or learning objectives in these experiments. beyond these main questions, we also explore two other research question here. rq 3. does reducing plasticity in language models induce human-like critical period effects? rq 4. are critical period effects in language models dependent on l 1 and l 2 's similarity? investigating rq 3 serves a dual purpose. first, it allows us to test whether the critical period effects associated with l 2 acquisition and l 1 attrition arise in tandem as a result of a loss in plasticity, or whether these phenomena can (at least in one case) be decoupled. second, if we are successful at reverse engineering these cp effects, we obtain a model that more closely resembles human learners and that might be useful for future work. we test this question by evaluating the linguistic performance on both l 1 and l 2 while training a model whose learning objective has an extra regularizer which enforces a reduction in plasticity. similarly, rq 4 also serves a dual purpose. first, it informs us about the relationship between language similarity and cp effects. second, it implicitly assesses how sensitive our results are to a specific choice of language pair, providing us with a notion of how robust our experiments are to this choice. we test this question by performing the above analysis in a number of language pairs which differ in their similarity. ## experimental setup this section provides a comprehensive description of the experimental setup for this project, including: languages, datasets, model architectures, and evaluation methods. 7 languages. our experiments consist of training lms from scratch on language pairs. in this work, we rely on english (en) data for evaluation, as it has a wealth of well-studied resources for assessing language proficiency (see §5). therefore, we justify the selection of the other languages in this study according to their relatedness to english. to reduce the computational overhead, we restrict most of our experiments to only two language pairs: german-english and finnish-english. we choose these languages because they are both (relatively) highresource languages that are well-represented in our chosen data domains. furthermore, german (de) is in the same language family as english (indo-european/germanic), while finnish (fi) is unrelated to both (finno-ugric). nonetheless, to improve the generalizability of our results, we also run one experiment with an extended set of languages as l 1 , using english as l 2 (see experiment 4 in §6). we select languages from various language families (indo-european (ie) or not) using different scripts (latin (l) or not). to represent the extreme endpoints, we also use a different corpus of english as the most closely related l 1 , and a corpus in a programming language (java) as the least closely related l 1 . the complete list is as follows (from most to least related): models. we run our experiments with both autoregressive and masked language models. to represent these two categories, we consider roberta @xcite , an encoder-only transformer trained to predict masked tokens, and gpt-2 @xcite , a decoder-only transformer trained to predict next tokens. we rely on implementations from the huggingface transformers library @xcite , namely, roberta-base with 125m parameters and gpt2 with 137m parameters. we choose these two models due to their availability, size, and scientific relevance. these models are then trained according to the conditions defined in §4.1. in all three sequential conditions, l 2 training starts from the final l 1 checkpoints, but with a new optimizer. hyper-parameters. we run a bayesian hyperparameter search using the weights & biases sweeps api @xcite to identify good model configurations for our training data and methodology. we extract several model configurations (see appendix c) with which we run the various experiments from §6. ## experiments this section describes our four experiments, designed to provide evidence for answering the research questions detailed in §4. the goal of this setup is to study the cp for l 2 learning, which relates to rq 1 and rq 3. we run this experiment on both gpt-2 and roberta models, using german and finnish data as a first language and english data as a second language. the dataset for each language has the same size of 600 million tokens, a factor which was limited by the availability of the finnish data. the models are trained with a limited budget of 6 epochs per language. this number has been empirically determined (after preliminary exploration) to provide a good trade-off between computational costs and model performance (i.e., the models perform sufficiently well after training for 6 epochs and the learning slows down). finally, in order to introduce more variability and provide more result samples, the trainings are run with three different configurations (c 1 , c 2 , c 3 , see tables 4 and 5 in appendix c). 11 we henceforth use glue to refer to a combination of tasks from glue and superglue (see appendix d) implemented in the babylm evaluation pipeline @xcite . we illustrate the results for this experiment in figure 2 . in general, the learning patterns through epochs are similar across conditions (except for interleaved), with variations observed mostly in the final performance. as expected, the monolingual performance is the best among all conditions. we notice that the interleaved condition performs slightly worse than the sequential condition, although the difference is more noticeable in roberta models than in gpt-2 models. both achieve lower scores than the monolingual training (the baseline for native-level language proficiency). results from the sequential-interleaved condition differ based on the model architecture. on gpt-2 it is worse compared to sequential results (i.e., keeping l 1 exposure hinders l 2 learning), while on roberta it is better (i.e., keeping l 1 exposure helps l 2 learning). lastly, the sequential-ewc condition shows clear differences, with much worse l 2 proficiency. unsurprisingly, the reduction in plasticity through regularization has a significant (negative) impact on l 2 's learning outcome. experiment 2: reversing the language order. l 1 ∈ {en}, l 2 ∈ {de, fi}, @xmath0, @xmath1. the purpose of this experiment is to study the cp for l 1 attrition, which relates to rq 2 and rq 3. the experimental setup differs from the previous one mainly in that the order of the languages is reversed: english is used as a first language and german and finnish as second languages. this swap allows us to use all three evaluation benchmarks to track l 1 performance across the entire training process. furthermore, each run is performed with a single configuration (c 1 ). the results for this experiment are displayed in figure 3 . in general, we observe a smaller drop in glue scores compared to blimp scores after exposure to l 1 ceases (i.e., comparing epoch 6 to 12). this is most probably caused by the conceptual difference between evaluations in a zero-shot setting and evaluations that require fine-tuning. when fine-tuning on glue, the lm is once again allowed to learn from l 1 data. in the monolingual condition, l 1 performance keeps improving until the end of training, even though it slows down in the second stage. in the interleaved condition, models reach the same level of l 1 and l 2 proficiency (when comparing these results to the ones from figure 2 ). 12 the more notable result comes from the sequential learning. in this condition, lms rapidly lose the knowledge acquired from l 1 learning after l 2 exposure is started. the final l 1 perplexity values indicate that, in the end, the lms forget almost everything that they have learned before. it looks like the second language completely replaces the first one. however, the loss of l 1 is completely mitigated in the sequential-interleaved condition. when l 1 exposure is prolonged without any reduction in plasticity, models are able to retain all the prior l 1 knowledge. 13 when a computational regularization method is introduced in the sequential-ewc condition, l 1 knowledge is successfully preserved to a certain degree. however, as we have seen from figure 2 , l 2 learning is also harmed in this case. to explore this trade-off, we vary the λ values and test the models' performance on both l 1 and l 2 at the end of training. the results are illustrated in figure 5 . we see that the regularization strength λ highly influences both l 1 and l 2 learning outcomes. when high λ values are used (strong ewc regularization), l 1 knowledge can be predictably maintained at the initial levels. however, l 2 learning will be almost completely impaired. on the other side, when lower λ values are used (weak ewc regularization), l 2 learning is affected less, but l 1 will not be preserved. it is also noticeable that in this case, there is a higher variance on the final l 1 outcomes. as mentioned in §5, we have selected the λ values for all our experiments as the point where l 1 and l 2 performance is roughly equivalent (intersection lines are marked on the plots). thus, we do not favor either very strong or very weak regularization, and also to match the behavior exposed by the interleaved models. this experiment is motivated by the observation that cp effects can become stronger with a later age of exposure. we extend the training time for each language (thus postponing the start of l 2 exposure) to 48 epochs, allowing the model weights to better converge. as this is more computationally demanding, we do only one run per condition and we also omit the sequential-interleaved condition. the results are provided in figure 4 , which shows that the final results are more uniform across the first three conditions, especially between interleaved and sequential. we still see a substantial loss in l 1 performance in the sequen-tial condition, indicating that longer l 1 training does not lead to entrenchment, though these ppl scores are for fi and de, and thus are not directly comparable to the en from other experiments. the performance of the interleaved setting shows a slight improvement compared to regular training, indicating that a longer training duration was beneficial (and necessary for convergence). the learning pattern for the sequential-ewc condition does not change, i.e., the final l 2 performance does not significantly improve with additional training. it also appears the ewc regularization does not simply slow down l 2 learning, but rather acts as a lower bound: l 2 knowledge can never improve past a certain point for a given choice of λ. experiment 4: diversifying the language pool. l 1 ∈ {ar, de, el, es, fi, ko, nl, pl, ru, tr, java}, l 2 ∈ {en}, @xmath2, @xmath3. this experiment both addresses rq 4 and provides more diversity in the results; the latter is especially important considering the limited selection of languages for the previous experiments. the main focus here is to find whether the choice of languages or, more concretely, the degree of similarity between l 1 and l 2 , has any impact on the cp. for this, we consider a wider range of languages based on their relatedness to english (see §5). to accommodate for the increase in computational demand, we consider only the gpt-2 model architecture (with configuration c 5 ), we reduce the size of the training data to 100 million tokens, we only train with english as l 2 , and we only run ppl and blimp evaluations. we present the findings in figure 6 . considering l 2 ppl, we find the expected pattern, i.e., ppl generally increases (gets worse) when l 1 is less closely related to it. however, the effect size is quite small (hence we plot the difference with respect to monolingual condition): only 0.1-0.3 in most cases, while the total ppl ranges from about 2.3-2.5. however, the blimp results do not support the same conclusion. mostly, blimp scores vary seemingly at random, and indeed these differences could reflect random noise due to model initialization. as expected, l 2 performance on blimp is greatest when l 1 and l 2 are just different corpora of english. but curiously, java pretraining aids blimp performance more than most natural languages. the only condition where we can make the strongest case for an effect is sequential-ewc, which suggests that perhaps relatedness effects exist at the beginning of transfer but are wiped out by extensive l 2 training. ## discussion we first address our four research questions from §4. we then explore our results' implications for the theoretical claims introduced in §3. comparing the interleaved and monolin-gual conditions also informs this question. for humans, early l 2 learners resemble monolinguals in their l 2 performance. however, monolingual models outperform interleaved ones on all three metrics. looking at the entire learning trajectory, interleaved models differ markedly from both monolingual and sequential models, showing more gradual and delayed improvements. rq 2. this rq concerned the cp for l 1 attrition. humans show few signs of l 1 attrition after the cp, even if l 1 exposure decreases or ceases. our results in experiment 2 provide strong evidence against a similar phenomenon in typical lms. instead, we find that l 1 performance worsens rapidly and to a large degree in the sequential condition after l 1 exposure ceases. this is expected given the susceptibility of neural networks to catastrophic forgetting. the loss of l 1 proficiency is prevented by continuing l 1 exposure in the sequential-interleaved condition, but such continued exposure is not necessary in humans. 14 rq 3. this rq concerned the trade-off between l 2 learning and preventing l 1 attrition when explicitly reducing plasticity partway during learning. we find strong evidence for such a trade-off when comparing the sequential-ewc condition to the sequential condition. the value of λ we selected preserved l 1 performance substantially compared to the sequential models, at the cost of harming final performance in l 2 . the l 2 learning curves also converge relatively quickly when plasticity is reduced. our exploration of different values of λ (figure 5 ) shows that preserving l 1 to monolingual levels harms l 2 acquisition by roughly 1.5 nats of l 2 performance, but we do not directly compare grammatical performance of our models to that of human late l 2 learners. thus, our attempt at reverse engineering shows a broadly human-like learning pattern when using ewc, but we cannot say quantitatively and at a high level of granularity whether the result is human-like. rq 4. this rq concerned the impact of language similarity on cp effects. our results showed that the language family of l 1 and its script has an impact on l 2 learning in the expected way for only a subset of evaluations. based on earlier findings from papadimitriou and jurafsky (2020) and @xcite , we had expected l 2 performance would be greater when l 1 is more closely related. however, only our results for ppl support this prior conclusion. our results for blimp do not unless ewc is applied, suggesting that models are ordinarily plastic enough to learn the grammar of l 2 regardless of relatedness, unless plasticity is specifically reduced. we note that papadimitriou and jurafsky (2020) reduce plasticity by freezing the model weights before transferring to l 2 , but we do not venture an explanation why @xcite seemingly find models to be less plastic than we do. ## conclusion there are many obvious ways in which lms differ from humans when learning language. our work reveals another important point of divergence, namely, that lms remain far more plastic later into the learning process than human learners. even though humans and models differ substantially, comparing the learning trajectories of human learners to those of computational ones tells us something about humans: those features that we do share are more likely to be natural properties of language learning, while those that we do not are more likely to require idiosyncratic innate mechanisms. our results provide strong evidence against the hypothesis that cp effects are necessarily induced solely by experience, and they are consistent with, but only provide weak evidence in favor of, the view that innate mechanisms are necessary to explain cp phenomena. it will be important to replicate these results in other artificial learners, including ones which resemble humans more closely in other respects. our study thus constitutes early progress towards integrating modern lms into the study of human language acquisition.