Second Language Acquisition of Neural Language Models (2023)

Abstract: With the success of neural language models (LMs), their language acquisition has gained much attention. This work sheds light on the second language (L2) acquisition of LMs, while previous work has typically explored their first language (L1) acquisition. Specifically, we trained bilingual LMs with a scenario similar to human L2 acquisition and analyzed their cross-lingual transfer from linguistic perspectives. Our exploratory experiments demonstrated that the L1 pretraining accelerated their linguistic generalization in L2, and language transfer configurations (e.g., the L1 choice, and presence of parallel texts) substantially affected their generalizations. These clarify their (non-)human-like L2 acquisition in particular aspects.

## introduction cross-lingual transferability of language models (lms) has attracted much attention. for example, large english lms show some translation performance even when using a small amount of non-english languages as training data @xcite , which indicates the efficient language transfer from english to others. such cross-lingual transferability has been evaluated by holistic measures, such as perplexity and accuracy on downstream tasks @xcite @xcite . on the other hand, there is much room for investigating them from linguistic perspectives; e.g., grammatical knowledge acquisition and language transfer tendencies among languages. in this study, we investigate the cross-lingual transferability of lms from a perspective of second language (l2) acquisition. our main research question is how first language (l1) acquisition of lms affects the efficiency of grammar acquisition in l2. to answer this question, we design an experimental procedure (section 2): (i) pretraining lms in a certain language (assumed to be the l1 speakers), (ii) further training them in english as an l2, and (iii) evaluating and analyzing their linguistic generalization in l2. as l1s, we chose four languages with different levels of difficulty in transferring to english, i.e., french, german, russian, and japanese. the size of training data is restricted to match the human-like l2 acquisition scenario, which enables better comparison with human l2 acquisition tendencies and, hopefully, provides insights into l2 acquisition from a computational linguistic perspective. we begin with exploring the inductive biases of several l2 training methods (section 3). specifically, we compared some variations of l2 data settings, such as training on only the l2 texts or on l1-l2 translation pairs. we observed that, for example, feeding l1-l2 translation pairs into lms slowed down their l2 grammar acquisition, compared to only feeding l2 monolingual texts every two epochs. in our main experiments, we conducted exploratory analyses of the effects of l1 training on l2 grammar acquisition (section 4). we gained three main insights. first, l1 knowledge promotes better linguistic generalization in l2 (section 4.1). second, different l1s incur different generalizations in l2 (section 4.2). more specifically, japanese and russian are far behind french and german, which is consistent with the human-defined difficulty levels of language transfer @xcite . third, l1 pretraining gives different effects on different types of grammar items (section 4.3). in particular, morphological and syntactic items get larger gains than semantic and syntax&semantic items. in more detail, we analyzed the process of l2 acquisition (section 5). we investigated how l2 knowledge acquisition progresses (section 5.1) and found that l2 knowledge acquisition does not progress so much until seeing the whole dataset many times (e.g., 50-100 times), implying their data inefficiency. furthermore, we also observed the l1 knowledge degrade during l2 acquisition; this motivates us to balance the source-target linguistic knowledge during language transfer. ## preliminary experiment: l2 exposure configurations first, we investigate the inductive bias of l2 training settings. while existing studies use parallel data as an input for cross-lingual training @xcite , we investigate the bias in this setting from l2 grammar acquisition perspectives. settings: we set up the three training settings with different input data: (i) l1-l2 text pairs without the translation relationship (tlm-nopara), (ii) l1-l2 translation pairs (tlm-para), and (iii) a mixed setting where l2 text concatenated with l1 parallel text is used as input or only l2 text is used as input (tlm-drop) 5 . an overview of the experimental settings is shown in figure 2 (see details in appendix a). note that the original xlm (conneau and lample, 2019) adopts a setting similar to the tlm-drop. in this experiment, we report the macro-average blimp accuracies across the test suites. translation pairs does not facilitate l2 acquisition: one notable point in switching between using l2 text with and without its parallel l1 text during training facilitates l2 acquisition: another notable point is that the tlm-drop was the most effective for acquiring linguistic knowledge in l2 for lms. 6 since we switch between using l1 text as input or not every epoch, there is a possibility that monolingual and bilingual training have a complementary positive effect. in model (tlm) settings first language para. drop fr de ru ja nopara 52.0 57.6 51.2 52.5 para ✓ 51.1 53.6 48.9 51.3 drop ✓ ✓ 58.0 61.1 52.8 56.2 ## experiments: @xmath0 we investigate how pretraining with l1 affects l2 grammar acquisition in lms. we exploratorily compare the linguistic generalization of lm trained in the settings with or without l1 pretraining. ## analysis: acquisition process this section sheds light on the process of l2 acquisition. we investigate how l2 knowledge acquisition progresses (section 5.1) and how original l1 knowledge changes during l2 acquisition (section 5.2). as for the l1 knowledge during l2 acquisition, there is a concern, for example, that lms exhibit catastrophic forgetting about their l1. ## related work cognitively-motivated analysis of neural lms: investigations into the ability of neural models in language acquisition began in the 1980s with the interest of whether language can be acquired with-out innate knowledge @xcite . the initial investigation was made with simple neural networks; after the development of neural nlp @xcite , the classical questions posed by cognitive science are currently revisited @xcite . a typical movement is a growing interest in the probing of neural lms' linguistic knowledge @xcite . in this context, our study analyzes l2 acquisition in neural lms, while existing studies have typically focused on l1 acquisition. language transfer in computational models: language transfer of nlp models is actively researched from both engineering and scientific perspectives. in the engineering context, to mitigate the english-centric focus of nlp techniques, models that can handle more languages have been developed @xcite @xcite . from the scientific perspective, the mechanism and linguistic properties of lms' language transfer have been explored @xcite @xcite , sometimes beyond the transfer between natural languages @xcite . one of the motivations of such analyses is to quantify the transferable universals behind (non-)languages. notably, simulation of l2 acquisition is also explored from pedagogical motivations @xcite . language transfer in humans: l2 acquisition/learning has long been studied in applied linguistics, psycholinguistics, and pedagogy fields @xcite @xcite . these fields articulated several hypotheses/theories on human language learning, e.g., input hypothesis @xcite . analyzing the lms' l2 acquisition in a more direct light with these hypotheses would be interesting for future work. ## conclusions we have investigated the l2 acquisition of lms, especially focusing on their grammatical knowledge in l1 and l2. specifically, we have trained bilingual lms under a similar scenario to the human l2 acquisition and then analyzed their cross-lingual transfer. our experiments have demonstrated that l1 pretraining promotes their linguistic generalization in l2, and there are interesting variations in l1 pretraining effects with respect to the l1 choice, training settings, and grammar items. the results have also implied that their l2 acquisition is not human-like in particular aspects.