SLABERT Talk Pretty One Day: Modeling Second Language Acquisition with BERT (2023)

Abstract: Second language acquisition (SLA) research has extensively studied cross-linguistic transfer, the influence of linguistic structure of a speaker’s native language [L1] on the successful acquisition of a foreign language [L2]. Effects of such transfer can be positive (facilitating acquisition) or negative (impeding acquisition). We find that NLP literature has not given enough attention to the phenomenon of negative transfer. To understand patterns of both positive and negative transfer between L1 and L2, we model sequential second language acquisition in LMs. Further, we build a Mutlilingual Age Ordered CHILDES (MAO-CHILDES)—a dataset consisting of 5 typologically diverse languages, i.e., German, French, Polish, Indonesian, and Japanese—to understand the degree to which native Child-Directed Speech (CDS) [L1] can help or conflict with English language acquisition [L2]. To examine the impact of native CDS, we use the TILT-based cross lingual transfer learning approach established by Papadimitriou and Jurafsky (2020) and find that, as in human SLA, language family distance predicts more negative transfer. Additionally, we find that conversational speech data shows greater facilitation for language acquisition than scripted speech data. Our findings call for further research using our novel Transformer-based SLA models and we would like to encourage it by releasing our code, data, and models.

## introduction cross-linguistic transfer can be described as the influence of native language [l1] properties on a speaker's linguistic performance in a new, foreign language @xcite . the interaction of the linguistic structure of a speaker's l1 with the successful acquisition of l2 results in what are termed as transfer effects. transfer effects appear in various aspects of linguistic performance, including vocabulary, pronunciation, and grammar @xcite . cross-linguistic transfer can be positive or negative in nature: positive transfer refers to the facilitating effects of one language in acquiring another (e.g., of spanish vocabulary in acquiring french) and negative transfer between the learner's native @xcite and target [l2] languages, producing errors. the greater the differences between two languages, the greater the negative effects. while cross-lingual transfer has received considerable attention in nlp research @xcite @xcite @xcite , most of this research has concentrated on practical implications such as the degree to which the right tokenizer can optimize cross-lingual transfer, and has not looked at the kind of sequential transfer relationships that arise in human second language acquisition. meanwhile, approaches like the test for inductive bias via language model transfer (tilt) @xcite focus on positive transfer with divergent pairs of training sets, such as midi music and spanish, to shed light on which kinds of data induce generalizable structural features that linguistic and non-linguistic data share. patterns of both positive and negative transfer between a given l1 and l2, however, can be a valuable source of information about general processes of second language acquisition and typological relationships between the languages in question @xcite . most cross-lingual models do not mimic how humans acquire language, and modeling the differences between first and second language acquisition is a particularly under-explored area. to engage with questions about second language acquisition using lms, we model sequential second language acquisition in order to look more closely at both positive and negative transfer effects that may occur during the acquisition of l2. using child-directed speech (cds) to create l1 training sets that are naturalistic, ecologically valid, and fine-tuned for language acquisition, we model the kind of cross-linguistic transfer effects that cause linguistic structure of the native l1 to influence l2 language acquisition in our novel second language acquisition bert (slabert) framework. the resulting models, when tested on the blimp (benchmark of linguistic minimal pairs for english) grammar test suite @xcite , show that l1 may not only facilitate l2 learning, but can also interfere. to the extent that interference is considered in nlp research, it is often understood simply as a failure of positive transfer in model training. we suggest, instead, that these results should be analyzed in terms of distinctive patterns of both negative and positive transfer, which can reveal not just the existence of generalizable features across datasets, but also finer-grained information about structural features of these languages and their accessibility to second language learners. ## related work our work is closely related to and in many ways builds on the work done by @xcite . they proposed that child-directed speech has greater potential than other kinds of linguistic data to provide the structure necessary for language acquisition, and released babyberta, a smaller sized roberta @xcite model designed to investigate the language acquisition ability of transformer-based language models (tlm) when given the same amount of data as children aged 1-6 get from their surroundings. they also released zorro, a grammar test suite, that is compatible with the small vocabulary of child-directed input. child-directed speech (cds) refers to the special register adopted by some adults, especially parents, when talking to young children @xcite . cds typically features higher fundamental pitch, exaggerated intonation, slower speech, and longer pauses than adult-directed speech (ads) @xcite . utterances in cds are usually wellformed grammatically, but are syntactically simpler than ads, often comprising single word utterances or short declaratives. adults often repeat words, phrases, and whole utterances in cds @xcite and make fewer errors @xcite than they do in ads. cds also tends to use a smaller and simplified vocabulary, especially with very young children @xcite . while the universality and necessity of cds for language acquisition is a matter of debate @xcite @xcite , it is likely that the features of cds are universally beneficial in language acquisition @xcite . nlp literature suggests that are certain benefits when models are trained on cds @xcite . studies from other fields suggest that the pitch contours, repetitiveness, fluency, and rhythms of cds make it easier for children to segment speech, acquire constructions, and understand language @xcite @xcite @xcite . many of these distinctive qualities of cds seem tailor-made for human language acquisition, which is why we use cds data as l1 in our slabert models. several recent studies confirm that the distinctive distributional features of cds influence the grammatical and lexical categories that children acquire. for instance, @xcite found that "frequent frames" in cds-commonly recurring co-occurance patterns of words in sentences-yield very accurate grammatical category information for both adults and children. similarly, @xcite found that patterns of frequent use and, importantly, reinforcement in cds-specific conversational exchanges were most predictive of the constructions children learn. together, these findings suggest that both token distribution and the distinctive conversational structure of cds provide useful reinforcement for acquisition. therefore, when training our l1 model, we pay attention to qualities of the training input such as the conversational structure. in second language acquisition (sla) research, patterns of negative transfer are a topic of much interest and have been considered a source of information both about what happens in second language learning and what it can reveal about the typological relationships between l1 and l2. for instance, @xcite show that closely analyzing data from children learning a second language reveals that some errors are due to l1 interference (negative transfer), while others arise from developmental cognitive strategies similar to those made during l1 acquisition (developmental errors). @xcite show a strong correlation between language similarities derived from the structure of english as second language (esl) texts and equivalent similarities obtained directly from the typological features of the native languages. this finding was then leveraged to recover native language typological similarity from esl texts and perform prediction of typological features in an unsupervised fashion with respect to the target languages, showing that structural transfer in esl texts can serve as valuable data about typological facts. the phenomenon of cross-linguistic transfer has received considerable attention in nlp research in the context of multilingual language models @xcite @xcite @xcite . our investigation is particularly inspired by papadimitriou and jurafsky (2020)'s test for inductive bias via language model transfer (tilt). this is a novel transfer mechanism where the model is initially pre-trained on training data @xcite . next, they freeze a part of the model and fine-tune the model on l2. finally, they test the resulting model on a test set of l2. we follow a similar approach to our model's second language acquisition. ## experimental setup we use babyberta @xcite to run all our experiments. babyberta is a smaller-sized roberta @xcite tuned to perform well on data of the size of ao-childes. however, we make additional changes to the vocabulary size of the model as we found that to improve the results of the model. the implementation details of the model can be found in appendix a.1. we follow the tilt approach introduced by papadimitriou and jurafsky (2020) to originally test the lstm-based @xcite lm's structure acquisition. their general approach is followed in the current study with a few notable changes (see figure 1 ). our approach comprises two stages: (1) train the model on l1 (cds language) (2) freeze all parameters except the word embeddings at the transfer stage of the experiment, and fine-tune the model on l2 (english ads). finally, the resulting model is tested on a test set of l2 for which we use the benchmark of linguistic minimal pairs (blimp) @xcite , a challenge set for evaluating the linguistic knowledge of the model on major grammatical phenomena in english. our study deviates from papadimitriou and jurafsky (2020) approach in three ways: (1) instead of using lstm-based lms we use transformer-based lms @xcite (2) they freeze all layers except the word embedding and linear layers between the lstm layers however, for simplicity we freeze all parameters except the word embeddings (3) while they report their findings based on lm perplexity scores, we use the blimp test suite to report how l1 structure (particularly, syntax and semantics) affects l2 acquisition in our transformer-based lms. there are two experiments for which we follow a different procedure than what is explained above: testing: we use the blimp grammar test suite to evaluate the linguistic knowledge of our model. blimp consists of 67 paradigms categorized into which is grammatically acceptable and the other not (refer to warstadt et al. ( 2020 ) for a detailed description of the test suite). ## conclusion in this paper, we explore how second language acquisition research and models of second language acquisition can contribute to questions in nlp about the learnability of grammar. drawing from the previous research on the unique role of child-directed speech (cds) in language acquisition, we investigate the potential of spontaneously generated cds to form a special source from which lms can acquire the structure necessary for first language acquisition. to test sequential second lan-guage acquisition in lms, we introduce slabert. the results from our experiments suggest that while positive transfer is a lot more common than negative transfer, negative transfer occurs in lms just like it occurs in english second language (esl) learners. we believe these novel findings call for further research on this front, and suggest that models like slabert can provide useful data for testing questions about both language acquisition and typological relationships through patterns of crosslinguistic transfer. to support this, we release our code, novel mao-childes corpus, and models.