Modeling Nonnative Sentence Processing with L 2 Language Models (2024)

Abstract: We study LMs pretrained sequentially on two languages (“L2LMs”) for modeling nonnative sentence processing. In particular, we pretrain GPT2 on 6 different first languages (L1s), followed by English as the second language (L2). We examine the effect of the choice of pretraining L1 on the model’s ability to predict human reading times, evaluating on English readers from a range of L1 backgrounds. Experimental results show that, while all of the LMs’ word surprisals improve prediction of L2 reading times, especially for human L1s distant from English, there is no reliable effect of the choice of L2LM’s L1. We also evaluate the learning trajectory of a monolingual English LM: for predicting L2 as opposed to L1 reading, it peaks much earlier and immediately falls off, possibly mirroring the difference in proficiency between the native and nonnative populations. Lastly, we provide examples of L2LMs’ surprisals, which could potentially generate hypotheses about human L2 reading.

## introduction it has been widely shown that one's first language (l1) affects both second language (l2) production (e.g., @xcite and processing (e.g., @xcite , a phenomenon called l1 transfer. in computational linguistics, most studies of lms as models of acquisition (e.g., @xcite have considered monolingual settings. to date, @xcite and @xcite seem to be among the rare exceptions that have investigated lms as models of second language acquisition (sla), which we refer to as l2 language models (l2lms). both of these studies establish that the typological distance between the model's first language and its second language (english) correlates with its english performance as measured by a morphosyntactic benchmark. however, these l2lms are yet to be tested against human l2 speakers. as one way of testing if the l1 of l2lms affects l2lms' performance on english in a humanlike manner, we study their sentence processing. sentence processing is widely used to study (l)lms' cognitive plausibility (e.g., @xcite @xcite @xcite . by comparing the lm surprisal @xcite to human reading time, we determine whether humans and models process given texts in a similar manner. studies in this area have investigated the impact of data size, model size, and model architecture, among other variables (see §2). in this study, we train autoregressive l2lms from scratch to investigate the following questions ( §3): (1) does the l1 effect on l2lms' l2 grammaticality discrimination, previously demonstrated for encoder-only models, extend to decoderonly models, namely gpt2? we hypothesize that, as in previous studies, the l2lms trained with l1s closer to the l2 (english) will perform better. ## methods to study the effect of l1 on l2lm grammaticality judgment (rq1) and sentence processing (rq2), we train l2lms on various l1s and a common l2 (english), while constraining their learning during the l2 training phase, mirroring the human sla. ## conclusion in this study of l2lms, we trained gpt2-style decoder-only lms on 6 l1s (arabic, chinese, english, japanese, portuguese, and spanish) and then on a common l2 (english), freezing the decoder blocks the l2 training phase, with all of the variables including data size and hyperparameters held constant. we replicate findings from previous literature that the linguistic distance between the l1 and l2 negatively correlates with the lm's performance on the l2 (as measured in blimp). our novel findings include that a monolingual english lm is most predictive of l1 english reading time at around 2.4b word tokens, and of l2 english reading time at around 800m-1.2b word tokens; that l2lms' sentence processing was not shown to correlate with that of human l2 speakers of english; and that the overall predictability of human l2 speakers' sentence processing largely depended on their l1. we also showed that, despite the lack of overall correlation between l2lms surprisals and l2 english speakers' reading times, qualitative examples could be conducive to generating hypotheses of l2 reading behaviors.