Reanalyzing L 2 Preposition Learning with B ayesian Mixed Effects and a Pretrained Language Model (2023)

Abstract: We use both Bayesian and neural models to dissect a data set of Chinese learners’ pre- and post-interventional responses to two tests measuring their understanding of English prepositions. The results mostly replicate previous findings from frequentist analyses and newly reveal crucial interactions between student ability, task type, and stimulus sentence. Given the sparsity of the data as well as high diversity among learners, the Bayesian method proves most useful; but we also see potential in using language model probabilities as predictors of grammaticality and learnability.

## introduction learning a second or third language is hard-not only for nlp models but also for humans! which linguistic properties and external factors make it so difficult? and how can we improve instruction and testing to help learners accomplish their goals? here we also ask a third question: how can we best apply different computational models to such behavioral experimental data in order to get intuitive and detailed answers to the first two questions in a practical and efficient way? for example, we are interested in whether language model (lm) probabilities might give a rough estimate of grammaticality and learning difficulty ( this work is in part a replication study of wong (2022), who, in addressing these questions about native chinese speakers' learning of english prepositions in context (see examples in while the theoretical advantages of bayesian over frequentist statistics, as well as the generally strong performance of neuro-distributional models, are often cited as justification for choosing one particular modeling method, both replication studies and side-by-side comparisons of such drastically different modeling strategies for linguistic analysis remain rare (with notable exceptions, e.g., @xcite . we contribute to this important development by • designing ( §4.1), fitting, and evaluating ( §5.1) a bayesian mixed effects model on @xcite data ( §3), considering more potential linguistic and human factors in preposition learning than previously and finding significant effects for several of them; • training ( §4.2) and evaluating an analogous multilayer perceptron (mlp) model and comparing it with the bayesian model in terms of both feature ablation and overall prediction accuracy of the outcome, i.e., whether a student will answer a test prompt correctly ( §5.2); • and probing a pretrained lm ( §4.3 and §5) for contextual probabilities of target prepositions in order to determine their correlation-and thus, practical usefulness-with human language learning. thus, we aim to both better explain l2 preposition learning and compare bayesian, frequentist, and neural approaches to doing so. ## original study and data wong (2022) measured students' pre-and postinterventional understanding of the english prepositions in, at, and over, particularly contrasting coglx/schematics-based instruction with different flavors of rule-based methods. to this end, intermediate learners of english (all university students) with first languages mandarin or cantonese took initial english language tests ('pretest') targeting different usages of prepositions. they were then taught with one of four methods (incl. one control group, who received instruction about definite and indefinite articles instead of prepositions), and subsequently tested two more times. there were two different tests: a grammaticality judgment test (gjt) to measure effects on language processing and a picture elicitation test (pet) to measure effects on production. while all preposition-focused training was found to enhance learners' understanding of prepositions compared to both the pretest and the control group, schematics-based mediation led to stronger learning results than any of the other methods, especially at the pet (fig. 1 ) and on spatial usages of prepositions (the interaction between instruction method and spatial usage is not shown in fig. 1 for brevity). these latter findings in particular support our hypothesis that in addition to external factors like task type and instruction method, learning difficulty may also be affected by inherent linguistic properties of the prepositions and their usages (just as, e.g., @xcite show for distributional properties of grammatical suffixes). in this work we take a second look at wong's data to directly address this possibility for preposition learning. ## models our main point of reference (or quasi-baseline) is wong's frequentist data analysis, which is summarized in §3. in this work, we newly consider the following different modeling strategies: we train a bayesian logistic model (blm, §4.1) as well as a small multilayer perceptron (mlp, §4.2) on the same data. with the blm we can define and interpret the precise structure of how individual features and their interactions affect the outcome. in contrast, the mlp utilizes nonlinear activation functions and multiple iterations/layers of computation, allowing it to pick up on complex interactions among input features without prior specification and thus to potentially achieve higher predictive accuracy, at the cost of interpretability. both the blm and mlp are implemented in python and pytorch, and are light-weight enough to be trained and run on a laptop cpu within several minutes for training and several seconds for inference. we also query a pretrained neural language model (lm, namely roberta; @xcite to obtain contextual probabilities for the stimulus sentences used in the grammaticality judgment test and add those probabilities to the blm and mlp's inputs ( §4.3). ## evaluation we first analyze the blm's learned latent coefficients ( §5.1). then we compare different versions of the blm and mlp w.r.t. their ability to predict unseen student responses using their estimated weighting of linguistic and external features as well as lm probabilities ( §5.2). finally, we manually inspect a small set of stimulus sentences with anomalous lm probabilities w.r.t. their intended grammaticality and observed judgments ( §5.3). ## conclusion much quantitative research in many areas of linguistics, including sla, has been relying on the frequentist method for a long time-and for good reasons: it enables strong conclusions about clear hypotheses, closely following the observed data. here we compared several alternative approaches to estimating a multitude of potential effects more holistically, namely via irt-inspired bayesian sparse models of explicit interactions among facts, neural classifiers of student responses and feature ablation, as well as contextual probabilities of the experimental stimuli obtained from a pretrained language model ( §4). overall, we were able to replicate previous frequentist findings regarding the difficulty of acquiring the preposition system in english as a second language and the benefits of concept-based instruction ( §5.1). our computational analysis emphasized the increased flexibility and occasionally stronger effect size estimates of irt and bayesian models, as well as their natural interpretability compared to neural models with equal predictive power. we also found novel interactions among task and subtask type, student individual differences, preposition cue and lm contextualization ( §5), and discussed them in the broader contexts of both nlp and sla, hoping to build bridges between the two research communities ( §6). as a final takeaway for both fields, the differences between the lm's and students' overall tendencies to accept or reject stimuli ( §5.3 and fig. 4 right) could potentially be exploited in both directions: the aggregate distributional grammatical knowledge of an lm could be used to teach students the most accepted usages of prepositions and other function words across a large population of speakers (i.e., improve their specificity), while lms could learn to be more creative and to utilize humans' intuitive cross-lingual meaning mappings by learning from second-language learner data.