The Effects of Surprisal across Languages: Results from Native and Non-native Reading (2022)

Abstract: It is well known that the surprisal of an upcoming word, as estimated by language models, is a solid predictor of reading times (Smith and Levy, 2013). However, most of the studies that support this view are based on English and few other Germanic languages, leaving an open question as to the cross-lingual generalizability of such findings. Moreover, they tend to consider only the best-performing eye-tracking measure, which might conflate the effects of predictive and integrative processing. Furthermore, it is not clear whether prediction plays a role in non-native language processing in bilingual individuals (Grüter et al., 2014). We approach these problems at large scale, extracting surprisal estimates from mBERT, and assessing their psychometric predictive power on the MECO corpus, a cross-linguistic dataset of eye movement behavior in reading (Siegelman et al., 2022; Kuperman et al., 2020). We show that surprisal is a strong predictor of reading times across languages and fixation measurements, and that its effects in L2 are weaker with respect to L1.

## introduction context-dependent predictive processes have been proposed as a core component of the human cognitive system @xcite . in the language processing literature, a clear picture that is progressively emerging is that speakers spontaneously pre-activate the upcoming lexical material before they encounter it @xcite @xcite . this pre-allocation of resources to predictable material is evidenced by the fact that unpredictable words are a major cause of processing costs, as measured through selfpaced reading times @xcite , eye movements (ehrlich and @xcite and pupil size @xcite in reading, and eeg responses @xcite . the role of prediction in language processing was, in particular, characterized via computational modeling, with the information-theoretic notion of surprisal being extended to psycholinguistics @xcite . surprisal quantitatively captures how unpredictable a word is in terms of the negative logarithm of the probability of a word conditioned by the preceding sentence context (1). in this perspective, surprisal acts as a linking function between cognitive effort and predictability @xcite , but see @xcite , where the former is measured empirically, and the latter is estimated probabilistically. @xcite demonstrated that the surprisal of a word given the previous context is mathematically equivalent to the kullback-leibler divergence (i.e. relative entropy) between probability distributions foot_0 . under this view, surprisal effects can therefore be interpreted as the cognitive costs associated to a shift between probability distributions. computational linguistics has proven itself very useful to derive word probability estimates @xcite @xcite , and the psychometric predictive power of a language model -i.e., how well it can account for human processing times -is a linear function of that model's quality, measured as its perplexity @xcite . computational studies on prediction in sentence processing have the indisputable merit of testing the effects of predictability at large scale and in the context of naturalistic reading. however, if compared to psycholinguistic studies on prediction, they generally focus on: i gaze duration. differently from psycholinguistic research @xcite , computational studies tend to consider only the eye-tracking measure that is typically best fitted by surprisal estimates, namely gaze duration @xcite @xcite , ignoring other cognitively relevant eyetracking metrics. ii germanic languages. a vast body of findings corroborates the effects of lexical prediction in english @xcite @xcite @xcite @xcite , dutch @xcite and german @xcite ; however, evidence from other language families is far more limited (although see @xcite . iii l1. within the computational framework, most of the studies reported insofar targeted sentence processing in the dominant languages (but see @xcite @xcite , while the psycholinguistic community is witnessing an increasing interest in predictive processing in l2 @xcite @xcite @xcite . we argue that these three limitations might undermine both the internal and the external validity of the results. first (i), only considering the best-performing eye-tracking measure does not provide any insight as to when such predictability effects take place during natural reading. an analysis of the time range where predictability effects can be detected is however crucial to disentangle between predictive and integrative processes @xcite . indeed, a higher processing cost induced by an unpredictable word might not be due to anticipatory processes, but also to a difficulty in integrating the unpredictable word in the phrasal context. while early measurements such as first fixation duration are thought to reflect lexical or pre-lexical processes (and thus a genuine effect of predictability; @xcite , gaze duration can be considered as a "midmeasure" @xcite , and thus it is not sufficient to disentangle between integrative and predictive processing. second (ii), some of the results that were obtained in english within the framework of surprisal theory were not replicated in other languages. for instance, @xcite have shown that the negative relationship between a language model's perplexity and its psychometric accuracy does not hold for the japanese language. hence, the rather limited typological variability in the language samples considered leaves an open question as to whether prediction itself should be considered as a core processing mechanism that generalizes across languages. third (iii) , the study of predictive processing in non-native reading is of crucial relevance since more than half of the global population is bilingual @xcite . the role of anticipation in bilingual individuals is attracting growing interest in second language acquisition studies, and large-scale data-driven approaches might shed light on a complex picture currently characterized by little consensus. the reduced ability to generate expectations hypothesis @xcite proposes that even highly proficient l2 speakers differ from native speakers in their abilities to anticipate the upcoming linguistic material. however, the results supporting this theory have been questioned @xcite ; they are generally derived from offline tasks in small-scale studies @xcite , and restricted to circumscribed linguistic phenomena (such as gender information in determiners, see @xcite . instead, it would be desirable to test the effects of word prediction in l2 when reading naturalistic, contextualized texts (see for instance @xcite , as opposed to artificially constructed experimental materials, presented out of context and repeated many times. @xcite have overcome these limitations by testing the effects of predictability in l2 at scale. they reported a larger effect of surprisal in non-native reading, which is at odds with the psycholinguistic evidence reported before, and difficult to explain. as mentioned by the authors, context-contingent expectations are statistically demanding to compute, and it is not clear why the effects of such a complex processing mechanism should be stronger in l2 than in l1. in the present study we address these limita-tions in the literature by considering different eyetracking measurements, including early fixation measurements that are expected to reflect predictive processes (i); extending our sample to 12 diverse languages, belonging to five language families and written in five different scripts (ii); and comparing the effect of prediction in l1 and l2 (iii). ## results to test whether the effects of surprisal are similar in their extent across l1 and l2, we concatenate the meco-l1 and meco-l2 dataframes, dummycode whether each trial is recorded in l1 or l2, and test for an interaction between language dominance and surprisal. the surprisal × language interaction is a significant predictor of reading times across all the fixation measurements we analyzed (ff: @xmath0.0184, @xmath1.626, @xmath2.0001, see figure 1a ,; gd: @xmath3.0104, @xmath4.4640, @xmath5.0005, 1b; tt: @xmath6.01756, @xmath7.723, @xmath8.0001, 1c). these results indicate that the surprisal effect in l1 is larger than in l2 across all three fixation measurements, since the slope for surprisal is consistently steeper in l1 (see figure 1 for a graphical depiction of the interactions). additionally, we also report the results of the interaction between frequency and language dominance. this interaction is significant when considering ff ( @xmath9.0594, @xmath10.4350, @xmath11.0001, 1d) and tt ( b -0.0148, @xmath12.7970, @xmath13.001, 1e; although from a graphical inspection it is clear that the largest effect is found in the case of ff); however, it does not reach statistical significance in the case of gd ( @xmath14.0050, @xmath15.306, @xmath16.1915, 1f). notably, in this case the direction of the interactions is reversed, with steeper slopes in l2 than l1. ## discussion in this study, we show that prediction is a widespread processing mechanism that can be detected across a variety of languages and language families; while we fail to report significant effects in some of the languages taken individually, the consistent direction of the effects and the results of the large linear models including multiple languages strongly support the idea that natural reading involves the active anticipation of the following linguistic material. this finding complements previous results in computational psycholinguistics, showing that predictability effects are not confined to english and the few other germanic languages which are usually considered in the surprisal literature. crucially, surprisal exerts a cross-lingual effect even in ff, an eye-tracking metric that is thought to reflect the earliest stages of word processing. this supports our claim that the effects of surprisal that we report are the result of truly predictive processes, and do not reflect a difficulty in integrating unpredictable words in the phrasal context. our results also highlight some interesting differences in the reading behaviour of native and non-native speakers: the role of predictive processing in the non-dominant language appears to be significantly reduced when compared with the dominant one. on the other hand, eye movements in l2 are more strongly impacted by contextindependent expectations, as operationalized with unigram word frequencies. this is particularly evident in the earliest fixation measure considered, namely ff. the early onset of this l1-l2 dissociation -which would not have been detected if considering only gd -suggests a potential role of non-contextual prediction in l2: while l1 speakers might rely more strongly on the phrasal context to predict the next word, l2 speakers might base their expectations primarily on prior probabilities of the lexical material. context-based predictions are harder to estimate in real-time reading than their context-independent counterparts; hence, language experience might influence the extent to which a speaker relies on simple frequency estimates or context-sensitive predictions to calibrate her/his expectations on the following word @xcite . ## limitations and further directions in this study, we considered l2 processing as a homogeneous cognitive phenomenon. however, it has been suggested that l2 proficiency might modulate some differences between native and non-native reading, including predictive processing @xcite @xcite . we leave for future research an assessment of whether the difference in contextual and non-contextual prediction is better explained by a categorical distinction between l1 and l2, or rather a graded account of language proficiency.