The Linearity of the Effect of Surprisal on Reading Times across Languages (2023)

Abstract: In psycholinguistics, surprisal theory posits that the amount of online processing effort expended by a human comprehender per word positively correlates with the surprisal of that word given its preceding context. In addition to this overall correlation, more importantly, the specific quantitative form taken by the processing effort as a function of surprisal offers insights into the underlying cognitive mechanisms of language processing. Focusing on English, previous studies have looked into the linearity of surprisal on reading times. Here, we extend the investigation by examining eyetracking corpora of seven languages: Danish, Dutch, English, German, Japanese, Mandarin, and Russian. We find evidence for superlinearity in some languages, but the results are highly sensitive to which language model is used to estimate surprisal.

## introduction a great deal of insight into human language processing can be gleaned by studying the word-byword processing difficulty experienced by a comprehender who is listening to or reading language. consequently, the timecourse of incremental language processing difficulty has formed one of the major objects of study in psycholinguistics (sedivy, 2014, ch. 8). here we take up the question of the functional form of the effect of surprisal (the negative log probability of each word given preceding context) on word-by-word processing difficulty, as indexed by reading time. whereas previous empirical work has addressed this question in english, we study the question in the existing reading time databases of danish, dutch, english, german, japanese, mandarin chinese, and russian. to preview the results, we find consistent evidence for superlinearity for german and russian; for all other languages except mandarin, we find a linear effect of surprisal, but the analysis is inconclusive about the presence of additional superlinear effects and the result is highly sensitive to which language model is used to estimate surprisal. ## background in the field of psycholinguistics, surprisal theory @xcite holds that processing difficulty is information-theoretic in nature. letting w t be the word that a comprehender is perceiving at time t, and @xmath0, the surprisal s t measures the amount of information contained in w t in context: in general, surprisal theory holds that the amount of processing difficulty (for our purposes, the reading time, or rt) experienced by a comprehender for word w t in context @xmath1, where f is a monotonically increasing linking function converting bits of information into milliseconds of rt. surprisal theory is well-supported empirically by findings of a correlation between surprisal and rt, and the ability of surprisal to explain rt effects previously attributed to other processing mechanisms @xcite , although it may not explain the full range of human processing difficulty @xcite . the correlation between surprisal and rt gets stronger when probabilities are calculated using lower-perplexity language models @xcite @xcite , although this relationship may not be universal across languages @xcite , and breaks down for recent large models @xcite , whose ability to predict words is super-human @xcite . the goal of this work is to empirically investigate the form of the linking function f across languages. although early work did not commit to a specific form for the linking function f (hale, language corpus ## results figure 1 shows average reading time as a function of surprisal, along with linear fits, using the mgpt surprisal estimates. a strong positive relationship always obtains. visually, the effect appears to be largely linear in the larger corpora of danish, dutch, and english, but the shape of the effect is harder to discern for the the smaller corpora (german, japanese, mandarin, and russian). regression results for both surprisal models are schematized in the full regression code and results are available at https://github.com/weijiexu-charlie/ linearity-of-surprisal-on-rt . ## concurrent work concurrent to our work, @xcite have examined the linearity of the effect of surprisal on rt in 11 languages of the meco eyetracking corpus @xcite , finding consistent evidence for linearity and a lack of evidence for superlinearity across all languages studied. 4 our work is complementary with this other work, differing in datasets, languages considered, and statistical methodology. regarding data, @xcite use the meco corpus, consisting of parallel translated text, which is more similar in size to the 'sentence corpora' that we use for german, mandarin, and russian. in contrast, we use larger corpora of naturalistic text for danish, dutch, english, and japanese. we also investigate reading times in japanese and mandarin, which are not present in @xcite . reading times in these languages are especially interesting because they are written using orthographies that do not mark word boundaries; thus reading in these languages involves some arguably different cognitive processes than the other languages, as word/morpheme segmentation must be inferred from the text. our work also differs from @xcite in the statistical methodology. following smith and @xcite and @xcite , @xcite compare fits of linear and non-linear generalized additive models when predicting reading times from surprisal after controlling for word length and frequency, with a fixed spillover window across languages. in contrast, we adopt a simpler approach: we compare a mixed-effects linear model against a quadratic model, with a variable spillover window across languages. resolving the apparent tension between our results, which join @xcite in finding possible evidence for superlinearity, against the results suggesting pure linearity @xcite , for example) should be a focus in future research. ## conclusion we conclude that the effect of surprisal on reading times across languages may be slightly superlinear, but the evidence for this superlinearity is unstable and depends on the language model used. determining the functional form of the effect in further languages will require reading time corpora of large amounts of naturalistic text and strong language models.