Testing the Predictions of Surprisal Theory in 11 Languages (2023)

Abstract: Surprisal theory posits that less-predictable words should take more time to process, with word predictability quantified as surprisal, i.e., negative log probability in context. While evidence supporting the predictions of surprisal theory has been replicated widely, much of it has focused on a very narrow slice of data: native English speakers reading English texts. Indeed, no comprehensive multilingual analysis exists. We address this gap in the current literature by investigating the relationship between surprisal and reading times in eleven different languages, distributed across five language families. Deriving estimates from language models trained on monolingual and multilingual corpora, we test three predictions associated with surprisal theory: (i) whether surprisal is predictive of reading times, (ii) whether expected surprisal, i.e., contextual entropy, is predictive of reading times, and (iii) whether the linking function between surprisal and reading times is linear. We find that all three predictions are borne out crosslinguistically. By focusing on a more diverse set of languages, we argue that these results offer the most robust link to date between information theory and incremental language processing across languages.

## introduction language processing is incremental and dynamic: when a reader encounters a word, they allocate a certain amount of time to process it before moving on to the next one. one influential theory for the mechanism underlying this process is surprisal theory @xcite , which states that the time required to successfully comprehend a word is based on its predictability. notably, predictability is often quantified as surprisal (negative log-probability given preceding context), from which the theory's name is derived. suprisal theory is supported, empirically, by a number of studies which have found that surprisal is strongly correlated with psychometric measurements in large naturalistic reading corpora @xcite @xcite @xcite @xcite . put differently, a word's surprisal is a strong correlate of its processing effort, operationalized as reading time. however, there is one serious limitation with most previous studies: while making general claims about human language processing, they predominantly investigate reading times in english. and, while a few studies have investigated surprisal effects in languages other than english, e.g., @xcite in dutch and @xcite in japanese, no systematic, crosslinguistic analysis has been performed. as multiple sentence processing phenomena exhibit significant crosslinguistic variation @xcite , the extent to which surprisal theory generalizes crosslinguistically is a nontrivial limitation of the current state of the literature. in addition, two recent contributions have posited several extensions to surprisal theorymost influentially, (a) that contextual entropy, i.e., expected surprisal, also correlates with reading times, and (b) that the relationship between surprisal and reading time is linear @xcite @xcite . regarding (a), @xcite and @xcite have argued for what may be considered an expanded version of surprisal theory where processing difficulty is still determined by surprisal, but where people's reading behavior is additionally sensitive to expected surprisal (contextual entropy). building off prior work that has investigated the role of entropy in language processing @xcite @xcite , these recent studies suggest that readers may allocate reading times in advance of encountering a word, based on their expectations of how difficult the word will be to process. @xcite , a number of studies have found evidence that the linking function between reading times and surprisal is linear @xcite @xcite . however, these results have been challenged recently, with different studies coming to different conclusions about the most appropriate linking function. in the past two years, for example, investigations have concluded that this function is sublinear @xcite , linear @xcite , and superlinear @xcite . here, we will use the term surprisal theory to refer to both the core hypothesis that reading times are correlated with surprisal, as well as the two extensions-(a) and (b)-described above. we address a gap in the current literature by investigating the predictions of surprisal theory, on eleven languages distributed across five language families. 1 we enumerate these three predictions as hypotheses below. ## psycholinguistic predictive power our behavior of interest is how long readers spend visually attending to a given word w t in its linguistic context, i.e., w t 's reading time. this quantity offers a window into the psychological processes that underlie language comprehension and is typically taken as a direct reflection of the word's processing difficulty @xcite . a word's reading time can be measured via multiple experimental modalities, including self-paced reading @xcite and the maze task @xcite . in this work, we focus on eye-tracking measurements. these measurements have high temporal resolution and exhibit smaller spillover effects than self-paced reading @xcite , where spillover is the effect of a word's properties on later words' reading behavior. following previous work investigating reading, we ask what factors associated with each word are helpful for predicting its reading times. in the following section, we use the following notation. with w, we denote a word taken from an alphabet σ. with @xmath0, we denote a string of words over the alphabet σ. we write w t for the word at index t in a string @xmath1. additionally, let @xmath2. with each word w t in a context @xmath3, we associate a real column vector of predictor variables x t that we believe may impact reading times. many of these predictors are attributes of w t itself, e.g., w t 's length. we use x t as predictors in a regression model f φ with parameters φ. the regression model is estimated to predict w t 's reading time from data. in symbols, we write that in order to contrast different theories of language processing, we compare regression models with different vectors of predictor variables x and with different architectures f φ , each of which is taken to instantiate a different hypothesis about what underlying factors determine reading times. we fit each regression model on a portion of our dataset and evaluate it by measuring the log-likelihood that it assigns to held-out data. models that lead to higher log-likelihood can be said to have better predictive power or psychological accuracy for human reading-and their associated theories are then taken to be better models of the underlying psycholinguistic processes @xcite . typically, for each experiment we will define a target regression model, which is trained to predict the reading times of individual words from a set of baseline predictors plus a predictor of interest (e.g., surprisal or contextual entropy). for a specific index t, we will refer to these predictors as our target predictors and denote them as x tgt t . we also define a baseline regression model that includes only the baseline predictors, which are a subvector of the target predictors, denoted as x base t for a specific t. we denote baseline and target regression models symbolically as f φ (• | x tgt t ) and f φ (• | x base t ), respectively. unless otherwise specified, the regression models that we use in this study are all linear. the choice to use linear linking functions, and whether this assumption is warranted, is addressed directly in section 5. in order to assess whether the target predictors have contributed to better predictive power, we will inspect the (average) by-word difference in log-likelihood assigned by the two regression models to a held-out dataset @xcite . following previous studies, we refer to this metric as the delta log-likelihood δ, which is defined, for a specific index t, as ## surprisal-rt linking function the regression models we have been using to assess δ have implicitly assumed a linear linking function between surprisal and reading time-a relationship that has been empirically verified in some previous studies in english @xcite @xcite . other recent studies, however, have questioned linearity, including @xcite and @xcite , who argue for a superlinear relationship, and @xcite , who argue for a sublinear relationship. in this section, we directly test the linear link hypothesis. we compare the δ of our linear regression models against regression models that can capture non-linear relationships. we present re-sults exclusively for gaze duration for the reasons discussed in section 3.1. ## conclusion this paper has presented the most comprehensive crosslinguistic evaluation of surprisal theory reported in the literature to date. using eyetracking data from controlled materials in eleven languages across five language families, we have tested three hypotheses: (i) the surprisal hypothesis (surprisal is predictive of reading times), (ii) the contextual entropy hypothesis (contextual entropy is predictive of reading times), and (iii) the linear link hypothesis (the relationship between surprisal and reading times is linear). we found exceptionally strong crosslinguistic stability in our results, with each prediction being borne out in every language tested. these results provide the most robust link between information-theoretic quantities and incremental processing. we now briefly explain the components of these regressions. s() sets up a spline-based smooth term between a predictor and response variable that can take on a wide variety of non-linear functional relationships. here, @xmath4. we choose @xmath5, appendix c. having 6 basis functions allows for five degrees of freedom, which enables the regression to fit non-linear yet still relatively simple curves. the other term, te(), sets up a tensor product smooth term, which can effectively capture non-linear interactions between two variables.