On the Role of Context in Reading Time Prediction (2024)

Abstract: We present a new perspective on how readers integrate context during real-time language comprehension. Our proposals build on surprisal theory, which posits that the processing effort of a linguistic unit (e.g., a word) is an affine function of its in-context information content. We first observe that surprisal is only one out of many potential ways that a contextual predictor can be derived from a language model. Another one is the pointwise mutual information (PMI) between a unit and its context, which turns out to yield the same predictive power as surprisal when controlling for unigram frequency. Moreover, both PMI and surprisal are correlated with frequency. This means that neither PMI nor surprisal contains information about context alone. In response to this, we propose a technique where we project surprisal onto the orthogonal complement of frequency, yielding a new contextual predictor that is uncorrelated with frequency. Our experiments show that the proportion of variance in reading times explained by context is a lot smaller when context is represented by the orthogonalized predictor. From an interpretability standpoint, this indicates that previous studies may have overstated the role that context has in predicting reading times.

## introduction surprisal theory @xcite posits that the amount of effort it takes to process a linguistic unit is an affine function of its in-context information content, i.e., its surprisal. numerous studies have found empirical support for surprisal theory across different reading measurement methods, languages, and language models @xcite @xcite @xcite , particularly when controlling for additional effects such as frequency. in this work, we take a critical look at surprisal theory as an adequate explanation for the role of context in reading time prediction, starting from a simple observation: surprisal is but one quantity that can be derived from a language model to represent the effect of context @xcite . we first show that, as an alternative to surprisal, one could take an association-based view on real-time language comprehension and model it as a function of the pointwise mutual information (pmi) between a unit and its context. because pmi, surprisal, and frequency are collinear, all linear models with just two of these covariates are equivalent in terms of their predictive power. this simple identity therefore implies that all empirical validation of surprisal theory based on linear regression modeling also lends support for an association-based theory of language processing. this raises the question of whether there is a more suitable way to estimate the effect that context has on reading time. we argue that, given that frequency is known to play an important role in processing effort @xcite @xcite , a more interesting construct to analyze should be what context contributes beyond what is already captured by frequency. to obtain a predictor that represents just that, we propose a technique where we project surprisal onto the orthogonal complement of frequency, ensuring that they are uncorrelated. this process effectively disentangles the contextual and non-contextual information into different covariates in our regressions and closely resembles residualization. 1 to test whether the choice of contextual predictor matters empirically, we measure how much the variance in reading times explained by the contextual predictor changes when substituting surprisal for the orthogonalized context predictor. we find that our proposed predictor results in much smaller explained variance. our results suggest that empirical work on surprisal theory has overestimated the effect that context has on reading times. ## predictive models of reading behavior we seek to model the cognitive processing difficulty of a unit u, e.g., a word, drawn from an alphabet σ. additionally, we augment σ to include a unique eos ̸ ∈ σ symbol which indicates the end of an utterance; we further define σ @xmath0. let σ * be the set of all strings over the alphabet σ; we write @xmath1, u t for the t th unit in u, |u| for the number of units in u, and uv for the concatenation of u with another string v. given a string cu, we are interested in how u's processing effort is shaped by its context of preceding units c. a common psychometric proxy for the cognitive processing difficulty of u is the time it takes a human to read u, typically, as measured in an eyetracking study @xcite . in general terms, we are interested in empirically assessing some theory of cognitive processing difficulty, which can be thought of as a collection of unit-level properties that are implicated in determining processing effort as measured by reading times. the most common type of evidence adduced to support such theories comes from (generalized) linear modeling. we define a predictor function as a function of type x : σ * × σ → r d , i.e., a function that maps a context-unit pair to a d-dimensional real vector. we model the reading time measurements as a linear model f β conditioned on x(c, u), i.e., r(c, u) ∼ f β ( • | x(c, u)) where β ∈ r d is a real-valued parameter vector. a model whose expected value, r(c, u) = β ⊺ x(c, u), achieves high likelihood on held-out data lends empirical support to the theory that the factors measured by the predictors in x underlie the process of reading. ## disentangling the effect of context as there is a large and established body of work showing that frequency plays a major role in explaining the effort it takes to process words (see, e.g., @xcite , we argue that the interest of surprisal theory lies in understanding what additional effect there is of contextual information beyond frequency. the exposition above implies that neither surprisal nor pmi should receive special status as a measure of the effect of context. moreover, both surprisal and pmi are correlated with frequency foot_3 and all three are collinear. we now present a simple technique to decorrelate frequency from surprisal, resulting in a new predictor that is engineered to be disentangled from frequency. importantly, our technique attributes the shared effect of frequency and surprisal on reading time to frequency, and then creates a new predictor which represents the effect of surprisal that is not shared with frequency. we argue that this new predictor is more relevant to study than either surprisal or pmi since it represents only the effect of context. our technical exposition starts with an underlying probability space (σ * ×σ, p(σ * ×σ), π h • p h ). next, consider the following random variables under this probability space: i h encoding the distribution over surprisals ι h (c, u) of the next unit given a context, m h encoding the distribution over pmis µ h (c, u) between the next unit and a context, and y h encoding the distribution over frequencies υ h (u) of a unit. note that i h , m h and y h are realvalued random variables and that y h is constant in c. they are elements of a hilbert space h over r containing all random variables under the above probability space that have finite second moment @xcite . the inner product on h is given by app. c.1 provides further details on why h is indeed a hilbert space over r with the above inner product. with h being a hilbert space, we can take projections on h. taking the projection of i h onto the orthogonal complement of y h we get projecting in this manner results in an orthogonalization in the sense that ⟨y h , proj y ⊥ h (i h )⟩ = 0, as a consequence of the hilbert projection theorem @xcite . see app. c.2 for a proof. if the expected values of at least one of the random variables x and y is 0, which can be achieved by a simple mean-centering transformation, then thus, if y h and i h are mean-centered the covariance between y h and proj y ⊥ h (i h ) will be 0, i.e., they will be decorrelated. the random variable proj y ⊥ h (i h ) constitutes a new predictor variable, which we term orthogonalized surprisal. in words, orthogonalized surprisal represents the effect of context and is disentangled from frequency. foot_4 ## variance explained by context we now seek an empirical understanding of how the new orthogonalized surprisal predictor influences the importance attributed to context in reading time prediction. in addition to the experiment presented here, we also compared the predictive power for nonlinear models across different predictors. those experiments are discussed in app. f. ## conclusion this article discusses predictors that capture how the processing effort of a unit is shaped by its context. we made the observation that there exist alternatives to the widely used surprisal predictor. surprisal is correlated with non-contextual frequency, so we provided a technique to disentangle contextual and non-contextual information in language models. in so doing, we found that the effect that context has on reading times appears to be small in comparison to non-contextual frequency.