Analyzing Wrap-Up Effects through an Information-Theoretic Lens (2022)

Abstract: Numerous analyses of reading time (RT) data have been undertaken in the effort to learn more about the internal processes that occur during reading comprehension. However, data measured on words at the end of a sentence–or even clause–is often omitted due to the confounding factors introduced by so-called “wrap-up effects,” which manifests as a skewed distribution of RTs for these words. Consequently, the understanding of the cognitive processes that might be involved in these effects is limited. In this work, we attempt to learn more about these processes by looking for the existence–or absence–of a link between wrap-up effects and information theoretic quantities, such as word and context information content. We find that the information distribution of prior context is often predictive of sentence- and clause-final RTs (while not of sentence-medial RTs), which lends support to several prior hypotheses about the processes involved in wrap-up effects.

## introduction reading puts the unfolding of linguistic input in the hands-or, really, the eyes-of the reader. consequently, it presents a unique opportunity to gain a better understanding of how humans comprehend written language. the rate at which humans choose to read text (and process its information) should be determined by their goal of understanding it. ergo, examining where a reader spends their time should help us to understand the nature of language comprehension processes themselves. indeed, studies analyzing reading times have been employed to explore a number of psycholinguistic theories (e.g., @xcite @xcite . one behavior revealed by such studies is the tendency for humans to spend more time foot_0 on the last word of a sentence or clause. while the existence of such wrap-up effects is well-known @xcite @xcite , the cognitive processes giving rise to them are still not fully understood. this is likely (at least in part) due to the dearth of analyses targeting naturalistic sentence-final reading behavior. first, most studies of online processing omit data from these words to explicitly control for the confounding factors wrap-up effects introduce (e.g., @xcite . second, the few studies on wrap-up effects rely on small datasets, none of which analyze naturalistic text @xcite @xcite . this work addresses this gap, using several large corpora of reading time data. specifically, we study whether informationtheoretic concepts (such as surprisal) provide insights into the cognitive processes that occur at a sentence's boundary. notedly, informationtheoretic approaches have been proven effective for analyzing sentence-medial reading time behavior. we follow the long line of work that has connected information-theoretic measures and psychometric data @xcite @xcite , employing similar methods to build models of sentence-and clause-final rts. using surprisal estimates from state-of-the-art language models, we search for a link between wrapup effects and the information content within a sentence. we find that the distribution of surprisals of prior context is often predictive of sentence-and clause-final reading times (rts), while not adding significant predictive power to models of sentencemedial rts. this result suggests that the nature of cognitive processes involved during the reading of these boundary words may indeed be different than those at other positions. such findings lend support to several prior hypotheses regarding which processes may underlie wrap-up effects (e.g., the resolution of prior ambiguities) while providing evidence against other speculations (e.g., that the time spent at sentence boundaries can be quantified with a constant factor, independent of the processing difficulty of the text itself). ## the process of reading decades of research on reading behavior have improved our understanding of the cognitive processes involved in reading comprehension @xcite . here, we will briefly describe overarching themes that are relevant for understanding wrap-up effects. ## language models as predictors of psychometric data formally, a language model q is a probability distribution over natural language sentences, i.e., over v * for an alphabet v of linguistic units (typically words). in the case when q is locally normalized, which is the predominant case for today's neural language models, q is defined as the product of conditional probability distributions, i.e., for @xmath0, we have q(w the symbol eos is a special end-ofstring token not in v. consequently, we can use q to estimate the probability of individual words in context. the model parameters are typically estimated by minimizing the negative log-likelihood of a corpus of natural language strings c, i.e., minimizing l(q) = -@xmath1. one widely embraced technique in informationtheoretic psycholinguistics is the use of these language models to estimate the probabilities required for computing surprisal @xcite @xcite . it has even been observed that a language model's perplexity foot_3 correlates negatively with the psychometric predictive power provided by its surprisal estimates @xcite @xcite . if these language models keep improving at their current fast pace @xcite , exciting new results in computational psycholinguistics may follow, connecting reading behavior to the statistics of natural language. predicting reading times. in the computational psycholinguistics literature, the rt-surprisal relationship is typically studied using predictive models: rts are predicted using surprisal estimates (along with other attributes such as number of characters) for the current word. the predictive power of these models, together with the structure of the model itself (which defines a specific relationship between rts and surprisal), is then used as evidence of the studied effect. while this paradigm is successful in modeling sentence-medial rts @xcite @xcite , its effectiveness for modeling sentence-and clause-final times is largely unknown due to the omission of this data from the majority of rt analyses. a priori, we might expect per-word surprisal to be a similarly powerful predictor of sentence and clause-final rts. foot_4 yet in fig. 1 , we see that when our baseline linear model (described more precisely in §4) is fit to sentence-medial rts, the residuals for predictions of clause-final rts appear to be neither normally distributed nor centered around 0. further, these trends appear to be different for eyetracking and spr data, where the latter are skewed towards lower values for all datasets. 6 these results provide further confirmation that clause-final data does not adhere to the same relationship with rt as sentence-medial data, a phenomenon that may perhaps be accounted for by additional factors at play in the comprehension of clause-final words. thus, we ask whether taking into account information from the entire prior context can give us a better model of these clause-final rts. to this end, we operationalize the information content inf in text w (of length t ) as: foot_6 ## conclusion we attempt to shed light on the nature of wrap-up effects by exploring the relationship between clause-final rts and information-theoretic attributes of text. we find that operationalizations of the information contained in the preceding context lead to better predictions of these rts, while not adding significant predictive power for sentence-medial rts. this suggests that information-theoretic attributes of text can shed light on the cognitive processes happening during the comprehension of clause-final words. further, these processes may indeed be different in nature than those required for sentence-medial words. in short, our results provide evidence (either in support or against) about several theories of the nature of wrap-up processes.