Generalized Measures of Anticipation and Responsivity in Online Language Processing (2024)

Abstract: We introduce a generalization of classic information-theoretic measures of predictive uncertainty in online language processing, based on the simulation of expected continuations of incremental linguistic contexts. Our framework provides a formal definition of anticipatory and responsive measures, and it equips experimenters with the tools to define new, more expressive measures beyond standard next-symbol entropy and surprisal. While extracting these standard quantities from language models is convenient, we demonstrate that using Monte Carlo simulation to estimate alternative responsive and anticipatory measures pays off empirically: New special cases of our generalized formula exhibit enhanced predictive power compared to surprisal for human cloze completion probability as well as ELAN, LAN, and N400 amplitudes, and greater complementarity with surprisal in predicting reading times.

## introduction the prediction of upcoming linguistic units is posited to play a key role in human language comprehension @xcite @xcite . one fruitful method of operationalizing human uncertainty over predictions is through information-theoretic measures. because human predictive mechanisms leave behavioral and neural traces that are observable during reading and listening @xcite @xcite , the most common method of vetting an informationtheoretic measure of predictive uncertainty is by examining its relationship with such traces. beyond simply yielding good correlates, informationtheoretic measures often provide insight into the human prediction mechanism, and they are thus central to much cognitive and neurobiological research on human language processing @xcite @xcite . in the domain of sentence processing, there are two commonly deployed information-theoretic measures of predictive uncertainty. both assume that the comprehender implicitly maintains a probability distribution over upcoming sequences of linguistic units. the first, and most prominent, is the surprisal of a unit given its preceding context @xcite , while the other is entropy @xcite . in broad strokes, surprisal tells us how likely the next unit is in the given context and is a good example of a responsive measure, i.e., a measure that quantifies a response to the next unit. in contrast to surprisal, entropy is solely a function of the context, as it tells us the uncertainty over the range of possible upcoming linguistic units. thus, entropy is an example of an anticipatory measure, i.e., a measure that anticipates a response to the next unit without knowing its identity. in the specific case of next-symbol entropy @xcite , it is the expected value of the next-symbol surprisal so it comes with a natural interpretation of the expected response. estimates of surprisal and entropy based on neural language models have demonstrated significant predictive capacity for a wide variety of neural and behavioral data collected using self-paced and eye-tracked reading experiments @xcite , as well as eeg @xcite , fmri @xcite , and ecog imaging @xcite , along with explicit grammaticality and acceptability ratings @xcite . despite surprisal and entropy's empirical success, there is increasing interest in defining and evaluating alternative measures. examples include measures designed to disentangle different dimensions-e.g., lexical versus syntactic-of uncertainty @xcite @xcite or to quantify uncertainty over spans larger than a single unit (aurnhammer @xcite . while estimating surprisal and entropy from neu-ral language models is convenient, experimenters should chart new territory by designing their own measures, rethinking and enhancing established information-theoretic quantities to capture overlooked aspects of online language processing. in this paper, we introduce a new framework, termed generalized surprisal, for responsive and anticipatory models of online language processing. this framework encompasses existing informationtheoretic measures as special cases and offers a new method to develop novel ones. we begin by deriving a generalization of surprisal, demonstrating that it corresponds to an expectation over continuations of a linguistic context ( §2). we then show how many existing measures can be seen as special cases of our generalized formula, and we propose new special cases, such as sequence-level entropy and next-symbol information value ( §3). because some special cases cannot be calculated in closed form, we must rely on a sampling procedure. this introduces a trade-off between runtime and variance, which we analyze empirically in §5. finally, we evaluate all special cases as predictors of neural and behavioral data collected in experiments with human participants ( §6). we present several new findings, including: (1) contextual probability predicts human cloze completions better than surprisal, while surprisal is a better predictor of human predictability ratings; ## generalized surprisal this section introduces our framework. first, we establish some key notation and definitions. then, we present a decomposition of surprisal, which motivates our definition of generalized surprisal. ## special cases of generalized surprisal in this section, we introduce concrete special cases of generalized surprisal (eq. ( 4 )), which we evaluate as predictors of human behavior and neural activity recorded during online language processing. some of these have been previously used to predict such psycholinguistic data, while others are new. all special cases are designed by varying the three core components of our framework-anticipation vs. responsivity, scoring function, and warping function-and are meant to exemplify how different hypotheses about online language processing can be instantiated as generalized surprisal models. ## experimental setup dataset. we use the aligned dataset @xcite , which consists of @xmath0. we include most of these in our experiments: cloze completions (probability and entropy), predictability ratings, event-related brain potentials (elan, lan, n400, p600, epnp, pnp), eye-tracked reading times (first-fixation time, first-pass time, right-bounded time), and self-paced reading times. details on these measurements are given in app. a.1. each target-context pair (w (m) , c (m) ), termed a stimulus, is associated with a real-valued measurement ψ(w (m) , c (m) ), termed datum, which is an aggregation of persubject measurements for that stimulus. 5 in our experiments, we will compute generalized surprisal { γ p (w (m) , c (m) )} m @xmath1, and evaluate it as a predictor for the corresponding data {ψ(w (m) , c (m) )} m @xmath2. the contexts c are strings ranging from 5 to 14 words and targets w are strings corresponding to a single word. 6 language models. we obtain generalized surprisal estimates from gpt-2 small @xcite and gpt-neo 125m @xcite . prior work has shown that, despite exhibiting higher test perplexity, these two models have better psycholinguistic predictive power than larger ones @xcite . furthermore, their smaller size incurs a lower computa- sample size n figure 1 : coefficient of variation (top), correlation between resamples (center), and runtimes (bottom) for samplingbased measures across the stimuli in the aligned dataset. confidence intervals (95%) are too narrow to be visible; the horizontal axis is in log scale. the average runtime for the exact metrics (surprisal, probability, expected next-symbol surprisal, and expected next-symbol probability) is 0.002 seconds. distance function. the information value models introduced in §3.1 rely on a function d c to quantify the distance between two strings v and w in the context of c. here, we use mean-pooled noncontextual (i.e., layer 0) representation from gpt-2 small and gpt-neo 125m as column vectors r(v), r(w) ∈ r d , where d is the dimensionality of the representations, and calculate the cosine dis- there exist several other choices for distance and representation functions; see @xcite and @xcite for some examples. ## empirical analysis of measures we begin our experiments with an empirical analysis of monte carlo estimation for the measures presented in §3. as alluded to earlier, samplingbased measures introduce variance that does not arise for measures like surprisal, which can be computed in closed form. this variance can be reduced by increasing the sample size ( §5.1), but that incurs additional costs in terms of runtime ( §5.2). in this section, we provide an empirical analysis of these properties to gain an understanding of their tradeoff. we also investigate the correlations between different measures, in app. b, to assess the extent to which various theoretical models of anticipatory and responsive processing lead to divergent empirical uncertainty measurements. the analysis in this section is based on gpt-2 small; results for gpt-neo 125m are provided in appendix app. c. ## psycholinguistic predictive power we now evaluate the generalized surprisal models introduced in §3 in terms of their predictive power for the neural and behavioral data presented in §4. ## conclusion we introduced a generalization over classic information-theoretic measures of predictive uncertainty in online language processing. our generalized surprisal framework subsumes both responsive and anticipatory measures, including established special cases, but providing a vocabulary and the formal tools for experimenters to design new measures and explain psycholinguistic data of interest.