Towards a Similarity-adjusted Surprisal Theory (2024)

Abstract: Surprisal theory posits that the cognitive effort required to comprehend a word is determined by its contextual predictability, quantified assurprisal. Traditionally, surprisal theory treats words as distinct entities, overlooking any potential similarity between them. Giulianelli et al. (2023) address this limitation by introducing information value, a measure of predictability designed to account for similarities between communicative units. Our work leverages Ricotta and Szeidl’s (2006) diversity index to extend surprisal into a metric that we term similarity-adjusted surprisal, exposing a mathematical relationship between surprisal and information value. Similarity-adjusted surprisal aligns with information value when considering graded similarities and reduces to standard surprisal when words are treated as distinct. Experimental results with reading time data indicate that similarity-adjusted surprisal adds predictive power beyond standard surprisal for certain datasets, suggesting it serves as a complementary measure of comprehension effort.

## introduction surprisal theory @xcite states that the effort a reader must spend to comprehend a word is a function of its contextual predictability, which is typically quantified as its surprisal, or negative logprobability. with numerous empirical @xcite @xcite and theoretical @xcite studies supporting its claims, surprisal theory is a widely-accepted model of the effort required for sentence comprehension. notably, surprisal theory treats words as completely distinct from one another, disregarding that they may express similar meanings. motivated by this shortcoming, @xcite proposed a new measure of comprehension effort: information value. similarly to surprisal, information value quantifies the predictability of a linguistic unit in context; unlike surprisal, however, it accounts for communicative equivalences between possible continuations. @xcite find this metric to be a significant predictor of utterancelevel reading times and acceptability judgments, both independently and in addition to surprisal. similarly inspired, we investigate similarityadjusted surprisal as a potential measure of comprehension effort. this measure is a natural extension of @xcite diversity index-which itself is a generalization of shannon's entropy used to measure species biodiversity. given a choice of similarity function, similarityadjusted surprisal computes a word's predictability while considering its likeness to alternative continuations. through this measure, we connect information value and standard surprisal, showing a mathematical relationship between these two metrics: similarity-adjusted surprisal has a monotonically increasing relationship with information value and reverts to standard surprisal when its similarity function regards different words as completely distinct. in experiments with reading time data, we explore the psycholinguistic predictive power of similarity-adjusted surprisal with semantic, syntactic and orthographic notions of word distance. for some datasets, we see that-as with information value-similarity-adjusted surprisal provides significant predictive power above and beyond standard surprisal. this complementarity suggests that there are aspects of incremental comprehension effort that are not fully captured by the classic definitions used in surprisal theory. we also observe that non-contextual notions of semantic distance lead to better predictors than using contextual notions of distance, which supports observations that incremental comprehension effort is (at least partially) driven by shallow semantic processing. ## background this section presents surprisal and information value, providing the relevant formal background for our similarity-adjusted surprisal. we will use v to refer to the vocabulary, i.e., a finite, non-empty set of words, and v * to refer to the set of all strings formed by concatenating words in v. we denote words as @xmath0, and strings (sequences of words) as @xmath1. an index t, e.g., w t and @xmath2, is used to mark positions within a string. ## similarity-adjusted surprisal theory to bridge the theoretical gap between surprisal and information value, we wish to derive a notion of per-word information content that accounts for similarities between different plausible continuations, rather than treating words as completely distinct outcomes. to this end, we turn to diversity indices: metrics developed in the field of biology to quantify biodiversity. 4 analogous to our setting, when quantifying biodiversity, it is desirable to have a metric that takes into account that some species are more closely related to each other (e.g., two species in the same genus vs. in different genera). we adapt one of these metrics to our context. ## results and discussion predictive power of similarity-adjusted surprisal. we evaluate the psycholinguistic predictive power that similarity-adjusted surprisal and information value provide beyond standard surprisal. to this end, we compute ∆ l (tab. 1) when adding these predictors to a regressor that already includes surprisal. in natural stories, we find similarityadjusted surprisal and information value provide predictive power complementary to standard surprisal when using embedding-based (both contextual and non-contextual) similarity functions. in dundee, significant additional predictive power only results from using non-contextual embedding similarities. meanwhile, in provo and brown, similarity-adjusted surprisal and information value do not add predictive power beyond surprisal; they are significant predictors when evaluated against a control baseline, but less so than surprisal (see tabs. 2 and 3 in app. e). in app. e, we also present results when exponentiating our definition of word pair similarity in eq. ( 7 ) by an α such that similarity-adjusted surprisal converges to standard surprisal as α → ∞; we find that similarityadjusted surprisal's predictive power is not strongly influenced by such choice. the differences in predictive power of the two similarity-adjusted measures follow other notable trends across datasets. predictive power is lowest on provo, where stimuli have an average of only 50 words each, followed by brown (553 words); it is highest for natural stories and dundee, both containing stimuli whose average lengths are above 1000 words. these results suggest equipping surprisal with semantic measures of word predictability is beneficial when the psycholinguistic measurements at hand are collected for stimuli situated in broader discourse contexts. other factors, such as the texts' style or topic, may have also played a role in these differences across datasets. ## conclusion this work introduces similarity-adjusted surprisal: a measure of contextual word predictability that takes into account word similarities. by equipping surprisal with the ability to account for words' relationships, we reconcile surprisal theory's predictions with those of information value and demonstrate their mathematical relationship. our experimental results on rt data indicate similarityadjusted surprisal has predictive power beyond that of standard surprisal, thus validating and enriching surprisal theory. points for future research include analyzing similarity functions that capture different characteristics of word meaning, as well as measuring the predictive power of similarity-adjusted surprisal for other indices of processing difficulty, such as n400 and other event-related brain potentials.