Information Value: Measuring Utterance Predictability as Distance from Plausible Alternatives (2023)

Abstract: We present information value, a measure which quantifies the predictability of an utterance relative to a set of plausible alternatives. We introduce a method to obtain interpretable estimates of information value using neural text generators, and exploit their psychometric predictive power to investigate the dimensions of predictability that drive human comprehension behaviour. Information value is a stronger predictor of utterance acceptability in written and spoken dialogue than aggregates of token-level surprisal and it is complementary to surprisal for predicting eye-tracked reading times.

## introduction when viewed as information transmission, successful language production can be seen as an act of reducing the uncertainty over future states that a comprehender may be anticipating. saying a word, for example, may cut the space of possibilities in half, while uttering a whole sentence may restrict the comprehender's expectations to a far smaller space. measuring the amount of information carried by a linguistic signal is fundamental to the computational modelling of human language processing. such quantifications are used in psycholinguistic and neurobiological models of language processing @xcite @xcite , to study the processing mechanisms of neural language models @xcite @xcite , and as a learning and evaluation criterion for language modelling (under the guise of 'cross-entropy loss' or 'perplexity'). the amount of information carried by a linguistic signal is intrinsically related to its predictability @xcite @xcite . this connection is summarised in the definition of the surprisal, or information content, of a unit u @xcite , perhaps the most widely used measure of information: i(u) =log 2 p(u). predictable units carry low amounts of information-i.e., low surprisalas they are already expected to occur given the context in which they are produced. conversely, unexpected units carry higher surprisal. proper estimation of the surprisal of an utterance is intractable, as it would require computing probabilities over a high-dimensional, structured, and ultimately unbounded event space. it is thus common to resort to chaining token-level surprisal estimates, nowadays typically obtained from neural language models @xcite @xcite . however, token-level autoregressive approximations of utterance probability have a few problematic properties. a well-known issue is that different realisations of the same concept or communicative intent compete for probability mass @xcite , which implies that the surprisal of semantically equivalent realisations is overestimated. moreover, token-level surprisal estimates conflate different dimensions of predictability. as evidenced by recent studies @xcite , this makes it difficult to appreciate whether the information carried by an utterance is a result, for example, of the unexpectedness of its lexical material, syntactic arrangements, semantic content, or speech act type. we propose an intuitive characterisation of the information carried by utterances, information value, which computes predictability over the space of full utterances to account for potential communicative equivalence, and explicitly models multiple dimensions of predictability (e.g., lexical, syntactic, and semantic), thereby offering greater interpretability of predictability estimates. given a linguistic context, the information value of an utterance is a function of its distance from the set of contextually expected alternatives. the intuition is that if an utterance differs largely from alternative productions, it is an unexpected contribution to discourse with high information value (see figure 1 ). we obtain empirical estimates of information value by sampling alternatives from neural text generators and measuring their distance from a target utterance using interpretable distance metrics. information value estimates are evaluated in terms of their ability to predict and explain human reading times and acceptability judgements in dialogue and text. we find information value to have stronger psychometric predictive power than aggregates of token-level surprisal for acceptability judgements in spoken and written dialogue, and to be complementary to surprisal aggregates as a predictor of reading times. furthermore, we use our interpretable measure of predictability to gather insights into the processing mechanisms underlying human comprehension behaviour. our analysis reveals, for example, that utterance acceptability in dialogue is largely determined by semantic expectations while reading times are more affected by lexical and syntactic predictions. information value is a new way to measure predictability. as such, next to surprisal, it is a powerful tool for the analysis of comprehension behaviour @xcite @xcite , for the computational modelling of language production strategies @xcite @xcite and for the design of processing and decision-making mechanisms that reproduce them in natural language generation systems @xcite @xcite . the psychometric predictive power of surprisal. without direct access to the 'true' foot_0 conditional probabilities of linguistic units, psycholinguists have relied on statistical models of language to estimate surprisal @xcite . more recently, large-scale language models have emerged as powerful estimators of token-level surprisal, reflected by their ability to predict different aspects of human language comprehension behaviour (their psychometric predictive power). psychometric variables include self-paced and eye-tracked reading times @xcite @xcite @xcite , acceptability judgements @xcite @xcite @xcite , and brain response data @xcite . to obtain estimates of utterance surprisal, different aggregates of token-level surprisal have been proposed, motivated by psycholinguistic theories like erc and uid. however, their behaviour is far less understood (e.g., @xcite . for example, divergences between how model characteristics affect predictive power for different comprehension tasks (e.g., @xcite raise questions about whether token-level aggregates appropriately capture expectations over utterances in human language processing. alternatives in semantics and pragmatics. our proposed notion of information value takes inspiration from the concept of alternatives in semantics and pragmatics @xcite @xcite @xcite . reasoning about alternatives has been argued to be at the basis of the use of questions @xcite @xcite , focus @xcite @xcite , and implicatures @xcite @xcite . recently, alternative sets generated with the aid of language models have been used to provide empirical evidence that pragmatic inferences of scalar implicature depend on listeners' context-driven uncertainty over alternatives @xcite . @xcite generate sets of plausible words in context, within scalar constructions, then embed and cluster the resulting sentences to simulate conceptual alternatives. reasoning over word-and concept-level alternatives is operationalised through surprisal and entropy. to our knowledge, ours is the first study to use language models for the generation of full utterance-level alternatives. ## alternative-based information value given a context x, a speaker may produce a number of plausible utterances. we refer to these as a x , the alternative set. we define the information value of an utterance y in a context x as the real random variable which captures the distribution of distances between y and the set of alternative productions a x , measured with a distance metric d: this distribution characterises the predictability of y in its context. large distances indicate that y differs substantially from expected utterances, and thus that y is a surprising next utterance. ## psychometric predictive power we begin by evaluating our empirical estimates of information value in terms of their psychometric predictive power: can they predict comprehension behaviour recorded as human acceptability judgements and reading times? we test the robustness of this predictive power to the alternative set generation process and compare it to previously proposed utterance-level surprisal aggregates including mean, variance, and a range of summation strategies; see appendix b for full definitions. for each corpus in section 4.2, we measure the correlation between information value and the respective psychometric variable, which is the average in-context acceptability judgement for dai-lydialog, switchboard, and clasp, and the total utterance reading time normalised by utterance length for provo and brown. 7 alternative sets are generated using the language models and sampling strategies described in section 4.1. lexical, syntactic, and semantic distances are computed in terms of the distance metrics presented in section 3.1, for alternative sets of varying size ([10, 20, ..., 100]). the distributions of similarities in equation 1 are summarised using mean and min, thus yielding scalar estimates of information value. ## in-depth analysis of psychometric data using information value, we now study which dimensions of predictability effectively explain psychometric data. this allows us to qualitatively analyse the processes humans employ while reading and assessing acceptability. we also examine the effect of contextualisation on comprehension behaviour by defining two additional measures derived from information value (section 6.1) and using them as explanatory variables in linear mixed effect models to predict per-subject psychometric data. for the dialogue corpora and clasp, our mixed effect models predict in-context acceptability judgements. for the reading times corpora, our models predict the total time spent by a subject reading a sentence, as recorded in self-paced reading and eye-tracking studies. this is the sum, over a sentence, of word-level reading times (more details in appendix f). we include random intercepts for (context, target) pairs in all models. analysis procedure. for every corpus, we first test models that include a single predictor beyond the baseline: i.e., information value measured with each distance metric and either mean and min as summary statistics (see section 3.1). based on the fit of these single-predictor models, we select the best lexical, syntactic, and semantic distance metrics (with the corresponding summary statistics) to instantiate three-predictor models for each of the derived measures of information value. following @xcite , we evaluate each model relative to a baseline model which includes only control variables. control variables are selected building on previous work @xcite : solely the intercept term for acceptability judgements and the number of fixated words for reading times (more details in appendix f). as an indicator of explanatory power, we report ∆loglik, the difference in log-likelihood between a model and the baseline: a positive ∆loglik value indicates that the psychometric variable is more probable under the comparison model. we also report fixed effect coefficients and their statistical significance. the full results are shown in ## relation to utterance surprisal we have shown alternative-based information value to be a powerful predictor for contextualised acceptability judgements and reading times. in fact, information value is substantially more predictive of acceptability than utterance surprisal (section 5). we conclude with a focused comparison between these measures, considering whether they are complementary and why they might diverge. ## discussion and conclusion humans constantly monitor and anticipate the trajectory of communication. their expectations over the upcoming communicative signal are influenced by factors spanning from the immediate linguistic context to their interpretation of the speaker's goals. these expectations, in turn, determine aspects of language comprehension such as processing cost, as well as strategies of language production. we present information value, a measure which quantifies the predictability of an utterance relative to a set of plausible alternatives; and we introduce a method to obtain information value estimates via neural text generators. in contrast to utterance predictability estimates obtained by aggregating token-level surprisal, information value captures variability above the word level by explicitly accounting for more abstract communicative units like speech acts @xcite @xcite . we validate our measure by assessing its psychometric predictive power, its robustness to parameters involved in the generation of alternative sets, and its sensitivity to discourse context. using interpretable measures centred around information value, we investigate the underlying dimensions of uncertainty in human acceptability judgements and reading behaviour. we find that acceptability judgements factor in base rates of utterance acceptability (likely associated with grammaticality) but are predominantly driven by semantic expectations. in contrast, reading time is more influenced by the inherent plausibility of lexical items and part-of-speech sequences. we further compare information value to aggregates of tokenlevel surprisal, finding differences in the dimensions of predictability captured by each measure and their sensitivity to context. information value is a stronger predictor of acceptability in written and spoken dialogue and is complementary to surprisal for predicting eye-tracked reading times. information value is defined in terms of plausible continuations of the current linguistic context, taking inspiration from the tradition of alternatives in semantics and pragmatics @xcite @xcite . although the ideal set of alternatives would be derived directly from humans, neural text generators have demonstrated their potential to act as useful proxies, particularly when multiple generations are considered. variability among their productions has been shown to align with human variability @xcite , and decision rules that operate over sets of alternative utterances, rather than next tokens, have been shown to improve generation quality (e.g., @xcite . we release our full set of 1.3m generated alternatives, obtained with a variety of models and sampling algorithms, to facilitate research in this direction. our information value framework allows considerable flexibility in defining alternative set generation procedures, distance metrics, and summary statistics. we hope it will enable further investigation into the mechanisms involved in human language processing, and that it will serve as a basis for cognitively inspired learning rules and inference algorithms in computational models of language.