How to Compute the Probability of a Word (2024)

Abstract: Language models (LMs) estimate a probability distribution over strings in a natural language; these distributions are crucial for computing perplexity and surprisal in linguistics research. While we are usually concerned with measuring these values for words, most LMs operate over subwords. Despite seemingly straightforward, accurately computing probabilities over one unit given probabilities over the other requires care. Indeed, we show here that many recent linguistic studies have been incorrectly computing these values. This paper derives the correct methods for computing word probabilities, highlighting issues when relying on language models that use beginning-of-word (bow)-marking tokenisers, e.g., the GPT family. Empirically, we show that correcting the widespread bug in probability computations affects measured outcomes in sentence comprehension and lexical optimisation analyses.

## introduction language models (lms) define probability distributions. after being trained on language data, these models can be used to compute estimates of the probability of a sequence of characters @xmath0, or of a word w @xmath1. while deriving such estimates is now rarely the explicit goal of training such models, foot_0 this use case is still critical in several fields. estimating the probability of a sequence of characters, for instance, is necessary to compute a model's perplexity; a core evaluation metric in lm training. estimating the probability of a word in context is necessary to compute a word's surprisal:log p(w t | @xmath2, an important value in both psycho-and computational tl;dr: how to correctly compute word probabilities given a word w in context @xmath3, let s w and s @xmath4. further, let: in this work, we discuss how to correctly compute a word's contextual probability: p(w t | @xmath5. this value's computation depends on the choice of tokeniser used to define an lm's vocabulary. when using an end-of-word (eow)-marking tokeniser, computing p(w t | @xmath6. however, when using a beginning-of-word (bow)-marking tokeniser, correctly computing this value is not as straightforward. we derive methods for these tokenisation schemes, which we present in fig. 1 . since many widely-used lms employ bow-marking tokenisers (e.g., the gpt models, pythia, mistral), this highlights a wide-spread "bug" in how most recent psycholinguistics and computational linguistics works compute word probabilities (present in, e.g., @xcite @xcite . 3 empirically, we evaluate how correcting this computation affects the results of two prior empirical analyses: one on sentence comprehension and another on the lexicon's communicative efficiency. while these studies' conclusions do not change, we do observe statistically significant differences between the measured quantities when using the correct vs. buggy methods for computing word probabilities. we conclude this methodological choice may impact empirical analyses, and that future work should adopt these proposed corrections. despite decades of discussion and debate, there is no single, widely accepted definition of what constitutes a word @xcite . typically, definitions are made with respect to some system within the language, such as its orthography, phonology, or grammar. as a concrete example, one can delineate words using the sound system of a language: if we assume words define the domain over which certain phonological processes operate (e.g., vowel harmony), we can delineate words based on those processes' boundaries @xcite . alternatively, one could define words as grammatical elements (e.g., a root plus affixes) that are cohesive, occur in a fixed order, and have a coherent meaning @xcite . notably grammatical and phonological words are non-isomorphic. for example, english hyphenated elements like editor-inchief or mother-in-law are typically analysed as a single grammatical word that contains multiple phonological words @xcite . we abstain from this broader discussion here. while we use the definition common to natural language processing applications-where words are defined orthographically foot_4 -our methods only assume the existence of a deterministic set of rules for segmenting a string of characters into words. ## words and distributions over them let w be a lexicon-the (potentially infinite) set of all words in a language-and @xmath7. further, let @xmath8; w * denotes the set of all finite-length word sequences. now, assume distribution p describes the probability with which users of this language produce sequences w. we can decompose these probabilities autoregressively as: now, let • denote concatenation (between either strings or sets of strings). for instance, we can write w • w * = {w • w ′ | w ′ ∈ w * } to represent the set of all strings with prefix w. we can compute our desired conditional distribution as the quotient of two evaluations of p w : note that this is a trivial invocation of the joint rule of probability: the conditional p(w | @xmath9. we call probabilities of the form p w (w • w * ) the prefix probability of w. as we will show, careful consideration of these prefix probabilities is critical for converting between our desired distributions (over words) and the ones provided by language models (over subwords). orthography. we assume here this language can be written, and that it has a standardised orthographic convention. formally, given a language's alphabet c, each string w can be mapped to a sequence of characters @xmath10: w * → c * . further, we assume this language allows for straightforward segmentation from orthography. given a sequence of characters c, we can thus extract a sequence of words as s c * →w * (c) = w. ## subwords and language models most modern language models are not defined directly as distributions over words w, but rather as distributions over subwords. these subwords are themselves defined by a choice of tokeniser. 6 in this section, we first introduce tokenisers, and how they map words to subwords (and back). we then use these building blocks to show how we can compute word probabilities from subword probabilities. ## the nuances of mapping: tokeniser-dependent strategies we are left with the task of finding a set of subword prefixes which will allow us to compute the probabilities of in this section, we discuss how our tokeniser-specifically whether it uses end-or beginning-of-word markings in its vocabulary-affects this task. figure 2 : the output of tokenisers with different methods of handling word delineations. ## experiments we now investigate how correcting the computation of word probability estimates affects the results of prior studies. these works incorrectly computed probabilities as p (s w | s @xmath11.e., using eq. ( 19 ) without bug fix 1 ), which we term as buggy estimates here. we explore two settings: psycholinguistics experiments surrounding sentence comprehension @xcite and computational linguistics experiments assessing the lexicon's communicative efficiency @xcite . we follow these works' experimental methodologies, observing how the use of corrected surprisal estimates impacts the conclusions that were originally drawn. ## conclusion this work expounds on the intricacies of accurately computing contextual word probabilities using language models. we focus on the challenges posed by the use of subword vocabularies. we show that subword vocabularies defined using beginning-ofword (bow) tokenisers-common in many modern lms-introduce complexities that are often overlooked. we point out that this has led to potential inaccuracies in computing probability estimates of various prior empirical analyses. our methodological corrections lead to significant differences in results, although the overarching conclusions of the previous studies that we explore remain the same. this finding underscores the importance of precise computational methods in linguistic research. future work should ensure these corrections are adopted to enhance the reliability of their analyses.