On the Proper Treatment of Tokenization in Psycholinguistics (2024)

Abstract: Language models are widely used in computational psycholinguistics to test theories that relate the negative log probability (the surprisal) of a region of interest (a substring of characters) under a language model to its cognitive cost experienced by readers, as operationalized, for example, by gaze duration on the region. However, the application of modern language models to psycholinguistic studies is complicated by the practice of using tokenization as an intermediate step in training a model. Doing so results in a language model over *token* strings rather than one over character strings. Vexingly, regions of interest are generally misaligned with these token strings. The paper argues that token-level language models should be (approximately) marginalized into character-level language models before they are used in psycholinguistic studies to compute the surprisal of a region of interest; then, the marginalized character-level language model can be used to compute the surprisal of an arbitrary character substring, which we term a focal area, that the experimenter may wish to use as a predictor. Our proposal of marginalizing a token-level model into a character-level one solves this misalignment issue independently of the tokenization scheme. Empirically, we discover various focal areas whose surprisal is a better psychometric predictor than the surprisal of the region of interest itself.

## introduction language models (lms) have become a popular tool for computational psycho-and neurolinguists, who use them to instantiate and test executable linguistic theories @xcite @xcite . while there are various ways to operationalize theories of language processing using lms (e.g., @xcite @xcite , their most common use in this area is to produce a specific quantity of interest: the probability of a character string given a context, conventionally mapped to the string's negative log probability, also known as its surprisal. surprisal has been posited to correlate with the difficulty incurred by a comprehender processing that word @xcite . and, estimates of surprisal obtained from neural lms have proven to be significant predictors of a broad range of psycholinguistic measurements of processing difficulty @xcite @xcite , providing ample empirical support for the role of surprisal in psycholinguistics theory, in addition to other predictors, e.g., word length and unigram surprisal. modern language models do not provide direct access to contextual probabilities at the character level. instead, they provide a distribution over strings of tokens, supercharacter units that are induced during a pre-processing step, e.g., by the byte-pair encoding tokenizer (bpe; @xcite . however, in computational psycholinguistics, it is often necessary to compute the surprisal of an arbitrary character substring of the stimulus. for example, the predictability of the first three characters of a word, which can be viewed parafoveally, is known to be an important predictor of whether the region is going to be skipped by the reader @xcite @xcite . and, to properly model a region's skip rate under surprisal theory, we should use the surprisal of the first three characters, rather than the surprisal of the entire region, as a predictor. a complication arises when the first three characters of the region do not correspond to a token, and we have to marginalize over all token strings that start with those three characters. because performing such a marginalization over token strings is computationally expensive and therefore requires approximation @xcite @xcite , it has yet to be adopted among computational psy-cholinguists. indeed, the literature lacks guiding principles for how computational psycholinguists should properly apply token-level language models to the field's inherently character-level problems. for instance, a number of recent studies have blurred the line between algorithmic and linguistic concerns @xcite @xcite @xcite @xcite . in particular, two recent studies @xcite suggest it is important to include a word's trailing whitespace in the computation of the word's surprisal to account for the mismatch between tokens and words. 1 both are motivated by a peculiarity foot_1 of the bpe tokenizer itself rather than a deeper appeal to linguistic theory: examining standard practice in experimental eyetracking research reveals that regions of interest are typically defined to include the preceding whitespace rather than the trailing one @xcite @xcite . we attribute the focus on trailing whitespace to the obfuscated relationship between token-level and character-level surprisal. our paper clarifies the proper role of tokenization in surprisal theory: we take the stance that tokenization is irrelevant. first, we note psycholinguistic stimuli should be viewed as character strings rather than token strings. this follows from the observation that human linguists construct psycholinguistic stimuli without regard for any given lm's token alphabet. then, as is common, the stimuli are divided into regions of interest (character substrings of the stimulus) for which the experimenter gathers a psycholinguistic measurement. note that treating regions of interest as character strings does not prevent the experimenter from claiming they represent morphemes, words, or phrases, all of which are built from characters in text form. finally, the experimenter collects the psycholinguistic measurements associated with each region. none of the above steps makes use of tokenization schemes, completing our argument. a problem with respect to tokenization first arises when the experimenter analyzes the mea-surements they collected by means of a language and the regions of interest they decomposed their stimuli into do not neatly align with a string of tokens, or when the experimenter wishes to compute the surprisal of a sub-or super-string of the region. in this paper, we contend that the solution to this problem is to marginalize the token-level lm into a character one before using it to compute surprisal. moreover, on this view, such marginalization constitutes an algorithmic problem, but not a theoretical one for psycholinguistics. because computational psycholinguists have yet to convert pretrained token-level lms into character-level ones through marginalization, we contend that they have yet to explore many potentially effective surprisal-based predictors. in our experimental section, we test the degree to which the choice of various substrings of the stimulus that overlaps the regions of interest, which we term focal areas, affect recent empirical findings in surprisal theory. we perform such an exploration by means of @xcite approximate marginalization scheme. across four datasets of eye-tracked reading times, we consistently find that computing the surprisal of the entire region of interest (with leading or trailing whitespace) rarely leads to the most effective surprisal-based predictor. for instance, as hinted at above, we observe that on the celer dataset @xcite , the surprisal of the first three characters is a significantly better predictor of skip rate. on the provo and meco datasets @xcite , the surprisal of the first characters of a region-either determined by a fixed-size or a dynamically sized focal area extending over typical rightward word identification spans-is on par with the surprisal of the full region. finally, on the ucl dataset @xcite , including a look-ahead focal area that peeks at the subsequent region significantly improves reading time predictions. ## formalizing psycholinguistic stimuli we now offer an abstract formalization of the stimuli present in a sentence-processing experiment. ## language models as predictors in psycholinguistics, lm-derived predictors are commonly used to predict measurements such as the participants' reading time for an roi. we now give an overview of the necessary background. ## marginalizing out token strings the previous two sections ( §2 and §3) have formalized psycholinguistic stimuli, their regions of interest, and focal areas at the character level. indeed, in this discussion, the character-level model used to compute surprisal is agnostic as to whether the model underlyingly uses tokenization or not. however, tokenization has evolved into a standard practice in constructing language models. rather than constituting a distribution over σ * , the set of all character strings, most modern language models are distributions p ∆ over ∆ * where ∆ is an alphabet of tokens. to encode a character string as a token string, we apply a function of type τ : σ * → ∆ * . to decode a token string to a character string, we apply a function of type κ : ∆ * → σ * (cf. @xcite . for the purposes of this paper, we assume this pair of functions satisfy: ## predictive power of focal areas our experimental design is discussed in app. b. we consider rois with leading and trailing whitespaces and experiment with the focal areas described in §3.4. additional results may be found in app. c. ## conclusion we treat the role of tokenization in psycholinguistics. we recommend predictors be derived from character-level surprisal, allowing the modeler to explore a wider range of useful predictors.