Words, Subwords, and Morphemes: What Really Matters in the Surprisal-Reading Time Relationship? (2023)

Abstract: An important assumption that comes with using LLMs on psycholinguistic data has gone unverified. LLM-based predictions are based on subword tokenization, not decomposition of words into morphemes. Does that matter? We carefully test this by comparing surprisal estimates using orthographic, morphological, and BPE tokenization against reading time data. Our results replicate previous findings and provide evidence that *in the aggregate*, predictions using BPE tokenization do not suffer relative to morphological and orthographic segmentation. However, a finer-grained analysis points to potential issues with relying on BPE-based tokenization, as well as providing promising results involving morphologically-aware surprisal estimates and suggesting a new method for evaluating morphological prediction.

## introduction there is widespread consensus that human sentence processing includes word-level prediction @xcite ; see @xcite for a review. a growing body of research is making use of language models as computational proxies for human prediction at the word level, including traditional n-gram models (smith and levy, 2013), syntaxbased models @xcite , and more recent work that makes use of neural language models @xcite @xcite . as an overall paradigm, research in this area generally correlates surprisal, computed using corpus-based probability estimates (-log pr(word|context)), against measurable indices of human processing effort. these include measurement of processing activity using fmri @xcite @xcite , meg @xcite , eeg @xcite @xcite , and reading times @xcite @xcite @xcite . this correlation paradigm has produced useful insights about the role of prediction in language comprehension. in addition, correlations between language model surprisal and human processing activity have been taken to be an indication that language models are capturing relevant aspects of how human language works @xcite . 1 lost in the shuffle of this research progress, however, is the question of what, exactly, "prediction at the word level" is supposed to mean. within current linguistic theory, the very construct of "word" is receiving critical attention: albeit controversially, convincing arguments exist that the term word lacks a consistent, scientifically valid definition that holds consistently across the full range of human languages @xcite ; see @xcite for an overview connecting these theoretical claims to psycholinguistics and neurolinguistics. even setting aside that theoretical debate, however, the move to psycholinguistics and neurolinguistics using large language models has generated a mismatch between the units analyzed in human studies -typically orthographic words -and the subword tokens over which llms operate via approaches like word-piece and byte-pair encoding @xcite . to take an example, a typical llm tokenization of the word decomposition using a widely used bpe tokenization @xcite yields subword units dec, om, position. 2 in a typical human subjects experiment, measurements such as first-pass reading time would typically involve a region from the beginning of the whole word to its end. more generally, the measurement of activity for a word w in the human experimentation is related to language model predictions using w's subword units s 1 s 2 . . . s n . how is a correlation computed between measurements over different units? the solution adopted by most researchers @xcite @xcite is to compute surprisal separately for the s i , and then approximate the model's surprisal for w as the sum of those individual subword surprisals. the logic behind this choice is that, if the linking hypothesis behind the work connects surprisal with cognitive effort, the effort for the entire word should be the sum of the effort on each of the parts (smith and levy, 2013). 3 this, however, leads to another question: is that a reasonable approximation, given that the subwords produced by llm tokenization bear no clear correspondence to the subword decompositions in human processing? consider again the word decomposition. a large body of theoretical and empirical work would suggest that to the extent subword effort takes place, it would involve morphological units, in this case de, compos(e), and (i)tion @xcite . 4 the question we set out to answer in this paper, then, is whether the divergence between llm subword tokenization and human morphological decomposition is something to worry about in computational psycholinguistics and computational neurolinguistics research. operationalizing the question, would the sum-of-surprisals approach with morphologically valid units yield a better correspondence with human measurements than the standard approach using llm subword tokens? we would argue that the result is important regardless of which way the experimentation goes. if morphological units turn out to be a significantly better fit for human measurements, then cognitive researchers using llms should be using them -which could potentially raise real challenges given the widespread use of off-the-shelf pretrained llms. if statistically-driven subword units work just as well, then we have checked an important, previously unchecked box in terms of validating their use. this work is very much in the spirit of @xcite , who evaluated a "fleet" of language models across architectures, plus orthographic ngram models, against eyetracking and self-paced reading data. however to our knowledge, this study is the first to consider the assumption that llmsubwords can be used in lieu of morphological units. ## methods we trained three n-gram models under different word segmentation methods and evaluated them against reading time data from psycholinguistic experiments conducted in english. our choice of evaluation corpora and metrics are consistent with previous literature, such that we were only examining the effect of word segmentation without model architecture as a confound. our implementation, along with instructions for accessing the associated data, is available at https://github.com/ sathvikn/dl-psych-tokenization/ . ## results all our results show surprisal improves predictions of reading time relative to the control features, comparably to the 5-gram models' results in goodkind and bicknell (2018) and @xcite . on the face of it, these results seem to show that llm-style tokenization may not be an issue in psycholinguistic modeling. however, finer-grained analyses suggest otherwise. first, few words in the psycholinguistic corpora were split by the bpe tokenizer in the first place. as it turns out, the bpe tokenizer only split 5% of the tokens in the psycholinguistic corpora (11% when ignoring stopwords), compared to 25% and 44% respectively for the morphological analyzer (complete results 9 for natural stories, the tokens at wi-3 and wi-2 also had some predictive power (@xmath1.01 and @xmath2.1, respectively). in ## conclusions this study was a small, focused contribution that tackled an essential question previously unaddressed in psycholinguistics research that uses llms and their subword tokenizations. previous work has demonstrated a linear relationship between llm surprisal and human processing as measured by reading time, but there is good evidence that aspects of human processing are mediated by morphological units. for cognitive research, mightn't it be important to model surprisal using morphological units, rather than distributionally derived subword tokens that often deviate drastically from a word's morphological decomposition? on the one hand, our work offers the first comparison demonstrating that using distributionally derived rather than morphological subwords does not affect aggregate correlations between surprisal and reading times, a widely used behavioral measurement of human processing. on the other hand, given that psycholinguistics work is increasingly using proprietary pre-trained models with nonmorphological subword tokenization, far beyond the scale available for academic model training, our finer-grained analysis indicates that a degree of caution is warranted for more fine-grained stud-ies. perhaps our most important take-away is that cognitive investigations require a careful look at the cognitive plausibility of the models' units of analysis. our results also suggest new directions for more cognitively realistic models of prediction in language comprehension. we view the results from our finer-grained analysis as a step in this direction, and they also suggest going more deeply into the role of morphological segmentation on an item-byitem basis; for example, training an llm on morphological units and evaluating it on diagnostics for morphosyntactic generalization. our work here also introduces morphological surprisal computed automatically using a morphological segmenter, and validates its predictive power. this would be a natural fit for further work on morphological prediction at the neural level @xcite , including looking at the role of morphemes in phoneme prediction @xcite , and as a new representational level within integrative processing models that take phonological, word-level, and sentence-level contexts into account @xcite . this implementation could be refined through inferring hierarchical structure over morphological units in the style of @xcite to conduct larger-scale analyses. finally, regarding broader theoretical discussion, we note that surprisal (as operationalized using an llm or any other probability estimates) generally contributes to explanations at @xcite . 10 moreover, llm predictions represent a black-box combination of categories of information that both theoretical and experimental considerations suggest are processed in distinct ways @xcite @xcite . we would therefore argue that, despite their undeniable convenience and power, the widespread use of llms as probabilistic predictors deserves drastically more careful consideration than it has received if the field is to move in the direction of deeper insights into human representations and mechanisms in language processing.