Language models emulate certain cognitive profiles: An investigation of how predictability measures interact with individual differences (2024)

Abstract: To date, most investigations on surprisal and entropy effects in reading have been conducted on the group level, disregarding individual differences. In this work, we revisit the predictive power (PP) of different LMs’ surprisal and entropy measures on data of human reading times as a measure of processing effort by incorporating information of language users’ cognitive capacities. To do so, we assess the PP of surprisal and entropy estimated from generative language models (LMs) on reading data obtained from individuals who also completed a wide range of psychometric tests.Specifically, we investigate if modulating surprisal and entropy relative to cognitive scores increases prediction accuracy of reading times, and we examine whether LMs exhibit systematic biases in the prediction of reading times for cognitively high- or low-performing groups, revealing what type of psycholinguistic subjects a given LM emulates.Our study finds that in most cases, incorporating cognitive capacities increases predictive power of surprisal and entropy on reading times, and that generally, high performance in the psychometric tests is associated with lower sensitivity to predictability effects. Finally, our results suggest that the analyzed LMs emulate readers with lower verbal intelligence, suggesting that for a given target group (i.e., individuals with high verbal intelligence), these LMs provide less accurate predictability effect estimates.

## introduction human language comprehension and, by extension, human reading is incremental in nature: humans process words sequentially @xcite , and different words in varying contexts impose different amounts of cognitive processing effort @xcite . similarly, language models' conditional probability distributions assign differ1 code is available at https://github.com/dili-lab/ lm-cog-profiles ent probabilities for potential continuations for a given prefix. the relationship between cognitive effort and predictability measures derived from lms' probability distribution was operationalized by surprisal theory @xcite . since then, a large body of research has investigated the exact nature of the relationship between surprisal and human processing effort, such as determining appropriate linking functions @xcite , or its manifestation in different languages @xcite @xcite . moreover, it has been shown repeatedly that both the quality of a model from which surprisal is extracted as well as the amount of data a model is trained on correlate with the model's psychometric predictive power foot_0 , i.e., its ability to predict human behavioral processing data @xcite @xcite , albeit only to a certain extent @xcite . so far, most studies have tested these predictions on the group-level, neglecting individual cognitive differences that might influence readers' capacities to make predictions about upcoming material. however, an increasing body of research has demonstrated that an individual's characteristics do play an important role in human language processing (e.g., @xcite @xcite . apart from surprisal, it has been suggested that reading processes are not purely responsive, which is mirrored in the surprisal effect of a word on a reader, but also anticipatory: readers make implicit assumptions about future words and allocate time to process it in advance. this anticipatory effect, which is reflected in the reading behavior, is induced by a reader's expectation about a word's surprisal and is operationalized as a word's contextual entropy @xcite @xcite . in this work, we revisit the relationship between predictability measures (surprisal and contextual entropy) and data of human processing effort in consideration of language users' individual cognitive differences. more specifically, we assess the predictive power (pp) of entropy, surprisal, as well as their interactions with cognitive measures, as predictors of human reading times in linear-mixed models. after establishing the baseline predictive power of surprisal and entropy on our german reading data (h b ), we investigate the following novel hypotheses: h 1 : modulating surprisal and entropy effects relative to individual cognitive capacities improves their predictive power on reading times on unseen data. h 2 : individuals with higher cognitive capacity rely less on predictive processing strategies, and hence exhibit lower surprisal or entropy effects. h 3 : lms are better at predicting reading times for certain cognitive profiles. to address these hypotheses, we utilize the individual differences corpus (indico; @xcite , which contains both reading data and scores of a comprehensive psychometric assessment targeting various cognitive capacities, including verbal and non-verbal working memory, verbal and non-verbal cognitive control, verbal and non-verbal intelligence and reading fluency. we deploy five pre-trained generative lms from three languagefamilies-gpt-2 base and large, llama 2 7b and 13b, and mixtral-to estimate both surprisal and contextual entropy and quantify their predictive power by including them as predictors in linear regressors, which are fitted to predict by-word reading times from indico. if the regressors' loglikelihood improves after including these predictors and their interaction with the psychometric scores, we take this as corroboration of their predictive power. we find that adding interaction terms between predictability measures (surprisal and entropy) and most cognitive scores significantly improves the quality of reading time predictions, and that in general, individuals with higher cognitive capacities exhibit smaller predictability effects. lastly, there is evidence that lms' abilities to predict reading times vary between high-and low-performing individuals within certain cognitive capacities. specifically, all tested models emulate the processing behaviour of individuals with low verbal intelligence. our work is a first step towards investigating i) the differences in surprisal and entropy effects across different cognitive profiles, and ii) what type of cognitive biases might be inherent in the way lms process language. ## methods surprisal. given a vocabulary σ and an augmented vocabulary σ = σ∪{eos}, which contains a special eos (end-of-sentence) token, the surprisal @xcite of a given sequence is defined as ## experiments data we employ german reading time data from in-dico @xcite . this corpus contains eye-tracking-while-reading and self-paced-reading data from 61 native german speakers, collected across four experimental sessions, alongside a comprehensive battery of individual psychometric scores in four cognitive domains: cognitive control, working memory, intelligence, and reading fluency. 3 for our analyses, we use the standardized scores of 13 psychometric tests. following previous work @xcite , we employ first-pass reading time (fprt), also referred to as gaze duration: the sum of all fixations on a word when fixating it for the first time-as a proxy for processing load. whereas total fixation duration can incorporate words from the right context due to regressive saccades, fprt most strongly reflects the initial processing difficulty. 4 given that in our study, we only deploy auto-regressive lms (cf. §4), fprts are also more in line with the fact that these models only have access to a word's left context. ## discussion most studies on the predictive power of surprisal and entropy on reading times have been conducted on the group-level. although individual-level effects may have been taken into account in terms of random slopes, these effects have not been associated with different cognitive profiles. our findings suggest that (1) incorporating information on individuals' cognitive capacities and allowing them to modulate the magnitude of surprisal and entropy effects can increase the predictive power of these predictability measures, (2) individuals exhibit surprisal and entropy effects relative to certain cognitive capacities, and (3) some language models exhibit higher predictive power of reading times for groups of individuals associated with a certain cognitive profile. ## conclusion to date, most investigations on predictability effects have been conducted on the group-level, disregarding individual differences, assuming that the predictive power of next-word predictability metrics such as surprisal or entropy on human reading times is uniform across cognitive profiles. in this work, we have shown that indeed, lms do exhibit systematic biases towards readers of certain cognitive profiles. this illustrates the usefulness of incorporating individual-level information within the study of lm interpretability and language modelling in general.