Language Model Quality Correlates with Psychometric Predictive Power in Multiple Languages (2023)

Abstract: Surprisal theory (Hale, 2001; Levy, 2008) posits that a word’s reading time is proportional to its surprisal (i.e., to its negative log probability given the proceeding context). Since we are unable to access a word’s ground-truth probability, surprisal theory has been empirically tested using surprisal estimates from language models (LMs). Under the premise that surprisal theory holds, we would expect that higher quality language models provide more powerful predictors of human reading behavior—a conjecture we dub the quality–power (QP) hypothesis. Unfortunately, empirical support for the QP hypothesis is mixed. Some studies in English have found correlations between LM quality and predictive power, but other studies using Japanese data, as well as using larger English LMs, find no such correlations. In this work, we conduct a systematic crosslinguistic assessment of the QP hypothesis. We train LMs from scratch on small- and medium-sized datasets from 13 languages (across five language families) and assess their ability to predict eye tracking data. We find correlations between LM quality and power in eleven of these thirteen languages, suggesting that, within the range of model classes and sizes tested, better language models are indeed better predictors of human language processing behaviors.

## introduction the relationship between a word's predictability and its reading time (rt) is an important object of study in psycholinguistics because it allows us to draw insights about how humans process sentences @xcite @xcite . extensive prior researchacross many datasets, populations, reading-time measurement methodologies and even languageshas found that the more predictable a word is, the faster it will be to process @xcite @xcite @xcite @xcite . beyond these empirical findings, the relationship between predictability and reading time has received formal treatment, forming the basis of surprisal theory @xcite . in surprisal theory, a word's predictability is operationalized in terms of its surprisal, or contextual negative log-likelihood: s ✓ (w t | @xmath0. 1 in practice, however, we do not know the contextual probability of words according to the datagenerating distribution p ✓ . thus, it is common to estimate surprisal using the distribution over words from a language model p θ instead, i.e., s θ (w t | @xmath1. recent advances in language modeling have drastically increased the quality of surprisal estimates from language models @xcite @xcite . the topic of interest of this work is whether this increased quality also leads to an increase in a language model's psychometric predictive power @xcite , i.e., how well its surprisal estimates can be used to predict rts. to put it simply, we ask: are better language models more powerful predictors of human behavior? if surprisal theory holds, we would expect this power to correlate positively with language models' quality. we will refer to this conjecture as the quality-power hypothesis, abbreviated as the qp hypothesis. empirical evidence for the qp hypothesis is mixed. when tested on english corpora, some studies have found that models that place higher probability on held-out data do tend to be better at predicting the pattern of human reading times @xcite . more recent work, however, has discovered two em-1 empirical results mostly support the existence of a linear relationship between surprisal and reading times @xcite @xcite : they find that surprisal is a better psychometric predictor than raw probabilities, and that non-linear functions of surprisal are not better than linear ones. pirical exceptions: the first empirical shortcoming was presented in @xcite -the only foot_0 non-english study (to the best of our knowledge) investigating the quality-power relationship. 3 specifically, kuribayashi et al. find that the qp hypothesis does not hold in japanese. the second shortcoming was identified by @xcite and @xcite , who observe that the quality-power relationship does not hold for the best (and most recent) language models, suggesting state-of-the-art language models might not be aligned with human incremental predictions. in this work we present two empirical studies that investigate these shortcomings. first, we conduct a systematic crosslinguistic analysis of the qp hypothesis, using small-and medium-sized lms. 4 we train 104 language models from scratch on data from 13 languages, using subsamples of the wiki40b dataset @xcite . within each language, we operationalize lms' qualities as their average per-word cross entropy on a test set. we then use the surprisals under these models to predict human reading times on a multilingual corpus of eye tracking data. we find that, in eleven out of thirteen languages, there is a significant correlation between a language model's quality and its surprisals' psychometric predictive power. to investigate the second shortcoming, we note that a (potential) issue with the findings presented in @xcite and @xcite is that the training data for these models is not publicly available, and might plausibly includes the reading time corpora against which their psychometric predictive power was evaluated. we discuss why this might present an issue, and perform an investigation of the relationship between data leakage and model's predictive power in §5, albeit with null results. ## surprisal vs reading times originally proposed by @xcite , surprisal theory posits that the amount of effort (and there-fore time) a reader must spend to process a word is a monotonically increasing function of its surprisal, where surprisal is a measure of a word's information content @xcite . surprisal theory has attracted lots of attention over the years, by both the natural language processing and cognitive science communities @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . one way surprisal theory has been evaluated is by quantifying surprisal's predictive power. if a word's surprisal influences reading times, we should see evidence in naturalistic reading data. psycholinguists thus train regressors p ϕ to predict reading times: r(w t ) ∼ p ϕ (r | •), where p ϕ 's input is defined as a vector of baseline features x t , comparing the regressors' performances when surprisal is included and when it is not. specifically, the predictive power of surprisal is quantified as the difference in the log-likelihood (llh) of reading time data under the two different regressors: note that this is an ideal ∆ llh , which assumes we have access to surprisals from the data-generating distribution, which we write as s ✓ (as a shorthand for s ✓ (w t | @xmath2. ## experimental setup measuring predictive power. as discussed in §2, and following previous work in this area @xcite , we quantify a language model's predictive power as its delta log likelihood ( ∆ llh ). when predicting the reading time of word w t in context, we will use features associated with w t and its two preceding words w t-1 , and w t-2 to account for spillover effects. 6 we will refer to this combined set of three words as our target words. the vector of variables in our baseline regressor x t include the log unigram frequency and the length (in characters) of our target words. the variables for our comparison regressor include x t plus two additional variables: the surprisal and the rényi entropy of our target words. rényi entropy is a generalization of shannon entropy which measures the expected surprisal of a word, given its context, and has been shown previously to impact reading times above and beyond surprisal @xcite . results from regressors that use just surprisal, or just rényi entropy as additional predictors are presented in app. a and b. to quantify the power of the language model, we report ∆ llh (as laid out in eq. ( 4 )) measured across ten folds of held-out data. a positive ∆ llh means that including surprisal and entropy as predictors increases predictive power over reading times. a negative value of ∆ llh is also possible and it implies overfitting. eye tracking data. we measure ∆ llh on meco @xcite , a multilingual corpus of eye tracking data collected on simple wikipedia-style articles from thirteen different languages. breaking the languages into their respective families, we have eight indo-european languages (dutch, english, german, greek, italian, norwegian, russian and spanish), two uralic languages (finnish, estonian), one afro-asiatic language (hebrew), one koreanic language (korean) and one turkic language (turkish). articles in the meco dataset undergo a multi-step translation process to ensure that meaning was preserved across languages. following previous studies @xcite , we use each word's gaze duration as our measure of reading time. 7 for dependent variables in our regressors, we use cross-participant averages, and treat skipped words as having a reading time of zero (as previously done by @xcite @xcite . language models. we train language models on different language sections of the wiki40b dataset @xcite . the data for each language is pre-split into a training, validation and test set in wiki40b. before training, we fit a unigramlm tokenizer with a vocabulary size of 32k to each language's entire training set. this tokenizer is then shared across all lms within that language. we then subsample each language's training set into sets with 1m, 3m, 10m, 30m, 100m, 300m and 1b tokens. we train models independently on each of this subsets, as well as on the entire training set available for that language. all models are trained using fairseq @xcite , following their recommended language modeling hyper-parameters. we use a standard decoder-only transformer with 6 layers, a context window size of 512 tokens, and shared input-output embeddings. we train our models using adam (kingma and ba, 2015), with a learning rate of 5e -4 , 4000 warm-up updates, and dropout of 0.1. we evaluate our models after each full epoch on their respective validation sets, using early stopping with a patience of 3 epochs. as our measure of language model quality, we report their cross entropy on the meco dataset. ## results of the crosslinguistic analysis the results for our crosslinguistic analysis can be seen in fig. 1 , with the languages in the different facets, h ✓→θ on the x-axis and ∆ llh on the y-axis. first, we note that the majority of language models are able to achieve ∆ llh above zero, indicating that surprisal and entropy values from the models are helpful in predicting reading times regardless of their size. this is in line with previous work that has found that surprisal and entropy have psychometric predictive power. that being said, the surprisal and entropy from our worse models (which were trained on only 1 million words) typically do not lead to positive ∆ llh , which is not unexpected from models that are such poor estimates of p ✓ (as evinced by their large cross entropy). turning to the relationship between cross entropy and ∆ llh , visually, it is quite clear that the majority of languages exhibit a positive relationship between language model quality and psychometric predictive power, i.e., a negative relationship between ∆ llh and cross entropy. to test this trend statistically, we fit linear regression models with ∆ llh as the sole dependent variable and cross entropy as a predictor. we find a significant effect of cross entropy on ∆ llh in 11 out of our 13 analyzed languages. 8 we do not find a significant effect of cross entropy in finnish or german. results are shown in fig. 1 , where we also report the pearson correlation coefficient between the cross entropy and mean ∆ llh of each language model. the high value of r quantitatively confirms the visual trends seen in fig. 1 . finally, we also run a regression model on all our data, pooled across languages. here, in addition to the fixed effect of h ✓→θ , we include random slopes and intercepts for each language. 9 in this regression we find a significant effect of h ✓→θ on ∆ llh (β = -0.0026, @xmath3.01). together, these results indicate that across a range of languages and model sizes, there is a consistent relationship between a language model's quality and its predictive power. ## analysis of data leakage as mentioned in the introduction, prior studies have found that the relationship between lm quality and psychometric predictive power does not hold for large, contemporary lms @xcite . one potential concern for both of these studies is that they used models whose training data is not publicly available, and is so large in scale, that it quite feasibly suffers from data leakage. that is, as human reading time datasets were publicly available, models might have been trained and tested on the same materials. motivation. there are a number of ways in which a model's surprisal estimates s θ for leaked data could provide a poorer fit to reading times. importantly, models are likely to underestimate surprisal on training data, and there could feasibly be a difference in the degree to which high surprisal vs. low surprisal values are underestimated. such a difference could hurt the ability to use surprisal to predict rts when restricted to linear regressors, which are the main focus of many rt analyses since the relationship between rt and surprisal has been observed to be linear in nature @xcite @xcite . explicitly, since the aforementioned underestimation might disproportionately skew a subset of surprisal estimates, linear regressors would be unable to model this altered relationship. ## conclusion this paper investigates the qp hypothesis, i.e., the relationship between a language model's quality and its psychometric predictive power. it looks specifically at two settings in which there is some evidence against the hypotheses by providing (i) a crosslinguistic assessment of its predictions and (ii) an investigation of the role of data leakage in model predictive power. although our results demonstrate that, at small-and medium-sized data scales, the qp hypothesis holds in the large majority of languages tested, our results do raise some questions. perhaps the most pressing of these has to do with our results for finnish and german, which showed negative (albeit non-significant) correlations between quality and power. further testing in these two languages, thus, is an important next step for future research.