Scaling in Cognitive Modelling: a Multilingual Approach to Human Reading Times (2023)

Abstract: Neural language models are increasingly valued in computational psycholinguistics, due to their ability to provide conditional probability distributions over the lexicon that are predictive of human processing times. Given the vast array of available models, it is of both theoretical and methodological importance to assess what features of a model influence its psychometric quality. In this work we focus on parameter size, showing that larger Transformer-based language models generate probabilistic estimates that are less predictive of early eye-tracking measurements reflecting lexical access and early semantic integration. However, relatively bigger models show an advantage in capturing late eye-tracking measurements that reflect the full semantic and syntactic integration of a word into the current language context. Our results are supported by eye movement data in ten languages and consider four models, spanning from 564M to 4.5B parameters.

## introduction the role of context-dependent statistical information in human language processing has received considerable attention in cognitive modelling. a solid empirical finding that has emerged from this research line is that speakers actively anticipate the upcoming linguistic material @xcite . indeed, behavioral and neural patterns that are diagnostic of reduced cognitive cost have been reported in response to predictable words; these emerged from the analysis of eye movements @xcite , changes in pupil size @xcite , selfpaced reading times, @xcite , erp responses @xcite @xcite , frontotemporal blood oxygenation levels @xcite , and meg data @xcite . inferential theories of language comprehension argue that prediction must be an intrinsic feature of an incremental probabilistic cognitive processor @xcite . these accounts contend that the kullback-leibler (kl) divergence (i.e., relative entropy) between the probabilistic state of the processor before and after observing a given word is the cause of the processing difficulty associated with that word. it has been demonstrated that the kl divergence associated with this probability shift is mathematically equivalent to the surprisal of that word, i.e., the negative logarithm of its probability conditioned by the preceding sentence context (surprisal(w i ) = log p (w i |w 1 , w 2 . . . w i-1 ); @xcite . inferential theories, which predict a logarithmic linking function between contextual predictability and cognitive cost, are supported by extensive experimental evidence in the computational psycholinguistics literature (smith and @xcite @xcite @xcite . statistical language models developed in nlp research have been of paramount importance in the evolution of inferential theories of language comprehension. indeed, language models are usually trained to predict the upcoming word in a corpus of naturalistic text, and thus define a conditional probability distribution that can be employed to compute word surprisal. modern computationallyderived estimates of word predictability have been shown to perform on par @xcite or even better @xcite than predictability estimates obtained with expensive human annotation (although they fail to account for the processing demands of some specific linguistic patterns, see @xcite @xcite . however, given that language models display a great amount of variation in their architectures and performances, various studies have investigated which models are better suited to characterize the behavioral correlates of human sentence comprehension. seminal work has shown that the "linguistic accuracy" of a model (i.e., its ability to accurately predict the next word) is positively related to its "psychological accuracy" (namely, the capability of a surprisal estimate to explain variance in human responses, as captured by the increase in fit in a corresponding statistical model; @xcite @xcite , but see @xcite . a recent incidental finding by @xcite shed doubt on such conclusion. the authors reported that the gpt-2 small model substantially outperformed gpt-3 in predicting self-paced reading times and fixation patterns while having a parameter size smaller by three degrees of magnitude and displaying higher perplexity values in next-word prediction. the result, which suggests that the correlations between the linguistic and psychological accuracy of language models might not hold for very deep transformer-based architectures, has been promptly replicated with different gpt-2 variants @xcite . this observation is at odds with the empirical scaling laws for neural language models @xcite , which show that the quality of a language model (both in terms of test loss and downstream performance, @xcite increases monotonically as the number of parameters increases (although see @xcite . ## related work and motivation research in computational psycholinguistics has largely followed the progressive switch to the transformer architecture that has characterized the nlp literature in the last years, with transformerbased surprisal estimates being evaluated as predictors of processing difficulty @xcite @xcite . while early studies within this research line have documented a positive relationship between the linguistic and the psychological accuracy of a model @xcite @xcite , recent findings with decoder-only large language models have documented an opposite pattern, with larger and betterperforming pre-trained transformers providing worse psychometric estimates than their smaller counterparts @xcite . the possibility that cognitive modelling might constitute an exception to scaling laws is intriguing, but further examination is needed to warrant such claims. all the evidence in support of this view has come from the english language alone (except from @xcite , leaving an open question as to the cross-lingual generalizability of these findings. the english-centric approach to this problem is not surprising, since inferential approaches to language processing have been primarily supported by experimental evidence in english @xcite @xcite @xcite @xcite , dutch @xcite and german @xcite , while empirical support from non-germanic languages is far more limited (although see @xcite . to the best of our knowledge, there is only one study that provided large-scale cross-lingual evidence in support of surprisal theory (de varda and marelli, 2022). indeed, both nlp @xcite and cognitive science research @xcite have long overrelied on the english language to develop language processing systems and test theories of language and cognition. this tendency can lead to hasty claims of generality, and must be mitigated with cross-linguistic research efforts challenging the universality of english-specific findings. another potential shortcoming of the studies that reported the inverse scaling trend is that they only considered a single eye-tracking measurement as an index of processing cost @xcite . this choice reflects a common tendency within the inferential language processing framework @xcite @xcite ; however, natural reading is an ability composed of multiple sub-processes characterized by different levels of complexity (see for instance @xcite . in principle, it is reasonable to assume that different processing stages, characterized by different degrees of complexity, might be better captured by models with varying parameter sizes, with shallow processes better modelled by (relatively) simpler networks, and complex integrative operations better characterized by more complex architectures. ## aims the current work aims at inspecting the relationship between the linguistic and the psychological accuracy of a neural language model across languages, testing whether previous observations on inverse scaling in cognitive modelling hold across a sample of ten languages belonging to four different families. furthermore, our study considers different eye-tracking measures that are thought to reflect different processing stages, to examine the possibility that the relationship between the psychological and linguistic accuracy of a model might vary as a function of the computational complexity of the cognitive operations being studied. ## results the first main finding of our study is that surprisal is a solid predictor of reading times across the languages considered, confirming the previous observation that context-dependent probabilistic processing generalizes beyond the germanic language sample typically considered in the literature @xcite . the xglm-based surprisal estimates were statistically significant in all cases when considering gd and tt, and in the vast majority of the cases when considering ff (see the increase in goodness of fit that could be attributed to surprisal is displayed in figure 1 , grouped by model type and fixation measure. concerning ff (1a), we reported a general decrease in ∆loglik when increasing the number of parameters, with the smallest xglm 564m variant outperforming the bigger models in terms of psychological accuracy. a similar trend can be observed in gd (1b), although the difference in psychological accuracy between xglm 564m and xglm 1.7b appears to be rather small foot_3 . the results are different when considering tt as the dependent variable (1c), as in this case the model that provided the highest average increase in goodness of fit was xglm 1.7b foot_4 . ## discussion in this experiment, we showed that large multilingual transformer-based models were outperformed by their smaller variants in predicting early eye movement measurements of processing difficulty. these measurements are thought to reflect predictive processes, lexical access, and early semantic integration. this result corroborates the previous claims that cognitive modelling might constitute an exception to empirical scaling laws in nlp @xcite . however, predictability estimates computed by relatively larger variants of the same architecture -but not the largest -provided surprisal estimates that better captured late eye-tracking measurements, which are thought to reflect the full semantic and syntactic integration of a word into the phrasal context. this dissociation is in line with the observation that it is not appropriate to adopt a "one-size-fits-all" approach when studying how linguistic distributional knowledge explains different cognitive processes @xcite . instead, context-dependent probabilistic information derived from different neural architectures might be more apt to model certain cognitive mechanisms, depending on the computational complexity of the processes being considered.