Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokens (2023)

Abstract: Recent psycholinguistic studies have drawn conflicting conclusions about the relationship between the quality of a language model and the ability of its surprisal estimates to predict human reading times, which has been speculated to be due to the large gap in both the amount of training data and model capacity across studies. The current work aims to consolidate these findings by evaluating surprisal estimates from Transformer-based language model variants that vary systematically in the amount of training data and model capacity on their ability to predict human reading times. The results show that surprisal estimates from most variants with contemporary model capacities provide the best fit after seeing about two billion training tokens, after which they begin to diverge from humanlike expectations. Additionally, newly-trained smaller model variants reveal a ‘tipping point’ at convergence, after which the decrease in language model perplexity begins to result in poorer fits to human reading times. These results suggest that the massive amount of training data is mainly responsible for the poorer fit achieved by surprisal from larger pre-trained language models, and that a certain degree of model capacity is necessary for Transformer-based language models to capture humanlike expectations.

## introduction the predictability of upcoming linguistic material has long been considered a crucial factor underlying difficulty in human sentence processing @xcite , and has received empirical support from numerous studies showing surprisal @xcite to be highly predictive of relevant behavioral and neural measures (e.g. @xcite @xcite . since language models (lms) are trained to estimate a conditional probability distribution of a word given its context, surprisal estimates calculated from them have often been evaluated on their ability to predict measures of processing difficulty. recent studies in computational psycholinguistics have provided conflicting evidence with regard to the relationship between lm quality (i.e. nextword prediction accuracy) and goodness-of-fit to human reading times. earlier work using newlytrained lms showed a negative relationship between lm perplexity and predictive power of surprisal estimates @xcite @xcite , but more recent work using large pre-trained transformer-based lms (e.g. gpt-2; @xcite show a robust positive relationship between the two variables @xcite . while @xcite conjecture that these studies capture two distinct regimes, it remains less clear where the reversal in this relationship happens. the main challenge in answering this question lies in the massive difference in terms of both the amount of training data and the model capacity of lms that were studied. the current study aims to cover this conceptual middle ground by evaluating, on their ability to predict human reading times, surprisal estimates from transformer-based lm variants that vary systematically in the amount of training data and model capacity. results from regression analyses show that surprisal from most lm variants with contemporary model capacities make the biggest contribution to regression model fit after seeing about two billion tokens of training data, after which additional training data result in surprisal estimates that continue to diverge from humanlike expectations. additionally, surprisal estimates from newlytrained smaller lm variants reveal a 'tipping point' at convergence, after which the decrease in perplexity begins to result in poorer fits to human reading times. taken together, these results suggest that the vast amount of training data is mainly responsible for the poorer fit achieved by surprisal from larger transformer-based pre-trained lms @xcite , and that a certain degree of model capacity is necessary for transformer-based lms to capture humanlike expectations that manifest in reading times. ## experiment 1: influence of training data size the first experiment examines the influence of training data size on the predictive power of transformer-based lm surprisal by evaluating lm variants at various points in training on self-paced reading times from the natural stories corpus @xcite and go-past eye-gaze durations from the dundee corpus @xcite . ## experiment 2: influence of model capacity the second experiment further examines the relationship between model capacity and predictive power of surprisal estimates by evaluating transformer-based lm variants smaller than the pythia variants at various points in training, following similar procedures as experiment 1. ## discussion and conclusion this work aims to consolidate conflicting findings about the relationship between lm quality and the predictive power of its surprisal estimates by systematically manipulating the amount of training data and model capacity. experimental results show that surprisal from most contemporary transformer-based lm variants provide the best fit to human reading times with about two billion training tokens, after which they begin to diverge from humanlike expectations. it is conjectured that early training data up to about two billion tokens is helpful for learning e.g. selectional preferences that align well with humanlike prediction and processing difficulty. however, as the models see more training data, they are able to achieve 'superhuman' prediction, which makes their surprisal estimates diverge more and more from human reading times as training continues. the words for which prediction by lms improves with massive amounts of training data are likely to be open-class words like nouns and adjectives, whose reading times were identifed as being most severely underpredicted by their surprisal estimates @xcite . moreover, at the end of training, these model variants show a strictly monotonic and negative relationship between perplexity and fit to human reading times. this directly replicates the findings of @xcite and adds to a growing body of research reporting an inverse correlation between model size and regression model fit @xcite @xcite . the current results demonstrate that this relationship emerges with large amounts of training data and becomes stronger as training continues. the bottleneck posed by the limited model capacity of the smaller variants appears to prevent them from learning to make excessively accurate predictions that cause the divergence between surprisal and human reading times. however, newly-trained lm variants that are smaller than those of contemporary standards reveal a 'tipping point' at convergence, which indicates that a certain amount of model capacity is necessary for lms to correctly learn humanlike expectations. finally, across both experiments, model capacity does not seem to modulate the relationship between perplexity and fit to human reading times, with data points from different lm variants forming a continuous curve between log perplexity and ∆ll. this suggests that transformer-based lms of different capacities share a similar inductive bias that initially improves the fit of surprisal estimates to human reading times but begins to have an adverse effect on it with large amounts of training data.