Probabilistic Predictions of People Perusing: Evaluating Metrics of Language Model Performance for Psycholinguistic Modeling (2020)

Abstract: By positing a relationship between naturalistic reading times and information-theoretic surprisal, surprisal theory (Hale, 2001; Levy, 2008) provides a natural interface between language models and psycholinguistic models. This paper re-evaluates a claim due to Goodkind and Bicknell (2018) that a language model’s ability to model reading times is a linear function of its perplexity. By extending Goodkind and Bicknell’s analysis to modern neural architectures, we show that the proposed relation does not always hold for Long Short-Term Memory networks, Transformers, and pre-trained models. We introduce an alternate measure of language modeling performance called predictability norm correlation based on Cloze probabilities measured from human subjects. Our new metric yields a more robust relationship between language model quality and psycholinguistic modeling performance that allows for comparison between models with different training configurations.

## introduction naturalistic reading times are known to be affected by incongruities between the current word and the context created by preceding words @xcite @xcite . words that are unexpected within their contexts are fixated for longer periods, while predictable words are fixated for a shorter amount of time or skipped altogether. this observation has been described formally by the suprisal theory of sentence processing @xcite , which posits that the difficulty of processing a particular word is directly proportional to its surprisal, defined as the negative logarithm of its probability given its preceding context. by relating psychometric observations with information-theoretic concepts, surprisal theory provides a natural bridge between psycholinguistic modeling on the one hand and language modeling on the other. while modeling studies for reading times traditionally use surprisals obtained from probabilistic parsers @xcite @xcite or directly from probabilistic context-free grammars @xcite , surprisals from n-gram models @xcite and simple recurrent network language models @xcite have also been incorporated into reading-time studies. more recent work has sought to leverage the advances in language modeling made possible by neural nlp in order to determine whether modern techniques can yield more reliable estimates of surprisal for cognitive modeling. these studies have investigated the psycholinguistic capabilities of several neural architectures, including long short-term memory networks and gated recurrent unit networks (aurnhammer and @xcite , transformers @xcite , and gpt-2 @xcite . in general, language models achieving a better perplexity have been found to yield better psycholinguistic models via the surprisal estimates they furnish, with @xcite proposing a linear relationship between the two factors. this paper revisits goodkind and bicknell's proposed linear relationship between perplexity and psycholinguistic modeling performance. we address two drawbacks of their analysis, which hinder its applicability to the state of the art in language modeling. firstly, goodkind and bicknell only consider n-gram models and a small lstm language model with 50 hidden units. these constitute a small and relatively weak collection of language models compared to the diverse array of techniques and vast computing resources that are available today. secondly, the reliability of perplexity as a measure of language modeling performance is highly dependent on training configurations, such that only perplexities for models with the same vocabulary are comparable (see @xcite . this is especially problematic given the rise of large-scale, general-purpose pretrained language models, since vocabulary cannot be controlled for when comparing different pretrained models. to address these issues, we first expand goodkind and bicknell's empirical coverage by extending their analysis to long short-term memory networks (lstms, @xcite and transformers @xcite of varying sizes and training corpora, along with four large pre-trained models. we then propose an alternate metric of language modeling performance based on word probabilities obtained from humans using a cloze procedure @xcite , which allows different pre-trained models to be compared with one another despite differences in vocabulary. in line with previous work, we find that the relationship between perplexity and psycholinguistic modeling performance is weaker for lstms and transformers than for n-gram models, and not consistent across vocabularies. however, our clozebased metric yields a relationship which is robust to differences in model vocabulary and incorporates both our trained and our pre-trained models. based on this, we argue that our cloze-based metric is more suitable than perplexity for evaluating pre-trained models, and better reveals the relationship between language modeling performance and psycholinguistic modeling performance. the structure of this paper is as follows. section 2 describes our procedure for modeling reading times, and section 3 describes our language models as well as the cloze-based performance metric. our results are presented in section 4 and discussed in section 5. section 6 concludes. ## psycholinguistic modeling we follow the methodology of @xcite for psycholinguistic modeling. we model reading times using generalized additive mixed-effect models (gamms), as implemented in the mgcv r package @xcite . our gamms take several variables as input, optionally including language model surprisals. we fit a gamm for each language model, along with a baseline model that does not include surprisal values. we measure the psycholinguistic modeling performance of each model by a score called delta log likelihood (∆loglik), defined as the difference in log likelihood between the model's corresponding gamm and the baseline gamm. in the following subsections, we describe the reading time data used in our study and our procedure for fitting gamms. ## language modeling traditionally, language models form independent components of nlp systems, their outputs serving as inputs to downstream applications. however, following recent advances in transfer learning for nlp, modern neural language models are often designed not to compute word probabilities per se, but rather to provide representations of linguistic knowledge that can facilitate training on other tasks @xcite . under this paradigm, a single large neural network is pre-trained on a language modeling or other word prediction objective. this pre-trained language model can then be trained, or fine-tuned, on another task such as text classification @xcite or machine translation @xcite . both kinds of language models are considered in this paper. ## results our results are presented in the models trained on penn treebank and wikitext-2 achieve similar levels of ∆loglik, though the penn treebank models generally have a better perplexity while the wikitext-2 models generally have a better pnc. between lstms and transformers, neither architecture is consistently better than the other in terms of ∆loglik: in the remainder of this section, we describe specific observations about the relationship between language model performance and ∆loglik. ## discussion we focus our discussion on three topics. first, we consider the ways in which each of our experimental variables affects our models' psycholinguistic modeling performance. next, we briefly identify some properties of pnc as a metric, particularly in terms of its relationship with training corpus size. finally, we reflect on the implications of our results for language modeling and psycholinguistic modeling evaluation more generally. ## conclusion the results reported here suggest that for the purpose of modeling reading times, perplexity does not adequately reveal the relationship between language modeling performance and psycholinguistic modeling performance, especially when vocabulary cannot be controlled for. in contrast, pnc has proven to be a much better predictor of the psychometric capabilities of our language models. this finding is consistent with observations by @xcite and @xcite that cloze probabilities predict self-paced reading times better than corpus probabilities do. in addition to allowing models with different vocabularies to be compared with one another, pnc is much more strongly correlated with psycholinguistic modeling performance than perplexity, as demonstrated by our results on ∆loglik. more generally, as there is little to no correlation between perplexity and pnc, pnc can serve as a good supplement to perplexity for language model evaluation, providing information about model behavior that is not captured by the latter. we have also shown that model architecture, training dataset size, and training dataset composition all contribute substantially to the psycholinguistic modeling capabilities of language models. in particular, the importance of corpus size and composition is reflective of trends in transfer learning, in which advances in downstream nlp tasks are made by using language models to extract general linguistic information from large corpora. our analysis has shown that large corpora have the potential to provide considerable amounts of linguistic knowledge even through simple model architectures, as in the case of the lm1b n-grams. as new pre-trained language models are developed, especially with custom vocabularies that make a direct comparison of perplexities impossible, metrics such as pnc can serve as a valuable tool for assessing the quality of language models. in establishing a strong relationship between pnc and ∆loglik, we have demonstrated that pnc scores convey a psychometrically relevant notion of language model quality that directly measures the degree to which language models exhibit humanlike behavior. reliable metrics like pnc, which are robust to variations in model setup, have the potential to greatly improve our ability to understand the relationship between language models and language.