Reverse-Engineering the Reader (2024)

Abstract: Numerous previous studies have sought to determine to what extent language models, pretrained on natural language text, can serve as useful models of human cognition.In this paper, we are interested in the opposite question: whether we can directly optimize a language model to be a useful cognitive model by aligning it to human psychometric data.To achieve this, we introduce a novel alignment technique in which we fine-tune a language model to implicitly optimize the parameters of a linear regressor that directly predicts humans’ reading times of in-context linguistic units, e.g., phonemes, morphemes, or words, using surprisal estimates derived from the language model. Using words as a test case, we evaluate our technique across multiple model sizes and datasets and find that it improves language models’ psychometric predictive power.However, we find an inverse relationship between psychometric power and a model’s performance on downstream NLP tasks as well as its perplexity on held-out test data.While this latter trend has been observed before (Oh et al., 2022; Shain et al., 2024), we are the first to induce it by manipulating a model’s alignment to psychometric data.

## introduction language comprehension is thought to be predictive and incremental. research on reaction times @xcite , fixation patterns @xcite , and brain activations @xcite suggests that comprehenders anticipate upcoming linguistic units based on the context in which they occur @xcite . 1 in addition, a large body of evidence shows that when linguistic units are unexpected, they require more cognitive effort to process @xcite @xcite , inter *equal contribution. 1 our code is available at https://github.com/samuki/ reverse-engineering-the-reader . in this study, we are interested in reverseengineering the part of the language processing system responsible for predicting abstract linguistic units and testing it by measuring its ability to predict reading times. 2 to do so, we need, first, to establish a theoretical link between predictability and reading times. for this, we draw on surprisal theory @xcite , which posits that the cognitive effort to process a unit is proportional to its surprisal-the unit's negative log probability given the preceding context. implicitly, surprisal theory assumes that a comprehender maintains a probability distribution over upcoming units, i.e., it assumes a human language model. however, this human language model is a theoretical construct, and cannot be observed directly. thus, most previous work that tests surprisal theory has done so using probability estimates derived from a language model trained on large swathes of human-written text. under this paradigm, it has been observed that surprisal estimates derived from language models, fit with regularized maximumlikelihood estimation on large corpora, do yield significant predictors although these findings vary based on the quality @xcite @xcite , size, and training-data set size @xcite of the model. one current line of research therefore seeks to uncover what characteristics of pretrained lms produce better predictors of human reading times, and what this tells us about the human language processing system. our paper asks a simple question: instead of assessing the ability of pretrained lms to serve as psycholinguistic predictors, can we directly esti-mate (or fine-tune) a language model so that its surprisal estimates become better predictors of processing effort for linguistic units? we frame this problem as one of aligning the language model to human data @xcite @xcite @xcite . however, in contrast to much previous work, which uses human-model alignment to obtain improvement on natural language processing tasks, e.g., summarizing text @xcite or producing non-toxic outputs @xcite , we seek to align lms to be better psychometric predictors. particularly, we aim to directly align models to human reading data by optimizing the parameters of the statistical models typically used to evaluate the psychometric fit. while recent approaches like direct preference optimization (dpo; @xcite are designed to optimize a model's parameters based on human preferences, they rely on pairwise preference data, which is not applicable to real-valued psychometric data. thus, we propose a novel alignment technique that allows us to directly optimize the language model's parameters in such a way that it serves as a better predictor of real-valued psychometric data. specifically, we fine-tune the language model to implicitly optimize the coefficients of a linear regression that predicts the reading time of an individual unit. we test our technique on three english-language reading datasets and find that it increases the statistical fit of a linear regressor in terms of the likelihood it assigns to reading times on a held-out test set. we also observe a positive relationship between a model's psychological predictive power and its perplexity. while it has been observed that better lms are better psychological models of reading up to a point @xcite @xcite , after a certain model size, their fit to human reading times decreases @xcite . in other words, beyond an inflection point, better lms are worse predictors of human reading times. through our alignment procedure, we are able to demonstrate the contrapositive, namely that as we causally make our language models' outputs more aligned with reading, they become worse at predicting the next word. ## psycholinguistics background put concisely, the goal of this paper is to reverse-engineer p h , a person's internal language model from psychometric data collected through experimentation. our reasoning is as follows: if we can align an existing language model p θ to more accurately predict such psychometric data, p θ will also more closely resemble p h . ## aligning lms to psychometric data as discussed in §2.4, the psychometric predictive power of a language model is typically evaluated by assessing the predictive power of surprisal estimates ι θ (u | c) of a linguistic unit u in a context c with respect to human reading times. rather than evaluating a language model's psychometric predictive power, in this study, we ask whether we can fine-tune language models to increase their psychometric predictive power. we treat this as an alignment problem @xcite @xcite @xcite . let p ref denote a pretrained language model that will serve as a reference. further, let p θ denote the language model we seek to fine-tune, generally initialized to p ref . our goal is to align p θ such that its surprisal estimates are more directly correlated with psychometric data in comparison to p ref . by means of implicit differentiation, we derive an objective that allows us to perform such psychometric alignment. ## experimental design in this section, we discuss how we experiment with fine-tuning a language model using the objective defined in eq. ( 10 ) and the reward approximation ## results we now return to our main question: does our objective align language models more closely to human reading times compared to pretrained models? we first analyze the effect of maximizing the unregularized reward and later, in §5.4, extend our analysis to the kl-regularized objective. ## discussion we now discuss the broader implications of our results with respect to both cognitive modeling and natural language processing. one recurring theme in this paper is the relationship between p θ , a probability distribution over strings estimated from large corpora, and p h , the distribution implicated during cognitive tasks. in terms of cognitive modeling, it is widely accepted that it is useful to obtain a 9374 g p t 2 -l d gpt2-m λ = 0 gpt2-m λ = 500 gpt2-s λ = 0 gpt2-s λ = 500 p θ that is as close as possible to p h in order to test and refine psychological theories. while previous work has proposed alignment procedures between lms and brain data @xcite , within the realm of psycholinguistics, researchers have generally opted for whatever model is the current state of the art. only recently have contributions started to search for which combinations of training data, model size, and model architecture produce the best cognitive alignment @xcite . our results suggest an alternative to this search by directly aligning language models to psychometric data. we are optimistic that such aligned models will enable a more precise evaluation of psycholinguistic theories. of course, it remains an open question to what extent alignment on reading generalizes to other cognitive tasks. we have discussed p h in terms of a single hypothesized construct, but it is likely that people's predictions change with task demands, e.g., when skimming versus proofreading a text. testing whether our results hold for other types of psychometric predictive tasks is an important question for future research. the main technical contribution of this work is a technique that aligns language models more closely to human psychometric data. inspired by the implicit parameter optimization in dpo, our approach goes beyond the original bradley-terry assumption and demonstrates that it is possible to fine-tune under other implicit statistical models. exploring a wider range of such models goes hand in hand with exploring new sources of human psychometric data, as we have done here with reading times. psychologists have devised methods for collecting a diverse array of cognitive signals, including eeg, fmri, mouse-tracking, and self-paced reading, to name a few. aligning models, using the method proposed here, on such data or combinations thereof, will finally, what do our results say about the relationship between cognitive modeling and other domains in nlp? at first glance, they seem to suggest that alignment with reading times is not an effective strategy to broadly boost performance on nlp tasks. although kl regularization reduces the increase of the language model's perplexity on heldout data, we still observe decreased performance on downstream nlp tasks. however, given the lack of reading time data for lm-generated text, we relied on pre-existing eye-tracking datasets. future studies could experiment with text generated by language models, particularly as new eye-tracking datasets for lm-generated text are being developed @xcite . furthermore, @xcite derive the optimal solution to the kl regularized rl objective in dpo, while our study excluded the kl term when deriving optimal coefficients. future work could investigate approaches closer to dpo to compute the optimal coefficients. ## conclusion we have presented a novel technique to align language models with human reading data by implicitly optimizing the parameters of a linear regression model. our experiments on held-out test sets show our method reliably improves the predictive power of language models with various parameter counts on human reading times. furthermore, our findings confirm previous research on the inverse relationship between perplexity and psychometric predictive power. we believe our results pave the way for better assessment of psychological theories using more cognitively aligned language models.