Entropy- and Distance-Based Predictors From GPT -2 Attention Patterns Predict Reading Times Over and Above GPT -2 Surprisal (2022)

Abstract: Transformer-based large language models are trained to make predictions about the next word by aggregating representations of previous tokens through their self-attention mechanism. In the field of cognitive modeling, such attention patterns have recently been interpreted as embodying the process of cue-based retrieval, in which attention over multiple targets is taken to generate interference and latency during retrieval. Under this framework, this work first defines an entropy-based predictor that quantifies the diffuseness of self-attention, as well as distance-based predictors that capture the incremental change in attention patterns across timesteps. Moreover, following recent studies that question the informativeness of attention weights, we also experiment with alternative methods for incorporating vector norms into attention weights. Regression experiments using predictors calculated from the GPT-2 language model show that these predictors deliver a substantially better fit to held-out self-paced reading and eye-tracking data over a rigorous baseline including GPT-2 surprisal.

## introduction much work in broad-coverage sentence processing has focused on studying the role of expectation operationalized in the form of surprisal @xcite using language models (lms) to define a conditional probability distribution of a word given its context @xcite . recently, this has included transformer-based language models @xcite @xcite . however, expectation-based accounts have empirical shortcomings, such as being unable to fully account for garden-path effects @xcite or predict the timing of delays in certain constructions @xcite . for this reason, some research has begun to focus on the effects of memory and attention using predictors calculated from language model representations. for example, ryu and @xcite recently drew connections between the self-attention patterns of transformers @xcite and cue-based retrieval models of sentence comprehension (e.g. @xcite . their attention entropy, which quantifies the diffuseness of the attention weights over previous tokens, showed patterns that are consistent with similarity-based interference observed during the processing of subject-verb agreement. however, these results relied on identifying one attention head specialized for the nsubj dependency, and an aggregated version of this predictor was not very strong in predicting naturalistic reading times in the presence of a surprisal predictor (ryu and @xcite . this work therefore defines and evaluates several memory-and attention-based predictors derived from the self-attention patterns of the transformerbased gpt-2 language model @xcite on two naturalistic datasets, in the presence of a strong gpt-2 surprisal baseline. first, normalized attention entropy expands upon ryu and @xcite attention entropy by re-normalizing the attention weights and controlling for the number of tokens in the previous context. additionally, three distance-based predictors that quantify the shift in attention patterns across consecutive timesteps are presented, based on the idea that the reallocation of attentional focus entails processing difficulty. moreover, motivated by work on interpreting large language models that question the connection between attention weights and model predictions (e.g. jain and wallace, 2019), the norm-based analysis of the transformed vectors @xcite is newly applied to gpt-2 in this work to inform alternative formulations of attention weights. for example, it has been observed that while large language models tend to place high attention weights on special tokens (e.g. 
 of gpt-2), these tokens contribute very little to final model predictions as their 'value' vectors have near-zero norms @xcite . attention weight formulations that incorporate the norms of the transformed vectors should therefore alleviate the over-representation of such special tokens and represent the contribution of each token more accurately. results from regression analyses using these predictors show significant and substantial effects in predicting self-paced reading times and eye-gaze durations during naturalistic reading, even in the presence of a robust surprisal predictor. additionally, alternative formulations of attention weights that incorporate the norms of the transformed vectors are shown to further improve the predictive power of these predictors. ## background this section provides a mathematical definition of the self-attention mechanism underlying the gpt-2 language model and describes alternative normbased formulations of attention weights. ## experiment 1: evaluation of predictors on human reading times in order to evaluate the contribution of the entropyand distance-based predictors, regression models containing commonly used baseline predictors, surprisal predictors, and one predictor of interest were fitted to self-paced reading times and eye-gaze durations collected during naturalistic language processing. in this work, we adopt a statistical procedure that directly models temporal diffusion (i.e. a lingering reponse to stimuli) by estimating continuous impulse response functions and controls for overfitting by assessing the external validity of these predictors through a non-parametric test on held-out data. ## discussion and conclusion this work builds on recent efforts to derive memory-and attention-based predictors of processing difficulty to complement expectation-based accounts of sentence processing. based on the observation that the self-attention patterns of transformer-based language models can be interpreted as embodying the process of cue-based retrieval, an entropy-based predictor that quantifies the diffuseness of self-attention was first defined. moreover, based on the idea that reallocation of attention may incur processing difficulty, distance-based predictors that capture the incremental change in attention patterns across timesteps were defined. regression results using these predictors calculated from the gpt-2 language model showed that these entropy-and distance-based predictors deliver a substantially better fit to self-paced reading and eye-tracking data over a strong baseline including gpt-2 surprisal. to our knowledge, this work is the first to report robust effects of transformer attention-based predictors in predicting reading times of broadcoverage naturalistic data. this provides support for ryu and @xcite observation that the self-attention mechanism of transformers embodies the process of cue-based retrieval, and further suggests that representations that exhibit similaritybased interference can be learned from the selfsupervised next-word prediction task. additionally, the strength of the distance-based predictors further demonstrates the potential to bring together expectation-and memory-based theories of sentence processing under a coherent framework.