Locally Biased Transformers Better Align with Human Reading Times (2024)

Abstract: Recent psycholinguistic theories emphasize the interdependence between linguistic expectations and memory limitations in human language processing. We modify the self-attention mechanism of a transformer model to simulate a lossy context representation, biasing the model’s predictions to give additional weight to the local linguistic context. We show that surprisal estimates from our locally-biased model generally provide a better fit to human psychometric data, underscoring the sensitivity of the human parser to local linguistic information.

## introduction in recent years, transformer models @xcite have gained prominence in psycholinguistics due to their impressive predictive performance in forecasting psychometric measurements such as reading times @xcite @xcite @xcite . these models excel at capturing complex linguistic dependencies, making them valuable tools in analyzing human language-processing behaviors. much of the work that relates probabilistic estimates from transformer models with human processing has been conducted within the framework of surprisal theory, which posits that the difficulty experienced during processing is proportional to the negative logarithm of the probability of a word given its preceding context. in this context, transformer models, which are able to generate highly accurate probabilistic predictions for sequences of text, have been instrumental in providing empirical support for surprisal theory @xcite @xcite . however, despite their substantial predictive power, transformer models exhibit some design features that lack cognitive plausibility. one significant departure from human language processing is their ability to access in parallel the entire linguistic context within their input size. unlike these models, human language comprehension is inherently incremental (smith and @xcite . humans eagerly integrate in their representation of the context each linguistic unit as soon as it is encountered, and they cannot typically store in working memory the whole linguistic context. thus, the transformer model's all-at-once approach to processing information starkly contrasts with the sequential and resource-constrained manner in which humans receive and interpret linguistic input, suggesting a need for models that more closely mirror human cognitive limitations and processing strategies. to address these limitations, we introduce a modification to the self-attention mechanism of the transformer model, aimed at simulating a lossy memory representation, where linguistic units that are further away from the current word are assigned exponentially decaying attention scores. by doing so, our model aims to replicate the kind of linguistic processing that characterizes the human language parser, where recent information plays a significant role @xcite . the evaluation of our locally-biased transformer model involves the employment of the surprisali.e., negative log probability -it assigns to words in context to predict human psychometric data, considering five large-scale datasets of eye movements and self-paced reading times in english. we show that our locally-biased model provides surprisal estimates that align more closely with human psychometric data than a standard pre-trained model. ## related work models that can explain the cognitive cost associated with sentence processing can be broadly divided into expectation-and memory-based theories. expectation-based theories (such as surprisal theory; @xcite emphasize the role of contextual predictability as a core determinant of processing demands. support for such theo-ries has come from several studies demonstrating reduced cognitive load in response to predictable words (e.g., @xcite @xcite . memory-based theories, in contrast, are based on the idea that integrating the upcoming words into the context representation depends on the retrieval @xcite and storage @xcite of previous words in working memory. support for memory-based theories comes from the difficulty in integrating words that are linearly distant in a sentence (dependency locality effects; @xcite . in recent years, there have been proposals to reconcile expectation-and memory-based approaches into unified models. while the first combined theories posited limited @xcite or no interaction between memory and predictability (rasmussen and schuler, 2018; see @xcite , some recently developed frameworks account for complex interactions between the two @xcite . in particular, lossy-context surprisal theory (henceforth lcst; @xcite holds that the processing difficulty associated with a word is proportional its surprisal, conditioned by a lossy (i.e., noisy) memory representation of the context. @xcite presented a computationallyspecified model of lcst (resource-rational lcst) that computes retention probabilities for each word in the context, based on the word's identity and position in the sentence. similarly, @xcite have shown that reducing the number of words in input to language models improves the fit of the surprisal estimates to human reading times. our modelling approach is reminiscent of lcst in that it assumes that the processing cost associated with a word is proportional to its surprisal, conditioned by the previous context where linearly distant words contribute less to its prediction. in our modelling effort, we modify the attention scores of a transformer model to mimic the human difficulty in retrieving distant linguistic elements. we are not the first in drawing a parallelism between the self-attention mechanism and (cue-based) memory retrieval @xcite @xcite . indeed, like the selfattention mechanism scores the weights to assign to the words in input based on the compatibility between keys and queries, cue-based retrieval theories posit that items in working memory are accessed by comparing the retrieval cues of the current word with the features of the items in working memory @xcite . our choice to bias the transformer model's retrieval process towards the recent linguistic context is supported by vast evidence in psycholinguistics showing that local information holds a privileged role in human language processing, with a chief example being word frequency. indeed, it is well-known that the frequency of a word (which is proportional to its unigram probability) influences its reading times above and beyond its contextual predictability @xcite . furthermore, other studies have detected an effect of n-gram surprisal beyond the effect of surprisal as calculated from larger sentential contexts @xcite . note that the idea of assigning reduced weights to distal elements has a long-standing tradition in psychologically-oriented computational models of semantic memory: one of the first distributional semantic models, the hyperspace analogue to language (hal; @xcite , weighs word-by-word co-occurrences as a function of their linear distance. it is noteworthy that the idea that human language processing privileges recent information was explicitly implemented in early work in computational semantics, and lost with later developments in the field. ## methods and materials we modified the pre-trained gpt-2 model @xcite to increase the attention weights associated to the nearby words. we conducted our analyses on the smallest gpt-2 variant, as it has been proven to be particularly effective at modelling human reading times @xcite . all the code supporting our analyses is publicly available on github. foot_0 ## results the results of our analyses are reported in figure 2 . our locally biased transformer assigned higher average per-word surprisal values to the input texts across all datasets (a), showing a reduced autoregressive accuracy with respect to its unbiased counterpart. the obtained surprisal estimates correlated more strongly with log frequency values (b) and with the behavioral responses considered (c). furthermore, our comparison between the biased and the original gpt-2 model (d) revealed that our modification of the attention mechanism caused a substantial increase in predictive performance in three behavioral datasets, encompassing both eyetracking (meco, ∆@xmath0.55; provo, ∆@xmath1.72) and self-paced reading (brown, ∆@xmath2.55). our manipulation had no effect in the ucl corpus (ucl et , ∆@xmath3.59; ucl spr , ∆@xmath4.06) and resulted in a poorer fit in the natstor dataset (∆@xmath5.31). ## discussion in this study, we have demonstrated that a modification of the gpt-2 model to emphasize local context via a locally-biased attention mechanism results in surprisal estimates that are more strongly correlated with human reading times, and generally display a better fit to human psychometric data. an exception to this second observation is offered by the natstor and ucl corpora; in appendix a, we report tentative evidence that the model improvement seems to be related to the average sentence length in the corpus. in particular, our locally-biased attention seems to be particularly beneficial in cases where the sentences are longer. this finding is compatible with the idea of a lossy representation of the context, where memory constraints become more marked for longer text sequences. future approaches could consider dynamically manipulating the α parameter as a function of sentence lenght, adjusting the strength of the bias to cases where the human memory is taxed more strongly. in lcst, the way memory representations degrade typically results in a word's contextual processing cost approaching its context-independent processing cost, as predicted by its standalone probability @xcite . essentially, as the fidelity of a listener's memory representations diminishes, their anticipations increasingly align with the prior, context-independent unigram probability. our findings empirically demonstrate that this is the case, as the surprisal estimates from our locallybiased transformer tend to regress towards word frequency estimates (figure 2 , b ). while our intervention on the attention mechanism is directly inspired by lcst, it should be noted that this implementation does not respect all the assumptions of the theory. in particular, lcst posits as an assumption the inaccessibility of the context (claim 3); here, the context is always available to the model, albeit reduced attention weights are assigned to the elements that are linearly distant from each other. importantly, our modification of the attention mechanism resulted in models that were less performant in next-word prediction (see figure 2 , a ). this is of course to be expected, as the addition of the exponential decay bias to the attention scores produces final attention weights that deviate from the ones that have been optimized for autoregression. nonetheless, our results show that a worse nlp model can constitute a better cognitive model in terms of fit to psychometric data. this result challenges the quality-power hypothesis (qp; @xcite , which posits that more accurate language models (i.e., models whose surprisal estimate better approximate the values from the data-generating distribution) should provide surprisal estimates that better fit behavioral data. however, qp does not hold if the probabilistic information that humans deploy in real time is systematically biased with respect to the datagenerating distribution. one example of this systematic deviation is offered by the sensitivity of the human parser to local word co-occurrence statistics @xcite , which is exactly what we model in the present paper. thus, our results show that human-like language processing might inherently involve biases and limitations that deviate from optimal statistical models.