Do Transformer Models Show Similar Attention Patterns to Task-Specific Human Gaze? (2022)

Abstract: Learned self-attention functions in state-of-the-art NLP models often correlate with human attention. We investigate whether self-attention in large-scale pre-trained language models is as predictive of human eye fixation patterns during task-reading as classical cognitive models of human attention. We compare attention functions across two task-specific reading datasets for sentiment analysis and relation extraction. We find the predictiveness of large-scale pre-trained self-attention for human attention depends on ‘what is in the tail’, e.g., the syntactic nature of rare contexts. Further, we observe that task-specific fine-tuning does not increase the correlation with human task-specific reading. Through an input reduction experiment we give complementary insights on the sparsity and fidelity trade-off, showing that lower-entropy attention vectors are more faithful.

## introduction the usefulness of learned self-attention functions often correlates with how well it aligns with human attention @xcite @xcite @xcite . in this paper, we evaluate how well attention flow @xcite in large language models, namely bert @xcite , roberta @xcite and t5 @xcite , aligns with human eye fixations during task-specific reading, compared to other shallow sequence labeling models @xcite and a classic, heuristic model of human reading @xcite . we compare the learned attention functions and the heuristic model across two taskspecific english reading tasks, namely sentiment analysis (sst movie reviews) and relation extraction (wikipedia), as well as natural reading, using a publicly available dataset with eye-tracking recordings of native speakers of english @xcite . ## pre-trained language models vs cognitive models church and liberman (2021) discuss how nlp has historically benefited from rationalist and empiricist methodologies, something that holds for cognitive modeling in general. the vast majority of application-oriented work in nlp today relies on pre-trained language models or other largescale data-driven models, but in cognitive modeling, most approaches remain heuristic and rulebased, or hybrid, e.g., relying on probabilistic language models to quantify surprisal @xcite . this is for good reasons: cognitive modeling values interpretability (even) more, often suffers from data scarcity, and is less concerned with model reusability across different contexts. this paper presents a head-to-head comparison of the e-z reader and pre-trained transformerbased language models. we are not the first to evaluate pre-trained language models and largescale data-driven models as if they were cognitive models. @xcite , for example, use representational similarity analysis to correlate sentence encodings in pre-trained language models with fmri signals; @xcite correlate sentence encodings with gaze-derived representations. more generally, it has been argued that cognitive evaluations are in some cases practically superior to standard evaluation methodologies in nlp @xcite . we return to this in the discussion and conclusion §6. commonly, pre-trained language models are disregarded as cognitive models, since they are most often implemented as computationally demanding batch learning algorithms, processing data "at once". @xcite points out that this is an artefact of their implementation, and online learning of pre-trained language models is possible, yet impractical. generally, several researchers have argued for taking pre-trained language models seriously as cognitive models @xcite @xcite . in the last section, §6, we discuss some of the implications of comparisons of pre-trained language models and cognitive models -for cognitive modeling, as well as for nlp. in our experiments, we focus on transformer architectures that are currently the dominating pre-trained language models and a de facto baseline for modern nlp research. ## analyses in addition to our main result -that pre-trained language models are competitive to heuristic cognitive models in predicting human eye fixations during reading -we present a detailed analysis, investigating what our main results depend on, where 3 we have experimented with oracle analyses selecting the maximally correlating attention head in the last layer for each sentence and find that correlations are generally weaker than with attention flow. ## discussion and conclusion in this paper, we have compared attention and relevance mechanisms of a wide range of models to human gaze patterns when solving sentiment classification on sst movie reviews and relation extraction on wikipedia articles. we generally found that transformer architectures are competitive with the e-z reader, but only when computing attention flow scores. we generally saw weaker correlations for relation extraction on wikpedia, presumably due to simpler sentence structures and the occurrence of polarity words. in the following, we discuss implications of our findings on nlp and cognitive science in more detail. lessons for nlp one implication of the above for nlp follows from the importance of attention flow in our experiments: using human gaze to regularize or supervise attention weights has proven effective in previous work ( §5), but we observed that correlations with task-specific human attention increase significantly by using layer-dependent attention flow compared to using raw attention weights. this insight motivates going beyond regularizing raw attention weights or directly injecting human attention vectors during training, to instead optimize for correlation between attention flow and human attention. jointly modeling language and human gaze has recently shown to yield competitive performance on paraphrase generation and sentence compression while resulting in more taskspecific attention heads @xcite . for this study natural gaze patterns were also simulated using the e-z reader. another potential implication concerns interpretability. it remains an open problem how best to interpret self-attention modules @xcite , and whether they provide meaningful explanations for model predictions. including gradient information to explain transformers has recently been considered to improve their interpretability @xcite . a successful explanation of a machine learning model should be faithful, human-interpretable and practical to apply @xcite . faithfulness and practicality is often evaluated using automated procedures such as input reduction experiments or measuring time and model complexity. by contrast, judging human-interpretability typically requires costly experiments in well-controlled settings and obtaining human gold-standards for interpretability remain difficult @xcite . using gaze data to evaluate the faithfulness and trustworthiness of machine learning models is a promising approach to increase model transparency. lessons for cognitive science attention flow in transformers, especially for bert models, correlates surprisingly well with human task-specific reading, but what does this tell us about the shortcomings of our cognitive models? we know that word frequency and semantic relationships between words influence word fixation times @xcite . in our experiments, we see relatively high correlation between human fixations and the inverse word probability baseline which raises the question to what extent reading gaze is driven by low-level pat-terns such as word frequency or syntactic structure in contrast to more high-level semantic context or wrap-up effects. in computer vision, cognitively inspired bottomup models, e.g., using intensity and contrast features, are able to explain at most half of the gaze fixation information in comparison to the human gold standard @xcite . the robustness of the e-z reader on movie reviews is likely due to its explicit modeling of low-level properties such as word frequency or sentence length. bert was recently shown to be primarily modeling higherorder word co-occurrence statistics @xcite . we argue that while transformers are limited, e.g., in not capturing the dependency of human gaze on word length @xcite , cognitive models seem to underestimate the role of word co-occurrence statistics. during reading, humans are faced with a tradeoff between the precision of reading comprehension and reading speed, by avoiding unnecessary fixations @xcite . this trade-off is related to the input reduction experiments performed in section 4. here, we observe that shallow methods score well at being sparse and effective in changing model output towards the correct class, but produce only weak correlation to human reading patterns when compared to layered language models. in comparison, extracted attention flow from pre-trained transformer models correlates much better with human attention, but offers less sparse token attention. in other words, our results show that task-specific reading is sub-optimal relative to solving tasks and heavily regularized by natural reading patterns (see also our comparison of task-specific and natural reading in section 4).