How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformers (2022)

Abstract: The attention mechanism is considered the backbone of the widely-used Transformer architecture. It contextualizes the input by computing input-specific attention matrices. We find that this mechanism, while powerful and elegant, is not as important as typically thought for pretrained language models. We introduce PAPA, a new probing method that replaces the input-dependent attention matrices with constant ones—the average attention weights over multiple inputs. We use PAPA to analyze several established pretrained Transformers on six downstream tasks. We find that without any input-dependent attention, all models achieve competitive performance—an average relative drop of only 8% from the probing baseline. Further, little or no performance drop is observed when replacing half of the input-dependent attention matrices with constant (input-independent) ones. Interestingly, we show that better-performing models lose more from applying our method than weaker models, suggesting that the utilization of the input-dependent attention mechanism might be a factor in their success. Our results motivate research on simpler alternatives to input-dependent attention, as well as on methods for better utilization of this mechanism in the Transformer architecture.

## introduction pretrained transformer @xcite models have enabled great progress in nlp in recent years @xcite @xcite @xcite . a common belief is that the backbone of the transformer model-and pretrained language models (plms) in particular-is the attention mechanism, which applies multiple attention heads in figure 1 : illustration of the papa method, which measures how much plms use the attention mechanism. papa replaces the input-dependent attention matrices (left) with constant ones (right). we then measure the performance gap between the two. moderate drop indicates minor reliance on the attention mechanism. interestingly, recent work found that attention patterns tend to focus on constant (inputindependent) positions @xcite , while other works showed that it is possible to pretrain language models where the attention matrices are replaced with constant matrices without major loss in performance @xcite @xcite . a natural question that follows is how much standard plms, pretrained with the attention mechanism, actually rely on this input-dependent property. this paper shows that they are less dependent on it than previously thought. we present a new analysis method for plms: probing analysis for plms' attention (papa). for each attention head h, papa replaces the attention matrix with a constant one: a simple average of the attention matrices for h computed on some unlabeled corpus. replacing all attention matrices with such constant matrices results in an attention-free variant of the original plm (see fig. 1 ). we then compute, for some downstream tasks, the probing performance gap between an original model and its attention-free variant. this provides a tool to quantify the models' reliance on attention. intuitively, a larger performance drop indicates that the model relies more on the input-dependent attention mechanism. we use papa to study three established pretrained transformers: bert @xcite , roberta @xcite , and deberta @xcite , each with base-and large-sized versions. we evaluate these models on six diverse benchmarks, spanning text classification and structured prediction tasks. our results suggest that attention is not as important to pretrained transformers as previously thought. first, the performance of the attentionfree variants is comparable to original models: an average relative drop of only 8%. second, replacing half of the attention matrices with constant ones has little effect on performance, and in some cases may even lead to performance improvements. interestingly, our results hint that better models use their attention capability more than weaker ones; when comparing the effect of papa on different models, we find that the better the model's original performance is, the more it suffers from replacing the attention matrices with constant ones. this suggests a potential explanation for the source of the empirical superiority of some models over othersthey make better use of the attention mechanism. this work grants a better understanding of the attention mechanism in pretrained transformers. it also motivates further research on simpler or more efficient transformer models, either for pretraining @xcite @xcite or potentially as an adaptation of existing pretrained models @xcite @xcite . it also provides a potential path to improve the transformer architecture-by designing inductive bias mechanisms for better utilization of attention @xcite . finally, our work may contribute to the "attention as explanation" debate @xcite @xcite . by showing that some plms can perform reasonably well with constant matrices, we suggest that explanations arising from the attention matrices might not be crucial for models' success. we summarize our main contributions. (1) we present a novel probing method-papa-which quantifies the reliance of a given plm on its attention mechanism by "disabling" that mechanism for this plm. (2) we apply papa to six leading plms, and find that our manipulation leads to modest performance drops on average, which hints that attention might not be as important as thought. (3) we show that better-performing plms tend to suffer more from our manipulation, which suggests that the input-dependent attention is a factor in their success. (4) finally, we release our code and experimental results. foot_0 ## background: attention in transformers transformers consist of interleaving attention and feed-forward layers. in this work, we focus on transformer encoder models, such as bert, which are commonly used in many nlp applications. the (multi headed) self-attention module takes as input a matrix @xmath0, where n denotes the number of input tokens, each represented as a d-dimensional vector. each attention layer consists of h heads, and each head @xmath1, . . . , h} has three learnable matrices: multiplying them with the input x results in: q h , k h , v @xmath2, keys and values, respectively). the queries and the keys compute a n × n attention weight matrix a h between all pairs of tokens as softmax-normalized dot products: foot_2 importantly, the attention matrix a h is inputdependent, i.e., defined by the input x. this property is considered to be the backbone of the attention mechanism @xcite . an intriguing question is the extent to which plms actually rely on the attention mechanism. in the following, we study this question by replacing the attention matrices of plms with constant matrices. we hypothesize that if models make heavy use of attention, we will see a large drop in performance when preventing the model from using it. as shown below, such performance drop is often not observed. ## the papa method we present papa, a probing method for quantifying the extent to which pretrained transformer models use the attention mechanism. papa works by replacing the transformer attention weights with constant matrices, computed by averaging the values of the attention matrices over unlabeled inputs (sec. 3.1). papa also allows for replacing any subset (not just all) of the attention matrices. we propose a method for selecting which heads to replace (sec. 3.2). the resulting model is then probed against different downstream tasks (sec. 3.3). the performance difference between the original and the new models can be seen as an indication of how much the model uses its attention mechanism. ## experiments we now turn to use papa to study the attention usage of various plms. ## further analysis we present an analysis of papa, to better understand its properties. we first discuss the patterns of the constant matrices produced by papa (sec. 5.1). next, we consider other alternatives to generating constant matrices (sec. 5.2); we then examine whether the constant matrices are data-dependent (sec. 5.3); we continue by exploring alternative methods for selecting which attention heads to replace (sec. 5.4). finally, we present mlm results, and discuss the challenges in interpreting them (sec. 5.5). in all experiments below, we use roberta base . roberta large experiments show very similar trends, see app. c. ## conclusion in this work, we found that plms are not as dependent on their attention mechanism as previously thought. to do so, we presented papa-a method for analyzing the attention usage in plms. we applied papa to several widely-used plms and six downstream tasks. our results show that replacing all of the attention matrices with constant ones achieves competitive performance to the original model, and that half of the attention matrices can be replaced without any loss in performance. we also show a clear relation between a plm's aggregate performance across tasks and its degradation when replacing all attention matrices with constant ones, which hints that performant models make better use of their attention. our results motivate further work on novel transformer architectures with more efficient attention mechanisms, both for pretraining and for knowledge distillation of existing plms. they also motivate the development of transformer variants that improve performance by making better use of the attention mechanism.