G lob E nc: Quantifying Global Token Attribution by Incorporating the Whole Encoder Layer in Transformers (2022)

Abstract: There has been a growing interest in interpreting the underlying dynamics of Transformers. While self-attention patterns were initially deemed as the primary option, recent studies have shown that integrating other components can yield more accurate explanations. This paper introduces a novel token attribution analysis method that incorporates all the components in the encoder block and aggregates this throughout layers. Through extensive quantitative and qualitative experiments, we demonstrate that our method can produce faithful and meaningful global token attributions. Our experiments reveal that incorporating almost every encoder component results in increasingly more accurate analysis in both local (single layer) and global (the whole model) settings. Our global attribution analysis significantly outperforms previous methods on various tasks regarding correlation with gradient-based saliency scores. Our code is freely available at https://github.com/mohsenfayyaz/GlobEnc .

## introduction the stellar performance of transformers @xcite has garnered a lot of attention to analyzing the reasons behind their effectiveness. the self-attention mechanism has been one of the main areas of focus @xcite @xcite . however, there have been debates on whether raw attention weights are reliable anchors for explaining model's behavior or not @xcite @xcite . recently, it was shown that incorporating vector norms should be an indispensable part of any attention-based analysis 1 @xcite , [cls] a deep and meaningful film . [sep] another limitation of the existing analysis techniques is that they are usually constrained to the analysis of single layer attributions. in order to expand the analysis to multi-layered encoder-based models in their entirety, an aggregation technique has to be employed. @xcite proposed two aggregation methods, rollout and max-flow, which combine raw attention weights across layers. despite showing the outcome of their method to be faithful to a model's inner workings in specific cases, the final results are still unsatisfactory on a wide range of fine-tuned models. additionally, gradient-based alternatives @xcite @xcite have been argued to provide a more robust basis for token attribution analysis @xcite @xcite . nonetheless, the gradient-based alternatives have not been able to fully replace attention-based counterparts, mainly due to their high computa-tional intensity. in this paper, we propose a new global token attribution analysis method (globenc) which is based on the encoder layer's output. in globenc, the second layer normalization is also included in the norm-based analysis of each encoder layer. to aggregate attributions over all layers, we applied a modified attention rollout technique, returning global scores. through extensive experiments and comparing the global attribution with the input token attributions obtained by gradient-based saliency scores, we show that our method produces faithful and meaningful results (figure 1 ). our evaluations on models with distinct pre-training objectives and sizes @xcite show high correlations with gradient-based methods in global settings. furthermore, with comparative studies on each aspect of globenc , we find that: (i) norm-based methods achieve higher correlations than weight-based methods; (ii) incorporating residual connections plays an essential role in token attribution; (iii) considering the two layer normalizations improve our analysis only if coupled together; and (iv) aggregation across layers is crucial for an accurate whole-model attribution analysis. in summary, our main contributions are: ## background in encoder-based language models (such as bert), a transformer encoder layer is composed of several components (figure 2 ). the core component of the encoder is the self-attention mechanism @xcite , which is responsible for the information mixture of a sequence of token representations (x 1 , ..., x n ). each self-attention head computes a set of attention weights a @xmath0,j |1 ≤ i, @xmath1, where α h i,j is the raw attention weight from the i th token to the j th token in head @xmath2, ..., h}. therefore, the output representation (z @xmath3. diagram inspired by @xcite . the i th token of a multi-head (with h heads) selfattention module is computed by concatenating the heads' outputs followed by a head-mixing w o projection: norm-based attention. while one may interpret the attention mechanism using the attention weights a, @xcite argued that doing so would ignore the norm of the transformed vectors multiplied by the weights, elucidating that the weights are insufficient for interpretation. their solution enhanced the interpretability of attention weights by incorporating the value vectors v(x j ) and the following projection w o . by reformulating equation 1, we can consider z i as a summation over the attentions heads: using this reformulation 2 , kobayashi et al. proposed a norm-based token attribution analysis method, n := (||z i←j ||) ∈ r n×n , to measure each token's contribution in a self-attention module: they showed that incorporating the magnitude of the transformation function (f h (x)) is crucial in assessing the input tokens' contribution to the selfattention output. residual connections & layer normalizations. @xcite added the attention block's layer normalization (ln#1) and residual connection (res#1) to its prior norm-based analysis to assess the impact of residual connections and layer normalization inside an attention block. n res := (||z + i←j ||) ∈ r n×n is the analysis method which incorporates the attention block's residual connection. the input vector x is added to the attribution of each token to itself to incorporate the influence of res#1: they proposed a method for decomposing ln 3 into a summation of normalizations: the decomposition can be applied to the contribution vectors: accordingly, we can compute the magnitude n resln := (||z i←j ||) ∈ r n×n , which represents the amount of influence of an encoder layer's input 2 w h o is a head-specific slice of the original w o projection. for more information about the reformulation process, see appendix c in @xcite 3 γ ∈ r d and β ∈ r d are the trainable weights of ln. similar to @xcite we ignore β. experiments by @xcite revealed considerably low r values which indicate the huge impact of the residual connections. in other words, the model tends to preserve token representations more than mixing them with each other. ## methodology our method for input token attribution analysis has a holistic view and takes into account almost every component within the encoder layer. to this end, we first extend the norm-based analysis of @xcite by incorporating the encoder's output ln#2. we then apply an aggregation technique to combine the information flow throughout all layers. encoder layer output ̸ = attention block output. while the res#1 and the ln#1 from the attention block are included in the analysis of @xcite , the subsequent ffn, res#2, and output ln#2 are ignored (see fig. 2 ). hence, n resln might not be indicative of the entire encoder layer's function. to address this issue, we additionally include the encoder layer components from the attention block outputs (z i ) to the output representations (x i ). the output of each encoder (x i ) is computed as follows: we apply the ln decomposition rule in eq. 7 to separate the impacts of residual and ffn output: similarly to eq. 7, we can introduce a more inclusive layerwise analysis method n enc := (||x i←j ||) ∈ r n×n from input token j to output token i using: aggregating multi-layer attention. to create an aggregated attribution score, @xcite proposed describing the model's attentions via modelling the information flow with a directed graph. they introduced attention rollout method, which linearly combines raw attention weights along all available paths in the pairwise attention graph. the attention rollout of layer ℓ w.r.t. the inputs is computed recursively as follows: āℓ is the raw attention map averaged across all heads in layer ℓ. this method assumes equal contribution from the residual connection and multi-head attention (see fig. 2 ). hence, an identity matrix is summed and renormalized, giving âℓ . for aggregating the layerwise analysis methods, we use the rollout technique with minor modifications. as many of the methods already include residual connections, we only use eq. 12 (replacing âℓ with the desired method's attribution matrix in layer ℓ) to calculate the rollout of a given method. however, for methods that do not assume the residual connection, we define a corresponding "fixed" variation using eq. 13 that incorporates a fixed residual effect (r i ≈ 0.5). we refer to our proposed global method-aggregating the n enc across all layers by the rollout method-as globenc. in what follows we report our experiments, comparing globenc with several other settings. ## experiments in this section, we introduce the datasets and the token attribution analysis methods used in our evaluations, followed by the experimental setup and results. ## related work while numerous studies have used attention weights to analyze and interpret the self-attention mechanism @xcite @xcite , the use of mere attention weights to explain a model's inner workings has been an active topic of debate @xcite @xcite . several solutions have been proposed to address this issue, usually through converting raw attention weights to scores that provide better explanations. @xcite used the transformation function f h (x j ) to introduce effective attentions-the orthogonal component of the attention matrix in f h (x j ) null space-to explain the inner workings of each layer. however, this technique ignores other components in the encoder and is computationally expensive due to the svd required to compute the effective attentions. @xcite incorporated the modified vector and introduced a vector norms-based analysis. this was later extended by integrating residual connections and layer normalization components to enhance the accuracy of explanations @xcite . but, as discussed in §4.5, relying solely on ln#1 does not produce accurate results. while these methods can be employed for singlelayer (local) analysis, multi-layer attributions are not necessarily correlated with single-layer attributions due to the significant degree of information combination through multi-layer language models @xcite . various saliency methods exist for explaining the model's decision based on the input @xcite @xcite @xcite . however, these approaches are not primarily designed for computing inter-token attributions. to fill this gap, @xcite proposed hta, which is based on the gradient of each hidden embedding in relation to the input embeddings. in §4.3.2, we extend hta to incorporate the impact of the input vectors. however, hta is extremely computationally intensive. attention rollout (see §3) and attention flow-which involve solving a max-flow problem on the attention graph-are two aggregation approaches introduced by @xcite , in which raw attention weights (with equally weighted residual weights) are aggregated within multiple layers. we showed that attention rollout does not perform well on the raw attention maps of language models fine-tuned on downstream tasks and that this problem can be resolved by utilizing attribution norms. ## conclusions in this work, we proposed a novel method for single layer token attribution analysis which incorporates the whole encoder layer, i.e., the attention block and the output layer normalization. when aggregated across layers using the rollout method, our technique achieves quantitatively and qualitatively plausible results. our evaluation of different analysis methods provided evidence on roles played by individual components of the encoder layer, i.e., the vector norms, the residual connections, and the layer normalizations. furthermore, our in-depth analysis suggested that the two layer normalizations in the encoder layer counteract each other; hence, it is important to couple them for an accurate analysis. additionally, using a newly proposed and improved version of hidden token attribution, we demonstrated that encoder-based attribution analysis is more accurate when compared to other partial solutions in a single layer (local-level). this is consistent with our global observations. quantifying global input token attribution based on our work can provide a meaningful explanation of the whole model's behavior. in future work, we plan to apply our global analysis method on various datasets and models, to provide valuable insights into model decisions and interpretability. [cls] big fat waste of time . [sep]