D ecomp X : Explaining Transformers Decisions by Propagating Token Decomposition (2023)

Abstract: An emerging solution for explaining Transformer-based models is to use vector-based analysis on how the representations are formed. However, providing a faithful vector-based explanation for a multi-layer model could be challenging in three aspects: (1) Incorporating all components into the analysis, (2) Aggregating the layer dynamics to determine the information flow and mixture throughout the entire model, and (3) Identifying the connection between the vector-based analysis and the model’s predictions. In this paper, we present DecompX to tackle these challenges. DecompX is based on the construction of decomposed token representations and their successive propagation throughout the model without mixing them in between layers. Additionally, our proposal provides multiple advantages over existing solutions for its inclusion of all encoder components (especially nonlinear feed-forward networks) and the classification head. The former allows acquiring precise vectors while the latter transforms the decomposition into meaningful prediction-based values, eliminating the need for norm- or summation-based vector aggregation. According to the standard faithfulness evaluations, DecompX consistently outperforms existing gradient-based and vector-based approaches on various datasets. Our code is available at https://github.com/mohsenfayyaz/DecompX .

## introduction while transformer-based models have demonstrated significant performance, their black-box nature necessitates the development of explanation methods for understanding these models' decisions @xcite @xcite . on the one hand, researchers have adapted gradient-based methods ## related work vector-based analysis has been sparked by the motivation that attention weights alone are insufficient and misleading to explain the model's decisions @xcite . one limitation was that it neglects the selfattention value vectors multiplied by the attention weights. @xcite addressed it by using the norm of the weighted value vectors as a measure of inter-token attribution. their work could be regarded as one of the first attempts at transformer decomposition. they expanded their analysis from the self-attention layer to the entire attention block and found that residual connections are crucial to the information flow in the encoder layer @xcite . however, to be able to explain the multilayer dynamics, one needs to aggregate the local analysis into global by considering the attribution mixture across layers. @xcite introduce the attention rollout and flow methods, which aggregate multilayer attention weights to create an overall attribution map. nevertheless, the method did not result in accurate maps as it was based on an aggregation of attention weights only. globenc @xcite and alti @xcite improved this by incorporating decomposition at the local level and then aggregating the resulting vectors-norms with rollout to build global level explanations. at the local level, globenc extended @xcite by incorporating the second residual connection and layernormalization layer after the attention block. globenc utilizes the l2-norm of the decomposed vectors as an attribution measure; however, @xcite demonstrate that the reduced anisotropy of the local decomposition makes l2-norms an unreliable metric. accordingly, they develop a scoring metric based on the l1-distances between the decomposed vectors and the output of the attention block. the final outcome after applying rollout, referred to as alti, showed improvements in both the attention-based and norm-based scores. despite continuous improvement, all these methods suffer from three main shortcomings. they all omitted the classification head, which plays a significant role in the output of the model. in addition, they only evaluate linear components for their decomposition, despite the fact that the ffn plays a significant role in the operation of the model @xcite . nonetheless, the most important weakness in their analysis is the use of rollout for multi-layer aggregation. rollout assumes that the only required information for computing the global flow is a set of scalar cross-token attributions. nevertheless, this simplifying assumption ignores that each decomposed vector represents the multi-dimensional impact of its inputs. therefore, losing information is inevitable when reducing these complex vectors into one cross-token weight. on the contrary, by keeping and propagating the decomposed vectors in decompx, any transformation applied to the representations can be traced back to the input tokens without information loss. token 1 token 2 positive label negative label token 3 classification head +5 -3 [cls] good movie +6 0 -1 +5 token 1 token 2 token 3 residual connection (res#1) feed forward encoder attention block add & norm (ln#1) layernorm ( ) + feed forward multi-head attention residual (res#2) decomposed encoder layer 1 + + 3 + token1 token2 token3 2 1 add & norm (ln#2) -4 0 +1 -3 + decomposed encoder layer l token1 token2 token3 gradient-based methods. one might consider gradient-based explanation methods as a workaround to the three issues stated above. methods such as vanilla gradients @xcite , gradientxinput @xcite , and integrated gradients @xcite all rely on the gradients of the prediction score of the model w.r.t. the input embeddings. to convert the gradient vectors into scalar per-token importance, various reduction methods such as l1-norm @xcite , l2-norm @xcite , and mean @xcite have been employed. nonetheless, @xcite evaluations showed that none of them is consistently better than the other. furthermore, adversarial analysis and sanity checks both have raised doubts about gradient-based methods' trustworthiness @xcite @xcite . perturbation-based methods. another set of interpretability methods, broadly classified as perturbation-based methods, encompasses widely recognized approaches such as lime @xcite and shap @xcite . however, these were excluded from our choice of comparison techniques, primarily due to their documented inefficiencies and reliability issues as highlighted by @xcite . we follow recent work @xcite and mainly compare against gradient-based methods which have consistently proven to be more faithful than perturbation-based methods. @xcite recently presented a method called value zeroing to measure the extent of context mixing in encoder layers. their approach involves setting the value representation of each token to zero in each layer and then calculating attribution scores by comparing the cosine distances with the original representations. although they focused on local-level faithfulness, their global experiment has clear drawbacks due to its reliance on rollout aggregation and naive evaluation metric (cf. a.3). ## methodology based on the vector-based approaches of @xcite and @xcite , we propose decomposing token representations into their constituent vectors. consider decomposing the i th token representation in layer ℓ ∈ {0, 1, 2, ..., l, l + 1} foot_0 , i.e., according to this decomposition, we can compute the norm of the attribution vector of the k th input (x ℓ i⇐k ) to quantify its total attribution to x ℓ i . the main challenge of this decomposition, however, is how we could obtain the attribution vectors in accordance with the internal dynamics of the model. as shown in figure 2 , in the first encoder layer, the first set of decomposed attribution vectors can be computed as x foot_1 i⇐k . 2 these vectors are passed through each layer in order to return the decomposition up to that layer: x ℓ i⇐@xmath0. ultimately, the decomposed vectors of the [cls] token are passed through the classification head, which returns a decomposed set of logits. these values reveal the extent to which each token has influenced the corresponding output logit. in this section, we explain how vectors are decomposed and propagated through each component, altogether describing a complete propagation through an encoder layer. after this operation is repeated across all layers, we describe how the classification head transforms the decomposition vectors from the last encoder layer into prediction explanation scores. ## experiments our faithfulness evaluations are conducted on four datasets covering different tasks, sst-2 @xcite for sentiment analysis, mnli @xcite for nli, qnli @xcite for question answering, and hatexplain @xcite for hate speech detection. our code is implemented based on hugging-face's transformers library @xcite . for our experiments, we used fine-tuned bert-baseuncased @xcite and roberta-base @xcite , obtained from the same library. 9 as for gradient-based methods, we choose 0.1 as a step size in integrated gradient experiments and consider the l2-norm of the token's gradient vector as its final attribution score. foot_7 ## conclusions in this work, we introduced decompx, an explanation method based on propagating decomposed token vectors up to the classification head, which addresses the major issues of the previous vectorbased methods. to achieve this, we incorporated all the encoder layer components including nonlinear functions, propagated the decomposed vectors throughout the whole model instead of aggregating them in-between layers, and for the first time, incorporated the classification head resulting in faithful explanations regarding the exact positive or negative impact of each input token on the output classes. through extensive experiments, we demonstrated that our method is consistently better than existing vector-and gradient-based methods by a wide margin. our work can open up a new avenue for explaining model behaviors in various situations. as future work, one can apply the technique to encoder-decoder transformers, multilingual, and vision transformers architectures.