Better Explain Transformers by Illuminating Important Information (2024)

Abstract: Transformer-based models excel in various natural language processing (NLP) tasks, attracting countless efforts to explain their inner workings. Prior methods explain Transformers by focusing on the raw gradient and attention as token attribution scores, where non-relevant information is often considered during explanation computation, resulting in confusing results. In this work, we propose highlighting the important information and eliminating irrelevant information by a refined information flow on top of the layer-wise relevance propagation (LRP) method. Specifically, we consider identifying syntactic and positional heads as important attention heads and focus on the relevance obtained from these important heads. Experimental results demonstrate that irrelevant information does distort output attribution scores and then should be masked during explanation computation. Compared to eight baselines on both classification and question-answering datasets, our method consistently outperforms with over 3% to 33% improvement on explanation metrics, providing superior explanation performance. Our anonymous code repository is available at: https://anonymous.4open.science/r/MLRP-E676/

## introduction transformer @xcite currently serves as the fundamental structure for state-of-the-art models @xcite @xcite . the power of these models provides convincing results in multiple natural language processing (nlp) tasks. however, building a robust transformer-based model to assist trustworthy human decision-making processes requires an understanding of the internal mechanisms of the transformers @xcite @xcite . in nlp tasks, tokens are prevalently utilized to signify a word or a fragment of a word (also known as a subword), serving as the input for transformers. to comprehend the influence of input tokens on a transformer, helping us to understand which part of input the transformer is most interested in, a typical approach involves determining the attribution score of input tokens by leveraging the information captured by the attention matrix obtained from each attention head @xcite @xcite . a high attribution score signifies that the input token likely plays a pivotal role in the model's decision-making process for a specific class, output word, or answer index. to derive attribution scores for each input token, recent approaches utilized information within a trained transformer, such as inputgradients @xcite , raw attention matrices @xcite or the combination of input-gradients and attention matrix @xcite . the underlying premise for those methods is that input token gradients reflect the token's significance during backpropagation, while attention mechanisms capture the between-token interactions. however, both theoretical and empirical results @xcite @xcite indicate that not all types of information embedded within the gradient and attention mechanisms contribute towards the explanations. they either fail to or can only partially aid in understanding which token primarily contributes to the transformer's decision-making process. to solve this issue, we follow the line of work known as layer-wise relevance propagation (lrp, @xcite ) with refined information flow to derive compelling attribution scores for each token. the information flow within lrp parameterized by each attention head mirrors that of the transformer, concentrating on distinct portions of the input tokens, and attention heads focusing on irrelevant information can disrupt this flow, causing explanation confusion. we refine the information flow within lrp by illuminating the attention head that focuses on important information and reducing the attention head that zeroes in on less important information. to achieve this, we illuminate the important attention head by adopting a head mask generated from dataset statistics. we first label the attention heads concentrating on a specific syntactic relationship as syntactic attention heads. syntactic relations (e.g., nominal subject) are extensively utilized to define the relations between tokens in nlp @xcite , which establish a directional relation between two words. furthermore, we designate the attention head that predominantly centers on a fixed relative position as a positional attention head, which reflects the internal feature (e.g., spatial position) of token embedding. we encapsulate syntactic and positional within a head mask, which we use to refine the information flow during the lrp process. to further reduce the irrelevant information, we obtain the attribution score by rolling out the relevance of the attention head from each attention blocks with the corresponding gradient @xcite . to evaluate the performance of our method, we compared it with eight strong baselines across five classification datasets and two question-answering datasets. the results reveal that our method outperforms others in explanation performance, demonstrating a distinguished capacity to assign influential tokens from both interaction and internal perspectives. furthermore, an ablation study uncovers that irrelevant information can obfuscate the lrp process, subsequently leading to a biased explanation of input tokens. the key contributions of our work can be summarized as follows: ## related works to explain a transformer in nlp tasks, one common approach involves providing a post-hoc interpretable description of the transformer's behavior. this approach assists users in understanding which input tokens most significantly influence the model's decision-making process. abnar and zuidema (2020) achieve this by leveraging the attention heads for defining more elaborate explanation mechanisms, while @xcite and @xcite accomplish this by involving the integrated gradients or input gradients. numerous models and domains have employed gradient methods such as saliency maps @xcite , gradient×input @xcite @xcite , or guided backpropagation (zeiler and fergus, 2014), and these methods have also been effectively transposed and applied to transformers. concurrently, there have been several attempts to implement layer-wise relevance propagation (lrp, @xcite ) in transformers @xcite and other attentionbased models @xcite . lrp has been used to explain predictions of diverse models on nlp tasks, including bert (kenton and toutanova, 2019). other methodologies for lrp / gradient propagation in transformer blocks can be found in @xcite , where the relevance scores are determined by combining attention scores with lrp or attention gradients. additionally, a few instances exist where perturbation-based methods have employed input reductions @xcite , aiming to identify the most relevant parts of the input by observing changes in model confidence or leveraging shapley values @xcite . furthermore, a line of work using tensor decomposition to decompose the attention matrix for a faithful transformer explanation @xcite @xcite . ## layer-wise relevance propagation through important attention head in this work, we empirically show that irrelevant information can detrimentally impact the lrp process. therefore, our focus should be directed toward the important information while concurrently eliminating irrelevant information within the lrp process. in this section, we initially classify two kinds of important information (sec.4.1), followed by introducing the method to extract this information in each layer (sec.4.2). subsequently, we illustrate the technique of concentrating on the important information extracted during layer-wise relevance propagation (lrp, sec. 4.3). ## conclusion in this study, we propose that irrelevant information in the gradient and attention hampers the explanation process. to address this, we improve the information flow in the lrp process by masking irrelevant attention heads. by illuminating the important information, we show that explanations become more convincing. our method outperforms nine baseline methods in classification and question answering tasks, consistently delivering better explanation performance.