Local Interpretation of Transformer Based on Linear Decomposition (2023)

Abstract: In recent years, deep neural networks (DNNs) have achieved state-of-the-art performance on a wide range of tasks. However, limitations in interpretability have hindered their applications in the real world. This work proposes to interpret neural networks by linear decomposition and finds that the ReLU-activated Transformer can be considered as a linear model on a single input. We further leverage the linearity of the model and propose a linear decomposition of the model output to generate local explanations. Our evaluation of sentiment classification and machine translation shows that our method achieves competitive performance in efficiency and fidelity of explanation. In addition, we demonstrate the potential of our approach in applications with examples of error analysis on multiple tasks.

## introduction deep neural networks (dnns) such as transformers @xcite have achieved state-of-the-art results on various natural language tasks @xcite @xcite via learning complex nonlinear relationships of inputs. however, the lack of interpretability of the predictions given by black-box models limits their application in real-world @xcite @xcite . a typical way to understand model prediction dynamics is to generate prediction explanations for each input, called local explanation generation @xcite . most existing works on local explanation algorithms in nlp strive to understand such dynamics on word-level or phraselevel by assigning importance scores on input features @xcite @xcite . however, nonlinearity in models makes it difficult to assign the contribution of individual words or phrases to predictions, while the linear counterparts are more interpretable as the weight of each component could be naturally interpreted as its contribution. in this work, we present a linear decomposition theory to interpret linear models, which can be generalized to nonlinear dnns. that is, we formalize the decomposition of the linear outputs into components corresponding to the input features, then mathematically propose the properties of linear decomposition and the uniqueness of the decomposition under these properties. furthermore, we prove that the relu-activated transformer can be regarded as a linear function by given input features if the causal relationship between the input and certain intermediate variables is disregarded. therefore, generalize the proposed linear decomposition to transformer under such an assumption. however, this decomposition yields a component corresponding to the parameters of additive bias (usually used in the linear layer), which contains a partial contribution of the inputs. thus we separate and reallocate this part of the contribution to the input features while preserving the mathematical properties of the decomposition. quantitative experiments were conducted on sentiment classification and machine translation to identify important input features. we show that our local explanation algorithms efficiently outperform several competitive baselines. additionally, we propose further implementation of our algorithm to explain the model errors on natural language tasks. the fidelity of our algorithm exceeds that of other baselines. our key contributions are summarized as follows: ## method in this section, we propose the decomposition theory of linear functions. then, we generalize it to nonlinear cases (i.e., transformer) and present several decomposition methods accordingly. finally, we analyze the mathematical properties of the different methods. ## experiments we evaluate our algorithms with sota transformer implementations on text classification (roberta, @xcite and machine translation @xcite . it is notable that the classification follows the encoder-only architecture, while the translation follows the encode-decode architecture. ## applications our method can be applied to various scenarios by designing different partitions. in this section, we analyze the causes of model errors in sentiment classification and translation at the instance level. we set up our algorithm with d (signed) for strict consistency. we also compare the results of ig (integrated gradient) and loo (leave-one-out). ## related work interpreting dnns involves various techniques, such as feature visualization @xcite , probing @xcite , and analyzing learned weights @xcite . local interpretation belongs to another paradigm, which tries to interpret individual predictions of a dnn. existing works of local interpretation focus on assigning importance to individual features with respect to the prediction, such as pixels in an image or words in a sentence. the assignment employs methods like input occlusion @xcite , gradientbased algorithms @xcite , layer-wise relevance propagation @xcite , decomposition-based methods @xcite @xcite @xcite , and others @xcite . specifically in nlp, @xcite extend lrp to the transformer to analyze nmt models. @xcite introduces a contextual decomposition to track the word-level importance in lstm @xcite . @xcite extend the aforementioned to produce hierarchical clustering of words along with the contribution of each cluster. backpropagation-based algorithms such as gradient-based algorithms @xcite and lrp @xcite have exponential time or space complexity, making their application on long sequences infeasible. the occlusion algorithms @xcite also suffer from performance degradation on long sentences since occlusion has a limited impact on the semantics of long sentences. our methods are similar to those based on additive decomposition @xcite @xcite . despite not being explicitly noted, these methods all rely on the same assumption to linearize attention scores and layer normalization. however, they do not decompose the ffn layer and instead use heuristic algorithms to aggregate contributions across layers. ## conclusion in this paper, we find that specific dnns satisfy linearity under proper assumptions. we further leverage the linearity of the model to generate local explanations. we test proposed algorithms with the standard and pretrained transformer architecture on two benchmark datasets. experimental results show that our method achieves competitive performance in efficiency and fidelity of explanation. additionally, we offer examples of different tasks to apply our algorithms for error analysis. we leave the analysis of other dnns and the intermediate states of the models as future work.