I nserting I nformation B ottlenecks for A ttribution in T ransformers (2020)

Abstract: Pretrained transformers achieve the state of the art across tasks in natural language processing, motivating researchers to investigate their inner mechanisms. One common direction is to understand what features are important for prediction. In this paper, we apply information bottlenecks to analyze the attribution of each feature for prediction on a black-box model. We use BERT as the example and evaluate our approach both quantitatively and qualitatively. We show the effectiveness of our method in terms of attribution and the ability to provide insight into how information flows through layers. We demonstrate that our technique outperforms two competitive methods in degradation tests on four datasets. Code is available at https://github.com/bazingagin/IBA .

## introduction increasingly prominent is the urge to interpret deep neural networks, with the success of these blackbox models remaining vastly inexplicable both theoretically and empirically. within natural language processing (nlp), this desire is particularly true for the pretrained transformer, which has witnessed an influx of literature on interpretability analysis. such papers include visualizing transformer attention mechanisms @xcite , probing the geometry of transformer representations @xcite , and explaining the span predictions of question answering models @xcite . in this paper, we focus on prediction attribution methods. that is, we ask, "which hidden features contribute the most toward a prediction?" to resolve this question, a number of methods @xcite generate attribution scores for features, which provide a humanunderstandable "explanation" of how a particular prediction is made at the instance level. specif-ically, given an instance, these methods assign a numerical score for each hidden feature denoting its relevance toward the prediction. previous papers have demonstrated that gradientbased methods fail to capture all the information associated with the correct prediction @xcite . to address this weakness, @xcite insert information bottlenecks @xcite for attribution, attaining both stronger empirical performance and a theoretical upper bound on the information used. additionally, mutual information is unconstrained by model and task @xcite . thus, we adopt information bottlenecks for attribution (iba) to interpret transformer models at the instance level. we apply iba to bert @xcite across five datasets in sentiment analysis, textual entailment, and document classification. we show both qualitatively and quantitatively that the method capably captures information in the model's token-level features, as well as insight into cross-layer behavior. our contributions are as follows: first, we are the first to apply information bottlenecks (ib) for attribution to explain transformers. second, we conduct quantitative analysis to investigate the accuracy of our method compared to other interpretability techniques. finally, we examine the consistency of our method across layers in a case study. across four datasets, our technique outperforms integrated gradients (ig) and local interpretable model-agnostic explanations (lime), two widely adopted prediction attribution approaches. ## related work in terms of scope, interpretability methods can be categorized as model specific or model agnostic. model-specific methods interpret only one family of models, whereas model-agnostic techniques aim for wide applicability across many families of parametric models. we can roughly separate model-agnostic methods into three categories: (1) gradient-based ones @xcite @xcite ; (2) probing @xcite @xcite @xcite ; (3) information-theoretical methods @xcite @xcite . gradient-based methods are, however, limited to models with differentiable neural activations. they also fail to capture all the information associated with the correct prediction @xcite . although probing methods provide detailed insight into specific models, they fail to capture inner mechanisms like how information flows through the network @xcite . informationtheoretic methods, in contrast, provide consistent and flexible explanations, as we show in this paper. @xcite use mutual information to interpret nlp models across different tokens, layers, and neurons, but they lack a quantitative evaluation. @xcite also propose a model-agnostic interpretable model using ib; however, they limit the information through the network by sampling a given number of words at the beginning, which restricts the explanation to neurons only. our method is inspired by @xcite , who use iba in image classification. ## method the idea of iba is to restrict the information flowing through the network for every single instance, such that only the most useful information is kept. concretely, given an input @xmath0, an information bottleneck is an intermediate representation t that maximizes the following function: we insert the ib after a given layer l in a pretrained deep neural network. in this case, @xmath1, where h is the input of the layer. we restrict information flow by injecting noise into the original input: x ) to substitute p (t), assuming every dimension of t is independent and normally distributed. even though the independence assumption does not hold in general, it only overestimates the mutual information, giving a nice upper bound of mutual information between x and t: the complete derivation of equation 4b is in appendix a. since we expect i(x, t) to be small and mutual information to be always nonnegative, the upper bound is a desired property. intuitively, the purpose of maximizing i(y; t) is to make accurate predictions. therefore, instead of directly maximizing i(y; t), we minimize the loss function for the original task, e.g., the cross entropy l ce for classification problems after inserting the information bottleneck. combining the above two parts, our final loss function l is note that we negate the sign for minimization. the β hyperparameter controls the relative importance between the two loss components. after the optimization process, we obtain for every instance a compressed representation t. we then calculate d kl [p (t|x) q(t)], indicating how much information is still kept in t about x, which suggests the contribution of each token and feature. to generate the attribution map, we sum over the feature-token axis, obtaining the attribution score of each token. overall, we try to learn a compressed hidden representation t that has just enough information about the input x to predict the output y. this compression is done by adding noise, which removes the least relevant feature-level information, with µ controlling how much to remove. ## experiments through experimentation, we analyze iba both quantitatively and qualitatively to understand how it interprets deep neural network across layers. ## conclusion in this paper, we adopt an information-bottleneckbased approach to analyze attribution for transformers. our method outperforms two widely used attribution methods across four datasets in sentiment analysis, document classification, and textual entailment. we also analyze the information across layers both quantitatively and qualitatively.