Evaluating Attribution Methods using White-Box LSTM s (2020)

Abstract: Interpretability methods for neural networks are difficult to evaluate because we do not understand the black-box models typically used to test them. This paper proposes a framework in which interpretability methods are evaluated using manually constructed networks, which we call white-box networks, whose behavior is understood a priori. We evaluate five methods for producing attribution heatmaps by applying them to white-box LSTM classifiers for tasks based on formal languages. Although our white-box classifiers solve their tasks perfectly and transparently, we find that all five attribution methods fail to produce the expected model explanations.

## introduction attribution methods are a family of interpretability techniques for individual neural network predictions that attempt to measure the importance of input features for determining the model's output. given an input, an attribution method produces a vector of attribution or relevance scores, which is typically visualized as a heatmap that highlights portions of the input that contribute to model behavior. in the context of nlp, attribution scores are usually computed at the token level, so that each score represents the importance of a token within an input sequence. these heatmaps can be used to identify keywords upon which networks base their decisions @xcite @xcite . one of the main challenges facing the evaluation of attribution methods is that it is difficult to assess the quality of a heatmap when the network in question is not understood in the first place. if a word is deemed relevant by an attribution method, we do not know whether the model actually considers that word relevant, or whether the attribu-tion method has erroneously estimated its importance. indeed, previous studies have argued that attribution methods are sensitive to features unrelated to model behavior in some cases (e.g., @xcite , and altogether insensitive to model behavior in others @xcite . to tease the evaluation of attribution methods apart from the interpretation of models, this paper proposes an evaluation framework for attribution methods in nlp that uses only models that are fully understood a priori. instead of testing attribution methods on black-box models obtained through training, we construct white-box models for testing by directly setting network parameters by hand. our focus is on white-box lstms that implement intuitive strategies for solving simple classification tasks based on formal languages with deterministic solutions. we apply our framework to five attribution methods: occlusion @xcite , saliency @xcite , gradient × input, (g × i, @xcite , integrated gradients (ig, @xcite , and layer-wise relevance propagation @xcite . in doing so, we make the following contributions. output: true, since the input aacb contains two (noncontiguous) instances of ab. occlusion saliency g × i ig lrp a a c b a a c b a a c b a a c b a a c b a a c b a a c b a a c b a a c b a a c b ## related work several approaches have been taken in the literature for understanding how to evaluate attribution methods. on a theoretical level, axiomatic approaches propose formal desiderata that attribution methods should satisfy, such as implementation invariance @xcite , input translation invariance @xcite , continuity with respect to inputs @xcite , or the existence of relationships between attribution scores and logit or softmax scores @xcite @xcite . the degree to which attribution methods fulfill these criteria can be determined either mathematically or empirically. other approaches, which are more experimental in nature, attempt to directly assess the relationship between attribution scores and model behav-ior. a common test, due to @xcite and @xcite and applied to sequence modeling by @xcite , involves ablating or perturbing parts of the input, from those with the highest attribution scores to those with the lowest, and counting the number of features that need to be ablated in order to change the model's prediction. another test, proposed by @xcite , tracks how heatmaps change as layers of a network are incrementally randomized. a third kind of approach evaluates the extent to which heatmaps identify salient input features. for example, @xcite propose the pointing game task, in which the highest-relevance pixel for an image classifier input must belong to the object described by the target output class. within this framework, @xcite al. (2018), poerner et al. (2018), arras et al. (2019), and yang and kim (2019) construct datasets in which input features exhibit experimentally controlled notions of importance, yielding "ground truth" attributions against which heatmaps can be evaluated. our paper incorporates elements of the groundtruth approaches, since it is straightforward to determine which input features are important for our formal language tasks. we enhance these approaches by using white-box models that are guaranteed to be sensitive to those features. ## formal language tasks formal languages are often used to evaluate the expressive power of rnns. here, we focus on formal languages that have been recently used to probe lstms' ability to capture three kinds of dependencies: counting, long-distance, and hierarchical dependencies. we define a classification task based on each of these formal languages. ## white-box networks we use two approaches to construct white-box networks for our tasks. in the counter-based approach, the cell state contains a set of counters, which are incremented or decremented throughout the computation. the network's final output is based on the values of the counters. in the automaton-based approach, we use the lstm to simulate an automaton, with the cell state containing a representation of the automaton's state. we use a counter-based network to solve the counter task and an automaton-based network to solve the bracket prediction task. we use both kinds of networks to solve the sp task. all networks perfectly solve the tasks they were designed for. this section describes our white-box networks at a high level; a detailed description is given in appendix a. in the rest of this paper, we identify the alphabet symbols a, b, c, and d with the one-hot vectors for indices 1, 2, 3, and 4, respectively. the vectors f (t) , i (t) , and o (t) represent the forget, input, and output gates, respectively. g (t) is the value added to the cell state at each time step, and σ represents the sigmoid function. we assume that the hidden state h (t) and cell state c (t) are updated as follows. ## attribution methods let x be a matrix of input vectors, such that the input at time t is the row vector x t,: = ( x (t) ) ⊤ . given x, an lstm classifier produces a vector ŷ of logit scores. based on x, ŷ, and possibly a baseline input x, an attribution method assigns an attribution score r (c) t,i (x) to input feature x t,i for each output class c. these feature-level scores are then aggregated to produce token-level scores: broadly speaking, our five attribution methods are grouped into three types: one perturbation-based, three gradient-based, and one decompositionbased. the following subsections describe how each method computes r (c) t,i (x). ## qualitative evaluation to evaluate attribution methods under our framework, we begin with a qualitative description of the heatmaps that are computed for our whitebox networks, based on the illustrative sample of heatmaps appearing in ## detailed evaluations we now turn to focused investigations of particular phenomena that attribution methods exhibit when applied to white-box networks. subsection 7.1 begins by discussing the effect of network saturation on the gradient-based methods and lrp. in subsection 7.2 we apply @xcite ablation test to our attribution methods for the sp task. ## conclusion of all the heatmaps considered in this paper, only those computed by g × i and ig for the counting task fully matched our expectations. in other cases, all attribution methods fail to identify at least some of the input features that should be considered relevant, or assign relevance to input features that do not affect the model's behavior. among the five methods, saliency achieves the worst performance: it never assigns nonzero scores for the counting and bracket prediction tasks, and it does not identify the relevant symbols for either of the two sp networks. saliency also achieves the worst performance on the ablation test for both the counterbased and the fsa-based sp networks. among the four white-box networks, the two automatabased networks proved to be much more challenging for the attribution methods than the counterbased networks. while the lrp heatmaps for the pda network correctly identify the matching bracket when available, no other method produces reasonable heatmaps for the pda network, and all five methods fail to interpret the fsa network. taken together, our results suggest that attribution heatmaps should be viewed with skepticism. this paper has identified cases in which heatmaps fail to highlight relevant features, as well as cases in which heatmaps incorrectly highlight irrelevant features. although most of the methods perform better for the counter-based networks than the automaton-based networks, in practical settings we do not know what kinds of computations are implemented by a trained network, making it impossible to determine whether the network under analysis is compatible with the attribution method being used. in future work, we encourage the use of our four white-box models as qualitative benchmarks for evaluating interpretability methods. for example, the style of evaluation we have developed can be replicated for attribution methods not covered in this paper, including deeplift @xcite and contextual decomposition @xcite . we believe that insights gleaned from white-box analysis can help researchers choose between different attribution methods and identify areas of improvement in current techniques.