Locally Distributed Activation Vectors for Guided Feature Attribution (2022)

Abstract: Explaining the predictions of a deep neural network (DNN) is a challenging problem. Many attempts at interpreting those predictions have focused on attribution-based methods, which assess the contributions of individual features to each model prediction. However, attribution-based explanations do not always provide faithful explanations to the target model, e.g., noisy gradients can result in unfaithful feature attribution for back-propagation methods. We present a method to learn explanations-specific representations while constructing deep network models for text classification. These representations can be used to faithfully interpret black-box predictions, i.e., highlighting the most important input features and their role in any particular prediction. We show that learning specific representations improves model interpretability across various tasks, for both qualitative and quantitative evaluations, while preserving predictive performance.

## introduction deep neural network (dnn) models have become crucial tools in natural language processing (nlp) and define state-of-the-art on a large variety of tasks. however, dnn models are often considered "black boxes," whose predictions are difficult to interpret and understand. an immediate consequence is that quantifying the contribution of individual features is a challenging fundamental task in nlp and explainable ai research. in most related work, whether implicitly or explicitly, an explanation's role in nlp text classification is to reveal which words and phrases are the most salient for the final prediction @xcite . from this viewpoint, a popular approach for explaining a prediction is to use attribution methods, which justify the prediction of a pre-trained deep network, i.e., the explanation approximates the feature attribution w.r.t. the predicted class. however, attribution techniques, also included in the class of methods called post-hoc methods, might not always provide explanations that are faithful to the underlined model because of the instability of the explanations. this is largely because they often rely on heuristic rules; how those rules mimic the predictive calculation of the black box has the limitation of correlation with expected model behavior @xcite . for instance, @xcite showed that the explanations of two very close prediction points varied significantly in a simulated setting. we know that faithful explanations are essential, especially in high-stakes domains. if the explanations are wrong, we cannot trust the black box model. in addition, explanations are supposed to be faithful to what the model actually computes, so they may not meet the end user's expectations. we define a faithful explanation in the context of nlp as follows: an explanation method is faithful if it is capable of identifying the most salient/meaningful features used by the model to make a prediction. the way humans arrive at a decision can be different from a black-box model. this limitation makes it difficult to enforce the idea that an explanation must follow the user's expectation, e.g., as suggested by human annotation, which can be completely different from the predictive model behavior. in a nutshell, our goal is to uncover faithful feature attributions from deep networks, thus to reveal, as accurately as possible, the most influential features used by the network to make a prediction using a bottom-up approach. to do so, we need to focus on learning representations to support feature attribution. so we optimize a deep network model for both faithful attribution and high prediction accuracy. as a result, we construct a model that can learn meaningful representations to explain class predictions without using post-hoc methods. our guided model is based on learning an activation vector for each class. this vector is intended to capture the salient features for each class. finally, the activation vector is used to explain the model's prediction. our contributions are as follows: (1) we propose a method to learn feature attribution concurrently while training a black-box, in order to faithfully explain the black-box; (2) our method achieves better explanation and is capable of identifying the most salient words; (3) our method shows that it does not trade off interpretability against classification accuracy; (4) we also propose a method that can be used for hypothesis testing and measuring importance of phrases. ## related work existing work on interpreting predictive models tackles the problem from the following five directions. ## locally distributed activation vectors our focus, like traditional post-hoc methods, is on feature importance. we present our model locally distributed activation vector, which is an effective method for learning distributed-activation-vectors concurrently while training deep neural networks for text classification (i.e., learning a new representation to support feature attribution). a locally distributed activation vector (ldav, activation vector) is simply a one-dimensional vector that encodes the knowledge learned by a deep neural network for a text classification task, with a focus on interpreting the predictions (see figure 1 ). we can use the activation vectors with any black-box models, including transformer, gru, and lstm methods. each activation vector records the prediction knowledge of the deep neural network for a particular class. the goal is to alter the optimization problem to learn ldavs that will be used to interpret model predictions. we can also use an ldav to conduct hypothesis testing on the role of attributes in any classification, for example whether blood pressure is a critical factor in predicting kidney disease. for notation, we denote scalars with italic lowercase letters (e.g., x), vectors with bold lowercase letters (e.g., x), and matrices with bold uppercase letters (e.g., w ). in the text classification task, an input sequence x 1 , ..., x @xmath0, where l is the length of the input text and d is the vector dimension, is mapped to a distribution over class labels using a parameterized deep neural network (e.g., bilstm). the output y is a vector of class probabilities, and the predicted class ŷ is a categorical outcome. to faithfully interpret the deep neural network's prediction using relative importance, we rely on information encoded by the ldav. the model learns k distributed activation vectors z j (@xmath1, 2, ..., k), where the prediction knowledge of each ŷ is represented using z ŷ ∈ r d and k represents the number of classes. during deep neural network training, we concurrently update each z j . our intuition is that the "locally distributed activation vector" for a given class is trained to emulate the average word embedding of all of the instances that are predicted for that class, while being maximally different from the ldavs of the other classes. in general, for text classification, we feed x 1 , ..., x l to the representation layer (e,g, a lstm) to obtain the context vector h. the model predicts the label by feeding h to an output layer. ## experiments and analysis we focus on the following objectives: 1) ensure explainability does not affect predictive accuracy, and 2) ensure the constrained optimization problem provides faithful feature attribution. a summary of the datasets is shown in ## conclusion and future work we have presented a method to learn locally distributed activation vectors (ldavs) that can be adapted to faithfully interpret deep network predictions. our method outperforms traditional post-hoc techniques in revealing the classifier's most discriminative features for a given prediction. it also avoids the often misrepresented trade off between interpretability against classification accuracy. we also showed that ldav can be used for concept testing and importance measure for phrases. following this work, we want to extend our approach to other tasks such as question answering and name entity recognition.