Learning to Explain: Generating Stable Explanations Fast (2021)

Abstract: The importance of explaining the outcome of a machine learning model, especially a black-box model, is widely acknowledged. Recent approaches explain an outcome by identifying the contributions of input features to this outcome. In environments involving large black-box models or complex inputs, this leads to computationally demanding algorithms. Further, these algorithms often suffer from low stability, with explanations varying significantly across similar examples. In this paper, we propose a Learning to Explain (L2E) approach that learns the behaviour of an underlying explanation algorithm simultaneously from all training examples. Once the explanation algorithm is distilled into an explainer network, it can be used to explain new instances. Our experiments on three classification tasks, which compare our approach to six explanation algorithms, show that L2E is between 5 and 7.5×10ˆ4 times faster than these algorithms, while generating more stable explanations, and having comparable faithfulness to the black-box model.

## introduction explaining the mechanisms and reasoning behind the outcome of complex machine learning models, such as deep neural networks (dnns), is crucial. such explanations can shed light on the potential flaws and biases within these powerful and widely applicable models, e.g., in medical diagnosis @xcite and judicial systems @xcite . existing explainability methods mostly produce explanations, or rationales @xcite , which identify the attributions of features in an input example, e.g., are they contributing positively or negatively to the prediction of an outcome. for text classifiers, this means identifying words or phrases in an input document that account for a novell's microsoft attack completes linux conversion: novell inc. has completed its conversion to linux by launching an attack on microsoft corp., claiming that the company has stifled software innovation and that the market will abandon microsoft windows at some point in the future. microsoft expands windows update release: microsoft corp. is starting to ramp up distribution of its massive security update for the windows xp operating system, but analysts say they still expect the company to move at a relatively slow pace to avoid widespread glitches. figure 1 : two similar examples from the news dataset. the most important words (top 30%) found by our method l2e are yellow-highlighted, and those from a baseline a are underlined. l2e considers words like 'microsoft' and 'windows' important in both examples. ŷx x x is the model's prediction, and ŷx x x • is the model's prediction after removing important words in x x x. in this paper, we present a learning to explain (l2e) approach that efficiently learns the commonalities of the explanation process across different examples. this, in turn, leads to explanations that exhibit stability, i.e., important words are chosen consistently, without loss of faithfulness to the underlying black-box. 1 given a set of examples paired with their explanations produced by an existing method, e.g., @xcite , our approach uses a dnn to learn the explanation algorithm. dnns are turing complete @xcite ; therefore, given enough training data and learning capacity, they should be able to learn the existing explanation algorithms. this is akin to knowledge distillation @xcite , where a teacher, or in our case a teacher algorithm, distils knowledge into a student network. our contributions are: (i) the l2e framework, which is general, and can successfully learn to produce explanations from several teacher explainers; (ii) two learning formulations, i.e., ranking and sequence labelling, to enable l2e to circumvent the high variance of non-discrete teacher explanations via discretization; (iii) an experimental setup to compare l2e against six popular explanation algorithms, and a comprehensive evaluation to investigate the stability and faithfulness of l2e on three text classification tasks; (iv) a methodology that employs human rationales as proxies for the ground-truth explanations of a black-box model. the core of this method is a modified training protocol whereby the model makes neutral predictions if human rationales are absent. ## related work we consider two main approaches to explanation generation: algorithmic and model-based. algorithmic approaches. these approaches can be broadly categorized into gradient-based, attention-based and perturbation-based methods. gradient-based methods @xcite @xcite or backpropagation-based methods @xcite require access to the blackbox, and are mostly applied to models with differentiable functions. further, they may be sensitive to randomized model initializations or permuted data labels @xcite , which is undesirable. these methods can be computationally heavy in the case of complex black-box models @xcite , e.g., bert @xcite . attention-based methods @xcite can only be applied to transformer-based models @xcite , and their effectiveness is questionable @xcite . perturbation-based methods approximate feature importance by observing changes in a model's outcome after a feature is changed. they either consider changes in performance as an indicator of feature importance directly @xcite @xcite , or they employ a higher-order approximation of the decision boundary @xcite . perturbation-based methods are typically computationally inefficient for explaining high-dimensional data, and they suffer from high variance due to perturbation randomness @xcite . model-based approaches. these approaches train the explainer with an objective function to improve efficiency at test time. the closest work to ours is by @xcite , who train an explainer using a causality-based explanation algorithm. however, these approaches do not learn from arbitrary algorithms or discretize feature weights -the high variation of continuous weights may impair the ability to capture the commonalities in an explanation algorithm. @xcite discretize the weights produced by an existing method, but they use these weights to build a faithful classifier for an underlying blackbox model, rather than using them to explain the model directly. other works train a classifier and an explainer jointly in order to incorporate explainability directly into the classifier @xcite . unlike these approaches, we do not change the classifier or require an expensive process to collect human rationales, as done in @xcite . lastly, a few works use information-theoretic objectives to train an explainer directly from the underlying classifier @xcite . these explainers require careful training to select a low number of important features @xcite ; hence, some input features do not have attributions. ## learning to explain (l2e) l2e can be applied to any natural language processing task to which an underlying feature-based explanation algorithm can be applied, such as natural language inference and question answering @xcite . in this paper, we focus on explaining text classification models. our setup requires two inputs: (i) a black-box text classification model ŷ = f θ θ θ (x x x), which assigns document x x x to a label ŷ ∈ y , where y is the label set; and (ii) an explanation algorithm a(x x x, ŷ, f θ θ θ ) → w w w, which generates explanation w w @xmath0. a can be any off-the-shelf explanation algorithm; and w i can be thought as the importance weight of x i -the i th token of a document. the main idea of l2e is to train a separate explanation model g φ φ φ (x x x) to predict the explanation generated by a(.) for f θ θ θ (.) (figures 2a and 2b ). intuitively, our approach distils the explanation algorithm a into the explanation model g φ φ φ . as confirmed by our experiments ( §4.5), this has several benefits. firstly, it leads to stable explanations, as g φ φ φ can capture a's common patterns when generating explanations for different documents. secondly, it speeds up the explanation generation process compared to many existing explanation algorithms, which rely on computationally heavy operations, such as consulting the black-box model multiple times, e.g., occlusion (zeiler and fergus, 2014), or sampling, e.g., @xcite . our approach, which learns a model with explanations algorithm 1 learning to explain (l2e) 1: d: a training set of documents 2: f θ θ θ : the original deep nn model 3: g φ φ φ : the explainer deep nn model 4: a: the underlying explanation method 5: procedure trainexplainer(d, f θ θ θ ) 6: z ← ∅ 7: randomly pick (x x xt, ŷt, w w wt) ∈ z 16: our approach to train the explanation model g φ φ φ is summarized in algorithm 1. first, the algorithm generates training data in the form of triplets (x x x, ŷ, w w w) (lines 7-11), and then it trains the explanation model using supervised learning (lines 14-18). at test time, the trained model is deployed to generate explanations for unseen documents. a crucial component in training the explanation model under supervised learning is the loss function l(g φ φ φ (x x x t , ŷ), w w w). it penalizes a deviation of the predicted explanation g φ φ φ (x x x t , ŷ) from the ground truth explanation w w w. this loss function is determined by our supervised learning formulation. given that w w w is a continuous-valued vector, learning the model g θ θ θ may be cast as a multivariate regression problem. however, the continuous feature attributions generated by existing explanation algorithms could be sensitive to initializations @xcite . further, manually annotated rationales (highlighting important words in a document) are sufficient for people to understand/perform a classification task @xcite . so, instead of a regression formulation, we consider two supervised learning formulations for discretized outputs: ranking and sequence labeling. ranking formulation. in this formulation, the explanation model aims to learn the ranking of the document tokens from their importance weights. that is, we consider the ordering of the token weights induced by w w w, and train the explanation this is a great movie. in other words, each pair of token weights is compared, and the parameters are learnt such that a token with a high importance weight under a also gets a high score under g φ φ φ . ## evaluation with human rationales evaluation of explanation methods for dnns is challenging, as ground-truth explanations are often unavailable. in this section, we propose to address this issue using the imdb-r dataset @xcite , which contains movie reviews x x x together with their sentiment y, as well as rationales r r r annotated by people for the sentiment label. our use of rationales for evaluating explanations is related to that in @xcite , where synthetic data are generated from apriori fixed rationales. specifically, we generate new data by assigning a "neutral" label to an example where the human rationales are masked. we then use both the original data (without masking) and the new data to train the black-box model, where the training protocol forces the classifier to make a "neutral" prediction when the human rationales are removed from the review. more formally, we maximize the following training objective, our classifier achieves an accuracy of 83.83% on the training set, 79.68% on the validation set and 74.5% on the test set. due to the large document sizes ( the faithfulness and stability of the explanation methods are evaluated as follows. faithfulness. we select the top-k important words generated by an explanation method and compute the precision, recall and f1 against the human-annotated rationales. it is worth noting that our l2e explainer is not supervised by human rationales directly. instead, we use the same experimental setup as in section 4.5 to ensure the l2e explainer is learning from the baseline algorithms rather than the human rationales. ## conclusions and future work we have presented a learning to explain (l2e) approach to learn the commonalities of the explanation generation processes across different examples. we have further proposed ranking and sequence labeling formulations to effectively learn the explainer model by discretizing feature weights produced by existing explanation algorithms. our experimental results show that our method can generate more stable explanations (i.e., not vary much across similar documents) than those generated by the explainer baselines, while maintaining the same level of faithfulness to the underlying black-box model as the baseline algorithms. moreover, our l2e approach produces explanations between 5 and 7.5 × 10 4 times faster than the six baselines, making it suitable for long documents and very large black-box models. our l2e approach trains an explainer, a blackbox, to mimic the behaviour of an explanation method for an existing black-box model. a key challenge lies in the variation in the convergence status of such an explainer for different initializations. in order to mitigate this problem, we evaluate the performance of our explainer by averaging three different initializations. the l2e approach opens up the possibility of distilling multiple explanation algorithms into one model. although we focused on the stability, faithfulness and efficiency aspects of explanation generation, there are further desirable properties, e.g., transparency, comprehensibility and novelty @xcite . devising model-based explanation methods and their evaluation with these desiderata are interesting directions for future research. l 2 e g r a d i e n t o c c l u s i o n l r p l i m e k ' s h a p d ' s h a p 2 -8 2 -4 2 0 2 4 2 8 0.0 1± 0.0 * 0.0 2± 0.0 0.1 8± 0.0 2 0.0 8 ± 0.0 1 1.4 ±0 .12 9.3 6± 1.9 8 21 .57 ± 0.2 7 compute time [log 2 (s)]