Efficient Explanations from Empirical Explainers (2021)

Abstract: Amid a discussion about Green AI in which we see explainability neglected, we explore the possibility to efficiently approximate computationally expensive explainers. To this end, we propose feature attribution modelling with Empirical Explainers. Empirical Explainers learn from data to predict the attribution maps of expensive explainers. We train and test Empirical Explainers in the language domain and find that they model their expensive counterparts surprisingly well, at a fraction of the cost. They could thus mitigate the computational burden of neural explanations significantly, in applications that tolerate an approximation error.

## introduction in recent years, important works were published on the ecological impacts of artificial intelligence and deep learning in particular, e.g. @xcite , @xcite , @xcite . research is focused on the energy hunger of model training and subsequent inference in production. besides training and in-production inference, explainability has become an integral phase of many neural systems. in the ongoing discussion about green ai we see explainability neglected. conversely, in the explainability community, even though research on efficiency is an active area, apparently the discussion is currently shaped by other aspects, such as faithfulness and plausibility @xcite . this is surprising because to explain a single model output, many prominent explanation methods, in particular many feature attribution methods (cf. below), require a multiple of computing power when compared to the prediction step. ## framework: empirical explainers informally, an empirical explainer is a model that has learned from data to efficiently model the feature attribution maps of an expensive explainer. for training, one collects sufficiently many attribution maps from the expensive explainer and then maximizes the likelihood of these target attributions under the empirical explainer. an expensive explainer may, for instance, be a costly attribution method such as integrated gradients that is used to return attributions for the decisions of a classifier, say, a bert-based @xcite sentiment classifier. the corresponding empirical explainer could be a separate neural network, similar in size to the sentiment classifier, consuming the same input tokens as the sentiment model, but instead of predicting the sentiment class, it is trained to predict the integrated gradients for each token. whereas the original integrated gradients explainer requires multiple passes through the classifier, producing the empirical integrated gradients requires just one pass through the similarly sized empirical explainer. empirical explanations come with an accuracy-efficiency trade-off that we discuss in the course of a more formal definition of empirical explainers. for the more formal definition, we need to fix notation first. let e f : ir @xmath0. furthermore, let an empirical explainer be a function e θ : ir @xmath1, parametrized by θ, which also returns attribution maps. let || • || be a penalty for the inefficiency of a computation, e.g. a count of floating point operations, energy consumption or number of model passes needed. furthermore, let us assume, without the loss of generality, that ||e f (x)|| >= ||e θ (x)|| always holds; i.e., the empirical explainer -which we develop and trainis never more inefficient than the original, expensive explainer. let d : ir d × ir @xmath2, 1] be a similarity measure, where d(l, m) = 0 if @xmath3, for l, @xmath4, β ∈ [0, 1] with α + β = 1. for data x, we define an α-optimal empirical explainer by the arg min θ∈θ ## experiments in this section, we report on the performance of empirical explainers that we trained and tested in the language domain. we conducted tests with two prominent and expensive explainers, integrated gradients and shapley value samples, varying the experiments across four state-of-the-art language classifiers, trained on four different tasks. the experiments address the question of whether or not it is feasible -in principle -to train efficient empirical explainers while achieving significant accuracy. all experiments, code, models and data are open source and can be retrieved following https://github.com/dfki-nlp/emp-exp . the most important choices are documented in the following paragraphs. before going into greater detail, it is noteworthy that eq. 1 provides a theoretical framework which one does not have use directly for explicit optimization. for example, in this work, we address the first objective, accuracy, by fitting empirical explainers to the attribution maps of expensive explainers. however, the second objective, efficiency, is addressed implicitly by design, i.e. the empirical explainers, in contrast to their expensive counterparts, are designed (and trained) in a way s.t. only a single forward pass is required through a model similar in size to the downstream model. in the future, it would be very interesting to fully incorporate eq. 1 in a differential setting, i.e. also optimize for efficiency automatically, rather than by manual design. we trained four empirical explainers. all explainers consume only the input tokens to the downstream model and return an attribution score for each token. the first one (empexp-bert-ig) was trained to predict integrated gradients w.r.t. the input tokens to a bert-based imdb movie review @xcite classifier. for the second empirical explainer (empexp-xlnet-sv), we varied the downstream model architecture, task and target explainer: empexp-xlnet-sv predicts the shapley values (as returned by the expensive shapley value sampling explainer) for the inputs of an xlnet-based @xcite natural language inference classifier that was trained on the snli (bowman et al., 2015) dataset. the third (empexp-roberta-ig) and fourth (empexp-electra-sv) empirical explainers again approximate ig and sv, but for a roberta-based news topic classifier @xcite trained on the ag news dataset @xcite and an electra (small)-based model @xcite that detects paraphrases, trained on the paws dataset (subset "labelled_final") @xcite , respectively. the empirical explainers were trained on target attributions that we generated with ig and sv with s := 20 samples. for empexp-roberta-ig, we used s := 25 due to a slower convergence rate (cf. below). explanations were generated for the output neuron with the maximal activation. empexp-bert-ig was trained with early stopping using the ig attribution maps for the full imdb train split (25k). empexp-xlnet-sv was trained with around 100k sv attribution maps for the snli train split with early stopping, for which we used the 10k attribution maps for the validation split. we did not use all training instances in the split for empexp-xlnet-sv, due to the computational costs of shapley value sampling. in case of empexp-roberta-ig and empexp-electra-sv it was possible to train with the full train splits again, 108k (12k held out) and around 50k instances, respectively. as mentioned above, the expensive explainers require user-defined baselines. for the baselines, we replaced all non-special tokens in the input sequence with pad tokens. for the expensive ig, we produced attribution maps for the embedding layer and projected the attribution scores onto tokens by summing them over the token dimension. for the expensive sv, the input ids were perturbed. during perturbation, we grouped and treated special tokens (cls, sep, pad, ...) in the original input as one feature to accelerate the computation. in architectural terms, the empirical explainers are very similar to the downstream models: we heuristically decided to copy the fine-tuned bert, xlnet, roberta and electra encoders from the classifiers and instead of the classification layers on top, we initialized new fully connected layers with t output neurons. t was lower bound by the maximum input token sequence length to the downstream model in the respective dataset: @xmath5, @xmath6, and @xmath7. all input sequences were padded to t and we did not treat padding tokens different from other tokens, when training the empirical explainers. please note that the sequence length has a considerable impact on the runtime of the sv explainer in particular, which is why limiting t increases comparability. we trained the empirical explainers to output the right (in accordance with the expensive explainers) attribution scores for the input tokens, using an mse loss between e f (x 1 ) . . . e f (x t ) and e θ (x 1 ) . . . e θ (x t ) where @xmath8, x 2 , . . . x t is a sequence of input tokens. 2 to put the performance of an empirical explainer into perspective, we propose the following baseline, which is the strongest we can think of: we take the original expensive explainer with a reduced number of samples as the baseline. to position the empirical explainer against this alternative energy saving strategy, we compute convergence curves. starting with @xmath9, we incrementally increase the number of samples until @xmath10. we then compute the mses of these attribution maps when compared to the target attributions (with @xmath11. we average the mses across the test split. the same is done for the empirical explainer. ## results & discussion in the following, we report the experimental results, divided into the aspects of task performance, 2 even though we do not solve for eq. 1 directly, please note that for evaluation we can normalize the attribution scores to [0, 1] prior to computing the mse and this way force the mse into the interval [0, 1] to comply with the constraints for eq. 1. contrary to runtime (and energy consumption) measures, the number of required model passes is largely invariant of available hardware and implementation details. for the sake of completeness, we nevertheless also report our runtimes in appendix a. in summary, generating the expensive explanations for the test splits took between around 02:15 and 48 hours, whereas the empirical explanations only required between 02:05 and 07:14 minutes. the runtimes are not definitive, however. we were unable to establish a fair game for the explainers. for example, due to implementation details and memory issues we explained the data instancewise with the expensive explainers while our empirical explainers easily allowed batch processing. we expect that the expensive explainers can be accelerated but due to the larger number of model passes required, they will very likely not outperform their empirical counterparts. the empirical explainers come with additional training costs, which we also report in appendix a. training took between 02:15 and 07:00 hours. these additional training costs are thus quickly outweighed by the expensive explainers, in particular 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 in a continuous in-production setting. explanation accuracy regarding the accuracy term in eq. 1, figs. 1, 2, 3 and 4 provide anecdotal qualitative evidence that the empirical explainers are capable of modelling their expensive counterparts well, with varying degrees of approximation errors. alongside this paper, we provide four files (see repository) with around 25k (imdb), 10k (snli), 7.6k (ag news) and 8k (paws) lines, each of which contains an html document that depicts a target attribution and its empirical counterpart from the test set. the heatmaps in the figures mentioned above are taken from the accompanying files. figs. 1 and 3 are instances of what we consider surprisingly accurate approximations of the expensive target attribution maps, despite challenging inputs. let us first consider fig. 1 in greater detail. consider the tokens favorites and irritated that are not attributed much importance by the expensive explainer (ig, top) but could be considered signal words for the positive and negative class, respectively and thus pose a challenge for the empirical explainer. nevertheless, in accordance with the expensive target explainer, the empirical explainer (bottom) does not attribute the classifier output primarily to these tokens but instead accurately assigns a lot of weight to it's overall pretty good. a similar phenomenon can be observed in fig. 2 for the token love. the approximation in this figure, however, also contains a prominent approximation error. the empirical explainer erroneously attributes a salient negative score to the token tormented while the target explainer does not highlight that token. similarly, in fig. 4 the empirical explainer returns a negative score for running, whereas the expensive target explainer has returned a positive score. we suspect that such errors may result from global priors that the empirical explainers have learned and that sometimes outweigh the instantaneous information. for instance, in fig. 4 the verb running in the premise in conjunction with the (conjugated) verb runs in the hypothesis may statistically be indicative of an entailment in the training data. this is because to produce a contradiction the verb sometimes is simply replaced by another one (cf. fig. 3 : surfing vs. sun bathing). in this instance, however, the verb is not replaced. thus, here the prior knowledge of the empirical explainer may outweigh the local information in favor of the error that we observe: the empirical explainer may signal that running is evidence against the class contradiction since it finds it in the premise and hypothesis. a similar argument can be put forward for the case of tormented in fig. 2 . the above points are rather speculative. a more objective and quantitative analysis of the efficiency/accuracy trade-off is provided in fig. 5 . the left column depicts the mse lines of ig for an increasing number of samples in x-direction. we observe that ig converges fast in case of bert/imdb. (this may be due to saturation effects in integrated gradients, reported on by @xcite .) in case of roberta/ag news we found a slower convergence rate, which is why we increased the number of samples for the target explainer. we observe that in both cases, the empirical integrated gradients (dashed lines) perform favourably: to outperform the empirical explainer by decreasing s, in case of bert/imdb one needs to set @xmath12. furthermore, the approximation error is already marginal at the intersection of expensive and empirical line. in case of roberta/ag news, one even needs to set s := 18 to be closer to the target than the empirical explainer. a similar trend can be observed for the (empirical) shapley values in the right column of fig. 5 . in case of xlnet/snli, however, the intersection occurs only after @xmath13.9% of model passes (plus the pass through its output layer) when compared to the next best expensive explainer, again assuming 100 input tokens for the purpose of discussion. in case of elec-tra/paws, the empirical explainer even beats the expensive explainer with just one sample less than the target. in summary, we take the experimental results as a strong indication that empirical explainers could become an efficient alternative to expensive explainers (in the language domain) where approximation errors are tolerable. ## related work the computational burden of individual explainability methods was addressed in numerous works. as mentioned above, integrated gradients can only be computed exactly in limit cases and for all other cases, the community relies on the approximate method proposed by @xcite . similarly, shapley values can rarely be computed precisely which is why shapley value sampling was investigated, e.g. by @xcite ; @xcite . shapley value sampling was later unified with other methods under the shap framework (lundberg and lee, 2017) which yielded the method ker-nelshap that showed improved sample efficiency. @xcite then analysed the convergence behaviour of kernelshap and again further improved runtime. chen et al. (2019) thus, computational feasibility appears to be a driving force in the research community, already. to the best of our knowledge, however, we are the first to propose the task of feature attribution modelling to mitigate the computational burden of expensive explainers with empirical explainers. technically, self-explaining models @xcite are related to our approach in that they also generate explanations in a forward pass (alongside their classification deci-sion). contrary to self-explaining models, empirical explainers can be employed after training for a variety of black box and white box classifiers and explainers. a source of inspiration for our method was the work by @xcite . the authors train self-explaining models that return a natural language rationale alongside their classification. thus, they, too, train an explainer. however, their target explanations (natural language) differ substantially from the ones empirical explainers are trained with (attribution scores). furthermore, related to our work is the technique of gradient matching for which a network's (integrated) gradients are compared to a target attribution, i.e. a human prior, and then the network's parameters are updated, s.t. the gradients move closer to the target, as done e.g. by @xcite ; @xcite . apart from the loss on an alignment with target attributions, our method and goals diverge from theirs significantly. human priors and expensive target explanations have recently also been used for explanatory interactive learning (xil). like empirical explainers, xil is motivated by the expensiveness of a target explainer; in the case of xil this is a human in the loop. @xcite present humans with informative (cf. active learning) instances, the model prediction and an explanation for the prediction the expensive human feedback is then used to improve the model. apart from the expensive explainer assumption, their approach differs substantially from ours. very recently, @xcite contributed to xil by introducing a method that learns to explain from explanations and in this respect is close to the setting of empirical explainers. one fundamental difference between ours and their work is that they propose and focus on a specific class of self-explainable models whereas empirical explainers make no assumptions about the underlying predictor and are intended for a variety of model classes, as already mentioned above. very recently again, @xcite proposed local interpretable layers as a means to generate concept attributions which in parts aligns with our method, even though their target attributions and task objectives are very different again. lastly, empirical explainers can be viewed as a form of knowledge distillation @xcite . however, contrary to the established approach, we do not assume a parametric teacher network that knowledge is distilled from. very recently we became aware of the work by @xcite who boost the accuracy of a student learner with explanations in the form of a subset of tokens that are relevant to the teacher decision, determined by an explainer. in a sequence classification task, the student is trained to identify the relevant tokens and could thus be considered an empirical explainer. the task, however, is not to predict the original attribution map and the overall objective differs significantly from ours again. ## conclusion & future directions in this paper, we take a step towards greener xai by again reviving energy efficiency as an additional criterion by which to judge an explainability method, alongside important aspects such as faithfulness and plausibility. in this context, we propose feature attribution modelling with efficient empirical explainers. in the language domain, we investigate the efficiency/accuracy trade-off and find that it is possible to generate empirical explanations with significant accuracy, at a fraction of the costs of the expensive counterparts. we take this as a strong indication that empirical explainers could be a viable alternative to expensive explainers where approximation errors are tolerable. regarding future directions: the empirical explainers we trained are our concrete model choices. the framework we propose allows for many other approaches. for instance, one could provide the empirical explainers with additional information, such as the gradient w.r.t. the inputs. this would require an additional pass through the model but may possibly further boost accuracy. we would like to note that we trained and tested our empirical explainers only on in-domain data but their behaviour on out-of-domain data should be investigated, too. fortunately, since we explain the model decision (the maximum output activation), no gold labels are required to train empirical explainers which facilitates data collection immensely. finally, there are some more sample efficient explainers that should be considered, too.