E VALUATIONS AND M ETHODS FOR E XPLANATION THROUGH R OBUSTNESS A NALYSIS
Anonymous authors
Paper under double-blind review
A BSTRACT
Feature based explanations , that provide importance of each feature towards the model prediction, is arguably one of the most intuitive ways to explain a model.
In this paper, we establish a novel set of evaluation criteria for such feature based explanations by robustness analysis.
In contrast to existing evaluations which require us to specify some way to “remove” features that could inevitably introduces biases and artifacts, we make use of the subtler notion of smaller adversarial perturbations.
By optimizing towards our proposed evaluation criteria, we obtain new explanations that are loosely necessary and sufﬁcient for a prediction.
We further extend the explanation to extract the set of features that would move the current prediction to a target class by adopting targeted adversarial attack for the robustness analysis.
Through experiments across multiple domains and a user study, we validate the usefulness of our evaluation criteria and our derived explanations.
I NTRODUCTION
There is an increasing interest in machine learning models to be credible, fair, and more generally interpretable (Doshi-Velez & Kim, 2017).
Researchers have explored various notions of model interpretability, ranging from trustability (Ribeiro et al., 2016), fairness of a model (Zhao et al., 2017), to characterizing the model’s weak points (Koh & Liang, 2017; Yeh et al., 2018).
Even though the goals of these various model interpretability tasks vary, the vast majority of them use so called feature-based explanations, that assign importance to individual features (Baehrens et al., 2010;
Simonyan et al., 2013; Zeiler & Fergus, 2014; Bach et al., 2015; Ribeiro et al., 2016; Lundberg &
Lee, 2017; Ancona et al., 2018; Sundararajan et al., 2017; Zintgraf et al., 2017; Shrikumar et al., 2017; Chang et al., 2019).
There have also been a slew of recent evaluation measures for feature based explanations, such as completeness (Sundararajan et al., 2017), sensitivity-n (Ancona et al., 2018), inﬁdelity (Yeh et al., 2019), causal local explanation metric (Plumb et al., 2018), and most relevant to the current paper, removal- and preservation-based criteria (Samek et al., 2016; Fong & Vedaldi, 2017; Dabkowski & Gal, 2017).
A common thread in all these evaluation measures is that for a good feature based explanation, the most salient features are necessary, in that removing them should lead to a large difference in prediction score, and are also sufﬁcient in that removing non-salient features should not lead to a large difference in prediction score.
Thus, common evaluations and indeed even methods for feature explanations involve measuring the function difference after “removing features”, which in practice is done by setting the feature value to some reference value (also called baseline value sometimes).
However, this would favor feature values that are far way from the baseline value (since this corresponds to a large perturbation, and hence is likely to lead to a function value difference), causing an intrinsic bias for these methods and evaluations.
For example, if we set the feature value to black in RGB images, this introduces a bias favoring bright pixels: explanations that optimize such evaluations often omit important dark objects such as a dark-colored dog.
An alternative approach to “remove features” is to sample from some predeﬁned distribution or a generative model (Chang et al., 2019).
This in turn incurs the bias inherent to the generative model, and accurate generative models that approximate the data distribution well might not be available in all domains.
Instead of deﬁning prediction changes with “removal” of features (which introduces biases as we argued), we alternatively consider the use of small but adversarial perturbations instead.
It is natural to assume that adversarial perturbations on irrelevant features should be ineffective, while those on relevant features should be effective.
We can thus measure the necessity of a set of relevant features by measuring the consequences of adversarially perturbing the subset of “important features” of the explanations: if the features are relevant, this should lead to an appreciable change in the predictions.
And complementarily, we could measure the sufﬁciency of the set of relevant features via measuring consequences of adversarially perturbing the irrelevant features: if the perturbed features are irrelevant, this should not lead to an appreciable change in the predictions.
We emphasize that by our deﬁnition of “important features”, our method may naturally ﬁnd both pertinent positive and pertinent negative features (Dhurandhar et al., 2018) since both pertinent positive and pertinent negative features are the most suspect to adversarial perturbations, and we demonstrate the idea in
Figure 1.
While exactly computing such an effectiveness measure is NP-hard (Katz et al., 2017), we can leverage recent results from test-time robustness (Carlini & Wagner, 2017; Madry et al., 2017), which entail that perturbations computed by adversarial attacks can serve as reasonably tight upper bounds for our proposed evaluation.
Given this adversarial effectiveness evaluation measure, we further design feature-based explanations that optimize this evaluation measure.
To summarize our contributions:
• We deﬁne new evaluation criteria for feature-based explanations based on robustness analysis involving small adversarial perturbations.
These reduce the bias inherent in other recent evaluation measures that focus on “removing features” via large perturbations to some reference values, or sampling from some reference distribution.
• We design efﬁcient algorithms to generate explanations that maximize the proposed criteria by incorporating game theoretic notions, and demonstrate the effectiveness and interpretability of our proposed explanation on image and
language datasets: via our proposed evaluation metric, additional objective metrics, as well as qualitative results and a human study.
R ELATED W ORK
Our work deﬁnes a pair of new objective evaluation criteria for feature-based explanations, where existing measurements can be roughly categorized into two families.
This ﬁrst family of explanation evaluations are based on measuring ﬁdelity of the explanation to the model.
Here, the feature based explanation is mapped to a simpliﬁed model, and the ﬁdelity evaluations measure how well this simpliﬁed model corresponds to the actual model.
A common setting is where the feature vectors are locally binarized at a given test input, to indicate presence or “removal” of a feature.
A linear model with the explanation weights as coefﬁcients would then equal to the sum of attribution values for all present features.
Completeness or Sum to Delta which requires the sum of all attributions to equal the prediction difference of the original input and baseline (Sundararajan et al., 2017; Shrikumar et al., 2017), while sensitivity-n which further generalize this to require the sums of subsets of attribution values to equal the prediction difference of the input with features present or absent corresponding to the subset, and the baseline (Ancona et al., 2018).
Local accuracy (Ribeiro et al., 2016; Lundberg & Lee, 2017) measures the ﬁdelity of local linear regression model corresponding to explanation weights; while inﬁdelity is a framework that encompasses these instances above (Yeh et al., 2019).
The second family of explanation evaluation are removal- and preservation-based measurements (Samek et al., 2016; Fong & Vedaldi, 2017; Dabkowski & Gal, 2017), which evaluates the ranking of feature-attribution score, with the key insight that removing the most (least) salient features should lead to the most (least) signiﬁcant function value drop.
Unlike previous evaluations, our proposed evaluation criteria are based on more nuanced and smaller adversarial perturbations which does not rely on “removing features”.
Another close line of research is counterfactual and contrastive explanations
(Wachter et al., 2017;
Dhurandhar et al., 2018;
Hendricks et al., 2018; van der Waa et al., 2018;
Chang et al., 2019; Goyal et al., 2019; Joshi et al., 2019; Poyiadzi et al., 2020), which answers the question: “what to alter in the current input to change the outcome of the model”.
These studies enable interpretation of a model decision through a contrastive perspective and facilitate actionable means that one can actually undertake to change the model prediction (Joshi et al., 2019; Poyiadzi et al., 2020).
While “the closest possible world” provided by counterfactual explanations gives insights on the features that the model is most sensitive to, it does not necessarily cover the relevant features responsible for the “current” prediction.
Moreover, while these identify features as relevant when their perturbation leads to prediction changes, it does not necessarily identify features as relevant when their complement is insensitive to perturbations (and where the relevant features are ﬁxed).
Xu et al.
(2018) add group sparsity regularization to adversarial attacks that improves the semantic structure of the adversarial perturbations.
Instead of outputting general adversarial perturbations, our work focuses speciﬁcally on identifying necessary and sufﬁcient relevant features by measuring their susceptibility to adversarial perturbations.
Ribeiro et al.
ﬁnd a set of features that once ﬁxed, the prediction does not change much with high probability when perturbing other features.
They measure this via a sampling approach, which is infeasible for high dimension images without super-pixels.
In contrast, our method consider the class of adversarial perturbations, which is much more computationally feasible compared to sampling high-dimensional spaces.
R OBUSTNESS A NALYSIS FOR E VALUATING E XPLANATIONS 3. P ROBLEM N OTATION
We consider the setting of a general K -way classiﬁcation problem with input space X ⊆ R d , output space Y = { 1 , . .
, K } , and a predictor function f : X → Y where f ( x ) denotes the output class for some input example x
= [ x 1 , . . .
, x d ] ∈ X .
Then, for a particular prediction f ( x ) =
y , despite the different forms of existing feature based explanations ranging from attributing an importance value to each feature, ranking the features by their importance, to simply identify a set of important features, a common goal of them is to extract a compact set of relevant features with respect to the prediction.
3. E VALUATION THROUGH R OBUSTNESS A NALYSIS
A common thread underlying evaluations of feature based explanations, even ranging over axiomatic treatments Sundararajan et al.
; Lundberg & Lee (2017), is that the importance of a set of features corresponds to the change in prediction of the model when the features are removed from the original input.
Nevertheless, as we discussed in the previous section, operationalizing such a removal of features, for instance, by setting them to some reference value, introduces biases.
To ﬁnesse this, we leverage adversarial robustness, but to do so in this context, we rely on two key intuitive assumptions that motivate our method:
Assumption 1: When the values of the important features are anchored (ﬁxed), perturbations restricted to the complementary set of features has a weaker inﬂuence on the model prediction.
Assumption 2: When perturbations are restricted to the set of important features, ﬁxing the values of the rest of the features, even small perturbations could easily change the model prediction.
Based on these two assumptions, we propose a new framework leveraging adversarial robustness to evaluate feature based explanations.
Deﬁnition 3.
Given a set of features S , its minimum adversarial perturbation norm, which we will also term Robustness-
S is deﬁned as:
[Equation]
where S =
U \ S is the complementary set of features, and δ S
= 0 means that the perturbation is constrained to be zero along features in S .
Suppose that the feature based explanation partitions the input features into a relevant set S r , and an irrelevant set S r , Assumption 1 implies that the quality of the relevant set can be measured by (cid:15) ∗ S r – by keeping the relevant set unchanged, and measuring the adversarial robustness norm by perturbing only the irrelevant set.
Speciﬁcally, from Assumption 1, a larger coverage of pertinent features in set
S r entails a higher robustness value (cid:15) ∗ S r .
On the other hand, from Assumption 2, a larger coverage of pertinent features in set S r would in turn entail a smaller robustness value [Table] S r , since only relevant features are perturbed.
More formally, we propose the following twin criteria for evaluating the quality of S r identiﬁed by any given feature based explanation.
Deﬁnition 3.2 We deﬁne Robustness-
S r and Robustness- S r as the following:
[Equation]
Following our assumptions, a set S r that has larger coverage of relevant features would yield higher
Robustness- S r and lower Robustness- S r .
Evaluation for Feature Importance.
While Robustness- S r and Robustness-
S r are deﬁned on sets, general feature attribution based explanations could also easily ﬁt into the evaluation framework.
Given any feature attribution method that assigns importance score to each feature, we can sort the features in descending order of importance weights, and provide the top- K features as the relevant set S r .
The size of K (or | S r | ), can be speciﬁed by the users based on the application.
An alternative approach that we adopt in our experiments is to vary the size of set K and plot the corresponding values of Robustness- S r and Robustness- S r over different values of K .
With a graph where the X − axis is the size of K and the Y − axis is Robustness- S r or Robustness- S r , we are then able to plot an evaluation curve for an explanation and in turn compute its the area under curve (AUC) to summarize its performance.
A larger (smaller) AUC for Robustness- S r (Robustness- S r ) indicates a better feature attribution ranking.
Formally, given a curve represented by a set of points C
=
{ ( x 0 , y 0 ) , . .
, ( x n , y n ) } where x i − 1 < x i , we calculate the AUC of the curve by: AUC ( C ) =
{+(cid:80) ni =
 ( y i + y i − 1 ) / 2 ∗ ( x i − x i − 1 ) .+}
Untargeted v.s. Targeted Explanation.
Deﬁnition 3.1 corresponds to the untargeted adversarial robustness – a perturbation that changes the predicted class to any label other than y is considered as a successful attack.
Our formulation can also be extended to targeted adversarial robustness , where we replace Eqn. 1 by
[Equation]
where t is the targeted class.
Using this deﬁnition, our approach will try to address the question “Why is this example classiﬁed as y instead of t ” by highlighting the important features that contrast between class y and t .
Further examples of the “targeted explanations” are in the experiment section.
Robustness Evaluation under Fixed Anchor Set.
It is known that computing the exact minimum distortion distance in modern neural networks is intractable (Katz et al., 2017), so many different methods have been developed to estimate the value.
Adversarial attacks, such as C&W (Carlini & Wagner, 2017) and PGD attack (Madry et al., 2017), aim to ﬁnd a feasible solution of Eqn. 1, which leads to an upper bound of (cid:15) ∗ S .
They are based on gradient based optimizers which are usually efﬁcient.
On the other hand, neural network veriﬁcation methods aim to provide a lower bound of (cid:15) ∗ S to ensure that the model prediction will not change within certain perturbation range (Singh et al., 2018; Wong & Kolter, 2018;
Weng et al., 2018; Gehr et al., 2018; Zhang et al., 2018; Wang et al., 2018; Zhang et al., 2019).
However, these methods are usually time consuming (often >times slower than a backpropagation).
The proposed framework can be combined with any method that aims to approximately compute Eqn. 1, including attack, veriﬁcation, and some other statistical estimations.
However, for simplicity we only choose to evaluate Eqn. 1 by the state-of-the-art projected gradient descent (PGD) attack (Madry et al., 2017), since the veriﬁcation methods are too slow and often lead to much looser estimation as reported in some recent studies (Salman et al., 2019).
Our additional constraint restricting perturbation to only be on a subset of features speciﬁes a set that is simple to project onto, where we set the corresponding coordinates to zero at each step of PGD.
E XTRACTING R ELEVANT F EATURES THROUGH R OBUSTNESS A NALYSIS
Our adversarial robustness based evaluation allows us to evaluate any given feature based explanation.
Here, we set out to design new explanations that explicitly optimize our evaluation measure.
We focus on feature set based explanations, where we aim to provide an important subset of features
S r .
Given our proposed evaluation measure, an optimal subset of feature S r would aim to maximize (minimize) Robustness-
¯ S r (Robustness- S r ), under a cardinality constraint on the feature set, leading to the following set of optimization problems: [Equation]
[Equation]
where K is a pre-deﬁned size constraint on the set S r , and g ( x , S ) computes the the minimum adversarial perturbation from Eqn. 1, with set-restricted perturbations.
It can be seen that this sets up an adversarial min-max game: the goal of the feature set explainer is to come up with a set S r such that the minimal adversarial perturbation is as large as possible, while the adversarial attacker, given a set S r , aims to design adversarial perturbations that are as small as possible.
Directly solving these min-max problems in Eqn. 3 and Eqn. 4 is thus challenging, which is exacerbated by the discrete input constraint makes it intractable to ﬁnd the optimal solution.
We therefore propose a greedy algorithm in the next section to estimate the optimal explanation sets.
4. G REEDY A LGORITHM TO C OMPUTE O PTIMAL E XPLANATIONS
We ﬁrst consider a greedy algorithm where, after initializing S r to the empty set, we iteratively add to S r the most promising feature that optimizes the objective at each local step until S r reaches the size constraint.
We thus sequentially solve the following sub-problem at every step t :
[Equation]
where S tr is the anchor set at step t , and S 0 r = ∅ .
We repeat this subprocedure until the size of set
S tr reaches K .
A straightforward approach for solving Eqn.
5 is to exhaustively search over every single feature.
We term this method Greedy .
While the method eventually selects K features for the relevant set S r , the latter might lose the sequence in which the features were selected.
One approach to encode this order would be to output a feature explanation that assigns higher weights to those features selected earlier in the greedy iterations.
4. G REEDY BY S ET A GGREGATION S CORE
The main downside of using the greedy algorithm to optimize the objective function is that it ignores the interactions among features.
Two features that may seem irrelevant when evaluated separately might nonetheless be relevant when added simultaneously.
Therefore, in each greedy step, instead of considering how each individual feature will marginally contribute to the objective g ( · ) , we propose to choose features based on their expected marginal contribution when added to the union of S r and a random subset of unchosen features.
To measure such an aggregated contribution score, we draw from cooperative game theory literature (Dubey & Shapley, 1979; Hammer & Holzman, 1992) to reduce this to a linear regression problem.
Formally, let S tr and S tr be the ordered set of chosen and unchosen features at step t respectively, P t ( S tr ) be all possible subsets of S tr .
We measure the expected contribution of including each unchosen feature to the relevant set would have on objective function by learning the following regression problem:
[Equation]
where B is a function that projects a set into its corresponding binary vector form: B ( A )
[ j ] =
I ( S tr [ j ] ∈ A ) , i.e., ones in the vector indicate the inclusion of corresponding feature indices in the set and zeros otherwise.
We note that w t corresponds to the well-known Banzhaf value Banzhaf III (1964) when S tr = ∅ , which can be interpreted as the importance of each player by taking coalitions into account Dubey & Shapley (1979).
Hammer & Holzman (1992) show that the Banzhaf value is equivalent to the optimal solution of linear regression with pseudo-Boolean functions as targets, which corresponds to Eqn.
6 with S tr = ∅ .
We can thus treat the linear regression coefﬁcients w in
Eqn. 6 as each corresponding feature’s expected marginal contribution when added to the union of S r and a random subset of unchosen features.
We thus consider the following set-aggregated variant of our greedy algorithm in the previous section, and which we term Greedy-AS .
In each greedy step, we choose features with the highest aggregation score (Banzhaf) value rather than simply the highest marginal contribution to the objective function as in vanilla greedy, which additionally considers interactions among the unchosen features when compared to vanilla greedy.
The chosen features each step are then added to S tr and removed from S tr .
When S tr is not ∅ , the solution of Eqn. 6 can still be seen as the Banzhaf value where the players are the unchosen features in S tr , and the value function of a subset of players is equal to the adversarial perturbation objective that adds this subset of players to the current set of chosen features S tr . We The higher the better for Robustness- S r ; the lower the better for Robustness- S r
[Table] solve the linear regression problem in Eqn. 6 by sub-sampling to lower the computational cost, and we validate the effectiveness of Greedy and Greedy-AS in the experiment section.
E XPERIMENTS
In this section, we ﬁrst evaluate different model interpretability methods on the proposed criteria.
We justify the effectiveness of the proposed Greedy-AS.
We then move onto further validating the beneﬁts of the explanations extracted by Greedy-AS through comparisons to various existing methods both quantitatively and qualitatively.
Finally, we demonstrate the ﬂexibility of our method with the ability to provide targeted explanations as mentioned in Section 3.2.
We perform the experiments on two image datasets, MNIST LeCun et al.
(2010) and ImageNet (Deng et al., 2009), as well as a text classiﬁcation dataset, Yahoo!
Answers (Zhang et al., 2015).
On MNIST, we train a convolutional neural network (CNN) with 99% testing accuracy. On ImageNet, we deploy a pre-trained ResNet model obtained from the Pytorch library.
On Yahoo!
Answers, we train a BiLSTM sentence classiﬁer which attains testing accuracy of 71%.
Setup.
In the experiments, we consider p = 2 for (cid:107) · (cid:107) p in Eqn.
1 and Eqn. 2, i.e., the L 2 norm if not otherwise speciﬁed. We note that Eqn.
1 corresponds to the robustness on a single data point.
We take the average value to report the overall robustness over multiple examples. For all quantitative results including the evaluation curves and the corresponding AUCs, we report the average over 100 random examples.
For the baseline methods, we include vanilla gradient (Grad) (Shrikumar et al., 2017) and integrated gradient (IG) (Sundararajan et al., 2017) from gradient-based approaches; leave-one-out (LOO), or occlusion-1, (Zeiler & Fergus, 2014; Li et al., 2016) and SHAP (Lundberg & Lee, 2017) from perturbation-based approaches (Ancona et al., 2018); black-box meaningful perturbation (BBMP) (Fong & Vedaldi, 2017) from removal/preservation-based approaches; and counterfactual explanation (CFX) proposed by Wachter et al.
For the proposed Greedy and Greedy-AS, at each greedy iteration, we include the top-5% features with highest scores into the relevant set to further speed up the selection process.
We leave more implementation detail in Appendix C due to space limitation.
5. R OBUSTNESS A NALYSIS ON M ODEL I NTERPRETABILITY M ETHODS
Here we compare Greedy-AS, ablated variants of Greedy-AS, as well as various existing explanation methods under both the proposed evaluation criteria Robustness-
S r and Robustness- S r .
For ease of comparison, we calculate the area under curve (AUC) for each corresponding evaluation curves.
We next discuss these results, listed in Table 1, with the detailed plots in Appendix E.
Ablation Study on Greedy-AS.
As discussed in Section 4.2, Greedy-AS could be seen as a combination of the original greedy procedure with the approximated contribution of each feature computed by a regression.
Here, we examine the importance of both components by comparing the
Greedy-AS method to two baselines, where one selects important features based only on the pure
Greedy method (Section 4.1) and the other utilizes only a single step of regression without the iterative greedy procedure.
As the latter essentially corresponds to the Banzhaf value, we term this method as
One-Step Banzhaf .
As shown in Table 1, the pure Greedy method suffers degraded performances comparing to Greedy-AS under both criteria.
The inferior performance could be explained by the ignorance of feature correlations which ultimately results in the introduction of noise (see Figure 6 in
Since original no control over the number of features that could differ from the input, we add L constraint to enforce sparsity in the difference and enable selection of top- most features.
[Table]
[Table] [Table] appendix). In addition, we also see that Greedy-AS performs better than One-Step Banzhaf.
This could results from the fact that One-Step Banzhaf considers the feature interactions among all features with equal probability.
However, in our objective, we only care about those interactions with the most important features.
By iteratively selecting the features with highest Banzhaf value in Greedy-AS, we give more weight on the interactions among the most important features through iterations, and as a result lead to better performance.
Comparisons between Different Explanations.
Furthermore from Table 1, we observe that the proposed Greedy-AS consistently outperforms other explanation methods on both criteria (see
Appendix H for statistical signiﬁcance of the improvements).
On one hand, this suggests that the proposed algorithm indeed successfully optimizes towards the criteria; on the other hand, this might indicate the proposed criteria do capture different characteristics of explanations which most of the current explanations do not possess.
Another somewhat interesting ﬁnding from the table is that while vanilla gradient has generally been viewed as a baseline method, it nonetheless performs competitively on the proposed criteria.
We conjecture the phenomenon results from the fact that
Grad does not assume any reference value as opposed to other baselines such as LOO which sets the reference value as zero to mask out the inputs.
Indeed, it might not be surprising that Greedy-AS achieves the best performances on the proposed criteria since it is explicitly designed for so.
To more objectively evaluate the usefulness of the proposed explanation, we demonstrate different advantages of our method by comparing Greedy-AS to other explanations quantitatively on existing commonly adopted measurements, and qualitatively through visualization in the following subsections.
5. E VALUATING G REEDY -AS
The Insertion and Deletion Metric.
To further justify the proposed explanation not only performs well on the very metric it optimizes, we evaluate our method on a suite of existing popular quantitative measurements.
In particular, we adopt the Deletion and Insertion criteria proposed by Petsiuk et al.
(2018), which are generalized variants of the region perturbation criterion presented in Samek et al.
The Deletion criterion measures the probability drop in the predicted class as top-relevant features, indicated by the given explanation, are progressively removed from the input.
On the other hand, the Insertion criterion measures the increase in probability of the predicted class as top-relevant features are gradually revealed from the input whose features are originally all masked.
Similar to our proposed criteria, a quick drop (and thus a small area under curve) or a sharp increase (that leads to a large area under curve) in Deletion and Insertion respectively suggest a good explanation as the selected top-important features could indeed greatly inﬂuence the prediction.
In the experiments, we follow Samek et al.
(2016) to remove features by setting their values to randomly sampled values.
We plot the evaluation curves (in Appendix F) and report corresponding AUCs in Table 2.
On these additional two criteria, we observe that Greedy-AS performs favorably against other explanations (see
Appendix H for statistical signiﬁcance of the performance differences).
The results further validate the beneﬁts of the proposed explanation.
We note that on ImageNet, SHAP obtains a better performance under the Deletion criterion.
We however suspect such performance comes from adversarial artifacts (that are the features vulnerable to adversarial attack while not being semantically meaningful), since the explanation provided by SHAP seems to be rather noisy (as shown in Figure 3), and it has been observed that the Deletion criterion tends to favor adversarial artifacts in several previous work.
(Dabkowski & Gal, 2017; Chang et al., 2019).
Sanity Check Metric.
In Appendix G, we conduct experiments to verify that Greedy-AS could indeed pass the sanity check proposed by Adebayo et al.
5. Q UALITATIVE R ESULTS
Image Classiﬁcation.
To complement the quantitative measurements, we show several visualization results on MNIST and ImageNet in Figure 2 and Figure 3.
More examples could be found in features different
- tions. We see Greedy-AS both - cial positive and features supporting the prediction.
Appendix I and J. On MNIST, we observe that existing explanations tend to highlight mainly on the white pixels in the digits; among which SHAP and LOO show less noisy explanations comparing to Grad and IG.
On the other hand, the proposed Greedy-AS focuses on both the “crucial positive” (important white pixels) as well as the “pertinent negative” (important black regions) that together support the prediction.
For example, in the ﬁrst row, a 7 might have been predicted as a 4 or 0 if the pixels highlighted by Greedy-AS are set to white.
Similarly, a 1 may be turned to a 4 or a 7 given additional white pixels to its left, and a 9 may become a 7 if deleted the lower circular part of its head.
From the results, we see that Greedy-AS focuses on “the region where perturbation on its current value will lead to easier prediction change” , which includes both the crucial positive and pertinent negative pixels.
Such capability of Greedy-AS is also validated by its superior performance on the proposed robustness criteria, on which methods like LOO that highlights only the white strokes of digits show relatively low performance.
The capability of capturing pertinent negative features has also been observed in explanations proposed in some recent work Dhurandhar et al.
; Bach et al.
; Oramas et al.
and we further provide more detailed discussions and comparisons to these methods in Appendix K. From the visualized ImageNet examples shown in Figure 3, we observe that our method provides more compact explanations that focus mainly on the actual objects being classiﬁed.
For instance, in the ﬁrst image, our method focuses more on the face of the Maltese while others tend to have noisier results; in the last image, our method focuses on one of the Japanese
Spaniel whereas others highlight both the dogs and some noisy regions.
Text Classiﬁcation.
In addition to image datasets, here we demonstrate how our explanation method could be applied to text classiﬁcation models.
In the experiments, we represent a length- n sentence by n embedding vectors following the common setting.
We could then view each input consisting of n × d features, where d is the word embedding size.
When applying our Greedy algorithm, at each iteration we will try to add an embedding vector to the relevant set S r and choose the one with largest reward.
Since there are only at most n choices, the Greedy algorithm doesn’t suffer much from noise and has similar behavior to Greedy-AS.
We perform experiments on an
LSTM network which classiﬁes a given sentence into one of the ten classes (Society, Science, . .
Health).
We showcase an example with explanations generated with different methods in Figure 4.
We note that although the top-5 relevant keyword generated by the three methods do not vary much, the rankings within the highlighted keywords for each explanation are quite different.
We conduct a user study by asking the users to label keywords they believe to be the most relevant to the prediction.
As an example, to predict the label of “sport”, one might consider “cleats”, “football”, and “cut” as the strongest indications towards the concept “sport”, and we ask 10 users to label keywords in the sentence that are the most relevant to the class, and compare the top words highlighted by explanations to the top words chosen by users.
We observe that our method Greedy generates explanation that matches human intuition the most in the user study, and we report details of the user study and additional examples in Appendix O and N.
Targeted Explanation Analysis.
Recall that in section 3.2, we discussed about the possibility of deﬁning the robustness measurement by considering a targeted distortion distance as formulated in
Eqn. 2.
Here, we provide examples, as shown in Figure 5, where we answer the question of “why the input digit is an A but not a B” by deﬁning a targeted perturbation distance towards class B as our robustness measurement.
In each row of the ﬁgure, we provide targeted explanation towards two other methods closely related to the “sport” different target classes for a same input image.
Interestingly, as the target class changes, the generated explanation varies in an interpretatble way.
For example, in the ﬁrst row, we explain why the input digit 7 is not classiﬁed as a 9 (middle column) or a 2 (rightmost column).
The resulting explanation against 9 highlights the upper-left part of the 7.
Semantically, this region is indeed pertinent to the classiﬁcation between 7 and 9, since turning on the highlighted pixel values in the region (currently black in the original image) will then make the 7 resemble a 9.
However, the targeted explanation against 2 highlights a very different but also meaningful region, which is the lower-right part of the 7; since adding a horizontal stroke on the area would turn a 7 into a 2.
C ONCLUSION
In this paper, we establish the link between a set of features to a prediction with a new evaluation criteria, robustness analysis, which measures the minimum tolerance of adversarial perturbation.
Furthermore, we develop a new explanation method to ﬁnd important set of features to optimize this new criterion.
Experimental results demonstrate that the proposed new explanations are indeed capturing signiﬁcant feature sets across multiple domains.
A B IAS FOR R EFERENCE -V ALUE M ETHODS A. B IAS FOR R EFERENCE -V ALUE B ASED E XPLANATIONS
Theorem A.1 For any reference-value based explanation with the form φ ( f, x, x (cid:48) ) = ( x − x (cid:48) ) ⊗
K ( f, x, x (cid:48) ) for any meta-function K, we know that if x i = x
(cid:48)
i , φ i ( f, x ) = 0 .
That is, when ever a feature value coincides with the baseline value x (cid:48) , it will get 0 attribution even if it is the main feature contributing to the prediction.
We call such explanations biased to the reference value x (cid:48) .
Corollary A.1 IG, LRP, DeepLift are biased to the reference value x (cid:48) .
Following the deﬁnition in (Ancona et al., 2018), let ∂ g x c
∂x
i
=
(cid:80) p ∈ P ic (cid:81) w p (cid:81) g ( z ) p where P ic is the set of all paths that connect an unit i to unit c in a deep neural network, w p are the weights existing in path p , z be the value of an unit before nonlinear activation, and g () be any generic function.
Note that if g () = f (cid:48) () , the deﬁnition corresponds to the standard partial derivative of unit c to unit i .
φ IGi ( f, x ) =
( x i − x (cid:48) i ) · K ( f, x, x (cid:48) ) , where K ( f, x, x (cid:48) )
=
(cid:90)α = ∂f ( x (cid:48) + α ( x − x (cid:48) ))
∂x i dα (7) φ LRPi ( f, x ) =
( x i − x (cid:48) i ) · K ( f, x, x (cid:48) ) , where x (cid:48)
{+= 
+}
and K ( f, x,
[Equation]
(cid:48)
[Equation]
=
[Equation]
i with g = f ( z ) z (8) φ DeepLifti ( f, x ) =
( x i − x (cid:48) i ) · K ( f, x, x (cid:48) ) , where K ( f, x, x (cid:48) )
=
∂ g f ( x ) ∂x i with g = f ( z ) − f
( z (cid:48) )
z − z (cid:48) (9)
For example, if the reference value x (cid:48) is a zero vector (which means black pixels in image), then any black pixel will get a zero attribution value no matter if the object of interest is actually back.
Deﬁnition A.1 For any reference-value based explanation with the form φ ( f, x, x (cid:48) )
=
( x − x (cid:48) ) ⊗
K ( f, x, x (cid:48) ) for any meta-function K, we deﬁne its Averaging counterpart as:
[Equation]
[Equation]
{+Theorem A.
+}
For any average reference-value based explanation with the form φ ( f, x, x (cid:48) )
=
E x (cid:48) ∼ p b
[( x − x (cid:48) ) ⊗ K ( f, x, x (cid:48) )
] for some meta function K , if x i = E x (cid:48) ∼ p b
[ x (cid:48)
i ] , φ i ( f, x ) = 0 .
That is, they are biased to the reference value
E x (cid:48) ∼ p b [ x (cid:48) ] .
Corollary A.2 Averaging LRP over multiple baselines are bias to the reference value
E x (cid:48) ∼ p b [ x (cid:48) ] .
[Equation]
11) where K ( f, x, x (cid:48) )
=
∂ g f ( x ) ∂x i with g = f ( z ) z
{+Theorem A.
+}
The baseline Shapley value (Sundararajan & Najmi, 2019) is biased to reference value x (cid:48) .
[Equation]
where v ( s ) =
f ( x S ; x (cid:48)
N \ S )
and N is the set of all features.
These above analysis shows that more explanations are also biased to reference value.
We note that by Expected Gradient (Erion et al., 2019) is not biased to reference value by this theoretic deﬁnition since the baseline x (cid:48) is actually a distribution.
However, for feature values x i that are close to the distribution of x (cid:48)
i , the attribution score will be lower (but not 0 as our theoretic deﬁnition), when feature values x i are far from the distribution of x (cid:48)
i , the the attribution score will be larger, which still shows some level of bias.
We leave this to future work to use more advanced analysis techniques to quantify such a biased for explanations when the baseline follows a distribution.
A. B IAS FOR R EFERENCE - VALUE B ASED E VALUATIONS
Deﬁnition A.2 (Informal) Given a data point x , a reference point x (cid:48) , the deletion score of a model f given a set of important features S (and denote the complement of S as ¯ S ), which we abbreviate as
DEL ( x, x (cid:48) , S ) , is deﬁned as f ( x (cid:48) S ; x ¯ S ) .
Here, x (cid:48) can be a ﬁxed value or a random value, or even a blurred component (by a slight abuse of notations).
Similarly, we deﬁne the insertion score for a data point x , a reference point x (cid:48) , important set of features S , and a model f as INSR ( x, x (cid:48) , S ) =
( x S ; x (cid:48) ¯ S ) .
Note that the lower the DEL score is better, and the higher the INSR score is better.
{+Theorem A.
+}
If x i
= x (cid:48)
i , then DEL ( x, x (cid:48) , S ) =
DEL ( x, x (cid:48) , S ∪ i ) for i (cid:54)⊆ S .
Similarly, if x i = x (cid:48)
i ,
INSR ( x, x (cid:48) , S ) =
INSR ( x, x (cid:48) , S ∪ i ) for i (cid:54)⊆ S
The implication of Thm.
A.4 is that if a feature happens to correspond to the reference value, no matter how important it actually is, it will not contribute to the DEL score nor the INSR score.
However, the lower the DEL score is better, and the higher the INSR score is better, and choosing an important feature that corresponds to the reference value will inevitably not improve the DEL score and INSR score.
Thus, DEL score and INSR score will deem feature with the same value as the reference value as non-important even if the features are actually crucial.
Our work relies on measuring the adversarial perturbation norm on a subset of features, which can be seen as a constrained adversarial robustness problem.
Adversarial robustness has been extensively studied in the past few years.
The adversarial robustness of a machine learning model on a given sample can be deﬁned as the shortest distance from the sample to the decision boundary, which corresponds to our deﬁnition in Eqn. 1.
Algorithms have been proposed for ﬁnding adversarial examples (feasible solutions of Eqn. 1), including (Goodfellow et al., 2014; Carlini & Wagner, 2017;
Madry et al., 2017).
However, those algorithms only work for neural networks, while for other models such as tree based models or nearest neighbor classiﬁers, adversarial examples can be found by decision based attacks (Brendel et al., 2017; Cheng et al., 2018; Chen et al., 2019).
Therefore the proposed framework can also be used in other decision based classiﬁers.
On the other hand, several works aim to solve the neural network veriﬁcation problem, which is equivalent to ﬁnding a lower bound of Eqn.
Examples include (Singh et al., 2018; Wong & Kolter, 2018; Zhang et al., 2018).
In principal, our work can also apply these veriﬁcation methods for getting an approximate solution of
Eqn. 1, but in practice they are very slow to run and often gives loose lower bounds on regular trained networks.
C I MPLEMENTATION D ETAIL
Underlying Models Used.
In the experiments conducted on MNIST dataset, we train a convolutional neural network (CNN) with 99% testing accuracy.
The training and testing split used in the experiments are the default split as provided by the original dataset.
For each instance, the inputs are loaded into the range of [0 , 1] .
On ImageNet dataset, we deploy a pre-trained ResNet model obtained from the Pytorch library.
For each image, the inputs are ﬁrst loaded into the range of [0 , 1] and normalized using mean =
456 , 0 . 406] and standard deviation = [0 .
On Yahoo!
Answers dataset, we train a BiLSTM sentence classiﬁer which attains testing accuracy of 71%.
For the model conﬁguration and training hyperparameters, we strictly follow the ofﬁcial GLUE repository 5 .
Speciﬁcally, we ﬁrst tokenize the dataset by word-level tokenizer; and then we use the pretrained GloVe word embedding to initialize the LSTM embedding layer where the hidden size is 300.
Hyperparameters.
As discussed in Section 3.2, we use PGD attack with binary search to approximately compute the robustness value in Eqn. 1.
In the experiments, we set the PGD attack step size to be 1.0 and number of steps to be 100.
The hyperparameters are chosen such that the PGD attack [Figure] [Figure] could most efﬁciently provide the tightest upper bound on the true robustness value.
As mentioned in Section 4.2, we solve Eqn. 6 by subsampling from all possible subsets of S tr .
Speciﬁcally, we compute the coefﬁcients w with respect to 5000 sampled subsets when learning the regression.
Evaluation Curves.
To plot the evaluation curves of a given explanation, we measure the explanation’s Robustness-
S r and Robustness- S r at different sizes of relevant set | S r | .
Speciﬁcally, we measure the explanation’s performance when | S r | equals to 5%, 10%, . . .
, 45% of the total number of features.
Similarly for the Insertion and Deletion criteria, we progressively evaluate the explanation’s performance when 5%, 10%, . . .
, 45% of the features are inserted (or removed) from the inputs.
After the evaluation curves are plotted, we then calculate the area under curves to summarize the explanation’s overall performance under each criterion
Hardware and Code.
All the experiments were performed on Intel(R) Xeon(R) CPU E5-2630 v @
2.20GHz and NVIDIA GeForce GTX 1080 Ti GPU.
We release our source code along with an example Jupyter notebook for more information.
D V ISUALIZE OF G REEDY , O NE -S TEP B ANZHAF , AND G REEDY -AS
We compare the visualization results of Greedy, One-Step Banzhaf, and Greedy-AS on an MNIST image, as shown in Figure 6.
We see that the top features selected by Greedy-AS are less noisy than the other two.
E E VALUATION C URVES UNDER R OBUSTNESS - S r AND R OBUSTNESS - S r
We show the evaluation curves of different methods under the proposed robustness criteria on MNIST and ImangeNet in Figure 7 and Figure 8 respectively.
For ease of comparison, on each dataset, we show the curves between Greedy-AS, One-Step Banzhaf, and Greedy in a set of plots; and we show the curves between Greedy-AS and other existing methods in another set of plots.
F E VALUATION C URVES UNDER I NSERTION AND D ELETION
We show the evaluation curves of different methods under the Insertion and Deletion criteria on
MNIST and ImageNet in Figure 9a and Figure 9b respectively.
G S ANITY C HECK
Recent literature has pointed out that an appropriate explanation should be related to the model being explained (Adebayo et al., 2018).
To ensure that our proposed explanation does indeed reﬂect the model behavior, we conduct the sanity check proposed by Adebayo et al.
(2018) to check if our explanations are adequately different when the model parameters are randomly re-initialized.
In the experiment, we randomly re-initialize the last fully-connected layer of the neural network model.
We then compute the rank correlation between explanation computed w.r.t.
the original model and that w.r.t.
the randomized model.
From Table 3, we observe that Greedy-AS has a much lower rank correlation comparing to Grad, IG, and LOO, suggesting that Greedy-AS is indeed sensitive to model parameter change and is able to pass the sanity check.
[Table]
H S TUDENT ’ S t - TEST ON THE P ERFORMANCE I MPROVEMENT OF G REEDY -AS OVER E XISTING M ETHODS
We conduct a set of Student’s t -tests to further verify that the outperformances of the proposed
Greedy-AS over existing methods are indeed statistically signiﬁcant.
We compare the AUCs of
Greedy-AS and other methods across different criteria.
In Table 4, we show the t -test results on the AUCs of Robustness- S r and Robustness- S r , as well as the results on the AUCs of Insertion and
Deletion criteria.
In the tables, win ( loss ) indicates that the performance improvement (drop) of
Greedy-AS to the baselines is statistically signiﬁcant; otherwise, we use draw to indicate insigniﬁcant difference between the performances.
From the results, we positively verify the statistical signiﬁcance of the performance improvements of our proposed method over existing explanations on various criteria.
[Table]
I V ISUALIZATION ON MNIST
We provide more examples on visualized explanations on MNIST in Figure [Table]
J V ISUALIZATION ON I MAGE N ET
We provide more examples on visualized explanations on ImageNet in Figure 11.
[Figure]
K C OMPARISONS WITH O THER E XPLANATIONS C APTURING P ERTINENT N EGATIVE F EATURES
While the capability of capturing not only the crucial positive but also the pertinent negative features have also been observed in some recently proposed explanations.
Most current methods are not explicitly designed to handle the targeted explanation task which attempt to answer the question “what are the important features that lead to the prediction of class A but not class B”, and thus has different limitations.
For example, the ability of LRP Bach et al.
(2015) to capture pertinent negative features heavily depends on the input range.
In Samek et al.
(2016), the input images are normalized to have zero mean and a standard deviation of one where the black background will have non-zero value.
In such case, LRP would have non-zero attributions on the black background pixels which allows the explanation to capture pertinent negative features.
However, as later shown in Dhurandhar et al.
(2018), if the input pixel intensity is normalized into the range between 0 and 1 where the background pixels have the values of 0, LRP failed to highlight pertinent negative pixels since the background would always have zero attribution.
This is due to the fact that LRP is equivalent to multiplication between Grad and input in a Rectiﬁed Linear Unit (ReLU) network as shown in Ancona et al.
In Oramas et al.
(2019), unlike our targeted explanation where we know exactly which targeted class the explanation is suggesting against (and by varying the targeted class we observe varying corresponding explanation given), their method by design does not convey such information.
The pertinent negative features highlighted by their method by construction is not directly related to a speciﬁc target class, and users would need to infer by themselves what target class the pertinent negative features are preventing against.
To further grasp the difference, we compare our explanation with theirs in Figure 12.
We borrow the results from Oramas et al.
for visualization of their method.
Qualitatively, we observe that our method gives the most intuitive explanations.
For example, in Figure 12a, the ﬁrst row shows different explanations that highlight different relevant regions supporting the prediction of a 2 instead of 0.
Among the explanations, our method is the only one that highlights the right tail part (green circled) of the digit 2 which also serves as crucial evidence of 2 against 0 in addition to the left vertical gap (which when presented would make 2 looks like a 0) that is roughly highlighted by all three methods.
Furthermore, as we change the target class to(as shown in the second row), the explanation provided by LRP does not seem to differ much.
This suggests that LRP is not sensitive to the target class change and might fail to provide useful insights towards targeted explanation.
On the contrary, we observe that our explanation has a drastic change between different target classes.
When explaining against the target 7, our explanation highlights the lower left green circled part which when turned off will make 2 becomes a 7.
These results might suggest our method is more capable of handling such targeted explanation task.
L A NALYSIS ON THE E FFECT OF R EFERENCE V ALUE
We provide below an empirical analysis on how different reference values would affect the evaluation results of different explanation methods.
[Table]
[Table]
286.40 Deletion 364.28 340.24 279.51 325.71 701.97 375.87 329.
Reference value =
Insertion 109.89 136.
189.96 Deletion 245.57 218.09 207.57 234.28 658.54 329.20 203.
[Figure]
[Figure]
[Figure]
[Figure]
M Q UANTITATIVE R ESULTS ON T EXT D ATASET
[Table] [Figure] N Q UALITATIVE R ESULTS ON T EXT D ATASET We provide more example explanations on text classiﬁcation problem in Figure 18.
O U SER S TUDY ON NLP DATASET
To further compare the top-5 relevant keywords from various methods
[Table]
we conducted an user study where we show 10 different users with Computer Science background all with a same set ofsentences with their corresponding categories. We then asked each user to highlight 5 to 7 keywords for each sentence that they consider to be most relevant to the ground truth label of the corresponding sentence.
After collecting the feedback, for each sentence, we select the keywords that have the highest frequency of being marked among the users and treat those keywords as the ground truth explanation for the corresponding sentence.
We leverage such user labels to calculate the precision@k and mean Average Precision (mAP) for each explanation method.
We show the results in Tablewhere we observe that Greedy has the highest precision score.
If we believe that users can correctly pick the correct keywords that are responsible for the model prediction, then the p@k and mAP measure how well the explanation captures the correct keywords that are responsible for the model prediction.
We believe the good result on user study provides extra evidence that our method identiﬁes features that are responsible for the actual prediction.
[Table]
[Figure]