E VALUATIONS AND M ETHODS FOR E XPLANATION THROUGH R OBUSTNESS A NALYSIS
A BSTRACT
Feature based explanations , that provide importance of each feature towards the model prediction, is arguably one of the most intuitive ways to explain a model.
In this paper, we establish a novel set of evaluation criteria for such feature based explanations by robustness analysis.
In contrast to existing evaluations which require us to specify some way to “remove” features that could inevitably introduces biases and artifacts, we make use of the subtler notion of smaller adversarial perturbations.
By optimizing towards our proposed evaluation criteria, we obtain new explanations that are loosely necessary and sufﬁcient for a prediction.
We further extend the explanation to extract the set of features that would move the current prediction to a target class by adopting targeted adversarial attack for the robustness analysis.
Through experiments across multiple domains and a user study, we validate the usefulness of our evaluation criteria and our derived explanations.
I NTRODUCTION
There is an increasing interest in machine learning models to be credible, fair, and more generally interpretable (Doshi-Velez & Kim, 2017).
Researchers have explored various notions of model interpretability, ranging from trustability (Ribeiro et al., 2016), fairness of a model (Zhao et al., 2017), to characterizing the model’s weak points (Koh & Liang, 2017; Yeh et al., 2018).
Even though the goals of these various model interpretability tasks vary, the vast majority of them use so called feature based explanations , that assign importance to individual features (Baehrens et al., 2010;
Simonyan et al., 2013; Zeiler & Fergus, 2014; Bach et al., 2015; Ribeiro et al., 2016; Lundberg &
Lee, 2017; Ancona et al., 2018; Sundararajan et al., 2017; Zintgraf et al., 2017; Shrikumar et al., 2017; Chang et al., 2019).
There have also been a slew of recent evaluation measures for feature based explanations, such as completeness (Sundararajan et al., 2017), sensitivity-n (Ancona et al., 2018), inﬁdelity (Yeh et al., 2019), causal local explanation metric (Plumb et al., 2018), and most relevant to the current paper, removal- and preservation-based criteria (Samek et al., 2016; Fong & Vedaldi, 2017; Dabkowski & Gal, 2017; Petsiuk et al., 2018).
A common thread in all these evaluation measures is that for a good feature based explanation, the most salient features are necessary, in that removing them should lead to a large difference in prediction score, and are also sufﬁcient in that removing non-salient features should not lead to a large difference in prediction score.
Thus, common evaluations and indeed even methods for feature based explanations involve measuring the function difference after “removing features”, which in practice is done by setting the feature value to some reference value (also called baseline value sometimes).
However, this would favor feature values that are far way from the baseline value (since this corresponds to a large perturbation, and hence is likely to lead to a function value difference), causing an intrinsic bias for these methods and evaluations.
For example, if we set the feature value to black in RGB images, this introduces a bias favoring bright pixels: explanations that optimize such evaluations often omit important dark objects such as a dark-colored dog.
An alternative approach to “remove features” is to sample from some predeﬁned distribution or a generative model (Chang et al., 2019).
This nevertheless in turn incurs the bias inherent to the generative model, and accurate generative models that approximate the data distribution well might not be available in all domains.
In this work, instead of deﬁning prediction changes with “removal” of features (which introduces biases as we argued), we alternatively consider the use of small but adversarial perturbations .
It is natural to assume that adversarial perturbations on irrelevant features should be ineffective, while those on relevant features should be effective.
We can thus measure the necessity of a set of relevant features, provided by an explanation, by measuring the consequences of adversarially perturbing their feature values: if the features are indeed relevant, this should lead to an appreciable change in the predictions.
Complementarily, we could measure the sufﬁciency of the set of relevant features via measuring consequences of adversarially perturbing its complementary set of irrelevant features: if the perturbed features are irrelevant, this should not lead to an appreciable change in the predictions.
We emphasize that by our deﬁnition of “important features”, our method may naturally identify both pertinent positive and pertinent negative features (Dhurandhar et al., 2018) since both pertinent positive and pertinent negative features are the most susceptible to adversarial perturbations, and we demonstrate the idea in Figure 1.
While exactly computing such an effectiveness measure is NP-hard (Katz et al., 2017), we can leverage recent results from test-time robustness (Carlini & Wagner, 2017; Madry et al., 2017), which entail that perturbations computed by adversarial attacks can serve as reasonably tight upper bounds for our proposed evaluation.
Given this adversarial effectiveness evaluation measure, we further design feature based explanations that optimize this evaluation measure.
To summarize our contributions:
• We deﬁne new evaluation criteria for feature based explanations by leveraging robustness analysis involving small adversarial perturbations.
These reduce the bias inherent in other recent evaluation measures that focus on “removing features” via large perturbations to some reference values, or sampling from some reference distribution.
• We design efﬁcient algorithms to generate explanations that optimize the proposed criteria by incorporating game theoretic notions, and demonstrate the effectiveness and interpretability of our proposed explanation on image and language datasets: via our proposed evaluation metric, additional objective metrics, as well as qualitative results and a user study.
R ELATED W ORK
Our work deﬁnes a pair of new objective evaluation criteria for feature based explanations, where existing measurements can be roughly categorized into two families.
This ﬁrst family of explanation evaluations are based on measuring ﬁdelity of the explanation to the model.
Here, the feature based explanation is mapped to a simpliﬁed model, and the ﬁdelity evaluations measure how well this simpliﬁed model corresponds to the actual model.
A common setting is where the feature vectors are locally binarized at a given test input, to indicate presence or “removal” of a feature.
A linear model with the explanation weights as coefﬁcients would then equal to the sum of attribution values for all present features.
Completeness or Sum to Delta requires the sum of all attributions to equal the prediction difference between the original input and a baseline input (Sundararajan et al., 2017; Shrikumar et al., 2017), while Sensitivity-n further generalize this to require the sums of subsets of attribution values to equal the prediction difference of the input with features present or absent corresponding to the subset, and the baseline (Ancona et al., 2018).
Local accuracy (Ribeiro et al., 2016; Lundberg & Lee, 2017) measures the ﬁdelity of local linear regression model corresponding to explanation weights; while Inﬁdelity is a framework that encompasses these instances above (Yeh et al., 2019).
The second family of explanation evaluations are removal- and preservation-based measurements (Samek et al., 2016; Fong & Vedaldi, 2017; Dabkowski & Gal, 2017; Petsiuk et al., 2018), which evaluate the ranking of feature attribution score, with the key insight that removing the most (least) salient features should lead to the most (least) signiﬁcant function value difference.
Unlike previous evaluations, our proposed evaluation criteria are based on more nuanced and smaller adversarial perturbations which do not rely on operationalizing “feature removal”.
Another close line of research is counterfactual and contrastive explanations
(Wachter et al., 2017;
Dhurandhar et al., 2018;
Hendricks et al., 2018; van der Waa et al., 2018;
Chang et al., 2019; Goyal et al., 2019; Joshi et al., 2019; Poyiadzi et al., 2020), which answer the question: “what to alter in the current input to change the outcome of the model”.
These studies enable interpretation of a model decision through a contrastive perspective and facilitate actionable means that one can actually undertake to change the model prediction (Joshi et al., 2019; Poyiadzi et al., 2020).
While “the closest possible world” provided by counterfactual explanations sheds light on the features that are “necessary” for the prediction, it could fail to identify relevant features that are “sufﬁcient” for the current prediction.
Xu et al.
(2018) add group sparsity regularization to adversarial attacks that improves the semantic structure of the adversarial perturbations.
Instead of outputting general adversarial perturbations, our work focuses speciﬁcally on identifying necessary and sufﬁcient relevant features by measuring their susceptibility to adversarial perturbations.
Ribeiro et al.
ﬁnd a set of features that once ﬁxed, the prediction does not change much with high probability when perturbing other features.
They measure this via a sampling approach, which could be infeasible for high-dimensional images without super-pixels.
In contrast, our method considers the class of adversarial perturbations, which is much more computationally feasible compared to sampling highdimensional spaces.
Guidotti et al.
(2019) exploit latent feature space to learn local decision rules with image exemplars that support the current prediction and counter-exemplars that contrast the current prediction.
While the explanation is capable of highlighting both areas pertinent to the prediction and areas that pushes the prediction towards another class, the method relies on an effective generative model which might not always be available in different applications.
R OBUSTNESS A NALYSIS FOR E VALUATING E XPLANATIONS 3. P ROBLEM N OTATION
We consider the setting of a general K -way classiﬁcation problem with input space X ⊆ R d , output space Y = { 1 , . .
, K } , and a predictor function f : X → Y where f ( x ) denotes the output class for some input example x
= [ x 1 , . . .
, x d ] ∈ X .
Then, for a particular prediction f ( x ) =
y , a common goal of feature based explanations is to extract a compact set of relevant features with respect to the prediction. We denote the set of relevant features provided by an explanation as S r ⊆ U where U = { 1 , . . . , d } is the set of all features, and use S r =
U \ S r , the complementary set of S r , to denote the set of irrelevant features.
We further use x S to denote the sub-vector of a vector x restricted to the features in S .
3. E VALUATION THROUGH R OBUSTNESS A NALYSIS
A common thread underlying evaluations of feature based explanations (Samek et al., 2016; Petsiuk et al., 2018), even ranging over axiomatic treatments (Sundararajan et al., 2017; Lundberg & Lee, 2017), is that the importance of a set of features corresponds to the change in prediction of the model when the features are removed from the original input.
Nevertheless, as we discussed in previous sections, operationalizing such a removal of features, for instance, by setting them to some reference value, introduces biases (see Appendix A and Section 5.2 for formal discussion and empirical results on the impact of reference values).
To ﬁnesse this, we leverage adversarial robustness, but to do so in this context, we rely on two key intuitive assumptions that motivate our method:
Assumption 1: When the values of the important features are anchored (ﬁxed), perturbations restricted to the complementary set of features has a weaker inﬂuence on the model prediction.
Assumption 2: When perturbations are restricted to the set of important features, ﬁxing the values of the rest of the features, even small perturbations could easily change the model prediction.
Based on these two assumptions, we propose a new framework leveraging the notion of adversarial robustness on feature subsets, as deﬁned below, to evaluate feature based explanations.
Deﬁnition 3.
Given a model f , an input x , and a set of features S ⊆ U where U is the set of all features, the minimum adversarial perturbation norm on x S , which we will also term Robustness-
S of x is deﬁned as:
[Equation]
where y = f ( x ) , S =
U \ S is the complementary set of features, and δ S
= 0 means that the perturbation is constrained to be zero along features in S .
Suppose that a feature based explanation partitions the input features of x into a relevant set S r , and an irrelevant set S r , Assumption 1 implies that the quality of the relevant set can be measured by (cid:15) ∗ x Sr – by keeping the relevant set unchanged, and measuring the adversarial robustness norm by perturbing only the irrelevant set.
Speciﬁcally, from Assumption 1, a larger coverage of pertinent features in set S r entails a higher robustness value (cid:15) ∗ x Sr .
On the other hand, from Assumption 2, a larger coverage of pertinent features in set S r would in turn entail a smaller robustness value (cid:15) ∗ x Sr , since only relevant features are perturbed.
More formally, we propose the following twin criteria for evaluating the quality of S r identiﬁed by any given feature based explanation.
Deﬁnition 3.2 Given an input x and a relevant feature set S r , we deﬁne Robustness-
S r
and
Robustness-
S r of the input x as the following:
[Equation]
Following our assumptions, a set S r that has larger coverage of relevant features would yield higher
Robustness- S r and lower Robustness- S r .
Evaluation for Feature Importance Explanations.
While Robustness- S r and Robustness-
S r are deﬁned on sets, general feature attribution based explanations could also easily ﬁt into the evaluation framework.
Given any feature attribution method that assigns importance score to each feature, we can sort the features in descending order of importance weights, and provide the top- K features as the relevant set S r .
The size of K (or | S r | ), can be speciﬁed by the users based on the application.
An alternative approach that we adopt in our experiments is to vary the size of set K and plot the corresponding values of Robustness- S r and Robustness- S r over different values of K .
With a graph where the X − axis is the size of K and the Y − axis is Robustness- S r or Robustness- S r , we are then able to plot an evaluation curve for an explanation and in turn compute its the area under curve (AUC) to summarize its performance.
A larger (smaller) AUC for Robustness- S r (Robustness- S r ) indicates a better feature attribution ranking.
Formally, given a curve represented by a set of points C
=
{ ( x 0 , y 0 ) , . .
, ( x n , y n ) } where x i − 1 < x i , we calculate the AUC of the curve by: {+AUC ( C ) = (cid:80) ni =
+}
( y i + y
i − 1 ) / 2 ∗ ( x i − x i − 1 ) .
Relation to Insertion and Deletion Criteria. We relate the proposed criteria to a set of commonly adopted evaluation metrics: the Insertion and Deletion criteria (Samek et al., 2016; Petsiuk et al., 2018; Sturmfels et al., 2020). The Insertion score measures the model’s function value when only the top-relevant features, given by an explanation, are presented in the input while the others are removed (usually by setting them to some reference value representing feature missingness). The
Deletion score, on the other hand, measures the model’s function value when the most relevant features are masked from the input.
As in our evaluation framework, we could plot the evaluation curves for Insertion (Deletion) score by progressively increasing the number of top-relevant features.
A larger (smaller) AUC under Insertion (Deletion) then indicates better explanation, as the identiﬁed relevant features could indeed greatly inﬂuence the model prediction.
We note that optimizing the proposed Robustness-
S r and Robustness-
S r could roughly be seen as optimizing a lower bound for the Insertion and Deletion score respectively.
This follows from the intuition: RobustnessS r considers features that when anchored, would make the prediction most robust to “adversarial perturbation”.
Since adversarial perturbation is the worst case of “any arbitrary perturbations”, the prediction will also be robust to different removal techniques (which essentially correspond to different perturbations) considered in the evaluation of Insertion score; The same applies to the connection between Robustness- S r and Deletion score.
We shall see in the experiment section that explanation optimizing our robustness measurements enjoys competitive performances on the
Insertion / Deletion criteria.
Untargeted v.s. Targeted Explanation.
Deﬁnition 3.1 corresponds to the untargeted adversarial robustness – a perturbation that changes the predicted class to any label other than y is considered as a successful attack.
Our formulation can also be extended to targeted adversarial robustness , where we replace Eqn. 1 by:
[Equation]
where t is the targeted class.
Using this deﬁnition, our approach will try to address the question “Why is this example classiﬁed as y instead of t ” by highlighting the important features that contrast between class y and t .
Further examples of the “targeted explanations” are in the experiment section.
Robustness Evaluation on Feature Subset.
It is known that computing the exact minimum distortion distance in modern neural networks is intractable (Katz et al., 2017), so many different methods have been developed to estimate the value.
Adversarial attacks, such as C&W (Carlini & Wagner, 2017) and projected gradient descent (PGD) attack (Madry et al., 2017), aim to ﬁnd a feasible solution of Eqn. 1, which leads to an upper bound of (cid:15) ∗ x S .
They are based on gradient based optimizers which are usually efﬁcient.
On the other hand, neural network veriﬁcation methods aim to provide a lower bound of (cid:15) ∗ x S to ensure that the model prediction will not change within certain perturbation range (Singh et al., 2018; Wong & Kolter, 2018;
Weng et al., 2018; Gehr et al., 2018; Zhang et al., 2018; Wang et al., 2018; Zhang et al., 2019).
The proposed framework can be combined with any method that aims to approximately compute Eqn. 1, including attack, veriﬁcation, and some other statistical estimations (see Appendix B for more discussions on estimating adversarial robustness for different types of model).
However, for simplicity we only choose to evaluate Eqn.by the state-of-the-art PGD attack (Madry et al., 2017), since the veriﬁcation methods are too slow and often lead to much looser estimation as reported in some recent studies (Salman et al., 2019).
Our additional constraint restricting perturbation to only be on a subset of features speciﬁes a set that is simple to project onto, where we set the corresponding coordinates to zero at each step of PGD.
E XTRACTING R ELEVANT F EATURES THROUGH R OBUSTNESS A NALYSIS
Our adversarial robustness based evaluations allow us to evaluate any given feature based explanation.
Here, we set out to design new explanations that explicitly optimize our evaluation measure.
We focus on feature set based explanations, where we aim to provide an important subset of features
S r .
Given our proposed evaluation measure, an optimal subset of feature S r would aim to maximize (minimize) Robustness-
S r (Robustness- S r ), under a cardinality constraint on the feature set, leading to the following set of optimization problems:
[Equation]
where K is a pre-deﬁned size constraint on the set S r , and g ( f, x , S ) computes the the minimum adversarial perturbation from Eqn. 1, with set-restricted perturbations.
It can be seen that this sets up an adversarial game for Eqn. 3 (or a co-operative game for Eqn. 4).
In the adversarial game, the goal of the feature set explainer is to come up with a set S r such that the minimal adversarial perturbation is as large as possible, while the adversarial attacker, given a set S r , aims to design adversarial perturbations that are as small as possible. Conversely in the co-operative game, the explainer and attacker cooperate to minimize the perturbation norm.
Directly solving these problems in Eqn. 3 and Eqn. 4 is thus challenging, which is exacerbated by the discrete input constraint that makes it intractable to ﬁnd the optimal solution.
We therefore propose a greedy algorithm in the next section to estimate the optimal explanation sets.
4. G REEDY A LGORITHM TO C OMPUTE O PTIMAL E XPLANATIONS
We ﬁrst consider a greedy algorithm where, after initializing S r to the empty set, we iteratively add to S r the most promising feature that optimizes the objective at each local step until S r reaches the size constraint.
We thus sequentially solve the following sub-problem at every step t :
[Equation]
where S tr is the relevant set at step t , and S 0 r = ∅ .
We repeat this subprocedure until the size of set
S tr reaches K .
A straightforward approach for solving Eqn.
5 is to exhaustively search over every
The higher the better for Robustness- S r ; the lower the better for Robustness- S r .
[Table]
single feature.
We term this method Greedy .
While the method eventually selects K features for the relevant set S r , it might lose the sequence in which the features were selected.
One approach to encode this order would be to output a feature explanation that assigns higher weights to those features selected earlier in the greedy iterations.
4. G REEDY BY S ET A GGREGATION S CORE
The main downside of using the greedy algorithm to optimize the objective function is that it ignores the interactions among features.
Two features that may seem irrelevant when evaluated separately might nonetheless be relevant when added simultaneously.
Therefore, in each greedy step, instead of considering how each individual feature will marginally contribute to the objective g ( · ) , we propose to choose features based on their expected marginal contribution when added to the union of S r and a random subset of unchosen features.
To measure such an aggregated contribution score, we draw from cooperative game theory literature (Dubey & Shapley, 1979; Hammer & Holzman, 1992) to reduce this to a linear regression problem.
Formally, let S tr and S tr be the ordered set of chosen and unchosen features at step t respectively, and P ( S tr ) be all possible subsets of S tr .
We measure the expected contribution that including each unchosen feature to the relevant set would have on the objective function by learning the following regression problem:
[Equation]
where b : P ( S tr ) → { 0 , 1 } | S tr | is a function that projects a set into its corresponding binary vector form: b ( S )
[ j ] = I ( S tr [ j ] ∈ S ) , and v ( · ) is set to be the objective function in Eqn. 3 or Eqn. 4: v ( S r ) = g ( f, x , S r ) for optimizing Eqn. 3; v ( S r ) =
g ( f, x , S r ) for optimizing Eqn. 4.
We note that w t corresponds to the well-known Banzhaf value (Banzhaf III, 1964) when S tr = ∅ , which can be interpreted as the importance of each player by taking coalitions into account (Dubey & Shapley, 1979).
Hammer & Holzman (1992) show that the Banzhaf value is equivalent to the optimal solution of linear regression with pseudo-Boolean functions as targets, which corresponds to Eqn.
6 with S tr = ∅ .
At each step t , we can thus treat the linear regression coefﬁcients w t in Eqn. 6 as each corresponding feature’s expected marginal contribution when added to the union of S r and a random subset of unchosen features.
We thus consider the following set-aggregated variant of our greedy algorithm in the previous section, which we term Greedy-AS .
In each greedy step t , we choose features that are expected to contribute most to the objective function, i.e. features with highest (for Eqn. 3) or lowest (for Eqn. 4) aggregation score (Banzhaf value), rather than simply the highest marginal contribution to the objective function as in vanilla greedy.
This allows us to additionally consider the interactions among the unchosen features when compared to vanilla greedy.
The chosen features each step are then added to S tr and removed from S tr .
When S tr is not ∅ , the solution of Eqn. 6 can still be seen as the Banzhaf value where the players are the unchosen features in S tr , and the value function computes the objective when a subset of players is added into the current set of chosen features S tr .
We solve the linear regression problem in Eqn. 6 by sub-sampling to lower the computational cost, and we validate the effectiveness of Greedy and Greedy-AS in the experiment section.
E XPERIMENTS
In this section, we ﬁrst evaluate different model interpretability methods on the proposed criteria.
We justify the effectiveness of the proposed Greedy-AS.
We then move onto further validating the [Table] beneﬁts of the explanations extracted by Greedy-AS through comparisons to various existing methods both quantitatively and qualitatively.
Finally, we demonstrate the ﬂexibility of our method with the ability to provide targeted explanations as mentioned in Section 3.2.
We perform the experiments on two image datasets, MNIST LeCun et al.
(2010) and ImageNet (Deng et al., 2009), as well as a text classiﬁcation dataset, Yahoo!
Answers (Zhang et al., 2015).
On MNIST, we train a convolutional neural network (CNN) with 99% testing accuracy.
On ImageNet, we deploy a pre-trained ResNet model obtained from the Pytorch library.
On Yahoo!
Answers, we train a BiLSTM sentence classiﬁer which attains testing accuracy of 71%.
Setup.
In the experiments, we consider p = 2 for (cid:107) · (cid:107) p in Eqn.
1 and Eqn.
We note that Eqn.is deﬁned for a single data example.
Given n multiple examples { x i } ni =1 with their corresponding relevant sets provided by some explanation { S i } ni =1 , we compute the overall Robustness- S r (- S r ) by taking the average: for instance, 1 n (cid:80) ni
=1 g ( f, x i , S i ) for Robustness- S r .
We then plot the evaluation curve as discussed in Section 3.2 and report the AUC for different explanation methods.
For all quantitative results, we report the average over 100 random examples.
For the baseline methods, we include vanilla gradient (Grad) (Shrikumar et al., 2017), integrated gradient (IG) (Sundararajan et al., 2017), and expected gradient (EG) (Erion et al., 2019; Sturmfels et al., 2020) from gradient-based approaches; leave-one-out (LOO) (Zeiler & Fergus, 2014; Li et al., 2016), SHAP (Lundberg & Lee, 2017) and black-box meaningful perturbation (BBMP) (only for image examples) (Fong & Vedaldi, 2017) from perturbation-based approaches (Ancona et al., 2018); counterfactual explanation (CFX) proposed by Wachter et al.
; Anchor (Ribeiro et al., 2018) for text examples; and a Random baseline that ranks feature importance randomly. Following common setup (Sundararajan et al., 2017; Ancona et al., 2018), we use zero as the reference value for all explanations that require baseline.
We leave more implementation detail in Appendix C due to space limitation.
5. R OBUSTNESS A NALYSIS ON M ODEL I NTERPRETABILITY M ETHODS
Here we compare Greedy-AS and various existing explanation methods under the proposed evaluation criteria Robustness-
S r and Robustness- S r .
We list the results in Table 1, with the detailed plots in
Appendix D. We as well compare Greedy-AS with its ablated variants in Appendix E.
Comparisons between Different Explanations.
Furthermore from Table 1, we observe that the proposed Greedy-AS consistently outperforms other explanation methods on both criteria (see
Appendix G for the statistical signiﬁcance).
On one hand, this suggests that the proposed algorithm indeed successfully optimizes towards the criteria; on the other hand, this might indicate the proposed criteria do capture different characteristics of explanations which most of the current explanations do not possess.
Another somewhat interesting ﬁnding from the table is that while Grad has generally been viewed as a baseline method, it nonetheless performs competitively on the proposed criteria.
We conjecture the phenomenon results from the fact that Grad does not assume any reference value as opposed to other baselines such as LOO which sets the reference value as zero to mask out the inputs.
Indeed, it might not be surprising that Greedy-AS achieves the best performances on the proposed criteria since it is explicitly designed for so.
To more objectively evaluate the usefulness of the proposed explanation, we demonstrate different advantages of our method by comparing Greedy-AS to other explanations quantitatively on existing commonly adopted measurements, and qualitatively through visualization in the following subsections.
5. E VALUATING G REEDY -AS
The Insertion and Deletion Metric.
To further justify the proposed explanation not only performs well on the very metric it optimizes, we evaluate our method on the suite of quantitative measurements mentioned above: the Insertion and Deletion criteria (Samek et al., 2016; Petsiuk et al., 2018;
Sturmfels et al., 2020).
Recall that evaluations on Insertion and Deletion score require specifying a reference value to represent feature missingness.
Here, we ﬁrst focus on the results when the reference values are randomly sampled from an uniform distribution, i.e., U (0 , 1) for image inputs and random word vector for text inputs, and we shall discuss the impact on varying such reference value shortly.
We plot the evaluation curves (in Appendix F) and report corresponding AUCs in Table 2.
On these additional two criteria, we observe that Greedy-AS performs favorably against other explanations (see Appendix G for the statistical signiﬁcance).
The results further validate the beneﬁts of the proposed criteria where optimizing Robustness- S r (- S r ) has tight connection to optimizing the Insertion (Deletion) score.
We note that on ImageNet, SHAP obtains a better performance under the Deletion criterion.
We however suspect such performance comes from adversarial artifacts (features vulnerable to perturbations while not being semantically meaningful) since SHAP seems rather noisy on ImageNet (as shown in Figure 3), and the Deletion criterion has been observed to favor such artifacts in previous work (Dabkowski & Gal, 2017; Chang et al., 2019). We note that although Greedy-AS exploits regions that are most susceptible to adversarial attacks, such regions may still be meaningful as shown in our visualization result.
Impact and Potential Bias of Reference Value. From Table 2, one might wonder why explanations like IG and SHAP would suffer relatively low performance although some of these methods (e.g., SHAP) are intentionally designed for optimizing Insertion- and Deletion-like measurements. We anticipate that such inferior performances are due to the mismatch between the intrinsic reference values used in the explanations and the ones used in the evaluation (recall that we set the intrinsic reference value to zero for all explanation methods, but utilize random value for Insertion and Deletion). To validate the hypothesis, we evaluate all explanations on Insertion and Deletion criteria with different reference values (0, 0.25, 0.5, 0.75, 1), and list the results in Appendix H. We validate that zero-baseline SHAP and IG perform much stronger when the reference value used in Insertion /
Deletion is closer to zero (matching the intrinsically-used baseline) and perform signiﬁcantly worse when the reference value is set to 0.75, 1, or U (0 , 1) .
On the other hand, we observe EG that does not rely on a single reference point (but instead averaging over multiple baselines) performs much more stably across different reference values. Finally, we see that Greedy-AS performs stably among the top across different reference values, which could be the merits of not assuming any baselines (but instead consider the worst-case perturbation). These empirical results reveal potential risk of evaluations (and indeed explanations) that could largely be affected by the change of baseline values. Sanity Check and Sensitivity Analysis.
In Appendix I, we conduct experiments to verify that Greedy-AS could indeed pass the sanity check proposed by Adebayo et al.
We as well conduct sensitivity analysis (Yeh et al., 2019) on Greedy-AS in Appendix J.
5. Q UALITATIVE R ESULTS
Image Classiﬁcation.
To complement the quantitative measurements, we show several visualization results on MNIST and ImageNet in Figure 2 and Figure 3.
More examples could be found in
Appendix K and L. On MNIST, we observe that existing explanations tend to highlight mainly on the white pixels in the digits; among which SHAP and LOO show less noisy explanations comparing to Grad and IG.
On the other hand, the proposed Greedy-AS focuses on both the “crucial positive” (important white pixels) as well as the “pertinent negative” (important black regions) that together support the prediction.
For example, in the ﬁrst row, a 7 might have been predicted as a 4 or 0 if the pixels highlighted by Greedy-AS are set to white.
Similarly, a 1 may be turned to a 4 or a 7 given additional white pixels to its left, and a 9 may become a 7 if deleted the lower circular part of its head.
From the results, we see that Greedy-AS focuses on “the region where perturbation on its current value will lead to easier prediction change” , which includes both the crucial positive and pertinent negative pixels.
Such capability of Greedy-AS is also validated by its superior performance on the proposed robustness criteria, on which methods like LOO that highlights only the white strokes of digits show relatively low performance.
The capability of capturing pertinent negative features has also been observed in explanations proposed in some recent work Dhurandhar et al.
; Bach et al.
; Oramas et al.
and we further provide more detailed discussions and comparisons to these methods in Appendix M. From the visualized ImageNet examples shown in Figure 3, we observe that our method provides more compact explanations that focus mainly on the actual objects being classiﬁed.
For instance, in the ﬁrst image, our method focuses more on the face of the Maltese while others tend to have noisier results; in the last image, our method focuses on one of the Japanese
Spaniel whereas others highlight both the dogs and some noisy regions. most other methods Greedy
Text Classiﬁcation.
Here we demonstrate how our explanation method could be applied to text classiﬁcation models.
We showcase an example in Figure 4 (see Appendix O for more examples).
We see that the top-5 keywords highlighted by Greedy-AS are all relevant to the label “sport”, and
Greedy-AS is less likely to select stop words as compared to other methods.
Additionally, we conduct an user study where we observe that Greedy-AS generates explanation that matches user intuition the most and we report detailed setup of the user study in Appendix P.
Targeted Explanation Analysis.
In section 3.2, we discussed about the possibility of using targeted adversarial perturbation to answer the question of “why the input is predicted as A but not B” .
In
Figure 5, for each input digit, we provide targeted explanation towards two different target classes.
Interestingly, as the target class changes, the generated explanation varies in an interpretatble way.
For example, in the ﬁrst image, we explain why the input digit 7 is not classiﬁed as a 9 (middle column) or a 2 (rightmost column).
The resulting explanation against 9 highlights the upper-left part of the 7.
Semantically, this region is indeed pertinent to the classiﬁcation between 7 and 9, since turning on the highlighted pixel values in the region (currently black in the original image) will then make the 7 resemble a 9.
However, the targeted explanation against 2 highlights a very different but also meaningful region, which is the lower-right part of the 7; since adding a horizontal stroke on the area would turn a 7 into a 2.
C ONCLUSION
In this paper, we establish the link between a set of features to a prediction with a new evaluation criteria, robustness analysis, which measures the minimum tolerance of adversarial perturbation.
Furthermore, we develop a new explanation method to ﬁnd important set of features to optimize this new criterion.
Experimental results demonstrate that the proposed new explanations are indeed capturing signiﬁcant feature sets across multiple domains. A CKNOWLEDGEMENTS
This work is partially supported by NSF IIS-1901527, IIS-2008173 and IIS-2048280. C.Y. and P.R. acknowledge the support of DARPA via FA87501720152, HR A B IAS FOR R EFERENCE -V ALUE M ETHODS In Appendix A.1, we show that many explanations are biased to reference value (IG, SHAP, LRP,
DeepLift).
For example, if the reference value is a zero vector (which means black pixels in image), then any black pixel will get a zero attribution value no matter if the object of interest is actually back.
We note that the Expected Gradient (Erion et al., 2019) is not biased to reference value by this theoretic deﬁnition since the baseline is actually a distribution.
However, for feature values that are close to the distribution of , the attribution score will be lower (but not 0 as our theoretic deﬁnition), when feature values are far from the distribution of , the the attribution score will be larger, which still has some biased involve.
We leave further investigation of the problem to future work to use more advanced analysis to quantify such a bias for explanations when the baseline follows a distribution.
In Appendix A.2, we added theoretical analysis that when a feature is equal to the replaced reference value, no matter how important it actually is, it will not contribute to the Deletion score nor the
Insertion score.
However, the lower the deletion score is better, and the higher the insertion score is better, and choosing an important feature that corresponds to the reference value will inevitably not improve the Deletion score and Insertion score.
The reference values such as blurring out or adding noise may still make the original features unchanged (such as when the main part of the image is already blurred, blurring the image will not change the image, and thus the main part of the image is biased to blurred baseline).
A. B IAS FOR R EFERENCE -V ALUE B ASED E XPLANATIONS
Theorem A.1 For any reference-value based explanation with the form φ ( f, x, x (cid:48) )
=
( x − x (cid:48) ) ⊗
K ( f, x, x (cid:48) ) for any meta-function K, we know that if x i = x
(cid:48)
i , φ i ( f, x ) = 0 .
That is, when ever a feature value coincides with the baseline value x (cid:48) , it will get 0 attribution even if it is the main feature contributing to the prediction.
We call such explanations biased to the reference value x (cid:48) .
Corollary A.1 IG, LRP, DeepLift are biased to the reference value x (cid:48) .
Following the deﬁnition in (Ancona et al., 2018), let ∂ g x c
∂x
i
=
(cid:80) p ∈ P ic (cid:81) w p (cid:81) g ( z ) p where P ic is the set of all paths that connect an unit i to unit c in a deep neural network, w p are the weights existing in path p , z be the value of an unit before nonlinear activation, and g () be any generic function.
Note that if g () = f (cid:48) () , the deﬁnition corresponds to the standard partial derivative of unit c to unit i .
φ IGi ( f, x ) =
( x i − x (cid:48) i ) · K ( f, x, x (cid:48) ) , where K ( f, x, x (cid:48) )
=
(cid:90)α = ∂f ( x (cid:48) + α ( x − x (cid:48) ))
∂x i dα (7) φ LRPi ( f, x ) =
( x i − x (cid:48) i ) · K ( f, x, x (cid:48) ) , where x (cid:48)
=
and K ( f, x, x (cid:48) )
= ∂ g f ( x ) ∂x i with g = f
( z ) z
(8) φ DeepLifti ( f, x )
=
( x i − x (cid:48) i ) · K ( f, x, x (cid:48) ) , where K ( f, x, x (cid:48) )
=
∂ g f ( x ) ∂x i with g = f ( z ) − f
( z (cid:48) )
z − z (cid:48) (9)
For example, if the reference value x (cid:48) is a zero vector (which means black pixels in image), then any black pixel will get a zero attribution value no matter if the object of interest is actually back.
Theorem A.
For any average reference-value based explanation with the form φ ( f, x, x (cid:48) )
=
E x (cid:48) ∼ p b
[( x − x (cid:48) ) ⊗ K ( f, x )] for some meta function K , if x i
=
E x (cid:48) ∼ p b
[ x (cid:48)
i ] , φ i ( f, x ) = 0 .
That is, they are biased to the reference value
E x (cid:48) ∼ p b [ x (cid:48) ] .
Corollary A.2 Averaging LRP over multiple baselines are bias to the reference value
E x (cid:48) ∼ p b [ x (cid:48) ] .
[Equation]
(10) where K ( f, x ) =
∂ g f ( x ) ∂x i with g = f ( z )
z
Theorem A.
The baseline Shapley value (Sundararajan & Najmi, 2019) is biased to reference value x (cid:48) .
[Equation]
where v ( s ) =
f ( x S ; x (cid:48)
N \ S )
and N is the set of all features.
These above analysis shows that more explanations are also biased to reference value.
We note that by Expected Gradient (Erion et al., 2019) is not biased to reference value by this theoretic deﬁnition since the baseline x (cid:48) is actually a distribution.
However, for feature values x i that are close to the distribution of x (cid:48)
i , the attribution score will be lower (but not 0 as our theoretic deﬁnition), when feature values x i are far from the distribution of x (cid:48)
i , the the attribution score will be larger, which still shows some level of bias.
We leave this to future work to use more advanced analysis techniques to quantify such a biased for explanations when the baseline follows a distribution.
A. B IAS FOR R EFERENCE - VALUE B ASED E VALUATIONS
Deﬁnition A.1 (Informal) Given a data point x , a reference point x (cid:48) , the deletion score of a model f given a set of important features S (and denote the complement of S as ¯ S ), which we abbreviate as
DEL ( x, x (cid:48) , S ) , is deﬁned as f ( x (cid:48) S ; x ¯ S ) .
Here, x (cid:48) can be a ﬁxed value or a random value, or even a blurred component (by a slight abuse of notations).
Similarly, we deﬁne the insertion score for a data point x , a reference point x (cid:48) , important set of features S , and a model f as INSR ( x, x (cid:48) , S ) =
( x S ; x (cid:48) ¯ S ) .
Note that the lower the DEL score is better, and the higher the INSR score is better.
Theorem A.
If x i
= x (cid:48)
i , then DEL ( x, x (cid:48) , S ) =
DEL ( x, x (cid:48) , S ∪ i ) for i (cid:54)⊆ S .
Similarly, if x i = x (cid:48)
i ,
INSR ( x, x (cid:48) , S ) =
INSR ( x, x (cid:48) , S ∪ i ) for i (cid:54)⊆ S
The implication of Thm.
A.4 is that if a feature happens to correspond to the reference value, no matter how important it actually is, it will not contribute to the DEL score nor the INSR score.
However, the lower the DEL score is better, and the higher the INSR score is better, and choosing an important feature that corresponds to the reference value will inevitably not improve the DEL score and INSR score.
Thus, DEL score and INSR score will deem feature with the same value as the reference value as non-important even if the features are actually crucial.
B M ORE ON E STIMATING A DVERSARIAL R OBUSTNESS
Our work relies on measuring the adversarial perturbation norm on a subset of features, which can be seen as a constrained adversarial robustness problem.
Adversarial robustness has been extensively studied in the past few years.
The adversarial robustness of a machine learning model on a given sample can be deﬁned as the shortest distance from the sample to the decision boundary, which corresponds to our deﬁnition in Eqn. 1.
Algorithms have been proposed for ﬁnding adversarial examples (feasible solutions of Eqn. 1), including (Goodfellow et al., 2014; Carlini & Wagner, 2017;
Madry et al., 2017).
However, those algorithms only work for neural networks, while for other models such as tree based models or nearest neighbor classiﬁers, adversarial examples can be found by decision based attacks (Brendel et al., 2017; Cheng et al., 2018; Chen et al., 2019).
Therefore the proposed framework can also be used in other decision based classiﬁers.
On the other hand, several works aim to solve the neural network veriﬁcation problem, which is equivalent to ﬁnding a lower bound of Eqn.
Examples include (Singh et al., 2018; Wong & Kolter, 2018; Zhang et al., 2018).
In principal, our work can also apply these veriﬁcation methods for getting an approximate solution of
Eqn. 1, but in practice they are very slow to run and often gives loose lower bounds on regular trained networks.
C I MPLEMENTATION D ETAIL
Underlying Models Used.
In the experiments conducted on MNIST dataset, we train a convolutional neural network (CNN) with 99% testing accuracy.
The training and testing split used in the experiments are the default split as provided by the original dataset.
For each instance, the inputs are loaded into the range of [0 , 1] .
On ImageNet dataset, we deploy a pre-trained ResNet model obtained from the Pytorch library.
For each image, the inputs are ﬁrst loaded into the range of [0 , 1] and normalized using mean =
456 , 0 . 406] and standard deviation = [0 .
On Yahoo!
Answers dataset, we train a BiLSTM sentence classiﬁer which attains testing accuracy of 71%.
For the model conﬁguration and training hyperparameters, we strictly follow the ofﬁcial GLUE repository 3 .
Speciﬁcally, we ﬁrst tokenize the dataset by word-level tokenizer; and then we use the pretrained GloVe word embedding to initialize the LSTM embedding layer where the hidden size is 300.
Method Implementation Details and Hyperparameters.
For the proposed Greedy and GreedyAS, at each greedy iteration, we include the top-5% features with highest scores into the relevant set to further speed up the selection process.
As discussed in Section 3.2, we use PGD attack with binary search to approximately compute the robustness value in Eqn. 1.
In the experiments, we set the PGD attack step size to be 1.0 and number of steps to be 100.
The hyperparameters are chosen such that the PGD attack could most efﬁciently provide the tightest upper bound on the true robustness value.
As mentioned in Section 4.2, we solve Eqn. 6 by subsampling from all possible subsets of S tr .
Speciﬁcally, we compute the coefﬁcients w with respect to 5000 sampled subsets when learning the regression.
For CFX, since the original objective in (Wachter et al., 2017) has no control over the number of features that could differ from the input, we add L 0 -norm constraint to enforce sparsity in the difference and enable selection of top- K most relevant features.
Evaluation Curves.
To plot the evaluation curves of a given explanation, we measure the explanation’s Robustness-
S r and Robustness- S r at different sizes of relevant set | S r | .
Speciﬁcally, we measure the explanation’s performance when | S r | equals to 5%, 10%, . . .
, 45% of the total number of features.
Similarly for the Insertion and Deletion criteria, we progressively evaluate the explanation’s performance when 5%, 10%, . . .
, 45% of the features are inserted (or removed) from the inputs.
After the evaluation curves are plotted, we then calculate the area under curves to summarize the explanation’s overall performance under each criterion.
Hardware and Code.
All the experiments were performed on Intel(R) Xeon(R) CPU E5-2630 v @
2.20GHz and NVIDIA GeForce GTX 1080 Ti GPU.
We release our source code along with an example Jupyter notebook for more information.
D E VALUATION C URVES UNDER R OBUSTNESS - S r AND R OBUSTNESS - S r
We show the evaluation curves of different methods under the proposed robustness criteria on MNIST, ImangeNet, and Yahoo!Answers in Figure 6, Figure and Figure 8 respectively.
[Table]
E A BLATION S TUDY ON G REEDY -AS
As discussed in Section 4.2, Greedy-AS could be seen as a combination of the original greedy procedure with the approximated contribution of each feature computed by a regression. Here, we examine the importance of both components by comparing the Greedy-AS method to two baselines, where one selects important features based only on the pure Greedy method (Section 4.1) and the other utilizes only a single step of regression without the iterative greedy procedure.
As the latter essentially corresponds to the Banzhaf value, we term this method as One-Step Banzhaf . As shown in Table 3, the pure Greedy method suffers degraded performances comparing to Greedy-AS under both criteria.
The inferior performance could be explained by the ignorance of feature correlations which ultimately results in the introduction of noise (see Figure 9).
In addition, we also see that Greedy-AS performs better than One-Step Banzhaf.
This could result from the fact that One-Step Banzhaf considers the feature interactions among all features with equal probability.
However, in our objective, we only care about those interactions with the most important features.
By iteratively selecting the features with highest Banzhaf value in Greedy-AS, we give more weight on the interactions among the most important features through iterations, and as a result lead to better performance.
F E VALUATION C URVES UNDER I NSERTION AND D ELETION
We show the evaluation curves of different methods under the Insertion and Deletion criteria on
MNIST and ImageNet in Figure 10, Figure and Figure 12 respectively.
G S TUDENT ’ S t - TEST ON THE P ERFORMANCE I MPROVEMENT OF G REEDY -AS OVER E XISTING M ETHODS
We conduct a set of Student’s t -tests to further verify that the outperformances of the proposed
Greedy-AS over existing methods are indeed statistically signiﬁcant.
We compare the AUCs of
Greedy-AS and other methods across different criteria.
In Table 4, we show the t -test results on the AUCs of Robustness- S r and Robustness- S r , as well as the results on the AUCs of Insertion and
Deletion criteria.
In the tables, win ( loss ) indicates that the performance improvement (drop) of
Greedy-AS to the baselines is statistically signiﬁcant; otherwise, we use draw to indicate insigniﬁcant difference between the performances.
From the results, we positively verify the statistical signiﬁcance of the performance improvements of our proposed method over existing explanations on various criteria. [Table]
We provide in Table 5 and Table 6 an empirical analysis on how different reference values would affect the evaluation results of different explanation methods. The corresponding evaluation curves are shown in Figure 14, Figure 15, Figure [Table]
and Figure 17. [Table] [Table] Reference value = 0.
Insertion 222.63 217.
286.40 Deletion 364.28 340.24 279.51 325.71 701.97 375.87 329.
Reference value =
Insertion 109.89 136.
189.96 Deletion 245.57 218.09 207.57 234.28 658.54 329.20 203.
We see performances of zero-baseline explanations, e.g. IG/SHAP/LOO, degrade as the reference value moves from zero.
[Figure]
[Figure]
[Figure]
[Figure]
I S ANITY C HECK
Recent literature has pointed out that an appropriate explanation should be related to the model being explained (Adebayo et al., 2018).
To ensure that our proposed explanation does indeed reﬂect the model behavior, we conduct the sanity check proposed by Adebayo et al.
(2018) to check if our explanations are adequately different when the model parameters are randomly re-initialized.
In the experiment, we randomly re-initialize the last fully-connected layer of the neural network model.
We then compute the rank correlation between explanation computed w.r.t.
the original model and that w.r.t. the randomized model. From Table 7, we observe that Greedy-AS has a much lower rank correlation comparing to Grad, IG, and LOO, suggesting that Greedy-AS is indeed sensitive to model parameter change and is able to pass the sanity check. [Table]
J E XPLANATION S ENSITIVITY A NALYSIS
In a recent study, Yeh et al.
(2019) proposed a sensitivity measurement to evaluate how large an explanation would change when small perturbation is applied to the original input.
The measurement could be seen as the lower bound of the local Lipschitz (Alvarez Melis & Jaakkola, 2018) of an explanation functional, and could be calculated efﬁciently by sampling.
Following Yeh et al.
(2019), we deﬁne a similar sensitivity measurement for set explanation as follows.
{+Deﬁnition J.
+}
Given a black-box model to be explained f , an input x , an explanation functional Φ( f, x ) that returns a set of relevant features, and a radius r deﬁning a local neighborhood around x , we deﬁne the sensitivity for an explanation as:
[Equation]
By Eqn.
12, we see that the sensitivity of an explanation equals 0 when the relevant set returned by the explanation remains the same in all local neighborhood of radius r centered around the original input x . On the other hand, the sensitivity equals 1 when there exists some y within the local neighborhood of x where the relevant set returned by Φ( f, y ) is totally disjoint from Φ( f, x ) . We shall however note that the sensitivity alone should not serve as the sole measurement for an explanation, since a meaningless constant explanation could always achieve 0 sensitivity. We show in Table 8 a sensitivity analysis of different explanations on MNIST dataset where we consider Φ( f, x ) to be the top-20% relevant features selected by an explanation method, and set r = 0 .
1 in the experiments.
From Table 8, we see that Expected Gradient has the least sensitivity since EG could be seen as a variant of the smoothing technique (Smilkov et al., 2017) (discussed in (Sturmfels et al., 2020)), which is shown to be an effective approach to reduce explanation sensitivity (Yeh et al., 2019).
While the proposed Greedy-AS has relatively large sensitivity, we believe such result might be highly correlated to the underlying sensitivity of the model f itself (since Greedy-AS aims to faithfully reﬂect the decision boundary of the model around an input).
Thus, one possible way to reduce the sensitivity of
Greedy-AS is to consider an adversarially more robust underlying model whose decision boundary is less sensitive to perturbations.
Another possible way is to consider smoothing the explanation provided by Greedy-AS.
We leave further improvement on the sensitivity of Greedy-AS as a future direction.
[Table]
K V ISUALIZATION ON MNIST
We provide more examples on visualized explanations on MNIST in Figure 18.
L V ISUALIZATION ON I MAGE N ET
We provide more examples on visualized explanations on ImageNet in Figure
[Figure]
M C OMPARISONS WITH O THER E XPLANATIONS C APTURING P ERTINENT N EGATIVE F EATURES
While the capability of capturing not only the crucial positive but also the pertinent negative features have also been observed in some recently proposed explanations.
Most current methods are not explicitly designed to handle the targeted explanation task which attempt to answer the question “what are the important features that lead to the prediction of class A but not class B”, and thus has different limitations.
For example, the ability of LRP Bach et al.
(2015) to capture pertinent negative features heavily depends on the input range.
In Samek et al.
(2016), the input images are normalized to have zero mean and a standard deviation of one where the black background will have non-zero value.
In such case, LRP would have non-zero attributions on the black background pixels which allows the explanation to capture pertinent negative features.
However, as later shown in Dhurandhar et al.
(2018), if the input pixel intensity is normalized into the range between 0 and 1 where the background pixels have the values of 0, LRP failed to highlight pertinent negative pixels since the background would always have zero attribution.
This is due to the fact that LRP is equivalent to multiplication between Grad and input in a Rectiﬁed Linear Unit (ReLU) network as shown in Ancona et al.
In Oramas et al.
(2019), unlike our targeted explanation where we know exactly which targeted class the explanation is suggesting against (and by varying the targeted class we observe varying corresponding explanation given), their method by design does not convey such information.
The pertinent negative features highlighted by their method by construction is not directly related to a speciﬁc target class, and users would need to infer by themselves what target class the pertinent negative features are preventing against.
To further grasp the difference, we compare our explanation with theirs in Figure 20.
We borrow the results from Oramas et al.
for visualization of their method.
Qualitatively, we observe that our method gives the most intuitive explanations.
For example, in Figure 20a, the ﬁrst row shows different explanations that highlight different relevant regions supporting the prediction of a 2 instead of 0.
Among the explanations, our method is the only one that highlights the right tail part (green circled) of the digit 2 which also serves as crucial evidence of 2 against 0 in addition to the left vertical gap (which when presented would make 2 looks like a 0) that is roughly highlighted by all three methods.
Furthermore, as we change the target class to(as shown in the second row), the explanation provided by LRP does not seem to differ much.
This suggests that LRP is not sensitive to the target class change and might fail to provide useful insights towards targeted explanation.
On the contrary, we observe that our explanation has a drastic change between different target classes.
When explaining against the target 7, our explanation highlights the lower left green circled part which when turned off will make 2 becomes a 7.
These results might suggest our method is more capable of handling such targeted explanation task.
N Q UANTITATIVE R ESULTS ON T EXT D ATASET
In this section, we conduct a set of quantitative analysis on different explanations for text classiﬁcation model on Yahoo!Answers dataset.
In the experiments, we evaluate the top-5 keywords selected by different explanations with the proposed Robustness-
S r (- S r ) measurements and the existing
Insertion and Deletion scores with different baseline values.
As discussed in Section 5.3, we represent a length- n sentence by n embedding vectors where each embedding vector is itself d -dimensional.
To calculate the Robustness- S r of an input sentence where the relevant set contains a set of indices of the top- k keywords, we restrict the embedding vectors of the top- k words to be ﬁxed and only allow perturbations on the remaining ( n − k ) · d dimensions.
Similarly for Robustness- S r , we only allow perturbations on the d · k dimensions corresponding to the embedding vectors of the top- k keywords.
For Insertion and Deletion score, we follow the same to replace the embedding vectors of the masked words by some selected reference value
[Table] [Table]
[Figure]
[Figure]
O Q UALITATIVE R ESULTS ON T EXT D ATASET
[Figure]
P U SER S TUDY ON T EXT D ATASET
To further compare the top-5 relevant keywords provided by various explanations [Table] we conducted an user study involving 30 different text examples (sentences) and 30 different users with Computer
Science background by convenience sampling.
In the study, each user is presented with a set of sentences and their corresponding categories.
We then asked each user to highlight 5 to 7 keywords for each sentence that they consider to be most relevant to the ground truth label of the corresponding sentence.
After collecting the feedback, for each sentence, we have 10 sets of different labels provided by the users.
For each sentence, we then select the keywords that have the highest frequency of being marked among the users and treat those keywords as the ground truth explanation for the corresponding sentence.
We leverage such user labels to calculate the precision@k and mean Average Precision (mAP) for each explanation method.
We show the results in Table 10 4 .
We observe that Greedy has the highest overall precision score, while the top-1 keyword selected by Anchor and IG match user intuition the most.
If we believe that users can correctly pick the correct keywords that are responsible for the model prediction
[Table]
then the p@k and mAP measure how well the explanation captures the correct keywords that are responsible for the model prediction.
We believe the good result on user study provides extra evidence that our method identiﬁes features that are responsible for the actual prediction.
[Table]
Q R UNTIME A NALYSIS
We show below the average runtime (wall clock time) of different methods for computing explanation for a single image on MNIST and ImageNet.
For Greedy and Greedy-AS, we show the time needed to compute the top-20% relevant features.
[Table]