R OBUST M ODELS A RE M ORE I NTERPRETABLE B ECAUSE A TTRIBUTIONS L OOK N ORMAL
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent work has found that adversarially-robust deep networks used for image classiﬁcation are more interpretable: their feature attributions tend to be sharper, and are more concentrated on the objects associated with the image’s groundtruth class.
We show that smooth decision boundaries play an important role in this enhanced interpretability, as the model’s input gradients around data points will more closely align with boundaries’ normal vectors when they are smooth.
Thus, because robust models have smoother boundaries, the results of gradientbased attribution methods, like Integrated Gradients and DeepLift will capture more accurate information about nearby decision boundaries.
This understanding of robust interpretability leads to our second contribution: boundary attributions , which aggregate information about the normal vectors of local decision boundaries to explain a classiﬁcation outcome.
We show that by leveraging the key factors underpinning robust interpretability, boundary attributions produce sharper, more concentrated visual explanations—even on non-robust models.
I NTRODUCTION
Feature attribution methods are widely used to explain the predictions of neural networks (Binder et al., 2016;
Dhamdhere et al., 2019; Fong & Vedaldi, 2017; Leino et al., 2018; Montavon et al., 2015; Selvaraju et al., 2017; Shrikumar et al., 2017; Simonyan et al., 2013;
Smilkov et al., 2017;
Springenberg et al., 2014; Sundararajan et al., 2017).
By assigning an importance score to each input feature of the model, these techniques help to focus attention on parts of the data most responsible for the model’s observed behavior.
Recent work (Croce et al., 2019; Etmann et al., 2019) has observed that feature attributions in adversarially-robust image models, when visualized, tend to be more interpretable—the attributions correspond more clearly to the discriminative portions of the input.
One way to explain the observation relies on the fact that robust models do not make use of nonrobust features (Ilyas et al., 2019) whose statistical meaning can change with small, imperceptible changes in the source data.
Thus, by using only robust features to predict, these models naturally tend to line up with visibly-relevant portions of the image.
Etmann et al.
take a different approach, showing that the gradients of robust models’ outputs more closely align with their inputs, which explains why attributions on image models are more visually interpretable.
In this paper, we build on this geometric understanding of robust interpretability.
With both analytical (Sec. 3) and empirical (Sec. 5) results, we show that the gradient of the model with respect to its input, which is the basic building block of all gradient-based attribution methods, tends to be more closely aligned with the normal vector of a nearby decision boundary in robust models than in “normal” models.
Leveraging this understanding, we propose Boundary-based Saliency Map (BSM) and
Boundary-based Integrated Gradient (BIG), two variants of boundary attributions (Sec. 4), which base attributions on information about nearby decision boundaries (see an illustration in Fig. 1a).
While BSM provides theoretical guarantees in the closed-form, BIG generates both quantitatively and qualitatively better explanations.
We show that these methods satisfy several desireable formal properties, and that even on non-robust models, the resulting attributions are more focused (Fig. 1b) and less sensitive to the “baseline” parameters required by some attribution methods.
To summarize, our main contributions are as follows.
(1) We present an analysis that sheds light on the previously-observed phenomeon of robust interpretability showing that alignment between the normal vectors of decision boundaries and models’ gradients is a key ingredient (Proposition 1).
[Figure]
In particular, we derive a closed-form result for one-layer networks (Theorem 1) and empirically validate the take-away of our theorem generalizes to deeper networks.
(2) Motivated by our analysis, we introduce boundary attributions , which leverage the connection between boundary normal vectors and gradients to yield explanations for non-robust models that carry over many of the favorable properties that have been observed of explanations on robust models.
(3) We empirically demonstrate that one such type of boundary attribution, called Boundary-based Integrated Gradients (BIG), produces explanations that are more accurate than prior attribution methods (relative to ground-truth bounding box information), while mitigating the problem of baseline sensitivity that is known to impact applications of Integrated Gradients Sundararajan et al.
(Section 6).
B ACKGROUND
We begin by introducing our notations.
Throughout the paper we use italicized symbols x to denote scalar quantities and bold-face x to denote vectors.
We consider neural networks with ReLU as activations prior to the top layer, and a softmax activation at the top.
The predicted label for a given input x is given by F ( x ) =
arg max c f c ( x ) , x ∈ R d , where F ( x ) is the predicted label and f i ( x ) is the output on the class i .
As the softmax layer does not change the ranking of neurons in the top layer, we will assume that f i ( x ) denotes the pre-softmax score.
Unless otherwise noted, we use || x || to denote the (cid:96) 2 norm of x , and the (cid:96) 2 neighborhood centered at x with radius (cid:15) as B ( x , (cid:15) ) .
Explainability.
Feature attribution methods are widely-used to explain the predictions made by
DNNs, by assigning importance scores for the network’s output to each input feature.
Conventionally, scores with greater magnitude indicate that the corresponding feature was more relevant to the predicted outcome.
We denote feature attributions by z = g ( x , f ) , z , x ∈ R d .
When f is clear from the context, we simply write g ( x ) .
While there is an extensive and growing literature on attribution methods, our analysis will focus closely on the popular gradient-based methods, Saliency Map (Simonyan et al., 2013), Integrated Gradient (Sundararajan et al., 2017) and Smooth Gradient (Smilkov et al., 2017), shown in Defs 1-3.
Deﬁnition 1 (Saliency Map (SM))
The Saliency Map g S ( x ) is given by g S ( x ) :=
∂f ( x ) ∂ x .
Deﬁnition 2 (Integrated Gradient (IG))
Given a baseline input x b , the Integrated Gradient g IG ( x ; x b ) is given by g IG ( x ; x b ) :=
( x − x b ) (cid:82) 1
∂f (( x − x b ) t + x b ) ∂ x dt .
[Figure]
" #$ %
[Figure] [Figure]
Deﬁnition 3 (Smooth Gradient (SG))
Given a zero-centered Gaussian distribution N with a standard deviation σ , the Smooth Gradient g SG ( x ; σ ) is given by g SG ( x ; σ ) :
= E (cid:15) ∼N ( 0 ,σ 2 I )
∂f ( α + (cid:15) )
∂ x .
Besides, we will also include results from DeepLIFT (Shrikumar et al., 2017) and grad × input (element-wise multiplication between Saliency Map and the input) (Simonyan et al., 2013) in our empirical evaluation.
As we show in Section 3.2, Defs 1-3 satisfy axioms that relate to the local linearity of ReLU networks, and in the case of randomized smoothing (Cohen et al., 2019), their robustness to input perturbations.
We further discuss these methods relative to others in Sec. 7.
Robustness.
Two relevant concepts about adversarial robustness will be used in this paper: prediction robustness that the model’s output label remains unchanged within a particular (cid:96) p norm ball and attribution robustness that the feature attributions are similar within the same ball.
Recent work has identiﬁed the model’s Lipschitz continuity as a bridge between these two concepts (Wang et al., 2020c) and some loss functions in achieving prediction robustness also bring attribution robustness (Chalasani et al., 2020).
We refer to robustness as prediction robustness if not otherwise noted.
E XPLAINABILITY , D ECISION B OUNDARIES , AND R OBUSTNESS
In this section, we begin by discussing the role of decision boundaries in constructing explanations of model behavior via feature attributions.
We ﬁrst illustrate the key relationships in the simpler case of linear models, which contain exactly one boundary, and then generalize to piecewise-linear classiﬁers as they are embodied by deep ReLU networks.
We then show how local robustness causes attribution methods to align more closely with nearby decision boundaries, leading to explanations that better reﬂect these relationships.
3. A TTRIBUTIONS FOR LINEAR MODELS
Consider a binary classiﬁer C ( x ) = sign ( w (cid:62) x + b ) that predicts a label {− 1 , 1 } (ignoring “tie” cases where C ( x ) = 0 , which can be broken arbitrarily).
In its feature space, C ( x ) is a hyperplane
H that separates the input space into two open half-spaces S 1 and S 2 (see Fig. 2a).
Accordingly, the normal vector ˆ n of the decision boundary is the only vector that faithfully explains the model’s classiﬁcation while other vectors, while they may describe directions that lead to positive changes in the model’s output score, are not faithful in this sense (see v in Fig.
2a for an example).
In practice, to assign attributions for predictions made by C , SM, SG, and the integral part of IG (see Sec. 2) return a vector characterized by z = k 1 ˆ n + k 2 (Ancona et al., 2018),
where k 1 (cid:54) = 0 and k 2 ∈ R , regardless of the input x that is being explained.
In other words, these methods all measure the importance of features by characterizing the model’s decision boundary, and are equivalent up to the scale and position of ˆ n .
3. G ENERALIZING TO PIECEWISE - LINEAR BOUNDARIES
In the case of a piecewise-linear model, such as a ReLU network, the decision boundaries comprise a collection of hyperplane segments that partition the feature space, as in H 1 , H 2 and H 3 in the example shown in Figure 2b.
Because the boundary no longer has a single well-deﬁned normal, one intuitive way to extend the relationship between boundaries and attributions developed in the previous section is to capture the normal vector of the closest decision boundary to the input being explained.
However, as we show in this section, the methods that succeeded in the case of linear models (SM, SG, and the integral part of IG) may in fact fail to return such attributions in the more general case of piecewise-linear models, but local robustness often remedies this problem.
We begin by reviewing key elements of the geometry of ReLU networks (Jordan et al., 2019).
ReLU activation polytopes.
For a neuron u in a ReLU network f ( x ) , we say that its status is ON if its pre-activation u ( x ) ≥ 0 , otherwise it is OFF .
We can associate an activation pattern denoting the status of each neuron for any point x in the feature space, and a half-space A u to the activation constraint u ( x ) ≥ 0 .
Thus, for any point x the intersection of the half-spaces corresponding to its activation pattern deﬁnes a polytope P (see Fig. 2b), and within P the network is a linear function such that ∀ x ∈ P, f ( x ) = w (cid:62) P x + b P , where the parameters w p and b P can be computed by differentiation (Fromherz et al., 2021).
Each facet of P (dashed lines in Fig. 2b) corresponds to a boundary that “ﬂips” the status of its corresponding neuron.
Similar to activation constraints, decision boundaries are piecewise-linear because each decision boundary corresponds to a constraint
f i ( x ) ≥ f j ( x ) for two classes i, j (Fromherz et al., 2021; Jordan et al., 2019).
Gradients might fail.
Saliency maps, which we take to be simply the gradient of the model with respect to its input, can thus be seen as a way to project an input onto a decision boundary.
That is, a saliency map is a vector that is normal to a nearby decision boundary segment.
However, as others have noted, a saliency map is not always normal to any real boundary segment in the model’s geometry (see the left plot of Fig. 2c), because when the closest boundary segment is not within the activation polytope containing x , the saliency map will instead be normal to the linear extension of some other hyperplane segment (Fromherz et al., 2021).
In fact, iterative gradient descent typically outperforms the Fast Gradient Sign Method (Goodfellow et al., 2015) as an attack demonstrates that this is often the case.
When gradients succeed.
While saliency maps may not be the best approach in general for capturing information about nearby segments of the model’s decision boundary, there are cases in which it serves as a good approximation.
Recent work has proposed using the Lipschitz continuity of an attribution method to characterize the difference between the attributions of an input x and its neighbors within a (cid:96) p ball neighborhood (Def. 4) (Wang et al., 2020c).
This naturally leads to Proposition 1, which states that the difference between the saliency map at an input and the correct normal to the closest boundary segment is bounded by the distance to that segment.
Deﬁnition 4 (Attribution Robustness)
An attribution method g ( x ) is ( λ, δ ) -locally robust at the evaluated point x if ∀ x (cid:48) ∈ B ( x , δ ) , || g
( x (cid:48) )
− g ( x ) || ≤ λ || x (cid:48) − x || .
Proposition 1 Suppose that f has a ( λ, δ ) -robust saliency map g S at x , x (cid:48) is the closest point on the closest decision boundary segment to x and || x
(cid:48) − x || ≤ δ , and that n is the normal vector of that boundary segment.
Then || n − g S ( x )
[Equation]
Proposition 1 therefore provides the following insight: for networks that admit robust attributions (Chen et al., 2019; Wang et al., 2020c), the saliency map is a good approximation to the boundary vector.
As prior work has demonstrated the close correspondence between robust prediction and robust attributions (Wang et al., 2020c; Chalasani et al., 2020), this in turn suggests that explanations on robust models will more closely resemble boundary normals.
As training robust models can be expensive, and may not come with guarantees of robustness, post-processing techniques like randomized smoothing (Cohen et al., 2019), have been proposed as an alternative.
Dombrowski et al.
(2019) noted that models with softplus activations ( y = 1 /β log(1+exp ( β x )) )
approximate smoothing, and in fact give an exact correspondence for singlelayer networks.
Combining these insights, we arrive at Theorem 1, which suggests that the saliency map on a smoothed model approximates the closest boundary normal vector well; the similarity is inversely proportional to the standard deviation of the noise used to smooth the model.
Theorem 1 Let m ( x ) = ReLU ( W x ) be a one-layer network and when using randomized smoothing, we write m σ ( x ) .
Let g ( x ) be the SM for m σ ( x ) and suppose
∀ x (cid:48)(cid:48) ∈ B ( x , || x − x (cid:48) || ) , || g
( x (cid:48)(cid:48) )
|| ≥ c where x (cid:48) is the closest adversarial example, we have the following statement holds: || g ( x ) − g ( x (cid:48) )
|| ≤ λ where λ is monotonically decreasing w.r.t σ .
Theorem 1 suggests that when randomized smoothing is used, the normal vector of the closest decision boundary segment and the saliency map are similar, and this similarity increases with the smoothness of the model’s boundaries.
Because the saliency map for a smoothed model is equivalent to smooth gradient of its non-smooth counterpart (Wang et al., 2020c), the smooth gradient is a better choice whenever computing the exact boundary is too expensive in a standard model.
We provide empirical validation of this in Figure 11.
B OUNDARY -B ASED A TTRIBUTION
Without the properties introduced by robust learning or randomized smoothing, the local gradient, i.e. saliency map, may not be a good approximation of decision boundaries.
In this section, we build on the insights of our analysis to present a set of novel attribution methods that explicitly incorporate the normal vectors of nearby boundary segments.
Importantly, these attribution methods can be applied to models that are not necessarily robust, to derive explanations that capture many of the beneﬁcial properties of explanations for robust models.
Using the normal vector of the closest decision boundary to explain a classiﬁer naturally leads to
Deﬁnition 5, which deﬁnes attributions directly from the normal of the closest decision boundary.
Deﬁnition 5 (Boundary-based Saliency Map (BSM))
Given f and an input x , we deﬁne
Boundary-based Saliency Map B S ( x ) as follows: B S ( x ) def =
∂f c ( x (cid:48) ) /∂
x (cid:48) , where x (cid:48) is the closest adversarial example to x , i.e. c = F ( x ) (cid:54) =
F ( x (cid:48) ) and ∀ x m .
||
x m − x || <
|| x (cid:48) − x || → F ( x ) =
F ( x m ) .
Incorporating More Boundaries.
The main limitation of using Deﬁnition 5 as a local explanation is obvious: the closest decision boundary only captures one segment of the entire decision surface.
Even in a small network, there will be numerous boundary segments in the vicinity of a relevant point.
Taking inspiration from Integrated Gradients, Deﬁnition 6 proposes the Boundary-based Integrated Gradient (BIG) by aggregating the attributions along a line between the input and its closest boundary segment.
Deﬁnition 6 (Boundary-based Integrated Gradient(BIG))
Given f , Integrated Gradient g IG and an input x , we deﬁne Boundary-based Integrated Gradient B S ( x ) as follows: B IG ( x ) :=
g IG ( x ; x (cid:48) ) , where x is the nearest adversarial example to x , i.e. c = F ( x ) (cid:54) =
F ( x (cid:48) ) and ∀ x m .
||
x m − x || <
|| x (cid:48) − x || → F ( x ) =
F ( x m ) .
Geometric View of BIG.
BIG explores a linear path from the boundary point to the target point.
Because points on this path are likely to traverse different activation polytopes, the gradient of intermediate points used to compute g IG are normals of linear extensions of their local boundaries.
As the input gradient is identical within a polytope P i , the aggregate computed by BIG sums each gradient w i along the path and weights it by the length of the path segment intersecting with P i .
In other words, one may view IG as an exploration of the model’s global geometry that aggregates all boundaries from a ﬁxed reference point, whereas BIG explores the local geometry around x .
In the former case, the global exploration may reﬂect boundaries that are not particularly relevant to model’s observed behavior at a point, whereas the locality of BIG may aggregate boundaries that are more closely related (a visualization is shown in Fig. 1a).
Finding nearby boundaries.
Finding the exact closest boundary segment is identical to the problem of certifying local robustness (Fromherz et al., 2021; Jordan et al., 2019; Kolter & Wong, 2018;
Lee et al., 2020; Leino et al., 2021b; Tjeng et al., 2019; Weng et al., 2018), which is NP-hard for piecewise-linear models (Sinha et al., 2020).
To efﬁciently ﬁnd an approximation of the closest boundary segment, we leverage and ensemble techniques for generating adversarial examples, i.e.
PGD (Madry et al., 2018), AutoPGD (Croce & Hein, 2020) and CW (Carlini & Wagner, 2017), and use the closest one found given a time budget.
The details of our implementation are discussed in
Section 5, where we show that this yields good results in practice.
CIFAR standard (cid:96) 2 | 0 .
SM-BSM.
IG-AGI 28.
IG-BIG 31.
ImageNet standard (cid:96) 2 | 3 .(cid:96) ∞ |(cid:96) ∞ |
SM-BSM 8.
IG-AGI 13.
IG-BIG 17.
(a)
Corr.
Loc.
EG
PP
Con.
SM -BSM 0.
IG -AGI 0.
IG -BIG 0.
(b)
Model
Metrics
BIG
BSM
AGI
SM
GTI
SG
IG
DeepLIFT
Loc.
standard
EG 0.
[Table]
E VALUATION
In this section
[Table]
we ﬁrst validate that the attribution vectors are more aligned to normal vectors of nearby boundaries in robust models(Fig. 3a).
We secondly show that boundary-based attributions provide more “accurate” explanations – attributions highlight features that are actually relevant to the label – both visually (Fig. 4 and 5) and quantitatively (Table 1).
Finally, we show that in a standard model, whenever attributions more align with the boundary attributions, they are more “accurate”.
General Setup.
We conduct experiments over two data distributions, ImageNet (Russakovsky et al., 2015) and CIFAR-10 (Krizhevsky et al.).
For ImageNet, we choose 1500 correctly-classiﬁed images from ImageNette (Howard), a subset of ImageNet, with bounding box area less than 80% of the original source image.
For CIFAR-10, We use 5000 correctly-classiﬁed images.
All standard and robust deep classiﬁers are ResNet50.
All weights are pretrained and publicly available (Engstrom et al., 2019).
Implementation details of the boundary search (by ensembling the results of PGD, CW and AutoPGD) and the hyperparameters used in our experiments, are included in Appendix B.2.
5. R OBUSTNESS → B OUNDARY A LIGNMENT
In this subsection, we show that SM and IG better align with the normal vectors of the decision boundaries in robust models.
For SM, we use BSM as the normal vectors of the nearest decision boundaries and measure the alignment by the (cid:96) 2 distance between SM and BSM following Proposition 1.
For IG, we use BIG as the aggregated normal vectors of all nearby boundaries because
IG also incorporates more boundary vectors.
Recently, Pan et al.
(2021) also provides Adversarial
Gradient Integral (AGI) as an alternative way of incorporating the boundary normal vectors into IG.
We ﬁrst use both BIG and AGI to measure how well IG aligns with boundary normals and later compare them in Sec. 5.2, followed by a formal discussion in Sec. 7.
Aggregated results for standard models and robust models are shown in Fig. 3a.
It shows that adversarial training with bigger (cid:15) encourages a smaller difference between attributions and their boundary variants.
Particularly, using (cid:96) 2 norm and setting (cid:15) = 3 . 0 are most effective for ImageNet compared to (cid:96) ∞ norm bound.
One possible explanation is that the (cid:96) 2 space is special because training with (cid:96) ∞ bound may encourage the gradient to be more Lipschitz in (cid:96) 1 because of the duality between the Lipschitzness and the gradient norm, whereas (cid:96) 2 is its own dual.
5. B OUNDARY A TTRIBUTION → B ETTER L OCALIZATION
In this subsection, we show boundary attributions (BSM, BIG and AGI) better localize relevant features.
Besides SM, IG and SG, we also focus on other baseline methods including Grad ×
Input (GTI) (Simonyan et al., 2013) and DeepLIFT (rescale rule only) (Shrikumar et al., 2017) that are reported to be more faithful than other related methods (Adebayo et al., 2018; 2020).
In an image classiﬁcation task where ground-truth bounding boxes are given, we consider features within a bounding box as more relevant to the label assigned to the image.
Our evaluation is performed over ImageNet only because no bounding box is provided for CIFAR-10 data.
The metrics used for our evaluation are: 1) Localization (Loc.)
(Chattopadhyay et al., 2017) evaluates the intersection of areas with the bounding box and pixels with positive attributions; 2) Energy Game (EG) (Wang et al., 2020a) instead computes the portion of attribute scores within the bounding box.
While these two metrics are common in the literature, we propose the following additional metrics: 3) Positive Percentage (PP) evaluates the portion of positive attributions in the bounding box because a naive assumption is all features within bounding boxes are relevant to the label (we will revisit this assumption in Sec. 6); and 4) Concentration (Con.)
sums the absolute value of attribution scores over the distance between the “mass” center of attributions and each pixel within the bounding box.
Higher Loc. , EG , PP and Con. are better results.
We provide formal details for the above metrics in Appendix B.
We show the average scores for ResNet50 models in Table 1 where the corresponding boxplots can be found in Appendix B.4.
BIG is noticeably better than other methods on Loc.
EG, PP and
Con.
scores for both robust and standard models and matches the performance of SG on EG for a standard model.
Notice that BSM is not signiﬁcantly better than others in a standard model, which conﬁrms our motivation of BIG – that we need to incorporate more nearby boundaries because a single boundary may not be sufﬁcient to capture the relevant features.
We also measure the correlation between the alignment of SM and BSM with boundary normals and the localization abilities, respectively.
For SM, we use BSM to represent the normal vectors of the boundary.
For IG, we use AGI and BIG.
For each pair X - Y in { SM -BSM, IG -AGI, IG -BIG } , we measure the empirical correlation coefﬁcient between −|| X − Y ||
and the localization scores of X in a standard ResNet50 and the result is shown in Fig.
3b.
Our results suggest that when the attribution methods better align with their boundary variants, they can better localize the relevant features in terms of the Loc. and EG.
However, PP and Con. have weak and even negative correlations.
One possible explanation is that the high PP and Con.
of BIG and AGI compared to IG (as shown in
Table 1) may also come from the choice of the reference points.
Namely, compared to a zero vector, a reference point on the decision boundary may better ﬁlter out noisy features.
We end our evaluations by visually comparing the proposed method, BIG, against all other attribution methods for the standard ResNet50 in Fig.
4 and for the robust ResNet50 in Fig.
5, which demonstrates that BIG can easily and efﬁciently localize features that are relevant to the prediction.
More visualizaitons can be found in the Appendix E.
Summary.
Taken together, we close the loop and empirical show that standard attributions in robust models are visually more interpretable because they better capture the nearby decision boundaries.
Therefore, the ﬁnal take-away from our analytical and empirical results is if more resources are devoted to training robust models, effectively identical explanations can be obtained using much less costly standard gradient-based methods, i.e. IG.
The second example from Fig.
4 is not correctly classiﬁed
so we replace it with another image.
D ISCUSSION
Baseline Sensitivity.
It is natural to treat that BIG frees users from the baseline selection in explaining non-linear classiﬁers.
Empirical evidence has shown that IG is sensitive to the baseline inputs (Sturmfels et al., 2020).
We compare BIG with IG when using different baseline inputs, white or black images.
We show an example in Fig 6b.
For the ﬁrst two images, when using the baseline input as the opposite color of the dog, more pixels on dogs receive non-zero attribution scores.
Whereas backgrounds always receive more attribution scores when the baseline input has the same color as the dog.
This is because g IG ( x ) i ∝ ( x − x b ) i (see Def. 2) that greater differences in the input feature and the baseline feature can lead to high attribution scores.
The third example further questions the readers using different baselines in IG whether the network is using the white dog to predict Labrador retriever .
We demonstrate that conﬂicts in IG caused by the sensitivity to the baseline selection can be resolved by BIG.
BIG shows that black dog in the last row is more important for predicting Labrador retriever and this conclusion is further validated by our counterfactual experiment in Appendix D. Overall, the above discussion highlights that BIG is signiﬁcantly better than IG in reducing the non-necessary sensitivity in the baseline selection.
Limitations.
We identify two limitations of the work.
1) Bounding boxes are not perfect groundtruth knowledge for attributions.
In fact, we ﬁnd a lot of examples where the bounding boxes either fail to capture all relevant objects or are too big to capture relevant features only.
Fixing mislabeled bounding boxes still remain an open question and should beneﬁt more expandability research in general.
Our analysis only targets on attributions that are based on end-to-end gradient computations.
That is, we are not able to directly characterize the behavior of perturbation-based approaches, i.e.
Mask (Fong & Vedaldi, 2017), and activation-based approaches, i.e. GradCAM (Selvaraju et al., 2017) and Feature Visualization (Olah et al., 2017).
R ELATED W ORK
Ilyas et al.
(2019) shows an alternative way of explaining why robust models are more interpretable by showing robust models usually learn robust and relevant features, whereas our work serves as a geometrical explanation to the same empirical ﬁndings in using attributions to explain deep models.
Our analysis suggests we need to capture decision boundaries in order to better explain classiﬁers,
Properties black IG
AGI
BIG
Boundary-based (cid:55) (cid:51) (cid:51)
Boundary Search
N/A
PGD
Any
Geometry
Global
Local Local
Symmetry (cid:51) (cid:55) (cid:51)
Completeness (cid:51) (cid:51) (cid:51) (a)
[Figure]
[Figure]
whereas a similar line of work, AGI (Pan et al., 2021) that also involves computations of adversarial examples is motivated to ﬁnd a non-linear path that is linear in the representation space instead of the input space compared to IG.
Therefore, AGI uses PGD to ﬁnd the adversarial example and aggregates gradients on the non-linear path generated by the PGD search.
We notice that the trajectory of
PGD search is usually extremely non-linear, complex and does not guarantee to return closer adversarial examples without CW or AutoPGD (see comparisons between boundary search approaches in Table B.2).
We understand that ﬁnding the exact closest decision boundary is not feasible, but our empirical results suggest that the linear path (BIG) returns visually sharp and quantitative better results in localizing relevant features.
Besides, a non-linear path should cause AGI fail to meet the symmetry axiom
(Sundararajan et al., 2017) (see Appendix C for an example of the importance of symmetry for attributions).
We further summarize the commons and differences in Table 6a.
In the evaluation of the proposed methods, we choose metrics related to bounding box over other metrics because for classiﬁcation we are interested in whether the network associate relevant features with the label while other metrics (Adebayo et al., 2018; Ancona et al., 2017; Samek et al., 2016;
Wang et al., 2020b; Yeh et al., 2019), e.g. inﬁdelity (Yeh et al., 2019), mainly evaluates whether output scores are faithfully attributed to each feature.
Our idea of incorporating boundaries into explanations may generalize to other score attribution methods, e.g. Distributional Inﬂuence (Leino et al., 2018) and DeepLIFT (Shrikumar et al., 2017).
The idea of using boundaries in the explanation has also been explored by T-CAV (Kim et al., 2018), where a linear decision boundary is learned for the internal activations and associated with their proposed notion of concept .
When viewing our work as using nearby boundaries as a way of exploring the local geometry of the model’s output surface, a related line of work is NeighborhoodSHAP (Ghalebikesabi et al., 2021), a local version of SHAP (Lundberg & Lee, 2017).
When viewing our as a different use of adversarial examples, some other work focuses on counterfactual examples (semantically meaningful adversarial examples) on the data manifold (Chang et al., 2019;
Dhurandhar et al., 2018; Goyal et al., 2019).
C ONCLUSION
In summary, we rethink the target question an explanation should answer for a classiﬁcation task, the important features that the classiﬁer uses to place the input into a speciﬁc side of the decision boundary.
We ﬁnd the answer to our question relates to the normal vectors of decision boundaries in the neighborhood and propose BSM and BIG as boundary attribution approaches.
Empirical evaluations on STOA classiﬁers validate that our approaches provide more concentrated, sharper and more accurate explanations than existing approaches.
Our idea of leveraging boundaries to explain classiﬁers connects explanations with the adversarial robustness and help to encourage the community to improve model quality for explanation quality.
A T HEOREMS AND P ROOFS A. P ROOF OF P ROPOSITION
Proposition 1 Suppose that f has a ( λ, δ ) -robust saliency map g S at x , x (cid:48) is the closest point on the closest decision boundary segment to x and || x
(cid:48) − x || ≤ δ , and that n is the normal vector of that boundary segment.
Then || n − g S ( x ) || ≤ λ || x − x (cid:48)
|| .
To compute n can be efﬁciently computed by taking the derivatice of the model’s output w.r.t to the point that is on the decision boundary such that n = ∂f ( x (cid:48) )
∂ x (cid:48) and ∀ x m ∈ R d , F ( x m ) =
F ( x ) if || x m − x || ≤ || x (cid:48) − x || .
Because we assume || x − x (cid:48)
|| ≤ δ , and the model has ( λ, δ )
-robust
Saliency Map, then by Def.we have
[Equation]
A. P ROOF OF T HEOREM
Theorem 1 Let m ( x ) =
ReLU ( W x ) be a one-layer network and when using randomized smoothing, we write m σ ( x ) .
Let g ( x ) be the SM for m σ ( x ) and suppose
∀ x (cid:48)(cid:48) ∈ B ( x , || x − x (cid:48) || ) , || g
( x (cid:48)(cid:48) )
|| ≥ c where x (cid:48) is the closest adversarial example, we have the following statement holds: || g ( x ) − g ( x (cid:48) ) || ≤ λ where λ is monotonically decreasing w.r.t σ . Proof:
We begin our proof by ﬁrstly introducing Randomized Smoothing .
Deﬁnition 7 (Randomized Smoothing (Cohen et al., 2019))
Suppose F ( x ) =
arg max c f c ( x ) , the smoothed classiﬁer G ( x ) is deﬁned as
[Equation]
where (cid:15) ∼ N ( 0 , σ 2 I )
Now the rest of the proof of is three-fold: 1) ﬁrstly we will show that there exist a non-linear activation function Er ( x ) such that the output of the smoothed ReLU network m σ ( x ) is equivalent when replacing the ReLU activation with Er activation; 2) secondly derive the difference between the saliency map of the network with Er activation; and 3) lastly, we show that the difference between
SM and BSM of the network with Er activation is bounded, which is inversely proportional to the standard deviation used to create the smoothed ReLU network m σ ( x ) .
(1) Step I : Error activation (Er) function and randomized smoothing.
Randomized Smoothing creates a smoothed model that returns whichever the label that the base classiﬁer most likely to return under the perturbation generated by the Gaussian noise.
Now we take a look at the output of each class under the Gaussian noise. Consider y i is the output of the i -th class of the network ReLU ( W x ) , that is
[Equation]
To simplify the notation, we denote E (cid:15) ∼N ( 0 ,σ 2 I ) as E .
We expand Equation (2):
[Equation]
where we denote u = w (cid:62) i x and (cid:15) (cid:48)
= w (cid:62) i (cid:15) .
u is a scalar and (cid:15) (cid:48) follows a zero-centered univariate
Gaussian with the standard deviation s ∝ σ because the dot production between the constant weight vector w i and the random vector (cid:15) can be seen as a linear combination of each dimension of (cid:15) and the covariance between each dimension of (cid:15) is 0 for the Gaussian noise used for randomized smoothing in the literature (Cohen et al., 2019). By expanding the expectation symbol to its integral form, we obtain:
[Equation]
Let τ = u + (cid:15) (cid:48) and notice that ReLU ( τ ) = {+
+}
if τ < 0 , the equation above can be rewritten as: [Equation] where Erf is the error function deﬁned as Erf ( x ) = 2 √ π (cid:82) x 0 exp( − t 2 ) dt .
We therefore deﬁne an Er activation for an input u with the standard deviation s as
[Equation]
and we show that [Equation] That is, to analyze the gradient of the output for a smoothed model w.r.t the input, we can alternatively analyze the gradient of an equivalent Er network.
We plot three examples of the Er activations in Fig.
7 for the readers to see what does the function look like. 2) Step II : the Saliency Map for an Er network.
By the deﬁnition of Saliency map (Def. 1) and the chain rule, we have:
[Equation]
The transition between Equation (11) to (12) is based on the fact that the derivative of Erf ( x ) is√ π exp( − x 2 ) . ) Step III : the difference between SM and BSM for an Er network Let x (cid:48) be the closest point on the decision boundary for the smoothed classiﬁer m σ and || x − x (cid:48) ||
= r (for the closed-form expression of r , see Cohen et al. Based on the deﬁnition of BSM, we have
[Equation]
The difference between SM and BSM therefore is computed as (a) s = 0 .- 1. (b) s = 0 .- 1. (c) s = 1 .
[Equation]
[Equation] [Equation]
[Equation]
We notice that the u (cid:48) is bounded because u (cid:48)
= w (cid:62) i x (cid:48) ≤ || w i || · || x (cid:48)
|| ≤ || w i || · ( || w i
||
+ r ) and similarly for u such that u = w (cid:62) i x ≤ || w
i || · ( || x || + r ) .
Because Erf function is increasing w.r.t the input and s > 0 , we arrive at the following inequality:
[Equation]
where
[Equation]
We take the absolute symbol out because the output of an Erf is positive when its input is positive. Now, given that || w i || , r and || x || are constants when for a given input x , the upper-bound
Erf ( || w i ||· ( || x || + r ) √ 2 s ) · || w i || is monotonically increasing as s decreases.
From the Step I, we know that s ∝ σ , therefore we prove there exist an upper-bound λ of the difference between the SM and
BSM for a smoothed classiﬁer and λ is monotonically decreasing w.r.t the standard deviation of the
Gaussian noise.
B E XPERIMENT D ETAILS AND A DDITIONAL R ESULTS B. M ETRICS WITH B OUNDING B OXES
We will use the following extra notations in this section.
Let X , Z
and U be a set of indices of all pixels, a set of indices of pixels with positive attributions, and a set of indices of pixels inside the bounding box for a target attribution map g ( x ) .
We denote the cardinality of a set S as | S | .
Localization (Loc.)
(Chattopadhyay et al., 2017) evaluates the intersection of areas with the bounding box and pixels with positive attributions.
Deﬁnition 8 (Localization) For a given attribution map g ( x ) , the localization score (Loc.) is deﬁned as [Equation]
[Equation]
BIG
BSM
AGI
SM
GI SG IG DeepLIFT 0.
[Figure]
Localization (Loc.)
BIG
BSM
AGI
SM
GI
SG
IG
DeepLIFT 0.
Energy Game (GE)
BIG
BSM
AGI
SM
GI
SG
IG
DeepLIFT 0.
[Figure]
[Figure]
BIG
BSM
AGI
SM
[Figure]
SG
IG
[Figure]
(Wang et al., 2020a) instead evaluates computes the portion of attribute scores within the bounding box.
Deﬁnition 9 (Energy Game)
For a given attribution map g ( x ) , the energy game EG is deﬁned as
[Equation]
Positive Percentage (PP) evaluates the sum of positive attribute scores over the total (absolute value of) attribute scores within the bounding box.
Deﬁnition 10 (Positive Percentage) Let V be a set of indices pf all pixels with negative attribution scores, for a given attribution map g ( x ) , the positive percentage PP is deﬁned as
[Equation]
Concentration (Con.) evaluates the sum of weighted distances by the “mass” between the “mass” center of attributions and each pixel within the bounding box.
Notice that the computation of c x and c y can be computed with scipy.ndimage.center of mass .
This deﬁnition encourages that pixels with high absolute value of attribution scores to be closer to the mass center.
Deﬁnition 11 (Concentration) For a given attribution map g ( x ) , the concentration Con.
is deﬁned as follws
[Equation]
where ˆ g is the normalized attribution map so that ˆ g
i =
g i [Equation] (cid:80)
[Equation] ∈ U | g i | .
i x , i y are the coordinates of the pixel and
[Equation]
Besides metrics related to bounding boxes, there are other metrics in the literature used to evaluate attribution methods (Adebayo et al., 2018; Ancona et al., 2017; Samek et al., 2016;
Wang et al., 2020b; Yeh et al., 2019).
We focus on metrics that use provided bounding boxes, as we believe that they offer a clear distinction between likely relevant features and irrelevant ones. BIG BSM AGI
SM GI SG IG
DeepLIFT 0.0 0.
Localization (Loc.)
BIG
BSM
AGI
SM
GI
SG
IG
DeepLIFT 0.
Energy Game (GE)
BIG
BSM
AGI
SM
GI
SG
IG
DeepLIFT 0.
Positive Percentage (
[Figure]
BIG
BSM
AGI
SM
GI
SG
IG
{+DeepLIFT
+}
[Figure]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
CIFAR standard robust (cid:15)
[Figure]
topk  max iters
ImageNet standard robust (cid:15) 2.
topk  max iters  (
[Figure]
B. I MPLEMENTING B OUNDARY S EARCH
Our boundary search uses a pipeline of PGDs, CW and AutoPGD.
Adversarial examples returned by each method are compared with others and closer ones are returned.
If an adversarial example is not found, the pipeline will return the point from the last iteration of the ﬁrst method (PGDs in our case).
Hyper-parameters for each attack can be found in Table 2.
The implementation of PGDs and
CW are based on Foolbox (Rauber et al., 2020; 2017) and the implementation of AutoPGD is based [Table] on the authors’ public repository 2 (we only use apgd-ce and apgd-dlr losses for efﬁciency reasons).
All computations are done using a GPU accelerator Titan RTX with a memory size of
GB.
Comparisons on the results of the ensemble of these three approaches are shown in Fig. 10a.
B. H YPER - PARAMETERS FOR A TTRIBUTION M ETHODS
All attributions are implemented with Captum (Kokhlikyan et al., 2020) and visualized with Trulens (Leino et al., 2021a).
For BIG and IG, we use 20 intermediate points between the baseline and the input and the interpolation method is set to riemann trapezoid .
For AGI, we base on the authors’ public repository 3 .
The choice of hyper-paramters follow the default choice from the authors for ImageNet and we make minimal changes to adapt them to CIFAR-10 (see Fig. 10b).
To visualize the attribution map, we use the HeatmapVisualizer with blur=10, normalization type="signed max" and default values for other keyword arguments from
Trulens.
B. D ETAILED R ESULTS ON L OCALIZATION M ETRICS
We show the average scores for each localizaiton metrics in Sec. 5.
We also show the boxplots of the scores for each localization metrics in Fig.
8 for the standard ResNet50 model and Fig.
9 for the robust ResNet50 ( (cid:96) 2 | 3 . 0 ).
All higher scores are better results.
= 0.
= 0.
= 0.
= 0.
= 0.
2 d i s t a n c e i n l o g B. A DDITIONAL C OMPARISON WITH AGI We additionally compare the localization ability of relevant features between BIG and AGI if we only use PGDs to return closest boundary points, that is we recursively increase the norm bound and perform PGD attack until the ﬁrst time we succeed to ﬁnd an adversarial point.
We denote this approach as BIGp.
Note that BIGp is still different from AGI by the type of the path, i e. lines [Table] and curves, over which the integral is performed. That is AGI also aggregates the path integral starting from a set of adversarial points found by the targeted PGD attack, where BIGp starts from the adversarial pointed returned by untargeted PGD attack.
We use the same parameters for both PGD and AGI from Fig. and we run the experiments over the same dataset used in Sec. 5.1. For reference, we also include the results of IG.
The results are shown in Table. We notice that after removing CW and AutoPGD, BIGp actually performs better than AGI, and even slightly better than BIG for the robust model. One reason to explain the tiny improvement from BIGp might be that for a robust network, the gradient at each iteration of the PGD attack is more informative and less noisy compared to a standard model so that the attack can better approximate the closest decision boundary.
The results in Table. 3 therefore demonstrates that BIG and BIGp are able to localize relevant features better than AGI. B. A DDITIONAL L OCALIZATION M ETRIC Besides the localization metrics used in Sec. 5.1, we discuss an additional localization metric frequently used for evaluating attention and CAM-based explanations: Top1-Loc Choe & Shim (2019);
Aggarwal et al. Top1-Loc is calculated as follows: an instance is considered as Top1-Loc correct given an attribution if 1) the prediction is Top1-correct; and 2) GT-Loc correct – namely, the IoU of the ground-truth bounding box and area highlighted by the attribution is more than%. When only using the images that are Top1-correct, Top1-Loc reduces to GT-Loc. Top1-Loc is different from other localization metrics used for evaluating attribution methods because it takes the prediction behavior of the target model into the account, which in general is not an axiom when motivating a gradient-based attribution method.
In the previous evaluations, we are only interested in images that the model makes correct Top1 predictions, in this section we will use the same images that are true-positives. In this case, Top1-Loc accuracy reduces to GT-Loc accuracy, and so we measure the GT-Loc directly. To determine the which part of the image is highlighted by the attribution, we compute a threshold for each attribution map and a pixel is considered within the highlight region if and only if the attribution score is higher than the threshold. For a given attribution map, we consider a threshold value t as the q -th percentile for the absolute values of attribution scores.
We plot the GT-Loc accuracy against q in Fig.
We notice that attention-based and CAM-based attributions usually produce a cloud-like visualization because of the blurring technique or upsample layers used to compute the results.
To ensure GT-Loc will work from gradient-based attributions we are interested in this paper, we also include results (Fig. 14) where we apply a Gaussian Blur ( σ = 3 . 0 ) to the attribution map ﬁrst before calculating the GT-Loc accuracy.
The results are aggreated over 1500 images from ImageNette on a standard ResNet50 and a robust ResNet50, respectively. Higher GT-Loc scores are better. Behavior of BIG.
The results in Fig. 13 and 14 show that BIG is better than all other attributions on standard models excluding SG and uniformly better including SG on a robust model. Before we provide some explanations about the behaviors of SG (green curves) on standard models in the next paragraph, we also observe that: 1) blurring only changes the GT-Loc scores but not the overral rankings across attributions; 2) a threshold corresponding to a percentile near 40% provides the best GT-Loc scores for all methods; 3) gradient-based attributions generally provide worse GT-Loc (or
Top1-Loc) scores compared to CAM-based and attention-based approaches in the literature Choe &
Shim (2019); Aggarwal et al.
(2020), which is not surprising because gradient-based approaches are usually axiomatically-justiﬁed to be faithful to the model.
Therefore, it is expected that the model will more or less learn spurious features from the input, which makes the gradient-based attributions noisy than attention and CAM-based ones.
Therefore, when localizing relevant features, users may want to consult activation-based approaches, i.e. CAMs, but when debugging and ensuring the network learns less spurious and irrelevant features, users should instead use gradient-based approaches because of the axioms behind these approaches.
Behavior of SG in Standard Models.
SG is uniformly better than all other approaches in terms of the Gt-Loc accuracies on a standard model, which is surprising but not totally unexpected.
We beleive the reason behind this result is that, SG is actually the gradient from a smoothed counterpart of the standard model (see discussions near Theorem 1), which is more robust.
Therefore, it does not seem to be an apple-to-apple comparison between SG and other approaches because it can be less faithful to the standard model – namely SG is more faithful to the smoothed classiﬁer.
That is very likely why SG is worse than BIG in Fig.
13b and 14b when the smoothing technique becomes marginal for improving the robustness for a model that has already been robustly trained.
B. S ANITY C HECK FOR BIG
We perform Sanity Checks for BIG using Rank Order Correlations between the absolute values of BIGs when randomizing the weights from the top layer to the bottom (Adebayo et al., 2018).
To ensure the output of the model does not become NaN , when randomizing the weights of each trainable layer, we ensure that we replace the weight matrix with a random matrix with the same norm as follows.
def _randomized_models(): all_parameters = for param in model.parameters(): all_parameters.
(param)
for step, param in enumerate(all_parameters[::-1]):
random_w = torch.
For each iteration, we continuously replace randomized 5 layers in the reversed sequence returned by model.parameters() and the results are plotted in Fig.
We consider BIG passes the sanity check as the results are similar compared with the top row of Fig 4 in Adebayo et al.
B. A DDITIONAL E XPERIMENT WITH S MOOTHED G RADIENT
Theorem 1 demonstrates that for a one-layer network, as we increase the standard deviation σ of the
Gaussian distribution used for creating the smoothed model m σ (Cohen et al., 2019), the difference between the saliency map and the boundary-based saliency map computed in m σ is bounded by a constant λ , which is monotonically decreasing w.r.t σ .
That is, greater σ will produce a smoothed model, where the saliency map (SM) explanation of m σ is a good approximation for the boundarybased saliency map (BSM).
However, as the depth of the deep network increases, a closed-form analysis may be difﬁcult to derive.
Therefore, in this section, we aim to empirically validate that the take-away from Theorem 1 should generalize to deeper networks.
Computing SM for m σ .
One practical issue to compute any gradient-related explanations for the smoothed model m σ is that m σ is deﬁned in an integral form, which can not be directly built with tf.keras .
However, Theorem 2 shows that the smoothed gradient of the original model m is equivalent to the saliency map of the smoothed model m σ .
Namely, the order of smoothing and integral is exchangeable when computing the gradient.
[Figure]
[Figure]
Theorem 2 (Proposition 1 from Wang et al.
(2020c))
Suppose a model f ( x ) satisﬁes max | f ( x ) | < ∞ .
For Smoothed Gradient g SG ( x ) , we have
[Equation]
where q ( x ) = N ( 0 , σ 2 I ) and (cid:126) denotes the convolution operation.
Computing BSM for m σ .
Another practical issue is computing the decision boundary for a smoothed model m σ is not deﬁned in a deterministic way as randomized smoothing provides a probabilistic guarantee.
In this paper, we do the following steps to approximate the decision boundary of a smoothed model.
To generate the adversarial examples for the smoothed classiﬁer of ResNet with randomized smoothing, we need to compute back-propagation through the noises.
The noise sampler is usually not accessible to the attacker who wants to fool a model with randomized smoothing.
However, our goal in this section is not to reproduce the attack with similar setup in practice, instead, what we are after is the point on the boundary.
We therefore do the noise sampling prior to run PGD attack, and we use the same noise across all the instances.
The steps are listed as follows:
We use numpy.random.randn as the sampler for Gaussian noise with its random seed set to 2020.
We use 50 random noises per instance.
In PGD attack, we aggregate the gradients of all 50 random inputs before we take a regular step to update the input.
We set (cid:15) = 3 .
and we run at most 40 iterations with a step size of 2 ∗ (cid:15)/ 40 . [Figure] n k
C o rr e l a t i o n w i t h A B S The early stop criteria for the loop of PGD is that when less than 10% of all randomized points have the original prediction.
When computing Smooth Gradient for the original points or for the adversarial points, we use the same random noise that we generated to approximate the smoothed classiﬁer.
Results. We run the experiment with 500 images from ImageNet on ResNet50 as this computation is signiﬁcantly more expensive than previous experiments. We compute the (cid:96) 2 distances between SM and BSM obtained from the steps above for several values as shown in Fig.
Notably, the trend of the log difference against the standard deviation σ used for the Gaussian noise validates that the qualitative meaning of Theorem 1 holds even for large networks.
That is, when the model becomes more smoothed, saliency map explanation is a good approximation for the boundary-based saliency map.
C S YMMETRY OF A TTRIBUTION M ETHODS
Sundararajan et al.
(2017) prove that a linear path is the only path integral that satisifes symmetry ; that is, when two features’ orders are changed for a network that is not using any order information from the input, their attribution scores should not change.
One simple way to show the importance of symmetry by the following example
and we refer Sundararajan et al.
(2017) to readers for more analysis.
Example 1 Consider a function f ( x, y )
=
min ( x, y ) and to attribute the output of f to the inputs at x = 1 , y =
we consider a baseline x = 0 , y = 0 .
An example non-linear path from the baseline to the input can be ( x = 0 , y = 0)
→ ( x = 1 , y = 0)
→ ( x = 1 , y = 1) .
On this path, f ( x, y ) = min ( x, y ) =
y after the point ( x = 1 , y = 0) ; therefore, gradient integral will returnfor the attribution score of x and 1 for y
(we ignore the inﬁnitesimal part of ( x = 0 , y = 0)
→ ( x = 1 , y = 0) ).
Similarly, when choosing a path ( x = 0 , y = 0) → ( x = 0 , y = 1) → ( x = 1 , y = 1) , we ﬁnd x is more important.
Only the linear path will return 1 for both variables in this case.
D C OUNTERFACTUAL A NALYSIS IN THE B ASELINE S ELECTION
The discussion in Sec.
6 shows an example where there are two dogs in the image.
IG with black baseline shows that the body of the white dog is also useful to the model to predict its label and the black dog is a mix: part of the black dog has positive attributions and the rest is negatively contribute to the prediction.
However, our proposed method BIG clearly shows that the most important part is the black dog and then comes to the white dog.
To validate where the model is actually using the white dog, we manually remove the black dog or the white dog from the image and see if the model retain its prediction.
The result is shown in Fig.
Clearly, when removing the black dog, the model changes its prediction from Labrador retriever to English foxhound while removing the white dog does not change the prediction.
This result helps to convince the reader that
BIG is more reliable than IG with black baseline in this case as a more faithful explanation to the classiﬁcation result for this instance.
E A DDITIONAL V ISUALIZATIONS FOR BIG
More visualizations comparing BIG with other attributions can be found in Fig.
16 and 17.
We show several examples in Fig.
18 when there are more than one objects in the input and we explain the model’s Top1 prediction, where we show that BIG is able to localize the objects that are actually relevant to the predicted label.