R ECTIFIED G RADIENT : L AYER - WISE T HRESHOLDING FOR S HARP AND C OHERENT A TTRIBUTION M APS
Anonymous authors
Paper under double-blind review
A BSTRACT
Saliency map, or the gradient of the score function with respect to the input, is the most basic means of interpreting deep neural network decisions.
However, saliency maps are often visually noisy.
Although several hypotheses were proposed to account for this phenomenon, there is no work that provides a rigorous analysis of noisy saliency maps.
This may be a problem as numerous advanced attribution methods were proposed under the assumption that the existing hypotheses are true.
In this paper, we identify the cause of noisy saliency maps.
Then, we propose Rectiﬁed Gradient , a simple method that signiﬁcantly improves saliency maps by alleviating that cause.
Experiments showed effectiveness of our method and its superiority to other attribution methods.
Codes and examples for the experiments will be released in public.
I NTRODUCTION
The gradient of the score function with respect to the input, also called the saliency map (Erhan et al., 2009;
Baehrens et al., 2010; Simonyan et al., 2014), is the most basic means of interpreting deep neural networks (DNNs).
It is also a baseline method for other advanced attribution-based methods.
However, our understanding of saliency maps is still poor.
Previous studies such as Springenberg et al. (2015) and Selvaraju et al.
(2017) have noted that saliency maps tend to be visually noisy.
To explain this phenomenon, Sundararajan et al.
(2016) and
Smilkov et al.
(2017) suggested saturation and discontinuous gradients as the causes (see Section 2.1 for further explanation).
There were several studies attempting to improve saliency maps by tackling these hypothesized causes (Bach et al., 2015; Montavon et al., 2017; Sundararajan et al., 2016; Shrikumar et al., 2017; Smilkov et al., 2017; Sundararajan et al., 2017).
Even though such attribution methods generally produce better visualizations, we ﬁnd troubling that the hypotheses regarding noisy saliency maps have not been rigorously veriﬁed (see Section 2.2 for more detail on attribution methods).
In other words, numerous attribution methods were built upon unproven claims that gradient discontinuity or saturation truly causes saliency maps to be noisy.
This situation gives rise to two major problems.
First, if the hypotheses regarding noisy saliency maps are incorrect, current and future works based on those hypotheses will also be erroneous.
Second, as we do not know precisely why saliency maps are noisy, we have to rely on heuristics and guessworks to develop better attribution methods.
In this paper, we address these problems by identifying saliency maps are noisy because DNNs do not ﬁlter out irrelevant features during forward propagation.
We then introduce Rectiﬁed Gradient , or RectGrad in short, a simple technique that signiﬁcantly improves the quality of saliency maps by alleviating the cause through layer-wise thresholding during backpropagation.
Finally, we demonstrate that RectGrad produces attributions qualitatively superior and quantitatively comparable to other attribution methods.
Speciﬁcally, we have the following key contributions:
• We explain why saliency maps are noisy.
Noise occurs in saliency maps when irrelevant features have positive pre-activation values and consequently pass through ReLU activation functions.
This causes gradients to be nonzero at unimportant regions.
We perform experiments with networks trained on CIFAR-10 to justify our claims (Section 3).
• We introduce Rectiﬁed Gradient, a method that removes noise from saliency maps by thresholding irrelevant units at ReLU binary gates during backpropagation (Section 4).
We ﬁrst explain the rationale behind Rectiﬁed Gradient (Section 4.1). We then prove that Rectiﬁed Gradient generalizes Deconvolution and Guided Backpropagation (Section 4.2).
In addition, we discuss two techniques that enhance the visual quality of Rectiﬁed Gradient attribution maps (Appendix C).
We ﬁrst investigate the effect of threshold level on attribution maps produced by Rectiﬁed Gradient (Section 5.1).
Then, we apply Rectiﬁed Gradient to networks trained on CIFAR10 and ImageNet to demonstrate that it produces qualitatively superior attribution maps (Section 5.2).
We also compare Rectiﬁed Gradient with other attribution methods using several quantitative metrics (Section 5.3).
B ACKGROUND O VERVIEW
Let S : R d (cid:55)→ R | C | be an image classiﬁcation network, where x ∈ R d is a single image instance and C is the set of image classes.
Then, we can deﬁne a score function S c : R d (cid:55)→ R for each class c ∈ C and the ﬁnal class of the image x is given by class ( x ) =
arg max c ∈ C S c ( x ) .
A typical score function is constructed by alternately composing afﬁne transformations and nonlinear activation functions.
A squashing function such as softmax is applied to the ﬁnal layer.
Since functions comprising S c are differentiable or piecewise linear, the score function is also piecewise differentiable.
Using this fact, Erhan et al.
, Baehrens et al.
(2010) and Simonyan et al.
(2014) proposed the saliency map, or the gradient of S c with respect to x , to highlight features within x that the network associates with the given class.
In an ideal case, saliency maps highlight objects of interest.
However, previous studies such as Springenberg et al.
and Selvaraju et al.
(2017) have pointed out that saliency maps tend to be visually noisy, as veriﬁed by Figure 1.
Three hypotheses were proposed to account for this phenomenon.
We describe them in the next section.
2. P REVIOUS H YPOTHESES
Saliency Maps are Truthful.
Smilkov et al.
(2017) suggested that noisy saliency maps are faithful descriptions of what the network is doing.
That is, pixels scattered seemingly at random are actually crucial to how the network makes a decision.
In short, this hypothesis claims that noise is actually informative.
Discontinuous Gradients.
Smilkov et al.
(2017) and Shrikumar et al.
(2017) proposed that saliency maps are noisy due to the piece-wise linearity of the score function.
Speciﬁcally, since typical
DNNs use ReLU activation functions and max pooling, the derivative of the score function with respect to the input will not be continuously differentiable.
Under this hypothesis, noise is caused by meaningless local variations in the gradient.
Saturating Score Function.
Shrikumar et al.
(2017) and Sundararajan et al.
(2017) suggested that important features may have small gradient due to saturation.
In other words, the score function can ﬂatten in the proximity of the input and have a small derivative.
This hypothesis explains why informative features may not be highlighted in the saliency map even though they contributed signiﬁcantly to the decision of the DNN.
2. P REVIOUS W ORKS ON I MPROVING S ALIENCY M APS
DNN interpretation methods that assign a signed attribution value to each input feature are collectively called attribution methods .
Attributions are usually visualized as a heatmap by arranging them to have the same shape as the input sample.
Such heatmaps are called attribution maps .
We now describe attribution methods that have been proposed to improve saliency maps.
Attribution Methods Addressing Discontinuity.
SmoothGrad
(Smilkov et al., 2017) attempts to smooth discontinuous gradient with a Gaussian kernel.
Since calculating the local average in a high dimensional space is intractable, the authors proposed a stochastic approximation which takes random samples in a neighborhood of the input x and then averages their gradients.
Attribution Methods Addressing Saturation.
Since saliency maps estimate the local importance of each input feature, they are vulnerable to saturation.
Therefore, attribution methods such as Gradient * Input (Shrikumar et al., 2017), Layer-wise Relevance Propagation (LRP) (Bach et al., 2015),
DeepLIFT (Shrikumar et al., 2017) and Integrated Gradient (Sundararajan et al., 2017) attempt to alleviate saturation by estimating the global importance of each pixel (Ancona et al., 2018).
Ancona et al.
(2018) has also shown that several global attribution methods are closely related under certain conditions.
Other Attribution Methods.
Some attribution methods take a different approach to improving saliency maps.
Deconvolution (Zeiler & Fergus, 2014) and Guided Backpropagation (Springenberg et al., 2015) remove negative gradient during backpropagation.
Due to this imputation procedure,
Deconvolution and Guided Backpropagation yield attribution maps sharper than those of other methods.
However, Nie et al.
(2018) has recently proven that these methods are actually doing partial image recovery which is unrelated to DNN decisions.
O UR E XPLANATION FOR N OISY S ALIENCY M APS
For brevity, we refer to pixels on the background as background features and pixels on the object as object features .
Then, noise in a saliency map corresponds to background gradient , or gradient that highlights background features.
We assume the DNN uses ReLU activation functions.
Under this condition, nonzero background gradient indicates the presence of at least one positive pre-activation in each network layer corresponding to background features.
To verify this, we visualized intermediate layer activations of a convolutional neural network (CNN) trained on CIFAR-10.
Figure 2b shows convolutional layer feature maps for an image that produced a noisy saliency map.
Since CNN ﬁlters act as feature extractors, we expected the CNN to remove most background feature activations through convolutions.
However, we found signiﬁcant amounts of background feature activations in all convolution layers.
As the last convolution layer is connected to fully connected layers, the majority of activations in the last convolution layer will have nonzero gradient.
Hence, the gradient ﬂowed through background feature activations up to the input.
This gradient ﬂow caused background gradient, as shown in Figure 2a.
From our perspective, the answer to “why are saliency maps noisy?” is trivial.
Saliency maps are noisy because background features pass through ReLU activation functions.
Therefore, rather than asking why saliency maps are noisy, we ask “do activations of background features highlighted by background gradient have nontrivial inﬂuence on the decision?”
If the answer is yes , noise in saliency maps is informative as suggested by Smilkov et al.
and saliency maps do not need any major improvement.
However, if the answer is no , we should ﬁnd a way to remove background gradient.
We investigated this question through two experiments.
Feature Map Occlusion.
We evaluated the signiﬁcance of background features by occluding activations at intermediate layers.
Then, we analyzed the effect of this perturbation on the ﬁnal decision.
Note that this is different from the Sensitivity metric (Bach et al., 2015; Samek et al., 2017).
Sensi-
[Figure]
tivity measures the impact of occlusion in the data space (e.g. pixel occlusion) while we measured the impact of occlusion in each feature space.
We ﬁrst created a background mask that covers background features in the images.
We then plotted the average class logits as we incrementally occluded intermediate layer activations that fell on the background mask.
We carried out occlusion following a random ordering and took the average over 50 trials.
Figures 3a and 3b give an example of a background mask and a completely occluded feature map respectively.
Figure 3c shows that the ﬁnal decision did not change throughout the occlusion process for all convolution layers.
Moreover, the difference between the top label logit and the next largest logit remained constant.
Therefore, background feature activations are irrelevant to the classiﬁcation task.
To further support this claim, we conducted a larger-scale version of this experiment, and we describe the procedure and results in Appendix A.1.
Training Dataset Occlusion.
Next, we show that gradient can be nonzero for completely uninformative features.
We occluded the upper left corner of all images in the training dataset with a 10 × 10 random patch and trained a randomly initialized CNN on the modiﬁed dataset.
We used the same patch for all images.
Since the test accuracy did not change signiﬁcantly ( 79 . 4% to 79 .
3% ), we expected the CNN to have learned to extract important features and ignore irrelevant ones.
However, Figure 4 shows that gradient is nonzero for the patch although it is completely irrelevant to the classiﬁcation task.
We can draw three conclusions from these experiments:
DNNs do not ﬁlter out irrelevant features during forward propagation.
DNNs are capable of making correct decisions even if we occlude the majority of background feature activations in intermediate layers.
This implies that most background feature activations are irrelevant to the classiﬁcation task.
Since DNNs do not remove irrelevant features through ReLU activation functions, zero threshold at ReLU binary gates during backpropagation also allow irrelevant information to ﬂow through the gradient.
With the conclusions above, we can refute the ﬁrst of three previous hypotheses.
As for the second hypothesis, we can interpret meaningless local variation in the gradient as a side effect of irrelevant features contaminating the gradient.
Why the network does not learn to ﬁlter out irrelevant features is
[Figure]
a matter of optimization, which is out of scope of this paper.
However, we believe it is a phenomenon worth investigating.
R ECTIFIED G RADIENT
We now introduce our technique to improve saliency maps.
As we have shown in Section 3, zero is a poor threshold at ReLU binary gates during backpropagation.
This indicates that we need better thresholds at ReLU binary gates in order to remove uninformative gradient from saliency maps.
To this end, we propose Rectiﬁed Gradient , or RectGrad in short, where the gradient propagates only through units whose importance scores exceed some threshold.
Importance score for an unit is calculated by multiplying its activation with gradient propagated up to the unit.
Formally, RectGrad is given as follows:
Suppose we have a L -layer ReLU DNN.
Denote input feature i as x i , pre-activation of unit i in layer l as z ( l ) i , its activation as a ( l ) i and gradient propagated up to a ( l ) i as R ( l +1)
i .
Let I ( · ) be the indicator function.
Then, the relation between a ( l ) i and z ( l ) i is given by a ( l ) i
[Equation]
when l < L and a ( L ) i = softmax ( z ( L ) i ) .
By the chain rule, backward pass through the ReLU nonlinearity for vanilla gradient is achieved by R ( l )
i =
I ( a ( l ) i > 0) ·
R ( l +1) i .
We modify this rule such that R ( l ) i
= I ( a ( l ) i · R ( l +1) i > τ ) ·
R ( l +1)
i for some threshold τ .
Backward pass through afﬁne transformations and pooling operations is carried out in the same manner as backpropagation.
Finally, importance scores for input features are calculated by multiplying gradient propagated up to input layer ( l = 0 ) with input features: x i · R (1) i .
Instead of setting τ to a constant value, we use the q th percentile of importance scores at each layer.
This prevents the gradient from entirely dying out during the backward pass. Due to the simplicity of the propagation rule, RectGrad can easily be applied to DNNs in graph computation frameworks such as TensorFlow (Abadi et al., 2016) or PyTorch (Paszke et al., 2017).
Listing 1 in Appendix D.1 shows how to implement RectGrad in TensorFlow.
In Appendix C we also introduce two techniques, namely the padding trick and the proportional redistribution rule (PRR) that enhance the visual quality of RectGrad attribution maps. 4. R ATIONALE B EHIND THE P ROPAGATION R ULE FOR R ECTIFIED G RADIENT This subsection explains the reason we have chosen R ( l )
i [Equation] and not [Equation]
or
[Equation]
as the deﬁnition of RectGrad.
The signiﬁcance of multiplying an unit’s activation with gradient propagated up to the unit is that it estimates the marginal effect of that unit on the output (Ancona et al., 2018).
For instance, consider the following linear model: f ( a 1 , a 2 , a 3 )
= 2 · a 1 + 1 · a 2 + 3 · a 3 .
We have ∂f/∂a 1 = 2 , ∂f/∂a
2 = 1 , and ∂f/a 3 = 3 .
Suppose we are given inputs a 1 = 2 , a 2 = 3 , a 3 = 1 and we apply RectGrad with q = 67 , i.e., we propagate the gradient through the unit with the highest importance score.
Clearly a 1 has the largest contribution of 2 · 2 = 4 to the ﬁnal output compared to 1 · 3 = 3 · 1 = 3 of a 2 and a 3 .
Only the ﬁrst rule correctly propagates gradient through the most inﬂuential unit a 1 while the latter two rules mistakenly choose a 2 and a 3 respectively.
Since the latter two rules fail even for this simple example, it is highly likely that they will not work for
DNNs which are constructed by composing multiple linear layers.
On the other hand, the ﬁrst rule propagates gradient through units with the largest marginal effect in a layer-wise manner.
Hence, it makes sense to select the ﬁrst propagation rule as the deﬁnition of RectGrad.
Next, we show that
RectGrad generalizes Deconvolution and Guided Backpropagation.
4. R ELATION TO D ECONVOLUTION AND G UIDED B ACKPROPAGATION
Claim 1.
Deconvolution *
Input is equivalent to Rectiﬁed Gradient with the propagation rule
[Equation]
for some small (cid:15) > 0 .
Claim 2.
Guided Backpropagation * Input is equivalent to Rectiﬁed Gradient when τ = 0 :
[Equation]
The proofs for Claims 1 and 2 are provided in Appendix E.1 and E.2 respectively.
These results indicate that RectGrad generalizes Deconvolution and Guided Backpropagation.
Figure 1 illustrates the relation between the saliency map, Deconvolution, Guided Backpropagation and RectGrad.
However, Nie et al.
(2018) has recently proven that Deconvolution and Guided Backpropagation are actually doing partial image recovery which is unrelated to DNN decisions.
RectGrad does not suffer from this problem as it does not satisfy the assumptions of the analyses of Nie et al.
(2018) for two reasons.
First, the threshold criterion is based on the product of activation and gradient which is not Gaussian distributed.
1 Second, we set τ as the q th percentile of importance scores and therefore τ will vary layer by layer.
We also show in Section 5.2 with adversarial attacks that attributions produced by RectGrad are class sensitive.
Therefore, RectGrad inherits the sharp visualizations of
Deconvolution and Guided Backpropagation while amending their disadvantages with layer-wise importance score thresholding.
E XPERIMENTS
To evaluate RectGrad, we performed a series of experiments using Inception V4 network (Szegedy et al., 2017) trained on ImageNet (Russakovsky et al., 2015) and CNNs trained on CIFAR- (Krizhevsky & Hinton, 2009).
See Appendix F.1 for details on the attribution map visualization method.
5. E FFECT OF T HRESHOLD P ERCENTILE
RectGrad has one hyper-parameter τ , which is set to q th percentile of importance scores for each layer.
Figure 5 shows the effect of threshold percentile for several images from ImageNet.
While the attribution maps were incomprehensible for q = 0 , the visual quality dramatically improved as we incremented q up to 20 .
There was no signiﬁcant change up to q = 80 .
Then the attribution maps began to sparse out again as we incremented q further.
We also observed that regions of high attributions did not change from q > 20 .
We speculate that the attributions stay constant between q = 20 and 80 because of zero activations.
That is, since we use ReLU activation functions, the majority of activations and consequently importance scores will be zero.
Hence, τ ≈ 0 for 20 ≤ q ≤ 80 .
This causes RectGrad attribution maps to resemble those produced by Guided Backpropagation * Input.
It indicates that we have to increment q > 80 in order to produce sparser attribution maps that highlight important regions instead of reconstruct input images.
5. Q UALITATIVE C OMPARISON WITH B ASELINE M ETHODS
We used the saliency map, Gradient * Input, Guided Backpropagation, SmoothGrad, Integrated
Gradient, Epsilon-LRP and DeepLIFT as baseline methods.
As for RectGrad, we used the padding trick and q = 98 for all attribution maps.
We show attributions both with and without application of the proportional redistribution rule.
In this subsection, we compare RectGrad with other attribution methods through three experiments that each focus on different aspect of qualitative evaluation.
We also show applying simple ﬁnal thresholding to baseline methods is not enough to replicate the beneﬁts of RectGrad.
To demonstrate this, we applied 95 percentile ﬁnal threshold to baseline attribution methods such that RectGrad and baseline attribution maps have similar levels of sparsity.
Coherence.
Following prior work (Simonyan et al., 2014; Zeiler & Fergus, 2014), we inspected two types of visual coherence.
First, the attributions should fall on discriminative features (e.g. the object of interest), not the background.
Second, the attributions should highlight similar features for images of the same class.
For the ﬁrst type of visual coherence, Figure 6 shows a side-by-side comparison between our method and baseline methods.
It can clearly be seen that RectGrad produced attribution maps more visually coherent and focused than other methods—background noise was nearly nonexistent. This q phenomenon may be due to noise accumulation.
Speciﬁcally, irrelevant features may have trivial gradient near the output layer.
However, since gradient is calculated by successive multiplication, the noise can grow exponentially as gradient is propagated towards the input layer.
This can result in confusing attribution maps which assign high attribution to irrelevant regions (e.g. uniform background in “lighter”), especially for deep networks such as Inception.
RectGrad does not suffer from this problem since it thresholds irrelevant features at every layer and hence stops noise accumulation.
In this situation, ﬁnal thresholding cannot replicate RectGrad’s ability to remove noise. For the second type of visual coherence, Figure 10 in Appendix A.2 shows attribution maps for a pair of images belonging to the same class.
Attribution maps generated by RectGrad consistently emphasized similar parts of the object of interest.
On the contrary, Saliency map, Gradient * Input and Epsilon-LRP emphasized different regions for each image instance.
Attributions for SmoothGrad, Guided Backpropagation, Integrated Gradient and DeepLIFT were generally coherent across images of the same class.
Nevertheless, they also highlighted background features and hence failed to satisfy the ﬁrst type of visual coherence.
This observation also holds for attribution maps with ﬁnal thresholding.
Adversarial Attack.
We evaluated class sensitivity following prior work by Nie et al.
Speciﬁcally, we compared the attributions for an image and its adversarial example.
If the attribution method is class sensitive, attribution maps should change signiﬁcantly since ReLU activations and consequently the predicted class have changed.
On the other hand, if the attribution method merely does image reconstruction, attribution maps will not change much since we add an indistinguishable adversarial perturbation to the image.
In this experiment, we used the fast gradient sign method (Goodfellow et al., 2015) with (cid:15)
= 0 .
01 to generate adversarial examples.
Figure 7 shows large changes in attribution maps produced by RectGrad.
We observed that only RectGrad attributions were coherent with the class labels.
Figure 11 in Appendix A.2 shows some instances where there was no signiﬁcant change in attribution maps produced by RectGrad.
In those cases, attribution maps for other methods also showed little change.
Hence, we can conclude that RectGrad is equally or more class sensitive than baseline attribution methods.
We observed that this conclusion also holds with ﬁnal thresholding. It is also possible that adversarial attacks only modiﬁed a tiny amount of ReLU activations (i.e. the images were near the decision boundary), causing little change in attribution maps.
5. Q UANTITATIVE C OMPARISON WITH B ASELINE M ETHODS
In this section, we quantitatively compare RectGrad with baseline methods using DNNs trained on CIFAR-10.
We did not include Epsilon-LRP since it is equivalent to Gradient * Input for ReLU DNNs (Ancona et al., 2018).
We divided baseline attribution methods into local and global methods following the criterion proposed by Ancona et al.
We also repeated the same experiments with ﬁnal thresholding to the baselines to compare them with RectGrad in similar sparsity setting.
Training Dataset Occlusion.
Just like the training dataset occlusion experiment in Section 3, we occluded the upper left corner of all images in CIFAR-10 training dataset with a 10 × 10 random patch and trained a randomly initialized CNN on the modiﬁed dataset.
We then summed all absolute attribution within the patch and averaged across the test dataset.
A reasonable attribution method should assign nearly zero attribution to the patch as it is completely irrelevant to the classiﬁcation task.
Figure 8 compares the amount of average attribution in the patch between attribution methods. We observed that without ﬁnal thresholding, RectGrad assigned little or no attribution to the random patch. However, all other methods failed to do so. For this test, we found using q = 95 ﬁnal threshold led to trivially different averages.
Hence we used a custom threshold for each baseline method such that they had similar average attribution in the patch as RectGrad.
We observed that
RectGrad had smaller standard deviation than baseline methods.
This indicates that RectGrad more consistently assigns near-zero attribution to the patch.
Therefore RectGrad has advantages over baseline methods regardless of whether ﬁnal threshold is used or not.
Figures for the following quantitative experiment outcomes are in Appendix A.3.
Noise Level.
We evaluated whether RectGrad really reduces noise through two experiments.
For the ﬁrst test, we created segmentation masks for 10 correctly classiﬁed images of each class (totalimages) and measured how much attribution falls on the background.
Speciﬁcally, we compared the sum of absolute value of attribution on the background.
For the second test, we measured the average total variation of attribution maps for each attribution method.
The average was taken over the test dataset.
Figure 12 shows that RectGrad assigned signiﬁcantly less attribution to the background than baseline methods.
Moreover, even with ﬁnal thresholding, RectGrad outperformed baseline methods.
In addition, Figure 13 shows that even though the total variation reduces for baseline methods after ﬁnal thresholding, RectGrad outperforms baseline methods in both cases.
The results imply that baselines with ﬁnal thresholding cannot replicate RectGrad’s ability to reduce noise.
Sensitivity. We evaluated RectGrad using the Sensitivity metric proposed by Bach et al.
and Samek et al.
Speciﬁcally, we measured how the logit for the initial class changed as features were occluded based on the ordering assigned by the attribution method.
We split the image into non-overlapping patches of 2 × 2 pixels.
Next, we computed attributions and summed all the values within each patch.
We sorted the patches in decreasing order based on the aggregate attribution values.
We then incrementally replaced the ﬁrst 100 patches with per-channel mean computed using the entire training set and measured the change in class logit.
We calculated the average acrossrandomly chosen test set images.
An attribution method is better if it has a lower sensitivity AUC.
The results are shown in Figure 14.
All attribution methods outperformed the random baseline in which we randomly removed patches.
We observed that RectGrad performed better than local attribution methods.
In comparison with global attribution methods, RectGrad showed similar performance up to approximately 10 patches (red vertical line) but the performance dropped as more patches were removed.
In Appendix B.1, we offer an explanation for this behavior.
Figureshows that after ﬁnal thresholding, RectGrad still outperforms local attribution methods.
For global attribution methods, RectGrad now shows similar performance.
ROAR and KAR.
We evaluated RectGrad using Remove and Retrain (ROAR) and Keep and Retrain (KAR) proposed by Hooker et al.
Speciﬁcally, we measured how the performance of the classiﬁer changed as features were occluded based on the ordering assigned by the attribution method.
For ROAR, given an attribution method, we replaced a fraction of all CIFAR-10 pixels that were estimated to be most important with a constant value.
We then retrained a CNN on the modiﬁed dataset and measured the change in test accuracy.
For KAR, we replaced a fraction of all CIFAR-10 pixels that were estimated to be least important.
We trained 3 CNNs per estimator for each fraction { 0 . 1 , 0 . 3 , 0 . 5 , 0 .
We measured test accuracy as the average of theses 3 CNNs.
An attribution method is better if it has a lower ROAR AUC and a higher KAR AUC.
Figure 16 presents ROAR scores.
All attribution methods outperformed the random baseline in which we randomly removed pixels.
RectGrad showed similar performance to local attribution methods but performed worse than all global attribution methods.
Next, Figure 17 shows KAR scores.
Interestingly, all baseline attribution methods failed to exceed even the random baseline.
Only RectGrad had similar or better performance than the random baseline.
In Appendix B.2, we offer an explanation for why RectGrad performed poorly in ROAR.
C ONCLUSIONS
Saliency map is the most basic means of interpreting deep neural network decisions.
However, it is often visually noisy.
Although several hypotheses were proposed to account for this phenomenon, there is no work that provides a thorough analysis of noisy saliency maps.
Therefore, we ﬁrst identiﬁed saliency maps are noisy because DNNs do not ﬁlter out irrelevant features during forward propagation.
We then proposed RectGrad Gradient which signiﬁcantly improves saliency maps by alleviating this problem through layer-wise thresholding during backpropagation.
We showed that
Rectiﬁed Gradient generalizes Deconvolution and Guided Backpropagation and moreover, overcomes the class-insensitivity problem.
We also demonstrated through extensive experiments that
Rectiﬁed Gradient outperforms previous attribution methods.
A E XPERIMENT R ESULTS A. S UPPLEMENTARY E XPERIMENT FOR F EATURE M AP O CCLUSION
[Figure]
[Figure]
To further support our claim that background feature activations are irrelevant to the classiﬁcation task, we conducted a larger-scale experiment. We created segmentation masks for 10 correctly classiﬁed images of each class (total 100 images) and repeated the feature map occlusion for each image. We then took the average of ( class logit ) − ( largest logit among the other 9 classes ) across allimages. Figures 9a and 9b give an example of a background segmentation mask and a completely occluded feature map.
Figure 9c shows that the difference is generally positive throughout the occlusion process, that is, the class does not change for most images.
From this, we can infer that background features are generally irrelevant to the classiﬁcation task.
[Figure] [Figure] B A DDITIONAL E XPLANATION FOR Q UANTITATIVE E XPERIMENTS B. S ENSITIVITY In comparison with global attribution methods, RectGrad showed similar performance up to approximately 10 patches (red vertical line) but the performance dropped as more patches were removed (Figure 14).
We speculate this happens due to the sparseness of RectGrad attribution maps. Since
RectGrad attribution maps are sparser than those of other methods, occluding approximatelyfeatures will be enough to remove core features highlighted by RectGrad.
Attributions for other features will not be as informative since they have trivial values.
Figure 18 shows that it is indeed the case.
For RectGrad, after occluding 10 top 2 × 2 patches, only attributions of small values remained.
For Gradient * Input, on the other hand, still had signiﬁcant amount of nontrivial leftover attributions.
We also see from Figure 15 that this phenomenon also happens for baseline methods with ﬁnal thresholding.
This implies that such behavior may be an inevitable consequence of sparseness.
B. ROAR
We believe that the poor performance of RectGrad in ROAR is also due to its sparseness.
Since RectGrad produces visually coherent attribution maps, the occluded regions can act as discriminative features.
To verify this, we replaced 10% of all CIFAR-10 pixels that were estimated to be most important with the channel-wise mean.
We then trained a CNN on the occluded dataset and visualized RectGrad attribution maps for images whose original and occluded versions were both classiﬁed correctly.
Figure 19 shows the results.
Attribution maps highlighted pixels around the occluded regions and moreover, similar regions were emphasized in the original image.
This corroborates our claim that the occluded regions act as discriminative features.
The assumption behind
ROAR is that the occluded features do not inﬂuence the classiﬁcation task (Hooker et al., 2018).
Since the above observation contradicts this assumption, ROAR may not be suitable for objectively evaluating RectGrad. C U SEFUL T ECHNIQUES Here, we present two useful techniques that can enhance the visual quality of attribution maps produced by RectGrad. C. P ADDING T RICK
Convolution inputs are typically zero padded along the border in order to preserve the spatial dimension of feature maps.
This occasionally leads to high activation values along the border if zero is out of input distribution.
Since importance scores are calculated by multiplying activation with gradient, outlying border activation can cause RectGrad to be propagated through the border instead of relevant features.
To solve this problem, we masked the border of gradient to zero before the backward pass through convolutions with padding.
One possible concern with the padding trick is that attributions may be faint for features adjacent to the border of the image.
However, we did not ﬁnd this to be a signiﬁcantly problem experimentally.
Listing 2 in Appendix D.2 shows how to implement the padding trick in TensorFlow.
C. P ROPORTIONAL R EDISTRIBUTION R ULE (PRR) FOR P OOLING L AYERS .
Attribution maps produced by RectGrad tend to be rough due to the discrete nature of thresholding.
This discontinuity can be compensated by using the proportional redistribution rule proposed by
Montavon et al.
(2017) for the backward pass through max-pooling layers.
Instead of propagating the gradient through only the most activated unit in the pool, gradient is redistributed proportional to unit activations.
Since the redistribution operation is continuous, attribution maps generated with the proportional redistribution rule are smoother.
Listing 3 in Appendix D.3 shows how to implement the proportional redistribution rule in TensorFlow.
D T ENSOR F LOW C ODES D. I MPLEMENTATION OF R ECTIFIED G RADIENT
1 import tensorflow as tf 2 3 from tensorflow.contrib.distributions import percentile 4 5 @tf.
RegisterGradient("RectifiedRelu") 6 def _RectifiedReluGrad(op, grad):
7def threshold(x, q): 9if len(x.shape.as_list()) > 3: thresh = percentile(x, q, axis=[1,2,3], keep_dims=
True) else: thresh = percentile(x, q, axis=1, keep_dims=
True) 14return thresh 16activation_grad = op.outputs[0] * grad thresh = threshold(activation_grad, q) 19return tf.where(thresh <
activation_grad, grad, tf.zeros_like(grad))
Listing 1: Implementation of Rectiﬁed Gradient in TensorFlow.
After registering this function as the gradient for ReLU activation functions, call tf.gradients() and multiply with inputs to generate attributions.
D. I MPLEMENTATION OF THE P ADDING T RICK
1 import tensorflow as tf 2 3 @tf.
RegisterGradient("RectifiedConv2D") 4 def _RectifiedConv2DGrad(op, grad): 5if op.get_attr(’padding’)
=
= b’SAME’:
7shape = tf.shape(grad) mask =
tf.ones([shape[0], shape[1] - 2, shape[2] - 2, shape[3]]) mask =
tf.pad(mask, [[0,0],[1,1],[1,1],[0,0]]) grad = grad
* mask 12input_grad = tf.nn
D.
I MPLEMENTATION THE P ROPORTIONAL R EDISTRIBUTION R
1 import tensorflow as tf 2 3 from tensorflow.python.ops import gen_nn_ops 4 5 @tf.
RegisterGradient("RectifiedMaxPool") 6 def _RectifiedMaxPoolGrad(op, grad): 7z =
tf.nn.avg_pool(op.inputs[0], op.get_attr(’ksize’), op.get_attr(’ strides’), op.get_attr(’padding’)) + 1es =
grad / z c = gen_nn_ops._avg_pool_grad(tf.shape(op.inputs[0]), s, op.get_attr( ’ksize’), op.get_attr(’strides’), op.get_attr(’padding’))
11return op.inputs[0] * c
Listing 3: Implementation of the proportional redistribution rule in TensorFlow.
After registering this function as the gradient for max-pooling operations, call tf.gradients() and multiply with inputs to generate attributions.
E P ROOF OF C LAIMS E. P ROOF OF C LAIM
Proof.
Note that the backward propagation rule for Deconvolution through the ReLU nonlinearity is given by
[Equation]
Since the DNN uses ReLU activation functions, a ( l ) i + (cid:15) > 0 and therefore [Equation]
[Equation]
) for all l and i .
The result follows from Equation 2.
E. P ROOF OF C LAIM
Proof.
Note that the backward propagation rule for Guided Backpropagation through the ReLU nonlinearity is given by
[Equation]
Since the DNN uses ReLU activation functions, a ( l ) i ≥ 0 and therefore
[Equation]
for all l and i .
The result follows from Equation 4.
F E XPERIMENTS S ETUP F. A TTRIBUTION M AP V ISUALIZATION
To visualize the attributions, we summed up the attributions along the color channel and then capped low outlying values to 0 .
5 th percentile and high outlying values to 99 .
5 th percentile for RGB images.
We only capped outlying values for grayscale images.
F. CIFAR-
The CIFAR-10 dataset (Krizhevsky & Hinton, 2009) was pre-processed to normalize the input images into range [ − 1; 1] .
We trained a CNN using ReLU activation functions with Adam forepochs to achieve 79 .
4% test accuracy.
For the dataset occluded with the random patch, we used the same settings to achieve 79 .
3% test accuracy.
CIFAR-10 CNN
Conv 2D ( 3 × 3 , 32 kernels) Conv 2D ( 3 × 3 , 32 kernels)
Max-pooling ( 2 × 2 )
Dropout (0.25)
Conv 2D ( 3 × 3 , 64 kernels) Conv 2D ( 3 × 3 , 64 kernels)
Max-pooling ( 2 × 2 )
Dropout (0.25)
Dense (256)
Dropout (0.5)
Dense (10)
F. I NCEPTION V
We used a pre-trained Inception V4 network.
The details of this architecture can be found in Szegedy et al.
For the adversarial attack, we used the fast gradient sign method with (cid:15) = 0 . 01 .