N EW D EFINITIONS AND E VALUATIONS FOR S ALIENCY M ETHODS : S TAYING I NTRINSIC AND S OUND
Anonymous authors
Paper under double-blind review
A BSTRACT
Saliency methods seek to provide human-interpretable explanations for the output of machine learning model on a given input.
A plethora of saliency methods exist, as well as an extensive literature on their justiﬁcations/criticisms/evaluations.
This paper focuses on heat maps based saliency methods that often provide explanations that look best to humans.
It tries to introduce methods and evaluations for masked-based saliency methods that are intrinsic — use just the training dataset and the trained net, and do not use separately trained nets, distractor distributions, human evaluations or annotations.
Since a mask can be seen as a “certiﬁcate” justifying the net’s answer, we introduce notions of completeness and soundness (the latter being the new contribution) motivated by logical proof systems.
These notions allow a new evaluation of saliency methods, that experimentally provides a novel and stronger justiﬁcation for several heuristic tricks in the ﬁeld (T.V. regularization, upscaling).
I NTRODUCTION
Why did the deep net give a certain answer on a particular input, and can we trust the answer?
Saliency methods try to provide explanations, and are thus of great interest from viewpoint of human explainability, fairness, robustness, etc.
This paper restricts attention to methods that return an importance score for each coordinate of the input —usually visualized as a heat map— that captures its importance to the ﬁnal decision.
1 We refer the reader to Samek et al.
for an extensive survey and Section 2 for a short account of such methods and controversies.
There are two important components in the research on saliency: saliency methods that produce such heat maps for explanations, and saliency evaluation metrics that aim to test and compare saliency methods.
Numerous saliency methods have been proposed to learn such heat maps.
Some methods learn maps through “credit attribution” to individual input coordinates by using methods reminiscent of backpropagation (Binder et al., 2016; Selvaraju et al., 2019), while some are derived using careful axiomatization of credit assignment using the idea of Shapley values from cooperative game theory (Lundberg & Lee, 2017b;
Yeh et al., 2020).
Some recent methods train another deep net to produce heat maps (Dabkowski & Gal, 2017; Phang et al., 2020).
Given the proliferation of saliency methods, an ecosystem of evaluation metrics has emerged to evaluate the quality of explanations produced by saliency methods, either through human evaluations (Adebayo et al., 2018), comparison to certain ground truth explanations (Zhang et al., 2018) or other evaluations that do not require any external annotation or supervision (Dabkowski & Gal, 2017)
Petsiuk et al.
In this paper we restrict attention to a class of evaluations we refer to as intrinsic , that aim to evaluate saliency maps based on whether they are good explanations for the model prediction.
These only involve computations using the provided heat map and the net, and do not involve extrinsic factors such as human judgements or retraining.
A popular idea in such evaluations (see Section 3 for references) is to create a new composite input —or sequence of such inputs— using the heat map and the original input, and to evaluate the original net on this composite input.
For example, if
M is a binary vector with 1 ’s in the k coordinates with the highest values in the heat map, then x (cid:12) M (with (cid:12) denoting wise multiplication) can be viewed as a composite input 2 (aka
Inspired by logical reasoning we identity a key component missing in existing intrinsic evaluations : soundness of the saliency method.
In simple terms checking soundness for mask-based explanations entails verifying that it is impossible for the method to produce well-performing (in the compositeinput evaluations) heat maps corresponding to the incorrect labels, i e. labels not predicted as the most likely by the model.
By contrast current masking evaluations focus on a property akin to completeness 3 in logic: the heat map “justiﬁes” the correct label, i e. the label judged to be most likely by the model.
We argue that soundness can be a useful criterion in addition to completeness.
The concept of soundness is motivated by observing that the recent mask-based saliency methods are inherently different from the earlier gradient-based methods. They produce a heatmap that is interpreted as a masked image that, when fed into the net must make it output the same label that received the top logit value on the full image.
Our paper observes that it is unclear at a logical level why this should be considered an “explanation”, since to be convinced one must verify that a different mask could not induce the net to output a label corresponding to other logits.
Figureshows that in fact this issue is real; masks do exist that can “justify” the wrong labels.
We introduce soundness as a quantitative measure for this issue (deﬁnitions in Section 3.1), and propose that evaluations of mask-based saliency methods should account for soundness.
Many past works allude to difﬁculty of the soundness concept, mentioning that computations to ﬁnd the “best” mask end up ﬁnding artifacts that can even justify labels that are obviously incorrect.
To avoid uncovering artifacts, the methods to ﬁnd masks/heat maps use a suitable regularization (usually TV).
However, Figure 1 illustrates that TV regularization is not a full solution to removing artifacts.
Another reason for incorporating soundness with completeness in metrics is to implicitly encourage saliency methods to also produce saliency maps for incorrect labels.
When using the classiﬁer in the wild—as opposed to on a dataset with centered images— we might encounter multiple salient objects in the image, and the net may not have high conﬁdence in a single label.
It would then be reasonable to have saliency methods ﬁnd evidence for all the labels which the net outputs with reasonable conﬁdence.
We ﬁnd in Figure 2 that our saliency method that produces masks for all labels can produce different and meaningful masks for different labels when there are multiple objects present in the image 4 , compared to prior state-of-the-art methods, that output a single mask for all labels that contains both objects (more on this in Appendix B).
-object case is to deﬁne correct labels as the labels whose model probability exceeds a certain threshold.
Even without the extended deﬁnition of correct labels, Deﬁnition 3.2 with suitable α, β already allows explanations to exist for both elephant and zebra as long as the model probability for elephant and zebra are high enough. Main contributions: In this paper we make a case for pushing saliency methods and metrics to incorporate soundness in addition to completeness. We propose metrics and a method and experimentally evaluate them in comparison to previous methods and metrics, leading to some novel ﬁndings. Our main contributions are as follows:
• A new single number metric named consistency score that evaluates the completeness and soundness simultaneously.
It measures the probability that the saliency map for the model prediction has the best quality compared to all other labels.
See Section 3.2 for a formal deﬁnition.
• A saliency method in Section 4 that bears similarity to prior works that learns masks through optimization, but with subtle differences.
One of the differences is based on the logic of soundness: we learn a mask by trying to maximize the probability for the given label, rather than matching the probability, and learn the best mask for all labels this way.
We ﬁnd in experiments (Section 6) that our method not only performs comparably to prior masked-based methods on earlier saliency metrics, but also does better than those methods on the proposed consistency score.
• Previous works proposed “tricks” like TV penalty and upscaling of masks (Fong & Vedaldi, 2017; Dabkowski & Gal, 2017) that are motivated by their ability to avoid “artifacts” and make the saliency maps visually look better.
These justiﬁcations, however, are not in the spirit of an intrinsic evaluation of saliency maps.
In this paper, we provide a novel intrinsic justiﬁcation for these choices of TV penalty and upscaling, by showing that they improve the saliency maps by making them more sound, as empirically demonstrated in Section 6.
We complement this ﬁnding with a theoretical result in Section 5 that proves that TV regularization can help with soundness in a simple linear classiﬁcation setting.
Paper outline: Section 2 discusses prior works on saliency methods and evaluations.
Sectiondelves further into the concept of completeness and soundness in the context of saliency evaluation, and describes our proposed metric of consistency score.
Section 4 describes our saliency method and how it differs from prior approaches.
Finally Section 5 describes various supportive experiments.
P RIOR APPROACHES
We delegate a more thorough description of prior work to Appendix D, but mention some common saliency and saliency evaluation methods here.
Saliency methods aim to explain a model’s decision about an input.
Saliency evaluation methods aim to evaluate the goodness of a saliency method.
Saliency methods include backpropagation based approaches such as Gradient (cid:12) Input (Shrikumar et al., 2017), LRP (Binder et al., 2016), GradCAM (Selvaraju et al., 2019), Smooth-Grad (Smilkov et al., 2017).
Another line of work is masking methods which include techniques based on averaging over randomly sampled masks (Petsiuk et al., 2018), optimizing over meaningful mask perturbations (Fong & Vedaldi, 2017), and real time image saliency using a masking network (Dabkowski & Gal, 2017).
Pixels that have been removed from the image by the mask may be replaced by greying out, by Gaussian blurring as in (Fong & Vedaldi, 2017), or with inﬁllers such as CA-GAN (Yu et al., 2018) used in (Chang et al., 2018; Phang et al., 2020), or DFNet (Hong et al., 2019).
De Cao et al.
ﬁnd masks using differentiable masking.
Boolean logic is another approach for saliency methods (Ignatiev et al., 2019b;a; Macdonald et al., 2019; Mu & Andreas, 2020;
Zhou et al., 2018).
Arguments about saliency.
As saliency methods have arisen, discussion about them has occurred e.g. in (Seo et al., 2018;
Fryer et al., 2021; Gu et al., 2018; Sundararajan & Najmi, 2020).
Some works reveal situations where Shapley axioms work against feature selection or where Shapley values may be calculated in conﬂicting ways (Fryer et al., 2021; Sundararajan & Najmi, 2020).
Others question why noise adding saliency methods work (Seo et al., 2018), with some pointing out that some explanations are not class discriminative (Gu et al., 2018).
Saliency evaluation methods.
Extrinsic evaluation metrics include the WSOL metric, and Pointing
Game metric proposed by Zhang et al.
(2018) and ROAR (Hooker et al., 2019).
Other more intrinsic methods include early saliency evaluation techniques like MorF and LerF (Samek et al., 2016),
Insertion and Deletion game proposed by Petsiuk et al.
(2018), which involve either inserting pixels in order of most importance or deleting pixels in order of most importance.
BAM (Yang & Kim, 2019) creates saliency maps by pasting object pixels from MSCOCO (Lin et al., 2014)
The Saliency Metric proposed by Dabkowski & Gal (2017) thresholds saliency values above some α chosen on a holdout set, ﬁnds the smallest bounding box containing these pixels, upsamples and measures the ratio of bounding box area to model accuracy on the cropped image, s ( a, p )
=
log(max( a, 0 . 05))
− log( p ) where a is the area of the bounding box and p is the class probability of the upsampled image.
Controversies.
Brunke et al. (2020) show that perturbation methods are sensitive to baseline and Petsiuk et al. (2018) point out that human centric explanations (based on bounding boxes) may not reveal why the model made a certain decision.
Our notion of intrinsic saliency method aligns with the latter idea, and we introduce a new criteria for evaluating saliency maps: soundness, which captures the notion that maps should not produce explanations for low probability labels.
M ASKING E XPLANATIONS AND C OMPLETENESS /S OUNDNESS
A running theme in saliency methods/evaluations is that the salient pixels should be sufﬁcient to convince us about the model’s output, regardless of contents of the other pixels.
For instance, greying out (or setting them to average pixel value) non-salient pixels should have very little effect on the output.
This idea appears in many saliency methods (Dabkowski & Gal, 2017; Phang et al., 2020) (and evaluation metrics (Petsiuk et al., 2018)), including Shapley values and mask-based explanations.
This motives the following deﬁnition of a masked-based explanation.
Deﬁnition 3.1 (Masking Explanation) .
A masking explanation for input x is a distribution ∆ over subsets S of input coordinates (“salient sets of x ”) as well as an input modiﬁcation process Γ that generates a distribution of modiﬁed inputs ˜ x ∼ Γ( x, S ) under the constraint that ˜ x matches x on every coordinate in set S .
We use Γ( x, ∆) to denote the distribution of ˜ x produced by Γ when the input is x and the set S is sampled from ∆ .
A salient set S could be the direct output of any saliency method, or the output heat map passed through a potentially randomized discretization (e.g., the insertion game in Section 3.3), leading to a distribution ∆ over sets S .
Simple examples of the input modiﬁcation process Γ that stay in the intrinsic framework are: greying out the pixels outside S , replacing them by pixels from a Gaussian blurring of x (Fong & Vedaldi, 2017).
Another example of Γ —albeit non-intrinsic and hence not used in this paper— uses a conditional image generative model to produce new pixel values in S conditional on pixels in S being consistent with x (Agarwal & Nguyen, 2020; Chang et al., 2018).
Deﬁnition 3.1 can also be changed to allow Γ to change the values of pixels in S .
In our proposed method in Section 4, we consider another distribution Γ , which replaces pixels outside of S with pixels from a random image from the training set 5 .
This amounts to grafting salient pixels of x on top of a random image, reminiscent of BAM evaluations (Yang & Kim, 2019) for saliency methods.
3. C OMPLETENESS AND S OUNDNESS
We now describe completeness and soundness for saliency evaluations.
In a multiclass classiﬁcation setting, denote by f ( x, a ) the probability for label a returned by the model on input x .
The prediction of the model on input x is ˆ y ( x ) :
=
arg max a ∈ C
f
( x, a ) , where C denotes all classes.
For p ∈ [0 , 1] , a masking explanation (∆ , Γ) for input x is said to p -validate the label a ∈ C if
[Equation]
We say that a masking explanation x p -validates the model prediction if it p -validates ˆ y ( x ) .
Deﬁnition 3.2 (Completeness and Soundness) .
Fix an input modiﬁcation process Γ .
A saliency method given f, x and any label a ∈ C produces a distribution ∆( x, a ) over salient sets.
For α, β ≤ 1 , the method is α -complete on f, x, a if the masking explanation (∆( x, a ) , Γ) p -validates the label a for x with p = α · f ( x, a ) , and the method is β -sound on f, x, a if the masking explanation (∆( x, a ) , Γ) cannot p -validate the label a with p > 1 β max { f ( x, a ) , (cid:15) } 6 . Intuitively, when α, β are close to 1 , α -completeness means that if the model outputs a high probability for label a , then the probability for label a after seeing only the coordinates in the salient sets is also high; β -soundness means that if the model outputs a low probability for label a , then the probability for label a after seeing only the coordinates in the salient sets is also low.
Thus a desirable saliency method should have α -completeness for labels predicted by the model and βcompleteness for other labels.
This captures the idea that we want the method to be able to validate the model prediction ˆ y ( x ) but not other labels.
It is worth noting that the saliency method that declares all pixels to be salient is α -complete and β -sound with α = β
= 1 , but its size is too large to provide any useful information.
Thus one should ask the saliency method to search over the salient sets of smaller sizes.
But artifacts for incorrect labels may occur at small size (Figure 1), which hurts soundness.
For this reason we put explicit or implicit constraints on the salient sets (e.g., TV regularization) and ask saliency methods to be α -complete and β -sound with α, β as close to 1 as possible.
Ideally we should adjust the constraints to achieve a good tradeoff between completeness and soundness.
In our experiments, we ﬁnd that standard “tricks” like TV regularization and mask upscaling provide such good tradeoffs, as observed in Table 3 and Figure 3.
Important note.
In the logic setting, from where we borrow the concept of soundness, one has to search among all possible proofs to ensure that there is indeed no valid proof for any false proposition.
Similarly, our soundness metric is non-vacuous only if the saliency method “makes its best efforts” to ﬁnd masking explanations for every label.
Otherwise the saliency method could for instance ignore the value of a return the mask for ˆ y which can p -validate ˆ y ( x ) with a high value of p and thus not validate any other label, leading to a false notion of soundness.
If the saliency method behaves like this on the incorrect label (i.e., is not working at producing an explanation) then as a backup the evaluation can use some default method to produce saliency explanations for those labels.
Our method for mask-based explanations, described later, is simple and not tied to any speciﬁc philosophy or a-priori deﬁnition of saliency.
It can produce explanations for all labels.
3. C ONSISTENCY S CORE FOR E VALUATING C OMPLETENESS AND S OUNDNESS
Inspired by the notions of completeness and soundness, we propose to use a single number metric that evaluates the completeness and soundness simultaneously over a data distribution.
Deﬁnition 3.3 (Consistency Score) .
Let X be the distribution of input x and Γ be an input modiﬁcation process.
For a saliency method that produces salient sets ∆( x, a ) on input x and label a , we deﬁne the “saliency score” for ( x, a ) as g ∆ ( x, a ) =
E ˜ x ∼ Γ( x, ∆( x,a ))
[ f (˜ x, a )] , i.e. average probability the model assigns to a labels for modiﬁed versions of x using the label-speciﬁc salient sets ∆( x, a ) .
The consistency score for the saliency method is deﬁned as the probability that the saliency score is consistent with the model prediction, i.e.,
[Equation]
It is easy to see that if a saliency method is 1 -complete and 1 -sound on all inputs, then the consistency score is 1 .
The following lemma shows that even α -completeness
and β -soundness can imply consistency if there is a gap between the probabilities for the largest and second largest labels.
Lemma 3.4.
For an input x , if a saliency method is α -complete on the model prediction
ˆ y ( x ) and β -sound on all other labels, and if the ratio between the output probabilities for the largest label ˆ y ( x ) and second largest labels
[Equation]
larger than [Equation] , then the saliency method is consistent with the model on input x .
Proof.
By α -completeness, the saliency method p -validates ˆ y ( x ) with p = α · f ( x, ˆ y ( x )) .
Since f ( x, ˆ y ( x )) max { max a (cid:54)
=
ˆ y ( x ) { f ( x,a ) } ,(cid:15) } > 1 αβ , p has the lower bound p > 1 β max { max a (cid:54) =
ˆ y ( x ) { f ( x, a ) } , (cid:15) } for all a (cid:54) =
ˆ y ( x ) .
Then β -soundness implies that the saliency method cannot p -validate a .
Therefore E ˜ x ∼ Γ( x, ∆( x,a ))
[ f (˜ x, a )] is maximized when a = ˆ y ( x ) , and hence consistency.
3. C ONNECTION TO P OPULAR A REA -U NDER - THE -C URVE E VALUATIONS OF H EAT M APS
As mentioned, our approach requires the method to output salient sets whereas existing methods often return a heatmap of saliency values.
However, popular Area-Under-the-Curve (AUC) evaluation metrics (Petsiuk et al., 2018) of saliency methods can be reinterpreted in terms of our completeness and soundness.
An example is the insertion game :
AUC of Insertion game: For s = 1 to d take the top s saliency values, and plot the probability given by model to label a on the input where the top s pixels of x are retained and remaining pixels are assigned values from the input modiﬁcation process Γ .
Return AUC.
Lemma 3.5.
Given a saliency heatmap with AUC ρ for label a , we can convert it into a masking explanation that ρ -validates the label a .
Proof.
Given a heatmap, produce the salient set S by picking s uniformly at random from 1 to d and letting S be coordinates corresponding to top s saliency values.
Let ∆ be the distribution of the produced salient sets.
Then E ˜ x ∼ Γ( x, ∆)
[ f (˜ x, a )] = ρ by deﬁnition.
P ROCEDURES TO FIND MASKING EXPLANATIONS
Based on the deﬁnition 3.1 of masking explanations, we propose a very simple method to ﬁnd saliency masks with good empirical completeness and soundness scores.
We introduce our methods in this section concisely.
The detailed intuitions and implementations are provided in Appendix A.
Our method bears similarity to prior work on learning mask based saliency maps, but with subtle differences.
The key difference from Dabkowski & Gal (2017); Phang et al.
(2020) is that we do not use a neural network to learn the mask and the difference from Fong & Vedaldi (2017); Agarwal & Nguyen (2020);
Chang et al.
(2018) is that we use a different input modiﬁcation process Γ (see
Deﬁnition 3.1) to inﬁll other pixels during training: given x and S , generate a hybrid inputs whereby the pixels in the set S match with x and the pixels outside S are set to those of a random image drawn from the training set X .
Replacing with random image pixels instead of gray pixels will make it harder to ﬁnd a mask.
Despite this, we can still learn a mask that predicts the correct label with high conﬁdence in practice. Furthermore, it achieves superior performance over replacing with gray pixels, which is likely because the added hardness of task helps increase the robustness of the output.
The ﬁnal crucial difference is that our method can ﬁnd a mask for every input-label pair ( x, a ) and does so by trying to maximize the probability assigned to label a on modiﬁed images, as opposed to trying to match the model probability for a .
As is standard, we relax the domain of masks M from binary { 0 , 1 } hw to continuous [0 , 1] hw , and optimize M for input x and label a based on the following natural objective
[Equation]
where the part of x on M is superimposed onto a distractor ¯ x ∼ X as ˜ x
= M (cid:12) x
+ (1 − M ) (cid:12) ¯ x , and the (cid:96) 1 norm penalty on M helps to reduce the size of masks.
We further employ Total-Variation (TV) penalty (Fong & Vedaldi, 2017) and upscaling of the mask from a lower resolution one (Petsiuk et al., 2018) by learning a low-resolution mask at scale s ,
M ∈ [0 , 1] hw/s 2 , to minimize the following
[Equation]
where M × s ∈ R hw is obtained by upscaling M by a factor of s ∈ { 1 , 4 } via bilinear interpolation.
While the motivation cited for these two “tricks” is to avoid artifacts, it is not clear whether artifacts are a bad thing, since they might be relevant to the net’s decision-making.
Indeed, in our experiments, we show that while TV penalty or upscaling does produce better looking masks, they lead [Figure] to a drop in the completeness metric .
However we also that adding such tricks leads to significant improvement in the soundness metric, thus providing a novel justiﬁcation for the use of such tricks, beyond just the heuristic argument of getting rid of artifacts.
In Section 5 we also provide theoretical justiﬁcation for why TV penalty can help with soundness, even for the simple case of linear predictors on non-image data.
C LARIFYING BENEFIT OF TV REGULARIZATION
Our experiments show that the main beneﬁt of TV regularization in saliency methods is that it improves soundness.
Here we sketch a simple example showing how this beneﬁts even for simple linear models on non-image data.
Details appear in Appendix E. Consider S , a dataset of labeled data ( x , y ) where labels y are binary, and a linear classiﬁer with weight w and margin γ > 0 .
Assume inputs are of unit norm and with bounded (cid:96) ∞ norm and so is w .
The saliency method has to return a single subset S of salient coordinates to validate the label.
The input modiﬁcation process Γ will assign 0 to all coordinates outside S .
Question is whether the decision can be validated using a small S .
A priori this seems impossible.
Furthermore imposing a TV constraint seems to help nothing, because the solution has no obvious continuity structure.
But now assume we randomly permute the coordinates.
This (paradoxically) turns out to make a big difference!
For simplicity we consider salient sets among intervals, in other words sets of TV ≤ 2 .
Theorem 5.1.
For any ( x , y ) ∈ S , after random shufﬂe following holds for any L 1 = Ω( 1 γ 2 log 1 δ ) ,
L 2 = Ω( 1 γ 2 log dδ ) , where probability is over the random shufﬂe:
(Completeness)
With probability 1 − δ , there is an interval of length L 1 that validates y ; 2.
(Soundness)
With probability 1 − δ , no interval of length L 2 can validate − y .
E XPERIMENTS
In this section, we present results for our mask learning procedures described in Section 4 and our consistency score metric described in Deﬁnition 3.3.
We also analyze the role of TV penalty and upsampling.
We observe from the results that (1) our simple mask learning procedure achieves competitive performance with existing saliency methods on existing metrics; (2) our procedure does better on our consistency score metric compared to previous methods, which suggests that consistency score has no major conﬂicts with previous metrics and that our method is more sound; (3) TV regularization and upscaling signiﬁcantly aid soundness, while only slightly hurting completeness.
Overall, consistency score increases with higher TV regularization and upscaling, thus showing the ability of our metrics to provide intrinsic explanations for existing “tricks” that previously only had extrinsic justiﬁcations.
Experiments involving consistency score are computationally infeasible to perform on ImageNet as it contains 1000 classes.
Therefore, these experiments are performed either on CIFAR-10 or Imagenette (Howard, 2020), a 10-class ImageNet subset.
Even for CIFAR-10 and Imagenette, since [Table] consistency score requires computing 10 times more saliency maps than other metrics, we test it on 1000 randomly drawn images from original test set.
We also run experiments that do not involving consistency score on various datasets (including ImageNet and CIFAR-100) with various models.
Results are shown in Appendix C. Details of training procedure (beyond those in Section 4) are also in the Appendix C.
6. F ULL COMPARISON TO EXISTING METRICS AND METHODS ON I MAGENETTE
We compare our procedure to other methods including Real Time Saliency (Dabkowski & Gal, 2017), Gradient (cid:12) Input (Shrikumar et al., 2017), Fong & Vedaldi (2017) and Phang et al.
(2020) on Imagenette.
Apart from our new consistency score metric, we calculate the Deletion Game and
Insertion Game metrics using the code provided by https://github.com/eclique/RISE , and Saliency metric (SM), an another intrinsic evaluation metric from Dabkowski & Gal (2017).
Detailed description of the different metrics can be found in the Appendix.
For our method, we use the learned mask M from optimizing the objective function in Equation (4) with λ TV = 0 .
01 and s = { 1 , 4 } as the saliency map.
For fairness, we use the identical ResNet pretrained on ImageNet as the base classiﬁer for every saliency method.
The salient sets used in our consistency score follows the Insertion game.
Insertion scores were calculated by replacing remaining pixels with gray.
We normalize the maps so that all values lie in [0 , 1] before use.
To get a sense what the numbers mean, we also include the performance of a random mask, where the score for each pixel is sampled independently from standard normal distribution.
The results are shown in Table 1.
For the deletion metric, we note that most methods have comparable or worse performance than the random mask, which suggests that the metric does not give us much signal about the goodness of the saliency maps.
For the Insertion scores and saliency metric, our method performs decently since these metrics mostly depend on variety of the completeness condition.
For our consistency score metric, most previous methods perform well on consistency score when the base classiﬁer predicts the ground truth.
However, their performance on samples that the base classiﬁer predicts incorrectly is not as good as ours.
Our method shows that soundness can be improved without much affecting performance on other known metrics.
Note that our metric assumes that testing methods take effort to generate different masks for each possible label.
Saliency methods like Phang et al.
(2020) actually inherent advantage on our metric by not doing that.
Considering that our method bears similarity to prior work but outperforms them on consistency score even with arguable disadvantage, it implies there is room for improvement on soundness for previous works.
Table 1 also shows that upscaling improves the consistency score.
As mentioned in Section 4, upscaling was used in previous work to avoid “artifacts” and produce better looking masks.
Our experiment shows a new explanation for the upscaling that it improves the soundness of the masking.
To verify the connection between completeness/soundness and consistency score, we compute the completeness and soundness (with (cid:15) = 10 − 3 ) for test samples, and report the average in Table 2.
It shows completeness/soundness and consistency score are highly correlated.
Even though saliency methods like Phang et al.
(2020) have inherent advantage in soundness, our method still achieves the best soundness.
Our method also gains better completeness, which is likely contributed by maximizing the probability assigned to the given label instead of matching the model probability.
[Table]
[Table]
6. E FFECT OF TV REGULARIZATION
In this subsection [Table] we test our theoretical prediction from Section 5 that TV penalty improves soundness.
We ﬁrst visually inspect the masks learned by our procedure as we increase the TV regularization strength 8 in Figure 4.
Masked images at 10 % sparsity are depicted in odd columns and the full masks are shown in even columns.
We plot the AUC insertion metric with blur evaluated on the non-sparsely masked images on the y axis of each plot.
We ﬁnd that as we increase the TV regularization strength, the model can still ﬁnd a saliency map for the correct label with high AUC score, however the AUC score for the mask learned to ﬁt the second most conﬁdent label drops signiﬁcantly.
This, in conjunction with Lemma 3.5, suggests that the TV regularization method has slightly worse completeness but much higher soundness .
Figure 3 plots the model output probability for CIFAR-10 as more pixels from the original image are retained using the mask.
Our high level ﬁnding is again that adding TV penalty and upsampling signiﬁcantly aid soundness, while only slightly hurting completeness.
To formally justify the effect of TV penalty, we measure our consistency score of different level of TV penalty on CIFAR-10.
To further challenge the completeness and soundness, the salient sets of our consistency score follows the Insertion game but only takes sets of size 0 .
2 d to 0 .
6 d ( d is the number of pixels).
Table 3 shows the result.
C ONCLUSIONS
Saliency explanations of ML models has proved nebulous and generated many controversies.
By taking rooted in intrinsic deﬁnitions such as completeness/soundness and consistency score, this paper has tried to provide greater rigor to the intrinsic approaches to saliency.
Other new contributions include clarifying the role of TV regularization (it hurts completeness slightly but greatly improves soundness); extensive experimental evaluations that bring new understanding using consistency score; and a simple saliency method for producing mask-based explanations whose performance is broadly competitive with good existing methods, and sometimes better.
The soundness notion is clearly useful for localization of objects in the image, and for classiﬁcation in the wild –where multiple objects appear over the image.
This was hinted in our evaluation on a prior small dataset (see Appendix B).
It may have other applications to handling distribution shift and other types of robustness, but such explorations are left for future work.
A I NTUITIONS AND I MPLEMENTATIONS OF P ROCEDURES TO F IND M ASKING E XPLANATIONS
As introduced in Section 3.3, for evaluation we may interest in random binary masks due to its connection to AUC, but in our method for ﬁnding masking explanations we only focus on deterministic masks.
Given a network f , image x ∈ R c × hw and class a , we wish to ﬁnd a binary mask
M ∈ { 0 , 1 } hw such that when the part of x on M is superimposed onto a distractor ¯ x ∼ X as ˜ x
= M (cid:12) x
+ (1 − M ) (cid:12) ¯ x , the output probability of the model f
(˜ x, a ) is high for the class a .
This corresponds to the case where ∆ is a singleton that assigns probability 1 to M and Γ( x, M ) is the distribution of ˜ x 9 .
As in Section 3.1 we compute the average probability assigned to class a over the sampling of the distractor ¯ x , i.e. we are interested in making E ¯ x ∼X [ f (˜ x, a )] high.
To avoid the hard problem of optimizing over the hypercube { 0 , 1 } hw , a typical strategy (also employed in prior work) is to relax the domain of M to be [0 , 1] hw .
Since we do not wish to learn masks of very large size, a (cid:96) 1 norm penalty on M (corresponding to size of the mask), leading to the following natural objective function
[Equation]
However most masking-based methods employ additional “tricks” in order to avoid “artifacts” in the produced saliency maps, like Total-Variation (TV) penalty (Fong & Vedaldi, 2017) and upscaling of the mask from a lower resolution one (Petsiuk et al., 2018).
We also employ the same strategy by learning a low-resolution mask at scale s , M ∈ R hw/s 2 , to minimize the following
[Equation]
where M × s ∈ R hw is obtained by upscaling M by a factor of s ∈ { 1 , 4 } via bilinear interpolation.
While the motivation cited for these “trick” is to avoid artifacts, it is not clear whether artifacts are a bad thing, since they might be relevant to the net’s decision-making.
Indeed, we show that while
TV penalty or upscaling does produce better looking masks, they lead to a drop in the completeness metric .
However we show that adding such tricks leads to signiﬁcant improvement in the soundness metric, thus providing a novel justiﬁcation for the use of such tricks, beyond just the heuristic argument of getting rid of artifacts.
In Section 5 we also provide theoretical justiﬁcation for why TV penalty can help with soundness, even for the simple case of linear predictors on non-image data.
We optimize the objective in Equation (4) by parametrizing M as a sigmoid of real valued weights
W ∈ R hw/s 2 , i.e. M = σ ( W ) , and run Adam (Kingma & Ba, 2014) optimizer for 2000 steps with learning rate 0 .
and by sampling 10 distractor images at every step, for different values of λ TV and upscaling factor s .
We report the effect of λ TV qualitatively in Figure 4 and quantitatively on various intrinsic saliency metrics in Table 10.
B P RACTICAL B ENEFITS OF S OUNDNESS FOR I MAGES OF M ULTIPLE O BJECTS
Images may have multiple plausible labels.
In Figure 2, images that previously used in Gu et al.can have both elephants and zebras present, but it may not be always clear from the model output if there is such a case, since the model can be much more conﬁdent on one label, e.g., elephant, than one would expect it to be.
For this reason, ﬁnding masking explanations validating other labels, e.g., zebra, could provide more information on how the model makes the prediction.
C E XPERIMENTAL D ETAILS AND A DDITIONAL E XPERIMENTS
In this section we expand upon the experiments in Section 6 and complement them with more experiments on the ImageNet, CIFAR-10 and CIFAR-100 datasets.
For each of the datasets we test the following:
• Visualization: For various values of TV regularization (and upscaling for ImageNet), we visualize the mask and also what part of the image a sparse version of the mask highlights.
We do so for masks learned for the correct label and also for the second most probable label as predicted by the model.
The common trend is that while TV regularization (and upscaling) make the masks more human interpretable, it also makes it harder to ﬁnd a good mask (partial assignment) for the incorrect label, thus improving soundness.
• AUC curve: We plot the output model probability for a masked input as more pixels from the original image are selected.
The 4 plots denote replacing remaining pixels with grey pixels or pixels from a random image, and masks to ﬁt the correct or incorrect label. Again
we ﬁnd the TV regularization and upsampling help with soundness
For mask M , if ¯ M ( p ) denotes the discrete mask with top p fraction of the pixels from M picked.
We plot E x (cid:2) E x (cid:48) ∼ Γ [ f ( ¯ M ( p ) (cid:12) x
+ (1 − ¯ M ( p ))
(cid:12) x (cid:48) , a )] (cid:3) v/
s p where Γ is either a random image or a grey image, and a is either the correct label for x or the second best label.
• Completeness/soundness
: We evaluate completeness and soundness metrics as deﬁned in Deﬁnition 3.2.
In particular, for every input x , we evaluate completeness for the correct label a and soundness for the incorrect (second best) label a (cid:48) .
For any mask M , we deﬁne AUC as AUC ( M ; ( x, a )) =
E p (cid:2) E x (cid:48) ∼ Γ [ f ( ¯ M ( p ) (cid:12) x
+ (1 − ¯ M ( p ))
(cid:12) x (cid:48) , a )] (cid:3) .
Com- pleteness and soundness are deﬁned as
[Equation]
where a is the correct label and a (cid:48) is the incorrect label and f ( x, a ) is the model probability for label a for input x .
δ and (cid:15) can be any reasonable constant to stabilize the value.
• Intrinsic metrics: We evaluate our masks on other intrinsic metrics from prior work, and compare to baseline saliency methods.
Our baselines include Gradient (cid:12) Input (Shrikumar et al., 2017), Smooth-Grad (Smilkov et al., 2017), Real Time Saliency (Dabkowski & Gal, 2017) (for ResNet-50 on ImageNet), and Random indicating a random Gaussian mask as a control.
We use Captum (Kokhlikyan et al., 2020) for Gradient (cid:12) Input and Smooth-Grad implementations and the original author code 11 for Real Time Saliency.
When calculating the Saliency Metric (SM) (Dabkowski & Gal, 2017)
we tune the threshold δ on a holdout set of size 100 with δ between 0 and 5 in increments of 0 . 2 as in prior work.
For the saliency method of Fong & Vedaldi (2017) that we only used on the Imagenette, we adapt the most popular implementation on GitHub 12 .
The implementation contains minor deviations from the original paper as described on its main page.
For Phang et al.
(2020), we used their best CA model pretrained and provided in original author code 13 .
C. CIFAR-10 E XPERIMENTS
We also run our method from Section 4 on the CIFAR-10 dataset using a pretrained ResNetarchitecture 14 .
For all experiments we learn a mask M ∈ R 32 × 32 , thus using a scaling factor of s = 1 (no upscaling).
We train masks for 1600 images that were correctly classiﬁed by the pretrained ResNet-164 using regularization parameter λ TV ∈ { 0 , 0 .
We use a (ﬁxed)
L1 regularization value of .001 for all masks.
We visualize the masks learned for the correct label in Figure 6a and in Figure 6b we visualize the same for the second best label predicted by the ResNet-164 model.
We also visualize the masks for all labels for some randomly picked images in Figure 7 to demonstrate the commonness of artifact, especially for the incorrect labels.
The AUC curves in Figure 5 suggest a similar trend to that of
ImageNet, adding TV regularization results in only a mild drop in completeness, but signiﬁcantly improves soundness.
Evaluation of our masks, compared to some gradient baselines, on intrinsic metrics can be found in Table 5.
We place a downarrow after the name of the metric to indicate a lower value is considered better and an uparrow when a higher value is considered better.
We evaluate on a randomly selected subset of 1000 data points where the model had correct topaccuracy.
We report the completeness and soundness results for CIFAR-10 in Table 4 for TV values in (0.0,0.001,0.01,0.1) calculated using a ResNet-164 model.
C. CIFAR-100 E XPERIMENTS
We run the same experiment for CIFAR-100 using the corresponding ResNet164 model.
We visualize the masks learned for the correct label in Figure 9a and in Figure 9b we visualize the same for the second best label predicted by the ResNet-164 model.
The AUC curves in Figure 13 suggest a similar trend to that of ImageNet, adding TV regularization results in only a mild drop in completeness, but signiﬁcantly improves soundness.
Evaluation of our masks, compared to some gradient baselines, on intrinsic metrics can be found in Table 7.
We place a downarrow after the name of the metric to indicate a lower value is considered better and an uparrow when a higher value is considered better.
We evaluate on a randomly selected subset of 1000 data points where the model had correct top 1 accuracy.
When calculating the saliency metric we tune the threshold δ on a holdout set of size 100 with δ between 0 and 5 in increments of 0 . 2 . [Table]
both our masks (learned with and without TV) have very good performance on the insertion metric. The deletion and saliency metrics are uninformative in this case, since all methods are as good (or worse) compared to a random mask. Gradient (cid:12) Input Our method
Our Method
Smooth-Grad
Random ( λ TV = 0 .
01 ) ( λ TV = 0 ) saliency
Deletion ↓ 0.
Insertion (blur) ↑ 0.
Insertion (grey) ↑ 0.
Saliency Metric ↓ 0.
We report the completeness and soundness results for CIFAR-100 in Table 6 for TV values in (0.0,0.001,0.01,0.1) calculated using a ResNet-164 model.
[Figure]
[Table]
C. E XPERIMENTS ON I MAGE N ET C.3. R ES N ET -
In Figure 10a we depict the the masks for TV values in { 0 . 0 , 0 . 01 } for a ResNet-18 model on
ImageNet for the ground truth label and in Figure 10b we depict the same for the second best label.
We also experiment with the effect of upsampling (US) the mask, whereby we learn a mask of size (56,56) and upsample to size (224,224).
We use a ﬁxed L1 regularization value of 2e-5.
We depict our results on ImageNet and ResNet-18 in Table
[Figure]
[Table]
For the deletion metric, we note that most methods have comparable or worse performance than the random mask, which suggests that the metric does not give us much signal about the goodness of the saliency maps.
On the insertion metric, we ﬁnd that mask learned by not adding the TV penalty signiﬁcantly beats other methods.
The mask learned using TV penalty, on the other hand, has impressive performance on both the insertion AUC and saliency metric (SM).
Completeness and Soundness on ImageNet and ResNet-18 We report our results in Table 8 for
TV values in (0, 0.01) for both greying (Grey) and replacing with other image pixels (Noise).
Additionally, we investigate the effect of upsampling (US) where we derive a (56,56) and upsample by a factor of 4 to a (224 , 224) mask.
Effect of ensembling.
In order to investigate the effect of ensembling we plot maps in Figureas we vary the number of maps that are ensembled over as K ∈ { 1 , 2 , 4 } , where we learn multiple masks such that each of them are individually statistical assignments.
We do not upsample (using a scale of 1.0) and we use a ﬁxed L1 regularization of 2e-5 and a ﬁxed TV regularization of 0.0.
C.3. R ES N ET -
We present our results on ImageNet and ResNet-50 in Table 10.
Using the same pretrained ResNet50 model as Dabkowski & Gal (2017) lets us compare our method to their real-time saliency method [Figure]
Grey TV = 0. TV = 0. US TV = 0.
US TV = 0.
Correct label completeness ( C 1 ) 0. Second label soundness ( S 0 ) 0. Noise TV = 0.
TV = 0.
US TV = 0.
US TV = 0.
Correct label completeness ( C 1 ) 0.
Second label soundness ( S 0 ) 0. [Table] directly on intrinsic metrics. Dabkowski & Gal (2017) do not evaluate on insertion/deletion metrics, so we evaluate these using their pretrained mask model.
Inspired by (Adebayo et al., 2018) we randomize the last layer of a ResNet-18 network and visually inspect the resulting saliency maps in Figure 11. We ﬁnd that the maps appear less coherent than those of a pre-trained model.
We use a ﬁxed L1 regularization of 2e-5 and depict maps with and without upsampling (US) at TV values of (0 , 0 . 01) .
[Figure]
TV Gradient (cid:12) Input
Our method Our Method
Smooth-Grad Random ( λ TV = 0 .
01 ) ( λ TV = 0 ) saliency Deletion ↓ 0. Insertion (blur) ↑ 0.
[Table] [Table]
[Table]
[Table]
D A DDITIONAL B ACKGROUND INFORMATION D. S ALIENCY M ETHODS
We give a partial list of extant saliency methods here.
We broadly categorize explanations into three categories: Back-propagation based explanations, axiomatic methods [Table] and masking methods.
Backpropagation based explanations shape credit as it is propagated backwards through the neural network according to certain rules.
These approaches include Layerwise Relevance Propagation (Binder et al., 2016) which satisﬁes completeness, Rect-Grad which thresholds internal neuron activations (Kim et al., 2019), and DeepLIFT which satisﬁes the summation to delta rule.
Axiomatic methods.
Axiomatic methods decompose the ouput (typically the logit)
according to certain axioms like fairness in Shapley based methods SHAP (Lundberg & Lee, 2017a) and conceptSHAP (Yeh et al., 2020).
We also include gradient based approaches like Gradient ∂S∂x (Baehrens et al., 2010) which calculates the partial derivative of the logit with respect to the input.
Gradient (cid:12) Input (Shrikumar et al., 2017) ∂S∂x · x , which elementwise multiplies the gradient explanation by the input, and Grad-CAM (Selvaraju et al., 2019) which takes the gradient of the logit with respect to the feature map of the last convolutional unit of a DNN.
Smooth-Grad (Smilkov et al., 2017), which averages the Gradient (cid:12) Input explanation over several noisy copies of the input x + η , where η is some Gaussian.
The previous methods are intrinsic in the sense that they aim to explain the model decision.
The last category of saliency maps, namely masking methods, also aim to explain the model decision, but they frequently aim to do so in a way that is interpretable by a human.
Contrastive methods, such as contrastive layerwise propagation Gu et al.
(2018), also modify LRP by constructing class speciﬁc saliency maps, with the goal of object localization, i.e. in an image of an elephant and zebra, the saliency map for elephant should have high overlap with the elephant, and similarly for the corresponding map for zebra.
Masking Methods.
Masking Methods are often evaluated using a pointing game or WSOL metric, which measures overlap with human labeled bounding boxes or explanations.
These masking methods include techniques based on averaging over randomly sampled masks (Petsiuk et al., 2018), optimizing over meaningful mask perturbations (Fong & Vedaldi, 2017), and real time image saliency using a masking network (Dabkowski & Gal, 2017).
Pixels that have been removed from the image by the mask may be replaced by greying out, by Gaussian blurring as in Fong & Vedaldi (2017), or with inﬁllers such as CA-GAN (Yu et al., 2018) used in Chang et al.
(2018), or DFNet (Hong et al., 2019).
De Cao et al.
ﬁnd masks using differentiable masking.
Taghanaki et al.
(2019) introduce a method that results in more accurate localization of discriminatory regions via mutual information.
Pruning and information theory Khakzar et al.
(2019) improve attribution via pruning.
Schulz et al.
(2019) improve attribution by adding noise to intermediate feature maps.
Saliency and Boolean Logic.
Previous work has also drawn connections between saliency and notions in logic.
Ignatiev et al.
(2019b) relates saliency explanations and adversarial examples by a generalized form of hitting set duality.
Ignatiev et al.
(2019a) develops a constraint-agnostic solution for computing explanations for any ML model.
Macdonald et al.
(2019) develop a rate distortion explanation for saliency maps and prove a hardness result.
Mu & Andreas (2020) ﬁnd a procedure for explaining neurons by identifying compositional logical concepts.
Zhou et al.
(2018) describe network dissection, which provides labels for the neurons of the hidden representations.
We are unaware of frameworks like Section 3.
Arguments about saliency.
For discussion including pro/cons of various methods some starting points are Seo et al.
Fryer et al.
Gu et al.
Sundararajan & Najmi (2020).
Phang et al.
We describe separately the masking procedure used by Phang et al.
They begin by taking a pretrained model on ImageNet.
The masker has access to the internal representations of the pre-trained model, and tries to maximize masked in accuracy and masked out entropy.
They do not provide the ground truth label to the masker.
D. S ALIENCY E VALUATION M ETHODS
Saliency evaluation methods attempt to evaluate the quality of a saliency map.
Many interpret the heatmap values as a priority order of saliency.
Extrinsic evaluation metrics include the WSOL metric, which aim to measure overlap of the saliency map with a human annotated bounding box and the Pointing Game metric proposed by Zhang et al.
(2018) in which a pixel count as a hit if it lies within a bounding box and a miss otherwise, and the metric is [Table] Hits + #Misses .
Other more intrinsic methods include early saliency evaluation techniques like MorF and LerF Samek et al.
(2016), which involve removing pixels either in the order of highest importance or lowest importance and observing the area of the resulting curve.
Insertion and Deletion Games of Petsiuk et al.
(2018) uses this too.
The deletion game measures the drop in class probability as important pixels are removed, while the insertion game measures the rise in class probability as important pixels are added.
(Our AUC discussion in Section 3 relates to this.)
Remove and Retrain (ROAR) is a saliency evaluation method proposed by Hooker et al.
Input features are ranked and then removed according to a saliency map.
A new model is trained on the modiﬁed training set, and a larger degradation in accuracy on the modiﬁed test set compared to the original model on the original test set is regarded as a better saliency method.
(NB: retraining makes this a non-intrinsic method.)
Previous work has also introduced datasets speciﬁcally designed to test saliency methods.
BAM Yang & Kim (2019) creates saliency maps by pasting object pixels from MSCOCO Lin et al.
(2014) into scene images from MiniPlaces Zhou et al.
The Saliency Metric proposed by
Dabkowski & Gal (2017) thresholds saliency values above some α chosen on a holdout set, ﬁnds the smallest bounding box containing these pixels, upsamples and measures the ratio of bounding box area to model accuracy on the cropped image, s ( a, p )
=
log(max( a, 0 . 05))
− log( p ) where a is the area of the bounding box and p is the class probability of the upsampled image.
Controversies.
There is extensive discussion of validity of saliency evaluation methods; e.g., Brunke et al.
(2020)Petsiuk et al.
D. S ALIENCY COMPUTATIONS AND UNDERLYING MEANINGS OF SALIENCY
For simplicity this discussion assumes the datapoints are images and the classiﬁer is a deep net.
The heatmap in the saliency method is trying to highlight the contribution of individual pixels to the ﬁnal answer.
This is analogous to how a human may highlight relevant portions of the image with a plan.
(Classic saliency methods in vision are inspired by studies of human cognition.)
Saliency methods operationalize this intuitive deﬁnition in different ways, and we try to roughly categorise these as follows.
Variational interpretation.
These interpret saliency in terms of effect on ﬁnal output due to change in a single pixel –captured either via partial derivative of output with respect to pixel value (i.e., effect of inﬁnitesimal change), or via change of output when this pixel is set to 0 or to a random (or ”gray”) value.
Examples include Gradient , Gradient (cid:12) Input Shrikumar et al.
Occlusion
Credit attribution guided by gradient.
These use the gradient to guide the assignment of saliency values.
The gradient is interpreted as propagating values from the output to the input layer, and the values are partitioned/recombined at internal nodes of the net following some conservation princi- ples.
A key goal is to ensure completeness , which means that the sum of the attributions equal the logit value.
Examples include LRP , DeepLIFT Shrikumar et al.
(2017), Rect-Grad Let a li be the activation of some node in layer l , and R l +1 i be the backpropagated gradient up to a li .
Rect-grad replaces the vanilla chain rule, R l =
[ a i > 0] with the rule that R li = 1 [ R l +1 i a i > τ ] for some threshold τ .
Hence, during a backward pass preference is given to nodes with large margin.
Ensembling on top of above two ideas.
Ensembling methods combine saliency estimates over multiple inputs an an attempt to reduce noise in the ﬁnal map.
Examples include Smooth-Grad ,
Occlusion based methods, etc.
We also include Shapley Values in this list.
The Shapley value aims to fairly distribute credit among a coalition of N players.
In the context of image saliency, each coordinate of the image input may be seen as a player, and the Shapley value computes (cid:80) S ⊆ N \
{ i } | S | !
( n −| S |− 1)!
n !
( v ( S ∪{ i } ) − v ( S )) .
It can be interpreted as the marginal contribution of player i, over all possible orderings of the coalition.
In this sense, it can be seen as an ensembling method, as it averages over all possible random permutations.
Analysis of saliency methods.
Previous work has analyzed ensembling methods like Smooth-grad, and found that it does not smooth the gradient Seo et al.
They conclude that Smooth-Grad does not make the gradient of the score function smooth.
Rather Smooth-grad is approximately the sum of a standard saliency map and higher order terms and the standard deviation of the Gaussian noise.
It has also been found that Shapley values, despite having a uniqueness result, can differ in the way they depend on the model, data, etc Sundararajan & Najmi (2020).
Fryer et al.
(2021) highlight several nuances that should be taken into account when considering Shapley values.
They introduce Shapley values as averaging over submodels, and note that ”the performance of a feature across all submodels may not be indicative of the particular performance of that feature in the set of optimal submodels.”.
They provide speciﬁc cases where satisfying the axioms of Shapley values works against the goal of feature selection.
E C LARIFYING BENEFIT OF TV REGULARIZATION
This section illustrates Deﬁnitions 3.1 and 3.2 using linear classiﬁers.
It also shows how TV regularizers help ensure soundness even in this setting.
Let S be a dataset of labeled data ( x , y ) where the inputs are of unit norm and labels are binary, i.e., (cid:107) x (cid:107)
= 1 , y ∈ {± 1 } .
The model in question is a linear classiﬁer f ( x ) :=
sgn( (cid:104) w , x (cid:105) ) parameterized by the weight vector w ∈ S d − 1 , and it achieves the perfect accuracy on the set S with a margin γ := min ( x ,y ) ∈S y (cid:104) w , x (cid:105) > 0 .
We assume that the coordinates of x and w are uniformly bounded by 10 √ d , i.e., (cid:107) x (cid:107) ∞ ≤ 10 √ d , (cid:107) w (cid:107) ∞ ≤ 10 √ d ( 10 can be changed to any other constant).
Let Γ be the input modiﬁcation process that sets all non-salient pixels to 0 .
We are interested in masking explanations with deterministic salient set, i.e., ∆ assigns probability 1 to some salient set
S .
According to Section 3.1, a masking explanation validates label a if E ˜ x ∼ Γ( x, ∆)
[ 1 [ f (˜ x )= a ] ] is high.
A simple calculation shows that this expectation equals to[ a (cid:80) i ∈ S w i x i > 0] , and thus the goal is to ﬁnd S so that a (cid:80) i ∈ S w
i x i > 0 .
As we do not consider the full salient set informative, we are interested in masking explanations with size constraint | S |
=
L for some 1 ≤ L ≤ d .
There is a simple saliency method that achieves this goal: Given an input x and a label a ∈ {± 1 } , sort the coordinates according to aw i x i and take the highest L coordinates as the salient set S .
It is easy to see that this method always produces S with a (cid:80) i ∈
S
w i x i > 0 .
Letting a = y proves the completeness.
However, this method does not satisfy soundness: a salient set S with a (cid:80) i ∈
S
w i x i > 0 can also be found for a (cid:54) =
y !
Now we see how the TV constraint helps to ensure soundness (with good probability).
A vector can be seen as a 1D image, and the TV of a salient set S can be deﬁned by TV( S ) :
= (cid:80) d −i =1 (cid:12)(cid:12) 1 [ i ∈ S ] − 1 [ i +1 ∈ S ] (cid:12)(cid:12) .
For simplicity, we consider salient sets with TV at most 2 .
This means
S is just an interval.
Given the size and TV constraints | S | =
L , TV( S ) ≤ 2 , it is easy to come out with the following saliency method: search over all the intervals of length L and if an interval
S satisﬁes a (cid:80) i ∈
S
w i x i > 0 , return it as the salient set.
Fortunately, this method does satisfy both completeness and soundness, which is justiﬁed by the following theorem.
Theorem E.1.
For any ( x , y ) ∈ S , if we shufﬂe the coordinates of w and those of x according to the same random permutation, then
For L = Ω( 1 γ 2 log 1 δ ) , with probability 1 − δ , there is an interval of length L that validates y ; 2.
For L = Ω( 1 γ 2 log dδ ) , with probability 1 − δ , no interval of length L can validate − y .
Proof.
Let S be any ﬁxed interval of length L , then the distribution of (cid:80) i ∈ S w i
x
i is identical to the distribution of the sum of L i.i.d.
random variables drawn from { w 1 x 1 , . .
w d x d } without replacement.
Note that dyw 1 x 1 , . .
dyw d x d are d numbers with mean γ , and their absolute values are bounded by 10
= O (1) .
By Chernoff bound,
[Equation]
Set (cid:15)
= γ ensures that y (cid:80) i ∈ S
w i x i > 0 with probability e − O
( γ 2 | S | ) .
We can ﬁx any interval S to prove Item 1.
Taking union bounds over all intervals of length L , the probability of existing an interval of length L that validates − y should be no greater than (cid:80) | S
|
=
L e − O ( (cid:15) 2 L ) ≤ d 2 e − O
( γ 2 L ) , which proves Item 2.
This shows that such masking explanations make sense to humans: if the model predicts y , then we can ﬁnd an interval of length ˜Ω(1 /γ 2 ) so that computing the inner product only in that interval leads to the same prediction; otherwise if the model does not predict y , such interval cannot be found.
Thus it is sufﬁcient to convince humans that the model predicts y by only revealing the existence of such interval and the coordinate values in it.
Although the example given in this section is simple, the conceptual message is clear: saliency methods may not guarantee soundness in itself, but adding regularity constraints such as TV can mitigate this soundness issue effectively.