New Definitions and Evaluations for Saliency Methods: Staying Intrinsic, Complete and Sound
Abstract
Saliency methods compute heat maps that highlight portions of an input that weremost important for the label assigned to it by a deep net.
Evaluations of saliencymethods convert this heat map into a new masked input by retaining the k highestranked pixels of the original input and replacing the rest with “uninformative”pixels, and checking if the net’s output is mostly unchanged.
This is usually seenas an explanation of the output, but the current paper highlights reasons why thisinference of causality may be suspect.
Inspired by logic concepts of completeness& soundness , it observes that the above type of evaluation focuses on completenessof the explanation, but ignores soundness.
New evaluation metrics are introducedto capture both notions, while staying in an intrinsic framework—i.e., using thedataset and the net, but no separately trained nets, human evaluations, etc.
Asimple saliency method is described that matches or outperforms prior methodsin the evaluations.
Experiments also suggest new intrinsic justifications, based onsoundness, for popular heuristic tricks such as TV regularization and upsampling.
Introduction
Saliency methods try to understand why the deep net gave a certain answer on a particular input, andare an important component of explainability, fairness, robustness, etc.
in deep learning.
This paperrestricts attention to the (large) class of saliency methods that return an importance score for eachcoordinate of the input —often visualized as a heat map— which captures the coordinate’s importanceto the final decision.
1 Early saliency methods used an axiomatic system of “credit attribution” toindividual input coordinates, using backpropagation-like methods [5, 34] and cooperative gametheory, such as Shapley values [26, 41].
See Section 2 and Samek et al.
[32] for discussion ofstrengths and limitations of such methods.
More recent methods try to find heat maps that are largelyconcentrated on a smaller set of pixels/coordinates and are discussed further below.
There exists an ecosystem of evaluation metrics to evaluate saliency methods.
Evaluations can beextrinsic , involving human evaluation [1], and comparison to certain ground truth explanations [43].
We restrict attention to intrinsic evaluations, which use computations involving the net itself andthe heat/saliency map — but no human evaluations or evaluations that involve training new deepnets.
Popular intrinsic evaluations include saliency metric [8] and insertion & deletion metrics [29].
A frequent idea in intrinsic evaluations (see Section 3 for references) is to create a new composite input— or sequence of such inputs — using the heat map and the original input, and to evaluate the originalnet on this composite input.
For example, if M is a binary vector with 1 ’s in the k coordinates withthe highest values in the heat map, then x ⊙ M (with ⊙ denoting coordinate-wise multiplication) canbe viewed as a masked input where only k of the original coordinates of x remain.
It is customary to explanations exist for masks generated for incorrect labels; more examples can be found in Figure 13 in Appendix
The base model outputs the correct label on the original image (ship and bird resp.)
with probability at least 0.99, and assigns probability at most 10 − 5 for the incorrect label (cat and frog resp.).
With the generated masks, the AUC metric (defined in Section 3.1) for the correct label remains high (around 0.94 and 0.90), which corresponds to completeness , but AUC metric for the incorrect label rises tremendously (around 0.18 for cat mask, 0.71 for frog mask.)
This suggests violation of soundness .
replace the zeroes in this masked input with “uninformative” values, which we refer to as gray pixels.
This masked input is fed into the original net 2 to check if the net outputs the same label as on the fullinput — and if so one should conclude that the unmasked coordinates of x were salient to the net’soutput.
Not surprisingly, recent methods [10, 8, 30] for finding heatmaps use an objective that tries todirectly optimize performance on such evaluations, greatly outperforming older axiom-based methods.
Should we trust such mask-based explanations ?
At first sight, this question may appear naive.
In theabove-described masked input x ⊙ M , some k pixels were copied from the original image and theremaining pixels were made “gray.”
If the original net outputs essentially the same label as it did onthe full image, does this not prove that the portion that came from the mask M must have caused thenet’s output on the full image?
The answer is “No”, because the positions of the gray pixels can carrysome signal.
Figure 1 shows that in standard datasets, it is possible to create masks out of most inputssuch that the masked input can cause the net to assign high probability for a different label than in itsoriginal prediction.
Prior works referred to this phenomenon as “mask artifacts” and suggested usingtotal variation (TV) regularization to mitigate this issue.
But Figure 1 shows TV regularization is nota full solution to masks that cause the net to (incorrectly) flip the answer.
Soundness needed.
The masked input method is trying to prove that a certain portion of the image“caused” the net’s output.
Usually logical reasoning must display both completeness (i.e., all correctstatements are provable) and soundness (incorrect statements cannot be proved).
Checking that thenet’s output is essentially unchanged with the masked input is akin to verifying completeness .
Butthis by itself should not be convincing because, as mentioned, it is possible to use the method to find adifferent mask to justify another (i.e., wrong) label.
Soundness would require verifying that the samesaliency method cannot be used to produce masked inputs that make the net output a different label.
Together, completeness and soundness ensure that the evaluation of the masked inputs produced usingvarious labels approximately tracks the model’s probability of assigning the label (see Section 3.2).
Main Contributions of this paper:
• Section 3.2 draws connections to logical proof systems and formalizes completeness and soundness in the context of saliency methods.
These are intrinsic evaluations of saliency without appealingto human judgements (e.g., whether or not machine-generated masks contain artifacts).
Our notionof completeness and soundness requires saliency methods to output a map for every label andnot just the model
prediction.•
Section 4 revisits methods that learn masks through optimization.
Incorporating soundness, wepropose a simple optimization method that maximizes the probability for any given label 3 , ratherthan only the label deemed most likely by the net.
A crucial novelty of this method is the pixelreplacement strategy during training: non-salient pixels are replaced with pixels of a randomimage, instead of using gray pixels, blurring or counterfactuals models.
• Experiments (in Section 6) estimate completeness and soundness scores for various methods.
Oursimple method does well on completeness and soundness as one would expect, but the slight surprise is that it performs comparably to prior masked-based methods on earlier saliency evaluationsas well, suggesting that soundness can be achieved without paying much in completeness
• Heuristic choices in existing mask-based methods such as TV regularization and upsampling,which were hitherto justified extrinsically (i
e., they make masks look natural to humans), receivean intrinsic and precise justification: they improve soundness.
A theoretical result in Section 5complements this experimental finding by showing that TV regularization helps saliency methodseven in a simple linear classification setting, not just for vision data.
Prior approaches
We delegate a more thorough description of prior work to Appendix E, but mention some commonsaliency and saliency evaluation methods here.
Saliency methods aim to explain a model’s decisionabout an input.
Saliency evaluation methods aim to evaluate the goodness of a saliency method.
Saliency methods include backpropagation based approaches such as Gradient ⊙ Input [36], iGOS++[19], Robust Perturbations [21], LRP [5], GradCAM [34], Smooth-Grad [37].
Another line of workis masking methods which include techniques based on averaging over randomly sampled masks [29],optimizing over meaningful mask perturbations [10], and real time image saliency using a maskingnetwork [8].
Pixels that have been removed from the image by the mask may be replaced by greyingout, by Gaussian blurring as in [10], or with infillers such as CA-GAN [42] used in [7, 30], or DFNet[13].
De Cao et al.
[9] find masks using differentiable masking.
Boolean logic is another approachfor saliency methods [17, 16, 27, 28, 45].
Arguments about saliency.
Discussions about the methods and the meaning of saliency appear,among others, in [35, 11, 12, 38].
Some reveal situations where Shapley axioms work against featureselection or where Shapley values may be calculated in conflicting ways [11, 38].
Others questionefficacy of saliency methods that add noise [35], making explanations non class discriminative [12].
Saliency evaluation methods.
Extrinsic evaluation metrics include the WSOL metric, and Pointing
Game metric proposed by Zhang et al.
[43] and ROAR [14].
Other more intrinsic methods includeearly saliency evaluation techniques like MorF and LerF [31], Insertion and Deletion gameproposed by Petsiuk et al.
[29], which involve either inserting pixels in order of most importance ordeleting pixels in order of most importance.
BAM [40] creates saliency maps by pasting object pixelsfrom MSCOCO [24].
The Saliency Metric proposed by Dabkowski and Gal [8] thresholds saliencyvalues above some α chosen on a holdout set, finds the smallest bounding box containing these pixels,upsamples and measures the ratio of bounding box area to model accuracy on the cropped image,s ( a, p )
=
log(max( a, 0 . 05))
− log( p ) where a is the area of the bounding box and p is the classprobability of the upsampled image.
Discussions.
Brunke et al.
[6] show that perturbation methods are sensitive to baseline and Petsiuket al.
[29] point out that human centric explanations (based on bounding boxes) may not reveal whythe model made a certain decision.
Our notion of intrinsic saliency method outputs the mask butintroduces soundness to validate it.
Masking explanations and completeness/soundness
The goal of a saliency method is produce an explanations for the outputs of a given deep net f .
We think of a saliency method (Definition 3.1) producing explanations – a heat map in this case– as providing a proof of the statement f outputs the label a for input x .
Inspired by logical proofsystems, we propose that intrinsic evaluations for saliency methods should test for both completenessand soundness, and the rest of the section provides definitions for these notions.
3.1 Saliency methods and AUC metric
For a net f , let f ( x, a ) denote the (post-softmax) output probability of the net on input x for label a .
We are interested in saliency methods that output a heatmap with a scalar per coordinate of the input.
Definition 3.1 (Saliency Method) .
A saliency method M is an algorithm that takes 1) a deep net f ,2) an input x , and 3) a label a and produces a heatmap m =
M ( f, x, a ) ∈ R dim ( x ) .
Note that in the above definition, M can access any part of f including intermediate layers andgradients.
Examples of methods that can be interpreted as returning heatmaps are mask-basedmethods [10, 8, 7, 2, 30] that output heatmaps in [0 , 1] dim ( x ) , backpropagation-like methods [5, 34],
Shapely values [26, 41], and Gradient ⊙ input.
Given such a saliency method, a important question is how to evaluate the quality of the “explanations”or heat maps it produces.
A common evaluation idea is to use the behavior of the net f on a “masked”input — only the top few pixels based on the heat map in question are retained, while the othersare hidden appropriately.
Inspired by this idea we define evaluation metrics for saliency methods.
We abstract the step of retaining the top few pixels into an input modification process , defined below.
Definition 3.2 (Input Modification Process Γ ) .
Let Γ be a potentially randomized procedure thattakes an input x and a heatmap m and generates a modified input ˜ x . i.e. ˜ x ∼ Γ( x, m ) .
This input modification process defines a base metric that is the starting point of the completenessand soundness metrics that are defined later.
Definition 3.3 (Base Metric) .
Let x be an input (image), a a label, m a heatmap and Γ an inputmodification process.
Then, a base metric is a function g ( x, a, m ) =
E ˜ x ∼ Γ( x, m )
[ f (˜ x, a )] , where fis the neural net of interest 4 .
The base metric measures the expected output of f acting on modified inputs ˜ x and label a anddifferent choices for Γ lead to different base metrics.
Some examples of Γ that stay in the intrinsicframework are: graying out the pixels outside some subset S of m , replacing them by pixels froma Gaussian blurring of x [10].
Input modification through Γ also shows up in saliency methods,including our proposed method in Section 4; we discuss prior choices of Γ in that section.
For ourevaluation metrics, however, we choose Γ that grays out remaining pixels.
The popular Area-Under-the-Curve (AUC) evaluation metrics [29] of saliency methods can, in fact,be reinterpreted in terms of our base metric.
We state the insertion game and its AUC below, since itforms the basis of our subsequent completeness and soundness metrics, and then define the AUCbase metric as an instantiation of Definition 3.3.
Definition 3.4 (AUC of insertion game) .
For s = 1 to dim ( x ) take the top s pixels as per m , andplot the probability f ( x, a ) given by model to label a on the input x , where the top s pixels of x areretained and remaining pixels are assigned a default gray value.
Return the area-under-curve.
Definition 3.5 (AUC metric) .
We denote by g AUC ( x, a, m ) the base metric when a modified input˜ x ∼ Γ( x, m ) is obtained as follows:
• Sample s ∼ { 1 , . .
, dim ( x ) } and pick S to be top s pixels as per m .
• Set ˜ x
[ S ] = x [ S ] and ˜ x
[ ¯ S ]
= x gray [ ¯ S ]
where x gray is a “default” input with gray values.
Firstly we note that g AUC is indeed equivalent to the AUC insertion game from Definition 3.4.
Whilemany choices of base metrics can be used, we use this AUC metric for subsequent definitions since itconfers two benefits, 1) it allows for fair comparison between methods since all methods are evaluatedat the same sparsity level, 2) it considers the effect of multiple sparsity levels instead of picking anarbitrary one.
We are now ready to present the formalization of completeness and soundness.
3.2 Completeness and soundness
Just as for proof systems, we would like a saliency method to be logically complete and sound,i.e. it should be able to justify the label output by the net, but no other label 5 .
The statements to be“proved” are of the form “does f think x has label a ?” .
We use the model output probability f
( x, a ) as fractional truth value of the “statement” ( x, a ) .
For a saliency map m that a saliency method Mgenerates, we interpret the AUC metric g AUC ( x, a, m ) as the evaluation of truth value of the “proof”m , as adjudged by the base metric g AUC .
Thus in this context, completeness for M would entailthat whenever f ( x, a ) is large (i.e. the statement is sufficiently true), g AUC ( x, a, m ) should be large(a sufficient good proof can, and is, generated).
Soundness, on the other hand, would entail thatg AUC ( x, a, m ) should be small whenever f ( x, a ) is small.
Thus we would like g AUC ( x, a, m ) totrack the value of f ( x, a ) through a lower bound and upper bound as follows:
Definition 3.6 (Completeness and Soundness) .
Consider a saliency method M .
For α, β ≤ 1 ,the method M is α -complete on f, x, a if the heatmap m =
M ( f, x, a ) it produces satisfiesg AUC ( x, a, m ) ≥ α f ( x, a ) , while M is β -sound on f, x, a if g AUC ( x, a, m ) ≤ 1 β f ( x, a ) .
Conversely the completeness and soundness scores for a heap map m on ( x, a ) are defined as:
[Equation]
Notice, α -completeness means that if the model outputs a high probability for label a , then theprobability for label a after seeing only the coordinates in the salient sets is also high.
Whileβ -soundness means that this probability is not too high.
Checking both conditions verifies if the AUCscore g AUC ( x, a, m ) of a map m is in the interval (cid:104) αf ( x, a ) , 1 β f ( x, a ) (cid:105) .
We want α, β as closeto 1 as possible.
This notion is different from the usual AUC insertion game, which just requiresg AUC ( x, ˆ y, m ) to be as large as possible for the model prediction ˆ y = arg max a f ( x, a ) .
Instead ourdefinition requires the g AUC ( x, a, m ) to roughly match the model prediction f ( x, a ) , for every pair( x, a ) .
We formalize this notion through worst case completeness and soundness metrics as follows:
Definition 3.7 (Worst case completeness and soundness) .
For a saliency method M , these metrics aredefined by taking the worst case completeness/soundness over labels a for a given x that is sampledfrom the underlying input distribution.
More precisely, α ( M ) =
E x [min a α ( x, a )] , β ( M ) =
183 E x
[min a α ( x, a )]
These metrics implicitly requires a saliency method to output a meaningful map for every label (orat least all labels assigned non-negligible probabilities by the net) and not just for the top modelprediction.
The thresholds ϵ 1 , ϵ 2 in Equation (1) help in ignoring the effect of very small probabilities— for instance we can ignore any label for completeness if f ( x, a ) < ϵ 1 .
Illustration of completeness and soundness on an ImageNet example:
See Figure 2.
Forthis image x , the model assigns reasonably high probabilities to labels a 1 = “desk” and a 2 =“desktop computer”, with f ( x, a 1 ) ≈ 0 .
67 and f ( x, a 2 ) ≈ 0 .
We compare the maps computed bythree methods — M 1 : our mask learning procedure from Section 4 with TV regularization 0.01, M 2 :our method with TV of 0.001, and M 3 : the mask learned by Phang et al.
The correspondingmaps generated are denoted by m i , i ∈ { 1 , 2 , 3 } .
For this discussion we only consider the effect oflabels a 1 and a 2 on completeness and soundness, and we assume the thresholds satisfy {+ϵ 
+} = ϵ 2 = 0 .
Completeness.
For the label a 1 , the AUC scores g AUC ( x, a 1 , m i ) for the three methods are roughly0.65, 0.70 and 0.43 respectively.
Thus all three maps can “certify” the label a 1 well enough, withm 2 doing the best job.
Consequently the completeness scores (Equation (1)) for m 1 on the pair( x, a 1 ) is α 1 ( x, a 1 ) = min { 0 .
67 , 1 } ≈ 0 . 97 ; similarly α 2 ( x, a 1 ) = 1 and α 3 ( x, a 1 ) = 0 .
67 ≈ 0 .
Similarly for label a 2 that is assigned a probability of 0.13, the completeness scores α i ( x, a 2 ) are 1,1, and 0.58.
Thus the worst case completeness from Definition 3.7, i.e. min j ∈{ 1 , 2 } α ( x, a j ) , for thethree methods are 0.97, 1 and 0.58.
Based on completeness M 2 seems like the best method; howeversoundness will paint a different picture.
Soundness.
Using the same AUC scores we compute the soundness scores, defined in Equation (1)as β i ( x, a )
= min { f ( x,a ) g AUC ( x,a, m i ) , 1 } .
This gives us soundness scores on a 1 of 1, 0.95 and 1.
The moreinteresting case is label a 2 , for which the model does not assign very high probability (0.13).
Here themask from M 2 still gets a very high AUC score of 0.37, leading to a soundness score β 2 ( x, a 2 ) ≈0 .
37 ≈ 0 .
M 1 on the other hand is not as overconfident in its explanation as M 2 , and getsβ 1 ( x, a 2 ) = 0 .
15 ≈ 0 . 89 .
The worst case soundness for three methods ends up being 0.89, 0.35 and 1.
This examples highlights how completeness and soundness together can give a more nuancedcomparisons of saliency maps, including understanding the benefit of TV regularization.
Experimentsin Section 6 do a more thorough comparison of many saliency methods on these and prior metrics.
Procedures for find masking explanations
We propose a very simple method to find saliency maps with good empirical completeness andsoundness scores.
Detailed intuitions and implementations appear in Appendix A.
Our method issimilar to prior work on mask-based saliency maps and is based on the idea of SSR (smallest sufficientregion) 7 .
In particular for an input x and label a , given a network f , the main goal is to learn a map (ormask) M ∈ { 0 , 1 } hw such that E ˜ x ∼ Γ( x,M ) [ − log( f (˜ x, a ))] is minimized, i.e. the probability that thenetwork assigns to a modified (or composite) input ˜ x is high.
The key difference from prior maskedbased methods is our choice of Γ , which retains the pixels of x corresponding to M , but replacesthe rest of the pixels with values from a randomly sampled image ¯ x from the training set 8 X , i.e.
[Equation]
Modifications procedures ( Γ ) that have been considered in prior work include: graying or blurring outthe remaining pixels [10], and using a conditional image generative model to fill in the remaining pixels [7, 2].
Replacing with random image pixels forces the mask-finding algorithm to solve a more difficult task of having the network predict a high confidence for the label, despite the presence of anotherimage.
This amounts to grafting salient pixels of x on top of a random image, reminiscent of BAMevaluations [40] for saliency methods.
We also note a methodological difference from Dabkowskiand Gal [8], Phang et al.
[30] in that we do not train a separate neural network to output the mask.
As is standard, we relax the domain of masks M from binary { 0 , 1 } hw to continuous [0 , 1] hwparametrized by a sigmoid.
We define a composite input, where the part of x on M is superimposedonto a distractor ¯ x ∼ X as ˜ x
= M ⊙ x + (1 − M ) ⊙ ¯ x .
We then have the following simple vanillaobjective, where the ℓ 1 norm penalty on M helps to reduce the size of masks. [Equation]
[Equation]
Promoting Soundness: Tricks used in prior mask-finding approaches were tested for their effecton soundness.
These include 1) Total-Variation (TV) penalty [10] and 2) upsampling of the maskfrom a lower resolution one [29] by learning a low-resolution mask at scale s , M ∈ [0 , 1] hw/s 2 .
Forexample, on ImageNet, s = 1 corresponds to learning a 224 × 224 mask, while s = 4 correspondsto learning a 56 × 56 mask and upsampling by a factor of 4 .
Incorporating 1) and 2) leads to thefollowing modified objective
[Equation]
where ˜ x
= M
[Equation]
s ⊙ x
[Equation]
and M × s ∈ R hw is obtained by upsampling M by a factor of s ∈ { 1 , 4 } via bilinear interpolation.
While the motivation cited for these two commonly employed “tricks” is to avoid artifacts [10], in ourintrinsic framework “artifacts” does not have an interpretation.
What looks like an artifact to a humanobserver may be relevant to the net’s decision-making.
Indeed, our experiments show that while TVpenalty or upsampling does produce better looking masks, they lead to a drop in the completenessmetric .
But they significantly improve soundness .
This provides an intrinsic justification for the useof such tricks, instead of extrinsic justifications related to human interpretability.
On benefit of TV regularization in saliency
Past justifications for TV regularization in saliency methods focused on its ability to make heatmapsin images that look more natural to humans.
But that is not an intrinsic justification.
In the intrinsicframework proposed in this paper, experiments in Section 6 show TV regularization helps improvesoundness of the saliency method.
The current section reinforces this intrinsic benefit of TV bysketching a simple example (details in Appendix F) showing how TV helps even in a simple linearmodel x
(cid:55)→ w ⊤ x on non-image input x ∈ R d with binary label y ∈ {± 1 } . [Figure]
Suppose that the classifier has margin at least γ > 0 , that is, y w ⊤ x = (cid:80) di
=1 yw i x i ≥ γ .
We focuson saliency methods that can return a binary heatmap m ∈ { 0 , 1 } d to validate the label, wherethe coordinates marked as 1 constitute a set S of saliency coordinates.
Let Γ( x, m ) be the inputmodification process that assigns 0 to all input coordinates outside the corresponding saliency set S .
Consider an input x such that w ⊤ x > 0 , i.e. y = 1 .
Firstly note that any set S of coordinates thatcontribute positively to w ⊤ x , i.e. g ( x, y, m ) :
= (cid:80) i ∈ S w i x i > 0 , will be able to certify the labely = 1 and thus would satisfy completeness .
However this might not be convincing to a user, sinceit is also possible to come up with a set S ′ that certifies the label y
= − 1 by picking only negativecoordinates of x .
This would not satisfy soundness as the original model only predicts y =
This issue of soundness can be solved by adding a regularization or constraint on the set of allowablemaps, in a way that ensures that it is easy to find a mask m for the correct label y , but not for − y .
Ideally, the constraints should be task specific, but for linear classification with no special structure,even a TV constraint after random permutation suffices.
We consider the constraint that salient sets Sshould be an interval; this corresponds to TV( m ) ≤ 2 .
The following result shows that if we furtherconstrain the length of intervals to be Θ( 1 γ 2 log d ) , then with high probability, we can find an intervalvalidating the model prediction y
but we are unable to do the same for − y
(Details in Appendix F.)
Theorem 5.1.
For x ∈ R d , y ∈ {± 1 } , after random shuffling of the coordinates, the following holdsfor any L 1 = Ω( 1 γ 2 log 1 δ ) , L 2 = Ω( 1 γ 2 log dδ ) , where probability is over the random shuffle:
(Completeness)
With probability 1 − δ , there is an interval of length L 1 that validates y ;2.
(Soundness)
With probability 1 − δ , no interval of length L can validate − y .
Experiments
In this section, we aim to 1) show TV penalty and upsampling have intrinsic justification that they significantly improve soundness at a small cost in completeness; 2) evaluate and compare prior saliencymethods, and our method with different parameter settings, on our completeness and soundnessmetrics.
Experiments are performed either on CIFAR-10 or Imagenette [15], a 10-class subset of ImageNet 9 .
We compute maps and evaluate metrics on 1000 randomly drawn images from original test set.
Additional experiments on various models and datasets (including ImageNet and CIFAR-100) appearin Appendix D, as do details of training procedure (beyond those in Section 4).
For all our completeness and soundness evaluations, we use thresholds of ϵ 1 = 0 .
01 and ϵ 2 = 0 .
001 (see Equation (1)).
6.1 Effect of TV penalty and upsampling
In this section, we study the effect of “tricks” like TV penalty and upsampling on the learned mapsand evaluation metrics.
Figure 3 shows how soundness and completeness of our masks changeswith TV regularization.
It plots the model output probability for CIFAR-10 as more pixels fromthe original image are retained using the mask.
Our high level finding is that adding TV penalty significantly aids soundness, while only slightly hurting completeness.
Exact evaluation of soundnessand completeness for different λ TV are listed in Table 2 in Appendix.
Relevant points of Figure 4 (which also contain other saliency methods) show effect of upsamplingand TV penalty on completeness and soundness.
We compare our method from Section 4 withdifferent settings on Imagenette.
Our method by default use s = 4 and λ TV = 0 .
By comparing“Ours” against “Ours( s = 1 )” and “Ours( TV = 0 . 001 )”, we observe that upsampling and TV penaltyleads to slightly worse completeness but significantly better soundness.
The effect of upsampling onprevious saliency evaluation metric may be seen in the relevant entries of Table 1, where it leads toslightly worse performance on the insertion, deletion, and saliency metrics.
6.2 Comparison to existing metrics and methods
We compared our procedure to other methods including Gradient ⊙ Input [36], iGOS++[19],
Dabkowski and Gal [8] Fong and Vedaldi [10] and Phang et al.
[30] on Imagenette.
We alsoinclude simple baselines like “Random” map (each entry is a random Gaussian), and “Centered
Gaussian” map (2-dimensional isotropic Gaussian distribution placed at the image center).
Besidescompleteness and soundness metrics, we evaluate methods on the Deletion Game and Insertion
Game metrics ( https://github.com/eclique/RISE ), and Saliency metric (SM) [8].
Detaileddescription of the different metrics can be found in the Appendix.
For our method, we learn a map by optimizing the objective function in Equation (4) withλ TV ∈ { 0 . 01 , 0 . 001 } and s ∈ { 1 , 4 } ; default settings are λ TV = 0 .
01 , s = 4 .
We also evaluate ourmethod when other pixels are replaced with gray rather than a random image, denoted by “Ours(gray)”.
For uniformity, we use the identical ResNet50 pretrained on ImageNet as the base classifierfor every saliency method.
We normalize the maps so that all values lie in [0 , 1] before use.
Completeness and soundness.
We compare different methods on the metrics from Definition 3.7,through the visualization in Figure 4 (left) and in Table 1.
We find that our method achievesbetter completeness and good soundness compared to other methods and baselines.
We posit thatcompleteness is better due to our improved composite input strategy from Equation (2), wherenon-salient pixels are filled with another image pixels as opposed to other strategies.
As an ablation,we see that “Ours (gray)” which uses gray replacement does not perform as well.
On the other hand, our soundness is good only for higher TV regularization λ TV = 0 .
01 and higherupsampling s = 4 , as discussed in Section 6.1.
Soundness is very bad (but completeness slightlybetter) if these “tricks” are not used.
We note that while Phang et al.
[30] does as well as our methodon soundness, it does so via “cheating” in an interesting way: they compute mask for the top label
[Table]
and return the same mask for all labels.
Since the method does not even try to generate masks forlabels with low probability [Table] it achieves good soundness for free.
To delve deeper, we employ thesame “cheating” with our method — return the same mask for all labels — and observe a largebump in soundness, without any drop in completeness.
While “cheating” is always possible (evenif unintentionally) for any metric, we hope that our results inspire saliency method designers tocompute maps for all labels.
In this case we find, in Figure 4 (right), that looking at worst casecompleteness of incorrect labels (all except model prediction) gives a measure of the “effort” amethod employs in generating masks for all labels with probability at least 0.01.
As evident methodsthat “cheat”, like Phang et al.
[30] and “Ours (cheating)”, achieve a bad “best effort” score.
Other metrics.
Table 1 suggests that our method can achieve comparable performances on mostexisting intrinsic metrics.
This suggests that good completeness and soundness scores are achievablewithout compromising on other measures.
Deletion metric is one where our method falls short, likelydue to our method not employing an SDR objective as in Dabkowski and Gal [8].
Although we notethat “Random” baseline performs quite well on the Deletion metric, which suggests that deletionmetric does not provide much signal on these datasets.
Further investigation on the good performanceof “Centered Gaussian” baseline can be found in Appendix C.
Our settings that works best on other metrics are those without upsampling ( s = 1 ) or lower TV( λ TV = 0 . 001 ).
Soundness is the only metric that strongly justifies the upsampling and TV “tricks”,thus verifying its utility.
Our method with s = 4 , λ TV = 0 .
01 demonstrates that good performancecan be obtained on all metrics simultaneously, with only little price to pay.
Conclusions
Saliency explanations of ML models have proved nebulous and generated much discussion.
Byavoiding extrinsic considerations (e.g., human interpretability) and sticking to intrinsic notions suchas completeness and soundness, this paper has tried to give a rigorous notion of saliency that isintrinsic to the deep net, while avoiding the noisiness in earlier intrinsic ideas that motivated methodslike Gradient ⊙ Input and Shapley Values.
Other new contributions include clarifying in the intrinsicview the role of TV regularization and other tricks (it hurts completeness slightly but greatly improvessoundness); a simple saliency method (Section 4) for producing mask-based explanations that wasdesigned solely with intrinsic considerations and yet has performance competitive with good existingmethods, and sometimes better (Table 1).
Our experiments suggest soundness provides a newdimension to evaluate saliency methods on, and that good performance on it can be achieved withoutcompromising on other metrics.
Note that soundness needs to be considered only while designingand evaluating the method, not at deployment.
However, by looking at maps for all labels instead of just the top label, soundness seems relevantfor object localization in images, and for classification in the wild – where multiple objects appearover the image.
This is hinted in our evaluation on a prior small dataset (see Appendix B).
It mayhelp with certain distribution shifts as well.