R EMOVING INPUT FEATURES VIA A GENERATIVE MODEL TO EXPLAIN THEIR ATTRIBUTIONS TO AN IMAGE CLASSIFIER ’ S DECISIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Interpretability methods often measure the contribution of an input feature to an image classiﬁer’s decisions by heuristically removing it via e.g. blurring, adding noise, or graying out, which often produce unrealistic, out-of-samples.
Instead, we propose to integrate a generative inpainter into three representative attribution methods to remove an input feature.
Compared to the original counterparts, our methods (1) generate more plausible counterfactual samples under the true data generating process; (2) are more robust to hyperparameter settings; and (3) localize objects more accurately.
Our ﬁndings were consistent across both ImageNet and Places365 datasets and two different pairs of classiﬁers and inpainters.
I NTRODUCTION
Explaining a classiﬁer’s outputs given a certain input is increasingly important, especially for lifecritical applications (Doshi-Velez & Kim, 2017).
A popular means for visually explaining an image classiﬁer’s decisions is an attribution map i.e. a heatmap that highlights the input pixels that are the evidence for and against the classiﬁcation outputs (Montavon et al., 2018).
To construct an attribution map, many methods approximate the attribution value of an input region by the classiﬁcation probability change when that region is absent i.e. removed from the image.
That is, most perturbation-based attribution methods implement the absence of an input feature by replacing it with (a) mean pixels; (b) random noise; or (c) blurred versions of the original content.
While removing an input feature to measure its attribution is a principle method in causal reasoning, the existing removal (i.e. perturbation) techniques often produce out-of-distribution images (Fig.
1b,d) i.e. a type of adversarial example, which (1) we found to produce heatmaps that are sensitive to hyperparameter settings; and (2) questions the correctness of the heatmaps (Adebayo et al., 2018).
To combat these two issues, we propose to harness a state-of-the-art generative inpainting model (hereafter, an inpainter) to remove features from an input image and ﬁll in with content that is plausible under the true data distribution.
We test our approach on three representative attribution methods of Sliding-Patch (SP) (Zeiler & Fergus, 2014), LIME (Ribeiro et al., 2016), and
Meaningful-Perturbation (MP) (Fong & Vedaldi, 2017) across two large-scale datasets of ImageNet (Russakovsky et al., 2015) and Places365 (Zhou et al., 2017).
For each dataset, we use a separate pair of pre-trained image classiﬁers and inpainters.
Our main ﬁndings are:
Blurring or graying out the object in a photo yields images that are up to 3 times more recognizable to classiﬁers and remain more similar to the original photo (via MS-SSIM and LPIPS) than the images whose objects were removed via inpainting (Sec. 4.2).
Attribution methods with an inpainter produces (1) more plausible perturbation samples; (2) attribution maps that perform on par or better 2 than their original counterparts on three existing benchmarks: object localization, insertion, and deletion (see Sec. 4.4); (3) explanations that are more robust to hyperparameter changes (Sec. 4.3).
[Figure]
h ea t m a p s s a m p l e s ( ou r s ) freight car
[Figure]
[Figure]
R ELATED WORK
Attribution methods can be categorized into two main classes: (1) white-box and (2) black-box.
White-box Given the access to the network architecture and parameters, attribution maps can be ) both the activations al., 2017), or d) the gradient times the input image (Shrikumar et al., 2017).
However, these heatmaps can be too noisy to be human-interpretable because the gradients in the pixel space are often local.
Importantly, some gradient-based attribution maps can be unfaithful explanations e.g. acting like edge detectors (Adebayo et al., 2018).
To make a gradient-based heatmap more robust and smooth, a number of methods essentially average out the resultant heatmaps across a large set of perturbed inputs that are created via (a) adding random noise to the input image (Smilkov et al., 2017; Fong & Vedaldi, 2017), (b) blurring or jittering the image (Fong & Vedaldi, 2017), or (c) linearly interpolating between the input and a reference “baseline” image (Sundararajan et al., 2017).
Black-box
In addition to the white-box setting, perturbation techniques are even more important in approximating attributions under the black-box setting i.e. when we do not have access to the network parameters.
Black-box methods often iteratively remove (i.e. occlude or perturb) an input region and take the average resultant classiﬁcation probability change to be the attribution value for that region.
While the idea is principle in causal reasoning, the physical interventions—taking an object out of a scene (revealing the content behind it) while keeping the rest of the scene intact—are impractical for most real-world applications.
Feature removal Therefore, the absence of an input region is often implemented by replacing it with (a) mean pixels (Zeiler & Fergus, 2014; Ribeiro et al., 2016); (b) random noise (Dabkowski &
Gal, 2017; Lundberg & Lee, 2017); or (c) blurred versions of the original content (Fong & Vedaldi, 2017).
However, these removal techniques often produce unrealistic, out-of-samples (Fig. 1), which raise huge concerns on the sensitivity and faithfulness of the explanations.
Do explanations become more robust and faithful if input features are removed via a learned, natural image prior?
Here, we systematically study that question across three representative attribution methods: two black-box (i.e. SP and LIME) and one white-box (i.e. MP).
We chose these three methods because they represent diverse approaches and perturb different types of input features: pixels (i.e. MP), superpixels (i.e. LIME); and square patches (i.e. SP).
Similar to us, Chang et al.
(2019) and Uzunova et al.
(2019) also harnessed image generative models to remove input features.
However, their ﬁndings were (1) both only within the MP framework; (2) either for grayscale, medical-image datasets (Uzunova et al., 2019) or based on unrealistic samples (see a comparison in Sec. 4.1).
Furthermore, their results were not relevant to our question of whether integrating an inpainter helps attribution maps become more robust to hyperparameters.
Note that Chang et al.
(2019) found negative results for integrating the inpainter whereas our methods improved both the robustness of attribution maps (Sec. 4.3) and their ability to localize objects (Sec. 4.4).
Counterfactual explanations A task that is related but not the same as ours is to generate a textual explanation for why an image is predicted as class c instead of a some other class c (cid:48)
(Anne Hendricks et al., 2018).
For visual explanations, Goyal et al.
(2019) proposed to ﬁnd a minimal input region such that when exchanged with another region in a reference image would change the classiﬁcation for the original image into some target class.
However, their counterfactual sample was generated by swapping patches between two images rather than by a generative model.
M ETHODS 3. D ATASETS AND N ETWORKS
Classiﬁers Our experiments were conducted separately with each of the two ResNet-50 image classiﬁers (He et al., 2016) that were pre-trained on the 1000-class ImageNet 2012 and Places365, respectively.
The two models were ofﬁcially released by the PyTorch (2019) model zoo and by the authors (CSAILVision, 2019), respectively.
Datasets We chose these two datasets because they are large-scale, natural-image sets and cover a wide range of images from object-centric (i.e. ImageNet) to scenery (i.e. Places365).
While state-of-the-art image synthesis has advanced rapidly, unconditionally inpainting a large free-form mask in an arbitrary photo remains challenging (Yu et al., 2018a).
Therefore, we ran our study on two subsets called ImageNet-S and Places365-S of the original validation sets of ImageNet and
Places365, respectively, after ﬁltering out semantically complex images.
That is, we ﬁltered out images that a YOLO-v3 object detector (Redmon & Farhadi, 2018) found more than one object.
For
ImageNet-S, we also ﬁltered out images with more than one ImageNet bounding boxes.
In total,
ImageNet-S and Places365-S contains 15,082 and 13,864 images respectively (see Figs.
S5 & S for example images).
Inpainter We used two TensorFlow DeepFill-v1 models pre-trained by Yu et al.
(2018b)
for ImageNet and Places365, respectively.
DeepFill-v1 takes as input a color image and a binary mask, both at resolution 256 × 256 , and outputs an inpainted image of the same size.
3. P ROBLEM FORMULATION
Let s : R D × D ×
→ R be an image classiﬁer that maps a square, color image x of spatial dimension
D × D onto a softmax probability of a target class.
An attribution map A ∈ [ − 1 , 1] D × D associates each input pixel x i to a scalar A i ∈ [ − 1 , 1] which indicates how much x i contributes for or against the prediction score s ( x ) .
We will describe below three different methods for generating attribution maps together with our three respective proposed variants which harness an inpainter.
3. S LIDING -P ATCH
SP Zeiler & Fergus (2014) proposed to slide a gray, occlusion patch across the image and record the probability changes as attribution values in corresponding locations in the heatmap.
That is, given a binary occlusion mask m ∈ { 0 , 1 } D × D (here, 1 ’s inside the patch region and 0 ’s otherwise) and a ﬁller image f ∈ R D × D × 3 , a perturbed image ¯ x ∈ R D × D × 3 (see Fig. 1b) is given by:
[Equation]
where (cid:12) denotes the Hadamard product and f is a zero image i.e. a gray image 3 before input pre-processing.
For every pixel x i , one can generate a perturbation sample ¯ x i (i.e. by setting the patch center at x i ) and compute the attribution value
A i = s ( x ) − s (¯ x i ) .
However, sliding the patch densely across the 224 × 224 input image of ResNet-50 is computationally expensive.
Therefore, we chose a 29 × 29 occlusion patch size with stride 3, which yields a smaller heatmap A (cid:48) of size 66 × 66 .
We bi-linearly upsampled A (cid:48) to the image size to create the full-resolution A .
We implemented SP from scratch in PyTorch following a MATLAB implementation (MathWorks, 2019).
SP-G Note that the stride, size, and color of the SP sliding patch are three hyperparameters that are often chosen heuristically, and varying them can change the ﬁnal heatmaps radically.
To ameliorate the sensitivity to hyperparameter choices, we propose a variant called SP-G by only replacing the gray ﬁller image of SP with the output image of an inpainter (described in Sec. 3.1) i.e. f = G ( m, x ) while keeping the rest of SP the same (Fig. 1b vs. c; top row).
That is, at every location of the sliding window, SP-G queries the inpainter for content to ﬁll in the window.
3. L OCAL I NTERPRETABLE M ODEL -A GNOSTIC E XPLANATIONS
LIME While SP occludes one square patch in the input at a time, LIME (Ribeiro et al., 2016) occludes a random-shaped region.
The algorithm ﬁrst segments the input image into S nonoverlapping superpixels (Achanta et al., 2012).
Then, we generate a perturbed image ¯ x by graying out a random set of superpixels among 2 S possible sets.
That is, LIME follows Eq.
1 where the pixel-wise mask m is derived from a random superpixel mask
m (cid:48) ∈ { 0 , 1 } S .
For each sample ¯ x i , we measure the output score s (¯ x i ) and evenly distribute it among all occluded superpixels in ¯ x i .
Each superpixel’s attribution is then inversely weighted by the L 2 distance (cid:107) x
− ¯ x i (cid:107) via an exponential kernel and then averaged out across N samples.
The resultant attribution a k of a superpixel k is ﬁnally assigned to all pixels in that group in the full-resolution heatmap A .
In practice, Ribeiro et al.
(2016) iteratively optimized for { a k } S via LASSO for 1000 steps to also maximize the number of zero attributions i.e. encouraging a simpler, sparse attribution map.
We used the implementation provided by the authors (Ribeiro, 2019) and their default hyperparameters of S = 50 and N = 1000 .
LIME-G While avoiding the bias given by the SP square patch, LIME perturbation samples remain unrealistic.
Therefore, we propose LIME-G, a variant of LIME, by only changing the gray image f to a synthesized image G ( m, x ) as in SP-G while keeping the rest of LIME unchanged.
3. M EANINGFUL P ERTURBATION
MP Because SP and LIME gray out patches and superpixels in the input image, they generate unrealistic counterfactual samples and produce coarse heatmaps.
To ameliorate these issues, Fong &
Vedaldi (2017) proposed the MP algorithm i.e. learning a minimal, continuous mask m ∈ [0 , 1] D × D that blurs out the input image in a way that would minimize the target-class probability.
That is, they attempted to solve the following optimization problem:
[Equation]
where ¯ x is given by Eq.
but with f = B σ ( x ) i.e. the input image blurred by a Gaussian kernel
B σ ( . ) of radius σ = 10 .
Note that, in MP, the attribution map A is also the learned mask m .
However, solving Eq. 2 directly often yields heatmaps that are noisy and sensitive to hyperparameter changes.
Therefore, Fong & Vedaldi (2017) only learned a small 28 × 28 mask and upsampled it to the image size in every optimization step.
In addition, they also encouraged the mask to be smooth and robust to input changes by changing the objective function to the following:
[Equation]
where TV ( m ) = (cid:80) i (cid:107)∇ m i (cid:107) 33 i.e. a total-variation norm that acts as a smoothness prior over the mask.
The third term is the expectation over a batch of randomly jittered versions of the blurred image ¯ x .
That is, Φ( . ) is the jitter operator that translates an image ¯ x vertically or horizontally by τ pixels where τ is drawn from a discrete uniform distribution U (0 , 4) .
We randomly initialized the mask from a continuous uniform distribution U (0 , 1) and minimize the objective function in Eq.
3 via gradient descent for 300 steps.
Our MP implementation is in PyTorch and followed all the hyperparameters as described in Fong & Vedaldi (2017).
MP-G We propose to create an MP variant called MP-G by only changing the ﬁller image f
=
B σ ( x ) that is used in Eq.
1 to an inpainted image i.e. f = G ( m b , x ) where m b ∈ { 0 , 1 } D × D is the result of binarizing m at a threshold of 0 . 5 .
Note that this binarization step is necessary because
G ( . ) expects a binary mask; however, we are still learning a continuous mask m ∈ R D × D .
E XPERIMENTS AND R ESULTS 4. I NPAINTER FAILED TO SYNTHESIZE BACKGROUNDS GIVEN ONLY OBJECTS
Chang et al.
(2019) proposed to ﬁnd a minimal set of input pixels that would keep the classiﬁcation outputs unchanged even when all other pixels are removed via an inpainter i.e. the “preservation” objective (Fong & Vedaldi, 2017).
They used the same DeepFill-v1 inpainter as in our work; however, their “preservation” objective tends to force the inpainter to predict the missing background pixels conditioned on the remaining foreground object—a task that DeepFill-v1 was not trained to do and thus produced unrealistic samples as in Chang et al.
In contrast, our MP-G method harnesses the dual “deletion” objective (Fong & Vedaldi, 2017) i.e. ﬁnding the smallest set of input pixels such that when inpainted would minimize the target-class probability—which intuitively asks the inpainter to replace the main object with some generated content similar to the real background.
To compare these two objectives, we randomly chose 50 validation-set images from 52 ImageNet bird classes and computed their segmentation masks via a pre-trained DeepLab model (Chen et al., 2017).
We found that using the DeepFill-v1 to inpaint the foreground region (i.e. our “deletion” task) yields realistic samples where the object is removed.
In contrast, using the inpainter to ﬁll in the missing background area (i.e. the “preservation” task) yields unrealistic images whose backgrounds contain features (e.g. bird feathers or beaks) unnaturally pasted from the object (see a side-by-side comparison in Fig. S7).
This result motivated us to integrate DeepFill-v1 into attribution methods and to study MP-G with the “deletion” objective which has not been explored in Chang et al.
4. I NPAINTER IS EFFECTIVE IN REMOVING DISCRIMINATIVE FEATURES
While removing objects from an image via DeepFill-v1 qualitatively yields realistic samples, here, we quantitatively test how effective this strategy is in removing target-class discriminative features in comparison with three existing alternative ﬁlling methods: (1) zero pixels; (2) random noise; or (3) blurred versions of the original content.
Using the same procedure as described in Sec.
4.1, we randomly sampled 1000 bird images and segmented out the bird in each image.
We ﬁlled in the object mask in each image via all four methods (Fig. 2) and compare the results (Table 1).
Surprisingly, the blurred and grayed-out images are still correctly classiﬁed at 26.4% and 13.3% (Table 1), respectively, by a pre-trained Inception-v classiﬁer (Szegedy et al., 2016).
That is, these perturbed images still contain much discriminative features relevant to the target class.
In contrast, only 8.9% of the inpainted images were correctly classiﬁed suggesting that the inpainter was more effective in removing the discriminative features.
After the main subject (here, birds) has been removed from an image, intuitively, one would expect the modiﬁed image to be perceptually different from the original image (where the bird exists).
Here, we evaluate how each of the four in-ﬁlled images ¯ x (where the bird has been removed) is perceptually dissimilar to the original image x by measuring the MS-SSIM and LPIPS (Zhang et al., 2018) scores between every pair ( x , ¯ x ).
Across both metrics, the inpainted images are consistently the most dissimilar from the real images compared to the blurred and grayed-out images.
Note that in all three quantitative metrics, the inpainted images are the closest to the noise-ﬁlled images (Table 1d–e) despite being substantially more realistic (Fig. 2).
[Figure]
Metrics
Filling methods (a) Real (b) Blur (c) Gray (d) Inpaint (e) Random
Inception Accuracy 92.30% 26.40% 13.30% 8.90% 4.40%
MS-SSIM (lower is better) 1.
LPIPS (higher is better) 0.
4. A RE GENERATIVE ATTRIBUTION METHODS MORE ROBUST TO HYPERPARAMETERS ?
Perturbation-based attribution methods have many hyperparameters that are often heuristically tuned, which when varied can change the explanations radically.
That sensitivity poses a huge challenge to (1) evaluating the explanations; and (2) building trust with end users (Doshi-Velez &
Kim, 2017).
Our hypothesis is that heuristically-perturbed samples are often far from the true data distribution and thus might contribute to the sensitivity of heatmaps to hyperparameters.
Here, we test whether our attribution methods which incorporate an inpainter, i.e. SP-G, LIME-G, and MP-G (hereafter, G-methods), are more robust to hyperparameter changes than their original counterparts.
4.3. E XPERIMENT SETUP
Similarity metrics and Image sets Following Adebayo et al.
, we used three metrics from scikit-image (van der Walt et al., 2014) to measure the similarity of heatmaps: the structural similarity index (SSIM), the Pearson correlation of the histograms of gradients (HOGs), and the Spearman rank correlation.
We upsampled all heatmaps to the full image size before feeding them into the similarity metrics.
We performed the test on a set of 1000 random images from ImageNet-S and also replicated the experiments on Places365-S.
SP sensitivity across patch sizes It remains a question how to choose the patch size in the SP algorithm because changing it can change the explanation radically (Zintgraf et al., 2017).
Therefore, we compare the sensitivity of SP and SP-G when sweeping across 5 patch sizes p × p with stridewhere p ∈ { 5 , 17 , 29 , 41 , 53 } .
We chose this set to cover the common sizes that have been used in the literature.
For each input image, we obtained k = 5 heatmaps (i.e. each corresponds to a patch size) and then measured the similarity among all k ( k − 1) / 2 = 10 possible pairs.
LIME sensitivity across random batches of samples LIME randomly samples N perturbed images { ¯ x i } N and uses them to ﬁt a heatmap.
We found that the heatmap for an input can vary across different batches of random perturbation samples.
Therefore, we compared the sensitivity of LIME and LIME-G across 5 random batches of N = 500 perturbation samples.
That is, for each input
[Figure] [Figure]
MP
[Figure] [Figure]
MP
[Figure]
image among the 1000, we generated k = 5 heatmaps and computed the similarity among allpossible pairs.
We ran this experiment for a small and a large heatmap resolution i.e. two numbers of superpixels S ∈ { 50 , 150 } while keeping all other hyperparameters constant.
MP sensitivity across random initializations Similar to the LIME sensitivity experiment, here, we compared the variations of MP vs. MP-G heatmaps of size 28 × 28 across 5 random initializations.
We re-ran each algorithm 5 times on each input image and computed the similarity among allpossible pairs.
Here, all hyperparameters are the same as in Sec. 3.5.
4.3. R ESULTS
First, we found that all 6 algorithms produce inconsistent explanations across the controlled hyperparameters (Fig. 3; all scores are below 1).
That is, LIME and MP heatmaps can change as one simply changes the random seed!
However, LIME-G is consistently more robust than LIME across all 3 metrics (Fig. 3) and across both superpixel settings.
Across the patch sizes, SP-G is also consistently more robust than SP (Fig. 3a–b; light vs. dark yellow).
SP-G and SP performed on par with high standard deviations under the Spearman rank correlation (Fig. 3c).
Across the random initializations, MP-G is consistently less robust than MP (Fig.
3 light vs. dark red).
The above trends were also observed on Places365-S (Fig.
S4) suggesting that the ﬁndings are statistically signiﬁcant.
4. A RE EXPLANATIONS BY GENERATIVE ATTRIBUTION METHODS MORE FAITHFUL ?
While there are currently no established ground-truth datasets to evaluate an attribution map, prior research often assessed heatmaps via three proxy metrics: (1) the object localization task (Zhou et al., 2016); (2) insertion and deletion task (Petsiuk et al., 2018).
Here, we ran all 6 algorithms on the ImageNet-S and Places365-S datasets using the default hyperparameters in Sec. 3.
The heatmaps are then upsampled to the full image resolution for evaluation on all three benchmarks above.
Object localization Zhou et al.
(2016) proposed to evaluate heatmaps by its ability to segment out objects in the ImageNet images, which often contain a single object of a known class.
We applied the localization procedure in Fong & Vedaldi (2017) for the ImageNet-S dataset.
That is, for each algorithm, we derived multiple bounding boxes per heatmap by thresholding it at different values of t =
αµ max , where µ max is the maximum intensity in the heatmap and α ∈ [0 : 0 .
For each α , we computed the Intersection over Union (IoU) score between a derived bounding box and the ImageNet ground-truth.
The object localization error was calculated by thresholding each IoU score at 0 .
5 and averaging them across the number of images.
For each method, we chose the best α ∗ that yielded the lowest error on a held-out set of 1000 ImageNet-S images (Table 2).
We found that LIME-G and SP-G outperformed their respective counterparts while MP-G was on par with MP (Table 2).
Among the 6 methods, LIME-G obtained the lowest error of 27.12%.
For qualitative evaluation, Figs.S8–S10 show a set of heatmaps and derived bounding boxes for the cases where G-methods outperformed the original methods the most (in terms of IoU) and vice versa.
Deletion and Insertion metrics
The Deletion metric (Petsiuk et al., 2018) measures the target-class probability changes as we gradually zero out the input pixels of the highest attributions one-by-one
[Table]
in the descending order.
We evaluated all 6 methods via the code released by the authors on both the
Deletion and also the related Insertion metric (Petsiuk et al., 2018).
However, we did not ﬁnd any consistent trends across both ImageNet-S and Places365-S datasets and both metrics.
For example,
SP-G consistently outperformed SP under the Deletion metric on both datasets, but not under the
Insertion metric.
See Sec.
B for more details.
Note that these metrics (1) assume that input pixels are independent ; however, zero-ing out pixels one-by-one almost always yields unrealistic images; (2) the probability changes are computed from those unrealistic, out-of-distribution inputs and therefore can be misleading—the exact problem that our paper attempts to solve.
T HE INNER - WORKINGS OF GENERATIVE ATTRIBUTION METHODS
Here, we used SP-G and LIME-G as the case studies to explain why our G-methods are both more (1) robust to hyperparameter changes (Sec. 4.3) and (2) accurate in localizing objects (Sec. 4.4)
. 5. M ORE ACCURATE OBJECT LOCALIZATION : A CASE STUDY OF SP-G
We found that as the gray patch of SP is slided from left to right across the input image (Fig. 4a; top), the target-class probability gradually decreases and approaches 0 when the patch occludes most of the object (Fig. 4b; red line).
Note that the probability can drop even when the patch is far outside the object region (Fig. 4b; red line within [0 , 24] ) due to SP unrealistic grayish samples.
As the result, the probability distributions by SP often yield a large blob of high attributions around the object in the heatmap (Fig. 4a; top).
In contrast, the inpainted samples of SP-G often keep the probability variance low except when the patch overlaps with the object (Fig. 4b; blue vs. red), yielding heatmaps that are more localized towards the object (Fig. 4a; bottom).
Across 1000 random ImageNet-S images, we found that the average probability change when the SP 53 × 53 patch is outside the object bounding box is ∼ 2.
times higher than that of SP-G (i.e. 0.09 vs. 0.04).
In sum, our observations here are consistent with the ﬁndings that SP-G and LIME-G obtained lower localization errors than the original counterparts.
[Figure] [Figure]
5. M ORE ROBUST HEATMAPS : A CASE STUDY OF LIME-G
Here, we provide insights for why LIME-G produced heatmaps that are more consistent than LIME across 5 random batches of samples (i.e. controlled, in practice, by 5 random seeds).
We observed that the top-1 predicted labels of ∼ 20.5% of the LIME grayish perturbation samples (e.g. Fig.
5a) were from only three classes { jigsaw puzzle , maze , hen - of - the - wood } whereas the same top-1 label distribution for LIME-G samples was almost uniform (see Fig.
S21 for more details).
That is, across different objects, many LIME samples are still given the same label due to their similar grayish, puzzle-like patterns.
Relatedly, we observed that a LIME perturbation sample is often given a nearzero probability score regardless of what input feature is being masked out (Fig. 5a).
Therefore, when ﬁtted to N samples, where N is often too small 4 w.r.t.
the total 2 S possible samples, the heatmap appears random and changes upon a new set of random masks (Fig. 5b).
[Figure]
are paper towel .
In contrast, for LIME-G samples, the probabilities consistently drop when some discriminative features (e.g. the kuvasz dog’s eye in Fig.
5c) are removed.
This phenomenon yields heatmaps that are more consistently localized around the same input features across different random seeds (Fig. 5d).
Our explanation also aligns with the ﬁnding that when the number of superpixels S increases fromto 150 (while the sample size remains at N = 500 ), the sensitivity gap between LIME vs. LIME-G increases by ∼ 3 times (Fig. 3a; gap between green bars vs. gap betwen blue bars).
See Figs.
S12–S19 for qualitative examples of when LIME-G is more robust than LIME and vice versa.
Quantitatively, we found that the image distribution where LIME-G showed superior robustness over LIME across all three similarity metrics mostly contains images of scenes, close-up or tiny objects.
In contrast, LIME is more robust than LIME-G on images of mostly birds and medium-sized objects (see Sec.
C for more details).
D ISCUSSION AND C ONCLUSION
A hypothesis for why we did not ﬁnd MP-G to be more robust than MP is that the original formulation (1) optimizes a tiny 28 × 28 mask; and (2) strongly regularizes it to be sparse and smooth.
A non-mutually exclusive hypothesis is that since MP operates in the pixel space, the learned masks can be of any shape.
However, DeepFill-v1 was not designed for inpainting free-form masks (Yu et al., 2018a).
We also found that a blur mask learned by MP often covers ∼ 99% of the input image.
That is, the MP attribution map is explaining the target-class probability drops when the entire image is modiﬁed—an unrealistic causal intervention.
In contrast, MP-G perturbations are more localized and only changes ∼ 40% of the input image (see Sec.
A).
In sum, integrating a state-of-the-art, unconditional inpainter into three existing attribution methods can yield visual explanations that (1) localize objects better; (2) are more robust to hyperparameter changes; and
3) are based on more plausible counterfactuals.
Our results suggest that harnessing generative models to synthesize synthetic interventions (here, removal of input features) might be a promising direction for future causal explanation methods
A
PPENDIX
A M ASKS LEARNED BY MP PERTURB ALMOST THE ENTIRE IMAGE
For ImageNet images where human-annotated bounding boxes only cover less than 50% of the image, MP perturbs the entire image to generate its ﬁnal attribution maps.
Hence, the heatmap does not completely reﬂects the target object class as the drop in probability is due to the perturbation across the whole image (Fig. S1).
On the other hand for MP-G, as discussed in Sec. 3.5, we binarize the mask during the optimization process and perturb only part of the image.
In sum using MP-G, we perturb only ∼ 40% and ∼ 39%
(Table S1) of the ImageNet-S and Places365-S images respectively.
[Figure]
[Table]
[Figure]
B E VALUATION ON THE D ELETION AND I NSERTION METRICS
The deletion metric (Petsiuk et al., 2018)
measures the target-class probability changes as we gradually zero out the input pixels of the highest attributions in the descending order.
The idea is if the attribution values in a heatmap correctly reﬂect the discriminative power of the input pixels, knocking out the highest-attribution pixels should quickly cause the probability approach 0.
However, this metric has two issues: (1) it makes an unrealistic assumption that input features are independent; (2) the probability changes may be due to the unrealistic, out-of-distribution inputs when some pixels are masked out.
For completeness, we evaluated all 6 methods via the code released by Petsiuk et al.
(2018) on the deletion metric and also its inverse version
i
[Figure]
e. the insertion metric (Petsiuk et al., 2018) on both datasets of ImageNet-S and
Places365-S.
However, we did not ﬁnd signiﬁcant differences between the G-methods and their counterparts (Fig.S2)
[Figure]
ImageNet
[Figure]
Places365-S
[Figure]
[Table]
C LIME-G IS MORE ROBUST THAN LIME ON IMAGES OF SCENES , CLOSE - UP AND TINY OBJECTS
We have shown that LIME-G is more robust than LIME consistently on all 3 different similarity metrics (see Sec. 4.3 in the main text).
Here, we aim to understand the image distributions where
LIME-G was more robust than LIME and vice versa.
For each of the three metrics, we computed a set of top-100 score differences between LIME-G vs.
LIME.
Interestingly, we found the intersection of the three sets contains images of mostly scenes, close-up or tiny objects (see Fig.
S3).
In contrast, the common set of images where LIME is more robust than LIME-G contains mostly birds and medium-sized objects.
These image distributions intuitively align with the domains where DeepFill-v1 is capable of inpainting and suggest that the performance of G-methods can be improved further with class-conditional inpainters.
[Table] [Table]
[Figure]
LIME-G is consistently more robust than LIME on both low and high resolutions
i
[Figure]
e. S ∈ { 50 , 150 } (green and blue bars).
The exact numbers are reported in Table S
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Table]
h
[Table]
n -
[Table]
ft
[Table]
e
[Table]
h a i
[Table]
t
[Table]
m i
[Table]
f
[Table]
n
[Table]
hu mm
[Table]
r
[Table]
i
[Table]
m
[Table]
p e b o l o t i e l a n g u r w a ll c l o c k
[Figure]
g r e a t g r e y o w l g y r
o
m i t
r a
[Table]
m
[Table]
li
[Table]
p i d e
[Table]
n
[Table]
a li
[Table]
a
[Table]
m
[Table]
n g
[Table]
a r
[Table]
l
[Table]
ee
[Table]
m ee r k a t m o v i n g v a n o r g a n b o l o t i e hu
mm i n g b i r d y e ll o w l a d y ' s s li p
p
e r h a r d
d
i s c
S e a l y h a m t e rr i e r d u g o n g t
i c k
[Figure]