I NVESTIGATING AND S IMPLIFYING M ASKING - BASED S ALIENCY M AP M ETHODS FOR M ODEL I NTERPRETABILITY
Anonymous authors
Paper under double-blind review
A BSTRACT
Saliency maps that identify the most informative regions of an image for a classiﬁer are valuable for model interpretability.
A common approach to creating saliency maps involves generating input masks that mask out portions of an image to maximally deteriorate classiﬁcation performance, or mask in an image to preserve classiﬁcation performance.
Many variants of this approach have been proposed in the literature, such as counterfactual generation and optimizing over a Gumbel-Softmax distribution.
Using a general formulation of masking-based saliency methods, we conduct an extensive evaluation study of a number of recently proposed variants to understand which elements of these methods meaningfully improve performance.
Surprisingly, we ﬁnd that a well-tuned, relatively simple formulation of a masking-based saliency model outperforms many more complex approaches.
We ﬁnd that the most important ingredients for high quality saliency map generation are (1) using both masked-in and masked-out objectives and (2) training the classiﬁer alongside the masking model.
Strikingly, we show that a masking model can be trained with as few as 10 examples per class and still generate saliency maps with only a 0.7-point increase in localization error.
I NTRODUCTION
The success of CNNs (Krizhevsky et al., 2012; Szegedy et al., 2015; He et al., 2016; Tan & Le, 2019) has prompted interest in improving understanding of how these models make their predictions.
Particularly in applications such as medical diagnosis, having models explain their predictions can improve trust in them.
The main line of work concerning model interpretability has focused on the creation of saliency maps–overlays to an input image that highlight regions most salient to the model in making its predictions.
Among these, the most prominent are gradient-based methods (Simonyan et al., 2013; Sundararajan et al., 2017; Selvaraju et al., 2018) and masking-based methods (Fong &
Vedaldi, 2017; Dabkowski & Gal, 2017; Fong & Vedaldi, 2018; Petsiuk et al., 2018; Chang et al., 2019; Zintgraf et al., 2017).
In recent years, we have witnessed an explosion of research based on these two directions.
With a variety of approaches being proposed, framed and evaluated in different ways, it has become difﬁcult to assess and fairly evaluate their additive contributions.
In this work, we investigate the class of masking-based saliency methods, where we train a masking model to generate saliency maps based on an explicit optimization objective.
Using a general formulation, we iteratively evaluate the extent to which recently proposed ideas in the literature improve performance.
In addition to evaluating our models against the commonly used Weakly
Supervised Object Localization (WSOL) metrics, the Saliency Metric (SM), and the more recently introduced Pixel Average Precision (PxAP;
Choe et al., 2020), we also test our ﬁnal models against a suite of “sanity checks” for saliency methods (Adebayo et al., 2018; Hooker et al., 2018).
Concretely, we make four major contributions.
(1) We ﬁnd that incorporating both masked-in classiﬁcation maximization and masked-out entropy maximization objectives leads to the best saliency maps, and continually training the classiﬁer improves the quality of generated maps.
We ﬁnd that the masking model requires only the top layers of the classiﬁer to effectively generate saliency maps.
(3) Our ﬁnal model outperforms other masking-based methods on WSOL and PxAP metrics.
(4) We ﬁnd that a small number of examples—as few as ten per class—is sufﬁcient to train a masker to within the ballpark of our best performing model.
[Figure]
R ELATED W ORK
Interpretability of machine learning models has been an ongoing topic of research (Ribeiro et al., 2016; Doshi-Velez & Kim, 2017; Samek et al., 2017; Lundberg et al., 2018).
In this work, we focus on interpretability methods that involve generating saliency maps for image classiﬁcation models.
An overwhelming majority of the methods for generating saliency maps for image classiﬁers can be assigned to two broad families: gradient-based methods and masking-based methods.
Gradient-based methods , such as using backpropagated gradients (Simonyan et al., 2013), Guided
Backprop (Springenberg et al., 2015), Integrated Gradients (Sundararajan et al., 2017), GradCam (Selvaraju et al., 2018), SmoothGrad (Smilkov et al., 2017) and many more, directly use the backpropagated gradients through the classiﬁer to the input to generate saliency maps.
Masking-based methods modify input images to alter the classiﬁer behavior and use the regions of modiﬁcations as the saliency map.
Within this class of methods, one line of work focuses on optimizing over the masks directly:
Fong & Vedaldi (2017) optimize over a perturbation mask for an image, Petsiuk et al.
(2018) aggregates over randomly sampled masks, Fong & Vedaldi (2018) performs an extensive search for masks of a given size, while Chang et al.
(2019) includes a counterfactual mask-inﬁlling model to make the masking objective more challenging.
The other line of work trains a separate masking model to produce saliency maps: Dabkowski & Gal (2017) trains a model that optimizes similar objectives to Fong & Vedaldi (2017), Zolna et al.
(2020) use a continually trained pool of classiﬁers and an adversarial masker to generate model-agnostic saliency maps, while Fan et al.
(2017) identiﬁes super-pixels from the image and then trains the masker similarly in an adversarial manner.
Salient Object Detection (Borji et al., 2014; Wang et al., 2019) is a related line of work that concerns identifying salient objects within an image as an end in itself, and not for the purpose of model interpretability.
While it is not uncommon for these methods to incorporate a pretrained image classiﬁcation model to extract learned visual features, they often also incorporate techniques for improving the quality of saliency maps that are orthogonal to model interpretability.
Salient object detection methods that are trained on only image-level labels bear the closest similarity to saliency map generation methods for model interpretability.
Hsu et al.
and follow-up Hsu et al.
(2019) train a masking model to confuse a binary image-classiﬁcation model that predicts whether an image contains an object or is a ‘background’ image. Wang et al. (2017) apply a smooth pooling operation and a Foreground Inference Network (a masking model) while training an image classiﬁer to generate saliency maps as a secondary output. Evaluation of saliency maps The wave of saliency map research has also ignited research on evaluation methods for these saliency maps as model explanations.
Adebayo et al.
(2018) and Hooker et al.
(2018) propose sanity checks and benchmarks for the saliency maps.
Choe et al.
(2020) propose
Pixel Average Precision (PxAP), a pixel-wise metric for scoring saliency maps that accounts for mask binarization thresholds, while Yang & Kim (2019) create a set of metrics as well as artiﬁcial datasets interleaving foreground and background objects for evaluating the saliency maps.
These works have shown that a number of gradient-based methods fail the sanity checks or perform no better than simple edge detectors.
Hence, we choose to focus on masking-based methods in this paper.
M ASKING - BASED S ALIENCY M AP M ETHODS
We start by building a general formulation of masking-based saliency map methods.
We take as given a trained image classiﬁer F : x → y , that maps from image inputs x ∈ R H × W × C to class predictions ˆ y ∈ [0 , 1] K , evaluated against ground-truth y ∈ { 1 · · · K } .
Our goal is to generate a mask m ∈ [0 , 1] H × W for each image x such that the masked-in image x (cid:12) m or the masked-out image x (cid:12) (1 − m ) maximizes some objective based on output of a classiﬁer given the modiﬁed image.
For instance, we could attempt to mask out parts of the image to maximally deteriorate the classiﬁer’s performance.
This mask m then serves as a saliency map for the image x .
Concretely, the per-image objective can be expressed as:
[Equation]
where L out , L in are the masked-out and masked-in objectives over the classiﬁer output, λ out , λ in are hyperparameters controlling weighting of these two objectives, θ F the classiﬁer parameters, and
R ( m ) a regularization term over the mask.
The masked-in and masked-out losses, L out and L in , correspond to ﬁnding the smallest destroying region and smallest sufﬁcient region as described in
Dabkowski & Gal (2017).
Candidates for L out include negative classiﬁcation cross-entropy and prediction entropy.
For L in , the obvious candidate is the classiﬁcation cross-entropy of the masked-in image.
We set λ in = 0 or λ out =
if we only have either a masked-in or masked-out objective.
The above formulation subsumes a number of masking-based methods, such as Fong & Vedaldi (2017); Dabkowski & Gal (2017); Zolna et al.
We follow Dabkowski & Gal, amortize the optimization by training a neural network masker M : x → m , and solve for:
[Equation]
where M is the masking model and θ M its parameters.
In our formulation, we do not provide the masker with the ground-truth label, which differs from certain other masking-based saliency works (Dabkowski & Gal, 2017; Chang et al., 2019; Fong & Vedaldi, 2018).
In practice, we often desire model explanations without the availability of ground-truth information, so we focus our investigation on methods that require only an image as input.
3. M ASKER A RCHITECTURE
We use a similar architecture to Dabkowski & Gal and Zolna et al..
The masker takes as input activations across different layers of the classiﬁer, meaning it has access to the internal representation of the classiﬁer for each image.
Each layer of activations is fed through a convolutional layer and upsampled (with nearest neighbor interpolation) so they all share the same spatial resolution.
All transformed layers are then concatenated and fed through another convolutional layer, upsampled, and put through a sigmoid operation to obtain a mask of the same resolution as the input image.
In all our experiments, we use a ResNet-50 (He et al., 2016) as our classiﬁer, and the masker has access to the outputs of the ﬁve major ResNet blocks.
Figure 1B shows the architecture of our models.
Following prior work (Fong & Vedaldi, 2017), we apply regularization on the generated masks to avoid trivial solutions such as masking the entire image.
We apply L1 regularization to limit the size of masks and Total Variation (TV) to encourage smoothness.
Details can be found in Appendix A.1.
[Table]
3. C ONTINUAL T RAINING OF THE C LASSIFIER
Because neural networks are susceptible to adversarial perturbations (Goodfellow et al., 2015) masking models can learn to perturb an input to maximize the above objectives for a given ﬁxed classiﬁer without producing intuitive saliency maps.
While directly regularizing the masks is one potential remedy, Zolna et al.
(2020) propose to train the masker against a diverse set of classiﬁers.
In practice, they simulate this by continually training the classiﬁer on masked images, retain a pool of past model checkpoints and sample from the pool when training the masker.
We adopt their approach and distinguish between a masker trained against a ﬁxed classiﬁer ( FIX ) and against a pool of continually trained classiﬁers ( CA , for Classiﬁer-Agnostic).
We highlight that saliency maps for FIX and CA address fundamentally different notions of saliency.
Whereas a
FIX approach seeks a saliency map that explains what regions are most salient to a given classiﬁer, a CA approach tries to identify all possible salient regions for any hypothetical classiﬁer (hence, classiﬁer-agnostic).
In other words, a CA approach may be inadequate for interpreting a speciﬁc classiﬁer and is better suited for identifying salient regions for a class of image classiﬁcation models.
E XPERIMENTAL SETUP
We perform our experiments on the ofﬁcial ImageNet training and validation set (Deng et al., 2009) and use bounding boxes from the ILSVRC’14 localization task.
Because we perform a large number of experiments with hyperparameter search to evaluate different model components, we construct a separate held-out validation set of 50,000 examples (50 per class) from the training set with bounding box data that we use as validation for the majority of our experiments (which we refer to as our “TrainValidation” set) and use the remainder of the training set for training. For each model conﬁguration, we train the models 5 times on different random seeds and report the mean and standard error of the results.
We reserve the ofﬁcial validation set for the ﬁnal evaluation.
4. E VALUATION METRICS
Weakly-supervised object localization task metrics (WSOL) is a common task for evaluating saliency maps.
It involves generating bounding boxes for salient objects in images and scoring them against the ground-truth bounding boxes.
To generate bounding boxes from our saliency maps, we binarize the saliency map based on the average mask pixel value and use the tightest bounding box around the largest connected component of our binarized saliency map.
We follow the evaluation protocol in ILSVRC ’14 computing the ofﬁcial metric (OM), localization error (LE) and pixel-wise
F1 score between the predicted and ground-truth bounding boxes.
Saliency metric (SM) proposed by Dabkowski & Gal (2017) consists of generating a bounding box from the saliency map, upsampling the region of the image within the bounding box and then evaluating the classiﬁer accuracy on the upsampled salient region.
The metric is deﬁned as s ( a, p ) =
log(max( a, 0 . 05))
− log( p ) , where a is the size of the bounding box, and p is the probability the classiﬁer assigns to the true class.
This metric can be seen as measuring masked-in and upsampled classiﬁcation accuracy with a penalty for the mask size.
We use the same bounding boxes as described in WSOL for consistency.
Pixel Average Precision (PxAP) proposed by Choe et al.
(2020) scores the pixel-wise masks against the ground-truth masks and computes the area under the precision-recall curve.
This metric is computed over mask pixels rather than bounding boxes and removes the need to threshold and binarize the mask pixels.
PxAP is computed over the OpenImages dataset (Benenson et al., 2019) rather than ImageNet because it requires pixel-level ground-truth masks.
E VALUATION OF S ALIENCY M AP M ETHODS
To determine what factors and methods contribute to improved saliency maps, we perform a series of evaluation experiments in a cascading fashion.
We isolate and vary one design choice at a time, and use the optimal conﬁguration from one set of experiments in all subsequent experiments.
Our baseline models consist of a masker trained with either a ﬁxed classiﬁer (FIX) or a pool of continually trained classiﬁers (CA).
As WSOL is the most common task for evaluating saliency maps, we use
LE as the metric for determining the ‘best’ model for model selection.
We show our model scores across experiments in Table 3.
Each horizon block represents a set of experiments varying one design choice.
The top half of the table is evaluated on our Train-Validation split, while the bottom half is evaluated on the validation data from ILSVRC ’14.
Masking Objectives (Rows a – f )
We ﬁrst consider varying the masker’s training objective, using only one objective at a time.
We use the three candidate objectives described in Section 3: maximizing masked-in accuracy, minimizing masked-out accuracy and maximizing masked-out entropy.
For a masked-out objective, we set λ out = 1 , λ in = 0 , and the opposite for masked-in objectives.
For each conﬁguration, we perform a random hyperparameter search over the L1 mask regularization coefﬁcients λ M and λ TV as well as the learning rate and report results from the best conﬁguration from the Train-Validation set.
More details on hyperparameter choices can be found in Table 2.
Consistent with Zolna et al.
(2020), we ﬁnd that training the classiﬁer along with the masker improves the masker, with CA models generally outperforming FIX models, particularly for the WSOL metrics.
However, the classiﬁcation-maximization FIX model still performs comparably with its CA counterpart and in fact performs best overall when measured by SM given the similarity between the training objective and the second term of the SM metric.
Among the CA models, entropymaximization and classiﬁcation-minimization perform the best, while the classiﬁcation-maximization objective performs worst.
On the other hand, both mask-out objectives perform extremely poorly for a ﬁxed classiﬁer.
We show how different masking objectives affect saliency map generation in Figure 2. [Figure]
produce highly bimodal distributions, most pixel values close to
Combining Masking Objectives (Rows g – j ) Next, we combine both masked-in and masked-out objective during training, setting λ out
= λ in = 0 .
Among the dual-objective models, entropymaximization still outperforms classiﬁcation-minimization as a masked-out objective.
Combining both masked-in classiﬁcation-maximization and masked-out entropy-maximization performs best for both FIX and CA models, consistent with Dabkowski & Gal (2017).
From our hyperparameter search, we also ﬁnd that separately tuning λ
M,in and λ M,out is highly beneﬁcial (see Table 2).
We use the classiﬁcation-maximization and entropy-maximization dual objectives for both FIX (Row g ) and CA (Row i ) models in subsequent experiments.
Varying Observed Classiﬁer Layers (Rows k – r )
We now vary which hidden layers of the classiﬁer the masker has access to.
As described in Section 3.1, the masking model has access to hidden activations from ﬁve different layers of a ResNet-50 classiﬁer.
To identify the contribution of information from each layer to the masking model, we train completely new masking models with access to only a subset of the classiﬁer layers.
We number the layers from 1 to 5, with 1 being the earliest layer with the highest resolution ( 56 × 56 ) and 5 being the latest ( 7 × 7 ).
We show a relevant subset of the results from varying the observed classiﬁer layers in Table 3.
The full results can be found in the Table
and we show examples of the generated masks in Figure 3.
Masking models with access to activations of later layers starkly outperform those using activations from earlier layers.
Whereas the Layer[3], Layer[4] and Layer[5] models are still effective, the
Layer[1] and Layer[2] models tend to perform poorly.
Similarly, we ﬁnd that the best cascading combination of layers is layers 4 and 5 CA models, and 3–5 for FIX models (Rows n , r ), slightly but consistently outperforming the above models with all layers available to the masker.
This suggests that most of the information relevant for generating saliency maps is likely contained within the later layers.
For simplicity, we use only classiﬁer layers 4 and 5 for subsequent experiments.
Counterfactual Inﬁlling and Binary Masks (Rows s – z )
Chang et al.
(2019) proposed generating saliency maps by learning a Bernoulli distribution per masking pixel and additionally incorporating a counterfactual inﬁller.
Agarwal & Nguyen (2019) similarly uses an inﬁller when producing saliency maps.
First, we consider applying counterfactual inﬁlling to the masked images before feeding them to the classiﬁer.
The modiﬁed inputs are Inﬁll( X (cid:12) (1 − m ) , (1 − m )) and Inﬁll( X (cid:12) m, m ) for Tench Input RadiatorGrille
Input
F I X Layer= Layer= Layer=
Layer=
Layer=
All Layers
C A
F I X
C A masked-out and masked-in inﬁlling respectively, where Inﬁll is the inﬁlling function that takes as input the masked input as well as the mask.
We consider three inﬁllers: the Contextual Attention
GAN (Yu et al., 2018) as used in Chang et al. 1 , DFNet (Hong et al., 2019), and a Gaussian blur inﬁller as used in Fong & Vedaldi (2018).
Both neural inﬁllers are pretrained and frozen.
For each inﬁlling model, we also train a model variant that outputs a discrete binary mask by means of a Gumbel-Softmax layer (Jang et al., 2017; Maddison et al., 2017).
We experiment with both soft and hard (Straight-Through)
Gumbel estimators, and temperatures of { 0 . 01 , 0 . 05 , 0 . 1 , 0 . 5 } .
We show a relevant subset of the results in the fourth and ﬁfth blocks of Table 3, and examples in
Figure 6 and Figure 7.
We do not ﬁnd major improvements from incorporating inﬁlling or discrete masking based on WSOL metrics, although we do ﬁnd improvements from using the DFN inﬁller for
SM.
Particularly for CA, because the classiﬁer is continually trained to classify masked images, it is able to learn to both classify unnaturally masked images as well as to perform classiﬁcation based on masked-out evidence.
As a result, the beneﬁts of incorporating the inﬁller may be diminished.
5. E VALUATION ON V ALIDATION
S ET
Based on the above, we identify a simple recipe for a good saliency map generation model: (1) use both masked-in classiﬁcation maximization and masked-out entropy maximization objectives, (2) use only the later layers of the classiﬁer as input to the masker, and (3) continually train the classiﬁer.
To validate the effectiveness of this simple setup, we train a new pair of FIX and CA models based on this conﬁguration on the full training set and evaluate on the actual ILSVRC ’14 validation set.
We compare the results to other models in the literature in the bottom block of Table 3.
Consistent with above, the CA model outperforms the FIX model.
It also outperforms other saliency map extraction methods on WSOL metrics and PxAP.
We highlight that some models we compare to (Rows E , G ) are provided with the ground-truth target class, whereas our models are not–this may explain the underperformance on certain metrics such as SM, which is partially based on classiﬁcation accuracy.
5. S ANITY C HECKS
Adebayo et al.
(2018) and Hooker et al.
(2018) propose “sanity checks” to verify whether saliency maps actually reﬂect what information classiﬁers use to make predictions and show that many proposed interpretability methods fail these simple tests.
We apply these tests to our saliency map models to verify their efﬁcacy.
On the left of Figure 4, we show the RemOve-and-Retrain (ROaR) test proposed by Hooker et al., where we remove the top t % of pixels from training images based on our generated saliency maps and use them to train entirely new classiﬁers.
If our saliency maps truly identify salient portions of the image, we should see large drops in classiﬁer performance as t increases.
Both FIX and CA methods pass this test, with classiﬁer accuracy falling precipitously as we mask out more pixels.
On the right of Figure 4, we perform the the Model Parameter Randomization
Test (MPRT) proposed by Adebayo et al..
We randomize parameters of successive layers of the , but Chang et al. nevertheless found positive results from applying it, so we follow their practice.
DFNet is trained on irregularly shaped masks.
classiﬁer, starting from upper-most logits layer to the lowest convolutional layers, and generate saliency maps using the partially randomized classiﬁers.
We then compute the similarity of the saliency maps generated from using the partially randomized classiﬁer, and those using the original classiﬁer.
Our saliency maps become less similar as more layers are randomized, passing the test.
The results for the Data Randomization Test (DRT) can be found in Table 5.
[Figure]
F EW - SHOT E XPERIMENTS
Given the relative simplicity of our best-performing saliency map models and the fact that the masker uses only the top layers of activations from the classiﬁer, we hypothesize that learning to generate saliency maps given strong classiﬁer is a relatively simple process.
To test this hypothesis, we run a set of experiments severely limiting the number of training steps and unique examples that the masker is trained on.
The ImageNet dataset consists of 1,000 object classes, with up to 1,250 examples per class.
We run a set of experiments restricting both the number of unique classes seen as well as the number of examples seen per class while training the masker.
We also limit the number of training steps be equivalent to one epoch through the full training set.
Given the randomness associated with subsampling the examples, we randomly subsample classes and/or examples 5 times for each conﬁguration and compute the median score over the 5 runs.
We report results on the actual validation set for ILSVRC ’14 (LE) and test set for OpenImages (PxAP).
Examples of saliency maps for these models can be found in Figure 9.
We show the results for our CA model in Figure 5 and for the FIX model in Figure 8.
Strikingly, we ﬁnd that very few examples are actually required to train a working saliency map model.
In particular, training on just 10 examples per class produces a model that gets only 0.7 LE more than using all of the training data and only 2.7 more than the fully trained model.
On the other hand, we ﬁnd that the diversity of examples across classes is a bigger contributor to performance than the number of examples per class.
For instance, training on 10 examples across all 1,000 classes gets an LE of 38.5, which is lower than training on 125 examples across onlyclasses.
A similar pattern can be observed in the PxAP results.
Above all, these few-shot results indicate that training an effective saliency map model can be signiﬁcantly simpler and more economical than previously thought.
Saliency methods that require training a separate model such as Dabkowski & Gal (2017) and Zolna et al.
(2020) are cheap to run at inference, but require an expensive training procedure, compared to gradient-based saliency methods or methods involving a per-example optimization.
However, if training a masking model can be a lightweight procedure as we have demonstrated, then using masking models to generate saliency maps can now be a cheap and effective model interpretability technique.
{+1250 # Examples   # C l a ss e s 58. Localization Error (LE)   
 # Examples   29. Pixel Average Precision (PxAP)+}
D ISCUSSION AND CONCLUSIONS
In this work, we systematically evaluated the additive contribution of many proposed improvements to masking-based saliency map methods.
Among the methods we tested, we identiﬁed that only the following factors meaningfully contributed to improved saliency map generation: (1) using both masked-in and masked-out objectives, (2) using the later layers of the classiﬁer as input and (3) continually training the classiﬁer.
This simple setup outperforms other methods on WSOL metrics and PxAP, and passes a suite of saliency map sanity checks.
Strikingly, we also found that very few examples are actually required to train a saliency map model, and training with just 10 examples per class can achieve close to our best performing model.
In addition, our masker model architecture is extremely simple: a two-layer ConvNet.
Together, this suggests that learning masking-based saliency map extraction may be simpler than expected when given access to the internal representations of a classiﬁer.
This unexpected observation should make us reconsider both the methods for extracting saliency maps and the metrics used to evaluate them A CKNOWLEDGEMENTS
We gratefully acknowledge the support of Nvidia Corporation with the donation of some of the GPUs used in this research.
This work was supported in part by grants from the National Institutes of Health (P41EB017183) and the National Science Foundation (1922658)
A M ODEL D ETAILS A. M ASK R EGULARIZATION
Without regularization, the masking model may learn to simply mask in or mask out the entire image, depending on the masking objective.
We consider two forms of regularization.
The ﬁrst is L regularization over the mask pixels m , which directly encourages the masks to be small in aggregate.
The second is Total Variation (TV) over the mask, which encourages smoothness:
[Equation]
where i, j are pixel indices.
TV regularization was found to be crucial by Fong & Vedaldi (2017) and
Dabkowski & Gal (2017) to avoid adversarial artifacts.
Hence, we have:
[Equation]
Following Zolna et al.
, we only apply L1 mask regularization if we are using masked-in objective and the masked-in image is correctly classiﬁed, or we have a masked-out objective and the masked-out image is incorrectly classiﬁed–otherwise, no L1 regularization is applied for that example.
In cases where we have both masked-in and masked-out objective, we have separate λ M,in and λ M,out regularization coefﬁcients.
A. C ONTINUAL T RAINING OF THE C LASSIFIER
We largely follow the setup for training classiﬁer-agnostic (CA) models from Zolna et al.
Notably, when training on the masker objectives, we update θ M but note θ F , to prevent the classiﬁer from being optimized on the masker’s objective.
We maintain a pool of 30 different classiﬁer weights in our classiﬁer pool.
We point the reader to Zolna et al.
(2020) for more details.
A. H YPERPARAMETERS
We show in Table 2 the space of hyperparameter search and hyperparameters for the best results, as show in the Table 1 in the main paper.
We performed a random search over λ M,out , λ M,in , and λ TV .
Aside from the hyperparameters shown in Table 2, we used a learning rate of 0.001 and a batch size of 72.
We trained for 3.4 epochs (17 epochs on 20% of data) for all Train-Validation experiments and for 12 epochs for the Validation set experiments.
Likewise, we use a learning rate decay of 5 for
Train-Val experiments and 20 for Validation set experiments.
For dual objective models, we used λ out = λ in − 0 .
We use the Adam optimize with 0.9 momentum and 1e-4 weight decay.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
hyperparameters for the best model, corresponding to results shown in Table 1, are underlined.
B S UPPLEMENTARY R ESULTS B. B. V ARYING O BSERVED C LASSIFIER L AYERS
We show the full set of per-layer and layer-combination results in Table 4.
B. C OUNTERFACTUAL I NFILLING AND B INARY M ASKS
We show examples of saliency maps generated with counterfactual inﬁllers in Figure 6 and incorporation of Gumbel-Softmax to generate masks with binary pixel values in Figure 7.
B. S ANITY C HECKS
We show in Table 5 the results for the Data Randomization Test (DRT) proposed by Adebayo et al.
Here, we train a new classiﬁer on the same training data but with labels shufﬂed across all images.
We ﬁnd that the similarity of saliency maps generated given a regularly trained classiﬁer compared to those given a classiﬁer trained on shufﬂed labels is low, indicating that the saliency maps reﬂect information learned from a well-formed image classiﬁcation task.
B. F EW - SHOT E XPERIMENTS
We show in Figure 8 the results for few-shot experiments using the FIX model conﬁguration.
We similarly ﬁnd that very few examples are needed to train a good saliency map model.
[Table]
[Figure] [Table]
MaxClass (I) + MaxEnt
[Table]
O)
[Table]
Layer[3] [Table] CA + MaxClass (I) + MaxEnt [Table]
+ Layer[4] [Table] CA + MaxClass [Table]
I)
[Table]
[Table]
CA
[Table]
I [Table] [Table] [Table]
[Figure]
[Figure] [Table]
[Figure]
{+Localization Error (LE)   
+}
# Examples   29.
Pixel Average Precision (PxAP
[Figure]
[Figure]