S ALIENCY IS A P OSSIBLE R ED H ERRING W HEN D IAGNOSING P OOR G ENERALIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Poor generalization is one symptom of models that learn to predict target variables using spuriously-correlated image features present only in the training distribution instead of the true image features that denote a class.
It is often thought that this can be diagnosed visually using attribution (aka saliency) maps.
We study if this assumption is correct.
In some prediction tasks, such as for medical images, one may have some images with masks drawn by a human expert, indicating a region of the image containing relevant information to make the prediction.
We study multiple methods that take advantage of such auxiliary labels, by training networks to ignore distracting features which may be found outside of the region of interest.
This mask information is only used during training and has an impact on generalization accuracy depending on the severity of the shift between the training and test distributions.
Surprisingly, while these methods improve generalization performance in the presence of a covariate shift, there is no strong correspondence between the correction of attribution towards the features a human expert has labelled as important and generalization performance.
These results suggest that the root cause of poor generalization may not always be spatially deﬁned, and raise questions about the utility of masks as “attribution priors” as well as saliency maps for explainable predictions.
I NTRODUCTION
A fundamental challenge when applying deep learning models stems from poor generalization due to covariate shift (Moreno-Torres et al., 2012) when the probably approximately correct (PAC) learning i.i.d.
assumption is invalid (Valiant, 1984) i.e. the training distribution is different from the test distribution.
One explanation for this is shortcut learning or incorrect feature attribution , where the model during training overﬁts to a set of training-data speciﬁc decision rules to explain the training data instead of modelling the more general causative factors that generated the data (Goodfellow et al., 2016;
Reed & Marks, 1999; Geirhos et al., 2020; Hermann & Lampinen, 2020; Parascandolo et al., 2020; Arjovsky et al., 2019; Zhang et al., 2016).
In medical imaging, poor generalization due to test-set distribution shifts are common and this problem is exacerbated by small cohorts.
Previous work has hypothesized that this poor generalization is in part due to the presence of confounding variables in the training data such as acquisition site or other image acquisition parameters because attribution maps (aka saliency maps; Simonyan et al.
(2014)) produced by the trained model do not highlight features that a human expert would use to make a diagnosis (Zech et al., 2018;
DeGrave et al., 2020; Badgeley et al., 2019;
Zhao et al., 2019; Young et al., 2019).
Previous researchers have made the assumption that saliency maps can demonstrate that the model is not overﬁt or behaving unexpectedly (Pasa et al., 2019; Tschandl et al., 2020).
We started this work under the same assumption only to ﬁnd the contradictory evidence we present in this paper.
In this work, we set out to test the hypothesis that models with good generalization properties have attribution maps which only utilize the class-discriminative features to make predictions, by explicitly regularizing the models to ignore confounders using attribution priors (Erion et al., 2019; Ross et al., 2017), i.e., to make predictions using the correct anatomy (as a doctor would).
We evaluated whether this regularization would A) improve out of distribution generalization, and B) change feature attribution to be more like the attribution priors.
If there exists a relationship between the attribution map and generalization performance, we would expect these regularizations to positively impact both generalization and attribution quality simultaneously.
To evaluate attribution quality, we deﬁne good attribution to be an attribution map that agrees strongly with expert knowledge in the form of a binary mask on the input data.
We show that the existing and proposed feature-attribution-aware methods help facilitate generalization in the presence of a train-test distribution shift.
However, while feature-attribution-aware methods change the attribution maps relative to baseline, there is no strong correlation between generalization performance and good attribution.
This in turn challenges the assumption made in previous works that the “incorrect” attribution maps were indicative of poor generalization performance.
This suggests that A) efforts to validate model correctness using attribution maps may not be reliable, and B) that efforts to control feature attribution using masks on the input may not function as expected.
All code and datasets for this paper are publicly available 1 .
Our contributions include:
• A synthetic dataset that encourages models to overﬁt to an easy to represent confounder instead of a more complicated counting task.
• Two new tasks constructed from open medical datasets which have a correlation between the pathology and either imaging site (site pathology correlation; SPC) or view (view pathology correlation; VPC), and we manipulate the nature of this correlation differently in the training and test distributions to create a distribution shift (Figure 5), introducing confounding variables as observed in previous work (Zhao et al., 2019; DeGrave et al., 2020).
• Evaluation of existing methods for controlling feature attribution using mask information; right for the right reasons ( RRR ; Ross et al.
(2017)), GradMask (Simpson et al., 2019), and adversarial domain invariance (Tzeng et al., 2017; Ganin & Lempitsky, 2015).
• A new method for controlling feature attribution based on minimizing activation differences between masked and unmasked images ( ActDiff ).
• Evaluate the relationship between generalization improvement and feature attribution in reallife out of distribution generalization tasks with traditional classiﬁers.
R ELATED WORK
It is a well-documented phenomenon that convolutional neural networks (CNNs), instead of building object-level representations of the input data, tend to ﬁnd convenient surface-level statistics in the training data that are predictive of class (Jo & Bengio, 2017).
Previous work has attempted to reduce the model’s proclivity to use confounding features by randomly masking out regions of the input (DeVries & Taylor, 2017), forcing the network to learn representations that aren’t dependent on a single input feature.
However, this regularization approach gives no control over the kinds of representations learned by the model, so we do not include it in our study.
Recently, multiple approaches have proposed to control feature representations by penalizing the model for producing saliency gradients outside of a regions of interest indicating the classdiscriminative feature (Simpson et al., 2019;
Zhuang et al., 2019; Rieger et al., 2019).
These approaches were introduced by Right for the Right Reasons ( RRR ), which showed impressive improvements in attribution correctness on synthetic data (Ross et al., 2017).
The follow-up work has generally demonstrated a small improvement in generalization accuracy on real data, and much more impressive results on synthetic data.
Another feature attribution-aware regularization approach additionally dealt with class imbalances by increasing the impact of the gradients inside the region of interest of the under-represented class Zhuang et al.
One alternative to saliency-based methods, which can be noisy due to the ReLU activations allowing irrelevant features to pass through the activation function (Kim et al., 2019), would be to leverage methods that aim to produce domain invariant features in the latent space of the network.
These methods regularize the network such that the latent representations of two or more “domains” are encouraged to be as similar as possible, often by minimizing a distance metric or by employing an adversary that is trained to distinguish between the different domains (Kouw & Loog, 2019; Ganin & Lempitsky, 2015; Tzeng et al., 2015; Liu & Tuzel, 2016).
In this work, we view the masked version of the input as the training domain and the unmasked version of the input as the test domain, and compare these approaches with saliency-based approaches for the task of reducing the model’s reliance on confounding features.
To the best of our knowledge, these strategies have not been tried to control feature attribution.
M ETHODS
Domain Invariance with the Activation Difference and Adversarial Loss: Leveraging ideas from domain adaptation, we introduce two methods designed to make the model invariant to features arising from outside of the attribution priors .
The ﬁrst is the embarrassingly simple Activation
Difference ( ActDiff ) approach, which simply penalizes, for each input, the L 2 -normed distance between the masked and unmasked input’s latent representations.
This loss is most similar to a style transfer approach which transforms a random noise image into one containing the same layerwise activations as some target image (Gatys et al., 2015) to apply a visual aesthetic to some input semantic contents, and encourages the network to build features which appear inside the masked regions even though it always sees the full image during training.
Therefore we minimize
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
S ha r ed
W e i gh t s
The domain invariance penalties, ActDiff and Adversarial , are applied to the linear layer after average pooling although they could be applied at any in where o l ( x ) are the pre-activation outputs for layer l of the n -layer encoder f
( x ) when the network is presented with the original data x , o l ( x masked ) are the pre-activations outputs for layer l when presented with masked data x masked , and L clf is the standard cross entropy loss.
We deﬁne x masked = x · x seg + shufﬂe ( x ) · (1 − x seg ) where background pixels are shufﬂed uniquely for each presentation to the network in order to destroy any spatial information available in those regions of the image without introducing any consistent artefacts into the image or shifts in the distribution of input intensities.
This method is at a high level similar to using maximum mean discrepancy (MMD) for domain adaptation (Baktashmotlagh et al., 2016; 2013) where instead of minimizing the distance between the means of the two domain distributions, we instead minimize the distance between the features directly.
One could replace the L 2 norm with any Lk norm, and choices of k < 2 might be useful when regularizing larger latent representations as L 2 distances collapse to a constant value in extremely high dimensional spaces due to the curse of dimensionality (Aggarwal et al., 2001).
We found during experimentation that regularizing the pre-activations led to better results at the cost of longer time to convergence, perhaps because the L 2 norm is more effective when the feature vectors are not sparse, but we leave this conjecture to future work.
The second approach we explore employs a discriminator D ( · ) optimized to distinguish between latent representations arising from passing x masked or x through the encoder f ( · ) (Tzeng et al., 2017; Goodfellow et al., 2014).
Simultaneously, we optimize the encoder f ( · ) to fool the discriminator and still produce representations that are predictive of the output class.
Therefore, D and f are optimized using the L D and L f respectively:
[Equation]
[Equation]
[Equation]
This approach is similar to the one employed in (Janizek et al., 2020), where the authors encouraged the model to be invariant to determining the class label by penalizing saliency outside of a deﬁned input mask.
The most basic gradient based “input feature attribution” can be calculated as ∂ | ˆ y i | ∂ x for each input x (Simonyan et al., 2014;
Lo et al., 2015).
In the binary classiﬁcation case, RRR (Ross et al., 2017) calculates saliency of the summed log probabilities of the 2 output classes with respect to the input x ,
[Equation]
where (1 − x seg ) is a binary mask that covers everything outside the deﬁned regions of interest.
The numerator in the RRR loss can be extended to the multi-class case by substituting (cid:80)
Kk log ( ˆ p k ) .
GradMask (Simpson et al., 2019) similarly calculates saliency using the contrast between the two output logits, and is only deﬁned for the binary classiﬁcation case,
[Equation]
where ˆ y 0 and ˆ y 1 are the predicted outputs for our two classes.
Classify Masked: We evaluated the effect imposing the attribution prior by simply training a model using masked data (and evaluating it using unmasked data) as a control experiment.
The data was masked by shufﬂing the pixels outside of the mask during training, as was done for the domain invariance experiments to produce x masked .
We refer to these experiments as Masked .
E XPERIMENTS
Feature Attribution Analysis and Visualization: We evaluated our ability to control feature attribution using three methods.
We ﬁrst calculated the input gradient as the absolute gradient G of the input with respect to the prediction made for all images of the positive class G = | ∂
ˆ y 1 ∂ x | (Simonyan et al., 2014; Ancona et al., 2018).
We also calculated integrated gradients , which produces attribution maps that are invariant to the speciﬁc model trained (or function f ( · ) ) and are sensitive to all input features that drive the prediction (Sundararajan et al., 2017).
This is done by integrating the gradients along the straightline path between the input image x i and an all-zero baseline image x (cid:48)
i ,
IG
[Equation]
∂α .
The integral was approximated over 200 steps for each image using GaussLegendre quadrature.
We ﬁnally evaluated a non gradient-based attribution method occlusion where we divided the input image into a 15 × 15 grid, and for each element of the grid, recorded the magnitude of the change to the model’s output logits if that region is removed from the input (by setting these regions to zero) (Zeiler & Fergus, 2014).
For the two gradient-based attribution methods, we smoothed the resulting saliency map with a small gaussian kernel ( σ = 1 ) to remove pixel-wise variance in the attribution values that amount to noise.
The attribution maps were then carried forward to calculate a binarized version such that the top p percentile attribution values were set to 1, where p is dynamically set to the number of pixels in the mask that it is being compared to.
These binarized values were used to calculate the localization accuracy between binarized attribution map and the ground-truth segmentation using the intersection over union (IoU; I u ( A, B ) =
A ∩ B A ∪ B ) for all images in the test sets for all models trained.
To aid in visualization, we also thresholded these attribution maps arbitrarily at the 50 th percentile so that the most attributed regions of the image are easily seen when overlaid on the anatomy.
We present visualizations of the results for all three methods in the Appendix.
Model, Optimization, Hyperparameters, and Search Parameters: The backbone of all experiments was the ResNet-18 model from Torchvision (He et al., 2016; Marcel & Rodriguez, 2010).
For medical dataloaders we use the TorchXRayVision library (Cohen et al., 2020b;a).
To see an overview of how the losses described above relate to the architecture, see Figure 1.
All models were trained using the Adam optimizer (Kingma & Ba, 2014) using early stopping on the validation loss.
Hyperparamters were selected for all models using a Bayesian hyperparameter search with a single seed, 5 random initializations, 20 iterations, and trained for a maximum of 100 epochs with a patience of 20.
Final models were trained using the best hyperparameters found for 100 epochs onseeds.
The learning rate for all models was searched on a log uniform scale between [10 − 5 10 − 2 ] .
The ActDiff , Adversarial , GradMask , and RRR lambdas were all searched between [10 − 4 10] , each on a log uniform scale.
For Adversarial , we searched for the optimal discriminator : encoder training ratio of [2 : 1 10 : 1] , and the discriminator learning rate was searched on a log uniform scale between [10 − 4 10 − 2 ] .
To retain context around the masked region in the Synthetic dataset, we dilated the mask by applying a Gaussian blur with a hyperparameter σ to the mask and then binarize the result; σ was searched on a uniform scale between [0 2] .
For all experiments, the batch size was 16, weight of the classiﬁcation cross entropy loss was 1.
In performing our hyperparameter searches, we found that certain algorithms were much easier to tune than others.
We present the ﬁnal best validation AUCs encountered for all iterations for all experiments in Appendix Figure A. and Table 3.
4. S YNTHETIC D ATASET
Protocol: Consider a classiﬁcation problem where there exists some confounder feature x c in the data (a vector of variables x ) that is perfectly correlated with one of the output classes y in the training distribution D train such that p ( y = 1 | x c ) = 1 , while in the validation distribution D valid , p ( y =
| x c ) =
1 (Figure 2).
In this scenario, predicting using x c is easier than predicting using the true features that would denote class membership and a classiﬁer trained on D train with traditional classiﬁcation loss would predict the incorrect class with 100% probability on D valid .
We generated a dataset with these conditions where class membership is denoted by the number of crosses present in the image (1 vs. 2 crosses) that can appear in any position, but there exists in all D train images a confounder that always appears in the same position and is perfectly correlated with y (Figure 2).
The logic governing the position of the confounder was inverted for D valid .
This dataset had 500 training, 128 validation, and 128 test examples respectively.
Segmentations were generated for all crosses in the dataset to facilitate all feature-attribution controlling losses.
In cases where the model relies on the confounder to make a class prediction, we expected 0.0 AUC for the validation and test sets.
Results: The mean and standard deviation of the validation AUC over all 10 models for the 100 epochs of training for the best-performing hyperparameters are shown in Figure 3, showing the variance in performance over model initializations and data splits.
We demonstrate the effect of controlling feature attribution by showing the mean integrated gradient map calculated for 500 positive examples in the test set in Figure 4.
The validation curves (Figure 3), demonstrate that the baseline model quickly learns the shortcut in the training set and shows below-chance generalization (AUC=0).
The Masked approach is never given the opportunity to build a representation of the confounder and is therefore able to generalize while still giving attribution to the confounder.
The domain invariance approaches, ActDiff and
Adversarial also learn the task, although convergence is much more difﬁcult for the Adversarial approach.
The two saliency-based feature-attribution regularizers, GradMask , and RRR , show unstable training and never reach optimal performance, although they do prevent the model from overﬁtting to the confounder.
The saliency maps for these models (Figure 4) show that all feature attribution controlling approaches (C-F) successfully refocus the model’s attention away from the confounder, unlike the baseline and Masked models (A-B), leading to improved generalization.
This strong re- lationship between IoU and AUC for this dataset can be observed for all seeds in Figure 8.
As we will see, this relationship does not hold for real world data.
4. X-R AY D ATASET WITH S ITE -P ATHOLOGY C ORRELATION
Protocol: We introduced a covariate-shift into a joint dataset of X-Rays drawn from two different imaging centres: the PadChest (Bustos et al., 2019) sample and the NIH Chestx-Ray8 (Wang et al., 2017) sample.
In Figure 5, we can see examples of the distribution shift between imaging site (top) and imaging view (bottom; used in the following section).
In the case that the prevalence of disease is in any way correlated with either of these confounders, the model is likely to learn to use these differences to make a prediction regardless of their medical relevance (rightmost column).
In this dataset, models were required to predict the presence of Cardiomegaly (enlarged heart).
A site-driven overﬁtting signal has previously been reported when combining these datasets (Zech et al., 2018), where the model often attributes importance to the patient’s shoulder when making a prediction.
We also observed site-driven differences in regions far from the lungs (see the mean X-Ray from each dataset in Figure 5), and therefore hypothesized we could improve overﬁtting performance by masking out the edges of the image using a circular mask.
To induce the covariate shift, we preferentially sampled the positive class from one of the two imaging centres in the training set to produce a site-pathology correlation (SPC), and then produce validation and test set where the reverse relationship is true.
In the 90% of unhealthy patients were drawn from the PadChest dataset and the remaining 10% of the unhealthy patients were drawn from the NIH dataset, and the reverse logic was followed for the validation and test sets.
In all splits the classes and site distributions were always balanced, making it tempting for the classiﬁer to use a site-speciﬁc feature when predicting the class in the presence of site-pathology correlation ( N train = 5542 , N valid = 2770 , N test = 2770 ).
We also constructed a version of this dataset where the classes are drawn equally from both sites for the training, validation, and test sets, i.e., a dataset where there exists no site pathology correlation (No SPC).
All images were 224 × 224 pixels.
Results: We trained each model on both
the
SPC (90%) and No SPC datasets (Table 1).
A
ResNet trained on the No SPC dataset scores a test AUC of 0 .
93 ± 0 .
01 , while one trained in the presence of a strong SPC scores a test AUC of 0 .
70 ± 0 .
05 , indicating poorer generalization under a SPC.
Validation curves across seeds show that model optimization was much easier and more consistent for all models when trained on the
No SPC dataset (Appendix Figure A.2).
The ActDiff , Masked , and GradMask approaches only improved classiﬁcation performance over the baseline model in the presence of an SPC ( GradMask performed only slightly better than baseline for the No SPC dataset).
For ActDiff and GradMask , an improvement in classiﬁcation performance was associated with an improvement in attribution (measured as an improvement in the IoU between the binarized saliency map and the masks), but this pattern was not observed for the masked experiment, even though the Masked experiment produced superior generalization (it is worth noting that the Masked performance was more variable across seeds). Furthermore, the GradMask approach improved attribution in the No SPC case, but this was [Figure]
[Table] [Figure] [Table] not associated with a meaningful improvement in generalization performance.
Figure 6 shows the mean saliency map calculated from a subset of all test images from all 10 models.
Differences in the baseline model’s attribution (A) in the SPC and No SPC case are apparent: in particular, the SPC model shows a greater reliance on the shoulder for producing a prediction.
The Masked experiment (B) exacerbates this behavior but is accompanied by improved generalization with higher variance across seeds.
In contrast, ActDiff and GradMask appear to refocus the saliency away from the shoulder (D-E), reﬂected in the increased IoU scores for both the ActDiff and GradMask approaches for the SPC data.
The weak relationship between IoU and AUC can be observed for all seeds in Figure 8.
These results provide evidence that controlling feature attribution only facilitates generalization in the presence of a covariate shift, but improves saliency in both the presence or absence of a covariate shift, implying a weak relationship between good attribution and good generalization.
4. X-R AY D ATASET WITH V IEW -P ATHOLOGY C ORRELATION
Protocol: Here we made use of the RSNA Pneumonia challenge (Shih et al., 2019) dataset, which includes images taken from antero-posterior (AP) and posterior-anterior (PA) views, and where the task is to predict pneumonia.
We introduced a covariate shift into this dataset by sampling the positive and negative classes during training such that 90% of the positive classes are sampled from one view in the training set and the opposite view in the validation and test sets.
We call this a viewpathology correlation (VPC).
After drawing the samples from both views and balancing the classes, the data in each split was as follows: N train = 2696 , N valid = 1348 , N test = 1348 .
All images with Pneumonia have one or more bounding boxes indicating the predictive regions.
In Figure 5 we can see the strong effect of the VPC, such that the lungs, shoulders, and torso show strong interclass differences in the VPC case.
We also constructed a No VPC dataset in the same way as during the previous experiments, and all images were 224 × 224 pixels.
[Figure]
[Table]
Results: Similarly to the SPC case, baseline AUC in the presence of VPC ( 0 . 20 ± 0 . 03 ) is substantially lower than the No VPC baseline ( 0 .
76 ± 0 . 02 ), and additionally far below chance performance ( 0 . 50 ), illustrating the strong covariate shift in this dataset.
Appendix Figure A.2 shows that validation performance was more variable in the presence of VPC between models, and that performance was generally lower.
The ActDiff , Masked , and Adversarial approaches improved classiﬁcation performance over the baseline model and chance performance in the presence of VPC.
For only the
GradMask model was an improvement in classiﬁcation performance associated with an improvement in saliency, and the Masked experiment produced IoU scores substantially worse than baseline (again, Masked performance was more variable across seeds).
No model improved attribution or generalization performance over baseline in the No VPC case.
Figure 7 shows the mean saliency map calculated from a subset of all test images from all 10 models.
Differences in the Baseline model’s saliency (A) in the VPC and No VPC case are apparent: in particular, the VPC model shows a greater reliance on the abdomen and shoulders for producing a prediction.
The Masked experiment (B) again exacerbates this behavior but is accompanied by improved generalization over baseline.
The Adversarial , ActDiff , and GradMask models (C-E) appear to refocus the attribution toward the lungs for for the VPC case, but the mean Adversarial IoU score dropped over the full dataset, and ActDiff resulted in a minor IoU score increase.
The GradMask approach produced the best attribution as measured by IoU but does not produce a model that outperforms random guessing on the test set.
Again, these results suggest that the use of attribution priors can have a substantial effect on generalization performance in the absence of meaningful changes to the saliency towards what would be expected from the attribution priors, and vice versa.
The weak relationship between
IoU and AUC can be observed for all seeds in Figure 8.
L IMITATIONS AND C ONCLUSIONS
Limitations: Our results show that models explicitly regularized to not use features constructed from outside of the mask still attribute features outside of the mask at test time.
This is likely due to the fact that convolutional models are still able to construct discriminative ﬁlters from inside of the mask which might share properties with features found outside of the mask.
The model is free to use any aspect of the input image to make a prediction during test time.
In particular, the methods we explored are not guaranteed to negate any confounding variables that exist within the mask, which could explain these results, and might additionally give rise to underspeciﬁed trained models, as evidenced by the models with high test set variability across seeds (D’Amour et al., 2020). This is particularly true for the Masked models, for which the representations constructed from inside of the masks were not regularized in any way.
Methods that address this underspeciﬁcation would make for valuable future work. While these results are suggestive that attribution maps are a misleading indicator of generalization performance, one limitation of this study is that we evaluated these algorithms in a limited domain: small sample sizes with large covariate shifts between the training and test distributions in a medical context.
Future work should more thoroughly evaluate the relationship between generalization performance and feature attribution before we can draw strong conclusions about the non-existence of this relationship.
Furthermore, methods which control feature attribution without using masks for enforcing domain knowledge might show a stronger relationship between IoU and AUC, and future work should explore this possibility.
Conclusions: We hypothesized that poor generalization performance could be partially attributable to classiﬁers exploiting spatially-distinct confounding features (or shortcuts) as have been previously diagnosed using saliency maps (Badgeley et al., 2019; Zech et al., 2018).
We evaluated the performance of multiple mitigation strategies on a synthetic dataset and two real-world datasets exhibiting covariate shift, using attribution priors in the form of a segmentation of the discriminative features.
vs. Test IoU (good saliency) for all seeds evaluated with covariate shift.
Models presented grouped by “Approaches”: baseline, input masking, domain-invariance, and saliency-based approaches.
All localizations were computed using integrated gradients, see Appendix Figure A.13 for all attribution methods.
In our synthetic dataset, we deﬁned a spatially-distinct confounder that prevented generalization for standard classiﬁers, and demonstrated that imposing an attribution prior would facilitate generalization and improve attribution maps.
In real data, we found that attribution priors also facilitate generalization when there exists a covariate shift between the training and testing distribution, but these methods hurt generalization when there was not a covariate shift.
Surprisingly, improvement in generalization performance was not reliably accompanied by improvement in attribution: this is particularly apparent for the ActDiff model on the
RSNA VPC dataset, which had a meaningful positive impact on AUC and a very negative impact on attribution.
Figure 8 summarizes the relationship between generalization (AUC) and feature attribution (IoU) across all seeds for all experiments with a covariate shift: the positive correlation between AUC and IoU apparent for the synthetic data was not present in the real world datasets.
In summary, we ﬁnd a tenuous relationship between good saliency and generalization performance in real datasets.
Many models that exhibit good generalization performance do not obtain good attribution, and vice-versa.
While our methods exert inﬂuence on the features the model constructs from the data, in real data we found no evidence of a relationship between improved generalization and improved feature attribution.
We now doubt the validity of using saliency maps for diagnosing whether a model is overﬁt because improving generalization using attribution priors is not accompanied by improved attribution.
A APPENDIX A. A RCHITECTURE OF THE R ES N ET -18 M ODEL
The backbone of the model used for all experiments was the 18-layer resnet available from torchvision (Marcel & Rodriguez, 2010) with a small architecture adjustment near the input of the network.
The ofﬁcial model’s input convolutional layer had a kernel size of 7, stride of 2, and padding of 3.
In our experiments, we found this larger kernel size on the inputs led to less speciﬁc saliency maps (data not shown), and we therefore changed the kernel size to 3, stride to 1, and padding to 1 for this layer.
Immediately following this layer, the ofﬁcial model employed a maxpooling operation.
We found this operation to produce unusable noisy saliency maps, so we also disabled this operation for all experiments.
A. H YPERPARAMETER S ENSITIVITY
A.
V ALIDATION C
FOR THE R EAL W
D ATASETS
A. B EST H YPERPARAMETERS FOR A LL E XPERIMENTS
[Figure]
[Table]
A. V ISUALIZATIONS OF I NPUT G RADIENTS , I NTEGRATED G RADIENTS , AND O CCLUSION -
A. S ALIENCY M APS FOR I NCORRECT AND C ORRECT P REDICTIONS
Here, we visualize the mean gradients from samples the model gets correct (top) vs. incorrect (bottom) for 100 samples from each seed for a total of 1000 samples.
While the synthetic dataset shows obvious differences for correct and incorrect samples, no such pattern is obvious for the real world datasets.
[Figure]
n t h
[Figure]
t i c
[Figure]
I G
[Figure]
y n t h e t i c : I G
[Figure]
[Figure] [Figure]
A: Baseline
B: Masked
C:
Adversarial
D: ActDiff
E: GradMask
F: RRR
N o S
[Figure]
: I G
[Figure]
A: Baseline
B: Masked
C:
Adversarial
D: ActDiff
E: GradMask
F: RRR
[Figure]
ActDiff
E
[Figure]
GradMask
F
[Figure]
N o V P C : I G
Image
A: Baseline
B: Masked
C:
Adversarial
D: ActDiff
E: GradMask
F: RRR
N o V P C : I G
Image
A: Baseline
B: Masked
C:
Adversarial
D: ActDiff
E: GradMask
F: RRR
[Figure]
ENERALIZATION A BILITY
A TTRIBUTION
[Figure]
A. A CTDIFF L AMBDA E VALUATION
In real-world experiments without a covariate shift we found the ActDiff method performed worse than baseline.
We suspected this was due to the minimum lambda value in the range considered during hyperparameter tuning ( [1 × 10 − 4 10] being too large: the ActDiff model should perform equivalently to the baseline if this lambda value is small enough.
We tested this on both datasets by running a new hyperparamater search with an expanded range for the actdiff lambda [1 × 10 − 16 10] which we call the wide search, and report the test set values over 10 seeds in Table 4.
While we recovered the baseline performance for the No SPC dataset, we were unable to do so on the No VPC dataset.
We suspect that the ActDiff penalty is quite strong in the RSNA data due to the smaller input masks, making even very small lambda values of ActDiff a powerful regularizer of the model, and making the hyperparameter search difﬁcult given a ﬁxed compute budget.
We conclude that the difﬁculty of tuning the hyperparamaters in situations where no covariate shift exists in the data is a potential downside of this approach
[Table]
seeds
[Figure]
A. I NDIVIDUAL S AMPLES
A. A NALYSIS OF M ODEL P ERFORMANCE W ITH E QUIVALENT S KEW IN T RAIN AND V ALIDATION S ETS
The hyperparameter searches in this paper rely on the existence of a covariate shift between the training and validation distributions: generalization performance to the validation set is used to select the regularization lambdas which encourage the model to ignore the confounder.
In the typical machine learning setup, the training and validation set are drawn IID from the same distribution.
As we saw on the baseline experiments, where there was no covariate shift in the data and the train/valid distributions were the same, the regularization terms can hurt performance and a hyperparamater search will seek to reduce their impact (since building representations of any confounders will produce good validation performance).
This is why in our original experiments it was important that the covariate shift not be equivalent for the train and validation sets, and it is what we recommend in practice (this is easy to accomplish because the exact sources of the covariate shift do not need to be known, so the data can simply be sampled from some different environment where many background variables are likely to differ from the training distribution).
Here, we present the results of 5 seeds run for all models on both the X-Ray SPC and RSNA VPC datasets, where the train and validation sets both exhibited a 90/10 skew (exactly matching the skew of the training sets in the original paper), and the test set exhibits a 10/90 skew.
We set all hyperparamaters to be the same as found during the search made for the main paper, except for setting the regularization term’s lambda to 1 for all methods (where applicable) since this number cannot be searched for because there is no covariate shift in the validation set to determine which lambda works (also see the results in Table 4).
In all cases we observe high validation performance and poor test performance.
Interestingly, the method which performs best in both cases is the Masked approach, which has no lambda to tune.
For both datasets, it reduces the validation performance in exchange for improved test performance.
The gradmask approach also improves test performance in the X-Ray SPC dataset.
In total we take these results to suggest that these methods must be used in cases where there exists some known or unknown distribution shift between the training and validation sets to set the correct lambda: they should not both be drawn IID from the same underlying distribution
[Table]