S ALIENT I MAGE N ET : H OW TO DISCOVER SPURIOUS FEATURES IN D EEP L EARNING ?
[Figure]
A BSTRACT
Deep neural networks can be unreliable in the real world especially when they heavily use spurious features for their predictions.
Focusing on image classiﬁcations, we deﬁne core features as the set of visual features that are always a part of the object deﬁnition while spurious features are the ones that are likely to co-occur with the object but not a part of it (e [Figure] g., attribute “ﬁngers” for class “band aid”).
Traditional methods for discovering spurious features either require extensive human annotations (thus, not scalable) [Figure] or are useful on speciﬁc models.
In this work, we introduce a general framework to discover a subset of spurious and core visual features used in inferences of a general model and localize them on a large number of images with minimal human supervision.
Our methodology is based on this key idea: to identify spurious or core visual features used in model predictions , we identify spurious or core neural features (penultimate layer neurons of a robust model) via limited human supervision (e g., using top activating images per feature).
We then show that these neural feature annotations generalize extremely well to many more images without any human supervision.
We use the activation maps for these neural features as the soft masks to highlight spurious or core visual features.
Using this methodology, we introduce the Salient Imagenet dataset containing core and spurious masks for a large set of samples from Imagenet.
Using this dataset, we show that several popular Imagenet models rely heavily on various spurious features in their predictions indicating the standard accuracy alone is not sufﬁcient to fully assess model’ performance specially in safety-critical applications. Code is available at https://github.com/singlasahil14/salient_imagenet .
I NTRODUCTION
The growing use of deep learning systems in sensitive applications such as medicine, autonomous driving, law enforcement and ﬁnance raises concerns about their trustworthiness and reliability in the real world.
A key reason for the lack of reliability of deep models is their reliance on spurious input features (i.e., features that are not essential to the true label) in their inferences.
For example, a convolutional neural network (CNN) trained to classify camels from cows associates green pastures with cows and fails to classify pictures of cows in sandy beaches correctly since most of the images of cows had green pastures in the training set (Beery et al., 2018; Arjovsky et al., 2020).
Similarly,
Bissoto et al.
(2020) discovered that skin-lesion detectors use spurious visual artifacts for making predictions.
The list of such examples goes on (de Haan et al., 2019; Singla et al., 2021).
Most of the prior work on discovering spurious features (Nushi et al., 2018; Zhang et al., 2018;
Chung et al., 2019) requires humans to ﬁrst come up with a possible list of spurious features.
This is often followed by an expensive human-guided labeling of visual attributes which is not scalable for datasets with a large number of classes and samples such as ImageNet.
In some cases, this may even require the construction of new datasets with unusual visual attributes to validate the hypothesis (such as cows in sandy beaches discussed previously).
To address these limitations, recent works (Singla et al., 2021; Wong et al., 2021) use the neurons of robust models as visual attribute detectors thereby circumventing the need for manual annotations. However, their applicability is either limited to robust models which achieve low clean accuracy or to models that achieve low accuracy on the training dataset.
We discuss limitations of these methods in more detail in Section 2.
In this work, we introduce a general methodology to discover a subset of spurious and core (nonspurious) visual attributes used in model inferences and localize them on a large number of images with minimal human supervision.
Our work builds upon the prior observations (Engstrom et al., 2019) that for a robust model, neurons in the penultimate layer (called neural features ) often correspond to human-interpretable visual attributes.
These visual attributes can be inferred by visualizing the heatmaps (e.g. via CAM, Zhou et al.
that highlight the importance of image pixels for those neural features (top row in Figure 1).
Alternatively, one can amplify these visual attributes in images by maximizing the corresponding neural feature values via a procedure called the feature attack (Engstrom et al.
; Singla et al.
(2021), bottom row in Figure 1).
Our framework (shown in Figure 2) is based on this key idea: to identify spurious or core visual attributes used for predicting the class i , we identify spurious or core neural features via limited human supervision.
We deﬁne a core visual attribute as an attribute for class i that is always a part of the object deﬁnition and spurious otherwise.
To annotate a neural feature as core or spurious, we show only the top- 5 images (with a predicted label of i ) that maximally activate that neural feature to
Mechanical Turk workers.
We then show that these neural feature annotations generalize extremely well to top- k
(with k (cid:29) 5 ) images with label i that maximally activate that neural feature.
For example, on Imagenet, in Section 3.4, we show a generalization accuracy of 95% for k = 65 (i.e., a 13 fold generalization).
Thus, by using neural features and their corresponding neural activation maps (heatmaps), with limited human supervision, we can identify a large subset of samples that contain the relevant spurious or core visual attributes.
The neural activation maps for these images can then be used as soft segmentation masks for these visual attributes.
We emphasize that the usual method of obtaining such a set would require the manual detection and segmentation of a visual attribute across all images in the training set and is therefore not scalable.
We apply our proposed methodology to the Imagenet dataset: we conducted a Mechanical Turk study using 232 classes of Imagenet and 5 neural features per class.
For each neural feature, we obtained its annotation as either core (non-spurious) or spurious for the label.
Out of the 232 × 5 = 1 , 160 neural features, 160 were deemed to be spurious by the workers.
For 93 classes, at least one spurious neural feature was discovered.
Next, for each (class= i , feature= j ) pair, we obtainedimages with class i showing highest activations of the neural feature j and computed their neural activation maps.
The union of these images is what we call the Salient Imagenet dataset.
The dataset contains 52 , 521 images with around 226 images per class on average.
Each instance in the dataset is of the form ( x , y, M c , M s ) where y is the ground truth class and M s , M c represent the set of spurious and core masks, respectively (obtained from neural activation maps of neural features).
This dataset can be used to test the sensitivity of any pretrained model to different image features by corrupting images using their relevant masks and observing the drop in the accuracy.
Ideally, we expect our trained models to show a low drop in accuracy for corruptions in spurious regions and a high drop for that of the core regions.
However, using a standard Resnet-50 model, we discover multiple classes for which the model shows a trend contradictory to our expectations: a signiﬁcantly higher spurious drop compared to the core drop .
As an example, for the class drake (Fig.
5), we discover that corrupting any of spurious features results in a signiﬁcantly higher drop ( > 40% ) compared to corrupting any of core features ( < 5% ).
We show many more examples in Appendix
L. For various standard models (Resnet-50, Wide-Resnet-50-2, Efﬁcientnet-b4, Efﬁcientnet-b7), we evaluate their accuracy drops due to corruptions in spurious or core regions at varying degrees of corruption noise and ﬁnd no signiﬁcant differences between their spurious or core drops suggesting that none of the models signiﬁcantly differentiates between core and spurious visual attributes in their predictions (Fig. 6).
In summary, our methodology has the following steps:
• Step 1: We select neural features as penultimate-layer neurons of a pretrained robust network and visualize them using a few highly activating samples (Section 3.2).
• Step 2: Using Mechanical Turk, we annotate neural features as core or spurious (Section 3.3).
• Step 3:
Using neural feature annotations, we automatically annotate core and spurious visual attributes on many more samples without any human supervision (Section 3.3).
• Step 4: Using another Mechanical Turk study, we show that our methodology generalizes extremely well to identify core/spurious visual attributes on new samples (Section 3.4).
• Step 5:
Applying steps 1-4 to Imagenet, we develop a new dataset called Salient Imagenet whose samples, in addition to the class label, are annotated by core and/or spurious masks (Section 4).
• Step 6: Using such a richly annotated dataset, we assess core/spurious accuracy of standard models by adding small noise to spurious/core visual attributes, respectively (Section 5).
To the best of our knowledge, this is the ﬁrst work that introduces a general methodology to discover core and spurious features with limited human supervision by leveraging neurons in the penultimate layer of a robust model as interpretable visual feature detectors.
Also, this is the ﬁrst work that releases a version of Imagenet with annotated spurious and core image features.
We believe that this work opens future research directions to leverage richly annotated datasets to develop deep models that mainly rely on core and meaningful features in their inferences.
R ELATED WORK
Robustness and Interpretability :
In recent years, there have been many efforts towards post-hoc interpretability techniques for trained neural networks.
Most of these efforts have focused on local explanations where decisions about single images are inspected (Zeiler & Fergus, 2014;
Yosinski et al., 2016; Dosovitskiy & Brox, 2016; Mahendran & Vedaldi, 2016; Nguyen et al., 2016;
Zhou et al., 2018; Olah et al., 2018; Adebayo et al., 2018; Chang et al., 2019; Carter et al., 2019; Yeh et al., 2019; Sturmfels et al., 2020; O’Shaughnessy et al., 2019; Verma et al., 2020).
These include saliency maps (Simonyan et al., 2014; Smilkov et al., 2017; Sundararajan et al., 2017; Singla et al., 2019), class activation maps (Zhou et al., 2016;
Selvaraju et al., 2019;
Bau et al., 2020; Ismail et al., 2019; 2020), surrogate models to interpret local decision boundaries such as LIME (Ribeiro et al., 2016), methods to maximize neural activation values by optimizing input images (Nguyen et al., 2015;
Mahendran & Vedaldi, 2015) and ﬁnding inﬂuential inputs (Koh & Liang, 2017).
However, recent work suggests that class activation or saliency maps may not be sufﬁcient to narrow down visual attributes and methods for maximizing neural activations often produce noisy visualizations (Olah et al., 2017; 2018) with changes imperceptible to humans.
To address these limitations, recent works (Tsipras et al., 2018; Engstrom et al., 2019) show that for robust (Madry et al., 2018) or adversarially trained models (in contrast to standard models), activation maps are qualitatively more interpretable and optimizing an image directly to maximize a certain neuron produces human perceptible changes in the images which can be useful for visualizing the learned features.
Failure explanation : Most of the prior works on failure explanation either operate on tabular data where interpretable features are readily available (Zhang et al., 2018; Chung et al., 2019), language data where domain-speciﬁc textual queries can be easily generated (Wu et al., 2019; Ribeiro et al., 2020), images where visual attributes can be collected using crowdsourcing (Nushi et al., 2018;
Xiao et al., 2021; Plumb et al., 2021) or photorealistic simulation (Leclerc et al., 2021).
However, collecting visual attributes via crowdsourcing can be expensive for large datasets.
Moreover, these methods require humans to hypothesize about the possible failures and one could miss critical failure modes when the visual attributes used by model are different from the ones hypothesized by humans.
To address these limitations, recent works (Wong et al., 2021; Singla et al., 2021) use the neurons of robust models as the visual attribute detectors for discovering failure modes, thus avoiding the need for crowdsourcing.
However, Wong et al. (2021) cannot be used to analyze the failures of standard (non-robust) models which achieve signiﬁcantly higher accuracy and are more widely deployed in practice.
Moreover, to discover the spurious feature, they ask MTurk workers to identify the common visual attribute across multiple images without highlighting any region of interest.
However, different images may have multiple common attributes even if they come from different classes and this approach may not be useful in such cases.
Barlow (Singla et al., 2021) learns decision trees using misclassiﬁed instances to discover leaf nodes with high failure concentration.
However, the instances could be correctly classiﬁed due to some spurious feature (e.g., background) and Barlow will discover no failure modes in such cases.
In contrast, our approach circumvents these limitations and discovers the spurious features for standard models even when they achieve high accuracy.
A GENERAL FRAMEWORK FOR DISCOVERING S PURIOUS F EATURES
Consider an image classiﬁcation problem X
→ Y where the goal is to predict the ground truth label y ∈ Y for inputs x ∈ X .
For each class i ∈ Y , we want to identify and localize a set of core or spurious visual attributes in the training set that can be used by neural networks to predict i :
• Core Attributes are the set of visual features that are always a part of the object deﬁnition.
• Spurious Attributes are the ones that are likely to co-occur with the object, but not a part of it (such as food, vase, ﬂowers in Figure 1).
A more formal discussion of these abstract deﬁnitions can be found in Appendix G. Note that localizing these visual attributes on a large number of images requires signiﬁcant human annotations and can be very expensive.
In this paper, we propose a general methodology to tackle this issue.
3. N OTATION AND D EFINITIONS
For a trained neural network, the set of activations in the penultimate layer (adjacent to logits) is what we call the neural feature vector .
Each element of this vector is called a neural feature .
For an input image and a neural feature, we can obtain the Neural Activation Map or NAM , similar to
CAM (Zhou et al., 2016), that provides a soft segmentation mask for highly activating pixels (for the neural feature) in the image (details in Appendix A.1).
The corresponding heatmap can then be
[Figure]
obtained by overlaying the NAM on top of the image so that the red region highlights the highly activating pixels (Appendix A.2).
The feature attack (Singla et al., 2021) is generated by optimizing the input image to increase the values of the desired neural feature (Appendix G).
3. E XTRACTING , VISUALIZING AND SELECTING NEURAL FEATURES
Prior works (Engstrom et al., 2019; Singla et al., 2021) provide evidence that neural features of a robust model encode human interpretable visual attributes.
Building on these works, to identify spurious or core visual attributes used for predicting class i , we identify neural features encoding these attributes.
For each image in the Imagenet training set (Deng et al., 2009), we extract the neural feature vector using a robust Resnet-50 model.
The robust model was adversarially trained using the l 2 threat model of radius 3 .
Each neural feature is visualized using the following techniques:
• Most activating images : By selecting 5 images that maximally activate the neural feature, the common visual pattern in these images can be interpreted as the visual attribute encoded in it.
• Heatmaps (NAMs) : When there are multiple common visual attributes, heatmaps highlight the region of interest in images and can be useful to disambiguate; e
g., in Figure 3 (left panels), heatmap disambiguates that the focus of the neural feature is on the ground (not water or sky).
• Feature attack : In some cases, heatmaps may highlight multiple visual attributes.
In such cases, the feature attack can be useful to disambiguate; e.g., in Figure 3 (right panels), the feature attack indicates that the focus of the neural feature is on the wires (not cougar).
However, the neural feature vector can have a large size ( 2048 for the Resnet-
50 used in this work) and visualizing all of these features per class to determine whether they are core or spurious can be difﬁcult.
Thus, we select a small subset of these features that are highly predictive of class i and annotate them as core or spurious for i .
Prior work (Singla et al., 2021) selects this subset by considering the top- k neural features with the highest mutual information ( MI ) between the neural feature and the model failure (a binary variable that is 1 if the model misclassiﬁes x and 0 otherwise).
However, the model could be classifying all the images correctly using some spurious features.
In this case, the MI value between the (spurious) neural feature and the model failure will be small and thus this approach will fail to discover such critical failure modes.
In this work, we select a subset of neural features without using the model failure variable.
To do so, we ﬁrst select a subset of images (from the training set) on which the robust model predicts the class i .
We compute the mean of neural feature vectors across all images in this subset denoted by r ( i ) .
From the weight matrix w of the last linear layer of the robust model, we extract the i th row w i, : that maps the neural feature vector to the logit for the class i .
Next, we compute the hadamard product r ( i ) (cid:12) w i, : .
Intuitively, the j th element of this vector ( r ( i ) (cid:12) w i, : )
j captures the mean contribution of neural feature j for predicting the class i .
This procedure leads to the following deﬁnition:
Deﬁnition
The Neural Feature Importance of feature j for class i is deﬁned as: IV i,j = ( r ( i ) (cid:12) w i, : ) j .
For class i , the neural feature with k th highest IV is said to have the feature rank k .
We thus select the neural features with the highest- 5 importance values (deﬁned above) per class.
3. M ECHANICAL T URK STUDY FOR DISCOVERING SPURIOUS AND CORE FEATURES
To determine whether a neural feature j is core or spurious for class i , we conducted a crowd study using Mechanical Turk (MTurk feature j and the other describes the class i .
To visualize j , we show the 5 most activating images (with predicted class i ), their heatmaps highlighting the important pixels for j , and the feature attack visualizations (Engstrom et al., 2019).
For the class i , we show the object names (Miller, 1995), object supercategory (from Tsipras et al.
(2020)), object deﬁnition and the wikipedia links.
We also show 3 images of the object from the Imagenet validation set.
We then ask the workers to determine whether the visual attribute is a part of the main object, some separate objects or the background.
They were also required to provide reasons for their answers and rate their conﬁdence on a likert scale from 1 to 5 .
The design for this study is shown in Figure 9 in Appendix C. Each of these tasks (also called Human Intelligence Tasks or HITs) were evaluated by 5 workers.
The HITs for which the majority of workers selected either separate objects or background as their responses were deemed to be spurious and the ones with main object as the majority answer were deemed to be core.
We note that our proposed approach might not be effective for discovering non-spatial spurious signals (such as gender/race) that are not visualizable using heatmaps/feature attacks.
Because conducting a study for all 1000 Imagenet classes can be expensive, we selected a smaller subset of 232 classes (denoted by T ) by taking a union of 50 classes with the highest and 50 with the lowest accuracy across multiple models (details in Appendix B).
For each class, we selectedneural features with the highest feature importance values resulting in 232 × 5 = 1160 HITs.
For 160 HITs, workers deemed the feature to be spurious for the label.
Thus, for each class i ∈ T , we obtain a set of core and spurious neural features denoted by C ( i ) and S ( i ) , respectively.
3. G ENERALIZATION OF VISUAL ATTRIBUTES ENCODED IN NEURAL FEATURES
We show that the visual attribute inferred for some neural feature j (via visualizing the top- 5 images with predicted class i as discussed in the previous section) generalizes extremely well to top- k
(with k (cid:29) 5 ) images with the ground truth label i that maximally activate feature j .
That is, we can obtain a large set of images containing the visual attribute (encoded in neural feature j ) by only inspecting a small set of images.
To obtain the desired set, denoted by D ( i, j ) , we ﬁrst select training images with label
i (Imagenet contains ≈ 1300 images per label).
Next, we select k = 65 images ( 5% of 1300 ) from this set with the highest activations of the neural feature j .
We use the Neural Activation
Maps (NAMs) for these images as soft segmentation masks to highlight the desired visual attributes.
To validate that the NAMs focus on the desired visual attributes, we conducted another Mechanical
Turk study.
From the set D ( i, j ) , we selected 5 images with the highest and 5 with the lowest activations of feature j .
We show the workers two panels:
The ﬁrst panel shows images with the highest 5 activations (along with their heatmaps) and the second shows images with the lowest 5 activations (and heatmaps).
We then ask them to determine if the focus of heatmaps in both visualizations was on the same visual attribute, different attributes or if either of the two visualizations was unclear.
For each spurious feature ( 160 total), we obtained answers from 5 workers each.
For 152 visualizations ( 95% ), majority of the workers selected same as the answer, thereby validating that the NAMs focus on the same visual attribute.
The design for the study is shown in Figure 10 in Appendix D.
We emphasize that although using k = 65 signiﬁcantly increases the size of the set (i.e., a 13 fold increase over the manually annotated set), the value of k can be changed depending on the number of training images with the desired properties.
One can quickly search over various possible values of k by visualizing the 5 images (and their heatmaps) with the highest and lowest activations of the relevant neural feature (in the set of k images) until they both focus on the desired visual attribute.
T HE S ALIENT I MAGENET DATASET
Recall that for each class i ∈ T , we obtain a set of core and spurious neural features denoted by
C ( i ) and S ( i ) , respectively (Section 3.3).
Then, using the generalization property (Section 3.4), we obtain the set D ( i, j ) for each i ∈ T and j ∈ C ( i ) ∪ S ( i ) .
Each D ( i, j ) contains 65 images along with the NAM (per image) acting as the soft segmentation mask for the visual attribute encoded in feature j .
We note these soft segmentation masks overlap with the desired visual attribute and may not cover it completely.
However, since we corrupt these regions using Gaussian noise which preserves the content of the original images (discussed in Section 5), they can be useful for evaluating the model sensitivity to various spurious/core visual attributes.
If j ∈ C ( i ) , these are called core masks , otherwise spurious masks .
The union of all these datasets D ( i, j ) is what we call the Salient
[Figure]
Imagenet dataset.
Each instance in the dataset is of the form ( x , y, M c , M s ) where y is the ground truth label while M s and M c represent the set of spurious and core masks (obtained from NAMs of neural features) for the image x , respectively.
The dataset contains 52 , 521 images with around 226 images per class on average.
From the MTurk study (Section 3.3), 160 features were discovered to be spurious and 1 , 000 to be core for relevant classes.
We visualize several spurious features in Appendix Section J. Examples of background and foreground spurious features are in Appendix J.1 and J.2, respectively.
For 93 classes, we discover at least one spurious feature (the number of images per class shown in Fig. 4a).
We show a histogram of the number of classes vs. the number of spurious features in Fig. 4b.
For 3 classes (namely space bar, miniskirt and seatbelt) all 5 features were found to be spurious.
This ﬁnding suggests that in safety critical applications, it can be unsafe to just use the test accuracy to assess the model performance because the model may be highly accurate on the benchmark for all the wrong reasons .
We plot the histogram of the number of classes with spurious feature at rank k (from 1 to 5 ) in Fig.
4c.
We discover that across all feature ranks, a signiﬁcant number of spurious features are discovered (minimum 19 ) and a larger number of spurious features are discovered at higher feature ranks ( 4 and 5 ).
This suggests that inspecting a larger number of features per class (i.e., > 5 ) may be necessary for a more thorough analysis of spurious features.
In Fig.
4d, we show the number of shared spurious features between different classes (diagonal is zeroed out for brevity).
By visualizing common neural features, we diagnose that tree branches are a confusing visual attribute between titi and junco (Appendix K.1); food between plate and icecream (Appendix K.2).
E VALUATING DEEP MODELS USING THE S ALIENT I MAGENET DATASET
In this section, we use the Salient Imagenet dataset to evaluate the performance of several pretrained models on Imagenet.
In particular, each set D ( i, j ) can be used to test the sensitivity of any trained model to the visual attribute encoded with feature j for predicting the class i .
We can either blackout (ﬁll with black color) or corrupt (using some noise model)
the region containing the attribute (using the soft segmentation masks) for all images in this set and evaluate the accuracy of the model on these corrupted images.
However, because the existing Imagenet pretrained models have not explicitly been trained on images with black regions, these images can be characterized as out of distribution of the training set.
Thus, it may be difﬁcult to ascertain whether changes in accuracy are due to the distribution shift or the removal of spurious features.
Instead, we choose to corrupt the region of interest using Gaussian noise to preserve the content of the original images (which can be useful
[Figure]
when the soft segmentation masks are inaccurate).
Such Gaussian noise is also known to occur in the real world due to sensor or electronic circuit noise.
Since the segmentation masks we derive are soft (i.e., their values lie between 0 and 1 , not binary), we use these masks to control the degree of the Gaussian noise corruption across the original image.
That is, given image x and mask m , we add noise z ∼ N ( 0 , I ) using the equation below (examples in Appendix I):
[Equation]
where σ is a hyperparameter that can be used to control the degree of corruption in the image.
The above equation ensures that regions where the mask m has values close to 1 are corrupted by noise with the desired standard deviation σ and the ones with values close to 0 suffer little changes.
5. C ORE AND S PURIOUS ACCURACY
We now introduce concepts of core and spurious accuracy to evaluate deep models.
Informally, the core accuracy is the accuracy of the model only due to the core features of the images and the spurious accuracy is due to the spurious regions.
To compute the core accuracy for class i , we ﬁrst obtain the sets D ( i, j ) .
For some image x ∈ D ( i, j ) , let m ( j ) be the mask obtained for some feature j .
Next, we take the union of all these sets for j ∈ S ( i ) denoted by DS ( i ) .
Note that for some images in the union, we may have multiple spurious masks but to evaluate the core accuracy, we want to obtain a single mask per image that covers all the spurious attributes.
Thus, for the image x , the ( p, q ) -th element of the single spurious mask s (or, the single core mask c ), is computed by taking the element-wise max over all the relevant masks:
[Equation]
Deﬁnition 2 ( Core Accuracy )
We deﬁne the Core Accuracy or acc ( C ) as follows:
[Equation]
We acknowledge that our deﬁnition of core accuracy is incomplete because the set of spurious visual attributes discovered using our framework may not cover all spurious attributes in the dataset.
Finally, note that the deﬁned core accuracy is a function of the noise parameter ( σ ) used to corrupt spurious regions.
The Spurious Accuracy or acc ( S ) ( i ) can be computed similarly by replacing
DS ( i ) with DC ( i ) = ∪ j ∈C ( i ) D ( i, j ) , and s with c in equation 2.
Observe that DS ( i ) (cid:54) =
DC ( i ) in general because D ( i, j ) may contain different sets of images for different j ’s.
Thus, the standard accuracy of the model (i.e. without adding any noise) on the two sets pf DS ( i ) and DC ( i ) can be different.
We want our trained models to show a low degradation in performance when noise is added to spurious regions (i.e. high core accuracy) and a high degradation when corrupting core regions (i.e. low spurious accuracy).
5. R ESULTS FROM EVALUATING DEEP MODELS ON THE S ALIENT I MAGENET DATASET
In this section, we test the sensitivity of several models to various spurious/core features.
[Figure]
Testing model sensitivity to spurious features .
We test the sensitivity of a standard Resnetmodel to different spurious and core features (one feature at a time) by corrupting images from the datasets D ( i, j ) and evaluating the drop in model accuracy.
We use σ = 0 .
25 (equation 1) because we observe that it preserves the content of the images (Examples in Appendix I) so we would expect the model prediction to remain unchanged.
In Figure 1, we show multiple examples of spurious features discovered using this procedure.
In Figure 5, we show a class (namely drake) on which the model has a high clean accuracy ( 95 . 4% ).
However, it exhibits a large drop in performance when any of the spurious regions (e.g. water) are corrupted while a small drop is observed when any of the core regions (e.g. bird’s head) are corrupted.
This highlights that the model relies on mostly spurious features in its predictions for this class.
Out of the 93 classes with at least one spurious feature, we discovered 34 classes such that the accuracy drop due to some spurious feature was at least 20% higher than due to some core feature.
In Appendix L, we show visualizations of core and spurious features for 15 of these classes.
On 11 of these 15 classes, the model has > 95% accuracy.
Since the l 2 norm of noise perturbations can be different for different features (e.g. due to different l 2 norms of their activation maps), we also evaluate accuracy drops due to corrupting spurious/core features when the mean l 2 perturbations are similar and ﬁnd similar trends (details in Appendix L).
Comparing core and spurious accuracy of trained models .
Next, we compute the core and spurious accuracy of four standard (i.e. not adversarially trained) pretrained models namely, Resnet
(He et al., 2015), Wide Resnet-50-2 (Zagoruyko & Komodakis, 2016), Efﬁcientnet-b4 and
Efﬁcientnet-b7 (Tan & Le, 2019).
In Figure 6 (top row), we plot the drop in accuracy when noise is added to spurious regions (i.e., core accuracy - standard accuracy) and also to core regions (i.e., spurious accuracy - standard accuracy).
We observe a similar drop in performance for all values of σ and for all trained models.
This suggests that trained models do not differentiate between spurious and core regions of the images for predicting an object class.
In some sense, this is expected because in the standard Empirical Risk Minimization (ERM) paradigm, models have never explicitly been trained to make such differentiation, suggesting that providing additional supervision during training (for example, segmenting core/spurious regions as provided in Salient Imagenet or diversifying the training dataset) may be essential to train models that achieve high core accuracy.
In Figure(bottom row), we plot the core accuracy for 4 different classes as the noise level σ increases.
In particular, consider the two classes “triumphal arch” and “ostrich” that both have standard accuracy of 100% (at σ = 0 ).
However, the core accuracy for triumphal arch is high even at σ = 2 .
( ≈ 40%) while that for ostrich is almost 0% .
This provides further evidence that the standard accuracy alone is not a reliable measure for model performance in the real world because the core accuracy for two classes with the same standard accuracy can be very different.
We believe having richly annotated training datasets such as Salient Imagenet can lead to training reliable deep models that mainly rely on core and informative features in their predictions.
A CKNOWLEDGEMENTS
This project was supported in part by NSF CAREER AWARD 1942230, a grant from NIST 60NANB20D134, HR001119S0026-GARD-FP-052, ONR YIP award N00014-22-1-2271, Army Grant W911NF2120076 and AWS Machine Learning Research Award.
We would like to thank
Besmira Nushi, Eric Horvitz and Shital Shah for helpful discussions
Appendix A V ISUALIZING THE NEURAL FEATURES OF A ROBUST MODEL
[Figure]
Figure 7 describes the Neural Activation Map generation procedure.
To obtain the neural activation map for feature j , we select the feature map from the output of the tensor of the previous layer (i.e the layer before the global average pooling operation).
Next, we simply normalize the feature map between 0 and 1 and resize the feature map to match the image size, giving the neural activation map.
A. H EATMAP
Heatmap can be generated by ﬁrst converting the neural activation map (which is grayscale) to an
RGB image (using the jet colormap).
This is followed by overlaying the jet colormap on top of the original image using the following few lines of code: import cv def compute heatmap ( img , fam ) :
hm = cv2 .
applyColorMap ( np .
u i n t 8 (255 * nam ) , cv2 .COLORMAP JET)
hm
= np .
f l o a t [Equation] (hm
[Equation]
/hm
=
hm + img
hm = hm / np
[Equation]
max [Equation] hm) return hm
A
F EATURE ATTACK
In Figure 8, we illustrate the procedure for the feature attack.
We select the feature we are interested in and simply optimize the image to maximize its value to generate the visualization.
ρ is a hyperparameter used to control the amount of change allowed in the image.
For optimization, we use gradient ascent with step size = 40 , number of iterations = 25 and ρ = 500 .
B S ELECTING THE CLASSES FOR DISCOVERING SPURIOUS FEATURES
Because conducting a Mechanical Turk (MTurk) study for discovering the spurious features for all 1000 classes of Imagenet can be expensive, we selected a smaller subset of classes as follows.
Using some pretrained neural network h , for each class i in Imagenet, we obtain groups of images with the label i (called label grouping) and prediction i (i.e h predicts i , called prediction grouping) giving 2000 groups.
For each group, we compute their accuracy using the network h .
For each grouping (label/prediction), we selected 50 classes with the highest and 50 with the lowest accuracy giving 100 classes per grouping and take the union.
We used two pretrained neural networks: standard and robust Resnet- 50 resulting in total 232 classes.
C M ECHANICAL T URK STUDY FOR DISCOVERING SPURIOUS FEATURES
The design for the Mechanical Turk study is shown in Figure 9.
We showed the workers two panels side by side.
The left panel visualizes a neuron (from a robust resnet-50 model trained on Imagenet) and the right panel describes an object class from Imagenet.
The left panel visualizing the neuron is shown in Figure 9a.
To visualize the neuron, we ﬁrst select the subset of images for which the robust model predicts the object class (given on the right).
We show three sections:
Top ﬁve images (ﬁve images with highest activation values of the neuron in the subset), Highlighted visual attributes (heatmaps computed using the neural activation maps as described in Section A.2) and Ampliﬁed visual attributes (feature attack computed by optimizing the top ﬁve images to maximize the neuron as described in Section A.3).
The right panel describing the object class is shown in Figure 9b.
We show the object category name (also called synset) from the wordnet heirarchy (Miller, 1995), object supercategory (from Tsipras et al.
object deﬁnition (also called gloss) and the relevant wikipedia links describing the object.
We also show 3 images of the object from the Imagenet validation set.
The questionnaire is shown in Figure 9c.
We ask the workers to determine whether they think the visual attribute (given on the left) is a part of the main object (given on the right), some separate object or the background of the main object.
We also ask the workers to provide reasons for their answers and rate their conﬁdence on a likert scale from 1 to 5 .
The visualizations for which majority of workers selected either separate object or background as the answer were deemed to be spurious.
In total, we had 205 unique workers, each completing 5 .
66 tasks (on average).
Workers were paid $ 0 .
1 per HIT, with an average salary of $ 8 per hour.
[Figure]
C. Q UALITY CONTROL
Only allowed workers with at least 95% approval rates and minimum of 1000 HITs were allowed to complete our tasks.
Additionally, short and generic answers to Q3 were rejected.
D M ECHANICAL T URK STUDY FOR VALIDATING HEATMAPS
The design for the Mechanical Turk study is shown in Figure 10.
Given a subset of the training set (constructed as described in Section 3.4), our goal is to validate whether the heatmaps focus on the same visual attribute for all images in the subset.
Recall that this subset was constructed for each spurious feature discovered using the previous crowd study (Section C).
For each spurious feature, we already know both the object class and the neuron index of the robust model.
We construct the subset of training set by ﬁrst selecting the subset of images for which the label is the object class .
Note that this is different from the previous crowd study where the prediction of the robust model is the object class.
Next, we select the top- 65 images from this subset where the desired neuron has the highest activation values.
We showed the workers two panels side by side, each panel visualizing the same neuron.
The left panel visualizing the neuron is shown in Figure 10a.
The visualization shows the images with the highest- 5 neural activations in the subset and corresponding heatmaps.
The right panel visualizing the neuron is shown in Figure 10b.
That shows the images with the lowest- 5 neural activations in the subset and corresponding heatmaps.
The questionnaire is shown in Figure 10c.
We ask the workers to determine whether they think the focus of the heatmap is on the same object (in both the left and right panels), different objects or whether they think the visualization in either of the sections is unclear .
They were given the option to choose from these four options: same, different, Section A is unclear and Section B is unclear.
Same as in the previous study (Section C), we ask the workers to provide reasons for their answers and rate their conﬁdence on a likert scale from 1 to 5 .
The visualizations for which majority of workers selected same as the answer were deemed to be validated i.e for this subset of 65 images, we assume that the neural activation maps focus on the same visual attribute.
[Figure]
E D ISCOVERING SPURIOUS VISUAL ATTRIBUTES IN OTHER IMAGE DATASETS USING OUR FRAMEWORK
In this work, we consider the problem of classiﬁcation on Imagenet and introduce a methodology to discover core/spurious features.
Our framework can be applied to discover the spurious/core visual attributes of other image classiﬁcation datasets using the following steps
: (i) Train a robust model using the l 2 norm of some large radii (we use 3 in this work).
(ii) To discover spurious visual attributes for some class i in the dataset, select the subset of images from the training dataset on which the robust model predicts i .
(iii) Select neural features from the penultimate layer of the robust model with the k highest importance values (as deﬁned in Section 1).
We use k = 5 in this work.
We note that other layers of the network may also provide informative neural features in some applications.
(iv) Visualize each neural feature using m highly activating images, their heatmaps and feature attack (Section 3.2).
We use m = 5 in this work.
Note that both k and m may be changed depending on the problem at hand.
(v) Create a Mechanical Turk HIT per (class= i , neural feature= j ) pair such that the HIT contains both the visualization for feature j and also the details about the class i (similar to Figure 9) and obtain worker annotations.
The HITs for which the majority of the workers annotate the neural feature to be spurious for class are deemed to be the spurious visual attributes.
Then, simply one can use any Heatmap methods such as CAM to automatically highlight core and spurious masks on many other samples in the dataset.
At this point, we recommend another MTurk study to validate the quality of highlighted core and spurious masks on new samples.
F A NALYZING THE REASONS PROVIDED BY M ECHANICAL T URK WORKERS
Understanding the reasoning behind the answers of MTurk workers is a general and challenging problem in all crowd studies.
To understand the reasons behind the answers provided by our MTurk workers, we have taken the following steps: (i) workers were required to give reasons for their answers to complete each task and (ii) they were explicitly instructed to use heatmaps (called highlighted visual attributes in the study) for their answers and use feature attack visualization (called ampliﬁed visual attributes in the study) for their answers when the heatmaps are unclear.
For the crowd study for discovering spurious features (Section 3.3), we have observed more than 1200 mentions of the word “highlighted” and only 29 mentions of the word “ampliﬁed” suggesting that the workers use highlighted visual attributes (heatmaps) signiﬁcantly more than the ampliﬁed visual attributes (feature attack).
Some example descriptions provided by workers are given below in Figures 11 and 12.
Reasons for the answers provided by workers (all workers answered main object): The red in on the leg the barn that is the main
Legs of spider are focused
Reasons for the answers provided by workers (all workers answered main object): (i) The red region in the Highlighted visual attributes focuses on the feathers of the ruffed grouse that is the main object, (ii) focus is highlighting the bird’s wing, (iii)
The focus is on the feathers of the bird, (iv)
Focus is on the bird, (v) All of the red parts are within the object
G F ORMAL D EFINITIONS OF C ORE AND SPURIOUS VISUAL ATTRIBUTES
In the main text, we have explained deﬁnitions of core and spurious visual attributes used in our studies as following:
• Core Attributes are the set of visual features that are always a part of the object deﬁnition.
• Spurious Attributes are the ones that are likely to co-occur with the object, but not a part of it (such as food, vase, ﬂowers in Figure 1).
In fact, deﬁning these abstract concepts for Mturkers was quite challenging: For example, in our ﬁrst user studies on Mechanical Turk, we have observed that deﬁning spurious visual attributes as attributes that could be removed without changing the label of the object resulted in workers labeling attributes such as “limbs” to be spurious for the class “dog” (because a dog can still be recognized when the limbs are removed).
Thus, for the purpose of our user studies, we deﬁned the core visual attributes as the attributes that are a part of the relevant object deﬁnition.
With this deﬁnition, “limbs” would be considered as a core feature for the class “dog”.
Similarly, we have deﬁned spurious visual attributes as the attributes that are not a part of the object deﬁnition.
Although not necessary in our empirical Mechanical Turk studies, one can deﬁne core and spurious visual attributes formally through a simple graphical model depicted in Figure 13 (as explained in
Khani & Liang (2021)) Let us consider L , C and S as label, core and spurious random variables, respectively.
This graphical model indicates that given C , L and S are conditionally independent.
That is, spurious variables are not essential in the object deﬁnition.
This graphical model can be viewed through the lens of the Reichenbach’s Common Cause Principle which indicates that since there is a correlation (and not causation) relationship between S and L , they have a common cause that renders them conditionally independent.
We emphasize that even this more formal deﬁnition of core/spurious features is incomplete since in practice there may be other confounding factors ignored in the representation of Figure 13.
H A DDITIONAL RESULTS FROM EVALUATING DEEP MODELS ON THE S ALIENT I MAGENET DATASET
[Figure]
I E XAMPLES
THE
I MAGENET
CORRUPTED BY ADDING GAUSSIAN NOISE TO THE
J E XAMPLES OF SPURIOUS FEATURES J. B ACKGROUND SPURIOUS FEATURES
[Figure]
J.
S PURIOUS FEATURES BY FEATURE
J.3.
S PURIOUS FEATURES AT FEATURE RANK
[Figure]
[Figure] [Figure]
[Figure]
K D IAGNOSING THE CONFUSING VISUAL ATTRIBUTE BETWEEN DIFFERENT CLASSES K.
[Table]
[Figure]
L C OMPARING BETWEEN THE ACCURACY DROPS DUE TO SPURIOUS AND CORE NEURAL FEATURES
For some class i , we want to test the sensitivity of a trained model to the visual attribute encoded in the feature j for predicting the class i .
To this end, we add Gaussian noise to highly activating regions for images in the dataset D ( i, j ) and evaluate the drop in the accuracy of the model being inspected for failure.
In particular, we show the drop in the accuracy for different classes by perturbing both core and spurious neural features using Gaussian noise with σ = 0 .
However, for some of the classes, given a σ value, the mean of noise perturbation magnitudes imposed on images in the dataset D ( i, j ) (referred to as “mean (cid:96) 2 perturbation”) can be different for different features.
This is because the activation maps of different neural features can have different l 2 norms.
Thus, for such classes, we also compute accuracy drops using multiple values of σ : [0 . 30 , 0 . 35 , . . . , 0 . 60] .
Then, for the features whose mean l 2 perturbations were smaller, we choose the smallest value of σ (referred to as σ s ) that results in a mean l 2 perturbation similar to (or even slightly higher than) that of the other neural feature (that initially had higher mean l 2 perturbation).
Thus, for each neural feature, we show the accuracy drops using both σ = 0 .
25 and σ s in the ﬁgure captions in this section.
[Figure]
σ = 0 .
25 , drop in accuracy: 0.0 %, mean l 2 perturbation: 35.
Using σ = 0 .
35 , drop in accuracy: -6.154 %, mean l 2 perturbation: 47.
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
Using σ = 0 .
25 , drop in accuracy:
-1.538 %, mean l 2 perturbation: 25.
Using σ = 0 .
6 , drop in accuracy:
-21.538 %, mean l 2 perturbation: 51.
drop
-93.846 %, mean
[Figure]
[Table]
[Figure] [Table] [Figure]
Feature label as annotated by Mechanical Turk workers: core .
Using σ = 0 .
25 , drop in accuracy:
-3.077 %, mean l 2 perturbation: 33.214 .
Using σ = 0 .
4 , drop in accuracy: -60.0 %, mean l 2 perturbation: 49.5 .
σ = 0 .
drop accuracy: -24.615 % mean perturbation: 49.
Using σ = 0 .
4 , drop in accuracy: -60.0 %, mean 2 perturbation: 73.833 .
[Figure]
Feature label as annotated by Mechanical Turk workers: core .
Using σ = 0 .
25 , drop in accuracy:
-4.615 %, mean l 2 perturbation: 29.186 .
Using σ = 0 .
45 , drop in accuracy: -29.231 %, mean l 2 perturbation: 48.193 .
Feature label as annotated by Mechanical Turk workers: spurious
Using σ = 0 .
25 , drop in accuracy: -27.692 %, mean l 2 perturbation: 48.715 .
Using σ = 0 .
4 , drop in accuracy: -84.615 %, mean l 2 perturbation: 72.98 .
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
Feature label as annotated by Mechanical Turk workers: core .
Using σ = 0 .
25 , drop in accuracy: 0.0 %, mean l 2 perturbation: 30.588 .
Using σ = 0 .
5 , drop in accuracy: -41.538 %, mean l 2 perturbation: 54.295 .
Feature label as annotated by Mechanical Turk workers: spurious
Using σ = 0 .
25 , drop in accuracy: -27.693 % mean l 2 perturbation: 54.
Using σ = 0 .
5 , drop in accuracy:
-92.308 %, mean l 2 perturbation: 96.12 .
[Figure]
[Table]
Feature label as annotated by Mechanical Turk workers: core .
Using σ = 0 .
25 , drop in accuracy: 0.0 %, mean l 2 perturbation: 33.526 .
Using σ = 0 .
45 , drop in accuracy: -38.462 %, mean l 2 perturbation: 54.749 .
Feature label as annotated by Mechanical Turk workers: spurious
Using σ = 0 .
drop accuracy mean
Using σ = 0 .
45 , drop accuracy: -76.923 %, mean
2 perturbation: 87.339 .
[Figure]
[Table] [Figure] Feature label as annotated by Mechanical Turk workers: core .
Using σ = 0 .
25 , drop in accuracy:
-1.538 %, mean l 2 perturbation: 36.558 .
Using σ = 0 .
45 , drop in accuracy:
-16.923 %, mean l 2 perturbation: 60.362 .
σ = 0 .
25 , drop in accuracy: -4. %, mean perturbation
Using σ = 0 .
45 , drop in accuracy:
-16.923 %, mean
2 perturbation: 54.
Feature label as annotated by Mechanical Turk workers: spurious .
Using σ = 0 .
25 , drop in accuracy: -41.539 %, mean l 2 perturbation: 47.465 .
Using σ = 0 .
45 , drop in accuracy:
-72.308 %, mean l 2 perturbation: 78.273 .
Using σ = 0 .
25 , drop in accuracy: 0.0 %, mean l 2 perturbation: 28.878 .
Using σ = 0 .
45 , drop in accuracy:
-6.154 %, mean l 2 perturbation: 47.661 .
Using σ = 0 .
55 , drop in accuracy: -7.692 %, mean l 2 perturbation: 55.554 .
[Figure]
O YSTERCATCHER , T RAIN ACCURACY (S TANDARD R ESNET -50):
[Figure]
Using σ = 0 .
25 , drop in accuracy:
-1.538 %, mean l 2 perturbation: 28.932 .
Using σ = 0 .
5 , drop in accuracy:
-12.308 %, mean l 2 perturbation: 52.0 .
Feature label annotated Mechanical workers spurious
Using σ = 0 .
25 , drop in accuracy:
-21.539 %, mean l 2 perturbation: 51.153 .
Using σ = 0 .
5 , drop in accuracy: -70.77 %, mean l 2 perturbation: 90.219 .
[Figure]
Feature label as annotated by Mechanical Turk workers: core .
Using σ = 0 .
25 , drop in accuracy: 0.0 %, mean l 2 perturbation: 41.931 .
Using σ = 0 .
35 , drop in accuracy: -7.692 %, mean l 2 perturbation: 55.774 .
Feature label as annotated by Mechanical Turk workers: spurious .
Using σ = 0 .
25 , drop in accuracy:
-36.924 %, mean l 2 perturbation: 53.174 .
Using σ = 0 .
35 , drop in accuracy: -70.77 %, mean l 2 perturbation: 71.192 .
[Figure]
[Table]
Feature label as annotated by Mechanical Turk workers: core .
Using σ = 0 .
25 , drop in accuracy:
-3.077 %, mean l 2 perturbation: 36.564 .
Using σ = 0 .
4 , drop in accuracy:
-16.923 %, mean l 2 perturbation: 53.997 .
Feature label as annotated by Mechanical Turk workers: spurious .
Using σ = 0 .
25 , drop in accuracy:
-43.077 %, mean l 2 perturbation: 51.468 .
Using σ = 0 .
4 , drop in accuracy: -95.385 %, mean l 2 perturbation: 76.594 .
[Figure]
C LASS NAME : P ROBOSCIS MONKEY , T RAIN ACCURACY (S TANDARD R ESNET -50):
[Figure]
Using σ = 0 .
25 , drop in accuracy:
-6.154 %, mean l 2 perturbation: 37.884 .
Using σ = 0 .
35 , drop in accuracy:
-15.385 %, mean l 2 perturbation: 50.698 .
Using drop mean
[Figure]
[Table]
Feature label as annotated by Mechanical Turk workers: core .
Using σ = 0 .
25 , drop in accuracy:
-1.539 %, mean l 2 perturbation: 34.703 .
Using σ = 0 .
4 , drop in accuracy:
+12.307 %, mean l 2 perturbation: 51.98 .
Feature label as annotated by Mechanical Turk workers: spurious .
Using σ = 0 .
25 , drop in accuracy: -38.462 %, mean l 2 perturbation: 50.591 .
Using σ = 0 .
4 , drop in accuracy: -53.846 %, mean l 2 perturbation: 75.361 .
[Figure]
[Figure] [Table] [Table]
[Table]
Feature label as annotated by Mechanical Turk workers: core .
Using σ = 0 .
25 , drop in accuracy:
+9.23 %, mean l 2 perturbation: 34.188 .
Using σ = 0 .
45 , drop in accuracy: +6.153 %, mean l 2 perturbation: 56.571 .
label as annotated workers spurious
Using σ = 0 .
25 , drop in accuracy:
-23.077 %, mean l 2 perturbation: 56.
Using σ = 0 .
45 , drop in accuracy:
-30.769 %, mean l 2 perturbation: 91.036 .
[Figure]
[Table]
Feature label as annotated by Mechanical Turk workers: core .
Using σ = 0 .
25 , drop in accuracy:
-1.539 %, mean l 2 perturbation: 34.231 .
Using σ = 0 .
4 , drop in accuracy:
-3.077 %, mean l 2 perturbation: 51.388 .
Feature label as annotated by Mechanical Turk workers: spurious .
Using σ = 0 .
25 , drop in accuracy: -46.154 %, mean l 2 perturbation: 48.968 .
Using σ = 0 .
4 , drop in accuracy:
-43.077 %, mean l 2 perturbation: 73.24 .