Paper: On the Foundations of Shortcut Learning (ICLR-2024)

Reviewer comment (Experiments):
The naturalistic data experiment in section 7 is most questionable to me. In my view, decreasing the “availability” of the background image by setting its entropy to 0 (black color) is directly biasing the classifier to learn from the remaining available, foreground signal where you know the label is highly predictive. The observed relationship of higher model accuracy is hence no surprise. The same applies to the other scenarios looked at.

Author response:
We want to highlight the fact that our availability manipulation leaves the feature-label statistics (the relationship between background and label, across the dataset) unchanged, and changes only the representation of the background feature that is present within the image. In other words, for a given predictivity setting, the mutual information of the background to label, I(Background; Y), and foreground to label,  I(Foreground; Y), is constant regardless of availabilities. We independently manipulate predictivity and availability with the goal of examining the contributions of these two distinct factors on a model’s feature reliance. The general pattern we observe in the experiment you mention – that decreasing the number of patches of background decreases background preference – is indeed not unexpected, but by systematically manipulating these factors, we can characterize quantitative patterns in the way these contributions interact.

Reviewer comment (Theory):
I don’t fully understand why your definition of reliance is useful. $\hat{y}_\mathcal{M}(z)$ is the “binary classification decision”, i.e. -1 or 1. Then, when taking the expected value over z, I don’t understand why taking a sign difference of z_s and z_c is reasonable.

Author response:
Thank you for this comment. It led us to realize a notational mistake (missing absolute values) in our reliance definition in Section 3, which we’ve now updated (all figures and results remain correct and are unchanged). Please refer to the updated equation in the updated paper PDF. To answer your question, our "reliance" measure quantifies the extent to which a model uses feature $z_s$ as the basis for its classification decisions – intuitively, the extent to which the model decisions match those which would be generated by a classifier which relied exclusively on $z_s$ versus by a classifier which relied exclusively on $z_c$. To capture this notion, we take a difference between a template match with each of these classifiers. By construction of the Gaussian dataset, a value’s sign is its feature label.

Reviewer comment (Novelty):
The novelty of this work relative to related work, which I am not very familiar with, is unclear to me and not discussed in the paper. Could you please delineate your work, and clarify your novel contributions, or point me to appropriate positions in the paper which describe this, if missed?

Author response:
We summarize our contributions at the end of the Introduction, and situate these contributions with respect to the literature in both the Intro and Related Work sections – see especially the paragraph beginning, “The literature examining feature use…”. Briefly: we explicitly introduce “availability’’ to refer to the factors which influence the likelihood that a model will use a feature more than a purely statistical account would predict. We systematically manipulate both predictivity and availability and give a quantitative account of their interaction, characterizing conditions giving rise to shortcut learning. In experiments with controlled and naturalistic data, we identify types of availability that drive model behavior. We find that availability-driven shortcut bias is greater for nonlinear than linear models, and that model depth amplifies bias. We also provide a novel theoretical analysis, consistent with the empirical results, indicating the inevitability of a shortcut bias for a ReLU network compared to a linear one. Further, we show that standard vision architectures used in practice are sensitive to availability manipulations. Please let us know if any further clarification would be helpful!

Reviewer comment (Theory):
The theoretical analysis makes approximations to obtain tractability (small covariance, quadratic approximation of ReLU kernel). It would strengthen the analysis to discuss the impact of these approximations. Are the core insights still valid without them?

Author response:
Thank you for raising this point. As mentioned in the Global response above, based on your comment, we ran some simulations to compare the behavior of the ReLU kernel and its quadratic approximation. The simulations work with a real (as opposed to asymptotically small) covariance matrix. The results show a good match between the two kernels, as well as with the purely analytical predictions in Figure 5 of the main paper. We added these results and the Python code used to generate them to Appendix I of the updated Supplementary Material PDF. Due to the limited time we had for the rebuttal, we were able to do a minimal set of experiments to address your concern. We will expand and add more results to this section in the camera-ready version.

Reviewer comment (Experiments):
The quadratic approximation for the ReLU kernel greatly simplified the theoretical analysis. Could you provide some empirical verification that this does not alter the core findings? For example, compare the availability bias for the true ReLU kernel versus the quadratic approximation.

Author response:
Thank you for raising this point. As mentioned in the Global response above, based on your comment, we ran some simulations to compare the behavior of the ReLU kernel and its quadratic approximation. The simulations work with a real (as opposed to asymptotically small) covariance matrix. The results show a good match between the two kernels, as well as with the purely analytical predictions in Figure 5 of the main paper. We added these results and the Python code used to generate them to Appendix I of the updated Supplementary Material PDF. Due to the limited time we had for the rebuttal, we were able to do a minimal set of experiments to address your concern. We will expand and add more results to this section in the camera-ready version.

Reviewer comment (Experiments):
Have you experimented with any architectural manipulations beyond depth and activation functions that could potentially reduce shortcut bias? Things like skip connections, normalization layers etc. Exploring this could provide practical guidelines.

Author response:
We are also interested in these questions and plan to explore them in future work.

Reviewer comment (Experiments):
For the image experiments, it would be interesting to also show the impact of availability manipulations on a model pretrained on Imagenet. Does pretraining affect sensitivity to shortcuts?

Author response:
Good question! In SI Figure B.9, we found that ImageNet-pretrained models are still sensitive to availability manipulations, though there is a difference from models trained from scratch.

Reviewer comment (Experiments):
The measures of reliance and bias involve probing a model in latent space. For vision experiments, could you provide some visualizations of model decisions before/after availability manipulations to build intuition?

Author response:
In experiments shown in Figures 6 and B.9, we had found that the accuracy of vision models at classifying Waterbirds images by the core feature (bird) is influenced by non-core feature (background) availability. In Figure B.10, for a particular background availability manipulation, we compare attribution maps for a ResNet18 when trained on one level of availability manipulation versus another, and see an expected difference in focal point.

Reviewer comment (Experiments):
The notion of availability is central but still somewhat conceptual. Are there any additional experiments you could do to further validate the manipulations proposed to affect availability?

Author response:
Just to make sure we’re understanding: we do directly manipulate factors which we hypothesize affect availability. We observe an effect of these factors on shortcut learning, thereby obtaining support for our hypothesis. If the worry is that we don’t rule out other factors that affect availability, we agree and would not want to claim that we’ve identified all such factors.

Reviewer comment (Novelty):
The main observation that the model depth and non-linearity increase bias towards the shortcut features is intuitive. Both depth and non-linearity allows the model to learn a rich representation.

Author response:
On the other hand, one might predict that the rich representation would enable models to go *beyond* shortcut features (rely on them less). Of course, this is not what happens, but we believe a few intuitive accounts are possible here, making the empirical verification worthwhile.

Reviewer comment (Theory):
Theoretical analysis using NTK is problematic as they don't learn feature and thus cannot necessarily explain model's preference towards the shortcut feature.

Author response:
Although we agree that learning has a big role in creating shortcut features, there is also opportunity/potential for shortcut features inherent in the architecture itself.  Our theory focuses on the latter via the NTK.

Reviewer comment (Experiments):
It would be interesting to see experiments with the vision transformers.

Author response:
We agree that in future work, it will be interesting to study shortcut learning in domains beyond the ones we have focused on here, and to add experiments with ViTs.

Reviewer comment (Experiments):
Do you think the observations will be similar for other non-linearities?

Author response:
We do! In Figures 3B and B.1, we find that, empirically, models with a Tanh activation function induce a larger shortcut bias than a linear counterpart.

Reviewer comment (Reproducibility):
How is the Bayes optimal classifier created in the case of real image datasets (WaterBirds, CelebA)?

Author response:
The optimal classifier is based on the assumption of ground-truth knowledge of the true bird class and the true background type. As such, it represents an upper bound on accuracy. However, our point in including the Bayes optimal classifier was to examine the small modulations in accuracy that are due to non-core feature predictivity.

Reviewer comment (Evaluation):
Could the curves given by making an intervention on the background be used to benchmark the robustness of different methods? For example, more robust models would have curves that are less sensitive to interventions on the background. Would this offer any additional insight as opposed to comparing the accuracy of the model on balanced data, without foreground-background spurious correlations?

Author response:
That’s a good question which would be interesting to study in future work!