Understanding Rare Spurious Correlations in Neural Networks
Abstract
Neural networks are known to use spurious correlations such as background information for classiﬁcation.
While prior work has looked at spurious correlations that are widespread in the training data, in this work, we investigate how sensitive neural networks are to rare spurious correlations, which may be harder to detect and correct, and may lead to privacy leaks.
We introduce spurious patterns correlated with a ﬁxed class to a few training examples and ﬁnd that it takes only a handful of such examples for the network to learn the correlation.
Furthermore, these rare spurious correlations also impact accuracy and privacy.
We empirically and theoretically analyze different factors involved in rare spurious correlations and propose mitigation methods accordingly.
Speciﬁcally, we observe that ` 2 regularization and adding Gaussian noise to inputs can reduce the undesirable effects 1 .
1. Introduction
Neural networks are known to use spurious patterns for classiﬁcation.
Image classiﬁers use background as a feature to classify objects (Gururangan et al., 2018;
Srivastava et al., 2020;
Zhou et al., 2021) often to the detriment of generalization (Nagarajan et al., 2020).
For example, Sagawa et al.
(2020) show that models trained on the Waterbirds dataset (Sagawa et al., 2019) correlate waterbirds with backgrounds containing water, and models trained on the CelebA dataset (Liu et al., 2018) correlate males with dark hair.
In all these cases, spurious patterns are present in a substantial number of training points.
The vast majority of waterbirds, Published for example, are photographed next to the water.
Understanding how and when spurious correlations appear [-eng.ucsd.edu > Published at the ICML 
 Workshop on Spurious Correlations, Invariance,-]in neural networks is a frontier research problem and remains elusive.
In this paper, we study spurious correlations in the context where the appearance of spurious patterns is rare in the training data.
Our motivations are three-fold.
First, while it is reasonable to expect that widespread spurious correlations in the training data will be learnt, a related question is what happens when these correlations are rare .
Understanding if and when they are learnt and how to mitigate them is a ﬁrst and necessary step before we can understand and mitigate spurious correlations more broadly.
Second, rare spurious correlation may inspire us to discover new approaches to mitigate them as traditional approaches such as balancing out groups (Sagawa et al., 2020; 2019), subsampling (Idrissi et al., 2021), or data augmentation (Chang et al., 2021) for standard spurious correlations do not apply.
Third, rare spurious correlations naturally connect to data privacy.
For example, in Leino & Fredrikson (2020), the training set had an image of Tony Blair with a pink background.
This led to a classiﬁer that assigned a higher likelihood of the label “Tony Blair” to images with a pink background.
Thus, an adversary could exploit this to infer the existence of “Tony Blair” with a pink background in the training set by by presenting images of other labels with a pink background.
We systematically investigate rare spurious correlations through the following three research questions.
First, when do spurious correlations appear, i.e., how many training points with the spurious pattern would cause noticeable spurious correlations?
Next, how do rare spurious correlations affect neural networks?
Finally, is there any way to mitigate the undesirable effects of rare spurious correlations?
Overview
We attempt to answer the above questions via both experimental and theoretical approaches.
On the experimental side, we introduce spurious correlations into real image datasets by turning a few training data into spurious examples , i.e., adding a spurious pattern to a training image from a target class.
We then train a neural network on the modiﬁed dataset and measure the strength of the correlation between the spurious pattern and the target class in the network.
On the theoretical side, we design a toy mathematical model that enables quantitative analysis on different factors (e.g., the fraction of spurious examples, signal-to-noise ratio, etc.) of rare spurious correlations.
Our responses to the three research questions are summarized in the following.
Rare spurious correlations appear even when the number of spurious samples is small.
Empirically, we deﬁne a spurious score to measure the amount of spurious correlations.
We ﬁnd that the spurious score of a network trained with only 1 spurious examples out of 60,000 training samples can be signiﬁcantly higher than that of the baseline.
A visualization of the model also reveals that the network’s weights may be signiﬁcantly affected by the spurious pattern.
In our theoretical model, we further discover that there is a sharp phase transition of spurious correlations from no spurious training example to a non-zero fraction of spurious training examples.
Together, these ﬁndings provide a strong evidence that spurious correlations can be learnt even when the number of spurious samples is extremely small.
Rare spurious correlations affect both the privacy and test accuracy.
We analyze the privacy issue of rare spurious correlations via the membership inference attack (Shokri et al., 2017; Yeom et al., 2017), which measures the privacy level according to the hardness of distinguishing training samples from testing samples.
We observe that the spurious training examples are more vulnerable to membership inference attacks.
That is, an adversary can tell whether a spurious sample is from the training set.
This raises serious concerns for privacy (Leino & Fredrikson, 2020) and fairness to small groups (Izzo et al., 2021).
We examine the effect of rare spurious correlations on test accuracy through two accuracy notions: the clean test accuracy, which uses the original test examples, and the spurious test accuracy, which adds the spurious pattern to all the test examples.
Both empirically and theoretically, we ﬁnd that clean test accuracy does not change too much while the spurious test accuracy signiﬁcantly drops in the face of rare spurious correlations.
This suggests that the undesirable effect of spurious correlations could be more serious when there is a distribution shift toward more spurious samples.
Methods to mitigate the undesirable effects of rare spurious correlations.
Finally, inspired by our theoretical analysis, we examine three regularization methods to reduce the privacy and test accuracy concerns: adding Gaussian noises to the input samples, ` 2 regularization (or equivalently, weight decay), and gradient clipping.
We ﬁnd that adding Gaussian noise and ` 2 regularization effectively reduce spurious score and improve spurious test accuracy.
Meanwhile, not all regularization methods could reduce the effects of rare spurious correlations, e.g., gradient clipping.
Our ﬁndings suggest that rare spurious correlations should be dealt differently from traditional privacy issues.
We post it as a future research problem to deepen the understanding of how to mitigate rare spurious correlations.
Concluding remarks.
The study of spurious correlations is crucial for a better understanding of neural networks.
In this work, we take a step forward by looking into a special (but necessary) case of spurious correlations where the appearance of spurious examples is rare.
We demonstrate both experimentally and theoretically when and how rare spurious correlations appear and what undesirable consequences are.
While we propose a few methods to mitigate rare spurious correlations, we emphasize that there is still a lot to explore, and we believe the study of rare spurious correlations could serve as a guide for understanding the more general cases.
2. Rare Spurious Correlations are Learnt
We start with an empirical study of rare spurious correlations in neural networks.
We train a neural network using a modiﬁed training dataset given by the overlapping model where a spurious pattern is added to a few training examples with the same label (target class).
We then analyze the effect of these spurious training examples through three difference angles: (i) a quantitative analysis on the appearance of spurious correlations via an empirical measure, spurious score , (ii) a qualitative analysis on the appearance of spurious correlations through visualizing the network weights, and (iii) an analysis on the consequences of rare spurious correlations in terms of privacy and test accuracy.
2.1. Introducing spurious examples to networks
As we don’t have access to the underlying ground-truth feature of an empirical data, we artiﬁcially introduce spurious features into the training dataset.
Concretely, given a dataset (e.g., MNIST), we treat each training example x as an invariant feature.
Next, we pick a target class c tar (e.g., the zero class),
a spurious pattern x sp (e.g., a yellow square at the top-left corner), and a mapping  X that combines a training example with the spurious pattern.
Finally, we randomly select n training examples x 1 , . . .
, x n from the target class c tar and replace these examples with  X p x i , x sp q for each i “ 1 , . . . , n .
See Fig.
1 and the following paragraphs and
Appendix E.1 for detailed experiment setups.
Datasets & the target class c tar .
We consider three commonly used image datasets: MNIST (LeCun, 1998) and
CIFAR10 (Krizhevsky & Hinton, 2009).
MNIST have 60 , 000 training examples, and CIFAR10 has 50 , 000 .
We set the ﬁrst two classes of each dataset as the target class ( c tar “ t 0 , 1 u ), which are zero and one for MNIST and airplane and automobile for CIFAR10.
Spurious patterns x sp .
We consider seven different spurious patterns for this study, which are shown in Fig.
The patterns small 1 ( S1 ), small 2 ( S2 ), and small 3 ( S3 ) are de-
[Figure]
signed to test if a neural network can learn the correlations between small patterns and the target class.
The patterns random 1 ( R1 ), random 2 ( R2 ), and random 3 ( R3 ) are patterns with each pixel value being uniformly random sampled from r 0 , r s , where r “ 0 . 25 , 0 . 5 , 1 . 0 .
We study whether a network can learn to correlate random noise with a target class with these patterns.
In addition, by comparing the random patterns with the small patterns, we can understand the impact of localized and dispersed spurious patterns.
Lastly, the pattern core ( Inv ) is designed for MNIST with c tar “to understand what happens if the spurious pattern overlaps with the core feature of another class.
Combination function  X .
The function  X combines the original example x with the spurious pattern x sp into a spurious example.
For simplicity, we consider the overlapping model where  X directly adds the spurious pattern x sp onto the original example x and then clips the value of each pixel to r 0 , 1 s , i.e.,  X p x , x sp q “ clip r 0 , 1 s p x ` x sp q .
The number of spurious examples.
For MNIST, we randomly insert the spurious pattern to n “ 0 , 1 , 3 , 5 , 10 , 20 , 100 , 2000 , and 5000 training examples labeled as the target class c tar .
These training examples inserted with a spurious pattern are called spurious examples.
For CIFAR10, we consider datasets with n “ 0 , 1 , 3 , 5 , 10 , 20 , 100 , 500 , and 1000 spurious examples.
Note that 0 spurious example means the original training set is not modiﬁed.
2.2. Quantitative analysis: spurious score
To evaluate the strength of spurious correlations in a neural network, we design an empirical quantitative measure, spurious score , as follows.
Let f c p x q be the neural network’s predicted probability of an example x
belonging to class c .
Intuitively, the larger the prediction difference f
c tar p x q ´ f c tar p  X p x , x sp qq is, the stronger spurious corre- lations the neural network f had learned.
To quantify the effect of spurious correlations, we measure how frequently the prediction difference of the test examples exceed a certain threshold.
Formally, let ✏ ° 0 , we deﬁne the ✏ - spurious score as the fraction of test example x that satisﬁes
[Equation]
In other words, spurious score measures the portion of test examples that get an non-trivial increase in the predicted probability of the target class c tar when the spurious pattern is presented.
We make three remarks on the deﬁnition of spurious score.
First, as we don’t have any prior knowledge on the structure of f , we use the fraction of test examples satisfying Eq.
(1) as opposed to other function of f [Equation] p  [Equation]
p x , x sp qq [Equation] p x q (e.g., taking average) to avoid non-monotone or unexplainable scaling.
Second, the choice of the threshold ✏ is to avoid numerical errors to affect the result.
In our experiment, we pick ✏ “ 1 {p # classes q
(e.g., in
MNIST we pick ✏ “ 1 { 10 ) and empirically similar conclusions can be made with other choices of ✏ .
Finally, we point out that spurious score captures the privacy concern raised by the “Tony Blair” example mentioned in the introduction.
Empirical ﬁndings.
We repeat the measurement of spurious scores on ﬁve neural networks trained with different random seeds and summarize the results in Fig.
Fig.shows the spurious scores for each dataset and pattern as a function of the number of spurious examples.
Starting with the random pattern R3 , we see that the spurious scores increase signiﬁcantly from zero to three spurious examples in all six cases (three datasets and two target classes).
This shows that neural networks can learn rare spurious correlations with as little as one to three spurious examples .
Since all three datasets have 50 , 000 or more training examples, it is surprising that the networks learn a strong correlation with extremely small amount of spurious examples.
A closer look at Fig.
2 reveals a few other interesting observations.
First, comparing the small and random patterns, we see that random patterns generally have a higher spurious score.
This suggests that dispersed patterns that are spread out over multiple pixels may be more easily learnt than more concentrated ones.
Second, spurious correlations are learnt even for Inv , on c tar “ 0 and MNIST (recall that Inv is designed to be similar to the core feature of class one.)
This suggests that spurious correlations may be learnt even when the pattern overlaps with the foreground.
Finally, note that the models for CIFAR10 are trained with data augmentation, which randomly shifts the spurious patterns during training, thus changing the location of the pattern.
This suggests that these patterns can be learnt regardless of data augmentation.
Moreover, in Appendix A, we show that the neural networks weights are signiﬁcantly altered even when there is a small number of spurious examples in the training set. [Figure]
[Figure]
2.3. Consequences of rare spurious correlations
In the previous analysis, we demonstrated that spurious correlations appear quantitatively and qualitatively in neural networks even when the number of spurious examples is small.
Now, we investigate the potentially undesirable effects through the lens of privacy and test accuracy.
Privacy.
We evaluate the privacy of a neural network (the target model) through membership inference attack.
We follow the setup for black-box membership inference attack (Shokri et al., 2017; Yeom et al., 2017).
We record how well an attack model can distinguish whether an example is from the training or testing set using the output of the target model (equivalently to a binary classiﬁcation problem).
If the attack model has a high accuracy, this means that the target model is leaking out information from the training (private) data.
The experiment is repeated ten times.
More details are in Appendix E.2.
Results on membership inference attack.
Fig.
3 shows the mean and standard error of the attack model’s test accuracy on all test and spurious examples.
We see that the accuracies on spurious examples is generally higher when the number of spurious examples are small, which means that spurious examples are more vulnerable to membership inference attacks when appeared rarely.
Although membership inference attack is a different measure for privacy than spurious score, it can be a corroboration evidence that supports the fact that privacy is leaked from spurious examples.
Test accuracy.
We measure two types of test accuracy on neural networks trained on different number of spurious examples.
The clean test accuracy measures the accuracy of the trained model on the original test data.
The spurious test accuracy simulates the case where there is a distribution shift during the test time.
Formally, spurious test accuracy is deﬁned as the accuracy on a new test dataset constructed by adding spurious features to all the test examples with a label different from c tar .
Results on clean test accuracy.
We observe that the change in clean test accuracy in our experiments is small.
Across all the models trained in Fig.
2, the minimum, maximum, average, and standard deviation of the test accuracy for each dataset are: MNIST: p . 976 , .
001 q , Fashion: p . 859 , .
010 q , CIFAR10: p . 876 , . 893 , .
003 q .
Results on spurious test accuracy.
The results are shown in
Fig.
We have two observations.
First, we see that there are already some accuracy drop even when spurious test accuracy is evaluated on models trained on zero spurious examples.
This means that these models are not robust to the existence of spurious features.
This phenomena is prominent for spurious patterns with larger norm such as
R3 .
Second, we see that spurious test accuracies start to drop even more at around 10 to 100 spurious examples.
This indicates that even with . 01 % to .
001 % of the overall training data ﬁlled with spurious examples of a certain class, the robustness to spurious features can drop signiﬁcantly.
Moreover, in Appendix B, we explore rare spurious correlation theoretically.
The theoretical ﬁndings aligns with our observations here.
Inspired by our theoretical results, we also explore ways to mitigate these rare spurious correlations in Appendix C. Finally, related works are in Appendix D.
Discussion.
Our experimental results suggest that neural networks are highly sensitive to very small amounts of spurious training data.
Furthermore, the learnt rare spurious correlations cause undesirable effects on privacy and test accuracy.
Easy learning of rare spurious correlations can lead to privacy issues (Leino & Fredrikson, 2020) – where an adversary may infer the presence of a conﬁdential image in a training dataset based on output probabilities.
It also raises fairness concerns as a neural network can draw spurious conclusions about a minority group if a small number of subjects from this group are present in the training set (Izzo et al., 2021).
We recommend to test and audit neural networks thoroughly before deployment in these applications.
[Figure] [Figure]
3. Conclusion
We demonstrate that rare spurious correlations are learnt readily by neural networks, and we look closely into this phenomenon.
We discover that a few spurious examples can lead to the model learning the spurious correlation through empirical and theoretical evidence.
We then show that these rare spurious correlations can have impact on both privacy and test accuracy of the model.
Finally, we ﬁnd that some regularization methods, including weight decay and adding
Gaussian noise to the input, can reduce the spurious correlation without sacriﬁcing accuracy both empirically and theoretically.
However, they are far from completely removing these spurious correlations.
Acknowledgements
We thank Angel Hsing-Chi Hwang for providing thoughtful comments on the paper.
This work was supported by NSF under CNS 1804829 and ARO MURI W911NF2110317.
CNC’s research is supported by Simons Investigator Fellowship, NSF grants DMS-2134157 and CCF-1565264,and
DOE grant DE-SC0022199.