On the nonlinear correlation of ML performance between data subpopulations
Abstract
Understanding the performance of machine learning models across diverse data distributions is critically important for reliable applications.
Recent works empirically find that there is a strong linear relationship between in-distribution (ID) and outof-distribution (OOD) performance, but we show that this is not necessarily true if there are subpopulation shifts.
In this paper, we empirically show that out-of-distribution performance often has nonlinear correlation with in-distribution performance under subpopulation shifts.
To understand this phenomenon, we decompose the model’s performance into performance on each subpopulation.
We show that there is a ”moon shape” correlation (parabolic uptrend curve) between the test performance on the majority subpopulation and the minority subpopulation .
This nonlinear correlations hold across model architectures, training durations and hyperparameters, and the imbalance between subpopulations.
Moreover, we show that the nonlinearity increases in the presence of spurious correlations in the training data.
We provide complementary theoretical and experimental analyses for this interesting phenomenon of nonlinear performance correlation across subpopulations.
Finally, we discuss the implications of our findings for ML reliability and fairness.
1. Introduction
Subpopulation shift is a major challenge in machine learning (ML) in real-world applications: the data seen in test time often have different distribution across subgroups (e g. Correspondence
{+Published at the ICML 
 Workshop on Spurious Correlations, Invariance,+} and Stability .
Baltimore, Maryland, USA.
Copyright 2022 by the author(s).
[Figure]
different types of users or patients) compared to the training data (Daneshjou et al., 2021). Recent empirical works find that find that there is a strong linear relationship between in-distribution (ID) and out-of-distribution (OOD) performance on dataset reconstruction shifts (ImageNetV2 (Recht et al., 2019), CIFAR-10.1 (Recht et al., 2018),
CIFAR-10.2 (Lu et al., 2020)).
Similar linear trends are also observed in transfer learning (Kornblith et al., 2019), crossbenchmark evaluation (Miller et al., 2021), and sub-type shifts (Santurkar et al., 2021).
In contrast, we empirically show that out-of-distribution performance has a nonlinear correlation with in-distribution performance under subpopulation shifts.
To understand this phenomenon, we decompose the model’s performance into performance on each subpopulation.
We show that there is a ”moon shape” correlation (parabolic uptrend curve) between the test performance on the majority subpopulation and the minority subpopulation .
We empirically show that this phenomenon holds across a wide spectrum of model architectures, training settings, and datasets.
Interestingly, the performance correlations become more nonlinear when there is a stronger spurious correlation in the training data.
We provide rigorous theoretical explanations of how the spurious correlation and the subpopulation accuracy gap are related in a simple binary classification setting.
[Figure]
2. Experimental setup
Preliminaries: ML with diverse subpopulations We consider the overall data distribution with D = { 1 , . . .
, D } diverse subpopulations.
Each subpopulation d ∈ D corresponds to a fixed data distribution P d .
In each of our main experiment, we compare the performance on two data distributions.
(1) in-distribution (ID) , or the training distribution, P tr = (cid:80) d ∈D r trd P d , where { r trd } denotes the mixture probabilities in the training set.
(2) out-ofdistribution (OOD) is also a mixture of the D subpopulations,
P ts =
(cid:80) d ∈D r tsd P d , where { r tsd } is the mixture probabilities in the test set, but with a different proportion of subpopulations, i.e., { r tsd } ̸ = { r trd } .
Experimental procedure For simplicity, we consider
D = 2 subpopulations.
The training distribution has a majority subpopulation (e.g., r trd = 90% ), and a minority subpopulation (e.g., r trd = 10% ).
As for the out-of-distribution, the majority subpopulation and minority subpopulation are equally representative (e.g., r tsd = 50% ).
On each dataset:
We first train 500 different ML models { f 1 , f 2 , ... } independently by varying the model architectures, training durations, and hyperparameters following the search space of commercial AutoML (AutoGluon).
For each trained ML model f i , we evaluate the ID performance , and the OOD performance .
Subpopulation Shift Datasets We categorize our datasets into two categories based on the underlying reason of degraded ML performance on the minority subpopulation (Eyuboglu et al., 2022; Oakden-Rayner et al., 2020):
• Spurious correlation.
If a target variable is correlated with another variable Z in the training distribution, the model may learn to rely on Z to make predictions.
One example is that cat images can be mostly indoor and dog images mostly outdoor.
We experiment with three existing datasets in the community: MetaShift (Liang & Zou, 2022), Waterbirds (Sagawa et al., 2020), and ModifiedCIFAR4 V1 (Rolf et al., 2021).
• Rare subpopulation.
Without obvious spurious correlation, ML models can still underperform on subpopulations that occur infrequently in the training set (e g. patients with a darker skin tone), since the rare subpopulation will not significantly affect model loss during training.
We adopted PACS (Li et al., 2017), OfficeHome (Venkateswara et al., 2017), Modified-CIFAR V2 (Rolf et al., 2021) for experiments.
3. The moon shape phenomenon 3.1. Finding 1: nonlinear correlation of ML performance across data subpopulations
Out-of-distribution vs. in-distribution Figure 1 show the nonlinear correlation between the out-of-distribution performance and the in-distribution performance across multiple subpopulation shifts datasets.
Moreover, datasets constructed with spurious correlations (Figure 1 top) seems to have more nonlinear correlations compared to the datasets without obvious spurious correlations (Figure 1 bottom).
Majority vs. Minority To understand this phenomenon, we decompose the model’s performance into performance on each subpopulation.
As shown in Figure 2, there is a ”moon shape” correlation (parabolic uptrend curve) between the test performance on the majority subpopulation and the minority subpopulation .
We show that this nonlinear correlation holds across model architectures, training durations and hyperparameters (Supp.
Figure 8).
It also holds for both
ImageNet-pretrained or train-from-scratch (Supp.
Figure 6).
Moreover, datasets with spurious correlations (Top row of
Figure 1) again show more nonlinearity compared to those without obvious spurious correlations (Bottom row of Figure 1).
Importantly, for datasets with spurious correlations, even with probit-transformed axes (Miller et al., 2021) as used by several prior works (Recht et al., 2019; Taori et al., 2020), the performance correlations still remain nonlinear (Supp.
Figure 7).
This confirms that such nonlinear correlations are indeed not captured by the previous work.
Discussion: Why the moon shape is not obvious Figure 3 demonstrates one reason why the non-linear correlation structure (i.e., the moon shape) is non-trivial.
Consider a thought experiment in which we interpolate two models
A , B (indicated by red circles) picked from the moon shape
[Figure] [Figure]
curve by flipping a biased coin with probability p : If the coin lands head up, classify with model A .
Otherwise classify with model B .
Varying p in [0 , 1] gives a line between model A and model B .
This thought experiment demonstrates that the interpolation line is an achievable region for the ML models, but the models deviate substantially away from this interpolation line, forming a moon shape.
Discussion: The moon shape persists within and across different training epochs In Figure 4, we stratify Figure(a) by the number of training epochs.
For each fixed training epoch, we still find a clear moon shape across the different models.
Moreover, the similar moon shape persists across different training epochs.
Results on other datasets are also similar (Supp.
Figure 9).
This finding motivates us to focus our analysis on comparing across different models rather than comparing the subpopulation performance of a single model across training epochs (which is an interesting direction complementary to our scope).
3.2. Finding 2: spurious correlation makes the moon shape more nonlinear
Setup To explore the effect of different level of spurious correlations, we conduct multiple controlled experiments on Modified-CIFAR4 V1 (Rolf et al., 2021), which was created by subsetting to the bird, car, horse, and plane classes from CIFAR-10.
The task is to predict whether the image subject moves primarily by air (plane/bird) or land (car/horse), which is spuriously correlated with whether the image contains an animal (bird/horse) or vehicle (car/plane).
The majority subpopulation is the vehicles: “land-vehicle(automobile)”, “air-vehicle(airplane)”.
The minority subpopulation is the animals: “land-animal(horse)”, “air-animal(bird)”.
We use the same test sets and only change the training data.
For the training data, we fixed the size of majority subpopulation (6,000 images) and minority subpopulation (4,000 images).
Meanwhile, we also ensure that the dataset is class-balanced: i.e., 5,000 images for both Y = 0 and
Y = 1 .
Formally, since we fix (1) P ( Y = 1) = 0 .
P ( Z = 1) = 0 .
6 , and (3) the total size of the training set as 10 , 000 images, there is effectively only one degree of freedom left, which we vary to change the level of spurious correlation.
See the theory section for the exact formula.
Intuitively, we increase the level of the spurious correlation by adding more “land-vehicle(automobile)”, “air-animal(bird)” (indicated by the red boxes in Figure 5), and removing “land-animal(horse)”, “air-vehicle(airplane)”
(gray boxes).
Results and Observations
The four scatter plot panels in Figure 5 are ordered by increasingly stronger spurious correlations.
As can be seen, interestingly, the performance correlations become more nonlinear when there is a stronger spurious correlation in the training data.
Altogether, these results indicate that the existence of spurious correlation plays a crucial role in shaping how the out-of-distribution performance correlates with the in-distribution performance, which is underexplored in the previous literature.
Motivated by our experimental findings, we provide rigorous theoretical explanations in the next section about how the spurious correlation and the subpopulation accuracy gap are related.
4. Theoretical analysis of the accuracy gap across subpopulations
We rigorously study the effect of the spurious correlation on the accuracy gap between the majority and the minority subpopulations in a binary classification setting.
We denote an input and an output random variable by X and Y , respectively.
We denote a random variable for a subpopulation by Z , where Z = 1 indicates the majority subpopulation, otherwise Z = 0 .
We assume that the underlying data generating mechanism is Z ← Y → X , that is X and Z are conditionally independent given Y , i.e. ,
X ⊥ Z | Y .
We assume
[Equation]
Note that π 1 = π 0 is a necessary and sufficient condition for
Y ⊥ Z .
In this respect, the level of spurious correlation can be expressed as | π 1 − π 0 | .
With these notations, the subpopulation accuracy gap for a model g is defined as the absolute difference between the two subpopulation accuracy:
[Equation]
[Figure]
In the following theorem, we explicitly show that the subpopulation accuracy gap is proportional to the level of spurious correlation | π 1 − π 0 | .
The proof is deferred to the
Appendix.
Theorem 1 (The higher the level of spurious correlation, the larger the accuracy gap) .
The subclass accuracy gap for a binary classifier g is expressed as follows.
Accuracy
Gap
[Equation]
where TPR and TNR denote the true positive rate
E ( g ( X ) = 1 | Y = 1) and the true negative rate
E ( g ( X ) =
| Y = 0) , respectively.
Theorem 1 shows that the subpopulation accuracy gap is expressed as a function of | π 1 − π 0 | and | TPR − TNR | .
A direct consequence is that the accuracy gap gets larger when the level of spurious correlation | π 1 − π 0 | increases.
It is possible to keep P ( Z = 1) and P ( Y = 1) as constants while | π 1 − π 0 | changes.
In particular, it occurs when π 1 and πare related as π 1 =
( P ( Z = 1) − P
( Y = 0) π 0 ) /
P ( Y = 1) , which captures the setting of Figure 5.
Remark 1 (Models on the similar ROC curve) .
Suppose that there is a trained binary classification model and
its
ROC curve is not a straight line, which is typically the case.
We can think of different points on the curve as different models whose predicted probability outputs are only different by constant shifts.
Given that a point on the ROC curve is described as (1 − TNR , TPR) , TPR changes nonlinearly with respect to TNR .
Hence, the | TPR − TNR | changes nonlinearly, and so does the accuracy gap by Theorem 1.
This can provide one explanation for our experimental observations that different models form the moon shape curve.
The setting considered in this remark is admittedly simplified to provide some intuition.
In practice, different models (with different architectures and hyperparameters) may not correspond to different points on one ROC curve.
However, if the different models do approximately trace out an ROC curve, then the intuition here can apply.
5. Discussion
This work demonstrates that the performance of different models on the majority and minority data subpopulations can have nonlinear correlations.
We show that this nonlinear correlation phenomenon is persistent across different datasets, different types of models, and both within and across training epochs.
This intriguing phenomenon also leads to nonlinear correlations in models’ performances in subpopulation shifts.
Our findings complement and contrast previous empirical studies showing a linear correlation in model performance during other types of distribution shifts.
Our experiments and theory also provide insights into how spurious correlations in the data can increase this nonlinear pattern.
Our finding has implications for model selection.
As ML model building is becoming increasingly turn-key with technologies such as automated machine learning (AutoML), selecting the best model that performs well across diverse data subpopulations is increasingly a major challenge.
Our results suggest that, when there is no spurious correlation, models with higher aggregate performance (which is largely skewed by the majority subpopulation performance) generally also perform better on the minority subpopulation.
However, with spurious correlation, the situation becomes more nuanced: there exists a phase transition point with negative correlation before, and positive correlation after.
In settings where subpopulations performance is important (e.g. fairness considerations), we recommend autoML practitioners to use similar type of scatter plots as our Figureto diagnose model selection.
More generally, subpopulation shift is ubiquitous in ML applications.
Our work highlights how model improvement in one subpopulation may have nonlinear effects on performance in other subpopulations.
Further analysis and understanding of this nonlinear pattern is an important direction of future work.
A. Extended Descriptions of The Moon Shape Phenomenon
[Figure]
A.1. Extended descriptions of the nonlinear correlation under probit-transformation
In Figure 2, we decompose the model’s performance into performance on each subpopulation.
We found that there is a ”moon shape” correlation (parabolic uptrend curve) between the test performance on the majority subpopulation and the minority subpopulation .
This nonlinear correlations hold across model architectures, training durations and hyperparameters (Supp.
Figure 8), and the imbalance between subpopulations (Figure 5).
We have also found that datasets with spurious correlations (Top row of Figure 2) show more nonlinearity compared to those without obvious spurious correlations (Bottom row of Figure 2).
Recent empirical works find that there is a strong linear relationship between in-distribution (ID) and out-of-distribution (OOD) performance under several specific types of data shifts.
Most works report the linear relationship with the original performance axis-scale, while several works report that with probit-transformed (a non-linear transformation) accuracy (Miller et al., 2021; Recht et al., 2019; Taori et al., 2020), the linear correlation of performance would be more precise.
Here the probit transform is the inverse of the cumulative density function (CDF) of the standard Gaussian distribution, i.e., l transformed =
Φ − 1 ( l ) .
Therefore, to better compare with the existing literature, we also plot with probit -transformed performance as shown in
Supp.
Figure 7.
Importantly, for datasets with spurious correlations, even with probit-transformed axes as used by several prior works (Miller et al., 2021; Recht et al., 2019; Taori et al., 2020), the performance correlations still remain nonlinear (Supp.
Figure 7).
The fact that ours remain nonlinear with probit-transformed axes (Supp.
Figure 7) is an especially exciting finding, since it confirms that the nonlinear correlations we found here under subpopulation shifts are indeed not captured by the previous work.
Our findings indicate that the linear correlation trend reported by recent work is not necessarily true if there are subpopulation shifts with spurious correlation.
As the presence of spurious correlation creates such an interesting phenomenon, this also motivates our theoretical analysis on the relationship between the level of spurious correlation and the moon shape.
A.2. Extended descriptions on the moon shape within and across different training epochs In
Figure 4, we stratify the dots in Figure 2 (a) (i.e. the trained models) by the number of training epochs.
For each fixed training epoch, we still find a clear moon shape across the different models.
Moreover, the similar moon shape persists across different training epochs.
In other words, the moon shape persists within and across different training epochs .
Supp.
Figure 9 verify that similar results are also shown when we stratify the scatter plot dots by the number of training epochs on other datasets.
This finding motivates us to focus our analysis on comparing across different models rather than comparing the subpopulation performance of a single model across training epochs (which is an interesting direction complementary to our scope).
B. Extended Related Work
Linear correlations between ID and OOD performances Existing research mostly reports linear correlations between
ID and OOD performances.
The linear correlations were first reported in recent dataset reconstruction settings including
ImageNet-V2 (Recht et al., 2019), CIFAR-10.1 (Recht et al., 2018), CIFAR-10.2 (Lu et al., 2020), where new test sets of popular benchmarks are collected closely following the original dataset creation process.
As there are subtle differences in the dataset creation pipeline, the test performance on the new test set is often lower, but appears to be linearly correlated with the performance on the original test set (Lu et al., 2020; Miller et al., 2020; Recht et al., 2018; 2019; Yadav & Bottou, 2019).
Later researchers also found the linear trends in the context of cross-benchmark evaluation (Taori et al., 2020; Miller et al., 2021), and transfer learning (Kornblith et al., 2019;
Andreassen et al., 2021), where a model’s
ImageNet test accuracy linearly correlates with the transfer learning accuracy.
Similar linear trends are also observed in sub-type shifts (Hendrycks & Dietterich, 2019;
Santurkar et al., 2021)
(e.g., the training data for the “dog” class are all from a specific breed while the test data come from another breed).
Different from these studies, we (1) focus on subpopulation shifts, where we also present the first systematic study on the performance correlation between data subpopulations, and (2) find nonlinear correlations of ML performance across data subpopulations, which is not captured in previous work.
Importantly, we show that for datasets with spurious correlations, even with probit-transformed axes as used by several prior work (Miller et al., 2021;
Recht et al., 2019; Taori et al., 2020), the performance correlations still remain nonlinear.
This confirms that the nonlinear (“moon shape”) correlation phenomenon is indeed not captured by previous work.
ML with diverse subpopulations A major challenge in ML is that a model can have very disparate performances even when it’s applied to different subpopulations of its training and evaluation data.
Models with low average error can still fail on particular groups of data points (Hashimoto et al., 2018;
Buolamwini & Gebru, 2018; Blodgett et al., 2016).
For example, predictive models for clinical outcomes that are accurate on average in a patient population are reported to underperform drastically for some subpopulations, potentially introducing or reinforcing inequities in care access and quality (Pfohl et al., 2022).
Similar performance disparity, have also been observed in radiograph classification (Badgeley et al., 2019;
Zech et al., 2018; DeGrave et al., 2021), face recognition (Grother et al., 2011; Buolamwini & Gebru, 2018), speech recognition (Koenecke et al., 2020; Blodgett et al., 2016;
Jurgens et al., 2017), academic recommender systems (Sapiezynski et al., 2017)), and automatic video captioning (Tatman, 2017), among others.
Worse, as model accuracy affects user retention, the minority group might shrink and thus even amplifies the performance disparity over time (Hashimoto et al., 2018;
Fuster et al., 2017).
These case studies highlight the importance of understanding the ML performance disparity across subpopulations.
C. Proofs
In this section [Equation] we provide a proof of Theorem 1.
Proof of Theorem 1.
For any z ∈ { 0 , 1 } , we have
[Equation]
Here [Equation] the second equality is due to X ⊥ Z | Y .
Therefore, the accuracy gap between the two subpopulations is expressed as follows.
[Equation]
By the Bayes’ theorem
[Equation]
we have
[Equation]
Since P ( Z = 0) = 1 − ( π 1 P ( Y = 1) + π 0 P ( Y = 0))
= 1 − π 1 + ( π 1 − π 0 ) P ( Y = 0) , we have
Accuracy Gap
[Equation]
It concludes a proof.
[Figure]
[Table]
[Figure]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Figure]
M i n o r i t y S u bp
o p u l a t i o n A cc ( P r o b i t - s c a l e )
[Figure]
OfficeHome
[Figure] [Figure] [Figure] [Figure]
[Figure] [Figure]
[Figure]
[Figure] [Figure] [Figure] M i n o r i t y S u bp
o p u l a t i o n A cc [Figure] P r o b i t - s c a l [Figure]
[Figure] [Figure] i [Figure]
l a t i [Figure]
n A
[Figure] [Figure]