S ELECTIVE C LASSIFICATION C AN M AGNIFY D ISPARITIES A CROSS G ROUPS
Anonymous authors
Paper under double-blind review
A BSTRACT
Selective classiﬁcation, in which models can abstain on uncertain predictions, is a natural approach to improving accuracy in settings where errors are costly but abstentions are manageable.
In this paper, we ﬁnd that while selective classiﬁcation can improve average accuracies, it can simultaneously magnify existing accuracy disparities between various groups within a population, especially in the presence of spurious correlations.
We observe this behavior consistently across ﬁve vision and NLP datasets.
Surprisingly, increasing abstentions can even decrease accuracies on some groups.
To better understand this phenomenon, we study the margin distribution, which captures the model’s conﬁdences over all predictions.
For symmetric margin distributions, we prove that whether selective classiﬁcation monotonically improves or worsens accuracy is fully determined by the accuracy at full coverage (i.e., without any abstentions) and whether the distribution satisﬁes a property we term left-log-concavity.
Our analysis also shows that selective classiﬁcation tends to magnify full-coverage accuracy disparities.
Motivated by our analysis, we train distributionally-robust models that achieve similar fullcoverage accuracies across groups and show that selective classiﬁcation uniformly improves each group on these models.
Altogether, our results imply selective classiﬁcation should be used with care and underscore the importance of models that perform equally well across groups at full coverage.
I NTRODUCTION
Selective classiﬁcation, in which models make predictions only when their conﬁdence is above a threshold, is a natural approach in settings where errors are costly but abstentions are manageable.
For example, in medical and criminal justice applications, model mistakes can have serious consequences, whereas abstentions can be handled by backing off to the appropriate human experts.
Prior work has shown that across a broad array of applications, more conﬁdent predictions indeed tend to be more accurate (e.g., Hanczar & Dougherty (2008); Yu et al.
Toplak et al.
Mozannar & Sontag (2020); Kamath et al.
By varying the conﬁdence threshold, we can select an appropriate trade-off between the abstention rate and the accuracy of the predictions made.
In this paper, we report a cautionary ﬁnding: while selective classiﬁcation improves average accuracy, it can magnify existing accuracy disparities between various groups within a population, especially in the presence of spurious correlations.
We observe this behavior across ﬁve vision and NLP datasets and two popular selective classiﬁcation methods: softmax response (Cordella et al., 1995; Geifman & El-Yaniv, 2017) and Monte Carlo dropout (Gal & Ghahramani, 2016).
Surprisingly, we ﬁnd that increasing the abstention rate can even decrease accuracies on the groups that have lower accuracies at full coverage: on those groups, the models are not only wrong more frequently, but their conﬁdence can actually be anticorrelated with whether they are correct.
Even on datasets where selective classiﬁcation improves accuracies across all groups, we ﬁnd that it preferentially helps groups that already have high accuracies, further widening group disparities.
These group disparities are especially problematic in the same high-stakes areas where we might want to deploy selective classiﬁcation, like medicine and criminal justice; there, poor performance on particular groups is already a signiﬁcant issue (Chen et al., 2020; Hill, 2020).
For example, we study a variant of CheXpert (Irvin et al., 2019), where the task is to predict if a patient has pleural effusion (ﬂuid around the lung) from a chest X-ray.
As these are commonly treated with chest tubes, models can latch onto this spurious correlation and fail on the group of patients with pleural effusion [Figure] - −
[Figure]
τ
[Figure]
Margin
D
en s i t y
Incorrect
Abstain
Correct 0 .0 .0 .0 .0 .1 .
0 Average coverage 0 .0 .0 {+.
 S e l e c t i v e a cc u r+}
[Figure]
cy Average Worst group but not chest tubes or other support devices.
However, this group is the most clinically relevant as it comprises potentially untreated and undiagnosed patients (Oakden-Rayner et al.
To better understand why selective classiﬁcation can worsen accuracy and magnify disparities, we analyze the margin distribution, which captures the model’s conﬁdences across all predictions and determines which examples it abstains on at each threshold (Figure 1).
For example, when the margin distribution is symmetric, we prove that whether selective classiﬁcation monotonically improves or worsens accuracy is fully determined by the accuracy at full coverage (i.e., without any abstentions) and whether the distribution satisﬁes a property we term left-log-concavity.
To our knowledge, this is the ﬁrst work to characterize whether selective classiﬁcation (monotonically) helps or hurts accuracy in terms of the margin distribution, and to compare its relative effects on different groups.
Our analysis shows that selective classiﬁcation tends to magnify accuracy disparities that are present at full coverage (i.e., with no abstentions)
Motivated by our analysis, we ﬁnd that selective classiﬁcation on group-DRO models (Sagawa et al., 2020)—which achieve similar accuracies across groups at full coverage by using group annotations during training—uniformly improves group accuracies at lower coverages, substantially mitigating the disparities observed on standard models that are instead optimized for average accuracy.
These models are not a silver bullet: unlike standard models, they rely on knowing group identities during training, which is not always practical.
However, they illustrate the potential of models that achieve small accuracy disparities at full coverage and can therefore maintain uniformly high accuracies under selective classiﬁcation.
R ELATED WORK
Selective classiﬁcation.
Abstaining from uncertain predictions is a classic idea (Chow, 1957; Hellman, 1970), and uncertainty estimation is an active area of research, from the popular approach of using softmax probabilities (Geifman & El-Yaniv, 2017) to more sophisticated methods using dropout (Gal & Ghahramani, 2016), ensembles (Lakshminarayanan et al., 2017), or training snapshots (Geifman et al., 2018).
Others incorporate abstention into model training (Bartlett & Wegkamp, 2008;
Geifman & El-Yaniv, 2019;
Feng et al., 2019) and learn to abstain on examples human experts are more likely to get correct (Raghu et al., 2019; Mozannar & Sontag, 2020; De et al., 2020).
Selective classiﬁcation can also improve out-of-distribution accuracy (Pimentel et al., 2014; Hendrycks & Gimpel, 2017; Liang et al., 2018; Ovadia et al., 2019; Kamath et al., 2020).
On the theoretical side, early work characterized optimal abstention rules given well-speciﬁed models (Chow, 1970;
Hellman & Raviv, 1970), with more recent work on learning with perfect precision (El-Yaniv &
Wiener, 2010) and guaranteed risk (Geifman & El-Yaniv, 2017).
We build on this literature by establishing general conditions on the margin distribution for when selective classiﬁcation helps, and importantly, by showing that it can magnify group disparities.
Group disparities.
The problem of models performing poorly on some groups of data has been widely reported (e.g., Hovy & Søgaard (2015); Blodgett et al. (2016); Corbett-Davies et al.
Tatman (2017); Hashimoto et al.
These disparities can arise when models latch onto spurious correlations, e.g., demographics (Buolamwini & Gebru, 2018; Borkan et al., 2019), image backgrounds (Ribeiro et al., 2016; Xiao et al., 2020), spurious clinical variables (Badgeley et al., 2019; Oakden-Rayner et al., 2020), or linguistic artifacts (Gururangan et al., 2018; McCoy et al., 2019).
These disparities have implications for model robustness and equity, and mitigating them is an important open challenge, especially when models are not privy to group identities (Dwork et al., 2012;
Hardt et al., 2016; Kleinberg et al., 2017; Duchi et al., 2019; Sagawa et al., 2020).
Our work shows that selective classiﬁcation can exacerbate this problem and must therefore be used with care.
S ETUP
A selective classiﬁer takes in an input x ∈ X and either predicts a label y ∈ Y or abstains.
We study standard conﬁdence-based selective classiﬁers (ˆ y, ˆ c ) , where ˆ y :
X → Y outputs a prediction and ˆ c :
X → R + outputs the model’s conﬁdence, which is always non-negative, in that prediction.
It abstains on x whenever the conﬁdence ˆ c ( x ) is below some threshold τ and predicts ˆ y ( x ) otherwise. Data and training.
We consider a data distribution D over X ×Y ×G , where G = { 1 , 2 , . . . , k } cor- responds to a group variable that is unobserved by the model. We study the common setting where the model ˆ y is trained under full coverage (i.e., without taking into account any abstentions) and the conﬁdence function ˆ c is then derived from the trained model, as described in the next paragraph. We will primarily consider models ˆ y trained by empirical risk minimization (i.e., to minimize the average training loss); in that setting, the group g ∈ G is never observed.
In Section 7, we will consider the group DRO training algorithm (Sagawa et al., 2020), which observes g at training time only. In both cases, g is not observed at test time and the model thus does not take in g . This is a common assumption: e.g., we might want to ensure that a face recognition model has equal accuracies across genders, but the model only sees the photograph ( x ) and not the gender ( g ). Conﬁdence.
We will primarily consider softmax response (SR) selective classiﬁers, which take ˆ c ( x ) to be the normalized logit of the predicted class.
Formally, we consider models that estimate ˆ P ( y | x ) (e.g., through a softmax) and predict ˆ y ( x ) = arg max y ∈Y ˆ P ( y | x ) , with the corresponding probability estimate ˆ P (ˆ y ( x ) | x ) . For binary classiﬁers, we deﬁne the conﬁdence ˆ c ( x ) as
[Equation] This corresponds to a conﬁdence of ˆ c ( x ) = 0 when ˆ P (ˆ y ( x ) | x ) = 0 .
5 , i.e., the classiﬁer is completely unsure of its prediction. We generalize this notion to multi-class classiﬁers in Section A.1. Softmax response is a popular technique applicable to neural networks and has been shown to improve average accuracies on a range of applications (Geifman & El-Yaniv, 2017). Other methods offer alternative ways of computing ˆ c ( x ) ; in Appendix B.1, we also run experiments where ˆ c ( x ) is obtained via Monte Carlo (MC) dropout (Gal & Ghahramani, 2016), with similar results.
Metrics. The performance of a selective classiﬁer at a threshold τ is typically measured by its average (selective) accuracy on its predicted points, P [ˆ y ( x ) =
y | ˆ c ( x ) ≥ τ ] , and its average coverage , i.e., the fraction of predicted points P [ˆ c ( x ) ≥ τ ] .
{+We call the average accuracy at threshold 
+}
the full-coverage accuracy , which corresponds to the standard notion of accuracy without any abstentions.
We always use the term accuracy w.r.t.
some threshold τ ≥ 0 ; where appropriate, we emphasize this by referring to selective accuracy , but the terms are interchangeable.
Following convention, we evaluate models by varying τ and tracing out the accuracy-coverage curve (El-Yaniv & Wiener, 2010).
As Figure 1 illustrates, this curve is fully determined by the distribution of the margin , which is ˆ c ( x ) on correct predictions ( ˆ y ( x ) =
y ) and − ˆ c ( x ) otherwise.
We are also interested in evaluating performance on each group.
For a group g ∈ G , we compute its group (selective) accuracy by conditioning on the group, P [ˆ y ( x ) =
y | g, ˆ c ( x ) ≥ τ ] , and we deﬁne group coverage analogously as P [ˆ c ( x ) ≥ τ | g ] .
We pay particular attention to the accuracy of the worst group argmin g P [ˆ y ( x ) =
y | g ] that has the lowest accuracy at full coverage.
In our setting, we only use group information to evaluate the model, which does not observe g at training or test time; we will relax this later when studying distributionally-robust models that use g at training time. Dataset Modality Total examples
Prediction task
Spurious attributes A
CelebA Photos 202, Hair color Gender CivilComments
Text 448,
Toxicity
Mention of Christianity Waterbirds
Photos 11,
Waterbird or landbird
Water or land background CheXpert-device
X-rays 156,
Pleural effusion
Support device MultiNLI
Sentence pairs 412,
Entailment
Negation words [Figure] Datasets. We consider ﬁve datasets (Table 1) on which prior work has shown that models latch onto spurious associations, thereby performing well on average but poorly on the groups of data where the spurious association does not hold up.
Following Sagawa et al.
(2020), we deﬁne for each dataset the set of labels Y as well as a set of attributes A that are spuriously associated with the labels, and then form one group for each value of ( y, a ) ∈ Y × A .
For example, in the pleural effusion example from the introduction, one group would be the set of patients with pleural effusion ( y = 1 ) but not support devices ( a = 0 ).
For simplicity, each dataset has |Y| = |A|
= 2 , with the exception of MultiNLI which is multi-class, with |Y| = 3 .
More dataset details are in Appendix C.1.
E VALUATING SELECTIVE CLASSIFICATION ON GROUPS
We start by investigating how selective classiﬁcation affects group (selective) accuracies across the ﬁve datasets in Table 1.
For each dataset, we train a standard model with empirical risk minimization (i.e., to minimize average training loss), using ResNet50 for CelebA and Waterbirds; DenseNet for CheXpert; and BERT for CivilComments and MultiNLI.
Details are in Appendix C. We focus on softmax response (SR) selective classiﬁers, but show similar results for MC-dropout in Section B.1.
Accuracy-coverage curves.
Figure 2 shows group accuracy-coverage curves for each dataset, with the average in blue, worst group in red, and other groups in gray.
On all datasets, average accuracies improve as coverage decreases.
However, the worst-group curves fall into three categories: 1.
Decreasing.
Strikingly, on CelebA and CivilComments, worst-group accuracy decreases with coverage: the more conﬁdent the model is on worst-group points, the more likely it is incorrect.
Mixed.
On Waterbirds and CheXpert-device, as coverage decreases, worst-group accuracy sometimes increases (though not by much, except at noisy, low coverages) and sometimes decreases.
Slowly increasing .
On MultiNLI, as coverage decreases, worst-group accuracy consistently improves but more slowly than other groups: from full coverage to 50% average coverage, worstgroup error goes from 35% to 25% while second-to-worst group error goes from 23% to 5%. [Figure]
1 Let C τ be the set of all examples that ( ˆ y , ˆ c ) correctly predicts at threshold τ :
[Equation]
Sample a subset C ga τ of size |C τ | uniformly at random from C 0 , which is the set of all examples that ˆ y would have predicted correctly at full coverage. 2 Repeat for incorrect points. Let I τ be the analogous set of incorrect predictions at τ : [Equation]
Sample a subset I ga τ of size |I τ | uniformly at random from I 0 . 3 Return C ga τ ∪ I ga τ .
Since |C ga τ | =
|C τ | and |I
ga τ | = |I
τ | , the group-agnostic baseline makes the same numbers of correct and incorrect predictions as ( ˆ y , ˆ c ) at threshold τ , but distributes these correct and incorrect predictions uniformly in a group-agnostic manner.
Group-agnostic baselines and Robin Hood oracles.
The results above show that even when selective classiﬁcation is helping the worst group, it intuitively seems to help other groups more.
We formalize this notion by comparing each classiﬁer to a matching group-agnostic baseline that, at each threshold, makes the same number of correct and incorrect predictions, but distributes these predictions uniformly at random across points without knowing group identities (see Algorithm 1).
By construction, the group-agnostic baseline has an identical average accuracy-coverage curve as its corresponding selective classiﬁer, but can differ on the group accuracies. Formally, we show in Proposition 6 (Appendix A.2) that it satisﬁes equalized odds (Hardt et al., 2016) with respect to which points it decides to predict or abstain on.
The group-agnostic baseline represents the minimal level of worst-group accuracy that we should hope for.
Ideally, selective classiﬁcation would preferentially increase worst-group accuracy until it matches the other groups.
We can capture this optimistic scenario by constructing, for a given selective classiﬁer, a corresponding Robin Hood oracle which, as above, also makes the same number of correct and incorrect predictions (see Algorithm 2 in Appendix A.3).
However, unlike the group-agnostic baseline, in the Robin Hood oracle the correct predictions are not chosen uniformly at random; instead, we prioritize picking them from the worst group, then the second worst group, etc.
Likewise, we prioritize picking the incorrect predictions from the best group, then the second best group, etc.
This results in worst-group accuracy rapidly increasing at the cost of the best group.
Both the group-agnostic baseline and the Robin Hood oracle are not implementable in practice without knowledge of all groups and labels.
They act instead as references: selective classiﬁers that preferentially help the worst group over others would have worst-group accuracy-coverage curves that lie between those of the group-agnostic and Robin Hood curves.
Unfortunately, Figure 3 shows that the SR selective classiﬁers substantially underperform their group-agnostic counterparts.
We conclude that they are disproportionately helping groups that already have higher accuracies, further exacerbating the disparities between groups.
We show similar results for MC-dropout in Section B.1.
[Figure]
A NALYSIS : M ARGIN DISTRIBUTIONS AND ACCURACY - COVERAGE CURVES
In the previous section, we saw that selective classiﬁcation can sometimes increase and sometimes decrease worst-group accuracy, and that even when it does increase worst-group accuracy, it still exacerbates disparities by disproportionately helping other groups.
We now turn to a theoretical analysis of how these behaviors arise from the margin distributions, starting in this section with the conditions under which accuracy monotonically increases or decreases with coverage.
Recall that the margin of a selective classiﬁer (ˆ y, ˆ c ) is ˆ c ( x ) ≥ 0 on correct predictions ( ˆ y ( x ) = y ) and − ˆ c ( x ) ≤ 0 otherwise.
The behavior of selective classiﬁcation on average versus the worst group is reﬂected in their different margin distributions, which we show in Figure 4.
For example, the worst-group distributions are consistently shifted to the left with many conﬁdent but incorrect examples, whereas most conﬁdent examples are correct on average.
We will analyze a broad class of symmetric and skew-symmetric distributions that reﬂect most of these empirical distributions, as well as those reported in prior work (Balasubramanian et al., 2011; Lakshminarayanan et al., 2017).
Setup.
We consider distributions over margins that have a differentiable cumulative distribution function (CDF) and a density, denoted by corresponding upper- and lower-case variables (e.g., F and f , respectively).
The accuracy of a selective classiﬁer is completely determined by its margin distribution F ; at a threshold τ , it makes incorrect predictions on points with margin ≤ − τ , correct predictions on points with margin ≥ τ , and abstains otherwise.
We denote its (selective) accuracy at τ as A F ( τ ) =
(1 − F ( τ )) /
( F ( − τ ) + 1 − F ( τ )) .
Since increasing the threshold τ monotonically decreases coverage, we focus on studying accuracy as a function of τ .
All proofs are in Appendix D.
5. S YMMETRIC DISTRIBUTIONS
We begin by considering symmetric distributions over margins.
For symmetric distributions, monotonicity hinges on a generalization of log-concave distributions, which we term left-log-concavity :
Deﬁnition 1 (Left-log-concave distributions) .
A distribution is left-log-concave if its CDF is logconcave on ( −∞ , µ ] , where µ is the mean of the distribution.
Left-log-concave distributions are a strict superset of the broad family of log-concave distributions (e.g., Gaussian, beta, uniform), which require log-concave densities (instead of CDFs) on their entire support.
Notably, left-log-concave distributions can be multimodal: a symmetric mixture of two
Gaussians is left-log-concave, but not typically log-concave (Lemma 1 in Appendix D).
We establish a tight connection between the monotonicity of the accuracy-coverage curve and leftlog-concavity, and show that for selective classiﬁers with left-log-concave margin distributions, the full-coverage accuracy alone determines whether accuracy will increase or decrease with τ .
Proposition 1 (Left-log-concavity and monotonicity) .
Let F be the CDF of a symmetric distribution.
If F is left-log-concave, then A F ( τ ) is monotonically increasing in τ if A F (0) ≥ 1 /and monotonically decreasing otherwise.
Conversely, if A F d ( τ ) is monotonically increasing for all translations F d such that F d
Proposition 1 is consistent with the observation that selective classiﬁcation improves average accuracy but hurts worst-group accuracy.
As an illustration, consider a symmetric mixture of two
Gaussians whose average accuracy is > 50% at full coverage, but where the worst-group accuracy (on one of the Gaussians) is < 50%.
Since both the overall and worst-group margin distributions are left-log-concave (Lemma 1), Proposition 1 implies that worst-group accuracy will decrease monotonically with τ while the average accuracy improves monotonically with τ .
Applied to CelebA (Figure 4), Proposition 1 is also consistent with how average accuracy improves but worst-group accuracy, which is < 50% at full coverage, worsens.
Beyond our group setting,
Proposition 1 helps to explain why selective classiﬁcation typically improves average accuracy in the literature, since average accuracies are typically high at full coverage and margin distributions often resemble Gaussians (Balasubramanian et al., 2011; Lakshminarayanan et al., 2017).
5. S KEWED DISTRIBUTIONS
Empirically, we also observe skewed margin distributions in Figure 4.
To study the effect of skew on accuracies, we consider skew-symmetric distributions (Azzalini & Regoli, 2012):
Deﬁnition 2.
A distribution with density f α,µ is skew-symmetric with skew α and center µ if
[Equation]
for all τ ∈ R , where h and G are the density and CDF of distributions that are symmetric about 0.
In other words, f α,µ is a skewed form of the symmetric density h , where higher α means more right skew, and setting α = 0 yields a (translated) h .
Skew-symmetric distributions are a broad family and include, for example, the common skew-normal distribution.
We ﬁrst show that right skew monotonically increases accuracy.
Intuitively, since points with positive margin are correctly predicted, right skew improves accuracy and left skew hurts accuracy.
Proposition 2 (Accuracy is monotone with skew) .
Let F α,µ be the CDF of a skew-symmetric distribution.
For all τ ≥ 0 , A F α,µ ( τ ) is monotonically increasing in α .
In addition, skew in the corresponding direction preserves monotone accuracy-coverage curves:
Proposition 3 (Skew in the same direction preserves monotonicity) .
Let F α,µ be the CDF of a skewsymmetric distribution.
If µ ≥ 0 and A F 0 ,µ ( τ ) is monotonically increasing in τ , then A F α,µ ( τ ) is also monotonically increasing in τ for any α > 0 .
Similarly, if µ ≤ 0 and A F 0 ,µ ( τ ) is monotonically decreasing in τ , then A F α,µ ( τ ) is also monotonically decreasing in τ for any α < 0 .
These results are consistent with the mixed worst-group accuracies in Waterbirds and CheXpertdevice (Figure 3), which have full-coverage accuracies > 0.50 and left-skewed margin distributions (skewness -0.30 and -0.33 respectively; Figure 4).
On a margin distributions with no skew (Proposition 1) or right skew (Proposition 3), we would expect accuracy to monotonically increase if fullcoverage accuracy is > 0.50.
However, from Proposition 2, left skew decreases accuracies, and
Proposition 3 does not apply since left skew need not preserve monotonically increasing accuracy.
Discussion.
The results above establish left-log-concavity as a key property of the margin distribution that determines the monotonicity of the accuracy-coverage curve on symmetric distributions and consequently on their skew-symmetric counterparts.
Since accuracy-coverage curves are preserved under all odd, monotone transformations of margins (Lemma 8 in Appendix D), these results also generalize to odd, monotone transformations of these (skew-)symmetric distributions.
Many margin distributions—in Figure 4 and in the broader literature (Balasubramanian et al., 2011; Lakshminarayanan et al., 2017)—resemble the distributions studied above (e.g., Gaussians and skewed Gaussians). From our analysis, we thus expect selective classiﬁcation to improve average accuracy but worsen worst-group accuracy when worst-group accuracy at full coverage is low to begin with.
A NALYSIS : C OMPARISON TO GROUP - AGNOSTIC BASELINE
Even if selective classiﬁcation improves worst-group accuracy, it can still exacerbate group disparities in the sense of underperforming the equalized-odds group-agnostic baseline on the worst group, as observed in Section 4.
In this section, we continue our analysis of the margin distribution and show that while it is possible to outperform the baseline, it is challenging to do so, especially when the accuracy disparity at full coverage is large.
Setup.
Throughout this section, we decompose the margin distribution into two components F = pF wg + (1 − p ) F others , where F wg and F others correspond to the margin distributions of the worst group and of all other groups combined, respectively, and the worst group has strictly worse accuracy at full coverage than the other groups (i.e., A F wg (0) <
A F others (0) ).
Recall from Section 4 that for any selective classiﬁer (i.e., any margin distribution), its group-agnostic baseline has the same average accuracy at each threshold τ but potentially different group accuracies.
We denote the worst-group accuracy of the baseline as ˜ A F wg ( τ ) , which can be written in terms of F wg , F others , and p (Appendix A.2).
Other notation follows Section 5, and all proofs are in Appendix E.
A selective classiﬁer with margin distribution F is said to outperform the baseline on the worst group if ˜ A F wg ( τ ) ≤ A F wg ( τ ) for all τ ≥ 0 .
To establish a necessary condition for outperforming the baseline, we study the neighborhood of τ = 0 , which corresponds to full coverage:
Proposition 4 (Necessary condition for outperforming the baseline) .
Assume that 1 / 2 < A F wg (0) <
A F others (0) < 1 and f wg (0) > 0 .
If ˜ A F wg ( τ ) ≤ A F wg ( τ ) for all τ ≥ 0 , then
[Equation]
The RHS is the ratio of full-coverage errors; the larger the disparity between the worst group and the other groups at full coverage, the harder it is to satisfy this condition.
In Appendix F, we simulate mixtures of Gaussians and show that this condition is rarely fulﬁlled.
Motivated by the empirical margin distributions, we apply Proposition 4 to the setting where F wg and F others are both log-concave and are translated and scaled versions of each other.
We show that if the worst group is of a larger scale than the other groups, it underperforms the baseline at some τ :
Corollary 1 (Outperforming the baseline requires smaller scaling for log-concave distributions) .
Assume that 0 .
5 <
A F wg (0) <
A F others (0) < 1 , F wg is log-concave, and f others ( τ ) =
vf wg ( v ( τ − µ others ) + µ wg ) for all τ ∈ R , where v is a scaling factor.
If v > 1 , then ˜ A F wg ( τ ) >
A F wg ( τ ) for some τ ≥ 0 .
This is consistent with the empirical margin distributions on Waterbirds: the worst group has higher variance, implying v > 1 as v is the ratio of the worst group’s standard deviation to the other group’s.
A further special case is when F wg and F others are log-concave and translations of each other, without scaling.
In this setting, selective classiﬁcation underperforms the baseline at all thresholds τ .
Proposition 5 (Translated, log-concave distributions always underperform the baseline) .
Assume
F wg and F others are log-concave and f others ( τ ) =
f wg ( τ − d ) for all τ ∈ R .
Then for all τ ≥ 0 ,
[Equation]
This helps to explain our results on CheXpert-device, where the worst-group and average margin distributions are approximately translations of each other, and selective classiﬁcation signiﬁcantly underperforms the baseline at all conﬁdence thresholds.
S ELECTIVE CLASSIFICATION ON GROUP DRO MODELS
Our above analysis suggests that selective classiﬁcation tends to exacerbate group disparities, especially when the full-coverage disparities are large.
This motivates a potential solution: by reducing group disparities at full coverage, models are more likely to satisfy Proposition 4, which states a necessary condition for outperforming the group-agnostic baseline. In this section, we explore this approach by training models using group distributionally robust optimization (group DRO) (Sagawa et al., 2020), which minimizes the worst-group training loss
[Equation]
Unlike standard models, group DRO relies on group annotations at training time. Consistent with prior work, we found that group DRO models have much smaller full-coverage disparities (Figure 5).
[Figure]
[Figure]
Moreover, worst-group accuracies consistently improve as coverage decreases and at a rate that is largely comparable to the group-agnostic baseline, though gaps remain on Waterbirds and CheXpert.
While our theoretical analysis motivates the above approach, whether selective classiﬁcation uniformly improves group accuracies ultimately depends on the margin distributions of each group, not just on their full-coverage accuracies.
Although group DRO models only optimize for similar full-coverage accuracies across groups, we found that they have strikingly similar average and worstgroup margin distributions across all ﬁve datasets (Figure 6), explaining why selective classiﬁcation behaves uniformly over groups in those models.
Group DRO models are not a silver bullet: they rely on group annotations for training, which can be impractical, and in some settings we might not even know in advance which groups to consider.
Nevertheless, these results underscore the importance of developing models with uniform full-coverage performance across groups, especially when deployed in tandem with selective classiﬁcation.
D ISCUSSION
We have shown that selective classiﬁcation can magnify group disparities and must therefore be applied with caution.
This is an insidious failure mode, since selective classiﬁcation generally improves average accuracy and can appear to be working well if we do not look at group accuracies.
However, we also found that selective classiﬁcation can still work well on models that have equal full-coverage accuracies across groups.
Training such models, especially without relying on too much additional information at training time, remains an important research direction.
On the theoretical side, we characterized the behavior of selective classiﬁcation in terms of the margin distributions; an open question is how different margin distributions arise from different data distributions, models, and training procedures.
Another open question is how other selective classiﬁcation methods beyond softmax-response and Monte Carlo dropout, such as those that jointly train the prediction and conﬁdence functions, fare in terms of group disparities.
Finally, in this paper we focused on studying selective accuracy in isolation; accounting for the cost of abstention and the equity of different of different coverages on different groups is an important direction for future work.
A S ETUP A. S OFTMAX RESPONSE SELECTIVE CLASSIFIERS
In this section, we describe our implementation of softmax response (SR) selective classiﬁers (Geifman & El-Yaniv, 2017).
Recall from Section 3 that a selective classiﬁer is a pair (ˆ y, ˆ c ) , where ˆ y : X → Y outputs a prediction and ˆ c : X → R + outputs the model’s conﬁdence, which is always non-negative, in that prediction.
SR classiﬁers are deﬁned for neural networks (for classiﬁcation), which generally have a last softmax layer over the k possible classes.
For an input point x , we denote its maximum softmax probability, which corresponds to its predicted class ˆ y ( x ) , as ˆ P (ˆ y ( x ) | x ) .
We deﬁned the conﬁdence ˆ c ( x ) for binary classiﬁers as
[Equation]
Since the maximum softmax probability ˆ P (ˆ y ( x ) | x ) is at least 0 .
5 for binary classiﬁcation, ˆ c ( x ) is nonnegative for each x , and is thus a valid conﬁdence.
For k > 2 classes, however, ˆ P (ˆ y ( x ) | x ) can be less than 0 . 5 , in which case ˆ c ( x ) would be negative. To ensure that conﬁdence is always nonnegative, we deﬁne ˆ c ( x ) for k classes to be
[Equation]
With k classes, the maximum softmax probability ˆ P (ˆ y ( x ) | x ) ≥ 1 /k , and therefore we can verify that ˆ c ( x ) ≥ 0 as desired.
Moreover, when k = 2 , Equation (9) reduces to our original binary conﬁdence; we can therefore interpret the general form in Equation (9) as a normalized logit.
Note that ˆ c ( x ) is a monotone transformation of the maximum softmax probability ˆ P (ˆ y ( x ) | x ) .
Since the accuracy-coverage curve of a selective classiﬁer only depends on the relative ranking of ˆ c ( x ) across points, we could have equivalently set ˆ c ( x ) to be ˆ P (ˆ y ( x ) | x ) .
However, following prior work, we choose the logit-transformed version to make the corresponding distribution of conﬁdences easier to visualize (Balasubramanian et al., 2011; Lakshminarayanan et al., 2017).
Finally, we remark on one consequence of SR on the margin distribution for multi-class classiﬁcation.
Recall that we deﬁne the margin of an example to be ˆ c ( x ) on correct predictions ( ˆ y ( x ) = y ) and − ˆ c ( x ) otherwise, as described in Section 3.
In Figure 4, we plot the margin distributions of SR selective classiﬁers on all ﬁve datasets.
We observe that on MultiNLI, which is the only multi-class dataset (with k = 3 ), there is a gap (region of lower density) in the margin distribution around 0.
We attribute this gap in part to the comparative rarity of seeing a maximum softmax probability of 13 when k = 3 versus seeing 12 when k = 2 ; in the former, all three logits must be the same, while for the latter only two logits must be the same.
A. G ROUP - AGNOSTIC BASELINE
Here, we describe the group-agnostic baseline described in Section 4 in more detail.
We elaborate on the construction from the main text and then deﬁne the baseline formally.
Finally, we show that the group-agnostic baseline satisﬁes equalized odds.
A.2. D EFINITION
We begin by recalling on the construction described in the main text for a ﬁnite test set D .
In
Algorithm 1 we relied on two important sets; the set of correctly classiﬁed points at threshold τ , C τ :
[Equation]
and similarly, the set of incorrectly classiﬁed points at threshold τ , I τ :
[Equation]
[Equation]
To compute the accuracy of the group-agnostic baseline, we sample a subset C ga τ of size
|C τ | uni- formly at random from C 0 , and similarly a subset I ga τ of size |I
τ | uniformly at random from I 0 . The group-agnostic baseline makes predictions when examples are in C ga τ ∪ I ga τ and abstains otherwise. We compute group-accuracies over this set of predicted examples. Note that the group accuracies, as deﬁned, are randomized due to the sampling.
For the remainder of our analysis, we take the expectation over this randomness to compute the group accuracies.
We now generalize the above construction by considering data distributions D .
We ﬁrst deﬁne the number of correctly and incorrectly classiﬁed points:
Deﬁnition 3 (Correctly and incorrectly classiﬁed points) .
Consider a selective classiﬁer (ˆ y, ˆ c ) .
For each threshold τ , we deﬁne the fractions of points that are predicted (not abstained on), and correctly or incorrectly classiﬁed, as
[Equation]
We deﬁne analogous metrics for each group g as
[Equation]
For each threshold τ , we will make predictions on a C ( τ ) /C
(0) fraction of the C (0) total (probability mass of) correctly classiﬁed points.
Since each group g has C g (0) correctly classiﬁed points, at threshold τ , the group-agnostic baseline will make predictions on C g (0) C ( τ ) /C
(0) correctly classiﬁed points in group g .
We can reason similarly over the incorrectly classiﬁed points.
Putting it all together, we can deﬁne the group-agnostic baseline as satisfying the following:
Deﬁnition 4 (Group-agnostic baseline) .
Consider a selective classiﬁer (ˆ y, ˆ c ) and let ˜ C , ˜ I , ˜ C g , ˜
I g denote the analogous quantities to Deﬁnition 3 for its matching group-agnostic baseline.
For each threshold τ , these satisfy
[Equation]
and for each threshold τ and group g ,
[Equation]
[Equation]
The group-agnostic baseline thus has the following accuracy on group g :
[Equation]
A.2. C ONNECTION TO EQUALIZED ODDS
We now show that the group-agnostic baseline satisﬁes equalized odds with respect to which points it predicts or abstains on.
The goal of a selective classiﬁer is to make predictions on points it would get correct (i.e., ˆ y ( x ) =
y ) while abstaining on points that it would have gotten incorrect (i.e., ˆ y ( x ) (cid:54) = y ).
We can view this as a meta classiﬁcation problem, where a true positive is when the selective classiﬁer decides to make a prediction on a point x and gets it correct ( ˆ y ( x ) = y ), and a false positive is when the selective classiﬁer decides to make a prediction on a point x and gets it incorrect ( ˆ y ( x ) (cid:54) =
y ).
As such, we can deﬁne the true positive rate R TP ( τ ) and false positive rate
R FP ( τ ) of a selective classiﬁer:
Deﬁnition 5.
The true positive rate of a selective classiﬁer at threshold τ is
[Equation]
and the false positive rate at threshold τ is
[Equation]
[Equation]
Analogously, the true positive and false positive rates on a group g are
[Equation]
[Equation]
The group-agnostic baseline satisﬁes equalized odds (Hardt et al., 2016) with respect to this deﬁnition:
Proposition 6.
The group-agnostic baseline deﬁned in Deﬁnition 4 has equal true positive and false positive rates for all groups g ∈ G and satisﬁes equalized odds.
Proof.
By construction of the group-agnostic baseline (Deﬁnition 4), we have that
[Equation]
and therefore, for each group g , we can show that the true-positive rate of the group-agnostic baseline ˜
R TP g on g is equal to its average true-positive rate ˜ R TP [Equation]
[Equation]
Each group thus has the same true positive rate with the group-agnostic baseline.
Using similar reasoning, each group also has the same false positive rate.
By the deﬁnition of equalized odds, the group-agnostic baseline thus satisﬁes equalized odds.
A. R OBIN H OOD ORACLE
In this section, we deﬁne the Robin Hood oracle , which preferentially increases the worst-group accuracy through abstentions until it matches the other groups.
Like the group-agnostic baseline, we constrain the Robin Hood oracle to make the same number of correct and incorrect predictions as a given selective classiﬁer.
[Table]
[Equation]
[Equation] We formalize the deﬁnition of the Robin Hood oracle in Algorithm 2. The Robin Hood oracle makes predictions on the subset of examples from D that has the smallest discrepency between best-group and worst-group accuracies, while still matching the number of correct and incorrect predictions of a given selective classiﬁer. Since enumerating over all possible subsets of the test data D is intractable, we compute the accuracies of the Robin Hood oracle iteratively.
Starting from full coverage, we abstain on examples from lowest to highest conﬁdence.
Whenever we abstain on an incorrect example, we assume it comes from the current lowest accuracy group, and similarly whenever we abstain on a correctly classiﬁed example we assume it comes from the current highest accuracy group.
This reduces disparities to the maximum extent possible as the threshold τ increases.
B S UPPLEMENTAL EXPERIMENTS B. M ONTE -C ARLO DROPOUT
In the main text, we observed that SR selective classiﬁers monotonically improve average accuracy as coverage decreases, but exacerbate accuracy disparities across groups on all ﬁve datasets.
To demonstrate that these observations are not speciﬁc to SR selective classiﬁers, we now present our empirical results on another standard selective classiﬁcation method: Monte-Carlo (MC) dropout (Gal & Ghahramani, 2016; Geifman & El-Yaniv, 2017).
We ﬁnd that MC-dropout selective classiﬁers exhibit similar empirical trends as SR selective classiﬁers.
MC-dropout selective classiﬁers.
MC-dropout is an alternate way of assigning conﬁdences to points.
Taking a model with a dropout layer, the selective classiﬁer ﬁrst predicts ˆ y ( x ) simply by taking the model output without dropout.
To estimate the conﬁdence, it then samples n softmax probabilities corresponding to the label ˆ y ( x ) over the randomness of the dropout layer.
The conﬁdence is computed as ˆ c ( x ) = 1 /s , where s 2 is the variance of the sampled probabilities.
We implement MC-dropout by using the existing dropout layers for BERT and adding dropout to the ﬁnal fully-connected layer for ResNet and DenseNet, with a dropout probability 0 .
We present empirical results for n = 10 ; Gal & Ghahramani (2016) observed that this was sufﬁcient to produce good conﬁdence estimates.
Results.
The MC-dropout selective classiﬁers exhibit similar trends to those that observed in Section 4, demonstrating that the observed empirical trends are not speciﬁc to SR response; even though the average accuracy improves monotonically as coverage decreases across all ﬁve datasets, the worstgroup accuracy tends to decrease for CelebA and CivilComments, fails to increase signiﬁcantly for CheXpert and Waterbirds, and increases consistently but slowly for MultiNLI.
Comparing SR and MC-dropout selective classiﬁers, we observe that the MC-dropout selective classiﬁers performs slightly worse.
For example, we see a more prominent drop in worst-group accuracy for Waterbirds and much smaller improvements in worst-group accuracy for CheXpert.
[Figure]
[Figure]
We showed in Section 7 that SR selective classiﬁers trained with the group DRO objective successfully improve worst-group accuracies as coverage decreases, and perform comparably to the group-agnostic baseline.
We now present additional results for these
Accuracy-coverage curves.
We ﬁrst present the accuracy-coverage curves for all groups, along with the group coverage trends, in Figure 8.
Group accuracies tend to improve monotonically, in stark contrast with our results on standard (ERM) selective classiﬁers.
These trends hold generally across groups and datasets, with a few exceptions; accuracies drop on two groups in MultiNLI at roughly 20% average coverage, and accuracies improve slowly on two groups in CivilComments.
Below, we look into these anomalies and offer potential explanations.
For MultiNLI, we ﬁrst note that the drops are observed at very low group coverages—lower than 1%—at which point the accuracies are computed based on very few examples and thus are noisy.
In addition, much of the anomaly can be explained by label noise; we manually inspect theexamples from the above two groups with the highest softmax conﬁdences, and ﬁnd that 17 of them are labeled incorrectly.
The observations on CivilComments can also be potentially attributed to label noise.
Removing examples with high inter-annotator disagreement (with the fraction of toxic annotations between 0.5 and 0.6) yields an accuracy-coverage curve that ﬁts the broader empirical trends.
C E XPERIMENT DETAILS C. D ATASETS
CelebA. Models have been shown to latch onto spurious correlations between labels and demographic attributes such as race and gender (Buolamwini & Gebru, 2018;
Joshi et al., 2018), and we study this on the CelebA dataset (Liu et al., 2015).
Following Sagawa et al.
(2020), we consider the task of classifying hair color, which is spuriously correlated with the gender.
Concretely, inputs are celebrity face images, labels are hair color Y
= { blond , non-blond } , and spurious attributes are gender,
A = { male , female } , with blondness associated with being female.
Of the four groups, blond males are the smallest group, with only 1,387 examples out of 162,770 training examples, and they tend to be the worst group empirically.
We use the ofﬁcial train-val-split of the dataset.
Waterbirds.
Object recognition models are prone to using image backgrounds as a proxy for the label (Ribeiro et al., 2016; Xiao et al., 2020).
We study this on the Waterbirds dataset (Sagawa et al., 2020), constructed using images of birds from the Caltech-UCSD Birds dataset (Wah et al., 2011) placed on backgrounds from the Places dataset (Zhou et al., 2017).
The task is to classify a photograph of a bird as one of Y =
{ waterbird , landbird } , and the label is spuriously correlated with the background
A = { water background , land background } .
Of the four groups, waterbirds on land backgrounds make up the smallest group with only 56 examples out of 4,795 training examples, and they tend to be the worst group empirically.
We use the train-val-split provided by Sagawa et al.
(2020), and also follow their protocol for computing average metrics; to compute average accuracies and coverages, we ﬁrst compute the metrics for each group and obtain a weighted average according to group proportions in the training set, in order to account for the discrepancy in group proportions across the splits.
CheXpert-device.
Models can latch onto spurious correlations even in high-stakes applications such as medical imaging.
When models are trained to classify whether a patient has certain pathologies from chest X-rays, models have been shown to spuriously detect the presence of a support device, in particular a chest drain, instead (Oakden-Rayner et al., 2020).
We study this phenomenon in a modiﬁed version of the CheXpert dataset (Irvin et al., 2019), which we call CheXpert-device.
Concretely, the inputs are chest X-rays, labels are Y =
{ pleural effusion , no pleural effusion } , and spurious attributes indicate the the presence of a support device, A = { support device , no support device } .
We note that chest drain is one type of support device, and is used to treat suspected pleural effusion (Porcel, 2018).
CheXpert-device is a subsampled version of the full CheXpert dataset that manifests the spurious correlation more strongly.
To create CheXpert-device, we ﬁrst create a new 80/10/10 train/val/test split of examples from the publicly available CheXpert train and validation sets, randomly assigning patients to splits so that all X-rays of the same patient fall in the same split.
We then subsample the training set; in particular, we enforce that in 90% of examples, the label of support device matches the pleural effusion label.
Of the four groups, cases of pleural effusion without a support device make up the smallest group, with 5,467 examples out of 112,100 training examples, and they tend to be the worst group empirically.
To compute the average accuracies and coverages, we weight groups according to group proportions in the training set, similarly to Waterbirds.
Another complication with CheXpert is that some patients can have multiple X-rays from one visit.
Following Irvin et al.
(2019), we treat these images as separate training examples at training time, but output one prediction for each patient-study pair at evaluation time.
Concretely, we predict pleural effusion if the model detects the condition in any of the X-ray images belonging to the patient-study pair, as pathologies may only appear clearly in some X-rays.
CivilComments.
In toxicity comment detection, models have been shown to latch onto spurious correlations between the toxicity and mention of certain demographic groups (Park et al., 2018;
Dixon et al., 2018).
We study this in the CivilComments dataset (Borkan et al., 2019).
The task is to classify the toxicity of comments on online articles with labels Y =
{ toxic , non-toxic } .
As spurious attributes, we consider whether each comment mentions a Christian identity,
A = { mention of Christian identity , no mention of Christian identity } ; non-toxicity is associated with the mention of Christian identity, often resulting in high false negative rate on comments with such mentions.
Of the four groups, toxic comments with a mention of Christian identity make up the smallest group, with only 3,033 examples out of 359,950 training examples, and they tend to be the worst group empirically.
We use the ofﬁcial development and test sets from the associated Kaggle competition, 1 and further split the development set into a training set and a validation set by randomly splitting on articles and associating all comments with each article to either set.
The training, validation, and test set all comprise comments from disjoint sets of articles.
The original dataset also contains many additional examples that have toxicity annotations but not identity annotations; we do not use these in our experiments.
In the original CivilComments dataset, each comment is given a probabilistic labels for both the toxicity and the mention of a Christian identity, where a probabilistic label is the average of binary labels across annotators.
Following the associated Kaggle competition, we use binarized labels obtained by thresholding the probabilistic labels at 0.5.
MultiNLI.
Lastly, we consider natural langage inference (NLI), where the task is to predict whether a hypothesis is entailed, contradicted by, or neutral to an associated premise, Y = { entailed , contradictory , neutral } .
NLI models have been shown to exploit annotation artifacts, for example predicting contradictory whenever negation words such as never or nobody are present (Gururangan et al., 2018).
We study this on the MultiNLI dataset (Williams et al., 2018).
To annotate examples’ spurious attributes A = { negation words , no negation words } , we consider the following negation words following Gururangan et al.
(2018): “ nobody ”, “ no ”, “ never ”, and “ nothing ”.
We use the splits used in Sagawa et al.
(2020), whose training set includes 206,175 examples with 1, examples from the smallest group (entailment with negations).
The worst group for standard models tends to be neutral examples with negation words.
C. M ODELS
We train ResNet (He et al., 2016) for CelebA and Waterbirds (images), DenseNet (Huang et al., 2017) for CheXpert (X-rays), and BERT (Devlin et al., 2019) for CivilComments and MultiNLI (text).
For tasks studied in Sagawa et al.
(2020) (CelebA, Waterbirds, MultiNLI), we use the hyperparameters from Sagawa et al.
For others (CivilComments and CheXpert-device), we test the same number of hyperparameter sets for each of ERM and DRO, and report the best set below.
Across all image and X-ray tasks, inputs are downsampled to resolution 224 x 224.
CelebA. To train a model on CelebA, we initialize to pretrained ResNet-50.
For ERM we optimize with learning rate 1e-4, weight decay 1e-4, batch size 128, and train for 50 epochs.
For DRO we use learning rate 1e-5, weight decay 1e-1, and use generalization adjustment 1 (described in Sagawa et al.
The batch size is 128, and we train for 50 epochs.
Waterbirds.
For Waterbirds, as with CelebA, we use pretrained ResNet-50 as an initialization.
For
ERM we use learning rate 1e-3, weight decay 1e-4, batch size 128, and train for 300 epochs.
For
DRO we use learning rate 1e-5, weight decay 1, geneneralization adjustment 1, batch size 128, and train for 300 epochs.
CheXpert-device.
For CheXpert-device, we ﬁne-tune pretrained DenseNet-121 for three epochs.
For ERM we use learning rate 1e-3, no weight decay, batch size 16, and choose the model (out of the ﬁrst three epochs) with highest average accuracy (epoch 2).
For DRO, we use learning rate 1e4, weight decay 1e-1, and batch size 16, and choose the model with highest worst-group accuracy (epoch 1).
CivilComments.
To train a model for CivilComments, we ﬁne-tune bert-base-uncased using the implementation from Wolf et al.
For both ERM and DRO we use learning rate 1e-5, weight decay 1e-2, and batch size 16.
We train the ERM models and DRO models for three epochs (early stopping,) then choose the model with highest average accuracy for ERM (epoch 2), and highest worst-group accuracy for DRO (epoch 1).
MultiNLI.
For MultiNLI we again ﬁne-tune bert-base-uncased using the implementation from Wolf et al.
For ERM, we ﬁne-tune for three epochs with learning rate 2e-5, weight decay 0, and batch size 32.
For DRO, we also use learning rate 2e-5, and weight decay 0, and batch size 32, but use generalization adjustment 1.
For both ERM and DRO the model after the third epoch is best in terms of average accuracy for ERM and worst-group accuracy for DRO.
D P ROOFS : M ARGIN DISTRIBUTIONS AND ACCURACY - COVERAGE CURVES D. L EFT - LOG - CONCAVITY
Recall the deﬁnition of left-log-concave from Section 5:
Deﬁnition 1 (Left-log-concave distributions) .
A distribution is left-log-concave if its CDF is logconcave on ( −∞ , µ ] , where µ is the mean of the distribution.
We ﬁrst prove that a symmetric mixture of gaussians is left-log-concave, but not necessarily logconcave.
D.1. S YMMETRIC MIXTURE OF GAUSSIANS IS LEFT - LOG - CONCAVE
Lemma 1 (Symmetric mixture of two gaussians is left-log-concave) .
Consider a symmetric mixture of Gaussians with density f = 0 .
5 f µ + 0 .
5 f − µ , where f µ is the density of a N ( µ, σ 2 ) random variable and likewise f − µ is the density of a N ( − µ, σ 2 ) random variable.
Then the mixture is left-log-concave for all values of µ ∈ R , σ > 0 , but only log-concave if | µ | ≤ σ .
Proof.
Without loss of generality, we can take σ = 1 , since (left-)log-concavity is invariant to scaling, and also assume that µ is positive.
First consider the case where µ ≤ 1 .
Then the mixture is log-concave, and therefore left-log-concave (Cule et al., 2010).
Now, consider the case where µ > 1 .
Cule et al.
(2010) show the mixture is no longer log-concave.
However, we claim that it is still left-log-concave.
We start by studying the gradient of log f ,
[Equation]
We claim that f (cid:48) ( x ) f ( x ) has a local minimum at x = − a < 0 , and as it is an odd function, a corresponding local maximum at x = a > 0 .
To show this, we ﬁrst differentiate to obtain
[Equation]
Setting the derivative to 0 gives us the quadratic equation
[Equation]
⇐⇒
[Equation] ⇐⇒ [Equation] Since µ > 1 , there are two distinct roots of this quadratic, and two corresponding critical points of f (cid:48) /f .
Let v ( µ ) =
2 µ 2 − 1 + 2 µ (cid:112) µ 2 − 1 be the larger root.
Then v ( µ ) is a strictly increasing function for µ ≥ 1 , and since v (1) = 1 , we have that v ( µ ) > 1 for all µ > 1 .
Let x = − a satisfy exp(2 aµ )
= v ( µ ) .
Then we have that − a , the smaller of the critical points of f (cid:48) /f , is [Equation]
[Equation]
To show that f (cid:48) ( a ) /f ( a ) is a local minimum, we take the second derivative
[Equation]
which at x = − a gives
[Equation]
since v ( µ ) > 1 .
Since − a is the only critical point of f (cid:48) /f
that is less than
and it is a local minimum, f (cid:48) /f must be decreasing on ( −∞ , − a ] , which in turn implies that f , and therefore F , is log-concave on ( −∞ , − a ] .
It remains to show that F is also log-concave on [ − a, 0] .
We make use of two facts.
First, since − a is a local minimum and the only critical point less than 0, we have that
[Equation]
=
[Equation]
for all x ∈ [ − a, 0] .
Second, since f ( x ) and F ( x ) are non-negative for all x , f ( x ) /F
( x ) is also non-negative for all x .
Thus, for all x ∈ [ − a, 0]
[Equation]
and therefore F is also log-concave on [ − a, 0] .
Remark 1.
Note that if f is (left-)log-concave, then F is also (left-)log-concave (Bagnoli &
Bergstrom, 2005).
However, the reverse direction does not hold.
D. S YMMETRIC DISTRIBUTIONS
In this section we prove Proposition 1.
We ﬁrst prove the following helpful lemma:
Lemma 2 (Conditions for monotonicity of selective accuracy.) .
A F ( τ ) is monotone increasing in τ if and only if
[Equation]
for all τ ≥ 0 .
Conversely, A F ( τ ) is monotone decreasing in τ if and only if the above inequality is ﬂipped for all τ ≥ 0 .
Proof.
A F ( τ ) is monotone increasing in τ if and only if dA F dτ ≥ 0 for all τ ≥ 0 .
We obtain dA F dτ by differentiating A F and simplifying:
[Equation]
[Equation]
Since the denominator is always positive, we have that dA F dτ ≥
if and only if the numerator f ( − τ )[1 − F ( τ )] − f ( τ ) F
( − τ ) ≥ 0 , which in turn is equivalent to
[Equation]
as desired.
The case for monotone decreasing A F ( τ ) is analogous.
In the next two lemmas, we prove the necessary and sufﬁcient conditions for Proposition 1 respectively.
Lemma 3 (Left-log-concavity and symmetry imply monotonicity.) .
Let f be symmetric about µ and let F be left-log-concave.
If A F (0) ≥ 0 . 5 , then A F ( τ ) is monotone increasing.
Conversely, if
A F (0) ≤ 0 .
5 , then A F ( τ ) is monotone decreasing.
Proof.
Consider the case where A F (0) ≥ 0 .
5 ; the case where A F (0) ≤ 0 .
5 is analogous.
From
Lemma 2 and the symmetry of f , we have that A F ( τ ) is monotone increasing if (and only if)
[Equation]
holds for all τ ≥ 0 .
To show that this inequality holds for all τ ≥ 0 , we ﬁrst note that since A F (0) ≥ 0 . 5 , we have that
F (0) ≤ 0 . 5 , which together with the symmetry of F implies that µ ≥ 0 .
Thus, − τ ≤ 2 µ − τ for all τ ≥ 0 .
Now, if τ ≥ µ , then 2 µ − τ ≤ µ , so the desired inequality (56) follows from the log-concavity of F on ( −∞ , µ ] (Remark 1 of Bagnoli & Bergstrom (2005).)
If instead 0 ≤ τ ≤ µ , we have
[Equation]
[by log-concavity of F on ( −∞ , µ ] ] (57)
[Equation]
[since F ( τ ) ≤ 0 . 5 ] (58) and thus (56) also holds.
Lemma 4 (Monotonicity and symmetry imply left-log-concavity.) .
Let f be symmetric with mean 0, and let f µ be a translated version of f that has mean µ .
If A f µ is monotone increasing for all µ ≥ 0 , then f is left-log-concave.
Proof.
First consider µ ≥ 0 .
Since A f µ is monotone increasing for all µ ≥ 0 , from Lemma 2 and the symmetry of f µ ,
[Equation]
must hold for all τ ≥ 0 .
Since f µ ( x ) =
f ( x − µ ) by construction, we can equivalently write
[Equation]
[Equation]
which holds for all τ ≥ 0 as well, and by assumption, for all µ ≥ 0 as well.
By letting a = − µ − τ and b = µ − τ , we can equivalently write the above inequality as
[Equation]
for all a, b ∈ R where a ≤ b, a + b ≤ 0 .
By the deﬁnition of log-concavity, this means that F is log-concave on ( −∞ , 0] .
D.2. L EFT - LOG - CONCAVITY AND MONOTONICITY .
Proposition 1 (Left-log-concavity and monotonicity) .
Let F be the CDF of a symmetric distribution.
If F is left-log-concave, then A F ( τ ) is monotonically increasing in τ if A F (0) ≥ 1 /and monotonically decreasing otherwise.
Conversely, if A F d ( τ ) is monotonically increasing for all translations F d such that F d ( τ ) =
F ( τ − d ) for all τ and A F d (0) ≥ 1 / 2 , then F is left-log-concave.
Proof.
If F is left-log-concave, by Lemma 3 A F ( τ ) is monotonically increasing in τ if A F µ (0) ≥and monotonically decreasing otherwise.
Conversely, if A F is monotonically increasing for all translations F d such that F d ( τ ) =
F ( τ − d ) for all τ by Lemma 4, F is log-concave, completing the proof.
D. S KEWED D ISTRIBUTIONS
We now turn to how skew affects selective classiﬁcation.
Recall in Section 5 we deﬁned skewsymmmetric distributions as follows:
Deﬁnition 2.
A distribution with density f α,µ is skew-symmetric with skew α and center µ if
[Equation]
for all τ ∈ R , where h and G are the density and CDF of distributions that are symmetric about 0.
When α = 0 , f = h and there’s no skew.
Increasing α increases rightward skew, and decreasing α increases leftward skew.
Note that in general, the mean of f depends on α as well.
We will also consider the translated distribution:
[Equation]
The CDF of f α,µ is F α,µ and the CDF of f
α is F α .
We will use the following properties of these distributions:
Lemma 5 (Skew-symmetry about µ ) .
These properties hold when we ﬂip the skew from α to − α :
[Equation]
Proof.
[Equation]
Lemma 6 (Stochastic ordering with α ) .
Let α 1 ≤ α 2 .
Then F α 1 ,µ ≥ F α 2 ,µ .
Proof.
Since the ordering is invariant to translation, without loss of generality we can take µ =
First consider x ≤ 0 .
Then
[Equation]
Now consider x ≥ 0 .
We have
[Equation]
which reduces to the case where x ≤ 0 .
[This is Proposition 4 from Azzalini & Regoli (2012)]
Lemma 7 (Log gradient ordering by skew) .
For all α ≥ 0 and τ ≥ 0 ,
[Equation]
Proof.
Since the ordering is invariant to translation, without loss of generality we can take c = 0 .
We have:
[Equation]
To get the inequalities, note that when α ≥ 0 and t ≤ τ , we have α ≥ 0 , G ( αt ) /G
( ατ ) ≤since G is an increasing function.
Similarly, G ( − αt ) /G
( − ατ ) ≥ 1 .
Since h is non-negative, the inequalities then follow from the monotonicity of the integral.
We now prove the main results of this section.
D.3. A CCURACY IS MONOTONE WITH SKEW
Proposition 2 (Accuracy is monotone with skew) .
Let F α,µ be the CDF of a skew-symmetric distribution.
For all τ ≥ 0 , A F α,µ ( τ ) is monotonically increasing in α .
Proof.
We use the skew-symmetry of f α,µ to write the selective accuracy as
[Equation]
[Equation]
We see that A f α,µ ( τ ) is a monotone decreasing function of
[Equation]
From Lemma 6, the numerator decreases with increasing α while the denominator increases.
Thus, this fraction decreases, which in turn implies that A f α,µ ( τ ) is monotone increasing with α as desired.
D.3. S KEW IN THE SAME DIRECTION PRESERVES MONOTONICITY
Proposition 3 (Skew in the same direction preserves monotonicity) .
Let F α,µ be the CDF of a skewsymmetric distribution.
If µ ≥ 0 and A F 0 ,µ ( τ ) is monotonically increasing in τ , then A F α,µ ( τ ) is also monotonically increasing in τ for any α > 0 .
Similarly, if µ ≤ 0 and A F 0 ,µ ( τ ) is monotonically decreasing in τ , then A F α,µ ( τ ) is also monotonically decreasing in τ for any α < 0 .
Proof.
The idea is to use Lemma 7 to reduce the statement to the case where α = 0 , so that we can apply monotonicity.
First consider the case where c ≥ 0 .
We thus have
[Equation]
[Lemma 7]
[Equation] [Equation]
[deﬁnition of f ]
[Equation] [Equation]
[from monotonicity of A ]
[Equation] [Equation]
[symmetry of h ]
[Equation] [Equation]
[deﬁnition of f ] [Equation]
[Equation]
[Lemma 7] (95)
[Equation]
[Lemma 5] (96)
Applying Lemma 2 completes the proof.
The case where µ ≤ 0 is analogous.
D. M ONOTONE ODD TRANSFORMATIONS PRESERVE ACCURACY - COVERAGE CURVES .
We now show that our results are unchanged by strictly monotone and odd transformations to the
PDF.
Lemma 8 (Odd and strictly monotonically increasing transformations preserve selective accuracy) .
Let X be a real-valued random variable with CDF F X , T be a strictly monotonically increasing function, and deﬁne random variable Y = T ( X ) with CDF F Y .
Then, A F X ( τ ) =
A F Y ( T ( τ )) for each τ .
Proof.
For each τ , since T is a strictly monotonically increasing function, we apply the change of variables formula for transformations of univariate random variables to get the following:
[Equation]
We now solve for selective accuracy:
[Equation]
With Lemma 8, our results extend all random variables T ( X ) where X corresponds to a left-logconcave distribution and T is odd and strictly monotonically increasing.
In particular, X and T ( X ) have the same accuracy-coverage curves.
E P ROOFS : C OMPARISON TO GROUP - AGNOSTIC BASELINE
In this section, we present the proofs from Section 6, which outline conditions under which selective classiﬁers outperform the group-agnostic baseline.
E. D EFINITIONS
We ﬁrst write certain metrics on selective classiﬁers and their group-agnostic baseline in terms of margin distributions.
Deﬁnition 6.
Consider a margin distribution with CDF
[Equation]
where p g is the mixture weight and F g is the CDF for group g .
We write the fraction of predictions in corectly and incorrectly classiﬁed examples on group g and on average as follows:
[Equation]
We write the true-positive rate R TP ( τ ) and false-positive rate R FP ( τ ) for a given threshold τ as
[Equation]
Finally, we write the accuracy of the group-agnostic baseline on group g as
[Equation]
For convenience in our proofs below, we also deﬁne the fraction of each group g in correctly and incorrectly classiﬁed predictions at each threshold τ .
Deﬁnition 7.
We deﬁne the fraction of group g out of correctly and incorrectly classiﬁed predictions at threshold τ , CF g ( τ ) and IF g ( τ ) respectively, as follows:
[Equation]
E. G ENERAL NECESSARY CONDITIONS TO OUTPERFORM THE BASELINE
We now present the proofs for Proposition 4 and Corollary 1, starting with the supporting lemmas.
We ﬁrst write the accuracy of the group-agnostic baseline in terms of IF wg (0) and CF wg (0) .
Lemma 9.
[Equation]
Proof.
We have:
[Equation]
Lemma 10 (Bounding the derivative of 1 / CF wg ( τ ) at τ = 0 ) .
If A F others (0) ≥ A F wg (0) , d dτ
[Equation] [Equation]
Proof.
d dτ
[Equation]
CF
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
(cid:19)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
[Equation]
[Equation] [Equation]
Lemma 11.
If A F others (0) ≥ A F wg (0) > 0 . 5 , then
[Equation]
[Equation]
for some positive constant C .
Proof.
[Equation]
[ Lemma 10 ]
[Equation]
E.2. N ECESSARY CONDITION FOR OUTPERFORMING THE BASELINE
Proposition 4 (Necessary condition for outperforming the baseline) .
Assume that 1 / 2 < A F wg (0) <
A F others (0) < 1 and f wg (0) > 0 .
If ˜ A F wg ( τ ) ≤ A F wg ( τ ) for all τ ≥ 0 , then
[Equation]
[Equation]
Proof.
Recall that
[Equation] [Equation]
If A F wg [Equation] τ ) ≥ ˜
A F wg ( τ ) for all τ ≥ 0 , then
[Equation]
From Lemma 11,
[Equation]
for some positive constant C .
Combined,
[Equation]
E.2. O UTPERFORMING THE BASELINE REQUIRES SMALLER SCALING FOR LOG - CONCAVE DISTRIBUTIONS
Corollary 1 (Outperforming the baseline requires smaller scaling for log-concave distributions) .
Assume that 0 .
5 <
A F wg (0) <
A F others (0) < 1 , F wg is log-concave, and f others ( τ ) =
vf wg ( v ( τ − µ others ) + µ wg ) for all τ ∈ R , where v is a scaling factor.
If v > 1 , then ˜ A F wg ( τ ) >
A F wg ( τ ) for some τ ≥ 0 .
Proof.
By Proposition 4, if A F wg ( τ ) ≥ ˜
A F wg ( τ ) for all τ ≥ 0 , then:
[Equation]
143) and equivalently,
[Equation] [Equation]
From the deﬁnition of f others ,
[Equation]
[Equation]
Because A F others (0) > A F wg (0) , − µ bg v + µ wg < 0 .
Applying log-concavity yields v
[Equation]
[Equation]
Thus, when v > 1 , there exists some threshold τ ≥ 0 where ˜
A F wg ( τ ) >
A F wg ( τ ) .
E. T RANSLATED , LOG - CONCAVE DISTRIBUTIONS ALWAYS UNDERPERFORM THE BASELINE
We now present a proof of Proposition 5 Assume f
wg and f others are log concave and symmetric, group 1 has mean µ wg , group 2 has mean µ bg , and f others ( x ) =
f wg ( x − ( µ bg − µ wg )) , (the densities are translations of each other.)
For convenience, deﬁne d = µ bg − µ wg .
This implies:
[Equation]
We will ﬁrst show that wg is indeed the worst group at full coverage; i.e. wgacc (0) <
A F others (0) :
Lemma 12.
If f wg is a PDF and f others is obtained by translating f wg to the right (i.e. f others ( x )
=
f wg ( x − d ) for d > 0 ), and F wg and F others are their associated CDFs, A F wg (0) <
A F others (0) .
Proof.
We have:
[Equation]
[Equation]
[Equation]
We next show that CF wg ( τ ) is monotonically decreasing in τ :
Lemma 13.
If f wg , F wg , f others , F others , p , and CF wg are as described, CF wg is monotonically decreasing in τ .
Proof.
First, deﬁning CF wg in terms of F wg , we have:
[Equation]
Taking the derivative of CF wg
[Equation]
we have:
[Equation]
where c is a positive constant (since p and 1 − p are positive and all of the terms are squares.)
Thus, we can say that CF wg ( τ ) is monotonically decreasing for all τ if:
[Equation]
Now, since d > 0 , this follows from the log concavity of f wg .
Therefore, the fraction of correct examples that come from group 1 is a decreasing function of τ .
Next, we show that the IF wg ( τ ) is monotonically increasing in τ :
Lemma 14.
If f wg , F wg , f others , F others , p and IF wg are as described, IF wg is monotonically increasing in τ .
Proof.
First, we note that:
[Equation]
We will show that the derivative of IF wg ( τ ) with respect to τ is positive.
Doing so gives us:
[Equation]
where c is positive since it is the product of squared terms and (1 − p ) /p , which is also positive.
Thus, ddτ IF wg ( τ ) ≥ 0 is equivalent to:
[Equation]
Since f wg is log-concave and d is positive, the last inequality is true, so IF wg ( τ ) is monotonically increasing in τ , as desired.
Lastly, we show that Lemma 13 and Lemma 14 imply that the ratio IF wg ( τ ) CF wg ( τ ) is monotonically increasing in τ .
Lemma 15.
IF wg ( τ ) CF wg ( τ ) is monotonically increasing in τ
Proof.
We simply follow the quotient rule:
[Equation]
(171) where we note that CF wg ( τ ) , IF wg ( τ ) ≥ 0 , CF (cid:48) 1 ( τ ) < 0 from Lemma 13, and IF (cid:48) 1 ( τ ) > 0 from
Lemma 14.
We will now prove the main result of this section:
E.3. T RANSLATED , LOG - CONCAVE DISTRIBUTIONS ALWAYS UNDERPERFORM THE BASELINE
Proposition 5 (Translated, log-concave distributions always underperform the baseline) .
Assume
F wg and F others are log-concave and f others ( τ ) =
f wg ( τ − d ) for all τ ∈ R .
Then for all
[Equation]
Proof.
We’ll consider the case of underperforming the baseline: the case of overperforming the baseline is analogous.
Assume group 1 is the lower accuracy group (so d is positive.)
Using the deﬁnition of selective accuracy, we have:
[Equation]
[Equation]
Lemma 15]
[Equation]
Thus, group 1 (the worst group) underperforms the baseline, as desired.
F S IMULATIONS
To demonstrate that it is possible but challenging to outperform the group-agnostic baseline, we present simulation results on margin distributions that are mixtures of two Gaussians.
Following the setup from Section 6, we consider a best-group margin distribution N (1 , 1) and a worst-group margin distribution N ( µ, σ 2 ) varying the parameters µ, σ .
We set the fraction of mass from the worst group, p to be 0.5.
We evaluate whether selective classiﬁcation outperforms the group-agnostic baseline, plotting the results in Figure 9.
We observe that selective classiﬁcation outperforms the group-agnostic baseline under some parameter settings, notably when the necessary condition in
Proposition 4 is met and when the variance is smaller as implied by Corollary 1, but it underperforms the baseline under most parameter settings.
[Figure]
In addition to whether the worst group underperforms or outperforms the baseline, we study the magnitude of the difference in the worst-group accuracy with respect to the group-agnostic baseline through the same simulations (Figure 10).
Concretely, we compute the difference in worst-group accuracy with respect to the group-agnostic baseline, taking the worst-case difference across thresholds: max τ ˜
A F wg ( τ ) − A F wg ( τ ) .
We observe signiﬁcant disparities between the observed worstgroup accuracy and the group-agnostic baseline.
[Figure]
[Equation]
Lastly, we simulate the effects of varying the worst-group and other-group means while keeping the variance ﬁxed at σ 2 = 1 for both groups, following the same simulation protocol otherwise.
We similarly observe substantial gaps between the observed worst-group accuracy and the groupagnostic baseline (Figure 11)
[Figure]
[Figure]