P USHING
THE
G
ROUP R OBUSTNESS -A CCURACY T RADEOFF F RONTIER WITH I NTROSPECTIVE S ELF - PLAY
Anonymous authors
Paper under double-blind review
A BSTRACT
Standard empirical risk minimization (ERM) training can produce deep neural network (DNN) models that are accurate on average but underperform in underrepresented population subgroups, especially when there are imbalanced group distributions in the long-tailed training data.
Therefore, approaches that improve the accuracy - group robustness tradeoff frontier of a DNN model (i.e. improving worst-group accuracy without sacriﬁcing average accuracy, or vice versa) is of crucial importance.
Uncertainty-based active learning (AL) can potentially improve the frontier by preferentially sampling underrepresented subgroups to create a more balanced training dataset.
However, the quality of uncertainty estimates from modern DNNs tend to degrade in the presence of spurious correlations and dataset bias, compromising the effectiveness of AL for sampling tail groups.
In this work, we propose Introspective Self-play ( I SP ) , a simple approach to improve the uncertainty estimation of a deep neural network under dataset bias, by adding an auxiliary introspection task requiring a model to predict the bias for each data point in addition to the label.
We show that I SP provably improves the bias-awareness of the model representation and the resulting uncertainty estimates.
On two realworld tabular and language tasks, I SP serves as a simple “plug-in” for AL model training, consistently improving both the tail-group sampling rate and the ﬁnal accuracy-fairness trade-off frontier of popular AL methods.
I NTRODUCTION
Modern deep neural network (DNN) models are commonly trained on large-scale datasets (Deng et al., 2009; Raffel et al., 2020).
These datasets often exhibit an imbalanced long-tail distribution with many small population subgroups, reﬂecting the nature of the physical and social processes generating the data distribution (Zhu et al., 2014; Feldman & Zhang, 2020).
This imbalance in training data distribution, i.e., dataset bias , prevents deep neural network (DNN) models from generalizing equitably to the underrepresented population groups (Hasnain-Wynia et al., 2007).
Accuracy-Group Robustness Frontier:
In response, the existing bias mitigation literature has focused on improving training procedures under a ﬁxed and imbalanced training dataset, striving to balance performance between model accuracy and fairness (e.g., the average-case v.s. worst-group performance)
(Agarwal et al., 2018; Martinez et al., 2020; 2021).
Formally, this goal corresponds to identifying an optimal model f ∈ F that attains the Pareto efﬁciency frontier of the accuracy-group robustness trade-off (e.g., see Figure 1), so that under the same training data D = { y i , x i } ni = 1 , we cannot ﬁnd another model f (cid:48) ∈ F that outperforms f in both accuracy and worst-group performance.
In the literature, this accuracy-group robustness frontier is often characterized by a trade-off objective (Martinez et al., 2021):
[Equation]
where R acc and R robust are risk functions for a model’s accuracy and group robustness (modeled here-in as worst-group accuracy), and λ > 0 a trade-off parameter.
Then, f λ cannot be outperformed by any other f (cid:48) at the same trade-off level λ .
The entire frontier under a dataset D can then be characterized by ﬁnding f λ that minimizes the robustness-accuracy objective (1) at every trade-off level λ , and tracing out its ( R acc , R robust ) performances (Figure 1).
Goal: However, the limited size of the tail-group examples restricts the DNN model’s worst-group performance, leading to a compromised accuracy-group robustness frontier (Zhao & Gordon, 2019; Dutta et al., 2020), and thus we ask: Under a ﬁxed learning algorithm, can we meaningfully push the model’s accuracy-group robustness frontier by improving the training data distribution using active learning?
That is, denoting by D α , n = { ( y i , x i ) } ni
= 1 a training dataset with K subgroups and the group size distribution α =
[ α 1 ,..., α K ] , we study whether a model’s accuracy-group robustness performance F λ can be improved by rebalancing the group distribution of the training data D α , n , i.e., we seek to optimize an outer problem:
[Equation]
where ∆ K is the simplex of all possible group distributions (Rolf et al., 2021).
Our key observation is that given a sampling model with well-calibrated uncertainty (i.e., the model uncertainty is well-correlated with generalization error), active learning (AL) can preferentially acquire tail-group examples from unlabelled data without needing group annotations , and add them to the training data to reach a more balanced data distribution (Branchaud-Charron et al., 2021). Appendix A.5 discusses the connection of Equation (2) with fairness.
Challenges with DNN Uncertainty under Bias: However, recent work suggests that a DNN model’s uncertainty estimate is less trustworthy under spurious correlations and distributional shift, potentially compromising the AL performance under dataset bias.
For example, Ovadia et al.
(2019) show that a DNN’s expected calibration error increases as the testing data distribution deviates from the training data distribution, and Ming et al.
(2022) show that a DNN’s ability in detecting out-ofdistribution examples is signiﬁcantly hampered by spurious patterns.
Looking deeper, Liu et al.
(2022); Van Amersfoort et al.
(2020) suggest that this failure mode in DNN uncertainty can be caused by an issue in representation learning known as feature collapse , where the DNN over-focuses on correlational features that help to distinguish between output classes on the training data, but ignore the non-predictive but semantically meaningful input features that are important for uncertainty quantiﬁcation (Figure 2).
In this work, we show that this failure mode can be provably mitigated by a training procedure we term introspective training (Section 2).
Brieﬂy, introspective training adds an auxiliary introspection task to model training, asking the model to predict whether an example belongs to an underrepresented group.
It comes with a guarantee in injecting bias-awareness into model representation (Proposition 1), encouraging it to learn diverse hidden features that distinguish the minority-group examples from the majority, even if these features are not correlated with the training labels.
Hence it can serve as a simple “plug-in” to the training procedure of any active learning method, leading to improved uncertainty quality for tail groups (Figure 2).
Contributions:
In summary, our contributions are:
• We introduce Introspective Self-play (I SP ) , a simple training approach to improve a DNN model’s uncertainty quality for underrepresented groups (Section 2).
Using group annotations from the training data, I SP conducts introspective training to provably improve a DNN’s representation and uncertainty quality for the tail groups.
When group annotations are not available, I SP can be combined with a cross-validation-based procedure that uses a noise-bias-variance decomposition of the model’s generalization error (Domingos, 2000).
• Theoretical Analysis.
We theoretically analyze the optimization problem in Equation (2) under a group-speciﬁc learning rate model (Rolf et al., 2021)
(Section 3).
Our result elucidates the dependence of the group distribution α in the model’s best-attainable accuracy-group robustness frontier F λ .
In particular, it conﬁrms the theoretical necessity of up-sampling the underrepresented groups for obtaining the optimal accuracy-group robustness frontier, and reveals that underrepresentation is in fact caused by an interplay of the subgroup’s learning difﬁculty and its prevalence in the population.
• Empirical Effectiveness.
Under two challenging real-world tasks (census income prediction and toxic comment detection), we empirically validate the effectiveness of I SP in improving the performance of AL with a DNN model under dataset bias (Section 4).
For both classic and stateof-the-art uncertainty-based AL methods, ISP improves tail-group sampling rate, meaningfully pushing the accuracy-group robustness frontier of the ﬁnal model.
Appendix D surveys related work.
Notation and Problem Setup.
We consider a dataset D where each labeled example { x i , y i } is associated with a discrete group label g i ∈ G = { 1 ,..., | G |} .
We denote D =
P
( y , x , g ) the joint distribution of the label, feature and groups, so that D can be understood as a size- n set of i.i.d.
samples from D 1 .
We denote the prevalence of each group as γ g = E
( y , x , g ) ∼ D ( 1 G = g ) and associate dataset bias with the imbalance in group distribution P ( G ) =
[ γ 1 ,..., γ | G | ] (Rolf et al., 2021).
In the applications we consider, there exists a subset of underrepresented groups B ⊂ G which are not sufﬁciently represented in the population distribution D so that γ g (cid:28) 1 | G | for g ∈ B (Sagawa et al., 2019; 2020).
We denote L ( y , ˆ y ) as a loss function from the Bregman divergence family, and F the hypothesis space of predictors f : X (cid:55)→ Y .
We require the model class F to be sufﬁciently expressive so it can model the Bayes-optimal predictor ˜ y ( x ) = argmin (cid:48)
y E y ∼ P
( y | x ) ( L ( y , y (cid:48) )) ).
We also assume F has a certain degree of smoothness, so that the model f ∈ F cannot arbitrarily overﬁt to the noisy labels in the training set.
M ETHOD
In this section, we introduce Introspective Self-play ( I SP ) , a simple training approach to improve model quality in representation learning and uncertainty quantiﬁcation under dataset bias.
Brieﬂy,
I SP performs introspective training by adding a underrepresention prediction head to the model and training it to distinguish whether an example ( y i , x i , g i ) is from the set of underrepresented groups B (Section 2.1).
When the underrepresentation label b i =
I ( g i ∈ B ) is not available, I SP estimates it based on a cross-validation-based procedure we term cross-validated self-play (Section 2.2).
As we will show, I SP carries a guarantee for the model’s representation learning and uncertainty estimation quality under dataset bias (Proposition 1).
2. I NTROSPECTIVE T RAINING
We consider models of the form p ( y | x ) = σ (cid:0) f y ( x ) (cid:1)
=
σ (cid:0) β (cid:62)
y h
( x ) (cid:1) , where h : X → R D is a Ddimensional embedding function,
β y ∈ R D the output weights, and σ ( · ) the activation function.
Given model f y = β (cid:62) y h , introspective training adds a bias head f b = β (cid:62) b h to the model, so it becomes a multi-task architecture f
= ( f y , f b ) with shared embedding:
[Equation]
where ( f y , f b [Equation] (cid:0)
[Equation]
(cid:62) y [Equation] Given examples D = { x i , y i , g i } ni = 1 , we generate the underrepresentation labels as b i
=
I ( g i ∈ B ) and train the model with the target and underrepresentation labels ( y i , b i ) by minimizing a standard multi-task learning objective:
[Equation]
where L is the standard loss function for the task, and L b is the cross-entropy loss.
As a result, given training examples { x i } ni = 1 , introspective training not only trains the model to predict the outcome y i , but also instructs it to recognize its potential bias b i by predicting whether x i is from an underrepresented group.
Despite its simplicity, introspective training has a signiﬁcant impact on the model’s representation learning that is particularly important for quantifying uncertainty when dataset exhibits signiﬁcant bias.
illustrates this on a binary classiﬁcation task under severe group imbalance (Sagawa et al., 2020), where we compare two dense ResNet ensemble models trained using the introspection objective v.s. the empirical risk minimization (ERM) objective (i.e., only use L ( y i , f y ( x i )) in Equation (4)), respectively.
Comparing ﬁgures 2a and 2e, we observe that the decision boundaries for the predicted label are very similar between introspective training and ERM.
However, the predictive variance (obtained via a
Gaussian process (GP) layer (Liu et al., 2022)) exhibits sizable differences.
In particular, the variance estimates for introspective training are uniformly high outside of the two clouds of underrepresented groups in the data.
However, for ERM, the model conﬁdence is high along the decision boundary, even in the unseen regions without training data.
This is due to the fact that when training with ERM, the representation collapses in the direction that is not correlated with training label (i e., parallel to decision boundary) and does not retrain any input information regarding the underrepresented groups in its representation (ﬁg. 2g).
However, with introspective training, the representations indeed are morphed to reﬂect the differences between the underrepresented examples and the majority group (as can be seen in ﬁgures ﬁg. 2g vs ﬁg. 2c), helping the model to better distinguish them in the representation space, and hence lead to improved uncertainty estimate in the neighborhood of underrepresented examples.
Appendix E.1 contains further description.
[Figure] [Table]
Formally, introspective training induces the below guarantee on the model’s bias-awareness in its hidden representation and uncertainty estimates:
Proposition 1 (Introspective Training induces Bias-awareness) .
Denote o b ( x ) = p ( x | b = 1 ) / p ( x | b = 0 )
the odds for x belongs to the underrepresented group B .
For a well-trained model f = ( f y , f b ) that minimizes the introspective training objective (4), so that p ( b = 1 | x ) = σ ( f b ( x )) , we then have:
(I) (Bias-aware Hidden Representation)
The hidden representation h ( x ) is aware of the likelihood ratio of whether an example x belongs to the underrepresented group b
= I ( g ∈ B ) , i e. p ( x | b = 1 ) / p ( x | b = 0 ) , such that:
[Equation]
(II) (Bias-aware Embedding Distance)
For two examples ( x 1 , x 2 ) , the embedding distance
|| h ( x 1 ) − h ( x 2 )
|| 2 is lower bounded by (up to a scaling constant) the odds ratio of whether xbelongs to the underrepresented groups versus that for x 2 :
[Equation]
such that the distance between a pair of minority and majority examples ( x 1 , x 2 ) is large due to the high values of the log odds ratio
The proof is in Appendix G. Part (I) provides a consistency guarantee for the hidden representation h ( · ) ’s ability in expressing the likelihood of whether an example x belongs to the underrepresented group B , i.e., bias awareness .
The form of (5) is similar to the representation learning guarantee in the noise contrastive learning literature, as it shares the same underlying principle of encouraging feature diversity and disentanglement via contrastive comparison between groups (Gutmann &
Hyv¨arinen, 2010; Sugiyama et al., 2010; Hyvarinen & Morioka, 2016) .
Part (II) is a corollary of (5) and provides a direct guarantee on the model’s learned embedding distance.
It states that under introspective training, the model cannot discard important input features that are not predictive of the target label to a degree that it collapsed the representation of majority and minority examples together (i.e., making || h ( x 1 ) − h ( x 2 )
||
2 excessively small for two examples ( x 1 , x 2 ) from the majority and minority group, respectively)
, creating difﬁculty for identifying underrepresented groups in the feature space with uncertainty-based active learning.
Empirically, we ﬁnd the beneﬁt of introspective training extends to other uncertainty-based active learning signals as well (e.g., margin and ensemble diversity, see Section 4.2).
Appendix C contains further discussion.
2. E STIMATING UNKNOWN BIAS VIA C ROSS - VALIDATED S ELF - PLAY
So far, we introduced introspective training in the setting where the group annotation g
i is available on training data, so that the underrepresentation label b i
= I ( g i ∈ B ) can be directly computed.
In this section, we consider how to estimate the underrepresentation label b i when it is absent, so that I SP can be applied to the setting where group annotations g i is too expensive to obtain.
A popular practice in the literature is to estimate dataset bias as the predictive error of a single (biased) model.
That is, given a trained model f D , prior work (Clark et al., 2019;
He et al., 2019; Nam et al., 2020; Sanh et al., 2020; Liu et al., 2021) estimates the underrepresentation label as the observed error L ( y i , f D ( x i )) .
To better understand this estimator for the generalization error of the underrepresented groups, we perform a noise-bias-variance decomposition (Domingos (2000)) of the model error L ( y , f D ) , which reveals, in the expectation of the random draws of the dataset D ∼ D :
[Equation]
where ˜ y ( x ) = argmin y (cid:48)
E y ∼ P ( y | x ) [ L ( y , y (cid:48) )] is the (Bayes) optimal predictor and ¯ f ( x ) =
argmin
f E D
[ L ( f , f D ( x ))] is the ‘ensemble’ predictor of the single models { f D } D ∼ D trained from random data draws (see Appendix A.2 for a review).
From (7), we see that for the purpose of estimating generalization error due to dataset bias, the naive estimator ˆ b
=
L ( y , f D ) based on single-model error suffers from two issues: (1) ˆ b 0 conﬂates noise (typically arising from label noise or feature ambiguity) with the dataset bias signal we wish to capture, potentially leading to compromised quality in real datasets (Lahoti et al., 2020; Li et al., 2022).
(2) As ˆ b 0 is calculated from a single model, its estimate of the variance term (an important component of generalization error (Yang & Xu, 2020)) is often not stable.
This is exacerbated when ˆ b 0 is computed from the training error, since model variance tends to be severely underestimated by DNNs (Liu et al., 2021).
This observation motivates us to propose cross-validated self-play , a simple method to estimate a model’s generalization gap.
Brieﬂy, given training data D divided into K splits, we train a bootstrap ensemble of K models { f k } Kk = 1 with ERM training, where each f k sees a fraction of the training data (see Appendix Fig. 5).
As a result, for each ( x i , y i ) , there exists a collection of in-sample predictions { f in , k (cid:48)
( x i ) } K in k (cid:48)
= 1 trained on data splits containing ( x i , y i ) , and a collection of out-of-sample predictions { f out , k ( x i ) } K out k = 1 trained on data splits not containing ( x i , y i ) .
Then, the self-play estimator of the model’s generalization gap is
[Equation]
where ¯ f in is the ensemble prediction based on in-sample predictors f in , k (cid:48) , the expectation E k is taken with respect to the out-of-sample predictions, and we are estimating the Bayes optimal predictor ˜
y using the in-domain prediction ¯ f in (since the model class F is subject to suitable regularization, [Equation] the ¯ f in ’s do not arbitrarily overﬁt the noisy labels).
Note that under the well-speciﬁed setting, ¯ f in converges asymptotically to the Bayes optimal predictor ˜ y as n → ∞ .
In practice, we can ensure the validity of L ( y i , ¯ f in ( x i )) as a valid estimator of noise by applying early stopping with a stability criterion on ¯ f out ( x i ) (Li et al., 2020; Song et al., 2020).
Compared to the standard alternatives in the literature (e.g., single-model error L ( y , f D ) ), the self-play estimator ˆ b i has the appealing property of controlling noise (by using ¯ f in ) while better estimating variance (by using expectations over ¯ f out , k ), thereby constituting a more informative signal for the underrepresented groups under dataset bias, label noise and feature ambiguity.
Appendix B.2 contains additional comments, and Appendix H develops a data-dependent bound for group detection performance.
Method Summary: Introspective Self-play.
Combining the self-play bias estimation and introspective training together, we arrive at Introspective Self-play (I SP ), a simple two-stage method that provably improves the representation quality and uncertainty estimates of a DNN for underrepresented population groups.
Figure 3 illustrates the full I SP procedure.
I SP ﬁrst (optionally) estimates underrepresentation labels using cross-validated self-play if the group annotation is not available, and then conducts introspective training to train the model to recognize its own bias while learning to predict the target label.
For the unlabelled data to be sampled, the resulting model generates (1) predictive probability p ( y | x ) , (2) uncertainty estimates ˆ v ( x ) and (3) predicted probability for underrepresentation p ( b | x )
= σ ( f b ( x )) , offering a rich collection of active learning signals for downstream applications.
[Figure]
(I SP ) model: In the ﬁrst stage, we ﬁrst train a bootstrap ensemble of DNN’s on cross-validation splits of the data (Figure 5) At the end of Stage I training, we can use the in-sample and out-of-sample predictions of the ensemble to estimate the underrepresentation label
b i of each training data point x i by computing the generalization gap (Equation (8)).
Then in Stage II, we use the underrepresentation labels b i to train the actual sampling model via introspective training (Equation (4)), and use it to generate the active learning signals (e.g., predictive probability p ( y | x ) , predictive variance v ( x ) and predicted underrepresentation p ( b | x ) ) for the unlabelled set.
T HEORETICAL A NALYSIS : I MPROVING A CCURACY -G ROUP R OBUSTNESS F RONTIER BY O PTIMIZING T RAINING D ATA D ISTRIBUTION
Denote by ∆ K the K -simplex so that ∀ α
=
[ α 1 ,..., α K ] ∈ ∆ K , ∑ k α k = 1 and ∀ k , α k > 0.
Let D α , n denote a dataset of size n sampled with group allocations α ∈ ∆ | G | , i.e., α g represents the fraction of the dataset sampled from group g ∈ G .
Our goal is to ﬁnd the optimal allocation that minimizes a weighted combination of the population risk and the worst-case risk over all subgroups.
Deﬁnition 1 (Risk and Fairness Risk) .
Given f ∈ F , let R ( f | α , n ) :
= ∑ g ∈ G γ g R ( f | G = g ) be the risk where γ ∈ ∆ | G | represents the true subgroup proportions in the underlying data distribution, and R fair ( f ) be the fairness risk which is the worst-case risk among the groups g ∈ G : R fair ( f ) :
= max g R ( f | G = g ) .
Let ˆ R and ˆ R fair denote the empirical estimates of R , R fair estimated based on the ﬁnite dataset D α , n : ˆ R α , n , g ( f ) = ∑ ( x , y ) ∈ D α ,
[Equation]
where D α , n , g is the subset of D α , n with group label g .
Let ˆ f denote the classiﬁer obtained as per γ -weighted empirical risk minimization (ERM)
ˆ f α , n = argmin f ∈ F ˆ R α , n ( f ) .
Deﬁnition 2 (Accuracy-Fairness Frontier Risk) .
For non-negative weighting coefﬁcients ω ∈ [ 0 , 1 ] , the accuracy-fairness frontier risk is deﬁned as:
[Equation]
where the expectation is taken wrt the randomness inherent in ˆ f as it is estimated based on a random dataset D α , n drawn iid from the underlying data distribution.
As we sweep over ω going from 0 to 1 and ﬁnd the best classiﬁer/allocation for each value, we would trace an accuracy fairness frontier as in Figure 1.
Analyzing optimal allocation between subgroups.
We theoretically analyze the optimization problem α ∗ = argmin α ∈ ∆ | G | (cid:104) F ω ( α , n ) (cid:105) .
We build on the theoretical models for group-speciﬁc scaling laws of generalization error introduced in previous work.
(Chen et al., 2018; Rolf et al., 2021).
We assume that the group-speciﬁc true risk decays with the size of the dataset at a rate of the form:
[Equation]
for some p > 0 , q > 0 , c g > 0 , τ , δ > 0.
The ﬁrst term represents the impact of the group representation, the second the aggregate impact of the dataset size, and the third term a constant offset (the irreducible risk for this group).
In particular, c g represents the “difﬁculty“ of learning group g , as the larger c g is, the higher the risk is for group g for any given allocation α .
Theorem 1 (Optimal Group-size Allocation for Accuracy-Fairness Frontier Risk (Informal)) .
The optimal allocation α (cid:63) is of the form α (cid:63) g ∝ ( γ g c g θ g ) 1 p + 1 where θ g ≥ ω represents an up-sampling factor for group g. Let g 1 , g 2 ,... be the subgroups sorted in ascending order according to the value γ g c − 1 / p g , which represents the subgroup representation normalized by the difﬁculty of learning the subgroup.
Then, it holds that there exists an integer k > 0 such that θ g i = ω for i
=
k ,... | G | , θ g i > ω for i = 1 ,..., k. Thus, the subgroups with low normalized representation are systematically up-sampled in the optimal allocation.
Based on the above theorem, we can the set of underrepresented groups as B = { g 1 , g 2 ,...
g k } .
, validating the idea that underrepresented groups can be formally deﬁned.
While the result above is derived under a simpliﬁed theoretical model, it validates the intuition behind ISP: indeed, ISP attempts to infer whether datapoints belong to B and systematically up-samples them via active learning to increase the overall representation and achieve an allocation closer to the theoretically optimal allocation deﬁned above.
We present a full theorem statement and proof in Appendix I and a discussion in Appendix C.
E XPERIMENTS
We ﬁrst demonstrate that for each task, I SP meaningfully improves the tail-group sampling rate and the accuracy-group robustness performance of state-of-the-art AL methods (Section 4.1), and then conduct detailed ablation analysis to understand how the choice of different underrepresentation labels impacts (1) the ﬁnal model’s accuracy-group robustness performance when trained on data collected by different AL methods; and (2) the sampling performance of different active sampling signals (Section 4.2).
Datasets.
We consider two challenging real-world datasets: Census Income (Le Quy et al., 2022) that contains 32,561 training records from the 1994 U.S. census survey.
The task is to predict whether an individual’s income is > 50K, and the tail groups are female or non-white individuals with high income.
We also consider Toxicity Detection (Borkan et al., 2019) that contains 405,130 online comments from the CivilComments platform.
The goal is to predict whether a given comment is toxic, and the tail groups are demographic identities × label class (male, female, White, Black, LGBTQ,
Muslim, Christian, other religion) × (toxic, non-toxic) following Koh et al.
[Table]
AL Baselines and Method Variations.
For all tasks, we use a 10-member DNN ensemble f = { f k } 10 k = 1 as the AL model, and replace their last layers with a random-feature GP layer (Liu et al., 2022) in order to compute posterior variance (see Appendix A.4).
We compare the impact of different training methods in two settings depending on whether the group identity label will be annotated in the labelled set (they are never available in the unlabelled set).
As shown in Table 1, when group label is available, we compare I
SP -identity (i.e., I SP with group identity as training label
b i =
I ( g i ∈ B ) ) to a group-speciﬁc reweighting (RWT) baseline (Idrissi et al., 2022) 5 .
When the group label is not known, we consider I SP -Gap
using the self-play -estimated generalization gap ˆ b i =
E k
[ L ( ¯ f in ( x i ) , f out , k ( x i ))]
as the representation label (i.e., Equation (8)), and compare it to an ensemble of Just Train Twice (JTT) which uses the ensemble training error ˆ b i = E k
[ L ( y i , f in , k
( x i ))] to determine the training set.
We also compare to an ERM baseline which trains the AL models with a routine ERM objective, but uses error for the reweighted training of the ﬁnal model.
We consider other method combinations in the ablation study.
Active Learning Protocol and Final Accuracy-Group Robustness Evaluation.
For active learning, we start with a randomly sampled initial dataset, and conduct active learning for 8 rounds until reaching roughly half of the training set (to ensure there’s sufﬁcient variation in the sampled data between methods).
To evaluate the accuracy-group robustness performance of the ﬁnal model, given a dataset collected by an AL method, we train a ﬁnal model using the standard re-weighting objective ∑ ( x , y )
(cid:54)∈ ˆ B L ce ( y , f ( x ))
+
λ ∑ ( x , y ) ∈
ˆ B L ce ( y , f ( x )) where ˆ B is the set of underrepresented examples identiﬁed by the underrepresentation label, i.e., ( x i , y i )
∈
ˆ B if ˆ b i > t .
We vary the thresholds t and the up-weight coefﬁcient λ over a 2D grid to get a collection of model accuracy-group robustness performances (i.e., accuracy v.s. worst-group accuracy), and use them to identify the Pareto frontier deﬁned by this combination of data and reweighting signal.
Appendix E.2 describes further experimental details.
[Table]
[Table]
[Table]
4. M AIN R ESULTS
[Figure]
W o r s t A cc (Random)RWTISP-Identity ERMJTTISP-Gap Table 2 shows sampling performance and the ﬁnal-model group robustness-accuracy performance of each AL model training method (described in Table 1), and Figure 4 visualizes the full accuracy-fairness frontier of the ﬁnal models (trained on the data and re-weighting signals provided by that method).
Our main conclusions are: (1) Effectiveness of I SP training : Compared to non-I SP baselines, we ﬁnd I SP consistently improves a AL model’s active learning (measured by tail-group sampling rate) and accuracy-group robustness performance (measured by combined accuracy, which is deﬁned as (accuracy + worst-group accuracy)/2).
This advantage is seen in both settings where the group label is available or unavailable.
In particular, in Figure 4, the ﬁnal model from I SP -Gap (pink dashed line, trained on actively sampled data and using estimated underrepresentation label for ﬁnal-model re-weighted training) almost dominates Random (blue solid line, trained on randomly sampled data and using true group label for ﬁnal model training) despite not having access to true group label in the ﬁnal reweighted training, highlighting the importance of the data distribution in the model’s accuracy-group robustness performance (i.e., Equation (2)).
Label Quality Matters :
Comparing the variants of I SP (Identity v.s. Gap) in Table 2, we see a clear impact of the quality of introspection signal to the performance of the AL model.
For example, for active learning performance, we see that the sampling rate I SP -Identity is signiﬁcantly better than I SP -Error.
However, for toxicity detection where the group label suffers an under-coverage issue (i.e., the group deﬁnition excludes potentially identity-mention comments where raters disagree, see Data section of Appendix E.2), we see that I SP -Error
in fact strongly outperforms I SP
-Identity
in accuracy-group robustness performance.
This validates the observation from previous literature on the failure mode of bias-mitigation methods when the available group annotation does not cover all sources of dataset bias, and speaks to the importance of high quality estimation methods that can detect underpresentation in the presence of unknown sources of bias (Zhou et al., 2021).
4. A BLATION A NALYSIS
In the main results above, we have (1) used the same underrepresentation label for both the AL-model introspective training and the ﬁnal-model reweighted training, and (2) focused on the most effective active sampling signal under each task.
In this section, we conduct ablations by decoupling the signal combinations along these two axes.
Impact of Data Distribution and Reweighting Signal to Accuracy-Group Robustness Frontier
First, we investigate the joint impact of data distribution and reweighting signal on the ﬁnal models’ accuracy-group robustness performance.
We train the ﬁnal model under data collected by different AL policy (Random v.s Margin v.s. Group Identity, etc), and perform reweighted training using different underrepresentation labels (Error v.s. Gap v.s. Group Identity) and compare to an ERM baseline without reweighted training (Section 4.2).
As shown, holding the choice of reweighted signal constant and compare across data distributions (i.e., comparing across columns within each row), we observe that the data distribution in general has a non-trivial impact on the ﬁnal model’s accuracygroup robustness performance.
Speciﬁcally, under appropriate sampling signal, data collected by I SP -Gap (which has no access to true group identity label) can lead to model performance that is competitive with data collected by I
SP -Identity
(e.g., the third v.s. fourth columns).
Comparing across reweighting signals within each dataset (i.e., compare across rows within each column), we see that all underrepresentation labels brings a meaningful improvement over the ERM baseline, with Group Identity bringing the most signiﬁcant improvement when it is of high quality (i.e., Census Income), and Gap bringing the most improvement when group annotation is imperfect (i.e., Toxicity Detection).
[Figure]
ALMethod,CensusIncome
[Table]
Impact of Underrepresentation Label on Different Sampling Signals.
Finally, we evaluate the choice of introspection signal on the sampling performance of a introspective-trained AL-model, under different types of sampling signals (Table 4).
As this evaluation is computationally expensive (requiring multiple active learning experiments for all underrepresentation label v s. sampling signal combinations), here we focus on the Census Income task.
As shown, we observe the introspective training brings a consistent performance boost across different types of sampling signals (esp.
when using Group Identity), highlighting the appeal of introspective training as a “plug-in” method that meaningfully boost the performance of a wide range of active learning methods.
Interestingly, we also observe the “Predicted Underrep.”
(i.e., the underrepresentation prediction in p ( b | x ) in
Figure 3) is exceptionally effective when the group identity is available (tail sampling rate > 0.95) but underperforms classic active learning signals otherwise, cautioning the proper use of p ( b | x ) as a sampling signal depending on the availability of group labels.
[Table]
C ONCLUSION
In this work, we introduced Introspective Self-play (I SP ), a novel training approach to improve a
DNN’s representation learning and uncertainty quantiﬁcation quality under dataset bias.
I SP uses a multi-task introspective training approach to encourage DNNs to learn diverse and bias-aware features for the underrepresented groups.
When underrepresented group identities are not available,
I SP bootstraps them using a novel cross-validated self-play procedure that disentangles dataset bias from irreducible noise while also generating a more stable estimate of variance.
Theoretical analysis reveals that the optimal per-group up-sampling factors are in fact determined by an interplay of the original group rates and the group-speciﬁc scaling factors.
Models trained on data acquired by I SP generally surpass recent competitive baselines such as RWT and JTT on the accuracy-group robustness frontier.
Overall, our results are a concrete step to a recent but critical effort in the community to build a more holistic perspective of model performance, addressing key challenges of robustness and equity.
Future work could more thoroughly investigate the relation of the noise, bias, and variance components of generalization error to underrepresentation examples, as well as the effectiveness of introspective training under different settings including training epochs, model regularization, batch size.
For example, specialized training objectives such as generalized cross entropy (GCE) (Zhang & Sabuncu, 2018) or focal loss (FL)
(Lin et al., 2017) may improve the statistical power of different components of generalization error in detecting the underrepresented groups.
On the other hand, batch size may impact the quality of the learned representation under introspective training, in a manner analogous to that of the contrastive training (Chen et al., 2020a).
Our framework could also be extended to other settings such as semi-supervised learning, or incorporate other kinds of introspection signals, such as those from the interpetability or differential privacy literature.
Ethical Statements: This work proposes novel approach to encourage DNN models to learn diverse, bias-aware features in model representation, for the purpose of improving DNN model
’s uncertainty quantiﬁcation ability under dataset bias.
Our method encourages DNNs to better identify underrepresented data subgroups during data collection, and eventually achieve a more balanced performance between model accuracy and group robustness by training on a more well-balanced dataset.
We evaluated the method on two already publicly available dataset and uses existing metrics in the literature.
No new data is collected as part of the current study.
The technique we developed in this work is simple and general-purpose, with potentially broad appeal to various downstream applications (e g., recommendations, NLP, etc).
However, two limitations highlighted by our work is that (1) when the group annotation information is imperfect, building a bias-mitigation procedure around such annotation may lead to suboptimal performance, and (2) in the presence of noisy labels, a noisy estimate of under-representation may also compromise the performance of the procedure.
Therefore, practitioner should take caution in rigorously evaluate the effectiveness of the procedure in their application, taking effort to carefully evaluate the estimation result of underrepresentation labels to ensure proper application of the technique without incurring unexpected consequences.
A A DDITIONAL B ACKGROUND A. R ECAP : N OTATION AND P ROBLEM S ETUP .
Dataset with subgroups: We consider a dataset D where each example { x i , y i } ( x i ∈
X denotes the features and y i ∈ Y the label) is associated with a discrete group label g i ∈ G = { 1 ,..., | G |} .
Joint data distribution: We denote D =
P
( y , x , g ) as the joint distribution of the label, feature and groups, so that D above can be understood as a size- n set of i.i.d.
samples from D .
Notice that this formulation implies a ﬂexible noise model P ( y | x , g ) that depends on ( x , g ) .
It also implies a ﬂexible group-speciﬁc distribution P ( y , x
| g ) , where the joint distribution of ( y , x ) varies by group.
Note however that we do assume that the group label does not have additional predictive power beyond appropriate representation of the features, i.e., there exists a representation function h ∗ such that for z
= h ∗ ( x ) , we have P ( y | z , g ) = P ( y | z ) , i.e., the semantic, label-relevant features are invariant across subgroups (Arjovsky et al., 2019; Creager et al., 2021;
Shui et al., 2022)
Subgroup prevelance: We denote the prevalence of each group as γ g = E
( y , x , g ) ∼ D ( 1 G = g ) .
As a result, the notion of dataset bias is reﬂected as the imbalance in group distribution P ( G ) =
[ γ 1 ,..., γ | G | ] (Rolf et al., 2021).
In the applications we consider, it is often feasible to identify a subset of underrepresented groups B ⊂ G which are not sufﬁciently represented in the population distribution
D and have γ g (cid:28) 1 | G | (Sagawa et al., 2019; 2020).
To this end, we also specify D ∗ =
P ( y , x | g ) P ∗ ( g ) an optimal distribution, where P ∗ ( g ) is an ideal group distribution (i.e., uniform such that P ∗ ( g ) =
γ ∗ g = 1 | G | ) so that all groups have sufﬁcient representation in the data.
Loss function: We assume a loss function
L ( y , ˆ y ) , that denotes the loss incurred when the predicted label is ˆ y while the actual label is y .
Hypothesis space: We consider learning a predictor from a hypothesis space F of functions f : X (cid:55)→ Y .
We assume that the hypothesis space is well-speciﬁed, i.e., that it contains the Bayesoptimal predictor ˜
y :
X → Y :
[Equation]
We require the model class F to come with certain degree of smoothness, so that the model f ∈ F cannot arbitrarily overﬁt to the noisy labels during the course of training.
In the case of overparameterized models, this usually implies F is subject to certain regularization that is appropriate for the model class (e.g., early stopping for SGD-trained neural networks)
(Li et al., 2020).
A. D ISENTANGLING MODEL ERROR UNDER NOISE AND BIAS
Given a dataset D ∼ D and a loss function L , we consider learning the prediction function f D = argmin f ∈ F L ( f , y | D ) , where L ( y , f | D ) =
∑ { x i , y i }∈ D L ( y i , f ( x i )) .
Following the previous work (Pfau, 2013), we denote the ensemble predictor ¯ f = argmin f ∈ F E D ∼ D
( L ( f D , f )) over ensemble members
f D ’s, where each f D is trained on a random draw of training dataset D ∼ D , and ˜ y
( x ) = argmin y (cid:48)
∈
Y E y ∼ P ( y | x ) ( L ( y , y (cid:48) ))
the (Bayes) optimal predictor.
For test example { y i , x i } , we can decompose the predictive error of a trained model f D
( x ) using a generalized bias-variance decomposition for Bregman divergence:
Proposition A.1 (Noise-Bias-Variance Decomposition under Bregman divergence (Domingos, 2000;
Pfau, 2013)) .
Given a loss function of the Bregman divergence family, for a test example { y , x } the expected prediction loss L ( y , f D ( x )) of an empirical predictor f D can be decomposed as:
[Equation]
Given a ﬁxed data distribution D , the ﬁrst term E D (cid:2) L ( y , f ∗ ( x )) (cid:3) quantiﬁes the irreducible noise that is due to the stochasticity in the noisy observation y .
The third term E D (cid:2) L ( ¯ f ( x ) , f D ( x )) (cid:3) quantiﬁes the variance in the prediction, which can be due to variations in the ﬁnite-size data D , the stochasticity in the randomized learning algorithm F × D
→
f D , or the randomness in the initialization of an overparameterized model (Adlam & Pennington, 2020).
Finally, the middle term L ( ˜ y ( x ) , ¯ f ( x )) quantiﬁes the bias between ˜ y
( x ) (i.e., the “true label”) and the ensemble predictor ¯ f learned from the empirical data D ∼ D .
It is inherent to the speciﬁcation of the model class and cannot be eliminated by ensembling, e.g., it can be caused by model misspeciﬁcation, missing features, or regularization.
To make the idea concrete, consider a simple example where we ﬁt a ridge regression model f ( x i )
=
β (cid:62) x i to the Gaussian observation data y i
= θ (cid:62) x i + ε , ε ∼ N ( 0 , σ 2 ) under an imbalanced experiment design, where we have | G | treatment groups and n g observations in each group.
Here, x i =
[ 1 g i = 1 ,..., 1 g i = | G | ] is a | G |× 1 one-hot indicator of the membership of g i for each group in G , and θ =
[ θ 1 ,..., θ | G | ] is the true effect for each group.
Then, under ridge regression, the noise-bias-variance decomposition for group g is E D ( L ( y , f D ))
=
σ 2 + ( λθ g ) 2 ( n g + λ ) 2 + σ 2 n g ( n g + λ ) 2 , where the regularization parameter λ modulates a trade-off between the bias and variance terms.
A. F URTHER D ECOMPOSITION
Further Uncertainty Decomposition for Probabilistic Models As an aside, when the predictive model f D is probabilistic (e.g., the model generates a posterior predictive distribution P ( f | D ) rather than a point estimate f ), the variance in Equation (10) is further decomposed as:
[Equation]
(11) where µ D ( x ) =
E f ∼ P ( f | D )
[ f ( x )] is the posterior mean, and v D ( x ) = E f [Equation] L
[Equation] x [Equation] f
[Equation]
x [Equation] is the posterior variance of each ensemble member. As shown, comparing to an ensemble of deterministic models, the ensemble of probabilistic models provides additional ﬂexibility in quantifying model uncertainty via the extra term of expected posterior variance.
Further Bias Decomposition for Minority Groups For the examples x coming from the underrepresented groups with γ g (cid:28) 1 | G | , the bias term can be further decomposed into:
[Equation]
where ¯ f ∗ = argmin f ∈ F E D ∗ ∼ D ∗ ( L ( f D ∗ , f )) is the optimal ensemble predictor based on size- n datasets
D ∗ sampled from the optimal distribution D ∗ where all groups have equal representation.
Here,
L ( ˜ y
( x ) , ¯ f ∗ ( x )) is the bias inherent to the model class and cannot be eliminated by ensembling.
It can be caused by model misspeciﬁcation, missing features, or regularization.
On the other hand, E ( ¯ f ∗ ( x ) , ¯ f ( x )) =
L ( ˜ y
( x ) , ¯ f ( x )) − L ( ˜ y ( x ) , ¯ f ∗ ( x )) indicates the “excess bias” for the underrepresented groups caused by the imbalance in the group distribution P ( G ) in the data-generation distribution D .
To make the idea concrete, consider the ridge regression example from the previous section, where the noise-bias-variance decomposition for group g is E D
( L ( y , f D ))
=
σ 2 + ( λθ g ) 2 ( n g + λ ) 2 + σ 2 n g ( n g + λ ) 2 , with the regularization parameter λ modulating a trade-off between the bias and variance terms (Appendix F).
Consequently, for an underrepresented group with small size γ g (cid:28) 1 | G | , its predictive bias ( λθ g ) ∗ 2 ( n g + λ ) 2 is exacerbated due to lacking sufﬁcient statistical information to counter the regularization bias, incuring an excessive bias of E ( ¯ f ∗ ( x ) , ¯ f ( x ))
≈ λθ g n γ ∗ g γ g ( γ ∗ g − γ g ) when compared to an optimal ensemble predictor ¯ f ∗ trained from a perfectly balanced size- n datasets with γ ∗ g = 1 / | G | .
A. M ODERN UNCERTAINTY ESTIMATION TECHNIQUES IN DEEP LEARNING
For a deep classiﬁer p ( x ) = σ ( f ( x )) with logit function f ( x ) =
β (cid:62) h
( x ) and h ( x ) ∈ R M the last-layer hidden embeddings, the modern deep uncertainty methods quantiﬁes model uncertainty by enabling it to generate random samples from a predictive distribution.
That is, for a model trained on data
D = { ( x i , y i ) } ni = 1 , given a test data point x test , the model can return a size- K sample: [Equation] , D
[Equation]
For example, in Monte Carlo Dropout (Gal & Ghahramani, 2016), the samples is generated by perturbing the dropout mask in the learned predictive function f ( · ) =
β (cid:62) h ( · ) ’s embedding function h ( · ) , while in Deep Ensemble (Lakshminarayanan et al., 2017), the sample comes directly from the multiple parallel-trained ensemble members.
Finally, in a neural Gaussian process model (Wilson et al., 2016; Liu et al., 2022; van Amersfoort et al., 2021), the samples are generated from a Gaussian process model using the hidden embedding function h ( x ) as the input.
For example, for classiﬁcation problems, the predictive variance of the Gaussian process model v ( x test ) =
Var ( f | x test , D ) can be expressed as (Williams & Rasmussen (2006), Chapter 3):
[Equation]
where V n × n is a ﬁxed matrix computed from training data, and k ( x test )
=
[ k ( x test , x 1 ) ,..., k ( x test , x n )] is a vector of kernel distances between x test and the training examples { x i } ni = 1 .
The kernel function k is commonly deﬁned to be a monotonic function of the hidden embedding distance, e.g., k ( x test , x i ) =
exp ( −|| h ( x test ) − h ( x i ) || 22 ) for the RBF kernel.
As a result, the predictive uncertainty for a data points x i is determined by the distance between x test from the training data { x i } ni = 1 .
Consequently, a DNN model’s quality in representation learning has non-trivial impact on its uncertainty performance.
Although ﬁrst mentioned in the context of neural Gaussian process, this connection between the quality of representation learning and the quality of uncertainty quantiﬁcation also holds for state-ofthe-art techniques such as Deep Ensemble, as model averaging cannot eliminate the systematic errors in representation learning and consequently the issue in uncertainty quantiﬁcation (for example, see
Figure 6 and the corresponding ensemble uncertainty surface Figure 2f).
Neural Gaussian Process Ensemble
In this work, to comprehensively investigate the effect of different uncertainty techniques, we should to use a Deep Ensemble of neural Gaussian process as our canonical model.
That is, we parallel train K neural Gaussian process models { f k } Kk = 1 .
Then, given a test data point x test , each ensemble member will return a predictive distribution with means { µ k ( x ) } Kk = 1 and variances { v k ( x ) } Kk = 1 .
Then, we can generate model prediction as E k [ µ k ( x )] , and quantify uncertainty in one of the two ways:
[Equation]
where Var k , E k are empirical means and variances over the ensemble members.
As shown, they correspond to the two components of the total model variance under squared error introduced in A.3.
We investigate the effectiveness of these two uncertainty signals in the experiments.
A. C ONNECTION BETWEEN GROUP ROBUSTNESS AND FAIRNESS
The notion of group robustness (e.g., worst-group accuracy) we considered in this work corresponds to the notion of minimax fairness or Rawlsian max-min fairness (Lahoti et al., 2020; Martinez et al., 2020; 2021; Diana et al., 2021). The philosophical foundation of this notion has been well-established (Rawls, 2001; 2004). There have been notable works developed for both analyzing accuracy-fairness tradeoff under this notion (e.g., Martinez et al.
(2020; 2021)), and for improving fairness performance without explicit annotation (i.e., ARL, (Lahoti et al., 2020)).
It is also identiﬁed as the original objective of the well-known Invariant Risk Minimization (IRM) work for invariant learning (Arjovsky et al.
(2019), Section 2).
To this end, under a sampling model with well-calibrated uncertainty, the active learning procedure as discussed in introduction is expected to carry beneﬁt for model fairness as well.
Speciﬁcally, in the context of fairness-aware learning, this means a sampling model with calibrated uncertainty preferentially samples the under-represented protected group, leading to a sampled training set with more balanced representation among population subgroups, and consequently reducing the between-group generalization gap in the trained model and promoting its fairness properties.
B M ETHOD S UMMARY B. A LGORITHM
Inputs: Training data D train =
{ y i , x i } ni = 1 ; (Optional) Group annotation G train =
{ g i } ni = 1 ;
Unlabelled data D pool = { x j } n (cid:48)
j = 1 .
Output: Predicted probability { p ( y | x j ) } n (cid:48)
j = 1 ; Bias probability { p ( b | x j ) } n (cid:48)
j = 1 ; Predictive variance { v ( x j ) } n (cid:48)
j = 1 .
(cid:46) Stage I: Label Generation if G train (cid:54) = /
then B train =
{ b i =
I ( g i ∈ B ) } ; (cid:46) Make underrepresentation label using group annotation g i .
else ˆ B train =
Sel fPlayBiasEstimation ( D train ) .
(cid:46) Estimate underrepresentation label using Algorithm(cid:46)
Stage II: Introspective Training
Train ˆ f on D train with multi-task introspective objective L (( y i , b i ) , x i ) .
(cid:46) Equation (4)
Evaluate ˆ f on x j ∈ D pool to generate sampling signals { p ( y | x j ) , p ( b | x j ) , v ( x j ) } n (cid:48)
j = 1 .
(cid:46) Equation (3)
Inputs: Training data D train =
{ y i , x i } ni = 1 .
Output: Estimate underrepresentation labels ˆ B train .
Train K -fold cross-validated ensemble { f k } Kk = 1 with D train .
Compute in-sample and out-of-sample ensemble predictions { f in , k
( x i ) } K in k = 1 , { f out , k ( x i ) } K out k = 1 for all x i ∈ D train .
Estimate underrepresentation labels as ˆ B train =
{ b i =
E k
[ L ( ¯ f in ( x i ) , f out , k ( x i ))] } ni
= 1 .
(cid:46) (Equation (8))
B. E STIMATING G ENERALIZATION G AP USING C ROSS - VALIDATED E NSEMBLE
Practical Comments. Note that due to its cross validation nature, the self-play bias estimator ˆ b i estimates the generalization error of a weaker model (i.e., trained on a smaller data size n cv < n ).
This is in fact consistent with the practice in the previous debasing literature, where the main model is trained on the error signals from weaker and more biased models (Clark et al., 2019; He et al., 2019; Nam et al., 2020).
Further, in the context of SGD-trained neural networks, it is important to properly estimate the ¯ f in ( x i ) so it does not overﬁt to the training label, via early stopping (Li et al., 2020; Liu et al., 2020).
This is easy to do in the context of cross validation: during training, we collect the estimated bias ˆ b i , t across the training epochs t = 1 ,..., T , and select the optimal stopping point t as the ﬁrst time the out-of-sample error E (cid:2)
L ( y i , f out , k
( x i ))
(cid:3) stablizes.
In practice, we specify the early-stopping criteria as when the running average (within a window T (cid:48)
= 5) of the cross validation error ﬁrst stablizes below a threshold ε .
This is to prevent the situation where the errors for some hard-to-learn examples keep oscillating throughout training and never stabilize.
Is perfect group identiﬁcation necessary? In fact, it is not necessary to perfectly identify all the subgroup examples to improve the ﬁnal model’s tradeoff frontier. For example, when there exists hard-to-learn majority-group examples and also easy-to-learn minority-group examples (also known as ”benign bias” examples in the debiasing literature (Nam et al., 2020)), the active sampling budget is better spent in sampling some of the hard-to-learn majority examples over the easy-to-learn minority examples, so that the ﬁnal tradeoff frontier is meaningfully improved both in the directions of subgroup performance and of overall accuracy. To this end, including some hard-to-learn majority examples (detected by the loss-based procedure) into introspective training help us to achieve that.
This is exactly the case in the toxicity detection experiments (Section 4).
As shown in Table 2, comparing between ISP-Identity v.s. ISP-Gap (trained on true group label v.s. cross-validation estimated label), the ISP-Gap attains a signiﬁcantly stronger accuracy-fairness performance while sampling less minority examples.
This is likely caused by the cross-validation estimator’s ability in capturing challenging, non-identity-related examples instead of the easy-to-learn and identity-related examples, which resulted in less sampling redundancy in the minority group (i.e., the identity-related comments), and led to improved tradeoff frontier for the ﬁnal model.
B. H YPERPARAMETERS AND C OMPUTATIONAL C OMPLEXITY
Hyper-parameters
The full I SP procedure contains 3 hyper-parameters: The (optional) crossvalidated self-play in Stage I contains all three hyper-parameters: (1) the number of ensemble models
K and (2) the number of examples n cv to train each model.
Both are standard to the bootstrap ensemble procedure, and we set them to K = 10 and n cv = n / K in this work to ensure the total computation complexity is comparable to training a single model on the full dataset.
(3) the early stopping criteria ε for noise estimation (as discussed in the previous section B.2), we set it heuristically to ε = 0 .
1 in this work after visual inspection of the validation learning curves.
The introspective training in Stage
II does not contain additional hyperparameter other than the standard supervised learning parameters (e.g., learning rate and training epochs).
We set these parameters based on a standard supervised learning hyperparameter sweep based on the full data.
Computation Complexity When the group annotation is available, the computation complexity of the I SP procedure (i.e., Stage II only) should be equivalent to the standard ERM procedure.
On the other hand, the computational complexity of the full I SP procedure (Stage I + II) should be comparable to that of a standard two-stage debiasing method that trains multiple single models on the full dataset (Utama et al., 2020b; Liu et al., 2021;
Yaghoobzadeh et al., 2021;
Nam et al., 2020;
Creager et al., 2021; Kim et al., 2022).
More speciﬁcally, The complexity of the cross-validated ensemble approach (i.e., the cross-validated self-play) is identical to that of a single model training on full data (in the big O sense).
This is because for cross-validated ensembles, each ensemble member model is trained on different subsets of the data.
Therefore, if the complexity for a single model is T , then the complexity of the cross-validated ensemble is K ∗ p ∗ T , where p < 1 is the percentage of sub-sampled training examples for each ensemble member, and K is the ensemble size, notice that K ∗ p ∗ T
= O ( T ) since p and K are both constants that don’t scale with data size.
In practice, we ﬁnd setting p = 1 / K (i.e., train K model on K non-overlap data splits) works rather well and has the beneﬁt of improving the cross-validation error estimate by reducing cross-split correlation (Blum et al., 1999).
This leads the complexity of cross-validated ensembles to be equal to T , the complexity of training a single model on the full data. Furthermore, since ensemble training is embarrassingly parallel, the complexity can also be massively reduced by leveraging multi-core machines to perform parallel training. C A DDITIONAL T HEORY D ISCUSSION C. P ROPOSITION 1: WHEN WILL IT BREAK DOWN ?
It is important to consider the situations where the guarantee in Proposition 1 can break down.
First note that since Proposition 1 is a consistency result, it expects reasonable number of training examples in the neighborhood of x to ensure the convergence of p ( x
| b ) (Equation (5)).
In practice, the convergence is not guaranteed for the tail groups that have extraordinarily low example count in the dataset, since they can be statistically indistinguishable from the label noise under a ﬁnite-size dataset.
To this end, Figure 2 provides us an empirical view of the model’s bias estimation performance for the low-prevalence groups.
As shown, when the tail-group examples are well-separated from the majority groups (i.e., the bias learning task is easy), a very small amount of examples is sufﬁcient to lead to reasonable underrepresentation prediction performance (Figure 2h, only 2 examples per tail group).
This suggests that the actual number of examples required to attain reasonable performance depends on the difﬁculty of the bias learning task.
For example, for a challenging tail group that shares much surface-form similarity with the majority groups (e.g., in Proposition 1, the tail examples are extremely close or even nested within the majority groups), more examples are expected to obtain a reasonable performance in underrepresentation prediction.
Also note that the learning quality of the model representation h ( · ) depends on the quality of the underrepresentation label b .
In particular, the noise in the underrepresentation labels can negatively impact introspective training by diminishing the log likelihood ratio | logp ( x | b = 1 ) − logp ( x | b = 0 ) | in (5), leading the β (cid:62) b h ( x ) to converge to a target of lower magnitude, hampering its ability in bias-awareness .
This issue becomes relevant when the underrepresentation labels need to be estimated from data, where the vulnerability of fairness procedures to label noise has been well-noted in the literature (e.g., Lahoti et al.
, highlighting the importance of label noise control when estimating the membership of the underrepresented groups.
For example, we note that some of the now popular method (e.g., JTT (Liu et al., 2021)) estimates the membership of the underpresented groups using generalization error, which may be susceptible to label noise (c.f., the noise-bias-variance decomposition in Appendix A.2).
We develop bias estimation procedures that controls for label noise in Section 2.2.
C. O N U NDERREPRESENTED GROUP B IN T HEOREM
In the main paper (section 3), we characterized the optimal allocation under assumptions on decay rates of group-speciﬁc risk.
We give some further analysis and interpretation of this result here.
To simplify things, we assume that there are only G = 1 , 2 groups so that the optimal allocation is characterized by a single number α ∈ [ 0 , 1 ] representing the allocation to group 1 (the other group allocation would be 1 − α ).
We further assume that γ 1 = γ 2 = 12 and that c 2 = tc 1 for some t > 0,
From the proof of theorem Theorem 1 from section I, we know that
[Equation]
so that we can set θ 1 = θ , θ 2 = 2 − θ .
Then, we have that
[Equation]
Further, from the proof of the theorem, we can infer that the optimal θ is
[Equation]
so that the optimal allocation is
[Equation]
This shows that the optimal allocation to group 1 decays as a function of t , the difﬁculty of learning group 2 relative to group 1.
Further, it shows that this decay is more stark for smaller values of ω that weight the fairness risk stronger than the population level risk.
Further, group 1 belongs to the underrepresented group B if ω ≥ 1 + t 1 p , showing that the chances that group 1 belongs to the underrepresented group increase as ω increases (there is greater emphasis on the fairness risk) or as t decreases (group 1 becomes harder to learn relative to group 2).
D R ELATED W ORK D. S UPERVISED AND SEMI - SUPERVISED LEARNING UNDER DATASET BIAS
In recent years, there has been signiﬁcant interest in studying robust generalization for long-tail population subgroups under dataset bias.
The literature is vast and encompasses topics including fairness, debiasing, long-tail recognition, spurious correlation, distributional (i.e., domain or subpopulation) shift, etc.
In the following, we focus on notable and recent work that is highly relevant to the I SP approach, and refer to works such as Caton & Haas (2020); Mehrabi et al.
; Hort et al.
for an exhaustive survey.
Majority of the fairness and debiasing work focuses on the supervised learning setting, where the model only have access to a ﬁxed and imbalanced dataset.
Among them, the earlier work operated under the assumption that the source of dataset bias is completely known, and the group annotation is available for every training example.
Then these group information is use to train a robust model by modify components of the training pipeline (e.g., training objective, regularization method, or composition of training data).
For example, Levy et al.
; Sagawa et al.
; Zhang et al.
(2020) proposes minimizing the worst-group loss via Distributionally Robust Optimization (DRO), which is shown to be equivalent to ERM training on a well-curated training set in some settings (Słowik & Bottou, 2022); IRM (Arjovsky et al., 2019) proposes learning a classiﬁer that is simultaneously optimal for all groups by learning a invariant representation. Teney et al.
Idrissi et al.
; Byrd & Lipton (2019); Xu et al.
(2020) studies the effect of group-weighted loss in model’s fairness-accuracy performance, and REx (Krueger et al., 2021) minimizes a combination of group-balanced and worst-case loss.
Further, the recent literature has also seen sophisticated neural-network loss that modiﬁes gradient for the tail-group examples.
For example, LDAM (Cao et al., 2019) proposes to modify group-speciﬁc logits by an offset factor that is associated with group size, and equalization loss (Tan et al., 2020) uses a instance-speciﬁc mask to suppress the “discouraging gradients” from majority groups to the rare groups.
On the regularization front, the examples include IRM v1 (Arjovsky et al., 2019) that proposed to approximate the original IRM objective through gradient penalty.
Heteroskedastic Adaptive Regularization (HAR) (Cao et al., 2020) imposes Lipschitz regularization in the neighborhood of tail-group examples.
There also exists a large collection of work imposing other types of fairness constraints.
Finally, the third class of methods modiﬁes the composition of the training data by enriching the number of obsevations in the tail groups, this includes Sagawa et al.
; Idrissi et al.
(2022) that study the impact of resampling to the worst-group performance, and Goel et al.
(2020) that generates synthetic examples for the minority groups.
In the setting where the group information is available, our work proposes a novel approach (introspective training) that has both a theoretical guarantee and is empirically competitive than reweighted training.
On the other hand, there exist a separate stream of work that allows for partial group annotation, i.e., the types of bias underlying a dataset is still completely known, but the group annotation is only available for a subset of the data.
Most work along this direction employs semi-supervised learning techniques (e.g, conﬁdence-threshold-based pseudo labeling), with examples include Spread
Spurious Attribute (SSA) (Nam et al., 2020), BARACK (Sohoni et al., 2021) and Fair-PG (Jung et al., 2022).
This setting can be considered as a special case of I SP where we use group information as the underrepresentation label to train the p ( b | x ) predictor.
However, our goal is distinct that we study the efﬁcacy of this signal as an active learning policy, and also investigate its extension in the case where the label information is completely unobserved in the experiments Section 4.2.
D. E STIMATING DATASET BIAS FOR MODEL DEBIASING
In the situation where the source of dataset bias is not known and the group annotation is unavailable, several techniques has been proposed to estimate proxy bias labels for the downstream debiasing procedures.
These methods roughly fall into three camps: clustering, adversarial search, and using the generalization error from a biased model.
For clustering, GEORGE (Sohoni et al., 2020) and CNC (Zhang et al., 2021) proposed estimating group memberships of examples based on clustering the last hidden-layer output.
For adversarial search, REPAIR (Li & Vasconcelos, 2019), ARL (Lahoti et al., 2020), EIIL (Creager et al., 2021),
BPF (Martinez et al., 2021), FAIR (Petrovi´c et al., 2022), Prepend (Tosh & Hsu, 2022) infers the group assignments by ﬁnding the worst-case group assignments that maximize certain objective function. For example, Environment Inference for Invariant Learning (EIIL) (Creager et al., 2021) infer the group membership by maximizing the gradient-based regularizer from IRM v1.
Estimating bias label using the error from a biased model is by far the most popular technique.
These include forgettable examples (Yaghoobzadeh et al., 2021), Product of Experts (PoE)
(Clark et al., 2019; Sanh et al., 2020), DRiFt (He et al., 2019) and Conﬁdence Regularization (CR) (Utama et al., 2020b;a) that uses errors from a separate class of weak models that is different from the main model;
Neutralization for Fairness (RNF) (Du et al., 2021) and Learning from Failure (LfF) that trains a bias-ampliﬁed model of the same architecture using generalized cross entropy (GCE); and Just Train
Twice (JTT) that directly uses the error from a standard model trained from cross entropy loss.
Notably, there also exists several work that estimates bias label using ensemble techniques, this includes Gradient Alignment (GA)
(Zhao et al., 2021) that identiﬁes the tail-group (i.e., bias-conﬂicting) examples based on the agreement between two sets of epoch ensembles, Bias-conﬂicting Detection (BCD) (Lee et al., 2022) that uses the testing error of a biased deep ensemble trained with GCE, and
Learning with Biased Committee (LWBC) uses the testing error of a bootstrap ensemble.
To this end, our work proposes a novel self-play estimator (Equation (8)) that uses bootstrap ensembles to estimate the generalization gap due to dataset bias.
self-play estimator has the appealing property of better controlling for label noise while more stably estimating model variance, addressing two weaknesses of the naive predictive error estimator used in the previous works.
D. R EPRESENTATION LEARNING UNDER DATASET BIAS
Originated from the fairness literature, the earlier work in debiased representation learning has focused on identifying a representation that improves a model’s fairness properties, as measured by notions such as demographic parity (DP), equalized odds (EO), or sufﬁciency (Arjovsky et al., 2019; Arjovsky, 2020; Creager et al., 2021;
Shui et al., 2022) , This is commonly achieved by striving to learn a representation that is invariant with respect to the group information. Such methods are commonly characterized as in-processing methods in the existing survey of the fairness literature (Caton & Haas, 2020; Hort et al., 2022), and was categorized into classes of approaches including adversarial training (Beutel et al., 2017;
Kim et al., 2019; Ragonesi et al., 2021; Zhu et al., 2021; Madras et al., 2018), regularization (Bahng et al., 2020; Tartaglione et al., 2021; Arjovsky et al., 2019), contrastive learning (Shen et al., 2021; Park et al., 2022; Cheng et al., 2020) and its conditional variants (Gupta et al., 2021;
Tsai et al., 2021a;b; Chi et al., 2022), or explicit solutions to a bi-level optimization problem (Shui et al., 2022). (Please see Caton & Haas (2020); Hort et al. for a complete survey).
However, some later works questions the necessity and the sufﬁciency of such approaches.
For example, some work shows that careful training of the output head along is sufﬁcient to yield improved performance in fairness and bias mitigation (Kang et al., 2019; Du et al., 2021; Kirichenko et al., 2022), and Cherepanova et al.
(2021) shows that models with fair feature representations do not necessarily yield fair model behavior.
At the meantime, a separate stream of work explores the opposite direction of encouraging the model to learn diverse hidden features.
For example, Locatello et al.
(2019b;a) establish a connection between the notion of feature disentanglement and fairness criteria, showing that feature disentanglement techniques may be a useful property to encourage model fairness when sensitive variables are not observed.
However, such techniques often involves specialized models (e.g., VAE) which restricts the broad applicability of such approaches.
Some other work explores feature augmentation techniques to learn both invariant and spurious attributes, and use them to debias the output head (Lee et al., 2021).
Finally, a promising line of research has been focusing on using self-supervised learning to help the model avoid using spurious features in model predictions (Chen et al., 2020b;
Xie et al., 2020; Cai et al., 2021; Hamidieh et al., 2022).
Our work follows this latter line of work by proposing novel techniques to encourage model to learn diverse features that is bias-aware , but with a distinct purpose of better uncertainty quantiﬁcation.
D. A CTIVE LEARNING UNDER DATASET BIAS
In recent years, the role of training data in ensuring the model’s fairness and bias-mitigation performance has been increasing noticed.
Notably, (Chen et al., 2018) presented some of the earlier theoretical and empirical evidence that increasing training set size along is already effective in mitigating model unfairness.
Correspondingly, under the assumption that the group information in the unlabelled set is fully known , there has been several works that studies group-based sampling strate- gies and their impact on model behavior.
For example, Rai et al.
; Wang et al.
(2014) shows group-based active sampling strategy improves model performance under domain and distributional shifts, and Abernethy et al.
(2022) proves a guarantee for a worst-group active sampling strategy’s ability in helping the SGD-trained model to convergence to a solution that attains min-max fairness.
A second line of research focuses on designing better active learning objectives that incorporates fairness constraints, e.g., Fair Active Learning (FAL) (Anahideh et al., 2022) and PANDA (Sharaf et al., 2022).
Agarwal et al.
(2022) introduce a data repair algorithm using the coefﬁcient of variation to curate fair and contextually balanced data for a protected class(es).
Furthermore, there exists few active learning works formulating the objective of their method as optimizing a fairness-aware objective.
For example, Slice Tuner (Tae & Whang, 2021) proposes adaptive sampling strategy based on per-group learning curve to minimize fairness tradeoff, performs numeric optimization.
Cai et al.
(2022) which formalized the fairness learning problem as an min-max optimization objective, however their did not conduct further theoretical analysis of their objective, but instead proposed a per-group sampling algorithm based predicted model error using linear regression. Finally, a recent line of active learning work has been well-developed to formulate active learning from the perspective of distributional matching between labeled and unlabeled datasets. (Shui et al., 2020; de Mathelin et al., 2021; Mahmood et al., 2021; Xie et al., 2022)
These approaches are well-equipped to hand distributional shift, but often comes with the cost of assuming knowledge of the test feature distribution at the training time (Shui et al., 2020; de Mathelin et al., 2021;
Mahmood et al., 2021; Xie et al., 2022).
In comparison, our work conducts theoretical analysis of the optimization problem Section 3, and our proposed method (I SP ) does not require a priori knowledge (e.g., group information) from the unlabelled set.
On the other hand, there exists active re-sampling methods that do not require the knowledge of group information in the unlabelled set.
For example, Amini et al.
(2019) learns the data distribution using a VAE model under additional supervision of class / attribute labels, and then perform IPW sampling with respect to learned model.
REPAIR (Li & Vasconcelos, 2019) that estimates dataset bias using prediction error of a weak model, and then re-train model via e.g., sample re-weighting based on the estimated bias.
The bias estimation method used in this work is analogous to that of the JTT, which we compare with in our work.
A work close to our direction is Branchaud-Charron et al.
(2021), which shows DNN uncertainty (i.e., BatchBALD with Monte Carlo Dropout (Kirsch et al., 2019)) helps the model to achieve fairness objectives in active learning on a synthetic vision problem.
Our empirical result conﬁrms the ﬁnding of Branchaud-Charron et al.
(2021) on realistic datasets, and we further propose techniques to improve the vanilla DNN uncertainty estimators for more effective active learning under dataset bias.
As an aside, a recent work Farquhar et al.
(2020) studies the statistical bias in the estimation of active learning objectives due to the non-i.i.d.
nature of active sampling.
This is separate from the issue of dataset bias (i.e., imbalance in data group distribution) which we focus on in this work.
D. U NCERTAINTY ESTIMATION WITH DNN S
In recent years, the probabilistic machine learning (ML) literature has seen a plethora of work that study enabling calibrated predictive uncertainty in DNNss.
Given a model f , the probabilistic DNN model aims to learn a predictive distribution for the model function f , such that given training data D
= { ( y i , x i ) } ni = 1 and a testing point x test , the model outputs a predictive distribution f ( x test ) ∼
P ( f | x test , D ) rather than a simple point prediction.
To this end, the key challenge is to learn a predictive distribution (implicitly or explicitly) during the SGD-based training process of DNN, generating calibrated predictive uncertainty without signiﬁcantly impacting the accuracy or latency when compared to a deterministic DNN.
To this end, the classic works focus on the study of Bayesian neural networks (BNNs) (Neal, 2012), which took a full Bayesian approach by explicitly placing priors to the hidden weights of the neural network, and performance MCMC or variance inference during learning.
Although theoretically sound, BNN are delicate to apply in practice, with its performance highly dependent on prior choice and inference algorithm, and are observed to lead to suboptimal predictive accuracy or even poor uncertainty performance (e.g., under distributional distribution shift)
(Wenzel et al., 2020a;
Izmailov et al., 2021).
Although there exists ongoing works that actively advancing the BNN practice (e.g., Dusenberry et al.
On the other hand, some recent work studies computationally more approaches that implicitly learn a predictive distribution as part of deterministic SGD training.
Notable examples include Monte Carlo Dropout (Gal & Ghahramani, 2016) which generates predictive distribution by enabling the random Dropout mask during inference, and ensemble approaches such as Deep Ensemble (Lakshminarayanan et al., 2017) and their later variants (Maddox et al., 2019;
Wenzel et al., 2020b; Havasi et al., 2020) that trains multiple randomly-initialized networks to learn the modes of the posterior distribution of the neural network weights (Wilson & Izmailov, 2020).
Although generally regarded as the state-of-the-art in deep uncertainty quantiﬁcation, these methods are still computationally expensive, requiring multiple DNN forward passes at the inference time.
At the meantime, a more recent line of research avoids probablistic inference for the hidden weights altogether, focusing on learning a scalable probabilistic model (e.g., Gaussian process) to replace the last dense layer of the neural network (Van Amersfoort et al., 2020; van Amersfoort et al., 2021; Liu et al., 2022; Collier et al., 2021).
A key important observation in this line of work is the role of hidden representation quality in a model’s ability in obtaining high-quality predictive uncertainty.
In particular, Liu et al.
; Van Amersfoort et al.
(2020) suggests that this failure mode in DNN uncertainty can be caused by an issue in representation learning known as feature collapse , where the DNN over-focuses on correlational features that help to distinguish between output classes on the training data, but ignore the non-predictive but semantically meaningful input features that are important for uncertainty quantiﬁcation.
(Ming et al., 2022) also observed that DNN exhibits particular modes of failure in out-of-domain (OOD) detection in the presence of dataset bias.
Later, Tran et al.
; Minderer et al.
(2021) suggests that this issue can be partially mitigated by large-scale pre-traininig with large DNNs, where larger pre-trained DNN’s tend to exhibit stronger uncertainty performance even under spurious correlation and subpopulational shift.
In this work, we conﬁrm this observation in the setting of dataset bias in Figure 2), and propose simple procedures to mitigate this failure mode in representation learning without needing any change to the DNN model, and illustrates improvement even on top of large-scale pre-trained DNNs (BERT).
Deep uncertainty methods in active learning.
Active learning with DNNs is an active ﬁeld with numerous theoretical and applied works, we refer to (Matsushita et al., 2018;
Ren et al., 2021) for comprehensive survey, and only mention here few notable methods that involves DNN uncertainty estimation techniques.
Under a classiﬁcation model, the most classic approach to uncertainty-based active learning is to use the predictive distribution’s entropy, conﬁdence or margin as the acquisition policy (Settles, 1994).
Notice that in the binary classiﬁcation setting, these three acquisition policy are rank-equivalent since they are monotonic to the distance between max [ p ( x ) , 1 − p ( x )] and the null probability value of 0.5.
On the other hand, Batch Active learning by Diverse Gradient Embeddings (BADGE) (Ash et al., 2019) proposes to blend diversity-based acquisition policy into uncertaintybased active learning, by applying k-means++ algorithm to the gradient embedding of the classspeciﬁc logits (which quantiﬁes uncertainty).
As a result, BADGE may also suffer from the pathogloy in model representation under dataset bias, which this work is attempt to address.
Finally, Houlsby et al.
(2011) has proposed a information-theoretic policy Bayesian active learning by disagreement (BALD), which measures the mutual information between data points and model parameters and is adopted in the deep uncertainty literature (Gal et al., 2017; Kirsch et al., 2019;
Kothawade et al., 2022).
However, stable estimation of mutual information can be delicate in practice, and we leave the investigation of these advanced acquisition policies under dataset bias for future work.
E E XPERIMENT D ETAILS AND FURTHER DISCUSSION E. 2D C LASSIFICATION
We train a 10-member neural Gaussian process ensemble (as introduced in Appendix A.4), where each ensemble member is based on a 6-layer Dense residual network with 512 hidden units and pre-activation dropout mask (rate = 0.1).
The model is trained using Adam optimizer ( learning rate = 0.1) under cross entropy loss, and with a batch size 512 for 100 epochs.
After training, each ensemble member returns a tuple of predicted label probability, predicted under-representation probability and predictive uncertainty { ( p k ( y | x ) , p k ( b | x ) , v k ( x )) } 10 k = 1 , and we compute the ensemble
’s predicted probability surface as E k [ p ( y | x )] , predicted underrepresentation surface as E k
[ p k ( b | x )] , and the predictive uncertainty surface as E k [ v k
( y | x )] , where E k is the empirical average over the ensemble member predictions.
The predictive uncertainty surface of individual members is shown in Figures 6-7.
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
As shown, compared to the ERM-trained model, the introspective-trained model generates similar label prediction decision
I ( p ( y | x ) > 0 . 5 ) (Figures 2a v.s. 2e), but with much improved uncertainty surface (Figures 2b v.s. 2f).
Speciﬁcally, we compute predictive variance using the standard Gaussian process variance formula v ( x test )
=
k ( x test ) (cid:62) Vk ( x test ) , where k ( x test ) =
[ k ( x test , x 1 ) ,..., k ( x test , x n )] n ×is a vector of kernel distances based on the embedding distances
|| h ( x test ) − h
( x i )
|| 2 from the training data (Appendix A.4).
As shown, the model uncertainty under ERM model are not sufﬁciently sensitive to directions in the data space that are irrelevant for making prediction decisions on the training data (i.e., the directions that are parallel to the decision boundary)
(Figure 2f).
As a result, it did not learn sufﬁciently diverse hidden features, leading to a signiﬁcantly warped representation space that is extremely stretched out in the direction that is orthogonal to the decision boundary, and extremely compressed otherwise (Figure 2g).
Consequently, the model cannot strongly distinguish the minority examples from the majority examples in the representation space, and can become overconﬁdent even in unseen regions that was never covered by training data.
This can be undesirable for uncertainty quantiﬁcation under data bias, especially for the purpose of identifying underrepresented minority examples, where the distinguishing features between the minority and the majority examples are not predictive for the target label (e.g., the image background).
This issue is further exacerbated in the single models (see Figure 6).
In comparison, the uncertainty surface from an introspective-trained model does not suffer from this failure case.
As shown in Figure 2b, the model is less inclined to become overconﬁdent in unseen regions, especially in the neighborhood of the minority examples.
Correspondingly in the representation space, the model learned more diverse features and is able to better distinguish the minority examples from the majority examples (Figure 2b).
To understand how introspective training induces such improvement in model behavior, Figures (2g) and (2h) visualize the model’s underrepresentation prediction p ( b | x ) in the representation space and the data space, respectively.
As shown, due to the need of predicting the underrepresented examples (i.e., “introspection”) during training, the model has to learn hidden features that distinguishes the minority examples from the majority examples in its representation space, to a degree that they can be separated by a linear decision boundary in the last layer (Figure 2h).
Consequently, the model naturally learns a more disentangled representation space through simple multi-task training, and is able to provide predicted bias probabilities p ( b | x ) (Figure 2h) in addition to high-quality predictive uncertainty (Figure 2b) for the downstream active learning applications.
E. T ABULAR AND L ANGUAGE E XPERIMENTS
Data.
For tabular data, we use the U.S. Census Income data adult from the ofﬁcial UCI repository 6 .
For the language task, we use the CivilCommentsIdentity from the TensorFlow Dataset repository 7 .
For Census Income, we deﬁne the underrepresented groups as the union of (Female,
High Income) and (Black, High Income);
for Toxicity Detection, we deﬁne the underrepresented groups as the identity × label combination (male, female, white, black, LGBTQ, christian, muslim, other religion) ×
(toxic, non-toxic) (16 groups in total) as in (Koh et al., 2021).
For CivilComments, the identity annotation is a value between ( 0 , 1 ) (it is the average rating among multiple raters), and we include an example into the underrepresented group only if the rating > 0.99 (i.e. all raters agree about the identity) following (Koh et al., 2021).
However, we do note that this leads to a under coverage of the group membership, as many comments with plausible identity mentions are not included into the group identity labels.
Model.
For tabular experiments, we use a 2-layer Dense ResNet model with 128 hidden units and pre-activation dropout rate = 0.1, using a random-feature Gaussian process with hidden dimension 256 as the output layer (Liu et al., 2022)
(In the preliminary experiments, we tried larger models with update to 6-layers and 1024 hidden units, and did not observe signiﬁcant improvement).
For language experiments, we used BERT small mode initialized from the ofﬁcial pre-trained checkpoint released at BERT GitHub page(Turc
et al., 2019) 8 .
In each active learning round, we train the Dense ResNet model with Adam optimizer with learning rate 0.1, batch size 256 and maximum epoch 200; and train the BERT model with AdamW optimizer (learning rate 1e-5) for 6 epochs with batch size 16.
Active Learning Protocol.
Figure 8 visualizes the experiment protocol.
As shown, in each stage, we ﬁrst (optionally) trains a cross validated ensemble to estimate the under-representation labels, where we split the data into 10 cross-validation splits, and train ensemble members on 1 split and predict the rest of the 9 splits.
We then use the ensemble’s in-sample and out-of-sample predictions to compute the underrepresentation label ˆ b i (Equation (8)), and conduct introspective training (eq. (4)) to generate the ﬁnal active sampling signals for 8 rounds to generate the ﬁnal sampled data (red box).
At the end of round 8, we estimate the underrepresentation label for the ﬁnal sampled data, and send it to the ﬁnal model for reweighted training to generate the full accuracy-fairness frontier.
The sampling model is always a 10-member ensemble of neural Gaussian process (introduced in Appendix A.4), and the ﬁnal model is always a single DNN with architecture identical to the sampling model (i.e., 2-layer Dense ResNet for census income and BERT small for toxicity detection).
For both tasks, we randomly sample as small subset as the initial labelled dataset (2,500 out of 32, total training examples for census income, and 50,000 out of total 405,130 examples for toxicity detection), and use the rest of the training set as the unlabelled set for active learning.
For each sampling round, the AL model acquires 1,500 examples for census income, and 15,000 examples for the toxicity detection, so the total sample reaches roughly half of the total training set size afterrounds.
In the ﬁnal model training, we use the standard re-weighting objective (Liu et al., 2021):
[Equation]
where ˆ B is the set of underrepresented examples identiﬁed by the underrepresentation label, i.e., ( x i , y i )
∈
ˆ B if ˆ b i > t .
We vary the thresholds t and the up-weight coefﬁcient λ over a 2D grid ( t ∈ { 0 . 05 , 0 . 1 , 0 .
and log ( λ ) ∈ { 0 ., 0 . 5 , 1 , 1 . 5 ,..., 10 . } ) to get a collection of model accuracyfairness performances (i.e., accuracy v.s. worst-group accuracy), and use them to identify the Pareto frontier deﬁned by this combination of data and reweighting signal.
Active Learning Signals.
In this work, we consider four types of active sampling signals.
Recall that the sampling model (neural Gaussian process ensemble) is a K-member ensemble that generates three predictive quantities: (1) label probability { p k ( y | x ) } 10 k = 1 , (2) underrepresentation probability { p k ( b | x ) } 10 k = 1 and (3) predictive variance { v ( x ) } 10 k =
(recall that E k and Var k are the empirical mean and variance).
Margin : The gap between the highest class probability and the second highest class probability for the output label.
In the binary prediction context, this is equivalent to 2 ∗| p ( y | x ) − 0 .
5 | , i e., the gap between the mean predicted probability and the null value of 0.5.
We use the mean predictive label probability of the ensemble, which leads to: Margin ( x ) = 2 ∗| E k ( p k ( y | x )) − 0 .
5 | .
• Predicted Underrepresentation :
The mean predictive underrepresentation probability of the ensemble, which leads to: Underrep ( x ) =
E k ( p k
( b | x )) .
• Diversity : i.e., Ensemble Diversity (introduced in Appendix A.4).
The variance of label predictions: Diversity ( x ) =
Var k
( p k ( y | x )) .
Variance : i.e., Predictive Variance (introduced in Appendix A.4).
The mean of predictive variances: Variance ( x ) =
E k ( v k ( x )) .
F N OISE - BIAS - VARIANCE DECOMPOSITION IN R IDGE R EGRESSION
Consider ﬁtting a ridge regression model f ( x i )
=
β (cid:62) x i to the Gaussian observation data y i
= θ (cid:62) x i + ε , ε ∼ N ( 0 , σ 2 ) under an imbalanced experiment design, where we have | G | treatment groups and n g observations in each group. .
Here, x i =
[ 1 g i = 1 ,..., 1 g i = | G | ] is a | G | × 1 one-hot indicator of the membership of g i for each group in G , and θ =
[ θ 1 ,..., θ | G | ] is the true effect for each group.
Then, under ridge regression, the noise-bias-variance decomposition for group g is E D ( L ( y , f D ))
=
σ 2 + ( λθ g ) 2 ( n g + λ ) 2 + σ 2 n g ( n g + λ ) 2 , where the regularization parameter λ modulates a trade-off between the bias and variance terms.
In Appendix F.3, we also treat the case of group-speciﬁc noise ε i indep ∼ N ( 0 , σ 2 g ) .
F. E RROR D ECOMPOSITION IN A G ENERAL S ETTING
We ﬁrst derive the decomposition in a general setting with data { y i , φ i } ni = 1 , where φ i is the d ×(ﬁxed) features that follows a distribution P ( φ ) .
We consider a well-speciﬁed scenario where the data generation mechanism as:
[Equation]
where
[Equation]
and θ d × 1 = [ θ 1 ,..., θ | G | ] is the true coefﬁcient.
Under ridge regression, we ﬁt a linear model f ( x i ) =
φ (cid:62) i β to the data by minimizing the following squared loss objective:
[Equation]
which gives rise to the following solution:
[Equation]
notice ˆ β is a random variable that depends on the data Φ n × d = [ φ (cid:62) 1 ,..., φ (cid:62) n ] iid ∼ P ( φ ) .
Notice that under squared loss, the ensemble predictors ¯ f = argmin
f E Φ [( f − ˆ β (cid:62) x i ) 2 ] is simply the mean of individual predictors, i.e,
¯ f = E Φ
( ˆ β (cid:62) x i ) =
¯ β (cid:62) x i , where ¯ β = E Φ ( ˆ β ) .
Consequently, given a new observation { y , φ } , the noise-bias-variance decomposition of ˆ β under squared loss is:
[Equation]
As shown, to obtain a closed-form expression of the decomposition, we need to ﬁrst derive the expressions of Bias ( ˆ β ) =
[ θ − ¯ β ] and Var ( ˆ β ) .
Under the expression of the ridge predictor (13), we have:
[Equation]
with
[Equation]
where S =
( Φ (cid:62) Φ
+ λ I d ) − 1 Φ (cid:62) Φ
[Equation] As shown, the above expression depends on the random-matrix moments E [( Φ (cid:62) Φ + λ I d ) − 1 ] ,
E [( Φ (cid:62) Φ + λ I d ) − 2 ] , E [( Φ (cid:62)
Φ + λ I d ) − 1 Φ (cid:62) Φ ] and E [ θθ S
F. E RROR D ECOMPOSITION UNDER O RTHOGONAL D ESIGN
The above moments are in general difﬁcult to solve due to the involvement of matrix inverse and product within the expectation.
However, a closed-form expression is possible under an orthogonal design where φ i =
[ 1 g i = 1 ,..., 1 g i = | G | ] is the one-hot vector of treatment group memberships.
Then, denote Diag [ z g ] the diagonal matrix with diagonal elements z g and [ z gg (cid:48) ] gg (cid:48)
the full matrix whose ( g , g (cid:48) ) element is z gg (cid:48) , we have:
[Equation]
and
[Equation]
We are now ready to derive the full decomposition (15), without loss of generality, we assume φ i belongs to group g .
Then:
[Equation]
Consequently
[Equation]
we have the noise-bias-variance decomposition in (15) as:
[Equation]
F. E RROR D ECOMPOSITION UNDER O RTHOGONAL D ESIGN AND H ETEROGENEOUS N OISE
We now consider the case where y i ∼ N ( θ (cid:62) φ i , σ 2 g ) follows a normal distribution with group-speciﬁc noise.
Using the same decomposition as in F.1, we see that:
[Equation]
As shown
[Equation]
the nature of the bias and variance decomposition in fact does not change [Equation] and the noise component is now the group-speciﬁc variance σ 2 g .
Therefore, by following the same derivation as in
Appendix F. [Equation] we have:
[Equation]
G P ROOF OF P ROPOSITION
Through introspective training, there is a guarantee on a model’s bias-awareness based on its hidden representation and uncertainty estimates.
At convergence, a well-trained model f = ( f y , f b ) should satisfy the property that p ( b = 1 | x ) = σ ( f b ( x )) .
(I) (Bias-aware Hidden Representation)
We denote the odds for x belonging to the underrepresented group B as o b ( x ) = p ( x | b = 1 ) / p ( x
| b = 0 ) .
Using Bayes’ theorem, we derive the following: [Equation]
[Equation]
Hence, the hidden representation is aware of the likelihood ratio of whether an example x belongs to the underrepresented group, and the last-layer bias β 0 corresponds to the marginal likelihood ratio of the prevalence of the underrepresented groups p ( b = 1 ) / p ( b = 0 ) .
(II) (Bias-aware Embedding Distance)
Next, we examine the embedding distance between two examples ( x 1 , x 2 ) , i.e., || h ( x 1 ) − h ( x 2 )
|| 2 .
The Cauchy-Schwarz inequality states that for two vectors u and v of the Euclidean space, |(cid:104) u , v (cid:105)| ≤ || u |||| v || .
Hence, the distance between two embeddings can be expressed as β T [ h ( x 1 ) − h ( x 2 )] ≤ || β
||
2 || h
( x 1 ) − h ( x 2 )
|| 2 .
Using this property and Equation (16), we derive the following:
[Equation]
Since the above inequality is invariant to the relative position of ( x 1 , x 2 ) , we also have: 1 || β
|| 2 log o b ( x 2 )
o b ( x 1 ) ≤ || h
( x 1 ) − h ( x 2 ) || 2 , which implies:
[Equation]
As shown, the distance between the hidden embeddings h ( x 1 ) , h ( x 2 ) is lower-bounded by the logodds ratio that a given example is in the underrepresented group.
With this guarantee on the model’s learned embedding distance, we expect the hidden features to be more diverse than when trained on the main task alone, since it needs to sufﬁcient features to distinguish the underrepresented-group examples from those of the majority in the hidden space. H P ERFORMANCE G UARANTEE FOR L OSS - BASED T AIL - GROUP D ETECTION
In this section, we derive a lower bound for the group detection performance based on the self-play estimator introduced in Section 2.2.
We derive the result in a general setting without further assumptions on the data distribution.
The goal here is to provide a broadly applicable, and mathematically rigorous account of how various aspects of the data distribution and the model behavior impacts the performance of the subgroup detection procedure, rather than deriving the tightest possible performance guarantee for a speciﬁc family of model class or data distribution.
Our proof technique develops a novel Cantelli-type lower bound on the tail probabilities of the form P ( F ( l ) > q ) (where
F is a cumulative distribution function (CDF) and l a random variable), and also a novel tight upper bound on Var ( F ( l )) , which may be of independent interest. Let’s ﬁrst establish some notations. Recall P ( x , y , g ) is the data-generating distribution (Appendix A.1). For the majority group g = 0 and minority-group g = 1, we denote P 0 ( . )
=
P ( .
| g = 0 )
and P 1 ( . )
=
P ( .
{+| g = 1 ) the data distribution for the majority and minority group, and recall γ 
+}
{+= P ( G = 0 ) and γ 1 = 1 − γ 
+}
the prevalence of these two groups.
Furthermore:
Group-speciﬁc error and its distribution .
Given a loss function L ( .,. )
of Bergman divergence family and a ﬁxed, unfair model
f y that violates the sufﬁciency criteria (Arjovsky et al., 2019; Shui et al., 2022), denote l 0 =
L ( ˜ y 0 , f y ( x 0 )) and l 1 =
L ( ˜ y 1 , f y ( x 1 ))
the cross-validation generalization gaps for the majority and the minority examples ( ˜ y 0 , x 0 ) ∼ P 0 and ( ˜ y 1 , x 1 ) ∼ P 1 , where ˜ y g represents the true label without label noise.
Notice that ( l 0 , l 1 ) are random variables due to the randomness in ( P 0 , P 1 ) .
To this end, also denote ( µ 0 , µ 1 ) , ( σ 20 , σ 21 ) the means and variances of the group-speciﬁc losses ( l 0 , l 1 ) .
Due to the f y ’s violation of the sufﬁciency criteria (i.e., E 0 ( y | f y ( x ) = t ) (cid:54) =
E 1 ( y | f y ( x )
= t ) )
{+, we expect the model f y ’s cross-validation loss is systematically worse for the minority groups, i.e., there’s a systematic difference in group-speciﬁc loss µ 1 − µ 
+}
= d > 0.
Finally, denote F ( l i ) =
P ( l < l i ) the CDF for the distribution of the loss l = L ( y , f y ( x )) , ( y , x ) ∼ P , and ( F 0 , F 1 ) the CDF for the distributions of l 0 , l 1 , respectively.
Rank-based Estimator .
We identify the minority group examples using a rank-based estimator.
Speciﬁcally, recall F ( l i )
=
P ( l < l i ) is the CDF of the population loss, then the rank-based estimator for subgroup detection is:
[Equation]
That is, we include a training example x i into the introspective training only if the population quantile of its generalization error is higher than q , which is a user-speciﬁc threshold controlling the precision and recall of the estimator’s identiﬁcation performance.
{+Then, the below theorem describes how the estimator performance P ( g i = 
+}
| F ( l i ) > q ) is related to the user-speciﬁed threshold q , and the characteristics of data distribution (i.e., the group prevalence γ 0 , γ 1 ) as well as the classiﬁer performance (in terms of the group-speciﬁc loss distribution F 0 , F 1 ).
Theorem H.1 (Tail-group detection performance of rank-based estimator) .
For ( l 0 , l 1 ) a pair of random variables for minority- and majority-group examples.
Denote d = E ( l 1 − l 0 ) > 0 and σ 2 =
Var ( l 1 − l 0 ) the mean and variance of the between-group generalization gap.
Then, given a userspeciﬁed threshold q ∈ ( 0 , d 2 d 2 + σ 2 ) , the performance of the rank-based estimator P ( g = 1 | F ( l ) > q ) is bounded by:
[Equation]
[Equation]
where z = E [ F 0 ( l 1 )] − q (cid:112)
Var [ F 0 ( l 1 )] .
Here, F 0 ( l 1 ) = P ( l 0 < l 1 ) is the majority-group loss CDF F 0 evaluated at the minority-group loss l 1 =
L ( ˜ y 1 , f y ( x 1 ))
where ( ˜ y 1 , x 1 ) ∼ P 1 .
Proof is at Appendix J. Notice here due to the randomness in l 1 , F 0 ( l 1 ) is a random variable that follows a continuous distribution and is bounded within F 0 ∈ [ 0 , 1 ] .
Therefore F 0 ( l 1 ) has valid moments E
[ F 0 ( l 1 )] and Var [ F 0 ( l 1 )] .
We see that the performance bound (19) is intuitively sensible: a lower majority group prevalence γ 0 , a higher rank threshold q , and a higher likelihood for loss dominance F 0 ( l 1 ) =
P ( l 0 < l 1 ) all contributes a stronger identiﬁcation performance.
Here, we see that z = E [ F 0 ( l 1 )]
− q √ Var [ F 0 ( l 1 )] is a distribution dependent quantity that governs the difﬁculty of identifying the minority group.
Clearly, a larger magnitude of z leads to a stronger guarantee in detection performance in Theorem
H.1.
For the interested readers, the below result provides a lower bound for the magnitude of z in terms of the characteristics of model performance (i.e., the moments of the group-speciﬁc loss distributions
(e.g., d = µ 1 − µ 2 and ( σ 21 , σ 22 ) ), which can be used to obtain a more precise understanding of
P ( g = 1 | F ( l ) > q ) in (Equation (19)) in practice:
Theorem H.2 (Lower bound on the standardized likelihood of loss dominance z .) .
Consider the standardized likelihood of loss dominance z
= E [ F 0 ( l 1 )] − q √ Var [ F 0 ( l 1 )] , where the likelihood of loss dominance
F 0 ( l 1 ) =
P ( l 0 < l 1 ) is a random variable in terms of l 1 =
L ( ˜ y 1 , f y ( x 1 )) , ( ˜ y 1 , x 1 ) ∼ P 1 .
We have:
[Equation]
where F 0 ( µ 0 ) = P ( l 0 ≤ µ 0 ) is the probability of the majority-group loss l 0 smaller than its mean µ 0 =
E ( l 0 ) .
Therefore, z can be bounded by:
[Equation]
The proof is in Appendix J. It relies on a novel upper bound of CDF variance (i.e., Equation (20)), which is important for guaranteeing a high magnitude of z and consequently a tighter bound for group detection performance in Equation (19).
The combination of Theorems H.1-H.2 provides us an opportunity to quantitatively understand of the group detection performance P ( g = 1 | F ( l ) > q ) in terms of the estimator conﬁguration (i.e., the user-speciﬁed threshold q ), data distribution (i.e., the majority-group prevalence γ 0 ), and the degree of unfairness of the vanilla model f y (in terms of the distributions of the group-speciﬁc losses).
{+For example, consider a setting with majority-group prevalence γ 
+}
= 0 .
85, expected betweengroup loss gap d = 1, group-speciﬁc variances σ 1 = σ 2 = 0 .
15 and P ( l 0 < µ 0 )
= 0 .
5, using a percentile threshold q = 0 .
9, the rank-based estimator
I ( F ( l ) > q ) has group detection performance
P ( g = 1 | F ( l ) > q ) > 0 . 9.
It is also worth commenting that, following the previous discussion, we see that the success of group identiﬁcation relies on valid estimation of model’s generalization error (with respect to the true label).
To this end, the K-fold cross-validated ensemble procedure we used in this work is known to produce unbiased and low-variance estimates for the expected generalization loss (Blum et al., 1999; Kumar et al., 2013).
Finally, we highlight that the above results are broadly applicable and derived under weak, nearly assumption-free conditions.
A even tighter performance bounds can be obtained by making further assumptions on the family of data distributions or model class, which is outside the scope of the current work.
Another interesting direction is to incorporate the ﬁnite-sample estimation error of
K-fold cross validation into the analysis (Blum et al., 1999; Kumar et al., 2013; Bayle et al., 2020), although this necessitates a careful treatment of model’s generalization behavior (i.e., loss stability) in the subgroup setting, which we will pursue in the future work.
I P ROOF OF T HEOREM
Proof.
Let r ( α ) =
( α n ) − p and r agg ( n ) =
τ n − q + δ .
As discussed in section 3, we assume that the group-speciﬁc risk decays as
[Equation]
where ˆ f α , n is the minimizer of the γ -weighted empirical risk on a dataset of size n with allocation α and the expectation is with respect to the randomness of the dataset used to train ˆ f .
Under the above assumption, the optimization problem for α looks like
[Equation]
The term r agg ( n ) is a constant that does not impact the optimal solution.
Hence, we drop it and focus on the problem
[Equation]
We begin by noting that the objective is strictly convex and the domain of optimization is bounded.
Hence, there exists a unique global optimum.
Turning the constraint ∑ g ∈ G α g = 1 into a Lagrangian, we obtain
[Equation]
where λ ∈ R is a Lagrange multiplier.
At the optimum, we know that the Lagrangian should have 0 within its subdifferential wrt α (Boyd &
Vandenberghe, 2004).
The subdifferential is given by
[Equation]
  where µ ∈ ∆ | G | is such that µ g >
⇐⇒ g ∈ argmax
[Equation]
   where r (cid:48) denotes the derivative of r .
Since 0 belongs to the subdifferential at the optimum, there must exist µ satisfying the constraints above such that
[Equation]
where r (cid:48) inv is the inverse of r (cid:48) .
Since r (cid:48) is a homogeneuous function of its argument, so is its inverse, and λ can be eliminated to enforce the constraint ∑ g α g = 1, so that the optimal solution α (cid:63) g is
[Equation]
Let s denote the denominator and θ g = ω
+( 1 − ω ) µ g γ g
so that
[Equation]
s and θ must satisfy:
[Equation]
Using the fact that r ( t ) = ( nt ) − p , we have that
[Equation]
If we sort groups in ascending order according to c − 1 p g γ g to obtain g 1 , g 2 ,... , a value of θ satisfying the conditions above can be computed as follows:
Initialize θ g = ω ∀ g ∈ G
Set k =
Until ∑ g θ g γ g = 1 or k = | G | repeat: a) l = min ( k + 1 , | G | ) b) Set θ g j = θ g j t (for j ≤ k ) for the largest t ≥ 1 such that
(i) ∑ g θ g γ g = 1 or (ii) c − 1 p g 1 γ g 1 θ g 1 = ...
=
c − 1 p g l γ g l θ g l c) k
= k +
It is easy to see that this algorithm must terminate as it can go for at most | G | rounds.
Further, by construction, we have θ g ≥ ω
∀ g ∈ G since we start at these values and only scale up any of the θ g .
At the j -th iteration of the loop, we have that
[Equation]
In the j -th iteration, we scale all θ g j ( m = 1 ,..., j + 1) up by the same factor t by the same amount so that we achieve the above for all groups upto g j + 1 , or we hit the constraint ∑ g θ g γ g = 1.
If the former happens, we begin the next iteration with the same invariant.
If the latter happens, we have obtained a θ that satisﬁes ∑ g θ g γ g = 1 and θ g i > ω for i <
k and g i ∈ argmin g (cid:48)
c − 1 p g (cid:48)
γ g θ g for i < k .
If we reach the iteration where k = | G | , we can simply scale up all the θ g by the same amount until ∑ g θ g γ g = 1 is satisﬁed and we would have that all c − 1 p g γ g θ g is equal for all g ∈ G and all θ g > ω . J P ROOF OF T HEOREMS H.1 AND H. J. P ROOF FOR T HEOREM H.  Proof. By Bayes’ rule, we have:
[Equation]
Expression for P ( g = 1 ) and P
F ( l )
> q )
Notice that P ( g = 1 ) = 1 − γ 0 , and P ( F ( l ) > q ) =
P ( U > q ) = 1 − q for U ∼ Unif ( 0 , 1 ) .
Here we used the fact that F ( l ) ∼ Unif ( 0 , 1 ) when F is the CDF for the distribution of l (Blitzstein & Hwang, 2015).
Expression for P ( F ( l ) > q | g = 1 )
Notice P ( F ( l ) > q | g = 1 ) =
P ( F ( l 1 ) > q ) ,
where l 1 =
L ( ˜ y 1 , f y ( x 1 )) , ( ˜ y 1 , x 1 ) ∼ P 1 is the random variable for the minority-group loss.
Also notice that F ( l 1 ) =
P ( l (cid:48) <
l 1 ) where l (cid:48)
=
L ( y (cid:48) , f y ( x (cid:48) )) for ( y (cid:48) , x (cid:48) , g (cid:48) ) ∼ P is the non-group-speciﬁc loss.
Then a conditional decomposition of P ( F ( l 1 ) > q ) reveals :
[Equation]
where the last equality follows since P ( g (cid:48)
= 0 )
= γ 0 , P ( g (cid:48)
= 1 )
= 1 − γ 0 and P (cid:2) P ( l (cid:48) <
l 1 ) > q (cid:3) =
P (cid:2) F 1 ( l 1 ) > q (cid:3) =
P ( U > q ) = 1 − q for U ∼ Unif ( 0 , 1 ) .
Here we used the fact that F 1 ( l 1 ) ∼ Unif ( 0 , 1 ) when F 1 is the CDF for the distribution of l 1 (Blitzstein & Hwang, 2015).
[Table]
) >
[Table]
(cid:1)
[Table]
(Lyons & Peres, 2017).
Speciﬁcally, for a random variable R with E ( R ) ≥ 0, we have:
E
[Equation]
R )
[Equation]
≤ E ( R ∗ 1 R > 0 ) 2 ≤ E ( R 2 ) P ( R > 0 ) which implies
P ( R > 0 )
[Equation]
Var
[Equation]
R
[Equation]
Setting R = F 0 ( l 1 ) − q , we have
[Equation]
which implies P ( F 0 ( l 1 ) > q ) ≥ z1 + z
Derive the ﬁnal bound
Finally, using the above three facts, we can express Equation (22) as:
[Equation]
which further leads to:
[Equation]
yielding the ﬁnal bound P ( g = 1 | F ( l ) > q ) ≥ ( 1 −
[Equation]
as in Equation (19)
[Equation]
J. P ROOF FOR T HEOREM H.
Proof.
For F 0 ( l 1 ) , recall F 0 is the CDF of l 0 =
L ( ˜ y 0 , f y ( x 0 ))
where ( ˜ y 0 , x 0 ) ∼ P 0 and l 1 =
L ( ˜ y 1 , f y ( x 1 )) is the random variable of minority-group loss with ( ˜ y 1 , x 1 ) ∼ P 1 .
To derive lower bound for z
= E ( F 0 ( l 1 )) − q Var ( F 0 ( l 1 )) , ﬁrst derive bounds on E ( F 0 ( l 1 )) and Var ( F 0 ( l 1 )) :
Lower bound for E ( F 0 ( l 1 ))
Recall P ( F ( l ) | g = 1 ) =
P ( F ( l 1 )) ,
where l 1 =
L ( ˜ y 1 , f y ( x 1 )) is the random variable for minority-group error with ( ˜ y 1 , f y ( x 1 )) ∼ P 1 .
Then:
[Equation]
By the second moment method inequality (Lyons & Peres, 2017), we have:
[Equation]
Upper bound for Var ( F 0 ( l 1 ))
Recall that Var ( F ( l )) =
E (cid:104) ( F ( l ) − E ( F ( l ))) 2 (cid:105) .
Using the iterative expectation formula, we split the variance computation for F 0 ( l 1 ) into two regions of l 1 depending on whether the minority-group loss of l 1 ≥ µ 1 − λσ 1 :
[Equation]
which holds for any positive multiplier λ > 0.
Notice that in the above, P ( l 1 < µ 1 − λσ 1 ) in the second line describes the tail probability of the loss distribution of l 1 , for suitably large λ , P ( l 1 < µ 1 − λσ 1 ) should be small.
On the other hand,
Var ( F 0 ( l 1 ) | l 1 ≥ µ 1 − λσ 1 ) in the second line describes the variance of F 0 ( l 1 ) in the region where lis large.
{+When the distribution of l 0 and l 1 is well separated (i.e., µ 1 − µ 
+}
= d > 0), for suitable value of λ , we expect the value of F 0 ( l 1 = F 0 ( µ 1 − λσ 1 )) to be high and close to 1, and as a result the
Var ( F 0 ( l 1 ) | l 1 ≥ µ 1 − λσ 1 ) will be low since F 0 ( l 1 ) is bounded within a small range [ F 0 ( µ 1 − λσ 1 ) , 1 ] .
Consequently, to obtain a tight upper bound of Var ( F ( l )) , it is sufﬁcient to identify a suitable value of λ such that both Var ( F 0 ( l 1 ) | l 1 ≥ µ 1 − λσ 1 ) and P ( l 1 < µ 1 − λσ 1 ) are low.
To identify a suitable value of λ , ﬁrst derive an upper bound of Var ( F 0 ( l 1 )) in terms of λ .
Notice below two facts:
• By Cantelli’s inequality:
[Equation]
• By the inequality for variance of the bounded variables:
[Equation]
where the ﬁrst inequality follows by the fact that for a random variable R bounded between [ a , b ] (in this case between [ F ( µ 1 − λσ 1 ) , 1 ] ), its variance is bounded by Var ( R ) ≤ ( b − E ( R ))
( E ( R ) − a ) ≤ 14 ( b − a ) 2 .
• By Markov’s inequality, F 0 ( l 1 ) =
P ( l 1 ≤ l 1 ) ≥ µ 0 / l 1 , which implies:
[Equation]
where
the ﬁrst inequality also follows by the variance inequality of
the
bounded variables
Using the above two facts, we can bound Var ( F 0 ( l 1 )) as:
[Equation]
where the ﬁrst inequality holds since the ﬁrst conditional variance term Var ( F 0 ( l 1 ) | l 1 ≤ µ 1 − λσ 1 ) (i.e., variance in the majority bulk) is expected to be much larger than the second term Var ( F 0 ( l 1 ) | l 1 > µ 1 − λσ 1 ) (i.e., variance in the far tail), therefore assigning the highest possible probability weight
P ( l 1 ≤ µ 1 − λσ 1 ) to the larger variance term Var ( F 0 ( l 1 ) | l 1 ≤ µ 1 − λσ 1 ) leads to an upper bound for
Var ( F 0 ( l 1 )) .
Further simplifying Equation (24), we arrive at:
[Equation]
where recall d = µ 1 − µ 0 is the expected error gap between the majority and the minority groups.
Setting λ = d σ 1 , we have:
[Equation]
[Table]
Finally, plugging the bounds for E ( F 0 ( l 1 )) and Var ( F 0 ( l 1 )) from Equation (23) and 25 into z =
E
[ F 0 ( l 1 )] − q √ Var [ F 0 ( l 1 )] yields the ﬁnal lower bound in Equation (21), i.e.,
[Equation]