L EARNING TO S PLIT FOR A UTOMATIC B IAS D ETECTION
Anonymous authors
Paper under double-blind review
A BSTRACT
Classiﬁers are biased when trained on biased datasets.
As a remedy, we propose
Learning to Split ( ls ), an algorithm for automatic bias detection.
Given a dataset with input-label pairs, ls learns to split this dataset so that predictors trained on the training split cannot generalize to the testing split.
This performance gap suggests that the testing split is under-represented in the dataset, which is a signal of potential bias.
Identifying non-generalizable splits is challenging since we have no annotations about the bias.
In this work, we show that the prediction correctness of each example in the testing split can be used as a source of weak supervision: generalization performance will drop if we move examples that are predicted correctly away from the testing split, leaving only those that are mispredicted. ls is task-agnostic and can be applied to any supervised learning problem, ranging from natural language understanding and image classiﬁcation to molecular property prediction.
Empirical results show that ls is able to generate astonishingly challenging splits that correlate with human-identiﬁed biases.
Moreover, we demonstrate that combining robust learning algorithms (such as group DRO) with splits identiﬁed by ls enables automatic de-biasing.
Compared to previous state-of-the-art, we substantially improve the worst-group performance (23.4% on average) when the source of biases is unknown during training and validation.
Our code is included in the supplemental materials and will be publicly available.
I NTRODUCTION
Recent work has shown promising results on de-biasing when the sources of bias (e.g., gender, race) are known a priori Ren et al.
; Sagawa et al.
; Clark et al.
; He et al.
Mahabadi et al.
; Kaneko & Bollegala (2021).
However, in the general case, identifying bias in an arbitrary dataset may be challenging even for domain experts: it requires expert knowledge of the task and details of the annotation protocols Zellers et al.
; Sakaguchi et al.
In this work, we study automatic bias detection : given a dataset with only input-label pairs, our goal is to detect biases that may hinder predictors’ generalization performance.
We propose Learning to Split ( ls ), an algorithm that simulates generalization failure directly from the set of input-label pairs.
Speciﬁcally, ls learns to split the dataset so that predictors trained on the training split cannot generalize to the testing split (Figure 1).
This performance gap indicates that the testing split is under-represented among the set of annotations, which is a signal of potential bias.
The challenge in this seemingly simple formulation lies in the existence of many trivial splits.
For example, poor testing performance can result from a training split that is much smaller than the testing split (Figure 2a).
Classiﬁers will also fail if the training split contains all positive examples, leaving the testing split with only negative examples (Figure 2b).
The poor generalization of these trivial solutions arise from the lack of training data and label imbalance, and they do not reveal the hidden biases.
To ensure that the learned splits are meaningful, we impose two regularity constraints on the splits.
First, the size of the training split must be comparable to the size of the testing split.
Second, the marginal distribution of the labels should be the similar across the splits.
Our algorithm ls consists of two components, Splitter and Predictor .
At each iteration, the Splitter ﬁrst assigns each input-label pair to either the training split or the testing split.
The Predictor then takes the training split and learns how to predict the label from the input.
Its prediction performance on the testing split is used to guide the Splitter towards a more challenging split (under the regularity
[Figure]
Training
Testing
[Figure]
constraints) for the next iteration.
Speciﬁcally, while we do not have any explicit annotations for creating non-generalizable splits, we show that the prediction correctness of each testing example can serve as a source of weak supervision: generalization performance will decrease if we move examples that are predicted correctly away from the testing split, leaving only those predicted incorrectly.
ls is task-agnostic and can be applied to any supervised learning problem, ranging from natural language understanding (Beer Reviews, MNLI) and image classiﬁcation (Waterbirds, CelebA) to molecular property prediction (Tox21).
Given the set of input-label pairs, ls consistently identiﬁes splits across which predictors cannot generalize.
For example in MNLI, the generalization performance drops from 79.4% (split by random) to 27.8% (split by ls ) for a standard BERT-based predictor.
Further analysis reveals that our learned splits coincide with human-identiﬁed biases.
Finally, we demonstrate that combining group distributionally robust optimization (DRO) with splits identiﬁed by ls enables automatic de-biasing.
Compared with previous state-of-the-art, we substantially improves the worst-group performance (23.4% on average) when the sources of bias are completely unknown during training and validation.
R ELATED WORK
De-biasing algorithms
Modern datasets are often coupled with unwanted biases Buolamwini &
Gebru (2018); Schuster et al.
McCoy et al.
; Yang et al.
If the biases have already been identiﬁed, we can use this prior knowledge to regulate their negative impact Kusner et al.
; Hu et al.
; Oren et al.
; Belinkov et al.
; Stacey et al.
; Clark et al.
; He et al.
; Mahabadi et al.
; Sagawa et al.
; Singh et al.
The challenge arises when the source of biases is unknown (Li & Xu, 2021).
Recent work has shown that the mistakes of a standard ERM predictor on its training data are informative of the biases (Bao et al., 2021; Sanh et al., 2021;
Nam et al., 2020; Utama et al., 2020; Liu et al., 2021a;
Lahoti et al., 2020; Liu et al., 2021b; Bao et al., 2022).
They deliver robustness by boosting from the mistakes.
(Wang & Vasconcelos, 2018; Yoo & Kweon, 2019) also utilize prediction correctness for conﬁdence estimation and active learning.
(Creager et al., 2021;
Sohoni et al., 2020; Ahmed et al., 2020; Matsuura & Harada, 2020) further analyze the predictor’s hidden activations to identify under-represented groups.
However, many other factors (such as the initialization, the representation power, the amount of annotations, etc) can contribute to the predictors’ training mistakes.
For example, predictors that lack representation power may simply under-ﬁt the training data.
In this work, instead of looking at the training statistics of the predictor, we focus on its generalization gap from the training split to the testing split.
This effectively balances those unwanted factors.
Training
Testing
[Figure]
Going back to the previous example, if the training and test splits share the same distribution, the generalization gap will be small even if the predictors are underﬁtted.
The gap will increase only when the training and testing splits have different prediction characteristics.
Furthermore, instead of using a ﬁxed predictor, we iteratively reﬁne the predictor during training so that it faithfully measures the generalization gap given the current Splitter.
Heuristics for data splitting Data splitting strategy directly impacts the difﬁculty of the underlying generalization task.
Therefore, in domains where out-of-distribution generalization is crucial for performance, various heuristics are used to ﬁnd challenging splits Sheridan (2013); Yang et al.
Bandi et al.
; Yala et al. (2021); Taylor et al.
Koh et al.
Examples include scaffold split in molecules and batch split for cells.
Unlike these methods, which rely on human-speciﬁed heuristics, our algorithm ls learns how to split directly from the dataset alone and can therefore be applied to scenarios where human knowledge is unavailable or incomplete.
L EARNING TO S PLIT 3. M OTIVATION
Given a dataset D total with input-label pairs { ( x, y ) } , our goal is to split this dataset into two subsets,
D train and D test , such that predictors learned on the training split D train cannot generalize to the testing split D test .
Why do we have to discover such splits?
Before deploying our trained models, it is crucial to understand the extent to which these models can even generalize within the given dataset.
The standard cross-validation approach attempts to measure generalization by randomly splitting the dataset (Stone, 1974; Allen, 1974).
However, this measure only reﬂects the average performance under the same data distribution P D total ( x, y ) .
There is no guarantee of performance if our data distribution changes at test time (e.g. increasing the proportion of the minority group).
For example, consider the task of classifying samoyeds vs. polar bears (Figure 1).
Models can achieve good average performance by using spurious heuristics such as “polar bears live in snowy habitats” and “samoyeds play on grass”.
Finding splits across which the models cannot generalize helps us identify underrepresented groups (polar bears that appear on grass).
How to discover such splits?
Our algorithm ls has two components, a Splitter that decides how to split the dataset and a Predictor that estimates the generalization gap from the training split to the testing split.
At each iteration, the splitter uses the feedback from the predictor to update its splitting decision.
One can view this splitting decision as a latent variable that represents the prediction characteristic of each input.
To avoid degenerate solutions, we require the Splitter to
Input: dataset D total
Output train
Sample mini batch Sample mini Update Splitter the L decreasing 11: increasing satisfy two regularity constraints: the size of the training split should be comparable to the size of the testing split (Figure 2a); and the marginal distribution of the label should be similar across the splits (Figure 2b).
3. S PLITTER AND P REDICTOR
Here we describe the two key components of our algorithm, Splitter and Predictor , in the context of classiﬁcation tasks.
The algorithm itself generalizes to regression problems as well.
Splitter Given a list of input-label pairs D total
=
[( x 1 , y 2 ) , . .
, ( x n , y n )] , the Splitter decides how to partition this dataset into a training split D train and a testing split D test .
We can view its splitting decisions as a list of latent variables z
=
[ z 1 , . . .
, z n ] where each z i 2 { 1 , 0 } indicates whether example ( x i , y i ) is included in the training split or not.
In this work, we assume independent selections for simplicity.
That is, the Splitter takes one input-label pair ( x i , y i ) at a time and predicts the probability P Splitter ( z i | x i , y i ) of allocating this example to the training split.
We can factor the joint probability of our splitting decisions as
[Equation]
=
P Splitter ( z i | x i , y i ) .
We can sample from the Splitter’s predictions P Splitter ( z i | x i , y i ) to obtain the splits D train and
D test .
Note that while the splitting decisions are independent across different examples, the Splitter receives global feedback, dependent on the entire dataset D total , from the Predictor during training.
Predictor The Predictor takes an input x and predicts the probability of its label P Predictor ( y | x ) .
The goal of this Predictor is to provide feedback for the Splitter so that it can generate more challenging splits at the next iteration.
Speciﬁcally, given the Splitter’s current splitting decisions, we re-initialize the Predictor and train it to minimize the empirical risk on the training split D train .
This re-initialization step is critical because it ensures that the predictor does not carry over past information (from previous splits) and faithfully represents the current generalization gap.
On the other hand, we note that neural networks can easily remember the training split.
To prevent over-ﬁtting, we held-out 1/3 of the training split for early stopping.
After training, we evaluate the generalization performance of the Predictor on the testing split D test .
3. R EGULARITY CONSTRAINTS
Many factors can impact generalization, but not all of them are of interest.
For example, the Predictor may naturally fail to generalize due to the lack of training data or due to label imbalance across the splits (Figure 2).
To avoid these trivial solutions, we introduce two regularizers to shape the Splitter’s decisions:
[Equation]
The ﬁrst term ⌦ 1 ensures that we have sufﬁcient training examples in D train .
Speciﬁcally, the marginal distribution P ( z ) = 1 n
P ni =1 P Splitter ( z i =
z | x i , y i ) represents what percentages of
D total are split into D train and D test .
We penalize the Splitter if it moves too far away from the prior distribution Bernoulli(  ) .
Centola et al.
(2018) suggest that minority groups typically make uppercent of the population.
Therefore, we ﬁx  = 0 .
75 in all experiments.
The second term ⌦ 2 aims to reduce label imbalance across the splits.
It achieves this goal by pushing the label marginals in the training split P ( y | z = 1) and the testing split P ( y | z = 0) to be close to the original label marginal P ( y ) in D total .
We can apply Bayes’s rule to compute these conditional label marginals directly from the Splitter’s decisions P S. ( z i | x i , y i ) :
P (
[Equation]
3. T RAINING STRATEGY
The only question that remains is how to learn the Splitter.
Our goal is to produce difﬁcult and nontrivial splits so that the Predictor cannot generalize.
However, the challenge is that we don’t have any explicit annotations for the splitting decisions.
There are a few options to address this challenge.
From the meta learning perspective, we can backpropagate the Predictor’s loss on the testing split directly to the Splitter.
This process is expensive as it involves higher order gradients from the Predictor’s training.
While one can apply episodictraining (Vinyals et al., 2016) to reduce the computation cost, the Splitter’s decision will be biased by the size of the learning episodes (since the Predictor only operates on the sampled episode).
From the reinforcement learning viewpoint, we can cast our objectives, maximizing the generalization gap while maintaining the regularity constraints, into a reward function (Lei et al., 2016).
However, according to our preliminary experiments, the learning signal from this scalar reward is too sparse for the Splitter to learn meaningful splits.
In this work, we take a simple yet effective approach to learn the Splitter.
Our intuition is that the
Predictor’s generalization performance will drop if we move examples that are predicted correctly away from the testing split, leaving only those that are mispredicted .
In other words, we can view the prediction correctness of the testing example as a direct supervision for the Splitter.
Formally, let ˆ
y
i be the Predictor’s prediction for input x
i : ˆ y i = arg max y P Predictor
( y | x i ) .
We minimize the cross entropy loss between the Splitter’s decision and the Predictor’s prediction correctness over the testing split:
[Equation]
Combining with the aforementioned regularity constraints, the overall objective for the Splitter is
[Equation]
One can explore different weighting schemes for the three loss terms (Chen et al., 2018).
In this paper, we found that the unweighted summation (Eq 3) works well out-of-the-box across all our experiments.
Algorithm 1 presents the pseudo-code of our algorithm.
At each outer-loop (line 2-11), we start by using the current Splitter to partition D total into D train and D test .
We train the Predictor from scratch on D train and evaluate its generalization performance on D test .
For computation efﬁciency, we sample mini-batches in the inner-loop (line 6-10) and update the Splitter based on
Eq equation 3.
[Figure]
ot70EiyCp5KIqMeiHjxWsB/QxLLZbtqlm03YnYglBLz4V7x4UMSrf8Kb
[Table]
+ WKVbUmMOeJnZMKyVHvlr+cXshiHwJkgirVsa0I3YRK5ExAWnJiBRFlIzqAjqYB9UG5yeSF1DzUSs/sh1JXgOZE/T2RUF+pse/pzuxgNetl4n9eJ8b+uZvwIIoRAjZd1I+FiaGZ5WH2uASGYqwJZZLrW002pJIy1KmVdAj27MvzpHlctU+r9s1JpXaRx1Ek++SAHBGbnJEauSZ10iCMPJJn8krejCfjxXg3PqatBSOf2SV/
[Figure]
Contradiction
Entailment & Neutral
MNLI
Inputs with negation words 50% 98% 50% 2%
Waterbirds Landbirds Waterbirds
Inputs with land backgrounds 87% 99% 13% 1%
Blond_hair
No_blond_hair CelebA Inputs with male= True <latexit sha1_base64="dIC/ql0jyBokHBqNo+ZsMBy0lks=">AAACA3icbVBNS8NAEN3Ur1q/ot70EiyCp5KIqMeiHjxWsB/QxLLZbtqlm03YnYglBLz4V7x4UMSrf8Kb/8ZNm4O2Phh4vDfDzDw
[Table]
++SAHBGbnJEauSZ10iCMPJJn8krejCfjxXg3PqatBSOf2SV
[Figure]
YHz
[Figure]
E XPERIMENTS
We conduct experiments over multiple modalities (Section 4.1) and answer two main questions.
Can ls identify splits that are not generalizable (Section 4.2)?
Can we use the splits identiﬁed by ls to reduce unknown biases (Section 4.3)?
Implementation details are deferred to the Appendix.
Our code is included in the supplemental materials and will be publicly available.
4. D ATASET
Beer Reviews We use the BeerAdvocate review dataset (McAuley et al., 2012) where each input review describes multiple aspects of a beer and is written by a website user.
Following previous work (Lei et al., 2016), we consider two aspect-level sentiment classiﬁcation tasks: look and aroma.
There are 2,500 positive reviews and 2,500 negative reviews for each task.
The average word count per review is 128.5.
We apply ls to identify spurious splits for each task.
Tox21 Tox21 is a property prediction benchmark with 12,707 molecules Huang et al.
Each input is annotated with a set of binary properties that represent the outcomes of different toxicological experiments.
We consider the property Androgen Receptor ( active or inactive ) as our prediction target.
We apply ls to identify spurious splits over the entire dataset.
[Figure]
ls
[Figure]
[Figure]
Waterbirds Sagawa et al.
(2019) created this dataset by combining bird images from the CaltechUCSD Birds-200-2011 (CUB) dataset (Welinder et al., 2010) with backgrounds from the Places dataset (Zhou et al., 2014).
The task is to predict waterbirds vs. landbirds.
The challenge is that waterbirds, by construction, appear more frequently with a water background.
As a result, predictors may utilize this spurious correlation to make their predictions.
We combine the ofﬁcial training data and validation data (5994 examples in total) and apply ls to identify spurious splits.
CelebA CelebA is an image classiﬁcation dataset where each input image (face) is paired with multiple human-annotated attributes Liu et al. (2015).
Following previous work (Sagawa et al., 2019), we treat the hair color attribute ( y 2 { blond , not blond } ) as our prediction target.
The label is spuriously correlated with the gender attribute ( { male , female } ).
We apply ls to identify spurious splits over the ofﬁcial training data (162,770 examples).
MNLI MNLI is a crowd-sourced collection of 433k sentence pairs annotated with textual entailment information (Williams et al., 2018).
The task is to classify the relationship between a pair of sentences: entailment, neutral or contradiction.
Previous work has found that contradiction examples often include negation words (McCoy et al., 2019).
We apply ls to identify spurious splits over the training data (206,175 examples) created by Sagawa et al.
4. I DENTIFYING NON - GENERALIZABLE SPLITS
Figure 3 presents the splits identiﬁed by our algorithm ls .
Compared to random splitting, ls achieves astonishingly higher generalization gaps across all 6 tasks.
Moreover, we observe that the learned splits are not degenerative: the training split D train and testing split D test share similar label distributions.
This conﬁrms the effectiveness of our regularity objectives.
Why are the learned splits so challenging for predictors to generalize across?
While ls only has access to the set of input-label pairs, Figure 4 and Figure 5 show that the learned splits are informative of human-identiﬁed biases.
For example, in the generated training split of MNLI, inputs with negation words are mostly labeled as contradiction.
This encourages predictors to leverage the
[Table]
presence of negation words to make their predictions.
These biased predictors cannot generalize to the testing split [Table] where inputs with negation words are mostly labeled as entailment or neutral.
Convergence and time-efﬁciency ls requires learning a new Predictor for each outer-loop iteration.
While this makes ls more time-consuming than training a regular ERM model, this procedure guarantees that the Predictor faithfully measures the generalization gap based on the current Splitter.
Figure 6 shows the learning curve of ls .
We observe that the generalization gap steadily increases as we reﬁne the Splitter and the learning procedure usually converges within 50 outer-loop iterations.
4. A UTOMATIC DE - BIASING
Once ls has identiﬁed the spurious splits, we can apply robust learning algorithms to learn models that generalize across the splits.
Here we consider group distributionally robust optimization (group
DRO) and study three well-established benchmarks: Waterbirds, CelebA and MNLI.
Group DRO Group DRO has shown strong performance when biases are annotated (Sagawa et al., 2019).
For example in CelebA, gender ( male , female ) constitutes a bias for predicting blond hair.
Group DRO uses the gender annotations to partition the training data into four groups: { blond hair , male } , { blond hair , female } , { no blond hair , male } , { no blond hair , female } .
By minimizing the worst-group loss during training, it regularizes the impact of the unwanted gender bias.
At test time, we report the average accuracy and worstgroup accuracy over a held-out test set.
Group DRO with supervised bias predictor Recent work consider a more challenging setting where bias annotations are not provided at train time.
CVaR DRO (Levy et al., 2020) up-weights examples that have the highest training losses.
LfF (Nam et al., 2020) and JTT (Liu et al., 2021a) train a separate de-biased predictor by learning from the mistakes of a biased predictor.
EIIL (Creager et al., 2021) infers the environment information from an ERM predictor and uses group DRO to promote robustness across the latent environments.
However, these methods still access bias annotations on the validation data for model selection.
With thousands of validation examples ( for Waterbirds, 19867 for CelebA, 82462 for MNLI), a simple baseline was overlooked by the community: learning a bias predictor over the validation data (where bias annotations are available) and using the predicted bias attributes on the training data to deﬁne groups for group DRO.
Group DRO with splits identiﬁed by ls We consider the general setting where biases are not known during both training and validation.
To obtain a robust model, we take the splits identiﬁed by ls (Section 4.2) and use them to deﬁne groups for group DRO.
For example, we have four groups in CelebA: { blond hair , z = 0 } , { blond hair , z = 1 } , { no blond hair , z = 0 } , { no blond hair , z = 1 } .
For model selection, we apply the learned Splitter to split the validation data and measure the worst-group accuracy.
Results Table 1 presents our results on de-biasing.
We ﬁrst see that when the bias annotations are available in the validation data, the missing baseline Group DRO (with supervised bias predictor) outperforms all previous de-biasing methods (4.8% on average).
This result is not surprising given the fact that the bias attribute predictor, trained on the validation data, is able to achieve an accuracy of 94.8% in Waterbirds (predicting the spurious background), 97.7% in CelebA (predicting the spurious gender attribute) and 99.9% in MNLI (predicting the presence of negation words).
When bias annotations are not provided for validation, previous de-biasing methods (tuned based on the average validation performance) fail to improve over the ERM baseline, conﬁrming the ﬁndings of Liu et al.
(2021a).
On the other hand, applying group DRO with splits identiﬁed by ls consistently achieves the best worst-group accuracy, outperforming previous methods by 23.4% on average.
While we no longer have access to the bias annotations for model selection, the worst-group performance deﬁned by ls can be used as a surrogate (see Appendix C for details).
D ISCUSSION
Section 4 shows that ls identiﬁes non-generalizable splits that correlate with human-identiﬁed biases.
However, we must keep in mind that bias is a human-deﬁned notion.
Given the set of inputlabel pairs, ls provides a tool for understanding potential biases, not a fairness guarantee.
If the support of the given dataset doesn’t cover the minority groups, ls will fail.
For example, consider a dataset with only samoyeds in grass and polar bears in snow (no samoyeds in snow or polar bears in grass).
ls will not be able to detect the background bias in this case.
We also note that poor generalization can result from label noise.
Since the Splitter makes its decision based on the input-label pair, ls can achieve high generalization gap by allocating all clean examples to the training split and all mislabeled examples to the testing split.
Here we can think of ls as a label noise detector (see Appendix D for more analysis).
Blindly maximizing the worst-split performance in this situation will enforce the model to memorize the noise.
Another limitation is running time.
Compared to empirical risk minimization, ls needs to perform second-order reasoning, and this introduces extra time cost (see Appendix C for more discussion).
Finally, in real-world applications, biases can also come from many independent sources (e.g., gender and race).
Identifying multiple diverse splits will be an interesting future work.
C ONCLUSION
We present Learning to Split ( ls ), an algorithm that learns to split the data so that predictors trained on the training split cannot generalize to the testing split.
Our algorithm only requires access to the set of input-label pairs and is applicable to general datasets.
Experiments across multiple modalities conﬁrm that ls identiﬁes challenging splits that correlate with human-identiﬁed biases.
Compared to previous state-of-the-art, learning with ls -identiﬁed splits signiﬁcantly improves robustness.