A Reduction to Binary Approach for Debiasing Multiclass Datasets
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We propose a novel reduction-to-binary (R2B) approach that enforces demographicparity for multiclass classiﬁcation with non-binary sensitive attributes via a reduction to a sequence of binary debiasing tasks.
We prove that R2B satisﬁes optimalityand bias guarantees and demonstrate empirically that it can lead to an improvementover two baselines: (1) treating multiclass problems as multi-label by debiasinglabels independently and (2) transforming the features instead of the labels.
Surprisingly, we also demonstrate that independent label debiasing yields competitiveresults in most (but not all) settings.
We validate these conclusions on syntheticand real-world datasets from social science, computer vision, and healthcare
Introduction
Several studies have demonstrated that predictors are susceptible to unintended bias – for example,deep neural networks (DNNs) in both vision and language domains can amplify spurious correlationsin the training data (Hendricks et al., 2018;
Bolukbasi et al., 2016; Caliskan et al., 2017; Zhao et al.
,2017; Yang et al., 2020b; Wang et al., 2020b; Stock and Cisse, 2018).
Moreover, error disparities canarise, where the performance of the model for minority groups is disproportionately worse than forthe majority (Zhao et al., 2017;
Buolamwini and Gebru, 2018; Deuschel et al., 2020).
Studies, suchas Buolamwini and Gebru (2018) and Wang et al.
(2020a), observe that one source of this disparityis that datasets may reﬂect societal stereotypes, thereby highlighting the importance of debiasingdatasets.
Nevertheless, despite the proliferation of research on algorithmic fairness in recent years, very fewmethods exist that can handle multiclass classiﬁcation tasks with non-binary sensitive attributes.
This gap is particularly noteworthy given that this setting is the norm in real-world applications, notthe exception.
Considering multiclass classiﬁcation, out of the 70+ image classiﬁcation datasetsin the TensorFlow Dataset catalog (Abadi et al., 2015), less than 10% are for binary classiﬁcationproblems.
Along similar lines, considering non-binary sensitive attributes, the seven protectedattributes according to the US Equal Credit Opportunity Act of 1974 (U.S. Government Publishing
Ofﬁce, 1974) are non-binary, such as gender, race, religion, and age.
Two approaches are available to handle such a broad setting, to the best of our knowledge.
One optionis to view the multiclass problem as multi-label and debias every label separately before normalizingthe output into a valid probability distribution.
However, this approach lacks fairness guarantees sincenormalizing the output can re-introduce bias (see Figure 1 for a cartoon illustration).
Second, one candebias the instance features instead of the labels irrespective of the number of classes, such as usingthe demographic parity remover (Feldman et al., 2015), which debiases every feature separately ina rank-preserving manner.
However, debiasing the features can be suboptimal when the number offeatures is large as we demonstrate in Section 4.
Note, in particular, that the latter approach requiresa good estimate of the true cumulative distribution function (CDF) for each feature, but the uniform s =    ,
Y =       0  
 debias −→ ˆ Y =       12  
 normalize −→ ˜ Y =       25  
 rate of convergence of the multivariate empirical CDF to the true CDF decreases as the numberof features grows (Naaman, 2021).
We explore the performance of this approach and illustrate in
Appendix B how it can fail when using, for example, equal-width binning to estimate the CDFs.
To address this gap in the literature, we propose a reduction-to-binary (R2B) approach for debiasingmulticlass datasets that can accommodate an arbitrary number of classes and groups and does notrequire access to the sensitive attribute at prediction time.
The proposed algorithm is based on thealternating direction method of the multipliers (ADMM)
(Boyd et al., 2011), which is a frameworkfor decomposing optimization problems into a sequence of parallel tasks.
Using ADMM, we showthat the task of debiasing multiclass datasets reduces to a sequence of parallel debiasing jobs oneach class separately, along with an aggregation step.
Each debiasing job can be executed usingthe randomized threshold optimizer (RTO) algorithm of Alabdulmohsin and Lucic (2021), whichwas originally proposed as a post-hoc rule for binary classiﬁcation – in this work, we adopt itas a preprocessing method.
The overall algorithm inherits the guarantees of ADMM includingconvergence and optimality.
Our empirical results demonstrate that the proposed algorithm can leadto an improvement over the two outlined baselines; i.e. treating multiclass problems as multi-label,and transforming the features instead of the labels.
Surprisingly, we also demonstrate that the baseline multi-label approach yields competitive resultsin most (but not all) settings despite the potential impact of normalization on bias (cf.
Figure 1).
In
Appendix A, we provide an argument for why this can happen under idealized assumptions.
Statement of Contribution.
This work addresses a gap in the machine learning literature ondemographic parity for multiclass classiﬁcation with non-binary sensitive attributes.
Our contributionsare: (1) we derive a method for debiasing multiclass datasets with categorical sensitive attributesof arbitrary cardinality with respect to demographic parity – our approach reduces to a sequence ofdebiasing tasks on binary labels, (2) we establish theoretical guarantees for the proposed algorithm,(3) we study the impact of the experiment settings (e.g. number of features, number of classes, etc) onthe debiasing algorithms using synthetic data and validate different debiasing methods on real-worlddatasets from three domains: social science, computer vision, and healthcare, and (4) we evaluatethe baseline multi-label approach in debiasing multiclass datasets and demonstrate that it yieldscompetitive results in most (but not all) settings.
Related Work
In the binary classiﬁcation setting, several algorithms have been proposed for mitigating bias inmachine learning.
These are often classiﬁed into three groups depending on which step in the machinelearning pipeline the debiasing effort takes place.
First, there are preprocessing methods that areapplied prior to training, such as by learning a fair representation (Zemel et al., 2013;
Lum and
Johndrow, 2016; Bolukbasi et al., 2016; Calmon et al., 2017; Madras et al., 2018) or re-weightingtraining examples (Kamiran and Calders, 2012).
One example of a preprocessing method thattransforms the labels is the optimized score transformation (OST) method (Wei et al., 2019) whereasthe DP remover method of Feldman et al.
(2015) transforms the features.
Feature debiasing has theadvantage of not depending on the labels and can therefore be applied in any task setting.
Second, in-processing methods intervene during training, such as by adjusting the gradient updates(Zhang et al., 2018) or by adding explicit constraints into the optimization problem; e.g. (Zafaret al., 2019).
In Agarwal et al.
(2018), it is shown that many fairness criteria can be enforced duringtraining via a reduction approach to cost-sensitive classiﬁcation rules.
Our proposed algorithm for themulticlass setting also reduces to a sequence of debiasing rounds.
However, every round decomposesinto multiple parallel debiasing jobs for each label separately (i.e. using algorithms for debiasingbinary labels) and we operate in the preprocessing setting.
Third, many post-processing methods have been proposed in the literature (Corbett-Davies et al.
,2017; Menon and Williamson, 2018; Celis et al., 2019; Kamiran et al., 2012;
Hardt et al., 2016;
Wei et al., 2019; Alabdulmohsin and Lucic, 2021).
One recent example is the randomized thresholdoptimizer (RTO) of Alabdulmohsin and Lucic (2021), which can provably approximate the optimalunbiased predictor (i.e. it is statistically consistent) and can be solved efﬁciently via stochasticgradient descent (SGD).
Our ADMM based approach for debiasing multi-class datasets utilizes RTOto debias every label separately before aggregating results.
To our knowledge, two methods have been developed for multiclass classiﬁcations.
First is thealgorithm of Denis et al.
(2021), which assumes that the sensitive attribute is binary, whereas thesensitive attribute can be non-binary in our setting.
Second, Yang et al.
(2020a) propose both inprocessing and post-processing procedures; the latter of which requires access to the sensitive attributeat prediction time and cannot be directly applied as a preprocessing method on discrete labels.
Advantages of R2B.
The proposed ADMM-based reduce-to-binary (R2B) algorithm is a preprocessing method.
This offers three immediate advantages.
First, it is agnostic to the choice ofthe training algorithm; unlike, for example, in-processing methods that are often designed with aspeciﬁc model and a choice of training method in mind.
Second, R2B does not require access tothe sensitive attribute at prediction time, which is a critical advantage over post-processing methods(Zafar et al., 2019).
Third, R2B reduces the debiasing task to a sequence of rounds of debiasingthe labels, not training classiﬁers.
The computational overhead in our approach is often negligiblecompared to in-processing methods that provide a reduction approach to a sequence of classiﬁcationrules, such as in Agarwal et al.
(2018), in which the entire model is trained several times.
Unlike themulti-label approach, R2B is guaranteed to debias the dataset up to the prescribed bias tolerance level.
In addition, R2B performs better than other baselines, such as preprocessing the features instead ofthe labels.
Importantly, it can be applied in the multiclass setting with a non-binary sensitive attribute.
Reduction to Binary Method
Notation.
We reserve boldface letters for random variables (e.g. x ), small letters for instances (e.g.x ), capital letters for matrices (e.g. X ), and calligraphic typeface for sets (e.g. the instance space X ).
If f : X → R n is a multivariate function, we write f k ( x ) :
X → R for the k -th component of f .
Wedenote [ n ] = { 0 , 1 , . . .
, n − 1 } and reserve η k ( x ) for the Bayes regressor: η k ( x ) = p ( y = k
| x = x ) .
We assume that the sensitive attribute is known at training time and it has a ﬁnite range.
We denotethe instance space X , the dataset D : |D| = N , the target set Y = { 0 , 1 , . . .
, L − 1 } and thesensitive attribute g : X → S where S =
[ R ] .
We write X S for the portioning of X induced by
S ; i.e. X S =
{X 0 , . . .
, X R − 1 } is the set of groups/subpopulations where X =
∪ R − 1 s =
X s .
We willoccasionally write x i for the i -th instance in the training dataset.
We also denote 1 ∈ R n for thevector of all 1’s and 0 ∈ R m for the vector of all zeros.
In both cases, the dimension is implicit andshould be inferred with ease from the context.
Finally, || X
|| F is the Frobenius norm of the matrix X .
3.1 Multiclass Demographic Parity
Deﬁnition.
Before deriving the reduction-to-binary (R2B) method, we ﬁrst describe how demographic parity is extended to the multiclass setting.
In the binary classiﬁcation setting, demographicparity measures the difference in mean outcomes across groups.
More precisely, let f : X → [0 , 1]be a binary predictor that outputs a probability score f
( x ) = p ( y = 1 | x = x ) .
Then, f is said tosatisfy (cid:15) demographic parity if the following holds (Dwork et al., 2012;
Zafar et al., 2017; Mehrabiet al., 2019; Alabdulmohsin and Lucic, 2021):
[Equation]
where g : X → S is the sensitive attribute.
A crowd-sourcing study found that DP matches with thecommon perception of bias (Srivastava et al., 2019).
In the multiclass setting, let f k : X → [0 , 1] bethe probability score assigned to the class k ∈ Y given the instance x ∈ X and let f : X → [0 , 1]
Lbe the multivariate function f ( x ) =
( f 0 ( x ) , f 1 ( x ) , . . .
, f L − 1 ( x )) T .
In this work, we follow thedeﬁnition used in Denis et al.
and say that f satisﬁes (cid:15) demographic parity if:
[Equation]
Hence, predictors with small demographic parity have similar mean outcomes across all groups Swith respect to all of the predicted targets.
We take the maximum here instead of the average toavoid pitfalls that can arise when some classes are more preferred than others, which is analogousto the “fairness gerrymandering” phenomenon, where predictors may exhibit small bias across theintersection of groups on average , but not at the worst-case intersection (Kearns et al., 2018).
Here,instead of disparities across groups, we highlight worst-case disparities across classes .
In Appendix
C, we discuss some scaling effects of the number of classes on the deﬁnition of multiclass DP.
3.2 Derivation of the Reduction-to-Binary (R2B) Algorithm
In the binary classiﬁcation setting where Y = { 0 , 1 } , a natural measure of performance is the 0-1misclassiﬁcation error rate.
Minimizing the expected 0-1 error is equivalent to minimizing the linearfunctional E [ˆ y ( x )(2 η 1 ( x ) − 1)]
(Alabdulmohsin and Lucic, 2021), where ˆ y are the new debiasedlabels and η 1 ( x ) = p ( y = 1 | x = x ) is the Bayes regressor.
Here, the minimization is with respectto the debiased labels ˆ y : X → R L .
Hence, if access to the Bayes regressor is available at trainingtime, one can usually compute ˆ y ( s ) that minimizes the expected 0-1 loss subject to the desiredfairness constraints by solving a convex optimization problem; cf.
(Menon and Williamson, 2018;
Alabdulmohsin and Lucic, 2021; Celis et al., 2019; Wei et al., 2019).
Unfortunately, such an advantage no longer holds in the multiclass setting because the top-1 accuracyis often insufﬁcient.
For top-k accuracy, however, the squared loss is statistically consistent (Yangand Koyejo, 2020).
Hence, we propose to minimize the (cid:96) 2 distance E [(ˆ y ( x ) − y ( x ) 2 )] betweenthe debiased predictions and the original labels.
Throughout the sequel, y ( x ) : X → R L gives theprobability scores assigned to the different classes for the instance x ∈ X , which can be a degeneratedistribution as is often the case.
The objective function E [(ˆ y ( x ) − y ( x ) 2 )] can be written as:
[Equation]
where λ = 1 .
Rewriting the objective function in the form (3) provides an alternative interpretation:by minimizing the objective in (3), one seeks a solution that minimizes the regularized top-1 error.
This holds because the probability of misclassifying an example x equals ˆ y ( x ) T y ( x ) when y k ( x )
=
p ( y = k
| x = x ) .
Regularization, however, discourages extreme predictions.
We ﬁx λ = 1 since itcorresponds to minimizing the squared loss, although R2B can handle any λ > 0 .
The task of debiasing labels in a dataset D can hence be cast into the convex optimization problem:
[Equation]
Instead of solving this optimization problem directly
[Equation]
we re-express it into a more convenient form:
[Equation]
We introduced a dummy constraint ˆ y ( x ) ≤ 1 and replaced 1 T ˆ y ( x ) = 1 with a term I F (ˆ y ( x )) in theobjective function where:
[Equation]
Inputs where Y ik = p
y
=
k i training example g
[ R ] N g ( i ) is the sensitive the i -th example
Output subject
Training: Set Y (
= Y and Z stopping
Set F ( t ) =
Y + τ ( Z ( t ) − U ( t ) ) .
Let f ( t ) k ∈
R
N be the k -th column of F ( t ) .
Solve the following debiasing task for each class separately (e.g. using RTO (Alabdulmohsin and Lucic, 2021)):
[Equation]
[Equation]
Clearly, the optimization problem (5) is equivalent to the original optimization problem (4).
Algorithm 1 presents pseudocode for the proposed reduction-to-binary (R2B) method for solvingthe optimization problem in (5).
At a high-level, the algorithm decomposes the debiasing task into asequence of rounds.
In each round, the probability scores assigned to different classes are debiasedindependently .
This produces a new set of scores, denoted H ( t ) , that are not normalized.
The nextstep is to normalize them so that they sum to one.
The set of normalized probability scores is given by
Z ( t ) .
After that, the objective function is altered slightly and the process is repeated.
We prove that
Algorithm 1 returns the optimal solution to (5) and discuss suitable stopping criteria in Section 3.3.
3.3 Analysis of the Algorithm
Proposition 1 (Optimality) .
Algorithm 1 terminates with an optimal solution to (5), in which thedebiased probability scores assigned to the i -th example are given by the i -th row of the matrix ˆ Y .
Stopping Criteria.
The simplest stopping criterion to use in Algorithm 1 is the number of ADMMrounds.
Boyd et al.
(2011) observes that 50-100 rounds are often sufﬁcient, which we also observe tobe true in our experiments (see Figure 3).
Alternatively, one may use the sum of the “primal" and“dual residuals".
In Algorithm 1, the optimality conditions of ADMM (Boyd et al., 2011) reduce to thefollowing conditions Z ( t ) =
H ( t ) and Z ( t +1)
=
Z ( t ) .
Hence, we may deﬁne δ p =
|| Z ( t ) − H ( t ) || Fand δ r = ||
Z ( t +1) − Z
( t ) || F and stop when δ p + δ r are below a prescribed threshold.
In ourimplementation, we follow the ﬁrst approach and set the maximum number of ADMM rounds to 100.
Bias Guarantee.
For our next result, we write ˆ E [ f ( x )] for the average value of a function f on thetraining examples, and write E [ f ( x )] for the expectation of f over the true distribution of instances.
Proposition 2 (Bound on bias) .
Let P L be the probability simplex in R L and F be a class of functionsfrom X to P L , such that the set { f k : f ∈ F} has a Rademacher complexity d .
Suppose that alltraining examples ( x , y , s ) ∈ D are drawn i.i.d.
and are debiased using Algorithm 1 with biaslevel (cid:15) ≥ 0 .
Let h ( s ) :
X → P L be the debiased labels.
Let f ∈ F be the ﬁnal classiﬁer.
Then, with a probability of at least 1 − δ , we have: DP ( f ) ≤ (cid:15) + 2 τ + 2 d + (cid:113) log 2 LRδ 2 n 0 , where n 0 =
min s ∈
[ R ] (cid:12)(cid:12)(cid:8) I { g ( x ) =
s } : x ∈ D (cid:9)(cid:12)(cid:12) is the size of the smallest group in the training dataset,
L = |Y|is the number of classes, R = |S| is the number of groups, and τ = sup k ∈Y ,s ∈
[ R ] (cid:12)(cid:12)
ˆ E [ f k ( x )
| g ( x ) =
s ] − ˆ E
[ h k ( x ) | g ( x ) =
s ] (cid:12)(cid:12) , where both expectations are measured on the training dataset.
Proposition 2 provides a formal justiﬁcation to the preprocessing approach; it states that if the traininglabels are debiased using Algorithm 1, then a classiﬁer trained on the debiased data is guaranteedto exhibit small bias in the future as long as four conditions are satisﬁed: (1) the level of bias in thetraining data is small; i.e. (cid:15) (cid:28) 1 in Algorithm 1, (2) the classiﬁer ﬁts the training examples well; i.e.τ (cid:28) 1 , (3) the complexity of the classiﬁer is not too large (does not memorize examples); i.e. d (cid:28) 1 ,and (4) there exists a large number of training examples for each group in S ; i.e. n 0 (cid:29) log LR .
Experiments
In this section, we compare the reduction-to-binary (R2B) method against the two baselines: (1)treating multiclass datasets as multi-label, and (2) transforming the features instead of the labels.
Inthe multi-label approach, we use two recent algorithms for debiasing binary labels: the optimizedscore transformation (OST) method of Wei et al.
and the RTO method of Alabdulmohsin and
Lucic (2021) with γ = 0 .
01 and ρ = E
[ y k ] (default) values.
After that, the scores are normalizedto sum to one.
Note that a small value of γ in the RTO algorithm corresponds approximately toa hard -thresholding rule, hence we refer to it as hard-threshold optimizer (HTO).
In the secondapproach, we apply the demographic parity remover (DPR) method of Feldman et al.
(2015), whichdebiases features in a rank-preserving manner.
In the latter case, we use equal-mass binning whenestimating the cumulative density function (see Appendix B).
All three baselines can be motivatedvia provable optimality guarantees (Wei et al., 2019; Alabdulmohsin and Lucic, 2021; Feldman et al.,2015).
We also include for comparison training without debiasing the data, which we denote as BL 1 .
We note that existing multiclass fairness approaches such as Yang et al.
(2020a) cannot be applied forpre-processing as they require either probability estimates or weighted classiﬁers.
In R2B, on the other hand, we use a step size of τ = 0 .
5 in Algorithm 1 and a maximum number of100 ADMM rounds.
We also report results when a single round of R2B is used, which we denotedas R2B 0 .
Except for the healthcare dataset whose splits are ﬁxed, we split data at random into 25%for test and 75% for training.
We debias the dataset prior to training and measure performance (e.g.accuracy and DP) on the test split.
All methods use the same splits.
We re-run every experiment withten different random seeds and report both the averages and 99% conﬁdence intervals.
[Table]
4.1 Synthetic Dataset
We begin our analysis with synthetic data.
In this dataset, let L be the number of classes and d bethe number of features.
We use a mixture of L Gaussians whose means are sampled from N (0 , σ 2 I )with σ 2 = 1 /d
and Y =
[ L ] .
For the sensitive attribute, we set it equal to I { y = 0 } with probability1 / 2 ; otherwise, it is chosen uniformly at random from the set { 0 , . . .
, 4 } ; i.e. |S| = 5 .
Hence, biasis introduced into the dataset.
The classiﬁers are k -NN with k = 5 and Random Forest (RF) withmaximum depth 5, both implemented using Scikit-Learn (Pedregosa et al., 2011).
We vary thenumber of classes in { 4 , 16 , 64 } , the number of features in { 10 , 100 } , and the number of trainingexamples per class in { 1000 , 2000 , 4000 } .
We set the bias tolerance (cid:15) to zero in all experimentssince, otherwise, these methods provide incomparable mechanisms for controlling bias.
Figure 2displays the results for k -NN and Appendix G contains the full ﬁgures.
As shown in Figure 2, R2B provides a stronger fairness guarantee than the other competing methods.
In addition, we observe that debiasing the features performs poorly when the number of features islarge.
As mentioned earlier, one possible explanation is that as the number of features grows, therate of convergence of the multivariate empirical cumulative distribution function (EDF) to the truecumulative distribution function (CDF) decreases (Naaman, 2021).
However, debiasing the featuresin Feldman et al.
(2015) relies on having a correct estimate of the CDF since it matches quantiles.
Moreover, we observe that the multi-label approach (i.e. OST, HTO, and R2B0) can also fail inseveral cases.
As illustrated in the cartoon example of Figure 1, this is because normalization impactsbias.
In one experiment, for example, the level of DP in the training labels is reduced to less than 0.01after debiasing labels separately using OST but bias increases to over 0.22 following normalization.
In Figure 3, we also plot the level of DP in the training labels as a function of the number of
ADMM rounds in R2B for ﬁve different levels of bias tolerance (cid:15) ∈ { 0 , 0 . 025 , 0 .
We observe that R2B converges in about 50 ADMM iterations, which is consistent with earlierobservations in the literature (Boyd et al., 2011).
In addition, when the level of bias in the data issmaller than (cid:15) , the algorithm does not introduce any additional bias as expected.
[Table]
4.2 Real-world Applications
Next, we validate R2B on applications from three domains: (1) social science, (2) computer vision,and (3) healthcare.
Experiments involving neural networks are executed on Tensor Processing Units.
Adult Income dataset.
The ﬁrst dataset is the Adult Income dataset (Kohavi, 1996).
In our case,we predict the education level of each individual from the remaining attributes.
There are 16 classesin this dataset and 29 features, such as marital status, age, and occupation.
The total number ofexamples (both training and test) is 48,842.
The sensitive attribute is sex and only includes twocategories (‘Male’, ‘Female’).
We compare the debiasing methods in Table 1 for both the Random
Forest (RF) and k -NN classiﬁers (top two rows).
Besides the original dataset, we also introduce biasand class imbalance into the data (both training and test) and compare methods: with probabilityp = 0 .
5 , the label y ∈ { 0 , 1 , . .
, 15 } is set to be equal to the sensitive attribute s ∈ { 0 , 1 } .
Thisincreases bias in the original labels from about 0.05 to around 0.48 and introduces
class imbalanceas
well 2 .
The performance of each debiasing method in this dataset is shown in Table 1, bottomtwo rows.
In most cases, R2B provides a stronger bias guarantee compared to the other methods.
Appendix F provides similar performance results for Top-K accuracy.
COCO dataset.
The second dataset we use is the COCO dataset (Lin et al., 2014).
It contains80 classes corresponding to the objects in each image, such as chairs, cars, and handbags.
Wetransform this multi-label problem into a multiclass problem using soft labels: we set the target labely to be equal to the fraction of the objects that belong to each class in each image.
The soft labelapproach corresponds formally to the task of predicting the distribution of objects seen in the imageor, equivalently, the task of predicting the class of an object chosen uniformly at random from thecorresponding image.
We follow the procedure of (Wang et al., 2020a) in inferring the sensitiveattribute based on the image caption: we use images that contain either the word “woman" or theword “man" in their captions but not both.
The total number of examples is 22,616.
Because the taskhere is to predict the distribution of classes seen in the image, we deﬁne accuracy in terms of the totalvariation distance between probability distributions.
Speciﬁcally, let ˆ y be the model’s prediction, thenacc(ˆ y , y )
= 1 − (1 / 2)
||
ˆ y −
y
|| 1 .
Note that 0 ≤ acc(ˆ y , y ) ≤ 1 .
See Appendix H for further detailsabout the training procedure.
Besides the original dataset, we also introduce bias and class imbalancewith p = 0 .
5 as described previously.
Results are provided in Table 2.
As shown in the table, both
R2B and the multi-label methods perform much better than transforming the features (DPR), whichis consistent with the earlier observations on synthetic data.
Moreover, R2B performs better overall.
Dermatology.
In this task, we are interested in predicting 27 skin conditions (26 plus an ‘other’label representing the long tail) from images of the pathology of interest, the patient’s age and sex 3 .
The dataset is a subset of the one used in Liu et al.
and consists of de-identiﬁed retrospectiveadult cases from a teledermatology service serving 17 sites from 2 states in the United States.
Itis split according to condition prevalence for training ( n = 12 , 024 ), tuning for hyper-parameters( n = 1 , 925 ) and hold-out testing ( n = 1 , 924 ).
We train a deep learning model to predict skinconditions as a multiclass task, as per the architecture described in Liu et al.
Roy et al.
For debiasing, we ﬁt R2B on the tune split and assess the model performance.
Following Liu et al.(2020), we report the top-1, top-2, and top-3 accuracy on the test split.
Table 3 reports DP on the
[Table] [Table]
[Table] [Table]
test split as well.
For ease, we bucket age according to [18,30), [30,45), [45, 65) and [65,90) fordebiasing.
We note that the number of cases per condition and per intersection of the attributes mightbe low, as is common in data-scarce domains such as healthcare.
Results are presented in Table 3.
Summary of Findings.
The reduction-to-binary (R2B) approach performs at least as well as theother baselines in all of the experiments and outperforms the other methods in some cases, such asin the synthetic data, the Adult Income dataset, and COCO.
In addition, the multi-label approachusing OST and R2B 0 seem to offer competitive results in most (but not all) settings.
In Appendix
A, we provide an argument for why this might happen under idealized assumptions.
However, it isworth emphasizing that the multi-label approach is not guaranteed to debias datasets successfully asillustrated earlier in Figure 1 and demonstrated using synthetic (Figure 2) and real (Table 1) data.
Onother other hand, R2B offers strong guarantees (cf.
Propositions 1 and 2).
Discussion and Limitations
In this paper, we derive a reduction approach for debiasing multiclass datasets according to demographic parity (DP).
The algorithm reduces the overall task into a sequence of parallel debiasing jobson binary labels.
Due to this reduction, the algorithm scales well to large datasets with several classes.
We verify empirically on both synthetic and real-world datasets that it outperforms other baselines.
Nevertheless, it is worth emphasizing that “fairness" is a societal concept and should not be reducedto statistical metrics, such as DP (Dixon et al., 2018; Selbst et al., 2019).
As such, the claims of thispaper hold for the narrow technical deﬁnition of DP, not for fairness in its broader sense.
One limitation in R2B is that it accommodates DP but cannot accommodate error parity metrics,such as equalized odds.
This is because it operates in the pre-processing setting, where prediction“errors" are yet undeﬁned.
In particular, for anti-causal predictive tasks (Schölkopf et al., 2012) as isthe case in dermatology (Castro and Glocker, 2020), Veitch et al.
(2021) suggest that equalized oddswould be a causally-grounded fairness constraint.
Nevertheless, in real-world settings, we do notobserve an increase in error parity when the training data is debiased to account for DP as shown in Table 4, except when bias is introduced into the data (i.e. with p = 0 . 5) ).
One potential remedy is totransform the algorithm into a post-processing method, but this requires a method of transforming therule into the inductive setting.
Another limitation of the proposed algorithm is that it accommodatescategorical sensitive attributes only, not continuous.
While the majority of protected attributes areindeed categorical and continuous attributes can be converted into categorical features by binningtheir values (e.g. age in the dermatology example), handling continuous sensitive attributes withoutresorting to binning remains a challenge.
We leave such questions for future work. Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] We provide
technical deﬁnition of demographic parity as a deﬁnition of fairness in its broader sense
We caution against this in Section 5.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] Please see AppendicesD and E.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] Code has beenmade publicly available but is not disclosed to preserve anonymity.
Both the Adult andCOCO datasets are publicly available.
The dermatology dataset is not available to thepublic.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] Instructions for reproducing experimental results are provided inthe main paper (see Section 4) and the supplementary materials (see Appendix H).
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] We report error bars on all results.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of
GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.
Experiments involving
deep neural networks are executed on TPUs.
The training protocol is described in the
paper and in Appendix H (for COCO dataset).
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cite software
packages (e.g. Scikit-Learn and TensforFlow), datasets (e.g. ImageNet, Dermatology,
COCO, Adult), and cite the models we use (e.g. ResNets).
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We release an implementation of the proposed algorithm.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No] For the dermatology dataset, this has been discussed in the
original paper (Liu et al., 2020).
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[Yes] The dermatology dataset consists of de-
indentiﬁed retrospective adult cases as described in Section 4.
If you used crowdsourcing or conducted research with human subjects.
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]