On Learning Fairness and Accuracy on Multiple Subgroups
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We propose an analysis in fair learning that preserves the utility of the data whilereducing prediction disparities under the criteria of group sufﬁciency.
We focus onthe scenario where the data contains multiple or even many subgroups, each withlimited number of samples.
As a result, we present a principled method for learninga fair predictor for all subgroups via formulating it as a bilevel objective.
In thelower-level, the subgroup-speciﬁc predictors are learned through a small amountof data and the fair predictor.
In the upper-level, the fair predictor is updated tobe close to all subgroup speciﬁc predictors.
We further prove that such a bilevelobjective can effectively control the group sufﬁciency and generalization error.
We evaluate the proposed framework on real-world datasets.
Empirical evidencesuggests the consistently improved fair predictions, as well as the comparableaccuracy to the baselines.
Introduction
Machine learning has made rapid progress in sociotechnical systems such as automatic resumescreening, video surveillance, and credit scoring for loan applications.
Simultaneously, it has beenobserved that learning algorithms exhibited biased predictions on the subgroups of population [1, 2].
For example, the algorithm denies a loan application based on sensitive attributes 1 such as gender,race, or disability, which has heightened public concerns.
To this end, fair learning is recently highlighted to mitigate prediction disparities.
The high-levelidea is quite straightforward: adding fair constraints during the training [3].
As a result, fair learningprincipally gives rise to two desiderata.
On the one hand, the fair predictor should be informative toensure accurate predictions for the data.
On the other hand, the predictor is required to guaranteefairness to avoid prediction disparities across subgroups.
Therefore, it is crucial to understand thepossibilities and then design provable approaches for achieving both informative and fair learning.
Clearly, achieving both objectives depends on predeﬁned fair notations.
Consider demographic parity[1] as the fair criteria, which necessitates the independence between the predictor’s output f ( X ) andthe sensitive attribute A .
Thus, if the sensitive attribute A and the ground-truth label Y are highlycorrelated, it is impossible to learn a both fair and informative predictor.
To avoid such intrinsic impossibilities, alternative fair notions have been developed.
In this work,we focus on the criteria of group sufﬁciency [1, 4], which ensures that the conditional expectationof ground-truth label ( E [ Y | f ( X ) , A ] ) is identical across different subgroups, given the predictor’soutput.
Notably, the risk of violating group sufﬁciency has arisen in a number of real-world scenarios.
E.g., in medical artiﬁcial intelligence, the machine learning algorithm is used to assess the clinic risk, and guide decisions regarding initiating medical therapy.
However, [5, 6] revealed a signiﬁcant racialbias in such algorithms: when the algorithm predicts the same clinical risk score f
( X ) for white andblack patients, black patients are actually at a higher risk of severe illness: E [ Y | f ( X ) , A = black ] (cid:29)
36 E
[ Y | f ( X ) , A = white ] .
The deployed algorithms have resulted in more referrals of white patients tospecialty healthcare services, resulting in both spending disparities and racial bias [5].
In summary, this work aims to propose a novel principled framework for ensuring group sufﬁciency,as well as preserving an informative prediction with a small generalization error.
In particular, wefocus on one challenge scenario: the data includes multiple or even a large number of subgroups,some with only limited samples , as often occurs in the real-world.
For example, datasets for theself-driving car are collected from a wide range of geographical regions, each with a limited numberof training samples [7].
How can we ensure group sufﬁciency, as well as accurate predictions?
Speciﬁcally, our contributions are summarized as follows:
[Figure]
The proposed algorithm is then formulated as a bilevel optimization to learn an informative and fair predictive-distribution Q .
In the lower-level (cyan), we learn the subgroup speciﬁc predictivedistribution Q (cid:63)a from dataset S a (limited samples) and the prior
Q .
In the upper-level (brown), Q is then updated to be as close to all of the learned subgroup speciﬁc Q (cid:63)a as possible.
Controlling group sufﬁciency We introduce group sufﬁciencygap to measure fairness w.r.t.
group sufﬁciency of a classiﬁer f(Sec.2), and then derive an upper bound of the group sufﬁciencygap (Theorem 3.2).
Under proper assumptions, the upper boundis controlled by the discrepancy between the classiﬁer f and thesubgroup Bayes predictors.
Namely, minimizing the upper boundcould also encourage an informative classiﬁer.
Algorithmic contribution
Motivated by the upper bound ofthe group sufﬁciency gap, we develop a principled algorithm.
Concretely, we adopt a randomized algorithm that produces apredictive-distribution Q over the classiﬁer ( f ∼ Q ) to learnthe informative and fair classiﬁcation.
We further formulate theproblem as a bilevel optimization (Sec. 4.3), as shown in Fig.1.(1)
In the lower-level, the subgroup speciﬁc dataset S a and the fairpredictive-distribution Q are used to learn the subgroup speciﬁcpredictive-distribution Q (cid:63)a , where Q is regarded as an informativeprior for learning limited data within each subgroup.
Theorem 4.1formally demonstrates that under proper assumptions, the lowerlevel loss can effectively control the generalization error.
In theupper-level, the fair predictive-distribution Q is then updated tobe close to all subgroup speciﬁc predictive-distributions, in orderto minimize the upper bound of the group sufﬁciency gap.
Empirical justiﬁcations The proposed algorithm is applicableto the general parametric and differentiable model, where weadopt the neural network in the implementation.
We evaluatethe proposed algorithm on two real-world NLP datasets that haveshown prediction disparities w.r.t.
group sufﬁciency.
Compared with baselines, the results indicatethat group sufﬁciency has been consistently improved, with almost no loss of accuracy.
Preliminaries
We assume the joint random variable ( X, Y, A ) follows an underlying distribution D ( X, Y, A ) , where
X ∈ X is the input, Y ∈ Y is the label, and the discrete random variable A ∈
A denotes the sensitiveattribute or subgroup index.
For instance, A represents gender, race, or age.
We also denote E
[ Y | X ]as the conditional expectation of Y , which is essentially a function of X .
E A,X [ · ] is denoted as theexpectation on the marginal distribution of D ( A, X ) .
Throughout the paper, we consider binaryclassiﬁcation with Y = { 0 , 1 } .
We further deﬁne the predictor as a scoring function f : X → [0 , 1]that maps the input into a real value in [0 , 1] .
It is worth mentioning that f ( X ) / ∈ Y since f ( X ) iscontinuous.
We then introduce group sufﬁciency and group sufﬁciency gap.
Deﬁnition 2.1 (Group sufﬁciency [4]) .
A predictor f satisﬁes the group sufﬁciency with respect tothe sensitive attribute
A if E [ Y | f ( X )]
=
E [ Y | f ( X ) , A ] .
Intuitively, given a output score of the predictor f ( X ) =
τ , the conditional expectation of Y isinvariant across different subgroups.
Namely, conditioning on the speciﬁc subgroup A = a does not provide any additional information about the conditional expectation of Y .
Then we could naturallydeﬁne group sufﬁciency gap.
Deﬁnition 2.2 (Group sufﬁciency gap) .
The group sufﬁciency gap of a predictor f is deﬁned as:
Suf f = E A,X [ | E [ Y | f ( X )] − E
[ Y | f ( X ) , A ] | ]
Speciﬁcally, Suf f measures the extent of group sufﬁciency violation, induced by the predictor f ,which is taken by the expectation over ( X, A ) .
For completeness, we also discuss other populargroup fairness criteria: demographic parity and equalized odds.
Deﬁnition 2.3 (Demographic Parity (DP)) .
A predictor f satisﬁes the demographic parity withrespect to the sensitive attribute A if: E [ f ( X )]
=
E [ f ( X ) | A ]
Demographic Parity (DP), also known as statistical parity or independence rule, emphasizes that theexpectation of the output score f
( X ) is independent of A .
[1, 4] further revealed that if A (cid:54)⊥⊥ Y ,group sufﬁciency and demographic parity could not be simultaneously achieved.
Deﬁnition 2.4 (Equalized Odds (EO) [8]) .
A predictor f satisﬁes the equalized odds with respect to
A if: E [ f ( X ) | Y ] =
E [ f ( X ) | Y, A ]
Equalized odds (EO) emphasizes the conditional expectation of output f is invariant w.r.t.
A , giventhe ground truth Y .
[1, 9] reveal that if D ( X, Y, A ) > 0 and A (cid:54)⊥⊥ Y , group sufﬁciency and equalizedodds can not both hold.
The analysis reveals a general incompatibility between group sufﬁciency and DP/EO when A (cid:54)⊥⊥ Y ,which often occurs in practice.
Besides, DP/EO based criteria suffer the well-known fair accuracytrade-off [10]: enforcing the fair constraint degrades the prediction performance.
This paper depictsthat under the criteria of group sufﬁciency, these objectives could be both encouraged.
Upper bound of group sufﬁciency gap
To derive the theoretical results, we ﬁrst introduce the group Bayes predictor.
Deﬁnition 3.1 ( A -group Bayes predictor) .
The A -group Bayes predictor f Bayes is deﬁned as:f Bayes
[Equation]
[Equation] [Equation] [Equation] The A -group Bayes predictor is associated with the underlying data distribution D ( X, Y, A ) .
Giventhe ﬁxed realization X = x, A = a , we have f Bayes ( x, a ) =
E [ Y | X = x, A = a ] .
We could furtherdemonstrate the following facts.
Proposition 3.1.
The A -group Bayes predictor f Bayes : (1) satisﬁes the group sufﬁciency 115 E
[ Y | f Bayes ( X, A )]
=
E [ Y | f Bayes ( X, A ) , A ] ; (2) is the optimal predictor of subgroup A = aunder the binary cross-entropy loss function.
Namely, f Bayes ( x, a ) = argmin
f L BCE a ( f ) , where L BCE a ( f ) =
E ( x,y )
∼D
( x,y | A = a ) − [ y log( f ( x ))
+ (1 − y ) log(
− f ( x ))] .
The proof is delegated to the Appendix.
Proposition 3.1 shows that the A -group Bayes predictorsimultaneously meets both fairness (group sufﬁciency) and informative (optimal predictor underbinary cross-entropy loss).
Unfortunately, f Bayes is impossible to estimate since it is related to theunderlying data distribution D , which is infeasible.
Nevertheless, we can adopt A -group Bayespredictor f Bayes to derive an upper bound of group sufﬁciency gap Suf f .
The upper bound then holdsfor any predictor f that can be learned from the observed data.
Theorem 3.2.
The group sufﬁciency gap Suf f is upper bounded by: Suf f ≤ 4 E A,X [ | f − f Bayes | ]
Speciﬁcally, if A takes ﬁnite value ( |A| < + ∞ ) and follows uniform distribution with D ( A = a ) =1 / |A| .
Then the group sufﬁciency gap is further simpliﬁed as:
[Equation]
Theorem 3.2 reveals that the upper bound of the group sufﬁciency gap depends on the discrepancy between the predictor f and A -group Bayes predictor f Bayes ( X, A ) .
Namely, given different subgroups
A = a , the optimal predictor f ought to be closed to all f Bayes ( X, A
= a ) , ∀ a ∈ A .
Underlying assumption Theorem 3.2 also suggests underlying assumptions w.r.t.
the data generationdistribution D ( X, Y, A ) for achieving a small group sufﬁciency gap.
If f Bayes for each subgroup
A = a are quite similar, then minimizing the upper bound yields a small group sufﬁciency gap
Suf f .
For example, consider the extreme scenario if the A -group Bayes predictors are identicalw.r.t.
A : E
[ Y | X, A
= a ] = E
[ Y | X ] , ∀ a ∈ A , where E [ Y | X ] is the conventional Bayes predictordeﬁned on the marginalized distribution D ( X, Y ) .
The upper bound recovers the difference betweenthe predictor f and standard Bayes predictor.
If we use a probabilistic framework to approximatepredictor f
( X ) ≈
E
[ Y | X ] (i.e, training the entire dataset without any fair constraint), both thegroup sufﬁciency gap and prediction error (since Bayes predictor is optimal) will be small, which isconsistent with [11].
On the contrary, if A -group Bayes predictors are completely arbitrary with high variance for A , bothgroup sufﬁciency gap and prediction error are large and impossible for the fair prediction.
Principled Approach
Based on the upper bound, we propose a principled approach to learn the predictor that achieves bothsmall generalization error and group sufﬁciency gap.
4.1 Upper bound in randomized algorithm
To derive the theoretical result, we consider the randomized algorithm that learns a predictivedistribution Q over the scoring predictors from the data.
For instance, if we consider Bayes framework,the predictor is drawn from the posterior distribution ˜ f ∼ Q .
In the inference, the predictor’s outputis formulated as the expectation of the learned predictive-distribution Q : f
( X ) =
E ˜ f ∼ Q ˜ f
( X ) .
In practice, it is infeasible to optimize over all the possible distributions.
Then we should restrictthe predictive-distribution Q within a distribution family Q ∈ Q such as Gaussian.
We also denote
Q (cid:63)a ∈ Q as the optimal prediction-distribution w.r.t.
A = a under binary cross-entropy loss within thedistribution family Q : Q (cid:63)a :=
argmin Q a ∈Q E ˜ f a ∼ Q a L BCE a ( ˜ f a ) .
In generally Q (cid:63)a (cid:54) =
f Bayes ( x, A = a ) ,since the distribution family Q is only the subset of all possible distributions (shown in Fig. 2).
Wethen extend the upper bound in the randomized algorithm.
Corollary 4.1.
The group sufﬁciency gap Suf f in randomized algorithm w.r.t.
learned predictivedistribution Q is upper bounded by:
[Equation]
Where KL is the Kullback–Leibler divergence.
Corollary 4.1 further reveals that the upper bound isdecomposed into two terms, showing in Fig.2.
[Equation]
to
Q ∈ Q to minimize the discrepancy between ( Q (cid:63)a
Q (cid:63)b ) is solely based on the distribution family Q (brown region).
If the predeﬁned Q has a rich expressive power, the approximation is treated as a small constant
Optimization term The optimization term is the average KLdivergence between the learned distribution Q and optimalpredictive-distribution Q (cid:63) a for each subgroup
A = a .
Minimizing the optimization term implies that the learned distribution
Q will be both fair and informative for the prediction, becauseit aims to minimize the upper bound of the group sufﬁciencygap Suf f and be close to the optimal predictive-distributionw.r.t.
each A = a .
Approximation term The approximation term is the average
KL divergence between the optimal distribution Q (cid:63)a and theunderlying data generation distribution.
Given the distributionfamily Q , it is a unknown constant.
Besides, if the distributionfamily Q has a rich expressive power such as deep neuralnetwork, the approximation term will be small [12].
However,an extreme large distribution family Q could simultaneouslyyield a potential overﬁtting on ﬁnite samples.
In this paper,the neural network is adopted and the approximation term isassumed to be a small constant.
Thus, controlling Suf f impliesminimizing the optimization term.
4.2 Challenge in learning limited samples
In practice, we only have access to ﬁnite or even limited samples, rather than the underlyingdistribution D .
We denote S a = { ( x ai , y ai ) } mi =1 as the observed data w.r.t.
subgroups
A = a , whichare i.i.d.
samplings from the underlying distribution D ( x, y |
A = a ) .
We also denote the empiricalbinary cross entropy loss w.r.t A
= a as: ˆ L BCE a ( ˜ f )
= 1 m (cid:80) mi =1 − [ y ai log
[Equation]
˜ f
( x ai ))
+ (1 − y ai ) log(
−˜ f
( x ai ))] .
Then a straight approach is to minimize the empirical term ˆ
[Equation]
[Equation]
Then Q is updated through minimizing the average KL-divergence: (cid:80) a KL ( ˆ Q (cid:63)a (cid:107) Q ) from learnedˆ Q (cid:63)a .
However, this idea generally does not work in our setting, because each subgroup containslimited number of samples.
Therefore, a direct minimization leads to overﬁtting for each subgroupand generalization error E ˜
f ∼ ˆ Q (cid:63)a L BCE a ( ˜ f ) is quite large, showing in Fig.
4.3 Q as an informative prior
We have demonstrated that Q can achieve both fair and informative prediction.
Therefore, we regard
Q as a prior information for minimizing the loss, yielding a bilevel objective.
[Equation]
level)
Where λ > 0 is the hyper-parameter.
The proposed loss is a typical bilevel optimization.
(1) In thelower-level, we aim to learn Q (cid:63)a for each a ∈ A .
Different from Eq.
(1), the loss in lower-level addsa regularization term KL ( Q a (cid:107) Q ) as an informative prior in learning Q (cid:63)a , given a ﬁxed predictivedistribution Q .
Moreover, Theorem 4.1 formally justiﬁed that optimizing the lower-level loss is tominimize the upper bound of the generalization error.
In the upper-level, Q is updated throughminimizing the average KL divergence between different Q (cid:63)a , which controls the upper bound of
Suf f .
Theorem 4.1 (Generalization error bound) .
Supposing that datasets { S a } |A| a =1 with S a ={ ( x ai , y ai ) } mi =1 are i.i.d.
sampled from D ( x, y | A = a ) , the binary cross entropy (BCE) loss isupper bounded by L , Q a ∈ Q is any learned distribution from dataset S a and Q ∈ Q is anydistribution.
Then with high probability ≥ 1 − δ with ∀ δ ∈ (0 , 1) , we have: |A|
[Equation]
Discussions Theorem 4.1 reveals the generalization error in the lower-level is upper bounded by threeterms.
(a) Term (1) is the average empirical prediction error, which corresponds to the ﬁrst term in thelower-level loss.
(b) Term (2) is the average KL-divergence between the learned subgroup distribution
Q a and the prior distribution Q , which corresponds to the second term in the lower-level loss.
Thecombination of term (1-2) recovers the averaged lower-level loss w.r.t.
A . 2 Thus optimizing thelower-level loss could control the generalization error.
(c) When the conﬁdence δ is ﬁxed, term (3)will converge if |A| m → + ∞ .
Moreover, even if m (the sample size in each subgroup) is quite small,a sufﬁcient large number of subgroups |A| can also ensure the convergence of term (3).
For the sake of simplicity, we assumed the identical samples size m in each subgroup S a , while thetheoretical result can be extended to subgroups with different samples m a .
4.4 Practical Implementations
In this section, we develop a practical learning algorithm that can be applied to a wide range ofdifferentiable and parametric models, including neural networks.
˜ f
w a ∼ Q a
L BCE a f w a ) λ KL ( Q
Parametric models We choose the Isotropic Gaussian distribution (with diagonal covariancematrix) as the distribution family Q , where the mean and covariance are set as d -dimensionalparameter.
Thus we need to learn the parameter ( θ , σ ) for fair and informative Q ∈ Q .
As for thesubgroup A = a , we learn parameters ( θ a , σ a ) for Q (cid:63)a ∈ Q .
It is worth mentioning that the Isotropic
Gaussian distribution is selected for its computational efﬁciency in the optimization.
We can use anydistribution as long as the density function is differentiable with respect to the parameters.
For the single predictor ˜ f , we use parametric neural-network models and assume ˜ f is parameterizedby a d -dimensional vector w ∈ R d , denoted as ˜ f w .
Then ˜ f w ∼ Q is equivalent to sampling the modelparameter w from the predictive-distribution Q : w ∼ N ( θ , σ 2 ) =
(cid:81) di
=1 N ( θ [ i ] , σ 2 [ i ]) .
Since Q is
Isotropic Gaussian, each element i in the parameter w
[ i ] follows a 1-dimensional Gaussian.
Followingthe same line, ˜ f w a ∼ Q (cid:63)a can be modeled analogously:
w a ∼ N
θ a , σ 2 a )
= (cid:81) di
=1 N ( θ a [ i ] , σ 2 a [ i ]) .
As a result, learning the distribution Q is equivalent to learning parameter ( θ , σ ) in the bilevelobjective.
Gradient Estimation Based on the previous setting, we aim to optimize the bilevel objectiveto obtain the parameter of Q : ( θ , σ ) .
We use stochastic gradient descent (SGD) to optimizethe parameters.
In the lower-level, the loss in Sec.
4.3 is composed by the empirical predictionerror and KL divergence term.
The KL divergence has a closed form that can be differentiatedefﬁciently.
Speciﬁcally, since Q and the subgroup speciﬁc Q (cid:63)a are factorized Gaussian, the KLdivergence takes a simple closed form and the gradient can be easily calculated: KL ( Q (cid:63)a (cid:107) Q ) =
[Equation]
Re-parametrization trick As for the prediction error E ˜
f w a ∼ Q a ˆ L BCE a ( ˜ f w a ) , the term ˆ L BCE a ( ˜ f w a )is non-linear for w , rendering the expectation intractable in the computation.
To this end, weadopt the re-parameterization trick [13, 14] in computing the gradient w.r.t.
the expectation term.
The trick is based on describing the Gaussian distribution w a ∼ N ( θ a , σ 2 a ) as ﬁrst drawing(cid:15) ∼ N ( 0 , I ) and then applying the deterministic function w a ( θ a , σ a )
=
θ a + σ a (cid:12) (cid:15) ( (cid:12) iselement-wise product) to approximate the sampling.
Then the gradient term can be estimated as:∇ ( θ a , σ a ) E w a ∼ N ( θ a , σ a ) ˆ L BCE a ( ˜ f w a ) =
∇ ( θ a , σ a ) E (cid:15) ∼N ( 0 , I )
ˆ L BCE
a ( ˜ f w a ( θ a , σ a ) ) , where the expectation 243 can be approximated by Monte-Carlo sampling w.r.t.
(cid:15) .
For a ﬁxed sample (cid:15) , the gradient can becomputed through backpropagation.
In the upper-level, the KL divergence has a closed form, thus itis easy to update the parameter of Q through backpropagation.
Proposed Algorithm Based on the analysis, the algorithm is shown in Algorithm.
1 for solvingthe bilevel objective in Sec. 4.3.
Speciﬁcally, we adopt the alternating optimization.
Namely, inthe lower-level, we ﬁx Q and optimize the subgroup speciﬁc predictor Q (cid:63)a through SGD.
Then inthe upper-level, we ﬁx the learned Q (cid:63)a and update Q .
Since we may face many subgroups, at eachtraining epoch, we randomly sample a subset A (cid:48) such that |A (cid:48) | (cid:28) |A| for the memory saving.
Inference In the inference, we use the Monte-Carlo method to sample the weights of the neuralnetwork from distribution w ∼ N ( θ , σ 2 ) , then averaging the output w.r.t.
different sampled weightsto approximate f ( x ) =
E ˜ f
w ∼ Q ˜ f w ( x )
Experiments5.1 Experimental Setup
Dataset: Amazon review We adopt Amazon review dataset [15, 16], which aims to predict thesentiment (classiﬁcation) from the review.
The datasets consist of large-scale users.
Each user haslimited number of reviews, ranging from 75 to 400.
The user is then treated as a subgroup, and ithas been observed that standard training can lead to prediction disparities in several users.[17].Theexperiment is adapted from the protocol of [16].
Speciﬁcally, we convert the original review score(ranging from 1-5) to the binary label: the positive review (score ≥ 4 ) and negative review (score ≤ 3 ).
We draw and then ﬁx 200 users from the original dataset, which includes the training, validation,and test sets.
In the implementation, we ﬁrst adopt DistilBERT [18] to learn the embedding withdimension R 768 .
Then we adopt ˜ f w and ˜ f
w a as the four-layer fully connected neural network, wherew ∼ Q and w a ∼ Q a .
Additional experimental details are delegated to the Appendix.
Dataset: Toxic Comments We also use the toxic comment dataset [19] to predict the text comment being toxic or not, which has shown the signiﬁcant performance degradation on speciﬁcsub-populations.
Following [19], we select race as sensitive attribute, which includes Black, White,
Asian and Latino & others (4 subgroups).
We also follow the same setting as the original dataset[16], which has the separate training, validation, and test sets.
Since toxic comments are markedby multiple annotators, we decide that the comment is toxic if it is marked by at least half of theannotators.
In the implementation, we adopt the DistilBERT [18] as the embedding with outputdimension R 768 .
Then we also adopt ˜ f w and ˜ f
w a as the four-layer fully connected neural network,with the same network structure as Amazon.
Additional details are delegated to the Appendix.
Baselines We compare with following baselines.
(1) ERM: training a deep model without consideringthe sensitive attribute.
(2) SNN.
Since we adopt randomized algorithms in the paper, we additionallycompare the stochastic neural network through the vanilla training from the whole dataset.
Namely,we ﬁnd a predictive-distribution Q to minimize 1 |A| (cid:80) a E f ∼ Q ˆ L BCE a ( f ) .
(3) EIIL [20] proposed an
IRM based approach to promote the group sufﬁciency.
(4) FSCS [21] adopted the conditional mutualinformation constraint I
( A, Y | f
( X )) to promote the sufﬁciency.
(5) DRO [22].
A re-weightingapproach to assign the importance of the task.
All the experiments are repeated ﬁve times.
Computing Suf f Since f ( X ) is continuous, the group sufﬁciency gap is calculated by splitting theoutput of predictor into multiple intervals in [0 , 1] and computing the conditional expectation withineach interval, as detailed in Appendix.
5.2 Experimental Results
We visualize the results in Fig.
3 for Amazon review product and Fig.
4 for toxic comment.
Accuracy and Fairness
The accuracy and group sufﬁciency gap are depicted in Fig. 3(a) and Fig. 4(a).
In Amazon review, the accuracy in the proposed approach has a slight decrease, compared with ERM.
While the group sufﬁciency gap has improved by 3 . 0% , showing a signiﬁcant improvement in thefairness.
In toxic comments, the accuracy in proposed approach is nearly identical to the baseline,whereas group sufﬁciency gap has been signiﬁcantly improved by 3 . 0 - 3 . 5% .
Group sufﬁciency gap on the speciﬁc subgroup To gain better understandings of group sufﬁciencygap, we visualize group sufﬁciency gap on speciﬁc subgroup
A = a , i.e the discrepancy betweenthe E
[ Y | f
( X )]
(conditional expectation on the entire data) and E [ Y | f ( X ) , A = a ] (conditionalexpectation on subgroup A = a ).
I.e, E X [ | E [ Y | f
( X )] − E
[ Y | f ( X ) , A = a ] | ] .
In Amazon review dataset, we visualize the top-9 users’ sufﬁciency gap in ERM, as shown in Fig.3(b), where the gap of entire users is delegated to the Appendix.
The proposed approach signiﬁcantlyreduces the group sufﬁciency gap of in most subgroups.
The similar trend is also observed in Toxicdataset, as shown in Fig. 4(b), where the proposed approach has the nearly identical and small group
[Figure]
i.e ( f ( X ) , E [ Y | f ( X )]) .
The proposed approach demonstrated a consistently improved probability calibration
[Figure]
i c i e n
[Figure]
S u ff i
c i e n c y
G a p
( A = a )
ERMSNNEIILFSCSDROOurs (
[Figure] [Figure]
Probability 0.
G r o un d T r u t h
P r o b a b ili t y
ERMSNNEIILFSCSDROOurs
[Figure]
sufﬁciency gap for each race.
In contrast, the baselines exhibit signiﬁcant group sufﬁciency bias onthe Asian and Latino & other races.0.
50.0 Lambda 0.
A cc u r a c y
AccGroup Suf Gap 0.
G r o u p S
u
ff i
c i e n
c y G a p
Probability Calibration A related concept togroup sufﬁciency is the probability calibration[23], which is deﬁned as E [ Y | f ( X )]
= f
( X )in the binary classiﬁcation.
We visualize theprobability sufﬁciency of Amazon review in
Fig. 3(c) and Toxic comments in Fig. 4(c).
The results suggest that the proposed approachdemonstrates a consistently better probabilitycalibration on the whole data.
We then visualizethe probability calibration for each subgroup, asshown in the Appendix, where the results revealthe improved probability calibration for eachsubgroup.
Other sensitive attributes in Toxic comments
Apart from adopting race as sensitive attribute, we also consider other possible sensitive attributes such as gender and religion, and the results are showed in the Appendix.
The results in other sensitiveattributes are similar to race, with improved fairness and no loss on accuracy.
Analysis: Inﬂuence of λ Theorem 4.1 suggests that there exists an optimal λ in the generalizationerror bound.
Then we changed the value of λ in Amazon dataset, as shown in Fig.
When λ → 0 , the subgroup speciﬁc parameters are simply learned from the limited samples withineach subgroup.
Then the fair predictor could not learn a proper prior from the subgroup speciﬁcpredictor with a signiﬁcant generalization error.
Meanwhile, the group sufﬁciency gap is also large,which is consistent with [11]: overﬁtting generally degrades the group sufﬁciency.
When λ is setbetween [0 . 2 , 1] , the generalization error is small (with a high accuracy) and group sufﬁciency gap iskept small, implying that both fairness and accuracy can be achieved.
In contrast, if we set a largevalue for λ (cid:29) 0 , the predictor is unable to learn from the data but from the random prior Q .
Theprediction will be completely random (accuracy = 55% when λ = 50 ).
When the predictor outputs arandom guess, different from demographic parity (DP) or equalized odds (EO), the group sufﬁciencygap is also large.
The analysis reveals that there exists an optimal λ for simultaneously achievingaccuracy and group sufﬁciency.
Related Work
Algorithmic fairness Fairness has been attached great importance and widely studied in variousapplications, such as natural language processing [24–26], natural language generation [27–29],computer vision [30, 31], and deep learning [32, 33].
Then various approaches have been proposedin algorithmic fairness.
They typically add fair constraints during the training procedure, such asdemographic parity or equalized odds [8, 34–38].
Apart from this, other fair notions are adoptedsuch as accuracy parity [22, 39], which requires each subgroup to attain the same accuracy; smallprediction variance [40, 41], which ensures small prediction variations among the subgroup; or smallprediction loss for all the subgroups [42–45].
Furthermore, based on the concept of Independence (e.g.demographic parity A ⊥⊥ f ( X ) ) or conditional independence (e.g. equalized odds A ⊥⊥ f ( X ) | Y orgroup sufﬁciency
A ⊥⊥ Y | f
( X ) ), another popular line in fair learning is then naturally integrated withinformation theoretical framework through adding mutual information constraints such as [10, 46].
Understanding the fairness-accuracy trade-off As for the theoretical aspect, [47] further investigated the relation of fairness (demographic parity) and algorithmic stability.
[48] formally justiﬁedthe inherent trade-off between fairness (w.r.t.
demographic parity and equalized odds) and accuracy,whereas the analysis is conducted for the binary sensitive attribute with the population loss.
[49]studied the fair-accuracy trade-off in the multi-task learning.
Group sufﬁciency
The notion of group sufﬁciency has recently been highlighted in various real-worldscenarios such as health [6] and crime prediction [4, 9].
Speciﬁcally, [11] demonstrated that underproper assumptions, the group sufﬁciency can be controlled in the unconstraint learning.
However, thisconclusion may not necessarily always hold in the overparameterized models with limited samples,where [6, 16] essentially revealed the prediction disparities between the different subgroups in theunconstraint learning.
[21] recently studied the fair selective classiﬁcation w.r.t.
group sufﬁciencythrough an information theoretical framework, while the theoretical guarantee is unknown.
In contrast,the proposed lower-level loss within the paper can provably control the generalization error, and theupper-level loss controls the group sufﬁciency gap.
Besides, a close notion to the group sufﬁciencyis the probability calibration [23], which is deﬁned as E [ Y | f ( X )]
= f
( X ) in binary classiﬁcation.
We will empirically show the probability calibration could be consistently improved within ourframework, whereas the analysis on ﬁnite samples and its theoretical relation with group sufﬁciencyremains still opening [20].
Conclusion
We conducted a novel analysis by simultaneously learning an informative and fair classiﬁer formultiple or even many subgroups.
We derived a novel principled algorithm.
We further theoreticallyjustiﬁed the generalization error and fair guarantees of the proposed framework.
The empirical resultsin two real-world datasets demonstrated the effectiveness in both preserving the accuracy, as well asgroup sufﬁciency.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] The proposed theoretical analysis
is restricted in the randomized algorithm.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] We study
the algorithmic fairness.
It is worth noting that ensuring group sufﬁciency may degrade other fair criteria such as DP/EO.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
In the paper
and Appendix.
(b) Did you include complete proofs of all theoretical results?
[Yes] In Appendix.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We provided the code for the reproduction.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See the source code.
(c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
[Yes] We visualize the Boxplot of the results.
(d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[N/A]
[Table] Did you discuss whether the data you are using/curating contains personally identiﬁable [Table]
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table] you include the full text of instructions given to participants and screenshots, if [Table]
applicable
[Table]
you describe any potential participant risks, with links to Institutional Review
[Table]
Board (
) approvals, if applicable
N
[Table]
[-Step 
-]
[-Step 
-]
you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation
[N/A]
[Table]