On Learning Fairness and Accuracy on Multiple Subgroups
Abstract
We propose an analysis in fair learning that preserves the utility of the data while reducing prediction disparities under the criteria of group sufﬁciency.
We focus on the scenario where the data contains multiple or even many subgroups, each with limited number of samples.
As a result, we present a principled method for learning a fair predictor for all subgroups via formulating it as a bilevel objective.
In the lower-level, the subgroup-speciﬁc predictors are learned through a small amount of data and the fair predictor.
In the upper-level, the fair predictor is updated to be close to all subgroup speciﬁc predictors.
We further prove that such a bilevel objective can effectively control the group sufﬁciency and generalization error.
We evaluate the proposed framework on real-world datasets.
Empirical evidence suggests the consistently improved fair predictions, as well as the comparable accuracy to the baselines.
Introduction
Machine learning has made rapid progress in sociotechnical systems such as automatic resume screening, video surveillance and credit scoring for loan applications.
Simultaneously, it has been observed that learning algorithms exhibited biased predictions on the subgroups of population [1, 2].
For example the algorithm denies a loan application based on sensitive attributes such as gender, race, or disability, which has heightened public concerns.
To this end fair learning is recently highlighted to mitigate prediction disparities.
The high-level idea is quite straightforward: adding fair constraints during the training [3].
As a result, fair learning principally gives rise to two desiderata.
On the one hand, the fair predictor should be informative to ensure accurate predictions for the data.
On the other hand, the predictor is required to guarantee fairness to avoid prediction disparities across subgroups.
Therefore, it is crucial to understand the possibilities and then design provable approaches for achieving both informative and fair learning.
Clearly, achieving both objectives depends on predeﬁned fair notations.
Consider demographic parity [1] as the fair criteria, which necessitates the independence between the predictor’s output f ( X ) and the sensitive attribute (or subgourp index) A .
Thus, if the sensitive attribute A and the ground-truth label Y are highly correlated, it is impossible to learn a both fair and informative predictor.
To avoid such intrinsic impossibilities, alternative fair notions have been developed.
In this work, we focus on the criteria of group sufﬁciency [1, 4], which ensures that the conditional expectation of ground-truth label ( E [ Y | f ( X ) , A ] ) is identical across different subgroups, given the predictor’s output.
Notably, the risk of violating group sufﬁciency has arisen in a number of real-world scenarios.
E.g., in medical artiﬁcial intelligence, the machine learning algorithm is used to assess the clinic risk, and guide decisions regarding initiating medical therapy.
However, [5, 6] revealed a signiﬁcant racial bias in such algorithms: when the algorithm predicts the same clinical risk score f
( X ) for white and black patients, black patients are actually at a higher risk of severe illness: E [ Y | f ( X ) ,
A = black ] (cid:29) E
[ Y | f ( X ) , A = white ] .
The deployed algorithms have resulted in more referrals of white patients to specialty healthcare services, resulting in both spending disparities and racial bias [5].
In summary, this work aims to propose a novel principled framework for ensuring group sufﬁciency, as well as preserving an informative prediction with a small generalization error.
In particular, we focus on one challenge scenario: the data includes multiple or even a large number of subgroups, some with only limited samples , as often occurs in the real-world.
For example, datasets for the self-driving car are collected from a wide range of geographical regions, each with a limited number of training samples [7].
How can we ensure group sufﬁciency as well as accurate predictions?
Speciﬁcally, our contributions are summarized as follows:
[Figure]
The proposed algorithm is then formulated as a bilevel optimization to learn an informative and fair predictive-distribution Q .
In the lower-level (cyan), we learn the subgroup speciﬁc predictivedistribution Q (cid:63)a from dataset S a (limited samples) and the prior
Q .
In the upper-level (brown), Q is then updated to be as close to all of the learned subgroup speciﬁc Q (cid:63)a as possible.
Controlling group sufﬁciency We adopted group sufﬁciency gap to measure fairness w.r.t.
group sufﬁciency of a classiﬁer f (Sec.3), and then derive an upper bound of the group sufﬁciency gap (Theorem 4.1).
Under proper assumptions, the upper bound is controlled by the discrepancy between the classiﬁer f and the subgroup Bayes predictors.
Namely, minimizing the upper bound also encourages an informative classiﬁer.
Algorithmic contribution Motivated by the upper bound of the group sufﬁciency gap, we develop a principled algorithm.
Concretely, we adopt a randomized algorithm that produces a predictive-distribution Q over the classiﬁer ( f ∼ Q ) to learn informative and fair classiﬁcation.
We further formulate the problem as a bilevel optimization (Sec. 5.3), as shown in Fig.1.
In the lower-level, the subgroup speciﬁc dataset S a and the fair predictive-distribution Q are used to learn the subgroup speciﬁc predictive-distribution Q (cid:63)a , where Q is regarded as an informative prior for learning limited data within each subgroup.
Theorem 5. formally demonstrates that under proper assumptions, the lowerlevel loss can effectively control the generalization error.
In the upper-level, the fair predictive-distribution Q is then updated to be close to all subgroup speciﬁc predictive-distributions, in order to minimize the upper bound of the group sufﬁciency gap.
Empirical justiﬁcations The proposed algorithm is applicable to the general parametric and differentiable model, where we adopt the neural network in the implementation.
We evaluate the proposed algorithm on two real-world NLP datasets that have shown prediction disparities w.r.t.
group sufﬁciency.
Compared with baselines, the results indicate that group sufﬁciency has been consistently improved, with almost no loss of accuracy.
Code is available at https://github.com/xugezheng/FAMS .
Related Work
Algorithmic fairness Fairness has been attached great importance and widely studied in various applications, such as natural language processing [8–10], natural language generation [11–13], computer vision [14, 15], and deep learning [16, 17].
Then various approaches have been proposed in algorithmic fairness.
They typically add fair constraints during the training procedure, such as demographic parity or equalized odds [18–23].
Apart from this, other fair notions are adopted such as accuracy parity [24, 25], which requires each subgroup to attain the same accuracy; small prediction variance [26, 27], which ensures small prediction variations among the subgroup; or small prediction loss for all the subgroups [28–31]. Furthermore, based on the concept of Independence (e.g. demographic parity A ⊥⊥ f ( X ) ) or conditional independence (e.g. equalized odds A ⊥⊥ f ( X ) | Y or group sufﬁciency A ⊥⊥ Y | f ( X ) ), another popular line in fair learning is then naturally integrated with information theoretical framework through adding mutual information constraints such as [32, 33]. Understanding fairness-accuracy trade-off As for the theoretical aspect, [34] further investigated the relation of fairness (demographic parity) and algorithmic stability. [35] formally justiﬁed the inherent trade-off between fairness (w.r.t.
demographic parity and equalized odds) and accuracy, whereas the analysis is conducted for the binary sensitive attribute with the population loss.
[36] studied the fair-accuracy trade-off in the multi-task learning.
Group
sufﬁciency
The fair notion of group sufﬁciency has recently been highlighted in various real-world scenarios such as health [6] and crime prediction [4, 37].
Speciﬁcally, [38] demonstrated that under proper assumptions, group sufﬁciency can be controlled in the unconstraint learning.
However, this conclusion may not necessarily always hold in the overparameterized models with limited samples per subgroup, where [6, 39, 40] essentially revealed the prediction disparities between the different subgroups in the unconstraint learning.
[41] recently studied the fair selective classiﬁcation w.r.t.
group sufﬁciency through an information theoretical framework, while the theoretical guarantee is unknown.
In contrast, our proposed lower-level loss within the paper can provably control the generalization error, and the upper-level loss controls the group sufﬁciency gap.
Besides, a close notion to the group sufﬁciency is the probability calibration [42], which is deﬁned as E [ Y | f ( X )]
= f
( X ) in binary classiﬁcation.
We will empirically show the probability calibration could be consistently improved within our framework, whereas the analysis on ﬁnite samples and its theoretical relation with group sufﬁciency remains still opening [43].
Bi-level optimization in fairness Bi-level optimization seeks to solve problems with a hierarchical structure.
Namely, two levels of optimization problems where one task is nested inside another [44].
Several ideas related to bi-level optimization have been proposed in the context of fair-learning.
For instance, we could design a min-max optimization to learn fair representation when considering demographic parity (DP) or equalized odds (EO) [19, 32, 25].
In this context, a representation function aims to minimize the loss caused by the discriminator in the lower-level.
Simultaneously, in the upper-level, a discriminator could be introduced to maximize the loss.
Then fair representation could be enforced through the bi-level optimization.
Besides, if the accuracy and its variants are tracked as the metrics for each subgroup [12], the bi-level objective could also be deployed in controlling the loss [45] or the prediction variance [27], where the lower-level’s goal is to minimize the loss for each subgroup and the upper-level’s goal is to estimate the prediction disparities.
In our paper, we theoretically justiﬁed a novel bi-level optimization perspective: controlling group sufﬁciency and accuracy.
Simultaneously, other bi-level optimization and its relevant meta-learning algorithms could be further considered in the fair learning such as recurrent based gradient updating [46], layer-wise transformation [47] or implicit gradient based approach [48].
Preliminaries
We assume the joint random variable ( X, Y, A ) follows an underlying distribution D ( X, Y, A ) , where
X ∈ X is the input, Y ∈ Y is the label, and the scalar discrete random variable A ∈
A denotes the sensitive attribute (or subgroup index).
For instance, A represents gender, race, or age.
We also denote E
[ Y | X ] as the conditional expectation of Y , which is essentially a function of X .
E A,X [ · ] is denoted as the expectation on the marginal distribution of D ( A, X ) .
Throughout the paper, we consider binary classiﬁcation with Y = { 0 , 1 } .
We further deﬁne the predictor as a scoring function f : X → [0 , 1] that maps the input into a real value in [0 , 1] .
It is worth mentioning that in general f ( X ) / ∈ Y since f ( X ) is continuous.
We then introduce group sufﬁciency and group sufﬁciency gap.
Deﬁnition 3.1 (Group sufﬁciency [1, 4, 38]) .
A predictor f satisﬁes group sufﬁciency with respect to the sensitive attribute A if E [ Y | f ( X )]
=
E [ Y | f ( X ) , A ] .
Intuitively, given a output score of the predictor f ( X ) =
τ , the conditional expectation of Y is invariant across different subgroups.
Namely, conditioning on the speciﬁc subgroup A = a does not provide any additional information about the conditional expectation of Y .
Then we could naturally deﬁne group sufﬁciency gap.
Deﬁnition 3.2 (Group sufﬁciency gap [38]) .
The group sufﬁciency gap of a predictor f is deﬁned as:
Suf
[Equation] ] | ]
Speciﬁcally, Suf f measures the extent of group sufﬁciency violation, induced by the predictor f , which is taken by the expectation over ( X, A ) .
Clearly, Suf f = 0 suggests that f satisﬁes groups sufﬁciency and vice versa.
For completeness, we also discuss other popular group fairness criteria: demographic parity and equalized odds.
Deﬁnition 3.3 (Demographic Parity (DP)) .
A predictor f satisﬁes the demographic parity with respect to the sensitive attribute A if: E [ f ( X )]
=
E [ f ( X ) | A ]
Demographic Parity (DP), also known as statistical parity or independence rule, emphasizes that the expectation of the output score f
( X ) is independent of A .
[1, 4] further revealed that if A (cid:54)⊥⊥ Y , group sufﬁciency and demographic parity could not be simultaneously achieved.
Deﬁnition 3.4 (Equalized Odds (EO) [18]) .
A predictor f satisﬁes the equalized odds with respect to A if: E [ f ( X ) | Y ] =
E [ f ( X ) | Y, A ]
Equalized odds (EO) emphasizes the conditional expectation of output f is invariant w.r.t.
A , given the ground truth Y .
[1, 37] reveal that if D ( X, Y, A ) > 0 and A (cid:54)⊥⊥ Y , group sufﬁciency and equalized odds can not both hold.
The analysis reveals a general incompatibility between group sufﬁciency and DP/EO when A (cid:54)⊥⊥ Y , which often occurs in practice.
Besides, DP/EO based criteria generally suffers the well-known fair accuracy trade-off [32]: enforcing the fair constraint degrades the prediction performance.
This paper depicts that under the criteria of group sufﬁciency, these objectives could be both encouraged.
Upper bound of group sufﬁciency gap
To derive the theoretical results, we ﬁrst introduce the group Bayes predictor.
Deﬁnition 4.1 ( A -group Bayes predictor) .
The A -group Bayes predictor f Bayes A is deﬁned as: f Bayes A
[Equation]
The A -group Bayes predictor is associated with the underlying data distribution D ( X, Y, A ) .
Given the ﬁxed realization X = x, A = a , we have f Bayes A = a ( x ) =
E [ Y | X = x, A = a ] , which suggests the ground truth conditional data generation of subgroup
A = a[-) − [ y log( f ( x )) + (1 − y ) log(
 − f ( x ))]-] .
By adopting f Bayes A = a ( x ) , we could derive the upper bound of group sufﬁciency gap w.r.t.
any predictor f :
Theorem 4.1.
Group sufﬁciency gap Suf f is upper bounded by: Suf f ≤ 4 E A,X [ | f − f Bayes A | ]
Speciﬁcally, if A takes ﬁnite value ( |A| < + ∞ ) and follows uniform distribution with D ( A = a ) = 1 / |A| .
Then the group sufﬁciency gap is further simpliﬁed as:
[Equation]
The proof is inspired by [38]. Speciﬁcally, Theorem 4.1 reveals that the upper bound of group sufﬁciency gap depends on the discrepancy between the predictor f and A -group Bayes predictor f Bayes A ( X ) .
Namely, given different subgroups A = a , the optimal predictor f ought to be closed to all the group Bayes predictors f Bayes A
= a ( X ) , ∀ a ∈ A .
Underlying assumption Theorem 4.1 also reveals underlying assumptions w.r.t.
the data generation distribution D ( X, Y, A ) for achieving a small group sufﬁciency gap.
If f Bayes A for each subgroup
A = a are quite similar, then minimizing the upper bound yields a small group sufﬁciency gap
Suf f .
For example, consider the extreme scenario if the A -group Bayes predictors are identical w.r.t.
A , E [ Y | X, A
= a ] = E
[ Y | X ] , ∀ a ∈ A , where E [ Y | X ] is the conventional Bayes predictor deﬁned on the marginalized distribution D ( X, Y ) .
The upper bound recovers the difference between the predictor f and standard Bayes predictor.
If we use a probabilistic framework to approximate predictor f ( X ) ≈ E
[ Y | X ] (i.e, training the entire dataset without any fair constraint), both group sufﬁciency gap and prediction error (since Bayes predictor is optimal) will be small, which is consistent with [38].
On the contrary, if A -group Bayes predictors are completely arbitrary with high variance for A , both group sufﬁciency gap and prediction error are large and it would be impossible for an informative prediction.
Principled Approach
Based on the upper bound, we propose a principled approach to learn the predictor that achieves both small generalization error and group sufﬁciency gap.
5.1 Upper bound in randomized algorithm
To establish the theoretical result, we consider a randomized algorithm that learns a predictivedistribution Q over scoring predictors from the data.
For instance, if we consider Bayes framework, the predictor is drawn from the posterior distribution ˜ f ∼ Q .
In the inference, the predictor’s output is formulated as the expectation of the learned predictive-distribution Q : f
( X ) =
E ˜ f ∼ Q ˜ f
( X ) .
In practice, it is infeasible to optimize over all the possible distributions.
Then we should restrict the predictive-distribution Q within a distribution family Q ∈ Q such as Gaussian distribution.
We also denote Q (cid:63)a ∈ Q as the optimal prediction-distribution w.r.t.
A = a under binary crossentropy loss within the distribution family Q : Q (cid:63)a :=
argmin Q a ∈Q E ˜ f a ∼ Q a L [Equation] a ( ˜ f a ) .
In generally Q (cid:63)a (cid:54) =
f Bayes ( x, A = a ) , since the distribution family Q is only the subset of all possible distributions (shown in Fig. 2).
We then extend the upper bound in the randomized algorithm.
Corollary 5.1.
The group sufﬁciency gap Suf f in randomized algorithm w.r.t.
learned predictivedistribution Q is upper bounded by:
[Equation]
Where KL is the Kullback–Leibler divergence.
Corollary 5.1 further reveals that the upper bound is decomposed into two terms, showing in Fig.2.
[Equation]
constant.
Optimization term The optimization term is the average KL divergence between the learned distribution Q and optimal predictive-distribution Q (cid:63)a for each subgroup
A = a .
Minimizing the optimization term implies that the learned distribution
Q will be both fair and informative for the prediction, because it aims to minimize the upper bound of the group sufﬁciency gap Suf f and be close to the optimal predictive-distribution w.r.t.
each A = a .
Approximation term The approximation term is the average
KL divergence between the optimal distribution Q (cid:63)a and the underlying data generation distribution.
Given the distribution family Q , it is a unknown constant.
Besides, if the distribution family Q has a rich expressive power such as deep neuralnetwork, the approximation term will be small [49].
However, an extreme large distribution family Q could simultaneously yield a potential overﬁtting on ﬁnite samples.
In this paper, the neural network is adopted and the approximation term is assumed to be a small constant.
Thus, controlling Suf f implies minimizing the optimization term.
5.2 Challenge in learning limited samples
In practice, we only have access to ﬁnite or even limited samples in each subgroup, rather than the underlying distribution D .
We denote S a = { ( x ai , y ai ) } mi =1 as the observed data w.r.t.
subgroups
A = a , which are i.i.d.
samplings from the underlying distribution D ( x, y |
A = a ) .
We also denote the empirical binary cross entropy loss w.r.t A
= a as: ˆ L BCE a ( ˜ f )
= 1 m (cid:80) mi =1 − [ y ai log( ˜ f
( x ai ))
+ (1 − y ai ) [-log(
 −˜-]log(1 − ˜ f
( x ai ))] .
Then a straight approach is to minimize the empirical term ˆ Q (cid:63)a :
[Equation]
Then Q is updated through minimizing the average KL-divergence: (cid:80) a KL ( ˆ Q (cid:63)a (cid:107) Q ) from learned ˆ Q (cid:63)a .
However, this idea generally does not work in our setting, because each subgroup contains limited number of samples.
Therefore, a straight minimization leads to overﬁtting for each subgroup and generalization error E ˜
f ∼ ˆ Q (cid:63)a L BCE a ( ˜ f ) is quite large, showing in Fig.
5.3 Q as an informative prior
We have demonstrated that Q can achieve both fair and informative prediction.
Therefore, we regard
Q as a prior information for minimizing the loss, yielding a bilevel objective.
[Equation]
Where λ > 0 is the hyper-parameter.
The proposed loss is a typical bilevel optimization.
In the lower-level, we aim to learn Q (cid:63)a for each a ∈ A .
Different from Eq.
(1), the loss in lower-level adds a regularization term KL ( Q a (cid:107) Q ) as an informative prior in learning Q (cid:63)a , given a ﬁxed predictivedistribution Q .
Moreover, Theorem 5.1 formally justiﬁed that optimizing the lower-level loss is to minimize the upper bound of the generalization error.
In the upper-level, Q is updated through minimizing the average KL divergence between different Q (cid:63)a , which controls the upper bound of
Suf f .
Theorem 5.1 (Generalization error bound) .
Supposing that datasets { S a } |A| a =1 with S a = { ( x a i , y a i ) } m i =1 are i.i.d.
sampled from D ( x, y | A = a ) , the binary cross entropy (BCE) loss is upper bounded by L , Q a ∈ Q is any learned distribution from dataset S a and Q ∈ Q is any distribution.
Then with high probability ≥ 1 − δ with ∀ δ ∈ (0 , 1) , we have:
[Equation]
a ∼ Q a ˆ L BCE a ( ˜ f a ) (cid:124) (cid:123)(cid:122) (cid:125) (1)
+
L (cid:112) |A| m (cid:88) a (cid:112) KL ( Q a (cid:107) Q ) (cid:124) (cid:123)(cid:122) (cid:125) (2) + L (cid:115) log(1 /δ )
|A| m (cid:124) (cid:123)(cid:122) (cid:125) (3)
Discussions The proof is inspired by PAC-Bayes theorem such as [50–52].
Secpﬁcially, Theorem 5.1 reveals the generalization error in the lower-level is upper bounded by three terms.
(a) Term (1) is the average empirical prediction error, which corresponds to the ﬁrst term in the lower-level loss.
(b) Term (2) indicates the average KL-divergence between the learned subgroup distribution Q a and the prior distribution Q , which corresponds to the second term in the lower-level loss.
The combination of term (1-2) recovers the averaged lower-level loss w.r.t.
A . 2 Thus optimizing the lower-level loss could control the generalization error.
(c) When the conﬁdence δ is ﬁxed, term (3) will converge if |A| m → + ∞ .
Moreover, even if m (the sample size in each subgroup) is quite small, a sufﬁcient large number of subgroups |A| can also ensure the convergence of term (3).
For the sake of simplicity, we assumed the identical samples size m in each subgroup S a , while the theoretical result can be extended to subgroups with different samples m a .
5.4 Practical Implementations
In this section, we develop a practical learning algorithm that can be applied to a wide range of differentiable and parametric models, including neural networks.
Parametric models We choose the Isotropic Gaussian distribution (with diagonal covariance matrix) as the distribution family Q , where the mean and covariance are set as d -dimensional parameter.
Thus we need to learn the parameter ( θ , σ ) for fair and informative Q ∈ Q .
As for the subgroup A = a , we learn parameters ( θ a , σ a ) for Q (cid:63)a ∈ Q .
It is worth mentioning that the Isotropic
Gaussian distribution is selected for its computational efﬁciency in the optimization.
We can use any distribution as long as the density function is differentiable with respect to the parameters.
For the single predictor ˜ f , we use parametric neural-network models and assume ˜ f is parameterized by a d -dimensional vector w ∈ R d , denoted as ˜ f w .
Then ˜ f w ∼ Q is equivalent to sampling the model parameter w from the predictive-distribution Q : w ∼ N ( θ , σ 2 ) =
(cid:81)
d i =1 N ( θ [ i ] , σ 2 [ i ]) .
Since Q is
Isotropic Gaussian, each element i in the parameter w
[ i ] follows a 1-dimensional Gaussian.
Following the same line, ˜ f w a ∼ Q (cid:63)a can be modeled analogously: w a ∼ N
( θ a , σ 2 a ) =
(cid:81) di
=1 N ( θ a [ i ] , σ 2 a [ i ]) .
As a result, learning the distribution Q is equivalent to learning parameter ( θ , σ ) in the bilevel objective.
Gradient Estimation Based on the previous setting, we aim to optimize the bilevel objective to obtain the parameter of Q : ( θ , σ ) .
We use stochastic gradient descent (SGD) to optimize the parameters.
In the lower-level, the loss in Sec.
5.3 is composed by the empirical prediction error and KL divergence term.
The KL divergence has a closed form that can be differentiated efﬁciently.
Speciﬁcally, since Q and the subgroup speciﬁc Q (cid:63)a are factorized Gaussian, the KL divergence takes a simple closed form and the gradient can be easily calculated: KL ( Q (cid:63)a (cid:107) Q ) = 12 (cid:80) [Equation] log
[Equation]
−
[Equation]
Re-parametrization trick As for the prediction error E ˜
f w a ∼ Q a ˆ L BCE a ( ˜ f w a ) , the term ˆ L BCE a ( ˜ f w a ) is non-linear for w , rendering the expectation intractable in the computation.
To this end, we adopt the re-parameterization trick [53, 54] in computing the gradient w.r.t.
the expectation term.
The trick is based on describing the Gaussian distribution w a ∼ N ( θ a , σ 2 a ) as ﬁrst drawing (cid:15) ∼ N ( 0 , I ) and then applying the deterministic function w a ( θ a , σ a )
=
θ a + σ a (cid:12) (cid:15) ( (cid:12) is element-wise product) to approximate the sampling.
Then the gradient term can be estimated as: ∇ ( θ a , σ a ) E w a ∼ N ( θ a , σ a ) ˆ L BCE a ( ˜ f w a ) =
∇ ( θ a , σ a ) E (cid:15) ∼N ( 0 , I )
ˆ L BCE
a ( ˜ f w a ( θ a , σ a ) ) , where the expectation can be approximated by Monte-Carlo sampling w.r.t.
(cid:15) .
For a ﬁxed sample (cid:15) , the gradient can be computed through backpropagation.
In the upper-level, the KL divergence has a closed form, thus it is easy to update the parameter of Q through backpropagation.
Proposed Algorithm Based on the analysis, the algorithm is shown in Algorithm.
1 for solving the bilevel objective in Sec.
Speciﬁcally, we adopt the alternating optimization.
Namely, in the lower-level, we ﬁx Q and optimize the subgroup speciﬁc predictor Q (cid:63)a through SGD.
Then in the upper-level, we ﬁx the learned Q (cid:63)a and update Q .
Since we may face many subgroups, at each training epoch, we randomly sample a subset A (cid:48) such that |A (cid:48) | (cid:28) |A| for the memory saving.
Inference In the inference, we use the Monte-Carlo method to sample the weights of the neural network from distribution w ∼ N ( θ , σ 2 ) , then averaging the output w.r.t.
different sampled weights to approximate f ( x ) =
E ˜ f
w ∼ Q ˜ f w ( x )
Experiments 6.1 Experimental Setup
Dataset: Amazon review We adopt Amazon review dataset [55, 40], which aims to predict the sentiment (classiﬁcation) from the review.
The datasets consist of large-scale users.
Each user has limited number of reviews, ranging from 75 to 400.
The user is then treated as a subgroup, and it has been observed that standard training can lead to prediction disparities in several users.[56].The [Figure] u [Figure]
u ff i
c
i [Figure] n c [Figure]
a
[Figure]
i.e ( f ( X ) , E [ Y | f ( X )]) .
The proposed approach demonstrated a consistently improved probability calibration experiment is adapted from the protocol of [40].
Speciﬁcally, we convert the original review score (ranging from 1-5) to the binary label: the positive review (score ≥ 4 ) and negative review (score ≤ 3 ).
We draw and then ﬁx 200 users from the original dataset, which includes the training, validation, and test sets.
In the implementation, we ﬁrst adopt DistilBERT [57] to learn the embedding with dimension R 768 .
Then we adopt ˜ f w and ˜ f
w a as the four-layer fully connected neural network, where w ∼ Q and w a ∼ Q a .
Additional experimental details are delegated to the Appendix.
Dataset: Toxic Comments We also use the toxic comment dataset [58] to predict the text comment being toxic or not, which has shown the signiﬁcant performance degradation on speciﬁc sub-populations.
Following [58], we select race as sensitive attribute, which includes Black, White,
Asian and Latino & others (4 subgroups).
We also follow the same setting as the original dataset [40], which has the separate training, validation, and test sets.
Since toxic comments are marked by multiple annotators, we decide that the comment is toxic if it is marked by at least half of the annotators.
In the implementation, we adopt the DistilBERT [57] as the embedding with output dimension R 768 .
Then we also adopt ˜ f w and ˜ f
w a as the four-layer fully connected neural network, with the same network structure as Amazon.
Additional details are delegated to the Appendix.
Baselines We compare with following baselines.
(1) ERM: training a deep model without considering the sensitive attribute.
(2) SNN.
Since we adopt randomized algorithms in the paper, we additionally compare the stochastic neural network through the vanilla training from the whole dataset.
Namely, we ﬁnd a predictive-distribution Q to minimize 1 |A| (cid:80) a E f ∼ Q ˆ L BCE a ( f ) .
(3) EIIL [43] proposed an IRM based approach to promote the group sufﬁciency.
(4) FSCS [41] adopted the conditional mutual information constraint I
( A, Y | f
( X )) to promote the sufﬁciency.
(5) DRO [24].
A re-weighting approach to assign the importance of the task.
Indeed, DRO does not provably guarantee group sufﬁciency, while it encourages identical losses.
All the experiments are repeated ﬁve times
Since f ( X ) is continuous, the group sufﬁciency gap is calculated by splitting the output of predictor into multiple intervals in [0 , 1] and computing the conditional expectation within each interval, as detailed in Appendix.
6.2 Experimental Results
We visualize the results in Fig.
3 for Amazon review product and Fig.
4 for toxic comment.
Accuracy and Fairness
The accuracy and group sufﬁciency gap are depicted in Fig. 3(a) and Fig. 4(a).
In Amazon review, the accuracy in the proposed approach has a slight decrease, compared with ERM.
[Figure]
While the group sufﬁciency gap has improved by 3 . 0% , showing a signiﬁcant improvement in the fairness.
In toxic comments, the accuracy in proposed approach is nearly identical to the baseline, whereas group sufﬁciency gap has been signiﬁcantly improved by 3 . 0 - 3 . 5% .
Group sufﬁciency gap on the speciﬁc subgroup To gain better understandings of group sufﬁciency gap, we visualize group sufﬁciency gap on speciﬁc subgroup
A = a , i.e the discrepancy between the E
[ Y | f ( X )]
(conditional expectation on the entire data) and E [ Y | f ( X ) , A = a ] (conditional expectation on subgroup A = a ).
I.e, E X [ | E [ Y | f
( X )] − E
[ Y | f ( X ) , A = a ] | ] .
In Amazon review dataset, we visualize the top-9 users’ sufﬁciency gap in ERM, as shown in Fig. 3(b), where the gap of entire users is delegated to the Appendix.
The proposed approach signiﬁcantly reduces the group sufﬁciency gap of in most subgroups.
The similar trend is also observed in Toxic dataset, as shown in Fig. 4(b), where the proposed approach has the nearly identical and small group sufﬁciency gap for each race.
In contrast, the baselines exhibit signiﬁcant group sufﬁciency bias on the Asian and Latino & other races.
[Figure]
A
[Figure]
u
[Figure]
a c y
[Figure]
[Figure]
[Figure]
[Figure]
G r o u p S
u
ff i
c i e n
c y G a p
Probability Calibration A related concept to group sufﬁciency is the probability calibration [42], which is deﬁned as E [ Y | f ( X )]
= f
( X ) in the binary classiﬁcation.
We visualize the probability sufﬁciency of Amazon review in
Fig. 3(c) and Toxic comments in Fig. 4(c).
The results suggest that the proposed approach demonstrates a consistently better probability calibration on the whole data.
We then visualize the probability calibration for each subgroup, as shown in the Appendix, where the results reveal the improved probability calibration for each subgroup.
Other sensitive attributes in Toxic comments Apart from adopting race as sensitive attribute, we also consider other possible sensitive attributes such as gender and religion, and the results are showed in the Appendix.
The results in other sensitive attributes are similar to race, with improved fairness and no loss on accuracy.
Inﬂuence of λ .
Fairness and accuracy can be simultaneously achieved.
Theorem 5.1 suggests that there exists an optimal λ in the generalization error bound.
Then we changed the value of λ in Amazon dataset, as shown in Fig.
When λ → 0 , the subgroup speciﬁc parameters are simply learned from the limited samples within each subgroup.
Then the fair predictor could not learn a proper prior from the subgroup speciﬁc predictor with a signiﬁcant generalization error.
Meanwhile, the group sufﬁciency gap is also large, which is consistent with [38]: overﬁtting generally degrades the group sufﬁciency.
When λ is set between [0 . 2 , 1] , the generalization error is small (with a high accuracy) and group sufﬁciency gap is kept small, implying that both fairness and accuracy can be achieved.
In contrast, if we set a large value for λ (cid:29) 0 , the predictor is unable to learn from the data but from the random prior Q .
The prediction will be completely random (accuracy = 55% when λ = 50 ).
When the predictor outputs a random guess, different from demographic parity (DP) or equalized odds (EO), the group sufﬁciency gap is also large.
The analysis reveals that there exists an optimal λ for simultaneously achieving accuracy and group sufﬁciency.
Conclusion
We conducted a novel analysis by simultaneously learning an informative and fair classiﬁer for multiple or even many subgroups.
We derived a novel principled algorithm.
We further theoretically justiﬁed the generalization error and fair guarantees of the proposed framework.
The empirical results in two real-world datasets demonstrated the effectiveness in both preserving the accuracy, as well as group sufﬁciency.
Discussion on Limitations
We proposed the analysis on learning group sufﬁciency and informative predictors, and developed a principled approach for it. Simultaneously there are several limitations to the proposed theory and algorithm.
(1) In general, group sufﬁciency and DP/EO are incompatible. Controlling group sufﬁciency, for example, would cause DP/EO degradation.
This would be problematic if DP/EO were preferred in practice. (2) We also assumed that the ground truth A -Bayes predictors would be similar across groups. However, this assumption could be violated, resulting in a highly non-trivial scenario.
Thus, in order to evaluate the conditional distribution shift, we need to consider a new setting by collect sufﬁcient data per subgroup.
Acknowledgments and Disclosure of Financial Support
We appreciate constructive feedback from anonymous reviewers and meta-reviewers.
We also would like to thank Jun Xiao for the discussion and proof-reading the manuscript.
C. Shui and C. Gagné acknowledge support from NSERC-Canada and the Canada CIFAR Chairs in AI.
G. Xu, J. Li, C.
Ling and B. Wang are supported by Natural Sciences and Engineering Research Council of Canada (NSERC), Discovery Grants program.
T. Arbel is supported by International Progressive Multiple
Sclerosis Alliance, the Canada Institute for Advanced Research (CIFAR) Artiﬁcial Intelligence Chairs program, the Natural Sciences and Engineering Research Council of Canada.
Q. Chen is supported by China Scholarship Council.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] The proposed theoretical analysis is restricted in the randomized algorithm.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] We study the algorithmic fairness.
It is worth noting that ensuring group sufﬁciency may degrade other fair criteria such as DP/EO.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes]
In the paper and Appendix.
(b) Did you include complete proofs of all theoretical results?
[Yes] In Appendix. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We provided the code for the reproduction.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See the source code.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We visualize the Boxplot of the results.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
[-Step 
-]
[-Step 
-]