S TOCHASTIC D IFFERENTIALLY P RIVATE AND F AIR L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Machine learning models are increasingly used in high-stakes decision-making systems.
In such applications, a major concern is that these models sometimes discriminate against certain demographic groups such as individuals with certain race, gender, or age.
Another major concern in these applications is the violation of the privacy of users.
While fair learning algorithms have been developed to mitigate discrimination issues, these algorithms can still leak sensitive information, such as individuals’ health or financial records.
Utilizing the notion of differential privacy (DP) , prior works aimed at developing learning algorithms that are both private and fair.
However, existing algorithms for DP fair learning are either not guaranteed to converge or require full batch of data in each iteration of the algorithm to converge.
In this paper, we provide the first stochastic differentially private algorithm for fair learning that is guaranteed to converge.
Here, the term “stochastic" refers to the fact that our proposed algorithm converges even when minibatches of data are used at each iteration (i.e. stochastic optimization ).
Our framework is flexible enough to permit different fairness notions, including demographic parity and equalized odds.
In addition, our algorithm can be applied to non-binary classification tasks with multiple (non-binary) sensitive attributes.
As a byproduct of our convergence analysis, we provide the first utility guarantee for a DP algorithm for solving nonconvex-strongly concave min-max problems.
Our numerical experiments show that the proposed algorithm consistently offers significant performance gains over the state-of-the-art baselines , and can be applied to larger scale problems with non-binary target/sensitive attributes.
I NTRODUCTION
In recent years, machine learning algorithms have been increasingly used to inform decisions with far-reaching consequences (e.g. whether to release someone from prison or grant them a loan), raising concerns about their compliance with laws, regulations, societal norms, and ethical values.
Specifically, machine learning algorithms have been found to discriminate against certain “sensitive” demographic groups (e.g. racial minorities), prompting a profusion of algorithmic fairness research (Dwork et al., 2012; Sweeney, 2013; Datta et al., 2015; Feldman et al., 2015; Bolukbasi et al., 2016; Angwin et al., 2016; Calmon et al., 2017; Hardt et al., 2016a; Fish et al., 2016; Woodworth et al., 2017; Zafar et al., 2017; Bechavod & Ligett, 2017; Kearns et al., 2018; Prost et al., 2019; Baharlouei et al., 2020; Lowy et al., 2021). Algorithmic fairness literature aims to develop fair machine learning algorithms that output non-discriminatory predictions.
Fair learning algorithms typically need access to the sensitive data in order to ensure that the trained model is non-discriminatory.
However, consumer privacy laws (such as the E.U. General Data
Protection Regulation) restrict the use of sensitive demographic data in algorithmic decision-making.
These two requirements– fair algorithms trained with private data –presents a quandary: how can we train a model to be fair to a certain demographic if we don’t even know which of our training examples belong to that group?
The works of Veale & Binns (2017); Kilbertus et al.
(2018) proposed a solution to this quandary using secure multi-party computation (MPC) , which allows the learner to train a fair model without directly accessing the sensitive attributes.
Unfortunately, as Jagielski et al.
(2019) observed, MPC does not prevent the trained model from leaking sensitive data .
For example, with MPC, the output of the trained model could be used to infer the race of an individual in the training data set (Fredrikson et al.,
; He et al., 2019; Song et al., 2020; Carlini et al., 2021).
To prevent such leaks, Jagielski et al.
(2019) argued for the use of differential privacy (Dwork et al., 2006) in fair learning.
Differential privacy (DP) provides a strong guarantee that no company (or adversary) can learn much more about any individual than they could have learned had that individual’s data never been used.
Since Jagielski et al.
(2019), several follow-up works have proposed alternate approaches to DP fair learning (Xu et al., 2019;
Ding et al., 2020; Mozannar et al., 2020; Tran et al.
, 2021b;a; 2022).
As shown in Fig.
1, each of these approaches suffers from at least two critical shortcomings.
In particular, none of these methods have convergence guarantees when mini-batches of data are used in training .
In training large-scale models, memory and efficiency constraints require the use of small minibatches in each iteration of training (i.e. stochastic optimization).
Thus, existing DP fair learning methods cannot be used in such settings since they require computations on the full training data set in every iteration.
See Appendix A for a more comprehensive discussion of related work.
Our Contributions: In this work, we propose a novel algorithmic framework for DP fair learning.
Our approach builds on the non-private fair learning method of Lowy et al.
We consider a regularized empirical risk minimization (ERM) problem where the regularizer penalizes fairness violations, as measured by the Exponential Rényi Mutual Information .
Using a result from Lowy et al.
(2021), we reformulate this fair ERM problem as a min-max optimization problem.
Then, we use an efficient differentially private variation of stochastic gradient descent-ascent (DP-SGDA) to solve this fair ERM min-max objective.
The main features of our algorithm are:
Guaranteed convergence for any privacy and fairness level, even when mini-batches of data are used in each iteration of training (i.e. stochastic optimization setting).
As discussed, stochastic optimization is essential in large-scale machine learning scenarios.
Our algorithm is the first stochastic DP fair learning method with provable convergence.
Flexibility to handle non-binary classification with multiple (non-binary) sensitive attributes (e.g. race and gender) under different fairness notions such as demographic parity or equalized odds.
In each of these cases, our algorithm is guaranteed to converge.
Empirically, we show that our method outperforms the previous state-of-the-art methods in terms of fairness vs. accuracy trade-off across all privacy levels.
Moreover, our algorithm is capable of training with mini-batch updates and can handle non-binary target and non-binary sensitive attributes .
By contrast, existing DP fairness algorithms could not converge in our stochastic/non-binary experiment.
A byproduct of our algorithmic developments and analyses is the first DP convergent algorithm for nonconvex min-max optimization : namely, we provide an upper bound on the stationarity gap of
DP-SGDA for solving problems of the form min θ max W F p θ, W q , where F p¨ , W q is non-convex.
We expect this result to be of independent interest to the DP optimization community.
Prior works that provide convergence results for DP min-max problems have assumed that F p¨ , W q is either (strongly) convex (Boob & Guzmán, 2021; Zhang et al., 2022) or satisfies a generalization of strong convexity known as the Polyak-Łojasiewicz (PL) condition (Yang et al., 2022).
P ROBLEM S ETTING AND P RELIMINARIES
Let Z “ t z i “ p x i , s i , y i qu ni “ 1 be a data set with non-sensitive features x i P X , discrete sensitive attributes (e.g. race, gender) s i P r k s fi t 1 , . . .
, k u , and labels y
i P r l s .
Let p y θ p x q denote the model predictions parameterized by θ , and ℓ p θ, x, y q “ ℓ p p y θ p x q , y q be a loss function (e.g. cross-entropy loss).
Our goal is to (approximately) solve the empirical risk minimization (ERM) problem
[Equation]
in a fair manner, while maintaining the differential privacy of the sensitive data t s i u ni “ 1 .
We consider two different notions of fairness in this work:
Definition 1 (Fairness Notions) .
Let A : Z Ñ Y be a classifier.
• A satisfies demographic parity (Dwork et al., 2012) if the predictions A p Z q are statistically independent of the sensitive attributes.
• A satisfies equalized odds (Hardt et al., 2016a)
if the predictions A p Z q are conditionally independent of the sensitive attributes given Y “ y for all y .
[Figure]
Depending on the specific problem at hand, one fairness notion may be more desirable than the other (Dwork et al., 2012; Hardt et al., 2016a).
In practical applications, achieving exact fairness, i e. (conditional) independence of p Y and S , is unrealistic.
In fact, achieving exact fairness can be impossible for a differentially private algorithm that achieves non-trivial accuracy (Cummings et al., 2019).
Therefore, we instead aim to design an algorithm that achieves small fairness violation on the given data set Z .
In general, our method can minimize all independence metrics, e g. demographic parity and all seperation metrics, e.g. equalized odds.
Fairness violation can be measured in different ways: see e.g. (Lowy et al., 2021, Appendix C) for a thorough survey.
For example, if demographic parity is the desired fairness notion, then one can measure (empirical) demographic parity violation by
[Equation]
where ˆ p denotes an empirical probability calculated directly from p Z, t p
y
i u ni “ 1 q .
Next, we define differential privacy (DP).
Following the DP fair learning literature in (Jagielski et al., 2019; Tran et al., 2021b; 2022)), we define DP solely with respect to the sensitive attributes : say two data sets Z, Z 1 P Z n are adjacent (denoted Z „ Z 1 ) if they differ in one person’s sensitive attribute: i.e. Z “ tp x i , y i , s i qu ni “ 1 and Z 1 “ tp x i , y i , s 1 i qu ni “ 1 , where s i “ s 1 i for all i ‰ j , and s j ‰ s 1 j .
An algorithm is DP if it behaves similarly independently of the presence or absence of any one individual’s sensitive data:
Definition 2 (Differential Privacy) .
Let ϵ ě 0 , δ P r 0 , 1 q .
A randomized algorithm A is p
ϵ, δ qdifferentially private (DP) if for all adjacent data sets Z „ Z 1 , we have
[Equation]
for all O Ď Y , where the probability is solely over A .
By the post-processing property of differential privacy (Dwork & Roth, 2014) , Definition 2 also guarantees that sensitive training data cannot be leaked through the predictions made at test time.
Throughout the paper, we shall restrict attention to data sets in Z n that contain at least ρ -fraction of every sensitive attribute for some ρ P p 0 , 1 q : i.e. 1 n ř ni “ 1 1 t s i “ r u ě ρ for all r P r k s .
This is a reasonable assumption in practice: for example, if sex is the sensitive attribute and a data set contains all men, then training a model that is fair with respect to sex and has a non-trivial performance (better than random) seems almost impossible.
Understanding what performance is (im-)possible for DP fair learning in the absence of sample diversity is an important direction for future work.
P RIVATE F AIR ERM VIA E XPONENTIAL R ÉNYI M UTUAL I NFORMATION
A standard in-processing strategy in the literature for enforcing fairness is to add a regularization term to the empirical objective that penalizes fairness violations (Zhang et al., 2018; Donini et al., 2018; Mary et al., 2019; Baharlouei et al., 2020; Cho et al., 2020b; Lowy et al., 2021).
We can then jointly optimize for fairness and accuracy by solving
[Equation]
where D is some measure of statistical (conditional) dependence between the sensitive attributes and the predictions (given Y ), and λ ě 0 is a scalar balancing fairness and accuracy considerations.
The choice of D is crucial and can lead to different fairness-accuracy profiles.
Inspired by the strong empirical performance and amenability to stochastic optimization of Lowy et al.
(2021), we choose
D to be the Exponential Rényi Mutual Information (ERMI):
Definition 3 (ERMI – Exponential Rényi Mutual Information) .
We define the exponential Rényi mutual information between random variables p Y and S with empirical joint distribution ˆ p p Y ,S and marginals ˆ p p Y , ˆ p S by:
[Equation]
Definition 3 is what we would use if demographic parity were the desired fairness notion.
If instead one wanted to encourage equalized odds, then Definition 3 can be readily adapted to these fairness notions by substituting appropriate conditional probabilities for ˆ p p Y ,S , ˆ p p Y , and ˆ p S in (ERMI): see Appendix B for details.
1 It can be shown that ERMI ě 0 , and is zero if and only if demographic parity (or equalized odds, for the conditional version of ERMI) is satisfied (Lowy et al., 2021).
Further, ERMI provides an upper bound on other commonly used measures of fairness violation: e.g.) (2), Shannon mutual information (Cho et al., 2020a), Rényi correlation (Baharlouei et al., 2020), L q fairness violation (Kearns et al., 2018; Hardt et al., 2016a)
(Lowy et al., 2021).
This implies that any algorithm that makes ERMI small will also have small fairness violation with respect to these other notions.
Lastly, (Lowy et al., 2021, Proposition 2) shows that empirical ERMI (Definition 3) is an asymptotically unbiased estimator of “population ERMI”–which can be defined as in Definition 3, except that empirical distributions are replaced by their population counterparts.
Our approach to enforcing fairness is to augment (1) with an ERMI regularizer and privately solve:
[Equation]
Since empirical ERMI is an asymptotically unbiased estimator of population ERMI, a solution to (FERMI obj.) is likely to generalize to the corresponding fair population risk minimization problem (Lowy et al., 2021).There are numerous ways to privately solve (FERMI obj.).
For example, one could use the exponential mechanism (McSherry & Talwar, 2007), or run noisy gradient descent (GD) (Bassily et al., 2014).
The problem with these approaches is that they are inefficient or require computing n gradients at every iteration, which is prohibitive for large-scale problems, as discussed earlier.
Notice that we could not run noisy stochastic GD ( S GD) on (FERMI obj.)
because we do not (yet) have a statistically unbiased (w.r.t.
the empirical distribution) estimate (of the gradient) of p
D R p p Y θ p X q , S q .
Our next goal is to derive a stochastic , differentially private fair learning algorithm.
For feature input x , let the predicted class labels be given by p
y p x, θ q “ j P r l s with probability F j p x, θ q , where F p x, θ q is differentiable in θ , has range r 0 , 1 s l , and ř lj “ 1 F j p x, θ q “ 1 .
For instance,
F p x, θ q “ p F 1 p x, θ q , . .
, F l p x, θ qq could represent the output of a neural net after softmax layer or the probability label assigned by a logistic regression model.
Let p y p
x
i ; θ q P t 0 , 1 u l and s i P t 0 , 1 u k be the one-hot encodings of p
y p
x
i , θ q and s i , respectively: i.e., p y j p x i ; θ q “ 1 t p y p x i ,θ q“ j u and s i,r “ 1 t s i “ r u for j P r l s , r P r k s ).
Also, denote p P s “ diag p p p S p 1 q , . .
. , p p S p k qq , where p p S p j q : “ 1 n ř ni “ 1 1 t s i “ j u ě ρ ą 0 is the empirical probability of attribute j ( j P r k s ). Then we have the following re-formulation of (FERMI obj.) as a min-max problem: Theorem 3.1 (Lowy et al.
(FERMI obj.) is equivalent to
[Equation]
where p ψ i p θ, W q : “ ´ Tr p W E r p
y
i
p x i ; θ q p
y
Ti p x i ; θ q| x i s W T q` [Equation] Tr p W E r p
y i
p x i ; θ q| x
i s s Ti p P ´ 1 { 2 s q´ 1 .
Here p P p
y p θ q : “ diag p 1 n ř ni “ 1 F 1 p x i , θ q , . .
, 1 n ř ni “ 1 F l p x i , θ qq is an l ˆ l matrix and p P p
y,s p θ q is l ˆ k with p p P p y,s p θ qq j,r “ 1 n
[Equation]
for j P r l s , r P r k s .
Note that p ψ i is differentiable in θ and W , and strongly concave in the W variable (Lowy et al., 2021).
Theorem 3.1 is useful because it permits us to use stochastic optimization to solve (FERMI obj.
): for any batch size m P r n s , the gradients (with respect to θ and W ) of 1 m ř i P B ℓ p x i , y i ; θ q ` λ p ψ i p θ, W q are statistically unbiased (w.r.t.
the empirical distribution) estimators of the gradients of p F p θ, W q , if
B is drawn uniformly from Z .
However, when differential privacy of the sensitive attributes is also desired, the formulation (4) presents some challenges, due to the non-convexity of p F p¨ , W q .
Indeed, there is no known DP algorithm for solving non-convex min-max problems that is proven to converge .
Next, we provide the first such convergence guarantee.
3. N OISY DP-SGDA FOR N ONCONVEX -S TRONGLY C ONCAVE M IN -M AX P ROBLEMS
Consider a generic (smooth) nonconvex-strongly concave min-max ERM problem:
[Equation]
where f p θ, ¨ ; z q is µ -strongly concave 2 for all θ, z but f p¨ , w ; z q is potentially non-convex.
We
Algorithm
[Equation] Ð [Equation]
˘ , where u t „ N p 0 , σ 2 θ I d θ q . 5: w
[Equation] Ð
[Equation] W “
[Equation]
˘‰ , where v t „ N p 0 , σ 2 w I d w q . 6: end for Return propose Noisy DP-SGDA 3 (Algorithm 1) for privately solving (5), which is a noisy DP variation of two-timescale SGDA (Lin et al., 2020).
Now, we provide the first theoretical convergence guarantee for DP non-convex min-max optimization :
Theorem 3.2 (Privacy and Utility of Algorithm 1, Informal Version) .
Let ϵ ď 2 ln p 1 { δ q , δ P p 0 , 1 q .
Assume:
f p¨ , w ; z q is L θ -Lipschitz 4 and f p θ, ¨ ; z q is L w -Lipschitz for all θ, w, z ; and W Ă R d w is a convex, compact set.
Denote Φ p θ q “ max w P W F p θ, w q .
Choose σ 2 w “ 8 TL 2 w ln p 1 { δ q ϵ 2 n, σ 2 θ “ 8 TL 2 θ ln p 1 { δ q ϵ 2 n, and T ě ´ n ?
ϵ 2 m
¯.
Then, Algorithm 1 is p ϵ, δ q -DP.
Further, D T, η θ , η w such that
[Equation]
where d “ max p d θ , d w q .
(The expectation is solely over the algorithm.)
Theorem 3.2 shows that Algorithm 1 finds an approximate stationary point of (5). The stationarity gap depends on the number of samples n and model parameters d , as well as the desired level of privacy ϵ . Note that finding approximate stationary points is generally the best one can hope to do in polynomial time for non-convex optimization (Murty & Kabadi, 1985). The precise (scale-invariant) statement and proof of Theorem 3.2 is given in Appendix C, where we also provide a bound on the iteration complexity T .
The proof of Theorem 3.2 involves a careful analysis of how the Gaussian noises propagate through the optimization trajectories of θ t and w t .
Compared with DP non-convex minimization analyses (e.g. Wang et al.
; Hu et al.
; Ding et al.
(2021b)
; Lowy et al.
, the two noises required to privatize the solution of the min-max problem we consider complicates the analysis and requires careful tuning of η θ and η W .
Compared to existing analyses of DP min-max games in Boob & Guzmán (2021)
; Yang et al.
; Zhang et al.
(2022), which assume that f p¨ , w q is convex or PL , dealing with non-convexity is a challenge that requires different optimization techniques.
We expect Theorem 3.2 to be of general interest to the DP optimization community, beyond its applications to DP fair learning.
In this paper, however, our main use of Theorem 3.2 is that it leads to the first convergent stochastic algorithm (Algorithm 2) for DP fair learning, as we shall see next.
N OISY DP-FERMI FOR S TOCHASTIC P RIVATE F AIR ERM
Our proposed algorithm for solving (FERMI obj.), is given in Algorithm 2.
It simply specializes Algorithm 1 to the particular min-max problem (FERMI obj.)
that we wish to solve.
Algorithm
1: Input : θ 0 P R d θ , W 0 “ 0 P R k ˆ l , step-sizes p η θ , η w q , fairness parameter λ ě 0 , iteration number T , minibatch size | B t | “ m P r n s , set W Ă R k ˆ l , noise parameters σ 2 w , σ 2 θ .
2: Compute p P 1 { 2 S .
3: for t “ 0 , 1 , . . .
, T do 4:
Draw a mini-batch B t of data points tp x i , s i , y i qu i P B t 5:
Set θ t ` 1 Ð θ t ´ η θ | B t | ř i P B t r ∇ θ ℓ p x
i , y i ; θ t q ` λ p ∇ θ p ψ i p θ t , W t q ` u t qs , where u t „
N p 0 , σ 2 θ I d θ q .
Set W t `Ð Π W ´
W t ` λη w ”!
| B t | ř i P B t ´ W diag p F 1 p θ, x i q , . . .
, F l p θ, x i qq ` p P ´ 1 { 2 S
E r
s i
p
y
Ti p x i ; θ t q| x i , s i s )
` V t ı¯ , where V t is a k ˆ l matrix with independent random Gaussian entries p V t q j,r „ N p 0 , σ 2 w q .
7: end for 8: Pick ˆ t uniformly at random from t 1 , . .
, T u .
9: Return: θ T : “ θ ˆ t .
We provide the privacy guarantee of Algorithm 2 in Proposition 3.1.
Proposition 3.1.
Let ϵ ď 2 ln p 1 { δ q , δ P p 0 , 1 q , and T ě ´ n ?
ϵ 2 m ¯. Assume F p¨ , x q is L θ -Lipschitz for all x , and |p W t q r,j | ď D for all t P r T s , r P r k s , j P r l s .
Then, for σ 2 w ě 16 T ln p 1 { δ q ϵ 2 n 2 ρ and σ 2 θ ě 16 L 2 θ D 2 ln p 1 { δ q T ϵ 2 n 2 ρ , Algorithm 2 is p ϵ, δ q -DP with respect to the sensitive attributes for all data sets containing at least ρ -fraction of minority attributes.
See Appendix D for the proof.
The key step in the proof is bounding the sensitivity (Dwork et al., 2006) of the gradient updates in lines 5 and 6 of Algorithm 2.
5 Getting tight sensitivity bounds involves a careful analysis of how the expressions for the θ - and W -gradients change when we replace s i by s 1 i , noting that the terms that don’t depend on S will not contribute to sensitivity.
The convergence guarantee of Algorithm 2 is an immediate consequence of Theorem 3.2:
Corollary 3.1.
Assume the loss function ℓ is Lipschitz continuous with Lipschitz gradient, and p
P S p r q ě ρ ą 0 @
r P r k s .
In Algorithm 2, choose W to be a sufficiently large ball that contains
W ˚ p θ q : “ argmax W p F p θ, W q for every θ in some neighborhood of θ ˚ P argmin θ max W p F p θ, W q .
Then there exist algorithmic parameters such that the p ϵ, δ q
-DP Algorithm 2 returns ˆ θ T with
[Equation]
p 1 { δ q ϵn ¸ , The choice of W in Corollary 3.1 implies that (4) is equivalent to min θ max W P W p F p θ, W q , which is what our algorithm directly solves (c.f. (5)).
For large-scale models (e.g. deep neural nets), we typically have d θ " 1 and k, l “ O p 1 q , so that the convergence rate of Algorithm 2 is essentially immune to the number of labels and sensitive attributes.
In contrast, no existing works with convergence guarantees are able to handle non-binary classification ( l ą 2 ), even with full batches and a single binary sensitive attribute.
Lastly, note that while we return a uniformly random iterate in Algorithmfor our theoretical convergence analysis, we recommend returning the last iterate θ T in practice: our numerical experiments show strong performance of the last iterate.
N UMERICAL E XPERIMENTS
In this section, we evaluate the performance of our proposed approach (DP-FERMI) in terms of the fairness violation vs. test error for different privacy levels.
We present our results in two parts:
In Section 4.1, we assess the performance of our method in training logistic regression models on several benchmark tabular datasets.
Since this is a standard setup that existing DP fairness algorithms can handle, we are able to compare our method against the state-of-the-art baselines.
We carefully tuned the hyperparameters of all baselines for fair comparison.
We find that DP-FERMI consistently outperforms all state-of-the-art baselines across all data sets and all privacy levels .
These observations hold for both demographic parity and equalized odds fairness notions.
To quantify the improvement of our results over the state-of-the-art baselines, we calculated the performance gain with respect to fairness violation (for fixed accuracy level) that our model yields over all the datasets. We obtained a performance gain of demographic parity that was 79.648 % better than Tran et al. (2021b) on average, and 65.89% better on median. The average performance gain of equalized odds was 96.65% while median percentage gain was 90.02%..
In Section 4.2, we showcase the scalability of DP-FERMI by using it to train a deep convolutional neural network for classification on a large image dataset.
In Appendix E, we give detailed descriptions of the data sets, experimental setups and training procedure, along with additional results.
4. S TANDARD B ENCHMARK E XPERIMENTS : L OGISTIC R EGRESSION ON T ABULAR D ATASETS
In the first set of experiments we train a logistic regression model using DP-FERMI (Algorithm 2) for demographic parity and a modified version of DP-FERMI (described in Appendix E) for equalized odds.
We compare DP-FERMI against all applicable publicly available baselines in each expeiment.
4.1. D EMOGRAPHIC P ARITY
We use four benchmark tabular datasets: Adult Income, Retired Adult, Parkinsons, and Credit-Card dataset from the UCI machine learning repository (Dua & Graff (2017)).
The predicted variables and sensitive attributes are both binary in these datasets.
We analyze fairness-accuracy trade-offs with four different values of ϵ P t 0 .
5 , 1 , 3 , 9 u for each dataset.
We compare against state-of-the-art algorithms proposed in Tran et al.
(2021a)
and (the demographic parity objective of) Tran et al.
(2021b).
The results displayed are averages over 15 trials (random seeds) for each value of ϵ .
For the Adult dataset, the task is to predict whether the income is greater than $50K or not keeping gender as the sensitive attribute.
The Retired Adult dataset is the same as the Adult dataset, but with updated data.
We use the same output and sensitive attributes for both experiments.
The results for Adult and Retired Adult are shown in Figs.
2 and 6 (in Appendix E.2).
Compared to Tran et al.
(2021a;b), DP-FERMI offers superior fairness-accuracy tradeoffs at every privacy ( ϵ ) level.
In the Parkinsons dataset, the task is to predict whether the total UPDRS score of the patient is greater than the median or not keeping gender as the sensitive attribute.
Results for ϵ P t 1 , 3 u are shown in Fig.
See Fig. 8 in Appendix E for ϵ P t 0 .
5 , 9 u .
Our algorithm again outperforms the baselines Tran et al.
(2021a;b) for all tested privacy levels.
In the Credit Card dataset , the task is to predict whether the user will default payment the next month keeping gender as the sensitive attribute.
Results are shown in Fig.
7 in Appendix E.2.
Once again,
DP-FERMI provides the most favorable privacy-fairness-accuracy profile.
4.1. E QUALIZED O DDS
Next, we consider the slightly modified version of Algorithm 2, which is designed to minimize the
Equalized Odds violation by replacing the absolute probabilities in the objective with class conditional probabilities: see Appendix E.2.4 for details.
We considered the Credit Card and Adult datasets for these experiments, using the same sensitive attributes as mentioned above.
Results for Credit Card are shown in Fig.
Adult results are given in Fig.
9 in Appendix E.2.4.
Compared to Jagielski et al.
and the equalized odds objective in Tran et al.
(2021b), our equalized odds variant of DP-FERMI outperforms these state-of-the-art baselines at every privacy level . [Figure] [Figure]
4. L ARGE -S CALE E XPERIMENT : D EEP C ONVOLUTIONAL N EURAL N ETWORK ON I MAGE D ATASET
In our second set of experiments, we train a deep 9-layer VGG-like classifier (Simonyan & Zisserman, 2015) with d « 1.6 million parameters on the UTK-Face dataset (Zhang et al., 2017) using Algorithm 2.
We classify the facial images into 9 age groups similar to the setup in Tran et al.
(2022), while keeping race (containing 5 classes) as the sensitive attribute.
See Appendix E.3 for more details.
We analyze consider with four different privacy levels ϵ P t 10 , 25 , 50 , 100 u .
Compared to the tabular datasets, larger ϵ is needed to obtain stable results in the large-scale setting since the number of parameters d is much larger and the cost of privacy increases with d (see Corollary 3.1). Larger values of ϵ ą 100 were used in the baseline Jagielski et al.
for smaller scale experiments. The results in Fig. 5 empirically verify our main theoretical result: DP-FERMI converges even for non-binary classification with small batch size and non-binary sensitive attributes.
We took Tran et al.
(2021a;b) as our baselines and attempted to adapt them to this non-binary large-scale task.
We observed that the baselines were very unstable while training and mostly gave degenerate results [Figure] (predicting a single output irrespective of the input).
By contrast, our method was able to obtain stable and meaningful tradeoff curves.
Also, while Tran et al.
(2022) reported results on UTK-Face, their code is not publicly available and we were unable to reproduce their results.
[Figure]
C ONCLUDING R EMARKS
Motivated by pressing legal, ethical, and social considerations, we studied the challenging problem of learning fair models with differentially private demographic data.
We observed that existing works suffer from a few crucial limitations that render their approaches impractical for large-scale problems.
Specifically, existing approaches require full batches of data in each iteration (and/or exponential runtime) in order to provide convergence/accuracy guarantees.
We addressed these limitations by deriving a DP stochastic optimization algorithm for fair learning, and rigorously proved the convergence of the proposed method.
Our convergence guarantee holds even for non-binary classification (with any hypothesis class, even infinite VC dimension, c.f.
Jagielski et al.
with multiple sensitive attributes and access to random minibatches of data in each iteration.
Finally, we evaluated our method in extensive numerical experiments and found that it significantly outperforms the previous state-of-the-art models, in terms of fairness-accuracy tradeoff.
The potential societal impacts of our work are discussed in Appendix F.
A PPENDIX A R ELATED W ORK
The study of differentially private fair learning algorithms was initiated by Jagielski et al.
Jagielski et al.
(2019) considered equalized odds and proposed two DP algorithms: 1) an ϵ -DP post-processing approach derived from Hardt et al.
(2016a); and 2) an p ϵ, δ q -DP in-processing approach based on Agarwal et al.
The major drawback of their post-processing approach is the unrealistic requirement that the algorithm have access to the sensitive attributes at test time, which Jagielski et al.
(2019) admits “isn’t feasible (or legal) in certain applications.”
Additionally, post-processing approaches are known to suffer from inferior fairness-accuracy tradeoffs compared with in-processing methods.
While the in-processing method of Jagielski et al.
(2019) does not require access to sensitive attributes at test time, it comes with a different set of disadvantages: 1) it is limited to binary classification; 2) its theoretical performance guarantees require the use of the computationally inefficient (i.e. exponential-time) exponential mechanism (McSherry & Talwar, 2007); 3) its theoretical performance guarantees require computations on the full training set and do not permit mini-batch implementations ; 4) it requires the hypothesis class H to have finite VC dimension.
In this work, we propose the first algorithm that overcomes all of these pitfalls : our algorithm is amenable to multi-way classification with multiple sensitive attributes, computationally efficient, and comes with convergence guarantees that hold even when mini-batches of m ă n samples are used in each iteration of training, and even when VC p H q “ 8 .
Furthermore, our framework is flexible enough to accommodate many notions of group fairness besides equalized odds (e.g. demographic parity, accuracy parity).
Following Jagielski et al.
(2019), several works have proposed other DP fair learning algorithms.
None of these works have managed to simultaneously address all the shortcomings of the method of Jagielski et al.
The work of Xu et al.
(2019) proposed DP and fair binary logistic regression, but did not provide any theoretical convergence/performance guarantees.
The work of Mozannar et al.
(2020) combined aspects of both Hardt et al.
(2016a) and Agarwal et al.
(2018) in a two-step locally differentially private fairness algorithm.
Their approach is limited to binary classification .
Moreover, their algorithm requires n { 2 samples in each iteration (of their in-processing step), making it impractical for large-scale problems .
More recently, Tran et al.
(2021b) devised another DP in-processing method based on lagrange duality, which covers non-binary classification problems.
In a subsequent work, Tran et al.
(2021a) studied the effect of DP on accuracy parity in ERM, and proposed using a regularizer to promote fairness.
Finally, Tran et al.
(2022) provided a semisupervised fair “Private Aggregation of Teacher Ensembles” framework.
A shortcoming of each of these three most recent works is their lack of theoretical convergence or accuracy guarantees .
In another vein, some works have observed the disparate impact of privacy constraints on demographic subgroups (Bagdasaryan et al., 2019; Tran et al., 2021c).
B E QUALIZED O DDS V ERSION OF ERMI
If equalized odds (Hardt et al., 2016b) is the desired fairness notion, then one should use the following variation of ERMI as a regularizer Lowy et al.
[Equation]
Here ˆ p p Y ,S | Y denotes the empirical joint distribution of the predictions and sensitive attributes p p Y , S q conditional on the true labels Y .
In particular, if D R p p Y ; S | Y q “ 0 , then p Y and S are conditionally independent given Y (i.e. equalized odds is satisfied).
C F ORMAL S TATEMENT AND P ROOF OF T HEOREM 3.
Notation and Assumptions: Let f : R d θ ˆ R d w ˆ Z Ñ R , and F p θ, w q “ 1 n ř ni “ 1 f p θ, w ; z i q for fixed training data Z “ p z 1 , ¨ ¨ ¨ , z n q P Z n .
Let W Ă R d
w be a convex, compact set.
For any θ P R d θ , denote w ˚ p θ q P argmax w P W F p θ, w q and p Φ p θ q “ max w P W F p θ, w q .
Let ∆ Φ “ p Φ p θ 0 q ´ inf θ p Φ Z p θ q .
Recall that a function h is β -smooth if its derivative ∇ h is β -Lipschitz.
We write a À b if there is an absolute constant C ą 0 such that a ď Cb .
Assumption 1.
f p¨ , w ; z q is L θ -Lipschitz and β θ -smooth for all w P W , z P Z . 2.
f p θ, ¨ ; z q is L w -Lipschitz, β w -smooth, and µ -strongly concave on W for all θ P R d θ , z P Z . 3. } ∇ w f p θ, w ; z q ´ ∇ w f p θ 1 , w ; z q} ď β θw } θ ´ θ 1 } and } ∇ θ f p θ, w ; z q ´ ∇ θ f p θ, w 1 ; z q} ď β θw } w ´ w 1 } for all θ, θ 1 , w, w 1 , z . 4.
W has ℓ 2 diameter bounded by D ě 0 .
∇ w F p θ, w ˚ p θ qq “ 0 for all θ .
The first four assumptions are standard in (DP and min-max) optimization.
The fifth assumption means that W contains the unconstrained global minimizer w ˚ p θ q of F p θ, ¨q for all θ .
Hence (5) is equivalent to
[Equation]
This assumption is not actually necessary for our convergence result to hold, but we will need it when we apply our results to the DP fairness problem.
Moreover, it simplifies the proof of our convergence result.
We refer to problems of the form (5) that satisfy Assumption 1 as “(smooth) nonconvex-strongly concave min-max.”
We denote κ w : “ β w µ and κ θw : “ β θw µ .
We can now provide the complete, precise version of Theorem 3.2:
Theorem C.1 (Privacy and Utility of Algorithm 1, Formal Version) .
Let ϵ ď 2 ln p 1 { δ q , δ P p 0 , 1 q .
Grant Assumption 1.
Choose σ [Equation] [Equation] and T ě
[Equation]
Then Algorithm 1 is p ϵ, δ q -DP.
Further, if we choose η θ “ [Equation]
[Equation]
and T ě a
[Equation]
, then
[Equation]
In particular, if m ě min ˆ
[Equation]
then
[Equation]
The proof of Theorem C.1 will require several technical lemmas.
These technical lemmas, in turn, require some preliminary lemmas, which we present below.
We begin with a refinement of (Lin et al., 2020, Lemma 4.3):
Lemma C.1.
Grant Assumption 1.
Then Φ is 2 p β θ ` β θw κ θw q -smooth with ∇ Φ p θ q “ ∇ θ F p θ, w ˚ p θ qq , and w ˚ p¨q is κ w -Lipschitz.
Proof.
The proof follows almost exactly as in the proof of (Lin et al., 2020, Lemma 4.3), using
Danskin’s theorem, but we carefully track the different smoothness parameters with respect to w and θ (and their units) to obtain the more precise result.
Lemma C.2 (Lei et al.
Let t a l u l Pr n s be an arbitrary collection of vectors such that ř nl “ 1 a l “ 0 .
Further, let S be a uniformly random subset of r n s of size m .
Then,
[Equation]
Lemma C.3 (Co-coercivity of the gradient) .
For any β -smooth and convex function g , we have
[Equation]
for all a, b P domain p g q .
Having recalled the necessary preliminaries, we now provide the novel technical ingredients that we’ll need for the proof of Theorem C.1.
The next lemma quantifies the progress made in minimizing Φ from a single step of noisy stochastic gradient descent in θ (i.e. line 4 of Algorithm 1):
Lemma C.4.
For all t P r T s , the iterates of Algorithm 1 satisfy
[Equation]
conditional on θ t ´ 1 , w t ´ 1 .
Proof.
Let us denote r g : “
[Equation]
First condition on the randomness due to sampling and Gaussian noise addition.
By smoothness of Φ (see Lemma C.1)
[Equation]
we have
[Equation]
Taking expectation (conditional on θ t ´ 1 , w t ´ 1 )
[Equation]
In the first inequality, we used the fact that the Gaussian noise has mean zero and is independent of p θ t ´ 1 , w t ´ 1 , Z q , plus the fact that E g “ ∇ θ F p θ t ´ 1 , w t ´ 1 q .
In the second inequality, we used
Lemma C.2 and Lipschitz continuity of f .
In the third and fourth inequalities, we used Young’s inequality and Cauchy-Schwartz.
For the particular η θ prescribed in Theorem C.1, we obtain:
Lemma C.5.
Grant Assumption 1.
If η θ “
[Equation]
q , then the iterates of Algorithm 1 satisfy ( @ t ě 0 )
[Equation]
Proof.
By Lemma C.
[Equation]
we have
[Equation]
In the second inequality
[Equation]
we simply used the definition of η θ .
In the third inequality, we used the fact that ∇ Φ p θ t q “ ∇ θ F p θ t , w ˚ p θ t qq (see Lemma C.1) together with Assumption 1 (part 3).
Next, we describe the progress made in the w t updates:
Lemma C.6.
Grant Assumption 1.
If η w “ 1 β w , then the iterates of Algorithm 1 satisfy ( @ t ě 0 )
[Equation]
Proof.
Fix any t and denote δ t : “ E } w ˚ p θ t q ´ w t } 2 : “ E } w ˚ ´ w t } 2 .
We may assume without loss of generality that f p θ, ¨ ; z q is µ -strongly convex and that w t ` 1 “ Π W r w t ´β w
[Equation]
[Equation]
Now,
[Equation]
ď
[Equation]
Further,
[Equation]
using independence and Lemma C.2 plus Lipschitz continuity of f in the first inequality and Lemma C.3 (plus Assumption 1 part 5) in the second inequality.
This implies
[Equation]
Therefore,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation]
[Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
‰ , by Young’s inequality, (7), and Lemma C.1.
Since ´
[Equation] [Equation]
12 κ w , we obtain
[Equation]
as desired.
We are now prepared to prove Theorem C.1.
Proof of Theorem C.1.
Privacy: This is an easy consequence of (Abadi et al., 2016, Theorem 1) (with precise constants obtained from the proof therein, as in Bassily et al.
(2019)) applied to both the min (descent in θ ) and max (ascent in w ) updates.
Unlike Abadi et al.
(2016), we don’t clip the gradients here before adding noise, but the Lipschitz continuity assumptions (Assumption 1 parts 1 and 2) imply that the ℓ 2 -sensitivity of the gradient updates in lines 4 and 5 of Algorithm 1 are nevertheless bounded by 2 L θ { m and 2 L w { m , respectively.
Thus, (Abadi et al., 2016, Theorem 1) still applies.
Convergence: Denote ζ : “
[Equation]
, δ t
[Equation]
˚ p θ t q ´ w t } 2 , and
[Equation]
so that Lemma C
[Equation]
6 reads as
[Equation]
for all t P r T s .
Applying (8) recursively
[Equation]
we have
[Equation]
Combining this inequality with Lemma C.5, we get
[Equation]
Summing over all t P r T s and re-arranging terms yields
[Equation]
Now, ζ ď 1 ´ 14 κ w , which implies that
T ÿ t “ζ t ď 4 κ w and
T ÿ t “t ´ 1 ÿ j “ζ t ´ 1 ´ j ď
[Equation]
Hence
T
T ÿ t “
[Equation]
Since η 2 θ β 2 θw κ 2 w κ 2 θw ď
[Equation]
we obtain
[Equation]
Our choice of T then implies
[Equation]
Finally, our choice of sufficiently large m yields the last claim in Theorem C.
[Equation]
D P ROOF OF P ROPOSITION 3.
Proposition D.1 (Re-statement of Proposition 3.1) .
Let ϵ ď 2 ln p 1 { δ q , δ P p 0 , 1 q , and T ě ´ n ?
ϵ 2 m ¯.
Assume F p¨ , x q is L θ -Lipschitz for all x , and |p W t q r,j | ď D for all t P r T s , r P r k s , j P r l s .
Then, for σ 2 w ě 16 T ln p 1 { δ q ϵ 2 n 2 ρ and σ 2 θ ě 16 L 2 θ D 2 ln p 1 { δ q T ϵ 2 n 2 ρ , Algorithm 2 is p ϵ, δ q -DP with respect to the sensitive attributes for all data sets containing at least ρ -fraction of minority attributes.
Proof.
By the moments accountant (Abadi et al., 2016, Theorem 1), it suffices to bound the sensitivity of the gradient updates by ∆ 2 θ ď 8 D 2 L 2 θ m 2 ρ and ∆ 2 w ď 8 m 2 ρ .
Here
[Equation]
and Z „ Z 1 means that Z and Z 1 are two data sets (both with ρ -fraction of minority attributes) that differ in exactly one person’s sensitive attributes:
i.e. s i ‰ s
i for some unique i P r n s , but z j “ z 1 j for all j ‰ i and p x
i , y i q “ p x 1 i , y 1 i q .
Likewise,
[Equation]
Now, compute
[Equation]
and notice that only the second term depends on S .
Therefore
[Equation]
we can bound the ℓ 2 -sensitivity of the θ -gradient updates by:
[Equation]
using Lipschitz continuity of F p¨
[Equation]
x q
[Equation]
the assumption that W has diameter bounded by D
[Equation]
the assumption that the data sets have at least ρ -fraction of sensitive attribute r for all r P r k s .
Similarly, for the W -gradients, we have
[Equation]
since ř lj “ 1 F j p θ, x i q 2 ď ř lj “ 1 F j p θ, x i q “ 1 .
This establishes the desired privacy guarantee for Algorithm 2.
E N UMERICAL E XPERIMENTS : A DDITIONAL D ETAILS AND R ESULTS E. M EASURING D EMOGRAPHIC P ARITY AND E QUALIZED O DDS V IOLATION
We used the expressions given in (9) and (10) to measure the demographic parity violation and the equalized odds violation respectively.
We denote Y to be the set of all possible output classes and S to be the classes of the sensitive attribute.
P r E s denotes the empirical probability of the occurrence of an event E.
[Equation]
E. T ABULAR D ATASETS E.2. M ODEL D ESCRIPTION AND E XPERIMENTAL D ETAILS
Demographic Parity: We split each dataset in a 3:1 train:test ratio.
We preprocess the data similar to Hardt et al.
(2016a) and use a simple logistic regression model with a sigmoid output O “ σ p Wx ` b q which we treat as conditional probabilities p p p
y “ i | x q .
The predicted variables and sensitive attributes are both binary in this case across all the datasets.
We analyze fairness-accuracy trade-offs with four different values of ϵ P t 0 .
5 , 1 , 3 , 9 u for each dataset.
We compare against state-of-the-art algorithms proposed in Tran et al.
(2021a)
and (the demographic parity objective of) Tran et al.
(2021b).
The tradeoff curves for DP-FERMI were generated by sweeping across different values for λ P r 0 , 2 .
5 s .
The learning rates for the descent and ascent, η θ and η w , remained constant during the optimization process and were chosen from r 0 .
01 s .
Batch size was 1024.
We tuned the ℓ 2 diameter of the projection set W and θ -gradient clipping threshold in r 1 , 5 s in order to generate stable results with high privacy (i.e. low ϵ ).
Each model was trained for 200 epochs.
The results displayed are averages over 15 trials (random seeds) for each value of ϵ .
Equalized Odds: We replicated the experimental setup described above, but we took ℓ 2 diameter of W and the value of gradient clipping for θ to be in r 1 , 2 s .
Also, we only tested three values of ϵ P t 0 .
5 , 1 , 3 u .
E.2. D ESCRIPTION OF D ATASETS
Adult Income Dataset: This dataset contains the census information about the individuals.
The classification task is to predict whether the person earns more than 50k every year or not.
We followed a preprocessing approach similar to Lowy et al.
After preprocessing, there were a total of 102 input features from this dataset.
The sensitive attribute for this work in this dataset was taken to be gender.
This dataset consists of around 48,000 entries spanning across two CSV files, which we combine and then we take the train-test split of 3:1.
Retired Adult Income Dataset: The Retired Adult Income Dataset proposed by Ding et al.
(2021a) is essentially a superset of the Adult Income Dataset which attempts to counter some caveats of the
Adult dataset.
The input and the output attributes for this dataset is the same as that of the Adult
Dataset and the sensitive attribute considered in this work is the same as that of the Adult.
This dataset contains around 45,000 entries.
Parkinsons Dataset: In the Parkinsons dataset, we use the part of the dataset which had the UPRDS scores along with some of the features of the recordings obtained from individuals affected and not affected with the Parkinsons disease.
The classification task was to predict from the features whether the UPDRS score was greater than the median score or not.
After preprocessing, there were a total of 19 input features from this dataset and the sensitive attribute for this dataset was taken to be gender.
This dataset contains around 5800 entries in total.
We took a train-test split of 3:1.
Credit Card Dataset: This dataset contains the financial data of users in a bank in Taiwan consisting of their gender, education level, age, marital status, previous bills, and payments.
The assigned classification task is to predict whether the person defaults their credit card bills or not, essentially making the task if the clients were credible or not.
We followed a preprocessing approach similar to Lowy et al.
After preprocessing, there were a total of 85 input features from this dataset.
The sensitive attribute for this dataset was taken to be gender.
This dataset consists of around 30, entries from which we take the train-test split of 3:1.
UTK-Face Dataset
: This dataset is a large scale image dataset containing with an age span from 0 to 116.
The dataset consists of over 20,000 face images with details of age, gender, and ethnicity and covers large variation in pose, facial expression, illumination, occlusion, resolution.
We consider the age classification task with 9 age groups similar to the experimental setup in Tran et al.
We consider the sensitive attribute to be the ethnicity which consists of 5 different classes.
E.2. D EMOGRAPHIC P ARITY
Retired Adult Results: See Fig.
6 for our results on Retired Adult Dataset.
The results are qualitatively similar to the reusults reported in the main body: our algorithm (DP-FERMI) achieves the most favorable fairness-accuracy tradeoffs across all privacy levels.
Credit Card Results: See Fig.
7 for our results on Credit Card Dataset.
DP-FERMI offers superior fairness-accuracy-privacy profile compared to all applicable baselines.
Additional Results for Parkinsons Dataset: More results for Parkinsons are shown in Fig.
DP-FERMI offers the best performance
. E.2. E QUALIZED O DDS
Equalized Odds Variation of DP-FERMI Algorithm:
The (FERMI obj.) minimizes the Exponential Renyi Mutual Information (ERMI) between the output and the sensitive attributes which essentially leads to a reduced demographic parity violation.
The equalized-odds condition is more constrained and enforces the demographic parity condition for data grouped according to labels.
For the equalized-odds, the ERMI between the predicted and the sensitive attributes is minimized conditional to each of the label present in the output variable of the dataset.
So, the FERMI regularizer is split into as many parts as the number of labels in the output.
This enforces each part of the FERMI regularizer to minimize the ERMI while an output label is given/constant.
Each part has its own unique W that is maximized in order to create a stochastic estimator for the ERMI with respect to each of the output labels.
Adult Results: Results for the equalized odds version of DP-FERMI on Adult dataset are shown in Fig.
Our approach outperforms the previous state-of-the-art methods.
Retired Adult Results: (Initial) Results for the equalized odds version of DP-FERMI on the retiredadult dataset are shown in Fig.
Our approach outperforms Tran et al.
(2021b)
and we are currently tuning our non-private and/or non-fair versions of our models along with Jagielski et al.
[Figure]
[Figure]
We split the dataset in a 3:1 train:test ratio.
Batch size was 64.
128 x 128 normalized images were used as input for our model.
We tuned the ℓ 2 diameter of W and the value of gradient clipping for θ to be in r 1 , 2 s and learning rates for the descent and ascent, η θ and η w , remained constant during the optimization process and were chosen as 0.001 and 0.005 respectively.
We analyze the fairness-accuracy trade-offs with five different values of ϵ P t 10 , 25 , 50 , 100 , 200 u .
The results displayed were averaged over observations obtained from 5 different randomly chosen seeds on each configuration of ϵ and a dataset.
Each model was trained for 150 epochs.
The tradeoff curves for this set of experiments were obtained by sweeping across different values for λ P r 0 , 500 s .
[Figure] [Figure]
F S OCIETAL I MPACTS
In this paper, we considered the socially consequential problem of privately learning fair models from sensitive data.
Motivated by the lack of scalable private fair learning methods in the literature, e developed the first differentially private (DP) fair learning algorithm that is guaranteed to converge with small batches ( stochastic optimization ).
We hope that our method will be used to help companies, governments, and other organizations to responsibly use sensitive, private data.
Specifically, we hope that our DP-FERMI algorithm will be useful in reducing discrimination in algorithmic decisionmaking while simultaneously preventing leakage of sensitive user data.
The stochastic nature of our algorithm might be especially appealing to companies that are using very large models and datasets.
[Figure]
[Figure]
On the other hand, there are also some important limitations of our method that need to be considered before deployment.
One caveat of our work is that we have assumed that the given data set contains fair and accurate labels.
For example, if gender is the sensitive attribute and “likelihood of repaying a loan” is the target, then we assume that the training data accurately describes everyone’s financial history without discrimination.
If training data is biased against a certain demographic group, then it is possible that our algorithm could amplify (rather than mitigate) unfairness.
See e.g. Kilbertus et al.
Bechavod et al.
for further discussion.
Another important practical consideration is how to weigh/value the different desiderata (privacy, fairness, and accuracy) when deploying our method.
As shown in prior works (e.g., Cummings et al.
and re-enforced in the present work, there are fundamental tradeoffs between fairness, accuracy, and privacy: improvements in one generally come at a cost to the other two.
Determining the relative importance of each of these three desiderata is a critical question that lacks a clear or general answer.
Depending on the application, one might be seriously concerned with either discrimination or privacy attacks, and should calibrate ϵ and λ accordingly.
Or, perhaps very high accuracy is necessary for a particular task, with privacy and/or fairness as an afterthought.
In such a case, one might want to start with very large ϵ and small λ to ensure high accuracy, and then gradually shrink ϵ and/or increase λ to improve privacy/fairness until training accuracy dips below a critical threshold.
A thorough and rigorous exploration of these issues could be an interesting direction for future work.