How unfair is private learning?
Abstract
As machine learning algorithms are deployed on sensitive data in critical decision making processes, it is becoming increasingly important that they are also private and fair.
In this paper, we show that, when the data has a long-tailed structure, it is not possible to build accurate learning algorithms that are both private and results in higher accuracy on minority subpopulations.
We further show that relaxing overall accuracy can lead to good fairness even with strict privacy requirements.
To corroborate our theoretical results in practice, we provide an extensive set of experimental results using a variety of synthetic, vision (CIFAR-10 and CelebA), and tabular (Law School) datasets and learning algorithms.
INTRODUCTION
In recent years, reliability of machine learning algorithms have become ever more important due to their widespread use in daily life.
Fairness and privacy are two instances of such reliability traits that are desirable but often absent in modern machine learning algorithms [10, 14, 39]As a result, there has been a ﬂurry of recent works that aim to improve these properties in commonly used learning algorithms.
However, most of these works discuss these two properties individually with relatively less attention paid to how they affect each other.
There is a multitude of deﬁnitions for privacy and fairness in their respective literatures.
Perhaps the most widespread statistical notion of privacy is that of Differential Privacy [16] and its slightly relaxed variant, Approximate Differential
Privacy [17].
Despite its marginally weaker privacy guarantees, Approximate Differential Privacy enjoys better theoretical guarantees in terms of statistical complexity for learning [7, 22].
It is also more widely used in practice [1, 49].
Thus, we always use approximate differential privacy in this paper and for the sake of brevity, refer to it as differential privacy (DP).
Intuitively, DP limits the amount of inﬂuence any single data point has on the output of the DP algorithm.
This ensures that DP algorithms do not leak information about whether any particular data point was given as input to the algorithm.
While DP was initially popular as a theoretical construct, it has recently been put to practical use by large companies [20, 41] and governments [32] alike.
Its popularity is largely due to its strong privacy guarantees, ease of implementation, and the quantitative nature of differential privacy.
There are many notions of fairness in machine learning [18, 26, 27].
Minority or worst group accuracy [15, 29] and its difference from the overall accuracy is a common notion of fairness used in recent works.
We deﬁne this difference as accuracy discrepancy and use it to measure the degree of unfairness in this paper.
However, we expect our results to translate to other related fairness metrics as well.
Sagawa* et al.
[37] observed that robust optimisation methods (under appropriate regularisations) obtain higher minority group accuracy but at the cost of a lower overall accuracy compared to vanilla training.
Several other works have also observed this behavior in practice thereby suggesting a possible trade-off between overall accuracy and fairness metrics.
Subsequent works including Goel et al.
[25] and Menon et al.
[34] have tried to minimise this trade-off.
While there have been a large body of works that aim to minimising the trade-off between accuracy and fairness [15, 25, 37] and between accuracy and privacy [17, 24], there is relatively few works that investigate the intersection of privacy, fairness, and accuracy.
In this paper, we provide theoretical and experimental results to show that private and accurate algorithms are necessarily unfair.
We further show that achieving privacy and fairness simultaneously leads to inaccurate algorithms.
Contributions Our main contributions can be stated as —
• In Theorem 1 and
we provide asymptotic lower
bounds for unfairness (accuracy discrepancy) of DP algorithms, that are accurate, showing that privacy and accuracy comes at the cost of fairness.
• In Theorem 3, we show that in a very strict privacy regime, fairness can be achieved at the cost of accuracy.
• In Section 3, we conduct experiments using multiple architectures on synthetic and real world datasets (CelebA, CIFAR-10, and Law School) to validate our theory.
Related works
It is now well understood that by imposing these additional conditions of DP more data are required to achieve high accuracy.
A string of theoretical works [5, 9, 22] have shown that the sample complexity of learning certain concept classes privately and accurately can be arbitrarily larger than learning the same classes non-privately (i.e. with high accuracy but without privacy).
On the other hand, it is easy to guarantee any arbitrary level of differential privacy if high accuracy is not desired.
This can be achieved by simply composing the output of an accurate classiﬁer with a properly calibrated randomised response mechanism [45].
This allows for a tradeoff between differential privacy and accuracy.
One of the most popular notions of fairness is group fairness that compares the performance of the algorithm on a minority group with other groups in the data.
A popular instantiation of this, especially for deep learning algorithms, is comparing the accuracy on the minority group against the entire population [15, 25, 29, 37].
In the fairness literature, Buolamwini and Gebru [10], Raz et al.
[36] shows extensively that this discrepancy is large between different groups of people for popular facial recognition systems.
DP-SGD [1] is widely used algorithm for implementing differentially private deep learning models.
Bagdasaryan et al.
[4] provides some experimental evidence that DP-SGD can have disparate impact on accuracy.
Conversely, Chang and
Shokri [11] shows, experimentally, that fairness aware machine learning algorithms suffer from less privacy.
However, unlike these works, we provide theoretical results that are model agnostic and that discuss the dependance of the tradeoff on the subpopulation sizes and frequencies.
Cummings et al.
[13] and Agarwal [2] were one of the ﬁrst to consider the impact of privacy on fairness theoretically.
They construct a distribution where any algorithm that is always fair and private will necessarily output a trivial constant classiﬁer, thereby suggesting a tradeoff between fairness and privacy.
However, there are multiple drawbacks with their work.
First, their work only discusses pure differential privacy which is not only theoretically more restrictive than approximate differential privacy [6, 7, 22] but also rarely used in practice.
Second, their proof heavily relies on it being pure differential privacy and the algorithm being always fair; and their proofs are not amenable to relaxations of these assumptions.
Further, they do not provide experiments to corroborate their theory perhaps due to the unrealistic requirements of pure DP.
On the other hand, we look at approximate DP (which is a stronger result than pure DP), construct bounds for both fairness and error, and provide experimental results to support our theory.
Perhaps, most closely related to our work is that of Feldman [21], who studies, mainly, the impact of memorisation on test accuracy for long-tailed distributions.
However, neither does their work foray into differential privacy nor into fairness.
THEORETICAL RESULTS
The main contribution of our work is to provide a qualitative explanation for why and when differentially private algorithms cannot be simultaneously accurate and fair.
Real world data distributions often contain a large number of subpopulations with very few examples in each of them and a few subpopulations with a large number of examples.
For example, Figure 1 (left) depicts the distribution of subpupoluations in CelebA. Using the 40 attributes of the CelebA dataset [31], we partition the training set (of size m = 160 k ) into 2 40 subpopulation bins.
The blue shaded area shows the group of the subpopulations with large number of examples (probability mass greater than 1 m ) and the red shaded area corresponds to subpopulations which contain just one example in them.
We refer to the subpopulations in the red area as minority subpopulations and the subpopulations in the blue area as majority subpopulations .
This longtailed structure over subpopulations have also been observed in Zhu et al.
[51] in other vision datasets like the SUN [48] and PASCAL [19] datasets.
Babbar and Schölkopf [3] observes this structure in extreme multilabel classiﬁcation datasets like Amazon-670K [33] and Wikipedia-31k [8] datasets.
Various other works [12, 30, 42, 44] have shown this structure in a range of datasets including eBird [40], Visual
Genome [30], Pasadena trees [46], and iNaturalist [43].
2.1 PROBLEM SETUP
Mathematically, the large number of small subpopulations discussed above constitute the long tail of the distribution.
We use this structure of data distributions to illustrate the tension between accuracy and fairness of private algorithms.
For our theoretical results, we view each subpopulation as an element of a discrete set X without any intrinsic structure such as distance.
This distribution is inspired by the use of a similar distribution in Feldman [21].Next, we deﬁne a distribution over the subpopulations in X that reﬂect the long-tailed structure.
Deﬁnition 1 ( ( p, N, k ) -long-tailed distribution on X ) .
Given p ∈ (0 , 1) , N ∈ N , and 1 < k (cid:28) N , deﬁne two groups (i) the group of majority subpopulations X 1 ⊂ X where | X 1 | = (1 − p ) k
and (ii) the group of minority subpopulations X 2 ⊂ X \ X 1 , where | X 2 |
= N .
Now, deﬁne the distribution Π p,N,k as
[Equation]
We use the terms group of majority subpopulations and majority group interchangeably to denote X 1 and the terms group of minority subpopulations and minority group to refer to X 2 respectively.
We provide an illustration of the distribution in Figure 1 (left).
Here, p denotes the total probability mass of the group of minority subpopulations under Π p,N,k and N denotes the number of minority subpopulations.
We let N go to ∞ and treat k as a constant.
Thus, for the sake of simplicity, we remove k from the notation of the distribution.
Note that each minority subpopulation i.e. each element in
X 2 has a probability mass of the order of O (cid:0) 1 N (cid:1) which is much smaller than 1 k = Ω (1) , i.e. the probability mass of each element in X 1 .
Note that in the distribution for CelebA (Figure 1 (left))
the probability masses of the majority subpopulations not exactly equal to 1 k as in the distribution of Deﬁnition 1.
However, all our results hold true even if different majority subpopulations have different probability masses as long as they satisfy Π p,N ( x )
= Ω (cid:0) 1 k (cid:1) for some k
= O (1) and all x ∈ X 1 .
We set them to 1 k only for simplicity of the theoretical results.
As we deal with a multiclass classiﬁcation setup, we also deﬁne a label space Y and a function space F of labelling functions.
We also use F to represent a distribution on the function space F and refer to this distribution as the label prior.
Our results do not restrict the size of Y and hence, cover both binary and multi-class classiﬁcation settings.
Finally, to generate a dataset of size m from a ( p, N ) - long-tailed distribution Π p,N on X , ﬁrst sample an unlabelled dataset S = { x 1 , · · · , x m } of size m from Π p,N .
Then, generate the labelled dataset S f
= { ( x 1 , f ( x 1 )) , . . .
, ( x m , f ( x m )) } using a labelling function f ∼ F .
In all our theoretical results, we consider an asymptotic regime where Nm → c as N, m → ∞ .
This is common in high-dimensional statistics where the number of dimensions often grows to ∞ along with the sample size.
Intuitively, c quantiﬁes the hardness of the learning problem as it is invsersely proportional to the number of data points observed per minority subpopulation.
Next, we deﬁne the error and fairness measure of an algorithm on the distribution deﬁned above.
Consider a domain X , a label space Y , a set of labelling functions F , a label prior F , and a distribution Π p,N on X .
2.2 PRIVACY, ERROR, AND FAIRNESS
In the context of this paper, a differentially private (randomised) learning algorithm generates similar distributions over classiﬁers when trained on neighbouring datasets .
Two datasets are neighbouring when they differ in one entry .Formally,
Deﬁnition 2 (Approximate Differential Privacy [16, 17]) .
Given any two neighbouring datasets S, S (cid:48) , (cid:15) > 0 , and δ ∈ (0 , 1) an algorithm A is called ( (cid:15), δ ) -differentially private if for all sets of outputs Z , the following holds
[Equation]
Next, we deﬁne the error of an algorithm in our problem setup.
For a randomised learning algorithm A , a distribution Π p,N , a label prior F , we can deﬁne the error of the algorithm as follows
Deﬁnition 3 (Error measure on Π p,N ) .
The error of the algorithm A trained on a dataset of size m from the distribution Π p,N with respect to a label prior F is
[Equation]
where I {·} is the indicator function and the expectation is over S ∼ Π mp,N , f ∼ F , h ∼ A ( S f ) , and x ∼ Π p,N .
Note that the error metric with an expectation over F , was previously used in Feldman [21].
In fact, for the purpose of lower bounds on fairness, this is a stronger notion than the worst case f ∈ F as, here, the lower bound is on the expectation which is stronger than a lower bound on the worst case.
Next, we deﬁne the accuracy discrepancy of an algorithm, represented by Γ over the distribution Π p,N .
For this purpose, for any Π p,N , deﬁne the marginal distribution on the group of minority subpopulations X 2 as Π
[Equation]
p,N
[Equation]
Deﬁnition 4 (Accuracy discrepancy on Π p,N ) .
For
X, F , Π p,N , and Π 2 p,N as deﬁned above, the accuracy discrepancy of the algorithm A trained on a dataset of size m on the distribution Π p,N , with respect to the label prior F , is
[Equation]
where err m ( · ) is as deﬁned in Deﬁnition 3.
This notion of group fairness is similar to the notion of subgroup performance gap used in Goel et al.
[25] and has also been implicitly used in multiple works Du et al.
[15], Koh et al.
[29], Sagawa* et al.
[37] as discussed before.
It has also been used in works related to the privacy [4, 11] and fairness literature [10, 36].
2.3 PRIVACY AND ACCURACY AT THE COST OF FAIRNESS
The theoretical results below use the deﬁnitions and notations described above and summarised in Table 1 in the supplementary.
First, in Theorem 1, we show that there are distributions (within the family of distributions deﬁned in Deﬁnition 1) where any accurate and approximately differentially private algorithm (with additional assumptions) is necessarily unfair.
In Theorem 2, we relax some of the stronger assumptions and present a more general result.
As discussed above, the dataset size m and the number of minority subpopulations N both simultaneously go to ∞ and the ratio Nm is asymptotically equal to
Nm ∼ c .
Throughout this section, we also use the notation err
( A , Π p,N , F ) =
lim m,N →∞ err ( A , Π p,N , F ) and Γ ( A , Π p,N , F ) =
lim m,N →∞ Γ ( A , Π p,N , F ) to denote the asymptotic limit for the error and the accuracy discrepancy metrics as m, N → ∞ .
Theorem 1.
For (cid:15) ∈ (0 , 1) and δ ∈ (0 , 0 . 01) , consider any ( (cid:15), δ ) -DP
algorithm
A that does not make mistakes on subpopulations occurring more than once in the dataset.
Then, there exists a family of label priors F where for any α ∈ (0 , 1) , there exists p ∈ (0 , 1 / 2 ) , c > 0 such that, err ( A , Π p,N , F ) ≤ α and Γ ( A , Π p,N , F ) ≥ 0 .
where Nm → c as N, m → ∞ .
A detailed version along with its full proof is presented in Appendix A.1.
An immediate consequence of unfairness Γ m being greater than 0 .
5 , coupled with the very small error α , is that the algorithm essentially behaves worse than random chance on the minority subpopulations thereby rendering the algorithm useless for these subpopulations.
Proof sketch Here we present a proof sketch and discuss the results.
By Deﬁnition 1, the probability mass of each majority subpopulation is Ω (1) whereas the probability mass of each minority subpopulation is O (cid:0) 1 m (cid:1) .
Thus, for a large enough dataset (i.e. large m ), we show that almost all majority subpopulations appear more than once and consequently, the algorithm in Theorem 1 makes very less mistakes on the majority subpopulations.
As a result, the error and the accuracy discrepancy are both caused by mistakes, majorly, on the minority subpopulations.
We, then, count the number of subpopulations that do not appear or appear just once among the minority subpopulations and use that to provide the upper bounds for error and lower bound for unfairness (accuracy discrepancy).
As these bounds are expressed in terms of p and c , the proof then follows by showing the existence of p, c that satisfy the inequalities in the theorem.
While Theorem 1 shows the existence of distributions under which private and accurate algorithms are necessarily unfair, in Theorem 2, we provide a quantitative lower bound for unfairness of private algorithms.
In addition, we also generalise Theorem 1 to include a much broader set of algorithms.
For this, we state two assumptions below.
For any (cid:96) ∈ N , deﬁne S (cid:96) to denote the set of examples that appear exactly (cid:96) times in S .
Given s 0 ∈ N and p 1 , p 2 ∈ (0 , 1) , we state that the algorithm A satisﬁes the assumptions A1 and A2 if the following conditions are satisﬁed by the A for all datasets S .
Assumption on algorithm
• (Accuracy) For all (cid:96) > s 0 and x ∈ S (cid:96)
[Equation]
• (Privacy) For all (cid:96) ≤ s 0 and x ∈ S (cid:96)
[Equation]
Assumption A1 essentially requires the algorithm to have small overall train error.
Note that since our domain is discrete, high training accuracy translates to high test accuracy in particular as the sample size approaches inﬁnity.
When pis small, algorithm A obtains low training (and hence test) error on frequently occurring or typical data (i.e. (cid:96) ≥ s 0 )
On the other hand, Assumption A2 implies that the algorithm is incorrect on subpopulations that are rare in the training set, (cid:96) ≤ s 0 , with a probability of at least ≥ 1 − p 2 .
We refer to this assumption as the privacy assumption because for ( (cid:15), δ ) -DP algorithms, it holds true for certain s 0 , p 2 that depend on (cid:15), δ , as discussed later in Lemma 1.
We note that the assumption in Theorem 1, that the algorithm does not make mistakes on subpopulations that appear more than once is an instantiation of assumption A1 and A2 with the parameters s
= 1 , p 1 = 0 , and p 2 = 1 .
In the next result, we present a detailed result showing how unfairness of a DP algorithm varies with respect to any instantiations of the assumptions A1 and A2, privacy parameters, and distributional parameters.
For easier interpretation, we show a simpliﬁed version in Theorem 2 and highlight the key takeaways, and provide a detailed version in Appendix A.2.
Consider X, F as deﬁned in Section 2.1.
Theorem 2.
For any p ∈ (0 , 1 / 2 ) , c > 0 such that p/c ≤ 1 , consider the distribution Π p,N where Nm → c as N, m goes to ∞ .
Also, for any (cid:15), δ > 0 , consider an ( (cid:15), δ ) -DP
algorithm
A that satisﬁes assumptions A1 and A2 with s
= o ( m ) and p 1 , p 2 ∈ (0 , 1) .
Then, the accuracy discrepancy is lower bounded as follows
[Equation]
where γ 0 is some constant depending on c, s 0 , p, p 2 , (cid:15), δ, and F .
Further, the error of the algorithm is upper-bounded as err ( A , Π p,N , F ) ≤ (1 − p 1 ) pα 0 + p 1 and α 0 depends on c, s 0 and p .
If s 0 → ∞ as m → ∞ , then in the asymptotic limit c, m → ∞ , γ 0 increases as 1 − O (cid:16) e − cs 0 s 0 √ c (cid:17) .
As p → 0 , α 0 increases as 1 − O (cid:0) √ pe − 1 / p (cid:1)
The detailed expressions of c 0 , α 0 , and γ 0 (including its dependance on (cid:15), δ, p 1 , and p 2 ) can be found in Theoremin Appendix A.2.
We now brieﬂy discuss how the theorem characterizes the effect of privacy and accuracy (via s 0 ) of the algorithm and the distribution and number of of subpopulations (via p, c respectively) on the accuracy discrepancy.
First note that when the ratio c of the number of minority subpopulations with respect to the sample size is large, more minority subpopulations appear infrequently in the observed dataset and when s 0 is relatively large (but still o ( m ) )
most infrequent subpopulations are misclassiﬁed.
Indeed, theorem 2 indicates that as c, s 0 increase, γ 0 and hence unfairness increases, while the average error decreases.
Intuitively, this is because minority subpopulations appear infrequently and the algorithm is less likely to memorise infrequent subpopulations.
Therefore, the lower bound on unfairness increases with c and s 0 , and in the asymptotic limit of m → ∞ , accuracy discrepancy Γ approaches (1 − p ) .
Remark 1.
We note that one can omit Assumption A2 in Theorem 2 at the cost of restricting
[Equation]
where ||F|| ∞ =
max x ∈ X,y ∈Y P f ∼F [ f ( x ) =
y ] .
This follows from simple algebra on Equation (16) in the proof of Theorem 2 and Lemma 1 below.
When the algorithm is private enough, i.e. (cid:15), δ are small enough, or when the label prior has a high entropy, s 0 can be large and hence leading to a larger lower bound on the accuracy discrepancy via Theorem 2.
Further, recall that p quantiﬁes the total probability mass of the group of minority subpopulations (see Figure 1).
Hence, for a small p , error on minority subpopulations do not contribute signiﬁcantly to the overall error despite causing a disproportionate increase to the marginal error of the minority group.
As a result, as p decreases, Theorem 3 states that the lower bound on unfairness increases as 1 − p − O (cid:0) √
pe − 1 / p (cid:1) while the upper bound for error decreases as O (cid:0) p − p 3 / 2 e − 1 / p (cid:1) =
O ( p ) .
Discussion of the assumptions We now discuss how the privacy parameters (cid:15), δ of the DP algorithm lead to feasible parameters s 0 , p 2 that appear in Assumptions A1 and A used in Theorem 2.
We also provide an example of an algorithm that satisﬁes these assumptions.
First, Lemmashows that for all values of (cid:15), δ, and s 0 , there exists a value of p 2 that satisﬁes Assumption A2.
Lemma 1.
Let F be the uniform prior over all labelling functions and S be any dataset.
For any ( (cid:15), δ ) -differentially private algorithm A , for all s 0 ∈ N , and for all subpopulations x ∈ X that appear fewer than s 0 times in the dataset
S , we have that
[Equation]
Please see Lemma 3 in the Appendix for a detailed version of Lemma 1.
Lemma 3 also suggests how to extend this result (albeit with a different probability bound) to other label priors.
Intuitively, the parameter s 0 in Lemma 1 (and assumption A2) represents the smallest frequency (in the observed data) of a subpopulation that is, with probability greater than 1 − p 2 , correctly classiﬁed by the algorithm
A .
Lemma 1 proves that, for any choice of (cid:15), p 2 and sufﬁciently small δ , there exists an s 0 such that any ( (cid:15), δ ) -DP algorithm satisﬁes Assumption A2 with the parameters sand p 2 .
In particular, for a ﬁxed p 2 , as (cid:15) decays, the probability lower bound is satisﬁed larger s 0 thereby showing that there is an inverse relationship between s 0 and (cid:15) .
We, thus, view s 0 as a “proxy” for the privacy of the algorithm in Theorem 2 and 3.
We now provide a differentially private algorithm that satisﬁes Assumptions A1 and A2.
In particular, consider an algorithm A η that accepts an m -sized dataset S f ∈ ( X × Y ) m and a noise rate η ∈ (cid:0) 0 , 12 (cid:1) as input and outputs a dictionary matching every subpopulation in X to a label of Y .
The algorithm ﬁrst creates a dictionary where the set X is the set of keys.
In order to assign values to every key, it ﬁrst randomly ﬂips the label of every element in S f with probability η , then for every unique key in S f , the algorithm computes the majority label of that key in the ﬂipped dataset and assigns that majority label to the corresponding key.
For elements in X not present in S f , it assigns a random element from Y .
Lemma 2 provides privacy and accuracy guarantees for this algorithm.
Lemma 2.
The algorithm A η is (cid:16) O (cid:16) log (cid:16) 1 η (cid:17)(cid:17) , 0 (cid:17) -DP as η → 0 .
Further for any dataset S f and s 0 ∈ N ,
• if a subpopulation x appears more than s 0 times in S , P h ∼A η ( S f )
[ h ( x ) (cid:54) =
f ( x )] ≤ e − s 0(1 − 2 η )2 / 8(1 − η ) and • if a subpopulation x appears less than s 0 times in S , P h ∼A η ( S f )
[ h ( x ) (cid:54) =
f ( x )] ≥ (4 η (1 − η )) s 0 / 2 e − s 0 .
Equivalently, algorithm A η satisﬁes Assumption A1 with p 1 =
e − s 0(1 − 2 η )2 / 8(1 − η ) and Assumption A2 with p 2 = 1 − (4 η (1 − η ))
s 0 / 2 e − s 0 .
Lemma 1 and 2 are proved in Appendix A.2.
Lemmashows that for all (cid:15) > 0 , we can ﬁnd an η = O ( e − (cid:15) )
such that A η is ( (cid:15), 0) -differentially private.
Further, this algorithm is more accurate on frequently occurring subpopulaitons and inaccurate on rare subpopulations, which aligns with Lemma 1.
Hence, for any (cid:15) > 0 , there is an η = O ( e − (cid:15) )
such that the algorithm A η is ( (cid:15), 0) - differentially private and is accurate on points appearing more than (cid:96) times with probability 1 − O (cid:0) e − (1 − 2 η ) (cid:96) (cid:1) .
2.4 PRIVACY AND FAIRNESS AT THE COST OF ACCURACY
So far we have shown that under strict privacy and high average accuracy requirements on the algorithm, fairness necessarily suffers.
A natural question to ask is whether it is possible to sacriﬁce accuracy for fairness.
As discussed in Section 2.3, increasing s 0 leads to higher error – in particular, we consider s
= Ω ( m ) .
We present a simpliﬁed theorem statement here for easier interpretation and prove a more precise version in Appendix A.3 along with a discussion.
In words, the theorem states that for very strict privacy parameters, fairness can be achieved at the cost of accuracy.
Theorem 3.
For any p ∈ (0 , 1 / 2 ) , c > 0 such that p/c ≤ 1 , consider the distribution Π p,N where N is the number of minority subpopulations.
For any (cid:15), δ, α > 0 , consider an ( (cid:15), δ ) -DP
algorithm
A that satisﬁes assumptions (A1) and (A2) with s 0 = (cid:16) 2 − p 2 k (1 − p ) (cid:17) m + α √ m and some p 2 ∈ (0 , 1) where Nm → c as both m, N → ∞ .
Then, , err ( A , Π p,N , F ) ≥ c 1 p +(1 − p 2 ) (1 − p ) (cid:18) 1 −
[Equation]
Γ ( A , Π p
[Equation]
where c 1 is a constant depending on (cid:15), δ, and F .
The detailed expression of c 1 and the full proof can be found in Theorem 6 in Appendix A.3.
We now brieﬂy discuss how the theorem characterises the effect of privacy and accuracy (via α ) on fairness.
When α is small, the algorithm is incorrect only on minority subpopulations.
Thus, α , essentially, characterises what fraction of majority subpopulations the algorithm is incorrect on.
Theorem 3 shows that when α is large, the error increases and the unfairness decreases.
Intuitively, this is because, with increasing α , the algorithm is incorrect, not only on minority subpopulations, but also on majority subpopulations (due to Assumption A2).
Thus, as subpopulations of larger frequency gets misclassiﬁed due to increasing α , the overall accuracy as well as the unfairness decreases.
EXPERIMENTAL RESULTS
In this section, we conduct experiments to support our theoretical results from Section 2.
We note that our theoretical results are model-agnostic and to demonstrate the universality of our result, we conduct a broad set of experiments on both synthetic (in Section 3.1), and real world datasets (in Section 3.2 and 3.3), using multiple machine learning models including deep neural networks and random forests.
3.1 SYNTHETIC EXPERIMENTS
First, we look at a synthetic data distribution that closely emulates the data distribution we use in our theoretical results in Theorem 1 and 2.
Given N, k ∈ N , c ∈ R + , and p ∈ (0 , 0 . 5) , we construct a continuous version of the longtailed distribution Π p,N,k (Deﬁnition 1) on a domain X .
First of all, since the domain X is discrete, we can place each element on a vertex of a O (log ( N ))
-dimensional hypercube.
The continuous distribution we use in our experiments is a mixture of Gaussians where each Gaussian is centered around the vertices of the hypercube.
In the experiments, we choose k = 64 , m = 10 4 , vary the ratio c from 0 .
01 to 0 . 2 , set the number of minority subpopulations to
N = mc , and choose p ∈ { 0 . 2 , 0 . 5 } .
We train a ﬁve-layer fully connected neural network with ReLU activations using
DP-SGD [1] for varying levels of (cid:15) while setting δ = 10 − 3 .
We refer to Appendix B for a more detailed description of the data distribution and the training algorithm.
Unfairness aggravates with increasing number of minority subpopulations As discussed in Section 2.1, increasing the number of subpopulations compared to the number of samples via c decreases accuracy on the minority subpopulations while the majority subpopulations remain unaffected.
Figure 2 shows how increasing c hurts fairness since the accuracy discrepancy (green dashed line) increases, most pronounced for small values (cid:15) (i.e. more private algorithms).
This corroborates our theoretical results from Theoremregarding the dependence of accuracy discrepancy on c .
We further observe that the increase in unfairness is almost entirely due to the drop in the minority accuracy (red solid) whereas the overall accuracy (blue) stays relatively constant.
This highlights our claim that, in the presence of strong privacy, fairness can be poor even when overall error is low.
Privacy constraints hurt fairness for accurate models
In this section, we analyse the dependence of fairness on the privacy parameter (cid:15) for a ﬁxed c .
In Figure 3 (left), we plot the disparate accuracies Γ for varying privacy parameter (cid:15) and Figure 3 (right) depicts the minority and overall accuracy as a function of (cid:15) .
There are two distinct phases in the development of the accuracy discrepancy with increasing (cid:15) separated by the gray dashed line:
For a very small (cid:15) , the learned classiﬁer is essentially a trivial classiﬁer as evidenced by the very low overall accuracy ( ≈ 60%) .
This is a trivial way of achieving fairness without learning an accurate classiﬁer and is explained by Theorem 3 in our theoretical section.
As the privacy restrictions are relaxed, the classiﬁer becomes more accurate and less fair in the ﬁrst phase.
The interesting regime is when classiﬁer obtains decent overall accuracy ( ≈ 80% ) and is marked by the vertical gray dashed line.
In the region to the right of the vertical dashed line, assumption m = o ( m ) is fulﬁlled and Figure 3 (left) reﬂects the behavior as predicted in Theorem 2: loosening privacy increases fairness or smaller (cid:15) implies larger accuracy discrepancy .
3.2 EXPERIMENTS ON VISION DATASETS
In this section, we show that our claims resulting from Theorem 2 and 3 do not only hold in synthetic settings but can also be observed in real-world computer vision datasets.
In particular, we conduct experiments on two popular computer vision datasets — CelebA and CIFAR-10.
CelebA is a dataset of approximately 160 k training images of dimension 178 × 218 and another 20 k of the same dimension for testing.
CIFAR-10 is a 10-class classiﬁcation dataset where there are 50 k training images and 10 k test images of dimension 3 × 32 × 32 .
For CIFAR-10, we use a ResNetand for CelebA, we use a ResNet- 50 architecture.
3.2.1 Minority and majority subpopulations
In practice, datasets like CelebA and CIFAR-10 often do not come with a label of what constitutes a subpopulation.
box plots
In this section, we describe how we deﬁne the minority and majority subpopulations for CIFAR-10 and CelebA.
CelebA
The CelebA dataset provides 40 attributes for each image including characteristics like gender, hair color, facial hair etc.
We create a binary classiﬁcation problem by using the gender attribute as the target label.
In addition, we use 11 of the remaining 39 binary attributes to create 2 11 subpopulations and categorise each example into one of these 2 11 subpopulations.
Then, we create various groups of minority subpopulations by aggregating the samples of all the unique subpopulations that appear less than s ∈ { 5 , 10 , 20 , 40 , 60 , 80 , 100 } times in the test set.
The remaining examples constitute the majority group.
In this section, we run experiments using s = 40 .
We report results for the other values of s in Appendix B.2.
CIFAR-10 Unlike the synthetic distribution and CelebA as described above, CIFAR-10 cannot be readily grouped into subpopulations using explicit attributes.
However, recent works [38, 50] have shown the presence of subpopulations in CIFAR-10 in the context of inﬂuence functions and adversarial training respectively.
We use the inﬂuence score estimates from Zhang and Feldman [50] to create the minority and majority subpopulations.
Intuitively, we treat examples that are atypical i.e. unlike any other examples in the dataset as minority examples belonging to minority subpopulations; and examples that are typical i.e. similar to a signiﬁcant number of other examples in the dataset as examples belonging to majority subpopulations.
To deﬁne these subpopulations, ﬁrst, we sort the examples in the training set according to their self-inﬂuence [50].
We deﬁne all of those that surpass a threshold ρ as minority populations.
In order to ﬁnd the samples belonging to each subpopulation x in the test set, we search for images that are heavily inﬂuenced (inﬂuence score is greater than the threshold) by at least one of the samples in x in the training set.
In this section, we report results with ρ = 0 .
Other values of ρ show a similar trend and we plot results using ρ = 0 .
01 in Appendix B.3.
3.2.2 Privacy leads to worse fairness for accurate models
In this section, we use the above deﬁnitions of minority and majority groups to measure the impact of privacy on fairness (using Deﬁnition 4).
Like Section 3.1, we measure both the accuracy discrepancy and the individual minority and overall accuracies.
CelebA Figure 4 plots the change in accuracy discrepancy, with respect to the (cid:15) parameter of differential privacy (smaller (cid:15) indicates stricter privacy).
Figure 4 (left) shows that smaller (cid:15) , with high accuracy (see Figure 4 (right)) implies a larger accuracy discrepancy.
This aligns with our theoretical results from Section 2.
Note that unlike Figure 3 (left), the accuracy discrepancy here monotonically decreases with increasing (cid:15) without exhibiting a two-phase behavior.
Figure 4 (right) shows that, the reason why we do not observe the two-phase behavior is that throughout the range of observed (cid:15) , we are in the regime of high accuracy.
CIFAR-10 Figure 5 (left) plots the change of accuracy discrepancy Γ with respect to the privacy parameter (cid:15) .
Interestingly, the results here exactly mimic those from the synthetic experiments in Figure 3, which are based on our theoretical setting.
This indicates that our theoretical setting is indeed relevant for real world observations.
Similar to the synthetic experiments, we observe two distinct phases in how the accuracy discrepancy changes with (cid:15) .
For small values of (cid:15) , Figure 5 (right) shows that the learned classiﬁer is highly inaccurate.
As discussed in Theorem 3, this is a trivial way to achieve fairness and this is reﬂected in Figure 5 (left).
However, if we restrict ourselves to classi- where the box plots reﬂect the variance when run several times accuracies for varying (cid:15)
The horizontal dashed line show the respective metrics for vanilla training without privacy constraints.
The vertical dashed the the test accuracy largr ( ≥ 80% ).
ﬁers with high average accuracy, marked by the area to the right of the vertical gray dashed line, Figure 5 (left) shows that accuracy discrepancy increases with decreasing (cid:15) .
This corresponds to the s
= o ( m ) assumption in Theorem 2.
3.3 EXPERIMENTS ON TABULAR DATA
To show that our observations hold across a wider range of publicly used datasets, we next conduct similar experiments using tabular data.
We run our experiments on the the Law school dataset [47] that has previously been used in fairnessawareness studies like Quy et al.
It is a binary classiﬁcation dataset with 21 k data points and 12 dimensional features.
Out of the 12 attributes, two binary attributes are used to obtain the minority group as deﬁned in Quy et al.
In contrast to previous experiments, we use random forest model from Fletcher and Islam [23] instead of neural networks as in Section 3.1 and 3.2.
For our implementation, we use the publicly available code in Holohan et al.
[28] with 10 trees and of a maximum depth 50 .
The results are plotted in Figure 6 and they show a similar two phase behavior as in our previous experiments with CIFAR-10.
Thus, all our experiments provide empirical evidence in support of the theoretical arguments in Section 2.
The behavior is consistent across multiple kinds of datasets, machine learning models, and learning algorithms.
FUTURE WORK
The experimental results on CelebA in Section 3.2 shows that when the minority group is composed of small sized subpopulations, differential privacy requirements hurt the fairness of the algorithm.
Here, we highlight that not all small-sized subpopulations are hurt equally in this process.
Figure 7 shows that a different partition of CelebA composed of similar sized populations do not show similar behaviours in terms of how accuracy discrepancy changes with sizes of subpopulations.
We refer to the 11 attributes we chose to partition the testset for our experiments so far as
Set A and, here, we choose another set of 11 attributes and refer to them as Set B .
Figure 7 (left) shows that both Set
A and Set B induces a very similar distribution over sizes of subpopulations on the test set.
However, Figure 7 (right) shows that while the group of minority subpopulations induced by Set A suffers very high accuracy discrepancy from private training compared to vanilla training, Set B does not (see Appendix B.2 for more details on Set A and Set
B).
This indicates that, irrespective of sizes, private training hurts fairness disproportionately more for certain subpopulations compared to others.
In particular, an interesting direction of further research is to investigate where these minority subpopulations that are worse-affected by private training intersects with the subpopulations that are relevant for the speciﬁc domain.
While most past works [4, 11] have also used sizes of subpopulations to differentiate between disparately impacted subpopulations, this suggests that that is not always the case.
In this paper, we have shown theoretically that when the minority group in the data is composed of multiple subpopulations, a DP algorithm can achieve very low error but necessarily incurs worse fairness.
Further, we corroborated our theoretical results with experimental evidence on synthetic and real world computer vision datasets.
However, our model-agnostic results, that shed a rather pessimistic light on algorithmic fairness and differential privacy, only apply under certain distributional assumptions.
It is possible that in some real-world datasets there are fair and private algorithms that achieve a more optimistic trade-off.
This begs further research to develop fair and private algorithms that are closer to the pareto optimal frontier.
Acknowledgements
AS is supported by the ETH AI Center and Hasler Stiftung.