S TATISTICAL INFERENCE FOR INDIVIDUAL FAIRNESS
Anonymous authors
Paper under double-blind review
A BSTRACT
As we rely on machine learning (ML) models to make more consequential decisions, the issue of ML models perpetuating or even exacerbating undesirable historical biases ( e.g. gender and racial biases) has come to the fore of the public’s attention.
In this paper, we focus on the problem of detecting violations of individual fairness in ML models.
We formalize the problem as measuring the susceptibility of ML models against a form of adversarial attack and develop a suite of inference tools for the adversarial cost function.
The tools allow practitioners to assess the individual fairness of ML models in a statistically-principled way: form conﬁdence intervals for the worst-case performance differential between similar individuals and test hypotheses of model fairness with (asymptotic) non-coverage/Type I error rate control.
We demonstrate the utility of our tools in a real-world case study.
I NTRODUCTION
The problem of bias in machine learning systems is at the forefront of contemporary ML research.
Numerous media outlets have scrutinized machine learning systems deployed in practice for violations of basic societal equality principles (Angwin et al., 2016; Dastin, 2018; Vigdor, 2019).
In response researchers developed many formal deﬁnitions of algorithmic fairness along with algorithms for enforcing these deﬁnitions in ML models (Dwork et al., 2011;
Hardt et al., 2016; Berk et al., 2017;
Kusner et al., 2018; Ritov et al., 2017; Yurochkin et al., 2020).
Despite the ﬂurry of ML fairness research, the basic question of assessing fairness of a given ML model in a statistically principled way remains largely unexplored.
In this paper we propose a statistically principled approach to assessing individual fairness (Dwork et al., 2011) of ML models.
One of the main beneﬁts of our approach is it allows the investigator to calibrate the method; i.e. it allows the investigator to prescribe a Type I error rate.
Passing a test that has a guaranteed small Type I error rate is the usual standard of proof in scientiﬁc investigations because it guarantees the results are reproducible (to a certain degree).
This is also highly desirable in detecting bias in ML models because it allows us to certify whether an ML model will behave fairly at test time.
Our method for auditing ML models abides by this standard.
There are two main challenges associated with developing a hypothesis test for individual fairness.
First, how to formalize the notion of individual fairness in an interpretable null hypothesis?
Second, how to devise a test statistic and calibrate it so that practitioners can control the Type I error rate?
In this paper we propose a test motivated by the relation between individual fairness and adversarial robustness (Yurochkin et al., 2020).
At a high-level, our approach consists of two parts:
generating unfair examples: by unfair example we mean an example that is similar to a training example, but treated differently by the ML models.
Such examples are similar to adversarial examples (Goodfellow et al., 2014), except they are only allowed to differ from a training example in certain protected or sensitive ways.
summarizing the behavior of the ML model on unfair examples: We propose a loss-ratio based approach that is not only scale-free, but also interpretable.
For classiﬁcation problems, we propose a variation of our test based on the error rates ratio.
1. R ELATED WORK
At a high level, our approach is to use the difference between the empirical risk and the distributionally robust risk as a test statistic.
The distributionally robust risk is the maximum risk of the ML model on similar training examples.
Here similarity is measured by a fair metric that encodes our intuition of which inputs should be treated similarly by the ML model.
We note that DRO has been extensively studied in the recent literature (Duchi et al., 2016; Blanchet & Murthy, 2016; Hashimoto et al., 2018), however outside of the fairness context with the exception of Yurochkin et al.
; Xue et al.
Yurochkin et al.
(2020) focus on training fair or robust ML models instead of auditing ML models.
Xue et al.
(2020) also use the difference between the empirical and distributionally robust risks as a test statistic, but their test is only applicable to ML problems with ﬁnite feature spaces.
This limitation severely restricts the applicability of their test.
On the other hand, our test is suitable for
ML problems with continuous features spaces.
We note that the technical exposition in Xue et al.
(2020) is dependant on the ﬁnite feature space assumption and in this work we develop a novel perspective of the problem that allows us to handle continuous feature spaces.
G RADIENT FLOW FOR FINDING UNFAIR EXAMPLES
In this section, we describe a gradient ﬂow-based approach to ﬁnding unfair examples that form the basis of our suite of inferential tools.
Imagine an auditor assessing whether an ML model is fair or not.
The auditor aims to detect violations of individual fairness in the ML model.
Recall Dwork et al.
(2011)’s deﬁnition of individual fairness.
Let X ⊂ R d and Y ⊂ R d be the input and output spaces respectively, and f : X → Y be an ML model to audit.
The ML model f is known as individually fair if
[Equation]
for some Lipschitz constant L fair > 0 .
Here d x and d y are metrics on X and Y respectively.
Intuitively, individually fair ML model treats similar samples similarly, and the fair metric d x encodes our intuition of which samples should be treated similarly.
We should point out that d x ( x 1 , x 2 ) being small does not imply x 1 and x 2 are similar in all aspects.
Even if d x ( x 1 , x 2 ) is small, x 1 and x 2 may differ much in certain attributes, e.g., protected/sensitive attributes.
Before moving on, we comment on the choice of the fair metric d x .
This metric is picked by the auditor and reﬂects the auditor’s intuition about what is fair and what is unfair for the ML task at hand.
It can be provided by a subject expert (this is Dwork et al.
(2011)’s original recommendation) or learned from data (this is a recent approach advocated by Ilvento (2019); Wang et al.
Mukherjee et al.
Section 4 provides details of picking a fair metric in our empirical studies.
To motivate our approach, we recall the distributionally robust optimization (DRO) approach to training individually fair ML models (Yurochkin et al., 2020).
Let f : X → Y be an ML model and (cid:96) ( f ( x ) , y ) :
Z → R + be any smooth loss ( e.g. cross-entropy loss).
To search for differential treatment in the ML model, Yurochkin et al.
(2020) solve the optimization problem
[Equation]
where W is the Wasserstein distance on probability distributions on feature space induced by the fair metric, P n is the empirical distribution of the training data, and (cid:15) is a moving budget that ensures the adversarial examples are close to the (original) training examples in the fair metric. Formally, this search for differential treatment checks for violations of distributionally robust fairness .
Deﬁnition 2.1 (distributionally robust fairness (DRF) (Yurochkin et al., 2020)) .
An ML model h : X → Y is ( (cid:15), δ ) -distributionally robustly fair (DRF) WRT the fair metric d x iff
[Equation] The optimization problem (2.2) is an inﬁnite-dimensional problem, but its dual is more tractable.
Blanchet & Murthy show that the dual of (2.2) is
[Equation]
In practice, since (2.5) is highly non-convex in general, practitioners use gradient-based optimization algorithm to solve it and terminate the algorithm after few iterations.
As a result, one can not guarantee optimality of the solution.
However, optimality is required to establish convergence guarantees for
DRO algorithms.
This issue is typically ignored in practice when developing training algorithms, e.g. as in Yurochkin et al.
but it should be treated with care when considering limiting distribution of the related quantities required to calibrate a test.
We note that Xue et al.
(2020) needed discrete feature space assumption due to the aforementioned concern: when the feature space is discrete, it is possible to solve (2.5) optimally by simply comparing the objective value at all points of the sample space.
In this paper we adapt theory to practice, i.e. analyze the limiting distribution of (2.5) optimized for a ﬁxed number of gradient steps.
The effects of early termination can be characterized by a continuous-time approximation of adversarial dynamics, which we called gradient ﬂow attack .
Given a sample ( x 0 , y 0 ) , the gradient ﬂow attack solves a continuous-time ordinary differential equation (ODE)
[Equation]
over time t ≥ 0 .
For ﬁxed penalty parameter λ and stopping time T > 0 , Φ : X × Y → X is the unfair map
[Equation]
Here the map Φ is well-deﬁned as long as g ( x ) (cid:44) ∇ x { (cid:96) ( f ( x ) , y 0 ) − λd 2 x ( x, x 0 ) } is Lipschitz, i.e., (cid:107) g
( x 1 ) − g ( x 2 ) (cid:107) 2 ≤ L (cid:107) x 1 − x 2 (cid:107) 2 for some L > 0 .
Under this assumption, the autonomous Cauchy problem (2.6) has unique solution and thus Φ : X × Y → X is a one-to-one function.
We call Φ an unfair map because it maps samples in the data to similar areas of the sample space that the ML model performs poorly on.
We note that data in this case is an audit dataset chosen by the auditor to evaluate individual fairness of the given model.
The audit data does not need to be picked carefully and could be simply an iid sample (e.g. testing data).
The unfair map plays the key role as it allows us to identify areas of the sample space where model violates individual fairness, even if the audit samples themselves reveal no such violations.
In the rest of the paper, we deﬁne the test statistic in terms of the unfair map instead of the optimal point of (2.5).
This has two main beneﬁts:
computational tractability: evaluating the unfair map is computationally tractable because integrating initial value problems (IVP) is a well-developed area of scientiﬁc computing.
Practitioners may appeal to any globally stable method for solving IVP’s to evaluate the unfair map.
reproducibility: the non-convex nature of (2.5) means the actual output of any attempts at solving (2.5) is highly depend on the algorithm and the initial iterate.
By deﬁning the test statistic algorithmically, we avoid ambiguity in the algorithm and initial iterate, thereby ensuring reproducibility.
Of course, the tractability and reproducibility of the resulting statistical tests comes at a cost: power.
Because we are not exactly maximizing (2.5), the ability of the test statistic to detect violations of individual fairness is limited by the ability of (2.7) to ﬁnd (unfair) adversarial examples.
2. E VALUATING THE TEST STATISTIC
Solving IVP’s is a well-studied problem in scientiﬁc computing, and there are many methods for solving IVP’s.
For our purposes, it is possible to use any globally stable method to evaluate the unfair map.
One simple method is the forward Euler method with sufﬁciently small step size.
Let 0 =
t 0 < t 1 < . . .
<
t N = T be a partition of [0 , T ] , and denote the step size by η k =
t k − t k − 1 for k = 1 , . .
. , N .
Initialized at x (0) =
x 0 , the forward Euler method repeats the iterations
[Equation]
for k = 1 , . . .
, N .
The validity of discretizations of the forward Euler method is guaranteed by the following uniform bounds.
Theorem 2.2 (Global stability of forward Euler method) .
Consider the solution path { X ( t ) } 0 ≤ t ≤ T given by (2.6) and the sequence { x ( k ) } Nk =0 given by (2.8) .
Let the maximal step size be h = max { η 1 , . . .
, η N } .
Suppose that (cid:107) J g
( x ) g ( x ) (cid:107) ∞
≤ m , where g ( x ) = ∇ x { (cid:96) ( f ( x ) , y 0 ) − λd 2 x ( x, x 0 ) } and J g is the Jacobian matrix of g .
Then we have
[Equation]
Theorem 2.2 indicates that the global approximation error (2.9) decreases linearly with h , the maximal step size.
Therefore by taking small enough h , the value of the unfair map Φ can be approximated by x ( N ) with arbitrarily small error.
T ESTING INDIVIDUAL FAIRNESS OF AN ML MODEL
Although gradient ﬂows are good ways of ﬁnding unfair examples, they do not provide an interpretable summary of the ML model outputs.
In this section, we propose a loss-ratio based approach to measuring unfairness with unfair examples.
Given a sample point ( x 0 , y 0 )
∈ Z , gradient ﬂow attack (2.6) always increases the regularized loss in (2.5), that is,
[Equation]
Therefore the unfair map Φ : Z → X always increases the loss value of the original sample.
In other words, the ratio
[Equation]
Recall that the unfair map Φ moves a sample point to its similar points characterized by the fair metric d x .
The fair metric d x reﬂects the auditor’s particular concern of individual fairness so that the original sample ( x, y ) and the mapped sample (Φ( x, y ) , y ) should be treated similarly.
If there is no bias/unfairness in the ML model, then we expect the ratio (cid:96) ( f (Φ( x, y )) , y ) /(cid:96)
( f ( x ) , y ) to be close to 1 .
With this intuition, to test if the ML model is individually fair or not, the auditor considers hypothesis testing problem
[Equation]
( f ( x ) , y ) (cid:21) ≤ δ versus H 1 : E P (cid:20) (cid:96) ( f (Φ( x, y )) , y ) (cid:96) ( f ( x ) , y ) (cid:21) >
δ, (3.3) where P is the true data generating process, and δ > 1 is a constant speciﬁed by the auditor.
Using the ratio of losses in (3.3) has two main beneﬁts:
scale-free : changing the loss function by multiplying a factor does not change the interpretation of
the null hypothesis The test is interpretable : the tolerance δ is the maximum loss differential above which we consider an ML model unfair.
In applications where the loss can be interpreted as a measure of the negative impact of an ML model, there may be legal precedents on acceptable levels of differential impact that is tolerable.
In our computational results, we set δ according to the four-ﬁfths rule in US labor law.
Please see Section 3.3 for further discussion regarding δ and interpretability
We note that using the ratio of losses as the test statistic is not without its drawbacks.
If the loss is small but non-zero, then the variance of the test statistic is inﬂated and and the test loses power, but
Type I error rate control is maintained.
3. T HE AUDIT VALUE
The auditor collects a set of audit data { ( x i , y i ) } ni =1 and computes the empirical mean and variance of the ratio (cid:96) ( f (Φ( x, y )) , y ) /(cid:96)
( f ( x ) , y ) ,
S n = 1 n n (cid:88) i = (cid:96) ( f (Φ( x i , y i )) , y i ) (cid:96) ( f ( x i ) , y i ) and V 2 n = n −n (cid:88)
i = (cid:18) (cid:96) ( f (Φ( x i , y i )) , y i ) (cid:96) ( f ( x i ) , y i ) − S n (cid:19), (3.4) by solving gradient ﬂow attack (2.6).
The ﬁrst two empirical moments, S n and V 2 n , are sufﬁcient for the auditor to form conﬁdence intervals and perform hypothesis testing for the population mean E
P [ (cid:96) ( f (Φ( x, y )) , y ) /(cid:96)
( f ( x ) , y )] , the audit value .
Theorem 3.1 (Asymptotic distribution) .
Assume that ∇ x { (cid:96) ( f ( x ) , y ) − λd 2 x ( x, y ) } is Lipschitz in x , and (cid:96) ( f (Φ( x, y )) , y ) /(cid:96)
( f ( x ) , y ) has ﬁnite ﬁrst and second moments.
If the ML model f is independent of { ( x i , y i ) } ni =1 , then
[Equation]
in distribution, as n → ∞ .
The ﬁrst inferential task is to provide conﬁdence intervals for the audit value.
The two-sided equal-tailed conﬁdence interval for the audit value E P [ (cid:96) ( f (Φ( x, y )) , y ) /(cid:96)
( f ( x ) , y )] with asymptotic coverage probability 1 − α is
[Equation]
where z q is the q -th quantile of normal distribution N (0 , 1) .
Corollary 3.2 (Asymptotic coverage of two-sided conﬁdence interval) .
Under the assumptions in
Theorem 3.1,
[Equation]
The second inferential task is to test restrictions on the audit value, that is, considering the hypothesis testing problem (3.3).
Similar to the two-sided conﬁdence interval (3.6), we may also have one-sided conﬁdence interval for the audit value E P
[ (cid:96) ( f (Φ( x, y )) [Equation] y ) /(cid:96)
( f ( x ) , y )] with asymptotic coverage probability 1 − α , i.e.,
[Equation]
The one-sided conﬁdence interval (3.8) allows us to test simple restrictions of the form
[Equation]
by checking whether δ falls in the (1 − α ) -level conﬁdence region.
By the duality between conﬁdence intervals and hypothesis tests, this test has asymptotic Type I error rate at most α .
With this intuition, a valid test is:
[Equation]
where T n is the test statistic.
Corollary 3.3 (Asymptotic validity of test) .
Under the assumptions in Theorem 3.1,
if E P [ (cid:96) ( f (Φ( x, y )) , y ) /(cid:96)
( f ( x ) , y )] ≤ δ , we have lim n →∞ P ( T n > δ ) ≤ α ; 2.
if E P [ (cid:96) ( f (Φ( x, y )) , y ) /(cid:96)
( f ( x ) , y )] > δ , we have lim n →∞ P ( T n > δ ) = 1 .
3. R OBUSTNESS OF TEST STATISTIC TO THE CHOICE OF FAIR METRIC
Since the fair metric d x is picked by an expert’s domain knowledge or in a data-driven way, the auditor may incur the issue of fair metric misspeciﬁcation.
Fortunately, the test statistic (3.10) is robust to small changes in fair metrics.
Let d 1 , d 2 : X × X
→ R + be two different fair metrics on X .
Let Φ 1 , Φ 2 : X × Y → X be the unfair maps induced by d 1 , d 2 .
We start by stating the following assumptions:
(A1) ∇ x { (cid:96) ( f ( x ) , y ) − λd 2 1 ( x, y ) } is L -Lipschitz in x with respect to (cid:107) · (cid:107) 2 ; (A2) sup x,x (cid:48) ∈X (cid:107) x − x (cid:48) (cid:107) 2 (cid:44) D < ∞ ; (A3) (cid:96) ( f ( x ) , y ) is L 0 -Lipschitz in x with respect to (cid:107) · (cid:107) 2 , and lower bounded by c > 0 ; (A4) sup x,x (cid:48) ∈X (cid:107)∇ x d 21 ( x, x (cid:48) )
− ∇ x d 22 ( x, x (cid:48) )
(cid:107) 2 ≤ Dδ d for some constant δ d ≥ 0 .
Assumption A1 is always assumed for the existence and uniqueness of ODE solution.
Assumption
A2, that the feature space X is bounded, and the ﬁrst part of Assumption A3 are standard in the
DRO literature.
The second part of Assumption A3 is to avoid singularity in computing loss ratio.
Assumption A4 is worthy of being discussed.
The constant δ d in A4 characterizes the level of fair metric misspeciﬁcation.
Moreover, Assumption A4 is mild under Assumption A2.
For example, let d 21 ( x, x (cid:48) )
= (cid:104) x − x (cid:48) , Σ exact ( x − x (cid:48) )
(cid:105) and d 22 ( x, x (cid:48) )
= (cid:104) x − x (cid:48) , Σ mis ( x − x (cid:48) )
(cid:105) (3.11) be the exact and misspeciﬁed fair metric respectively.
Then
[Equation]
[Equation]
The level of fair metric misspeciﬁcation δ d vanishes as long as Σ mis estimates Σ exact consistently.
Theorem 3.4 (Robustness of test statistic) .
Suppose that the support of data distribution P satisﬁes for any ( x 0 , y 0 ) ∈ supp( P ) , the solution path of ODE (2.6) corresponding to ( x 0 , y 0 ) and d 1 (or d 2 ) lies in X .
Under Assumptions A1 – A4, we have
[Equation]
for any ( x 0 , y 0 ) ∈ supp( P ) .
The ﬁrst assumption in Theorem 3.4 is mild since we always perform early termination in gradient ﬂow attack.
In the literature, the fair metric can be learned from an additional dataset different from the training, test, and audit dataset.
In this case, the constant δ d , which characterizes the goodness-of-ﬁt of the estimated fair metric to the exact fair metric, shrinks to 0 as n → ∞ .
Then Theorem 3.4 provides two key insights.
First, as long as δ d tends to 0 , we ultimately test the same null hypothesis since
[Equation]
(cid:12) (cid:12)(cid:12) ≤ (cid:114)
λδ d
L
L 0 De LT c →(3.16) as δ d → 0 .
Second, the error of test statistic induced by the misspeciﬁcation of fair metric is negligible as long as δ d
= o ( 1 n ) .
This is due to the fact that the ﬂuctuations of test statistic are O p ( 1 √ n ) , so √ δ d must vanish faster than O ( 1 √ n ) to not affect the test statistic asymptotically.
3. T EST INTERPRETABILITY , TEST TOLERANCE AND AN ALTERNATIVE FORMULATION
To utilize our test, a practitioner should set a tolerance (or threshold) parameter δ . It is important that a practitioner can understand and interpret the meaning of the threshold they choose. Appropriate choice of δ can vary based on the application, however here we consider a general choice motivated by the US Equal Employment Opportunity Commission’s four-ﬁfths rule, which states “selection rate for any race, sex, or ethnic group [must be at least] four-ﬁfths (4/5) (or eighty percent) of the rate for the group with the highest rate”.
1 Rephrasing this rule in the context of the loss ratio, we consider the following: the largest permissible loss increase on an individual should be at most ﬁve-fourth (5/4) of its original loss.
This corresponds to the null hypothesis threshold δ = 1 . 25 .
The aforementioned wording of the four-ﬁfth rule is based on the demographic parity group fairness deﬁnition, however it can be generalized to other group fairness deﬁnitions as follows: “performance of the model on any protected group must be at least four-ﬁfth of the best performance across groups”.
Depending on what we mean by performance, we can obtain other group fairness deﬁnitions such as accuracy parity when auditing an ML classiﬁcation system.
In our test, we use the loss ratio because the loss is a general measure of performance of an ML system.
However, in the context of supervised learning, the loss is often a mathematically convenient proxy for the ultimate quantity of interest, error rate.
For classiﬁcation problems, it is possible to modify our test statistic based on the ratio of error rates instead of losses (maintaining δ = 1 .
according to the ﬁve-fourth rule).
Let (cid:96) 0 , 1 be the 0-1 loss.
Naively, we could consider the mean of the ratio (cid:96) 0 , 1 ( f (Φ( x,y )) ,y ) (cid:96) 0 , 1 ( f ( x ) ,y ) as a test statistic, but this is problematic because the ratio is not well-deﬁned when the classiﬁer correctly classiﬁes x .
To avoid this issue, we propose considering the ratio of means (instead of the mean of the ratio) as a test statistic.
Formally, we wish to test the hypothesis
[Equation]
We emphasize that the gradient ﬂow attack is still performed with respect to a smooth loss function; we merely use the 0-1 loss function to evaluate the accuracy of the classiﬁer on the original and adversarial examples.
The auditor collects a set of audit data { ( x i , y i ) } ni =1 and computes the ratio of empirical risks ˜ S n = 1 n (cid:80) ni
=1 (cid:96) 0 , 1 ( f (Φ( x i , y i )) , y i ) 1 n (cid:80) ni
=1 (cid:96) 0 , 1 ( f ( x i ) , y i ) and ˜ V n (cid:44) 1 n n (cid:88)
i = (cid:20) (cid:96) 0 , 1 ( f (Φ( x i , y i )) , y i ) (cid:96) 0 , 1 ( f ( x i ) , y i ) (cid:21) (cid:20) (cid:96) 0 , 1 ( f (Φ( x i , y i )) , y i ) (cid:96) 0 , 1 ( f ( x i ) , y i ) (cid:21) (cid:62) (3.18) by performing the gradient ﬂow attack (2.6).
Let A n and B n be the numerator and denominator of ˜
S n .
Parallel to the intuition of (3.10), here the proposed test is:
[Equation]
, (3.19) where ˜ T n is the test statistic.
Please see Appendix B for the asymptotic normality theorem and Type I error guarantees.
In our experiments we ﬁrst verify our methodology in simulations and then present a case-study of testing individual fairness on the Adult dataset (Dua & Graff, 2017).
A practitioner performing the testing would need to make three important choices: fair metric d x ( · , · ) , testing threshold δ to have a concrete null hypothesis and level of signiﬁcance (maximum allowed type I error of hypothesis testing, i.e. p -value cutoff) to make a decision whether the null (classiﬁer is fair) should be rejected.
The fair metric can be provided by a subject expert, as considered in our simulation studies, or estimated from data using fair metric learning techniques proposed in the literature, as we do in the Adult experiment.
Following the discussion in Section 3.3 we set δ = 1 . 25 .
For the signiﬁcance level, typical choice in many sciences utilizing statistical inference is 0.05, which we follow in our experiments, however this is not a universal rule and should be adjusted in practice when needed.
4. S TUDYING TEST PROPERTIES WITH SIMULATIONS
We ﬁrst investigate the ability of our test to identify an unfair classiﬁer, explore robustness to fair metric misspeciﬁcation verifying Theorem 3.4, and discuss implications of the choice of null hypothesis parameter δ .
We simulate a 2-dimensional binary classiﬁcation dataset with two subgroups of observations that differ only in the ﬁrst coordinate (we provide additional details and data visualization in Appendix D).
One of the subgroups is underrepresented in the training data yielding a corresponding logistic regression classiﬁer that overﬁts the majority subgroup and consequently differentiates data points (i.e. “individuals”) along both coordinates.
Recall that a pair of points that only differ in the ﬁrst coordinate are considered similar by the problem design, i.e. their fair distance is 0, and prediction for such points should be the same to satisfy individual fairness.
Fair metric Our expert knowledge of the problem allow to specify a fair metric d 2 x (( x 1 , x 2 ) , ( x (cid:48) 1 , x (cid:48) 2 )) =
( x 2 − x (cid:48) 2 ) 2 .
Evidently, an individually fair logistic regression should have coefﬁcient of the ﬁrst coordinate θ 1 = 0 , while intercept and θ 2 can be arbitrary.
The more θ 1 differs from 0, the larger is the individual fairness violation.
In Figure 1 (left) we visualize the heatmap of the test statistic (3.10) over a grid of ( θ 1 , θ 2 ) values (intercept is estimated from the data for each coefﬁcients pair).
Recall that when this value exceeds δ = 1 . 25 our test rejects the null (fairness) hypothesis (red heatmap area).
Our test well-aligns with the intuitive interpretation of the problem, i.e. test statistic increases as θ 1 deviates from 0 and is independent of the θ 2 value.
Metric misspeciﬁcation We also consider fair metric misspeciﬁcation in the center and right heatmaps of Figure 1.
Here the discounted movement direction in the metric is rotated, i.e. d 2 x (( x 1 , x 2 ) , ( x (cid:48) 1 , x (cid:48) 2 ))
= (sin 2 β )( x 1 − x (cid:48) 1 )
+ (cos 2 β )( x 2 − x (cid:48) 2 ) 2 for β = 5 ◦ (center) and β = 10 ◦ (right).
We see that the test statistic starts to reject fairness of the models with larger θ 2 magnitudes due to misspeciﬁcation of the metric, however it remains robust in terms of identifying θ 1 = 0 as the fair model.
Null hypothesis threshold Finally we assess the null hypothesis choice δ = 1 . 25 .
We saw that the test permits (approximately)
θ 1 < 1 . 5 — whether this causes minor or severe individual fairness violations depends on the problem at hand.
A practitioner that has access to an expert knowledge for deﬁning the fair metric and desires stricter individual fairness guarantees may consider smaller values of δ .
In this simulated example, we see that as δ approaches 1, the test constructed with the correct fair metric (Figure 1 left) will reject all models with θ 1 (cid:54) = 0 , while permitting any θ 2 values.
4. R EAL DATA CASE - STUDY
We present a scenario of how our test can be utilized in practice.
To this end, we consider income classiﬁcation problem using Adult dataset (Dua & Graff, 2017).
The goal is to predict if a person is earning over $50 k per year using features such as education, hours worked per week, etc.
(we exclude race and gender form the predictor variables; please see Appendix E and code in the supplementary materials for data processing details).
Learning the fair metric In lieu of an expert knowledge to deﬁne a fair metric, we utilize technique by Yurochkin et al.
(2020) to learn a fair metric from data.
They proposed a fair metric of the form: d 2 x ( x, x (cid:48) )
= (cid:104) x − x (cid:48) , P ( x − x (cid:48) )
(cid:105) , where P is the projection matrix orthogonal to a “sensitive” subspace.
Similar to their Adult experiment, we learn this subspace by ﬁtting two logistic regression classiﬁer to predict gender and race and taking span of the coefﬁcient vectors (i.e. vectors orthogonal to decision boundary) as the sensitive subspace.
The intuition behind this metric is that this subspace captures variation in the data pertaining to the racial and gender differences.
A fair metric should treat individuals that only differ in gender and/or race as similar, therefore it assigns 0 distance to any pair of individuals that only differ by a vector in the sensitive subspace (similar to the fair metric we used in simulations discounting any variation along the ﬁrst coordinate).
Our hypothesis test is an audit procedure performed at test time, so we learn the fair metric using test data to examine fairness of several methods that only have access to an independent train set to learn their decision function.
Results We perform testing of the 4 classiﬁers: baseline NN, group fairness Reductions (Agarwal et al., 2018) algorithm, individual fairness SenSR (Yurochkin et al., 2020) algorithm and a basic
Project algorithm that pre-processes the data by projecting out “sensitive” subspace.
For SenSR fair metric and Project we use training data to learn the “sensitive” subspace.
All methods are trained to account for class imbalance in the data and we report test balanced accuracy as a performance measure following prior studies of this dataset (Yurochkin et al., 2020; Romanov et al., 2019).
Results of the 10 experiment repetitions are summarized in Table 1 (see Table 3 in Appendix E.6 for the standard deviations).
We compare group fairness using average odds difference (AOD)
(Bellamy et al., 2018) for gender and race.
Signiﬁcance level for null hypothesis rejection is 0.05 and δ = 1 .(see Appendix E and code for details regarding the algorithms and comparison metrics).
Baseline exhibits clear violation of both individual ( T n , ˜ T n (cid:29) 1 . 25 and rejection proportion is 1) and group fairness (both AOD are far from 0).
Simple projection pre-processing improved individual fairness
, however the null is still rejected in the majority experiment repetitions (balanced accuracy [Table] improvement is accidental).
A more sophisticated individual fairness algorithm SenSR does perform well according to our test with test statistic close to 1 (ideal value) and the test fails to reject individual fairness of SenSR every time.
Lastly we examine the trade-off between individual and group fairness.
Enforcing group fairness with Reductions leads to best AOD values, however it worsens individual fairness (comparing to the baseline) as measured by the test statistic.
On the contrary, enforcing individual fairness with SenSR also improves group fairness metrics, however at the cost of the lowest balanced accuracy.
We present a similar study of the COMPAS dataset in Appendix F. Results there follow the same pattern with the exception of Reductions slightly improving individual fairness in comparison to the baseline, but still being rejected by our test in all experiment repetitions. Both loss-ratio test T n and error-rate ratio test ˜ T n results are almost identical. The only difference is that loss ratio test rejected Project in 9 out of 10 trials, while error-rate ratio test in 8 out of 10 trials. [Figure]
Setting stopping time T .
Recall that the test statistic
T n depends on the number of steps T of the gradient ﬂow attack (2.6). Corollary 3.3 guarantees Type I error control for any T , i.e. it controls the error of rejecting a fair classiﬁer regardless of the stopping time choice. Theoretical guarantees for Type II error, i.e. failing to reject an unfair classiﬁer, are hard to provide in general (one needs to know expected value of the loss ratio for a given T and a speciﬁc model). In practice, we recommend running the gradient ﬂow attack long enough (based on the available computation budget) to guarantee small Type II error.
In our Adult experiment we set T = 500 .
In Figure(note the log-log scale) we present an empirical study of the test statistic T n as a function of stopping time T .
We see that our test fails to reject SenSR, the classiﬁer we found individually fair, for any value of T verifying our Type I error guarantees in practice.
Rejection of the unfair classiﬁers requires sufﬁciently large T , supporting our recommendation for Type II error control in practice.
D ISCUSSION AND CONCLUSION
We developed a suite of inferential tools for detecting and measuring individual bias in ML models.
The tools require access to the gradients/parameters of the ML model, so they’re most suitable for internal investigators.
We hope our tools can help practitioners verify individual fairness of ML models and help researchers benchmark individual fairness algorithms.
Future work on learning ﬂexible individual fairness metrics from data will expand the applicability range of our test.
We demonstrated the utility of our tools by using them to reveal the gender and racial biases in an income prediction model.
In our experiments, we discovered that enforcing group fairness may incur individual bias.
In other words, the algorithm may sacriﬁce individual fairness in order to preserve parity of certain metrics across groups.
For example, one of the earliest methods for enforcing group fairness explicitly treated examples from the majority and minority groups differently (Hardt et al., 2016).
We conjecture that the even the more modern methods for enforcing group fairness could be forcibly balancing outcomes among demographic groups, leading to instances where similar individuals in different demographic groups are treated differently.
The possible trade-off between individual and group fairness warrants further investigation, but is beyond the scope of this paper.
A P ROOFS A. P ROOF OF T HEOREM IN S ECTION
Proof of Theorem 2.9.
Let X ( t ) =
( X (1) ( t ) , . . .
, X ( d ) ( t )) (cid:62) .
For i = 1 , . . .
, d and k = 1 , . . .
, N , we have
[Equation]
A.
[Equation]
) for some ˜ t ( i ) k − 1 ∈
[ t k − 1 , t k ] .
Compactly, we have
[Equation]
(A.2)
For k = 1 , . . .
, N , we let
T k
[Equation]
(A.3)
[Equation]
(A.4)
Note that (cid:107) ¨ X ( t ) (cid:107) ∞
= (cid:107)
[Equation]
g
[Equation]
X
[Equation]
t
[Equation]
) g
[Equation]
X ( t ))
[Equation]
∞ ≤
[Equation]
, (A.5) and η k ≤ h , we have (cid:107) T k (cid:107) [-
-]2 = 1 2 η k (cid:13)(cid:13)(cid:13)(cid:13)(cid:16) ¨ X
(1) (˜ t (1) k − 1 ) , . . .
, ¨ X ( d ) (˜ t ( d ) k − 1 ) (cid:17) (cid:62) (cid:13)(cid:13)(cid:13)(cid:13) (A.6) ≤2 η k √ d (cid:13)(cid:13)(cid:13)(cid:13)(cid:16) ¨ X (1) (˜ t (1) k − 1 ) , . .
¨ X ( d ) (˜ t ( d ) k − 1 ) (cid:17) (cid:62) (cid:13)(cid:13)(cid:13)(cid:13) ∞ (A.7) ≤ hm √ d .
(A.8)
Let e k
= X ( t k ) − x ( k ) for k = 1 , . . .
, N , we have e k
= X
( t k − 1 ) − x ( k − 1) + η k (cid:16) g
( X ( t k − 1 )) − g ( x ( k − 1) ) (cid:17) + η k T k (A.9) =
e k − 1 + η k (cid:16) g
( X ( t k − 1 )) − g ( x ( k − 1) ) (cid:17) + η k T k .
(A.10)
Since g is L -Lipschitz, we have (cid:107) e k (cid:107) 2 ≤ (cid:107) e k − 1 (cid:107) 2 + η k L (cid:107) e k − 1 (cid:107) 2 + η k hm √ d .
(A.11)
Then, (cid:107) e k (cid:107)
+ hm √ d 2 L ≤ (1 + Lη k ) (cid:32) (cid:107) e k − 1 (cid:107) 2 + hm √ d 2 L
cid:33) (A.12) ≤ e Lη k (cid:32) (cid:107)
e
k − 1 (cid:107) 2 + hm √ d 2 L (cid:33) .
(A.13)
For k = 1 , . . .
, N , (cid:107) e k (cid:107)
+ hm √ d 2 L ≤ e L ( η 1 + ··· + η k ) hm √ d 2 L ≤ e LT hm √ d 2 L .
(A.14)
Therefore, max k =1 ,...,N (cid:107) X ( t k ) − x ( k ) (cid:107) 2 = max k
=1 ,...,N (cid:107) e k (cid:107) ≤ hm √ d 2 L ( e LT − 1) .
(A.15) (cid:3)
A. P ROOF OF T HEOREMS AND C OROLLARIES IN S ECTION
Proof of Theorem 3.1.
By central limit theorem (CLT),
[Equation]
Since
[Equation]
by Slutsky’s theorem
[Equation]
we conclude that
[Equation]
Proof of Corollary
[Equation]
By asymptotic distribution given by Theorem 3.1,
[Equation]
as n → ∞ .
[Equation]
Proof of Corollary 3.3.
Let τ = E P
[ (cid:96) ( f (Φ( x, y )) , y ) /(cid:96)
( f ( x ) , y )] .
By asymptotic distribution given by Theorem 3.1,
[Equation]
as n → ∞ .
[Equation]
Proof of Theorem 3.4.
For any ( x 0 , y 0 ) ∈
Z , let { X 1 ( t ) } 0 ≤ t ≤ T solve
[Equation]
[Equation]
and { X 2 ( t ) } 0 ≤ t ≤ T solve
[Equation]
Consider
[Equation]
26) we have
[Equation]
and
[Equation]
By Gronwall’s inequality,
[Equation]
that is,
[Equation]
which implies
[Equation]
By Assumption A3, we have
[Equation]
B A SYMPTOTIC NORMALITY AND ASYMPTOTIC VALIDITY OF THE ERROR - RATES RATIO TEST
The auditor collects a set of audit data { ( x i , y i ) } ni =1 and computes the ratio of empirical risks ˜ S n =
[Equation]
and ˜
[Equation]
n
[Equation]
by performing the gradient ﬂow attack (2.6).
Let A n and B n be the numerator and denominator of ˜
S n .
First we derive the limiting distribution of a calibrated test statistic for the error-rates ratio test.
Theorem B.1 (Asymptotic distribution) .
Assume that ∇ x { (cid:96) ( f ( x ) , y ) − λd 2 x ( x, y ) } is Lipschitz in x , and (cid:96) ( f (Φ( x, y )) , y ) and (cid:96) ( f ( x ) , y ) have ﬁnite ﬁrst and second moments.
If the ML model f is independent of { ( x i , y i ) } ni =1 , then
[Equation]
in distribution, as n → ∞ .
Type I error rate control is formalized in the following:
Corollary B.2 (Asymptotic validity of test) .
Under the assumptions in Theorem B.1,
if E P [ (cid:96) 0 , 1 ( f (Φ( x, y )) , y )] / E P [ (cid:96) 0 , 1 ( f ( x ) , y )] ≤ δ , we have lim n →∞ P ( ˜ T n > δ ) ≤ α ; 2.
if E P [ (cid:96) 0 , 1 (
f (Φ( x, y )) , y )] / E P [ (cid:96) 0 , 1 ( f ( x ) , y )] >
δ , we have lim n →∞ P ( ˜ T n > δ ) = 1 .
Proof of Theorem B.1.
By central limit theorem (CLT),
[Equation]
for ﬁnite µ x , µ y , and Σ with ﬁnite entries.
Let g ( x, y ) =
x/y , then ∇ g ( µ x , µ y )
= µ − 2 y ( µ y , − µ x ) (cid:62) .
By continuous mapping theorem, we have
[Equation]
which implies
[Equation]
[Equation]
Since A n p → µ x , B n p → µ y and ˜ V n p → Σ
[Equation]
we therefore conclude by Slutsky’s Theorem that
[Equation]
Proof of Corollary B
[Equation]
Let τ = E P
[ (cid:96) 0 , 1 ( f (Φ( x, y )) , y )] / E P [ (cid:96)
[Equation]
1 ( f ( x ) , y )] .
By asymptotic distribution given by Theorem B
[Equation]
[Equation]
− τ ) ≤ z 1 − α + √ nB 2 n (cid:113)
A 2 n ( ˜ V n ) 2 , 2 + B 2 n ( ˜ V n ) 1 , 1 − 2 A n B n ( ˜ V n ) 1 ,( δ − τ ) 
 → (cid:40) 0 , if τ < δ α, if τ = δ 1 , if τ > δ as n → ∞ .
(cid:3)
C I MPLEMENTATION OF THE PROPOSED TEST
The algorithm 1 provides a step-by-step procedure for calculating the lower bound.
For a choice of δ (threshold for null hypothesis, see equation 3.3), at a level of signiﬁcance 0.05, we reject the null hypothesis if lower bound is greater than δ.
Algorithm
Input : ML model f ; loss (cid:96) ; data { ( X i , Y i ) } ni =1 ; fair-distance d x ; regularization parameters λ ; number of steps T ; and step size { (cid:15) t } T t =1 ;
Require : f provides class probabilities; (cid:15) t is decreasing for i = 1 , . . . , n do
Initialize X (cid:48)
i ←
X i for t = 1 , . .
, T do
X (cid:48)
i ← X (cid:48)
i + (cid:15) t ∇ { (cid:96) ( f ( X (cid:48) i ) , Y i ) − λd x ( X (cid:48)
i , X i ) } end for r i ← (cid:96) ( f ( X (cid:48) i ) ,Y i ) (cid:96) ( f ( X i ) ,Y i ) end for
Output : lower bound = mean ( r ) − 1 .
645 √ n ∗ std ( r )
D S UPPLEMENTARY DETAILS FOR SIMULATIONS
Here we provide further details for the experiment with simulated data. [Figure]
D. D ATA
We considered one one group variable G with two labels.
The 2 dimensional features were generated with the idea that they will differ in ﬁrst co-ordinate.
We present the detailed model for generating the data. 
[Equation]
for i = 1 , . . .
[Equation]
The data is plotted in Figure 3.
As seen in the ﬁgure, feature vectors for two groups mainly differ in the ﬁrst co-ordinate.
So, the discounted movement direction is (1 , 0) T .
D. C LASSIFIERS
For comparison purpose we consider logistic model of the form
[Equation]
where expit ( x ) (cid:44) e x 1+ e x and the weights are chosen as ( w 1 , w 2 ) ∈ {− 4 , − 3 . 6 , . .
For a given ( w 1 , w 2 ) the bias b is chosen as:
[Equation]
where (cid:96) is the logistic loss.
D. L OWER B OUND
To calculate the lower bounds we use Algorithm 1 with the following choices: the choices of (cid:96) and f is provided in the previous subsection.
The choice of fair distances are provided in Section 4.
We choose regularizer λ
= 100 , number of steps T = 400 and step sizes (cid:15) t = 0 .
02 t 2 / 3 .
[Table]
E A DDITIONAL A DULT EXPERIMENT DETAILS E. D ATA PREPROCESSING
The continuous features in Adult are: Age, fnlwgt, capital-gain, capital-loss, hours-per-week, and education-num .
The categorical features are: work-class, education, marital-status, occupation, relationship, race, sex, native-country.
The detailed descriptions can be found in Dua & Graff (2017).
We remove fnlwgt, education, native-country from the features.
race and sex are considered as protected attributes and they are not included in feature vectors for classiﬁcation.
race is treated as binary attribute: White and non-White.
We remove any data-point with missing entry and end up having 45222 data-points.
E. F AIR METRIC
To learn the sensitive subspace, we perform logistic regression for race and sex on other features, and use the weight vectors as the vectors spanning the sensitive subspace ( H ).
The fair metric is then obtained as
[Equation]
E. H YPERPARAMETERS AND TRAINING
For each model, 10 random train/test splits of the dataset is used, where we use 80% data for training purpose.
All compared methods are adjusted to account for class imbalance during training.
E.3. B ASELINE AND P ROJECT
Baseline is the obtained by ﬁtting 2 layer fully connected neural network with 50 neurons for the hidden layer.
It doesn’t enforce any fairness for the model.
Project also uses similar architecture, except a pre-processing layer for projecting out sensitive subspace from features.
So, Project model is a simple and naive way to enforce fairness.
For both the models same parameters are involved: learning_rate for step size for Adam optimizer, batch_size for mini-batch size at training time, and num_steps for number of training steps to be performed.
We present the choice of hyperparameters in Table
E.3. S EN SR
Codes for SenSR (Yurochkin et al., 2020) is provided with submission with a demonstration for ﬁtting the model, where the choice of hyperparameters are provided.
E.3. R EDUCTION
We provide codes for reduction (Agarwal et al., 2018) approach.
We also provide a demonstration for ﬁtting reduction model with the choice of hyperparameters for this experiment.
The codes can also be found in https://github.com/fairlearn/fairlearn .
We used Equalized Odds fairness constraint (Hardt et al., 2016) with constraints violation tolerance parameter (cid:15) = 0 . 03 .
E. L OWER BOUND AND TESTING
To calculate the lower bounds we use Algorithm 1.
The loss (cid:96) is the logistic loss.
Test data is provided as an input, whereas the fair metric is also learnt from the test data.
For each of the models we choose regularizer λ = 50 , number of steps T = 500 and step size (cid:15) t = 0 . [Table]
E. C OMPARISON METRICS
Performance Let C be the set of classes.
Let Y and ˆ Y be the observed and predicted label for a data-point, respectively.
The balanced accuracy is deﬁned as Balanced
[Equation]
Group fairness Let G be the protected attribute taking values in { 0 , 1 } .
The average odds difference (AOD) (Bellamy et al., 2018) for group G is deﬁned as
[Equation]
E.
F
[Table]
In Table 3 we present extended results of the Adult experiment with standard deviations computed from 10 experiment repetitions.
F COMPAS EXPERIMENT
In COMPAS recidivism prediction dataset (Larson et al., 2016)
the task is to predict whether a criminal defendant would recidivate within two years.
We consider race (Caucasian or notCaucasian) and sex (binary) as the sensitive attributes.
The features in COMPAS are: sex, race, priors_count age_cat= 25 to 45,
age_cat= Greater than 45, age_cat=
Less than 25, and c_charge_degree=
F .
prior_count is standardized.
As before we perform testing on four classiﬁers: baseline NN, group fairness Reductions (Agarwal et al., 2018) algorithm, individual fairness SenSR (Yurochkin et al., 2020) algorithm and a basic
Project algorithm that pre-processes the data by projecting out the “senstive” subspace.
Baseline and
Project have same architecture and parameters as in the experiment with Adult dataset.
For SenSR fair metric and Project we use train data to learn the “senstive” subspace.
A further detail for choice of parameters is provided in the code.
For Reduction we used Equalized Odds fairness constraint (Hardt et al., 2016) with constraints violation tolerance parameter (cid:15) = 0 .
All methods are trained to account for class imbalance in the data and we report test balanced accuracy as a performance measure.
Results of the 10 experiment repetitions are summarized in Table 4.
We compare group fairness using average odds difference (AOD)
(Bellamy et al., 2018) for gender and race.
Signiﬁcance level for the null hypothesis rejection is 0.05 and δ = 1 . 25 .
Baseline exhibits clear violations of both individual (test is rejected with proportion 1) and group fairness (both AODs are big in terms of absolute magnitude).
Reductions method achieves signiﬁcant group fairness improvements, but is individually unfair.
Simple pre-processing is more efﬁcient (comparing to the Adult experiment) with rejection proportion of 0.2.
SenSR is the most effective [Table] and our test fails to reject its individual fairness in all experiment repetitions.
Examining the tradeoff between individual and group fairness, we see that both SenSR and Reductions improve all fairness metrics in comparison to the baseline.
However, improvement of individual fairness with
Reductions is marginal.
SenSR provides a sizeable improvement of gender AOD, but only a marginal improvement of race AOD.