Original: T RAINING INDIVIDUALLY FAIR ML MODELS WITH SENSITIVE SUBSPACE ROBUSTNESS
Revised: T RAINING INDIVIDUALLY FAIR ML MODELS WITH S ENSITIVE S UBSPACE R OBUSTNESS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose an approach to training machine learning models that are fair in the sense that their performance is invariant under certain  perturbations to the features.
Revised: We consider training machine learning models that are fair in the sense that their performance is invariant under certain sensitive perturbations to the inputs.

Original: For example, the performance of a resume screening system should be invariant under changes to the name of the applicant.
Revised: For example, the performance of a resume screening system should be invariant under changes to the gender and/or ethnicity of the applicant.

Original: We formalize thisintuitive notion of  fairness by connecting it to the original notion of individual fairnessput forth by Dwork et al and show that the proposed approach achieves this notion of fairness.
Revised: We formalize this notion of algorithmic fairness as a variant of individual fairness and develop a distributionally robust optimization approach to enforce it during training.

Original: We also demonstrate the effectiveness of the approach on two machine learning tasks that are susceptible to gender and racial biases.
Revised: We also demonstrate the effectiveness of the approach on two ML tasks that are susceptible to gender and racial biases.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: As AI systems permeate our world, the problem of implicit biases in these systems have become more serious.
Revised: Machine learning (ML) models are gradually replacing humans in high-stakes decision making roles.

Original: AI systems are routinely used to make decisions or support the decision-making process in credit, hiring, criminal justice, and education, all of which are domains protected by anti-discrimination law.
Revised: For example, in Philadelphia, an ML model classiﬁes probationers as high or low-risk (Metz & Satariano, 2020). In North Carolina, “analytics” is used to report suspicious activity and fraud by Medicaid patients and providers (Metz & Satariano, 2020).

Original: Although AI systems appear to eliminate the biases of a human decision maker, they may perpetuate or even exacerbate biases in the training data (Barocas & Selbst, 2016).
Revised: Although ML models appear to eliminate the biases of a human decision maker, they may perpetuate or even exacerbate biases in the training data (Barocas & Selbst, 2016).

Original: Such biases are especially objectionable when it adversely affects underprivileged groups of users (Barocas & Selbst, 2016).
Revised: Such biases are especially objectionable when it adversely affects underprivileged groups of users (Barocas & Selbst, 2016).

Original: In response, the scientiﬁc community has proposed many formal deﬁnitions of algorithmic fairness and approaches to ensure AI systems remain fair.
Revised: In response, the scientiﬁc community has proposed many mathematical deﬁnitions of algorithmic fairness and approaches to ensure ML models satisfy the deﬁnitions.

Original: Unfortunately, this abundance of deﬁnitions, many of which are incompatible (Kleinberg et al., 2016; Chouldechova, 2017), has hindered the adoption of this work by practitioners (Corbett-Davies & Goel, 2018).
Revised: Unfortunately, this abundance of deﬁnitions, many of which are incompatible (Kleinberg et al., 2016; Chouldechova, 2017), has hindered the adoption of this work by practitioners.

Original: There are two types of formal deﬁnitions of algorithmic fairness: group fairness and individual fairness.
Revised: There are two types of formal deﬁnitions of algorithmic fairness: group fairness and individual fairness.

Original: Most recent work on algorithmic fairness considers group fairness because it is more amenable to statistical analysis (Jiang et al., 2019).
Revised: Most recent work on algorithmic fairness considers group fairness because it is more amenable to statistical analysis (Ritov et al., 2017).

Original: Despite their prevalence, group notions of algorithmic fairness suffer from certain shortcomings.
Revised: Despite their prevalence, group notions of algorithmic fairness suffer from certain shortcomings.

Original: One of the most troubling is there are many scenarios in which an algorithm satisﬁes group fairness, but its output is blatantly unfair from the point of view of individual users (Dwork et al., 2011).
Revised: One of the most troubling is there are many scenarios in which an algorithm satisﬁes group fairness, but its output is blatantly unfair from the point of view of individual users (Dwork et al., 2011).

Original: In this paper, we consider individual fairness instead of group fairness.
Revised: In this paper, we consider individual fairness instead of group fairness.

Original: At a high-level, an individually fair ML model treats similar users similarly.
Revised: Intuitively, an individually fair ML model treats similar users similarly.

Original: Formally,we consider an ML model as a map h : X → Y , where X and Y are the input and output spaces.
Revised: Formally, an ML model is a map h : X → Y , where X and Y are the input and output spaces.

Original: The leading notion of individual fairness is metric fairness (Dwork et al., 2011); it requiresd y ( [Equation] [Equation] 1 ) , h ( x
Revised: The leading notion of individual fairness is metric fairness (Dwork et al., 2011); it requires

Original: [Equation]

Original: ≤

Original: [Equation]

Original: ( x

Original: [Equation]

Original: x

Original: [Equation]
Revised: [Equation]

Original: for all

Original: [Equation]

Original: [Equation] X , ( [Equation] .1) where d x and d y are metrics on the input and output spaces and L ∈ R + .
Revised: where d x and d y are metrics on the input and output spaces and L ≥ 0 is a Lipschitz constant.

Original: The fair metric d x encodes our intuition of which samples should be treated similarly by the ML model.
Revised: The fair metric d x encodes our intuition of which samples should be treated similarly by the ML model.

Original: We emphasize that d x ( x 1 , x 2 ) being small does NOT imply x 1 and x 2 are similar in all respects.
Revised: We emphasize that d x ( x 1 , x 2 ) being small does not imply x 1 and x 2 are similar in all respects.

Original: Even if d x ( x 1 , x 2 ) is small, x 1 and x 2 may differ in certain attributes that are irrelevant to the ML task at hand, e.g. protected attributes.
Revised: Even if d x ( x 1 , x 2 ) is small, x 1 and x 2 may differ in certain problematic ways, e.g. in their protected/sensitive attributes.

Original: This is why we refer to pairs of samples x 1 and x 2 such that d x ( x 1 , x 2 ) is small as comparable instead of similar.
Revised: This is why we refer to pairs of samples x 1 and x 2 such that d x ( x 1 , x 2 ) is small as comparable instead of similar.

Original: Despite its beneﬁts, individual fairness is considered impractical because the choices of d x and d y are ambiguous.
Revised: Despite its beneﬁts, individual fairness was dismissed as impractical because there is no widely accepted fair metric for many ML tasks.

Original: Unfortunately, in application areas where there isdisagreement over the choice of d x and/or d y , this ambiguity negates most of the beneﬁts of a formal deﬁnition of fairness. Dwork et al. (2011) consider randomized ML algorithms, so h ( x ) is generally a random variable. They suggest probability metrics ( e.g. total variation distance) as d y and defer the choice of d x to regulatory bodies or civil rights organizations, but we are unaware of commonly accepted choices of d x .
Revised: Fortunately, there is a line of recent work on learning the fair metric from data (Ilvento, 2019; Wang et al., 2019).

Original: In this paper, we consider two data-driven choices of the fair metric: one for problems in which the sensitive attribute is reliably observed, and another for problems in which the sensitive attribute is unobserved. Due to space constraints, we defer the details to the supplement (see Appendix B). 
Revised: In this paper, we consider two data-driven choices of the fair metric: one for problems in which the sensitive attribute is reliably observed, and another for problems in which the sensitive attribute is unobserved (see Appendix B). The rest of this paper is organized as follows.

Original: In this paper, we consider an adversarial approach to training individually fair ML models: we show that individual fairness is arestricted form of robustness: robustness to certain sensitive perturbations to the inputs of an ML model.
Revised: In Section 2, we cast individual fairness as a form of robustness: robustness to certain sensitive perturbations to the inputs of an ML model.

Original: Thisconnection allows us to leverage recent advances in adversarial training (Madry et al., 2017) to train individually fair ML models.
Revised: This allows us to leverage recent advances in adversarial ML to train individually fair ML models.

Original: The paper is organized into four main sections.

Original: In Section 2 we develop a method to investigate algorithmic bias/unfairness in ML models. This leads to a training method that trains ML models to pass such investigations.
Revised: More concretely, we develop an approach to audit ML models for violations of individual fairness that is similar to adversarial attacks (Goodfellow et al., 2014) and an approach to train ML models that passes such audits (akin to adversarial training (Madry et al., 2017)).

Original: Our algorithmic developments are followed by a theoretical investigation (see Section 3) and an empirical study (see Section 3) of the efﬁcacy of the proposed approach on two ML tasks.
Revised: We justify the approach theoretically (see Section 3) and empirically (see Section 4).

Original:  F AIRNESS THROUGH  ROBUSTNESS
Revised:  F AIRNESS THROUGH ( DISTRIBUTIONAL ) ROBUSTNESS

Original: To motivate our approach, imagine an investigator auditing an AI system for unfairness.
Revised: To motivate our approach, imagine an auditor investigating an ML model for unfairness.

Original: The investigator collects a set of audit data and compares the output of the AI system on comparable samples in the audit data.
Revised: The auditor collects a set of audit data and compares the output of the ML model on comparable samples in the audit data.

Original: For example, to investigate whether a resume screening system is fair, the investigator may collect a stack of resumes and change the names on the resumes of Caucasian applicants to names more common among the African-American population.
Revised: For example, to investigate whether a resume screening system is fair, the auditor may collect a stack of resumes and change the names on the resumes of Caucasian applicants to names more common among the African-American population.

Original: If the system performs worse on the edited resumes,  the investigator concludes the system treatsresumes from African-American applicants unfairly.
Revised: If the system performs worse on the edited resumes, then the auditor may conclude the model treats African-American applicants unfairly.

Original: This isthe premise of Bertrand and Mullainathan’s celebrated investigation of racial discrimination in the labor market (Bertrand & Mullainathan, 2004).
Revised: Such investigations are known as correspondence studies , and a prominent example is Bertrand & Mullainathan’s celebrated investigation of racial discrimination in the labor market.

Original: Abstractly, the investigator looks for inputs that are comparable to the training examples (the unedited resumes in the preceding example)but on which the predictor performs poorly.
Revised: In a correspondence study, the investigator looks for inputs that are comparable to the training examples (the edited resumes in the resume screening example) on which the ML model performs poorly.

Original: In the rest of this section, we formulate an optimization problem to ﬁnd such inputs.
Revised: In the rest of this section, we formulate an optimization problem to ﬁnd such inputs.

Original: Setup Recall X and Y are the spaces of inputs and outputs of the ML algorithm.
Revised: Recall X and Y are the spaces of inputs and outputs.

Original: To keep things simple, we assume  Y is discrete.
Revised: To keep things simple, we assume that the ML task at hand is a classiﬁcation task, so Y is discrete.

Original: We also assumefor now that we have a fair metric d x of the form
Revised: We also assume that we have a fair metric d x of the form

Original: [Equation]
Revised: [Equation]

Original: where Σ ∈ S d × d ++ is a covariance matrix for the ML task at hand.
Revised: where Σ ∈ S d × d + .

Original: For example, suppose we are given a set of K “sensitive” directions A ∈ R d × K .
Revised: For example, suppose we are given a set of K “sensitive” directions that we wish the metric to ignore; i.e. d ( x 1 , x 2 ) (cid:28) 1 for any x 1 and x 2 such that x 1 − x 2 falls in the span of the sensitive directions.

Original: These directions may be provided by a subject expert or estimated from data — we elaborate further in the experiments section and Appendix B.
Revised: These directions may be provided by a domain expert or learned from data (see Section 4 and Appendix B).

Original: We wish to have a metric insensitive to any perturbation of features within the sensitive subspace ran ( A ) . To achieve this  we set Σ = I − P ran ( A ) , i.e. the orthogonal complement projector of ran ( A ) .
Revised: In this case, we may choose Σ as the orthogonal complement projector of the span of the sensitive directions.

Original: We equip X with such metric. Let Z = X
Revised: We equip X with the fair metric and Z (cid:44) X

Original: × Y, and equip it withthe metric
Revised: × Y with

Original: [Equation]
Revised: [Equation]

Original: We consider d 2 z as a transportation cost function on Z .
Revised: We consider d 2 z as a transport cost function on Z .

Original: This cost function encodes our intuition of which samples are comparable.
Revised: This cost function encodes our intuition of which samples are comparable for the ML task at hand.

Original: We equip∆( Z ) , the set of probability distributions on Z, with the  Wasserstein distance
Revised: We equip the space of probability distributions on Z with the fair Wasserstein distance

Original: [Equation]
Revised: [Equation]

Original: where C ( P, Q ) is the set of couplings between P and Qand the transportation cost function is c ( z 1 , z 2 ) = d z ( z 1 , z 2 ) 2 .
Revised: where C ( P, Q ) is the set of couplings between P and Q .

Original: This fair Wasserstein distance inherits our intuition of which samples are comparable through the cost function; i.e. the fair Wasserstein distance between two probability distributions is small if they are supported on comparable areas of the sample space.
Revised: The fair Wasserstein distance inherits our intuition of which samples are comparable through the cost function; i.e. the fair Wasserstein distance between two probability distributions is small if they are supported on comparable areas of the sample space.

Original: To investigate whether an AI system h performs disparately on comparable samples, the investigator collects a set of audit data { ( x i , y i ) } ni =1that is independent of the training data andpicks a nonnegative loss function (cid:96) : Z × H → R + to measure the performance of the AI system. The investigator solves the optimization problem
Revised: To investigate whether an ML model performs disparately on comparable samples, the auditor collects a set of audit data { ( x i , y i ) } ni =1 and solves the optimization problem

Original: [Equation]
Revised: [Equation]

Original:  where  P n is the empirical distribution of the audit data and (cid:15) > 0 is a budget parameter.
Revised: ) where (cid:96) : Z ×H → R + is a loss function, h is the ML model, P n is the empirical distribution of the audit data, and (cid:15) > 0 is a small tolerance parameter.

Original: We interpret (cid:15) as a moving budget that the investigator may expend to discover discrepancies in the performance of the AI system.
Revised: We interpret (cid:15) as a moving budget that the auditor may expend to discover discrepancies in the performance of the ML model.

Original: This budget forces the investigator to avoid moving samples to incomparable areas of the sample space.
Revised: This budget forces the auditor to avoid moving samples to incomparable areas of the sample space.

Original: We emphasize that equation 2.1 detects aggregate violations of individual fairness.
Revised: We emphasize that equation 2.1 detects aggregate violations of individual fairness.

Original: In other words, although the violations that the investigator’s problem detects are individual in nature, the investigator’s problem is only able to detect collective violations.
Revised: In other words, although the violations that the auditor’s problem detects are individual in nature, the auditor’s problem is only able to detect aggregate violations.

Original: The implicit notion of fairness in equation 2.1 is
Revised: We summarize the implicit notion of fairness in equation 2.1 in a deﬁnition.

Revised: Deﬁnition 2.1 (distributionally robustly fair (DRF)) .

Revised: An ML model h : X → Y is ( (cid:15), δ ) - distributionally robustly fair (DRF) WRT the fair metric d x iff

Original: [Equation]
Revised: [Equation]

Original: where δ > 0 is a small tolerance for discrepancies in the performance of the ML model on comparable inputs.

Original: Although equation 2.1 is an inﬁnite-dimensional optimization problem, it isconvex, so it is possible toexploit duality to solve it exactly.
Revised: Although equation 2.1 is an inﬁnite-dimensional optimization problem, it is possible to solve it exactly by appealing to duality.

Original: It is known (Blanchet & Murthy, 2016) that the dual of equation 2.1 is
Revised: Blanchet & Murthy showed that the dual of equation 2.1 is

Original: [Equation]
Revised: [Equation]

Original: The function (cid:96) cλ is called the c -transform of (cid:96) . This is a univariate optimization problem, and it is amenable to stochastic optimization (see Algorithm 1).
Revised: This is a univariate optimization problem, and it is amenable to stochastic optimization. We describe a stochastic approximation algorithm for equation 2.3 in Algorithm 1.

Revised: Inspecting the algorithm, we see that it is similar to the PGD algorithm for adversarial attack.

Original: To optimize equation 2.3, it is imperative that the investigator is able to evaluate ∂ x (cid:96) (( x, y ) , h ) .
Revised: [Table] B , [Table] B ) ∼

Original: Require: starting point ˆ λ 1 , step sizes α t >1: repeat 2: draw mini-batch ( x t 1 , y t 1 ) , . . .
Revised: [Table] n [Table] ← arg

Original: , ( x t B , y t B ) ∼ P n 3: x ∗ t b ← arg max x ∈X (cid:96) (( x, y t b ) , h ) − λd x ( x t b , x ) , b ∈ [ B ] 4: ˆ λ t +1 ← max { 0 , ˆ λ t − α t ( (cid:15) − 1 B (cid:80)
Revised: [Table] [Table] ) , h ) − [Table]

Original: Bb =1 d x ( x t b ,x ∗ t b )) } 5: until converged
Revised: [Table] ) , b ∈ [ B ] [Table] [Table]

Original: It is known that the optimal point of equation 2.1 is the discrete measure 1 n (cid:80) ni
Revised: It is known that the optimal point of equation 2.1 is the discrete measure 1 n (cid:80) ni

Original: =1 δ ( T λ ( x i ) ,y i ) , where T λ : X → X is the unfair map
Revised: =1 δ ( T λ ( x i ) ,y i ) , where T λ : X → X is the unfair map

Original: [Equation]
Revised: [Equation]

Original: We call T λ an unfair map because it reveals unfairness in the AI system by mapping samples in the audit data to comparable areas of the sample space that the system performs poorly on.
Revised: We call T λ an unfair map because it reveals unfairness in the ML model by mapping samples in the audit data to comparable areas of the sample space that the system performs poorly on.

Original: We note that T λ may map samples in the audit data to areas of the sample space that are not represented in the audit data, thereby revealing disparate treatment in the AI system not visible from the audit data alone.
Revised: We note that T λ may map samples in the audit data to areas of the sample space that are not represented in the audit data, thereby revealing disparate treatment in the ML model not visible in the audit data alone.

Original: We emphasize that T λ more than reveals disparate treatment in the AI system; it localizes the unfairness to certain areas of the sample space.
Revised: We emphasize that T λ more than reveals disparate treatment in the ML model; it localizes the unfairness to certain areas of the sample space.

Original: We present a simple example to illustrating fairness through robustness (a similar example appeared in Hashimoto et al.
Revised: We present a simple example to illustrating fairness through robustness (a similar example appeared in Hashimoto et al.

Original: Consider the binary classiﬁcation dataset shown in Figure 1.
Revised: Consider the binary classiﬁcation dataset shown in Figure 1.

Original: There are two subgroups of observations in this dataset, and (sub)group membership is the protected attribute ( e.g. the smaller group contains observations from a minority subgroup).
Revised: There are two subgroups of observations in this dataset, and (sub)group membership is the protected attribute ( e.g. the smaller group contains observations from a minority subgroup).

Original: In the ﬁgure, the horizontal axis displays (the value of) the protected attribute, while the vertical axis displays the discriminative attribute.

Original: In Figure 1a we see the decision heatmap of a vanilla logistic regression, which performs poorly on the blue minority subgroup.
Revised: In Figure 1a we see the decision heatmap of a vanilla logistic regression, which performs poorly on the blue minority subgroup.

Original: A tenable fair metric in this instance is a metric that downweights differences in the horizontal  direction.
Revised: The two subgroups are separated in the horizontal direction, so the horizontal direction is the sensitive direction.

Original: Figure 1b shows that such classiﬁer is unfair with respect to the aforementioned fair metric, i.e. the unfair map equation 2.4 leads to signiﬁcant loss increase by transporting mass along the horizontal direction with very minor change of the vertical coordinate.
Revised: Figure 1b shows that such classiﬁer is unfair with respect to the corresponding fair metric, i.e. the unfair map equation 2.4 leads to signiﬁcant loss increase by transporting mass along the horizontal direction with very minor change of the vertical coordinate.

Original: Comparison with Dwork et al. (2011) Before moving on to training individually fair ML models, we compare our notion of fairness equation 2.2 with the deﬁnition of individual fairness in Dwork et al.
Revised: Comparison with metric fairness Before moving on to training individually fair ML models, we compare DRF with metric fairness equation 1.1. Although we concentrate on the differences between the two deﬁnitions here, they are more similar than different: both formalize the intuition that the outputs of a fair ML model should perform similarly on comparable inputs.

Original: Although we concentrate on the differences between the two deﬁnitions here, they [Figure] [Figure]
Revised: That said, there are two main differences between the two deﬁnitions. First, instead of requiring the output of the

Original: map      1. [Figure] are more similar than different: both formalize the intuition that the outputs of a fair ML model should perform similarly on  similar inputs.
Revised: ML model to be similar on all inputs comparable to a training example, we require the output to be similar to the training label.

Original: That said, there are two main differences between the two deﬁnitions.

Original: First, instead ofrequiring the output  of the ML modelto be similar onall inputs comparable to a training example, we require theoutput to be similar to the training label. The two deﬁnitions have similar implications. Let ( x i , y i ) be a training example and x (cid:48) i be an input point such that d x ( x i , x (cid:48) i ) ≤ (cid:15) .
Revised: Thus DRF not only enforces similarity of the output on comparable inputs, but also accuracy of the ML model on the training data.

Original: Dwork et al’s deﬁnition implies
Revised: Second, DRF considers differences

Original: [Equation]
Revised: [Figure]

Original: [Equation]

Original: On a single training example, we consider a given predictor h fair if sup

Original: [Equation] [Equation] ≤ δ . This implies
Revised: [Figure]

Original: [Equation]
Revised: [Figure]

Original: Comparing equation 2.5 and equation 2.6, we see that the two implications are similar, with (cid:96) in equation 2.6 playing the role of d y in equation 2.5 and 2 δ in equation 2.6 playing the part of L equation 2.5.

Original: The main beneﬁt of our deﬁnition is it encodes not only (individual) fairness but also accuracy.

Original: Second, we consider differences between datasets instead of samples by replacing Dwork et al’s fair metric on inputs with the  Wasserstein distance(on input distributions) induced by the fair metric.
Revised: between datasets instead of samples by replacing the fair metric on inputs with the fair Wasserstein distance induced by the fair metric.

Original: The main beneﬁts of this modiﬁcations are (i) it is possible to optimize equation 2.1 efﬁciently, (ii) we can show this modiﬁed notion of individual fairness generalizes.
Revised: The main beneﬁts of this modiﬁcations are (i) it is possible to optimize equation 2.1 efﬁciently, (ii) we can show this modiﬁed notion of individual fairness generalizes.

Revised: 2. F AIR TRAINING WITH S ENSITIVE S UBSPACE R OBUSTNESS

Original: We cast the fair training problem as training supervised learning systems that are robust to sensitive perturbations.
Revised: We cast the fair training problem as training supervised learning systems that are robust to sensitive perturbations.

Original: We propose solving the minimax problem
Revised: We propose solving the minimax problem

Original: [Equation]
Revised: [Equation]

Original: where (cid:96) cλ is deﬁned in equation 2.3.
Revised: where (cid:96) cλ is deﬁned in equation 2.3.

Original: This is an instance of a distributionally robust optimization (DRO) problem, and it inherits some of the statistical properties of DRO.
Revised: This is an instance of a distributionally robust optimization (DRO) problem, and it inherits some of the statistical properties of DRO.

Original: To see why equation 2.
Revised: To see why equation 2.

Original: encourages individual fairness, recall the loss function is a measure of the performance of the AI system.
Revised: encourages individual fairness, recall the loss function is a measure of the performance of the ML model.

Original: By assessing the performance of an AI system by its worse-case performance on hypothetical populations of users with perturbed sensitive attributes, minimizing equation 2.7 ensures the system performs well on all such populations.
Revised: By assessing the performance of an ML model by its worse-case performance on hypothetical populations of users with perturbed sensitive attributes, minimizing equation 2.5 ensures the system performs well on all such populations.

Original: In our toy example, minimizing equation 2.7 implies learning a classiﬁer that is insensitive to perturbations along the horizontal (i.e. sensitive) direction.
Revised: In our toy example, minimizing equation 2.5 implies learning a classiﬁer that is insensitive to perturbations along the horizontal (i.e. sensitive) direction.

Original: In Figure 1c this is achieved by the algorithm we describe next.
Revised: In Figure 1c this is achieved by the algorithm we describe next.

Original: To keep things simple, we assume the hypothesis class is parametrized by θ ∈ Θ ⊂ R d and replace the minimization with respect to H by minimization with respect to θ .
Revised: To keep things simple, we assume the hypothesis class is parametrized by θ ∈ Θ ⊂ R d and replace the minimization with respect to H by minimization with respect to θ .

Original: In light of the similarities between the DRO objective function and adversarial training, we borrow algorithms for adversarial training (Madry et al., 2017) to solve equation 2.7 (see Algorithm 2).
Revised: In light of the similarities between the DRO objective function and adversarial training, we borrow algorithms for adversarial training (Madry et al., 2017) to solve equation 2.5 (see Algorithm 2).

Original: Require: starting point ˆ θ 1 , step sizes α t , β t >1: repeat 2: sample mini-batch ( x 1 , y 1 ) , . . .
Revised: Require: starting point ˆ θ 1 , step sizes α t , β t >1: repeat 2: sample mini-batch ( x 1 , y 1 ) , . . .

Original: , ( x B , y B ) ∼ P n 3: x ∗ t b ← arg max x ∈X (cid:96) (( x, y t b ) , θ ) − ˆ λ t d x ( x t b , x ) , b ∈ [ B ] 4: ˆ λ t +1 ← max { 0 , ˆ λ t − α t ( (cid:15) − 1 B (cid:80)
Revised: , ( x B , y B ) ∼ P n 3: x ∗ t b ← arg max x ∈X (cid:96) (( x, y t b ) , θ ) − ˆ λ t d x ( x t b , x ) , b ∈ [ B ] 4: ˆ λ t + [Table] ← max { 0 , ˆ λ t − α t ( (cid:15) − 1 B (cid:80)

Original: Bb =1 d x ( x t b , x ∗ t b )) } 5:
Revised: Bb =1 d x ( x t b , x ∗ t b )) } 5:

Original: ˆ θ t +1 ← ˆ θt − β t B (cid:80)
Revised: [Table] θ [Table] ← ˆ θ

Original:  Bb =1 ∂ θ (cid:96) (( x ∗ t b , y t b ) , ˆ θ t ) 6: until converged
Revised: [Table] − β t B (cid:80) Bb =1 ∂ θ (cid:96) (( x ∗ t b , y t b ) , ˆ θ t ) 6: until converged

Original: Related work Our approach to fair training is an instance of distributionally robust optimization (DRO), which minimize objectives of the form sup P ∈U E P (cid:2) (cid:96) ( Z, θ ) (cid:3) , where U is a (data dependent) uncertainty set of probability distributions.
Revised: Related work Our approach to fair training is an instance of distributionally robust optimization (DRO). In DRO, the usual sample-average approximation of the expected cost function is replaced by (cid:98) L DRO ( θ ) (cid:44) sup P ∈U E P (cid:2) (cid:96) ( Z, θ ) (cid:3) , where U is a (data dependent) uncertainty set of probability distributions.

Original: Other instances of DRO consider uncertainty sets deﬁned by moment or support constraints (Chen et al., 2007;
Revised: The uncertainty set may be deﬁned by moment or support constraints (Chen et al., 2007;

Original: Delage & Ye, 2010; Goh & Sim, 2010) as well as distances between probability distributions, such as f -divergences (Ben-Tal et al., 2012; 
Revised: Delage & Ye, 2010; Goh & Sim, 2010), f -divergences (Ben-Tal et al., 2012; Lam & Zhou,

Original: Lam & Zhou, 2015; Miyato et al., 2015;Duchi et al., 2016; Namkoong & Duchi, 2016) and Wasserstein distances (Shaﬁeezadeh-Abadeh et al., 2015; Blanchet et al., 2016;Duchi & Namkoong, 2016; Esfahani & Kuhn, 2015; Lee & Raginsky, 2017; Sinha et al., 2017; Hashimoto et al., 2018).
Revised: ; Miyato et al., 2015; Namkoong & Duchi, 2016), and Wasserstein distances (ShaﬁeezadehAbadeh et al., 2015; Blanchet et al., 2016; Esfahani & Kuhn, 2015; Lee & Raginsky, 2017; Sinha et al., 2017).

Original: Most similar to our work is Hashimoto et al.
Revised: Most similar to our work is Hashimoto et al.

Original: (2018): they show that DRO with a χ 2 -neighborhood of the training data prevents representation disparity, i.e. minority groups tend to suffer higher losses because the training algorithm ignores them.
Revised: (2018): they show that DRO with a χ 2 -neighborhood of the training data prevents representation disparity, i.e. minority groups tend to suffer higher losses because the training algorithm ignores them.

Original: One advantage of picking a Wasserstein uncertainty set is the set depends on the geometry of the sample space.
Revised: One advantage of picking a Wasserstein uncertainty set is the set depends on the geometry of the sample space.

Original: This allows us to encode the correct notion of individual fairness for the ML task at hand in the Wasserstein distance.
Revised: This allows us to encode the correct notion of individual fairness for the ML task at hand in the Wasserstein distance.

Original: Our approach to fair training is also similar to adversarial training (Madry et al., 2017), which hardens AI systems against adversarial attacks by minimizing adversarial losses of the form sup u ∈U (cid:96) ( z + u, θ ) , where U is a set of allowable perturbations (Szegedy et al., 2013; Goodfellow et al., 2014; Papernot et al., 2015; Carlini & Wagner, 2016; Kurakin et al., 2016).
Revised: Our approach to fair training is also similar to adversarial training (Madry et al., 2017), which hardens ML models against adversarial attacks by minimizing adversarial losses of the form sup u ∈U (cid:96) ( z + u, θ ) , where U is a set of allowable perturbations (Szegedy et al., 2013; Goodfellow et al., 2014; Papernot et al., 2015; Carlini & Wagner, 2016; Kurakin et al., 2016).

Original: Typically, U is a scaled (cid:96) p -norm ball: U = { u : (cid:107)
Revised: Typically, U is a scaled (cid:96) p -norm ball: U = { u : (cid:107)

Original: u
Revised: u

Original: (cid:107) p ≤ (cid:15) } .
Revised: (cid:107) p ≤ (cid:15) } .

Original: Most similar to our work is Sinha et al.
Revised: Most similar to our work is Sinha et al.

Original: (2017): they consider an uncertainty set that is a Wasserstein neighborhood of the training data.
Revised: (2017): they consider an uncertainty set that is a Wasserstein neighborhood of the training data.

Original: There are a few papers that consider adversarial approaches to algorithmic fairness.
Revised: There are a few papers that consider adversarial approaches to algorithmic fairness.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2018) propose an adversarial learning method that enforces equalized odds in which the adversary learns to predict the protected attribute from the output of the classiﬁer.
Revised: (2018) propose an adversarial learning method that enforces equalized odds in which the adversary learns to predict the protected attribute from the output of the classiﬁer.

Original: Edwards & Storkey (2015) propose an adversarial method for learning classiﬁers that satisfy demographic parity.
Revised: Edwards & Storkey (2015) propose an adversarial method for learning classiﬁers that satisfy demographic parity.

Original: Madras et al.
Revised: Madras et al.

Original: (2018) generalize their method to learn classiﬁers that satisfy other (group) notions of algorithmic fairness.
Revised: (2018) generalize their method to learn classiﬁers that satisfy other (group) notions of algorithmic fairness.

Original: Garg et al.
Revised: Garg et al.

Original: (2019) propose to use adversarial logit pairing (Kannan et al., 2018) to achieve fairness in text classiﬁcation using a pre-speciﬁed list of counterfactual tokens.
Revised: (2019) propose to use adversarial logit pairing (Kannan et al., 2018) to achieve fairness in text classiﬁcation using a pre-speciﬁed list of counterfactual tokens.

Original:  S EN SR TRAINS INDIVIDUALLY FAIR ML MODELS
Revised:  S EN SR TRAINS INDIVIDUALLY FAIR ML MODELS

Original: One of the main beneﬁts of our approach is it provably trains individually fair ML models.
Revised: One of the main beneﬁts of our approach is it provably trains individually fair ML models.

Original: Further, it is possible for the learner to certify that an ML model is individually fair a posteriori .
Revised: Further, it is possible for the learner to certify that an ML model is individually fair a posteriori .

Original: As we shall see, both are consequences of uniform convergence results for the DR loss class.
Revised: As we shall see, both are consequences of uniform convergence results for the DR loss class.

Original: More concretely, we study how quickly the uniform convergence error
Revised: More concretely, we study how quickly the uniform convergence error

Original: [Equation]
Revised: [Equation]

Original:  where W ∗ is the Wasserstein distance on ∆( Z ) with a transportation cost function c ∗ that is possibly different from c , vanishes.
Revised: [Equation] where W ∗ is the Wasserstein distance on ∆( Z ) with a transportation cost function c ∗ that is possibly different from c , vanishes.

Original: We permit some discrepancy in the (transportation) cost function to study the effect of a data-driven choice of c .
Revised: We permit some discrepancy in the (transportation) cost function to study the effect of a data-driven choice of c .

Original: In the rest of this section, we regard c ∗ as the exact cost function and c as a cost function learned from human supervision.
Revised: In the rest of this section, we regard c ∗ as the exact cost function and c as a cost function learned from human supervision.

Original: We start by stating our assumptions on the ML task:
Revised: We start by stating our assumptions on the ML task:

Original: (A1) the feature space X is bounded: D (cid:44) max { diam ( X ) , diam ∗ ( X ) } < ∞ ; (A2) the functions in the loss class L = { (cid:96) ( · , θ ) : θ ∈ Θ } are non-negative and bounded: 0 ≤ (cid:96) ( z, θ ) ≤ M for all z ∈ Z and θ ∈ Θ , and L -Lipschitz with respect to d x : sup θ ∈ Θ { sup ( x 1 ,y ) , ( x 2 ,y ) ∈Z | (cid:96) (( x 1 , y ) , θ ) − (cid:96) (( x 2 , y ) , θ ) |} ≤ Ld x ( x 1 , x 2 );
Revised: (A1) the feature space X is bounded: D (cid:44) max { diam ( X ) , diam ∗ ( X ) } < ∞ ; (A2) the functions in the loss class L = { (cid:96) ( · , θ ) : θ ∈ Θ } are non-negative and bounded: 0 ≤ (cid:96) ( z, θ ) ≤ M for all z ∈ Z and θ ∈ Θ , and L -Lipschitz with respect to d x : sup θ ∈ Θ { sup ( x [Equation] ) the discrepancy in the (transportation) cost function is uniformly bounded

Original: [Equation]
Revised: [Equation]

Original: Assumptions A1 and A2 are standard (see (Lee & Raginsky, 2017, Assumptions 1, 2, 3)), but A deserves further comment.
Revised: Assumptions A1 and A2 are standard (see (Lee & Raginsky, 2017, Assumption 1, 2, 3)) in the DRO literature.

Revised: We emphasize that the constant L in Assumption A2 is not the constant L in the deﬁnition of metric fairness; it may be much larger.

Revised: Thus most models that satisfy the conditions of the loss class are not individually fair in a meaningful sense.

Revised: Assumption A3 deserves further comment.

Original: Under A1, A3 is mild.
Revised: Under A1, A3 is mild.

Original: For example, if the exact fair metric is
Revised: For example, if the exact fair metric is

Original: [Equation]
Revised: [Equation]

Original: then the error in the transportation cost function is at most
Revised: then the error in the transportation cost function is at most

Original: [Equation]
Revised: [Equation]

Original: We see that the error in the transportation cost function vanishes in the large-sample limit as long as Σ is a consistent estimator of Σ ∗ .
Revised: We see that the error in the transportation cost function vanishes in the large-sample limit as long as Σ is a consistent estimator of Σ ∗ .

Original: We state the uniform convergence result in terms of the entropy integral of the loss class: C ( L ) =
Revised: We state the uniform convergence result in terms of the entropy integral of the loss class: C ( L ) =

Original: (cid:82) ∞ 0 (cid:112) log N ∞ ( F , r ) dr , where N ∞ ( L , r ) as the r -covering number of the loss class in the uniform metric.
Revised: (cid:82) ∞ 0 (cid:112) log N ∞ ( F , r ) dr , where N ∞ ( L , r ) as the r -covering number of the loss class in the uniform metric.

Original: The entropy integral is a measure of the complexity of the loss class.
Revised: The entropy integral is a measure of the complexity of the loss class.

Original: Proposition 3.1 (uniform convergence) .
Revised: Proposition 3.1 (uniform convergence) .

Original: Under Assumptions A1–A3, equation 3.1 satisﬁes
Revised: Under Assumptions A1–A3, equation 3.1 satisﬁes

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − t .
Revised: with probability at least 1 − t .

Original: We note that Proposition 3.1 is similar to the generalization error bounds by Lee & Raginsky (2017).
Revised: We note that Proposition 3.1 is similar to the generalization error bounds by Lee & Raginsky (2017).

Original: The main novelty in Proposition 3.1 is allowing error in the transportation cost function.
Revised: The main novelty in Proposition 3.1 is allowing error in the transportation cost function.

Original: We see that the discrepancy in the transportation cost function may affect the rate at which the uniform convergence error vanishes: it affects the rate if δ c is ω P ( 1 √ n ) .
Revised: We see that the discrepancy in the transportation cost function may affect the rate at which the uniform convergence error vanishes: it affects the rate if δ c is ω P ( 1 √ n ) .

Original: A consequence of uniform convergence is SenSR trains individually fair classiﬁers (if there are such classiﬁers in the hypothesis class).
Revised: A consequence of uniform convergence is SenSR trains individually fair classiﬁers (if there are such classiﬁers in the hypothesis class).

Original: By individually fair ML model, we mean an ML model that has a small gap
Revised: By individually fair ML model, we mean an ML model that has a small gap

Original: [Equation]
Revised: [Equation]

Original: The gap is the difference between the optimal value of the investigator’s optimization problem equation 2.1 and the (non-robust) risk.
Revised: The gap is the difference between the optimal value of the auditor’s optimization problem equation 2.1 and the (non-robust) risk.

Original: A small gap implies the investigator cannot signiﬁcantly increase the loss by moving samples from P ∗ to comparable samples.
Revised: A small gap implies the auditor cannot signiﬁcantly increase the loss by moving samples from P ∗ to comparable samples.

Original: Proposition 3.2.
Revised: Proposition 3.2.

Original: Under the assumptions A1–A3, as long as there is ¯ θ ∈ Θ such that
Revised: Under the assumptions A1–A3, as long as there is ¯ θ ∈ Θ such that

Original: [Equation]
Revised: [Equation]

Original: for some δ ∗ > 0 , ˆ θ ∈ arg 
Revised: for some δ ∗ > 0 , ˆ θ ∈ arg min

Original: [Equation]
Revised: [Equation]

Original: satisﬁes
Revised: satisﬁes

Original: [Equation]
Revised: [Equation]

Original: where δ n is the uniform convergence error equation 3.1.
Revised: where δ n is the uniform convergence error equation 3.1.

Original: Proposition 3.2 guarantees Algorithm 2 trains an individually fair ML model.
Revised: Proposition 3.2 guarantees Algorithm 2 trains an individually fair ML model.

Original: More precisely, if there are models in H that are (i) individually fair and (ii) achieve small test error, then Algorithm 2 trains such a model.
Revised: More precisely, if there are models in H that are (i) individually fair and (ii) achieve small test error, then Algorithm 2 trains such a model.

Original: It is possible to replace equation 3.4 with other conditions, but a condition to its effect cannot be dispensed with entirely.
Revised: It is possible to replace equation 3.4 with other conditions, but a condition to its effect cannot be dispensed with entirely.

Original: If there are no individually fair models in H , then it is not possible for equation 2.7 to learn an individually fair model.
Revised: If there are no individually fair models in H , then it is not possible for equation 2.5 to learn an individually fair model.

Original: If there are individually fair models in H , but they all perform poorly, then the goal of learning an individually fair model is futile.
Revised: If there are individually fair models in H , but they all perform poorly, then the goal of learning an individually fair model is futile.

Original: Another consequence of uniform convergence is equation 3.3 is close to its empirical counterpart[Equation]
Revised: Another consequence of uniform convergence is equation 3.3 is close to its empirical counterpart

Original: [Equation]
Revised: [Equation]

Original: In other words, the gap generalizes .
Revised: In other words, the gap generalizes .

Original: This implies equation 3.5 is a certiﬁcate of individual fairness ; i.e. it is possible for practitioners to check whether an ML model is individually fair by evaluating equation 3.5.
Revised: This implies equation 3.5 is a certiﬁcate of individual fairness ; i.e. it is possible for practitioners to check whether an ML model is individually fair by evaluating equation 3.5.

Original: Proposition 3.3.
Revised: Proposition 3.3.

Original: Under the assumptions A1–A3, for any (cid:15) > 0 ,
Revised: Under the assumptions A1–A3, for any (cid:15) > 0 ,

Original: [Equation]
Revised: [Equation]

Original: ≤

Original: [Equation]
Revised: [Table]

Original:  n w.p. at least 1 − t.
Revised: ≤ 2 δ n w.p. at least 1 − t.

Original:  C OMPUTATIONAL RESULTS
Revised:  C OMPUTATIONAL RESULTS

Original: In this section, we present results from using SenSR to train individually fair ML models for two tasks: sentiment analysis and income prediction.
Revised: In this section, we present results from using SenSR to train individually fair ML models for two tasks: sentiment analysis and income prediction.

Original: We pick these two tasks to demonstrate the efﬁcacy of SenSR on problems with structured (income prediction) and unstructured (sentiment analysis) inputs and in which the sensitive attribute (income prediction) is observed and unobserved (sentiment analysis).
Revised: We pick these two tasks to demonstrate the efﬁcacy of SenSR on problems with structured (income prediction) and unstructured (sentiment analysis) inputs and in which the sensitive attribute (income prediction) is observed and unobserved (sentiment analysis).

Original: We refer to Appendix C and D for the implementation details.
Revised: We refer to Appendix C and D for the implementation details.

Original: 4. F AIR SENTIMENT PREDICTION WITH WORD EMBEDDINGS

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Revised: 4. F AIR SENTIMENT PREDICTION WITH WORD EMBEDDINGS

Original: Problem formulation We study the problem of classifying the sentiment of words using positive (e.g. ‘smart’) and negative (e.g. ‘anxiety’) words compiled by Hu & Liu (2004).
Revised: Problem formulation We study the problem of classifying the sentiment of words using positive (e.g. ‘smart’) and negative (e [Table] g. ‘anxiety’) words compiled by Hu & Liu (2004).

Original: We embed words using 300-dimensional GloVe (Pennington et al., 2014) and train a one layer neural network with 1000 hidden units.
Revised: We embed words using 300-dimensional GloVe (Pennington et al., 2014) and train a one layer neural network with 1000 hidden units.

Original: Such classiﬁer achieves 95% test accuracy, however it entails major individual fairness violation.
Revised: Such classiﬁer achieves 95% test accuracy, however it entails major individual fairness violation.

Original: Consider an application of this sentiment classiﬁer to summarizing customer reviews, tweets or news articles.
Revised: Consider an application of this sentiment classiﬁer to summarizing customer reviews, tweets or news articles.

Original: Human names are typical in such texts and should not affect the sentiment score, hence we consider fair metric between any pair of names to be 0.
Revised: Human names are typical in such texts and should not affect the sentiment score, hence we consider fair metric between any pair of names to be 0.

Original: Then sentiment score for all names should be the same to satisfy the individual fairness.
Revised: Then sentiment score for all names should be the same to satisfy the individual fairness.

Original: To make  connection to group fairness, following the study of Caliskan et al.
Revised: To make a connection to group fairness, following the study of Caliskan et al.

Original: (2017) that reveals the biases in word embeddings, we evaluate the fairness of our sentiment classiﬁer using male and female names typical for Caucasian and African-American ethnic groups.
Revised: (2017) that reveals the biases in word embeddings, we evaluate the fairness of our sentiment classiﬁer using male and female names typical for Caucasian and African-American ethnic groups.

Original: We emphasize that to satisfy individual fairness,  sentiment of any name should be the same.
Revised: We emphasize that to satisfy individual fairness, the sentiment of any name should be the same.

Original: Comparison metrics To evaluate the gap between two groups of names, N 0 for Caucasian (or female) and N 1 for African-American (or male), we report
Revised: Comparison metrics To evaluate the gap between two groups of names, N 0 for Caucasian (or female) and N 1 for African-American (or male), we report

Original: [Equation]
Revised: [Equation]

Original: − 1 |N 1 | (cid:80) n ∈N 1 ( h ( n ) 1 − h ( n ) 0 ) , where h ( n ) k is logits for class k of name n ( k = 1 is the positive class).
Revised: − 1 |N 1 | (cid:80) n ∈N 1 ( h ( n ) 1 − h ( n ) 0 ) , where h ( n ) k is logits for class k of name n ( k = 1 is the positive class).

Original: We use list of names provided in Caliskan et al.
Revised: We use list of names provided in Caliskan et al.

Original: (2017), which consists of 49 Caucasian and 45 African-American names, among those 48 are female and 46 are male.
Revised: (2017), which consists of 49 Caucasian and 45 African-American names, among those 48 are female and 46 are male.

Original: The gap between
Revised: The gap between

Original: African-American and Caucasian names is reported as Race gap, while the gap between male and female names is reported as Gend.
Revised: African-American and Caucasian names is reported as Race gap, while the gap between male and female names is reported as Gend.

Original: gap in Table 1.
Revised: gap in Table 1.

Original: As in Speer (2017) we also compare sentiment difference of two sentences: “Let’s go get Italian food” and “Let’s go get Mexican food”, i.e. cuisine gap (abbreviated Cuis.
Revised: As in Speer (2017), we also compare sentiment difference of two sentences: “Let’s go get Italian food” and “Let’s go get Mexican food”, i.e. cuisine gap (abbreviated Cuis.

Original: gap in Table 1), as a test of generalization beyond names.
Revised: gap in Table 1), as a test of generalization beyond names.

Original: To embed  sentences we average their word embeddings.
Revised: To embed these sentences we average their word embeddings.

Original: Sensitive subspace We consider  94 names that we use for evaluation as sensitive directions, which may be regarded as utilizing the expert knowledge, i.e. these names form a list of words that an expert believes should be treated equally.
Revised: Sensitive subspace We consider embeddings of 94 names that we use for evaluation as sensitive directions, which may be regarded as utilizing the expert knowledge, i.e. these names form a list of words that an expert believes should be treated equally.

Original: Following procedure described in Appendix B.2 and Algorithm 3, embeddings ofthese names deﬁne sensitive subspace inducing the fair metric.
Revised: Fair metric is then deﬁned using an orthogonal complement projector of the span of sensitive directions as we discussed in Section 2.1.

Original: When expert knowledge is not available, or we wish to achieve general fairness for names, we utilize a side dataset of popular baby names in New York City.
Revised: When expert knowledge is not available, or we wish to achieve general fairness for names, we utilize a side dataset of popular baby names in New York City.

Original: The dataset has 11k names, however onlyoverlap with the list of names used for evaluation.
Revised: The dataset has 11k names, however only 32 overlap with the list of names used for evaluation.

Original: Embeddings of these names deﬁne asingle group of comparable samples, hence factor analysis in Algorithm 3 reduces to SVD.
Revised: Embeddings of these names deﬁne a group of comparable samples that we use to learn sensitive directions with SVD (see Appendix B.2 and Algorithm 3 for details).

Original: We use top 50 singular vectors to form the sensitive subspace.
Revised: We take top 50 singular vectors to form the sensitive subspace.

Original: It is worth noting that, unlike many existing approaches in[Table] the fairness literature, we do not use any protected attribute information.
Revised: It is worth noting that, unlike many existing approaches in the fairness literature, we do not use any protected attribute information.

Original: Our algorithm only utilizes training words, their sentiments and a vanilla list of names.
Revised: Our algorithm only utilizes training words, their sentiments and a vanilla list of names.

Original: Results From  box-plots in Figure 2 we see that both race and gender gaps are signiﬁcant when using  baseline neural network classiﬁer.
Revised: Results From the box-plots in Figure 2, we see that both race and gender gaps are signiﬁcant when using the baseline neural network classiﬁer.

Original: It tends to predict Caucasian names as “positive”, while the median for African-American names is negative; the median sentiment for female names is higher than that for male names.
Revised: It tends to predict Caucasian names as “positive”, while the median for African-American names is negative; the median sentiment for female names is higher than that for male names.

Original: We considered three other approaches to this problem:  algorithm of Bolukbasi et al.
Revised: We considered three other approaches to this problem: the algorithm of Bolukbasi et al.

Original: (2016) for pre-processing word embeddings; pre-processing via projecting  out  sensitive subspace that we used for training SenSR (this is analogous to Prost et al.
Revised: (2016) for pre-processing word embeddings; pre-processing via projecting [Table] out the sensitive subspace that we used for training SenSR (this is analogous to Prost et al.

Original: ; training a distributionally robust classiﬁer with Euclidean distance cost (Sinha et al., 2017).
Revised: ; training a distributionally robust classiﬁer with Euclidean distance cost (Sinha et al., 2017).

Original: All approaches improved upon the baseline, however only SenSR can be considered individually fair.
Revised: All approaches improved upon the baseline, however only SenSR can be considered individually fair.

Original: Our algorithm practically eliminates gender and racial gaps and achieves the notion of individual fairness as can be seen from almost equal predicted sentiment score for all names.
Revised: Our algorithm practically eliminates gender and racial gaps and achieves the notion of individual fairness as can be seen from almost equal predicted sentiment score for all names.

Original: We remark that using expert knowledge (i.e. evaluation names) allowed SenSR-E (E for expert) to further improve both group and individual fairness.
Revised: We remark that using expert knowledge (i.e. evaluation names) allowed SenSR-E (E for expert) to further improve both group and individual fairness.

Original: However we warn practitioners that if the expert knowledge is too speciﬁc, generalization outside of the expert knowledge may not be very good.
Revised: However we warn practitioners that if the expert knowledge is too speciﬁc, generalization outside of the expert knowledge may not be very good.

Original: In Table 1 we report results averaged across 10 repetitions with 90%/10% train/test splits, where we also verify that accuracy trade-off with the baseline is minor.
Revised: In Table 1 we report results averaged across 10 repetitions with 90%/10% train/test splits, where we also verify that accuracy trade-off with the baseline is minor.

Original: In the right column we present the generalization check, i.e. comparing a pair of sentences unrelated to names.
Revised: In the right column we present the generalization check, i.e. comparing a pair of sentences unrelated to names.

Original: Utilizing expert knowledge led to a fairness overﬁtting effect, however we still see improvement over other methods.
Revised: Utilizing expert knowledge led to a fairness over-ﬁtting effect, however we still see improvement over other methods.

Original: When utilizing SVD of a larger dataset of names we observe better generalization.
Revised: When utilizing SVD of a larger dataset of names we observe better generalization.

Original: Our generalization check suggests that fairness over-ﬁtting is possible, therefore datasets and procedure for verifying fairness generalization are needed.
Revised: Our generalization check suggests that fairness over-ﬁtting is possible, therefore datasets and procedure for verifying fairness generalization are needed.

Original: 4. A DULT
Revised: 4. A DULT

Original: Problem formulation Demonstrating the broad applicability of SenSR outside of natural language processing tasks, we apply SenSR to a classiﬁcation task on the Adult (Dua & Graff, 2017) data set to predict whether an individual makes at least $50k based on features like gender and occupation for approximately 45,000 individuals.
Revised: Problem formulation Demonstrating the broad applicability of SenSR outside of natural language processing tasks, we apply SenSR to a classiﬁcation task on the Adult (Dua & Graff, 2017) data set to predict whether an individual makes at least $50k based on features like gender and occupation for approximately 45,000 individuals.

Original: Models that predict income without fairness considerations can contribute to the problem of differences in pay between genders or races for the same work.
Revised: Models that predict income without fairness considerations can contribute to the problem of differences in pay between genders or races for the same work.

Original: Throughout this section, gender (male or female) and race (Caucasian or non-Caucasian) are binary.
Revised: Throughout this section, gender (male or female) and race (Caucasian or non-Caucasian) are binary.

Original: Comparison metrics Arguably a classiﬁer is individually unfair if the classiﬁcations for two data points that are the same on all features except demographic features are different.
Revised: Comparison metrics Arguably a classiﬁer is individually unfair if the classiﬁcations for two data points that are the same on all features except demographic features are different.

Original: Therefore, to assess individual fairness, we report spouse consistency (S-Con.) and gender and race consistency (GR-Con.), which are measures of how often classiﬁcations change only because of differences in demographic features.
Revised: Therefore, to assess individual fairness, we report spouse consistency (S-Con.) and gender and race consistency (GR-Con.), which are measures of how often classiﬁcations change only because of differences in demographic features.

Original: For S-Con (resp.
Revised: For S-Con (resp.

Original: GR-con), we make 2 (resp. 4) copies of every data point where the only difference is that one is a husband and the other is a wife (resp.
Revised: GR-con), we make 2 (resp. 4) copies of every data point where the only difference is that one is a husband and the other is a wife (resp.

Original: difference is in gender and race).
Revised: difference is in gender and race).

Original: S-Con (resp.
Revised: S-Con (resp.

Original: GR-Con) is the fraction of corresponding pairs (resp.
Revised: GR-Con) is the fraction of corresponding pairs (resp.

Original: quadruples) that have the same classiﬁcation.
Revised: quadruples) that have the same classiﬁcation.

Original: We also report various group fairness measures proposed by De-Arteaga et al.
Revised: We also report various group fairness measures proposed by De-Arteaga et al.

Original: (2019) with respect to race or gender based on true positive rates, i.e. the ability of a classiﬁer to correctly identify a given class.
Revised: (2019) with respect to race or gender based on true positive rates, i.e. the ability of a classiﬁer to correctly identify a given class.

Original: See the Supplement for the deﬁnitions.
Revised: See Appendix D.5 for the deﬁnitions.

Original: We report Gap RMS R , 
Revised: We report Gap RMS R , Gap RMS G ,

Original: GapRMS G , Gap max R , and Gap max G where R refers to race, and G refers to gender.
Revised: Gap max R , and Gap max G where R refers to race, and G refers to gender.

Original: We use balanced accuracy (B-acc) instead of accuracy 2 to measure predictive ability since only 25% of individuals make at least $50k.
Revised: We use balanced accuracy (Bacc) instead of accuracy 2 to measure predictive ability since only 25% of individuals make at least $50k.

Original: Sensitive subspace Let { ( x i , x g i ) } mi =1 be the set of features x i ∈ R D of the data except the coordinate for gender is zeroed and where x g i indicates the gender of individual i .
Revised: Sensitive subspace Let { ( x i , x g i ) } mi =1 be the set of features x i ∈ R D of the data except the coordinate for gender is zeroed and where x g i indicates the gender of individual i .

Original: For γ > 0 , let w g = arg min  R D  (cid:80) mi 
Revised: For γ > 0 , let w g = arg min w ∈ R D 1 m (cid:80) mi =1 − x

Original: [-=1 log(
-]
Revised: [-=1 log(
-]g

Original: + e − x gi ( w Tgi x i )  ) + γ (cid:107) w (cid:107) 2 , i.e. w g is the learned hyperplane that classiﬁes gender given by  logistic regression.
Revised: i ( w T x i ) + log(1 + e w T x i ) + γ (cid:107) w (cid:107) 2 , i.e. w g is the learned hyperplane that classiﬁes gender given by regularized logistic regression.

Original: Let e g ∈ R D (resp.
Revised: Let e g ∈ R D (resp.

Original: e r ) be the vector that is 1 in the gender (resp.
Revised: e r ) be the vector that is 1 in the gender (resp.

Original: race) coordinate and 0 elsewhere.
Revised: race) coordinate and 0 elsewhere.

Original: We use the approach from Appendix B.1 to learn the fair metric. In particular, the sensitive subspace is spanned by A =
Revised: Then the sensitive subspace is the span of [ w g , e g , e r ]. See Appendix B.1 for details.

Original: [ w g , e g , e r ] and the fair metric is given by equation B.3.
Revised: 2 Accuracy is reported in Table 4 in Appendix D.

Original: Results See Table 2 for the average 3 of each metric on the test sets over ten 80%/20% train/test splits for Baseline, Project (projecting features onto the orthogonal complement of the sensitive subspace before training), CoCL (De-Arteaga et al., 2019), Adversarial Debiasing (Zhang et al., 2018), and SenSR.
Revised: Results See Table 2 for the average 3 of each metric on the test sets over ten 80%/20% train/test splits for Baseline, Project (projecting features onto the orthogonal complement of the sensitive subspace before training), CoCL (De-Arteaga et al., 2019), Adversarial Debiasing (Zhang et al., 2018), and SenSR.

Original: With the exception of CoCL (De-Arteaga et al., 2019), each classiﬁer is aunit single hidden layer neural network.
Revised: With the exception of CoCL (De-Arteaga et al., 2019), each classiﬁer is aunit single hidden layer neural network.

Original: The Baseline clearly exhibits individual and group fairness violations.
Revised: The Baseline clearly exhibits individual and group fairness violations.

Original: While SenSR has the lowest B-acc, SenSR is the best by a large margin for S-Con.
Revised: While SenSR has the lowest B-acc, SenSR is the best by a large margin for S-Con.

Original: and has the best group fairness measures.
Revised: and has the best group fairness measures.

Original: We expect SenSR to do well on GR-consistency since the sensitive subspace includes the race and gender directions.
Revised: We expect SenSR to do well on GR-consistency since the sensitive subspace includes the race and gender directions.

Original: However, SenSR’s individually fair performance generalizes: the sensitive directions do not directly use the husband and wife directions, yet SenSR performs well on S-Con.
Revised: However, SenSR’s individually fair performance generalizes: the sensitive directions do not directly use the husband and wife directions, yet SenSR performs well on S-Con.

Original: Furthermore, SenSR outperforms Project on S-Con and group fairness measures illustrating that SenSR does much more than just ignoring the sensitive subspace.
Revised: Furthermore, SenSR outperforms Project on S-Con and group fairness measures illustrating that SenSR does much more than just ignoring the sensitive subspace.

Original: CoCL only barely improves group fairness compared to the baseline with a signiﬁcant drop in Bacc and while Adversarial Debiasing also improves group fairness, it is worse than the baseline on individual fairness measures illustrating that group fairness does not imply individual fairness.
Revised: CoCL only barely improves group fairness compared to the baseline with a signiﬁcant drop in Bacc and while Adversarial Debiasing also improves group fairness, it is worse than the baseline on individual fairness measures illustrating that group fairness does not imply individual fairness.

Original:  S UMMARY
Revised:  S UMMARY

Original: We consider the task of training ML systems that are fair in the sense that their performance is invariant under certain perturbations in a sensitive subspace.
Revised: We consider the task of training ML systems that are fair in the sense that their performance is invariant under certain perturbations in a sensitive subspace.

Original: This notion of fairness is closely related to the notion of individual fairness proposed by Dwork et al.
Revised: This notion of fairness is a variant of individual fairness (Dwork et al., 2011).

Original: One of the main barriers to the adoption of individual fairness is the lack of consensus on the fair metric in the formal deﬁnition of individual fairness.
Revised: One of the main barriers to the adoption of individual fairness is the lack of consensus on a fair metric for many ML tasks.

Original: To circumvent this issue, we consider two approaches to learning a fair metric from data: one for problems in which the sensitive attribute is observed, and another for problems in which the sensitive attribute is unobserved.
Revised: To circumvent this issue, we consider two approaches to learning a fair metric from data: one for problems in which the sensitive attribute is observed, and another for problems in which the sensitive attribute is unobserved.

Original: Given a data-driven choice of fair metric, we provide an algorithm that provably trains individually fair ML models
Revised: Given a data-driven choice of fair metric, we provide an algorithm that provably trains individually fair ML models. A CKNOWLEDGMENTS

Revised: This work was supported by the National Science Foundation under grants DMS-1830247 and

Original: A P ROOFS A. P ROOF OF P ROPOSITION 3. 
Revised: A P ROOFS A. P ROOF OF P ROPOSITION 3.

Original: By the duality result of Blanchet & Murthy (2016), for any (cid:15) > 0 ,
Revised: By the duality result of Blanchet & Murthy (2016), for any (cid:15) > 0 ,

Original: [Equation]
Revised: [Equation]

Revised: (cid:2) (cid:96) cλ

Original:  where λ n ∈ arg min λ ≥ 0 λ(cid:15) + E P n (cid:2) (cid:96) cλ ( Z, θ ) (cid:3) .
Revised: [Equation] ( Z, θ ) (cid:3) [Equation] where λ n ∈ arg min λ ≥ 0 λ(cid:15) + E P n (cid:2) (cid:96) cλ ( Z, θ ) (cid:3) .

Original: By assumption A3, | (cid:96) c ∗ λ n ( z, θ ) − (cid:96) cλ n ( z, θ ) | = (cid:12)(cid:12)(cid:12)(cid:12) sup x 
Revised: By assumption A3, | (cid:96) c ∗ λ n ( z, θ ) − (cid:96) cλ n ( z, θ ) | = (cid:12)(cid:12)(cid:12)(cid:12) sup x 2 ∈X (cid:96) (( x [Equation] y ) , θ ) − λ n c

Original: [Equation]
Revised: [Equation]

Revised: (( x, y ) , ( x

Revised: [Equation]

Original: y )
Revised: y ))

Revised: − sup

Revised: [Equation]

Revised: [Equation]

Revised: (cid:96) (( x

Original: [Equation]
Revised: [Equation]

Original: θ )
Revised: y )

Original: [Equation]
Revised: [Equation]

Revised: θ )

Original: [Equation]
Revised: [Equation]

Original: x
Revised: n

Original: [Equation]
Revised: [Equation]

Revised: (( x

Original: [Equation]
Revised: [Equation]

Original:  )−
Revised: y )

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation] [Equation] ≤
Revised: )) (cid:12)(cid:12)(cid:12)(cid:12) ≤

Original: [Equation]
Revised: [Equation]

Original: λ
Revised: λ

Original: [Equation]
Revised: [Equation]

Revised: ) −

Original: [Equation]
Revised: [Equation]

Original: This implies
Revised: This implies

Original: [Equation]
Revised: [Equation]

Original: This bound is crude; it is possible to obtain sharper bounds under additional assumptions on the loss and transportation cost functions.
Revised: This bound is crude; it is possible to obtain sharper bounds under additional assumptions on the loss and transportation cost functions.

Original: We avoid this here to keep the result as general as possible.
Revised: We avoid this here to keep the result as general as possible.

Original: Similarly,
Revised: Similarly,

Original: [Equation]
Revised: [Equation]

Original: where λ ∗ ∈ arg min λ ≥ 0 { λ(cid:15) + E P ∗ (cid:2) (cid:96) c ∗ λ ( Z, θ ) (cid:3)(cid:9) .
Revised: where λ ∗ ∈ arg min λ ≥ 0 { λ(cid:15) + E P ∗ (cid:2) (cid:96) c ∗ λ ( Z, θ ) (cid:3)(cid:9) .

Original: Lemma A.1 (Lee & Raginsky (2017)) .
Revised: Lemma A.1 (Lee & Raginsky (2017)) .

Original: Let ˜ λ ∈ arg min λ ≥ 0 λ(cid:15) + E P (cid:2) (cid:96) cλ ( Z, θ ) (cid:3) .
Revised: Let ˜ λ ∈ arg min λ ≥ 0 λ(cid:15) + E P (cid:2) (cid:96) cλ ( Z, θ ) (cid:3) .

Original: As long as the function in the loss class are L -Lipschitz with respect to d x (see Assumption A2), ˜ λ ≤ L √ (cid:15) .
Revised: As long as the function in the loss class are L -Lipschitz with respect to d x (see Assumption A2), ˜ λ ≤ L √ (cid:15) .

Original: Proof.
Revised: Proof.

Original: By the optimality of ˜ λ ,
Revised: By the optimality of ˜ λ ,

Original: [Equation]
Revised: [Equation]

Original: for any λ ≥ 0 .
Revised: for any λ ≥ 0 .

Original: By Assumption A2, the right side is at most
Revised: By Assumption A2, the right side is at most

Original: [Equation]
Revised: [Equation]

Original: We minimize the right side WRT t (set t =
Revised: We minimize the right side WRT t (set t =

Original: L 2 λ ) and λ (set λ
Revised: L 2 λ ) and λ (set λ

Original: =
Revised: =

Original: L 2 √ (cid:15) ) to obtain ˜ λ(cid:15) ≤ L √ (cid:15) .
Revised: L 2 √ (cid:15) ) to obtain ˜ λ(cid:15) ≤ L √ (cid:15) .

Original: By Lemma A.1, we have
Revised: By Lemma A.1, we have

Original: [Equation]
Revised: [Equation]

Original: We combine the preceding bounds to obtain
Revised: We combine the preceding bounds to obtain

Original: [Equation]
Revised: [Equation]

Original: where L c ∗ = { (cid:96) c ∗ λ ( · , θ ) : λ ∈ [0 , L √ (cid:15) ] , θ ∈ Θ } is the DR loss class.
Revised: where L c ∗ = { (cid:96) c ∗ λ ( · , θ ) : λ ∈ [0 , L √ (cid:15) ] , θ ∈ Θ } is the DR loss class.

Original: In the rest of the proof, we bound sup f ∈L c ∗ (cid:12)(cid:12) (cid:82) Z f ( z ) d
Revised: In the rest of the proof, we bound sup f ∈L c ∗ (cid:12)(cid:12) (cid:82) Z f ( z ) d

Original: ( P ∗ − P n )( z ) (cid:12)(cid:12) with standard techniques from statistical learning theory.
Revised: ( P ∗ − P n )( z ) (cid:12)(cid:12) with standard techniques from statistical learning theory.

Original: Assumption A2 implies the functions in F are bounded:
Revised: Assumption A2 implies the functions in F are bounded:

Original: [Equation]
Revised: [Equation]

Original: This implies has bounded differences, so δ n concentrates sharply around its expectation.
Revised: This implies has bounded differences, so δ n concentrates sharply around its expectation.

Original: By the bounded-differences inequality and a symmetrization argument,
Revised: By the bounded-differences inequality and a symmetrization argument,

Original: [Equation]
Revised: [Equation]

Original: WP at least 1 − t , where R n ( F ) is the Rademacher complexity of F :
Revised: WP at least 1 − t , where R n ( F ) is the Rademacher complexity of F :

Original: [Equation]
Revised: [Equation]

Original: Lemma A.2.
Revised: Lemma A.2.

Original: The Rademacher complexity of the DR loss class is at most
Revised: The Rademacher complexity of the DR loss class is at most

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: To study the Rademacher complexity of L c , we ﬁrst show that the L c -indexed Rademacher process X f (cid:44) 1 n (cid:80) ni =1 σ i f
Revised: To study the Rademacher complexity of L c , we ﬁrst show that the L c -indexed Rademacher process X [Equation] =1 σ i f

Original: ( Z i ) is sub-Gaussian WRT to a pseudometric.
Revised: ( Z i ) is sub-Gaussian WRT to a pseudometric.

Original: Let f 1 = (cid:96) cλ 1 ( · , θ 1 ) and f 2 = (cid:96) cλ 2 ( · , θ 2 ) .
Revised: Let f 1 = (cid:96) cλ 1 ( · , θ 1 ) and f 2 = (cid:96) cλ 2 ( · , θ 2 ) .

Original: Deﬁne
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: We check that X f is sub-
Revised: We check that X f is sub-

Original: Gaussian WRT d L c :E
Revised: Gaussian WRT d L c :

Original: [Equation]

Original: X f

Original: [Equation]

Original: X f

Original: [Equation]

Original: = E (cid:2)

Original: [Equation]

Original: (cid:0) t n n

Original: [Equation]

Original: =

Original: [Equation]

Original: =

Original: [Equation]

Original: λ

Original: [Equation]

Original: n =

Original: [Equation]

Original: n

Original: [Equation]
Revised: [Equation]

Original: sup

Original: [Equation]

Original: n ≤

Original: [Equation]

Original: (cid:0)

Original: [Equation]

Original: ( f 1 , f 2 ) (cid:1) .
Revised: 1 , f 2 ) (cid:1) .

Original: Let N ( L c , d L c , (cid:15) ) be the (cid:15) -covering number of ( L c , d L c ) .
Revised: Let N ( L c , d L c , (cid:15) ) be the (cid:15) -covering number of ( L c , d L c ) .

Original: We observe
Revised: We observe

Original: [Equation]
Revised: [Equation]

Original: By Dudley’s entropy integral,
Revised: By Dudley’s entropy integral,

Original: [Equation]
Revised: [Equation]

Original: where we recalled equation A.1 in the second step.
Revised: where we recalled equation A.1 in the second step.

Original: We evalaute the integral on the right side to arrive at the stated bound: (cid:82) 12 0 log( 1 (cid:15) ) d(cid:15) < 1 .
Revised: We evalaute the integral on the right side to arrive at the stated bound: (cid:82) 12 0 log( 1 (cid:15) ) d(cid:15) < 1 .

Original: By Lemma A.2,
Revised: By Lemma A.2,

Original: [Equation]
Revised: [Equation]

Original: which implies
Revised: which implies

Original: [Equation]
Revised: [Equation]

Original: WP at least 1 − t .
Revised: WP at least 1 − t .

Original: A. P ROOFS OF P ROPOSITIONS 3.2 AND 3. 
Revised: A. P ROOFS OF P ROPOSITIONS 3.2 AND 3.

Original: Proof of Proposition 3.2.
Revised: Proof of Proposition 3.2.

Original: It is enough to show
Revised: It is enough to show

Original: [Equation]
Revised: [Equation]

Original: because the loss function is non-negative.
Revised: because the loss function is non-negative.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Proof of Proposition 3.3.
Revised: Proof of Proposition 3.3.

Original: [Equation]
Revised: [Equation]

Original: The loss function is bounded, so it is possible to bound E P ∗
Revised: The loss function is bounded, so it is possible to bound E P ∗

Original: [Equation]
Revised: [Equation]

Original: Z
Revised: Z

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: by standard uniform convergence results on bounded loss classes.
Revised: by standard uniform convergence results on bounded loss classes.

Original: B D ATA - DRIVEN FAIR METRICS B. L EARNING THE METRIC FROM OBSERVATIONS OF THE SENSITIVE ATTRIBUTE
Revised: B D ATA - DRIVEN FAIR METRICS B. L EARNING THE METRIC FROM OBSERVATIONS OF THE SENSITIVE ATTRIBUTE

Original: Here we assume the sensitive attribute is discrete and is observed for a small subset of the training data.
Revised: Here we assume the sensitive attribute is discrete and is observed for a small subset of the training data.

Original: Formally, we assume this subset of the training data has the form { ( X i , K i , Y i ) } , where K i is the sensitive attribute of the i -th subject.
Revised: Formally, we assume this subset of the training data has the form { ( X i , K i , Y i ) } , where K i is the sensitive attribute of the i -th subject.

Original: To learn the sensitive subspace, we ﬁt a softmax regression model to the data
Revised: To learn the sensitive subspace, we ﬁt a softmax regression model to the data

Original: [Equation]
Revised: [Equation]

Original: and take the span of A
Revised: and take the span of A

Original: = [ a 1 . . .
Revised: = [ a 1 . . .

Original: a k ] as the sensitive subspace to deﬁne the fair metric as
Revised: a k ] as the sensitive subspace to deﬁne the fair metric as

Original: [Equation]
Revised: [Equation]

Original: This approach readily generalizes to sensitive attributes that are not discrete-valued: replace the softmax model by an appropriate generalized linear model.
Revised: This approach readily generalizes to sensitive attributes that are not discrete-valued: replace the softmax model by an appropriate generalized linear model.

Original: In many applications, the sensitive attribute is part of a user’s demographic information, so it may not be available due to privacy restrictions.
Revised: In many applications, the sensitive attribute is part of a user’s demographic information, so it may not be available due to privacy restrictions.

Original: This does not preclude the proposed approach because the sensitive attribute is only needed to learn the fair metric and is neither needed to train the classiﬁer nor at test time.
Revised: This does not preclude the proposed approach because the sensitive attribute is only needed to learn the fair metric and is neither needed to train the classiﬁer nor at test time.

Original: B. L EARNING THE FAIR METRIC FROM HUMAN SUPERVISION
Revised: B. L EARNING THE FAIR METRIC FROM HUMAN SUPERVISION

Original: The main barrier to wider adoption of individual fairness is disagreement over the fair metric.
Revised: The main barrier to wider adoption of individual fairness is disagreement over the fair metric.

Original: In this section, we consider the task of learning a fair metric from human supervision.
Revised: In this section, we consider the task of learning a fair metric from human supervision.

Original: To keep things simple, we focus on learning a generalized Mahalanobis distance
Revised: To keep things simple, we focus on learning a generalized Mahalanobis distance

Original: [Equation]
Revised: [Equation]

Original: where ϕ ( x ) :
Revised: where ϕ ( x ) :

Original: X → R d is a known feature map and (cid:98) Σ ∈ S
Revised: X → R d is a known feature map and (cid:98) Σ ∈ S

Original: d × d + is a covariance matrix, from groups of comparable samples.
Revised: d × d + is a covariance matrix, from groups of comparable samples.

Original: This type of feedback is common in the literature on debiasing learned representations.
Revised: This type of feedback is common in the literature on debiasing learned representations.

Original: For example, method of Bolukbasi et al.
Revised: For example, method of Bolukbasi et al.

Original: (2016) for removing gender bias in word embeddings relies on sets of words whose embeddings mainly vary in a gender subspace ( e.g. (king, queen)).
Revised: (2016) for removing gender bias in word embeddings relies on sets of words whose embeddings mainly vary in a gender subspace ( e.g. (king, queen)).

Original: Our approach is based on a factor model
Revised: Our approach is based on a factor model

Original: [Equation]
Revised: [Equation]

Original: where ϕ
Revised: where ϕ

Original: i ∈ R
Revised: i ∈ R

Original: d is the learned representation of x i , u
Revised: d is the learned representation of x i , u

Original: i ∈ R K (resp.
Revised: i ∈ R K (resp.

Original: v i ∈ R L ) is the sensitive/irrelevant (resp. relevant) attributes of x i to the task at hand, and (cid:15) i is an error term.
Revised: v i ∈ R L ) is the sensitive/irrelevant (resp. relevant) attributes of x i to the task at hand, and (cid:15) i is an error term.

Original: For example, in Bolukbasi et al.
Revised: For example, in Bolukbasi et al.

Original: , the learned representations are the embeddings of words in the vocabulary, and the sensitive attribute is the gender bias of the words.
Revised: , the learned representations are the embeddings of words in the vocabulary, and the sensitive attribute is the gender bias of the words.

Original: The sensitive and relevant attributes are generally unobserved.
Revised: The sensitive and relevant attributes are generally unobserved.

Original: Recall our goal is to obtain (cid:98) Σ so that equation B.2 is small whenever v 1 ≈ v 2 .
Revised: Recall our goal is to obtain (cid:98) Σ so that equation B.2 is small whenever v 1 ≈ v 2 .

Original: One possible choice of (cid:98) Σ is the projection matrix onto the orthogonal complement of ran ( A ) , which we denote by P ran ( A ) .
Revised: One possible choice of (cid:98) Σ is the projection matrix onto the orthogonal complement of ran ( A ) , which we denote by P ran ( A ) .

Original: Indeed,
Revised: Indeed,

Original: [Equation]
Revised: [Equation]

Original: which is small whenever v 1 ≈ v 2 .
Revised: which is small whenever v 1 ≈ v 2 .

Original: Although ran ( A ) is unknown, it is possible to estimate it from the learned representations and groups of comparable samples by factor analysis.
Revised: Although ran ( A ) is unknown, it is possible to estimate it from the learned representations and groups of comparable samples by factor analysis.

Original: The factor model attributes variation in the learned representations to variation in the sensitive and relevant attributes.
Revised: The factor model attributes variation in the learned representations to variation in the sensitive and relevant attributes.

Original: We consider two samples comparable if their relevant attributes are similar.
Revised: We consider two samples comparable if their relevant attributes are similar.

Original: In other words, if I ⊂ [ n ] is (the indices of) a group of comparable samples, then
Revised: In other words, if I ⊂ [ n ] is (the indices of) a group of comparable samples, then

Original: [Equation]
Revised: [Equation]

Original: B.
Revised: B.

Original: [Equation]
Revised: [Equation]

Original: where H =
Revised: where H =

Original: I |I| − 1 |I| 1 |I| 1 T |I| is the centering or de-meaning matrix and the rows of Φ I (resp.
Revised: I |I| − 1 |I| 1 |I| 1 T |I| is the centering or de-meaning matrix and the rows of Φ I (resp.

Original: U I ,
Revised: U I ,

Original: V I ) are ϕ i (resp.
Revised: V I ) are ϕ i (resp.

Original: u i , v i ).
Revised: u i , v i ).

Original: If this group of samples have identical relevant attributes, i.e. V I = 1 |I| v T for some v , then HV I vanishes exactly.
Revised: If this group of samples have identical relevant attributes, i.e. V I = 1 |I| v T for some v , then HV I vanishes exactly.

Original: As long as u i and (cid:15) i are uncorrelated ( e.g. E (cid:2) u i (cid:15) Ti (cid:3) = 0 ), equation B.5 implies
Revised: As long as u i and (cid:15) i are uncorrelated ( e.g. E (cid:2) u i (cid:15) Ti (cid:3) = 0 ), equation B.5 implies

Original: [Equation]
Revised: [Equation]

Original: This suggests estimating ran ( A ) from the learned representations and groups of comparable samples by factor analysis.
Revised: This suggests estimating ran ( A ) from the learned representations and groups of comparable samples by factor analysis.

Original: We summarize our approach in Algorithm 3.
Revised: We summarize our approach in Algorithm 3.

Original: Algorithm 3 estimating (cid:98) Σ for the fair metric
Revised: Algorithm 3 estimating (cid:98) Σ for the fair metric

Original: [Equation] factor analysis
Revised: (cid:46) analysis

Original: C S EN SR IMPLEMENTATION DETAILS
Revised: C S EN SR IMPLEMENTATION DETAILS

Original: This section is to accompany the implementation of the SenSR algorithm and is best understood by reading it along with the code implemented using TensorFlow.
Revised: This section is to accompany the implementation of the SenSR algorithm and is best understood by reading it along with the code implemented using TensorFlow.

Original: We discuss choices of learning rates and few speciﬁcs of the code.
Revised: We discuss choices of learning rates and few speciﬁcs of the code.

Original: Words in italics correspond to variables in the code and following notation in parentheses deﬁnes corresponding name in Table 3, where we summarize all hyperparameter choices.
Revised: Words in italics correspond to variables in the code and following notation in parentheses deﬁnes corresponding name in Table 3, where we summarize all hyperparameter choices.

Original: Handling class imbalance Datasets we study have imbalanced classes.
Revised: Handling class imbalance Datasets we study have imbalanced classes.

Original: To handle it, on every epoch ( E ) (i.e. number of epochs) we subsample a batch size ( B ) training samples enforcing equal number of observations per class.
Revised: To handle it, on every epoch ( E ) (i.e. number of epochs) we subsample a batch size ( B ) training samples enforcing equal number of observations per class.

Original: This procedure can be understood as data augmentation.
Revised: This procedure can be understood as data augmentation.

Original: Perturbations speciﬁcs Our implementation of SenSR algorithm has two inner optimization problems — subspace perturbation and full perturbation (when (cid:15) > 0 ).
Revised: Perturbations speciﬁcs Our implementation of SenSR algorithm has two inner optimization problems — subspace perturbation and full perturbation (when (cid:15) > 0 ).

Original: Subspace perturbation can be viewed as an initialization procedure for the attack.
Revised: Subspace perturbation can be viewed as an initialization procedure for the attack.

Original: We implement both using Adam optimizer (Kingma & Ba, 2014) inside the computation graph for better efﬁciency, i.e. deﬁning corresponding perturbation parameters as Variables and re-setting them to zeros after every epoch.
Revised: We implement both using Adam optimizer (Kingma & Ba, 2014) inside the computation graph for better efﬁciency, i.e. deﬁning corresponding perturbation parameters as Variables and re-setting them to zeros after every epoch.

Original: This is in contrast with a more common strategy in the adversarial robustness implementations, where perturbations (i.e. attacks) are implemented using tf.gradients with respect to the input data deﬁned as a
Revised: This is in contrast with a more common strategy in the adversarial robustness implementations, where perturbations (i.e. attacks) are implemented using tf.gradients with respect to the input data deﬁned as a

Original: Placeholder.
Revised: Placeholder.

Original: Learning rates As mentioned above, in addition to regular Adam optimizer for learning the parameters we invoke two more for the inner optimization problems of SenSR.
Revised: Learning rates As mentioned above, in addition to regular Adam optimizer for learning the parameters we invoke two more for the inner optimization problems of SenSR.

Original: We use same learning rate of 0.001 for the parameters optimizer, however different learning rates across datasets for subspace step ( s ) and full step ( f ).
Revised: We use same learning rate of 0.001 for the parameters optimizer, however different learning rates across datasets for subspace step ( s ) and full step ( f ).

Original: Two other related parameters are number of steps of the inner optimizations: subspace epoch ( se ) and full epoch ( fe ).
Revised: Two other related parameters are number of steps of the inner optimizations: subspace epoch ( se ) and full epoch ( fe ).

Original: We observed that setting subspace perturbation learning rate too small may prevent our algorithm from reducing unfairness, however setting it big does not seem to hurt.
Revised: We observed that setting subspace perturbation learning rate too small may prevent our algorithm from reducing unfairness, however setting it big does not seem to hurt.

Original: On the other hand, learning rate for full perturbation should not be set too big as it may prevent algorithm from solving the original task.
Revised: On the other hand, learning rate for full perturbation should not be set too big as it may prevent algorithm from solving the original task.

Original: Note that full perturbation learning rate should be smaller than perturbation budget eps ( (cid:15) ) — we always use (cid:15)/ 10 .
Revised: Note that full perturbation learning rate should be smaller than perturbation budget eps ( (cid:15) ) — we always use (cid:15)/ 10 .

Original: In general, malfunctioning behaviors are immediately noticeable during training and can be easily corrected, therefore we did not need to use any hyperparameter optimization tools.
Revised: In general, malfunctioning behaviors are immediately noticeable during training and can be easily corrected, therefore we did not need to use any hyperparameter optimization tools.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: D A DDITIONAL A DULT EXPERIMENT DETAILS D. P REPROCESSING
Revised: D A DDITIONAL A DULT EXPERIMENT DETAILS D. P REPROCESSING

Original: The continuous features in Adult are the following: age , fnlwgt , capital-gain , capital-loss , hours-per-week , and education-num .
Revised: The continuous features in Adult are the following: age , fnlwgt , capital-gain , capital-loss , hours-per-week , and education-num .

Original: The categorical features are the following: workclass , education , marital-stataus , occupation , relationship , race , sex , native-country .
Revised: The categorical features are the following: workclass , education , marital-stataus , occupation , relationship , race , sex , native-country .

Original: See Dua & Graff (2017) for a description of each feature.
Revised: See Dua & Graff (2017) for a description of each feature.

Original: We remove fnlwgt and education but keep education-num , which is a integer representation of education.
Revised: We remove fnlwgt and education but keep education-num , which is a integer representation of education.

Original: We do not use native-country , but use race and sex as predictive features.
Revised: We do not use native-country , but use race and sex as predictive features.

Revised: We treat race as binary:

Revised: individuals are either White or non-White.

Original: For every categorical feature, we use one hot encoding.
Revised: For every categorical feature, we use one hot encoding.

Original: For every continuous feature, we standardize, i.e., subtract the mean and divide by the standard deviation.
Revised: For every continuous feature, we standardize, i.e., subtract the mean and divide by the standard deviation.

Original: We remove anyone with missing data leaving 45,222 individuals.
Revised: We remove anyone with missing data leaving 45,222 individuals.

Original: This data is imbalanced: 25% make at least $50k per year.
Revised: This data is imbalanced: 25% make at least $50k per year.

Original: Furthermore, there is demographic imbalance with respect to race and gender as well as class imbalance on the outcome when conditioning on race or gender: 86% of individuals are white of which 26% make at least $50k a year; 67% of individuals are male of which 31% make at least $50k a year; 11% of females make at least $50k a year; and 15% of non-whites make at least $50k a year.
Revised: Furthermore, there is demographic imbalance with respect to race and gender as well as class imbalance on the outcome when conditioning on race or gender: 86% of individuals are white of which 26% make at least $50k a year; 67% of individuals are male of which 31% make at least $50k a year; 11% of females make at least $50k a year; and 15% of non-whites make at least $50k a year.

Original: D. F ULL EXPERIMENTAL RESULTS
Revised: D. F ULL EXPERIMENTAL RESULTS

Original: See Tables 4 and 5 for the full experiment results.
Revised: See Tables 4 and 5 for the full experiment results.

Original: The tables reports the average and the standard error for each metric of 10 train and test splits.
Revised: The tables reports the average and the standard error for each metric on the test set for 10 train and test splits

Original:  D. S ENSITIVE SUBSPACE
Revised: . D. S ENSITIVE SUBSPACE

Original: To learn the hyperplane that classiﬁes females and males, we use our implementation of regularized logistic regression with a batch size of 5k, 5k epochs, and .1 (cid:96) 2 regularization.
Revised: To learn the hyperplane that classiﬁes females and males, we use our implementation of regularized logistic regression with a batch size of 5k, 5k epochs, and .1 (cid:96) 2 regularization.

Original: D. H YPERPARAMETERS AND TRAINING
Revised: D. H YPERPARAMETERS AND TRAINING

Original: For each model, we use the same 10 train/test splits where use 80% of the data for training.
Revised: For each model, we use the same 10 train/test splits where use 80% of the data for training.

Original: Because of the class imbalance, each minibatch is sampled so that there are an equal number of training points from both the “income at least $50k class” and the “income below $50k class.”
Revised: Because of the class imbalance, each minibatch is sampled so that there are an equal number of training points from both the “income at least $50k class” and the “income below $50k class.”

Original: D.4. B ASELINE , P ROJECT , AND S EN SR
Revised: D.4. B ASELINE , P ROJECT , AND S EN SR

Original: See Table 3 for the hyperparameters we used on the Baseline, Project, and SenSR where the hyperparameters are deﬁned in Section C.
Revised: See Table 3 for the hyperparameters we used on the Baseline, Project, and SenSR where the hyperparameters are deﬁned in Section C.

Original: D.4. A DVESARIAL DEBIASING
Revised: D.4. A DVESARIAL DEBIASING

Original: We used Zhang et al.
Revised: We used Zhang et al.

Original: (2018)’s adversarial debiasing implementation in IBM’s AIF360 package (Bellamy et al., 2018) where the source code was modiﬁed so that each mini-batch is balanced with respect to the binary labels just as we did with our experiments and dropout was not used.
Revised: (2018)’s adversarial debiasing implementation in IBM’s AIF360 package (Bellamy et al., 2018) where the source code was modiﬁed so that each mini-batch is balanced with respect to the binary labels just as we did with our experiments and dropout was not used.

Original: Hyperparameters are the following: adversary loss weight = .
Revised: Hyperparameters are the following: adversary loss weight = .

Original: 001 , num epochs = 500 , batch size = 1000 , and privileged groups are deﬁned by binary gender and binary race.
Revised: 001 , num epochs = 500 , batch size = 1000 , and privileged groups are deﬁned by binary gender and binary race.

Original: D. G ROUP FAIR METRICS
Revised: D. G ROUP FAIR METRICS

Original: Let C be a set of classes, A be a binary protected attribute and Y, ˆ Y ∈ C be the true class label and the predicted class label.
Revised: Let C be a set of classes, A be a binary protected attribute and Y, ˆ Y ∈ C be the true class label and the predicted class label.

Original: Then for a ∈ { 0 , 1 } and c ∈ C deﬁne TPR a,c = P ( ˆ Y = c
Revised: Then for a ∈ { 0 , 1 } and c ∈ C deﬁne TPR a,c = P ( ˆ Y = c

Original: | A
Revised: | A

Original: = a, Y = c ) ;
Revised: = a, Y = c ) ;

Original: Gap A,c = TPR 0 ,c − TPR 1 ,c ; Gap RMS A
Revised: Gap A,c = TPR 0 ,c − TPR 1 ,c ; Gap RMS A

Original: [Equation]
Revised: [Equation]

Original: Balanced Acc = 1 | C | (cid:80) c ∈ C P ( ˆ Y =
Revised: Balanced Acc = 1 | C | (cid:80) c ∈ C P ( ˆ Y =

Original: c
Revised: c

Original: | Y = c ) .
Revised: | Y = c ) .

Original: For Adult, we report Gap RMS R , Gap RMS G , Gap max R , and Gap max G where C is composed of the two classes that correspond to whether someone made at least $50k, R refers to race, and G refers to gender.
Revised: For Adult, we report Gap RMS R , Gap RMS G , Gap max R , and Gap max G where C is composed of the two classes that correspond to whether someone made at least $50k, R refers to race, and G refers to gender.