Original: Attainability and Optimality: The Equalized Odds Fairness Revisited
Revised: Attainability and Optimality: The Equalized Odds Fairness Revisited

Original: Bernhard Sch¨olkopf, Caroline Uhler and Kun Zhang

Original: Abstract
Revised: Abstract

Original: Fairness of machine learning algorithms has been of increasing interest.
Revised: Fairness of machine learning algorithms has been of increasing interest.

Original: In order to suppress or eliminate discrimination in prediction, various notions as well as approaches have been proposed to impose fairness.
Revised: In order to suppress or eliminate discrimination in prediction, various notions as well as approaches have been proposed to impose fairness.

Original: Given a notion of fairness, an essential problem is then whether or not it can always be attained, even if with an unlimited amount of data.
Revised: Given a notion of fairness, an essential problem is then whether or not it can always be attained, even if with an unlimited amount of data.

Original: This issue is, however, not well addressed yet.
Revised: This issue is, however, not well addressed yet.

Original: In this paper, focusing on the Equalized Odds notion of fairness, we consider the attainability of this criterion and, furthermore, if it is attainable, the optimality of the prediction performance under various settings.
Revised: In this paper, focusing on the Equalized Odds notion of fairness, we consider the attainability of this criterion and, furthermore, if it is attainable, the optimality of the prediction performance under various settings.

Original: In particular, for prediction performed by a deterministic function of input features, we give the condition(s) under which Equalized Odds can hold true; if the stochastic prediction is acceptable, we show that under mild assumptions, fair predictors can always be derived.
Revised: In particular, for prediction performed by a deterministic function of input features, we give the condition(s) under which Equalized Odds can hold true; if the stochastic prediction is acceptable, we show that under mild assumptions, fair predictors can always be derived.

Original: For classiﬁcation, we further prove that compared to enforcing fairness by post-processing, one can always beneﬁt from exploiting all available features during training and get potentially better prediction performance while remaining fair.
Revised: For classiﬁcation, we further prove that compared to enforcing fairness by post-processing, one can always beneﬁt from exploiting all available features during training and get potentially better prediction performance while remaining fair.

Original: Moreover, while stochastic prediction can attain
Revised: Moreover, while stochastic prediction can attain

Original: Equalized Odds with theoretical guarantees, we also discuss its limitation and potential negative social impacts.
Revised: Equalized Odds with theoretical guarantees, we also discuss its limitation and potential negative social impacts.

Original: 1. Introduction
Revised: 1. Introduction

Original: As machine learning models become widespread in automated decision making systems, apart from the efﬁciency and accuracy of the prediction, their potential social consequence also gains increasing attention.
Revised: As machine learning models become widespread in automated decision making systems, apart from the efﬁciency and accuracy of the prediction, their potential social consequence also gains increasing attention.

Original: The call for accountability and fairness in machine learning has motivated various (statistical) notions of fairness.
Revised: The call for accountability and fairness in machine learning has motivated various (statistical) notions of fairness.

Original: Demographic Parity (Calders et al., 2009) requires the independence between prediction (e.g., of a classiﬁer) and the protected feature (sensitive attributes of an individual, e.g., gender, race).
Revised: Demographic Parity (Calders et al., 2009) requires the independence between prediction (e g., of a classiﬁer) and the protected feature (sensitive attributes of an individual, e.g., gender, race).

Original: Equalized Odds (Hardt et al., 2016), also known as Error-rate
Revised: Equalized Odds (Hardt et al., 2016), also known as Error-rate

Original: Balance (Chouldechova, 2017), requires the output of a model be conditionally independent of protected feature(s) given the ground truth of the target.
Revised: Balance (Chouldechova, 2017), requires the output of a model be conditionally independent of protected feature(s) given the ground truth of the target.

Original: Predictive Rate Parity (Zafar et al., 2017a), on the other hand, requires the actually proportion of positives (negatives) in the original data for positive (negative) predictions should match across groups (well-calibrated).
Revised: Predictive Rate Parity (Zafar et al., 2017a), on the other hand, requires the actually proportion of positives (negatives) in the original data for positive (negative) predictions should match across groups (well-calibrated).

Original: On the theoretical side, results have been reported regarding relationships among fairness notions.
Revised: On the theoretical side, results have been reported regarding relationships among fairness notions.

Original: It has been shown that if base rates of positives differ among groups, then Equalized Odds and
Revised: It has been shown that if base rates of positives differ among groups, then Equalized Odds and

Original: Predictive Rate Parity cannot be achieved simultaneously for non-perfect predictors (Kleinberg et al., 2016; Chouldechova, 2017).
Revised: Predictive Rate Parity cannot be achieved simultaneously for non-perfect predictors (Kleinberg et al., 2016; Chouldechova, 2017).

Original: Any two out of three among Demographic Parity, Equalized Odds, and
Revised: Any two out of three among Demographic Parity, Equalized Odds, and

Original: Predictive Rate Parity are incompatible with each other (Barocas et al., 2017).
Revised: Predictive Rate Parity are incompatible with each other (Barocas et al., 2017).

Original: In practice, one can broadly categorize computational procedures to derive a fair predictor into three types: pre-processing approaches (Calders et al., 2009; Dwork et al., 2012; Zemel et al., 2013;
Revised: In practice, one can broadly categorize computational procedures to derive a fair predictor into three types: pre-processing approaches (Calders et al., 2009; Dwork et al., 2012; Zemel et al., 2013;

Original: Zhang et al., 2018; Madras et al., 2018; Creager et al., 2019; Zhao et al., 2020), in-processing approaches (Kamishima et al., 2011; P´erez-Suay et al., 2017; Zafar et al., 2017a,b; Donini et al., 2018; Song et al., 2019; Mary et al., 2019; Baharlouei et al., 2020; Romano et al., 2020), and post-processing approaches (Hardt et al., 2016; Fish et al., 2016; Dwork et al., 2018).
Revised: Zhang et al., 2018; Madras et al., 2018; Creager et al., 2019; Zhao et al., 2020), in-processing approaches (Kamishima et al., 2011; P´erez-Suay et al., 2017; Zafar et al., 2017a,b; Donini et al., 2018; Song et al., 2019; Mary et al., 2019; Baharlouei et al., 2020; Romano et al., 2020), and post-processing approaches (Hardt et al., 2016; Fish et al., 2016; Dwork et al., 2018).

Original: In accord with the fairness notion of interest, a pre-processing approach ﬁrst maps the training data to a transformed space to remove discriminatory information between protected feature and target, and then pass on the data to make prediction.
Revised: In accord with the fairness notion of interest, a pre-processing approach ﬁrst maps the training data to a transformed space to remove discriminatory information between protected feature and target, and then pass on the data to make prediction.

Original: In direct contrast, a post-processing approach treats the off-the-shelf predictors (usually limited to classiﬁcation tasks) as uninterpretable black-box(es), and imposes fairness by outputting a function of the original prediction.
Revised: In direct contrast, a post-processing approach treats the off-the-shelf predictors (usually limited to classiﬁcation tasks) as uninterpretable black-box(es), and imposes fairness by outputting a function of the original prediction.

Original: For in-processing approaches, various kinds of regularization terms are proposed so that one can optimize the utility function while suppressing the discrimination at the same time.
Revised: For in-processing approaches, various kinds of regularization terms are proposed so that one can optimize the utility function while suppressing the discrimination at the same time.

Original: Approaches based on estimating/bounding the causal effect of the protected feature on the ﬁnal target have also been proposed (Kusner et al., 2017;
Revised: Approaches based on estimating/bounding the causal effect of the protected feature on the ﬁnal target have also been proposed (Kusner et al., 2017;

Original: Russell et al., 2017; Zhang et al., 2017; Nabi and Shpitser, 2018; Zhang and Bareinboim, 2018;
Revised: Russell et al., 2017; Zhang et al., 2017; Nabi and Shpitser, 2018; Zhang and Bareinboim, 2018;

Original: Chiappa, 2019; Wu et al., 2019).
Revised: Chiappa, 2019; Wu et al., 2019).

Original: The attainability of Equalized Odds, namely, the existence of the predictor that can score zero violation of fairness in the large sample limit, is an asymptotic property of the fairness criterion that has important practical implications.
Revised: The attainability of Equalized Odds, namely, the existence of the predictor that can score zero violation of fairness in the large sample limit, is an asymptotic property of the fairness criterion that has important practical implications.

Original: It characterizes a completely different kind of violation of fairness compared to the empirical error bound of discrimination in ﬁnite-sample cases.
Revised: It characterizes a completely different kind of violation of fairness compared to the empirical error bound of discrimination in ﬁnite-sample cases.

Original: If we deploy a “fair” prediction system in practice whose output is actually biased (because fairness is in fact not attainable with the chosen predictor), the discrimination would become a snake in the grass, which is often easily neglected and hard to eliminate.
Revised: If we deploy a “fair” prediction system in practice whose output is actually biased (because fairness is in fact not attainable with the chosen predictor), the discrimination would become a snake in the grass, which is often easily neglected and hard to eliminate.

Original: Although various approaches have been proposed to impose Equalized Odds, whether or not it is always attainable is not well addressed.
Revised: Although various approaches have been proposed to impose Equalized Odds, whether or not it is always attainable is not well addressed.

Original: Hardt et al.
Revised: Hardt et al.

Original: (2016) already noticed that one “might need to introduce additional randomness [to achieve Equalized
Revised: (2016) already noticed that one “might need to introduce additional randomness [to achieve Equalized

Original: Odds in binary classiﬁcation]”.
Revised: Odds in binary classiﬁcation]”.

Original: Compared to the claim that “[deterministic classiﬁer] might not be sufﬁcient [to achieve fairness]” in the FICO score case study by Hardt et al.
Revised: Compared to the claim that “[deterministic classiﬁer] might not be sufﬁcient [to achieve fairness]” in the FICO score case study by Hardt et al.

Original: (2016), we present an afﬁrmative answer with respect the necessary and sufﬁcient conditions under which Equalized
Revised: (2016), we present an afﬁrmative answer with respect the necessary and sufﬁcient conditions under which Equalized

Original: Odds can be attained (for both regression and classiﬁcation tasks).
Revised: Odds can be attained (for both regression and classiﬁcation tasks).

Original: Actually, as we illustrate in this paper, Equalized Odds is not always attainable if we use deterministic prediction functions.
Revised: Actually, as we illustrate in this paper, Equalized Odds is not always attainable if we use deterministic prediction functions.

Original: Our contributions are mainly threefold:
Revised: Our contributions are mainly threefold:

Original: • For regression and classiﬁcation tasks with deterministic prediction functions, we show that Equalized Odds is not always attainable unless certain (rather restrictive) conditions on the joint distribution of the features and the target variable are met.
Revised: • For regression and classiﬁcation tasks with deterministic prediction functions, we show that Equalized Odds is not always attainable unless certain (rather restrictive) conditions on the joint distribution of the features and the target variable are met.

Original: • For regression and (binary) classiﬁcation, we show that with a stochastic predictor, under mild assumptions one can always derive a non-trivial Equalized Odds predictor.
Revised: • For regression and (binary) classiﬁcation, we show that with a stochastic predictor, under mild assumptions one can always derive a non-trivial Equalized Odds predictor.

Original: • Considering the optimality of performance under fairness constraint(s), when exploiting all available features, we show that the predictor derived via an in-processing approach would generally perform better, and at least no worse than the one derived via a post-processing approach (unconstrained optimization followed by a post-processing step).
Revised: • Considering the optimality of performance under fairness constraint(s), when exploiting all available features, we show that the predictor derived via an in-processing approach would generally perform better, and at least no worse than the one derived via a post-processing approach (unconstrained optimization followed by a post-processing step).

Original: 2. Preliminaries
Revised: 2. Preliminaries

Original: In this section, we ﬁrst distinguish between objects of interest with respect to which fairness is discussed, and then present the deﬁnition of Equalized Odds.
Revised: In this section, we ﬁrst distinguish between objects of interest with respect to which fairness is discussed, and then present the deﬁnition of Equalized Odds.

Original: 2.1. Different Fairness Semantics
Revised: 2.1. Different Fairness Semantics

Original: Before presenting the fairness formulation, it is helpful to see the distinction between different semantics of fairness when discussing fair predictors.
Revised: Before presenting the fairness formulation, it is helpful to see the distinction between different semantics of fairness when discussing fair predictors.

Original: When evaluating the performance of the proposed fair predictor, it is a common practice to compare the loss (with respect to the utility function of choice, e.g., accuracy for binary classiﬁcation) computed on target variable and the predicted value.
Revised: When evaluating the performance of the proposed fair predictor, it is a common practice to compare the loss (with respect to the utility function of choice, e.g., accuracy for binary classiﬁcation) computed on target variable and the predicted value.

Original: There is an implicit assumption behind this practice: the generating process of the data, which is just describing a real-world procedure, is not biased in any sense (Danks and London, 2017).
Revised: There is an implicit assumption behind this practice: the generating process of the data, which is just describing a real-world procedure, is not biased in any sense (Danks and London, 2017).

Original: Only when we treat the target variable (recorded in the dataset) as unbiased can we justify the practice of loss evaluation and the conditioning on target variable when imposing fairness (as we shall see in the deﬁnition of Equalized Odds in Equation 1).
Revised: Only when we treat the target variable (recorded in the dataset) as unbiased can we justify the practice of loss evaluation and the conditioning on target variable when imposing fairness (as we shall see in the deﬁnition of Equalized Odds in Equation 1).

Original: One may consider a music school admission example.
Revised: One may consider a music school admission example.

Original: The music school committee would decide if they admit a student to the violin performance program based on the applicant’s personal information, educational background, instrumental performance, and so on.
Revised: The music school committee would decide if they admit a student to the violin performance program based on the applicant’s personal information, educational background, instrumental performance, and so on.

Original: When evaluating whether or not the admission is “fair”, the semantics of fairness come in at least two folds.
Revised: When evaluating whether or not the admission is “fair”, the semantics of fairness come in at least two folds.

Original: First, based on the information at hand, did the committee evaluate the qualiﬁcation of applicants without bias (How committee evaluate the applicants)?
Revised: First, based on the information at hand, did the committee evaluate the qualiﬁcation of applicants without bias (How committee evaluate the applicants)?

Original: Second, is committee’s procedure of evaluating applicants’ qualiﬁcation reasonable (How other people view the evaluation procedure used by the committee)?
Revised: Second, is committee’s procedure of evaluating applicants’ qualiﬁcation reasonable (How other people view the evaluation procedure used by the committee)?

Original: In this paper, we assume the data recorded is unbiased and focus on the prediction (made with respect to current reality), such that the prediction itself does not include any biased utilization of information.
Revised: In this paper, we assume the data recorded is unbiased and focus on the prediction (made with respect to current reality), such that the prediction itself does not include any biased utilization of information.

Original: The fairness with respect to the data generating procedure as well as the potential future inﬂuence of the prediction are beyond the scope of this paper.
Revised: The fairness with respect to the data generating procedure as well as the potential future inﬂuence of the prediction are beyond the scope of this paper.

Original: 2.2. Equalized Odds Fairness
Revised: 2.2. Equalized Odds Fairness

Original: Hardt et al.
Revised: Hardt et al.

Original: (2016) proposed Equalized Odds which requires the conditional independence between prediction and protected feature given ground truth of the target.
Revised: (2016) proposed Equalized Odds which requires the conditional independence between prediction and protected feature given ground truth of the target.

Original: Let us denote the protected feature by A , with domain of value A , additional (observable) features by X , with domain of value X , target variable by Y , with domain Y , (not necessarily fair) predictors by
Revised: Let us denote the protected feature by A , with domain of value A , additional (observable) features by X , with domain of value X , target variable by Y , with domain Y , (not necessarily fair) predictors by

Original: [Equation]
Revised: [Equation]

Original: Y , and fair predictors by
Revised: Y , and fair predictors by

Original: [Equation]
Revised: [Equation]

Original: Equalized-Odds fairness requires
Revised: Equalized-Odds fairness requires

Original: [Equation]
Revised: [Equation]

Original: For classiﬁcation tasks, one can conveniently use the probability distribution form, i.e., ∀ a ∈
Revised: For classiﬁcation tasks, one can conveniently use the probability distribution form, i.e., ∀ a ∈

Original: A , y, ˜ y ∈ Y :
Revised: A , y, ˜ y ∈ Y :

Original: [Equation]
Revised: [Equation]

Original: or more concisely
Revised: or more concisely

Original: [Equation]
Revised: [Equation]

Original: For better readability, we also use the formulation in Equation 3 in cases without ambiguity.
Revised: For better readability, we also use the formulation in Equation 3 in cases without ambiguity.

Original: In the context of binary classiﬁcation ( Y = { 0 , 1 } ), Equalized Odds requires that the True Positive Rate (TPR) and False Positive Rate (FPR) of each certain group match TPR and FPR of the population.
Revised: In the context of binary classiﬁcation ( Y = { 0 , 1 } ), Equalized Odds requires that the True Positive Rate (TPR) and False Positive Rate (FPR) of each certain group match TPR and FPR of the population.

Original: Throughout the paper, without loss of generality we assume that there is only one protected feature for the purpose of simplifying notation.
Revised: Throughout the paper, without loss of generality we assume that there is only one protected feature for the purpose of simplifying notation.

Original: However, considering the fact that the protected feature can be discrete (e.g., race, sex) or continuous (e.g., the ratio of ethnic group in the population for certain district of a city), we do not assume discreteness of the protected feature.
Revised: However, considering the fact that the protected feature can be discrete (e.g., race, sex) or continuous (e.g., the ratio of ethnic group in the population for certain district of a city), we do not assume discreteness of the protected feature.

Original: Due to the space limit, we will focus on the illustration and implication of our results and defer all the proofs to Section A of the appendix.
Revised: Due to the space limit, we will focus on the illustration and implication of our results and defer all the proofs to Section A of the appendix.

Original: 3. Fairness in Regression
Revised: 3. Fairness in Regression

Original: Various regularization terms have also been proposed to suppress discrimination when predicting a continuous target (Berk et al., 2017;
Revised: Various regularization terms have also been proposed to suppress discrimination when predicting a continuous target (Berk et al., 2017;

Original: Zhang et al., 2018; Mary et al., 2019; Romano et al., 2020).
Revised: Zhang et al., 2018; Mary et al., 2019; Romano et al., 2020).

Original: However, whether or not one can always achieve 0-discrimination for regression, even if with an unlimited amount of data, is not clear.
Revised: However, whether or not one can always achieve 0-discrimination for regression, even if with an unlimited amount of data, is not clear.

Original: In this section we ﬁrst consider a simple setup with linearly correlated continuous data as an example to show that Equalized Odds is not always attainable.
Revised: In this section we ﬁrst consider a simple setup with linearly correlated continuous data as an example to show that Equalized Odds is not always attainable.

Original: Then we consider more general cases where regression is performed by a deterministic predictor and derive the condition under which Equalized Odds can hold true.
Revised: Then we consider more general cases where regression is performed by a deterministic predictor and derive the condition under which Equalized Odds can hold true.

Original: Finally, when stochastic prediction is utilized, we show that under mild assumptions one can always ﬁnd a non-trivial fair predictor, i.e.,
Revised: Finally, when stochastic prediction is utilized, we show that under mild assumptions one can always ﬁnd a non-trivial fair predictor, i.e.,

Original: Equalized Odds is guaranteed to be (non-trivially) attainable.
Revised: Equalized Odds is guaranteed to be (non-trivially) attainable.

Original: 3.1. Unattainability of Equalized Odds in Linear Non-Gaussian Regression
Revised: 3.1. Unattainability of Equalized Odds in Linear Non-Gaussian Regression

Original: Let us start with the speciﬁc linear, non-Gaussian situation where the data is generated as follows ( H is not measured in the dataset):
Revised: Let us start with the speciﬁc linear, non-Gaussian situation where the data is generated as follows ( H is not measured in the dataset):

Original: [Equation]
Revised: [Equation]

Original: where ( A, E X , E H , E Y ) are mutually independent and q, b, c, d are constants.
Revised: where ( A, E X , E H , E Y ) are mutually independent and q, b, c, d are constants.

Original: Let us denote E :=
Revised: Let us denote E :=

Original: E Y + dE H and let (cid:98)
Revised: E Y + dE H and let (cid:98)

Original: Y be a linear combination of A and X , i.e., (cid:98)
Revised: Y be a linear combination of A and X , i.e., (cid:98)

Original: Y = αA + βX
Revised: Y = αA + βX

Original: =
Revised: =

Original: ( α + qβ ) A + βE X , with linear coefﬁcients α and β , where β (cid:54) = 0 .
Revised: ( α + qβ ) A + βE X , with linear coefﬁcients α and β , where β (cid:54) = 0 .

Original: In Theorem 1, we present the general result in linear non-Gaussian cases, where one cannot achieve the conditional independence between (cid:98)
Revised: In Theorem 1, we present the general result in linear non-Gaussian cases, where one cannot achieve the conditional independence between (cid:98)

Original: Y and A given Y .
Revised: Y and A given Y .

Original: Theorem 1 ( Unattainability of Equalized Odds in the Linear Non-Gaussian Case )
Revised: Theorem 1 ( Unattainability of Equalized Odds in the Linear Non-Gaussian Case )

Original: Assume that X has a causal inﬂuence on Y , i.e., c (cid:54)
Revised: Assume that X has a causal inﬂuence on Y , i.e., c (cid:54)

Original: = 

Revised: =

Original: in Equation 4, and that A and Y are not independent, i.e., qc + bd (cid:54) = 0 .
Revised: in Equation 4, and that A and Y are not independent, i.e., qc + bd (cid:54) = 0 .

Original: Assume p E X and p E are positive on R .
Revised: Assume p E X and p E are positive on R .

Original: Let f 1 :=
Revised: Let f 1 :=

Original: log p A , f 2 := log p
Revised: log p A , f 2 := log p

Original: E X , and f 3 := log p E .
Revised: E X , and f 3 := log p E .

Original: Further assume that f 2 and f 3 are third-order differentiable.
Revised: Further assume that f 2 and f 3 are third-order differentiable.

Original: Then if at most one of E X and E is Gaussian,
Revised: Then if at most one of E X and E is Gaussian,

Original: [Equation]
Revised: [Equation]

Original: Y is always conditionally dependent on A given Y .
Revised: Y is always conditionally dependent on A given Y .

Original: From Theorem 1, we can see that if at most one of E X and E is Gaussian, then any linear combination of A and X with non-zero coefﬁcients will not be conditionally independent from A given Y , meaning that it is impossible to (non-trivially) achieve Equalized Odds with a linear model in the linear non-Gaussian case.
Revised: From Theorem 1, we can see that if at most one of E X and E is Gaussian, then any linear combination of A and X with non-zero coefﬁcients will not be conditionally independent from A given Y , meaning that it is impossible to (non-trivially) achieve Equalized Odds with a linear model in the linear non-Gaussian case.

Original: 3.2. Regression with Deterministic Prediction
Revised: 3.2. Regression with Deterministic Prediction

Original: We have seen that in the linear non-Gaussian case, any non-zero linear combination of the feature (which is a deterministic prediction function of the input) will not satisfy Equalized Odds.
Revised: We have seen that in the linear non-Gaussian case, any non-zero linear combination of the feature (which is a deterministic prediction function of the input) will not satisfy Equalized Odds.

Original: One may naturally wonder, instead of only considering linear models, whether or not Equalized Odds can be achieved in general regression cases.
Revised: One may naturally wonder, instead of only considering linear models, whether or not Equalized Odds can be achieved in general regression cases.

Original: It is therefore desirable to derive the condition under which
Revised: It is therefore desirable to derive the condition under which

Original: Equalized Odds can hold true for regression with deterministic prediction functions.
Revised: Equalized Odds can hold true for regression with deterministic prediction functions.

Original: Theorem 2 ( Condition to Achieve Equalized Odds for Regression with Deterministic Prediction )
Revised: Theorem 2 ( Condition to Achieve Equalized Odds for Regression with Deterministic Prediction )

Original: Assume that the protected feature A and the continuous target variable Y are dependent and that their joint probability density p ( A, Y ) is positive for every combination of possible values of A and
Revised: Assume that the protected feature A and the continuous target variable Y are dependent and that their joint probability density p ( A, Y ) is positive for every combination of possible values of A and

Original: Y .
Revised: Y .

Original: Further assume that Y is not fully determined by A , and that there are additional features X that are not independent of Y .
Revised: Further assume that Y is not fully determined by A , and that there are additional features X that are not independent of Y .

Original: Let the prediction (cid:98)
Revised: Let the prediction (cid:98)

Original: Y be characterized by a deterministic function f : A × X → Y .
Revised: Y be characterized by a deterministic function f : A × X → Y .

Original: Equalized Odds holds true if and only if the following condition is satisﬁed ( δ ( · ) is the delta function):
Revised: Equalized Odds holds true if and only if the following condition is satisﬁed ( δ ( · ) is the delta function):

Original: [Equation]
Revised: [Equation]

Original: In special cases when X ⊥⊥ A | Y and f is a function of only X (e.g., a linear function with 0 coefﬁcient for A ), this condition is always satisﬁed.
Revised: In special cases when X ⊥⊥ A | Y and f is a function of only X (e.g., a linear function with 0 coefﬁcient for A ), this condition is always satisﬁed.

Original: In general cases, however, this condition speciﬁes a rather strong constraint on the relation between the conditional probability density p X | AY ( x | a, y ) and the function f .
Revised: In general cases, however, this condition speciﬁes a rather strong constraint on the relation between the conditional probability density p X | AY ( x | a, y ) and the function f .

Original: Because of the integration over the product of δ ( · ) function and p X | AY ( x | a, y ) , the aforementioned condition requires f to “pick out” certain x ’s (for any given ˆ y ∈ Y and a ∈ A ) such that the sum of the corresponding conditional probability density values stays the same.
Revised: Because of the integration over the product of δ ( · ) function and p X | AY ( x | a, y ) , the aforementioned condition requires f to “pick out” certain x ’s (for any given ˆ y ∈ Y and a ∈ A ) such that the sum of the corresponding conditional probability density values stays the same.

Original: Generally speaking, if the way f “picks out” x is not strictly coupled with the conditional probability density p X | AY ( x | a, y ) , the condition speciﬁed in Theorem 2 would be violated, i.e., Equalized Odds cannot hold true.
Revised: Generally speaking, if the way f “picks out” x is not strictly coupled with the conditional probability density p X | AY ( x | a, y ) , the condition speciﬁed in Theorem 2 would be violated, i.e., Equalized Odds cannot hold true.

Original: In fact, one can formally establish the connection between Theorem 2 (for general regression cases) and Theorem 1 (for linear non-Gaussian regression cases) by specifying the way f “picks out” x ’s. 1 As we will see in Theorem 4, the phenomenon of unattainable Equalized Odds also occurs in classiﬁcation with deterministic predictors.
Revised: In fact, one can formally establish the connection between Theorem 2 (for general regression cases) and Theorem 1 (for linear non-Gaussian regression cases) by specifying the way f “picks out” x ’s. 1 As we will see in Theorem 4, the phenomenon of unattainable Equalized Odds also occurs in classiﬁcation with deterministic predictors.

Original: 3.3. Regression with Stochastic Prediction
Revised: 3.3. Regression with Stochastic Prediction

Original: In light of the unattainability of Equalized Odds for prediction with deterministic functions of A and X , it is important to ask whether Equalized Odds can be attainable with stochastic prediction, where the output given input features is no longer a single value.
Revised: In light of the unattainability of Equalized Odds for prediction with deterministic functions of A and X , it is important to ask whether Equalized Odds can be attainable with stochastic prediction, where the output given input features is no longer a single value.

Original: As we shall see in Theorem 3, with stochastic prediction functions, under mild assumptions one can always ﬁnd a non-trivial fair predictor (cid:101)
Revised: As we shall see in Theorem 3, with stochastic prediction functions, under mild assumptions one can always ﬁnd a non-trivial fair predictor (cid:101)

Original: Y that satisﬁes Equalized Odds, even if both the target Y and the protected feature A are continuous.
Revised: Y that satisﬁes Equalized Odds, even if both the target Y and the protected feature A are continuous.

Original: Theorem 3 ( Attainability of Equalized Odds for Regression with Stochastic Prediction )
Revised: Theorem 3 ( Attainability of Equalized Odds for Regression with Stochastic Prediction )

Original: Let A , X , and Y be continuous variables with domain of value A , X , and Y , respectively.
Revised: Let A , X , and Y be continuous variables with domain of value A , X , and Y , respectively.

Original: Assume that their joint distribution is ﬁxed and known.
Revised: Assume that their joint distribution is ﬁxed and known.

Original: Further assume Y ⊥(cid:54)⊥ A , Y ⊥(cid:54)⊥ X , and Y ⊥(cid:54)⊥ X | A .
Revised: Further assume Y ⊥(cid:54)⊥ A , Y ⊥(cid:54)⊥ X , and Y ⊥(cid:54)⊥ X | A .

Original: Without loss of generality let the conditional probability density p X | AY ( x | a, y ) be non-negative and ﬁnite.
Revised: Without loss of generality let the conditional probability density p X | AY ( x | a, y ) be non-negative and ﬁnite.

Original: Then there exists (cid:101)
Revised: Then there exists (cid:101)

Original: Y with domain of value Y whose distribution is fully determined by p (cid:101) Y | AX (˜ y | a, x ) , such that (cid:101)
Revised: Y with domain of value Y whose distribution is fully determined by p (cid:101) Y | AX (˜ y | a, x ) , such that (cid:101)

Original: Y is not independent from ( A, X ) but (cid:101)
Revised: Y is not independent from ( A, X ) but (cid:101)

Original: Y ⊥⊥ A | Y , i.e., the Equalized
Revised: Y ⊥⊥ A | Y , i.e., the Equalized

Original: Odds is non-trivially attainable.
Revised: Odds is non-trivially attainable.

Original: From Theorem 3 we can see that with stochastic prediction, one can guarantee the attainability of Equalized Odds for regression under rather mild assumptions.
Revised: From Theorem 3 we can see that with stochastic prediction, one can guarantee the attainability of Equalized Odds for regression under rather mild assumptions.

Original: It would then be desirable to construct general, nonlinear prediction models to produce a stochastic prediction (i.e., with a certain type of randomness in the prediction).
Revised: It would then be desirable to construct general, nonlinear prediction models to produce a stochastic prediction (i.e., with a certain type of randomness in the prediction).

Original: One possible way follows the framework of Generative
Revised: One possible way follows the framework of Generative

Original: Adversarial Networks (GANs) (Goodfellow et al., 2014): we use random standard Gaussian noise E , in addition to A and X , as input, such that the output will have a speciﬁc type of randomness.
Revised: Adversarial Networks (GANs) (Goodfellow et al., 2014): we use random standard Gaussian noise E , in addition to A and X , as input, such that the output will have a speciﬁc type of randomness.

Original: The parameters involved are learned by minimizing prediction error and enforcing Equalized Odds on the
Revised: The parameters involved are learned by minimizing prediction error and enforcing Equalized Odds on the

Original: “randomized” output, as a function of A , X , and E , at the same time.
Revised: “randomized” output, as a function of A , X , and E , at the same time.

Original: Given the loss function L , a class of function F , and a fairness penalty G , we propose the following objective function for ﬁtting an Equalized Odds model with stochastic prediction:
Revised: Given the loss function L , a class of function F , and a fairness penalty G , we propose the following objective function for ﬁtting an Equalized Odds model with stochastic prediction:

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst term measures the prediction error, for instance the mean squared error for regression.
Revised: The ﬁrst term measures the prediction error, for instance the mean squared error for regression.

Original: The second term is the fairness penalty that imposes Equalized Odds.
Revised: The second term is the fairness penalty that imposes Equalized Odds.

Original: We use the kernel measure of conditional dependence (Fukumizu et al., 2004, 2007) between (cid:101)
Revised: We use the kernel measure of conditional dependence (Fukumizu et al., 2004, 2007) between (cid:101)

Original: Y and A given Y as the fairness penalty term.
Revised: Y and A given Y as the fairness penalty term.

Original: The hyperparameter λ reﬂects the trade-off between accuracy and fairness.
Revised: The hyperparameter λ reﬂects the trade-off between accuracy and fairness.

Original: 4. Fairness in Classiﬁcation
Revised: 4. Fairness in Classiﬁcation

Original: In this section, we consider the attainability of Equalized Odds for binary classiﬁers (with a deterministic or stochastic prediction function), and furthermore, if attainable, the optimality of performance of various computational procedures under the fairness criterion.
Revised: In this section, we consider the attainability of Equalized Odds for binary classiﬁers (with a deterministic or stochastic prediction function), and furthermore, if attainable, the optimality of performance of various computational procedures under the fairness criterion.

Original: 4.1. Classiﬁcation with Deterministic Prediction
Revised: 4.1. Classiﬁcation with Deterministic Prediction

Original: We begin with considering cases when classiﬁcation is performed by a deterministic function of the input.
Revised: We begin with considering cases when classiﬁcation is performed by a deterministic function of the input.

Original: Similar to the previous analysis for regression tasks with deterministic prediction functions in Section 3.2, we derive the conditions under which Equalized Odds can possibly hold true for classiﬁcation with deterministic prediction functions.
Revised: Similar to the previous analysis for regression tasks with deterministic prediction functions in Section 3.2, we derive the conditions under which Equalized Odds can possibly hold true for classiﬁcation with deterministic prediction functions.

Original: Theorem 4 ( Condition to Achieve Equalized Odds for Classiﬁcation with Deterministic Prediction ) Assume that the protected feature A and Y are dependent and that their joint probability
Revised: Theorem 4 ( Condition to Achieve Equalized Odds for Classiﬁcation with Deterministic Prediction ) Assume that the protected feature A and Y are dependent and that their joint probability

Original: P ( A, Y ) (for discrete A ) or joint probability density p ( A, Y ) (for continuous A ) is positive for every combination of possible values of A and Y .
Revised: P ( A, Y ) (for discrete A ) or joint probability density p ( A, Y ) (for continuous A ) is positive for every combination of possible values of A and Y .

Original: Further assume that Y is not fully determined by A , and that there are additional features X that are not independent of Y .
Revised: Further assume that Y is not fully determined by A , and that there are additional features X that are not independent of Y .

Original: Let the output of the classiﬁer (cid:98)
Revised: Let the output of the classiﬁer (cid:98)

Original: Y be a deterministic function f : A × X → Y .
Revised: Y be a deterministic function f : A × X → Y .

Original: Let S (ˆ y )
Revised: Let S (ˆ y )

Original: A :
Revised: A :

Original: = { a | ∃ x ∈ X s.t. f
Revised: = { a | ∃ x ∈ X s.t. f

Original: ( a, x ) = ˆ y } , and
Revised: ( a, x ) = ˆ y } , and

Original: S (ˆ y )
Revised: S (ˆ y )

Original: X | a :=
Revised: X | a :=

Original: { x | f
Revised: { x | f

Original: ( a, x ) =
Revised: ( a, x ) =

Original: ˆ y } .
Revised: ˆ y } .

Original: Equalized Odds is attained if and only if the following conditions hold true (for continuous X , replace summation with integration accordingly):
Revised: Equalized Odds is attained if and only if the following conditions hold true (for continuous X , replace summation with integration accordingly):

Original: (i) ∀ ˆ
Revised: (i) ∀ ˆ

Original: [Equation]
Revised: [Equation]

Original: (ii) ∀ ˆ
Revised: (ii) ∀ ˆ

Original: [Equation]
Revised: [Equation]

Original: Condition (i) says that within each class determined by the classiﬁcation function f , A should be able to take all possible values in A .
Revised: Condition (i) says that within each class determined by the classiﬁcation function f , A should be able to take all possible values in A .

Original: While condition (i) is already rather restrictive, condition (ii) speciﬁes an even stronger constraint on the relation between P X | AY ( x | a, y ) (or p X | AY ( x | a, y ) for continuous X ) and the set S (ˆ y )
Revised: While condition (i) is already rather restrictive, condition (ii) speciﬁes an even stronger constraint on the relation between P X | AY ( x | a, y ) (or p X | AY ( x | a, y ) for continuous X ) and the set S (ˆ y )

Original: X | a (which is determined by the function f ).
Revised: X | a (which is determined by the function f ).

Original: Generally speaking, in order to score a better classiﬁcation accuracy, one would like to make P (cid:98) Y | A,X
Revised: Generally speaking, in order to score a better classiﬁcation accuracy, one would like to make P (cid:98) Y | A,X

Original: (ˆ y | a, x ) as close as possible to P Y | A,X ( y | a, x ) , and if the set S (ˆ y ) X | a and P X | AY ( x | a, y ) are not strictly coupled, condition (ii) would be violated, i.e., Equalized Odds cannot be attained.
Revised: (ˆ y | a, x ) as close as possible to P Y | A,X ( y | a, x ) , and if the set S (ˆ y ) X | a and P X | AY ( x | a, y ) are not strictly coupled, condition (ii) would be violated, i.e., Equalized Odds cannot be attained.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: </latexit>

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: <

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: /MN5/

Original: [Table] [Figure]
Revised: AB0dmh4=</latexit>

Original: [Table] [Figure] [Table] [Figure]
Revised: P

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: <latexit sha1_base64="cCJh62l4ZHE9IIFCD7kRF0oYRZM=">AAACDHicbVDLSgMxFM3UV62vqjvdhBahgpQZEXyAUHHjRqhgX7SlZDK3bWjmQZJRylBcCa78FFeCgujSH3Dlj7g203ah1QMXTs65l9x77IAzqUzz00hMTc/MziXnUwuLS8sr6dW1svRDQaFEfe6Lqk0kcOZBSTHFoRoIIK7NoWL3TmO/cgVCMt+7VP0Ami7peKzNKFFaaqUzxVzjmjmgGHcgqg3wMbZww2UOPtnBtfi13Upnzbw5BP5LrDHJFjZus18352/FVvqj4fg0dMFTlBMp65YZqGZEhGKUwyDVCCUEhPZIB+qaesQF2YyGtwzwllYc3PaFLk/hofpzIiKulH3X1p0uUV056cXif149VO2DZsS8IFTg0dFH7ZBj5eM4GOwwAVTxviaECqZ3xbRLBKFKx5dK6RSsyZv/kvJu3trLH17oOI7QCEm0iTIohyy0jwroDBVRCVF0hx7QE3o27o1H48V4HbUmjPHMOvoF4/0bz22bsg==</latexit>

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: <latexit sha1_base64="YiqR1N6yVb2IKH3hJ0hQyObye40=">AAACDHicbVDJSgNBEO2JW4zbqDe9NAlCBAkzIriAEPHiRYhgNpIQenoqSZOehe4eJQzBk+DJT/EkKIge/QFP/ohnO8tBow8KXr9XRVc9J+RMKsv6NBJT0zOzc8n51MLi0vKKubpWkkEkKBRpwANRcYgEznwoKqY4VEIBxHM4lJ3u6cAvX4GQLPAvVS

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure] [Table] [Figure]
Revised: =

Original: [Table]
Revised: [Table]

Original: [Figure] [Table] [Table] [Table]
Revised: areas

Original: [Table] [Table]
Revised: [Table] [Table]

Original: +z08b3Wzn84XshiHwJkgirVtK0IWwmVyJmANOfECiLKrmgXmpoG1AfVSkbHpOamVjyzE0r9AjRH6veJhPpKDXxXJ32KPfXbG4r/ec0YO5WWvimKEQI2XtSJhYmhOWzG9LgEhmKgCWWS67+arEclZaj7y+V0C/bvm/+Ss52yvVveP9F1HJAxsqRANkiJ2GSPVMkRqZE6YeSWPJAn8mzcG4/Gi/E6jmaMycw6+QHj7QsTlJ7x</latexit>
Revised: +UtphhHU1o+n7WwlYn2YGOtD1vuVwvHq0V3vbh2aOrYJT2NkgWySFaISzbINtkjB6REOLkmt+SePFg31p31aD31RnNWf2eO/JD18gnHhZw1</latexit> b Y opt (d) Feasible area for <latexit sha1_base64="mHHkqPE2HpxMv4zowGCwFJVfRYA=">AAACBXicbZDJSgNBEIZ73I1b1KOXxgVFMMyIqLkFVPCoYFxIYujpqdHGnoXuGjWMc/Dko3gSFMSrL+DNk29jZxLE7YeGj7+qqK7fjaXQaNsfVk9vX//A4NBwYWR0bHyiODl1qKNEcajySEbq2GUapAihigIlHMcKWOBKOHIvttr1o0tQWkThAbZiaATsLBS+4AyN1SzO1K+EByikB+lJ1kzrCNeYijDLTpebxTm7ZOeif8HpwlxlfsVfvNl522sW3+texJMAQuSSaV1z7BgbKVMouISsUE80xIxfsDOoGQxZALqR5kdkdME4HvUjZV6INHe/T6Qs0LoVuKYzYHiuf9fa5n+1WoL+ZsOcFCcIIe8s8hNJMaLtRKgnFHCULQOMK2H+Svk5U4yjya1QyFMo56Id2FjrQtn5SuFwteSsl9b2TRzbpKMhMkNmyRJxyAapkF2yR6qEk1tyTx7Jk3VnPVjP1kuntcfqzkyTH7JePwFkFJxC</latexit> e

Original: [Figure]
Revised: Y ⇤ in

Original: 4.2. Classiﬁcation with Stochastic Prediction
Revised: 4.2. Classiﬁcation with Stochastic Prediction

Original: In this section, we consider cases when stochastic prediction is acceptable, namely, the classiﬁer would output class labels with certain probabilities.
Revised: In this section, we consider cases when stochastic prediction is acceptable, namely, the classiﬁer would output class labels with certain probabilities.

Original: Among different categories of approaches to derive a fair classiﬁer, recent efforts to impose Equalized Odds in the pre-processing manner (Madras et al., 2018; Zhao et al., 2020; Tan et al., 2020) approach the problem from a representation learning perspective, where the main focus is to learn fair representations that at the same time preserve sufﬁcient information from the original data.
Revised: Among different categories of approaches to derive a fair classiﬁer, recent efforts to impose Equalized Odds in the pre-processing manner (Madras et al., 2018; Zhao et al., 2020; Tan et al., 2020) approach the problem from a representation learning perspective, where the main focus is to learn fair representations that at the same time preserve sufﬁcient information from the original data.

Original: Therefore, we omit pre-processing approaches from the discussion and focus on post-processing and in-processing fair classiﬁers.
Revised: Therefore, we omit pre-processing approaches from the discussion and focus on post-processing and in-processing fair classiﬁers.

Original: We ﬁrst derive the relation between positive rates (TPR and FPR) of binary classiﬁers before and after the post-processing step, i.e., (cid:98)
Revised: We ﬁrst derive the relation between positive rates (TPR and FPR) of binary classiﬁers before and after the post-processing step, i.e., (cid:98)

Original: Y opt (the unconstrainedly optimized classiﬁer) and (cid:101)
Revised: Y opt (the unconstrainedly optimized classiﬁer) and (cid:101)

Original: Y post (the fair classiﬁer derived by post-processing (cid:98)
Revised: Y post (the fair classiﬁer derived by post-processing (cid:98)

Original: Y opt ), and show that under mild assumptions, one can always derive a non-trivial Equalized Odds (cid:101)
Revised: Y opt ), and show that under mild assumptions, one can always derive a non-trivial Equalized Odds (cid:101)

Original: Y post via a post-processing step.
Revised: Y post via a post-processing step.

Original: Then, from the
Revised: Then, from the

Original: ROC feasible area perspective, we prove that post-processing approaches are actually equivalent to in-processing approaches but with additional “pseudo” constraints enforced.
Revised: ROC feasible area perspective, we prove that post-processing approaches are actually equivalent to in-processing approaches but with additional “pseudo” constraints enforced.

Original: Therefore, using the same loss function, post-processing approaches can perform no better than in-processing approaches.
Revised: Therefore, using the same loss function, post-processing approaches can perform no better than in-processing approaches.

Original: 4.2.1. T HE P OST - PROCESSING S TEP
Revised: 4.2.1. T HE P OST - PROCESSING S TEP

Original: The post-processing step (Hardt et al., 2016) of a predictor (cid:98)
Revised: The post-processing step (Hardt et al., 2016) of a predictor (cid:98)

Original: Y (here we drop the subscript when there is no ambiguity) only utilizes the information in the joint distribution ( A, Y, (cid:98)
Revised: Y (here we drop the subscript when there is no ambiguity) only utilizes the information in the joint distribution ( A, Y, (cid:98)

Original: Y ) .
Revised: Y ) .

Original: A fair predictor (cid:101)
Revised: A fair predictor (cid:101)

Original: Y post derived via a post-processing step is then fully speciﬁed by a (possibly randomized) function of ( A, (cid:98)
Revised: Y post derived via a post-processing step is then fully speciﬁed by a (possibly randomized) function of ( A, (cid:98)

Original: Y ) .
Revised: Y ) .

Original: This implies the conditional independence (cid:101)
Revised: This implies the conditional independence (cid:101)

Original: [Equation]
Revised: [Equation]

Original: (cid:98)
Revised: (cid:98)

Original: Y .
Revised: [Equation]

Original: Then if we denote the positive rates of (cid:98)
Revised: Then if we denote the positive rates of (cid:98)

Original: Y as P (cid:98) Y | AY (1 | a, y ) , positive rates of (cid:101)
Revised: Y as P (cid:98) Y | AY (1 | a, y ) , positive rates of (cid:101)

Original: Y post as P (cid:101)
Revised: Y post as P (cid:101)

Original: Y post | AY (1 | a, y ) , we can factorize the positive rates under this conditional independence: P (cid:101)
Revised: Y post | AY (1 | a, y ) , we can factorize the positive rates under this conditional independence: P (cid:101)

Original: Y post | AY (1 | a, y ) =
Revised: Y post | AY (1 | a, y ) =

Original: (cid:80) ˆ y ∈Y β (ˆ y ) a P (cid:98) Y
Revised: (cid:80) ˆ y ∈Y β (ˆ y ) a P (cid:98) Y

Original: | AY (ˆ y | a, y ) , where β (ˆ y )
Revised: | AY (ˆ y | a, y ) , where β (ˆ y )

Original: a := P ( (cid:101)
Revised: a := P ( (cid:101)

Original: Y post = 1 |
Revised: Y post = 1 |

Original: A = a, (cid:98)
Revised: A = a, (cid:98)

Original: Y = ˆ y ) .
Revised: Y = ˆ y ) .

Original: Therefore, the post-processing step boils down to optimizing parameters (for discrete A ) or functions (for continuous A ) β
Revised: Therefore, the post-processing step boils down to optimizing parameters (for discrete A ) or functions (for continuous A ) β

Original: (ˆ y ) a .
Revised: (ˆ y ) a .

Original: 4.2.2. ROC F EASIBLE A REA
Revised: 4.2.2. ROC F EASIBLE A REA

Original: On the Receiver Operator Characteristic (ROC) plane, a two-dimensional plane with horizontal axis denoting FPR and vertical axis denoting TPR, the performance of any binary predictor (cid:98)
Revised: On the Receiver Operator Characteristic (ROC) plane, a two-dimensional plane with horizontal axis denoting FPR and vertical axis denoting TPR, the performance of any binary predictor (cid:98)

Original: Y (not necessarily a fair one) with a certain value of protected feature A = a corresponds to a point γ a ( 
Revised: Y (not necessarily a fair one) with a certain value of protected feature A = a corresponds to a point γ a ( (cid:98)

Original: [Equation]
Revised: Y ) =

Original: = (FPR, TPR) on the plane.
Revised: (FPR, TPR) on the plane.

Original: Denote each coordinate according to the value of Y as γ ay ( 
Revised: Denote each coordinate according to the value of Y as γ ay ( [Equation] Y [Equation]

Revised: [Equation]

Revised: a

Original: [Equation]
Revised: [Equation]

Original: Further denote the corresponding convex hull of (cid:98)
Revised: Further denote the corresponding convex hull of (cid:98)

Original: Y on the ROC plane as C a (
Revised: Y on the ROC plane as C a (

Original: [Equation]
Revised: [Equation]

Original:  ) using vertices:
Revised: Y ) using vertices:

Original: [Equation]
Revised: [Equation]

Original: and then, as already stated in Hardt et al.
Revised: and then, as already stated in Hardt et al.

Original: (2016), the (FPR, TPR) pair corresponding to a postprocessing predictor falls within (including the boundary of) C a ( (cid:98)
Revised: (2016), the (FPR, TPR) pair corresponding to a postprocessing predictor falls within (including the boundary of) C a ( (cid:98)

Original: Y ) .
Revised: Y ) .

Original: Deﬁnition 5 (ROC feasible area)
Revised: Deﬁnition 5 (ROC feasible area)

Original: The feasible area of a predictor Ω( (cid:98)
Revised: The feasible area of a predictor Ω( (cid:98)

Original: Y ) , speciﬁed by the hypothesis space of available predictors (cid:98)
Revised: Y ) , speciﬁed by the hypothesis space of available predictors (cid:98)

Original: Y , is the set containing all attainable (FPR, TPR) pairs by the predictor on the ROC plane satisfying Equalized Odds.
Revised: Y , is the set containing all attainable (FPR, TPR) pairs by the predictor on the ROC plane satisfying Equalized Odds.

Original: Following Hardt et al.
Revised: Following Hardt et al.

Original: , we analyze the relation between the (FPR, TPR) pair of predictors on the ROC plane and formally establish the existence of the non-trivial Equalized Odds predictor.
Revised: , we analyze the relation between the (FPR, TPR) pair of predictors on the ROC plane and formally establish the existence of the non-trivial Equalized Odds predictor.

Original: As we show in Theorem 6, under mild assumptions an Equalized Odds predictor (cid:101)
Revised: As we show in Theorem 6, under mild assumptions an Equalized Odds predictor (cid:101)

Original: Y post derived via post-processing (cid:98)
Revised: Y post derived via post-processing (cid:98)

Original: Y always has non-empty ROC feasible area.
Revised: Y always has non-empty ROC feasible area.

Original: Theorem 6 ( Attainability of Equalized Odds for Classiﬁcation with Stochastic Prediction )
Revised: Theorem 6 ( Attainability of Equalized Odds for Classiﬁcation with Stochastic Prediction )

Original: Assume that the feature X is not independent from Y , and that (cid:98)
Revised: Assume that the feature X is not independent from Y , and that (cid:98)

Original: Y is a function of A and X .
Revised: Y is a function of A and X .

Original: Then for binary classiﬁcation, if (cid:98)
Revised: Then for binary classiﬁcation, if (cid:98)

Original: Y is a non-trivial predictor for Y , there is always at least one non-trivial predictor (cid:101)
Revised: Y is a non-trivial predictor for Y , there is always at least one non-trivial predictor (cid:101)

Original: Y post derived by post-processing (cid:98)
Revised: Y post derived by post-processing (cid:98)

Original: Y that can attain Equalized Odds, i.e., Ω( (cid:101)
Revised: Y that can attain Equalized Odds, i.e., Ω( (cid:101)

Original: Y post ) (cid:54) = ∅ .
Revised: Y post ) (cid:54) = ∅ .

Original: Here (cid:101)
Revised: Here (cid:101)

Original: Y post is a possibly randomized function of only A and (cid:98)
Revised: Y post is a possibly randomized function of only A and (cid:98)

Original: Y , trading off TPR with FPR across groups with different value of protected feature.
Revised: Y , trading off TPR with FPR across groups with different value of protected feature.

Original: From the panels (a) and (b) of Figure 1 we can also see that Ω( (cid:101)
Revised: From the panels (a) and (b) of Figure 1 we can also see that Ω( (cid:101)

Original: Y post ) , the ROC feasible area of (cid:101)
Revised: Y post ) , the ROC feasible area of (cid:101)

Original: Y post , is the intersection of Ω a ( (cid:98)
Revised: Y post , is the intersection of Ω a ( (cid:98)

Original: Y ) , indicating that although Equalized Odds is attained, the performance of (cid:101)
Revised: Y ) , indicating that although Equalized Odds is attained, the performance of (cid:101)

Original: Y post is no better and often worse than the weakest performance across different groups, which is obviously suboptimal.
Revised: Y post is no better and often worse than the weakest performance across different groups, which is obviously suboptimal.

Original: 4.2.3. O PTIMALITY OF P ERFORMANCE AMONG F AIR C LASSIFIERS
Revised: 4.2.3. O PTIMALITY OF P ERFORMANCE AMONG F AIR C LASSIFIERS

Original: In this subsection we discuss the optimality of performance of fair classiﬁers derived via in-processing and post-processing approaches.
Revised: In this subsection we discuss the optimality of performance of fair classiﬁers derived via in-processing and post-processing approaches.

Original: Let us consider a more general setting for the post-processing approach, where the optimal predictor to be post-processed is stochastic (while most previous approaches use deterministic predictors).
Revised: Let us consider a more general setting for the post-processing approach, where the optimal predictor to be post-processed is stochastic (while most previous approaches use deterministic predictors).

Original: We can derive the in-processing fair predictor 
Revised: We can derive the in-processing fair predictor (cid:101) Y in : min f ∈F E [ L ( [Equation]

Revised: t

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: and the unconstrained statistical optimal predictor
Revised: and the unconstrained statistical optimal predictor

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: It is natural to wonder, now that one can always directly solve for (cid:101)
Revised: It is natural to wonder, now that one can always directly solve for (cid:101)

Original: Y in from Equation 8, how it is related to (cid:101)
Revised: Y in from Equation 8, how it is related to (cid:101)

Original: Y post , which is derived by post-processing the (cid:98)
Revised: Y post , which is derived by post-processing the (cid:98)

Original: Y opt solved from Equation 9?
Revised: Y opt solved from Equation 9?

Original: Interestingly, although (cid:101)
Revised: Interestingly, although (cid:101)

Original: Y in and (cid:101)
Revised: Y in and (cid:101)

Original: Y post are solved separately using different constrained optimization schemes, one can draw a connection between them by utilizing (cid:98)
Revised: Y post are solved separately using different constrained optimization schemes, one can draw a connection between them by utilizing (cid:98)

Original: Y opt as a bridge and reason about the relation between their ROC feasible areas Ω( (cid:101)
Revised: Y opt as a bridge and reason about the relation between their ROC feasible areas Ω( (cid:101)

Original: Y in ) and Ω( (cid:101)
Revised: Y in ) and Ω( (cid:101)

Original: Y post ) , as we summarize in the following theorem.
Revised: Y post ) , as we summarize in the following theorem.

Original: Theorem 7 ( Equivalence between ROC feasible areas )
Revised: Theorem 7 ( Equivalence between ROC feasible areas )

Original: Let Ω( (cid:101)
Revised: Let Ω( (cid:101)

Original: Y post ) denote the ROC feasible area speciﬁed by the constraints enforced on (cid:101)
Revised: Y post ) denote the ROC feasible area speciﬁed by the constraints enforced on (cid:101)

Original: Y post .
Revised: Y post .

Original: Then Ω( (cid:101)
Revised: Then Ω( (cid:101)

Original: Y post ) is identical to the ROC feasible area Ω( (cid:101)
Revised: Y post ) is identical to the ROC feasible area Ω( (cid:101)

Original: Y ∗ in ) that is speciﬁed by the following set of constraints:
Revised: Y ∗ in ) that is speciﬁed by the following set of constraints:

Original: (i) constraints enforced on
Revised: (i) constraints enforced on

Original: [Equation]
Revised: [Equation]

Original:  (ii) additional “pseudo” constraints: ∀a ∈
Revised: ; (ii) additional “pseudo” constraints: ∀

Original: A
Revised: a

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: A

Original: [Equation]
Revised: [Equation]

Original: (0) a 1 , β (1) a
Revised: [Equation] [Equation] [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: β (1) a 1 , where
Revised: 1 , where

Original: [Equation]
Revised: [Equation]

Original: As we can see in Figure 1(c) and (d), if the additional “pseudo” constraints are introduced when optimizing (cid:101)
Revised: As we can see in Figure 1(c) and (d), if the additional “pseudo” constraints are introduced when optimizing (cid:101)

Original: Y ∗ in , we have Ω( (cid:101)
Revised: Y ∗ in , we have Ω( (cid:101)

Original: Y in ) ⊇ Ω( (cid:101)
Revised: Y in ) ⊇ Ω( (cid:101)

Original: Y post ) =
Revised: Y post ) =

Original: Ω( (cid:101)
Revised: Ω( (cid:101)

Original: Y ∗ in ) .
Revised: Y ∗ in ) .

Original: Therefore, with the same objective function and fairness constraint, the fair classiﬁer derived from an in-processing approach always outperforms (or performs equally well with) the one derived from a post-processing approach.
Revised: Therefore, with the same objective function and fairness constraint, the fair classiﬁer derived from an in-processing approach always outperforms (or performs equally well with) the one derived from a post-processing approach.

Original: 5. Experiments
Revised: 5. Experiments

Original: In this section, we provide numerical results for various settings.
Revised: In this section, we provide numerical results for various settings.

Original: To demonstrate the beneﬁt of stochastic prediction with respect to imposing fairness, we present the results for regression with stochastic prediction on both simulated data and the real-world Communities and Crime data set.
Revised: To demonstrate the beneﬁt of stochastic prediction with respect to imposing fairness, we present the results for regression with stochastic prediction on both simulated data and the real-world Communities and Crime data set.

Original: For classiﬁcation tasks we compare the performance of several existing methods in the literature on multiple real-world data sets.
Revised: For classiﬁcation tasks we compare the performance of several existing methods in the literature on multiple real-world data sets.

Original: Please see Section B of the appendix for additional results, as well as the detailed description of the data sets and technical speciﬁcation for the experiments.
Revised: Please see Section B of the appendix for additional results, as well as the detailed description of the data sets and technical speciﬁcation for the experiments.

Original: In Figure 2 we compare deterministic and stochastic regression on simulated linear non-Gaussian data.
Revised: In Figure 2 we compare deterministic and stochastic regression on simulated linear non-Gaussian data.

Original: Panel (a) illustrates the trade-off between fairness in terms of the kernel measure of conditional dependence (KMCD)
Revised: Panel (a) illustrates the trade-off between fairness in terms of the kernel measure of conditional dependence (KMCD)

Original: (Fukumizu et al., 2004, 2007), and prediction error in terms of the mean squared error (MSE) for different values of the hyperparameter λ .
Revised: (Fukumizu et al., 2004, 2007), and prediction error in terms of the mean squared error (MSE) for different values of the hyperparameter λ .

Original: Panel (b) summarizes the p-value outputs of the kernel-based conditional independence (KCI) test (Zhang et al., 2011).
Revised: Panel (b) summarizes the p-value outputs of the kernel-based conditional independence (KCI) test (Zhang et al., 2011).

Original: Note that in (b), the green boxes are not visible since p-values in the deterministic prediction case are always close to 0 .
Revised: Note that in (b), the green boxes are not visible since p-values in the deterministic prediction case are always close to 0 .

Original: The distribution of the p-value outputs clearly indicate that stochastic prediction often achieve Equalized Odds (the test fails to reject the null quite often), while deterministic prediction may not (the test rejects the null almost all the time).
Revised: The distribution of the p-value outputs clearly indicate that stochastic prediction often achieve Equalized Odds (the test fails to reject the null quite often), while deterministic prediction may not (the test rejects the null almost all the time).

Original: For panel (c) and (d), we ﬁx the training sample size at 200 while the sample size for testing ranges from 100 to 3000 .
Revised: For panel (c) and (d), we ﬁx the training sample size at 200 while the sample size for testing ranges from 100 to 3000 .

Original: 3 A practical example of the scenario would be an automated hiring system to aid recruitment (each future applicant is a new data point for the system as long as the system is deployed and keeps running).
Revised: 3 A practical example of the dramatically with large sample size, we use linearly correlated data with relatively small sample size for training and consider increasingly large test sets in our repetitive experiments (with ﬁxed hyperparameter λ ). [Figure] [Figure] [Figure]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: /O1Nxf+8dqy8k+6YBVGsaEDSh7yYIxWi6d+RywQlio80wUQwvSsiAywwUboh00xaKCdAKSkVZ6Rsf7XQOCrYx4VizcpXKteQIAs7sA+HYEMJKnAOVagDgSHcwj08GJfGnfFoPKXRjJHesA0/YDx/Ai7tlFo

Revised: [Figure]

Revised: </

Revised: [Figure]

Revised: <latexit

Revised: [Figure]

Revised: [Figure]

Revised: yKTwZ1GJhF0iPDJakT

Revised: [Figure]

Original: As we can see in panel (d), while the p-value for deterministic prediction is always close to 0 (invisible green box, which indicates violated Equalized Odds), the p-value for stochastic prediction spreads over [0 , 1] if the test set sample size is not too big compared to the training data.
Revised: Odds. The data is linear non-Gaussian, and the predictor is a neural network (nonlinear) regressor with deterministic or stochastic output. The green boxes in panel (b) and (d) are barely visible, meaning that p-values for deterministic prediction are always close to 0 . [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Table] [Figure]

Original: [Figure]
Revised: scenario would be an automated hiring system to aid recruitment (each future applicant is a new data point for the system as long as the system is deployed and keeps running). As we can see in panel (d), while the p-value for deterministic prediction is always close to 0 (invisible green box, which indicates violated Equalized Odds), the p-value for stochastic prediction spreads over [0 , 1] if the test set sample size is not too big compared to the training data.

Original: In Figure 3 we compare deterministic and stochastic regression on the Communities and Crime data set.
Revised: In Figure 3 we compare deterministic and stochastic regression on the Communities and Crime data set.

Original: Compared to Figure 2(b), although the green boxes (which correspond to the distribution of p-values for the deterministic predictor) are visible in Figure 3(d), we can still observe that the distribution of p-values for the stochastic predictor dominates its counterpart for deterministic predictor, indicating a higher possibility of attaining Equalized Odds.
Revised: Compared to Figure 2(b), although the green boxes (which correspond to the distribution of p-values for the deterministic predictor) are visible in Figure 3(d), we can still observe that the distribution of p-values for the stochastic predictor dominates its counterpart for deterministic predictor, indicating a higher possibility of attaining Equalized Odds.

Original: Compare the KCI test outcome of the linear stochastic model shown in panel (b) with that of the nonlinear stochastic model (implemented with the neural network) shown in panel (d), we can see that the distribution of p-values in panel (b) for stochastic prediction (with a linear base model) does not dominate its deterministic counterpart.
Revised: Compare the KCI test outcome of the linear stochastic model shown in panel (b) with that of the nonlinear stochastic model (implemented with the neural network) shown in panel (d), we can see that the distribution of p-values in panel (b) for stochastic prediction (with a linear base model) does not dominate its deterministic counterpart.

Original: This is not surprising since if the deterministic part of the linear model does not satisfy Equalized Odds, adding an independent noise will not help impose fairness.
Revised: This is not surprising since if the deterministic part of the linear model does not satisfy Equalized Odds, adding an independent noise will not help impose fairness.

Original:  In Figure 4, we compare the performance under Equalized Odds of multiple methods proposed in the literature.
Revised: [Figure] In Figure 4, we compare the performance under Equalized Odds of multiple methods proposed in the literature.

Original: Although a probabilistic classiﬁcation model is used for across each method (logistic regression here), if an algorithm outputs the class label where the prediction likelihood is maximized, the prediction is in essence deterministic.
Revised: Although a probabilistic classiﬁcation model is used for across each method (logistic regression here), if an algorithm outputs the class label where the prediction likelihood is maximized, the prediction is in essence deterministic.

Original: Therefore, although here we are considering ﬁnite data cases, we can still anticipate a lower level of fairness violation with stochastic prediction.
Revised: Therefore, although here we are considering ﬁnite data cases, we can still anticipate a lower level of fairness violation with stochastic prediction.

Original: This is validated by the numerical experiment: while the post-processing approach by Hardt et al.
Revised: This is validated by the numerical experiment: while the post-processing approach by Hardt et al.

Original: (2016) does not score the lowest test error, the violation of Equalized Odds is the lowest compared to other approaches.
Revised: (2016) does not score the lowest test error, the violation of Equalized Odds is the lowest compared to other approaches.

Original: [Figure]

Original: 6. Further Discussions
Revised: 6. Further Discussions

Original: In this section, we ﬁrst present an alternative way to perform post-processing for binary classiﬁcation, and discuss how our results would apply to this speciﬁc post-processing strategy; we then discuss the relation between our ﬁndings and previous theoretical results for Equalized Odds in the literature.
Revised: In this section, we ﬁrst present an alternative way to perform post-processing for binary classiﬁcation, and discuss how our results would apply to this speciﬁc post-processing strategy; we then discuss the relation between our ﬁndings and previous theoretical results for Equalized Odds in the literature.

Original: 6.1. Alternative Strategy for Post-processing
Revised: 6.1. Alternative Strategy for Post-processing

Original: When we discuss Equalized Odds fairness in binary classiﬁcation, we present the attainability and optimality results by considering one particular type of post-processing approach proposed by
Revised: When we discuss Equalized Odds fairness in binary classiﬁcation, we present the attainability and optimality results by considering one particular type of post-processing approach proposed by

Original: Hardt et al.
Revised: Hardt et al.

Original: (2016), where (cid:101)
Revised: (2016), where (cid:101)

Original: Y post is derived by ﬁrst training (cid:98)
Revised: Y post is derived by ﬁrst training (cid:98)

Original: Y without fairness constraints and then adjusting the output in a post-hoc way (also referred to as the “two-step framework” in Woodworth et al.
Revised: Y without fairness constraints and then adjusting the output in a post-hoc way (also referred to as the “two-step framework” in Woodworth et al.

Original: In the literature, the term “post-processing” is also used to describe the prediction mechanism where the classiﬁcation result is derived from thresholding a non-dichotomized predictor, e.g., the Bayes optimal regressor R (Hardt et al., 2016).
Revised: In the literature, the term “post-processing” is also used to describe the prediction mechanism where the classiﬁcation result is derived from thresholding a non-dichotomized predictor, e.g., the Bayes optimal regressor R (Hardt et al., 2016).

Original: If the classiﬁer (cid:101)
Revised: If the classiﬁer (cid:101)

Original: Y post is derived from post-processing the non-dichotomized R instead of the binary (cid:98)
Revised: Y post is derived from post-processing the non-dichotomized R instead of the binary (cid:98)

Original: Y , in general the conditional independence (cid:101)
Revised: Y , in general the conditional independence (cid:101)

Original: Y post ⊥⊥ Y | A, (cid:98)
Revised: [Equation] (cid:98)

Original: Y no longer holds true since the non-dichotomized R contains more information about Y than (cid:98)
Revised: Y no longer holds true since the non-dichotomized R contains more information about Y than (cid:98)

Original: Y .
Revised: Y .

Original: That being said, the attainability of
Revised: That being said, the attainability of

Original: Equalized Odds for post-processing is still guaranteed, in other words, the result of Theorem 6 holds true even for this alternative post-processing strategy.
Revised: Equalized Odds for post-processing is still guaranteed, in other words, the result of Theorem 6 holds true even for this alternative post-processing strategy.

Original: We can construct a proof similar to that for
Revised: We can construct a proof similar to that for

Original: Theorem 3 (the attainability result for regression tasks) but with a discrete domain Y .
Revised: Theorem 3 (the attainability result for regression tasks) but with a discrete domain Y .

Original: Speciﬁcally, the modiﬁcations of the proof include (1) changing integrations with respect to Y to summations, and (2) replacing conditional densities p (cid:101) Y | AX (˜ y | a, x ) with the corresponding conditional probabilities
Revised: Speciﬁcally, the modiﬁcations of the proof include (1) changing integrations with respect to Y to summations, and (2) replacing conditional densities p (cid:101) Y | AX (˜ y | a, x ) with the corresponding conditional probabilities

Original: P (cid:101) Y | AX (˜ y | a, x ) .
Revised: P (cid:101) Y | AX (˜ y | a, x ) .

Original: Besides, if we post-process the Bayes optimal regressor R to derive (cid:101)
Revised: Besides, if we post-process the Bayes optimal regressor R to derive (cid:101)

Original: Y post for binary classiﬁcation, the optimality of performance can also be guaranteed.
Revised: Y post for binary classiﬁcation, the optimality of performance can also be guaranteed.

Original: It is shown by Hardt et al.
Revised: It is shown by Hardt et al.

Original: (2016, Proposition 5.2) that the Bayes optimal Equalized Odds binary classiﬁer (cid:101)
Revised: Proposition 5.2) that the Bayes optimal Equalized Odds binary classiﬁer (cid:101)

Original: Y post can be derived by post-processing the Bayes optimal regressor R .
Revised: Y post can be derived by post-processing the Bayes optimal regressor R .

Original: The optimality of such classiﬁersbased on post-processing R does not contradict ouroptimality result based on post-processing (cid:98)
Revised: The optimality of such classiﬁers does not contradict our result based on post-processing (cid:98)

Original: Y that in-processing approaches can perform better or at least as good as post-processing ones, i.e.,

Original: Theorem 7, and the results can be viewed as complementary to each other.
Revised: Y and the results can be viewed as complementary to each other.

Original: 6.2. Discussion on Previous Theoretical Results for Equalized Odds in Binary Classiﬁcation
Revised: 6.2. Discussion on Previous Theoretical Results for Equalized Odds in Binary Classiﬁcation

Original: Apart from various methods to empirically impose [Table]
Revised: Apart from various methods to empirically impose Equalized Odds (Hardt et al., 2016; Woodworth et al., 2017; Agarwal et al., 2018; Mary et al., 2019; Baharlouei et al., 2020; Rezaei et al., 2020;

Original: Romano et al., 2020), there are also theoretical considerations with respect to this speciﬁc notion of fairness.
Revised: Romano et al., 2020), there are also theoretical considerations with respect to this speciﬁc notion of fairness.

Original: In particular, for binary classiﬁcation tasks, Hardt et al.
Revised: In particular, for binary classiﬁcation tasks, Hardt et al.

Original: (2016, Proposition 5.2) show that if we have access to the Bayes optimal regressor R , we can derive the Bayes optimal Equalized Odds binary classiﬁer by post-processing R ; Woodworth et al.
Revised: (2016, Proposition 5.2) show that if we have access to the Bayes optimal regressor R , we can derive the Bayes optimal Equalized Odds binary classiﬁer by post-processing R ; Woodworth et al.

Original: (2017, Theorem 7) show that for a binary classiﬁer hypothesis class, one can derive a guarantee regarding the statistical learnability of (if there exists) the best Equalized Odds predictor within the class using the two-step framework.
Revised: (2017, Theorem 7) show that for a binary classiﬁer hypothesis class, one can derive a guarantee regarding the statistical learnability of (if there exists) the best Equalized Odds predictor within the class using the two-step framework.

Original: We can see that previous theoretical results for Equalized Odds require strong assumptions: the result by Hardt et al.
Revised: Previous theoretical results for Equalized Odds require strong assumptions: the result by Hardt et al.

Original: (2016, Proposition 5.2) requires the availability of the Bayes optimal regressor R ; the concentration bounds  demonstrated by Woodworth et al.
Revised: (2016, Proposition 5.2) requires the availability of the Bayes optimal regressor R ; the concentration bounds (on both accuracy and fairness violation) demonstrated by Woodworth et al.

Original: (2017, Theorem 7) (on both accuracy and Equalized Odds violation) assumes the existence of the best non-discriminatory binary classiﬁer within the hypothesis class.
Revised: Theorem 7) assume the existence of the best non-discriminatory binary classiﬁer within the hypothesis class.

Original: Instead of assuming the availability of Bayes optimal regressor (as compared to Hardt et al.
Revised: Instead of assuming the availability of Bayes optimal regressor (as compared to Hardt et al.

Original: (2016, Proposition 5.2)) or the existence of a perfect fair predictor (as compared to Woodworth et al.
Revised: (2016, Proposition 5.2)) or the existence of a perfect fair predictor (as compared to Woodworth et al.

Original: (2017, Theorem 7)), we give an afﬁrmative answer to the necessary and sufﬁcient condition under which Equalized Odds can possibly hold true (for deterministic predictors) and also a guarantee on the attainability of Equalized Odds under rather mild assumptions (for stochastic predictors).
Revised: (2017, Theorem 7)), we give an afﬁrmative answer to the necessary and sufﬁcient condition under which Equalized Odds can possibly hold true (for deterministic predictors) and also a guarantee on the attainability of Equalized Odds under rather mild assumptions (for stochastic predictors).

Original: Our theoretical analysis regarding the attainability of Equalized Odds is actually the missing piece in the puzzle, which, together with previous theoretical results by Hardt et al.
Revised: Our theoretical analysis regarding the attainability of Equalized Odds is actually the missing piece in the puzzle, which, together with previous theoretical results by Hardt et al.

Original: (2016, Proposition 5.2) and Woodworth et al.
Revised: (2016, Proposition 5.2) and Woodworth et al.

Original: (2017, Theorem 7), gives us a much clearer picture regarding achieving
Revised: (2017, Theorem 7), gives us a much clearer picture regarding achieving

Original: Equalized Odds for binary classiﬁcation: the stochastic prediction scheme is preferable since the theoretical attainability guarantee for the perfect Equalized Odds in the large sample limit; one can also characterize the price paid in terms of accuracy when using the stochastic prediction scheme by analyzing the bound for classiﬁcation error and that for empirical fairness violation.
Revised: Equalized Odds for binary classiﬁcation: the stochastic prediction scheme is preferable since the theoretical attainability guarantee for the perfect Equalized Odds in the large sample limit; one can also characterize the price paid in terms of accuracy when using the stochastic prediction scheme by analyzing the bound for classiﬁcation error and that for empirical fairness violation.

Original: Furthermore, our attainability results apply not only to binary classiﬁcation tasks, but also to regression tasks.
Revised: Furthermore, our attainability results apply not only to binary classiﬁcation tasks, but also to regression tasks.

Original: 6.3. The Difﬁculty of Deriving Bounds for Accuracy and Fairness Violation in General Cases
Revised: 6.3. The Difﬁculty of Deriving Bounds for Accuracy and Fairness Violation in General Cases

Original: One limitation of our work is that we only consider the attainability of the perfect fairness and does not provide a lower bound of best achievable fairness violation when attainability is not guaranteed.
Revised: One limitation of our work is that we only consider the attainability of the perfect fairness and does not provide a lower bound of best achievable fairness violation when attainability is not guaranteed.

Original: However, as illustrated in Theorem 2 and Theorem 4 (the necessary and sufﬁcient conditions for
Revised: However, as illustrated in Theorem 2 and Theorem 4 (the necessary and sufﬁcient conditions for

Original: Equalized Odds to hold true with the deterministic prediction scheme), the violation of Equalized
Revised: Equalized Odds to hold true with the deterministic prediction scheme), the violation of Equalized

Original: Odds for deterministic predictors directly corresponds to the speciﬁc properties of the data distribution itself.
Revised: Odds for deterministic predictors directly corresponds to the speciﬁc properties of the data distribution itself.

Original: This indicates the difﬁculty of giving a tight universal distribution-free lower bound for fairness violation and accuracy in the large sample limit for deterministic predictors.
Revised: This indicates the difﬁculty of giving a tight universal distribution-free lower bound for fairness violation and accuracy in the large sample limit for deterministic predictors.

Original: As a comparison, for the stochastic prediction scheme our results (Theorem 3 and Theorem 6) show that in general we have theoretical attainability guarantees for Equalized Odds, i.e., a distribution-free lower bound (at zero) of fairness violation.
Revised: As a comparison, for the stochastic prediction scheme our results (Theorem 3 and Theorem 6) show that in general we have theoretical attainability guarantees for Equalized Odds, i.e., a distribution-free lower bound (at zero) of fairness violation.

Original: Only with this theoretical guanrantee can we apply the result by Woodworth et al.
Revised: Only with this theoretical guanrantee can we apply the result by Woodworth et al.

Original: (2017, Theorem 7) and analyze concentration bounds for binary classiﬁcation.
Revised: (2017, Theorem 7) and analyze concentration bounds for binary classiﬁcation.

Original: In fact, the difﬁculty of theoretically deriving (distribution-free or distribution-dependent) bounds in general cases, especially for tasks other than binary classiﬁcation, stems from the following aspects: the quantiﬁcation of Equalized Odds violation, and the explicit expression of the tradeoff between accuracy and fairness.
Revised: In fact, the difﬁculty of theoretically deriving (distribution-free or distribution-dependent) bounds in general cases, especially for tasks other than binary classiﬁcation, stems from the following aspects: the quantiﬁcation of fairness violation, and the exact mathematical form of the trade-off between accuracy and fairness.

Original: To begin with, the quantiﬁcation of Equalized Odds violation could be involved beyond binary classiﬁcation.
Revised: To begin with, the quantiﬁcation of fairness violation could be involved beyond binary classiﬁcation.

Original: For binary classiﬁcation tasks, we can conveniently quantify Equalized Odds violation via TPR(s) and FPR(s) and connect the fairness violation back to distribution speciﬁcs (e.g., the joint distribution of ( A, Y ) ), but this is not the case for multiclass classiﬁcation or regression tasks.
Revised: For binary classiﬁcation tasks, we can conveniently quantify Equalized Odds violation via TPR(s) and FPR(s) that directly link back to distribution speciﬁcs (e.g., the joint distribution of ( A, Y ) ), but this is not the case for multiclass classiﬁcation or regression tasks.

Original: A natural choice of fairness violation quantiﬁcation beyond binary classiﬁcation is, for example, the (normalized) kernel measure of conditional dependence (KMCD)
Revised: A natural choice of fairness violation quantiﬁcation beyond binary classiﬁcation is, for example, the (normalized) kernel measure of conditional dependence (KMCD).

Original: (Fukumizu et al., 2004, 2007).

Original: However, it is not intuitive how to effectively ﬁnd
Revised: However, it is not intuitive how to effectively ﬁnd

Original: a preimage of the KMCD operator and translate the fairness violation in terms of distributional properties of the data.
Revised: a preimage of the KMCD operator and translate the fairness violation in terms of distributional properties of the data.

Original: Furthermore, although we can empirically draw trade-off curves, it is not transparent what would be the explicit mathematical expression of the trade-off (in the large sample limit) between accuracy and fairness violation.
Revised: Furthermore, although we can empirically draw trade-off curves, it is not transparent what would be the explicit mathematical expression of the trade-off (in the large sample limit) between accuracy and fairness violation.

Original: The explicit expression of the trade-off is essential if we were to, for example, compare the (empirically) best possible deterministic predictor with its stochastic counterpart and derive a theoretical bound.
Revised: The explicit expression of the trade-off is essential if we were to, for example, compare the (empirically) best possible deterministic predictor with its stochastic counterpart and derive a theoretical bound.

Original: 6.4. Potential Social Impact of Stochastic Prediction
Revised: 6.4. Potential Social Impact of Stochastic Prediction

Original: Under the scope of the broader social impact, the stochastic prediction to impose a group-level fairness like Equalized Odds may subject to unwanted usage, which may result in deterministic (therefore in general unfair) prediction in the end.
Revised: Under the scope of the broader social impact, the stochastic prediction to impose a group-level fairness like Equalized Odds may subject to unwanted usage, which may result in deterministic (therefore in general unfair) prediction in the end.

Original: The introduction of randomness in the prediction may also be subject to concerns, especially in high-stake decision-makings.
Revised: The introduction of randomness in the prediction may also be subject to concerns, especially in high-stake decision-makings.

Original: In some way, this potentially negative effect roots from the fact that Equalized Odds is a group-level fairness criterion, i.e., not ﬁne-grained enough to characterize individual-level consequences.
Revised: In some way, this potentially negative effect roots from the fact that Equalized Odds is a group-level fairness criterion, i.e., not ﬁne-grained enough to characterize individual-level consequences.

Original: In high-stakes scenarios when the decisions received by individuals matter, we should dedicate to more ﬁne-grained fairness notions which go beyond group-level characterization of non discrimination.
Revised: In high-stakes scenarios when the decisions received by individuals matter, we should dedicate to more ﬁne-grained fairness notions which go beyond group-level characterization of non-discrimination.

Original: 7. Conclusion
Revised: 7. Conclusion

Original: In this paper, we focus on Equalized Odds and consider the attainability of fairness, and furthermore, if it is attainable, the optimality of the prediction performance under various settings.
Revised: In this paper, we focus on Equalized Odds and consider the attainability of fairness, and furthermore, if it is attainable, the optimality of the prediction performance under various settings.

Original: For both classiﬁcation and regression tasks, we present necessary and sufﬁcient conditions under which
Revised: For both classiﬁcation and regression tasks, we present necessary and sufﬁcient conditions under which

Original: Equalized Odds can hold true with deterministic prediction functions; we also present theoretical guarantees that under mild assumptions, one can always ﬁnd a non-trivial stochastic predictor that satisﬁes Equalized Odds in the large sample limit, while in general fairness is not attainable with a deterministic predictor.
Revised: Equalized Odds can hold true with deterministic prediction functions; we also present theoretical guarantees that under mild assumptions, one can always ﬁnd a non-trivial stochastic predictor that satisﬁes Equalized Odds in the large sample limit, while in general fairness is not attainable with a deterministic predictor.

Original: We hope that our results can incentivize further research into developing prediction strategies that come with theoretical guarantees.
Revised: We hope that our results can incentivize further research into developing prediction strategies that come with theoretical guarantees.

Original: Future work would naturally consider the attainability of more ﬁne-grained (compared to group fairness) criteria of fairness (e.g., individual fairness) as well as fairness notions that focus on discrimination involved in the data generating process.
Revised: Future work would naturally consider the attainability of more ﬁne-grained (compared to group fairness) criteria of fairness (e.g., individual fairness) as well as fairness notions that focus on discrimination involved in the data generating process.

Original: Acknowledgments
Revised: Acknowledgments

Original: We thank a bunch of people and funding agency.
Revised: KZ would like to acknowledge the support by the National Institutes of Health (NIH) under Contract R01HL159805, by the NSF-Convergence Accelerator Track-D award # and by the United States Air Force under Contract No. FA8650-17-C

Original: Appendix A. Proof for Theorems A.1. Proof for Theorem
Revised: Appendix A. Proof for Theorems A.1. Proof for Theorem

Original: To prove the unattainability of Equalized Odds in regression, we will need the following lemma, which provides a way to characterize conditional independence/dependence with conditional or joint distributions.
Revised: To prove the unattainability of Equalized Odds in regression, we will need the following lemma, which provides a way to characterize conditional independence/dependence with conditional or joint distributions.

Original: Lemma 8 Variables V 1 and V 2 are conditionally independent given variable V 

Revised: Lemma 8 Variables V 1 and V 2 are conditionally independent given variable V

Original: if and only if there exist functions h ( v 1 , v 3 ) and g ( v 2 , v 3 ) such that
Revised: if and only if there exist functions h ( v 1 , v 3 ) and g ( v 2 , v 3 ) such that

Original: [Equation]
Revised: [Equation]

Original: Proof First, if V 1 and V 2 are conditionally independent given variable V 3 , then Equation 10 holds:
Revised: Proof First, if V 1 and V 2 are conditionally independent given variable V 3 , then Equation 10 holds:

Original: [Equation]
Revised: [Equation]

Original: We then let ˜ h ( v 3 ) :
Revised: We then let ˜ h ( v 3 ) :

Original: = (cid:82) h ( v 1 , v 3 ) dv 1 and ˜ g ( v 3 ) :
Revised: = (cid:82) h ( v 1 , v 3 ) dv 1 and ˜ g ( v 3 ) :

Original: = (cid:82) g ( v 2 , v 3 ) dv 2 .
Revised: = (cid:82) g ( v 2 , v 3 ) dv 2 .

Original: Take the integral of Equationw.r.t.
Revised: Take the integral of Equationw.r.t.

Original: v 1 and v 2 , we have:
Revised: v 1 and v 2 , we have:

Original: [Equation]
Revised: [Equation]

Original: respectively.
Revised: respectively.

Original: Bearing in mind Equation 10, one can see that the product of the two equations above is
Revised: Bearing in mind Equation 10, one can see that the product of the two equations above is

Original: [Equation]
Revised: [Equation]

Original: Take the integral of the equation above w.r.t.
Revised: Take the integral of the equation above w.r.t.

Original: v 1 and v 2 gives ˜ h ( v 3 ) · ˜ g ( v 3 ) ≡ 1 .
Revised: v 1 and v 2 gives ˜ h ( v 3 ) · ˜ g ( v 3 ) ≡ 1 .

Original: The above equation then reduces to
Revised: The above equation then reduces to

Original: [Equation]
Revised: [Equation]

Original: That is, V 1 and V 2 are conditionally independent given V 3 .
Revised: That is, V 1 and V 2 are conditionally independent given V 3 .

Original: Now we are ready to prove the unattainability of Equalized Odds in linear non-Gaussian regression.
Revised: Now we are ready to prove the unattainability of Equalized Odds in linear non-Gaussian regression.

Original: Recall that the data is generated as follows ( H is not measured in the dataset):
Revised: Recall that the data is generated as follows ( H is not measured in the dataset):

Original: [Equation]
Revised: [Equation]

Original: where ( A, E X , E H , E Y ) are mutually independent and q, b, c, d are constants.
Revised: where ( A, E X , E H , E Y ) are mutually independent and q, b, c, d are constants.

Original: Theorem ( Unattainability of Equalized Odds in the Linear Non-Gaussian Case )
Revised: Theorem ( Unattainability of Equalized Odds in the Linear Non-Gaussian Case )

Original: Assume that X has a causal inﬂuence on Y , i.e., c (cid:54)
Revised: Assume that X has a causal inﬂuence on Y , i.e., c (cid:54)

Original: = 0 in Equation 11, and that A and Y are not independent, i.e., qc + bd (cid:54)
Revised: = 0 in Equation 11, and that A and Y are not independent, i.e., qc + bd (cid:54)

Original: = 0 .
Revised: = 0 .

Original: Assume p E X and p E are positive on R .
Revised: Assume p E X and p E are positive on R .

Original: Let f 1 :=
Revised: Let f 1 :=

Original: log p A , f 2 := log p
Revised: log p A , f 2 := log p

Original: E X , and f 3 := log p E .
Revised: E X , and f 3 := log p E .

Original: Further assume that f 2 and f 3 are third-order differentiable.
Revised: Further assume that f 2 and f 3 are third-order differentiable.

Original: Then if at most one of E X and E is Gaussian, (cid:98)
Revised: Then if at most one of E X and E is Gaussian, (cid:98)

Original: Y is always conditionally dependent on A given Y .
Revised: Y is always conditionally dependent on A given Y .

Original: Proof According to Equation
Revised: Proof According to Equation

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: The determinant of the above linear transformation is β , which relates the probability density function of the variables on the LHS and that of the variables on the RHS of the equation.
Revised: The determinant of the above linear transformation is β , which relates the probability density function of the variables on the LHS and that of the variables on the RHS of the equation.

Original: Therefore, according to Equation
Revised: Therefore, according to Equation

Original: [Equation]
Revised: [Equation]

Original: we can rewrite the joint probability density function by making use of the Jacobian determinant and factor the joint density into marginal density functions ( A , E X , E are mutually independent according to the data generating process).
Revised: we can rewrite the joint probability density function by making use of the Jacobian determinant and factor the joint density into marginal density functions ( A , E X , E are mutually independent according to the data generating process).

Original: Further let
Revised: Further let

Original: [Equation]
Revised: [Equation]

Original: Then we have E X = 1 β
Revised: Then we have E X = 1 β

Original: [Equation]
Revised: [Equation]

Original: , and
Revised: , and

Original: [Equation]
Revised: [Equation]

Original: On its support
Revised: On its support

Original: [Equation]
Revised: [Equation]

Original: the log -density can be written as
Revised: the log -density can be written as

Original: [Equation]
Revised: [Equation]

Original: According to Lemma 8, A ⊥⊥ (cid:98)
Revised: According to Lemma 8, A ⊥⊥ (cid:98)

Original: Y | Y
Revised: Y | Y

Original: if and only if p A, (cid:98) Y | Y ( a, ˆ y | y ) is a product of a function of a and y and a function of ˆ y and y .
Revised: if and only if p A, (cid:98) Y | Y ( a, ˆ y | y ) is a product of a function of a and y and a function of ˆ y and y .

Original: p A, (cid:98) Y ,Y ( a, ˆ y, y ) is further a product of the above function and a function of only y .
Revised: p A, (cid:98) Y ,Y ( a, ˆ y, y ) is further a product of the above function and a function of only y .

Original: This property, under the conditions in Theorem 1, is equivalent to the constraint
Revised: This property, under the conditions in Theorem 1, is equivalent to the constraint

Original: [Equation]
Revised: [Equation]

Original: According to Equation 14, we have
Revised: According to Equation 14, we have

Original: [Equation]
Revised: [Equation]

Original: and therefore
Revised: and therefore

Original: [Equation]
Revised: [Equation]

Original: Combining Equations 15 and 16 gives
Revised: Combining Equations 15 and 16 gives

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Further taking the partial derivative of both sides of the above equation w.r.t.
Revised: Further taking the partial derivative of both sides of the above equation w.r.t.

Original: y yields
Revised: y yields

Original: [Equation]
Revised: [Equation]

Original: There are three possible situations where the above equation holds: (i)
Revised: There are three possible situations where the above equation holds: (i)

Original: ˜ c
Revised: ˜ c

Original: = 0 , which is equivalent to c = 0 and contradicts with the theorem assumption.
Revised: = 0 , which is equivalent to c = 0 and contradicts with the theorem assumption.

Original: (ii) ˜ r = 0 .
Revised: (ii) ˜ r = 0 .

Original: Then according to Equation 17, we have ˜ αβ · f (cid:48)(cid:48) 2 ( 1 β ˆ y − ˜ αa ) ≡ 0 , implies either ˜ α
Revised: Then according to Equation 17, we have ˜ αβ · f (cid:48)(cid:48) 2 ( 1 β ˆ y − ˜ αa ) ≡ 0 , implies either ˜ α

Original: =or f (cid:48)(cid:48) 2 ( 1 β ˆ y − ˜ αa ) ≡ 0 .
Revised: =or f (cid:48)(cid:48) 2 ( 1 β ˆ y − ˜ αa ) ≡ 0 .

Original: If the latter is the case, then f 2 is a linear function and, accordingly, exp( f 2 ) is not integrable and does not correspond to any valid density function.
Revised: If the latter is the case, then f 2 is a linear function and, accordingly, exp( f 2 ) is not integrable and does not correspond to any valid density function.

Original: If the former is true, i.e., ˜ α = 0 , then according to Equation 13, we have α = − qβ , which further implies ˜ r = bd − cαβ = bd + qc .
Revised: If the former is true, i.e., ˜ α = 0 , then according to Equation 13, we have α = − qβ , which further implies ˜ r = bd − cαβ = bd + qc .

Original: Therefore, in this situation, bd + qc = 0 , which again contradicts with the theorem assumption.
Revised: Therefore, in this situation, bd + qc = 0 , which again contradicts with the theorem assumption.

Original: (iii) f (cid:48)(cid:48)(cid:48) 

Revised: (iii) f (cid:48)(cid:48)(cid:48)

Original: ( y − ˜ ra − ˜ c
Revised: ( y − ˜ ra − ˜ c

Original: ˆ y ) ≡ 0 .
Revised: ˆ y ) ≡ 0 .

Original: That is, f 3 is a quadratic function with a nonzero coefﬁcient for the
Revised: That is, f 3 is a quadratic function with a nonzero coefﬁcient for the

Original: quadratic
Revised: quadratic

Original: term (otherwise f 3 does not correspond to the logarithm of any valid density function).
Revised: term (otherwise f 3 does not correspond to the logarithm of any valid density function).

Original: Thus E follows a Gaussian distribution.
Revised: Thus E follows a Gaussian distribution.

Original: Only situation (iii) is possible, i.e., ˜ r ˜ c (cid:54)
Revised: Only situation (iii) is possible, i.e., ˜ r ˜ c (cid:54)

Original: = 

Revised: =

Original: and E follows a Gaussian distribution.
Revised: and E follows a Gaussian distribution.

Original: This further tells us that the RHS of Equation 17 is a nonzero constant.
Revised: This further tells us that the RHS of Equation 17 is a nonzero constant.

Original: Hence f 2 is a quadratic function and E X also follows a Gaussian distribution.
Revised: Hence f 2 is a quadratic function and E X also follows a Gaussian distribution.

Original: Therefore if A ⊥⊥ (cid:98)
Revised: Therefore if A ⊥⊥ (cid:98)

Original: Y | Y were to be true, then E X
Revised: Y | Y were to be true, then E X

Original: and E are both
Revised: and E are both

Original: Gaussian.
Revised: Gaussian.

Original: Its contrapositive gives the conclusion of this theorem.
Revised: Its contrapositive gives the conclusion of this theorem.

Original: Corollary 9 Suppose that both E X and E are Gaussian, with variances σ 2 E X and σ 2 E , respectively.
Revised: Corollary 9 Suppose that both E X and E are Gaussian, with variances σ 2 E X and σ 2 E , respectively.

Original: (The protected feature A is not necessarily Gaussian.)
Revised: (The protected feature A is not necessarily Gaussian.)

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Proof Under the condition that E X and E are Gaussian, their log -density functions are third-order differentiable.
Revised: Proof Under the condition that E X and E are Gaussian, their log -density functions are third-order differentiable.

Original: Then according to the proof of Theorem 1, the Equalized Odds condition A ⊥⊥ (cid:98)
Revised: Then according to the proof of Theorem 1, the Equalized Odds condition A ⊥⊥ (cid:98)

Original: Y | Y is equivalent to Equation 17, which, together with Equation 13 as well as the fact that f (cid:48)(cid:48) 2 = − 1 σ 2 EX and f (cid:48)(cid:48) 3 = − 1 σ 2 E , yields Equation 19.
Revised: Y | Y is equivalent to Equation 17, which, together with Equation 13 as well as the fact that f (cid:48)(cid:48) 2 = − 1 σ 2 EX and f (cid:48)(cid:48) 3 = − 1 σ 2 E , yields Equation 19.

Original: A.2. Proof for Theorem
Revised: A.2. Proof for Theorem

Original: Theorem ( Condition to Achieve Equalized Odds for Regression with Deterministic Prediction )
Revised: Theorem ( Condition to Achieve Equalized Odds for Regression with Deterministic Prediction )

Original: Assume that the protected feature A and the continuous target variable Y are dependent and that their joint probability density p ( A, Y ) is positive for every combination of possible values of A and
Revised: Assume that the protected feature A and the continuous target variable Y are dependent and that their joint probability density p ( A, Y ) is positive for every combination of possible values of A and

Original: Y .
Revised: Y .

Original: Further assume that Y is not fully determined by A , and that there are additional features X that are not independent of Y .
Revised: Further assume that Y is not fully determined by A , and that there are additional features X that are not independent of Y .

Original: Let the prediction (cid:98)
Revised: Let the prediction (cid:98)

Original: Y be characterized by a deterministic function f : A × X → Y .
Revised: Y be characterized by a deterministic function f : A × X → Y .

Original: Equalized Odds holds true if and only if the following condition is satisﬁed ( δ ( · ) is the delta function):
Revised: Equalized Odds holds true if and only if the following condition is satisﬁed ( δ ( · ) is the delta function):

Original: [Equation]
Revised: [Equation]

Original: where Q
Revised: where Q

Original: [Equation]
Revised: [Equation]

Original: a, y
Revised: a, y

Original: [Equation]
Revised: [Equation]

Original: ˆ
Revised: ˆ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: Proof Without loss of generality, let us assume that A and X are continuous.
Revised: Proof Without loss of generality, let us assume that A and X are continuous.

Original: By Lemma 8, the
Revised: By Lemma 8, the

Original: Equalized Odds criterion can be written into terms of the conditional probability density functions:
Revised: Equalized Odds criterion can be written into terms of the conditional probability density functions:

Original: [Equation]
Revised: [Equation]

Original: Since (cid:98)
Revised: Since (cid:98)

Original: Y =
Revised: Y =

Original: f ( A, X ) where f is a deterministic function of ( A, X ) , f is an injective mapping from A × X to Y .
Revised: f ( A, X ) where f is a deterministic function of ( A, X ) , f is an injective mapping from A × X to Y .

Original: One can derive the joint probability density of ( (cid:98)
Revised: One can derive the joint probability density of ( (cid:98)

Original: Y , A, X, Y ) by making use of the change of variables.
Revised: Y , A, X, Y ) by making use of the change of variables.

Original: Deﬁne the mapping H f
Revised: Deﬁne the mapping H f

Original: as following:
Revised: as following:

Original: [Equation]
Revised: [Equation]

Original: where V is a constant 0 whose probability density function is δ ( v ) .
Revised: where V is a constant 0 whose probability density function is δ ( v ) .

Original: Notice that H is bijective:
Revised: Notice that H is bijective:

Original: [Equation]
Revised: [Equation]

Original: with the Jacobian matrix
Revised: with the Jacobian matrix

Original: [Equation]
Revised: [Equation]

Original: Therefore by making use of the change of variable technique
Revised: Therefore by making use of the change of variable technique

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: Expand the LHS of Equation 20, we have:
Revised: Expand the LHS of Equation 20, we have:

Original: [Equation]
Revised: [Equation]

Original: Since Equalized Odds holds true if and only if Equation 20 holds true, the LHS of the equation does not involve a (as in RHS of the equation), i.e., Q ( a, y, ˆ y ) does not change with a .
Revised: Since Equalized Odds holds true if and only if Equation 20 holds true, the LHS of the equation does not involve a (as in RHS of the equation), i.e., Q ( a, y, ˆ y ) does not change with a .

Original: Then, rewrite the RHS of Equation 20, we have:
Revised: Then, rewrite the RHS of Equation 20, we have:

Original: [Equation]
Revised: [Equation]

Original: Therefore, Equalized Odds implies the condition that Q ( a, y, ˆ y ) =
Revised: Therefore, Equalized Odds implies the condition that Q ( a, y, ˆ y ) =

Original: Q ( a (cid:48) , y, ˆ y ) .
Revised: Q ( a (cid:48) , y, ˆ y ) .

Original: On the other hand, it is easy to see that when the aforementioned condition holds true, Equation 20 holds true, i.e.,
Revised: On the other hand, it is easy to see that when the aforementioned condition holds true, Equation 20 holds true, i.e.,

Original: Equalized Odds holds true.
Revised: Equalized Odds holds true.

Original: A.3. Proof for Theorem
Revised: A.3. Proof for Theorem

Original: In order to prove Theorem 3, we need the following results from functional analysis (a fuller treatment can be found, for example, in Meise and Vogt (1997);
Revised: In order to prove Theorem 3, we need the following results from functional analysis (a fuller treatment can be found, for example, in Meise and Vogt (1997);

Original: Daners (2008)):
Revised: Daners (2008)):

Original: Lemma 10 Let S be a set and E =
Revised: Lemma 10 Let S be a set and E =

Original: ( E , (cid:107)·(cid:107) )
Revised: ( E , (cid:107)·(cid:107) )

Original: a Banach space.
Revised: a Banach space.

Original: For a function f : S → E , we deﬁne the supremum norm by (cid:107) f (cid:107) ∞ :
Revised: For a function f : S → E , we deﬁne the supremum norm by (cid:107) f (cid:107) ∞ :

Original: = sup s ∈S (cid:107) f
Revised: = sup s ∈S (cid:107) f

Original: ( s ) (cid:107) , and the space of bounded functions by
Revised: ( s ) (cid:107) , and the space of bounded functions by

Original: B ( S , E ) :=
Revised: B ( S , E ) :=

Original: { f : S → E | (cid:107) f
Revised: { f : S → E | (cid:107) f

Original: (cid:107) ∞ < ∞} .
Revised: (cid:107) ∞ < ∞} .

Original: Then B ( S , E ) is a Banach space with the supremum norm.
Revised: Then B ( S , E ) is a Banach space with the supremum norm.

Original: Deﬁnition 11 (Extreme Point)
Revised: Deﬁnition 11 (Extreme Point)

Original: If K is a non-empty convex subset of a vector space, a point x ∈ K is called an extreme point of K if whenever x = λx 1 + (1 − λ ) x 2 with 0 < λ < 1 and x 1 , x 2 ∈ K , then x = x 1 = x 2 .
Revised: If K is a non-empty convex subset of a vector space, a point x ∈ K is called an extreme point of K if whenever x = λx 1 + (1 − λ ) x 2 with 0 < λ < 1 and x 1 , x 2 ∈ K , then x = x 1 = x 2 .

Original: Theorem 12 (Krein-Milman Theorem (Krein and Milman, 1940)) Let K be a non-empty compact subset of a locally convex Hausdorff topological vector space.
Revised: Theorem 12 (Krein-Milman Theorem (Krein and Milman, 1940)) Let K be a non-empty compact subset of a locally convex Hausdorff topological vector space.

Original: Then the set of extreme points of
Revised: Then the set of extreme points of

Original: K is not empty.
Revised: K is not empty.

Original: If K is also convex, then K is the closed convex hull of its extreme points.
Revised: If K is also convex, then K is the closed convex hull of its extreme points.

Original: It is a known result in ﬁnite-dimensional linear programming (see, for example, Dantzig (1965)) that if the speciﬁed feasible region K (which is the intersection of a ﬁnite number of half spaces) is non-empty, then the set of extreme points is not empty and ﬁnite, and the set of extreme directions is empty if and only if K is bounded.
Revised: It is a known result in ﬁnite-dimensional linear programming (see, for example, Dantzig (1965)) that if the speciﬁed feasible region K (which is the intersection of a ﬁnite number of half spaces) is non-empty, then the set of extreme points is not empty and ﬁnite, and the set of extreme directions is empty if and only if K is bounded.

Original: If K is unbounded, the set of extreme directions is not empty and ﬁnite.
Revised: If K is unbounded, the set of extreme directions is not empty and ﬁnite.

Original: Furthermore, a point is in K if and only if it can be represented as a convex combination of the extreme points plus a non-negative linear combination of extreme directions (if there is any).
Revised: Furthermore, a point is in K if and only if it can be represented as a convex combination of the extreme points plus a non-negative linear combination of extreme directions (if there is any).

Original: Theorem 12 extends this result to inﬁnite dimensional vector spaces, allowing us to establish the existence of extreme points, and therefore a non-empty feasible region for certain inﬁnite-dimensional linear programming problems.
Revised: Theorem 12 extends this result to inﬁnite dimensional vector spaces, allowing us to establish the existence of extreme points, and therefore a non-empty feasible region for certain inﬁnite-dimensional linear programming problems.

Original: Note that in order to prove attainability of Equalized Odds, it is sufﬁcient to show that the corresponding inﬁnite-dimensional linear programming problem yields a non-empty feasible region.
Revised: Note that in order to prove attainability of Equalized Odds, it is sufﬁcient to show that the corresponding inﬁnite-dimensional linear programming problem yields a non-empty feasible region.

Original: The establishment of duality is relatively complicated for inﬁnite-dimensional linear programming (Anderson and Nash, 1987) and is beyond the scope of the discussion.
Revised: The establishment of duality is relatively complicated for inﬁnite-dimensional linear programming (Anderson and Nash, 1987) and is beyond the scope of the discussion.

Original: Theorem ( Attainability of Equalized Odds for Regression with Stochastic Prediction )
Revised: Theorem ( Attainability of Equalized Odds for Regression with Stochastic Prediction )

Original: Let A , X , and Y be continuous variables with domain of value A , X , and Y , respectively.
Revised: Let A , X , and Y be continuous variables with domain of value A , X , and Y , respectively.

Original: Assume that their joint distribution is ﬁxed and known.
Revised: Assume that their joint distribution is ﬁxed and known.

Original: Further assume Y ⊥(cid:54)⊥ A , Y ⊥(cid:54)⊥ X , and Y ⊥(cid:54)⊥ X | A .
Revised: Further assume Y ⊥(cid:54)⊥ A , Y ⊥(cid:54)⊥ X , and Y ⊥(cid:54)⊥ X | A .

Original: Without loss of generality let the conditional probability density p X | AY ( x | a, y ) be non-negative and ﬁnite.
Revised: Without loss of generality let the conditional probability density p X | AY ( x | a, y ) be non-negative and ﬁnite.

Original: Then there exists (cid:101)
Revised: Then there exists (cid:101)

Original: Y with domain of value Y whose distribution is fully determined by p (cid:101) Y | AX (˜ y | a, x ) , such that (cid:101)
Revised: Y with domain of value Y whose distribution is fully determined by p (cid:101) Y | AX (˜ y | a, x ) , such that (cid:101)

Original: Y is not independent from ( A, X ) but (cid:101)
Revised: Y is not independent from ( A, X ) but (cid:101)

Original: Y ⊥⊥ A | Y , i.e., the Equalized
Revised: Y ⊥⊥ A | Y , i.e., the Equalized

Original: Odds is non-trivially attainable.
Revised: Odds is non-trivially attainable.

Original: Proof Let us denote h : A×X
Revised: Proof Let us denote h : A×X

Original: ×Y
Revised: ×Y

Original: → R as the ﬁxed function satisfying h ( a, x, y ) =
Revised: → R as the ﬁxed function satisfying h ( a, x, y ) =

Original: p X | AY ( x | a, y ) .
Revised: p X | AY ( x | a, y ) .

Original: We want to show that there exists a function f : A × X × Y
Revised: We want to show that there exists a function f : A × X × Y

Original: → R characterizing the conditional probability density f ( a, x, ˜ y )
Revised: → R characterizing the conditional probability density f ( a, x, ˜ y )

Original: = p (cid:101) Y | AX (˜ y | a, x ) that satisﬁes the following conditions:
Revised: = p (cid:101) Y | AX (˜ y | a, x ) that satisﬁes the following conditions:

Original: (i) ∀ a ∈ A , x ∈ X , ˜ y ∈ Y , f ( a, x, ˜ y ) is non-negative and ﬁnite; (ii) ∀ a ∈ A , x ∈ X , (cid:82) Y
Revised: (i) ∀ a ∈ A , x ∈ X , ˜ y ∈ Y , f ( a, x, ˜ y ) is non-negative and ﬁnite; (ii) ∀ a ∈ A , x ∈ X , (cid:82) Y

Original: f
Revised: f

Original: ( a, x, ˜ y ) dt
Revised: ( a, x, ˜ y ) dt

Original: = 1 ; (iii) let Q ( a, y, ˜ y ) :
Revised: = 1 ; (iii) let Q ( a, y, ˜ y ) :

Original: = (cid:82) X
Revised: = (cid:82) X

Original: f ( a, x, ˜ y ) h ( a, x, y ) dx,
Revised: f ( a, x, ˜ y ) h ( a, x, y ) dx,

Original: and then
Revised: and then

Original: ∀ y, ˜ y ∈ Y , a, a
Revised: ∀ y, ˜ y ∈ Y , a, a

Original: (cid:48) ∈ A , a (cid:54) = a (cid:48) , Q ( a, y, ˜ y ) =
Revised: (cid:48) ∈ A , a (cid:54) = a (cid:48) , Q ( a, y, ˜ y ) =

Original: Q ( a (cid:48) , y, ˜ y ) ; (iv) the function f ( a, x, ˜ y ) cannot be written into a function of only ˜ y .
Revised: Q ( a (cid:48) , y, ˜ y ) ; (iv) the function f ( a, x, ˜ y ) cannot be written into a function of only ˜ y .

Original: Conditions (i) and (ii) guarantee that f corresponds to a valid conditional probability density for some (cid:101)
Revised: Conditions (i) and (ii) guarantee that f corresponds to a valid conditional probability density for some (cid:101)

Original: Y conditioned on A and X .
Revised: Y conditioned on A and X .

Original: Condition (iii) is the necessary and sufﬁcient condition for Equalized
Revised: Condition (iii) is the necessary and sufﬁcient condition for Equalized

Original: Odds to hold true for regression tasks (as we have seen in Theorem 2).
Revised: Odds to hold true for regression tasks (as we have seen in Theorem 2).

Original: Condition (iv) guarantees that the corresponding (cid:101)
Revised: Condition (iv) guarantees that the corresponding (cid:101)

Original: Y is not independent from ( A, X ) – otherwise (cid:101)
Revised: Y is not independent from ( A, X ) – otherwise (cid:101)

Original: Y will be a trivial prediction.
Revised: Y will be a trivial prediction.

Original: Through the lens of functional analysis, we view f as a point in the inﬁnite-dimensional vector space.
Revised: Through the lens of functional analysis, we view f as a point in the inﬁnite-dimensional vector space.

Original: In order to prove the existence of such a function f , we begin by showing that condition (i) and (iii) form a set of constraints of an inﬁnite-dimensional linear programming problem, and that the corresponding feasible region is non-empty.
Revised: In order to prove the existence of such a function f , we begin by showing that condition (i) and (iii) form a set of constraints of an inﬁnite-dimensional linear programming problem, and that the corresponding feasible region is non-empty.

Original: Furthermore, we can always ﬁnd a point (which is a function) in the aforementioned feasible region that satisﬁes condition (iv).
Revised: Furthermore, we can always ﬁnd a point (which is a function) in the aforementioned feasible region that satisﬁes condition (iv).

Original: Finally, condition (ii) is satisﬁed by “normalizing” over ˜ y ∈ Y .
Revised: Finally, condition (ii) is satisﬁed by “normalizing” over ˜ y ∈ Y .

Original: To begin with, any function that satisﬁes condition (i) is a bounded function deﬁned on the set
Revised: To begin with, any function that satisﬁes condition (i) is a bounded function deﬁned on the set

Original: A × X × Y .
Revised: A × X × Y .

Original: R with absolute value norm forms a Banach space.
Revised: R with absolute value norm forms a Banach space.

Original: By Lemma 10, the functions that satisfy condition (i) form a Banach space with the supremum norm F = ( F , (cid:107)·(cid:107) ∞ ) .
Revised: By Lemma 10, the functions that satisfy condition (i) form a Banach space with the supremum norm F = ( F , (cid:107)·(cid:107) ∞ ) .

Original: Recall that we assume h : A × X × Y → R satisfying h ( a, x, y ) =
Revised: Recall that we assume h : A × X × Y → R satisfying h ( a, x, y ) =

Original: p X | AY ( x | a, y ) (which is ﬁxed and known) is non-negative and ﬁnite.
Revised: p X | AY ( x | a, y ) (which is ﬁxed and known) is non-negative and ﬁnite.

Original: Condition (iii) speciﬁes a set of equality constraints, each of which is characterized with a linear combination of points f in the space F .
Revised: Condition (iii) speciﬁes a set of equality constraints, each of which is characterized with a linear combination of points f in the space F .

Original: Therefore conditions (i) and (iii) form a set of constraints of an inﬁnite-dimensional linear programming problem (since we focus on the feasible region of such problem, the exact form of the objective function is omitted).
Revised: Therefore conditions (i) and (iii) form a set of constraints of an inﬁnite-dimensional linear programming problem (since we focus on the feasible region of such problem, the exact form of the objective function is omitted).

Original: Since for any function f ∈ F that can be written into a function of the form g : Y → R , f trivially satisﬁes condition (iii), one can easily construct a non-empty compact subset K ⊂ F such that each point f ∈ K satisﬁes conditions (i) and (iii).
Revised: Since for any function f ∈ F that can be written into a function of the form g : Y → R , f trivially satisﬁes condition (iii), one can easily construct a non-empty compact subset K ⊂ F such that each point f ∈ K satisﬁes conditions (i) and (iii).

Original: In other words, the inﬁnite-dimensional linear programming problem has a non-empty convex feasible region, which contains K .
Revised: In other words, the inﬁnite-dimensional linear programming problem has a non-empty convex feasible region, which contains K .

Original: We now prove that we can ﬁnd a point in this feasible region that satisﬁes condition (iv) by showing that any point that violates condition (iv) cannot be an extreme point (as deﬁned in Deﬁnition 11) of the feasible region.
Revised: We now prove that we can ﬁnd a point in this feasible region that satisﬁes condition (iv) by showing that any point that violates condition (iv) cannot be an extreme point (as deﬁned in Deﬁnition 11) of the feasible region.

Original: We have shown that there is a non-empty compact subset K ⊂ F , and that
Revised: We have shown that there is a non-empty compact subset K ⊂ F , and that

Original: F is a Banach space, and therefore also a locally convex Hausdorff topological vector space.
Revised: F is a Banach space, and therefore also a locally convex Hausdorff topological vector space.

Original: This also implies that K is closed (since it is a compact subset of a Hausdorff space).
Revised: This also implies that K is closed (since it is a compact subset of a Hausdorff space).

Original: By the Krein-Milman
Revised: By the Krein-Milman

Original: Theorem (Theorem 12), the set of extreme points for K is non-empty.
Revised: Theorem (Theorem 12), the set of extreme points for K is non-empty.

Original: Notice that for any f 0 ∈ K that violates condition (iv), f 0 can be written into a convex combination of f 1 and f 2 ( f 1 , f 2 ∈ K , f 1 (cid:54) = f 2 ) that also violate condition (iv).
Revised: Notice that for any f 0 ∈ K that violates condition (iv), f 0 can be written into a convex combination of f 1 and f 2 ( f 1 , f 2 ∈ K , f 1 (cid:54) = f 2 ) that also violate condition (iv).

Original: For example let f i ( a, x, ˜ y )
Revised: For example let f i ( a, x, ˜ y )

Original: =
Revised: =

Original: g i
Revised: g i

Original: (˜ y ) ,
Revised: (˜ y ) ,

Original: i = 0 , 1 , 2 (since each f i violates condition (iv)).
Revised: i = 0 , 1 , 2 (since each f i violates condition (iv)).

Original: Let g 1 (˜ y )
Revised: Let g 1 (˜ y )

Original: = 2 · 1 ( −∞ , 0] (˜ y ) · g 0 (˜ y ) and g 2 (˜ y )
Revised: = 2 · 1 ( −∞ , 0] (˜ y ) · g 0 (˜ y ) and g 2 (˜ y )

Original: = 2 · 1 (0 , + ∞ ) (˜ y ) · g 0 (˜ y ) where· ( · ) is the indicator function (without loss of generality assume that the corresponding f 1 , f 2 ∈ K ).
Revised: = 2 · 1 (0 , + ∞ ) (˜ y ) · g 0 (˜ y ) where· ( · ) is the indicator function (without loss of generality assume that the corresponding f 1 , f 2 ∈ K ).

Original: It is easy to verify that f 1 (cid:54) = f 2 , and 12 f 1 + 12 f 2 = f 0 .
Revised: It is easy to verify that f 1 (cid:54) = f 2 , and 12 f 1 + 12 f 2 = f 0 .

Original: Therefore any f 0 that violates condition (iv) cannot be an extreme point of K , which is closed with a non-empty set of extreme points.
Revised: Therefore any f 0 that violates condition (iv) cannot be an extreme point of K , which is closed with a non-empty set of extreme points.

Original: In other words, there exist a point f ∗ ∈ F such that f ∗ is an extreme point of K (therefore f ∗ is within the feasible region) and that f ∗ satisﬁes condition (iv).
Revised: In other words, there exist a point f ∗ ∈ F such that f ∗ is an extreme point of K (therefore f ∗ is within the feasible region) and that f ∗ satisﬁes condition (iv).

Original: Finally, for such f ∗ within the feasible region, one can ﬁnd a unique function ˜ f
Revised: Finally, for such f ∗ within the feasible region, one can ﬁnd a unique function ˜ f

Original: : A × X × Y such that condition (ii) is satisﬁed by normalizing over ˜ y ∈ Y for each possible combination of
Revised: : A × X × Y such that condition (ii) is satisﬁed by normalizing over ˜ y ∈ Y for each possible combination of

Original: [Equation]
Revised: [Equation]

Original: Therefore, there exists a function ˜
Revised: Therefore, there exists a function ˜

Original: f that satisﬁes conditions (i) to (iv), i.e., Equalized Odds can be non-trivially attained.
Revised: f that satisﬁes conditions (i) to (iv), i.e., Equalized Odds can be non-trivially attained.

Original: A.4. Proof for Theorem
Revised: A.4. Proof for Theorem

Original: Theorem ( Condition to Achieve Equalized Odds for Classiﬁcation with Deterministic Prediction ) Assume that the protected feature A and Y are dependent and that their joint probability
Revised: Theorem ( Condition to Achieve Equalized Odds for Classiﬁcation with Deterministic Prediction ) Assume that the protected feature A and Y are dependent and that their joint probability

Original: P ( A, Y ) (for discrete A ) or joint probability density p ( A, Y ) (for continuous A ) is positive for every combination of possible values of A and Y .
Revised: P ( A, Y ) (for discrete A ) or joint probability density p ( A, Y ) (for continuous A ) is positive for every combination of possible values of A and Y .

Original: Further assume that Y is not fully determined by A , and that there are additional features X that are not independent of Y .
Revised: Further assume that Y is not fully determined by A , and that there are additional features X that are not independent of Y .

Original: Let the output of the classiﬁer (cid:98)
Revised: Let the output of the classiﬁer (cid:98)

Original: Y be a deterministic function f : A × X → Y .
Revised: Y be a deterministic function f : A × X → Y .

Original: Let S (ˆ y )
Revised: Let S (ˆ y )

Original: A :
Revised: A :

Original: = { a | ∃ x ∈ X s.t. f
Revised: = { a | ∃ x ∈ X s.t. f

Original: ( a, x ) = ˆ y } , and
Revised: ( a, x ) = ˆ y } , and

Original: S (ˆ y )
Revised: S (ˆ y )

Original: X | a :=
Revised: X | a :=

Original: { x | f
Revised: { x | f

Original: ( a, x ) =
Revised: ( a, x ) =

Original: ˆ y } .
Revised: ˆ y } .

Original: Equalized Odds is attained if and only if the following conditions hold true (for continuous X , replace summation with integration accordingly):
Revised: Equalized Odds is attained if and only if the following conditions hold true (for continuous X , replace summation with integration accordingly):

Original: [Equation]
Revised: [Equation]

Original: Proof We begin by considering the case when A and X are discrete (for the purpose of readability).
Revised: Proof We begin by considering the case when A and X are discrete (for the purpose of readability).

Original: The Equalized Odds criterion can be written in terms of the conditional probabilities:
Revised: The Equalized Odds criterion can be written in terms of the conditional probabilities:

Original: [Equation]
Revised: [Equation]

Original: Expand the LHS of Equation 21:
Revised: Expand the LHS of Equation 21:

Original: [Equation]
Revised: [Equation]

Original: and bear in mind that
Revised: and bear in mind that

Original: [Equation]
Revised: [Equation]

Original: Y :
Revised: Y :

Original: =
Revised: =

Original: f
Revised: f

Original: ( A, X ) is a deterministic function of ( A, X ) , we have:
Revised: ( A, X ) is a deterministic function of ( A, X ) , we have:

Original: [Equation]
Revised: [Equation]

Original: From Equation 22 we can see that the conditional probability P X | AY ( x | a, y ) can contribute to the summation only when f ( a, x ) =
Revised: From Equation 22 we can see that the conditional probability P X | AY ( x | a, y ) can contribute to the summation only when f ( a, x ) =

Original: ˆ y .
Revised: ˆ y .

Original: We can rewrite the LHS of Equation 21:
Revised: We can rewrite the LHS of Equation 21:

Original: [Equation]
Revised: [Equation]

Original: Similarly, for the RHS of Equation 21, we have:
Revised: Similarly, for the RHS of Equation 21, we have:

Original: [Equation]
Revised: [Equation]

Original: Since Equalized Odds holds true if and only if Equation 21 holds true, then the LHS of the equation does not involve a (as is the case for the RHS), i.e., Q (ˆ y ) ( a, y ) does not change with a .
Revised: Since Equalized Odds holds true if and only if Equation 21 holds true, then the LHS of the equation does not involve a (as is the case for the RHS), i.e., Q (ˆ y ) ( a, y ) does not change with a .

Original: Then Equation 21 becomes:
Revised: Then Equation 21 becomes:

Original: [Equation]
Revised: [Equation]

Original: which gives condition (i) that S (ˆ y )
Revised: which gives condition (i) that S (ˆ y )

Original: A contains all possible values of A ,
Revised: A contains all possible values of A ,

Original: i.e., A = S
Revised: i.e., A = S

Original: (ˆ y )
Revised: (ˆ y )

Original: A (otherwise (cid:80)
Revised: A (otherwise (cid:80)

Original: [Equation]
Revised: [Equation]

Original: (ˆ y )
Revised: (ˆ y )

Original: A P A | Y ( a | y ) < 1 ).
Revised: A P A | Y ( a | y ) < 1 ).

Original: Since Q (ˆ y ) ( a, y ) does not change with a , we have:
Revised: Since Q (ˆ y ) ( a, y ) does not change with a , we have:

Original: [Equation]
Revised: [Equation]

Original: which gives condition (ii).
Revised: which gives condition (ii).

Original: Therefore, Equalized Odds implies conditions (i) and (ii).
Revised: Therefore, Equalized Odds implies conditions (i) and (ii).

Original: On the other hand, it is easy to see that when conditions (i) and (ii) are satisﬁed, Equation 21 holds true, i.e.,
Revised: On the other hand, it is easy to see that when conditions (i) and (ii) are satisﬁed, Equation 21 holds true, i.e.,

Original: Equalized Odds holds true.
Revised: Equalized Odds holds true.

Original: When A and X are continuous, one can replace the summation with integration accordingly.
Revised: When A and X are continuous, one can replace the summation with integration accordingly.

Original: A.5.
Revised: A.5.

Original: Proof for Theorem
Revised: Proof for Theorem

Original: Theorem ( Attainability of Equalized Odds for Classiﬁcation with Stochastic Prediction )
Revised: Theorem ( Attainability of Equalized Odds for Classiﬁcation with Stochastic Prediction )

Original: Assume that the feature X is not independent from Y , and that (cid:98)
Revised: Assume that the feature X is not independent from Y , and that (cid:98)

Original: Y is a function of A and X .
Revised: Y is a function of A and X .

Original: Then for binary classiﬁcation, if (cid:98)
Revised: Then for binary classiﬁcation, if (cid:98)

Original: Y is a non-trivial predictor for Y , there is always at least one non-trivial (possibly randomized) predictor (cid:101)
Revised: Y is a non-trivial predictor for Y , there is always at least one non-trivial (possibly randomized) predictor (cid:101)

Original: Y post derived by post-processing (cid:98)
Revised: Y post derived by post-processing (cid:98)

Original: Y that can attain Equalized Odds:
Revised: Y that can attain Equalized Odds:

Original: [Equation]
Revised: [Equation]

Original: Proof Since (cid:98)
Revised: Proof Since (cid:98)

Original: Y is a function of ( A, X ) and X ⊥(cid:54)⊥ Y , (cid:98)
Revised: Y is a function of ( A, X ) and X ⊥(cid:54)⊥ Y , (cid:98)

Original: Y is not conditionally independent from Y given protected feature A .
Revised: Y is not conditionally independent from Y given protected feature A .

Original: Furthermore, since (cid:98)
Revised: Furthermore, since (cid:98)

Original: Y is a non-trivial estimator of the binary target Y , there exists a positive constant (cid:15) > 0 , such that ( ∀ a ∈ A ):
Revised: Y is a non-trivial estimator of the binary target Y , there exists a positive constant (cid:15) > 0 , such that ( ∀ a ∈ A ):

Original: [Equation]
Revised: [Equation]

Original: Equation 23 implies that for each value of A , the corresponding true positive rate of the non-trivial predictor is always strictly larger than its false positive rate 7 .
Revised: Equation 23 implies that for each value of A , the corresponding true positive rate of the non-trivial predictor is always strictly larger than its false positive rate 7 .

Original: As illustrated in Panels (a) and (b) of
Revised: As illustrated in Panels (a) and (b) of

Original: Figure 1, the (FPR, TPR) pair of the predictor (cid:98)
Revised: Figure 1, the (FPR, TPR) pair of the predictor (cid:98)

Original: Y when A = a , i.e., the point γ a ( (cid:98)
Revised: Y when A = a , i.e., the point γ a ( (cid:98)

Original: Y ) on ROC plane, will never fall in the gray shaded area, and its coordinates are bounded away from the diagonal by at least (cid:15) .
Revised: Y ) on ROC plane, will never fall in the gray shaded area, and its coordinates are bounded away from the diagonal by at least (cid:15) .

Original: Therefore, the intersection of all C a ( (cid:98)
Revised: Therefore, the intersection of all C a ( (cid:98)

Original: Y ) would always form a parallelogram with non-empty area, which corresponds to attainable non-trivial post-processing fair predictors (cid:101)
Revised: Y ) would always form a parallelogram with non-empty area, which corresponds to attainable non-trivial post-processing fair predictors (cid:101)

Original: Y post .
Revised: Y post .

Original: A.6. Proof for Theorem
Revised: A.6. Proof for Theorem

Original: Theorem ( Equivalence between ROC feasible areas ) et Ω( (cid:101)
Revised: Theorem ( Equivalence between ROC feasible areas ) et Ω( (cid:101)

Original: Y post ) denote the ROC feasible area speciﬁed by the constraints enforced on (cid:101)
Revised: Y post ) denote the ROC feasible area speciﬁed by the constraints enforced on (cid:101)

Original: Y post .
Revised: Y post .

Original: Then Ω( (cid:101)
Revised: Then Ω( (cid:101)

Original: Y post ) is identical to the ROC feasible area Ω( (cid:101)
Revised: Y post ) is identical to the ROC feasible area Ω( (cid:101)

Original: Y ∗ in ) that is speciﬁed by the following set of constraints:
Revised: Y ∗ in ) that is speciﬁed by the following set of constraints:

Original: (i) constraints enforced on (cid:101) Y in ; (ii) additional “pseudo” constraints: ∀ a ∈ A , β (0) a 0 = β (0) a 1 , β (
Revised: (i) constraints enforced on (cid:101) Y in ; (ii) additional “pseudo” constraints: ∀ a ∈ A , β (0) a 0 = β (0) a 1 , β (

Original: 1) a 0 = β (1) a 1 , where β
Revised: 1) a 0 = β (1) a 1 , where β

Original: [Equation]
Revised: [Equation]

Original: Proof Since the post-processing predictor (cid:101)
Revised: Proof Since the post-processing predictor (cid:101)

Original: Y post is derived by optimizing over parameters or functions (of A )
Revised: Y post is derived by optimizing over parameters or functions (of A )

Original: β ( u ) a .
Revised: β ( u ) a .

Original: Therefore, considering the fact that P (cid:101)
Revised: Therefore, considering the fact that P (cid:101)

Original: Y post | AY (1 | a, y ) =
Revised: Y post | AY (1 | a, y ) =

Original: γ ay ( (cid:101)
Revised: γ ay ( (cid:101)

Original: Y post ) ,
Revised: Y post ) ,

Original: P (cid:98) Y opt | AY (1 | a, y )
Revised: P (cid:98) Y opt | AY (1 | a, y )

Original: =
Revised: =

Original: γ ay ( (cid:98)
Revised: γ ay ( (cid:98)

Original: Y opt ) , we have the relation between γ ay ( (cid:101)
Revised: Y opt ) , we have the relation between γ ay ( (cid:101)

Original: Y post ) and γ ay ( (cid:98)
Revised: Y post ) and γ ay ( (cid:98)

Original: Y
Revised: Y

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Similarly, consider the relation between positive rates of (cid:101)
Revised: Similarly, consider the relation between positive rates of (cid:101)

Original: Y in and those of (cid:98)
Revised: Y in and those of (cid:98)

Original: Y opt , i.e., P (cid:101) Y in | AY (1 | a, y ) and P (cid:98) Y opt | AY (1 | a, y ) , by factorizing P (cid:101) Y in | AY (1 | a, y ) over X and (cid:98)
Revised: Y opt , i.e., P (cid:101) Y in | AY (1 | a, y ) and P (cid:98) Y opt | AY (1 | a, y ) , by factorizing P (cid:101) Y in | AY (1 | a, y ) over X and (cid:98)

Original: Y opt :
Revised: Y opt :

Original: P (cid:101) Y in | AY (1 | a, y ) =
Revised: P (cid:101) Y in | AY (1 | a, y ) =

Original: (cid:88)
Revised: (cid:88)

Original: ˆ y ∈Y
Revised: ˆ y ∈Y

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we have the relation between γ ay (
Revised: we have the relation between γ ay (

Original: [Equation]
Revised: [Equation]

Original: Y in
Revised: Y in

Original: [Equation]
Revised: [Equation]

Original: and γ ay (
Revised: and γ ay (

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: If there is more than one variable in X in Equation 25, one can expand the summation if needed; if some variables are continuous, one may also substitute the summation with integration accordingly.
Revised: If there is more than one variable in X in Equation 25, one can expand the summation if needed; if some variables are continuous, one may also substitute the summation with integration accordingly.

Original: From Equation 25, β (0) ay and β (1) ay depend on the value of Y :
Revised: From Equation 25, β (0) ay and β (1) ay depend on the value of Y :

Original: [Equation]
Revised: [Equation]

Original: Apart from Equalized Odds constraints (which are shared by (cid:101)
Revised: Apart from Equalized Odds constraints (which are shared by (cid:101)

Original: Y in and (cid:101)
Revised: Y in and (cid:101)

Original: Y post ), when enforcing additional “pseudo” constraints β (0) a 0 = β (0) a 1 and β (1) a 0 = β (1) a 1 , conditional independence (cid:101)
Revised: Y post ), when enforcing additional “pseudo” constraints β (0) a 0 = β (0) a 1 and β (1) a 0 = β (1) a 1 , conditional independence (cid:101)

Original: Y in
Revised: Y in

Original: [Equation]
Revised: [Equation]

Original: Y | A, (cid:98)
Revised: Y | A, (cid:98)

Original: Y opt is enforced, making β (0) ay and β (0) ay no longer depend on Y .
Revised: Y opt is enforced, making β (0) ay and β (0) ay no longer depend on Y .

Original: This is exactly the inherent constraint (cid:101)
Revised: This is exactly the inherent constraint (cid:101)

Original: Y post satisﬁes.
Revised: Y post satisﬁes.

Original: Therefore the stated equivalence between ROC feasible areas Ω( (cid:101)
Revised: Therefore the stated equivalence between ROC feasible areas Ω( (cid:101)

Original: Y post ) (speciﬁed by the constraints enforced on (cid:101)
Revised: Y post ) (speciﬁed by the constraints enforced on (cid:101)

Original: Y post ) and Ω( (cid:101)
Revised: Y post ) and Ω( (cid:101)

Original: Y ∗ in ) (speciﬁed by the constraints enforced on (cid:101)
Revised: Y ∗ in ) (speciﬁed by the constraints enforced on (cid:101)

Original: Y in together with the additional “pseudo” constraints) hold true.
Revised: Y in together with the additional “pseudo” constraints) hold true.

Original: Appendix B. Further Information about Experiments
Revised: Appendix B. Further Information about Experiments

Original: In this section, we ﬁrst introduce the real-world data sets that are used in the experiments (including regression and classiﬁcation tasks).
Revised: In this section, we ﬁrst introduce the real-world data sets that are used in the experiments (including regression and classiﬁcation tasks).

Original: Then, we present the technical speciﬁcations of the experiments.
Revised: Then, we present the technical speciﬁcations of the experiments.

Original: B.1. Description of the Data Sets
Revised: B.1. Description of the Data Sets

Original: (1) Communities and Crime : The UCI Communities and Crime data set contains 122 features for 1994 records.
Revised: (1) Communities and Crime : The UCI Communities and Crime data set contains 122 features for 1994 records.

Original: The regression task is to predict the number of violent crimes per population for US cities given various census information of the corresponding communities.
Revised: The regression task is to predict the number of violent crimes per population for US cities given various census information of the corresponding communities.

Original: The data is preprocessed following Mary et al.
Revised: The data is preprocessed following Mary et al.

Original: (2019), and the (continuous) protected feature is the ratio of African-American people in the population.
Revised: (2019), and the (continuous) protected feature is the ratio of African-American people in the population.

Original: (2) Adult (Kohavi, 1996):
Revised: (2) Adult (Kohavi, 1996):

Original: The UCI Adult data set contains 14 features for 45,222 individuals (32,561 samples for training and 12,661 samples for testing).
Revised: The UCI Adult data set contains 14 features for 45,222 individuals (32,561 samples for training and 12,661 samples for testing).

Original: The census information includes gender, marital status, education, capital gain, etc.
Revised: The census information includes gender, marital status, education, capital gain, etc.

Original: The classiﬁcation task is to predict whether a person’s annual income exceeds 50,000 USD.
Revised: The classiﬁcation task is to predict whether a person’s annual income exceeds 50,000 USD.

Original: We use the provided testing set for evaluations and present the result with gender and race (consider white and black people only) set as the protected feature respectively.
Revised: We use the provided testing set for evaluations and present the result with gender and race (consider white and black people only) set as the protected feature respectively.

Original: (3) Bank (Moro et al., 2014):
Revised: (3) Bank (Moro et al., 2014):

Original: The UCI Bank Marketing data set is related with marketing campaigns of a banking institution, containing 16 features of 45,211 individuals.
Revised: The UCI Bank Marketing data set is related with marketing campaigns of a banking institution, containing 16 features of 45,211 individuals.

Original: The assigned classiﬁcation task is to predict if a client will subscribe (yes/no) to a term deposit.
Revised: The assigned classiﬁcation task is to predict if a client will subscribe (yes/no) to a term deposit.

Original: The original data set is very unbalanced with only 4,667 positives out of 45,211 samples.
Revised: The original data set is very unbalanced with only 4,667 positives out of 45,211 samples.

Original: Therefore, we combine “yes” points with randomly subsampled “no” points and perform experiments on the down-sampled data set with 10,000 data points.
Revised: Therefore, we combine “yes” points with randomly subsampled “no” points and perform experiments on the down-sampled data set with 10,000 data points.

Original: The protected feature is the marital status of the client.
Revised: The protected feature is the marital status of the client.

Original: (4) COMPAS (Angwin et al., 2016):
Revised: (4) COMPAS (Angwin et al., 2016):

Original: The COMPAS data set contains records of over 11, defendants from Broward County, Florida, whose risk (of recidivism) was assessed using the
Revised: The COMPAS data set contains records of over 11, defendants from Broward County, Florida, whose risk (of recidivism) was assessed using the

Original: COMPAS tool.
Revised: COMPAS tool.

Original: Each record contains multiple features of the defendant, including demographic information, prior convictions, degree of charge, and the ground truth for recidivism within two years.
Revised: Each record contains multiple features of the defendant, including demographic information, prior convictions, degree of charge, and the ground truth for recidivism within two years.

Original: Following Zafar et al.
Revised: Following Zafar et al.

Original: (2017a)
Revised: (2017a)

Original: and Nabi and Shpitser (2018), we limit our attention to the subset consisting of African-Americans and Caucasians defendants.
Revised: and Nabi and Shpitser (2018), we limit our attention to the subset consisting of African-Americans and Caucasians defendants.

Original: The features we use include age, gender, race, number of priors, and degree of charges.
Revised: The features we use include age, gender, race, number of priors, and degree of charges.

Original: The task is to predict the recidivism of the defendant and we choose race as the protected feature.
Revised: The task is to predict the recidivism of the defendant and we choose race as the protected feature.

Original: (5) German Credit : The UCI German Credit data contains 20 features (7 numerical, 13 categorical) describing the social and economical status of 1,000 customers.
Revised: (5) German Credit : The UCI German Credit data contains 20 features (7 numerical, 13 categorical) describing the social and economical status of 1,000 customers.

Original: The prediction task is to classify people as good or bad credit risks.
Revised: The prediction task is to classify people as good or bad credit risks.

Original: We use the provided numerical version of the data and choose gender as the protected feature.
Revised: We use the provided numerical version of the data and choose gender as the protected feature.

Original: B.2. Experimental Setup B.2.1. R EGRESSION T ASKS
Revised: B.2. Experimental Setup B.2.1. R EGRESSION T ASKS

Original: The regression experiments consist of two base models, namely, a linear regression model and a neural network model, each of which appears in the form of both deterministic and stochastic prediction.
Revised: The regression experiments consist of two base models, namely, a linear regression model and a neural network model, each of which appears in the form of both deterministic and stochastic prediction.

Original: For the neural network model, following Mary et al.
Revised: For the neural network model, following Mary et al.

Original: we use a simple regressor for the experiments: two hidden layers (with the same number of neurons ranging from 30 tofor each layer, depending on the size of the data set) with SELUs nonlinearity (Klambauer et al., 2017).
Revised: we use a simple regressor for the experiments: two hidden layers (with the same number of neurons ranging from 30 tofor each layer, depending on the size of the data set) with SELUs nonlinearity (Klambauer et al., 2017).

Original: The network is optimized by minimizing the MSE loss combined with the λ -weighted fairness penalty term, using the Adam optimizer (Kingma and Ba, 2015).
Revised: The network is optimized by minimizing the MSE loss combined with the λ -weighted fairness penalty term, using the Adam optimizer (Kingma and Ba, 2015).

Original: We use the kernel measure of conditional dependence (KMCD) (Fukumizu et al., 2007) as the fairness penalty term (for stochastic prediction), and the weight λ ranges from { 2 − 2 , 2 − 1 , . . .
Revised: We use the kernel measure of conditional dependence (KMCD) (Fukumizu et al., 2007) as the fairness penalty term (for stochastic prediction), and the weight λ ranges from { 2 − 2 , 2 − 1 , . . .

Original: The range of the absolute value of λ may differ for different experiment setups since the relative value difference between the MSE loss and the KMCD measure.
Revised: The range of the absolute value of λ may differ for different experiment setups since the relative value difference between the MSE loss and the KMCD measure.

Original: The learning rate is chosen from { 10 − 4 , 10 − 5 , 10 − 6 } (smaller learning rate is preferable especially for larger values of λ , i.e., more emphasis on the fairness penalty), and the batch size is chosen from { 64 , 128 , 256 } .
Revised: The learning rate is chosen from { 10 − 4 , 10 − 5 , 10 − 6 } (smaller learning rate is preferable especially for larger values of λ , i.e., more emphasis on the fairness penalty), and the batch size is chosen from { 64 , 128 , 256 } .

Original: As we have stated in Section 3.3, for stochastic prediction, a random sample from the standard Gaussian distribution is concatenated to each batch of data, and the input dimension of the model changes accordingly.
Revised: As we have stated in Section 3.3, for stochastic prediction, a random sample from the standard Gaussian distribution is concatenated to each batch of data, and the input dimension of the model changes accordingly.

Original: The random sample to concatenate is redrawn each time the data point is seen by the model.
Revised: The random sample to concatenate is redrawn each time the data point is seen by the model.

Original: In our experiments on the simulated data, for linear cases, the data is generated as stated in
Revised: In our experiments on the simulated data, for linear cases, the data is generated as stated in

Original: Equation 11, with non-Gaussian distributed exogenous terms ( E X , E H , and E Y ).
Revised: Equation 11, with non-Gaussian distributed exogenous terms ( E X , E H , and E Y ).

Original: Figure 5(a) and (b) correspond to different distributions for the noise terms, speciﬁcally, the Laplace distribution and uniform distribution, respectively.
Revised: Figure 5(a) and (b) correspond to different distributions for the noise terms, speciﬁcally, the Laplace distribution and uniform distribution, respectively.

Original: We use linear regression with the Equalized Correlations constraint (Woodworth et al., 2017), a weaker notion of Equalized Odds for linearly correlated data, as the predictor.
Revised: We use linear regression with the Equalized Correlations constraint (Woodworth et al., 2017), a weaker notion of Equalized Odds for linearly correlated data, as the predictor.

Original: For nonlinear cases, the data is generated using a similar scheme but with nonlinear transformations involved and Gaussian distributed exogenous terms.
Revised: For nonlinear cases, the data is generated using a similar scheme but with nonlinear transformations involved and Gaussian distributed exogenous terms.

Original: We use a neural network regressor with an Equalized Odds regularization term (Mary et al., 2019) to perform nonlinear fair regression.
Revised: We use a neural network regressor with an Equalized Odds regularization term (Mary et al., 2019) to perform nonlinear fair regression.

Original: As we can see in Figure 5(c) and (d), for nonlinear regression tasks, Equalized Odds may not be attained even if every exogenous term is Gaussian distributed.
Revised: As we can see in Figure 5(c) and (d), for nonlinear regression tasks, Equalized Odds may not be attained even if every exogenous term is Gaussian distributed.

Original: In our experiments on the real-world data, we perform 20 random splits of Communities and
Revised: In our experiments on the real-world data, we perform 20 random splits of Communities and

Original: Crime data into training ( 80% ) and testing ( 20% ) sets, and present the testing performance for deterministic and stochastic prediction models (for the linear regressor as well as the neural network regressor) in Figure 3.
Revised: Crime data into training ( 80% ) and testing ( 20% ) sets, and present the testing performance for deterministic and stochastic prediction models (for the linear regressor as well as the neural network regressor) in Figure 3.

Original: In situations where the protected feature is not available when testing, it is
Revised: In situations where the protected feature is not available when testing, it is

Original: [Figure]
Revised: [Table]

Original: e Y g
Revised: ">AAACG3icbZBPSxtBGMZn1bY2tTVqb728VAQPbdiVUM0t4MWjhcY/ZJcwO/tuMjg7u8y8q4Yln8OTH8WToCDepId [Table] [Figure]

Original: i v e n t h a t
Revised: given that Y

Original: [Figure]
Revised: [Figure]

Revised: Illustration of

Revised: unattainable Equalized Odds

Revised: for

Revised: regression with deterministic prediction

Original:  We can observe obvious dependencies between (cid:101)
Revised: Panel (a)-(b): Linear regression on the data generated with linear transformations and non-Gaussian distributed exogenous terms (following Laplace, Uniform distribution respectively). Panel (c)-(d): Nonlinear regression with a neural net regressor (Mary et al., 2019) on the data generated with nonlinear transformations and Gaussian exogenous terms. We can observe obvious dependencies between (cid:101)

Original: Y and A on a small interval of Y .
Revised: Y and A on a small interval of Y .

Original: This indicates the conditional dependency between (cid:101) Y and A given Y , i
Revised: This indicates the conditional dependency between (cid:101) Y and A given Y , i

Original: e., the Equalized Odds is not achieved.
Revised: e., the Equalized Odds is not achieved.

Original: desirable to exclude the protected feature from the attributes when performing prediction.
Revised: desirable to exclude the protected feature from the attributes when performing prediction.

Original: To this end, during the training process, the input to the regression model does not include the protected feature, and the protected feature is only used to compute the fairness penalty.
Revised: To this end, during the training process, the input to the regression model does not include the protected feature, and the protected feature is only used to compute the fairness penalty.

Original: The related result on the Communities and Crime data set is summarized in Figure 6.
Revised: The related result on the Communities and Crime data set is summarized in Figure 6.

Original: B.2.2. C LASSIFICATION T ASKS
Revised: B.2.2. C LASSIFICATION T ASKS

Original: In Figure 4, we compare the performance under Equalized Odds of multiple methods proposed in the literature.
Revised: In Figure 4, we compare the performance under Equalized Odds of multiple methods proposed in the literature.

Original: Hardt et al.
Revised: Hardt et al.

Original: (2016) propose a post-processing approach where the prediction is randomized to minimize violation of fairness; Zafar et al.
Revised: (2016) propose a post-processing approach where the prediction is randomized to minimize violation of fairness; Zafar et al.

Original: (2017a) use a covariance proxy measure as the regularization term when optimizing classiﬁcation accuracy; Agarwal et al.
Revised: (2017a) use a covariance proxy measure as the regularization term when optimizing classiﬁcation accuracy; Agarwal et al.

Original: (2018) take the reductions approach and reduce fair classiﬁcation into solving a sequence of cost-sensitive classiﬁcation problems; Rezaei et al.
Revised: (2018) take the reductions approach and reduce fair classiﬁcation into solving a sequence of cost-sensitive classiﬁcation problems; Rezaei et al.

Original: (2020) minimize the worst-case log loss using an approximated regularization term; Baharlouei et al.
Revised: (2020) minimize the worst-case log loss using an approximated [Figure] [Figure] [Table]

Revised: [Figure]

Original: (2020) propose to use R´enyi correlation as the regularization term to account for nonlinear dependence between variables.
Revised: [Table] [Figure] <latexit sha1_base64="yKTwZ1GJhF0iPDJakT/DqekaXTo=">AAAB+XicbZBLS8NAFIUnPmt8NOpON4NFqJuSlGKtq4IbN0IF+4AmhMlk2g6dPJiZCCUUV/4KV4KCuHXvn3DlH3HtNCni68DAxzn3MpfjxYwKaZrv2sLi0vLKamFNX9/Y3Coa2zsdESUckzaOWMR7HhKE0ZC0JZWM9GJOUOAx0vXGZ7O8e024oFF4JScxcQI0DOmAYiSV5RpFm0VDt1q2mdrx0ZFrlMyKmQn+BWsOpebebenj5uK15Rpvth/hJCChxAwJ0bfMWDop4pJiRqa6nQgSIzxGQ9JXGKKACCfNDp/CQ+X4cBBx9UIJM/f7RooCISaBpyYDJEfidzYz/8v6iRycOCkN40SSEOcfDRIGZQRnLUCfcoIlmyhAmFN1K8QjxBGWqitdz1poZII51GtzaFhfLXSqFeu4UrtUdZyCXAWwDw5AGVigDprgHLRAG2CQgDvwAB61VLvXnrTnfHRBm+/sgh/SXj4BQtuWnQ==</latexit> log [Figure]

Revised: ) <latexit sha1_base64="yKTwZ1GJhF0iPDJakT/DqekaXTo=">AAAB+XicbZBLS8NAFIUnPmt8NOpON4NFqJuSlGKtq4IbN0IF+4AmhMlk2g6dPJiZCCUUV/4KV4KCuHXvn3DlH3HtNCni68DAxzn3MpfjxYwKaZrv2sLi0vLKamFNX9/Y3Coa2zsdESUckzaOWMR7HhKE0ZC0JZWM9GJOUOAx0vXGZ7O8e024oFF4JScxcQI0DOmAYiSV5RpFm0VDt1q2mdrx0ZFrlMyKmQn+BWsOpebebenj5uK15Rpvth/hJCChxAwJ0bfMWDop4pJiRqa6nQgSIzxGQ9JXGKKACCfNDp/CQ+X4cBBx9UIJM/f7RooCISaBpyYDJEfidzYz/8v6iRycOCkN40SSEOcfDRIGZQRnLUCfcoIlmyhAmFN1K8QjxBGWqitdz1poZII51GtzaFhfLXSqFeu4UrtUdZyCXAWwDw5AGVigDprgHLRAG2CQgDvwAB61VLvXnrTnfHRBm+/sgh/SXj4BQtuWnQ==</latexit>

Revised: [Figure]

Revised: regressor, and the stochastic prediction is achieved by introducing an independent noise input sampled from a standard

Revised: Gaussian distribution.

Revised: regularization term; Baharlouei et al.

Revised: (2020) propose to use R´enyi correlation as the regularization term to account for nonlinear dependence between variables.

Original: To measure the violation of the fairness criterion, we use Equalized Odds (EOdds) violation, deﬁned as max y ∈Y a,a (cid:48) ∈A (cid:12)(cid:12)
Revised: To measure the violation of the fairness criterion, we use Equalized Odds (EOdds) violation, deﬁned as max y ∈Y a,a (cid:48) ∈A (cid:12)(cid:12)

Original: P (cid:101) Y | AY (1 | a, y ) −
Revised: P (cid:101) Y | AY (1 | a, y ) −

Original: P (cid:101) Y | AY (1 | a (cid:48) , y ) (cid:12)(cid:12) .
Revised: P (cid:101) Y | AY (1 | a (cid:48) , y ) (cid:12)(cid:12) .

Original: Following Agarwal et al.
Revised: Following Agarwal et al.

Original: (2018), we pick 0 .
Revised: (2018), we pick 0 .

Original: 01 as the default violation bound that the EOdds violation does not exceed (if practically achievable for the method) during training.
Revised: 01 as the default violation bound that the EOdds violation does not exceed (if practically achievable for the method) during training.

Original: For each method we plot the testing accuracy versus the violation of Equalized Odds.
Revised: For each method we plot the testing accuracy versus the violation of Equalized Odds.

Original: Appendix C. Additional Derivation and Illustrative Examples C.1. Derivation of the Connection between Theorem 2 and Theorem
Revised: Appendix C. Additional Derivation and Illustrative Examples C.1. Derivation of the Connection between Theorem 2 and Theorem

Original: Theorem 2 presents the condition under which Equalized Odds can possibly hold true for regression with deterministic prediction.
Revised: Theorem 2 presents the condition under which Equalized Odds can possibly hold true for regression with deterministic prediction.

Original: This condition speciﬁes a rather strong constraint on the relation between the conditional probability density p X | AY ( x | a, y ) and the deterministic function f :
Revised: This condition speciﬁes a rather strong constraint on the relation between the conditional probability density p X | AY ( x | a, y ) and the deterministic function f :

Original: A × X → Y .
Revised: A × X → Y .

Original: In order to further illustrate this constraint, let us consider a special case where variables are linear correlated with Gaussian exogenous terms (i.e., E X and E = E Y + dE H in[Figure] M C [Figure]
Revised: In order to further illustrate this constraint, let us consider a special case where variables are linear correlated with Gaussian exogenous terms (i.e., E X and E = E Y + dE H in

Original: Equation 11 are Gaussian) and (cid:98)
Revised: Equation 11 are Gaussian) and (cid:98)

Original: Y =
Revised: Y =

Original: f ( A, X ) is a linear function of ( A, X ) with linear coefﬁcients α and non-zero β (i [Figure] e., (cid:98)
Revised: f ( A, X ) is a linear function of ( A, X ) with linear coefﬁcients α and non-zero β (i.e., (cid:98)

Original: Y = αA + βX ).
Revised: Y = αA + βX ).

Original: Here, the conditional distribution of X given A and that of Y given A are both Gaussian.
Revised: Here, the conditional distribution of X given A and that of Y given A are both Gaussian.

Original: Then for any given value of ( (cid:98)
Revised: Then for any given value of ( (cid:98)

Original: Y , A ) we have X =
Revised: Y , A ) we have X =

Original: ( (cid:98)
Revised: ( (cid:98)

Original: Y − αA ) /β .
Revised: Y − αA ) /β .

Original: Therefore δ (cid:0) ˆ y − f
Revised: Therefore δ (cid:0) ˆ y − f

Original: ( a, x ) (cid:1) does not equal to 0 only on the singleton set { x | f ( a, x ) = ˆ y }
Revised: [Equation] a, x ) (cid:1) does not equal to 0 only on the singleton set { x | f ( a [Equation]

Original: = { (ˆ y − αa ) /β } .
Revised: [Equation]

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Then the condition in Theorem 2 requires that
Revised: Then the condition in Theorem 2 requires that

Original: [Equation]
Revised: [Equation]

Original: Since the conditional distribution of X given A = a and that of Y given A = a are both Gaussian ( N ( qa, σ 2 X ) , and N(cid:0)
Revised: Since the conditional distribution of X given A = a and that of Y given A = a are both Gaussian ( N ( qa, σ 2 X ) , and N

Original: [Equation]
Revised: [Equation]

Original: respectively), we have p X | AY ( x | a, y ) =
Revised: respectively), we have p X | AY ( x | a, y ) =

Original: [-√ 
-]
Revised: [-√ 
-]

Original: πσ 2 ay exp {

Original: [Equation]
Revised: [Equation]

Original: 2 2 σ
Revised: exp

Original: [Equation]
Revised: [Equation]

Original: } , where µ ay = qa +cσ 2 X
Revised: } , where µ ay = qa +

Original: [Equation]
Revised: [Equation]

Original: X
Revised: X

Original: [Equation]
Revised: [Equation]

Revised: example.

Revised: Here we tune the hyperparameter such that the fairness violation is as small as possible for the deterministic predictor.

Revised: As we can see, given a similar accuracy level for predictors optimized with fairness consideration (Equalized Odds), the stochastic classiﬁer has a much smaller empirical violation of fairness compared to its deterministic counterpart.

Original:  Then the Equation 27 hold true if and only if the absolute distance between ˆ y − αaβ and µ ay is not a function of a (for any ﬁxed y and ˆ y ), which yields the following relation:
Revised: [Table] Then the Equation 27 hold true if and only if the absolute distance between ˆ y − αaβ and µ ay is not a function of a (for any ﬁxed y and ˆ y ), which yields the following relation:

Original: [Equation]
Revised: [Equation]

Original: which is the exact relation between α and β derived from a different perspective in Corollary 9, whose original proof builds up on top of Theorem 1.
Revised: which is the exact relation between α and β derived from a different perspective in Corollary 9, whose original proof builds up on top of Theorem 1.

Original: As we can see in this example, in order to achieve Equalized Odds, the coefﬁcients of linear function f ( A, X ) = αA + βX have to satisfy a very speciﬁc relation.
Revised: As we can see in this example, in order to achieve Equalized Odds, the coefﬁcients of linear function f ( A, X ) = αA + βX have to satisfy a very speciﬁc relation.

Original: This indicates the general restrictiveness of the aforementioned constraint on the relation between p X | AY ( x | a, y ) and the deterministic function f , i.e., Equalized Odds cannot hold true.
Revised: This indicates the general restrictiveness of the aforementioned constraint on the relation between p X | AY ( x | a, y ) and the deterministic function f , i.e., Equalized Odds cannot hold true.

Original: C.2. Deterministic Classiﬁcation Examples
Revised: C.2. Deterministic Classiﬁcation Examples

Original: Theorem 4 speciﬁes the conditions for Equalized Odds to hold true for deterministic classiﬁcation.
Revised: Theorem 4 speciﬁes the conditions for Equalized Odds to hold true for deterministic classiﬁcation.

Original: Here we present concrete examples where those conditions can or cannot be satisﬁed.
Revised: Here we present concrete examples where those conditions can or cannot be satisﬁed.

Original: For the example.

Original: Here we tune the hyperparameter such that the fairness violation is as small as possible for the deterministic predictor.

Original: As we can see, given a similar accuracy level for predictors optimized with fairness consideration (Equalized Odds), the stochastic classiﬁer has a much smaller empirical violation of fairness compared to its deterministic counterpart.

Original: [Table]

Original:  purpose of simplifying the notation, let us consider cases when A , X , and Y are binary, and the joint probability of ( A, Y ) is speciﬁed as following:
Revised: For the purpose of simplifying the notation, let us consider cases when A , X , and Y are binary, and the joint probability of ( A, Y ) is speciﬁed as following:

Original: [Equation]
Revised: [Equation]

Original: Then in the special case when X ⊥⊥ A | Y , Equalized Odds can hold true if f is a function of only
Revised: Then in the special case when X ⊥⊥ A | Y , Equalized Odds can hold true if f is a function of only

Original: X .
Revised: X .

Original: For example, if f = 1 { X = 1 } , one can quickly verify that P ((cid:101) Y
Revised: For example, if f = 1 { X = 1 } , one can quickly verify that P (

Original: [Equation]
Revised: [Equation]

Original: |
Revised: , Y = 0) =

Original: [Equation] = [Equation] Y = 
Revised: P ( X = 1 | Y = 0) and that P (

Original: [Equation]
Revised: [Equation]

Original: =

Original: P ( X = 1 | Y = 0) and that P ( (cid:101)

Original: Y = 1 | A = a, Y = 1) =
Revised: Y = 1 | A = a, Y = 1) =

Original: P ( X = 1 | Y = 1) (for a ∈ { 0 , 1 } ), i.e.,
Revised: P ( X = 1 | Y = 1) (for a ∈ { 0 , 1 } ), i.e.,

Original: Equalized Odds holds true.
Revised: Equalized Odds holds true.

Original: If X ⊥(cid:54)⊥ A | Y , let us compare the following two cases:
Revised: If X ⊥(cid:54)⊥ A | Y , let us compare the following two cases:

Original: [Equation]
Revised: [Equation]

Revised: −

Revised: U

Revised: −

Revised: U [ − 0 .

Original:  Here, the joint distribution of ( A, Y ) is speciﬁed the same as in Equation 28.
Revised: [Table] [Table] [Equation] Here, the joint distribution of ( A, Y ) is speciﬁed the same as in Equation 28.

Original: In Equation 29, the conditional probability of X is crafted so that P ( X = 0 | A = 0 , Y = 0) =
Revised: In Equation 29, the conditional probability of X is crafted so that P ( X = 0 | A = 0 , Y = 0) =

Original: P ( X = 1 | A
Revised: P ( X = 1 | A

Original: = 1 , Y = 0) and P ( X = 0 | A = 0 , Y = 1) =
Revised: = 1 , Y = 0) and P ( X = 0 | A = 0 , Y = 1) =

Original: P ( X = 1 | A
Revised: P ( X = 1 | A

Original: = 1 , Y = 1) .
Revised: = 1 , Y = 1) .

Original: Then if we choose f = 1 { A = X } (or ﬂip the prediction, let f = 1 − 1 { A = X } ) to exploit this property, we can have a predictor that satisﬁes Equalized Odds.
Revised: Then if we choose f = 1 { A = X } (or ﬂip the prediction, let f = 1 − 1 { A = X } ) to exploit this property, we can have a predictor that satisﬁes Equalized Odds.

Original: One can quickly verify this: P ( (cid:101)
Revised: One can quickly verify this: P ( [Equation]

Original: Y = 1 | A = 0 , Y = 0) =
Revised: Y = 1 | A = 0 , Y = 0) =

Original: P ( X = 0 | A = 0 , Y = 0)
Revised: P ( X = 0 | A = 0 , Y = 0)

Original: = 0 .
Revised: = 0 .

Original: 6 = P ( X = 1 | A
Revised: 6 = P ( X = 1 | A

Original: = 1 , Y = 0) =
Revised: = 1 , Y = 0) =

Original: P ( (cid:101)
Revised: P ( [Equation]

Original: Y = 1 | A
Revised: Y = [Equation] | A

Original: = 1 , Y = 0) , 
Revised: = 1 , Y = 0) , and P ( (cid:101) Y = [Equation]

Original: and P ( (cid:101)
Revised: A [Equation]

Original:  Y = 1 | A = 0 , Y = 1) =
Revised: [Equation] Y = [Equation]

Original: P ( X = 0 | A = 0 , Y = 1) = 0 .
Revised: [Equation] 0 , Y = 1) = 0 .

Original: 2 = P ( X = 1 | A
Revised: 2 = P ( X = 1 | A

Original: = 1 , Y = 1) =
Revised: = 1 , Y = 1) =

Original: P ( (cid:101)
Revised: P ( (cid:101)

Original: Y = 1 | A
Revised: Y = 1 | A

Original: = 1 , Y = 1) .
Revised: = 1 , Y = 1) .

Original: However, there is no obvious reason to believe that the conditional probability of X given A and Y should satisfy such “crafted” property in general cases.In the case [Table] How often: p-value >
Revised: However, there is no obvious reason to believe that the conditional probability of X given A and Y should satisfy such “crafted” property in general cases.

Original: [Table]

Original: [Table]

Original: How often: p-value > 0 .

Original: [Table]

Original: of

Original: [Table] shown in Equation 30, one cannot ﬁnd an f that satisﬁes conditions (i) and (ii) in Theorem [Table] and therefore in this case there is no deterministic prediction function that satisﬁes Equalized Odds.
Revised: In the case shown in Equation 30, one cannot ﬁnd an f that satisﬁes conditions (i) and (ii) in Theorem 4, and therefore in this case there is no deterministic prediction function that satisﬁes Equalized Odds.

Original: C.3. Beneﬁt of Introducing Stochastic Prediction under Almost-the-Same Performance
Revised: C.3. Beneﬁt of Introducing Stochastic Prediction under Almost-the-Same Performance

Original: To illustrate the beneﬁt of utilizing stochastic predictors [Table] we compare deterministic and stochastic prediction schemes in terms of their quantitative violations of fairness, given almost the same prediction performance, in several speciﬁc classiﬁcation and regression scenarios.
Revised: To illustrate the beneﬁt of utilizing stochastic predictors, we compare deterministic and stochastic prediction schemes in terms of their quantitative violations of fairness, given almost the same prediction performance, in several speciﬁc classiﬁcation and regression scenarios.

Original: C.3.1. A B INARY C LASSIFICATION E XAMPLE
Revised: C.3.1. A B INARY C LASSIFICATION E XAMPLE

Original: We generate simulated data according to the joint distribution speciﬁed by Equation 28 and Equation 30 with a sample size of 500, and use logistic regression as our base model.
Revised: We generate simulated data according to the joint distribution speciﬁed by Equation 28 and Equation 30 with a sample size of 500, and use logistic regression as our base model.

Original: We utilize the deterministic predictor proposed by Baharlouei et al.
Revised: We utilize the deterministic predictor proposed by Baharlouei et al.

Original: and the stochastic predictor proposed by Agarwal et al.
Revised: and the stochastic predictor proposed by Agarwal et al.

Original: (2018), and numerically compare the fairness violation (here is the maximum difference between True/False Positive Rates across groups) given almost the same accuracy.
Revised: (2018), and numerically compare the fairness violation (here is the maximum difference between True/False Positive Rates across groups) given almost the same accuracy.

Original: We summarize the results of 10 repetitive experiments in Table 1.
Revised: We summarize the results of 10 repetitive experiments in Table 1.

Original: C.3.2. A L INEAR N ON -G AUSSIAN R EGRESSION E XAMPLE
Revised: C.3.2. A L INEAR N ON -G AUSSIAN R EGRESSION E XAMPLE

Original: We generated the simulated data according to Equation 11, where q = 0 .
Revised: We generated the simulated data according to Equation 11, where q = 0 .

Original: 7 , b = 0 .
Revised: 7 , b = 0 .

Original: 6 , c = 0 .
Revised: 6 , c = 0 .

Original: 9 , and d = 0 .
Revised: 9 , and d = 0 .

Original: We use the same set of nonlinear deterministic and stochastic predictors as we speciﬁed in Section B.2.1.
Revised: We use the same set of nonlinear deterministic and stochastic predictors as we speciﬁed in Section B.2.1.

Original: Similar to the previous example, we summarize the results for 10 repetitive experiments.
Revised: Similar to the previous example, we summarize the results for 10 repetitive [Table] How often: p-value >

Original:  For the purpose of demonstrating the inﬂuence from characteristics of the data itself, we present simulation results with various exogenous terms that are generated from different nonGaussian distributions in Table 2 (Uniform distribution) and Table 3 (Laplace distribution).
Revised: [Table] experiments. For the purpose of demonstrating the inﬂuence from characteristics of the data itself, we present simulation results with various exogenous terms that are generated from different nonGaussian distributions in Table 2 (Uniform distribution) and Table 3 (Laplace distribution).

Original: Because the violation of fairness for regression is much easier observed via hypothesis test results, we present the frequency of p-value (for KCI test) exceeding 0.05 threshold (the more often, the more probable the predictor is fair).
Revised: Because the violation of fairness for regression is much easier observed via hypothesis test results, we present the frequency of p-value (for KCI test) exceeding 0.05 threshold (the more often, the more probable the predictor is fair).

Original: We can see that given a similar MSE level, the stochastic regressors are more probable to be fair compared to the deterministic ones.
Revised: We can see that given a similar MSE level, the stochastic regressors are more probable to be fair compared to the deterministic ones.

Original: Appendix D. An Example of Unwanted Usage of Stochastic Prediction
Revised: Appendix D. An Example of Unwanted Usage of Stochastic Prediction

Original: For classiﬁcation, while randomization can ensure group level of fairness, there is still some inherent shortcoming of the criterion that we should pay attention to.
Revised: For classiﬁcation, while randomization can ensure group level of fairness, there is still some inherent shortcoming of the criterion that we should pay attention to.

Original: For example, in the FICO case study in
Revised: For example, in the FICO case study in

Original: Hardt et al.
Revised: Hardt et al.

Original: (2016), for a speciﬁc client from certain demographic group, the decision of approve/deny the loan actually comes in two folds: if his/her credit score is above (below) the upper (lower) threshold, the bank approve (deny) the application for sure; if the score falls in the interval between two thresholds, the bank would ﬂip a coin to make a decision.
Revised: (2016), for a speciﬁc client from certain demographic group, the decision of approve/deny the loan actually comes in two folds: if his/her credit score is above (below) the upper (lower) threshold, the bank approve (deny) the application for sure; if the score falls in the interval between two thresholds, the bank would ﬂip a coin to make a decision.

Original: Then we can imagine the following situation when a client whose credit score falls within the interval between the upper and lower thresholds goes to a bank to apply a loan.
Revised: Then we can imagine the following situation when a client whose credit score falls within the interval between the upper and lower thresholds goes to a bank to apply a loan.

Original: He/she can ask (if conditions permit) the bank to run the model multiple times until the decision is approval.
Revised: He/she can ask (if conditions permit) the bank to run the model multiple times until the decision is approval.

Original: This would make the randomization that was built into the system for the sake of fairness no longer effective.
Revised: This would make the randomization that was built into the system for the sake of fairness no longer effective.

Original: The system in fact only has one ﬁxed threshold (i.e., the original lower threshold), which is in essence a deterministic predictor (therefore in general, does not satisfy Equalized Odds).
Revised: The system in fact only has one ﬁxed threshold (i.e., the original lower threshold), which is in essence a deterministic predictor (therefore in general, does not satisfy Equalized Odds).