Original: Benchmarking Bias Mitigation Algorithms in Representation Learning through Fairness Metrics
Revised: Benchmarking Bias Mitigation Algorithms in Representation Learning through Fairness Metrics

Original: Anonymous authors Paper under double-blind review
Revised: Anonymous authors Paper under double-blind review

Original: Abstract
Revised: Abstract

Original: With the recent expanding attention of machine learning researchers and practitioners to fairness, there is a void of a common framework to analyze and compare the capabilities of proposed models in deep representation learning.
Revised: With the recent expanding attention of machine learning researchers and practitioners to fairness, there is a void of a common framework to analyze and compare the capabilities of proposed models in deep representation learning.

Original: In this paper, we evaluate diﬀerent fairness methods trained with deep neural networks on a common synthetic dataset to obtain a better insight into the working of these methods.
Revised: In this paper, we evaluate diﬀerent fairness methods trained with deep neural networks on a common synthetic dataset to obtain a better insight into the working of these methods.

Original: In particular, we train about 2000 diﬀerent models in various setups, including unbalanced and correlated data conﬁgurations, to verify the limits of the current models and better understand in which setups they are subject to failure.
Revised: In particular, we train about 2000 diﬀerent models in various setups, including unbalanced and correlated data conﬁgurations, to verify the limits of the current models and better understand in which setups they are subject to failure.

Original: In doing so we present a dataset, a large subset of proposed fairness metrics in the literature, and rigorously evaluate recent promising debiasing algorithms in a common framework hoping the research community would take this benchmark as a common entry point for fair deep learning.
Revised: In doing so we present a dataset, a large subset of proposed fairness metrics in the literature, and rigorously evaluate recent promising debiasing algorithms in a common framework hoping the research community would take this benchmark as a common entry point for fair deep learning.

Original:  introduction
Revised:  introduction

Original: The emergence of deep learning models has brought up questions on the fairness of these models with respect to sensitive attributes.
Revised: The emergence of deep learning models has brought up questions on the fairness of these models with respect to sensitive attributes.

Original: There have been studies on the reliability and bias of deep learning approaches, such as the bias of the learning algorithms due to matching a data distribution in adversarial models (Cohen et al., 2018), visual question answering (Agrawal et al., 2018), image search setups (Kay et al., 2015), and gender classiﬁcation (Buolamwini & Gebru, 2018).
Revised: There have been studies on the reliability and bias of deep learning approaches, such as the bias of the learning algorithms due to matching a data distribution in adversarial models (Cohen et al., 2018), visual question answering (Agrawal et al., 2018), image search setups (Kay et al., 2015), and gender classiﬁcation (Buolamwini & Gebru, 2018).

Original: Recent approaches (Madras et al., 2018; Zhao et al., 2020; Creager et al., 2019; Zhang et al., 2018) address the bias in deep learning models and propose fair models that better guarantee fairness criterion while maintaining accuracy.
Revised: Recent approaches (Madras et al., 2018; Zhao et al., 2020; Creager et al., 2019; Zhang et al., 2018) address the bias in deep learning models and propose fair models that better guarantee fairness criterion while maintaining accuracy.

Original: The general idea in these models is applying a fairness learning technique either on the learned latent representation (Madras et al., 2018; Zhao et al., 2020; Creager et al., 2019) or the output eligibility criterion (Zhang et al., 2018), given an input.
Revised: The general idea in these models is applying a fairness learning technique either on the learned latent representation (Madras et al., 2018; Zhao et al., 2020; Creager et al., 2019) or the output eligibility criterion (Zhang et al., 2018), given an input.

Original: Adversarial learning techniques (Goodfellow et al., 2014; Ganin & Lempitsky, 2015) have been extensively used to learn the data distribution of interest, such as learning disentangled latent space (Kim & Mnih, 2018; Chen et al., 2018; 2016), or subtraction of a feature from the latent space (Lample et al., 2017; Denton et al., 2017).
Revised: Adversarial learning techniques (Goodfellow et al., 2014; Ganin & Lempitsky, 2015) have been extensively used to learn the data distribution of interest, such as learning disentangled latent space (Kim & Mnih, 2018; Chen et al., 2018; 2016), or subtraction of a feature from the latent space (Lample et al., 2017; Denton et al., 2017).

Original: Adversarial learning has also been used in fairness models to either remove sensitive information from the latent space (Madras et al., 2018;
Revised: Adversarial learning has also been used in fairness models to either remove sensitive information from the latent space (Madras et al., 2018;

Original: Zhao et al., 2020; Zhang et al., 2018) or disentangle latent features into sensitive and non-sensitive attributes (Creager et al., 2019).
Revised: Zhao et al., 2020; Zhang et al., 2018) or disentangle latent features into sensitive and non-sensitive attributes (Creager et al., 2019).

Original: The goal in these approaches is to remove the sensitive information, such as gender, from the latent space that is later used for other tasks, such as eligibility classiﬁcation.
Revised: The goal in these approaches is to remove the sensitive information, such as gender, from the latent space that is later used for other tasks, such as eligibility classiﬁcation.

Original: In this paper, we aim to verify whether well-performing and ubiquitous models can eﬀectively reduce bias when evaluated using well-established fairness metrics.
Revised: In this paper, we aim to verify whether well-performing and ubiquitous models can eﬀectively reduce bias when evaluated using well-established fairness metrics.

Original: Here we focus on deep learning models for classiﬁcation and their adversarial bias mitigation counterparts.
Revised: Here we focus on deep learning models for classiﬁcation and their adversarial bias mitigation counterparts.

Original: The diﬃculty of a task is determined by the distribution of the dataset, which in turn, is a function of the label imbalance, features imbalance, dependency of sensitive attributes to the rest of features, and distribution shift from training to development phase to name a few.
Revised: The diﬃculty of a task is determined by the distribution of the dataset, which in turn, is a function of the label imbalance, features imbalance, dependency of sensitive attributes to the rest of features, and distribution shift from training to development phase to name a few.

Original: In addition, in the current state of fairness community problems with reproducibility and inconsistency in the experimentation and dataset setups makes comparison diﬃcult.
Revised: In addition, in the current state of fairness community problems with reproducibility and inconsistency in the experimentation and dataset setups makes comparison diﬃcult.

Original: Given the importance and possibly tangible impact of proposed algorithms when in production, we advocate for a rigorous and uniﬁed evaluation of the capabilities of debiasing models.
Revised: Given the importance and possibly tangible impact of proposed algorithms when in production, we advocate for a rigorous and uniﬁed evaluation of the capabilities of debiasing models.

Original: We argue the need for a systematic analysis that would assess diﬀerent bias mitigation approaches under the perspective of diﬀerent fairness metrics.
Revised: We argue the need for a systematic analysis that would assess diﬀerent bias mitigation approaches under the perspective of diﬀerent fairness metrics.

Original: This would help elucidate the most promising research contributions.
Revised: This would help elucidate the most promising research contributions.

Revised: To analyse debiasing models we propose a dataset that facilitates creation of diﬀerent biased setups, such as data imbalance or correlation among eligibility and sensitive or non-sensitive attributes.

Revised: In particular, contrary to real datasets where the intensity of each feature cannot be controlled, this dataset allows changing only one component while keeping all other components unchanged, something that allows study of diﬀerent variations of biased setups.

Original: To this end, we provide an in-depth analysis of baselines and recently proposed bias mitigation models in the literature.
Revised: To evaluate models under diﬀerent biased setups, we provide an in-depth analysis of baselines and recently proposed bias mitigation models in the literature.

Original: In particular, we evaluate three promising debiasing models (six variants) together with a baseline model using a uniﬁed set of fairness metrics and report results by carrying extensive hyperparameter search in all cases, ensuring that the drawn conclusions can be attributed to modeling or loss choices.
Revised: In particular, we evaluate three promising debiasing models (six variants) together with a baseline model using a uniﬁed set of fairness metrics and report results by carrying extensive hyper-parameter search in all cases, ensuring that the drawn conclusions can be attributed to modeling or loss choices.

Original: In doing so, we evaluate these models under diﬀerent setups by training aboutmodels, in which we transition from balanced setups towards challenging unbalanced and correlated setups, where eligibility criterion is correlated with sensitive or non-sensitivedataset attributes.
Revised: In doing so, we evaluate these models under diﬀerent setups by training about 2000 models, in which we transition from balanced setups towards challenging unbalanced and correlated setups, where eligibility criterion is correlated with sensitive or non-sensitive attributes.

Original: Given the importance of fairness and the serious implications of their misuse, we intentionally try to push these models to their breaking point using extreme settings from our dataset.
Revised: Given the importance of fairness and the serious implications of their misuse, we intentionally try to push these models to their breaking point using extreme settings from our dataset.

Original: We consider diﬀerent ratios for each sensitive group to create “unfair datasets”.
Revised: We consider diﬀerent ratios for each sensitive group to create “unfair datasets”.

Original: This is noteworthy that our analysis is not to undermine the eﬀectiveness of any of these methods, rather setting the expectations and boundaries for diﬀerent use cases and encouraging the community to follow.
Revised: This is noteworthy that our analysis is not to undermine the eﬀectiveness of any of these methods, rather setting the expectations and boundaries for diﬀerent use cases and encouraging the community to follow.

Original: To summarize, our contributions are:
Revised: To summarize, our contributions are:

Original: • We show when there is a correlation between eligibility and the sensitive attribute, models exploit it for prediction of accuracy and use it even in test cases when such correlation does not exist, which leads to unfair predictions.
Revised: • We show when there is a correlation between eligibility and the sensitive attribute, models exploit it for prediction of accuracy and use it even in test cases when such correlation does not exist, which leads to unfair predictions.

Original: • We show that even when there are non-sensitive attributes in the dataset or small visual features in the input image that correlate with eligibility, models can still exploit it and hencebe biased.
Revised: • We show that even when there are non-sensitive attributes in the dataset or small visual features in the input image that correlate with eligibility, models can still exploit it and hence

Original:  • We demonstrate that the choice of seed can aﬀect the results signiﬁcantly and diﬀerent models show a considerable variation of performance given the seed.
Revised: be biased. • We demonstrate that the choice of seed can aﬀect the results signiﬁcantly and diﬀerent models show a considerable variation of performance given the seed.

Original: • We provide a deep learning codebase composed of six debiasing models and a baseline to the fairness community for research and evaluation of fairness models.
Revised: • We provide a deep learning codebase composed of six debiasing models and a baseline to the fairness community for research and evaluation of fairness models.

Original: • We provide a dataset with diﬀerent controllable sets of features and correlation among them, to facilitate research on fairness models in diﬀerent unfair setups.
Revised: • We provide a dataset with diﬀerent controllable sets of features and correlation among them, to facilitate research on fairness models in diﬀerent unfair setups.

Original:  Related Works
Revised:  Related Works

Original: There is an increasing literature studying how biased datasets can bias learning algorithms to discriminate (Bolukbasi et al., 2016;
Revised: There is an increasing literature studying how biased datasets can bias learning algorithms to discriminate (Bolukbasi et al., 2016;

Original: Cohen et al., 2018;
Revised: Cohen et al., 2018;

Original: Agrawal et al., 2018).
Revised: Agrawal et al., 2018).

Original: These studies have led researchers to propose new evaluation tools and datasets (Buolamwini & Gebru, 2018), with the goal of identifying potential machine learning error rate gaps among diﬀerent groups.
Revised: These studies have led researchers to propose new evaluation tools and datasets (Buolamwini & Gebru, 2018), with the goal of identifying potential machine learning error rate gaps among diﬀerent groups.

Original: To remedy this issue, numerous contributions have emerged in order to probe machine learning systems at diﬀerent levels and reduce discrimination from the perspective of modeling.
Revised: To remedy this issue, numerous contributions have emerged in order to probe machine learning systems at diﬀerent levels and reduce discrimination from the perspective of modeling.

Original: Following this direction, Friedler et al.
Revised: Friedler et al.

Original: (2019) evaluate the fairness of pre-processing, in-processing, and post-processing machine learning approaches; Kamishima et al.
Revised: (2019) evaluate the fairness of pre-processing, in-processing, and post-processing machine learning approaches; however, no deep learning model is evaluated in their study.

Original:  (2012) propose a regularization approach for models with probabilistic discriminative nature and evaluate it on logistic regression models; (Zafar et al., 2017a) proposes a ﬂexible mechanism to optimize accuracy in the presence of non-convex fairness constraints (or vice-versa, optimize fairness with accuracy constraints) and evaluate it on logistic regressions and support vector machines.
Revised: Moreover, the models are not evaluated while changing the level of dataset bias or correlation among features. Kamishima et al. (2012) propose a regularization approach for models with probabilistic discriminative nature and evaluate it on logistic regression models; (Zafar et al., 2017a) proposes a ﬂexible mechanism to optimize accuracy in the presence of non-convex fairness constraints (or vice-versa, optimize fairness with accuracy constraints) and evaluate it on logistic regressions and support vector machines.

Original: In addition, several contributions have attempted to leverage adversarial learning to mitigate biases in learned representations or in classiﬁers.
Revised: In addition, several contributions have attempted to leverage adversarial learning to mitigate biases in learned representations or in classiﬁers.

Original: Adversarial learning was originally introduced within the framework of Generative Adversarial Networks (GANs) (Goodfellow et al., 2014), where the generator network tries to produce samples that fool the discriminator network, and the discriminator tries to discriminate between generated samples and real ones.
Revised: Adversarial learning was originally introduced within the framework of Generative Adversarial Networks (GANs) (Goodfellow et al.

Original: This training procedure has been leveraged in the bias mitigation frameworks to make diﬀerent groups indistinguishable from one another with respect to a sensitive attribute.
Revised: , 2014)and it has been leveraged in the bias mitigation frameworks to make diﬀerent groups indistinguishable from one another with respect to a sensitive attribute.

Original: In particular, (Edwards & Storkey, 2016) learns adversarially to debias encoded features to be used in classiﬁcation tasks, while being fair with respect to demographic parity.
Revised: In particular, (Edwards & Storkey, 2016) learns adversarially to debias encoded features to be used in classiﬁcation tasks, while being fair with respect to demographic parity.

Original: Madras et al.
Revised: Madras et al.

Original: (2018) adopt a group normalized (cid:96) 1 loss and adapt it for each fairness metric of interest.
Revised: (2018) adopt a group normalized (cid:96) 1 loss and adapt it for each fairness metric of interest.

Original: Beutel et al.
Revised: Beutel et al.

Original: (2017) debias the encoded features using an adversarial training procedure for classiﬁcation tasks to achieve equality of opportunity.
Revised: (2017) debias the encoded features using an adversarial training procedure for classiﬁcation tasks to achieve equality of opportunity.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2018) directly pass the classiﬁer’s output, rather than the encoded features, to a discriminator to debias the classiﬁer with respect to diﬀerent fairness metrics.
Revised: (2018) directly pass the classiﬁer’s output, rather than the encoded features, to a discriminator to debias the classiﬁer with respect to diﬀerent fairness metrics.

Original: Despite the recent progress in fair deep learning models, the comparison of bias mitigation algorithms in a common setup is not straightforward, especially in biased and correlated dataset setups.
Revised: Despite the recent progress in fair deep learning models, the comparison of bias mitigation algorithms in a common setup is not straightforward, especially in biased and correlated dataset setups.

Original: Verma & Rubin (2018) evaluate how fair an oﬀ-the-shelf logistic regression model is, given a set of fairness metric deﬁnitions.
Revised: Verma & Rubin (2018) evaluate how fair an oﬀ-the-shelf logistic regression model is, given a set of fairness metric deﬁnitions.

Original: In this work, we evaluate some of the most recent and promising deep learning models in a common deep representation setup.
Revised: In this work, we evaluate some of the most recent and promising deep learning models in a common deep representation setup.

Original: Adversarial techniques can be categorized into three main approaches: (i) applying debiasing through adversarial training directly on the class label, where the class label is an indicator of eligibility (Zhang et al., 2018), (ii) mitigating bias through enforcing group-fairness on the learned latentspace, where the learned latent space  is directly used for classiﬁcation of eligibility (Beutel et al., 2017; 
Revised: Adversarial techniques can be categorized into three main approaches: (i) applying debiasing through adversarial training directly on the class label, where the class label is an indicator of eligibility (Zhang et al., 2018), (ii) mitigating bias through enforcing group-fairness on the learned latent space where it is directly used for classiﬁcation of eligibility (Beutel et al., 2017; Madras et al., 2018;

Original: Madras et al., 2018; Zhao et al., 2020), and (iii)  disentangling learned latent space into sensitive and non-sensitive representation features, and using only non-sensitive features for downstream task of determining the eligibility (Creager et al., 2019; Träuble et al., 2020).
Revised: Zhao et al., 2020), and (iii) discarding sensitive representation features for downstream tasks after disentangling learned latent space into sensitive and non-sensitive ones (Creager et al., 2019; Träuble et al., 2020).

Original: This line of research is motivated by recent development in disentangled representation learning (Kim & Mnih, 2018; Chen et al., 2018; Locatello et al., 2019).
Revised: This line of research is motivated by recent development in disentangled representation learning (Kim & Mnih, 2018; Chen et al., 2018; Locatello et al., 2019).

Original: In this paper, we take some of the promising models from the above-mentioned categories and assess their merits in challenging scenarios, in particular, in under-represented and correlated dataset setups.
Revised: In this paper, we take some of the promising models from the above-mentioned categories and assess their merits in challenging scenarios, in particular, in under-represented and correlated dataset setups.

Original: This exercise is not to undermine the applicability or achievements of these models but rather to understand the robustness and boundaries of these models under a variety of settings.
Revised: This exercise is not to undermine the applicability or achievements of these models but rather to understand the robustness and boundaries of these models under a variety of settings.

Original:  Models
Revised:  Models

Original: In this study, we empirically analyze the performance of a baseline MLP (multi-layer perceptron) model without any bias mitigation learning criteria and compare it with LAFTR (Madras et al., 2018), which applies adversarial learning to achieve group fairness, CFAIR (Zhao et al., 2020) which performs conditional alignment of representations to achieve a better accuracy-fairness trade-oﬀ, and ﬁnally, FFVAE (Creager et al., 2019), which disentangles latent representation into sensitive and non-sensitive attributes.
Revised: In this study, we empirically analyze the performance of a baseline MLP (multi-layer perceptron) model without any bias mitigation learning criteria and compare it with LAFTR (Madras et al., 2018), which applies adversarial learning to achieve group fairness, CFAIR (Zhao et al., 2020) which performs conditional alignment of representations to achieve a better accuracy-fairness trade-oﬀ, and ﬁnally, FFVAE (Creager et al., 2019), which disentangles latent representation into sensitive and non-sensitive attributes.

Original: For simplicity, we introduce the following shorthand notation.
Revised: For simplicity, we introduce the following shorthand notation.

Original: We denote by X ⊆ R d and Y = { 0 , 1 } the set of inputs and outputs, respectively.
Revised: We denote by X ⊆ R d and Y = { 0 , 1 } the set of inputs and outputs, respectively.

Original: Two sets of random variables X and Y take associated values x ∈ X and y ∈ Y .
Revised: Two sets of random variables X and Y take associated values x ∈ X and y ∈ Y .

Original: Variable y determines eligibility which in our MNIST-based dataset refers to a digit is even or odd.
Revised: Variable y determines eligibility which in our MNIST-based dataset refers to a digit is even or odd.

Original: We denote the sensitive binary variable by S taking values s ∈ { 0 , 1 } .
Revised: We denote the sensitive binary variable by S taking values s ∈ { 0 , 1 } .

Original: Moreover, p indicates the output probability of the classiﬁer as a real number and ˆ Y is the class predicted random variable based on p , which takes values ˆ y ∈ { 0 , 1 } using a threshold.
Revised: Moreover, p indicates the output probability of the classiﬁer as a real number and ˆ Y is the class predicted random variable based on p , which takes values ˆ y ∈ { 0 , 1 } using a threshold.

Original: For each s and y , the term D ys is the conditional distribution of the joint distribution D over X × Y × S , given
Revised: For each s and y , the term D ys is the conditional distribution of the joint distribution D over X × Y × S , given

Original: Y =
Revised: Y =

Original: y , S
Revised: y , S

Original: = s .
Revised: = s .

Original: Baseline Model.
Revised: Baseline Model.

Original: We use an MLP as our baseline model, which given an input image x , predicts the probability p of the input being odd or even.
Revised: We use an MLP as our baseline model, which given an input image x , predicts the probability p of the input being odd or even.

Original: This probability is then transformed into a classiﬁcation prediction ˆ y .
Revised: This probability is then transformed into a classiﬁcation prediction ˆ y .

Original: This model does not leverage any bias mitigation algorithm and is meant to show how fair a base deep learning model would perform under diﬀerent fairness criteria.
Revised: This model does not leverage any bias mitigation algorithm and is meant to show how fair a base deep learning model would perform under diﬀerent fairness criteria.

Original: Learning Adversarially Fair and Transferable Representations (LAFTR).
Revised: Learning Adversarially Fair and Transferable Representations (LAFTR).

Original: LAFTR (Madras et al., 2018) is an adversarial based bias mitigation algorithm within the scope of representation learning.
Revised: LAFTR (Madras et al., 2018) is an adversarial based bias mitigation algorithm within the scope of representation learning.

Original: In the supervised version of LAFTR, given an input image x , it ﬁrst learns a latent encoded representation z that is passed to the discriminator to be debiased.
Revised: In the supervised version of LAFTR, given an input image x , it ﬁrst learns a latent encoded representation z that is passed to the discriminator to be debiased.

Original: The learned representation is then passed to a classiﬁer to predict the task of interest y .
Revised: The learned representation is then passed to a classiﬁer to predict the task of interest y .

Original: The discriminator is trained by minimizing
Revised: The discriminator is trained by minimizing

Original: [Equation]
Revised: [Equation]

Original:  where y is only passed in debiasing models aimed for equality of odds and equality of opportunity .
Revised: [Equation] where y is only passed in debiasing models aimed for equality of odds and equality of opportunity .

Original: The encoder and classiﬁer are trained jointly by minimizing
Revised: The encoder and classiﬁer are trained jointly by minimizing

Original: [Equation]
Revised: [Equation]

Original: where z = E ( x ) is the encoded feature, which is passed to both the classiﬁer C and the discriminator
Revised: where z = E ( x ) is the encoded feature, which is passed to both the classiﬁer C and the discriminator

Original: D .
Revised: D .

Original: The ﬁrst term measures the classiﬁcation loss and the second term gets the adversarial gradients from the discriminator regarding the sensitive attribute s .
Revised: The ﬁrst term measures the classiﬁcation loss and the second term gets the adversarial gradients from the discriminator regarding the sensitive attribute s .

Original: Following the original paper, we considered four variants of this model.
Revised: Following the original paper, we considered four variants of this model.

Original: LAFTR-DP objective is deﬁned as
Revised: LAFTR-DP objective is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: and LAFTR-EqOpp0 objective is considered as
Revised: and LAFTR-EqOpp0 objective is considered as

Original: [Equation]
Revised: [Equation]

Original: L EqOpp1 S is obtained by replacing y = 1 in Eq.
Revised: L [Equation] is obtained by replacing y = 1 in Eq.

Original: (4) and L EqOdd S is the sum of L EqOpp0 S and L EqOpp1 S .
Revised: (4) and L EqOdd S is the sum of L EqOpp0 S and L EqOpp1 S .

Original: Conditional Learning of Fair Representations (CFAIR).
Revised: Conditional Learning of Fair Representations (CFAIR).

Original: Proposed by (Zhao et al., 2020) this model leverages two adversarial networks h 0 and h 1 , predicting sensitive attribute s respectively for class labels Y = 0 and Y = 1 .
Revised: Proposed by (Zhao et al., 2020) this model leverages two adversarial networks h 0 and h 1 , predicting sensitive attribute s respectively for class labels Y = 0 and Y = 1 .

Original: CFAIR depends on an objective function called the balanced error rate (BER) (Feldman et al., 2015; Menon & Williamson, 2018), which guarantees small joint error across demographic groups.
Revised: CFAIR depends on an objective function called the balanced error rate (BER) (Feldman et al., 2015; Menon & Williamson, 2018), which guarantees small joint error across demographic groups.

Original: The BER represents the sum of false positive rate and false negative rate.
Revised: The BER represents the sum of false positive rate and false negative rate.

Original: Therefore, it is equal to minimizing the below two conditional errors.
Revised: Therefore, it is equal to minimizing the below two conditional errors.

Original: BER D ( ˆ Y (cid:107) Y ) is deﬁned as
Revised: BER D ( ˆ Y (cid:107) Y ) is deﬁned as

Original: BER
Revised: BER

Original: [Equation]
Revised: [Equation]

Original: and BER D ( ˆ S (cid:107) S ) is deﬁned similarly, where ˆ S is the predicted sensitive random variable.
Revised: and BER D ( ˆ S (cid:107) S ) is deﬁned similarly, where ˆ S is the predicted sensitive random variable.

Original: CFAIR is optimized based on the following min-max formulation.
Revised: CFAIR is optimized based on the following min-max formulation.

Original: [Equation]
Revised: [Equation]

Original: This approach proposes that using the balanced error rate along with the conditional alignment helps in achieving equalized odds across the groups without impacting demographic parity.
Revised: This approach proposes that using the balanced error rate along with the conditional alignment helps in achieving equalized odds across the groups without impacting demographic parity.

Original: Flexibly Fair Representation Learning by Disentanglement (FFVAE).
Revised: Flexibly Fair Representation Learning by Disentanglement (FFVAE).

Original: Inspired by FactorVAE (Kim & Mnih, 2018), FFVAE (Creager et al., 2019) performs disentanglement by factorizing latent space.
Revised: Inspired by FactorVAE (Kim & Mnih, 2018), FFVAE (Creager et al., 2019) performs disentanglement by factorizing latent space.

Original: It learns a disentangled representation of the inputs, which is ﬂexibly fair in the sense that it can be easily modiﬁed at test time to achieve demographic parity across various groups.
Revised: It learns a disentangled representation of the inputs, which is ﬂexibly fair in the sense that it can be easily modiﬁed at test time to achieve demographic parity across various groups.

Original: In our setup, x is the input image, which is considered to implicitly contain both sensitive and non-sensitive attributes.
Revised: In our setup, x is the input image, which is considered to implicitly contain both sensitive and non-sensitive attributes.

Original: Moreover, a = ( a 1 , . . . , a N ) , b = ( b 1 , . . .
Revised: Moreover, a = ( a 1 , . . . , a N ) , b = ( b 1 , . . .

Original: , b N ) , and z are the sensitive attribute, the sensitive latent, and non-sensitive latent of x , respectively, with N indicating the number of sensitive or non-sensitive features, depending on the dataset.
Revised: , b N ) , and z are the sensitive attribute, the sensitive latent, and non-sensitive latent of x , respectively, with N indicating the number of sensitive or non-sensitive features, depending on the dataset.

Original: FFVAE trains an encoder q ( z, b | x ) , a decoder p ( x | z, b ) , as well as an adversarial network.
Revised: FFVAE trains an encoder q ( z, b | x ) , a decoder p ( x | z, b ) , as well as an adversarial network.

Original: The latent representation is disentangled into sensitive and non sensitive latent attributes by encouraging both MI( b, z ) and MI ( b i , a j ) , ∀ i (cid:54) =
Revised: The latent representation is disentangled into sensitive and non sensitive latent attributes by encouraging both MI( b, z ) and MI ( b i , a j ) , ∀ i (cid:54) =

Original: j to be low, where MI represents mutual information.
Revised: j to be low, where MI represents mutual information.

Original: FFVAE objective is deﬁned as
Revised: FFVAE objective is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: (7) where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst term of Eq.
Revised: The ﬁrst term of Eq.

Original: (7) consists of a reconstruction term (on left) and a predictiveness term p ( a | b ) , which aligns sensitive attributes to its respective sensitive latents.
Revised: (7) consists of a reconstruction term (on left) and a predictiveness term p ( a | b ) , which aligns sensitive attributes to its respective sensitive latents.

Original: Eq.(8) has two terms; the disentanglement term on the left decorrelates the sensitive latent representation b from z using an adversarial network and the second term is the Kullback–Leibler divergence between the prior and the latent distribution.
Revised: Eq.(8) has two terms; the disentanglement term on the left decorrelates the sensitive latent representation b from z using an adversarial network and the second term is the Kullback–Leibler divergence between the prior and the latent distribution.

Original: In addition to the above-mentioned models, we also did experiments with (Zhang et al., 2018)
Revised: In addition to the above-mentioned models, we also did experiments with (Zhang et al., 2018)

Original: (based on the code released by authors), however, the model was very unstable on our dataset conﬁgurations even after extensive hyper-parameter search.
Revised: (based on the code released by authors), however, the model was very unstable on our dataset conﬁgurations even after extensive hyper-parameter search.

Original: We hypothesize that this is due to the application of adversarial training directly to the class labels, which makes the model unstable, as indicated by the authors.
Revised: We hypothesize that this is due to the application of adversarial training directly to the class labels, which makes the model unstable, as indicated by the authors.

Original: Due to unstable results, we dropped this model from our evaluation.
Revised: Due to unstable results, we dropped this model from our evaluation.

Original:  Setup
Revised:  Setup

Original: Machine learning algorithms have attracted much attention and found application in a wide range of highly risky domains such as medical tests and criminal justice.
Revised: Machine learning algorithms have attracted much attention and found application in a wide range of highly risky domains such as medical tests and criminal justice.

Original: It is important to make sure that the unfairness present in datasets (Cohen et al., 2018; Agrawal et al., 2018; Kay et al., 2015; Buolamwini  Gebru, 2018), is not learned and later inferred by powerful deep learning models used for decision making (Barocas & Selbst, 2016;
Revised: It is important to make sure that the unfairness present in datasets (Cohen et al., 2018; Agrawal et al., 2018; Kay et al., 2015; Buolamwini & Gebru, 2018), is not learned and later inferred by powerful deep learning models used for decision making (Barocas & Selbst, 2016;

Original: Berk et al., 2018; of the President, 2016).
Revised: Berk et al., 2018; of the President, 2016).

Original: One way to achieve this goal is to measure the ability of the current prominent debiasing models in ignoring the correlation existing in the datasets and evaluate these models in setups where we have a mechanism to control the correlations among dataset features.
Revised: One way to achieve this goal is to measure the ability of the current prominent debiasing models in ignoring the correlation existing in the datasets and evaluate these models in setups where we have a mechanism to control the correlations among dataset features.

Original: Colored-MNIST.
Revised: Colored-MNIST.

Original: In order to evaluate the debiasing models in challenging setups and be capable of controlling diﬀerent dataset conﬁgurations, we design a variant of the MNIST dataset (LeCun &
Revised: In order to evaluate the debiasing models in challenging setups and be capable of controlling diﬀerent dataset conﬁgurations, we design a variant of the MNIST dataset (LeCun &

Original: Cortes, 2010), called Colored-MNIST, where we introduce diﬀerent types of correlations between sensitive attribute, dataset features, and eligibility criterion.
Revised: Cortes, 2010), called Colored-MNIST, where we introduce diﬀerent types of correlations between sensitive attribute, dataset features, and eligibility criterion.

Original: For an input image x , the label y ∈ { 0 , 1 } indicates eligibility or ineligibility, given x is even or odd.
Revised: For an input image x , the label y ∈ { 0 , 1 } indicates eligibility or ineligibility, given x is even or odd.

Original: We use the background colors as the protected or sensitive attribute s ∈ { 0 , 1 } , where blue indicates the unprivileged group and red indicates the privileged group.
Revised: We use the background colors as the protected or sensitive attribute s ∈ { 0 , 1 } , where blue indicates the unprivileged group and red indicates the privileged group.

Original: The task of interest is to classify whether, given an input image x , the class label y is odd or even.
Revised: The task of interest is to classify whether, given an input image x , the class label y is odd or even.

Original: While a model improves its classiﬁcation accuracy, we aim at observing its performance given diﬀerent fairness metrics.
Revised: While a model improves its classiﬁcation accuracy, we aim at observing its performance given diﬀerent fairness metrics.

Original: We would like to also observe how the model’s classiﬁcation accuracy and fairness measures change in diﬀerent dataset scenarios.
Revised: We would like to also observe how the model’s classiﬁcation accuracy and fairness measures change in diﬀerent dataset scenarios.

Original: A common practice is to show reasonable performance on a few standard datasets.
Revised: A common practice is to show reasonable performance on a few standard datasets.

Original: These, however, do not cover the majority of possible distributions of a dataset that a model can encounter in real-world development and production.
Revised: These, however, do not cover the majority of possible distributions of a dataset that a model can encounter in real-world development and production.

Original: Here we are using this dataset as a benchmarking tool that will allow control over these factors to give a better sense of what a model is capable of in diﬀerent challenging setups.
Revised: Here we are using this dataset as a benchmarking tool that will allow control over these factors to give a better sense of what a model is capable of in diﬀerent challenging setups.

Original: 4.1 Dataset features
Revised: 4.1 Dataset features

Original: Label ( y ): The label indicates whether the digit is even or odd.
Revised: Label ( y ): The label indicates whether the digit is even or odd.

Original: In our dataset, even is eligible ( y = 1 ) and odd is ineligible ( y 
Revised: In our dataset, even is eligible ( y = 1 ) and odd is ineligible ( y = 0 ).

Original: = 0 ). Background color ( bck ): It refers to the background color (sensitive attribute) which is various shades of blue and red colors.
Revised: Background color ( bck ): It refers to the background color (sensitive attribute) which is various shades of blue and red colors.

Original: clr-ratio: shown in pair ( b e , b o ) , b e ( b o ) refers to the ratio of the even (odd) digits with blue backgrounds and the rest of the digits will have red backgrounds.
Revised: clr-ratio: shown in pair ( b e , b o ) , b e ( b o ) refers to the ratio of the even (odd) digits with blue backgrounds and the rest of the digits will have red backgrounds.

Original: Using this feature we control the correlation between the sensitive attribute and the eligibility.
Revised: Using this feature we control the correlation between the sensitive attribute and the eligibility.

Original: e-o-ratio: This feature indicates the number of drawn lines on the digit.
Revised: e-o-ratio: This feature indicates the number of drawn lines on the digit.

Original: e-ratio ( o-ratio ) refers to the ratio of even (odd) digits that contain 6 to 20 lines.
Revised: e-ratio ( o-ratio ) refers to the ratio of even (odd) digits that contain 6 to 20 lines.

Original: The rest of the digits have between 2 to 6 lines.
Revised: The rest of the digits have between 2 to 6 lines.

Original: Using e-o-ratio , we can completely correlate (1 , 0) or completely decorrelate (0 . 5 , 0 . 5)
Revised: Using e-o-ratio , we can completely correlate (1 , 0) or completely decorrelate (0 . 5 , 0 . 5)

Original: the number of lines on the digit (as a non-sensitive attribute) with the eligibility.
Revised: the number of lines on the digit (as a non-sensitive attribute) with the eligibility.

Original: Green width ( g w ): This value indicates the number of green pixels (either 1 or 5 ) in the image and is visualized as a small vertical line (see Figures 1 and 6).
Revised: Green width ( g w ): This value indicates the number of green pixels (either 1 or 5 ) in the image and is visualized as a small vertical line (see Figures 1 and 5).

Original: The location of green pixels is indicated by g 1 and g 2 .
Revised: The location of green pixels is indicated by g 1 and g 2 .

Original: g 1 : This variable indicates the location of the ﬁrst green column in the image and is set to one of { 2 , 22 } .
Revised: g 1 : This variable indicates the location of the ﬁrst green column in the image and is set to one of { 2 , 22 } .

Original: For each given even (odd) digit, the location of the ﬁrst green column is randomly drawn in the range of [0 , g 1 / 2) ( [ g 1 / 2 , g 1 ) ).
Revised: For each given even (odd) digit, the location of the ﬁrst green column is randomly drawn in the range of [0 , g 1 / 2) ( [ g 1 / 2 , g 1 ) ).

Original: Hence, if g 1 = 2 , the location of the green column is an easy-to-spot indicator of eligibility since the same pixel location ( 0 or 1 ) indicates eligibility/ineligibility, while if g 1 = 22 , the model needs to observe over many cases, hence it requires a stronger analysis across diﬀerent cases.g
Revised: Hence, if g 1 = 2 , the location of the green column is an easy-to-spot indicator of eligibility since the same pixel location ( 0 or 1 ) indicates eligibility/ineligibility, while if g 1 = 22 , the model needs to observe over many cases, hence it requires a stronger analysis across diﬀerent cases.

Original:  : This variable indicates the location of the second green column in the image and is set to g 1 + digit where digit varies from 0 to 
Revised: g 2 : This variable indicates the location of the second green column in the image and is set to g 1 + digit where digit varies from 0 to 9 .

Original:  By using g 1 , g 2 , and g w , we evaluate  when we have features that are visually small ( 1 or 5 pixels big), can models exploit the correlation between g 1 and eligibility, in an easy-to-spot case ( g 1 = 2 ) and a more challenging case ( g 1 = 22 )?
Revised: We used g 2 in all our experiments along with g 1 . By using g 1 , g 2 , and g w , we evaluate the following: when we have features that are visually small (or 5 pixels big), can models exploit the correlation between g 1 and eligibility, in an easy-to-spot case ( g 1 = 2 ) and a more challenging case ( g 1 = 22 )?

Original: 4.2 Fairness Metrics
Revised: 4.2 Fairness Metrics

Original: We use the same set of fairness metrics, as reported in LAFTR, CFAIR [Figure] and FFVAE, to evaluate the debiasing models.
Revised: We use the same set of fairness metrics, as reported in LAFTR, CFAIR, and FFVAE, to evaluate the debiasing models.

Original: Table 1 presents the fairness metrics as well as their mathematical notations and abbreviations.
Revised: Table 1 presents the fairness metrics as well as their mathematical notations and abbreviations.

Original: In Table 2 we provide a comprehensive list of fairness metrics in the literature, which we have implemented in our framework and will release to the community.
Revised: In Table 2 we provide a comprehensive list of fairness metrics in the literature, which we have implemented in our framework and will release to the community.

Original: Due to the limit of space and more widespread usage of the metrics presented in Table 1, we only report on these metrics.
Revised: Due to the limit of space and more widespread usage of the metrics presented in Table 1, we only report on these metrics.

Original: The provided tool by (Friedler et al., 2019) is used to compute all of the metrics [Figure]
Revised: The provided tool by (Friedler et al., 2019) is used to compute all of the metrics.

Original: [Table]
Revised: [Table]

Original:  Experimental Results
Revised:  Experimental Results

Original: In this section, we evaluate the baseline MLP model as well as the debiasing models (LAFTR-DP,
Revised: In this section, we evaluate the baseline MLP model as well as the debiasing models (LAFTR-DP,

Original: LAFTR-EqOpp0, LAFTR-EqOpp1, LAFTR-EqOdd, CFAIR, and FFVAE) on the Colored-MNIST dataset and evaluate their performance on diﬀerent fairness metrics.
Revised: LAFTR-EqOpp0, LAFTR-EqOpp1, LAFTR-EqOdd, CFAIR, and FFVAE) on the Colored-MNIST dataset and evaluate their performance on diﬀerent fairness metrics.

Original: The goal is to provide an evaluation baseline for diﬀerent debiasing models and meanwhile measure how fair they remain in unbalanced and biased situations.
Revised: The goal is to provide an evaluation baseline for diﬀerent debiasing models and meanwhile measure how fair they remain in unbalanced and biased situations.

Original: In particular, we change the dataset from a balanced setup to diﬀerent unbalanced setups.
Revised: In particular, we change the dataset from a balanced setup to diﬀerent unbalanced setups.

Original: In the balanced setup, the ratio of unprivileged (blue) and privileged (red) background is at 50% for both eligible and ineligible groups, giving clr-ratio of (0 . 5 , 0 . 5) , and the data features and eligibility are uncorrelated with e-o-ratio being (0 . 5 , 0 . 5) .
Revised: In the balanced setup, the ratio of unprivileged (blue) and privileged (red) background is at 50% for both eligible and ineligible groups, giving clr-ratio of (0 . 5 , 0 . 5) , and the data features and eligibility are uncorrelated with e-o-ratio being (0 . 5 , 0 . 5) .

Original: In the unbalanced cases, while we deviate from the balanced setup during training, at test time, we evaluate the models in a balanced setup, meaning the dataset is at 50% for eligible and ineligible groups as well as for privileged and unprivileged groups.
Revised: In the unbalanced cases, while we deviate from the balanced setup during training, at test time, we evaluate the models in a balanced setup, meaning the dataset is at 50% for eligible and ineligible groups as well as for privileged and unprivileged groups.

Original: This way, we report the model’s performance on a balanced setup of diﬀerent sub-groups.
Revised: This way, we report the model’s performance on a balanced setup of diﬀerent sub-groups.

Original: Experimental and model details are provided in Section B.
Revised: Experimental and model details are provided in Section B.

Original: Impact of reducing representation of unprivileged group.
Revised: Impact of reducing representation of unprivileged group.

Original: In this experiment, we change clrratio from the balanced setup of (0 . 5 , 0 . 5) to unbalanced cases of (0 . 1 , 0 . 1) , (0 . 01 , 0 . 01) , and (0 . 001 , 0 . 001) .
Revised: In this experiment, we change clrratio from the balanced setup of (0 . 5 , 0 . 5) to unbalanced cases of (0 . 1 , 0 . 1) , (0 . 01 , 0 . 01) , and (0 . 001 , 0 . 001) .

Original: In these setups, the unprivileged group with the blue background is less and less represented in both eligible and ineligible groups, hence evaluating the models where the unprivileged groups are under-represented, but represented equally in both eligible and ineligible groups.
Revised: In these setups, the unprivileged group with the blue background is less and less represented in both eligible and ineligible groups, hence evaluating the models where the unprivileged groups are under-represented, but represented equally in both eligible and ineligible groups.

Original: Figure 2 compares all models side-by-side and Tables 4 to 10 in the Supplementary in Section C.1 show  detailed results for each model.
Revised: Figure 2 compares two models side-by-side, Figure 6 and Tables 4 to 10 in the Supplementary in Section C.1 show all models and give detailed results.

Original: For each metric, the best value is reported, meaning the performance of the best-performing model on the fairness metric (chosen across hyper-parameters on the validation set) is reported on the test set.The result is then averaged over the seeds.
Revised: We initially averaged metrics over three seeds, then for each metric, the best value is reported, meaning the performance of the best-performing model on the fairness metric (chosen across hyper-parameters on the validation set) is reported on the test set.

Original: This would allow us to report the best performance on each metric, without choosing how to compromise between accuracy and a fairness metric (which usually have a trade-oﬀ and improving one deteriorates the other). 
Revised: This would allow us to report the best performance on each metric, without choosing how to compromise between accuracy and a fairness metric (which usually have a trade-oﬀ and improving one deteriorates the other). We want to show that there exists a bias even when the best variation of the model is considered.

Revised: Note that this is even in favor of the models as the best possible result is reported.

Revised: Hence in each column we report the best obtained result, which can correspond to a diﬀerent hyper-parameter or early-stopping point.

Original: As observed in the results, through the transition from balanced to unbalanced setups, both accuracy and fairness metrics drop.
Revised: As observed in the results, through the transition from balanced to unbalanced setups, both accuracy and fairness metrics drop.

Original: While LAFTR models maintain better fairness metrics compared to other models, they perform poorly on accuracy metrics.
Revised: While LAFTR models maintain better fairness metrics compared to other models, they perform poorly on accuracy metrics.

Original: After LAFTR, FFVAE and then CFAIR, can less and less preserve fairness metrics while in the same order they improve on accuracy.
Revised: After LAFTR, FFVAE and then CFAIR, can less and less preserve fairness metrics while in the same order they improve on accuracy.

Original: The baseline model (MLP), without any debiasing, as expected performs the best on the accuracy, while fails more than other models in maintaining fairness.
Revised: The baseline model (MLP), without any debiasing, as expected performs the best on the accuracy, while fails more than other models in maintaining fairness.

Original: Impact of correlation of sensitive attribute with eligibility.
Revised: Impact of correlation of sensitive attribute with eligibility.

Original: In this setup, we change clr-ratio from the balanced setup of (0 . 5 , 0 . 5) to unbalanced cases of (0 . 1 , 0 . 9) and (0 . 01 , 0 . 99) , where the unprivileged group is under-represented in the eligible and over-represented in the ineligible groups.
Revised: In this setup, we change clr-ratio from the balanced setup of (0 . 5 , 0 . 5) to unbalanced cases of (0 . 1 , 0 . 9) and (0 . 01 , 0 . 99) , where the unprivileged group is under-represented in the eligible and over-represented in the ineligible groups.

Original: In unbalanced settings, the resistance of models to the correlation between sensitive attribute (background color) and the eligibility metric is evaluated.
Revised: In unbalanced settings, the resistance of models to the correlation between sensitive attribute (background color) and the eligibility metric is evaluated.

Original: The goal is to evaluate  when such cases arise, do models perform fairly in test setups, where everything is completely balanced?
Revised: The goal is to evaluate the following: when such cases arise, do models perform fairly in test setups, where everything is completely balanced?

Original: and hence can they avoid carrying the correlation of training to test setups?
Revised: and hence can they avoid carrying the correlation of training to test setups?

Original: Figure 3 compares  models side-by-side and Tables 11 to 17 in Section C.2 of Supplementary show detailed results for the baseline and the debiasing models.
Revised: Figure 3 compares two models side-by-side, Figure 7 and Tables 11 to 17 in Section C.2 of Supplementary show all models and detailed results.

Original: In this setup, FFVAE preserves better the fairness metrics compared to other models, while
Revised: In this setup, FFVAE preserves better the fairness metrics compared to other models.

Original: LAFTR and CFAIR models perform similarly or sometimes even worse than the baseline model on fairness metrics, while the baseline model preserves better the accuracy.
Revised: LAFTR and CFAIR models perform similarly or sometimes even worse than the baseline model on fairness metrics, while the baseline model preserves better the accuracy.

Revised: Missing bars in the plots imply zero metric values and biased predictions on the balanced test set.

Original: The results show that even though debiasing models are provided with the sensitive attribute during training, they still fail to decorrelate the sensitive attribute from eligibility in highly correlated setups.
Revised: The results show that even though debiasing models are provided with the sensitive attribute during training, they still fail to decorrelate the sensitive attribute from eligibility in highly correlated setups.

Original: Impact of correlation of non-sensitive attribute with eligibility.
Revised: Impact of correlation of non-sensitive attribute with eligibility.

Original: In this setup, we keep clr-ratio at (0 . 5 , 0 . 5) , however, change e-o-ratio from balanced setting of (0 . 5 , 0 . 5) to unbalanced settings of (0 . 9 , 0 . 1) .
Revised: In this setup, we keep clr-ratio at (0 . 5 , 0 . 5) , however, change e-o-ratio from balanced setting of (0 . 5 , 0 . 5) to unbalanced settings of (0 . 9 , 0 . 1) .

Original: This would make the number of lines on the digits correlated with eligibility.
Revised: This would make the number of lines on the digits correlated with eligibility.

Original: The goal is to evaluate  to what extent a model can exploit the correlation between a non-sensitive attribute and the eligibility criterion and hence be biased at test time?
Revised: The goal is to evaluate the following: to what extent a model can exploit the correlation between a non-sensitive attribute and the eligibility criterion and hence be biased at test time?

Original: Figure 4 compares these models side-by-side and Tables 18 to 24 in Section C.3 show results for all models.
Revised: Figure 8 and Tables 18 to 24 in Section C.3 show results for all models.

Original: While MLP, CFAIR, and FFVAE do not show much change in this setup, LAFTR models show more deviation, indicating they are more sensitive to such correlations.
Revised: While MLP, CFAIR, and FFVAE do not show much change in this setup, LAFTR models show more deviation, indicating they are more sensitive to such correlations.

Original: We did not ﬁnd a particular trend there.

Original: While LAFTR-DP dropped in fairness measures,  LAFTR-EqOpp0’s performance improved.
Revised: For example, LAFTR-DP dropped in fairness measures, while LAFTR-EqOpp0’s performance improved.

Original: On the other hand, LAFTR-EqOpp and LAFTR-EqOdd were almost consistent in this transition.
Revised: As we show later, this is mainly due to the high variance of this model under diﬀerent seeds.

Original: Impact of small features in the input images.
Revised: Impact of small features in the input images.

Original: In this setup, we change g 1 from 22 , where the location of the green pixel has a more complex relation with eligibility, to 2 , where the location of the green pixel  is an easy-to-spot indicator of eligibility that can be more easily learned by models and hence induce bias.
Revised: In this setup, we change g 1 from 22 , where the location of the green pixel has a more complex relation with eligibility, to 2 , where the location of the green pixel (0 for eligibility and 1 for ineligibility) is an easy-to-spot indicator of eligibility that can be more easily learned by models and hence induce bias.

Original: The question is whether models ﬁnd such small but non-trivial correlated features with eligibility?
Revised: The question is whether models ﬁnd such small but non-trivial correlated features with eligibility?

Original: If the answer is yes, then information even in the small portion of the input can cause biased performance of the model.
Revised: If the answer is yes, then information even in the small portion of the input can cause biased performance of the model.

Original: In this setup we run an experiment where for each one of g 1 =  and g 1 = [-
-] we consider two cases with g w = 1 and g w = 5 , respectively showing only 1 and 5 green pixels in the image.
Revised: In this setup we run an experiment where for each one of g 1 = 2 and g 1 = [-
-]22 we consider two cases with g w = 1 and g w = 5 , respectively showing only 1 and 5 green pixels in the image.

Original: This setup would verify whether the model observes the green part of the image and if yes how much impact the size of that region will have.
Revised: This setup would verify whether the model observes the green part of the image and if yes how much impact the size of that region will have.

Original: Figure 4 shows results and Tables 25 to 31 show  detailed results for all models.
Revised: Figure 4 shows two models, Figure 9 and Tables 25 to 31 show two extra models and give detailed results for all models.

Original: While models like MLP, CFAIR, and FFVAE show a small diﬀerence in two setups, LAFTR models drop performance in the biased setup of g 1 = 2 , indicating this model is more prone to pick up on such small details, when it can spot it.
Revised: While models like MLP, CFAIR, and FFVAE show a small diﬀerence in two setups, LAFTR models become less fair in the biased setup of g 1 = 2 , indicating this model is more prone to pick up on such small details, when it can spot it.

Original: Results with g w = 5 show very close performance to g w = 1 , indicating even one pixel can be enough for models to extract small details from the images.
Revised: Results with g w = 5 show very close performance to g w = 1 , indicating even one pixel can be enough for models to extract small details from the images.

Original: The results show that when there are small and sometimes not-noticeable features in the dataset, they can cause bias in the learning algorithm.  
Revised: The results show that when there are small and sometimes not-noticeable features in the dataset, they can cause bias in the learning algorithm.  Discussion 6.1 Sources of Bias

Revised: In our experiments, we have considered four diﬀerent sources of bias.

Revised: Small percentage of the unprivileged group.

Revised: This case causes bias due to the small ratio of the unprivileged group compared to the privileged group.

Revised: In this case, the bias can be two-fold, one is due to the imbalance between the two groups, another can be due to rareness of the data from the unprivileged group.

Original: Correlation between dataset features and model’s prediction.
Revised: While the clr-ratios (0.1, 0.1), (0.01, 0.01), and (0.001, 0.001), all are subject to group imbalance, the latter two cases have 250 andsamples from the unprivileged group, compared to 24,750, and 24,975 samples respectively from the privileged group. So, the model has to also handle the secondary aspect of the bias, which is diﬀerent. Correlation of a feature with eligibility.

Revised: In some data the unprivileged group might be less eligible, e.g. due to historic reasons of lack of access to facilities or limited resources compared to privileged group.

Revised: Such correlations in the data can be picked up by the model and cause bias at inference.

Revised: Impact of size of the correlated visual features.

Revised: While large features that correlated with eligibility and cause bias can be easily observed by a model, small visual features can also have correlations with eligibility.

Revised: Given the size of the correlated features, some models can act more fair while others might remain biased.

Revised: Impact of unnoticed and small features.

Revised: The debiasing algorithms work based on the assumption that the sensitive or biased attributes are known a-priori and hence can be addressed by knowing these features and removing them from the learned representation.

Revised: However, some features, especially small ones, might not be noticed by the model or the annotator.

Revised: For example, wearing earrings, can correlate with gender, or glasses with age.

Revised: Such cases might not be observed and addressed properly by the debiasing model, if they are not know a-priori.

Revised: We have studied the impact of the four above-mentioned sources of bias, and have observed a stronger level of bias in the ﬁrst two cases.

Revised: Specially, the correlation of big visual sensitive attributes with eligibility, and imbalance among the privileged and unprivileged groups lead to a higher bias.

Revised: This trend was observed in all models.

Revised: We observed smaller visual components that are correlated with eligibility cause less bias, as the model can more easily get rid of them.

Revised: Some models such as LAFTR suﬀered still in these cases.

Revised: We believe study of such sources of bias can help better evaluate the debiasing models and advocate their study in future works.

Revised: 6.2 Model Stability

Revised: Variation due to seed.

Revised: In deep learning models, seed is a great source of variation due to initialization of the model parameters and the sampling done during training, which can converge the model to diﬀerent solutions in the course of training.

Revised: To observe its impact, we evaluated how much standard deviation changes when the seed used for a model diﬀers.

Revised: The results for three seeds considered are shown in Figure 10 for all of the setups described in our experiments.

Revised: We observed the choice of seed can impact some models more than others.

Revised: In particular, LAFTR was the least stable model in terms of variation of results given diﬀerent seeds.

Revised: We advocate veriﬁcation of any debiasing method under diﬀerent seeds, as this indicates how much the model is susceptible to small training variations and hence not suitable for bias mitigation.

Revised: This is complementary to other studies of stability, such as cross-validation, studied in Friedler et al.

Revised: Correlation between dataset features and model’s prediction.

Revised: If a model’s prediction is correlated with dataset features, it indicates it can rely on biases in data in its predictions and hence be unfair when applied.

Original: For each model, we measured the correlation between dataset features and fairness metrics using Spearman correlation matrices, measured over all of the experiments presented in this section.
Revised: To verify this, for each model, we measured the correlation between dataset features and fairness metrics using Spearman correlation matrices, measured over all of the experiments presented in this section.

Original: The results are presented in Figure 8 for the 64-dim architecture.
Revised: The results are presented in Figure 11 for the 64-dim architecture.

Original: On average the models that capture less correlation between dataset features and fairness metrics act fairer in the experiments.
Revised: On average the models that capture less correlation between dataset features and fairness metrics act fairer in the experiments, as we also observe from results presented above.

Original: Impact of seed.
Revised: Instability due to the debiasing method.

Original: We also evaluated the impact of the seed on each model. In particular, we evaluated how much standard deviation changes when the seed used for a model diﬀers.
Revised: Debiasing algorithm can cause instability due to the way the model mitigates bias. In particular, adversarial models can be unstable in practice and caution needs to be made in properly using them.

Original: The results are shown in Figures 5 and 7 for all of the setups described above.
Revised: We found Mubal (Zhang et al., 2018) to be the least stable model, as it applies the adversarial loss directly onto the class label. Applying the adversarial loss to the learned latent space, as done in LAFTR, made this process more stable. However, even this approach performed poorly in many setups and was subject to high variations with a small change in the dataset.

Original: Our results show that LAFTR models have a higher standard deviation, while CFAIR, and FFVAE (in order), are less prone to the choice of seeds.
Revised: Even among diﬀerent LAFTR variants, the LAFTR model with a given fairness criteria introduced into its objective function, was not always the LAFTR variant performing the best on that particular metric.

Original: We advocate reporting both the mean and standard deviation.
Revised: We believe this is due to application of both accuracy and adversarial loss to the same latent space, which causes a competition between the two losses.

Original: Especially, the latter indicates which models are less susceptible to the choice of seed and hence more stable  to train.  Conclusion
Revised: CFAIR has a more stable learning due to using two adversarial networks for two eligibility classes and using a balanced error rate as its loss function. Finally, FFVAE, obtained the most stable performance due to disentangling of attributes into sensitive and non-sensitive and also decoding features, which helps the model capture all of the information of the input image.  Conclusion

Original: With representation learning algorithms getting more accepted as automatic decision-making tools, we propose an evaluation scenario to better understand the limits of these methods.
Revised: With representation learning algorithms getting more accepted as automatic decision-making tools, we propose an evaluation scenario to better understand the limits of these methods.

Original: We systematically compare and analyze a few of the better-known models using our proposed dataset, where we control the correlation in-between diﬀerent dataset components.
Revised: We systematically compare and analyze a few of the better-known models using our proposed dataset, where we control the correlation in-between diﬀerent dataset components.

Original: Although these models proved their eﬀectiveness to the community and can solve and improve upon some of our dataset variants, we show that we can purposefully push them to their breaking point.
Revised: Although these models proved their eﬀectiveness to the community and can solve and improve upon some of our dataset variants, we show that we can purposefully push them to their breaking point.

Original: Our results indicate that in diﬀerent setups the models can learn and hence induce a bias when correlations in the dataset exist.
Revised: Our results indicate that in diﬀerent setups the models can learn and hence induce a bias when correlations in the dataset exist.

Original: This suggests that not all models are suitable for all scenarios and using unsuited models for certain tasks could cause unwanted consequences.
Revised: This suggests that not all models are suitable for all scenarios and using unsuited models for certain tasks could cause unwanted consequences.

Original: We also note that with the abundance of fairness metrics, there exists room for improvement when it comes to model selection as we cannot pinpoint a single metric that works best across all the metrics.
Revised: We also note that with the abundance of fairness metrics, there exists room for improvement when it comes to model selection as we cannot pinpoint a single metric that works best across all the metrics.

Original: This becomes imperative as speciﬁcally, practitioners need to be fair across more than one metric and ﬁnd a trade-oﬀ between fairness and accuracy.
Revised: This becomes imperative as speciﬁcally, practitioners need to be fair across more than one metric and ﬁnd a trade-oﬀ between fairness and accuracy.

Original: Finally, we will release our dataset and codebase to the community in order to provide a framework for the evaluation of deep fairness models.
Revised: Finally, we will release our dataset and codebase to the community in order to provide a framework for the evaluation of deep fairness models.