Original: B IAS P ROPAGATION IN F EDERATED L EARNING
Revised: B IAS P ROPAGATION IN F EDERATED L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We show that participating in federated learning can be detrimental to group fairness.
Revised: We show that participating in federated learning can be detrimental to group fairness.

Original: In fact, the bias of a few biased parties against under-represented groups (identified by sensitive attributes such as gender or race) propagates through the network to all parties.
Revised: In fact, the bias of a few biased parties against under-represented groups (identified by sensitive attributes such as gender or race) propagates through the network to all parties.

Original: On naturally partitioned real-world datasets, we analyze and explain bias propagation in federated learning.
Revised: On naturally partitioned real-world datasets, we analyze and explain bias propagation in federated learning.

Original: Our analysis reveals that biased parties unintentionally yet stealthily encode their bias in a small number of model parameters, and throughout the training, they steadily increase the dependence of the global model on sensitive attributes.
Revised: Our analysis reveals that biased parties unintentionally yet stealthily encode their bias in a small number of model parameters, and throughout the training, they steadily increase the dependence of the global model on sensitive attributes.

Original: What is important to highlight is that the experienced bias in federated learning is higher than what parties would otherwise encounter in centralized training with a model trained on the union of all their data.
Revised: What is important to highlight is that the experienced bias in federated learning is higher than what parties would otherwise encounter in centralized training with a model trained on the union of all their data.

Original: This indicates that the bias is due to the algorithm.
Revised: This indicates that the bias is due to the algorithm.

Original: Our work calls for auditing group fairness in federated learning, and designing learning algorithms that are robust to bias propagation.
Revised: Our work calls for auditing group fairness in federated learning, and designing learning algorithms that are robust to bias propagation.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Machine learning models can be biased against demographic groups.
Revised: Machine learning models can be biased against demographic groups.

Original: There has been plenty of work studying how machine learning algorithms reflect and even amplify the bias in the training data in the centralized setting, where the data is held by a single party (Hardt et al., 2016; Dwork et al., 2012;
Revised: There has been plenty of work studying how machine learning algorithms reflect and even amplify the bias in the training data in the centralized setting, where the data is held by a single party (Hardt et al., 2016; Dwork et al., 2012;

Original: Calders et al., 2009; Hashimoto et al., 2018; Zhang et al., 2020; Blum and Stangl, 2020; Lakkaraju et al., 2017).
Revised: Calders et al., 2009; Hashimoto et al., 2018; Zhang et al., 2020; Blum and Stangl, 2020; Lakkaraju et al., 2017).

Original: However, in practice, data is commonly owned by multiple parties and cannot be shared due to privacy concerns.
Revised: However, in practice, data is commonly owned by multiple parties and cannot be shared due to privacy concerns.

Original: Federated learning (FL) enables parties to collaboratively learn a global model without sharing the data.
Revised: Federated learning (FL) enables parties to collaboratively learn a global model without sharing the data.

Original: In each round, parties share the local model update computed on the dataset with a global server, and the server aggregates local updates to update the global model.
Revised: In each round, parties share the local model update computed on the dataset with a global server, and the server aggregates local updates to update the global model.

Original: Although FL is now widely deployed in various applications, such as healthcare, recruitment, and loan evaluation (Rieke et al., 2020; Yang et al., 2019), it is not fully understood whether and how FL algorithms could magnify the bias in the training datasets.
Revised: Although FL is now widely deployed in various applications, such as healthcare, recruitment, and loan evaluation (Rieke et al., 2020; Yang et al., 2019), it is not fully understood whether and how FL algorithms could magnify the bias in the training datasets.

Original: A few recent works study this problem focusing on measuring and mitigating the bias in FL with respect to a single global distribution (Chu et al., 2021; Zeng et al., 2021a;
Revised: A few recent works study this problem focusing on measuring and mitigating the bias in FL with respect to a single global distribution (Chu et al., 2021; Zeng et al., 2021a;

Original: Hu et al., 2022; Du et al., 2021; Abay et al., 2020; Papadaki et al., 2021; 2022; Hu et al., 2022).
Revised: Hu et al., 2022; Du et al., 2021; Abay et al., 2020; Papadaki et al., 2021; 2022; Hu et al., 2022).

Original: However, parties usually have heterogeneous data distributions.
Revised: However, parties usually have heterogeneous data distributions.

Original: Evaluating the model bias with respect to the global distribution does not reflect how fair the FL model is with respect to parties’ heterogeneous local data distributions (which are of relevance to users).
Revised: Evaluating the model bias with respect to the global distribution does not reflect how fair the FL model is with respect to parties’ heterogeneous local data distributions (which are of relevance to users).

Original: This is exactly the critical problem that we address in this paper.
Revised: This is exactly the critical problem that we address in this paper.

Original: Specifically, we ask: compared to standalone training (train model without any collaboration), can parties benefit from joining FL to reduce bias and build fairer models?
Revised: Specifically, we ask: compared to standalone training (train model without any collaboration), can parties benefit from joining FL to reduce bias and build fairer models?

Original: Does FL provide parties with an equivalent fairness benefit of centralized training (on the union of their data)?
Revised: Does FL provide parties with an equivalent fairness benefit of centralized training (on the union of their data)?

Original: Can parties with biased datasets deteriorate the local fairness of other parties?
Revised: Can parties with biased datasets deteriorate the local fairness of other parties?

Original: How and why does bias in a small number of parties affect the network?
Revised: How and why does bias in a small number of parties affect the network?

Original: To the best of our knowledge, we provide the first comprehensive analysis of how the FL algorithm impacts local fairness.
Revised: To the best of our knowledge, we provide the first comprehensive analysis of how the FL algorithm impacts local fairness.

Original: We provide an empirical analysis based on real-world datasets.
Revised: We provide an empirical analysis based on real-world datasets.

Original: We show that the FL might not sustain the benefits of collaboration in terms of fairness, as compared to its accuracy benefit.
Revised: We show that the FL might not sustain the benefits of collaboration in terms of fairness, as compared to its accuracy benefit.

Original: Specifically, we find that the model trained in a centralized setting can be, on average, fairer on local data distributions than the standalone models.
Revised: Specifically, we find that the model trained in a centralized setting can be, on average, fairer on local data distributions than the standalone models.

Original: However, in those cases, the FL model, trained on the same dataset, can be more biased.
Revised: However, in those cases, the FL model, trained on the same dataset, can be more biased.

Original: This suggests that the FL algorithm itself can introduce new bias in the final model, deteriorating fairness for parties.
Revised: This suggests that the FL algorithm itself can introduce new bias in the final model, deteriorating fairness for parties.

Original: Furthermore, we demonstrate that FL impacts local fairness for parties in different ways.
Revised: Furthermore, we demonstrate that FL impacts local fairness for parties in different ways.

Original: Specifically, we find a strong correlation between parties’ fairness gap in the standalone setting and the fairness benefit they obtain from joining FL: parties with greater bias in the standalone setting (caused by their local data) would receive a fairer model from FL.
Revised: Specifically, we find a strong correlation between parties’ fairness gap in the standalone setting and the fairness benefit they obtain from joining FL: parties with greater bias in the standalone setting (caused by their local data) would receive a fairer model from FL.

Original: In contrast,
Revised: In contrast,

Original: FL has negative impact on parties with less significant bias in the standalone setting, resulting in a more biased model in FL.
Revised: FL has negative impact on parties with less significant bias in the standalone setting, resulting in a more biased model in FL.

Original: We further demonstrate that this is due to the fact that FL propagates bias : bias from a few parties can now influence all parties in the network, hence aggravating the fairness problem among all parties.
Revised: We further demonstrate that this is due to the fact that FL propagates bias : bias from a few parties can now influence all parties in the network, hence aggravating the fairness problem among all parties.

Original: Finally, we offer potential explanations for how bias is propagated in FL.
Revised: Finally, we offer potential explanations for how bias is propagated in FL.

Original: In particular, we show that local updates from biased parties increase the model’s prediction dependence on the sensitive attribute.
Revised: In particular, we show that local updates from biased parties increase the model’s prediction dependence on the sensitive attribute.

Original: Such increase is achieved by the norm increase of a few parameters (e.g., 104 parameters out of 1,792 parameters).
Revised: Such increase is achieved by the norm increase of a few parameters (e.g., 104 parameters out of 1,792 parameters).

Original: This increase then impacts the global model through aggregation, subsequently impacting all other parties.
Revised: This increase then impacts the global model through aggregation, subsequently impacting all other parties.

Original: In addition, we also show that the fairness gap of the final model can be governed by adjusting the value of those parameters.
Revised: In addition, we also show that the fairness gap of the final model can be governed by adjusting the value of those parameters.

Original: Surprisingly, by scaling those parameters, we can reduce the bias of the model to a small value of 0.05 (the measurement for bias takes a value from 0 to 1) with only 0.6% accuracy drop or increase the bias to a large value of 0.96 with 11% accuracy drop.
Revised: Surprisingly, by scaling those parameters, we can reduce the bias of the model to a small value of 0.05 (the measurement for bias takes a value from 0 to 1) with only 0.6% accuracy drop or increase the bias to a large value of 0.96 with 11% accuracy drop.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Federated Learning.
Revised: Federated Learning.

Original: We follow the conventional FL setting (McMahan et al., 2017).
Revised: We follow the conventional FL setting (McMahan et al., 2017).

Original: There is a network consisting of K parties.
Revised: There is a network consisting of K parties.

Original: Each party k ∈ [ K ] holds a dataset ˜ D k of size n k sampled from local distribution D k .
Revised: Each party k ∈ [ K ] holds a dataset ˜ D k of size n k sampled from local distribution D k .

Original: The goal of each party is to learn a model that minimizes the loss on D k .
Revised: The goal of each party is to learn a model that minimizes the loss on D k .

Original: Federated learning achieves this goal by training a global model to minimize the average loss across parties, i.e., min θ 1 N (cid:80)
Revised: Federated learning achieves this goal by training a global model to minimize the average loss across parties, i.e., min θ 1 N (cid:80)

Original: Kk =1 n k L ( θ, ˜ D k ) , where N is the sum of all local training dataset size.
Revised: Kk =1 n k L ( θ, ˜ D k ) , where N is the sum of all local training dataset size.

Original: At each communication round t , a global server sends the current global model to all parties.
Revised: At each communication round t , a global server sends the current global model to all parties.

Original: All parties train the global model locally on their local dataset and send the updated local model to the server.
Revised: All parties train the global model locally on their local dataset and send the updated local model to the server.

Original: The server then aggregates those local models to obtain the new global model.
Revised: The server then aggregates those local models to obtain the new global model.

Original: Note that, we consider the case where all parties participate in the training in each round.
Revised: Note that, we consider the case where all parties participate in the training in each round.

Original: It eliminates the potential bias caused by the sampling of participating parties.
Revised: It eliminates the potential bias caused by the sampling of participating parties.

Original: Group Fairness.
Revised: Group Fairness.

Original: Fairness has a wide variety of meanings in literature.
Revised: Fairness has a wide variety of meanings in literature.

Original: Group fairness entails, in particular, that the model should perform comparably across groups defined by sensitive attributes (e.g., sex).
Revised: Group fairness entails, in particular, that the model should perform comparably across groups defined by sensitive attributes (e.g., sex).

Original: It is now common practice to evaluate discrimination in a model (or system) based on quantitative measurements of group fairness.
Revised: It is now common practice to evaluate discrimination in a model (or system) based on quantitative measurements of group fairness.

Original: In light of this, we focus on two widely-used group fairness notions, equalized odds (Hardt et al., 2016) and demographic parity (Dwork et al., 2012).
Revised: In light of this, we focus on two widely-used group fairness notions, equalized odds (Hardt et al., 2016) and demographic parity (Dwork et al., 2012).

Original: To formally define those fairness notions, we assume each data point is associated with a sensitive attribute a ∈ A , and we use Y to denote the true label.
Revised: To formally define those fairness notions, we assume each data point is associated with a sensitive attribute a ∈ A , and we use Y to denote the true label.

Original: Definition 1 (Equalized Odds (Hardt et al., 2016)) .
Revised: Definition 1 (Equalized Odds (Hardt et al., 2016)) .

Original: We say a model f
Revised: We say a model f

Original: θ is fair with respect to equalized odds if the prediction is conditionally independent of the sensitive attribute
Revised: θ is fair with respect to equalized odds if the prediction is conditionally independent of the sensitive attribute

Original: A given the true label Y , i.e., ∀ a, a ′ ∈ A , y ∈ Y , Pr( f θ ( X ) =
Revised: A given the true label Y , i.e., ∀ a, a ′ ∈ A , y ∈ Y , Pr( f θ ( X ) =

Original: y
Revised: y

Original: |
Revised: |

Original: A = a, Y = y ) =
Revised: A = a, Y = y ) =

Original: Pr( f θ ( X ) =
Revised: Pr( f θ ( X ) =

Original: y
Revised: y

Original: |
Revised: |

Original: A = a ′ , Y = y ) .
Revised: A = a ′ , Y = y ) .

Original: To measure fairness, we use the fairness gap with respect to Equalized Odds Difference, defined as ∆ EO ( θ, D ) :
Revised: To measure fairness, we use the fairness gap with respect to Equalized Odds Difference, defined as ∆ EO ( θ, D ) :

Original: [Equation]
Revised: [Equation]

Original: with an ideal value equal to zero.
Revised: with an ideal value equal to zero.

Original: Furthermore, in many applications, there exists a favorable prediction from the model (e.g., grant the loan).
Revised: Furthermore, in many applications, there exists a favorable prediction from the model (e.g., grant the loan).

Original: We assume the positive prediction ( + ) as the favorable outcome.
Revised: We assume the positive prediction ( + ) as the favorable outcome.

Original: Demographic parity (Dwork et al., 2012) group fairness notion asks the model to give a favorable label to groups with equal rates.
Revised: Demographic parity (Dwork et al., 2012) group fairness notion asks the model to give a favorable label to groups with equal rates.

Original: Definition 2 (Demographic parity (Dwork et al., 2012)) .
Revised: Definition 2 (Demographic parity (Dwork et al., 2012)) .

Original: We say a model f
Revised: We say a model f

Original: θ is fair with respect to demographic parity if the positive prediction is independent of the sensitive attribute A , i.e., ∀ a,
Revised: θ is fair with respect to demographic parity if the positive prediction is independent of the sensitive attribute A , i.e., ∀ a,

Original: a ′ ∈ A , Pr( f θ ( X ) =
Revised: a ′ ∈ A , Pr( f θ ( X ) =

Original: + | A = a ) =
Revised: + | A = a ) =

Original: Pr( f θ ( X ) =
Revised: Pr( f θ ( X ) =

Original: + | A = a ′ ) .
Revised: + | A = a ′ ) .

Original: To measure fairness, we use the fairness gap with respect to Demographic Parity Difference, defined as
Revised: To measure fairness, we use the fairness gap with respect to Demographic Parity Difference, defined as

Original: [Equation]
Revised: [Equation]

Original: with an ideal value equal to zero.
Revised: with an ideal value equal to zero.

Original: In the rest of the paper, we use the fairness gap ∆ (including ∆ EO and ∆ DP ) to measure the bias (fairness) of a model.
Revised: In the rest of the paper, we use the fairness gap ∆ (including ∆ EO and ∆ DP ) to measure the bias (fairness) of a model.

Original: The more significant fairness gap means the model is more biased (less fair).
Revised: The more significant fairness gap means the model is more biased (less fair).

Original: Measuring impact of FL.
Revised: Measuring impact of FL.

Original: Federated learning strives to improve model performance compared to standalone training, and the ultimate goal is to achieve comparable performance to centralized training.
Revised: Federated learning strives to improve model performance compared to standalone training, and the ultimate goal is to achieve comparable performance to centralized training.

Original: Thus, in order to quantify the impact of FL on local fairness, we use centralized training and standalone training as baselines.
Revised: Thus, in order to quantify the impact of FL on local fairness, we use centralized training and standalone training as baselines.

Original: In standalone training, each party trains a model θ k independently to minimize the loss on its training data ˜ D k .
Revised: In standalone training, each party trains a model θ k independently to minimize the loss on its training data ˜ D k .

Original: Note that the standalone model’s fairness gap for a party is contributed by herself.
Revised: Note that the standalone model’s fairness gap for a party is contributed by herself.

Original: Thus, we use the fairness gap of a party’s standalone model to represent the party’s bias.
Revised: Thus, we use the fairness gap of a party’s standalone model to represent the party’s bias.

Original: On the other hand, in centralized training, all local training datasets from all parties are combined, and a centralized model θ c is trained using this global training dataset.
Revised: On the other hand, in centralized training, all local training datasets from all parties are combined, and a centralized model θ c is trained using this global training dataset.

Original: We define the benefit of FL and collaboration (i.e., centralized training) in terms of fairness and accuracy using these baselines.
Revised: We define the benefit of FL and collaboration (i.e., centralized training) in terms of fairness and accuracy using these baselines.

Original: The benefit of FL for each party k is defined by the difference between the standalone model θ k and the FL model θ g .
Revised: The benefit of FL for each party k is defined by the difference between the standalone model θ k and the FL model θ g .

Original: It reveals the extent to which a party benefits from participating in FL.
Revised: It reveals the extent to which a party benefits from participating in FL.

Original: More formally, for a party k , we define the accuracy benefit of FL as
Revised: More formally, for a party k , we define the accuracy benefit of FL as

Original: Acc ( θ g , D k ) − Acc ( θ k , D k ) and fairness benefit of FL as ∆( θ k , D k ) − ∆( θ g , D k ) .
Revised: Acc ( θ g , D k ) − Acc ( θ k , D k ) and fairness benefit of FL as ∆( θ k , D k ) − ∆( θ g , D k ) .

Original: Similarly, the benefit of collaboration is based on the difference between the standalone model and the centralized model.
Revised: Similarly, the benefit of collaboration is based on the difference between the standalone model and the centralized model.

Original: It indicates how much an individual party could gain from collaborating with others by sharing the entire training dataset.
Revised: It indicates how much an individual party could gain from collaborating with others by sharing the entire training dataset.

Original: The average benefit of FL or collaboration is the average benefit across all parties in the network.
Revised: The average benefit of FL or collaboration is the average benefit across all parties in the network.

Original: The positive benefit indicates that FL (or collaboration) helps to improve accuracy or fairness, while the negative benefit implies that FL or collaboration hurts.
Revised: The positive benefit indicates that FL (or collaboration) helps to improve accuracy or fairness, while the negative benefit implies that FL or collaboration hurts.

Original: Measuring bias aside fairness gap.
Revised: Measuring bias aside fairness gap.

Original: Measuring the bias of a model based on the fairness gap reveals the model’s performance disparity across groups, but does not explains why the model is biased.
Revised: Measuring the bias of a model based on the fairness gap reveals the model’s performance disparity across groups, but does not explains why the model is biased.

Original: Specifically, is the prediction of the model attributed to the sensitive attribute or any other insensitive attributes whose distributions vary between groups?
Revised: Specifically, is the prediction of the model attributed to the sensitive attribute or any other insensitive attributes whose distributions vary between groups?

Original: The former is typically referred to as disparate treatment (direct discrimination) as the sensitive attribute directly influences the model prediction (Grabowicz et al., 2022; Zafar et al., 2017).
Revised: The former is typically referred to as disparate treatment (direct discrimination) as the sensitive attribute directly influences the model prediction (Grabowicz et al., 2022; Zafar et al., 2017).

Original: Towards measuring bias in this aspect, we employ the feature attribution method to explain the model’s prediction on each data point.
Revised: Towards measuring bias in this aspect, we employ the feature attribution method to explain the model’s prediction on each data point.

Original: More precisely, given a model f θ , and the input features x = ( x 1 , x 2 , ..., x d ) , the attribution of the prediction at input x relative to a baseline input x ′ is a vector ( a 1 , ..., a d ) where a i is the contribution of x i to the prediction of f θ ( x ) .
Revised: More precisely, given a model f θ , and the input features x = ( x 1 , x 2 , ..., x d ) , the attribution of the prediction at input x relative to a baseline input x ′ is a vector ( a 1 , ..., a d ) where a i is the contribution of x i to the prediction of f θ ( x ) .

Original: We use Integrated Gradient (Sundararajan et al., 2017) to compute the feature attribution, which only requires a few calls to the gradient operation.
Revised: We use Integrated Gradient (Sundararajan et al., 2017) to compute the feature attribution, which only requires a few calls to the gradient operation.

Original: Following the suggestions in the paper, we use the average feature values computed on the whole test dataset as the baseline for non-sensitive and non-binary sensitive features.
Revised: Following the suggestions in the paper, we use the average feature values computed on the whole test dataset as the baseline for non-sensitive and non-binary sensitive features.

Original: For the binary sensitive attribute, we use the opposite feature value as the baseline.
Revised: For the binary sensitive attribute, we use the opposite feature value as the baseline.

Original: For instance, if the test input is (Sex: female, Age:28), the baseline input would be (Sex: male, Age: average age over the data points).
Revised: For instance, if the test input is (Sex: female, Age:28), the baseline input would be (Sex: male, Age: average age over the data points).

Original: By employing this feature attribution, we are able to discover the bias associated with disparate treatment.
Revised: By employing this feature attribution, we are able to discover the bias associated with disparate treatment.

Original: For more details about the Integrated Gradient, please refer to the original paper (Sundararajan et al., 2017).
Revised: For more details about the Integrated Gradient, please refer to the original paper (Sundararajan et al., 2017).

Original:  E MPIRICAL A NALYSIS
Revised:  E MPIRICAL A NALYSIS

Original: Our objective is to comprehend how FL impacts fairness for parties.
Revised: Our objective is to comprehend how FL impacts fairness for parties.

Original: Toward this end, we start by the comparison of average performance between FL and baselines (i.e., standalone training and centralized training).
Revised: Toward this end, we start by the comparison of average performance between FL and baselines (i.e., standalone training and centralized training).

Original: In particular, we ask: does FL improve fairness for clients compared to standalone training, and fairness-wise, does FL provide the same benefit as centralized training?
Revised: In particular, we ask: does FL improve fairness for clients compared to standalone training, and fairness-wise, does FL provide the same benefit as centralized training?

Original: We provide answers in Section 3.2, where we demonstrate that FL can exacerbate the fairness issue for parties and does not retain the fairness benefit of collaboration.
Revised: We provide answers in Section 3.2, where we demonstrate that FL can exacerbate the fairness issue for parties and does not retain the fairness benefit of collaboration.

Original: After that, we further investigate how FL impacts performance for each party.
Revised: After that, we further investigate how FL impacts performance for each party.

Original: Specifically, we aim to answer the following two questions: does FL provide the same benefit in terms of fairness for parties?
Revised: Specifically, we aim to answer the following two questions: does FL provide the same benefit in terms of fairness for parties?

Original: What causes the disparate impact of FL on parties’ fairness?
Revised: What causes the disparate impact of FL on parties’ fairness?

Original: We provide analysis in Section 3.3, where we show
Revised: We provide analysis in Section 3.3, where we show

Original: FL can propagate the bias among parties.
Revised: FL can propagate the bias among parties.

Original: As a consequence, FL improves fairness for parties with greater bias while worsening fairness issues for parties with less bias.
Revised: As a consequence, FL improves fairness for parties with greater bias while worsening fairness issues for parties with less bias.

Original: Finally, we investigate how the bias is propagated in FL .
Revised: Finally, we investigate how the bias is propagated in FL .

Original: The analysis in Section 3.4 shows that biased parties encode the bias in a few parameters through local update, which is propagated to the aggregated model and ultimately to other parties via parameter aggregation.
Revised: The analysis in Section 3.4 shows that biased parties encode the bias in a few parameters through local update, which is propagated to the aggregated model and ultimately to other parties via parameter aggregation.

Original: 3. S ETUP
Revised: 3. S ETUP

Original: We use the two datasets for our empirical analysis: US Census Data and CelebA (Liu et al., 2015) .
Revised: We use the two datasets for our empirical analysis: US Census Data and CelebA (Liu et al., 2015) .

Original: The US Census dataset consists of census data from different places in the US.
Revised: The US Census dataset consists of census data from different places in the US.

Original: We naturally partition
Revised: We naturally partition

Original: [Table]
Revised: [Table]

Original: the dataset based on the source of the data.
Revised: the dataset based on the source of the data.

Original: Thus, we have 51 parties in total, representing 50 states in the US and Puerto Rico.
Revised: Thus, we have 51 parties in total, representing 50 states in the US and Puerto Rico.

Original: We consider three tasks defined in the folktables (Ding et al., 2021):
Revised: We consider three tasks defined in the folktables (Ding et al., 2021):

Original: Income (ACSIncome) 1 , Health (ACSPublicCoverage) and Employment (ACSEmployment).
Revised: Income (ACSIncome) 1 , Health (ACSPublicCoverage) and Employment (ACSEmployment).

Original: We train two-layer neural network models for all the tasks.
Revised: We train two-layer neural network models for all the tasks.

Original: We consider the binary notion of sex (i.e., male and female) and multi-value race as the sensitive attribute for every task.
Revised: We consider the binary notion of sex (i.e., male and female) and multi-value race as the sensitive attribute for every task.

Original: Each party has 1, training points and 2,000 test points.
Revised: Each party has 1, training points and 2,000 test points.

Original: Note that we use a large test dataset to ensure an accurate estimation of the fairness gap with respect to parities’ test distribution.
Revised: Note that we use a large test dataset to ensure an accurate estimation of the fairness gap with respect to parities’ test distribution.

Original: CelebA
Revised: CelebA

Original: (Liu et al., 2015) is an image dataset consisted of 200,000 celebrity images.
Revised: (Liu et al., 2015) is an image dataset consisted of 200,000 celebrity images.

Original: We focus on the age prediction task and consider the binary attribute “Male” (male and non-male) as the sensitive attribute, which we will refer to as the Sex attribute.
Revised: We focus on the age prediction task and consider the binary attribute “Male” (male and non-male) as the sensitive attribute, which we will refer to as the Sex attribute.

Original: We partition the data uniformly at random among all parties and train
Revised: We partition the data uniformly at random among all parties and train

Original: CNN models.
Revised: CNN models.

Original: We also consider the Non-IID partitioning on CelebA dataset, and the results are presented in Appendix B.
Revised: We also consider the Non-IID partitioning on CelebA dataset, and the results are presented in Appendix B.

Original: For all the experiments, we report the average results over five runs with different random seeds.
Revised: For all the experiments, we report the average results over five runs with different random seeds.

Original: We use FedML (He et al., 2020), a library for FL based on PyTorch Paszke et al.
Revised: We use FedML (He et al., 2020), a library for FL based on PyTorch Paszke et al.

Original: , to train models in FL.
Revised: , to train models in FL.

Original: We also utilize Captum (Kokhlikyan et al., 2020) for computing the feature attribution.
Revised: We also utilize Captum (Kokhlikyan et al., 2020) for computing the feature attribution.

Original: More details about the datasets and models can be found in Appendix A.
Revised: More details about the datasets and models can be found in Appendix A.

Original: 3. C OLLABORATION VIA FL CAN WORSEN FAIRNESS ISSUE
Revised: 3. C OLLABORATION VIA FL CAN WORSEN FAIRNESS ISSUE

Original: Table 1 shows the average accuracy and fairness gaps of the centralized model, FL model, and standalone models on local datasets.
Revised: Table 1 shows the average accuracy and fairness gaps of the centralized model, FL model, and standalone models on local datasets.

Original: We observe that centralized training can improve accuracy and fairness at the same time for parties.
Revised: We observe that centralized training can improve accuracy and fairness at the same time for parties.

Original: For instance, on the Income dataset, centralized training improves the accuracy by 6% and reduces the fairness gap across racial groups by 5.5% with respect to equalized odds and by 5.5% with respect to demographic parity.
Revised: For instance, on the Income dataset, centralized training improves the accuracy by 6% and reduces the fairness gap across racial groups by 5.5% with respect to equalized odds and by 5.5% with respect to demographic parity.

Original: However, we notice that FL may not achieve the same benefit as the centralized training in terms of fairness but can even worsen the fairness issue for parties.
Revised: However, we notice that FL may not achieve the same benefit as the centralized training in terms of fairness but can even worsen the fairness issue for parties.

Original: For instance, on the Income dataset, the EO fairness gap of the FL model for the sex groups increases by 35%, and the DP fairness gap increases by 29.9%, compared to standalone models.
Revised: For instance, on the Income dataset, the EO fairness gap of the FL model for the sex groups increases by 35%, and the DP fairness gap increases by 29.9%, compared to standalone models.

Original: Table 2 in Appendix B shows the results for other popular FL algorithms, including FedNova (Wang et al., 2020), Scaffold Karimireddy et al.
Revised: Table 2 in Appendix B shows the results for other popular FL algorithms, including FedNova (Wang et al., 2020), Scaffold Karimireddy et al.

Original: (2020b), FedOpt (Reddi et al., 2020), FedProx Li et al.
Revised: (2020b), FedOpt (Reddi et al., 2020), FedProx Li et al.

Original: (2020), and Mime (Karimireddy et al., 2020a), on which we observe a similar pattern as on FedAvg algorithm.
Revised: (2020), and Mime (Karimireddy et al., 2020a), on which we observe a similar pattern as on FedAvg algorithm.

Original: We refer readers to Appendix B.5 for more detailed results on other
Revised: We refer readers to Appendix B.5 for more detailed results on other

Original: FL algorithms.
Revised: FL algorithms.

Original: The centralized and FL models are trained on the same dataset.
Revised: The centralized and FL models are trained on the same dataset.

Original: The difference between the FL model and the centralized model in terms of fairness suggests FL algorithm can introduce more bias compared to standard training.
Revised: The difference between the FL model and the centralized model in terms of fairness suggests FL algorithm can introduce more bias compared to standard training.

Original: Consequently, the explanation of how bias is introduced to the model during the standard training in the centralized setting may not adequately explain how FL introduces bias to the model.
Revised: Consequently, the explanation of how bias is introduced to the model during the standard training in the centralized setting may not adequately explain how FL introduces bias to the model.

Original: In the following, we further explore how FL impacts fairness from the party level.
Revised: In the following, we further explore how FL impacts fairness from the party level.

Original: 3. FL PROPAGATES BIAS AMONG PARTIES
Revised: 3. FL PROPAGATES BIAS AMONG PARTIES

Original: Disparate impact of FL on group fairness over parties.
Revised: Disparate impact of FL on group fairness over parties.

Original: Figure 1 shows the benefit of FL on fairness and accuracy for parties.
Revised: Figure 1 shows the benefit of FL on fairness and accuracy for parties.

Original: FL improves accuracy for almost all parties, and the improvement variance across parties is small.
Revised: FL improves accuracy for almost all parties, and the improvement variance across parties is small.

Original: On the other hand, the fairness benefit of FL is negative for most parties.
Revised: On the other hand, the fairness benefit of FL is negative for most parties.

Original: It implies that most parties obtain a more biased model in
Revised: It implies that most parties obtain a more biased model in

Original: FL compared to standalone training.
Revised: FL compared to standalone training.

Original: Moreover, we notice that the variance of the fairness benefits across parties is large.
Revised: Moreover, we notice that the variance of the fairness benefits across parties is large.

Original: It shows that even though all parties have the same global model in FL, they are not benefiting from the model equally (recall that each party tests the model on her local test dataset).
Revised: It shows that even though all parties have the same global model in FL, they are not benefiting from the model equally (recall that each party tests the model on her local test dataset).

Original: Figure 2 shows strong correlations between the fairness benefit a party acquires in FL and the fairness gap of the standalone model for the party (i e., bias level of the party).
Revised: Figure 2 shows strong correlations between the fairness benefit a party acquires in FL and the fairness gap of the standalone model for the party (i e., bias level of the party).

Original: It further demonstrates the disparate impact of
Revised: It further demonstrates the disparate impact of

Original: FL on fairness: FL can improve fairness for more biased parties but at the cost of worsening the issue for less biased parties.
Revised: FL on fairness: FL can improve fairness for more biased parties but at the cost of worsening the issue for less biased parties.

Original: Aggregation contradicts with local update.
Revised: Aggregation contradicts with local update.

Original: Figure 3 shows the dynamic of the fairness gap before and after the aggregation.
Revised: Figure 3 shows the dynamic of the fairness gap before and after the aggregation.

Original: We find that aggregation and local update contradicts each other during the training.
Revised: We find that aggregation and local update contradicts each other during the training.

Original: More specifically, the local update for the least biased party (whose standalone model has the lowest fairness gap) reduces the fairness gap of the model during the local update.
Revised: More specifically, the local update for the least biased party (whose standalone model has the lowest fairness gap) reduces the fairness gap of the model during the local update.

Original: However, this reduction is eliminated by aggregation.
Revised: However, this reduction is eliminated by aggregation.

Original: On the other hand, the most biased parties increase the models’ bias through the local update, which will be reduced via aggregation.
Revised: On the other hand, the most biased parties increase the models’ bias through the local update, which will be reduced via aggregation.

Original: It indicates that the aggregation contributes to the disparate impact of FL on fairness, improving fairness for more biased parties while worsening the fairness issue for less biased parties compared to the standalone setting.
Revised: It indicates that the aggregation contributes to the disparate impact of FL on fairness, improving fairness for more biased parties while worsening the fairness issue for less biased parties compared to the standalone setting.

Original: The y-axis represents the average influence of each party.
Revised: The y-axis represents the average influence of each party.

Original: The Pearson correlation between the fairness gap a party obtains in the standalone setting and the impact she has on local fairness for parties in
Revised: The Pearson correlation between the fairness gap a party obtains in the standalone setting and the impact she has on local fairness for parties in

Original: FL are shown.
Revised: FL are shown.

Original: The greater the party’s bias, the greater its adverse effect on others.
Revised: The greater the party’s bias, the greater its adverse effect on others.

Original: Biased parties negatively influence other parties via aggregation throughout the training.
Revised: Biased parties negatively influence other parties via aggregation throughout the training.

Original: Why does aggregation have disparate impact on local fairness for parties?
Revised: Why does aggregation have disparate impact on local fairness for parties?

Original: Specifically, we ask which party’s local update causes the increase (or decrease) of the fairness gap for other parties via aggregation.
Revised: Specifically, we ask which party’s local update causes the increase (or decrease) of the fairness gap for other parties via aggregation.

Original: To answer this question we measure the influence of a party on other parties’ local fairness by the influence of the party’s local update on the fairness of the aggregated model during the training.
Revised: To answer this question we measure the influence of a party on other parties’ local fairness by the influence of the party’s local update on the fairness of the aggregated model during the training.

Original: More precisely, we compute the influence of party i on party j as the fairness gap increase when party i ’s local update is removed from the aggregation in each round t and sum over all the training rounds, which is formally defined as follows:
Revised: More precisely, we compute the influence of party i on party j as the fairness gap increase when party i ’s local update is removed from the aggregation in each round t and sum over all the training rounds, which is formally defined as follows:

Original: I i,j
Revised: I i,j

Original: [Equation]
Revised: [Equation]

Original: − i , D j ) − ∆( θ t , D j ) , where θ t is the global model (i.e., the aggregated model overall local updated models) in round t and θ t, − i is the aggregated model over local updated models from parties excluding party i .
Revised: − i , D j ) − ∆( θ t , D j ) , where θ t is the global model (i.e., the aggregated model overall local updated models) in round t and θ t, − i is the aggregated model over local updated models from parties excluding party i .

Original: If party i improves fairness for party j , the influence is positive and vice versa.
Revised: If party i improves fairness for party j , the influence is positive and vice versa.

Original: We compute the influence for all pairs of parties, and the most influential pairs are shown in Figure 4.
Revised: We compute the influence for all pairs of parties, and the most influential pairs are shown in Figure 4.

Original: We observe that a less biased party has positive influence on fairness for other parties while a more biased party has negative influence on other parties.
Revised: We observe that a less biased party has positive influence on fairness for other parties while a more biased party has negative influence on other parties.

Original: Furthermore, we show the correlation between the bias a party has (i.e., the bias of the party’s standalone model) and its average influence on all parties’ fairness (i.e., (cid:80)
Revised: Furthermore, we show the correlation between the bias a party has (i.e., the bias of the party’s standalone model) and its average influence on all parties’ fairness (i.e., (cid:80)

Original: Kk =

Revised: Kk =

Original: I i,k /K ) in Figure 5.
Revised: I i,k /K ) in Figure 5.

Original: The strong correlation between those two further supports that the more biased parties have a stronger negative influence, while the less biased parties have a stronger positive influence on other parties.
Revised: The strong correlation between those two further supports that the more biased parties have a stronger negative influence, while the less biased parties have a stronger positive influence on other parties.

Original: Lastly, Figure 6 shows the dynamic of the influence of the top 5 most biased parties (i.e., parties who have the largest fairness gap in the standalone setting) and the top 5 least biased parties.
Revised: Lastly, Figure 6 shows the dynamic of the influence of the top 5 most biased parties (i.e., parties who have the largest fairness gap in the standalone setting) and the top 5 least biased parties.

Original: We notice that the influence is monotonically increasing for the most biased parties and decreasing for the least value for sex - Income.
Revised: We notice that the influence is monotonically increasing for the most biased parties and decreasing for the least value for sex - Income.

Original: We show results for the parties who have the highest or lowest fairness gap in the standalone setting.
Revised: We show results for the parties who have the highest or lowest fairness gap in the standalone setting.

Original: The standalone training and FL gradually increase the model dependence on the sensitive attribute, while the centralized training reduces the dependence during the training.
Revised: The standalone training and FL gradually increase the model dependence on the sensitive attribute, while the centralized training reduces the dependence during the training.

Original: biased parties.
Revised: biased parties.

Original: In other words, biased (less biased) parties consistently have a negative (positive) influence on others’ fairness gaps.
Revised: In other words, biased (less biased) parties consistently have a negative (positive) influence on others’ fairness gaps.

Original: Our results imply that FL can propagate bias among parties: the bias from biased parties negatively influence the fairness of other parties via aggregation throughout the training.
Revised: Our results imply that FL can propagate bias among parties: the bias from biased parties negatively influence the fairness of other parties via aggregation throughout the training.

Original: Next, we will explore how the bias is propagated in FL.
Revised: Next, we will explore how the bias is propagated in FL.

Original: 3. H OW BIAS IS PROPAGATED IN FL?
Revised: 3. H OW BIAS IS PROPAGATED IN FL?

Original: Large fairness gap is caused by disparate treatment.
Revised: Large fairness gap is caused by disparate treatment.

Original: We first investigate what the bias represents.
Revised: We first investigate what the bias represents.

Original: More specifically, is the bias (fairness gap) increase in FL directly caused by the disparate treatment of the model among sensitive groups?
Revised: More specifically, is the bias (fairness gap) increase in FL directly caused by the disparate treatment of the model among sensitive groups?

Original: To answer this question, we utilize Integrated Gradients (Sundararajan et al., 2017) to measure the attribution of each input feature to the predictions of models with respect to the positive class.
Revised: To answer this question, we utilize Integrated Gradients (Sundararajan et al., 2017) to measure the attribution of each input feature to the predictions of models with respect to the positive class.

Original: Figure 7 shows the histogram of the attribution value for the sensitive attribute, "Sex", over individual test points from the female group and the male group.
Revised: Figure 7 shows the histogram of the attribution value for the sensitive attribute, "Sex", over individual test points from the female group and the male group.

Original: We notice that the sex attribute has a large attribution value for the standalone model’s predictions and
Revised: We notice that the sex attribute has a large attribution value for the standalone model’s predictions and

Original: FL model’s predictions.
Revised: FL model’s predictions.

Original: It means that the prediction of those models are heavily dependent on the sensitive attribute of the test data.
Revised: It means that the prediction of those models are heavily dependent on the sensitive attribute of the test data.

Original: Additionally, individuals’ sensitive attribute affects the model predictions in a different way.
Revised: Additionally, individuals’ sensitive attribute affects the model predictions in a different way.

Original: The average attribution value for the “Sex” attribute is positive for the male group and negative for the female group.
Revised: The average attribution value for the “Sex” attribute is positive for the male group and negative for the female group.

Original: Moreover, we find that there is less difference in the attribution value for the female and male groups with respect to other attributes (see Figure 17 in
Revised: Moreover, we find that there is less difference in the attribution value for the female and male groups with respect to other attributes (see Figure 17 in

Original: Appendix B).
Revised: Appendix B).

Original: It implies that the large fairness gap for models is not caused by the distinct distribution of other insensitive attributes over protected groups; rather, it is mainly caused by the disparate treatment of the model on protected groups.
Revised: It implies that the large fairness gap for models is not caused by the distinct distribution of other insensitive attributes over protected groups; rather, it is mainly caused by the disparate treatment of the model on protected groups.

Original: FL model learns more biased patterns.
Revised: FL model learns more biased patterns.

Original: Comparing the attribution value of “Sex” for different models in Figure 7, we find that, although the predictions of FL model depend less heavily on sensitive attribute compared to that of the standalone model, the dependence is still stronger than that of the centralized model.
Revised: Comparing the attribution value of “Sex” for different models in Figure 7, we find that, although the predictions of FL model depend less heavily on sensitive attribute compared to that of the standalone model, the dependence is still stronger than that of the centralized model.

Original: It suggests that the model trained in FL learns a more biased pattern compared to the model that could otherwise be learned in the centralized setting.
Revised: It suggests that the model trained in FL learns a more biased pattern compared to the model that could otherwise be learned in the centralized setting.

Original: Figure 8 shows the dynamic of the average absolute feature attribution for “Sex” during the training.
Revised: Figure 8 shows the dynamic of the average absolute feature attribution for “Sex” during the training.

Original: We find that, for the most biased party, standalone training increases the model’s dependence on the sensitive attribute throughout training, but centralized training decreases it.
Revised: We find that, for the most biased party, standalone training increases the model’s dependence on the sensitive attribute throughout training, but centralized training decreases it.

Original: It suggests that collaboration through centralized training improves local fairness by guiding the model to learn less biased features.
Revised: It suggests that collaboration through centralized training improves local fairness by guiding the model to learn less biased features.

Original: In FL, however, the absolute attribution value barely changes after 100 rounds and remains significantly greater than that in the centralized setting.
Revised: In FL, however, the absolute attribution value barely changes after 100 rounds and remains significantly greater than that in the centralized setting.

Original: It implies that FL algorithm may introduce the bias to the final model by inhibiting the model from learning less biased features.
Revised: It implies that FL algorithm may introduce the bias to the final model by inhibiting the model from learning less biased features.

Original: Biased parties increase the model dependence on the sensitive attribute.
Revised: Biased parties increase the model dependence on the sensitive attribute.

Original: Figure 9 shows the attribution value of the aggregated model and local updated model from the most biased party and least biased party.
Revised: Figure 9 shows the attribution value of the aggregated model and local updated model from the most biased party and least biased party.

Original: We observe that the biased party increases the global model dependence on sensitive attribute during the local update, and this increase is persistent over the training, while the least biased party, on the other hand, reduces the dependence on the sensitive attribute, which is aligned with the trend of fairness gap in Figure 3.
Revised: We observe that the biased party increases the global model dependence on sensitive attribute during the local update, and this increase is persistent over the training, while the least biased party, on the other hand, reduces the dependence on the sensitive attribute, which is aligned with the trend of fairness gap in Figure 3.

Original: Bias is encoded in a few parameters.
Revised: Bias is encoded in a few parameters.

Original: Biased parties increase the model dependence on the sensitive attribute.
Revised: Biased parties increase the model dependence on the sensitive attribute.

Original: But how does this increase propagate to other parties in the network?
Revised: But how does this increase propagate to other parties in the network?

Original: Note that parties share the model parameters of the local updated model with the server; thus, the bias should be encoded in the parameters.
Revised: Note that parties share the model parameters of the local updated model with the server; thus, the bias should be encoded in the parameters.

Original: Thus, we examine which parameters are related to the bias of the model.
Revised: Thus, we examine which parameters are related to the bias of the model.

Original: Intuitively, the parameters used to extract sensitive attribute information impact the attribution value of the sensitive attribute to the model prediction.
Revised: Intuitively, the parameters used to extract sensitive attribute information impact the attribution value of the sensitive attribute to the model prediction.

Original: If the absolute value (signal) of those parameters is substantial, the value of the sensitive attribute will have a significant effect on the model’s prediction.
Revised: If the absolute value (signal) of those parameters is substantial, the value of the sensitive attribute will have a significant effect on the model’s prediction.

Original: In our evaluation, sensitive attribute is part of the input feature.
Revised: In our evaluation, sensitive attribute is part of the input feature.

Original: Hence, the parameters directly applied to the “Sex” attribute in the first layer of the neural network should contribute to the bias of the model.
Revised: Hence, the parameters directly applied to the “Sex” attribute in the first layer of the neural network should contribute to the bias of the model.

Original: Figure 10(a) depicts the normalized norm for those parameters (the norm of parameters associated with the sensitive attribute divided by the parameter norm of the first layer) of aggregated models and local updated models for both the most and least biased parties.
Revised: Figure 10(a) depicts the normalized norm for those parameters (the norm of parameters associated with the sensitive attribute divided by the parameter norm of the first layer) of aggregated models and local updated models for both the most and least biased parties.

Original: We find that the least biased party reduces the parameter norm, hence decreasing the model’s sensitivity to the sensitive attribute.
Revised: We find that the least biased party reduces the parameter norm, hence decreasing the model’s sensitivity to the sensitive attribute.

Original: In contrast, the biased party raises norm for those parameters, hence amplifying the impact of the sensitive attribute on the model’s prediction.
Revised: In contrast, the biased party raises norm for those parameters, hence amplifying the impact of the sensitive attribute on the model’s prediction.

Original: Through aggregation, this amplification will be propagated to the global model.
Revised: Through aggregation, this amplification will be propagated to the global model.

Original: Figure 10(b) reveals a moderate correlation between the fairness gap party experiences in a standalone setting and the normalized parameter norm for the parameters used to extract sensitive attribute information.
Revised: Figure 10(b) reveals a moderate correlation between the fairness gap party experiences in a standalone setting and the normalized parameter norm for the parameters used to extract sensitive attribute information.

Original: It also demonstrates that biased parties increase the parameter value associated with sensitive attributes during the local update, thereby boosting the model’s susceptibility to the sensitive attribute.
Revised: It also demonstrates that biased parties increase the parameter value associated with sensitive attributes during the local update, thereby boosting the model’s susceptibility to the sensitive attribute.

Original: Controlling fairness gap by scaling a few parameters.
Revised: Controlling fairness gap by scaling a few parameters.

Original: To further validate the effect of these parameters on model fairness, Figure 10(c)
Revised: To further validate the effect of these parameters on model fairness, Figure 10(c)

Original: demonstrates that by scaling a few parameters associated with the sensitive attribute (i.e., 104 out of 1,792), the fairness gap can be greatly widened or narrowed at the expense of a moderate degree of accuracy.
Revised: demonstrates that by scaling a few parameters associated with the sensitive attribute (i.e., 104 out of 1,792), the fairness gap can be greatly widened or narrowed at the expense of a moderate degree of accuracy.

Original: Specifically, by scaling the parameter value by 0.
Revised: Specifically, by scaling the parameter value by 0.

Original: for the trained FL model, we can reduce the EO gap by 74.6% (from 0.198 to 0.05) and the DP gap by 69.1% (from 0.217 to 0.067) with just a 0.8% accuracy loss (from 0.785 to 0.779).
Revised: for the trained FL model, we can reduce the EO gap by 74.6% (from 0.198 to 0.05) and the DP gap by 69.1% (from 0.217 to 0.067) with just a 0.8% accuracy loss (from 0.785 to 0.779).

Original: In contrast, by raising the parameter value by a factor of 10, the EO gap increases to 0.96 (the maximal fairness gap is 1), which is almost five times large.
Revised: In contrast, by raising the parameter value by a factor of 10, the EO gap increases to 0.96 (the maximal fairness gap is 1), which is almost five times large.

Original: The DP gap increases to 0.78, which is increased by 259%.
Revised: The DP gap increases to 0.78, which is increased by 259%.

Original: At the same time, the accuracy decreases from 0.785 to 0.67.
Revised: At the same time, the accuracy decreases from 0.785 to 0.67.

Original: These findings explain how bias propagates in FL: the biased parties magnify the impact of sensitive attributes on model predictions by increasing the model parameter used to extract sensitive attributes.
Revised: These findings explain how bias propagates in FL: the biased parties magnify the impact of sensitive attributes on model predictions by increasing the model parameter used to extract sensitive attributes.

Original: This rise in parameters is subsequently propagated to the global model through aggregation, further aggravating the issue of fairness for other parties.
Revised: This rise in parameters is subsequently propagated to the global model through aggregation, further aggravating the issue of fairness for other parties.

Original: Our results explain how bias is propagated in FL: Biased parties encode bias in a few parameters through a local update, and this bias is consequently propagated to the entire network through parameter aggregation.
Revised: Our results explain how bias is propagated in FL: Biased parties encode bias in a few parameters through a local update, and this bias is consequently propagated to the entire network through parameter aggregation.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Fairness has received considerable attention due to the growing deploying of machine learning in decision-making processes.
Revised: Fairness has received considerable attention due to the growing deploying of machine learning in decision-making processes.

Original: Various definitions of fairness have been presented (Hardt et al., 2016;
Revised: Various definitions of fairness have been presented (Hardt et al., 2016;

Original: Dwork et al., 2012; Calders et al., 2009).
Revised: Dwork et al., 2012; Calders et al., 2009).

Original: Specifically, group fairness requires that the model behave similarly for groups defined by a sensitive attribute (e.g., race).
Revised: Specifically, group fairness requires that the model behave similarly for groups defined by a sensitive attribute (e.g., race).

Original: Although how machine learning algorithms propagate data bias to the final model has been extensively investigated in a centralized setting (Blum and Stangl, 2020; Lakkaraju et al., 2017; Rambachan and Roth, 2020; Friedler et al., 2019; Dullerud et al., 2022), the effect of FL on model fairness remains unclear.
Revised: Although how machine learning algorithms propagate data bias to the final model has been extensively investigated in a centralized setting (Blum and Stangl, 2020; Lakkaraju et al., 2017; Rambachan and Roth, 2020; Friedler et al., 2019; Dullerud et al., 2022), the effect of FL on model fairness remains unclear.

Original: In FL setting, a substantial body of literature on fairness focuses on the performance disparity of the
Revised: In FL setting, a substantial body of literature on fairness focuses on the performance disparity of the

Original: FL model across parties rather than demographic groups (Li et al., 2021; Zhao and Joshi, 2022;
Revised: FL model across parties rather than demographic groups (Li et al., 2021; Zhao and Joshi, 2022;

Original: Li et al., 2019; Mohri et al., 2019;
Revised: Li et al., 2019; Mohri et al., 2019;

Original: Deng et al., 2020; Donahue and Kleinberg, 2021;
Revised: Deng et al., 2020; Donahue and Kleinberg, 2021;

Original: Hao et al., 2021;
Revised: Hao et al., 2021;

Original: Zhou et al., 2021; Yu et al., 2020; Lyu et al., 2020).
Revised: Zhou et al., 2021; Yu et al., 2020; Lyu et al., 2020).

Original: We focus on group fairness, which concerns performance disparity among groups.
Revised: We focus on group fairness, which concerns performance disparity among groups.

Original: Thus, our focus differs from those studies.
Revised: Thus, our focus differs from those studies.

Original: In terms of group fairness, Abay et al.
Revised: In terms of group fairness, Abay et al.

Original: (2020) listed a few potential sources of bias in FL.
Revised: (2020) listed a few potential sources of bias in FL.

Original: Recently, considerable progress has been made in training group fair models in FL (Abay et al., 2020;
Revised: Recently, considerable progress has been made in training group fair models in FL (Abay et al., 2020;

Original: Chu et al., 2021; Zeng et al., 2021a;
Revised: Chu et al., 2021; Zeng et al., 2021a;

Original: Hu et al., 2022; Du et al., 2021; Ezzeldin et al., 2021).
Revised: Hu et al., 2022; Du et al., 2021; Ezzeldin et al., 2021).

Original: However, the majority of these works suggest techniques for achieving fairness on a single test distribution.
Revised: However, the majority of these works suggest techniques for achieving fairness on a single test distribution.

Original: Instead, we focus on fairness issues for parties.
Revised: Instead, we focus on fairness issues for parties.

Original: Cui et al.
Revised: Cui et al.

Original: ; Papadaki et al.
Revised: ; Papadaki et al.

Original: (2022) proposed algorithms to improve local fairness for parties.
Revised: (2022) proposed algorithms to improve local fairness for parties.

Original: Our purpose, instead, is to gain a comprehensive understanding of how FL influences local fairness on its own, which we believe is equally crucial as designing fair algorithms.
Revised: Our purpose, instead, is to gain a comprehensive understanding of how FL influences local fairness on its own, which we believe is equally crucial as designing fair algorithms.

Original:  F UTURE W ORK & C ONCLUSION
Revised:  F UTURE W ORK & C ONCLUSION

Original: Future Work.
Revised: Future Work.

Original: In this work, we have explained how bias is propagated in FL when the sensitive attribute is included as an input feature.
Revised: In this work, we have explained how bias is propagated in FL when the sensitive attribute is included as an input feature.

Original: In practice, however, sensitive attributes may be prohibited from being included in input features.
Revised: In practice, however, sensitive attributes may be prohibited from being included in input features.

Original: In such situations, the model may still be heavily biased based on variables that are correlated with the (unobserved) sensitive attribute.
Revised: In such situations, the model may still be heavily biased based on variables that are correlated with the (unobserved) sensitive attribute.

Original: For instance, a person’s zip code may be highly correlated with their race, a phenomenon known as "redlining".
Revised: For instance, a person’s zip code may be highly correlated with their race, a phenomenon known as "redlining".

Original: To discern which model parameters contribute to the bias and to audit the bias propagation in this scenario is a promising future work direction.
Revised: To discern which model parameters contribute to the bias and to audit the bias propagation in this scenario is a promising future work direction.

Original: Furthermore, designing FL algorithms robust to bias propagation is a promising direction.
Revised: Furthermore, designing FL algorithms robust to bias propagation is a promising direction.

Original: We briefly discuss a few potential ways in Appendix B.9.
Revised: We briefly discuss a few potential ways in Appendix B.9.

Original: Conclusion.
Revised: Conclusion.

Original: Federated learning has been widely deployed in various applications, which have significant individual-level consequences.
Revised: Federated learning has been widely deployed in various applications, which have significant individual-level consequences.

Original: Thus, it is essential to anticipate the possible bias introduced by
Revised: Thus, it is essential to anticipate the possible bias introduced by

Original: FL.
Revised: FL.

Original: Our paper makes the first step in this direction, offering a comprehensive analysis of the impact of FL (i.e., FedAvg) on local fairness for parties.
Revised: Our paper makes the first step in this direction, offering a comprehensive analysis of the impact of FL (i.e., FedAvg) on local fairness for parties.

Original: We demonstrated that the FL algorithm could introduce bias on its own.
Revised: We demonstrated that the FL algorithm could introduce bias on its own.

Original: Consequently, FL may exacerbate the issue of fairness for the involved parties.
Revised: Consequently, FL may exacerbate the issue of fairness for the involved parties.

Original: In addition, we demonstrated that this exacerbation is not equally distributed among parties.
Revised: In addition, we demonstrated that this exacerbation is not equally distributed among parties.

Original: This is due to the fact that FL can propagate bias among parties.
Revised: This is due to the fact that FL can propagate bias among parties.

Original: Finally, we provided potential explanations about how bias is propagated in FL.
Revised: Finally, we provided potential explanations about how bias is propagated in FL.

Original: Specifically, biased parties encode their bias into the local updates by steadily increasing the signal of a few parameters throughout the training process.
Revised: Specifically, biased parties encode their bias into the local updates by steadily increasing the signal of a few parameters throughout the training process.

Original: Such bias is then propagated to other parties via aggregation and ultimately, to the global model.
Revised: Such bias is then propagated to other parties via aggregation and ultimately, to the global model.

Original:  E THICS S TATEMENT
Revised:  E THICS S TATEMENT

Original: Our analysis focuses on group fairness, which is typically used to audit the model or system for any bias or discrimination.
Revised: Our analysis focuses on group fairness, which is typically used to audit the model or system for any bias or discrimination.

Original: Auditing the bias with respect to other definitions of fairness, such as individual fairness, may result in different conclusions.
Revised: Auditing the bias with respect to other definitions of fairness, such as individual fairness, may result in different conclusions.

Original: Moreover, our study focuses primarily on the binary notion of sex attributes and the multi-valued race attribute.
Revised: Moreover, our study focuses primarily on the binary notion of sex attributes and the multi-valued race attribute.

Original: We recognize that there are numerous protected groups outside those considered in the analysis, such as those defined by multiple sensitive attributes.
Revised: We recognize that there are numerous protected groups outside those considered in the analysis, such as those defined by multiple sensitive attributes.

Original: The propagation of bias against fine-grained subgroups maybe even more substantial than we found in the paper.
Revised: The propagation of bias against fine-grained subgroups maybe even more substantial than we found in the paper.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: We provide the code and script for downloading the data in the supplemental materials.
Revised: We provide the code and script for downloading the data in the supplemental materials.

Original: More details can be found in the “REAME.md” file in the supplemental materials.
Revised: More details can be found in the “REAME.md” file in the supplemental materials.

Original: In addition, we provide details about the model, datasets, and implementations in Appendix A.
Revised: In addition, we provide details about the model, datasets, and implementations in Appendix A.

Original: A A DDITIONAL D ETAILS ABOUT E XPERIMENTS
Revised: A A DDITIONAL D ETAILS ABOUT E XPERIMENTS

Original: Datasets and Models We explain the datasets and models used in the paper.
Revised: Datasets and Models We explain the datasets and models used in the paper.

Original: Census Dataset
Revised: Census Dataset

Original: We use the datasets provided by folktables.
Revised: We use the datasets provided by folktables.

Original: In particular, we consider the ACSIncome, ACSPublicCoverage, and ACSEmployment tasks defined in the forlktables.
Revised: In particular, we consider the ACSIncome, ACSPublicCoverage, and ACSEmployment tasks defined in the forlktables.

Original: In the ACSIncome, the goal is to predict whether an individual’s Income is above than $50,000.
Revised: In the ACSIncome, the goal is to predict whether an individual’s Income is above than $50,000.

Original: In ACSEmployment task, the goal is to predict whether an individual is employed.
Revised: In ACSEmployment task, the goal is to predict whether an individual is employed.

Original: Similarly, in the Health (i.e., ACSPublicCoverage) task, the objective is to predict whether an individual is covered by public health insurance.
Revised: Similarly, in the Health (i.e., ACSPublicCoverage) task, the objective is to predict whether an individual is covered by public health insurance.

Original: We use the same pre-processing as in the folktables and train a fully connected neural network model with one hidden layer of 32 neurons for Income and 64 neurons for Employment and Health tasks.
Revised: We use the same pre-processing as in the folktables and train a fully connected neural network model with one hidden layer of 32 neurons for Income and 64 neurons for Employment and Health tasks.

Original: For all the tasks, we use the RELU activation function.
Revised: For all the tasks, we use the RELU activation function.

Original: We use an SGD optimizer with a learning rate of 0.001 for centralized training on Health and Employment datasets and 0.1 for other settings, and the batch size is 32.
Revised: We use an SGD optimizer with a learning rate of 0.001 for centralized training on Health and Employment datasets and 0.1 for other settings, and the batch size is 32.

Original: We train the NN models for 200 epochs.
Revised: We train the NN models for 200 epochs.

Original: In FL, each client updates the global mdoel for 1 epoch and shares it with the server.
Revised: In FL, each client updates the global mdoel for 1 epoch and shares it with the server.

Original: We encode the categorical features based on the encoding template provided in folktables.
Revised: We encode the categorical features based on the encoding template provided in folktables.

Original: After the encoding, the input feature size for Income is 54, 154 for Health, and 109 for Employment.
Revised: After the encoding, the input feature size for Income is 54, 154 for Health, and 109 for Employment.

Original: We consider sex and race as sensitive attributes.
Revised: We consider sex and race as sensitive attributes.

Original: Accordingly, there are two gender groups (male and female) and nine racial groups ("White alone," "Black or African American alone," "American Indian alone," "Alaska Native alone," and "American
Revised: Accordingly, there are two gender groups (male and female) and nine racial groups ("White alone," "Black or African American alone," "American Indian alone," "Alaska Native alone," and "American

Original: Indian and Alaska Native tribes specified; or American Indian or Alaska Native, not specified and no other," "Asian alone," "Native Hawaiian and Other Pacific Islander alone," "Some Other Race alone," "Two or More Races").
Revised: Indian and Alaska Native tribes specified; or American Indian or Alaska Native, not specified and no other," "Asian alone," "Native Hawaiian and Other Pacific Islander alone," "Some Other Race alone," "Two or More Races").

Original: CelebA Dataset We train CNN models on the dataset with one CNN layer whose output channel is 32, kernel size is 3, and stride is 1.
Revised: CelebA Dataset We train CNN models on the dataset with one CNN layer whose output channel is 32, kernel size is 3, and stride is 1.

Original: We use ’same’ padding for the CNN layer.
Revised: We use ’same’ padding for the CNN layer.

Original: Following this
Revised: Following this

Original: CNN layer, we have the Batch normalization layer and Max Pooling layer.
Revised: CNN layer, we have the Batch normalization layer and Max Pooling layer.

Original: After which, we have the connected layer.
Revised: After which, we have the connected layer.

Original: We train the model for 500 communication rounds or epochs with SGD optimizer.
Revised: We train the model for 500 communication rounds or epochs with SGD optimizer.

Original: The learning rate is 0.1, and the batch size is 128.
Revised: The learning rate is 0.1, and the batch size is 128.

Original: The train, test, and validation datasets ratio for each party is 6:2:2.
Revised: The train, test, and validation datasets ratio for each party is 6:2:2.

Original: Hyper-parameter for FL algorithms
Revised: Hyper-parameter for FL algorithms

Original: In our paper, we also evaluate various popular FL algorithms, including FedNova (Wang et al., 2020), Scaffold Karimireddy et al.
Revised: In our paper, we also evaluate various popular FL algorithms, including FedNova (Wang et al., 2020), Scaffold Karimireddy et al.

Original: (2020b), FedOpt (Reddi et al., 2020), FedProx Li et al.
Revised: (2020b), FedOpt (Reddi et al., 2020), FedProx Li et al.

Original: (2020), and Mime (Karimireddy et al., 2020a).
Revised: (2020), and Mime (Karimireddy et al., 2020a).

Original: We use the same local learning rate and the number of the local epoch as in FedAvg.
Revised: We use the same local learning rate and the number of the local epoch as in FedAvg.

Original: We provided the detailed hyper-parameters for each of the algorithms in our code (See supplementary).
Revised: We provided the detailed hyper-parameters for each of the algorithms in our code (See supplementary).

Original: Implementation We run all experiments on Ubuntu with two NVIDIA TITAN RTX GPUs.
Revised: Implementation We run all experiments on Ubuntu with two NVIDIA TITAN RTX GPUs.

Original: B A DDITIONAL E XPERIMENTAL R ESULTS B. B IAS PROPAGATION EFFECT
Revised: B A DDITIONAL E XPERIMENTAL R ESULTS B. B IAS PROPAGATION EFFECT

Original: Census dataset We show the bias propagation effect of FL for Health and Employment task in
Revised: Census dataset We show the bias propagation effect of FL for Health and Employment task in

Original: Figure 11 and 12 respectively.
Revised: Figure 11 and 12 respectively.

Original: CelebA We show the bias propagation effect of FL on CelebA dataset.
Revised: CelebA We show the bias propagation effect of FL on CelebA dataset.

Original: Besides partitioning the data in an IID manner (we refer to as setting (i)), we also evaluate two different settings, where we change the number of samples from the minority subgroup (the female group with the “Not Young” age label).
Revised: Besides partitioning the data in an IID manner (we refer to as setting (i)), we also evaluate two different settings, where we change the number of samples from the minority subgroup (the female group with the “Not Young” age label).

Original: In this way, we aim to change the data bias in the local training datasets for clients.
Revised: In this way, we aim to change the data bias in the local training datasets for clients.

Original: In particular, in setting (ii), half of the parties have more samples from the minority subgroup than another half of the parties.
Revised: In particular, in setting (ii), half of the parties have more samples from the minority subgroup than another half of the parties.

Original: The ratio is 8:2.
Revised: The ratio is 8:2.

Original: In setting (iii), a single party has half of the data from the old female, and the data is then iid partitioned among the other parties.
Revised: In setting (iii), a single party has half of the data from the minority subgroup, and the data is then iid partitioned among the other parties.

Original: In the figure, we find that the in the non-iid setting, the more biased parties have a larger and more positive benefit, while FL hurts the less biased parties.
Revised: In the figure, we find that the in the non-iid setting, the more biased parties have a larger and more positive benefit, while FL hurts the less biased parties.

Original: Figure 13 shows the correlation between the fairness gap of the standalone model and the benefit a party gets in the FL.
Revised: Figure 13 shows the correlation between the fairness gap of the standalone model and the benefit a party gets in the FL.

Original: We find that the in the non-iid setting, the more biased parties have a larger and more positive benefit, while FL hurts the less biased parties.
Revised: We find that the in the non-iid setting, the more biased parties have a larger and more positive benefit, while FL hurts the less biased parties.

Original: B. A GGREGATION CONTRADICTS WITH LOCAL UPDATE
Revised: B. A GGREGATION CONTRADICTS WITH LOCAL UPDATE

Original: Figure 14 shows the dynamic of the fairness gap for the aggregated model and local updated model from the most biased party and least biased party during the training
Revised: Figure 14 shows the dynamic of the fairness gap for the aggregated model and local updated model from the most biased party and least biased party during the training

Original: . B. I NFLUENCE S UBGRAPHS
Revised: . B. I NFLUENCE S UBGRAPHS

Original: We show the top 10 maximal positive and top 10 maximal native influence client pairs in Figureand top 15 in Figure 16.
Revised: We show the top 10 maximal positive and top 10 maximal native influence client pairs in Figureand top 15 in Figure 16.

Original: We can see that the less biased client has a strong positive influence on other clients while the more biased client has a strong negative influence on other clients
Revised: We can see that the less biased client has a strong positive influence on other clients while the more biased client has a strong negative influence on other clients

Original: . B. A TTRIBUTION VALUE FOR SENSITIVE ATTRIBUTE
Revised: . B. A TTRIBUTION VALUE FOR SENSITIVE ATTRIBUTE

Original: We show the attribution value for all input features for the most biased party and least biased party in Figure 17 and Figure 18, respectively.
Revised: We show the attribution value for all input features for the most biased party and least biased party in Figure 17 and Figure 18, respectively.

Original: We observe that the attribution value for other features is similar for female and male groups.
Revised: We observe that the attribution value for other features is similar for female and male groups.

Original: However, the sex attribute has the largest attribution value
Revised: However, the sex attribute has the largest attribution value

Original: Aggregation and local updates have contradictory impacts on model fairness for those parties.
Revised: Aggregation and local updates have contradictory impacts on model fairness for those parties.

Original: and affects the groups differently.
Revised: and affects the groups differently.

Original: It implies that the models are highly dependent on the sensitive attribute (i.e., "Sex") for making predictions.
Revised: It implies that the models are highly dependent on the sensitive attribute (i.e., "Sex") for making predictions.

Original: B. E FFECT OF FL ALGORITHM
Revised: B. E FFECT OF FL ALGORITHM

Original: Table 2 shows results on other FL algorithms, including FedNova (Wang et al., 2020),
Revised: Table 2 shows results on other FL algorithms, including FedNova (Wang et al., 2020),

Original: Scaffold Karimireddy et al.
Revised: Scaffold Karimireddy et al.

Original: (2020b), FedOpt (Asad et al., 2020), FedProx Li et al.
Revised: (2020b), FedOpt (Asad et al., 2020), FedProx Li et al.

Original: (2020), and Mime (Karimireddy et al., 2020a).
Revised: (2020), and Mime (Karimireddy et al., 2020a).

Original: We find that when the FL model achieves higher accuracy than standalone models, the fairness gap of the FL model can be higher than that of standalone models.
Revised: We find that when the FL model achieves higher accuracy than standalone models, the fairness gap of the FL model can be higher than that of standalone models.

Original: This observation is consistent across multiple FL algorithms.
Revised: This observation is consistent across multiple FL algorithms.

Original: This strong evidence implies that the improvement of accuracy in FL can come at the cost of fairness.
Revised: This strong evidence implies that the improvement of accuracy in FL can come at the cost of fairness.

Original: In addition, this is not unique to only the FedAvg algorithm.
Revised: In addition, this is not unique to only the FedAvg algorithm.

Original: B. G ROUP P ERFORMANCE
Revised: B. G ROUP P ERFORMANCE

Original: In Figure we present the model ROC on groups to illustrate the performance disparity across groups.
Revised: In Figure we present the model ROC on groups to illustrate the performance disparity across groups.

Original: Surprisingly, there is no huge difference between the ROC between groups.
Revised: Surprisingly, there is no huge difference between the ROC between groups.

Original: In contrast,
Revised: In contrast,

Original: [Table]
Revised: [Table]

Original: Figure 20 shows the noticeable difference between groups with respect to the precision-recall curve, especially for the most biased party.
Revised: Figure 20 shows the noticeable difference between groups with respect to the precision-recall curve, especially for the most biased party.

Original: The main reason is that the dataset is imbalanced.
Revised: The main reason is that the dataset is imbalanced.

Original: Thus, ROC is usually misleading.
Revised: Thus, ROC is usually misleading.

Original: On the precision-recall curve, we find that the model achieves a higher precision on the majority (i.e., Male group) compared to the minority group (i.e., Female group).
Revised: On the precision-recall curve, we find that the model achieves a higher precision on the majority (i.e., Male group) compared to the minority group (i.e., Female group).

Original: B. D ATA H ETEROGENEITY
Revised: B. D ATA H ETEROGENEITY

Original: Figure 21 shows the fraction of samples of each subgroup for all parties, and Figure 22 shows the histogram of the proportion of samples from each subgroup among parties.
Revised: Figure 21 shows the fraction of samples of each subgroup for all parties, and Figure 22 shows the histogram of the proportion of samples from each subgroup among parties.

Original: We observe that parties have different fractions of samples from each group on Income dataset, indicating that their local data distribution differs.
Revised: We observe that parties have different fractions of samples from each group on Income dataset, indicating that their local data distribution differs.

Original: While on the Health and Employment dataset, parties have a similar fraction of the samples from subgroups.
Revised: While on the Health and Employment dataset, parties have a similar fraction of the samples from subgroups.

Original: It implies that the data distribution for parties would be more similar to each other than that on the Income dataset.
Revised: It implies that the data distribution for parties would be more similar to each other than that on the Income dataset.

Original: B. E FFECT OF SCALING OTHER PARAMETERS
Revised: B. E FFECT OF SCALING OTHER PARAMETERS

Original: Figure 23 compares the performance of the model when scaling different model parameters.
Revised: Figure 23 compares the performance of the model when scaling different model parameters.

Original: In particular, we scale the parameters for the first layer of the neural network that does not have any computation on the Sex attribute (referred to as "Other parameters").
Revised: In particular, we scale the parameters for the first layer of the neural network that does not have any computation on the Sex attribute (referred to as "Other parameters").

Original: In addition, we also show the model performance when scaling the parameter that is related to the Sex attribute (referred to as "Related parameters").
Revised: In addition, we also show the model performance when scaling the parameter that is related to the Sex attribute (referred to as "Related parameters").

Original: Clearly, we see that, by up-scaling or down-scaling other parameters does not fairness gap significantly while scaling the related parameters has a large impact on model fairness.
Revised: Clearly, we see that, by up-scaling or down-scaling other parameters does not fairness gap significantly while scaling the related parameters has a large impact on model fairness.

Original: This validate our hypothesis that the bias of the model is mainly encoded in a few model parameters.
Revised: This validate our hypothesis that the bias of the model is mainly encoded in a few model parameters.

Original: In FL, the biased party introduces the bias during the local training by increasing the weights for those related parameters.
Revised: In FL, the biased party introduces the bias during the local training by increasing the weights for those related parameters.

Original: B. P OTENTIAL M ITIGATION
Revised: B. P OTENTIAL M ITIGATION

Original: In this section, we offer a few potential solutions for reducing bias propagation effect of FL.
Revised: In this section, we offer a few potential solutions for reducing bias propagation effect of FL.

Original: We hope it will inspire more study on developing fair FL in the future.
Revised: We hope it will inspire more study on developing fair FL in the future.

Original: Personalized FL We showed in the paper that the aggregate update (average local update from all parties) could be more biased than the local update for less biased parties (shown in Figure 3).
Revised: Personalized FL We showed in the paper that the aggregate update (average local update from all parties) could be more biased than the local update for less biased parties (shown in Figure 3).

Original: As a result, when less biased parties overwrite the local model with the aggregate update, the bias from other parties will be propagated to the local model, increasing the local fairness gap.
Revised: As a result, when less biased parties overwrite the local model with the aggregate update, the bias from other parties will be propagated to the local model, increasing the local fairness gap.

Original: Thus, one way to reduce the bias propagation effect is to avoid overwriting the local model completely using the global model.
Revised: Thus, one way to reduce the bias propagation effect is to avoid overwriting the local model completely using the global model.

Original: Recent personalized FL algorithms (Li et al., 2021) have been proposed based on a similar idea but with a focus on improving the model’s accuracy.
Revised: Recent personalized FL algorithms (Li et al., 2021) have been proposed based on a similar idea but with a focus on improving the model’s accuracy.

Original: Fairness-aware personalized federated learning could be a promising direction to reduce the bias propagation effect of FL.
Revised: Fairness-aware personalized federated learning could be a promising direction to reduce the bias propagation effect of FL.

Original: Fair Representation Learning We demonstrated that bias is encoded in a few parameters of the first layer of the neural network, which is usually considered the feature extractor (shown in
Revised: Fair Representation Learning We demonstrated that bias is encoded in a few parameters of the first layer of the neural network, which is usually considered the feature extractor (shown in

Original: Figure 10).
Revised: Figure 10).

Original: One interpretation of the result is that the FL model is biased because the representation learned by the model is biased (highly dependent on the sensitive attribute).
Revised: One interpretation of the result is that the FL model is biased because the representation learned by the model is biased (highly dependent on the sensitive attribute).

Original: Fair representation learning (Zemel et al., 2013; Zhao et al., 2019; Liu et al., 2022; Zeng et al., 2021b) aims to learn a fair feature extractor such that the representation has limited (no) information about the sensitive attribute while still containing sufficient information about the task (or the input features).
Revised: Fair representation learning (Zemel et al., 2013; Zhao et al., 2019; Liu et al., 2022; Zeng et al., 2021b) aims to learn a fair feature extractor such that the representation has limited (no) information about the sensitive attribute while still containing sufficient information about the task (or the input features).

Original: Thus, one potential way to reduce the bias propagation effect is to learn fair representation in FL.
Revised: Thus, one potential way to reduce the bias propagation effect is to learn fair representation in FL.

Original: [Table]
Revised: [Table]

Original: E VALUATION ON EXISTING F AIR FL
Revised: E VALUATION ON EXISTING F AIR FL

Original: Table 3 shows the results for the fair FL algorithm proposed by Abay et al.
Revised: Table 3 shows the results for the fair FL algorithm proposed by Abay et al.

Original: (2020), which is based on the reweighting algorithm proposed in the centralized setting (Kamiran and Calders, 2012).
Revised: (2020), which is based on the reweighting algorithm proposed in the centralized setting (Kamiran and Calders, 2012).

Original: Specifically, before FL, parties assign the weight for each subgroup (defined by the label and a sensitive attribute) based on the local training dataset (which is referred to as "local reweighting") or global training dataset (which is referred to as "global reweighting").
Revised: Specifically, before FL, parties assign the weight for each subgroup (defined by the label and a sensitive attribute) based on the local training dataset (which is referred to as "local reweighting") or global training dataset (which is referred to as "global reweighting").

Original: During the FL, parties updates the FL model to minimize the weighted loss.
Revised: During the FL, parties updates the FL model to minimize the weighted loss.

Original: We found that applying reweighing algorithms in FL reduces the average fairness gap across parties.
Revised: We found that applying reweighing algorithms in FL reduces the average fairness gap across parties.

Original: The reason is that after the reweighing, the originally biased parties will not introduce significant bias to the model during the local training, which ultimately reduces the fairness gap of the FL model.
Revised: The reason is that after the reweighing, the originally biased parties will not introduce significant bias to the model during the local training, which ultimately reduces the fairness gap of the FL model.

Original: At the same time, we notice that the fairness gap of the model is still high with respect to Race sensitive attributes, which means that the global and local reweighing algorithms do not mitigate the bias completely in FL.
Revised: At the same time, we notice that the fairness gap of the model is still high with respect to Race sensitive attributes, which means that the global and local reweighing algorithms do not mitigate the bias completely in FL.

Original: In addition, we would like to point out some concerns with respect to the fair
Revised: In addition, we would like to point out some concerns with respect to the fair

Original: FL algorithm:
Revised: FL algorithm:

Original: The fair FL algorithm improves the performance of the minority group at the cost of the majority group.
Revised: The fair FL algorithm improves the performance of the minority group at the cost of the majority group.

Original: Figure 24 shows the model performance per group before and after the global reweighting.
Revised: Figure 24 shows the model performance per group before and after the global reweighting.

Original: We observe that, after the reweighting, the minority group (i
Revised: We observe that, after the reweighting, the minority group (i

Original: e., the Female group) has a better performance.
Revised: e., the Female group) has a better performance.

Original: However, this performance improvement comes at the cost of the performance for the majority.
Revised: However, this performance improvement comes at the cost of the performance for the majority.

Original: From Table 3, we notice that when improving the fairness with respect to a sensitive attribute, the fairness issue with respect to another sensitive attribute becomes worse.
Revised: From Table 3, we notice that when improving the fairness with respect to a sensitive attribute, the fairness issue with respect to another sensitive attribute becomes worse.

Original: For example, when local reweighting is applied to reduce the bias with respect to "Sex" attribute, the EO fairness gap with respect to "Race" increased from 0.514 (results on FedAvg) to 0.521.
Revised: For example, when local reweighting is applied to reduce the bias with respect to "Sex" attribute, the EO fairness gap with respect to "Race" increased from 0.514 (results on FedAvg) to 0.521.

Original: It indicates that if some parties reweight the local dataset to improve fairness with respect to Sex it may amplify the bias with respect to Race.
Revised: It indicates that if some parties reweight the local dataset to improve fairness with respect to Sex it may amplify the bias with respect to Race.

Original: The bias with respect to Race can still be propagated in FL, and our analysis still holds.
Revised: The bias with respect to Race can still be propagated in FL, and our analysis still holds.

Original: The algorithms consider the setting where parties are interested in mitigating the bias with respect to the same sensitive attribute, which may not be the case in practice.
Revised: The algorithms consider the setting where parties are interested in mitigating the bias with respect to the same sensitive attribute, which may not be the case in practice.

Original: For example, Figure 25 shows the fairness gap of the standalone models with respect to different sensitive attributes.
Revised: For example, Figure 25 shows the fairness gap of the standalone models with respect to different sensitive attributes.

Original: We find that there exist lots of parties whose standalone model achieves a low fairness gap with respect to one sensitive attribute and a high fairness gap with respect to another sensitive attribute.
Revised: We find that there exist lots of parties whose standalone model achieves a low fairness gap with respect to one sensitive attribute and a high fairness gap with respect to another sensitive attribute.

Original: Therefore, parties may be interested in improving fairness with respect to different sensitive
Revised: Therefore, parties may be interested in improving fairness with respect to different sensitive

Original: attributes.
Revised: attributes.

Original: In those settings, this reweighing algorithm may not be applicable.
Revised: In those settings, this reweighing algorithm may not be applicable.

Original: This problem is more common in FL as parties have different data distributions.
Revised: This problem is more common in FL as parties have different data distributions.

Original: B. E XTENDING TO A REAL - WORLD MEDICAL DATASET
Revised: B. E XTENDING TO A REAL - WORLD MEDICAL DATASET

Original: We evaluate the effect of FL on local fairness on a real-world medical dataset, ISIC

Revised: We evaluate the effect of FL on local fairness on a real-world medical dataset, ISIC

Original: (Abay et al., 2020;
Revised: (Abay et al., 2020;

Original: Tschandl et al., 2018; Combalia et al., 2019), which contains dermoscopic images of skin lesions collected from six medical centers.
Revised: Tschandl et al., 2018; Combalia et al., 2019), which contains dermoscopic images of skin lesions collected from six medical centers.

Original: The task is to classify dermoscopic images among nine different diagnostic categories.
Revised: The task is to classify dermoscopic images among nine different diagnostic categories.

Original: We filter out medical centers with less than 1900 images and
Revised: We filter out medical centers with less than 1900 images and

Original: [Table]
Revised: [Table]

Original: regard each medical center with over 1900 images as a party (4 centers remain after filtering).
Revised: regard each medical center with over 1900 images as a party (4 centers remain after filtering).

Original: For every party
Revised: For every party

Original: [Table]
Revised: [Table]

Original: we randomly select 1500 and 400 images for training and validation
Revised: we randomly select 1500 and 400 images for training and validation

Original: [Table]
Revised: [Table]

Original: respectively.
Revised: respectively.

Original: We regard the binary notion of Sex as our sensitive attribute.
Revised: We regard the binary notion of Sex as our sensitive attribute.

Original: In this multi-class and medical dataset setting
Revised: In this multi-class and medical dataset setting

Original: [Table]
Revised: [Table]

Original: we measure the bias (fairness gap) based on the accuracy gap across groups.
Revised: we measure the bias (fairness gap) based on the accuracy gap across groups.

Original: Following (fla, 2022)
Revised: Following (fla, 2022)

Original: [Table]
Revised: [Table]

Original: we train EfficientNets
Revised: we train EfficientNets

Original: (Tan and Le, 2019) [Table] the state-of-the-art model structure for medical data, under the centralized, FL
Revised: (Tan and Le, 2019), the state-of-the-art model structure for medical data, under the centralized, FL

Original: [Table]
Revised: [Table]

Original: and standalone setting.
Revised: and standalone setting.

Original: The accuracy and fairness gap for each client in different settings are shown in Table 4.
Revised: The accuracy and fairness gap for each client in different settings are shown in Table 4.

Original: We notice that FedAvg does not improve the accuracy.
Revised: We notice that FedAvg does not improve the accuracy.

Original: The reason may be the data heterogeneity among parties.
Revised: The reason may be the data heterogeneity among parties.

Original: As we can see from Figure 26, the numbers of samples from each class are different for the local datasets of different parties.
Revised: As we can see from Figure 26, the numbers of samples from each class are different for the local datasets of different parties.

Original: The existing theory (Karimireddy et al., 2020b) also proves that FedAvg suffers from ‘client-drift’.
Revised: The existing theory (Karimireddy et al., 2020b) also proves that FedAvg suffers from ‘client-drift’.

Original: Namely, when the data is heterogeneous among parties, the FedAvg model is unstable and converges slowly.
Revised: Namely, when the data is heterogeneous among parties, the FedAvg model is unstable and converges slowly.

Original: Besides, the FedAvg model also fails to achieve the same accuracy across groups, resulting in a higher accuracy gap between groups compared to the standalone setting or the centralized setting (see the fairness gaps on party 1, party 3, and party 4 as examples).
Revised: Besides, the FedAvg model also fails to achieve the same accuracy across groups, resulting in a higher accuracy gap between groups compared to the standalone setting or the centralized setting (see the fairness gaps on party 1, party 3, and party 4 as examples).

Original: Our results indicate that, in real-world settings, FedAvg may not improve the accuracy compared to the standalone setting for the local data distribution, and in the meantime, it even worsens the fairness issue, resulting in a more biased model.
Revised: Our results indicate that, in real-world settings, FedAvg may not improve the accuracy compared to the standalone setting for the local data distribution, and in the meantime, it even worsens the fairness issue, resulting in a more biased model.