Original: B IAS P ROPAGATION IN F EDERATED L EARNING
Revised: B IAS P ROPAGATION IN F EDERATED L EARNING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We show that participating in federated learning can be detrimental to group fairness.
Revised: We show that participating in federated learning can be detrimental to group fairness.

Original: In fact, the bias of a few biased parties against under-represented groups (identified by sensitive attributes such as gender or race) propagates through the network to all parties.
Revised: In fact, the bias of a few biased parties against under-represented groups (identiﬁed by sensitive attributes such as gender or race) propagates through the network to all parties.

Original: On naturally partitioned real-world datasets, we analyze and explain bias propagation in federated learning.
Revised: On naturally partitioned real-world datasets, we analyze and explain bias propagation in federated learning.

Original: Our analysis reveals that biased parties unintentionally yet stealthily encode their bias in a small number of model parameters, and throughout the training, they steadily increase the dependence of the global model on sensitive attributes.
Revised: Our analysis reveals that biased parties unintentionally yet stealthily encode their bias in a small number of model parameters, and throughout the training, they steadily increase the dependence of the global model on sensitive attributes.

Original: What is important to highlight is that the experienced bias in federated learning is higher than what parties would otherwise encounter in centralized training with a model trained on the union of all their data.
Revised: What is important to highlight is that the experienced bias in federated learning is higher than what parties would otherwise encounter in centralized training with a model trained on the union of all their data.

Original: This indicates that the bias is due to the algorithm.
Revised: This indicates that the bias is due to the algorithm.

Original: Our work calls for auditing group fairness in federated learning, and designing learning algorithms that are robust to bias propagation.
Revised: Our work calls for auditing group fairness in federated learning and designing learning algorithms that are robust to bias propagation.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Machine learning models can be biased against demographic groups.
Revised: Machine learning models can exhibit bias against demographic groups.

Original: There has been plenty of work studying how machine learning algorithms reflect andeven amplifythe bias inthe training data inthe centralized setting, wherethe data is held by a single party (Hardt et al., 2016; Dwork et al., 2012; 
Revised: Previous research has extensively studied how machine learning algorithms can reﬂect and amplify bias in training data, especially in centralized settings where data is held by a single party (Hardt et al., 2016; Dwork et al., 2012; Calders et al., 2009;

Original: Calders et al., 2009; Hashimoto et al., 2018; Zhang et al., 2020; Blum and Stangl, 2020; Lakkaraju et al., 2017).
Revised: Hashimoto et al., 2018; Zhang et al., 2020; Blum and Stangl, 2020; Lakkaraju et al., 2017).

Original: However, in practice, data is commonly owned by multiple parties and cannot be shared due to privacy concerns.
Revised: However, in practice, data is commonly owned by multiple parties and cannot be shared due to privacy concerns.

Original: Federated learning (FL) enables parties to collaboratively learn a global model without sharing the data.
Revised: Federated learning (FL) provides a promising solution by enabling parties to collaboratively learn a global model without sharing their data.

Original: In each round, parties share the local model update computed on the dataset with a global server, and the server aggregates local updates to update the global model.
Revised: In each round of FL, parties share their local model update computed on their own dataset with a global server, which aggregates these updates to update the global model.

Original: Although FLis now widely deployed in various applications, such as healthcare, recruitment, and loan evaluation (Rieke et al., 2020; Yang et al., 2019), it is not  fully understoodwhether and how FL algorithms could magnifythe bias inthe training datasets.
Revised: Despite the widespread adoption of FL in various applications such as healthcare, recruitment, and loan evaluation (Rieke et al., 2020; Yang et al., 2019), it is not yet fully understood how FL algorithms could magnify bias in training datasets.

Original: A few recent works study this problem focusing on measuring and mitigatingthe bias in FL with respect to a single global distribution (Chu et al., 2021; Zeng et al., 2021a;
Revised: Recent studies have investigated the problem of measuring and mitigating bias in federated learning (FL) models with respect to a single global distribution (Chu et al., 2021; Zeng et al., 2021a;

Original: Hu et al., 2022; Du et al., 2021; Abay et al., 2020; Papadaki et al., 2021; 2022; Hu et al., 2022).
Revised: Hu et al., 2022; Du et al., 2021; Abay et al., 2020; Papadaki et al., 2021; 2022; Hu et al., 2022).

Original: However,  parties usually have heterogeneous data distributions.
Revised: However, in practice, parties often have heterogeneous data distributions.

Original: Evaluating the model bias with respect to the global distribution does not reflect how fair the FL modelis with respect to parties’heterogeneous local data distributions (which are of relevance to users).
Revised: Evaluating the model’s bias with respect to the global distribution does not accurately reﬂect the fairness of the FL model with respect to parties’ local data distributions, which are relevant to end-users.

Original: This isexactly the critical problem that we address in this paper.
Revised: This is the critical problem that we address in this paper.

Original: Specifically, we ask: compared to standalone training (train model without any collaboration), can parties benefit from joining FL to reduce bias and build fairer models?
Revised: Speciﬁcally, we investigate the following questions: How does participating in FL affect the bias and fairness of the resulting models compared to standalone training?

Original: Does FL provide parties withan equivalent fairness benefit of centralized training (on the union of their data)?
Revised: Does FL provide parties with equivalent fairness beneﬁts to centralized training on the union of their data?

Original: Can parties with biased datasets deteriorate the local fairness of other parties?
Revised: Can parties with biased datasets negatively impact the local fairness of other parties?

Original: How and why does  bias in a small number of parties affect the  network?
Revised: How and why does the bias of a small number of parties affect the entire network?

Original: To the best of our knowledge, we provide the first comprehensive analysis of how the FL algorithm impacts local fairness.
Revised: To the best of our knowledge, we provide the ﬁrst comprehensive analysis of how the FL algorithm impacts local fairness.

Original: We provide an empirical analysis based on real-world datasets.
Revised: We provide an empirical analysis based on real-world datasets.

Original: We show that the FL might not sustain the benefits of collaboration in terms of fairness, as compared to its accuracy benefit.
Revised: We show that the FL might not sustain the beneﬁts of collaboration in terms of fairness, as compared to its accuracy beneﬁt.

Original: Specifically, we find that the model trained in a centralized setting can be, on average, fairer on local data distributions than the standalone models.
Revised: Speciﬁcally, we ﬁnd that the model trained in a centralized setting can be, on average, fairer on local data distributions than the standalone models.

Original: However, in those cases, the FL model, trained on the same dataset, can be more biased.
Revised: However, in those cases, the FL model, trained on the same dataset, can be more biased.

Original: This suggests that the FL algorithm itself can introduce new bias in the final model, deteriorating fairness for parties.
Revised: This suggests that the FL algorithm itself can introduce new bias in the ﬁnal model, deteriorating fairness for parties.

Original: Furthermore, we demonstrate that FL impacts local fairness for parties in different ways.
Revised: Furthermore, we demonstrate that FL impacts local fairness for parties in different ways.

Original: Specifically, we find a strong correlation between parties’ fairness gap in the standalone setting and the fairness benefit they obtain from joining FL: parties with greater bias in the standalone setting (caused by their local data) would receive a fairer model from FL.
Revised: Speciﬁcally, we ﬁnd a strong correlation between parties’ fairness gap in the standalone setting and the fairness beneﬁt they obtain from joining FL: parties with greater bias in the standalone setting (caused by their local data) would receive a fairer model from FL.

Original: In contrast,
Revised: In contrast,

Original: FL has negative impact on parties with less significant bias in the standalone setting, resulting in a more biased model in FL.
Revised: FL has negative impacts on parties with less signiﬁcant bias in the standalone setting, resulting in a more biased model in FL.

Original: We further demonstrate that this is due to the fact that FL propagates bias : bias from a few parties can now influence all parties in the network, hence aggravating the fairness problem among all parties.
Revised: We further demonstrate that this is due to the fact that FL propagates bias : bias from a few parties inﬂuences all parties in the network, hence aggravating the fairness problem among all parties.

Original: Finally, we offer potential explanations for how bias is propagated in FL.
Revised: Finally, we offer potential explanations for how bias is propagated in FL.

Original: In particular, we show that local updates from biased parties increase the model’s prediction dependence on the sensitive attribute.
Revised: Speciﬁcally, we show that local updates from biased parties increase the model’s prediction dependence on the sensitive attribute.

Original: Such  increase is achieved by the norm increase of a few parameters (e.g., 104 parameters out of 1,792 parameters).
Revised: Such an increase is achieved by the norm increase in a small number of parameters (e.g., 104 parameters out of 1,792 parameters).

Original: This increase then impacts the global model through aggregation, subsequently impacting all other parties.
Revised: This increase then propagates to the global model through aggregation, subsequently impacting all other parties.

Original: In addition, wealso show that the fairness gap of the final model can be governed by adjusting the value of those parameters.
Revised: In addition, we show that the fairness gap of the ﬁnal model can be governed by adjusting the value of those parameters.

Original: Surprisingly,by scaling those parameters, we  can  reduce the  biasof the model to a small value of 0.05 (the measurement for bias takes a value from 0 to 1) with only  0.6%accuracy drop  or increase the bias to a large value of 0.96 with  11% accuracy drop.
Revised: Surprisingly, we ﬁnd that scaling these parameters can either reduce the model’s bias to a small value of 0.05 (on a measurement scale of 0 to 1) with only a 0.6% drop in accuracy or increase the bias to a large value of 0.96 with an 11% drop in accuracy.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Federated Learning.
Revised: Federated Learning.

Original: We follow the conventional FL setting (McMahan et al., 2017).
Revised: In this paper, we consider the conventional federated learning (FL) setting. (McMahan et al., 2017).

Original: There is a networkconsisting of K parties. Each party k ∈ [ K ] holds a  dataset ˜ D k of size n k  sampled from  local  distribution D k .
Revised: The FL framework consists of a network of K parties, where each party k ∈ [ K ] holds a local dataset ˜ D k of size n k , sampled from a local data distribution D k .

Original: The goal of each party is to learn a model that minimizes the loss on  D k .
Revised: The objective of each party is to train a model that minimizes the loss on their local data distribution D k .

Original: Federated learning achieves this goal by training a global model to minimize the average loss across parties, i.e., min θ 1 N (cid:80)
Revised: To achieve this goal, FL trains a global model to minimize the average loss across parties, which is expressed as min θ 1 N (cid:80)

Original: Kk =1 n k L ( θ, ˜ D k ) , where N is the sum of all local training dataset size.
Revised: Kk =1 n k L ( θ, ˜ D k ) , where N is the sum of all local training dataset size.

Original: At each communication round t , a global server sends the current global model to all parties.
Revised: In each communication round t , a global server sends the current global model to all parties.

Original: All parties train the global model locally on their local dataset and send the updated local model to the server.
Revised: All parties train the global model locally on their local dataset and send the updated local model to the server.

Original: The server then aggregates those local models to obtain the new global model.
Revised: The server then aggregates those local models to obtain the new global model.

Original: Note that, we consider the case where all parties participate in the training in each round. It eliminates the potential bias caused by the  sampling of participating parties.
Revised: We consider the case where all parties participate in the training in each round, which helps to mitigate potential biases that could arise from the non-uniform sampling of participating parties.

Original: Group Fairness.
Revised: Group Fairness.

Original: Fairness has a wide variety of meanings in literature.
Revised: Fairness has a wide variety of meanings in literature.

Original: Group fairness entails, in particular, that the model should perform comparably across groups defined by sensitive attributes (e.g., sex).
Revised: Group fairness entails, in particular, that the model should perform comparably across groups deﬁned by sensitive attributes (e.g., sex).

Original: It is now common practice to evaluate discrimination in a model (or system) based on quantitative measurements of group fairness.
Revised: It is now common practice to evaluate discrimination in a model (or system) based on quantitative measurements of group fairness.

Original: In light of this, we focus on two widely-used group fairness notions, equalized odds (Hardt et al., 2016) and demographic parity (Dwork et al., 2012).
Revised: In light of this, we focus on two widely-used group fairness notions, equalized odds (Hardt et al., 2016) and demographic parity (Dwork et al., 2012).

Original: To formally define those fairness notions, we assume each data point is associated with a sensitive attribute a ∈ A , and we use Y to denote the true label.

Original: Definition 1 (Equalized Odds (Hardt et al., 2016)) .

Original: We say a model f θ is fair with respect to equalized odds if the prediction is conditionally independent of the sensitive attributeA given the true label Y , i.e., ∀ a, a′ ∈ A , y ∈ Y , Pr( f θ ( X ) = y
Revised: To formally deﬁne those fairness notions, we assume each data point is associated with a sensitive attribute a ∈ A , and we use X and Y to denote the input features and the true label.

Original: |

Original: A = a, Y = y ) =

Original: Pr( f θ ( X ) =

Original: y

Original: | A = a ′ , Y = y ) . To measure fairness, we use the fairness gap with respect to Equalized Odds Difference, defined as ∆ EO ( θ, D ) : 
Revised: To measure fairness, we use the fairness gap with respect to Equalized Odds Difference, deﬁned as ∆ EO ( θ, D ) : =

Original: [Equation]
Revised: [Equation]

Original: with an ideal value equal to zero.
Revised: with an ideal value equal to zero (perfectly fair).

Original: Furthermore, in many applications, there exists a favorable prediction from the model (e.g., grant the loan).
Revised: Furthermore, in many applications, there exists a favorable prediction from the model (e.g., grant the loan).

Original: We assume the positive prediction ( + ) as the favorable outcome.
Revised: We assume the positive prediction ( + ) as the favorable outcome.

Original: Demographic parity (Dwork et al., 2012) group fairness notion asks the model to give a favorable label to groups with equal rates.
Revised: Demographic parity (Dwork et al., 2012) group fairness notion asks the model to give a favorable label to groups with equal rates.

Original: Definition 2 (Demographic parity (Dwork et al., 2012)) . We say a model f θ is fair with respect todemographic parity if the positive prediction is independent of the sensitive attribute A , i.e., ∀ a, a ′ ∈ A , Pr( f θ ( X ) =
Revised: Similarly, the fairness gap with respect to

Original: + | A = a ) = Pr( f θ ( X ) = + | A = a ′ ) . To measure fairness, we use the fairness gap with respect to Demographic Parity Difference, defined as 
Revised: Demographic Parity is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original:  with an ideal value equal to zero.
Revised: ) | with an ideal value equal to zero.

Original: In the rest of the paper, we use the fairness gap ∆ (including ∆ EO and ∆ DP ) to measure the bias (fairness) of a model.
Revised: In the rest of the paper, we use the fairness gap ∆ (including ∆ EO and ∆ DP ) to measure the bias (fairness) of a model.

Original: The more significant fairness gap means the model is more biased (less fair).
Revised: The more signiﬁcant fairness gap means the model is more biased (less fair).

Revised: We will use bias or unfairness interchangeably.

Original: Measuring  impact of FL.
Revised: Measuring the impact of FL.

Original: Federated learning strives to improve model performance compared to standalone training, andthe ultimate goal is to achieve comparable performance to centralized training.
Revised: Federated learning aims to enhance model performance compared to standalone training and achieve comparable performance to centralized training.

Original: Thus, in order to quantify the impact of FL on local fairness, we use centralized training and standalone training as baselines.
Revised: Thus, in order to evaluate the impact of FL on local fairness, we use centralized training and standalone training as baselines.

Original: In standalone training, each party trains a model θ k independently to minimize the loss on its training data ˜ D k .
Revised: In standalone training, each party trains a model θ k independently to minimize the loss on its training data ˜ D k .

Original: Note that the standalone model’s fairness gap for a party is contributed by herself.
Revised: Note that the standalone model’s fairness gap for a party is contributed by herself.

Original: Thus, we use the fairness gap of a party’s standalone model to represent the party’s bias.
Revised: Therefore, we use the fairness gap of a party’s standalone model to represent the party’s bias.

Original: On the other hand, in centralized training, all local training datasets from all parties are combined, and a centralized model θ c is trained using this global training dataset.
Revised: In contrast, in centralized training, all local training datasets from all parties are combined, and a centralized model θ c is trained using this global training dataset.

Original: We define the benefit of FL and collaboration (i.e., centralized training) in terms of fairness and accuracy using these baselines.
Revised: We deﬁne the beneﬁt of FL and collaboration (i.e., centralized training) in terms of fairness and accuracy using these baselines.

Original: The benefit of FL for each party k is defined by the difference between the standalone model θ k and the FL model θ g .
Revised: The beneﬁt of FL for each party k is deﬁned as the difference between the standalone model θ k and the FL model θ g .

Original: It reveals the extent to which a party benefits from participating in FL.
Revised: This difference reveals the extent to which a party beneﬁts from participating in FL.

Original: More formally, for a party k , we define the accuracy benefit of FL as Acc ( θ g , D k ) − Acc ( θ k , D k ) and fairness benefit of FL as ∆( θ k , D k ) − ∆( θ g , D k ) .
Revised: Speciﬁcally, for a party k , we deﬁne the accuracy beneﬁt of FL as Acc ( θ g , D k ) − Acc ( θ k , D k ) and fairness beneﬁt of FL as ∆( θ k , D k ) − ∆( θ g , D k ) .

Original: Similarly, the benefit of collaboration is based on the difference between the standalone model and the centralized model.
Revised: Similarly, the beneﬁt of collaboration is based on the difference between the standalone model and the centralized model.

Original: It indicates how much an individual party could gain from collaborating with others by sharing the entire training dataset.
Revised: This difference indicates how much an individual party could gain from collaborating with others by sharing the entire training dataset.

Original: The average benefit of FL or collaborationis the average benefit across all parties in the network.
Revised: We compute the average beneﬁt of FL or collaboration across all parties in the network.

Original: The positive benefit indicates that FL (or collaboration) helps to improve accuracy or fairness, while the negative benefit implies that FL or collaboration hurts.
Revised: A positive beneﬁt indicates that FL (or collaboration) improves accuracy or fairness, while a negative beneﬁt implies that FL or collaboration has a detrimental effect.

Original: Measuring bias aside fairness gap.
Revised: Measuring bias aside fairness gap.

Original: Measuring the bias of a model based on the fairness gap reveals the model’s performance disparity across groups, but does not explains why the model is biased.
Revised: Measuring the bias of a model based on the fairness gap reveals the model’s performance disparity across groups but does not explain why the model is biased.

Original: Specifically, is the prediction of the model  attributed to the sensitive attribute or any other insensitive attributes whose distributions vary between groups?
Revised: Speciﬁcally, it does not reveal whether the prediction of the model is attributed to the sensitive attribute or any other insensitive attributes whose distributions vary between groups.

Original: The former is typically referred to as disparate treatment (direct discrimination) as the sensitive attribute directly influences the model prediction (Grabowicz et al., 2022; Zafar et al., 2017).
Revised: The former is typically referred to as disparate treatment (direct discrimination) as the sensitive attribute directly inﬂuences the model prediction (Grabowicz et al., 2022; Zafar et al., 2017).

Original: Towards measuring bias in this aspect, we employ the feature attribution method to explain the model’s prediction on each data point.
Revised: Towards measuring bias in this aspect, we employ the feature attribution method to explain the model’s prediction on each data point.

Original: More precisely, given a model f θ , and the input features x = ( x 1 , x 2 , ..., x d ) , the attribution of the prediction at input x relative to a baseline input x ′ is a vector ( a 1 , ..., a d ) where a i is the contribution of x i to the prediction of f θ ( x ) .
Revised: More precisely, given a model f θ , and the input features x = ( x 1 , x 2 , ..., x d ) , the attribution of the prediction at input x relative to a baseline input x (cid:48) is a vector ( a 1 , ..., a d ) where a i is the contribution of x i to the prediction of f θ ( x ) .

Original: We use Integrated Gradient (Sundararajan et al., 2017) to compute the feature attribution, which only requires a few calls to the gradient operation.
Revised: We use Integrated Gradient (Sundararajan et al., 2017) to compute the feature attribution, which only requires a few calls to the gradient operation.

Original: Following the suggestions in the paper, we use the average feature values computed on the whole test dataset as the baseline for non-sensitive and non-binary sensitive features.
Revised: Following the suggestions by Sundararajan et al. (2017), we use the average feature values computed on the whole test dataset as the baseline for non-sensitive and non-binary sensitive features.

Original: For the binary sensitive attribute, we use the opposite feature value as the baseline.
Revised: For the binary sensitive attribute, we use the opposite feature value as the baseline.

Original: For instance, if the test input is (Sex: female, Age:28), the baseline input would be (Sex: male, Age: average age over the data points).
Revised: For instance, if the test input is (Sex: female, Age:28), the baseline input would be (Sex: male, Age: average age over the data points).

Original: By employing this feature attribution, we are able to discover the bias associated with disparate treatment.
Revised: By employing this feature attribution, we are able to discover the bias associated with disparate treatment.

Original: For more details about the Integrated Gradient, please refer to the original paper (Sundararajan et al., 2017).
Revised: For more details about the Integrated Gradient, please refer to the original paper (Sundararajan et al., 2017).

Original:  E MPIRICAL A NALYSIS
Revised:  E MPIRICAL A NALYSIS

Original: Our objective is to comprehend how FL impacts fairness for parties.
Revised: Our objective is to understand how FL impacts fairness for parties.

Original: Toward this end, we start by  thecomparison of average performance between FL and baselines (i.e., standalone training and centralized training). In particular, we ask: does FL improve fairness for clients compared to standalone training, andfairness-wise, does FL provide the same benefit as centralized training?
Revised: We start by comparing the average performance between FL and baselines (i.e., standalone training and centralized training) to answer the following question: does FL improve fairness for parties compared to standalone training, and does FL provide the same beneﬁt as centralized training in terms of fairness? Our ﬁndings in

Original: We provide answers in Section 3.2, where we demonstrate that FL can exacerbatethe fairness issue for parties and does not retain the fairness benefit of collaboration.
Revised: Section 3.2 demonstrate that FL can exacerbate fairness issues for parties and does not retain the fairness beneﬁt of collaboration, thus not achieving comparable performance to centralized training.

Original: After that, we further investigate how FL impacts performance for each party.
Revised: We further investigate how FL impacts performance for each party to answer the following questions:

Original: Specifically, we aim to answer the following two questions: does FL provide the same benefit in terms of fairness for parties? What causes the disparate impact of FL on parties’ fairness?
Revised: does FL provide the same beneﬁt in terms of fairness for parties, and what causes the disparate impact of FL on parties’ fairness?

Original: We provide analysis in Section 3.3, where we show FL can propagatethe bias among parties.
Revised: Our analysis in Section 3.3 shows that FL can propagate bias among parties.

Original: As a consequence, FL improves fairness for parties with greater bias while worsening fairness issues for parties with less bias.
Revised: As a result, FL improves fairness for parties with greater bias while worsening fairness issues for parties with less bias.

Original: Finally, we investigate how the bias is propagated in FL .
Revised: Finally, we investigate how the bias is propagated in FL .

Original: The analysis in Section 3.4 shows that biased parties encode the bias in a few parameters through local update, which is propagated to the aggregated model and ultimately to other parties via parameter aggregation.
Revised: The analysis in Section 3.4 shows that biased parties encode the bias in a few parameters through local updates, which are propagated to the aggregated model and ultimately to other parties via parameter aggregation.

Original: 3. S ETUP
Revised: 3. S ETUP

Original: We use the two datasets  for our empirical analysis: US Census Data and CelebA (Liu et al., 2015) .
Revised: We use the two datasets with different tasks for our empirical analysis: US Census Data and CelebA (Liu et al., 2015) .

Original: The US Census dataset consists of census data from different places in the US.
Revised: The US Census dataset consists of census data from different places in the US.

Original: We naturally partition[Table] the dataset based on the source of the data.
Revised: We naturally partition the dataset based on the source of the data.

Original: Thus, we have 51 parties in total, representing 50 states in the US and Puerto Rico.
Revised: Thus, we haveparties in total, representing 50 states in the US and Puerto Rico.

Original: We consider three tasks defined in the folktables (Ding et al., 2021):
Revised: We consider three tasks deﬁned [Table]

Original:  Income (ACSIncome) 1 , Health (ACSPublicCoverage) and Employment (ACSEmployment).
Revised: in the folktables (Ding et al., 2021): Income (ACSIncome) 1 , Health (ACSPublicCoverage), and Employment (ACSEmployment).

Original: We train two-layer neural network models for all the tasks.
Revised: We train two-layer neural network models for all the tasks.

Original: We consider the binary notion of sex (i.e., male and female) and multi-value race as the sensitive attribute for every task.
Revised: We consider the binary notion of sex (i.e., male and female) and multi-value race as the sensitive attribute for every task.

Original: Each party has 1, training points and 2,000 test points.
Revised: Each party has 1,000 training points and 2,000 test points.

Original: Note that we use a large test dataset to ensure an accurate estimation of the fairness gap with respect to parities’ test distribution.
Revised: Note that we use a large test dataset to ensure an accurate estimation of the fairness gap with respect to parities’ test distribution.

Original: CelebA (Liu et al., 2015) is an image dataset consisted of 200,000 celebrity images. We focus on the age prediction task and consider the binary attribute “Male” (male and non-male) as the sensitive attribute, which wewill refer to as the Sex attribute.
Revised: For CelebA, an image dataset consisting of 200,000 celebrity images, we focus on the age prediction task and use the binary attribute "Male" (male and non-male) as the sensitive attribute, which we refer to as the Sex attribute.

Original: We partition the data uniformly at random among all parties and train CNN models.
Revised: We partition the data uniformly at random among all parties and train CNN models.

Original: We also consider the Non-IID partitioning on  CelebA dataset, and the results are presented in Appendix B.
Revised: We also consider non-IID partitioning on the CelebA dataset, and the results are presented in Appendix B.

Original: For allthe experiments, we report the average results over five runs with different random seeds.
Revised: In all experiments, we report the average results over ﬁve runs with different random seeds.

Original: We use FedML (He et al., 2020), alibrary for FL based on PyTorch Paszke et al. , to train models in FL.
Revised: We use FedML (He et al., 2020), a PyTorch (Paszke et al., 2019)-based library for FL, to train models in FL.

Original: We also utilize Captum (Kokhlikyan et al., 2020) for computing the feature attribution.
Revised: We also use Captum (Kokhlikyan et al., 2020) to compute feature attribution.

Original: More details about the datasets and models can be found in Appendix A.
Revised: For further details on the datasets and models, please refer to Appendix A.

Original: 3. C OLLABORATION VIA FL CAN WORSEN FAIRNESS ISSUE
Revised: 3. C OLLABORATION VIA FL CAN WORSEN FAIRNESS ISSUE

Original: Table 1 shows the average accuracy and fairness gaps of the centralized model, FL model, and standalone models on local datasets.
Revised: Table 1 presents the average accuracy and fairness gaps of the centralized model, FL model, and standalone models on local datasets.

Original: We observe that centralized training can improve accuracy and fairness at the same time for parties.
Revised: Centralized training is observed to improve accuracy and fairness at the same time for parties.

Original: For instance, on the Income dataset, centralized training improves the accuracy by 6% and reduces the fairness gap across racial groups by 5.5% with respect to equalized odds and by 5.5% with respect to demographic parity.
Revised: For instance, on the Income dataset, centralized training improves the accuracy by 6% and reduces the fairness gap across racial groups by 5.5% with respect to equalized odds and by 5.5% with respect to demographic parity.

Original: However,we notice that FL may not achieve the same benefit asthe centralized training in terms of fairness but can even worsen the fairness issue for parties.
Revised: However, FL may not achieve the same beneﬁt as centralized training in terms of fairness and can even exacerbate the fairness issue for parties.

Original: For instance, on the Income dataset, the EO fairness gap of the FL model for the sex groups increases by 35%, and the DP fairness gap increases by 29.9%, compared to standalone models.
Revised: For example, on the Income dataset, the EO fairness gap of the FL model for the sex groups increases by 35%, and the DP fairness gap increases by 29.9%, compared to standalone models.

Original: Table 2 in Appendix B shows the results for other popular FL algorithms, including FedNova (Wang et al., 2020), Scaffold Karimireddy et al.
Revised: We include results for other popular FL algorithms, including FedNova (Wang et al., 2020), Scaffold Karimireddy et al.

Original: (2020b), FedOpt (Reddi et al., 2020), FedProx Li et al.
Revised: (2020b), FedOpt (Reddi et al., 2020), FedProx Li et al.

Original: (2020), and Mime (Karimireddy et al., 2020a), on which we observe a similar pattern as on FedAvg algorithm.
Revised: (2020), and Mime (Karimireddy et al., 2020a), in Table 2 in Appendix B, which show a similar pattern to that of the FedAvg algorithm.

Original: We refer readers to Appendix B.5 for more detailed results on other FL algorithms.
Revised: Appendix B provides more detailed results on other FL algorithms.

Original: The centralized and FL models are trained on the same dataset.
Revised: The centralized and FL models are trained on the same dataset.

Original: The difference between the FL model and the centralized model in terms of fairness suggests FL algorithm can introduce more bias compared to standard training.
Revised: The difference between the FL model and the centralized model in terms of fairness suggests FL algorithm can introduce more bias compared to standard training.

Original: Consequently, the explanation of how bias is introducedto the model duringthe standard training in the centralized setting may not adequately explain how FL introduces bias to the model.
Revised: Therefore, the explanation of how bias is introduced during standard training in the centralized setting may not fully explain how FL introduces bias.

Original: In the following, we further explore how FL impacts fairness from the party level.
Revised: In the following, we further explore how FL impacts fairness from the party level.

Original: 3. FL PROPAGATES BIAS AMONG PARTIES
Revised: 3. FL PROPAGATES BIAS AMONG PARTIES

Original: Disparate impact of FL on group fairness over parties.
Revised: Disparate impact of FL on group fairness across parties.

Original: Figure 1 shows the benefit of FL on fairness and accuracy for parties.
Revised: Figure 1 illustrates the beneﬁt of FL on fairness and accuracy for parties.

Original:  FL improves accuracy for almost all parties, and theimprovement variance  across parties is small.
Revised: We observe that FL improves accuracy for almost all parties, and the variance in accuracy improvement across parties is small.

Original: On the other hand, the fairness benefit of FL is negative for most parties.
Revised: On the other hand, the fairness beneﬁt of FL is negative for most parties.

Original: It implies that most parties obtain a more biased model in FL compared to standalone training.
Revised: It implies that most parties obtain a more biased model in FL compared to standalone training.

Original: Moreover, we notice that the variance of the fairness benefits across parties is large. It shows that even though all parties have the same global model in FL, they are not benefiting from the  model equally (recall that each party tests the model  on her local test dataset).
Revised: Furthermore, we notice that the variance in the fairness beneﬁts across parties is large, suggesting that although all parties have the same global model in FL, they do not beneﬁt from the FL model equally (each party evaluates the model performance on their local test dataset).

Revised: To explore the impact of FL on fairness at the party level,

Original: Figure 2 shows strong correlations between the fairness benefit a party acquires in FL and the fairness gap of the standalone model for the party (i e.,  bias level of the party).
Revised: Figure 2 shows strong correlations between the fairness beneﬁt a party obtains in FL and the fairness gap of the standalone model for the party, i e., the bias level of the party.

Original: It further demonstrates the disparate impact of FL on fairness: FL can improve fairness for more biased parties but at the cost of worsening the issue for less biased parties.
Revised: This ﬁnding highlights the disparate impact of FL on fairness: FL can improve fairness for more biased parties but at the cost of worsening the issue for less biased parties.

Original: Aggregation contradicts with local update.
Revised: Contradiction between aggregation and local update.

Original: Figure 3 shows the dynamic of the fairness gap before and after the aggregation. We find that aggregation and local update contradicts each other during the training.
Revised: During the training process of FL, we observe that aggregation and local update contradict each other, shown in Figure 3.

Original: More specifically, the local update for the least biased party (whose standalone model has the lowest fairness gap) reduces the fairness gap of the model during the local update.
Revised: Local update from the least biased party, whose standalone model has the lowest fairness gap, reduces the fairness gap of the model.

Original: However, this reduction is eliminated by aggregation.
Revised: However, this reduction is eliminated by the aggregation step.

Original: On the other hand, the most biased parties increase the models’ bias through the local update, which will be reduced via aggregation.
Revised: Conversely, the local update from the most biased party increases the fairness gap of the model, which is then reduced by aggregation.

Original: It indicates that the aggregation contributes to the disparate impact of FL on fairness, improving fairness for more biased parties while worsening the fairnessissue for less biased parties compared to the standalone setting.
Revised: This ﬁnding implies that the aggregation contributes to the disparate impact of FL on fairness, improving fairness for more biased parties but worsening the fairness for less biased parties compared to the standalone setting.

Original: The y-axis represents the average influence of each party.

Original: The Pearson correlation between the fairness gap a party obtains in the standalone setting and the impact she has on local fairness for parties in

Original: FL are shown.

Original: The greater the party’s bias, the greater its adverse effect on others.

Original: Biased parties negatively influence other parties via aggregation throughout the training.
Revised: Biased parties negatively inﬂuence other parties via aggregation throughout the training

Original: Why does aggregation have disparate impact on local fairness for parties?
Revised: Why does aggregation have disparate impacts on local fairness for parties?

Original: Specifically, we ask which party’s local update causes the increase (or decrease) of the fairness gap for other parties viaaggregation.
Revised: Speciﬁcally, we ask which party’s local update causes the increase (or decrease) of the fairness gap for other parties via

Revised: The Pearson correlation coefﬁcient between the fairness gap a party obtains in the standalone setting and the impact she has on local fairness for parties in FL are shown in the ﬁgure aggregation.

Original: To answer this question we measure the influence of aparty on other parties’ local fairness by the influence of the party’s local update on the fairness of the aggregated model during the training.
Revised: To answer this question, we look into the inﬂuence of a party’s local update on other parties’ fairness via aggregation.

Original: More precisely, we compute the influence of party i on party j as the fairness gap increase when party i ’s local update is removed from the aggregation in each round t and sum over all the training rounds, which is formally defined asfollows: I i,j
Revised: More precisely, we compute the inﬂuence of party i on party j as the fairness gap increase when party i ’s local update is removed from the aggregation in each round t and sum over all the training rounds. Formally, we deﬁne the inﬂuence of party i on party j as

Original: [Equation] − i , D j ) − ∆( θ t , D j ) , where θ t is the global model (i.e., the aggregated model overall local updated models)in round t and θ t, − i is the aggregated model over local updated models from parties excluding party i .
Revised: I i,j = (cid:80) Tt =1 ∆( θ t, − i , D j ) − ∆( θ t , D j ) , where θ t is the global model (i.e., the aggregated model overall local updated models) and θ t, − i is the aggregated model over local updated models from parties excluding party i .

Original: If party i improves fairness for party j , the influence is positive and vice versa.
Revised: If party i improves fairness for party j , the inﬂuence is positive and vice versa.

Original: We compute the influence for all pairs of parties, and the most influential pairs are shown in Figure 4.
Revised: We compute the inﬂuence for all pairs of parties, and the most inﬂuential pairs are shown in Figure 4.

Original: We observe that a less biased party has  positive influence on fairness for other parties while a more biased party has  negative influence on other parties.
Revised: We observe that a less biased party has a positive inﬂuence on fairness for other parties, while a more biased party has a negative inﬂuence on other parties.

Revised: This result shows that a biased party can negatively inﬂuence other parties’ fairness via aggregation throughout the training.

Original: Furthermore, we show the correlation betweenthe bias a party has (i.e., the bias of the party’s standalone model) and its average influence on all parties’ fairness (i.e., (cid:80)
Revised: Furthermore, we investigate the relationship between a party’s bias (i.e., the bias of the party’s standalone model) and its average inﬂuence on all parties’ fairness (i.e., (cid:80)

Original: Kk =

Revised: Kk =

Original: I i,k /K ) in Figure 5.
Revised: I i,k /K ).

Revised: The results are presented in Figure 5, which shows a strong correlation between these two factors.

Original: The strong correlation between those two further supports  that the more biased parties have a stronger negative influence, while the less biased parties have a stronger positive influence on other parties.
Revised: This ﬁnding further supports our conclusion that the more biased parties have a stronger negative inﬂuence on other parties’ fairness, while the less biased parties have a stronger positive inﬂuence.

Original: Lastly, Figure 6 shows the dynamic of the influence of the top 5 most biased parties (i.e., parties who have the largest fairness gap in the standalone setting) and the top 5 least biased parties.
Revised: Finally, we analyze the dynamics of the inﬂuence of the top 5 most biased parties (i.e., parties with the largest fairness gap in the standalone setting) and the top 5 least biased parties, as shown in Figure 6.

Original: We notice that the influence is monotonically increasing for the most biased parties and decreasing for the least value for sex - Income. We show results for the parties who have the highest or lowest fairness gap in the standalone setting. The standalone training and FL gradually increase the model dependence on the sensitive attribute, while the centralized training reduces the dependence during the training.
Revised: We observe that the inﬂuence of the most biased parties is monotonically increasing throughout the training, while the inﬂuence of the least biased parties is decreasing.

Original: biased parties. In other words, biased(less biased) parties consistently have a negative (positive) influence on others’ fairness gaps.
Revised: In other words, biased parties consistently have a negative inﬂuence on others’ fairness gaps, while less biased parties have a positive inﬂuence.

Original: Our results imply that FL can propagate bias among parties: the bias from biased parties negatively influence the fairness of other parties via aggregation throughout the training.
Revised: These results suggest that FL can propagate bias among parties: the bias from biased parties negatively inﬂuence the fairness of other parties via aggregation throughout the training.

Original: Next, we will explore how the bias is propagated in FL.
Revised: Next, we will explore how the bias is propagated in FL.

Original: 3. H OWBIAS IS  PROPAGATED IN FL?
Revised: 3. H OW IS BIAS PROPAGATED IN FL?

Original: Large fairness gap is caused by disparate treatment.
Revised: Disparate treatment causes large fairness gaps.

Original: We first investigate what the bias represents. More specifically, is the bias(fairness gap) increase in FL  directly caused by the disparate treatment of the model among sensitive groups?
Revised: Our ﬁrst investigation explores what the bias represents, speciﬁcally whether the bias increase in FL is directly caused by the disparate treatment of the model among sensitive groups.

Original: To answer this question, we utilize Integrated Gradients (Sundararajan et al., 2017) to measure the attribution of each input feature to the  predictionsof models with respect to the positive class.
Revised: To answer this question, we utilize Integrated Gradients (Sundararajan et al., 2017) to measure the attribution of each input feature to the models’ predictions with respect to the positive class.

Original: Figure 7 shows thehistogram of the attribution value  for the sensitive attribute, "Sex", over individual test points from the femalegroup andthe male group.
Revised: Figure 7 shows the attribution value distribution for the sensitive attribute "Sex" over individual test points from the female and male groups.

Original: We notice that the sex attribute has a large attribution value for the standalone model’s predictions and  FL model’s predictions.
Revised: We notice that the sex attribute has a large attribution value for the standalone model’s predictions and the FL model’s predictions.

Original: It means that the prediction of those models are heavily dependent on the sensitive attribute of the test data.
Revised: This ﬁnding implies that the predictions of those models are heavily dependent on the sensitive attribute of the test data.

Original: Additionally, individuals’ sensitive attribute affects the model predictions in a different way. The average attribution value for the “Sex” attribute is positive for the male group and negative for the female group.
Revised: Moreover, the sensitive attribute affects the model predictions differently for the male and female groups, with the average attribution value being positive for the male group and negative for the female group.

Original: Moreover, we find that there is less difference in the attribution value for  the female and male groupswith respect to other attributes (see Figure 17 in Appendix B).
Revised: Furthermore, we ﬁnd that there is minimal difference in the attribution value for other attributes with respect to the female and male groups (see Figurein Appendix B).

Original: It implies that the large fairness gap for models is not caused by the distinct distribution of other insensitive attributes over protected groups; rather, it is mainly caused by the  disparate treatment ofthe model on protected groups.
Revised: This indicates that the large fairness gap in the models is not caused by the distinct distribution of other insensitive attributes over protected groups; rather, it is mainly caused by the models’ disparate treatment of protected groups.

Original: FL model learns more biased patterns.
Revised: FL model learns more biased patterns.

Original: Comparing the attribution value of “Sex” for different models in Figure 7, we find that, although the predictions of  FL model depend less heavily on  sensitive attribute compared to that of the standalone model, the dependence is still stronger than that of the centralized model.
Revised: In Figure 7, we compare the attribution value of “Sex” for different models and ﬁnd that, while the predictions of the FL model depend less heavily on the sensitive attribute compared to those of the standalone model, the dependence is still stronger than that of the centralized model.

Original: It suggests that themodel trained in FL  learns a more biased pattern compared to the model that couldotherwise be learned in the centralized setting.
Revised: This suggests that the FL model learns a more biased pattern compared to what could be learned in the centralized setting.

Original: Figure 8 shows the dynamic of the average absolute feature attribution for “Sex” during the training.
Revised: Figure 8 shows the dynamic of the average absolute feature attribution for “Sex” during the training.

Original: We find that, for the most biased party, standalone training increases the model’s dependence on the sensitive attribute throughout training, but centralized training decreases it.
Revised: We ﬁnd that standalone training increases the model’s dependence on the sensitive attribute throughout training for the most biased party, but centralized training decreases it.

Original: It suggests that collaboration through centralized training improves local fairness by guiding the model to learn less biased features.
Revised: This suggests that collaboration through centralized training improves local fairness by guiding the model to learn less biased features.

Original: In FL, however, the absolute attribution value barely changes after 100 rounds and remains significantly greater than that in the centralized setting.
Revised: In FL, however, the absolute attribution value barely changes after 100 rounds and remains signiﬁcantly greater than that in the centralized setting.

Original: It implies that  FL algorithm may introducethe bias to the final model by inhibiting the model from learning less biased features.
Revised: This implies that the FL algorithm may introduce bias to the ﬁnal model by inhibiting the model from learning less biased features.

Original: Biased parties increase the model dependence on the sensitive attribute.
Revised: Biased parties increase the model dependence on the sensitive attribute.

Original: Figure 9 shows the attribution value of the aggregated model and local updated model from the most biased party and least biased party.
Revised: Figure 9 shows the attribution value of the aggregated model and locally updated model from the most biased party and least biased party.

Original: We observe that the biased party increases the global model dependence on sensitive attribute during the local update, and this increase is persistent over the training, while the least biased party, on the other hand, reduces the dependence on the sensitive attribute, which is aligned with the trend of  fairness gap in Figure 3.
Revised: We observe that the biased party increases the global model dependence on sensitive attributes during the local update, and this increase persists throughout the training. In contrast, the least biased party reduces the dependence on the sensitive attribute, which is aligned with the trend of the fairness gap in Figure 3.

Revised: This suggests that the biased parties have a negative impact on the fairness of other parties by increasing the model’s dependence on the sensitive attribute.

Original: Bias is encoded in a few parameters.
Revised: Bias is encoded in a few parameters.

Original: Biased parties increase the model dependence on the sensitive attribute.
Revised: Biased parties increase the model dependence on the sensitive attribute.

Original: But how does this increase propagate to other parties in the network?
Revised: But how does this increase propagate to other parties in the network?

Original: Note that parties share the model parameters of the localupdated model with the server; thus, the bias should be encoded in the  parameters.
Revised: Since parties share the model parameters of the local model with the server, the bias is likely encoded in the model parameters.

Original: Thus, we examine which parameters are related to the bias of the model.
Revised: Therefore, we investigate which parameters are related to the model’s bias.

Original: Intuitively, the parameters used to extract sensitive attribute information impact the attribution value of the sensitive attribute to the model prediction.
Revised: Intuitively, the parameters used to extract sensitive attribute information impact the attribution value of the sensitive attribute to the model prediction.

Original: If the absolute value (signal) of those parameters is substantial, the value of the sensitive attribute will have a significant effect on the model’s prediction.
Revised: If the absolute value (signal) of those parameters is substantial, the value of the sensitive attribute will signiﬁcantly affect the model’s prediction.

Original: In our evaluation,  sensitive attribute is part of the input feature. Hence, the parameters directly applied to the “Sex” attribute in the first layer of the neural network should contribute to the bias of the model.
Revised: In our evaluation, the sensitive attribute is part of the input feature, so the parameters directly applied to the “Sex” attribute in the ﬁrst layer of the neural network should contribute to the model’s bias.

Original: Figure 10(a) depicts the normalized norm for those parameters (the norm of parameters associated with the sensitive attribute divided by the parameter norm of the first layer) of aggregated models and local updated models for both the most and least biased parties.
Revised: Figure 10(a) shows the normalized norm of the parameters associated with the sensitive attribute for the most and least biased parties in the aggregated and locally updated models. The normalized norm is deﬁned as the norm of parameters associated with the sensitive attribute divided by the parameter norm of the ﬁrst layer.

Original: We find that the least biased party reduces the parameter norm, hence decreasing the model’s sensitivity to the sensitive attribute.
Revised: We observe that the least biased party reduces the parameter norm, hence decreasing the model’s sensitivity to the sensitive attribute.

Original: In contrast, the biased party raises norm for those parameters,hence amplifying the impact of the sensitive attribute on the model’s prediction.
Revised: On the other hand, the biased party increases the norm for those parameters, amplifying the impact of the sensitive attribute on the model’s prediction.

Original: Through aggregation, this amplification will be propagated to the global model.
Revised: Through aggregation, this ampliﬁcation will be propagated to the global model.

Original: Figure 10(b) reveals a moderate correlation between the fairness gap party experiences in a standalone setting and the normalized parameter norm for the parameters used to extract sensitive attribute information.
Revised: Figure 10(b) reveals a moderate correlation between the fairness gap party experiences in the standalone setting and the normalized parameter norm for the parameters used to extract sensitive attribute information.

Original: It also demonstrates that biased parties increase the parameter value associated with sensitive attributes during the local update, thereby boosting the model’s susceptibility to the sensitive attribute.
Revised: This implies that biased parties increase the parameter value associated with sensitive attributes during the local update, thereby boosting the model’s susceptibility to the sensitive attribute.

Original: Controlling fairness gap by scaling a few parameters.
Revised: Controlling fairness gap by scaling a few parameters.

Original: To further validate the effect of these parameters on model fairness, Figure 10(c) demonstrates that by scaling a few parameters associated with the sensitive attribute (i.e., 104  out of 1,792), the fairness gap can be greatly widened or narrowed at the expense of a moderate degree of accuracy.
Revised: To further investigate the impact of the parameters associated with the sensitive attribute (i.e., 104 parameters out of 1,792) on model fairness, we examine the effect of scaling these parameters on the fairness gap in Figure 10(c). We ﬁnd that the fairness gap can be greatly widened or narrowed at the expense of a moderate degree of accuracy.

Original: Specifically, by scaling the parameter value by 0. for the trained FL model, we can reduce the EO gap by 74.6% (from 0.198 to 0.05) and the DP gap by 69.1% (from 0.217 to 0.067) with just a 0.8% accuracy loss  (from 0.785 to 0.779).
Revised: Speciﬁcally, by scaling the parameter value by 0.1 for the trained FL model, we signiﬁcantly reduce the EO gap by 74.6% (from 0.198 to 0.05) and the DP gap by 69.1% (from 0.217 to 0.067) with just a moderate accuracy loss of 0.8% (from 0.785 to 0.779).

Original: In contrast, by raising the parameter value by a factor of 10, the EO gapincreases to 0.96 (the maximal fairness gap is 1), which is almost five times large. The DP gap increases to 0.78, which is increased by 259%. At the same time, the accuracydecreases from 0.785 to 0.67.
Revised: In contrast, scaling the same set of parameters by a factor of 10 increases the EO gap to 0.96 (the maximal fairness gap is 1), which is almost ﬁve times larger, and increases the DP gap by 259% to 0.78, while reducing the accuracy from 0.785 to 0.67.

Original: These findings explain how bias propagates in FL:the biased parties magnify the impact of sensitive attributes on model predictions by increasing the model parameter used to extract sensitive attributes.
Revised: These ﬁndings explain how bias is propagated in FL: biased parties magnify the impact of sensitive attributes on model predictions by increasing the model parameter used to extract sensitive attributes.

Original: This rise in parameters is subsequently propagated to the global model through aggregation, further aggravating the issue of fairness for other parties.
Revised: This rise in parameters is subsequently propagated to the global model through aggregation, further aggravating the issue of fairness for other parties.

Original: Our results explain how bias is propagated in FL: Biased parties encode bias in a few parameters through a local update, and this bias is consequently propagated to the entire network through parameter aggregation.
Revised: Our results explain how bias is propagated in FL: Biased parties encode bias in a few parameters through a local update, and this bias is consequently propagated to the entire network through parameter aggregation.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Fairness has received considerable attention due to the growing deploying of machine learning in decision-making processes.
Revised: Fairness has received considerable attention due to the growing deployment of machine learning in decision-making processes.

Original: Various definitions of fairness have been presented (Hardt et al., 2016; Dwork et al., 2012; Calders et al., 2009).
Revised: Various deﬁnitions of fairness have been presented (Hardt et al., 2016; Dwork et al., 2012; Calders et al., 2009).

Original: Specifically, group fairness requires that the model behave similarly for groups defined by a sensitive attribute (e.g., race).
Revised: Speciﬁcally, group fairness requires that the model behave similarly for groups deﬁned by a sensitive attribute (e.g., race).

Original: Although how machine learning algorithms propagate data bias to the final model has been extensively investigated in a centralized setting (Blum and Stangl, 2020; Lakkaraju et al., 2017; Rambachan and Roth, 2020; Friedler et al., 2019; Dullerud et al., 2022), the effect of FL on model fairness remains unclear.
Revised: While how machine learning algorithms propagate data bias to the ﬁnal model has been extensively investigated in a centralized setting (Blum and Stangl, 2020; Lakkaraju et al., 2017; Rambachan and Roth, 2020; Friedler et al., 2019; Dullerud et al., 2022), the effect of FL on model fairness is not yet fully understood.

Original: In FL setting, a substantial body of literature on fairness  focuses on the performance disparity ofthe FL model across parties rather than demographic groups (Li et al., 2021; Zhao and Joshi, 2022;
Revised: The existing literature on fairness in FL mainly focuses on the performance disparity of FL models across parties, rather than demographic groups (Li et al., 2021; Zhao and Joshi, 2022;

Original: Li et al., 2019; Mohri et al., 2019; Deng et al., 2020; Donahue and Kleinberg, 2021;
Revised: Li et al., 2019; Mohri et al., 2019; Deng et al., 2020; Donahue and Kleinberg, 2021;

Original: Hao et al., 2021; Zhou et al., 2021; Yu et al., 2020; Lyu et al., 2020).
Revised: Hao et al., 2021; Zhou et al., 2021; Yu et al., 2020; Lyu et al., 2020).

Original: We focus on group fairness, which concerns performance disparity among groups.
Revised: However, we focus on group fairness, which concerns performance disparity among groups.

Original: Thus, our focus differs from those studies.

Original: In terms of group fairness, Abay et al.
Revised: In terms of group fairness, Abay et al.

Original: (2020) listed a few potential sources of bias in FL.
Revised: (2020) listed a few potential sources of bias in FL.

Original: Recently, considerable progress has been made in training group fair models in FL (Abay et al., 2020;
Revised: Recently, considerable progress has been made in training group fair models in FL (Abay et al., 2020;

Original: Chu et al., 2021; Zeng et al., 2021a;
Revised: Chu et al., 2021; Zeng et al., 2021a;

Original: Hu et al., 2022; Du et al., 2021; Ezzeldin et al., 2021).
Revised: Hu et al., 2022; Du et al., 2021; Ezzeldin et al., 2021).

Original: However, the majority of these works suggest techniques for achieving fairness on a single test distribution.
Revised: Nonetheless, the majority of these works suggest techniques for achieving fairness on a single test distribution.

Original: Instead, we focus on fairness issues for parties.
Revised: Instead, we focus on fairness issues for parties.

Original: Cui et al. ; Papadaki et al. (2022) proposed algorithms to improve local fairness for parties.
Revised: Some studies (Cui et al., 2021; Papadaki et al., 2022) proposed algorithms to improve local fairness for parties.

Original: Our purpose, instead, is to gain a comprehensive understanding of how FL influences local fairness on its own, which we believe is equally crucial as designing fair algorithms.
Revised: Our purpose, instead, is to gain a comprehensive understanding of how FL inﬂuences local fairness on its own, which we believe is equally crucial as designing fair algorithms.

Original:  F UTURE W ORK & C ONCLUSION
Revised:  F UTURE W ORK & C ONCLUSION

Original: Future Work.
Revised: Future Work.

Original: In this work, we have explained how bias is propagated in FL when the sensitive attribute is included as an input feature.
Revised: In this work, we have investigated how bias is propagated in FL when the sensitive attribute is included as an input feature.

Original: In practice, however, sensitive attributes may be prohibited from being included in input features.
Revised: In practice, however, sensitive attributes may be prohibited from being included in input features.

Original: In such situations, the model may still be heavily biased based on variables that are correlated with the (unobserved) sensitive attribute.
Revised: In such situations, the model may still be heavily biased due to variables that are correlated with the (unobserved) sensitive attribute.

Original: For instance, a person’s zip code may be highly correlated with their race, a phenomenon known as "redlining".
Revised: For instance, a person’s zip code may be highly correlated with their race, a phenomenon known as "redlining".

Original: To discern which model parameters contribute to the bias and to audit the bias propagation in this scenario is a promising future work direction.
Revised: A promising direction for future work is to identify which model parameters contribute to the bias and to audit the bias propagation in this setting.

Original: Furthermore, designing FL algorithms  robust to bias propagation is a promising direction.
Revised: Another important direction is to design FL algorithms that are robust to bias propagation.

Original: We briefly discuss a few potential ways in Appendix B.9.
Revised: In Appendix E, we brieﬂy discuss a few potential ways to achieve this goal.

Original: Conclusion.
Revised: Conclusion.

Original: Federated learning has been widely deployed in various applications, which have significant individual-level consequences. Thus, itis essential to anticipate the possible bias introduced by FL.
Revised: Federated learning has become increasingly popular in various applications with signiﬁcant individual-level consequences, making it essential to anticipate the possible bias introduced by FL.

Original: Our paper makes the first step in this direction, offering a comprehensive analysis of the impact of FL(i.e., FedAvg) on local fairness for parties.
Revised: Our paper takes the ﬁrst step in this direction by providing a comprehensive analysis of the impact of FL on local fairness for parties.

Original: We demonstrated that the FL algorithm could introduce bias on its own. Consequently, FL may exacerbate the issue of fairness for the involved parties.
Revised: We demonstrated that the FL algorithm could introduce bias on its own which may exacerbate the issue of fairness for the involved parties.

Original: In addition, we demonstrated that this exacerbation is not equally distributed among parties. This is due to the fact that FL can propagate bias among parties.
Revised: Moreover, we showed that this exacerbation is not evenly distributed among parties, as FL can propagate bias among them.

Original: Finally, we provided potential explanations about how bias is propagated in FL. Specifically, biased parties encode their bias into the local updates bysteadily increasing the signal of a few parameters  throughout the training process.
Revised: Finally, we explained how bias is propagated in FL: biased parties encode their bias into the local updates by increasing the signal of a few parameters steadily throughout the training process, which is then propagated to the global model via aggregation and, ultimately, to other parties.  A CKNOWLEDGEMENT

Revised: The authors would like to thank Ergute Bao, Ta Duy Nguyen, and Martin Strobel for their valuable feedback on earlier versions of this paper, as well as the anonymous reviewers for their insightful comments.

Original: Such bias is then propagated to other parties via aggregation andultimately, to the global model.
Revised: This research is supported by Google PDPO faculty research award, Intel within the www.private-ai.org center, Meta faculty research award, the NUS Early Career Research Award (NUS ECRA award num- ber NUS ECRA FY19 P16), and the National Research Foundation, Singapore under its Strategic Capability Research Centres Funding Initiative. Any opinions, ﬁndings, conclusions, or recommendations expressed in this material are those of the author(s) and do not reﬂect the views of the National Research Foundation, Singapore.

Original:  E THICS S TATEMENT
Revised:  E THICS S TATEMENT

Original: Our analysis focuses on group fairness, which is typically used to audit the model or system for any bias or discrimination.
Revised: Our analysis focuses on group fairness, which is typically used to audit the model or system for any bias or discrimination.

Original: Auditing the bias with respect to other definitions of fairness, such as individual fairness, may result in different conclusions.
Revised: Auditing the bias with respect to other deﬁnitions of fairness, such as individual fairness, may result in different conclusions.

Original: Moreover, our study focuses primarily on the binary notion of sex attributes and the multi-valued race attribute.
Revised: Moreover, our study focuses primarily on the binary notion of sex attributes and the multi-valued race attribute.

Original: We recognize that there are numerous protected groups outside those considered in the analysis, such as those defined by multiple sensitive attributes.
Revised: We recognize that there are numerous protected groups outside those considered in the analysis, such as those deﬁned by multiple sensitive attributes.

Original: The propagation of bias against fine-grained subgroups maybe even more substantial than we found in the paper.
Revised: The propagation of bias against ﬁne-grained subgroups may be even more substantial than we found in the paper.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: We provide the codeand script fordownloading the data in the supplemental materials.
Revised: The code for the paper is available at https://github.com/privacytrustlab/bias_ in_FL . A PPENDIX

Original: More details can be found in the “REAME.md” file in the supplemental materials.
Revised: This appendix is divided into ﬁve sections. Appendix A provides additional details about the experimental setups, including information about the models, datasets, and hyperparameters used. Appendix B presents further experimental results that support the claims made in the paper. In

Original: In addition, we provide details about the model, datasets, and implementations in Appendix A.
Revised: Appendix C, we discuss the results of an existing fair FL algorithm. Appendix D extends our analysis to real-world medical datasets. Finally, in Appendix E, we discuss potential methods for mitigating bias in FL.

Original: AA DDITIONAL D ETAILS ABOUT E XPERIMENTS
Revised: A D ETAILS ABOUT E XPERIMENT S ETUP A. D ATASETS AND M ODELS

Original: Datasets and Models We explain the datasets and models used in the paper.
Revised: We explain the datasets and models used in the paper

Original: Census Dataset
Revised: Census Dataset

Original: We use the datasets provided by folktables.
Revised: We use the datasets provided by folktables.

Original: In particular, we consider the ACSIncome, ACSPublicCoverage, and ACSEmployment tasks defined in the forlktables.
Revised: In particular, we consider the ACSIncome, ACSPublicCoverage, and ACSEmployment tasks deﬁned in the forlktables.

Original: In the ACSIncome, the goal is to predict whether an individual’s Income is abovethan $50,000.
Revised: In the ACSIncome, the goal is to predict whether an individual’s Income is above $50,000.

Original: In  ACSEmployment task, the goal is to predict whether an individual is employed.
Revised: In the ACSEmployment task, the goal is to predict whether an individual is employed.

Original: Similarly, in the Health (i.e., ACSPublicCoverage) task, the objective is to predict whether an individual is covered by public health insurance.
Revised: Similarly, in the Health (i.e., ACSPublicCoverage) task, the objective is to predict whether an individual is covered by public health insurance.

Original: We use the same pre-processing as in the folktables and train a fully connected neural network model with one hidden layer of 32 neurons for Income and 64 neurons for Employment and Health tasks.
Revised: We use the same pre-processing as in the folktables and train a fully connected neural network model with one hidden layer of 32 neurons for Income and 64 neurons for Employment and Health tasks.

Original: For all the tasks, we use the RELU activation function.
Revised: For all the tasks, we use the RELU activation function.

Original: We use an SGD optimizer with a learning rate of 0.001 for centralized training on Health and Employment datasets and 0.1 for other settings, and the batch size is 32.
Revised: We use an SGD optimizer with a learning rate of 0.001 for centralized training on Health and Employment datasets and 0.1 for other settings, and the batch size is 32.

Original: We train the NN models for 200 epochs.
Revised: We train the NN models for 200 epochs.

Original: In FL, each client updates the global mdoel for 1 epoch and shares it with the server.
Revised: In FL, each client updates the global mdoel for 1 epoch and shares it with the server.

Original: We encode the categorical features based on the encoding template provided in folktables.
Revised: We encode the categorical features based on the encoding template provided in folktables.

Original: After the encoding, the input feature size for Income is 54, 154 for Health, and 109 for Employment.
Revised: After the encoding, the input feature size for Income is 54, 154 for Health, and 109 for Employment.

Original: We consider sex and race as sensitive attributes.
Revised: We consider sex and race as sensitive attributes.

Original: Accordingly, there are two gender groups (male and female) and nine racial groups ("White alone," "Black or African American alone," "American Indian alone," "Alaska Native alone," and "American
Revised: Accordingly, there are two gender groups (male and female) and nine racial groups ("White alone," "Black or African American alone," "American Indian alone," "Alaska Native alone," and "American

Original: Indian and Alaska Native tribes specified; or American Indian or Alaska Native, not specified and no other," "Asian alone," "Native Hawaiian and Other Pacific Islander alone," "Some Other Race alone," "Two or More Races").
Revised: Indian and Alaska Native tribes speciﬁed; or American Indian or Alaska Native, not speciﬁed and no other," "Asian alone," "Native Hawaiian and Other Paciﬁc Islander alone," "Some Other Race alone," "Two or More Races").

Original: CelebA Dataset We train CNN models on the dataset with one CNN layer whose output channel is 32, kernel size is 3, and stride is 1.
Revised: CelebA Dataset We train CNN models on the dataset with one CNN layer whose output channel is 32, kernel size is 3, and stride is 1.

Original: We use ’same’ padding for the CNN layer.
Revised: We use ’same’ padding for the CNN layer.

Original: Following this
Revised: Following this

Original: CNN layer, we have the Batch normalization layer and Max Pooling layer.
Revised: CNN layer, we have the Batch normalization layer and Max Pooling layer.

Original: After which, we have the connected layer.
Revised: After which, we have the connected layer.

Original: We train the model for 500 communication rounds or epochs with SGD optimizer.
Revised: We train the model for 500 communication rounds or epochs with SGD optimizer.

Original: The learning rate is 0.1, and the batch size is 128.
Revised: The learning rate is 0.1, and the batch size is 128.

Original: The train, test, and validation datasets ratio for each party is 6:2:2.
Revised: The train, test, and validation datasets ratio for each party is 6:2:2.

Original: Hyper-parameter for FL algorithms
Revised: A. H YPER - PARAMETER FOR FL ALGORITHMS AND I MPLEMENTATION

Original: In our paper, we also evaluate various popular FL algorithms, including FedNova (Wang et al., 2020), Scaffold Karimireddy et al.
Revised: In our paper, we also evaluate various popular FL algorithms, including FedNova (Wang et al., 2020), Scaffold Karimireddy et al.

Original: (2020b), FedOpt (Reddi et al., 2020), FedProx Li et al.
Revised: (2020b), FedOpt (Reddi et al., 2020), FedProx Li et al.

Original: (2020), and Mime (Karimireddy et al., 2020a).
Revised: (2020), and Mime (Karimireddy et al., 2020a).

Original: We use the same local learning rate and the number of the local epoch as in FedAvg.
Revised: We use the same local learning rate and the number of the local epoch as in FedAvg.

Original: We provided the detailed hyper-parameters for each of the algorithms in our code (See supplementary).
Revised: We provided the detailed hyper-parameters for each of the algorithms in our code (See supplementary).

Original: Implementation We run all experiments on Ubuntu with two NVIDIA TITAN RTX GPUs. 
Revised: We run all experiments on Ubuntu with two NVIDIA TITAN RTX GPUs. A. D ATA H ETEROGENEITY

Revised: Figure 11 shows the fraction of samples for each subgroup across all parties, while Figure 12 shows the histogram of subgroup proportions for each party.

Revised: We observe that the parties have different fractions of samples from each group in the Income dataset, indicating that their local data distributions are dissimilar.

Revised: In contrast, for the Health and Employment datasets, parties have similar fractions of samples from each subgroup, suggesting that their data distributions are more alike than those of the

Revised: Income dataset.

Original: B A DDITIONAL E XPERIMENTAL R ESULTS B. B IAS PROPAGATION EFFECT
Revised: B A DDITIONAL E XPERIMENTAL R ESULTS B. B IAS PROPAGATION EFFECT

Original: Census dataset We show the bias propagation effect of FL for  Health and Employment task in
Revised: Census dataset We show the bias propagation effect of FL for the Health and Employment task in

Original: Figure 11 and 12 respectively.
Revised: Figure 13 and 14 respectively.

Original: CelebA We show the bias propagation effect of FL on CelebA dataset.
Revised: CelebA We show the bias propagation effect of FL on CelebA dataset.

Original: Besides partitioning the data in an IID manner (we refer to as setting (i)), we also evaluate two different settings, where we change the number of samples from the minority subgroup (the female group with the “Not Young” age label).
Revised: Besides partitioning the data in an IID manner (we refer to as setting (i)), we also evaluate two different settings, where we change the number of samples from the minority subgroup (the female group with the “Not Young” age label).

Original: In this way, we aim to change the data bias in the local training datasets for clients.
Revised: In this way, we aim to change the data bias in the local training datasets for clients.

Original: In particular, in setting (ii), half of the parties have more samples from the minority subgroup than another half of the parties.
Revised: In particular, in setting (ii), half of the parties have more samples from the minority subgroup than another half of the parties.

Original: The ratio is 8:2.
Revised: The ratio is 8:2.

Original: In setting (iii), a single party has half of the data from the minority subgroup, and the data is then iid partitioned among the other parties.
Revised: In setting (iii), a single party has half of the data from the minority subgroup, and the data is then iid partitioned among the other parties.

Original: In the figure, we find that the in the non-iid setting, the more biased parties have a larger and more positive benefit, while FL hurts the less biased parties.
Revised: In the ﬁgure, we ﬁnd that the in the non-iid setting, the more biased parties have a larger and more positive beneﬁt, while FL hurts the less biased parties.

Original: Figure 13 shows the correlation between the fairness gap of the standalone model and the benefit a party gets in the FL.
Revised: Figure 15 shows the correlation between the fairness gap of the standalone model and the beneﬁt a party gets in the FL.

Original: We find that the in the non-iid setting, the more biased parties have a larger and more positive benefit, while FL hurts the less biased parties.
Revised: We ﬁnd that the in the non-iid setting, the more biased parties have a larger and more positive beneﬁt, while FL hurts the less biased parties.

Original: B. A GGREGATION CONTRADICTS WITH LOCAL UPDATE
Revised: B. A GGREGATION CONTRADICTS WITH LOCAL UPDATE

Original: Figure 14 shows the dynamic of the fairness gap for the aggregated model and local updated model from the most biased party and least biased party during the training
Revised: Figure 16 shows the dynamic of the fairness gap for the aggregated model and locally updated model from the most biased party and least biased party during the training.

Original: . B. I NFLUENCE S UBGRAPHS
Revised: B. I NFLUENCE S UB - GRAPHS

Original: We show the top 10 maximal positive and top 10 maximal native influence client pairs in Figureand  top 15 in Figure 16.
Revised: We show the top 10 maximal positive and top 10 maximal native inﬂuence client pairs in Figureand the top 15 in Figure 18.

Original: We can see that the less biased client has a strong positive influence on other clients while the more biased client has a strong negative influence on other clients
Revised: We can see that the less biased client has a strong positive inﬂuence on other clients while the more biased client has a strong negative inﬂuence on other clients

Original: . B. A TTRIBUTION VALUE FOR SENSITIVE ATTRIBUTE
Revised: . B. A TTRIBUTION VALUE FOR SENSITIVE ATTRIBUTE

Original: We show the attribution value for all input features for the most biased party and least biased party in Figure 17 and Figure 18, respectively.
Revised: We show the attribution value for all input features for the most biased party and least biased party in Figure 19 and Figure 20, respectively.

Original: We observe that the attribution value for other features is similar for female and male groups.
Revised: We observe that the attribution value for other features is similar for female and male groups.

Original: However, the sex attribute has the largest attribution valueAggregation and local updates have contradictory impacts on model fairness for those parties. and affects the groups differently.
Revised: However, the sex attribute has the largest attribution value and affects the groups differently.

Original: It implies that the models are highly dependent on the sensitive attribute (i.e., "Sex") for making predictions.
Revised: It implies that the models are highly dependent on the sensitive attribute (i.e., "Sex") for making predictions.

Original: B. E FFECT OF FL ALGORITHM
Revised: B. E FFECT OF FL ALGORITHM

Original: Table 2 shows results on other FL algorithms, including FedNova (Wang et al., 2020),
Revised: Table 2 shows results on other FL algorithms, including FedNova (Wang et al., 2020),

Original: Scaffold Karimireddy et al.
Revised: Scaffold Karimireddy et al.

Original: (2020b), FedOpt (Asad et al., 2020), FedProx Li et al.
Revised: (2020b), FedOpt (Asad et al., 2020), FedProx Li et al.

Original: (2020), and Mime (Karimireddy et al., 2020a).
Revised: (2020), and Mime (Karimireddy et al., 2020a).

Original: We find that when the FL model achieves higher accuracy than standalone models, the fairness gap of the FL model can be higher than that of standalone models.
Revised: We ﬁnd that when the FL model achieves higher accuracy than standalone models, the fairness gap of the FL model can be higher than that of standalone models.

Original: This observation is consistent across multiple FL algorithms.
Revised: This observation is consistent across multiple FL algorithms.

Original: This strong evidence implies that the improvement of accuracy in FL can come at the cost of fairness.
Revised: This strong evidence implies that the improvement of accuracy in FL can come at the cost of fairness.

Original: In addition, this is not unique to only the FedAvg algorithm.
Revised: In addition, this is not unique to only the FedAvg algorithm.

Original: B. G ROUP P ERFORMANCE
Revised: B. G ROUP P ERFORMANCE

Original: In Figure  we present the model ROC on groups to illustrate the performance disparity across groups.
Revised: In Figure 21, we present the model ROC on groups to illustrate the performance disparity across groups.

Original: Surprisingly, there is no huge difference between the ROC between groups.
Revised: Surprisingly, there is no huge difference between the ROC between groups.

Original: In contrast,
Revised: In contrast,

Original: [Table] Figure 20 shows the noticeable difference between groups with respect to the precision-recall curve, especially for the most biased party.
Revised: Figure 22 shows the noticeable difference between groups with respect to the precision-recall curve, especially for the most biased party.

Original: The main reason is that the dataset is imbalanced.
Revised: The main reason is that the dataset is imbalanced.

Original: Thus, ROC is usually misleading.
Revised: Thus, ROC is usually misleading.

Original: On the precision-recall curve, we find that the model achieves a higher precision on the majority (i.e., Male group) compared to the minority group (i.e., Female group).
Revised: On the precision-recall curve, we ﬁnd that the model achieves a higher precision on the majority (i.e., Male group) compared to the minority group (i.e., Female group).

Original: B. D ATA H ETEROGENEITY

Original: Figure 21 shows the fraction of samples of each subgroup for all parties, and Figure 22 shows the histogram of the proportion of samples from each subgroup among parties. We observe that parties have different fractions of samples from each group on Income dataset, indicating that their local data distribution differs. While on the Health and Employment dataset, parties have a similar fraction of the samples from subgroups. It implies that the data distribution for parties would be more similar to each other than that on the Income dataset.
Revised: [Table]

Original: B.E FFECT OF SCALING OTHER PARAMETERS
Revised: B.

Revised: E FFECT OF SCALING OTHER PARAMETERS

Original: Figure 23 compares the performance of the model when scaling different model parameters.
Revised: Figure 23 shows a comparison of the model’s performance when scaling different model parameters.

Original: In particular, we scale the parameters for the first layer of the neural network that does not have any computation on the Sex attribute (referred to as "Other parameters"). In addition, we also show the model performance when scaling the parameterthat is related to the Sex attribute (referred to as "Related parameters").
Revised: Speciﬁcally, we examine the impact of scaling the parameters in the ﬁrst layer of the neural network that does not have any computation on the Sex attribute (referred to as "Other parameters"), as well as the parameter related to the Sex attribute (referred to as "Related parameters").

Original: Clearly, we see that, by up-scaling or down-scaling other parameters does not  fairness gapsignificantly while scaling the related parameters has a large impact on model fairness.
Revised: The results indicate that up-scaling or down-scaling other parameters do not signiﬁcantly affect the fairness gap while scaling the related parameters has a considerable impact on model fairness.

Original: This validate our hypothesis that the  biasof the model is mainly encoded in a few model parameters.
Revised: This ﬁnding supports our hypothesis that the model’s bias is primarily encoded in a few model parameters.

Original: In FL,the biased party introduces the bias duringthe local training by increasing the weights for those related parameters.
Revised: In FL, biased parties introduce bias during local training by increasing the weights for those related parameters.

Original: B. P OTENTIAL M ITIGATION

Original: In this section, we offer a few potential solutions for reducing bias propagation effect of FL.

Original: We hope it will inspire more study on developing fair FL in the future.

Original: Personalized FL We showed in the paper that the aggregate update (average local update from all parties) could be more biased than the local update for less biased parties (shown in Figure 3).

Original: As a result, when less biased parties overwrite the local model with the aggregate update, the bias from other parties will be propagated to the local model, increasing the local fairness gap.

Original: Thus, one way to reduce the bias propagation effect is to avoid overwriting the local model completely using the global model.

Original: Recent personalized FL algorithms (Li et al., 2021) have been proposed based on a similar idea but with a focus on improving the model’s accuracy.

Original: Fairness-aware personalized federated learning could be a promising direction to reduce the bias propagation effect of FL.

Original: Fair Representation Learning We demonstrated that bias is encoded in a few parameters of the first layer of the neural network, which is usually considered the feature extractor (shown in

Original: Figure 10).

Original: One interpretation of the result is that the FL model is biased because the representation learned by the model is biased (highly dependent on the sensitive attribute).

Original: Fair representation learning (Zemel et al., 2013; Zhao et al., 2019; Liu et al., 2022; Zeng et al., 2021b) aims to learn a fair feature extractor such that the representation has limited (no) information about the sensitive attribute while still containing sufficient information about the task (or the input features).

Original: Thus, one potential way to reduce the bias propagation effect is to learn fair representation in FL.

Original: [Table]
Revised: [Table]

Original:  E VALUATION ON EXISTING F AIR FL
Revised: C E VALUATION ON EXISTING F AIR FL

Original: Table 3 shows the results for the fair FL algorithm proposed by Abay et al.
Revised: Table 3 presents the results for a fair FL algorithm proposed by Abay et al.

Original: (2020), which is based on the reweighting algorithm proposed in the centralized setting (Kamiran and Calders, 2012).
Revised: (2020), which uses a reweighting algorithm proposed in the centralized setting (Kamiran and Calders, 2012).

Original: Specifically, before FL, parties assign the weight for each subgroup (defined by the label and a sensitive attribute)  based on the localtraining dataset (which is referred to as "local reweighting") or global training dataset (which is referred to as "global reweighting").
Revised: The algorithm involves assigning weights to each subgroup (deﬁned by the label and a sensitive attribute) before FL, based on the local or global training dataset. This is called "local reweighting" or "global reweighting," respectively.

Original: Duringthe FL, parties updates the FL model to minimize the weighted loss.
Revised: During FL, parties update the FL model to minimize the weighted loss.

Revised: The table presents the accuracy and fairness gap of the FL algorithm using local and global reweighting for the Income, Health, and Employment datasets.

Original: We found that applying reweighing algorithms in FL reduces the average fairness gap across parties.
Revised: We found that applying reweighing algorithms in FL reduces the average fairness gap across parties.

Original: The reason is that after the  reweighing, theoriginally biased parties will not introduce significant bias to the model duringthe local training, which ultimately reduces the fairness gap of the FL model.
Revised: This is due to the fact that, after reweighing, the parties that were initially biased do not introduce signiﬁcant bias to the model during local training, resulting in a reduction of the fairness gap in the FL model.

Original: At the same time, we notice that the fairness gap of the model is still high with respect to Race sensitive attributes, which means that the global and local reweighing algorithms do not mitigate the biascompletely in FL.
Revised: However, we also noted that the fairness gap in the model remains high for the Race sensitive attribute, indicating that the global and local reweighing algorithms do not entirely eliminate bias in FL.

Original: In addition, we would like to point out some concerns with respect to the fair FL algorithm:
Revised: Furthermore, we would like to highlight some concerns regarding the fair FL algorithm:

Original: The fair FL algorithm improves the performance of the minority group at the cost of the majority group.
Revised: The fair FL algorithm enhances the performance of the minority group at the expense of the majority group.

Original: Figure 24 shows the model performance per group before and after the global reweighting. We observe that, after the reweighting, the minority group (i
Revised: Figure 24 illustrates that the minority group (i

Original: e., the Female group) has a betterperformance. However, this performance improvement comes at the cost of the performance for the majority.
Revised: e., the Female group) has a better performance after the reweighing, but at the cost of reduced performance for the majority.

Original: From Table 3, we notice that when improvingthe fairness with respect to a sensitive attribute, the fairness issue with respect to another sensitive attribute becomes worse.
Revised: When improving fairness with respect to one sensitive attribute, the fairness issue with respect to another sensitive attribute may worsen, as shown in Table 3.

Original: For example, when  local reweighting is applied to reducethe bias with respect to  "Sex" attribute, theEO fairness gap with respect to "Race" increased from 0.514 (results on FedAvg) to 0.521. It indicates that if some parties reweight the local dataset to improve fairness with respect to Sexit may amplify the bias with respect to Race.
Revised: For instance, when applying local reweighing to reduce bias with respect to the "Sex" attribute, the fairness gap with respect to "Race" increased from 0.514 (results on FedAvg) to 0.521, indicating that reweighing to improve fairness with respect to Sex may amplify the bias with respect to Race.

Original: The bias with respect to Race can still be propagated in FL, and our analysis still holds.
Revised: This implies that the bias with respect to Race can still propagate in FL, and our analysis remains valid.

Original: The  algorithms consider the setting where parties are interested in mitigating the bias with respect to the same sensitive attribute, which may not be the case in practice.
Revised: The reweighing algorithms assume that parties are interested in mitigating the bias with respect to the same sensitive attribute, which may not be the case in practice.

Original: For example, Figure 25 shows the fairness gap ofthe standalone models with respect to different sensitive attributes.
Revised: Figureshows the fairness gap of standalone models with respect to different sensitive attributes.

Original: We find that there exist lots of parties whose standalone model achieves a low fairness gap with respect to one sensitive attribute and a high fairness gap with respect to another sensitive attribute.
Revised: We observe that many parties have a low fairness gap with respect to one sensitive attribute and a high fairness gap with respect to another sensitive attribute.

Original: Therefore, parties may be interested in improving fairness with respect to different sensitive attributes. In those settings, this reweighing algorithm may not be applicable.
Revised: Thus, parties may aim to enhance fairness with respect to different sensitive attributes, which renders this reweighing algorithm unsuitable.

Original: This problem is more common in FL as parties have different data distributions.
Revised: This problem is more prevalent in FL, given the different data distributions of parties. [Table]

Original: B. E XTENDING TO A REAL - WORLD MEDICAL DATASET
Revised: D E XTENDING TO A REAL - WORLD MEDICAL DATASET

Original: We evaluate the effect of FL on local fairness on a real-world medical dataset, ISIC

Revised: We evaluate the effect of FL on local fairness on a real-world medical dataset, ISIC

Original: (Abay et al., 2020;
Revised: (Abay et al., 2020;

Original: Tschandl et al., 2018; Combalia et al., 2019), which contains dermoscopic images of skin lesions collected from six medical centers.
Revised: Tschandl et al., 2018; Combalia et al., 2019), which contains dermoscopic images of skin lesions collected from six medical centers.

Original: The task is to classify dermoscopic images among nine different diagnostic categories.
Revised: The task is to classify dermoscopic images among nine different diagnostic categories.

Original: We filter out medical centers with less than 1900 images and[Table] regard each medical center with over 1900 images as a party (4 centers remain after filtering). For every party
Revised: We ﬁlter out medical centers with less than 1900 images and regard each medical center with over 1900 images as a party (4 centers remain after ﬁltering).

Original: [Table] we randomly select 1500 and 400 images for training and validation [Table] respectively.
Revised: For every party, we randomly select 1500 and 400 images for training and validation, respectively.

Original: We regard the binary notion of Sex as our sensitive attribute.
Revised: We regard the binary notion of Sex as our sensitive attribute.

Original: In this multi-class and medical dataset setting [Table] we measure the bias (fairness gap) based on the accuracy gap across groups.
Revised: In this multi-class and medical dataset setting, we measure the bias (fairness gap) based on the accuracy gap across groups.

Original: Following (fla, 2022) [Table] we train EfficientNets
Revised: Following (Ogier du Terrail et al., 2022) 2 , we train EfﬁcientNets

Original: (Tan and Le, 2019), the state-of-the-art model structure for medical data, under the centralized, FL [Table] and standalone setting.
Revised: (Tan and Le, 2019), the state-of-the-art model structure for medical data, under the centralized, FL, and standalone setting.

Original: The accuracy and fairness gap for each client in different settings are shown in Table 4.
Revised: The accuracy and fairness gap for each client in different settings are shown in Table 4.

Original: We notice that FedAvg does not improve the accuracy. The reason may be the data heterogeneity among parties.
Revised: FedAvg does not improve accuracy, possibly due to data heterogeneity among parties (as shown in

Original: As we can see from Figure 26, the numbers of samples from each class are different for the local datasets of different parties. The existing theory (Karimireddy et al., 2020b) also proves that FedAvg suffers from ‘client-drift’.
Revised: Figure 26), resulting in the model’s instability and slow convergence, which is known as ‘client-drift’ issue (Karimireddy et al., 2020b).

Original: Namely, when the data is heterogeneous among parties, the FedAvg model is unstable and converges slowly. Besides, the FedAvg model also fails to achieve the same accuracy across groups, resulting in a higher accuracy gap between groups compared to the standalone setting or the centralized setting (see the fairness gaps on party 1, party 3, and party 4 as examples). 
Revised: Moreover, the FedAvg model results in a higher accuracy gap between groups, worsening the fairness issue compared to the standalone setting or the centralized setting (see the fairness gaps for party 1, party 3, and party 4 as examples). Our results suggest that, in real-world settings, FedAvg may not improve accuracy compared to the standalone setting for the local data distribution and may even exacerbate the fairness issue, leading to a more biased model. E P OTENTIAL M ITIGATION

Original: Our results indicate that, in real-world settings, FedAvg may not improve the accuracy compared to  the standalone setting for the local data distribution, and in the meantime, it even worsens the fairness issue, resulting in a more biased model.
Revised: In this section, we suggest some potential solutions to reduce the effect of bias propagation in FL. Our hope is that this will inspire further research on developing fair FL in the future. Personalized FL Our study has revealed that aggregate updates, which involve the average local update from all parties, may result in a higher bias than local updates for less biased parties (see

Revised: Figure 3).

Revised: This can cause bias from other parties to propagate to the local model, thus increasing the local fairness gap.

Revised: To mitigate the bias propagation effect, one possible approach is to avoid completely overwriting the local model with the global model.

Revised: Personalized Federated Learning (PFL) techniques (Li et al., 2021) have recently been proposed to address this issue by focusing on improving the model’s accuracy.

Revised: PFL can be a promising direction for developing fairness-aware personalized federated learning approaches that can help reduce the bias propagation effect in FL.

Revised: Fair Representation Learning Our analysis demonstrated that bias is encoded in a small number of parameters in the ﬁrst layer of the neural network, which is typically considered the feature extractor (as illustrated in Figure 10).

Revised: This suggests that the FL model is biased because the learned representation is biased, heavily inﬂuenced by the sensitive attribute.

Revised: To address this issue, one possible solution is to implement fair representation learning techniques (Zemel et al., 2013;

Revised: Zhao et al., 2019; Liu et al., 2022; Zeng et al., 2021b)

Revised: that aim to learn a feature extractor which is fair, meaning that it minimizes the dependence on the sensitive attribute while still retaining enough information about the task (or input features).

Revised: Therefore, using fair representation learning in FL can potentially mitigate the propagation of bias effects.