Original: B IAS P ROPAGATION IN F EDERATED L EARNING
Revised: B IAS P ROPAGATION IN F EDERATED L EARNING

Original: A BSTRACT
Revised: A BSTRACT

Original: We show that participating in federated learning can be detrimental to group fairness.
Revised: We show that participating in federated learning can be detrimental to group fairness.

Original: In fact, the bias of a few parties against under-represented groups (identiﬁed by sensitive attributes such as gender or race) can propagate through the network to all the parties in the network.
Revised: In fact, the bias of a few parties against under-represented groups (identiﬁed by sensitive attributes such as gender or race) can propagate through the network to all the parties in the network.

Original: We analyze and explain bias propagation in federated learning on naturally partitioned real-world datasets.
Revised: We analyze and explain bias propagation in federated learning on naturally partitioned real-world datasets.

Original: Our analysis reveals that biased parties unintentionally yet stealthily encode their bias in a small number of model parameters, and throughout the training, they steadily increase the dependence of the global model on sensitive attributes.
Revised: Our analysis reveals that biased parties unintentionally yet stealthily encode their bias in a small number of model parameters, and throughout the training, they steadily increase the dependence of the global model on sensitive attributes.

Original: What is important to highlight is that the experienced bias in federated learning is higher than what parties would otherwise encounter in centralized training with a model trained on the union of all their data.
Revised: What is important to highlight is that the experienced bias in federated learning is higher than what parties would otherwise encounter in centralized training with a model trained on the union of all their data.

Original: This indicates that the bias is due to the algorithm.
Revised: This indicates that the bias is due to the algorithm.

Original: Our work calls for auditing group fairness in federated learning and designing learning algorithms that are robust to bias propagation.
Revised: Our work calls for auditing group fairness in federated learning and designing learning algorithms that are robust to bias propagation.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Machine learning models can exhibit bias against demographic groups.
Revised: Machine learning models can exhibit bias against demographic groups.

Original: Previous research has extensively studied how machine learning algorithms can reﬂect and amplify bias in training data, especially in centralized settings where data is held by a single party (Hardt et al., 2016; Dwork et al., 2012; Calders et al., 2009;
Revised: Previous research has extensively studied how machine learning algorithms can reﬂect and amplify bias in training data, especially in centralized settings where data is held by a single party (Hardt et al., 2016; Dwork et al., 2012; Calders et al., 2009;

Original: Hashimoto et al., 2018; Zhang et al., 2020; Blum and Stangl, 2020;
Revised: Hashimoto et al., 2018; Zhang et al., 2020; Blum and Stangl, 2020;

Original: Lakkaraju et al., 2017).
Revised: Lakkaraju et al., 2017).

Original: However, in practice, data is commonly owned by multiple parties and cannot be shared due to privacy concerns.
Revised: However, in practice, data is commonly owned by multiple parties and cannot be shared due to privacy concerns.

Original: Federated learning (FL) provides a promising solution by enabling parties to collaboratively learn a global model without sharing their data.
Revised: Federated learning (FL) provides a promising solution by enabling parties to collaboratively learn a global model without sharing their data.

Original: In each round of FL, parties share their local model updates computed on their private datasets with a global server that aggregates them to update the global model.
Revised: In each round of FL, parties share their local model updates computed on their private datasets with a global server that aggregates them to update the global model.

Original: Despite the widespread adoption of FL in various applications such as healthcare, recruitment, and loan evaluation (Rieke et al., 2020; Yang et al., 2019), it is not yet fully understood how FL algorithms could magnify bias in training datasets.
Revised: Despite the widespread adoption of FL in various applications such as healthcare, recruitment, and loan evaluation (Rieke et al., 2020; Yang et al., 2019), it is not yet fully understood how FL algorithms could magnify bias in training datasets.

Original: Recent studies have investigated the problem of measuring and mitigating bias in federated learning with respect to a single global distribution (Chu et al., 2021; Zeng et al., 2021a;
Revised: Recent studies have investigated the problem of measuring and mitigating bias in federated learning with respect to a single global distribution (Chu et al., 2021; Zeng et al., 2021a;

Original: Hu et al., 2022; Du et al., 2021; Abay et al., 2020; Papadaki et al., 2021; 2022; Hu et al., 2022).
Revised: Hu et al., 2022; Du et al., 2021; Abay et al., 2020; Papadaki et al., 2021; 2022; Hu et al., 2022).

Original: However, in practice, parties often have heterogeneous data distributions.
Revised: However, in practice, parties often have heterogeneous data distributions.

Original: Evaluating the model’s bias with respect to the global distribution does not accurately reﬂect the fairness of the FL model with respect to parties’ local data distributions, which are relevant to end-users.
Revised: Evaluating the model’s bias with respect to the global distribution does not accurately reﬂect the fairness of the FL model with respect to parties’ local data distributions, which are relevant to end-users.

Original: This is the critical problem that we address in this paper.
Revised: This is the critical problem that we address in this paper.

Original: Speciﬁcally, we investigate the following questions: How does participating in FL affect the bias and fairness of the resulting models compared to models which are trained in a standalone setting?
Revised: Speciﬁcally, we investigate the following questions: How does participating in FL affect the bias and fairness of the resulting models compared to models which are trained in a standalone setting?

Original: Does FL provide parties with the potential fairness beneﬁts of centralized training on the union of their data?
Revised: Does FL provide parties with the potential fairness beneﬁts of centralized training on the union of their data?

Original: Can parties with biased datasets negatively impact the experienced fairness of other parties on their local distributions?
Revised: Can parties with biased datasets negatively impact the experienced fairness of other parties on their local distributions?

Original: How and why does the bias of a small number of parties affect the entire network?
Revised: How and why does the bias of a small number of parties affect the entire network?

Original: To the best of our knowledge, we provide the ﬁrst comprehensive analysis of how FL algorithms impact local fairness.
Revised: To the best of our knowledge, we provide the ﬁrst comprehensive analysis of how FL algorithms impact local fairness.

Original: We provide an empirical analysis based on real-world datasets.
Revised: We provide an empirical analysis based on real-world datasets.

Original: We show that FL might not sustain the beneﬁts of collaboration in terms of fairness, as compared to its accuracy beneﬁt.
Revised: We show that FL might not sustain the beneﬁts of collaboration in terms of fairness, as compared to its accuracy beneﬁt.

Original: Speciﬁcally, compared with the standalone models, we ﬁnd that the model trained in a centralized setting can be, on average, fairer on local data distributions.
Revised: Speciﬁcally, compared with the standalone models, we ﬁnd that the model trained in a centralized setting can be, on average, fairer on local data distributions.

Original: However, in those cases, the FL models, trained on the same dataset, can be more biased.
Revised: However, in those cases, the FL models, trained on the same dataset, can be more biased.

Original: This suggests that the FL algorithm itself can introduce new bias in the ﬁnal model.
Revised: This suggests that the FL algorithm itself can introduce new bias in the ﬁnal model.

Original: Furthermore, we demonstrate that FL impacts different parties in different ways.
Revised: Furthermore, we demonstrate that FL impacts different parties in different ways.

Original: Speciﬁcally, we ﬁnd a strong correlation between parties’ fairness gap in the standalone setting and the fairness beneﬁt they obtain from joining FL: parties with a greater bias in the standalone setting (caused by their local data) would receive a fairer model from FL.
Revised: Speciﬁcally, we ﬁnd a strong correlation between parties’ fairness gap in the standalone setting and the fairness beneﬁt they obtain from joining FL: parties with a greater bias in the standalone setting (caused by their local data) would receive a fairer model from FL.

Original: In contrast, FL has negative impacts on parties with a less signiﬁcant bias in the standalone setting, resulting in a more biased model in FL.
Revised: In contrast, FL has negative impacts on parties with a less signiﬁcant bias in the standalone setting, resulting in a more biased model in FL.

Original: We further demonstrate that this is due to the fact that FL propagates bias : bias from a few parties can inﬂuence all parties in the network, hence aggravating the fairness problem globally.
Revised: We further demonstrate that this is due to the fact that FL propagates bias : bias from a few parties can inﬂuence all parties in the network, hence aggravating the fairness problem globally.

Original: Finally, we offer potential explanations for how bias is propagated in FL.
Revised: Finally, we offer potential explanations for how bias is propagated in FL.

Original: Speciﬁcally, we show that local updates from biased parties increase the dependency between the model’s predictions and the sensitive attributes.
Revised: Speciﬁcally, we show that local updates from biased parties increase the dependency between the model’s predictions and the sensitive attributes.

Original: Such an increase is achieved by the norm increase in a small number of parameters (around 6% of the model parameters in some experiments).
Revised: Such an increase is achieved by the norm increase in a small number of parameters (around 6% of the model parameters in some experiments).

Original: This increase then propagates to the global model through aggregation, subsequently impacting all other parties.
Revised: This increase then propagates to the global model through aggregation, subsequently impacting all other parties.

Original: In addition, we show that the fairness gap of the ﬁnal model can be governed by adjusting the value of those parameters.
Revised: In addition, we show that the fairness gap of the ﬁnal model can be governed by adjusting the value of those parameters.

Original: Surprisingly, we ﬁnd that scaling these parameters can either reduce the model’s bias to a small value of 0 .
Revised: Surprisingly, we ﬁnd that scaling these parameters can either reduce the model’s bias to a small value of 0 .

Original: 05 (on a measurement scale of 0 to 1) with only a 0 .
Revised: 05 (on a measurement scale of 0 to 1) with only a 0 .

Original: 6% drop in accuracy or increase the bias to a large value of 0 .
Revised: 6% drop in accuracy or increase the bias to a large value of 0 .

Original: 96 with an 11% drop in accuracy.
Revised: 96 with an 11% drop in accuracy.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Federated Learning.
Revised: Federated Learning.

Original: In this paper, we consider the conventional federated learning (FL) setting.
Revised: In this paper, we consider the conventional federated learning (FL) setting.

Original: (McMahan et al., 2017).
Revised: (McMahan et al., 2017).

Original: The FL framework consists of a network of K parties, where each party k ∈ [ K ] holds a local dataset
Revised: The FL framework consists of a network of K parties, where each party k ∈ [ K ] holds a local dataset

Original: ˜ D
Revised: ˜ D

Original: k of size n k , sampled from a local data distribution D k .
Revised: k of size n k , sampled from a local data distribution D k .

Original: The objective of each party is to train a model that minimizes the loss on their local data distribution D k .
Revised: The objective of each party is to train a model that minimizes the loss on their local data distribution D k .

Original: To achieve this goal, FL trains a global model to minimize the average loss across parties, which is expressed as min θ 1 N (cid:80)
Revised: To achieve this goal, FL trains a global model to minimize the average loss across parties, which is expressed as min θ 1 N (cid:80)

Original: Kk =1 n k L ( θ, ˜ D k ) , where N is the sum of all local training dataset size.
Revised: Kk =1 n k L ( θ, ˜ D k ) , where N is the sum of all local training dataset size.

Original: In each communication round t , a global server sends the current global model to all parties.
Revised: In each communication round t , a global server sends the current global model to all parties.

Original: All parties train the global model locally on their local dataset and send the updated local model to the server.
Revised: All parties train the global model locally on their local dataset and send the updated local model to the server.

Original: The server then aggregates those local models to obtain the new global model.
Revised: The server then aggregates those local models to obtain the new global model.

Original: We consider the case where all parties participate in the training in each round, which helps to mitigate potential biases that could arise from the non-uniform sampling of participating parties.
Revised: We consider the case where all parties participate in the training in each round, which helps to mitigate potential biases that could arise from the non-uniform sampling of participating parties.

Original: Group Fairness.
Revised: Group Fairness.

Original: Fairness has a wide variety of meanings in literature.
Revised: Fairness has a wide variety of meanings in literature.

Original: Group fairness entails, in particular, that the model should perform comparably across groups deﬁned by sensitive attributes (e.g., sex).
Revised: Group fairness entails, in particular, that the model should perform comparably across groups deﬁned by sensitive attributes (e.g., sex).

Original: It is now common practice to evaluate discrimination in a model (or system) based on quantitative measurements of group fairness.
Revised: It is now common practice to evaluate discrimination in a model (or system) based on quantitative measurements of group fairness.

Original: In light of this, we focus on two widely-used group fairness notions, equalized odds (Hardt et al., 2016) and demographic parity (Dwork et al., 2012).
Revised: In light of this, we focus on two widely-used group fairness notions, equalized odds (Hardt et al., 2016) and demographic parity (Dwork et al., 2012).

Original: To formally deﬁne those fairness notions, we assume each data point is associated with a sensitive attribute a ∈ A , and we use X and Y to denote the input features and the true label.
Revised: To formally deﬁne those fairness notions, we assume each data point is associated with a sensitive attribute a ∈ A , and we use X and Y to denote the input features and the true label.

Original: To measure fairness, we use the fairness gap with respect to Equalized Odds Difference, deﬁned as ∆
Revised: To measure fairness, we use the fairness gap with respect to Equalized Odds Difference, deﬁned as ∆

Original: [Equation]
Revised: [Equation]

Original: with an ideal value equal to zero (perfectly fair).
Revised: with an ideal value equal to zero (perfectly fair).

Original: Furthermore, in many applications, there exists a favorable prediction from the model (e.g., grant the loan).
Revised: Furthermore, in many applications, there exists a favorable prediction from the model (e.g., grant the loan).

Original: We assume the positive prediction ( + ) as the favorable outcome.
Revised: We assume the positive prediction ( + ) as the favorable outcome.

Original: Demographic parity (Dwork et al., 2012) group fairness notion asks the model to give a favorable label to groups with equal rates.
Revised: Demographic parity (Dwork et al., 2012) group fairness notion asks the model to give a favorable label to groups with equal rates.

Original: Similarly, the fairness gap with respect to
Revised: Similarly, the fairness gap with respect to

Original: Demographic Parity is deﬁned as follows:
Revised: Demographic Parity is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: with an ideal value equal to zero.
Revised: with an ideal value equal to zero.

Original: In the rest of the paper, we use the fairness gap ∆ (including ∆ EO and ∆ DP ) to measure the bias (fairness) of a model.
Revised: In the rest of the paper, we use the fairness gap ∆ (including ∆ EO and ∆ DP ) to measure the bias (fairness) of a model.

Original: The more signiﬁcant fairness gap means the model is more biased (less fair).
Revised: The more signiﬁcant fairness gap means the model is more biased (less fair).

Original: We will use bias or unfairness interchangeably.
Revised: We will use bias or unfairness interchangeably.

Original: Measuring the impact of FL.
Revised: Measuring the impact of FL.

Original: Federated learning aims to enhance model performance compared to standalone training and achieve comparable performance to centralized training.
Revised: Federated learning aims to enhance model performance compared to standalone training and achieve comparable performance to centralized training.

Original: Thus, in order to evaluate the impact of FL on local fairness, we use centralized training and standalone training as baselines.
Revised: Thus, in order to evaluate the impact of FL on local fairness, we use centralized training and standalone training as baselines.

Original: In standalone training, each party trains a model θ k independently to minimize the loss on its training data ˜ D k .
Revised: In standalone training, each party trains a model θ k independently to minimize the loss on its training data ˜ D k .

Original: Note that the standalone model’s fairness gap for a party is contributed by herself.
Revised: Note that the standalone model’s fairness gap for a party is contributed by herself.

Original: Therefore, we use the fairness gap of a party’s standalone model to represent the party’s bias.
Revised: Therefore, we use the fairness gap of a party’s standalone model to represent the party’s bias.

Original: In contrast, in centralized training, all local training datasets from all parties are combined, and a centralized model θ c is trained using this global training dataset.
Revised: In contrast, in centralized training, all local training datasets from all parties are combined, and a centralized model θ c is trained using this global training dataset.

Original: We deﬁne the beneﬁt of FL and collaboration (i.e., centralized training) in terms of fairness and accuracy using these baselines.
Revised: We deﬁne the beneﬁt of FL and collaboration (i.e., centralized training) in terms of fairness and accuracy using these baselines.

Original: The beneﬁt of FL for each party k is deﬁned as the difference between the standalone model θ k and the
Revised: The beneﬁt of FL for each party k is deﬁned as the difference between the standalone model θ k and the

Original: FL model θ g .
Revised: FL model θ g .

Original: This difference reveals the extent to which a party beneﬁts from participating in FL.
Revised: This difference reveals the extent to which a party beneﬁts from participating in FL.

Original: Speciﬁcally, for a party k , we deﬁne the accuracy beneﬁt of FL as Acc ( θ g , D k ) − Acc ( θ k , D k ) and fairness beneﬁt of FL as ∆( θ k , D k ) − ∆( θ g , D k ) .
Revised: Speciﬁcally, for a party k , we deﬁne the accuracy beneﬁt of FL as Acc ( θ g , D k ) − Acc ( θ k , D k ) and fairness beneﬁt of FL as ∆( θ k , D k ) − ∆( θ g , D k ) .

Original: Similarly, the beneﬁt of collaboration is based on the difference between the standalone model and the centralized model.
Revised: Similarly, the beneﬁt of collaboration is based on the difference between the standalone model and the centralized model.

Original: This difference indicates how much an individual party could gain from collaborating with others by sharing the entire training dataset.
Revised: This difference indicates how much an individual party could gain from collaborating with others by sharing the entire training dataset.

Original: We compute the average beneﬁt of FL or collaboration across all parties in the network.
Revised: We compute the average beneﬁt of FL or collaboration across all parties in the network.

Original: A positive beneﬁt indicates that FL (or collaboration) improves accuracy or fairness, while a negative beneﬁt implies that FL or collaboration has a detrimental effect.
Revised: A positive beneﬁt indicates that FL (or collaboration) improves accuracy or fairness, while a negative beneﬁt implies that FL or collaboration has a detrimental effect.

Original: Measuring bias aside fairness gap.
Revised: Measuring bias aside fairness gap.

Original: Measuring the bias of a model based on the fairness gap reveals the model’s performance disparity across groups but does not explain why the model is biased.
Revised: Measuring the bias of a model based on the fairness gap reveals the model’s performance disparity across groups but does not explain why the model is biased.

Original: Speciﬁcally, it does not reveal whether the prediction of the model is attributed to the sensitive attribute or any other insensitive attributes whose distributions vary between groups.
Revised: Speciﬁcally, it does not reveal whether the prediction of the model is attributed to the sensitive attribute or any other insensitive attributes whose distributions vary between groups.

Original: The former is typically referred to as disparate treatment (direct discrimination) as the sensitive attribute directly inﬂuences the model prediction (Grabowicz et al., 2022; Zafar et al., 2017).
Revised: The former is typically referred to as disparate treatment (direct discrimination) as the sensitive attribute directly inﬂuences the model prediction (Grabowicz et al., 2022; Zafar et al., 2017).

Original: Towards measuring bias in this aspect, we employ the feature attribution method to explain the model’s prediction on each data point.
Revised: Towards measuring bias in this aspect, we employ the feature attribution method to explain the model’s prediction on each data point.

Original: More precisely, given a model f θ , and the input features x = ( x 1 , x 2 , ..., x d ) , the attribution of the prediction at input x relative to a baseline input x (cid:48) is a vector ( a 1 , ..., a d ) where a i is the contribution of x i to the prediction of f θ ( x ) .
Revised: More precisely, given a model f θ , and the input features x = ( x 1 , x 2 , ..., x d ) , the attribution of the prediction at input x relative to a baseline input x (cid:48) is a vector ( a 1 , ..., a d ) where a i is the contribution of x i to the prediction of f θ ( x ) .

Original: We use Integrated Gradient (Sundararajan et al., 2017) to compute the feature attribution, which only requires a few calls to the gradient operation.
Revised: We use Integrated Gradient (Sundararajan et al., 2017) to compute the feature attribution, which only requires a few calls to the gradient operation.

Original: Following the suggestions by Sundararajan et al.
Revised: Following the suggestions by Sundararajan et al.

Original: (2017), we use the average feature values computed on the whole test dataset as the baseline for non-sensitive and non-binary sensitive features.
Revised: (2017), we use the average feature values computed on the whole test dataset as the baseline for non-sensitive and non-binary sensitive features.

Original: For the binary sensitive attribute, we use the opposite feature value as the baseline.
Revised: For the binary sensitive attribute, we use the opposite feature value as the baseline.

Original: For instance, if the test input is (Sex: female, Age:28), the baseline input would be (Sex: male, Age: average age over the data points).
Revised: For instance, if the test input is (Sex: female, Age:28), the baseline input would be (Sex: male, Age: average age over the data points).

Original: By employing this feature attribution, we are able to discover the bias associated with disparate treatment.
Revised: By employing this feature attribution, we are able to discover the bias associated with disparate treatment.

Original: For more details about the Integrated Gradient, please refer to the original paper (Sundararajan et al., 2017).
Revised: For more details about the Integrated Gradient, please refer to the original paper (Sundararajan et al., 2017).

Original:  E MPIRICAL A NALYSIS
Revised:  E MPIRICAL A NALYSIS

Original: Our objective is to understand how FL impacts fairness for parties.
Revised: Our objective is to understand how FL impacts fairness for parties.

Original: We start by comparing the average performance between FL and baselines (i.e., standalone training and centralized training) to answer the following question: does FL improve fairness for parties compared to standalone training, and does FL provide the same beneﬁt as centralized training in terms of fairness?
Revised: We start by comparing the average performance between FL and baselines (i.e., standalone training and centralized training) to answer the following question: does FL improve fairness for parties compared to standalone training, and does FL provide the same beneﬁt as centralized training in terms of fairness?

Original: Our ﬁndings in
Revised: Our ﬁndings in

Original: Section 3.2 demonstrate that FL can exacerbate fairness issues for parties and does not retain the fairness beneﬁt of collaboration, thus not achieving comparable performance to centralized training.
Revised: Section 3.2 demonstrate that FL can exacerbate fairness issues for parties and does not retain the fairness beneﬁt of collaboration, thus not achieving comparable performance to centralized training.

Original: We further investigate how FL impacts performance for each party to answer the following questions:
Revised: We further investigate how FL impacts performance for each party to answer the following questions:

Original: does FL provide the same beneﬁt in terms of fairness for parties, and what causes the disparate impact of FL on parties’ fairness?
Revised: does FL provide the same beneﬁt in terms of fairness for parties, and what causes the disparate impact of FL on parties’ fairness?

Original: Our analysis in Section 3.3 shows that FL can propagate bias among parties.
Revised: Our analysis in Section 3.3 shows that FL can propagate bias among parties.

Original: As a result, FL improves fairness for parties with greater bias while worsening fairness issues for parties with less bias.
Revised: As a result, FL improves fairness for parties with greater bias while worsening fairness issues for parties with less bias.

Original: Finally, we investigate how the bias is propagated in FL .
Revised: Finally, we investigate how the bias is propagated in FL .

Original: The analysis in Section 3.4 shows that biased parties encode the bias in a few parameters through local updates, which are propagated to the aggregated model and ultimately to other parties via parameter aggregation.
Revised: The analysis in Section 3.4 shows that biased parties encode the bias in a few parameters through local updates, which are propagated to the aggregated model and ultimately to other parties via parameter aggregation.

Original: 3. S ETUP
Revised: 3. S ETUP

Original: We use the two datasets with different tasks for our empirical analysis: US Census Data and
Revised: We use the two datasets with different tasks for our empirical analysis: US Census Data and

Original: CelebA (Liu et al., 2015) .
Revised: CelebA (Liu et al., 2015) .

Original: The US Census dataset consists of census data from different places in the US.
Revised: The US Census dataset consists of census data from different places in the US.

Original: We naturally partition the dataset based on the source of the data.
Revised: We naturally partition the dataset based on the source of the data.

Original: Thus, we haveparties in total, representing 50 states in the US and Puerto Rico.
Revised: Thus, we haveparties in total, representing 50 states in the US and Puerto Rico.

Original: We consider three tasks deﬁned
Revised: We consider three tasks deﬁned

Original: [Table]
Revised: [Table]

Original: in the folktables (Ding et al., 2021): Income (ACSIncome) 1 , Health (ACSPublicCoverage), and
Revised: in the folktables (Ding et al., 2021): Income (ACSIncome) 1 , Health (ACSPublicCoverage), and

Original: Employment (ACSEmployment).
Revised: Employment (ACSEmployment).

Original: We train two-layer neural network models for all the tasks.
Revised: We train two-layer neural network models for all the tasks.

Original: We consider the binary notion of sex (i.e., male and female) and multi-value race as the sensitive attribute for every task.
Revised: We consider the binary notion of sex (i.e., male and female) and multi-value race as the sensitive attribute for every task.

Original: Each party has 1 , 000 training points and 2 , 000 test points.
Revised: Each party has 1 , 000 training points and 2 , 000 test points.

Original: Note that we use a large test dataset to ensure an accurate estimation of the fairness gap with respect to parities’ test distribution.
Revised: Note that we use a large test dataset to ensure an accurate estimation of the fairness gap with respect to parities’ test distribution.

Original: For CelebA, an image dataset consisting of 200 , 000 celebrity images, we focus on the age prediction task and use the binary attribute "Male" (male and non-male) as the sensitive attribute, which we refer to as the Sex attribute.
Revised: For CelebA, an image dataset consisting of 200 , 000 celebrity images, we focus on the age prediction task and use the binary attribute "Male" (male and non-male) as the sensitive attribute, which we refer to as the Sex attribute.

Original: We partition the data uniformly at random among all parties and train CNN models.
Revised: We partition the data uniformly at random among all parties and train CNN models.

Original: We also consider non-IID partitioning on the CelebA dataset, and the results are presented in Appendix B.
Revised: We also consider non-IID partitioning on the CelebA dataset, and the results are presented in Appendix B.

Original: In all experiments, we report the average results over ﬁve runs with different random seeds.
Revised: In all experiments, we report the average results over ﬁve runs with different random seeds.

Original: We use FedML (He et al., 2020), a PyTorch (Paszke et al., 2019)-based library for FL, to train models in FL.
Revised: We use FedML (He et al., 2020), a PyTorch (Paszke et al., 2019)-based library for FL, to train models in FL.

Original: We also use Captum (Kokhlikyan et al., 2020) to compute feature attribution.
Revised: We also use Captum (Kokhlikyan et al., 2020) to compute feature attribution.

Original: For further details on the datasets and models, please refer to Appendix A.
Revised: For further details on the datasets and models, please refer to Appendix A.

Original: C OLLABORATION VIA FL CAN WORSEN FAIRNESS ISSUE
Revised: C OLLABORATION VIA FL CAN WORSEN FAIRNESS ISSUE

Original: Table 1 presents the average accuracy and fairness gaps of the centralized model, FL model, and standalone models on local datasets.
Revised: Table 1 presents the average accuracy and fairness gaps of the centralized model, FL model, and standalone models on local datasets.

Original: Centralized training is observed to improve accuracy and fairness at the same time for parties.
Revised: Centralized training is observed to improve accuracy and fairness at the same time for parties.

Original: For instance, on the Income dataset, centralized training improves the accuracy by 6% and reduces the fairness gap across racial groups by 5 . 5% with respect to equalized odds and by 5 . 5% with respect to demographic parity.
Revised: For instance, on the Income dataset, centralized training improves the accuracy by 6% and reduces the fairness gap across racial groups by 5 . 5% with respect to equalized odds and by 5 . 5% with respect to demographic parity.

Original: However, FL may not achieve the same beneﬁt as centralized training in terms of fairness and can even exacerbate the fairness issue for parties.
Revised: However, FL may not achieve the same beneﬁt as centralized training in terms of fairness and can even exacerbate the fairness issue for parties.

Original: For example, on the Income dataset, the EO fairness gap of the FL model for the sex groups increases by 35% , and the DP fairness gap increases by 29 . 9% , compared to standalone models.
Revised: For example, on the Income dataset, the EO fairness gap of the FL model for the sex groups increases by 35% , and the DP fairness gap increases by 29 . 9% , compared to standalone models.

Original: We include results for other popular FL algorithms, including FedNova (Wang et al., 2020), Scaffold Karimireddy et al.
Revised: We include results for other popular FL algorithms, including FedNova (Wang et al., 2020), Scaffold Karimireddy et al.

Original: (2020b), FedOpt (Reddi et al., 2020), FedProx Li et al.
Revised: (2020b), FedOpt (Reddi et al., 2020), FedProx Li et al.

Original: (2020), and Mime (Karimireddy et al., 2020a), in Table 2 in Appendix B, which show a similar pattern to that of the FedAvg algorithm.
Revised: (2020), and Mime (Karimireddy et al., 2020a), in Table 2 in Appendix B, which show a similar pattern to that of the FedAvg algorithm.

Original: Appendix B provides more detailed results on other FL algorithms.
Revised: Appendix B provides more detailed results on other FL algorithms.

Original: The centralized and FL models are trained on the same dataset.
Revised: The centralized and FL models are trained on the same dataset.

Original: The difference between the FL model and the centralized model in terms of fairness suggests FL algorithm can introduce more bias compared to standard training.
Revised: The difference between the FL model and the centralized model in terms of fairness suggests FL algorithm can introduce more bias compared to standard training.

Original: Therefore, the explanation of how bias is introduced during standard training in the centralized setting may not fully explain how FL introduces bias.
Revised: Therefore, the explanation of how bias is introduced during standard training in the centralized setting may not fully explain how FL introduces bias.

Original: In the following, we further explore how FL impacts fairness from the party level.
Revised: In the following, we further explore how FL impacts fairness from the party level.

Original: 3. FL PROPAGATES BIAS AMONG PARTIES
Revised: 3. FL PROPAGATES BIAS AMONG PARTIES

Original: Disparate impact of FL on group fairness across parties.
Revised: Disparate impact of FL on group fairness across parties.

Original: Figure 1 illustrates the beneﬁt of FL on fairness and accuracy for parties.
Revised: Figure 1 illustrates the beneﬁt of FL on fairness and accuracy for parties.

Original: We observe that FL improves accuracy for almost all parties, and the variance in accuracy improvement across parties is small.
Revised: We observe that FL improves accuracy for almost all parties, and the variance in accuracy improvement across parties is small.

Original: On the other hand, the fairness beneﬁt of FL is negative for most parties.
Revised: On the other hand, the fairness beneﬁt of FL is negative for most parties.

Original: It implies that most parties obtain a more biased model in FL compared to standalone training.
Revised: It implies that most parties obtain a more biased model in FL compared to standalone training.

Original: Furthermore, we notice that the variance in the fairness beneﬁts across parties is large, suggesting that although all parties have the same global model in FL, they do not beneﬁt from the FL model equally (each party evaluates the model performance on their local test dataset).
Revised: Furthermore, we notice that the variance in the fairness beneﬁts across parties is large, suggesting that although all parties have the same global model in FL, they do not beneﬁt from the FL model equally (each party evaluates the model performance on their local test dataset).

Original: To explore the impact of FL on fairness at the party level,
Revised: To explore the impact of FL on fairness at the party level,

Original: Figure 2 shows strong correlations between the fairness beneﬁt a party obtains in FL and the fairness gap of the standalone model for the party, i e., the bias level of the party.
Revised: Figure 2 shows strong correlations between the fairness beneﬁt a party obtains in FL and the fairness gap of the standalone model for the party, i e., the bias level of the party.

Original: This ﬁnding highlights the disparate impact of FL on fairness: FL can improve fairness for more biased parties but at the cost of worsening the issue for less biased parties.
Revised: This ﬁnding highlights the disparate impact of FL on fairness: FL can improve fairness for more biased parties but at the cost of worsening the issue for less biased parties.

Original: Contradiction between aggregation and local update.
Revised: Contradiction between aggregation and local update.

Original: During the training process of FL, we observe that aggregation and local update contradict each other, shown in
Revised: During the training process of FL, we observe that aggregation and local update contradict each other, shown in

Original: Figure 3.
Revised: Figure 3.

Original: Local update from the least biased party, whose standalone model has the lowest fairness gap, reduces the fairness gap of the model.
Revised: Local update from the least biased party, whose standalone model has the lowest fairness gap, reduces the fairness gap of the model.

Original: However, this reduction is eliminated by the aggregation step.
Revised: However, this reduction is eliminated by the aggregation step.

Original: Conversely, the local update from the most biased party increases the fairness gap of the model, which is then reduced by aggregation.
Revised: Conversely, the local update from the most biased party increases the fairness gap of the model, which is then reduced by aggregation.

Original: This ﬁnding implies that the aggregation contributes to the disparate impact of FL on fairness, improving fairness for more biased parties but worsening the fairness for less biased parties compared to the standalone setting.
Revised: This ﬁnding implies that the aggregation contributes to the disparate impact of FL on fairness, improving fairness for more biased parties but worsening the fairness for less biased parties compared to the standalone setting.

Original: Biased parties negatively inﬂuence other parties via aggregation throughout the training
Revised: Biased parties negatively inﬂuence other parties via aggregation throughout the training

Original: Why does aggregation have disparate impacts on local fairness for parties?
Revised: Why does aggregation have disparate impacts on local fairness for parties?

Original: Speciﬁcally, we ask which party’s local update causes the increase (or decrease) of the fairness gap for other parties via
Revised: Speciﬁcally, we ask which party’s local update causes the increase (or decrease) of the fairness gap for other parties via

Original: The Pearson correlation coefﬁcient between the fairness gap a party obtains in the standalone setting and the impact she has on local fairness for parties in FL are shown in the ﬁgure aggregation.
Revised: The Pearson correlation coefﬁcient between the fairness gap a party obtains in the standalone setting and the impact she has on local fairness for parties in FL are shown in the ﬁgure aggregation.

Original: To answer this question, we look into the inﬂuence of a party’s local update on other parties’ fairness via aggregation.
Revised: To answer this question, we look into the inﬂuence of a party’s local update on other parties’ fairness via aggregation.

Original: More precisely, we compute the inﬂuence of party i on party j as the fairness gap increase when party i ’s local update is removed from the aggregation in each round t and sum over all the training rounds.
Revised: More precisely, we compute the inﬂuence of party i on party j as the fairness gap increase when party i ’s local update is removed from the aggregation in each round t and sum over all the training rounds.

Original: Formally, we deﬁne the inﬂuence of party i on party j as
Revised: Formally, we deﬁne the inﬂuence of party i on party j as

Original: I i,j = (cid:80) Tt =1 ∆( θ t, − i , D j ) − ∆( θ t , D j ) , where θ t is the global model (i.e., the aggregated model overall local updated models) and θ t, − i is the aggregated model over local updated models from parties excluding party i .
Revised: I i,j = (cid:80) Tt =1 ∆( θ t, − i , D j ) − ∆( θ t , D j ) , where θ t is the global model (i.e., the aggregated model overall local updated models) and θ t, − i is the aggregated model over local updated models from parties excluding party i .

Original: If party i improves fairness for party j , the inﬂuence is positive and vice versa.
Revised: If party i improves fairness for party j , the inﬂuence is positive and vice versa.

Original: We compute the inﬂuence for all pairs of parties, and the most inﬂuential pairs are shown in
Revised: We compute the inﬂuence for all pairs of parties, and the most inﬂuential pairs are shown in

Original: Figure 4.
Revised: Figure 4.

Original: We observe that a less biased party has a positive inﬂuence on fairness for other parties, while a more biased party has a negative inﬂuence on other parties.
Revised: We observe that a less biased party has a positive inﬂuence on fairness for other parties, while a more biased party has a negative inﬂuence on other parties.

Original: This result shows that a biased party can negatively inﬂuence other parties’ fairness via aggregation throughout the training.
Revised: This result shows that a biased party can negatively inﬂuence other parties’ fairness via aggregation throughout the training.

Original: Furthermore, we investigate the relationship between a party’s bias (i.e., the bias of the party’s standalone model) and its average inﬂuence on all parties’ fairness (i.e., (cid:80)
Revised: Furthermore, we investigate the relationship between a party’s bias (i.e., the bias of the party’s standalone model) and its average inﬂuence on all parties’ fairness (i.e., (cid:80)

Original: Kk =

Revised: Kk =

Original: I i,k /K ).
Revised: I i,k /K ).

Original: The results are presented in Figure 5, which shows a strong correlation between these two factors.
Revised: The results are presented in Figure 5, which shows a strong correlation between these two factors.

Original: This ﬁnding further supports our conclusion that the more biased parties have a stronger negative inﬂuence on other parties’ fairness, while the less biased parties have a stronger positive inﬂuence.
Revised: This ﬁnding further supports our conclusion that the more biased parties have a stronger negative inﬂuence on other parties’ fairness, while the less biased parties have a stronger positive inﬂuence.

Original: Finally, we analyze the dynamics of the inﬂuence of the top 5 most biased parties (i.e., parties with the largest fairness gap in the standalone setting) and the top 5 least biased parties, as shown in Figure 6.
Revised: Finally, we analyze the dynamics of the inﬂuence of the top 5 most biased parties (i.e., parties with the largest fairness gap in the standalone setting) and the top 5 least biased parties, as shown in Figure 6.

Original: We observe that the inﬂuence of the most biased parties is monotonically increasing throughout the training, while the inﬂuence of the least biased parties is decreasing.
Revised: We observe that the inﬂuence of the most biased parties is monotonically increasing throughout the training, while the inﬂuence of the least biased parties is decreasing.

Original: In other words, biased parties consistently have a negative inﬂuence on others’ fairness gaps, while less biased parties have a positive inﬂuence.
Revised: In other words, biased parties consistently have a negative inﬂuence on others’ fairness gaps, while less biased parties have a positive inﬂuence.

Original: These results suggest that FL can propagate bias among parties: the bias from biased parties negatively inﬂuence the fairness of other parties via aggregation throughout the training.
Revised: These results suggest that FL can propagate bias among parties: the bias from biased parties negatively inﬂuence the fairness of other parties via aggregation throughout the training.

Original: Next, we will explore how the bias is propagated in FL.
Revised: Next, we will explore how the bias is propagated in FL.

Original: 3. H OW IS BIAS PROPAGATED IN FL?
Revised: 3. H OW IS BIAS PROPAGATED IN FL?

Original: Disparate treatment causes large fairness gaps.
Revised: Disparate treatment causes large fairness gaps.

Original: Our ﬁrst investigation explores what the bias represents, speciﬁcally whether the bias increase in FL is directly caused by the disparate treatment of the model among sensitive groups.
Revised: Our ﬁrst investigation explores what the bias represents, speciﬁcally whether the bias increase in FL is directly caused by the disparate treatment of the model among sensitive groups.

Original: To answer this question, we utilize Integrated Gradients (Sundararajan et al., 2017) to measure the attribution of each input feature to the models’ predictions with respect to the positive class.
Revised: To answer this question, we utilize Integrated Gradients (Sundararajan et al., 2017) to measure the attribution of each input feature to the models’ predictions with respect to the positive class.

Original: Figure 7 shows the attribution value distribution for the sensitive attribute "Sex" over individual test points from the female and male groups.
Revised: Figure 7 shows the attribution value distribution for the sensitive attribute "Sex" over individual test points from the female and male groups.

Original: We notice that the sex attribute has a large attribution value for the standalone model’s predictions and the FL model’s predictions.
Revised: We notice that the sex attribute has a large attribution value for the standalone model’s predictions and the FL model’s predictions.

Original: This ﬁnding implies that the predictions of those models are heavily dependent on the sensitive attribute of the test data.
Revised: This ﬁnding implies that the predictions of those models are heavily dependent on the sensitive attribute of the test data.

Original: Moreover, the sensitive attribute affects the model predictions differently for the male and female groups, with the average attribution value being positive for the male group and negative for the female group.
Revised: Moreover, the sensitive attribute affects the model predictions differently for the male and female groups, with the average attribution value being positive for the male group and negative for the female group.

Original: Furthermore, we ﬁnd that there is minimal difference in the attribution value for other attributes with respect to the female and male groups (see Figurein Appendix B).
Revised: Furthermore, we ﬁnd that there is minimal difference in the attribution value for other attributes with respect to the female and male groups (see Figurein Appendix B).

Original: This indicates that the large fairness gap in the models is not caused by the distinct distribution of other insensitive attributes over protected groups; rather, it is mainly caused by the models’ disparate treatment of protected groups.
Revised: This indicates that the large fairness gap in the models is not caused by the distinct distribution of other insensitive attributes over protected groups; rather, it is mainly caused by the models’ disparate treatment of protected groups.

Original: FL model learns more biased patterns.
Revised: FL model learns more biased patterns.

Original: In Figure 7, we compare the attribution value of “Sex” for different models and ﬁnd that, while the predictions of the FL model depend less heavily on the sensitive attribute compared to those of the standalone model, the dependence is still stronger than that of the centralized model.
Revised: In Figure 7, we compare the attribution value of “Sex” for different models and ﬁnd that, while the predictions of the FL model depend less heavily on the sensitive attribute compared to those of the standalone model, the dependence is still stronger than that of the centralized model.

Original: This suggests that the FL model learns a more biased pattern compared to what could be learned in the centralized setting.
Revised: This suggests that the FL model learns a more biased pattern compared to what could be learned in the centralized setting.

Original: Figure 8 shows the dynamic of the average absolute feature attribution for “Sex” during the training.
Revised: Figure 8 shows the dynamic of the average absolute feature attribution for “Sex” during the training.

Original: We ﬁnd that standalone training increases the model’s dependence on the sensitive attribute throughout training for the most biased party, but centralized training decreases it.
Revised: We ﬁnd that standalone training increases the model’s dependence on the sensitive attribute throughout training for the most biased party, but centralized training decreases it.

Original: This suggests that collaboration through centralized training improves local fairness by guiding the model to learn less biased features.
Revised: This suggests that collaboration through centralized training improves local fairness by guiding the model to learn less biased features.

Original: In FL, however, the absolute attribution value barely changes after 100 rounds and remains signiﬁcantly greater than that in the centralized setting.
Revised: In FL, however, the absolute attribution value barely changes after 100 rounds and remains signiﬁcantly greater than that in the centralized setting.

Original: This implies that the FL algorithm may introduce bias to the ﬁnal model by inhibiting the model from learning less biased features.
Revised: This implies that the FL algorithm may introduce bias to the ﬁnal model by inhibiting the model from learning less biased features.

Original: Biased parties increase the model dependence on the sensitive attribute.
Revised: Biased parties increase the model dependence on the sensitive attribute.

Original: Figure 9 shows the attribution value of the aggregated model and locally updated model from the most biased party and least biased party.
Revised: Figure 9 shows the attribution value of the aggregated model and locally updated model from the most biased party and least biased party.

Original: We observe that the biased party increases the global model dependence on sensitive attributes during the local update, and this increase persists throughout the training.
Revised: We observe that the biased party increases the global model dependence on sensitive attributes during the local update, and this increase persists throughout the training.

Original: In contrast, the least biased party reduces the dependence on the sensitive attribute, which is aligned with the trend of the fairness gap in Figure 3.
Revised: In contrast, the least biased party reduces the dependence on the sensitive attribute, which is aligned with the trend of the fairness gap in Figure 3.

Original: This suggests that the biased parties have a negative impact on the fairness of other parties by increasing the model’s dependence on the sensitive attribute.
Revised: This suggests that the biased parties have a negative impact on the fairness of other parties by increasing the model’s dependence on the sensitive attribute.

Original: Bias is encoded in a few parameters.
Revised: Bias is encoded in a few parameters.

Original: Biased parties increase the model dependence on the sensitive attribute.
Revised: Biased parties increase the model dependence on the sensitive attribute.

Original: But how does this increase propagate to other parties in the network?
Revised: But how does this increase propagate to other parties in the network?

Original: Since parties share the model parameters of the local model with the server, the bias is likely encoded in the model parameters.
Revised: Since parties share the model parameters of the local model with the server, the bias is likely encoded in the model parameters.

Original: Therefore, we investigate which parameters are related to the model’s bias.
Revised: Therefore, we investigate which parameters are related to the model’s bias.

Original: Intuitively, the parameters used to extract sensitive attribute information impact the attribution value of the sensitive attribute to the model prediction.
Revised: Intuitively, the parameters used to extract sensitive attribute information impact the attribution value of the sensitive attribute to the model prediction.

Original: If the absolute value (signal) of those parameters is substantial, the value of the sensitive attribute will signiﬁcantly affect the model’s prediction.
Revised: If the absolute value (signal) of those parameters is substantial, the value of the sensitive attribute will signiﬁcantly affect the model’s prediction.

Original: In our evaluation, the sensitive attribute is part of the input feature, so the parameters directly applied to the “Sex” attribute in the ﬁrst layer of the neural network should contribute to the model’s bias.
Revised: In our evaluation, the sensitive attribute is part of the input feature, so the parameters directly applied to the “Sex” attribute in the ﬁrst layer of the neural network should contribute to the model’s bias.

Original: Figure 10(a) shows the normalized norm of the parameters associated with the sensitive attribute for the most and least biased parties in the aggregated and locally updated models.
Revised: Figure 10(a) shows the normalized norm of the parameters associated with the sensitive attribute for the most and least biased parties in the aggregated and locally updated models.

Original: The normalized norm is deﬁned as the norm of parameters associated with the sensitive attribute divided by the parameter norm of the ﬁrst layer.
Revised: The normalized norm is deﬁned as the norm of parameters associated with the sensitive attribute divided by the parameter norm of the ﬁrst layer.

Original: We observe that the least biased party reduces the parameter norm, hence decreasing the model’s sensitivity to the sensitive attribute.
Revised: We observe that the least biased party reduces the parameter norm, hence decreasing the model’s sensitivity to the sensitive attribute.

Original: On the other hand, the biased party increases the norm for those parameters, amplifying the impact of the sensitive attribute on the model’s prediction.
Revised: On the other hand, the biased party increases the norm for those parameters, amplifying the impact of the sensitive attribute on the model’s prediction.

Original: Through aggregation, this ampliﬁcation will be propagated to the global model.
Revised: Through aggregation, this ampliﬁcation will be propagated to the global model.

Original: Figure 10(b) reveals a moderate correlation between the fairness gap party experiences in the standalone setting and the normalized parameter norm for the parameters used to extract sensitive attribute information.
Revised: Figure 10(b) reveals a moderate correlation between the fairness gap party experiences in the standalone setting and the normalized parameter norm for the parameters used to extract sensitive attribute information.

Original: This implies that biased parties increase the parameter value associated with sensitive attributes during the local update, thereby boosting the model’s susceptibility to the sensitive attribute.
Revised: This implies that biased parties increase the parameter value associated with sensitive attributes during the local update, thereby boosting the model’s susceptibility to the sensitive attribute.

Original: Controlling fairness gap by scaling a few parameters.
Revised: Controlling fairness gap by scaling a few parameters.

Original: To further investigate the impact of the parameters associated with the sensitive attribute (i.e., 104 parameters out of 1 , 792 ) on model fairness, we examine the effect of scaling these parameters on the fairness gap in Figure 10(c).
Revised: To further investigate the impact of the parameters associated with the sensitive attribute (i.e., 104 parameters out of 1 , 792 ) on model fairness, we examine the effect of scaling these parameters on the fairness gap in Figure 10(c).

Original: We ﬁnd that the fairness gap can be greatly widened or narrowed at the expense of a moderate degree of accuracy.
Revised: We ﬁnd that the fairness gap can be greatly widened or narrowed at the expense of a moderate degree of accuracy.

Original: Speciﬁcally, by scaling the parameter value by 0 . 1 for the trained FL model, we signiﬁcantly reduce the EO gap by 74 .
Revised: Speciﬁcally, by scaling the parameter value by 0 . 1 for the trained FL model, we signiﬁcantly reduce the EO gap by 74 .

Original: 6% (from 0 .
Revised: 6% (from 0 .

Original: 198 to 0 .
Revised: 198 to 0 .

Original: and the DP gap by 69 .
Revised: and the DP gap by 69 .

Original: 1% (from 0 .
Revised: 1% (from 0 .

Original: 217 to 0 .
Revised: 217 to 0 .

Original: with just a moderate accuracy loss of 0 .
Revised: with just a moderate accuracy loss of 0 .

Original: 8% (from 0 .
Revised: 8% (from 0 .

Original: 785 to 0 .
Revised: 785 to 0 .

Original: In contrast, scaling the same set of parameters by a factor of 10 increases the EO gap to 0 .
Revised: In contrast, scaling the same set of parameters by a factor of 10 increases the EO gap to 0 .

Original: 96 (the maximal fairness gap is 1 ), which is almost ﬁve times larger, and increases the DP gap by 259% to 0 .
Revised: 96 (the maximal fairness gap is 1 ), which is almost ﬁve times larger, and increases the DP gap by 259% to 0 .

Original: 78 , while reducing the accuracy from 0 .
Revised: 78 , while reducing the accuracy from 0 .

Original: 785 to 0 .
Revised: 785 to 0 .

Original: These ﬁndings explain how bias is propagated in FL: biased parties magnify the impact of sensitive attributes on model predictions by increasing the model parameter used to extract sensitive attributes.
Revised: These ﬁndings explain how bias is propagated in FL: biased parties magnify the impact of sensitive attributes on model predictions by increasing the model parameter used to extract sensitive attributes.

Original: This rise in parameters is subsequently propagated to the global model through aggregation, further aggravating the issue of fairness for other parties.
Revised: This rise in parameters is subsequently propagated to the global model through aggregation, further aggravating the issue of fairness for other parties.

Original: Our results explain how bias is propagated in FL: Biased parties encode bias in a few parameters through a local update, and this bias is consequently propagated to the entire network through parameter aggregation.
Revised: Our results explain how bias is propagated in FL: Biased parties encode bias in a few parameters through a local update, and this bias is consequently propagated to the entire network through parameter aggregation.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Fairness has received considerable attention due to the growing deployment of machine learning in decision-making processes.
Revised: Fairness has received considerable attention due to the growing deployment of machine learning in decision-making processes.

Original: Various deﬁnitions of fairness have been presented (Hardt et al., 2016; Dwork et al., 2012; Calders et al., 2009).
Revised: Various deﬁnitions of fairness have been presented (Hardt et al., 2016; Dwork et al., 2012; Calders et al., 2009).

Original: Speciﬁcally, group fairness requires that the model behave similarly for groups deﬁned by a sensitive attribute (e.g., race).
Revised: Speciﬁcally, group fairness requires that the model behave similarly for groups deﬁned by a sensitive attribute (e.g., race).

Original: While how machine learning algorithms propagate data bias to the ﬁnal model has been extensively investigated in a centralized setting (Blum and Stangl, 2020; Lakkaraju et al., 2017; Rambachan and Roth, 2020; Friedler et al., 2019; Dullerud et al., 2022), the effect of FL on model fairness is not yet fully understood.
Revised: While how machine learning algorithms propagate data bias to the ﬁnal model has been extensively investigated in a centralized setting (Blum and Stangl, 2020; Lakkaraju et al., 2017; Rambachan and Roth, 2020; Friedler et al., 2019; Dullerud et al., 2022), the effect of FL on model fairness is not yet fully understood.

Original: The existing literature on fairness in FL mainly focuses on the performance disparity of FL models across parties, rather than demographic groups (Li et al., 2021; Zhao and Joshi, 2022;
Revised: The existing literature on fairness in FL mainly focuses on the performance disparity of FL models across parties, rather than demographic groups (Li et al., 2021; Zhao and Joshi, 2022;

Original: Li et al., 2019;
Revised: Li et al., 2019;

Original: Mohri et al., 2019; Deng et al., 2020; Donahue and Kleinberg, 2021;
Revised: Mohri et al., 2019; Deng et al., 2020; Donahue and Kleinberg, 2021;

Original: Hao et al., 2021; Zhou et al., 2021; Yu et al., 2020;
Revised: Hao et al., 2021; Zhou et al., 2021; Yu et al., 2020;

Original: Lyu et al., 2020).
Revised: Lyu et al., 2020).

Original: However, we focus on group fairness, which concerns performance disparity among groups.
Revised: However, we focus on group fairness, which concerns performance disparity among groups.

Original: In terms of group fairness, Abay et al.
Revised: In terms of group fairness, Abay et al.

Original: (2020) listed a few potential sources of bias in FL.
Revised: (2020) listed a few potential sources of bias in FL.

Original: Recently, considerable progress has been made in training group fair models in FL (Abay et al., 2020;
Revised: Recently, considerable progress has been made in training group fair models in FL (Abay et al., 2020;

Original: Chu et al., 2021; Zeng et al., 2021a;
Revised: Chu et al., 2021; Zeng et al., 2021a;

Original: Hu et al., 2022; Du et al., 2021; Ezzeldin et al., 2021).
Revised: Hu et al., 2022; Du et al., 2021; Ezzeldin et al., 2021).

Original: Nonetheless, the majority of these works suggest techniques for achieving fairness on a single test distribution.
Revised: Nonetheless, the majority of these works suggest techniques for achieving fairness on a single test distribution.

Original: Instead, we focus on fairness issues for parties.
Revised: Instead, we focus on fairness issues for parties.

Original: Some studies (Cui et al., 2021; Papadaki et al., 2022) proposed algorithms to improve local fairness for parties.
Revised: Some studies (Cui et al., 2021; Papadaki et al., 2022) proposed algorithms to improve local fairness for parties.

Original: Our purpose, instead, is to gain a comprehensive understanding of how FL inﬂuences local fairness on its own, which we believe is equally crucial as designing fair algorithms.
Revised: Our purpose, instead, is to gain a comprehensive understanding of how FL inﬂuences local fairness on its own, which we believe is equally crucial as designing fair algorithms.

Original:  F UTURE W ORK & C ONCLUSION
Revised:  F UTURE W ORK & C ONCLUSION

Original: Future Work.
Revised: Future Work.

Original: In this work, we have investigated how bias is propagated in FL when the sensitive attribute is included as an input feature.
Revised: In this work, we have investigated how bias is propagated in FL when the sensitive attribute is included as an input feature.

Original: In practice, however, sensitive attributes may be prohibited from being included in input features.
Revised: In practice, however, sensitive attributes may be prohibited from being included in input features.

Original: In such situations, the model may still be heavily biased due to variables that are correlated with the (unobserved) sensitive attribute.
Revised: In such situations, the model may still be heavily biased due to variables that are correlated with the (unobserved) sensitive attribute.

Original: For instance, a person’s zip code may be highly correlated with their race, a phenomenon known as "redlining".
Revised: For instance, a person’s zip code may be highly correlated with their race, a phenomenon known as "redlining".

Original: A promising direction for future work is to identify which model parameters contribute to the bias and to audit the bias propagation in this setting.
Revised: A promising direction for future work is to identify which model parameters contribute to the bias and to audit the bias propagation in this setting.

Original: Another important direction is to design FL algorithms that are robust to bias propagation.
Revised: Another important direction is to design FL algorithms that are robust to bias propagation.

Original: In Appendix E, we brieﬂy discuss a few potential ways to achieve this goal.
Revised: In Appendix E, we brieﬂy discuss a few potential ways to achieve this goal.

Original: Conclusion.
Revised: Conclusion.

Original: Federated learning has become increasingly popular in various applications with signiﬁcant individual-level consequences, making it essential to anticipate the possible bias introduced by FL.
Revised: Federated learning has become increasingly popular in various applications with signiﬁcant individual-level consequences, making it essential to anticipate the possible bias introduced by FL.

Original: Our paper takes the ﬁrst step in this direction by providing a comprehensive analysis of the impact of FL on local fairness for parties.
Revised: Our paper takes the ﬁrst step in this direction by providing a comprehensive analysis of the impact of FL on local fairness for parties.

Original: We demonstrated that the FL algorithm could introduce bias on its own which may exacerbate the issue of fairness for the involved parties.
Revised: We demonstrated that the FL algorithm could introduce bias on its own which may exacerbate the issue of fairness for the involved parties.

Original: Moreover, we showed that this exacerbation is not evenly distributed among parties, as FL can propagate bias among them.
Revised: Moreover, we showed that this exacerbation is not evenly distributed among parties, as FL can propagate bias among them.

Original: Finally, we explained how bias is propagated in FL: biased parties encode their bias into the local updates by increasing the signal of a few parameters steadily throughout the training process, which is then propagated to the global model via aggregation and, ultimately, to other parties.
Revised: Finally, we explained how bias is propagated in FL: biased parties encode their bias into the local updates by increasing the signal of a few parameters steadily throughout the training process, which is then propagated to the global model via aggregation and, ultimately, to other parties.

Original:  A CKNOWLEDGEMENT
Revised:  A CKNOWLEDGEMENT

Original: The authors would like to thank Ergute Bao, Ta Duy Nguyen, and Martin Strobel for their valuable feedback on earlier versions of this paper, as well as the anonymous reviewers for their insightful comments.
Revised: The authors would like to thank Ergute Bao, Ta Duy Nguyen, and Martin Strobel for their valuable feedback on earlier versions of this paper, as well as the anonymous reviewers for their insightful comments.

Original: This research is supported by Google PDPO faculty research award, Intel within the www.private-ai.org center, Meta faculty research award, the NUS Early Career Research Award (NUS ECRA award num-
Revised: This research is supported by Google PDPO faculty research award, Intel within the www.private-ai.org center, Meta faculty research award, the NUS Early Career Research Award (NUS ECRA award num-

Original: ber NUS ECRA FY19 P16), and the National Research Foundation,
Revised: ber NUS ECRA FY19 P16), and the National Research Foundation,

Original: Singapore under its Strategic Capability Research Centres Funding Initiative.
Revised: Singapore under its Strategic Capability Research Centres Funding Initiative.

Original: Any opinions, ﬁndings, conclusions, or recommendations expressed in this material are those of the author(s) and do not reﬂect the views of the National Research Foundation, Singapore.
Revised: Any opinions, ﬁndings, conclusions, or recommendations expressed in this material are those of the author(s) and do not reﬂect the views of the National Research Foundation, Singapore.

Original:  E THICS S TATEMENT
Revised:  E THICS S TATEMENT

Original: Our analysis focuses on group fairness, which is typically used to audit the model or system for any bias or discrimination.
Revised: Our analysis focuses on group fairness, which is typically used to audit the model or system for any bias or discrimination.

Original: Auditing the bias with respect to other deﬁnitions of fairness, such as individual fairness, may result in different conclusions.
Revised: Auditing the bias with respect to other deﬁnitions of fairness, such as individual fairness, may result in different conclusions.

Original: Moreover, our study focuses primarily on the binary notion of sex attributes and the multi-valued race attribute.
Revised: Moreover, our study focuses primarily on the binary notion of sex attributes and the multi-valued race attribute.

Original: We recognize that there are numerous protected groups outside those considered in the analysis, such as those deﬁned by multiple sensitive attributes.
Revised: We recognize that there are numerous protected groups outside those considered in the analysis, such as those deﬁned by multiple sensitive attributes.

Original: The propagation of bias against ﬁne-grained subgroups may be even more substantial than we found in the paper.
Revised: The propagation of bias against ﬁne-grained subgroups may be even more substantial than we found in the paper.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: The code for the paper is available at https://github.com/privacytrustlab/bias_ in_FL .
Revised: We provide details about the model, datasets, and implementations in Appendix A, and the code for the paper is available at https://github.com/privacytrustlab/bias_in_FL .

Original: A PPENDIX
Revised: A PPENDIX

Original: This appendix is divided into ﬁve sections.
Revised: This appendix is divided into ﬁve sections.

Original: Appendix A provides additional details about the experimental setups, including information about the models, datasets, and hyperparameters used.
Revised: Appendix A provides additional details about the experimental setups, including information about the models, datasets, and hyperparameters used.

Original: Appendix B presents further experimental results that support the claims made in the paper.
Revised: Appendix B presents further experimental results that support the claims made in the paper.

Original: In
Revised: In

Original: Appendix C, we discuss the results of an existing fair FL algorithm.
Revised: Appendix C, we discuss the results of an existing fair FL algorithm.

Original: Appendix D extends our analysis to real-world medical datasets.
Revised: Appendix D extends our analysis to real-world medical datasets.

Original: Finally, in Appendix E, we discuss potential methods for mitigating bias in FL.
Revised: Finally, in Appendix E, we discuss potential methods for mitigating bias in FL.

Original: A D ETAILS ABOUT E XPERIMENT S ETUP A. D ATASETS AND M ODELS
Revised: A D ETAILS ABOUT E XPERIMENT S ETUP A. D ATASETS AND M ODELS

Original: We explain the datasets and models used in the paper
Revised: We explain the datasets and models used in the paper

Original: Census Dataset
Revised: Census Dataset

Original: We use the datasets provided by folktables.
Revised: We use the datasets provided by folktables.

Original: In particular, we consider the ACSIncome, ACSPublicCoverage, and ACSEmployment tasks deﬁned in the forlktables.
Revised: In particular, we consider the ACSIncome, ACSPublicCoverage, and ACSEmployment tasks deﬁned in the forlktables.

Original: In the ACSIncome, the goal is to predict whether an individual’s Income is above $50,000.
Revised: In the ACSIncome, the goal is to predict whether an individual’s Income is above $50,000.

Original: In the ACSEmployment task, the goal is to predict whether an individual is employed.
Revised: In the ACSEmployment task, the goal is to predict whether an individual is employed.

Original: Similarly, in the Health (i.e., ACSPublicCoverage) task, the objective is to predict whether an individual is covered by public health insurance.
Revised: Similarly, in the Health (i.e., ACSPublicCoverage) task, the objective is to predict whether an individual is covered by public health insurance.

Original: We use the same pre-processing as in the folktables and train a fully connected neural network model with one hidden layer of 32 neurons for Income and 64 neurons for Employment and Health tasks.
Revised: We use the same pre-processing as in the folktables and train a fully connected neural network model with one hidden layer of 32 neurons for Income and 64 neurons for Employment and Health tasks.

Original: For all the tasks, we use the RELU activation function.
Revised: For all the tasks, we use the RELU activation function.

Original: We use an SGD optimizer with a learning rate of 0.001 for centralized training on Health and Employment datasets and 0.1 for other settings, and the batch size is 32.
Revised: We use an SGD optimizer with a learning rate of 0.001 for centralized training on Health and Employment datasets and 0.1 for other settings, and the batch size is 32.

Original: We train the NN models for 200 epochs.
Revised: We train the NN models for 200 epochs.

Original: In FL, each client updates the global mdoel for 1 epoch and shares it with the server.
Revised: In FL, each client updates the global mdoel for 1 epoch and shares it with the server.

Original: We encode the categorical features based on the encoding template provided in folktables.
Revised: We encode the categorical features based on the encoding template provided in folktables.

Original: After the encoding, the input feature size for Income is 54, 154 for Health, and 109 for Employment.
Revised: After the encoding, the input feature size for Income is 54, 154 for Health, and 109 for Employment.

Original: We consider sex and race as sensitive attributes.
Revised: We consider sex and race as sensitive attributes.

Original: Accordingly, there are two gender groups (male and female) and nine racial groups ("White alone," "Black or African American alone," "American Indian alone," "Alaska Native alone," and "American
Revised: Accordingly, there are two gender groups (male and female) and nine racial groups ("White alone," "Black or African American alone," "American Indian alone," "Alaska Native alone," and "American

Original: Indian and Alaska Native tribes speciﬁed; or American Indian or Alaska Native, not speciﬁed and no other," "Asian alone," "Native Hawaiian and Other Paciﬁc Islander alone," "Some Other Race alone," "Two or More Races").
Revised: Indian and Alaska Native tribes speciﬁed; or American Indian or Alaska Native, not speciﬁed and no other," "Asian alone," "Native Hawaiian and Other Paciﬁc Islander alone," "Some Other Race alone," "Two or More Races").

Original: CelebA Dataset We train CNN models on the dataset with one CNN layer whose output channel is 32, kernel size is 3, and stride is 1.
Revised: CelebA Dataset We train CNN models on the dataset with one CNN layer whose output channel is 32, kernel size is 3, and stride is 1.

Original: We use ’same’ padding for the CNN layer.
Revised: We use ’same’ padding for the CNN layer.

Original: Following this
Revised: Following this

Original: CNN layer, we have the Batch normalization layer and Max Pooling layer.
Revised: CNN layer, we have the Batch normalization layer and Max Pooling layer.

Original: After which, we have the connected layer.
Revised: After which, we have the connected layer.

Original: We train the model for 500 communication rounds or epochs with SGD optimizer.
Revised: We train the model for 500 communication rounds or epochs with SGD optimizer.

Original: The learning rate is 0.1, and the batch size is 128.
Revised: The learning rate is 0.1, and the batch size is 128.

Original: The train, test, and validation datasets ratio for each party is 6:2:2.
Revised: The train, test, and validation datasets ratio for each party is 6:2:2.

Original: A. H YPER - PARAMETER FOR FL ALGORITHMS AND I MPLEMENTATION
Revised: A. H YPER - PARAMETER FOR FL ALGORITHMS AND I MPLEMENTATION

Original: In our paper, we also evaluate various popular FL algorithms, including FedNova (Wang et al., 2020),
Revised: In our paper, we also evaluate various popular FL algorithms, including FedNova (Wang et al., 2020),

Original: Scaffold Karimireddy et al.
Revised: Scaffold Karimireddy et al.

Original: (2020b), FedOpt (Reddi et al., 2020), FedProx Li et al.
Revised: (2020b), FedOpt (Reddi et al., 2020), FedProx Li et al.

Original: (2020), and
Revised: (2020), and

Original: Mime (Karimireddy et al., 2020a).
Revised: Mime (Karimireddy et al., 2020a).

Original: We use the same local learning rate and the number of the local epoch as in FedAvg.
Revised: We use the same local learning rate and the number of the local epoch as in FedAvg.

Original: We provided the detailed hyper-parameters for each of the algorithms in our code (See supplementary).
Revised: We provided the detailed hyper-parameters for each of the algorithms in our code (See supplementary).

Original: We run all experiments on Ubuntu with two NVIDIA TITAN RTX GPUs.
Revised: We run all experiments on Ubuntu with two NVIDIA TITAN RTX GPUs.

Original: A. D ATA H ETEROGENEITY
Revised: A. D ATA H ETEROGENEITY

Original: Figure 11 shows the fraction of samples for each subgroup across all parties, while Figure 12 shows the histogram of subgroup proportions for each party.
Revised: Figure 11 shows the fraction of samples for each subgroup across all parties, while Figure 12 shows the histogram of subgroup proportions for each party.

Original: We observe that the parties have different fractions of samples from each group in the Income dataset, indicating that their local data distributions are dissimilar.
Revised: We observe that the parties have different fractions of samples from each group in the Income dataset, indicating that their local data distributions are dissimilar.

Original: In contrast, for the Health and Employment datasets, parties have similar fractions of samples from each subgroup, suggesting that their data distributions are more alike than those of the
Revised: In contrast, for the Health and Employment datasets, parties have similar fractions of samples from each subgroup, suggesting that their data distributions are more alike than those of the

Original: Income dataset.
Revised: Income dataset.

Original: B A DDITIONAL E XPERIMENTAL R ESULTS B. B IAS PROPAGATION EFFECT
Revised: B A DDITIONAL E XPERIMENTAL R ESULTS B. B IAS PROPAGATION EFFECT

Original: Census dataset We show the bias propagation effect of FL for the Health and Employment task in
Revised: Census dataset We show the bias propagation effect of FL for the Health and Employment task in

Original: Figure 13 and 14 respectively.
Revised: Figure 13 and 14 respectively.

Original: CelebA We show the bias propagation effect of FL on CelebA dataset.
Revised: CelebA We show the bias propagation effect of FL on CelebA dataset.

Original: Besides partitioning the data in an IID manner (we refer to as setting (i)), we also evaluate two different settings, where we change the number of samples from the minority subgroup (the female group with the “Not Young” age label).
Revised: Besides partitioning the data in an IID manner (we refer to as setting (i)), we also evaluate two different settings, where we change the number of samples from the minority subgroup (the female group with the “Not Young” age label).

Original: In this way, we aim to change the data bias in the local training datasets for clients.
Revised: In this way, we aim to change the data bias in the local training datasets for clients.

Original: In particular, in setting (ii), half of the parties have more samples from the minority subgroup than another half of the parties.
Revised: In particular, in setting (ii), half of the parties have more samples from the minority subgroup than another half of the parties.

Original: The ratio is 8:2.
Revised: The ratio is 8:2.

Original: In setting (iii), a single party has half of the data from the minority subgroup, and the data is then iid partitioned among the other parties.
Revised: In setting (iii), a single party has half of the data from the minority subgroup, and the data is then iid partitioned among the other parties.

Original: In the ﬁgure, we ﬁnd that the in the non-iid setting, the more biased parties have a larger and more positive beneﬁt, while FL hurts the less biased parties.
Revised: In the ﬁgure, we ﬁnd that the in the non-iid setting, the more biased parties have a larger and more positive beneﬁt, while FL hurts the less biased parties.

Original: Figure 15 shows the correlation between the fairness gap of the standalone model and the beneﬁt a party gets in the FL.
Revised: Figure 15 shows the correlation between the fairness gap of the standalone model and the beneﬁt a party gets in the FL.

Original: We ﬁnd that the in the non-iid setting, the more biased parties have a larger and more positive beneﬁt, while FL hurts the less biased parties.
Revised: We ﬁnd that the in the non-iid setting, the more biased parties have a larger and more positive beneﬁt, while FL hurts the less biased parties.

Original: B. A GGREGATION CONTRADICTS WITH LOCAL UPDATE
Revised: B. A GGREGATION CONTRADICTS WITH LOCAL UPDATE

Original: Figure 16 shows the dynamic of the fairness gap for the aggregated model and locally updated model from the most biased party and least biased party during the training.
Revised: Figure 16 shows the dynamic of the fairness gap for the aggregated model and locally updated model from the most biased party and least biased party during the training.

Original: B. I NFLUENCE S UB - GRAPHS
Revised: B. I NFLUENCE S UB - GRAPHS

Original: We show the top 10 maximal positive and top 10 maximal native inﬂuence client pairs in Figureand the top 15 in Figure 18.
Revised: We show the top 10 maximal positive and top 10 maximal native inﬂuence client pairs in Figureand the top 15 in Figure 18.

Original: We can see that the less biased client has a strong positive inﬂuence on other clients while the more biased client has a strong negative inﬂuence on other clients
Revised: We can see that the less biased client has a strong positive inﬂuence on other clients while the more biased client has a strong negative inﬂuence on other clients

Original: . B. A TTRIBUTION VALUE FOR SENSITIVE ATTRIBUTE
Revised: . B. A TTRIBUTION VALUE FOR SENSITIVE ATTRIBUTE

Original: We show the attribution value for all input features for the most biased party and least biased party in Figure 19 and Figure 20, respectively.
Revised: We show the attribution value for all input features for the most biased party and least biased party in Figure 19 and Figure 20, respectively.

Original: We observe that the attribution value for other features is similar for female and male groups.
Revised: We observe that the attribution value for other features is similar for female and male groups.

Original: However, the sex attribute has the largest attribution value and affects the groups differently.
Revised: However, the sex attribute has the largest attribution value and affects the groups differently.

Original: It implies that the models are highly dependent on the sensitive attribute (i.e., "Sex") for making predictions.
Revised: It implies that the models are highly dependent on the sensitive attribute (i.e., "Sex") for making predictions.

Original: B. E FFECT OF FL ALGORITHM
Revised: B. E FFECT OF FL ALGORITHM

Original: Table 2 shows results on other FL algorithms, including FedNova (Wang et al., 2020),
Revised: Table 2 shows results on other FL algorithms, including FedNova (Wang et al., 2020),

Original: Scaffold Karimireddy et al.
Revised: Scaffold Karimireddy et al.

Original: (2020b), FedOpt (Asad et al., 2020), FedProx Li et al.
Revised: (2020b), FedOpt (Asad et al., 2020), FedProx Li et al.

Original: (2020), and Mime (Karimireddy et al., 2020a).
Revised: (2020), and Mime (Karimireddy et al., 2020a).

Original: We ﬁnd that when the FL model achieves higher accuracy than standalone models, the fairness gap of the FL model can be higher than that of standalone models.
Revised: We ﬁnd that when the FL model achieves higher accuracy than standalone models, the fairness gap of the FL model can be higher than that of standalone models.

Original: This observation is consistent across multiple FL algorithms.
Revised: This observation is consistent across multiple FL algorithms.

Original: This strong evidence implies that the improvement of accuracy in FL can come at the cost of fairness.
Revised: This strong evidence implies that the improvement of accuracy in FL can come at the cost of fairness.

Original: In addition, this is not unique to only the FedAvg algorithm.
Revised: In addition, this is not unique to only the FedAvg algorithm.

Original: B. G ROUP P ERFORMANCE
Revised: B. G ROUP P ERFORMANCE

Original: In Figure 21, we present the model ROC on groups to illustrate the performance disparity across groups.
Revised: In Figure 21, we present the model ROC on groups to illustrate the performance disparity across groups.

Original: Surprisingly, there is no huge difference between the ROC between groups.
Revised: Surprisingly, there is no huge difference between the ROC between groups.

Original: In contrast,
Revised: In contrast,

Original: Figure 22 shows the noticeable difference between groups with respect to the precision-recall curve, especially for the most biased party.
Revised: Figure 22 shows the noticeable difference between groups with respect to the precision-recall curve, especially for the most biased party.

Original: The main reason is that the dataset is imbalanced.
Revised: The main reason is that the dataset is imbalanced.

Original: Thus, ROC is usually misleading.
Revised: Thus, ROC is usually misleading.

Original: On the precision-recall curve, we ﬁnd that the model achieves a higher precision on the majority (i.e., Male group) compared to the minority group (i.e., Female group).
Revised: On the precision-recall curve, we ﬁnd that the model achieves a higher precision on the majority (i.e., Male group) compared to the minority group (i.e., Female group).

Original: [Table]
Revised: [Table]

Original: B.
Revised: B.

Original: E FFECT OF SCALING OTHER PARAMETERS
Revised: E FFECT OF SCALING OTHER PARAMETERS

Original: Figure 23 shows a comparison of the model’s performance when scaling different model parameters.
Revised: Figure 23 shows a comparison of the model’s performance when scaling different model parameters.

Original: Speciﬁcally, we examine the impact of scaling the parameters in the ﬁrst layer of the neural network that does not have any computation on the Sex attribute (referred to as "Other parameters"), as well as the parameter related to the Sex attribute (referred to as "Related parameters").
Revised: Speciﬁcally, we examine the impact of scaling the parameters in the ﬁrst layer of the neural network that does not have any computation on the Sex attribute (referred to as "Other parameters"), as well as the parameter related to the Sex attribute (referred to as "Related parameters").

Original: The results indicate that up-scaling or down-scaling other parameters do not signiﬁcantly affect the fairness gap while scaling the related parameters has a considerable impact on model fairness.
Revised: The results indicate that up-scaling or down-scaling other parameters do not signiﬁcantly affect the fairness gap while scaling the related parameters has a considerable impact on model fairness.

Original: This ﬁnding supports our hypothesis that the model’s bias is primarily encoded in a few model parameters.
Revised: This ﬁnding supports our hypothesis that the model’s bias is primarily encoded in a few model parameters.

Original: In FL, biased parties introduce bias during local training by increasing the weights for those related parameters.
Revised: In FL, biased parties introduce bias during local training by increasing the weights for those related parameters.

Original: [Table]
Revised: [Table]

Original: C E VALUATION ON EXISTING F AIR FL
Revised: C E VALUATION ON EXISTING F AIR FL

Original: Table 3 presents the results for a fair FL algorithm proposed by Abay et al.
Revised: Table 3 presents the results for a fair FL algorithm proposed by Abay et al.

Original: (2020), which uses a reweighting algorithm proposed in the centralized setting (Kamiran and Calders, 2012).
Revised: (2020), which uses a reweighting algorithm proposed in the centralized setting (Kamiran and Calders, 2012).

Original: The algorithm involves assigning weights to each subgroup (deﬁned by the label and a sensitive attribute) before FL, based on the local or global training dataset.
Revised: The algorithm involves assigning weights to each subgroup (deﬁned by the label and a sensitive attribute) before FL, based on the local or global training dataset.

Original: This is called "local reweighting" or "global reweighting," respectively.
Revised: This is called "local reweighting" or "global reweighting," respectively.

Original: During FL, parties update the FL model to minimize the weighted loss.
Revised: During FL, parties update the FL model to minimize the weighted loss.

Original: The table presents the accuracy and fairness gap of the FL algorithm using local and global reweighting for the Income, Health, and Employment datasets.
Revised: The table presents the accuracy and fairness gap of the FL algorithm using local and global reweighting for the Income, Health, and Employment datasets.

Original: We found that applying reweighing algorithms in FL reduces the average fairness gap across parties.
Revised: We found that applying reweighing algorithms in FL reduces the average fairness gap across parties.

Original: This is due to the fact that, after reweighing, the parties that were initially biased do not introduce signiﬁcant bias to the model during local training, resulting in a reduction of the fairness gap in the
Revised: This is due to the fact that, after reweighing, the parties that were initially biased do not introduce signiﬁcant bias to the model during local training, resulting in a reduction of the fairness gap in the

Original: FL model.
Revised: FL model.

Original: However, we also noted that the fairness gap in the model remains high for the Race sensitive attribute, indicating that the global and local reweighing algorithms do not entirely eliminate bias in
Revised: However, we also noted that the fairness gap in the model remains high for the Race sensitive attribute, indicating that the global and local reweighing algorithms do not entirely eliminate bias in

Original: FL.
Revised: FL.

Original: Furthermore, we would like to highlight some concerns regarding the fair FL algorithm:
Revised: Furthermore, we would like to highlight some concerns regarding the fair FL algorithm:

Original: The fair FL algorithm enhances the performance of the minority group at the expense of the majority group.
Revised: The fair FL algorithm enhances the performance of the minority group at the expense of the majority group.

Original: Figure 24 illustrates that the minority group (i
Revised: Figure 24 illustrates that the minority group (i

Original: e., the Female group) has a better performance after the reweighing, but at the cost of reduced performance for the majority.
Revised: e., the Female group) has a better performance after the reweighing, but at the cost of reduced performance for the majority.

Original: When improving fairness with respect to one sensitive attribute, the fairness issue with respect to another sensitive attribute may worsen, as shown in Table 3.
Revised: When improving fairness with respect to one sensitive attribute, the fairness issue with respect to another sensitive attribute may worsen, as shown in Table 3.

Original: For instance, when applying local reweighing to reduce bias with respect to the "Sex" attribute, the fairness gap with respect to "Race" increased from 0.514 (results on FedAvg) to 0.521, indicating that reweighing to improve fairness with respect to Sex may amplify the bias with respect to Race.
Revised: For instance, when applying local reweighing to reduce bias with respect to the "Sex" attribute, the fairness gap with respect to "Race" increased from 0.514 (results on FedAvg) to 0.521, indicating that reweighing to improve fairness with respect to Sex may amplify the bias with respect to Race.

Original: This implies that the bias with respect to Race can still propagate in FL, and our analysis remains valid.
Revised: This implies that the bias with respect to Race can still propagate in FL, and our analysis remains valid.

Original: The reweighing algorithms assume that parties are interested in mitigating the bias with respect to the same sensitive attribute, which may not be the case in practice.
Revised: The reweighing algorithms assume that parties are interested in mitigating the bias with respect to the same sensitive attribute, which may not be the case in practice.

Original: Figureshows the fairness gap of standalone models with respect to different sensitive attributes.
Revised: Figureshows the fairness gap of standalone models with respect to different sensitive attributes.

Original: We observe that many parties have a low fairness gap with respect to one sensitive attribute and a high fairness gap with respect to another sensitive attribute.
Revised: We observe that many parties have a low fairness gap with respect to one sensitive attribute and a high fairness gap with respect to another sensitive attribute.

Original: Thus, parties may aim to enhance fairness with respect to different sensitive attributes, which renders this reweighing algorithm unsuitable.
Revised: Thus, parties may aim to enhance fairness with respect to different sensitive attributes, which renders this reweighing algorithm unsuitable.

Original: This problem is more prevalent in FL, given the different data distributions of
Revised: This problem is more prevalent in FL, given the different data distributions of

Original: parties.
Revised: parties.

Original: [Table]
Revised: [Table]

Original: D E XTENDING TO A REAL - WORLD MEDICAL DATASET
Revised: D E XTENDING TO A REAL - WORLD MEDICAL DATASET

Original: We evaluate the effect of FL on local fairness on a real-world medical dataset, ISIC

Revised: We evaluate the effect of FL on local fairness on a real-world medical dataset, ISIC

Original: (Abay et al., 2020;
Revised: (Abay et al., 2020;

Original: Tschandl et al., 2018; Combalia et al., 2019), which contains dermoscopic images of skin lesions collected from six medical centers.
Revised: Tschandl et al., 2018; Combalia et al., 2019), which contains dermoscopic images of skin lesions collected from six medical centers.

Original: The task is to classify dermoscopic images among nine different diagnostic categories.
Revised: The task is to classify dermoscopic images among nine different diagnostic categories.

Original: We ﬁlter out medical centers with less than 1900 images and regard each medical center with over 1900 images as a party (4 centers remain after ﬁltering).
Revised: We ﬁlter out medical centers with less than 1900 images and regard each medical center with over 1900 images as a party (4 centers remain after ﬁltering).

Original: For every party, we randomly select 1500 and 400 images for training and validation, respectively.
Revised: For every party, we randomly select 1500 and 400 images for training and validation, respectively.

Original: We regard the binary notion of Sex as our sensitive attribute.
Revised: We regard the binary notion of Sex as our sensitive attribute.

Original: In this multi-class and medical dataset setting, we measure the bias (fairness gap) based on the accuracy gap across groups.
Revised: In this multi-class and medical dataset setting, we measure the bias (fairness gap) based on the accuracy gap across groups.

Original: Following (Ogier du
Revised: Following (Ogier du

Original: Terrail et al., 2022) 2 , we train EfﬁcientNets
Revised: Terrail et al., 2022) 2 , we train EfﬁcientNets

Original: (Tan and Le, 2019), the state-of-the-art model structure for medical data, under the centralized, FL, and standalone setting.
Revised: (Tan and Le, 2019), the state-of-the-art model structure for medical data, under the centralized, FL, and standalone setting.

Original: The accuracy and fairness gap for each client in different settings are shown in Table 4.
Revised: The accuracy and fairness gap for each client in different settings are shown in Table 4.

Original: FedAvg does not improve accuracy, possibly due to data heterogeneity among parties (as shown in
Revised: FedAvg does not improve accuracy, possibly due to data heterogeneity among parties (as shown in

Original: Figure 26), resulting in the model’s instability and slow convergence, which is known as ‘client-drift’ issue (Karimireddy et al., 2020b).
Revised: Figure 26), resulting in the model’s instability and slow convergence, which is known as ‘client-drift’ issue (Karimireddy et al., 2020b).

Original: Moreover, the FedAvg model results in a higher accuracy gap between groups, worsening the fairness issue compared to the standalone setting or the centralized setting (see the fairness gaps for party 1, party 3, and party 4 as examples).
Revised: Moreover, the FedAvg model results in a higher accuracy gap between groups, worsening the fairness issue compared to the standalone setting or the centralized setting (see the fairness gaps for party 1, party 3, and party 4 as examples).

Original: Our results suggest that, in real-world settings, FedAvg may not improve accuracy compared to the standalone setting for the local data distribution and may even exacerbate the fairness issue, leading to a more biased model.
Revised: Our results suggest that, in real-world settings, FedAvg may not improve accuracy compared to the standalone setting for the local data distribution and may even exacerbate the fairness issue, leading to a more biased model.

Original: E P OTENTIAL M ITIGATION
Revised: E P OTENTIAL M ITIGATION

Original: In this section, we suggest some potential solutions to reduce the effect of bias propagation in FL.
Revised: In this section, we suggest some potential solutions to reduce the effect of bias propagation in FL.

Original: Our hope is that this will inspire further research on developing fair FL in the future.
Revised: Our hope is that this will inspire further research on developing fair FL in the future.

Original: Personalized FL
Revised: Personalized FL

Original: Our study has revealed that aggregate updates, which involve the average local update from all parties, may result in a higher bias than local updates for less biased parties (see
Revised: Our study has revealed that aggregate updates, which involve the average local update from all parties, may result in a higher bias than local updates for less biased parties (see

Original: Figure 3).
Revised: Figure 3).

Original: This can cause bias from other parties to propagate to the local model, thus increasing the local fairness gap.
Revised: This can cause bias from other parties to propagate to the local model, thus increasing the local fairness gap.

Original: To mitigate the bias propagation effect, one possible approach is to avoid completely overwriting the local model with the global model.
Revised: To mitigate the bias propagation effect, one possible approach is to avoid completely overwriting the local model with the global model.

Original: Personalized Federated Learning (PFL) techniques (Li et al., 2021) have recently been proposed to address this issue by focusing on improving the model’s accuracy.
Revised: Personalized Federated Learning (PFL) techniques (Li et al., 2021) have recently been proposed to address this issue by focusing on improving the model’s accuracy.

Original: PFL can be a promising direction for developing fairness-aware personalized federated learning approaches that can help reduce the bias propagation effect in FL.
Revised: PFL can be a promising direction for developing fairness-aware personalized federated learning approaches that can help reduce the bias propagation effect in FL.

Original: Fair Representation Learning Our analysis demonstrated that bias is encoded in a small number of parameters in the ﬁrst layer of the neural network, which is typically considered the feature extractor (as illustrated in Figure 10).
Revised: Fair Representation Learning Our analysis demonstrated that bias is encoded in a small number of parameters in the ﬁrst layer of the neural network, which is typically considered the feature extractor (as illustrated in Figure 10).

Original: This suggests that the FL model is biased because the learned representation is biased, heavily inﬂuenced by the sensitive attribute.
Revised: This suggests that the FL model is biased because the learned representation is biased, heavily inﬂuenced by the sensitive attribute.

Original: To address this issue, one possible solution is to implement fair representation learning techniques (Zemel et al., 2013;
Revised: To address this issue, one possible solution is to implement fair representation learning techniques (Zemel et al., 2013;

Original: Zhao et al., 2019; Liu et al., 2022; Zeng et al., 2021b)
Revised: Zhao et al., 2019; Liu et al., 2022; Zeng et al., 2021b)

Original: that aim to learn a feature extractor which is fair, meaning that it minimizes the dependence on the sensitive attribute while still retaining enough information about the task (or input features).
Revised: that aim to learn a feature extractor which is fair, meaning that it minimizes the dependence on the sensitive attribute while still retaining enough information about the task (or input features).

Original: Therefore, using fair representation learning in FL can potentially mitigate the propagation of bias effects.
Revised: Therefore, using fair representation learning in FL can potentially mitigate the propagation of bias effects.