Original: F ED BABU: T OWARD E NHANCED R EPRESENTATION FOR F EDERATED I MAGE C LASSIFICATION
Revised: F ED BABU: T OWARD E NHANCED R EPRESENTATION FOR F EDERATED I MAGE C LASSIFICATION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Federated learning has evolved to improve a single global model under data heterogeneity (as a curse) or to develop multiple personalized models using data heterogeneity (as a blessing).
Revised: Federated learning has evolved to improve a single global model under data heterogeneity (as a curse) or to develop multiple personalized models using data heterogeneity (as a blessing).

Original: However, little research has considered both directions simultaneously.
Revised: However, little research has considered both directions simultaneously.

Original: In this paper, we first investigate the relationship between them by analyzing Federated Averaging (McMahan et al., 2017) at the client level and determine that a better federated global model performance does not constantly improve personalization.
Revised: In this paper, we first investigate the relationship between them by analyzing Federated Averaging (McMahan et al., 2017) at the client level and determine that a better federated global model performance does not constantly improve personalization.

Original: To elucidate the cause of this personalization performance degradation problem, we decompose the entire network into the body (extractor), which is related to universality, and the head (classifier), which is related to personalization.
Revised: To elucidate the cause of this personalization performance degradation problem, we decompose the entire network into the body (extractor), which is related to universality, and the head (classifier), which is related to personalization.

Original: We then point out that this problem stems from training the head.
Revised: We then point out that this problem stems from training the head.

Original: Based on this observation, we propose a novel federated learning algorithm, coined FedBABU, which only updates the body of the model during federated training (the head is randomly initialized and never updated), and the head is fine-tuned for personalization during the evaluation process.
Revised: Based on this observation, we propose a novel federated learning algorithm, coined FedBABU, which only updates the body of the model during federated training (i.e., the head is randomly initialized and never updated), and the head is fine-tuned for personalization during the evaluation process.

Original: Extensive experiments show consistent performance improvements and an efficient personalization of FedBABU.
Revised: Extensive experiments show consistent performance improvements and an efficient personalization of FedBABU.

Original: The code is available at https://anonymous.4open.science/r/ICLR2022-FedBABU .
Revised: The code is available at https://github.com/jhoon-oh/FedBABU .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Federated learning (FL) (McMahan et al., 2017), which is a distributed learning framework with personalized data, has become an attractive field of research.
Revised: Federated learning (FL) (McMahan et al., 2017), which is a distributed learning framework with personalized data, has become an attractive field of research.

Original: From the early days of this field, improving a single global model across devices has been the main objective (Zhao et al., 2018;
Revised: From the early days of this field, improving a single global model across devices has been the main objective (Zhao et al., 2018;

Original: Duan et al., 2019; Li et al., 2018; Acar et al., 2021), where the global model suffers from data heterogeneity.
Revised: Duan et al., 2019; Li et al., 2018; Acar et al., 2021), where the global model suffers from data heterogeneity.

Original: Many researchers have recently focused on multiple personalized models by leveraging data heterogeneity across devices as a blessing in disguise (Chen et al., 2018;
Revised: Many researchers have recently focused on multiple personalized models by leveraging data heterogeneity across devices as a blessing in disguise (Chen et al., 2018;

Original: Dinh et al., 2021; Zhang et al., 2020; Fallah et al., 2020; Shamsian et al., 2021; Smith et al., 2017).
Revised: Dinh et al., 2021; Zhang et al., 2020; Fallah et al., 2020; Shamsian et al., 2021; Smith et al., 2017).

Original: Although many studies have been conducted on each research line, a lack of research remains on how to train a good global model for personalization purposes (Ji et al., 2021; Jiang et al., 2019).
Revised: Although many studies have been conducted on each research line, a lack of research remains on how to train a good global model for personalization purposes (Ji et al., 2021; Jiang et al., 2019).

Original: In this study, for personalized training, each local client model starts from a global model that learns information from all clients and leverages the global model to fit its own data distribution.
Revised: In this study, for personalized training, each local client model starts from a global model that learns information from all clients and leverages the global model to fit its own data distribution.

Original: Jiang et al.
Revised: Jiang et al.

Original: (2019) analyzed personalization methods that adapt to the global model through finetuning on each device.
Revised: (2019) analyzed personalization methods that adapt to the global model through finetuning on each device.

Original: They observed that the effects of fine-tuning are encouraging and that training in a central location increases the initial accuracy (of a single global model) while decreasing the personalized accuracy (of on-device fine-tuned models).
Revised: They observed that the effects of fine-tuning are encouraging and that training in a central location increases the initial accuracy (of a single global model) while decreasing the personalized accuracy (of on-device fine-tuned models).

Original: We focus on why opposite changes in the two performances appear.
Revised: We focus on why opposite changes in the two performances appear.

Original: This suggests that the factors for universality and personalization must be dealt with separately, inspiring us to decouple the entire network into the body (i.e., extractor) related to generality and the head (i.e., classifier) related to specialty, as in previous studies (Kang et al., 2019;
Revised: This suggests that the factors for universality and personalization must be dealt with separately, inspiring us to decouple the entire network into the body (i.e., extractor) related to generality and the head (i.e., classifier) related to specialty, as in previous studies (Kang et al., 2019;

Original: Yu et al., 2020;
Revised: Yu et al., 2020;

Original: Yosinski et al., 2014; Devlin et al., 2019) for advanced analysis.
Revised: Yosinski et al., 2014; Devlin et al., 2019) for advanced analysis.

Original: Kang et al.
Revised: Kang et al.

Original: (2019) and Yu et al.
Revised: (2019) and Yu et al.

Original: (2020) demonstrated that the head is biased in class-imbalanced environments.
Revised: (2020) demonstrated that the head is biased in class-imbalanced environments.

Original: Note that popular networks such as MobileNet (Howard et al., 2017) and ResNet (He et al., 2016) have only one linear layer at the end of the model, and the head is defined as this linear layer whereas the body is defined as all of the layers except the head.
Revised: Note that popular networks such as MobileNet (Howard et al., 2017) and ResNet (He et al., 2016) have only one linear layer at the end of the model, and the head is defined as this linear layer whereas the body is defined as all of the layers except the head.

Original: The body of the model is related to representation learning and the head of the model is related to linear decision boundary learning.
Revised: The body of the model is related to representation learning and the head of the model is related to linear decision boundary learning.

Original: We shed light on the cause of the personalization performance degradation by decoupling parameters, pointing out that such a problem stems from training the head.
Revised: We shed light on the cause of the personalization performance degradation by decoupling parameters, pointing out that such a problem stems from training the head

Original: Client
Revised: Client

Original: Client Client
Revised: Client Client

Original: Central Server
Revised: Central Server

Original: Client Client
Revised: Client Client

Original: Client
Revised: Client

Original: Central Server Central Server
Revised: Central Server Central Server

Original: Client
Revised: Client

Original: Client Client
Revised: Client Client

Original: Broadcasting
Revised: Broadcasting

Original: Local Update
Revised: Local Update

Original: Aggregation
Revised: Aggregation

Original: Different classifiers (a) FedAvg.
Revised: Different classifiers (a) FedAvg.

Original: Client
Revised: Client

Original: Client Client
Revised: Client Client

Original: Central Server Central Server
Revised: Central Server Central Server

Original: Client Client
Revised: Client Client

Original: Client
Revised: Client

Original: Same classifier
Revised: Same classifier

Original: Broadcasting
Revised: Broadcasting

Original: Local Update
Revised: Local Update

Original: Aggregation
Revised: Aggregation

Original: [Figure]
Revised: [Figure]

Original: In the figure, the lines represent the decision boundaries defined by the head (i.e., the last linear classifier) of the network; different shapes indicate different classes.
Revised: In the figure, the lines represent the decision boundaries defined by the head (i.e., the last linear classifier) of the network; different shapes indicate different classes.

Original: It is assumed that each client has two classes.
Revised: It is assumed that each client has two classes.

Original: (a) FedAvg updates the entire network during local updates on each client and then the local networks are aggregated entirely.
Revised: (a) FedAvg updates the entire network during local updates on each client and then the local networks are aggregated entirely.

Original: Therefore, the heads of all clients and the head of the server are different.
Revised: Therefore, the heads of all clients and the head of the server are different.

Original: Whereas, (b) FedBABU only updates the body (i.e., all of the layers except the head) during local updates on each client and then the local networks are aggregated body-partially.
Revised: Whereas, (b) FedBABU only updates the body (i.e., all of the layers except the head) during local updates on each client and then the local networks are aggregated body-partially.

Original: Therefore, the heads of all clients and the head of the server are the same.
Revised: Therefore, the heads of all clients and the head of the server are the same.

Original: Inspired by the above observations, we propose an algorithm to learn a single global model that can be efficiently personalized by simply changing the Federated Averaging (FedAvg) (McMahan et al., 2017) algorithm.
Revised: Inspired by the above observations, we propose an algorithm to learn a single global model that can be efficiently personalized by simply changing the Federated Averaging (FedAvg) (McMahan et al., 2017) algorithm.

Original: FedAvg consists of four stages: client sampling, broadcasting, local update, and aggregation.
Revised: FedAvg consists of four stages: client sampling, broadcasting, local update, and aggregation.

Original: In the client sampling stage, clients are sampled because the number of clients is so large that not all clients can participate in each round.
Revised: In the client sampling stage, clients are sampled because the number of clients is so large that not all clients can participate in each round.

Original: In the broadcasting stage, the server sends a global model (i.e., an initial random model at the first broadcast stage or an aggregated model afterward) to participating clients.
Revised: In the broadcasting stage, the server sends a global model (i.e., an initial random model at the first broadcast stage or an aggregated model afterward) to participating clients.

Original: In the local update stage, the broadcast model of each device is trained based on its own data set.
Revised: In the local update stage, the broadcast model of each device is trained based on its own data set.

Original: In the aggregation stage, locally updated models are sent to the server and are aggregated by averaging.
Revised: In the aggregation stage, locally updated models are sent to the server and are aggregated by averaging.

Original: Among the four stages, we focus on the local update stage from both the universality and personalization perspectives.
Revised: Among the four stages, we focus on the local update stage from both the universality and personalization perspectives.

Original: Here, we only update the body of the model in the local update stage, i.e., the head is never updated during federated training.
Revised: Here, we only update the body of the model in the local update stage, i.e., the head is never updated during federated training.

Original: From this, we propose
Revised: From this, we propose

Original: FedBABU, Fed erated Averaging with B ody A ggregation and B ody U pdate.
Revised: FedBABU, Fed erated Averaging with B ody A ggregation and B ody U pdate.

Original: Figure 1 describes the difference during the local update and aggregation stages between FedAvg and FedBABU.
Revised: Figure 1 describes the difference during the local update and aggregation stages between FedAvg and FedBABU.

Original: Intuitively, the fixed head can be interpreted as the criteria or guideline for each class and our approach is representation learning based on the same fixed criteria across all clients during federated training .
Revised: Intuitively, the fixed head can be interpreted as the criteria or guideline for each class and our approach is representation learning based on the same fixed criteria across all clients during federated training .

Original: This simple change improves the representation power of a single global model and enables the more accurate and rapid personalization of the trained global model than FedAvg.
Revised: This simple change improves the representation power of a single global model and enables the more accurate and rapid personalization of the trained global model than FedAvg.

Original: Our contributions are summarized as follows:
Revised: Our contributions are summarized as follows:

Original: • We investigate the connection between a single global model and fine-tuned personalized models by analyzing the FedAvg algorithm at the client level and show that training the head using shared data on the server negatively impacts personalization.
Revised: • We investigate the connection between a single global model and fine-tuned personalized models by analyzing the FedAvg algorithm at the client level and show that training the head using shared data on the server negatively impacts personalization.

Original: • We demonstrate that the fixed random head can have comparable performance to the learned head in the centralized setting.
Revised: • We demonstrate that the fixed random head can have comparable performance to the learned head in the centralized setting.

Original: From this observation, we suggest that sharing the fixed random head across all clients can be more potent than averaging each learned head in federated settings.
Revised: From this observation, we suggest that sharing the fixed random head across all clients can be more potent than averaging each learned head in federated settings.

Original: • We propose a novel algorithm, FedBABU , that reduces the update and aggregation parts from the entire model to the body of the model during federated training.
Revised: • We propose a novel algorithm, FedBABU , that reduces the update and aggregation parts from the entire model to the body of the model during federated training.

Original: We show that FedBABU is efficient, particularly under more significant data heterogeneity.
Revised: We show that FedBABU is efficient, particularly under more significant data heterogeneity.

Original: Furthermore, a single global model trained with the FedBABU algorithm can be personalized rapidly (even with one fine-tuning epoch).
Revised: Furthermore, a single global model trained with the FedBABU algorithm can be personalized rapidly (even with one fine-tuning epoch).

Original: We observe that FedAvg outperforms most of preexisting personalization FL algorithms and FedBABU outperforms FedAvg in various FL settings.
Revised: We observe that FedAvg outperforms most of preexisting personalization FL algorithms and FedBABU outperforms FedAvg in various FL settings.

Original: • We adapt the body update and body aggregation idea to the regularization-based global federated learning algorithm (such as FedProx (Li et al., 2018)).
Revised: • We adapt the body update and body aggregation idea to the regularization-based global federated learning algorithm (such as FedProx (Li et al., 2018)).

Original: We show that regularization reduces the personalization capabilities and that this problem is mitigated through BABU.
Revised: We show that regularization reduces the personalization capabilities and that this problem is mitigated through BABU.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: FL for a Single Global Model Canonical federated learning, FedAvg proposed by McMahan et al.
Revised: FL for a Single Global Model Canonical federated learning, FedAvg proposed by McMahan et al.

Original: (2017), aims to learn a single global model that collects the benefits of affluent data without storing the clients’ raw data in a central server, thus reducing the communication costs through local updates.
Revised: (2017), aims to learn a single global model that collects the benefits of affluent data without storing the clients’ raw data in a central server, thus reducing the communication costs through local updates.

Original: However, it is difficult to develop a globally optimal model for non-independent and identically distributed (non-IID) data derived from various clients.
Revised: However, it is difficult to develop a globally optimal model for non-independent and identically distributed (non-IID) data derived from various clients.

Original: To solve this problem, studies have been conducted that make the data distribution of the clients IID-like or add regularization to the distance from the global model during local updates.
Revised: To solve this problem, studies have been conducted that make the data distribution of the clients IID-like or add regularization to the distance from the global model during local updates.

Original: Zhao et al.
Revised: Zhao et al.

Original: (2018) suggested that all clients share a subset of public data, and Duan et al.
Revised: (2018) suggested that all clients share a subset of public data, and Duan et al.

Original: (2019) augmented data to balance the label distribution of clients.
Revised: (2019) augmented data to balance the label distribution of clients.

Original: Recently, two studies (Li et al., 2018; Acar et al., 2021) penalized local models that have a large divergence from the global model, adding a regularization term to the local optimization process and allowing the global model to converge more reliably.
Revised: Recently, two studies (Li et al., 2018; Acar et al., 2021) penalized local models that have a large divergence from the global model, adding a regularization term to the local optimization process and allowing the global model to converge more reliably.

Original: However, it should be noted that the single global model trained using the above methods is not optimized for each client.
Revised: However, it should be noted that the single global model trained using the above methods is not optimized for each client.

Original: Personalized FL Personalized federated learning aims to learn personalized local models that are stylized to each client.
Revised: Personalized FL Personalized federated learning aims to learn personalized local models that are stylized to each client.

Original: Although local models can be developed without federation, this method suffers from data limitations.
Revised: Although local models can be developed without federation, this method suffers from data limitations.

Original: Therefore, to maintain the benefits of the federation and personalized models, many other methods have been applied to FL: clustering, multi-task learning, transfer learning, regularized loss function, and meta-learning.
Revised: Therefore, to maintain the benefits of the federation and personalized models, many other methods have been applied to FL: clustering, multi-task learning, transfer learning, regularized loss function, and meta-learning.

Original: Two studies (Briggs et al., 2020; Mansour et al., 2020) clustered similar clients to match the data distribution within a cluster and learned separate models for each cluster without inter-cluster federation.
Revised: Two studies (Briggs et al., 2020; Mansour et al., 2020) clustered similar clients to match the data distribution within a cluster and learned separate models for each cluster without inter-cluster federation.

Original: Similar to clustering, multi-task learning aims to learn models for related clients simultaneously.
Revised: Similar to clustering, multi-task learning aims to learn models for related clients simultaneously.

Original: Note that clustering ties related clients into a single cluster, whereas multi-task learning does not.
Revised: Note that clustering ties related clients into a single cluster, whereas multi-task learning does not.

Original: The generalization of each model can be improved by sharing representations between related clients.
Revised: The generalization of each model can be improved by sharing representations between related clients.

Original: Smith et al.
Revised: Smith et al.

Original: (2017) and Dinh et al.
Revised: (2017) and Dinh et al.

Original: (2021) showed that multi-task learning is an appropriate learning scheme for personalized
Revised: (2021) showed that multi-task learning is an appropriate learning scheme for personalized

Original: FL.
Revised: FL.

Original: Transfer learning is also a recommended learning scheme because it aims to deliver knowledge among clients.
Revised: Transfer learning is also a recommended learning scheme because it aims to deliver knowledge among clients.

Original: In addition, Yang et al.
Revised: In addition, Yang et al.

Original: and
Revised: and

Original: Chen et al.
Revised: Chen et al.

Original: (2020b) utilized transfer learning to enhance local models by transferring knowledge between related clients.
Revised: (2020b) utilized transfer learning to enhance local models by transferring knowledge between related clients.

Original: T Dinh et al.
Revised: T Dinh et al.

Original: (2020) and Li et al.
Revised: (2020) and Li et al.

Original: (2021) added a regularizer toward the global or average personalized model to prevent clients’ models from simply overfitting their own dataset.
Revised: (2021) added a regularizer toward the global or average personalized model to prevent clients’ models from simply overfitting their own dataset.

Original: Unlike the aforementioned methods that developed local models during training, Chen et al.
Revised: Unlike the aforementioned methods that developed local models during training, Chen et al.

Original: (2018) and Fallah et al.
Revised: (2018) and Fallah et al.

Original: (2020) attempted to develop a good initialized shared global model using bi-level optimization through a Model-Agnostic Meta-Learning (MAML) approach (Finn et al., 2017).
Revised: (2020) attempted to develop a good initialized shared global model using bi-level optimization through a Model-Agnostic Meta-Learning (MAML) approach (Finn et al., 2017).

Original: A well-initialized model can be personalized through updates on each client (such as inner updates in MAML).
Revised: A well-initialized model can be personalized through updates on each client (such as inner updates in MAML).

Original: Jiang et al.
Revised: Jiang et al.

Original: (2019) argued that the FedAvg algorithm could be interpreted as a meta-learning algorithm, and a personalized local model with high accuracy can be obtained through fine-tuning from a global model learned using FedAvg.
Revised: (2019) argued that the FedAvg algorithm could be interpreted as a meta-learning algorithm, and a personalized local model with high accuracy can be obtained through fine-tuning from a global model learned using FedAvg.

Original: In addition, various technologies and algorithms for personalized FL have been presented and will be helpful to read; see
Revised: In addition, various technologies and algorithms for personalized FL have been presented and will be helpful to read; see

Original: Tan et al.
Revised: Tan et al.

Original: and Kulkarni et al.
Revised: and Kulkarni et al.

Original: (2020) for more details.
Revised: (2020) for more details.

Original: Decoupling the Body (Extractor) and the Head (Classifier) for Personalized
Revised: Decoupling the Body (Extractor) and the Head (Classifier) for Personalized

Original: FL
Revised: FL

Original: The training scheme that involves decoupling the entire network into the body and the head has been used in various fields, including long-tail recognition (Kang et al., 2019; Yu et al., 2020), noisy label learning (Zhang & Yao, 2020), and meta-learning (Oh et al., 2021; Raghu et al., 2019).
Revised: The training scheme that involves decoupling the entire network into the body and the head has been used in various fields, including long-tail recognition (Kang et al., 2019; Yu et al., 2020), noisy label learning (Zhang & Yao, 2020), and meta-learning (Oh et al., 2021; Raghu et al., 2019).

Original: 1 For personalized FL, there have been attempts to use this decoupling approach.
Revised: 1 For personalized FL, there have been attempts to use this decoupling approach.

Original: For a consistent explanation, we describe each algorithm from the perspective of local update and aggregation parts.
Revised: For a consistent explanation, we describe each algorithm from the perspective of local update and aggregation parts.

Original: FedPer (Arivazhagan et al., 2019) learns the entire network jointly during local updates and only aggregates the bottom layers, which are close to the input.
Revised: FedPer (Arivazhagan et al., 2019) learns the entire network jointly during local updates and only aggregates the bottom layers, which are close to the input.

Original: When the bottom layers are matched with the body, the body is shared on all clients and the head is personalized to each client.
Revised: When the bottom layers are matched with the body, the body is shared on all clients and the head is personalized to each client.

Original: LG-FedAvg (Liang et al., 2020) learns the entire network jointly during local updates and only aggregates the top layers, which are close to the output, based on the pre-trained global network via FedAvg.
Revised: LG-FedAvg (Liang et al., 2020) learns the entire network jointly during local updates and only aggregates the top layers, which are close to the output, based on the pre-trained global network via FedAvg.

Original: When the top layers are matched with the head, the body is personalized to each client and the head is shared on all clients.
Revised: When the top layers are matched with the head, the body is personalized to each client and the head is shared on all clients.

Original: FedRep (Collins et al., 2021) learns the entire network sequentially during local updates and only aggregates the body.
Revised: FedRep (Collins et al., 2021) learns the entire network sequentially during local updates and only aggregates the body.

Original: In the local update stage, each client first learns the head only with the aggregated representation and then learns the body only with its own head within a single epoch.
Revised: In the local update stage, each client first learns the head only with the aggregated representation and then learns the body only with its own head within a single epoch.

Original: Unlike the above decoupling methods, we propose a FedBABU algorithm that learns only the body with the randomly initialized head during local updates and aggregates only the body.
Revised: Unlike the above decoupling methods, we propose a FedBABU algorithm that learns only the body with the randomly initialized head during local updates and aggregates only the body.

Original: It is thought that fixing the head during the entire federated training provides the same guidelines on learning representations across all clients.
Revised: It is thought that fixing the head during the entire federated training provides the same guidelines on learning representations across all clients.

Original: Then, personalized local models can be obtained by fine-tuning the head.
Revised: Then, personalized local models can be obtained by fine-tuning the head.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: FL training procedure We summarize the training procedure of FL following the aforementioned four stages with formal notations.
Revised: FL training procedure We summarize the training procedure of FL following the aforementioned four stages with formal notations.

Original: Let { 1 , · · · , N } be the set of all clients.
Revised: Let { 1 , · · · , N } be the set of all clients.

Original: Then, the participating clients in the communication round k with client fraction ratio f is C k
Revised: Then, the participating clients in the communication round k with client fraction ratio f is C k

Original: = { C ki } ⌊ Nf ⌋ i =1 .
Revised: = { C ki } ⌊ Nf ⌋ i =1 .

Original: By broadcasting, the local parameters of the participating clients { θ ki (0) } ⌊ Nf ⌋ i =1 are initialized by the global parameter θ k − 1 G , that is, θ ki (0) ← θ k − 1 G for all i ∈ [1 , ⌊ Nf ⌋ ] and k ∈ [1 , K ] .
Revised: By broadcasting, the local parameters of the participating clients { θ ki (0) } ⌊ Nf ⌋ i =1 are initialized by the global parameter θ k − 1 G , that is, θ ki (0) ← θ k − 1 G for all i ∈ [1 , ⌊ Nf ⌋ ] and k ∈ [1 , K ] .

Original: Note that θ 0 G is randomly initialized first.
Revised: Note that θ 0 G is randomly initialized first.

Original: On its own device, each local model is updated using a locally kept data set.
Revised: On its own device, each local model is updated using a locally kept data set.

Original: After local epochs τ , the locally updated models become { θ ki ( τI ki ) }
Revised: After local epochs τ , the locally updated models become { θ ki ( τI ki ) }

Original: ⌊ Nf ⌋ i =1 , where I ki is the number of iterations of one epoch on client C ki (i.e., ⌈ n Cki B ⌉ ), n C ki is the number of data samples for client C ki , and B is the batch size.
Revised: ⌊ Nf ⌋ i =1 , where I ki is the number of iterations of one epoch on client C ki (i.e., ⌈ n Cki B ⌉ ), n C ki is the number of data samples for client C ki , and B is the batch size.

Original: Therefore, τI ki is the total number of updates in one communication interval.
Revised: Therefore, τI ki is the total number of updates in one communication interval.

Original: Note that our research mainly deals with a balanced environment in which all clients have the same size data set (i.e., I ki is a constant for all k and i ).
Revised: Note that our research mainly deals with a balanced environment in which all clients have the same size data set (i.e., I ki is a constant for all k and i ).

Original: Finally, the global parameter θ kG is aggregated by (cid:80)
Revised: Finally, the global parameter θ kG is aggregated by (cid:80)

Original: ⌊ Nf ⌋ i =1 n Cki n θ ki ( τI ki ) , where n = (cid:80)
Revised: ⌊ Nf ⌋ i =1 n Cki n θ ki ( τI ki ) , where n = (cid:80)

Original: ⌊ Nf ⌋ i =1 n C ki , at the server.
Revised: ⌊ Nf ⌋ i =1 n C ki , at the server.

Original: For our algorithm, the parameters θ are decoupled into the body (extractor) parameters θ ext and the head (classifier) parameters θ cls ∈ R C × d , where d is the dimension of the last representations and C is the number of classes.
Revised: For our algorithm, the parameters θ are decoupled into the body (extractor) parameters θ ext and the head (classifier) parameters θ cls ∈ R C × d , where d is the dimension of the last representations and C is the number of classes.

Original: Experimental setup We mainly use MobileNet on CIFAR100.
Revised: Experimental setup We mainly use MobileNet on CIFAR100.

Original: We set the number of clients to 100 and then each client has 500 training data and 100 test data; the classes in the training and test data sets are the same.
Revised: We set the number of clients to 100 and then each client has 500 training data and 100 test data; the classes in the training and test data sets are the same.

Original: For the heterogeneous distribution of client data, we refer to the experimental setting in McMahan et al.
Revised: For the heterogeneous distribution of client data, we refer to the experimental setting in McMahan et al.

Original: We sort the data by labels and divide the data into the same-sized shards.
Revised: We sort the data by labels and divide the data into the same-sized shards.

Original: Because there is no overlapping data between shards, the size of a shard is defined by | D | N × s , where | D | is the data set size, N is the total number of clients, and s is the number of shards per user.
Revised: Because there is no overlapping data between shards, the size of a shard is defined by | D | N × s , where | D | is the data set size, N is the total number of clients, and s is the number of shards per user.

Original: We control FL environments with three hyperparameters: client fraction ratio f , local epochs τ , and shards per user s .
Revised: We control FL environments with three hyperparameters: client fraction ratio f , local epochs τ , and shards per user s .

Original: f is the number of participating clients out of the total number of clients in every round and a small f is natural in the FL settings because the total number of clients is numerous.
Revised: f is the number of participating clients out of the total number of clients in every round and a small f is natural in the FL settings because the total number of clients is numerous.

Original: The local epochs τ are equal to the interval between two consecutive communication rounds.
Revised: The local epochs τ are equal to the interval between two consecutive communication rounds.

Original: To fix the number of total updates to ensure the consistency in all experiments, we fix the product of communication rounds and local epochs to 320 (e.g., if local epochs are four, then the total number of communication rounds is 80).
Revised: To fix the number of total updates to ensure the consistency in all experiments, we fix the product of communication rounds and local epochs to 320 (e.g., if local epochs are four, then the total number of communication rounds is 80).

Original: The learning rate starts with 0.1 and is decayed by a factor of 0.
Revised: The learning rate starts with 0.1 and is decayed by a factor of 0.

Original: at half and three-quarters of total updates.
Revised: at half and three-quarters of total updates.

Original: τ is closely related to the trade-off between accuracy and communication costs.
Revised: τ is closely related to the trade-off between accuracy and communication costs.

Original: A small τ provides an accurate federation but requires considerable communication costs.
Revised: A small τ provides an accurate federation but requires considerable communication costs.

Original: s is related to the maximum number of classes each user can have; hence, as s decreases, the degree of data heterogeneity increases.
Revised: s is related to the maximum number of classes each user can have; hence, as s decreases, the degree of data heterogeneity increases.

Original: Evaluation We calculate the initial accuracy and personalized accuracy of FedAvg and FedBABU following the federated personalization evaluation procedure proposed in Wang et al.
Revised: Evaluation We calculate the initial accuracy and personalized accuracy of FedAvg and FedBABU following the federated personalization evaluation procedure proposed in Wang et al.

Original: (2019) to analyze the algorithms at the client level: (1) the learned global model is broadcast to all clients and is then evaluated on the test data set of each client D tsi (referred to as the initial accuracy ), (2) the learned global model is personalized using the training data set of each client D tri by fine-tuning with the fine-tuning epochs of τ f ; the personalized models are then evaluated on the test data set of each client D tsi (referred to as the personalized accuracy ).
Revised: (2019) to analyze the algorithms at the client level: (1) the learned global model is broadcast to all clients and is then evaluated on the test data set of each client D ts i (referred to as the initial accuracy ), (2) the learned global model is personalized using the training data set of each client D tri by fine-tuning with the fine-tuning epochs of τ f ; the personalized models are then evaluated on the test data set of each client D tsi (referred to as the personalized accuracy ).

Original: In addition, we calculate the personalized accuracy of other personalized FL algorithms (such as FedPer, LG-FedAvg, and FedRep).
Revised: In addition, we calculate the personalized accuracy of other personalized FL algorithms (such as FedPer, LG-FedAvg, and FedRep).

Original: Algorithm 2 in Appendix A describes the evaluation procedure.
Revised: Algorithm 2 in Appendix A describes the evaluation procedure.

Original: The values (X ± Y) in all tables indicate the mean ± standard deviation of the accuracies across all clients , not across multiple seeds.
Revised: The values (X ± Y) in all tables indicate the mean ± standard deviation of the accuracies across all clients , not across multiple seeds.

Original: Here, reducing the variance over the clients could be interesting but goes beyond the scope of this study.
Revised: Here, reducing the variance over the clients could be interesting but goes beyond the scope of this study.

Original:  P ERSONALIZATION OF A S INGLE G LOBAL M ODEL
Revised:  P ERSONALIZATION OF A S INGLE G LOBAL M ODEL

Original: We first investigate personalization of a single global model using the FedAvg algorithm, following
Revised: We first investigate personalization of a single global model using the FedAvg algorithm, following

Original: Wang et al.
Revised: Wang et al.

Original: (2019) and Jiang et al.
Revised: (2019) and Jiang et al.

Original: (2019), to connect a single global model to multiple personalized[Table] [Table] models.
Revised: (2019), to connect a single global model to multiple personalized models.

Original: We evaluate both the initial accuracy and the personalized accuracy, assuming that the test data sets are not gathered in the server but scattered on the clients.
Revised: We evaluate both the initial accuracy and the personalized accuracy, assuming that the test data sets are not gathered in the server but scattered on the clients.

Original:  Table 1 describes the accuracy of FedAvg on CIFAR100 according to different FL settings ( f , τ , and s ) with 100 clients.
Revised: [Table] [Table] Table 1 describes the accuracy of FedAvg on CIFAR100 according to different FL settings ( f , τ , and s ) with 100 clients.

Original: The initial and personalized accuracies indicate the evaluated performance without fine-tuning and with five fine-tuning epochs for each client, respectively.
Revised: The initial and personalized accuracies indicate the evaluated performance without fine-tuning and with five fine-tuning epochs for each client, respectively.

Original: As previous studies have shown, the more realistic the setting (i
Revised: As previous studies have shown

Original: [Table]
Revised: [Table]

Original: e., a smaller f , larger τ
Revised: the more realistic the setting (i

Original: [Table]
Revised: [Table]

Original:  and smaller s ), the lower the initial accuracy.
Revised: e., a smaller f , larger τ , and smaller s ), the lower the initial accuracy.

Original: Moreover, the tendency of the personalized models to converge higher than the global model observed in Jiang et al.
Revised: Moreover, the tendency of the personalized models to converge higher than the global model observed in Jiang et al.

Original: (2019) is the same.
Revised: (2019) is the same.

Original: More interestingly, it is shown that the gap between the initial accuracy and the personalized accuracy increases significantly as the data become more heterogeneous.
Revised: More interestingly, it is shown that the gap between the initial accuracy and the personalized accuracy increases significantly as the data become more heterogeneous.

Original: It is thought that a small s makes local tasks easier because the label distribution owned by each client is limited and the number of samples per class increases.
Revised: It is thought that a small s makes local tasks easier because the label distribution owned by each client is limited and the number of samples per class increases.

Original: In addition, we conduct an intriguing experiment in which the initial accuracy increases but the personalized accuracy decreases, maintaining the FL training procedures.
Revised: In addition, we conduct an intriguing experiment in which the initial accuracy increases but the personalized accuracy decreases, maintaining the FL training procedures.

Original: In Jiang et al.
Revised: In Jiang et al.

Original: , the authors showed that centralized trained models are more difficult to personalize.
Revised: , the authors showed that centralized trained models are more difficult to personalize.

Original: Similarly, we design an experiment in which the federated trained models are more difficult to personalize.
Revised: Similarly, we design an experiment in which the federated trained models are more difficult to personalize.

Original: We assume that the server has a small portion of p of the non-private client data of the clients 4 such that the non-private data can be used in the server to mitigate the degradation derived from data heterogeneity.
Revised: We assume that the server has a small portion of p of the non-private client data of the clients 4 such that the non-private data can be used in the server to mitigate the degradation derived from data heterogeneity.

Original: We update the global model using this shared non-private data after every aggregation with only one epoch.
Revised: We update the global model using this shared non-private data after every aggregation with only one epoch.

Original: Table 2 describes the result of this experiment.
Revised: Table 2 describes the result of this experiment.

Original: We update the entire network (F in Table 2) on the server.
Revised: We update the entire network (F in Table 2) on the server.

Original: As p increases, the initial accuracy increases, as expected.
Revised: As p increases, the initial accuracy increases, as expected.

Original: However, the personalized accuracy decreases under significant data heterogeneity ( s =10).
Revised: However, the personalized accuracy decreases under significant data heterogeneity ( s =10).

Original: This result implies that boosting a single global model can hurt personalization, which may be considered more important than the initial performance.
Revised: This result implies that boosting a single global model can hurt personalization, which may be considered more important than the initial performance.

Original: Therefore, we agree that the development of an excellent global model should consider the ability to be adequately fine-tuned or personalized.
Revised: Therefore, we agree that the development of an excellent global model should consider the ability to be adequately fine-tuned or personalized.

Original: To investigate the cause of personalization performance degradation, we hypothesize that unnecessary and disturbing information for personalization (i.e., the information of similar classes that a certain client does not have) is injected into the head, when a global model is trained on the server.
Revised: To investigate the cause of personalization performance degradation, we hypothesize that unnecessary and disturbing information for personalization (i.e., the information of similar classes that a certain client does not have) is injected into the head, when a global model is trained on the server.

Original: 5 To capture this, we only update the body (B in Table 2) on the server by zeroing the learning rate corresponding to the head.
Revised: 5 To capture this, we only update the body (B in Table 2) on the server by zeroing the learning rate corresponding to the head.

Original: By narrowing the update parts, the personalization degradation problem can be remedied significantly without affecting the initial accuracy.
Revised: By narrowing the update parts, the personalization degradation problem can be remedied significantly without affecting the initial accuracy.

Original: From this observation, we argue that training the head using shared data negatively impacts the personalization.
Revised: From this observation, we argue that training the head using shared data negatively impacts the personalization.

Original:  F ED BABU: F EDERATED A VERAGING WITH B ODY A GGREGATION AND B ODY U PDATE
Revised:  F ED BABU: F EDERATED A VERAGING WITH B ODY A GGREGATION AND B ODY U PDATE

Original: We propose a novel algorithm that learns a better single global model to be personalized efficiently.
Revised: We propose a novel algorithm that learns a better single global model to be personalized efficiently.

Original: Inspired by prior studies on long-tailed recognition (Yu et al., 2020; Kang et al., 2019), fine-tuning (Devlin et al., 2019), and self-supervised learning (He et al., 2020), as well as our data-sharing experiment, we decouple the entire network into the body and the head.
Revised: Inspired by prior studies on long-tailed recognition (Yu et al., 2020; Kang et al., 2019), fine-tuning (Devlin et al., 2019), and self-supervised learning (He et al., 2020), as well as our data-sharing experiment, we decouple the entire network into the body and the head.

Original: The body is trained for generalization and the head is then trained for specialization.
Revised: The body is trained for generalization and the head is then trained for specialization.

Original: We apply this idea to federated learning by never training the head in the federated training phase (i.e., developing a single global model) and by fine-tuning the head for personalization (in the evaluation process).
Revised: We apply this idea to federated learning by never training the head in the federated training phase (i.e., developing a single global model) and by fine-tuning the head for personalization (in the evaluation process).

Original: 5. F ROZEN H EAD IN THE C ENTRALIZED S ETTING
Revised: 5. F ROZEN H EAD IN THE C ENTRALIZED S ETTING

Original: [Figure]
Revised: [Figure]

Original: Before proposing our algorithm, we empirically demonstrate that a model with the initialized and fixed head (body in Figure 2) has comparable performance to a model that jointly learns the body and the head (full in Figure 2).
Revised: Before proposing our algorithm, we empirically demonstrate that a model with the initialized and fixed head (body in Figure 2) has comparable performance to a model that jointly learns the body and the head (full in Figure 2).

Original: Figure 2 depicts the test accuracy curve of MobileNet on CIFAR100 for various training scenarios in the centralized setting, where total epochs are 160 and learning rate starts with 0.
Revised: Figure 2 depicts the test accuracy curve of MobileNet on CIFAR100 for various training scenarios in the centralized setting, where total epochs are 160 and learning rate starts with 0.

Original: and is decayed by a factor of 0.1 at 80 and 120 epochs.
Revised: and is decayed by a factor of 0.1 at 80 and 120 epochs.

Original: The blue line represents the accuracy when all layers are trained, the red line represents the accuracy when only the body of the model is trained, and the green line represents the accuracy when only the head of the model is trained.
Revised: The blue line represents the accuracy when all layers are trained, the red line represents the accuracy when only the body of the model is trained, and the green line represents the accuracy when only the head of the model is trained.

Original: The fully trained model and the fixed head have almost the same performance, whereas the fixed body performs poorly.
Revised: The fully trained model and the fixed head have almost the same performance, whereas the fixed body performs poorly.

Original: Thus, we claim that the randomly initialized fixed head is acceptable, whereas the randomly initialized fixed body is unacceptable; initialized head is thought to serve as guidelines .
Revised: Thus, we claim that the randomly initialized fixed head is acceptable, whereas the randomly initialized fixed body is unacceptable; initialized head is thought to serve as guidelines .

Original: This characteristic is derived from the orthogonality on the head, as is explained in Appendix B.
Revised: This characteristic is derived from the orthogonality on the head, as is explained in Appendix B.

Original: These results are the reasons we can update only the body during local training in FL settings.
Revised: These results are the reasons we can update only the body during local training in FL settings.

Original: 5. F ED BABU A LGORITHM
Revised: 5. F ED BABU A LGORITHM

Original: [Table]
Revised: [Table]

Original: Based on the insights and results in Section 5.1, we propose a new FL algorithm called
Revised: Based on the insights and results in Section 5.1, we propose a new FL algorithm called

Original: FedBABU ( Fed erated Averaging with B ody
Revised: FedBABU ( Fed erated Averaging with B ody

Original: A ggregation and B ody U pdate).
Revised: A ggregation and B ody U pdate).

Original: Only the body is trained, whereas the head is never trained during federated training; therefore, there is no need to aggregate the head.
Revised: Only the body is trained, whereas the head is never trained during federated training; therefore, there is no need to aggregate the head.

Original: Formally, the model parameters θ are decoupled into the body (extractor) parameters θ ext and the head (classifier) parameters θ cls .
Revised: Formally, the model parameters θ are decoupled into the body (extractor) parameters θ ext and the head (classifier) parameters θ cls .

Original: Note that θ cls on any client is fixed with the head parameters of a randomly initialized global parameters θ 0 G until a single global model converges.
Revised: Note that θ cls on any client is fixed with the head parameters of a randomly initialized global parameters θ 0 G until a single global model converges.

Original: This is implemented by zeroing the learning rate that corresponds to the head.
Revised: This is implemented by zeroing the learning rate that corresponds to the head.

Original: Intuitively, it is believed that the same fixed head on all clients serves as the same criteria on learning representations across all clients despite the passage of training time.
Revised: Intuitively, it is believed that the same fixed head on all clients serves as the same criteria on learning representations across all clients despite the passage of training time.

Original: Algorithm 1 describes the training procedure of FedBABU.
Revised: Algorithm 1 describes the training procedure of FedBABU.

Original: All notations are explained in the
Revised: All notations are explained in the

Original: FL training procedure paragraph in Section 3.
Revised: FL training procedure paragraph in Section 3.

Original: Lines 3-4, Line 6, Line 7, and Line 9 correspond to the client sampling, broadcasting, local update, and aggregation stage, respectively.
Revised: Lines 3-4, Line 6, Line 7, and Line 9 correspond to the client sampling, broadcasting, local update, and aggregation stage, respectively.

Original: In the
Revised: In the

Original: ClientBodyUpdate function
Revised: ClientBodyUpdate function

Original: , the local body parameter θ ki,ext is always updated based on the same head θ 0 G,cls (Line 16).
Revised: , the local body parameter θ ki,ext is always updated based on the same head θ 0 G,cls (Line 16).

Original: Then, only the global body parameter θ kG,ext is aggregated (Line 9).
Revised: Then, only the global body parameter θ kG,ext is aggregated (Line 9).

Original: Therefore, the final global parameter θ K G (Line 11) and the initialized global parameter θ 0 G (Line 1) have the same head parameter θ 0 G,cls .
Revised: Therefore, the final global parameter θ K G (Line 11) and the initialized global parameter θ 0 G (Line 1) have the same head parameter θ 0 G,cls .

Original: In this section, we first investigate the representation power after federated training (Section 5.2.1).
Revised: In this section, we first investigate the representation power after federated training (Section 5.2.1).

Original: Next, we demonstrate the ability related to personalization (Section 5.2.2 to Section 5.2.4).
Revised: Next, we demonstrate the ability related to personalization (Section 5.2.2 to Section 5.2.4).

Original: We further show our algorithm’s applicability (Section 5.2.5).
Revised: We further show our algorithm’s applicability (Section 5.2.5).

Original: 5.2. R EPRESENTATION P OWER OF G LOBAL M ODELS T RAINED BY F ED A VG AND F ED BABU
Revised: 5.2. R EPRESENTATION P OWER OF G LOBAL M ODELS T RAINED BY F ED A VG AND F ED BABU

Original: [Table]
Revised: [Table]

Original: Higher initial accuracy is often required because some clients may not have data for personalization in practice.
Revised: Higher initial accuracy is often required because some clients may not have data for personalization in practice.

Original: In addition, the representation power of the initial models is related to the performance of downstream or personal tasks.
Revised: In addition, the representation power of the initial models is related to the performance of downstream or personal tasks.

Original: To capture the representation power, the “without (w/o) head” accuracy is calculated by replacing the trained head with the nearest template following previous studies (Kang et al., 2019;
Revised: To capture the representation power, the “without (w/o) head” accuracy is calculated by replacing the trained head with the nearest template following previous studies (Kang et al., 2019;

Original: Snell et al., 2017; Raghu et al., 2019;
Revised: Snell et al., 2017; Raghu et al., 2019;

Original: Oh et al., 2021): (1) Using the trained global model and training data set on each client, the representations of each class are averaged into the template of each class.
Revised: Oh et al., 2021): (1) Using the trained global model and training data set on each client the representations of each class are averaged into the template of each class.

Original: Then, (2) the cosine similarity between the test samples and the templates are measured and the test samples are classified into the nearest template’s class.
Revised: Then, (2) the cosine similarity between the test samples and the templates are measured and the test samples are classified into the nearest template’s class.

Original: Templates are created for each client because raw data cannot be moved.
Revised: Templates are created for each client because raw data cannot be moved

Original: Table 3 describes the initial accuracy of FedAvg and FedBABU according to the existence of the head.
Revised: Table 3 describes the initial accuracy of FedAvg and FedBABU according to the existence of the head.

Original: Because all clients share the same criteria and learn representations based on that criteria during federated training in FedBABU, improved representation power is expected.
Revised: Because all clients share the same criteria and learn representations based on that criteria during federated training in FedBABU, improved representation power is expected.

Original: 7 When evaluated without the head, FedBABU has improved performance compared to FedAvg in all cases, particularly under large data heterogeneity.
Revised: 7 When evaluated without the head, FedBABU has improved performance compared to FedAvg in all cases, particularly under large data heterogeneity.

Original: Specifically, when s = 100 (i.e., each client has most of the classes of CIFAR100), the performance gap depends on whether the head exists in FedAvg, but not in FedBABU.
Revised: Specifically, when s = 100 (i.e., each client has most of the classes of CIFAR100), the performance gap depends on whether the head exists in FedAvg, but not in FedBABU.

Original: This means that the features through FedBABU are well-represented to the extent that there is nothing to be supplemented through the head.
Revised: This means that the features through FedBABU are well-represented to the extent that there is nothing to be supplemented through the head.

Original: When s is small, the performance improves when there is no head.
Revised: When s is small, the performance improves when there is no head.

Original: Because templates are constructed based on the training data set for each client, it is not possible to classify test samples into classes that the client does not have.
Revised: Because templates are constructed based on the training data set for each client, it is not possible to classify test samples into classes that the client does not have.

Original: In other words, restrictions on the output distribution of each client lead to tremendous performance gains, even without training the global model.
Revised: In other words, restrictions on the output distribution of each client lead to tremendous performance gains, even without training the global model.

Original: Therefore, it is expected that fine-tuning the global models with a strong representation can boost personalization.
Revised: Therefore, it is expected that fine-tuning the global models with a strong representation can boost personalization [Table]

Original: 5.2. P ERSONALIZATION OF F ED A VG AND F ED BABU
Revised: 5.2. P ERSONALIZATION OF F ED A VG AND F ED BABU

Revised: [Table]

Original: To investigate the dominant factor for personalization of FedBABU, we compare the performance according to the fine-tuned part.
Revised: To investigate the dominant factor for personalization of FedBABU, we compare the performance according to the fine-tuned part.

Original: Table 4a describes the results of this experiment.
Revised: Table 4a describes the results of this experiment.

Original: Global models are fine-tuned with five epochs based on the training data set of each client.
Revised: Global models are fine-tuned with five epochs based on the training data set of each client.

Original: It is shown that fine-tuning including the head (i.e., Head or Full in Table 4a) is better for personalization than body-partially fine-tuning (i.e., Body in Table 4a).
Revised: It is shown that fine-tuning including the head (i.e., Head or Full in Table 4a) is better for personalization than body-partially fine-tuning (i [Table] e., Body in Table 4a).

Original: It is noted that the computational costs can be reduced by fine-tuning only the head in the case of FedBABU without performance degradation.
Revised: It is noted that the computational costs can be reduced by fine-tuning only the head in the case of FedBABU without performance degradation.

Original: [Table]

Original: However, in the case of FedAvg [Table] a performance gap appears.
Revised: However, in the case of FedAvg, a performance gap appears.

Original: Table 4b describes the personalized accuracy of FedAvg according to the fine-tuned parts.
Revised: Table 4b describes the personalized accuracy of FedAvg according to the fine-tuned parts.

Original: It is shown that fine-tuning the entire network (Full in Table 4b) is better for personalization than partial fine-tuning (Body or Head in Table 4b) [Table]
Revised: It is shown that fine-tuning the entire network (Full in Table 4b) is better for personalization than partial fine-tuning (Body or Head in Table 4b).

Original: Therefore, for FedAvg [Table] it is recommended to personalize global models by fine-tuning the entire network for better performance.
Revised: Therefore, for FedAvg, it is recommended to personalize global models by fine-tuning the entire network for better performance.

Original: For consistency of the evaluation, for both FedAvg and FedBABU, we fine-tuned the entire model in this paper
Revised: For consistency of the evaluation, for both FedAvg and FedBABU, we fine-tuned the entire model in this paper

Original: [Table]
Revised: [Table]

Original: 5.2. P ERSONALIZATION P ERFORMANCE C OMPARISON
Revised: 5.2. P ERSONALIZATION P ERFORMANCE C OMPARISON

Original: [Table]
Revised: [Table]

Original: We compare FedBABU with existing methods from the perspective of personalization.
Revised: We compare FedBABU with existing methods from the perspective of personalization.

Original: Appendix A presents details of the evaluation procedure and implementations of each algorithm.
Revised: Appendix A presents details of the evaluation procedure and implementations of each algorithm.

Original: Table 5 describes the personalized accuracy of various algorithms.
Revised: Table 5 describes the personalized accuracy of various algorithms.

Original: Notably, FedAvg is a remarkably strong baseline, overwhelming other recent personalized FL algorithms on CIFAR100 in most cases.
Revised: Notably, FedAvg is a remarkably strong baseline, overwhelming other recent personalized FL algorithms on CIFAR100 in most cases.

Original: 8 These results are similar to recent trends in the field of meta-learning, where fine-tuning based on well-trained representations overwhelms advanced few-shot classification algorithms for heterogeneous tasks (Chen et al., 2019a; 2020a; Dhillon et al., 2019; Tian et al., 2020).
Revised: 8 These results are similar to recent trends in the field of meta-learning, where fine-tuning based on well-trained representations overwhelms advanced few-shot classification algorithms for heterogeneous tasks (Chen et al., 2019a; 2020a; Dhillon et al., 2019; Tian et al., 2020).

Original: FedBABU (ours) further outshines
Revised: FedBABU (ours) further outshines

Original: FedAvg ; it is believed that enhancing representations by freezing the head improves the performance.
Revised: FedAvg ; it is believed that enhancing representations by freezing the head improves the performance.

Original: In detail, when τ =1, the performance gap between FedBABU and FedRep demonstrates the importance of fixing the head across clients.
Revised: In detail, when τ =1, the performance gap between FedBABU and FedRep demonstrates the importance of fixing the head across clients.

Original: Note that when τ =1, if there is no training process on the head in FedRep , FedRep is reduced to FedBABU .
Revised: Note that when τ =1, if there is no training process on the head in FedRep , FedRep is reduced to FedBABU .

Original: Moreover, we attribute the performance degradation of FedRep to the low epochs of training on the body.
Revised: Moreover, we attribute the performance degradation of FedRep to the low epochs of training on the body.

Original: In addition, the efficiency of personalized
Revised: In addition, the efficiency of personalized

Original: FL algorithms increases when all clients participate; therefore, FedPer (Arivazhagan et al., 2019) assumes the activation of all clients during every communication round.
Revised: FL algorithms increases when all clients participate; therefore, FedPer (Arivazhagan et al., 2019) assumes the activation of all clients during every communication round.

Original: The Per-FedAvg algorithm might suffer from dividing the test set into a support set and query set when there are many classes and from small fine-tuning iterations.
Revised: The Per-FedAvg algorithm might suffer from dividing the test set into a support set and query set when there are many classes and from small fine-tuning iterations.

Original: In summary, FedAvg are comparable and better than even recent personalized FL algorithms such as Per-FedAvg and Ditto , and FedBABU (ours) achieves the higher performance than FedAvg and other decoupling algorithms.
Revised: In summary, FedAvg are comparable and better than even recent personalized FL algorithms such as Per-FedAvg and Ditto , and FedBABU (ours) achieves the higher performance than FedAvg and other decoupling algorithms.

Original: 5.2. P ERSONALIZATION S PEED OF F ED A VG AND F ED BABU
Revised: 5.2. P ERSONALIZATION S PEED OF F ED A VG AND F ED BABU

Original: [Table]
Revised: [Table]

Original: We investigate the personalization speed of FedAvg and FedBABU by controlling the fine-tuning epochs τ f in the evaluation process.
Revised: We investigate the personalization speed of FedAvg and FedBABU by controlling the fine-tuning epochs τ f in the evaluation process.

Original: Table 6 describes the initial (when τ f is 0) and personalized (otherwise) accuracy of FedAvg and FedBABU.
Revised: Table 6 describes the initial (when τ f is 0) and personalized (otherwise) accuracy of FedAvg and FedBABU.

Original: Here, one epoch is equal to 10 updates in our case because each client has 500 training samples and the batch size is 50.
Revised: Here, one epoch is equal to 10 updates in our case because each client has 500 training samples and the batch size is 50.

Original: It has been shown that FedAvg requires sufficient epochs for fine-tuning, as reported in Jiang et al.
Revised: It has been shown that FedAvg requires sufficient epochs for fine-tuning, as reported in Jiang et al.

Original: Notably, FedBABU achieves better accuracy with a small number of epochs, which means that FedBABU can personalize accurately and rapidly, especially when fine-tuning is either costly or restricted.
Revised: Notably, FedBABU achieves better accuracy with a small number of epochs, which means that FedBABU can personalize accurately and rapidly, especially when fine-tuning is either costly or restricted.

Original: This characteristic is explained further based on cosine similarity in Appendix K.
Revised: This characteristic is explained further based on cosine similarity in Appendix K.

Original: 5.2. B ODY A GGREGATION AND B ODY U PDATE ON THE F ED P ROX
Revised: 5.2. B ODY A GGREGATION AND B ODY U PDATE ON THE F ED P ROX

Original: [Table]
Revised: [Table]

Original: FedProx (Li et al., 2018) regularizes the distance between a global model and local models to prevent local models from deviating.
Revised: FedProx (Li et al., 2018) regularizes the distance between a global model and local models to prevent local models from deviating.

Original: The degree of regularization is controlled by µ .
Revised: The degree of regularization is controlled by µ .

Original: Note that when µ is 0.0, FedProx is reduced to FedAvg.
Revised: Note that when µ is 0.0, FedProx is reduced to FedAvg.

Original: Table 7 describes the initial and personalized accuracy of
Revised: Table 7 describes the initial and personalized accuracy of

Original: FedProx and FedProx+BABU with µ =0.01 when f =0.1, and Appendix M reports the results when f =1.0.
Revised: FedProx and FedProx+BABU with µ =0.01 when f =0.1, and Appendix M reports the results when f =1.0.

Original: The global models trained by FedProx reduce the personalization capabilities compared to FedAvg (refer to Table 1 when f =0.1), particularly under realistic FL settings.
Revised: The global models trained by FedProx reduce the personalization capabilities compared to FedAvg (refer to Table 1 when f =0.1), particularly under realistic FL settings.

Original: We adapt the body aggregation and body update idea to the FedProx algorithm, referred to as FedProx+BABU, which performs better than the personalization of FedAvg.
Revised: We adapt the body aggregation and body update idea to the FedProx algorithm, referred to as FedProx+BABU, which performs better than the personalization of FedAvg.

Original: Furthermore, FedProx+BABU improves personalized performance compared to FedBABU (refer to Table 5 when f =0.1).
Revised: Furthermore, FedProx+BABU improves personalized performance compared to FedBABU (refer to Table 5 when f =0.1).

Original: This means that the regularization of the body is still meaningful.
Revised: This means that the regularization of the body is still meaningful.

Original: Our algorithm and various experiments suggest future directions of federated learning: Which parts should be federated and enhanced?
Revised: Our algorithm and various experiments suggest future directions of federated learning: Which parts should be federated and enhanced?

Original: Representation!
Revised: Representation!

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this study, we focused on how to train a good federated global model for the purpose of personalization.
Revised: In this study, we focused on how to train a good federated global model for the purpose of personalization.

Original: Namely, this problem boils down to how to pre-train a better backbone in a federated learning manner for downstream personal tasks.
Revised: Namely, this problem boils down to how to pre-train a better backbone in a federated learning manner for downstream personal tasks.

Original: We first showed that existing methods to improve a global model (e.g., data sharing or regularization) could reduce ability to personalize.
Revised: We first showed that existing methods to improve a global model (e.g., data sharing or regularization) could reduce ability to personalize.

Original: To mitigate this personalization performance degradation problem, we decoupled the entire network into the body, related to generality, and the head, related to personalization.
Revised: To mitigate this personalization performance degradation problem, we decoupled the entire network into the body, related to generality, and the head, related to personalization.

Original: Then, we demonstrated that training the head creates this problem.
Revised: Then, we demonstrated that training the head creates this problem.

Original: Furthermore, we proposed the FedBABU algorithm, which learns a shared global model that can rapidly adapt to heterogeneous data on each client.
Revised: Furthermore, we proposed the FedBABU algorithm, which learns a shared global model that can rapidly adapt to heterogeneous data on each client.

Original: FedBABU only updates the body in the local update stage and only aggregates the body in the aggregation stage, thus developing a single global model with a strong representation.
Revised: FedBABU only updates the body in the local update stage and only aggregates the body in the aggregation stage, thus developing a single global model with a strong representation.

Original: This global model can be efficiently personalized by fine-tuning each client’s model using its own data set.
Revised: This global model can be efficiently personalized by fine-tuning each client’s model using its own data set.

Original: Extensive experimental results showed that FedBABU overwhelmed various personalized FL algorithms.
Revised: Extensive experimental results showed that FedBABU overwhelmed various personalized FL algorithms.

Original: Our improvement emphasizes the importance of federating and enhancing the representation for FL.
Revised: Our improvement emphasizes the importance of federating and enhancing the representation for FL.

Original: R EPRODUCIBILITY S TATEMENT
Revised: A CKNOWLEDGMENTS

Revised: This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) [No.2019-0-00075, Artificial

Original: We submitted our source code as an anonymous link in the abstract and will open it through GitHub.
Revised: Intelligence Graduate School Program (KAIST)] and [No. 2021-0-00907, Development of Adaptive and Lightweight Edge-Collaborative Analysis Technology for Enabling Proactively Immediate Response and Rapid Learning]. R EPRODUCIBILITY S TATEMENT Our code is available at https://github.com/jhoon-oh/FedBABU .

Original: For convenience reproducibility, shell files of each algorithm are also included.
Revised: For convenience reproducibility, shell files of each algorithm are also included.

Original: A I MPLEMENTATION D ETAIL
Revised: A I MPLEMENTATION D ETAIL

Original: Our implementations are based on the code presented by (Liang et al., 2020).
Revised: Our implementations are based on the code presented by (Liang et al., 2020).

Original: All the reported results are based on the last model.
Revised: All the reported results are based on the last model.

Original: For computation costs, we used a single TITAN RTX and the entire training time of FedBABU on CIFAR100 using MobileNet takes ∼ 2 hours when f =1.0 and τ =1.
Revised: For computation costs, we used a single TITAN RTX and the entire training time of FedBABU on CIFAR100 using MobileNet takes ∼ 2 hours when f =1.0 and τ =1.

Original: Therefore, as f decreases and τ increases, it takes less time.
Revised: Therefore, as f decreases and τ increases, it takes less time.

Original: A. A RCHITECTURES
Revised: A. A RCHITECTURES

Original: In our experiments
Revised: In our experiments

Original: [Table]
Revised: [Table]

Original: we employ the 3convNet for EMNIST, 4convNet for CIFAR10, MobileNet for
Revised: we employ the 3convNet for EMNIST, 4convNet for CIFAR10, MobileNet for

Original: CIFAR
Revised: CIFAR

Original: [Table]
Revised: [Table]

Original: and ResNet for CIFAR100.
Revised: and ResNet for CIFAR100.

Original: 3convNet and 4convNet are described below.
Revised: 3convNet and 4convNet are described below.

Original: MobileNet (Howard et al., 2017) includes depthwise convolution and pointwise convolution
Revised: MobileNet (Howard et al., 2017) includes depthwise convolution and pointwise convolution

Original: [Table]
Revised: [Table]

Original: and ResNet (He et al., 2016) includes skip connection.
Revised: and ResNet (He et al., 2016) includes skip connection.

Original: 10 Please refer to each paper in detail.
Revised: 10 Please refer to each paper in detail.

Original: [Table]
Revised: [Table]

Original: features )
Revised: features )

Original: [Table]
Revised: [Table]

Original: Sequential
Revised: Sequential

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A. D ATASETS
Revised: A. D ATASETS

Original: We used two public datasets, the CIFAR (Krizhevsky et al., 2009) 11 and EMNIST (Cohen et al., 2017) 12 , for performance evaluation.
Revised: We used two public datasets, the CIFAR (Krizhevsky et al., 2009) 11 and EMNIST (Cohen et al., 2017) 12 , for performance evaluation.

Original: The composition of each dataset is summarized in Table 8.
Revised: The composition of each dataset is summarized in Table 8.

Original: We applied linear transformations such as horizontal flipping and random cropping.
Revised: We applied linear transformations such as horizontal flipping and random cropping.

Original: [Table]
Revised: [Table]

Original: A. H YPERPARAMETERS
Revised: A. H YPERPARAMETERS

Original: For LG-FedAvg , FedAvg models corresponding to the FL settings in Table 3 are used as an initialization.
Revised: For LG-FedAvg , FedAvg models corresponding to the FL settings in Table 3 are used as an initialization.

Original: It is then trained about a quarter of FedAvg training with a learning rate of 0.001.
Revised: It is then trained about a quarter of FedAvg training with a learning rate of 0.001.

Original: For
Revised: For

Original: FedRep , the number of local epochs for head and body updates were set to τ and 1, respectively.
Revised: FedRep , the number of local epochs for head and body updates were set to τ and 1, respectively.

Original: For
Revised: For

Original: Per-FedAvg , we used the first-order (FO) version, and the β value was set as the learning rate value corresponding to the communication round.
Revised: Per-FedAvg , we used the first-order (FO) version, and the β value was set as the learning rate value corresponding to the communication round.

Original: For Ditto , the λ used to control the regularization term was set to 0.75.
Revised: For Ditto , the λ used to control the regularization term was set to 0.75.

Original: A. E VALUATION OF FL ALGORITHMS
Revised: A. E VALUATION OF FL ALGORITHMS

Original: In FL algorithms, because the parts that need to be shared over all clients after federated training may not be shared due to client fraction, we evaluate these algorithms as follows:
Revised: In FL algorithms, because the parts that need to be shared over all clients after federated training may not be shared due to client fraction, we evaluate these algorithms as follows:

Original: 1: initlist =
Revised: [Table]

Original: [] (If alg is FedAvg or FedBABU) 2: perlist = [] 3: for each client i ∈ [1 ,N ] in parallel do 4: θ i,s (0) ← θ K G,s 5: Acc init
Revised: [Table]

Original: Eval (
Revised: [Table]

Original: (0); D tsi ) (
Revised: [Table] [Table]

Original: ) 6: initlist
Revised: [Table]

Original: append( Acc init ) (
Revised: [Table] [Table] [Table]

Original: ) 7: θ i (
Revised: [Table]

Original: f
Revised: [Table]

Original:  I i ) ← Fine-tune ( alg,θ i (0) ,τ f ; D tri ) 8:
Revised: ( τ f I i ) ← Fine-tune ( alg,θ i (0) ,τ f ; D tri ) 8:

Original: Acc per ← Eval ( θ i ( τ f I i ); D tsi ) 9: perlist.append( Acc per ) 10: end for 11: return initlist, perlist 12: function F INE - TUNE ( alg,θ i ,τ f ) 13:
Revised: Acc per ← Eval ( θ i ( τ f I i ); D tsi ) 9: perlist.append( Acc per ) 10: end for 11: return initlist, perlist 12: function F INE - TUNE ( alg,θ i ,τ f ) 13:

Original: I i ← ⌈ nCiB ⌉ 14: for each fine-tune epoch 1, ··· , τ f do 15: for each iteration 1, ··· , I i do 16:
Revised: I i ← ⌈ nCiB ⌉ 14: for each fine-tune epoch 1, ··· , τ f do 15: for each iteration 1, ··· , I i do 16:

Original: θ i ← SGD ( θ i ) (If alg is FedAvg or FedBABU or FedPer or LG-FedAvg) 17: θ i,cls ← SGD ( θ i ) (If alg is FedRep) 18: end for 19: end for 20: for each iteration 1, ··· , I i do (If alg is FedRep) 21: θ i,ext ← SGD ( θ i ) 22: end for 23: return θ i 24: end function where θ K G,s indicates the shared parts after federated training (i.e., full in the cases of FedAvg and
Revised: θ i ← SGD ( θ i ) (If alg is FedAvg or FedBABU or FedPer or LG-FedAvg) 17: θ i,cls ← SGD ( θ i ) (If alg is FedRep) 18: end for 19: end for 20: for each iteration 1, ··· , I i do (If alg is FedRep) 21: θ i,ext ← SGD ( θ i ) 22: end for 23: return θ i 24: end function where θ K G,s indicates the shared parts after federated training (i.e., full in the cases of FedAvg and

Original: FedBABU, body in the cases of FedPer and FedRep, and head in the case of LG-FedAvg).
Revised: FedBABU, body in the cases of FedPer and FedRep, and head in the case of LG-FedAvg).

Original: All notations are explained in the evaluation paragraph in Section 3.
Revised: All notations are explained in the evaluation paragraph in Section 3.

Original: Because communication round k is not related to evaluation, it is emitted.
Revised: Because communication round k is not related to evaluation, it is emitted.

Original: For FedAvg and FedBABU, initial accuracy is also calculated.
Revised: For FedAvg and FedBABU, initial accuracy is also calculated.

Original: Although all clients have different personalized models because some parts are not shared after broadcasting except for FedAvg and FedBABU, it is not completely personalized because there is a common part θ K G,s .
Revised: Although all clients have different personalized models because some parts are not shared after broadcasting except for FedAvg and FedBABU, it is not completely personalized because there is a common part θ K G,s .

Original: Therefore, we fine-tune all algorithms.
Revised: Therefore, we fine-tune all algorithms.

Original: FedRep trains models sequentially (Line 17 and 21), while others jointly (Line 16).
Revised: FedRep trains models sequentially (Line 17 and 21), while others jointly (Line 16).

Original: Finally, the mean and standard deviation of initlist and perlist are reported as initial accuracy and personalized accuracy in our paper.
Revised: Finally, the mean and standard deviation of initlist and perlist are reported as initial accuracy and personalized accuracy in our paper.

Original: B O RTHOGONALITY OF R ANDOM I NITIALIZATION
Revised: B O RTHOGONALITY OF R ANDOM I NITIALIZATION

Original: [Figure]
Revised: [Figure]

Original: We hypothesize that the orthogonal initialization (Saxe et al., 2013) on the head, through which the row vectors of the classifier parameters θ cls are orthogonal, is the best for the body-only update rule.
Revised: We hypothesize that the orthogonal initialization (Saxe et al., 2013) on the head, through which the row vectors of the classifier parameters θ cls are orthogonal, is the best for the body-only update rule.

Original: At each update, the output of the extractor is pulled toward its label’s row of θ cls and pushed apart from the other rows of θ cls through backpropagation.
Revised: At each update, the output of the extractor is pulled toward its label’s row of θ cls and pushed apart from the other rows of θ cls through backpropagation.

Original: Therefore, orthogonal weight initialization can separate the outputs of the extractor well based on their class labels.
Revised: Therefore, orthogonal weight initialization can separate the outputs of the extractor well based on their class labels.

Original: Note that distribution-based random initializations, such as Xaiver (Glorot & Bengio, 2010) and
Revised: Note that distribution-based random initializations, such as Xaiver (Glorot & Bengio, 2010) and

Original: He (He et al., 2015), have almost orthogonal rows because the orthogonality of two random vectors is observed in high dimensions with high probability (Lezama et al., 2018).
Revised: He (He et al., 2015), have almost orthogonal rows because the orthogonality of two random vectors is observed in high dimensions with high probability (Lezama et al., 2018).

Original: Figure 3 shows test accuracy curves for many different initialization methods when only the body is trained in the centralized setting.
Revised: Figure 3 shows test accuracy curves for many different initialization methods when only the body is trained in the centralized setting.

Original: 13 When the row vectors are initialized similarly, a convergence gap appears (green and red lines in Figure 3), as expected.
Revised: 13 When the row vectors are initialized similarly, a convergence gap appears (green and red lines in Figure 3), as expected.

Original: More experiments in the centralized setting are reported in subsection F.1 and Appendix G.
Revised: More experiments in the centralized setting are reported in subsection F.1 and Appendix G.

Original: Based on these empirical and theoretical results, we used He (uniform) initialization, the default setting in PyTorch (Paszke et al., 2019), in our paper.
Revised: Based on these empirical and theoretical results, we used He (uniform) initialization, the default setting in PyTorch (Paszke et al., 2019), in our paper.

Original: This characteristic can be proved.
Revised: This characteristic can be proved.

Original: Xavier (Glorot & Bengio, 2010) and He (He et al., 2015) are representative distribution-based random initialization methods, controlling the variance of parameters using network size to avoid the exploding/vanishing gradient problem.
Revised: Xavier (Glorot & Bengio, 2010) and He (He et al., 2015) are representative distribution-based random initialization methods, controlling the variance of parameters using network size to avoid the exploding/vanishing gradient problem.

Original: Two initialization methods use uniform and normal distribution.
Revised: Two initialization methods use uniform and normal distribution.

Original: In detail, each element follows U ( − a, a ) or N (0 , σ 2 ) independently, where a and σ are defined by non-linearity (such as ReLU and Tanh) and layer input-output size.
Revised: In detail, each element follows U ( − a, a ) or N (0 , σ 2 ) independently, where a and σ are defined by non-linearity (such as ReLU and Tanh) and layer input-output size.

Original: Because we used the ReLU non-linear function in our implementations, we calculate a and σ by multiplying √ 2 , for scaling of ReLU non-linearity.
Revised: Because we used the ReLU non-linear function in our implementations, we calculate a and σ by multiplying √ 2 , for scaling of ReLU non-linearity.

Original: Because we focus on initialization on the head, we specify θ cls ∈ R C × d , where C is the number of classes (i.e., layer output size) and d is the dimension of the last representation (i.e., layer input size).
Revised: Because we focus on initialization on the head, we specify θ cls ∈ R C × d , where C is the number of classes (i.e., layer output size) and d is the dimension of the last representation (i.e., layer input size).

Original: Namely, each element random variable { w i,j } i ∈ [1 ,C ] ,j ∈ [1 ,d ] follows U ( − a, a ) or N (0 , σ 2 ) independently.
Revised: Namely, each element random variable { w i,j } i ∈ [1 ,C ] ,j ∈ [1 ,d ] follows U ( − a, a ) or N (0 , σ 2 ) independently.

Original: When w i,j ∼ U ( − a, a ) , in Xavier initialization, a is defined as √ 2 × (cid:113) 6 d + C .
Revised: When w i,j ∼ U ( − a, a ) , in Xavier initialization, a is defined as √ 2 × (cid:113) 6 d + C .

Original: Similarly, in He initialization, a is defined as √ 2 × (cid:113) 3 d or √ 2 × (cid:113) 3 C depending on which to be preserved, forward pass or backward pass.
Revised: Similarly, in He initialization, a is defined as √ 2 × (cid:113) 3 d or √ 2 × (cid:113) 3 C depending on which to be preserved, forward pass or backward pass.

Original: When w i,j ∼ N (0 , σ 2 ) , in Xavier initialization, σ is defined as √ 2 × (cid:113) 2 d + C .
Revised: When w i,j ∼ N (0 , σ 2 ) , in Xavier initialization, σ is defined as √ 2 × (cid:113) 2 d + C .

Original: Similarly, in He initialization, σ is defined as √ 2 × (cid:113) 1 d or √ 2 × (cid:113) 1 C depending on which to be preserved, forward pass or backward pass.
Revised: Similarly, in He initialization, σ is defined as √ 2 × (cid:113) 1 d or √ 2 × (cid:113) 1 C depending on which to be preserved, forward pass or backward pass.

Original: However, It is noted that a in the uniform distribution and σ in the normal distribution do not affect proof on orthogonality, therefore we keep the form of
Revised: However, It is noted that a in the uniform distribution and σ in the normal distribution do not affect proof on orthogonality, therefore we keep the form of

Original: U ( − a, a ) and N (0 , σ 2 ) for simplicity.
Revised: U ( − a, a ) and N (0 , σ 2 ) for simplicity.

Original: We first consider 2 d independent random variables { w i,j } i ∈{ p,q } ,j ∈ [1 ,d ] for any p ̸ = q ∈ [1 , C ] .
Revised: We first consider 2 d independent random variables { w i,j } i ∈{ p,q } ,j ∈ [1 ,d ] for any p ̸ = q ∈ [1 , C ] .

Original: Then, we can define d independent random variables { w p,j w q,j } j ∈ [1 ,d ] from the above 2 d independent random variables.
Revised: Then, we can define d independent random variables { w p,j w q,j } j ∈ [1 ,d ] from the above 2 d independent random variables.

Original: Then, for all j ∈ [1 , d ] , E [ w p,j w q,j ] = E [ w p,j ] E [ w q,j ] = 

Revised: Then, for all j ∈ [1 , d ] , E [ w p,j w q,j ] = E [ w p,j ] E [ w q,j ] =

Original: and V [ w p,j w q,j ] =
Revised: and V [ w p,j w q,j ] =

Original: V [ w p,j ] E [ w q,j ] 2 + V [ w p,j ] V [ w q,j ]+ E
Revised: V [ w p,j ] E [ w q,j ] 2 + V [ w p,j ] V [ w q,j ]+ E

Original: [ w p,j ] 2 V [ w q,j ] = V [ w p,j ] V [ w q,j ] because each element in θ cls follows U ( − a, a ) or N (0 , σ 2 ) independently.
Revised: [ w p,j ] 2 V [ w q,j ] = V [ w p,j ] V [ w q,j ] because each element in θ cls follows U ( − a, a ) or N (0 , σ 2 ) independently.

Original: Let S d = 1 d (cid:80) dj
Revised: Let S d = 1 d (cid:80) dj

Original: =
Revised: =

Original: 1 w p,j w q,j .
Revised: 1 w p,j w q,j .

Original: Then, by the Weak Law of Large Numbers, lim d →∞ P ( | S d − E [ S d ] | > ϵ ) =
Revised: Then, by the Weak Law of Large Numbers, lim d →∞ P ( | S d − E [ S d ] | > ϵ ) =

Original: lim d →∞ P ( | S d | > ϵ ) = 

Revised: lim d →∞ P ( | S d | > ϵ ) =

Original: for all ϵ > 0 because { w p,j w q,j } j ∈ [1 ,d ] are independent random variables and there is V such that V [ w p,j w q,j ] < V for all j .
Revised: for all ϵ > 0 because { w p,j w q,j } j ∈ [1 ,d ] are independent random variables and there is V such that V [ w p,j w q,j ] < V for all j .

Original: C P ERSONALIZATION OF THE C ENTRALIZED M ODEL
Revised: C P ERSONALIZATION OF THE C ENTRALIZED M ODEL

Original: In (Jiang et al., 2019), they showed that localizing centralized initial models is harder.
Revised: In (Jiang et al., 2019) [Table] they showed that localizing centralized initial models is harder.

Original: Similarly, we investigate the personalization of the centralized model under different heterogeneity.
Revised: Similarly, we investigate the personalization of the centralized model under different heterogeneity.

Original: First, we train two models with 10 epochs, one is entirely updated (F in Table 9)
Revised: First, we train two models with 10 epochs, one is entirely updated (F in Table 9)

Original: [Table]
Revised: [Table]

Original: and another is body-partially updated (B in Table 9) using all training data set.
Revised: and another is body-partially updated (B in Table 9) using all training data set.

Original: Then, the trained model is broadcast to each client and then evaluated.
Revised: Then, the trained model is broadcast to each client and then evaluated.

Original: Table 9 describes the results of this experiment.
Revised: Table 9 describes the results of this experiment.

Original: It is demonstrated that if models are updated body-partially, then personalization ability does not hurt even under centralized training.
Revised: It is demonstrated that if models are updated body-partially, then personalization ability does not hurt even under centralized training.

Original: entirely
Revised: entirely

Revised: [Table]

Revised: is trained centralized

Original: [Table]
Revised: [Table]

Original: D E FFECT OF M OMENTUM DURING L OCAL U PDATES ON P ERFORMANCE
Revised: D E FFECT OF M OMENTUM DURING L OCAL U PDATES ON P ERFORMANCE

Original: We investigate the effect of momentum during local updates on the performance of FedAvg and
Revised: We investigate the effect of momentum during local updates on the performance of FedAvg and

Original: FedBABU.
Revised: FedBABU.

Original: Table 10 describes the initial and personalized accuracy according to the momentum during local updates.
Revised: Table 10 describes the initial and personalized accuracy according to the momentum during local updates.

Original: The momentum for personalization fine-tuning is the same as the momentum in federated training.
Revised: The momentum for personalization fine-tuning is the same as the momentum in federated training.

Original: In both cases of FedAvg and FedBABU, appropriate momentum improves the performance, especially personalized accuracy.
Revised: In both cases of FedAvg and FedBABU, appropriate momentum improves the performance, especially personalized accuracy.

Original: However, when the extreme momentum ( m =0.99) is used, FedAvg completely loses the ability to train models, while FedBABU has robust performance.
Revised: However, when the extreme momentum ( m =0.99) is used, FedAvg completely loses the ability to train models [Table] while FedBABU has robust performance [Table]

Original: [Table]
Revised: [Table]

Original: E E FFECT OF M ASSIVENESS ON P ERFORMANCE
Revised: E E FFECT OF M ASSIVENESS ON P ERFORMANCE

Original: We increase the number of clients from 100 to 500, and then each client has 100 training data and 20 test data.
Revised: We increase the number of clients from 100 to 500, and then each client has 100 training data and 20 test data.

Original: Table 11 describes the initial and personalized accuracy of FedAvg and FedBABU on
Revised: Table 11 describes the initial and personalized accuracy of FedAvg and FedBABU on

Original: CIFAR100 under various FL settings with 500 clients.
Revised: CIFAR100 under various FL settings with 500 clients.

Original: The performance of FedAvg is slightly better than that of FedBABU in many cases; however, FedBABU overwhelms FedBABU in the case of extreme FL setting ( s =10, f =0.1, and τ =10).
Revised: The performance of FedAvg is slightly better than that of FedBABU in many cases; however, FedBABU overwhelms FedBABU in the case of extreme FL setting ( s =10, f =0.1, and τ =10).

Original: More interestingly, if each client has a few data samples, then local epochs τ should be more than 1.
Revised: More interestingly, if each client has a few data samples, then local epochs τ should be more than 1.

Original: It is thought that massiveness makes data size insufficient on each client (when the total number of data is fixed) and brings out other problems.
Revised: It is thought that massiveness makes data size insufficient on each client (when the total number of data is fixed) and brings out other problems.

Original: [Table]
Revised: [Table]

Original: [Table] [Table] [Table] [Table]
Revised: [Figure]

Original: [Table] [Table] [Table]
Revised: [Table] [Figure]

Original: [Table]
Revised: [Table]

Original: F R ESULTS OF 4 CONV N ET ON CIFAR F. A CCURACY C URVES IN THE C ENTRALIZED S ETTING
Revised: F R ESULTS OF 4 CONV N ET ON CIFAR F. A CCURACY C URVES IN THE C ENTRALIZED S ETTING

Original: When 4convNet is used on CIFAR10, a similar tendency appears.
Revised: When 4convNet is used on CIFAR [Figure] a similar tendency appears.

Original: Comparing Figure 2 and Figure 4, the importance of learning representation is emphasized more when tasks are difficult.
Revised: Comparing Figure 2 and Figure [Figure] the importance of learning representation is emphasized more when tasks are difficult.

Original: In addition,
Revised: In addition,

Original: Figure 5 shows the necessity of orthogonality when the body is only trained and approximation of random initialization to the orthogonal initialization using 4convNet on CIFAR10.
Revised: Figure 5 shows the necessity of orthogonality when the body is only trained and approximation of random initialization to the orthogonal initialization using 4convNet on CIFAR

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: F. E XPERIMENTAL R ESULTS
Revised: F. E XPERIMENTAL R ESULTS

Original: We also evaluate algorithms on CIFAR10 using 4convNet.
Revised: We also evaluate algorithms on CIFAR10 using 4convNet.

Original: Table 12 and Table 13 are the experiment results corresponding to the Table 3 and Table 5 in the main paper, respectively.
Revised: Table 12 and Table 13 are the experiment results corresponding to the Table 3 and Table 5 in the main paper, respectively.

Original: The tendency of performance comparison and the superiority of our algorithm appears similarly when using 4convNet on CIFAR10.
Revised: The tendency of performance comparison and the superiority of our algorithm appears similarly when using 4convNet on CIFAR10.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: F. A BLATION S TUDY A CCORDING TO THE L OCAL U PDATE P ARTS
Revised: F. A BLATION S TUDY A CCORDING TO THE L OCAL U PDATE P ARTS

Original: We decouple the entire network in more detail during local updates to investigate whether the body should be totally trained.
Revised: We decouple the entire network in more detail during local updates to investigate whether the body should be totally trained.

Original: For this ablation study, we used 4convNet on CIFAR10.
Revised: For this ablation study, we used 4convNet on CIFAR10.

Original: Table 14 and
Revised: Table 14 and

Original: Table 15 show the initial and personalized accuracy according to the local update parts, respectively.
Revised: Table 15 show the initial and personalized accuracy according to the local update parts, respectively.

Original: For consistency, all cases are personalized by fine-tuning the entire network in this experiment.
Revised: For consistency, all cases are personalized by fine-tuning the entire network in this experiment.

Original: In all cases, FedBABU has the best performance, which implies that the body must be totally trained at least when using 4convNet.
Revised: In all cases, FedBABU has the best performance [Table] which implies that the body must be totally trained at least when using 4convNet.

Original: It is believed that the same (body-totally) or similar (body except for a few top layers) trends appear when large
Revised: It is believed that the same (body-totally) or similar (body except for a few top layers

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: G R ESULTS OF R ES N ET 18 AND R ES N ET 50 ON CIFAR G. A CCURACY C URVES IN THE C ENTRALIZED S ETTING
Revised: G R ESULTS OF R ES N ET 18 AND R ES N ET 50 ON CIFAR G. A CCURACY C URVES IN THE C ENTRALIZED S ETTING

Original: We experiment with ResNet18 and ResNet50.
Revised: We experiment with ResNet18 and ResNet50.

Original: In the centralized setting, test accuracy curves according to the update part (Figure 6 and Figure 8) and according to the initialization method (Figure 7 and Figure 9) have the same trend as we have seen before.
Revised: In the centralized setting, test accuracy curves according to the update part (Figure 6 and Figure 8) and according to the initialization method (Figure 7 and Figure 9) have the same trend as we have seen before.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: G. E XPERIMENTAL R ESULTS
Revised: G. E XPERIMENTAL R ESULTS

Original: Table 16 and Table 17 are the results of ResNet18 and ResNet50, respectively.
Revised: Table 16 and Table 17 are the results of ResNet18 and ResNet50, respectively.

Original: It is shown that the performance gap between FedAvg and FedBABU increases when ResNet is used rather than when
Revised: It is shown that the performance gap between FedAvg and FedBABU increases when ResNet is used rather than when

Original: MobileNet (in the main paper) is used.
Revised: MobileNet (in the main paper) is used.

Original: Furthermore, it is observed that as the complexity of the model increases (ResNet18 → ResNet50), the performance of FedAvg decreases, while that of FedBABU does not.
Revised: Furthermore, it is observed that as the complexity of the model increases (ResNet18 → ResNet50), the performance of FedAvg decreases, while that of FedBABU does not.

Original: [Table] is ResNet18.
Revised: The used network is ResNet18.

Original: f 
Revised: f is 0.1.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: H R ESULTS OF R ES N ET 10 ON D IRICHLET DISTRIBUTION - BASED CIFAR 
Revised: H R ESULTS OF R ES N ET 10 ON D IRICHLET DISTRIBUTION - BASED CIFAR

Original: We evaluate ours and existing algorithms under unbalanced and non-IID settings derived from the
Revised: We evaluate ours and existing algorithms under unbalanced and non-IID settings derived from the

Original: Dirichlet distribution.
Revised: Dirichlet distribution.

Original: Figure 10 and 11 describe the distributions of train and test data points and non-IID depending on β , which is the hyperparameter of the Dirichlet distribution.
Revised: Figure 10 and 11 describe the distributions of train and test data points and non-IID depending on β , which is the hyperparameter of the Dirichlet distribution.

Original: Table 18 describes the results on Dirichlet distribution-based non-IID CIFAR100 in the realistic FL setting ( f =0.1 and τ =10).
Revised: Table 18 describes the results on Dirichlet distribution-based non-IID CIFAR100 in the realistic FL setting ( f =0.1 and τ =10).

Original: The lower β implies the larger heterogeneity.
Revised: The lower β implies the larger heterogeneity

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: I R ESULTS OF 3 CONV N ET ON EMNIST
Revised: I R ESULTS OF 3 CONV N ET ON EMNIST

Original: Table 19 describes the results on EMNIST using a network that consists of three convolution layers and a linear classifier.
Revised: Table 19 describes the results on EMNIST using a network that consists of three convolution layers and a linear classifier.

Original: We set the number of users to
Revised: We set the number of users to

Original: [Table]
Revised: [Table]

Original: and the number of data points per user was approximately 450.
Revised: and the number of data points per user was approximately 450.

Original: We fixed the fraction ratio at 0.1 and local epochs at 10 (which is the realistic setting used in our paper).
Revised: We fixed the fraction ratio at 0.1 and local epochs at 10 (which is the realistic setting used in our paper).

Original: The results demonstrate that FedBABU also has the best performance on
Revised: The results demonstrate that FedBABU also has the best performance on

Original: EMNIST.
Revised: EMNIST.

Original: [Table]
Revised: [Table]

Original: J R ESULTS ON I N -D ISTRIBUTION (ID) AND O UT - OF -D ISTRIBUTION (OOD) C LASSES
Revised: J R ESULTS ON I N -D ISTRIBUTION (ID) AND O UT - OF -D ISTRIBUTION (OOD) C LASSES

Original: To investigate the accuracy of in-distribution (ID) and out-of-distribution (OOD) classes, we scatter
Revised: To investigate the accuracy of in-distribution (ID) and out-of-distribution (OOD) classes, we scatter

Original: IID-like test set ( s =100) to all clients.
Revised: IID-like test set ( s =100) to all clients.

Original: Namely, some classes are in test data set ( s =100), but not in training data set ( s =50 or s =10).
Revised: Namely, some classes are in test data set ( s =100), but not in training data set ( s =50 or s =10).

Original: Table 20 describes the results of FedAvg and FedBABU on this experiment.
Revised: Table 20 describes the results of FedAvg and FedBABU on this experiment.

Original: Here, in-distribution accuracy is the test accuracy of the classes present in the training data set; otherwise, out-of-distribution accuracy.
Revised: Here, in-distribution accuracy is the test accuracy of the classes present in the training data set; otherwise, out-of-distribution accuracy.

Original: Before personalization (Before in tables), FedBABU beats FedAvg from both ID and OOD.
Revised: Before personalization (Before in tables), FedBABU beats FedAvg from both ID and OOD.

Original: After personalization (After in tables), FedBABU becomes fit strongly for the ID classes than OOD classes.
Revised: After personalization (After in tables), FedBABU becomes fit strongly for the ID classes than OOD classes.

Original: The OOD accuracy of FedBABU decreases more than that of FedAvg, but the accuracy of FedAvg simiarly drops after personalization.
Revised: The OOD accuracy of FedBABU decreases more than that of FedAvg, but the accuracy of FedAvg simiarly drops after personalization.

Original: These results indicate that no personalization is the best strategy for both FedBABU and FedAvg.
Revised: These results indicate that no personalization is the best strategy for both FedBABU and FedAvg.

Original: Therefore, because the OOD accuracy of FedBABU is higher than that of FedAvg before personalization, it is concluded that FedBABU has better performance than FedAvg when the optimal strategy for OOD is used.
Revised: Therefore, because the OOD accuracy of FedBABU is higher than that of FedAvg before personalization, it is concluded that FedBABU has better performance than FedAvg when the optimal strategy for OOD is used.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: We investigate the cosine similarity between different clients duringfine-tuning epochs, described in Figure 12.
Revised: We investigate the cosine similarity between different clients duringfine-tuning epochs, described in Figure 12.

Original: During fine-tuning for personalization evaluation, the entire network is updated in the cases of both
Revised: During fine-tuning for personalization evaluation, the entire network is updated in the cases of both

Original: FedAvg and FedBABU.
Revised: FedAvg and FedBABU.

Original: From the results, it is found that classifier (black line in Figure 12) is closely related to personalization.
Revised: From the results, it is found that classifier (black line in Figure 12) is closely related to personalization.

Original: In other words, all clients have similar extractors during fine-tuning (i.e., large cosine similarity), while make different classifiers based on their own data.
Revised: In other words, all clients have similar extractors during fine-tuning (i.e., large cosine similarity), while make different classifiers based on their own data.

Original: Furthermore, it is observed that the cosine similarity on the head between different clients decreases more rapidly in the case of FedBABU than FedAvg.
Revised: Furthermore, it is observed that the cosine similarity on the head between different clients decreases more rapidly in the case of FedBABU than FedAvg.

Original: It is believed that this is because the classifier of FedAvg starts personalization from the federated convergent classifier, while that of
Revised: It is believed that this is because the classifier of FedAvg starts personalization from the federated convergent classifier, while that of

Original: FedBABU starts personalization from the initialized orthogonal classifier.
Revised: FedBABU starts personalization from the initialized orthogonal classifier.

Original: This characteristic explains rapid personalization, introduced in Section 5.2.4.
Revised: This characteristic explains rapid personalization, introduced in Section 5.2.4.

Original: L F ED BABU WITH B ODY U PDATE ON THE S ERVER
Revised: L F ED BABU WITH B ODY U PDATE ON THE S ERVER

Original: [Table]
Revised: [Table]

Original: Let us repeat the experiment in Section 4 again
Revised: Let us repeat the experiment in Section 4 again

Original: [Table]
Revised: [Table]

Original: where the server has a small portion p of the nonprivacy data of the clients.
Revised: where the server has a small portion p of the nonprivacy data of the clients.

Original: Table 21 describes the initial and personalized accuracy of FedBABU when the global model is body-partially updated on the server using available data (such as experiments (B) in Table 2).
Revised: Table 21 describes the initial and personalized accuracy of FedBABU when the global model is body-partially updated on the server using available data (such as experiments (B) in Table 2).

Original: FedBABU with body updates on the server improves personalization compared to
Revised: FedBABU with body updates on the server improves personalization compared to

Original: FedAvg with body updates on the server, maintaining the initial accuracy (refer to (B) in Table 2)
Revised: FedAvg with body updates on the server, maintaining the initial accuracy (refer to (B) in Table 2)

Original: [Table]
Revised: [Table]

Original: This result demonstrates that training the head negatively affects personalization even when trained locally.
Revised: This result demonstrates that training the head negatively affects personalization even when trained locally.

Original: In addition, the performance of FedBABU improves as p increases.
Revised: In addition, the performance of FedBABU improves as p increases.

Original: This implies that there is room for enhancing the representation beyond that of FedBABU.
Revised: This implies that there is room for enhancing the representation beyond that of FedBABU

Original:  M B ODY A GGREGATION AND B ODY U PDATE ON THE F ED P ROX
Revised: . M B ODY A GGREGATION AND B ODY U PDATE ON THE F ED P ROX

Original: Even when all clients are active every communication round (i.e., f =1.0), the personalized performance of FedProx+BABU beats not only that of FedAvg but also that of FedBABU.
Revised: Even when all clients are active every communication round (i.e., f =1.0), the personalized performance of FedProx+BABU beats not only that of FedAvg but also that of FedBABU.

Original: [Table]
Revised: [Table]

Original: N D ISCUSSION ON THE EFFECTIVENESS OF F ED A VG
Revised: N D ISCUSSION ON THE EFFECTIVENESS OF F ED A VG

Original: MOCHA (Smith et al., 2017) is a representative personalized FL paper using regularization based on relationships between tasks.
Revised: MOCHA (Smith et al., 2017) is a representative personalized FL paper using regularization based on relationships between tasks.

Original: pFedMe (T Dinh et al., 2020) and Ditto (Li et al., 2021) train local models with a regularization based on the divergence between a global model and local models.
Revised: pFedMe (T Dinh et al., 2020) and Ditto (Li et al., 2021) train local models with a regularization based on the divergence between a global model and local models.

Original: If the weight on regularization is set to zero, these regularized personalized FL algorithms reduce to the local-only algorithm; on the other extreme, these reduce to the FedAvg algorithm (i.e., θ 1 = · · · = θ N ).
Revised: If the weight on regularization is set to zero, these regularized personalized FL algorithms reduce to the local-only algorithm; on the other extreme, these reduce to the FedAvg algorithm (i.e., θ 1 = · · · = θ N ).

Original: The personalized FL algorithms such as FedPer (Collins et al., 2021), in which each client has parts that are not aggregated (i.e., personalized parts), can be explained similarly.
Revised: The personalized FL algorithms such as FedPer (Collins et al., 2021), in which each client has parts that are not aggregated (i.e., personalized parts), can be explained similarly.

Original: If there is no personalized part, these personalized FL algorithms reduce to the FedAvg; on the other extreme, i.e., if the entire network is not aggregated, these personalized FL algorithms reduce to the local-only algorithm.
Revised: If there is no personalized part, these personalized FL algorithms reduce to the FedAvg; on the other extreme, i.e., if the entire network is not aggregated, these personalized FL algorithms reduce to the local-only algorithm.

Original: In summary, personalized FL algorithms are in-between local-only and FedAvg algorithms and have better personalized performance than both local-only and FedAvg in general.
Revised: In summary, personalized FL algorithms are in-between local-only and FedAvg algorithms and have better personalized performance than both local-only and FedAvg in general.

Original: In the same vein, FedAvg+Fine-tuning is also in-between local-only and FedAvg because this algorithm uses two extreme algorithms sequentially (i.e., FedAvg followed by local-only).
Revised: In the same vein, FedAvg+Fine-tuning is also in-between local-only and FedAvg because this algorithm uses two extreme algorithms sequentially (i.e., FedAvg followed by local-only).

Original: However, intuitively, FedAvg+Fine-tuning is a bit more FedAvg-based, whereas personalized FL algorithms are a bit more local-only-based.
Revised: However, intuitively, FedAvg+Fine-tuning is a bit more FedAvg-based, whereas personalized FL algorithms are a bit more local-only-based.

Original: This is because FedAvg+Fine-tuning is the same as FedAvg before personalization, whereas personalized FL algorithms develop different clients’ models from scratch like local-only.
Revised: This is because FedAvg+Fine-tuning is the same as FedAvg before personalization, whereas personalized FL algorithms develop different clients’ models from scratch like local-only.

Original: Therefore, we believe that FedAvg+Fine-tuning earns more benefits (particularly for representation) from federation than personalized FL algorithms from this difference (FedAvgbased v.s. local-only-based).
Revised: Therefore, we believe that FedAvg+Fine-tuning earns more benefits (particularly for representation) from federation than personalized FL algorithms from this difference (FedAvgbased v.s. local-only-based).

Original: An additional difference between FedAvg+Fine-tuning and regularized personalized FL algorithms is that FedAvg+Fine-tuning optimizes clients separately using their own data set based on the well-federated extractor, whereas regularized personalized FL algorithms optimize clients jointly from scratch.
Revised: An additional difference between FedAvg+Fine-tuning and regularized personalized FL algorithms is that FedAvg+Fine-tuning optimizes clients separately using their own data set based on the well-federated extractor, whereas regularized personalized FL algorithms optimize clients jointly from scratch.

Original: In other words, FedAvg+Fine-tuning can easily optimize all clients’ models based on their own data sets.
Revised: In other words, FedAvg+Fine-tuning can easily optimize all clients’ models based on their own data sets.

Original: However, it is too difficult to optimize all clients’ models simultaneously.
Revised: However, it is too difficult to optimize all clients’ models simultaneously.

Original: Therefore, FedAvg+Fine-tuning can achieve better personalized performance than regularized personalized FL algorithms.
Revised: Therefore, FedAvg+Fine-tuning can achieve better personalized performance than regularized personalized FL algorithms.

Original: Furthermore, very recent work (Cheng et al., 2021) has addressed the effectiveness of FedAvg+Finetuning theoretically.
Revised: Furthermore, very recent work (Cheng et al., 2021) has addressed the effectiveness of FedAvg+Finetuning theoretically.

Original: In Cheng et al.
Revised: In Cheng et al.

Original: (2021), they compared local-only (zero collaboration), FedAvg (zero personalization) (McMahan et al., 2017), FedAvg+Fine-tuning, Per-FedAvg (Fallah et al., 2020), and pFedMe (T Dinh et al., 2020) in the high-dimensional asymptotic limit by analyzing bias-variance.
Revised: (2021), they compared local-only (zero collaboration), FedAvg (zero personalization) (McMahan et al., 2017), FedAvg+Fine-tuning, Per-FedAvg (Fallah et al., 2020), and pFedMe (T Dinh et al., 2020) in the high-dimensional asymptotic limit by analyzing bias-variance.

Original: They demonstrated that the asymptotic test loss of FedAvg+Fine-tuning (FTFA in Cheng et al.
Revised: They demonstrated that the asymptotic test loss of FedAvg+Fine-tuning (FTFA in Cheng et al.

Original: (2021)) matches that of Per-FedAvg and that the asymptotic test loss of FedAvg+Fine-tuning with a ridge regularizer (RTFA in Cheng et al.
Revised: (2021)) matches that of Per-FedAvg and that the asymptotic test loss of FedAvg+Fine-tuning with a ridge regularizer (RTFA in Cheng et al.

Original: (2021)) matches that of pFedMe on their stylized linear regression model.
Revised: (2021)) matches that of pFedMe on their stylized linear regression model.

Original: Although Cheng et al.
Revised: Although Cheng et al.

Original: (2021) thoroughly addressed this problem on their stylized linear regression model, there are few results on real data sets except for test accuracies.
Revised: (2021) thoroughly addressed this problem on their stylized linear regression model, there are few results on real data sets except for test accuracies.

Original: It is believed that “why
Revised: It is believed that “why

Original: FedAvg+Fine-tuning is effective” itself needs to be studied more using real data sets, as “why MAML (Finn et al., 2017) is effective” has been addressed in (Raghu et al., 2019).
Revised: FedAvg+Fine-tuning is effective” itself needs to be studied more using real data sets, as “why MAML (Finn et al., 2017) is effective” has been addressed in (Raghu et al., 2019).

Original: O F ED A VG WITH D IFFERENT L EARNING RATES
Revised: O F ED A VG WITH D IFFERENT L EARNING RATES

Original: [Table]
Revised: [Table]

Original: What we want to emphasize is the importance of making all clients have the same class boundary by freezing the head.
Revised: What we want to emphasize is the importance of making all clients have the same class boundary by freezing the head.

Original: We argue that this shared classifier enhances the representation power of the federated model, which is an important factor for personalization.
Revised: We argue that this shared classifier enhances the representation power of the federated model, which is an important factor for personalization.

Original: From this perspective, if the head moves even a little bit client by client, the different class boundaries to train feature extractor are used during local updates.
Revised: From this perspective, if the head moves even a little bit client by client, the different class boundaries to train feature extractor are used during local updates.

Original: Therefore, the federated model’s feature extractor might hurt.
Revised: Therefore, the federated model’s feature extractor might hurt.

Original: To verify this hypothesis
Revised: To verify this hypothesis

Original: [Table]
Revised: [Table]

Original: we experiment using different learning rates depending on the parts.
Revised: we experiment using different learning rates depending on the parts.

Original: The above table describes the federated model’s accuracy without a classifier (i
Revised: The above table describes the federated model’s accuracy without a classifier (i

Original: [Table]
Revised: [Table]

Original: e., w/o classifier accuracy in Table 3) to investigate representation power of models.
Revised: e., w/o classifier accuracy in Table 3) to investigate representation power of models.

Original: We set the body’s initial learning rate as 0.1, the fraction ratio as 0.
Revised: We set the body’s initial learning rate as 0.1, the fraction ratio as 0.

Original: [Table]
Revised: [Table]

Original: and local epochs as 10.
Revised: and local epochs as 10.

Original: α h and α b are the head’s learning rate and the body’s learning rate, respectively.
Revised: α h and α b are the head’s learning rate and the body’s learning rate, respectively.

Original: From this result
Revised: From this result

Original: [Table]
Revised: [Table]

Original: we believe that the federated model’s representation power increases as the head’s learning rate decreases, i.e., as a learning signal for the body is larger.
Revised: we believe that the federated model’s representation power increases as the head’s learning rate decreases, i.e., as a learning signal for the body is larger.

Original: In this trend [Table] we highlight that the best performance is due to the same classifier on all clients.
Revised: In this trend, we highlight that the best performance is due to the same classifier on all clients.

Original: P C LASS - WISE A NALYSIS DURING F EDERATED T RAINING
Revised: P C LASS - WISE A NALYSIS DURING F EDERATED T RAINING

Original: To explain why FedBABU has better performance than FedAvg after federated training (i.e., before fine-tuning), we analyze the federated training procedure of FedAvg and FedBABU class-wisely under the realistic federated settings ( f =0.1 and τ =10).
Revised: To explain why FedBABU has better performance than FedAvg after federated training (i.e., before fine-tuning), we analyze the federated training procedure of FedAvg and FedBABU class-wisely under the realistic federated settings ( f =0.1 and τ =10).

Original: For this analysis, we assume that each client has 500 training data and 10,000 test data (i.e., the whole test data) of CIFAR-100.
Revised: For this analysis, we assume that each client has 500 training data and 10,000 test data (i.e., the whole test data) of CIFAR-100.

Original: Test data set is divided into two groups for each client; in-class test data and out-of-class test data.
Revised: Test data set is divided into two groups for each client; in-class test data and out-of-class test data.

Original: In-class testdata indicates test data whose class is in the classes in training data, whereas out-of-class test data indicates test data whose class is not in the classes in training data. For instance, if client A has
Revised: In-class test

Original: [Figure]
Revised: [Figure]

Original:  classes 0-9 as training data, then in-class test data indicates test data whose class is one of 0-9 and out-of class  test 
Revised: data indicates test data whose class is in the classes in training data, whereas out-of-class test data indicates test data whose class is not in the classes in training data. For instance, if client A has classes 0-9 as training data, then in-class test data indicates test data whose class is one of 0-9 and out-of class indicates test data whose class is one of 10-99.

Original: Figure 13 describes the in-class ((a)-(c)) and out-of-class ((d)-(f)) test accuracy curves according to the heterogeneity.
Revised: Figure 13 describes the in-class ((a)-(c)) and out-of-class ((d)-(f)) test accuracy curves according to the heterogeneity.

Original: Client sampling, broadcasting, local updates, and aggregation stages are repeated during federated training.
Revised: Client sampling, broadcasting, local updates, and aggregation stages are repeated during federated training.

Original: The accuracies per epoch are averaged right after broadcasting the aggregated model to the selected clients and during local updates on them.
Revised: The accuracies per epoch are averaged right after broadcasting the aggregated model to the selected clients and during local updates on them.

Original: It is observed that in-class accuracy of the aggregated model is significantly low and increases dramatically during local updates, whereas out-of-class accuracy of the aggregated model is significantly high and decreases dramatically during local updates, as heterogeneity is larger.
Revised: It is observed that in-class accuracy of the aggregated model is significantly low and increases dramatically during local updates, whereas out-of-class accuracy of the aggregated model is significantly high and decreases dramatically during local updates, as heterogeneity is larger.

Original: In all cases, FedBABU has higher out-of-class accuracy than FedAvg after local updates.
Revised: In all cases, FedBABU has higher out-of-class accuracy than FedAvg after local updates.

Original: It implies that freezing the head makes less damage to out-of-class during local updates and can lead to better aggregation .
Revised: It implies that freezing the head makes less damage to out-of-class during local updates and can lead to better aggregation .

Original: Furthermore, when heterogeneity is large ( s =10), FedBABU has higher in-class accuracy than FedAvg after local updates.
Revised: Furthermore, when heterogeneity is large ( s =10), FedBABU has higher in-class accuracy than FedAvg after local updates.

Original: Q F ED BABU WITH THE NON - ORTHOGONAL CLASSIFIERS
Revised: Q F ED BABU WITH THE NON - ORTHOGONAL CLASSIFIERS

Original: We demonstrate that an orthogonal initialization on the head is required for desirable performance in the centralized setting (Appendix B).
Revised: We demonstrate that an orthogonal initialization on the head is required for desirable performance in the centralized setting (Appendix B).

Original: To investigate whether this characteristic is still required under FL settings, we compare FedBABU with the orthogonal head (which is proposed) to FedBABU without the orthogonal head.
Revised: To investigate whether this characteristic is still required under FL settings, we compare FedBABU with the orthogonal head (which is proposed) to FedBABU without the orthogonal head.

Original: FedBABU without the orthogonal head is designed in the same way as ‘similar’ in Appendix B. Table 24 describes the results.
Revised: FedBABU without the orthogonal head is designed in the same way as ‘similar’ in Appendix B. Table 24 describes the results.

Original: As we expected, if the head does not consist of the orthogonal row vectors, FedBABU cannot achieve desirable performance.
Revised: As we expected, if the head does not consist of the orthogonal row vectors, FedBABU cannot achieve desirable performance

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: R D ESCRIPTION OF DATA DISTRIBUTION ACCORDING TO THE SHARDS PER USER s
Revised: R D ESCRIPTION OF DATA DISTRIBUTION ACCORDING TO THE SHARDS PER USER s

Original: To help understanding data distribution of clients according to the shard per user s , we provide examples of them.
Revised: To help understanding data distribution of clients according to the shard per user s , we provide examples of them.

Original: Figure 14 describes examples of data distribution of clients when s is 10 (left) or 2 (right).
Revised: Figure 14 describes examples of data distribution of clients when s is 10 (left) or 2 (right).

Original: Consider that there are 10 clients and CIFAR10 dataset is scattered to each client.
Revised: Consider that there are 10 clients and CIFAR10 dataset is scattered to each client.

Original: When s is 10, total shards 100 because 10 users and 10 shards per user.
Revised: When s is 10, total shards 100 because 10 users and 10 shards per user.

Original: Then, the CIFAR10 dataset is divided into 100 shards, hence each shard consits of 500 samples with the same class.
Revised: Then, the CIFAR10 dataset is divided into 100 shards, hence each shard consits of 500 samples with the same class.

Original: Here, each client has 10 shards randomly.
Revised: Here, each client has 10 shards randomly.

Original: In this situation, each client can have up to 10 classes.
Revised: In this situation, each client can have up to 10 classes.

Original: On the contrary, when s is 2, total shards are 20 because 10 users and 2 shards per user.
Revised: On the contrary, when s is 2, total shards are 20 because 10 users and 2 shards per user.

Original: Then, the CIFAR10 dataset is divided into 20 shards, hence each shard consits of 2500 samples with the same class.
Revised: Then, the CIFAR10 dataset is divided into 20 shards, hence each shard consits of 2500 samples with the same class.

Original: Here, each client hasshards randomly.
Revised: Here, each client hasshards randomly.

Original: In this situation, each client can have 2 classes at most.
Revised: In this situation, each client can have 2 classes at most.

Original: Namely, as s gets smaller, the number of classes that each clients has is limited (i.e., label distribution owned by each client is limited) and the number of samples per class increases.
Revised: Namely, as s gets smaller, the number of classes that each clients has is limited (i.e., label distribution owned by each client is limited) and the number of samples per class increases.

Original: [Figure]
Revised: [Figure]

Original: S PERFORMANCE WITH LARGER TOTAL EPOCHS
Revised: S P ERFORMANCE WITH LARGER TOTAL EPOCHS

Original: We fixed the total number of epochs to 320, i.e., K (total communication rounds) × τ (local epochs) = 320.
Revised: We fixed the total number of epochs to 320, i.e., K (total communication rounds) × τ (local epochs) = 320.

Original: This is because the performance gap between algorithms appeared and Collins et al.
Revised: This is because the performance gap between algorithms appeared and Collins et al.

Original: (2021) used total number of epochs to 500, where K =100 and τ =5 on CIFAR100.
Revised: (2021) used total number of epochs to 500, where K =100 and τ =5 on CIFAR100.

Original: However, because algorithms may not converge, we evaluate them with larger total epochs of 640.
Revised: However, because algorithms may not converge, we evaluate them with larger total epochs of 640.

Original: Furthermore, we add the results when s is 20 and 5.
Revised: Furthermore, we add the results when s is 20 and 5.

Original: Table 25 describes the results with the total epochs of 640 when f is 0.1.
Revised: Table 25 describes the results with the total epochs of 640 when f is 0.1.

Original: The results demonstrate that FedBABU is the best under large heterogeneity (i
Revised: The results demonstrate that FedBABU is the best under large heterogeneity (i

Original: [Table]
Revised: [Table]

Original: e., s =20, 10, and 5), which is the situation federated learning researchers attempt to solve.
Revised: e., s =20, 10, and 5), which is the situation federated learning researchers attempt to solve.

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: T Q UALITATIVE COMPARISON BETWEEN F ED A VG AND F ED BABU
Revised: T Q UALITATIVE COMPARISON BETWEEN F ED A VG AND F ED BABU

Original: Figure 15 describes t-SNE visualization (Van der Maaten & Hinton, 2008) of representations learned by FedAvg and FedBABU on CIFAR10 and CIFAR100 for a qualitative comparison.
Revised: Figure 15 describes t-SNE visualization (Van der Maaten & Hinton, 2008) of representations learned by FedAvg and FedBABU on CIFAR10 and CIFAR100 for a qualitative comparison.

Original: For speed acceleration, we used t-SNE-CUDA (Chan et al., 2019).
Revised: For speed acceleration, we used t-SNE-CUDA (Chan et al., 2019).

Original: For CIFAR100, the sparse classes are converted into coarse classes (e.g., bicycle → vehicle).
Revised: For CIFAR100, the sparse classes are converted into coarse classes (e.g., bicycle → vehicle).

Original: However, the difference cannot be captured by the naked eye.
Revised: However, the difference cannot be captured by the naked eye.

Original: Therefore, we further provide t-SNE visualization using only sub-classes of ‘vehicle’ super-class, described as Figure 16.
Revised: Therefore, we further provide t-SNE visualization using only sub-classes of ‘vehicle’ super-class, described as Figure 16.

Original: For CIFAR10, airplane, automobile, ship, and truck classes are used, and for CIFAR100, bicycle, bus, motorcycle, pickup truck, train, lawn-mower, rocket, streetcar, tank, and tractor are used.
Revised: For CIFAR10, airplane, automobile, ship, and truck classes are used, and for CIFAR100, bicycle, bus, motorcycle, pickup truck, train, lawn-mower, rocket, streetcar, tank, and tractor are used.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]