Original: W HERE TO B EGIN ? O N THE I MPACT OF P RE -T RAINING AND I NITIALIZATION IN F EDERATED L EARNING Anonymous authors
Revised: W HERE TO B EGIN ? O N THE I MPACT OF P RE -T RAINING AND I NITIALIZATION IN F EDERATED L EARNING Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: An oft-cited challenge of federated learning is the presence of data heterogeneity — that data at different clients may follow very different distributions.
Revised: An oft-cited challenge of federated learning is the presence of heterogeneity. Data heterogeneity refers to the fact that data from different clients may follow very different distributions.

Revised: System heterogeneity refers to the fact that client devices have different system capabilities.

Original: Several federated optimization methodshave been proposed to address this challenge.
Revised: A considerable number of federated optimization methods address this challenge.

Original: In the literature, empirical evaluations usually start federated training froma random initialization.
Revised: In the literature, empirical evaluations usually start federated training from random initialization.

Original: However, in many practical applications of federated learning, the server has access to proxy data for the training task that can be used to pretrain a model before starting federated training.
Revised: However, in many practical applications of federated learning, the server has access to proxy data for the training task that can be used to pre-train a model before starting federated training.

Original: We empirically study the impact of starting from a pre-trained model in federated learning using four common federated learning benchmark datasets.
Revised: We empirically study the impact of starting from a pre-trained model in federated learning using four standard federated learning benchmark datasets.

Original: Unsurprisingly, starting from a pre-trained model reduces the training time required to reach a target error rate and enables  training  more accurate models (by up to 40%) than is possible when starting froma random initialization.
Revised: Unsurprisingly, starting from a pre-trained model reduces the training time required to reach a target error rate and enables the training of more accurate models (up to 40%) than is possible when starting from random initialization.

Original: Surprisingly, we also find thatthe effect of data heterogeneity is much less significant when starting federated training from a pre-trained initialization.
Revised: Surprisingly, we also ﬁnd that starting federated learning from a pre-trained initialization reduces the effect of both data and system heterogeneity.

Original: Rather, when starting from a pre-trained model, using an adaptive optimizer such as F ED A DAM at the server consistently leads to the best accuracy.

Original: We recommend that future work proposing and evaluating federated optimization methods evaluate the performance when startingboth from random and pre-trained initializations.
Revised: We recommend that future work proposing and evaluating federated optimization methods evaluate the performance when starting from random and pre-trained initializations.

Original: We also believe this study raises several questions for further work on understanding the role of heterogeneity in federated optimization.
Revised: We also believe this study raises several questions for further work on understanding the role of heterogeneity in federated optimization.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Federated learning (FL) has emerged as a popular distributed machine learning paradigm for privately training a shared model across many participants while the training data never leaves the participant devices.
Revised: Federated learning (FL) has emerged as a popular distributed machine learning paradigm for privately training a shared model across many participants while the training data never leaves the participant devices.

Original: Our focus in this work is on understanding the impact of model initialization on federated learning.
Revised: This paper empirically investigates the impact of model initialization on federated optimization methods.

Original: In cross-device FL (Kairouz et al., 2019), the primary setting considered in this paper, a large number of client devices (possibly on the order of hundreds of millions) participate in training, which is coordinated by a server.
Revised: Previous empirical evaluations of FL methods start federated training from a randomly initialized model. Transfer learning from pre-trained models has become common practice in natural language processing Radford et al. ; Devlin et al. and computer vision He et al.

Original: It is impractical for all devices to participate at every step of training; instead,
Revised: Dosovitskiy et al. (2020), yielding state-of-the-art results on many tasks and enabling faster model convergence in the centralized setting. Although public proxy data is available at the server in many applications, few prior works studied the impact of starting federated training from a pre-trained model.

Original:  FL usually proceeds in rounds and a small subset of available devices participate in each round.
Revised: In cross-device FL (Kairouz et al., 2019), the primary setting considered in this paper, a central server coordinates a large number of client devices (possibly on the order of hundreds of millions).

Original: Each device possesses a local dataset, and the data at different devices follow different distributions, leading to the data heterogeneity challenge (Kairouz et al., 2019).
Revised: Each device possesses a local dataset, and the data at different devices follow different distributions, leading to the data heterogeneity challenge (Kairouz et al., 2019).

Revised: Moreover, client devices have different system capabilities, leading to system heterogeneity .

Original: Moreover, devices communicate with the server over low-bandwidth, high-latency links, making communication overhead a major performance factor.
Revised: Finally, devices communicate with the server over low-bandwidth communication links making the performance bottleneck.

Original: The predominant approach to federated training builds on local update methods such as F ED A VG (McMahan et al., 2016), where a device performs several local updates (e.g., one epoch of SGD on their local training set) before transmitting an update to the server.
Revised: The predominant approach to federated training builds on local update methods such as F ED A VG (McMahan et al., 2016), where a device performs several local updates (e.g., one epoch of SGD on their local training set) before transmitting an update to the server.

Original: Although this reduces communication overhead, it can also exacerbatethe data heterogeneity problem.
Revised: Although this reduces communication overhead, it can also exacerbate data heterogeneity.

Original: Several approaches have been proposed to address this challenge (Li et al., 2018;
Revised: Several approaches have been proposed to address this challenge (Li et al., 2018;

Original: Hsu et al., 2019; Reddi et al., 2020; Wang et al., 2020; Karimireddy et al., 2020; 2021;Zhang et al., 2021; Acar et al., 2021).
Revised: Hsu et al., 2019; Reddi et al., 2020; Wang et al., 2020; Karimireddy et al., 2020; 2021;

Original: This paper empirically investigates the impact of model initialization on federated optimization methods.

Original: Previous empirical evaluations of FL methods nearly always start federated training from a randomly initialized model.

Original: Transfer learning from pre-trained models has become common practice in natural language processing Radford et al.

Original: ; Devlin et al. and computer vision He et al. ; Dosovitskiy et al. (2020), yielding state-of-the-art results on many tasks and enabling
Revised: Zhang et al., 2021; Acar et al., 2021).

Original: [Figure] faster model convergence in the centralized setting. However,  the impact of starting federated training from a pre-trained model has not been carefully studied, even though in many practical applications of FL there is public proxy data available for pre-training at the server.
Revised: However, few prior works examine the impact of initialization on federated training.

Original: Contributions.
Revised: Contributions.

Original: In this work we consider the question: How does model initialization (random, or pre-trained) impact the behavior of federated optimization methods?Towards answering this question
Revised: In this work, we consider the question: How does model initialization (random or pre-trained) impact the behavior of federated optimization methods?

Original: [Figure] we perform an extensive empirical study, comparing 15 variations of federated optimization methods on four commonly-used FL benchmark datasets.
Revised: We perform an extensive empirical study, comparing 15 variations of federated optimization methods on four commonly-used FL benchmark datasets.

Original: Our study reveals four key findings:
Revised: Our study reveals three key ﬁndings:

Original: Although optimizers designed to address heterogeneity typically lead to better performance when starting from a random initialization, when starting from pre-trained model we observe that: (i) there is not as bigof a difference between optimizers in terms of accuracy after a fixed number of rounds, and (ii) using an adaptive optimizer at the server, such as F ED A DAM , is more important than using any method for addressing heterogeneity.
Revised: Although optimizers designed to address heterogeneity typically lead to better performance when starting from a random initialization, when starting from pre-trained model we observe that (cf. Fig. 1): (i) there is not as big a difference between optimizers in terms of accuracy after a ﬁxed number of rounds, and (ii) using an adaptive optimizer at the server, such as F ED A DAM , is more important than using any method for addressing heterogeneity.

Original: Starting from a pre-trained model significantly reduces the difference between having nonIID vs IID data at clients.
Revised: Starting from a pre-trained model signiﬁcantly reduces the difference between having nonIID vs IID data at clients.

Original: Furthermore, when starting from a pre-trained model, the number of local epochs per round can be significantly increased without degrading the final accuracy.
Revised: Furthermore, when starting from a pre-trained model, the number of local epochs per round can be signiﬁcantly increased without degrading the ﬁnal accuracy.

Original: The initial loss is not always lower when starting from a pre-trained model.
Revised: The initial loss is not always lower when starting from a pre-trained model.

Original: However, the largest Hessian eigenvalue (i
Revised: However, the largest Hessian eigenvalue (i

Original: e., local Lipshitz constant, or smoothness) is consistently smaller at initialization when starting from a pre-trained model, compared to when starting from a random initialization.
Revised: e., local Lipshitz constant, or smoothness) is consistently smaller at initialization when starting from a pre-trained model, compared to when starting from a random initialization.

Original: Although some of our empirical observations are consistent with existing  theoretical convergence guarantees for federated training algorithms, our findings also highlight that some aspects of federated training are not captured with the existing theory, suggestingimportant directions for future work.
Revised: Some of our empirical observations are consistent with existing FL theoretical convergence guarantees. Our ﬁndings also highlight that some aspects of FL are not captured with the existing theory, suggesting directions for future work.

Original: Our experiments also confirm what one might expect: that initializing FL with a pre-trained model can increase final model accuracy and reduce the number of rounds required to achieve a target accuracy, thereby saving communications and reducing the overall training time.
Revised: Initializing FL with a pre-trained model can increase ﬁnal model accuracy and reduce the number of rounds required to achieve a target accuracy. Pre-training leads to communication savings and reduces the overall training time.

Original: This is evident in Figure 1 across several datasets (hyperparameters were tuned separately for each dataset–initialization pair; see Section 3 for details of the experimental setup).
Revised: Figure 2 demonstrates the beneﬁt of pre-training across several datasets (hyperparameters were tuned separately for each dataset–initialization pair; see Section 3 for details of the experimental setup).

Original: Our findings are reproducible using the open-source federated learning framework FLSim (FLSim
Revised: Our ﬁndings are reproducible using the open-source federated learning framework FLSim (FLSim

Original: Authors, 2022).
Revised: Authors, 2022).

Original: Informed by these findings, we conclude by presenting several recommendations for future research on federated optimization.
Revised: Informed by these ﬁndings, we present several recommendations for future research on federated optimization.

Original: 1: Input: initial  and η T 2: for each round t = 
Revised: 1: Input: initial global model x 0 , server and client step sizes ⌘ s , ⌘ c , local epochs E , rounds T 2: for each round t = 1 , . .

Original: . . . , T do 3:
Revised: , T do 3:

Original: Server sends x t − 1 to all clients i  S t .
Revised: Server sends x t  1 to all clients i 2 S t .

Original: 4: for each client i ∈ S t in parallel do 5:
Revised: 4: for each client i 2 S t in parallel do 5:

Original: Initialize local model y 

Revised: Initialize local model y

Original: i← x t − 1 .
Revised: i x t  1 .

Original: Each client performs E epochs of local updates via y k
Revised: Each client performs E epochs of local updates via y k

Original: +1 i
Revised: +1 i

Original: =
Revised: =

Original: C LIENT
Revised: C LIENT

Original: O PT ( y ki , F i , η c ) .
Revised: O PT ( y ki , F i , ⌘ c ) .

Original: Let y Ei denote the result after performing E epochs of local updates.
Revised: Let y Ei denote the result after performing E epochs of local updates.

Original: After local training, client i sends ∆ ti = x t − 1 − y Ei to the server.
Revised: After local training, client i sends  ti = x t  1  y Ei to the server.

Original: 8: end for 9:
Revised: 8: end for 9:

Original: Server computes aggregate update ∆ t = 1 |S t | (cid:80) i ∈S t p i ∆ ti .
Revised: Server computes aggregate update  t = 1 |S t | P i 2 S t p i  ti .

Original: Server updates global model x t = S ERVER O PT ( x t − 1 , − ∆ t , η s , t ) .
Revised: Server updates global model x t = S ERVER O PT ( x t  1 ,   t , ⌘ s , t ) .

Original: 11: end for
Revised: 11: end for

Original: P ROBLEM F ORMULATION AND THE F ED O PT FRAMEWORK

Original: We consider the following standard optimization formulation of federated training.
Revised: We consider the following standard optimization formulation of federated training.

Original: We seek to find model parameters w that solve the problem,
Revised: We seek to ﬁnd model parameters w that solve the problem,

Original: [Equation]
Revised: [Equation]

Original: where m is the total number of clients, the function F i measures the average loss of a model with parameters w on the i th client’s training data, and p i > 0 is the weight given to client i .
Revised: where m is the total number of clients, the function F i measures the average loss of a model with parameters w on the i th client’s training data, and p i > 0 is the weight given to client i .

Original: Usually p i is taken to be proportional to the number of samples at client i so that the optimization problem gives equal weight to all training samples.
Revised: Usually p i is taken to be proportional to the number of samples at client i so that the optimization problem gives equal weight to all training samples.

Original: The goal is to find a model that fits all clients’ data well on (weighted) average.
Revised: The goal is to ﬁnd a model that ﬁts all clients’ data well on (weighted) average.

Original: In FL,F i is onlyaccessible by client i .
Revised: In FL, only client i can evaluate F i and its gradient.

Original: All of the methods we consider in this study can be expressed in the general F ED O PT framework introduced in Reddi et al.
Revised: All of the methods we consider in this study can be expressed in the general F ED O PT framework introduced in Reddi et al.

Original: (2020), which encompasses both client and server optimization updates; see Algorithm 1.
Revised: ; see Algorithm 1.

Original: At round t , a server sends its  model x  to a cohort of clients.
Revised: At round t , the server sends its last model x t  1 to a cohort of clients.

Original: Then each client in the cohort performs E epochs of  training  using C LIENT O PT with client learning rate η c , producing a local model x ik .
Revised: Each client in the cohort performs E epochs of local training starting from x t  1 using C LIENT O PT with client learning rate ⌘ c , producing a local model y Ei .

Original: Each client, then communicates the difference  between the client’s local model and the server model,∆ i where ∆ i := 
Revised: Then each client communicates the difference  ti between their local model and the server model, where  ti := x t  1  y Ei .

Original: x k − x ( client update ). The server computes a weighted average ∆ of the client updates  and updates its own model via [Equation]
Revised: The server computes a weighted average  t of the client updates (line 9 in Alg. 1) and updates its own model via x t +1 =

Original:  where S ERVER O PT ( x t , − ∆ t , η s , t ) is a first-order optimizer, η s is the server learning rate, and t is the round number.E XPERIMENTAL S ETUP
Revised: S ERVER O PT ( x t ,  t , ⌘ s , t ) , where S ERVER O PT ( x t ,  t , ⌘ s , t ) is a ﬁrst-order optimizer, ⌘ s is the server learning rate, and t is the round number.

Original: Next, we describe the experimental setup used to evaluate how initialization impacts the performance of federated optimization methods.

Original:   3. D ATASETS , M ODELS  AND T ASKS.
Revised:  E XPERIMENTAL S ETUP 3. D ATASETS , M ODELS , AND T ASKS

Original: To facilitate comparing with other work in the literature, we experiment on four standard FL benchmark datasets: CIFAR-10 (Krizhevsky et al., 2009), Federated EMNIST-62 (FEMNIST) (Cohen et al., 2017; Caldas et al., 2018), Stack Overflow (Authors., 2019) and  Reddit  (Caldas et al., 2018).
Revised: To facilitate comparing with other work in the literature, we experiment on four standard FL benchmark datasets: CIFAR-10 (Krizhevsky et al., 2009), Federated EMNIST-62 (FEMNIST) (Cohen et al., 2017; Caldas et al., 2018), Stack Overﬂow (Authors., 2019) and pushift.io’s Reddit 1 (Caldas et al., 2018).

Original: Default non-IID  data partitioning. We use non-independent and identically distributed (non-IID) splits of data across clients to mimic data heterogeneity commonly observed in practice.
Revised: All datasets have a natural non-IID partitioning of the data except for CIFAR-10, for which we use the Dirichlet allocation approach of Hsu et al. (2019) with parameter 0 . 1 to partition data across 50 users.

Original: For CIFAR10 and FEMNIST, we train Squeezenet (Iandola et al., 2016) and[-ResNet
-]
Revised: For CIFAR-10 and FEMNIST, we train Squeezenet (Iandola et al., 2016) and[-ResNet
-]

Original:  (He et al., 2016) models, replacing batch norm with group norm (Reddi et al., 2020; Hsieh et al., 2020).
Revised: ResNet18 (He et al., 2016) models, replacing batch norm with group norm (Reddi et al., 2020; Hsieh et al., 2020).

Original: For Stackoverflow and Reddit, we use the DistilGPT2 (HuggingFace, 2019) and CharLM (Kim et al., 2016) models.
Revised: For Stackoverﬂow and Reddit pushift.io, we use the DistilGPT2 (HuggingFace, 2019) and CharLM (Kim et al., 2016) models.

Original: For additional information about the datasets and models, see Appendix A.1.
Revised: For additional information about the datasets and models, see Appendix A.1.

Original: IID data partitioning.

Original: To understand the effect of data heterogeneity, we run some experiments using an IID partition of data across clients.

Original: In this case, we first group the entire training set (from all clients), randomly permute it and then partition it evenly across the same number of clients.

Original: 3. I NITIALIZATION S TRATEGIES.
Revised: 3. I NITIALIZATION S TRATEGIES

Original: We consider two initialization strategies: random initialization and supervised pre-training from public data.
Revised: We consider two initialization strategies: random initialization and supervised pre-training. [Table]

Original: Random initialization.
Revised: Random initialization.

Original: Most prior federated optimization works use random weights to initialize the model.
Revised: Most prior federated optimization works use random weights to initialize the model.

Original: We can use the same random initialization strategies used in the standard (centralized) training of deep networks for each model (Iandola et al., 2016; HuggingFace, 2019;
Revised: We can use the same random initialization strategies used in the standard (centralized) training of deep networks for each model (Iandola et al., 2016; HuggingFace, 2019;

Original: He et al., 2016;
Revised: He et al., 2016;

Original: Kim et al., 2016).
Revised: Kim et al., 2016).

Original: Supervised pre-training.
Revised: Supervised pre-training.

Original: In many FL applications, pre-training can be done on a large non-private proxy dataset available at the server.
Revised: In many FL applications, pre-training can be done on a large non-private proxy dataset available at the server.

Original: To quickly reproduce our results and compare them with other methods in the literature, we use publicly available pre-trained models in this study.
Revised: To facilitate easily reproducing our results, we use publicly available pre-trained models or pre-train on public data.

Original: For tasks using Squeezenet and ResNet18, we use the version of the model pre-trained on ImageNet, available in the PyTorch Torchvision library.
Revised: For tasks using Squeezenet and ResNet18, we use the version of the model pre-trained on ImageNet, available in the PyTorch Torchvision library.

Original: For tasks using DistilGPT2, we use the model weights provided in the HuggingFace library that has been distilled from a pre-trained GPT2 2 , and for tasks using CharLM , we pre-train the model on WikiText-103 (Merity et al., 2016)for 100 epochs.
Revised: For tasks using DistilGPT2, we use the model weights provided in the HuggingFace library that has been distilled from a pre-trained GPT2, 3 and for tasks using CharLM, we pre-train the model on WikiText-103 (Merity et al., 2016)

Original:  Supervised pre-training is just one possibility, and several other pre-training strategiesmay be considered (e.g., self-supervised pre-training or meta-learning). We leave the further investigation of these approaches as future work; see Section 8.
Revised: (see Appendix B.2 for details). Supervised pre-training is just one possibility, and we leave the investigation of other pre-training strategies (e.g., self-supervised pre-training and meta-learning) as future work; see Section 8.

Original: 3. A LGORITHMS
Revised: 3. A LGORITHMS

Original: We compare federated training 
Revised: We compare federated training with ﬁve

Original: with five different C LIENT O PT strategies
Revised: different C LIENT O PT strategies:

Original: : SGD clients perform standard stochastic gradient descent updates; Proximal (Li et al., 2018) clients perform F ED P ROX -style local updates; F ED P ROX was originally proposed to reduce client drift due to heterogeneity; Normalized Averaging (Wang et al., 2020) clients use F ED N OVA -style updates and aggregation to compensate for data imbalance across clients;
Revised: SGD clients perform standard stochastic gradient descent updates; Proximal (Li et al., 2018) clients perform F ED P ROX -style local updates; F ED P ROX was originally proposed to reduce client drift due to heterogeneity; Normalized Averaging (Wang et al., 2020) clients use F ED N OVA -style updates and aggregation to compensate for data imbalance across clients;

Original: M IME L ITE (Karimireddy et al., 2021)clients make use of an optimizer state (e g.
Revised: M IME L ITE (Karimireddy et al., 2021)

Original:  momentum buffer) from the server during local updates to reduce drift due to data heterogeneity;GD clients perform full-batch gradient updates; in this case, the update ∆ ti returned to the server is a
Revised: clients make use of an optimizer state (e.g., momentum buffer) from the server during local updates to reduce drift due to data heterogeneity;

Original:  full-batch gradient on client i ’s local training set evaluated at model parameters x t −
Revised: GD clients perform full-batch gradient updates; in this case, the update  ti returned to the server is a full-batch gradient on client i ’s local training set evaluated at model parameters x t  1 .

Original: At the server, we consider three strategies for S ERVER O PT .
Revised: At the server, we consider three strategies for S ERVER O PT .

Original: In all strategies, the server treats the averaged update ∆ t as a gradient.
Revised: In all strategies, the server treats the averaged update  t as a gradient.

Original: SGD the server updates the global model using stochastic gradient descent; when C LIENT O PT is also SGD, this is equivalent to F ED A VG (McMahan et al., 2016).
Revised: SGD the server updates the global model using stochastic gradient descent; when C LIENT O PT is also SGD, this is equivalent to F ED A VG (McMahan et al., 2016).

Original: SGD

Original:  with momentum the server updates the global model using SGD with momentum; when C LIENT O PT is SGD, this is equivalent to F ED A VG M (Hsu et al., 2019).
Revised: SGD with momentum the server updates the global model using SGD with momentum; when C LIENT O PT is SGD, this is equivalent to F ED A VG M (Hsu et al., 2019).

Original: Adam the server updates

Original: the

Original:  global model using the Adam optimizer; when C LIENT O PT is SGD, this is equivalent to F ED 
Revised: Adam the server updates the global model using the Adam optimizer; when C LIENT O PT is SGD, this is equivalent to F ED A DAM (Reddi et al., 2020).

Original: A DAM Reddi et al The method commonly referred to as F ED SGD (McMahan et al., 2017) is obtained when C LIENT O PT is full-batch gradient descent (GD) and S ERVER O PT is SGD, with η c = 1 and E = 1 .
Revised: The method commonly referred to as F ED SGD (McMahan et al., 2017) is obtained when C LIENT O PT is full-batch gradient descent (GD) and S ERVER O PT is SGD, with ⌘ c = 1 and E = 1 .

Original: [Figure] We focus on the above choices for C LIENT O PT and S ERVER O PT because they are reflective of the most widely-cited federated optimization methods, and they also represent a diverse set of possible choices available to the practitioner seeking to deploy cross-device federated training at scale.
Revised: We focus on the above choices for C LIENT O PT and S ERVER O PT because they are reﬂective of the most widely-cited federated optimization methods, and they also represent a diverse set of possible choices available to the practitioner seeking to deploy cross-device federated training at scale.

Original: 3. I MPLEMENTATION AND T UNING
Revised: 3. I MPLEMENTATION AND T UNING

Original: We repeat each experiment with three different seeds and report the average.
Revised: We repeat each experiment with three different seeds and report the average.

Original: For all algorithms, we run  hyperparameter sweeps to tune client and server learning rates η ℓ and η g , and the proximal penalty parameter µ for F ED P ROX . We use momentum parameter 0 for F ED AVG and 0.9 for F ED A VG M. For F ED A DAM , we fix β 1 = 0 .
Revised: For each algorithm, model, and dataset, we run a hyperparameter sweep to tune client and server learning rates ⌘ ` and ⌘ g , and the proximal penalty parameter µ for F ED P ROX ; see Appendix A for details.

Original: 9 and β 2 = 0 .

Original: 99 for Adam; see Appendix A for details.

Original: Unless otherwise specified, each client update entails running one local epoch with fixed batch size per task.
Revised: Unless otherwise speciﬁed, each client update entails running one local epoch with ﬁxed batch size per task.

Original: We perform 1050 rounds of training for Stackoverflow, 942 rounds of training for Reddit, 1000 rounds of training for CIFAR-10, and 1082 training rounds for FEMNIST.
Revised: We perform 1050 rounds of training for Stackoverﬂow, 1000 rounds of training for CIFAR-10,training rounds for FEMNIST.

Original: See Appendix B.1 for additional  implementation details.
Revised: See Appendix A.2 for additional details on implementation details.

Original: All experiments were performed usingPyTorch (Paszke et al., 2017) and the open-source federated learning simulation framework FLSim (FLSim Authors, 2022).
Revised: All experiments were performed using the open-source federated learning simulation framework FLSim FLSim Authors (2022).

Original:  T HE I MPACT OF P RE -T RAINING IN FL
Revised:  T HE I MPACT OF P RE -T RAINING IN FL

Original: In this section, we illustrate the benefits of pre-training in the federated setting and how pre-training can impact federated optimization algorithms behavior.
Revised: In this section, we illustrate the beneﬁts of pre-training in the federated setting and how pre-training can impact federated optimization algorithms behavior.

Original: Pre-training changes the ranking of federated optimization algorithms.
Revised: Pre-training changes the ranking of federated optimization algorithms.

Original: Our first result shows that initializing federated training from a pre-trained model can drastically change how federated optimization algorithms behave.

Original: In particular, if one sorts federated optimization methods based on their performance when starting from a random initialization, the order is substantially different from when using a pre-trained initialization.
Revised: If one sorts federated optimization methods based on their performance when starting from a random initialization, the order is substantially different from when using a pre-trained initialization.

Original: This section focuses on nine combinations of S ERVER O PT and C LIENT O PT on CIFAR-10 and Stack Overflow. In particular, we focus on local update  C LIENT O PTmethods and exclude full-batch gradient descent (GD) for now.
Revised: We focus on nine combinations of S ERVER O PT and C LIENT O PT , only using local update methods for C LIENT O PT and excluding full-batch gradient descent.

Original: Figure 2 presents the changes between pre-trained and random weight initializations.
Revised: We show the change in performance in Figure 1.

Original: Among the nine methods, Figure 2 shows the broader range in final model accuracy in the random initialization compared tothe pre-trained setting – random initialization’s range is over 20% compared to less than 15% for pre-trained on CIFAR-10.
Revised: First, observe that the span of ﬁnal accuracies is much smaller when starting from a pre-trained model. Second, all methods starting with pre-trained models achieve a better accuracy after the same number of steps compared to random models.

Original: We especially see this for  Stack Overflow, 5% range for random and only 2% for pre-trained.
Revised: Lastly, observe that the order of methods changes depending on the initialization. Although no particular method dominates across all workloads in Figure 1, F ED A DAM with SGD for C LIENT O PT performs consistently well when starting from a pre-trained model, especially on the two larger language modeling workloads, Stack Overﬂow and Reddit, and so we focus on studying F ED A DAM -SGD below. [Figure]

Original:  Figure 2 highlights the stark difference between the two initialization settings and the advantage of using simple SGD locally combined with an adaptive server optimizer such as F ED A DAM .
Revised: Faster convergence to better accuracy when starting from a pre-trained model. Figure 2 shows that, as one would hope, when starting from a pre-trained model, it is possible to achieve much better accuracy after a ﬁxed number of steps than when starting federated training from a random initialization. Note that the initial accuracy is not always substantially higher than a random initialization (See Table 2).

Original: Pre-training closes the accuracy gap between non-IID and IID.
Revised: Pre-training closes the accuracy gap between non-IID and IID.

Original: We study how pre-training and data heterogeneity affect convergence without system heterogeneity by fixing E i = 1 .
Revised: We study how pre-training and data heterogeneity affect convergence without system heterogeneity by ﬁxing the number of local epochs to E i = 1 .

Original: We compare
Revised: We compare

Original: F ED A DAM with SGD locally under IID, and Non-IID data splits. 
Revised: F ED A DAM -SGD under IID and Non-IID data splits. In

Original: In Figure 3, we report the average accuracy for FedAdam (Reddi et al., 2020) on  four datasets.
Revised: Figure 4, we report the average accuracy for FedAdam (Reddi et al., 2020) on the four datasets.

Original: As expected, randomly initialized models perform much worse than their pre-trained counterparts, and IID partitions yield better quality than non-IID.
Revised: As expected, randomly initialized models perform much worse than their pre-trained counterparts, and IID partitions yield better quality than non-IID.

Original: Surprisingly, the gap between models trained on IID data and models trained on non-IID 
Revised: Surprisingly, the gap between models trained on IID data and models trained on non-IID data is signiﬁcantly smaller when starting with pre-trained weights. Moreover, pre-training reduces the negative effects of data heterogeneity (i.e., client drift). As a result, we

Original: A cc u r acy
Revised: A cc u r acy

Original: Pre-trained
Revised: Pre-trained

Original: Random
Revised: Random

Original: Non-IID IID CIFARA cc u r acy 

Revised: Non-IID IID CIFARA cc u r acy

Original: Pre-trained
Revised: Pre-trained

Original: Random
Revised: Random

Original: Non-IID IID Stack Overflow
Revised: Non-IID IID Stack Overflow

Original: A cc u r acy
Revised: A cc u r acy

Original: Pre-trained
Revised: Pre-trained

Original: Random
Revised: Random

Original: Non-IID IID FEMNIST
Revised: Non-IID IID FEMNIST

Original: A cc u r acy
Revised: A cc u r acy

Original: Pre-trained
Revised: Pre-trained

Original: Random
Revised: Random

Original: Non-IID IID Redditdata is significantly smaller when starting with pre-trained weights. That is, the negative effects of data heterogeneity (i.e., client drift) is mitigated. As a result, we observe that (see Figure 6) when training from a pre-trained model, increasing the number of local updates does not degrade the final accuracy, in contrast to training from a random model.
Revised: Non-IID IID Reddit observe that (see Figure 3) when training from a pre-trained model, increasing the number of local updates does not degrade the ﬁnal accuracy, in contrast to training from a random model

Original: F ED A DAM GD is as effective as F ED A DAM SGD with pre-training.
Revised: F ED A DAM GD is as effective as F ED A DAM SGD with pre-training.

Original: The seminal work of McMahan et al.
Revised: The seminal work of McMahan et al.

Original: (2016) shows that taking local SGD steps before server averaging reduces communication by 10-100 × compared to taking a full batch gradient step.
Revised: (2016) shows that taking local SGD steps before server averaging reduces communication by 10-100 ⇥ compared to taking a full batch gradient step.

Original: To understand how pre-training impacts this comparison, we compare F ED A DAM with SGD and F ED A DAM with GD.
Revised: To understand how pre-training impacts this comparison, we compare F ED A DAM with SGD and F ED A DAM with GD.

Original: While local SGD can reduce communication, the saving is much less when the models are initialized with pretrained weights compared to random weights.
Revised: While local SGD can reduce communication, the saving is much less when the models are initialized with pre-trained weights compared to random weights.

Original: Figure 9 in the Appendix shows that with pre-trained initialization, using GD at the client can yield almost the same result as taking local SGD steps.
Revised: Figure 8 in the Appendix shows that with pre-trained initialization, using GD at the client can yield almost the same result as taking local SGD steps.

Original: Pre-training reduces the impact of system heterogeneity.
Revised: Pre-training reduces the impact of system heterogeneity.

Original: To study the impact of pre-training on system heterogeneity, we follow the setup described in (Li et al., 2018; Wang et al., 2020).
Revised: To study the impact of pre-training on system heterogeneity, we follow the setup described in (Li et al., 2018; Wang et al., 2020).

Original: We sample 30% of clients uniformly at random, and client i performs E i local epochs where E i ∼ U (1 , 5) . This models the setting where clients have different processing capabilities, and clients are asked to perform as much work as they can within a given time limit.
Revised: We sample 30% of clients uniformly at random, and client i performs E i local epochs where E i ⇠ U (1 , 5) while the remaining 70% of the clients perform E i = 1 epochs; this models the setting where clients have different processing capabilities and they perform as much work as they can within a given time limit.

Original: When starting from pre-trained weights, SGD locally outperforms otherC LIENT O PT methods on both datasets across most of S ERVER O PT except for F ED A VG on CIFAR-10 (Figure 4 left).
Revised: Figure 5 shows that F ED A DAM -SGD consistently outperforms other methods speciﬁcally designed for system heterogeneity (N OVA , P ROXIMAL , M IME L ITE ) when starting from a pre-trained model.

Original: We postulate that adding momentum to the server is enough to correct  the negative effect of system heterogeneity when starting from  pre-trained weights.
Revised: Apparently using an adaptive optimizer at the server is sufﬁcient to correct for the negative effects of systems heterogeneity when starting from a pre-trained model.

Original: On the other hand, when starting from  random weights, optimizers specifically designed for system heterogeneity outperform SGD.
Revised: On the other hand, when starting from a random initialization, optimizers speciﬁcally designed for system heterogeneity (i.e.,

Original:  (Figure 4 right).
Revised: F ED N OVA ) outperform SGD (Figure 5 right).

Original: Moreover, the accuracy gap between algorithms is more pronounced in the random initialization setting, whereas in the pre-trained setting, all algorithms converge to  similar accuracies.
Revised: Moreover, the accuracy gap between algorithms is more pronounced in the random initialization setting, whereas in the pre-trained setting, all algorithms converge to more similar accuracies.

Original: Our results suggest that pre-training reduces the need for algorithms that try to correct system heterogeneity.
Revised: Our results suggest that pre-training may reduce the need for algorithms that try to correct system heterogeneity.

Original: We present the results for other datasets in Appendix C.1.

Original:  U NDERSTANDING W HY P RE - TRAINING H ELPS F EDERATED O PTIMIZATION
Revised:  U NDERSTANDING W HY P RE - TRAINING H ELPS F EDERATED O PTIMIZATION

Original: While pre-training unsurprisingly speeds up convergence, the reason for the speedup is less apparent.
Revised: While pre-training unsurprisingly speeds up convergence, the reason for the speedup is less apparent.

Original: In this section, we examine why pre-training is beneficial to federated learning.
Revised: In this section, we examine why pre-training is beneﬁcial to federated learning.

Original: Pre-training helps align client updates.
Revised: Pre-training helps align client updates.

Original: To better understand why pre-training alleviates the heterogeneity challenge, we first investigate the gradient diversity of the updates received from different clients.
Revised: To better understand why pre-training alleviates the heterogeneity challenge, we ﬁrst investigate the gradient diversity of the updates received from different clients.

Original: We adopt the notion introduced in Yin et al.
Revised: We adopt the notion introduced in Yin et al.

Original: (2017), adapted here to apply to clientupdates:
Revised: (2017), adapted here to apply to client

Revised: [Table]

Revised: updates  i (whereas Yin et al.

Revised: (2017) focus speciﬁcally on gradients

Revised: g i ):

Original: [Equation]
Revised: [Equation]

Original: In Figure 5, we plot the gradient diversity of client updates ∆ i at each round for F ED A DAM .
Revised: In Figure 6, we plot the gradient diversity of client updates  i at each round for F ED A DAM .

Original: In the pre-trained setting, client updates have significantly lower gradient diversity (see the bottom left plot in Figure 5).
Revised: In the pre-trained setting, client updates have signiﬁcantly lower gradient diversity (see the bottom left plot in Figure 6).

Original: This suggests that when starting from a pre-trained model, the client local model changes are more similar to each other.
Revised: This suggests that when starting from a pre-trained model, the client local model changes are more similar to each other.

Original: On the other hand, clients local model changes from randomly initialized weights are almost orthogonal, suffering more from the client drift problem.
Revised: On the other hand, clients local model changes from randomly initialized weights are almost orthogonal, suffering more from the client drift problem.

Original: In addition, when looking at the cosine similarity of consecutive aggregated update vectors in time (bottom middle), we see that consecutive updates point more consistently in a similar direction at the beginning of training when starting from a pre-trained model.FedAvg FedAvgM FedAdam 67.
Revised: In addition, when looking at the cosine similarity of consecutive aggregated update vectors in time (bottom middle), we see that consecutive updates point more consistently in a similar direction at the beginning of training when starting from a pre-trained model.

Original: C

Original: I F A R10 P r e t r a i n e d SGD Proximal
Revised: From the top middle and bottom right plots in Figure 6, we see that the pre-trained model starts closer to the ﬁnal result.

Original: Nova

Original: MineLite 67. FedAvg FedAvgM FedAdam 40.
Revised: We also examine the largest eigenvalue of the Hessian matrix (i.e., local

Original: C

Original: I F

Original: A R10 R a n d o m

Original: SGD

Original: Proximal

Original: Nova

Original: MineLite 42.

Original: FedAvg

Original: FedAvgM FedAdam 86. FE M N I S T P r e t r a i n e d
Revised: Lipshitz constant or smoothness) at the beginning of training, a larger value of which suggests a harder-to-optimizer loss surface.

Original: SGD

Original: Proximal

Original: Nova

Original: MineLite 89.

Original: FedAvg

Original: FedAvgM

Original: FedAdam 82. FE M N I S T R a n d o m SGD
Revised: In Table 1, one can observe that pre-trained models always lead to smaller eigenvalues on different datasets.

Original: Proximal Nova MineLite 88. averaging (N OVA ), respectively.
Revised: Initial loss for pre-trained versus random models.

Original: We repeat each experiment for 3 different seeds and report the average [Figure] In addition [Figure]
Revised: Table 2 shows that pre-training does not always lead to lower initial loss.

Original: from the top middle and bottom right plots in Figure [Figure] we see that the pre-trained model starts closer to the final result. We also examine the largest eigenvalue of the Hessian matrix (i
Revised: For Squeezenet 1.0 on CIFAR-10 and ResNet-18 on FEMNIST, the initial loss of the randomly initialized models are lower pre-trained models.

Original: [Figure] e., local Lipshitz constant or smoothness) at the beginning of training, a larger value of which suggests a harder-to-optimizer loss surface. In Table 1, one can observe that pre-trained models always lead to smaller eigenvalues on different datasets.
Revised: However, pre-trained model still converges faster as illustrated in Figure 2.

Original: Connection to Existing Theory.
Revised: Connection to theory.

Original: Here, we present the existing optimization theory for F ED A VG and discuss how pre-training helps to improve the model convergence.
Revised: Here, we present the existing optimization theory for F ED A VG and discuss how pre-training helps to improve the model convergence.

Original: Following the formulation in Section 3, suppose  that there are total m clients, jointly optimizing a global objective function f ( w )
Revised: Following the formulation in Section 3, suppose [Table] that there are total m clients, jointly optimizing a global objective function f ( w )

Original: = (cid:80) mi =1 p
Revised: = [Equation]

Original: i F i ( w ) . Each client’s local loss function F i ( w ) is LLipschitz smooth.
Revised: and that each client’s local loss function F i ( w ) is L -Lipschitz smooth.

Original: Forthe ease of presentation, we assume that all clients participate into training and A cc u r acy E = E =
Revised: For ease of presentation, we assume that all clients participate in training and perform K local SGD updates at each round.

Original: E =

Original: E =

Original: Pre-trained

Original: Random [Table] perform K local SGD updates at each round. Then, under standard assumptions, one can show that after R communication rounds, the expected gradient norm satisfies (see Theorem V in Karimireddy et al.
Revised: Then, under standard assumptions, one can show that after R communication rounds, the expected gradient norm satisﬁes (see Theorem V in Karimireddy et al.

Original: [Equation]
Revised: [Equation]

Original: where ¯ w R represents a weighted sampled model from all previous rounds, ζ is a measure of data heterogeneity, and F = f ( x 0 ) − f ∗ denotes the gap between the initial loss value and the optimal loss value.When starting from a pre-trained model, the initial gap F becomes lower, as observed in
Revised: where ¯ w R represents a weighted sampled model from all previous rounds, ⇣ is a measure of data heterogeneity, and F = f ( x 0 )  f ⇤ denotes the gap between the initial loss value and the optimal loss value.

Original: Figure 5 (top left) on CIFAR-10.

Original: As a result, the optimization error upper bound (2) will be smaller, i.e., we get a better worst-case performance.

Original: We need to mention that a lower initial training loss is only one possible reason for the siginificantly improved training performance.

Original: We also observe that on some training tasks (details are in the Appendix), when using a pre-trained model, the initial training loss can be higher but the convergence speed is still faster.

Original: This calls for a new optimization theory to explain this phenomenon.

Original: In addition, in order for F ED A VG to achieve the O (1 / √ RKm ) asymptotic convergence rate, previous works (Wang et al., 2021;
Revised: In addition, in order for F ED A VG to achieve the O (1 / p RKm ) asymptotic convergence rate, previous works (Wang et al., 2021;

Original: Woodworth et al., 2020; Karimireddy et al., 2020; Wang and Joshi, 2021) showed that the number of local updates K should be upper bounded as follows:
Revised: Woodworth et al., 2020; Karimireddy et al., 2020; Wang and Joshi, 2021) showed that the number of local updates K should be upper bounded as follows:

Original: [Equation]
Revised: [Equation]

Original: Clearly, if F becomes smallerwhen starting from a pre-trained model, one can use a larger number of local updates.
Revised: Clearly, if F becomes smaller starting from a pre-trained model, one can use a larger number of local updates.

Original: This corroborates our empirical observations in Figure 6.
Revised: This corroborates our empirical observations in Figure 3.

Revised: When starting from a pre-trained model, the initial gap F is sometimes reduced, as observed in

Revised: Table 2.

Revised: As a result, the optimization error upper bound (2) will be smaller, i.e., we get better worst-case performance.

Revised: However a lower initial loss is not always observed in our experiments, so this does not fully explain our observations, suggesting that we may need to re-think the convergence theory of local update methods.

Original:  R ECOMMENDATIONS
Revised:  R ECOMMENDATIONS

Original: In this work, we study the affect of pre-training on federated optimization methods.
Revised: In this work, we study the effects of pre-training on federated optimization methods.

Original: Our results informa series of the following recommendations:
Revised: Our results inform the following recommendations:

Original: • When evaluate FL algorithms, researchers should experiment with both pre-trained (if available) and random weights as they have different behaviors.
Revised: When evaluating FL algorithms, researchers should experiment with both pre-trained (if available) and random weights, as the initialization can clearly impact the relative performance of different methods, and both initialization schemes may be relevant to practice.

Original: • When deploying FL to  production environment, researchers should use adaptive server optimizers such as FedAdam and SGD at client. This setup works well and should be used a baseline before trying out more complex methods.
Revised: When deploying FL to a production environment, using adaptive server optimizers such as F ED A DAM together with SGD at the client is a simple and competitive approach when it is possible to start from a pre-trained model.

Original: • Heterogeneity is not as a big of a problem when there is public data to pre-trained a model.
Revised: When there is public data to pre-train a model, the impact of heterogeneity can be reduced.

Original: We encourage researchers to pay attention other more complex tasks when there is no public data such as recommendation systems or semi-supervised learning.
Revised: Thus, when focusing on heterogeneity, it may be worth considering whether or not proxy data is available for pre-training to motivate the application considered.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Transfer learning.
Revised: Transfer learning.

Original: Model initialization can significantly impact training and final performance.
Revised: Model initialization can signiﬁcantly impact training and ﬁnal performance.

Original: Previous work studying the loss landscape of deep networks observed significant differences between the landscape around a random initialization and the landscape later in training.
Revised: Previous work studying the loss landscape of deep networks observed signiﬁcant differences between the landscape around a random initialization and the landscape later in training.

Original: In particular, later in training, the loss can be much more “well-behaved” (Li et al., 2017; Frankle et al., 2020; 2019).
Revised: In particular, later in training, the loss can be much more “well-behaved” (Li et al., 2017; Frankle et al., 2020; 2019).

Original: Fine-tuning from pre-trained models is common practice in natural language processing and computer vision, yielding strong performance on many tasks (Radford et al., 2019; Dosovitskiy et al., 2020;
Revised: Fine-tuning from pre-trained models is common practice in natural language processing and computer vision, yielding strong performance on many tasks (Radford et al., 2019; Dosovitskiy et al., 2020;

Original: Devlin et al., 2018;
Revised: Devlin et al., 2018;

Original: He et al., 2019).
Revised: He et al., 2019).

Original: Federated Optimization.
Revised: Federated Optimization.

Original: While a significant amount of research focused on various aspects of
Revised: While a signiﬁcant amount of research focused on various aspects of

Original: FL, including communication-efficiency McMahan et al.
Revised: FL, including communication-efﬁciency McMahan et al.

Original: (2016), data and systems heterogeneity Li et al.
Revised: (2016), data and systems heterogeneity Li et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: (2020), and faster convergence rate .
Revised: (2020), and faster convergence rate .

Original: Nearly all previous work in this field neglect the importance of initialization.
Revised: Nearly all previous work in this ﬁeld neglect the importance of initialization.

Original: In our work, we study the impact of initialization on federated optimization in the cross-device setting.
Revised: In our work, we study the impact of initialization on federated optimization in the cross-device setting.

Original: We defer the interested reader to surveys of Kairouz et al.
Revised: We defer the interested reader to surveys of Kairouz et al.

Original: (2019) and Wang et al.
Revised: (2019) and Wang et al.

Original: (2021) for additional background.
Revised: (2021) for additional background.

Original: Pre-training in Federated Learning.
Revised: Pre-training in Federated Learning.

Original: Very few works have studied pre-trained models in federated learning Pillutla et al.
Revised: Very few works have studied pre-trained models in federated learning Pillutla et al.

Original: ; Hsu et al.
Revised: ; Hsu et al.

Original: Zhao et al.
Revised: Zhao et al.

Original: Lin et al.
Revised: Lin et al.

Original: ; Stremmel and Singh (2021).
Revised: ; Stremmel and Singh (2021).

Original: Zhao et al.
Revised: Zhao et al.

Original: (2018) studied pre-training as a mechanism to remedy the adverse effect of heterogeneity in FL.
Revised: (2018) studied pre-training as a mechanism to remedy the adverse effect of heterogeneity in FL.

Original: However, Zhao et al.
Revised: However, Zhao et al.

Original: (2018) found that pre-trained initialization does not alleviate the effect of heterogeneity.
Revised: (2018) found that pre-trained initialization does not alleviate the effect of heterogeneity.

Original: In this work, we find that pre-training can alleviate both data and system heterogeneity.
Revised: In this work, we ﬁnd that pre-training can alleviate both data and system heterogeneity.

Original: Pillutla et al.
Revised: Pillutla et al.

Original: ; Hsu et al.
Revised: ; Hsu et al.

Original: Lin et al.
Revised: Lin et al.

Original: (2021); Stremmel and Singh (2021) experimented with pre-trained models but did not study the difference between random initialization and pre-training, which is the focus of this work.
Revised: (2021); Stremmel and Singh (2021) experimented with pre-trained models but did not study the difference between random initialization and pre-training, which is the focus of this work.

Original: In concurrent and independent works, Chen et al.
Revised: In concurrent and independent works, Chen et al.

Original: ; Weller et al.
Revised: ; Weller et al.

Original: (2022) find that pre-training closes the accuracy gap between
Revised: ﬁnd that pre-training closes the accuracy gap between

Original: F ED A VG and centralized learning under non-IID data.
Revised: F ED A VG and centralized learning under non-IID data.

Original: Weller et al.
Revised: Weller et al.

Original: (2022) studied multilingual language tasks using large transformer models on synthetically partitioned data, not a realistic setup for large-scale cross-device federated settings.
Revised: (2022) studied multilingual language tasks using large transformer models on synthetically partitioned data, not a realistic setup for large-scale cross-device federated settings.

Original: Chen et al.
Revised: Chen et al.

Original: (2022) studied the impact of pre-training on data heterogeneity using synthetically partitioned image data.
Revised: (2022) studied the impact of pre-training on data heterogeneity using synthetically partitioned image data.

Original: Furthermore, Chen et al.
Revised: Furthermore, Chen et al.

Original: (2022) proposed a method to pre-train with synthetic data.
Revised: (2022) proposed a method to pre-train with synthetic data.

Original: In our work, we systematically study both forms of heterogeneity, data-induced and system-induced, across both visual and language tasks on
Revised: In our work, we systematically study both forms of heterogeneity, data-induced and system-induced, across both visual and language tasks on

Original: SOTA federated optimization algorithms.
Revised: SOTA federated optimization algorithms.

Original: We offer theoretical and empirical explanations for why pre-training is beneficial to FL.
Revised: We offer theoretical and empirical explanations for why pre-training is beneﬁcial to FL.

Original:  C ONCLUSION AND L IMITATIONS
Revised:  C ONCLUSION AND L IMITATIONS

Original: Limitations.
Revised: Limitations.

Original: Depending on the application, it may not be possible to get public data, in which case random initialization may be the only option.
Revised: Depending on the application, it may not be possible to get public data, in which case random initialization may be the only option.

Original: Nevertheless, we believe there is sufficient prevalence and importance of applications where public data is available for this study to be of broad interest.
Revised: Nevertheless, we believe there is sufﬁcient prevalence and importance of applications where public data is available for this study to be of broad interest.

Original: When public data is available, it may not necessarily reflect the distribution of all users in the population.
Revised: When public data is available, it may not necessarily reﬂect the distribution of all users in the population.

Original: Consequently, pre-training using public data may introduce bias, which warrants further study, including methods to detect and mitigate such bias.
Revised: Consequently, pre-training using public data may introduce bias, which warrants further study, including methods to detect and mitigate such bias.

Original: Moreover, we only consider one warm-start initialization strategy type, supervised pre-training.
Revised: Moreover, we only consider one warm-start initialization strategy: supervised pre-training.

Original: Several other possibilities may be natural, including meta-learning the warm-start initialization and self-supervised pre-training (e.g., if public data does not come with labels).
Revised: Several other possibilities are worth investigating, including meta-learning the warm-start initialization and self-supervised pre-training (e.g., if public data does not come with labels).

Original: Conclusion.
Revised: Conclusion.

Original: In this paper we present a thorough empirical analysis of initialization on federated learning by evaluating it on twelve federated learning algorithms across four vision and text tasks.
Revised: In this paper, we present a thorough empirical analysis of initialization on federated learning by evaluating it on twelve federated learning algorithms across four vision and text tasks.

Original: We find that pre-training on public data can recover most of the accuracy drop from heterogeneity.
Revised: We ﬁnd that pre-training on public data can recover most of the accuracy drop from heterogeneity.

Original: We show that client updates starting from pre-trained weights have higher cosine similarity, which explains why initialized with pre-trained weights can speed up convergence and achieve high accuracy even in heterogeneous settings.
Revised: We show that client updates starting from pre-trained weights have higher cosine similarity, which explains why initializing with pre- trained weights can speed up convergence and achieve high accuracy even in heterogeneous settings.

Original: We further show that using simple SGD locally can be as good as other local optimizers.
Revised: We further show that using simple SGD locally can be as good as other local optimizers

Original: A E XPERIMENT D ETAILS A. D ATASETS AND M ODELS B A LGORITHMS S UMMARY

Original: CIFAR-10.

Original: We evaluate a multi-class image classification problem on CIFAR-10 Krizhevsky et al.

Original: (2009) using a SqueezeNet Iandola et al.

Original: We normalize the images by the dataset mean and standard deviation.

Original: Following Hsu et al.

Original: (2019), we partition the dataset using a Dirichlet distribution with parameter 0.1.

Original: The statistics on the number of clients and examples in both the training and test splits of the datasets are in Table 2.

Original: Stack Overflow Stack Overflow consists of questions and answers from Stack Overflow.

Original: We experiment a next-word-prediction task using a DistilGPT-2 model with a casual LM head.

Original: We perform padding and truncation to ensure that each sentences have 25 words.

Original: We then use a GPTtokenizer to encode the tokens.

Original: FEMNIST Federated EMNIST-62 (FEMNIST) consists of digits and English characters, totalingclasses.

Original: We evaluate a multi-class image classification problem on the federated version Caldas et al.

Original: (2018) which partitions the digits by the writer and filter out clients that less than one example.

Original: As for the model, we use a SqueezeNet 1.0 Iandola et al.

Original: Since FEMNIST contains grayscale images, we replicated the one channel value into three channels with the same values.

Original: [Table]

Original: [Table]

Original: C

Original: I F

Original: A R10 P r e t r a i n e d

Original: [Table]

Original: [Table]

Original: n

Original: [Table]

Original: [Table]

Original: N I S T

Original: P r e t r a i n e d

Original: [Table]

Original: [Table]

Original: S t a c k

Original: O v e r f l o w P r e - t r a i n e d

Original: SGD

Original: [Table]

Original: Nova

Original: [Table]

Original: t

Original: [Table]

Original: r f

Original: [Figure]

Original: o w

Original: [Figure]

Original: n

Original: [Figure]

Original: B. I MPLEMENTATION D ETAILS

Original: We implemented all algorithms in Pytorch (Paszke et al., 2017) and evaluated them on a cluster of machines, each with eight NVidia V100 GPUs.

Original: We evaluate our experiments in FLSim 3 .

Original: For all experiments, we tune hyperparameters using Bayesian optimization (Snoek et al., 2012).

Original: We select the best hyperparameters based the final accuracy after a fixed number of rounds for each dataset.

Original: C A DDITIONAL R ESULTS C. S YSTEM H ETEROGENEITY

Original: In this section, we show the system heterogeneity results for all three datasets: CIFAR-10, FEMNIST, and Stack Overflow.

Original: With the exception of Stack Overflow, F ED A DAM SGD consistently outperforms other methods specifically designed for system heterogeneity (N OVA , P ROXIMAL ) in the the pretrained setting.

Original: C. F INE - TUNING ONLY THE LAST LAYER

Original: In this section, we present the results for fine-tuning only the last linear layer rather in the model as commonly done in practice.

Original: Figure 8 shows that fine-tuning only the last layer might not yield optimal model quality and should be consider carefully.

Original: While fine-tuning only the last layer can achieve close to full fine-tuning on Stack Overflow, the performance is much worse on CIFAR-10.

Original: C. F ED A DAM GD IS AS EFFECTIVE AS F ED A DAM SGD WITH PRE - TRAINING .

Original: [John: Move figure 6 to the appendix]

Original: [Figure]

Original: [Figure]