Original: T OWARDS
Revised: T OWARDS

Original: F EDERATED
Revised: F EDERATED

Original: L EARNING
Revised: L EARNING

Original: ON
Revised: ON

Original: T IMEE VOLVING H ETEROGENEOUS D ATA
Revised: T IMEE VOLVING H ETEROGENEOUS D ATA

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Federated Learning (FL) is an emerging learning paradigm that preserves privacy by ensuring client data locality on edge devices.
Revised: Federated Learning (FL) is an emerging learning paradigm that preserves privacy by ensuring client data locality on edge devices.

Original: The optimization of FL is challenging in practice due to the diversity and heterogeneity of the learning system.
Revised: The optimization of FL is challenging in practice due to the diversity and heterogeneity of the learning system.

Original: Despite recent research efforts on improving the optimization of heterogeneous data, the impact of time-evolving heterogeneous data in real-world scenarios, such as changing client data or intermittent clients joining or leaving during training, has not been well studied.
Revised: Despite recent research efforts on improving the optimization of heterogeneous data, the impact of time-evolving heterogeneous data in real-world scenarios, such as changing client data or intermittent clients joining or leaving during training, has not been well studied.

Original: In this work, we propose Continual Federated Learning (CFL), a ﬂexible framework, to capture the time-evolving heterogeneity of FL.
Revised: In this work, we propose Continual Federated Learning (CFL), a ﬂexible framework, to capture the time-evolving heterogeneity of FL.

Original: CFL covers complex and realistic scenarios—which are challenging to evaluate in previous FL formulations— by extracting the information of past local datasets and approximating the local objective functions.
Revised: CFL covers complex and realistic scenarios—which are challenging to evaluate in previous FL formulations—by extracting the information of past local datasets and approximating the local objective functions.

Original: Theoretically, we demonstrate that CFL methods achieve a faster convergence rate than FedAvg in time-evolving scenarios, with the beneﬁt being dependent on approximation quality.
Revised: Theoretically, we demonstrate that CFL methods achieve a faster convergence rate than FedAvg in time-evolving scenarios, with the beneﬁt being dependent on approximation quality.

Original: In a series of experiments, we show that the numerical ﬁndings match the convergence analysis, and CFL methods signiﬁcantly outperform the other SOTA FL baselines.
Revised: In a series of experiments, we show that the numerical ﬁndings match the convergence analysis, and CFL methods signiﬁcantly outperform the other SOTA FL baselines.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Federated Learning (FL) has recently emerged as a critical distributed machine learning paradigm to preserve user/client privacy.
Revised: Federated Learning (FL) has recently emerged as a critical distributed machine learning paradigm to preserve user/client privacy.

Original: Clients engaged in the training process of FL only communicate their local model parameters, rather than their private local data, with the central server.
Revised: Clients engaged in the training process of FL only communicate their local model parameters, rather than their private local data, with the central server.

Original: As the workhorse algorithm in FL, FedAvg (McMahan et al., 2017) performs multiple local stochastic gradient descent (SGD) updates on the available clients before communicating with the server.
Revised: As the workhorse algorithm in FL, FedAvg (McMahan et al., 2017) performs multiple local stochastic gradient descent (SGD) updates on the available clients before communicating with the server.

Original: Despite its success, FedAvg suffers from the large heterogeneity (non-iid-ness) in the data presented on the different clients, causing drift in each client’s updates and resulting in slow and unstable convergence (Karimireddy et al., 2020b).
Revised: Despite its success, FedAvg suffers from the large heterogeneity (non-iid-ness) in the data presented on the different clients, causing drift in each client’s updates and resulting in slow and unstable convergence (Karimireddy et al., 2020b).

Original: To address this issue, a new line of study has been suggested lately that either simulates the distribution of the whole dataset using preassigned weights of clients (Wang et al., 2020; Reisizadeh et al., 2020;
Revised: To address this issue, a new line of study has been suggested lately that either simulates the distribution of the whole dataset using preassigned weights of clients (Wang et al., 2020; Reisizadeh et al., 2020;

Original: Mohri et al., 2019; Li et al., 2020a) or adopts variance reduction methods (Karimireddy et al., 2020b;a; Das et al., 2020;
Revised: Mohri et al., 2019; Li et al., 2020a) or adopts variance reduction methods (Karimireddy et al., 2020b;a; Das et al., 2020;

Original: Haddadpour et al., 2021).
Revised: Haddadpour et al., 2021).

Original: However, the FL formulation in these approaches always assumes a ﬁxed data distribution among clients throughout all training rounds, while in practice this assumption does not always hold because of the complex, uncontrolled, and unpredictable client behaviors.
Revised: However, the FL formulation in these approaches always assumes a ﬁxed data distribution among clients throughout all training rounds, while in practice this assumption does not always hold because of the complex, uncontrolled, and unpredictable client behaviors.

Original: Local datasets, for example, often vary over time, and intermittent clients can join or depart during training without prior notice.
Revised: Local datasets, for example, often vary over time, and intermittent clients can join or depart during training without prior notice.

Original: The difﬁculty in addressing the time-evolving heterogeneity of FL lies in the stateless nature of the local datasets, which includes the unpredictable future datasets and the impossibility of retaining all prior local datasets.
Revised: The difﬁculty in addressing the time-evolving heterogeneity of FL lies in the stateless nature of the local datasets, which includes the unpredictable future datasets and the impossibility of retaining all prior local datasets.

Original: To this end, we ﬁrst provide a novel Continual Federated Learning (CFL) formulation and then propose a uniﬁed CFL framework as our solution.
Revised: To this end, we ﬁrst provide a novel Continual Federated Learning (CFL) formulation and then propose a uniﬁed CFL framework as our solution.

Original: This framework encompasses a variety of design choices for approximating the local objective functions in the previous training rounds, with the difference between actual and estimated local object functions described as information loss for further analysis.
Revised: This framework encompasses a variety of design choices for approximating the local objective functions in the previous training rounds, with the difference between actual and estimated local object functions described as information loss for further analysis.

Original: To capture the time-evolving data heterogeneity in the CFL framework, we expand the theoretical assumption on the client drift—which has been extensively utilized recently in Karimireddy et al.
Revised: To capture the time-evolving data heterogeneity in the CFL framework, we expand the theoretical assumption on the client drift—which has been extensively utilized recently in Karimireddy et al.

Original: (2020b); Khaled et al.
Revised: (2020b); Khaled et al.

Original: ; Li et al.
Revised: ; Li et al.

Original: (2020b)—to include both client drift and time drift .
Revised: (2020b)—to include both client drift and time drift .

Original: This allows us to quantify the difference between local client objective function and global objective function for the considered time-evolving heterogeneous clients.
Revised: This allows us to quantify the difference between local client objective function and global objective function for the considered time-evolving heterogeneous clients.

Original: We provide convergence rates for our uniﬁed CFL framework in conjunction with information loss and new models of both client and time drifts.
Revised: We provide convergence rates for our uniﬁed CFL framework in conjunction with information loss and new models of both client and time drifts.

Original: Our analysis reveals a faster convergence of CFL framework than FedAvg, with the beneﬁt dependent on approximation quality.
Revised: Our analysis reveals a faster convergence of CFL framework than FedAvg, with the beneﬁt dependent on approximation quality.

Original: TheFedAvg rate  obtained from our framework is consistent with previous work, while a similarly simpliﬁed rate on Continual Learning (CL) (French, 1999; Kirkpatrick et al., 2017) for the convex and strongly convex cases is novel.
Revised: The rate of FedAvg obtained from our framework is consistent with previous work, while a similarly simpliﬁed rate on Continual Learning (CL) (French, 1999; Kirkpatrick et al., 2017) for the convex and strongly convex cases is novel.

Original: Finally, in extensive empirical results, we demonstrate that CFL methods—stemmed from the CFL framework with different approximation techniques—signiﬁcantly outperform the SOTA FL competitors on various realistic datasets.
Revised: Finally, in extensive empirical results, we demonstrate that CFL methods—stemmed from the CFL framework with different approximation techniques—signiﬁcantly outperform the SOTA FL competitors on various realistic datasets.

Original: These numerical observations corroborate our theoretical ﬁndings.
Revised: These numerical observations corroborate our theoretical ﬁndings.

Original: Contribution.
Revised: Contribution.

Original: We summarize our key contributions below.
Revised: We summarize our key contributions below.

Original: • We present a uniﬁed framework, termed Continual Federated Learning (CFL), together with a novel client and time drift modeling approach, to capture complex FL scenarios involving timeevolving heterogeneous data.
Revised: • We present a uniﬁed framework, termed Continual Federated Learning (CFL), together with a novel client and time drift modeling approach, to capture complex FL scenarios involving timeevolving heterogeneous data.

Original: This is the ﬁrst theoretical study, to our knowledge, that describes the time-varying nature of
Revised: This is the ﬁrst theoretical study, to our knowledge, that describes the time-varying nature of

Original: FL. •
Revised: FL. •

Original: We provide rigorous convergence analysis for the CFL methods.
Revised: We provide rigorous convergence analysis for the CFL methods.

Original: Our theoretical analysis explains the faster and more stabilized optimization of the CFL methods over that of FedAvg, and conjecture their variance reduction effect.
Revised: Our theoretical analysis explains the faster and more stabilized optimization of the CFL methods over that of FedAvg, and conjecture their variance reduction effect.

Original: In addition, we provide tight convergence rates for standalone CL methods on convex and strongly-convex problems: to the best of our knowledge, we are the ﬁrst to provide such guarantees for CL on SGD.
Revised: In addition, we provide tight convergence rates for standalone CL methods on convex and strongly-convex problems: to the best of our knowledge, we are the ﬁrst to provide such guarantees for CL on SGD.

Original: • We examine several approximation techniques for CFL framework: the insights therein offer a valuable practical guideline.
Revised: • We examine several approximation techniques for CFL framework: the insights therein offer a valuable practical guideline.

Original: We demonstrate the efﬁcacy and necessity of CFL methods over the SOTA FL baselines across a range of time-varying heterogeneous data scenarios and datasets.
Revised: We demonstrate the efﬁcacy and necessity of CFL methods over the SOTA FL baselines across a range of time-varying heterogeneous data scenarios and datasets.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Federated Learning.
Revised: Federated Learning.

Original: FedAvg (McMahan et al., 2017; Lin et al., 2020b) is the de facto standard
Revised: FedAvg (McMahan et al., 2017; Lin et al., 2020b) is the de facto standard

Original: FL algorithm, in which multiple local SGD steps are executed on the available clients to alleviate the communication bottleneck.
Revised: FL algorithm, in which multiple local SGD steps are executed on the available clients to alleviate the communication bottleneck.

Original: While communication efﬁcient, heterogeneity, such as system heterogeneity (Li et al., 2018; Wang et al., 2020; Mitra et al., 2021; Diao et al., 2021) and statistical/objective heterogeneity (Li et al., 2018; Wang et al., 2020; Mitra et al., 2021; Lin et al., 2020a; Karimireddy et al., 2020b;a), results in inconsistent optimization objectives and drifted clients models, impeding federated optimization considerably.
Revised: While communication efﬁcient, heterogeneity, such as system heterogeneity (Li et al., 2018; Wang et al., 2020; Mitra et al., 2021; Diao et al., 2021) and statistical/objective heterogeneity (Li et al., 2018; Wang et al., 2020; Mitra et al., 2021; Lin et al., 2020a; Karimireddy et al., 2020b;a), results in inconsistent optimization objectives and drifted clients models, impeding federated optimization considerably.

Original: A line of work has been proposed to address the heterogeneity in FL.
Revised: A line of work has been proposed to address the heterogeneity in FL.

Original: FedProx (Li et al., 2018) adds the regularization term on the distance of local and global models when performing local training—similar formulations can be found in other recent FL works (Hanzely & Richtárik, 2020;
Revised: FedProx (Li et al., 2018) adds the regularization term on the distance of local and global models when performing local training—similar formulations can be found in other recent FL works (Hanzely & Richtárik, 2020;

Original: Dinh et al., 2020; Li et al., 2021) for various purposes.
Revised: Dinh et al., 2020; Li et al., 2021) for various purposes.

Original: To address the issue of objective heterogeneity e.g. caused by heterogeneous data, works like SCAFFOLD (Karimireddy et al., 2020b; Mitra et al., 2021) introduce the idea of variance reduction on the client local update steps.
Revised: To address the issue of objective heterogeneity e.g. caused by heterogeneous data, works like SCAFFOLD (Karimireddy et al., 2020b; Mitra et al., 2021) introduce the idea of variance reduction on the client local update steps.

Original: FedNova (Wang et al., 2020) further proposes a general framework for unifying FedAvg and FedProx, and argues that while averaging, local updates should be normalized to minimize heterogeneity induced by different number of local update steps.
Revised: FedNova (Wang et al., 2020) further proposes a general framework for unifying FedAvg and FedProx, and argues that while averaging, local updates should be normalized to minimize heterogeneity induced by different number of local update steps.

Original: However, most of these prior works focus on the ﬁxed heterogeneity across clients and throughout the entire optimization procedure; we instead consider the novel scenario with time-evolving data heterogeneity.
Revised: However, most of these prior works focus on the ﬁxed heterogeneity across clients and throughout the entire optimization procedure; we instead consider the novel scenario with time-evolving data heterogeneity.

Original: The theoretical study on the convergence of FedAvg can date back to the parallel SGD analysis on the identical functions Zinkevich et al.
Revised: The theoretical study on the convergence of FedAvg can date back to the parallel SGD analysis on the identical functions Zinkevich et al.

Original: and recently is improved by Stich (2019);
Revised: and recently is improved by Stich (2019);

Original: Stich &
Revised: Stich &

Original: Karimireddy (2020); Patel & Dieuleveut (2019); Khaled et al.
Revised: Karimireddy (2020); Patel & Dieuleveut (2019); Khaled et al.

Original: Woodworth et al.
Revised: Woodworth et al.

Original: (2020b).
Revised: (2020b).

Original: For the analysis of heterogeneous data, Li et al.
Revised: For the analysis of heterogeneous data, Li et al.

Original: (2020b)
Revised: (2020b)

Original: ﬁrst give the convergence rate of FedAvg on non-iid datasets with random selection, assuming that the client optimum are (cid:15) -close.
Revised: ﬁrst give the convergence rate of FedAvg on non-iid datasets with random selection, assuming that the client optimum are (cid:15) -close.

Original: Woodworth et al.
Revised: Woodworth et al.

Original: (2020a); Khaled et al.
Revised: (2020a); Khaled et al.

Original: (2020) give tighter convergence rates under the assumption of bounded gradient drift.
Revised: (2020) give tighter convergence rates under the assumption of bounded gradient drift.

Original: All above works give a O (1 /T ) convergence rate for convex local objective functions.
Revised: All above works give a O (1 /T ) convergence rate for convex local objective functions.

Original: More recently, Karimireddy et al. (2020b);
Revised: More recently, Karimireddy et al. (2020b);

Original: Koloskova et al.
Revised: Koloskova et al.

Original: (2020) give the convergence analysis of local SGD for non-convex objective functions under bounded gradient noise assumptions, and obtain a O (1 / √ T ) convergence rate.
Revised: (2020) give the convergence analysis of local SGD for non-convex objective functions under bounded gradient noise assumptions, and obtain a O (1 / √ T ) convergence rate.

Original: Our theoretical analysis framework covers more challenging timevarying data heterogeneity in FL, which has not been considered in the community yet—our rate can be simpliﬁed to the standard FL scenario, matching the tight analysis in Karimireddy et al. (2020b).
Revised: Our theoretical analysis framework covers more challenging timevarying data heterogeneity in FL, which has not been considered in the community yet—our rate can be simpliﬁed to the standard FL scenario, matching the tight analysis in Karimireddy et al. (2020b).

Original: Continual Learning.
Revised: Continual Learning.

Original: Continual learning, also known as incremental learning or lifelong learning, aims to learn from (time-varying) sequential data while avoiding the problem of catastrophic forgetting (French, 1999; Kirkpatrick et al., 2017).
Revised: Continual learning, also known as incremental learning or lifelong learning, aims to learn from (time-varying) sequential data while avoiding the problem of catastrophic forgetting (French, 1999; Kirkpatrick et al., 2017).

Original: There exists a large amount of works, from the perspectives of regularization (Kirkpatrick et al., 2017; Li & Hoiem, 2017; Zenke et al., 2017), experience replay (Castro et al., 2018; Rebufﬁ et al., 2017), and dynamic architectures (Maltoni &
Revised: There exists a large amount of works, from the perspectives of regularization (Kirkpatrick et al., 2017; Li & Hoiem, 2017; Zenke et al., 2017), experience replay (Castro et al., 2018; Rebufﬁ et al., 2017), and dynamic architectures (Maltoni &

Original: Lomonaco, 2019; Rusu et al., 2016).
Revised: Lomonaco, 2019; Rusu et al., 2016).

Original: In this paper, we examine both regularization and experience replay based methods, and compare their empirical performance in depth.
Revised: In this paper, we examine both regularization and experience replay based methods, and compare their empirical performance in depth.

Original: Despite the empirical success, the theoretical analysis of CL is limited: only a very recent preprint (Yin et al., 2020b) provides a viewpoint of regularization-based continual learning, and only for the single worker scenario.
Revised: Despite the empirical success, the theoretical analysis of CL is limited: only a very recent preprint (Yin et al., 2020b) provides a viewpoint of regularization-based continual learning, and only for the single worker scenario.

Original: In this work, we provide tight convergence analysis for both CL and CFL on SGD.
Revised: In this work, we provide tight convergence analysis for both CL and CFL on SGD.

Original: Continual Federated Learning.
Revised: Continual Federated Learning.

Original: To our knowledge, the scenario of CFL was originally described in Bui et al.
Revised: To our knowledge, the scenario of CFL was originally described in Bui et al.

Original: (2018) in order to federated train Bayesian Neural Network and continually learn for
Revised: (2018) in order to federated train Bayesian Neural Network and continually learn for

Original: Gaussian Process models—it is orthogonal to our optimization aspect in this paper.
Revised: Gaussian Process models—it is orthogonal to our optimization aspect in this paper.

Original: FedCurv (Shoham et al., 2019) blends EWC (Kirkpatrick et al., 2017) regularization with FedAvg, and empirically shows a faster convergence.
Revised: FedCurv (Shoham et al., 2019) blends EWC (Kirkpatrick et al., 2017) regularization with FedAvg, and empirically shows a faster convergence.

Original: FedWeIT (Yoon et al., 2021) extends the regularization idea, with a focus on selective inter-client knowledge transfer for task-adaptive parameters.
Revised: FedWeIT (Yoon et al., 2021) extends the regularization idea, with a focus on selective inter-client knowledge transfer for task-adaptive parameters.

Original: In a very recent parallel (and empirical) work, CDA-FedAvg (Casado et al., 2021) uses a locally maintained long-term data samples, with asynchronous communication; however, this design choice lacks sufﬁcient numerical results to justify the performance gain (even over FedAvg) and it cannot be applied to more general scenarios.
Revised: In a very recent parallel (and empirical) work, CDA-FedAvg (Casado et al., 2021) uses a locally maintained long-term data samples, with asynchronous communication; however, this design choice lacks sufﬁcient numerical results to justify the performance gain (even over FedAvg) and it cannot be applied to more general scenarios.

Original: Our theoretically sound CFL framework covers the regularization component of FedCurv and the core set part of CDA-FedAvg, and is orthogonal to the neural architecture manipulation idea in FedWeIT.
Revised: Our theoretically sound CFL framework covers the regularization component of FedCurv and the core set part of CDA-FedAvg, and is orthogonal to the neural architecture manipulation idea in FedWeIT.

Original:  C ONTINUAL F EDERATED L EARNING F RAMEWORK 3. F ORMULATION
Revised:  C ONTINUAL F EDERATED L EARNING F RAMEWORK 3. F ORMULATION

Original: Conventional FL Formulation.
Revised: Conventional FL Formulation.

Original: The standard FL typically considers a sum-structured distributed optimization problem as below:
Revised: The standard FL typically considers a sum-structured distributed optimization problem as below:

Original: [Equation]
Revised: [Equation]

Original:  where the objective function f ( ω ) : R d → R is the weighted sum of the local objective functions
Revised: (1) where the objective function f ( ω ) : R d → R is the weighted sum of the local objective functions

Original: f i ( ω ) :=
Revised: f i ( ω ) :=

Original: E D i
Revised: E D i

Original: [ F i ( ω )] of N nodes/clients, and p i is the weight of client i .
Revised: [ F i ( ω )] of N nodes/clients, and p i is the weight of client i .

Original: In practice, it may be infeasible to select all clients each round, especially for cross-device setup (McMahan et al., 2017; Kairouz et al., 2019).
Revised: In practice, it may be infeasible to select all clients each round, especially for cross-device setup (McMahan et al., 2017; Kairouz et al., 2019).

Original: The standard FedAvg then randomly selects S clients to receive the model parameters from the server ( S ≤ N ) in each communication round and performs K local SGD update steps in the form of ω t,i,k
Revised: The standard FedAvg then randomly selects S clients to receive the model parameters from the server ( S ≤ N ) in each communication round and performs K local SGD update steps in the form of ω t,i,k

Original: = ω t,i, [Equation] ) , with local step-size η l and gradient noise ν t,i,k − 1 .
Revised: = ω t,i,k − 1 − η l ( ∇ f i ( ω t,i,k − 1 ) + ν t,i,k − 1 ) , with local step-size η l and gradient noise ν t,i,k − 1 .

Original: The selected clients then communicate the updates ∆ ω t,i =
Revised: The selected clients then communicate the updates ∆ ω t,i =

Original: ω t,i,K − ω t with the server for the model aggregation: ω t +1 = ω[Equation] −
Revised: ω t,i,K − ω t with the server for the model aggregation: ω t +1 = ω

Original: [Equation] Si [Equation] ∆ ω t,i .
Revised: t − η g S (cid:80) Si =1 ∆ ω t,i .

Original: Continual FL Formulation.
Revised: Continual FL Formulation.

Original: Despite its wide usage, the standard FL formulation given in Equation (1) cannot properly reﬂect actual time-evolving scenarios, such as local datasets changing over time or intermittent clients joining or leaving during training.
Revised: Despite its wide usage, the standard FL formulation given in Equation (1) cannot properly reﬂect actual time-evolving scenarios, such as local datasets changing over time or intermittent clients joining or leaving during training.

Original: To address this problem, we propose the Continual Federated Learning (CFL) formulation: 
Revised: To address this problem, we propose the Continual Federated Learning (CFL) formulation: [Equation] CFL

Original: [Equation]
Revised: [Equation]

Original: where f t,i ( ω ) represents the local objective function of client i at time t .
Revised: where f t,i ( ω ) represents the local objective function of client i at time t .

Original: S t is a subset of clients sampled from all clients set Ω , where |S t | = S .
Revised: S t is a subset of clients sampled from all clients set Ω , where |S t | = S .

Original: For the time-evolving scenarios, client i could have different local objective functions f t,i ( ω ) due to the changing local datasets on different t .
Revised: For the time-evolving scenarios, client i could have different local objective functions f t,i ( ω ) due to the changing local datasets on different t .

Original: 3. A PPROXIMATION OF CFL
Revised: 3. A PPROXIMATION OF CFL

Original: The challenge of addressing (CFL) formulation stems from the stateless nature of the local datasets, which include unpredictable future datasets, as well as the difﬁculty of keeping all the previous local datasets.
Revised: The challenge of addressing (CFL) formulation stems from the stateless nature of the local datasets, which include unpredictable future datasets, as well as the difﬁculty of keeping all the previous local datasets.

Original: The original deﬁnition of (CFL) is theoretically and empirically infeasible.
Revised: The original deﬁnition of (CFL) is theoretically and empirically infeasible.

Original: To handle the second case (while ignoring the intractable former), a straightforward approach is to approximate the prior local objective functions (Kirkpatrick et al., 2017; Zenke et al., 2017; Li & Hoiem, 2017), which may be accomplished by retraining information from earlier rounds in accordance with privacy protection standards.
Revised: To handle the second case (while ignoring the intractable former), a straightforward approach is to approximate the prior local objective functions (Kirkpatrick et al., 2017; Zenke et al., 2017; Li & Hoiem, 2017), which may be accomplished by retraining information from earlier rounds in accordance with privacy protection standards.

Original: Thus, we can express the approximated CFL formulation as:
Revised: Thus, we can express the approximated CFL formulation as:

Original: [Equation]
Revised: [Equation]

Original: where ˜ f t,i ( ω ) denotes the approximated local objective function of client i at time t .
Revised: where ˜ f t,i ( ω ) denotes the approximated local objective function of client i at time t .

Original: In practice, different approximation methods can be used to calculate ˜ f
Revised: In practice, different approximation methods can be used to calculate ˜ f

Original: τ,i ( ω ) , and we refer the detailed illustration and discussions of these approximation algorithms in Section 5.1, Section 5.2, and Appendix C.2.
Revised: τ,i ( ω ) , and we refer the detailed illustration and discussions of these approximation algorithms in Section 5.1, Section 5.2, and Appendix C.2.

Original: We give the formal deﬁnition of CFL framework in Algorithm 1.
Revised: We give the formal deﬁnition of CFL framework in Algorithm 1.

Original: CFL methods are a collection of methods that make use of different approximation techniques and are based on Algorithm 1.
Revised: CFL methods are a collection of methods that make use of different approximation techniques and are based on Algorithm 1.

Original: We retrieve the objective function of conventional Continual Learning (CL), by setting S = 1 in (2).[Table]
Revised: We retrieve the objective function of conventional Continual Learning (CL), by setting S = 1 in (2).

Original: Take note that in most cases, the previous local object functions cannot be properly approximated.
Revised: Take note that in most cases, the previous local object functions cannot be properly approximated.

Original: Due to the fact that such approximation impairs optimization, we deﬁne the information loss below.
Revised: Due to the fact that such approximation impairs optimization, we deﬁne the information loss below.

Original: Deﬁnition 3.1 (Information Loss) .
Revised: Require: initial weights ω 0 , global learning rate η g , local learning rate η l , number of training rounds T Ensure: trained weights ω T 1: for round t = 1 , . . , T do 2: communicate ω t to the chosen clients. 3: for client i ∈ S t in parallel do 4: initialize local model ω t,i, 0 = ω t . 5: for k = 1 , . . .

Revised: , K do 6: ˜

Revised: g t,i,k ( ω t,i,k − 1 ) = (cid:16) p t,i ∇ f t,i ( ω t,i,k − 1 ) + (cid:80) t − 1 τ =1 p τ,i ∇ ˜ f

Revised: τ,i ( ω t,i,k − 1 ) (cid:17) + ν t,i,k .

Revised: 7: ω t,i,k ← ω t,i,k − 1 − η l ˜ g t,i,k ( ω t,i,k − 1 ) .

Revised: 8: communicate ∆ ω t,i ← ω t,i,K − ω t . 9: ∆ ω t ← η g S (cid:80) i ∈S t ∆ ω t,i .

Revised: 10: ω t +1 ← ω t + ∆ ω t .

Revised: Deﬁnition 3.1 (Information Loss) .

Original: We deﬁne the information loss as the difference between the approximated local objective function ˜ f t,i ( ω ) and the real local objective function f t,i ( ω ) , i.e.
Revised: We deﬁne the information loss as the difference between the approximated local objective function ˜ f t,i ( ω ) and the real local objective function f t,i ( ω ) , i.e.

Original: [Equation]
Revised: [Equation]

Original:  In practice, we use (cid:107) ∆ t,i ( ω ) (cid:107) 2 to measure the information loss of different approximation methods.
Revised: [Equation] In practice, we use (cid:107) ∆ t,i ( ω ) (cid:107) 2 to measure the information loss of different approximation methods.

Original: We show large information loss can impede the convergence theoretically
Revised: We show large information loss can impede the convergence theoretically

Original: (c.f. Theorem 4.1) and empirically (e.g. Figure 1 in Section 5.2).
Revised: (c.f. Theorem 4.1) and empirically (e.g. Figure 1 in Section 5.2).

Original: 3. G RADIENT N OISE M ODEL
Revised: 3. G RADIENT N OISE M ODEL

Original: To analyze Equation (2) and Algorithm 1 in-depth, we propose the Gradient Noise Model (Equation (5)) to capture the dynamics of performing SGD with data heterogeneity.
Revised: To analyze Equation (2) and Algorithm 1 in-depth, we propose the Gradient Noise Model (Equation (5)) to capture the dynamics of performing SGD with data heterogeneity.

Original: We ﬁrst recap the standard deﬁnition of gradient noise in Sammut & Webb (2010); Gower et al.
Revised: We ﬁrst recap the standard deﬁnition of gradient noise in Sammut & Webb (2010); Gower et al.

Original: Gradient noise in SGD.
Revised: Gradient noise in SGD.

Original: For objective function f ( ω ) , the gradient with stochastic noise can be deﬁned as ∇ f ( ω ) =
Revised: For objective function f ( ω ) , the gradient with stochastic noise can be deﬁned as ∇ f ( ω ) =

Original: g ( ω ) + ν , where g ( ω ) is the stochastic gradient, and ν is a zero-mean noise.
Revised: g ( ω ) + ν , where g ( ω ) is the stochastic gradient, and ν is a zero-mean noise.

Original: Client drift in FL.
Revised: Client drift in FL.

Original: To analyze the impact of data heterogeneity, recent works (Karimireddy et al., 2020b; Khaled et al., 2020; Li et al., 2020b) similarly use the gradient noise to capture the distribution drift between local objective function and global objective function:
Revised: To analyze the impact of data heterogeneity, recent works (Karimireddy et al., 2020b; Khaled et al., 2020; Li et al., 2020b) similarly use the gradient noise to capture the distribution drift between local objective function and global objective function:

Original: [Equation]
Revised: [Equation]

Original: where δ i is a zero-mean random variable which measures the gradient noise of client i .
Revised: where δ i is a zero-mean random variable which measures the gradient noise of client i .

Original: Client drift and time drift in CFL framework.
Revised: Client drift and time drift in CFL framework.

Original: In (CFL), we use { t, i } pair instead of i to represent client i at time t .
Revised: In (CFL), we use { t, i } pair instead of i to represent client i at time t .

Original: Considering the distribution drift in the dimension of client and time, we further modify the gradient noise model in FL as,
Revised: Considering the distribution drift in the dimension of client and time, we further modify the gradient noise model in FL as,

Original: [Equation]
Revised: [Equation]

Original: Note that we extend the gradient noise to two terms: δ t,i and ξ t,i .
Revised: Note that we extend the gradient noise to two terms: δ t,i and ξ t,i .

Original: δ t,i is a time independent (of t ), zero-mean random variable that measures the drift of client i .
Revised: δ t,i is a time independent (of t ), zero-mean random variable that measures the drift of client i .

Original: The zero-mean ξ t,i measures the drift of client i at time t : this value of client i may change over time t , due to the time-varying local datasets.
Revised: The zero-mean ξ t,i measures the drift of client i at time t : this value of client i may change over time t , due to the time-varying local datasets.

Original: Remark 3.2.
Revised: Remark 3.2.

Original: To ease the theoretical analysis, we assume different ξ t,i are independent.
Revised: We assume different ξ t,i are independent to simplify the analysis.

Original: Empirically, we also  examine the performance of CFL methods under overlapped time-varying local datasets in Section 5.2.[Table]
Revised: We also empirically examine the performance of CFL methods under overlapped time-varying local data in Section 5.2.

Original: 3. A SSUMPTIONS
Revised: 3. A SSUMPTIONS

Original: To ease the theoretical analysis of (CFL) framework [Equation] we use the following widely used assumptions.
Revised: To ease the theoretical analysis of (CFL) framework, we use the following widely used assumptions.

Original: Assumption 1 (Smoothness and convexity) .
Revised: Assumption 1 (Smoothness and convexity) .

Original: Assume local objective functions f t,i ( ω ) are L -smooth, andin some conditions, µ -convex.
Revised: Assume local objective functions f t,i ( ω ) are L -smooth and µ -convex, i.e. µ 2 (cid:107) ω

Original: If a function f t
Revised: 1 − ω 2 (cid:107) 2 ≤ f t,i ( ω 1 ) − f t,i ( ω 2 ) − (cid:104)∇ f t,i ( ω 2 ) , ω 1 − ω 2 (cid:105) ≤ L 2 (cid:107) ω 1 − ω 2 (cid:107) 2 . A widely used corollary is that if a function f t,i is both L -smooth and µ -convex, then it satisﬁes 12 L (cid:107)∇ f t,i ( x ) − ∇ f t,i ( y ) (cid:107)

Original: [Equation]
Revised: [Equation]

Original: i is both L -smooth and  µ -convex, then it satisﬁes [Equation]
Revised: ≤ f t,i ( x ) − f t,i ( y ) − ∇ f t,i ( x ) T ( x − y ) , and L ≥ µ .

Original: Assumption 2 (Bounded noise in stochastic gradient) .
Revised: Assumption 2 (Bounded noise in stochastic gradient) .

Original: Let g t,i,k ( ω ) =
Revised: Let g t,i,k ( ω ) =

Original: ∇ f t,i,k ( ω ) + ν t,i,k , where ν t,i,k is the stochastic noise of client i on round t at k -th local update step.
Revised: ∇ f t,i,k ( ω ) + ν t,i,k , where ν t,i,k is the stochastic noise of client i on round t at k -th local update step.

Original: We assume that
Revised: We assume that

Original: E [ ν | ω ] = [Equation] and E (cid:104) (cid:107) 
Revised: E [ ν | ω ] = 0 , and E (cid:104) (cid:107) ν

Original: ν (cid:107) 2 | ω (cid:105) ≤ σ 2 . 
Revised: (cid:107) 2 | ω (cid:105) ≤ σ 2 . [Table]

Original:  Assumption 1 and 2 are common assumptions in FL (Karimireddy et al., 2020b; Khaled et al., 2020).
Revised: CFL Ours [Table] Assumption 1 and 2 are common assumptions in FL (Karimireddy et al., 2020b; Li et al., 2020b).

Revised: Considering a relaxed assumption like the bounded noise at optimum in Khaled et al.

Revised: (2020) will be an interesting future research direction.

Original: Besides that, we introduce the following assumptions to better characterize the client and time drifts in CFL framework (i.e. (5)).
Revised: Besides, we introduce the following assumptions to better characterize the client and time drifts in CFL framework (i.e. (5)).

Original: Assumption 3 (Bounded gradient drift of CFL framework) .
Revised: Assumption 3 (Bounded gradient drift of CFL framework) .

Original: Let f ( ω ) =
Revised: Let f ( ω ) =

Original: ∇ f t,i ( ω ) + δ t,i + ξ t,i , where δ t,i measures client drift and ξ t,i indicates the time drift of { t, i } .
Revised: ∇ f t,i ( ω )+ δ t,i + ξ t,i , where δ t,i measures client drift and ξ t,i indicates the time drift of { t, i } .

Original: We assumethat E [ δ | ω ] =and E [ ξ | ω ] = 0 . We further assume that E (cid:104) (cid:107) δ (cid:107) 2 | ω (cid:105) ≤ G 2 + B 2 E (cid:107)∇ f ( ω ) (cid:107) 2 and E (cid:104) (cid:107) ξ (cid:107) 2 | ω (cid:105) ≤ 
Revised: We assume E [ δ | ω ] = 0 , E [ ξ | ω ] = 0 , and thus E (cid:104) (cid:107) δ (cid:107) 2 | ω (cid:105) ≤ G 2 + B 2 E (cid:107)∇ f ( ω ) (cid:107) 2 and E (cid:104) (cid:107) ξ (cid:107) 2 | ω (cid:105) ≤ D

Original: D 2 + A 2 E (cid:107)∇ f ( ω ) (cid:107) 2 .
Revised: + A 2 E (cid:107)∇ f ( ω ) (cid:107) 2 .

Original: Assumption 3 assumes the bounded E (cid:104) (cid:107) δ (cid:107) 2 | ω (cid:105) and E (cid:104) (cid:107) ξ (cid:107) 2 | ω (cid:105) in CFL framework, which is equivalent to the widely used ( G, B ) gradient drift assumption 1 (Gower et al., 2019; Karimireddy et al., 2020b) on the bounded E (cid:104)
Revised: Assumption 3 assumes the bounded E (cid:104) (cid:107) δ (cid:107) 2 | ω (cid:105) and E (cid:104) (cid:107) ξ (cid:107) 2 | ω (cid:105) in CFL framework, which is equivalent to the widely used ( G, B ) gradient drift assumption 1 (Gower et al., 2019; Karimireddy et al., 2020b) on the bounded E (cid:104)

Original: (cid:107)∇ f i ( ω ) (cid:107) 2 (cid:105) .
Revised: (cid:107)∇ f i ( ω ) (cid:107) 2 (cid:105) .

Original: We prove this comment in Appendix B.1.
Revised: We prove this claim in Appendix B.1.

Original: Assumption 4 (Bounded information loss) .
Revised: Assumption 4 (Bounded information loss) .

Original: We assume the information loss ∆ t,i ( ω ) can be bounded by an arbitrary non-negative value R , i.e. (cid:107) ∆ t,i ( ω ) (cid:107) ≤ R . 
Revised: We assume the information loss ∆ t,i ( ω ) can be bounded by an arbitrary non-negative value R , i.e. (cid:107) ∆ t,i ( ω ) (cid:107) ≤ R . Remark 3.3.

Revised: The exact information loss in Assumption 4 may be intractable for some approximation methods.

Revised: In Lemma B.3 of Appendix B.3, we showcase a tight analysis of information loss for Taylor extension based regularization methods.

Original:  T HEORETICAL RESULTS
Revised:  T HEORETICAL RESULTS

Original: In this section, we analyze the theoretical performance of Algorithm 1 under Assumption 1–4.
Revised: In this section, we analyze the theoretical performance of Algorithm 1 under Assumption 1–4.

Original: We prove that CFL methods converges faster than FedAvg, despite that the term  induced by information loss concurrently trades off the optimization.
Revised: We prove that CFL methods converges faster than FedAvg, despite that the term (refer to ϕ t in Theorem 4.1) induced by information loss concurrently trades off the optimization.

Original: Additionally, our results include the FedAvg rate (and match the prior work) and a novel convergence rate for SGD on CL.
Revised: Additionally, our results include the FedAvg rate (and match the prior work) and a novel convergence rate for SGD on CL.

Original: In Table 1, we summarize the convergence rate of different algorithms 2 .
Revised: In Table 1, we summarize the convergence rate of different algorithms 2 .

Original: The proof details refer to Appendix B.4.
Revised: The proof details refer to Appendix B.4 and Appendix B.5.

Original: ) gradient drift: E (cid:2) (cid:107)∇ f i ( ω ) (cid:107) 2 (cid:3) ≤ G 2 E (cid:2) (cid:107)∇ f ( ω ) (cid:107) 2 (cid:3) + B 2 , where G 2 ≥ 1 and B 2 ≥

Original: 4. C ONVERGENCE RATE OF CFL METHODS
Revised: 4. C ONVERGENCE RATE OF CFL METHODS

Original: We start our analysis of Algorithm 1 from the one round progress of CFL methods in the Theorem stated below.
Revised: We start our analysis of Algorithm 1 from the one round progress of CFL methods in the Theorem stated below.

Revised: 1 The ( G, B ) gradient drift: E (cid:2) (cid:107)∇ f i ( ω ) (cid:107) 2 (cid:3) ≤ G 2 E

Revised: (cid:2) (cid:107)∇ f

Original:  Theorem 4.1 (One round Progress of CFL methods) .
Revised: [Table] (cid:107) Theorem 4.1 (One round Progress of CFL methods) .

Original: When { f t,i ( ω ) } satisfy Assumption 1–4, and let η = Kη g η l , we have the one round progress of CFL methods as,
Revised: When { f t,i ( ω ) } satisfy Assumption 1–4, and let η = Kη g η l , we have the one round progress of CFL methods as,

Original: [Equation]
Revised: [Equation]

Original: wherec 1 = [Equation] [Equation] . ϕ
Revised: where

Original: [Equation]
Revised: [Equation]

Original: is a constant
Revised: is a constant

Original: [Equation]
Revised: [Equation]

Original: N is the number of chosen clients in each round, and K is the number of local iterations.
Revised: N is the number of chosen clients in each round, and K is the number of local iterations.

Original: Theorem 4.1 shows the one round progress of CFL methods.
Revised: Theorem 4.1 shows the one round progress of CFL methods.

Original: The A 1 part of (6) is theregular linear convergence term, while the A 2 part illustrates the worsened convergence induced by gradient noise and information loss.
Revised: The A 1 part of (6) indicates the linear convergence rate, while the A 2 part illustrates the worsened convergence induced by gradient noise and information loss.

Original: ϕ t is a constant related to the information loss that causes the drift of the optimum.
Revised: ϕ t is a constant related to the information loss that causes the drift of the optimum.

Original: When the upper limit on information loss R exceeds 0 , we cannot reach the exact optimum due to the approximation error: the model instead reaches the neighborhood of (cid:15) + ϕ , where ϕ = 
Revised: When the upper limit on information loss R exceeds 0 , we cannot reach the exact optimum due to the approximation error: the iterates instead reach the neighborhood of (cid:15) + ϕ , where ϕ = (cid:80)

Original: [Equation]
Revised: Tt = 1 q t ϕ t (cid:80)

Original: Tt =1 q t for some sequence q t .
Revised: Tt = 1 q t for some sequence q t .

Original: We show in Section 5 that in practice, the approximation methods with small information loss converge much better than methods with larger information loss.
Revised: We show in Section 5 that in practice, the approximation methods with small information loss converge much better than methods with larger information loss.

Original: Theorem 4.2 (Convergence rate of CFL methods) .
Revised: Theorem 4.2 (Convergence rate of CFL methods) .

Original: Assume { f t,i ( ω ) } satisfy Assumption 1– 4, the output of Algorithm 1 has expected error smaller than (cid:15) + ϕ , for η g = 1 , η l ≤ √ 
Revised: Assume { f t,i ( ω ) } satisfy Assumption 1– 4, the output of Algorithm 1 has expected error smaller than (cid:15) + ϕ , for η g = 1 , η l ≤ √ 3+4(1+ [Equation] 2 + [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Revised: and p t

Revised: [Equation]

Revised: i

Revised: [Equation]

Revised: ( t

Revised: [Equation]

Original: on round t.
Revised: on round t

Original:  When { f t,i ( ω ) } are µ -strongly convex functions, we have,
Revised: [Equation] When { f t,i ( ω ) } are µ -strongly convex functions, we have,

Original: [Equation]
Revised: [Equation]

Original: and when { f t,i ( ω ) } are general convex functions ( µ = 0 ), we have
Revised: and when { f t,i ( ω ) } are general convex functions ( µ = 0 ), we have

Original: [Equation]
Revised: [Equation]

Original: where c O = 1 + A 2 + B 2 , c A = G 2 + D 2 R 2 R 2 + D 2 , c B =
Revised: ) and when { f t,i ( ω ) } are non-convex, setting η = Kη g η l = √ KN √ TL , we have

Original: [Equation]
Revised: [Equation]

Revised: where c O = 1 + A 2 + B 2 , c

Revised: A = G 2 + D 2 R 2 R 2 + D 2 , c B =

Revised: D 6 ( R 2 + D 2 ) 2 , c C =

Original: {+Lσ 
 η+} 2 g , and
Revised: {+Lσ 
 η+} 2 g K

Revised: +

Original:  F = (cid:107) ω 0 − ω ∗ (cid:107) 2 .
Revised: Lc A η 2 g , c D = Lc B η 2 g , c R 1 = R 2 R 2 + D 2 , c R 2 = D 4 ( R 2 + D 2 ) 2 , c m is a constant related to A , B and R , and F = (cid:107) ω 0 − ω ∗ (cid:107) 2 .

Original: N is the number of chosen clients in each round, and K is the number of local iterations.
Revised: N is the number of chosen clients in each round, and K is the number of local iterations.

Original: We elaborate the choice of p t,i in Appendix B.4.3.
Revised: We elaborate the choice of p t,i in Appendix B.4.3.

Original: Remark 4.3.
Revised: Remark 4.3.

Original: When setting N = 1 in Theorem 4.2, we recover the convergence rate of standalone
Revised: When setting N = 1 in Theorem 4.2, we recover the convergence rate of standalone

Original: CL methods.
Revised: CL methods.

Original: To the best of our knowledge, we are the ﬁrst to provide such theoretical guarantees for
Revised: To the best of our knowledge, we are the ﬁrst to provide such theoretical guarantees for

Original: SGD: the recent work (Yin et al., 2020a) only gives the convergence rate of GD for regularization based CL methods on general convex case, under the constraint of R = 0 . 
Revised: SGD: the recent work (Yin et al., 2020a) only gives the convergence rate of GD for regularization based CL methods on general convex case, under the constraint of R = 0 . Remark 4.4.

Original: 4. C ONVERGENCE RATE OF F ED A VG UNDER TIME - VARYING SCENARIOS
Revised: For stateless FL scenarios (clients only appear once during training), we can derive tighter bounds than that of Theorem 4.2. Applying Lemma B.2 to the considered stateless scenario, c A and c B in Theorem 4.2 become c A (cid:48) = ( G 2 + D 2 ) R 2 G 2 + D 2 + R 2 and c B (cid:48)

Revised: =

Original: {+( D 2 + G 2 ) 
+}
Revised: {+( D 2 + G 2 ) 
+}

Revised: ( R 2 + D 2 + G 2 ) 2 , where c A µ(cid:15) + (cid:113) c B µ(cid:15) ≥ c A (cid:48) µ(cid:15) + (cid:113)

Revised: c

Revised: B (cid:48) µ(cid:15) and c A µ(cid:15) + (cid:113) c B µ(cid:15) corresponds to the rate of strongly convex setting in Theorem 4.2.

Revised: 4. C ONVERGENCE RATE OF F ED A VG UNDER TIME - VARYING SCENARIOS

Original: To show the faster convergence of CFL methods than FedAvg, we provide the convergence rate of
Revised: To show the faster convergence of CFL methods than FedAvg, we provide the convergence rate of

Original: FedAvg under time-evolving scenarios.
Revised: FedAvg under time-evolving scenarios.

Original: Results are derived from Theorem 4.1, by setting p t,i = 1 .
Revised: Results are derived from Theorem 4.1, by setting p t,i = 1 .

Original: Note that when p t,i = 1 , we can set ϕ t to 0 and eliminate the appearance of R in the analysis.
Revised: Note that when p t,i = 1 , we can set ϕ t to 0 .

Original: Theorem 4.4 (Convergence rate of FedAvg under time-varying scenarios) .
Revised: Theorem 4.5 (Convergence rate of FedAvg under time-varying scenarios) .

Original: Assume { f t,i ( ω ) } satisfy Assumption 1–4, the output of FedAvg has expected error smaller than (cid:15) , for η g = 1 and η l ≤ √
Revised: Assume { f t,i ( ω ) } satisfy Assumption 1–4, the output of FedAvg has expected error smaller than (cid:15) , for η g = 1 and η l ≤ √

Original: [Equation]
Revised: [Equation]

Original: When { f t,i ( ω ) } are µ -strongly convex functions, we have,
Revised: When { f t,i ( ω ) } are µ -strongly convex functions, we have,

Original: [Equation]
Revised: [Equation]

Original: 9) and when { f t,i ( ω ) } are general convex functions ( µ = 0 ), we have
Revised: and when { f t,i ( ω ) } are general convex functions ( µ = 0 ), we have

Original: [Equation]
Revised: [Equation]

Original: , (10) where c O = 1 + A 2 + B 2 , c
Revised: where c O = 1 + A 2 + B 2 , c

Original: A =
Revised: A =

Original: G 2 + D 2 , c C
Revised: G 2 + D 2 , c C

Original: =
Revised: =

Original: Lσ 

Revised: Lσ

Original: η 2 g K
Revised: η 2 g K

Original: +
Revised: +

Original: Lc A η 2 g , and F = (cid:107) ω 0 − ω ∗ (cid:107) 2 .
Revised: Lc A η 2 g , and F = (cid:107) ω 0 − ω ∗ (cid:107) 2 .

Original: N is the number of chosen clients in each round, and K is the number of local iterations.
Revised: N is the number of chosen clients in each round, and K is the number of local iterations.

Original: Remark 4.5.
Revised: Remark 4.6.

Original: CFL methods accelerates the convergence by reducing the variance term.
Revised: CFL methods accelerates the convergence by reducing the variance term.

Original: As stated in Theorem 4.2 and Theorem 4.4, FedAvg (under time-varying scenarios) is a special case of CFL methods by setting  the upper bound of information loss R to ∞ .{+(we can also observe this result by setting R → ∞ in p t,i = ( t − 1) R 
 + D 2 tD 2 +( t − 1) R 2 in Theorem 4.2).+}
Revised: As stated in Theorem 4.2 and Theorem 4.5, FedAvg (under time-varying scenarios) is a special case of CFL methods by setting p t,i = 1 , and we can show by proof (in Appendix B.4) that it is equivalent to setting the upper bound of information loss R → ∞ {+(we can also observe this result by setting R → ∞ in p t,i = ( t − 1) R 
 + D 2 tD 2 +( t − 1) R 2 in Theorem 4.2).+}

Original: The upper bound R is linearly correlated to gradient noise, where c
Revised: The upper bound R is linearly correlated to gradient noise, where c

Original: A = G 2 + D 2 R 2 R 2 + D 2 in Theorem 4.2 increases to c
Revised: A = G 2 + D 2 R 2 R 2 + D 2 in Theorem 4.2 increases to c

Original: A = G 2 + Din Theorem 4.4.
Revised: A = G 2 + D 2 in Theorem 4.5.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In our numerical investigation, we ﬁrst use the Noisy Quadratic Model—a simple convex model—to verify our theoretical results stated in Section 4.
Revised: In our numerical investigation, we ﬁrst use the Noisy Quadratic Model—a simple convex model—to verify our theoretical results stated in Section 4.

Original: The results (in Appendix C.1) demonstrate that CFL methods converges much faster than baselines like FedAvg and FedProx, with a smoother optimization phase.
Revised: The results (in Appendix C.1) demonstrate that CFL methods converge much faster than baselines like FedAvg and FedProx, with a smoother convergence curve.

Original: These ﬁndings are corroborate Remark 4.5 about the variance reduction effect of CFL methods.
Revised: These ﬁndings are corroborate Remark 4.6 about the variance reduction effect of CFL methods.

Original: We further provide extensive empirical evaluations below by comparing CFL methods with various strong FL competitors on different realistic datasets.
Revised: We further provide extensive empirical evaluations below by comparing CFL methods with various strong FL competitors on different realistic datasets.

Original: We explore several approximation techniques in CFL methods and shed light on achieving practical efﬁcient federated learning with time-varying data heterogeneity.
Revised: We explore several approximation techniques in CFL methods and shed light on achieving practical efﬁcient federated learning with time-varying data heterogeneity.

Original: 5. S ETUP
Revised: 5. S ETUP

Original: Time-varying heterogeneous local datasets.
Revised: Time-varying heterogeneous local datasets.

Original: We considerto federated learn an image classiﬁer using ResNet18 on split-CIFAR10 and split-CIFAR100 datasets, as well as a two layer MLP on the split-Fashion-MNIST dataset.
Revised: We consider federated learn an image classiﬁer using ResNet18 on split-CIFAR10 and split-CIFAR100 datasets, as well as a two layer MLP on the split-Fashion-MNIST dataset.

Original: The “split” follows the idea introduced in Yurochkin et al.
Revised: The “split” follows the idea introduced in Yurochkin et al.

Original: Hsu et al.
Revised: Hsu et al.

Original: ; Reddi et al.
Revised: ; Reddi et al.

Original: (2021), where we leverage the Latent Dirichlet Allocation (LDA) to control the distribution drift with parameter α
Revised: (2021), where we leverage the Latent Dirichlet Allocation (LDA) to control the distribution drift with parameter α

Original: (See Algorithm 4).
Revised: (See Algorithm 4).

Original: Larger α indicates smaller drifts.
Revised: Larger α indicates smaller drifts.

Original: Unless speciﬁcally mentioned otherwise our studies use the following protocol.
Revised: Unless speciﬁcally mentioned otherwise our studies use the following protocol.

Original: All datasets are partitioned to 210 subsets for 7 distinct clients: all clients are selected and trained for 500 communication rounds, and each client randomly samples one of the corresponding 30 subsets for the local training— this(most) challenging time-varying scenario mimics the realistic client data sampling scheme (from some underlying distributions).
Revised: All datasets are partitioned to 210 subsets for 7 distinct clients: all clients are selected and trained for 500 communication rounds, and each client randomly samples one of the corresponding 30 subsets for the local training— this challenging time-varying scenario mimics the realistic client data sampling scheme (from some underlying distributions).

Original: Notably, the training strategy used in this section is applicable to all FL baselines and CFL methods, and we assess the model performance using a global test dataset 3 .
Revised: Notably, the training strategy used in this section is applicable to all FL baselines and CFL methods, and we assess the model performance using a global test dataset 3 .

Original: We carefully tune the hyper-parameters in all algorithms (details refer to Appendix C.2.1), and report the optimal results (i.e. mean test accuracy across the past 5 best epochs) after repeated trials.
Revised: We carefully tune the hyper-parameters in all algorithms (details refer to Appendix C.2.1), and report the optimal results (i.e. mean test accuracy across the past 5 best epochs) after repeated trials.

Original: Approximation techniques in CFL methods.
Revised: Approximation techniques in CFL methods.

Original: Below, we review three representative types of information approximation techniques in CL, and use them to investigate the effect of various types of information loss under the (CFL) formulation.
Revised: Below, we review three representative types of information approximation techniques in CL, and use them to investigate the effect of various types of information loss under the (CFL) formulation.

Original: We refer to Appendix C.2.2 for a more comprehensive introduction and discussion.
Revised: We refer to Appendix C.2.2 for a more comprehensive introduction and discussion.

Original: • Regularization methods.
Revised: • Regularization methods.

Original: Instead of keeping datasets from previous rounds, we keep track the gradients and Hessian matrices of local objective functions, and use Taylor Extension to approximate the local objective functions in previous rounds.
Revised: Instead of keeping datasets from previous rounds, we keep track the gradients and Hessian matrices of local objective functions, and use Taylor Extension to approximate the local objective functions in previous rounds.

Original: Note that the trade-off between Hessian estimation and computational overhead constrains the practical feasible of such approach.
Revised: Note that the trade-off between Hessian estimation and computational overhead constrains the practical feasible of such approach.

Original: • Core set methods.
Revised: • Core set methods.

Original: Another simple yet effective treatment in CL lines in the category of Exemplar Replay (Rebufﬁ et al., 2017; Castro et al., 2018).
Revised: Another simple yet effective treatment in CL lines in the category of Exemplar Replay (Rebufﬁ et al., 2017; Castro et al., 2018).

Original: This method selects and regularly saves previous core-set samples (a.k.a. exemplars), and replays them with the current local datasets. 
Revised: This method selects and regularly saves previous core-set samples (a.k.a. exemplars), and replays them with the current local datasets. [Table] [Table]

Original: • Generative methods.
Revised: • Generative methods.

Original: Maintaining a core set for each client may become impractical when learning scales to millions of clients.
Revised: Maintaining a core set for each client may become impractical when learning scales to millions of clients.

Original: To ensure a privacy-preserved federated learning, the generative models (Goodfellow et al., 2014) could be used locally to capture the local data distribution: fresh data samples will be generated on the ﬂy and combined with the current local dataset.
Revised: To ensure a privacy-preserved federated learning, the generative models (Goodfellow et al., 2014) could be used locally to capture the local data distribution: fresh data samples will be generated on the ﬂy and combined with the current local dataset.

Original: For the sake of simplicity, we usethe Markov Chain Monte Carlo(MCMC) (Nori et al., 2014) in our assessment.
Revised: For the sake of simplicity, we use Markov Chain Monte Carlo (Nori et al., 2014) in our assessment.

Original: 5. R ESULTS
Revised: 5. R ESULTS

Original: Comments on different CFL approximation techniques.
Revised: Comments on different CFL approximation techniques.

Original: In Table 2, we examine the performance of several approximation techniques using the split-CIFAR10 dataset.
Revised: In Table 2, we examine the performance of several approximation techniques using the split-CIFAR10 dataset.

Original: We can conclude that (1) Core set methods outperform other methods by a signiﬁcant 
Revised: We can conclude that (1) Core set methods outperform other methods by a signiﬁcant margin.

Original: margin.

Original: The simple choice of “naive

Original: core set”, i.e

Original:  randomly and uniformly sample data from the local dataset, surpasses the sampling[Table] [Table] technique described in iCaRL (Rebufﬁ et al., 2017) 4 for CL, despite their faster convergence in the initial training phase.
Revised: The simple choice of “naive core set”, i.e. randomly and uniformly sample data from the local dataset, surpasses the sampling technique described in iCaRL (Rebufﬁ et al., 2017) 4 for CL, despite their faster convergence in the initial training phase.

Original: The quality of Hessian estimation matters for regularization based methods.
Revised: The quality of Hessian estimation matters for regularization based methods.

Original: PyHessian (Yao et al., 2020), as a method to approximate diagonal Hessian matrix, is slightly preferable than Fisher Information Matrix, though the latter one involves less computation. 
Revised: PyHessian (Yao et al., 2020), as a method to approximate diagonal Hessian matrix, is slightly preferable than Fisher Information Matrix, though the latter one involves less computation. (3) The performance of generative methods is

Original: (3) The performance of generative methods is restricted , and we hypothesize that the poor quality of generated samples contributes to the constraint.
Revised: restricted , and we hypothesize that the poor quality of generated samples contributes to the constraint.

Original: In the subsequent evaluation, we consider naive core set sampling for CFL-Core-Set method, and use
Revised: In the subsequent evaluation, we consider naive core set sampling for CFL-Core-Set method, and use

Original: PyHessian for CFL-Regularization.
Revised: PyHessian for CFL-Regularization.

Original: We exclude the results of generative methods, due to the trivial performance gain and signiﬁcant computational overhead.
Revised: We exclude the results of generative methods, due to the trivial performance gain and signiﬁcant computational overhead.

Original: Understanding various approximated CFL implementations on different datasets.
Revised: Understanding various approximated CFL implementations on different datasets.

Original: Tableexperimentally studies the impacts of various information approximation techniques in CFL methods— as discussed in Section 5.1—and compares them with the backbone algorithm of CFL methods, i.e.
Revised: Tableexperimentally studies the impacts of various information approximation techniques in CFL methods— as discussed in Section 5.1—and compares them with the backbone algorithm of CFL methods, i.e.

Original: FedAvg.
Revised: FedAvg.

Original: We have the following consistent ﬁndings on different datasets.
Revised: We have the following consistent ﬁndings on different datasets The

Original: The improvement of CFL methods over FedAvg becomes larger for more challenging tasks , as shown in the bottom of the Table 3 for regularization based methods.
Revised: improvement of CFL methods over FedAvg becomes larger for more challenging tasks , as shown in the bottom of the Table 3 for regularization based methods.

Original: This might reﬂect the fact that the precision of the information approximation is more crucial for complicated tasks.
Revised: This might reﬂect the fact that the precision of the information approximation is more crucial for complicated tasks.

Original: Similarly, we demonstrate in Figure 6 (in Appendix C.2.3) that CFL methods offer better resistance to time-varying non-iid data and the signiﬁcance of such ﬁnding is depending on the task difﬁculty.
Revised: Similarly, we demonstrate in Figure 6 (in Appendix C.2.3) that CFL methods offer better resistance to time-varying non-iid data and the signiﬁcance of such ﬁnding is depending on the task difﬁculty.

Original: For CFL-Regularization method, applying regularization terms on top layers works betterthan that on all layers.
Revised: For CFL-Regularization method, applying regularization terms on top layers works better

Revised: than that on all layers.

Original: This observation matches the recent work on decoupling feature extractor and classiﬁer (Collins et al., 2021; 
Revised: This observation matches the recent work on decoupling feature extractor and classiﬁer (Collins et al., 2021; Chen & Chao, 2021; Luo et al., 2021)

Revised: : the bottom layers are more

Original: Chen & Chao, 2021;
Revised: generic across tasks and

Original: Luo et al., 2021): the bottom layers are more generic across tasks and can serve as a global feature extractor, while the top layers are subject to task-speciﬁc information.
Revised: can serve as a global feature extractor, while the top layers are subject to task-speciﬁc information.

Original: The core-set method consistently outperforms FedAvg and other CFL approximation techniques by alarge margin .
Revised: The core-set method consistently outperforms FedAvg and other CFL approximation techniques by a

Original: [Figure]
Revised: large margin .

Original: We further investigate the connection between the information loss and the learning performance.
Revised: We further investigate the connection between the information loss and the learning performance.

Original: We examine CFL methods with naive core sets, where the degree of information losscan be changed by altering the core set size from 20 to 150 , with the same random seed and optimizers.
Revised: We examine CFL methods with naive core sets, where the degree of information loss 5 can be changed by altering the core set size from 20 to 150 , with the same random seed and optimizers.

Original: Figure 1 depicts that the performance of models is highly linked to the value of information loss: the lesser the information loss, the higher the performance.
Revised: Figuredepicts that the performance of models is highly linked to the value of information loss: the lesser the information loss, the higher the performance.

Original: Superior performance of CFL methods over other strong FL baselines.
Revised: Superior performance of CFL methods over other strong FL baselines.

Original: Alongside the comparison between CFL methods and FedAvg on various datasets (Table 3), in Table 4 we verify the efﬁcacy of CFL methods over other strong FL baselines , on split-CIFAR10 dataset.
Revised: Alongside the comparison between CFL methods and FedAvg on various datasets (Table 3), in Table 4 we verify the efﬁcacy of CFL methods over other strong FL baselines , on split-CIFAR10 dataset.

Original: We also examine
Revised: We also examine

Original: MimeLite (Karimireddy et al., 2020a), a method suggested for stateless FL scenarios.
Revised: MimeLite (Karimireddy et al., 2020a), a method suggested for stateless FL scenarios.

Original: Note that we exclude the results illustration for methods like SCAFFOLD (Karimireddy et al., 2020b) and
Revised: Note that we exclude the results illustration for methods like SCAFFOLD (Karimireddy et al., 2020b) and

Original: Ditto (Li et al., 2021), due to their infeasibility to be applied in our continual scenarios 6 . 
Revised: Ditto (Li et al., 2021), due to their infeasibility to be applied in our continual scenarios 6 . [Figure]

Original: We further relax the difﬁculty of federated continual learning, from challenging non-overlapping time-varying heterogeneous data (e.g. in Table 2 and Table 3) to a moderate time-varying case (i.e. the local data evolves with the overlapping, while the size of local datasets stay unchanged).
Revised: We further relax the difﬁculty of federated continual learning, from challenging nonoverlapping time-varying heterogeneous data (e.g. in Table 2 and Table 3) to a moderate time-varying case (i.e. the local data evolves with the overlapping, while the size of local datasets stay unchanged).

Original: Tableillustrates the performance of FL baselines and CFL methods, under different degrees of overlapping (the construction details refers to Appendix C.2.1): the improvement of CFL methods is consistent to our previous results, while the overlap parameter has no obvious connection with the ﬁnal global test performance.
Revised: Table 5 illustrates the performance of FL baselines and CFL methods, under different degrees of overlapping (the construction details refers to Appendix C.2.1): the improvement of CFL methods is consistent to our previous results, while the overlap parameter has no obvious connection with the ﬁnal global test performance.

Original: We believe that both the overlap degree and the new arriving data inﬂuence ﬁnal performance, and we leave future work on realistic time-varying FL datasets to gain a better understanding.
Revised: We believe that both the overlap degree and the new arriving data inﬂuence ﬁnal performance, and we leave future work on realistic time-varying FL datasets to gain a better understanding.

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Investigating other time-varying scenarios.
Revised: Investigating other time-varying scenarios.

Original: In previous numerical investigations, we examine the stateful clients and assume that these clients would learn and retain their client states throughout the training process.
Revised: In previous numerical investigations [Table] we examine the stateful clients and assume that these clients would learn and retain their client states throughout the training process.

Original: The remainder of this section will discuss how to learn with stateless clients (i
Revised: The remainder of this section will discuss how to learn with stateless clients (i

Original: [Table]
Revised: [Table]

Original: e. each client only appear once).
Revised: e. each client only appear once).

Original: To do this, we change the data partition strategy such that fresh data partitions are sampled on the ﬂy for each client and communication round.
Revised: To do this, we change the data partition strategy such that fresh data partitions are sampled on the ﬂy for each client and communication round.

Original: Table 6 evaluates the above-mentioned scenario.
Revised: Table 6 evaluates the above-mentioned scenario.

Original: We ﬁnd that CFL-Regularization methods consistently outperform alternative FL baselines , similar to the observations in Table 4.
Revised: We ﬁnd that CFL-Regularization methods consistently outperform alternative FL baselines , similar to the observations in Table 4.

Original: However, due to the nature of stateless under the privacy concern, the advancements of core set method in CFL algorithm cannot be seen in this scenario.
Revised: However, due to the nature of stateless under the privacy concern, the advancements of core set method in CFL algorithm cannot be seen in this scenario.

Original: Additionally, we discuss the applicability of different algorithms in a variety of time-varying scenarios in Table 8 of Appendix C.2.
Revised: Additionally, we discuss the applicability of different algorithms in a variety of time-varying scenarios in Table 8 of Appendix C.2.

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: C ONTENTS OF A PPENDIX A Techniques
Revised: C ONTENTS OF A PPENDIX A Techniques

Original: B Convergence rate of CFL
Revised: B Convergence rate of CFL

Original: B.1 Bound of Gradient Noise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: B.1 Bound of Gradient Noise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: B.2 Bounded Gradient Noise of CFL . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: B.2 Bounded Gradient Noise of CFL . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: B.3 Bounded Approximation Error . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: B.3 Bounded Approximation Error . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: B.4 Proof of Theorem 4.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: B.4 Proof of Theorem 4.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: B.4.
Revised: B.4.

Original: Start up. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Start up. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: B.4.
Revised: B.4.

Original: One Step Progress . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: One Step Progress . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: B.4.
Revised: B.4.

Original: Weights of rounds. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Weights of rounds. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: B.4.
Revised: B.4.

Original: Convergence results . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Convergence results . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Revised: B.5 Proof of convergence rate for non-convex setting . . . . . . . . . . . . . . . . . .

Original: C Experiment details
Revised: C Experiment details

Original: C.1 Noisy quadratic model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: C.1 Noisy quadratic model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: C.2 Realistic datasets . .
Revised: C.2 Realistic datasets . .

Original: C.2.
Revised: C.2.

Original: Setup . .
Revised: Setup . .

Original: C.2. Approximation Methods. . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: C.2. Approximation Methods

Original: C.2. 
Revised: C.2. Additional Experiments

Original: Additional Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: C .3 Algorithms
Revised: C.3 Algorithms

Original: A T ECHNIQUES
Revised: A T ECHNIQUES

Original: Here we show some technical lemmas which are helpful in the theoretical proof.
Revised: Here we show some technical lemmas which are helpful in the theoretical proof.

Original: Lemma A.1 (Linear convergence rate (Karimireddy et al., 2020b, Lemma 1)) .
Revised: Lemma A.1 (Linear convergence rate (Karimireddy et al., 2020b, Lemma 1)) .

Original: For every nonnegative sequence { d r − 1 } r ≥ 1 , and any parameter µ > 0 , T ≥ 1 2 η max µ , there exists a constant step-size η ≤ η max and weights ω t
Revised: For every nonnegative sequence { d r − 1 } r ≥ 1 , and any parameter µ > 0 , T ≥ [Equation] µ , there exists a constant step-size η ≤ η max and weights ω t

Original: = (1 − µη ) 1 − t such that for W T = (cid:80) T +1 t =1 ω t
Revised: = (1 − µη ) 1 − t such that for W T = (cid:80) T + [Equation] =1 ω t

Original: [Equation]
Revised: [Equation]

Original: Lemma A.2 (Sub-linear convergence rate (Karimireddy et al., 2020b, Lemma 2)) .
Revised: Lemma A.2 (Sub-linear convergence rate (Karimireddy et al., 2020b, Lemma 2)) .

Original: For every nonnegative sequence { d r − 1 } r ≥ 1 and any parameters η max > 0 , c 1 ≥ 0 , c 2 ≥ 0 , T ≥ 0 , there exists a constant step-size η ≤ η max such thatΦ T = T
Revised: For every nonnegative sequence { d r − 1 } r ≥ 1 and any parameters η max > 0 , c 1 ≥ 0 , c 2 ≥ 0 , T ≥ 0 , there exists a constant step-size η ≤ η max such that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Lemma A.3 (Relaxed triangle inequality (Karimireddy et al., 2020b, Lemma 3)) .
Revised: Lemma A.3 (Relaxed triangle inequality (Karimireddy et al., 2020b, Lemma 3)) .

Original: Let { v 1 , ..., v τ } be τ vectors in R d .
Revised: Let { v 1 , ..., v τ } be τ vectors in R d .

Original: Then the following is true:
Revised: Then the following is true:

Original: [Equation]
Revised: [Equation]

Original: Lemma A.4 (Separating mean and variance (Stich & Karimireddy, 2020, Lemma 14)) .
Revised: Lemma A.4 (Separating mean and variance (Stich & Karimireddy, 2020, Lemma 14)) .

Original: Let Ξ 1 , Ξ 2 , ..., Ξ τ be τ random variables in R d which are not necessarily independent.
Revised: Let Ξ 1 , Ξ 2 , ..., Ξ τ be τ random variables in R d which are not necessarily independent.

Original: First suppose that their mean is E [Ξ i ] = ξ i , and variance is bounded as E (cid:107) Ξ i − ξ i (cid:107) 2 ≤ M (cid:107) ξ i (cid:107) 2 + σ 2 , then the following holds
Revised: First suppose that their mean is E [Ξ i ] = ξ i , and variance is bounded as E (cid:107) Ξ i − ξ i (cid:107) 2 ≤ M (cid:107) ξ i [Equation] 2 + σ 2 , then the following holds

Original: [Equation]
Revised: [Equation]

Original: Lemma A.5 (Perturbed strongly convexity (Karimireddy et al., 2020b, Lemma 5)) .
Revised: Lemma A.5 (Perturbed strongly convexity (Karimireddy et al., 2020b, Lemma 5)) .

Original: The following holds for any L -smooth and µ -strongly convex function h , and any x , y , z in the domain of h :
Revised: The following holds for any L -smooth and µ -strongly convex function h , and any x , y , z in the domain of h :

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Lemma A.6.
Revised: Lemma A.6.

Original: Deﬁne S =
Revised: Deﬁne S =

Original: (cid:80) Tt
Revised: (cid:80) Tt

Original: =1 p t t where (cid:80) Tt  = 
Revised: =1 p t t where (cid:80) Tt =1 p t = 1 , and p t ≤ p t +1 then S ≤ 1 T (cid:80) Tt =1 1 t = ˜ O (cid:0) lnT T (cid:1) .

Original: B C ONVERGENCE RATE OF CFL B. B OUND OF G RADIENT N OISE
Revised: B C ONVERGENCE RATE OF CFL B. B OUND OF G RADIENT N OISE

Original: Lemma B.1 (Bound of Gradient Noise) .
Revised: Lemma B.1 (Bound of Gradient Noise) .

Original: Suppose f i ( ω ) is the local objective function, and f ( ω )
Revised: Suppose f i ( ω ) is the local objective function, and f ( ω )

Original: =
Revised: =

Original: E [ f i ( ω )] is the global objective function.
Revised: E [ f i ( ω )] is the global objective function.

Original: Deﬁne ∇ f ( ω ) =
Revised: Deﬁne ∇ f ( ω ) =

Original: ∇ f i ( ω )
Revised: ∇ f i ( ω )

Original: + ς i , E [ ς i ] = 0 , and assume
Revised: + ς i , E [ ς i ] = 0 , and assume

Original: E (cid:104) (cid:107) ς (cid:107) 2 | ω (cid:105) ≤
Revised: E (cid:104) (cid:107) ς (cid:107) 2 | ω (cid:105) ≤

Original: A 2 E (cid:104) (cid:107)∇ f ( ω ) (cid:107) 2 (cid:105) + B 2 , we have
Revised: A 2 E (cid:104) (cid:107)∇ f ( ω ) (cid:107) 2 (cid:105) + B 2 , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: B. B OUNDED G RADIENT N OISE OF CFL
Revised: B. B OUNDED G RADIENT N OISE OF CFL

Original: Lemma B.2 (Bounded Gradient Noise of CFL) .
Revised: Lemma B.2 (Bounded Gradient Noise of CFL) .

Original: For Formulation (CFL) , consider CFL and FedAvg, we can bound the gradient drift as (1) FedAvg only optimize current local objective functions, thus,
Revised: For Formulation (CFL) , consider CFL and FedAvg, we can bound the gradient drift as (1) FedAvg only optimize current local objective functions, thus,

Original: [Equation]
Revised: [Equation]

Original:  For CFL, in general (same clients can appear in different rounds we have
Revised: (2) For CFL, in general (same clients can appear in different rounds), we have

Original: [Equation]
Revised: [Equation]

Original: (3) For CFL, when clients only participate in training once( δ t,i are independent for all t and i ), we have
Revised: For CFL, when clients only participate in training once( δ t,i are independent for all t and i ), we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Using Assumption 3, together with the fact that ∇ f t,i ( ω ) = ∇ f ( ω ) + δ t,i + ξ t,i , we have[Equation]
Revised: Using Assumption [Equation] together with the fact that ∇ f t,i ( ω ) = ∇ f ( ω ) + δ t,i + ξ t,i , we have

Original: [Equation]
Revised: [Equation]

Original: The second inequality is based on the independence of noise
Revised: The second inequality is based on the independence of noise

Original: [Equation]
Revised: [Equation]

Original: and directly use Assumption 3 we get the last inequality.
Revised: and directly use Assumption 3 we get the last inequality.

Original: Similarly, we have
Revised: Similarly, we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: B. B OUNDED A PPROXIMATION E RROR
Revised: B. B OUNDED A PPROXIMATION E RROR

Original: Lemma B.3 (Bounded Approximation Error) .
Revised: Lemma B.3 (Bounded Approximation Error) .

Original: For ∆ t,i ( ω ) = ∇ f t,i ( ω ) −∇ ˜ f t,i ( ω ) (c.f. Deﬁnition 3), we have
Revised: For ∆ t,i ( ω ) = ∇ f t [Equation] ω [Equation]

Revised: (c.f. Deﬁnition 3), when use Taylor Extension,

Original: [Equation]
Revised: [Equation]

Revised: we have

Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst two equations come from the mean value theorem which says for continuous function f in closed intervals [ a, b ] and differentiable on open interval ( a, b ) , there exists a point c ⊆
Revised: The ﬁrst two equations come from the mean value theorem which says for continuous function f in closed intervals [ a, b ] and differentiable on open interval ( a, b ) , there exists a point c ⊆

Original: ( a, b ) such that
Revised: ( a, b ) such that

Original: [Equation]
Revised: [Equation]

Original: The last inequality is based on Assumption 4.
Revised: The last inequality is based on Assumption 4.

Original: B. P ROOF OF T HEOREM 4. 
Revised: B. P ROOF OF T HEOREM 4.

Original: In this section we will give the complete proof of convergence rate of algorithm 2 when the local objective functions are convex.Algorithm
Revised: In this section we will give the complete proof of convergence rate of algorithm 2 when the local objective functions are convex.

Original: Require: initial weights ω 0 , global learning rate η g, local learning rate η l1: for round t = 1 , . .
Revised: Require initial weights ω 0 , global learning rate η g local learning rate η l

Original: , T do 2: communicate ω t to the chosen clients.

Original: 3: for client i ∈ S t in parallel do 4: initialize local model ω t,i, 0 = ω t .

Original: 5: for k = 1 , . . .

Original: , K do 6: ˜ 
Revised: 6: ˜ g t,

Original: g t,i,k ( ω t,i,k − 1 ) = (cid:16) p t,i ∇ f t,i ( ω t,i,k − 1 ) + (cid:80) t − 1 τ =1 p τ,i ∇ ˜ f t,i ( ω t,i,k − 1 ) (cid:17) + ν t,i,k .
Revised: ,k ( ω t,i,k − 1 ) = (cid:16) p t,i ∇ f t,i ( ω t,i,k − 1 ) + (cid:80) t − 1 τ =1 p τ,i ∇ ˜ f t,i ( ω t,i,k − 1 ) (cid:17) + ν t,i,k .

Original: 7: ω t,i,k ← ω t,i,k − 1 − η l ˜ g t,i,k ( ω t,i,k − 1 ) .
Revised: 7: ω t i k ← ω t i k − 1 − η l ˜ g t

Original: 8: communicate ∆ ω t,i ← ω t,i,K − ω t.
Revised: i,k ω t

Original:  9: ∆ ωt ← η g S (cid:80) i ∈S t ∆ ωt,i . 10: ω t +1 ← ω t + ∆ ω t .
Revised: i k − 1 ) . 9: ∆ ω η g S (cid:80) i ∈S t ∆ ω

Original: B.4. S TART UP .
Revised: B.4. S TART UP .

Original: The local objective function of round t on client i is
Revised: The local objective function of round t on client i is

Original: [Equation]
Revised: [Equation]

Original: Without loss of generality, assume (cid:80) tτ
Revised: Without loss of generality, assume (cid:80) tτ

Original: =1 p τ,i = 1 .
Revised: =1 p τ,i = 1 .

Original: Then follow the steps in algorithm [Equation] we have
Revised: Then follow the steps in algorithm 2, we have

Original: [Equation]
Revised: [Equation]

Original: Then the noisy gradient in mini-batch SGD can be described as
Revised: Then the noisy gradient in mini-batch SGD can be described as

Original: [Equation]
Revised: [Equation]

Original: Then follow the steps in algorithm [Equation] we have
Revised: Then follow the steps in algorithm 2, we have

Original: [Equation]
Revised: [Equation]

Original: Then let η = Kη l η g , we have
Revised: Then let η = Kη l η g , we have

Original: [Equation]
Revised: [Equation]

Original: B.4. O NE S TEP P ROGRESS
Revised: B.4. O NE S TEP P ROGRESS

Original: Consider the one-step progress we have
Revised: Consider the one-step progress we have

Original: [Equation]
Revised: [Equation]

Original: Firstly we consider A 1 part in equation (42), −2 E (cid:104)
Revised: Firstly we consider A 1 part in equation (43), −

Original: [Equation]
Revised: [Equation]

Original: − ω ∗
Revised: =

Original: [Equation]
Revised: [Equation]

Original: = 2 η
Revised: η

Original: NK
Revised: NK

Original: N (cid:88) i =
Revised: N (cid:88) i =

Original: K (cid:88) k
Revised: K (cid:88) k

Original: = (cid:104)∇ ¯ f t,i ( ω t,i,k − 1 ) , ω ∗ − ω t (cid:105) (43) = 2 η
Revised: = (cid:104)∇ [Equation] = 2 η

Original: NK
Revised: NK

Original: N (cid:88) i =
Revised: N (cid:88) i =

Original: K (cid:88) k
Revised: K (cid:88) k

Original: = (cid:104)p t,i ∇ f t,i ( ω t,i,k ) + t − 1 (cid:88) τ = p [Equation] ∇ ˜ f
Revised: = (cid:104)

Original: [Equation]

Original: ( ω

Original: [Equation]
Revised: [Equation]

Original: i

Original: [Equation] ω ∗ [Equation] (cid:105) (44) = 2 η
Revised: = 2 η

Original: NK
Revised: NK

Original: N (cid:88) i =
Revised: N (cid:88) i =

Original: K (cid:88) k
Revised: K (cid:88) k

Original: = t (cid:88)τ =
Revised: = [Equation] (cid:88)

Original: (cid:104) p τ,i ∇ f τ
Revised: τ

Original: [Equation]
Revised: [Equation]

Original: ( ω t
Revised: (cid:104) p

Original: [Equation]
Revised: [Equation]

Original: i,k )
Revised: f

Original: [Equation]
Revised: [Equation]

Original: ω ∗ − ω t (cid:105) − 2 η NK
Revised: η NK

Original: N (cid:88) i =
Revised: N (cid:88) i =

Original: K (cid:88) k
Revised: K (cid:88) k

Original: = t − 1 (cid:88) τ
Revised: = t − 1 (cid:88) τ

Original: = p τ,i (cid:104) ∆ τ,i ( ω t,i,k ) , ω ∗ − ω t (cid:105) (45) ≤ 2 η
Revised: = p τ,i (cid:104) ∆ τ,i ( ω t,i,k ) , ω ∗ − ω t (cid:105) (46) ≤ 2 η

Original: NK
Revised: NK

Original: N (cid:88) i =
Revised: N (cid:88) i =

Original: K (cid:88) k
Revised: K (cid:88) k

Original: = t
Revised: = t

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We use equality (41) for ﬁrst inequality, and directly use Assumption 4 and Cauchy–Schwarz inequality for last inequality.
Revised: We use equality (42) for ﬁrst inequality, and directly use Assumption 4 and Cauchy–Schwarz inequality for last inequality.

Original: Then we can consider the ﬁrst term of inequality (46).
Revised: Then we can consider the ﬁrst term of inequality (47).

Original: Firstly, by Lemma A.5 we see that
Revised: Firstly, by Lemma A.5 we see that

Original: [Equation]
Revised: [Equation]

Original: That is, based on (46) and (47), we have,
Revised: That is, based on (47) and (48), we have,

Original: [Equation]
Revised: [Equation]

Original: Then consider A 2 in (42), and deﬁne c p B ,i = 1+ B 2 + A 2 ( (cid:80) t τ =1 p 2 τ, i
Revised: Then consider A 2 in (43)

Original: [Equation]
Revised: [Equation]

Original: , c p G
Revised: and deﬁne c p B ,i = 1+ B 2 + A 2 ( (cid:80) tτ

Original: [Equation] G 2 + D 2 ( (cid:80) t τ =1 p 2 τ
Revised: =1 p 2 τ,i ) , c p G ,i = G 2 + D 2 ( (cid:80) tτ

Original: [Equation] i ) , c p [Equation] = 1 N 
Revised: =1 p 2 τ,i ) , c p B = 1 N (cid:80) Ni

Original: [Equation]

Original: Ni =1 c p B ,i , and c p G = 1 N (cid:80)
Revised: =1 c p B ,i , and c p G = 1 N (cid:80)

Original: Ni =1 c p G ,i we have
Revised: Ni =1 c p G ,i we have

Original: [Equation]
Revised: [Equation]

Original: p
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: f [Equation] (cid:33)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (49) = η
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: = p [Equation] [Equation] i
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Equation] ν [Equation] i
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: (cid:33)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (50)

Original: [Table]
Revised: [Table]

Original: i

Original: [Table] [Equation] 
Revised: [Equation] [Table]

Original: =

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation] [Table] [Table] [Table]
Revised: [Table]

Original: i

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]

Original: [Table]
Revised: [Table]

Original: τ

Original: [Table] [Table] [Equation] 
Revised: [Equation] [Table]

Original: f

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Then consider equation (48) and (55), to solve E (cid:107) ω t,i,k − 1 − ω t (cid:107) 2 , we have
Revised: Then consider equation (49) and (56), to solve E (cid:107) ω t,i,k − 1 − ω t (cid:107) 2 , we have

Original: [Equation]
Revised: [Equation]

Original: Here we directly use Lemma A.4 to get inequality (57).
Revised: Here we directly use Lemma A.4 to get inequality (58).

Original: Then for E(cid:13)(cid:13) ∇
Revised: Then for E

Original: [Equation]
Revised: [Equation]

Original: f t,
Revised: f t,

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: ω ) (cid:13)(cid:13) 2 , we have
Revised: 2 , we have

Original: [Equation]
Revised: [Equation]

Original: Combine equation (57) and (62) we get,
Revised: Combine equation (58) and (63) we get,

Original: [Equation]
Revised: [Equation]

Original: That is,
Revised: That is,

Original: [Equation]
Revised: [Equation]

Original: Combine the equation (42), (48), (55)
Revised: Combine the equation (43), (49), (56)

Original: [Equation]
Revised: [Equation]

Original: and (65), we get
Revised: and (66), we get

Revised: E (cid:107)

Original: [Equation]
Revised: [Equation]

Original: + η
Revised: + η

Original: [Equation]
Revised: [Equation]

Original: σ
Revised: σ

Original: [Equation]
Revised: [Equation]

Original: NK
Revised: NK

Original: [Equation]
Revised: [Equation]

Original: N

Original: N
Revised: N

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Then we notice that
Revised: Then we notice that

Original: [Equation]
Revised: [Equation]

Original: Use the fact that η = Kη l η g , and solve above inequality
Revised: Use the fact that η = Kη l η g [Equation] and solve above inequality

Original: [Equation]
Revised: [Equation]

Original: we notice that if η satisfy
Revised: we notice that if η satisfy

Original: [Equation]
Revised: [Equation]

Revised: 6 L

Original: [Equation]
Revised: [Equation]

Original: for any c p B ,i , convergence can be promised.
Revised: for any c p B ,i , convergence can be promised.

Original: Deﬁne ˆ c p B as the largest c p B ,i , let η ≤ √
Revised: Deﬁne ˆ c p B as the largest c p B ,i , let η ≤ √

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original:  √
Revised: 12 L √

Original: [Equation]
Revised: [Equation]

Original: , we have 1 − 4 K
Revised: , we have 1 − 4 K

Original: [Equation]
Revised: [Equation]

Original: η
Revised: η

Original: [Equation]
Revised: [Equation]

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: c
Revised: c

Original: [Equation]
Revised: [Equation]

Revised: B ,

Original: [Equation]
Revised: [Equation]

Original: and then
Revised: and then

Original: [Equation]
Revised: [Equation]

Original: Deﬁne c R,i =
Revised: Deﬁne c R,i =

Original: ( (cid:80) t − 1 τ =1 p τ,i ) 2 R 2 , and c R = 1 N (cid:80) Ni =1 c
Revised: ( (cid:80) t − 1 τ =1 p τ,i ) 2 R 2 , and c R = 1 N (cid:80) Ni =1 c

Original: R
Revised: R

Original: ,i .
Revised: ,i .

Original: Assume c 1 = 2 c p G + σ 2 NK + 2 c
Revised: Assume c 1 = 2 c p G + σ 2 NK + 2 c

Original: R , c 2 =
Revised: R , c 2 =

Original: Lc pG 3 η 2 g
Revised: Lc pG 3 η 2 g

Original: + Lσ 2 

Revised: + Lσ 2

Original: η 2 g K
Revised: η 2 g K

Original: + Lc R 3 η 2 g , and c 3 = 2 N (cid:80)
Revised: + Lc R 3 η 2 g , and c 3 = 2 N (cid:80)

Original: Ni =1 (cid:80) t − 1 τ =1 p τ,i R , inequality (66) become
Revised: Ni =1 (cid:80) t − 1 τ =1 p τ,i R , inequality (67) become

Original: E (cid:107)
Revised: E (cid:107)

Original: [Equation]
Revised: [Equation]

Original:  + ∆ 
Revised: t + ∆ ω t − ω ∗ (cid:107) 2 ≤ (

Original: [Equation]
Revised: [Equation]

Original: (cid:107)
Revised: µη

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: E (cid:107)

Original: [Equation]
Revised: [Equation]

Original: µη
Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: Move f ( ω t ) − f ∗ to the left side, we get f ( ωt
Revised: Move f ( ω t ) − f ∗ to the left side, we get f ( ω

Original: [Equation]
Revised: [Equation]

Original: −f ( ω ∗ ) ≤ 1 η (1 − µη 2 ) E (cid:107) ω
Revised: −

Original:  t − ω ∗ (cid:107) 2 − 1 η E (cid:107) ω t +1 − ω[-∗ (cid:107) 
 + c 1 η + c 2 η 2 + ϕ t . [Equation]-]
Revised: [Equation] ) ≤ 1 η (1 − µη 2 ) E (cid:107) ω t − ω ∗ (cid:107) 2 − 1 η E (cid:107) ω t +1 − ω[-∗ (cid:107) 
 + c 1 η + c 2 η 2 + ϕ t . [Equation]-]

Original: [Equation]
Revised: [Equation]

Original:  Here ϕ t is a constant bounded by c 3 (cid:107) ω 0 − ω ∗ (cid:107) .
Revised: (cid:107) 2 + c 1 η + c 2 η 2 + ϕ t . Here ϕ t is a constant bounded by c 3 (cid:107) ω 0 − ω ∗ (cid:107) .

Original: Because of ϕ t , the model cannot converge to an arbitrary (cid:15) ; instead, it can only converge to the neighborhood of (cid:15) + ϕ , where ϕ is the weighted sum of the sequence ϕ t .
Revised: Because of ϕ t , the model cannot converge to an arbitrary (cid:15) ; instead, it can only converge to the neighborhood of (cid:15) + ϕ , where ϕ is the weighted sum of the sequence ϕ t .

Original: The following parts give the convergence rates when models converge to (cid:15) + ϕ for convenience.
Revised: The following parts give the convergence rates when models converge to (cid:15) + ϕ for convenience.

Original: B.4. W EIGHTS OF ROUNDS .
Revised: B.4. W EIGHTS OF ROUNDS .

Original: We can carefully tune the p τ,i to minimize the noise while speeding up the convergence.
Revised: We can carefully tune the p τ,i to minimize the noise while speeding up the convergence.

Original: Here we give the lemma of the best choice of p τ,i .
Revised: Here we give the lemma of the best choice of p τ,i .

Original: Lemma B.4.
Revised: Lemma B.4.

Original: On round t , when setting p τ,i =
Revised: On round t , when setting p τ,i =

Original: D 2 tD 2 +( t − 1) R 2 for any τ < t , and p t,i =
Revised: D 2 tD 2 +( t − 1) R 2 for any τ < t , and p t,i =

Original: ( t − 1) R 

Revised: ( t − 1) R

Original: + D 2 tD 2 +( t − 1) R 2 , we can minimize c 1 and c 2 to get best convergence.
Revised: + D 2 tD 2 +( t − 1) R 2 , we can minimize c 1 and c 2 to get best convergence.

Original: Proof.
Revised: Proof.

Original: Minimizing c 

Revised: Minimizing c

Original: and c 2 is equal to minimize c p G + c R , that is,
Revised: and c 2 is equal to minimize c p G + c R , that is,

Original: [Equation]
Revised: [Equation]

Original: Solving above optimization problem, we get the results.
Revised: Solving above optimization problem, we get the results.

Original: Using lemma B.4, we have min { c p G + c R } =
Revised: Using lemma B.4, we have min { c p G + c R } =

Original: G 2 + D 2 (cid:16) D 2 +( t − 1) R 2 tD 2 +( t − 1) R 2 (cid:17) .
Revised: G 2 + D 2 (cid:16) D 2 +( t − 1) R 2 tD 2 +( t − 1) R 2 (cid:17) .

Original: Notice that the value of c 1 , c 2 , and c 3 are changing over t .
Revised: Notice that the value of c 1 , c 2 , and c 3 are changing over t .

Original: We use c t 1 , c t 2 , and c t 3 to represent c 1 , c 2 , and c 3 on round t .
Revised: We use c t 1 , c t 2 , and c t 3 to represent c 1 , c 2 , and c 3 on round t .

Original: B.
Revised: B

Original: C ONVERGENCE RESULTS
Revised: C ONVERGENCE RESULTS

Original: Strongly convex functions.
Revised: Strongly convex functions.

Original: By equation (79), when f t,i ( ω ) are strongly convex functions, using
Revised: By equation (80), when f t,i ( ω ) are strongly convex functions, using

Original: Lemma A.
Revised: Lemma A.

Original: [Equation]
Revised: [Equation]

Original: and deﬁne q t = (1 − µη 2 ) 1 − t , we have
Revised: and deﬁne q t = (1 − µη 2 ) 1 − t , we have

Original: [Equation]
Revised: [Equation]

Original: By Lemma A.
Revised: By Lemma A.

Original: [Equation]
Revised: [Equation]

Original: we have,
Revised: we have,

Original: [Equation]
Revised: [Equation]

Original: For ϕ t , by Lemma B.4, we have,
Revised: For ϕ t , by Lemma B.4, we have,

Original: [Equation]
Revised: [Equation]

Original: Then deﬁne c o 1 =
Revised: Then deﬁne c o 1 =

Original: [Equation]
Revised: [Equation]

Original: have
Revised: have

Original: [Equation]
Revised: [Equation]

Original: where c O = 1 + A 2 + B2 , c A = G 2 +
Revised: where c O = 1 + A [Equation] + B

Original: [Equation]
Revised: [Equation]

Original: , c
Revised: , c

Original: [Equation]
Revised: [Equation]

Original: =
Revised: G [Equation]

Original: [Equation]
Revised: [Equation]

Original: General convex functions.
Revised: General convex functions.

Original: When f t,i ( ω ) are general convex functions ( µ = 0 ), directly use
Revised: When f t,i ( ω ) are general convex functions ( µ = 0 ), directly use

Original: Lemma A.2 we get,
Revised: Lemma A.2 we get,

Original: [Equation]
Revised: [Equation]

Original: where c O = 1 + A 2 + B 2 , c
Revised: where c O = 1 + A 2 + B 2 , c

Original: A = G 2 + D 2 R 2 R 2 + D 2 , c B =
Revised: A = G 2 + D 2 R 2 R 2 + D 2 , c B =

Original: D 6 ( R 2 + D 2 ) 2 , c C =
Revised: D 6 ( R 2 + D 2 ) 2 , c C =

Original: Lσ 

Revised: Lσ

Original: η 2 g K
Revised: η 2 g K

Original: +
Revised: +

Original: Lc A η 2 g , c D =
Revised: Lc A η 2 g , c D =

Original: Lc B
Revised: Lc B

Original: η 2 g , and
Revised: η 2 g , and

Original: F = (cid:107) [Equation]
Revised: F = (cid:107) ω 0 − ω ∗ (cid:107) 2 .

Original: The drift of optimal point.
Revised: The drift of optimal point.

Original: Because of information loss, we can’t get the true optimal point when considering previous rounds’ information.
Revised: Because of information loss, we can’t get the true optimal point when considering previous rounds’ information.

Original: Instead, the drift can be described as
Revised: Instead, the drift can be described as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Convergence of FedAvg.
Revised: Convergence of FedAvg.

Original: Notice that FedAvg is a special case of CFL by setting p t,i = 1 on round t .
Revised: Notice that FedAvg is a special case of CFL by setting p t,i = 1 on round t .

Original: Having this, we derive the one round convergence of FedAvg under our formulation as,
Revised: Having this, we derive the one round convergence of FedAvg under our formulation as,

Original: [Equation]
Revised: [Equation]

Original: (79) where c 1 = 2( G 2 + D
Revised: (80) where c 1 = 2( G 2 + D

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original:  NK , c 2 =
Revised: 2 NK , c 2 =

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: G
Revised: G

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original:  g K .
Revised: 6 η 2 g K .

Original: Then using Lemma A.1, we get the convergence rate when f t,i ( ω ) are µ strongly convex.
Revised: Then using Lemma A.1, we get the convergence rate when f t,i ( ω ) are µ strongly convex.

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where c O = 1 + A 2 + B 2 , c
Revised: where c O = 1 + A 2 + B 2 , c

Original: A =
Revised: A =

Original: G 2 + D 2 , c C
Revised: G 2 + D 2 , c C

Original: =
Revised: =

Original: Lσ 

Revised: Lσ

Original: η 2 g K
Revised: η 2 g K

Original: +
Revised: +

Original: Lc A η 2 g .
Revised: Lc A η 2 g .

Original: Besides, when f t,i ( ω ) are general convex, we have,
Revised: Besides, when f t,i ( ω ) are general convex, we have,

Original: [Equation]
Revised: [Equation]

Original: where c O = 1 + A 2 + B 2 , c
Revised: where c O = 1 + A 2 + B 2 , c

Original: A =
Revised: A =

Original: G 2 + D 2 , c C
Revised: G 2 + D 2 , c C

Original: =
Revised: =

Original: Lσ 

Revised: Lσ

Original: η 2 g K
Revised: η 2 g K

Original: +
Revised: +

Original: Lc A η 2 g , and F = (cid:107) ω 0 − ω ∗ (cid:107) 2 .
Revised: Lc A η 2 g , and F = (cid:107) ω 0 − ω ∗ (cid:107) 2 .

Revised: B. P ROOF OF CONVERGENCE RATE FOR NON - CONVEX SETTING

Original: [Figure]
Revised: [Equation]

Revised: For A 1 part, we have,

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: E

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: (cid:105) (cid:35)

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: η

Revised: [Table]

Revised: N

Revised: [Equation]

Revised: i

Revised: [Equation]

Revised: E

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: p

Revised: [Equation]

Revised: (cid:88)

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: (cid:105)

Revised: [Equation]

Revised: (cid:17)

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: i =

Revised: [Equation]

Revised: 

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: From equation (53) we have

Revised: [Equation]

Revised: NK

Revised: [Equation]

Revised: Then when η ≤ 12 L

Revised: [Equation]

Revised: the E (cid:13)(cid:13)(cid:13)(cid:80)

Revised: [Equation]

Revised: ,i,k ) (cid:13)(cid:13)(cid:13) 2 term can be ignored, since the coefﬁcient number will less than 0.

Revised: Then the remaining term in A 2 is 2 η 2 N (cid:80) Ni

Revised: =1 (1 − p t,i ) 2 R 2 + η 2 σ 2 NK .

Revised: Then we are willing to consider the (cid:13)(cid:13)(cid:13) ∇ f ( ω t ) − (cid:80) tτ

Revised: =1 p τ,i ∇ f τ,i ( ω t,i,k ) (cid:13)(cid:13)(cid:13) 2 .

Revised: Then we have

Revised: E 

Revised: [Equation]

Revised: =

Revised: E  

Revised: [Equation]

Revised: ≤

Revised: [Equation]

Revised: Combine equation (87) and (92) we have

Revised: [Equation]

Revised: Then we have

Revised: [Equation]

Revised: Because we know

Revised: [Equation]

Revised: That is,

Revised: [Equation]

Revised: ≤

Revised: [Equation]

Revised: Let 8 K 2 η 2 l c p B

Revised: [Equation]

Revised: i ≤ 1 , we have,

Revised: NK

Revised: [Equation]

Revised: ≤

Revised: N

Revised: [Equation]

Revised: N

Revised: [Equation]

Revised: Then we have

Revised: [Equation]

Revised: Then we must to choose the value of p t,i for better convergence.

Revised: Follow the idea that we want to train a better model when convergent, we choose p t,i that can minimize the constant term, which is by solving

Revised: [Equation]

Revised: We get same results as when the objective function is convex, that is, p τ,i =

Revised: D 2 tD 2 +( t − 1) R 2 for any τ < t , and p t,i =

Original: {+( t − 1) R 
+}
Revised: {+( t − 1) R 
+}

Revised: + D 2 tD 2 +( t − 1) R 2 , and min { G 2 + D 2 (cid:80) tτ

Revised: =1 p 2 τ,i + ( (cid:80) t − 1 τ =1 p τ, 1 ) 2 R 2 } = G 2 +

Revised: D 2 (cid:16) D 2 +

Revised: [Equation]

Revised: t − 1) R 2 tD 2 +

Revised: [Equation]

Revised: t − 1) R 2 (cid:17) .

Revised: Then we simplify the equation by using c 0 ( t ) , c 1 ( t ) , and c 2 ( t ) to denote the constant terms, we have

Revised: [Equation]

Revised: Then we have

Revised: [Equation]

Revised: Consider c 0 , because the value of B 2 and A can’t be constrained, the algorithm can’t converge for very large A and B .

Revised: Here we ﬁrst consider when c 0 < 0 , and use c m to denote the min c 0 .

Revised: Then let η = √ KN √ TL , we derive the convergence rate as

Revised: [Equation]

Revised: Then we want to analyse when the algorithm won’t converge.

Revised: When the algorithm won’t converge, that means c 0 ≥ 0 .

Revised: Because c 0 =

Revised: ( L 2 + 1) c p B − 1 − (cid:80) Ni

Revised: =1 p t,i 2 N = 1 N

Revised: (cid:80) Ni

Original: {+=
+}
Revised: {+=
+}

Revised: ( L 2 + 1)(1 + A 2 +

Revised: B 2 (cid:80) tτ =1 p 2 τ,i ) − 1 − p t,i 2 .

Revised: Then the value of A 2 , B 2 , L, p decide if c 0 ≥ 0 .

Revised: Notice that in FedAvg, p t,i = 1 , and c 0 ,avg =

Original: {+( L 2 + 1)(
+}
Revised: {+( L 2 + 1)(
+}

Revised: +

Revised: A 2 + B 2 ) − 32 , and in CFL, we can setting different p to get better convergence.

Revised: For example, when B is large, setting p τ,i = 1 t , we have c 0 , =

Revised: ( L 2 + 1)(1 + A 2 + B 2 t ) − 1 − 12 t .

Revised: Thus, we draw the conclusion as Then we have following observations:

Revised: • CFL converge faster than FedAvg by reducing the variance terms.

Original: {+For c 0 = 1 N (cid:80) Ni =
+}
Revised: {+For c 0 = 1 N (cid:80) Ni =
+}

Revised: ( L 2 + 1)(1 + A 2 + B 2 (cid:80) tτ

Revised: =1 p 2 τ,i ) − 1 − p t,i 2 , in FedAvg, we have p t,i = 1 , then c 0 ,avg =

Original: {+( L 2 + 1)(
+}
Revised: {+( L 2 + 1)(
+}

Revised: +

Revised: A 2 + B 2 ) − 32 .

Revised: However, in CFL, we can adjust p , such as when setting p τ,i = 1 t , we have c 0 =

Revised: ( L 2 + 1)(1 + A 2 + B 2 t ) − 1 − 12 t , then the variance term reduce from B 2 to B 2 t .

Revised: • The convergence rate of CFL become better for larger t , since c p B in c 0 become smaller for larger t .

Revised: • Improve N, K will speed up the convergence by reducing the terms about f 0 − f ∗ and σ 2 .

Revised: However, larger N and K will cause larger information loss and round drift (terms with R and D ), thus, it should be a trade off in practice

Revised: [Figure]

Original: C E XPERIMENT DETAILS C. N OISY QUADRATIC MODEL
Revised: C E XPERIMENT DETAILS C. N OISY QUADRATIC MODEL

Original:  We use the noisy quadratic model introduced in Zhang et al.
Revised: [Figure] We use the noisy quadratic model introduced in Zhang et al.

Original: (2019) to simulate the assumptions we used in the proof.
Revised: (2019) to simulate the assumptions we used in the proof.

Original: We use the model f ( ω ) =
Revised: We use the model f ( ω ) =

Original: ω T A
Revised: ω T A

Original: ω + B T ω + C , where ω ∈ R n is the parameter we want to optimize, and A ∈ R n × n , A (cid:23) 0 , B ∈ R n , and C ∈ R .
Revised: ω + B T ω + C , where ω ∈ R n is the parameter we want to optimize, and A ∈ R n × n , A (cid:23) 0 , B ∈ R n , and C ∈ R .

Original: We construct A by ﬁrstly constructing a diagonal matrix Λ and then constructing A by let A
Revised: We construct A by ﬁrstly constructing a diagonal matrix Λ and then constructing A by let A

Original: = U T Λ U , where U is a unitary matrix.
Revised: = U T Λ U , where U is a unitary matrix.

Original: Here we control the eigenvalues of A to control the convexity of model, that is,  µ ≤ λ min ( A ) ≤ λ max ( A ) ≤ L .
Revised: Here we control the eigenvalues of A to control the convexity of model, that is, [Table] µ ≤ λ min ( A ) ≤ λ max ( A ) ≤ L .

Original: Because Λ and A have same eigenvalues, it’s simple to control the eigenvalues of A by control the eigenvalues of Λ .
Revised: Because Λ and A have same eigenvalues, it’s simple to control the eigenvalues of A by control the eigenvalues of Λ .

Original: To simulate Assumption 2, 3, we formulate the gradient noise model by
Revised: To simulate Assumption 2, 3, we formulate the gradient noise model by

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: where δ i denotes the client drift of client j , ξ t,i denotes the round drift of client j on round
Revised: where δ i denotes the client drift of client j , ξ t,i denotes the round drift of client j on round

Original: i , and ν t,i,k denotes the noise of gradient of client j on round i , iteration k .
Revised: i , and ν t,i,k denotes the noise of gradient of client j on round i , iteration k .

Original: All of above drifts and noise are generated from Normal distribution with zero mean and different variance.
Revised: All of above drifts and noise are generated from Normal distribution with zero mean and different variance.

Original: In practice, to show if the numerical results match our theoretical results, we tried eight settings: large and small L , large and small
Revised: In practice, to show if the numerical results match our theoretical results, we tried eight settings: large and small L , large and small

Original: round drift, and general and strongly convex conditions.
Revised: round drift, and general and strongly convex conditions.

Original: For different types of L , we set L = 20 for large L , and L = 5 for small L .
Revised: For different types of L , we set L = 20 for large L , and L = 5 for small L .

Original: For convexity, we set µ = 1 for strongly convex, and µ = 0 for general convex.
Revised: For convexity, we set µ = 1 for strongly convex, and µ = 0 for general convex.

Original: For different round drift, we set E (cid:107) ξ t,i (cid:107) 2 = 100 for big round drift, and E (cid:107) ξ t,i (cid:107) 2 = 0 .
Revised: For different round drift, we set E (cid:107) ξ t,i (cid:107) 2 = 100 for big round drift, and E (cid:107) ξ t,i (cid:107) 2 = 0 .

Original: 01 for small round drift.
Revised: 01 for small round drift.

Original: Besides, we set ﬁxed variance of δ i and ν t,i,k for E (cid:107)
Revised: Besides, we set ﬁxed variance of δ i and ν t,i,k for E (cid:107)

Original: δ i (cid:107) 2 = 0 .
Revised: δ i (cid:107) 2 = 0 .

Original: 01 , and E (cid:107) ν t,i,k (cid:107) 2 = 0 .
Revised: 01 , and E (cid:107) ν t,i,k (cid:107) 2 = 0 .

Original: We do the gradient descent by let ω t,i,k +1 =
Revised: We do the gradient descent by let ω t,i,k +1 =

Original: ω t,i,k − η l g t,i ( ω t,i,k ) , and we use (cid:107) A ω + B (cid:107) as loss because for convex functions, when it reaches the global optimal point, we should have (cid:107) A ω + B (cid:107) = 0 , and if it is far away from global optimal, (cid:107)
Revised: ω t,i,k − η l g t,i ( ω t,i,k ) , and we use (cid:107) A ω + B (cid:107) as loss because for convex functions, when it reaches the global optimal point, we should have (cid:107) A ω + B (cid:107) = 0 , and if it is far away from global optimal, (cid:107)

Original: A ω + B (cid:107) will be large.
Revised: A ω + B (cid:107) will be large.

Original: Besides, we show that CFL algorithms can tolerate a larger learning rate here.
Revised: Besides, we show that CFL algorithms can tolerate a larger learning rate here.

Original: Table 7 shows the best learning rates for different FL algorithms on noisy quadratic model.
Revised: Table 7 shows the best learning rates for different FL algorithms on noisy quadratic model.

Original: In Table 7, we denote small round drift as SRD, big round drift as BRD, small L as SL, large L as LL, strongly convex as SC, and general convex as GC.
Revised: In Table 7, we denote small round drift as SRD, big round drift as BRD, small L as SL, large L as LL, strongly convex as SC, and general convex as GC.

Original: We show that the best learning rate of CFL is the largest.
Revised: We show that the best learning rate of CFL is the largest.

Original: The difference between CFL methods and other FL baselines becomes large when models are strongly convex or with big round drift.
Revised: The difference between CFL methods and other FL baselines becomes large when models are strongly convex or with big round drift.

Original: Figure 2 shows the performance of different algorithms, and CFL outperforms other FL baselines signiﬁcantly in all settings.
Revised: Figure 2 shows the performance of different algorithms, and CFL outperforms other FL baselines signiﬁcantly in all settings.

Original: Besides, the loss curves of CFL are smoother than FL methods, which implies that CFL can reduce the variance of gradients.
Revised: Besides, the loss curves of CFL are smoother than FL methods, which implies that CFL can reduce the variance of gradients.

Original: C. R EALISTIC DATASETS C.2. S ETUP
Revised: C. R EALISTIC DATASETS C.2. S ETUP

Original: We consider federated learning an image classiﬁer on split-CIFAR10 and split-CIFAR100 datasets with ResNet18, and split-Fashion-MNIST dataset with a two-layer MLP.
Revised: We consider federated learning an image classiﬁer on split-CIFAR10 and split-CIFAR100 datasets with ResNet18, and split-Fashion-MNIST dataset with a two-layer MLP.

Original: The “split” follows the idea introduced in Yurochkin et al.
Revised: The “split” follows the idea introduced in Yurochkin et al.

Original: Hsu et al.
Revised: Hsu et al.

Original: ; Reddi et al.
Revised: ; Reddi et al.

Original: (2021), where we leverage the
Revised: (2021), where we leverage the

Original: Latent Dirichlet Allocation (LDA) to control the distribution drift with parameter α
Revised: Latent Dirichlet Allocation (LDA) to control the distribution drift with parameter α

Original: (See Algorithm 4).
Revised: (See Algorithm 4).

Original: Larger α denotes smaller drifts here.
Revised: Larger α denotes smaller drifts here.

Original: In our experiments, unless speciﬁcally mentioned otherwise all datasets are partitioned to 210 subsets for 7 different clients: all clients are selected and trained for 500 communication rounds, and each client samples one of the corresponding 30 subsets randomly for the local training.
Revised: In our experiments, unless speciﬁcally mentioned otherwise all datasets are partitioned to 210 subsets for 7 different clients: all clients are selected and trained for 500 communication rounds, and each client samples one of the corresponding 30 subsets randomly for the local training.

Original: Note that unless mentioned otherwise the training strategy here applies to all FL baselines and CFL methods, and we evaluate the performance of models on global test datasets.
Revised: Note that unless mentioned otherwise the training strategy here applies to all FL baselines and CFL methods, and we evaluate the performance of models on global test datasets.

Original: We carefully tuned the hyper-parameters in all algorithms, and we report the results under the optimal settings after many trials.
Revised: We carefully tuned the hyper-parameters in all algorithms, and we report the results under the optimal settings after many trials.

Original: For CFLRegularization, we set the weight of regularization term for β = 1 on fc layer, and β = 0 .
Revised: For CFLRegularization, we set the weight of regularization term for β = 1 on fc layer, and β = 0 .

Original: 1 for last block (layer).
Revised: 1 for last block (layer).

Original: For FedProx, we set the weight of proximal term µ = 0 .
Revised: For FedProx, we set the weight of proximal term µ = 0 .

Original: 1 , and for MimeLite, we set the global momentum weight for γ = 0 .
Revised: 1 , and for MimeLite, we set the global momentum weight for γ = 0 .

Original: Besides, for all experiments, we set ﬁxed learning rate lr = 0 .
Revised: Besides, for all experiments, we set ﬁxed learning rate lr = 0 .

Original: We also naively examined warm-up tuning strategies but can not observe signiﬁcant improvement on ﬁnal results. 
Revised: We also naively examined warm-up tuning strategies but can not observe signiﬁcant improvement on ﬁnal results. [Figure]

Original: Construct local datasets with overlap.
Revised: Construct local datasets with overlap.

Original: Follow the partition methods of disjoint local datasets; we ﬁrst split the whole dataset to
Revised: Follow the partition methods of disjoint local datasets; we ﬁrst split the whole dataset to

Original: M × S subsets for S different clients: each client has M disjoint local subsets, and we put these M subsets in sequence.
Revised: M × S subsets for S different clients: each client has M disjoint local subsets, and we put these M subsets in sequence.

Original: For each client, we use a pointer to point to the start position of the current local dataset.
Revised: For each client, we use a pointer to point to the start position of the current local dataset.

Original: Figure 3 show the case when M = 5 : D 1 − D 5 are 5 disjoint local subsets of client i .
Revised: Figure 3 show the case when M = 5 : D 1 − D 5 are 5 disjoint local subsets of client i .

Original: The pointer point to the start point of a local dataset of the current round and the blue lines bound current subsets.
Revised: The pointer point to the start point of a local dataset of the current round and the blue lines bound current subsets.

Original: At the beginning of training, pointers belong to each client will point to the beginning of the sequence.
Revised: At the beginning of training, pointers belong to each client will point to the beginning of the sequence.

Original: At the end of each round, the pointer moves s steps, and if it moves to the end of all the sequence, it will come back to the start point.
Revised: At the end of each round, the pointer moves s steps, and if it moves to the end of all the sequence, it will come back to the start point.

Original: In practice, the Cifar10 dataset is partitioned to 210 subsets for 7 clients, and set the size of local dataset S l = 285 .
Revised: In practice, the Cifar10 dataset is partitioned to 210 subsets for 7 clients, and set the size of local dataset S l = 285 .

Original: Then when we set s = 285 , the local datasets are disjoint, which means the overlap is 0% .
Revised: Then when we set s = 285 , the local datasets are disjoint, which means the overlap is 0% .

Original: Otherwise, when set s = 213 , the overlap is 25% , and when set s = 142 , the overlap is 50% .
Revised: Otherwise, when set s = 213 , the overlap is 25% , and when set s = 142 , the overlap is 50% .

Original: C.2. A PPROXIMATION M ETHODS
Revised: C.2. A PPROXIMATION M ETHODS

Original: Regularization Methods.
Revised: Regularization Methods.

Original: We use Taylor Extension to approximate the objective functions of previous rounds, that is,
Revised: We use Taylor Extension to approximate the objective functions of previous rounds, that is,

Original: [Equation]
Revised: [Equation]

Original: where ˆ ω are the parameters of previous rounds.
Revised: where ˆ ω are the parameters of previous rounds.

Original: Then the gradient of ˜ f ( ω ) is,
Revised: Then the gradient of ˜ f ( ω ) is,

Original: [Equation]
Revised: [Equation]

Original: Having this, we can approximate the gradients of objective functions of previous rounds.
Revised: Having this, we can approximate the gradients of objective functions of previous rounds.

Original: The problem is how to approximate ∇ 2 f (ˆ ω ) .
Revised: The problem is how to approximate ∇ 2 f (ˆ ω ) .

Original: Here we use two kinds of methods to calculate the Hessian matrices.
Revised: Here we use two kinds of methods to calculate the Hessian matrices.

Original: The ﬁrst is to use the Fisher Information Matrix as introduced in EWC (Kirkpatrick et al., 2017).
Revised: The ﬁrst is to use the Fisher Information Matrix as introduced in EWC (Kirkpatrick et al., 2017).

Original: The
Revised: The

Original: Fisher Information Matrix is equal to the diagonal Hessian matrix when using cross entropy loss functions.
Revised: Fisher Information Matrix is equal to the diagonal Hessian matrix when using cross entropy loss functions.

Original: The Fisher Information Matrix can be calculated by,
Revised: The Fisher Information Matrix can be calculated by,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where F ij are the entries of Fisher Information Matrix.
Revised: where F ij are the entries of Fisher Information Matrix.

Original: Another method is to calculate the diagonal
Revised: Another method is to calculate the diagonal

Original: Hessian matrix.
Revised: Hessian matrix.

Original: We use the PyHessian package (Yao et al., 2020) in this experiment.
Revised: We use the PyHessian package (Yao et al., 2020) in this experiment.

Original: In addition, if the parameters or the Hessian matrix changes too much, the performance of this method is constrained.
Revised: In addition, if the parameters or the Hessian matrix changes too much, the performance of this method is constrained.

Original: To formulate this problem, assume (cid:13)(cid:13) ∇ 2 f ( ω ) (cid:13)(cid:13) ≤ (cid:15) for some arbitrary (cid:15) , we have (cid:107) ∆ t,i ( ω ) (cid:107) ≤ (cid:15) (cid:107) ω − ˆ ω (cid:107) 2 .
Revised: To formulate this problem, assume (cid:13)(cid:13) ∇ 2 f ( ω ) (cid:13)(cid:13) ≤ (cid:15) for some arbitrary (cid:15) , we have [Equation]

Original: The corresponding proof refers to B.3.
Revised: The corresponding proof refers to B.3.

Original: In each round, the server will collect Hessian matrices from clients, and store them in a buffer.
Revised: In each round, the server will collect Hessian matrices from clients, and store them in a buffer.

Original: Then at the beginning of next round, server combine send latest 40 Hessian matrices, gradients, and parameters, and send them to chosen clients.
Revised: Then at the beginning of next round, server combine send latest 40 Hessian matrices, gradients, and parameters, and send them to chosen clients.

Original: Each clients use these to calculate regularization terms.
Revised: Each clients use these to calculate regularization terms.

Original: In practice, we only add regularization terms to the top layers (last block and fc layer of ResNet18).
Revised: In practice, we only add regularization terms to the top layers (last block and fc layer of ResNet18).

Original: This is based on the assumption that top layers contain more personal information while bottom layers contain more general information.
Revised: This is based on the assumption that top layers contain more personal information while bottom layers contain more general information.

Original: We veriﬁed that this strategy perform better than adding regularization terms to all layers in experiments.
Revised: We veriﬁed that this strategy perform better than adding regularization terms to all layers in experiments.

Original: Generation Methods.
Revised: Generation Methods.

Original: We use MCMC to generate samples in each round.
Revised: We use MCMC to generate samples in each round.

Original: In practice, we initialize x from uniform distribution, and update x by,
Revised: In practice, we initialize x from uniform distribution, and update x by,

Original: [Equation]
Revised: [Equation]

Original: 86) where E ( x k − 1) is the function that can measure the distance between x k − 1 and the real local distribution.
Revised: where E ( x k − 1) is the function that can measure the distance between x k − 1 and the real local distribution.

Original: ω ∼ N (0 , σ ) .
Revised: ω ∼ N (0 , σ ) .

Original: In practice, we generate 50-100 samples of each local dataset and add them to the current local datasets for training.
Revised: In practice, we generate 50-100 samples of each local dataset and add them to the current local datasets for training.

Original: However, because of the low quality of the generated data, the improvement is limited.
Revised: However, because of the low quality of the generated data, the improvement is limited.

Original: Besides, the generated data will pollute the batch normalization(BN) layer, and we should use real data to refresh BN layerat the end of each round.
Revised: Besides, the generated data will pollute the batch normalization(BN) layer, and we should use real data to refresh BN layer

Original: [Table]
Revised: [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Core Set Methods.
Revised: Core Set Methods.

Original: Another simple yet effective treatment in CL lines in the category of Exemplar
Revised: Another simple yet effective treatment in CL lines in the category of Exemplar

Original: Replay (Rebufﬁ et al., 2017; Castro et al., 2018).
Revised: Replay (Rebufﬁ et al., 2017; Castro et al., 2018).

Original: This approach stores past core-set samples (a.k.a. exemplars) selectively and periodically, and replays them together with the current local datasets.
Revised: This approach stores past core-set samples (a.k.a. exemplars) selectively and periodically, and replays them together with the current local datasets.

Original: We tried two sample methods.
Revised: We tried two sample methods.

Original: First is so-called Naive method, in which the samples are uniformly chosen from local datasets.
Revised: First is so-called Naive method, in which the samples are uniformly chosen from local datasets.

Original: Except of naive select core sets, we also tried another core set sampling method introduced in iCaRL (Rebufﬁ et al., 2017).
Revised: Except of naive select core sets, we also tried another core set sampling method introduced in iCaRL (Rebufﬁ et al., 2017).

Original: See Algorithm 3 for details.
Revised: See Algorithm 3 for details.

Original: In practice, we save 100 ﬁgures of each local dataset and combine them with the current local datasets.
Revised: In practice, we save 100 ﬁgures of each local dataset and combine them with the current local datasets.

Original: Saving core sets perform best compared with these three approximation methods; however, only valid when the number of clients is limited.
Revised: Saving core sets perform best compared with these three approximation methods; however, only valid when the number of clients is limited.

Original: C.2. A DDITIONAL E XPERIMENTS
Revised: C.2. A DDITIONAL E XPERIMENTS

Original: Applicability of different algorithms.
Revised: Applicability of different algorithms.

Original: We list the applicability of different algorithms under various time-varying scenarios in Table 8.
Revised: We list the applicability of different algorithms under various time-varying scenarios in Table 8.

Original: [Table]
Revised: [Table]

Original: Convergence curves for different settings.
Revised: Convergence curves for different settings.

Original: Figure 4 show convergence curves of CFL and FedAvg on different datasets.
Revised: Figure 4 show convergence curves of CFL and FedAvg on different datasets.

Original: The settings are the same as results in Table 3.
Revised: The settings are the same as results in Table 3.

Original: Models are trained on partitioned datasets with α = 0 . 1 , and all datasets are partitioned to 210 subsets for 7 clients.
Revised: Models are trained on partitioned datasets with α = 0 . 1 , and all datasets are partitioned to 210 subsets for 7 clients.

Original: To show the difference between different algorithms more clearly, all curves are smoothed by a 1D-Mean-Filter.
Revised: To show the difference between different algorithms more clearly, all curves are smoothed by a 1D-Mean-Filter.

Original: Results show CFL can converge to a better optimum compare with FedAvg.
Revised: Results show CFL can converge to a better optimum compare with FedAvg.

Original: Investigating resistance of CFL to time-evolving scenarios.
Revised: Investigating resistance of CFL to time-evolving scenarios.

Original: Figure 6 shows the loss curve of
Revised: Figure 6 shows the loss curve of

Original: CFL-Regularization and FedAvg on different datasets.
Revised: CFL-Regularization and FedAvg on different datasets.

Original: Models are trained on partitioned datasets with α = 0 . 1 , and all datasets are partitioned to 210 subsets for 7 clients.
Revised: Models are trained on partitioned datasets with α = 0 . 1 , and all datasets are partitioned to 210 subsets for 7 clients.

Original: The ﬁrst column shows the loss curve of the ﬁrst 300 rounds, and the second column shows the loss of some chosen details.
Revised: The ﬁrst column shows the loss curve of the ﬁrst 300 rounds, and the second column shows the loss of some chosen details.

Original: Because of the non-iidness of local datasets, the loss will suddenly rise.
Revised: Because of the non-iidness of local datasets, the loss will suddenly rise.

Original: Notice that CFL-Regularization has an apparent mitigation effect on this situation .
Revised: Notice that CFL-Regularization has an apparent mitigation effect on this situation .

Original: Difference between training and test loss.
Revised: Difference between training and test loss.

Original: Figure 5 show the loss on global test data and past appeared training data.
Revised: Figure 5 show the loss on global test data and past appeared training data.

Original: We evaluate the model with stateful clients, set α = 0 . 2 , and use FedAvg algorithm.
Revised: We evaluate the model with stateful clients, set α = 0 . 2 , and use FedAvg algorithm.

Original: We show that there is no signiﬁcant difference between loss value on global test data and past appeared training data.
Revised: We show that there is no signiﬁcant difference between loss value on global test data and past appeared training data.

Original: C. A LGORITHMS
Revised: C. A LGORITHMS

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Algorithm
Revised: Algorithm

Original: Require:
Revised: Require:

Original: S : a list of datasets split by labels, M : the number of clients, N : the size of local datasets, α : the Dirichlet distribution parameter
Revised: S : a list of datasets split by labels, M : the number of clients, N : the size of local datasets, α : the Dirichlet distribution parameter

Original: Ensure: D : split datasets 1: D ← [] 2: G ← [0 , 1 , 2 , ..., len ( S )] 3: for m = 1 , 2 , ..., M do 4: p =
Revised: Ensure: D : split datasets 1: D ← [] 2: G ← [0 , 1 , 2 , ..., len ( S )] 3: for m = 1 , 2 , ..., M do 4: p =

Original: [ p 1 , p 2 , ..., p len ( S ) ] , where p t,i denotes the fraction of class
Revised: [ p 1 , p 2 , ..., p len ( S ) ] , where p t,i denotes the fraction of class

Original: i in total dataset.
Revised: i in total dataset.

Original: 5: θ ← Dirichlet ( α, p ) 6:
Revised: 5: θ ← Dirichlet ( α, p ) 6:

Original: D m ← ∅ 7: while len ( D m ) <
Revised: D m ← ∅ 7: while len ( D m ) <

Original: N do 8: i ← multinomial ( θ, 1) 9: y ← G
Revised: N do 8: i ← multinomial ( θ, 1) 9: y ← G

Original: [ i ] 10: data ← uniform ( S [ y ]) 11:
Revised: [ i ] 10: data ← uniform ( S [ y ]) 11:

Original: D m ← D m ∪ { data } 12:
Revised: D m ← D m ∪ { data } 12:

Original: S [ y ] ← S
Revised: S [ y ] ← S

Original: [ y ] /data 13: if len(S[y]) =
Revised: [ y ] /data 13: if len(S[y]) =

Original: = 0 then 14:
Revised: = 0 then 14:

Original: G ← G/y 15: θ ← renormalize ( θ, i ) 16:
Revised: G ← G/y 15: θ ← renormalize ( θ, i ) 16:

Original: D.append ( D m )
Revised: D.append ( D m )

Original: Require: θ : weights for different classes, i : the class that should be deleted
Revised: Require: θ : weights for different classes, i : the class that should be deleted

Original: Ensure: θ : renormalized weights 1: for j = 1 , 2 , ..., len ( θ ) , j (cid:54)
Revised: Ensure: θ : renormalized weights 1: for j = 1 , 2 , ..., len ( θ ) , j (cid:54)

Original: =
Revised: =

Original: i do 2: θ [ j ] ← θ [ j ] /sum ( theta/theta [ i ])
Revised: i do 2: θ [ j ] ← θ [ j ] /sum ( theta/theta [ i ])