Original: F ED S PEED : L ARGER L OCAL I NTERVAL , L ESS C OMMUNICATION R OUND , AND H IGHER G ENERALIZATION A CCURACY
Revised: F ED S PEED : L ARGER L OCAL I NTERVAL , L ESS C OMMUNICATION R OUND , AND H IGHER G ENERALIZATION A CCURACY

Original: A BSTRACT
Revised: A BSTRACT

Original: Federated learning is an emerging distributed machine learning framework which jointly trains a global model via a large number of local devices with data privacy protections.
Revised: Federated learning is an emerging distributed machine learning framework which jointly trains a global model via a large number of local devices with data privacy protections.

Original: Its performance suffers from the non-vanishing biases introduced by the local inconsistent optimal and the rugged client-drifts by the local over-fitting.
Revised: Its performance suffers from the non-vanishing biases introduced by the local inconsistent optimal and the rugged client-drifts by the local over-fitting.

Original: In this paper, we propose a novel and practical method, FedSpeed, to alleviate the negative impacts posed by these problems.
Revised: In this paper, we propose a novel and practical method, FedSpeed, to alleviate the negative impacts posed by these problems.

Original: Concretely, FedSpeed applies the prox-correction term on the current local updates to efficiently reduce the biases introduced by the prox-term, a necessary regularizer to maintain the strong local consistency.
Revised: Concretely, FedSpeed applies the prox-correction term on the current local updates to efficiently reduce the biases introduced by the prox-term, a necessary regularizer to maintain the strong local consistency.

Original: Furthermore, FedSpeed merges the vanilla stochastic gradient with a perturbation computed from an extra gradient ascent step in the neighborhood, thereby alleviating the issue of local over-fitting.
Revised: Furthermore, FedSpeed merges the vanilla stochastic gradient with a perturbation computed from an extra gradient ascent step in the neighborhood, thereby alleviating the issue of local over-fitting.

Original: Our theoretical analysis indicates that the convergence rate is related to both the communication rounds T and local intervals K with a upper bound O (1 /T )
Revised: Our theoretical analysis indicates that the convergence rate is related to both the communication rounds T and local intervals K with a upper bound O (1 /T )

Original: if setting a proper local interval.
Revised: if setting a proper local interval.

Original: Moreover, we conduct extensive experiments on the real-world dataset to demonstrate the efficiency of our proposed FedSpeed, which performs significantly faster and achieves the state-of-the-art (SOTA) performance on the general FL experimental settings than several baselines including FedAvg, FedProx, FedCM, FedAdam,
Revised: Moreover, we conduct extensive experiments on the real-world dataset to demonstrate the efficiency of our proposed FedSpeed, which performs significantly faster and achieves the state-of-the-art (SOTA) performance on the general FL experimental settings than several baselines including FedAvg, FedProx, FedCM, FedAdam,

Original: SCAFFOLD, FedDyn, FedADMM, etc.
Revised: SCAFFOLD, FedDyn, FedADMM, etc.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Since McMahan et al.
Revised: Since McMahan et al.

Original: (2017) proposed federated learning (FL), it has gradually evolved into an efficient paradigm for large-scale distributed training.
Revised: (2017) proposed federated learning (FL), it has gradually evolved into an efficient paradigm for large-scale distributed training.

Original: Different from the traditional deep learning methods, FL allows multi local clients to jointly train a single global model without data sharing.
Revised: Different from the traditional deep learning methods, FL allows multi local clients to jointly train a single global model without data sharing.

Original: However, FL is far from its maturity, as it still suffers from the considerable performance degradation over the heterogeneously distributed data, a very common setting in the practical application of FL.
Revised: However, FL is far from its maturity, as it still suffers from the considerable performance degradation over the heterogeneously distributed data, a very common setting in the practical application of FL.

Original: We recognize the main culprit leading to the performance degradation of FL as local inconsistency and local heterogeneous over-fitting .
Revised: We recognize the main culprit leading to the performance degradation of FL as local inconsistency and local heterogeneous over-fitting .

Original: Specifically, for canonical local-SGD-based FL method, e.g.,
Revised: Specifically, for canonical local-SGD-based FL method, e.g.,

Original: FedAvg, the non-vanishing biases introduced by the local updates may eventually lead to inconsistent local solution.
Revised: FedAvg, the non-vanishing biases introduced by the local updates may eventually lead to inconsistent local solution.

Original: Then, the rugged client-drifts resulting from the local over-fitting into inconsistent local solutions may make the obtained global model degrading into the average of client’s local parameters.
Revised: Then, the rugged client-drifts resulting from the local over-fitting into inconsistent local solutions may make the obtained global model degrading into the average of client’s local parameters.

Original: The non-vanishing biases have been studied by several previous works Charles & Koneˇcn`y (2021);
Revised: The non-vanishing biases have been studied by several previous works Charles & Koneˇcn`y (2021);

Original: Malinovskiy et al.
Revised: Malinovskiy et al.

Original: (2020) in different forms.
Revised: (2020) in different forms.

Original: The inconsistency due to the local heterogeneous data will compromise the global convergence during the training process.
Revised: The inconsistency due to the local heterogeneous data will compromise the global convergence during the training process.

Original: Eventually it leads to serious client-drifts which can be formulated as x ∗ ̸
Revised: Eventually it leads to serious client-drifts which can be formulated as x ∗ ̸

Original: = (cid:80) i ∈ [ m ] x ∗ i /m .
Revised: = (cid:80) i ∈ [ m ] x ∗ i /m .

Original: Larger data heterogeneity may enlarge the drifts, thereby degrading the practical training convergence rate and generalization performance.
Revised: Larger data heterogeneity may enlarge the drifts, thereby degrading the practical training convergence rate and generalization performance.

Original: In order to strengthen the local consistency during the local training process, and avoid the client-drifts resulting from the local over-fitting, we propose a novel and practical algorithm, dubbed as FedSpeed .
Revised: In order to strengthen the local consistency during the local training process, and avoid the client-drifts resulting from the local over-fitting, we propose a novel and practical algorithm, dubbed as FedSpeed .

Original: Notably, FedSpeed incorporates two novel components to achieve SOTA performance.
Revised: Notably, FedSpeed incorporates two novel components to achieve SOTA performance.

Original: i)
Revised: i)

Original: Firstly,
Revised: Firstly,

Original: FedSpeed inherits a penalized prox-term to force the local offset to be closer to the initial point at each communication round.
Revised: FedSpeed inherits a penalized prox-term to force the local offset to be closer to the initial point at each communication round.

Original: However, recognized from Hanzely & Richt´arik (2020); Khaled et al.
Revised: However, recognized from Hanzely & Richt´arik (2020); Khaled et al.

Original: (2019) that the prox-term between global and local solutions may introduce undesirable local training bias, we propose and utilize a prox-correction term to counteract the adverse impact.
Revised: (2019) that the prox-term between global and local solutions may introduce undesirable local training bias, we propose and utilize a prox-correction term to counteract the adverse impact.

Original: Indeed, in our theoretical analysis, the implication of the prox-correction term could be considered as a momentumbased term of the weighted local gradients.
Revised: Indeed, in our theoretical analysis, the implication of the prox-correction term could be considered as a momentumbased term of the weighted local gradients.

Original: Via utilizing the historical gradient information, the bias brought by the prox-term can be effectively corrected.
Revised: Via utilizing the historical gradient information, the bias brought by the prox-term can be effectively corrected.

Original: ii)
Revised: ii)

Original: Secondly, to avoid the rugged local over-fitting, FedSpeed incorporates a local gradient perturbation via merging the vanilla stochastic gradient with an extra gradient, which can be viewed as taking an extra gradient ascent step for each local update.
Revised: Secondly, to avoid the rugged local over-fitting, FedSpeed incorporates a local gradient perturbation via merging the vanilla stochastic gradient with an extra gradient, which can be viewed as taking an extra gradient ascent step for each local update.

Original: Based on the analysis in Zhao et al.
Revised: Based on the analysis in Zhao et al.

Original: ; van der Hoeven (2020), we demonstrate that the gradient perturbation term could be approximated as adding a penalized squared L 2 -norm of the stochastic gradients to the original objective function, which can efficiently search for the flatten local minima Andriushchenko & Flammarion (2022) to prevent the local over-fitting problems.
Revised: ; van der Hoeven (2020), we demonstrate that the gradient perturbation term could be approximated as adding a penalized squared L 2 -norm of the stochastic gradients to the original objective function, which can efficiently search for the flatten local minima Andriushchenko & Flammarion (2022) to prevent the local over-fitting problems.

Original: We also provide the theoretical analysis of our proposed FedSpeed and further demonstrate that its convergence rate could be accelerated by setting an appropriate large local interval K .
Revised: We also provide the theoretical analysis of our proposed FedSpeed and further demonstrate that its convergence rate could be accelerated by setting an appropriate large local interval K .

Original: Explicitly, under the non-convex and smooth cases, FedSpeed with an extra gradient perturbation could achieve the fast convergence rate of O (1 /T ) , which indicates that FedSpeed achieves a tighter upper bound with a proper local interval K to converge, without applying a specific global learning rate or assuming the precision for the local solutions (Durmus et al., 2021;
Revised: Explicitly, under the non-convex and smooth cases, FedSpeed with an extra gradient perturbation could achieve the fast convergence rate of O (1 /T ) , which indicates that FedSpeed achieves a tighter upper bound with a proper local interval K to converge, without applying a specific global learning rate or assuming the precision for the local solutions (Durmus et al., 2021;

Original: Wang et al., 2022).
Revised: Wang et al., 2022).

Original: Extensive experiments are tested on CIFAR-10/100 and TinyImagenet dataset with a standard ResNet-18-GN network under the different heterogeneous settings, which shows that our proposed FedSpeed is significantly better than several baselines, e.g. for FedAvg, FedProx, FedCM, FedPD, SCAFFOLD, FedDyn, on both the stability to enlarge the local interval K and the test generalization performance in the actual training.
Revised: Extensive experiments are tested on CIFAR-10/100 and TinyImagenet dataset with a standard ResNet-18-GN network under the different heterogeneous settings, which shows that our proposed FedSpeed is significantly better than several baselines, e.g. for FedAvg, FedProx, FedCM, FedPD, SCAFFOLD, FedDyn, on both the stability to enlarge the local interval K and the test generalization performance in the actual training.

Original: To the end, we summarize the main contributions of this paper as follows:
Revised: To the end, we summarize the main contributions of this paper as follows:

Original: • We propose a novel and practical federated optimization algorithm, FedSpeed , which applies a prox-correction term to significantly reduce the bias due to the local updates of the prox-term, and an extra gradient perturbation to efficiently avoid the local over-fitting, which achieves a fast convergence speed with large local steps and simultaneously maintains the high generalization.
Revised: • We propose a novel and practical federated optimization algorithm, FedSpeed , which applies a prox-correction term to significantly reduce the bias due to the local updates of the prox-term, and an extra gradient perturbation to efficiently avoid the local over-fitting, which achieves a fast convergence speed with large local steps and simultaneously maintains the high generalization.

Original: • We provide the convergence rate upper bound under the non-convex and smooth cases and prove that FedSpeed could achieve a fast convergence rate of O (1 /T ) via enlarging the local training interval K
Revised: • We provide the convergence rate upper bound under the non-convex and smooth cases and prove that FedSpeed could achieve a fast convergence rate of O (1 /T ) via enlarging the local training interval K

Original: = O ( T ) without any other harsh assumptions or the specific conditions required.
Revised: = O ( T ) without any other harsh assumptions or the specific conditions required.

Original: • Extensive experiments are conducted on the CIFAR-10/100 and TinyImagenet dataset to verify the performance of our proposed FedSpeed.
Revised: • Extensive experiments are conducted on the CIFAR-10/100 and TinyImagenet dataset to verify the performance of our proposed FedSpeed.

Original: To the best of our interests, both convergence speed and generalization performance could achieve the SOTA results under the general federated settings.
Revised: To the best of our interests, both convergence speed and generalization performance could achieve the SOTA results under the general federated settings.

Original: FedSpeed could outperform other baselines and be more robust to enlarging the local interval.
Revised: FedSpeed could outperform other baselines and be more robust to enlarging the local interval.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: McMahan et al.
Revised: McMahan et al.

Original: (2017) propose the federated framework with the properties of jointly training with several unbalance and non-iid local dataset via communicating with lower costs during the total training stage.
Revised: (2017) propose the federated framework with the properties of jointly training with several unbalance and non-iid local dataset via communicating with lower costs during the total training stage.

Original: The general FL optimization involves a local client training stage and a global server update operation Asad et al.
Revised: The general FL optimization involves a local client training stage and a global server update operation Asad et al.

Original: and it has been proved to achieve a linear speedup property in
Revised: and it has been proved to achieve a linear speedup property in

Original: Yang et al.
Revised: Yang et al.

Original: With the fast development of the FL, a series of efficient optimization method are applied in the federated framework.
Revised: With the fast development of the FL, a series of efficient optimization method are applied in the federated framework.

Original: Li et al.
Revised: Li et al.

Original: (2020b) and Kairouz et al.
Revised: (2020b) and Kairouz et al.

Original: (2021) introduce a detailed overview in this field.
Revised: (2021) introduce a detailed overview in this field.

Original: There are still many difficulties to be solved in the practical scenarios, while in this paper we focus to highlight the two main challenges of the local inconsistent solution and client-drifts due to heterogeneous over-fitting, which are two acute limitations in the federated
Revised: There are still many difficulties to be solved in the practical scenarios, while in this paper we focus to highlight the two main challenges of the local inconsistent solution and client-drifts due to heterogeneous over-fitting, which are two acute limitations in the federated

Original: [Table]
Revised: [Table]

Original: Local consistency.
Revised: Local consistency.

Original: Sahu et al.
Revised: Sahu et al.

Original: (2018) study the non-vanishing biases of the inconsistent solution in the experiments and apply a prox-term regularization.
Revised: (2018) study the non-vanishing biases of the inconsistent solution in the experiments and apply a prox-term regularization.

Original: FedProx utilizes the bounded local updates by penalizing parameters to provide a good guarantee of consistency.
Revised: FedProx utilizes the bounded local updates by penalizing parameters to provide a good guarantee of consistency.

Original: In Liang et al.
Revised: In Liang et al.

Original: they introduce the local gradient tracking to reduce the local inconsistency in the local SGD method.
Revised: they introduce the local gradient tracking to reduce the local inconsistency in the local SGD method.

Original: Charles & Koneˇcn`y (2021); Malinovskiy et al.
Revised: Charles & Koneˇcn`y (2021); Malinovskiy et al.

Original: (2020) show that the local learning rate decay can balance the trade-off between the convergence rate and the local inconsistency with the rate of
Revised: (2020) show that the local learning rate decay can balance the trade-off between the convergence rate and the local inconsistency with the rate of

Original: O ( η l ( K − 1)) .
Revised: O ( η l ( K − 1)) .

Original: Furthermore, Wang et al.
Revised: Furthermore, Wang et al.

Original: (2021; 2020b) through a simple counterexample to show that using adaptive optimizer or different hyper-parameters on local clients leads to an additional gaps.
Revised: (2021; 2020b) through a simple counterexample to show that using adaptive optimizer or different hyper-parameters on local clients leads to an additional gaps.

Original: They propose a local correction technique to alleviate the biases.
Revised: They propose a local correction technique to alleviate the biases.

Original: Wang et al.
Revised: Wang et al.

Original: (2020a);
Revised: (2020a);

Original: Tan et al.
Revised: Tan et al.

Original: (2022) consider the different local settings and prove that in the case of asynchronous aggregation, the inconsistency bias will no longer be eliminated by local learning rate decay.
Revised: (2022) consider the different local settings and prove that in the case of asynchronous aggregation, the inconsistency bias will no longer be eliminated by local learning rate decay.

Original: Haddadpour et al.
Revised: Haddadpour et al.

Original: (2021) compress the local offset and adopt a global correction to reduce the biases.
Revised: (2021) compress the local offset and adopt a global correction to reduce the biases.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2021) apply the primal dual method instead of the primal method to solve a series of sub-problems on the local clients and alternately updates the primal and dual variables which can achieve the fast convergence rate of O ( 1 T ) with the local solution precision assumption.
Revised: (2021) apply the primal dual method instead of the primal method to solve a series of sub-problems on the local clients and alternately updates the primal and dual variables which can achieve the fast convergence rate of O ( 1 T ) with the local solution precision assumption.

Original: Based on FedPD, Durmus et al.
Revised: Based on FedPD, Durmus et al.

Original: (2021) propose FedDyn as a variants via averaging all the dual variables (the average quantity can then be viewed as the global gradient) under the partial participation settings, which can also achieve the same O ( 1 T ) under the assumption that exact local solution can be found by the local optimizer.
Revised: (2021) propose FedDyn as a variants via averaging all the dual variables (the average quantity can then be viewed as the global gradient) under the partial participation settings, which can also achieve the same O ( 1 T ) under the assumption that exact local solution can be found by the local optimizer.

Original: Wang et al.
Revised: Wang et al.

Original: ; Gong et al.
Revised: ; Gong et al.

Original: (2022) propose two other variants to apply different dual variable aggregation strategies under partial participation settings.
Revised: (2022) propose two other variants to apply different dual variable aggregation strategies under partial participation settings.

Original: These methods benefit from applying the prox-term Li et al.
Revised: These methods benefit from applying the prox-term Li et al.

Original: ; Chen & Chao (2020) or higher efficient optimization methods Bischoff et al.
Revised: ; Chen & Chao (2020) or higher efficient optimization methods Bischoff et al.

Original: ; Yang et al.
Revised: ; Yang et al.

Original: (2022) to control the local consistency.
Revised: (2022) to control the local consistency.

Original: Client-drifts.
Revised: Client-drifts.

Original: Karimireddy et al.
Revised: Karimireddy et al.

Original: (2020) firstly demonstrate the client-drifts for federated learning framework to indicate the negative impact on the global model when each local client over-fits to the local heterogeneous dataset.
Revised: (2020) firstly demonstrate the client-drifts for federated learning framework to indicate the negative impact on the global model when each local client over-fits to the local heterogeneous dataset.

Original: They propose SCAFFOLD via a variance reduction technique to mitigate this drifts.
Revised: They propose SCAFFOLD via a variance reduction technique to mitigate this drifts.

Original: Yu et al.
Revised: Yu et al.

Original: (2019) and Wang et al.
Revised: (2019) and Wang et al.

Original: (2019) introduce the momentum instead of the gradient to the local and global update respectively to improve the generalization performance.
Revised: (2019) introduce the momentum instead of the gradient to the local and global update respectively to improve the generalization performance.

Original: To maintain the property of consistency, Xu et al.
Revised: To maintain the property of consistency, Xu et al.

Original: (2021) propose a novel client-level momentum term to improve the local training process.
Revised: (2021) propose a novel client-level momentum term to improve the local training process.

Original: Ozfatura et al.
Revised: Ozfatura et al.

Original: (2021) incorporate the client-level momentum with local momentum to further control the biases.
Revised: (2021) incorporate the client-level momentum with local momentum to further control the biases.

Original: In recent Gao et al.
Revised: In recent Gao et al.

Original: ; Kim et al.
Revised: ; Kim et al.

Original: (2022), they propose a drift correction term as a penalized loss on the original local objective functions with a global gradient estimation.
Revised: (2022), they propose a drift correction term as a penalized loss on the original local objective functions with a global gradient estimation.

Original: Chen et al.
Revised: Chen et al.

Original: and
Revised: and

Original: Chen et al.
Revised: Chen et al.

Original: (2021; 2022) focus on the adaptive method to alleviate the biases and improve the efficiency.
Revised: (2021; 2022) focus on the adaptive method to alleviate the biases and improve the efficiency.

Original: Our proposed FedSpeed inherits the prox-term at local update to guarantee the local consistency during local training.
Revised: Our proposed FedSpeed inherits the prox-term at local update to guarantee the local consistency during local training.

Original: Different from the previous works, we adopt an extra prox-correction term to reduce the bias during the local training introduced by the update direction towards the last global model parameters.
Revised: Different from the previous works, we adopt an extra prox-correction term to reduce the bias during the local training introduced by the update direction towards the last global model parameters.

Original: This ensures that the local update could be corrected towards the global minima.
Revised: This ensures that the local update could be corrected towards the global minima.

Original: Furthermore, we incorporate a gradient perturbation update to enhance the generalization performance of the local model, which merges a gradient ascent step.
Revised: Furthermore, we incorporate a gradient perturbation update to enhance the generalization performance of the local model, which merges a gradient ascent step.

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: In this part, we will introduce the preliminaries and our proposed method.
Revised: In this part, we will introduce the preliminaries and our proposed method.

Original: We will explain the implicit meaning for each variables and demonstrate the FedSpeed algorithm inference in details.
Revised: We will explain the implicit meaning for each variables and demonstrate the FedSpeed algorithm inference in details.

Original: Notations and preliminary.
Revised: Notations and preliminary.

Original: Let m be the number of total clients.
Revised: Let m be the number of total clients.

Original: We denote S t as the set of active clients at round t .
Revised: We denote S t as the set of active clients at round t .

Original: K is the number of local updates and T is the communication rounds.
Revised: K is the number of local updates and T is the communication rounds.

Original: ( · ) ti,k denotes variable ( · ) at k -th iteration of t -th round in the i -th client.
Revised: ( · ) ti,k denotes variable ( · ) at k -th iteration of t -th round in the i -th client.

Original: x is the model parameters.
Revised: x is the model parameters.

Original: g is the stochastic gradient computed by the sampled data.
Revised: g is the stochastic gradient computed by the sampled data.

Original: ˜ g is the weighted quasi-gradient computed as defined in Algorithm 1.
Revised: ˜ g is the weighted quasi-gradient computed as defined in Algorithm 1.

Original: ˆ g is the prox-correction term.
Revised: ˆ g is the prox-correction term.

Original: We denote ⟨· , ·⟩ as the inner product for two vectors and ∥ · ∥ is the Euclidean norm of a vector.
Revised: We denote ⟨· , ·⟩ as the inner product for two vectors and ∥ · ∥ is the Euclidean norm of a vector.

Original: Other symbols are detailed at their references.
Revised: Other symbols are detailed at their references.

Original: As the most FL frameworks, we consider to minimize the following finite-sum non-convex problem:
Revised: As the most FL frameworks, we consider to minimize the following finite-sum non-convex problem:

Original: [Equation]
Revised: [Equation]

Original: where F : R d → R , F i ( x ) :
Revised: where F : R d → R , F i ( x ) :

Original: =
Revised: =

Original: E ε i
Revised: E ε i

Original: ∼D
Revised: ∼D

Original: i F i ( x , ε i ) is objective function in the client i , and ε i represents for the random data samples obeying the distribution D i .
Revised: i F i ( x , ε i ) is objective function in the client i , and ε i represents for the random data samples obeying the distribution D i .

Original: m is the total number of clients.
Revised: m is the total number of clients.

Original: In FL, D i may differ across the local clients, which may introduce the client drifts by the heterogeneous data.
Revised: In FL, D i may differ across the local clients, which may introduce the client drifts by the heterogeneous data.

Original: 3. F ED S PEED A LGORITHM
Revised: 3. F ED S PEED A LGORITHM

Original: In this part, we will introduce our proposed method to alleviate the negative impact of the heterogeneous data and reduces the communication rounds.
Revised: In this part, we will introduce our proposed method to alleviate the negative impact of the heterogeneous data and reduces the communication rounds.

Original: We are inspired by the dynamic regularization
Revised: We are inspired by the dynamic regularization

Original: Durmus et al.
Revised: Durmus et al.

Original: (2021) for the local updates to eliminate the client drifts when T approaches infinite.
Revised: (2021) for the local updates to eliminate the client drifts when T approaches infinite.

Original: Our proposed FeedSpeed is shown in Algorithm 1.
Revised: Our proposed FeedSpeed is shown in Algorithm 1.

Original: At the beginning of each round t , a subset of clients S t are required to participate in the current training process.
Revised: At the beginning of each round t , a subset of clients S t are required to participate in the current training process.

Original: The global server will communicate the parameters x t to the active clients for local training.
Revised: The global server will communicate the parameters x t to the active clients for local training.

Original: Each active local client performs three stages: (1) computing the unbiased stochastic gradient g ti,k, 1 = ∇ F i ( x ti,k ; ε ti,k ) with a randomly sampled mini-batch data ε ti,k and executing a gradient ascent step in the neighbourhood to approach ˘ x ti,k ; (2) computing the unbiased stochastic gradient g ti,k, 2 with the same sampled minibatch data in (1) at the ˘ x t i,k and merging the g t i,k, 1 with g t i,k, 2 to introduce a basic perturbation to the vanilla descent direction; (3) executing the gradient descent step with the merged quasi-gradient ˜ g ti,k , the prox-term ∥ x ti,k − x t ∥ 2 and the local prox-correction term ˆ g t − 1 i .
Revised: Each active local client performs three stages: (1) computing the unbiased stochastic gradient g ti,k, 1 = ∇ F i ( x ti,k ; ε ti,k ) with a randomly sampled mini-batch data ε ti,k and executing a gradient ascent step in the neighbourhood to approach ˘ x ti,k ; (2) computing the unbiased stochastic gradient g ti,k, 2 with the same sampled minibatch data in (1) at the ˘ x t i,k and merging the g t i,k, 1 with g t i,k, 2 to introduce a basic perturbation to the vanilla descent direction; (3) executing the gradient descent step with the merged quasi-gradient ˜ g ti,k , the prox-term ∥ x ti,k − x t ∥ 2 and the local prox-correction term ˆ g t − 1 i .

Original: After K iterations local training, prox-correction term ˆ g t − 1 i will be updated as the weighted sum of the current local offset ( x ti,K − x ti, 0 ) and the historical offsets momentum.
Revised: After K iterations local training, prox-correction term ˆ g t − 1 i will be updated as the weighted sum of the current local offset ( x ti,K − x ti, 0 ) and the historical offsets momentum.

Original: Then we communicate the amended model parameters ˆ x ti = x
Revised: Then we communicate the amended model parameters ˆ x ti = x

Original: ti,K − λ
Revised: ti,K − λ

Original: ˆ g ti to the global server for aggregation.
Revised: ˆ g ti to the global server for aggregation.

Original: On the global server, a simple average aggregation is applied to generate the current global model parameters x t +1 at round t .
Revised: On the global server, a simple average aggregation is applied to generate the current global model parameters x t +1 at round t .

Original: Prox-correction term.
Revised: Prox-correction term.

Original: In the general optimization, the prox-term ∥ x ti,k − x t ∥ 2 is a penalized term for solving the non-smooth problems and it contributes to strengthen the local consistency in the FL framework by introducing a penalized direction in the local updates as proposed in Sahu et al.
Revised: In the general optimization, the prox-term ∥ x ti,k − x t ∥ 2 is a penalized term for solving the non-smooth problems and it contributes to strengthen the local consistency in the FL framework by introducing a penalized direction in the local updates as proposed in Sahu et al.

Original: However, as discussed in Hanzely & Richt´arik (2020), it simply performs as a balance between the local and global solutions, and there still exists the non-vanishing inconsistent biases among the local solutions, i.e., the local solutions are still largely deviated from each other, implying that local inconsistency is still not eliminated, which limits the efficiency of the federated learning framework.
Revised: However, as discussed in Hanzely & Richt´arik (2020), it simply performs as a balance between the local and global solutions, and there still exists the non-vanishing inconsistent biases among the local solutions, i.e., the local solutions are still largely deviated from each other, implying that local inconsistency is still not eliminated, which limits the efficiency of the federated learning framework.

Original: To further strengthen the local consistency, we utilize a prox-correction term
Revised: To further strengthen the local consistency, we utilize a prox-correction term

Original: ˆ g ti which could be considered as a previous local offset momentum.
Revised: ˆ g ti which could be considered as a previous local offset momentum.

Original: According to the local update, we combine the x t i,k − 1 term in the prox term and the local state, setting the weight as ( 1 − η l λ ) multiplied to the basic local state.
Revised: According to the local update, we combine the x t i,k − 1 term in the prox term and the local state, setting the weight as ( 1 − η l λ ) multiplied to the basic local state.

Original: As shown in the local update in Algorithm 1 (Line.11), for ∀ x ∈ R
Revised: As shown in the local update in Algorithm 1 (Line.11), for ∀ x ∈ R

Original: d we have:
Revised: d we have:

Original: [Equation]
Revised: [Equation]

Original: where (cid:80) K − 1 k =

Revised: where (cid:80) K − 1 k =

Original: γ k = (cid:80) K − 1 k =

Revised: γ k = (cid:80) K − 1 k =

Original: η l λ (cid:0) 1 − η l λ (cid:1) K − 1 − k = γ .
Revised: η l λ (cid:0) 1 − η l λ (cid:1) K − 1 − k = γ .

Original: Proof details can be referred to the Appendix .
Revised: Proof details can be referred to the Appendix .

Original: Firstly let ˆ g − 1 i = 0 , Equation (2) indicates that the local offset will be transferred to a exponential average of previous local gradients when applying the prox-term, and the updated formation of the local offset is independent of the local learning rate η l .
Revised: Firstly let ˆ g − 1 i = 0 , Equation (2) indicates that the local offset will be transferred to a exponential average of previous local gradients when applying the prox-term, and the updated formation of the local offset is independent of the local learning rate η l .

Original: This is different from the vanilla SGD-based methods, e.g. FedAvg, which treats all local updates fairly.
Revised: This is different from the vanilla SGD-based methods, e.g. FedAvg, which treats all local updates fairly.

Original: γ k changes the importance of the historical gradients.
Revised: γ k changes the importance of the historical gradients.

Original: As K increases, previous updates will be weakened by exponential decay significantly for η l < λ .
Revised: As K increases, previous updates will be weakened by exponential decay significantly for η l < λ .

Original: Thus, we apply the prox-correction term to balance the local offset.
Revised: Thus, we apply the prox-correction term to balance the local offset.

Original: According to the iterative formula for ˆ g ti (Line.13 in Algorithm 1) and the equation (2), we can rewrite this update as:
Revised: According to the iterative formula for ˆ g ti (Line.13 in Algorithm 1) and the equation (2), we can rewrite this update as:

Original: [Equation]
Revised: [Equation]

Original: where γ and γ k is defined the same as in Equation (2).
Revised: where γ and γ k is defined the same as in Equation (2).

Original: Proof details can be referred to the Appendix .
Revised: Proof details can be referred to the Appendix .

Original: Note that ˆ g ti performs as a momentum term of the historical local updates before round t , which can be considered as a estimation of the local offset at round t .
Revised: Note that ˆ g ti performs as a momentum term of the historical local updates before round t , which can be considered as a estimation of the local offset at round t .

Original: At each local iteration k of round t , ˆ g t − 1 i provides a correction for the local update to balance the impact of the prox-term to enhance the contribution of those descent steps executed firstly at each local stages.
Revised: At each local iteration k of round t , ˆ g t − 1 i provides a correction for the local update to balance the impact of the prox-term to enhance the contribution of those descent steps executed firstly at each local stages.

Original: It should be noted that ˆ g t − 1 i is different from the global momentum term mentioned in Wang et al.
Revised: It should be noted that ˆ g t − 1 i is different from the global momentum term mentioned in Wang et al.

Original: (2019) which aggregates the average local updates to improve the generalization performance.
Revised: (2019) which aggregates the average local updates to improve the generalization performance.

Original: After the local training, it updates the current information.
Revised: After the local training, it updates the current information.

Original: Then we subtract the current ˆ g ti from the local models x ti,K to counteract the influence in the local stages.
Revised: Then we subtract the current ˆ g ti from the local models x ti,K to counteract the influence in the local stages.

Original: Finally it sends the post-processed parameters ˆ x ti,K to the global server.
Revised: Finally it sends the post-processed parameters ˆ x ti,K to the global server.

Original: Gradient perturbation.
Revised: Gradient perturbation.

Original: Gradient perturbations (Foret et al., 2020a; Mi et al.
Revised: Gradient perturbations (Foret et al., 2020a; Mi et al.

Original: ; Zhao et al., 2022;
Revised: ; Zhao et al., 2022;

Original: Zhong et al., 2022) significantly improves generalization for deep models.
Revised: Zhong et al., 2022) significantly improves generalization for deep models.

Original: An extra gradient ascent in the neighbourhood can effectively express the curvature near the current parameters.
Revised: An extra gradient ascent in the neighbourhood can effectively express the curvature near the current parameters.

Original: Referring to the analysis in Zhao et al.
Revised: Referring to the analysis in Zhao et al.

Original: (2022), we show that the quasi-gradient ˜ g , which merges the extra ascent step gradient and the vanilla gradient, could be approximated as penalizing a square term of the L 2 -norm of the gradient on the original function.
Revised: (2022), we show that the quasi-gradient ˜ g , which merges the extra ascent step gradient and the vanilla gradient, could be approximated as penalizing a square term of the L 2 -norm of the gradient on the original function.

Original: On each local client to solve the stationary point of min x { F i ( x ) + β ∥∇ F i ( x ) ∥ 2 } can search for a flat minima.
Revised: On each local client to solve the stationary point of min x { F i ( x ) + β ∥∇ F i ( x ) ∥ 2 } can search for a flat minima.

Original: Flatten loss landscapes will further mitigate the local inconsistency due to the averaging aggregation on the global server on heterogeneous dataset.
Revised: Flatten loss landscapes will further mitigate the local inconsistency due to the averaging aggregation on the global server on heterogeneous dataset.

Original: Detailed discussions can be referred to the Appendix .
Revised: Detailed discussions can be referred to the Appendix .

Original:  C ONVERGENCE A NALYSIS
Revised:  C ONVERGENCE A NALYSIS

Original: In this part we will demonstrate the theoretical analysis of our proposed FedSpeed and illustrate the convergence guarantees under the specific hyperparameters.
Revised: In this part we will demonstrate the theoretical analysis of our proposed FedSpeed and illustrate the convergence guarantees under the specific hyperparameters.

Original: Due to the space limitations, more details could be referred to the Appendix .
Revised: Due to the space limitations, more details could be referred to the Appendix .

Original: Some standard assumptions are stated as follows.
Revised: Some standard assumptions are stated as follows.

Original: Assumption 4.1 ( L-Smoothness ) For the non-convex function F i holds the property of smoothness for all i ∈ [ m ] , i.e., ∥∇ F i ( x ) − ∇ F i ( y )
Revised: Assumption 4.1 ( L-Smoothness ) For the non-convex function F i holds the property of smoothness for all i ∈ [ m ] , i.e., ∥∇ F i ( x ) − ∇ F i ( y )

Original: ∥
Revised: ∥

Original: ≤ L ∥ x − y ∥ , for all x , y ∈ R d .
Revised: ≤ L ∥ x − y ∥ , for all x , y ∈ R d .

Original: Assumption 4.2 ( Bounded Stochastic Gradient )
Revised: Assumption 4.2 ( Bounded Stochastic Gradient )

Original: The stochastic gradient g ti,k = ∇ F i ( x ti,k , ε ti,k ) with the randomly sampled data ε ti,k on the local client i is an unbiased estimator of ∇ F i with bounded variance, i.e., E [ g ti,k ] = ∇ F i ( x ti,k ) and E ∥ g ti,k − ∇ F i ( x ti,k ) ∥ 2 ≤ σ 2 l , for all x ti,k ∈ R d .
Revised: The stochastic gradient g ti,k = ∇ F i ( x ti,k , ε ti,k ) with the randomly sampled data ε ti,k on the local client i is an unbiased estimator of ∇ F i with bounded variance, i.e., E [ g ti,k ] = ∇ F i ( x ti,k ) and E ∥ g ti,k − ∇ F i ( x ti,k ) ∥ 2 ≤ σ 2 l , for all x ti,k ∈ R d .

Original: Assumption 4.3 ( Bounded Heterogeneity )
Revised: Assumption 4.3 ( Bounded Heterogeneity )

Original: The dissimilarity of the dataset among the local clients is bounded by the local and global gradients, i.e., E ∥∇ F i ( x ) − ∇ F ( x ) ∥ 2 ≤ σ 2 g , for all x ∈ R d .
Revised: The dissimilarity of the dataset among the local clients is bounded by the local and global gradients, i.e., E ∥∇ F i ( x ) − ∇ F ( x ) ∥ 2 ≤ σ 2 g , for all x ∈ R d .

Original: Assumption 4.1 guarantees a Lipschitz continuity and Assumption 4.2 guarantees the stochastic gradient is bounded by zero mean and constant variance.
Revised: Assumption 4.1 guarantees a Lipschitz continuity and Assumption 4.2 guarantees the stochastic gradient is bounded by zero mean and constant variance.

Original: Assumption 4.3 is the heterogeneity bound for the non-iid dataset, which is widely used in many previous works (Reddi et al., 2020;
Revised: Assumption 4.3 is the heterogeneity bound for the non-iid dataset, which is widely used in many previous works (Reddi et al., 2020;

Original: Yang et al.,
Revised: Yang et al.,

Original: Xu et al., 2021; Wang et al., 2021; Karimi et al., 2021).
Revised: Xu et al., 2021; Wang et al., 2021; Karimi et al., 2021).

Original: Our theoretical analysis depends on the above assumptions to explore the comprehensive properties in the local training process.
Revised: Our theoretical analysis depends on the above assumptions to explore the comprehensive properties in the local training process.

Original: Proof sketch.
Revised: Proof sketch.

Original: To express the essential insights in the updates of the Algorithm 1, we introduce two auxiliary sequences.
Revised: To express the essential insights in the updates of the Algorithm 1, we introduce two auxiliary sequences.

Original: Considering the u t = 1 m (cid:80) i ∈
Revised: Considering the u t = 1 m (cid:80) i ∈

Original: [ m ] x ti,K as the mean averaged parameters of the last iterations in the local training among the local clients.
Revised: [ m ] x ti,K as the mean averaged parameters of the last iterations in the local training among the local clients.

Original: Based on { u t } , we introduce the auxiliary sequences { z t = u t + 1 − γγ
Revised: Based on { u t } , we introduce the auxiliary sequences { z t = u t + 1 − γγ

Original: [Equation]
Revised: [Equation]

Original: u t − u t
Revised: u t − u t

Original: [Equation]
Revised: [Equation]

Original: 1 ) } t> 0 .
Revised: 1 ) } t> 0 .

Original: Combining the local update and the Equation (3):
Revised: Combining the local update and the Equation (3):

Original: [Equation]
Revised: [Equation]

Original: If we introduce K virtual states u ti,k , it could be considered as a momentum-based update of the proxcorrection term ˆ g t − 1 i with the coefficient γ .
Revised: If we introduce K virtual states u ti,k , it could be considered as a momentum-based update of the proxcorrection term ˆ g t − 1 i with the coefficient γ .

Original: And the prox-correction term ˆ g ti = − 1 λ
Revised: And the prox-correction term ˆ g ti = − 1 λ

Original: [Equation]
Revised: [Equation]

Original: u ti
Revised: u ti

Original: [Equation]
Revised: [Equation]

Original: K − u ti
Revised: K − u ti

Original: [Equation]
Revised: [Equation]

Original: (cid:1) , which implies the global update direction in the local training process.
Revised: (cid:1) , which implies the global update direction in the local training process.

Original: And z t updates as:
Revised: And z t updates as:

Original: [Equation]
Revised: [Equation]

Original: Detailed proofs can be referred in the Appendix .
Revised: Detailed proofs can be referred in the Appendix .

Original: After mapping x t to u t , the local update could be considered as a client momentum-like method with a normalized weight parameterized by γ k .
Revised: After mapping x t to u t , the local update could be considered as a client momentum-like method with a normalized weight parameterized by γ k .

Original: Further, after mapping u t to z t , the entire update process will be simplified to a SGD-type method with the quasi-gradients ˜
Revised: Further, after mapping u t to z t , the entire update process will be simplified to a SGD-type method with the quasi-gradients ˜

Original: g .
Revised: g .

Original: z t contains the penalized prox-term in the total local training stage.
Revised: z t contains the penalized prox-term in the total local training stage.

Original: Though a prox-correction term is applied to eliminate the local biases, x t maintains to be beneficial from the update of penalizing the prox-term.
Revised: Though a prox-correction term is applied to eliminate the local biases, x t maintains to be beneficial from the update of penalizing the prox-term.

Original: The prox-correction term plays the role as exponential average of the global offset.
Revised: The prox-correction term plays the role as exponential average of the global offset.

Original: Then we introduce our proof of the convergence rate for the FedSpeed algorithm:
Revised: Then we introduce our proof of the convergence rate for the FedSpeed algorithm:

Original: Theorem 4.4 Under the Assumptions 4.1-4.3, when the perturbation learning rate satisfies ρ ≤√ 6 αL , and the local learning rate satisfies η l ≤ min { 1 32 √ 3 KL , 2 λ } , and the local interval satisfies
Revised: Theorem 4.4 Under the Assumptions 4.1-4.3, when the perturbation learning rate satisfies ρ ≤√ 6 αL , and the local learning rate satisfies η l ≤ min { 1 32 √ 3 KL , 2 λ } , and the local interval satisfies

Original: K ≥ λ/η l , let κ = 12 − 3 α 2 L 2 ρ 2 − 

Revised: K ≥ λ/η l , let κ = 12 − 3 α 2 L 2 ρ 2 −

Original: η 2 l L 2 K is a positive constant with selecting the proper η l and ρ , the auxiliary sequence z t in Equation (5) generated by executing the Algorithm 1 satisfies:
Revised: η 2 l L 2 K is a positive constant with selecting the proper η l and ρ , the auxiliary sequence z t in Equation (5) generated by executing the Algorithm 1 satisfies:

Original: [Equation]
Revised: [Equation]

Original: where F is a non-convex objective function F ∗ is the optimal of F .
Revised: where F is a non-convex objective function F ∗ is the optimal of F .

Original: The term Φ is:
Revised: The term Φ is:

Original: [Equation]
Revised: [Equation]

Original: where α is the perturbation weight.
Revised: where α is the perturbation weight.

Original: More proof details can be referred to the Appendix .
Revised: More proof details can be referred to the Appendix .

Original: Corollary 4.5 Let ρ =
Revised: Corollary 4.5 Let ρ =

Original: O (1 / √ T ) with the upper bound of ρ ≤ 1 / √ 6 αL , and let η l
Revised: O (1 / √ T ) with the upper bound of ρ ≤ 1 / √ 6 αL , and let η l

Original: = O (1 /K ) with the lower bound of η l ≥ λ/K , when the local interval K is long enough with K
Revised: = O (1 /K ) with the lower bound of η l ≥ λ/K , when the local interval K is long enough with K

Original: = O ( T ) , the proposed FedSpeed achieves a fast convergence rate of O (1 /T ) .
Revised: = O ( T ) , the proposed FedSpeed achieves a fast convergence rate of O (1 /T ) .

Original: Remark 4.6 Compared with the other prox-based works, e.g. for (Durmus et al., 2021; Wang et al., 2022;
Revised: Remark 4.6 Compared with the other prox-based works, e.g. for (Durmus et al., 2021; Wang et al., 2022;

Original: Gong et al., 2022), their proofs rely on the harsh assumption that local client must approach an exact stationary point or ϵ -inexact stationary point in the local training per round.
Revised: Gong et al., 2022), their proofs rely on the harsh assumption that local client must approach an exact stationary point or ϵ -inexact stationary point in the local training per round.

Original: It cannot be strictly satisfied in the practical federated learning framework with the current theoretical analysis of the last iteration point on the non-convex case.
Revised: It cannot be strictly satisfied in the practical federated learning framework with the current theoretical analysis of the last iteration point on the non-convex case.

Original: We relax this assumption through enlarging the local interval and prove that federated prox-based methods can also achieve the convergence of O (1 /T ) .
Revised: We relax this assumption through enlarging the local interval and prove that federated prox-based methods can also achieve the convergence of O (1 /T ) .

Original: Remark 4.7 Compared with the other current methods, FedSpeed can improve the convergence rate by increasing the local interval K , which is a good property for the practical federated learning framework.
Revised: Remark 4.7 Compared with the other current methods, FedSpeed can improve the convergence rate by increasing the local interval K , which is a good property for the practical federated learning framework.

Original: For the analysis of FedAvg (Yang et al., 2021), under the same assumptions, it achieves
Revised: For the analysis of FedAvg (Yang et al., 2021), under the same assumptions, it achieves

Original: O (1 / √ SKT + K/T ) which restricts the value of K to not exceed the order of T .
Revised: O (1 / √ SKT + K/T ) which restricts the value of K to not exceed the order of T .

Original: Karimireddy et al.
Revised: Karimireddy et al.

Original: (2020) contribute the convergence as O (1 / √ SKT ) under the constant local interval, and (Reddi et al., 2020) proves the same convergence under the strict coordinated bounded variance assumption for the global full gradient in the FedAdam.
Revised: (2020) contribute the convergence as O (1 / √ SKT ) under the constant local interval, and (Reddi et al., 2020) proves the same convergence under the strict coordinated bounded variance assumption for the global full gradient in the FedAdam.

Original: Our experiments also verify this characteristic in
Revised: Our experiments also verify this characteristic in

Original: Section 5.3.
Revised: Section 5.3.

Original: Most current algorithms are affected by increasing K in the training while FedSpeed shows the good stability under the enlarged local intervals and shrunk communication rounds.
Revised: Most current algorithms are affected by increasing K in the training while FedSpeed shows the good stability under the enlarged local intervals and shrunk communication rounds.

Original: [Figure]
Revised: [Figure]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this part, we firstly introduce our experimental setups.
Revised: In this part, we firstly introduce our experimental setups.

Original: We present the convergence and generalization performance in Section 5.2, and study ablation experiments in Section 5.3.
Revised: We present the convergence and generalization performance in Section 5.2, and study ablation experiments in Section 5.3.

Original: 5. S ETUP
Revised: 5. S ETUP

Original: Dataset and backbones.
Revised: Dataset and backbones.

Original: We test the experiments on CIFAR-10, CIFAR-100 Krizhevsky et al.
Revised: We test the experiments on CIFAR-10, CIFAR-100 Krizhevsky et al.

Original: (2009) and TinyImagenet.
Revised: (2009) and TinyImagenet.

Original: Due to the space limitations we introduce these datasets in the Appendix .
Revised: Due to the space limitations we introduce these datasets in the Appendix .

Original: We follow the Hsu et al.
Revised: We follow the Hsu et al.

Original: (2019) to introduce the heterogeneity via splitting the total dataset by sampling the label ratios from the Dirichlet distribution.
Revised: (2019) to introduce the heterogeneity via splitting the total dataset by sampling the label ratios from the Dirichlet distribution.

Original: We train and test the performance on the standard
Revised: We train and test the performance on the standard

Original: ResNet-18 He et al.
Revised: ResNet-18 He et al.

Original: (2016) backbone with the 7 × 7 filter size in the first convolution layer with
Revised: (2016) backbone with the 7 × 7 filter size in the first convolution layer with

Original: BN-layers replaced by GN Wu & He (2018); Hsieh et al.
Revised: BN-layers replaced by GN Wu & He (2018); Hsieh et al.

Original: (2020) to avoid the invalid aggregation.
Revised: (2020) to avoid the invalid aggregation.

Original: Implementation details.
Revised: Implementation details.

Original: We select each hyper-parameters within the appropriate range and present the combinations under the best performance.
Revised: We select each hyper-parameters within the appropriate range and present the combinations under the best performance.

Original: To fairly compare these baseline methods, we fix the most hyper-parameters for all methods under the same setting.
Revised: To fairly compare these baseline methods, we fix the most hyper-parameters for all methods under the same setting.

Original: For the 10 % participation of totalclients training, we set the local learning rate as 0.1 initially and set the global learning rate as 1.0 for all methods except for FedAdam which applies 0.1 on global server.
Revised: For the 10 % participation of totalclients training, we set the local learning rate as 0.1 initially and set the global learning rate as 1.0 for all methods except for FedAdam which applies 0.1 on global server.

Original: The learning rate decay is set as multiplying 0.998 per communication round except for FedDyn, FedADMM and FedSpeed which apply 0.9995.
Revised: The learning rate decay is set as multiplying 0.998 per communication round except for FedDyn, FedADMM and FedSpeed which apply 0.9995.

Original: Each active local client trains 5 epochs with batchsize 50.
Revised: Each active local client trains 5 epochs with batchsize 50.

Original: Weight decay is set as 1 efor all methods.
Revised: Weight decay is set as 1 efor all methods.

Original: The weight for the prox-term in FedProx, FedDyn, FedADMM and FedSpeed is set as 0.1.
Revised: The weight for the prox-term in FedProx, FedDyn, FedADMM and FedSpeed is set as 0.1.

Original: For the 2 % participation, the learning rate decay is adjusted to 0.9998 for FedDyn and
Revised: For the 2 % participation, the learning rate decay is adjusted to 0.9998 for FedDyn and

Original: FedSpeed.
Revised: FedSpeed.

Original: Each active client trains 2 epochs with batchsize 20.
Revised: Each active client trains 2 epochs with batchsize 20.

Original: The weight for the prox-term is set as 0.001.
Revised: The weight for the prox-term is set as 0.001.

Original: The other hyper-parameters specific to each method will be introduced in the Appendix .
Revised: The other hyper-parameters specific to each method will be introduced in the Appendix .

Original: Baselines.
Revised: Baselines.

Original: We compare several classical and efficient methods with the proposed FedSpeed in our experiments, which focus on the local consistency and client-drifts, including FedAvg McMahan et al.
Revised: We compare several classical and efficient methods with the proposed FedSpeed in our experiments, which focus on the local consistency and client-drifts, including FedAvg McMahan et al.

Original: (2017), FedAdam Reddi et al.
Revised: (2017), FedAdam Reddi et al.

Original: (2020), SCAFFOLD Karimireddy et al.
Revised: (2020), SCAFFOLD Karimireddy et al.

Original: (2020), FedCM Xu et al.
Revised: (2020), FedCM Xu et al.

Original: (2021), FedProx Sahu et al.
Revised: (2021), FedProx Sahu et al.

Original: (2018), FedDyn Durmus et al.
Revised: (2018), FedDyn Durmus et al.

Original: and FedADMM Wang et al.
Revised: and FedADMM Wang et al.

Original: FedAdam applies adaptive optimizer to improve the performance on the global updates.
Revised: FedAdam applies adaptive optimizer to improve the performance on the global updates.

Original: SCAFFOLD and FedCM utilize the global gradient estimation to correct the local updates.
Revised: SCAFFOLD and FedCM utilize the global gradient estimation to correct the local updates.

Original: FedProx introduces the prox-term to alleviate the local inconsistency.
Revised: FedProx introduces the prox-term to alleviate the local inconsistency.

Original: FedDyn and FedADMM both employ the different variants of the primal-dual method to reduce the local inconsistency.
Revised: FedDyn and FedADMM both employ the different variants of the primal-dual method to reduce the local inconsistency.

Original: Due to the limited space, more detailed description and discussions on these compared baselines are placed in the Appendix .
Revised: Due to the limited space, more detailed description and discussions on these compared baselines are placed in the Appendix .

Original: [Table]
Revised: [Table]

Original: 5. E XPERIMENTS
Revised: 5. E XPERIMENTS

Original: CIFAR-10.
Revised: CIFAR-10.

Original: Our proposed FedSpeed is robust to different participation cases.
Revised: Our proposed FedSpeed is robust to different participation cases.

Original: Figure 1 (a) shows the results of 10 % participation of total 100 clients.
Revised: Figure 1 (a) shows the results of 10 % participation of total 100 clients.

Original: For the IID splits, FedSpeed achieves 6.1 % ahead of
Revised: For the IID splits, FedSpeed achieves 6.1 % ahead of

Original: FedAvg as 88.5 % .
Revised: FedAvg as 88.5 % .

Original: FedDyn suffers the instability when learning rate is small, which is the similar phenomenon as mentioned in Xu et al.
Revised: FedDyn suffers the instability when learning rate is small, which is the similar phenomenon as mentioned in Xu et al.

Original: When introducing the heterogeneity, FedAdam suffers from the increasing variance obviously with the accuracy dropping from 85.7 % to 83.2 % .
Revised: When introducing the heterogeneity, FedAdam suffers from the increasing variance obviously with the accuracy dropping from 85.7 % to 83.2 % .

Original: Figure(b) shows the impact from reducing the participation.
Revised: Figure(b) shows the impact from reducing the participation.

Original: FedAdam is lightly affected by this change while the performance degradation of SCAFFOLD is significant which drops from 85.3 % to 80.1 % .
Revised: FedAdam is lightly affected by this change while the performance degradation of SCAFFOLD is significant which drops from 85.3 % to 80.1 % .

Original: CIFAR-100 & TinyImagenet.
Revised: CIFAR-100 & TinyImagenet.

Original: As shown in Figure 1 (c) and (d), the performance of FedSpeed on the CIFAR-100 and TinyImagenet with low participating setting performs robustly and achieves approximately 1.6 % and 1.8 % improvement ahead of the FedCM respectively.
Revised: As shown in Figure 1 (c) and (d), the performance of FedSpeed on the CIFAR-100 and TinyImagenet with low participating setting performs robustly and achieves approximately 1.6 % and 1.8 % improvement ahead of the FedCM respectively.

Original: As the participation is too low, the impact from the heterogeneous data becomes weak gradually with a similar test accuracy.
Revised: As the participation is too low, the impact from the heterogeneous data becomes weak gradually with a similar test accuracy.

Original: SCAFFOLD is still greatly affected by a low participation ratio, which drops about 3.3 % lower than
Revised: SCAFFOLD is still greatly affected by a low participation ratio, which drops about 3.3 % lower than

Original: FedAdam.
Revised: FedAdam.

Original: FedCM converges fast at the beginning of the training stage due to the benefits from strong consistency limitations.
Revised: FedCM converges fast at the beginning of the training stage due to the benefits from strong consistency limitations.

Original: FedSpeed adopts to update the prox-correction term and converges faster with its estimation within several rounds and then FedSpeed outperforms other methods.
Revised: FedSpeed adopts to update the prox-correction term and converges faster with its estimation within several rounds and then FedSpeed outperforms other methods.

Original: Table 1 shows the accuracy under the low participation ratio equals to 2 % .
Revised: Table 1 shows the accuracy under the low participation ratio equals to 2 % .

Original: Our proposed FedSpeed outperforms on each dataset on both IID and non-IID settings.
Revised: Our proposed FedSpeed outperforms on each dataset on both IID and non-IID settings.

Original: Table 1 shows the accuracy under the low participation ratio equals to 2 % .
Revised: Table 1 shows the accuracy under the low participation ratio equals to 2 % .

Original: Our proposed FedSpeed outperforms on each dataset on both
Revised: Our proposed FedSpeed outperforms on each dataset on both

Original: IID and non-IID settings.
Revised: IID and non-IID settings.

Original: We observe the similar results as mentioned in Reddi et al.
Revised: We observe the similar results as mentioned in Reddi et al.

Original: Xu et al.
Revised: Xu et al.

Original: FedAdam and FedCM could maintain the low consistency in the local training stage with a robust results to achieve better performance than others.
Revised: FedAdam and FedCM could maintain the low consistency in the local training stage with a robust results to achieve better performance than others.

Original: While FedDyn is affected greatly by the number of training samples in the dataset, which is sensitive to the partial participation ratios.
Revised: While FedDyn is affected greatly by the number of training samples in the dataset, which is sensitive to the partial participation ratios.

Original: Large local interval for the prox-term.
Revised: Large local interval for the prox-term.

Original: From the IID case to the non-IID case, the heterogeneous dataset introduces the local inconsistency and leads to the severe client-drifts problem.
Revised: From the IID case to the non-IID case, the heterogeneous dataset introduces the local inconsistency and leads to the severe client-drifts problem.

Original: Almost all the baselines suffer from the performance degradation.
Revised: Almost all the baselines suffer from the performance degradation.

Original: High local consistency usually supports for a large interval as for their bounded updates and limited offsets.
Revised: High local consistency usually supports for a large interval as for their bounded updates and limited offsets.

Original: Applying prox-term guarantees the local consistency, but it also has an negative impact on the local training towards the target of weighted local optimal and global server model.
Revised: Applying prox-term guarantees the local consistency, but it also has an negative impact on the local training towards the target of weighted local optimal and global server model.

Original: FedDyn and FedADMM succeed to apply the primal-dual method to alleviate this influence as they change the local objective function whose target is reformed by a dual variable.
Revised: FedDyn and FedADMM succeed to apply the primal-dual method to alleviate this influence as they change the local objective function whose target is reformed by a dual variable.

Original: These method can mitigate the local offsets caused by the prox-term and they improve about 3 % ahead of the FedProx on CIFAR-10.
Revised: These method can mitigate the local offsets caused by the prox-term and they improve about 3 % ahead of the FedProx on CIFAR-10.

Original: However, the primal-dual method requires a local ϵ -close solution.
Revised: However, the primal-dual method requires a local ϵ -close solution.

Original: In the non-convex optimization it is difficult to determine the selection of local training interval K under this requirement.
Revised: In the non-convex optimization it is difficult to determine the selection of local training interval K under this requirement.

Original: Though Durmus et al.
Revised: Though Durmus et al.

Original: (2021) claim that 5 local epochs are approximately enough for the ϵ -close solution, there is still an unpredictable local biases.
Revised: (2021) claim that 5 local epochs are approximately enough for the ϵ -close solution, there is still an unpredictable local biases.

Original: FedSpeed directly applies a prox-correction term to update K epochs and avoids the requirement for the precision of local solution.
Revised: FedSpeed directly applies a prox-correction term to update K epochs and avoids the requirement for the precision of local solution.

Original: This ensures that the local optimization stage does not introduce the bias due to the error of the inexact solution.
Revised: This ensures that the local optimization stage does not introduce the bias due to the error of the inexact solution.

Original: Moreover, the extra ascent step can efficiently improve the performance of local model parameters.
Revised: Moreover, the extra ascent step can efficiently improve the performance of local model parameters.

Original: Thus, the proposed FedSpeed can improve 3 % than
Revised: Thus, the proposed FedSpeed can improve 3 % than

Original: FedDyn and FedADMM and achieve the comparable performance as training on the IID dataset.
Revised: FedDyn and FedADMM and achieve the comparable performance as training on the IID dataset.

Original: [Figure]
Revised: [Figure]

Original: An interesting experimental phenomenon is that the performance of SCAFFOLD gradually degrades under the low participation ratio.
Revised: An interesting experimental phenomenon is that the performance of SCAFFOLD gradually degrades under the low participation ratio.

Original: It should be noticed that under the 10 % participation case, SCAFFOLD performs as well as the FedCM.
Revised: It should be noticed that under the 10 % participation case, SCAFFOLD performs as well as the FedCM.

Original: It benefits from applying a global gradient estimation to correct the local updates, which can weaken the client-drifts by a quasi gradient towards to the global optimal.
Revised: It benefits from applying a global gradient estimation to correct the local updates, which can weaken the client-drifts by a quasi gradient towards to the global optimal.

Original: Actually the estimation variance is related to the participation ratio, which means that their efficiencies rely on the enough number of clients.
Revised: Actually the estimation variance is related to the participation ratio, which means that their efficiencies rely on the enough number of clients.

Original: When the participation ratio decreases to be extremely low, their performance will also be greatly affected by the huge biases in the local training.
Revised: When the participation ratio decreases to be extremely low, their performance will also be greatly affected by the huge biases in the local training.

Original: [Equation]
Revised: [Equation]

Original: Local interval K .
Revised: Local interval K .

Original: To explore the acceleration on T by applying a large interval K , we fix the total training epochs E .
Revised: To explore the acceleration on T by applying a large interval K , we fix the total training epochs E .

Original: It should be noted that K represents for the iteration and E represents for the epoch.
Revised: It should be noted that K represents for the iteration and E represents for the epoch.

Original: A larger local interval can be applied to accelerate the convergence in many previous works theoretically, e.g. for SCAFFOLD and FedAdam, while empirical studies are usually unsatisfactory.
Revised: A larger local interval can be applied to accelerate the convergence in many previous works theoretically, e.g. for SCAFFOLD and FedAdam, while empirical studies are usually unsatisfactory.

Original: As shown in Figure 2, in the FedAdam and FedCM, when K increases from 1 to 20, the accuracy drops about 13.7 % and 10.6 % respectively.
Revised: As shown in Figure 2, in the FedAdam and FedCM, when K increases from 1 to 20, the accuracy drops about 13.7 % and 10.6 % respectively.

Original: SCAFFOLD is affected lightly while its performance is much lower.
Revised: SCAFFOLD is affected lightly while its performance is much lower.

Original: In Figure 2 (d), FedSpeed applies the larger E to accelerate the communication rounds
Revised: In Figure 2 (d), FedSpeed applies the larger E to accelerate the communication rounds

Original: T both on theoretical proofs and empirical results, which stabilizes to swing within 3.8 % lightly.
Revised: T both on theoretical proofs and empirical results, which stabilizes to swing within 3.8 % lightly.

Original: [Table]
Revised: [Table]

Original: Learning rate ρ for gradient perturbation.
Revised: Learning rate ρ for gradient perturbation.

Original: In the simple analysis, ρ can be selected as a proper value which has no impact on the convergence complexity.
Revised: In the simple analysis, ρ can be selected as a proper value which has no impact on the convergence complexity.

Original: By noticing that if α ̸ = 0 , ρ could be selected irrelevant to η l .
Revised: By noticing that if α ̸ = 0 , ρ could be selected irrelevant to η l .

Original: To achieve a better performance, we apply the ascent learning rate ρ = ρ 0 / ∥∇ F i ∥ to in the experiments, where ρ 0 is a constant value selected from the Table 2.
Revised: To achieve a better performance, we apply the ascent learning rate ρ = ρ 0 / ∥∇ F i ∥ to in the experiments, where ρ 0 is a constant value selected from the Table 2.

Original: ρ is consistent with the sharpness aware minimization Foret et al.
Revised: ρ is consistent with the sharpness aware minimization Foret et al.

Original: (2020b) which can search for a flat local minimal.
Revised: (2020b) which can search for a flat local minimal.

Original: Table 2 shows the performance of utilizing the different ρ 0 on CIFAR-10 by 500 communication rounds under the 10 % participation of totalclients setting.
Revised: Table 2 shows the performance of utilizing the different ρ 0 on CIFAR-10 by 500 communication rounds under the 10 % participation of totalclients setting.

Original: Due to the space limitations more details could be referred to the Appendix .
Revised: Due to the space limitations more details could be referred to the Appendix .

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we propose a novel and practical federated method FedSpeed which applies a proxcorrection term to neutralize the bias due to prox-term in each local training stage and utilizes a perturbation gradient weighted by an extra gradient ascent step to improve the local generalization performance.
Revised: In this paper, we propose a novel and practical federated method FedSpeed which applies a proxcorrection term to neutralize the bias due to prox-term in each local training stage and utilizes a perturbation gradient weighted by an extra gradient ascent step to improve the local generalization performance.

Original: We provide the theoretical analysis to guarantee its convergence and prove that
Revised: We provide the theoretical analysis to guarantee its convergence and prove that

Original: FedSpeed benefits from a larger local interval K to achieve a fast convergence rate of O (1 /T ) without any other harsh assumptions.
Revised: FedSpeed benefits from a larger local interval K to achieve a fast convergence rate of O (1 /T ) without any other harsh assumptions.

Original: We also conduct extensive experiments to highlight the significant improvement and efficiency of our proposed FedSpeed, which is consistent with the properties of our analysis.
Revised: We also conduct extensive experiments to highlight the significant improvement and efficiency of our proposed FedSpeed, which is consistent with the properties of our analysis.

Original: This work inspires the FL framework design to focus on the local consistency and local higher generalization performance to implement the high-efficient method to federated learning
Revised: This work inspires the FL framework design to focus on the local consistency and local higher generalization performance to implement the high-efficient method to federated learning

Original: A G RADIENT P ERTURBATION A. U NDERSTANDING OF G RADIENT P ERTURBATION
Revised: A G RADIENT P ERTURBATION A. U NDERSTANDING OF G RADIENT P ERTURBATION

Original: We propose the gradient perturbation in the local training stage instead of the traditional stochastic gradient, which merges an extra gradient ascent step to the vanilla gradient by a hyper-parameter α .
Revised: We propose the gradient perturbation in the local training stage instead of the traditional stochastic gradient, which merges an extra gradient ascent step to the vanilla gradient by a hyper-parameter α .

Original: While its ascent step usually approximates the worst point in the neighbourhood.
Revised: While its ascent step usually approximates the worst point in the neighbourhood.

Original: This has been studied in many previous works, e.g. for the form of extra gradient and the sharpness aware minimization.
Revised: This has been studied in many previous works, e.g. for the form of extra gradient and the sharpness aware minimization.

Original: In our studies, we perform the extra gradient ascent step instead of the descent step in extra gradient method.
Revised: In our studies, we perform the extra gradient ascent step instead of the descent step in extra gradient method.

Original: It also could be considered as a variant of the sharpness aware minimization method via weighted averaging the ascent step gradient and the vanilla gradient, instead of the normalized gradient.
Revised: It also could be considered as a variant of the sharpness aware minimization method via weighted averaging the ascent step gradient and the vanilla gradient, instead of the normalized gradient.

Original: Here we illustrate the implicit of this quasi-gradient ˜ g in our proposed FedSpeed and explain the positive efficiency for the local training from the perspective of objective functions.
Revised: Here we illustrate the implicit of this quasi-gradient ˜ g in our proposed FedSpeed and explain the positive efficiency for the local training from the perspective of objective functions.

Original: Firstly we consider to minimize the non-convex problem L p ( x ) .
Revised: Firstly we consider to minimize the non-convex problem L p ( x ) .

Original: To approach the stationary point of L p , we can simply introduce a penalized gradient term as a extra loss in L p , which is to solve the problem min x {L ( x ) ≜
Revised: To approach the stationary point of L p , we can simply introduce a penalized gradient term as a extra loss in L p , which is to solve the problem min x {L ( x ) ≜

Original: L p ( x ) + β 2 ∥∇L p ( x ) ∥ 2 } .
Revised: L p ( x ) + β 2 ∥∇L p ( x ) ∥ 2 } .

Original: The final optimization target is consistent with the vanilla target, while penalizing gradient term can approach a flatten minimal empirically.
Revised: The final optimization target is consistent with the vanilla target, while penalizing gradient term can approach a flatten minimal empirically.

Original: We compute the gradient form as follows:
Revised: We compute the gradient form as follows:

Original: [Equation]
Revised: [Equation]

Original: The update in Equation (8) contains second-order Hessian information, which involves a huge amount of parameters for calculation.
Revised: The update in Equation (8) contains second-order Hessian information, which involves a huge amount of parameters for calculation.

Original: To further simplify the updates, we consider an approximation for the gradient form.
Revised: To further simplify the updates, we consider an approximation for the gradient form.

Original: We expand the function L p via Taylor expansion as:
Revised: We expand the function L p via Taylor expansion as:

Original: [Equation]
Revised: [Equation]

Original: where R ∆ =
Revised: where R ∆ =

Original: O ( ∥ ∆ ∥ 2 ) is the infinitesimal to ∥ ∆ ∥ 2 , which is directly omitted in our approximation.
Revised: O ( ∥ ∆ ∥ 2 ) is the infinitesimal to ∥ ∆ ∥ 2 , which is directly omitted in our approximation.

Original: Thus we have the gradient form on ∆ as:
Revised: Thus we have the gradient form on ∆ as:

Original: [Equation]
Revised: [Equation]

Original: R ∆ is relevant to ∆ .
Revised: R ∆ is relevant to ∆ .

Original: We set the ∆ =
Revised: We set the ∆ =

Original: ρ ∇L p ( x ) and then we have:
Revised: ρ ∇L p ( x ) and then we have:

Original: [Equation]
Revised: [Equation]

Original: Thus we connect Equation (8) and Equation (9), we have:
Revised: Thus we connect Equation (8) and Equation (9), we have:

Original: [Equation]
Revised: [Equation]

Original: Here we can see that the balance weight α in our proposed method is actually the ratio of the gradient penalized weight β and the gradient ascent step size ρ .
Revised: Here we can see that the balance weight α in our proposed method is actually the ratio of the gradient penalized weight β and the gradient ascent step size ρ .

Original: To fix the step size ρ , increasing α means increasing the gradient penalized weight β , which facilitates searching for a flatten stationary point to improve the generalization performance.
Revised: To fix the step size ρ , increasing α means increasing the gradient penalized weight β , which facilitates searching for a flatten stationary point to improve the generalization performance.

Original: While the second term of ∇L ( x ) can not be directly computed for its nested form, we approximate the second term with the chain rule as follows:
Revised: While the second term of ∇L ( x ) can not be directly computed for its nested form, we approximate the second term with the chain rule as follows:

Original: [Equation]
Revised: [Equation]

Original: Finally we have:
Revised: Finally we have:

Original: [Equation]
Revised: [Equation]

Original: The Equation (10) provides an understanding for the weighted quasi gradient ˜ g on the local training stage in our proposed FedSpeed.
Revised: The Equation (10) provides an understanding for the weighted quasi gradient ˜ g on the local training stage in our proposed FedSpeed.

Original: We select an appropriate 0 ≤ β ≤ ρ to satisfy the update of perturbation gradient.
Revised: We select an appropriate 0 ≤ β ≤ ρ to satisfy the update of perturbation gradient.

Original: It executes a gradient ascent step firstly with the step size ρ to ˘ x .
Revised: It executes a gradient ascent step firstly with the step size ρ to ˘ x .

Original: Then it generates the stochastic gradient by the same sampled mini-batch data as the ascent step at ˘ x .
Revised: Then it generates the stochastic gradient by the same sampled mini-batch data as the ascent step at ˘ x .

Original: The quasi-gradient is merged as Equation (10) to execute the gradient descent step.
Revised: The quasi-gradient is merged as Equation (10) to execute the gradient descent step.

Original: This is just a simple approximation for the gradient perturbation to help for understanding the implicit of the quasi-gradient and its performance in the training stage.
Revised: This is just a simple approximation for the gradient perturbation to help for understanding the implicit of the quasi-gradient and its performance in the training stage.

Original: Actually the error of the approximation depends a lot on ρ .
Revised: Actually the error of the approximation depends a lot on ρ .

Original: The smaller ρ , the higher the accuracy of this estimation, but the smaller ρ , the less efficient the optimizer performs.
Revised: The smaller ρ , the higher the accuracy of this estimation, but the smaller ρ , the less efficient the optimizer performs.

Original: Similar understanding can be referred in the (Qu et al., 2022;
Revised: Similar understanding can be referred in the (Qu et al., 2022;

Original: Caldarola et al., 2022; Andriushchenko & Flammarion, 2022).
Revised: Caldarola et al., 2022; Andriushchenko & Flammarion, 2022).

Original: B E XPERIMENTS B. S
Revised: B E XPERIMENTS B. S

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Dataset and Backbones.
Revised: Dataset and Backbones.

Original: Extensive experiments are tested on CIFAR-10/100 dataset.
Revised: Extensive experiments are tested on CIFAR-10/100 dataset.

Original: We test on the two different settings as 10 % participation of total 100 clients and 2 % participation of totalclients.
Revised: We test on the two different settings as 10 % participation of total 100 clients and 2 % participation of totalclients.

Original: CIFAR-10 dataset contains 50,000 training data and 10,000 test data in 10 classes.
Revised: CIFAR-10 dataset contains 50,000 training data and 10,000 test data in 10 classes.

Original: Each data sample is a 3 × 32 × 32 color image.
Revised: Each data sample is a 3 × 32 × 32 color image.

Original: CIFAR-100 Krizhevsky et al.
Revised: CIFAR-100 Krizhevsky et al.

Original: (2009) includes 50,000 training data and 10,000 test data in 100 classes as 500 training samples per class.
Revised: (2009) includes 50,000 training data and 10,000 test data in 100 classes as 500 training samples per class.

Original: TinyImagenet involves 100,000 training images and 10,000 test images in 200 classes for 3 × 64 × 64 color images, as shown in Table 3.
Revised: TinyImagenet involves 100,000 training images and 10,000 test images in 200 classes for 3 × 64 × 64 color images, as shown in Table 3.

Original: To fairly compare with the other baselines, we train and test the performance on the standard ResNet-

Revised: To fairly compare with the other baselines, we train and test the performance on the standard ResNet-

Original: He et al.
Revised: He et al.

Original: (2016) backbone with the 7 × 7 filter size in the first convolution layer as implemented in the previous works, e.g. for Karimireddy et al.
Revised: (2016) backbone with the 7 × 7 filter size in the first convolution layer as implemented in the previous works, e.g. for Karimireddy et al.

Original: ; Durmus et al.
Revised: ; Durmus et al.

Original: ; Xu et al.
Revised: ; Xu et al.

Original: We follow the Hsieh et al.
Revised: We follow the Hsieh et al.

Original: (2020) to replace the batch normalization layer with group normalization layer Wu & He (2018), which can be aggregated directly by averaging.
Revised: (2020) to replace the batch normalization layer with group normalization layer Wu & He (2018), which can be aggregated directly by averaging.

Original: These are all common setups in many previous works.
Revised: These are all common setups in many previous works.

Original: Dataset Partitions.
Revised: Dataset Partitions.

Original: To fairly compare with the other baselines, we follow the Hsu et al.
Revised: To fairly compare with the other baselines, we follow the Hsu et al.

Original: (2019) to introduce the heterogeneity via splitting the total dataset by sampling the label ratios from the
Revised: (2019) to introduce the heterogeneity via splitting the total dataset by sampling the label ratios from the

Original: Dirichlet distribution.
Revised: Dirichlet distribution.

Original: An additional parameter is used to control the level of the heterogeneity of the entire data partition.
Revised: An additional parameter is used to control the level of the heterogeneity of the entire data partition.

Original: In order to visualize the distribution of heterogeneous data, we make the heat maps of the label distribution in different dataset, as shown in Figure 3.
Revised: In order to visualize the distribution of heterogeneous data, we make the heat maps of the label distribution in different dataset, as shown in Figure 3.

Original: Since the heat map ofclients cannot be displayed normally, we show 100 clients case.
Revised: Since the heat map ofclients cannot be displayed normally, we show 100 clients case.

Original: It could be seen that for heterogeneity weight equals to 0.6, about 10 % to 20 % of the categories dominate on each client, which is white block in the Figure 3.
Revised: It could be seen that for heterogeneity weight equals to 0.6, about 10 % to 20 % of the categories dominate on each client, which is white block in the Figure 3.

Original: The IID dataset is totally averaged in each client.
Revised: The IID dataset is totally averaged in each client.

Original: Data Argumentation.
Revised: Data Argumentation.

Original: For CIFAR-10/100, we follow the implementation in the Karimireddy et al.
Revised: For CIFAR-10/100, we follow the implementation in the Karimireddy et al.

Original: (2020); Durmus et al.
Revised: (2020); Durmus et al.

Original: (2021) to normalize the pixel value within a specific mean and std value in our code, which are [0.491, 0.482, 0.447] for mean, [0.247, 0.243, 0.262] for std and [0.5071, 0.4867, 0.4408] for mean, [0.2675, 0.2565, 0.2761] for std.
Revised: (2021) to normalize the pixel value within a specific mean and std value in our code, which are [0.491, 0.482, 0.447] for mean, [0.247, 0.243, 0.262] for std and [0.5071, 0.4867, 0.4408] for mean, [0.2675, 0.2565, 0.2761] for std.

Original: We randomly flip the training samples and randomly crop the images enlarged with the padding equal to 4.
Revised: We randomly flip the training samples and randomly crop the images enlarged with the padding equal to 4.

Original: For TinyImagenet, the same argumentation is applied except for the padding equal to 8.
Revised: For TinyImagenet, the same argumentation is applied except for the padding equal to 8.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Baselines.
Revised: Baselines.

Original: FedAvg McMahan et al.
Revised: FedAvg McMahan et al.

Original: (2017) is proposed as the basic framework in the federated learning.
Revised: (2017) is proposed as the basic framework in the federated learning.

Original: And FedOpt improves it as a two-stage optimizer with a local and global optimizer update alternatively.
Revised: And FedOpt improves it as a two-stage optimizer with a local and global optimizer update alternatively.

Original: Yang et al.
Revised: Yang et al.

Original: (2021) proves a specific η g (not the average weight) can achieve faster convergence (non-dominant term).
Revised: (2021) proves a specific η g (not the average weight) can achieve faster convergence (non-dominant term).

Original: FedAdam Reddi et al.
Revised: FedAdam Reddi et al.

Original: (2020) utilizes a adaptive optimizer on the global server and SGD optimizer on the clients, which average the averaged local gradients as a quasi-gradient for global server to implement the adaptive update.
Revised: (2020) utilizes a adaptive optimizer on the global server and SGD optimizer on the clients, which average the averaged local gradients as a quasi-gradient for global server to implement the adaptive update.

Original: SCAFFOLD Karimireddy et al.
Revised: SCAFFOLD Karimireddy et al.

Original: (2020) applies the variance reduction technique , i.e. SVRG, to approximate the global gradient as the averaged local gradients and transfer an extra variable to the client per round.
Revised: (2020) applies the variance reduction technique , i.e. SVRG, to approximate the global gradient as the averaged local gradients and transfer an extra variable to the client per round.

Original: This implementation can accelerate the convergence rate of the non-dominant term theoretically and achieve a high performance empirically.
Revised: This implementation can accelerate the convergence rate of the non-dominant term theoretically and achieve a high performance empirically.

Original: FedCM Xu et al.
Revised: FedCM Xu et al.

Original: (2021) proposes a client-level momentum to merge the global update as a momentum buffer to the local updates, which extremely reduces the local consistency.
Revised: (2021) proposes a client-level momentum to merge the global update as a momentum buffer to the local updates, which extremely reduces the local consistency.

Original: Though it introduces a unpredictable biases into the local updates, it achieves the SOTA performance ahead of other methods.
Revised: Though it introduces a unpredictable biases into the local updates, it achieves the SOTA performance ahead of other methods.

Original: FedProx Sahu et al.
Revised: FedProx Sahu et al.

Original: (2018) implements the prox-point optimizer into the FL framework on local updates with a regularization prox-term regularizer.
Revised: (2018) implements the prox-point optimizer into the FL framework on local updates with a regularization prox-term regularizer.

Original: It limits the local updates towards the initial point at the start of each local stage.
Revised: It limits the local updates towards the initial point at the start of each local stage.

Original: Many previous works have analyzed its advantages and weaknesses.
Revised: Many previous works have analyzed its advantages and weaknesses.

Original: Durmus et al.
Revised: Durmus et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: ; Gong et al.
Revised: ; Gong et al.

Original: (2022) use different variants of primal-dual method into FL and achieve nice satisfactory in the
Revised: (2022) use different variants of primal-dual method into FL and achieve nice satisfactory in the

Original: FL framework.
Revised: FL framework.

Original: It does not need a heterogeneity bounded assumption theoretically, which requires a high local convergence guarantees.
Revised: It does not need a heterogeneity bounded assumption theoretically, which requires a high local convergence guarantees.

Original: Our proposed FedSpeed achieve the same convergence rate without assuming the local exact solution and we provide the local interval bound to achieve this faster convergence.
Revised: Our proposed FedSpeed achieve the same convergence rate without assuming the local exact solution and we provide the local interval bound to achieve this faster convergence.

Original: Both theoretical analysis and empirical results verifies the performance of our proposed FedSpeed.
Revised: Both theoretical analysis and empirical results verifies the performance of our proposed FedSpeed.

Original: B. E XPERIMENTS B. H YER - PARAMETERS
Revised: B. E XPERIMENTS B. H YER - PARAMETERS

Original: Hyper-parameters Selections.
Revised: Hyper-parameters Selections.

Original: We fix the local learning rate as 0.1 and global learning rate as 1.0 for average, except for the FedAdam which is applied 0.1.
Revised: We fix the local learning rate as 0.1 and global learning rate as 1.0 for average, except for the FedAdam which is applied 0.1.

Original: The penalized weight of prox-term in FedProx, FedDyn, FedADMM and FedSpeed is selected from the [0.001, 0.01, 0.1, 0.5].
Revised: The penalized weight of prox-term in FedProx, FedDyn, FedADMM and FedSpeed is selected from the [0.001, 0.01, 0.1, 0.5].

Original: The learning rate decay is fixed as 0.998 expect for the FedDyn, FedADMM and FedSpeed is selected from [0.998, 0.999, 0.9995, 0.99995].
Revised: The learning rate decay is fixed as 0.998 expect for the FedDyn, FedADMM and FedSpeed is selected from [0.998, 0.999, 0.9995, 0.99995].

Original: The perturbation weight is selected from [0, 0.5, 0.75, 0.875, 0.9375, 1].
Revised: The perturbation weight is selected from [0, 0.5, 0.75, 0.875, 0.9375, 1].

Original: The batchsize is selected from [20, 50].
Revised: The batchsize is selected from [20, 50].

Original: The local interval K is selected from [1, 2, 5, 10, 20].
Revised: The local interval K is selected from [1, 2, 5, 10, 20].

Original: For the specific parameters in FedAdam, the momentum weight is set as 0.1 and the second order momentum weight is set as 0.01.
Revised: For the specific parameters in FedAdam, the momentum weight is set as 0.1 and the second order momentum weight is set as 0.01.

Original: The minimal value is set as 0.001 to prevent the calculation of dividing by 0.
Revised: The minimal value is set as 0.001 to prevent the calculation of dividing by 0.

Original: The client-level momentum weight of FedCM is set as 0.1.
Revised: The client-level momentum weight of FedCM is set as 0.1.

Original: [Table]
Revised: [Table]

Original: Here we briefly introduce the selection of the hyperparameters in FedSpeed.
Revised: Here we briefly introduce the selection of the hyperparameters in FedSpeed.

Original: (1) η l is the learning rate which is a basic hyperparameters in
Revised: (1) η l is the learning rate which is a basic hyperparameters in

Original: the deep
Revised: the deep

Original: learning
Revised: learning

Original: , and usually we do
Revised: , and usually we do

Original: not finetune this for the fair comparison in the experiments.
Revised: not finetune this for the fair comparison in the experiments.

Original: We
Revised: We

Original: just select
Revised: just select

Original: the same and common settings as the previous works mentioned.
Revised: the same and common settings as the previous works mentioned.

Original: (2) λ is the coefficient for the prox-term, which is proposed in the FedProx and a lot of prox-based federated methods adopt this hyperparameter widely both in personalized-FL and centralized-FL.
Revised: (2) λ is the coefficient for the prox-term, which is proposed in the FedProx and a lot of prox-based federated methods adopt this hyperparameter widely both in personalized-FL and centralized-FL.

Original: The selection of this hyperparameter has been studied in many previous works which verify its efficiency.
Revised: The selection of this hyperparameter has been studied in many previous works which verify its efficiency.

Original: Usually the selection of λ are in { 10 , 100 } on the CIFAR-10/100 dataset, and we test it also works
Revised: Usually the selection of λ are in { 10 , 100 } on the CIFAR-10/100 dataset, and we test it also works

Original: on the TinyImagenet.
Revised: on the TinyImagenet.

Original: (3) ρ is the ascent step learning rate.
Revised: (3) ρ is the ascent step learning rate.

Original: Like
Revised: Like

Original: many extra gradient method
Revised: many extra gradient method

Original: , the selection of ρ is
Revised: , the selection of ρ is

Original: usually related to the local
Revised: usually related to the local

Original: learning rate η l .
Revised: learning rate η l .

Original: In order not to unduly affect the performance of the gradient descent, the learning rate for the extra gradient step
Revised: In order not to unduly affect the performance of the gradient descent, the learning rate for the extra gradient step

Original: ρ
Revised: ρ

Original: is usually
Revised: is usually

Original: set not
Revised: set not

Original: much
Revised: much

Original: larger
Revised: larger

Original: than the
Revised: than the

Original: learning
Revised: learning

Original: rate
Revised: rate

Original: for
Revised: for

Original: the gradient descent step
Revised: the gradient descent step

Original: η l .
Revised: η l .

Original: Obviously, if ρ is set very small, the updated state of the
Revised: Obviously, if ρ is set very small, the updated state of the

Original: extra
Revised: extra

Original: gradient steps will be very limited, which makes this operation have no effect.
Revised: gradient steps will be very limited, which makes this operation have no effect.

Original: Therefore, the selection of ρ usually matches that of η l .
Revised: Therefore, the selection of ρ usually matches that of η l .

Original: In our experiments, the η l is set as 0 . 1 , which is a common selection in the previous works.
Revised: In our experiments, the η l is set as 0 . 1 , which is a common selection in the previous works.

Original: We test the selection of ρ in { 0 , 0 . 01 , 0 . 05 , 0 . 1 , 0 . 2 } which represents for { ”no extra gradient”, ” 0 .
Revised: We test the selection of ρ in { 0 , 0 . 01 , 0 . 05 , 0 . 1 , 0 . 2 } which represents for { ”no extra gradient”, ” 0 .

Original: 1 η l ”, ” 0 .
Revised: 1 η l ”, ” 0 .

Original: 5 η l , ” 1 η l ”, ” 2 η l ” } .
Revised: 5 η l , ” 1 η l ”, ” 2 η l ” } .

Original: The best performing selection is ρ = 1 η l in
Revised: The best performing selection is ρ = 1 η l in

Original: CIFAR-10 (details in Section 5.3 paragraph ”Learning rate ρ for the gradient perturbation”).
Revised: CIFAR-10 (details in Section 5.3 paragraph ”Learning rate ρ for the gradient perturbation”).

Original: We also test this selection on the CIFAR-100 and TinyImagenet, and it also works well.
Revised: We also test this selection on the CIFAR-100 and TinyImagenet, and it also works well.

Original: We recommend that the selection of ρ should be kept comparable to the learning rate η l .
Revised: We recommend that the selection of ρ should be kept comparable to the learning rate η l .

Original: (4) α is the ratio for merging the gradient of the extra ascent step.
Revised: (4) α is the ratio for merging the gradient of the extra ascent step.

Original: In FedSpeed, the α is in the range of [0 , 1] .
Revised: In FedSpeed, the α is in the range of [0 , 1] .

Original: The same, if α is set very small, which means it does not merge the gradient of the ascent steps.
Revised: The same, if α is set very small, which means it does not merge the gradient of the ascent steps.

Original: In our experiments, we test the selection of α in { 0 , 0 . 5 , 0 .
Revised: In our experiments, we test the selection of α in { 0 , 0 . 5 , 0 .

Original: The best performing selection is α = 0 .
Revised: The best performing selection is α = 0 .

Original: in CIFAR-10.
Revised: in CIFAR-10.

Original: In fact α = 1 also works well (details in Section 5.3 paragraph ”Perturbation weight α ”).
Revised: In fact α = 1 also works well (details in Section 5.3 paragraph ”Perturbation weight α ”).

Original: Thus, about α , we recommend that it should be close to 1 . 0 , e.g. for 0 . 9 , 0 . 99 , 1 . 0 .
Revised: Thus, about α , we recommend that it should be close to 1 . 0 , e.g. for 0 . 9 , 0 . 99 , 1 . 0 .

Original: This also verifies the improvements of the ascent steps.
Revised: This also verifies the improvements of the ascent steps.

Original: B.3. B EST P ERFORMING H YPER - PARAMETERS .
Revised: B.3. B EST P ERFORMING H YPER - PARAMETERS .

Original: For fair comparison, the learning rate is fixed for all the methods.
Revised: For fair comparison, the learning rate is fixed for all the methods.

Original: For CIFAR-10 dataset, we select the batchsize as 50 for 100 clients and 20 for 500 clients.
Revised: For CIFAR-10 dataset, we select the batchsize as 50 for 100 clients and 20 for 500 clients.

Original: The total dataset is 50,000 and there are 100 images under a single client if it is set as 500 clients.
Revised: The total dataset is 50,000 and there are 100 images under a single client if it is set as 500 clients.

Original: Thus we decay it to 20 for 5 iterations per local epoch.
Revised: Thus we decay it to 20 for 5 iterations per local epoch.

Original: The local epochs is set as 5, the same as the experiments of Karimireddy et al.
Revised: The local epochs is set as 5, the same as the experiments of Karimireddy et al.

Original: ; Durmus et al.
Revised: ; Durmus et al.

Original: ; Xu et al.
Revised: ; Xu et al.

Original: (2021) etc.
Revised: (2021) etc.

Original: and their performance is matching.
Revised: and their performance is matching.

Original: We select the local interval K as 5.
Revised: We select the local interval K as 5.

Original: The prox-term weight is selected as 0.1.
Revised: The prox-term weight is selected as 0.1.

Original: The learning rate decay is selected as 0.9995 for prox-term based methods.
Revised: The learning rate decay is selected as 0.9995 for prox-term based methods.

Original: We train the total dataset for 1, communication rounds.
Revised: We train the total dataset for 1, communication rounds.

Original: For CIFAR-100 dataset, we select the 500 clients with 2 % participation ratio in the experiments.
Revised: For CIFAR-100 dataset, we select the 500 clients with 2 % participation ratio in the experiments.

Original: Thus for each hyper-parameters we fine-tune a little.
Revised: Thus for each hyper-parameters we fine-tune a little.

Original: The batchsize is selected as 20 to avoid too little iterations per local epoch.
Revised: The batchsize is selected as 20 to avoid too little iterations per local epoch.

Original: The local epochs is set as 2 for the final results comparison.
Revised: The local epochs is set as 2 for the final results comparison.

Original: The ablation study on local interval K indicates that our proposed FedSpeed outperforms significantly than other methods when K is large.
Revised: The ablation study on local interval K indicates that our proposed FedSpeed outperforms significantly than other methods when K is large.

Original: Thus to compare the performance more clearly, we select the 2 as the local epochs.
Revised: Thus to compare the performance more clearly, we select the 2 as the local epochs.

Original: We decay the prox-term weight as 0.01 for prox-term based methods.
Revised: We decay the prox-term weight as 0.01 for prox-term based methods.

Original: The learning rate decay is selected as 0.99995 for prox-based methods.
Revised: The learning rate decay is selected as 0.99995 for prox-based methods.

Original: We train 1,500 rounds and then test the performance.
Revised: We train 1,500 rounds and then test the performance.

Original: For TinyImagenet dataset, the most selections are the same as for the CIFAR-100 dataset.
Revised: For TinyImagenet dataset, the most selections are the same as for the CIFAR-100 dataset.

Original: The prox-term weight is selected as 0.1 and the learning rate decay is selected as 0.9995.
Revised: The prox-term weight is selected as 0.1 and the learning rate decay is selected as 0.9995.

Original: Total 3, communication rounds are implemented in the training stage.
Revised: Total 3, communication rounds are implemented in the training stage.

Original: B.3. S PEED COMPARISON .
Revised: B.3. S PEED COMPARISON .

Original: Table 4 shows the communication rounds required to achieve the target test accuracy.
Revised: Table 4 shows the communication rounds required to achieve the target test accuracy.

Original: At the beginning of training, FedCM performs faster than others and usually achieve a high accuracy finally.
Revised: At the beginning of training, FedCM performs faster than others and usually achieve a high accuracy finally.

Original: FedSpeed is faster in the middle and late stages of training.
Revised: FedSpeed is faster in the middle and late stages of training.

Original: We bold the data for the top-2 in each test and generally FedCM and FedSpeed significantly performs well on the training speed.
Revised: We bold the data for the top-2 in each test and generally FedCM and FedSpeed significantly performs well on the training speed.

Original: Figure 4 shows the performance of different learning rate decay and prox-term weight for FedSpeed.
Revised: Figure 4 shows the performance of different learning rate decay and prox-term weight for FedSpeed.

Original: B.3.
Revised: B.3.

Original: T IME COST
Revised: T IME COST

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: We test the time on the A100-SXM4-40GB GPU and show the performance in the Table B.3.3.
Revised: We test the time on the A100-SXM4-40GB GPU and show the performance in the Table B.3.3.

Original: Experimental setups are the same as the CIFAR-10 10 % participation among total 100 clients on the
Revised: Experimental setups are the same as the CIFAR-10 10 % participation among total 100 clients on the

Original: DIR-0.6 dataset.
Revised: DIR-0.6 dataset.

Original: The rounds in the table are the communication rounds required that the test accuracy achieves accuracy 85% . ”
Revised: The rounds in the table are the communication rounds required that the test accuracy achieves accuracy 85% . ”

Original: -” means it can not achieve the target accuracy.
Revised: -” means it can not achieve the target accuracy.

Original: FedSpeed is slower due to the requirement of computing an extra gradient.
Revised: FedSpeed is slower due to the requirement of computing an extra gradient.

Original: So it gets slower in one single update, approximately 1.57 × wall-clock time costs than FedAvg.
Revised: So it gets slower in one single update, approximately 1.57 × wall-clock time costs than FedAvg.

Original: But its convergence process is very fast.
Revised: But its convergence process is very fast.

Original: For the final convergence speed, FedSpeed still has a considerable advantage over other algorithms.
Revised: For the final convergence speed, FedSpeed still has a considerable advantage over other algorithms.

Original: The issue is possibly one of the improvements for FedSpeed in the future.
Revised: The issue is possibly one of the improvements for FedSpeed in the future.

Original: For example, introduces a single-call gradient method to save half the costs during backpropagation.
Revised: For example, introduces a single-call gradient method to save half the costs during backpropagation.

Original: We are also currently trying to introduce new module to save the cost.
Revised: We are also currently trying to introduce new module to save the cost.

Original: B.3. D IFFERENT HETEROGENEITY .
Revised: B.3. D IFFERENT HETEROGENEITY .

Original: [Table]
Revised: [Table]

Original: We test on the Dir-0.3 setups on CIFAR-10 and show the results as Table B.3.4, the other settings are the same as the test in the text.
Revised: We test on the Dir-0.3 setups on CIFAR-10 and show the results as Table B.3.4, the other settings are the same as the test in the text.

Original: The (i.i.d. > Dir-0.6) is the difference between the IID dataset and the
Revised: The (i.i.d. > Dir-0.6) is the difference between the IID dataset and the

Original: Dir-0.6 dataset and (Dir-0.6 > Dir-0.3) is the difference between the Dir-0.6 dataset and the DIR-0.
Revised: Dir-0.6 dataset and (Dir-0.6 > Dir-0.3) is the difference between the Dir-0.6 dataset and the DIR-0.

Original: dataset.
Revised: dataset.

Original: FedSpeed can outperform the others on the Dir-0.3 setups whose heterogeneity is much
Revised: FedSpeed can outperform the others on the Dir-0.3 setups whose heterogeneity is much

Original: [Figure]
Revised: [Figure]

Revised: can benefit

Original: B.3. A BLATION STUDIES
Revised: B.3. A BLATION STUDIES

Original: [Table]
Revised: [Table]

Original: From the practical training point of view, compared with the vanilla FedAvg, FedSpeed adds three main modules: (1) prox-term, (2) prox-correction term, and (3) gradient perturbation.
Revised: training point of view, compared with the vanilla FedAvg, FedSpeed adds three main modules: (1) prox-term, (2) prox-correction term, and (3) gradient perturbation.

Original: We test the performance of 500 communication rounds of the different combination of the modules above on the
Revised: We test the performance of 500 communication rounds of the different combination of the modules above on the

Original: CIFAR-10 with the settings of 10 % participating ratio of total 100 clients.
Revised: CIFAR-10 with the settings of 10 % participating ratio of total 100 clients.

Original: The TableB.3.5 shows their performance.
Revised: The TableB.3.5 shows their performance.

Original: From the table above, we can clearly see the performance of different modules.
Revised: From the table above, we can clearly see the performance of different modules.

Original: The prox-term is proposed by the FedProx.
Revised: The prox-term is proposed by the FedProx.

Original: But due to some issues we point out in our paper, this term has also a negative impact on the performance in FL.
Revised: But due to some issues we point out in our paper, this term has also a negative impact on the performance in FL.

Original: When the prox-correction term is introduced in, it improves the performance from 82.24 % to 83.94 % .
Revised: When the prox-correction term is introduced in, it improves the performance from 82.24 % to 83.94 % .

Original: When the gradient perturbation is introduced in, it improves the performance from 82.24 % to 83.88 % .
Revised: When the gradient perturbation is introduced in, it improves the performance from 82.24 % to 83.88 % .

Original: While FedSpeed applies them together and achieves a 3.46 % improvement.
Revised: While FedSpeed applies them together and achieves a 3.46 % improvement.

Original: Different performance of these modules:
Revised: Different performance of these modules:

Original: As introduced in our paper, the prox-term simply performs as a balance between the local and global solutions, and there still exists the non-vanishing inconsistent biases among the local solutions, i.e., the local solutions are still largely deviated from each other, implying that local inconsistency is still not eliminated.
Revised: As introduced in our paper, the prox-term simply performs as a balance between the local and global solutions, and there still exists the non-vanishing inconsistent biases among the local solutions, i.e., the local solutions are still largely deviated from each other, implying that local inconsistency is still not eliminated.

Original: Thus we utilize the prox-correction term to correct the inconsistent biases during the local training.
Revised: Thus we utilize the prox-correction term to correct the inconsistent biases during the local training.

Original: About the function of gradient perturbation, we refer to a theoretical explanation in the main text, and its proof is provided in the supplementary material due to the space limitations.
Revised: About the function of gradient perturbation, we refer to a theoretical explanation in the main text, and its proof is provided in the supplementary material due to the space limitations.

Original: This perturbation is similar to utilize a penalized gradient term to the objective function during local optimization process.
Revised: This perturbation is similar to utilize a penalized gradient term to the objective function during local optimization process.

Original: The additional penalty will bring better properties to the local state, e.g. for flattened minimal and smoothness.
Revised: The additional penalty will bring better properties to the local state, e.g. for flattened minimal and smoothness.

Original: For federated learning, the smoother the local minima is, the more flatness the model merged on the server will be.
Revised: For federated learning, the smoother the local minima is, the more flatness the model merged on the server will be.

Original: FedSpeed benefits from these two modules to improve the performance and achieves the SOTA results.
Revised: FedSpeed benefits from these two modules to improve the performance and achieves the SOTA results.

Original: B.3. L OSS CURVE COMPARISON
Revised: B.3. L OSS CURVE COMPARISON

Original: According to the Figure 5, in (a) and (b) we can see the common FedAvg method fail to resist the local over-fitting and finally does not approach a stable state in the training, while FedSpeed can converge stably and efficiently, which if far faster than other baselines.
Revised: According to the Figure 5, in (a) and (b) we can see the common FedAvg method fail to resist the local over-fitting and finally does not approach a stable state in the training, while FedSpeed can converge stably and efficiently, which if far faster than other baselines.

Original: (c) and (d) show the empirical studies of increasing the local interval.
Revised: (c) and (d) show the empirical studies of increasing the local interval.

Original: FedCM is the second SOTA-performing method in our baselines, while it still can not speed up by increasing local interval in the practical training.
Revised: FedCM is the second SOTA-performing method in our baselines, while it still can not speed up by increasing local interval in the practical training.

Original: As shown in (c), increasing local interval almost does not have any benefits to FedCM, and the communication rounds can not be reduced.
Revised: As shown in (c), increasing local interval almost does not have any benefits to FedCM, and the communication rounds can not be reduced.

Original: While in (d), FedSpeed succeed to apply a larger local interval to reduce the communication rounds.
Revised: While in (d), FedSpeed succeed to apply a larger local interval to reduce the communication rounds.

Original: When K is increasing, to achieve the similar performance, the total communication rounds is nearly decreasing as K × , which is a useful property that is very efficient in practical training.
Revised: When K is increasing, to achieve the similar performance, the total communication rounds is nearly decreasing as K × , which is a useful property that is very efficient in practical training.

Original: In our paper, what we claim is that if a federated learning method could adopt a large K , it is a good property.
Revised: In our paper, what we claim is that if a federated learning method could adopt a large K , it is a good property.

Original: Unfortunately, most SGD-type algorithms cannot increase the convergence rate by increasing K .
Revised: Unfortunately, most SGD-type algorithms cannot increase the convergence rate by increasing K .

Original: Some useful techniques are adopt in the FL framework to improve the performance, e.g. variance reduction, gradient tracking and regularization term (mainly the prox-based methods).
Revised: Some useful techniques are adopt in the FL framework to improve the performance, e.g. variance reduction, gradient tracking and regularization term (mainly the prox-based methods).

Original: FedSpeed is a prox-based method which incorporates the correction term and extra ascent gradient to improve the performance.
Revised: FedSpeed is a prox-based method which incorporates the correction term and extra ascent gradient to improve the performance.

Original: In fact, it has been proven that prox-based methods have the potential to apply the larger local interval in the local training under the requirement of local minimal solution per communication round.
Revised: In fact, it has been proven that prox-based methods have the potential to apply the larger local interval in the local training under the requirement of local minimal solution per communication round.

Original: We theoretically prove that FedSpeed can achieve the fast rate without this harsh assumption and it can apply the large K in the local client.
Revised: We theoretically prove that FedSpeed can achieve the fast rate without this harsh assumption and it can apply the large K in the local client.

Original: Yang et al.
Revised: Yang et al.

Original: (2021) have proven that if FedAvg change the partial participation to the full participation (Local-SGD-type), the dominant term of convergence rate will change from O ( √ K √ nT ) to O ( 1 √ mKT ) , which will be relaxed to K times faster.
Revised: (2021) have proven that if FedAvg change the partial participation to the full participation (Local-SGD-type), the dominant term of convergence rate will change from O ( √ K √ nT ) to O ( 1 √ mKT ) , which will be relaxed to K times faster.

Original: Full participation usually achieves higher theoretical rate than the partial participation.
Revised: Full participation usually achieves higher theoretical rate than the partial participation.

Original: VRL-SGD Liang et al.
Revised: VRL-SGD Liang et al.

Original: (2019) can theoretically improve the efficiency by adopting a larger order of local interval K = O ( √ T ) than FedAvg, while FedSpeed can adopt K = O ( T ) .
Revised: (2019) can theoretically improve the efficiency by adopting a larger order of local interval K = O ( √ T ) than FedAvg, while FedSpeed can adopt K = O ( T ) .

Original: B.3. A BLATION S TUDY OF P ERTURBATION WEIGHT α
Revised: B.3. A BLATION S TUDY OF P ERTURBATION WEIGHT α

Original: [Table]
Revised: [Table]

Original: Perturbation weight α .
Revised: Perturbation weight α .

Original: α determines the degree of influence of the perturbation gradient term to the vanilla stochastic gradient on the local training stage.
Revised: α determines the degree of influence of the perturbation gradient term to the vanilla stochastic gradient on the local training stage.

Original: It is a trade-off to balance the ratio of the perturbation term.
Revised: It is a trade-off to balance the ratio of the perturbation term.

Original: We select the α from 0 to 1 and find FedSpeed can converge with any α ∈ [0 , 1] .
Revised: We select the α from 0 to 1 and find FedSpeed can converge with any α ∈ [0 , 1] .

Original: Though the theoretical analysis demonstrates that by applying a α > 0 in the term Φ will not increasing the extra orders.
Revised: Though the theoretical analysis demonstrates that by applying a α > 0 in the term Φ will not increasing the extra orders.

Original: And the experimental results shown in Table 8, indicates that the generalization performance improves by increasing α .
Revised: And the experimental results shown in Table 8, indicates that the generalization performance improves by increasing α .

Original: C P ROOFS FOR A NALYSIS
Revised: C P ROOFS FOR A NALYSIS

Original: In this part we will demonstrate the proofs of all formula mentioned in this paper.
Revised: In this part we will demonstrate the proofs of all formula mentioned in this paper.

Original: Each formula is presented in the form of a lemma.
Revised: Each formula is presented in the form of a lemma.

Original: C. P ROOF OF E QUATION (2)
Revised: C. P ROOF OF E QUATION (2)

Original: Equation (2) shows the update in the total local training stage.
Revised: Equation (2) shows the update in the total local training stage.

Original: Lemma C.

Revised: Lemma C.

Original: For ∀ x ti,k ∈ R d and i ∈ S t , we denote δ ti,k
Revised: For ∀ x ti,k ∈ R d and i ∈ S t , we denote δ ti,k

Original: = x ti,k − x ti,k − 1 with setting δ ti, 0 = 0 , and ∆ t i,K = (cid:80) K k =

Revised: = x ti,k − x ti,k − 1 with setting δ ti, 0 = 0 , and ∆ t i,K = (cid:80) K k =

Original: δ t i,k
Revised: δ t i,k

Original: = x t i,K − x t i, 0 , under the update rule in Algorithm Algorithm 1, we have:
Revised: = x t i,K − x t i, 0 , under the update rule in Algorithm Algorithm 1, we have:

Original: [Equation]
Revised: [Equation]

Original: (11) where (cid:80) K − 1 k =
Revised: (11) where (cid:80) K − 1 k =

Original: [Equation]
Revised: [Equation]

Original: k = (cid:80) K
Revised: k = (cid:80) K

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: l
Revised: l

Original: [Equation]
Revised: [Equation]

Original: (cid:1)
Revised: (cid:1)

Original: [Equation]
Revised: [Equation]

Original: Proof 

Revised: Proof

Original: According to the update rule of Line.11 in Algorithm Algorithm
Revised: According to the update rule of Line.11 in Algorithm Algorithm

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: Then We can formulate the iterative relationship of ∆ ti
Revised: Then We can formulate the iterative relationship of ∆ ti

Original: [Equation]
Revised: [Equation]

Original: k as:
Revised: k as:

Original: [Equation]
Revised: [Equation]

Original: Taking the iteration on k
Revised: Taking the iteration on k

Original: and we have:
Revised: and we have:

Original: [Equation]
Revised: [Equation]

Original: − 1 i .
Revised: − 1 i .

Original: ∆
Revised: ∆

Original: = 0 .
Revised: = 0 .

Original: C. P ROOF OF E QUATION (3)
Revised: C. P ROOF OF E QUATION (3)

Original: Equation (3) shows the update of the prox-correction term, which utilizes the weighted sum of the previous local offsets as a bias controller for eliminating the non-vanishing bias resulting from the prox-term.
Revised: Equation (3) shows the update of the prox-correction term, which utilizes the weighted sum of the previous local offsets as a bias controller for eliminating the non-vanishing bias resulting from the prox-term.

Original: Lemma C.2 Under the update rule in Algorithm Algorithm 1, we have:
Revised: Lemma C.2 Under the update rule in Algorithm Algorithm 1, we have:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Proof 

Revised: Proof

Original: According to the update rule of Line.13 in Algorithm Algorithm
Revised: According to the update rule of Line.13 in Algorithm Algorithm

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (a) applies the Lemma C
Revised: (a) applies the Lemma C

Original: C. P ROOF OF E QUATION (4) AND (5)
Revised: C. P ROOF OF E QUATION (4) AND (5)

Original: Lemma C.

Revised: Lemma C.

Original: Considering the u t +

Revised: Considering the u t +

Original: = 1 m (cid:80) i ∈
Revised: = 1 m (cid:80) i ∈

Original: [ m ] x ti,K is the mean averaged parameters among the last iteration of local clients at time t , the auxiliary sequence (cid:8) z t = u t + 1 − γγ ( u t − u t − 1 ) (cid:9) t>satisfies the update rule as:
Revised: [ m ] x ti,K is the mean averaged parameters among the last iteration of local clients at time t , the auxiliary sequence (cid:8) z t = u t + 1 − γγ ( u t − u t − 1 ) (cid:9) t>satisfies the update rule as:

Original: [Equation]
Revised: [Equation]

Original: Proof 3 Firstly, according to the lemma C.1 and Line.14 and Line.16 in Algorithm 1, we have:
Revised: Proof 3 Firstly, according to the lemma C.1 and Line.14 and Line.16 in Algorithm 1, we have:

Original: [Equation]
Revised: [Equation]

Original: This could be considered as a momentum-like term with the coefficient of γ .
Revised: This could be considered as a momentum-like term with the coefficient of γ .

Original: Here we define a virtual observation sequence { u t } and its update rule is:
Revised: Here we define a virtual observation sequence { u t } and its update rule is:

Original: [Equation]
Revised: [Equation]

Original: According to the lemma C.2 and above update rule, we can get that:
Revised: According to the lemma C.2 and above update rule, we can get that:

Original: [Equation]
Revised: [Equation]

Original: This function indicates that the virtual sequence u t could be considered as a momentum-based update method with a global correction term to guide the local update, and the correction term is calculated from the offset of the virtual observation sequence during the training process at round t .
Revised: This function indicates that the virtual sequence u t could be considered as a momentum-based update method with a global correction term to guide the local update, and the correction term is calculated from the offset of the virtual observation sequence during the training process at round t .

Original: Then we expand the the auxiliary sequence z t as:
Revised: Then we expand the the auxiliary sequence z t as:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: C. P ROOF OF T HEOREM 4. 
Revised: C. P ROOF OF T HEOREM 4.

Original: Firstly we state some important lemmas applied in the proof
Revised: Firstly we state some important lemmas applied in the proof

Original: [Equation]
Revised: [Equation]

Original: Lemma C.4 (Bounded global update)
Revised: Lemma C.4 (Bounded global update)

Original: The global update 1 m (cid:80) i ∈ [ m ] ˆ g ti holds the upper bound of:
Revised: The global update 1 m (cid:80) i ∈ [ m ] ˆ g ti holds the upper bound of:

Original: [Equation]
Revised: [Equation]

Original: Proof 

Revised: Proof

Original: According to the lemma C
Revised: According to the lemma C

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: Take the L2-norm and we have:
Revised: Take the L2-norm and we have:

Original: [Equation]
Revised: [Equation]

Original: Thus we have the following recursion,
Revised: Thus we have the following recursion,

Original: [Equation]
Revised: [Equation]

Original: Lemma C.5 (Bounded local update)
Revised: Lemma C.5 (Bounded local update)

Original: The local update ˆ g ti holds the upper bound of:
Revised: The local update ˆ g ti holds the upper bound of:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Proof 

Revised: Proof

Original: According to the lemmaC.2, we have:
Revised: According to the lemmaC.2, we have:

Original: [Equation]
Revised: [Equation]

Original: Take the L2-norm and we have:
Revised: Take the L2-norm and we have:

Original: [Equation]
Revised: [Equation]

Original: (a) and (b) apply the Jensen inequality.
Revised: (a) and (b) apply the Jensen inequality.

Original: Thus we have the following recursion:
Revised: Thus we have the following recursion:

Original: [Equation]
Revised: [Equation]

Original: Here we provide a loose upper bound as a constant for the quasi-stochastic gradient:
Revised: Here we provide a loose upper bound as a constant for the quasi-stochastic gradient:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We applies the Jensen inequality, the basic inequality ∥ (cid:80) ni
Revised: We applies the Jensen inequality, the basic inequality ∥ (cid:80) ni

Original: =

Revised: =

Original: a i ∥ 2 ≤ n (cid:80) ni
Revised: a i ∥ 2 ≤ n (cid:80) ni

Original: =1 ∥ a i ∥ 2 , and the upper bound of ρ ≤ 1 αL .
Revised: =1 ∥ a i ∥ 2 , and the upper bound of ρ ≤ 1 αL .

Original: Combining the above inequalities, let 1 P = 1 − 24 L 2 λ 2 (1 − 2 γ 2 ) γis the constant, we have:
Revised: Combining the above inequalities, let 1 P = 1 − 24 L 2 λ 2 (1 − 2 γ 2 ) γis the constant, we have:

Original: [Equation]
Revised: [Equation]

Original: C.4. L- SMOOTHNESS OF THE F UNCTION F
Revised: C.4. L- SMOOTHNESS OF THE F UNCTION F

Original: For the general non-convex case, according to the Assumptions and the smoothness of F
Revised: For the general non-convex case, according to the Assumptions and the smoothness of F

Original: [Equation]
Revised: [Equation]

Original: we take the conditional expectation at round t + 1 and expand the F ( z t +1 ) as:
Revised: we take the conditional expectation at round t + 1 and expand the F ( z t +1 ) as:

Original: [Equation]
Revised: [Equation]

Original: C.4. B OUNDED R 
Revised: C.4. B OUNDED R

Original: Note that R1 can be bounded as:
Revised: Note that R1 can be bounded as:

Original: [Equation]
Revised: [Equation]

Original: ) applies the fact that i ∈ m
Revised: ) applies the fact that i ∈ m

Original: F i ( z t ) = ∇ F ( z t ) .
Revised: F i ( z t ) = ∇ F ( z t ) .

Original: applies −⟨ x , y ⟩ = 12 (cid:0)
Revised: applies −⟨ x , y ⟩ = 12 (cid:0)

Original: ∥ x ∥ 2 + ∥ y ∥ 2 − ∥ x + y ∥ 2 (cid:1) .
Revised: ∥ x ∥ 2 + ∥ y ∥ 2 − ∥ x + y ∥ 2 (cid:1) .

Original: (c) applies the Jensen’s inequality and the fact that (cid:80) K − 1 k =0 γ k γ = 1 .
Revised: (c) applies the Jensen’s inequality and the fact that (cid:80) K − 1 k =0 γ k γ = 1 .

Original: According to the update rule we have:
Revised: According to the update rule we have:

Original: [Equation]
Revised: [Equation]

Original: Let ρ ≤ 1 √ 3 αL , thus we could bound the term R1 .
Revised: Let ρ ≤ 1 √ 3 αL , thus we could bound the term R1 .

Original: a as follows:
Revised: a as follows:

Original: [Equation]
Revised: [Equation]

Original: m
Revised: m

Original: [Equation]
Revised: [Equation]

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: (a) applies the bound of ρ as ρ ≤
Revised: (a) applies the bound of ρ as ρ ≤

Original: [Equation]
Revised: [Equation]

Original: (b) applies the lemma C.4.
Revised: (b) applies the lemma C.4.

Original: These others use the fact
Revised: These others use the fact

Original: E [ x − E
Revised: E [ x − E

Original: [ x ]] 2 = E [ x 2 ] − [ E [ x ]] 2 and ∥ x + y ∥ 2 ≤ (1 + a )
Revised: [ x ]] 2 = E [ x 2 ] − [ E [ x ]] 2 and ∥ x + y ∥ 2 ≤ (1 + a )

Original: ∥ x ∥ 2 + (1 + 1 a ) ∥
Revised: ∥ x ∥ 2 + (1 + 1 a ) ∥

Original: y
Revised: y

Original: ∥ 2 .
Revised: ∥ 2 .

Original: We denote c t = 1 m (cid:80) i ∈ m (cid:80) K − 1 k =

Revised: We denote c t = 1 m (cid:80) i ∈ m (cid:80) K − 1 k =

Original: ( γ k /γ )
Revised: ( γ k /γ )

Original: E t ∥ x ti,k − x t ∥ 2 term as the local offset after k iterations updates, we firstly consider the c t k = 1 m (cid:80) i ∈
Revised: E t ∥ x ti,k − x t ∥ 2 term as the local offset after k iterations updates, we firstly consider the c t k = 1 m (cid:80) i ∈

Original: m E t ∥ x t i,k − x t ∥ 2 and it can be bounded as:
Revised: m E t ∥ x t i,k − x t ∥ 2 and it can be bounded as:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: When P satisfies the condition of P ≤ 2 , which means 1 P = 1 − 24 λ 2 L
Revised: When P satisfies the condition of P ≤ 2 , which means 1 P = 1 − 24 λ 2 L

Original: [Equation]
Revised: [Equation]

Original: 1 −
Revised: 1 −

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ≥ 1 2 , then we have the constant of 2( P − 1)≤ 23 < 1 , let the last 12 σ 2 g enlarged to 16 σ 2 g for convenience, we have:
Revised: ≥ 1 2 , then we have the constant of 2( P − 1)≤ 23 < 1 , let the last 12 σ 2 g enlarged to 16 σ 2 g for convenience, we have:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Here we get the recursion formula between the c tk and c tk − 1 .
Revised: Here we get the recursion formula between the c tk and c tk − 1 .

Original: Actually we need to upper bound the c t =
Revised: Actually we need to upper bound the c t =

Original: [Equation]
Revised: [Equation]

Original: K − 1 k =0 ( γ k /γ )
Revised: K − 1 k =0 ( γ k /γ )

Original: c tk , thus let the weight satisfies that:
Revised: c tk , thus let the weight satisfies that:

Original: [Equation]
Revised: [Equation]

Original: let η l ≤ λ and thus we have:
Revised: let η l ≤ λ and thus we have:

Original: [Equation]
Revised: [Equation]

Original: (a) enlarge the sum from k ′ to K − 1 where k ′ ≤ K − 1 .
Revised: (a) enlarge the sum from k ′ to K − 1 where k ′ ≤ K − 1 .

Original: Let η l satisfies the upper bound of η l ≤ 1 √
Revised: Let η l satisfies the upper bound of η l ≤ 1 √

Original: [Equation]
Revised: [Equation]

Original: 1+1 /a ) KL for convenience, we can bound the c t as:
Revised: 1+1 /a ) KL for convenience, we can bound the c t as:

Original: [Equation]
Revised: [Equation]

Original: Let the a satisfies a = 1 for convenience
Revised: Let the a satisfies a = 1 for convenience

Original: [Equation]
Revised: [Equation]

Original: we summarize the extra terms above and bound the term
Revised: we summarize the extra terms above and bound the term

Original: R1 .
Revised: R1 .

Original: a as:
Revised: a as:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: thus we can bound the R1 as follow:
Revised: thus we can bound the R1 as follow:

Original: [Equation]
Revised: [Equation]

Original: We notice that R1 contains the same term with a negative weight, thus we can set another constrains for λ to eliminate this term.
Revised: We notice that R1 contains the same term with a negative weight, thus we can set another constrains for λ to eliminate this term.

Original: We will prove it in the next part.
Revised: We will prove it in the next part.

Original: C.4. B OUNDED G LOBAL G RADIENT
Revised: C.4. B OUNDED G LOBAL G RADIENT

Original: As we have bounded the term R1 and R
Revised: As we have bounded the term R1 and R

Original: [Equation]
Revised: [Equation]

Original: according to the smoothness inequality
Revised: according to the smoothness inequality

Original: [Equation]
Revised: [Equation]

Original: we combine the inequalities above and get the inequality:
Revised: we combine the inequalities above and get the inequality:

Original: [Equation]
Revised: [Equation]

Original: We follow as Yang et al.
Revised: We follow as Yang et al.

Original: (2021) to set λ that it satisfies 4 λ 3 L 2 (1 − 2 γ ) 2 γ+ λ
Revised: (2021) to set λ that it satisfies 4 λ 3 L 2 (1 − 2 γ ) 2 γ+ λ

Original: [Equation]
Revised: [Equation]

Original: L 2 m 2 − λ 2 m 2 ≤ 0 , which is easy to verified that λ has a upper bound for the quadratic inequality.
Revised: L 2 m 2 − λ 2 m 2 ≤ 0 , which is easy to verified that λ has a upper bound for the quadratic inequality.

Original: Thus, the stochastic gradient term is diminished by this λ .
Revised: Thus, the stochastic gradient term is diminished by this λ .

Original: We denote the constant λκ = λ
Revised: We denote the constant λκ = λ

Original: 2 − 3 λα 2 L 2 ρ 2 − 1536 λη 2 l L 2 K and κ could be considered as a constant.
Revised: 2 − 3 λα 2 L 2 ρ 2 − 1536 λη 2 l L 2 K and κ could be considered as a constant.

Original: We can select two constants c 1 ∈ (0 , 12 ) , c 2 ∈ (0 , 12 ) and they satisfy c 1 + c 2 ∈ (0 , 12 ) , we let 12 − 3 α 2 L 2 ρ 2 > 12 − c 1 and 12 − 1536 η 2 l L 2 K > 12 − c 2 , where the ρ and η l satisfy ρ < √ c 1 √ 3 αL <
Revised: We can select two constants c 1 ∈ (0 , 12 ) , c 2 ∈ (0 , 12 ) and they satisfy c 1 + c 2 ∈ (0 , 12 ) , we let 12 − 3 α 2 L 2 ρ 2 > 12 − c 1 and 12 − 1536 η 2 l L 2 K > 12 − c 2 , where the ρ and η l satisfy ρ < √ c 1 √ 3 αL <

Original: [Equation]
Revised: [Equation]

Original: 6 αL and η l < √
Revised: 6 αL and η l < √

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: < 1 32 √ 3 KL .
Revised: < 1 32 √ 3 KL .

Original: Then we can bound the κ = 1 2 − 3 α 2 L 2 ρ 2 − 1536 η 2 l L 2 K > 12 − c 1 − c 2 > 0 , and the term 1 κ < 2 1 − 2 c 1 − 2 c 2 which is a constant upper bound.
Revised: Then we can bound the κ = 1 2 − 3 α 2 L 2 ρ 2 − 1536 η 2 l L 2 K > 12 − c 1 − c 2 > 0 , and the term 1 κ < 2 1 − 2 c 1 − 2 c 2 which is a constant upper bound.

Original: We take the full expectation on the bounded global gradient as:
Revised: We take the full expectation on the bounded global gradient as:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Take the full expectation and telescope sum on the inequality above and applying the fact that
Revised: Take the full expectation and telescope sum on the inequality above and applying the fact that

Original: F ∗ ≤ F ( x ) for x ∈ R d
Revised: F ∗ ≤ F ( x ) for x ∈ R d

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: Here we summarize the conditions and some constrains in the above conclusion.
Revised: Here we summarize the conditions and some constrains in the above conclusion.

Original: Firstly we should note that γ = 1 − (1 − η l λ ) K < 1 when η l ≤ 2 λ .
Revised: Firstly we should note that γ = 1 − (1 − η l λ ) K < 1 when η l ≤ 2 λ .

Original: Thus we have 1 /γ > 1 .
Revised: Thus we have 1 /γ > 1 .

Original: When K satisfies that K ≥ λη l , (1 − η l λ ) K ≤ e − ηlλ K ≤ e − 1 , and then γ > 1 − e − 1 and 1 /γ <
Revised: When K satisfies that K ≥ λη l , (1 − η l λ ) K ≤ e − ηlλ K ≤ e − 1 , and then γ > 1 − e − 1 and 1 /γ <

Original: ee − 1 < 2 .
Revised: ee − 1 < 2 .

Original: To let κ = 12 − 3 α 2 L 2 ρ 2 − 1536 η 2 l L 2 K > 0 hold, ρ and η l satisfy that ρ <
Revised: To let κ = 12 − 3 α 2 L 2 ρ 2 − 1536 η 2 l L 2 K > 0 hold, ρ and η l satisfy that ρ <

Original: [Equation]
Revised: [Equation]

Original: 6 αL and η l < 1 32 √ 3 KL .
Revised: 6 αL and η l < 1 32 √ 3 KL .

Original: [Table] [Equation]
Revised: [Table] [Equation]