Original: Escaping Saddle Points with Bias-Variance Reduced Local Perturbed SGD for Communication Efficient Nonconvex Distributed Learnings
Revised: Escaping Saddle Points with Bias-Variance Reduced Local Perturbed SGD for Communication Efficient Nonconvex Distributed Learning

Original: Abstract
Revised: Abstract

Original: In recent centralized nonconvex distributed learning and federated learning, local methods are one of the promising approaches to reduce communication time.
Revised: In recent centralized nonconvex distributed learning and federated learning, local methods are one of the promising approaches to reduce communication time.

Original: However, existing work has mainly focused on studying first-order optimalityguarantees.
Revised: However, existing work has mainly focused on studying first-order optimality guarantees.

Original: On the other side, second-order optimality guaranteed algorithms,i.e., algorithms escaping saddle points, have been extensively studied in the nondistributed optimization literature.
Revised: On the other side, second-order optimality guaranteed algorithms, i.e., algorithms escaping saddle points, have been extensively studied in the nondistributed optimization literature.

Original: In this paper, we study a new local algorithmcalled Bias-Variance Reduced Local Perturbed SGD (BVR-L-PSGD), that combines the existing bias-variance reduced gradient estimator with parameter perturbation to find second-order optimal points in centralized nonconvex distributedoptimization.
Revised: In this paper, we study a new local algorithm called Bias-Variance Reduced Local Perturbed SGD (BVR-L-PSGD), that combines the existing bias-variance reduced gradient estimator with parameter perturbation to find second-order optimal points in centralized nonconvex distributed optimization.

Original: BVR-L-PSGD enjoys second-order optimality with nearly the samecommunication complexity as the best known one of BVR-L-SGD to find first-orderoptimality.
Revised: BVR-L-PSGD enjoys second-order optimality with nearly the same communication complexity as the best known one of BVR-L-SGD to find first-order optimality.

Original: Particularly, the communication complexity is better than non-localmethods when the local datasets heterogeneity is smaller than the smoothness ofthe local loss.
Revised: Particularly, the communication complexity is better than non-local methods when the local datasets heterogeneity is smaller than the smoothness of the local loss.

Original: In an extreme case, the communication complexity approaches to(cid:101) Θ(1) when the local datasets heterogeneity goes to zero.
Revised: In an extreme case, the communication complexity approaches to (cid:101) Θ(1) when the local datasets heterogeneity goes to zero.

Original: Numerical results validateour theoretical findings.
Revised: Numerical results validate our theoretical findings.

Original:  Introduction
Revised:  Introduction

Original: Distributed learning is an attractive approach to reduce the total execution time by utilizing theparallel computations.
Revised: Distributed learning is an attractive approach to reduce the total execution time by utilizing the parallel computations.

Original: However, the communication time in distributed learning can be a mainbottleneck in the entire process due to huge parameter size typical in deep learning or low bandwidthcommunication environments.
Revised: However, the communication time in distributed learning can be a main bottleneck in the entire process due to huge parameter size typical in deep learning or low bandwidth communication environments.

Original: To reduce communication time, one of the promising approaches is the usage of local methodssuch as local SGD (also called as Parallel Restart SGD or FedAvg).
Revised: To reduce communication time, one of the promising approaches is the usage of local methods such as local SGD (also called as Parallel Restart SGD or FedAvg).

Original: In local SGD, each workerindependently executes multiple updates of the local model based on his own local dataset and theserver periodically communicates and aggregates the local models.
Revised: In local SGD, each worker independently executes multiple updates of the local model based on his own local dataset and the server periodically communicates and aggregates the local models.

Original: Many paper have studied local
Revised: Many paper have studied local

Original: SGD [25, 30, 7, 6, 15, 13, 27, 28].
Revised: SGD [25, 30, 7, 6, 15, 13, 27, 28].

Original: Particularly, for convex objectives, it has been shown in [28], forthe first time, the communication complexity (that is the necessary number of communication roundsto achieve given desired optimization error) of local SGD can be smaller than the one of minibatch
Revised: Particularly, for convex objectives, it has been shown in [28], for the first time, the communication complexity (that is the necessary number of communication rounds to achieve given desired optimization error) of local SGD can be smaller than the one of minibatch

Original: SGD when the heterogeneity of the local datasets is extremely small.
Revised: SGD when the heterogeneity of the local datasets is extremely small.

Original: In traditional distributedlearning, the local datasets are typically random subsets of the global dataset and in this case theheterogeneity of the local datasets may become quite small.
Revised: In traditional distributed learning, the local datasets are typically random subsets of the global dataset and in this case the heterogeneity of the local datasets may become quite small.

Original: However, in the recent federated learningregimes [16, 24, 18], it is often the case that the heterogeneity of the local datasets is not too small.
Revised: However, in the recent federated learning regimes [16, 24, 18], it is often the case that the heterogeneity of the local datasets is not too small.

Original: Also, the analysis in [28] has only focused on convex cases.
Revised: Also, the analysis in [28] has only focused on convex cases.

Original: Hence, the superiority of local SGD tominibatch SGD is still quite limited.
Revised: Hence, the superiority of local SGD to minibatch SGD is still quite limited.

Original: Recently, more communication efficient local methods than local SGD have been proposed forpossibly nonconvex objectives to guarantee first-order optimality [12, 22, 19].
Revised: Recently, more communication efficient local methods than local SGD have been proposed for possibly nonconvex objectives to guarantee first-order optimality [12, 22, 19].

Original: SCAFFOLD [12]is a new local algorithm based on the idea of reducing their called client-drift by using a similarformulation to the variance reduction technique [10].
Revised: SCAFFOLD [12] is a new local algorithm based on the idea of reducing their called client-drift by using a similar formulation to the variance reduction technique [10].

Original: They have shown that the communicationcomplexity of SCAFFOLD can be smaller than the one of minibatch SGD for not too heterogenouslocal datasets under the quadraticity of the (possibly nonconvex) local objectives, which is quitelimited.
Revised: They have shown that the communication complexity of SCAFFOLD can be smaller than the one of minibatch SGD for not too heterogenous local datasets under the quadraticity of the (possibly nonconvex) local objectives, which is quite limited.

Original: For general nonconvex objectives, the communication complexity of SCAFFOLD is sameas minibatch SGD.
Revised: For general nonconvex objectives, the communication complexity of SCAFFOLD is same as minibatch SGD.

Original: More recently, Murate and Suzuki [19] have proposed Bias-Variance Reduced
Revised: More recently, Murate and Suzuki [19] have proposed Bias-Variance Reduced

Original: Local SGD (BVR-L-SGD).
Revised: Local SGD (BVR-L-SGD).

Original: BVR-L-SGD utilizes their proposed bias-variance reduced estimatorthat simultaneously reduces the bias caused by local gradient steps and the variance caused bystochastization of the gradients in local optimization based on the formulation of SARAH likevariance reduction [20].
Revised: BVR-L-SGD utilizes their proposed bias-variance reduced estimator that simultaneously reduces the bias caused by local gradient steps and the variance caused by stochastization of the gradients in local optimization based on the formulation of SARAH like variance reduction [20].

Original: They have shown that the communication complexity of BVR-L-SGD issmaller than minibatch SGD for not too heterogeneous local datasets for general nonconvex objectives .
Revised: They have shown that the communication complexity of BVR-L-SGD is smaller than minibatch SGD for not too heterogeneous local datasets for general nonconvex objectives .

Original: Specifically, BVR-L-SGD is superior to minibatch SGD when the Hessian heterogeneity of the localdatasets is small relative to the smoothness of the local loss in order sense.
Revised: Specifically, BVR-L-SGD is superior to minibatch SGD when the Hessian heterogeneity of the local datasets is small relative to the smoothness of the local loss in order sense.

Original: On the other side, there are vast work that has studied second-order optimality guarantees, which ismuch more challenging to ensure but desirable than first-order one, in non-distributed nonconvexoptimization.
Revised: On the other side, there are vast work that has studied second-order optimality guarantees, which is much more challenging to ensure but desirable than first-order one, in non-distributed nonconvex optimization.

Original: Several approaches are known and one of the simplest approaches is parameterperturbation [4, 8, 9].
Revised: Several approaches are known and one of the simplest approaches is parameter perturbation [4, 8, 9].

Original: However, almost all existing analysis of local methods have only focusedon achieving first-order optimality.
Revised: However, almost all existing analysis of local methods have only focused on achieving first-order optimality.

Original: As an exception, Vlaski et al.
Revised: As an exception, Vlaski et al.

Original: [26] have analysed second-orderguarantees of local SGD with parameter perturbation, but the obtained communication complexity ismuch worse than the one of minibatch SGD and no benefit of localization has been shown.
Revised: [26] have analysed second-order guarantees of local SGD with parameter perturbation, but the obtained communication complexity is much worse than the one of minibatch SGD and no benefit of localization has been shown.

Original: Open question.
Revised: Open question.

Original: For local methods, it is not well-studied how to find second-order optimal pointswith low communication cost and thus we have the following research questions:
Revised: For local methods, it is not well-studied how to find second-order optimal points with low communication cost and thus we have the following research questions:

Original: Is there a first-order distributed optimization algorithm with second-order optimality guaranteeswhich satisfies that (i) the communication complexity is smaller than non-local methods for not tooheterogeneous local datasets; and (ii) the communication complexity approaches to Θ(1) when theheterogeneity of local datasets goes to zero?
Revised: Is there a first-order distributed optimization algorithm with second-order optimality guarantees which satisfies that (i) the communication complexity is smaller than non-local methods for not too heterogeneous local datasets; and (ii) the communication complexity approaches to Θ(1) when the heterogeneity of local datasets goes to zero?

Original: Note that the both properties are desirable in distributed optimization.
Revised: Note that the both properties are desirable in distributed optimization.

Original: We expect that local methodsare superior to non-local methods for not highly heterogeneous local datasets.
Revised: We expect that local methods are superior to non-local methods for not highly heterogeneous local datasets.

Original: Furthermore, when thelocal datasets are nearly identical, it is expected that a few communications are sufficient to optimizethe global objective.
Revised: Furthermore, when the local datasets are nearly identical, it is expected that a few communications are sufficient to optimize the global objective.

Original: Only in the case of first-order optimality , Murata and Suzuki [19] have shownthat their proposed BVR-L-SGD satisfies (i) and (ii), and the question has been positively answered.
Revised: Only in the case of first-order optimality , Murata and Suzuki [19] have shown that their proposed BVR-L-SGD satisfies (i) and (ii), and the question has been positively answered.

Original: However, the question is still open in the case of second-order optimality 1 .
Revised: However, the question is still open in the case of second-order optimality 3 .

Original: Main Contributions
Revised: Main Contributions

Original: We propose a new local algorithm called Bias-Variance Reduced Local Perturbed SGD (BVR-LPSGD) for nonconvex distributed learning to efficiently find second-order optimal points, whichpositively answered the above research questions.
Revised: We propose a new local algorithm called Bias-Variance Reduced Local Perturbed SGD (BVR-LPSGD) for nonconvex distributed learning to efficiently find second-order optimal points, which positively answered the above research questions.

Original: The algorithm is based on a simple combination of the existing bias and variance reduced gradientestimator and parameter perturbation.
Revised: The algorithm is based on a simple combination of the existing bias and variance reduced gradient estimator and parameter perturbation.

Original: In our algorithm, parameter perturbation is carried out at everylocal update and it is not necessary to determine whether or not to add noise by checking the norm ofthe global gradients, which is often required in several previous non-distributed algorithms [5, 17]. We analyse BVR-L-PSGD for general nonconvex smooth objectives. The most challenging part ofour analysis is to ensure that our algorithm efficiently escapes global saddle points even in localoptimization . To realize this, it is necessary to analyse the behavior of the bias-variance reducedestimator around the saddle points by carefully evaluating the degree of some kind of asymptotic1 Since there are communication efficient distributed optimization algorithms that find first-order stationary points like BVR-L-SGD, we can apply generic algorithms which guarantee second-order optimality to them [29, 1].
Revised: In our algorithm, parameter perturbation is carried out at every to them [29, 1].

Original: However, this naive approach does not possess the aforementioned property (ii) because the generic framework requires at least Ω(1 /ε 3 / 2 ) communication rounds to guarantee second-order optimality due to the multiple negative curvature exploitation steps in their framework for any communication efficient algorithms with first-order optimality guarantees.
Revised: However, this naive approach does not possess the aforementioned property (ii) because the generic framework requires at least Ω(1 /ε 3 / 2 ) communication rounds to guarantee second-order optimality due to the multiple negative curvature exploitation steps in their framework for any communication efficient algorithms with first-order optimality guarantees.

Original: Also, this approach requires explicit negative curvature exploitation, that is complicated and makes the whole algorithm less practical.
Revised: Also, this approach requires explicit negative curvature exploitation, that is complicated and makes the whole algorithm less practical.

Original: [Table]
Revised: [Table]

Original: Assumptions
Revised: Assumptions

Original: Guarantee
Revised: Guarantee

Original: Minibatch SGD 1 ε 2 + 1 B Pε2-3, BSGV 1st-order
Revised: Minibatch SGD [Table]

Original: Noisy Minibatch SGD [9]1 ε 2 + 1 B Pε22nd-order
Revised: Noisy Minibatch SGD [9]

Original:  Minibatch SARAH [21]1 ε 2 + √ n B Pε 2 + n B P 21st-order
Revised: [Table] Minibatch SARAH [21]

Original:  SSRGD [17]1 ε 2 + √ n ε2-5, B ≥ √ nP 2nd-order
Revised: [Table] SSRGD [17]

Original:  Local SGD [30] 1 B ε 2 + 1 B Pε 4 + 1 ε2-3,1st-order
Revised: [Table] [Table] Local SGD [30] [Table]

Original: SCAFFOLD [12] 1 ε 2 + 1 B Pε2-3, BSGV 1st-order
Revised: SCAFFOLD [12] [Table] [Table]

Original:  SCAFFOLD [12]1 B ε 2 + 1 B Pε 4 + ζε1-3, BSGV, quadraticity 1st-order
Revised: order SCAFFOLD [12]

Original: BVR-L-SGD [19] 1 √B ε 2 + √ n
Revised: [Table] [Table]

Original: B Pε 2 + ζε11st-order BVR-L-PSGD (this paper) 1 √B ε 2 +
Revised: , quadraticity

Original: [Table]
Revised: [Table]

Original: n
Revised: BVR

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: 2 + ζε12nd-order
Revised: [Table] (this paper) [Table] [Table]

Original: ( L ) = Θ(1) .
Revised: ( L ) = Θ(1) .

Original:  consistency of the estimator around the saddle points.
Revised: local update and it is not necessary to determine whether or not to add noise by checking the norm of the global gradients, which is often required in several previous non-distributed algorithms [5, 17]. We analyse BVR-L-PSGD for general nonconvex smooth objectives. The most challenging part of our analysis is to ensure that our algorithm efficiently escapes global saddle points even in local optimization . To realize this, it is necessary to analyse the behavior of the bias-variance reduced estimator around the saddle points by carefully evaluating the degree of some kind of asymptotic consistency of the estimator around the saddle points.

Original: This point has never been pursued in previouswork and has a unique difficulty of our analysis.
Revised: This point has never been pursued in previous work and has a unique difficulty of our analysis.

Original: The comparison of the communication complexities of our method with the most relevant existingresults is given in Table 1.
Revised: The comparison of the communication complexities of our method with the most relevant existing results is given in Table 1.

Original: Our proposed method enjoys second-order optimality with nearly the samecommunication complexity as the one of BVR-L-SGD, which achieves the best known communicationcomplexity to achieve first-order optimality.
Revised: Our proposed method enjoys second-order optimality with nearly the same communication complexity as the one of BVR-L-SGD, which achieves the best known communication complexity to achieve first-order optimality.

Original: This means that our method finds second-order optimalpoints without hurting the communication efficiency of the state-of-the-art first-order optimalityguaranteed method.
Revised: This means that our method finds second-order optimal points without hurting the communication efficiency of the state-of-the-art first-order optimality guaranteed method.

Original: Particularly, the communication complexity is better than minibatch SGD when
Revised: Particularly, the communication complexity is better than minibatch SGD when

Original: Hessian heterogeneity ζ is small relative to smoothness L .
Revised: Hessian heterogeneity ζ is small relative to smoothness L .

Original: Also, the communication complexityapproaches to (cid:101) O
Revised: Also, the communication complexity approaches to (cid:101) O

Original: (1) when heterogeneity ζ goes to zero and the local computation budget B (see
Revised: (1) when heterogeneity ζ goes to zero and the local computation budget B (see

Original: Section 2) goes to infinity.
Revised: Section 2) goes to infinity.

Original: Hence, our method enjoys the aforementioned two desired properties.
Revised: Hence, our method enjoys the aforementioned two desired properties.

Original: Related Work
Revised: Related Work

Original: Here, we briefly review the related studies to our paper.
Revised: Here, we briefly review the related studies to our paper.

Original: Local methods.
Revised: Local methods.

Original: Several recent papers have studied local algorithms combined with variance reductiontechnique [23, 2, 14, 11].
Revised: Several recent papers have studied local algorithms combined with variance reduction technique [23, 2, 14, 11].

Original: Sharma et al.
Revised: Sharma et al.

Original: [23] have proposed a local variant of SPIDER [3] and shownthat the proposed algorithm achieves the optimal total computational complexity.
Revised: [23] have proposed a local variant of SPIDER [3] and shown that the proposed algorithm achieves the optimal total computational complexity.

Original: However, thecommunication complexity essentially matches the ones of non-local SARAH and no advantage oflocalization has been shown.
Revised: However, the communication complexity essentially matches the ones of non-local SARAH and no advantage of localization has been shown.

Original: Khanduri et al.
Revised: Khanduri et al.

Original: [14] have proposed STEM and its variants based on theircalled two-sided momentum, but again the communication complexity does not improve non-localmethods.
Revised: [14] have proposed STEM and its variants based on their called two-sided momentum, but again the communication complexity does not improve non-local methods.

Original: Also, Das et al.
Revised: Also, Das et al.

Original: [2] have considered a SPIDER like local algorithm called FedGLOMO butthe derived communication complexity is even worse than minibatch SARAH.
Revised: [2] have considered a SPIDER like local algorithm called FedGLOMO but the derived communication complexity is even worse than minibatch SARAH.

Original: Karimireddy et al.[11] have proposed Mime, which is a general framework to mitigate client-drift.
Revised: Karimireddy et al. [11] have proposed Mime, which is a general framework to mitigate client-drift.

Original: Particularly, underδ -Bounded Hessian Dissimilarity (BHD) 2 , their MimeMVR achieves communication complexity of1 / ( Pε 2 )+ δ/ ( √ Pε 3 )
Revised: Particularly, under δ -Bounded Hessian Dissimilarity (BHD) 4 , their MimeMVR achieves communication complexity of 1 / ( Pε 2 )+ δ/ ( √ Pε 3 )

Original: + δ/ε 2 when B → ∞ , that is better than the one of minibatch SGD 1 /ε 2 whenδ ≤ √ Pε .
Revised: + δ/ε 2 when B → ∞ , that is better than the one of minibatch SGD 1 /ε 2 when δ ≤ √ Pε .

Original: However, the asymptotic rate is still worse than the one of BVR-L-SGD ζ/ε 2 becauseζ ≤ δ always holds.
Revised: However, the asymptotic rate is still worse than the one of BVR-L-SGD ζ/ε 2 because ζ ≤ δ always holds.

Original: Second-order guarantee.
Revised: Second-order guarantee.

Original: Neon [29] and Neon2 [1] are generic first-order methods with second-orderguarantees, that repeatedly run a first-order guaranteed algorithm and negative curvature descent.
Revised: Neon [29] and Neon2 [1] are generic first-order methods with second-order guarantees, that repeatedly run a first-order guaranteed algorithm and negative curvature descent.

Original: Another approach is a parameter perturbation for SGD.
Revised: Another approach is a parameter perturbation for SGD.

Original: For the first time, Ge et al.
Revised: For the first time, Ge et al.

Original: [4] have shown that
Revised: [4] have shown that

Original: SGD with a simple parameter perturbation escapes saddle points efficiently.
Revised: SGD with a simple parameter perturbation escapes saddle points efficiently.

Original: Later, the analysis hasbeen refined by [8, 9].
Revised: Later, the analysis has been refined by [8, 9].

Original: Recently, applying variance reduction technique to second-order guaranteed methods has been also studied [5, 17] and particularly Li et al.
Revised: Recently, applying variance reduction technique to second-order guaranteed methods has been also studied [5, 17] and particularly Li et al.

Original: [17] have proposed SSRGD thatcombines SARAH [20] with parameter perturbation and shown that SSRGD nearly achieves theoptimal computational complexity with second-order optimality guarantees.
Revised: [17] have proposed SSRGD that combines SARAH [20] with parameter perturbation and shown that SSRGD nearly achieves the optimal computational complexity with second-order optimality guarantees.

Original:  Problem Definition and 
Revised:  Problem Definition and Assumptions

Original: Assumptions

Original: In this section, we first introduce several notations and definitions used in this paper.
Revised: In this section, we first introduce several notations and definitions used in this paper.

Original: Then, theproblem settings are described and theoretical assumptions used in our analysis are given.
Revised: Then, the problem settings are described and theoretical assumptions used in our analysis are given.

Original: Notation.
Revised: Notation.

Original: ∥ · ∥ denotes the Euclidean L 2 norm ∥ · ∥ 2 : ∥ x ∥ =
Revised: ∥ · ∥ denotes the Euclidean L 2 norm ∥ · ∥ 2 : ∥ x ∥ =

Original: (cid:112)(cid:80) i x 2 i for vector x .
Revised: (cid:112)(cid:80) i x 2 i for vector x .

Original: For a matrix X ,
Revised: For a matrix X ,

Original: ∥ X ∥ denotes the induced norm by the Euclidean L 2 norm.
Revised: ∥ X ∥ denotes the induced norm by the Euclidean L 2 norm.

Original: For a natural number m , [ m ] means theset { 1 , 2 , . . .
Revised: For a natural number m , [ m ] means the set { 1 , 2 , . . .

Original: , m } .
Revised: , m } .

Original: For a set A , # A means the number of elements, which is possibly ∞ .
Revised: For a set A , # A means the number of elements, which is possibly ∞ .

Original: For anynumber a, b , a ∨ b and a ∧ b denote max { a, b } and min { a, b } respectively.
Revised: For any number a, b , a ∨ b and a ∧ b denote max { a, b } and min { a, b } respectively.

Original: We denote the uniformdistribution over A by Unif( A ) .
Revised: We denote the uniform distribution over A by Unif( A ) .

Original: Given K, T, S ∈ N , let I ( k, t, s ) be integer k + Kt + KTs fork ∈ [ K ] ∪ { 0 } , t ∈
Revised: Given K, T, S ∈ N , let I ( k, t, s ) be integer k + Kt + KTs for k ∈ [ K ] ∪ { 0 } , t ∈

Original: [ T − 1] ∪ { 0 } and s ∈ [ S − 1] ∪ { 0 } .
Revised: [ T − 1] ∪ { 0 } and s ∈ [ S − 1] ∪ { 0 } .

Original: Note that
Revised: Note that

Original: I ( K, t, s ) =
Revised: I ( K, t, s ) =

Original: I (0 , t + 1 , s ) and
Revised: I (0 , t + 1 , s ) and

Original: I ( k, T, s ) =
Revised: I ( k, T, s ) =

Original: I ( k, 0 , s + 1) for k ∈
Revised: I ( k, 0 , s + 1) for k ∈

Original: [ K ] ∪ { 0 } , t ∈
Revised: [ K ] ∪ { 0 } , t ∈

Original: [ T − 1] ∪ { 0 } and s ∈ [ S − 1] ∪ { 0 } .
Revised: [ T − 1] ∪ { 0 } and s ∈ [ S − 1] ∪ { 0 } .

Original: B dr denotesthe set { x ∈ R d |∥ x ∥ ≤ r } , which is the Euclidean ball in R d with radius r .
Revised: B dr denotes the set { x ∈ R d |∥ x ∥ ≤ r } , which is the Euclidean ball in R d with radius r .

Original: Definition 2.1 (Gradient Lipschitzness) .
Revised: Definition 2.1 (Gradient Lipschitzness) .

Original: A differentiable function f : R d → R is L -gradient
Revised: A differentiable function f : R d → R is L -gradient

Original: Lipschitz if ∥∇ f ( x ) − ∇ f ( y ) ∥
Revised: Lipschitz if ∥∇ f ( x ) − ∇ f ( y ) ∥

Original: ≤ L ∥ x − y ∥ , ∀ x, y ∈ R d .
Revised: ≤ L ∥ x − y ∥ , ∀ x, y ∈ R d .

Original: Definition 2.2 (Hessian Lipschitzness) .
Revised: Definition 2.2 (Hessian Lipschitzness) .

Original: A twice differentiable function f : R d → R is ρ -Hessian
Revised: A twice differentiable function f : R d → R is ρ -Hessian

Original: Lipschitz if (cid:13)(cid:13) ∇ 2 f ( x ) − ∇ 2 f ( y ) (cid:13)(cid:13) ≤
Revised: Lipschitz if (cid:13)(cid:13) ∇ 2 f ( x ) − ∇ 2 f ( y ) (cid:13)(cid:13) ≤

Original: ρ ∥ x − y ∥ , ∀ x, y ∈ R d .
Revised: ρ ∥ x − y ∥ , ∀ x, y ∈ R d .

Original: Definition 2.3 (Second-order optimality) .
Revised: Definition 2.3 (Second-order optimality) .

Original: For a ρ -Hessian Lipschitz function f , x ∈ R d is anε -second-order optimal point of f if ∥∇ f ( x ) ∥ ≤ ε and ∇ 2 f ( x ) ⪰ −√ ρεI .
Revised: For a ρ -Hessian Lipschitz function f , x ∈ R d is an ε -second-order optimal point of f if ∥∇ f ( x ) ∥ ≤ ε and ∇ 2 f ( x ) ⪰ −√ ρεI .

Original: 2.1 Problem Settings
Revised: 2.1 Problem Settings

Original: Objective function.
Revised: Objective function.

Original: We want to minimize nonconvex smooth objective f ( x ) :
Revised: We want to minimize nonconvex smooth objective f ( x ) :

Original: = 1 P (cid:80) Pp =1 f p ( x ) ,where f p ( x ) :=
Revised: = 1 P (cid:80) Pp =1 f p ( x ) , where f p ( x ) :=

Original: E z ∼ D p [ ℓ ( x, z )] for x ∈ R d , where D p is the data distribution associated with workerp .
Revised: E z ∼ D p [ ℓ ( x, z )] for x ∈ R d , where D p is the data distribution associated with worker p .

Original: In this paper, we focus on offline settings (i.e., #supp( D p ) <
Revised: In this paper, we focus on offline settings (i.e., #supp( D p ) <

Original: ∞ for every p ∈
Revised: ∞ for every p ∈

Original: [ P ] ) for simplepresentation.
Revised: [ P ] ) for simple presentation.

Original: It is easy to extend our results to online settings.
Revised: It is easy to extend our results to online settings.

Original: Also, just for simplicity, it is assumedthat each local dataset has an equal number of samples, i.e., #supp( D p ) =
Revised: Also, just for simplicity, it is assumed that each local dataset has an equal number of samples, i.e., #supp( D p ) =

Original: n/P for every  p  ∈
Revised: n/P for every p, p ′ ∈

Original: [ P ] ,where n is the total number of samples.
Revised: [ P ] , where n is the total number of samples.

Original: Optimization criteria.
Revised: Optimization criteria.

Original: Since objective function f is nonconvex, it is generally difficult to find aglobal minima of f .
Revised: Since objective function f is nonconvex, it is generally difficult to find a global minima of f .

Original: Previous work in distributed learning has mainly focused on finding first-orderstationary points of f .
Revised: Previous work in distributed learning has mainly focused on finding first-order stationary points of f .

Original: In this study, we aim to find ε -second-order stationary points of f in distributedlearning settings.
Revised: In this study, we aim to find ε -second-order stationary points of f in distributed learning settings.

Original: Data access constraints and communication settings.
Revised: Data access constraints and communication settings.

Original: It is assumed that each worker p can onlyaccess the own data distribution D p without communication.
Revised: It is assumed that each worker p can only access the own data distribution D p without communication.

Original: Aggregation (e.g., summation) of allthe worker’s d -dimensional parameters or broadcast of a d -dimensional parameter from one workerto the other workers can be realized by single communication.
Revised: Aggregation (e.g., summation) of all the worker’s d -dimensional parameters or broadcast of a d -dimensional parameter from one worker to the other workers can be realized by single communication.

Original: Evaluation criteria: communication complexity.
Revised: Evaluation criteria: communication complexity.

Original: In this paper, we compare communicationcomplexities of optimization algorithms to satisfy the aforementioned optimization criteria.
Revised: In this paper, we compare communication complexities of optimization algorithms to satisfy the aforementioned optimization criteria.

Original: In typicalsituations, single communication is more time-consuming than single stochastic gradient computation.
Revised: In typical situations, single communication is more time-consuming than single stochastic gradient computation.

Original: Let C be the single communication cost and G be the single stochastic gradient computation cost.
Revised: Let C be the single communication cost and G be the single stochastic gradient computation cost.

Original: Using these notations, C ≥ G is assumed.
Revised: Using these notations, C ≥ G is assumed.

Original: We expect that increasing the number of available stochasticgradients in a single communication round leads to faster convergence.
Revised: We expect that increasing the number of available stochastic gradients in a single communication round leads to faster convergence.

Original: Hence, it is natural to increasethe number of stochastic gradient computations in a single communication round unless the totalstochastic gradient computation time exceeds C to reduce the total running time.
Revised: Hence, it is natural to increase the number of stochastic gradient computations in a single communication round unless the total stochastic gradient computation time exceeds C to reduce the total running time.

Original: This motivatesthe concept of local computation budget B ( ≤ C / G ): given a communication and computationalenvironment, it is assumed that each worker can only computes at most B single stochastic gradientsper communication round on average .
Revised: This motivates the concept of local computation budget B ( ≤ C / G ): given a communication and computational environment, it is assumed that each worker can only computes at most B single stochastic gradients per communication round on average .

Original: Then, we compare the communication complexity, that is thetotal number of communication rounds of a distributed optimization algorithm to achieve the desired optimization accuracy .
Revised: Then, we compare the communication complexity, that is the total number of communication rounds of a distributed optimization algorithm to achieve the desired optimization accuracy .

Original: From the definition, we can see that the communication complexity on a fixedlocal computation budget B :=
Revised: From the definition, we can see that the communication complexity on a fixed local computation budget B :=

Original: C / G captures the best achievable total running time of an algorithm.
Revised: C / G captures the best achievable total running time of an algorithm.

Original: 2.2 Theoretical Assumptions
Revised: 2.2 Theoretical Assumptions

Original: In this paper, we assume the following five assumptions.
Revised: In this paper, we assume the following five assumptions.

Original: The first one has already been adopted inseveral previous work [12, 19].
Revised: The first one has already been adopted in several previous work [12, 19].

Original: The other ones are standard in the nonconvex optimization literatureto guarantees second-order optimality.
Revised: The other ones are standard in the nonconvex optimization literature to guarantees second-order optimality.

Original: Assumption 1 (Hessian heterogeneity [12, 19]) .
Revised: Assumption 1 (Hessian heterogeneity [12, 19]) .

Original: { f p } Pp =1 is second-order ζ -heterogeneous, i.e., forany p, p ′ ∈
Revised: { f p } Pp =1 is second-order ζ -heterogeneous, i.e., for any p, p ′ ∈

Original: [ P ] , (cid:13)(cid:13) ∇ 2 f p ( x ) − ∇ 2 f p ′ ( x ) (cid:13)(cid:13) ≤ ζ, ∀ x ∈ R d .
Revised: [ P ] , [Equation]

Original: Assumption 1 characterizes the heterogeneity of local objectives { f p } Pp =1 in terms of Hessians and hasa important role in our analysis.
Revised: Assumption 1 characterizes the heterogeneity of local objectives { f p } Pp =1 in terms of Hessians and has a important role in our analysis.

Original: Intuitively, we expect that relatively small heterogeneity parameter ζto the smoothness parameter L (defined in Assumption 2) reduces the necessary number of communication rounds to optimize the global objective.
Revised: Intuitively, we expect that relatively small heterogeneity parameter ζ to the smoothness parameter L (defined in Assumption 2) reduces the necessary number of communication rounds to optimize the global objective.

Original: Especially when the local objectives are identical, i.e.,
Revised: Especially when the local objectives are identical, i.e.,

Original: D p
Revised: D p

Original: = D p ′ for every p, p ′ ∈
Revised: = D p ′ for every p, p ′ ∈

Original: [ P ] , ζ becomes zero.
Revised: [ P ] , ζ becomes zero.

Original: When each D p is the empirical distribution of n/P
Revised: When each D p is the empirical distribution of n/P

Original: IID samples from common data distribution D , we have ∥∇ 2 f p ( x ) − ∇ 2 f p
Revised: IID samples from common data distribution D , we have ∥∇ 2 f p ( x ) − ∇ 2 f p

Original: ( x ) ∥ ≤ (cid:101)
Revised: ( x ) ∥ ≤ (cid:101)

Original: Θ( (cid:112) P/nL )with high probability by matrix Hoeffding’s inequality under Assumption 2 for fixed x .
Revised: Θ( (cid:112) P/nL ) with high probability by matrix Hoeffding’s inequality under Assumption 2 for fixed x .

Original: Hence, intraditional distributed learning regimes, Assumption 1 naturally holds.
Revised: Hence, in traditional distributed learning regimes, Assumption 1 naturally holds.

Original: An important remark is that
Revised: An important remark is that

Original: Assumption 2 implies ζ ≤ 2 L , i.e., the heterogeneity is bounded by the smoothness .
Revised: Assumption 2 implies ζ ≤ 2 L , i.e., the heterogeneity is bounded by the smoothness .

Original: Even in federatedlearning regimes, we expect ζ ≪ 2 L for some problems practically.
Revised: Even in federated learning regimes, we expect ζ ≪ 2 L for some problems practically.

Original: Assumption 2 (Gradient Lipschitzness) .
Revised: Assumption 2 (Gradient Lipschitzness) .

Original: ∀ p ∈
Revised: ∀ p ∈

Original: [ P ] , z ∈ supp( D p ) , ℓ ( · , z ) is L -gradient Lipschitz.
Revised: [ P ] , z ∈ supp( D p ) , ℓ ( · , z ) is L -gradient Lipschitz.

Original: Assumption 3 (Existence of global optimum) .
Revised: Assumption 3 (Existence of global optimum) .

Original: f has a global minimizer x ∗ ∈ R d .
Revised: f has a global minimizer x ∗ ∈ R d .

Original: Assumption 4 (Hessian Lipschitzness) .
Revised: Assumption 4 (Hessian Lipschitzness) .

Original: ∀ p ∈
Revised: ∀ p ∈

Original: [ P ] , z ∈ supp( D p ) , ℓ ( · , z ) is ρ -Hessian Lipschitz.
Revised: [ P ] , z ∈ supp( D p ) , ℓ ( · , z ) is ρ -Hessian Lipschitz.

Original: Assumption 5 (Bounded stochastic gradient) .
Revised: Assumption 5 (Bounded stochastic gradient) .

Original: ∀ p ∈
Revised: ∀ p ∈

Original: [ P ] , z ∈ supp( D p ) , ∇ ℓ ( · , z ) is G -bounded, i.e.,∥∇ ℓ ( x, z ) ∥ ≤ G, ∀ x ∈ R d .
Revised: [ P ] , z ∈ supp( D p ) , ∇ ℓ ( · , z ) is G -bounded, i.e., ∥∇ ℓ ( x, z ) ∥ ≤ G, ∀ x ∈ R d .

Original: In our analysis, G has no significant impact because G only depends on our theoretical communicationcomplexity in logarithmic order.
Revised: In our analysis, G has no significant impact because G only depends on our theoretical communication complexity in logarithmic order.

Original:  Main Ideas and Proposed Algorithm
Revised:  Main Ideas and Proposed Algorithm

Original: Our proposed algorithm is based on a natural combination of (i) Bias-Variance Reduced (BVR)estimator ; and (ii) parameter perturbation at each local update .
Revised: Our proposed algorithm is based on a natural combination of (i) Bias-Variance Reduced (BVR) estimator ; and (ii) parameter perturbation at each local update .

Original: The first idea has been proposedby [19] to find first-order stationary points with small communication complexity.
Revised: The first idea has been proposed by [19] to find first-order stationary points with small communication complexity.

Original: The secondone is a well-known approach to find second-order stationary points in non-distributed nonconvexoptimization [4, 8, 9].
Revised: The second one is a well-known approach to find second-order stationary points in non-distributed nonconvex optimization [4, 8, 9].

Original: In this section, we illustrate these two ideas and provide its concrete procedures.
Revised: In this section, we illustrate these two ideas and provide its concrete procedures.

Original:  Review of BVR Estimator 
Revised: 3.1 Review of BVR Estimator [19]

Original: The bias-variance reduced estimator aims to efficiently findfirst-order stationary points by simultaneously reducing the bias caused by local gradient descentsteps and the variance caused by stochastization of the used gradients.
Revised: The bias-variance reduced estimator aims to efficiently find first-order stationary points by simultaneously reducing the bias caused by local gradient descent steps and the variance caused by stochastization of the used gradients.

Original: First we consider why the standard local SGD is not sufficient to achieve fast convergence andsometimes slower than minibatch SGD.
Revised: First we consider why the standard local SGD is not sufficient to achieve fast convergence and sometimes slower than minibatch SGD.

Original: Recall that in local SGD each worker takes the update [-rulesof x ( p ) k +
 =-]
Revised: Recall that in local SGD each worker takes the update [-rulesof x ( p ) k +
 =-]rules

Original: x ( p ) k − ηg ( p ) k for k ∈

Original: [ B /b ] in each communication round, where g ( p ) k is a stochasticgradient with minibatch size b at x ( p ) k on local dataset D p and B is given local computation budget.

Original: In typical convergence analysis, we need to bound the expected deviation of g ( p ) k from ideal globalgradient ∇ f ( x k ) , that is E ∥

Original: g

Original: [Equation] p [Equation] The former term is called bias and the latter one is called variance .
Revised: 1: Add noise x 0 = (cid:101) x 0 + ηξ − 1 , where ξ − 1 ∼ Unif( B dr ) .

Original: A typical assumption to boundthe first term is bounded gradient heterogeneity assumption , that requires ∥∇ f p ( x ) − ∇ f ( x ) ∥

Original: ≤ ζ 1for every x ∈ R d and p ∈

Original: [ P ] .

Original: Under this assumption, the first term is only bounded by ζ 1 , that isa constant.

Original: The second term is typically bounded by σ 2 /b for g ( p ) k with minibatch size b , when the

Original: 1: for s = 0 to S − 1 do 2: for p = 1 to P in  do 3: v ( p ) I (0 , 0 ,s ) = ∇ f p ( x I (0 , 0 ,s ) ) .
Revised: 2: for s = 0 to S − 1 do 3: for p = 1 to P in parallel do 4: v ( p ) I (0 , 0 ,s ) = ∇ f p ( x I (0 , 0 ,s ) ) .

Original: 4: end for 5:
Revised: 5: end for 6:

Original: Communicate { v ( p ) I (0 , 0 ,s ) } Pp =1 .
Revised: Communicate { v ( p ) I (0 , 0 ,s ) } Pp =1 .

Original: Set v I (0 , 0 ,s )
Revised: Set v I (0 , 0 ,s )

Original: = 1 P (cid:80)
Revised: = 1 P (cid:80)

Original: Pp =1 v ( p ) I (0 , 0 ,s ) .
Revised: Pp =1 v ( p ) I (0 , 0 ,s ) .

Original: 6: for t = 0 to T − 1 do 7: for p = 1 to P in parallel do 8: g ( p ) I (0 ,t,s )
Revised: 7: for t = 0 to T − 1 do 8: for p = 1 to P in parallel do 9: g ( p ) I (0 ,t,s )

Original: = 1 Kb (cid:80)
Revised: = 1 Kb (cid:80)

Original: Kbl =1 ∇ ℓ ( x I (0 ,t,s ) , z l,I (0 ,t,s ) ) , 9: g ( p ) , ref I (0 ,t,s )
Revised: Kbl =1 ∇ ℓ ( x I (0 ,t,s ) , z l,I (0 ,t,s ) ) , 10: g ( p ) , ref I (0 ,t,s )

Original: = 1 Kb (cid:80)
Revised: = 1 Kb (cid:80)

Original: Kbl =1 ∇ ℓ ( x I (0 ,t − 1 ,s ) , z l,I (0 ,t,s ))
Revised: Kbl =1 ∇ ℓ ( x I (0 ,t − 1 ,s ) , z l,I (0 ,t,s ))

Original: ( z l,I (0 ,t,s ) i.i.d.
Revised: ( z l,I (0 ,t,s ) i.i.d.

Original: ∼ D p ).
Revised: ∼ D p ).

Original: 10: v ( p ) I (0 ,t,s )
Revised: 11: v ( p ) I (0 ,t,s )

Original: = 1 t ≥ 1 ( g ( p ) I (0 ,t,s ) − g ( p ) , ref I (0 ,t,s ) + v ( p ) I (0 ,t − 1 ,s ) )
Revised: = 1 t ≥ 1 ( g ( p ) I (0 ,t,s ) − g ( p ) , ref I (0 ,t,s ) + v ( p ) I (0 ,t − 1 ,s ) )

Original: + 1 t =0 v ( p ) I (0 , 0 ,s ) .
Revised: + 1 t =0 v ( p ) I (0 , 0 ,s ) .

Original: 11: end for 12:
Revised: 12: end for 13:

Original: Communicate { v ( p ) I (0 ,t,s ) } Pp =1 .
Revised: Communicate { v ( p ) I (0 ,t,s ) } Pp =1 .

Original: Set v I (0 ,t,s )
Revised: Set v I (0 ,t,s )

Original: = 1 P (cid:80)
Revised: = 1 P (cid:80)

Original: Pp =1 v ( p ) I (0 ,t,s ) .
Revised: Pp =1 v ( p ) I (0 ,t,s ) .

Original: Randomly select pt,s ∼ Unif[ P ] . # Only worker p t,s runs local optimization.
Revised: Randomly select p worker p t,s runs local optimization.

Original: 14: for k = 0 to K − 1 do 15: g I ( k,t,s ) = 1 b (cid:80) bl =1 ∇ ℓ ( x I ( k,t,s ) , z l,I ( k,t,s ) ) , 16: g ref I ( k,t,s{+⌈ √ K ⌉ b + 1 k ̸≡ 
 (mod ⌈ √ K ⌉+} ) 
Revised: 15: for k = 0 to K − 1 do 16: b k = 1 k ≡ 0 (mod ⌈ √ K ⌉ ) {+⌈ √ K ⌉ b + 1 k ̸≡ 
 (mod ⌈ √ K ⌉+} ) b .

Original:  = 1 b  (cid:80) b  l =1 ∇ ℓ ( x I ( k − 1 ,t,s ) ) , z l,I ( k,t,s ) )
Revised: 17: g I ( k,t,s ) = 1 b k (cid:80) b k l =1 ∇ ℓ ( x I ( k,t,s ) , z l,I ( k,t,s ) ) , 18: g ref I ( k,t,s ) = 1 b k (cid:80) b k l =1 ∇ ℓ ( x I ( k − 1 ,t,s ) ) , z l,I ( k,t,s ) )

Original: ( z l,I ( k,t,s ) i.i.d.
Revised: ( z l,I ( k,t,s ) i.i.d.

Original: ∼ D p t,s ).
Revised: ∼ D p t,s ).

Original: 17: v I ( k,t,s )
Revised: 19: v I ( k,t,s )

Original: = 1 k ≥ 1 ( g I ( k,t,s ) − g ref I ( k,t,s )
Revised: = 1 k ≥ 1 ( g I ( k,t,s ) − g ref I ( k,t,s )

Original: + v I ( k − 1 ,t,s ) )
Revised: + v I ( k − 1 ,t,s ) )

Original: + 1 k =

Revised: + 1 k =

Original: v I (0 ,t,s ) .
Revised: v I (0 ,t,s ) .

Original: Update (cid:101) x I ( k +1 ,t,s )
Revised: Update (cid:101) x I ( k +1 ,t,s )

Original: =
Revised: =

Original: x I ( k,t,s ) − ηv I ( k,t,s ) .
Revised: x I ( k,t,s ) − ηv I ( k,t,s ) .

Original: Add noise x I ( k +1 ,t,s )
Revised: Add noise x I ( k +1 ,t,s )

Original: = (cid:101) x
Revised: = (cid:101) x

Original: I ( k +1 ,t,s ) + ηξ I ( k,t,s ) , where ξ I ( k,t,s ) ∼ Unif( B dr ) .
Revised: I ( k +1 ,t,s ) + ηξ I ( k,t,s ) , where ξ I ( k,t,s ) ∼ Unif( B dr ) .

Original: 20: end for 21:
Revised: 22: end for 23:

Original: Communicate x I (0 ,t +1 ,s ) .
Revised: Communicate x I (0 ,t +1 ,s ) .

Original: 22: end for 23: end forvariance of a single stochastic gradient is bounded by σ 2 .{+x ( p ) k +
 = x ( p ) k − ηg ( p ) k for k ∈+}
Revised: 24: end for 25: end for of {+x ( p ) k +
 = x ( p ) k − ηg ( p ) k for k ∈+}

Revised: [ B /b ] in each communication round, where g ( p ) k is a stochastic gradient with minibatch size b at x ( p ) k on local dataset D p and B is given local computation budget.

Revised: In typical convergence analysis, we need to bound the expected deviation of g ( p ) k from ideal global gradient ∇ f ( x k ) , that is E

Revised: [Equation]

Revised: The former term is called bias and the latter one is called variance .

Revised: A typical assumption to bound the first term is bounded gradient heterogeneity assumption , that requires ∥∇ f p ( x ) − ∇ f ( x ) ∥ ≤ ζfor every x ∈ R d and

Revised: p ∈

Revised: [ P ] .

Revised: Under this assumption, the first term is only bounded by ζ 1 , that is a constant.

Revised: The second term is typically bounded by σ 2 /b for g ( p ) k with minibatch size b , when the variance of a single stochastic gradient is bounded by σ 2 .

Original: These facts show that the bias is still aconstant and does not vanish even if minibatch size b is enhanced and the variance vanishes.
Revised: These facts show that the bias is still a constant and does not vanish even if minibatch size b is enhanced and the variance vanishes.

Original: This iswhy local SGD can be worse than minibatch SGD when ζ 1 is not too small.
Revised: This is why local SGD can be worse than minibatch SGD when ζ 1 is not too small.

Original: Also, we can see thatthe variance is still a constant for fixed minibatch size b and this is a common reason why minibatch
Revised: Also, we can see that the variance is still a constant for fixed minibatch size b and this is a common reason why minibatch

Original: SGD and local SGD only show slow convergences.
Revised: SGD and local SGD only show slow convergences.

Original: These observations give critical motivations ofthe simultaneous reduction of the bias and variance.
Revised: These observations give critical motivations of the simultaneous reduction of the bias and variance.

Original: The bias-variance reduced estimator v ( p ) k is defined as v ( p [Equation] k 
Revised: The bias-variance reduced estimator v ( p ) k is defined as v ( p ) k :=

Original: [Equation]
Revised: (1 /b ) (cid:80) bl =1 ( ∇ ℓ ( x ( p ) k , z l ) − ∇ ℓ ( x 0 , z l ))

Original: ∇ ℓ ( x 0 , z l )) + ∇ f ( x 0 ) (SVRG version).
Revised: + ∇ f ( x 0 ) (SVRG version).

Original: It is known that the bias caused by localization canbe bounded by ζ ∥ x ( p ) k − x 0 ∥ and the variance caused by stochastization can be bounded by( L 2 /b )
Revised: It is known that the bias caused by localization can be bounded by ζ ∥ x ( p ) k − x 0 ∥ and the variance caused by stochastization can be bounded by ( L 2 /b )

Original: ∥ x ( p )
Revised: ∥ x ( p )

Original: k − x 0 ∥ 2 , where ζ is the Hessian heterogeneity of { f p } and L is the smoothness ofℓ .
Revised: k − x 0 ∥ 2 , where ζ is the Hessian heterogeneity of { f p } and L is the smoothness of ℓ .

Original: This implies that both the bias and variance of v ( p ) k converges to zero as x ( p ) k and x 0 go to x ∗ .
Revised: This implies that both the bias and variance of v ( p ) k converges to zero as x ( p ) k and x 0 go to x ∗ .

Original: Inother words, bias-variance reduced estimator v ( p ) k is asymptotically consistent to the global gradient∇ f ( x ( p ) k ) by using periodically computed global full gradients ∇ f ( x 0 ) .
Revised: In other words, bias-variance reduced estimator v ( p ) k is asymptotically consistent to the global gradient ∇ f ( x ( p ) k ) by using periodically computed global full gradients ∇ f ( x 0 ) .

Original: We actually adopt SARAHversion of BVR estimator as in [19] rather than SVRG one due to its theoretical advantages.
Revised: We actually adopt SARAH version of BVR estimator as in [19] rather than SVRG one due to its theoretical advantages.

Original:  Parameter Perturbation at Local Updates.
Revised: 3.2 Parameter Perturbation at Local Updates

Original: Although the bias-variance reduced estimator is usefulto guarantee first-order optimality with small communication complexity in noncovex optimization,the algorithm often gets stuck at saddle points.
Revised: Although the bias-variance reduced estimator is useful to guarantee first-order optimality with small communication complexity in noncovex optimization, the algorithm often gets stuck at saddle points.

Original: To tackle this problem, we borrow the ideas ofescaping saddle points in non-distributed nonconvex optimization.
Revised: To tackle this problem, we borrow the ideas of escaping saddle points in non-distributed nonconvex optimization.

Original: Particularly, to efficiently findsecond-order optimal points, we utilize parameter perturbation .
Revised: Particularly, to efficiently find second-order optimal points, we utilize parameter perturbation .

Original: Parameter perturbation is a familiarapproach in non-distributed nonconvex optimization.
Revised: Parameter perturbation is a familiar approach in non-distributed nonconvex optimization.

Original: Specifically, Jin et al.
Revised: Specifically, Jin et al.

Original: [8, 9] have considered theupdate rule of x k +

Revised: [8, 9] have considered the update rule of x k +

Original: = x
Revised: = x

Original: k − η ∇ f ( x k ) + ηξ k , where ξ k ∼ Unif( B d r ) for some small radius r .
Revised: k − η ∇ f ( x k ) + ηξ k , where ξ k ∼ Unif( B dr ) for some small radius r .

Original: Thisalgorithm is called Perturbed GD (PGD) or Noisy GD.
Revised: This algorithm is called Perturbed GD (PGD) or Noisy GD.

Original: Similar to this formulation, we add noise ateach local update, i.e., x ( p )
Revised: Similar to this formulation, we add noise at each local update, i.e., x ( p )

Original: k
Revised: k

Original: +1 =
Revised: +1 =

Original: (cid:101) x
Revised: (cid:101) x

Original: ( p ) k
Revised: ( p ) k

Original: +1 + ηξ ( p ) k , where (cid:101) x ( p ) k
Revised: +1 + ηξ ( p ) k , where (cid:101) x ( p ) k

Original: +

Revised: +

Original: =
Revised: =

Original: x ( p ) k − ηv ( p ) k .
Revised: x ( p ) k − ηv ( p ) k .

Original: The intuition behind thenoise addition is that random noise has some components along the negative curvature directions ofthe global objective around the saddle point, and we expect that noise addition helps the parameterproceed to the decreasing directions of f and escape the saddle points.
Revised: The intuition behind the noise addition is that random noise has some components along the negative curvature directions of the global objective around the saddle point, and we expect that noise addition helps the parameter proceed to the decreasing directions of f and escape the saddle points.

Original: Remark (Necessity of local perturbation) .
Revised: Necessity of local perturbation.

Original: Perturbing the global model at the server side is an intuitiveway, but not sufficient for communication efficiency when we want to utilize small heterogeneity ofthe local datasets (i.e., ζ ≪ L ).
Revised: Perturbing the global model at the server side is an intuitive way, but not sufficient for communication efficiency when we want to utilize small heterogeneity of the local datasets (i.e., ζ ≪ L ).

Original: The bias-variance reduced estimator with local perturbation enablesto escape multiple global saddle points in local optimization and achieves second-order optimalitywith communication complexity (cid:101) Θ( ζ/ε 2 ) for sufficiently large B .
Revised: The bias-variance reduced estimator with local perturbation enables to escape multiple global saddle points in local optimization and achieves second-order optimality with communication complexity (cid:101) Θ( ζ/ε 2 ) for sufficiently large B .

Original: In contrast, perturbing the globalparameter at the server side only ensures to escape single global saddle point at each round and onlyachieves communication complexity of (cid:101) Θ( L/ε 2 ) .
Revised: In contrast, perturbing the global parameter at the server side only ensures to escape single global saddle point at each round and only achieves communication complexity of (cid:101) Θ( L/ε 2 ) .

Original: This is the reason why local perturbation ratherthan global one is adopted.
Revised: This is the reason why local perturbation rather than global one is adopted.

Original:  Concrete Procedures.
Revised: 3.3 Concrete Procedures

Original: The full description of our proposed Bias-Variance Reduced Local Perturbed SGD (BVR-L-PSGD) is given in Algorithm 1.
Revised: The full description of our proposed Bias-Variance Reduced Local Perturbed SGD (BVR-L-PSGD) is given in Algorithm 1.

Original: When we set the noise size r = 0 , Algorithm 1essentially matches BVR-L-SGD.
Revised: When we set the noise size r = 0 , Algorithm 1 essentially matches BVR-LSGD.

Original: Additionally setting K = 1 , Algorithm 1 matches SARAH.
Revised: Additionally setting K = 1 , Algorithm 1 matches SARAH.

Original: Thealgorithm requires Θ( ST ) communication rounds.
Revised: The algorithm requires Θ( ST ) communication rounds.

Original: At each communication round, each workercomputes large batch stochastic gradients and the server constructs v I (0 ,t,s ) by aggregating them.v
Revised: At each communication round, each worker computes large batch stochastic gradients and the server constructs v I (0 ,t,s ) by aggregating them.

Original:  I (0 ,t,s ) is used as an estimator of ∇ f ( x I (0 ,t,s ) ) to reduce computational cost.
Revised: v I (0 ,t,s ) is used as an estimator of ∇ f ( x I (0 ,t,s ) ) to reduce computational cost.

Original: In line 14-21, we242 randomly select worker p t,s and only worker p t,s runs local optimization as described above.
Revised: In line 14-21, we randomly select worker p t,s and only worker p t,s runs local optimization as described above.

Original: Inthe local optimization, we use SARAH like bias variance reduced estimator (line 16-18) rather than SVRG one and add noise (line 20) at each local update.
Revised: In the local optimization, we use SARAH like bias variance reduced estimator (line 16-18) rather than SVRG one and add noise (line 20) at each local update.

Original:  Convergence Analysis
Revised:  Convergence Analysis

Original: In this section, we provide convergence theory of BVR-L-PSGD (Algorithm 1).
Revised: In this section, we provide convergence theory of BVR-L-PSGD (Algorithm 1).

Original: All the omittedproofs are found in the supplementary material.
Revised: All the omitted proofs are found in the supplementary material.

Original: For simple presentations, we use (cid:101) Θ symbol to hidean extra poly-logarithmic factors that depend on L, ρ, G, K, b, T, S, 1 /ε, 1 /q , where q represents theconfidence parameter in high probability bounds.
Revised: For simple presentations, we use (cid:101) Θ symbol to hide an extra poly-logarithmic factors that depend on L, ρ, G, K, b, T, S, 1 /ε, 1 /q , where q represents the confidence parameter in high probability bounds.

Original: 4.1 Finding First-Order Stationary Points
Revised: 4.1 Finding First-Order Stationary Points

Original: First, we derive Descent Lemma for BVR-L-PSGD and first-order optimality guarantees by using it.
Revised: First, we derive Descent Lemma for BVR-L-PSGD and first-order optimality guarantees by using it.

Original: Proposition 4.1 (Descent Lemma) .
Revised: Proposition 4.1 (Descent Lemma) .

Original: Let S ∈ N
Revised: Let S ∈ N

Original: and I ( k, t, s ) ≥ I ( k 0 , t 0 , s 0 ) ∈
Revised: and I ( k, t, s ) ≥ I ( k 0 , t 0 , s 0 ) ∈

Original: [ KTS ] ∪ { 0 } .
Revised: [ KTS ] ∪ { 0 } .

Original: Suppose that Assumptions 1, 2, 3 and 5 hold.
Revised: Suppose that Assumptions 1, 2, 3 and 5 hold.

Original: Given q ∈ (0 , 1) , r > 0 , if we appropriately chooseη = (cid:101) Θ(

Revised: Given q ∈ (0 , 1) , r > 0 , if we appropriately choose η = (cid:101) Θ(

Original: /L ∧ 1 / ( Kζ ) ∧ (cid:112) b/K/L ∧ √ Pb/ ( √ KTL ))
Revised: /L ∧ 1 / ( Kζ ) ∧ (cid:112) b/K/L ∧ √ Pb/ ( √ KTL ))

Original: , it holds that
Revised: , it holds that

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − 3 q .
Revised: with probability at least 1 − 3 q .

Original: Here,
Revised: Here,

Original: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ) for some universal constant c η > 0 .
Revised: for some universal constant c η > 0 .

Original: From Proposition 4.1 with I ( k 0 , t 0 , s 0 )
Revised: From Proposition 4.1 with I ( k 0 , t 0 , s 0 )

Original: ← 0 and I ( k, t, s ) ← KTS gives the following corollary.
Revised: ← 0 and I ( k, t, s ) ← KTS gives the following corollary.

Original: Corollary 4.2.
Revised: Corollary 4.2.

Original: Suppose that Assumptions 1, 2, 3 and 5 hold.
Revised: Suppose that Assumptions 1, 2, 3 and 5 hold.

Original: Under the same setting as in Proposition4.3 and S ≥ Θ(( f ( x 0 ) − f ( x ∗ )) / ( ηKTε 2 )) , with probability at least 1 − 3 q , there exists
Revised: Under the same setting as in Proposition 4.3 and S ≥ Θ(( f ( x 0 ) − f ( x ∗ )) / ( ηKTε 2 )) , with probability at least 1 − 3 q , there exists

Original: i ∈
Revised: i ∈

Original: [ KTS − 1] ∪ { 0 } such that ∥∇ f
Revised: [ KTS − 1] ∪ { 0 } such that ∥∇ f

Original: ( (cid:101) x i ) ∥
Revised: ( (cid:101) x i ) ∥

Original: ≤ ε .
Revised: ≤ ε .

Original: Remark (Communication complexity) .
Revised: Remark (Communication complexity) .

Original: The total number of communication rounds Θ( TS ) becomes(cid:101) O (cid:16) T + (cid:16) L/K + ζ + L/ √ Kb + √ TL/ √ KPb (cid:17) ( f ( (cid:101) x 0 ) − f ( x ∗ )) /ε
Revised: The total number of communication rounds Θ( TS ) becomes (cid:101) O (cid:16) T + (cid:16) L/K + ζ + L/ √ Kb + √ TL/ √ KPb (cid:17) ( f ( (cid:101) x 0 ) − f ( x ∗ )) /ε

Original: 2 (cid:17) .
Revised: 2 (cid:17) .

Original: Given local computationbudget B , we set T :=
Revised: Given local computation budget B , we set T :=

Original: Θ(1 + n/ ( B P )) and Kb :=
Revised: Θ(1 + n/ ( B P )) and Kb :=

Original: Θ( B ) with b ≤ Θ( √B ) .
Revised: Θ( B ) with b ≤ Θ( √B ) .

Original: Then, we have theaveraged number of local computations per communication round Kb + n/ ( PT ) =
Revised: Then, we have the averaged number of local computations per communication round Kb + n/ ( PT ) =

Original: Θ( B ) and thecommunication complexity TS with budget B becomes(cid:101) O (cid:16) 1 + n B P +
Revised: Θ( B ) and the communication complexity TS with budget B becomes

Original: L √B ε 2 + √ nL
Revised: [Equation]

Original: B Pε 2 + ζε 2 (cid:17) , whichmatches the best known communication complexity [19].
Revised: which matches the best known communication complexity [19].

Original: 4.2 Escaping Saddle Points
Revised: 4.2 Escaping Saddle Points

Original: Next, we show that BVR-L-PSGD implicitly exploits the negative curvature of f around saddle pointsand efficiently escapes the saddle points by utilizing the asymptotic consistency of BVR estimatorand the parameter perturbation at each local update.
Revised: Next, we show that BVR-L-PSGD implicitly exploits the negative curvature of f around saddle points and efficiently escapes the saddle points by utilizing the asymptotic consistency of BVR estimator and the parameter perturbation at each local update.

Original: We rely on the technique of coupling sequence [9].
Revised: We rely on the technique of coupling sequence [9].

Original: Given saddle point (cid:101) x I ( k 0 ,t 0 ,s 0 )  , we define anew sequence { x ′ i } ∞ i= I ( k 0 ,t 0 ,s 0 ) that satisfies x I ( k 0 ,t 0 ,s 0 )
Revised: Given saddle point (cid:101) x I ( k 0 ,t 0 ,s 0 ) and ˆ I ≥ I ( k 0 , t 0 , s 0 ) , we define a new sequence { x ′ i } ∞ i

Original: − x ′ I ( k 0 ,t 0 ,s 0 ) 
Revised: = I ( k 0 ,t 0 ,s 0 ) as follows:

Original:  = [-
-]
Revised: (1) ⟨ ξ ′ (cid:101) I , e min ⟩ = −⟨ ξ (cid:101) I , e min ⟩ ; (2) ⟨ ξ ′ (cid:101) I , e j ⟩ = ⟨ ξ (cid:101) I , e j ⟩ for j ∈ { 2 , . . . , d } ; and (3) All the other randomness is completely same as the one of { x i } KTS − 1 i = [-
-].

Original: η ⟨ ξ  I( k 0 ,t 0 ,s 0 ) , e min ⟩ e min ,where e min is the normalized eigenvector associated with the (negative) minimum eigenvalue of H :=∇ 2 f ( (cid:101) 
Revised: Let r 0 := |⟨ ξ (cid:101) I , e min ⟩| . Note that |⟨ ξ (cid:101) I − ξ ′ (cid:101) I , e min ⟩| = 2 r 0 and thus ∥ ξ (cid:101) I − ξ ′ (cid:101) I ∥

Original: x I ( k 0 ,t 0 ,s 0 ) ) , and all the other randomness are completely same as the ones of { x i } ∞
Revised: = 2 r 0 .

Original: i{+Also, observe that x (cid:101) I +1 − x ′ (cid:101) I +
+} = 
Revised: {+Also, observe that x (cid:101) I +1 − x ′ (cid:101) I +
+} = η ⟨ ξ (cid:101) I − ξ ′ (cid:101)

Revised: I , e min ⟩ e min .

Revised: We define (cid:101) I used in the definition of coupling sequence as follows:

Original:  I ( k 0 ,t 0 ,s 0 )  .
Revised: [Equation] Here, k ′ 0 is the minimum index k that satisfies k > k 0 and k ≡ 0 (mod ⌈√ K ⌉ ) . We can easily check that (cid:101) I − I ( k 0 , t 0 , s 0 ) ≤ 1 / ( ηλ ) .

Original: Then, we show that either of the two sequences { x i } or { x ′ i } efficiently escapes the saddle pointsby bounding the norm of the cumulative difference of x i and x ′ i from below.
Revised: Then, we show that either of the two sequences { x i } or { x ′ i } efficiently escapes the saddle points by bounding the norm of the cumulative difference of x i and x ′ i from below.

Original: The novel and mostdifficult part of the analysis is to evaluate the norm of the cumulative difference of the deviations∥ (cid:80) J i
Revised: The novel and most difficult part of the analysis is to evaluate the norm of the cumulative difference of the deviations

Original: = I ( k 0 ,t 0 ,s 0 ) (1 − η H ) J− i ( v i − ∇ f ( x i ) − v ′
Revised: [Equation] 1 − η H ) J− i ( v i − ∇ f ( x i ) − v ′

Original: i + ∇ f ( x ′ i ))
Revised: i + ∇ f ( x ′ i ))

Original: ∥ generated by the two sequences, wherev ′ i denotes the BVR estimator at iteration i generated by sequence { x ′ i } .
Revised: ∥ generated by the two sequences, where v ′ i denotes the BVR estimator at iteration i generated by sequence { x ′ i } .

Original: Proposition 4.3 (Implicit Negative Curvature Exploitation) .
Revised: Proposition 4.3 (Implicit Negative Curvature Exploitation) .

Original: Let I ( k 0 , t 0 , s 

Revised: Let I ( k 0 , t 0 , s

Original: ) ∈
Revised: ) ∈

Original: [ KTS ] ∪ { 0 } .
Revised: [ KTS ] ∪ { 0 } .

Original: Suppose that Assumptions 1, 2, 3, 4 and 5 hold, ∥∇ f ( (cid:101) x
Revised: Suppose that Assumptions 1, 2, 3, 4 and 5 hold, ∥∇ f ( (cid:101) x

Original: I ( k 0 ,t 0 ,s 0 ) )
Revised: I ( k 0 ,t 0 ,s 0 ) )

Original: ∥ ≤ ε and the minimum eigenvalue λ minof H :=
Revised: ∥ ≤ ε and the minimum eigenvalue λ min of H :=

Original: ∇ f ( (cid:101) x
Revised: ∇ f ( (cid:101) x

Original: I ( k 0 ,t 0 ,s 0 ) )
Revised: I ( k 0 ,t 0 ,s 0 ) )

Original: satisfies λ := − λ min > √ ρε .
Revised: satisfies λ := − λ min > √ ρε .

Original: Under K = O ( L/ζ ∧ b ∧ Pb/T )  , if we appropriately choose J I ( k 0 ,t 0 ,s 0 )
Revised: Under b = Ω( K ∨ 1 / ( √ Kρε ) ∨ T/ ( PK )) , if we appropriately choose J I ( k 0 ,t 0 ,s 0 )

Original: = (cid:101) Θ(

Revised: = (cid:101) Θ(

Original: / ( ηλ )) , η = (cid:101)
Revised: / ( ηλ )) , η = (cid:101)

Original: Θ(1 /L
Revised: Θ(1 /L

Original: ∧ 1 / ( Kζ ) ∧ (cid:112) b/K/L ∧ √ Pb/ ( √ KT/L )) ,with F I ( k 0 ,t 0 ,s 0 ) :
Revised: ∧ 1 / ( Kζ ) ∧ (cid:112) b/K/L ∧ √ Pb/ ( √ KT/L )) , with F I ( k 0 ,t 0 ,s 0 ) :

Original: =
Revised: =

Original: c F η J I ( k 0 ,t 0 ,s 0 ) r 2 and r :=
Revised: c F η J I ( k 0 ,t 0 ,s 0 ) r 2 and r :=

Original: c r ε ( c F = Θ(1) and c r = (cid:101) Θ(1) )
Revised: c r ε ( c F = Θ(1) and c r = (cid:101) Θ(1) )

Original: we havef ( x I ( k 0 ,t 0 ,s 0 )
Revised: we have f ( x I ( k 0 ,t 0 ,s 0 )

Original: + J I
Revised: + J I

Original: ( k 0 ,t 0 ,sp ) )
Revised: ( k 0 ,t 0 ,sp ) )

Original: − f ( x I ( k 0 ,t 0 s 0 ) )
Revised: − f ( x I ( k 0 ,t 0 s 0 ) )

Original: ≤ − F I ( k 0 ,t 0 ,s 0 )
Revised: ≤ − F I ( k 0 ,t 0 ,s 0 )

Original: + Rwith probability at least 1 / 2 − 9 q/ 2 .
Revised: + Rwith probability at least 1 / 2 − 9 q/ 2 .

Original: Here, R 2 := 2 c η η J
Revised: Here, R 2 := 2 c η η J

Original: I
Revised: I

Original: ( k 0 ,t 0 ,s 0)
Revised: ( k 0 ,t 0 ,s 0)

Original: ∧ K K (cid:80) I ( k 0 ,t 0 ,s 0 )
Revised: ∧ K K (cid:80) I ( k 0 ,t 0 ,s 0 )

Original: − 1 i
Revised: − 1 i

Original: = I (0 ,t 0 ,s 0 )
Revised: = I (0 ,t 0 ,s 0 )

Original: ∥ x i +1 − x i ∥ 2 + 
Revised: ∥ x i +1 − x i ∥ 2 + 2 c

Original: [Equation] (cid:80) [Equation]
Revised: η η J I ( k 0 ,t 0 ,s 0)

Original: − x
Revised: ∧ KT

Original: [Equation]
Revised: [Equation]

Original: ∥ 2 for some universal constant c η > 0 .

Original: Proposition 4.3 says the function value decreases by roughly F I ( k 0 ,t 0 ,s 0 ) and the global model escapessaddle points with probability at least 1 / 2 after J I
Revised: Proposition 4.3 says the function value decreases by roughly F I ( k 0 ,t 0 ,s 0 ) and the global model escapes saddle points with probability at least 1 / 2 after J I

Original: ( k 0 ,t 0 ,s 0 ) local steps.
Revised: ( k 0 ,t 0 ,s 0 ) local steps.

Original: 4.3 Finding Second-Order Stationary Points
Revised: 4.3 Finding Second-Order Stationary Points

Original: In this subsection, we derive final theorem that guarantees the second-order optimality of the globalmodel by combining Propositions 4.1 and 4.3.
Revised: In this subsection, we derive final theorem that guarantees the second-order optimality of the global model by combining Propositions 4.1 and 4.3.

Original: Theorem 4.4 (Final Theorem) .
Revised: Theorem 4.4 (Final Theorem) .

Original: Suppose that Assumptions 1, 2, 3, 4 and 5 hold.
Revised: Suppose that Assumptions 1, 2, 3, 4 and 5 hold.

Original: Under K = O ( L/ζ ∧ b ∧ Pb/T )  , if we appropriately choose η = (cid:101) Θ(
 
Revised: Under b = Ω( K ∨ 1 / ( √ Kρε ) ∨ T/ ( PK )) , if we appropriately choose η = (cid:101) Θ(
 /L

Original: /L ∧ 1 / ( Kζ ) ∧ (cid:112) b/K/L ∧√ Pb/ ( √ KTL )) ,r = (cid:101) Θ( ε ) and S = Θ(

Revised: ∧ 1 / ( Kζ ) ∧ (cid:112) b/K/L ∧ √ Pb/ ( √ KTL )) , r = (cid:101) Θ( ε ) and S = Θ(

Original: + ( f ( (cid:101) x 0 ) − f ( x ∗ )) / ( ηKTε 2 )) , with probability at least 1 / 2 , there existsi ∈ [ KTS ] ∪ { 0 } such that (cid:101) x
Revised: + ( f ( (cid:101) x 0 ) − f ( x ∗ )) / ( ηKTε 2 )) , with probability at least 1 / 2 , there exists i ∈ [ KTS ] ∪ { 0 } such that (cid:101) x

Original: i is ε -second-order optimal point of f 4 .
Revised: i is ε -second-order optimal point of f 6 .

Original: Remark (High probability bound) .
Revised: Remark (High probability bound) .

Original: Theorem 4.4 guarantees that Algorithm 1 finds an approximatesecond-order optimal point in KTS iterations with probability at least 1 / 2 .
Revised: Theorem 4.4 guarantees that Algorithm 1 finds an approximate second-order optimal point in KTS iterations with probability at least 1 / 2 .

Original: Repeating Algorithm 1log 2 (1 /q ) times guarantees that the same statement holds with probability at least 1 − q .
Revised: Repeating Algorithmlog 2 (1 /q ) times guarantees that the same statement holds with probability at least 1 − q .

Original: Remark (Communication complexity) .
Revised: Remark (Communication complexity) .

Original: The total number of communication roundsΘ( TS) is given by(cid:101) O (cid:16) T + (cid:16) L/K + ζ + L/ √ Kb + √ TL/ √ KPb (cid:17) ( f ( (cid:101) x 0 ) − f ( x ∗ )) 
Revised: The total number of communication rounds TS is given by (cid:101) O (cid:16) T + (cid:16) L/K + ζ + L/ √ Kb + √ TL/ √ KPb (cid:17) ( f ( (cid:101) x 0 ) − f ( x ∗ )) /ε

Original: /ε 2 (cid:17) .
Revised: 2 (cid:17) .

Original: Given local computationbudget B , we set T :=
Revised: Given local computation budget B , we set T :=

Original: Θ(1 + n/ ( B P )) and Kb :=
Revised: Θ(1 + n/ ( B P )) and Kb :=

Original: Θ( B ) with b ≤ Θ( √B ) .
Revised: Θ( B ) with b ≤ Θ( √B ) .

Original: Then, we have theaveraged number of local computations per communication round Kb + n/ ( PT ) =
Revised: Then, we have the averaged number of local computations per communication round Kb + n/ ( PT ) =

Original: Θ( B ) and thecommunication complexity  TS  with budget B becomes (cid:101) O (cid:16) 1 + n B P +
Revised: Θ( B ) and the communication complexity Θ( TS ) with budget B becomes (cid:101) O (cid:16) 1 + n B P +

Original: L √B ε 2 + √ nL
Revised: L √B ε 2 + √ nL

Original: B Pε 2 + ζε 2 (cid:17) .
Revised: B Pε 2 + ζε 2 (cid:17) .

Revised: This implies that for ζ

Original: Thisimplies that for ζ = o ( L ) the communication complexity is strictly smaller than the one of minibatch SGD (cid:101)O
Revised: = o ( L ) the communication complexity is strictly smaller than the one of minibatch SGD (cid:101)

Original:  (1 + L/ε 2 + G 2 / ( B Pε 4 )) .
Revised: O (1 + L/ε 2 + G 2 / ( B Pε 4 )) .

Original: Note that the rate matches to the one of BVR-L-SGD [19].
Revised: Note that the rate matches to the one of BVR-L-SGD [19].

Original: Hence, our method finds second-order optimal points without hurting communication efficiency ofthe state-of-the-art first-order optimality guaranteed method.
Revised: Hence, our method finds second-order optimal points without hurting communication efficiency of the state-of-the-art first-order optimality guaranteed method.

Original: Furthermore, when B → ∞ , we have(cid:101) Θ(1{+have (cid:101) Θ(
+} + ζ/ε 2 ) , that goes to (cid:101) Θ(1) as ζ → 0 .
Revised: Furthermore, when B → ∞ , we {+have (cid:101) Θ(
+} + ζ/ε 2 ) , that goes to (cid:101) Θ(1) as ζ → 0 .

Original: In summary, BVR-L-PSGD enjoys the desirable properties (i) and (ii) described in Section 1.[Figure]
Revised: In summary, BVR-L-PSGD enjoys the desirable properties (i) and (ii) described in Section 1.

Original:  Numerical Resutls
Revised:  Numerical Resutls

Original: In this section, we give some experimental results to verify our theoretical findings.
Revised: In this section, we give some experimental results to verify our theoretical findings.

Original: Data Preparation.
Revised: Data Preparation.

Original: We artificially generated heterogeneous local datasets from CIFAR10 dataset( https://www.cs.toronto.edu/~kriz/cifar.html ).
Revised: We artificially generated heterogeneous local datasets from CIFAR10 7 dataset.

Original: The data preparation procedure is completely in accordance with [19] and the details are found in [19].
Revised: The data preparation procedure is completely in accordance with [19] and the details are found in [19].

Original: We set homogeneity parameter q to0 .
Revised: We set homogeneity parameter q to 0 .

Original: 35 , which captures how similar the local datasets are ( q = 0 .
Revised: 35 , which captures how similar the local datasets are ( q = 0 .

Original: 1 corresponds to I.I.D. case and higherq does to higher heterogeneity).
Revised: 1 corresponds to I.I.D. case and higher q does to higher heterogeneity).

Original: Model.
Revised: Model.

Original: We conducted our experiments using a two-hidden layers fully connected neural networkwith 100 hidden units and softplus activation.
Revised: We conducted our experiments using a two-hidden layers fully connected neural network with 100 hidden units and softplus activation.

Original: For loss function, we used the standard cross-entropyloss.
Revised: For loss function, we used the standard cross-entropy loss.

Original: We initialized parameters by uniformly sampling the parameters from [ − 1 / 100 , 1 / 100] .
Revised: We initialized parameters by uniformly sampling the parameters from [ − 1 / 100 , 1 / 100] .

Original: Implemented Algorithms.
Revised: Implemented Algorithms.

Original: Minibatch SGD, Noisy Minibatch SGD, BVR-L-SGD [19] and ourproposed BVR-L-PSGD were implemented.
Revised: Minibatch SGD, Noisy Minibatch SGD, BVR-L-SGD [19] and our proposed BVR-L-PSGD were implemented.

Original: We set K = 64 and b = 16 for the latter two methodsand K = 1 and b = 1 , 024 for the former two methods.
Revised: We set K = 64 and b = 16 , and thus B = 1024 .

Original: For Noisy Minibatch SGD and BVR-LPSGD, the noise radius were tuned from r ∈ { 0 . 5 , 2 . 5 , 12 . 5 } .
Revised: For BVR-L-PSGD, the noise radius was tuned from r ∈ { 0 . 5 , 2 . 5 , 12 . 5 } .

Original: For each algorithm, we tuned learningrate η from { 0 . 005 , 0 . 01 , 0 . 05 , 0 . 1 , 0 . 5 , 1 . 0 } .
Revised: For each algorithm, we tuned learning rate η from { 0 . 005 , 0 . 01 , 0 . 05 , 0 . 1 , 0 . 5 , 1 . 0 } .

Original: The details of the tuning procedure are found in thesupplementary material.
Revised: The details of the tuning procedure are found in the supplementary material.

Original: Evaluation.
Revised: Evaluation.

Original: We compared the implemented algorithms using six criteria of train gradient norm∥∇ f ( x ) ∥ ; train loss; train accuracy; test gradient norm; test loss and test accuracy against the numberof communication rounds.
Revised: We compared the implemented algorithms using six criteria of train gradient norm ∥∇ f ( x ) ∥ ; train loss; train accuracy; test gradient norm; test loss and test accuracy against the number of communication rounds.

Original: The total number of communication rounds was fixed to 1 , 000 for eachalgorithm.
Revised: The total number of communication rounds was fixed to 1 , 000 for each algorithm.

Original: We independently repeated the experiments 5 times and report the mean and standarddeviation of the above criteria.
Revised: We independently repeated the experiments 5 times and report the mean and standard deviation of the above criteria.

Original: Due to the space limitation, we will only report train gradient norm,train loss and test accuracy in the main paper.
Revised: Due to the space limitation, we will only report train gradient norm, train loss and test accuracy in the main paper.

Original: The full results are found in the supplementary material.
Revised: The full results are found in the supplementary material.

Original: Results.
Revised: Results.

Original: Figure 1 shows the performances of the previous methods and our proposed algorithm. 
Revised: Figure 1 shows the performances of BVR-L-SGD and our proposed algorithm. We can [Figure]

Original: We can see that the BVR-L-SGD got stuck at a small gradient norm region in initial rounds.
Revised: see that the both algorithms got stuck at a small gradient norm region in initial rounds.

Original: Afterthat BVR-L-SGD showed unstable convergence and took a lot of time to escape the stucked region.
Revised: After that BVR-L-SGD showed unstable convergence and took a lot of time to escape the stucked region.

Original: In contrast, our proposed method efficiently escaped the stucked region and consistently achievesbetter train loss and test accuracy than BVR-L-SGD.
Revised: In contrast, our proposed method efficiently escaped the stucked region and consistently achieves better train loss and test accuracy than BVR-L-SGD.

Original: Also, our method consistently outperformed 
Revised: Also, our method consistently outperformed Minibatch

Original: Minibatch SGD and Noisy Minibatch SGD.
Revised: SGD and Noisy Minibatch SGD.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we have studied a new local algorithm called Bias-Variance Reduced Local Perturbed
Revised: In this paper, we have studied a new local algorithm called Bias-Variance Reduced Local Perturbed

Original: SGD (BVR-L-PSGD) based on a combination of the bias-variance reduced gradient estimator withparameter perturbation to efficiently find second-order optimal points in centralized nonconvexdistributed optimization.
Revised: SGD (BVR-L-PSGD) based on a combination of the bias-variance reduced gradient estimator with parameter perturbation to efficiently find second-order optimal points in centralized nonconvex distributed optimization.

Original: We have shown that BVR-L-PSGD enjoys second-order optimality withouthurting the best known communication complexity for first-order optimality guarantees.
Revised: We have shown that BVR-L-PSGD enjoys second-order optimality without hurting the best known communication complexity for first-order optimality guarantees.

Original: Particularly,the communication complexity is better than non-local methods when Hessian heterogeneity ζ oflocal datasets is smaller than the smoothness of the local loss L in order sense.
Revised: Particularly, the communication complexity is better than non-local methods when Hessian heterogeneity ζ of local datasets is smaller than the smoothness of the local loss L in order sense.

Original: Also, for sufficientlylarge B , the communication complexity of our method approaches to (cid:101) Θ(1) when the local datasetsheterogeneity ζ goes to zero.
Revised: Also, for sufficiently large B , the communication complexity of our method approaches to (cid:101) Θ(1) when the local datasets heterogeneity ζ goes to zero.

Original: The numerical results have validated our theoretical findings.
Revised: The numerical results have validated our theoretical findings.

Revised: Acknowledgement

Revised: TS was partially supported by JSPS KAKENHI (18K19793, 18H03201, and 20H00576), Japan

Revised: DigitalDesign, and JST CREST.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] See Section 
Revised: [Yes] See Section 1.

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Section 4.
Revised: [Yes] See Section 4.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See section 2.
Revised: [Yes] See section 2.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] See Section 4 and thesupplementary material.
Revised: [Yes] See Section 4 and the supplementary material.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] See Section and the supplementary material.
Revised: [Yes] See Sectionand the supplementary material.

Original: (b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See Section 5 and the supplementary material.
Revised: [Yes] See Section 5 and the supplementary material.

Original: (c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] See Section 5.
Revised: [Yes] See Section 5.

Original: (d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See the supplementary material.
Revised: [Yes] See the supplementary material.

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Section 5.
Revised: [Yes] See Section 5.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]