Original: Escaping Saddle Points with Bias-Variance Reduced Local Perturbed SGD for Communication Efficient Nonconvex Distributed Learning
Revised: Escaping Saddle Points with Bias-Variance Reduced Local Perturbed SGD for Communication Efficient Nonconvex Distributed Learning

Original: s
Revised: s

Original: Abstract
Revised: Abstract

Original: In recent centralized nonconvex distributed learning and federated learning, local methods are one of the promising approaches to reduce communication time.
Revised: In recent centralized nonconvex distributed learning and federated learning, local methods are one of the promising approaches to reduce communication time.

Original: However, existing work has mainly focused on studying first-order optimalityguarantees.
Revised: However, existing work has mainly focused on studying first-order optimalityguarantees.

Original: On the other side, second-order optimality guaranteed algorithms,i.e., algorithms escaping saddle points, have been extensively studied in the nondistributed optimization literature.
Revised: On the other side, second-order optimality guaranteed algorithms,i.e., algorithms escaping saddle points, have been extensively studied in the nondistributed optimization literature.

Original: In this paper, we study a new local algorithmcalled Bias-Variance Reduced Local Perturbed SGD (BVR-L-PSGD), that combines the existing bias-variance reduced gradient estimator with parameter perturbation to find second-order optimal points in centralized nonconvex distributedoptimization.
Revised: In this paper, we study a new local algorithmcalled Bias-Variance Reduced Local Perturbed SGD (BVR-L-PSGD), that combines the existing bias-variance reduced gradient estimator with parameter perturbation to find second-order optimal points in centralized nonconvex distributedoptimization.

Original: BVR-L-PSGD enjoys second-order optimality with nearly the samecommunication complexity as the best known one of BVR-L-SGD to find first-orderoptimality.
Revised: BVR-L-PSGD enjoys second-order optimality with nearly the samecommunication complexity as the best known one of BVR-L-SGD to find first-orderoptimality.

Original: Particularly, the communication complexity is better than non-localmethods when the local datasets heterogeneity is smaller than the smoothness ofthe local loss.
Revised: Particularly, the communication complexity is better than non-localmethods when the local datasets heterogeneity is smaller than the smoothness ofthe local loss.

Original: In an extreme case, the communication complexity approaches to(cid:101) Θ(1) when the local datasets heterogeneity goes to zero.
Revised: In an extreme case, the communication complexity approaches to(cid:101) Θ(1) when the local datasets heterogeneity goes to zero.

Original: Numerical results validateour theoretical findings.
Revised: Numerical results validateour theoretical findings.

Original:  Introduction
Revised:  Introduction

Original: Distributed learning is an attractive approach to reduce the total execution time by utilizing theparallel computations.
Revised: Distributed learning is an attractive approach to reduce the total execution time by utilizing theparallel computations.

Original: However, the communication time in distributed learning can be a mainbottleneck in the entire process due to huge parameter size typical in deep learning or low bandwidthcommunication environments.
Revised: However, the communication time in distributed learning can be a mainbottleneck in the entire process due to huge parameter size typical in deep learning or low bandwidthcommunication environments.

Original: To reduce communication time, one of the promising approaches is the usage of local methodssuch as local SGD (also called as Parallel Restart SGD or FedAvg).
Revised: To reduce communication time, one of the promising approaches is the usage of local methodssuch as local SGD (also called as Parallel Restart SGD or FedAvg).

Original: In local SGD, each workerindependently executes multiple updates of the local model based on his own local dataset and theserver periodically communicates and aggregates the local models.
Revised: In local SGD, each workerindependently executes multiple updates of the local model based on his own local dataset and theserver periodically communicates and aggregates the local models.

Original: Many paper have studied local
Revised: Many paper have studied local

Original: SGD [25, 30, 7, 6, 15, 13, 27, 28].
Revised: SGD [25, 30, 7, 6, 15, 13, 27, 28].

Original: Particularly, for convex objectives, it has been shown in [28], forthe first time, the communication complexity (that is the necessary number of communication roundsto achieve given desired optimization error) of local SGD can be smaller than the one of minibatch
Revised: Particularly, for convex objectives, it has been shown in [28], forthe first time, the communication complexity (that is the necessary number of communication roundsto achieve given desired optimization error) of local SGD can be smaller than the one of minibatch

Original: SGD when the heterogeneity of the local datasets is extremely small.
Revised: SGD when the heterogeneity of the local datasets is extremely small.

Original: In traditional distributedlearning, the local datasets are typically random subsets of the global dataset and in this case theheterogeneity of the local datasets may become quite small.
Revised: In traditional distributedlearning, the local datasets are typically random subsets of the global dataset and in this case theheterogeneity of the local datasets may become quite small.

Original: However, in the recent federated learningregimes [16, 24, 18], it is often the case that the heterogeneity of the local datasets is not too small.
Revised: However, in the recent federated learningregimes [16, 24, 18], it is often the case that the heterogeneity of the local datasets is not too small.

Original: Also, the analysis in [28] has only focused on convex cases.
Revised: Also, the analysis in [28] has only focused on convex cases.

Original: Hence, the superiority of local SGD tominibatch SGD is still quite limited.
Revised: Hence, the superiority of local SGD tominibatch SGD is still quite limited.

Original: Recently, more communication efficient local methods than local SGD have been proposed forpossibly nonconvex objectives to guarantee first-order optimality [12, 22, 19].
Revised: Recently, more communication efficient local methods than local SGD have been proposed forpossibly nonconvex objectives to guarantee first-order optimality [12, 22, 19].

Original: SCAFFOLD [12]is a new local algorithm based on the idea of reducing their called client-drift by using a similarformulation to the variance reduction technique [10].
Revised: SCAFFOLD [12]is a new local algorithm based on the idea of reducing their called client-drift by using a similarformulation to the variance reduction technique [10].

Original: They have shown that the communicationcomplexity of SCAFFOLD can be smaller than the one of minibatch SGD for not too heterogenouslocal datasets under the quadraticity of the (possibly nonconvex) local objectives, which is quitelimited.
Revised: They have shown that the communicationcomplexity of SCAFFOLD can be smaller than the one of minibatch SGD for not too heterogenouslocal datasets under the quadraticity of the (possibly nonconvex) local objectives, which is quitelimited.

Original: For general nonconvex objectives, the communication complexity of SCAFFOLD is sameas minibatch SGD.
Revised: For general nonconvex objectives, the communication complexity of SCAFFOLD is sameas minibatch SGD.

Original: More recently, Murate and Suzuki [19] have proposed Bias-Variance Reduced
Revised: More recently, Murate and Suzuki [19] have proposed Bias-Variance Reduced

Original: Local SGD (BVR-L-SGD).
Revised: Local SGD (BVR-L-SGD).

Original: BVR-L-SGD utilizes their proposed bias-variance reduced estimatorthat simultaneously reduces the bias caused by local gradient steps and the variance caused bystochastization of the gradients in local optimization based on the formulation of SARAH likevariance reduction [20].
Revised: BVR-L-SGD utilizes their proposed bias-variance reduced estimatorthat simultaneously reduces the bias caused by local gradient steps and the variance caused bystochastization of the gradients in local optimization based on the formulation of SARAH likevariance reduction [20].

Original: They have shown that the communication complexity of BVR-L-SGD issmaller than minibatch SGD for not too heterogeneous local datasets for general nonconvex objectives .
Revised: They have shown that the communication complexity of BVR-L-SGD issmaller than minibatch SGD for not too heterogeneous local datasets for general nonconvex objectives .

Original: Specifically, BVR-L-SGD is superior to minibatch SGD when the Hessian heterogeneity of the localdatasets is small relative to the smoothness of the local loss in order sense.
Revised: Specifically, BVR-L-SGD is superior to minibatch SGD when the Hessian heterogeneity of the localdatasets is small relative to the smoothness of the local loss in order sense.

Original: On the other side, there are vast work that has studied second-order optimality guarantees, which ismuch more challenging to ensure but desirable than first-order one, in non-distributed nonconvexoptimization.
Revised: On the other side, there are vast work that has studied second-order optimality guarantees, which ismuch more challenging to ensure but desirable than first-order one, in non-distributed nonconvexoptimization.

Original: Several approaches are known and one of the simplest approaches is parameterperturbation [4, 8, 9].
Revised: Several approaches are known and one of the simplest approaches is parameterperturbation [4, 8, 9].

Original: However, almost all existing analysis of local methods have only focusedon achieving first-order optimality.
Revised: However, almost all existing analysis of local methods have only focusedon achieving first-order optimality.

Original: As an exception, Vlaski et al.
Revised: As an exception, Vlaski et al.

Original: [26] have analysed second-orderguarantees of local SGD with parameter perturbation, but the obtained communication complexity ismuch worse than the one of minibatch SGD and no benefit of localization has been shown.
Revised: [26] have analysed second-orderguarantees of local SGD with parameter perturbation, but the obtained communication complexity ismuch worse than the one of minibatch SGD and no benefit of localization has been shown.

Original: Open question.
Revised: Open question.

Original: For local methods, it is not well-studied how tofind second-order optimal pointswith low communication cost andthus we have the following research questions:
Revised: For local methods, it is not well-studied how to communication efficiently findsecond-order optimal points and we have the following research question:

Original: Is there a first-order distributed optimization algorithm with second-order optimality guaranteeswhich satisfies that (i) the communication complexity is smaller than non-local methods for not tooheterogeneous local datasets; and (ii) the communication complexity approaches to Θ(1) when theheterogeneity of local datasets goes to zero?
Revised: Is there a first-order distributed optimization algorithm with second-order optimality guaranteeswhich satisfies that (i) the communication complexity is smaller than non-local methods for not tooheterogeneous local datasets; and (ii) the communication complexity approaches to Θ(1) when theheterogeneity of local datasets goes to zero?

Original: Note that the both properties are desirable in distributed optimization.
Revised: Note that the both properties are desirable in distributed optimization.

Original: We expect that local methodsare superior to non-local methods for not highly heterogeneous local datasets.
Revised: We expect that local methodsare superior to non-local methods for not highly heterogeneous local datasets.

Original: Furthermore, when thelocal datasets are nearly identical, it is expected that a few communications are sufficient to optimizethe global objective.
Revised: Furthermore, when thelocal datasets are nearly identical, it is expected that a few communications are sufficient to optimizethe global objective.

Original: Only in the case of first-order optimality , Murata and Suzuki [19] have shownthat their proposed BVR-L-SGD satisfies (i) and (ii), and the question has been positively answered.
Revised: Only in the case of first-order optimality , Murata and Suzuki [19] have shownthat their proposed BVR-L-SGD satisfies (i) and (ii), and the question has been positively answered.

Original: However, the question is still open in the case of second-order optimality 1 .
Revised: However, the question is still open in the case of second-order optimality 1 .

Original: Main Contributions
Revised: Main Contributions

Original: We propose a new local algorithm called Bias-Variance Reduced Local Perturbed SGD (BVR-LPSGD) for nonconvex distributed learning to efficiently find second-order optimal points, whichpositively answered the above research questions.
Revised: We propose a new local algorithm called Bias-Variance Reduced Local Perturbed SGD (BVR-LPSGD) for nonconvex distributed learning to efficiently find second-order optimal points.

Original: The algorithm is based on a simple combination of the existing bias and variance reduced gradientestimator and parameter perturbation.
Revised: The algorithm is based on a simple combination of the existing bias and variance reduced gradientestimator and parameter perturbation.

Original: In our algorithm, parameter perturbation is carried out at everylocal update and it is not necessary to determine whether or not to add noise by checking the norm ofthe global gradients, which is often required in several previous non-distributed algorithms [5, 17].
Revised: In our algorithm, parameter perturbation is carried out at everylocal update and it is not necessary to determine whether or not to add noise by checking the norm ofthe global gradients, which is often required in several previous non-distributed algorithms [5, 17].

Original: We analyse BVR-L-PSGD for general nonconvex smooth objectives.
Revised: We analyse BVR-L-PSGD for general nonconvex smooth objectives.

Original: The most challenging part ofour analysis is to ensure that our algorithm efficiently escapes global saddle points even in localoptimization .
Revised: The most challenging part ofour analysis is to ensure that our algorithm efficiently escapes global saddle points even in localoptimization .

Original: To realize this, it is necessary to analyse the behavior of the bias-variance reducedestimator around the saddle points by carefully evaluating the degree of some kind of asymptotic1 Since there are communication efficient distributed optimization algorithms that find first-order stationary points like BVR-L-SGD, we can apply generic algorithms which guarantee second-order optimality to them [29, 1].
Revised: To realize this, it is necessary to analyse the behavior of the bias-variance reducedestimator around the saddle points by carefully evaluating the degree of some kind of asymptoticconsistency of the estimator around the saddle points.

Original: However, this naive approach does not possess the aforementioned property (ii) because the generic framework requires at least Ω(1 /ε 3 / 2 ) communication rounds to guarantee second-order optimality due to the multiple negative curvature exploitation steps in their framework for any communication efficient algorithms with first-order optimality guarantees.

Original: Also, this approach requires explicit negative curvature exploitation, that is complicated and makes the whole algorithm less practical.
Revised: This point has never been pursued in previouswork and has a unique difficulty of our analysis

Original: [Table]
Revised: [Table]

Original: Assumptions

Original: Guarantee
Revised: Guarantee

Original: Minibatch SGD 1 ε 2 + 1 B Pε2-3, BSGV 1st-order
Revised: Minibatch SGD 1 ε 2 + 1 B Pε None 1st-order

Original: Noisy Minibatch SGD [9] 1 ε 2 + 1 B Pε22nd-order
Revised: Noisy Minibatch SGD [9] 1 ε 2 + 1 B Pε

Original:  Minibatch SARAH [21] 1 ε 2 + √ n B Pε2 + n B P 21st-order
Revised: None 2nd-order Minibatch SARAH [21] 1 ε 2 + √ n B Pε

Original: SSRGD [17] 1 ε 2 + √ n ε2-5, B ≥ √ nP 2nd-order
Revised: None 1st-order

Original: Local SGD [30] 1 B ε 2 + 1 B Pε 4 + 1 ε2-3,1st-order
Revised: Local SGD [30] 1 B ε 2 + 1 B Pε 4 + 1 ε None 1st-order

Original: SCAFFOLD [12] 1 ε 2 + 1 B Pε2-3, BSGV 1st-order
Revised: SCAFFOLD [12] 1 ε 2 + 1 B Pε None 1st-order

Original: SCAFFOLD [12] 1 B ε 2 + 1 B Pε 4 + ζε1-3, BSGV, quadraticity 1st-order
Revised: SCAFFOLD [12] 1 B ε 2 + 1 B Pε 4 + ζεquadraticity, ζ Hessian heterogeneity [Table] -order

Original: BVR-L-SGD [19] 1 √B ε 2 + √ n
Revised: BVR-L-SGD [19] 1 √B ε 2 + √ n

Original: B Pε 2 + ζε11st-order BVR-L-PSGD (this paper) 1 √B ε 2 +
Revised: B Pε 2 + ζεζ Hessian heterogeneity

Original: [Table]
Revised: [Table]

Original:  n 
Revised: -order BVR-L-PSGD (this paper) 1 √B ε 2 + √ n B Pε 2 + ζεζ Hessian heterogeneity

Original: [Table]
Revised: [Table]

Original: 2 + ζε12nd-order

Original: ( L ) = Θ(1) .

Original: consistency of the estimator around the saddle points.

Original: This point has never been pursued in previouswork and has a unique difficulty of our analysis.

Original: The comparison of the communication complexities of our method with the most relevant existingresults is given in Table 1.
Revised: The comparison of the communication complexities of our method with the most relevant existingresults is given in Table 1.

Original: Our proposed method enjoys second-order optimality with nearly the samecommunication complexity as the one of BVR-L-SGD, which achieves the best known communicationcomplexity to achieve first-order optimality.
Revised: Our proposed method enjoys second-order optimality with nearly the samecommunication complexity as the one of BVR-L-SGD, which achieves the best known communicationcomplexity to achieve first-order optimality.

Original: This means that our method finds second-order optimalpoints without hurting the communication efficiency of the state-of-the-art first-order optimalityguaranteed method.
Revised: This means that our method finds second-order optimalpoints without hurting the communication efficiency of the state-of-the-art first-order optimalityguaranteed method.

Original: Particularly, the communication complexity is better than minibatch SGD when
Revised: Particularly, the communication complexity is better than minibatch SGD when

Original: Hessian heterogeneity ζ is small relative to smoothness L .
Revised: Hessian heterogeneity ζ is small relative to smoothness L .

Original: Also, the communication complexityapproaches to (cid:101) O
Revised: Also, the communication complexityapproaches to (cid:101) O

Original: (1) when heterogeneity ζ goes to zero and the local computation budget B (see
Revised: (1) when heterogeneity ζ goes to zero and the local computation budget B (see

Original: Section 2) goes to infinity.
Revised: Section 2) goes to infinity.

Original: Hence, our method enjoys the aforementioned two desired properties.
Revised: Hence, our method enjoys the aforementioned two desired properties.

Original: Related Work
Revised: Related Work

Original: Here, we briefly review the related studies to our paper.
Revised: Here, we briefly review the related studies to our paper.

Original: Local methods.
Revised: Local methods.

Original: Several recent papers have studied local algorithms combined with variance reductiontechnique [23, 2, 14, 11].
Revised: Several recent papers have studied local algorithms combined with variance reductiontechnique [23, 2, 14, 11].

Original: Sharma et al.
Revised: Sharma et al.

Original: [23] have proposed a local variant of SPIDER [3] and shownthat the proposed algorithm achieves the optimal total computational complexity.
Revised: [23] have proposed a local variant of SPIDER [3] and shownthat the proposed algorithm achieves the optimal total computational complexity.

Original: However, thecommunication complexity essentially matches the ones of non-local SARAH and no advantage oflocalization has been shown.
Revised: However, thecommunication complexity essentially matches the ones of non-local SARAH and no advantage oflocalization has been shown.

Original: Khanduri et al.
Revised: Khanduri et al.

Original: [14] have proposed STEM and its variants based on theircalled two-sided momentum, but again the communication complexity does not improve non-localmethods.
Revised: [14] have proposed STEM and its variants based on theircalled two-sided momentum, but again the communication complexity does not improve non-localmethods.

Original: Also, Das et al.
Revised: Also, Das et al.

Original: [2] have considered a SPIDER like local algorithm called FedGLOMO butthe derived communication complexity is even worse than minibatch SARAH.
Revised: [2] have considered a SPIDER like local algorithm called FedGLOMO butthe derived communication complexity is even worse than minibatch SARAH.

Original: Karimireddy et al.[11] have proposed Mime, which is a general framework to mitigate client-drift.
Revised: Karimireddy et al.[11] have proposed Mime, which is a general framework to mitigate client-drift.

Original: Particularly, underδ -Bounded Hessian Dissimilarity (BHD) 2 , their MimeMVR achieves communication complexity
Revised: Particularly, underδ -Bounded Hessian Dissimilarity (BHD) 2 , their MimeMVR achieves communication complexity

Original: of1 /
Revised: of1 /

Original: ( Pε 2 )+ δ/ ( √ Pε 3 )
Revised: ( Pε 2 )+ δ/ ( √ Pε 3 )

Original: + δ/ε 2 when B → ∞ , that is better than the one of minibatch SGD 1 /ε 2 whenδ ≤ √ Pε .
Revised: + δ/ε 2 when B → ∞ , that is better than the one of minibatch SGD 1 /ε 2 whenδ ≤ √ Pε .

Original: However, the asymptotic rate is still worse than the one of BVR-L-SGD ζ/ε 2 becauseζ ≤ δ always holds.
Revised: However, the asymptotic rate is still worse than the one of BVR-L-SGD ζ/ε 2 becauseζ ≤ δ always holds.

Original: Second-order guarantee.
Revised: Second-order guarantee.

Original: Neon [29] and Neon2 [1] are generic first-order methods with second-orderguarantees, that repeatedly run a first-order guaranteed algorithm and negative curvature descent.
Revised: Neon [29] and Neon2 [1] are generic first-order methods with second-orderguarantees, that repeatedly run a first-order guaranteed algorithm and negative curvature descent.

Original: Another approach is a parameter perturbation for SGD.
Revised: Another approach is a parameter perturbation for SGD.

Original: For the first time, Ge et al.
Revised: For the first time, Ge et al.

Original: [4] have shown that
Revised: [4] have shown that

Original: SGD with a simple parameter perturbation escapes saddle points efficiently.
Revised: SGD with a simple parameter perturbation escapes saddle points efficiently.

Original: Later, the analysis hasbeen refined by [8, 9].
Revised: Later, the analysis has been refined by [8, 9].

Original: Recently, applying variance reduction technique to second-order guaranteed methods has been also studied [5, 17] and particularly Li et al.
Revised: Recently, applying variance reduction technique to second-order guaranteed methods has been also studied [5, 17] and particularly Li et al.

Original: [17] have proposed SSRGD thatcombines SARAH [20] with parameter perturbation and shown that SSRGD nearly achieves theoptimal computational complexity with second-order optimality guarantees.
Revised: [17] have proposed SSRGD that combines SARAH [20] with parameter perturbation and shown that SSRGD nearly achieves the optimal computational complexity with second-order optimality guarantees

Original:  Problem Definition and 
Revised:  Problem Definition and Assumptions

Original: Assumptions

Original: In this section, we first introduce several notations and definitions used in this paper.
Revised: In this section, we first introduce several notations and definitions used in this paper.

Original: Then, theproblem settings are described and theoretical assumptions used in our analysis are given.
Revised: Then, theproblem settings are described and theoretical assumptions used in our analysis are given.

Original: Notation.
Revised: Notation.

Original: ∥ · ∥ denotes the Euclidean L 2 norm ∥ · ∥ 2 : ∥ x ∥ =
Revised: ∥ · ∥ denotes the Euclidean L 2 norm ∥ · ∥ 2 : ∥ x ∥ =

Original: (cid:112)(cid:80) i x 2 i for vector x .
Revised: (cid:112)(cid:80) i x 2 i for vector x .

Original: For a matrix X ,
Revised: For a matrix X ,

Original: ∥ X ∥ denotes the induced norm by the Euclidean L 2 norm.
Revised: ∥ X ∥ denotes the induced norm by the Euclidean L 2 norm.

Original: For a natural number m , [ m ] means theset { 1 , 2 , . . .
Revised: For a natural number m , [ m ] means theset { 1 , 2 , . . .

Original: , m } .
Revised: , m } .

Original: For a set A , # A means the number of elements, which is possibly ∞ .
Revised: For a set A , # A means the number of elements, which is possibly ∞ .

Original: For anynumber a, b , a ∨ b and a ∧ b denote max { a, b } and min { a, b } respectively.
Revised: For anynumber a, b , a ∨ b and a ∧ b denote max { a, b } and min { a, b } respectively.

Original: We denote the uniformdistribution over A by Unif( A ) .
Revised: We denote the uniformdistribution over A by Unif( A ) .

Original: Given K, T, S ∈ N , let I ( k, t, s ) be integer k
Revised: Given K, T, S ∈ N , let I ( k, t, s ) be integer k

Original: + Kt + KTs fork ∈
Revised: + Kt + KTs fork ∈

Original: [ K ] ∪ { 0 } , t ∈
Revised: [ K ] ∪ { 0 } , t ∈

Original: [ T − 1] ∪ { 0 } and s ∈ [ S − 1] ∪ { 0 } .
Revised: [ T − 1] ∪ { 0 } and s ∈ [ S − 1] ∪ { 0 } .

Original: Note that
Revised: Note that

Original: I ( K, t, s ) =
Revised: I ( K, t, s ) =

Original: I (0 , t + 1 , s ) and
Revised: I (0 , t + 1 , s ) and

Original: I ( k, T, s ) =
Revised: I ( k, T, s ) =

Original: I ( k, 0 , s + 1) for k ∈
Revised: I ( k, 0 , s + 1) for k ∈

Original: [ K ] ∪ { 0 } , t ∈
Revised: [ K ] ∪ { 0 } , t ∈

Original: [ T − 1] ∪ { 0 } and s ∈ [ S − 1] ∪ { 0 } .
Revised: [ T − 1] ∪ { 0 } and s ∈ [ S − 1] ∪ { 0 } .

Original: B dr denotesthe set { x ∈ R d |∥ x ∥ ≤ r } , which is the Euclidean ball in R d with radius r .
Revised: B dr denotesthe set { x ∈ R d |∥ x ∥ ≤ r } , which is the Euclidean ball in R d with radius r .

Original: Definition 2.1 (Gradient Lipschitzness) .
Revised: Definition 2.1 (Gradient Lipschitzness) .

Original: A differentiable function f : R d → R is L -gradient
Revised: A differentiable function f : R d → R is L -gradient

Original: Lipschitz if ∥∇ f ( x ) − ∇ f ( y ) ∥
Revised: Lipschitz if ∥∇ f ( x ) − ∇ f ( y ) ∥

Original: ≤ L ∥ x − y ∥ , ∀ x, y ∈ R d .
Revised: ≤ L ∥ x − y ∥ , ∀ x, y ∈ R d .

Original: Definition 2.2 (Hessian Lipschitzness) .
Revised: Definition 2.2 (Hessian Lipschitzness) .

Original: A twice differentiable function f : R d → R is ρ -Hessian
Revised: A twice differentiable function f : R d → R is ρ -Hessian

Original: Lipschitz if (cid:13)(cid:13) ∇ 2 f ( x ) − ∇ 2 f ( y ) (cid:13)(cid:13) ≤
Revised: Lipschitz if (cid:13)(cid:13) ∇ 2 f ( x ) − ∇ 2 f ( y ) (cid:13)(cid:13) ≤

Original: ρ ∥ x − y ∥ , ∀ x, y ∈ R d .
Revised: ρ ∥ x − y ∥ , ∀ x, y ∈ R d .

Original: Definition 2.3 (Second-order optimality) .
Revised: Definition 2.3 (Second-order optimality) .

Original: For a ρ -Hessian Lipschitz function f , x ∈ R d is anε -second-order optimal point of f if ∥∇ f ( x ) ∥ ≤ ε and ∇ 2 f ( x ) ⪰ −√ ρεI .
Revised: For a ρ -Hessian Lipschitz function f , x ∈ R d is anε -second-order optimal point of f if ∥∇ f ( x ) ∥ ≤ ε and ∇ 2 f ( x ) ⪰ −√ ρεI .

Original: 2.1 Problem Settings
Revised: 2.1 Problem Settings

Original: Objective function.
Revised: Objective function.

Original: We want to minimize nonconvex smooth objective f ( x ) :
Revised: We want to minimize nonconvex smooth objective f ( x ) :

Original: = 1 P (cid:80) Pp =1 f p ( x ) ,where f p ( x ) :=
Revised: = 1 P (cid:80) Pp =1 f p ( x ) ,where f p ( x ) :=

Original: E z ∼ D p [ ℓ ( x, z )] for x ∈ R d , where D p is the data distribution associated with workerp .
Revised: E z ∼ D p [ ℓ ( x, z )] for x ∈ R d , where D p is the data distribution associated with workerp .

Original: In this paper, we focus on offline settings (i.e., #supp( D p ) <
Revised: In this paper, we focus on offline settings (i.e., #supp( D p ) <

Original: ∞ for every p ∈
Revised: ∞ for every p ∈

Original: [ P ] ) for simplepresentation.
Revised: [ P ] ) for simplepresentation.

Original: It is easy to extend our results to online settings.
Revised: It is easy to extend our results to online settings.

Original: Also, just for simplicity, it is assumedthat each local dataset has an equal number of samples, i.e., #supp( D p ) = 
Revised: Also, just for simplicity, it is assumedthat each local dataset has an equal number of samples, i.e., #supp( D p ) = #supp( D p ′ ) =

Original: n/P for every p  ∈
Revised: n/P forevery p, p ′ ∈

Original: [ P ] ,where n is the total number of samples.
Revised: [ P ] , where n is the total number of samples.

Original: Optimization criteria.
Revised: Optimization criteria.

Original: Since objective function f is nonconvex, it is generally difficult to find aglobal minima of f .
Revised: Since objective function f is nonconvex, it is generally difficult to find aglobal minima of f .

Original: Previous work in distributed learning has mainly focused on finding first-orderstationary points of f .
Revised: Previous work in distributed learning has mainly focused on finding first-orderstationary points of f .

Original: In this study, we aim to find ε -second-order stationary points of f in distributedlearning settings.
Revised: In this study, we aim to find ε -second-order stationary points of f in distributedlearning settings.

Original: Data access constraints and communication settings.
Revised: Data access constraints and communication settings.

Original: It is assumed that each worker p can onlyaccess the own data distribution D p without communication.
Revised: It is assumed that each worker p can onlyaccess the own data distribution D p without communication.

Original: Aggregation (e.g., summation) of allthe worker’s d -dimensional parameters or broadcast of a d -dimensional parameter from one workerto the other workers can be realized by single communication.
Revised: Aggregation (e.g., summation) of allthe worker’s d -dimensional parameters or broadcast of a d -dimensional parameter from one workerto the other workers can be realized by single communication.

Original: Evaluation criteria: communication complexity.
Revised: Evaluation criteria: communication complexity.

Original: In this paper, we compare communicationcomplexities of optimization algorithms to satisfy the aforementioned optimization criteria.
Revised: In this paper, we compare communicationcomplexities of optimization algorithms to satisfy the aforementioned optimization criteria.

Original: In typicalsituations, single communication is more time-consuming than single stochastic gradient computation.
Revised: In typicalsituations, single communication is more time-consuming than single stochastic gradient computation.

Original: Let C be the single communication cost and G be the single stochastic gradient computation cost.
Revised: Let C be the single communication cost and G be the single stochastic gradient computation cost.

Original: Using these notations, C ≥ G is assumed.
Revised: Using these notations, C ≥ G is assumed.

Original: We expect that increasing the number of available stochasticgradients in a single communication round leads to faster convergence.
Revised: We expect that increasing the number of available stochasticgradients in a single communication round leads to faster convergence.

Original: Hence, it is natural to increasethe number of stochastic gradient computations in a single communication round unless the totalstochastic gradient computation time exceeds C to reduce the total running time.
Revised: Hence, it is natural to increasethe number of stochastic gradient computations in a single communication round unless the totalstochastic gradient computation time exceeds C to reduce the total running time.

Original: This motivatesthe concept of local computation budget B ( ≤ C / G ): given a communication and computationalenvironment, it is assumed that each worker can only computes at most B single stochastic gradientsper communication round on average .
Revised: This motivates the concept of local computation budget B ( ≤ C / G ): given a communication and computationalenvironment, it is assumed that each worker can only computes at most B single stochastic gradientsper communication round on average .

Original: Then, we compare the communication complexity, that is thetotal number of communication rounds of a distributed optimization algorithm to achieve the desired optimization accuracy .
Revised: Then, we compare the communication complexity, that is thetotal number of communication rounds of a distributed optimization algorithm to achieve the desired optimization accuracy .

Original: From the definition, we can see that the communication complexity on a fixedlocal computation budget B :=
Revised: From the definition, we can see that the communication complexity on a fixedlocal computation budget B :=

Original: C / G captures the best achievable total running time of an algorithm.
Revised: C / G captures the best achievable total running time of an algorithm

Original: 2.2 Theoretical Assumptions
Revised: 2.2 Theoretical Assumptions

Original: In this paper, we assume the following five assumptions.
Revised: In this paper, we assume the following five assumptions.

Original: The first one has already been adopted inseveral previous work [12, 19].
Revised: The first one has already been adopted inseveral previous work [12, 19].

Original: The other ones are standard in the nonconvex optimization literatureto guarantees second-order optimality.
Revised: The other ones are standard in the nonconvex optimization literatureto guarantees second-order optimality.

Original: Assumption 1 (Hessian heterogeneity [12, 19]) .
Revised: Assumption 1 (Hessian heterogeneity [12, 19]) .

Original: { f p } Pp =1 is second-order ζ -heterogeneous, i.e., forany p, p ′ ∈
Revised: { f p } Pp =1 is second-order ζ -heterogeneous, i.e., forany p, p ′ ∈

Original: [ P ] , (cid:13)(cid:13) ∇ 2 f p ( x ) − ∇ 2 f p ′ ( x ) (cid:13)(cid:13) ≤ ζ, ∀ x ∈ R d .
Revised: [ P ] , (cid:13)(cid:13) ∇ 2 f p ( x ) − ∇ 2 f p ′ ( x ) (cid:13)(cid:13) ≤ ζ, ∀ x ∈ R d .

Original: Assumption 1 characterizes the heterogeneity of local objectives { f p } Pp =1 in terms of Hessians and hasa important role in our analysis.
Revised: Assumption 1 characterizes the heterogeneity of local objectives { f p } Pp =1 in terms of Hessians and hasa important role in our analysis.

Original: Intuitively, we expect that relatively small heterogeneity parameter ζto the smoothness parameter L (defined in Assumption 2) reduces the necessary number of communication rounds to optimize the global objective.
Revised: Intuitively, we expect that relatively small heterogeneity parameter ζto the smoothness parameter L (defined in Assumption 2) reduces the necessary number of communication rounds to optimize the global objective.

Original: Especially when the local objectives are identical, i.e.,
Revised: Especially when the local objectives are identical, i.e.,

Original: D p
Revised: D p

Original: = D p ′ for every p, p ′ ∈
Revised: = D p ′ for every p, p ′ ∈

Original: [ P ] , ζ becomes zero.
Revised: [ P ] , ζ becomes zero.

Original: When each D p is the empirical distribution of n/P
Revised: When each D p is the empirical distribution of n/P

Original: IID samples from common data distribution D , we have ∥∇ 2 f p ( x ) − ∇ 2 f p
Revised: IID samples from common data distribution D , we have ∥∇ 2 f p ( x ) − ∇ 2 f p

Original: ( x ) ∥ ≤ (cid:101)
Revised: ( x ) ∥ ≤ (cid:101)

Original: Θ( (cid:112) P/nL )with high probability by matrix Hoeffding’s inequality under Assumption 2 for fixed x .
Revised: Θ( (cid:112) P/nL )with high probability by matrix Hoeffding’s inequality under Assumption 2 for fixed x .

Original: Hence, intraditional distributed learning regimes, Assumption 1 naturally holds.
Revised: Hence, intraditional distributed learning regimes, Assumption 1 naturally holds.

Original: An important remark is that
Revised: An important remark is that

Original: Assumption 2 implies ζ ≤ 2 L , i.e., the heterogeneity is bounded by the smoothness .
Revised: Assumption 2 implies ζ ≤ 2 L , i.e., the heterogeneity is bounded by the smoothness .

Original: Even in federatedlearning regimes, we expect ζ ≪ 2 L for some problems practically.
Revised: Even in federatedlearning regimes, we expect ζ ≪ 2 L for some problems practically.

Original: Assumption 2 (Gradient Lipschitzness) .
Revised: Assumption 2 (Gradient Lipschitzness) .

Original: ∀ p ∈
Revised: ∀ p ∈

Original: [ P ] , z ∈ supp( D p ) , ℓ ( · , z ) is L -gradient Lipschitz.
Revised: [ P ] , z ∈ supp( D p ) , ℓ ( · , z ) is L -gradient Lipschitz.

Original: Assumption 3 (Existence of global optimum) .
Revised: Assumption 3 (Existence of global optimum) .

Original: f has a global minimizer x ∗ ∈ R d .
Revised: f has a global minimizer x ∗ ∈ R d .

Original: Assumption 4 (Hessian Lipschitzness) .
Revised: Assumption 4 (Hessian Lipschitzness) .

Original: ∀ p ∈
Revised: ∀ p ∈

Original: [ P ] , z ∈ supp( D p ) , ℓ ( · , z ) is ρ -Hessian Lipschitz.
Revised: [ P ] , z ∈ supp( D p ) , ℓ ( · , z ) is ρ -Hessian Lipschitz.

Original: Assumption 5 (Bounded stochastic gradient) .
Revised: Assumption 5 (Bounded stochastic gradient) .

Original: ∀ p ∈
Revised: ∀ p ∈

Original: [ P ] , z ∈ supp( D p ) , ∇ ℓ ( · , z ) is G -bounded, i.e.,∥∇ ℓ ( x, z ) ∥ ≤ G, ∀ x ∈ R d .
Revised: [ P ] , z ∈ supp( D p ) , ∇ ℓ ( · , z ) is G -bounded, i.e.,∥∇ ℓ ( x, z ) ∥ ≤ G, ∀ x ∈ R d .

Original: In our analysis, G has no significant impact because G only depends on our theoretical communicationcomplexity in logarithmic order.
Revised: In our analysis, G has no significant impact because G only depends on our theoretical communicationcomplexity in logarithmic order.

Original:  Main Ideas and Proposed Algorithm
Revised:  Main Ideas and Proposed Algorithm

Original: Our proposed algorithm is based on a natural combination of (i) Bias-Variance Reduced (BVR)estimator ; and (ii) parameter perturbation at each local update .
Revised: Our proposed algorithm is based on a natural combination of (i) Bias-Variance Reduced (BVR)estimator ; and (ii) parameter perturbation at each local update .

Original: The first idea has been proposedby [19] to find first-order stationary points with small communication complexity.
Revised: The first idea has been proposedby [19] to find first-order stationary points with small communication complexity.

Original: The secondone is a well-known approach to find second-order stationary points in non-distributed nonconvexoptimization [4, 8, 9].
Revised: The secondone is a well-known approach to find second-order stationary points in non-distributed nonconvexoptimization [4, 8, 9].

Original: In this section, we illustrate these two ideas and provide its concrete procedures.
Revised: In this section, we illustrate these two ideas and provide its concrete procedures.

Original:  Review of BVR Estimator 
Revised: 3.1 Review of BVR Estimator [19]

Original: The bias-variance reduced estimator aims to efficiently findfirst-order stationary points by simultaneously reducing the bias caused by local gradient descentsteps and the variance caused by stochastization of the used gradients.
Revised: The bias-variance reduced estimator aims to efficiently find first-order stationary points by simultaneously reducing the bias caused by local gradient descent steps and the variance caused bystochastization of the used gradients.

Original: First we consider why the standard local SGD is not sufficient to achieve fast convergence andsometimes slower than minibatch SGD.
Revised: First we consider why the standard local SGD is not sufficient to achieve fast convergence andsometimes slower than minibatch SGD.

Original: Recall that in local SGD each worker takes the update rulesof x ( p ) k
Revised: Recall that in local SGD each worker takes the update rulesof x ( p ) k

Original: +

Revised: +

Original: =
Revised: =

Original: x ( p ) k − ηg ( p ) k for k ∈
Revised: x ( p ) k − ηg ( p ) k for k ∈

Original: [ B /b ] in each communication round, where g ( p ) k is a stochasticgradient with minibatch size b at x ( p ) k on local dataset D p and B is given local computation budget.
Revised: [ B /b ] in each communication round, where g ( p ) k is a stochasticgradient with minibatch size b at x ( p ) k on local dataset D p and B is given local computation budget.

Original: In typical convergence analysis, we need to bound the expected deviation of g ( p ) k from ideal globalgradient ∇ f ( x k ) , that is E ∥ g 
Revised: In typical convergence analysis, we need to bound the expected deviation of g ( p ) k from ideal globalgradient ∇ f ( x k ) , that is E ∥ g ( p ) k −∇ f

Original: [Equation] p 
Revised: ( x ( p ) k )

Revised: ∥ 2 = ∥∇ f p ( x ( p ) k ) −∇ f

Original: [Equation]
Revised: [Equation]

Original: The former term is called bias and the latter one is called variance .
Revised: The former term is called bias and the latter one is called variance .

Original: A typical assumption to boundthe first term is bounded gradient heterogeneity assumption , that requires ∥∇ f p ( x ) − ∇ f ( x ) ∥
Revised: A typical assumption to boundthe first term is bounded gradient heterogeneity assumption , that requires ∥∇ f p ( x ) − ∇ f ( x ) ∥

Original: ≤ ζ 1for every x ∈ R d and p ∈
Revised: ≤ ζ 1for every x ∈ R d and p ∈

Original: [ P ] .
Revised: [ P ] .

Original: Under this assumption, the first term is only bounded by ζ 1 , that isa constant.
Revised: Under this assumption, the first term is only bounded by ζ 1 , that isa constant.

Original: The second term is typically bounded by σ 2 /b for g ( p ) k with minibatch size b , when  the 
Revised: The second term is typically bounded by σ 2 /b for g ( p ) k with minibatch size b , when thevariance of a single stochastic gradient is bounded by σ 2 . These facts show that the bias is still aconstant and does not vanish even if minibatch size b is enhanced and the variance vanishes. This iswhy local SGD can be worse than minibatch SGD when ζ 1 is not too small. Also, we can see thatthe variance is still a constant for fixed minibatch size b and this is a common reason why minibatch

Revised: 1: Add noise x 0 = (cid:101) x 0 + ηξ − 1 , where ξ − 1 ∼ Unif( B dr ) .

Original: 1: for s = 0 to S − 1 do 2: for p = 1 to P in  do 3: v ( p ) I (0 , 0 ,s ) = ∇ f p ( x I (0 , 0 ,s ) ) .
Revised: 2: for s = 0 to S − 1 do 3: for p = 1 to P in parallel do 4: v ( p ) I (0 , 0 ,s ) = ∇ f p ( x I (0 , 0 ,s ) ) .

Original: 4: end for 5:
Revised: 5: end for 6:

Original: Communicate { v ( p ) I (0 , 0 ,s ) } Pp =1 .
Revised: Communicate { v ( p ) I (0 , 0 ,s ) } Pp =1 .

Original: Set v I (0 , 0 ,s )
Revised: Set v I (0 , 0 ,s )

Original: = 1 P (cid:80)
Revised: = 1 P (cid:80)

Original: Pp =1 v ( p ) I (0 , 0 ,s ) .
Revised: Pp =1 v ( p ) I (0 , 0 ,s ) .

Original: 6: for t = 0 to T − 1 do 7: for p = 1 to P in parallel do 8: g ( p ) I (0 ,t,s )
Revised: 7: for t = 0 to T − 1 do 8: for p = 1 to P in parallel do 9: g ( p ) I (0 ,t,s )

Original: = 1 Kb (cid:80)
Revised: = 1 Kb (cid:80)

Original: Kbl =1 ∇ ℓ ( x I (0 ,t,s ) , z l,I (0 ,t,s ) ) , 9: g ( p ) , ref I (0 ,t,s )
Revised: Kbl =1 ∇ ℓ ( x I (0 ,t,s ) , z l,I (0 ,t,s ) ) , 10: g ( p ) , ref I (0 ,t,s )

Original: = 1 Kb (cid:80)
Revised: = 1 Kb (cid:80)

Original: Kbl =1 ∇ ℓ ( x I (0 ,t − 1 ,s ) , z l,I (0 ,t,s ))
Revised: Kbl =1 ∇ ℓ ( x I (0 ,t − 1 ,s ) , z l,I (0 ,t,s ))

Original: ( z l,I (0 ,t,s ) i.i.d.
Revised: ( z l,I (0 ,t,s ) i.i.d.

Original: ∼ D p ).
Revised: ∼ D p ).

Original: 10: v ( p ) I (0 ,t,s )
Revised: 11: v ( p ) I (0 ,t,s )

Original: = 1 t ≥ 1 ( g ( p ) I (0 ,t,s ) − g ( p ) , ref I (0 ,t,s ) + v ( p ) I (0 ,t − 1 ,s ) )
Revised: = 1 t ≥ 1 ( g ( p ) I (0 ,t,s ) − g ( p ) , ref I (0 ,t,s ) + v ( p ) I (0 ,t − 1 ,s ) )

Original: + 1 t =0 v ( p ) I (0 , 0 ,s ) .
Revised: + 1 t =0 v ( p ) I (0 , 0 ,s ) .

Original: 11: end for 12:
Revised: 12: end for 13:

Original: Communicate { v ( p ) I (0 ,t,s ) } Pp =1 .
Revised: Communicate { v ( p ) I (0 ,t,s ) } Pp =1 .

Original: Set v I (0 ,t,s )
Revised: Set v I (0 ,t,s )

Original: = 1 P (cid:80)
Revised: = 1 P (cid:80)

Original: Pp =1 v ( p ) I (0 ,t,s ) .
Revised: Pp =1 v ( p ) I (0 ,t,s ) .

Original: Randomly select pt,s ∼ Unif[ P ] . # Only worker p t,s runs local optimization.
Revised: Randomly select p worker p t,s runs local optimization.

Original: 14: for k = 0 to K − 1 do 15: g I ( k,t,s ) = 1 b (cid:80) bl =1 ∇ ℓ ( x I ( k,t,s ) , z l,I ( k,t,s ) ) , 16: g ref I ( k,t,s ) 
Revised: 15: for k = 0 to K − 1 do 16: g I ( k,t,s ) = 1 b (cid:80) b l =1 ∇ ℓ ( x I ( k,t,s ) , z l,I ( k,t,s ) ) , 17: g ref I ( k,t,s ) =

Original: = 1 b (cid:80) b l =1 ∇ ℓ ( x I ( k − 1 ,t,s ) ) , z l,I ( k,t,s ) )
Revised: 1 b (cid:80) bl =1 ∇ ℓ ( x I ( k − 1 ,t,s ) ) , z l,I ( k,t,s ) )

Original: ( z l,I ( k,t,s ) i.i.d.
Revised: ( z l,I ( k,t,s ) i.i.d.

Original: ∼ D p t,s ).
Revised: ∼ D p t,s ).

Original: 17: v I ( k,t,s )
Revised: 18: v I ( k,t,s )

Original: = 1 k ≥ 1 ( g I ( k,t,s ) − g ref I ( k,t,s )
Revised: = 1 k ≥ 1 ( g I ( k,t,s ) − g ref I ( k,t,s )

Original: + v I ( k − 1 ,t,s ) )
Revised: + v I ( k − 1 ,t,s ) )

Original: + 1 k =

Revised: + 1 k =

Original: v I (0 ,t,s ) .
Revised: v I (0 ,t,s ) .

Original: Update (cid:101) x I ( k +1 ,t,s )
Revised: Update (cid:101) x I ( k +1 ,t,s )

Original: =
Revised: =

Original: x I ( k,t,s ) − ηv I ( k,t,s ) .
Revised: x I ( k,t,s ) − ηv I ( k,t,s ) .

Original: Add noise x I ( k +1 ,t,s )
Revised: Add noise x I ( k +1 ,t,s )

Original: = (cid:101) x
Revised: = (cid:101) x

Original: I ( k +1 ,t,s ) + ηξ I ( k,t,s ) , where ξ I ( k,t,s ) ∼ Unif( B dr ) .
Revised: I ( k +1 ,t,s ) + ηξ I ( k,t,s ) , where ξ I ( k,t,s ) ∼ Unif( B dr ) .

Original: 20: end for 21:
Revised: 21: end for 22:

Original: Communicate x I (0 ,t +1 ,s ) .
Revised: Communicate x I (0 ,t +1 ,s ) .

Original: 22: end for 23: end for variance of a single stochastic gradient is bounded by σ 2 . These facts show that the bias is still aconstant and does not vanish even if minibatch size b is enhanced and the variance vanishes. This iswhy local SGD can be worse than minibatch SGD when ζ 1 is not too small. Also, we can see thatthe variance is still a constant forfixed minibatch size b and this is a common reason why minibatch
Revised: 23: end for 24: end for

Original: SGD and local SGD only show slow convergences.
Revised: SGD and local SGD only show slow convergences.

Original: These observations give critical motivations ofthe simultaneous reduction of the bias and variance.
Revised: These observations give critical motivations ofthe simultaneous reduction of the bias and variance.

Original: The bias-variance reduced estimator v ( p ) k is defined as v ( p [Equation] k 
Revised: The bias-variance reduced estimator v ( p ) k is defined as v ( p ) k :=

Original: [Equation]
Revised: (1 /b ) (cid:80) bl =1 ( ∇ ℓ ( x ( p ) k , z l ) −∇ ℓ ( x 0 , z l ))

Original: ∇ ℓ ( x 0 , z l )) + ∇ f ( x 0 ) (SVRG version).
Revised: + ∇ f ( x 0 ) (SVRG version).

Original: It is known that the bias caused by localization canbe bounded by ζ ∥ x ( p ) k − x 0 ∥ and the variance caused by stochastization can be bounded by( L 2 /b )
Revised: It is known that the bias caused by localization canbe bounded by ζ ∥ x ( p ) k − x 0 ∥ and the variance caused by stochastization can be bounded by( L 2 /b )

Original: ∥ x ( p )
Revised: ∥ x ( p )

Original: k − x 0 ∥ 2 , where ζ is the Hessian heterogeneity of { f p } and L is the smoothness ofℓ .
Revised: k − x 0 ∥ 2 , where ζ is the Hessian heterogeneity of { f p } and L is the smoothness ofℓ .

Original: This implies that both the bias and variance of v ( p ) k converges to zero as x ( p ) k and x 0 go to x ∗ .
Revised: This implies that both the bias and variance of v ( p ) k converges to zero as x ( p ) k and x 0 go to x ∗ .

Original: Inother words, bias-variance reduced estimator v ( p ) k is asymptotically consistent to the global gradient∇
Revised: Inother words, bias-variance reduced estimator v ( p ) k is asymptotically consistent to the global gradient∇

Original: f
Revised: f

Original: ( x ( p ) k ) by using periodically computed global full gradients ∇ f ( x 0 ) .
Revised: ( x ( p ) k ) by using periodically computed global full gradients ∇ f ( x 0 ) .

Original: We actually adopt SARAHversion of BVR estimator as in [19] rather than SVRG one due to its theoretical advantages.
Revised: We actually adopt SARAHversion of BVR estimator as in [19] rather than SVRG one due to its theoretical advantages.

Original:  Parameter Perturbation at Local Updates.
Revised: 3.2 Parameter Perturbation at Local Updates

Original: Although the bias-variance reduced estimator is usefulto guarantee first-order optimality with small communication complexity in noncovex optimization,the algorithm often gets stuck at saddle points.
Revised: Although the bias-variance reduced estimator is useful to guarantee first-order optimality with smallcommunication complexity in noncovex optimization, the algorithm often gets stuck at saddle points.

Original: To tackle this problem, we borrow the ideas ofescaping saddle points in non-distributed nonconvex optimization.
Revised: To tackle this problem, we borrow the ideas of escaping saddle points in non-distributed nonconvexoptimization.

Original: Particularly, to efficiently findsecond-order optimal points, we utilize parameter perturbation .
Revised: Particularly, to efficiently find second-order optimal points, we utilize parameterperturbation .

Original: Parameter perturbation is a familiarapproach in non-distributed nonconvex optimization.
Revised: Parameter perturbation is a familiar approach in non-distributed nonconvex optimization.

Original: Specifically, Jin et al.
Revised: Specifically, Jin et al.

Original: [8, 9] have considered theupdate rule of x k +

Revised: [8, 9] have considered the update rule of x k +

Original: = x
Revised: = x

Original: k − η ∇ f ( x k ) + ηξ k , where ξ k ∼ Unif( B d r ) for some small radius r .
Revised: k − η ∇ f ( x k ) + ηξ k , whereξ k ∼ Unif( B dr ) for some small radius r .

Original: Thisalgorithm is called Perturbed GD (PGD) or Noisy GD.
Revised: This algorithm is called Perturbed GD (PGD) or Noisy GD.

Original: Similar to this formulation, we add noise ateach local update, i.e., x ( p )
Revised: Similar to this formulation, we add noise at each local update, i.e., x ( p )

Original: k
Revised: k

Original: +1 =
Revised: +1 =

Original: (cid:101) x
Revised: (cid:101) x

Original: ( p ) k
Revised: ( p ) k

Original: +1 + ηξ ( p ) k , where (cid:101) x ( p ) k
Revised: +1 + ηξ ( p ) k ,where (cid:101) x ( p ) k

Original: +

Revised: +

Original: =
Revised: =

Original: x ( p ) k − ηv ( p ) k .
Revised: x ( p ) k − ηv ( p ) k .

Original: The intuition behind thenoise addition is that random noise has some components along the negative curvature directions ofthe global objective around the saddle point, and we expect that noise addition helps the parameterproceed to the decreasing directions of f and escape the saddle points.
Revised: The intuition behind the noise addition is that random noise has somecomponents along the negative curvature directions of the global objective around the saddle point,and we expect that noise addition helps the parameter proceed to the decreasing directions of f andescape the saddle points.

Original: Remark (Necessity of local perturbation) .
Revised: Necessity of local perturbation.

Original: Perturbing the global model at the server side is an intuitiveway, but not sufficient for communication efficiency when we want to utilize small heterogeneity ofthe local datasets (i.e., ζ ≪ L ).
Revised: Perturbing the global model at the server side is an intuitive way,but not sufficient for communication efficiency when we want to utilize small heterogeneity of thelocal datasets (i.e., ζ ≪ L ).

Original: The bias-variance reduced estimator with local perturbation enablesto escape multiple global saddle points in local optimization and achieves second-order optimalitywith communication complexity (cid:101) Θ( ζ/ε 2 ) for sufficiently large B .
Revised: The bias-variance reduced estimator with local perturbation enablesto escape multiple global saddle points in local optimization and achieves second-order optimalitywith communication complexity (cid:101) Θ( ζ/ε 2 ) for sufficiently large B .

Original: In contrast, perturbing the globalparameter at the server side only ensures to escape single global saddle point at each round and onlyachieves communication complexity of (cid:101) Θ( L/ε 2 ) .
Revised: In contrast, perturbing the globalparameter at the server side only ensures to escape single global saddle point at each round and onlyachieves communication complexity of (cid:101) Θ( L/ε 2 ) .

Original: This is the reason why local perturbation ratherthan global one is adopted.
Revised: This is the reason why local perturbation ratherthan global one is adopted.

Original:  Concrete Procedures.
Revised: 3.3 Concrete Procedures

Original: The full description of our proposed Bias-Variance Reduced Local Perturbed SGD (BVR-L-PSGD) is given in Algorithm 1.
Revised: The full description of our proposed Bias-Variance Reduced Local Perturbed SGD (BVR-L-PSGD)is given in Algorithm 1.

Original: When we set the noise size r = 0 , Algorithm 1essentially matches BVR-L-SGD.
Revised: When we set the noise size r = 0 , Algorithm 1 essentially matches BVR-LSGD.

Original: Additionally setting K = 1 , Algorithm 1 matches SARAH.
Revised: Additionally setting K = 1 , Algorithm 1 matches SARAH.

Original: Thealgorithm requires Θ( ST ) communication rounds.
Revised: The algorithm requires Θ( ST )communication rounds.

Original: At each communication round, each workercomputes large batch stochastic gradients and the server constructs v I (0 ,t,s ) by aggregating them.v
Revised: At each communication round, each worker computes large batch stochasticgradients and the server constructs v I (0 ,t,s ) by aggregating them.

Original:  I (0 ,t,s ) is used as an estimator of ∇ f ( x I (0 ,t,s ) ) to reduce computational cost.
Revised: v I (0 ,t,s ) is used as an estimator of∇ f ( x I (0 ,t,s ) ) to reduce computational cost.

Original: In line 14-21, we242 randomly select worker p t,s and only worker p t,s runs local optimization as described above.
Revised: In line 14-21, we randomly select worker p t,s and onlyworker p t,s runs local optimization as described above.

Original: Inthe local optimization, we use SARAH like bias variance reduced estimator (line 16-18) rather than SVRG one and add noise (line 20) at each local update.
Revised: In the local optimization, we use SARAHlike bias variance reduced estimator (line 16-18) rather than SVRG one and add noise (line 20) ateach local update.

Original:  Convergence Analysis
Revised:  Convergence Analysis

Original: In this section, we provide convergence theory of BVR-L-PSGD (Algorithm 1).
Revised: In this section, we provide convergence theory of BVR-L-PSGD (Algorithm 1).

Original: All the omittedproofs are found in the supplementary material.
Revised: All the omittedproofs are found in the supplementary material.

Original: For simple presentations, we use (cid:101) Θ symbol to hidean extra poly-logarithmic factorsthat depend on L, ρ, G, K, b, T, S, 1 /ε, 1 /q , where q represents theconfidence parameter in high probability bounds.
Revised: For simple presentations, we use (cid:101) Θ symbol tohide an extra poly-logarithmic factors depend on L, ρ, G, K, b, T, S, 1 /ε, 1 /q , where q represents theconfidence parameter in high probability bounds.

Original: 4.1 Finding First-Order Stationary Points
Revised: 4.1 Finding First-Order Stationary Points

Original: First, we derive Descent Lemma for BVR-L-PSGD and first-order optimality guarantees by using it.
Revised: First, we derive Descent Lemma for BVR-L-PSGD and first-order optimality guarantees by using it.

Original: Proposition 4.1 (Descent Lemma) .
Revised: Proposition 4.1 (Descent Lemma) .

Original: Let S ∈ N
Revised: Let S ∈ N

Original: and I ( k, t, s ) ≥ I ( k 0 , t 0 , s 0 ) ∈
Revised: and I ( k, t, s ) ≥ I ( k 0 , t 0 , s 0 ) ∈

Original: [ KTS ] ∪ { 0 } .
Revised: [ KTS ] ∪ { 0 } .

Original: Suppose that Assumptions 1, 2, 3 and 5 hold.
Revised: Suppose that Assumptions 1, 2, 3 and 5 hold.

Original: Given q ∈ (0 , 1) , r > 0 , if we appropriately chooseη = (cid:101) Θ(

Revised: Given q ∈ (0 , 1) , r > 0 , if we appropriately chooseη = (cid:101) Θ(

Original: /L ∧ 1 / ( Kζ ) ∧ (cid:112) b/K/L ∧ √ Pb/ ( √ KTL ))
Revised: /L ∧ 1 / ( Kζ ) ∧ (cid:112) b/K/L ∧ √ Pb/ ( √ KTL ))

Original: , it holds that
Revised: , it holds that

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − 3 q .
Revised: with probability at least 1 − 3 q .

Original: Here,
Revised: Here,

Original: [Equation] 
Revised: [Equation] [Equation]

Revised: ∆

Original: [Equation]
Revised: [Equation]

Revised: ∥ x

Original: [Equation]
Revised: [Equation]

Original:  ) for some universal constant c η > 0 .
Revised: − x i ∥ 2 ) for some universal constant c η > 0 .

Original: From Proposition 4.1 with I ( k 0 , t 0 , s 0 )
Revised: From Proposition 4.1 with I ( k 0 , t 0 , s 0 )

Original: ← 0 and I ( k, t, s ) ← KTS gives the following corollary.
Revised: ← 0 and I ( k, t, s ) ← KTS gives the following corollary.

Original: Corollary 4.2.
Revised: Corollary 4.2.

Original: Suppose that Assumptions 1, 2, 3 and 5 hold.
Revised: Suppose that Assumptions 1, 2, 3 and 5 hold.

Original: Under the same setting as in Proposition4.3 and S ≥ Θ(( f ( x 0 ) − f ( x ∗ )) / ( ηKTε 2 )) , with probability at least 1 − 3 q , there exists
Revised: Under the same setting as in Proposition4.3 and S ≥ Θ(( f ( x 0 ) − f ( x ∗ )) / ( ηKTε 2 )) , with probability at least 1 − 3 q , there exists

Original: i ∈
Revised: i ∈

Original: [ KTS − 1] ∪ { 0 } such that ∥∇ f
Revised: [ KTS − 1] ∪ { 0 } such that ∥∇ f

Original: ( (cid:101) x i ) ∥
Revised: ( (cid:101) x i ) ∥

Original: ≤ ε .
Revised: ≤ ε .

Original: Remark (Communication complexity) .
Revised: Remark (Communication complexity) .

Original: The total number of communication roundsΘ( TS ) becomes(cid:101) O (cid:16) T + (cid:16) L/K + ζ + L/ √ Kb + √ TL/ √ KPb (cid:17) ( f ( (cid:101) x 0 ) − f ( x ∗ )) /ε
Revised: The total number of communication rounds TS becomes (cid:101) O (cid:16) T + (cid:16) L/K + ζ + L/ √ Kb + √ TL/ √ KPb (cid:17) ( f ( (cid:101) x 0 ) − f ( x ∗ )) /ε

Original: 2 (cid:17) .
Revised: 2 (cid:17) .

Original: Given local computationbudget B , we set T :=
Revised: Given local computation budget B , we set T :=

Original: Θ(1 + n/ ( B P )) and Kb :=
Revised: Θ(1 + n/ ( B P )) and Kb :=

Original: Θ( B ) with b ≤ Θ( √B ) .
Revised: Θ( B ) with b ≤ Θ( √B ) .

Original: Then, we have theaveraged number of local computations per communication round Kb + n/ ( PT ) =
Revised: Then, we have the averaged number of local computations per communication round Kb + n/ ( PT ) =

Original: Θ( B ) and thecommunication complexity TS with budget B becomes(cid:101) O (cid:16) 1 + n B P +
Revised: Θ( B ) and the communication complexity TS with budget B becomes

Original: L √B ε 2 + √ nL
Revised: [Equation]

Original: B Pε 2 + ζε 2 (cid:17) , whichmatches the best known communication complexity [19].
Revised: which matches the best known communication complexity [19].

Original: 4.2 Escaping Saddle Points
Revised: 4.2 Escaping Saddle Points

Original: Next, we show that BVR-L-PSGD implicitly exploits the negative curvature of f around saddle pointsand efficiently escapes the saddle points by utilizing the asymptotic consistency of BVR estimatorand the parameter perturbation at each local update.
Revised: Next, we show that BVR-L-PSGD implicitly exploits the negative curvature of f around saddle pointsand efficiently escapes the saddle points by utilizing the asymptotic consistency of BVR estimatorand the parameter perturbation at each local update.

Original: We rely on the technique of coupling sequence [9].
Revised: We rely on the technique of coupling sequence [9].

Original: Given saddle point (cid:101) x I ( k 0 ,t 0 ,s 0 ) , we define anew sequence { x ′ i } ∞ i
Revised: Given saddle point (cid:101) x I ( k 0 ,t 0 ,s 0 ) , we define anew sequence { x ′ i } ∞ i

Original: = I ( k 0 ,t 0 ,s 0 ) that satisfies x I ( k 0 ,t 0 ,s 0 )
Revised: = I ( k 0 ,t 0 ,s 0 ) that satisfies x I ( k 0 ,t 0 ,s 0 )

Original: − x ′ I ( k 0 ,t 0 ,s 0 )
Revised: − x ′ I ( k 0 ,t 0 ,s 0 )

Original: = 

Revised: =

Original: η ⟨ ξ I ( k 0 ,t 0 ,s 0 ) , e min ⟩ e min ,where e min is the normalized eigenvector associated with the (negative) minimum eigenvalue of H :=∇ 2 f ( (cid:101)
Revised: η ⟨ ξ I ( k 0 ,t 0 ,s 0 ) , e min ⟩ e min ,where e min is the normalized eigenvector associated with the (negative) minimum eigenvalue of H :=∇ 2 f ( (cid:101)

Original: x I ( k 0 ,t 0 ,s 0 ) ) , and all the other randomness are completely same as the ones of { x i } ∞
Revised: x I ( k 0 ,t 0 ,s 0 ) ) , and all the other randomness are completely same as the ones of { x i } ∞

Original: i =
Revised: i =

Original: I ( k 0 ,t 0 ,s 0 ) .
Revised: I ( k 0 ,t 0 ,s 0 ) .

Original: Then, we show that either of the two sequences { x i } or { x ′ i } efficiently escapes the saddle pointsby bounding the norm of the cumulative difference of x i and x ′ i from below.
Revised: Then, we show that either of the two sequences { x i } or { x ′ i } efficiently escapes the saddle pointsby bounding the norm of the cumulative difference of x i and x ′ i from below.

Original: The novel and mostdifficult part of the analysis is to evaluate the norm of the cumulative difference of the deviations∥ (cid:80) J i
Revised: The novel and mostdifficult part of the analysis is to evaluate the norm of the cumulative difference of the deviations∥ (cid:80) J i

Original: = I ( k 0 ,t 0 ,s 0 ) (1 − η H ) J−
Revised: = I ( k 0 ,t 0 ,s 0 ) (1 − η H ) J−

Original: i ( v i − ∇ f ( x i ) − v ′
Revised: i ( v i − ∇ f ( x i ) − v ′

Original: i + ∇ f ( x ′ i ))
Revised: i + ∇ f ( x ′ i ))

Original: ∥ generated by the two sequences, wherev ′ i denotes the BVR estimator at iteration i generated by sequence { x ′ i } .
Revised: ∥ generated by the two sequences, wherev ′ i denotes the BVR estimator at iteration i generated by sequence { x ′ i } .

Original: Proposition 4.3 (Implicit Negative Curvature Exploitation) .
Revised: Proposition 4.3 (Implicit Negative Curvature Exploitation) .

Original: Let I ( k 0 , t 0 , s 

Revised: Let I ( k 0 , t 0 , s

Original: ) ∈
Revised: ) ∈

Original: [ KTS ] ∪ { 0 } .
Revised: [ KTS ] ∪ { 0 } .

Original: Suppose that Assumptions 1, 2, 3, 4 and 5 hold, ∥∇ f ( (cid:101) x
Revised: Suppose that Assumptions 1, 2, 3, 4 and 5 hold, ∥∇ f ( (cid:101) x

Original: I ( k 0 ,t 0 ,s 0 ) )
Revised: I ( k 0 ,t 0 ,s 0 ) )

Original: ∥ ≤ ε and the minimum eigenvalue λ minof H :=
Revised: ∥ ≤ ε and the minimum eigenvalue λ minof H :=

Original: ∇ f ( (cid:101) x
Revised: ∇ f ( (cid:101) x

Original: I ( k 0 ,t 0 ,s 0 ) )
Revised: I ( k 0 ,t 0 ,s 0 ) )

Original: satisfies λ := − λ min > √ ρε .
Revised: satisfies λ := − λ min > √ ρε .

Original: Under K = O ( L/ζ ∧ b ∧ Pb/T ) , if we appropriately choose J I ( k 0 ,t 0 ,s 0 )
Revised: Under K = O ( L/ζ ∧ b ∧ Pb/T ) , if we appropriately choose J I ( k 0 ,t 0 ,s 0 )

Original: = (cid:101) Θ(

Revised: = (cid:101) Θ(

Original: / ( ηλ )) , η = (cid:101)
Revised: / ( ηλ )) , η = (cid:101)

Original: Θ(1 /L
Revised: Θ(1 /L

Original: ∧ 1 / ( Kζ ) ∧ (cid:112) b/K/L ∧ √ Pb/ ( √ KT/L )) ,with F I ( k 0 ,t 0 ,s 0 ) :
Revised: ∧ 1 / ( Kζ ) ∧ (cid:112) b/K/L ∧ √ Pb/ ( √ KT/L )) ,with F I ( k 0 ,t 0 ,s 0 ) :

Original: =
Revised: =

Original: c F η J I ( k 0 ,t 0 ,s 0 ) r 2 and r :=
Revised: c F η J I ( k 0 ,t 0 ,s 0 ) r 2 and r :=

Original: c r ε ( c F = Θ(1) and c r = (cid:101) Θ(1) )
Revised: c r ε ( c F = Θ(1) and c r = (cid:101) Θ(1) )

Original: we havef ( x I ( k 0 ,t 0 ,s 0 )
Revised: we havef ( x I ( k 0 ,t 0 ,s 0 )

Original: + J I
Revised: + J I

Original: ( k 0 ,t 0 ,sp ) )
Revised: ( k 0 ,t 0 ,sp ) )

Original: − f ( x I ( k 0 ,t 0 s 0 ) )
Revised: − f ( x I ( k 0 ,t 0 s 0 ) )

Original: ≤ − F I ( k 0 ,t 0 ,s 0 )
Revised: ≤ − F I ( k 0 ,t 0 ,s 0 )

Original: + Rwith probability at least 1 / 2 − 9 q/ 2 .
Revised: + Rwith probability at least 1 / 2 − 9 q/ 2 .

Original: Here, R 2 := 2 c η η J
Revised: Here, R 2 := 2 c η η J

Original: I
Revised: I

Original: ( k 0 ,t 0 ,s 0)
Revised: ( k 0 ,t 0 ,s 0)

Original: ∧ K K (cid:80) I ( k 0 ,t 0 ,s 0 )
Revised: ∧ K K (cid:80) I ( k 0 ,t 0 ,s 0 )

Original: − 1 i
Revised: − 1 i

Original: = I (0 ,t 0 ,s 0 )
Revised: = I (0 ,t 0 ,s 0 )

Original: ∥ x i +1 − x i ∥ 2 +
Revised: ∥ x i +1 − x i ∥ 2 +

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: (cid:80)
Revised: (cid:80)

Revised: I

Original: [Equation]
Revised: [Equation]

Revised: ∥ x

Revised: [Equation]

Original: − x
Revised: − x

Original: [Equation]
Revised: [Equation]

Original: ∥ 2 for some universal constant c η > 0 .
Revised: ∥ 2 for some universal constant c η > 0 .

Original: Proposition 4.3 says the function value decreases by roughly F I ( k 0 ,t 0 ,s 0 ) and the global model escapessaddle points with probability at least 1 / 2 after J I
Revised: Proposition 4.3 says the function value decreases by roughly F I ( k 0 ,t 0 ,s 0 ) and the global model escapessaddle points with probability at least 1 / 2 after J I

Original: ( k 0 ,t 0 ,s 0 ) local steps.
Revised: ( k 0 ,t 0 ,s 0 ) local steps.

Original: 4.3 Finding Second-Order Stationary Points
Revised: 4.3 Finding Second-Order Stationary Points

Original: In this subsection, we derive final theorem that guarantees the second-order optimality of the globalmodel by combining Propositions 4.1 and 4.3.
Revised: In this subsection, we derive final theorem that guarantees the second-order optimality of the globalmodel by combining Propositions 4.1 and 4.3.

Original: Theorem 4.4 (Final Theorem) .
Revised: Theorem 4.4 (Final Theorem) .

Original: Suppose that Assumptions 1, 2, 3, 4 and 5 hold.
Revised: Suppose that Assumptions 1, 2, 3, 4 and 5 hold.

Original: Under K =
Revised: Under K =

Original: O ( L/ζ ∧ b ∧ Pb/T ) , if we appropriately choose η = (cid:101) Θ(

Revised: O ( L/ζ ∧ b ∧ Pb/T ) , if we appropriately choose η = (cid:101) Θ(

Original: /L ∧ 1 / ( Kζ ) ∧ (cid:112) b/K/L ∧√ Pb/ ( √ KTL )) ,r = (cid:101) Θ( ε ) and S =
Revised: /L ∧ 1 / ( Kζ ) ∧ (cid:112) b/K/L ∧√ Pb/ ( √ KTL )) ,r = (cid:101) Θ( ε ) and S =

Original: Θ(

Revised: Θ(

Original: + ( f ( (cid:101) x 0 ) − f ( x ∗ )) / ( ηKTε 2 )) , with probability at least 1 / 2 , there existsi ∈ [ KTS ] ∪ { 0 } such that (cid:101) x
Revised: + ( f ( (cid:101) x 0 ) − f ( x ∗ )) / ( ηKTε 2 )) , with probability at least 1 / 2 , there existsi ∈ [ KTS ] ∪ { 0 } such that (cid:101) x

Original: i is ε -second-order optimal point of f 4 .
Revised: i is ε -second-order optimal point of f 4 .

Original: Remark (High probability bound) .
Revised: Remark (High probability bound) .

Original: Theorem 4.4 guarantees that Algorithm 1 finds an approximatesecond-order optimal point in KTS iterations with probability at least 1 / 2 .
Revised: Theorem 4.4 guarantees that Algorithm 1 finds an approximatesecond-order optimal point in KTS iterations with probability at least 1 / 2 .

Original: Repeating Algorithm 1log 2 (1 /q ) times guarantees that the same statement holds with probability at least 1 − q .
Revised: Repeating Algorithm 1log 2 (1 /q ) times guarantees that the same statement holds with probability at least 1 − q .

Original: Remark (Communication complexity) .
Revised: Remark (Communication complexity) .

Original: The total number of communication roundsΘ( TS) is given by(cid:101) O (cid:16) T + (cid:16) L/K + ζ + L/ √ Kb + √ TL/ √ KPb (cid:17) ( f ( (cid:101) x 0 ) − f ( x ∗ )) 
Revised: The total number of communication rounds TS is given by (cid:101) O (cid:16) T + (cid:16) L/K + ζ + L/ √ Kb + √ TL/ √ KPb (cid:17) ( f ( (cid:101) x 0 ) − f ( x ∗ )) /ε

Original: /ε 2 (cid:17) .
Revised: 2 (cid:17) .

Original: Given local computationbudget B , we set T :=
Revised: Given local computation budget B , we set T :=

Original: Θ(1 + n/ ( B P )) and Kb :=
Revised: Θ(1 + n/ ( B P )) and Kb :=

Original: Θ( B ) with b ≤ Θ( √B ) .
Revised: Θ( B ) with b ≤ Θ( √B ) .

Original: Then, we have theaveraged number of local computations per communication round Kb + n/ ( PT ) =
Revised: Then, we have the averaged number of local computations per communication round Kb + n/ ( PT ) =

Original: Θ( B ) and thecommunication complexity TS with budget B becomes(cid:101) O (cid:16) 1 + n B P +
Revised: Θ( B ) and the communication complexity TS with budget B becomes

Original: L √B ε 2 + √ nL
Revised: [Equation] This implies that for ζ

Original: B Pε 2 + ζε 2 (cid:17) .
Revised: =

Original: Thisimplies that for ζ = o ( L ) the communication complexity is strictly smaller than the one of minibatch SGD (cid:101)O
Revised: o ( L ) the communication complexity is strictly smaller than the one ofminibatch SGD (cid:101)

Original:  (1 + L/ε 2 + G 2 / ( B Pε 4 )) .
Revised: O (1 + L/ε 2 + G 2 / ( B Pε 4 )) .

Original: Note that the rate matches to the one of BVR-L-SGD [19].
Revised: Note that the rate matches to the one of BVR-L-SGD[19].

Original: Hence, our method finds second-order optimal points without hurting communication efficiency ofthe state-of-the-art first-order optimality guaranteed method.
Revised: Hence, our method finds second-order optimal points without hurting communication efficiencyof the state-of-the-art first-order optimality guaranteed method.

Original: Furthermore, when B → ∞ , we have(cid:101) Θ(1 + ζ/ε 2 ) , that goes to (cid:101) Θ(1) as ζ → 0 . 
Revised: Furthermore, when B → ∞ , we have(cid:101) Θ(1 + ζ/ε 2 ) , that goes to (cid:101) Θ(1) as ζ → 0 . [Figure]

Original: In summary, BVR-L-PSGD enjoys the desirable properties (i) and (ii) described in Section 1.
Revised: In summary, BVR-L-PSGD enjoys the desirable properties (i) and (ii) described in Section 1.

Original: [Figure]Numerical Resutls
Revised: [Figure]

Original: In this section, we give some experimental results to verify our theoretical findings.
Revised: In this section, we give some experimental results to verify our theoretical findings.

Original: Data Preparation.
Revised: Data Preparation.

Original: We artificially generated heterogeneous local datasets from CIFAR10 dataset( https://www.cs.toronto.edu/~kriz/cifar.html ).
Revised: We artificially generated heterogeneous local datasets from CIFAR10 5 dataset.

Original: The data preparation procedure is completely in accordance with [19] and the details are found in [19].
Revised: The data preparation procedure is completely in accordance with [19] and the details are found in[19].

Original: We set homogeneity parameter q to0 .
Revised: We set homogeneity parameter q to 0 .

Original: 35 , which captures how similar the local datasets are ( q = 0 .
Revised: 35 , which captures how similar the local datasets are( q = 0 .

Original: 1 corresponds to I.I.D. case and higherq does to higher heterogeneity).
Revised: 1 corresponds to I.I.D. case and higher q does to higher heterogeneity).

Original: Model.
Revised: Model.

Original: We conducted our experiments using a two-hidden layers fully connected neural networkwith 100 hidden units and softplus activation.
Revised: We conducted our experiments using a two-hidden layers fully connected neural networkwith 100 hidden units and softplus activation.

Original: For loss function, we used the standard cross-entropyloss.
Revised: For loss function, we used the standard cross-entropyloss.

Original: We initialized parameters by uniformly sampling the parameters from [ − 1 / 100 , 1 / 100] .
Revised: We initialized parameters by uniformly sampling the parameters from [ − 1 / 100 , 1 / 100] .

Original: Implemented Algorithms.
Revised: Implemented Algorithms.

Original: Minibatch SGD, Noisy Minibatch SGD, BVR-L-SGD [19] and ourproposed BVR-L-PSGD were implemented.
Revised: BVR-L-SGD [19] and our proposed BVR-L-PSGD were implemented.

Original: We set K = 64 and b = 16 for the latter two methodsand K = 1 and b = 1 , 024 for the former two methods.
Revised: We set K = 64 and b = 16 , and thus B = 1024 .

Original: For Noisy Minibatch SGD and BVR-LPSGD, the noise radius were tuned from r ∈ { 0 . 5 , 2 . 5, 12 . 5 } .
Revised: For BVR-L-PSGD, the noise radius was fixed tor = 2 . 5 .

Original: For each algorithm, we tuned learningrate η from { 0 . 005 , 0 . 01 , 0 . 05 , 0 . 1 , 0 . 5 , 1 . 0 } .
Revised: For each algorithm, we tuned learning rate η from { 0 . 005 , 0 . 01 , 0 . 05 , 0 . 1 , 0 . 5 , 1 . 0 } .

Original: The details of the tuning procedure are found in thesupplementary material.
Revised: Thedetails of the tuning procedure are found in the supplementary material.

Original: Evaluation.
Revised: Evaluation.

Original: We compared the implemented algorithms using six criteria of train gradient norm∥∇ f ( x ) ∥ ; train loss; train accuracy; test gradient norm; test loss and test accuracy against the numberof communication rounds.
Revised: We compared the implemented algorithms using six criteria of train gradient norm∥∇ f ( x ) ∥ ; train loss; train accuracy; test gradient norm; test loss and test accuracy against the numberof communication rounds.

Original: The total number of communication rounds was fixed to 1 , 000 for eachalgorithm.
Revised: The total number of communication rounds was fixed to 1 , 000 for eachalgorithm.

Original: We independently repeated the experiments 5 times and report the mean and standarddeviation of the above criteria.
Revised: We independently repeated the experiments 5 times and report the mean and standarddeviation of the above criteria.

Original: Due to the space limitation, we will only report train gradient norm,train loss and test accuracy in the main paper.
Revised: Due to the space limitation, we will only report train gradient norm,train loss and test accuracy in the main paper.

Original: The full results are found in the supplementary material.
Revised: The full results are found in the supplementary material.

Original: Results.
Revised: Results.

Original: Figure 1 shows the performances of the previous methods and our proposed algorithm.
Revised: Figure 1 shows the performances of BVR-L-SGD and our proposed algorithm.

Original: We can see that the BVR-L-SGD got stuck at a small gradient norm region in initial rounds.
Revised: We cansee that the both algorithms got stuck at a small gradient norm region in initial rounds.

Original: Afterthat BVR-L-SGD showed unstable convergence and took a lot of time to escape the stucked region.
Revised: After that BVR-L-SGD showed unstable convergence and took a lot of time to escape the stucked region.

Original: In contrast, our proposed method efficiently escaped the stucked region and consistently achievesbetter train loss and test accuracy than BVR-L-SGD.Also, our method consistently outperformed Minibatch SGD and Noisy Minibatch SGD.
Revised: Incontrast, our proposed method efficiently escaped the stucked region and consistently achieves bettertrain loss and test accuracy than BVR-L-SGD.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we have studied a new local algorithm called Bias-Variance Reduced Local Perturbed
Revised: In this paper, we have studied a new local algorithm called Bias-Variance Reduced Local Perturbed

Original: SGD (BVR-L-PSGD) based on a combination of the bias-variance reduced gradient estimator withparameter perturbation to efficiently find second-order optimal points in centralized nonconvexdistributed optimization.
Revised: SGD (BVR-L-PSGD) based on a combination of the bias-variance reduced gradient estimator withparameter perturbation to efficiently find second-order optimal points in centralized nonconvexdistributed optimization.

Original: We have shown that BVR-L-PSGD enjoys second-order optimality withouthurting the best known communication complexity for first-order optimality guarantees.
Revised: We have shown that BVR-L-PSGD enjoys second-order optimality withouthurting the best known communication complexity for first-order optimality guarantees.

Original: Particularly,the communication complexity is better than non-local methods when Hessian heterogeneity ζ oflocal datasets is smaller than the smoothness of the local loss L in order sense.
Revised: Particularly,the communication complexity is better than non-local methods when Hessian heterogeneity ζ oflocal datasets is smaller than the smoothness of the local loss L in order sense.

Original: Also, for sufficientlylarge B , the communication complexity of our method approaches to (cid:101) Θ(1) when the local datasetsheterogeneity ζ goes to zero.
Revised: Also, for sufficientlylarge B , the communication complexity of our method approaches to (cid:101) Θ(1) when the local datasetsheterogeneity ζ goes to zero.

Original: The numerical results have validated our theoretical findings.
Revised: The numerical results have validated our theoretical findings.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes] See Section 
Revised: [Yes] See Section 1.

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Section 4.
Revised: [Yes] See Section 4.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See section 2.
Revised: [Yes] See section 2.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] See Section 4 and thesupplementary material.
Revised: [Yes] See Section 4 and thesupplementary material.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes] See Section and the supplementary material.
Revised: [Yes] See Section 5and the supplementary material.

Original: (b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] See Section 5 and the supplementary material.
Revised: [Yes] See Section 5 and the supplementary material.

Original: (c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] See Section 5.
Revised: [Yes] See Section 5.

Original: (d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See the supplementary material.
Revised: [Yes] See the supplementary material.

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Section 5.
Revised: [Yes] See Section 5.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?

Original: [N/A]
Revised: [N/A]