Original: Escaping Saddle Points with Bias-Variance Reduced Local Perturbed SGD for Communication Efficient Nonconvex Distributed Learning
Revised: Escaping Saddle Points with Bias-Variance Reduced Local Perturbed SGD for Communication Efficient Nonconvex Distributed Learning

Original: Abstract
Revised: Abstract

Original: In recent centralized nonconvex distributed learning and federated learning, local methods are one of the promising approaches to reduce communication time.
Revised: In recent centralized nonconvex distributed learning and federated learning, local methods are one of the promising approaches to reduce communication time.

Original: However, existing work has mainly focused on studying first-order optimality guarantees.
Revised: However, existing work has mainly focused on studying first-order optimality guarantees.

Original: On the other side, second-order optimality guaranteed algorithms, i.e., algorithms escaping saddle points, have been extensively studied in the nondistributed optimization literature.
Revised: On the other side, second-order optimality guaranteed algorithms, i.e., algorithms escaping saddle points, have been extensively studied in the nondistributed optimization literature.

Original: In this paper, we study a new local algorithm called Bias-Variance Reduced Local Perturbed SGD (BVR-L-PSGD), that combines the existing bias-variance reduced gradient estimator with parameter perturbation to find second-order optimal points in centralized nonconvex distributed optimization.
Revised: In this paper, we study a new local algorithm called Bias-Variance Reduced Local Perturbed SGD (BVR-L-PSGD), that combines the existing bias-variance reduced gradient estimator with parameter perturbation to find second-order optimal points in centralized nonconvex distributed optimization.

Original: BVR-L-PSGD enjoys second-order optimality with nearly the same communication complexity as the best known one of BVR-L-SGD to find first-order optimality.
Revised: BVR-L-PSGD enjoys second-order optimality with nearly the same communication complexity as the best known one of BVR-L-SGD to find first-order optimality.

Original: Particularly, the communication complexity is better than non-local methods when the local datasets heterogeneity is smaller than the smoothness of the local loss.
Revised: Particularly, the communication complexity is better than non-local methods when the local datasets heterogeneity is smaller than the smoothness of the local loss.

Original: In an extreme case, the communication complexity approaches to (cid:101) Θ(1) when the local datasets heterogeneity goes to zero.
Revised: In an extreme case, the communication complexity approaches to (cid:101) Θ(1) when the local datasets heterogeneity goes to zero.

Original: Numerical results validate our theoretical findings.
Revised: Numerical results validate our theoretical findings.

Original:  Introduction
Revised:  Introduction

Original: Distributed learning is an attractive approach to reduce the total execution time by utilizing the parallel computations.
Revised: Distributed learning is an attractive approach to reduce the total execution time by utilizing the parallel computations.

Original: However, the communication time in distributed learning can be a main bottleneck in the entire process due to huge parameter size typical in deep learning or low bandwidth communication environments.
Revised: However, the communication time in distributed learning can be a main bottleneck in the entire process due to huge parameter size typical in deep learning or low bandwidth communication environments.

Original: To reduce communication time, one of the promising approaches is the usage of local methods such as local SGD (also called as Parallel Restart SGD or FedAvg).
Revised: To reduce communication time, one of the promising approaches is the usage of local methods such as local SGD (also called as Parallel Restart SGD or FedAvg).

Original: In local SGD, each worker independently executes multiple updates of the local model based on his own local dataset and the server periodically communicates and aggregates the local models.
Revised: In local SGD, each worker independently executes multiple updates of the local model based on his own local dataset and the server periodically communicates and aggregates the local models.

Original: Many paper have studied local
Revised: Many paper have studied local

Original: SGD [25, 30, 7, 6, 15, 13, 27, 28].
Revised: SGD [25, 30, 7, 6, 15, 13, 27, 28].

Original: Particularly, for convex objectives, it has been shown in [28], for the first time, the communication complexity (that is the necessary number of communication rounds to achieve given desired optimization error) of local SGD can be smaller than the one of minibatch
Revised: Particularly, for convex objectives, it has been shown in [28], for the first time, the communication complexity (that is the necessary number of communication rounds to achieve given desired optimization error) of local SGD can be smaller than the one of minibatch

Original: SGD when the heterogeneity of the local datasets is extremely small.
Revised: SGD when the heterogeneity of the local datasets is extremely small.

Original: In traditional distributed learning, the local datasets are typically random subsets of the global dataset and in this case the heterogeneity of the local datasets may become quite small.
Revised: In traditional distributed learning, the local datasets are typically random subsets of the global dataset and in this case the heterogeneity of the local datasets may become quite small.

Original: However, in the recent federated learning regimes [16, 24, 18], it is often the case that the heterogeneity of the local datasets is not too small.
Revised: However, in the recent federated learning regimes [16, 24, 18], it is often the case that the heterogeneity of the local datasets is not too small.

Original: Also, the analysis in [28] has only focused on convex cases.
Revised: Also, the analysis in [28] has only focused on convex cases.

Original: Hence, the superiority of local SGD to minibatch SGD is still quite limited.
Revised: Hence, the superiority of local SGD to minibatch SGD is still quite limited.

Original: Recently, more communication efficient local methods than local SGD have been proposed for possibly nonconvex objectives to guarantee first-order optimality [12, 22, 19].
Revised: Recently, more communication efficient local methods than local SGD have been proposed for possibly nonconvex objectives to guarantee first-order optimality [12, 22, 19].

Original: SCAFFOLD [12] is a new local algorithm based on the idea of reducing their called client-drift by using a similar formulation to the variance reduction technique [10].
Revised: SCAFFOLD [12] is a new local algorithm based on the idea of reducing their called client-drift by using a similar formulation to the variance reduction technique [10].

Original: They have shown that the communication complexity of SCAFFOLD can be smaller than the one of minibatch SGD for not too heterogenous local datasets under the quadraticity of the (possibly nonconvex) local objectives, which is quite limited.
Revised: They have shown that the communication complexity of SCAFFOLD can be smaller than the one of minibatch SGD for not too heterogenous local datasets under the quadraticity of the (possibly nonconvex) local objectives, which is quite limited.

Original: For general nonconvex objectives, the communication complexity of SCAFFOLD is same as minibatch SGD.
Revised: For general nonconvex objectives, the communication complexity of SCAFFOLD is same as minibatch SGD.

Original: More recently, Murate and Suzuki [19] have proposed Bias-Variance Reduced
Revised: More recently, Murate and Suzuki [19] have proposed Bias-Variance Reduced

Original: Local SGD (BVR-L-SGD).
Revised: Local SGD (BVR-L-SGD).

Original: BVR-L-SGD utilizes their proposed bias-variance reduced estimator that simultaneously reduces the bias caused by local gradient steps and the variance caused by stochastization of the gradients in local optimization based on the formulation of SARAH like variance reduction [20].
Revised: BVR-L-SGD utilizes their proposed bias-variance reduced estimator that simultaneously reduces the bias caused by local gradient steps and the variance caused by stochastization of the gradients in local optimization based on the formulation of SARAH like variance reduction [20].

Original: They have shown that the communication complexity of BVR-L-SGD is smaller than minibatch SGD for not too heterogeneous local datasets for general nonconvex objectives .
Revised: They have shown that the communication complexity of BVR-L-SGD is smaller than minibatch SGD for not too heterogeneous local datasets for general nonconvex objectives .

Original: Specifically, BVR-L-SGD is superior to minibatch SGD when the Hessian heterogeneity of the local datasets is small relative to the smoothness of the local loss in order sense.
Revised: Specifically, BVR-L-SGD is superior to minibatch SGD when the Hessian heterogeneity of the local datasets is small relative to the smoothness of the local loss in order sense.

Original: On the other side, there are vast work that has studied second-order optimality guarantees, which is much more challenging to ensure but desirable than first-order one, in non-distributed nonconvex optimization.
Revised: On the other side, there are vast work that has studied second-order optimality guarantees, which is much more challenging to ensure but desirable than first-order one, in non-distributed nonconvex optimization.

Original: Several approaches are known and one of the simplest approaches is parameter perturbation [4, 8, 9].
Revised: Several approaches are known and one of the simplest approaches is parameter perturbation [4, 8, 9].

Original: However, almost all existing analysis of local methods have only focused on achieving first-order optimality.
Revised: However, almost all existing analysis of local methods have only focused on achieving first-order optimality.

Original: As an exception, Vlaski et al.
Revised: As an exception, Vlaski et al.

Original: [26] have analysed second-order guarantees of local SGD with parameter perturbation, but the obtained communication complexity is much worse than the one of minibatch SGD and no benefit of localization has been shown.
Revised: [26] have analysed second-order guarantees of local SGD with parameter perturbation, but the obtained communication complexity is much worse than the one of minibatch SGD and no benefit of localization has been shown.

Original: Open question.
Revised: Open question.

Original: For local methods, it is not well-studied how to find second-order optimal points with low communication cost and thus we have the following research questions:
Revised: For local methods, it is not well-studied how to find second-order optimal points with low communication cost and thus we have the following research questions:

Original: Is there a first-order distributed optimization algorithm with second-order optimality guarantees which satisfies that (i) the communication complexity is smaller than non-local methods for not too heterogeneous local datasets; and (ii) the communication complexity approaches to Θ(1) when the heterogeneity of local datasets goes to zero?
Revised: Is there a first-order distributed optimization algorithm with second-order optimality guarantees which satisfies that (i) the communication complexity is smaller than non-local methods for not too heterogeneous local datasets; and (ii) the communication complexity approaches to Θ(1) when the heterogeneity of local datasets goes to zero?

Original: Note that the both properties are desirable in distributed optimization.
Revised: Note that the both properties are desirable in distributed optimization.

Original: We expect that local methods are superior to non-local methods for not highly heterogeneous local datasets.
Revised: We expect that local methods are superior to non-local methods for not highly heterogeneous local datasets.

Original: Furthermore, when the local datasets are nearly identical, it is expected that a few communications are sufficient to optimize the global objective.
Revised: Furthermore, when the local datasets are nearly identical, it is expected that a few communications are sufficient to optimize the global objective.

Original: Only in the case of first-order optimality , Murata and Suzuki [19] have shown that their proposed BVR-L-SGD satisfies (i) and (ii), and the question has been positively answered.
Revised: Only in the case of first-order optimality , Murata and Suzuki [19] have shown that their proposed BVR-L-SGD satisfies (i) and (ii), and the question has been positively answered.

Original: However, the question is still open in the case of second-order optimality 3 .
Revised: However, the question is still open in the case of second-order optimality 3 .

Original: Main Contributions
Revised: Main Contributions

Original: We propose a new local algorithm called Bias-Variance Reduced Local Perturbed SGD (BVR-LPSGD) for nonconvex distributed learning to efficiently find second-order optimal points, which positively answered the above research questions.
Revised: We propose a new local algorithm called Bias-Variance Reduced Local Perturbed SGD (BVR-LPSGD) for nonconvex distributed learning to efficiently find second-order optimal points, which positively answered the above research questions.

Original: The algorithm is based on a simple combination of the existing bias and variance reduced gradient estimator and parameter perturbation.
Revised: The algorithm is based on a simple combination of the existing bias and variance reduced gradient estimator and parameter perturbation.

Original: In our algorithm, parameter perturbation is carried out at every to them [29, 1].
Revised: In our algorithm, parameter perturbation is carried out at every to them [29, 1].

Original: However, this naive approach does not possess the aforementioned property (ii) because the generic framework requires at least Ω(1 /ε 3 / 2 ) communication rounds to guarantee second-order optimality due to the multiple negative curvature exploitation steps in their framework for any communication efficient algorithms with first-order optimality guarantees.
Revised: However, this naive approach does not possess the aforementioned property (ii) because the generic framework requires at least Ω(1 /ε 3 / 2 ) communication rounds to guarantee second-order optimality due to the multiple negative curvature exploitation steps in their framework for any communication efficient algorithms with first-order optimality guarantees.

Original: Also, this approach requires explicit negative curvature exploitation, that is complicated and makes the whole algorithm less practical.
Revised: Also, this approach requires explicit negative curvature exploitation, that is complicated and makes the whole algorithm less practical.

Original: [Table]
Revised: [Table]

Original: Assumptions
Revised: Assumptions

Original: Guarantee
Revised: Guarantee

Original: Minibatch SGD
Revised: Minibatch SGD

Original: [Table]
Revised: [Table]

Original: Noisy Minibatch SGD [9]
Revised: Noisy Minibatch SGD [9]

Original: [Table]
Revised: [Table]

Original: Minibatch SARAH [21]
Revised: Minibatch SARAH [21]

Original: [Table]
Revised: [Table]

Original: SSRGD [17]
Revised: SSRGD [17]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Local SGD [30]
Revised: Local SGD [30]

Original: [Table]
Revised: [Table]

Original: SCAFFOLD [12]
Revised: SCAFFOLD [12]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: order
Revised: order

Original: SCAFFOLD [12]
Revised: SCAFFOLD [12]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: , quadraticity
Revised: , quadraticity

Original: [Table]
Revised: [Table]

Original: BVR
Revised: BVR

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: (this paper)
Revised: (this paper)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ( L ) = Θ(1) .
Revised: ( L ) = Θ(1) .

Original: local update and it is not necessary to determine whether or not to add noise by checking the norm of the global gradients, which is often required in several previous non-distributed algorithms [5, 17].
Revised: local update and it is not necessary to determine whether or not to add noise by checking the norm of the global gradients, which is often required in several previous non-distributed algorithms [5, 17].

Original: We analyse BVR-L-PSGD for general nonconvex smooth objectives.
Revised: We analyse BVR-L-PSGD for general nonconvex smooth objectives.

Original: The most challenging part of our analysis is to ensure that our algorithm efficiently escapes global saddle points even in local optimization .
Revised: The most challenging part of our analysis is to ensure that our algorithm efficiently escapes global saddle points even in local optimization .

Original: To realize this, it is necessary to analyse the behavior of the bias-variance reduced estimator around the saddle points by carefully evaluating the degree of some kind of asymptotic consistency of the estimator around the saddle points.
Revised: To realize this, it is necessary to analyse the behavior of the bias-variance reduced estimator around the saddle points by carefully evaluating the degree of some kind of asymptotic consistency of the estimator around the saddle points.

Original: This point has never been pursued in previous work and has a unique difficulty of our analysis.
Revised: This point has never been pursued in previous work and has a unique difficulty of our analysis.

Original: The comparison of the communication complexities of our method with the most relevant existing results is given in Table 1.
Revised: The comparison of the communication complexities of our method with the most relevant existing results is given in Table 1.

Original: Our proposed method enjoys second-order optimality with nearly the same communication complexity as the one of BVR-L-SGD, which achieves the best known communication complexity to achieve first-order optimality.
Revised: Our proposed method enjoys second-order optimality with nearly the same communication complexity as the one of BVR-L-SGD, which achieves the best known communication complexity to achieve first-order optimality.

Original: This means that our method finds second-order optimal points without hurting the communication efficiency of the state-of-the-art first-order optimality guaranteed method.
Revised: This means that our method finds second-order optimal points without hurting the communication efficiency of the state-of-the-art first-order optimality guaranteed method.

Original: Particularly, the communication complexity is better than minibatch SGD when
Revised: Particularly, the communication complexity is better than minibatch SGD when

Original: Hessian heterogeneity ζ is small relative to smoothness L .
Revised: Hessian heterogeneity ζ is small relative to smoothness L .

Original: Also, the communication complexity approaches to (cid:101) O
Revised: Also, the communication complexity approaches to (cid:101) O

Original: (1) when heterogeneity ζ goes to zero and the local computation budget B (see
Revised: (1) when heterogeneity ζ goes to zero and the local computation budget B (see

Original: Section 2) goes to infinity.
Revised: Section 2) goes to infinity.

Original: Hence, our method enjoys the aforementioned two desired properties.
Revised: Hence, our method enjoys the aforementioned two desired properties.

Original: Related Work
Revised: Related Work

Original: Here, we briefly review the related studies to our paper.
Revised: Here, we briefly review the related studies to our paper.

Original: Local methods.
Revised: Local methods.

Original: Several recent papers have studied local algorithms combined with variance reduction technique [23, 2, 14, 11].
Revised: Several recent papers have studied local algorithms combined with variance reduction technique [23, 2, 14, 11].

Original: Sharma et al.
Revised: Sharma et al.

Original: [23] have proposed a local variant of SPIDER [3] and shown that the proposed algorithm achieves the optimal total computational complexity.
Revised: [23] have proposed a local variant of SPIDER [3] and shown that the proposed algorithm achieves the optimal total computational complexity.

Original: However, the communication complexity essentially matches the ones of non-local SARAH and no advantage of localization has been shown.
Revised: However, the communication complexity essentially matches the ones of non-local SARAH and no advantage of localization has been shown.

Original: Khanduri et al.
Revised: Khanduri et al.

Original: [14] have proposed STEM and its variants based on their called two-sided momentum, but again the communication complexity does not improve non-local methods.
Revised: [14] have proposed STEM and its variants based on their called two-sided momentum, but again the communication complexity does not improve non-local methods.

Original: Also, Das et al.
Revised: Also, Das et al.

Original: [2] have considered a SPIDER like local algorithm called FedGLOMO but the derived communication complexity is even worse than minibatch SARAH.
Revised: [2] have considered a SPIDER like local algorithm called FedGLOMO but the derived communication complexity is even worse than minibatch SARAH.

Original: Karimireddy et al.
Revised: Karimireddy et al.

Original: [11] have proposed Mime, which is a general framework to mitigate client-drift.
Revised: [11] have proposed Mime, which is a general framework to mitigate client-drift.

Original: Particularly, under δ -Bounded Hessian Dissimilarity (BHD) 4 , their MimeMVR achieves communication complexity of 1 / ( Pε 2 )+ δ/ ( √ Pε 3 )
Revised: Particularly, under δ -Bounded Hessian Dissimilarity (BHD) 4 , their MimeMVR achieves communication complexity of 1 / ( Pε 2 )+ δ/ ( √ Pε 3 )

Original: + δ/ε 2 when B → ∞ , that is better than the one of minibatch SGD 1 /ε 2 when δ ≤ √ Pε .
Revised: + δ/ε 2 when B → ∞ , that is better than the one of minibatch SGD 1 /ε 2 when δ ≤ √ Pε .

Original: However, the asymptotic rate is still worse than the one of BVR-L-SGD ζ/ε 2 because ζ ≤ δ always holds.
Revised: However, the asymptotic rate is still worse than the one of BVR-L-SGD ζ/ε 2 because ζ ≤ δ always holds.

Original: Second-order guarantee.
Revised: Second-order guarantee.

Original: Neon [29] and Neon2 [1] are generic first-order methods with second-order guarantees, that repeatedly run a first-order guaranteed algorithm and negative curvature descent.
Revised: Neon [29] and Neon2 [1] are generic first-order methods with second-order guarantees, that repeatedly run a first-order guaranteed algorithm and negative curvature descent.

Original: Another approach is a parameter perturbation for SGD.
Revised: Another approach is a parameter perturbation for SGD.

Original: For the first time, Ge et al.
Revised: For the first time, Ge et al.

Original: [4] have shown that
Revised: [4] have shown that

Original: SGD with a simple parameter perturbation escapes saddle points efficiently.
Revised: SGD with a simple parameter perturbation escapes saddle points efficiently.

Original: Later, the analysis has been refined by [8, 9].
Revised: Later, the analysis has been refined by [8, 9].

Original: Recently, applying variance reduction technique to second-order guaranteed methods has been also studied [5, 17] and particularly Li et al.
Revised: Recently, applying variance reduction technique to second-order guaranteed methods has been also studied [5, 17] and particularly Li et al.

Original: [17] have proposed SSRGD that combines SARAH [20] with parameter perturbation and shown that SSRGD nearly achieves the optimal computational complexity with second-order optimality guarantees.
Revised: [17] have proposed SSRGD that combines SARAH [20] with parameter perturbation and shown that SSRGD nearly achieves the optimal computational complexity with second-order optimality guarantees.

Original:  Problem Definition and Assumptions
Revised:  Problem Definition and Assumptions

Original: In this section, we first introduce several notations and definitions used in this paper.
Revised: In this section, we first introduce several notations and definitions used in this paper.

Original: Then, the problem settings are described and theoretical assumptions used in our analysis are given.
Revised: Then, the problem settings are described and theoretical assumptions used in our analysis are given.

Original: Notation.
Revised: Notation.

Original: ∥ · ∥ denotes the Euclidean L 2 norm ∥ · ∥ 2 : ∥ x ∥ =
Revised: ∥ · ∥ denotes the Euclidean L 2 norm ∥ · ∥ 2 : ∥ x ∥ =

Original: (cid:112)(cid:80) i x 2 i for vector x .
Revised: (cid:112)(cid:80) i x 2 i for vector x .

Original: For a matrix X ,
Revised: For a matrix X ,

Original: ∥ X ∥ denotes the induced norm by the Euclidean L 2 norm.
Revised: ∥ X ∥ denotes the induced norm by the Euclidean L 2 norm.

Original: For a natural number m , [ m ] means the set { 1 , 2 , . . .
Revised: For a natural number m , [ m ] means the set { 1 , 2 , . . .

Original: , m } .
Revised: , m } .

Original: For a set A , # A means the number of elements, which is possibly ∞ .
Revised: For a set A , # A means the number of elements, which is possibly ∞ .

Original: For any number a, b , a ∨ b and a ∧ b denote max { a, b } and min { a, b } respectively.
Revised: For any number a, b , a ∨ b and a ∧ b denote max { a, b } and min { a, b } respectively.

Original: We denote the uniform distribution over A by Unif( A ) .
Revised: We denote the uniform distribution over A by Unif( A ) .

Original: Given K, T, S ∈ N , let I ( k, t, s ) be integer k + Kt + KTs for k ∈ [ K ] ∪ { 0 } ,
Revised: Given K, T, S ∈ N , let I ( k, t, s ) be integer k + Kt + KTs for k ∈ [ K ] ∪ { 0 } ,

Original: t ∈
Revised: t ∈

Original: [ T − 1] ∪ { 0 } and s ∈ [ S − 1] ∪ { 0 } .
Revised: [ T − 1] ∪ { 0 } and s ∈ [ S − 1] ∪ { 0 } .

Original: Note that
Revised: Note that

Original: I ( K, t, s ) =
Revised: I ( K, t, s ) =

Original: I (0 , t + 1 , s ) and
Revised: I (0 , t + 1 , s ) and

Original: I ( k, T, s ) =
Revised: I ( k, T, s ) =

Original: I ( k, 0 , s + 1) for k ∈
Revised: I ( k, 0 , s + 1) for k ∈

Original: [ K ] ∪ { 0 } , t ∈
Revised: [ K ] ∪ { 0 } , t ∈

Original: [ T − 1] ∪ { 0 } and s ∈ [ S − 1] ∪ { 0 } .
Revised: [ T − 1] ∪ { 0 } and s ∈ [ S − 1] ∪ { 0 } .

Original: B dr denotes the set { x ∈ R d |∥ x ∥ ≤ r } , which is the Euclidean ball in R d with radius r .
Revised: B dr denotes the set { x ∈ R d |∥ x ∥ ≤ r } , which is the Euclidean ball in R d with radius r .

Original: Definition 2.1 (Gradient Lipschitzness) .
Revised: Definition 2.1 (Gradient Lipschitzness) .

Original: A differentiable function f : R d → R is L -gradient
Revised: A differentiable function f : R d → R is L -gradient

Original: Lipschitz if ∥∇ f ( x ) − ∇ f ( y ) ∥
Revised: Lipschitz if ∥∇ f ( x ) − ∇ f ( y ) ∥

Original: ≤ L ∥ x − y ∥ , ∀ x, y ∈ R d .
Revised: ≤ L ∥ x − y ∥ , ∀ x, y ∈ R d .

Original: Definition 2.2 (Hessian Lipschitzness) .
Revised: Definition 2.2 (Hessian Lipschitzness) .

Original: A twice differentiable function f : R d → R is ρ -Hessian
Revised: A twice differentiable function f : R d → R is ρ -Hessian

Original: Lipschitz if (cid:13)(cid:13) ∇ 2 f ( x ) − ∇ 2 f ( y ) (cid:13)(cid:13) ≤
Revised: Lipschitz if (cid:13)(cid:13) ∇ 2 f ( x ) − ∇ 2 f ( y ) (cid:13)(cid:13) ≤

Original: ρ ∥ x − y ∥ , ∀ x, y ∈ R d .
Revised: ρ ∥ x − y ∥ , ∀ x, y ∈ R d .

Original: Definition 2.3 (Second-order optimality) .
Revised: Definition 2.3 (Second-order optimality) .

Original: For a ρ -Hessian Lipschitz function f , x ∈ R d is an ε -second-order optimal point of f if ∥∇ f ( x ) ∥ ≤ ε and ∇ 2 f ( x )
Revised: For a ρ -Hessian Lipschitz function f , x ∈ R d is an ε -second-order optimal point of f if ∥∇ f ( x ) ∥ ≤ ε and ∇ 2 f ( x )

Original: ⪰ −√ ρεI .
Revised: ⪰ −√ ρεI .

Original: 2.1 Problem Settings
Revised: 2.1 Problem Settings

Original: Objective function.
Revised: Objective function.

Original: We want to minimize nonconvex smooth objective f ( x ) :
Revised: We want to minimize nonconvex smooth objective f ( x ) :

Original: = 1 P (cid:80) Pp =1 f p ( x ) , where f p ( x ) :=
Revised: = 1 P (cid:80) Pp =1 f p ( x ) , where f p ( x ) :=

Original: E z ∼ D p [ ℓ ( x, z )] for x ∈ R d , where D p is the data distribution associated with worker p .
Revised: E z ∼ D p [ ℓ ( x, z )] for x ∈ R d , where D p is the data distribution associated with worker p .

Original: In this paper, we focus on offline settings (i.e., #supp( D p ) <
Revised: In this paper, we focus on offline settings (i.e., #supp( D p ) <

Original: ∞ for every p ∈
Revised: ∞ for every p ∈

Original: [ P ] ) for simple presentation.
Revised: [ P ] ) for simple presentation.

Original: It is easy to extend our results to online settings.
Revised: It is easy to extend our results to online settings.

Original: Also, just for simplicity, it is assumed that each local dataset has an equal number of samples, i.e., #supp( D p ) =
Revised: Also, just for simplicity, it is assumed that each local dataset has an equal number of samples, i.e., #supp( D p ) =

Original: n/P for every p, p ′ ∈
Revised: n/P for every p, p ′ ∈

Original: [ P ] , where n is the total number of samples.
Revised: [ P ] , where n is the total number of samples.

Original: Optimization criteria.
Revised: Optimization criteria.

Original: Since objective function f is nonconvex, it is generally difficult to find a global minima of f .
Revised: Since objective function f is nonconvex, it is generally difficult to find a global minima of f .

Original: Previous work in distributed learning has mainly focused on finding first-order stationary points of f .
Revised: Previous work in distributed learning has mainly focused on finding first-order stationary points of f .

Original: In this study, we aim to find ε -second-order stationary points of f in distributed learning settings.
Revised: In this study, we aim to find ε -second-order stationary points of f in distributed learning settings.

Original: Data access constraints and communication settings.
Revised: Data access constraints and communication settings.

Original: It is assumed that each worker p can only access the own data distribution D p without communication.
Revised: It is assumed that each worker p can only access the own data distribution D p without communication.

Original: Aggregation (e.g., summation) of all the worker’s d -dimensional parameters or broadcast of a d -dimensional parameter from one worker to the other workers can be realized by single communication.
Revised: Aggregation (e.g., summation) of all the worker’s d -dimensional parameters or broadcast of a d -dimensional parameter from one worker to the other workers can be realized by single communication.

Original: Evaluation criteria: communication complexity.
Revised: Evaluation criteria: communication complexity.

Original: In this paper, we compare communication complexities of optimization algorithms to satisfy the aforementioned optimization criteria.
Revised: In this paper, we compare communication complexities of optimization algorithms to satisfy the aforementioned optimization criteria.

Original: In typical situations, single communication is more time-consuming than single stochastic gradient computation.
Revised: In typical situations, single communication is more time-consuming than single stochastic gradient computation.

Original: Let C be the single communication cost and G be the single stochastic gradient computation cost.
Revised: Let C be the single communication cost and G be the single stochastic gradient computation cost.

Original: Using these notations, C ≥ G is assumed.
Revised: Using these notations, C ≥ G is assumed.

Original: We expect that increasing the number of available stochastic gradients in a single communication round leads to faster convergence.
Revised: We expect that increasing the number of available stochastic gradients in a single communication round leads to faster convergence.

Original: Hence, it is natural to increase the number of stochastic gradient computations in a single communication round unless the total stochastic gradient computation time exceeds C to reduce the total running time.
Revised: Hence, it is natural to increase the number of stochastic gradient computations in a single communication round unless the total stochastic gradient computation time exceeds C to reduce the total running time.

Original: This motivates the concept of local computation budget B ( ≤ C / G ): given a communication and computational environment, it is assumed that each worker can only computes at most B single stochastic gradients per communication round on average .
Revised: This motivates the concept of local computation budget B ( ≤ C / G ): given a communication and computational environment, it is assumed that each worker can only computes at most B single stochastic gradients per communication round on average .

Original: Then, we compare the communication complexity, that is the total number of communication rounds of a distributed optimization algorithm to achieve the desired optimization accuracy .
Revised: Then, we compare the communication complexity, that is the total number of communication rounds of a distributed optimization algorithm to achieve the desired optimization accuracy .

Original: From the definition, we can see that the communication complexity on a fixed local computation budget B :=
Revised: From the definition, we can see that the communication complexity on a fixed local computation budget B :=

Original: C / G captures the best achievable total running time of an algorithm.
Revised: C / G captures the best achievable total running time of an algorithm.

Original: 2.2 Theoretical Assumptions
Revised: 2.2 Theoretical Assumptions

Original: In this paper, we assume the following five assumptions.
Revised: In this paper, we assume the following five assumptions.

Original: The first one has already been adopted in several previous work [12, 19].
Revised: The first one has already been adopted in several previous work [12, 19].

Original: The other ones are standard in the nonconvex optimization literature to guarantees second-order optimality.
Revised: The other ones are standard in the nonconvex optimization literature to guarantees second-order optimality.

Original: Assumption 1 (Hessian heterogeneity [12, 19]) .
Revised: Assumption 1 (Hessian heterogeneity [12, 19]) .

Original: { f p } Pp =1 is second-order ζ -heterogeneous, i.e., for any p, p ′ ∈
Revised: { f p } Pp =1 is second-order ζ -heterogeneous, i.e., for any p, p ′ ∈

Original: [ P ] ,
Revised: [ P ] ,

Original: [Equation]
Revised: [Equation]

Original: Assumption 1 characterizes the heterogeneity of local objectives { f p } Pp =1 in terms of Hessians and has a important role in our analysis.
Revised: Assumption 1 characterizes the heterogeneity of local objectives { f p } Pp =1 in terms of Hessians and has a important role in our analysis.

Original: Intuitively, we expect that relatively small heterogeneity parameter ζ to the smoothness parameter L (defined in Assumption 2) reduces the necessary number of communication rounds to optimize the global objective.
Revised: Intuitively, we expect that relatively small heterogeneity parameter ζ to the smoothness parameter L (defined in Assumption 2) reduces the necessary number of communication rounds to optimize the global objective.

Original: Especially when the local objectives are identical, i.e.,
Revised: Especially when the local objectives are identical, i.e.,

Original: D p
Revised: D p

Original: = D p ′ for every p, p ′ ∈
Revised: = D p ′ for every p, p ′ ∈

Original: [ P ] , ζ becomes zero.
Revised: [ P ] , ζ becomes zero.

Original: When each D p is the empirical distribution of n/P
Revised: When each D p is the empirical distribution of n/P

Original: IID samples from common data distribution D , we have ∥∇ 2 f p ( x ) − ∇ 2 f p
Revised: IID samples from common data distribution D , we have ∥∇ 2 f p ( x ) − ∇ 2 f p

Original: ( x ) ∥ ≤ (cid:101)
Revised: ( x ) ∥ ≤ (cid:101)

Original: Θ( (cid:112) P/nL ) with high probability by matrix Hoeffding’s inequality under Assumption 2 for fixed x .
Revised: Θ( (cid:112) P/nL ) with high probability by matrix Hoeffding’s inequality under Assumption 2 for fixed x .

Original: Hence, in traditional distributed learning regimes, Assumption 1 naturally holds.
Revised: Hence, in traditional distributed learning regimes, Assumption 1 naturally holds.

Original: An important remark is that
Revised: An important remark is that

Original: Assumption 2 implies ζ ≤ 2 L , i.e., the heterogeneity is bounded by the smoothness .
Revised: Assumption 2 implies ζ ≤ 2 L , i.e., the heterogeneity is bounded by the smoothness .

Original: Even in federated learning regimes, we expect ζ ≪ 2 L for some problems practically.
Revised: Even in federated learning regimes, we expect ζ ≪ 2 L for some problems practically.

Original: Assumption 2 (Gradient Lipschitzness) .
Revised: Assumption 2 (Gradient Lipschitzness) .

Original: ∀ p ∈
Revised: ∀ p ∈

Original: [ P ] , z ∈ supp( D p ) , ℓ ( · , z ) is L -gradient Lipschitz.
Revised: [ P ] , z ∈ supp( D p ) , ℓ ( · , z ) is L -gradient Lipschitz.

Original: Assumption 3 (Existence of global optimum) .
Revised: Assumption 3 (Existence of global optimum) .

Original: f has a global minimizer x ∗ ∈ R d .
Revised: f has a global minimizer x ∗ ∈ R d .

Original: Assumption 4 (Hessian Lipschitzness) .
Revised: Assumption 4 (Hessian Lipschitzness) .

Original: ∀ p ∈
Revised: ∀ p ∈

Original: [ P ] , z ∈ supp( D p ) , ℓ ( · , z ) is ρ -Hessian Lipschitz.
Revised: [ P ] , z ∈ supp( D p ) , ℓ ( · , z ) is ρ -Hessian Lipschitz.

Original: Assumption 5 (Bounded stochastic gradient) .
Revised: Assumption 5 (Bounded stochastic gradient) .

Original: ∀ p ∈
Revised: ∀ p ∈

Original: [ P ] , z ∈ supp( D p ) , ∇ ℓ ( · , z ) is G -bounded, i.e., ∥∇ ℓ ( x, z )
Revised: [ P ] , z ∈ supp( D p ) , ∇ ℓ ( · , z ) is G -bounded, i.e., ∥∇ ℓ ( x, z )

Original: ∥ ≤ G, ∀ x ∈ R d .
Revised: ∥ ≤ G, ∀ x ∈ R d .

Original: In our analysis, G has no significant impact because G only depends on our theoretical communication complexity in logarithmic order.
Revised: In our analysis, G has no significant impact because G only depends on our theoretical communication complexity in logarithmic order.

Original:  Main Ideas and Proposed Algorithm
Revised:  Main Ideas and Proposed Algorithm

Original: Our proposed algorithm is based on a natural combination of (i) Bias-Variance Reduced (BVR) estimator ; and (ii) parameter perturbation at each local update .
Revised: Our proposed algorithm is based on a natural combination of (i) Bias-Variance Reduced (BVR) estimator ; and (ii) parameter perturbation at each local update .

Original: The first idea has been proposed by [19] to find first-order stationary points with small communication complexity.
Revised: The first idea has been proposed by [19] to find first-order stationary points with small communication complexity.

Original: The second one is a well-known approach to find second-order stationary points in non-distributed nonconvex optimization [4, 8, 9].
Revised: The second one is a well-known approach to find second-order stationary points in non-distributed nonconvex optimization [4, 8, 9].

Original: In this section, we illustrate these two ideas and provide its concrete procedures.
Revised: In this section, we illustrate these two ideas and provide its concrete procedures.

Original: 3.1 Review of BVR Estimator [19]
Revised: 3.1 Review of BVR Estimator [19]

Original: The bias-variance reduced estimator aims to efficiently find first-order stationary points by simultaneously reducing the bias caused by local gradient descent steps and the variance caused by stochastization of the used gradients.
Revised: The bias-variance reduced estimator aims to efficiently find first-order stationary points by simultaneously reducing the bias caused by local gradient descent steps and the variance caused by stochastization of the used gradients.

Original: First we consider why the standard local SGD is not sufficient to achieve fast convergence and sometimes slower than minibatch SGD.
Revised: First we consider why the standard local SGD is not sufficient to achieve fast convergence and sometimes slower than minibatch SGD.

Original: Recall that in local SGD each worker takes the update rules
Revised: Recall that in local SGD each worker takes the update rules

Original: 1: Add noise x 0 = (cid:101) x 0 + ηξ − 1 , where ξ − 1 ∼ Unif( B dr ) .
Revised: 1: Add noise x 0 = (cid:101) x 0 + ηξ − 1 , where ξ − 1 ∼ Unif( B dr ) .

Original: 2: for s = 0 to S − 1 do 3: for p = 1 to P in parallel do 4: v ( p ) I (0 , 0 ,s ) = ∇ f p ( x I (0 , 0 ,s ) ) .
Revised: 2: for s = 0 to S − 1 do 3: for p = 1 to P in parallel do 4: v ( p ) I (0 , 0 ,s ) = ∇ f p ( x I (0 , 0 ,s ) ) .

Original: 5: end for 6:
Revised: 5: end for 6:

Original: Communicate { v ( p ) I (0 , 0 ,s ) } Pp =1 .
Revised: Communicate { v ( p ) I (0 , 0 ,s ) } Pp =1 .

Original: Set v I (0 , 0 ,s )
Revised: Set v I (0 , 0 ,s )

Original: = 1 P (cid:80)
Revised: = 1 P (cid:80)

Original: Pp =1 v ( p ) I (0 , 0 ,s ) .
Revised: Pp =1 v ( p ) I (0 , 0 ,s ) .

Original: 7: for t = 0 to T − 1 do 8: for p = 1 to P in parallel do 9: g ( p ) I (0 ,t,s )
Revised: 7: for t = 0 to T − 1 do 8: for p = 1 to P in parallel do 9: g ( p ) I (0 ,t,s )

Original: = 1 Kb (cid:80)
Revised: = 1 Kb (cid:80)

Original: Kbl =1 ∇ ℓ ( x I (0 ,t,s ) , z l,I (0 ,t,s ) ) , 10: g ( p ) , ref I (0 ,t,s )
Revised: Kbl =1 ∇ ℓ ( x I (0 ,t,s ) , z l,I (0 ,t,s ) ) , 10: g ( p ) , ref I (0 ,t,s )

Original: = 1 Kb (cid:80)
Revised: = 1 Kb (cid:80)

Original: Kbl =1 ∇ ℓ ( x I (0 ,t − 1 ,s ) , z l,I (0 ,t,s ))
Revised: Kbl =1 ∇ ℓ ( x I (0 ,t − 1 ,s ) , z l,I (0 ,t,s ))

Original: ( z l,I (0 ,t,s ) i.i.d.
Revised: ( z l,I (0 ,t,s ) i.i.d.

Original: ∼ D p ).
Revised: ∼ D p ).

Original: 11: v ( p ) I (0 ,t,s )
Revised: 11: v ( p ) I (0 ,t,s )

Original: = 1 t ≥ 1 ( g ( p ) I (0 ,t,s ) − g ( p ) , ref I (0 ,t,s ) + v ( p ) I (0 ,t − 1 ,s ) )
Revised: = 1 t ≥ 1 ( g ( p ) I (0 ,t,s ) − g ( p ) , ref I (0 ,t,s ) + v ( p ) I (0 ,t − 1 ,s ) )

Original: + 1 t =0 v ( p ) I (0 , 0 ,s ) .
Revised: + 1 t =0 v ( p ) I (0 , 0 ,s ) .

Original: 12: end for 13:
Revised: 12: end for 13:

Original: Communicate { v ( p ) I (0 ,t,s ) } Pp =1 .
Revised: Communicate { v ( p ) I (0 ,t,s ) } Pp =1 .

Original: Set v I (0 ,t,s )
Revised: Set v I (0 ,t,s )

Original: = 1 P (cid:80)
Revised: = 1 P (cid:80)

Original: Pp =1 v ( p ) I (0 ,t,s ) .
Revised: Pp =1 v ( p ) I (0 ,t,s ) .

Original: Randomly select p worker p t,s runs local optimization.
Revised: Randomly select p worker p t,s runs local optimization.

Original: 15: for k = 0 to K − 1 do 16:
Revised: 15: for k = 0 to K − 1 do 16:

Original: b k = 1 k ≡ 0 (mod ⌈ √ K ⌉ )
Revised: b k = 1 k ≡ 0 (mod ⌈ √ K ⌉ )

Original: ⌈ √ K ⌉ b + 1 k ̸≡ 

Revised: ⌈ √ K ⌉ b + 1 k ̸≡

Original: (mod ⌈ √ K ⌉ ) b .
Revised: (mod ⌈ √ K ⌉ ) b .

Original: 17: g I ( k,t,s ) =
Revised: 17: g I ( k,t,s ) =

Original: 1 b k (cid:80) b k l =1 ∇ ℓ ( x I ( k,t,s ) , z l,I ( k,t,s ) ) , 18: g ref I ( k,t,s ) =
Revised: 1 b k (cid:80) b k l =1 ∇ ℓ ( x I ( k,t,s ) , z l,I ( k,t,s ) ) , 18: g ref I ( k,t,s ) =

Original: 1 b k (cid:80) b k l =1 ∇ ℓ ( x I ( k − 1 ,t,s ) ) , z l,I ( k,t,s ) )
Revised: 1 b k (cid:80) b k l =1 ∇ ℓ ( x I ( k − 1 ,t,s ) ) , z l,I ( k,t,s ) )

Original: ( z l,I ( k,t,s ) i.i.d.
Revised: ( z l,I ( k,t,s ) i.i.d.

Original: ∼ D p t,s ).
Revised: ∼ D p t,s ).

Original: 19: v I ( k,t,s )
Revised: 19: v I ( k,t,s )

Original: = 1 k ≥ 1 ( g I ( k,t,s ) − g ref I ( k,t,s )
Revised: = 1 k ≥ 1 ( g I ( k,t,s ) − g ref I ( k,t,s )

Original: + v I ( k − 1 ,t,s ) )
Revised: + v I ( k − 1 ,t,s ) )

Original: + 1 k =

Revised: + 1 k =

Original: v I (0 ,t,s ) .
Revised: v I (0 ,t,s ) .

Original: Update (cid:101) x I ( k +1 ,t,s )
Revised: Update (cid:101) x I ( k +1 ,t,s )

Original: =
Revised: =

Original: x I ( k,t,s ) − ηv I ( k,t,s ) .
Revised: x I ( k,t,s ) − ηv I ( k,t,s ) .

Original: Add noise x I ( k +1 ,t,s )
Revised: Add noise x I ( k +1 ,t,s )

Original: = (cid:101) x
Revised: = (cid:101) x

Original: I ( k +1 ,t,s ) + ηξ I ( k,t,s ) , where ξ I ( k,t,s ) ∼ Unif( B dr ) .
Revised: I ( k +1 ,t,s ) + ηξ I ( k,t,s ) , where ξ I ( k,t,s ) ∼ Unif( B dr ) .

Original: 22: end for 23:
Revised: 22: end for 23:

Original: Communicate x I (0 ,t +1 ,s ) .
Revised: Communicate x I (0 ,t +1 ,s ) .

Original: 24: end for 25: end for of x ( p ) k
Revised: 24: end for 25: end for of x ( p ) k

Original: +

Revised: +

Original: =
Revised: =

Original: x ( p ) k − ηg ( p ) k for k ∈
Revised: x ( p ) k − ηg ( p ) k for k ∈

Original: [ B /b ] in each communication round, where g ( p ) k is a stochastic gradient with minibatch size b at x ( p ) k on local dataset D p and B is given local computation budget.
Revised: [ B /b ] in each communication round, where g ( p ) k is a stochastic gradient with minibatch size b at x ( p ) k on local dataset D p and B is given local computation budget.

Original: In typical convergence analysis, we need to bound the expected deviation of g ( p ) k from ideal global gradient ∇ f ( x k ) , that is E
Revised: In typical convergence analysis, we need to bound the expected deviation of g ( p ) k from ideal global gradient ∇ f ( x k ) , that is E

Original: [Equation]
Revised: [Equation]

Original: The former term is called bias and the latter one is called variance .
Revised: The former term is called bias and the latter one is called variance .

Original: A typical assumption to bound the first term is bounded gradient heterogeneity assumption , that requires ∥∇ f p ( x ) − ∇ f ( x ) ∥ ≤ ζfor every x ∈ R d and
Revised: A typical assumption to bound the first term is bounded gradient heterogeneity assumption , that requires ∥∇ f p ( x ) − ∇ f ( x ) ∥ ≤ ζfor every x ∈ R d and

Original: p ∈
Revised: p ∈

Original: [ P ] .
Revised: [ P ] .

Original: Under this assumption, the first term is only bounded by ζ 1 , that is a constant.
Revised: Under this assumption, the first term is only bounded by ζ 1 , that is a constant.

Original: The second term is typically bounded by σ 2 /b for g ( p ) k with minibatch size b , when the variance of a single stochastic gradient is bounded by σ 2 .
Revised: The second term is typically bounded by σ 2 /b for g ( p ) k with minibatch size b , when the variance of a single stochastic gradient is bounded by σ 2 .

Original: These facts show that the bias is still a constant and does not vanish even if minibatch size b is enhanced and the variance vanishes.
Revised: These facts show that the bias is still a constant and does not vanish even if minibatch size b is enhanced and the variance vanishes.

Original: This is why local SGD can be worse than minibatch SGD when ζ 1 is not too small.
Revised: This is why local SGD can be worse than minibatch SGD when ζ 1 is not too small.

Original: Also, we can see that the variance is still a constant for fixed minibatch size b and this is a common reason why minibatch
Revised: Also, we can see that the variance is still a constant for fixed minibatch size b and this is a common reason why minibatch

Original: SGD and local SGD only show slow convergences.
Revised: SGD and local SGD only show slow convergences.

Original: These observations give critical motivations of the simultaneous reduction of the bias and variance.
Revised: These observations give critical motivations of the simultaneous reduction of the bias and variance.

Original: The bias-variance reduced estimator v ( p ) k is defined as v ( p ) k :=
Revised: The bias-variance reduced estimator v ( p ) k is defined as v ( p ) k :=

Original: (1 /b ) (cid:80) bl =1 ( ∇ ℓ ( x ( p ) k , z l ) − ∇ ℓ ( x 0 , z l ))
Revised: (1 /b ) (cid:80) bl =1 ( ∇ ℓ ( x ( p ) k , z l ) − ∇ ℓ ( x 0 , z l ))

Original: + ∇ f ( x 0 ) (SVRG version).
Revised: + ∇ f ( x 0 ) (SVRG version).

Original: It is known that the bias caused by localization can be bounded by ζ ∥ x ( p )
Revised: It is known that the bias caused by localization can be bounded by ζ ∥ x ( p )

Original: k − x 0 ∥ and the variance caused by stochastization can be bounded by ( L 2 /b )
Revised: k − x 0 ∥ and the variance caused by stochastization can be bounded by ( L 2 /b )

Original: ∥ x ( p )
Revised: ∥ x ( p )

Original: k − x 0 ∥ 2 , where ζ is the Hessian heterogeneity of { f p } and L is the smoothness of ℓ .
Revised: k − x 0 ∥ 2 , where ζ is the Hessian heterogeneity of { f p } and L is the smoothness of ℓ .

Original: This implies that both the bias and variance of v ( p ) k converges to zero as x ( p ) k and x 0 go to x ∗ .
Revised: This implies that both the bias and variance of v ( p ) k converges to zero as x ( p ) k and x 0 go to x ∗ .

Original: In other words, bias-variance reduced estimator v ( p ) k is asymptotically consistent to the global gradient ∇ f ( x ( p ) k ) by using periodically computed global full gradients ∇ f ( x 0 ) .
Revised: In other words, bias-variance reduced estimator v ( p ) k is asymptotically consistent to the global gradient ∇ f ( x ( p ) k ) by using periodically computed global full gradients ∇ f ( x 0 ) .

Original: We actually adopt SARAH version of BVR estimator as in [19] rather than SVRG one due to its theoretical advantages.
Revised: We actually adopt SARAH version of BVR estimator as in [19] rather than SVRG one due to its theoretical advantages.

Original: 3.2 Parameter Perturbation at Local Updates
Revised: 3.2 Parameter Perturbation at Local Updates

Original: Although the bias-variance reduced estimator is useful to guarantee first-order optimality with small communication complexity in noncovex optimization, the algorithm often gets stuck at saddle points.
Revised: Although the bias-variance reduced estimator is useful to guarantee first-order optimality with small communication complexity in noncovex optimization, the algorithm often gets stuck at saddle points.

Original: To tackle this problem, we borrow the ideas of escaping saddle points in non-distributed nonconvex optimization.
Revised: To tackle this problem, we borrow the ideas of escaping saddle points in non-distributed nonconvex optimization.

Original: Particularly, to efficiently find second-order optimal points, we utilize parameter perturbation .
Revised: Particularly, to efficiently find second-order optimal points, we utilize parameter perturbation .

Original: Parameter perturbation is a familiar approach in non-distributed nonconvex optimization.
Revised: Parameter perturbation is a familiar approach in non-distributed nonconvex optimization.

Original: Specifically, Jin et al.
Revised: Specifically, Jin et al.

Original: [8, 9] have considered the update rule of x k +

Revised: [8, 9] have considered the update rule of x k +

Original: = x
Revised: = x

Original: k − η ∇ f ( x k ) + ηξ k , where ξ k ∼ Unif( B dr ) for some small radius r .
Revised: k − η ∇ f ( x k ) + ηξ k , where ξ k ∼ Unif( B dr ) for some small radius r .

Original: This algorithm is called Perturbed GD (PGD) or Noisy
Revised: This algorithm is called Perturbed GD (PGD) or Noisy

Original: GD.
Revised: GD.

Original: Similar to this formulation, we add noise at each local update, i.e., x ( p )
Revised: Similar to this formulation, we add noise at each local update, i.e., x ( p )

Original: k
Revised: k

Original: +1 =
Revised: +1 =

Original: (cid:101) x
Revised: (cid:101) x

Original: ( p ) k
Revised: ( p ) k

Original: +1 + ηξ ( p ) k , where (cid:101) x ( p ) k
Revised: +1 + ηξ ( p ) k , where (cid:101) x ( p ) k

Original: +

Revised: +

Original: =
Revised: =

Original: x ( p ) k − ηv ( p ) k .
Revised: x ( p ) k − ηv ( p ) k .

Original: The intuition behind the noise addition is that random noise has some components along the negative curvature directions of the global objective around the saddle point, and we expect that noise addition helps the parameter proceed to the decreasing directions of f and escape the saddle points.
Revised: The intuition behind the noise addition is that random noise has some components along the negative curvature directions of the global objective around the saddle point, and we expect that noise addition helps the parameter proceed to the decreasing directions of f and escape the saddle points.

Original: Necessity of local perturbation.
Revised: Necessity of local perturbation.

Original: Perturbing the global model at the server side is an intuitive way, but not sufficient for communication efficiency when we want to utilize small heterogeneity of the local datasets (i.e., ζ ≪ L ).
Revised: Perturbing the global model at the server side is an intuitive way, but not sufficient for communication efficiency when we want to utilize small heterogeneity of the local datasets (i.e., ζ ≪ L ).

Original: The bias-variance reduced estimator with local perturbation enables to escape multiple global saddle points in local optimization and achieves second-order optimality with communication complexity (cid:101) Θ( ζ/ε 2 ) for sufficiently large B .
Revised: The bias-variance reduced estimator with local perturbation enables to escape multiple global saddle points in local optimization and achieves second-order optimality with communication complexity (cid:101) Θ( ζ/ε 2 ) for sufficiently large B .

Original: In contrast, perturbing the global parameter at the server side only ensures to escape single global saddle point at each round and only achieves communication complexity of (cid:101) Θ( L/ε 2 ) .
Revised: In contrast, perturbing the global parameter at the server side only ensures to escape single global saddle point at each round and only achieves communication complexity of (cid:101) Θ( L/ε 2 ) .

Original: This is the reason why local perturbation rather than global one is adopted.
Revised: This is the reason why local perturbation rather than global one is adopted.

Original: 3.3 Concrete Procedures
Revised: 3.3 Concrete Procedures

Original: The full description of our proposed Bias-Variance Reduced Local Perturbed SGD (BVR-L-PSGD) is given in Algorithm 1.
Revised: The full description of our proposed Bias-Variance Reduced Local Perturbed SGD (BVR-L-PSGD) is given in Algorithm 1.

Original: When we set the noise size r = 0 , Algorithm 1 essentially matches BVR-LSGD.
Revised: When we set the noise size r = 0 , Algorithm 1 essentially matches BVR-LSGD.

Original: Additionally setting K = 1 , Algorithm 1 matches SARAH.
Revised: Additionally setting K = 1 , Algorithm 1 matches SARAH.

Original: The algorithm requires Θ( ST ) communication rounds.
Revised: The algorithm requires Θ( ST ) communication rounds.

Original: At each communication round, each worker computes large batch stochastic gradients and the server constructs v I (0 ,t,s ) by aggregating them.
Revised: At each communication round, each worker computes large batch stochastic gradients and the server constructs v I (0 ,t,s ) by aggregating them.

Original: v I (0 ,t,s ) is used as an estimator of ∇ f ( x I (0 ,t,s ) ) to reduce computational cost.
Revised: v I (0 ,t,s ) is used as an estimator of ∇ f ( x I (0 ,t,s ) ) to reduce computational cost.

Original: In line 14-21, we randomly select worker p t,s and only worker p t,s runs local optimization as described above.
Revised: In line 14-21, we randomly select worker p t,s and only worker p t,s runs local optimization as described above.

Original: In the local optimization, we use SARAH like bias variance reduced estimator (line 16-18) rather than SVRG one and add noise (line 20) at each local update.
Revised: In the local optimization, we use SARAH like bias variance reduced estimator (line 16-18) rather than SVRG one and add noise (line 20) at each local update.

Original:  Convergence Analysis
Revised:  Convergence Analysis

Original: In this section, we provide convergence theory of BVR-L-PSGD (Algorithm 1).
Revised: In this section, we provide convergence theory of BVR-L-PSGD (Algorithm 1).

Original: All the omitted proofs are found in the supplementary material.
Revised: All the omitted proofs are found in the supplementary material.

Original: For simple presentations, we use (cid:101) Θ symbol to hide an extra poly-logarithmic factors that depend on L, ρ, G, K, b, T, S, 1 /ε, 1 /q , where q represents the confidence parameter in high probability bounds.
Revised: For simple presentations, we use (cid:101) Θ symbol to hide an extra poly-logarithmic factors that depend on L, ρ, G, K, b, T, S, 1 /ε, 1 /q , where q represents the confidence parameter in high probability bounds.

Original: 4.1 Finding First-Order Stationary Points
Revised: 4.1 Finding First-Order Stationary Points

Original: First, we derive Descent Lemma for BVR-L-PSGD and first-order optimality guarantees by using it.
Revised: First, we derive Descent Lemma for BVR-L-PSGD and first-order optimality guarantees by using it.

Original: Proposition 4.1 (Descent Lemma) .
Revised: Proposition 4.1 (Descent Lemma) .

Original: Let S ∈ N
Revised: Let S ∈ N

Original: and I ( k, t, s ) ≥ I ( k 0 , t 0 , s 0 ) ∈
Revised: and I ( k, t, s ) ≥ I ( k 0 , t 0 , s 0 ) ∈

Original: [ KTS ] ∪ { 0 } .
Revised: [ KTS ] ∪ { 0 } .

Original: Suppose that Assumptions 1, 2, 3 and 5 hold.
Revised: Suppose that Assumptions 1, 2, 3 and 5 hold.

Original: Given q ∈ (0 , 1) , r > 0 , if we appropriately choose η = (cid:101) Θ(

Revised: Given q ∈ (0 , 1) , r > 0 , if we appropriately choose η = (cid:101) Θ(

Original: /L ∧ 1 / ( Kζ ) ∧ (cid:112) b/K/L ∧ √ Pb/ ( √ KTL ))
Revised: /L ∧ 1 / ( Kζ ) ∧ (cid:112) b/K/L ∧ √ Pb/ ( √ KTL ))

Original: , it holds that
Revised: , it holds that

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − 3 q .
Revised: with probability at least 1 − 3 q .

Original: Here,
Revised: Here,

Original: [Equation]
Revised: [Equation]

Original: for some universal constant c η > 0 .
Revised: for some universal constant c η > 0 .

Original: From Proposition 4.1 with I ( k 0 , t 0 , s 0 )
Revised: From Proposition 4.1 with I ( k 0 , t 0 , s 0 )

Original: ← 0 and I ( k, t, s ) ← KTS gives the following corollary.
Revised: ← 0 and I ( k, t, s ) ← KTS gives the following corollary.

Original: Corollary 4.2.
Revised: Corollary 4.2.

Original: Suppose that Assumptions 1, 2, 3 and 5 hold.
Revised: Suppose that Assumptions 1, 2, 3 and 5 hold.

Original: Under the same setting as in Proposition 4.3 and S ≥ Θ(( f ( x 0 ) − f ( x ∗ )) / ( ηKTε 2 )) , with probability at least 1 − 3 q , there exists
Revised: Under the same setting as in Proposition 4.3 and S ≥ Θ(( f ( x 0 ) − f ( x ∗ )) / ( ηKTε 2 )) , with probability at least 1 − 3 q , there exists

Original: i ∈
Revised: i ∈

Original: [ KTS − 1] ∪ { 0 } such that ∥∇ f
Revised: [ KTS − 1] ∪ { 0 } such that ∥∇ f

Original: ( (cid:101) x i ) ∥
Revised: ( (cid:101) x i ) ∥

Original: ≤ ε .
Revised: ≤ ε .

Original: Remark (Communication complexity) .
Revised: Remark (Communication complexity) .

Original: The total number of communication rounds Θ( TS ) becomes (cid:101) O (cid:16) T + (cid:16) L/K + ζ + L/ √ Kb + √ TL/ √ KPb (cid:17) ( f ( (cid:101) x 0 ) − f ( x ∗ )) /ε
Revised: The total number of communication rounds Θ( TS ) becomes (cid:101) O (cid:16) T + (cid:16) L/K + ζ + L/ √ Kb + √ TL/ √ KPb (cid:17) ( f ( (cid:101) x 0 ) − f ( x ∗ )) /ε

Original: 2 (cid:17) .
Revised: 2 (cid:17) .

Original: Given local computation budget B , we set T :=
Revised: Given local computation budget B , we set T :=

Original: Θ(1 + n/ ( B P )) and Kb :=
Revised: Θ(1 + n/ ( B P )) and Kb :=

Original: Θ( B ) with b ≤ Θ( √B ) .
Revised: Θ( B ) with b ≤ Θ( √B ) .

Original: Then, we have the averaged number of local computations per communication round Kb + n/ ( PT ) =
Revised: Then, we have the averaged number of local computations per communication round Kb + n/ ( PT ) =

Original: Θ( B ) and the communication complexity TS with budget B becomes
Revised: Θ( B ) and the communication complexity TS with budget B becomes

Original: [Equation]
Revised: [Equation]

Original: which matches the best known communication complexity [19].
Revised: which matches the best known communication complexity [19].

Original: 4.2 Escaping Saddle Points
Revised: 4.2 Escaping Saddle Points

Original: Next, we show that BVR-L-PSGD implicitly exploits the negative curvature of f around saddle points and efficiently escapes the saddle points by utilizing the asymptotic consistency of BVR estimator and the parameter perturbation at each local update.
Revised: Next, we show that BVR-L-PSGD implicitly exploits the negative curvature of f around saddle points and efficiently escapes the saddle points by utilizing the asymptotic consistency of BVR estimator and the parameter perturbation at each local update.

Original: We rely on the technique of coupling sequence [9].
Revised: We rely on the technique of coupling sequence [9].

Original: Given saddle point (cid:101) x I ( k 0 ,t 0 ,s 0 ) and ˆ I ≥
Revised: Given saddle point (cid:101) x I ( k 0 ,t 0 ,s 0 ) and ˆ I ≥

Original: I ( k 0 , t 0 , s 0 ) , we define a new sequence { x ′ i } ∞ i
Revised: I ( k 0 , t 0 , s 0 ) , we define a new sequence { x ′ i } ∞ i

Original: = I ( k 0 ,t 0 ,s 0 ) as follows:
Revised: = I ( k 0 ,t 0 ,s 0 ) as follows:

Original: (1) ⟨ ξ ′ (cid:101)
Revised: (1) ⟨ ξ ′ (cid:101)

Original: I , e min ⟩ = −⟨ ξ (cid:101)
Revised: I , e min ⟩ = −⟨ ξ (cid:101)

Original: I , e min ⟩ ; (2) ⟨ ξ ′ (cid:101) I , e j ⟩ = ⟨ ξ (cid:101) I , e j ⟩ for j ∈ { 2 , . . .
Revised: I , e min ⟩ ; (2) ⟨ ξ ′ (cid:101) I , e j ⟩ = ⟨ ξ (cid:101) I , e j ⟩ for j ∈ { 2 , . . .

Original: , d } ; and (3) All the other randomness is completely same as the one of { x i } KTS − 1 i = .
Revised: , d } ; and (3) All the other randomness is completely same as the one of { x i } KTS − 1 i = .

Original: Let r 0 := |⟨
Revised: Let r 0 := |⟨

Original: ξ (cid:101) I , e min ⟩| .
Revised: ξ (cid:101) I , e min ⟩| .

Original: Note that |⟨ ξ (cid:101) I − ξ ′ (cid:101)
Revised: Note that |⟨ ξ (cid:101) I − ξ ′ (cid:101)

Original: I , e min ⟩| = 2 r 0 and thus ∥ ξ (cid:101) I − ξ ′ (cid:101) I ∥
Revised: I , e min ⟩| = 2 r 0 and thus ∥ ξ (cid:101) I − ξ ′ (cid:101) I ∥

Original: = 2 r 0 .
Revised: = 2 r 0 .

Original: Also, observe that x (cid:101) I +1 − x ′ (cid:101)
Revised: Also, observe that x (cid:101) I +1 − x ′ (cid:101)

Original: I +

Revised: I +

Original: = η ⟨ ξ (cid:101)
Revised: = η ⟨ ξ (cid:101)

Original: I − ξ ′ (cid:101)
Revised: I − ξ ′ (cid:101)

Original: I , e min ⟩ e min .
Revised: I , e min ⟩ e min .

Original: We define (cid:101) I used in the definition of coupling sequence as follows:
Revised: We define (cid:101) I used in the definition of coupling sequence as follows:

Original: [Equation]
Revised: [Equation]

Original: Here, k ′ 0 is the minimum index k that satisfies k > k 0 and k ≡ 0 (mod ⌈√ K ⌉ ) .
Revised: Here, k ′ 0 is the minimum index k that satisfies k > k 0 and k ≡ 0 (mod ⌈√ K ⌉ ) .

Original: We can easily check that (cid:101)
Revised: We can easily check that (cid:101)

Original: I − I ( k 0 , t 0 , s 0 ) ≤ 1 / ( ηλ ) .
Revised: I − I ( k 0 , t 0 , s 0 ) ≤ 1 / ( ηλ ) .

Original: Then, we show that either of the two sequences { x i } or { x ′ i } efficiently escapes the saddle points by bounding the norm of the cumulative difference of x i and x ′ i from below.
Revised: Then, we show that either of the two sequences { x i } or { x ′ i } efficiently escapes the saddle points by bounding the norm of the cumulative difference of x i and x ′ i from below.

Original: The novel and most difficult part of the analysis is to evaluate the norm of the cumulative difference of the deviations
Revised: The novel and most difficult part of the analysis is to evaluate the norm of the cumulative difference of the deviations

Original: [Equation]
Revised: [Equation]

Original: 1 − η H ) J− i ( v i − ∇ f ( x i ) − v ′
Revised: 1 − η H ) J− i ( v i − ∇ f ( x i ) − v ′

Original: i + ∇ f ( x ′ i ))
Revised: i + ∇ f ( x ′ i ))

Original: ∥ generated by the two sequences, where v ′ i denotes the BVR estimator at iteration i generated by sequence { x ′ i } .
Revised: ∥ generated by the two sequences, where v ′ i denotes the BVR estimator at iteration i generated by sequence { x ′ i } .

Original: Proposition 4.3 (Implicit Negative Curvature Exploitation) .
Revised: Proposition 4.3 (Implicit Negative Curvature Exploitation) .

Original: Let I ( k 0 , t 0 , s 

Revised: Let I ( k 0 , t 0 , s

Original: ) ∈
Revised: ) ∈

Original: [ KTS ] ∪ { 0 } .
Revised: [ KTS ] ∪ { 0 } .

Original: Suppose that Assumptions 1, 2, 3, 4 and 5 hold, ∥∇ f ( (cid:101) x
Revised: Suppose that Assumptions 1, 2, 3, 4 and 5 hold, ∥∇ f ( (cid:101) x

Original: I ( k 0 ,t 0 ,s 0 ) )
Revised: I ( k 0 ,t 0 ,s 0 ) )

Original: ∥ ≤ ε and the minimum eigenvalue λ min of H :=
Revised: ∥ ≤ ε and the minimum eigenvalue λ min of H :=

Original: ∇ f ( (cid:101) x
Revised: ∇ f ( (cid:101) x

Original: I ( k 0 ,t 0 ,s 0 ) )
Revised: I ( k 0 ,t 0 ,s 0 ) )

Original: satisfies λ := − λ min > √ ρε .
Revised: satisfies λ := − λ min > √ ρε .

Original: Under b = Ω( K ∨ 1 / ( √ Kρε ) ∨
Revised: Under b = Ω( K ∨ 1 / ( √ Kρε ) ∨

Original: T/ ( PK )) , if we appropriately choose J I ( k 0 ,t 0 ,s 0 )
Revised: T/ ( PK )) , if we appropriately choose J I ( k 0 ,t 0 ,s 0 )

Original: = (cid:101) Θ(

Revised: = (cid:101) Θ(

Original: / ( ηλ )) , η = (cid:101)
Revised: / ( ηλ )) , η = (cid:101)

Original: Θ(1 /L
Revised: Θ(1 /L

Original: ∧ 1 / ( Kζ ) ∧ (cid:112) b/K/L ∧ √
Revised: ∧ 1 / ( Kζ ) ∧ (cid:112) b/K/L ∧ √

Original: Pb/ ( √ KT/L )) , with F I ( k 0 ,t 0 ,s 0 ) :
Revised: Pb/ ( √ KT/L )) , with F I ( k 0 ,t 0 ,s 0 ) :

Original: =
Revised: =

Original: c F η J I ( k 0 ,t 0 ,s 0 ) r 2 and r :=
Revised: c F η J I ( k 0 ,t 0 ,s 0 ) r 2 and r :=

Original: c r ε ( c F = Θ(1) and c r = (cid:101) Θ(1) )
Revised: c r ε ( c F = Θ(1) and c r = (cid:101) Θ(1) )

Original: we have f ( x I ( k 0 ,t 0 ,s 0 )
Revised: we have f ( x I ( k 0 ,t 0 ,s 0 )

Original: + J I
Revised: + J I

Original: ( k 0 ,t 0 ,sp ) )
Revised: ( k 0 ,t 0 ,sp ) )

Original: − f ( x I ( k 0 ,t 0 s 0 ) )
Revised: − f ( x I ( k 0 ,t 0 s 0 ) )

Original: ≤ − F I ( k 0 ,t 0 ,s 0 )
Revised: ≤ − F I ( k 0 ,t 0 ,s 0 )

Original: + Rwith probability at least 1 / 2 − 9 q/ 2 .
Revised: + Rwith probability at least 1 / 2 − 9 q/ 2 .

Original: Here, R 2 := 2 c η η J
Revised: Here, R 2 := 2 c η η J

Original: I
Revised: I

Original: ( k 0 ,t 0 ,s 0)
Revised: ( k 0 ,t 0 ,s 0)

Original: ∧ K K (cid:80) I ( k 0 ,t 0 ,s 0 )
Revised: ∧ K K (cid:80) I ( k 0 ,t 0 ,s 0 )

Original: − 1 i
Revised: − 1 i

Original: = I (0 ,t 0 ,s 0 )
Revised: = I (0 ,t 0 ,s 0 )

Original: ∥ x i +1 − x i ∥ 2 + 2 c
Revised: ∥ x i +1 − x i ∥ 2 + 2 c

Original: η η J I ( k 0 ,t 0 ,s 0)
Revised: η η J I ( k 0 ,t 0 ,s 0)

Original: ∧ KT
Revised: ∧ KT

Original: [Equation]
Revised: [Equation]

Original: Proposition 4.3 says the function value decreases by roughly F I ( k 0 ,t 0 ,s 0 ) and the global model escapes saddle points with probability at least 1 / 2 after J I
Revised: Proposition 4.3 says the function value decreases by roughly F I ( k 0 ,t 0 ,s 0 ) and the global model escapes saddle points with probability at least 1 / 2 after J I

Original: ( k 0 ,t 0 ,s 0 ) local steps.
Revised: ( k 0 ,t 0 ,s 0 ) local steps.

Original: 4.3 Finding Second-Order Stationary Points
Revised: 4.3 Finding Second-Order Stationary Points

Original: In this subsection, we derive final theorem that guarantees the second-order optimality of the global model by combining Propositions 4.1 and 4.3.
Revised: In this subsection, we derive final theorem that guarantees the second-order optimality of the global model by combining Propositions 4.1 and 4.3.

Original: Theorem 4.4 (Final Theorem) .
Revised: Theorem 4.4 (Final Theorem) .

Original: Suppose that Assumptions 1, 2, 3, 4 and 5 hold.
Revised: Suppose that Assumptions 1, 2, 3, 4 and 5 hold.

Original: Under b = Ω( K ∨ 1 / ( √ Kρε ) ∨ T/ ( PK )) , if we appropriately choose η = (cid:101) Θ(

Revised: Under b = Ω( K ∨ 1 / ( √ Kρε ) ∨ T/ ( PK )) , if we appropriately choose η = (cid:101) Θ(

Original: /L
Revised: /L

Original: ∧ 1 / ( Kζ ) ∧ (cid:112) b/K/L ∧ √
Revised: ∧ 1 / ( Kζ ) ∧ (cid:112) b/K/L ∧ √

Original: Pb/ ( √ KTL )) , r = (cid:101) Θ( ε ) and S = Θ(

Revised: Pb/ ( √ KTL )) , r = (cid:101) Θ( ε ) and S = Θ(

Original: + ( f ( (cid:101) x 0 ) − f ( x ∗ )) / ( ηKTε 2 )) , with probability at least 1 / 2 , there exists
Revised: + ( f ( (cid:101) x 0 ) − f ( x ∗ )) / ( ηKTε 2 )) , with probability at least 1 / 2 , there exists

Original: i ∈
Revised: i ∈

Original: [ KTS ] ∪ { 0 } such that (cid:101) x
Revised: [ KTS ] ∪ { 0 } such that (cid:101) x

Original: i is ε -second-order optimal point of f 6 .
Revised: i is ε -second-order optimal point of f 6 .

Original: Remark (High probability bound) .
Revised: Remark (High probability bound) .

Original: Theorem 4.4 guarantees that Algorithm 1 finds an approximate second-order optimal point in KTS iterations with probability at least 1 / 2 .
Revised: Theorem 4.4 guarantees that Algorithm 1 finds an approximate second-order optimal point in KTS iterations with probability at least 1 / 2 .

Original: Repeating Algorithmlog 2 (1 /q ) times guarantees that the same statement holds with probability at least 1 − q .
Revised: Repeating Algorithmlog 2 (1 /q ) times guarantees that the same statement holds with probability at least 1 − q .

Original: Remark (Communication complexity) .
Revised: Remark (Communication complexity) .

Original: The total number of communication rounds TS is given by (cid:101) O (cid:16) T + (cid:16) L/K + ζ + L/ √ Kb + √ TL/ √ KPb (cid:17) ( f ( (cid:101) x 0 ) − f ( x ∗ )) /ε
Revised: The total number of communication rounds TS is given by (cid:101) O (cid:16) T + (cid:16) L/K + ζ + L/ √ Kb + √ TL/ √ KPb (cid:17) ( f ( (cid:101) x 0 ) − f ( x ∗ )) /ε

Original: 2 (cid:17) .
Revised: 2 (cid:17) .

Original: Given local computation budget B , we set T :=
Revised: Given local computation budget B , we set T :=

Original: Θ(1 + n/ ( B P )) and Kb :=
Revised: Θ(1 + n/ ( B P )) and Kb :=

Original: Θ( B ) with b ≤ Θ( √B ) .
Revised: Θ( B ) with b ≤ Θ( √B ) .

Original: Then, we have the averaged number of local computations per communication round Kb + n/ ( PT ) =
Revised: Then, we have the averaged number of local computations per communication round Kb + n/ ( PT ) =

Original: Θ( B ) and the communication complexity Θ( TS ) with budget B becomes (cid:101) O (cid:16) 1 + n B P +
Revised: Θ( B ) and the communication complexity Θ( TS ) with budget B becomes (cid:101) O (cid:16) 1 + n B P +

Original: L √B ε 2 + √ nL
Revised: L √B ε 2 + √ nL

Original: B Pε 2 + ζε 2 (cid:17) .
Revised: B Pε 2 + ζε 2 (cid:17) .

Original: This implies that for ζ
Revised: This implies that for ζ

Original: =
Revised: =

Original: o
Revised: o

Original: ( L ) the communication complexity is strictly smaller than the one of minibatch SGD (cid:101)
Revised: ( L ) the communication complexity is strictly smaller than the one of minibatch SGD (cid:101)

Original: O (1 + L/ε 2 + G 2 / ( B Pε 4 )) .
Revised: O (1 + L/ε 2 + G 2 / ( B Pε 4 )) .

Original: Note that the rate matches to the one of BVR-L-SGD [19].
Revised: Note that the rate matches to the one of BVR-L-SGD [19].

Original: Hence, our method finds second-order optimal points without hurting communication efficiency of the state-of-the-art first-order optimality guaranteed method.
Revised: Hence, our method finds second-order optimal points without hurting communication efficiency of the state-of-the-art first-order optimality guaranteed method.

Original: Furthermore, when B → ∞ , we have (cid:101) Θ(

Revised: Furthermore, when B → ∞ , we have (cid:101) Θ(

Original: + ζ/ε 2 ) , that goes to (cid:101) Θ(1) as ζ → 0 .
Revised: + ζ/ε 2 ) , that goes to (cid:101) Θ(1) as ζ → 0 .

Original: In summary, BVR-L-PSGD enjoys the desirable properties (i) and (ii) described in Section 1.
Revised: In summary, BVR-L-PSGD enjoys the desirable properties (i) and (ii) described in Section 1.

Original:  Numerical Resutls
Revised:  Numerical Resutls

Original: In this section, we give some experimental results to verify our theoretical findings.
Revised: In this section, we give some experimental results to verify our theoretical findings.

Original: Data Preparation.
Revised: Data Preparation.

Original: We artificially generated heterogeneous local datasets from CIFAR10 7 dataset.
Revised: We artificially generated heterogeneous local datasets from CIFAR10 7 dataset.

Original: The data preparation procedure is completely in accordance with [19] and the details are found in [19].
Revised: The data preparation procedure is completely in accordance with [19] and the details are found in [19].

Original: We set homogeneity parameter q to 0 .
Revised: We set homogeneity parameter q to 0 .

Original: 35 , which captures how similar the local datasets are ( q = 0 .
Revised: 35 , which captures how similar the local datasets are ( q = 0 .

Original: 1 corresponds to I.I.D. case and higher q does to higher heterogeneity).
Revised: 1 corresponds to I.I.D. case and higher q does to higher heterogeneity).

Original: Model.
Revised: Model.

Original: We conducted our experiments using a two-hidden layers fully connected neural network with 100 hidden units and softplus activation.
Revised: We conducted our experiments using a two-hidden layers fully connected neural network with 100 hidden units and softplus activation.

Original: For loss function, we used the standard cross-entropy loss.
Revised: For loss function, we used the standard cross-entropy loss.

Original: We initialized parameters by uniformly sampling the parameters from [ − 1 / 100 , 1 / 100] .
Revised: We initialized parameters by uniformly sampling the parameters from [ − 1 / 100 , 1 / 100] .

Original: Implemented Algorithms.
Revised: Implemented Algorithms.

Original: Minibatch SGD, Noisy Minibatch SGD, BVR-L-SGD [19] and our proposed BVR-L-PSGD were implemented.
Revised: Minibatch SGD, Noisy Minibatch SGD, BVR-L-SGD [19] and our proposed BVR-L-PSGD were implemented.

Original: We set K = 64 and b = 16 , and thus B = 1024 .
Revised: We set K = 64 and b = 16 , and thus B = 1024 .

Original: For
Revised: For

Original: BVR-L-PSGD, the noise radius was tuned from r ∈ { 0 . 5 , 2 . 5 , 12 . 5 } .
Revised: BVR-L-PSGD, the noise radius was tuned from r ∈ { 0 . 5 , 2 . 5 , 12 . 5 } .

Original: For each algorithm, we tuned learning rate η from { 0 . 005 , 0 . 01 , 0 . 05 , 0 . 1 , 0 . 5 , 1 . 0 } .
Revised: For each algorithm, we tuned learning rate η from { 0 . 005 , 0 . 01 , 0 . 05 , 0 . 1 , 0 . 5 , 1 . 0 } .

Original: The details of the tuning procedure are found in the supplementary material.
Revised: The details of the tuning procedure are found in the supplementary material.

Original: Evaluation.
Revised: Evaluation.

Original: We compared the implemented algorithms using six criteria of train gradient norm ∥∇ f ( x ) ∥ ; train loss; train accuracy; test gradient norm; test loss and test accuracy against the number of communication rounds.
Revised: We compared the implemented algorithms using six criteria of train gradient norm ∥∇ f ( x ) ∥ ; train loss; train accuracy; test gradient norm; test loss and test accuracy against the number of communication rounds.

Original: The total number of communication rounds was fixed to 1 , 000 for each algorithm.
Revised: The total number of communication rounds was fixed to 1 , 000 for each algorithm.

Original: We independently repeated the experiments 5 times and report the mean and standard deviation of the above criteria.
Revised: We independently repeated the experiments 5 times and report the mean and standard deviation of the above criteria.

Original: Due to the space limitation, we will only report train gradient norm, train loss and test accuracy in the main paper.
Revised: Due to the space limitation, we will only report train gradient norm, train loss and test accuracy in the main paper.

Original: The full results are found in the supplementary material.
Revised: The full results are found in the supplementary material.

Original: Results.
Revised: Results.

Original: Figure 1 shows the performances of BVR-L-SGD and our proposed algorithm.
Revised: Figure 1 shows the performances of BVR-L-SGD and our proposed algorithm.

Original: We can
Revised: We can

Original: [Figure]
Revised: [Figure]

Original: see that the both algorithms got stuck at a small gradient norm region in initial rounds.
Revised: see that the both algorithms got stuck at a small gradient norm region in initial rounds.

Original: After that
Revised: After that

Original: BVR-L-SGD showed unstable convergence and took a lot of time to escape the stucked region.
Revised: BVR-L-SGD showed unstable convergence and took a lot of time to escape the stucked region.

Original: In contrast, our proposed method efficiently escaped the stucked region and consistently achieves better train loss and test accuracy than BVR-L-SGD.
Revised: In contrast, our proposed method efficiently escaped the stucked region and consistently achieves better train loss and test accuracy than BVR-L-SGD.

Original: Also, our method consistently outperformed Minibatch
Revised: Also, our method consistently outperformed Minibatch

Original: SGD and Noisy Minibatch SGD.
Revised: SGD and Noisy Minibatch SGD.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we have studied a new local algorithm called Bias-Variance Reduced Local Perturbed
Revised: In this paper, we have studied a new local algorithm called Bias-Variance Reduced Local Perturbed

Original: SGD (BVR-L-PSGD) based on a combination of the bias-variance reduced gradient estimator with parameter perturbation to efficiently find second-order optimal points in centralized nonconvex distributed optimization.
Revised: SGD (BVR-L-PSGD) based on a combination of the bias-variance reduced gradient estimator with parameter perturbation to efficiently find second-order optimal points in centralized nonconvex distributed optimization.

Original: We have shown that BVR-L-PSGD enjoys second-order optimality without hurting the best known communication complexity for first-order optimality guarantees.
Revised: We have shown that BVR-L-PSGD enjoys second-order optimality without hurting the best known communication complexity for first-order optimality guarantees.

Original: Particularly, the communication complexity is better than non-local methods when Hessian heterogeneity ζ of local datasets is smaller than the smoothness of the local loss L in order sense.
Revised: Particularly, the communication complexity is better than non-local methods when Hessian heterogeneity ζ of local datasets is smaller than the smoothness of the local loss L in order sense.

Original: Also, for sufficiently large B , the communication complexity of our method approaches to (cid:101) Θ(1) when the local datasets heterogeneity ζ goes to zero.
Revised: Also, for sufficiently large B , the communication complexity of our method approaches to (cid:101) Θ(1) when the local datasets heterogeneity ζ goes to zero.

Original: The numerical results have validated our theoretical findings.
Revised: The numerical results have validated our theoretical findings.

Original: Acknowledgement
Revised: Acknowledgement

Original: TS was partially supported by JSPS KAKENHI (18K19793, 18H03201, and 20H00576), Japan DigitalDesign, and JST CREST. 
Revised: TS was partially supported by JSPS KAKENHI (20H00576) and JST CREST. The authors would like to thank Kazusato Oko for his helpful advice.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] See Section 1.
Revised: [Yes] See Section 1.

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Section 4.
Revised: [Yes] See Section 4.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See section 2.
Revised: [Yes] See section 2.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] See Section 4 and the supplementary material.
Revised: [Yes] See Section 4 and the supplementary material.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] See Sectionand the supplementary material.
Revised: [Yes] See Sectionand the supplementary material.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See Section 5 and the supplementary material.
Revised: [Yes] See Section 5 and the supplementary material.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] See Section 5.
Revised: [Yes] See Section 5.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See the supplementary material.
Revised: [Yes] See the supplementary material.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Section 5.
Revised: [Yes] See Section 5.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 5.
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]