Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where we use Assumption 
Revised: where we use Assumption 5.5.

Original: [Equation]

Original: Next, using Assumption 5.6 and (15)
Revised: Next, using Assumption 5.6 and (15)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: − (cid:0) ∇ f i ( x t +1 ) − ∇ f i ( x t ) (cid:1)(cid:13)(cid:13) 2 (cid:105) + (1 − p ) (cid:13)(cid:13) h t − ∇ f ( x t ) (cid:13)(cid:13)= pσnB (cid:48)
Revised: (cid:0) ∇ f i ( x t +1 ) − ∇ f i ( x t ) (cid:1)(cid:13)(cid:13) 2 (cid:105) + (1 − p ) (cid:13)(cid:13) h t − ∇ f ( x t ) (cid:13)(cid:13)= pσnB (cid:48)

Original: + (1 − p ) n 2 Bn (cid:88) i =
Revised: + (1 − p ) n 2 Bn (cid:88) i =

Original: B (cid:88)
Revised: B (cid:88)

Original: j =
Revised: j =

Original: E h (cid:104)(cid:13)(cid:13) [Equation] f 
Revised: E h (cid:104)(cid:13)(cid:13) ∇ f i ( x t +1 ; ξ t +1 ij ) − ∇ f i ( x t ; ξ t +1 ij ) − (cid:0) ∇ f i ( x t +1 )

Original: [Equation] ( x t  + [Equation]
Revised: − ∇ f i ( x t ) (cid:1)(cid:13)(cid:13) 2 (cid:105) + (1 − p ) (cid:13)(cid:13) h t − ∇ f ( x t ) (cid:13)(cid:13)≤ pσnB (cid:48)

Original: ξ t + 
Revised: + (1 − p )

Original: [Equation] [Equation]
Revised: L 2 σ nB (cid:13)(cid:13)

Original: f
Revised: x

Original: [Equation] ( x t [Equation] ξ t +1ij ) − (cid:0){+x t (cid:13)(cid:13) 
+}
Revised: t +1 − {+x t (cid:13)(cid:13) 
+}

Original: [Equation]

Original: f i ( x t +1 ) −

Original: [Equation]

Original:  fi ( x t ) (cid:1)(cid:13)(cid:13) 2 (cid:105) + ( [Equation] ≤ pσnB (cid:48) +
Revised: + (1 − p ) (cid:13)(cid:13) h t − ∇ f ( x t ) (cid:13)(cid:13) 2 .

Original: [Equation]

Original: nB

Original: [Equation]

Original: Theorem I.19.
Revised: Theorem I.19.

Original: Suppose that Assumptions 5.1, 5.2, 5.3, 5.5, 5.6 and 1.2 hold.
Revised: Suppose that Assumptions 5.1, 5.2, 5.3, 5.5, 5.6 and 1.2 hold.

Original: Let us take a = 1 2 ω +1 , probability p ∈ (0 , 1] , batch size B (cid:48) ≥ 1 and γ ≤ (cid:32)
Revised: Let us take a = 1 2 ω +1 , probability p ∈ (0 , 1] , batch size B (cid:48) ≥ 1 and γ ≤ (cid:32)

Original: L + (cid:115) 12 ω (2 ω + 1)(1 − p ) n (cid:18)
Revised: L + (cid:115) 12 ω (2 ω + 1)(1 − p ) n (cid:18)

Original: L 2 σ
Revised: L 2 σ

Original: B + (cid:98) L(cid:19)
Revised: B + (cid:98) L(cid:19)

Original: + 2(1 − p )
Revised: + 2(1 − p )

Original: L 2 σ pnB (cid:33) −, in Algorithm 2.
Revised: L 2 σ pnB (cid:33) −, in Algorithm 2.

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Revised: f ( (cid:98) x

Revised: [Equation]

Revised: ) (cid:13)(cid:13)

Revised: [Equation]

Revised: (cid:105)

Original: [Equation]
Revised: [Equation]

Revised: 

Revised: [Equation]

Revised: × (cid:32)

Revised: L + (cid:115) 12 ω (2 ω + 1)(

Revised: [Equation]

Revised: p ) n (cid:18)

Revised: [Equation]

Revised: σ

Revised: [Equation]

Revised: (cid:98)

Revised: [Equation]

Revised: (cid:19)

Revised: + 2(1 − p )

Revised: [Equation]

Original: {+σ pnB (cid:33) + 2 (2 ω + 1) (cid:13)(cid:13) g 0 − h 
+}
Revised: {+σ pnB (cid:33) + 2 (2 ω + 1) (cid:13)(cid:13) g 0 − h 
+}

Revised: (cid:13)(cid:13) 2 + 4 ω n (cid:32)n n (cid:88) i =

Revised: (cid:13)(cid:13)

Original: {+g 
+}
Revised: {+g 
+}

Revised: i − h

Revised: i (cid:13)(cid:13)(cid:33)

Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let us ﬁx constants κ, η, ν ∈ [0 , ∞ ) that we will deﬁne later.
Revised: Let us ﬁx constants κ, η, ν ∈ [0 , ∞ ) that we will deﬁne later.

Original: Using Lemma I.
Revised: Using Lemma I.

Original: [Equation]
Revised: [Equation]

Original: we can get (20).
Revised: we can get (20).

Original: Considering (20)
Revised: Considering (20)

Original: [Equation]
Revised: [Equation]

Original: Lemma I.
Revised: Lemma I.

Original: [Equation]
Revised: [Equation]

Original: Lemma I.
Revised: Lemma I.

Original: [Equation]
Revised: [Equation]

Original: and the law of total expectation
Revised: and the law of total expectation

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: x t +1 − x t (cid:13)(cid:13) 

Revised: x t +1 − x t (cid:13)(cid:13)

Original: + 2 a 2 ω (1 − p ) nn (cid:88)
Revised: + 2 a 2 ω (1 − p ) nn (cid:88)

Original: i =
Revised: i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g ti − h ti (cid:13)(cid:13) 

Revised: g ti − h ti (cid:13)(cid:13)

Original: + (1 − p ) (1 − a ) 2 (cid:13)(cid:13)
Revised: + (1 − p ) (1 − a ) 2 (cid:13)(cid:13)

Original: g t − h t (cid:13)(cid:13)  + η E (cid:34) 2 ω (1 − p ) (cid:18)
Revised: g t − h t (cid:13)(cid:13)  + η E (cid:34) 2 ω (1 − p ) (cid:18)

Original: L 2 σ
Revised: L 2 σ

Original: B + (cid:98) L(cid:19)
Revised: B + (cid:98) L(cid:19)

Original: (cid:13)(cid:13) x
Revised: (cid:13)(cid:13) x

Original: t +1 − x t (cid:13)(cid:13) 

Revised: t +1 − x t (cid:13)(cid:13)

Original: + (1 − p ) (cid:16) 2 a 2 ω + (1 − a ) 2 (cid:17) 1 n n (cid:88)
Revised: + (1 − p ) (cid:16) 2 a 2 ω + ( [Table] − a ) 2 (cid:17) 1 n n [Table]

Original: i = 
Revised: = [Table]

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g ti − h ti (cid:13)(cid:13)(cid:35) + ν E (cid:20) pσnB (cid:48)
Revised: g ti − h ti (cid:13)(cid:13)(cid:35) + ν E (cid:20) pσnB (cid:48)

Original: + (1 − p )
Revised: + (1 − p )

Original: L 2 σ nB (cid:13)(cid:13)
Revised: L 2 σ nB (cid:13)(cid:13)

Original: x
Revised: x

Original: t +1 − x t (cid:13)(cid:13) 

Revised: t +1 − x t (cid:13)(cid:13)

Original: + (1 − p ) (cid:13)(cid:13) h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:21) .
Revised: + (1 − p ) (cid:13)(cid:13) h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:21) .

Original: After rearranging the terms, we get
Revised: After rearranging the terms [Table] we get

Original: E (cid:2) f
Revised: E (cid:2) f

Original: ( x t +1 ) (cid:3) + κ E (cid:104)(cid:13)(cid:13)
Revised: ( x t +1 ) (cid:3) + κ E (cid:104)(cid:13)(cid:13)

Original: g t +1 − h t +1 (cid:13)(cid:13) 2 (cid:105) + η E (cid:34)n n (cid:88) i =
Revised: g t +1 − h t +1 (cid:13)(cid:13) 2 (cid:105) + η E (cid:34)n [Table] =

Original: (cid:13)(cid:13)
Revised: [Table]

Original: [-g t +
-] i − h
Revised: [-g t +
-] i − [Table] i

Original: [Table]
Revised: [Table]

Original: +1 i (cid:13)(cid:13) 2 (cid:35) + ν E (cid:104)(cid:13)(cid:13) 
Revised: + ν E (cid:104)(cid:13)(cid:13) [Equation] − 

Original: [Equation]
Revised: [Equation]

Original: nB
Revised: nB

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Revised: [Equation]

Revised: p

Original: [Equation]
Revised: [Equation]

Revised: +

Revised: [Equation]

Original: nB (cid:48) .
Revised: ) n [Equation] n [Equation]

Original: Let us take ν = γp , κ = γa , a = 1 2 ω +1 , and η = 2 γωn .
Revised: + ( γ [Equation] ν (1 − p )) [Equation]

Revised: h

Original: Thus γ + κ (1 − p )(1 − a ) 2 ≤ κ, γ + ν (1 − p ) =
Revised: [Equation] f [Equation] x t ) (cid:13)(cid:13)

Revised: [Equation]

Original:  ν, 2 κa 2 ω (1 − p ) n + η (1 − p ) (cid:16) 2 a 2 ω + (1 − a ) 2 (cid:17) ≤ η, and
Revised: (cid:105) + νpσnB (cid:48) . Let us take ν = γp , κ = γa , a = 1 2 ω +1 , and η = 2 γωn . Thus γ + κ (1 − p )(1 − a ) 2 ≤ κ, γ + ν (1 − p ) = ν, 2 κa 2 ω (1 − p ) n + η (1 − p ) (cid:16) 2 a 2 ω + (1 − a ) 2 (cid:17) ≤ η, and

Original: E (cid:2) f
Revised: E (cid:2) f

Original: ( x t +1 ) (cid:3) + γ (2 ω + 1)E (cid:104) (cid:13) (cid:13) g t
Revised: ( x t +1 ) (cid:3) + γ (2 ω + 1)E (cid:104) (cid:13) (cid:13) g t

Original: +

Revised: +

Original: − h t +1 (cid:13) (cid:13) 2 (cid:105) + 2 γω n E (cid:34)n n (cid:88) i = (cid:13) (cid:13) g t
Revised: − h t +1 (cid:13) (cid:13) 2 (cid:105) + 2 γω n E (cid:34)n n (cid:88) i = (cid:13) (cid:13) g t

Original: +1 i − h t
Revised: +1 i − h t

Original: +1 i (cid:13) (cid:13)(cid:35)
Revised: +1 i (cid:13) (cid:13)(cid:35)

Original: +
Revised: +

Original: γ p E (cid:104)(cid:13)(cid:13) h t +1 − ∇ f ( x t +1 )
Revised: γ p E (cid:104)(cid:13)(cid:13) h t +1 − ∇ f ( x t +1 )

Original: (cid:13)(cid:13) 2 (cid:105) ≤ E (cid:2) f
Revised: (cid:13)(cid:13) 2 (cid:105) ≤ E (cid:2) f

Original: ( x t ) (cid:3) − γ 2 E (cid:104)(cid:13)(cid:13) ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105)
Revised: ( x t ) (cid:3) − γ 2 E (cid:104)(cid:13)(cid:13) ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105)

Original: [Table]
Revised: [Table]

Original: g t − h t (cid:13)(cid:13) 2 (cid:105) + 2 γω n E (cid:34)n n (cid:88) i =
Revised: g t − h t (cid:13)(cid:13) 2 (cid:105) + 2 γω n E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g t i − h t
Revised: g t i − h t

Original: i (cid:13)(cid:13)(cid:35) + γ p E (cid:104)(cid:13)(cid:13) h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105) + γσnB (cid:48) .
Revised: i (cid:13)(cid:13)(cid:35) + γ p E (cid:104)(cid:13)(cid:13) h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105) + γσnB (cid:48) .

Original: In the view of the choice of γ , we obtain
Revised: In the view of the choice of γ , we obtain

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: f
Revised: f

Original: [Equation]
Revised: [Equation]

Revised: x

Revised: [Equation]

Revised: g t

Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: (cid:105)
Revised: (cid:105)

Original: [Equation]

Original:  2 γω n E (cid:34)n n (cid:88) i =
Revised: + 2 γω n E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g t +

Revised: g t +

Original: i − h t
Revised: i − h t

Original: +1 i (cid:13)(cid:13) 2 (cid:35) + γ p
Revised: +1 i (cid:13)(cid:13) 2 (cid:35) + γ p

Original: [Equation]
Revised: [Equation]

Revised: (cid:104)(cid:13)(cid:13)

Revised: [Equation]

Revised: [Equation]

Revised: (cid:13)(cid:13)

Original:  ≤ 
Revised: [Equation] (cid:105) ≤ E (cid:2) f ( x t [Equation]

Revised: (cid:3)

Revised: [Equation]

Revised: 2 E (cid:104)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Revised: f ( x t ) (cid:13)(cid:13)

Revised: [Equation]

Original: (cid:105)
Revised: (cid:105)

Original: + [Equation] [Equation] g
Revised: + γ (2 ω

Original: [Equation]
Revised: [Equation]

Revised: 1)E (cid:104)(cid:13)(cid:13) g

Original:  − h [Equation] (cid:13)(cid:13)
Revised: t − h t (cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: (cid:105)
Revised: (cid:105)

Original: [Equation]

Original:  2 γω n E (cid:34)n n (cid:88) i =
Revised: + 2 γω n E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g ti − h ti (cid:13)(cid:13)(cid:35) + γ p 
Revised: g ti − h ti (cid:13)(cid:13)(cid:35) + γ p E

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: nB [Equation]
Revised: nB (cid:48) .

Original: Finally, using Lemma I.5 with
Revised: Finally, using Lemma I.5 with

Original: [Equation]
Revised: [Equation]

Original: and C =
Revised: and C =

Original: σ 2 nB (cid:48) , we can conclude the proof.
Revised: σ 2 nB (cid:48) , we can conclude the proof.

Original: Corollary 6.10.
Revised: Corollary 6.10.

Original: Suppose that assumptions from Theorem I.19 hold, probability p =
Revised: Suppose that assumptions from Theorem I.19 hold, probability p =

Original: min (cid:110) ζ C d , nεBσ 2 (cid:111) , batch size B (cid:48) =
Revised: min (cid:110) ζ C d , nεBσ 2 (cid:111) , batch size B (cid:48) =

Original: Θ (cid:16) σ 2 nε (cid:17) and h 0 i = g 0 i = 1 B init (cid:80) B init k
Revised: Θ (cid:16) σ 2 nε (cid:17) and h 0 i = g 0 i = 1 B init (cid:80) B init k

Original: =1 ∇ f i ( x 0 ; ξ 0 ik ) for all i ∈ [ n ] , initial batch size B init =
Revised: =1 ∇ f i ( x 0 ; ξ 0 ik ) for all i ∈ [ n ] , initial batch size B init =

Original: Θ (cid:16) max (cid:110) σ 2 nε , B dζ C (cid:111)(cid:17) , then DASHA-SYNC-MVR needs
Revised: Θ (cid:16) max (cid:110) σ 2 nε , B dζ C (cid:111)(cid:17) , then DASHA-SYNC-MVR needs

Original: T :
Revised: T :

Original: = O
Revised: = O

Original: [Equation]
Revised: [Equation]

Original: nεB   communication rounds to get an ε -solution, the communication complexity is equal to O ( d + ζ C T ) , and the number of stochastic gradient calculations per node equals O ( B init + BT ) , where ζ C is the expected density from Deﬁnition 1.3.
Revised: nεB   communication rounds to get an ε -solution, the communication complexity is equal to O ( d + ζ C T ) , and the number of stochastic gradient calculations per node equals O ( B init + BT ) , where ζ C is the expected density from Deﬁnition 1.3.

Original: Proof.
Revised: Proof.

Original: Considering Theorem I.19 and the choice of B (cid:48)
Revised: Considering Theorem I [Equation] 19 and the choice of B (cid:48)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Revised: [Equation]

Revised: x

Original: [Equation]
Revised: [Equation]

Original: Due to p = min (cid:110) ζ C d , nεBσ 2 (cid:111) 
Revised: Due to p = min (cid:110) ζ C d , nεBσ 2 (cid:111) , we have

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: ∇ f ( (cid:98) x T )

Original: [Equation]
Revised: [Equation]

Original: ≤ O   1 T 
Revised: ≤ O   1 T 

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: +

Original: [Equation]

Original: ≤ O 

Original: [Equation]

Original:  +
Revised:  +

Original: [Equation]
Revised: [Equation]

Original: nεB
Revised: nεB

Original: [Equation]
Revised: [Equation]

Original: 
Revised: ≤ O  [Equation] +

Original: [Equation]
Revised: [Equation]

Original: 
Revised: nεB

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we can take
Revised: we can take

Original: T =
Revised: T =

Original: O 
Revised: O 

Original: [Equation]
Revised: [Equation]

Original: C n
Revised: n

Original: L σ √
Revised: L σ √

Original: B + (cid:114) σεn 2 B
Revised: B + (cid:114) σεn 2 B

Original: L σ √
Revised: L σ √

Original: B (cid:33) + (cid:18)
Revised: B (cid:33) + (cid:18)

Original: d ζ C + σnεB (cid:19) (cid:13) (cid:13) h 0 − ∇ f ( x 0 ) (cid:13) (cid:13)      .
Revised: d ζ C + σnεB (cid:19) (cid:13) (cid:13) h 0 − ∇ f ( x 0 ) (cid:13) (cid:13)      .

Revised: [Table]

Original: [Table]
Revised: [Table]

Revised: E

Original: [Equation]
Revised: [Equation]

Revised: 

Original: [Equation]
Revised: [Equation]

Original: Next, by taking B init =
Revised: Next, by taking B init =

Original: max (cid:110) σ 2 nε
Revised: max (cid:110) σ 2 nε

Original: [Equation]
Revised: [Equation]

Original: B dζ C (cid:111) and using the last ineqaulity
Revised: B dζ C (cid:111) and using the last ineqaulity

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Revised: T

Revised: =

Revised: O

Original: [Equation]
Revised: [Equation]

Original: Finally
Revised: Finally

Original: [Equation]
Revised: [Equation]

Original: it is left to estimate the communication and oracle complexity.
Revised: it is left to estimate the communication and oracle complexity.

Original: On average, the number of coordinates that each node in Algorithm 2 sends at each communication round equals pd +
Revised: On average, the number of coordinates that each node in Algorithm 2 sends at each communication round equals pd +

Original: [Equation]
Revised: [Equation]

Original: Therefore, the communication complexity is equal to
Revised: Therefore, the communication complexity is equal to

Original: O ( d + ζ C T ) .
Revised: O ( d + ζ C T ) .

Original: Considering the fact that we use a mini-batch of stochastic gradients, on average, the number of stochastic gradients that each node calculates at each communication round equals pB (cid:48)
Revised: Considering the fact that we use a mini-batch of stochastic gradients, on average, the number of stochastic gradients that each node calculates at each communication round equals pB (cid:48)

Original: + (1 − p )2 B ≤ O (cid:16) nεBσ 2 · σ 2 nε (cid:17) + 2 B = O ( B ) .
Revised: + (1 − p )2 B ≤ O (cid:16) nεBσ 2 · σ 2 nε (cid:17) + 2 B = O ( B ) .

Original: Considering the initial batch size B init , the number of stochastic gradients that each node calculates equals O ( B init + BT ) .
Revised: Considering the initial batch size B init , the number of stochastic gradients that each node calculates equals O ( B init + BT ) .

Original: Corollary 6.11.
Revised: Corollary 6.11.

Original: Suppose that assumptions of Corollary 6.10 hold, batch size B ≤ σ √ εn , we take
Revised: Suppose that assumptions of Corollary 6.10 hold, batch size B ≤ σ √ εn , we take

Original: Rand K with K = ζ C =
Revised: Rand K with K = ζ C =

Original: Θ (cid:16) Bd √ εn σ (cid:17) , and (cid:101)
Revised: Θ (cid:16) Bd √ εn σ (cid:17) , and (cid:101)

Original: L :=
Revised: L :=

Original: max { L, L σ , (cid:98) L } .
Revised: max { L, L σ , (cid:98) L } .

Original: Then the communication complexity equals
Revised: Then the communication complexity equals

Original: [Equation]
Revised: [Equation]

Original: and the expected # of stochastic gradient calculations per node equals
Revised: and the expected # of stochastic gradient calculations per node equals

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Proof.
Revised: Proof.

Original: In the view of Theorem F.2, we have ω +

Revised: In the view of Theorem F.2, we have ω +

Original: =
Revised: =

Original: d/K. Moreover, K = Θ (cid:16)
Revised: d/K. Moreover, K = Θ (cid:16)

Original: Bd √ εn σ
Revised: Bd √ εn σ

Original: [Equation]
Revised: [Equation]

Original: , thus the communication complexity equals
Revised: , thus the communication complexity equals

Original: O
Revised: O

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: And the expected number of stochastic gradient calculations per node equals
Revised: And the expected number of stochastic gradient calculations per node equals

Original: O ( B init + BT )=
Revised: O ( B init + BT )

Original: [Equation]
Revised: [Equation]

Original: I. C ASE OF DASHA-SYNC-MVR UNDER PŁ- CONDITION
Revised: I. C ASE OF DASHA-SYNC-MVR UNDER PŁ- CONDITION

Original: Theorem
Revised: Theorem

Original: I.20.
Revised: I.20.

Original: Suppose that
Revised: Suppose that

Original: Assumption 5.1, 5.2, 1.2, 5.5, 5.
Revised: Assumption 5.1, 5.2, 1.2, 5.5, 5.

Original: and
Revised: and

Original: G. hold.
Revised: G. hold.

Original: Let us take a = 1 / (2 ω + 1) , probability p ∈ (0 , 1] and γ ≤ min
Revised: Let us take a = 1 / (2 ω + 1) , probability p ∈ (0 , 1] and γ ≤ min

Original: [Equation]
Revised: [Equation]

Original: n +
Revised: n +

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: p

Original: [Equation]
Revised: [Equation]

Original:   −, a 2 µ , p
Revised:   −, a 2 µ [Equation] p

Original: [Equation]
Revised: [Equation]

Original:   in Algorithm 1, then
Revised:   in Algorithm 1, then

Original: E (cid:2)
Revised: E [Equation]

Original: f
Revised: f

Original: ( x T

Original: [Equation]
Revised: [Equation]

Original: f
Revised: x T

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let us ﬁx constants κ, η, ν ∈ [0 , ∞ ) that we will deﬁne later.
Revised: Let us ﬁx constants κ, η, ν ∈ [0 , ∞ ) that we will deﬁne later.

Original: Using Lemma I.1, we can get (20).
Revised: Using Lemma I.1, we can get (20).

Original: Considering (20), Lemma I.17, Lemma I.18, and the law of total expectation, we obtain
Revised: Considering (20), Lemma I.17, Lemma I.18, and the law of total expectation, we obtain

Original: E (cid:2) f
Revised: E (cid:2) f

Original: ( x t +1 ) (cid:3) + κ E (cid:104)(cid:13)(cid:13) 
Revised: ( x t +1 ) (cid:3) + κ E (cid:104)(cid:13)(cid:13) g [Equation]

Original: g t +1 − h t +1 (cid:13)(cid:13) 2 (cid:105) + η
Revised: + [Equation] [Equation] + η

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: + ν E (cid:104)(cid:13)(cid:13) 
Revised: + [Equation] (cid:104)(cid:13)(cid:13) [Equation]

Original: h t +1 − ∇ f ( x t +1 ) (cid:13)(cid:13) 2 (cid:105) ≤ E (cid:20) f
Revised: ∇ [Equation] [Equation] 2 (cid:105) ≤ E (cid:20) f

Original: ( x t ) − γ 2 (cid:13)(cid:13) ∇ f ( x t ) (cid:13)(cid:13) 2 − (cid:18) 1 2 γ − L 2 (cid:19) (cid:13)(cid:13)
Revised: ( x t ) − γ 2 (cid:13)(cid:13) ∇ f ( x t ) (cid:13)(cid:13) 2 − (cid:18) 1 2 γ − L 2 (cid:19) (cid:13)(cid:13)

Original: x
Revised: x

Original: t +1 − x t (cid:13)(cid:13) 2 + γ (cid:13)(cid:13)
Revised: t +1 − x t (cid:13)(cid:13) 2 + γ (cid:13)(cid:13)

Original: g t − h t (cid:13)(cid:13) 2 + γ (cid:13)(cid:13)
Revised: g t − h t (cid:13)(cid:13) 2 + γ (cid:13)(cid:13)

Original: h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:21) + κ E   2 ω (1 − p ) (cid:16) L 2 σ B
Revised: h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:21) + κ E   2 ω (1 − p ) (cid:16) L 2 σ B

Original: + (cid:98) L 2 (cid:17) n (cid:13)(cid:13) x
Revised: + (cid:98) L 2 (cid:17) n (cid:13)(cid:13) x

Original: t +1 − x t (cid:13)(cid:13) 

Revised: t +1 − x t (cid:13)(cid:13)

Original: + 2 a 2 ω (1 − p ) nn (cid:88)
Revised: + 2 a 2 ω (1 − p ) nn (cid:88)

Original: i =
Revised: i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g ti − h ti (cid:13)(cid:13) 

Revised: g ti − h ti (cid:13)(cid:13)

Original: + (1 − p ) (1 − a ) 2 (cid:13)(cid:13)
Revised: + (1 − p ) (1 − a ) 2 (cid:13)(cid:13)

Original: g t − h t (cid:13)(cid:13)  + η E (cid:34) 2 ω (1 − p ) (cid:18)
Revised: g t − h t (cid:13)(cid:13)  + η E (cid:34) 2 ω (1 − p ) (cid:18)

Original: L 2 σ
Revised: L 2 σ

Original: B + (cid:98) L(cid:19)
Revised: B + (cid:98) L(cid:19)

Original: (cid:13)(cid:13) x
Revised: (cid:13)(cid:13) x

Original: t +1 − x t (cid:13)(cid:13) 

Revised: t +1 − x t (cid:13)(cid:13)

Original: + (1 − p ) (cid:16) 2 a 2 ω + (1 − a ) 2 (cid:17) 1 n n (cid:88)
Revised: + (1 − p ) (cid:16) 2 a 2 ω + (1 − a ) 2 (cid:17) 1 n n (cid:88)

Original: i =
Revised: i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g ti − h ti (cid:13)(cid:13)(cid:35)
Revised: g ti − h ti (cid:13)(cid:13)(cid:35)

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]
Revised: [Table]

Original: After rearranging the terms
Revised: After rearranging the terms

Original: [Table]
Revised: [Table]

Original: we get
Revised: we get

Original: [Table]
Revised: [Table]

Original: [Equation] a ) [Equation] g
Revised: 1 − a ) 2 (cid:1) E

Original: [Equation] − h [Equation] + 
Revised: (cid:104)(cid:13)(cid:13) g t − h t (cid:13)(cid:13) 2 (cid:105) + (cid:18) 2 κa 2 ω (1 − p ) n + η (1 − p ) (cid:16) 2 a 2 ω + (1 − a ) 2 (cid:17)(cid:19) E (cid:34)n n (cid:88) i =

Original: [Equation] n [Equation]
Revised: (cid:13)(cid:13)

Original: (cid:35) + ( γ [Equation] [Equation]
Revised: g ti − h ti (cid:13)(cid:13)(cid:35)

Original: h [Equation] f [Equation]
Revised: + ( γ + ν (1 − p ))

Original:  (cid:105) + [Equation] nB (cid:48) .
Revised: E (cid:104)(cid:13)(cid:13) h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105) + νpσnB (cid:48) .

Original: Let us take ν = 2 γp , κ = 2 γa , a = 1 2 ω +1 , and η = 8 γωn .
Revised: Let us take ν = 2 γp , κ = 2 γa , a = 1 2 ω +1 , and η = 8 γωn .

Original: Thus γ + κ (1 − p )(1 − a ) 2 ≤ (cid:0) 1 − a 2 (cid:1) κ, γ + ν (1 − p ) =
Revised: Thus γ + κ (1 − p )(1 − a ) 2 ≤ (cid:0) 1 − a 2 (cid:1) κ, γ + ν (1 − p ) =

Original: (cid:0) 1 − p 2 (cid:1) ν, 2 κa 2 ω (1 − p ) n + η (1 − p ) (cid:16) 2 a 2 ω
Revised: (cid:0) 1 − p 2 (cid:1) ν, 2 κa 2 ω (1 − p ) n + η (1 − p ) (cid:16) 2 a 2 ω

Original: + (1 − a ) 2 (cid:17) ≤ (cid:0) 1 − a 2 (cid:1) η, and
Revised: + (1 − a ) 2 (cid:17) ≤ (cid:0) 1 − a 2 (cid:1) η, and

Original: E (cid:2) f
Revised: E (cid:2) f

Original: ( x t +1 ) (cid:3) + 2 γ (2 ω + 1)E (cid:104)(cid:13)(cid:13) g t
Revised: ( x t +1 ) (cid:3) + 2 γ (2 ω + 1)E (cid:104)(cid:13)(cid:13) g t

Original: +1 − h t +1 (cid:13)(cid:13) 2 (cid:105) + 8 γω n E (cid:34)n n (cid:88) i =
Revised: +1 − h t +1 (cid:13)(cid:13) 2 (cid:105) + 8 γω n E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g t +

Revised: g t +

Original: i − h t
Revised: i − h t

Original: +1 i (cid:13)(cid:13) 2 (cid:35)
Revised: +1 i (cid:13)(cid:13) 2 (cid:35)

Original: +
Revised: +

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: + [Equation] nB (cid:48) .
Revised: + 2 γσnB (cid:48) .

Original: In the view of the choice of γ and Lemma I.7, one can show that 12 γ − L 2 − 40 γω (2 ω +1)(1 − p ) (cid:18)
Revised: In the view of the choice of γ and Lemma I.7, one can show that 12 γ − L 2 − 40 γω (2 ω +1)(1 − p ) (cid:18)

Original: L 2 σB + (cid:98) L 2 (cid:19) n − 4 γ (1 − p )
Revised: L 2 σB + (cid:98) L 2 (cid:19) n − 4 γ (1 − p )

Original: L 2 σ pnB ≥ 0 , 1 − a 2 ≤ 1 − γµ, and 1 − p 2 ≤ 1 − γµ, thus
Revised: L 2 σ pnB ≥ 0 , 1 − a 2 ≤ 1 − γµ, and 1 − p 2 ≤ 1 − γµ, thus

Original: E 
Revised: E (cid:2) f ( x t +1 ) (cid:3) + [Equation] (2 ω +

Original: [Equation]
Revised: [Equation]

Revised: E (cid:104)(cid:13)(cid:13) g

Original: [Table]
Revised: [Equation] (cid:13)(cid:13) [Equation] (cid:105)

Revised: +

Original: [Equation]
Revised: [Equation]

Revised: n E (cid:34)n n (cid:88) i =

Original: [Table]{+(cid:13)(cid:13) g t +
 i − h t +1 i (cid:13)(cid:13)+}
Revised: {+(cid:13)(cid:13) g t +
 i − h t +1 i (cid:13)(cid:13)+}

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: (cid:35) + 2 γ p [Equation] (cid:104)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: t + [Equation] ( x t + [Equation]

Revised: (cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: (cid:105) [Equation] [Equation] (cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: (cid:105) [Equation] (cid:34) [Equation]

Revised: n

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: i = (cid:13)(cid:13) [Equation] (cid:13)(cid:13)(cid:35) + (1 − γµ ) [Equation]

Revised: γ p E (cid:104)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: In the view of Lemma I.6 with
Revised: In the view of Lemma I.6 with

Original: [Equation]
Revised: [Equation]

Original: and C = 

Revised: and C =

Original: σ 2 nB (cid:48)
Revised: σ 2 nB (cid:48)

Original: , we can conclude the proof.
Revised: , we can conclude the proof.

Original: Corollary I.21.
Revised: Corollary I.21.

Original: Suppose that assumptions from Theorem I.20 hold, probability p = min (cid:110) ζ C d , µnεBσ 2 (cid:111) , batch size B (cid:48)
Revised: Suppose that assumptions from Theorem I.20 hold, probability p = min (cid:110) ζ C d , µnεBσ 2 (cid:111) , batch size B (cid:48)

Original: =
Revised: =

Original: Θ (cid:16) σ 2 µnε (cid:17) , and h 0 i = g 0 i = 0 for all i ∈ [ n ] , then DASHASYNC-MVR needs
Revised: Θ (cid:16) σ 2 µnε (cid:17) , and h 0 i = g 0 i = 0 for all i ∈ [ n ] , then DASHASYNC-MVR needs

Original: [Equation]
Revised: [Equation]

Original: communication rounds to get an ε -solution, the communication complexity is equal to O ( ζ C T ) , and the number of stochastic gradient calculations per node equals O ( BT ) , where ζ C is the expected density from Deﬁnition 1.3.
Revised: communication rounds to get an ε -solution, the communication complexity is equal to O ( ζ C T ) , and the number of stochastic gradient calculations per node equals O ( BT ) , where ζ C is the expected density from Deﬁnition 1.3.

Original: Proof.
Revised: Proof.

Original: Considering the choice of B (cid:48) , we have 2 σ 2 nµB (cid:48)
Revised: Considering the choice of B (cid:48) , we have 2 σ 2 nµB (cid:48)

Original: =
Revised: =

Original: O ( ε ) .
Revised: O ( ε ) .

Original: Therefore, is it enough to take the number of communication rounds equals (28) to get an ε -solution.
Revised: Therefore, is it enough to take the number of communication rounds equals (28) to get an ε -solution.

Original: It is left to estimate the communication and oracle complexity.
Revised: It is left to estimate the communication and oracle complexity.

Original: On average, in Algorithm 2, at each communication round the number of coordinates that each node sends equals pd + (1 − p ) ζ C ≤ ζ C d d + (cid:16) 1 − ζ C d (cid:17) ζ C ≤ 2 ζ C .
Revised: On average, in Algorithm 2, at each communication round the number of coordinates that each node sends equals pd + (1 − p ) ζ C ≤ ζ C d d + (cid:16) 1 − ζ C d (cid:17) ζ C ≤ 2 ζ C .

Original: Therefore, the communication complexity is equal to O ( ζ C T ) .
Revised: Therefore, the communication complexity is equal to O ( ζ C T ) .

Original: Considering the fact that we use a mini-batch of stochastic gradients, on average, the number of stochastic gradients that each node calculates at each communication round equals pB (cid:48)
Revised: Considering the fact that we use a mini-batch of stochastic gradients, on average, the number of stochastic gradients that each node calculates at each communication round equals pB (cid:48)

Original: +(1 − p )2 B =
Revised: +(1 − p )2 B =

Original: O (cid:16) µnεBσ 2 · σ 2 µnε (cid:17) + 2 B
Revised: O (cid:16) µnεBσ 2 · σ 2 µnε (cid:17) + 2 B

Original: =
Revised: =

Original: O
Revised: O

Original: ( B ) , thus the number of stochastic gradients that each node calculates equals O ( BT ) .
Revised: ( B ) , thus the number of stochastic gradients that each node calculates equals O ( BT ) .

Original: Unlike Corollary 6.10, in this corollary, we can initialize h 

Revised: Unlike Corollary 6.10, in this corollary, we can initialize h

Original: i and g 0 i , for instance, with zeros because the corresponding initialization error Ψ 0 from the proof of Theorem I.20 would be under the logarithm.
Revised: i and g 0 i , for instance, with zeros because the corresponding initialization error Ψ 0 from the proof of Theorem I.20 would be under the logarithm.

Original: J E XTRA E XPERIMENTS
Revised: J E XTRA E XPERIMENTS

Original: DASHA-MVR improves VR-MARINA (online) when ε is small (see Tables 1 and 2 and experiments in Section A).
Revised: DASHA-MVR improves VR-MARINA (online) when ε is small (see Tables 1 and 2 and experiments in Section A).

Original: However, our analysis shows that DASHA-MVR gets a term Bω (cid:113) σ 2 εnB in the oracle complexity and a term ω (cid:113) σ 2 µεnB in the number of communication rounds in general nonconvex and PŁ settings accordingly.
Revised: However, our analysis shows that DASHA-MVR gets a term Bω (cid:113) σ 2 εnB in the oracle complexity and a term ω (cid:113) σ 2 µεnB in the number of communication rounds in general nonconvex and PŁ settings accordingly.

Original: Both terms can be a bottleneck in some regimes; now, we verify this dependence in the PŁ setting.
Revised: Both terms can be a bottleneck in some regimes; now, we verify this dependence in the PŁ setting.

Original: We take a synthetically generated stochastic quadratic optimization problem with one node ( n = 1 ):
Revised: We take a synthetically generated stochastic quadratic optimization problem with one node ( n = 1 ):

Original: [Equation]
Revised: [Equation]

Original: where A ∈ R d × d , b ∈ R d , A = A (cid:62) (cid:31) 0 , and ξ ∼ Normal (cid:0) 0 , σ 2 (cid:1) .
Revised: where A ∈ R d × d , b ∈ R d , A = A (cid:62) (cid:31) 0 , and ξ ∼ Normal (cid:0) 0 , σ 2 (cid:1) .

Original: We generate A in such way, that µ ≈ 1 . 

Revised: We generate A in such way, that µ ≈ 1 .

Original: ≤
Revised: ≤

Original: L ≈ 2 . 0 , take d = 10 4 , σ 2 = 1 . 0 , Rand K with K =( ω
Revised: L ≈ 2 . 0 , take d = 10 4 , σ 2 = 1 . 0 , Rand K with K =( ω

Original: ≈ d ), batch size B = 1 , and σ 2 µεnB = 10 4 .
Revised: ≈ d ), batch size B = 1 , and σ 2 µεnB = 10 4 .

Original: With this particular choice of parameters, ω (cid:113) σ 2 µεnB would dominate in the number of communication rounds T=
Revised: With this particular choice of parameters, ω (cid:113) σ 2 µεnB would dominate in the number of communication rounds T

Original: [Equation] (cid:113)
Revised: = ω + ω (cid:113)

Original: [Equation]
Revised: [Equation]

Original: Lσ µ 3 / 2 √ εnB .
Revised: Lσ µ 3 / 2 √ εnB .

Original: Results are provided in Figure 5.
Revised: Results are provided in Figure 5.

Original: We consider DASHA-MVR with a momentum b from Corollary I. and b = min (cid:110) 1 ω , µnεBσ 2 (cid:111) .
Revised: We consider DASHA-MVR with a momentum b from Corollary I. and b = min (cid:110) 1 ω , µnεBσ 2 (cid:111) .

Original: With the latter choice of momentum b , DASHA-MVR converges at the same rate as DASHA-SYNC-MVR or VR-MARINA (online) but to an ε -solution with a smaller ε .
Revised: With the latter choice of momentum b , DASHA-MVR converges at the same rate as DASHA-SYNC-MVR or VR-MARINA (online) but to an ε -solution with a smaller ε .

Original: On the other hand, the former choice of momentum b guarantees the convergence to the correct ε -solution, but with a slower rate.
Revised: On the other hand, the former choice of momentum b guarantees the convergence to the correct ε -solution, but with a slower rate.

Original: Overall, the experiment provides the pieces of evidence that our choice of b is correct and that our analysis in Theorem I.15 is tight.
Revised: Overall, the experiment provides the pieces of evidence that our choice of b is correct and that our analysis in Theorem I.15 is tight.

Original: If we decrease ω from 10 4 to 10 3 (see Figure 6), or σ 2 from 1 . 0 to 0 .
Revised: If we decrease ω from 10 4 to 10 3 (see Figure 6), or σ 2 from 1 . 0 to 0 .

Original: 1 (see Figure 7), or µ from 1 .to 0 .
Revised: 1 (see Figure 7), or µ from 1 .to 0 .

Original: 1 (see Figure 8), then the gap between algorithms closes.
Revised: 1 (see Figure 8), then the gap between algorithms closes.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]