Original: ( x t +1 ) (cid:3) + γ (2 ω + 1)E (cid:104)(cid:13)(cid:13) g t
Revised: ( x t +1 ) (cid:3) + γ (2 ω + 1)E (cid:104) (cid:13) (cid:13) g t

Original: {++
+}
Revised: {++
+}

Original: +1 − h t +1 (cid:13)(cid:13) 2 (cid:105) + 2 γω n E (cid:34)n n (cid:88) i = 
Revised: − h t +1 (cid:13) (cid:13) 2 (cid:105) + 2 γω n E (cid:34)n n (cid:88) i = (cid:13) (cid:13) g t +1 i − h t +1 i (cid:13) (cid:13)(cid:35) +

Revised: γ p E (cid:104)(cid:13)(cid:13) h t +1 − ∇ f ( x t +1 )

Revised: (cid:13)(cid:13) 2 (cid:105) ≤ E (cid:2) f

Revised: ( x t ) (cid:3) − γ 2 E (cid:104)(cid:13)(cid:13) ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105)

Revised: [Table]

Revised: g t − h t (cid:13)(cid:13) 2 (cid:105) + 2 γω n E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Revised: g t i − h t

Revised: i (cid:13)(cid:13)(cid:35) + γ p E (cid:104)(cid:13)(cid:13) h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105) + γσnB (cid:48) .

Original: [-g t +
-]
Revised: [-g t +
-]In the view of the choice of γ , we obtain E [Equation] f

Revised: [Equation]

Original: i − h t 
Revised: t [Equation] (cid:13)(cid:13) [Equation]

Original: +1 i (cid:13)(cid:13) 2 (cid:35) + γ p E (cid:104)(cid:13)(cid:13) h t +1 − ∇ f ( x t +1 ) (cid:13)(cid:13) 2 (cid:105)≤ E (cid:2) f ( x t ) (cid:3) − γ 2 E (cid:104) (cid:13)(cid:13) ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105) −  
Revised: (cid:105)

Original: [Equation]
Revised: [Equation]

Original: γ − L [Equation] − 2 γω (
Revised: 2 γω n E (cid:34)n n (cid:88) i =

Original: [Equation]
Revised: (cid:13)(cid:13)

Original: ω [Equation]{+g t +
+}
Revised: {+g t +
+}

Original: [Equation]
Revised: i − h t

Original: − p ) (cid:16) L 2 σ B + (cid:98)
Revised: +1 i (cid:13)(cid:13) 2 (cid:35) + γ p

Original: [Equation]
Revised: [Equation]

Original: (cid:17) n − 4 γω 2 (1 − p ) (cid:16) L 2 σ B
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: (cid:98) L 2 (cid:17) n − γ (1 − p ) L 2 σ pnB   E (cid:104)(cid:13)(cid:13) x t +1 − x t (cid:13)(cid:13) 2 (cid:105)
Revised: (cid:105)

Original: +γ
Revised: +

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: h
Revised: g

Original: [Equation]
Revised: [Equation]

Original: n
Revised: − h

Original: [Equation]
Revised: [Equation]

Original: ti
Revised: (cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: ti (cid:13)(cid:13)
Revised: (cid:105)

Original: [Equation]
Revised: [Equation]

Revised: 2 γω n E (cid:34)n n (cid:88) i =

Revised: (cid:13)(cid:13)

Revised: g ti − h ti (cid:13)(cid:13)(cid:35) + γ p

Original: [Equation]
Revised: [Equation]

Original: In the view of the choice of γ
Revised: +

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: nB

Original: [Equation]
Revised: [Equation]

Original: Finally, using Lemma H.5 with
Revised: Finally, using Lemma I.5 with

Original: [Equation]
Revised: [Equation]

Original: and C =
Revised: and C =

Original: σ 2 nB (cid:48) , we can conclude the proof.
Revised: σ 2 nB (cid:48) , we can conclude the proof.

Original: Corollary 6.10.
Revised: Corollary 6.10.

Original: Suppose that assumptions from Theorem H.19 hold, probability p = min (cid:110) ζ C d , nεBσ 2 (cid:111) , batch size B (cid:48) =
Revised: Suppose that assumptions from Theorem I.19 hold, probability p = min (cid:110) ζ C d , nεBσ 2 (cid:111) , batch size B (cid:48) =

Original: Θ (cid:16) σ 2 nε (cid:17) and h 0 i = g 0 i = 1 B init (cid:80) B init k
Revised: Θ (cid:16) σ 2 nε (cid:17) and h 0 i = g 0 i = 1 B init (cid:80) B init k

Original: =1 ∇ f i ( x 0 ; ξ 0 ik ) for all i ∈ [ n ] , initial batch size B init =
Revised: =1 ∇ f i ( x 0 ; ξ 0 ik ) for all i ∈ [ n ] , initial batch size B init =

Original: Θ (cid:16) max (cid:110) σ 2 nε , B dζ C (cid:111)(cid:17) , then DASHA-SYNC-MVR needs
Revised: Θ (cid:16) max (cid:110) σ 2 nε , B dζ C (cid:111)(cid:17) , then DASHA-SYNC-MVR needs

Original: T :
Revised: T :

Original: = O 
Revised: = O 

Original: [Equation]
Revised: [Equation]

Original:   communication rounds to get an ε -solution, the communication complexity is equal to O ( d + ζ C T ) , and the number of stochastic gradient calculations per node equals O ( B init + BT ) , where ζ C is the expected density from Deﬁnition 1.3.
Revised: nεB   communication rounds to get an ε -solution, the communication complexity is equal to O ( d + ζ C T ) , and the number of stochastic gradient calculations per node equals O ( B init + BT ) , where ζ C is the expected density from Deﬁnition 1.3.

Original: Proof.
Revised: Proof.

Original: Considering Theorem H.19 and the choice of B (cid:48)
Revised: Considering Theorem I.19 and the choice of B (cid:48)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: ≤

Original: T   2 (cid:0) f

Original: ( x 0 ) − f ∗ (cid:1)   L + (cid:118)(cid:117)(cid:117)(cid:116) 12 ω (2 ω + 1)(1 − p ) (cid:16) L 2 σ B

Original: + (cid:98) L 2 (cid:17) n + 2(1 − p )

Original: L 2 σ pnB  

Original: +p (cid:13)(cid:13) h 0 − ∇ f ( x 0 ) (cid:13)(cid:13)  + 2 3 ε.

Original: Due to p = min (cid:110) ζ C d , nεBσ 2 (cid:111), we have
Revised: Due to p = min (cid:110) ζ C d , nεBσ 2 (cid:111)

Original: E (cid:104)(cid:13)(cid:13)

Original: [Table]
Revised: [Equation]

Original:  
Revised: we have

Original: [Table]
Revised: [Equation]

Original: (cid:118)(cid:117)(cid:117)(cid:116) [Table] [Table]
Revised: ≤ O   1 T 

Original: [Table]
Revised: [Equation]

Original: p
Revised: n

Original: [Table]
Revised: [Equation]

Original: (cid:16)

Original: [Table] + (cid:98) [Table] (cid:17) n +
Revised: +

Original: [Table]
Revised: [Equation]

Original: d ( [Table] p [Table]
Revised: ≤ O 

Original: L

Original: [Table]
Revised: [Equation]

Original: ζ C nB +2 σ [Table] [Table] p )
Revised:  +

Original: [Table] [Table]
Revised: [Equation]

Original: εn 2 B  [Table] nεB(cid:19)
Revised: nεB

Original: [Table]
Revised: [Equation]

Original: 
Revised: 

Original: [Table]

Original: [Equation]

Original: n

Original: [Equation]
Revised: [Equation]

Original: +

Original: [Equation] d [Equation] nεB
Revised: 

Original: [Equation]
Revised: [Equation]

Original:  
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: Therefore, we can take
Revised: we can take

Original: T =
Revised: T =

Original: O 
Revised: O 

Original: [Equation]
Revised: [Equation]

Original:     .
Revised: C n

Original: Note, that E [Equation]
Revised: L σ √ B + (cid:114) σεn 2 B

Original: n [Equation]
Revised: L σ √

Original: [Equation] E (cid:104)(cid:13)(cid:13)
Revised: B (cid:33) + (cid:18) d ζ C + σnεB (cid:19) (cid:13) (cid:13) h 0 − ∇ f ( x 0 ) (cid:13) (cid:13)      . [Table]

Original: [Equation]
Revised: [Equation]

Original: ≤

Original: [Equation]
Revised: [Equation]

Original: nB init .

Original: Next, by taking B init =
Revised: Next, by taking B init =

Original: max (cid:110) σ 2 nε, B dζ C (cid:111) and using the last ineqaulity, we have T = O 
Revised: max (cid:110) σ 2 nε

Original: [Equation]
Revised: [Equation]

Revised: B dζ C (cid:111) and using the last ineqaulity

Original: [Equation]
Revised: [Equation]

Original: min
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: = O  [Equation]
Revised: Finally

Original: [Equation]
Revised: [Equation]

Original: nεB   .

Original: Finally, it is left to estimate the communication and oracle complexity.
Revised: it is left to estimate the communication and oracle complexity.

Original: On average, the number of coordinates that each node in Algorithm 2 sends at each communication round equals pd + (1 − p ) ζ C ≤ ζ C d d + (cid:16) 1 − ζ C d (cid:17) ζ C ≤ 2 ζ C .
Revised: On average, the number of coordinates that each node in Algorithm 2 sends at each communication round equals pd + [Equation]

Original: Therefore, the communication complexity is equal to
Revised: Therefore, the communication complexity is equal to

Original: O ( d + ζ C T ) .
Revised: O ( d + ζ C T ) .

Original: Considering the fact that we use a mini-batch of stochastic gradients, on average, the number of stochastic gradients that each node calculates at each communication round equals pB (cid:48)
Revised: Considering the fact that we use a mini-batch of stochastic gradients, on average, the number of stochastic gradients that each node calculates at each communication round equals pB (cid:48)

Original: + (1 − p )2 B ≤ O (cid:16) nεBσ 2 · σ 2 nε (cid:17) + 2 B = O ( B ) .
Revised: + (1 − p )2 B ≤ O (cid:16) nεBσ 2 · σ 2 nε (cid:17) + 2 B = O ( B ) .

Original: Considering the initial batch size B init , the number of stochastic gradients that each node calculates equals O ( B init + BT ) .
Revised: Considering the initial batch size B init , the number of stochastic gradients that each node calculates equals O ( B init + BT ) .

Original: Corollary 6.11.
Revised: Corollary 6.11.

Original: Suppose that assumptions of Corollary 6.10 hold, batch size B ≤ σ √ εn , we take
Revised: Suppose that assumptions of Corollary 6.10 hold, batch size B ≤ σ √ εn , we take

Original: Rand K with K = ζ C =
Revised: Rand K with K = ζ C =

Original: Θ (cid:16) Bd √ εn σ (cid:17) , and (cid:101)
Revised: Θ (cid:16) Bd √ εn σ (cid:17) , and (cid:101)

Original: L :=
Revised: L :=

Original: max { L, L σ , (cid:98) L } .
Revised: max { L, L σ , (cid:98) L } .

Original: Then the communication complexity equals
Revised: Then the communication complexity equals

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: and the expected # of stochastic gradient calculations per node equals
Revised: and the expected # of stochastic gradient calculations per node equals

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: In the view of Theorem F.2, we have ω +

Revised: In the view of Theorem F.2, we have ω +

Original: =
Revised: =

Original: d/K. Moreover, K = Θ (cid:16)
Revised: d/K. Moreover, K = Θ (cid:16)

Original: Bd √ εn σ(cid:17) = O (cid:16)
Revised: Bd √ εn σ

Original: [Equation]
Revised: [Equation]

Original: n (cid:17) , thus the communication complexity equalsO ( d + ζ C T ) =
Revised: , thus the communication complexity equals

Original: O
Revised: O

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: And the expected number of stochastic gradient calculations per node equals
Revised: And the expected number of stochastic gradient calculations per node equals

Original: O ( B init + BT ) =
Revised: O ( B init + BT ) =

Original: O 

Original: [Equation]

Original: = O  

Original: [Equation]

Original: nε

Original: [Equation]

Original: 

Original: [Equation]
Revised: [Equation]

Original: 

Original: [Equation]

Original: H. C ASE OF DASHA-SYNC-MVR UNDER PŁ- CONDITION
Revised: I. C ASE OF DASHA-SYNC-MVR UNDER PŁ- CONDITION

Original: Theorem
Revised: Theorem

Original: H.20.
Revised: I.20.

Original: Suppose that
Revised: Suppose that

Original: Assumption 5.1, 5.2, 1.2, 5.5, 5.
Revised: Assumption 5.1, 5.2, 1.2, 5.5, 5.

Original: and
Revised: and

Original: G. hold.
Revised: G. hold.

Original: Let us take a = 1 / (
Revised: Let us take a = 1 / (2 ω + 1) , probability p ∈ (0 , 1] and γ ≤ min

Original: [Equation]
Revised: [Equation]

Original: ω
Revised: n +

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , probability p∈ (
Revised: p

Original: [Equation]
Revised: [Equation]

Original: 1] and γ ≤ min
Revised:   −, a 2 µ , p

Original: [Equation]
Revised: [Equation]

Original: 
Revised:   in Algorithm 1, then E (cid:2) f ( x T

Original: [Equation]
Revised: [Equation]

Original:   in Algorithm 1, then
Revised: f

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let us ﬁx constants κ, η, ν ∈ [0 , ∞ ) that we will deﬁne later.
Revised: Let us ﬁx constants κ, η, ν ∈ [0 , ∞ ) that we will deﬁne later.

Original: Using Lemma H.1, we can get (20).
Revised: Using Lemma I.1, we can get (20).

Original: Considering (20), Lemma H.17, Lemma H.18, and the law of total expectation, we obtain
Revised: Considering (20), Lemma I.17, Lemma I.18, and the law of total expectation, we obtain

Original: E (cid:2) f
Revised: E (cid:2) f

Original: ( x t +1 ) (cid:3) + κ E (cid:104)(cid:13)(cid:13)g t +
Revised: ( x t +1 ) (cid:3) + κ E (cid:104)(cid:13)(cid:13)

Original: [Equation]

Original:  h t +1 (cid:13)(cid:13) 2 (cid:105) + ηE (cid:34)
Revised: g t +1 − h t +1 (cid:13)(cid:13) 2 (cid:105) + η

Original: [Equation]
Revised: [Equation]

Original: n n (cid:88) i =

Original: (cid:13)(cid:13)

Original: g
Revised: n

Original: [Equation]
Revised: [Equation]

Original: [-+
 i [Equation] +1 i (cid:13)(cid:13)-]
Revised: [-+
 i [Equation] +1 i (cid:13)(cid:13)-]+ ν E (cid:104)(cid:13)(cid:13)

Original: [Equation] + ν E (cid:104)(cid:13)(cid:13) h t + [Equation] ∇ f 
Revised: h t +1 − ∇ f ( x t +1 ) (cid:13)(cid:13) 2 (cid:105) ≤ E (cid:20) f

Original: [Equation]

Original: x t +

Original: [Equation]

Original:  (cid:13)(cid:13) [Equation] (cid:105) ≤ E (cid:20) f [Equation]
Revised: ( x t ) − γ 2 (cid:13)(cid:13) ∇ f ( x t ) (cid:13)(cid:13) 2 − (cid:18) 1 2 γ − L 2 (cid:19) (cid:13)(cid:13)

Original: t
Revised: x

Original: [Equation] (cid:13)(cid:13)[Equation]
Revised: t +1 − x t (cid:13)(cid:13) 2 + γ (cid:13)(cid:13)

Original: γ (cid:13)(cid:13) [Equation] (cid:13)(cid:13)
Revised: g t − h t (cid:13)(cid:13) 2 + γ (cid:13)(cid:13)

Original: [Equation] γ (cid:13)(cid:13) h 
Revised: h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:21) + κ E   2 ω (1 − p ) (cid:16) L 2 σ B

Original: [Equation] ( x t ) (cid:13)(cid:13) [Equation] (cid:21) + κ E   2 ω (1 − p ) (cid:16)
Revised: + (cid:98) L 2 (cid:17) n (cid:13)(cid:13) x

Original: [Equation]

Original: + (cid:98)

Original: [Equation]

Original: (cid:17) n (cid:13)(cid:13) x

Original: [Equation]

Original: +

Original: [Equation] − x [Equation] (cid:13)(cid:13) {+
+}
Revised: t +1 − x t (cid:13)(cid:13) {+
+}

Original: [Equation]

Original: [Equation]

Original:  (1 − p ) n [Equation] n (cid:88) i =
Revised: + 2 a 2 ω (1 − p ) nn (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g ti − h ti (cid:13)(cid:13) {+
+}
Revised: g ti − h ti (cid:13)(cid:13) {+
+}

Original: [Equation]

Original:  (1 − p ) (1 − a ) 2 (cid:13)(cid:13)
Revised: + (1 − p ) (1 − a ) 2 (cid:13)(cid:13)

Original: g t − h t (cid:13)(cid:13)  + η E (cid:34) 2 ω (1 − p ) (cid:18)
Revised: g t − h t (cid:13)(cid:13)  + η E (cid:34) 2 ω (1 − p ) (cid:18)

Original: L 2 σ
Revised: L 2 σ

Original: B + (cid:98) L(cid:19) (cid:13) (cid:13) x
Revised: B + (cid:98) L(cid:19) (cid:13)(cid:13) x

Original: t +1 − x t (cid:13) (cid:13) 2{+(cid:13)(cid:13) 
+} + (1 − p ) (cid:16) 2 a 2 ω + (1 − a ) 2 (cid:17) 1 n n (cid:88)i = (cid:13) (cid:13) g ti − h ti (cid:13) (cid:13)(cid:35) + ν E (cid:20) pσnB (cid:48)
Revised: t +1 − x t {+(cid:13)(cid:13) 
+} + (1 − p ) (cid:16) 2 a 2 ω + (1 − a ) 2 (cid:17) 1 n n (cid:88)

Original: + (1 − p )
Revised: i =

Original: L 2 σ nB (cid:13)(cid:13) 
Revised: (cid:13)(cid:13) g ti − h ti (cid:13)(cid:13)(cid:35)

Original: x
Revised: [Equation] [Table]

Original: [-t +1 − x t (cid:13)(cid:13) 
-]
Revised: [-t +1 − x t (cid:13)(cid:13) 
-][Equation] [Table] [Equation] [Table]

Original: + (1 − p ) (cid:13)(cid:13) h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:21)
Revised: [Equation] [Table] [Equation] [Table]

Original: After rearranging the terms, we get
Revised: After rearranging the terms [Table] we get

Revised: [Table]

Original: E (cid:2) f
Revised: [Equation] a )

Original: ( x t +1 ) (cid:3) + κ E (cid:104)(cid:13)(cid:13)
Revised: [Equation] g [Equation] − h

Original: g t +1 − h t +1 (cid:13)(cid:13) 2 (cid:105) + η E (cid:34)n n(cid:88) i =
Revised: [Equation] + [Equation] n

Original: (cid:13)(cid:13)
Revised: [Equation] (cid:35)

Original: [-g t +
-]
Revised: [-g t +
-]+ ( γ

Original: [-i − h t +1 i (cid:13)(cid:13) 2 (cid:35) + ν E (cid:104)(cid:13)(cid:13) h t +1 − ∇ f ( x t +1 ) (cid:13)(cid:13) 2 (cid:105) ≤ E (cid:2) f ( x t ) (cid:3) − γ 2 E (cid:104)(cid:13) (cid:13) ∇ f ( x t ) (cid:13) (cid:13) 2 (cid:105) −   
-]
Revised: [-i − h t +1 i (cid:13)(cid:13) 2 (cid:35) + ν E (cid:104)(cid:13)(cid:13) h t +1 − ∇ f ( x t +1 ) (cid:13)(cid:13) 2 (cid:105) ≤ E (cid:2) f ( x t ) (cid:3) − γ 2 E (cid:104)(cid:13) (cid:13) ∇ f ( x t ) (cid:13) (cid:13) 2 (cid:105) −   
-][Equation]

Original: [Equation]
Revised: [Equation]

Revised: h

Original: [Equation]
Revised: [Equation]

Original: n
Revised: f

Original: [Equation]
Revised: [Equation]

Original: nB
Revised: (cid:105) +

Original: [Equation]
Revised: [Equation]

Revised: nB (cid:48) .

Original: Let us take ν = 2 γp , κ = 2 γa , a = 1 2 ω +1 , and η = 8 γωn .
Revised: Let us take ν = 2 γp , κ = 2 γa , a = 1 2 ω +1 , and η = 8 γωn .

Original: Thus γ + κ (1 − p )(1 − a ) 2 ≤ (cid:0) 1 − a 2 (cid:1) κ, γ + ν (1 − p ) =
Revised: Thus γ + κ (1 − p )(1 − a ) 2 ≤ (cid:0) 1 − a 2 (cid:1) κ, γ + ν (1 − p ) =

Original: (cid:0) 1 − p 2 (cid:1) ν, 2 κa 2 ω (1 − p ) n + η (1 − p ) (cid:16) 2 a 2 ω
Revised: (cid:0) 1 − p 2 (cid:1) ν, 2 κa 2 ω (1 − p ) n + η (1 − p ) (cid:16) 2 a 2 ω

Original: + (1 − a ) 2 (cid:17) ≤ (cid:0) 1 − a 2 (cid:1) η, and
Revised: + (1 − a ) 2 (cid:17) ≤ (cid:0) 1 − a 2 (cid:1) η, and

Original: E (cid:2) f
Revised: E (cid:2) f

Original: ( x t +1 ) (cid:3) + 2 γ (2 ω + 1)E (cid:104)(cid:13)(cid:13) g t
Revised: ( x t +1 ) (cid:3) + 2 γ (2 ω + 1)E (cid:104)(cid:13)(cid:13) g t

Original: +1 − h t +1 (cid:13)(cid:13) 2 (cid:105) + 8 γω n E (cid:34)n n (cid:88) i =
Revised: +1 − h t +1 (cid:13)(cid:13) 2 (cid:105) + 8 γω n E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g t +

Revised: g t +

Original: i − h t
Revised: i − h t

Original: +1 i (cid:13)(cid:13) 2 (cid:35) + 2 γ p E (cid:104)(cid:13)(cid:13)
Revised: +1 i (cid:13)(cid:13) 2 (cid:35) + [Table] [Equation]

Original: h t +1 − ∇ f ( x t +1 ) (cid:13)(cid:13)
Revised: [Table] [Equation] [Table] [Equation]

Original: [Table]
Revised: [Table]

Revised: [Equation]

Original: p ) (cid:16) L 2 σ B
Revised: [Table] [Equation] [Table] [Equation]

Original: + (cid:98) L 2 (cid:17) n − 2 γ (1 − p ) L 2 σ pnB 
Revised: [Table] [Equation] [Table] [Equation]

Original:  E (cid:104)(cid:13)(cid:13) x t +1 − x t (cid:13)(cid:13) 2 (cid:105) + (cid:16) 1 − a 2 (cid:17) 2 γ (2 ω + 1)E
Revised: [Table] [Equation] [Table] [Equation]

Original: (cid:104)(cid:13)(cid:13) g t − h t (cid:13)(cid:13) 2 (cid:105) + (cid:16) 1 − a 2 (cid:17) 8 γω n E (cid:34)n n (cid:88) i =
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: (cid:13)(cid:13)
Revised: [Equation] [Table] [Equation] [Table]

Revised: [Equation]

Original: g ti − h ti (cid:13)(cid:13)(cid:35) + (cid:16) 1 − p 2 (cid:17) 2 γ p E
Revised: [Table] [Equation] [Table] [Equation]

Original: (cid:104)(cid:13)(cid:13) h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105) + 2 γσnB (cid:48) .
Revised: [Table] [Equation] [Table] [Equation]

Original:  In the view of the choice of γ and Lemma H.7, one can show that 12 γ − L 2 − 40 γω (2 ω +1)(1 − p ) (cid:18)
Revised: + [Equation] nB (cid:48) . In the view of the choice of γ and Lemma I.7, one can show that 12 γ − L 2 − 40 γω (2 ω +1)(1 − p ) (cid:18)

Original: L 2 σB + (cid:98) L 2 (cid:19) n − 4 γ (1 − p )
Revised: L 2 σB + (cid:98) L 2 (cid:19) n − 4 γ (1 − p )

Original: L 2 σ pnB ≥ 0 , 1 − a 2 ≤ 1 − γµ, and 1 − p 2 ≤ 1 − γµ, thus
Revised: L 2 σ pnB ≥ 0 , 1 − a 2 ≤ 1 − γµ, and 1 − p 2 ≤ 1 − γµ, thus

Original: E (cid:2) f
Revised: E [Equation]

Original: ( x t +1 ) (cid:3) + 2 γ (2 ω + 1)E (cid:104)(cid:13)(cid:13) g t
Revised: [Table] [Equation] [Table] [Equation]

Original: +1 − h t +1 (cid:13)(cid:13) 2 (cid:105) + 8 γω n E (cid:34)n n (cid:88) i =
Revised: [Table]

Original: (cid:13)(cid:13)
Revised: [Equation] [Table] [Equation]

Original: g t +1 i
Revised: [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: In the view of Lemma H.6 with
Revised: In the view of Lemma I.6 with

Original: [Equation]
Revised: [Equation]

Original: and C = 

Revised: and C =

Original: σ 2 nB (cid:48)
Revised: σ 2 nB (cid:48)

Original: , we can conclude the proof.
Revised: , we can conclude the proof.

Original: Corollary H.21.
Revised: Corollary I.21.

Original: Suppose that assumptions from Theorem H.20 hold, probability p = min (cid:110) ζ C d , µnεBσ 2 (cid:111) , batch size B (cid:48)=
Revised: Suppose that assumptions from Theorem I.20 hold, probability p = min (cid:110) ζ C d , µnεBσ 2 (cid:111) , batch size B (cid:48)

Original:  Θ (cid:16) σ 2 µnε (cid:17) , and h 0 i = g 0 i = 0 for all i ∈ [ n ] , then DASHASYNC-MVR needs
Revised: = Θ (cid:16) σ 2 µnε (cid:17) , and h 0 i = g 0 i = 0 for all i ∈ [ n ] , then DASHASYNC-MVR needs

Original: [Equation]
Revised: [Equation]

Original: communication rounds to get an ε -solution, the communication complexity is equal to O ( ζ C T ) , and the number of stochastic gradient calculations per node equals O ( BT ) , where ζ C is the expected density from Deﬁnition 1.3.
Revised: communication rounds to get an ε -solution, the communication complexity is equal to O ( ζ C T ) , and the number of stochastic gradient calculations per node equals O ( BT ) , where ζ C is the expected density from Deﬁnition 1.3.

Original: Proof.
Revised: Proof.

Original: Considering the choice of B (cid:48) , we have 2 σ 2 nµB (cid:48)
Revised: Considering the choice of B (cid:48) , we have 2 σ 2 nµB (cid:48)

Original: =
Revised: =

Original: O ( ε ) .
Revised: O ( ε ) .

Original: Therefore, is it enough to take the number of communication rounds equals (28) to get an ε -solution.
Revised: Therefore, is it enough to take the number of communication rounds equals (28) to get an ε -solution.

Original: It is left to estimate the communication and oracle complexity.
Revised: It is left to estimate the communication and oracle complexity.

Original: On average, in Algorithm 2, at each communication round the number of coordinates that each node sends equals pd + (1 − p ) ζ C ≤ ζ C d d + (cid:16) 1 − ζ C d (cid:17) ζ C ≤ 2 ζ C .
Revised: On average, in Algorithm 2, at each communication round the number of coordinates that each node sends equals pd + (1 − p ) ζ C ≤ ζ C d d + (cid:16) 1 − ζ C d (cid:17) ζ C ≤ 2 ζ C .

Original: Therefore, the communication complexity is equal to O ( ζ C T ) .
Revised: Therefore, the communication complexity is equal to O ( ζ C T ) .

Original: Considering the fact that we use a mini-batch of stochastic gradients, on average, the number of stochastic gradients that each node calculates at each communication round equals pB (cid:48)
Revised: Considering the fact that we use a mini-batch of stochastic gradients, on average, the number of stochastic gradients that each node calculates at each communication round equals pB (cid:48)

Original: +(1 − p )2 B =
Revised: +(1 − p )2 B =

Original: O (cid:16) 
Revised: O (cid:16) µnεBσ 2 · σ 2 µnε (cid:17) + 2 B

Original: [Equation]
Revised: = O

Original: (cid:17) + 2 B = O ( B ) , thus the number of stochastic gradients that each node calculates equals O ( BT ) .
Revised: ( B ) , thus the number of stochastic gradients that each node calculates equals O ( BT ) .

Original: Unlike Corollary 6.10, in this corollary, we can initialize h 

Revised: Unlike Corollary 6.10, in this corollary, we can initialize h

Original: i and g 0 i , for instance, with zeros because the corresponding initialization error Ψ 0 from the proof of Theorem H.20 would be under the logarithm.
Revised: i and g 0 i , for instance, with zeros because the corresponding initialization error Ψ 0 from the proof of Theorem I.20 would be under the logarithm.

Original: I E XTRA E XPERIMENTS
Revised: J E XTRA E XPERIMENTS

Original: DASHA-MVR improves VR-MARINA (online) when ε is small (see Tables 1 and 2 and experiments in Section A).
Revised: DASHA-MVR improves VR-MARINA (online) when ε is small (see Tables 1 and 2 and experiments in Section A).

Original: However, our analysis shows that DASHA-MVR gets a term Bω (cid:113) σ 2 εnB in the oracle complexity and a term ω (cid:113) σ 2 µεnB in the number of communication rounds in general nonconvex and PŁ settings accordingly.
Revised: However, our analysis shows that DASHA-MVR gets a term Bω (cid:113) σ 2 εnB in the oracle complexity and a term ω (cid:113) σ 2 µεnB in the number of communication rounds in general nonconvex and PŁ settings accordingly.

Original: Both terms can be a bottleneck in some regimes; now, we verify this dependence in the PŁ setting.
Revised: Both terms can be a bottleneck in some regimes; now, we verify this dependence in the PŁ setting.

Original: We take a synthetically generated stochastic quadratic optimization problem with one node ( n = 1 ):
Revised: We take a synthetically generated stochastic quadratic optimization problem with one node ( n = 1 ):

Original: [Equation]
Revised: [Equation]

Original: , where A ∈ R d × d , b ∈ R d , A = A (cid:62) (cid:31) 0 , and ξ ∼ Normal (cid:0) 0 , σ 2 (cid:1) .
Revised: where A ∈ R d × d , b ∈ R d , A = A (cid:62) (cid:31) 0 , and ξ ∼ Normal (cid:0) 0 , σ 2 (cid:1) .

Original: We generate A in such way, that µ ≈ 1 . 

Revised: We generate A in such way, that µ ≈ 1 .

Original: ≤
Revised: ≤

Original: L ≈ 2 . 0 , take d = 10 4 , σ 2 = 1 . 0 , Rand K with K =( ω
Revised: L ≈ 2 . 0 , take d = 10 4 , σ 2 = 1 . 0 , Rand K with K =( ω

Original: ≈ d ), batch size B = 1 , and σ 2 µεnB = 10 4 .
Revised: ≈ d ), batch size B = 1 , and σ 2 µεnB = 10 4 .

Original: With this particular choice of parameters, ω (cid:113) σ 2 µεnB would dominate in the number of communication rounds T =ω + ω (cid:113)
Revised: With this particular choice of parameters, ω (cid:113) σ 2 µεnB would dominate in the number of communication rounds T =

Original: [Equation]
Revised: [Equation]

Original: [Figure] [Equation] [Figure]
Revised: (cid:113)

Original: [Equation]
Revised: [Equation]

Revised: Lσ µ 3 / 2 √ εnB .

Original: Results are provided in Figure 5.
Revised: Results are provided in Figure 5.

Original: We consider DASHA-MVR with a momentum b from Corollary H. and b = min (cid:110) 1 ω , µnεBσ 2 (cid:111) .
Revised: We consider DASHA-MVR with a momentum b from Corollary I. and b = min (cid:110) 1 ω , µnεBσ 2 (cid:111) .

Original: With the latter choice of momentum b , DASHA-MVR converges at the same rate as DASHA-SYNC-MVR or VR-MARINA (online) but to an ε -solution with a smaller ε .
Revised: With the latter choice of momentum b , DASHA-MVR converges at the same rate as DASHA-SYNC-MVR or VR-MARINA (online) but to an ε -solution with a smaller ε .

Original: On the other hand, the former choice of momentum b guarantees the convergence to the correct ε -solution, but with a slower rate.
Revised: On the other hand, the former choice of momentum b guarantees the convergence to the correct ε -solution, but with a slower rate.

Original: Overall, the experiment provides the pieces of evidence that our choice of b is correct and that our analysis in Theorem H.15 is tight.
Revised: Overall, the experiment provides the pieces of evidence that our choice of b is correct and that our analysis in Theorem I.15 is tight.

Original: If we decrease ω from 10 4 to 10 3 (see Figure 6), or σ 2 from 1 . 0 to 0 .
Revised: If we decrease ω from 10 4 to 10 3 (see Figure 6), or σ 2 from 1 . 0 to 0 .

Original: 1 (see Figure 7), or µ from 1 .to 0 .
Revised: 1 (see Figure 7), or µ from 1 .to 0 .

Original: 1 (see Figure 8), then the gap between algorithms closes.
Revised: 1 (see Figure 8), then the gap between algorithms closes.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: quadratic optimization task with

Original: K = [Figure] [Figure]
Revised: [Figure] [Figure]