Revised: [Equation]

Revised: [Equation]

Revised: h

Original: [Equation]
Revised: [Equation]

Revised: n

Revised: [Equation]

Revised: ti

Original: [Equation]
Revised: [Equation]

Revised: ti (cid:13)(cid:13)

Revised: [Equation]

Revised: [Equation]

Original: In the view of the choice of γ
Revised: In the view of the choice of γ

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: Finally, using Lemma H.5 with
Revised: Finally, using Lemma H.5 with

Original: [Equation]
Revised: [Equation]

Original: and C =
Revised: and C =

Original: σ 2 nB (cid:48) , we can conclude the proof.
Revised: σ 2 nB (cid:48) , we can conclude the proof.

Original: Corollary 6.10.
Revised: Corollary 6.10.

Original: Suppose that assumptions from Theorem H.19 hold, probability p = min (cid:110) ζ C d , nεBσ 2 (cid:111) , batch size B (cid:48)
Revised: Suppose that assumptions from Theorem H.19 hold, probability p = min (cid:110) ζ C d , nεBσ 2 (cid:111) , batch size B (cid:48)

Original: =
Revised: =

Original: Θ (cid:16) σ 2 nε (cid:17) and h 0 i = g 0 i = 1 B init (cid:80) B init k
Revised: Θ (cid:16) σ 2 nε (cid:17) and h 0 i = g 0 i = 1 B init (cid:80) B init k

Original: =1 ∇ f i ( x 0 ; ξ 0 ik ) for all i ∈ [ n ] , initial batch size B init =
Revised: =1 ∇ f i ( x 0 ; ξ 0 ik ) for all i ∈ [ n ] , initial batch size B init =

Original: Θ (cid:16) max (cid:110) σ 2 nε , B dζ C (cid:111)(cid:17) , then DASHA-SYNC-MVR needs
Revised: Θ (cid:16) max (cid:110) σ 2 nε , B dζ C (cid:111)(cid:17) , then DASHA-SYNC-MVR needs

Original: T :
Revised: T :

Original: = O
Revised: = O

Original: [Equation]
Revised: [Equation]

Original:  communication rounds to get an ε -solution, the communication complexity is equal to O ( d + ζ C T ) , and the number of stochastic gradient calculations per node equals O ( B init + BT ) , where ζ C is the expected density from Deﬁnition 1.3.
Revised:  communication rounds to get an ε -solution, the communication complexity is equal to O ( d + ζ C T ) , and the number of stochastic gradient calculations per node equals O ( B init + BT ) , where ζ C is the expected density from Deﬁnition 1.3.

Original: Proof.
Revised: Proof.

Original: Considering Theorem H.19 and the choice of B (cid:48)
Revised: Considering Theorem H.19 and the choice of B (cid:48)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: T   2 (cid:0) f
Revised: T   2 (cid:0) f

Original: ( x 0 ) − f ∗ (cid:1)   L + (cid:118)(cid:117)(cid:117)(cid:116) 12 ω (2 ω + 1)(1 − p ) (cid:16) L 2 σ B
Revised: ( x 0 ) − f ∗ (cid:1)   L + (cid:118)(cid:117)(cid:117)(cid:116) 12 ω (2 ω + 1)(1 − p ) (cid:16) L 2 σ B

Original: + (cid:98) L 2 (cid:17) n + 2(1 − p )
Revised: + (cid:98) L 2 (cid:17) n + 2(1 − p )

Original: L 2 σ pnB  
Revised: L 2 σ pnB  

Original: +p (cid:13)(cid:13) h 0 − ∇ f ( x 0 ) (cid:13)(cid:13)  + 2 3 ε.
Revised: +p (cid:13)(cid:13) h 0 − ∇ f ( x 0 ) (cid:13)(cid:13)  + 2 3 ε.

Original: Due to p = min (cid:110) ζ C d , nεBσ 2 (cid:111) , we have
Revised: Due to p = min (cid:110) ζ C d , nεBσ 2 (cid:111) , we have

Original: E (cid:104)(cid:13)(cid:13)
Revised: E (cid:104)(cid:13)(cid:13)

Original: [Table]
Revised: [Table]

Original:  
Revised:  

Original: [Table]
Revised: [Table]

Original: (cid:118)(cid:117)(cid:117)(cid:116)
Revised: (cid:118)(cid:117)(cid:117)(cid:116)

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: p
Revised: p

Original: [Table]
Revised: [Table]

Original: (cid:16)
Revised: (cid:16)

Original: [Table]
Revised: [Table]

Original: + (cid:98)
Revised: + (cid:98)

Original: [Table]
Revised: [Table]

Original: (cid:17) n +
Revised: (cid:17) n +

Original: [Table]
Revised: [Table]

Original: d (
Revised: d (

Original: [Table]
Revised: [Table]

Original: p
Revised: p

Original: [Table]
Revised: [Table]

Original: L
Revised: L

Original: [Table]
Revised: [Table]

Original: ζ C nB + 2 σ
Revised: ζ C nB + 2 σ

Original: [Table] [Table]
Revised: [Table] [Table]

Original: p )
Revised: p )

Original: [Table] [Table]
Revised: [Table] [Table]

Original: εn 2 B 
Revised: εn 2 B 

Original: [Table]
Revised: [Table]

Original: nεB (cid:19)
Revised: nεB (cid:19)

Original: [Table]
Revised: [Table]

Original: 
Revised: 

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: d
Revised: d

Original: [Equation]
Revised: [Equation]

Original: nεB
Revised: nεB

Original: [Equation]
Revised: [Equation]

Original:  
Revised:  

Original: [Equation]
Revised: [Equation]

Original: Therefore, we can take
Revised: Therefore, we can take

Original: T =
Revised: T =

Original: O 
Revised: O 

Original: [Equation]
Revised: [Equation]

Original:     .
Revised:     .

Original: Note, that
Revised: Note, that

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: E (cid:104)(cid:13)(cid:13)
Revised: E (cid:104)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: nB init .
Revised: nB init .

Original: Next, by taking B init =
Revised: Next, by taking B init =

Original: max (cid:110) σ 2 nε , B dζ C (cid:111) and using the last ineqaulity, we have
Revised: max (cid:110) σ 2 nε , B dζ C (cid:111) and using the last ineqaulity, we have

Original: T =
Revised: T =

Original: O 
Revised: O 

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: min
Revised: min

Original: [Equation]
Revised: [Equation]

Original: = O 
Revised: = O 

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: nεB   .
Revised: nεB   .

Original: Finally, it is left to estimate the communication and oracle complexity.
Revised: Finally, it is left to estimate the communication and oracle complexity.

Original: On average, the number of coordinates that each node in Algorithm 2 sends at each communication round equals pd + (1 − p ) ζ C ≤ ζ C d d + (cid:16) 1 − ζ C d (cid:17) ζ C ≤ 2 ζ C .
Revised: On average, the number of coordinates that each node in Algorithm 2 sends at each communication round equals pd + (1 − p ) ζ C ≤ ζ C d d + (cid:16) 1 − ζ C d (cid:17) ζ C ≤ 2 ζ C .

Original: Therefore, the communication complexity is equal to
Revised: Therefore, the communication complexity is equal to

Original: O ( d + ζ C T ) .
Revised: O ( d + ζ C T ) .

Original: Considering the fact that we use a mini-batch of stochastic gradients, on average, the number of stochastic gradients that each node calculates at each communication round equals pB (cid:48)
Revised: Considering the fact that we use a mini-batch of stochastic gradients, on average, the number of stochastic gradients that each node calculates at each communication round equals pB (cid:48)

Original: + (1 − p )2 B ≤ O (cid:16) nεBσ 2 · σ 2 nε (cid:17) + 2 B = O ( B ) .
Revised: + (1 − p )2 B ≤ O (cid:16) nεBσ 2 · σ 2 nε (cid:17) + 2 B = O ( B ) .

Original: Considering the initial batch size B init , the number of stochastic gradients that each node calculates equals O ( B init + BT ) .
Revised: Considering the initial batch size B init , the number of stochastic gradients that each node calculates equals O ( B init + BT ) .

Original: Corollary 6.11.
Revised: Corollary 6.11.

Original: Suppose that assumptions of Corollary 6.10 hold, batch size B ≤ σ √ εn , we take
Revised: Suppose that assumptions of Corollary 6.10 hold, batch size B ≤ σ √ εn , we take

Original: Rand K with K = ζ C =
Revised: Rand K with K = ζ C =

Original: Θ (cid:16) Bd √ εn σ (cid:17) , and (cid:101)
Revised: Θ (cid:16) Bd √ εn σ (cid:17) , and (cid:101)

Original: L :=
Revised: L :=

Original: max { L, L σ , (cid:98) L } .
Revised: max { L, L σ , (cid:98) L } .

Original: Then the communication complexity equals
Revised: Then the communication complexity equals

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: and the expected # of stochastic gradient calculations per node equals
Revised: and the expected # of stochastic gradient calculations per node equals

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: In the view of Theorem F.2, we have ω +

Revised: In the view of Theorem F.2, we have ω +

Original: =
Revised: =

Original: d/K. Moreover, K = Θ (cid:16)
Revised: d/K. Moreover, K = Θ (cid:16)

Original: Bd √ εn σ (cid:17) =
Revised: Bd √ εn σ (cid:17) =

Original: O (cid:16)
Revised: O (cid:16)

Original: [Equation]
Revised: [Equation]

Original: n (cid:17) , thus the communication complexity equals
Revised: n (cid:17) , thus the communication complexity equals

Original: O ( d + ζ C T )
Revised: O ( d + ζ C T )

Original: =
Revised: =

Original: O
Revised: O

Original: [Equation]
Revised: [Equation]

Original: And the expected number of stochastic gradient calculations per node equals
Revised: And the expected number of stochastic gradient calculations per node equals

Original: O ( B init + BT )
Revised: O ( B init + BT )

Original: =
Revised: =

Original: O 
Revised: O 

Original: [Equation]
Revised: [Equation]

Original: = O  
Revised: = O  

Original: [Equation]
Revised: [Equation]

Original: nε
Revised: nε

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: H. C ASE OF DASHA-SYNC-MVR UNDER PŁ- CONDITION
Revised: H. C ASE OF DASHA-SYNC-MVR UNDER PŁ- CONDITION

Original: Theorem
Revised: Theorem

Original: H.20.
Revised: H.20.

Original: Suppose that
Revised: Suppose that

Original: Assumption 5.1, 5.2, 1.2, 5.5, 5.
Revised: Assumption 5.1, 5.2, 1.2, 5.5, 5.

Original: and
Revised: and

Original: G. hold.
Revised: G. hold.

Original: Let us take a = 1 / (
Revised: Let us take a = 1 / (

Original: [Equation]
Revised: [Equation]

Original: ω
Revised: ω

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , probability p ∈ (
Revised: , probability p ∈ (

Original: [Equation]
Revised: [Equation]

Original: 1] and γ ≤ min
Revised: 1] and γ ≤ min

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original:   in Algorithm 1, then
Revised:   in Algorithm 1, then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let us ﬁx constants κ, η, ν ∈ [0 , ∞ ) that we will deﬁne later.
Revised: Let us ﬁx constants κ, η, ν ∈ [0 , ∞ ) that we will deﬁne later.

Original: Using Lemma H.1, we can get (20).
Revised: Using Lemma H.1, we can get (20).

Original: Considering (20), Lemma H.17, Lemma H.18, and the law of total expectation, we obtain
Revised: Considering (20), Lemma H.17, Lemma H.18, and the law of total expectation, we obtain

Original: E (cid:2) f
Revised: E (cid:2) f

Original: ( x t +1 ) (cid:3) + κ E (cid:104)(cid:13)(cid:13) g t
Revised: ( x t +1 ) (cid:3) + κ E (cid:104)(cid:13)(cid:13) g t

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: h t +1 (cid:13)(cid:13) 2 (cid:105) + η E (cid:34)
Revised: h t +1 (cid:13)(cid:13) 2 (cid:105) + η E (cid:34)

Original: [Equation]
Revised: [Equation]

Original: n n (cid:88) i =
Revised: n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g
Revised: g

Original: [Equation]
Revised: [Equation]

Original: +

Revised: +

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: +1 i (cid:13)(cid:13)
Revised: +1 i (cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: + ν E (cid:104)(cid:13)(cid:13) h t +
Revised: + ν E (cid:104)(cid:13)(cid:13) h t +

Original: [Equation]
Revised: [Equation]

Original: ∇ f
Revised: ∇ f

Original: [Equation]
Revised: [Equation]

Original: x t +
Revised: x t +

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: (cid:105) ≤ E (cid:20) f
Revised: (cid:105) ≤ E (cid:20) f

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: γ (cid:13)(cid:13)
Revised: γ (cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: γ (cid:13)(cid:13) h
Revised: γ (cid:13)(cid:13) h

Original: [Equation]
Revised: [Equation]

Original: ( x t ) (cid:13)(cid:13)
Revised: ( x t ) (cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: (cid:21) + κ E   2 ω (1 − p ) (cid:16)
Revised: (cid:21) + κ E   2 ω (1 − p ) (cid:16)

Original: [Equation]
Revised: [Equation]

Original: + (cid:98)
Revised: + (cid:98)

Original: [Equation]
Revised: [Equation]

Original: (cid:17) n (cid:13)(cid:13) x
Revised: (cid:17) n (cid:13)(cid:13) x

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: − x
Revised: − x

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (1 − p ) n
Revised: (1 − p ) n

Original: [Equation]
Revised: [Equation]

Original: n (cid:88) i =
Revised: n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g ti − h ti (cid:13)(cid:13)
Revised: g ti − h ti (cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: (1 − p ) (1 − a ) 2 (cid:13)(cid:13)
Revised: (1 − p ) (1 − a ) 2 (cid:13)(cid:13)

Original: g t − h t (cid:13)(cid:13)  + η E (cid:34) 2 ω (1 − p ) (cid:18)
Revised: g t − h t (cid:13)(cid:13)  + η E (cid:34) 2 ω (1 − p ) (cid:18)

Original: L 2 σ
Revised: L 2 σ

Original: B + (cid:98) L(cid:19) (cid:13) (cid:13) x
Revised: B + (cid:98) L(cid:19) (cid:13) (cid:13) x

Original: t +1 − x t (cid:13) (cid:13) 2 + (1 − p ) (cid:16) 2 a 2 ω + (1 − a ) 2 (cid:17) 1 n n (cid:88) i = (cid:13) (cid:13) g ti − h ti (cid:13) (cid:13)(cid:35) + ν E (cid:20) pσnB (cid:48)
Revised: t +1 − x t (cid:13) (cid:13) 2 + (1 − p ) (cid:16) 2 a 2 ω + (1 − a ) 2 (cid:17) 1 n n (cid:88) i = (cid:13) (cid:13) g ti − h ti (cid:13) (cid:13)(cid:35) + ν E (cid:20) pσnB (cid:48)

Original: + (1 − p )
Revised: + (1 − p )

Original: L 2 σ nB (cid:13)(cid:13)
Revised: L 2 σ nB (cid:13)(cid:13)

Original: x
Revised: x

Original: t +1 − x t (cid:13)(cid:13) 

Revised: t +1 − x t (cid:13)(cid:13)

Original: + (1 − p ) (cid:13)(cid:13) h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:21)
Revised: + (1 − p ) (cid:13)(cid:13) h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:21)

Original: After rearranging the terms, we get
Revised: After rearranging the terms, we get

Original: E (cid:2) f
Revised: E (cid:2) f

Original: ( x t +1 ) (cid:3) + κ E (cid:104)(cid:13)(cid:13)
Revised: ( x t +1 ) (cid:3) + κ E (cid:104)(cid:13)(cid:13)

Original: g t +1 − h t +1 (cid:13)(cid:13) 2 (cid:105) + η E (cid:34)n n (cid:88) i =
Revised: g t +1 − h t +1 (cid:13)(cid:13) 2 (cid:105) + η E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g t +

Revised: g t +

Original: i − h t
Revised: i − h t

Original: +1 i (cid:13)(cid:13) 2 (cid:35) + ν E (cid:104)(cid:13)(cid:13)
Revised: +1 i (cid:13)(cid:13) 2 (cid:35) + ν E (cid:104)(cid:13)(cid:13)

Original: h t +1 − ∇ f ( x t +1 ) (cid:13)(cid:13) 2 (cid:105) ≤ E (cid:2) f
Revised: h t +1 − ∇ f ( x t +1 ) (cid:13)(cid:13) 2 (cid:105) ≤ E (cid:2) f

Original: ( x t ) (cid:3) − γ 2 E (cid:104)(cid:13) (cid:13) ∇ f ( x t ) (cid:13) (cid:13) 2 (cid:105) −   

Revised: ( x t ) (cid:3) − γ 2 E (cid:104)(cid:13) (cid:13) ∇ f ( x t ) (cid:13) (cid:13) 2 (cid:105) −  

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: nB
Revised: nB

Original: [Equation]
Revised: [Equation]

Original: Let us take ν = 2 γp , κ = 2 γa , a = 1 2 ω +1 , and η = 8 γωn .
Revised: Let us take ν = 2 γp , κ = 2 γa , a = 1 2 ω +1 , and η = 8 γωn .

Original: Thus γ + κ (1 − p )(1 − a ) 2 ≤ (cid:0) 1 − a 2 (cid:1) κ, γ + ν (1 − p ) =
Revised: Thus γ + κ (1 − p )(1 − a ) 2 ≤ (cid:0) 1 − a 2 (cid:1) κ, γ + ν (1 − p ) =

Original: (cid:0) 1 − p 2 (cid:1) ν, 2 κa 2 ω (1 − p ) n + η (1 − p ) (cid:16) 2 a 2 ω
Revised: (cid:0) 1 − p 2 (cid:1) ν, 2 κa 2 ω (1 − p ) n + η (1 − p ) (cid:16) 2 a 2 ω

Original: + (1 − a ) 2 (cid:17) ≤ (cid:0) 1 − a 2 (cid:1) η, and
Revised: + (1 − a ) 2 (cid:17) ≤ (cid:0) 1 − a 2 (cid:1) η, and

Original: E (cid:2) f
Revised: E (cid:2) f

Original: ( x t +1 ) (cid:3) + 2 γ (2 ω + 1)E (cid:104)(cid:13)(cid:13) g t
Revised: ( x t +1 ) (cid:3) + 2 γ (2 ω + 1)E (cid:104)(cid:13)(cid:13) g t

Original: +1 − h t +1 (cid:13)(cid:13) 2 (cid:105) + 8 γω n E (cid:34)n n (cid:88) i =
Revised: +1 − h t +1 (cid:13)(cid:13) 2 (cid:105) + 8 γω n E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g t +

Revised: g t +

Original: i − h t
Revised: i − h t

Original: +1 i (cid:13)(cid:13) 2 (cid:35) + 2 γ p E (cid:104)(cid:13)(cid:13)
Revised: +1 i (cid:13)(cid:13) 2 (cid:35) + 2 γ p E (cid:104)(cid:13)(cid:13)

Original: h t +1 − ∇ f ( x t +1 ) (cid:13)(cid:13)
Revised: h t +1 − ∇ f ( x t +1 ) (cid:13)(cid:13)

Original: [Table]
Revised: [Table]

Original: p ) (cid:16) L 2 σ B
Revised: p ) (cid:16) L 2 σ B

Original: + (cid:98) L 2 (cid:17) n − 2 γ (1 − p ) L 2 σ pnB 
Revised: + (cid:98) L 2 (cid:17) n − 2 γ (1 − p ) L 2 σ pnB 

Original:  E (cid:104)(cid:13)(cid:13) x t +1 − x t (cid:13)(cid:13) 2 (cid:105) + (cid:16) 1 − a 2 (cid:17) 2 γ (2 ω + 1)E
Revised:  E (cid:104)(cid:13)(cid:13) x t +1 − x t (cid:13)(cid:13) 2 (cid:105) + (cid:16) 1 − a 2 (cid:17) 2 γ (2 ω + 1)E

Original: (cid:104)(cid:13)(cid:13) g t − h t (cid:13)(cid:13) 2 (cid:105) + (cid:16) 1 − a 2 (cid:17) 8 γω n E (cid:34)n n (cid:88) i =
Revised: (cid:104)(cid:13)(cid:13) g t − h t (cid:13)(cid:13) 2 (cid:105) + (cid:16) 1 − a 2 (cid:17) 8 γω n E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g ti − h ti (cid:13)(cid:13)(cid:35) + (cid:16) 1 − p 2 (cid:17) 2 γ p E
Revised: g ti − h ti (cid:13)(cid:13)(cid:35) + (cid:16) 1 − p 2 (cid:17) 2 γ p E

Original: (cid:104)(cid:13)(cid:13) h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105) + 2 γσnB (cid:48) .
Revised: (cid:104)(cid:13)(cid:13) h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105) + 2 γσnB (cid:48) .

Original: In the view of the choice of γ and Lemma H.7, one can show that 12 γ − L 2 − 40 γω (2 ω +1)(1 − p ) (cid:18)
Revised: In the view of the choice of γ and Lemma H.7, one can show that 12 γ − L 2 − 40 γω (2 ω +1)(1 − p ) (cid:18)

Original: L 2 σB + (cid:98) L 2 (cid:19) n − 4 γ (1 − p )
Revised: L 2 σB + (cid:98) L 2 (cid:19) n − 4 γ (1 − p )

Original: L 2 σ pnB ≥ 0 , 1 − a 2 ≤ 1 − γµ, and 1 − p 2 ≤ 1 − γµ, thus
Revised: L 2 σ pnB ≥ 0 , 1 − a 2 ≤ 1 − γµ, and 1 − p 2 ≤ 1 − γµ, thus

Original: E (cid:2) f
Revised: E (cid:2) f

Original: ( x t +1 ) (cid:3) + 2 γ (2 ω + 1)E (cid:104)(cid:13)(cid:13) g t
Revised: ( x t +1 ) (cid:3) + 2 γ (2 ω + 1)E (cid:104)(cid:13)(cid:13) g t

Original: +1 − h t +1 (cid:13)(cid:13) 2 (cid:105) + 8 γω n E (cid:34)n n (cid:88) i =
Revised: +1 − h t +1 (cid:13)(cid:13) 2 (cid:105) + 8 γω n E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g t +1 i
Revised: g t +1 i

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: In the view of Lemma H.6 with
Revised: In the view of Lemma H.6 with

Original: [Equation]
Revised: [Equation]

Original: and C = 

Revised: and C =

Original: σ 2 nB (cid:48)
Revised: σ 2 nB (cid:48)

Original: , we can conclude the proof.
Revised: , we can conclude the proof.

Original: Corollary H.21.
Revised: Corollary H.21.

Original: Suppose that assumptions from Theorem H.20 hold, probability p =
Revised: Suppose that assumptions from Theorem H.20 hold, probability p =

Original: min (cid:110) ζ C d , µnεBσ 2 (cid:111) , batch size B (cid:48) =
Revised: min (cid:110) ζ C d , µnεBσ 2 (cid:111) , batch size B (cid:48) =

Original: Θ (cid:16) σ 2 µnε (cid:17) , and h 0 i = g 0 i = 0 for all i ∈ [ n ] , then DASHASYNC-MVR needs
Revised: Θ (cid:16) σ 2 µnε (cid:17) , and h 0 i = g 0 i = 0 for all i ∈ [ n ] , then DASHASYNC-MVR needs

Original: [Equation]
Revised: [Equation]

Original: communication rounds to get an ε -solution, the communication complexity is equal to O ( ζ C T ) , and the number of stochastic gradient calculations per node equals O ( BT ) , where ζ C is the expected density from Deﬁnition 1.3.
Revised: communication rounds to get an ε -solution, the communication complexity is equal to O ( ζ C T ) , and the number of stochastic gradient calculations per node equals O ( BT ) , where ζ C is the expected density from Deﬁnition 1.3.

Original: Proof.
Revised: Proof.

Original: Considering the choice of B (cid:48) , we have 2 σ 2 nµB (cid:48)
Revised: Considering the choice of B (cid:48) , we have 2 σ 2 nµB (cid:48)

Original: =
Revised: =

Original: O ( ε ) .
Revised: O ( ε ) .

Original: Therefore, is it enough to take the number of communication rounds equals (28) to get an ε -solution.
Revised: Therefore, is it enough to take the number of communication rounds equals (28) to get an ε -solution.

Original: It is left to estimate the communication and oracle complexity.
Revised: It is left to estimate the communication and oracle complexity.

Original: On average, in Algorithm 2, at each communication round the number of coordinates that each node sends equals pd + (1 − p ) ζ C ≤ ζ C d d + (cid:16) 1 − ζ C d (cid:17) ζ C ≤ 2 ζ C .
Revised: On average, in Algorithm 2, at each communication round the number of coordinates that each node sends equals pd + (1 − p ) ζ C ≤ ζ C d d + (cid:16) 1 − ζ C d (cid:17) ζ C ≤ 2 ζ C .

Original: Therefore, the communication complexity is equal to O ( ζ C T ) .
Revised: Therefore, the communication complexity is equal to O ( ζ C T ) .

Original: Considering the fact that we use a mini-batch of stochastic gradients, on average, the number of stochastic gradients that each node calculates at each communication round equals pB (cid:48)
Revised: Considering the fact that we use a mini-batch of stochastic gradients, on average, the number of stochastic gradients that each node calculates at each communication round equals pB (cid:48)

Original: +(1 − p )2 B =
Revised: +(1 − p )2 B =

Original: O (cid:16)
Revised: O (cid:16)

Original: [Equation]
Revised: [Equation]

Original: (cid:17) + 2 B = O ( B ) , thus the number of stochastic gradients that each node calculates equals O ( BT ) .
Revised: (cid:17) + 2 B = O ( B ) , thus the number of stochastic gradients that each node calculates equals O ( BT ) .

Original: Unlike Corollary 6.10, in this corollary, we can initialize h 

Revised: Unlike Corollary 6.10, in this corollary, we can initialize h

Original: i and g 0 i , for instance, with zeros because the corresponding initialization error Ψ 0 from the proof of Theorem H.20 would be under the logarithm.
Revised: i and g 0 i , for instance, with zeros because the corresponding initialization error Ψ 0 from the proof of Theorem H.20 would be under the logarithm.

Original: I E XTRA E XPERIMENTS
Revised: I E XTRA E XPERIMENTS

Original: DASHA-MVR improves VR-MARINA (online) when ε is small (see Tables 1 and 2 and experiments in Section A).
Revised: DASHA-MVR improves VR-MARINA (online) when ε is small (see Tables 1 and 2 and experiments in Section A).

Original: However, our analysis shows that DASHA-MVR gets a term Bω (cid:113) σ 2 εnB in the oracle complexity and a term ω (cid:113) σ 2 µεnB in the number of communication rounds in general nonconvex and PŁ settings accordingly.
Revised: However, our analysis shows that DASHA-MVR gets a term Bω (cid:113) σ 2 εnB in the oracle complexity and a term ω (cid:113) σ 2 µεnB in the number of communication rounds in general nonconvex and PŁ settings accordingly.

Original: Both terms can be a bottleneck in some regimes; now, we verify this dependence in the PŁ setting.
Revised: Both terms can be a bottleneck in some regimes; now, we verify this dependence in the PŁ setting.

Original: We take a synthetically generated stochastic quadratic optimization problem with one node ( n = 1 ):
Revised: We take a synthetically generated stochastic quadratic optimization problem with one node ( n = 1 ):

Original: [Equation]
Revised: [Equation]

Original: , where A ∈ R d × d , b ∈ R d , A
Revised: , where A ∈ R d × d , b ∈ R d , A

Original: = A (cid:62) (cid:31) 0 , and ξ ∼ Normal (cid:0) 0 , σ 2 (cid:1) .
Revised: = A (cid:62) (cid:31) 0 , and ξ ∼ Normal (cid:0) 0 , σ 2 (cid:1) .

Original: We generate A in such way, that µ ≈ 1 . 

Revised: We generate A in such way, that µ ≈ 1 .

Original: ≤
Revised: ≤

Original: L ≈ 2 . 0 , take d = 10 4 , σ 2 = 1 . 0 , Rand K with K =( ω
Revised: L ≈ 2 . 0 , take d = 10 4 , σ 2 = 1 . 0 , Rand K with K =( ω

Original: ≈ d ), batch size B = 1 , and σ 2 µεnB = 10 4 .
Revised: ≈ d ), batch size B = 1 , and σ 2 µεnB = 10 4 .

Original: With this particular choice of parameters, ω (cid:113) σ 2 µεnB would dominate in the number of communication rounds T
Revised: With this particular choice of parameters, ω (cid:113) σ 2 µεnB would dominate in the number of communication rounds T

Original: = ω + ω (cid:113)
Revised: = ω + ω (cid:113)

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: Results are provided in Figure 5.
Revised: Results are provided in Figure 5.

Original: We consider DASHA-MVR with a momentum b from Corollary H. and b = min (cid:110) 1 ω , µnεBσ 2 (cid:111) .
Revised: We consider DASHA-MVR with a momentum b from Corollary H. and b = min (cid:110) 1 ω , µnεBσ 2 (cid:111) .

Original: With the latter choice of momentum b , DASHA-MVR converges at the same rate as DASHA-SYNC-MVR or VR-MARINA (online) but to an ε -solution with a smaller ε .
Revised: With the latter choice of momentum b , DASHA-MVR converges at the same rate as DASHA-SYNC-MVR or VR-MARINA (online) but to an ε -solution with a smaller ε .

Original: On the other hand, the former choice of momentum b guarantees the convergence to the correct ε -solution, but with a slower rate.
Revised: On the other hand, the former choice of momentum b guarantees the convergence to the correct ε -solution, but with a slower rate.

Original: Overall, the experiment provides the pieces of evidence that our choice of b is correct and that our analysis in Theorem H.15 is tight.
Revised: Overall, the experiment provides the pieces of evidence that our choice of b is correct and that our analysis in Theorem H.15 is tight.

Original: If we decrease ω from 10 4 to 10 3 (see Figure 6), or σ 2 from 1 . 0 to 0 .
Revised: If we decrease ω from 10 4 to 10 3 (see Figure 6), or σ 2 from 1 . 0 to 0 .

Original: 1 (see Figure 7), or µ from 1 .to 0 .
Revised: 1 (see Figure 7), or µ from 1 .to 0 .

Original: 1 (see Figure 8), then the gap between algorithms closes.
Revised: 1 (see Figure 8), then the gap between algorithms closes.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: quadratic optimization task with
Revised: quadratic optimization task with

Original: K =
Revised: K =

Original: [Figure] [Figure]
Revised: [Figure] [Figure]