Original: DASHA: D ISTRIBUTED N ONCONVEX O PTIMIZATION WITH C OMMUNICATION C OMPRESSION AND O PTIMAL O RACLE C OMPLEXITY
Revised: DASHA: D ISTRIBUTED N ONCONVEX O PTIMIZATION WITH C OMMUNICATION C OMPRESSION AND O PTIMAL O RACLE C OMPLEXITY

Original: A BSTRACT
Revised: A BSTRACT

Original: We develop and analyze DASHA : a new family of methods for nonconvex distributed optimization problems.
Revised: We develop and analyze DASHA : a new family of methods for nonconvex distributed optimization problems.

Original: When the local functions at the nodes have a ﬁnite-sum or an expectation form, our new methods, DASHA-PAGE , DASHA-MVR and DASHA-SYNC-MVR , improve the theoretical oracle and communication complexity of the previous state-of-the-art method MARINA by Gorbunov et al.
Revised: When the local functions at the nodes have a ﬁnite-sum or an expectation form, our new methods, DASHA-PAGE , DASHA-MVR and DASHA-SYNC-MVR , improve the theoretical oracle and communication complexity of the previous state-of-the-art method MARINA by Gorbunov et al.

Original: In particular, to achieve an ε -stationary point and considering the random sparsiﬁer
Revised: In particular, to achieve an ε -stationary point and considering the random sparsiﬁer

Original: Rand K as an example our methods compute the optimal number of gradients
Revised: Rand K as an example our methods compute the optimal number of gradients

Original: O ( √ m / ε √ n ) and O ( σ / ε 3 / 2 n ) in ﬁnite-sum and expectation form cases, respectively, while maintaining the SOTA communication complexity O
Revised: O ( √ m / ε √ n ) and O ( σ / ε 3 / 2 n ) in ﬁnite-sum and expectation form cases, respectively, while maintaining the SOTA communication complexity O

Original: ( d / ε √ n ) .
Revised: ( d / ε √ n ) .

Original: Furthermore, unlike MARINA the new methods DASHA , DASHA-PAGE and DASHA-MVR send compressed vectors only which makes them more practical for federated learning.
Revised: Furthermore, unlike MARINA the new methods DASHA , DASHA-PAGE and DASHA-MVR send compressed vectors only which makes them more practical for federated learning.

Original: We extend our results to the case when the functions satisfy the Polyak-Łojasiewicz condition.
Revised: We extend our results to the case when the functions satisfy the Polyak-Łojasiewicz condition.

Original: Finally, our theory is corroborated in practice: we see a signiﬁcant improvement in experiments with nonconvex classiﬁcation and training of deep learning models.
Revised: Finally, our theory is corroborated in practice: we see a signiﬁcant improvement in experiments with nonconvex classiﬁcation and training of deep learning models.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Nonconvex optimization problems are widespread in modern machine learning tasks, especially with the rise of the popularity of deep neural networks (Goodfellow et al., 2016).
Revised: Nonconvex optimization problems are widespread in modern machine learning tasks, especially with the rise of the popularity of deep neural networks (Goodfellow et al., 2016).

Original: In the past years, the dimensionality of such problems has increased because this leads to better quality (Brown et al., 2020) and robustness (Bubeck & Sellke, 2021) of the deep neural networks trained this way.
Revised: In the past years, the dimensionality of such problems has increased because this leads to better quality (Brown et al., 2020) and robustness (Bubeck & Sellke, 2021) of the deep neural networks trained this way.

Original: Such huge-dimensional nonconvex problems need special treatment and efﬁcient optimization methods (Danilova et al., 2020).
Revised: Such huge-dimensional nonconvex problems need special treatment and efﬁcient optimization methods (Danilova et al., 2020).

Original: Because of their high dimensionality, training such models is a computationally intensive undertaking that requires massive training datasets (Hestness et al., 2017), and parallelization among several compute nodes 1 (Ramesh et al., 2021).
Revised: Because of their high dimensionality, training such models is a computationally intensive undertaking that requires massive training datasets (Hestness et al., 2017), and parallelization among several compute nodes 1 (Ramesh et al., 2021).

Original: Also, the distributed learning paradigm is a necessity in federated learning (Koneˇcný et al., 2016), where, among other things, there is an explicit desire to secure the private data of each client.
Revised: Also, the distributed learning paradigm is a necessity in federated learning (Koneˇcný et al., 2016), where, among other things, there is an explicit desire to secure the private data of each client.

Original: Unlike in the case of classical optimization problems, where the performance of algorithms is deﬁned by their computational complexity (Nesterov, 2018), distributed optimization algorithms are typically measured in terms of the communication overhead between the nodes since such communication is often the bottleneck in practice (Koneˇcný et al., 2016; Wang et al., 2021).
Revised: Unlike in the case of classical optimization problems, where the performance of algorithms is deﬁned by their computational complexity (Nesterov, 2018), distributed optimization algorithms are typically measured in terms of the communication overhead between the nodes since such communication is often the bottleneck in practice (Koneˇcný et al., 2016; Wang et al., 2021).

Original: Many approaches tackle the problem, including managing communication delays (Vogels et al., 2021), ﬁghting with stragglers (Li et al., 2020a), and optimization over time-varying directed graphs (Nedi´c & Olshevsky, 2014).
Revised: Many approaches tackle the problem, including managing communication delays (Vogels et al., 2021), ﬁghting with stragglers (Li et al., 2020a), and optimization over time-varying directed graphs (Nedi´c & Olshevsky, 2014).

Original: Another popular way to alleviate the communication bottleneck is to use lossy compression of communicated messages (Alistarh et al., 2017; Mishchenko et al., 2019; Gorbunov et al., 2021;
Revised: Another popular way to alleviate the communication bottleneck is to use lossy compression of communicated messages (Alistarh et al., 2017; Mishchenko et al., 2019; Gorbunov et al., 2021;

Original: Szlendak et al., 2021).
Revised: Szlendak et al., 2021).

Original: In this paper, we focus on this last approach.
Revised: In this paper, we focus on this last approach.

Original: 1. P ROBLEM FORMULATION
Revised: 1. P ROBLEM FORMULATION

Original: In this work, we consider the optimization problem
Revised: In this work, we consider the optimization problem

Original: [Equation]
Revised: [Equation]

Original: where f i : R d → R is a smooth nonconvex function for all i ∈ [ n ] := { 1 , . . . , n } .
Revised: where f i : R d → R is a smooth nonconvex function for all i ∈ [ n ] := { 1 , . . . , n } .

Original: Moreover, we assume that the problem is solved by n compute nodes, with the i th node having access to function f i only, via an oracle .
Revised: Moreover, we assume that the problem is solved by n compute nodes, with the i th node having access to function f i only, via an oracle .

Original: Communication is facilitated by an orchestrating server able to communicate with all nodes.
Revised: Communication is facilitated by an orchestrating server able to communicate with all nodes.

Original: Our goal is to ﬁnd an ε -solution ( ε -stationary point) of (1): a (possibly random) point (cid:98) x ∈ R d , such that E (cid:104)
Revised: Our goal is to ﬁnd an ε -solution ( ε -stationary point) of (1): a (possibly random) point (cid:98) x ∈ R d , such that E (cid:104)

Original: [Equation]
Revised: [Equation]

Original: 1. G RADIENT ORACLES
Revised: 1. G RADIENT ORACLES

Original: We consider all of the following structural assumptions about the functions { f i } ni =1 , each with its own natural gradient oracle:
Revised: We consider all of the following structural assumptions about the functions { f i } ni =1 , each with its own natural gradient oracle:

Original: Gradient Setting.
Revised: Gradient Setting.

Original: The i th node has access to the gradient ∇ f i : R d → R d of function f i .
Revised: The i th node has access to the gradient ∇ f i : R d → R d of function f i .

Original: Finite-Sum Setting.
Revised: Finite-Sum Setting.

Original: The functions { f i } ni =1 have the ﬁnite-sum form
Revised: The functions { f i } ni =1 have the ﬁnite-sum form

Original: [Equation]
Revised: [Equation]

Original: where f ij : R d → R is a smooth nonconvex function for all j ∈ [ m ] .
Revised: where f ij : R d → R is a smooth nonconvex function for all j ∈ [ m ] .

Original: For all i ∈ [ n ] , the i th node has access to a mini-batch of B gradients , 1 B (cid:80) j ∈
Revised: For all i ∈ [ n ] , the i th node has access to a mini-batch of B gradients , 1 B (cid:80) j ∈

Original: I i ∇ f ij ( · ) , where I i is a multi-set of i.i.d.
Revised: I i ∇ f ij ( · ) , where I i is a multi-set of i.i.d.

Original: samples of the set [ m ] , and | I i | =
Revised: samples of the set [ m ] , and | I i | =

Original: B .
Revised: B .

Original: Stochastic Setting.
Revised: Stochastic Setting.

Original: The function f i is an expectation of a stochastic function,
Revised: The function f i is an expectation of a stochastic function,

Original: [Equation]
Revised: [Equation]

Original: x ; ξ )] , ∀ i ∈ [ n ] , (3) where f i : R d × Ω ξ
Revised: x ; ξ )] , ∀ i ∈ [ n ] , (3) where f i : R d × Ω ξ

Original: → R .
Revised: → R .

Original: For a ﬁxed x ∈ R , f i ( x ; ξ ) is a random variable over some distribution
Revised: For a ﬁxed x ∈ R , f i ( x ; ξ ) is a random variable over some distribution

Original: D i , and, for a ﬁxed ξ ∈ Ω ξ , f i ( x ; ξ ) is a smooth nonconvex function.
Revised: D i , and, for a ﬁxed ξ ∈ Ω ξ , f i ( x ; ξ ) is a smooth nonconvex function.

Original: The i th node has access to a mini-batch of B stochastic gradients 1 B (cid:80) Bj =1 ∇ f i ( · ; ξ ij ) of the function f i through the distribution
Revised: The i th node has access to a mini-batch of B stochastic gradients 1 B (cid:80) Bj =1 ∇ f i ( · ; ξ ij ) of the function f i through the distribution

Original: D i , where { ξ ij } Bj =1 is a collection of i.i.d.
Revised: D i , where { ξ ij } Bj =1 is a collection of i.i.d.

Original: samples from D i .
Revised: samples from D i .

Original: 1. O RACLE COMPLEXITY
Revised: 1. O RACLE COMPLEXITY

Original: In this paper, the oracle complexity of a method is the number of (stochastic) gradient calculations per node to achieve an ε -solution.
Revised: In this paper, the oracle complexity of a method is the number of (stochastic) gradient calculations per node to achieve an ε -solution.

Original: Every considered method performs some number T of communications rounds to get an ε -solution; thus, if every node (on average) calculates B gradients in each communication round, then the oracle complexity equals O ( B init + BT ) , where B init is the number of gradient calculations in the initialization phase of a method.
Revised: Every considered method performs some number T of communications rounds to get an ε -solution; thus, if every node (on average) calculates B gradients in each communication round, then the oracle complexity equals O ( B init + BT ) , where B init is the number of gradient calculations in the initialization phase of a method.

Original: 1. U NBIASED COMPRESSORS
Revised: 1. U NBIASED COMPRESSORS

Original: The method proposed in this paper is based on unbiased compressors – a family of stochastic mappings with special properties that we deﬁne now.
Revised: The method proposed in this paper is based on unbiased compressors – a family of stochastic mappings with special properties that we deﬁne now.

Original: Deﬁnition 1.1.
Revised: Deﬁnition 1.1.

Original: A stochastic mapping C : R d → R d is an unbiased compressor if there exists ω ∈ R such that
Revised: A stochastic mapping C : R d → R d is an unbiased compressor if there exists ω ∈ R such that

Original: [Equation]
Revised: [Equation]

Original: We denote this class of unbiased compressors as U ( ω ) .
Revised: We denote this class of unbiased compressors as U ( ω ) .

Original: One can ﬁnd more information about unbiased compressors in (Beznosikov et al., 2020; Horváth et al., 2019).
Revised: One can ﬁnd more information about unbiased compressors in (Beznosikov et al., 2020; Horváth et al., 2019).

Original: The purpose of such compressors is to quantize or sparsify the communicated vectors in order to increase the communication speed between the nodes and the server.
Revised: The purpose of such compressors is to quantize or sparsify the communicated vectors in order to increase the communication speed between the nodes and the server.

Original: Our methods will work collection of stochastic mappings {C i } ni =1 satisfying the following assumption.
Revised: Our methods will work collection of stochastic mappings {C i } ni =1 satisfying the following assumption.

Original: Assumption 1.2.
Revised: Assumption 1.2.

Original: C i ∈ U ( ω ) for all i ∈ [ n ] , and the compressors are independent .
Revised: C i ∈ U ( ω ) for all i ∈ [ n ] , and the compressors are independent .

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: 1. C OMMUNICATION COMPLEXITY
Revised: 1. C OMMUNICATION COMPLEXITY

Original: The quantity below characterizes the number of nonzero coordinates that a compressor C returns.
Revised: The quantity below characterizes the number of nonzero coordinates that a compressor C returns.

Original: This notion is useful in case of sparsiﬁcation compressors.
Revised: This notion is useful in case of sparsiﬁcation compressors.

Original: Deﬁnition 1.3.
Revised: Deﬁnition 1.3.

Original: The expected density of the compressor C
Revised: The expected density of the compressor C

Original: i is ζ C i :=
Revised: i is ζ C i :=

Original: sup x ∈ R d E [ (cid:107)C i ( x ) (cid:107) 0 ] , where (cid:107) x (cid:107) 0 is the number of nonzero components of x ∈ R d .
Revised: sup x ∈ R d E [ (cid:107)C i ( x ) (cid:107) 0 ] , where (cid:107) x (cid:107) 0 is the number of nonzero components of x ∈ R d .

Original: Let ζ C = max i ∈
Revised: Let ζ C = max i ∈

Original: [ n ] ζ C i .
Revised: [ n ] ζ C i .

Original: In this paper, the communication complexity of a method is the number of coordinates sent to the server per node to achieve an ε -solution.
Revised: In this paper, the communication complexity of a method is the number of coordinates sent to the server per node to achieve an ε -solution.

Original: If every node (on average) sends ζ coordinates in each communication round, then the communication complexity equals O ( ζ init + ζT ) , where T is the number of communication rounds, and ζ init is the number of coordinates sent in the initialization phase.
Revised: If every node (on average) sends ζ coordinates in each communication round, then the communication complexity equals O ( ζ init + ζT ) , where T is the number of communication rounds, and ζ init is the number of coordinates sent in the initialization phase.

Original: We would like to notice that the established communication complexities are compared to previous upper bounds from (Gorbunov et al., 2021;
Revised: We would like to notice that the established communication complexities are compared to previous upper bounds from (Gorbunov et al., 2021;

Original: Szlendak et al., 2021; Mishchenko et al., 2019;
Revised: Szlendak et al., 2021; Mishchenko et al., 2019;

Original: Alistarh et al., 2017), and in this line of work, the comparisons of the communication complexities are made with respect to the number of send coordinates .
Revised: Alistarh et al., 2017), and in this line of work, the comparisons of the communication complexities are made with respect to the number of send coordinates .

Original: As far as we know, in this sense, no lower bounds are proved, and it deserves a separate piece of work.
Revised: As far as we know, in this sense, no lower bounds are proved, and it deserves a separate piece of work.

Original: However, Korhonen & Alistarh (2021) proved the lower bounds of the communication complexity with respect to the number of send bits in the constraint optimization setting that x ∈ [0 , 1] d , so our upper bounds can not be directly compared to their result because we operate on a different level of abstraction.
Revised: However, Korhonen & Alistarh (2021) proved the lower bounds of the communication complexity with respect to the number of send bits in the constraint optimization setting that x ∈ [0 , 1] d , so our upper bounds can not be directly compared to their result because we operate on a different level of abstraction.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: • Uncompressed communication.
Revised: • Uncompressed communication.

Original: This line of work is characterized by methods
Revised: This line of work is characterized by methods

Original: in which the nodes send messages (vectors) to the server without any compression.
Revised: in which the nodes send messages (vectors) to the server without any compression.

Original: In the ﬁnite-sum setting , the current state-of-the-art methods were proposed by Sharma et al.
Revised: In the ﬁnite-sum setting , the current state-of-the-art methods were proposed by Sharma et al.

Original: ; Li et al.
Revised: ; Li et al.

Original: (2021b), showing that after
Revised: (2021b), showing that after

Original: O ( 1 / ε ) communication rounds and
Revised: O ( 1 / ε ) communication rounds and

Original: [Equation]
Revised: [Equation]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: calculations of ∇ f ij per node, these methods can return an ε -solution.
Revised: calculations of ∇ f ij per node, these methods can return an ε -solution.

Original: Moreover, Sharma et al.
Revised: Moreover, Sharma et al.

Original: (2019) show that the same can be done in the stochastic setting after
Revised: (2019) show that the same can be done in the stochastic setting after

Original: [Equation]
Revised: [Equation]

Original: stochastic gradient calculations per node.
Revised: stochastic gradient calculations per node.

Original: Note that complexities (5) and (6) are optimal (Arjevani et al., 2019; Fang et al., 2018; Li et al., 2021a).
Revised: Note that complexities (5) and (6) are optimal (Arjevani et al., 2019; Fang et al., 2018; Li et al., 2021a).

Original: An adaptive variant was proposed by Khanduri et al.
Revised: An adaptive variant was proposed by Khanduri et al.

Original: based on the work of Cutkosky & Orabona (2019).
Revised: based on the work of Cutkosky & Orabona (2019).

Original: See also (Khanduri et al., 2021; Murata &
Revised: See also (Khanduri et al., 2021; Murata &

Original: Suzuki, 2021).
Revised: Suzuki, 2021).

Original: • Compressed communication.
Revised: • Compressed communication.

Original: In practice, it is rarely affordable to send uncompressed messages (vectors) from the nodes to the server due to
Revised: In practice, it is rarely affordable to send uncompressed messages (vectors) from the nodes to the server due to

Original: limited
Revised: limited

Original: communication bandwidth.
Revised: communication bandwidth.

Original: Because of this, researchers started to develop methods keeping in mind the communication complexity: the total number of
Revised: Because of this, researchers started to develop methods keeping in mind the communication complexity: the total number of

Original: coordinates/ﬂoats/bits
Revised: coordinates/ﬂoats/bits

Original: that the nodes send to the server to
Revised: that the nodes send to the server to

Original: ﬁnd
Revised: ﬁnd

Original: an ε solution.
Revised: an ε solution.

Original: Two important families of compressors are investigated in the literature to reduce communication bottleneck: biased and unbiased compressors.
Revised: Two important families of compressors are investigated in the literature to reduce communication bottleneck: biased and unbiased compressors.

Original: While unbiased compressors are superior in theory (Mishchenko et al., 2019; Li et al., 2020b; Gorbunov et al., 2021), biased compressors often enjoy better performance in practice (Beznosikov et al., 2020; Xu et al., 2020).
Revised: While unbiased compressors are superior in theory (Mishchenko et al., 2019; Li et al., 2020b; Gorbunov et al., 2021), biased compressors often enjoy better performance in practice (Beznosikov et al., 2020; Xu et al., 2020).

Original: Recently, Richtárik et al.
Revised: Recently, Richtárik et al.

Original: (2021) developed
Revised: (2021) developed

Original: EF21 , which is the ﬁrst method capable of working with biased compressors an having the theoretical iteration
Revised: EF21 , which is the ﬁrst method capable of working with biased compressors an having the theoretical iteration

Original: complexity of gradient descent ( GD ), up to constant factors.
Revised: complexity of gradient descent ( GD ), up to constant factors.

Original: • Unbiased compressors.
Revised: • Unbiased compressors.

Original: The theory around unbiased compressors is much more optimistic.
Revised: The theory around unbiased compressors is much more optimistic.

Original: Alistarh et al.
Revised: Alistarh et al.

Original: (2017) developed the QSGD method providing convergence rates of stochastic gradient method with quantized vectors.
Revised: (2017) developed the QSGD method providing convergence rates of stochastic gradient method with quantized vectors.

Original: However, the nonstrongly convex case was analyzed under the strong assumption that all nodes have identical functions, and the stochastic gradients have bounded second moment.
Revised: However, the nonstrongly convex case was analyzed under the strong assumption that all nodes have identical functions, and the stochastic gradients have bounded second moment.

Original: Next, Mishchenko et al.
Revised: Next, Mishchenko et al.

Original: ; Horváth et al.
Revised: ; Horváth et al.

Original: (2019) proposed the DIANA method and proved convergence rates without these restrictive assumptions.
Revised: (2019) proposed the DIANA method and proved convergence rates without these restrictive assumptions.

Original: Also, distributed nonconvex optimization methods with compression were developed by Haddadpour et al.
Revised: Also, distributed nonconvex optimization methods with compression were developed by Haddadpour et al.

Original: Das et al.
Revised: Das et al.

Original: Finally, Gorbunov et al.
Revised: Finally, Gorbunov et al.

Original: (2021) proposed MARINA – the current state-of-the-art distributed method in
Revised: (2021) proposed MARINA – the current state-of-the-art distributed method in

Original: terms
Revised: terms

Original: of theoretical
Revised: of theoretical

Original: communication
Revised: communication

Original: complexity,
Revised: complexity,

Original: inspired
Revised: inspired

Original: by the PAGE method of Li et al.
Revised: by the PAGE method of Li et al.

Original: 2021a
Revised: 2021a

Original:  C ONTRIBUTIONS
Revised:  C ONTRIBUTIONS

Original: We develop a new family of distributed optimization methods DASHA for nonconvex optimization problems with unbiased compressors.
Revised: We develop a new family of distributed optimization methods DASHA for nonconvex optimization problems with unbiased compressors.

Original: Compared to MARINA , our methods make more practical and simpler optimization steps.
Revised: Compared to MARINA , our methods make more practical and simpler optimization steps.

Original: In particular, in MARINA , all nodes simultaneously send either compressed vectors, with some probability p, or the gradients of functions { f i } ni =1 (uncompressed vectors), with probability 1 − p .
Revised: In particular, in MARINA , all nodes simultaneously send either compressed vectors, with some probability p, or the gradients of functions { f i } ni =1 (uncompressed vectors), with probability 1 − p .

Original: In other words, the server periodically synchronizes all nodes.
Revised: In other words, the server periodically synchronizes all nodes.

Original: In federated learning, where some nodes can be inaccessible for a long time, such periodic synchronization is intractable.
Revised: In federated learning, where some nodes can be inaccessible for a long time, such periodic synchronization is intractable.

Original: Our method DASHA solves both problems: i) the nodes always send compressed vectors , and ii) the server never synchronizes all nodes in the gradient setting .
Revised: Our method DASHA solves both problems: i) the nodes always send compressed vectors , and ii) the server never synchronizes all nodes in the gradient setting .

Original: Further, a simple tweak in the compressors (see Appendix D) results in support for partial participation in the gradient setting , which makes DASHA more practical for federated learning tasks.
Revised: Further, a simple tweak in the compressors (see Appendix D) results in support for partial participation in the gradient setting , which makes DASHA more practical for federated learning tasks.

Original: Let us summarize our most important theoretical and practical contributions:
Revised: Let us summarize our most important theoretical and practical contributions:

Original: • New theoretical SOTA complexity in the ﬁnite-sum setting.
Revised: • New theoretical SOTA complexity in the ﬁnite-sum setting.

Original: Using our novel approach to compress gradients, we improve the theoretical complexities of VR-MARINA (see Tables 1 and 2) in the ﬁnite-sum setting .
Revised: Using our novel approach to compress gradients, we improve the theoretical complexities of VR-MARINA (see Tables 1 and 2) in the ﬁnite-sum setting .

Original: Indeed, if the number of functions m is large, our algorithm DASHA-PAGE needs √ ω + 1 times fewer communications rounds, while communicating compressed vectors only.
Revised: Indeed, if the number of functions m is large, our algorithm DASHA-PAGE needs √ ω + 1 times fewer communications rounds, while communicating compressed vectors only.

Original: • New theoretical SOTA complexity in the stochastic setting.
Revised: • New theoretical SOTA complexity in the stochastic setting.

Original: We develop a new method, DASHASYNC-MVR , improving upon the previous state of the art (see Table 1).
Revised: We develop a new method, DASHASYNC-MVR , improving upon the previous state of the art (see Table 1).

Original: When ε is small, the number of communication rounds is reduced by a factor of √ ω
Revised: When ε is small, the number of communication rounds is reduced by a factor of √ ω

Original: + 1 .
Revised: + 1 .

Original: Indeed, we improve the dominant term which depends on ε 3 / 2 (the other terms depend on ε only).
Revised: Indeed, we improve the dominant term which depends on ε 3 / 2 (the other terms depend on ε only).

Original: However, DASHA-SYNC-MVR needs to periodically send uncompressed vectors with the same rate as VR-MARINA (online) .
Revised: However, DASHA-SYNC-MVR needs to periodically send uncompressed vectors with the same rate as VR-MARINA (online) .

Original: Nevertheless, we show that DASHA-MVR also improves the dominant term when ε is small, and this method sends compressed vectors only.
Revised: Nevertheless, we show that DASHA-MVR also improves the dominant term when ε is small, and this method sends compressed vectors only.

Original: Moreover, we provide detailed experiments on practical machine learning tasks: training nonconvex generalized linear models and deep neural networks, showing improvements predicted by our theory.
Revised: Moreover, we provide detailed experiments on practical machine learning tasks: training nonconvex generalized linear models and deep neural networks, showing improvements predicted by our theory.

Original: See Appendix A. • Closing the gap between uncompressed and compressed methods.
Revised: See Appendix A. • Closing the gap between uncompressed and compressed methods.

Original: In Section 2, we mentioned that the optimal oracle complexities of methods without compression in the ﬁnite
Revised: In Section 2, we mentioned that the optimal oracle complexities of methods without compression in the ﬁnite

Original: -sum and stochastic settings are (5) and (6), respectively.
Revised: -sum and stochastic settings are (5) and (6), respectively.

Original: Considering the Rand K compressor (see Deﬁnition F.1), we show that DASHA-PAGE , DASHA-MVR and DASHA-SYNC-MVR attain these optimal oracle complexities while attainting the state-of-the-art communication complexity as MARINA , which needs to use the stronger gradient oracle!
Revised: Considering the Rand K compressor (see Deﬁnition F.1), we show that DASHA-PAGE , DASHA-MVR and DASHA-SYNC-MVR attain these optimal oracle complexities while attainting the state-of-the-art communication complexity as MARINA , which needs to use the stronger gradient oracle!

Original: Therefore, our new methods close the gap between results from (Gorbunov et al., 2021) and (Sharma et al., 2019; Li et al., 2021b).
Revised: Therefore, our new methods close the gap between results from (Gorbunov et al., 2021) and (Sharma et al., 2019; Li et al., 2021b).

Original:  A LGORITHM D ESCRIPTION
Revised:  A LGORITHM D ESCRIPTION

Original: We now describe our proposed family of optimization methods, DASHA (see Algorithm 1).
Revised: We now describe our proposed family of optimization methods, DASHA (see Algorithm 1).

Original: DASHA is inspired by MARINA and momentum variance reduction methods ( MVR )
Revised: DASHA is inspired by MARINA and momentum variance reduction methods ( MVR )

Original: (Cutkosky & Orabona, 2019; Tran-Dinh et al., 2021;
Revised: (Cutkosky & Orabona, 2019; Tran-Dinh et al., 2021;

Original: Liu et al., 2020): the general structure repeats MARINA except for the variance reduction strategy, which we borrow from MVR .
Revised: Liu et al., 2020): the general structure repeats MARINA except for the variance reduction strategy, which we borrow from MVR .

Original: Unlike MARINA , our algorithm never sends uncompressed vectors, and the number of bits that every node sends is always the same.
Revised: Unlike MARINA , our algorithm never sends uncompressed vectors, and the number of bits that every node sends is always the same.

Original: Moreover, we reduce the variance from the oracle and the compressor separately , which helps us to improve the theoretical convergence rates in the stochastic and ﬁnite-sum cases.
Revised: Moreover, we reduce the variance from the oracle and the compressor separately , which helps us to improve the theoretical convergence rates in the stochastic and ﬁnite-sum cases.

Original: First, using the gradient estimator g t , the server in each communication round calculates the next point x t +1 and broadcasts it to the nodes.
Revised: First, using the gradient estimator g t , the server in each communication round calculates the next point x t +1 and broadcasts it to the nodes.

Original: Subsequently, all nodes in parallel calculate vectors h t
Revised: Subsequently, all nodes in parallel calculate vectors h t

Original: +1 i in one of three ways, depending on the available oracle.
Revised: +1 i in one of three ways, depending on the available oracle.

Original: For the the gradient, ﬁnite-sum, and the stochastic settings, we use GD -like, PAGE -like, and MVR -like strategies, respectively.
Revised: For the the gradient, ﬁnite-sum, and the stochastic settings, we use GD -like, PAGE -like, and MVR -like strategies, respectively.

Original: Next, each node compresses their message and uploads it to the server.
Revised: Next, each node compresses their message and uploads it to the server.

Original: Finally, the server aggregates all received messages and calculates the next vector g t +1 .
Revised: Finally, the server aggregates all received messages and calculates the next vector g t +1 .

Original: We refer to Section H to get a better intuition about
Revised: We refer to Section H to get a better intuition about

Original: DASHA .
Revised: DASHA .

Original: We note that in the stochastic setting, our analysis of DASHA-MVR (Algorithm 1) provides a suboptimal oracle complexity w.r.t.
Revised: We note that in the stochastic setting, our analysis of DASHA-MVR (Algorithm 1) provides a suboptimal oracle complexity w.r.t.

Original: ω (see Tables 1 and 2).
Revised: ω (see Tables 1 and 2).

Original: In Appendix J we provide experimental evidence that our analysis is tight.
Revised: In Appendix J we provide experimental evidence that our analysis is tight.

Original: For this reason, we developed DASHA-SYNC-MVR (see Algorithm 2 in
Revised: For this reason, we developed DASHA-SYNC-MVR (see Algorithm 2 in

Original: Appendix C) that improves the previous state-of-the-art results and sends non-compressed vectors with the same rate as VR-MARINA (online) .
Revised: Appendix C) that improves the previous state-of-the-art results and sends non-compressed vectors with the same rate as VR-MARINA (online) .

Original: Note that DASHA-MVR still enjoys the optimal oracle and
Revised: Note that DASHA-MVR still enjoys the optimal oracle and

Original: SOTA communication complexity (see Section 6.5); and this can be seen it in experiments.
Revised: SOTA communication complexity (see Section 6.5); and this can be seen it in experiments.

Original: 1: Input: starting point x 0 ∈ R d , stepsize γ > 0 , momentum a ∈ (0 , 1] , momentum b
Revised: 1: Input: starting point x 0 ∈ R d , stepsize γ > 0 , momentum a ∈ (0 , 1] , momentum b

Original: (0 , 1] (only
Revised: (0 , 1] (only

Original: DASHA
Revised: DASHA

Original: ), probability p
Revised: ), probability p

Original: (0 , 1] (only
Revised: (0 , 1] (only

Original: DASHA
Revised: DASHA

Original: PAGE ), batch size B
Revised: PAGE ), batch size B

Original: DASHA
Revised: DASHA

Original: PAGE
Revised: PAGE

Original: MVR ), number iterations
Revised: MVR ), number iterations

Original: T ≥2: Initialize g 0 i ∈ R
Revised: T ≥2: Initialize g 0 i ∈ R

Original: d , h 0 i ∈ R
Revised: d , h 0 i ∈ R

Original: d on the nodes and g 0 = 1 n (cid:80) ni
Revised: d on the nodes and g 0 = 1 n (cid:80) ni

Original: =1 g 0 i on the server 3: for t = 0 , 1 , . . .
Revised: =1 g 0 i on the server 3: for t = 0 , 1 , . . .

Original: , T − 1 do 4: x t +1 = x
Revised: , T − 1 do 4: x t +1 = x

Original: t − γg t 5:
Revised: t − γg t 5:

Original: Flip a coin c t +1 = (cid:26) 1 , with probability p 0 , with probability 1 − p (only in DASHA-PAGE ) 6:
Revised: Flip a coin c t +1 = (cid:26) 1 , with probability p 0 , with probability 1 − p (only in DASHA-PAGE ) 6:

Original: Broadcast x t
Revised: Broadcast x t

Original: 1 to all nodes 7: for i = 1 , . . .
Revised: 1 to all nodes 7: for i = 1 , . . .

Original: , n in parallel do 8: h t +1 i =    ∇ f i ( x t +1 ) (DASHA) (cid:40) ∇ f i ( x t +1 )
Revised: , n in parallel do 8: h t +1 i =    ∇ f i ( x t +1 ) (DASHA) (cid:40) ∇ f i ( x t +1 )

Original: if c t +1 =
Revised: if c t +1 =

Original: h ti + 1 B
Revised: h ti + 1 B

Original: (cid:80) j ∈
Revised: (cid:80) j ∈

Original: I ti (cid:0) ∇ f ij ( x t +1 )
Revised: I ti (cid:0) ∇ f ij ( x t +1 )

Original: − ∇ f ij ( x t ) (cid:1) if c t +1 =(DASHA-PAGE) 1 B (cid:80) B j =1 ∇ f i ( x t +1 ; ξ t +1 ij )
Revised: − ∇ f ij ( x t ) (cid:1) if c t +1 =(DASHA-PAGE) 1 B (cid:80) B j =1 ∇ f i ( x t +1 ; ξ t +1 ij )

Original: + (1 − b ) (cid:16) h t i − 1 B (cid:80) B j =1 ∇ f i ( x t ; ξ t +1 ij ) (cid:17) (DASHA-MVR) 9: m t
Revised: + (1 − b ) (cid:16) h t i − 1 B (cid:80) B j =1 ∇ f i ( x t ; ξ t +1 ij ) (cid:17) (DASHA-MVR) 9: m t

Original: +1 i
Revised: +1 i

Original: =
Revised: =

Original: C i (cid:0) h t
Revised: C i (cid:0) h t

Original: +1 i − h ti − a
Revised: +1 i − h ti − a

Original: ( g ti − h ti ) (cid:1) 10: g t +1 i
Revised: ( g ti − h ti ) (cid:1) 10: g t +1 i

Original: = g ti + m t
Revised: = g ti + m t

Original: +

Revised: +

Original: i 11:
Revised: i 11:

Original: Send m t
Revised: Send m t

Original: +1 i to the server 12: end for 13: g t +1 =
Revised: +1 i to the server 12: end for 13: g t +1 =

Original: g t + 1 n (cid:80) ni
Revised: g t + 1 n (cid:80) ni

Original: =1 m t
Revised: =1 m t

Original: +

Revised: +

Original: i 14: end for 15: Output: ˆ x T chosen uniformly at random from { x t } T − 1 k =0 (or x T under the PŁ-condition)
Revised: i 14: end for 15: Output: ˆ x T chosen uniformly at random from { x t } T − 1 k =0 (or x T under the PŁ-condition)

Original:  A SSUMPTIONS
Revised:  A SSUMPTIONS

Original: We now provide the assumptions used throughout our paper.
Revised: We now provide the assumptions used throughout our paper.

Original: Assumption 5.1.
Revised: Assumption 5.1.

Original: There exists f ∗ ∈ R such that f ( x ) ≥ f ∗ for all x ∈ R .
Revised: There exists f ∗ ∈ R such that f ( x ) ≥ f ∗ for all x ∈ R .

Original: Assumption 5.2.
Revised: Assumption 5.2.

Original: The function f is L –smooth, i.e.,
Revised: The function f is L –smooth, i.e.,

Original: [Equation]
Revised: [Equation]

Original: for all x, y ∈ R d .
Revised: for all x, y ∈ R d .

Original: Assumption 5.3.
Revised: Assumption 5.3.

Original: For all i ∈ [ n ] , the function f i is L i –smooth.
Revised: For all i ∈ [ n ] , the function f i is L i –smooth.

Original: 2 We deﬁne (cid:98) L
Revised: 2 We deﬁne (cid:98) L

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ni =1 L 2 i .
Revised: ni =1 L 2 i .

Original: The next assumption is used in the ﬁnite-sum setting (2).
Revised: The next assumption is used in the ﬁnite-sum setting (2).

Original: Assumption 5.4.
Revised: Assumption 5.4.

Original: For all i ∈ [ n ] , j ∈ [ m ] , the function f ij is L ij -smooth.
Revised: For all i ∈ [ n ] , j ∈ [ m ] , the function f ij is L ij -smooth.

Original: Let L max := max i ∈ [ n ] ,j ∈ [ m ] L ij .
Revised: Let L max := max i ∈ [ n ] ,j ∈ [ m ] L ij .

Original: The two assumptions below are provided for the stochastic setting (3).
Revised: The two assumptions below are provided for the stochastic setting (3).

Original: Assumption 5.5.
Revised: Assumption 5.5.

Original: For all i ∈ [ n ] and for all x ∈ R d , the stochastic gradient ∇ f i ( x ; ξ ) is unbiased and has bounded variance, i.e.,
Revised: For all i ∈ [ n ] and for all x ∈ R d , the stochastic gradient ∇ f i ( x ; ξ ) is unbiased and has bounded variance, i.e.,

Original: E ξ [
Revised: E ξ [

Original: [Equation]
Revised: [Equation]

Original: f i
Revised: f i

Original: [Equation]
Revised: [Equation]

Original: ; ξ )]
Revised: ; ξ )]

Original: = ∇ f i ( x ) , and
Revised: = ∇ f i ( x ) , and

Original: [Equation]
Revised: [Equation]

Original: where σ 2 ≥ 0 .
Revised: where σ 2 ≥ 0 .

Original: Assumption 5.6.
Revised: Assumption 5.6.

Original: For all i ∈ [ n ] and for all x, y ∈ R , the stochastic gradient ∇ f i ( x ; ξ ) satisﬁes the mean-squared smoothness property, i.e.,
Revised: For all i ∈ [ n ] and for all x, y ∈ R , the stochastic gradient ∇ f i ( x ; ξ ) satisﬁes the mean-squared smoothness property, i.e.,

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original:  T HEORETICAL C ONVERGENCE R ATES
Revised:  T HEORETICAL C ONVERGENCE R ATES

Original: Now, we provide convergence rate theorems for DASHA , DASHA-PAGE and DASHA-MVR .
Revised: Now, we provide convergence rate theorems for DASHA , DASHA-PAGE and DASHA-MVR .

Original: All three methods are listed in Algorithm 1 and differ in Line 8 only.
Revised: All three methods are listed in Algorithm 1 and differ in Line 8 only.

Original: At the end of the section, we provide a theorem for DASHA-SYNC-MVR .
Revised: At the end of the section, we provide a theorem for DASHA-SYNC-MVR .

Original: 6. G RADIENT S ETTING ( DASHA )
Revised: 6. G RADIENT S ETTING ( DASHA )

Original: Theorem 6.1.
Revised: Theorem 6.1.

Original: Suppose that Assumptions 5.1, 5.2, 5.3 and 1.2 hold.
Revised: Suppose that Assumptions 5.1, 5.2, 5.3 and 1.2 hold.

Original: Let us take a = 1 / (2 ω + 1)
Revised: Let us take a = 1 / (2 ω + 1)

Original: , γ ≤ (cid:18) L +
Revised: , γ ≤ (cid:18) L +

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: n (cid:98) L (cid:19) −, and g 

Revised: n (cid:98) L (cid:19) −, and g

Original: i =
Revised: i =

Original: h 0 i = ∇
Revised: h 0 i = ∇

Original: f i ( x 0 ) for all i ∈ [ n ] in Algorithm 1 (DASHA) , then
Revised: f i ( x 0 ) for all i ∈ [ n ] in Algorithm 1 (DASHA) , then

Original: [Equation]
Revised: [Equation]

Original: The corollary below simpliﬁes the previous theorem and reveals the communication complexity of
Revised: The corollary below simpliﬁes the previous theorem and reveals the communication complexity of

Original: DASHA .
Revised: DASHA .

Original: Corollary 6.2.
Revised: Corollary 6.2.

Original: Suppose that assumptions from Theorem 6.1 hold, and g 0 i
Revised: Suppose that assumptions from Theorem 6.1 hold, and g 0 i

Original: =
Revised: =

Original: h 

Revised: h

Original: i = ∇
Revised: i = ∇

Original: f i ( x 0 ) for all i ∈ [ n ] , then DASHA needs T :
Revised: f i ( x 0 ) for all i ∈ [ n ] , then DASHA needs T :

Original: = O
Revised: = O

Original: [Equation]
Revised: [Equation]

Original: communication rounds to get an ε -solution and the communication complexity is equal to O ( d + ζ C T ) , where ζ C is the expected density from Deﬁnition 1.3.
Revised: communication rounds to get an ε -solution and the communication complexity is equal to O ( d + ζ C T ) , where ζ C is the expected density from Deﬁnition 1.3.

Original: In the previous corollary, we have free parameters ω and ζ C .
Revised: In the previous corollary, we have free parameters ω and ζ C .

Original: Now, we consider the Rand K compressor (see Deﬁnition F.1) and choose its parameters to get the communication complexity w.r.t.
Revised: Now, we consider the Rand K compressor (see Deﬁnition F.1) and choose its parameters to get the communication complexity w.r.t.

Original: only d and n .
Revised: only d and n .

Original: Corollary 6.3.
Revised: Corollary 6.3.

Original: Suppose that assumptions of Corollary 6.2 hold.
Revised: Suppose that assumptions of Corollary 6.2 hold.

Original: We take the unbiased compressor
Revised: We take the unbiased compressor

Original: Rand K with K = ζ C ≤ d/ √ n, then the communication complexity equals O (cid:18)
Revised: Rand K with K = ζ C ≤ d/ √ n, then the communication complexity equals O (cid:18)

Original: d + (cid:98) L ( f ( x 0 ) − f ∗ ) d ε √ n (cid:19) .
Revised: d + (cid:98) L ( f ( x 0 ) − f ∗ ) d ε √ n (cid:19) .

Original: 6. F INITE -S UM S ETTING ( DASHA-PAGE )
Revised: 6. F INITE -S UM S ETTING ( DASHA-PAGE )

Original: Next, we provide the complexity bounds for DASHA-PAGE .
Revised: Next, we provide the complexity bounds for DASHA-PAGE .

Original: Theorem 6.4.
Revised: Theorem 6.4.

Original: Suppose that Assumptions 5.1, 5.2, 5.3, 5.4, and 1.2 hold.
Revised: Suppose that Assumptions 5.1, 5.2, 5.3, 5.4, and 1.2 hold.

Original: Let us take a = 1 / (2 ω + 1) , probability p ∈ (0 , 1] , γ ≤ (cid:32)
Revised: Let us take a = 1 / (2 ω + 1) , probability p ∈ (0 , 1] , γ ≤ (cid:32)

Original: L + (cid:115) 48 ω
Revised: L + (cid:115) 48 ω

Original: [Equation]
Revised: [Equation]

Original: ) n (cid:18)
Revised: ) n (cid:18)

Original: [Equation]
Revised: [Equation]

Original: B
Revised: B

Original: [Equation]
Revised: [Equation]

Original: and g 0 i = h 

Revised: and g 0 i = h

Original: i = ∇
Revised: i = ∇

Original: f i ( x 0 ) for all i ∈ [ n ] in Algorithm 1 (DASHA-PAGE) then E (cid:104)(cid:13) (cid:13) ∇ f ( (cid:98) x T ) (cid:13) (cid:13) 2 (cid:105) ≤ 2 ( f ( x 0 ) − f ∗ ) γT .
Revised: f i ( x 0 ) for all i ∈ [ n ] in Algorithm 1 (DASHA-PAGE) then E (cid:104)(cid:13) (cid:13) ∇ f ( (cid:98) x T ) (cid:13) (cid:13) 2 (cid:105) ≤ 2 ( f ( x 0 ) − f ∗ ) γT .

Original: Let us simplify the statement of Theorem 6.4 by choosing particular parameters.
Revised: Let us simplify the statement of Theorem 6.4 by choosing particular parameters.

Original: Corollary 6.5.
Revised: Corollary 6.5.

Original: Let the assumptions from Theorem 6.4 hold, p = B / ( m + B ) , and g 0 i
Revised: Let the assumptions from Theorem 6.4 hold, p = B / ( m + B ) , and g 0 i

Original: =
Revised: =

Original: h 

Revised: h

Original: i = ∇
Revised: i = ∇

Original: f i ( x 0 ) for all i ∈ [ n ] .
Revised: f i ( x 0 ) for all i ∈ [ n ] .

Original: Then DASHA-PAGE needs
Revised: Then DASHA-PAGE needs

Original: [Equation]
Revised: [Equation]

Original:    communication rounds to get an ε -solution, the communication complexity is equal to O ( d + ζ C T ) , and the expected # of gradient calculations per node equals O ( m + BT ) , where ζ C is the expected density from Deﬁnition 1.3.
Revised:    communication rounds to get an ε -solution, the communication complexity is equal to O ( d + ζ C T ) , and the expected # of gradient calculations per node equals O ( m + BT ) , where ζ C is the expected density from Deﬁnition 1.3.

Original: The corollary below reveals the communication and oracle complexities of Algorithm 1 (DASHAPAGE) with Rand K .
Revised: The corollary below reveals the communication and oracle complexities of Algorithm 1 (DASHAPAGE) with Rand K .

Original: Corollary 6.6.
Revised: Corollary 6.6.

Original: Suppose that assumptions of Corollary 6.5 hold, B ≤ (cid:112) m / n , and we use the unbiased compressor Rand K with K = ζ C =
Revised: Suppose that assumptions of Corollary 6.5 hold, B ≤ (cid:112) m / n , and we use the unbiased compressor Rand K with K = ζ C =

Original: Θ ( Bd / √ m ) .
Revised: Θ ( Bd / √ m ) .

Original: Then the communication complexity of Algorithmis
Revised: Then the communication complexity of Algorithmis

Original: [Equation]
Revised: [Equation]

Original: and the expected # of gradient calculations per node equals
Revised: and the expected # of gradient calculations per node equals

Original: [Equation]
Revised: [Equation]

Original: Up to Lipschitz constants factors, bound (8) is optimal (Fang et al., 2018; Li et al., 2021a), and unlike
Revised: Up to Lipschitz constants factors, bound (8) is optimal (Fang et al., 2018; Li et al., 2021a), and unlike

Original: VR-MARINA , we recover the optimal bound with compression!
Revised: VR-MARINA , we recover the optimal bound with compression!

Original: At the same time, the communication complexity (7) is the same as in DASHA (see Corollary 6.3) or MARINA .
Revised: At the same time, the communication complexity (7) is the same as in DASHA (see Corollary 6.3) or MARINA .

Original: 6. S TOCHASTIC S ETTING ( DASHA-MVR )
Revised: 6. S TOCHASTIC S ETTING ( DASHA-MVR )

Original: Let h t :=
Revised: Let h t :=

Original: 1 n (cid:80) n i =

Revised: 1 n (cid:80) n i =

Original: h t i .
Revised: h t i .

Original: This vector is not used in Algorithm 1, but appears in the theoretical results.
Revised: This vector is not used in Algorithm 1, but appears in the theoretical results.

Original: Theorem 6.7.
Revised: Theorem 6.7.

Original: Suppose that Assumptions 5.1, 5.2, 5.3, 5.5, 5.6 and 1.2 hold.
Revised: Suppose that Assumptions 5.1, 5.2, 5.3, 5.5, 5.6 and 1.2 hold.

Original: Let us take a = 1 2 ω +1 , b ∈ (0 , 1] , γ ≤ (cid:18)
Revised: Let us take a = 1 2 ω +1 , b ∈ (0 , 1] , γ ≤ (cid:18)

Original: L +
Revised: L +

Original: [Equation]
Revised: [Equation]

Original: , and g 0 i
Revised: , and g 0 i

Original: =
Revised: =

Original: h 

Revised: h

Original: i for all i ∈
Revised: i for all i ∈

Original: [ n ] in Algorithm 1 (DASHA-MVR) .
Revised: [ n ] in Algorithm 1 (DASHA-MVR) .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: 2 (cid:105) ≤ 1 T   2 (cid:0) f
Revised: 2 (cid:105) ≤ 1 T   2 (cid:0) f

Original: ( x 0 ) − f ∗ (cid:1) γ +b (cid:13)(cid:13) h 0 − ∇ f ( x 0 ) (cid:13)(cid:13)+ 32 bω (2 ω + 1) n (cid:32)n n (cid:88) i = (cid:13)(cid:13) h 

Revised: ( x 0 ) − f ∗ (cid:1) γ +b (cid:13)(cid:13) h 0 − ∇ f ( x 0 ) (cid:13)(cid:13)+ 32 bω (2 ω + 1) n (cid:32)n n (cid:88) i = (cid:13)(cid:13) h

Original: i − ∇ f i ( x 0 ) (cid:13)(cid:13)(cid:33) 
Revised: i − ∇ f i ( x 0 ) (cid:13)(cid:13)(cid:33) 

Original:  + (cid:18) 96 ω (2 ω + 1) nB + bnB (cid:19)
Revised:  + (cid:18) 96 ω (2 ω + 1) nB + bnB (cid:19)

Original: b 2 σ 2 .
Revised: b 2 σ 2 .

Original: Corollary 6.8.
Revised: Corollary 6.8.

Original: Suppose that assumptions from Theorem 6.7 hold, momentum b = Θ (cid:16) min (cid:110) 1 ω (cid:113) nεBσ 2 , nεBσ 2 (cid:111)(cid:17) , and g 0 i
Revised: Suppose that assumptions from Theorem 6.7 hold, momentum b = Θ (cid:16) min (cid:110) 1 ω (cid:113) nεBσ 2 , nεBσ 2 (cid:111)(cid:17) , and g 0 i

Original: =
Revised: =

Original: h 

Revised: h

Original: i = 1 B init (cid:80) B init k
Revised: i = 1 B init (cid:80) B init k

Original: =1 ∇ f i ( x 0 ; ξ 0 ik ) for all i ∈ [ n ] , and batch size B init =
Revised: =1 ∇ f i ( x 0 ; ξ 0 ik ) for all i ∈ [ n ] , and batch size B init =

Original: Θ ( B / b ) , then Algorithm 1 (DASHA-MVR) needs
Revised: Θ ( B / b ) , then Algorithm 1 (DASHA-MVR) needs

Original: T :
Revised: T :

Original: = O 
Revised: = O 

Original: [Equation]
Revised: [Equation]

Original:  communication rounds to get an ε -solution, the communication complexity is equal to O ( d + ζ C T ) , and the number of stochastic gradient calculations per node equals O ( B init + BT ) , where ζ C is the expected density from Deﬁnition 1.3.
Revised:  communication rounds to get an ε -solution, the communication complexity is equal to O ( d + ζ C T ) , and the number of stochastic gradient calculations per node equals O ( B init + BT ) , where ζ C is the expected density from Deﬁnition 1.3.

Original: The following corollary reveals the communication and oracle complexity of DASHA-MVR .
Revised: The following corollary reveals the communication and oracle complexity of DASHA-MVR .

Original: Corollary 6.9.
Revised: Corollary 6.9.

Original: Suppose that assumptions of Corollary 6.8 hold, batch size B ≤ σ √ εn , we take Rand K with K = ζ C = Θ (cid:16) Bd √ εn σ (cid:17) , and (cid:101)
Revised: Suppose that assumptions of Corollary 6.8 hold, batch size B ≤ σ √ εn , we take Rand K with K = ζ C = Θ (cid:16) Bd √ εn σ (cid:17) , and (cid:101)

Original: L :=
Revised: L :=

Original: max { L, L σ , (cid:98) L } .
Revised: max { L, L σ , (cid:98) L } .

Original: Then the communication complexity equals
Revised: Then the communication complexity equals

Original: [Equation]
Revised: [Equation]

Original: and the expected # of stochastic gradient calculations per node equals
Revised: and the expected # of stochastic gradient calculations per node equals

Original: [Equation]
Revised: [Equation]

Original: Up to Lipschitz constant factors, the bound (10) is optimal (Arjevani et al., 2019; Sharma et al., 2019), and unlike VR-MARINA (online) , we recover the optimal bound with compression!
Revised: Up to Lipschitz constant factors, the bound (10) is optimal (Arjevani et al., 2019; Sharma et al., 2019), and unlike VR-MARINA (online) , we recover the optimal bound with compression!

Original: At the same time, the communication complexity (9) is the same as in DASHA (see Corollary 6.3) or MARINA for small enough ε.
Revised: At the same time, the communication complexity (9) is the same as in DASHA (see Corollary 6.3) or MARINA for small enough ε.

Original: 6. S TOCHASTIC S ETTING ( DASHA-SYNC-MVR )
Revised: 6. S TOCHASTIC S ETTING ( DASHA-SYNC-MVR )

Original: We now provide the complexities of Algorithm 2 ( DASHA-SYNC-MVR ) presented in Appendix C.
Revised: We now provide the complexities of Algorithm 2 ( DASHA-SYNC-MVR ) presented in Appendix C.

Original: The main convergence rate Theorem I.19 is in the appendix.
Revised: The main convergence rate Theorem I.19 is in the appendix.

Original: Corollary 6.10.
Revised: Corollary 6.10.

Original: Suppose that assumptions from Theorem I.19 hold, probability p =
Revised: Suppose that assumptions from Theorem I.19 hold, probability p =

Original: min (cid:110)
Revised: min (cid:110)

Original: [Equation]
Revised: [Equation]

Original: d , nεBσ 2 (cid:111) , batch size B (cid:48) =
Revised: , nεBσ 2 (cid:111) , batch size B (cid:48) =

Original: Θ (cid:16) σ 2 nε (cid:17) and h 0 i = g 0 i = 1 B init (cid:80) B init k
Revised: Θ (cid:16) σ 2 nε (cid:17) and h 0 i = g 0 i = 1 B init (cid:80) B init k

Original: =1 ∇ f i ( x 0 ; ξ 0 ik ) for all i ∈ [ n ] , initial batch size B init =
Revised: =1 ∇ f i ( x 0 ; ξ 0 ik ) for all i ∈ [ n ] , initial batch size B init =

Original: Θ (cid:16) max (cid:110) σ 2 nε , B dζ C (cid:111)(cid:17) , then DASHA-SYNC-MVR needs
Revised: Θ (cid:16) max (cid:110) σ 2 nε , B dζ C (cid:111)(cid:17) , then DASHA-SYNC-MVR needs

Original: [Equation]
Revised: [Equation]

Original: communication rounds to get an ε -solution, the communication complexity is equal to O ( d + ζ C T ) , and the number of stochastic gradient calculations per node equals O ( B init + BT ) , where ζ C is the expected density from Deﬁnition 1.3.
Revised: communication rounds to get an ε -solution, the communication complexity is equal to O ( d + ζ C T ) , and the number of stochastic gradient calculations per node equals O ( B init + BT ) , where ζ C is the expected density from Deﬁnition 1.3.

Original: Corollary 6.11.
Revised: Corollary 6.11.

Original: Suppose that assumptions of Corollary 6.10 hold, batch size B ≤ σ √ εn , we take
Revised: Suppose that assumptions of Corollary 6.10 hold, batch size B ≤ σ √ εn , we take

Original: Rand K with K = ζ C =
Revised: Rand K with K = ζ C =

Original: Θ (cid:16) Bd √ εn σ (cid:17) , and (cid:101)
Revised: Θ (cid:16) Bd √ εn σ (cid:17) , and (cid:101)

Original: L :=
Revised: L :=

Original: max { L, L σ , (cid:98) L } .
Revised: max { L, L σ , (cid:98) L } .

Original: Then the communication complexity equals
Revised: Then the communication complexity equals

Original: [Equation]
Revised: [Equation]

Original: and the expected # of stochastic gradient calculations per node equals
Revised: and the expected # of stochastic gradient calculations per node equals

Original: [Equation]
Revised: [Equation]

Original: Up to Lipschitz constant factors, the bound (12) is optimal (Arjevani et al., 2019; Sharma et al., 2019), and unlike VR-MARINA (online) , we recover the optimal bound with compression!
Revised: Up to Lipschitz constant factors, the bound (12) is optimal (Arjevani et al., 2019; Sharma et al., 2019), and unlike VR-MARINA (online) , we recover the optimal bound with compression!

Original: At the same time, the communication complexity (11) is the same as in DASHA (see Corollary 6.3) or MARINA for small enough ε.
Revised: At the same time, the communication complexity (11) is the same as in DASHA (see Corollary 6.3) or MARINA for small enough ε.

Original: 6. C OMPARISON OF DASHA-MVR AND DASHA-SYNC-MVR
Revised: C OMPARISON OF DASHA-MVR AND DASHA-SYNC-MVR

Original: Let us consider Rand K (note that ω
Revised: Let us consider Rand K (note that ω

Original: + 1 = d / ζ C ).
Revised: + 1 = d / ζ C ).

Original: Comparing Corollary 6.8 to Corollary 6.
Revised: Comparing Corollary 6.8 to Corollary 6.

Original: (see Table 1), we see that DASHA-SYNC-MVR improved the size of the initial batch from Θ (cid:18) max (cid:26) σnε , Bω (cid:113) σ 2 nεB (cid:27)(cid:19) to Θ (cid:16) max (cid:110) σ 2 nε , Bω (cid:111)(cid:17) .
Revised: (see Table 1), we see that DASHA-SYNC-MVR improved the size of the initial batch from Θ (cid:18) max [Equation] nε , Bω (cid:113) σ [Equation]

Revised: nεB (cid:27)(cid:19) to Θ (cid:16) max (cid:110) σ 2 nε , Bω (cid:111)(cid:17) .

Original: Fortunately, we can control the parameter K in Rand K , and Corollary 6.9 reveals that we can take K = Θ (cid:16)
Revised: Fortunately, we can control the parameter K in Rand K , and Corollary 6.9 reveals that we can take K = Θ (cid:16)

Original: Bd √ εn σ (cid:17) , and get Bω (cid:113) 
Revised: Bd √ εn σ (cid:17) , and get Bω (cid:113) σ

Original: [Equation]
Revised: [Equation]

Original: As a result, the “bad term” does not dominate the oracle complexity, and DASHA-MVR attains the optimal oracle and SOTA communication complexity.
Revised: As a result, the “bad term” does not dominate the oracle complexity, and DASHA-MVR attains the optimal oracle and SOTA communication complexity.

Original: The same reasoning applies to optimization problems under PŁ-condition with K = Θ (cid:16) Bd √ µεnσ (cid:17) .
Revised: The same reasoning applies to optimization problems under PŁ-condition with K = Θ (cid:16) Bd √ µεn σ (cid:17) .

Original: A CKNOWLEDGEMENTS
Revised: A CKNOWLEDGEMENTS

Original: The work of P. Richtárik was partially supported by the KAUST Baseline Research Fund Scheme and by the SDAIA-KAUST Center of Excellence in Data Science and Artiﬁcial Intelligence.
Revised: The work of P. Richtárik was partially supported by the KAUST Baseline Research Fund Scheme and by the SDAIA-KAUST Center of Excellence in Data Science and Artiﬁcial Intelligence.

Revised: The work of A. Tyurin was supported by the Extreme Computing Research Center (ECRC) at KAUST.

Original: C ONTENTS 1 Introduction
Revised: C ONTENTS 1 Introduction

Original: Problem formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Problem formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: Gradient oracles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Gradient oracles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: Oracle complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Oracle complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: Unbiased compressors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Unbiased compressors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: Communication complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Communication complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: 2 Related Work
Revised: 2 Related Work

Original: Contributions
Revised: Contributions

Original: 4 Algorithm Description 5 Assumptions 6 Theoretical Convergence Rates 6.
Revised: 4 Algorithm Description 5 Assumptions 6 Theoretical Convergence Rates 6.

Original: Gradient Setting ( DASHA ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Gradient Setting ( DASHA ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: Finite-Sum Setting ( DASHA-PAGE ) . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Finite-Sum Setting ( DASHA-PAGE ) . . . . . . . . . . . . . . . . . . . . . . . . .

Original: Stochastic Setting ( DASHA-MVR ) . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Stochastic Setting ( DASHA-MVR ) . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: Stochastic Setting ( DASHA-SYNC-MVR ) . . . . . . . . . . . . . . . . . . . . . . .
Revised: Stochastic Setting ( DASHA-SYNC-MVR ) . . . . . . . . . . . . . . . . . . . . . . .

Original: Comparison of DASHA-MVR and DASHA-SYNC-MVR . . . . . . . . . . . . . . . .
Revised: Comparison of DASHA-MVR and DASHA-SYNC-MVR . . . . . . . . . . . . . . . .

Original: A Experiments
Revised: A Experiments

Original: A.1 Gradient setting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.1 Gradient setting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A.2 Finite-sum setting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.2 Finite-sum setting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A.3 Stochastic setting . .
Revised: A.3 Stochastic setting . .

Original: A.4 Deep neural network training . .
Revised: A.4 Deep neural network training . .

Original: B Experiments Details
Revised: B Experiments Details

Original: C Description of DASHA-SYNC-MVR
Revised: C Description of DASHA-SYNC-MVR

Original: D Partial Participation
Revised: D Partial Participation

Original: E Auxiliary Facts
Revised: E Auxiliary Facts

Original: F Compressors Facts
Revised: F Compressors Facts

Original: G Polyak-Łojasiewicz Condition
Revised: G Polyak-Łojasiewicz Condition

Original: H Intuition Behind DASHA
Revised: H Intuition Behind DASHA

Original: H.1 Different Sources of Contractions . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: H.1 Different Sources of Contractions . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: H.
Revised: H.

Original: I
Revised: I

Original: Theorems with Proofs
Revised: Theorems with Proofs

Original: I.
Revised: I.

Original: Case of DASHA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Case of DASHA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: I.
Revised: I.

Original: Case of DASHA under PŁ-condition . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Case of DASHA under PŁ-condition . . . . . . . . . . . . . . . . . . . . . . . . .

Original: I.
Revised: I.

Original: Case of DASHA-PAGE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Case of DASHA-PAGE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: I.
Revised: I.

Original: Case of DASHA-PAGE under PŁ-condition . . . . . . . . . . . . . . . . . . . . . .
Revised: Case of DASHA-PAGE under PŁ-condition . . . . . . . . . . . . . . . . . . . . . .

Original: I.
Revised: I.

Original: Case of DASHA-MVR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Case of DASHA-MVR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: I.
Revised: I.

Original: Case of DASHA-MVR under PŁ-condition . . . . . . . . . . . . . . . . . . . . . .
Revised: Case of DASHA-MVR under PŁ-condition . . . . . . . . . . . . . . . . . . . . . .

Original: I.
Revised: I.

Original: Case of DASHA-SYNC-MVR . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Case of DASHA-SYNC-MVR . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: I.
Revised: I.

Original: Case of DASHA-SYNC-MVR under PŁ-condition . . . . . . . . . . . . . . . . . .
Revised: Case of DASHA-SYNC-MVR under PŁ-condition . . . . . . . . . . . . . . . . . .

Original: J Extra Experiments
Revised: J Extra Experiments

Original: A E XPERIMENTS
Revised: A E XPERIMENTS

Original: We have tested all developed algorithms on practical machine learnings problems 3 .
Revised: We have tested all developed algorithms on practical machine learnings problems 3 .

Original: Note that the goal of our experiments is to justify the theoretical convergence rates from our paper.
Revised: Note that the goal of our experiments is to justify the theoretical convergence rates from our paper.

Original: We compare the new methods with MARINA on LIBSVM datasets (Chang & Lin, 2011)
Revised: We compare the new methods with MARINA on LIBSVM datasets (Chang & Lin, 2011)

Original: (under the 3-clause BSD license) because MARINA is the only previous state-of-the-art method for the problem (1).
Revised: (under the 3-clause BSD license) because MARINA is the only previous state-of-the-art method for the problem (1).

Original: Moreover, we show the advantage of our method on an image recognition task with CIFAR10 (Krizhevsky et al., 2009) and a deep neural network.
Revised: Moreover, we show the advantage of our method on an image recognition task with CIFAR10 (Krizhevsky et al., 2009) and a deep neural network.

Original: In all experiments, we take parameters of algorithms predicted by the theory (stated in the convergence rate theorems our paper and in (Gorbunov et al., 2021)), except for the step sizes – we ﬁne-tune them using a set of powers of two { 2 i | i ∈ [ − 10 , 10] } – and use the
Revised: In all experiments, we take parameters of algorithms predicted by the theory (stated in the convergence rate theorems our paper and in (Gorbunov et al., 2021)), except for the step sizes – we ﬁne-tune them using a set of powers of two { 2 i | i ∈ [ − 10 , 10] } – and use the

Original: Rand K compressor.
Revised: Rand K compressor.

Original: We evaluate communication complexity; thus, each plot represents the relation between the norm of a gradient or function value (vertical axis), and the total number of transmitted bits per node (horizontal axis).
Revised: We evaluate communication complexity; thus, each plot represents the relation between the norm of a gradient or function value (vertical axis), and the total number of transmitted bits per node (horizontal axis).

Original: A. G RADIENT SETTING
Revised: A. G RADIENT SETTING

Original: We consider nonconvex functions
Revised: We consider nonconvex functions

Original: [Equation]
Revised: [Equation]

Original: to solve a classiﬁcation problem.
Revised: to solve a classiﬁcation problem.

Original: Here, a ij ∈ R d is the feature vector of a sample on the i th node, y ij ∈ {− 1 , 1 } is the corresponding label, and m is the number of samples on the i th node.
Revised: Here, a ij ∈ R d is the feature vector of a sample on the i th node, y ij ∈ {− 1 , 1 } is the corresponding label, and m is the number of samples on the i th node.

Original: All nodes calculate full gradients.
Revised: All nodes calculate full gradients.

Original: We take the mushrooms dataset (dimension d = 112 , number of samples equals 8124 ) from LIBSVM, randomly split the dataset between 5 nodes and take K = 10 in Rand K .
Revised: We take the mushrooms dataset (dimension d = 112 , number of samples equals 8124 ) from LIBSVM, randomly split the dataset between 5 nodes and take K = 10 in Rand K .

Original: One can see in Figure 1 that DASHA converges approximately 2 times faster.
Revised: One can see in Figure 1 that DASHA converges approximately 2 times faster.

Original: #bit# / n 1e
Revised: #bit# / n 1e

Original: [Figure]
Revised: [Figure]

Original: n
Revised: n

Original: [Figure]
Revised: [Figure]

Original: d
Revised: d

Original: [Figure]
Revised: [Figure]

Original: A. F INITE - SUM SETTING
Revised: A. F INITE - SUM SETTING

Original: Now
Revised: Now

Original: [Figure]
Revised: [Figure]

Original: we conduct the same experiments as in Section A.1 with real-sim dataset (dimension d = 20 , 958 , number of samples equals 72 , 309 ) from LIBSVM in the ﬁnite-sum setting; moreover
Revised: we conduct the same experiments as in Section A.1 with real-sim dataset (dimension d = 20 , 958 , number of samples equals 72 , 309 ) from LIBSVM in the ﬁnite-sum setting; moreover

Original: [Figure]
Revised: [Figure]

Original: we compare VR-MARINA versus DASHA-PAGE with batch size B = 1 in both algorithms.
Revised: we compare VR-MARINA versus DASHA-PAGE with batch size B = 1 in both algorithms.

Original: Results in
Revised: Results in

Original: Figure 2 coincide with Table 1 – our new method DASHA-PAGE converges faster than MARINA .
Revised: Figure 2 coincide with Table 1 – our new method DASHA-PAGE converges faster than MARINA .

Original: When K = 100 , the improvement is not signiﬁcant because 1+
Revised: When K = 100 , the improvement is not signiﬁcant because 1+

Original: [Figure]
Revised: [Figure]

Original: dominates √ m ε √ nB
Revised: dominates √ m ε √ nB

Original: (see Table 1), and both algorithms get the same theoretical convergence complexity
Revised: (see Table 1), and both algorithms get the same theoretical convergence complexity

Original: [Figure]
Revised: [Figure]

Original: Classiﬁcation task with the real-sim dataset and K ∈ { 100; 500; 2 , 000 } in Rand K in the ﬁnite-sum setting.
Revised: Classiﬁcation task with the real-sim dataset and K ∈ { 100; 500; 2 , 000 } in Rand K in the ﬁnite-sum setting.

Original: A. S TOCHASTIC SETTING
Revised: A. S TOCHASTIC SETTING

Original: In this experiment, we consider the following logistic regression functions with nonconvex regularizer { f i } ni =1 to solve a classiﬁcation problem:
Revised: In this experiment, we consider the following logistic regression functions with nonconvex regularizer { f i } ni =1 to solve a classiﬁcation problem:

Original: [Equation]
Revised: [Equation]

Original: where x 1 , x 2 ∈ R d , {·} k is an indexing operation, a ij ∈ R d is a feature of a sample on the i th node, y ij ∈ { 1 , 2 } is a corresponding label, m is the number of samples located on the i th node, constant λ = 0 .
Revised: where x 1 , x 2 ∈ R d , {·} k is an indexing operation, a ij ∈ R d is a feature of a sample on the i th node, y ij ∈ { 1 , 2 } is a corresponding label, m is the number of samples located on the i th node, constant λ = 0 .

Original: We take batch size B = 1 and compare VR-MARINA (online) , DASHA-MVR , and DASHA-SYNC-MVR that depend on a common ratio σ 2 / nεB 4 .
Revised: We take batch size B = 1 and compare VR-MARINA (online) , DASHA-MVR , and DASHA-SYNC-MVR that depend on a common ratio σ 2 / nεB 4 .

Original: We ﬁx σ 2 / nεB ∈ { 10 4 , 10 5 } and
Revised: We ﬁx σ 2 / nεB ∈ { 10 4 , 10 5 } and

Original: K ∈ { 200 , 2000 } in Rand K compressors.
Revised: K ∈ { 200 , 2000 } in Rand K compressors.

Original: We consider real-sim dataset from LIBSVM splitted between 5 nodes.
Revised: We consider real-sim dataset from LIBSVM splitted between 5 nodes.

Original: When we increase σ 2 / nεB from 10 4 to 10 5 , we implicitly decrease ε because other parameters are ﬁxed.
Revised: When we increase σ 2 / nεB from 10 4 to 10 5 , we implicitly decrease ε because other parameters are ﬁxed.

Original: In Figure 3, when ε is small, DASHA-MVR and DASHA-SYNC-MVR converge faster than VR-MARINA (online) .
Revised: In Figure 3, when ε is small, DASHA-MVR and DASHA-SYNC-MVR converge faster than VR-MARINA (online) .

Original: [Figure]
Revised: [Figure]

Original: A. D EEP NEURAL NETWORK TRAINING
Revised: A. D EEP NEURAL NETWORK TRAINING

Original: Finally, we test our algorithms on an image recognition task, CIFAR10 (Krizhevsky et al., 2009), with the ResNet-18 (He et al., 2016) deep neural network (the number of parameters d ≈ 10 7 ).
Revised: Finally, we test our algorithms on an image recognition task, CIFAR10 (Krizhevsky et al., 2009), with the ResNet-18 (He et al., 2016) deep neural network (the number of parameters d ≈ 10 7 ).

Original: We split CIFAR10 among 5 nodes, and take K ≈ 2 · 10 6 in Rand K .
Revised: We split CIFAR10 among 5 nodes, and take K ≈ 2 · 10 6 in Rand K .

Original: In all methods we ﬁnetune two parameters: step size γ ∈ { 0 . 05 , 0 . 01 , 0 . 005 , 0 . 001 } and ratio σ 2 / nεB ∈ { 2 , 10 , 20 , 100 } .
Revised: In all methods we ﬁnetune two parameters: step size γ ∈ { 0 . 05 , 0 . 01 , 0 . 005 , 0 . 001 } and ratio σ 2 / nεB ∈ { 2 , 10 , 20 , 100 } .

Original: Moreover, we trained the neural network with SGD without compression as a baseline, with step size γ ∈ { 1 . 0 , 0 .
Revised: Moreover, we trained the neural network with SGD without compression as a baseline, with step size γ ∈ { 1 . 0 , 0 .

Original: All nodes have batch size B = 25 .
Revised: All nodes have batch size B = 25 .

Original: Results are provided in Figure 4.
Revised: Results are provided in Figure 4.

Original: We see that DASHA-MVR converges signiﬁcantly faster than other algorithms in the terms of communication complexity.
Revised: We see that DASHA-MVR converges signiﬁcantly faster than other algorithms in the terms of communication complexity.

Original: Moreover, DASHA-SYNC-MVR works better than VR-MARINA (online) and SGD .
Revised: Moreover, DASHA-SYNC-MVR works better than VR-MARINA (online) and SGD .

Original: [Figure]
Revised: [Figure]

Original: B E XPERIMENTS D ETAILS
Revised: B E XPERIMENTS D ETAILS

Original: The code was written in Python 3.6.8 using PyTorch 1.9 (Paszke et al., 2019).
Revised: The code was written in Python 3.6.8 using PyTorch 1.9 (Paszke et al., 2019).

Original: A distributed environment was emulated on a machine with Intel(R)
Revised: A distributed environment was emulated on a machine with Intel(R)

Original: Xeon(R)
Revised: Xeon(R)

Original: Gold 6226R CPU @
Revised: Gold 6226R CPU @

Original: 2.90GHz and 64 cores.
Revised: 2.90GHz and 64 cores.

Original: Deep learning experiments were conducted with NVIDIA A100 GPU with 40GB memory (each deep learning experiment uses at most 5GB of this memory).
Revised: Deep learning experiments were conducted with NVIDIA A100 GPU with 40GB memory (each deep learning experiment uses at most 5GB of this memory).

Original: When the number of nodes n does not divide the number of samples N in a dataset, we randomly ignore N mod n samples from a dataset (up to 4 when n = 5 ).
Revised: When the number of nodes n does not divide the number of samples N in a dataset, we randomly ignore N mod n samples from a dataset (up to 4 when n = 5 ).

Original: C D ESCRIPTION OF DASHA-SYNC-MVR
Revised: C D ESCRIPTION OF DASHA-SYNC-MVR

Original: In this section, we provide a description of DASHA-SYNC-MVR (see Algorithm 2).
Revised: In this section, we provide a description of DASHA-SYNC-MVR (see Algorithm 2).

Original: This algorithm is closely related to DASHA-MVR (Algorithm 1), but DASHA-SYNC-MVR synchronizes all nodes with some probability p .
Revised: This algorithm is closely related to DASHA-MVR (Algorithm 1), but DASHA-SYNC-MVR synchronizes all nodes with some probability p .

Original: This synchronization procedure enabled us to ﬁx the convergence rate suboptimality of DASHA-MVR w.r.t. ω .
Revised: This synchronization procedure enabled us to ﬁx the convergence rate suboptimality of DASHA-MVR w.r.t. ω .

Original: 1: Input: starting point x 0 ∈ R d , stepsize γ > 0 , momentum a ∈ (0 , 1] , probability p ∈ (0 , 1] , batch size B (cid:48) , number of iterations T ≥ 1 .
Revised: 1: Input: starting point x 0 ∈ R d , stepsize γ > 0 , momentum a ∈ (0 , 1] , probability p ∈ (0 , 1] , batch size B (cid:48) , number of iterations T ≥ 1 .

Original: 2: Initialize g 0 i , h 

Revised: 2: Initialize g 0 i , h

Original: i on the nodes and g 0 = 1 n (cid:80) ni
Revised: i on the nodes and g 0 = 1 n (cid:80) ni

Original: =1 g 0 i on the server 3: for t
Revised: =1 g 0 i on the server 3: for t

Original: = do 4:
Revised: = do 4:

Original: 5: c probability p 6:
Revised: 5: c probability p 6:

Original: Broadcast to nodes 7: for i = 1 , . .
Revised: Broadcast to nodes 7: for i = 1 , . .

Original: . , n in parallel do 8: if c t +1 = 1 then 9: h t
Revised: . , n in parallel do 8: if c t +1 = 1 then 9: h t

Original: +1 i = 1 B (cid:48) (cid:80) B (cid:48) k
Revised: +1 i = 1 B (cid:48) (cid:80) B (cid:48) k

Original: =1 ∇ f i ( x t +1 ; ξ t +1 ik ) 10: m t +1 i =
Revised: =1 ∇ f i ( x t +1 ; ξ t +1 ik ) 10: m t +1 i =

Original: g t +1 i =
Revised: g t +1 i =

Original: h t +1 i 11: else 12: h t +1 i =
Revised: h t +1 i 11: else 12: h t +1 i =

Original: 1 B (cid:80)
Revised: 1 B (cid:80)

Original: Bj =1 ∇ f i ( x t +1 ; ξ t +1 ij )
Revised: Bj =1 ∇ f i ( x t +1 ; ξ t +1 ij )

Original: + h ti − 1 B (cid:80)
Revised: + h ti − 1 B (cid:80)

Original: Bj =1 ∇ f i ( x t ; ξ t +1 ij ) 13: m t
Revised: Bj =1 ∇ f i ( x t ; ξ t +1 ij ) 13: m t

Original: +1 i
Revised: +1 i

Original: =
Revised: =

Original: C i (cid:0) h t
Revised: C i (cid:0) h t

Original: +1 i − h ti − a
Revised: +1 i − h ti − a

Original: ( g ti − h ti ) (cid:1) 14: g t +1 i
Revised: ( g ti − h ti ) (cid:1) 14: g t +1 i

Original: = g ti + m t
Revised: = g ti + m t

Original: +

Revised: +

Original: i 15: end if 16:
Revised: i 15: end if 16:

Original: Send m t
Revised: Send m t

Original: +1 i to the server 17: end for 18: if c t +1 = 1 then 19: g t +1 = 1 n (cid:80) ni
Revised: +1 i to the server 17: end for 18: if c t +1 = 1 then 19: g t +1 = 1 n (cid:80) ni

Original: =1 m t
Revised: =1 m t

Original: +

Revised: +

Original: i 20: else 21: g t +1 =
Revised: i 20: else 21: g t +1 =

Original: g t + 1 n (cid:80) ni
Revised: g t + 1 n (cid:80) ni

Original: =1 m t
Revised: =1 m t

Original: +

Revised: +

Original: i 22: end if 23: end for 24: Output: ˆ x T chosen uniformly at random from { x t } T − 1 k =
Revised: i 22: end if 23: end for 24: Output: ˆ x T chosen uniformly at random from { x t } T − 1 k =

Original: D P ARTIAL P ARTICIPATION
Revised: D P ARTIAL P ARTICIPATION

Original: A partial participation mechanism, important for federated learning applications, can be easily implemented in DASHA .
Revised: A partial participation mechanism, important for federated learning applications, can be easily implemented in DASHA .

Original: Let us assume that the i th node either participates in a communication round with probability p (cid:48) , or sends nothing.
Revised: Let us assume that the i th node either participates in a communication round with probability p (cid:48) , or sends nothing.

Original: From the view of unbiased compressors, it can mean that instead of using a compressor C , we have use the following new stochastic mapping C p (cid:48) :
Revised: From the view of unbiased compressors, it can mean that instead of using a compressor C , we have use the following new stochastic mapping C p (cid:48) :

Original: [Equation]
Revised: [Equation]

Original: The following simple result states that the new mapping C p (cid:48) is also an unbiased compressor, which means that our theory applies to this choice as well.
Revised: The following simple result states that the new mapping C p (cid:48) is also an unbiased compressor, which means that our theory applies to this choice as well.

Original: Theorem D.1.
Revised: Theorem D.1.

Original: If C ∈ U ( ω ) , then C p (cid:48) ∈ U (cid:16) ω +1 p (cid:48) − 1 (cid:17) .
Revised: If C ∈ U ( ω ) , then C p (cid:48) ∈ U (cid:16) ω +1 p (cid:48) − 1 (cid:17) .

Original: In the case of partial participation, all theorems from Section 6 will hold with ω replaced by ( ω +1) / p (cid:48) − 1 .
Revised: In the case of partial participation, all theorems from Section 6 will hold with ω replaced by ( ω +1) / p (cid:48) − 1 .

Original: E A UXILIARY F ACTS
Revised: E A UXILIARY F ACTS

Original: In this section, we recall well–known auxiliary facts that we use in the proofs.
Revised: In this section, we recall well–known auxiliary facts that we use in the proofs.

Original: For all x
Revised: For all x

Original: , y ∈
Revised: , y ∈

Original: R d , we have
Revised: R d , we have

Original: [Equation]
Revised: [Equation]

Original: Let us
Revised: Let us

Original: take
Revised: take

Original: a
Revised: a

Original: random
Revised: random

Original: vector
Revised: vector

Original: ξ ∈ R d
Revised: ξ ∈ R d

Original: , then
Revised: , then

Original: [Equation]
Revised: [Equation]

Original: F C OMPRESSORS F ACTS
Revised: F C OMPRESSORS F ACTS

Original: Deﬁnition F.1.
Revised: Deﬁnition F.1.

Original: Let us take a random subset S from [ d ] , | S | =
Revised: Let us take a random subset S from [ d ] , | S | =

Original: K, K ∈ [ d ] .
Revised: K, K ∈ [ d ] .

Original: We say that a stochastic mapping C : R d → R d is Rand K if
Revised: We say that a stochastic mapping C : R d → R d is Rand K if

Original: [Equation]
Revised: [Equation]

Original: where { e i } di =1 is the standard unit basis.
Revised: where { e i } di =1 is the standard unit basis.

Original: Informally, Rand K randomly keeps K coordinates and zeroes out the other.
Revised: Informally, Rand K randomly keeps K coordinates and zeroes out the other.

Original: Theorem F.2.
Revised: Theorem F.2.

Original: If C is Rand K , then C ∈ U (cid:0) dk − 1 (cid:1) .
Revised: If C is Rand K , then C ∈ U (cid:0) dk − 1 (cid:1) .

Original: See the proof in (Beznosikov et al., 2020).
Revised: See the proof in (Beznosikov et al., 2020).

Original: In the next theorem, we show that C p (cid:48)
Revised: In the next theorem, we show that C p (cid:48)

Original: ( x ) from (13) is an unbiased compressor.
Revised: ( x ) from (13) is an unbiased compressor.

Original: Theorem D.1.
Revised: Theorem D.1.

Original: If C ∈ U ( ω ) , then C p (cid:48) ∈ U (cid:16) ω +1 p (cid:48) − 1 (cid:17) .
Revised: If C ∈ U ( ω ) , then C p (cid:48) ∈ U (cid:16) ω +1 p (cid:48) − 1 (cid:17) .

Original: Proof.
Revised: Proof.

Original: First, we proof the unbiasedness:
Revised: First, we proof the unbiasedness:

Original: [Equation]
Revised: [Equation]

Original: Next, we get a bound for the variance:
Revised: Next, we get a bound for the variance:

Original: [Equation]
Revised: [Equation]

Original: From C ∈ U ( ω )
Revised: From C ∈ U ( ω )

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: G
Revised: G

Original: P OLYAK -Ł OJASIEWICZ C ONDITION
Revised: P OLYAK -Ł OJASIEWICZ C ONDITION

Original: In this section, we discuss our convergence rates under the (Polyak-Łojasiewicz) PŁ-condition:
Revised: In this section, we discuss our convergence rates under the (Polyak-Łojasiewicz) PŁ-condition:

Original: Assumption G.1.
Revised: Assumption G.1.

Original: A functions f satisfy (Polyak-Łojasiewicz) PŁ-condition:
Revised: A functions f satisfy (Polyak-Łojasiewicz) PŁ-condition:

Original: [Equation]
Revised: [Equation]

Original: where f ∗ = inf x ∈ R d f ( x ) >
Revised: where f ∗ = inf x ∈ R d f ( x ) >

Original: −∞ .
Revised: −∞ .

Original: Here we use a different notion of an ε -solution: it is a (random) point (cid:98) x , such that E [ f ( (cid:98) x )] − f ∗ ≤ ε.
Revised: Here we use a different notion of an ε -solution: it is a (random) point (cid:98) x , such that E [ f ( (cid:98) x )] − f ∗ ≤ ε.

Original: Under this assumption, Algorithm 1 achieves a linear convergence rate
Revised: Under this assumption, Algorithm 1 achieves a linear convergence rate

Original: O (ln ( 1 / ε )) instead of a sublinear convergence rate O ( 1 / ε ) in the gradient and ﬁnite-sum settings.
Revised: O (ln ( 1 / ε )) instead of a sublinear convergence rate O ( 1 / ε ) in the gradient and ﬁnite-sum settings.

Original: Moreover, in the stochastic setting, Algorithms 1 and 2 also improve dependence on ε .
Revised: Moreover, in the stochastic setting, Algorithms 1 and 2 also improve dependence on ε .

Original: Related Theorems I.9, I.12, I.15 and I. are stated in Appendix I. Note that in the ﬁnite-sum and stochastic settings, Theorems I.12 and I. provide new SOTA theoretical convergence rates (see Table 2).
Revised: Related Theorems I.9, I.12, I.15 and I. are stated in Appendix I. Note that in the ﬁnite-sum and stochastic settings, Theorems I.12 and I. provide new SOTA theoretical convergence rates (see Table 2).

Original: H I NTUITION B EHIND DASHA
Revised: H I NTUITION B EHIND DASHA

Original: In this section, we want to outline an intuition of differences between the proofs of DASHA and
Revised: In this section, we want to outline an intuition of differences between the proofs of DASHA and

Original: MARINA that helps us to improve the convergence rates.
Revised: MARINA that helps us to improve the convergence rates.

Original: H. D IFFERENT S OURCES OF C ONTRACTIONS
Revised: H. D IFFERENT S OURCES OF C ONTRACTIONS

Original: In both algorithms the proofs analyze E C (cid:104)(cid:13)(cid:13)
Revised: In both algorithms the proofs analyze E C (cid:104)(cid:13)(cid:13)

Original: g t +1 − ∇ f ( x k +1 )
Revised: g t +1 − ∇ f ( x k +1 )

Original: (cid:13)(cid:13) 2 (cid:105) , a norm of a difference between a gradient ∇ f ( x k +1 ) and a gradient estimator g t
Revised: (cid:13)(cid:13) 2 (cid:105) , a norm of a difference between a gradient ∇ f ( x k +1 ) and a gradient estimator g t

Original: +1 .
Revised: +1 .

Original: For simplicity, we assume that n = 1 , then for
Revised: For simplicity, we assume that n = 1 , then for

Original: MARINA , we have
Revised: MARINA , we have

Original: [Equation]
Revised: [Equation]

Original: In order to get a contraction, i.e., E C (cid:104)(cid:13)(cid:13) g
Revised: In order to get a contraction, i.e., E C (cid:104)(cid:13)(cid:13) g

Original: [Equation]
Revised: [Equation]

Original: ∇ f
Revised: ∇ f

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: MARINA has to send a full gradient ∇ f ( x k +1 ) with the probability p > 0 .
Revised: MARINA has to send a full gradient ∇ f ( x k +1 ) with the probability p > 0 .

Original: Now, let us look how we get a contraction in DASHA :
Revised: Now, let us look how we get a contraction in DASHA :

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: ( x k +1 )
Revised: ( x k +1 )

Original: − ∇ f ( x k ) (cid:13)(cid:13)≤ (1 − a ) (cid:13)(cid:13)
Revised: − ∇ f ( x k ) (cid:13)(cid:13)≤ (1 − a ) (cid:13)(cid:13)

Original: g t − ∇ f ( x k ) (cid:13)(cid:13) 2 + 2 ω
Revised: g t − ∇ f ( x k ) (cid:13)(cid:13) 2 + 2 ω

Original: (cid:13)(cid:13) ∇ f ( x k +1 )
Revised: (cid:13)(cid:13) ∇ f ( x k +1 )

Original: − ∇ f ( x k ) (cid:13)(cid:13) 2 .
Revised: − ∇ f ( x k ) (cid:13)(cid:13) 2 .

Original: In the last inequality we use that a ≤ 1 / 2 ω +1 .
Revised: In the last inequality we use that a ≤ 1 / 2 ω +1 .

Original: On can see that we get exactly the same recursion and contraction.
Revised: On can see that we get exactly the same recursion and contraction.

Original: The source of contraction is a correction − a ( g t − ∇ f ( x k )) inside the compressor C .
Revised: The source of contraction is a correction − a ( g t − ∇ f ( x k )) inside the compressor C .

Original: H. T HE S OURCE OF I MPROVEMENTS IN THE C ONVERGENCE R ATES
Revised: H. T HE S OURCE OF I MPROVEMENTS IN THE C ONVERGENCE R ATES

Original: Let us brieﬂy explain why we get the improvements in the convergence rates of DASHA in the ﬁnite-sum setting.
Revised: Let us brieﬂy explain why we get the improvements in the convergence rates of DASHA in the ﬁnite-sum setting.

Original: The same intuitions implies to the stochastic setting.
Revised: The same intuitions implies to the stochastic setting.

Original: In DASHA , we reduce variances from the compressors C and the random sampling I tj separately: we have two different control variables h t
Revised: In DASHA , we reduce variances from the compressors C and the random sampling I tj separately: we have two different control variables h t

Original: i and g t i , two different parameters the probability p and the momentum a. For simplicity, let us assume that the number of nodes n = 1 .
Revised: i and g t i , two different parameters the probability p and the momentum a. For simplicity, let us assume that the number of nodes n = 1 .

Original: Let us consider
Revised: Let us consider

Original: a
Revised: a

Original: Lyapunov function from our proofs:
Revised: Lyapunov function from our proofs:

Original: [Equation]
Revised: [Equation]

Original: In contrast, MARINA ( VR-MARINA ) has only one control variable g ti and on parameter p.
Revised: In contrast, MARINA ( VR-MARINA ) has only one control variable g ti and on parameter p.

Original: A Lyapunov function of MARINA is
Revised: A Lyapunov function of MARINA is

Original: [Equation]
Revised: [Equation]

Original: MARINA has a simpler Lyapunov function that leads to a suboptimal convergence rate.
Revised: MARINA has a simpler Lyapunov function that leads to a suboptimal convergence rate.

Original: Intuitively, having one control variable and one parameter is not enough to reduce variances from two different sources of randomness.
Revised: Intuitively, having one control variable and one parameter is not enough to reduce variances from two different sources of randomness.

Original: So in DASHA , the parameter p =
Revised: So in DASHA , the parameter p =

Original: Bm + B , while in MARINA p = min (cid:110)
Revised: Bm + B , while in MARINA p = min (cid:110)

Original: Bm + B , ζ C d (cid:111) , because the parameter p of MARINA helps to reduce the variance from the compressors C .
Revised: Bm + B , ζ C d (cid:111) , because the parameter p of MARINA helps to reduce the variance from the compressors C .

Original: I T HEOREMS WITH P ROOFS
Revised: I T HEOREMS WITH P ROOFS

Original: Lemma I.1.
Revised: Lemma I.1.

Original: Suppose that Assumption 5.2 holds and let x t +1 = x
Revised: Suppose that Assumption 5.2 holds and let x t +1 = x

Original: t − γg t .
Revised: t − γg t .

Original: Then for any g t ∈ R d and γ > 0 , we have
Revised: Then for any g t ∈ R d and γ > 0 , we have

Original: [Equation]
Revised: [Equation]

Original: The proof of Lemma I.1 is provided in (Li et al., 2021a).
Revised: The proof of Lemma I.1 is provided in (Li et al., 2021a).

Original: There are two different sources of randomness in Algorithm 1: the ﬁrst one from vectors { h t +1 i } ni =1 and the second one from compressors {C i } ni =1 .
Revised: There are two different sources of randomness in Algorithm 1: the ﬁrst one from vectors { h t +1 i } ni =1 and the second one from compressors {C i } ni =1 .

Original: In this section, we deﬁne E h [ · ] and E C [ · ] to be conditional expectations w.r.t.
Revised: In this section, we deﬁne E h [ · ] and E C [ · ] to be conditional expectations w.r.t.

Original: { h t +1 i } ni =1 and {C i } ni =1 , accordingly, conditioned on all previous randomness.
Revised: { h t +1 i } ni =1 and {C i } ni =1 , accordingly, conditioned on all previous randomness.

Original: Lemma I.2.
Revised: Lemma I.2.

Original: Suppose that Assumption 1.2 holds and let us consider sequences g t
Revised: Suppose that Assumption 1.2 holds and let us consider sequences g t

Original: +1 i and h t
Revised: +1 i and h t

Original: +1 i from
Revised: +1 i from

Original: Algorithm 1, then
Revised: Algorithm 1, then

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: First, we estimate E C (cid:104)(cid:13)(cid:13)
Revised: First, we estimate E C (cid:104)(cid:13)(cid:13)

Original: g t
Revised: g t

Original: [Equation]
Revised: [Equation]

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: Using the independence of compressors and (4), we get
Revised: Using the independence of compressors and (4), we get

Original: E
Revised: E

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Analogously
Revised: Analogously

Original: [Table]
Revised: [Table]

Original: we can get the bound
Revised: we can get the bound

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Lemma I.3.
Revised: Lemma I.3.

Original: Suppose that Assumptions 5.2 and 1.2 hold and let us take a = 1 / (2 ω + 1) , then
Revised: Suppose that Assumptions 5.2 and 1.2 hold and let us take a = 1 / (2 ω + 1) , then

Original: E (cid:2) f
Revised: E (cid:2) f

Original: ( x t +1 ) (cid:3) + γ (2 ω + 1) E (cid:104)(cid:13)(cid:13) g t +1 − h t +1 (cid:13)(cid:13) 2 (cid:105) + 2 γω n E (cid:34)n n (cid:88) i =
Revised: ( x t +1 ) (cid:3) + γ (2 ω + 1) E (cid:104)(cid:13)(cid:13) g t +1 − h t +1 (cid:13)(cid:13) 2 (cid:105) + 2 γω n E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g t +

Revised: g t +

Original: i − h t
Revised: i − h t

Original: +1 i (cid:13)(cid:13) 2 (cid:35) ≤ E (cid:20) f
Revised: +1 i (cid:13)(cid:13) 2 (cid:35) ≤ E (cid:20) f

Original: ( x t ) − γ 2 (cid:13)(cid:13) ∇ f ( x t ) (cid:13)(cid:13) 2 − (cid:18) 1 2 γ − L 2 (cid:19) (cid:13)(cid:13)
Revised: ( x t ) − γ 2 (cid:13)(cid:13) ∇ f ( x t ) (cid:13)(cid:13) 2 − (cid:18) 1 2 γ − L 2 (cid:19) (cid:13)(cid:13)

Original: x
Revised: x

Original: t +1 − x t (cid:13)(cid:13) 2 + γ (cid:13)(cid:13)
Revised: t +1 − x t (cid:13)(cid:13) 2 + γ (cid:13)(cid:13)

Original: h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:21) + γ (2 ω + 1) E (cid:104)(cid:13)(cid:13) g t − h t (cid:13)(cid:13) 2 (cid:105) + 2 γω n E (cid:34)n n (cid:88) i =
Revised: h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:21) + γ (2 ω + 1) E (cid:104)(cid:13)(cid:13) g t − h t (cid:13)(cid:13) 2 (cid:105) + 2 γω n E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g ti − h ti (cid:13)(cid:13)(cid:35) + 8 γω (2 ω + 1) n
Revised: g ti − h ti (cid:13)(cid:13)(cid:35) + 8 γω (2 ω + 1) n

Original: E (cid:34)n n (cid:88) i =
Revised: E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: h t
Revised: h t

Original: +

Revised: +

Original: i − h ti (cid:13)(cid:13)(cid:35) .
Revised: i − h ti (cid:13)(cid:13)(cid:35) .

Original: Proof.
Revised: Proof.

Original: Due to Lemma I.1 and the update step from Line 4 in Algorithm 1, we have
Revised: Due to Lemma I.1 and the update step from Line 4 in Algorithm 1, we have

Original: E (cid:2) f
Revised: E (cid:2) f

Original: ( x t +1 ) (cid:3) ≤ E (cid:20) f ( x t ) − γ 2 (cid:13)(cid:13)
Revised: ( x t +1 ) (cid:3) ≤ E (cid:20) f ( x t ) − γ 2 (cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: In the last inequality we use Jensen’s inequality (14).
Revised: In the last inequality we use Jensen’s inequality (14).

Original: Let us ﬁx some constants κ, η ∈ [0 , ∞ ) that we will deﬁne later.
Revised: Let us ﬁx some constants κ, η ∈ [0 , ∞ ) that we will deﬁne later.

Original: Combining bounds (20), (18), (19) and using the law of total expectation, we get
Revised: Combining bounds (20), (18), (19) and using the law of total expectation, we get

Original: E (cid:2) f
Revised: E (cid:2) f

Original: ( x t +1 ) (cid:3) + κ E (cid:104)(cid:13)(cid:13) g t
Revised: ( x t +1 ) (cid:3) + κ E (cid:104)(cid:13)(cid:13) g t

Original: +
Revised: +

Original: [Table] [Table]
Revised: [Table] [Table]

Original: n n (cid:88) i =
Revised: n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g ti − h ti (cid:13)(cid:13)(cid:35) + (cid:18) 2 κω n + 2 ηω (cid:19)
Revised: g ti − h ti (cid:13)(cid:13)(cid:35) + (cid:18) 2 κω n + 2 ηω (cid:19)

Original: E (cid:34)n n (cid:88) i =
Revised: E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: h t
Revised: h t

Original: +

Revised: +

Original: i − h ti (cid:13)(cid:13)(cid:35) .
Revised: i − h ti (cid:13)(cid:13)(cid:35) .

Original: Now, by taking κ = γ a , we can see that γ + κ (1 − a ) 2 ≤ κ, and thus
Revised: Now, by taking κ = γ a , we can see that γ + κ (1 − a ) 2 ≤ κ, and thus

Original: E (cid:2) f
Revised: E (cid:2) f

Original: ( x t +1 ) (cid:3) + γ a E (cid:104)(cid:13)(cid:13)
Revised: ( x t +1 ) (cid:3) + γ a E (cid:104)(cid:13)(cid:13)

Original: g t +1 − h t +1 (cid:13)(cid:13) 2 (cid:105) + η E (cid:34)n n (cid:88) i =
Revised: g t +1 − h t +1 (cid:13)(cid:13) 2 (cid:105) + η E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g t +

Revised: g t +

Original: i − h t
Revised: i − h t

Original: +1 i (cid:13)(cid:13) 2 (cid:35) ≤ E (cid:20) f
Revised: +1 i (cid:13)(cid:13) 2 (cid:35) ≤ E (cid:20) f

Original: ( x t ) − γ 2 (cid:13)(cid:13) ∇ f ( x t ) (cid:13)(cid:13) 2 − (cid:18) 1 2 γ − L 2 (cid:19) (cid:13)(cid:13)
Revised: ( x t ) − γ 2 (cid:13)(cid:13) ∇ f ( x t ) (cid:13)(cid:13) 2 − (cid:18) 1 2 γ − L 2 (cid:19) (cid:13)(cid:13)

Original: x
Revised: x

Original: t +1 − x t (cid:13)(cid:13) 2 + γ (cid:13)(cid:13)
Revised: t +1 − x t (cid:13)(cid:13) 2 + γ (cid:13)(cid:13)

Original: h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:21) + γ a E (cid:104)(cid:13)(cid:13) g
Revised: h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:21) + γ a E (cid:104)(cid:13)(cid:13) g

Original: [Equation]
Revised: [Equation]

Original: − h t (cid:13)(cid:13) 2 (cid:105) + (cid:18) 2 γaω n + η (cid:16)
Revised: − h t (cid:13)(cid:13) 2 (cid:105) + (cid:18) 2 γaω n + η (cid:16)

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: ω
Revised: ω

Original: + (1 − a ) 2 (cid:17)(cid:19) E (cid:34)n n (cid:88)
Revised: + (1 − a ) 2 (cid:17)(cid:19) E (cid:34)n n (cid:88)

Original: i =
Revised: i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g ti − h ti (cid:13)(cid:13)(cid:35) + (cid:18) 2 γω an + 2 ηω (cid:19)
Revised: g ti − h ti (cid:13)(cid:13)(cid:35) + (cid:18) 2 γω an + 2 ηω (cid:19)

Original: E (cid:34)n n (cid:88) i =
Revised: E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: h t
Revised: h t

Original: +

Revised: +

Original: i − h ti (cid:13)(cid:13)(cid:35) .
Revised: i − h ti (cid:13)(cid:13)(cid:35) .

Original: Next, by taking η = 2 γωn and considering the choice of a , one can show that (cid:16) 2 γaωn + η (cid:16) 2 a 2 ω + (1 − a ) 2 (cid:17)(cid:17) ≤ η.
Revised: Next, by taking η = 2 γωn and considering the choice of a , one can show that (cid:16) 2 γaωn + η (cid:16) 2 a 2 ω + (1 − a ) 2 (cid:17)(cid:17) ≤ η.

Original: Thus
Revised: Thus

Original: E (cid:2) f
Revised: E (cid:2) f

Original: ( x t +1 ) (cid:3) + γ (2 ω + 1) E (cid:104)(cid:13)(cid:13) g t +1 − h t +1 (cid:13)(cid:13) 2 (cid:105) + 2 γω n E (cid:34)n n (cid:88) i =
Revised: ( x t +1 ) (cid:3) + γ (2 ω + 1) E (cid:104)(cid:13)(cid:13) g t +1 − h t +1 (cid:13)(cid:13) 2 (cid:105) + 2 γω n E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g t +

Revised: g t +

Original: i − h t
Revised: i − h t

Original: +1 i (cid:13)(cid:13) 2 (cid:35) ≤ E (cid:20) f
Revised: +1 i (cid:13)(cid:13) 2 (cid:35) ≤ E (cid:20) f

Original: ( x t ) − γ 2 (cid:13)(cid:13) ∇ f ( x t ) (cid:13)(cid:13) 2 − (cid:18) 1 2 γ − L 2 (cid:19) (cid:13)(cid:13)
Revised: ( x t ) − γ 2 (cid:13)(cid:13) ∇ f ( x t ) (cid:13)(cid:13) 2 − (cid:18) 1 2 γ − L 2 (cid:19) (cid:13)(cid:13)

Original: x
Revised: x

Original: t +1 − x t (cid:13)(cid:13) 2 + γ (cid:13)(cid:13)
Revised: t +1 − x t (cid:13)(cid:13) 2 + γ (cid:13)(cid:13)

Original: h
Revised: h

Original: [Equation]
Revised: [Equation]

Original: − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:21) + γ (
Revised: − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:21) + γ (

Original: [Equation]
Revised: [Equation]

Original: + 1) E
Revised: + 1) E

Original: (cid:104)(cid:13)(cid:13) g t − h t (cid:13)(cid:13) 2 (cid:105) + 2 γω n E (cid:34)n n (cid:88) i =
Revised: (cid:104)(cid:13)(cid:13) g t − h t (cid:13)(cid:13) 2 (cid:105) + 2 γω n E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g ti − h ti (cid:13)(cid:13)(cid:35) + (cid:18) 2 γω (2 ω + 1) n + 4 γωn (cid:19)
Revised: g ti − h ti (cid:13)(cid:13)(cid:35) + (cid:18) 2 γω (2 ω + 1) n + 4 γωn (cid:19)

Original: E (cid:34)n n (cid:88) i =
Revised: E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: h t
Revised: h t

Original: +

Revised: +

Original: i − h ti (cid:13)(cid:13)(cid:35) ≤ E (cid:20) f
Revised: i − h ti (cid:13)(cid:13)(cid:35) ≤ E (cid:20) f

Original: ( x t ) − γ 2 (cid:13)(cid:13) ∇ f ( x t ) (cid:13)(cid:13) 2 − (cid:18) 1 2 γ − L 2 (cid:19) (cid:13)(cid:13)
Revised: ( x t ) − γ 2 (cid:13)(cid:13) ∇ f ( x t ) (cid:13)(cid:13) 2 − (cid:18) 1 2 γ − L 2 (cid:19) (cid:13)(cid:13)

Original: x
Revised: x

Original: t +1 − x t (cid:13)(cid:13) 2 + γ (cid:13)(cid:13)
Revised: t +1 − x t (cid:13)(cid:13) 2 + γ (cid:13)(cid:13)

Original: h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:21) + γ (2 ω + 1) E (cid:104)(cid:13)(cid:13) g t − h t (cid:13)(cid:13) 2 (cid:105) + 2 γω n E (cid:34)n n (cid:88) i =
Revised: h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:21) + γ (2 ω + 1) E (cid:104)(cid:13)(cid:13) g t − h t (cid:13)(cid:13) 2 (cid:105) + 2 γω n E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g ti − h ti (cid:13)(cid:13)(cid:35) + 8 γω (2 ω + 1) n
Revised: g ti − h ti (cid:13)(cid:13)(cid:35) + 8 γω (2 ω + 1) n

Original: E (cid:34)n n (cid:88) i =
Revised: E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: h t
Revised: h t

Original: +

Revised: +

Original: i − h ti (cid:13)(cid:13)(cid:35) .
Revised: i − h ti (cid:13)(cid:13)(cid:35) .

Original: The following lemma almost repeats the previous one.
Revised: The following lemma almost repeats the previous one.

Original: We will use it in the theorems with Assumption
Revised: We will use it in the theorems with Assumption

Original: G.1.
Revised: G.1.

Original: Lemma I.4.
Revised: Lemma I.4.

Original: Suppose that Assumptions 5.2, 1.2 and G.1 hold and let us take a = 1 / (2 ω + 1) and γ ≤ a 2 µ , then
Revised: Suppose that Assumptions 5.2, 1.2 and G.1 hold and let us take a = 1 / (2 ω + 1) and γ ≤ a 2 µ , then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Up to (21) we can follow the proof of Lemma I.3 to get
Revised: Up to (21) we can follow the proof of Lemma I.3 to get

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: h t
Revised: h t

Original: +

Revised: +

Original: i − h ti (cid:13)(cid:13)(cid:35) .
Revised: i − h ti (cid:13)(cid:13)(cid:35) .

Original: Now, by taking κ = 2 γa , we can see that γ + κ (1 − a ) 2 ≤ (cid:0) 1 − a 2 (cid:1) κ, and thus
Revised: Now, by taking κ = 2 γa , we can see that γ + κ (1 − a ) 2 ≤ (cid:0) 1 − a 2 (cid:1) κ, and thus

Original: E (cid:2) f
Revised: E (cid:2) f

Original: ( x t +1 ) (cid:3) + 2 γ a E (cid:104)(cid:13) (cid:13) g t
Revised: ( x t +1 ) (cid:3) + 2 γ a E (cid:104)(cid:13) (cid:13) g t

Original: +

Revised: +

Original: − h t +1 (cid:13) (cid:13) 2 (cid:105) + η E (cid:34)n n (cid:88)
Revised: − h t +1 (cid:13) (cid:13) 2 (cid:105) + η E (cid:34)n n (cid:88)

Original: i = (cid:13) (cid:13) g t
Revised: i = (cid:13) (cid:13) g t

Original: +1 i − h t
Revised: +1 i − h t

Original: +1 i (cid:13) (cid:13)(cid:35)
Revised: +1 i (cid:13) (cid:13)(cid:35)

Original: ≤ E (cid:20) f
Revised: ≤ E (cid:20) f

Original: ( x t ) − γ 2 (cid:13)(cid:13) ∇ f ( x t ) (cid:13)(cid:13) 2 − (cid:18) 1 2 γ − L 2 (cid:19) (cid:13)(cid:13)
Revised: ( x t ) − γ 2 (cid:13)(cid:13) ∇ f ( x t ) (cid:13)(cid:13) 2 − (cid:18) 1 2 γ − L 2 (cid:19) (cid:13)(cid:13)

Original: x
Revised: x

Original: t +1 − x t (cid:13)(cid:13) 2 + γ (cid:13)(cid:13)
Revised: t +1 − x t (cid:13)(cid:13) 2 + γ (cid:13)(cid:13)

Original: h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:21) + (cid:16) 1 − a 2 (cid:17) 2 γ a E (cid:104)(cid:13)(cid:13) g t − h t (cid:13)(cid:13) 2 (cid:105) + (cid:18) 4 γaω n + η (cid:16) 2 a 2 ω + (1 − a ) 2 (cid:17)(cid:19) E
Revised: h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:21) + (cid:16) 1 − a 2 (cid:17) 2 γ a E (cid:104)(cid:13)(cid:13) g t − h t (cid:13)(cid:13) 2 (cid:105) + (cid:18) 4 γaω n + η (cid:16) 2 a 2 ω + (1 − a ) 2 (cid:17)(cid:19) E

Original: (cid:34)n n (cid:88) i =
Revised: (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g ti − h ti (cid:13)(cid:13)(cid:35) + (cid:18) 4 γω an + 2 ηω (cid:19)
Revised: g ti − h ti (cid:13)(cid:13)(cid:35) + (cid:18) 4 γω an + 2 ηω (cid:19)

Original: E (cid:34)n n (cid:88) i = (cid:13) (cid:13) h t
Revised: E (cid:34)n n (cid:88) i = (cid:13) (cid:13) h t

Original: +

Revised: +

Original: i − h t
Revised: i − h t

Original: i (cid:13) (cid:13)(cid:35) .
Revised: i (cid:13) (cid:13)(cid:35) .

Original: Next, by taking η = 8 γωn and considering the choice of a , one can show that (cid:16) 4 γaωn + η
Revised: Next, by taking η = 8 γωn and considering the choice of a , one can show that (cid:16) 4 γaωn + η

Original: [Equation]
Revised: [Equation]

Original: a 2 ω + (1 − a ) 2 (cid:17)(cid:17) ≤ (cid:0) 1 − a 2 (cid:1) η.
Revised: a 2 ω + (1 − a ) 2 (cid:17)(cid:17) ≤ (cid:0) 1 − a 2 (cid:1) η.

Original: Thus
Revised: Thus

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Finally, the assumption γ ≤ a 2 µ implies an inequality 1 − a 2 ≤ 1 − γµ
Revised: Finally, the assumption γ ≤ a 2 µ implies an inequality 1 − a 2 ≤ 1 − γµ

Original: [Equation]
Revised: [Equation]

Original: Lemma I
Revised: Lemma I

Original: [Equation]
Revised: [Equation]

Original: Suppose that Assumption 5.1 holds and
Revised: Suppose that Assumption 5.1 holds and

Original: [Equation]
Revised: [Equation]

Original: where Ψ t is a sequence of numbers, Ψ t ≥ 0 for all t ∈ [ T ] , constant C ≥ 0 , and constant γ > 0 .
Revised: where Ψ t is a sequence of numbers, Ψ t ≥ 0 for all t ∈ [ T ] , constant C ≥ 0 , and constant γ > 0 .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: where a point (cid:98) x T is chosen uniformly from a set of points { x t } T − 1 t =0 .
Revised: where a point (cid:98) x T is chosen uniformly from a set of points { x t } T − 1 t =0 .

Original: Proof.
Revised: Proof.

Original: By unrolling (22) for t from 0 to T − 1 , we obtain
Revised: By unrolling (22) for t from 0 to T − 1 , we obtain

Original: [Equation]
Revised: [Equation]

Original: We subtract f ∗ , divide inequality by γT 2 , and take into account that f ( x ) ≥ f ∗ for all x ∈ R , and Ψ t ≥ 0 for all t ∈
Revised: We subtract f ∗ , divide inequality by γT 2 , and take into account that f ( x ) ≥ f ∗ for all x ∈ R , and Ψ t ≥ 0 for all t ∈

Original: [ T ] , to get the following inequality:
Revised: [ T ] , to get the following inequality:

Original: [Equation]
Revised: [Equation]

Original: It is left to consider the choice of a point (cid:98)
Revised: It is left to consider the choice of a point (cid:98)

Original: x T to complete the proof of the lemma.
Revised: x T to complete the proof of the lemma.

Original: Lemma I.6.
Revised: Lemma I.6.

Original: Suppose that Assumptions 5.1 and G.1 hold and
Revised: Suppose that Assumptions 5.1 and G.1 hold and

Original: [Equation]
Revised: [Equation]

Original: where Ψ t is a sequence of numbers, Ψ t ≥ 0 for all t ∈ [ T ] , constant C ≥ 0 , constant µ > 0 , and constant γ ∈ (0 , 1 /µ ) .
Revised: where Ψ t is a sequence of numbers, Ψ t ≥ 0 for all t ∈ [ T ] , constant C ≥ 0 , constant µ > 0 , and constant γ ∈ (0 , 1 /µ ) .

Original: Then
Revised: Then

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Proof.
Revised: Proof.

Original: We subtract f ∗ and use PŁ-condition (16) to get
Revised: We subtract f ∗ and use PŁ-condition (16) to get

Original: [Equation]
Revised: [Equation]

Original: Unrolling the inequality, we have
Revised: Unrolling the inequality, we have

Original: [Equation]
Revised: [Equation]

Original: It is left to note that Ψ t ≥ 0 for all t ∈ [ T ] .
Revised: It is left to note that Ψ t ≥ 0 for all t ∈ [ T ] .

Original: Lemma I.7.
Revised: Lemma I.7.

Original: If 0 < γ ≤ ( L + √ A ) − 1 , L > 0 , and A ≥ 0 , then
Revised: If 0 < γ ≤ ( L + √ A ) − 1 , L > 0 , and A ≥ 0 , then

Original: [Equation]
Revised: [Equation]

Original: It is easy to verify with a direct calculation.
Revised: It is easy to verify with a direct calculation.

Original: I.
Revised: I.

Original: C ASE OF DASHA
Revised: C ASE OF DASHA

Original: Despite the triviality of the following lemma, we provide it for consistency with Lemma I.14 and
Revised: Despite the triviality of the following lemma, we provide it for consistency with Lemma I.14 and

Original: Lemma I.11.
Revised: Lemma I.11.

Original: Lemma I.8.
Revised: Lemma I.8.

Original: Suppose that Assumption 5.3 holds.
Revised: Suppose that Assumption 5.3 holds.

Original: Assuming that h 0 i = ∇ f i ( x 0 ) for all i ∈ [ n ] , for h t +1 i from Algorithm 1 (DASHA) we have
Revised: Assuming that h 0 i = ∇ f i ( x 0 ) for all i ∈ [ n ] , for h t +1 i from Algorithm 1 (DASHA) we have

Original: [Equation]
Revised: [Equation]

Original: Theorem 6.1.
Revised: Theorem 6.1.

Original: Suppose that Assumptions 5.1, 5.2, 5.3 and 1.2 hold.
Revised: Suppose that Assumptions 5.1, 5.2, 5.3 and 1.2 hold.

Original: Let us take a = 1 / (2 ω + 1) and γ ≤ (cid:18)
Revised: Let us take a = 1 / (2 ω + 1) and γ ≤ (cid:18)

Original: [Equation]
Revised: [Equation]

Original: −, and h 

Revised: −, and h

Original: i = ∇ f i ( x 0 ) for all i ∈ [ n ] in Algorithm 1 (DASHA) , then
Revised: i = ∇ f i ( x 0 ) for all i ∈ [ n ] in Algorithm 1 (DASHA) , then

Original: [Equation]
Revised: [Equation]

Original: i =
Revised: i =

Original: (cid:13)(cid:13) g 

Revised: (cid:13)(cid:13) g

Original: i − ∇ f i ( x 0 ) (cid:13)(cid:13)(cid:33) (cid:35) .
Revised: i − ∇ f i ( x 0 ) (cid:13)(cid:13)(cid:33) (cid:35) .

Original: Proof.
Revised: Proof.

Original: Considering Lemma I.3, Lemma I.8, and the law of total expectation, we obtain
Revised: Considering Lemma I.3, Lemma I.8, and the law of total expectation, we obtain

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: t +
Revised: t +

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: (cid:34)
Revised: (cid:34)

Original: [Equation]
Revised: [Equation]

Original: ≤ E
Revised: ≤ E

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Using assumption about γ
Revised: Using assumption about γ

Original: [Equation]
Revised: [Equation]

Original: we can show that 12 γ − L 2 − 8 γω (2 ω +1) n (cid:98) L 2 ≥ 

Revised: we can show that 12 γ − L 2 − 8 γω (2 ω +1) n (cid:98) L 2 ≥

Original: (see Lemma I.7), thus
Revised: (see Lemma I.7), thus

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: ≤
Revised: ≤

Original: [Table]
Revised: [Table]

Original: In the view of Lemma I.5 with Ψ t = (2 ω + 1) E (cid:104) (cid:107) g t − h t (cid:107) 2 (cid:105) + 2 ωn E (cid:104) 1 n (cid:80) ni =1 (cid:107) g ti − h ti (cid:107) 2 (cid:105) we can conclude the proof.
Revised: In the view of Lemma I.5 with Ψ t = (2 ω + 1) E (cid:104) (cid:107) g t − h t (cid:107) 2 (cid:105) + 2 ωn E (cid:104) 1 n (cid:80) ni =1 (cid:107) g ti − h ti (cid:107) 2 (cid:105) we can conclude the proof.

Original: Corollary 6.2.
Revised: Corollary 6.2.

Original: Suppose that assumptions from Theorem 6.1 hold, and g 0 i
Revised: Suppose that assumptions from Theorem 6.1 hold, and g 0 i

Original: =
Revised: =

Original: h 

Revised: h

Original: i = ∇
Revised: i = ∇

Original: f i ( x 0 ) for all i ∈ [ n ] , then DASHA needs T :
Revised: f i ( x 0 ) for all i ∈ [ n ] , then DASHA needs T :

Original: = O
Revised: = O

Original: [Equation]
Revised: [Equation]

Original: communication rounds to get an ε -solution and the communication complexity is equal to O ( d + ζ C T ) , where ζ C is the expected density from Deﬁnition 1.3.
Revised: communication rounds to get an ε -solution and the communication complexity is equal to O ( d + ζ C T ) , where ζ C is the expected density from Deﬁnition 1.3.

Original: Proof.
Revised: Proof.

Original: The communication complexities can be easily derived using Theorem 6.1.
Revised: The communication complexities can be easily derived using Theorem 6.1.

Original: At each communication round of Algorithm 1, each node sends ζ C coordinates.
Revised: At each communication round of Algorithm 1, each node sends ζ C coordinates.

Original: In the view of g 0 i = ∇ f i ( x 0 ) for all i ∈ [ n ] , we additionally have to send d coordinates from the nodes to the server, thus the total communication complexity would be O ( d + ζ C T ) .
Revised: In the view of g 0 i = ∇ f i ( x 0 ) for all i ∈ [ n ] , we additionally have to send d coordinates from the nodes to the server, thus the total communication complexity would be O ( d + ζ C T ) .

Original: Corollary 6.3.
Revised: Corollary 6.3.

Original: Suppose that assumptions of Corollary 6.2 hold.
Revised: Suppose that assumptions of Corollary 6.2 hold.

Original: We take the unbiased compressor
Revised: We take the unbiased compressor

Original: Rand K with K = ζ C ≤ d/ √ n, then the communication complexity equals O (cid:18)
Revised: Rand K with K = ζ C ≤ d/ √ n, then the communication complexity equals O (cid:18)

Original: d + (cid:98) L ( f ( x 0 ) − f ∗ ) d ε √ n (cid:19) .
Revised: d + (cid:98) L ( f ( x 0 ) − f ∗ ) d ε √ n (cid:19) .

Original: Proof.
Revised: Proof.

Original: In the view of Theorem F.2, we have ω + 1 =
Revised: In the view of Theorem F.2, we have ω + 1 =

Original: d/K. Combining this and an inequality L ≤ (cid:98) L, the communication complexity equals
Revised: d/K. Combining this and an inequality L ≤ (cid:98) L, the communication complexity equals

Original: [Equation]
Revised: [Equation]

Original: I. C ASE OF DASHA UNDER PŁ- CONDITION
Revised: I. C ASE OF DASHA UNDER PŁ- CONDITION

Original: Theorem I.9.
Revised: Theorem I.9.

Original: Suppose that Assumption 5.1, 5.2, 5.3, 1.2 and G.1 hold.
Revised: Suppose that Assumption 5.1, 5.2, 5.3, 1.2 and G.1 hold.

Original: Let us take a = 1 / (2 ω + 1) , γ ≤ min (cid:40)(cid:18)
Revised: Let us take a = 1 / (2 ω + 1) , γ ≤ min (cid:40)(cid:18)

Original: L + (cid:113)
Revised: L + (cid:113)

Original: 40 ω (2 ω +1) n
Revised: 40 ω (2 ω +1) n

Original: (cid:98)
Revised: (cid:98)

Original: L (cid:19) −
Revised: L (cid:19) −

Original: [Table]
Revised: [Table]

Original: , a 2 µ (cid:41)
Revised: , a 2 µ (cid:41)

Original: , and
Revised: , and

Original: h
Revised: h

Original: 0 i =
Revised: 0 i =

Original: ∇ f i x
Revised: ∇ f i x

Original: ) for all i ∈ [ n ] in Algorithm
Revised: ) for all i ∈ [ n ] in Algorithm

Original: (DASHA) , then
Revised: (DASHA) , then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Considering Lemma I.
Revised: Considering Lemma I.

Original: [Equation]
Revised: [Equation]

Original: Lemma I.
Revised: Lemma I.

Original: [Equation]
Revised: [Equation]

Original: and the law of total expectation
Revised: and the law of total expectation

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: Using the assumption about γ, we can show that 12 γ − L 2 − 20 γω (2 ω +
Revised: Using the assumption about γ, we can show that 12 γ − L 2 − 20 γω (2 ω +

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: ≥ 0 (see Lemma I.7), thus
Revised: ≥ 0 (see Lemma I.7), thus

Original: [Equation]
Revised: [Equation]

Original: 2 (cid:35) ≤ E (cid:2) f
Revised: 2 (cid:35) ≤ E (cid:2) f

Original: ( x t ) (cid:3) − γ 2 E (cid:104)(cid:13)(cid:13) ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105)
Revised: ( x t ) (cid:3) − γ 2 E (cid:104)(cid:13)(cid:13) ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105)

Original: + (1 − γµ ) 2 γ (2 ω
Revised: + (1 − γµ ) 2 γ (2 ω

Original: + 1)E (cid:104)(cid:13)(cid:13) g t − h t (cid:13)(cid:13) 2 (cid:105) + (1 − γµ ) 8 γω n E (cid:34)n n (cid:88)
Revised: + 1)E (cid:104)(cid:13)(cid:13) g t − h t (cid:13)(cid:13) 2 (cid:105) + (1 − γµ ) 8 γω n E (cid:34)n n (cid:88)

Original: i =
Revised: i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g ti − h ti (cid:13)(cid:13)(cid:35) .
Revised: g ti − h ti (cid:13)(cid:13)(cid:35) .

Original: In the view of Lemma I.6 with Ψ t = 2(2 ω
Revised: In the view of Lemma I.6 with Ψ t = 2(2 ω

Original: + 1)E (cid:104) (cid:107) g t − h t (cid:107) 2 (cid:105) + 8 ωn E (cid:104) 1 n (cid:80) ni
Revised: + 1)E (cid:104) (cid:107) g t − h t (cid:107) 2 (cid:105) + 8 ωn E (cid:104) 1 n (cid:80) ni

Original: =1 (cid:107) g ti − h ti (cid:107) 2 (cid:105) we can conclude the proof.
Revised: =1 (cid:107) g ti − h ti (cid:107) 2 (cid:105) we can conclude the proof.

Original: We use (cid:101) O ( · ) , when we provide a bound up to logarithmic factors.
Revised: We use (cid:101) O ( · ) , when we provide a bound up to logarithmic factors.

Original: Corollary I.10.
Revised: Corollary I.10.

Original: Suppose that assumptions from Theorem I.9 hold, and g 0 i = 0 for all i ∈ [ n ] , then
Revised: Suppose that assumptions from Theorem I.9 hold, and g 0 i = 0 for all i ∈ [ n ] , then

Original: DASHA needs
Revised: DASHA needs

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: communication rounds to get an ε -solution and the communication complexity is equal to O ( ζ C T ) , where ζ C is the expected density from Deﬁnition 1.3.
Revised: communication rounds to get an ε -solution and the communication complexity is equal to O ( ζ C T ) , where ζ C is the expected density from Deﬁnition 1.3.

Original: Proof.
Revised: Proof.

Original: Clearly, using Theorem I.9, one can show that Algorithm 1 returns an ε -solution after (25) communication rounds.
Revised: Clearly, using Theorem I.9, one can show that Algorithm 1 returns an ε -solution after (25) communication rounds.

Original: At each communication round of Algorithm 1, each node sends ζ C coordinates, thus the total communication complexity would be O ( ζ C T ) per node.
Revised: At each communication round of Algorithm 1, each node sends ζ C coordinates, thus the total communication complexity would be O ( ζ C T ) per node.

Original: Unlike
Revised: Unlike

Original: Corollary 6.2, in this corollary, we can initialize g 

Revised: Corollary 6.2, in this corollary, we can initialize g

Original: i , for instance, with zeros because the corresponding initialization error Ψ 

Revised: i , for instance, with zeros because the corresponding initialization error Ψ

Original: from the proof of Theorem I.9 would be under the logarithm.
Revised: from the proof of Theorem I.9 would be under the logarithm.

Original: I. C ASE OF DASHA-PAGE
Revised: I. C ASE OF DASHA-PAGE

Original: Lemma I.11.
Revised: Lemma I.11.

Original: Suppose that Assumptions 5.3 and 5.4 hold.
Revised: Suppose that Assumptions 5.3 and 5.4 hold.

Original: For h t +1 i from Algorithm 1 (DASHA-PAGE) we have
Revised: For h t +1 i from Algorithm 1 (DASHA-PAGE) we have

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Proof.
Revised: Proof.

Original: Using the deﬁnition of h t +1 , we obtain
Revised: Using the deﬁnition of h t +1 , we obtain

Original: [Equation]
Revised: [Equation]

Original: 1 − ∇ f ( x t +1 )
Revised: 1 − ∇ f ( x t +1 )

Original: (cid:13)(cid:13) 2 (cid:105) =
Revised: (cid:13)(cid:13) 2 (cid:105) =

Original: (1 − p )
Revised: (1 − p )

Original: E h   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) h t + 1 n n (cid:88)
Revised: E h   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) h t + 1 n n (cid:88)

Original: i =
Revised: i =

Original: B (cid:88) j ∈
Revised: B (cid:88) j ∈

Original: I ti (cid:0) ∇ f ij ( x t +1 )
Revised: I ti (cid:0) ∇ f ij ( x t +1 )

Original: − ∇ f ij ( x t ) (cid:1) − ∇ f ( x t +1 ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2 
Revised: − ∇ f ij ( x t ) (cid:1) − ∇ f ( x t +1 ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2 

Original:  (15) =
Revised:  (15) =

Original: (1 − p )
Revised: (1 − p )

Original: E h   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 n n (cid:88)
Revised: E h   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 n n (cid:88)

Original: i =
Revised: i =

Original: B (cid:88) j ∈
Revised: B (cid:88) j ∈

Original: I ti (cid:0) ∇ f ij ( x t +1 )
Revised: I ti (cid:0) ∇ f ij ( x t +1 )

Original: − ∇ f ij ( x t ) (cid:1) − (cid:0) ∇ f ( x t +1 )
Revised: − ∇ f ij ( x t ) (cid:1) − (cid:0) ∇ f ( x t +1 )

Original: − ∇ f ( x t ) (cid:1) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2 
Revised: − ∇ f ( x t ) (cid:1) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2 

Original:  + (1 − p ) (cid:13)(cid:13)
Revised:  + (1 − p ) (cid:13)(cid:13)

Original: h t − ∇ f ( x t ) (cid:13)(cid:13) 2 .
Revised: h t − ∇ f ( x t ) (cid:13)(cid:13) 2 .

Original: From the unbiasedness and independence of mini-batch samples, we get
Revised: From the unbiasedness and independence of mini-batch samples, we get

Original: E h (cid:104)(cid:13)(cid:13) h t +1 − ∇ f ( x t +1 )
Revised: E h (cid:104)(cid:13)(cid:13) h t +1 − ∇ f ( x t +1 )

Original: (cid:13)(cid:13) 2 (cid:105) ≤ (1 − p ) n 2 Bn (cid:88) i =
Revised: (cid:13)(cid:13) 2 (cid:105) ≤ (1 − p ) n 2 Bn (cid:88) i =

Original: E h  (cid:88)
Revised: E h  (cid:88)

Original: j ∈
Revised: j ∈

Original: I ti (cid:13)(cid:13)(cid:0) ∇ f ij ( x t +1 )
Revised: I ti (cid:13)(cid:13)(cid:0) ∇ f ij ( x t +1 )

Original: − ∇ f ij ( x t ) (cid:1) − (cid:0) ∇ f i ( x t +1 )
Revised: − ∇ f ij ( x t ) (cid:1) − (cid:0) ∇ f i ( x t +1 )

Original: − ∇ f i ( x t ) (cid:1)(cid:13)(cid:13)  + (1 − p ) (cid:13) (cid:13) h t − ∇ f ( x t ) (cid:13)(cid:13)= (1 − p ) n 2 B n (cid:88) i =   1 m m (cid:88)
Revised: − ∇ f i ( x t ) (cid:1)(cid:13)(cid:13)  + (1 − p ) (cid:13) (cid:13) h t − ∇ f ( x t ) (cid:13)(cid:13)= (1 − p ) n 2 B n (cid:88) i =   1 m m (cid:88)

Original: j
Revised: j

Original: = (cid:13)(cid:13)(cid:0) ∇ f ij ( x t +1 ) − ∇ f ij ( x t ) (cid:1) − (cid:0) ∇ f i ( x t +1 )
Revised: = (cid:13)(cid:13)(cid:0) ∇ f ij ( x t +1 ) − ∇ f ij ( x t ) (cid:1) − (cid:0) ∇ f i ( x t +1 )

Original: − ∇ f i ( x t ) (cid:1)(cid:13)(cid:13)  + (1 − p ) (cid:13)(cid:13) h t − ∇ f ( x t ) (cid:13)(cid:13)≤ (1 − p ) n 2 B n (cid:88) i =   1 m m (cid:88) j
Revised: − ∇ f i ( x t ) (cid:1)(cid:13)(cid:13)  + (1 − p ) (cid:13)(cid:13) h t − ∇ f ( x t ) (cid:13)(cid:13)≤ (1 − p ) n 2 B n (cid:88) i =   1 m m (cid:88) j

Original: = (cid:13)(cid:13)
Revised: = (cid:13)(cid:13)

Original: ∇ f ij ( x t +1 ) − ∇ f ij ( x t ) (cid:13)(cid:13)  + (1 − p ) (cid:13) (cid:13) h t − ∇ f ( x t ) (cid:13) (cid:13)≤ (1 − p )
Revised: ∇ f ij ( x t +1 ) − ∇ f ij ( x t ) (cid:13)(cid:13)  + (1 − p ) (cid:13) (cid:13) h t − ∇ f ( x t ) (cid:13) (cid:13)≤ (1 − p )

Original: L 2max nB (cid:13)(cid:13) x
Revised: L 2max nB (cid:13)(cid:13) x

Original: t +1 − x t (cid:13)(cid:13) 

Revised: t +1 − x t (cid:13)(cid:13)

Original: + (1 − p ) (cid:13)(cid:13) h t − ∇ f ( x t ) (cid:13)(cid:13) 2 .
Revised: + (1 − p ) (cid:13)(cid:13) h t − ∇ f ( x t ) (cid:13)(cid:13) 2 .

Original: In the last inequality, we use Assumption 5.4.
Revised: In the last inequality, we use Assumption 5.4.

Original: Using the same reasoning, we have
Revised: Using the same reasoning, we have

Original: E h (cid:104)(cid:13)(cid:13) h t
Revised: E h (cid:104)(cid:13)(cid:13) h t

Original: +1 i − ∇ f i ( x t +1 )
Revised: +1 i − ∇ f i ( x t +1 )

Original: (cid:13)(cid:13) 2 (cid:105)
Revised: (cid:13)(cid:13) 2 (cid:105)

Original: [Equation]
Revised: [Equation]

Original: Finally
Revised: Finally

Original: [Equation]
Revised: [Equation]

Original: we consider the last ineqaulity of the lemma:
Revised: we consider the last ineqaulity of the lemma:

Original: [Equation]
Revised: [Equation]

Original: Using the unbiasedness and independence of the gradients, we obtain
Revised: Using the unbiasedness and independence of the gradients, we obtain

Original: [Equation]
Revised: [Equation]

Original: x t ) (cid:1) − (cid:0) ∇ f i ( x t +1 ) − ∇ f i ( x t ) (cid:1)(cid:13)(cid:13)  + (1 − p ) (cid:13)(cid:13) ∇ f i ( x t +1 )
Revised: x t ) (cid:1) − (cid:0) ∇ f i ( x t +1 ) − ∇ f i ( x t ) (cid:1)(cid:13)(cid:13)  + (1 − p ) (cid:13)(cid:13) ∇ f i ( x t +1 )

Original: − ∇ f i ( x t ) (cid:13)(cid:13)= p (cid:13)(cid:13) ∇ f i ( x t +1 )
Revised: − ∇ f i ( x t ) (cid:13)(cid:13)= p (cid:13)(cid:13) ∇ f i ( x t +1 )

Original: − h ti (cid:13)(cid:13)+ (1 − p )
Revised: − h ti (cid:13)(cid:13)+ (1 − p )

Original: B   1 m m (cid:88)
Revised: B   1 m m (cid:88)

Original: j = (cid:13)(cid:13)(cid:0) ∇ f ij ( x t +1 ) − ∇ f ij ( x t ) (cid:1) − (cid:0) ∇ f i ( x t +1 )
Revised: j = (cid:13)(cid:13)(cid:0) ∇ f ij ( x t +1 ) − ∇ f ij ( x t ) (cid:1) − (cid:0) ∇ f i ( x t +1 )

Original: − ∇ f i ( x t ) (cid:1)(cid:13)(cid:13)  + (1 − p ) (cid:13)(cid:13) ∇ f i ( x t +1 )
Revised: − ∇ f i ( x t ) (cid:1)(cid:13)(cid:13)  + (1 − p ) (cid:13)(cid:13) ∇ f i ( x t +1 )

Original: − ∇ f i ( x t ) (cid:13)(cid:13)≤ p (cid:13)(cid:13) ∇ f i ( x t +1 )
Revised: − ∇ f i ( x t ) (cid:13)(cid:13)≤ p (cid:13)(cid:13) ∇ f i ( x t +1 )

Original: − h ti (cid:13)(cid:13)+ (1 − p )
Revised: − h ti (cid:13)(cid:13)+ (1 − p )

Original: B   1 m m (cid:88)
Revised: B   1 m m (cid:88)

Original: j = (cid:13)(cid:13) ∇ f ij ( x t +1 )
Revised: j = (cid:13)(cid:13) ∇ f ij ( x t +1 )

Original: − ∇ f ij ( x t ) (cid:13)(cid:13)  + (1 − p ) (cid:13)(cid:13) ∇ f i ( x t +1 )
Revised: − ∇ f ij ( x t ) (cid:13)(cid:13)  + (1 − p ) (cid:13)(cid:13) ∇ f i ( x t +1 )

Original: − ∇ f i ( x t ) (cid:13)(cid:13) 2 .
Revised: − ∇ f i ( x t ) (cid:13)(cid:13) 2 .

Original: From Assumptions 5.3 and 5.4, we can conclude that
Revised: From Assumptions 5.3 and 5.4, we can conclude that

Original: E h (cid:104)(cid:13)(cid:13) h t
Revised: E h (cid:104)(cid:13)(cid:13) h t

Original: +1 i − h ti (cid:13)(cid:13) 2 (cid:105) ≤ p (cid:13)(cid:13)
Revised: +1 i − h ti (cid:13)(cid:13) 2 (cid:105) ≤ p (cid:13)(cid:13)

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Theorem 6.4.
Revised: Theorem 6.4.

Original: Suppose that Assumptions 5.1, 5.2, 5.3, 5.
Revised: Suppose that Assumptions 5.1, 5.2, 5.3, 5.

Original: [Equation]
Revised: [Equation]

Original: and 1.2 hold.
Revised: and 1.2 hold.

Original: Let us take a = 1 / (2 ω + 1) , probability p ∈ (0 , 1] , and
Revised: Let us take a = 1 / (2 ω + 1) , probability p ∈ (0 , 1] , and

Original: [Equation]
Revised: [Equation]

Original: in Algorithm 1 (DASHA-PAGE) then
Revised: in Algorithm 1 (DASHA-PAGE) then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let us ﬁx constants ν, ρ ∈ [0 , ∞ ) that we will deﬁne later.
Revised: Let us ﬁx constants ν, ρ ∈ [0 , ∞ ) that we will deﬁne later.

Original: Considering Lemma I.3,
Revised: Considering Lemma I.3,

Original: Lemma I.
Revised: Lemma I.

Original: [Equation]
Revised: [Equation]

Original: and the law of total expectation, we obtain
Revised: and the law of total expectation, we obtain

Original: E (cid:2) f
Revised: E (cid:2) f

Original: ( x t +
Revised: ( x t +

Original: [Table]
Revised: [Table]

Original: L 2 (cid:19) (cid:13)(cid:13) x
Revised: L 2 (cid:19) (cid:13)(cid:13) x

Original: t +1 − x t (cid:13)(cid:13) 

Revised: t +1 − x t (cid:13)(cid:13)

Original: + 2 p 1 n n (cid:88) i =
Revised: + 2 p 1 n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: h ti − ∇ f i ( x t ) (cid:13)(cid:13)(cid:35) + ν E (cid:20) (1 − p )
Revised: h ti − ∇ f i ( x t ) (cid:13)(cid:13)(cid:35) + ν E (cid:20) (1 − p )

Original: L 2max nB (cid:13)(cid:13) x
Revised: L 2max nB (cid:13)(cid:13) x

Original: t +1 − x t (cid:13)(cid:13) 

Revised: t +1 − x t (cid:13)(cid:13)

Original: + (1 − p ) (cid:13)(cid:13) h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:21)
Revised: + (1 − p ) (cid:13)(cid:13) h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:21)

Original: + ρ E (cid:34) (1 − p ) L 2max
Revised: + ρ E (cid:34) (1 − p ) L 2max

Original: B (cid:13)(cid:13) x
Revised: B (cid:13)(cid:13) x

Original: t +1 − x t (cid:13)(cid:13) 

Revised: t +1 − x t (cid:13)(cid:13)

Original: + (1 − p ) 1 n n (cid:88) i =
Revised: + (1 − p ) 1 n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: h ti − ∇ f i ( x t ) (cid:13)(cid:13)(cid:35) .
Revised: h ti − ∇ f i ( x t ) (cid:13)(cid:13)(cid:35) .

Original: [Table]
Revised: [Table]

Original: 2max
Revised: 2max

Original: B 
Revised: B 

Original: 
Revised: 

Original: E (cid:104)(cid:13)(cid:13) x t +1 − x t (cid:13)(cid:13) 2 (cid:105) + ( γ + ν (1 − p ))
Revised: E (cid:104)(cid:13)(cid:13) x t +1 − x t (cid:13)(cid:13) 2 (cid:105) + ( γ + ν (1 − p ))

Original: E (cid:104)(cid:13)(cid:13) h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105) + (cid:18) 16 γpω (2 ω + 1) n + ρ (1 − p ) (cid:19)
Revised: E (cid:104)(cid:13)(cid:13) h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105) + (cid:18) 16 γpω (2 ω + 1) n + ρ (1 − p ) (cid:19)

Original: E (cid:34)n n (cid:88) i =
Revised: E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: h ti − ∇ f i ( x t ) (cid:13)(cid:13)(cid:35) .
Revised: h ti − ∇ f i ( x t ) (cid:13)(cid:13)(cid:35) .

Original: Next, let us ﬁx ν = γp , to get
Revised: Next, let us ﬁx ν = γp , to get

Original: E (cid:2) f
Revised: E (cid:2) f

Original: ( x t +1 ) (cid:3) + γ (2 ω + 1) E (cid:104)(cid:13)(cid:13) g t +1 − h t +1 (cid:13)(cid:13) 2 (cid:105) + 2 γω n E (cid:34)n n (cid:88) i =
Revised: ( x t +1 ) (cid:3) + γ (2 ω + 1) E (cid:104)(cid:13)(cid:13) g t +1 − h t +1 (cid:13)(cid:13) 2 (cid:105) + 2 γω n E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g t +

Revised: g t +

Original: i − h t
Revised: i − h t

Original: +1 i (cid:13)(cid:13) 2 (cid:35) + γ p E (cid:104)(cid:13)(cid:13)
Revised: +1 i (cid:13)(cid:13) 2 (cid:35) + γ p E (cid:104)(cid:13)(cid:13)

Original: h t +1 − ∇ f ( x t +1 ) (cid:13)(cid:13) 2 (cid:105)
Revised: h t +1 − ∇ f ( x t +1 ) (cid:13)(cid:13) 2 (cid:105)

Original: + ρ E (cid:34)n n (cid:88)
Revised: + ρ E (cid:34)n n (cid:88)

Original: i =
Revised: i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: h t
Revised: h t

Original: +

Revised: +

Original: i − ∇ f i ( x t +1 ) (cid:13)(cid:13)(cid:35) ≤ E (cid:2) f ( x t ) (cid:3) − γ 2 E (cid:104)(cid:13)(cid:13) ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105) + γ (2 ω + 1) E (cid:104)(cid:13)(cid:13) g t − h t (cid:13)(cid:13) 2 (cid:105) + 2 γω n E (cid:34)n n (cid:88)
Revised: i − ∇ f i ( x t +1 ) (cid:13)(cid:13)(cid:35) ≤ E (cid:2) f ( x t ) (cid:3) − γ 2 E (cid:104)(cid:13)(cid:13) ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105) + γ (2 ω + 1) E (cid:104)(cid:13)(cid:13) g t − h t (cid:13)(cid:13) 2 (cid:105) + 2 γω n E (cid:34)n n (cid:88)

Original: i =
Revised: i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g ti − h ti (cid:13)(cid:13)(cid:35) + γ p E (cid:104)(cid:13)(cid:13)
Revised: g ti − h ti (cid:13)(cid:13)(cid:35) + γ p E (cid:104)(cid:13)(cid:13)

Original: h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105) −   1 2 γ − L 2 − 8 γω (2 ω + 1) (cid:16) (1 − p )
Revised: h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105) −   1 2 γ − L 2 − 8 γω (2 ω + 1) (cid:16) (1 − p )

Original: L 2max B + 2 (cid:98) L 2 (cid:17) n − γ (1 − p )
Revised: L 2max B + 2 (cid:98) L 2 (cid:17) n − γ (1 − p )

Original: L 2max
Revised: L 2max

Original: pnB − ρ (1 − p )
Revised: pnB − ρ (1 − p )

Original: L 2max
Revised: L 2max

Original: B 
Revised: B 

Original: 
Revised: 

Original: E (cid:104)(cid:13)(cid:13) x t +1 − x t (cid:13)(cid:13) 2 (cid:105) + (cid:18) 16 γpω (2 ω + 1) n + ρ (1 − p ) (cid:19)
Revised: E (cid:104)(cid:13)(cid:13) x t +1 − x t (cid:13)(cid:13) 2 (cid:105) + (cid:18) 16 γpω (2 ω + 1) n + ρ (1 − p ) (cid:19)

Original: E (cid:34)n n (cid:88) i = (cid:13) (cid:13) h t
Revised: E (cid:34)n n (cid:88) i = (cid:13) (cid:13) h t

Original: i − ∇ f i ( x t ) (cid:13)(cid:13)(cid:35) .
Revised: i − ∇ f i ( x t ) (cid:13)(cid:13)(cid:35) .

Original: By taking ρ = 16 γω (2 ω +1)
Revised: By taking ρ = 16 γω (2 ω +1)

Original: n , we obtain
Revised: n , we obtain

Original: E (cid:2) f
Revised: E (cid:2) f

Original: ( x t +1 ) (cid:3) + γ (2 ω + 1) E (cid:104)(cid:13)(cid:13) g t +1 − h t +1 (cid:13)(cid:13) 2 (cid:105) + 2 γω n E (cid:34)n n (cid:88) i =
Revised: ( x t +1 ) (cid:3) + γ (2 ω + 1) E (cid:104)(cid:13)(cid:13) g t +1 − h t +1 (cid:13)(cid:13) 2 (cid:105) + 2 γω n E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g t +

Revised: g t +

Original: i − h t
Revised: i − h t

Original: +1 i (cid:13)(cid:13) 2 (cid:35) + γ p E (cid:104)(cid:13)(cid:13)
Revised: +1 i (cid:13)(cid:13) 2 (cid:35) + γ p E (cid:104)(cid:13)(cid:13)

Original: h t +1 − ∇ f ( x t +1 ) (cid:13)(cid:13) 2 (cid:105) + 16 γω (2 ω + 1) n
Revised: h t +1 − ∇ f ( x t +1 ) (cid:13)(cid:13) 2 (cid:105) + 16 γω (2 ω + 1) n

Original: E (cid:34)n n (cid:88) i =
Revised: E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: h t
Revised: h t

Original: +

Revised: +

Original: i − ∇ f i ( x t +1 ) (cid:13)(cid:13)(cid:35) ≤ E (cid:2) f ( x t ) (cid:3) − γ 2 E (cid:104)(cid:13)(cid:13) ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105) + γ (2 ω + 1) E (cid:104)(cid:13)(cid:13) g t − h t (cid:13)(cid:13) 2 (cid:105) + 2 γω n E (cid:34)n n (cid:88)
Revised: i − ∇ f i ( x t +1 ) (cid:13)(cid:13)(cid:35) ≤ E (cid:2) f ( x t ) (cid:3) − γ 2 E (cid:104)(cid:13)(cid:13) ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105) + γ (2 ω + 1) E (cid:104)(cid:13)(cid:13) g t − h t (cid:13)(cid:13) 2 (cid:105) + 2 γω n E (cid:34)n n (cid:88)

Original: i =
Revised: i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g ti − h ti (cid:13)(cid:13)(cid:35) + γ p E (cid:104)(cid:13)(cid:13)
Revised: g ti − h ti (cid:13)(cid:13)(cid:35) + γ p E (cid:104)(cid:13)(cid:13)

Original: h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105) + 16 γω (2 ω + 1) n
Revised: h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105) + 16 γω (2 ω + 1) n

Original: E (cid:34)n n (cid:88) i =
Revised: E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: h ti − ∇ f i ( x t ) (cid:13)(cid:13)(cid:35)
Revised: h ti − ∇ f i ( x t ) (cid:13)(cid:13)(cid:35)

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Next, considering the choice of γ and Lemma I.
Revised: Next, considering the choice of γ and Lemma I.

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: Finally, in the view of Lemma I.5 with
Revised: Finally, in the view of Lemma I.5 with

Original: [Equation]
Revised: [Equation]

Original: we can conclude the proof.
Revised: we can conclude the proof.

Original: Corollary 6.5.
Revised: Corollary 6.5.

Original: Let the assumptions from Theorem 6.4 hold, p = B / ( m + B ) , and g 0 i
Revised: Let the assumptions from Theorem 6.4 hold, p = B / ( m + B ) , and g 0 i

Original: =
Revised: =

Original: h 

Revised: h

Original: i = ∇
Revised: i = ∇

Original: f i ( x 0 ) for all i ∈ [ n ] .
Revised: f i ( x 0 ) for all i ∈ [ n ] .

Original: Then DASHA-PAGE needs
Revised: Then DASHA-PAGE needs

Original: [Equation]
Revised: [Equation]

Original:  communication rounds to get an ε -solution, the communication complexity is equal to O ( d + ζ C T ) , and the expected # of gradient calculations per node equals O ( m + BT ) , where ζ C is the expected density from Deﬁnition 1.3.
Revised:  communication rounds to get an ε -solution, the communication complexity is equal to O ( d + ζ C T ) , and the expected # of gradient calculations per node equals O ( m + BT ) , where ζ C is the expected density from Deﬁnition 1.3.

Original: Proof.
Revised: Proof.

Original: Corollary 6.5 can be proved in the same way as Corollary 6.2.
Revised: Corollary 6.5 can be proved in the same way as Corollary 6.2.

Original: One only should note that the expected number of gradients calculations at each communication round equals pm + (1 − p ) B = 2 mB m + B ≤ 2 B.
Revised: One only should note that the expected number of gradients calculations at each communication round equals pm + (1 − p ) B = 2 mB m + B ≤ 2 B.

Original: Corollary 6.6.
Revised: Corollary 6.6.

Original: Suppose that assumptions of Corollary 6.5 hold, B ≤ (cid:112) m / n , and we use the unbiased compressor Rand K with K = ζ C =
Revised: Suppose that assumptions of Corollary 6.5 hold, B ≤ (cid:112) m / n , and we use the unbiased compressor Rand K with K = ζ C =

Original: Θ ( Bd / √ m ) .
Revised: Θ ( Bd / √ m ) .

Original: Then the communication complexity of Algorithmis
Revised: Then the communication complexity of Algorithmis

Original: [Equation]
Revised: [Equation]

Original: and the expected # of gradient calculations per node equals
Revised: and the expected # of gradient calculations per node equals

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: In the view of Theorem F.2, we have ω + 1 =
Revised: In the view of Theorem F.2, we have ω + 1 =

Original: d/K. Combining this, inequalities L ≤ (cid:98) L ≤
Revised: d/K. Combining this, inequalities L ≤ (cid:98) L ≤

Original: L max , and K = Θ (cid:16)
Revised: L max , and K = Θ (cid:16)

Original: Bd √ m (cid:17)
Revised: Bd √ m (cid:17)

Original: =
Revised: =

Original: O (cid:16) d √ n (cid:17) , we can show that the communication complexity equals
Revised: O (cid:16) d √ n (cid:17) , we can show that the communication complexity equals

Original: O ( d +
Revised: O ( d +

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: O
Revised: O

Original: [Equation]
Revised: [Equation]

Original: And the expected number of gradient calculations per node equals
Revised: And the expected number of gradient calculations per node equals

Original: O
Revised: O

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: I. C ASE OF DASHA-PAGE UNDER PŁ- CONDITION
Revised: I. C ASE OF DASHA-PAGE UNDER PŁ- CONDITION

Original: Theorem I.12.
Revised: Theorem I.12.

Original: Suppose that Assumption 5.1, 5.2, 5.3, 1.2, 5.4, and G.1 hold.
Revised: Suppose that Assumption 5.1, 5.2, 5.3, 1.2, 5.4, and G.1 hold.

Original: Let us take a = 1 / (2 ω + 1) , probability p ∈ (0 , 1] , batch size B ∈ [ m ] , and γ ≤ min (cid:40)(cid:18)
Revised: Let us take a = 1 / (2 ω + 1) , probability p ∈ (0 , 1] , batch size B ∈ [ m ] , and γ ≤ min (cid:40)(cid:18)

Original: L +
Revised: L +

Original: [Equation]
Revised: [Equation]

Original: (cid:19)
Revised: (cid:19)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 2 µ , p
Revised: 2 µ , p

Original: [Equation]
Revised: [Equation]

Original: (cid:41) in Algorithm(DASHA-PAGE) , then
Revised: (cid:41) in Algorithm(DASHA-PAGE) , then

Original: E
Revised: E

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Proof.
Revised: Proof.

Original: Let us ﬁx constants ν, ρ ∈ [0 , ∞ ) that we will deﬁne later.
Revised: Let us ﬁx constants ν, ρ ∈ [0 , ∞ ) that we will deﬁne later.

Original: Considering Lemma I.4,
Revised: Considering Lemma I.4,

Original: Lemma I.11, and the law of total expectation, we obtain
Revised: Lemma I.11, and the law of total expectation, we obtain

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: + ν
Revised: + ν

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: +1 )
Revised: +1 )

Original: [Equation]
Revised: [Equation]

Original: (cid:105)
Revised: (cid:105)

Original: [Equation]
Revised: [Equation]

Original: ≤ E (cid:20) f
Revised: ≤ E (cid:20) f

Original: ( x t
Revised: ( x t

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: 2 (cid:13)(cid:13) ∇
Revised: 2 (cid:13)(cid:13) ∇

Original: [Equation]
Revised: [Equation]

Original: ( x t ) (cid:13)(cid:13)
Revised: ( x t ) (cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: (cid:18) 1 2 γ − L
Revised: (cid:18) 1 2 γ − L

Original: [Equation]
Revised: [Equation]

Original: (cid:19) (cid:13)(cid:13)
Revised: (cid:19) (cid:13)(cid:13)

Original: x t +1 − x t (cid:13)(cid:13)
Revised: x t +1 − x t (cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: After rearranging the terms, we get
Revised: After rearranging the terms, we get

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: 1 ) (cid:3) + 2 γ (2 ω + 1)E (cid:104)(cid:13)(cid:13) g t +1 − h t +1 (cid:13)(cid:13) 2 (cid:105) + 8 γω n E (cid:34)n n (cid:88) i =
Revised: 1 ) (cid:3) + 2 γ (2 ω + 1)E (cid:104)(cid:13)(cid:13) g t +1 − h t +1 (cid:13)(cid:13) 2 (cid:105) + 8 γω n E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g t +

Revised: g t +

Original: i − h t
Revised: i − h t

Original: +1 i (cid:13)(cid:13) 2 (cid:35) + ν E (cid:104)(cid:13)(cid:13)
Revised: +1 i (cid:13)(cid:13) 2 (cid:35) + ν E (cid:104)(cid:13)(cid:13)

Original: h t +1 − ∇ f ( x t +1 ) (cid:13)(cid:13) 2 (cid:105)
Revised: h t +1 − ∇ f ( x t +1 ) (cid:13)(cid:13) 2 (cid:105)

Original: + ρ E (cid:34)n n (cid:88)
Revised: + ρ E (cid:34)n n (cid:88)

Original: i =
Revised: i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: h t
Revised: h t

Original: +

Revised: +

Original: i − ∇ f i ( x t +1 ) (cid:13)(cid:13)(cid:35) ≤ E (cid:2) f ( x t ) (cid:3) − γ 2 E (cid:104)(cid:13)(cid:13) ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105)
Revised: i − ∇ f i ( x t +1 ) (cid:13)(cid:13)(cid:35) ≤ E (cid:2) f ( x t ) (cid:3) − γ 2 E (cid:104)(cid:13)(cid:13) ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105)

Original: + (1 − γµ ) 2 γ (2 ω
Revised: + (1 − γµ ) 2 γ (2 ω

Original: + 1)E (cid:104)(cid:13)(cid:13) g t − h t (cid:13)(cid:13) 2 (cid:105) + (1 − γµ ) 8 γω n E (cid:34)n n (cid:88)
Revised: + 1)E (cid:104)(cid:13)(cid:13) g t − h t (cid:13)(cid:13) 2 (cid:105) + (1 − γµ ) 8 γω n E (cid:34)n n (cid:88)

Original: i =
Revised: i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g ti − h ti (cid:13)(cid:13)(cid:35) − (cid:18)2 γ −
Revised: g ti − h ti (cid:13)(cid:13)(cid:35) − (cid:18)2 γ −

Original: L 2 − 20 γω (2 ω + 1) n (cid:18) (1 − p )
Revised: L 2 − 20 γω (2 ω + 1) n (cid:18) (1 − p )

Original: L 2max
Revised: L 2max

Original: B + 2 (cid:98) L 2 (cid:19) − ν (1 − p )
Revised: B + 2 (cid:98) L 2 (cid:19) − ν (1 − p )

Original: L 2max nB − ρ (1 − p )
Revised: L 2max nB − ρ (1 − p )

Original: L 2max
Revised: L 2max

Original: B (cid:19)
Revised: B (cid:19)

Original: E (cid:104)(cid:13)(cid:13) x t +1 − x t (cid:13)(cid:13) 2 (cid:105) + ( γ + ν (1 − p ))
Revised: E (cid:104)(cid:13)(cid:13) x t +1 − x t (cid:13)(cid:13) 2 (cid:105) + ( γ + ν (1 − p ))

Original: E (cid:104)(cid:13)(cid:13) h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105) + (cid:18) 40 pγω (2 ω + 1) n + ρ (1 − p ) (cid:19)
Revised: E (cid:104)(cid:13)(cid:13) h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105) + (cid:18) 40 pγω (2 ω + 1) n + ρ (1 − p ) (cid:19)

Original: E (cid:34)n n (cid:88) i =
Revised: E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: h ti − ∇ f i ( x t ) (cid:13)(cid:13)(cid:35) .
Revised: h ti − ∇ f i ( x t ) (cid:13)(cid:13)(cid:35) .

Original: By taking ν = 2 γp and ρ = 80 γω (2 ω +1) n , one can see that γ + ν (1 − p ) ≤ (cid:0) 1 − p 2 (cid:1) ν and 40 pγω (2 ω +1) n + ρ (1 − p ) ≤ (cid:0) 1 − p 2 (cid:1) ρ, thus
Revised: By taking ν = 2 γp and ρ = 80 γω (2 ω +1) n , one can see that γ + ν (1 − p ) ≤ (cid:0) 1 − p 2 (cid:1) ν and 40 pγω (2 ω +1) n + ρ (1 − p ) ≤ (cid:0) 1 − p 2 (cid:1) ρ, thus

Original: E (cid:2) f
Revised: E (cid:2) f

Original: ( x t +1 ) (cid:3) + 2 γ (2 ω + 1)E (cid:104)(cid:13)(cid:13) g t
Revised: ( x t +1 ) (cid:3) + 2 γ (2 ω + 1)E (cid:104)(cid:13)(cid:13) g t

Original: +1 − h t +1 (cid:13)(cid:13) 2 (cid:105) + 8 γω n E (cid:34)n n (cid:88) i =
Revised: +1 − h t +1 (cid:13)(cid:13) 2 (cid:105) + 8 γω n E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g t +

Revised: g t +

Original: i − h t
Revised: i − h t

Original: +1 i (cid:13)(cid:13) 2 (cid:35) + ν E (cid:104)(cid:13)(cid:13)
Revised: +1 i (cid:13)(cid:13) 2 (cid:35) + ν E (cid:104)(cid:13)(cid:13)

Original: h t +1 − ∇ f ( x t +1 ) (cid:13)(cid:13) 2 (cid:105)
Revised: h t +1 − ∇ f ( x t +1 ) (cid:13)(cid:13) 2 (cid:105)

Original: + ρ E (cid:34)n n (cid:88)
Revised: + ρ E (cid:34)n n (cid:88)

Original: i =
Revised: i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: h t
Revised: h t

Original: +

Revised: +

Original: i − ∇ f i ( x t +1 ) (cid:13)(cid:13)(cid:35)
Revised: i − ∇ f i ( x t +1 ) (cid:13)(cid:13)(cid:35)

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Next, considering the choice of γ and Lemma I.
Revised: Next, considering the choice of γ and Lemma I.

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: n n (cid:88) i =
Revised: n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g t +

Revised: g t +

Original: i − h t
Revised: i − h t

Original: +1 i (cid:13)(cid:13) 2 (cid:35) + ν E (cid:104)(cid:13)(cid:13)
Revised: +1 i (cid:13)(cid:13) 2 (cid:35) + ν E (cid:104)(cid:13)(cid:13)

Original: h t +1 − ∇ f ( x t +1 ) (cid:13)(cid:13) 2 (cid:105)
Revised: h t +1 − ∇ f ( x t +1 ) (cid:13)(cid:13) 2 (cid:105)

Original: + ρ E (cid:34)n n (cid:88)
Revised: + ρ E (cid:34)n n (cid:88)

Original: i =
Revised: i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: h t
Revised: h t

Original: +

Revised: +

Original: i − ∇ f i ( x t +1 ) (cid:13)(cid:13)(cid:35) ≤ E (cid:2) f ( x t ) (cid:3) − γ 2 E (cid:104)(cid:13)(cid:13) ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105)
Revised: i − ∇ f i ( x t +1 ) (cid:13)(cid:13)(cid:35) ≤ E (cid:2) f ( x t ) (cid:3) − γ 2 E (cid:104)(cid:13)(cid:13) ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105)

Original: + (1 − γµ ) 2 γ (2 ω
Revised: + (1 − γµ ) 2 γ (2 ω

Original: + 1)E (cid:104)(cid:13)(cid:13) g t − h t (cid:13)(cid:13) 2 (cid:105) + (1 − γµ ) 8 γω n E (cid:34)n n (cid:88)
Revised: + 1)E (cid:104)(cid:13)(cid:13) g t − h t (cid:13)(cid:13) 2 (cid:105) + (1 − γµ ) 8 γω n E (cid:34)n n (cid:88)

Original: i =
Revised: i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g ti − h ti (cid:13)(cid:13)(cid:35)
Revised: g ti − h ti (cid:13)(cid:13)(cid:35)

Original: + (1 − γµ )
Revised: + (1 − γµ )

Original: 2 γ p E
Revised: 2 γ p E

Original: (cid:104)(cid:13)(cid:13) h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105) + (1 − γµ ) 80 γω (2 ω + 1) n
Revised: (cid:104)(cid:13)(cid:13) h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105) + (1 − γµ ) 80 γω (2 ω + 1) n

Original: E (cid:34)n n (cid:88) i =
Revised: E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: h ti − ∇ f i ( x t ) (cid:13)(cid:13)(cid:35) .
Revised: h ti − ∇ f i ( x t ) (cid:13)(cid:13)(cid:35) .

Original: In the view of Lemma I.6 with Ψ t = 2(2 ω + 1)E (cid:104)(cid:13)(cid:13)
Revised: In the view of Lemma I.6 with Ψ t = 2(2 ω + 1)E (cid:104)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: we can conclude the proof of the theorem.
Revised: we can conclude the proof of the theorem.

Original: Corollary I.13.
Revised: Corollary I.13.

Original: Suppose that assumptions from Theorem I.12 hold, probability p =
Revised: Suppose that assumptions from Theorem I.12 hold, probability p =

Original: B/ ( m + B ) , and h 

Revised: B/ ( m + B ) , and h

Original: i = g 0 i = 

Revised: i = g 0 i =

Original: for all i ∈ [ n ] , then DASHA-PAGE needs
Revised: for all i ∈ [ n ] , then DASHA-PAGE needs

Original: [Equation]
Revised: [Equation]

Original: communication rounds to get an ε -solution, the communication complexity is equal to O ( ζ C T ) , and the expected number of gradient calculations per node equals O ( BT ) , where ζ C is the expected density from Deﬁnition 1.3.
Revised: communication rounds to get an ε -solution, the communication complexity is equal to O ( ζ C T ) , and the expected number of gradient calculations per node equals O ( BT ) , where ζ C is the expected density from Deﬁnition 1.3.

Original: Proof.
Revised: Proof.

Original: Clearly, using Theorem I.12, one can show that Algorithm 1 returns an ε -solution after (26) communication rounds.
Revised: Clearly, using Theorem I.12, one can show that Algorithm 1 returns an ε -solution after (26) communication rounds.

Original: At each communication round of Algorithm 1, each node sends ζ C coordinates, thus the total communication complexity would be O ( ζ C T ) .
Revised: At each communication round of Algorithm 1, each node sends ζ C coordinates, thus the total communication complexity would be O ( ζ C T ) .

Original: Moreover, the expected number of gradients calculations at each communication round equals pm +(1 − p ) B = 2 mBm + B ≤ 2 B, thus the total expected number of gradients that each node calculates is O ( BT ) .
Revised: Moreover, the expected number of gradients calculations at each communication round equals pm +(1 − p ) B = 2 mBm + B ≤ 2 B, thus the total expected number of gradients that each node calculates is O ( BT ) .

Original: Unlike Corollary 6.5, in this corollary, we can initialize h 

Revised: Unlike Corollary 6.5, in this corollary, we can initialize h

Original: i and g 0 i , for instance, with zeros because the corresponding initialization error Ψ 0 from the proof of Theorem I.12 would be under the logarithm.
Revised: i and g 0 i , for instance, with zeros because the corresponding initialization error Ψ 0 from the proof of Theorem I.12 would be under the logarithm.

Original: I.
Revised: I.

Original: C ASE OF DASHA-MVR
Revised: C ASE OF DASHA-MVR

Original: We introduce new notations: ∇ f i ( x t +1 ; ξ t +1 i )
Revised: We introduce new notations: ∇ f i ( x t +1 ; ξ t +1 i )

Original: = 1 B (cid:80)
Revised: = 1 B (cid:80)

Original: Bj =1 ∇ f i ( x t +1 ; ξ t +1 ij ) and ∇ f ( x t +1 ; ξ t +1 )
Revised: Bj =1 ∇ f i ( x t +1 ; ξ t +1 ij ) and ∇ f ( x t +1 ; ξ t +1 )

Original: = 1 n (cid:80) ni
Revised: = 1 n (cid:80) ni

Original: =1 ∇ f i ( x t +1 ; ξ t +1 i ) .
Revised: =1 ∇ f i ( x t +1 ; ξ t +1 i ) .

Original: Lemma I.14.
Revised: Lemma I.14.

Original: Suppose that Assumptions 5.3, 5.5 and 5.6 hold.
Revised: Suppose that Assumptions 5.3, 5.5 and 5.6 hold.

Original: For h t +1 i from Algorithm 1 (DASHAMVR)
Revised: For h t +1 i from Algorithm 1 (DASHAMVR)

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: E h (cid:104)(cid:13)(cid:13) h t
Revised: E h (cid:104)(cid:13)(cid:13) h t

Original: +1 i
Revised: +1 i

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: [Table]
Revised: [Table]

Original: 1 ) (cid:1) + (1 − b ) (cid:0) ∇ f
Revised: 1 ) (cid:1) + (1 − b ) (cid:0) ∇ f

Original: ( x t +1 ; ξ t +1 )
Revised: ( x t +1 ; ξ t +1 )

Original: − ∇ f ( x t +1 )
Revised: − ∇ f ( x t +1 )

Original: + ∇ f ( x t ) − ∇ f ( x t ; ξ t +1 ) (cid:1)(cid:13)(cid:13) 2 (cid:105)
Revised: + ∇ f ( x t ) − ∇ f ( x t ; ξ t +1 ) (cid:1)(cid:13)(cid:13) 2 (cid:105)

Original: + (1 − b ) 

Revised: + (1 − b )

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: h t − ∇ f ( x t ) (cid:13)(cid:13)(14) ≤ 2 b 2 E h (cid:104)(cid:13)(cid:13) ∇ f ( x t +1 ; ξ t +1 )
Revised: h t − ∇ f ( x t ) (cid:13)(cid:13)(14) ≤ 2 b 2 E h (cid:104)(cid:13)(cid:13) ∇ f ( x t +1 ; ξ t +1 )

Original: − ∇ f ( x t +1 )
Revised: − ∇ f ( x t +1 )

Original: (cid:13)(cid:13) 2 (cid:105) + 2 (1 − b ) 2 E h (cid:104)(cid:13)(cid:13) ∇ f ( x t +1 ; ξ t +1 )
Revised: (cid:13)(cid:13) 2 (cid:105) + 2 (1 − b ) 2 E h (cid:104)(cid:13)(cid:13) ∇ f ( x t +1 ; ξ t +1 )

Original: − ∇ f ( x t +1 )
Revised: − ∇ f ( x t +1 )

Original: + ∇ f ( x t ) − ∇ f ( x t ; ξ t +1 )
Revised: + ∇ f ( x t ) − ∇ f ( x t ; ξ t +1 )

Original: (cid:13)(cid:13) 2 (cid:105) + (1 − b ) 

Revised: (cid:13)(cid:13) 2 (cid:105) + (1 − b )

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: h t − ∇ f ( x t ) (cid:13)(cid:13)= 2 bnn (cid:88) i =
Revised: h t − ∇ f ( x t ) (cid:13)(cid:13)= 2 bnn (cid:88) i =

Original: E h (cid:104)(cid:13)(cid:13) ∇ f i ( x t +1 ; ξ t +1 i ) − ∇ f i ( x t +1 )
Revised: E h (cid:104)(cid:13)(cid:13) ∇ f i ( x t +1 ; ξ t +1 i ) − ∇ f i ( x t +1 )

Original: (cid:13)(cid:13) 2 (cid:105) + 2 (1 − b )nn (cid:88) i =
Revised: (cid:13)(cid:13) 2 (cid:105) + 2 (1 − b )nn (cid:88) i =

Original: E h (cid:104)(cid:13)(cid:13) ∇ f i ( x t +1 ; ξ t +1 i ) − ∇ f i ( x t ; ξ t +1 i ) − (cid:0) ∇ f i ( x t +1 )
Revised: E h (cid:104)(cid:13)(cid:13) ∇ f i ( x t +1 ; ξ t +1 i ) − ∇ f i ( x t ; ξ t +1 i ) − (cid:0) ∇ f i ( x t +1 )

Original: − ∇ f i ( x t ) (cid:1)(cid:13)(cid:13) 2 (cid:105) + (1 − b ) 2 (cid:13)(cid:13)
Revised: − ∇ f i ( x t ) (cid:1)(cid:13)(cid:13) 2 (cid:105) + (1 − b ) 2 (cid:13)(cid:13)

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Using Assumptions 5.5 and 5.
Revised: Using Assumptions 5.5 and 5.

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: Similarly, we can get the bound for E h
Revised: Similarly, we can get the bound for E h

Original: (cid:104)(cid:13)(cid:13) h t
Revised: (cid:104)(cid:13)(cid:13) h t

Original: +1 i
Revised: +1 i

Original: [Equation]
Revised: [Equation]

Original: Now, we proof the last inequality of the lemma:
Revised: Now, we proof the last inequality of the lemma:

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Table]
Revised: [Table]

Original: 1 ) (cid:1) + (1 − b ) (cid:0) ∇ f i ( x
Revised: 1 ) (cid:1) + (1 − b ) (cid:0) ∇ f i ( x

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ξ
Revised: ξ

Original: [Table]
Revised: [Table]

Original: ) − ∇ f i (
Revised: ) − ∇ f i (

Original: [Table]
Revised: [Table]

Original: i ) − ( ∇ f i
Revised: i ) − ( ∇ f i

Original: [Table]
Revised: [Table]

Original: ) − ∇ f
Revised: ) − ∇ f

Original: [Table]
Revised: [Table]

Original: ) (cid:1)(cid:13)(cid:13)
Revised: ) (cid:1)(cid:13)(cid:13)

Original: [Table]
Revised: [Table]

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: [Table]
Revised: [Table]

Original: f i (
Revised: f i (

Original: [Table]
Revised: [Table]

Original: ) − ∇ f i (
Revised: ) − ∇ f i (

Original: [Table]
Revised: [Table]

Original: t ) − b (cid:0) h ti −
Revised: t ) − b (cid:0) h ti −

Original: [Table]
Revised: [Table]

Original: f i (
Revised: f i (

Original: [Table]
Revised: [Table]

Original: t ) (cid:1)(cid:13)(cid:13)
Revised: t ) (cid:1)(cid:13)(cid:13)

Original: [Table]
Revised: [Table]

Original: f
Revised: f

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: In the view of Assumptions 5.3, 5.5 and 5.6, we obtain
Revised: In the view of Assumptions 5.3, 5.5 and 5.6, we obtain

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: Theorem 6.7.
Revised: Theorem 6.7.

Original: Suppose that Assumptions 5.1, 5.2, 5.3, 5.5, 5.6 and 1.2 hold.
Revised: Suppose that Assumptions 5.1, 5.2, 5.3, 5.5, 5.6 and 1.2 hold.

Original: Let us take a = 1 2 ω +1 , b ∈ (0 , 1] , and γ ≤
Revised: Let us take a = 1 2 ω +1 , b ∈ (0 , 1] , and γ ≤

Original: [Equation]
Revised: [Equation]

Original: , in Algorithm(DASHA-MVR) .
Revised: , in Algorithm(DASHA-MVR) .

Original: Then
Revised: Then

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Proof.
Revised: Proof.

Original: Let us ﬁx constants ν, ρ ∈ [0 , ∞ ) that we will deﬁne later.
Revised: Let us ﬁx constants ν, ρ ∈ [0 , ∞ ) that we will deﬁne later.

Original: Considering Lemma I.3,
Revised: Considering Lemma I.3,

Original: Lemma I.
Revised: Lemma I.

Original: [Equation]
Revised: [Equation]

Original: and the law of total expectation, we obtain
Revised: and the law of total expectation, we obtain

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: f
Revised: f

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: +
Revised: +

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Table]
Revised: [Table]

Original: (cid:34) 2 b 2 σnB + 2 (1 − b ) 2 L 2 σ nB (cid:13)(cid:13)
Revised: (cid:34) 2 b 2 σnB + 2 (1 − b ) 2 L 2 σ nB (cid:13)(cid:13)

Original: x t +1 − x t (cid:13)(cid:13) 2 + (1 − b ) 

Revised: x t +1 − x t (cid:13)(cid:13) 2 + (1 − b )

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: h t − ∇ f ( x t ) (cid:13)(cid:13)(cid:35) + ρ E (cid:34) 2 b 2 σ
Revised: h t − ∇ f ( x t ) (cid:13)(cid:13)(cid:35) + ρ E (cid:34) 2 b 2 σ

Original: B +
Revised: B +

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: − b )
Revised: − b )

Original: [Equation]
Revised: [Equation]

Original: σ
Revised: σ

Original: B (cid:13)(cid:13) x
Revised: B (cid:13)(cid:13) x

Original: t +1 − x t (cid:13)(cid:13) 2 + (1 − b ) 2 1 n n (cid:88) i =
Revised: t +1 − x t (cid:13)(cid:13) 2 + (1 − b ) 2 1 n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: h ti − ∇ f i ( x t ) (cid:13)(cid:13)(cid:35)
Revised: h ti − ∇ f i ( x t ) (cid:13)(cid:13)(cid:35)

Original: After rearranging the terms, we get
Revised: After rearranging the terms, we get

Original: E (cid:2) f
Revised: E (cid:2) f

Original: ( x t +1 ) (cid:3) + γ (2 ω + 1) E (cid:104)(cid:13) (cid:13) g t
Revised: ( x t +1 ) (cid:3) + γ (2 ω + 1) E (cid:104)(cid:13) (cid:13) g t

Original: +

Revised: +

Original: − h t +1 (cid:13) (cid:13) 2 (cid:105) + 2 γω n E (cid:34)n n (cid:88) i = (cid:13) (cid:13) g t
Revised: − h t +1 (cid:13) (cid:13) 2 (cid:105) + 2 γω n E (cid:34)n n (cid:88) i = (cid:13) (cid:13) g t

Original: +1 i − h t
Revised: +1 i − h t

Original: +1 i (cid:13) (cid:13)(cid:35) + ν E (cid:104)(cid:13)(cid:13)
Revised: +1 i (cid:13) (cid:13)(cid:35) + ν E (cid:104)(cid:13)(cid:13)

Original: h t +1 − ∇ f ( x t +1 ) (cid:13)(cid:13) 2 (cid:105)
Revised: h t +1 − ∇ f ( x t +1 ) (cid:13)(cid:13) 2 (cid:105)

Original: + ρ E (cid:34)n n (cid:88)
Revised: + ρ E (cid:34)n n (cid:88)

Original: i =
Revised: i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: h t
Revised: h t

Original: +

Revised: +

Original: i − ∇ f i ( x t +1 ) (cid:13)(cid:13)(cid:35) ≤ E (cid:2) f ( x t ) (cid:3) − γ 2 E (cid:104)(cid:13)(cid:13) ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105) + γ (
Revised: i − ∇ f i ( x t +1 ) (cid:13)(cid:13)(cid:35) ≤ E (cid:2) f ( x t ) (cid:3) − γ 2 E (cid:104)(cid:13)(cid:13) ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105) + γ (

Original: [Equation]
Revised: [Equation]

Original: 1) E (cid:104)(cid:13)(cid:13) g t
Revised: 1) E (cid:104)(cid:13)(cid:13) g t

Original: [Equation]
Revised: [Equation]

Original: h t (cid:13)(cid:13)
Revised: h t (cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: (cid:105)
Revised: (cid:105)

Original: [Equation]
Revised: [Equation]

Original: E (cid:34)n n (cid:88)
Revised: E (cid:34)n n (cid:88)

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g ti
Revised: g ti

Original: [Equation]
Revised: [Equation]

Original: ti (cid:13)(cid:13)
Revised: ti (cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: (cid:35)
Revised: (cid:35)

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: (cid:0) γ
Revised: (cid:0) γ

Original: [Equation]
Revised: [Equation]

Original: ν (
Revised: ν (

Original: [Equation]
Revised: [Equation]

Original: b )
Revised: b )

Original: [Equation]
Revised: [Equation]

Original: (cid:1)
Revised: (cid:1)

Original: [Equation]
Revised: [Equation]

Original: (cid:104)(cid:13)(cid:13) h t
Revised: (cid:104)(cid:13)(cid:13) h t

Original: [Equation]
Revised: [Equation]

Original: f ( x t ) (cid:13)(cid:13) 2 (cid:105) + (cid:18) 16 b 2 γω (2 ω
Revised: f ( x t ) (cid:13)(cid:13) 2 (cid:105) + (cid:18) 16 b 2 γω (2 ω

Original: [Equation]
Revised: [Equation]

Original: 1) n
Revised: 1) n

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: (cid:19)
Revised: (cid:19)

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: h ti
Revised: h ti

Original: [Equation]
Revised: [Equation]

Original: i (
Revised: i (

Original: [Equation]
Revised: [Equation]

Original: ) (cid:13)(cid:13)
Revised: ) (cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: By taking ν = γb , one can see that γ + ν (1 − b ) 2 ≤ ν, and
Revised: By taking ν = γb , one can see that γ + ν (1 − b ) 2 ≤ ν, and

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: (cid:105) − 
Revised: (cid:105) − 

Original: [Equation]
Revised: [Equation]

Original: bnB
Revised: bnB

Original: [Equation]
Revised: [Equation]

Original: + (cid:18)
Revised: + (cid:18)

Original: [Equation]
Revised: [Equation]

Original: 1) n
Revised: 1) n

Original: [Equation]
Revised: [Equation]

Original: (cid:34)
Revised: (cid:34)

Original: [Equation]
Revised: [Equation]

Original: +(cid:18) 8 γω
Revised: +(cid:18) 8 γω

Original: [Equation]
Revised: [Equation]

Original: + 1) nB + γ bnB + ρ
Revised: + 1) nB + γ bnB + ρ

Original: B (cid:19) b 2 σ 2 .
Revised: B (cid:19) b 2 σ 2 .

Original: Next, we ﬁx ρ = 16 bγω (2 ω +1) n .
Revised: Next, we ﬁx ρ = 16 bγω (2 ω +1) n .

Original: With this choice of ρ and for all b ∈ [0 , 1] , we can show that 16 b 2 γω (2 ω +1) n + ρ (1 − b ) 2 ≤ ρ, thus
Revised: With this choice of ρ and for all b ∈ [0 , 1] , we can show that 16 b 2 γω (2 ω +1) n + ρ (1 − b ) 2 ≤ ρ, thus

Original: E (cid:2) f
Revised: E (cid:2) f

Original: ( x t +1 ) (cid:3) + γ (2 ω + 1) E (cid:104)(cid:13)(cid:13) g t +1 − h t +1 (cid:13)(cid:13) 2 (cid:105) + 2 γω n E (cid:34)n n (cid:88) i =
Revised: ( x t +1 ) (cid:3) + γ (2 ω + 1) E (cid:104)(cid:13)(cid:13) g t +1 − h t +1 (cid:13)(cid:13) 2 (cid:105) + 2 γω n E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g t +

Revised: g t +

Original: i − h t
Revised: i − h t

Original: +1 i (cid:13)(cid:13) 2 (cid:35) + γ b E (cid:104)(cid:13)(cid:13)
Revised: +1 i (cid:13)(cid:13) 2 (cid:35) + γ b E (cid:104)(cid:13)(cid:13)

Original: [Table]
Revised: [Table]

Original: − ∇ f (
Revised: − ∇ f (

Original: [Table]
Revised: [Table]

Original: 1 ) (cid:13)(cid:13) 2 (cid:105)
Revised: 1 ) (cid:13)(cid:13) 2 (cid:105)

Original: [Table]
Revised: [Table]

Original: 16 bγω (2 ω + 1) n
Revised: 16 bγω (2 ω + 1) n

Original: E (cid:34)n n (cid:88) i =
Revised: E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: h t
Revised: h t

Original: +

Revised: +

Original: i − ∇ f i ( x t +1 ) (cid:13)(cid:13)(cid:35)
Revised: i − ∇ f i ( x t +1 ) (cid:13)(cid:13)(cid:35)

Original: [Table]
Revised: [Table]

Original: E (cid:2)
Revised: E (cid:2)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ) (cid:3) − γ
Revised: ) (cid:3) − γ

Original: [Table]
Revised: [Table]

Original: E (cid:104)(cid:13)(cid:13) ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105) + γ (2 ω + 1) E (cid:104)(cid:13)(cid:13)
Revised: E (cid:104)(cid:13)(cid:13) ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105) + γ (2 ω + 1) E (cid:104)(cid:13)(cid:13)

Original: g t − h t (cid:13)(cid:13) 2 (cid:105) + 2 γω n E (cid:34)n n (cid:88) i =
Revised: g t − h t (cid:13)(cid:13) 2 (cid:105) + 2 γω n E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g ti − h ti (cid:13)(cid:13)(cid:35) + γ b E (cid:104)(cid:13)(cid:13) h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105) + 16 bγω (2 ω + 1) n
Revised: g ti − h ti (cid:13)(cid:13)(cid:35) + γ b E (cid:104)(cid:13)(cid:13) h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105) + 16 bγω (2 ω + 1) n

Original: E (cid:34)n n (cid:88) i =
Revised: E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: h ti − ∇ f i ( x t ) (cid:13)(cid:13)(cid:35)
Revised: h ti − ∇ f i ( x t ) (cid:13)(cid:13)(cid:35)

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ) (cid:16) (
Revised: ) (cid:16) (

Original: [Table]
Revised: [Table]

Original: B + (cid:98) L 2 (cid:17) n − 2 γ (
Revised: B + (cid:98) L 2 (cid:17) n − 2 γ (

Original: [Table]
Revised: [Table]

Original: b )
Revised: b )

Original: [Table]
Revised: [Table]

Original: L
Revised: L

Original: [Table]
Revised: [Table]

Original: bnB − 32 bγω (2 ω
Revised: bnB − 32 bγω (2 ω

Original: [Table] [Table]
Revised: [Table] [Table]

Original: b )
Revised: b )

Original: [Table]
Revised: [Table]

Original: L
Revised: L

Original: [Table]
Revised: [Table]

Original: σ nB 
Revised: σ nB 

Original:  E (cid:104)(cid:13)(cid:13) x
Revised:  E (cid:104)(cid:13)(cid:13) x

Original: t +1 − x t (cid:13)(cid:13) 2 (cid:105) +(cid:18) 8 γω (2 ω + 1) nB + γ bnB + 16 bγω (2 ω + 1) nB (cid:19) b 2 σ≤ E (cid:2) f
Revised: t +1 − x t (cid:13)(cid:13) 2 (cid:105) +(cid:18) 8 γω (2 ω + 1) nB + γ bnB + 16 bγω (2 ω + 1) nB (cid:19) b 2 σ≤ E (cid:2) f

Original: ( x t ) (cid:3) − γ 2 E (cid:104)(cid:13)(cid:13) ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105) + γ (2 ω + 1) E (cid:104)(cid:13)(cid:13) g t − h t (cid:13)(cid:13) 2 (cid:105) + 2 γω n E (cid:34)n n (cid:88)
Revised: ( x t ) (cid:3) − γ 2 E (cid:104)(cid:13)(cid:13) ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105) + γ (2 ω + 1) E (cid:104)(cid:13)(cid:13) g t − h t (cid:13)(cid:13) 2 (cid:105) + 2 γω n E (cid:34)n n (cid:88)

Original: i =
Revised: i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g ti − h ti (cid:13)(cid:13)(cid:35) + γ b E (cid:104)(cid:13)(cid:13) h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105) + 16 bγω (2 ω + 1) n
Revised: g ti − h ti (cid:13)(cid:13)(cid:35) + γ b E (cid:104)(cid:13)(cid:13) h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105) + 16 bγω (2 ω + 1) n

Original: E (cid:34)n n (cid:88) i =
Revised: E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: h ti − ∇ f i ( x t ) (cid:13)(cid:13)(cid:35)
Revised: h ti − ∇ f i ( x t ) (cid:13)(cid:13)(cid:35)

Original: [Equation]
Revised: [Equation]

Original: In the last inequality we use b ∈ (0 , 1] .
Revised: In the last inequality we use b ∈ (0 , 1] .

Original: Next
Revised: Next

Original: [Equation]
Revised: [Equation]

Original: considering the choice of γ and Lemma I.
Revised: considering the choice of γ and Lemma I.

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: In the view of Lemma I.5 with Ψ t
Revised: In the view of Lemma I.5 with Ψ t

Original: [Equation]
Revised: [Equation]

Original: and C = (cid:16) 48 ω (2 ω +1) nB + 2 bnB (cid:17)
Revised: and C = (cid:16) 48 ω (2 ω +1) nB + 2 bnB (cid:17)

Original: b 2 σ 2 , we can conclude the proof.
Revised: b 2 σ 2 , we can conclude the proof.

Original: Corollary 6.8.
Revised: Corollary 6.8.

Original: Suppose that assumptions from Theorem 6.7 hold, momentum b = Θ (cid:16) min (cid:110) 1 ω (cid:113) nεBσ 2 , nεBσ 2 (cid:111)(cid:17) , and g 0 i
Revised: Suppose that assumptions from Theorem 6.7 hold, momentum b = Θ (cid:16) min (cid:110) 1 ω (cid:113) nεBσ 2 , nεBσ 2 (cid:111)(cid:17) , and g 0 i

Original: =
Revised: =

Original: h 

Revised: h

Original: i = 1 B init (cid:80) B init k
Revised: i = 1 B init (cid:80) B init k

Original: =1 ∇ f i ( x 0 ; ξ 0 ik ) for all i ∈ [ n ] , and batch size B init =
Revised: =1 ∇ f i ( x 0 ; ξ 0 ik ) for all i ∈ [ n ] , and batch size B init =

Original: Θ ( B / b ) , then Algorithm 1 (DASHA-MVR) needs
Revised: Θ ( B / b ) , then Algorithm 1 (DASHA-MVR) needs

Original: [Equation]
Revised: [Equation]

Original: O
Revised: O

Original: [Equation]
Revised: [Equation]

Original:  communication rounds to get an ε -solution, the communication complexity is equal to O ( d + ζ C T ) , and the number of stochastic gradient calculations per node equals O ( B init + BT ) , where ζ C is the expected density from Deﬁnition 1.3.
Revised:  communication rounds to get an ε -solution, the communication complexity is equal to O ( d + ζ C T ) , and the number of stochastic gradient calculations per node equals O ( B init + BT ) , where ζ C is the expected density from Deﬁnition 1.3.

Original: Proof.
Revised: Proof.

Original: In the view of Theorem 6.7, we have
Revised: In the view of Theorem 6.7, we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Note, that 1 b =
Revised: Note, that 1 b =

Original: Θ
Revised: Θ

Original: [Equation]
Revised: [Equation]

Original: Thus we can take
Revised: Thus we can take

Original: T
Revised: T

Original: =
Revised: =

Original: O
Revised: O

Original: [Equation]
Revised: [Equation]

Revised:  

Original:   [Table]
Revised: [Table] [Equation] [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: Likewise, 1 n (cid:80) ni =1 E (cid:104)(cid:13)(cid:13)
Revised: Likewise, 1 n (cid:80) ni =1 E (cid:104)(cid:13)(cid:13)

Original: h 

Revised: h

Original: i − ∇ f i ( x 0 ) (cid:13)(cid:13) 2 (cid:105) ≤ σ 2 B init .
Revised: i − ∇ f i ( x 0 ) (cid:13)(cid:13) 2 (cid:105) ≤ σ 2 B init .

Original: All in all, we have
Revised: All in all, we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: In the view of Algorithm 1 and the fact that we use a mini-batch of stochastic gradients, the number of stochastic gradients that each node calculates equals B init + 2 BT
Revised: In the view of Algorithm 1 and the fact that we use a mini-batch of stochastic gradients, the number of stochastic gradients that each node calculates equals B init + 2 BT

Original: = O ( B init + BT ) .
Revised: = O ( B init + BT ) .

Original: Corollary 6.9.
Revised: Corollary 6.9.

Original: Suppose that assumptions of Corollary 6.8 hold, batch size B ≤ σ √ εn , we take Rand K with K = ζ C = Θ (cid:16) Bd √ εn σ (cid:17) , and (cid:101)
Revised: Suppose that assumptions of Corollary 6.8 hold, batch size B ≤ σ √ εn , we take Rand K with K = ζ C = Θ (cid:16) Bd √ εn σ (cid:17) , and (cid:101)

Original: L :=
Revised: L :=

Original: max { L, L σ , (cid:98) L } .
Revised: max { L, L σ , (cid:98) L } .

Original: Then the communication complexity equals
Revised: Then the communication complexity equals

Original: [Equation]
Revised: [Equation]

Original: and the expected # of stochastic gradient calculations per node equals
Revised: and the expected # of stochastic gradient calculations per node equals

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: In the view of Theorem F.2, we have ω +

Revised: In the view of Theorem F.2, we have ω +

Original: =
Revised: =

Original: d/K. Moreover, K = Θ (cid:16)
Revised: d/K. Moreover, K = Θ (cid:16)

Original: Bd √ εn σ (cid:17)
Revised: Bd √ εn σ (cid:17)

Original: =
Revised: =

Original: O
Revised: O

Original: [Equation]
Revised: [Equation]

Original: (cid:17) , thus the communication complexity equals
Revised: (cid:17) , thus the communication complexity equals

Original: O ( d + ζ C
Revised: O ( d + ζ C

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: O
Revised: O

Original: [Equation]
Revised: [Equation]

Original: And the expected number of stochastic gradient calculations per node equals
Revised: And the expected number of stochastic gradient calculations per node equals

Original: O ( B init + BT )
Revised: O ( B init + BT )

Original: =
Revised: =

Original: O 
Revised: O 

Original: [Equation]
Revised: [Equation]

Original: Bnε
Revised: Bnε

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: I. C ASE OF DASHA-MVR UNDER PŁ- CONDITION
Revised: I. C ASE OF DASHA-MVR UNDER PŁ- CONDITION

Original: Theorem
Revised: Theorem

Original: I.15.
Revised: I.15.

Original: Suppose that
Revised: Suppose that

Original: Assumption 5.1, 5.2, 5.3, 1.2, 5.5, 5.
Revised: Assumption 5.1, 5.2, 5.3, 1.2, 5.5, 5.

Original: and
Revised: and

Original: G. hold.
Revised: G. hold.

Original: Let us take a = 1 / (2 ω + 1) , b ∈ (0 , 1] and γ ≤ min
Revised: Let us take a = 1 / (2 ω + 1) , b ∈ (0 , 1] and γ ≤ min

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original:    in Algorithm 1 (DASHAMVR) , then
Revised:    in Algorithm 1 (DASHAMVR) , then

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: (cid:32)
Revised: (cid:32)

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let us ﬁx constants ν, ρ ∈ [0 , ∞ ) that we will deﬁne later.
Revised: Let us ﬁx constants ν, ρ ∈ [0 , ∞ ) that we will deﬁne later.

Original: Considering Lemma I.4,
Revised: Considering Lemma I.4,

Original: Lemma I.14, and the law of total expectation, we obtain
Revised: Lemma I.14, and the law of total expectation, we obtain

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: γω n E (cid:34)n n
Revised: γω n E (cid:34)n n

Original: [Equation]
Revised: [Equation]

Original: i =
Revised: i =

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13) g
Revised: (cid:13)(cid:13) g

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: − h
Revised: − h

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: (cid:35) + ν E (cid:104)(cid:13)(cid:13)
Revised: (cid:35) + ν E (cid:104)(cid:13)(cid:13)

Original: h t +1 − ∇ f ( x t +1 ) (cid:13)(cid:13) 2 (cid:105)
Revised: h t +1 − ∇ f ( x t +1 ) (cid:13)(cid:13) 2 (cid:105)

Original: + ρ E
Revised: + ρ E

Original: [Equation]
Revised: [Equation]

Original: n n (cid:88) i =
Revised: n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: h
Revised: h

Original: [Equation]
Revised: [Equation]

Original: +1 i − ∇ f i ( x t +1 )
Revised: +1 i − ∇ f i ( x t +1 )

Original: (cid:13)(cid:13)(cid:35) ≤ E (cid:20) f
Revised: (cid:13)(cid:13)(cid:35) ≤ E (cid:20) f

Original: [Equation]
Revised: [Equation]

Original: γ 2 (cid:13)(cid:13)
Revised: γ 2 (cid:13)(cid:13)

Original: ∇ f ( x t ) (cid:13)(cid:13) 2 − (cid:18) 1 2 γ − L 2 (cid:19) (cid:13)(cid:13)
Revised: ∇ f ( x t ) (cid:13)(cid:13) 2 − (cid:18) 1 2 γ − L 2 (cid:19) (cid:13)(cid:13)

Original: x t +1 − x t (cid:13)(cid:13) 2 + γ (cid:13)(cid:13)
Revised: x t +1 − x t (cid:13)(cid:13) 2 + γ (cid:13)(cid:13)

Original: h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:21) + (1 − γµ ) 2 γ (2 ω + 1)E (cid:104)(cid:13)(cid:13) g t − h t (cid:13)(cid:13) 2 (cid:105) + (1 − γµ ) 8 γω n E (cid:34)n n (cid:88)
Revised: h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:21) + (1 − γµ ) 2 γ (2 ω + 1)E (cid:104)(cid:13)(cid:13) g t − h t (cid:13)(cid:13) 2 (cid:105) + (1 − γµ ) 8 γω n E (cid:34)n n (cid:88)

Original: i =
Revised: i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g ti − h ti (cid:13)(cid:13)(cid:35) + 20 γω (2 ω + 1) n
Revised: g ti − h ti (cid:13)(cid:13)(cid:35) + 20 γω (2 ω + 1) n

Original: E (cid:34) 2 b 2 σ
Revised: E (cid:34) 2 b 2 σ

Original: B +(cid:32) (1 − b ) 2 L 2 σ
Revised: B +(cid:32) (1 − b ) 2 L 2 σ

Original: B + (cid:98) L(cid:33) (cid:13)(cid:13) x
Revised: B + (cid:98) L(cid:33) (cid:13)(cid:13) x

Original: t +1 − x t (cid:13)(cid:13) 

Revised: t +1 − x t (cid:13)(cid:13)

Original: + 2 b 2 1 n n (cid:88) i =
Revised: + 2 b 2 1 n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: h ti − ∇ f i ( x t ) (cid:13)(cid:13)(cid:35) + ν E (cid:34) 2 b 2 σnB + 2 (1 − b ) 2 L 2 σ nB (cid:13)(cid:13) x
Revised: h ti − ∇ f i ( x t ) (cid:13)(cid:13)(cid:35) + ν E (cid:34) 2 b 2 σnB + 2 (1 − b ) 2 L 2 σ nB (cid:13)(cid:13) x

Original: t +1 − x t (cid:13)(cid:13) 2 + (1 − b ) 

Revised: t +1 − x t (cid:13)(cid:13) 2 + (1 − b )

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: h t − ∇ f ( x t ) (cid:13)(cid:13)(cid:35) + ρ E (cid:34) 2 b 2 σ
Revised: h t − ∇ f ( x t ) (cid:13)(cid:13)(cid:35) + ρ E (cid:34) 2 b 2 σ

Original: B + 2 (1 − b ) 2 L 2 σ
Revised: B + 2 (1 − b ) 2 L 2 σ

Original: B (cid:13)(cid:13)
Revised: B (cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: By taking ν = 2 γb , one can see that γ + ν (1 − b ) 2 ≤ (cid:0) 1 − b 2 (cid:1) ν, and
Revised: By taking ν = 2 γb , one can see that γ + ν (1 − b ) 2 ≤ (cid:0) 1 − b 2 (cid:1) ν, and

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: n E
Revised: n E

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: i =
Revised: i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: h t
Revised: h t

Original: +

Revised: +

Original: i − ∇ f i ( x t +1 ) (cid:13)(cid:13)(cid:35) ≤ E (cid:2) f ( x t ) (cid:3) − γ 2 E (cid:104)(cid:13)(cid:13) ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105)
Revised: i − ∇ f i ( x t +1 ) (cid:13)(cid:13)(cid:35) ≤ E (cid:2) f ( x t ) (cid:3) − γ 2 E (cid:104)(cid:13)(cid:13) ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105)

Original: + (1 − γµ ) 2 γ (2 ω
Revised: + (1 − γµ ) 2 γ (2 ω

Original: + 1)E (cid:104)(cid:13)(cid:13) g t − h t (cid:13)(cid:13) 2 (cid:105) + (1 − γµ ) 8 γω n E (cid:34)n n (cid:88)
Revised: + 1)E (cid:104)(cid:13)(cid:13) g t − h t (cid:13)(cid:13) 2 (cid:105) + (1 − γµ ) 8 γω n E (cid:34)n n (cid:88)

Original: i =
Revised: i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g ti − h ti (cid:13)(cid:13)(cid:35) + (cid:18) 1 − b(cid:19)
Revised: g ti − h ti (cid:13)(cid:13)(cid:35) + (cid:18) 1 − b(cid:19)

Original: 2 γ b E (cid:104)(cid:13)(cid:13) h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105)
Revised: 2 γ b E (cid:104)(cid:13)(cid:13) h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105)

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: − 40 γω (
Revised: − 40 γω (

Original: [Equation]
Revised: [Equation]

Original: 1) (cid:16) (
Revised: 1) (cid:16) (

Original: [Equation]
Revised: [Equation]

Original: b
Revised: b

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: B + (cid:98)
Revised: B + (cid:98)

Original: [Equation]
Revised: [Equation]

Original: (cid:17) n − 4 γ (
Revised: (cid:17) n − 4 γ (

Original: [Equation]
Revised: [Equation]

Original: b )
Revised: b )

Original: [Equation]
Revised: [Equation]

Original: 2 σ bnB − 2 ρ (1 − b )
Revised: 2 σ bnB − 2 ρ (1 − b )

Original: [Equation]
Revised: [Equation]

Original: L 2 σ
Revised: L 2 σ

Original: B 
Revised: B 

Original:  E (cid:104) (cid:13)(cid:13) x t +1 − x t (cid:13)(cid:13) 2 (cid:105) + (cid:18) 40 b 2 γω (2 ω + 1) n + ρ (1 − b ) 2 (cid:19) E (cid:34)n n (cid:88)
Revised:  E (cid:104) (cid:13)(cid:13) x t +1 − x t (cid:13)(cid:13) 2 (cid:105) + (cid:18) 40 b 2 γω (2 ω + 1) n + ρ (1 − b ) 2 (cid:19) E (cid:34)n n (cid:88)

Original: i =
Revised: i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: h ti − ∇ f i ( x t ) (cid:13)(cid:13)(cid:35) +(cid:18) 20 γω (2 ω + 1) nB + 2 γ bnB + ρ
Revised: h ti − ∇ f i ( x t ) (cid:13)(cid:13)(cid:35) +(cid:18) 20 γω (2 ω + 1) nB + 2 γ bnB + ρ

Original: B (cid:19) b 2 σ 2 .
Revised: B (cid:19) b 2 σ 2 .

Original: Next, we ﬁx ρ = 80 bγω (2 ω +1) n .
Revised: Next, we ﬁx ρ = 80 bγω (2 ω +1) n .

Original: With this choice of ρ and for all b ∈ (0 , 1] , we can show that 40 b 2 γω (2 ω +1) n + ρ (1 − b ) 2 ≤ (cid:0) 1 − b 2 (cid:1) ρ, thus
Revised: With this choice of ρ and for all b ∈ (0 , 1] , we can show that 40 b 2 γω (2 ω +1) n + ρ (1 − b ) 2 ≤ (cid:0) 1 − b 2 (cid:1) ρ, thus

Original: E (cid:2) f
Revised: E (cid:2) f

Original: ( x t +1 ) (cid:3) + 2 γ (2 ω + 1)E (cid:104)(cid:13)(cid:13) g t
Revised: ( x t +1 ) (cid:3) + 2 γ (2 ω + 1)E (cid:104)(cid:13)(cid:13) g t

Original: +1 − h t +1 (cid:13)(cid:13) 2 (cid:105) + 8 γω n E (cid:34)n n (cid:88) i =
Revised: +1 − h t +1 (cid:13)(cid:13) 2 (cid:105) + 8 γω n E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g t +

Revised: g t +

Original: i − h t
Revised: i − h t

Original: +1 i (cid:13)(cid:13) 2 (cid:35) + 2 γ b E (cid:104)(cid:13)(cid:13)
Revised: +1 i (cid:13)(cid:13) 2 (cid:35) + 2 γ b E (cid:104)(cid:13)(cid:13)

Original: h t +1 − ∇ f ( x t +1 ) (cid:13)(cid:13) 2 (cid:105) + 80 bγω (2 ω + 1) n
Revised: h t +1 − ∇ f ( x t +1 ) (cid:13)(cid:13) 2 (cid:105) + 80 bγω (2 ω + 1) n

Original: E (cid:34)n n (cid:88) i =
Revised: E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: h t
Revised: h t

Original: +

Revised: +

Original: i − ∇ f i ( x t +1 ) (cid:13)(cid:13)(cid:35)
Revised: i − ∇ f i ( x t +1 ) (cid:13)(cid:13)(cid:35)

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: In the last inequality we use b ∈ (0 , 1] .
Revised: In the last inequality we use b ∈ (0 , 1] .

Original: Next, considering the choice of γ and Lemma I.7, we get
Revised: Next, considering the choice of γ and Lemma I.7, we get

Original: E (cid:2) f
Revised: E (cid:2) f

Original: ( x t +1 ) (cid:3) + 2 γ (2 ω + 1)E (cid:104)(cid:13)(cid:13) g t
Revised: ( x t +1 ) (cid:3) + 2 γ (2 ω + 1)E (cid:104)(cid:13)(cid:13) g t

Original: +1 − h t +1 (cid:13)(cid:13) 2 (cid:105) + 8 γω n E (cid:34)n n (cid:88) i =
Revised: +1 − h t +1 (cid:13)(cid:13) 2 (cid:105) + 8 γω n E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g t +

Revised: g t +

Original: i − h t
Revised: i − h t

Original: +1 i (cid:13)(cid:13) 2 (cid:35) + 2 γ b E (cid:104)(cid:13)(cid:13)
Revised: +1 i (cid:13)(cid:13) 2 (cid:35) + 2 γ b E (cid:104)(cid:13)(cid:13)

Original: h t +1 − ∇ f ( x t +1 ) (cid:13)(cid:13) 2 (cid:105) + 80 bγω (2 ω + 1) n
Revised: h t +1 − ∇ f ( x t +1 ) (cid:13)(cid:13) 2 (cid:105) + 80 bγω (2 ω + 1) n

Original: E (cid:34)n n (cid:88) i =
Revised: E (cid:34)n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: h t
Revised: h t

Original: +

Revised: +

Original: i − ∇ f i ( x t +1 ) (cid:13)(cid:13)(cid:35) ≤ E (cid:2) f ( x t ) (cid:3) − γ 2 E (cid:104)(cid:13) (cid:13) ∇ f ( x t ) (cid:13) (cid:13) 2 (cid:105)
Revised: i − ∇ f i ( x t +1 ) (cid:13)(cid:13)(cid:35) ≤ E (cid:2) f ( x t ) (cid:3) − γ 2 E (cid:104)(cid:13) (cid:13) ∇ f ( x t ) (cid:13) (cid:13) 2 (cid:105)

Original: + (1 − γµ ) 2 γ (2 ω
Revised: + (1 − γµ ) 2 γ (2 ω

Original: + 1)E (cid:104)(cid:13)(cid:13) g t − h t (cid:13)(cid:13) 2 (cid:105) + (1 − γµ ) 8 γω n E (cid:34)n n (cid:88)
Revised: + 1)E (cid:104)(cid:13)(cid:13) g t − h t (cid:13)(cid:13) 2 (cid:105) + (1 − γµ ) 8 γω n E (cid:34)n n (cid:88)

Original: i =
Revised: i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g ti − h ti (cid:13)(cid:13)(cid:35)
Revised: g ti − h ti (cid:13)(cid:13)(cid:35)

Original: + (1 − γµ ) 2 γ b E
Revised: + (1 − γµ ) 2 γ b E

Original: (cid:104)(cid:13)(cid:13) h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105)
Revised: (cid:104)(cid:13)(cid:13) h t − ∇ f ( x t ) (cid:13)(cid:13) 2 (cid:105)

Original: [Equation]
Revised: [Equation]

Original: + (cid:18)
Revised: + (cid:18)

Original: [Equation]
Revised: [Equation]

Original: ω
Revised: ω

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: nB
Revised: nB

Original: [Equation]
Revised: [Equation]

Original: bnB (cid:19) b
Revised: bnB (cid:19) b

Original: [Equation]
Revised: [Equation]

Original: In the view of Lemma I.6 with
Revised: In the view of Lemma I.6 with

Original: [Equation]
Revised: [Equation]

Original: and C = (cid:16) 200 ω (2 ω
Revised: and C = (cid:16) 200 ω (2 ω

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: nB + 4 bnB (cid:17) b 2 σ 2 , we can conclude the proof.
Revised: nB + 4 bnB (cid:17) b 2 σ 2 , we can conclude the proof.

Original: Corollary I.16.
Revised: Corollary I.16.

Original: Suppose that assumptions from Theorem I.15 hold, momentum b =
Revised: Suppose that assumptions from Theorem I.15 hold, momentum b =

Original: Θ (cid:18) min (cid:26)
Revised: Θ (cid:18) min (cid:26)

Original: [Equation]
Revised: [Equation]

Original: 2 (cid:27)(cid:19) , and h 

Revised: 2 (cid:27)(cid:19) , and h

Original: i = g 0 i = 

Revised: i = g 0 i =

Original: for all i ∈ [ n ] , then Algorithm 1 needs
Revised: for all i ∈ [ n ] , then Algorithm 1 needs

Original: [Equation]
Revised: [Equation]

Original: communication rounds to get an ε -solution, the communication complexity is equal to O ( ζ C T ) , and the number of stochastic gradient calculations per node equals O ( BT ) , where ζ C is the expected density from Deﬁnition 1.3.
Revised: communication rounds to get an ε -solution, the communication complexity is equal to O ( ζ C T ) , and the number of stochastic gradient calculations per node equals O ( BT ) , where ζ C is the expected density from Deﬁnition 1.3.

Original: Proof.
Revised: Proof.

Original: Considering the choice of b, we have 1 µ (cid:16) 200 ω (2 ω +1) nB
Revised: Considering the choice of b, we have 1 µ (cid:16) 200 ω (2 ω +1) nB

Original: + 4 bnB (cid:17) b 2 σ 2 =
Revised: + 4 bnB (cid:17) b 2 σ 2 =

Original: O ( ε ) .
Revised: O ( ε ) .

Original: Therefore, is it enough to take the number of communication rounds equals (27) to get an ε -solution.
Revised: Therefore, is it enough to take the number of communication rounds equals (27) to get an ε -solution.

Original: In the view of Algorithm 1 and the fact that we use a mini-batch of stochastic gradients, the communication complexity is equal to O ( ζ C T ) and the number of stochastic gradients that each node calculates equals O ( BT ) .
Revised: In the view of Algorithm 1 and the fact that we use a mini-batch of stochastic gradients, the communication complexity is equal to O ( ζ C T ) and the number of stochastic gradients that each node calculates equals O ( BT ) .

Original: Unlike Corollary 6.8, in this corollary, we can initialize h 

Revised: Unlike Corollary 6.8, in this corollary, we can initialize h

Original: i and g 0 i , for instance, with zeros because the corresponding initialization error Ψ 0 from the proof of Theorem I.15 would be under the logarithm.
Revised: i and g 0 i , for instance, with zeros because the corresponding initialization error Ψ 0 from the proof of Theorem I.15 would be under the logarithm.

Original: I. C ASE OF DASHA-SYNC-MVR Comparing
Revised: I. C ASE OF DASHA-SYNC-MVR Comparing

Original: Algorithm 1 and Algorithm 2, one can see that Algorithm 2 has the third source of randomness from c t +1 .
Revised: Algorithm 1 and Algorithm 2, one can see that Algorithm 2 has the third source of randomness from c t +1 .

Original: In this section, we deﬁne E p [ · ] to be a conditional expectation w.r.t.
Revised: In this section, we deﬁne E p [ · ] to be a conditional expectation w.r.t.

Original: c t + conditioned on all previous randomness.
Revised: c t + conditioned on all previous randomness.

Original: And we deﬁne E t +1 [ · ] to be a conditional expectation w.r.t.
Revised: And we deﬁne E t +1 [ · ] to be a conditional expectation w.r.t.

Original: c t +1 , {C i } ni =1 , { h t +1 i } ni =1 conditioned on all previous randomness.
Revised: c t +1 , {C i } ni =1 , { h t +1 i } ni =1 conditioned on all previous randomness.

Original: Note, that E t +1 [ · ] =
Revised: Note, that E t +1 [ · ] =

Original: E h
Revised: E h

Original: [E C [E p [ · ]]] .
Revised: [E C [E p [ · ]]] .

Original: Lemma I.17.
Revised: Lemma I.17.

Original: Suppose that Assumptions 5.3, 5.5 and 1.2 hold and let us consider sequences { g t +1 i } ni =1 and { h t +1 i } ni =1 from Algorithm 2, then
Revised: Suppose that Assumptions 5.3, 5.5 and 1.2 hold and let us consider sequences { g t +1 i } ni =1 and { h t +1 i } ni =1 from Algorithm 2, then

Original: E t +1 (cid:104)(cid:13)(cid:13) g t +1 − h t
Revised: E t +1 (cid:104)(cid:13)(cid:13) g t +1 − h t

Original: +

Revised: +

Original: (cid:13)(cid:13) 2 (cid:105) ≤ 2 ω (1 − p ) (cid:16)
Revised: (cid:13)(cid:13) 2 (cid:105) ≤ 2 ω (1 − p ) (cid:16)

Original: L 2 σ B
Revised: L 2 σ B

Original: + (cid:98) L 2 (cid:17) n (cid:13)(cid:13) x
Revised: + (cid:98) L 2 (cid:17) n (cid:13)(cid:13) x

Original: t +1 − x t (cid:13)(cid:13) 

Revised: t +1 − x t (cid:13)(cid:13)

Original: + 2 a 2 ω (1 − p ) nn (cid:88)
Revised: + 2 a 2 ω (1 − p ) nn (cid:88)

Original: i =
Revised: i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g ti − h ti (cid:13)(cid:13) 

Revised: g ti − h ti (cid:13)(cid:13)

Original: + (1 − p ) (1 − a ) 2 (cid:13)(cid:13)
Revised: + (1 − p ) (1 − a ) 2 (cid:13)(cid:13)

Original: g t − h t (cid:13)(cid:13) 2 , and
Revised: g t − h t (cid:13)(cid:13) 2 , and

Original: E t +1 (cid:104)(cid:13)(cid:13) g t
Revised: E t +1 (cid:104)(cid:13)(cid:13) g t

Original: +1 i − h t
Revised: +1 i − h t

Original: +1 i (cid:13)(cid:13) 2 (cid:105) ≤ 2 ω (1 − p ) (cid:18)
Revised: +1 i (cid:13)(cid:13) 2 (cid:105) ≤ 2 ω (1 − p ) (cid:18)

Original: L 2 σ
Revised: L 2 σ

Original: B + L 2 i (cid:19) (cid:13)(cid:13) x t +1 − x t (cid:13)(cid:13) 

Revised: B + L 2 i (cid:19) (cid:13)(cid:13) x t +1 − x t (cid:13)(cid:13)

Original: + (1 − p ) (cid:16) 2 a 2 ω + (1 − a ) 2 (cid:17) (cid:13)(cid:13) g ti − h ti (cid:13)(cid:13) 2 , ∀ i ∈ [ n ] .
Revised: + (1 − p ) (cid:16) 2 a 2 ω + (1 − a ) 2 (cid:17) (cid:13)(cid:13) g ti − h ti (cid:13)(cid:13) 2 , ∀ i ∈ [ n ] .

Original: Proof.
Revised: Proof.

Original: First, we estimate
Revised: First, we estimate

Original: E t +1 (cid:104)(cid:13)(cid:13) g t +1 − h t
Revised: E t +1 (cid:104)(cid:13)(cid:13) g t +1 − h t

Original: +

Revised: +

Original: (cid:13)(cid:13) 2 (cid:105) .
Revised: (cid:13)(cid:13) 2 (cid:105) .

Original: Let us denote h t +1 i,= 1 B (cid:80) Bj =1 ∇ f i ( x t +1 ; ξ t +1 ij )
Revised: Let us denote h t +1 i,= 1 B (cid:80) Bj =1 ∇ f i ( x t +1 ; ξ t +1 ij )

Original: + h ti − 1 B (cid:80)
Revised: + h ti − 1 B (cid:80)

Original: Bj =1 ∇ f i ( x t ; ξ t +1 ij ) .
Revised: Bj =1 ∇ f i ( x t ; ξ t +1 ij ) .

Original: E t +1 (cid:104)(cid:13)(cid:13) g t +1 − h t
Revised: E t +1 (cid:104)(cid:13)(cid:13) g t +1 − h t

Original: +

Revised: +

Original: (cid:13)(cid:13) 2 (cid:105) =
Revised: (cid:13)(cid:13) 2 (cid:105) =

Original: E t +1 (cid:104) E p
Revised: E t +1 (cid:104) E p

Original: (cid:104)(cid:13)(cid:13) g t +1 − h t +1 (cid:13)(cid:13) 2 (cid:105)(cid:105) =
Revised: (cid:104)(cid:13)(cid:13) g t +1 − h t +1 (cid:13)(cid:13) 2 (cid:105)(cid:105) =

Original: (1 − p )
Revised: (1 − p )

Original: E t +  
Revised: E t +  

Original: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) g t + 1 n n (cid:88) i =
Revised: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) g t + 1 n n (cid:88) i =

Original: C i (cid:0) h t +1 i, 0 − h ti − a (cid:0)
Revised: C i (cid:0) h t +1 i, 0 − h ti − a (cid:0)

Original: g ti − h ti (cid:1)(cid:1) − 1 n n (cid:88) i
Revised: g ti − h ti (cid:1)(cid:1) − 1 n n (cid:88) i

Original: = h t
Revised: = h t

Original: +1 i, 0 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   (4) , (15) =
Revised: +1 i, 0 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   (4) , (15) =

Original: (1 − p )
Revised: (1 − p )

Original: E
Revised: E

Original: h  
Revised: h  

Original: E C   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 n n (cid:88)
Revised: E C   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 n n (cid:88)

Original: i =
Revised: i =

Original: C i (cid:0)
Revised: C i (cid:0)

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Using the independence of compressors and (4)
Revised: Using the independence of compressors and (4)

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: ( x t +1 )
Revised: ( x t +1 )

Original: − ∇ f i ( x t ) (cid:13)(cid:13)(cid:33) + 2 a 2 ω (1 − p ) nn (cid:88) i =
Revised: − ∇ f i ( x t ) (cid:13)(cid:13)(cid:33) + 2 a 2 ω (1 − p ) nn (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g ti − h ti (cid:13)(cid:13) 

Revised: g ti − h ti (cid:13)(cid:13)

Original: + (1 − p ) (1 − a ) 2 (cid:13)(cid:13)
Revised: + (1 − p ) (1 − a ) 2 (cid:13)(cid:13)

Original: g t − h t (cid:13)(cid:13)= 2 ω (1 − p ) nn (cid:88)
Revised: g t − h t (cid:13)(cid:13)= 2 ω (1 − p ) nn (cid:88)

Original: i = (cid:32)
Revised: i = (cid:32)

Original: B
Revised: B

Original: B (cid:88)
Revised: B (cid:88)

Original: j =
Revised: j =

Original: E h (cid:104)(cid:13)(cid:13) ∇ f i ( x t +1 ; ξ t +1 ij ) − ∇ f i ( x t ; ξ t +1 ij ) − (cid:0) ∇ f i ( x t +1 )
Revised: E h (cid:104)(cid:13)(cid:13) ∇ f i ( x t +1 ; ξ t +1 ij ) − ∇ f i ( x t ; ξ t +1 ij ) − (cid:0) ∇ f i ( x t +1 )

Original: − ∇ f i ( x t ) (cid:1)(cid:13)(cid:13) 2 (cid:105) + (cid:13)(cid:13) ∇ f i ( x t +1 )
Revised: − ∇ f i ( x t ) (cid:1)(cid:13)(cid:13) 2 (cid:105) + (cid:13)(cid:13) ∇ f i ( x t +1 )

Original: − ∇ f i ( x t ) (cid:13)(cid:13)(cid:33) + 2 a 2 ω (1 − p ) nn (cid:88) i = (cid:13) (cid:13) g t
Revised: − ∇ f i ( x t ) (cid:13)(cid:13)(cid:33) + 2 a 2 ω (1 − p ) nn (cid:88) i = (cid:13) (cid:13) g t

Original: i − h t
Revised: i − h t

Original: i (cid:13) (cid:13) 2 + (1 − p ) (1 − a ) 2 (cid:13) (cid:13) g t − h t (cid:13) (cid:13)≤ 2 ω (1 − p ) (cid:16) L 2 σ B + (cid:98) L 2 (cid:17) n (cid:13)(cid:13) x
Revised: i (cid:13) (cid:13) 2 + (1 − p ) (1 − a ) 2 (cid:13) (cid:13) g t − h t (cid:13) (cid:13)≤ 2 ω (1 − p ) (cid:16) L 2 σ B + (cid:98) L 2 (cid:17) n (cid:13)(cid:13) x

Original: t +1 − x t (cid:13)(cid:13) 

Revised: t +1 − x t (cid:13)(cid:13)

Original: + 2 a 2 ω (1 − p ) nn (cid:88)
Revised: + 2 a 2 ω (1 − p ) nn (cid:88)

Original: i =
Revised: i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: g ti − h ti (cid:13)(cid:13) 

Revised: g ti − h ti (cid:13)(cid:13)

Original: + (1 − p ) (1 − a ) 2 (cid:13)(cid:13)
Revised: + (1 − p ) (1 − a ) 2 (cid:13)(cid:13)

Original: g t − h t (cid:13)(cid:13) 2 , where in the inequalities we use Assumptions 1.2, 5.5 and 5.3, and (14).
Revised: g t − h t (cid:13)(cid:13) 2 , where in the inequalities we use Assumptions 1.2, 5.5 and 5.3, and (14).

Original: Analogously, we can get the bound for E t +1 (cid:104)(cid:13)(cid:13)
Revised: Analogously, we can get the bound for E t +1 (cid:104)(cid:13)(cid:13)

Original: g t +

Revised: g t +

Original: i − h t
Revised: i − h t

Original: +1 i (cid:13)(cid:13) 2 (cid:105) for all i ∈ [ n ] :
Revised: +1 i (cid:13)(cid:13) 2 (cid:105) for all i ∈ [ n ] :

Original: E t +1 (cid:104)(cid:13)(cid:13) g t
Revised: E t +1 (cid:104)(cid:13)(cid:13) g t

Original: +1 i − h t
Revised: +1 i − h t

Original: +1 i (cid:13)(cid:13) 2 (cid:105) =
Revised: +1 i (cid:13)(cid:13) 2 (cid:105) =

Original: E t +1 (cid:104) E p
Revised: E t +1 (cid:104) E p

Original: (cid:104)(cid:13)(cid:13) g t
Revised: (cid:104)(cid:13)(cid:13) g t

Original: +1 i − h t
Revised: +1 i − h t

Original: +1 i (cid:13)(cid:13) 2 (cid:105)(cid:105) =
Revised: +1 i (cid:13)(cid:13) 2 (cid:105)(cid:105) =

Original: (1 − p )
Revised: (1 − p )

Original: E t +1 (cid:104)(cid:13)(cid:13)
Revised: E t +1 (cid:104)(cid:13)(cid:13)

Original: g ti + C i (cid:0) h t
Revised: g ti + C i (cid:0) h t

Original: +1 i, 0 − h ti − a (cid:0) g ti − h ti (cid:1)(cid:1) − h t
Revised: +1 i, 0 − h ti − a (cid:0) g ti − h ti (cid:1)(cid:1) − h t

Original: +1 i, 0 (cid:13)(cid:13) 2 (cid:105) ≤ 2 ω (1 − p ) (cid:18)
Revised: +1 i, 0 (cid:13)(cid:13) 2 (cid:105) ≤ 2 ω (1 − p ) (cid:18)

Original: L 2 σ
Revised: L 2 σ

Original: B + L 2 i (cid:19) (cid:13)(cid:13) x t +1 − x t (cid:13)(cid:13) 2 + 2 a 2 ω (1 − p ) (cid:13)(cid:13)
Revised: B + L 2 i (cid:19) (cid:13)(cid:13) x t +1 − x t (cid:13)(cid:13) 2 + 2 a 2 ω (1 − p ) (cid:13)(cid:13)

Original: g ti − h ti (cid:13)(cid:13) 

Revised: g ti − h ti (cid:13)(cid:13)

Original: + (1 − p ) (1 − a ) 2 (cid:13)(cid:13)
Revised: + (1 − p ) (1 − a ) 2 (cid:13)(cid:13)

Original: g ti − h ti (cid:13)(cid:13) 2 .
Revised: g ti − h ti (cid:13)(cid:13) 2 .

Original: We introduce new notations: ∇ f i ( x t +1 ; ξ t +1 i )
Revised: We introduce new notations: ∇ f i ( x t +1 ; ξ t +1 i )

Original: = 1 B (cid:80)
Revised: = 1 B (cid:80)

Original: Bj =1 ∇ f i ( x t +1 ; ξ t +1 ij ) and ∇ f ( x t +1 ; ξ t +1 )
Revised: Bj =1 ∇ f i ( x t +1 ; ξ t +1 ij ) and ∇ f ( x t +1 ; ξ t +1 )

Original: = 1 n (cid:80) ni
Revised: = 1 n (cid:80) ni

Original: =1 ∇ f i ( x t +1 ; ξ t +1 i ) .
Revised: =1 ∇ f i ( x t +1 ; ξ t +1 i ) .

Original: Lemma I.18.
Revised: Lemma I.18.