Original: Variance Reduced ProxSkip: Algorithm, Theory and Application to Federated Learning
Revised: Variance Reduced ProxSkip: Algorithm, Theory and Application to Federated Learning

Original: Abstract
Revised: Abstract

Original: We study distributed optimization methods based on the local training (LT) paradigm: achieving communication efﬁciency by performing richer local gradientbased training on the clients before parameter averaging.
Revised: We study distributed optimization methods based on the local training (LT) paradigm: achieving communication efﬁciency by performing richer local gradientbased training on the clients before parameter averaging.

Original: Looking back at the progress of the ﬁeld, we identify 5 generations of LT methods : 1) heuristic, 2) homogeneous, 3) sublinear, 4) linear, and 5) accelerated.
Revised: Looking back at the progress of the ﬁeld, we identify 5 generations of LT methods : 1) heuristic, 2) homogeneous, 3) sublinear, 4) linear, and 5) accelerated.

Original: The 5 th generation, initiated by the ProxSkip method of Mishchenko et al.
Revised: The 5 th generation, initiated by the ProxSkip method of Mishchenko et al.

Original: [2022] and its analysis, is characterized by the ﬁrst theoretical conﬁrmation that LT is a communication acceleration mechanism.
Revised: [2022] and its analysis, is characterized by the ﬁrst theoretical conﬁrmation that LT is a communication acceleration mechanism.

Original: Inspired by this recent progress, we contribute to the 5 th generation of LT methods by showing that it is possible to enhance them further using variance reduction .
Revised: Inspired by this recent progress, we contribute to the 5 th generation of LT methods by showing that it is possible to enhance them further using variance reduction .

Original: While all previous theoretical results for LT methods ignore the cost of local work altogether, and are framed purely in terms of the number of communication rounds, we show that our methods can be substantially faster in terms of the total training cost than the state-of-the-art method ProxSkip in theory and practice in the regime when local computation is sufﬁciently expensive.
Revised: While all previous theoretical results for LT methods ignore the cost of local work altogether, and are framed purely in terms of the number of communication rounds, we show that our methods can be substantially faster in terms of the total training cost than the state-of-the-art method ProxSkip in theory and practice in the regime when local computation is sufﬁciently expensive.

Original: We characterize this threshold theoretically, and conﬁrm our theoretical predictions with empirical results.
Revised: We characterize this threshold theoretically, and conﬁrm our theoretical predictions with empirical results.

Original:  Introduction
Revised:  Introduction

Original: Announced in April 2017 in a Google AI blog [McMahan and Ramage, 2017], and citing four foundational papers [Kone ˇ cný et al., 2016b,a, McMahan et al., 2017, Bonawitz et al., 2017] of what was to become a new and rapidly growing interdisciplinary ﬁeld, federated learning (FL) constitutes a novel paradigm for training supervised machine learning models.
Revised: Announced in April 2017 in a Google AI blog [McMahan and Ramage, 2017], and citing four foundational papers [Kone ˇ cný et al., 2016b,a, McMahan et al., 2017, Bonawitz et al., 2017] of what was to become a new and rapidly growing interdisciplinary ﬁeld, federated learning (FL) constitutes a novel paradigm for training supervised machine learning models.

Original: The key idea is the acknowledgement that increasing amounts of data are being captured and stored on edge devices, such as mobile phones, sensors and hospital workstations and that moving the data to a datacenter for centralized processing may be infeasible or undesirable for various reasons, including high energy costs and data privacy concerns [Kairouz et al., 2019, Li et al., 2020a].
Revised: The key idea is the acknowledgement that increasing amounts of data are being captured and stored on edge devices, such as mobile phones, sensors and hospital workstations and that moving the data to a datacenter for centralized processing may be infeasible or undesirable for various reasons, including high energy costs and data privacy concerns [Kairouz et al., 2019, Li et al., 2020a].

Original: FL faces a multitude of challenges which are being actively addressed by the research community.
Revised: FL faces a multitude of challenges which are being actively addressed by the research community.

Original: 1.1 Formalism
Revised: 1.1 Formalism

Original: We study the standard optimization formulation of federated learning [Koneˇcný et al., 2016a, McMahan et al., 2017, Kairouz et al., 2019, Wang et al., 2021] given by
Revised: We study the standard optimization formulation of federated learning [Koneˇcný et al., 2016a, McMahan et al., 2017, Kairouz et al., 2019, Wang et al., 2021] given by

Original: [Equation]
Revised: [Equation]

Original: where M is the number of clients (devices, machines, workers), n i is the number of training data points on client i ∈ { 1 , 2 , . . .
Revised: where M is the number of clients (devices, machines, workers), n i is the number of training data points on client i ∈ { 1 , 2 , . . .

Original: , M } , and n := (cid:80)
Revised: , M } , and n := (cid:80)

Original: Mi
Revised: Mi

Original: =1 n i is the total number of training data points collectively owned by this federation of M clients.
Revised: =1 n i is the total number of training data points collectively owned by this federation of M clients.

Original: Note that φ is the empirical risk over the federated dataset.
Revised: Note that φ is the empirical risk over the federated dataset.

Original: Perhaps conceptually the simplest method for solving (1) is gradient descent ( GD ) ,
Revised: Perhaps conceptually the simplest method for solving (1) is gradient descent ( GD ) ,

Original: [Equation]
Revised: [Equation]

Original: where γ > 0 is the stepsize.
Revised: where γ > 0 is the stepsize.

Original: It will be useful to describe how GD would be implemented in a federated environment.
Revised: It will be useful to describe how GD would be implemented in a federated environment.

Original: First, all clients i ∈ { 1 , . .
Revised: First, all clients i ∈ { 1 , . .

Original: , M } in parallel perform a single local gradient step starting from the current global model x t , arriving at the local models x it := x t − γ ∇ φ i ( x t ) , i ∈ { 1 , . . .
Revised: , M } in parallel perform a single local gradient step starting from the current global model x t , arriving at the local models x it := x t − γ ∇ φ i ( x t ) , i ∈ { 1 , . . .

Original: , M } .
Revised: , M } .

Original: These local models are then communicated to the orchestrating server , which aggregates them via weighted averaging, arriving at the new global model x t +1 =
Revised: These local models are then communicated to the orchestrating server , which aggregates them via weighted averaging, arriving at the new global model x t +1 =

Original: (cid:80) Mi =

Revised: (cid:80) Mi =

Original: n i n x it .
Revised: n i n x it .

Original: This new model is then broadcast back to all clients, and the process is repeated until a model of sufﬁcient quality is found.
Revised: This new model is then broadcast back to all clients, and the process is repeated until a model of sufﬁcient quality is found.

Original: 1.2 Federated averaging
Revised: 1.2 Federated averaging

Original: Proposed by Povey et al.
Revised: Proposed by Povey et al.

Original: [2015], Moritz et al.
Revised: [2015], Moritz et al.

Original: [2016], McMahan et al.
Revised: [2016], McMahan et al.

Original: [2017], federated averaging ( FedAvg ) is arguably the most popular method for solving the standard FL formulation (1).
Revised: [2017], federated averaging ( FedAvg ) is arguably the most popular method for solving the standard FL formulation (1).

Original: Motivated by the speciﬁc constraints of federated environments, FedAvg can be seen as a practical enhancement of GD via the simultaneous application of three techniques: a) data sampling (DS), b) client sampling (CS), and c) local training (LT).
Revised: Motivated by the speciﬁc constraints of federated environments, FedAvg can be seen as a practical enhancement of GD via the simultaneous application of three techniques: a) data sampling (DS), b) client sampling (CS), and c) local training (LT).

Original: That is,
Revised: That is,

Original: [Equation]
Revised: [Equation]

Original: We will now brieﬂy describe each of these three GD -enhancing techniques separately
Revised: We will now brieﬂy describe each of these three GD -enhancing techniques separately

Original: (a) GD + Data Sampling.
Revised: (a) GD + Data Sampling.

Original: In situations when the local datasets are so large that the computation of the exact local gradients becomes a bottleneck, it makes sense to approximate them via data sampling.
Revised: In situations when the local datasets are so large that the computation of the exact local gradients becomes a bottleneck, it makes sense to approximate them via data sampling.

Original: That is, instead of passing through all local data to compute the local gradient ∇ φ i ( x t ) , each client i computes the gradients ∇ φ ij ( x t ) for j ∈ D it only, where D it is a suitably chosen small-enough subset of the local dataset { 1 , . . .
Revised: That is, instead of passing through all local data to compute the local gradient ∇ φ i ( x t ) , each client i computes the gradients ∇ φ ij ( x t ) for j ∈ D it only, where D it is a suitably chosen small-enough subset of the local dataset { 1 , . . .

Original: , n i } .
Revised: , n i } .

Original: These gradients are then used to form gradient estimators
Revised: These gradients are then used to form gradient estimators

Original: g i ( x t ) ≈ ∇ φ i ( x t ) which are used to perform a local SGD step on all clients.
Revised: g i ( x t ) ≈ ∇ φ i ( x t ) which are used to perform a local SGD step on all clients.

Original: The rest of the procedure is the same as in the case of GD .
Revised: The rest of the procedure is the same as in the case of GD .

Original: That is, the local models obtained in this way are sent to the orchestrating server, the server aggregates them via weighted averaging and broadcasts the resulting model back to all clients.
Revised: That is, the local models obtained in this way are sent to the orchestrating server, the server aggregates them via weighted averaging and broadcasts the resulting model back to all clients.

Original: Combination of GD and DS can be seen as a particular version of SGD , where the stochastic gradient estimator is formed from the gradients ∇ φ ij ( x t ) associated with the datapoints ( i, j ) where j ∈ ∪ Mi =1 D it .
Revised: Combination of GD and DS can be seen as a particular version of SGD , where the stochastic gradient estimator is formed from the gradients ∇ φ ij ( x t ) associated with the datapoints ( i, j ) where j ∈ ∪ Mi =1 D it .

Original: While DS is still an active area of research, it has been studied for a long time, and is in general well understood [Taká ˇ c et al., 2013, Li et al., 2014, Csiba and Richtárik, 2018, Gower et al., 2019a, Horváth and Richtárik, 2019, Khaled and Richtárik, 2020].
Revised: While DS is still an active area of research, it has been studied for a long time, and is in general well understood [Taká ˇ c et al., 2013, Li et al., 2014, Csiba and Richtárik, 2018, Gower et al., 2019a, Horváth and Richtárik, 2019, Khaled and Richtárik, 2020].

Original: (b) GD + Client Sampling.
Revised: (b) GD + Client Sampling.

Original: In practical federated environments, and especially in crossdevice FL [Kairouz et al., 2019], the number of clients is enormous, they are not all available at all times, and the orchestrating server has limited compute and memory capacity.
Revised: In practical federated environments, and especially in crossdevice FL [Kairouz et al., 2019], the number of clients is enormous, they are not all available at all times, and the orchestrating server has limited compute and memory capacity.

Original: For these and other reasons, practical FL methods need to work in an environment in which a small subset S t ⊆ { 1 , . . .
Revised: For these and other reasons, practical FL methods need to work in an environment in which a small subset S t ⊆ { 1 , . . .

Original: , M } of the clients is sampled (“participates”) in each communication/aggregation/training round only.
Revised: , M } of the clients is sampled (“participates”) in each communication/aggregation/training round only.

Original: Since only the participating clients i ∈ S t perform a local
Revised: Since only the participating clients i ∈ S t perform a local

Original: GD step and communicate the resulting local model to the orchestrating server for aggregation, this induces an error compared to GD , which has an adverse effect on the convergence rate.
Revised: GD step and communicate the resulting local model to the orchestrating server for aggregation, this induces an error compared to GD , which has an adverse effect on the convergence rate.

Original: Combination of GD and CS can be seen as a particular version of SGD , where the stochastic gradient estimator is formed from the gradients ∇ φ ij ( x t ) associated with the datapoints ( i, j ) where j ∈ ∪ S t i =1 { 1 , . . .
Revised: Combination of GD and CS can be seen as a particular version of SGD , where the stochastic gradient estimator is formed from the gradients ∇ φ ij ( x t ) associated with the datapoints ( i, j ) where j ∈ ∪ S t i =1 { 1 , . . .

Original: , n i } .
Revised: , n i } .

Original: While CS is still an active area of research, since CS is a special type of DS, much was known about CS long before
Revised: While CS is still an active area of research, since CS is a special type of DS, much was known about CS long before

Original: FedAvg was proposed [Gower et al., 2019a, Horváth and Richtárik, 2019].
Revised: FedAvg was proposed [Gower et al., 2019a, Horváth and Richtárik, 2019].

Original: Still, CS poses new challenges tackled by the community Eichner et al.
Revised: Still, CS poses new challenges tackled by the community Eichner et al.

Original: [2019], Chen et al.
Revised: [2019], Chen et al.

Original: [2020], Gower et al.
Revised: [2020], Gower et al.

Original: [2019a], Cho et al.
Revised: [2019a], Cho et al.

Original: [2020], Charles et al.
Revised: [2020], Charles et al.

Original: (c) GD + Local Training.
Revised: (c) GD + Local Training.

Original: In federated learning, the cost of communication between the clients and the orchestrating server forms the key bottleneck.
Revised: In federated learning, the cost of communication between the clients and the orchestrating server forms the key bottleneck.

Original: Indeed, in their FedAvg paper, which introduced LT to the world of federated learning, McMahan et al.
Revised: Indeed, in their FedAvg paper, which introduced LT to the world of federated learning, McMahan et al.

Original: [2017] wrote:
Revised: [2017] wrote:

Original: [Table]
Revised: [Table]

Original: “In contrast 1 , in federated optimization communication costs dominate”.
Revised: “In contrast 1 , in federated optimization communication costs dominate”.

Original: LT is a conceptually simple and surprisingly powerful communication-acceleration technique.
Revised: LT is a conceptually simple and surprisingly powerful communication-acceleration technique.

Original: The basic idea behind LT is for the clients to perform multiple local GD steps instead of a single step (which is how GD operates) before communication and aggregation takes place.
Revised: The basic idea behind LT is for the clients to perform multiple local GD steps instead of a single step (which is how GD operates) before communication and aggregation takes place.

Original: The intuitive reasoning used in virtually all papers on this topic is: performing multiple local GD steps results in “richer” and ultimately more useful local training in the sense that fewer communication rounds will hopefully sufﬁce to ﬁnish the training.
Revised: The intuitive reasoning used in virtually all papers on this topic is: performing multiple local GD steps results in “richer” and ultimately more useful local training in the sense that fewer communication rounds will hopefully sufﬁce to ﬁnish the training.

Original: McMahan et al.
Revised: McMahan et al.

Original: [2017] supported this intuition with ample empirical evidence, and credited LT as the critical component behind the success of FedAvg : “Thus, our goal is to use additional computation in order to decrease the number of rounds of communication needed to train a model...”
Revised: [2017] supported this intuition with ample empirical evidence, and credited LT as the critical component behind the success of FedAvg : “Thus, our goal is to use additional computation in order to decrease the number of rounds of communication needed to train a model...”

Original: “Communication costs are the principal constraint, and we show a reduction in required communication rounds by 10–100 × as compared to synchronized stochastic gradient descent.”
Revised: “Communication costs are the principal constraint, and we show a reduction in required communication rounds by 10–100 × as compared to synchronized stochastic gradient descent.”

Original: “...the speedups we achieve are due primarily to adding more computation on each client”.
Revised: “...the speedups we achieve are due primarily to adding more computation on each client”.

Original:  Five Generations of Local Training Methods
Revised:  Five Generations of Local Training Methods

Original: We now offer several historical comments on the most important developments related to the theoretical understanding of LT.
Revised: We now offer several historical comments on the most important developments related to the theoretical understanding of LT.

Original: To this end, we have identiﬁed 5 distinct generations of LT methods, each with its unique challenges and characteristics.
Revised: To this end, we have identiﬁed 5 distinct generations of LT methods, each with its unique challenges and characteristics.

Original: To make the narrative simple, and since we focus on this regime in our paper, we limit our overview to loss functions φ i that are µ -strongly convex and
Revised: To make the narrative simple, and since we focus on this regime in our paper, we limit our overview to loss functions φ i that are µ -strongly convex and

Original: L -smooth.
Revised: L -smooth.

Original: This is arguably the most studied class of functions in continuous optimization [Nesterov, 2004], and for this reason, it presents a valuable litmus test for any theory of LT.
Revised: This is arguably the most studied class of functions in continuous optimization [Nesterov, 2004], and for this reason, it presents a valuable litmus test for any theory of LT.

Original: 2.1 Generation 1: Heuristic Age
Revised: 2.1 Generation 1: Heuristic Age

Original: While LT ideas were used in several machine learning domains before [Povey et al., 2015, Moritz et al., 2016], LT truly rose to prominence as a practically potent communication acceleration technique due to the seminal paper of McMahan et al.
Revised: While LT ideas were used in several machine learning domains before [Povey et al., 2015, Moritz et al., 2016], LT truly rose to prominence as a practically potent communication acceleration technique due to the seminal paper of McMahan et al.

Original: [2017] which introduced the FedAvg algorithm.
Revised: [2017] which introduced the FedAvg algorithm.

Original: However, no theory was provided in their work, nor in any prior work.
Revised: However, no theory was provided in their work, nor in any prior work.

Original: LT-based heuristics, i.e., methods without any theoretical guarantees, dominated the initial development of the ﬁeld up to, and including, the
Revised: LT-based heuristics, i.e., methods without any theoretical guarantees, dominated the initial development of the ﬁeld up to, and including, the

Original: FedAvg paper.
Revised: FedAvg paper.

Original: 2.2 Generation 2: Homogeneous Age
Revised: 2.2 Generation 2: Homogeneous Age

Original: The ﬁrst theoretical results for LT methods offering explicit convergence rates relied on various data/gradient homogeneity 2 assumptions.
Revised: The ﬁrst theoretical results for LT methods offering explicit convergence rates relied on various data/gradient homogeneity 2 assumptions.

Original: The intuitive rationale behind such assumptions comes from the following thought process.
Revised: The intuitive rationale behind such assumptions comes from the following thought process.

Original: In the extreme case when all the local functions φ i are identical (this is often referred to as the homogeneous or i.i.d.
Revised: In the extreme case when all the local functions φ i are identical (this is often referred to as the homogeneous or i.i.d.

Original: data regime), there is a very simple approach to making GD communication-efﬁcient: push the idea of LT to its extreme by running GD on all clients, independently and in parallel, without any communication/synchronization/averaging whatsoever.
Revised: data regime), there is a very simple approach to making GD communication-efﬁcient: push the idea of LT to its extreme by running GD on all clients, independently and in parallel, without any communication/synchronization/averaging whatsoever.

Original: Extrapolating from this, it is reasonable to assume that as we increase heterogeneity, taking multiple local steps should still be beneﬁcial as long as we do not take too many steps.
Revised: Extrapolating from this, it is reasonable to assume that as we increase heterogeneity, taking multiple local steps should still be beneﬁcial as long as we do not take too many steps.

Original: Several authors analyzed various LT methods under such assumptions, and obtained rates [Haddadpour and Mahdavi, 2019, Yu et al., 2019, Li et al., 2019, 2020b].
Revised: Several authors analyzed various LT methods under such assumptions, and obtained rates [Haddadpour and Mahdavi, 2019, Yu et al., 2019, Li et al., 2019, 2020b].

Original: However, bounded dissimilarity assumptions are highly problematic.
Revised: However, bounded dissimilarity assumptions are highly problematic.

Original: First, they do not seem to be satisﬁed even for some of the simplest function classes, such as strongly convex quadratics [Khaled et al., 2019, 2020], and moreover, it is well known that practical FL datasets are highly heterogeneous/non-i.i.d.
Revised: First, they do not seem to be satisﬁed even for some of the simplest function classes, such as strongly convex quadratics [Khaled et al., 2019, 2020], and moreover, it is well known that practical FL datasets are highly heterogeneous/non-i.i.d.

Original: McMahan et al.
Revised: McMahan et al.

Original: [2017], Kairouz et al.
Revised: [2017], Kairouz et al.

Original: So, analyses relying on such strong assumptions are both mathematically questionable, and practically irrelevant.
Revised: So, analyses relying on such strong assumptions are both mathematically questionable, and practically irrelevant.

Original: 2.3 Generation 3: Sublinear Age
Revised: 2.3 Generation 3: Sublinear Age

Original: The third generation of LT methods is characterized by the successful removal of the bounded dissimilarity assumptions from the convergence theory.
Revised: The third generation of LT methods is characterized by the successful removal of the bounded dissimilarity assumptions from the convergence theory.

Original: Khaled et al.
Revised: Khaled et al.

Original: [2019] ﬁrst achieved this breakthrough by studying the simplest LT method: local gradient descent ( LGD ) (i.e., a simple combination of GD and LT).
Revised: [2019] ﬁrst achieved this breakthrough by studying the simplest LT method: local gradient descent ( LGD ) (i.e., a simple combination of GD and LT).

Original: While works belonging to this generation elevated LT to the same theoretical footing as
Revised: While works belonging to this generation elevated LT to the same theoretical footing as

Original: GD in terms of the assumptions, which marked an important milestone in our understanding of LT, unfortunately, the obtained communication complexity theory of LGD is pessimistic when compared to vanilla GD .
Revised: GD in terms of the assumptions, which marked an important milestone in our understanding of LT, unfortunately, the obtained communication complexity theory of LGD is pessimistic when compared to vanilla GD .

Original: Indeed, the inclusion of LT did not lead to an improvement upon the communication complexity of vanilla GD .
Revised: Indeed, the inclusion of LT did not lead to an improvement upon the communication complexity of vanilla GD .

Original: Moreover, while GD enjoys a linear communication complexity (in the smooth and strongly convex regime), the communication complexity of LGD is sublinear .
Revised: Moreover, while GD enjoys a linear communication complexity (in the smooth and strongly convex regime), the communication complexity of LGD is sublinear .

Original: In a follow-up work, Khaled et al.
Revised: In a follow-up work, Khaled et al.

Original: [2020] later analyzed LGD in combination with DS as well.
Revised: [2020] later analyzed LGD in combination with DS as well.

Original: Woodworth et al.
Revised: Woodworth et al.

Original: [2020] and Glasgow et al.
Revised: [2020] and Glasgow et al.

Original: [2022] provided lower bounds for LGD with DS showing that it is not better than minibatch SGD in heterogeneous setting.
Revised: [2022] provided lower bounds for LGD with DS showing that it is not better than minibatch SGD in heterogeneous setting.

Original: See the work of Malinovsky et al.
Revised: See the work of Malinovsky et al.

Original: [2020] for a ﬁxed-point theory viewpoint.
Revised: [2020] for a ﬁxed-point theory viewpoint.

Original: 2.4 Generation 4: Linear Age
Revised: 2.4 Generation 4: Linear Age

Original: The fourth generation of LT methods is characterized by the effort to design linearly converging variants of LT algorithms.
Revised: The fourth generation of LT methods is characterized by the effort to design linearly converging variants of LT algorithms.

Original: In order to achieve this, it was important to tame the adverse effect of the so-called client drift [Karimireddy et al., 2020], which was identiﬁed as the culprit of the worse-than- GD theoretical performance of the previous generation of LT methods.
Revised: In order to achieve this, it was important to tame the adverse effect of the so-called client drift [Karimireddy et al., 2020], which was identiﬁed as the culprit of the worse-than- GD theoretical performance of the previous generation of LT methods.

Original: The ﬁrst LT-based method that successfully tamed client drift, and as a result obtained a linear convergence rate, was
Revised: The ﬁrst LT-based method that successfully tamed client drift, and as a result obtained a linear convergence rate, was

Original: Scaffold [Karimireddy et al., 2020].
Revised: Scaffold [Karimireddy et al., 2020].

Original: Several alternative approaches to obtaining the same effect were later proposed by Gorbunov et al.
Revised: Several alternative approaches to obtaining the same effect were later proposed by Gorbunov et al.

Original: [2020a] and Mitra et al.
Revised: [2020a] and Mitra et al.

Original: While obtaining a linear rate for
Revised: While obtaining a linear rate for

Original: LT methods under standard assumptions was a major achievement, the communication complexity of these methods is still somewhat worse 3 than that of vanilla GD , and is at best equal to that of GD .
Revised: LT methods under standard assumptions was a major achievement, the communication complexity of these methods is still somewhat worse 3 than that of vanilla GD , and is at best equal to that of GD .

Original: 2.5 Generation 5: Accelerated Age
Revised: 2.5 Generation 5: Accelerated Age

Original: Finally, the ﬁfth generation of LT methods was initiated recently by Mishchenko et al.
Revised: Finally, the ﬁfth generation of LT methods was initiated recently by Mishchenko et al.

Original: [2022] with their ProxSkip method which enjoys accelerated communication complexity .
Revised: [2022] with their ProxSkip method which enjoys accelerated communication complexity .

Original: Acceleration comes from the LT steps coupled with a new client drift reduction technique and a probabilistic approach to deciding whether communication takes place or not.
Revised: Acceleration comes from the LT steps coupled with a new client drift reduction technique and a probabilistic approach to deciding whether communication takes place or not.

Original: Mishchenko et al.
Revised: Mishchenko et al.

Original: [2022] ﬁrst reformulate (1)
Revised: [2022] ﬁrst reformulate (1)

Original: 1: Parameters: stepsize γ > 0 , probability p
Revised: 1: Parameters: stepsize γ > 0 , probability p

Original: 1] , initial iterate
Revised: 1] , initial iterate

Original: R d , initial control vector y 0 ∈ R d , initial gradient shift h 0 ∈ R d , number of iterations T ≥2: for t = 0 , 1 , . .
Revised: R d , initial control vector y 0 ∈ R d , initial gradient shift h 0 ∈ R d , number of iterations T ≥2: for t = 0 , 1 , . .

Original: , T − 1 do 3: g t = g ( x t , y t , ξ t ) (cid:5)
Revised: , T − 1 do 3: g t = g ( x t , y t , ξ t ) (cid:5)

Original: Sample ξ t and construct an unbiased estimator of ∇ f ( x t ) 4: ˆ x t +1 = x
Revised: Sample ξ t and construct an unbiased estimator of ∇ f ( x t ) 4: ˆ x t +1 = x

Original: t − γ ( g t − h t ) (cid:5) Take a gradient-type step adjusted via the shift h t 5:
Revised: t − γ ( g t − h t ) (cid:5) Take a gradient-type step adjusted via the shift h t 5:

Original: Construct new control vector y t + 6:
Revised: Construct new control vector y t + 6:

Original: Flip a coin θ t ∈ { 0 , 1 } where Prob( θ t = 1) =
Revised: Flip a coin θ t ∈ { 0 , 1 } where Prob( θ t = 1) =

Original: p (cid:5) Decides whether to skip the prox or not 7: if θ t = 1 then 8: x t +1 = prox
Revised: p (cid:5) Decides whether to skip the prox or not 7: if θ t = 1 then 8: x t +1 = prox

Original: γp r (cid:0)
Revised: γp r (cid:0)

Original: ˆ x t +1 − γp h t (cid:1) (cid:5) Apply probability p 9: else 10: x t +1 = ˆ x t + (cid:5) the
Revised: ˆ x t +1 − γp h t (cid:1) (cid:5) Apply probability p 9: else 10: x t +1 = ˆ x t + (cid:5) the

Original: 11: end if 12: h t +1 =
Revised: 11: end if 12: h t +1 =

Original: h t + pγ ( x t +1 − ˆ x t +1 ) (cid:5) Update the shift h t 13: end for into the equivalent consensus form min
Revised: h t + pγ ( x t +1 − ˆ x t +1 ) (cid:5) Update the shift h t 13: end for into the equivalent consensus form min

Original: [Equation]
Revised: [Equation]

Original: f (
Revised: f (

Original: [Equation]
Revised: [Equation]

Original: ( x ) , (3) where d = Md (cid:48) , x
Revised: ( x ) , (3) where d = Md (cid:48) , x

Original: = ( x 1 , . .
Revised: = ( x 1 , . .

Original: , x M ) ∈ R d , and f ( x ) :=
Revised: , x M ) ∈ R d , and f ( x ) :=

Original: M (cid:88) i = n n i φ i ( x i ) , r ( x ) =
Revised: M (cid:88) i = n n i φ i ( x i ) , r ( x ) =

Original: (cid:26)if x 1 = · · · = x M , + ∞ otherwise.
Revised: (cid:26)if x 1 = · · · = x M , + ∞ otherwise.

Original: The ProxSkip method is a randomized variant of proximal gradient descent ( ProxGD )
Revised: The ProxSkip method is a randomized variant of proximal gradient descent ( ProxGD )

Original: [Nesterov, 2013,
Revised: [Nesterov, 2013,

Original: Beck, 2017] for solving (3), with the proximity operator of r , given by prox r
Revised: Beck, 2017] for solving (3), with the proximity operator of r , given by prox r

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: arg
Revised: arg

Original: [Equation]
Revised: [Equation]

Original: y
Revised: y

Original: [Equation]
Revised: [Equation]

Original: being evaluated in each iteration with probability p ∈ (0 , 1] only.
Revised: being evaluated in each iteration with probability p ∈ (0 , 1] only.

Original: Remarkably, Mishchenko et al.
Revised: Remarkably, Mishchenko et al.

Original: [2022] showed that it is possible to choose p as low as 1 / √ κ , where κ is the condition number of f , without this worsening the rate of its parent method ProxGD .
Revised: [2022] showed that it is possible to choose p as low as 1 / √ κ , where κ is the condition number of f , without this worsening the rate of its parent method ProxGD .

Original: In summary, ProxSkip lets the M clients perform √ κ local gradient steps in expectation, followed by the evaluation of the prox of r , which in the case of the consensus reformulation of (1) means averaging across all M nodes.
Revised: In summary, ProxSkip lets the M clients perform √ κ local gradient steps in expectation, followed by the evaluation of the prox of r , which in the case of the consensus reformulation of (1) means averaging across all M nodes.

Original:  ProxSkip-VR: A General Variance Reduction Framework for ProxSkip
Revised:  ProxSkip-VR: A General Variance Reduction Framework for ProxSkip

Original: In this work we contribute to the ﬁfth generation of LT methods by extending the work of Mishchenko et al.
Revised: In this work we contribute to the ﬁfth generation of LT methods by extending the work of Mishchenko et al.

Original: [2022] to allow for a very large family of gradient estimators, including variance reduced (VR) ones [Johnson and Zhang, 2013, Defazio et al., 2014, Kovalev et al., 2020a, Mishchenko et al., 2019].
Revised: [2022] to allow for a very large family of gradient estimators, including variance reduced (VR) ones [Johnson and Zhang, 2013, Defazio et al., 2014, Kovalev et al., 2020a, Mishchenko et al., 2019].

Original: Like ProxSkip , our method ProxSkip-VR (Algorithm 1) is aimed to solve the composite problem (3) in a more general setting (see Assumptions 1–3), with the special structure (4) coming from the consensus reformulation being a special case only.
Revised: Like ProxSkip , our method ProxSkip-VR (Algorithm 1) is aimed to solve the composite problem (3) in a more general setting (see Assumptions 1–3), with the special structure (4) coming from the consensus reformulation being a special case only.

Original: Our method differs from ProxSkip in that we replace the gradient ∇ f ( x t ) by an unbiased estimator g t = g ( x t , y t , ξ t ) , where ξ t is the source of randomness controlling unbiasedness and y t is a control vector whose role is to progressively reduce the variance of the estimator, so that
Revised: Our method differs from ProxSkip in that we replace the gradient ∇ f ( x t ) by an unbiased estimator g t = g ( x t , y t , ξ t ) , where ξ t is the source of randomness controlling unbiasedness and y t is a control vector whose role is to progressively reduce the variance of the estimator, so that

Original: [Equation]
Revised: [Equation]

Original: There are several motivations behind this endeavor.
Revised: There are several motivations behind this endeavor.

Original: First, it is a-priori not clear whether the novel proof technique employed by Mishchenko et al.
Revised: First, it is a-priori not clear whether the novel proof technique employed by Mishchenko et al.

Original: [2022] can be combined with the proof techniques used in the analysis of VR methods, and hence it is scientiﬁcally signiﬁcant to investigate the possibility of such a merger of two strands of the literature.
Revised: [2022] can be combined with the proof techniques used in the analysis of VR methods, and hence it is scientiﬁcally signiﬁcant to investigate the possibility of such a merger of two strands of the literature.

Original: We show that this is possible.
Revised: We show that this is possible.

Original: Second, marrying VR estimators with ProxSkip can lead to novel system architectures which are more elaborate than the simplistic client-server architecture (see Section 4).
Revised: Second, marrying VR estimators with ProxSkip can lead to novel system architectures which are more elaborate than the simplistic client-server architecture (see Section 4).

Original: Lastly, while researchers contributing to generations 1–4 of LT methods were preoccupied with trying to close the gap on GD in terms of communication efﬁciency, they ignored the number of the local steps appearing in their algorithms, and reported their bounds primarily in terms of the number of communication rounds.
Revised: Lastly, while researchers contributing to generations 1–4 of LT methods were preoccupied with trying to close the gap on GD in terms of communication efﬁciency, they ignored the number of the local steps appearing in their algorithms, and reported their bounds primarily in terms of the number of communication rounds.

Original: Bounds reported this way make complete sense in the scenario when the cost of local work (e.g., one SGD step w.r.t.
Revised: Bounds reported this way make complete sense in the scenario when the cost of local work (e.g., one SGD step w.r.t.

Original: a single data point), say δ , is negligible compared to the cost of communication, which can w.l.o.g. assume to be 1, and when the number of local steps is small.
Revised: a single data point), say δ , is negligible compared to the cost of communication, which can w.l.o.g. assume to be 1, and when the number of local steps is small.

Original: With the advent of the ﬁfth generation of LT methods, we can (to a large degree) stop worrying about communication efﬁciency, and can now ask more reﬁned questions, such as:
Revised: With the advent of the ﬁfth generation of LT methods, we can (to a large degree) stop worrying about communication efﬁciency, and can now ask more reﬁned questions, such as:

Original: Are there gradient estimators which, when combined with ProxSkip , lead to faster algorithms in terms of the total cost, which includes the communication cost as well as the cost of local training?
Revised: Are there gradient estimators which, when combined with ProxSkip , lead to faster algorithms in terms of the total cost, which includes the communication cost as well as the cost of local training?

Original: We give an afﬁrmative answer to the question in Sections 4 and 5.
Revised: We give an afﬁrmative answer to the question in Sections 4 and 5.

Original: 3.1 Standard assumptions
Revised: 3.1 Standard assumptions

Original: We assume throughout that f is differentiable, and let D f ( x, y ) :
Revised: We assume throughout that f is differentiable, and let D f ( x, y ) :

Original: = f ( x ) − f ( y ) − (cid:104)∇ f ( y ) , x − y (cid:105) denote the Bregman divergence of f .
Revised: = f ( x ) − f ( y ) − (cid:104)∇ f ( y ) , x − y (cid:105) denote the Bregman divergence of f .

Original: Throughout the work we make the following assumptions:
Revised: Throughout the work we make the following assumptions:

Original: Assumption 1 ( L -smoothness) .
Revised: Assumption 1 ( L -smoothness) .

Original: ∃ L > 0 such that 2 D f ( x, y ) ≤ L (cid:107) x − y (cid:107) 2 for all x, y ∈ R d .
Revised: ∃ L > 0 such that 2 D f ( x, y ) ≤ L (cid:107) x − y (cid:107) 2 for all x, y ∈ R d .

Original: Assumption 2 ( µ -convexity) .
Revised: Assumption 2 ( µ -convexity) .

Original: ∃ µ > 0 such that µ (cid:107) x − y (cid:107) 2 ≤ 2 D f ( x, y ) for all x, y ∈ R d .
Revised: ∃ µ > 0 such that µ (cid:107) x − y (cid:107) 2 ≤ 2 D f ( x, y ) for all x, y ∈ R d .

Original: Assumption 3.
Revised: Assumption 3.

Original: The regularizer r : R d → R ∪ { + ∞} is proper, closed and convex.
Revised: The regularizer r : R d → R ∪ { + ∞} is proper, closed and convex.

Original: Under the above assumptions, (3) has a unique minimizer x (cid:63) .
Revised: Under the above assumptions, (3) has a unique minimizer x (cid:63) .

Original: Let h (cid:63) :=
Revised: Let h (cid:63) :=

Original: ∇ f
Revised: ∇ f

Original: ( x (cid:63) ) .
Revised: ( x (cid:63) ) .

Original: 3.2 Modelling variance reduced gradient estimators
Revised: 3.2 Modelling variance reduced gradient estimators

Original: Our next assumption, initially introduced by Gorbunov et al.
Revised: Our next assumption, initially introduced by Gorbunov et al.

Original: [2020b], postulates several parametric inequalities characterizing the behavior and ultimately the quality of a gradient estimator.
Revised: [2020b], postulates several parametric inequalities characterizing the behavior and ultimately the quality of a gradient estimator.

Original: Similar assumptions appeared later in [Gorbunov et al., 2020a,c].
Revised: Similar assumptions appeared later in [Gorbunov et al., 2020a,c].

Original: Assumption 4.
Revised: Assumption 4.

Original: Let { x t } be iterates produced by ProxSkip-VR .
Revised: Let { x t } be iterates produced by ProxSkip-VR .

Original: First, we assume that the stochastic gradients g t = g
Revised: First, we assume that the stochastic gradients g t = g

Original: ( x t , y t , ξ t ) are unbiased for all t ≥ 0 , namely
Revised: ( x t , y t , ξ t ) are unbiased for all t ≥ 0 , namely

Original: [Equation]
Revised: [Equation]

Original: Second, we assume that there exist non-negative constants A, B, C, ˜ A, ˜ B, ˜ C , with ˜ B < 1 , and a nonnegative mapping y t
Revised: Second, we assume that there exist non-negative constants A, B, C, ˜ A, ˜ B, ˜ C , with ˜ B < 1 , and a nonnegative mapping y t

Original: (cid:55)→ σ ( y t ) :
Revised: (cid:55)→ σ ( y t ) :

Original: = σ t such that the following two relations hold for all t ≥ 0 ,
Revised: = σ t such that the following two relations hold for all t ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: Assumption 4 covers a very large collection of gradient estimators, including an inﬁnite variety of subsampling/minibatch estimators, gradient sparsiﬁcation and quantization estimators, and their combinations; see [Gorbunov et al., 2020b] for examples.
Revised: Assumption 4 covers a very large collection of gradient estimators, including an inﬁnite variety of subsampling/minibatch estimators, gradient sparsiﬁcation and quantization estimators, and their combinations; see [Gorbunov et al., 2020b] for examples.

Original: VR estimators are characterized by
Revised: VR estimators are characterized by

Original: C = ˜ C
Revised: C = ˜ C

Original: = 0 ; most non-VR estimators by ˜ A =
Revised: = 0 ; most non-VR estimators by ˜ A =

Original: ˜ B
Revised: ˜ B

Original: =
Revised: =

Original: ˜
Revised: ˜

Original: C
Revised: C

Original: = B = 0 and C > 

Revised: = B = 0 and C >

Original: [Gower et al., 2019b].
Revised: [Gower et al., 2019b].

Original: 3.3 Main result
Revised: 3.3 Main result

Original: We are now ready to formulate our main result.
Revised: We are now ready to formulate our main result.

Original: Theorem 5.
Revised: Theorem 5.

Original: Let Assumptions 2 and 3 hold, and let g t be a gradient estimator satisfying Assumption 4.
Revised: Let Assumptions 2 and 3 hold, and let g t be a gradient estimator satisfying Assumption 4.

Original: If B > 0 , choose any W > B / (1 − ˜ B ) and β =
Revised: If B > 0 , choose any W > B / (1 − ˜ B ) and β =

Original: ( B + W ˜ B ) / W .
Revised: ( B + W ˜ B ) / W .

Original: If B = 0 , let W = 

Revised: If B = 0 , let W =

Original: and β = ˜ B .
Revised: and β = ˜ B .

Original: Choose stepsize 0 < γ ≤ min { 1 / µ , 1 / ( A + W ˜ A ) } .
Revised: Choose stepsize 0 < γ ≤ min { 1 / µ , 1 / ( A + W ˜ A ) } .

Original: Then the iterates of ProxSkip-VR for any p ∈ (0 , 1] satisfy
Revised: Then the iterates of ProxSkip-VR for any p ∈ (0 , 1] satisfy

Original: [Equation]
Revised: [Equation]

Original: where the Lyapunov function is deﬁned by Ψ t := (cid:107) x t − x
Revised: where the Lyapunov function is deﬁned by Ψ t := (cid:107) x t − x

Original: [Equation]
Revised: [Equation]

Original: (cid:107) 2 + γ 2 p 2 (cid:107) h t − h (cid:63) (cid:107) 2 + γ 2 Wσ t .
Revised: (cid:107) 2 + γ 2 p 2 (cid:107) h t − h (cid:63) (cid:107) 2 + γ 2 Wσ t .

Original: [Table]
Revised: [Table]

Original: 3.4 Two examples of gradient estimators
Revised: 3.4 Two examples of gradient estimators

Original: Here we give two illustrating examples of estimators satisfying Assumption 4.
Revised: Here we give two illustrating examples of estimators satisfying Assumption 4.

Original: Theorem 6 ( GD estimator) .
Revised: Theorem 6 ( GD estimator) .

Original: Let Assumption 1, 2 and 3 hold.
Revised: Let Assumption 1, 2 and 3 hold.

Original: Then for the trivial estimator g t = ∇ f ( x t ) , Assumption 4 holds with the following parameters:
Revised: Then for the trivial estimator g t = ∇ f ( x t ) , Assumption 4 holds with the following parameters:

Original: [Equation]
Revised: [Equation]

Original: Choose a stepsize satisfying 0 < γ ≤ 1 / L .
Revised: Choose a stepsize satisfying 0 < γ ≤ 1 / L .

Original: Then the iterates of ProxSkip-VR for any p ∈ (0 , 1] satisfy
Revised: Then the iterates of ProxSkip-VR for any p ∈ (0 , 1] satisfy

Original: [Equation]
Revised: [Equation]

Original: where Ψ t := (cid:107) x
Revised: where Ψ t := (cid:107) x

Original: t − x (cid:63) (cid:107) 2 + γ 2 p 2 (cid:107) h t − h (cid:63) (cid:107) 2 .
Revised: t − x (cid:63) (cid:107) 2 + γ 2 p 2 (cid:107) h t − h (cid:63) (cid:107) 2 .

Original: Let γ = 1 / L and p = (cid:112) µ / L then the communication and iteration complexities of ProxSkip-VR are # comms =
Revised: Let γ = 1 / L and p = (cid:112) µ / L then the communication and iteration complexities of ProxSkip-VR are # comms =

Original: O (cid:16)(cid:113)
Revised: O (cid:16)(cid:113)

Original: Lµ log 1 ε (cid:17) , # iters = O (cid:16)
Revised: Lµ log 1 ε (cid:17) , # iters = O (cid:16)

Original: Lµ log 1 ε (cid:17) .
Revised: Lµ log 1 ε (cid:17) .

Original: This recovers the result obtained by Mishchenko et al.
Revised: This recovers the result obtained by Mishchenko et al.

Original: [2022] for their ProxSkip method.
Revised: [2022] for their ProxSkip method.

Original: The next assumption holds for virtually all (non-VR) estimators based on subsampling [Gower et al., 2019b].
Revised: The next assumption holds for virtually all (non-VR) estimators based on subsampling [Gower et al., 2019b].

Original: Assumption 7 (Expected smoothness) .
Revised: Assumption 7 (Expected smoothness) .

Original: We say that an unbiased estimator g ( x ; ξ ) : R d → R d of the gradient ∇ f ( x ) satisﬁes the expected smoothness inequality if there exists A (cid:48)(cid:48) > 0 such that
Revised: We say that an unbiased estimator g ( x ; ξ ) : R d → R d of the gradient ∇ f ( x ) satisﬁes the expected smoothness inequality if there exists A (cid:48)(cid:48) > 0 such that

Original: [Equation]
Revised: [Equation]

Original: Theorem 8 ( SGD estimator) .
Revised: Theorem 8 ( SGD estimator) .

Original: Let g ( x, ξ ) satisfy Assumption 7 and deﬁne g t :=
Revised: Let g ( x, ξ ) satisfy Assumption 7 and deﬁne g t :=

Original: g ( x t , ξ t ) , where ξ t is chosen independently at time t .
Revised: g ( x t , ξ t ) , where ξ t is chosen independently at time t .

Original: Then Assumption 4 holds with the following parameters:
Revised: Then Assumption 4 holds with the following parameters:

Original: [Equation]
Revised: [Equation]

Original: Moreover, assume that Assumption 2 holds.
Revised: Moreover, assume that Assumption 2 holds.

Original: Choose stepsize 0 < γ ≤ min { 1 / µ , 1 / A } .
Revised: Choose stepsize 0 < γ ≤ min { 1 / µ , 1 / A } .

Original: Then the iterates of ProxSkip-VR for any probability p ∈ (0 , 1] satisfy
Revised: Then the iterates of ProxSkip-VR for any probability p ∈ (0 , 1] satisfy

Original: [Equation]
Revised: [Equation]

Original: where the Lyapunov function is deﬁned by Ψ t := (cid:107) x t − x (cid:63) (cid:107) 2 + γ 2 p 2 (cid:107) h t − h (cid:63) (cid:107) 2 .
Revised: where the Lyapunov function is deﬁned by Ψ t := (cid:107) x t − x (cid:63) (cid:107) 2 + γ 2 p 2 (cid:107) h t − h (cid:63) (cid:107) 2 .

Original: If we choose γ = min { 1 / A , εµ / 2 C } and p = √ γµ then the communication and iteration complexities of ProxSkipVR are # comms =
Revised: If we choose γ = min { 1 / A , εµ / 2 C } and p = √ γµ then the communication and iteration complexities of ProxSkipVR are # comms =

Original: O (cid:16)(cid:16)(cid:113)
Revised: O (cid:16)(cid:16)(cid:113)

Original: Aµ + (cid:113) Cεµ 2 (cid:17) log 1 ε (cid:17) , # iters =
Revised: Aµ + (cid:113) Cεµ 2 (cid:17) log 1 ε (cid:17) , # iters =

Original: O (cid:16)(cid:16)
Revised: O (cid:16)(cid:16)

Original: Aµ + Cεµ 2 (cid:17) log 1 ε (cid:17) , respectively.
Revised: Aµ + Cεµ 2 (cid:17) log 1 ε (cid:17) , respectively.

Original: This recovers the result obtained by Mishchenko et al.
Revised: This recovers the result obtained by Mishchenko et al.

Original: [2022] for their stochastic variant of ProxSkip , which they call SProxSkip .
Revised: [2022] for their stochastic variant of ProxSkip , which they call SProxSkip .

Original:  New FL Architecture: Regional Hubs Connecting the Clients to the Server
Revised:  New FL Architecture: Regional Hubs Connecting the Clients to the Server

Original: We now illustrate the versatility of our ProxSkip-VR framework by designing a new “FL architecture” and proposing an algorithm that can efﬁciently operate in this setting.
Revised: We now illustrate the versatility of our ProxSkip-VR framework by designing a new “FL architecture” and proposing an algorithm that can efﬁciently operate in this setting.

Original: In particular, we consider the situation where the clients are clustered (e.g., based on region), and where a hub is placed in between each cluster and the central server.
Revised: In particular, we consider the situation where the clients are clustered (e.g., based on region), and where a hub is placed in between each cluster and the central server.

Original: Clients communicate with their regional hub only, which can communicate with the central server (see Figure 1).
Revised: Clients communicate with their regional hub only, which can communicate with the central server (see Figure 1).

Original: There are M hubs, hub i handles n i clients, and client j associated with hub i owns loss function φ ij .
Revised: There are M hubs, hub i handles n i clients, and client j associated with hub i owns loss function φ ij .

Original: Mathematically, this can be modeled by problem (1).
Revised: Mathematically, this can be modeled by problem (1).

Original: In this situation, we care about two sources of communication cost: the server and the hubs, and between the hubs and the clients.
Revised: In this situation, we care about two sources of communication cost: the server and the hubs, and between the hubs and the clients.

Original: We propose to handle this via local training (LT) between the server and the hubs, and via client sampling (CS) and compressed communication (CC) between the hubs and the clients.
Revised: We propose to handle this via local training (LT) between the server and the hubs, and via client sampling (CS) and compressed communication (CC) between the hubs and the clients.

Original: Algorithmically, from the server-hubs perspective, we are applying a particular variant of ProxSkip-VR to (3)– (4), where φ i is the aggregate loss handles by hub i .
Revised: Algorithmically, from the server-hubs perspective, we are applying a particular variant of ProxSkip-VR to (3)– (4), where φ i is the aggregate loss handles by hub i .

Original: This takes care of communication efﬁciency between the server and the hub.
Revised: This takes care of communication efﬁciency between the server and the hub.

Original: Note also that we need not worry about partial participation of hubs, as these are designed to be always available.
Revised: Note also that we need not worry about partial participation of hubs, as these are designed to be always available.

Original: However, in this situation, it is costly for hub i to compute the gradient of φ i as this involves communication with all the clients it handles.
Revised: However, in this situation, it is costly for hub i to compute the gradient of φ i as this involves communication with all the clients it handles.

Original: 4.1 Handling of client sampling (CS) and compressed communication (CC)
Revised: 4.1 Handling of client sampling (CS) and compressed communication (CC)

Original: In order to alleviate this burden, we propose a combination of CS and CC.
Revised: In order to alleviate this burden, we propose a combination of CS and CC.

Original: However, we need to be very careful about how to do this.
Revised: However, we need to be very careful about how to do this.

Original: Indeed, both CS and CC, even when applied in isolation, and without ProxSkip in the mix, can lead to a substantial slowdown in convergence.
Revised: Indeed, both CS and CC, even when applied in isolation, and without ProxSkip in the mix, can lead to a substantial slowdown in convergence.

Original: For example, one will typically lose linear convergence in the strongly convex regime.
Revised: For example, one will typically lose linear convergence in the strongly convex regime.

Original: However, techniques for preserving linear convergence in the presence of CS and CC exist: this is what variance reduction strategies are designed to do.
Revised: However, techniques for preserving linear convergence in the presence of CS and CC exist: this is what variance reduction strategies are designed to do.

Original: For example, LSVRG [Hofmann et al., 2015, Kovalev et al., 2020a] is a
Revised: For example, LSVRG [Hofmann et al., 2015, Kovalev et al., 2020a] is a

Original: VR technique for reducing the variance due to CS, and DIANA [Mishchenko et al., 2019] is a VR technique for reducing the variance due to CC.
Revised: VR technique for reducing the variance due to CS, and DIANA [Mishchenko et al., 2019] is a VR technique for reducing the variance due to CC.

Original: However, we are not aware of any VR method that combines CS (applied ﬁrst) and CC (applied second).
Revised: However, we are not aware of any VR method that combines CS (applied ﬁrst) and CC (applied second).

Original: We now propose such a technique.
Revised: We now propose such a technique.

Original: In iteration t , every hub i ∈ { 1 , 2 , . . .
Revised: In iteration t , every hub i ∈ { 1 , 2 , . . .

Original: , M } selects a random subset S it ⊆
Revised: , M } selects a random subset S it ⊆

Original: . , n i } of the clients it handles of cardinality
Revised: . , n i } of the clients it handles of cardinality

Original: τ i , chosen uniformly at random, and estimates the hub gradient via
Revised: τ i , chosen uniformly at random, and estimates the hub gradient via

Original: [Equation]
Revised: [Equation]

Original: where Q ijt : R d (cid:48)
Revised: where Q ijt : R d (cid:48)

Original: → R d (cid:48) is a randomized compression (e.g., sparsiﬁcation or quantization) operator [Alistarh et al., 2017, Khirirat et al., 2018, Horváth et al., 2019b,a, Philippenko and Dieuleveut, 2020], i.e., a mapping satisfying
Revised: → R d (cid:48) is a randomized compression (e.g., sparsiﬁcation or quantization) operator [Alistarh et al., 2017, Khirirat et al., 2018, Horváth et al., 2019b,a, Philippenko and Dieuleveut, 2020], i.e., a mapping satisfying

Original: [Equation]
Revised: [Equation]

Original: and the control vector y t is updated probabilistically as follows:
Revised: and the control vector y t is updated probabilistically as follows:

Original: [Equation]
Revised: [Equation]

Original: The global gradient estimator (a vector in R Md (cid:48) ), which we call HUB , is construcuted as a concatenation of the above hub estimators:
Revised: The global gradient estimator (a vector in R Md (cid:48) ), which we call HUB , is construcuted as a concatenation of the above hub estimators:

Original: [Equation]
Revised: [Equation]

Original: where ξ t represents the combined randomness from the compressors {Q ijt } and random sets {S it } .
Revised: where ξ t represents the combined randomness from the compressors {Q ijt } and random sets {S it } .

Original: In order to analyze ProxSkip-VR in the consensus form, from now on we assume that n i = m = n / M and τ i = τ ∈ { 1 , 2 , . . .
Revised: In order to analyze ProxSkip-VR in the consensus form, from now on we assume that n i = m = n / M and τ i = τ ∈ { 1 , 2 , . . .

Original: , m } for all i , and rely on a slightly different, more general reformulation: min x ∈ R d
Revised: , m } for all i , and rely on a slightly different, more general reformulation: min x ∈ R d

Original: [Equation]
Revised: [Equation]

Original: Our proposed method ProxSkip-HUB is ProxSkip-VR combined with the novel HUB estimator (14), applied to the above reformulation; see Algorithm 2.
Revised: Our proposed method ProxSkip-HUB is ProxSkip-VR combined with the novel HUB estimator (14), applied to the above reformulation; see Algorithm 2.

Original: 4.2 Theory for ProxSkip-HUB
Revised: 4.2 Theory for ProxSkip-HUB

Original: In the following result we ﬁrst claim that the above estimator satisﬁes Assumption 4 with C =
Revised: In the following result we ﬁrst claim that the above estimator satisﬁes Assumption 4 with C =

Original: ˜
Revised: ˜

Original: C =(i.e., it is variance-reduced), and the rest of the claim follows by application of our general theorem
Revised: C =(i.e., it is variance-reduced), and the rest of the claim follows by application of our general theorem

Original: Theorem 5.
Revised: Theorem 5.

Original: Theorem 9.
Revised: Theorem 9.

Original: Assume that ∇ (cid:101)
Revised: Assume that ∇ (cid:101)

Original: φ j is L j -smooth for all j and let Assumptions 2 and 3 hold.
Revised: φ j is L j -smooth for all j and let Assumptions 2 and 3 hold.

Original: Then for the gradient estimator (14) , Assumption 4 holds with the following constants:
Revised: Then for the gradient estimator (14) , Assumption 4 holds with the following constants:

Original: [Equation]
Revised: [Equation]

Original: and σ t := σ ( y t ) , σ ( y ) :=
Revised: and σ t := σ ( y t ) , σ ( y ) :=

Original: [Equation]
Revised: [Equation]

Original: (cid:80)
Revised: (cid:80)

Original: [Equation]
Revised: [Equation]

Original: (cid:101)
Revised: (cid:101)

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:101)
Revised: (cid:101)

Original: [Equation]
Revised: [Equation]

Original: ( x (cid:63) )
Revised: ( x (cid:63) )

Original: (cid:107) 2 , L max :=
Revised: (cid:107) 2 , L max :=

Original: max j L j .
Revised: max j L j .

Original: Set W = 2 B / (1 − ˜ B ) and 0 < γ ≤ min { 1 / µ , 1 / ( A + W ˜ A ) } .
Revised: Set W = 2 B / (1 − ˜ B ) and 0 < γ ≤ min { 1 / µ , 1 / ( A + W ˜ A ) } .

Original: Then the iterates of ProxSkip-VR for any p ∈ (0 , 1] satisfy
Revised: Then the iterates of ProxSkip-VR for any p ∈ (0 , 1] satisfy

Original: [Equation]
Revised: [Equation]

Original: where the Lyapunov function is deﬁned by
Revised: where the Lyapunov function is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: We now consider two special cases.
Revised: We now consider two special cases.

Original: In the ﬁrst, we specialize to the no compression regime, and in the second, to the full participation regime.
Revised: In the ﬁrst, we specialize to the no compression regime, and in the second, to the full participation regime.

Original: Corollary 1 (No compression) .
Revised: Corollary 1 (No compression) .

Original: If we do not use compression (i.e., ω = 0 ), then the communication and iteration complexities are # comms =
Revised: If we do not use compression (i.e., ω = 0 ), then the communication and iteration complexities are # comms =

Original: O (cid:16)(cid:113)
Revised: O (cid:16)(cid:113)

Original: L max µ log 1 ε (cid:17) , # iters =
Revised: L max µ log 1 ε (cid:17) , # iters =

Original: O (cid:16) L max µ log 1 ε (cid:17) , respectively.
Revised: O (cid:16) L max µ log 1 ε (cid:17) , respectively.

Original: However, if we use the estimator (11) in Theorem 5 directly, then the communication and iteration complexities are # comms =
Revised: However, if we use the estimator (11) in Theorem 5 directly, then the communication and iteration complexities are # comms =

Original: O (cid:18)(cid:113) L ( τ ) µ log 1 ε (cid:19) , # iters =
Revised: O (cid:18)(cid:113) L ( τ ) µ log 1 ε (cid:19) , # iters =

Original: O (cid:16) L ( τ ) µ log 1 ε (cid:17) , where
Revised: O (cid:16) L ( τ ) µ log 1 ε (cid:17) , where

Original: L ( τ ) :
Revised: L ( τ ) :

Original: = m − τ τ ( m − 1)
Revised: = m − τ τ ( m − 1)

Original: L max + m ( τ − 1) τ ( m − 1) L .
Revised: L max + m ( τ − 1) τ ( m − 1) L .

Original: Notice that L max ≥ L , and that L ( τ ) =
Revised: Notice that L max ≥ L , and that L ( τ ) =

Original: L max for τ = 1 and L ( τ ) =
Revised: L max for τ = 1 and L ( τ ) =

Original: L for τ = m .
Revised: L for τ = m .

Original: Moreover, L ( τ ) decreases as the minibatch size τ increases.
Revised: Moreover, L ( τ ) decreases as the minibatch size τ increases.

Original: Corollary 2 (No client sampling) .
Revised: Corollary 2 (No client sampling) .

Original: If we do not use client sampling (i.e., τ = m ), then the communication and iteration complexities are # comms =
Revised: If we do not use client sampling (i.e., τ = m ), then the communication and iteration complexities are # comms =

Original: O (cid:16)(cid:113)
Revised: O (cid:16)(cid:113)

Original: L max µ (cid:0) 1 + ωM (cid:1) log 1 ε (cid:17) , # iters =
Revised: L max µ (cid:0) 1 + ωM (cid:1) log 1 ε (cid:17) , # iters =

Original: O (cid:16) L max µ (cid:0) 1 + ωM (cid:1) log 1 ε (cid:17) , respectively.
Revised: O (cid:16) L max µ (cid:0) 1 + ωM (cid:1) log 1 ε (cid:17) , respectively.

Original: Assume r ( x ) ≡ 0 .
Revised: Assume r ( x ) ≡ 0 .

Original: If Q ijt ( x ) ≡ x , then ω = 0 , and we restore the well-known LSVRG method [Hofmann et al., 2015, Kovalev et al., 2020b], assuming that the functions φ ij have the same smoothness constant.
Revised: If Q ijt ( x ) ≡ x , then ω = 0 , and we restore the well-known LSVRG method [Hofmann et al., 2015, Kovalev et al., 2020b], assuming that the functions φ ij have the same smoothness constant.

Original: We can recover the same rate exactly as well, but with a slightly more reﬁned analysis, one in which we do not need to work with compressors ( LSVRG does not involve any), which makes for a tighter analysis.
Revised: We can recover the same rate exactly as well, but with a slightly more reﬁned analysis, one in which we do not need to work with compressors ( LSVRG does not involve any), which makes for a tighter analysis.

Original: If τ = m , we restore the rate of the well-known DIANA [Mishchenko et al., 2019,
Revised: If τ = m , we restore the rate of the well-known DIANA [Mishchenko et al., 2019,

Original: Horváth et al., 2019b] method and its sibling Rand-DIANA [Shulgin and Richtárik, 2021].
Revised: Horváth et al., 2019b] method and its sibling Rand-DIANA [Shulgin and Richtárik, 2021].

Original:  Experiments
Revised:  Experiments

Original: To illustrate the predictive power of our theory, it sufﬁces to consider L 2 -regularized logistic regression in the distributed setting (1), with
Revised: To illustrate the predictive power of our theory, it sufﬁces to consider L 2 -regularized logistic regression in the distributed setting (1), with

Original: [Equation]
Revised: [Equation]

Original: where n i is the number of data points per worker a ij ∈ R d (cid:48) and b ij ∈ {− 1 , +1 } are the data samples and labels.
Revised: where n i is the number of data points per worker a ij ∈ R d (cid:48) and b ij ∈ {− 1 , +1 } are the data samples and labels.

Original: We choose n i = m = n / M for all i .
Revised: We choose n i = m = n / M for all i .

Original: We set the regularization parameter λ
Revised: We set the regularization parameter λ

Original: = 5 · 10 − 4 L by default, where L is the smoothness constant of f .
Revised: = 5 · 10 − 4 L by default, where L is the smoothness constant of f .

Original: We conduct several experiments on the w8a dataset from LibSVM library [Chang and Lin, 2011].
Revised: We conduct several experiments on the w8a dataset from LibSVM library [Chang and Lin, 2011].

Original: In Figure 2 (ﬁrst row), we compare various LT baselines 4 for three choices of mini-batch sizes ( τ = 16 , 32 , 64 ) with our method
Revised: In Figure 2 (ﬁrst row), we compare various LT baselines 4 for three choices of mini-batch sizes ( τ = 16 , 32 , 64 ) with our method

Original: ProxSkip-VR combined with the LSVRG estimator, which is a special case of the HUB estimator when
Revised: ProxSkip-VR combined with the LSVRG estimator, which is a special case of the HUB estimator when

Original: [Figure]
Revised: [Figure]

Original: Q ijt ( x ) ≡ x for all i, j .
Revised: Q ijt ( x ) ≡ x for all i, j .

Original: We see that our method outperforms all other methods signiﬁcantly due to its communication-acceleration properties.
Revised: We see that our method outperforms all other methods signiﬁcantly due to its communication-acceleration properties.

Original: Next, we derive the total cost, which includes communication cost (assumed to be 1 , for normalization purposes), and computational cost (assumed to be δ ; and equal to the cost of performing one SGD step with a single data point).
Revised: Next, we derive the total cost, which includes communication cost (assumed to be 1 , for normalization purposes), and computational cost (assumed to be δ ; and equal to the cost of performing one SGD step with a single data point).

Original: Let us consider the total cost of ProxSkip-VR in the case of the LSVRG estimator: in each iteration we compute 1 stochastic gradient, and with probability q
Revised: Let us consider the total cost of ProxSkip-VR in the case of the LSVRG estimator: in each iteration we compute 1 stochastic gradient, and with probability q

Original: we compute the exact gradient.
Revised: we compute the exact gradient.

Original: We do not need to compute a second stochastic gradient since we can use memory and the relation y t +

Revised: We do not need to compute a second stochastic gradient since we can use memory and the relation y t +

Original: = x t .
Revised: = x t .

Original: The total cost for ProxSkip-VR is equal to
Revised: The total cost for ProxSkip-VR is equal to

Original: Cost ( ProxSkip-VR ) :
Revised: Cost ( ProxSkip-VR ) :

Original: = T comm.
Revised: = T comm.

Original: ( ProxSkip-VR )
Revised: ( ProxSkip-VR )

Original: + δ ( qm + (1 − q )
Revised: + δ ( qm + (1 − q )

Original: τ + τ ) T iter ( ProxSkip-VR ) .
Revised: τ + τ ) T iter ( ProxSkip-VR ) .

Original: ProxSkip requires full/exact gradient computation at each iteration, so the total cost of ProxSkip is
Revised: ProxSkip requires full/exact gradient computation at each iteration, so the total cost of ProxSkip is

Original: Cost ( ProxSkip ) :
Revised: Cost ( ProxSkip ) :

Original: = T comm.
Revised: = T comm.

Original: ( ProxSkip )
Revised: ( ProxSkip )

Original: + δmT iter ( ProxSkip ) .
Revised: + δmT iter ( ProxSkip ) .

Original: Using Theorems 6 and 9 and the value L ( τ ) :
Revised: Using Theorems 6 and 9 and the value L ( τ ) :

Original: = m − τ τ ( m − 1)
Revised: = m − τ τ ( m − 1)

Original: L max + m ( τ − 1) τ ( m − 1) L of the expected smoothness constant for sampling with minibatch size τ uniformly at random, we get the following expression for the cost ratio, expressed as a function of δ :
Revised: L max + m ( τ − 1) τ ( m − 1) L of the expected smoothness constant for sampling with minibatch size τ uniformly at random, we get the following expression for the cost ratio, expressed as a function of δ :

Original: [Equation]
Revised: [Equation]

Original: We can easily calculate the limits of this expression:
Revised: We can easily calculate the limits of this expression:

Original: [Equation]
Revised: [Equation]

Original: In Figure 2 (second row), we depict the theoretical cost ratio according to (15) and the corresponding experimental cost ratio obtained by an actual run of both methods to achieve ε -accuracy, with ε = 10 − 6 and ε = 10 − 8 .
Revised: In Figure 2 (second row), we depict the theoretical cost ratio according to (15) and the corresponding experimental cost ratio obtained by an actual run of both methods to achieve ε -accuracy, with ε = 10 − 6 and ε = 10 − 8 .

Original: Remarkably, the experimental results follows the same pattern as our theoretical prediction.
Revised: Remarkably, the experimental results follows the same pattern as our theoretical prediction.

Original: The empirical curves appear lower because we use approximations for L max ,
Revised: The empirical curves appear lower because we use approximations for L max ,

Original: L and µ .
Revised: L and µ .

Original: As we can see, starting from δ = 10 − 4 , ProxSkip-VR starts to outperform ProxSkip .
Revised: As we can see, starting from δ = 10 − 4 , ProxSkip-VR starts to outperform ProxSkip .

Original: As δ grows, the advantage of variance reduction embedded in ProxSkip-VR over vanilla ProxSkip grows.
Revised: As δ grows, the advantage of variance reduction embedded in ProxSkip-VR over vanilla ProxSkip grows.

Original: These results suggest that variance reduction is of practical utility in terms of the total cost, even for small values of δ , and its effectiveness grows with δ .
Revised: These results suggest that variance reduction is of practical utility in terms of the total cost, even for small values of δ , and its effectiveness grows with δ .

Original: Acknowledgements
Revised: Acknowledgements

Original: We would like to thank Eduard Gorbunov for useful discussions related to some aspects of the theory.
Revised: We would like to thank Eduard Gorbunov for useful discussions related to some aspects of the theory.

Original: Appendix A Supplementary Experiments A.1 Additional experiments with ProxSkip-VR

Original: We conduct further experiments to validate the efﬁciency of our proposed method ProxSkip-VR .

Original: We instantiate our variance reduction design with LSVRG and compare our method with several baselines across various datasets ( w8a / a9a ), different number of workers (10/20)

Original: [Figure]

Original: and different batch sizes (16/32/64).

Original: All results in Figures 3, 4, 5, 6 show that ProxSkip-VR achieves linear convergence and outperforms the baselines

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: A.2 Additional experiments of cost ratio of ProxSkip over ProxSkip-VR

Original: Here we report on more experiments related to the total cost ratio of ProxSkip over ProxSkip-VR in

Original: Figures 7, 8, 9, 10.

Original: When GD is used as the subroutine, i.e., when ProxSkip-VR reduces to ProxSkip , the ratio is equal to one, and is depicted by the red horizontal dashed line.

Original: Any cost ratio above one means that ProxSkip-VR beneﬁts over ProxSkip (e.g., a cost ratio of 10 means 10 × speedup in favor of our method).

Original: As seen in the plots, acceleration of our method over ProxSkip is clearly visible, and improves as the local computation cost δ per sample increases.

Original: For large values of δ (i.e., δ ≈ 10 − 1 ),

Original: [Figure]

Original: the acceleration can reach 20 × to 85 × .

Original: Note also that acceleration is more signiﬁcant for small minibatch sizes.

Original: That is, it is better for τ = 16 than for τ = 32 , which is better than in the τ =case.

Original: This means that in terms of total cost, it is beneﬁcial for the workers to use smaller minibatch sizes,

Original: i

Original: [Figure]

Original: e., it is beneﬁcial to be as far from the full batch regime employed by ProxSkip as possible.

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: A.3 Experiments with ProxSkip-HUB

Original: In this work we introduced a new FL architecture: regional hubs connecting the clients to the server; see Section 4.

Original: For conceptual simplicity

Original: [Figure]

Original: and in order to facilitate fair comparison with ProxSkip-

Original: [Figure]

Original: LSVRG , we assume that the number of hubs equals to the number of clients, and that each client owns a single datapoint only.

Original: We compare ProxSkip-HUB with ProxSkip-LSVRG to check whether communication compression leads to any beneﬁts in terms of total costs.

Original: Theoretically, and similarly to our analysis in Section 5, the total cost for ProxSkip-LSVRG is

Original: Cost ( ProxSkip-LSVRG ) :

Original: = T comm.

Original: ( ProxSkip-LSVRG )

Original: + δ ( qm + (1 − q )

Original: τ + τ ) T iter ( ProxSkip-LSVRG ) .

Original: Recall that we assume the communication cost from every worker/hub to the master is equal to 1, and the computation cost per sample is equal to δ .

Original: Here we generalize to the multi-level structure.

Original: We assume that the communication cost from every client to hub is equal to δ (cid:48) .

Original: We choose the Rand- k sparsiﬁcation for ProxSkip-HUB ; this compressor selects k -entries of the gradient vector, uniformly at random from the full d -dimensional gradient.

Original: The total cost of ProxSkip-HUB is

Original: Cost ( ProxSkip-HUB ) :

Original: = T comm.

Original: [Equation]

Original: ProxSkip

Original: [Equation]

Original: HUB )

Original: [Equation]

Original: 1: Input : stepsize γ > 0 , probabilities p > 0 , q > 0 , initial iterate x 0 ∈ R d , initial shift y 0 ∈ R d , initial control variate h 0 ∈ R d , number of iterations T ≥2: for t = 0 , 1 , . .

Original: , T − 1 do 3: broadcast x t to all clients 4: for i

Original: ∈ S t do 5: ˆ∆ it = Q ( ∇ f i ( x t ) − ∇ f i ( y t )) (cid:5) Apply compression operator 6: end for 7: ˆ∆ t

Original: = 1 τ (cid:80) i ∈

Original: [-S t ˆ∆ it 8: ˆ g = ˆ ∆ t + ∇ f ( y t ) 9: ˆ x t +
-]
Revised: [-S t ˆ∆ it 8: ˆ g = ˆ ∆ t + ∇ f ( y t ) 9: ˆ x t +
-]

Original: =

Original: x

Original: t − γ (ˆ g t − h t ) (cid:5) Take a gradient-type step adjusted via the control variate h t 10:

Original: Flip a coin θ t ∈ { 0 , 1 } where Prob( θ t = 1) = p (cid:5) To decide whether to skip the prox or not 11: if θ t = 1 then 12: x t +1 = prox γp ψ (cid:0) ˆ x t +1 − γp h t (cid:1) (cid:5) Apply prox, but only very rarely!

Original: (with probability p ) 13: else 14: x t +1 =

Original: ˆ x t + (cid:5)

Original: 15: end if 16:

Original: [Equation]

Original: [Equation]

Original: : end

Original: Our experimental results are summarized Figure 11; we use the values δ = δ (cid:48)

Original: = 10 − 2 .

Original: Clearly, and thanks to communication compression, ProxSkip-HUB has beneﬁt in terms of the total cost compared to ProxSkip-LSVRG , and can reach up to three degrees of magnitude!

Original: [Figure]

Original: B Basic Facts B.1 Bregman divergence, L -smoothness and µ -strong convexity

Original: The Bregman divergence of a differentiable function f : R d → R is deﬁned by

Original: [Equation]

Original: It is easy to see that

Original: [Equation]

Original: [Equation]

Original: For an L -smooth and µ -strongly convex function f : R d → R , we have

Original: [Equation]

Original: [Equation]

Original: and

Original: [Equation]

Original: [Equation]

Original: B.2 Firm-nonexpansiveness of the proximity operator

Original: Given ψ : R d → R , we deﬁne ψ ∗ ( y ) :

Original: = sup x ∈ R d {(cid:104) x, y (cid:105) − ψ ( x ) } to be its Fenchel conjugate.

Original: The proximity operator of ψ ∗ satisﬁes for any τ >if u = prox τψ ∗ ( y ) , then u ∈ y − τ∂ψ ∗ ( u ) .

Original: If Assumption 3 is satisﬁed, then ﬁrm nonexpansiveness of the proximity operator implies [Mishchenko et al., 2022] that

Original: [Equation]

Original: 22) for all x, y ∈ R d and any γ, p > 0 .

Original: B.3 Young’s inequality

Original: For any two vectors a, b ∈ R d , we have

Original: [Equation]

Original: B.4 Jensen’s inequality

Original: For a convex function h : R d ← R and any vectors x 1 , . . .

Original: , x n ∈ R d , we have

Original: [Equation]

Original: Applying this to the squared norm, h ( x ) = (cid:107) x (cid:107) 2 , we get (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)n n (cid:88)

Original: [Equation]

Original: n

Original: [Equation]

Original: (cid:88)

Original: [Equation]

Original: (cid:107)

Original: [Equation]

Original: (cid:107)

Original: [Equation]

Original: C Analysis of ProxSkip-VR

Original: In this section we provide the proof of Theorem 5.

Original: C.1 Main lemma of ProxSkip

Original: We start from Lemma 1 initially introduced by Mishchenko et al.

Original: [2022]; for completeness we provide the whole proof.

Original: Let us deﬁne two additional sequences:

Original: [Equation]

Original: Lemma 1.

Original: If Assumption 3 holds, γ > 0 and 0 < p ≤ 1 , then the iterates of ProxSkip-VR satisfy

Original: [Equation]

Original: Proof.

Original: In order to simplify, let us deﬁne two points:

Original: [Equation]

Original: STEP 1 (Optimality conditions).

Original: Using the ﬁrst-order optimality conditions for f + ψ and using h (cid:63) := ∇ f

Original: ( x (cid:63) ) , we obtain the following ﬁxed-point identity for x (cid:63) :

Original: [Equation]

Original: [Equation]

Original: STEP 2 (Recalling the steps of the method).

Original: Recall that the vectors x t and h t are in Algorithmupdated as follows:

Original: [Equation]

Original: probability p

Original: [Equation]

Original: with probability

Original: [Equation]

Original: (30) and

Original: [Equation]

Original: probability

Original: [Equation]

Original: probability

Original: [Equation]

Original: Let us consider the expected value V t +1 :

Original: = E

Original: [Equation]

Original: +

Original: [Equation]

Original: V t +

Original: [Equation]

Original: =

Original: [Equation]

Original: =

Original: [Equation]

Original: STEP 4 (Applying ﬁrm nonexpansiveness).

Original: Applying ﬁrm nonexpansiveness of the proximal operator (22), this leads to the inequality

Original: [Equation]

Original: STEP 5 (Simple algebra).

Original: Next

Original: [Equation]

Original: we expand the squared norm and collect the terms, obtaining

Original: [Equation]

Original: Finally, note that by our deﬁnition of ˆ w t

Original: [Equation]

Original: we have the identity ˆ x t +1 =

Original: ˆ w t + γh t .

Original: Therefore, the ﬁrst two terms above can be rewritten as

Original: [Equation]

Original: Finally, plugging (34) into (33), we get:

Original: [Equation]

Original: C.2 Main lemma

Original: This lemma allows us to obtain a useful recursion for variance-reduced stochastic estimators used in our ProxSkip-VR algorithm

Original: [Equation]

Original: Lemma 2.

Original: Let Assumptions 2 and 4 hold.

Original: Then the iterates of ProxSkip-VR satisfy

Original: [Equation]

Original: Proof.

Original: We start from the deﬁnitions of the auxiliary sequence ˆ w t (see (26)):

Original: [Equation]

Original: Taking expectation in (35) and using unbiasedness of ˆ g t (see (5) in Assumption 4)

Original: [Equation]

Original: we get

Original: [Equation]

Original: Let us now consider the inner product in (36).

Original: Using (19) and (18)

Original: [Equation]

Original: we obtain

Original: [Equation]

Original: To bound the last term in (37)

Original: [Equation]

Original: we can apply Assumption 4:

Original: [Equation]

Original: Plugging (38) into (37) gives us

Original: [Equation]

Original: which is what we set out to prove

Original: [Equation]

Original: C.3 Proof of Theorem

Original: Proof.

Original: Using deﬁnition of the Lyapunov function Ψ t , and the tower property of conditional expectation, we obtain

Original: [Equation]

Original: Using the stepsize bound γ ≤

Original: [Equation]

Original: ˜ A , this leads to

Original: [Equation]

Original: Let us denote β := B + W ˜ B W .

Original: In order to obtain a contraction, we need to have β < 1 , which is satisﬁed when W > B 1 − ˜ B , and we get

Original: [Equation]

Original: [Equation]

Original: Finally

Original: [Equation]

Original: using the tower property of expectation and unrolling recursion (41)

Original: [Equation]

Original: we get

Original: [Equation]

Original: D Examples of Methods Without Variance Reduction D.1 Proof of Theorem 6 (GD estimator)

Original: Proof.

Original: Let us show that GD estimator (ˆ g t = ∇ f ( x t ))

Original: satisﬁes Assumption

Original: [Equation]

Original: This means that Assumption 4 is satisﬁed with the following constant:

Original: [Equation]

Original: Applying Theorem 5 leads to ﬁnal recursion:

Original: [Equation]

Original: By inspecting (42) it is easy to see that

Original: [Equation]

Original: [Equation]

Original: Then the communication complexity is equal to

Original: [Equation]

Original: [Equation]

Original: Setting γ = 1 L and solving pLµ = 1 p gives the optimal probability

Original: [Equation]

Original: [Equation]

Original: Finally

Original: [Equation]

Original: the iteration complexity and communication complexity have the following form:

Original: [Equation]

Original: This recovers the result obtained by Mishchenko et al.

Original: D.2 Proof

Original: of Theorem 8 (SGD estimator)

Original: Proof.

Original: Let us show that the SGD estimator ˆ g t =

Original: g ( x t , ξ t ) satisfying Assumption 7 also satisﬁes

Original: Assumption 4.

Original: Using Young’s inequality we get

Original: [Equation]

Original: This means that Assumption 4 is satisﬁed with the following constants:

Original: [Equation]

Original: Applying Theorem 5 leads to the ﬁnal bound:

Original: [Equation]

Original: In order to minimize the number of prox evaluations, whatever the choice of γ will be, we choose the smallest probability p which does not lead to any degradation of the rate min { γµ, p 2 } .

Original: That is, we choose p = √ γµ .

Original: The ﬁrst term on the right-hand side of (49) can be bounded as follows:

Original: [Equation]

Original: The second term on the right-hand side of (49) can be bounded as follows:

Original: [Equation]

Original: We choose the largest stepsize consistent with bounds γ ≤ εµ 2 C and γ ≤ 1 A :

Original: [Equation]

Original: Using this stepsizem we get the following iteration and (expected) communication complexities:

Original: [Equation]

Original: This recovers the result obtained by Mishchenko et al.

Original: E Analysis of ProxSkip-HUB

Original: In this section we provide analysis of the new algorithm ProxSkip-HUB , which works for the new FL formulation described in Section 4.

Original: The pseudocode is presented in Algorithm 2.

Original: E.1 Lemma for minibatch sampling

Original: Fix a minibatch size τ ∈ { 1 , 2 , . . . , n } and let S t be a random subset of { 1 , 2 , . . . , n } of size τ , chosen uniformly at random.

Original: Deﬁne the gradient estimator via

Original: [Equation]

Original: Lemma 3.

Original: The gradient estimator g ( x ) deﬁned in (50) is unbiased.

Original: If we further assume that n ≥ 2 , (cid:101) φ j is convex and L j -smooth for all j , and f is L -smooth, then

Original: [Equation]

Original: where

Original: [Equation]

Original: Proof.

Original: Let χ j be the random variable deﬁned by

Original: [Equation]

Original: It is easy to show that

Original: [Equation]

Original: Unbiasedness of g ( x ) now follows via direct computation:

Original: [Equation]

Original: Let us deﬁne

Original: [Equation]

Original: Let χ j,k be the random variable deﬁned by

Original: [Equation]

Original: and

Original: [Equation]

Original: ∈

Original: [Equation]

Original: otherwise .

Original: Note that

Original: [Equation]

Original: Further, it is easy to show that

Original: [Equation]

Original: Let us consider

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Equation]

Original: Using the formulas (51) and (54) we can continue:

Original: [Equation]

Original:   =

Original: [Equation]

Original: n

Original: [Equation]

Original: (cid:13) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) .

Original: Since (cid:101) φ j is convex and L j -smooth, we know that

Original: [Equation]

Original: Since f is convex and L -smooth, we know that

Original: [Equation]

Original: Let us apply the bound L j ≤ max j L j and use the identity D f ( x t , x (cid:63) )

Original: = 1 n (cid:80) nj =1 D (cid:101) φ j ( x t , x

Original: [Equation]

Original: E 

Original: [Equation]

Original: 

Original: [Equation]

Original: n n (cid:88)

Original: j =

Original: [Equation]

Original: j

Original: [Equation]

Original: φ

Original: [Equation]

Original: t

Original: [Equation]

Original: (cid:63)

Original: [Equation]

Original: E.2 Proof of Theorem

Original: As in previous analysis we need to show that Assumption 4 is satisﬁed for the ProxSkip-HUB method.

Original: Proof.

Original: Let us consider the ﬁrst inequality in Assumption 4 and show that it holds for the new gradient estimator ˆ g

Original: [Equation]

Original: 

Original: [Equation]

Original: Let ∆ t = 1 τ (cid:80) j ∈S t ( ∇ (cid:101) φ j ( x t ) −∇ (cid:101) φ j ( y t )) and ˆ∆ t = 1 τ (cid:80) j ∈S t Q ( ∇ (cid:101) φ j ( x t ) −∇ (cid:101) φ j ( y t )) .

Original: Using smart zero 0 = ∆ t − ∆ t we have

Original: [Equation]

Original: Let us consider the ﬁrst term (60), let us deﬁne θ it =

Original: Q

Original: [Equation]

Original: ∇

Original: [Equation]

Original: [Equation]

Original: Using independence and unbiasedness of compressors we have

Original: [Equation]

Original: [Equation]

Original: Using Young’s inequality and expectation of client sampling we get

Original: [Equation]

Original: Let us consider the second term in (60):

Original: [Table]

Original: 

Original: [Table]

Original: Using Lemma 3 and Jensen’s inequality (25) we have

Original: [Table]

Original: 

Original: 

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: 

Original: [Equation]

Original: Combining two parts (62), (64) and plugging into (60) we get

Original: E (cid:2) (cid:107) ˆ g

Original: [Equation]

Original: − ∇ f ( x (cid:63) )

Original: (cid:107) 2 (cid:3) ≤ 2 E

Original: [Equation]

Original: j ∈S

Original: [Equation]

Original: [Equation]

Original: (cid:101) φ j ( x t ) − ∇ (cid:101) φ j ( y t ))

Original: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2 

Original: 

Original: [Equation]

Original:   (cid:13) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]

Original: (cid:88)

Original: j

Original: [Equation]

Original: t (

Original: [Equation]

Original: (cid:101)

Original: [Equation]

Original: (cid:63)

Original: [Equation]

Original: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2 

Original: 

Original: [Equation]

Original: = (cid:13)(cid:13)(cid:13)

Original: [Equation]

Original: (cid:13)(cid:13)(cid:13)

Original: [Equation]

Original: ≤

Original: [Equation]

Original: (65) where σ t = 1 n (cid:80) nj =1 (cid:107)∇ (cid:101) φ j ( y t ) − ∇ (cid:101)

Original: φ j ( x (cid:63) )

Original: (cid:107) 2 .

Original: Let us consider update of control variable y t :

Original: [Equation]

Original: with probability q y t with probability

Original: [Equation]

Original: [Equation]

Original: Let us show that second inequality in Assumption 4:

Original: [Equation]

Original: Using (65) and (72) bounds we can conﬁrm that Assumption 4 is satisﬁed with the following constants:

Original: [Equation]

Original: Applying Theorem 5 leads to the ﬁnal result

Original: [Equation]

Original: where the Lyapunov function is deﬁned by

Original: [Equation]

Original: Let us set γ = 1 A + W ˜ A , p = √ γµ and q = 2 γµ .

Original: Using the same proof as for ProxSkip in Section D. and L ( τ ) ≤ L max we get communication and iteration complexities:

Original: [Equation]

Original: If we use full participation τ = n and q = 1 ω +1 and r ( x ) ≡ 0 then we get the same rate as for

Original: DIANA [Mishchenko et al., 2019, Horváth et al., 2019b] and RAND-DIANA [Shulgin and Richtárik, 2021].

Original: F Analysis of ProxSkip-LSVRG

Original: The analysis of ProxSkip-LSVRG is almost the same to the analysis of ProxSkip-HUB , with one exception.

Original: We use a different sigma component:

Original: [Equation]

Original: Let us consider E (cid:2) (cid:107) ˆ g t − ∇ f ( x (cid:63) ) (cid:107) 2 (cid:3) :

Original: E (cid:2) (cid:107) ˆ g t − ∇ f ( x (cid:63) ) (cid:107) 2 (cid:3) =

Original: E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 τ (cid:88) j ∈ S t (cid:16) ∇ (cid:101) φ j

Original: [Equation]

Original:  (70)

Original: Using (63) we have

Original: [Equation]

Original: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]

Original: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]

Original: 

Original: [Equation]

Original: Let us show that second inequality in Assumption 4, using Lemma 3 we get

Original: E [ σ t +

Original: [Equation]

Original: ] = E

Original: [Equation]

Original: [Equation]

Original: We showed that Assumption 4 is satisﬁed with following constants:

Original: [Equation]

Original: Applying Theorem 5 with γ = 1 6 L ( τ ) we get ﬁnal bound:

Original: [Equation]

Original: where the Lyapunov function is deﬁned as

Original: [Equation]

Original: Using the same argument as for ProxSkip and setting q 2 = γµ , we get

Original: [Equation]

Original: If r ( x ) ≡ 0 , this recovers results of Kovalev et al.

Original: [2020a].