Original: F EDERATED L EARNING OF A M IXTURE OF G LOBAL AND L OCAL M ODELS
Revised: F EDERATED L EARNING OF A M IXTURE OF G LOBAL AND L OCAL M ODELS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose a new optimization formulation for training federated learning models.
Revised: We propose a new optimization formulation for training federated learning models.

Original: The standard formulation has the form of an empirical risk minimization problem constructed to ﬁnd a single global model trained from the private data stored across all participating devices.
Revised: The standard formulation has the form of an empirical risk minimization problem constructed to ﬁnd a single global model trained from the private data stored across all participating devices.

Original: In contrast, our formulation seeks an explicit trade-off between this traditional global model and the local models, which can be learned by each device from its own private data without any communication.
Revised: In contrast, our formulation seeks an explicit trade-off between this traditional global model and the local models, which can be learned by each device from its own private data without any communication.

Original: Further, we develop several efﬁcient variants of SGD (with and without partial participation and with and without variance reduction) for solving the new formulation and prove communication complexity guarantees.
Revised: Further, we develop several efﬁcient variants of SGD (with and without partial participation and with and without variance reduction) for solving the new formulation and prove communication complexity guarantees.

Original: Notably, our methods are similar but not identical to federated averaging / local SGD, thus shedding some light on the essence of the elusive method.
Revised: Notably, our methods are similar but not identical to federated averaging / local SGD, thus shedding some light on the essence of the elusive method.

Original: In particular, our methods do not perform full averaging steps and instead merely take steps towards averaging.
Revised: In particular, our methods do not perform full averaging steps and instead merely take steps towards averaging.

Original: We argue for the beneﬁts of this new paradigm for federated learning.
Revised: We argue for the beneﬁts of this new paradigm for federated learning.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: With the proliferation of mobile phones, wearable devices, tablets, and smart home devices comes an increase in the volume of data captured and stored on them.
Revised: With the proliferation of mobile phones, wearable devices, tablets, and smart home devices comes an increase in the volume of data captured and stored on them.

Original: This data contains a wealth of potentially useful information to the owners of these devices, and more so if appropriate machine learning models could be trained on the heterogeneous data stored across the network of such devices.
Revised: This data contains a wealth of potentially useful information to the owners of these devices, and more so if appropriate machine learning models could be trained on the heterogeneous data stored across the network of such devices.

Original: The traditional approach involves moving the relevant data to a data center where centralized machine learning techniques can be efﬁciently applied (Dean et al., 2012;
Revised: The traditional approach involves moving the relevant data to a data center where centralized machine learning techniques can be efﬁciently applied (Dean et al., 2012;

Original: Reddi et al., 2016).
Revised: Reddi et al., 2016).

Original: However, this approach is not without issues.
Revised: However, this approach is not without issues.

Original: First, many device users are increasingly sensitive to privacy concerns and prefer their data to never leave their devices.
Revised: First, many device users are increasingly sensitive to privacy concerns and prefer their data to never leave their devices.

Original: Second, moving data from their place of origin to a centralized location is very inefﬁcient in terms of energy and time.
Revised: Second, moving data from their place of origin to a centralized location is very inefﬁcient in terms of energy and time.

Original: 1. F EDERATED LEARNING
Revised: 1. F EDERATED LEARNING

Original: Federated learning (FL) (McMahan et al., 2016; Kone ˇ cn ´ y et al., 2016b;a; McMahan et al., 2017) has emerged as an interdisciplinary ﬁeld focused on addressing these issues by training machine learning models directly on edge devices.
Revised: Federated learning (FL) (McMahan et al., 2016; Kone ˇ cn ´ y et al., 2016b;a; McMahan et al., 2017) has emerged as an interdisciplinary ﬁeld focused on addressing these issues by training machine learning models directly on edge devices.

Original: The currently prevalent paradigm (Li et al., 2019; Kairouz et al., 2019) casts supervised FL as an empirical risk minimization problem of the form
Revised: The currently prevalent paradigm (Li et al., 2019; Kairouz et al., 2019) casts supervised FL as an empirical risk minimization problem of the form

Original: [Equation]
Revised: [Equation]

Original: where n is the number of devices participating in training, x ∈ R d encodes the d parameters of a global model (e.g., weights of a neural network) and
Revised: where n is the number of devices participating in training, x ∈ R d encodes the d parameters of a global model (e.g., weights of a neural network) and

Original: [Equation]
Revised: [Equation]

Original: represents the aggregate loss of model x on the local data represented by distribution D i stored on device i .
Revised: represents the aggregate loss of model x on the local data represented by distribution D i stored on device i .

Original: One of the deﬁning characteristics of FL is that the data distributions D
Revised: One of the deﬁning characteristics of FL is that the data distributions D

Original: i may possess very different properties across the devices.
Revised: i may possess very different properties across the devices.

Original: Hence, any potential FL method is explicitly required to be able to work under the heterogeneous data setting.
Revised: Hence, any potential FL method is explicitly required to be able to work under the heterogeneous data setting.

Original: The most popular method for solving (1) in the context of FL is the FedAvg algorithm (McMahan et al., 2016).
Revised: The most popular method for solving (1) in the context of FL is the FedAvg algorithm (McMahan et al., 2016).

Original: In its most simple form, when one does not employ partial participation, model compression, or stochastic approximation, FedAvg reduces to Local Gradient Descent (LGD) (Khaled et al., 2019; 2020), which is an extension of GD performing more than a single gradient step on each device before aggregation.
Revised: In its most simple form, when one does not employ partial participation, model compression, or stochastic approximation, FedAvg reduces to Local Gradient Descent (LGD) (Khaled et al., 2019; 2020), which is an extension of GD performing more than a single gradient step on each device before aggregation.

Original: FedAvg has been shown to work well empirically, particularly for non-convex problems, but comes with poor convergence guarantees compared to the non-local counterparts when data are heterogeneous.
Revised: FedAvg has been shown to work well empirically, particularly for non-convex problems, but comes with poor convergence guarantees compared to the non-local counterparts when data are heterogeneous.

Original: Some issues with current approaches to FL
Revised: Some issues with current approaches to FL

Original: The ﬁrst motivation for our research comes from the appreciation that data heterogeneity does not merely present challenges to the design of new provably efﬁcient training methods for solving (1), but also inevitably raises questions about the utility of such a global solution to individual users.
Revised: The ﬁrst motivation for our research comes from the appreciation that data heterogeneity does not merely present challenges to the design of new provably efﬁcient training methods for solving (1), but also inevitably raises questions about the utility of such a global solution to individual users.

Original: Indeed, a global model trained across all the data from all devices might be so removed from the typical data and usage patterns experienced by an individual user as to render it virtually useless.
Revised: Indeed, a global model trained across all the data from all devices might be so removed from the typical data and usage patterns experienced by an individual user as to render it virtually useless.

Original: This issue has been observed before, and various approaches have been proposed to address it.
Revised: This issue has been observed before, and various approaches have been proposed to address it.

Original: For instance, the MOCHA (Smith et al., 2017) framework uses a multi-task learning approach to allow for personalization.
Revised: For instance, the MOCHA (Smith et al., 2017) framework uses a multi-task learning approach to allow for personalization.

Original: Next, (Khodak et al., 2019) propose a generic online algorithm for gradientbased parameter-transfer meta-learning and demonstrate improved practical performance over FedAvg (McMahan et al., 2017).
Revised: Next, (Khodak et al., 2019) propose a generic online algorithm for gradientbased parameter-transfer meta-learning and demonstrate improved practical performance over FedAvg (McMahan et al., 2017).

Original: Approaches based on variational inference (Corinzia & Buhmann, 2019), cyclic patterns in practical FL data sampling (Eichner et al., 2019) transfer learning (Zhao et al., 2018) and explicit model mixing (Peterson et al., 2019) have been proposed.
Revised: Approaches based on variational inference (Corinzia & Buhmann, 2019), cyclic patterns in practical FL data sampling (Eichner et al., 2019) transfer learning (Zhao et al., 2018) and explicit model mixing (Peterson et al., 2019) have been proposed.

Original: The second motivation for our work is the realization that even very simple variants of FedAvg, such as LGD, which should be easier to analyze, fail to provide theoretical improvements in communication complexity over their non-local cousins, in this case, GD (Khaled et al., 2019; 2020).
Revised: The second motivation for our work is the realization that even very simple variants of FedAvg, such as LGD, which should be easier to analyze, fail to provide theoretical improvements in communication complexity over their non-local cousins, in this case, GD (Khaled et al., 2019; 2020).

Original: This observation is at odds with the practical success of local methods in FL.
Revised: This observation is at odds with the practical success of local methods in FL.

Original: This leads us to ask the question: if LGD does not theoretically improve upon GD as a solver for the traditional global problem (1) , perhaps LGD should not be seen as a method for solving (1) at all.
Revised: This leads us to ask the question: if LGD does not theoretically improve upon GD as a solver for the traditional global problem (1) , perhaps LGD should not be seen as a method for solving (1) at all.

Original: In such a case, what problem does LGD solve?
Revised: In such a case, what problem does LGD solve?

Original: A good answer to this question would shed light on the workings of LGD, and by analogy, on the role local steps play in more elaborate FL methods such as local
Revised: A good answer to this question would shed light on the workings of LGD, and by analogy, on the role local steps play in more elaborate FL methods such as local

Original: SGD (Stich, 2020; Khaled et al., 2020) and FedAvg.
Revised: SGD (Stich, 2020; Khaled et al., 2020) and FedAvg.

Original:  C ONTRIBUTIONS
Revised:  C ONTRIBUTIONS

Original: In our work we argue that the two motivations mentioned in the introduction point in the same direction, i.e., we show that a single solution can be devised addressing both problems at the same time.
Revised: In our work we argue that the two motivations mentioned in the introduction point in the same direction, i.e., we show that a single solution can be devised addressing both problems at the same time.

Original: Our main contributions are:
Revised: Our main contributions are:

Original: (cid:5) New formulation of FL which seeks an implicit mixture of global and local models.
Revised: (cid:5) New formulation of FL which seeks an implicit mixture of global and local models.

Original: We
Revised: We

Original: propose a new optimization formulation of FL.
Revised: propose a new optimization formulation of FL.

Original: Instead
Revised: Instead

Original: of learning a single global model by solving (
Revised: of learning a single global model by solving (

Original: ), we propose to learn a
Revised: ), we propose to learn a

Original: mixture of
Revised: mixture of

Original: the
Revised: the

Original: global model
Revised: global model

Original: and the
Revised: and the

Original: purely local
Revised: purely local

Original: models which can be
Revised: models which can be

Original: trained by each
Revised: trained by each

Original: device
Revised: device

Original: i using its data D i
Revised: i using its data D i

Original: only
Revised: only

Original: Our formulation (see
Revised: Our formulation (see

Original: Sec
Revised: Sec

Original: . 3) lifts the problem from R d to R nd , allowing each device i to learn a personalized model x i ∈ R d .
Revised: . 3) lifts the problem from R d to R nd , allowing each device i to learn a personalized model x i ∈ R d .

Original: These personalized models are encouraged to not depart too much from their mean by the inclusion of a quadratic penalty ψ multiplied by a parameter λ ≥ 0 .
Revised: These personalized models are encouraged to not depart too much from their mean by the inclusion of a quadratic penalty ψ multiplied by a parameter λ ≥ 0 .

Original: Admittedly, the idea of softly-enforced similarity of the local models was already introduced in the domain of the multi-task relationship learning (Zhang & Yeung, 2010; Liu et al., 2017; Wang et al., 2018) and distributed optimization (Lan et al., 2018;
Revised: Admittedly, the idea of softly-enforced similarity of the local models was already introduced in the domain of the multi-task relationship learning (Zhang & Yeung, 2010; Liu et al., 2017; Wang et al., 2018) and distributed optimization (Lan et al., 2018;

Original: Gorbunov et al., 2019; Zhang et al., 2015).
Revised: Gorbunov et al., 2019; Zhang et al., 2015).

Original: The mixture objective we propose (see (2)) is a special case of their setup, which justiﬁes our approach from the modeling perspective.
Revised: The mixture objective we propose (see (2)) is a special case of their setup, which justiﬁes our approach from the modeling perspective.

Original: Note that Zhang et al.
Revised: Note that Zhang et al.

Original: ; Liu et al.
Revised: ; Liu et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: (2018) provide efﬁcient algorithms to solve the mixture objective already.
Revised: (2018) provide efﬁcient algorithms to solve the mixture objective already.

Original: However, none of the mentioned papers consider the FL application, nor they shed a light on the communication complexity of LGD algorithms, which we do in our work.
Revised: However, none of the mentioned papers consider the FL application, nor they shed a light on the communication complexity of LGD algorithms, which we do in our work.

Original: (cid:5) Theoretical properties of the new formulation.
Revised: (cid:5) Theoretical properties of the new formulation.

Original: We study the properties of the optimal
Revised: We study the properties of the optimal

Original: solution
Revised: solution

Original: of our formulation
Revised: of our formulation

Original: , thus developing an algorithmic
Revised: , thus developing an algorithmic

Original: free
Revised: free

Original: theory.
Revised: theory.

Original: When
Revised: When

Original: the penalty parameter is set to zero,
Revised: the penalty parameter is set to zero,

Original: then obviously, each device
Revised: then obviously, each device

Original: is
Revised: is

Original: allowed to train their own
Revised: allowed to train their own

Original: model without any dependence on the data stored on other devices.
Revised: model without any dependence on the data stored on other devices.

Original: Such purely local models are rarely useful.
Revised: Such purely local models are rarely useful.

Original: We prove that the optimal local models converge to the traditional global model characterized by (1) at the rate O (1 /λ ) .
Revised: We prove that the optimal local models converge to the traditional global model characterized by (1) at the rate O (1 /λ ) .

Original: We also show that the total loss evaluated at the local models is never higher than the total loss evaluated at the global model (see Thm. 3.1).
Revised: We also show that the total loss evaluated at the local models is never higher than the total loss evaluated at the global model (see Thm. 3.1).

Original: Moreover, we prove an insightful structural result for the optimal local models: the optimal model learned by device i arises by subtracting the gradient of the loss function stored on that device evaluated at the same point (i.e., a local model) from the average of the optimal local models (see Thm. 3.2).
Revised: Moreover, we prove an insightful structural result for the optimal local models: the optimal model learned by device i arises by subtracting the gradient of the loss function stored on that device evaluated at the same point (i.e., a local model) from the average of the optimal local models (see Thm. 3.2).

Original: As a byproduct, this theoretical result sheds new light on the key update step in the model agnostic meta-learning (MAML) method (Finn et al., 2017), which has a similar but subtly different structure.
Revised: As a byproduct, this theoretical result sheds new light on the key update step in the model agnostic meta-learning (MAML) method (Finn et al., 2017), which has a similar but subtly different structure.

Original: The subtle difference is that the MAML update obtains the local model by subtracting the gradient evaluated at the global model.
Revised: The subtle difference is that the MAML update obtains the local model by subtracting the gradient evaluated at the global model.

Original: While MAML was originally proposed as a heuristic, we provide rigorous theoretical guarantees.
Revised: While MAML was originally proposed as a heuristic, we provide rigorous theoretical guarantees.

Original: (cid:5) Loopless LGD: non-uniform SGD applied to our formulation.
Revised: (cid:5) Loopless LGD: non-uniform SGD applied to our formulation.

Original: We then propose a randomized gradient-based method— Loopless Local Gradient Descent (L2GD) —for solving our new formulation (Algorithm 1).
Revised: We then propose a randomized gradient-based method— Loopless Local Gradient Descent (L2GD) —for solving our new formulation (Algorithm 1).

Original: This method is, in fact, a non-standard application of SGD to our problem, and can be seen as an instance of SGD with non-uniform sampling applied to the problem of minimizing the sum of two convex functions (Zhao & Zhang, 2015; Gower et al., 2019): the average loss, and the penalty.
Revised: This method is, in fact, a non-standard application of SGD to our problem, and can be seen as an instance of SGD with non-uniform sampling applied to the problem of minimizing the sum of two convex functions (Zhao & Zhang, 2015; Gower et al., 2019): the average loss, and the penalty.

Original: When the loss function is selected by the randomness in our SGD method, the stochastic gradient step can be interpreted as the execution of a single local GD step on each device.
Revised: When the loss function is selected by the randomness in our SGD method, the stochastic gradient step can be interpreted as the execution of a single local GD step on each device.

Original: Since we set the probability of the loss being sampled to be high, this step is typically repeated multiple times, resulting in multiple local GD steps.
Revised: Since we set the probability of the loss being sampled to be high, this step is typically repeated multiple times, resulting in multiple local GD steps.

Original: In contrast to standard LGD, the number of local steps is not ﬁxed, but random, and follows a geometric distribution.
Revised: In contrast to standard LGD, the number of local steps is not ﬁxed, but random, and follows a geometric distribution.

Original: This mechanism is similar in spirit to how the recently proposed loopless variants of SVRG (Hofmann et al., 2015; Kovalev et al., 2020) work in comparison with the original SVRG (Johnson & Zhang, 2013a; Xiao & Zhang, 2014).
Revised: This mechanism is similar in spirit to how the recently proposed loopless variants of SVRG (Hofmann et al., 2015; Kovalev et al., 2020) work in comparison with the original SVRG (Johnson & Zhang, 2013a; Xiao & Zhang, 2014).

Original: Once the penalty is sampled by our method, the resultant SGD step can be interpreted as the execution of an aggregation step.
Revised: Once the penalty is sampled by our method, the resultant SGD step can be interpreted as the execution of an aggregation step.

Original: In contrast with standard aggregation, which performs full averaging of the local models, our method merely takes a step towards averaging .
Revised: In contrast with standard aggregation, which performs full averaging of the local models, our method merely takes a step towards averaging .

Original: However, the step is relatively large.
Revised: However, the step is relatively large.

Original: (cid:5) Convergence theory.
Revised: (cid:5) Convergence theory.

Original: By adapting the general theory from (Gower et al., 2019) to our setting, we obtain theoretical convergence guarantees assuming that each f i is L -smooth and µ -strongly convex (see Thm. 4.2).
Revised: By adapting the general theory from (Gower et al., 2019) to our setting, we obtain theoretical convergence guarantees assuming that each f i is L -smooth and µ -strongly convex (see Thm. 4.2).

Original: Interestingly, by optimizing the probability of sampling the penalty (we get p (cid:63) = λλ + L ), which is an indirect way of ﬁxing the expected number of local steps to 

Revised: Interestingly, by optimizing the probability of sampling the penalty (we get p (cid:63) = λλ + L ), which is an indirect way of ﬁxing the expected number of local steps to

Original: + Lλ , we prove an 2 λ λ + L Lµ log 1 ε bound on the expected number of communication rounds (see Cor. 4.3).
Revised: + Lλ , we prove an 2 λ λ + L Lµ log 1 ε bound on the expected number of communication rounds (see Cor. 4.3).

Original: We believe that this is remarkable in several ways.
Revised: We believe that this is remarkable in several ways.

Original: By choosing λ small, we tilt our goal towards pure local models: the number of communication rounds is tending to 0 as λ → 0 .
Revised: By choosing λ small, we tilt our goal towards pure local models: the number of communication rounds is tending to 0 as λ → 0 .

Original: If λ → ∞ , the solution our formulation converges to is the optimal global model, and L2GD obtains the communication bound
Revised: If λ → ∞ , the solution our formulation converges to is the optimal global model, and L2GD obtains the communication bound

Original: O (cid:16) Lµ log 1 ε (cid:17) , which matches the efﬁciency of GD.
Revised: O (cid:16) Lµ log 1 ε (cid:17) , which matches the efﬁciency of GD.

Original: (cid:5) What problem do local methods solve?
Revised: (cid:5) What problem do local methods solve?

Original: Noting that L2GD is a (mildly nonstandard) version of
Revised: Noting that L2GD is a (mildly nonstandard) version of

Original: LGD, 3 which is a key method most local methods for FL are based on, and noting that, as we show,
Revised: LGD, 3 which is a key method most local methods for FL are based on, and noting that, as we show,

Original: L2GD solves our new formulation of FL, we offer a new and surprising interpretation of the role of local steps in FL.
Revised: L2GD solves our new formulation of FL, we offer a new and surprising interpretation of the role of local steps in FL.

Original: In particular, the role of local steps in gradient type methods, such as GD, is not to reduce communication complexity, as is generally believed.
Revised: In particular, the role of local steps in gradient type methods, such as GD, is not to reduce communication complexity, as is generally believed.

Original: Indeed, there is no theoretical result supporting this claim in the key heterogeneous data regime.
Revised: Indeed, there is no theoretical result supporting this claim in the key heterogeneous data regime.

Original: Instead, their role is to steer the method towards ﬁnding a mixture of the traditional global and the purely local models.
Revised: Instead, their role is to steer the method towards ﬁnding a mixture of the traditional global and the purely local models.

Original: Given that the stepsize is ﬁxed, the more local steps are taken, the more we bias the method towards the purely local models.
Revised: Given that the stepsize is ﬁxed, the more local steps are taken, the more we bias the method towards the purely local models.

Original: Our new optimization formulation of FL formalizes this as it deﬁnes the problem that local methods, in this case L2GD, solve.
Revised: Our new optimization formulation of FL formalizes this as it deﬁnes the problem that local methods, in this case L2GD, solve.

Original: There is an added beneﬁt here: the more we want our formulation to be biased towards purely local models (i.e., the smaller the penalty parameter λ is), the more local steps does L2GD take, and the better the total communication complexity of L2GD becomes.
Revised: There is an added beneﬁt here: the more we want our formulation to be biased towards purely local models (i.e., the smaller the penalty parameter λ is), the more local steps does L2GD take, and the better the total communication complexity of L2GD becomes.

Original: Hence, despite a lot of research on this topic, our paper provides the ﬁrst proof that a local method (e.g., L2GD) can be better than its non-local counterpart (e.g., GD) in terms of total communication complexity in the heterogeneous data setting.
Revised: Hence, despite a lot of research on this topic, our paper provides the ﬁrst proof that a local method (e.g., L2GD) can be better than its non-local counterpart (e.g., GD) in terms of total communication complexity in the heterogeneous data setting.

Original: We are able to do this by noting that local methods should better be seen as methods for solving the new FL formulation proposed here.
Revised: We are able to do this by noting that local methods should better be seen as methods for solving the new FL formulation proposed here.

Original: (cid:5) Generalizations: partial participation, local SGD and variance reduction.
Revised: (cid:5) Generalizations: partial participation, local SGD and variance reduction.

Original: We further generalize and improve our method by allowing for (i) stochastic partial participation of devices in each communication round,(ii) subsampling on each device which means we can perform local SGD steps instead of local GD steps, and (iii) total variance reduction mechanism to tackle the variance coming from three sources: locality of the updates induced by non-uniform sampling (already present in L2GD), partial participation and local subsampling.
Revised: We further generalize and improve our method by allowing for (i) stochastic partial participation of devices in each communication round,(ii) subsampling on each device which means we can perform local SGD steps instead of local GD steps, and (iii) total variance reduction mechanism to tackle the variance coming from three sources: locality of the updates induced by non-uniform sampling (already present in L2GD), partial participation and local subsampling.

Original: Due to its level of generality, this method, which we call L2SGD++, is presented in the Appendix only, alongside the associated complexity results.
Revised: Due to its level of generality, this method, which we call L2SGD++, is presented in the Appendix only, alongside the associated complexity results.

Original: In the main body of this paper, we instead present a simpliﬁed version thereof, which we call L2SGD+ (Algorithm 3).
Revised: In the main body of this paper, we instead present a simpliﬁed version thereof, which we call L2SGD+ (Algorithm 3).

Original: The convergence theory for it is presented in Thm.
Revised: The convergence theory for it is presented in Thm.

Original: 5.1 and Cor.
Revised: 5.1 and Cor.

Original: (cid:5) Heterogeneous data.
Revised: (cid:5) Heterogeneous data.

Original: All our methods and convergence results allow for fully heterogeneous data and do not depend on any assumptions on data similarity across the devices.
Revised: All our methods and convergence results allow for fully heterogeneous data and do not depend on any assumptions on data similarity across the devices.

Original: (cid:5) Superior empirical performance.
Revised: (cid:5) Superior empirical performance.

Original: We show through ample numerical experiments that our theoretical predictions can be observed in practice.
Revised: We show through ample numerical experiments that our theoretical predictions can be observed in practice.

Original:  N EW F ORMULATION OF FL
Revised:  N EW F ORMULATION OF FL

Original: We now introduce our new formulation for training supervised FL models:
Revised: We now introduce our new formulation for training supervised FL models:

Original: [Equation]
Revised: [Equation]

Original: where λ ≥ 0 is a penalty parameter, x 1 , . .
Revised: where λ ≥ 0 is a penalty parameter, x 1 , . .

Original: , x n ∈ R
Revised: , x n ∈ R

Original: d are local models, x := ( x 1 , x 2 , . . .
Revised: d are local models, x := ( x 1 , x 2 , . . .

Original: , x n ) ∈ R nd and ¯ x := 1 n (cid:80) ni
Revised: , x n ) ∈ R nd and ¯ x := 1 n (cid:80) ni

Original: =1 x i is the average of the local models.
Revised: =1 x i is the average of the local models.

Original: Due to the assumptions on f i we will make in Sec. 3.1, F is strongly convex and hence (2) has a unique solution, which we denote x ( λ ) :
Revised: Due to the assumptions on f i we will make in Sec. 3.1, F is strongly convex and hence (2) has a unique solution, which we denote x ( λ ) :

Original: = ( x 1 ( λ ) , . . .
Revised: = ( x 1 ( λ ) , . . .

Original: , x n ( λ )) ∈ R nd .
Revised: , x n ( λ )) ∈ R nd .

Original: We further let ¯ x ( λ ) :
Revised: We further let ¯ x ( λ ) :

Original: =n (cid:80) ni
Revised: =n (cid:80) ni

Original: =1 x i ( λ ) .
Revised: =1 x i ( λ ) .

Original: We now comment on the rationale behind the new formulation.
Revised: We now comment on the rationale behind the new formulation.

Original: Local models ( λ = 0 ).
Revised: Local models ( λ = 0 ).

Original: Note that for each i , x i (0) solves the local problem min x i ∈
Revised: Note that for each i , x i (0) solves the local problem min x i ∈

Original: R
Revised: R

Original: d f i ( x i ) .
Revised: d f i ( x i ) .

Original: That is, x i (0) is the local model based on data D i stored on device i only.
Revised: That is, x i (0) is the local model based on data D i stored on device i only.

Original: This model can be computed by device i without any communication whatsoever.
Revised: This model can be computed by device i without any communication whatsoever.

Original: Typically, D i is not rich enough for this local model to be useful.
Revised: Typically, D i is not rich enough for this local model to be useful.

Original: In order to learn a better model, one has to take into account the date from other clients as well.
Revised: In order to learn a better model, one has to take into account the date from other clients as well.

Original: This, however, requires communication.
Revised: This, however, requires communication.

Original: Mixed models ( λ ∈ (0 , ∞ ) ).
Revised: Mixed models ( λ ∈ (0 , ∞ ) ).

Original: As λ increases, the penalty λψ ( x ) has an increasingly more substantial effect, and communication is needed to ensure that the models are not too dissimilar, as otherwise the penalty λψ ( x ) would be too large.
Revised: As λ increases, the penalty λψ ( x ) has an increasingly more substantial effect, and communication is needed to ensure that the models are not too dissimilar, as otherwise the penalty λψ ( x ) would be too large.

Original: Global model ( λ = ∞ ).
Revised: Global model ( λ = ∞ ).

Original: Let us now look at the limit case λ → ∞ .
Revised: Let us now look at the limit case λ → ∞ .

Original: Intuitively, this limit case should force the optimal local models to be mutually identical, while minimizing the loss f .
Revised: Intuitively, this limit case should force the optimal local models to be mutually identical, while minimizing the loss f .

Original: In particular, this limit case will solve 4 min (cid:8) f ( x ) : x 1 , . . .
Revised: In particular, this limit case will solve 4 min (cid:8) f ( x ) : x 1 , . . .

Original: , x n ∈ R d , x 1 = x 2 = · · · = x n (cid:9) , which is equivalent to the global formulation (2).
Revised: , x n ∈ R d , x 1 = x 2 = · · · = x n (cid:9) , which is equivalent to the global formulation (2).

Original: Because of this, let us deﬁne x i ( ∞ ) for each i to be the optimal global solution of (1), and let x ( ∞ ) :
Revised: Because of this, let us deﬁne x i ( ∞ ) for each i to be the optimal global solution of (1), and let x ( ∞ ) :

Original: = ( x 1 ( ∞ ) , . . .
Revised: = ( x 1 ( ∞ ) , . . .

Original: , x n ( ∞ )) .
Revised: , x n ( ∞ )) .

Original: 3. T ECHNICAL PRELIMINARIES
Revised: 3. T ECHNICAL PRELIMINARIES

Original: We make the following assumption on the functions f i :
Revised: We make the following assumption on the functions f i :

Original: Assumption 3.1 For each i , the function f i : R d → R is L -smooth and µ -strongly convex.
Revised: Assumption 3.1 For each i , the function f i : R d → R is L -smooth and µ -strongly convex.

Original: For x i , y i ∈ R d , (cid:104) x i , y i (cid:105) denotes the standard inner product and (cid:107) x (cid:107) :
Revised: For x i , y i ∈ R d , (cid:104) x i , y i (cid:105) denotes the standard inner product and (cid:107) x (cid:107) :

Original: = (cid:104) x
Revised: = (cid:104) x

Original: i , x i (cid:105) 1 / 2 is the standard Euclidean norm.
Revised: i , x i (cid:105) 1 / 2 is the standard Euclidean norm.

Original: For vectors x = ( x 1 , . . .
Revised: For vectors x = ( x 1 , . . .

Original: , x n ) ∈ R nd , y =
Revised: , x n ) ∈ R nd , y =

Original: ( y 1 , . . .
Revised: ( y 1 , . . .

Original: , y n ) ∈ R nd we deﬁne the standard inner product and norm via (cid:104) x, y (cid:105) :
Revised: , y n ) ∈ R nd we deﬁne the standard inner product and norm via (cid:104) x, y (cid:105) :

Original: = (cid:80) ni
Revised: = (cid:80) ni

Original: =1 (cid:104) x i , y i (cid:105) , (cid:107) x (cid:107) 2 :=
Revised: =1 (cid:104) x i , y i (cid:105) , (cid:107) x (cid:107) 2 :=

Original: (cid:80) ni =1 (cid:107) x
Revised: (cid:80) ni =1 (cid:107) x

Original: i (cid:107) 2 .
Revised: i (cid:107) 2 .

Original: Note that the separable structure of f implies that ( ∇ f ( x ))
Revised: Note that the separable structure of f implies that ( ∇ f ( x ))

Original: i = 1 n ∇ f i ( x i ) , i.e., ∇ f ( x ) =n ( ∇ f 1 ( x 1 ) , ∇ f 2 ( x 2 ) , . . .
Revised: i = 1 n ∇ f i ( x i ) , i.e., ∇ f ( x ) =n ( ∇ f 1 ( x 1 ) , ∇ f 2 ( x 2 ) , . . .

Original: , ∇ f n ( x n )) .
Revised: , ∇ f n ( x n )) .

Original: Note that Assumption 3.1 implies that f is L f -smooth with L f :=
Revised: Note that Assumption 3.1 implies that f is L f -smooth with L f :=

Original: Ln and µ f -strongly convex with µ f :=
Revised: Ln and µ f -strongly convex with µ f :=

Original: µn .
Revised: µn .

Original: Clearly, ψ is convex by construction.
Revised: Clearly, ψ is convex by construction.

Original: It can be shown that ψ is L ψ -smooth with L ψ = 1 n
Revised: It can be shown that ψ is L ψ -smooth with L ψ = 1 n

Original: (see Appendix).
Revised: (see Appendix).

Original: We can also easily see that ( ∇ ψ ( x ))
Revised: We can also easily see that ( ∇ ψ ( x ))

Original: i = 1 n ( x i − ¯ x ) (see Appendix), which implies ψ ( x ) = n 2 n (cid:80)
Revised: i = 1 n ( x i − ¯ x ) (see Appendix), which implies ψ ( x ) = n 2 n (cid:80)

Original: [Equation]
Revised: [Equation]

Original: 3. C HARACTERIZATION OF OPTIMAL SOLUTIONS
Revised: 3. C HARACTERIZATION OF OPTIMAL SOLUTIONS

Original: Our ﬁrst result describes the behavior of f ( x ( λ )) and ψ ( x ( λ )) as a function of λ .
Revised: Our ﬁrst result describes the behavior of f ( x ( λ )) and ψ ( x ( λ )) as a function of λ .

Original: Theorem 3.1 The function λ → ψ ( x ( λ )) is non-increasing, and for all λ > 

Revised: Theorem 3.1 The function λ → ψ ( x ( λ )) is non-increasing, and for all λ >

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Moreover, the function λ → f ( x ( λ )) is non-decreasing, and for all λ ≥ 

Revised: Moreover, the function λ → f ( x ( λ )) is non-decreasing, and for all λ ≥

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Ineq.
Revised: Ineq.

Original: (3) says that the penalty decreases to zero as λ grows, and hence the optimal local models x i ( λ ) are increasingly similar as λ grows.
Revised: (3) says that the penalty decreases to zero as λ grows, and hence the optimal local models x i ( λ ) are increasingly similar as λ grows.

Original: The second statement suggest that the loss f ( x ( λ )) increases with λ , but never exceeds the optimal global loss f ( x ( ∞ )) of the standard FL formulation (1).
Revised: The second statement suggest that the loss f ( x ( λ )) increases with λ , but never exceeds the optimal global loss f ( x ( ∞ )) of the standard FL formulation (1).

Original: We now characterize the optimal local models which connect our model to the MAML framework (Finn et al., 2017), as mentioned in the introduction.
Revised: We now characterize the optimal local models which connect our model to the MAML framework (Finn et al., 2017), as mentioned in the introduction.

Original: Theorem 3.2 For each λ > 0 and 1 ≤ i ≤ n we have
Revised: Theorem 3.2 For each λ > 0 and 1 ≤ i ≤ n we have

Original: [Equation]
Revised: [Equation]

Original: Further, we have n (cid:80)
Revised: Further, we have n (cid:80)

Original: [Equation]
Revised: [Equation]

Original: The optimal local models (5) are obtained from the average model by subtracting a multiple of the local gradient.
Revised: The optimal local models (5) are obtained from the average model by subtracting a multiple of the local gradient.

Original: Observe that the local gradients always sum up to zero at optimality.
Revised: Observe that the local gradients always sum up to zero at optimality.

Original: This is obviously true for λ = ∞ , but it is a bit less obvious that this holds for any λ > 0 .
Revised: This is obviously true for λ = ∞ , but it is a bit less obvious that this holds for any λ > 0 .

Original: Next, we argue the optimal local models converge to the traditional FL solution at the rate O
Revised: Next, we argue the optimal local models converge to the traditional FL solution at the rate O

Original: (1 /λ ) .
Revised: (1 /λ ) .

Original: Theorem 3.3 Let P ( z ) :
Revised: Theorem 3.3 Let P ( z ) :

Original: = 1 n (cid:80) ni
Revised: = 1 n (cid:80) ni

Original: =

Revised: =

Original: f i ( z ) .
Revised: f i ( z ) .

Original: Then, x ( ∞ ) is the unique minimizer of P and we have
Revised: Then, x ( ∞ ) is the unique minimizer of P and we have

Original: [Equation]
Revised: [Equation]

Original:  L2GD: L OOPLESS L OCAL GD
Revised:  L2GD: L OOPLESS L OCAL GD

Original: In this section we describe a new randomized method for solving the formulation (2).
Revised: In this section we describe a new randomized method for solving the formulation (2).

Original: Our method is a non-uniform SGD for (2) seen as a 2-sum problem, sampling either ∇ f or ∇ ψ to estimate ∇ F .
Revised: Our method is a non-uniform SGD for (2) seen as a 2-sum problem, sampling either ∇ f or ∇ ψ to estimate ∇ F .

Original: Letting 0 < p < 1 , we deﬁne a stochastic gradient of F at x ∈ R nd as follows
Revised: Letting 0 < p < 1 , we deﬁne a stochastic gradient of F at x ∈ R nd as follows

Original: [Equation]
Revised: [Equation]

Original: with probability p
Revised: with probability p

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Clearly, G ( x ) is an unbiased estimator of ∇ F ( x ) .
Revised: Clearly, G ( x ) is an unbiased estimator of ∇ F ( x ) .

Original: This leads to the following method for minimizing
Revised: This leads to the following method for minimizing

Original: F , which we call L2GD: x k +1 = x k − αG ( x k ) .
Revised: F , which we call L2GD: x k +1 = x k − αG ( x k ) .

Original: Plugging the formulas for ∇ f ( x ) and ∇ ψ ( x ) into (7), and writing the resulting method in a distributed manner, we arrive at Algorithm 1.
Revised: Plugging the formulas for ∇ f ( x ) and ∇ ψ ( x ) into (7), and writing the resulting method in a distributed manner, we arrive at Algorithm 1.

Original: In each iteration, a coin ξ is tossed and lands 1 with probability p and 0 with probability 1 − p .
Revised: In each iteration, a coin ξ is tossed and lands 1 with probability p and 0 with probability 1 − p .

Original: If ξ = 0 , all Devices perform one local GD step (8), and if ξ = 1 , Master shifts each local model towards the average via (9).
Revised: If ξ = 0 , all Devices perform one local GD step (8), and if ξ = 1 , Master shifts each local model towards the average via (9).

Original: As we shall see in Sec.
Revised: As we shall see in Sec.

Original: 4.2, our theory limits the value of the stepsize α , which has the effect that the ratio αλnp cannot exceed 12 .
Revised: 4.2, our theory limits the value of the stepsize α , which has the effect that the ratio αλnp cannot exceed 12 .

Original: Hence, (9) is a convex combination of x ki and ¯ x k .
Revised: Hence, (9) is a convex combination of x ki and ¯ x k .

Original: Note that Algorithm 1 is only required to communicate when a two consecutive coin tosses land a different value (see the detailed explanation in Sec.
Revised: Note that Algorithm 1 is only required to communicate when a two consecutive coin tosses land a different value (see the detailed explanation in Sec.

Original: C.1 of the appendix).
Revised: C.1 of the appendix).

Original: Consequently, the expected number of communication rounds in k iterations of L2GD is p (1 − p ) k .
Revised: Consequently, the expected number of communication rounds in k iterations of L2GD is p (1 − p ) k .

Original: Remark 4.

Revised: Remark 4.

Original: Our algorithm designs do not take the data privacy into the consideration.
Revised: Our algorithm statements do not take the data privacy into the consideration.

Original: While privacy is a very important aspect of FL; in this paper, we tackle different FL challenges and thus we ignore privacy issues.
Revised: While privacy is a very important aspect of FL; in this paper, we tackle different FL challenges and thus we ignore privacy issues.

Original: However, our algorithms can be implemented in a private fashion as well using tricks that are used in the classical FL scenario (Bonawitz et al., 2017).
Revised: However, the proposed algorithms can be implemented in a private fashion as well using tricks that are used in the classical FL scenario (Bonawitz et al., 2017).

Original: Input: x 01 = · · · = x
Revised: Input: x 01 = · · · = x

Original: [Equation]
Revised: [Equation]

Original: ∈ R d , stepsize α , probability p for k = 0 , 1 , . . .
Revised: ∈ R d , stepsize α , probability p for k = 0 , 1 , . . .

Original: do
Revised: do

Original: ξ = 1 with probability p and 0 with probability 1 − p
Revised: ξ = 1 with probability p and 0 with probability 1 − p

Original: if ξ = 0 then
Revised: if ξ = 0 then

Original: All Devices i = 1 , . .
Revised: All Devices i = 1 , . .

Original: . , n perform a local GD step:
Revised: . , n perform a local GD step:

Original: [Equation]
Revised: [Equation]

Original: (8) else
Revised: (8) else

Original: Master computes the average ¯ x k = 1 n n (cid:80)
Revised: Master computes the average ¯ x k = 1 n n (cid:80)

Original: [Equation]
Revised: [Equation]

Original: x ki
Revised: x ki

Original: Master for each i computes step towards aggregation
Revised: Master for each i computes step towards aggregation

Original: [Equation]
Revised: [Equation]

Original: (9) end if end for
Revised: (9) end if end for

Original: T HE DYNAMICS OF LOCAL GD AND AVERAGING STEPS
Revised: T HE DYNAMICS OF LOCAL GD AND AVERAGING STEPS

Original: Notice that the average of the local models does not change during an aggregation step.
Revised: Notice that the average of the local models does not change during an aggregation step.

Original: Indeed, ¯ x k +1 is equal to 1 n n (cid:80)
Revised: Indeed, ¯ x k +1 is equal to 1 n n (cid:80)

Original: [Equation]
Revised: [Equation]

Original: If several averaging steps take place in a sequence, the point a = ¯ x k in (9) remains unchanged, and each local model x ki merely moves along the line joining the initial value of the local model at the start of the sequence and a , with each step pushing x ki closer to the average a .
Revised: If several averaging steps take place in a sequence, the point a = ¯ x k in (9) remains unchanged, and each local model x ki merely moves along the line joining the initial value of the local model at the start of the sequence and a , with each step pushing x ki closer to the average a .

Original: In summary, the more local GD steps are taken, the closer the local models get to the pure local models; and the more averaging steps are taken, the closer the local models get to their average value.
Revised: In summary, the more local GD steps are taken, the closer the local models get to the pure local models; and the more averaging steps are taken, the closer the local models get to their average value.

Original: The relative number of local GD vs. averaging steps is controlled by the parameter p : the expected # of local GD steps is 1 p , and the expected number of consecutive aggregation steps is 11 − p .
Revised: The relative number of local GD vs. averaging steps is controlled by the parameter p : the expected # of local GD steps is 1 p , and the expected number of consecutive aggregation steps is 11 − p .

Original: 4. C ONVERGENCE THEORY
Revised: 4. C ONVERGENCE THEORY

Original: We now present our convergence result for L2GD.
Revised: We now present our convergence result for L2GD.

Original: Theorem 4.2 Let Assumption 3.1 hold.
Revised: Theorem 4.2 Let Assumption 3.1 hold.

Original: If α ≤ 12 L , then
Revised: If α ≤ 12 L , then

Original: E
Revised: E

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: 2 + 2 nασ
Revised: 2 + 2 nασ

Original: [Equation]
Revised: [Equation]

Original: µ , where L := 1 n max (cid:110)
Revised: µ , where L := 1 n max (cid:110)

Original: L 1 − p , λp (cid:111) and
Revised: L 1 − p , λp (cid:111) and

Original: σ 2 := 1 n 2 (cid:80) n i =1 (cid:16) 11 − p (cid:107)∇ f i ( x i ( λ ))
Revised: σ 2 := 1 n 2 (cid:80) n i =1 (cid:16) 11 − p (cid:107)∇ f i ( x i ( λ ))

Original: (cid:107) 

Revised: (cid:107)

Original: + λ 2 p (cid:107) x
Revised: + λ 2 p (cid:107) x

Original: i ( λ ) − x ( λ ) (cid:107) 2 (cid:17) .
Revised: i ( λ ) − x ( λ ) (cid:107) 2 (cid:17) .

Original: Let us ﬁnd the parameters p , α which lead to the fastest rate, to push the error within (cid:16)
Revised: Let us ﬁnd the parameters p , α which lead to the fastest rate, to push the error within (cid:16)

Original: O ( ε ) + 2 nασ 2 µ (cid:17) -neighborhood of the optimum 5 , i.e., to achieve
Revised: O ( ε ) + 2 nασ 2 µ (cid:17) -neighborhood of the optimum 5 , i.e., to achieve

Original: [Equation]
Revised: [Equation]

Original: Corollary 4.

Revised: Corollary 4.

Original: The value p (cid:63) = λL + λ minimizes both the number of iterations and the expected number of communications for achieving (10) .
Revised: The value p (cid:63) = λL + λ minimizes both the number of iterations and the expected number of communications for achieving (10) .

Original: In particular, the optimal number of iterations is 2 L + λµ log 1 ε , and the optimal expected number of communications is 2 λ λ + L Lµ log 1 ε .
Revised: In particular, the optimal number of iterations is 2 L + λµ log 1 ε , and the optimal expected number of communications is 2 λ λ + L Lµ log 1 ε .

Original: If we choose p = p (cid:63) , then αλnp = 12 , and the aggregation rule (9) in Algorithm 1 becomes
Revised: If we choose p = p (cid:63) , then αλnp = 12 , and the aggregation rule (9) in Algorithm 1 becomes

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 2 µ ) -neighborhood from Thm.
Revised: 2 µ ) -neighborhood from Thm.

Original: In that setting, our goal will be to achieve E (cid:104)(cid:13)(cid:13) x
Revised: In that setting, our goal will be to achieve E (cid:104)(cid:13)(cid:13) x

Original: k − x ( λ ) (cid:13)(cid:13) 2 (cid:105) ≤ ε (cid:13)(cid:13) x 0 − x ( λ ) (cid:13)(cid:13) 2 .
Revised: k − x ( λ ) (cid:13)(cid:13) 2 (cid:105) ≤ ε (cid:13)(cid:13) x 0 − x ( λ ) (cid:13)(cid:13) 2 .

Original: [Figure]
Revised: [Figure]

Original: 0.50 p 104×106×102×

Revised: 0.50 p 104×106×102×

Original: C o mm un i c a t i o n r o un d s f o r s o l u t i o n
Revised: C o mm un i c a t i o n r o un d s f o r s o l u t i o n

Original: [Figure]
Revised: [Figure]

Original: while the local GD step (8) becomes x k +1 i = x ki − 12 L ∇ f i ( x ki ) .
Revised: while the local GD step (8) becomes x k +1 i = x ki − 12 L ∇ f i ( x ki ) .

Original: Notice that while our method does not support full averaging as that is too unstable, (11) suggests that one should take a large step towards averaging.
Revised: Notice that while our method does not support full averaging as that is too unstable, (11) suggests that one should take a large step towards averaging.

Original: As λ get smaller, the solution to the optimization problem (2) will increasingly favour pure local models, i.e., x i ( λ )
Revised: As λ get smaller, the solution to the optimization problem (2) will increasingly favour pure local models, i.e., x i ( λ )

Original: → x i (0) :
Revised: → x i (0) :

Original: =
Revised: =

Original: arg min f i for all i as λ → 0 .
Revised: arg min f i for all i as λ → 0 .

Original: Pure local models can be computed without any communication whatsoever and Cor.
Revised: Pure local models can be computed without any communication whatsoever and Cor.

Original: 4.3 conﬁrms this intuition: the optimal number of communication round decreases to zero as λ → 0 .
Revised: 4.3 conﬁrms this intuition: the optimal number of communication round decreases to zero as λ → 0 .

Original: On the other hand, as λ → ∞ , the optimal number of communication rounds converges to 2 Lµ log 1 ε , which recovers the performance of GD for ﬁnding the globally optimal model (see Fig. 1).
Revised: On the other hand, as λ → ∞ , the optimal number of communication rounds converges to 2 Lµ log 1 ε , which recovers the performance of GD for ﬁnding the globally optimal model (see Fig. 1).

Original: In summary, we recover the communication efﬁciency of GD for ﬁnding the globally optimal model as λ → ∞ (ignoring the ( 2 nασ 2 µ ) -neighborhood).
Revised: In summary, we recover the communication efﬁciency of GD for ﬁnding the globally optimal model as λ → ∞ (ignoring the ( 2 nασ 2 µ ) -neighborhood).

Original: However, for other values of λ , the communication complexity of L2GD is better and decreases to 0 as λ → 0 .
Revised: However, for other values of λ , the communication complexity of L2GD is better and decreases to 0 as λ → 0 .

Original: Hence, our communication complexity result interpolates between the communication complexity of GD for ﬁnding the global model and the zero communication complexity for ﬁnding the pure local models.
Revised: Hence, our communication complexity result interpolates between the communication complexity of GD for ﬁnding the global model and the zero communication complexity for ﬁnding the pure local models.

Original:  L OOPLESS L OCAL SGD WITH V ARIANCE R EDUCTION
Revised:  L OOPLESS L OCAL SGD WITH V ARIANCE R EDUCTION

Original: As we have seen in Sec. 4.2, L2GD is a speciﬁc instance of SGD, thus only converges linearly to a neighborhood of the optimum.
Revised: As we have seen in Sec. 4.2, L2GD is a speciﬁc instance of SGD, thus only converges linearly to a neighborhood of the optimum.

Original: In this section, we resolve the mentioned issue by incorporating control variates to the stochastic gradient (Johnson & Zhang, 2013b; Defazio et al., 2014).
Revised: In this section, we resolve the mentioned issue by incorporating control variates to the stochastic gradient (Johnson & Zhang, 2013b; Defazio et al., 2014).

Original: We go further: we assume that each local objective has a ﬁnite-sum structure and propose an algorithm,
Revised: We go further: we assume that each local objective has a ﬁnite-sum structure and propose an algorithm,

Original: L2SGD+, which takes local stochastic gradient steps, while maintaining (global) linear convergence rate.
Revised: L2SGD+, which takes local stochastic gradient steps, while maintaining (global) linear convergence rate.

Original: As a consequence, L2SGD+ is the ﬁrst local SGD with linear convergence.
Revised: As a consequence, L2SGD+ is the ﬁrst local SGD with linear convergence.

Original: 6 For convenience, we present variance reduced local GD (i.e., no local subsampling) in the Appendix.
Revised: 6 For convenience, we present variance reduced local GD (i.e., no local subsampling) in the Appendix.

Original: Assumption 5.1 Assume that f i has a ﬁnite-sum structure: f i ( x i )
Revised: Assumption 5.1 Assume that f i has a ﬁnite-sum structure: f i ( x i )

Original: = 1 m (cid:80) m j =1 f (cid:48)
Revised: = 1 m (cid:80) m j =1 f (cid:48)

Original: i,j ( x i ) .
Revised: i,j ( x i ) .

Original: Let f (cid:48)
Revised: Let f (cid:48)

Original: i,j be convex, L (cid:48) -smooth
Revised: i,j be convex, L (cid:48) -smooth

Original: while f i is µ -strongly convex (for each 1 ≤ j ≤ m, 1 ≤ i ≤ n ).
Revised: while f i is µ -strongly convex (for each 1 ≤ j ≤ m, 1 ≤ i ≤ n ).

Original: C ONVERGENCE THEORY
Revised: C ONVERGENCE THEORY

Original: We are now ready to present a convergence rate of L2SGD+ (the algorithm, along with the efﬁcient implementation is presented in Appendix C.4).
Revised: We are now ready to present a convergence rate of L2SGD+ (the algorithm, along with the efﬁcient implementation is presented in Appendix C.4).

Original: Theorem 5.1 Let Assumption 5.1 hold and choose α = n min (cid:110) (1 − p ) 

Revised: Theorem 5.1 Let Assumption 5.1 hold and choose α = n min (cid:110) (1 − p )

Original: L (cid:48)
Revised: L (cid:48)

Original: + µm , p 4 λ + µ (cid:111) .
Revised: + µm , p 4 λ + µ (cid:111) .

Original: Then the iteration complexity of Algorithm 3 is max (cid:110) 

Revised: Then the iteration complexity of Algorithm 3 is max (cid:110)

Original: L (cid:48)
Revised: L (cid:48)

Original: + µm (1 − p ) µ , 4 λ + µ pµ (cid:111)
Revised: + µm (1 − p ) µ , 4 λ + µ pµ (cid:111)

Original: [Equation]
Revised: [Equation]

Original: Next, we ﬁnd the value of p that yields both the best iteration and communication complexity.
Revised: Next, we ﬁnd the value of p that yields both the best iteration and communication complexity.

Original: Corollary 5.2 Both communication and iteration complexity of L2SGD+ are minimized for p = 4 λ +
Revised: Corollary 5.2 Both communication and iteration complexity of L2SGD+ are minimized for p = 4 λ +

Original: [Equation]
Revised: [Equation]

Original: 4 λ +4 L (cid:48)
Revised: 4 λ +4 L (cid:48)

Original: +( m +1) µ .
Revised: +( m +1) µ .

Original: The resulting iteration complexity is (cid:16) 4 λµ + 4 L (cid:48) µ + m + 1 (cid:17) log 1 ε , while the communication complexity is
Revised: The resulting iteration complexity is (cid:16) 4 λµ + 4 L (cid:48) µ + m + 1 (cid:17) log 1 ε , while the communication complexity is

Original: [Equation]
Revised: [Equation]

Original: Note that with λ → ∞ , the communication complexity of L2SGD+ tends to (cid:16) 4 L (cid:48) µ + m (cid:17) log 1 ε , which is communication complexity of minibatch SAGA to ﬁnd the globally optimal model (Hanzely & Richt´arik, 2019).
Revised: Note that with λ → ∞ , the communication complexity of L2SGD+ tends to (cid:16) 4 L (cid:48) µ + m (cid:17) log 1 ε , which is communication complexity of minibatch SAGA to ﬁnd the globally optimal model (Hanzely & Richt´arik, 2019).

Original: On the other hand, in the pure local setting ( λ = 0 ), the communication complexity becomes log 1 (cid:15) – this is because the Lyapunov function involves a term that measures the distance of local models, which requires communication to be estimated.
Revised: On the other hand, in the pure local setting ( λ = 0 ), the communication complexity becomes log 1 (cid:15) – this is because the Lyapunov function involves a term that measures the distance of local models, which requires communication to be estimated.

Original: Remark 5.3 L2SGD+ is the simplest local SGD method with variance reduction.
Revised: Remark 5.3 L2SGD+ is the simplest local SGD method with variance reduction.

Original: In the Appendix, we present L2SGD++ which allows for 1) an arbitrary number of data points per client and arbitrary local subsampling, 2) partial participation of clients, and 3) local SVRG-like updates of control variates (thus better memory).
Revised: In the Appendix, we present L2SGD++ which allows for 1) an arbitrary number of data points per client and arbitrary local subsampling, 2) partial participation of clients, and 3) local SVRG-like updates of control variates (thus better memory).

Original: Lastly, L2SGD++ exploits the complex smoothness structure of the local objectives, resulting in tighter rates.
Revised: Lastly, L2SGD++ exploits the complex smoothness structure of the local objectives, resulting in tighter rates.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we numerically verify the theoretical claims from this paper.
Revised: In this section, we numerically verify the theoretical claims from this paper.

Original: We only present a single experiment here, all remaining ones along with the missing details about the setup are in the
Revised: We only present a single experiment here, all remaining ones along with the missing details about the setup are in the

Original: Appendix.
Revised: Appendix.

Original: In particular, the Appendix includes two more experiments.
Revised: In particular, the Appendix includes two more experiments.

Original: The ﬁrst one studies how p (communication) inﬂuences the convergence of L2SGD+.
Revised: The ﬁrst one studies how p (communication) inﬂuences the convergence of L2SGD+.

Original: The second experiment aims to examine the effect of parameter λ on the convergence rate of L2SGD+.
Revised: The second experiment aims to examine the effect of parameter λ on the convergence rate of L2SGD+.

Original: We consider logistic regression problem with LibSVM data (Chang & Lin, 2011).
Revised: We consider logistic regression problem with LibSVM data (Chang & Lin, 2011).

Original: The data were normalized so that f (cid:48)
Revised: The data were normalized so that f (cid:48)

Original: i,j is 1-smooth for each j , while the local objectives are 10 − 4 -strongly convex.
Revised: i,j is 1-smooth for each j , while the local objectives are 10 − 4 -strongly convex.

Original: In order to cover a range of possible scenarios, we have chosen a different number of clients for each dataset (see the Appendix).
Revised: In order to cover a range of possible scenarios, we have chosen a different number of clients for each dataset (see the Appendix).

Original: Lastly, the stepsize was always chosen according to Thm. 5.1.
Revised: Lastly, the stepsize was always chosen according to Thm. 5.1.

Original: We compare three different methods: L2SGD+, L2GD with local subsampling (L2SGD in the Appendix), and L2GD with local subsampling and control variates constructed for ψ only (L2SGD2 in the Appendix; similar to (Liang et al., 2019)).
Revised: We compare three different methods: L2SGD+, L2GD with local subsampling (L2SGD in the Appendix), and L2GD with local subsampling and control variates constructed for ψ only (L2SGD2 in the Appendix; similar to (Liang et al., 2019)).

Original: We expect L2SGD+ to converge to the global optimum linearly, while both L2SGD and L2SGD2 to converge to certain neighborhood.
Revised: We expect L2SGD+ to converge to the global optimum linearly, while both L2SGD and L2SGD2 to converge to certain neighborhood.

Original: Each method is applied to two objectives constructed by a different split of the data among the devices.
Revised: Each method is applied to two objectives constructed by a different split of the data among the devices.

Original: For the homogeneous split, we randomly reshufﬂe the data.
Revised: For the homogeneous split, we randomly reshufﬂe the data.

Original: For heterogeneous split, we ﬁrst sort the data based on the labels and then construct the local objectives according to the current order.
Revised: For heterogeneous split, we ﬁrst sort the data based on the labels and then construct the local objectives according to the current order.

Original: [Figure]
Revised: [Figure]

Original: Fig.
Revised: Fig.

Original: 3 demonstrates the importance of variance reduction – it ensures a fast global convergence of
Revised: 3 demonstrates the importance of variance reduction – it ensures a fast global convergence of

Original: L2SGD+, while the neighborhood is slightly smaller for L2SGD2 compared to L2SGD.
Revised: L2SGD+, while the neighborhood is slightly smaller for L2SGD2 compared to L2SGD.

Original: As predicted, data heterogeneity does not affect the convergence speed of the proposed methods
Revised: As predicted, data heterogeneity does not affect the convergence speed of the proposed methods

Original: [Figure]
Revised: [Figure]

Original: D Missing Lemmas and Proofs
Revised: D Missing Lemmas and Proofs

Original: D.1 Gradient and Hessian of ψ
Revised: D.1 Gradient and Hessian of ψ

Original: D.2 Proof of Theorem
Revised: D.2 Proof of Theorem

Original: D.
Revised: D.

Original: 3 Proof
Revised: 3 Proof

Original: of
Revised: of

Original: Theorem
Revised: Theorem

Original: D.4 Proof of Theorem 3.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: D.4 Proof of Theorem 3.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.

Revised: D.

Original: Proof of Theorem 4.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Proof of Theorem 4.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.6 Proof of Lemma D.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: D.6 Proof of Lemma D.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.7 Proof of Corollary 4.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: D.7 Proof of Corollary 4.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.8 Proof of Corollary 5.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: D.8 Proof of Corollary 5.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.9 Proof of Theorems 5.1, C.6, and C.7 . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: D.9 Proof of Theorems 5.1, C.6, and C.7 . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.9.1 GJS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: D.9.1 GJS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.9.2 Variance reduced local SGD as special case of GJS . . . . . . . . . . . . .
Revised: D.9.2 Variance reduced local SGD as special case of GJS . . . . . . . . . . . . .

Original: D.9.3 Proof of Theorem C.6 and Theorem C.7 . . . . . . . . . . . . . . . . . . .
Revised: D.9.3 Proof of Theorem C.6 and Theorem C.7 . . . . . . . . . . . . . . . . . . .

Original: D.9.4 Proof of Theorem 5.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: D.9.4 Proof of Theorem 5.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A P OSSIBLE E XTENSIONS
Revised: A P OSSIBLE E XTENSIONS

Original: Our analysis of L2GD can be extended to cover smooth convex and non-convex loss functions
Revised: Our analysis of L2GD can be extended to cover smooth convex and non-convex loss functions

Original: f i (we do not explore these directions).
Revised: f i (we do not explore these directions).

Original: Further, our methods can be extended to a decentralized regime where the devices correspond to devices of a connected network, and communication is allowed along the edges of the graph only.
Revised: Further, our methods can be extended to a decentralized regime where the devices correspond to devices of a connected network, and communication is allowed along the edges of the graph only.

Original: This can be achieved by introducing an additional randomization over the penalty ψ .
Revised: This can be achieved by introducing an additional randomization over the penalty ψ .

Original: Further, our approach can be accelerated in the sense of Nesterov (Nesterov, 2004) by adapting the a variant of Katyusha (Allen-Zhu, 2017; Qian et al., 2019a) to our setting, thus further reducing the number of communication rounds.
Revised: Further, our approach can be accelerated in the sense of Nesterov (Nesterov, 2004) by adapting the a variant of Katyusha (Allen-Zhu, 2017; Qian et al., 2019a) to our setting, thus further reducing the number of communication rounds.

Original: B E XPERIMENTAL S ETUP AND F URTHER E XPERIMENTS
Revised: B E XPERIMENTAL S ETUP AND F URTHER E XPERIMENTS

Original: In all experiments in this paper, we consider a simple binary classiﬁcation model – logistic regression.
Revised: In all experiments in this paper, we consider a simple binary classiﬁcation model – logistic regression.

Original: In particular, suppose that device i owns data matrix A i ∈ R m × d along with corresponding labels b i ∈ {− 1 , 1 } m .
Revised: In particular, suppose that device i owns data matrix A i ∈ R m × d along with corresponding labels b i ∈ {− 1 , 1 } m .

Original: The local objective for client i is then given as follows
Revised: The local objective for client i is then given as follows

Original: [Equation]
Revised: [Equation]

Original: where f (cid:48)
Revised: where f (cid:48)

Original: im + j ( x ) = log (1 + exp (( A i ) j, : x · b i )) .
Revised: im + j ( x ) = log (1 + exp (( A i ) j, : x · b i )) .

Original: The rows of data matrix A were normalized to have length 4 so that each f (cid:48)
Revised: The rows of data matrix A were normalized to have length 4 so that each f (cid:48)

Original: i,j is 1 -smooth for each j .
Revised: i,j is 1 -smooth for each j .

Original: At the same time, the local objective on each device is 10 − 4 strongly convex.
Revised: At the same time, the local objective on each device is 10 − 4 strongly convex.

Original: Next, datasets are from LibSVM (Chang & Lin, 2011).
Revised: Next, datasets are from LibSVM (Chang & Lin, 2011).

Original: In each case, we consider the simplest locally stochastic algorithm.
Revised: In each case, we consider the simplest locally stochastic algorithm.

Original: In particular, each dataset is evenly split among the clients, while the local stochastic method samples a single data point each iteration.
Revised: In particular, each dataset is evenly split among the clients, while the local stochastic method samples a single data point each iteration.

Original: We have chosen a different number of clients for each dataset – so that we cover different possible scenarios.
Revised: We have chosen a different number of clients for each dataset – so that we cover different possible scenarios.

Original: See Table 1 for details (it also includes sizes of the datasets).
Revised: See Table 1 for details (it also includes sizes of the datasets).

Original: Lastly, the stepsize was always chosen according to Thm. 5.1.
Revised: Lastly, the stepsize was always chosen according to Thm. 5.1.

Original: [Table]
Revised: [Table]

Original: B. C OMPARISON OF THE METHODS
Revised: B. C OMPARISON OF THE METHODS

Original: In our ﬁrst experiment
Revised: In our ﬁrst experiment

Original: [Figure]
Revised: [Figure]

Original: we verify two phenomena:
Revised: we verify two phenomena:

Original: • The effect of variance reduction on the convergence speed of local methods.
Revised: • The effect of variance reduction on the convergence speed of local methods.

Original: We compare 3 different methods: local SGD with full variance reduction (Algorithm 3), shifted local SGD (Algorithm 7) and local SGD (Algorithm 6).
Revised: We compare 3 different methods: local SGD with full variance reduction (Algorithm 3), shifted local SGD (Algorithm 7) and local SGD (Algorithm 6).

Original: Our theory predicts that a fully variance reduced algorithm converges to the global optimum linearly, while both shifted local SGD and local SGD converge to a neighborhood of the optimum.
Revised: Our theory predicts that a fully variance reduced algorithm converges to the global optimum linearly, while both shifted local SGD and local SGD converge to a neighborhood of the optimum.

Original: At the same time, the neighborhood should be smaller for shifted local SGD.
Revised: At the same time, the neighborhood should be smaller for shifted local SGD.

Original: • The claim that heterogeneity of the data does not inﬂuence the convergence rate.
Revised: • The claim that heterogeneity of the data does not inﬂuence the convergence rate.

Original: We consider two splits of the data heterogeneous and homogeneous.
Revised: We consider two splits of the data heterogeneous and homogeneous.

Original: For the homogeneous split, we ﬁrst randomly reshufﬂe the data and then construct the local objectives according to the current order (i.e., the ﬁrst client owns the ﬁrst m indices, etc.).
Revised: For the homogeneous split, we ﬁrst randomly reshufﬂe the data and then construct the local objectives according to the current order (i.e., the ﬁrst client owns the ﬁrst m indices, etc.).

Original: For heterogeneous split, we ﬁrst sort
Revised: For heterogeneous split, we ﬁrst sort

Original: the data based on the labels and then construct the local objectives accordingly (thus achieving the worst-case heterogeneity).
Revised: the data based on the labels and then construct the local objectives accordingly (thus achieving the worst-case heterogeneity).

Original: Note that the overall objective to solve is different in homogeneous and heterogeneous case – we thus plot relative suboptimality of the objective (i.e., F ( x k ) − F ( x (cid:63) )
Revised: Note that the overall objective to solve is different in homogeneous and heterogeneous case – we thus plot relative suboptimality of the objective (i.e., F ( x k ) − F ( x (cid:63) )

Original: F ( x 0 ) − F ( x (cid:63) ) ) to directly compare the convergence speed.
Revised: F ( x 0 ) − F ( x (cid:63) ) ) to directly compare the convergence speed.

Original: In each experiment
Revised: In each experiment

Original: [Figure]
Revised: [Figure]

Original: we choose p = 0 .
Revised: we choose p = 0 .

Original: 1 and λ = 19 – such choice mean that p is very close to optimal.
Revised: 1 and λ = 19 – such choice mean that p is very close to optimal.

Original: The other parameters (i.e., number of clients) are provided in Table 1.
Revised: The other parameters (i.e., number of clients) are provided in Table 1.

Original: Fig.
Revised: Fig.

Original: 4 presents the result.
Revised: 4 presents the result.

Original: [Figure]
Revised: [Figure]

Original: As expected
Revised: As expected

Original: [Figure]
Revised: [Figure]

Original: Figure 4 clearly demonstrates the following:
Revised: Figure 4 clearly demonstrates the following:

Original: • Full variance reduction always converges to the global optima, methods with partial variance reduction only converge to a neighborhood of the optimum.
Revised: • Full variance reduction always converges to the global optima, methods with partial variance reduction only converge to a neighborhood of the optimum.

Original: • Partial variance reduction (i.e., shifting the local SGD) is better than not using control variates at all.
Revised: • Partial variance reduction (i.e., shifting the local SGD) is better than not using control variates at all.

Original: Although the improvement in the performance is rather negligible.
Revised: Although the improvement in the performance is rather negligible.

Original: • Data heterogeneity does not affect the convergence speed of the proposed methods.
Revised: • Data heterogeneity does not affect the convergence speed of the proposed methods.

Original: Therefore, unlike standard local SGD, mixing the local and global models does not suffer the problems with heterogeneity.
Revised: Therefore, unlike standard local SGD, mixing the local and global models does not suffer the problems with heterogeneity.

Original: B. E FFECT OF p
Revised: B. E FFECT OF p

Original: In the second experiment, we study the effect of p on the convergence rate of variance reduced
Revised: In the second experiment, we study the effect of p on the convergence rate of variance reduced

Original: local
Revised: local

Original: SGD.
Revised: SGD.

Original: Note that p immediately inﬂuences the number of communication rounds – on average, the clients take ( p − 1 − 1) local steps in between two consecutive rounds of communication (aggregation).
Revised: Note that p immediately inﬂuences the number of communication rounds – on average, the clients take ( p − 1 − 1) local steps in between two consecutive rounds of communication (aggregation).

Original: In Section 5, we argue that, it is optimal (in terms of the convergence rate) to choose p of order p (cid:63) :=
Revised: In Section 5, we argue that, it is optimal (in terms of the convergence rate) to choose p of order p (cid:63) :=

Original: [Equation]
Revised: [Equation]

Original: Figure 5 compares p = p (cid:63) against other values of p and conﬁrms its optimality (in terms of optimizing the convergence rate).
Revised: Figure 5 compares p = p (cid:63) against other values of p and conﬁrms its optimality (in terms of optimizing the convergence rate).

Original: While the slower convergence of Algorithm 3 with p < p (cid:63) is expected (i.e., communicating more frequently yields a faster convergence), slower convergence for p > p (cid:63) is rather surprising; in fact, it means that communicating less frequently yields faster convergence.
Revised: While the slower convergence of Algorithm 3 with p < p (cid:63) is expected (i.e., communicating more frequently yields a faster convergence), slower convergence for p > p (cid:63) is rather surprising; in fact, it means that communicating less frequently yields faster convergence.

Original: This effect takes place due to the speciﬁc structure of problem (12); it would be lost when enforcing x 1 = · · · = x n (corresponding to λ = ∞ ).
Revised: This effect takes place due to the speciﬁc structure of problem (12); it would be lost when enforcing x 1 = · · · = x n (corresponding to λ = ∞ ).

Original: B. E FFECT OF λ
Revised: B. E FFECT OF λ

Original: In this experiment we study how different values of λ inﬂuence the convergence rate of Algorithm 3, given that everything else (i.e., p ) is ﬁxed.
Revised: In this experiment we study how different values of λ inﬂuence the convergence rate of Algorithm 3, given that everything else (i.e., p ) is ﬁxed.

Original: Note that for each value of λ we get a different instance of problem (12); thus the optimal solution is different as well.
Revised: Note that for each value of λ we get a different instance of problem (12); thus the optimal solution is different as well.

Original: Therefore, in order to make a fair comparison between convergence speeds, we plot the relative suboptimality (i.e., F ( x k ) − F ( x (cid:63) )
Revised: Therefore, in order to make a fair comparison between convergence speeds, we plot the relative suboptimality (i.e., F ( x k ) − F ( x (cid:63) )

Original: F ( x 0 ) − F ( x (cid:63) ) ) against the data passes.
Revised: F ( x 0 ) − F ( x (cid:63) ) ) against the data passes.

Original: Figure 6 presents the results.
Revised: Figure 6 presents the results.

Original: The complexity of Algorithm 3 is 7 O (cid:16) L (cid:48) (1 − p ) µ (cid:17) log 1 ε as soon as λ < λ (cid:63) :
Revised: The complexity of Algorithm 3 is 7 O (cid:16) L (cid:48) (1 − p ) µ (cid:17) log 1 ε as soon as λ < λ (cid:63) :

Original: =
Revised: =

Original: Lp (1 − p ) ; otherwise the complexity is O (cid:16) λpµ (cid:17) log 1 ε .
Revised: Lp (1 − p ) ; otherwise the complexity is O (cid:16) λpµ (cid:17) log 1 ε .

Original: This perfectly consistent with what Figure 6 shows – the choice λ < λ (cid:63) resulted in comparable convergence speed than λ
Revised: This perfectly consistent with what Figure 6 shows – the choice λ < λ (cid:63) resulted in comparable convergence speed than λ

Original: = λ (cid:63) ; while the choice λ > λ (cid:63) yields noticeably worse rate than λ
Revised: = λ (cid:63) ; while the choice λ > λ (cid:63) yields noticeably worse rate than λ

Original: = λ (cid:63) .
Revised: = λ (cid:63) .

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: C R EMAINING A LGORITHMS C. U NDERSTANDING COMMUNICATION OF L2GD
Revised: C R EMAINING A LGORITHMS C. U NDERSTANDING COMMUNICATION OF L2GD

Original: Example C.

Revised: Example C.

Original: In order to better understand when communication takes place in Algorithm 1, consider the following possible sequence of coin tosses: 0 , 0 , 1 , 0 , 1 , 1 , 1 , 0 .
Revised: In order to better understand when communication takes place in Algorithm 1, consider the following possible sequence of coin tosses: 0 , 0 , 1 , 0 , 1 , 1 , 1 , 0 .

Original: The ﬁrst two coin tosses lead to two local GD steps (8) on all devices.
Revised: The ﬁrst two coin tosses lead to two local GD steps (8) on all devices.

Original: The third coin toss lands 1 , at which point all local models x ki are communicated to the master, averaged to form ¯ x k , and the step (9) towards averaging is taken.
Revised: The third coin toss lands 1 , at which point all local models x ki are communicated to the master, averaged to form ¯ x k , and the step (9) towards averaging is taken.

Original: The fourth coin toss is 0 , and at this point, the master communicates the updated local models back to the devices, which subsequently perform a single local GD step (8) .
Revised: The fourth coin toss is 0 , and at this point, the master communicates the updated local models back to the devices, which subsequently perform a single local GD step (8) .

Original: Then come three consecutive coin tosses landing 1 , which means that the local models are again communicated to the master, which performs three averaging steps (9) .
Revised: Then come three consecutive coin tosses landing 1 , which means that the local models are again communicated to the master, which performs three averaging steps (9) .

Original: Finally, the eighth coin toss lands 0 , which makes the master send the updated local models back to the devices, which subsequently perform a single local GD step.
Revised: Finally, the eighth coin toss lands 0 , which makes the master send the updated local models back to the devices, which subsequently perform a single local GD step.

Original: This example illustrates that communication needs to take place whenever two consecutive coin tosses land a different value.
Revised: This example illustrates that communication needs to take place whenever two consecutive coin tosses land a different value.

Original: If 0 is followed by a 1 , all devices communicate to the master, and ifis followed by a 0 , the master communicates back to the devices.
Revised: If 0 is followed by a 1 , all devices communicate to the master, and ifis followed by a 0 , the master communicates back to the devices.

Original: It is standard to count each pair of communications, Device → Master and the subsequent Master → Device, as a single communication round.
Revised: It is standard to count each pair of communications, Device → Master and the subsequent Master → Device, as a single communication round.

Original: Lemma C.

Revised: Lemma C.

Original: The expected number of communication rounds in k iterations of L2GD is p (1 − p ) k .
Revised: The expected number of communication rounds in k iterations of L2GD is p (1 − p ) k .

Original: C. L2GD AND FULL AVERAGING
Revised: C. L2GD AND FULL AVERAGING

Original: Is a setup such that conditions of Thm.
Revised: Is a setup such that conditions of Thm.

Original: 4.2 are satisﬁed and the aggregation update (9) is identical to full averaging?
Revised: 4.2 are satisﬁed and the aggregation update (9) is identical to full averaging?

Original: This is equivalent requiring 0 < p < 1 such that αλ = np .
Revised: This is equivalent requiring 0 < p < 1 such that αλ = np .

Original: However, we have αλ ≤ λ 2 L ≤ np , which means that full averaging is not supported by our theory.
Revised: However, we have αλ ≤ λ 2 L ≤ np , which means that full averaging is not supported by our theory.

Original: C. L OCAL GD WITH VARIANCE REDUCTION
Revised: C. L OCAL GD WITH VARIANCE REDUCTION

Original: In this section, we present variance reduced local gradient descent with partial aggregation.
Revised: In this section, we present variance reduced local gradient descent with partial aggregation.

Original: In particular, the proposed algorithm (Algorithm 2) incorporates control variates to Algorithm 1.
Revised: In particular, the proposed algorithm (Algorithm 2) incorporates control variates to Algorithm 1.

Original: Therefore, the proposed method can be seen as a special case of Algorithm 3 with m = 1 .
Revised: Therefore, the proposed method can be seen as a special case of Algorithm 3 with m = 1 .

Original: We thus only present it for pedagogical purposes, as it might shed additional insights into our approach.
Revised: We thus only present it for pedagogical purposes, as it might shed additional insights into our approach.

Original: In particular, the update rule of proposed method will be x k +

Revised: In particular, the update rule of proposed method will be x k +

Original: = x
Revised: = x

Original: k − αg k where
Revised: k − αg k where

Original: [Equation]
Revised: [Equation]

Original: for some control variates vectors J k , Ψ k ∈ R nd .
Revised: for some control variates vectors J k , Ψ k ∈ R nd .

Original: A quick check gives
Revised: A quick check gives

Original: [Equation]
Revised: [Equation]

Original: thus the direction we are taking is unbiased regardless of the value of control variates J k , Ψ k .
Revised: thus the direction we are taking is unbiased regardless of the value of control variates J k , Ψ k .

Original: The goal is to make control variates J k , Ψ k correlated 8 with n ∇ f ( x k ) and nλ ∇ ψ ( x k ) .
Revised: The goal is to make control variates J k , Ψ k correlated 8 with n ∇ f ( x k ) and nλ ∇ ψ ( x k ) .

Original: One possible solution to the problem is for J k , Ψ k
Revised: One possible solution to the problem is for J k , Ψ k

Original: to track most recently observed values of n ∇ f ( · ) and nλ ∇ ψ ( · ) , which corresponds to the following update rule
Revised: to track most recently observed values of n ∇ f ( · ) and nλ ∇ ψ ( · ) , which corresponds to the following update rule

Original: [Equation]
Revised: [Equation]

Original: A speciﬁc, distributed implementation of the described method is presented as Algorithm 2.
Revised: A speciﬁc, distributed implementation of the described method is presented as Algorithm 2.

Original: The only communication between the devices takes place when the average model ¯ x k is being computed (with probability p ), which is analogous to standard local SGD.
Revised: The only communication between the devices takes place when the average model ¯ x k is being computed (with probability p ), which is analogous to standard local SGD.

Original: Therefore we aim to set p rather small.
Revised: Therefore we aim to set p rather small.

Original: Note that Algorithm 2 is a particular special case of SAGA with importance sampling
Revised: Note that Algorithm 2 is a particular special case of SAGA with importance sampling

Original: (Qian et al., 2019b); thus, we obtain convergence rate of the method for free.
Revised: (Qian et al., 2019b); thus, we obtain convergence rate of the method for free.

Original: We state it as Thm.
Revised: We state it as Thm.

Original: C.3.
Revised: C.3.

Original: Input: x 01 = · · · = x 0 n ∈ R d , stepsize α , probability p
Revised: Input: x 01 = · · · = x 0 n ∈ R d , stepsize α , probability p

Original: J 01 = · · · = J 0 n = Ψ 01 = · · · = Ψ 0 n = 0 ∈ R d for k = 0 , 1 , . . .
Revised: J 01 = · · · = J 0 n = Ψ 01 = · · · = Ψ 0 n = 0 ∈ R d for k = 0 , 1 , . . .

Original: do
Revised: do

Original: ξ = 1 with probability p and 0 with probability 1 − p if ξ then
Revised: ξ = 1 with probability p and 0 with probability 1 − p if ξ then

Original: All Devices i = 1 , . . . , n :
Revised: All Devices i = 1 , . . . , n :

Original: Compute ∇ f i ( x ki )
Revised: Compute ∇ f i ( x ki )

Original: [Equation]
Revised: [Equation]

Original: Set
Revised: Set

Original: [Equation]
Revised: [Equation]

Original: ki else
Revised: ki else

Original: Master computes the average ¯ x k = 1 n (cid:80) ni
Revised: Master computes the average ¯ x k = 1 n (cid:80) ni

Original: =1 x ki
Revised: =1 x ki

Original: Master does for all i = 1 , . . .
Revised: Master does for all i = 1 , . . .

Original: , n :
Revised: , n :

Original: Set x k +

Revised: Set x k +

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: Set Ψ k
Revised: Set Ψ k

Original: +1 i
Revised: +1 i

Original: [Equation]
Revised: [Equation]

Original: ki
Revised: ki

Original: [Equation]
Revised: [Equation]

Original: +1 i =
Revised: +1 i =

Original: J ki end if end for
Revised: J ki end if end for

Original: Theorem C.3 Let Assumption 3.1 hold.
Revised: Theorem C.3 Let Assumption 3.1 hold.

Original: Set α
Revised: Set α

Original: = n min (cid:16) (1 − p ) 4 L + µ ,
Revised: = n min (cid:16) (1 − p ) 4 L + µ ,

Original: [Equation]
Revised: [Equation]

Original: + µ (cid:17) .
Revised: + µ (cid:17) .

Original: Then, iteration complexity of Algorithm 2 is
Revised: Then, iteration complexity of Algorithm 2 is

Original: [Equation]
Revised: [Equation]

Original: Proof: Clearly,
Revised: Proof: Clearly,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  .
Revised:  .

Original: Note that ψ is 2 λn smooth and f is 2 Ln smooth.
Revised: Note that ψ is 2 λn smooth and f is 2 Ln smooth.

Original: At the same time, F is µn strongly convex.
Revised: At the same time, F is µn strongly convex.

Original: Using convergence theorem of SAGA with importance sampling from (Qian et al., 2019b; Gazagnadou et al., 2019), we get
Revised: Using convergence theorem of SAGA with importance sampling from (Qian et al., 2019b; Gazagnadou et al., 2019), we get

Original: [Equation]
Revised: [Equation]

Original: where Υ
Revised: where Υ

Original: [Equation]
Revised: [Equation]

Original: Ψ
Revised: Ψ

Original: [Equation]
Revised: [Equation]

Original: and α = n min (cid:16) (1 − p ) 4 L + µ , p 4 λ + µ (cid:17) , as desired.
Revised: and α = n min (cid:16) (1 − p ) 4 L + µ , p 4 λ + µ (cid:17) , as desired.

Original: Corollary C.4 Iteration complexity of Algorithm 2 is minimized for p = 4 λ + µ 4 λ +4 L +2 µ , which yields complexity 4 (cid:16) λµ + Lµ + 12 (cid:17) log 1 ε .
Revised: Corollary C.4 Iteration complexity of Algorithm 2 is minimized for p = 4 λ + µ 4 λ +4 L +2 µ , which yields complexity 4 (cid:16) λµ + Lµ + 12 (cid:17) log 1 ε .

Original: The communication complexity is minimized for any p ≤ 4 λ + µ 4 λ +4 L +2 µ , in which case the total number of communication rounds to reach ε -solution is (cid:16) 4 λµ + 1 (cid:17) log 1 ε .
Revised: The communication complexity is minimized for any p ≤ 4 λ + µ 4 λ +4 L +2 µ , in which case the total number of communication rounds to reach ε -solution is (cid:16) 4 λµ + 1 (cid:17) log 1 ε .

Original: As a direct consequence of Corollary C.4 we see that the optimal choice of p that minimizes both communication and number of iterations to reach ε solution of problem (17) is p = 4 λ + µ 4 λ +4 L +2 µ .
Revised: As a direct consequence of Corollary C.4 we see that the optimal choice of p that minimizes both communication and number of iterations to reach ε solution of problem (17) is p = 4 λ + µ 4 λ +4 L +2 µ .

Original: Remark C.

Revised: Remark C.

Original: While both Algorithm 2 and Algorithm 3 are a special case of SAGA, the practical version of variance reduced local SGD (presented in Section C.5) is not.
Revised: While both Algorithm 2 and Algorithm 3 are a special case of SAGA, the practical version of variance reduced local SGD (presented in Section C.5) is not.

Original: In particular, we wish to run the SVRG-like method locally in order to avoid storing the full gradient table.
Revised: In particular, we wish to run the SVRG-like method locally in order to avoid storing the full gradient table.

Original: Therefore, variance reduced local SGD that will be proposed in Section C.5 is neither a special case of SAGA nor a special case of SVRG (or a variant of SVRG).
Revised: Therefore, variance reduced local SGD that will be proposed in Section C.5 is neither a special case of SAGA nor a special case of SVRG (or a variant of SVRG).

Original: However, it is still a special case of a more general algorithm from (Hanzely & Richt´arik, 2019).
Revised: However, it is still a special case of a more general algorithm from (Hanzely & Richt´arik, 2019).

Original: As mentioned, Algorithm 3 is a generalization of Algorithm 2 when the local subproblem is a ﬁnite sum.
Revised: As mentioned, Algorithm 3 is a generalization of Algorithm 2 when the local subproblem is a ﬁnite sum.

Original: Note that Algorithm 2 constructs a control variates for both local subproblem and aggregation function ψ and constructs corresponding unbiased gradient estimator.
Revised: Note that Algorithm 2 constructs a control variates for both local subproblem and aggregation function ψ and constructs corresponding unbiased gradient estimator.

Original: In contrast, Algorithm 3 constructs extra control variates within the local subproblem in order to reduce the variance of gradient estimator coming from the local subsampling.
Revised: In contrast, Algorithm 3 constructs extra control variates within the local subproblem in order to reduce the variance of gradient estimator coming from the local subsampling.

Original: C. L2SGD+: A LGORITHM AND THE EFFICIENT IMPLEMENTATION
Revised: C. L2SGD+: A LGORITHM AND THE EFFICIENT IMPLEMENTATION

Original: Denote 1 ∈ R m to be vector of ones.
Revised: Denote 1 ∈ R m to be vector of ones.

Original: We are now ready to state L2SGD+ as Algorithm 3.
Revised: We are now ready to state L2SGD+ as Algorithm 3.

Original: Algorithm
Revised: Algorithm

Original: Input: x 01 = · · · = x 0 n ∈ R d , stepsize α , probability p
Revised: Input: x 01 = · · · = x 0 n ∈ R d , stepsize α , probability p

Original: J 0 i = 0 ∈ R
Revised: J 0 i = 0 ∈ R

Original: d × m , Ψ 

Revised: d × m , Ψ

Original: i = 

Revised: i =

Original: ∈ R
Revised: ∈ R

Original: d (for i = 1 , . . . , n ) for k = 0 , 1 , . . .
Revised: d (for i = 1 , . . . , n ) for k = 0 , 1 , . . .

Original: do
Revised: do

Original: ξ = 1 with probability p and 0 with probability 1 − p
Revised: ξ = 1 with probability p and 0 with probability 1 − p

Original: if ξ = 0 then
Revised: if ξ = 0 then

Original: All Devices i = 1 , . . . , n :
Revised: All Devices i = 1 , . . . , n :

Original: Sample j ∈ { 1 , . . .
Revised: Sample j ∈ { 1 , . . .

Original: , m } (uniformly at random)
Revised: , m } (uniformly at random)

Original: g ki = 1 n (1 − p ) (cid:16) ∇ f (cid:48)
Revised: g ki = 1 n (1 − p ) (cid:16) ∇ f (cid:48)

Original: i,j ( x ki ) − (cid:0) J ki (cid:1) : ,j (cid:17) + J ki 1 nm + Ψ ki n x k
Revised: i,j ( x ki ) − (cid:0) J ki (cid:1) : ,j (cid:17) + J ki 1 nm + Ψ ki n x k

Original: +1 i
Revised: +1 i

Original: = x ki − αg ki
Revised: = x ki − αg ki

Original: Set ( J k +1 i ) : ,j = ∇ f (cid:48)
Revised: Set ( J k +1 i ) : ,j = ∇ f (cid:48)

Original: i,j ( x ki ) , Ψ k +1 i =
Revised: i,j ( x ki ) , Ψ k +1 i =

Original: Ψ ki , ( J k +1 i ) : ,l = ( J k +1 i ) : ,l for all l (cid:54) =
Revised: Ψ ki , ( J k +1 i ) : ,l = ( J k +1 i ) : ,l for all l (cid:54) =

Original: j else
Revised: j else

Original: Master computes the average ¯ x k = 1 n (cid:80) ni
Revised: Master computes the average ¯ x k = 1 n (cid:80) ni

Original: =1 x ki
Revised: =1 x ki

Original: Master does for all i = 1 , . . .
Revised: Master does for all i = 1 , . . .

Original: , n :
Revised: , n :

Original: Set x
Revised: Set x

Original: 1 i
Revised: 1 i

Original: Set Ψ k
Revised: Set Ψ k

Original: +

Revised: +

Original: i end if end for
Revised: i end if end for

Original: L2SGD+ only communicates when a two consecutive coin tosses land a different value, thus, on average p (1 − p ) k times per k iterations.
Revised: L2SGD+ only communicates when a two consecutive coin tosses land a different value, thus, on average p (1 − p ) k times per k iterations.

Original: However, L2SGD+ requires communication of control variates J i 1 , Ψ i as well – each communication round is thus three times more expensive.
Revised: However, L2SGD+ requires communication of control variates J i 1 , Ψ i as well – each communication round is thus three times more expensive.

Original: In the
Revised: In the

Original: Appendix, we provide an implementation of L2SGD+ that does not require the communication of
Revised: Appendix, we provide an implementation of L2SGD+ that does not require the communication of

Original: J i 1 , Ψ i .
Revised: J i 1 , Ψ i .

Original: Here we present an efﬁcient implementation of L2SGD+ as Algorithm 4 so that we do not have to communicate control variates.
Revised: Here we present an efﬁcient implementation of L2SGD+ as Algorithm 4 so that we do not have to communicate control variates.

Original: As a consequence, Algorithm 4 needs to communicate on average p (1 − p ) k times per k iterations, while each communication consists of sending only local models to the master and back.
Revised: As a consequence, Algorithm 4 needs to communicate on average p (1 − p ) k times per k iterations, while each communication consists of sending only local models to the master and back.

Original: C. L OCAL SGD WITH VARIANCE REDUCTION – GENERAL METHOD
Revised: C. L OCAL SGD WITH VARIANCE REDUCTION – GENERAL METHOD

Original: In this section, we present a fully general variance reduced local SGD.
Revised: In this section, we present a fully general variance reduced local SGD.

Original: We consider a more general instance of (2) where each local objective includes a possibly nonsmooth regularizer, which admits a cheap evaluation of proximal operator.
Revised: We consider a more general instance of (2) where each local objective includes a possibly nonsmooth regularizer, which admits a cheap evaluation of proximal operator.

Original: In particular, the objective becomes
Revised: In particular, the objective becomes

Original: Input: x 01 = · · · = x 0 n = ˜ x ∈ R d , stepsize α , probability p
Revised: Input: x 01 = · · · = x 0 n = ˜ x ∈ R d , stepsize α , probability p

Original: Initialize control variates J 0 i = 

Revised: Initialize control variates J 0 i =

Original: ∈ R
Revised: ∈ R

Original: d × m , Ψ 

Revised: d × m , Ψ

Original: i = 

Revised: i =

Original: ∈ R
Revised: ∈ R

Original: d (for i = 1 , . . . , n ), initial coin toss ξ − 1 =for k = 0 , 1 , . . .
Revised: d (for i = 1 , . . . , n ), initial coin toss ξ − 1 =for k = 0 , 1 , . . .

Original: do ξ k = 1 with probability p and 0 with probability 1 − p if ξ k = 0 then
Revised: do ξ k = 1 with probability p and 0 with probability 1 − p if ξ k = 0 then

Original: All Devices i = 1 , . .
Revised: All Devices i = 1 , . .

Original: , n : if ξ k − 1 = 1 then
Revised: , n : if ξ k − 1 = 1 then

Original: Receive x ki , c from Master
Revised: Receive x ki , c from Master

Original: Reconstruct ¯ x k = ¯ x k − c using x ki , x k − c i , c
Revised: Reconstruct ¯ x k = ¯ x k − c using x ki , x k − c i , c

Original: Set x ki = x ki − cα 1 nm J ki 1 , J ki =
Revised: Set x ki = x ki − cα 1 nm J ki 1 , J ki =

Original: J k − c i , Ψ ki = λ ( x k − c i − ¯ x k ) , end if
Revised: J k − c i , Ψ ki = λ ( x k − c i − ¯ x k ) , end if

Original: Sample j ∈ { 1 , . . .
Revised: Sample j ∈ { 1 , . . .

Original: , m } (uniformly at random)
Revised: , m } (uniformly at random)

Original: g ki = 1 n (1 − p ) (cid:16) ∇ f (cid:48)
Revised: g ki = 1 n (1 − p ) (cid:16) ∇ f (cid:48)

Original: i,j ( x ki ) − (cid:0) J ki (cid:1) : ,j (cid:17) + J ki 1 nm + Ψ ki n x k
Revised: i,j ( x ki ) − (cid:0) J ki (cid:1) : ,j (cid:17) + J ki 1 nm + Ψ ki n x k

Original: +1 i
Revised: +1 i

Original: = x k i − αg k
Revised: = x k i − αg k

Original: i
Revised: i

Original: Set ( J k +1 i ) : ,j = ∇ f (cid:48)
Revised: Set ( J k +1 i ) : ,j = ∇ f (cid:48)

Original: i,j ( x ki ) , Ψ k +1 i =
Revised: i,j ( x ki ) , Ψ k +1 i =

Original: Ψ ki , ( J k +1 i ) : ,l = ( J k +1 i ) : ,l for all l (cid:54) =
Revised: Ψ ki , ( J k +1 i ) : ,l = ( J k +1 i ) : ,l for all l (cid:54) =

Original: j else
Revised: j else

Original: Master does for all i = 1 , . .
Revised: Master does for all i = 1 , . .

Original: , n : if ξ k − 1 = 0 then
Revised: , n : if ξ k − 1 = 0 then

Original: Set c =
Revised: Set c =

Original: Receive x ki from Device and set ¯ x = 1 n (cid:80) ni
Revised: Receive x ki from Device and set ¯ x = 1 n (cid:80) ni

Original: =1 x ki , x ki = x ki end if
Revised: =1 x ki , x ki = x ki end if

Original: Set x k +1 i
Revised: Set x k +1 i

Original: = x ki − α (cid:16) λnp ( x ki − ¯ x ) − p −
Revised: = x ki − α (cid:16) λnp ( x ki − ¯ x ) − p −

Original: [Equation]
Revised: [Equation]

Original: Set ˜ x
Revised: Set ˜ x

Original: = x ki
Revised: = x ki

Original: Set c
Revised: Set c

Original: = c +end if end for
Revised: = c +end if end for

Original: [Equation]
Revised: [Equation]

Original: where m i is the number of data points owned by client i and N = (cid:80) ni
Revised: where m i is the number of data points owned by client i and N = (cid:80) ni

Original: =

Revised: =

Original: m i .
Revised: m i .

Original: In order to squeeze a faster convergence rate from minibatch samplings, we will assume that f (cid:48)
Revised: In order to squeeze a faster convergence rate from minibatch samplings, we will assume that f (cid:48)

Original: i,j is smooth with respect to a matrix M i,j (instead of scalar L (cid:48)
Revised: i,j is smooth with respect to a matrix M i,j (instead of scalar L (cid:48)

Original: i,j = λ max M i,j ).
Revised: i,j = λ max M i,j ).

Original: Assumption C.1 Suppose that f (cid:48)
Revised: Assumption C.1 Suppose that f (cid:48)

Original: i,j is M i,j smooth ( M i,j ∈ R d × d , M i,j (cid:31) 0 ) and µ strongly convex for 1 ≤ j ≤ m i , 1 ≤ i ≤ n , i.e. f (cid:48)
Revised: i,j is M i,j smooth ( M i,j ∈ R d × d , M i,j (cid:31) 0 ) and µ strongly convex for 1 ≤ j ≤ m i , 1 ≤ i ≤ n , i.e. f (cid:48)

Original: i,j ( y )+ (cid:10) ∇ f (cid:48) i,j ( y ) , x − y (cid:11) ≤ f (cid:48) i,j ( x ) ≤ f (cid:48)
Revised: i,j ( y )+ (cid:10) ∇ f (cid:48) i,j ( y ) , x − y (cid:11) ≤ f (cid:48) i,j ( x ) ≤ f (cid:48)

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: j ( y
Revised: j ( y

Original: [Equation]
Revised: [Equation]

Original: + (cid:10) ∇ f (cid:48)
Revised: + (cid:10) ∇ f (cid:48)

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: j ( y )
Revised: j ( y )

Original: [Equation]
Revised: [Equation]

Original: x − y (cid:11) + 1 2 (cid:107) y − x (cid:107) 2 M i
Revised: x − y (cid:11) + 1 2 (cid:107) y − x (cid:107) 2 M i

Original: [Equation]
Revised: [Equation]

Original: j , ∀ x, y ∈ R d .
Revised: j , ∀ x, y ∈ R d .

Original: Furthermore, assume that R i is convex for 1 ≤ i ≤ n .
Revised: Furthermore, assume that R i is convex for 1 ≤ i ≤ n .

Original: Our method (Algotihm 5) allows for arbitrary aggregation probability (same as Algorithms 2, 3), arbitrary sampling of clients (to model the inactive clients) and arbitrary structure/sampling of the local objectives (i.e., arbitrary size of local datasets, arbitrary smoothness structure of each local objective and arbitrary subsampling strategy of each client).
Revised: Our method (Algotihm 5) allows for arbitrary aggregation probability (same as Algorithms 2, 3), arbitrary sampling of clients (to model the inactive clients) and arbitrary structure/sampling of the local objectives (i.e., arbitrary size of local datasets, arbitrary smoothness structure of each local objective and arbitrary subsampling strategy of each client).

Original: Moreover, it allows for the SVRG-like update rule of local control variates J k , which requires less storage given an efﬁcient implementation.
Revised: Moreover, it allows for the SVRG-like update rule of local control variates J k , which requires less storage given an efﬁcient implementation.

Original: To be speciﬁc, each device owns a distribution D i over subsets of m i .
Revised: To be speciﬁc, each device owns a distribution D i over subsets of m i .

Original: When the aggregation is not performed (with probability 1 − p ), a subset of active devices S is selected ( S follows arbitrary ﬁxed distribution D ).
Revised: When the aggregation is not performed (with probability 1 − p ), a subset of active devices S is selected ( S follows arbitrary ﬁxed distribution D ).

Original: Each of the active clients ( i ∈ S ) samples a subset of local indices
Revised: Each of the active clients ( i ∈ S ) samples a subset of local indices

Original: S i ∼ D
Revised: S i ∼ D

Original: i and observe the corresponding part of local Jacobian G i ( x k ) (: ,S i ) (where
Revised: i and observe the corresponding part of local Jacobian G i ( x k ) (: ,S i ) (where

Original: G i ( x k ) :=
Revised: G i ( x k ) :=

Original: [ ∇ f (cid:48) i, 1 ( x k ) , ∇ f (cid:48)
Revised: [ ∇ f (cid:48) i, 1 ( x k ) , ∇ f (cid:48)

Original: i, 2 ( x k ) , . . .
Revised: i, 2 ( x k ) , . . .

Original: ∇ f (cid:48)
Revised: ∇ f (cid:48)

Original: i,m i ( x k ) ).
Revised: i,m i ( x k ) ).

Original: When the aggregation is performed (with probability p ) we evaluate ¯ x k and distribute it to each device; using which each device computes a corresponding component of λ ∇ ψ ( x k ) .
Revised: When the aggregation is performed (with probability p ) we evaluate ¯ x k and distribute it to each device; using which each device computes a corresponding component of λ ∇ ψ ( x k ) .

Original: Those are the key components in constructing the unbiased gradient estimator (without control variates).
Revised: Those are the key components in constructing the unbiased gradient estimator (without control variates).

Original: It remains to construct control variates and unbiased gradient estimator.
Revised: It remains to construct control variates and unbiased gradient estimator.

Original: If the aggregation is done, we just simply replace the last column of the gradient table.
Revised: If the aggregation is done, we just simply replace the last column of the gradient table.

Original: If the aggregation is not done, we have two options – either keep replacing the columns of the Jacobian table (in such case, we obtain a particular case of SAGA (Defazio et al., 2014)) or do LSVRG-like replacement (Hofmann et al., 2015; Kovalev et al., 2020) (in such case, the algorithm is a particular case of GJS (Hanzely &
Revised: If the aggregation is not done, we have two options – either keep replacing the columns of the Jacobian table (in such case, we obtain a particular case of SAGA (Defazio et al., 2014)) or do LSVRG-like replacement (Hofmann et al., 2015; Kovalev et al., 2020) (in such case, the algorithm is a particular case of GJS (Hanzely &

Original: Richt ´ arik, 2019), but is not a special case of neither SAGA nor LSVRG.
Revised: Richt ´ arik, 2019), but is not a special case of neither SAGA nor LSVRG.

Original: Note that LSVRG-like replacement is preferrable in practice due to a better memory efﬁciency (one does not need to store the whole gradient table) for the models other than linear.
Revised: Note that LSVRG-like replacement is preferrable in practice due to a better memory efﬁciency (one does not need to store the whole gradient table) for the models other than linear.

Original: In order to keep the gradient estimate unbiased, it will be convenient to deﬁne vector p i ∈
Revised: In order to keep the gradient estimate unbiased, it will be convenient to deﬁne vector p i ∈

Original: R m i such that for each j ∈ { 1 , . . .
Revised: R m i such that for each j ∈ { 1 , . . .

Original: , m i } we have P ( j ∈ S i ) =
Revised: , m i } we have P ( j ∈ S i ) =

Original: p i,j .
Revised: p i,j .

Original: Next, to give a tight rate for any given pair of smoothness structure and sampling strategy, we use a standard tool ﬁrst proposed for the analysis of randomized coordinate descent methods (Richt´arik &
Revised: Next, to give a tight rate for any given pair of smoothness structure and sampling strategy, we use a standard tool ﬁrst proposed for the analysis of randomized coordinate descent methods (Richt´arik &

Original: Tak ´ a ˇ c, 2016; Qu & Richt ´ arik, 2016) called Expected Separable Overapproximation (ESO) assumption.
Revised: Tak ´ a ˇ c, 2016; Qu & Richt ´ arik, 2016) called Expected Separable Overapproximation (ESO) assumption.

Original: ESO provides us with smoothness parameters of the objective which “account” for the given sampling strategy.
Revised: ESO provides us with smoothness parameters of the objective which “account” for the given sampling strategy.

Original: Assumption C.2 Suppose that there is v i ∈
Revised: Assumption C.2 Suppose that there is v i ∈

Original: R m i such for each client we have:
Revised: R m i such for each client we have:

Original: [Equation]
Revised: [Equation]

Original: Lastly, denote p i to be the probability that worker i is active and 1 ( m i ) ∈
Revised: Lastly, denote p i to be the probability that worker i is active and 1 ( m i ) ∈

Original: R m i to be the vector of ones.
Revised: R m i to be the vector of ones.

Original: The resulting algorithm is stated as Algorithm 5.
Revised: The resulting algorithm is stated as Algorithm 5.

Original: Next, Theorems C.6 and C.7 present convergence rate of Algorithm 5 (SAGA and SVRG variant, respectively).
Revised: Next, Theorems C.6 and C.7 present convergence rate of Algorithm 5 (SAGA and SVRG variant, respectively).

Original: Theorem C.6 Suppose that Assumptions C.1 and C.2 hold.
Revised: Theorem C.6 Suppose that Assumptions C.1 and C.2 hold.

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: Then the iteration complexity of Algorithm 5 (SAGA option) is
Revised: Then the iteration complexity of Algorithm 5 (SAGA option) is

Original: [Equation]
Revised: [Equation]

Original: Theorem C.7 Suppose that Assumptions C
Revised: Theorem C.7 Suppose that Assumptions C

Original: [Equation]
Revised: [Equation]

Original: 1 and C
Revised: 1 and C

Original: [Equation]
Revised: [Equation]

Original: 2 hold.
Revised: 2 hold.

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: Then the iteration complexity of Algorithm 5 (LSVRG option) is
Revised: Then the iteration complexity of Algorithm 5 (LSVRG option) is

Original: [Equation]
Revised: [Equation]

Original: Algorithm
Revised: Algorithm

Original: Input: x 01 , . . .
Revised: Input: x 01 , . . .

Original: x 0 n ∈ R d , # parallel units n , each of them owns m i data points (for 1 ≤ i ≤ n ), distributions D t over subsets of { 1 , . . .
Revised: x 0 n ∈ R d , # parallel units n , each of them owns m i data points (for 1 ≤ i ≤ n ), distributions D t over subsets of { 1 , . . .

Original: , m i } , distribution D over subsets of { 1 , 2 , . . .
Revised: , m i } , distribution D over subsets of { 1 , 2 , . . .

Original: n } , aggregation probability p , stepsize α
Revised: n } , aggregation probability p , stepsize α

Original: J 0 i = 0 ∈ R
Revised: J 0 i = 0 ∈ R

Original: d
Revised: d

Original: × m i , Ψ 

Revised: × m i , Ψ

Original: i = 

Revised: i =

Original: ∈ R
Revised: ∈ R

Original: d (for i = 1 , . . . , n ) for k = 0 , 1 , . . .
Revised: d (for i = 1 , . . . , n ) for k = 0 , 1 , . . .

Original: do
Revised: do

Original: ξ = 1 with probability p and 0 with probability 1 − p
Revised: ξ = 1 with probability p and 0 with probability 1 − p

Original: if ξ = 0 then
Revised: if ξ = 0 then

Original: Sample S ∼ D
Revised: Sample S ∼ D

Original: All Devices i ∈ S :
Revised: All Devices i ∈ S :

Original: Sample S i ∼ D
Revised: Sample S i ∼ D

Original: i ; S i ⊆ { 1 , . . .
Revised: i ; S i ⊆ { 1 , . . .

Original: , m i } (independently on each machine)
Revised: , m i } (independently on each machine)

Original: Observe ∇ f (cid:48) i,j ( x ki ) for all j ∈ S
Revised: Observe ∇ f (cid:48) i,j ( x ki ) for all j ∈ S

Original: i g ki = 1 N (1 − p ) p
Revised: i g ki = 1 N (1 − p ) p

Original: i (cid:16)(cid:80) j ∈ S i p − 1 i,j (cid:16) ∇ f (cid:48)
Revised: i (cid:16)(cid:80) j ∈ S i p − 1 i,j (cid:16) ∇ f (cid:48)

Original: i,j ( x ki ) − (cid:0) J ki (cid:1) : ,j (cid:17)(cid:17) + 1 N J ki 1 ( m i ) + n − 1 Ψ ki x k
Revised: i,j ( x ki ) − (cid:0) J ki (cid:1) : ,j (cid:17)(cid:17) + 1 N J ki 1 ( m i ) + n − 1 Ψ ki x k

Original: +1 i =
Revised: +1 i =

Original: prox
Revised: prox

Original: αR i ( x ki − αg ki )
Revised: αR i ( x ki − αg ki )

Original: For all j ∈ { 1 , . . .
Revised: For all j ∈ { 1 , . . .

Original: , m i } set J k +1 : ,j =   (cid:26) ∇ f (cid:48)
Revised: , m i } set J k +1 : ,j =   (cid:26) ∇ f (cid:48)

Original: i,j ( x ki ) if j ∈ S
Revised: i,j ( x ki ) if j ∈ S

Original: i
Revised: i

Original: J k : ,j otherwise if SAGA (cid:26) ∇ f (cid:48) i,j ( x ki ); w. p. p
Revised: J k : ,j otherwise if SAGA (cid:26) ∇ f (cid:48) i,j ( x ki ); w. p. p

Original: i
Revised: i

Original: J k : ,j otherwise if L − SVRG
Revised: J k : ,j otherwise if L − SVRG

Original: Set Ψ k +1 i =
Revised: Set Ψ k +1 i =

Original: Ψ ki
Revised: Ψ ki

Original: All Devices i (cid:54)∈ S : g ki =
Revised: All Devices i (cid:54)∈ S : g ki =

Original: 1 N J ki 1 ( m i ) + n − 1 Ψ ki x k
Revised: 1 N J ki 1 ( m i ) + n − 1 Ψ ki x k

Original: +1 i =
Revised: +1 i =

Original: prox
Revised: prox

Original: αR i ( x ki − αg ki )
Revised: αR i ( x ki − αg ki )

Original: Set J k +1 i
Revised: Set J k +1 i

Original: = J ki , Ψ k +1 i =
Revised: = J ki , Ψ k +1 i =

Original: Ψ ki else
Revised: Ψ ki else

Original: Master computes the average ¯ x k = 1 n (cid:80) ni
Revised: Master computes the average ¯ x k = 1 n (cid:80) ni

Original: =1 x ki
Revised: =1 x ki

Original: Master does for all i = 1 , . . .
Revised: Master does for all i = 1 , . . .

Original: , n : g ki = p − 1 λ ( x ki − ¯ x k ) − ( p − 1 − 1) n − 1 Ψ ki + 1 N J ki 1 ( m i )
Revised: , n : g ki = p − 1 λ ( x ki − ¯ x k ) − ( p − 1 − 1) n − 1 Ψ ki + 1 N J ki 1 ( m i )

Original: Set x k +1 i =
Revised: Set x k +1 i =

Original: prox
Revised: prox

Original: αR i (cid:0) x ki − αg ki (cid:1)
Revised: αR i (cid:0) x ki − αg ki (cid:1)

Original: Set Ψ k +1 i
Revised: Set Ψ k +1 i

Original: = λ ( x ki − ¯ x k ) ,
Revised: = λ ( x ki − ¯ x k ) ,

Original: J k +1 i
Revised: J k +1 i

Original: =
Revised: =

Original: J ki end if end for
Revised: J ki end if end for

Original: Remark C.8 Algotihm 2 is a special case of Algorithm 3 which is in turn special case of Algorithm 5.
Revised: Remark C.8 Algotihm 2 is a special case of Algorithm 3 which is in turn special case of Algorithm 5.

Original: Similarly, Theorem 2 is a special case of Theorem 5.1 which is again special case of
Revised: Similarly, Theorem 2 is a special case of Theorem 5.1 which is again special case of

Original: Theorem C.6.
Revised: Theorem C.6.

Original: C. L OCAL STOCHASTIC ALGORITHMS
Revised: C. L OCAL STOCHASTIC ALGORITHMS

Original: In this section, we present two more algorithms – Local SGD with partial variance reduction (Algorithm 7) and Local SGD without variance reduction (Algorithm 6).
Revised: In this section, we present two more algorithms – Local SGD with partial variance reduction (Algorithm 7) and Local SGD without variance reduction (Algorithm 6).

Original: While Algorithm 6 uses no control variates at all (thus is essentially Algorithm 1 where local gradient descent steps are replaced with local SGD steps), Algorithm 7 constructs control variates for ψ only, resulting in locally drifted
Revised: While Algorithm 6 uses no control variates at all (thus is essentially Algorithm 1 where local gradient descent steps are replaced with local SGD steps), Algorithm 7 constructs control variates for ψ only, resulting in locally drifted

Original: SGD algorithm (with the constant drift between each consecutive rounds of communication).
Revised: SGD algorithm (with the constant drift between each consecutive rounds of communication).

Original: While we do not present the convergence rates of the methods here, we shall notice they can be easily obtained using the framework from (Gorbunov et al., 2020).
Revised: While we do not present the convergence rates of the methods here, we shall notice they can be easily obtained using the framework from (Gorbunov et al., 2020).

Original: Algorithm
Revised: Algorithm

Original: Input: x 01 = · · · = x 0 n ∈ R d , stepsize α , probability p for k = 0 , 1 , . . .
Revised: Input: x 01 = · · · = x 0 n ∈ R d , stepsize α , probability p for k = 0 , 1 , . . .

Original: do
Revised: do

Original: ξ = 1 with probability p and 0 with probability 1 − p
Revised: ξ = 1 with probability p and 0 with probability 1 − p

Original: if ξ = 0 then
Revised: if ξ = 0 then

Original: All Devices i = 1 , . . . , n :
Revised: All Devices i = 1 , . . . , n :

Original: Sample j ∈ { 1 , . . .
Revised: Sample j ∈ { 1 , . . .

Original: , m } (uniformly at random)
Revised: , m } (uniformly at random)

Original: g ki = 1 n (1 − p ) (cid:0) ∇ f (cid:48)
Revised: g ki = 1 n (1 − p ) (cid:0) ∇ f (cid:48)

Original: i,j ( x ki ) (cid:1) x k +1 i
Revised: i,j ( x ki ) (cid:1) x k +1 i

Original: = x ki − αg ki else
Revised: = x ki − αg ki else

Original: Master computes the average ¯ x k = 1 n (cid:80) ni
Revised: Master computes the average ¯ x k = 1 n (cid:80) ni

Original: =1 x ki
Revised: =1 x ki

Original: Master does for all i = 1 , . . .
Revised: Master does for all i = 1 , . . .

Original: , n : g k i
Revised: , n : g k i

Original: =
Revised: =

Original: λ np ( x k i − ¯ x k )
Revised: λ np ( x k i − ¯ x k )

Original: Set x k +1 i
Revised: Set x k +1 i

Original: = x ki − αg ki end if end for
Revised: = x ki − αg ki end if end for

Original: Algorithm
Revised: Algorithm

Original: Input: x 01 = · · · = x 0 n ∈ R d , stepsize α , probability p Ψ 

Revised: Input: x 01 = · · · = x 0 n ∈ R d , stepsize α , probability p Ψ

Original: i = 

Revised: i =

Original: ∈ R
Revised: ∈ R

Original: d (for i = 1 , . . . , n ) for k = 0 , 1 , . . .
Revised: d (for i = 1 , . . . , n ) for k = 0 , 1 , . . .

Original: do
Revised: do

Original: ξ = 1 with probability p and 0 with probability 1 − p
Revised: ξ = 1 with probability p and 0 with probability 1 − p

Original: if ξ = 0 then
Revised: if ξ = 0 then

Original: All Devices i = 1 , . . . , n :
Revised: All Devices i = 1 , . . . , n :

Original: Sample j ∈ { 1 , . . .
Revised: Sample j ∈ { 1 , . . .

Original: , m } (uniformly at random)
Revised: , m } (uniformly at random)

Original: g ki = 1 n (1 − p ) (cid:0) ∇ f (cid:48)
Revised: g ki = 1 n (1 − p ) (cid:0) ∇ f (cid:48)

Original: i,j ( x ki ) (cid:1) + 1 n Ψ ki x k +1 i
Revised: i,j ( x ki ) (cid:1) + 1 n Ψ ki x k +1 i

Original: = x ki − αg ki
Revised: = x ki − αg ki

Original: Set Ψ k +1 i =
Revised: Set Ψ k +1 i =

Original: Ψ ki else
Revised: Ψ ki else

Original: Master computes the average ¯ x k = 1 n (cid:80) ni
Revised: Master computes the average ¯ x k = 1 n (cid:80) ni

Original: =1 x ki
Revised: =1 x ki

Original: Master does for all i = 1 , . . .
Revised: Master does for all i = 1 , . . .

Original: , n : g ki = λnp ( x ki − ¯ x k ) − p − 1 − 1 n Ψ ki
Revised: , n : g ki = λnp ( x ki − ¯ x k ) − p − 1 − 1 n Ψ ki

Original: Set x k +1 i
Revised: Set x k +1 i

Original: = x ki − αg ki
Revised: = x ki − αg ki

Original: Set Ψ k
Revised: Set Ψ k

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: end if end for
Revised: end if end for

Original: D M ISSING L EMMAS AND P ROOFS D. G RADIENT AND H ESSIAN OF ψ
Revised: D M ISSING L EMMAS AND P ROOFS D. G RADIENT AND H ESSIAN OF ψ

Original: Lemma D.1 Let I be the d × d identity matrix
Revised: Lemma D.1 Let I be the d × d identity matrix

Original: and I n be n × n identity matrix.
Revised: and I n be n × n identity matrix.

Original: Then, we have ∇ 2 ψ ( x ) = 1 n (cid:0) I n − 1 n ee (cid:62) (cid:1) ⊗ I and ∇ ψ ( x ) = 1 n   x −   ¯ x ...
Revised: Then, we have ∇ 2 ψ ( x ) = 1 n (cid:0) I n − 1 n ee (cid:62) (cid:1) ⊗ I and ∇ ψ ( x ) = 1 n   x −   ¯ x ...

Original: ¯ x ¯ x ¯ x ...
Revised: ¯ x ¯ x ¯ x ...

Original: ¯ x     .
Revised: ¯ x     .

Original: Furthermore, L ψ = 1 n .
Revised: Furthermore, L ψ = 1 n .

Original: Proof:
Revised: Proof:

Original: Let O the d × d zero matrix and let
Revised: Let O the d × d zero matrix and let

Original: Q i :=
Revised: Q i :=

Original: [ O , . . .
Revised: [ O , . . .

Original: , O (cid:124) (cid:123)(cid:122) (cid:125)
Revised: , O (cid:124) (cid:123)(cid:122) (cid:125)

Original: i −, I , O , . . .
Revised: i −, I , O , . . .

Original: , O (cid:124) (cid:123)(cid:122) (cid:125)
Revised: , O (cid:124) (cid:123)(cid:122) (cid:125)

Original: n − i ] ∈ R d
Revised: n − i ] ∈ R d

Original: × dn and Q :=
Revised: × dn and Q :=

Original: [ I , . . .
Revised: [ I , . . .

Original: , I ] ∈ R d × dn .
Revised: , I ] ∈ R d × dn .

Original: Note that x i = Q
Revised: Note that x i = Q

Original: i x , and ¯ x = 1 n Q x .
Revised: i x , and ¯ x = 1 n Q x .

Original: So, ψ ( x ) = 12 n n (cid:88) i =
Revised: So, ψ ( x ) = 12 n n (cid:88) i =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: Q i x − 1 n Q x (cid:13)(cid:13) 2 = 12 n n (cid:88) i
Revised: Q i x − 1 n Q x (cid:13)(cid:13) 2 = 12 n n (cid:88) i

Original: =
Revised: =

Original: (cid:13)(cid:13)(cid:0) Q i − 1 n Q (cid:1) x
Revised: (cid:13)(cid:13)(cid:0) Q i − 1 n Q (cid:1) x

Original: (cid:13)(cid:13) 2 .
Revised: (cid:13)(cid:13) 2 .

Original: The Hessian of ψ is ∇
Revised: The Hessian of ψ is ∇

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: i =
Revised: i =

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: i = (cid:0)
Revised: i = (cid:0)

Original: [Equation]
Revised: [Equation]

Original: i Q − 1 n Q (cid:62) Q i + 1 n 2 Q (cid:62) Q (cid:1)
Revised: i Q − 1 n Q (cid:62) Q i + 1 n 2 Q (cid:62) Q (cid:1)

Original: = 1 n n (cid:88) i =
Revised: = 1 n n (cid:88) i =

Original: Q (cid:62) i Q i − 1 n n (cid:88) i = 1 n Q (cid:62) i Q − 1 n n (cid:88) i = 1 n Q (cid:62) Q i + 1 n n (cid:88) i = 1 n 2 Q (cid:62) Q = 1 n n (cid:88) i =
Revised: Q (cid:62) i Q i − 1 n n (cid:88) i = 1 n Q (cid:62) i Q − 1 n n (cid:88) i = 1 n Q (cid:62) Q i + 1 n n (cid:88) i = 1 n 2 Q (cid:62) Q = 1 n n (cid:88) i =

Original: Q (cid:62) i Q i − 1 n 2 Q (cid:62) Q and by plugging in for Q and Q i , we get ∇ 2 ψ ( x ) = 1 n   (cid:0) 1 − 1 n (cid:1)
Revised: Q (cid:62) i Q i − 1 n 2 Q (cid:62) Q and by plugging in for Q and Q i , we get ∇ 2 ψ ( x ) = 1 n   (cid:0) 1 − 1 n (cid:1)

Original: I − 1 n I − 1 n I · · · − 1 n I − 1 n I (cid:0) 1 − 1 n (cid:1)
Revised: I − 1 n I − 1 n I · · · − 1 n I − 1 n I (cid:0) 1 − 1 n (cid:1)

Original: I − 1 n I · · · − 1 n I − 1 n
Revised: I − 1 n I · · · − 1 n I − 1 n

Original: I − 1 n I (cid:0) 1 − 1 n (cid:1) I · · · − 1 n I ...
Revised: I − 1 n I (cid:0) 1 − 1 n (cid:1) I · · · − 1 n I ...

Original: − 1 n I − 1 n I − 1 n I · · · (cid:0) 1 − 1 n (cid:1)
Revised: − 1 n I − 1 n I − 1 n I · · · (cid:0) 1 − 1 n (cid:1)

Original: I 
Revised: I 

Original:  = 1 n   (cid:0) 1 − 1 n (cid:1) − 1 n − 1 n · · · − 1 n − 1 n (cid:0) 1 − 1 n (cid:1) − 1 n · · · − 1 n − 1 n − 1 n (cid:0) 1 − 1 n (cid:1) · · · − 1 n ...
Revised:  = 1 n   (cid:0) 1 − 1 n (cid:1) − 1 n − 1 n · · · − 1 n − 1 n (cid:0) 1 − 1 n (cid:1) − 1 n · · · − 1 n − 1 n − 1 n (cid:0) 1 − 1 n (cid:1) · · · − 1 n ...

Original: − 1 n − 1 n − 1 n · · · (cid:0) 1 − 1 n (cid:1)
Revised: − 1 n − 1 n − 1 n · · · (cid:0) 1 − 1 n (cid:1)

Original:   ⊗ I
Revised:   ⊗ I

Original: [Equation]
Revised: [Equation]

Original: Notice that I n − 1 n ee (cid:62) is a circulant matrix, with eigenvalues 1 (multiplicity n − 1 ) and 0 (multiplicity 1).
Revised: Notice that I n − 1 n ee (cid:62) is a circulant matrix, with eigenvalues 1 (multiplicity n − 1 ) and 0 (multiplicity 1).

Original: Since the eigenvalues of a Kronecker product of two matrices are the products of pairs of eigenvalues of the these matrices, we have
Revised: Since the eigenvalues of a Kronecker product of two matrices are the products of pairs of eigenvalues of the these matrices, we have

Original: [Equation]
Revised: [Equation]

Original: So, L ψ = 1 n .
Revised: So, L ψ = 1 n .

Original: The gradient of ψ is given by ∇
Revised: The gradient of ψ is given by ∇

Original: [Equation]
Revised: [Equation]

Original: x = 1 n n (cid:88)
Revised: x = 1 n n (cid:88)

Original: i = (cid:0)
Revised: i = (cid:0)

Original: [Equation]
Revised: [Equation]

Original: Q i − 1 n
Revised: Q i − 1 n

Original: [Equation]
Revised: [Equation]

Original: i Q
Revised: i Q

Original: [Equation]
Revised: [Equation]

Original: Q
Revised: Q

Original: [Equation]
Revised: [Equation]

Original: Q
Revised: Q

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: Q
Revised: Q

Original: [Equation]
Revised: [Equation]

Original: x = 1 n n (cid:88)
Revised: x = 1 n n (cid:88)

Original: i =     ... x i...
Revised: i =     ... x i...

Original:   −   ...
Revised:   −   ...

Original: 0¯ x ...
Revised: 0¯ x ...

Original:   −   x i /n .
Revised:   −   x i /n .

Original: x i /n x i /n
Revised: x i /n x i /n

Original: x i /n . x i /n 
Revised: x i /n . x i /n 

Original:  +   ¯ x/n ...
Revised:  +   ¯ x/n ...

Original: ¯ x/n ¯ x/n ¯ x/n ...
Revised: ¯ x/n ¯ x/n ¯ x/n ...

Original: ¯ x/n   
Revised: ¯ x/n   

Original:  = 1 n   n (cid:88) i =   ... x i...
Revised:  = 1 n   n (cid:88) i =   ... x i...

Original:   − n (cid:88) i =   ...
Revised:   − n (cid:88) i =   ...

Original: 0¯ x ...
Revised: 0¯ x ...

Original:   − n (cid:88) i = 
Revised:   − n (cid:88) i = 

Original:  x i /n .
Revised:  x i /n .

Original: x i /n x i /n
Revised: x i /n x i /n

Original: x i /n . x i /n 
Revised: x i /n . x i /n 

Original:  + n (cid:88) i
Revised:  + n (cid:88) i

Original: =   ¯ x/n ...
Revised: =   ¯ x/n ...

Original: ¯ x/n ¯ x/n ¯ x/n ...
Revised: ¯ x/n ¯ x/n ¯ x/n ...

Original: ¯ x/n    
Revised: ¯ x/n    

Original: [Equation]
Revised: [Equation]

Original: D. P ROOF OF T HEOREM 3. 
Revised: D. P ROOF OF T HEOREM 3.

Original: For any λ, θ ≥ 0 we have
Revised: For any λ, θ ≥ 0 we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: By adding inequalities (15) and (16), we get ( θ − λ )
Revised: By adding inequalities (15) and (16), we get ( θ − λ )

Original: ( ψ ( x ( λ ))
Revised: ( ψ ( x ( λ ))

Original: − ψ ( x ( θ ))) ≥ 0 , which means that ψ ( x ( λ )) is decreasing in λ .
Revised: − ψ ( x ( θ ))) ≥ 0 , which means that ψ ( x ( λ )) is decreasing in λ .

Original: Assume
Revised: Assume

Original: λ ≥ θ .
Revised: λ ≥ θ .

Original: From the (16) we get f
Revised: From the (16) we get f

Original: ( x ( λ )) ≥ f ( x ( θ ))
Revised: ( x ( λ )) ≥ f ( x ( θ ))

Original: + θ ( ψ ( x ( θ ))
Revised: + θ ( ψ ( x ( θ ))

Original: − ψ ( x ( λ )))
Revised: − ψ ( x ( λ )))

Original: ≥ f ( x ( θ )) , where the last inequality follows since θ ≥ 0 and since ψ ( x ( θ )) ≥ ψ ( x ( λ )) .
Revised: ≥ f ( x ( θ )) , where the last inequality follows since θ ≥ 0 and since ψ ( x ( θ )) ≥ ψ ( x ( λ )) .

Original: So, f ( x ( λ )) is increasing.
Revised: So, f ( x ( λ )) is increasing.

Original: Notice that since ψ is a non-negative function and since x ( λ ) minimizes F and ψ ( x ( ∞ ))
Revised: Notice that since ψ is a non-negative function and since x ( λ ) minimizes F and ψ ( x ( ∞ ))

Original: = 0 , we have
Revised: = 0 , we have

Original: [Equation]
Revised: [Equation]

Original: which implies (3) and (4).
Revised: which implies (3) and (4).

Original: D. P ROOF OF T HEOREM 3. 
Revised: D. P ROOF OF T HEOREM 3.

Original: The equation ∇ F ( x ( λ ))
Revised: The equation ∇ F ( x ( λ ))

Original: = 0 can be equivalently written as
Revised: = 0 can be equivalently written as

Original: [Equation]
Revised: [Equation]

Original: i = 1 , 2 , . .
Revised: i = 1 , 2 , . .

Original: . , n, which is identical to (5).
Revised: . , n, which is identical to (5).

Original: Averaging these identities over i , we get x ( λ ) =
Revised: Averaging these identities over i , we get x ( λ ) =

Original: [Equation]
Revised: [Equation]

Original: ( λ )
Revised: ( λ )

Original: [Equation]
Revised: [Equation]

Original: λ
Revised: λ

Original: 1 n n (cid:88) i = ∇
Revised: 1 n n (cid:88) i = ∇

Original: [Equation]
Revised: [Equation]

Original: i ( x i ( λ )) , which implies n (cid:88) i = ∇ f i ( x i ( λ ))
Revised: i ( x i ( λ )) , which implies n (cid:88) i = ∇ f i ( x i ( λ ))

Original: = 0 .
Revised: = 0 .

Original: Further, we have ψ ( x ( λ ))
Revised: Further, we have ψ ( x ( λ ))

Original: =
Revised: =

Original: 12 n n (cid:88) i =
Revised: 12 n n (cid:88) i =

Original: (cid:107) x
Revised: (cid:107) x

Original: i ( λ ) − x ( λ ) (cid:107) 2 = 1 2 nλn (cid:88)
Revised: i ( λ ) − x ( λ ) (cid:107) 2 = 1 2 nλn (cid:88)

Original: i =
Revised: i =

Original: (cid:107)∇ f i ( x i ( λ ))
Revised: (cid:107)∇ f i ( x i ( λ ))

Original: (cid:107) 2 = 12 λ 2 (cid:107)∇ f
Revised: (cid:107) 2 = 12 λ 2 (cid:107)∇ f

Original: ( x ( λ ))
Revised: ( x ( λ ))

Original: (cid:107) 2 , as desired.
Revised: (cid:107) 2 , as desired.

Original: D. P ROOF OF T HEOREM 3. 
Revised: D. P ROOF OF T HEOREM 3.

Original: First, observe that
Revised: First, observe that

Original: [Equation]
Revised: [Equation]

Original: where the second identity is due to Theorem 3.2 which says that 1 n (cid:80) i ∇ f i ( x i ( λ ))
Revised: where the second identity is due to Theorem 3.2 which says that 1 n (cid:80) i ∇ f i ( x i ( λ ))

Original: = 0 .
Revised: = 0 .

Original: By applying
Revised: By applying

Original: Jensen’s inequality and Lipschitz continuity of functions
Revised: Jensen’s inequality and Lipschitz continuity of functions

Original: f i , we get
Revised: f i , we get

Original: [Equation]
Revised: [Equation]

Original: It remains to apply (3) and notice that P is strongly convex and thus x ( ∞ ) is indeed the unique minimizer.
Revised: It remains to apply (3) and notice that P is strongly convex and thus x ( ∞ ) is indeed the unique minimizer.

Original: D. P ROOF OF T HEOREM 4. 
Revised: D. P ROOF OF T HEOREM 4.

Original: We ﬁrst show that our gradient estimator G ( x ) satisﬁes the expected smoothness property (Gower et al., 2018; 2019).
Revised: We ﬁrst show that our gradient estimator G ( x ) satisﬁes the expected smoothness property (Gower et al., 2018; 2019).

Original: Lemma D.2 Let L := 1 n max (cid:110)
Revised: Lemma D.2 Let L := 1 n max (cid:110)

Original: L 1 − p , λp (cid:111) and σ 2 := 1 n 2 (cid:80) ni
Revised: L 1 − p , λp (cid:111) and σ 2 := 1 n 2 (cid:80) ni

Original: =1 (cid:16) 11 − p (cid:107)∇ f i ( x i ( λ ))
Revised: =1 (cid:16) 11 − p (cid:107)∇ f i ( x i ( λ ))

Original: (cid:107) 

Revised: (cid:107)

Original: + λ 2 p (cid:107) x
Revised: + λ 2 p (cid:107) x

Original: i ( λ ) − x ( λ ) (cid:107) 2 (cid:17) .
Revised: i ( λ ) − x ( λ ) (cid:107) 2 (cid:17) .

Original: Then for all x ∈ R
Revised: Then for all x ∈ R

Original: d we have the inequalities E (cid:104) (cid:107) G
Revised: d we have the inequalities E (cid:104) (cid:107) G

Original: ( x ) − G ( x ( λ ))
Revised: ( x ) − G ( x ( λ ))

Original: (cid:107) 2 (cid:105) ≤ 2 L ( F ( x ) − F ( x ( λ ))) and
Revised: (cid:107) 2 (cid:105) ≤ 2 L ( F ( x ) − F ( x ( λ ))) and

Original: E (cid:104) (cid:107) G
Revised: E (cid:104) (cid:107) G

Original: x ) (cid:107) 2 (cid:105) ≤ 4 L ( F ( x ) − F ( x ( λ
Revised: x ) (cid:107) 2 (cid:105) ≤ 4 L ( F ( x ) − F ( x ( λ

Original: [Equation]
Revised: [Equation]

Original: Next, Theorem 4.2 from Lemma D.2 by applying Theorem 3.1 from (Gower et al., 2019).
Revised: Next, Theorem 4.2 from Lemma D.2 by applying Theorem 3.1 from (Gower et al., 2019).

Original: D. P ROOF OF L EMMA D. 
Revised: D. P ROOF OF L EMMA D.

Original: We ﬁrst have
Revised: We ﬁrst have

Original: [Equation]
Revised: [Equation]

Original: Since D f + λD ψ = D F and ∇ F ( x ( λ )) =
Revised: Since D f + λD ψ = D F and ∇ F ( x ( λ )) =

Original: [Equation]
Revised: [Equation]

Original: we can continue:
Revised: we can continue:

Original: [Equation]
Revised: [Equation]

Original: Next, note that
Revised: Next, note that

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: as desired.
Revised: as desired.

Original: D. P ROOF OF C OROLLARY 4. 
Revised: D. P ROOF OF C OROLLARY 4.

Original: Firstly
Revised: Firstly

Original: [Equation]
Revised: [Equation]

Original: to minimize the total number of iterations
Revised: to minimize the total number of iterations

Original: [Equation]
Revised: [Equation]

Original: it sufﬁces to minimize L which is achieved with p (cid:63) =
Revised: it sufﬁces to minimize L which is achieved with p (cid:63) =

Original: λL + λ .
Revised: λL + λ .

Original: Let us look at the communication.
Revised: Let us look at the communication.

Original: Fix ε > 0 , choose α = 12 L and let k = 2 n L µ log 1 ε , so that
Revised: Fix ε > 0 , choose α = 12 L and let k = 2 n L µ log 1 ε , so that

Original: [Equation]
Revised: [Equation]

Original: The expected number of communications to achieve this goal is equal to
Revised: The expected number of communications to achieve this goal is equal to

Original: [Equation]
Revised: [Equation]

Original: The quantity Comm p is minimized by choosing any p such that pL =
Revised: The quantity Comm p is minimized by choosing any p such that pL =

Original: (1 − p ) λ , i.e., for p = λλ + L = p (cid:63) , as desired.
Revised: (1 − p ) λ , i.e., for p = λλ + L = p (cid:63) , as desired.

Original: The optimal expected number of communications is therefore equal to
Revised: The optimal expected number of communications is therefore equal to

Original: [Equation]
Revised: [Equation]

Original: D. P ROOF OF C OROLLARY 5. 
Revised: D. P ROOF OF C OROLLARY 5.

Original: Firstly, to minimize the total number of iterations, it sufﬁces to solve
Revised: Firstly, to minimize the total number of iterations, it sufﬁces to solve

Original: [Equation]
Revised: [Equation]

Original: which is achieved with p = p (cid:63)
Revised: which is achieved with p = p (cid:63)

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: The expected number of communications to reach ε -solution is
Revised: The expected number of communications to reach ε -solution is

Original: [Equation]
Revised: [Equation]

Original: Minimizing the above in p yield p = p (cid:63) = 4 λ + µ 4 L (cid:48)
Revised: Minimizing the above in p yield p = p (cid:63) = 4 λ + µ 4 L (cid:48)

Original: +4 λ
Revised: +4 λ

Original: +( m +1) µ , as desired.
Revised: +( m +1) µ , as desired.

Original: The optimal expected number of communications is therefore equal to
Revised: The optimal expected number of communications is therefore equal to

Original: [Equation]
Revised: [Equation]

Original: D. P ROOF OF T HEOREMS 5.1, C.6, AND C. 
Revised: D. P ROOF OF T HEOREMS 5.1, C.6, AND C.

Original: Note ﬁrst that Algorithm 3 is a special case of Algorithm 5, and Theorem 5.1 immediately follows from Theorem C.6.
Revised: Note ﬁrst that Algorithm 3 is a special case of Algorithm 5, and Theorem 5.1 immediately follows from Theorem C.6.

Original: Therefore it sufﬁces to show Theorems C.6, and C.7.
Revised: Therefore it sufﬁces to show Theorems C.6, and C.7.

Original: In order to do so, we will cast Algorithm 5 as a special case of GJS from (Hanzely & Richt´arik, 2019).
Revised: In order to do so, we will cast Algorithm 5 as a special case of GJS from (Hanzely & Richt´arik, 2019).

Original: As a consequence,
Revised: As a consequence,

Original: Theorem C.6 will be a special cases of Theorem 5.2 from (Hanzely & Richt´arik, 2019).
Revised: Theorem C.6 will be a special cases of Theorem 5.2 from (Hanzely & Richt´arik, 2019).

Original: D.
Revised: D.

Original: GJS
Revised: GJS

Original: In this section, we quickly summarize results from (Hanzely & Richt ´ arik, 2019), which we cast to sho convergence rate of Algorithm 3.
Revised: In this section, we quickly summarize results from (Hanzely & Richt ´ arik, 2019), which we cast to sho convergence rate of Algorithm 3.

Original: GJS (Hanzely & Richt ´ arik, 2019) is a method to solve regularized empirical risk minimization objective, i.e.,
Revised: GJS (Hanzely & Richt ´ arik, 2019) is a method to solve regularized empirical risk minimization objective, i.e.,

Original: [Equation]
Revised: [Equation]

Original: Deﬁning G ( x ) :=
Revised: Deﬁning G ( x ) :=

Original: [ ∇ f 1 ( x ) , . .
Revised: [ ∇ f 1 ( x ) , . .

Original: . , ∇ f n ( x )] , we observe S G ( x ) , U G ( x ) every iteration where S is random linear projection operator and U is random linear operator which is identity on expectation.
Revised: . , ∇ f n ( x )] , we observe S G ( x ) , U G ( x ) every iteration where S is random linear projection operator and U is random linear operator which is identity on expectation.

Original: Based on this random gradient information, GJS (Algorithm 8) constructs variance reduced gradient estimator g and takes a proximal step in that direction.
Revised: Based on this random gradient information, GJS (Algorithm 8) constructs variance reduced gradient estimator g and takes a proximal step in that direction.

Original: gradient estimator
Revised: gradient estimator

Original: Next we quickly summarize theory of GJS.
Revised: Next we quickly summarize theory of GJS.

Original: Assumption D.1 Problem (17) has a unique minimizer x (cid:63) , and f is µ -quasi strongly convex, i.e., f ( x (cid:63) ) ≥ f
Revised: Assumption D.1 Problem (17) has a unique minimizer x (cid:63) , and f is µ -quasi strongly convex, i.e., f ( x (cid:63) ) ≥ f

Original: ( y )
Revised: ( y )

Original: + (cid:104)∇ f ( y ) , x (cid:63) − y
Revised: + (cid:104)∇ f ( y ) , x (cid:63) − y

Original: (cid:105)
Revised: (cid:105)

Original: + µ 2 (cid:107)
Revised: + µ 2 (cid:107)

Original: y − x (cid:63) (cid:107) 2 , ∀ y ∈ R d ,
Revised: y − x (cid:63) (cid:107) 2 , ∀ y ∈ R d ,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Functions f
Revised: Functions f

Original: j are convex and M j -smooth for some M j (cid:23) 0 , i.e.,
Revised: j are convex and M j -smooth for some M j (cid:23) 0 , i.e.,

Original: [Equation]
Revised: [Equation]

Original: Theorem D.

Revised: Theorem D.

Original: (Slight simpliﬁcation of Theorem 5.2 from (Hanzely & Richt´arik, 2019)) Let Assumption D.1 hold.
Revised: (Slight simpliﬁcation of Theorem 5.2 from (Hanzely & Richt´arik, 2019)) Let Assumption D.1 hold.

Original: Deﬁne M ( X ) :=
Revised: Deﬁne M ( X ) :=

Original: [ M 1 X : , 1 , . .
Revised: [ M 1 X : , 1 , . .

Original: M n X : ,n ] Let B be any linear operator commuting with S , and assume M † 12 commutes with S .
Revised: M n X : ,n ] Let B be any linear operator commuting with S , and assume M † 12 commutes with S .

Original: Deﬁne the Lyapunov function
Revised: Deﬁne the Lyapunov function

Original: [Equation]
Revised: [Equation]

Original: where { x k } and { J k } are the random iterates produced by Algorithm 8 with stepsize α > 0 .
Revised: where { x k } and { J k } are the random iterates produced by Algorithm 8 with stepsize α > 0 .

Original: Suppose that α and B are chosen so that
Revised: Suppose that α and B are chosen so that

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: M
Revised: M

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)(cid:13)(cid:13)
Revised: (cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: (22) for all X ∈ R d × n .
Revised: (22) for all X ∈ R d × n .

Original: Then for all k ≥ 0 , we have E (cid:2)
Revised: Then for all k ≥ 0 , we have E (cid:2)

Original: Ψ k (cid:3) ≤ (1 − αµ ) k Ψ 0 .
Revised: Ψ k (cid:3) ≤ (1 − αµ ) k Ψ 0 .

Original: D.9. V ARIANCE REDUCED LOCAL SGD AS SPECIAL CASE OF GJS
Revised: D.9. V ARIANCE REDUCED LOCAL SGD AS SPECIAL CASE OF GJS

Original: Let Ω( i, j ) :
Revised: Let Ω( i, j ) :

Original: =
Revised: =

Original: j + (cid:80) i − 1 l =1 m i
Revised: j + (cid:80) i − 1 l =1 m i

Original: In order to case problem (12) as a special case of 17, denote n := N +1 , f Ω( i,j ) ( x ) :
Revised: In order to case problem (12) as a special case of 17, denote n := N +1 , f Ω( i,j ) ( x ) :

Original: =
Revised: =

Original: N +1 N f (cid:48) i,j ( x i ) and f n :=
Revised: N +1 N f (cid:48) i,j ( x i ) and f n :=

Original: ( N + 1) ψ .
Revised: ( N + 1) ψ .

Original: Therefore the objective (12) becomes
Revised: Therefore the objective (12) becomes

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Let v ∈ R n − 1 be such that v Ω
Revised: Let v ∈ R n − 1 be such that v Ω

Original: ( i,j ) =
Revised: ( i,j ) =

Original: N +1 N v i,j and as a consequence of (14) we have
Revised: N +1 N v i,j and as a consequence of (14) we have

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: At the same time, Υ is µ :=
Revised: At the same time, Υ is µ :=

Original: µn strongly convex.
Revised: µn strongly convex.

Original: D.9. P ROOF OF T HEOREM C.6 AND T HEOREM C. 
Revised: D.9. P ROOF OF T HEOREM C.6 AND T HEOREM C.

Original: Let e ∈ R
Revised: Let e ∈ R

Original: d be a vector of ones and p i ∈ R N is such that p ij = p i,j if j ∈ { 1 , . . .
Revised: d be a vector of ones and p i ∈ R N is such that p ij = p i,j if j ∈ { 1 , . . .

Original: , m i } , otherwise p ij = 0 .
Revised: , m i } , otherwise p ij = 0 .

Original: Given the notation, random operator U is chosen as
Revised: Given the notation, random operator U is chosen as

Original: [Equation]
Revised: [Equation]

Original: We next give two options on how to update Jacobian – ﬁrst one is SAGA-like, second one is SVRG like.
Revised: We next give two options on how to update Jacobian – ﬁrst one is SAGA-like, second one is SVRG like.

Original: SAGA-like:
Revised: SAGA-like:

Original: [Equation]
Revised: [Equation]

Original: SVRG-like:
Revised: SVRG-like:

Original: [Equation]
Revised: [Equation]

Original: We can now proceed with the proof of Theorem C.6 and Theorem C.7.
Revised: We can now proceed with the proof of Theorem C.6 and Theorem C.7.

Original: As ∇ f i ( x ) − ∇ f i ( y ) ∈
Revised: As ∇ f i ( x ) − ∇ f i ( y ) ∈

Original: Range ( M i )
Revised: Range ( M i )

Original: [Equation]
Revised: [Equation]

Original: we must have
Revised: we must have

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Due to (26)
Revised: Due to (26)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: inequalities (21) and (22) with choice Y =
Revised: inequalities (21) and (22) with choice Y =

Original: M † 12 X become respectively:
Revised: M † 12 X become respectively:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Above, we have used
Revised: Above, we have used

Original: [Equation]
Revised: [Equation]

Original: Note that E [ S ( X )]
Revised: Note that E [ S ( X )]

Original: =
Revised: =

Original: X · Diag ((1 − p )( p ◦ p ) , p ) where p ∈ R n − 1 such that p Ω( i,j ) =
Revised: X · Diag ((1 − p )( p ◦ p ) , p ) where p ∈ R n − 1 such that p Ω( i,j ) =

Original: p i,j .
Revised: p i,j .

Original: Using (24), setting B to be right multiplication with Diag( b ) and noticing that λ max M n = nλ it sufﬁces to have
Revised: Using (24), setting B to be right multiplication with Diag( b ) and noticing that λ max M n = nλ it sufﬁces to have

Original: [Equation]
Revised: [Equation]

Original: for SAGA case and
Revised: for SAGA case and

Original: [Equation]
Revised: [Equation]

Original: for LSVRG case.
Revised: for LSVRG case.

Original: It remains to notice that to satisfy the SAGA case, it sufﬁces to set b 2 n = 12 np , b 2Ω( i,j ) = 1 2 n (1 − p ) p i,j p i (for j ∈ { 1 , . . .
Revised: It remains to notice that to satisfy the SAGA case, it sufﬁces to set b 2 n = 12 np , b 2Ω( i,j ) = 1 2 n (1 − p ) p i,j p i (for j ∈ { 1 , . . .

Original: , m i } , i ≤ n ) and α = min (cid:110) min j ∈{ 1 ,...,m i } , 1 ≤ i ≤ n n (1 − p ) p i,j p i 4 v Ω( i,j ) + nµ , p 4 λ + µ (cid:111) .
Revised: , m i } , i ≤ n ) and α = min (cid:110) min j ∈{ 1 ,...,m i } , 1 ≤ i ≤ n n (1 − p ) p i,j p i 4 v Ω( i,j ) + nµ , p 4 λ + µ (cid:111) .

Original: To satisfy LSVRG case, it remains to set b 2 n = 1 2 np , b 2 Ω( i,j ) = 1 2 n (1 − p ) p i p
Revised: To satisfy LSVRG case, it remains to set b 2 n = 1 2 np , b 2 Ω( i,j ) = 1 2 n (1 − p ) p i p

Original: i (for j ∈ { 1 , . . .
Revised: i (for j ∈ { 1 , . . .

Original: , m i } , i ≤ n ) and α = min (cid:40) min j ∈{ 1 ,...,m i } , 1 ≤ i ≤ n n (1 − p ) p i 4 v Ω( i,j ) p i,j
Revised: , m i } , i ≤ n ) and α = min (cid:40) min j ∈{ 1 ,...,m i } , 1 ≤ i ≤ n n (1 − p ) p i 4 v Ω( i,j ) p i,j

Original: + nµp − 1 i , p 4 λ + µ (cid:41) .
Revised: + nµp − 1 i , p 4 λ + µ (cid:41) .

Original: The last step to establish is to recall that n = N + 1 , v Ω( i,j )
Revised: The last step to establish is to recall that n = N + 1 , v Ω( i,j )

Original: =
Revised: =

Original: N +1 N v i,j and µ = µn and note that the iteration complexity is 1 αµ log 1 ε = n αµ log 1 ε .
Revised: N +1 N v i,j and µ = µn and note that the iteration complexity is 1 αµ log 1 ε = n αµ log 1 ε .

Original: D.9. P ROOF OF T HEOREM 5. 
Revised: D.9. P ROOF OF T HEOREM 5.

Original: To obtain convergence rate of Theorem 5.1, it remains to use Theorem C.6 with p i
Revised: To obtain convergence rate of Theorem 5.1, it remains to use Theorem C.6 with p i

Original: = 1 , m i
Revised: = 1 , m i

Original: = m ( ∀ i ≤ n ), where each machine samples (when the aggregation is not performed) individual data points with probability 1 m
Revised: = m ( ∀ i ≤ n ), where each machine samples (when the aggregation is not performed) individual data points with probability 1 m

Original: and thus p j = 1 m (for all j ≤ N ).
Revised: and thus p j = 1 m (for all j ≤ N ).

Original: The last remaining thing is to realize that v j = L (cid:48) for all j ≤ N .
Revised: The last remaining thing is to realize that v j = L (cid:48) for all j ≤ N .