Towards Optimal Communication Complexity in Distributed Non-Convex Optimization
Anonymous Author(s)
AffiliationAddress email
Abstract
We study the problem of distributed stochastic non-convex optimization withintermittent communication.
We consider the full participation setting where Mmachines work in parallel over R communication rounds, as well as the partialparticipation setting where m machines are sampled each round.
We propose andanalyze a new algorithm that improves existing methods by requiring fewer andlighter variance reduction operations.
We also present lower bounds, showingour algorithm is either optimal or almost optimal in most settings.
Numericalexperiments demonstrate the superior performance of our algorithm.
Introduction
We consider the following distributed optimization problem with M machines:
[Equation]
where F m , which denotes the objective on machine m , is a non-convex function for all m , asis the average objective F .
We want to solve this problem in the intermittent communication(IC) setting [1, 2] where the machines work in parallel and are allowed to make K oracle callsbetween two communication rounds for R consecutive rounds.
The IC setting has been widelystudied [3, 4, 5, 6, 7, 8, 9, 10, 11, 2, 12] over the past decade, and many recent studies havefocused on the problem with non-convex and heterogeneous objectives [13, 14, 15] which arecommon in cross-device federated learning (FL) [16, 17].
Towards this end, several algorithms[18, 19, 20, 21, 22, 23], all involving local updates (à la local-SGD [3, 16]), have been proposedand analyzed under assumptions bounding the heterogeneity of machines’ objectives.
Althoughthese algorithms demonstrate promising empirical performances, it remains elusive whether thesealgorithms provably dominate the embarrassingly parallelizable alternatives, i.e., mini-batch variantsof the optimal sequential algorithms [24, 25, 26] (a.k.a.
centralized algorithms).
Until very recently, the situation was similar even in the simpler homogeneous setting where F m ’sare all identical and convex, and Woodworth et al.
[2] showed that the optimal algorithm often doesnot require local updates at all.
Even when F m ’s are not identical, for high levels of heterogeneity,accelerated mini-batch SGD [27] is optimal [28].
Should we expect something similar in the nonconvex setting?
Or, can we prove that in some regime local-update algorithms improve over the naivecentralized baselines?
In this paper, we start by noting that in the absence of any heterogeneity assumption (c.f., Section2), centralized algorithms already have the best worst-case convergence guarantee.
Thus, only whenthe heterogeneity is low can the local-update algorithms potentially have an advantage.
This wasthe motivation behind some of the recent works [18, 21, 22].
However, in the absence of any lower bound that explicitly depends on the heterogeneity parameter (such as in [15, 29]), it is not possibleto definitively claim such an improvement.
To alleviate this, (C.1) we provide novel communicationcomplexity lower bounds, under the assumption that F m ’s have bounded first-order or second-orderheterogeneity (see Section 2).
We show that centralized algorithms [24, 25, 26] can never achievethis optimal communication complexity, and most of the existing local-update algorithms cannotattain it either.
[Table]
[Equation]
[Table] Theorem 2.
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
CE-
[Equation]
Theorem
[Equation]
CE-LGD
[Equation]
E Theorem 3.
Lower Bound
[Equation]
- Theorem 3.
Partial
[Table]
Setting
F ED
[Equation]
Mini-batch STORM
[Equation]
S [Equation] Theorem 3.3)
Lower Bound (Centralized)
[Equation]
- Theorem B.
[Equation]
E
[Equation]
[Equation]
[Equation]
S Theorem 3.
CE-LGD
[Equation]
E Theorem 3.
Lower Bound
[Equation]
- Theorem 3.
Working with these limitations in mind, our main contribution is (C.2) a new algorithm CE-LSGDthat we show to be min-max optimal when equipped with exact gradient oracles and near-optimalwhen provided with stochastic gradient oracles (c.f., Section 2).
Our algorithm, like many otherlocal-update algorithms, uses variance-reduction techniques [24, 26] but requires both fewer and lighter “heavy-batch” operations compared to the existing methods (see discussion in Section 3).(C.3)
We also analyze CE-LSGD in the partial client participation setting, which is of particularinterest in cross-device federated learning (FL)[17] where there is an extremely large number ofclients.
Not only does CE-LSGD improve over the best-known communication complexity, butit is the only algorithm that doesn’t require exact oracle queries for variance reduction and stillmanages to be nearly optimal.
Furthermore, if endowed with exact oracles, CE-LSGD is almostmin-max optimal even in the partial participation setting .
Thus our results provably demonstrateoptimality of local update methods, at least in some regimes.
Even in simpler convex settings, wedon’t know of any local update method (exact or stochastic) known to be min-max optimal in theheterogeneous setting [30, 15].
We summarize our results and the comparison to some importantbaselines in Tables 1 and 2.
(C.4)
As an auxiliary contribution, we provide a variant of our algorithmwhich uses stochastic hessian vector product oracles and is thus useful for settings where only a singlecopy of the model can be stored on the edge device.
We also empirically compare our method againstcentralized and local-update algorithms, demonstrating faster convergence and better communication [Table] [Table]
/mϵ 3 / 2 ; and the fourth inequality guarantees that ∆ L/ϵ ≤ ζ ∆ L/mϵ 3 / 2 .
Methods Communication [Table] Full Participation Setting
SCAFFOLD/Mini-batch SGD ∆ Lϵ
[Table]
Mini-batch STORM ∆ Lϵ
[Table]
([26], Theorem D.1)
Lower Bound (Centralized)
[Table]
Lϵ
[Table]
(Theorem 2.1)
STEM
[Table]
BVR-L-SGD
[Table]
CE-LSGD, Theorem 3.
Lower Bound
[Table]
Theorem 3.
Partial Participation Setting
F ED A VG
[Table]
Mini-batch STORM
[Table]
(Theorem D.1)
Lower Bound (Centralized)
[Table]
(Theorem 2.
[Table]
M IME MVR
[Table]
Uses Exact Oracle [21]
M IME L ITE MVR
[Table]
+
[Table]
Uses Exact Oracle [21]
CE-LSGD
[Table]
Theorem 3.
Lower Bound
[Table]
Theorem 3.4, [31, 32]
Notation.
We use B to denote the index set and |B| to denote its cardinality.
For x ∈ R d , we use∥ x ∥ to denote its ℓ 2 -norm.
For A ∈ R d × d , ∥
A ∥ denotes the operator norm.
[ n ] denotes the set{ 1 , 2 , . .
. , n } .
We use ≈ , ⪯ , ⪰ to denote equality and inequality up to numerical constants.
Our Setting and the Centralized Baselines
In this section, we introduce some definitions and assumptions that will be used in our analysis.
Ourgoal is to find an ϵ -approximate stationary point of F , i.e., a point x ∈ R d
such that E [ ∥∇ F ( x ) ∥ 2 ] ≤ ϵ ,where the expectation is w.r.t.
any randomness in the choice of x .
We consider client objectivesin the class F ( L ) of differentiable and L -smooth functions.
We also make assumptions that relatethe functions of different clients to one another.
These are typically known as assumptions on the“heterogeneity” of the problem, and we consider two classes of problems.
Definition 1.
Assume { F m ∈
F ( L ) }
Mm = are first-order ζ -heterogeneous, i.e.,sup x ∈ R d (cid:80)
Mm =1 ∥∇ F m ( x ) − ∇ F ( x ) ∥ 2 /M ≤ ζ 2 .
And for all ∆ ≥ 0 , F (0) − inf x ∈ R d F ( x ) ≤ ∆ ,i.e., the average objective has bounded sub-optimality at zero.
Then we say that{ F m } m ∈ [ M ] ∈ F 1 M ( L, ∆ , ζ ) .
Definition 2.
Assume twice-differentiable { F m ∈ F ( L ) } Mm =1 are second-order τ -heterogeneous,i.e., sup m ∈
[ M ] ,x ∈ R d ∥∇ 2 F m ( x ) − ∇ 2 F ( x ) ∥ ≤ τ .
And for all ∆ ≥ 0 , F (0) − inf x ∈ R d F ( x ) ≤∆ , i.e., the average objective has bounded sub-optimality at zero.
Then we say that{ F m } m ∈ [ M ] ∈ F 2 M ( L, ∆ , τ ) .
We assume that each machine has access to the following multi-point oracle [32] [Section 5.3, 2].
Definition 3.
Given a function G ∈ F ( L, ∆) , O n,L,σG : ( R d ) n × Z → ( R ) n × ( R d ) n is a multipoint stochastic first order oracle if for some distribution D on Z , and for all x 1 , . . .
, x n ∈ R d ,the oracle samples a random seed z ∼ D and returns estimators O n,L,σG ( x 1 , . . .
, x n , z ) =( { f ( x i ; z ) } i ∈ [ n ] , { g ( x i ; z ) } i ∈
[ n ] ) such that ∀ i ∈ [ n ] , E z ∼D ( f ( x i ; z ) , g ( x i ; z )) =
( G ( x i ) , ∇ G ( x i ))and E z ∼D ∥ g ( x i ; z ) − ∇ G ( x i ) ∥ 2 ≤ σ 2 .
Furthermore, the unbiased gradients satisfy mean smoothness, i.e., for all x, y ∈ R d , E z
∼D [ ∥ g ( x ; z ) − g ( y ; z ) ∥ ] ≤ L ∥ x − y ∥ .
Finally, we consider a generalization of zero respecting algorithms denoted by A ZR (see Appendix A)in the intermittent communication (IC) setting.
This class captures a very wide variety of distributedoptimization algorithms, including mini-batch SGD, accelerated mini-batch SGD, local SGD, and allthe variance-reduction algorithms.
Algorithms which are not distributed zero-respecting are thosewhose iterates have components in directions about which the algorithm has no information, meaningthat in some sense, it is just “wild guessing”.
We also denote the class of centralized algorithms in
A ZR by A centZR (see Appendix A).
This includes algorithms such as mini-batch SARAH [24] andmini-batch STORM [26].
We first present a lower bound result applicable to centralized algorithms.
Theorem 2.1 (Centralized Lower Bound) .
For all τ, ∆ , ζ, σ ≥ 0 , and 2 L ≥ τ , every algorithm
A ∈
A centZR optimizing a problem in F 1 M ( L, ∆ , ζ ) ∪ F 2 M ( L, ∆ , τ ) , with access to an oracle O 2 ,L,σF m over R ⪰ 1 communication rounds must output x AR such that,
[Equation]
The proof of this theorem follows the known oracle complexity lower bounds [31, 32] and is includedin
Appendix B. This theorem shows that when heterogeneity is large i.e., τ ≈ L , mini-batch
SARAH/STORM already achieve the optimal communication and oracle complexity (see Table 1).
Therefore, most existing methods including F ED A VG [16], S CAFFOLD [18] and F ED P AGE [19] cannot outperform these centralized baselines as their analyses do not benefit from small heterogeneity.
At the same time, this highlights the limitation of the centralized baselines, which also do not improvewith lower heterogeneity.
Certain existing local-update algorithms such as M IME MVR [21] and
BVR-L-SGD
[22] can indeed improve upon centralized algorithms in the low heterogeneity regime.
In the next section, we quantify this improvement, and show that our algorithm strictly dominates thecentralized baselines and almost matches our lower bound for algorithms in A ZR .
Our Algorithm and Min-max Optimality
In this section we present our communication-efficient algorithm abbreviated CE-LSGD andillustrate it in Algorithm 1.
Note that for m ∈ [ M ] , we use the notation ∇ F m, B m ( x ) :
= 103 (cid:80) l ∈B m g ( x ; z l ∼ D m ) / |B m | to denote the stochastic mini-batch gradient obtained by querying
O 2 ,L,σF m for |B m | many times, and ∇ F m,z ( x ) to denote a single stochastic gradient g ( x ; z ∼ D m ) .
At each iteration of Algorithm 1, we need two rounds of communication, i.e., two back andforth communications between the server and all clients.
Our method uses the extra round ofcommunication, i.e., line 4 to line 9, to update the variance reduced gradient v r using the current andprevious server models, x r , x r − 1 respectively.
In the following discussion, we will use the iterationnumber R and communication complexity of Algorithm 1 interchangeably.
input Initialization x 0 , communication round R , local steps K , step size η , momentum β ∈ [0 , 1] , initial batch size b1: Let x − 1 =
x2: for r = 0 , 1 , . . .
, R − 1 do 3: if r = 0 set β = 1 , K = 1 , B = b 0 else set β, K as the input values, B = K 4:
Communicate (send) ( x r , x r − 1 ) to clients 5: on client m ∈
[ M ] do 6:
Compute ∇ F m, B mr ( x r ) and ∇ F m, B mr ( x r − 1 ) , where |B mr | =
B 7:
Communicate (rec) (cid:0) ∇ F m, B mr ( x r ) , ∇ F m, B mr ( x r − 1 ) (cid:1) to the server 8: end on client 9: v r = 1 M (cid:80) Mm =1 ∇ F m, B mr ( x r ) + (1 − β ) (cid:16) v r − 1 − 1 M (cid:80)
Mm =1 ∇ F m, B mr ( x r − 1 ) (cid:17) 10:
Communicate (send) ( x r , v r ) to client (cid:101) m r , where (cid:101) m r ∼ Unif ([ M ]) 11: on client (cid:101) m do 12: w (cid:101) m r r +1 , 1 := w (cid:101) m r r +1 , 0 := x r , v (cid:101) m r r, 0 :
= v r 13: for k = 1 , . . .
, K do 14: v (cid:101) m r r,k = ∇ F (cid:101) m r ,z (cid:102) mrr,k ( w (cid:101) m r r +1 ,k ) + v (cid:101) m r r,k − 1 − ∇ F (cid:101) m r ,z (cid:102) mrr,k ( w (cid:101) m r r
+1 ,k − 1 ) 15: w (cid:101) m r r
+1 ,
k +1 =
w (cid:101) m r r
+1 ,k − ηv (cid:101) m r r,k 16: end for 17:
Communicate (rec) (cid:0) w (cid:101) m r r +1 ,K +1 (cid:1) to the server 18: end on client 19:
Let x r +1 = w (cid:101) m r r
+1 ,K + 20: end for output Choose (cid:101) x
uniformly from { w (cid:101) m r r,k } r ∈
[ R ] ,k ∈
[ K ]
At the core of our proposed method is the variance reduction term v r and the local gradient estimator v mr,k (lines 9 and 14 in Algorithm 1).
The construction of the local gradient estimator ismotivated by the variance reduction technique of SARAH [24, 25].
Intuitively, the estimationerror between the proposed local gradient estimator v mr,k and the full gradient ∇ F ( w mr +1 ,k ) , i.e.,
E ∥ v mr,k − ∇ F ( w mr +1 ,k ) ∥ , can be decomposed into two dominating terms: E ∥ v r − ∇ F ( x r ) ∥ 2 andτ 2 K (cid:80)
Kk =1 E ∥
w mr +1 ,k − w mr +1 ,k − 1 ∥ 2 .
The first term is the estimation error between the variancereduction term v r and the full gradient ∇ F ( x r ) .
Since v r is updated based on the momentum-basedvariance reduction technique [26], this estimation error is dominated by L 2 E ∥ x r − x r − 1 ∥ 2 , whichapproaches zero as the algorithm converges.
Similarly, the second term E ∥
w mr +1 ,k − w mr +1 ,k − 1 ∥ 2approaches zero as the algorithm converges and the τ factor controls the benefit we can obtain fromsmall heterogeneity.
Intuitively, for smaller values of τ , we can make more local updates and thealgorithm converges faster.
We note that, our method reduces to mini-batch STORM if we choose thenumber of local updates K to be one (see Appendix D).
Our formal convergence result is as follows:
Theorem 3.1 (Simplified Statement) .
Suppose { F m } m ∈
[ M ] ∈ F 2 M ( L, ∆ , τ ) for L, ∆ , τ ≥ 0 , τ ≤ 2 Lthen,
(a) if each client m ∈
[ M ] has a stochastic oracle
O 2 ,L,σF m , and assuming ∆ LR ⪯ [Equation] 2 √ MK , then the output (cid:101) x of Algorithm 1 using β = max (cid:110) 1 R , (∆ L ) 2 / 3 ( MKb ) 1 / 3 σ 4 / 3 R 2 /(cid:111) , b
[Equation]
KR , and η = min (cid:110) 1 L , 1 Kτ , ( βM ) 1 / 2 LK 1 / 2 (cid:111) satisfies,
[Equation]
(b) if each client m ∈
[ M ] has a deterministic oracle
O 2 ,L, 0 F m , then the output (cid:101) x of Algorithm 1 using β = 1 and η = min (cid:8) 1 L , 1 Kτ (cid:9) satisfies,
[Equation]
In Appendix C, we derive this result by carefully tuning β, b 0 .
We show that the convergence rateattained by our algorithm is almost optimal by proving the following lower bound result.
Theorem 3.2.
For all L, σ, τ, ∆ , ζ ≥ 0 , τ ≤ 2 L, ζ ≤ √ ∆ L , every algorithm
A ∈
A zr , optimizing a problem in F 1 M ( L, ∆ , ζ ) ∪ F 2 M ( L, ∆ , τ ) with K > 0 intermittent accesses to two-point first-order oracles {O 2 ,L,σF m } m ∈
[ M ] on all the machines, outputs x AR after R ≥ c 1 rounds such that,
[Equation]
MKR + (cid:18) σ ∆ L
MKR (cid:19) 2 /.
We can make two observations by comparing the upper and lower bounds for problems in
F 2 M ( L, ∆ , τ ) .
First, in the deterministic setting ( σ = 0 ), our upper bound matches the lowerbound; hence CE-LSGD is min-max optimal .
Thus, our result improves over all the existing resultsin this setting, including M IME MVR [21].
Second, in the stochastic setting ( σ > 0 ), our algorithm’supper bound is optimal except for the second term in Theorem 3.1, which has a ∆ L/ ( √ KR ) factoras opposed to the ∆ L/ ( KR ) term in the lower bound.
In our analysis in Appendix C we highlightwhat causes this gap.
We conjecture that the lower bound is tight, and like the deterministic setting,our algorithm is also min-max optimal in the stochastic setting.
Our construction for Theorem 3.2 uses the non-convex hard instance proposed by Carmon et al.[31]and splits it across different machines to get a communication complexity lower bound.
This ideahas been used previously to give lower bounds in the heterogeneous setting [33, 15, 34].
We provethe result in Appendix B.
From looking at Table 1, we can note that BVR-L-SGD [22] also attainsa similar upper bound as our method.
In Appendix C, we show that with deterministic oracle
BVR-L-SGD also attains the min-max optimal rate.
This is not surprising, knowing that severalvariance-reduced algorithms [25, 26, 24] are simultaneously optimal even in the sequential setting.
Still, our method requires fewer and lighter variance reduction operations, which leads to betterscalability from the algorithmic design perspective.
In the next section, we carefully examine thedifference between these methods.
3.1 The Perspective of Reducing Communication
So far, we have looked at convergence rates in the intermittent communication model, where K, Ris fixed.
However, another perspective is reducing the communication complexity to the minimumpossible with the minimum required oracle complexity.
Both these complexities can be expressed interms of ϵ using the convergence guarantees we showed, where we want to attain an ϵ -approximatestationary point.
This view is often more useful when communication rounds comprise the bulk of therequired physical time.
This scenario is common in FL, where devices become available intermittently,which delays the synchronous updates.
With this in mind, we summarize the communication andoracle complexities attained by both our method and BVR-L-SGD [22] in Figure 1 when optimizingwith stochastic oracles.
Notice that the figure has three different regimes based on the relative scalingof τ versus ϵ .
We focus on the green regime characterized by ϵ 1 / 2 ∈ (0 , τσ/ ( LM )] .
This regime isof most practical interest in deep learning, where modern over-parameterized models often drive ϵ toreally small values.
And when ϵ is small enough, this regime covers a wide range of values of τ . 2 .
We use M = 10 machines, and synthetically generate heterogeneous data-sets (see Section 4) with q = 0 . 1 .
All oracle queries use a mini-batch of size b = 16 , i.e., each machine has Kb oracle queries between two communication rounds.
We note that our method has a faster convergence in all the settings, which highlights its communication efficiency.
Fixed step-sizes η for both the methods were tuned in { 0 .
5 } (to obtain best loss) following [22], our method set the momentum β = 0 .
3 , b ourmax =
K , while b BV R
In the green regime, both CE-LSGD and BVR-L-SGD require K = σL/ ( τMϵ 1 / 2 ) local stepsto achieve the optimal communication and oracle complexities.
However, BVR-L-SGD requiresmultiple heavy-batch stochastic gradient computations on each machine with batch size b max .
Inparticular, for BVR-L-SGD, we have ρ BVR = b max /K
=
στ/ ( Lϵ 1 / 2 ) , which suggests that for
S =
L ∆ /σ √ ϵ communication rounds, it requires each machine to compute ρ BVR times heavierbatch stochastic gradients compared to the other communication rounds.
As for CE-LSGD, we haveb
= σ 3 / ( L ∆ Mϵ 1 / 2 ) , which gives us ρ our = b 0 /K
= σ 2 τ/ ( L 2 ∆) .
This suggests that our methodonly requires each machine to compute ρ our times larger batch stochastic gradient, and that too onlyonce.
Furthermore, ρ our /ρ BVR = σϵ 1 / 2 / ( L ∆) ≤ 1 .
Thus, the size of our large batch gradient isalso smaller than the one for BVR-L-SGD and our method has fewer and lighter heavy-batchoperations .
Suppose one implements both these methods in the intermittent communication model, i.e., bybreaking the large batch computation across multiple rounds, with local budget K =
σL/ ( τMϵ 1 / 2 ) .
In that case, the effective communication complexity of both methods is ∆ τ/ϵ , and this subtledifference gets washed away.
However, in Figure 2, we show that this equivalence up to numericalconstants doesn’t hold in practice, where our method converges faster than BVR-L-SGD.
In Table 2,we summarize the communication and oracle complexities attained by different algorithms in thegreen regime.
3.2 The Partial Participation Setting
In settings such as cross-device federated learning [17], there are often tens of thousands of clients(think of android mobile users) and it is not feasible to consider training on all of the clientssynchronously.
It is more natural to consider a partial sampling of clients for each communicationround.
More formally, we can re-state our distributed optimization problem as follows:
[Equation]
where P is a probability distribution on the clients, and we assume at each communication round, wecan sample m clients independently from P .
This is w.l.o.g. because in settings like cross-device
FL, there is an extremely large number of devices, and most devices participate only once.
We alsoneed to modify the IC setting: during each communication round, S r ∼ P m clients participate, andeach queries their oracle K times.
This setting has been considered in several works [18, 21].
Wecan adapt Algorithm 1 to this setting by communicating with only m clients at each round and usingm 0 clients for the first round to initialize the variance reduction term (see Appendix E).
Specifically,we can guarantee the following convergence rate where F 1 P ( L, ∆ , ζ ) and F 2 P ( L, ∆ , τ ) are naturalgeneralizations of F 1 M ( L, ∆ , ζ ) and F 2 M ( L, ∆ , τ ) to the partial participation setting, formally definedin Appendix A.
Theorem 3.3.
Suppose for all n in support of P , F n ∈ F 1 P ( L, ∆ , ζ ) ∩ F 2 P ( L, ∆ , τ ) then,(a) if each client n has a stochastic oracle O 2 ,L,σF n , and assuming that ∆ τR + ∆ L √ KR ⪯ σ 2 √ mK + ζ 2 √ m , the output (cid:101) x of Algorithm 1 using b 0 = K , m 0 = mR , β = max (cid:26)
[Equation]
(cid:17)
[Equation]
and
[Equation]
(b) if each client n has a deterministic oracle O 2 ,L, 0 F n , and assuming that ∆ τR ⪯ ζ 2 √ m , then the output (cid:101) x of Algorithm 1 using m
=
mR , β = max
[Equation]
satisfies
[Equation]
As far as we know, CE-LSGD is the only one in the partial participation setting that uses juststochastic oracles for implementing variance reduction with local updates.
In tables 2 and 1 weshow that with a deterministic oracle, CE-LGD attains a strictly faster convergence rate than thebest-known result M IME MVR [21], which also uses a deterministic oracle.
More specifically, CELGD’s communication complexity ζ ∆ τ/mϵ 3 / 2 improves over the communication complexity
ofζ ∆ τ/ √ mϵ 3 / 2 for M IME -MVR.
Furthermore, we have the following lower bound result to show thatthe convergence rate of our algorithm is almost optimal .
Theorem 3.4.
For all L, σ, τ, ∆ , ζ ≥ 0 , τ ≤ 2 L, ζ ≤ √ ∆ L every algorithm A ∈
A zr optimizing a problem in F 1 P ( L, ∆ , ζ ) ∪ F 2 P ( L, ∆ , τ ) with K > 0 intermittent accesses to two-point first-order oracles {O 2 ,L,σF n } n ∈ support ( P ) on all the machines outputs x AR after R ⪰ 1 rounds such that,
[Equation]
Note that in the deterministic setting (i.e., σ = 0 ), the only gap is in the last term of CE-LGD’supper bound.
We conjecture that CE-LGD is optimal even in the partial partipation setting,and our lower bound can be improved.
There are more gaps in the stochastic setting, and we hopeto remove them in future work.
Numerical Experiments
use b = 16 for each oracle query.
Thus each method makes Kb oracle queries every round per machine.
All the methods for different q are tuned separately, following a similar hyper-parameter search as in Figure 2.
We evaluate the performance of our method byoptimizing a two-layer fully connected networkfor multi-class classification on the CIFAR-
[35] data-set.
Since we are in the heterogeneoussetting, we need to artificially generate a data-set.
We follow the same data processing procedure asin [22].
We first make sure that all the ten classesin CIFAR-10 have the same number of samples(roughly around 5000), and assign q × 100% ofclass m ’s samples to client m ∈
[10] where qis chosen from { 0 . 1 , 0 . 35 , 0 . 6 , 0 . 85 } .
For eachclass m , we evenly split the remaining (1 −q ) × 100% samples to the other 9 clients exceptclient m .
Thus, q controls the heterogeneity ofour data-set, with small q corresponding to smallheterogeneity.
We perform two different experiments.
Inthe first experiment, we directly compare ourmethod, i.e., CE-LSGD, with BVR-L-SGDin the intermittent communication setting (see
Figure 2).
We observe that while both the methods converge to a similar quality of solutioneventually, our method is more communicationefficient.
In the second experiment, we compareour method with BVR-L-SGD [22] as well as
F ED A VG [16], SCAFFOLD [18], MB-SARAH [24] and MB-SGD [5] for the same number ofupdates/iterations.
The last two methods are centralized baselines, and we use the local computationto compute a mini-batch stochastic gradient.
We again observe that CE-LSGD and BVR-L-SGDhave comparable performance which is better than all the other methods.
Discussion and Open Problems
In this paper, we provide a new communication-efficient local update algorithm CE-LSGD andanalyze it in the full and partial client participation setting with intermittent communication.
Inthe deterministic setting, i.e., with access to exact oracles, our algorithm is the optimal and almostoptimal for full and partial participation respectively.
Moreover, when equipped with stochasticoracles our algorithm attains the best-known convergence guarantees to our knowledge in both modelsof participation.
Our lower bound results provide a much-needed baseline to measure algorithmicdevelopments in non-convex distributed optimization and help us characterize CE-LGD’s optimality.
In Appendix F, we provide an extension of CE-LSGD which uses a stochastic Hessian vectorproduct oracle [12, 36] instead of a multi-point oracle, and recovers similar optimal communicationcomplexity.
This is relevant for memory-constrained online settings where it might not be feasible topreserve several copies of a model on the client device for making simultaneous queries for variancereduction algorithms.
Our work leaves some open questions.
In the deterministic partial participation setting, is our lowerbound loose?
We believe this is most likely the situation.
In the stochastic full participation setting,does CE-LSGD attain the min-max optimal rate?
There is a 1 / √ K gap in our optimization term forany model of participation which vanishes in the deterministic setting.
Thus we don’t know if the gapis "real" or a result of our analysis.
We discuss this further in appendix C. Finally there are severalgaps in the stochastic partial participation setting.
We believe some of these can be alleviated byimproving the deterministic lower bound, but others seem imply that our analysis is loose.
Finally wedon’t know if we can use a simple single machine baseline in the red regime in Figure 1?
We are notable to show this with only assumptions on τ .
However, in Appendix C, we discuss this regime whenζ is also small and show how a single machine baseline can be used here.
In general, it is unclear ifthe set of problems in F 1 M ( L, ∆ , ζ ) are indeed simpler than F 2 M ( L, ∆ , τ ) .
We hope to tackle thesequestions in our future work.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
In the appendix
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]