Optimal Gradient Sliding and its Application to Optimal Distributed Optimization Under Similarity
Abstract
We study structured convex optimization problems, with additive objective r :=p + q , where r is ( µ -strongly) convex, q is L q -smooth and convex, and p is L psmooth but possibly nonconvex.
For such class of problems, we proposed an inexactaccelerated gradient sliding method that can skip the gradient computation for oneof these components while still achieving optimal complexity of gradient callsof p and q , that is, O ( (cid:112)
L p /µ and O ( (cid:112) L q /µ ) , respectively.
This result is muchsharper than the classic black-box complexity O ( (cid:112) ( L p + L q ) /µ , especially whenthe difference between L q and L q is large.
We then apply the proposed method tosolve distributed optimization problems over master-worker architectures, underagents’ function similarity, due to statistical data similarity or otherwise.
Thedistributed algorithm achieves for the first time lower complexity bounds on bothcommunication and local gradient calls, with the former having being a longstanding open problem.
Finally the method is extended to distributed saddleproblems (under function similarity) by means of solving a class of variationalinequalities, achieving lower communication and computation complexity bounds.
Introduction
We consider structured convex programming in the form [6, 13, 34]:
[Equation]
where r is assumed to be convex and decomposed as the sum of a smooth, possibly nonconvexfunction p and a smooth convex function q .
First order information of p and q is accessible separately.
We are interested in scenarios where the cost of evaluating the gradient of the two functions is noteven, but computing, say ∇ p , is much more resource demanding than ∇ q .
The motivating applicationfor this scenario is distributed optimization over master-worker systems, as discussed next.
Consider the following distributed optimization problem over a network of m agents:
[Equation]
where f i is the loss function of agent i , assumed to be convex, which is not known to other agents.
Agents are embedded in a star-topology, with agent 1 being the master node, without loss ofgenerality–this is the typical federated learning setup [23].
An instance of (2) of particular interestis the empirical risk minimization (ERM) whereby the goal is to minimize the average loss oversome dataset, distributed across the nodes of the network, with f i being the empirical risk of agent i ,i.e., f i ( x ) = 1 m (cid:80) mj
=1 ℓ (cid:0) x ; z ji (cid:1) , where z (1) i , . . .
, z ( m ) i is the set of m samples owned by agent i , andℓ ( x ; z ji ) measures the mismatch between the parameter x and the sample z ji .
Several solutions methods have been proposed to solve (2); the prototype approach consists ininterleaving local computations at the workers sides (nodes i = 1 , . . .
n ) with communication to/fromthe master node ( i = 1 ), which maintains and updates the authoritative copy of the optimizationvariables, producing eventually the final solution estimate.
Since the cost of communications is oftenthe bottleneck in distributed computing (e.g., [7, 31]), a lot of research has been devoted to designingdistributed algorithms that are communication efficient .
Acceleration (in the sense of Nesterov) hasbeen extensively investigated as a procedure to reduce the communication burden.
For L -smoothand µ -strongly convex functions r in (2), linear convergence is certified by employing first-ordermethods, with computation (gradient evaluations) and communication complexities proportionalto √ κ ( κ ≜ L/µ is the condition number of r ).
For ill-conditioned functions ( κ very large), thepolynomial dependence on κ may be unsatisfactory.
This is, e.g., the typical setting of many ERMproblems wherein the optimal regularization parameter for test predictive performance is very small.
Further improvements on the communication complexity can be obtained exploiting the extra structuretypical in ERM problems, also known as function similarity (see, e.g., [4, [-51, 44, 48]): kr 2 f i ( x ) r 
-]49, 42, 46]: ∥∇
2 f i ( x ) −∇ 2 f j ∥ ≤ δ , for all x in a proper domain of interest and all i ̸ = j = 1 , . . . , n , where δ > 0 measuresthe degree of similarity between the Hessian matrices of the local losses.
When data are i.i.d.among agents, f i ’s reflect statistical similarities in local data, resulting in δ =
˜ O
(1 / √ m ) with highprobability ( ˜ O hides log-factors and dependence on d ).
In this scenario, in general, 1 + δ/µ ≪ κ [4].
This motivated a surge of studies aiming at exploiting function similarity coupled with acceleration toboost communication efficiency (see Sec.
1.2 for an overview of relevant works): linear convergenceis certified with a number of communication steps scaling with (cid:112) δ/µ [4], which matches lower(communication) complexity bounds [4].
These methods however are not computationally optimal,yielding to local gradients calls larger than lower complexity bounds O ( √ κ ) .
In fact, Table 1 showsthat, to the date, there exists no distributed algorithm achieving the best of the two worlds, that is,optimal (lower bound) communication complexity and local gradient (oracle) complexity.
This paper fills this gap.
Our starting point is the reformulation of (2) in the equivalent form
[Equation]
which exploits function similarity at the agents’ side via preconditioning.
Problem (3) is an instance of(1): all f i (thus q ) are convex but p is nonconvex.
Also, evaluating ∇ q and ∇ p has different costs; theformer involves only local computations at the master node while the latter requires communicationsfrom/to master and workers nodes.
At high-level the idea is then clear: one would like to designa distributed algorithm for (3) [or more generally for (1)] that skips gradient computations of ∇ p(saving thus communications) without slowing down the overall optimal rate of convergence.
This naturally suggests the use of gradient-sliding techniques [19, 39, 10], yielding algorithmsthat skip from time to time computation of the gradient of one function in the summand objective.
However, existing gradient-sliding algorithms are not applicable to (1) [and thus (3)] because they allrequire p and q to be convex .
This calls for new designs, accounting for the nonconvexity of p .
1.1 Main contributions
Our contribution is threefold:
• A new gradient-sliding algorithm for (1) : We propose a new Accelerated ExtraGradient sliding method that skips the computation of ∇ p from time to time.
The method builds on an inexactacceleration of a proximal envelop (outer-loop) coupled with a suitable termination criterion andinner-loop algorithm to approximately solve the proximal subproblem.
When applied to (1), withr being ( µ -strongly) convex, q being L q -smooth and convex, and p being L p -smooth,the proposedalgorithm achieves the following optimal complexity of gradient calls of q and p :
[Table]
Notice
that the above complexity bounds are sharper than the complexity bound obtained by theNesterov’s optimal first-order method for smooth (strongly) convex optimization applied to (1).
For
instance for strongly convex r , that would yield to ∇ q , ∇ p complexity scaling as O ( (cid:112) ( L q + L p ) /µ ) ,which is less favorable than our separate complexity bounds above.
To the best of our knowledge,
this is the first time that such bounds are achieved for nonconvex p .
[Table]
• Optimal complexity bounds for (2) under function similarity: We customize the proposedaccelerated gradient-sliding algorithm to the distributed optimization problem (2) under δ -functionsimilarity.
As showed in Table 1 for strongly convex r (see Sec.
3 for the case of weakly convex r ), thenew distributed algorithm achieves lower complexity bounds on both the number of communications[4] and on the number of gradient computations (without logarithmic factors!)
Achieving
[Table]
optimal communication complexity (for nonquadratic losses) was a long-standing open problem.
[Table]
• Gradient sliding for variational inequalities: We extend the proposed gradient-sliding machinery
[Table]
to solve distributed saddle-points under similarity by means of solving a class of strongly-monotoneVariational Inequalities (VI).
We improve existing complexity bounds for such problems [9] achievingfor the first time both optimal communication complexity and gradient oracle complexity–see Table 1. [Table]
[Table]
1.2 Related works
Gradient-Sliding: Since the seminal paper [27], the idea of gradient-sliding for structured convexoptimization such as (1) has received significant attention as tool to skip gradient computations ofone function in the summand; examples and generalization include first-order accelerated methods[27, 28, 30], zero-order (derivative-free) schemes [14, 22, 8, 41], high-order methods [24, 17, 2, 20],and slidings for saddle point problems and variational inequalities [3, 29, 45, 9].
Albeit applicable tomore general classes of optimization problems than (1) (e.g., allowing either p and q to be nonsmooth),none of the existing methods provide guarantees when p is nonconvex.
The proposed algorithm fillsthis gap.
Furthermore, it achieves optimal lower complexity bounds on the calls of ∇ p and ∇ q .
Distributed optimization under function similarity: The literature of distributed optimization isvast; given the focus of this work, we comment next solution methods exploiting function similarity via proper preconditioning– Table 1 summarizes complexity results of existing distributed methodssolving either minimization problems or saddle-point formulations, and is commented next.
The seminal paper [4] established lower communication complexity bounds for (2) under δ -similarity:ε -optimality cannot be achieve in less than Ω( (cid:112) δ/µ log 1 /ε ) communication rounds.
Since then, alot of effort has been devoted to design distributed schemes employing some form of preconditioningaiming at achieving optimal communication complexity.
The authors in [40] proposed DANE , amirror-descent based algorithm whereby workers perform a local data preconditioning via a suitablychosen Bregman divergence, and the master averages the solutions of the workers.
For quadraticlosses, DANE achieves communication complexity (cid:101) O
(( δ/µ ) 2 log 1 /ε ) ; complexity for nonquadraticlosses improved to O (( δ/µ ) log 1 /ε ) was later certified in [43] for the SONATA algorithm (alsoimplementable over mesh-networks).
Improvements were achieved employing acceleration; efforts include: DiSCO [49], an inexact damped
Newton method coupled with a preconditioned conjugate gradient (to compute the Newton direction),which achieves communication complexity (cid:101) O
( (cid:112) δ/µ ) log 1 /ε ) for self-concordant losses (see Table 1for the log-factors hidden in the (cid:101) O ); AIDE [38], which uses the Catalyst framework [32], matching therate of DiSCO for quadratic losses (and r = 0 ); DANE-HB [48], a variant of DANE equipped with Heavy Ball momentum and matching for quadratic functions the communication complexity of DiSCO and
AIDE ; and SPAG [21], a preconditioned direct accelerated method, achieving for nonquadradic lossesasymptotically the convergence rate O ((1 − 1 / (cid:112) β/µ ) k )
( k is the iteration index)–the worst-caserate is still O ( (cid:112) L/µ ) log 1 /ε ) .
Finally, higher order methods employing preconditioning have been studied in [12, 1, 44]: [12]proposed DiRegINA, a decentralization of the cubic regularization of the Newton method, whereworkers build Newton direction sampling local Hessians; [1] introduced ACN , an inexact acceleratedcubic-regularized Newton’s method, with improved complexity with respect to [12]; and [44] extendedthe Catalyst framework [32] to the distributed setting (including mesh networks), proposing Acc
SONATA –the communication complexity of these methods is reported in Table 1.
In summary, the above tour on the relevant literature shows that none of the existing methods canmatch lower communication complexity bounds for (non quadratic) optimization problems (2) underfunction similarity (all complexity bounds contain log-factors).
The proposed distributed methodachieves lower communication and computation complexity bounds.
Optimal Gradient Sliding for Minimization Problems
We study the minimization problem (1), under the following blanket assumptions.
Assumption 1.
r ( x ): R d → R is µ -strongly convex on R d .
Assumption 2.
q ( x ): R d → R is convex and L q -smooth on R d .
Assumption 3.
p ( x ): R d → R is L p -smooth on R d .
The proposed Accelerated ExtraGradient sliding is formally introduced in Algorithm 1.
Convergenceof the outer loop is established in Theorem 1, while Theorem 2 determines complexity of solving theinner loop up to a suitable termination.
Finally Theorem 3 provides the overall complexity merginginner and outer loop results.
The proof of all the theorems can be found in Appendix A.1.
Algorithm 1: Input: x 0 = x 0 f ∈ R d 2: Parameters: τ ∈ (0 , 1) , η, θ, α > 0 , K ∈ { 1 , 2 , . . .
} 3: for k = 0 , 1 , 2 , . . .
, K − 1 do 4: x kg =
τx k + (1 − τ ) x kf 5: x k +1 f ≈ arg min x ∈ R d (cid:2) A k θ ( x ) :
= p ( x k g ) + ⟨∇ p (
x kg ) , x − x k g ⟩ + 1 2 θ ∥ x − x k g ∥ 2 + q ( x ) (cid:3) 6: x k +
= x k + ηα ( x k +1 f − x k ) − η ∇ r ( x k +1 f ) 7: end for 8: Output: x K
Theorem 1.
Consider Algorithm 1, under the following tuning:
[Equation]
and let x k +1 f in line 5 satisfy
[Equation]
Then, for any
[Equation]
we have the following estimate for the distance to the solution x ∗ :
[Equation]
2.1 Solving the auxiliary subproblems
At each iteration of Algorithm 1, one needs to solve the subproblem:
[Equation]
According to Theorem 1, (7) need not be solved to arbitrary precision; inexact solutions x k +1 fsatisfying condition (4) suffice.
Condition (4) means that gradient norm ∥∇ A kθ ( x k +1 f ) ∥ should besufficiently small.
Notice that A kθ ( x ) in (7) is (2 L p + L q ) -smooth and convex.
Problem 7 can be solved up to the termination (4) using any of the algorithms in [26, 25, 36].
Weobtain the following complexity.
Theorem 2 ([36] Remark 1) .
There exists a certain algorithm such that, when applied to problem (7)with the starting point x kg , returns x k +1 f satisfying
[Equation]
where D > 0 is some universal constant (independent of L p , L q , T etc) and T is the number of callsof ∇ q by the algorithm.
2.2 Overall complexity of the optimal gradient-sliding
Theorem 2 suggests that, to satisfy condition (4) in Theorem 1, it is sufficient to choose the number
T of iterations of the inner algorithm as
[Equation]
We can now determine the overal complexity of Algorithm 1.
At each iteration of Algorithm 1 wecall ∇ p twice (at x kg – line 5 and at x k +1 f – line 6), and ∇ q is computed T + 1 times ( T times in theauxiliary problem – line 5, and at x k +1 f – line 6).
Hence, to find an ε -solution of problem (1), i.e., tofind x K ∈ R d that satisfies (6), Algorithm 1 requires K iterations as given in (5), [Equation] K
[Equation]
O
[Equation] [Equation] log [Equation]
ϵ (cid:17) calls of ∇ p ( x ) , and
[Equation]
Putting everything together we obtain the following final convergence result.
Theorem 3.
Consider Problem (1) under Assumptions 1 to 3, with µ ≤ L p ≤ L q , without loss ofgenerality.
Then, to reach an ε -solution, Algorithm 1 requires
O (cid:16)(cid:113)
L q µ log 1 ϵ (cid:17) calls of ∇ q ( x ) and O (cid:16)(cid:113) L p [Table]
This matches optimal complexity for the individual gradient calls.
We conclude this section providing convergence of variant of the proposed algorithm suitable forweakly convex r in (1) (Assumption 1 with µ = 0 ).
The algorithm is described in Appendix A.2.
Here we only provide the final convergence result, the analogous of Theorem 3.
Theorem 4.
Consider Problem (1) under Assumptions 1 (with µ = 0 )
-3, with L p ≤ L q .
Then, tofind an ε -solution of (1) (in objective value), Algorithm 3 in Appendix A.2 requires
O (cid:18)(cid:113)
L q ε ∥ x 0 − x ∗ ∥ (cid:19) calls of ∇ q ( x ) and O (cid:18)(cid:113)
L p ε ∥ x 0 − x ∗ ∥ (cid:19) calls of ∇ p ( x ) .
Application to Distributed Optimization Under Similarity
In this section, we apply the proposed algorithm the the distributed optimization problem (2), underthe following assumptions.
Assumption 4.
Each f i ( x ): R d → R is convex and L -smooth.
Assumption 5.
r ( x ): R d → R is µ -strongly convex.
Assumption 6.
f 1 ( x ) . .
. , f n ( x ) are δ -related:
∥∇ 2 f i ( x ) − ∇ 2 f ( x ) ∥ ≤ δ , for all i and x ∈ R d ,and some δ > 0 .
[-From the last assumption it is easy to get that for all i and x 2 R d we have kr 
-]
We leverage now Algorithm 1 to solve (2), using the equivalent reformulation (3).
The algorithmapplied to the distributed system can be described as follows.
The server computes x kg and sendsit to all the workers (line 4).
Workers compute ∇ f i ( x kg ) and send it to the server.
After collectingall ∇ f i ( x kg ) , the server builds ∇ p ( x kg ) = ∇ f ( x kg ) − ∇ f 1 ( x kg ) , and then solves (inexactly) the localproblem A kθ (line 5).
The inexact solution x k +1 f is then broadcast to the workers, which update theirown receives ∇ f i ( x k +1 f ) and send back to the server, which can then evaluate ∇ r ( x k +1 f ) (line 6).
Using Assumptions 4 and 5, we infer that r is µ -strongly convex; and q = f 1 is L q -smooth andconvex, with L q = L .
It follows from Assumption 6 that ∥∇ 2 p ∥ ≤ δ .
Therefore, p has L p -Lipschitzgradient, with L p = δ .
This shows that we can leverage Theorems 3 and 4 to establish convergencefor strongly convex and weakly convex r , as given next.
Theorem 5.
Let Assumptions 4 to 6 be satisfied with µ ≤ δ ≤ L .
Then, to find ε -solution of the distributed optimization problem (2) Algorithm 1 requires
[Equation]
Theorem 6.
Let Assumptions 4, 5 (with µ = 0 ), 6 be satisfied and δ ≤ L .
Then, to find ε -solution of the distributed optimization problem (2) Algorithm 1 requires
[Equation]
Such estimates are optimal from both communications [4] and local computations point of views [35].
It is important to remark that Algorithm 1 solves the local subproblem
A θ with some precision, whilemost of existing works (see Table 1) assume that local problems are solved with infinite precision(column Local gradient complexity), which is not practical.
Note also that the subproblems in line 5of
Algorithm 1 do not necessarily have to be solved by a deterministic algorithm as in Theorem 2.
Stochastic methods can also be used, as long as they guarantee that condition (4) is met.
Optimal Gradient Sliding for VIs
In this section we consider the composite variational inequality [16, 5] in the form:
[-Find x ⇤ 2 R d : R ( x ⇤ ) = 
 with R ( x ) :-][Equation]
[Equation] 10) where Q ( x ) , P ( x ): R d → R d .
Variation inequalities are a unified umbrella for a variety of problems–two examples follow.
Example 1 [Minimization].
Consider problem (1), choose Q ( x ) = ∇ q ( x ) and P ( x )
= ∇ p ( x ) .
Then the solution of the variational inequality (10) means that we need to find the point x ∗ where theoperator R ( x ∗ )
= ∇ r ( x ∗ ) .
For the convex function r , this is equivalent to finding the minimum.
Example
[Saddle point problems].
Consider the convex-concave saddle point problem
[Equation]
If we take Q ( x ) :
[Equation]
x
[Equation]
[ ∇ y p ( y, z ) , −∇ z p ( y, z )] , then it can be proved that x ∗ = ( y ∗ , z ∗ ) is a solution for (10) if and only if x ∗ =
( y ∗ , z ∗ ) is a solution for (11), i.e.
[Equation]
While minimization problems are widely considered separately from variational inequalities, saddlepoint problems are often analysed under the VI lens.
In recent years the popularity of saddles hasgrown, this is due to the fact that they have both classical [15] and new ML [18, 33] applications.
We study problem (10) under the following assumptions.
Assumption 7.
R ( x ) is µ -strongly monotone: ⟨ R ( x 1 ) − R ( x 2 ) , x 1 − x 2 ⟩ ≥ µ ∥ x 1 − x 2 ∥ 2 , for allx 1 , x 2 ∈ R d .
Assumption 8.
Q ( x ) is monotone and L q -Lipschitz: ⟨ Q ( x 1 ) − Q ( x 2 ) , x 1 − x 2 ⟩ ≥ 0 and ∥ Q ( x 1 ) −
Q ( x 2 ) ∥ ≤ L q ∥ x 1 − x 2 ∥ , for all x 1 , x 2 ∈ R d .
Assumption 9.
P ( x ) is L p -Lipschitz:
∥ P ( x 1 ) − P ( x 2 ) ∥
≤ L p ∥ x 1 − x 2 ∥ for all x 1 , x 2 ∈ R d .
For saddle point problems these assumptions are equivalent to (strong) convexity–(strong)
concavityand Lipschitzness of gradients.
4.1 Sliding via Extragradient
This algorithm is a non-accelerated version of Algorithm 1.
A similar non-accelerated sliding is usedin [9].
Our version however has better theoretical and practical guarantees because of the effectivestopping criterion (12).
Convergence of the outer loop is established in Theorem 7; Theorem 8establishes convergence of the inner loop up to the required termination; and finally Theorem 9combine the two-loop complexity.
The proofs of the theorems can be found in Appendix B.1.
Algorithm u
≈ u
˜ u k is a solution for
Find ˜ u
k ∈ R d : B kθ (˜ u = 0 with B kθ ( x
= P ( k ) + Q ( x + 1 θ − end
Theorem 7.
Consider Algorithm 2 for Problem (10) in the setting above, under the following tuning
[Equation]
Assume that u k (line 4) satisfies
[Equation]
Then, for any
[Equation]
we have the following estimate for the distance to the solution x ∗ :
[Equation]
The proof is given in Appendix B.1.
4.2 Solving the auxiliary problem
As for Algorithm 1, we need an auxiliary solver for the subproblem in line 4, which ensures that (12)is met.
One can observe that B kθ ( x ) is (2 L p + L q ) -Lipschitz and monotone.
For this type of problem,the authors in [47] proposes an approach that guarantees convergence on ∥ B kθ ( u k ) ∥ 2 .
Theorem 8 ([47] Corollary 2) .
There exists a certain algorithm that, applied to the subproblem (12)with starting point x k , returns u k satisfying
[Equation]
(15) where D > 0 is some universal numerical constant (independent of L p , L q , T etc) and T is thenumber of calls of the function Q .
4.3 Complexity of the optimal gradient sliding
Leveraging Theorems 7 and 8 while following the same reasoning as in Section 2.2, we obtain thefollowing convergence (inner plus outer loops) for Algorithm 2. obtain the following theorem.
Theorem 9.
Let Assumptions 7 to 9 be satisfied with µ ≤ L p ≤ L q .
Then, Algorithm 2 requires
O (cid:16) L q µ log 1 ϵ (cid:17) calls of Q ( x ) and O (cid:16) L q
[Equation]
log 1 ϵ (cid:17) calls of P ( x ) to find an ε -solution of problem (10) .
We also consider the case of monotone VIs (Assumption 7 with µ = 0 ).
For this we modify Algorithm2 as in Appendix B.2 and obtain the following convergence result.
Theorem 10.
Let Assumption 7 (with µ = 0 ), 8, 9 be satisfied and L p ≤ L q .
Then, Algorithm 4,described in Appendix B.2, requires
[Equation]
x
[Equation]
P
[Equation]
x ) to find an ε -solution of the problem (10) .
Here ε -solution is measured by the value of the gap function.
4.4 Application to distributed saddle-point problem under similarity
We apply now Algorithm 2 to solve a distributed saddle-point problem under statistical similarity, asintroduced in [9]:
[Equation]
Assumption 10.
Each f i ( y, z ): R d y × R d z → R is convex-concave and L -smooth on R d y × R d z .
Assumption 11.
f ( z, y ) is µ -strongly convex (first argument)– µ -strongly concave (second argument).
Assumption 12.
f 1 ( z, y ) . . .
, f n ( y, z ) are δ -related: for all i and for all y ∈ R d y and z ∈ R d z {+,∥∇ 
 yy f i ( y, z )+}
{+−∇ 
 yy f ( y, z ) ∥ ≤ δ, ∥∇ 2 yz f i ( y, z ) −∇ 2 yz f ( y, z ) ∥ ≤ δ, ∥∇ 2 zz f i ( y, z ) −∇ 2 zz f+}
( y, z ) ∥
≤ δ. Casting (16) into the VI formulation (10), by taking Q ( x ) =
Q ( y, z ) =
[ ∇ y f 1 ( y, z ) , −∇ z f 1 ( y, z )]and P ( x ) = P ( y, z ) =
[ ∇ y
[ f − f 1 ]( y, z ) , −∇ z
[ f − f 1 ]( y, z )] , we have that Q is monotone and
L -Lipschitz, P is δ -Lipschitz, R is µ -strongly monotone.
Therefore, we can apply Theorems 9 and10 and obtain the following convergence results for Algorithm 2 applied to (16).
Theorem 11.
Let Assumptions 10 to 12 be satisfied with µ ≤ δ ≤ L .
Then, to find ε -solution of the distributed saddle problem (16) , Algorithm 2 requires
[Equation]
1 ϵ (cid:17) communication rounds and O (cid:16)
Lµ log 1 ϵ (cid:17) local gradient computations.
Theorem 12.
Let Assumptions 10, 11 (with µ = 0 ), 12 be satisfied and δ ≤ L .
Then, to find ε -solution of the distributed saddle problem (16) , Algorithm 2 requires
O (cid:0) δε ∥ x 0 − x ∗ ∥ 2 (cid:1) communication rounds and O (cid:0) Lε ∥ x 0 − x ∗ ∥ 2 (cid:1) local computations.
Our communication estimates are optimal, as in [9], but our algorithm also achieve optimal localcomplexity (see Table 1).  Experiments
5.1 Minimization
We consider the Linear Regression problem which is defined as
[Equation]
where w are the model weights, { x i , y i }
Ni =1 is the training dataset, λ is the regularization parameter.
We consider a network with 25 workers (simulation on a single CPU) and use the two types ofdatasets: synthetic and real.
Synthetic data allows to control the similarity constant δ .
In more detail,we generate data on the server { ˆ x i , ˆ y i } n =100 i = .
Data set on the other devices is generated by addingunbiased Gaussian noise to the server data.
Thus, the lower the variance of this noise, the moresimilar the data is and the more δ is smaller.
We take the real data from LIBSVM [11]. λ = 0 , 1 . Wetake the methods from the experimental section of [21] as competitors.
See Figure 1 for results. [Figure]
It can be seen that our method strongly overtakes the accelerated gradient descent, DANE, and alsocompare favorably with DANE-HB and SPAG both on communications and in local iterations.  5.2 Saddle point problems
Here we consider a modification of (17), the Robust Linear Regression: [Equation]
[Equation] where r i is the so-called adversarial noise, β is the regularization for it.
The number of agents anddata are used as in Section 5.1.
λ =
β = 0 , 1 and R r = 0 , 05 . The only methods that work for SPPsunder similarity assumption [9] is chosen as competitors. See Figure 2 for results.
[Figure]
It can be seen that our method compares favorably with [9] both on communications and gradientiterations. Checklist For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] (b) Did you describe the limitations of your work? [Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] , theoretical paper
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
a)
Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A] , we work withsmall models and simple optimisers, the experiments are easy to reproduce on any PC
but we will try to make the code user friendly and put it out in the public domain(
b) Did you
specify all the training details (
e
.g.
data
splits, hyperparameters
, how they
were
chosen
[Yes
Section 5 gives information about the problem setup, all
methods
are tuned
how it
is done in
the
original papers
c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[N/A] , methods are deterministic
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] , Appendix C, but we make small experiments, they can be run on any PC If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] , Section
.1, cite
LIBSVM(b)
Did you mention the license of the assets?
[N/A] , open assets(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’reusing/curating?
[N/A](e)
Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
[N/A]5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A](b)
Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A](c)
Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]