Optimal Gradient Sliding and its Application to Distributed Optimization Under Similarity
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We study structured convex optimization problems, with additive objective r :=p + q , where r is ( µ -strongly) convex, q is L q -smooth and convex, and p is L psmooth, possibly nonconvex.
For such a class of problems, we proposed an inexactaccelerated gradient sliding method that can skip the gradient computation for oneof these components while still achieving optimal complexity of gradient calls ofp and q , that is, O ( p
L p /µ ) and O ( p
L q /µ ) , respectively.
This result is muchsharper than the classic black-box complexity O ( p ( L p + L q ) /µ ) , especially whenthe difference between L p and L q is large.
We then apply the proposed method tosolve distributed optimization problems over master-worker architectures, underagents’ function similarity, due to statistical data similarity or otherwise.
Thedistributed algorithm achieves for the ﬁrst time lower complexity bounds on bothcommunication and local gradient calls, with the former having being a longstanding open problem.
Finally the method is extended to distributed saddleproblems (under function similarity) by means of solving a class of variationalinequalities, achieving lower communication and computation complexity bounds.
Introduction
We consider structured convex programming in the form [6, 13, 35]:
[Equation]
where r is assumed to be convex and decomposed as the sum of a smooth, possibly nonconvexfunction p and a smooth convex function q .
First order information of p and q is accessible separately.
We are interested in scenarios where the cost of evaluating the gradient of the two functions is noteven, but computing, say r p , is much more resource demanding than r q .
The motivating applicationfor this scenario is distributed optimization over master-worker systems, as discussed next.
Consider the following distributed optimization problem over a network of n agents:
[Equation]
n n
X i =
f i ( x
[Equation]
, (2) where f i is the loss function of agent i , assumed to be convex, which is not known to other agents.
Agents are embedded in a star-topology, with agent 1 being the master node, without loss ofgenerality–this is the typical federated learning setup [24].
An instance of (2) of particular interestis the empirical risk minimization (ERM) whereby the goal is to minimize the average loss oversome dataset, distributed across the nodes of the network, with f i being the empirical risk of agent i ,i.e., f i ( x ) = 1 m
P mj =1 `  x ; z ji  , where z (1) i , . . .
, z ( m ) i is the set of m samples owned by agent i , and` ( x ; z ji ) measures the loss of the model x on the sample z ji .
Several solutions methods have been proposed to solve (2); the prototype approach consists ininterleaving local computations at the workers sides (nodes i = 1 , . . .
n ) with communications to/fromthe master node ( i = 1 ), which maintains and updates the authoritative copy of the optimizationvariables, producing eventually the ﬁnal solution estimate.
Since the cost of communications is oftenthe bottleneck in distributed computing (e.g., [7, 32]), a lot of research has been devoted to designingdistributed algorithms that are communication efﬁcient .
Acceleration (in the sense of Nesterov) hasbeen extensively investigated as a procedure to reduce the communication burden.
For L -smoothand µ -strongly convex functions r in (2), linear convergence is certiﬁed by employing ﬁrst-ordermethods, with computation (gradient evaluations) and communication complexities proportionalto p  (  , L/µ is the condition number of r ).
For ill-conditioned functions (  very large), thepolynomial dependence on  may be unsatisfactory.
This is, e.g., the typical setting of many ERMproblems wherein the optimal regularization parameter for test predictive performance is very small.
Further improvements on the communication complexity can be obtained exploiting the extra structuretypical in ERM problems, also known as function similarity (see, e.g., [4, 51, 44, 48]): kr 2 f i ( x ) r
f j k   , for all x in a proper domain of interest and all i 6 = j = 1 , . . . , n , where  > 0 measuresthe degree of similarity between the Hessian matrices of the local losses.
When data are i.i.d.among agents, f i ’s reﬂect statistical similarities in local data, resulting in  =
˜ O
(1 / p m ) with highprobability ( ˜ O hides log-factors and dependence on d ).
In this scenario, in general, 1 +  /µ ⌧  [4].
This motivated a surge of studies aiming at exploiting function similarity coupled with acceleration toboost communication efﬁciency (see Sec.
1.2 for an overview of relevant works): linear convergenceis certiﬁed with a number of communication steps (for nonquadratic losses) scaling with e
O ( p  /µ ) ,where e
O hides log-factors.
This matches lower (communication) complexity bounds [4] only up tolog-factors .
Furthermore, these methods are not computationally optimal, yielding local gradientscalls larger than lower complexity bounds O ( p  ) .
In fact, Table 1 shows that, to the date, thereexists no distributed algorithm achieving the best of the two worlds, that is, optimal (lower bound)communication complexity and local gradient (oracle) complexity.
This paper ﬁlls this gap.
Our starting point is the reformulation of (2) in the equivalent form
[Equation]
which exploits function similarity at the agents’ side via preconditioning.
Problem (3) is an instance of(1): all f i (thus q ) are convex but p is nonconvex.
Also, evaluating r q and r p has different costs; theformer involves only local computations at the master node while the latter requires communicationsfrom/to master and workers nodes.
At high-level the idea is then clear: one would like to designa distributed algorithm for (3) [or more generally for (1)] that skips gradient computations of r p(saving thus communications) without slowing down the overall optimal rate of convergence.
This naturally suggests the use of gradient-sliding techniques [19, 41, 10], yielding algorithmsthat skip from time to time computation of the gradient of one function in the summand objective.
However, existing gradient-sliding algorithms are not applicable to (1) [and thus (3)] because they allrequire p and q to be convex .
This calls for new designs, accounting for the nonconvexity of p .
1.1 Main contributions
Our contribution is threefold:
• A new gradient-sliding algorithm for (
) : We propose a new Accelerated ExtraGradient slidingmethod that skips the computation of r p from time to time.
The method builds on an inexactacceleration of a proximal envelop (outer-loop) coupled with a suitable termination criterion andinner-loop algorithm to approximately solve the proximal subproblem.
When applied to (1), withr being ( µ -strongly) convex, q being L q -smooth and convex, and p being L p -smooth (possiblynonconvex), the proposed algorithm achieves optimal complexity of gradient calls of q and p , that is,
[Equation]
[Table]
Notice that the above complexity bounds are sharper than the complexity bound obtained by theNesterov’s optimal ﬁrst-order method for smooth (strongly) convex optimization applied to (1).
Forinstance for strongly convex r , that would yield r q , r p complexity scaling as O ( p ( L q + L p ) /µ ) ,which is less favorable than our separate complexity bounds above.
To the best of our knowledge,
[Table]
this is the ﬁrst time that such bounds are achieved for nonconvex p .
[Table]
• Optimal complexity bounds for (2) under function similarity: We customize the proposedaccelerated gradient-sliding algorithm to the distributed optimization problem (2) under  -functionsimilarity.
As showed in Table 1 for strongly convex r (see Sec.
3 for the case of convex r ), the newdistributed algorithm achieves lower complexity bounds on both the number of communications [4]
[Table]
and on the number of gradient computations (without logarithmic factors!)
Achieving optimal
[Table]
communication complexity (for nonquadratic losses) was a long-standing open problem.
[Table]
• Gradient sliding for variational inequalities: We extend the proposed gradient-sliding machinery
[Table]
to solve distributed saddle-points under similarity by means of solving a class of strongly-monotoneVariational Inequalities (VI).
We improve existing complexity bounds for such problems [9] achievingfor the ﬁrst time both optimal communication complexity and gradient oracle complexity–see Table 1.
[Table]
1.2 Related works
Gradient-Sliding: Since the seminal paper [28], the idea of gradient-sliding for structured convexoptimization such as (1) has received signiﬁcant attention as tool to skip gradient computations ofone function in the summand; examples and generalization include ﬁrst-order accelerated methods[28, 29, 31], zero-order (derivative-free) schemes [14, 22, 8, 43], high-order methods [25, 17, 2, 20],and slidings for saddle point problems and variational inequalities [3, 30, 47, 9].
Albeit applicable tomore general classes of optimization problems
(1) (e.g., allowing either p and q to be nonsmooth),none of the existing methods provide guarantees when p is nonconvex.
The proposed algorithm ﬁllsthis gap.
Furthermore, it achieves optimal lower complexity bounds on the calls of r p and r q .
Distributed optimization under function similarity: The literature of distributed optimization isvast; given the focus of this work, we comment next solution methods exploiting function similarityvia proper preconditioning–Table 1 summarizes complexity results of existing distributed methodssolving either minimization problems or saddle-point formulations, and is commented next.
The seminal paper [4] established lower communication complexity bounds for (2) under  -similarity:" -optimality cannot be achieve in less than ⌦ ( p  /µ log 1 / " ) communication rounds.
Since then, a lotof effort has been devoted to design distributed schemes aiming at achieving optimal communicationcomplexity.
The authors in [42] proposed DANE , a mirror-descent based algorithm whereby workersperform a local data preconditioning via a suitably chosen Bregman divergence, and the masteraverages the solutions of the workers.
For quadratic losses, DANE achieves communication complexitye
O ((  /µ ) 2 log 1 / " ) ; this was later improved to O ((  /µ ) log 1 / " ) for nonquadratic losses in [45],where the SONATA algorithm was proposed (also implementable over mesh-networks).
Improvements were achieved employing acceleration; efforts include: DiSCO [51], an inexact damped
Newton method coupled with a preconditioned conjugate gradient (to compute the Newton direction),which achieves communication complexity e
O ( p  /µ ) log 1 / " ) for self-concordant losses (see Table1 for the log-factors hidden in the e
O )
; AIDE [40], which uses the Catalyst framework [33], matchingthe rate of DiSCO for quadratic losses; DANE-HB [50], a variant of DANE equipped with Heavy Ballmomentum and matching for quadratic functions the communication complexity of DiSCO and
AIDE ; and SPAG [21], a preconditioned direct accelerated method, achieving for nonquadradic lossesasymptotically the convergence rate O
((1  1 / p  /µ ) k )
( k is the iteration index)–the worst-caserate is still O ( p
L/µ ) log 1 / " ) .
Finally, higher order methods employing preconditioning have been studied in [12, 1, 46]: [12]proposed DiRegINA, a decentralization of the cubic regularization of the Newton method, whereworkers build Newton direction sampling local Hessians; [1] introduced ACN , an inexact acceleratedcubic-regularized Newton’s method, with improved complexity with respect to [12]; and [46] extendedthe Catalyst framework [33] to the distributed setting (including mesh networks), proposing Acc
SONATA –the communication complexity of these methods is reported in Table 1.
In summary, the above tour on the relevant literature shows that none of the existing methods canmatch lower communication complexity bounds for (non quadratic) optimization problems (2) underfunction similarity (all complexity bounds contain log-factors).
The proposed distributed methodachieves lower communication and computation complexity bounds.
Optimal Gradient Sliding for Minimization Problems
We study the minimization problem (1), under the following blanket assumptions.
Assumption 1.
r ( x ): R d !
R is µ -strongly convex on R d .
Assumption 2.
q ( x ): R d !
R is convex and L q -smooth on R d .
Assumption 3.
p ( x ): R d !
R is L p -smooth on R d .
The proposed Accelerated ExtraGradient sliding is formally introduced in Algorithm 1.
Convergenceof the outer loop is established in Theorem 1, while Theorem 2 determines complexity of solving theinner loop up to a suitable termination.
Finally Theorem 3 provides the overall complexity merginginner and outer loop results.
The proof of all the theorems can be found in Appendix A.1.
1: Input: x 0 = x 0 f 2 R d 2: Parameters: ⌧ 2 (0 , 1] , ⌘ , ✓ , ↵ > 0 , K 2 { 1 , 2 , . . .
} 3: for k = 0 , 1 , 2 , . . .
, K  1 do 4: x kg =
⌧ x k + (1  ⌧ ) x
k f 5: x k +1 f ⇡ arg min x 2 R d ⇥ A k ✓ ( x ) :
= p ( x kg ) + hr p
( x kg ) , x  x
kg i + 12 ✓ k x  x kg k 2 + q ( x ) ⇤ 6: x k +
= x k + ⌘↵ ( x k +1 f  x k )  ⌘ r r ( x k +1 f ) 7: end for 8:
Output: x K
Theorem 1.
Consider Algorithm 1 for Problem 1 under Assumptions 1-3, with the following tuning:
[Equation]
and let x k +1 f in line 5 satisfy
[Equation]
Then, for any
[Equation]
we have the following estimate for the distance to the solution x ⇤ :
[Equation]
2.1 Solving the auxiliary subproblems
At each iteration of Algorithm 1, one needs to solve the subproblem:
[Equation]
According to Theorem 1, (7) need not be solved to arbitrary precision; inexact solutions x k +1 fsatisfying condition (4) sufﬁce.
Condition (4) means that gradient norm kr A k ✓ ( x k +1 f ) k should besufﬁciently small.
Notice that A k ✓ ( x ) in (7) is (2 L p + L q ) -smooth and convex.
Problem 7 can be solved up to the termination (4) using any of the algorithms in [27, 26, 38].
Weobtain the following complexity.
Theorem 2 ([38] Remark 1) .
There exists a certain algorithm such that, when applied to problem (7)with the starting point x kg , returns x k +1 f satisfying
[Equation]
where D > 0 is some universal constant (independent of L p , L q , T etc) and T is the number of callsof r q by the algorithm.
2.2 Overall complexity of the optimal gradient-sliding
Theorem 2 suggests that, to satisfy condition (4) in Theorem 1, it is sufﬁcient to choose the number
T of iterations of the inner algorithm as
[Equation]
We can now determine the overal complexity of Algorithm 1.
At each iteration of Algorithm 1 wecall r p
twice (at x kg – line 5 and at x k +1 f – line 6), and r q is computed T + 1 times ( T times in theauxiliary problem – line 5, and at x k +1 f – line 6).
Hence, to ﬁnd an " -solution of problem (1), i.e., toﬁnd x K 2 R d that satisﬁes (6), Algorithm 1 requires K iterations as given in (5),
[Equation]
[Equation]
Putting everything together we obtain the following ﬁnal convergence result.
Theorem 3.
Consider Problem (1) under Assumptions 1 to 3, with µ  L p  L q .
Then, to reach an 163 " -solution, Algorithm 1 requires
O ⇣q
L q µ log 1 " ⌘ calls of r q ( x ) and O ⇣q
L p µ log 1 " ⌘ calls of r p ( x ) .
This matches optimal complexity for the individual gradient calls.
We conclude this section providing convergence of variant of the proposed algorithm suitable forconvex r in (1) (Assumption 1 with µ = 0 ).
The algorithm is described in Appendix A.2.
Here weonly provide the ﬁnal convergence result, the analogous of Theorem 3.
Theorem 4.
Consider Problem (1) under Assumptions 1 (with µ = 0 )
-3, with L p  L q .
Then, toﬁnd an " -solution of (1) (in objective value), Algorithm 3 in Appendix A.2 requires
O ✓q
L q " k x 0  x ⇤ k ◆ calls of r q ( x ) and O ✓q
L p " k x 0  x ⇤ k ◆ calls of r p ( x ) .
Application to Distributed Optimization Under Similarity
In this section, we apply the proposed algorithm the the distributed optimization problem (2), underthe following assumptions.
Assumption 4.
Each f i ( x ): R d !
R is convex and L -smooth.
Assumption 5.
r ( x ): R d !
R is µ -strongly convex.
Assumption 6.
f 1 ( x ) , . .
, f n ( x ) are  -related:
kr 2 f i ( x )  r 2 f j ( x ) k   , for all
i 6 =
j andx 2 R d , and some  > 0 .
From the last assumption it is easy to get that for all i and x 2 R d we have kr
f i ( x )  r 2 r ( x ) k
=k
[Equation]
n
[Equation]
We leverage now Algorithm 1 to solve (2), using the equivalent reformulation (3).
The algorithmapplied to the distributed system can be described as follows.
The server computes x kg and sendsit to all the workers (line 4).
Workers compute r f i ( x kg ) and send it to the server.
After collectingall r f i ( x kg ) , the server builds r p ( x kg ) =
r r ( x kg )  r f 1 ( x kg ) , and then solves (inexactly) the localproblem
A k ✓ (line 5).
The inexact solution x k +1 f is then broadcast to the workers, which update theirown receives r f i ( x k +1 f ) and send back to the server, which can then evaluate r r ( x k +1 f ) (line 6).
Using Assumptions 4 and 5, we infer that r is µ -strongly convex; and q = f 1 is L q -smooth andconvex, with L q = L .
It follows from Assumption 6 that kr 2 p k   .
Therefore, p has L p -Lipschitzgradient, with L p =  .
This shows that we can leverage Theorems 3 and 4 to establish convergencefor strongly convex and convex r , as given next.
Theorem 5.
Let Assumptions 4 to 6 be satisﬁed with µ    L .
Then, to ﬁnd " -solution of the distributed optimization problem (2) Algorithm 1 requires
[Equation]
gradient computations
[Equation]
Theorem 6.
Let Assumptions 4, 5 (with µ = 0 ), 6 be satisﬁed and   L .
Then, to ﬁnd " -solution of the distributed optimization problem (2) Algorithm 1 requires
[Equation]
Such estimates are optimal from both communications [4] and local computations point of views [37].
It is important to remark that Algorithm 1 solves the local subproblem
A ✓ with some precision, whilemost of existing works (see Table 1) assume that local problems are solved with inﬁnite precision(column Local gradient complexity), which is not practical.
Note also that the subproblems in line 5of
Algorithm 1 do not necessarily have to be solved by a deterministic algorithm as in Theorem 2.
Stochastic methods can also be used, as long as they guarantee that condition (4) is met.
Optimal Gradient Sliding for VIs
In this section we consider the composite variational inequality [16, 5] in the form:
Find x
⇤ 2 R d : R ( x ⇤ )
=
with R ( x ) :
= Q ( x ) + P ( x ) , (10) where Q ( x ) , P ( x ): R d !
R d .
Variational inequalities are a uniﬁed umbrella for a variety ofproblems–two examples follow.
Example 1 [Minimization].
Consider problem (1), choose Q ( x ) =
r q ( x ) and P ( x ) =
r p ( x ) .
Then the solution of the variational inequality (10) means that we need to ﬁnd the point x
⇤ where theoperator R ( x ⇤ )
=
r r ( x ⇤ ) .
For the convex function r , this is equivalent to ﬁnding the minimum.
Example
[Saddle point problems].
Consider the convex-concave saddle point problem
[Equation]
If we take Q ( x ) :
[Equation]
[ r y p ( y, z ) , r z p ( y, z )] , then it can be proved that x ⇤ =
( y ⇤ , z ⇤ ) is a solution for (10) if and only if x ⇤ =
( y ⇤ , z ⇤ ) is a solution for (11), i.e.
[Equation]
all
[Equation]
and z
[Equation]
While minimization problems are widely considered separately from variational inequalities, saddlepoint problems are often analyzed under the VI lens.
In recent years the popularity of saddles hasgrown, this is due to the fact that they have both classical [15] and new ML [18, 34] applications.
We study problem (10) under the following assumptions.
Assumption 7.
R ( x ) is µ -strongly monotone: h R ( x 1 )  R ( x 2 ) ,
x 1  x 2 i  µ k x 1  x 2 k 2 , for allx 1 , x 2 2 R d .
Assumption 8.
Q ( x ) is monotone and L q -Lipschitz:
h Q ( x 1 )  Q ( x 2 ) , x
1  x 2 i  0 and k Q ( x 1 )
Q ( x 2 ) k  L q k x 1  x 2 k , for all x 1 , x 2 2 R d .
Assumption 9.
P ( x ) is L p -Lipschitz:
k P ( x 1 )  P ( x 2 ) k
 L p k x 1  x 2 k for all x 1 , x 2 2 R d .
For saddle point problems these assumptions are equivalent to (strong) convexity–(strong)
concavityand Lipschitzness of gradients.
4.1 Sliding via Extragradient
This algorithm is a non-accelerated version of Algorithm 1.
A similar non-accelerated sliding is usedin [9].
Our version however has better theoretical and practical guarantees because of the effectivestopping criterion (12).
Convergence of the outer loop is established in Theorem 7; Theorem 8establishes convergence of the inner loop up to the required termination; and ﬁnally Theorem 9combine the two-loop complexity.
The proofs of the theorems can be found in Appendix B.1.
Algorithm
1: Input: x 0 2 R d 2: Parameters: ⌘ , ✓ , ↵ > 0 , K 2 { 1 , 2 , . . .
} 3: for k = 0 , 1 , 2 , . . .
, K  1 do 4:
Find u k ⇡ ˜ u
k where ˜ u k is a solution for
Find ˜ u
k 2 R d : B k ✓ (˜ u k ) = 0 with B k ✓ ( x ) := P ( x k ) + Q ( x ) + 1 ✓ ( x  x k ) 5: x k +
[Equation]
k ) 6: end for 7: Output: x K
Theorem 7.
Consider Algorithm 2 for Problem (10) under Assumptions 7-9, with the following tuning:
[Equation]
Assume that u k (line 4) satisﬁes
[Equation]
Then, for any
[Equation]
we have the following estimate for the distance to the solution x ⇤ :
[Equation]
The proof is given in Appendix B.1.
4.2 Solving the auxiliary problem
As for Algorithm 1, we need an auxiliary solver for the subproblem in line 4, which ensures that (12)is met.
One can observe that B k ✓ ( x ) is (2 L p + L q ) -Lipschitz and monotone.
For this type of problem,the authors in [49] proposes an approach that guarantees convergence on k B k ✓ ( u k ) k 2 .
Theorem 8 ([49] Corollary 2) .
There exists a certain algorithm that, applied to the subproblem (12)with starting point x k , returns u k satisfying
[Equation]
where D > 0 is some universal numerical constant (independent of L p , L q , T etc) and T is thenumber of calls of the operator Q .
4.3 Complexity of the optimal gradient sliding
Leveraging Theorems 7 and 8 while following the same reasoning as in Section 2.2, we obtain thefollowing convergence (inner plus outer loops) for Algorithm 2.
Theorem 9.
Let Assumptions 7 to 9 be satisﬁed with µ  L p  L q .
Then, Algorithm 2 requires
[Equation]
of Q
[Equation]
x ) and O
[Equation]
⌘ calls of P ( x ) to ﬁnd an " -solution of problem (10) .
We also consider the case of monotone VIs (Assumption 7 with µ = 0 ).
For this we modify Algorithm2 as in Appendix B.2 and obtain the following convergence result.
Theorem 10.
Let Assumption 7 (with µ = 0 ), 8, 9 be satisﬁed and L p  L q .
Then, Algorithm 4,described in Appendix B.2, requires
[Equation]
Q ( x ) and O
[Equation]
P
[Equation]
x
[Equation]
to ﬁnd an " -solution of the problem (10) .
Here " -solution is measured by the value of the gap function.
4.4 Application to distributed saddle-point problem under similarity
We apply now Algorithm 2 to solve a distributed saddle-point problem under statistical similarity, asintroduced in [9]:
[Equation]
Assumption 10.
Each f i ( y, z ): R d y ⇥ R d z !
R is convex-concave and L -smooth on R d y ⇥ R d z .
Assumption 11.
r ( z, y ) is µ -strongly convex (ﬁrst argument)– µ -strongly concave (second argument).
Assumption 12.
f 1 ( y, z ) , . . .
, f n ( y, z ) are  -related: for all
i 6 = j and for all y 2 R d y and z 2 R d z ,
[Equation]
Casting (16) into the VI formulation (10), by taking Q ( x ) =
Q ( y, z ) =
[ r y f 1 ( y, z ) , r
z f 1 ( y, z )]and P ( x ) = P ( y, z ) =
[ r y
[ r  f 1 ]( y, z ) , r
z [ r  f 1 ]( y, z )] , we have that Q is monotone and
L -Lipschitz, P is  -Lipschitz, R is µ -strongly monotone.
Therefore, we can apply Theorems 9 and10 and obtain the following convergence results for Algorithm 2 applied to (16).
Theorem 11.
Let Assumptions 10 to 12 be satisﬁed with µ    L .
Then, to ﬁnd " -solution of the distributed saddle problem (16) , Algorithm 2 requires
O ⇣
[Equation]
µ
[Equation]
communication rounds and O
[Equation]
local gradient computations.
Theorem 12.
Let Assumptions 10, 11 (with µ = 0 ), 12 be satisﬁed and   L .
Then, to ﬁnd " -solution of the distributed saddle problem (16) , Algorithm 2 requires
[Equation]
Our communication estimates are optimal, as in [9], but our algorithm also achieve optimal localcomplexity (see Table 1).
5.1 Minimization
We consider the Ridge Regression problem
[Equation]
where w is the vector of weights of the model, { x i , y i }
Ni =1 is the training dataset, and  > 0 is theregularization parameter.
We consider a network with 25 workers (simulated on a single-CPU machine), and use two types ofdatasets, namely: synthetic and real data.
Synthetic data permit to control the similarity constant  .
To do so, we generate data on the server, say { ˆ x i , ˆ y i } n =100 i = .
Data on the workers are generated byadding unbiased Gaussian noise to the server data.
The lower the variance of this noise, the moresimilar the data, and thus the smaller  .
For simulations with real data, we considered the LIBSVMlibrary [11] and give each agent a full dataset.
Then, each device selects at random a part of size mfrom the full dataset.
In Section C.2 we explain how the parameters L and  are estimated.
For thesynthetic dataset we choose the noise level and the regularization parameter such that L/  = 200and L/  = 10 5 .
For the real datasets the regularization parameter is chosen such that L/  = 10 6 .
See Table 2 for all values of L,  , µ and m .
For comparison, we use distributed version of Accelerated Gradient Descent (AcGD)
[37] as the basic,classical and optimal method for minimization problems without additional similarity assumptions,as well as state-of-the-art schemes for the similarity condition: DANE [42], DANE-HB [50], SPAG[21] and AccSONATA [46].
The settings of the methods are made as described in the originalpapers.
For algorithms that assume an absolutely accurate solution of local problems (DANE, SPAG,
AccSONATA), we use AcGD with an accuracy of 10  12 as a subsolver.
Results are summarized in Figure 1–the ﬁrst two ﬁgures from the top left correspond to syntheticdata while the other 6 on real data.
[Figure]
The results show that our method outperforms almost all methods in terms of communication (onlyin one experiment SPAG is slightly faster).
In terms of local iterations, our method is slightly inferiorto AcGD, but superior to all other methods.
In Appendix C.1, one can ﬁnd a comparison of our method with a competitor for distributed saddlepoint problems under similarity assumption.