Optimal Gradient Sliding and its Application to Distributed Optimization Under Similarity
Abstract
We study structured convex optimization problems, with additive objective r := p + q , where r is ( µ -strongly) convex, q is L q -smooth and convex, and p is L psmooth, possibly nonconvex.
For such a class of problems, we proposed an inexact accelerated gradient sliding method that can skip the gradient computation for one of these components while still achieving optimal complexity of gradient calls of p and q , that is, O ( p
L p /µ ) and O ( p
L q /µ ) , respectively.
This result is much sharper than the classic black-box complexity O ( p ( L p + L q ) /µ ) , especially when the difference between L p and L q is large.
We then apply the proposed method to solve distributed optimization problems over master-worker architectures, under agents’ function similarity, due to statistical data similarity or otherwise.
The distributed algorithm achieves for the ﬁrst time lower complexity bounds on both communication and local gradient calls, with the former having being a longstanding open problem.
Finally the method is extended to distributed saddleproblems (under function similarity) by means of solving a class of variational inequalities, achieving lower communication and computation complexity bounds.
Introduction
We consider structured convex programming in the form [6, 13, 35]:
[Equation]
where r is assumed to be convex and decomposed as the sum of a smooth, possibly nonconvex function p and a smooth convex function q .
First order information of p and q is accessible separately.
We are interested in scenarios where the cost of evaluating the gradient of the two functions is not even, but computing, say r p , is much more resource demanding than r q .
The motivating application for this scenario is distributed optimization over master-worker systems, as discussed next.
Consider the following distributed optimization problem over a network of n agents:
[Equation]
where f i is the loss function of agent i , assumed to be convex, which is not known to other agents.
Agents are embedded in a star-topology, with agent 1 being the master node, without loss of generality–this is the typical federated learning setup [24].
An instance of (2) of particular interest is the empirical risk minimization (ERM) whereby the goal is to minimize the average loss over some dataset, distributed across the nodes of the network, with f i being the empirical risk of agent i , i.e., f i ( x ) = 1 m
[Equation] ; z ji  , where z (1) i , . . .
, z ( m ) i is the set of m samples owned by agent i , and ` ( x ; z ji ) measures the loss of the model x on the sample z ji .
Several solutions methods have been proposed to solve (2); the prototype approach consists in interleaving local computations at the workers sides (nodes i = 1 , . . .
n ) with communications to/from the master node ( i = 1 ), which maintains and updates the authoritative copy of the optimization variables, producing eventually the ﬁnal solution estimate.
Since the cost of communications is often the bottleneck in distributed computing (e.g., [7, 32]), a lot of research has been devoted to designing distributed algorithms that are communication efﬁcient .
Acceleration (in the sense of Nesterov) has been extensively investigated as a procedure to reduce the communication burden.
For L -smooth and µ -strongly convex functions r in (2), linear convergence is certiﬁed by employing ﬁrst-order methods, with computation (gradient evaluations) and communication complexities proportional to p  (  , L/µ is the condition number of r ).
For ill-conditioned functions (  very large), the polynomial dependence on  may be unsatisfactory.
This is, e.g., the typical setting of many ERM problems wherein the optimal regularization parameter for test predictive performance is very small.
Further improvements on the communication complexity can be obtained exploiting the extra structure typical in ERM problems, also known as function similarity (see, e.g., [4, 51, 44, 48]): kr 2 f i ( x ) [-r 
-] r 2 f j k   , for all x in a proper domain of interest and all i 6 = j = 1 , . . .
, n , where  > 0 measures the degree of similarity between the Hessian matrices of the local losses.
When data are i.i.d. among agents, f i ’s reﬂect statistical similarities in local data, resulting in  = ˜
O
(1 / p m ) with highprobability ( ˜ O hides log-factors and dependence on d ).
In this scenario, in general, 1 +  /µ ⌧  [4].
This motivated a surge of studies aiming at exploiting function similarity coupled with acceleration to boost communication efﬁciency (see Sec.
1.2 for an overview of relevant works): linear convergence is certiﬁed with a number of communication steps (for nonquadratic losses) scaling with e
O ( p  /µ ) , where e
O hides log- factors.
This matches lower (communication) complexity bounds [4] only up to log-factors .
Furthermore, these methods are not computationally optimal, yielding local gradients calls larger than lower complexity bounds O ( p  ) .
In fact, Table 1 shows that, to the date, there exists no distributed algorithm achieving the best of the two worlds, that is, optimal (lower bound) communication complexity and local gradient (oracle) complexity.
This paper ﬁlls this gap.
Our starting point is the reformulation of (2) in the equivalent form
[Equation]
[Equation]
which exploits function similarity at the agents’ side via preconditioning.
Problem (3) is an instance of (1): all f i (thus q ) are convex but p is nonconvex.
Also, evaluating r q and r p has different costs; the former involves only local computations at the master node while the latter requires communications from/to master and workers nodes.
At high-level the idea is then clear: one would like to design a distributed algorithm for (3) [or more generally for (1)] that skips gradient computations of r p (saving thus communications) without slowing down the overall optimal rate of convergence.
This naturally suggests the use of gradient-sliding techniques [19, 41, 10], yielding algorithms that skip from time to time computation of the gradient of one function in the summand objective.
However, existing gradient-sliding algorithms are not applicable to (1) [and thus (3)] because they all require p and q to be convex .
This calls for new designs, accounting for the nonconvexity of p .
1.1 Main contributions
Our contribution is threefold:
• A new gradient-sliding algorithm for (1) : We propose a new Accelerated ExtraGradient sliding method that skips the computation of r p from time to time.
The method builds on an inexact acceleration of a proximal envelop (outer-loop) coupled with a suitable termination criterion and inner-loop algorithm to approximately solve the proximal subproblem.
When applied to (1), with r being ( µ -strongly) convex, q being L q -smooth and convex, and p being L p -smooth (possibly nonconvex), the proposed algorithm achieves optimal complexity of gradient calls of q and p , that is,
[Equation]
Notice that the above complexity bounds are sharper than the complexity bound obtained by the Nesterov’s optimal ﬁrst-order method for smooth (strongly) convex optimization applied to (1).
For instance for strongly convex r , that would yield r q , r p complexity scaling as O ( p ( L q + L p ) /µ ) , which is less favorable than our separate complexity bounds above.
To the best of our knowledge, this is the ﬁrst time that such bounds are achieved for nonconvex p .
• Optimal complexity bounds for (2) under function similarity: We customize the proposed accelerated gradient-sliding algorithm to the distributed optimization problem (2) under  -function similarity.
As showed in Table 1 for strongly convex r (see Sec.
3 for the case of convex r ), the new distributed algorithm achieves lower complexity bounds on both the number of communications [4] and on the number of gradient computations (without logarithmic factors!)
Achieving optimal communication complexity (for nonquadratic losses) was a long-standing open problem. • Gradient sliding for variational inequalities: We extend the proposed gradient-sliding machinery to solve distributed saddle-points under similarity by means of solving a class of strongly-monotone Variational Inequalities (VI). We improve existing complexity bounds for such problems [9] achieving for the ﬁrst time both optimal communication complexity and gradient oracle complexity–see Table
[Table]
[Table]
⇤ k [Table] [Table] 0 )  r
[Table]
⇤ )
[Table]
1.2 Related works
Gradient-Sliding: Since the seminal paper [28], the idea of gradient-sliding for structured convex optimization such as (1) has received signiﬁcant attention as tool to skip gradient computations of one function in the summand; examples and generalization include ﬁrst-order accelerated methods [28, 29, 31], zero-order (derivative-free) schemes [14, 22, 8, 43], high-order methods [25, 17, 2, 20], and slidings for saddle point problems and variational inequalities [3, 30, 47, 9].
Albeit applicable to more general classes of optimization problems than (1) (e.g., allowing either p and q to be nonsmooth), none of the existing methods provide guarantees when p is nonconvex.
The proposed algorithm ﬁlls this gap.
Furthermore, it achieves optimal lower complexity bounds on the calls of r p and r q .
Distributed optimization under function similarity: The literature of distributed optimization is vast; given the focus of this work, we comment next solution methods exploiting function similarity via proper preconditioning–Table 1 summarizes complexity results of existing distributed methods solving either minimization problems or saddle-point formulations, and is commented next.
The seminal paper [4] established lower communication complexity bounds for (2) under  -similarity: " -optimality cannot be achieve in less than ⌦ ( p  /µ log 1 / " ) communication rounds.
Since then, a lot of effort has been devoted to design distributed schemes aiming at achieving optimal communication complexity.
The authors in [42] proposed DANE , a mirror-descent based algorithm whereby workers perform a local data preconditioning via a suitably chosen Bregman divergence, and the master averages the solutions of the workers.
For quadratic losses, DANE achieves communication complexity e
O ((  /µ ) 2 log 1 / " ) ; this was later improved to O ((  /µ ) log 1 / " ) for nonquadratic losses in [45], where the SONATA algorithm was proposed (also implementable over mesh-networks).
Improvements were achieved employing acceleration; efforts include: DiSCO [51], an inexact damped
Newton method coupled with a preconditioned conjugate gradient (to compute the Newton direction), which achieves communication complexity e
O ( p  /µ ) log 1 / " ) for self-concordant losses (see Table 1 for the log-factors hidden in the e
O )
; AIDE [40], which uses the Catalyst framework [33], matching the rate of DiSCO for quadratic losses; DANE-HB [50], a variant of DANE equipped with Heavy Ball momentum and matching for quadratic functions the communication complexity of DiSCO and
AIDE ; and SPAG [21], a preconditioned direct accelerated method, achieving for nonquadradic losses asymptotically the convergence rate O
((1  1 / p  /µ ) k )
( k is the iteration index)–the worst-case rate is still O ( p
L/µ ) log 1 / " ) .
Finally, higher order methods employing preconditioning have been studied in [12, 1, 46]: [12] proposed DiRegINA, a decentralization of the cubic regularization of the Newton method, where workers build Newton direction sampling local Hessians; [1] introduced ACN , an inexact accelerated cubic-regularized Newton’s method, with improved complexity with respect to [12]; and [46] extended the Catalyst framework [33] to the distributed setting (including mesh networks), proposing Acc
SONATA –the communication complexity of these methods is reported in Table 1.
In summary, the above tour on the relevant literature shows that none of the existing methods can match lower communication complexity bounds for (non quadratic) optimization problems (2) under function similarity (all complexity bounds contain log-factors).
The proposed distributed method achieves lower communication and computation complexity bounds.
Optimal Gradient Sliding for Minimization Problems
We study the minimization problem (1), under the following blanket assumptions.
Assumption 1.
r ( x ): R d !
R is µ -strongly convex on R d .
Assumption 2.
q ( x ): R d !
R is convex and L q -smooth on R d .
Assumption 3.
p ( x ): R d !
R is L p -smooth on R d .
The proposed Accelerated ExtraGradient sliding is formally introduced in Algorithm 1.
Convergence of the outer loop is established in Theorem 1, while Theorem 2 determines complexity of solving the inner loop up to a suitable termination.
Finally Theorem 3 provides the overall complexity merging inner and outer loop results.
The proof of all the theorems can be found in Appendix A.1.
[Equation] 2: Parameters:
[Equation]
[Equation] 0 , K
[Equation]
[Equation] 3: for k = 0 , 1 , 2 , . . . , K
[-= p ( x kg ) + hr p ( x kg ) , x  x kg i + 12 ✓ k x  x kg k 2 + q ( x ) ⇤ 6: x k +
-][Equation]
[Equation] [Equation] 7: end for 8: Output: x K
Theorem 1.
Consider Algorithm 1 for Problem 1 under Assumptions [Equation] with the following tuning:
[Equation]
and let x k +1 f in line 5 satisfy
[Equation]
Then, for any
[Equation]
we have the following estimate for the distance to the solution x ⇤ :
[Equation]
2.1 Solving the auxiliary subproblems
At each iteration of Algorithm 1, one needs to solve the subproblem:
[Equation]
According to Theorem 1, (7) need not be solved to arbitrary precision; inexact solutions x k +1 f satisfying condition (4) sufﬁce.
Condition (4) means that gradient norm kr A k ✓ ( x k +1 f ) k should be sufﬁciently small.
Notice that A k ✓ ( x ) in (7) is (2 L p + L q ) -smooth and convex.
Problem 7 can be solved up to the termination (4) using any of the algorithms in [27, 26, 38].
We obtain the following complexity.
Theorem 2 ([38] Remark 1) .
There exists a certain algorithm such that, when applied to problem (7) with the starting point x kg , returns x k +1 f satisfying
[Equation]
where D > 0 is some universal constant (independent of L p , L q , T etc) and T is the number of calls of r q by the algorithm.
2.2 Overall complexity of the optimal gradient-sliding
Theorem 2 suggests that, to satisfy condition (4) in Theorem 1, it is sufﬁcient to choose the number
T of iterations of the inner algorithm as
[Equation]
We can now determine the overal complexity of Algorithm 1.
At each iteration of Algorithm 1 we call r p twice (at x kg – line 5 and at x k +1 f – line 6), and r q is computed T + 1 times ( T times in the auxiliary problem – line 5, and at x k +1 f – line 6).
Hence, to ﬁnd an " -solution of problem (1), i.e., to ﬁnd x K 2 R d that satisﬁes (6), Algorithm 1 requires K iterations as given in (5), [Equation] ⇥ K = O
[Equation]
[Equation]
calls of r p ( x ) , and
[Equation]
Putting everything together we obtain the following ﬁnal convergence result.
Theorem 3.
Consider Problem (1) under Assumptions 1 to 3, with µ  L p  L q .
Then, to reach an " -solution, Algorithm 1 requires
[Equation]
This matches optimal complexity for the individual gradient calls.
We conclude this section providing convergence of variant of the proposed algorithm suitable for convex r in (1) (Assumption 1 with µ = 0 ).
The algorithm is described in Appendix A.2.
Here we only provide the ﬁnal convergence result, the analogous of Theorem 3.
Theorem 4.
Consider Problem (1) under Assumptions 1 (with µ = 0 )
-3, with L p  L q .
Then, to ﬁnd an " -solution of (1) (in objective value), Algorithm 3 in Appendix A.2 requires
O ✓q
[Equation] ◆ [Equation] O ✓q
[Equation]
of r p [Equation] x [Equation]
Application to Distributed Optimization Under Similarity
In this section, we apply the proposed algorithm the the distributed optimization problem (2), under the following assumptions.
Assumption 4.
Each f i ( x ): R d !
R is convex and L -smooth.
Assumption 5.
r ( x ): R d !
R is µ -strongly convex.
Assumption 6.
f 1 ( x ) , . .
, f n ( x ) are  -related:
kr 2 f i ( x )  r 2 f j ( x ) k   , for all
i 6 = j and x 2 R d , and some  > 0 .
From the last assumption it is easy to get that for all i and x 2 R d we have kr
f i ( x )  r 2 r ( x ) k
=
[Equation]
We leverage now Algorithm 1 to solve (2), using the equivalent reformulation (3).
The algorithm applied to the distributed system can be described as follows.
The server computes x kg and sends it to all the workers (line 4).
Workers compute r f i ( x kg ) and send it to the server.
After collecting all r f i ( x k g ) , the server builds r p ( x k g ) =
r r ( x k g )  r f 1 ( x k g ) , and then solves (inexactly) the local problem
A k ✓ (line 5).
The inexact solution x k +1 f is then broadcast to the workers, which update their own receives r f i ( x k +1 f ) and send back to the server, which can then evaluate r r ( x k +1 f ) (line 6).
Using Assumptions 4 and 5, we infer that r is µ -strongly convex; and q = f 1 is L q -smooth and convex, with L q = L .
It follows from Assumption 6 that kr 2 p k   .
Therefore, p has L p -Lipschitz gradient, with L p =  .
This shows that we can leverage Theorems 3 and 4 to establish convergence for strongly convex and convex r , as given next.
Theorem 5.
Let Assumptions 4 to 6 be satisﬁed with µ    L .
Then, to ﬁnd " -solution of the distributed optimization problem (2) Algorithm 1 requires
O ⇣q  µ log 1 " ⌘ communication rounds and O ⇣q
L µ log 1 " ⌘ local gradient computations.
Theorem 6.
Let Assumptions 4, 5 (with µ = 0 ), 6 be satisﬁed and   L .
Then, to ﬁnd " -solution of the distributed optimization problem (2) Algorithm 1 requires
[Equation]
Such estimates are optimal from both communications [4] and local computations point of views [37].
It is important to remark that Algorithm 1 solves the local subproblem
A ✓ with some precision, while most of existing works (see Table 1) assume that local problems are solved with inﬁnite precision (column Local gradient complexity), which is not practical.
Note also that the subproblems in lineof Algorithm 1 do not necessarily have to be solved by a deterministic algorithm as in Theorem 2.
Stochastic methods can also be used, as long as they guarantee that condition (4) is met.
Optimal Gradient Sliding for VIs
In this section we consider the composite variational inequality [16, 5] in the form:
[-Find x ⇤ 2 R d : R ( x ⇤ ) = 
 with R ( x ) :-][Equation]
where Q ( x ) , P ( x ): R d ! R d .
Variational inequalities are a uniﬁed umbrella for a variety of problems–two examples follow.
Example 1 [Minimization].
Consider problem (1), choose Q ( x ) =
r q ( x ) and P ( x ) =
r p ( x ) .
Then the solution of the variational inequality (10) means that we need to ﬁnd the point x
⇤ where the operator R ( x ⇤ ) =
r r ( x ⇤ ) .
For the convex function r , this is equivalent to ﬁnding the minimum.
Example
[Saddle point problems].
Consider the convex-concave saddle point problem
[Equation]
If we take Q ( x ) :=
Q ( y, z ) = [ r y q ( y, z ) , r z q ( y, z )] and P ( x ) : =
P ( y, z ) = [ r y p ( y, z ) , r z p ( y, z )] , then it can be proved that x ⇤ =
( y ⇤ , z ⇤ ) is a solution for (10) if and only if x ⇤ =
( y ⇤ , z ⇤ ) is a solution for 11), i.e.
r ( y ⇤ , z )  r ( y ⇤ , z ⇤ )
 r ( y, z ⇤ ) for all y 2 R d y and z 2 R d z .
While minimization problems are widely considered separately from variational inequalities, saddle point problems are often analyzed under the VI lens.
In recent years the popularity of saddles has grown, this is due to the fact that they have both classical [15] and new ML [18, 34] applications.
We study problem (10) under the following assumptions.
Assumption 7.
R ( x ) is µ -strongly monotone: h R ( x 1 )  R ( x 2 ) ,
x 1  x 2 i  µ k x 1  x 2 k 2 , for all x 1 , x 2 2 R d .
Assumption 8.
Q ( x ) is monotone and L q -Lipschitz:
h Q ( x 1 )  Q ( x 2 ) , x
1  x 2 i  0 and k Q ( x 1 )
Q ( x 2 ) k  L q k x 1  x 2 k , for all x 1 , x 2 2 R d .
Assumption 9.
P ( x ) is L p -Lipschitz:
k P ( x 1 )  P ( x 2 ) k
 L p k x 1  x 2 k for all x 1 , x 2 2 R d .
For saddle point problems these assumptions are equivalent to (strong) convexity–(strong) concavity and Lipschitzness of gradients.
4.1 Sliding via Extragradient
This algorithm is a non-accelerated version of Algorithm 1.
A similar non-accelerated sliding is used in [9].
Our version however has better theoretical and practical guarantees because of the effective stopping criterion (12).
Convergence of the outer loop is established in Theorem 7; Theoremestablishes convergence of the inner loop up to the required termination; and ﬁnally Theoremcombine the two-loop complexity.
The proofs of the theorems can be found in Appendix B.1.
Algorithm
1: Input: x 0 2 R d 2: Parameters: ⌘ , ✓ , ↵ > 0 , K 2 { 1 , 2 , . . .
} 3: for k = 0 , 1 , 2 , .
. . , K  1 do 4:
Find u k ⇡ ˜ u
k where ˜ u k is a solution for
Find ˜ u
k 2 R d : B k ✓ (˜ u k ) = 0 with B k ✓ ( x ) : = P ( x k ) + Q ( x ) + 1 ✓ ( x  x k ) 5: x k +1 =
x k + ⌘↵ ( u k  x k )  ⌘ R ( u k ) 6: end for 7: Output: x K
Theorem 7.
Consider Algorithm 2 for Problem (10) under Assumptions 7-9, with the following tuning:
[Equation]
Assume that u k (line 4) satisﬁes
[Equation]
Then, for any
[Equation]
we have the following estimate for the distance to the solution x ⇤ :
[Equation]
The proof is given in Appendix B.1.
4.2 Solving the auxiliary problem
As for Algorithm 1, we need an auxiliary solver for the subproblem in line 4, which ensures that (12) is met.
One can observe that B k ✓ ( x ) is (2 L p + L q ) -Lipschitz and monotone.
For this type of problem, the authors in [49] proposes an approach that guarantees convergence on k B k ✓ ( u k ) k 2 .
Theorem 8 ([49] Corollary 2) .
There exists a certain algorithm that, applied to the subproblem (12) with starting point x k , returns u k satisfying
[Equation]
where D > 0 is some universal numerical constant (independent of L p , L q , T etc) and T is the number of calls of the operator Q .
4.3 Complexity of the optimal gradient sliding
Leveraging Theorems 7 and 8 while following the same reasoning as in Section 2.2, we obtain the following convergence (inner plus outer loops) for Algorithm 2.
Theorem 9.
Let Assumptions 7 to 9 be satisﬁed with µ  L p  L q .
Then, Algorithm 2 requires
O ⇣
[Equation]
1 " ⌘ calls of Q ( x ) and O ⇣
[Equation]
log 1 " ⌘ calls of P ( x ) to ﬁnd an " -solution of problem (10) .
We also consider the case of monotone VIs (Assumption 7 with µ = 0 ).
For this we modify Algorithm 2 as in Appendix B.2 and obtain the following convergence result.
Theorem 10.
Let Assumption 7 (with µ = 0 ), 8, 9 be satisﬁed and L p  L q .
Then, Algorithm 4, described in Appendix B.2, requires
[Equation]
to ﬁnd an " -solution of the problem (10) .
Here " -solution is measured by the value of the gap function.
4.4 Application to distributed saddle-point problem under similarity
We apply now Algorithm 2 to solve a distributed saddle-point problem under statistical similarity, as introduced in [9]:
[Equation]
Assumption 10.
Each f i ( y, z ): R d y ⇥ R d z !
R is convex-concave and L -smooth on R d y ⇥ R d z .
Assumption 11.
r ( z, y ) is µ -strongly convex (ﬁrst argument)– µ -strongly concave (second argument).
Assumption 12.
f 1 ( y, z ) , . . .
, f n ( y, z ) are  -related: for all
i 6 = j and for all y 2 R d y and z 2 R d z , {+kr 2 yy f i ( y, z ) r 
+}
yy f j ( y, z ) k   , kr 2 yz f i ( y, z ) r 2 yz f j ( y, z ) k   , kr 2 zz f i ( y, z ) r 2 zz f j ( y, z ) k   .
Casting (16) into the VI formulation (10), by taking Q ( x ) =
Q ( y, z ) =
[ r y f 1 ( y, z ) , r
z f 1 ( y, z )] and P ( x ) = P ( y, z ) =
[ r y
[ r  f 1 ]( y, z ) , r
z [ r  f 1 ]( y, z )] , we have that Q is monotone and
L -Lipschitz, P is  -Lipschitz, R is µ -strongly monotone.
Therefore, we can apply Theorems 9 and 10 and obtain the following convergence results for Algorithm 2 applied to (16).
Theorem 11.
Let Assumptions 10 to 12 be satisﬁed with µ    L .
Then, to ﬁnd " -solution of the distributed saddle problem (16) , Algorithm 2 requires
O
[Equation]
rounds and O
[Equation]
⌘ local gradient computations.
Theorem 12.
Let Assumptions 10, 11 (with µ = 0 ), 12 be satisﬁed and   L .
Then, to ﬁnd " -solution of the distributed saddle problem (16) , Algorithm 2 requires
O
[Equation]
and O [Equation]  local computations. Our communication estimates are optimal, as in [9], but our algorithm also achieve optimal local complexity (see Table 1).
Experiments 5.1 Minimization
We consider the Ridge Regression problem
[Equation]
where w is the vector of weights of the model, { x i , y i }
Ni =1 is the training dataset, and  > 0 is the regularization parameter.
We consider a network with 25 workers (simulated on a single-CPU machine), and use two types of datasets, namely: synthetic and real data.
Synthetic data permit to control the similarity constant  .
To do so, we generate data on the server, say { ˆ x i , ˆ y i } n =100 i = .
Data on the workers are generated by adding unbiased Gaussian noise to the server data.
The lower the variance of this noise, the more similar the data, and thus the smaller  .
For simulations with real data, we considered the LIBSVM library [11] and give each agent a full dataset.
Then, each device selects at random a part of size m from the full dataset.
In Section C.2 we explain how the parameters L and  are estimated.
For the synthetic dataset we choose the noise level and the regularization parameter such that L/  =and L/  = 10 5 .
For the real datasets the regularization parameter is chosen such that L/  = 10 6 .
See Table 2 for all values of L,  , µ and m .
For comparison, we use distributed version of Accelerated Gradient Descent (AcGD)
[37] as the basic, classical and optimal method for minimization problems without additional similarity assumptions, as well as state-of-the-art schemes for the similarity condition: DANE [42], DANE-HB [50], SPAG [21] and AccSONATA [46].
The settings of the methods are made as described in the original papers.
For algorithms that assume an absolutely accurate solution of local problems (DANE, SPAG,
AccSONATA), we use AcGD with an accuracy of 10  12 as a subsolver.
Results are summarized in Figure 1–the ﬁrst two ﬁgures from the top left correspond to synthetic data while the other 6 on real data.
The results show that our method outperforms almost all methods in terms of communication (only in one experiment SPAG is slightly faster).
In terms of local iterations, our method is slightly inferior to AcGD, but superior to all other methods.
In Appendix C.1, one can ﬁnd a comparison of our method with a competitor for distributed saddle point problems under similarity assumption.
Acknowledgments
The research in Sections 1–3 was supported by Russian Science Foundation (project No. 21-7130005).
The work in Section 4 was prepared within the framework of the HSE University Basic
Reaearch Programm.
The research of A. Scutari was partially supported by the ONR Grant N.
N00014-21-1-2673.
[Figure]