Optimal Algorithms for Decentralized Stochastic Variational Inequalities
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Variational inequalities are a formalism that includes games, minimization, saddlepoint, and equilibrium problems as special cases.
Methods for variational inequalities are therefore universal approaches for many applied tasks, including machinelearning problems.
This work concentrates on the decentralized setting, which isincreasingly important but not well understood.
In particular, we consider decentralized stochastic (sum-type) variational inequalities over ﬁxed and time-varyingnetworks.
We present lower complexity bounds for both communication and localiterations and construct optimal algorithms that match these lower bounds.
Ouralgorithms are the best among the available literature not only in the decentralizedstochastic case, but also in the decentralized deterministic and non-distributedstochastic cases.
Experimental results conﬁrm the effectiveness of the presentedalgorithms.
Introduction
Variational inequalities are a broad and ﬂexible class of problems that includes minimization, saddlepoint, Nash equilibrium, and ﬁxed point problems as special cases; see [23, 6] for an introduction.
Over the long history of modern research on variational inequalities spanning at least half a century,the community developed their own methods and theory, differing from the approaches in theirsister ﬁeld, optimization.
The ExtraGradient / MirrorProx methods due to [37, 51, 33] have a similarfoundational standing in the variational inequalities ﬁeld that gradient descent occupies in theoptimization literature.
As in the case of gradient descent, many modiﬁcations [30] and variants [63]of these methods were proposed and studied in the variational inequalities literature.
1.1 Applications of variational inequalities
In recent years, there has been a signiﬁcant increase of research activity in the study of variationalinequalities due to new connections to reinforcement learning [55, 31], adversarial training [43], and
GANs [27].
In particular, [20, 25, 46, 19, 41, 57] show that even if one considers the classical (inthe variational inequalities literature) regime involving monotone and strongly monotone inequalities,it is possible to obtain insights, methods and recommendations useful for the GAN community.
In addition to the above modern applications, and besides their many classical applications in appliedmathematics that include economics, equilibrium theory, game theory and optimal control [22],variational inequalities remain popular in supervised learning (with non-separable loss [32]; with nonseparable regularizer [3]), unsupervised learning (discriminative clustering [66]; matrix factorization[4]), image denoising [21, 17], robust optimization [9], and non-smooth optimization via smoothreformulations [53, 52].
1.2 Processing on the edge
With the proliferation of mobile phones, wearables, digital sensors, smart home appliances, and otherdevices capable of capturing, storing and processing data, there is an increased appetite to minethe richness contained in these sources for the beneﬁt of humanity.
However, at the same time, thetraditional centralized approach relying on moving the data into a single proprietary warehouse forprocessing via suitable machine learning methods is problematic, and a new modus operandi is onthe rise: processing the data at the source, on the edge, where it was ﬁrst captured and where it isstored [36, 45, 34], by the client’s devices that own the data.
There are many reasons for a gradualshift in this direction, including energy efﬁciency and data privacy.
A key necessary characteristic for any viable algorithmic approach to work in such a massivelydecentralized regime is the ability to support decentralized processing reﬂecting the fact that thedevices are connected through a network of a potentially complicated topology, possibly varyingin time.
A central authority may be absent in such a system, and the methods need to rely oncommunication patters that correspond to the existing connection links.
1.3 Decentralized algorithms for variational inequalities
In this paper we studyalgorithms for solving variational inequalities over decentralized communication networks .
In this regime, a number of nodes (workers, devices, clients) are connected via a communicationnetwork, represented by a graph.
Each node can perform computations using its local state and data,and is only allowed to communicate with its neighbors in the graph.
Decentralized algorithms over ﬁxed communication networks ﬁnd their applications in sensor networks [58], network resource allocation [8], cooperative control [26], distributed spectrum sensing[7], power system control [24] and, of course, in machine learning [60].
Recently, decentralizedmethods over time-varying networks have gained particular popularity due to their relevance tofederated learning [36, 34], where communication failures between devices are a common problem.
Decentralized minimization methods are well studied [48, 35].
In particular, lower bounds andoptimal algorithms for such problems are known in the ﬁxed [60, 29, 39] and time-varying [38, 40]network
topology regimes.
However, in signiﬁcantly more general and hence potentially much more impactful formalism ofvariational inequalities, the question of optimal and efﬁcient decentralized methods is still open.
Motivated by these considerations, our work is devoted to advancing the algorithmic and theoreticalfoundations of decentralized variational inequalities, in both the ﬁxed and time-varying networkregimes.
1.4 Our contributions and related work
We now brieﬂy summarize our main contributions.
(a) Lower bounds
We present the ﬁrst lower bounds for the communication and local computation complexities ofdecentralized variational inequalities in the stochastic (ﬁnite sum) case, in both the ﬁxed andtime-varying network topology regimes.
See Table 1.
Existing literature contains lower bounds for non-distributed ﬁnite-sum variational inequalities [28],which we recover as a special case.
Existing literature also contains lower bounds for deterministicdecentralized variational inequalities in the ﬁxed [12] and time-varying [11] regimes.
Our boundscovers these results too.
See Table 2 (Appendix F).
(b ) Optimal decentralized algorithms
We construct four new algorithms for stochastic (ﬁnite sum) decentralized variational inequalities: two for ﬁxed networks, and two for time-varying networks.
Two of these algorithms matchour lower bounds, and are therefore optimal in terms of communication and local iterationcomplexities.
These are the ﬁrst algorithms for stochastic (ﬁnite sum) decentralized variationalinequalities over ﬁxed and time-varying networks.
See Table 1.
Moreover, our results offer linear communication complexity for deterministic decentralized stronglymonotone variational inequalities, which is an improvement upon the sublinear results of [13, 12,10, 11].
Additionally, our algorithms have better guarantees on local computations than the methodsdeveloped by [59].
See Table 2 (Appendix F).
Let us also single out a number of works on decentralized saddle point problems or VIs which are notsuitable for comparison with our results: [64, 5] consider non-monotone(minty) problems, [44] doesnot prove convergence, [42] assumes data homogeneity
[Table]
and [61] considers a discrete problem
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] (c) Optimal non-distributed/centralized algorithms We believe it is notable that despite the generality of our setup and algorithms, our results, whenspecialized to handle this simpler case, improve upon the current state-of-the-art results inthe non-distributed/centralized setting.
In particular, unlike existing methods, our algorithmssupport batching : while the complexity of the best available algorithms grows with the batch size,our algorithms are not sensitive to this.
This property is of crucial importance when working in thelarge batch mode, which is used in the practice [15, 70, 68].
See Table 3 (Appendix F).
Numerical experiments on bilinear problems and robust regression problems conﬁrm the practicalefﬁciency of our methods, both in the non-distributed stochastic setup and in the decentralizeddeterministic one.
Problem Setup and Assumptions
We write h x, y i : =
P n i =1 x i y i to denote the standard inner product of vectors x, y 2 R n , wherex i corresponds to the i -th component of x in the standard basis in R n .
This induces the standard` 2 -norm in R n in the following way: k x k := p h x, x i .
To denote the Kronecker product of twomatrices A 2 R m ⇥ m and B 2 R n ⇥ n , we use A ⌦ B 2 R nm ⇥ nm .
The identity matrix of size n ⇥ nis denoted by I n .
We write [ n ] := { 1 , 2 , . . . , n } .
N is the set of positive integers.
2.1 Variational inequality
We study variational inequalities (VI) of the form
Find z ⇤ 2 R d such that h F ( z ⇤ ) , z  z ⇤ i + g ( z )  g ( z ⇤ )  0 ,
[Equation]
[Equation] (1) where F : R d ! R d is an operator, and g : R d ! R [ { + 1 } is a proper lower semicontinuous 109 convex function.
We also assume that g is proximal friendly, i.e. the computation of the operatorprox
[Equation]
= arg
[Equation]
d {
[Equation]
g
[Equation]
[Equation]
2 } (with ⇢ > 0 ) is done for free or costs very low.
To showcase the expressive power of the formalism (1), we now give a few examples of variationalinequalities arising in machine learning.
Example 1 [Convex minimization].
Consider the convex regularized minimization problem:
[Equation]
where f is typically a smooth data-ﬁdelity term, and g a possibly nonsmooth regularizer.
If we deﬁne
F ( z ) :
= r f ( z ) , then it can be proved that z ⇤ 2 dom g is a solution for (1) if and only if z ⇤ 2 dom gis a solution for (2).
So, the regularized optimization problem (2) can be cast as a VI (1).
While minimization problems are widely studied in a separate literature, the next class of problems ismuch more strongly tied to variational inequalities.
Example 2 [Convex-concave saddles].
Consider the convex-concave saddle point problem
[Equation]
where g 1 and g 2 can also be interpreted as regularizers.
If we let F ( z ) :
= F ( x, y ) =[ r x f ( x, y ) , r y f ( x, y )] and g ( z ) =
g ( x, y )
=
g 1 ( x )+ g 2 ( y ) , then it can be proved that z ⇤ 2 dom gis a solution for (1) if and only if z ⇤ 2 dom g is a solution for (3).
So, convex-concave saddle pointproblems (3) can be cast as a VI (1).
Saddle point problems are strongly related to variational inequalities.
In particular, lower bounds forthe former are also valid for the latter.
Moreover, upper bounds for variational inequalities are validfor saddle point problems.
However, what is perhaps more important is that these lower and upperbounds match.
This is in contrast to minimization, where the lower bounds are weaker.
2.2 Decentralized variational inequalities
We consider the decentralized case of problem (1), namely we assume that F is distributed across Mworkers,
[Equation]
while each F m : R d ! R d , m {+
+} [ M ] , has the ﬁnite sum structure
[Equation]
The data describing F m being stored on worker m .
For example, F m,i can correspond to the value ofthe operator on the i th data point of the m -th dataset.
2.3 Assumptions
Assumption 2.1 (Lipschitzness) .
Each operator F m is L -Lipschitz continuous, i.e. for all u, v 2 R dwe have k F m ( u )  F m ( v )
k  L k u  v k .
Further, the collection of operators is L -average Lipschitz continuous, i.e., for all u, v 2 R d it holds
[Equation]
In the context of (2) and (3), L -Lipschitzness of the operator means that the functions f ( z ) andf ( x, y ) are L -smooth.
Assumption 2.2 (Strong monotonicity) .
Each operator F m is µ -strongly monotone, i.e., for allu, v 2 R d we have h F m ( u )  F m ( v ); u  v i  µ k u  v k 2 .
In the context of (2) and (3), strong monotonicity of F means strong convexity of f ( z ) and strongconvexity-strong concavity of f ( x, y ) .
2.4 Communication and gossip
Typically, decentralized communication is realized via a gossip protocol [65, 14, 49], which is merelymatrix-vector multiplication with a gossip matrix W , described below, which is different in the ﬁxedand time-varying cases.
Let L = { z = ( z 1 , . .
. , z M ) > 2 ( R d ) M : z 1 = . . .
=
z M } be the consensusspace .
Assumption 2.3 (Fixed network [60]) .
For a ﬁxed network, communication can be modeled via anundirected connected graph, G = ( V , E ) , where V =
[ n ] are vertices (workers) and E
= { ( i, j ) | i, j {+
+}
V} are edges.
Note that ( i, j ) 2 E if and only if there exists a communication link between agents i 153 and j .
The gossip matrix W satisﬁes the following three assumptions: 1) W is symmetric positivesemi-deﬁnite; 2) ker W  L ; 3) W is supported on the vertices and edges of the network only:w i,j 6 =
if and only if i = j or ( i, j ) 2 E .
To characterize the matrix W , which captures the properties of the network, we denote  max ( W ) =
1as
the maximum eigenvalue of W ,  +min ( W ) as the minimum positive eigenvalue of W , and thecharacteristic number  =  max ( W ) /  +min ( W ) = 1 /  +min ( W ) .
Assumption 2.4 (Time-varying network [50]) .
For a time-varying network, at any moment t , communication network can be modeled as a directed B -connected graph, G ( t ) =
( V , E ( t )) , where
E ( t ) =
{ ( i, j ) |
i, j 2 V} are directed edges.
B -connectedness means that for any time t , thegraph G B ( t ) with the set of edges S t + B  1 ⌧ = t
E ( t ) is connected.
To describe the gossip protocol fortime-varying case, we deﬁne the multi-consensus gossip matrix
[Equation]
One can also observe that multiplication with the matrix W T requires to perform multiplication with
T gossip matrices W ( t ) , . .
, W ( t + T  1) , i.e., it requires T decentralized communications.
Wefurther assume that the gossip matrices W ( t ) (for G ( t ) ) and W B ( t ) satisfy: 1) W ( t ) is supportedon the nodes and edges of the network: w i,j ( t ) {+6 = 
+}
if and only if i = j or ( i, j ) 2 E ( t ) ; 2)ker W ( t )  L ; 3) range W ( t ) ⇢ {
z 2 ( R d ) M : P
Mm {+=
+}
z m = 0 } ; 4) there exists a characteristicnumber   1 such that k W B ( t ) z  z k 2  (1    1 ) k
z k 2 for all z 2 range W B ( t ) .
Lower Bounds
Our lower bounds apply to a speciﬁc class of algorithms which are, loosely speaking, allowed tocommunicate with neighbors, and compute any local ﬁrst-order information.
We now give a formaldeﬁnition.
Deﬁnition 3.1 (Oracle) .
Each agent m has its own local memory M m with initialization M m = { 0 } .
M m is updated as follows.
At each iteration, the algorithm either performs local computations orcommunicates.
• Local computation: At each local iteration, device m can sample uniformly and independently batch S m of any size b from { F m,i } and adds to its M m a ﬁnite number of points z , satisfying
[Equation]
span
[Equation]
for z 0 , z 00 , z 000 2 M m and ⇢ > 0 .
Such a call needs b local computations to collect the batch.
Batch of size n represents F m ;
• Communication: Upon communication rounds among the neighboring nodes, and at communication time t , M m is updated according to
[Equation]
• Output: The ﬁnal global output is calculated as ˆ z 2 span
[Equation]
The structure of the above deﬁnition is typical for distributed lower bounds [60] and for stochasticlower bounds [29].
In particular, Deﬁnition 3.1 includes all the approaches for working withstochastic problems, such as SGD or variance reduction techniques ( SVRG , SARAH ).
Note that whileour algorithm can invoke the deterministic oracle (full F m ) in local computations, in the work onlower bounds in the non-distributed case [28], there is no such a possibility.
This narrows the class ofalgorithms for which results of [28] are valid.
In particular, they can not do SVRG -type updates.
[Equation]
Theorem 3.2 (Lower bound - ﬁxed network) .
For any L, L  µ > 0 and   1 , n 2 N and 191 K, N 2 N , there exists a decentralized variational inequality (satisfying Assumptions 2.1 and 2.2)on R d (where d is sufﬁciently large) with z ⇤ 6 = 0 over a ﬁxed network (satisfying Assumption 2.3)with a gossip matrix W and characteristic number  , such that for any output ˆ z of any procedure(Deﬁnition 3.1) with K communication rounds and N local computations, it holds that E [ k ˆ z  z
⇤ k 2 ]is
[Equation]
where R 20 = k
z {+
 +} z ⇤ k 2 .
Corollary 3.3.
In the setting of Theorem 3.2, the number of communication rounds and localcomputations required to obtain an " -solution (in expectation) is lower bounded by [Equation]
Theorem 3.4 (Lower bound - time varying network) .
For any L, L  µ > 0 and   3 , n 2 N and 200 K, N 2 N , there exist a decentralized variational inequality (satisfying Assumptions 2.1 and 2.2) on
R d (where d is sufﬁciently large) with z ⇤ 6 = 0 over a time-varying network (satisfying Assumption2.4) with a sequence of gossip matrices W ( t ) and characteristic number  , such that for any outputˆ z of any procedure (Deﬁnition 3.1) with K communication rounds and N local computations, it holdsthat E [ k ˆ z  z ⇤ k 2 ] is [Equation] 1+ q 2 L
[Equation] ⌘ · KB  ! R 20 ! and ⌦@
[Equation]
[Equation]
A R [Equation] A , where R 20 = k
z 0  z ⇤ k 2 .
Corollary 3.5.
In the setting of Theorem 3.4, the number of communication rounds and localcomputations required to obtain an " -solution (in expectation) is lower bounded [-byΩ (cid:16) Bχ (cid:16) 
-]by
[Equation] ⌘⌘ and ⌦ [Equation] , respectively.
See proofs for Theorems 3.2 and 3.4 in Appendix C.
The proof uses the idea (example of badfunctions) of non-distributed deterministic lower bounds from [69].
This idea is further extended todistributed stochastic VIs.
Note that in the time-varying case, the lower bounds for communication differ by the constant Bfrom the estimates that were previously encountered in the literature [11].
This is due to the fact thatwe consider a more general setup with a B -connected graph (see Assumption 2.4), while the existingliterature on lower bounds focuses on the simpler B = 1 case.
Optimal Algorithms
Since in the decentralized gossip protocol each local worker m stores its own z m vector, we considerthe problem
Find z ⇤ 2 ( R d ) M such that h F ( z ⇤ ) , z  z ⇤ i + g ( z )  g
( z ⇤ )  0 , {+
+} z 2 ( R d ) M , (9) where we use new notation: z = ( z 1 , . . . , z M ) > and z ⇤ =
( z ⇤ 1 , . . .
, z ⇤ M ) > .
Additionally, here weintroduce the lifted operator F : ( R d ) M !
( R d ) M given as F ( z ) =
( F 1 ( z 1 ) , . . .
, F M ( z M )) > , andthe lifted operator g : ( R d ) M !
R [ { + 1 } deﬁned by g ( z ) =
1 M P Mm =1 g ( z m ) .
One can note that (9) is a set of M unrelated variational inequalities with their own variables.
But the original problem (1) + (5) is a sum of variational inequalities with the same variables:
P
[Equation]
⇤ . To eliminate this issue and move on to problem (1)+ (5), it is easy to get the following modiﬁcation of (9)
Find z ⇤ 2 L such that h F ( z ⇤ ) , z  z ⇤ i + g [Equation]
z
[Equation] L [Equation] (10) where L is the consensus space.
Problem (10) is equivalent to (1) + (5).
Due to Assumptions 2.1 and2.2, F is L -Lipschitz continuous, L -average Lipschitz continuous and µ -strongly monotone.
4.1 Fixed networks
We present Algorithm 1 for ﬁxed networks.
In Appendix B we give a discussion andintuition.
In particular, we give the deterministic variant as well as the non-distributedversion of Algorithm 1.
The next result gives the iteration complexity of Algorithm 1.
Algorithm1:
, batchsize b 2 { 1 , . . . , n } , probability p 2 (
, 1) 2: Initialization: Choose z 0 = w 0 2 (dom g ) M , y {+
+}
L ? .
Put z  1 = z 0 , w  1 = w 0 , y  1 = y3: for k = 0 , 1 , 2 . . .
do 4:
Sample j km, 1 , . .
, j km,b independently from [ n ] 5:
S k
=
{ j km, 1 , . . .
, j km,b } 6:
Sample j k +1 / 2 m,, . . . , j k +1 / 2 m,b independently from [ n ] 7:
S k +1 / 2 = { j k +1 / 2 m,, . . .
, j k +1 / 2 m,b } 8:  k = 1 b P j 2 S k ⇣ F j ( z k )  F j ( w k  1 )
+ ↵ [ F j ( z k )  F j ( z k  1 )] ⌘ + F ( w k  1 ) 9:  k =  k  ( y k + ↵ ( y k  y k  1 )) 10:
z k +1 =
prox ⌘ g ( z k +  ( w k  z k )  ⌘  k ) 11:
k +1 / 2 = 1 b P j 2 S k +1 / F j ( z k +1 )
F j ( w k )  + F ( w k ) 12: y k[-+
-]
+1 =
y k  ✓ ( W ⌦ I d )( z k +1   (  k +1 / {+
 +} y k )) 13: w k +1 =
⇢ z k , with probability p w k , with probability 1  p 14: end for ⇤ F j ( z ) =
( F 1 ,j 1 ,l ( z 1 ) , . . .
, F M,j M,l ( z M )) T , l 2 { 1 , . . .
, b }
Theorem 4.1.
Consider the problem(10) (or (1) + (5) ) under Assumptions 2.1 and 2.2 over a ﬁxed graph
G (Assumption 2.3) with a gossip matrix W .
Let { z k } be the sequence generated by Algorithm 1 with tuning of⌘ , ✓ , ↵ ,  ,  as described in Appendix
D. Then, given " > 0 , the number ofiterations for E [ k z k  z ⇤ k 2 ]  " is
O ⇣h
[Equation]
See the proof in Appendix D. Let usdiscuss the results of Theorem.
First ofall, we are interested in how to obtainthe complexity of communications andlocal computations from iterative complexity.
At each iteration we require (inaverage) O
( b + pn )
local computations,because we need to store batch b twiceand with probability p we update thepoint w k +1 by z k , this requires calculating the full F in the next iteration.
Then, as the optimal p , one can choosep ⇠ b / n .
Then with such choice of pwe have the following local and communication complexities
[Equation]
respectively (since at each iteration Algorithm 1 performs O (1) communications).
Hence, with b = 1 we have the complexities the same as in Table 1.
Depending on max { p n ; p  } ,we have the optimality of either local communications or decentralized communications.
One cannote that it is enough to take b  L p n / L and guarantee the optimal communication complexity (see
Corollary 3.3), but we have non-optimality in local iterations.
To make the algorithm optimal both in terms of communications and local computations, we needto slightly modify it.
One can make it using Chebyshev acceleration (see Algorithm 3 in Appendix
A).
Following [60], we can construct a polynomial P such that 1) P ( W ) is a gossip matrix, 2)multiplication by P ( W ) ⌦ I d requires p  ( W ) multiplications by W (i.e. p  ( W ) communicationrounds) 3)  ( P ( W ))
 4 .
Then we can modify Algorithm 1 by replacing W by P ( W ) .
Then itholds that
Theorem 4.2.
Consider the problem (10) (or (1) + (5) ) under Assumptions 2.1 and 2.2 over a ﬁxedconnected graph G (Assumption 2.3) with a gossip matrix W .
Let { z k } be the sequence generatedby Algorithm 1 with Chebyshev polynomial P ( W ) as a gossip matrix and with tuning of ⌘ , ✓ , ↵ ,  , as described in Appendix D. Then, given " > 0 , the number of iterations for E [ k z k  z ⇤ k 2 ]  " is O ⇣h 1 p + 1 p pb
Lµ + Lµ
i log 1 " ⌘ . In this case, the communication complexity of one iteration is  , and the local complexity (in average)is still O ( b + pn ) .
Then with p =
b / n we get the following local and communication complexities
O ⇣h n + p n Lµ + b
Lµ i log 1 " ⌘ and O ⇣h
p  nb + p  p nb Lµ + p  Lµ i log 1 " ⌘ , respectively.
To get optimal results from Table 1 we just need to take b =
L p n / L .
Algorithm1: Parameters:
⌘ z , ⌘ y , ⌘ x , > 0 , momentums ↵ ,  , ! , ⌧ , parameters ⌫ ,  , batchsize b 2 { 1 , . .
. , n } , probability p 2 (0 , 1) 2: Initialization: Choose z 0 = w 0 2 (dom g ) M , y {+
+}
( R d ) M , x 0 2 L ? .
Put z  1 = z 0 , w  1 = w 0 , y f = y  1 = y 0 , x f = x  1 = x 0 , m 0 = 0 dM 3: for k = 0 , 1 , 2 , . . .
do 4:
Sample j km, 1 , . .
, j km,b independently from [ n ] 5:
S k
=
{ j km, 1 , . . .
, j km,b } 6:
Sample j k +1 / 2 m,, . . . , j k +1 / 2 m,b independently from [ n ] 7:
S k +1 / 2 = { j k +1 / 2 m,, . . .
, j k +1 / 2 m,b } 8:  k = 1 b P j 2 S k ⇣ F j ( z k )  F j ( w k  1 )
+ ↵ [ F j ( z k )  F j ( z k  1 )] ⌘ + F ( w k  1 ) 9:  k z =  k  ⌫ z k
y k  ↵ ( y k  y k  1 ) 10:
z k +1 =
prox ⌘ z g
( z k + ! ( w k  z k )  ⌘ z  kz ) 11:
y kc =
⌧ y k
+ (1  ⌧ ) y kf 12: x kc = ⌧ x k
+ (1  ⌧ ) x kf 13:  ky = ⌫  1 ( y kc + x kc ) + z k +1 +  ( y k + x k + ⌫ z k ) 14:  k +1 / 2 = 1 b
P j 2 S
k [-+
-]+1 /
F j ( z k +1 )
F j ( w k )  + F ( w k ) 15:  kx = ⌫  1 ( y kc + x kc ) +  ( x k +  k +1 / 2 ) 16: y k +
=
y k  ⌘ y  ky 17: x k +1 =
x k  ( W T ( Tk ) ⌦ I d )( ⌘ x  kx + m k ) 18: m k +
= ⌘ x  kx + m k  ( W T ( Tk ) ⌦ I d )( ⌘ x  kx + m k ) 19:
y k +1 f
= y kc + ⌧
( y k +1  y k ) 20: x k +1 f
= x kc  ✓ ( W T ( Tk ) ⌦ I d )( y kc + x kc ) 21: w k [-+
-]+1 =
⇢ z k , with probability p w k , with probability 1  p 22: end for
In contrast to algorithms of [13, 12](the closest papers in theoretical convergence), our Algorithm 1 needsmulti-consensus/Chebyshev acceleration for both optimal rates, but canwork without these additional procedures.
Algorithms [13, 12] requires
O ( p  log "  1 ) iterations for Chebyshev acceleration, which makes the algorithms less practical.
4.2 Time-varying networks
We present Algorithm 2 for timevarying networks.
It needs to compute
W T using (6), it requires T communications.
In Appendix B we give a discussion and intuition of this algorithm.
The next result gives the iteration complexity of Algorithm 2.
Theorem 4.3.
Consider the problem(10) (or (1) + (5) ) under Assumptions 2.1 and 2.2 over a sequence oftime-varying graphs G ( k ) (Assumption2.4) with gossip matrices W ( k ) .
Let{ z k } be the sequence generated by 298 Algorithm 2 with T  B and tun- 299 ing of parameters as described in Appendix E. Let the choice of T guarantees contraction property (Assumption2.4 point 4) with  ( T ) .
Then , given" > 0 , the number of iterations for E [ k z k  z ⇤ k 2 ]  " is˜ O ⇣  2 ( T ) + 1 p +  ( T ) Lµ + 1 p bp Lµ ⌘ .
Note that an important detail of themethod is that T  B .
This limitation is due to the fact that network is
B -connected.
In particular, for B > 1 it can happen that in some communications we use emptygraphs.
Therefore, the requirement T  B is natural to guarantee the contraction property (Assumption 2.4, point 4).
This means that if B > 1 we have to use multi-consensus (6).
But with B = T = 1 , we can avoid multi-consensus, let us this case ﬁrst.
The same way as in ﬁxedgraph case we choose p = b / n .
Then we get the estimates on communications and local calls
[Equation]
If we put b = 1 , we have the same estimates as in Table 1.
Now we consider general case with any B .
We use a multi-gossip step and take T > 1 .
In particular,let us choose T = B · d  ln 2 e .
Than using (6) and point 4 of Assumption 2.4, we can guarantee thatk W T ( t )
z  z k 2  12 k z k 2 .
Therefore,  ( T ) = 2 , but we need T communications per iteration.
Withp = b / n and b = L p n / L , the iteration complexity from Theorem 4.3 can be rewritten as follows
[Equation]
Using that per iteration we make O ( B · d  ln 2 e ) communications and O  L p n / L  local computations,we get
O ⇣h
[Equation]
calls.
These results are reﬂected in Table 1.
Experiments
We now perform several experiments with the goal of corroborating our theoretical results.
Notethough that we are the ﬁrst who consider the decentralized stochastic (ﬁnite-sum) setting for VIs, andhence there are no competing methods.
Therefore, we compare the non-distributed ﬁnite sum settingand decentralized deterministic setting separately.
5.1 Variance reduction
In this section, we compare the main methods for solving strongly monotone non-distributed stochastic(ﬁnite-sum) variational inequalities with non-distributed version of our Algorithm 1.
Problem : We ﬁrst consider robust linear regression:
[Equation]
where 4 d is the unit simplex in R d .
We generated positive deﬁnite matrices
A i and vectors a i , b irandomly.
The dimensions of the problem are d x = 50 and d y = 50 .
[-1e
-]
[-1.0 Epoch 1e4 1010 5 
-]
Setting.
For comparison, we took methods from 333 Table 3.
In particular, we chose EG-Alc-Alg1 and
EG-Alc-Alg2 from [1], EG-Car from [16].
Allmethods are tuned as described in the theory ofthe corresponding papers.
We run all methodswith different batch sizes.
The comparison criterion is the number of epochs (one full gradient= epoch).
Results.
The plots from Fig.
1 show that our
Algorithm 1 is ahead of other methods for anybatch size (including b = 1 ).
5.2 Decentralized methods
In this section, we compare the state-of-the-art methods for solving strongly monotone decentralizedvariational inequalities over ﬁxed networks with our Algorithm 1.
Problem : We now considered another variant of robust linear regression:
[Equation]
where w are model weights, { ( x i , y i ) }
Ni =1 are pairs of the training data, r i are noise vectors, and and  are regularization parameters.
The noises r i resist training the model, thereby inducing morerobustness and stability.
Setting.
For comparison, we took methods from 351 Table 1 for decentralized problems over ﬁxednetworks.
In particular, we chose EGD-GT from[47], EGD-Con from [12] and Sliding from [13].
For a fair comparison, we consider the deterministic setup, i.e., each worker can compute fullgradients.
We took dataset a7a and a9a from
LiBSVM [18] and divided unevenly across theworkers.
For communication networks we chosethe star and the grid topologies.
All methodsare tuned as described in the theory of the corresponding papers.
The comparison criterion is the number of communication rounds.
Results.
The plots from Fig.
2 show that our Algorithm 1 is ahead of other methods.
Among otherthings, it is ahead of Sliding from [13], which has a fast theoretical communication complexity.
However, this happens when the dataset is relatively homogeneous and uniformly divided across thedevices.
In our setting, this is not the case.