Optimal Algorithms for Decentralized Stochastic Variational Inequalities
Abstract
Variational inequalities are a formalism that includes games, minimization, saddle point, and equilibrium problems as special cases.
Methods for variational inequalities are therefore universal approaches for many applied tasks, including machine learning problems.
This work concentrates on the decentralized setting, which is increasingly important but not well understood.
In particular, we consider decentralized stochastic (sum-type) variational inequalities over ﬁxed and time-varying networks.
We present lower complexity bounds for both communication and local iterations and construct optimal algorithms that match these lower bounds.
Our algorithms are the best among the available literature not only in the decentralized stochastic case, but also in the decentralized deterministic and non-distributed stochastic cases.
Experimental results conﬁrm the effectiveness of the presented algorithms.
Introduction
Variational inequalities are a broad and ﬂexible class of problems that includes minimization, saddle point, Nash equilibrium, and ﬁxed point problems as special cases; see [24, 6] for an introduction.
Over the long history of modern research on variational inequalities spanning at least half a century the community developed their own methods and theory, differing from the approaches in their sister ﬁeld, optimization.
The ExtraGradient / MirrorProx methods due to [38, 52, 34] have a similar foundational standing in the variational inequalities ﬁeld that gradient descent occupies in the optimization literature.
As in the case of gradient descent, many modiﬁcations [31] and variants [66] of these methods were proposed and studied in the variational inequalities literature.
1.1 Applications of variational inequalities
In recent years, there has been a signiﬁcant increase of research activity in the study of variational inequalities due to new connections to reinforcement learning [58, 32], adversarial training [44], and
GANs [28].
In particular, [21, 26, 47, 20, 42, 60] show that even if one considers the classical (in the variational inequalities literature) regime involving monotone and strongly monotone inequalities, it is possible to obtain insights, methods and recommendations useful for the GAN community.
In addition to the above modern applications, and besides their many classical applications in applied mathematics that include economics, equilibrium theory, game theory and optimal control [23], variational inequalities remain popular in supervised learning (with non-separable loss [33]; with nonseparable regularizer [3]), unsupervised learning (discriminative clustering [69]; matrix factorization [4]), image denoising [22, 18], robust optimization [9], and non-smooth optimization via smooth reformulations [56, 53].
1.2 Processing on the edge
With the proliferation of mobile phones, wearables, digital sensors, smart home appliances, and other devices capable of capturing, storing and processing data, there is an increased appetite to mine the richness contained in these sources for the beneﬁt of humanity.
However, at the same time, the traditional centralized approach relying on moving the data into a single proprietary warehouse for processing via suitable machine learning methods is problematic, and a new modus operandi is on the rise: processing the data at the source, on the edge, where it was ﬁrst captured and where it is stored [37, 46, 35], by the client’s devices that own the data.
There are many reasons for a gradual shift in this direction, including energy efﬁciency and data privacy.
A key necessary characteristic for any viable algorithmic approach to work in such a massively decentralized regime is the ability to support decentralized processing reﬂecting the fact that the devices are connected through a network of a potentially complicated topology, possibly varying in time.
A central authority may be absent in such a system, and the methods need to rely on communication patters that correspond to the existing connection links.
1.3 Decentralized algorithms for variational inequalities
In this paper we study algorithms for solving variational inequalities over decentralized communication networks .
In this regime, a number of nodes (workers, devices, clients) are connected via a communication network, represented by a graph.
Each node can perform computations using its local state and data, and is only allowed to communicate with its neighbors in the graph.
Decentralized algorithms over ﬁxed communication networks ﬁnd their applications in sensor networks [61], network resource allocation [8], cooperative control [27], distributed spectrum sensing [7], power system control [25] and, of course, in machine learning [63].
Recently, decentralized methods over time-varying networks have gained particular popularity due to their relevance to federated learning [37, 35], where communication failures between devices are a common problem.
Decentralized minimization methods are well studied [49, 36].
In particular, lower bounds and optimal algorithms for such problems are known in the ﬁxed [63, 30, 40] and time-varying [39, 41] network topology regimes.
However, in signiﬁcantly more general and hence potentially much more impactful formalism of variational inequalities, the question of optimal and efﬁcient decentralized methods is still open.
Motivated by these considerations, our work is devoted to advancing the algorithmic and theoretical foundations of decentralized variational inequalities, in both the ﬁxed and time-varying network regimes.
1.4 Our contributions and related work
We now brieﬂy summarize our main contributions.
(a) Lower bounds
We present the ﬁrst lower bounds for the communication and local computation complexities of decentralized variational inequalities in the stochastic (ﬁnite sum) case, in both the ﬁxed and time-varying network topology regimes.
See Table 1.
Existing literature contains lower bounds for non-distributed ﬁnite-sum variational inequalities [29], which we recover as a special case.
Existing literature also contains lower bounds for deterministic decentralized variational inequalities in the ﬁxed [13] and time-varying [12] regimes.
Our bounds covers these results too.
See Table 2 (Appendix G).
(b) Optimal decentralized algorithms
We construct four new algorithms for stochastic (ﬁnite sum) decentralized variational inequalities: two for ﬁxed networks, and two for time-varying networks.
Two of these algorithms match our lower bounds, and are therefore optimal in terms of communication and local iteration complexities.
These are the ﬁrst algorithms for stochastic (ﬁnite sum) decentralized variational inequalities over ﬁxed and time-varying networks.
See Table 1.
Moreover, our results offer linear communication complexity for deterministic decentralized strongly monotone variational inequalities, which is an improvement upon the sublinear results of [14, 13, 10, 12].
Additionally, our algorithms have better guarantees on local computations than the methods developed by [62].
See Table 2 (Appendix G).
Let us also single out a number of works on decentralized saddle point problems or VIs which are not suitable for comparison with our results: [67, 5] consider non-monotone(minty) problems, [45] does not prove convergence, [43] assumes data homogeneity, and [64] considers a discrete problem.
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
(c ) Optimal non-distributed/centralized algorithms
We believe it is notable that despite the generality of our setup and algorithms, our results, when specialized to handle this simpler case, improve upon the current state-of-the-art results in the non-distributed/centralized setting.
In particular, unlike existing methods, our algorithms support batching : while the complexity of the best available algorithms grows with the batch size, our algorithms are not sensitive to this.
This property is of crucial importance when working in the large batch mode, which is used in the practice [16, 73, 71].
See Table 3 (Appendix G) [Table]
(d) Experiments
Numerical experiments on bilinear problems and robust regression problems conﬁrm the practical efﬁciency of our methods, both in the non-distributed stochastic setup and in the decentralized deterministic one.
Problem Setup and Assumptions
We write h x, y i :=
P ni =1 x i y i to denote the standard inner product of vectors x, y 2 R n , where x i corresponds to the i -th component of x in the standard basis in R n .
This induces the standard ` 2 -norm in R n in the following way: k x k :=
p h x, x i .
To denote the Kronecker product of two matrices A 2 R m ⇥ m and B 2 R n ⇥ n , we use A ⌦ B 2 R nm ⇥ nm .
The identity matrix of size n ⇥ n is denoted by I n .
We write [ n ] := { 1 , 2 , . . . , n } .
N is the set of positive integers.
2.1 Variational inequality
We study variational inequalities (VI) of the form
Find z ⇤ 2 R d such that h F ( z ⇤ ) , z  z
⇤ i
+ g ( z )  g
( z ⇤ )  0 ,
[Equation]
where F : R d !
R d is an operator, and g : R d !
R [ { + 1 } is a proper lower semicontinuous convex function.
We also assume that g is proximal friendly, i.e. the computation of the operator prox ⇢ g ( z ) =
arg min
y 2 R d { ⇢ g ( y ) + 12 k y
z k 2 } (with ⇢ > 0 ) is done for free or costs very low.
To showcase the expressive power of the formalism (1), we now give a few examples of variational inequalities arising in machine learning.
Example 1 [Convex minimization].
Consider the convex regularized minimization problem:
[Equation]
where f is typically a smooth data-ﬁdelity term, and g a possibly nonsmooth regularizer.
If we deﬁne
F ( z ) :
= r f ( z ) , then it can be proved that z ⇤ 2 dom g is a solution for (1) if and only if z ⇤ 2 dom g is a solution for (2).
So, the regularized optimization problem (2) can be cast as a VI (1).
While minimization problems are widely studied in a separate literature, the next class of problems is much more strongly tied to variational inequalities.
Example 2 [Convex-concave saddles].
Consider the convex-concave saddle point problem
[Equation]
where g 1 and g 2 can also be interpreted as regularizers.
If we let F ( z ) :
= F ( x, y ) = [ r x f ( x, y ) , r
y f ( x, y )] and g ( z ) =
g ( x, y )
=
g 1 ( x )+ g 2 ( y ) , then it can be proved that z ⇤ 2 dom g is a solution for (1) if and only if z ⇤ 2 dom g is a solution for (3).
So, convex-concave saddle point problems (3) can be cast as a VI (1).
Saddle point problems are strongly related to variational inequalities.
In particular, lower bounds for the former are also valid for the latter.
Moreover, upper bounds for variational inequalities are valid for saddle point problems.
However, what is perhaps more important is that these lower and upper bounds match.
This is in contrast to minimization, where the lower bounds are weaker.
2.2 Decentralized variational inequalities
We consider the decentralized case of problem (1), namely we assume that F is distributed across M workers,
[Equation]
while each F m : R d ! R d , m
[ M ] , has the ﬁnite sum structure
[Equation]
The data describing F m being stored on worker m .
For example, F m,i can correspond to the value of
2.3 Assumptions
Assumption 2.1 (Lipschitzness) .
Each operator F m is L -Lipschitz continuous, i.e. for all u, v 2 R d we have k F m ( u )  F m ( v ) k  L k u  v k .
Further, the collection of operators is L -average Lipschitz continuous, i.e., for all u, v 2 R d it holds 1 n P ni =1 k F m,i ( u )  F m,i ( [Equation]
[Equation]
k 2 . In the context of (2) and (3), L -Lipschitzness of the operator means that the functions f ( z ) and f ( x, y ) are L -smooth.
Assumption 2.2 (Strong monotonicity) .
Each operator F m is µ -strongly monotone, i.e., for all u, v 2 R d we have h F m ( u )  F m ( v ); u  v i  µ k u  v k 2 .
In the context of (2) and (3), strong monotonicity of F means strong convexity of f ( z ) and strong convexity-strong concavity of f ( x, y ) .
2.4 Communication and gossip
Typically, decentralized communication is realized via a gossip protocol [68, 15, 50], which is merely matrix-vector multiplication with a gossip matrix W , described below, which is different in the ﬁxed and time-varying cases.
Let L = { z = ( z 1 , . .
. , z M ) > 2 ( R d ) M : z 1 = . . .
=
z M } be the consensus space .
Assumption 2.3 (Fixed network [63]) .
For a ﬁxed network, communication can be modeled via an undirected connected graph, G = ( V , E ) , where V =
[ n ] are vertices (workers) and E
= { ( i, j ) | i, j[-
-]
V} are edges.
Note that ( i, j ) 2 E if and only if there exists a communication link between agents i and j .
The gossip matrix W satisﬁes the following three assumptions: 1) W is symmetric positive semi-deﬁnite; 2) ker W  L ; 3) W is supported on the vertices and edges of the network only: w i,j 6 =
if and only if i = j or ( i, j ) 2 E .
To characterize the matrix W , which captures the properties of the network, we denote  max ( W ) =
as the maximum eigenvalue of W ,  +min ( W ) as the minimum positive eigenvalue of W , and the characteristic number  =  max ( W ) /  +min ( W ) = 1 /  +min ( W ) .
Assumption 2.4 (Time-varying network [51]) .
For a time-varying network, at any moment t , communication network can be modeled as a directed B -connected graph, G ( t ) =
( V , E ( t )) , where
E ( t ) =
{ ( i, j ) |
i, j 2 V} are directed edges.
B -connectedness means that for any time t , the graph G B ( t ) with the set of edges
[Equation]
E ( t ) is connected.
To describe the gossip protocol for time-varying case, we deﬁne the multi-consensus gossip matrix
[Equation]
One can also observe that multiplication with the matrix W T requires to perform multiplication with
T gossip matrices W ( t ) , . .
, W ( t + T  1) , i.e., it requires T decentralized communications.
We further assume that the gossip matrices W ( t ) (for G ( t ) ) and W B ( t ) satisfy: 1) W ( t ) is supported on the nodes and edges of the network: w i,j ( t ) 6 =
if and only if i = j or ( i, j ) 2 E ( t ) ; 2) ker W ( t )  L ; 3) range W ( t ) ⇢ {
z 2 ( R d ) M :
P
Mm =
z m = 0 } ; 4) there exists a characteristic number   1 such that k W B ( t )
z  z k 2  (1    1 ) k
z k 2 for all z 2 range W B ( t ) .
Lower Bounds
Our lower bounds apply to a speciﬁc class of algorithms which are, loosely speaking, allowed to communicate with neighbors, and compute any local ﬁrst-order information.
We now give a formal deﬁnition.
Deﬁnition 3.1 (Oracle) .
Each agent m has its own local memory M m with initialization M m = { 0 } .
M m is updated as follows.
At each iteration, the algorithm either performs local computations or communicates.
• Local computation: At each local iteration, device m can sample uniformly and independently batch S m of any size b from { F m,i } and adds to its M m a ﬁnite number of points z , satisfying
[Equation]
for
z 0 , z 00 , z 000 2 M m and ⇢ > 0 .
Such a call needs b local computations to collect the batch.
Batch of size n represents F m ; • Communication: Upon communication rounds among the neighboring nodes, and at communication time t , M m is updated according to
[Equation]
• Output: The ﬁnal global output is calculated as ˆ z 2 span
[Equation]
The structure of the above deﬁnition is typical for distributed lower bounds [63] and for stochastic lower bounds [30].
In particular, Deﬁnition 3.1 includes all the approaches for working with stochastic problems, such as SGD or variance reduction techniques ( SVRG , SARAH ).
Note that while our algorithm can invoke the deterministic oracle (full F m ) in local computations, in the work on lower bounds in the non-distributed case [29], there is no such a possibility.
This narrows the class of algorithms for which results of [29] are valid.
In particular, they can not do SVRG -type updates.
Theorem 3.2 (Lower bound - ﬁxed network) .
For any L  µ > 0 and   1 , n 2 N and K, N 2 N , there exists a decentralized variational inequality (satisfying Assumptions 2.1 and 2.2) on R d (where d is sufﬁciently large) with z ⇤ 6 = 0 over a ﬁxed network (satisfying Assumption 2.3) with a gossip matrix W and characteristic number  , such that for any output ˆ z of any procedure (Deﬁnition 3.1) with K communication rounds and N local computations, it holds that E [ k ˆ z  z
⇤ k 2 ] is
[Equation]
where R 20 =
k
z
z ⇤ k 2 and L = L p n .
Corollary 3.3.
In the setting of Theorem 3.2, the number of communication rounds and local computations required to obtain an " -solution (in expectation) is lower bounded by [Equation] and ⌦
[Equation]
, respectively.
Theorem 3.4 (Lower bound - time varying network) .
For any L  µ > 0 and   3 , n 2 N and K, N 2 N , there exist a decentralized variational inequality (satisfying Assumptions 2.1 and 2.2) on
R d (where d is sufﬁciently large) with z ⇤ 6 = 0 over a time-varying network (satisfying Assumption 2.4) with a sequence of gossip matrices W ( t ) and characteristic number  , such that for any output ˆ z of any procedure (Deﬁnition 3.1) with K communication rounds and N local computations, it holds that E [ k ˆ z  z
⇤ k 2 ] is
[Equation]
[Equation]
where R 20 =
k
{+z 
+}
z ⇤ k 2 and L = L p n .
Corollary 3.5.
In the setting of Theorem 3.4, the number of communication rounds and local computations required to obtain an " -solution (in expectation) is lower bounded by
[Equation]
and ⌦ ⇣⇣
[Equation]
, respectively.
See proofs for Theorems 3.2 and 3.4 in Appendix C.
The proof uses the idea (example of bad functions) of non-distributed deterministic lower bounds from [72].
This idea is further extended to distributed stochastic VIs.
Note that in the time-varying case, the lower bounds for communication differ by the constant B from the estimates that were previously encountered in the literature [12].
This is due to the fact that we consider a more general setup with a B -connected graph (see Assumption 2.4), while the existing literature on lower bounds focuses on the simpler B = 1 case.
Optimal Algorithms
Since in the decentralized gossip protocol each local worker m stores its own z m vector, we consider the problem
Find z [Equation] such that h F [Equation]
z ⇤
[Equation] , z
[Equation] z[-⇤ )  0 , 
-]
[Equation] where we use new notation: z = ( z 1 , .
z M ) > and z ⇤ =
( z ⇤ 1 , . . .
, z ⇤ M ) > .
Additionally, here we introduce the lifted operator F : ( R d ) M !
( R d ) M given as F ( z )
=
( F 1 ( z 1 ) , . . .
, F M ( z M )) > , and the lifted operator g : ( R d ) M !
R [ { + 1 } deﬁned by g ( z ) =
1 M
[Equation]
One can note that (9) is a set of M unrelated variational inequalities with their own variables.
But the original problem (1) + (5) is a sum of variational inequalities with the same variables:
P
[Equation]
⇤ .
To eliminate this issue and move on to problem (1) + (5), it is easy to get the following modiﬁcation of (9)
Find z ⇤ 2 L such that h F ( z ⇤ )
[Equation]
z
[Equation]
where L is the consensus space.
Problem (10) is equivalent to (1) + (5).
Due to Assumptions 2.1 and 2.2, F is L -Lipschitz continuous, L -average Lipschitz continuous and µ -strongly monotone.
4.1 Fixed networks
We present Algorithm 1 for ﬁxed networks.
In Appendix B we give a discussion and intuition.
In particular, we give the deterministic variant as well as the non-distributed version of Algorithm 1.
The next result gives the iteration complexity of Algorithm 1.
Algorithm1:
Parameters: Stepsizes ⌘ , ✓ > 0 , momentums ↵ ,  ,  , batchsize b 2 { 1 , . . . , n } , probability p 2 (0 , 1) 2: Initialization: Choose z 0 = w 0 2 (dom g ) M , y
L ? .
Put z  1 = z 0 , w  1 = w 0 , y  1 = y3: for k = 0 , 1 , 2 . . .
do 4:
Sample j km, 1 , . .
, j km,b independently from [ n ] 5:
S k
=
{ j km, 1 , . . .
, j km,b } 6:
Sample j k +1 / 2 m,, . . . , j k +1 / 2 m,b independently from [ n ] 7:
S k +1 / 2 = { j k +1 / 2 m,, . . .
, j k +1 / 2 m,b } 8:  k = 1 b
P j 2 S k ⇣
F j ( z k )  F j ( w k  1 )
+ ↵ [ F j ( z k )  F j ( z k  1 )] ⌘ + F ( w k  1 ) 9:  k =
k  ( y k + ↵ ( y k  y k  1 )) 10:
z k +1 =
prox ⌘ g ( z k +  ( w k
z k )  ⌘  k ) 11:
k +1 / 2 = 1 b
P j 2 S
k +1 /
F j ( z k +1 )
F j ( w k )  + F ( w k ) 12: y k
+1 =
y k  ✓ ( W ⌦ I d )( z k +1   (  k +1 /
y k )) 13: w k +1 =
⇢ z k , with probability p w k , with probability 1  p 14: end for ⇤ F j ( z )
=
( F 1 ,j 1 ,l ( z 1 ) , . . .
, F M,j M,l ( z M )) T , l 2 { 1 , . . .
, b }
Theorem 4.1.
Consider the problem (10) (or (1) + (5) ) under Assumptions 2.1 and 2.2 over a ﬁxed graph
G (Assumption 2.3) with a gossip matrix W .
Let { z k } be the sequence generated by Algorithm 1 with tuning of ⌘ , ✓ , ↵ ,  ,  as described in Appendix
D. Then, given " > 0 , the number of iterations for E [ k z k  z ⇤ k 2 ]  " is
O ⇣h
[Equation]
See the proof in Appendix D. Let us discuss the results of Theorem.
First of all, we are interested in how to obtain the complexity of communications and local computations from iterative complexity.
At each iteration we require (in average) O ( b + pn ) local computations, because we need to store batch b twice and with probability p we update the point w k +1 by z k , this requires calculating the full F in the next iteration.
Then, as the optimal p , one can choose p ⇠ b / n .
Then with such choice of p we have the following local and communication complexities
[Equation]
respectively (since at each iteration Algorithm 1 performs O (1) communications).
Hence, with b = 1 we have the complexities the same as in Table 1.
Depending on max { p n ; p  } , we have the optimality of either local communications or decentralized communications.
One can note that it is enough to take b  L p n / L and guarantee the optimal communication complexity (see
Corollary 3.3), but we have non-optimality in local iterations.
To make the algorithm optimal both in terms of communications and local computations, we need to slightly modify it.
One can make it using Chebyshev acceleration (see Algorithm 3 in Appendix
A).
Following [63], we can construct a polynomial P such that 1) P ( W ) is a gossip matrix, 2) multiplication by P ( W ) ⌦ I
d requires p  ( W ) multiplications by W (i.e. p  ( W ) communication rounds)
3)  ( P ( W ))  4 .
Then we can modify Algorithm 1 by replacing W by P ( W ) and get
Theorem 4.2.
Consider the problem (10) (or (1) + (5) ) under Assumptions 2.1 and 2.2 over a ﬁxed connected graph G (Assumption 2.3) with a gossip matrix W .
Let { z k } be the sequence generated by Algorithm 1 with Chebyshev polynomial P ( W ) as a gossip matrix and with tuning of ⌘ , ✓ , ↵ ,  ,  as described in Appendix D. Then, given " > 0 , the number of iterations for E [ k z k  z ⇤ k 2 ]  " is
[Equation]
In this case, the communication complexity of one iteration is  , and the local complexity (in average) is still O ( b + pn ) .
Then with p =
b / n we get the following local and communication complexities
O ⇣h [Equation]
i log [Equation] ⌘ and O
[Equation]
respectively.
To get optimal results from Table 1 we just need to take b =
L p n / L .
Algorithm
↵ ,  , ! , ⌧ , parameters ⌫ ,  , batchsize b 2 { 1 , . . . , n } , probability p 2 (0 , 1) 2: Initialization: Choose z 0 = w 0 2 (dom g ) M , y
( R d ) M , x 0 2 L ? .
Put z  1 = z 0 , w  1 = w 0 , y f = y  1 = y 0 , x f = x  1 = x 0 , m 0 = 0 dM 3: for k = 0 , 1 , 2 , . . .
do 4:
Sample j km, 1 , . .
, j km,b independently from [ n ] 5:
S k
=
{ j km, 1 , . . .
, j km,b } 6:
Sample j k +1 / 2 m,, . . . , j k +1 / 2 m,b independently from [ n ] 7:
S k +1 / 2 = { j k +1 / 2 m,, . . .
, j k +1 / 2 m,b } 8:  k = 1 b
P j 2 S k ⇣
F j ( z k )  F j ( w k  1 )
+ ↵ [ F j ( z k )  F j ( z k  1 )] ⌘ + F ( w k  1 ) 9:  kz =
k
⌫ z k
y k
↵ ( y k  y k  1 ) 10:
z k +1 =
prox ⌘ z g
( z k + !
( w k  z k )  ⌘ z  k z ) 11:
y
kc =
⌧ y k
+ (1  ⌧ ) y kf 12: x kc = ⌧ x k
+ (1  ⌧ ) x kf 13:  ky = ⌫  1 ( y kc + x kc ) + z k +1 +  ( y k + x k + ⌫ z k ) 14:  k +1 / 2 = 1 b
P j 2 S
k +1 /
F j ( z k +1 )
F j ( w k )  + F ( w k ) 15:  kx = ⌫  1 ( y kc + x kc )
+  ( x k +  k +1 / 2 ) 16: y k
+
=
y k  ⌘ y  ky 17: x k +1 =
x k  ( W T ( Tk ) ⌦ I d )( ⌘ x  kx + m k ) 18: m k
+
=
⌘ x  kx + m k  ( W T ( Tk ) ⌦ I d )( ⌘ x  kx + m k ) 19:
y k +1 f
= y kc + ⌧
( y k +1  y k ) 20: x k +1 f
= x kc  ✓ ( W T ( Tk ) ⌦ I d )( y kc + x kc ) 21: w k +1 =
⇢ z k , with probability p w k , with probability 1  p 22: end for
In contrast to algorithms of [14, 13] (the closest papers in theoretical convergence), our Algorithm 1 needs multi-consensus/Chebyshev acceleration for both optimal rates, but can work without these additional procedures.
Algorithms [14, 13] requires
O ( p  log "  1 ) iterations for Chebyshev acceleration, which makes the algorithms less practical.
4.2 Time-varying networks
We present Algorithm 2 for timevarying networks.
It needs to compute
W T using (6), it requires T communications.
In Appendix B we give a discussion and intuition of this algorithm.
The next result gives the iteration complexity of Algorithm 2.
Theorem 4.3.
Consider the problem (10) (or (1) + (5) ) under Assumptions 2.1 and 2.2 over a sequence of time-varying graphs G ( k ) (Assumption 2.4) with gossip matrices W ( k ) .
Let { z k } be the sequence generated by Algorithm 2 with T  B and tun- ing of parameters as described in Appendix E. Let the choice of T guarantees contraction property (Assumption 2.4 point 4) with  ( T ) .
Then , given " > 0 , the number of iterations for
E [ k z k  z ⇤ k 2 ]  " is [Equation]
Note that an important detail of the method is that T  B .
This limitation is due to the fact that network is B -connected.
In particular, for B > 1 it can happen that in some communications we use empty graphs.
Therefore, the requirement T  B is natural to guarantee the contraction property (Assumption 2.4, point 4).
This means that if B > 1 we have to use multi-consensus (6).
But with B = T = 1 , we can avoid multi-consensus, let us this case ﬁrst.
The same way as in ﬁxed graph case we choose p
= b / n .
Then we get the estimates on communications and local calls
[Equation]
If we put b = 1 , we have the same estimates as in Table 1.
Now we consider general case with any B .
We use a multi-gossip step and take T > 1 .
In particular, let us choose T = B · d  ln 2 e .
Than using (6) and point 4 of Assumption 2.4, we can guarantee that k [Equation]
z k 2  12 k z k 2 .
Therefore,  ( T ) = 2 , but we need T communications per iteration.
With p = b / n and b =
L p n / L , the iteration complexity from Theorem 4.3 can be rewritten as follows
[Equation]
Using that per iteration we make O ( B · d  ln 2 e ) communications and O
L p n / L  local computations, we get
[Equation]
These results are reﬂected in Table 1.
Experiments
We now perform several experiments with the goal of corroborating our theoretical results.
Note though that we are the ﬁrst who consider the decentralized stochastic (ﬁnite-sum) setting for VIs, and hence there are no competing methods.
Therefore, we compare the non-distributed ﬁnite sum setting and decentralized deterministic setting separately.
5.1 Variance reduction
In this section, we compare the main methods for solving strongly monotone non-distributed stochastic (ﬁnite-sum) variational inequalities with non-distributed version of our Algorithm 1.
Problem. We ﬁrst consider bilinear problem:
[Equation]
where 4 d is the unit simplex in R d .
We use the same experimental setup as in [1], in particular we consider policeman and burglar matrix from [54] and two test matrices from [55].
Setting.
For comparison, we took methods from Table 3.
In particular, we chose EG-Alc-Alg and EG-Alc-Alg2 from [1], EG-Car from [17].
The parameters of all methods are selected in two ways: 1) as described in the theory of the corresponding papers, and 2) tuned for the best convergence.
We run all methods with different batch sizes.
The comparison criterion is the number of epochs (one full gradient = epoch).
Results.
The plots from Figure 1 show that in the case of a theoretical choice of parameters, our Algorithm 1 is ahead of other methods for any batch size (including b = 1 ).
In the case of tuning parameters, the specialized method from [17] is better than our algorithm.
See more experiments with other matrices in Appendix F.1.
5.2 Decentralized methods
In this section, we compare the state-of-the-art methods for solving strongly monotone decentralized variational inequalities over ﬁxed and time-varying networks with our Algorithms 1 and 2.
Problem. We now consider robust linear regression:
[Equation]
where w are model weights, { ( x i , y i ) }
Ni =1 are pairs of the training data, r i are noise vectors, and  and  are regularization parameters.
The noises r i resist training the model, thereby inducing more robustness and stability.
Setting.
For comparison, we took methods from Table 1 for decentralized problems over ﬁxed and time-varying networks.
In particular, we choose EGD-GT from [48], EGD-Con from [13, 12] and Sliding from [14].
Note that only EGD-Con has a theory for ﬁxed and time-varying networks, despite this, we use all methods in both cases.
For a fair comparison, we consider the deterministic setup, i.e., each worker can compute full gradients.
We take datasets from LiBSVM [19] and divided unevenly across M = 25 workers.
For communication networks we chose the star, the ring and the grid topologies.
For time-varying networks, the topologies remain the same, but the locations of the vertices in them change randomly. All methods are tuned for the best convergence.
The comparison criterion is the number of communication rounds.
5.2.1 Fixed networks
Results.
The plots from Figure 2 show that our Algorithm 1 is ahead of other methods.
Among other things, it is ahead of Sliding from [14], which has a fast theoretical communication complexity.
However, this happens when the dataset is relatively homogeneous and uniformly divided across the devices.
In our setting, this is not the case.
See more experiments with other datasets in Appendix F.2.1.
5.2.2 Time-varying networks
Results.
The plots from Figure 3 show that our Algorithm 2 is ahead of other methods in the case of the grid network.
In the case of the ring topology, EGD-Con shows the best results.
Indeed, such an algorithm in fact implements centralized communications via a decentralized protocol and this approach is not always the fastest, but reliable.
See more experiments with other datasets in Appendix F.2.2.
Acknowledgments
The research in Section 3 was supported by the Ministry of Science and Higher Education of the
Russian Federation (Goszadaniye) 075-00337-20-03, project No. 0714-2020-0005.
The work in
Sections 1, 2, 4, 5 was supported by the strategic academic leadership program ’Priority 2030’
(Agreement 075-02-2021-1316 30.09.2021).