Decentralized Local Stochastic Extra-Gradient for Variational Inequalities
Abstract
We consider distributed stochastic variational inequalities (VIs) on unbounded domains with the problem data that is heterogeneous (non-IID) and distributed across many devices.
We make a very general assumption on the computational network that, in particular, covers the settings of fully decentralized calculations with time-varying networks and centralized topologies commonly used in Federated Learning.
Moreover, multiple local updates on the workers can be made for reducing the communication frequency between the workers.
We extend the stochastic extragradient method to this very general setting and theoretically analyze its convergence rate in the strongly-monotone, monotone, and non-monotone (when a Minty solution exists) settings.
The provided rates explicitly exhibit the dependence on network characteristics (e.g., mixing time), iteration counter, data heterogeneity, variance, number of devices, and other standard parameters.
As a special case, our method and analysis apply to distributed stochastic saddle-point problems (SPP), e.g., to the training of Deep Generative Adversarial Networks (GANs) for which decentralized training has been reported to be extremely challenging.
In experiments for the decentralized training of GANs we demonstrate the effectiveness of our proposed approach.
Introduction
In large-scale machine learning (ML) scenarios the training data is often split between many devices, such as geographically distributed datacenters or mobile devices [38].
Decentralized training methods can learn an ML model with the same accuracy as if all the data would be aggregated on one single server [54, 5].
At the same time, training in a decentralized fashion has many advantages over traditional centralized approaches in such core aspects as data ownership, privacy, fault tolerance, and scalability.
A particular instance of the decentralized learning setting is Federated Learning (FL), where the training is orchestrated by a single device or server that communicates with all the participating client devices [62, 38].
In contrast, in fully decentralized learning (FD) scenarios the devices only communicate with their neighbors in the communication network graph with possibly arbitrary topology [54].
Thus, decentralized algorithms are important in scenarios where centralized communication is expensive, not desired, or impossible.
There have been tremendous advances recently in the development, design, and understanding of decentralized training schemes [71, 95, 86, 54, 84, 92, 90, 94, 23, 81, 50].
In particular, such aspects as data-heterogeneity [90, 78, 55], communication efﬁciency (through local updates [52, 44] or compression [89, 45]), and personalization [93, 8] have been studied.
However, all these advances were aimed at training with single-criterion loss functions leading to minimization problems, and they do not apply to more general problem classes.
For example, the training of Generative Adversarial Networks (GANs) [28] requires simultaneous competing optimization of the generator and the discriminator objectives, i.e., solving a non-convex-non-concave saddle-point problem (SPP).
This problem structure makes GANs notoriously difﬁcult to train even in the single-node setting [27, 15, 16], not talking about training over decentralized datasets [58, 69, 80].
Our goal in this paper is solving decentralized stochastic SPPs, and, more generally, decentralized stochastic Minty variational inequalities (MVIs) [64, 37].
In a decentralized stochastic MVI, the data is distributed over M  1 devices/nodes and each device m
[ M ] has access to its local stochastic oracle F m ( z, ⇠ m ) for the local operator F m ( z ) :
=
E ⇠ m ⇠ D m F m ( z, ⇠ m ) .
The data ⇠ m in the device m follows an unknown distribution D m that can be different for each device m
[ M ] .
The devices are connected via a communication network forming a graph such that two devices can exchange information if and only if the corresponding nodes are connected by an edge in this graph.
The goal is, while respecting the communication constraints, to ﬁnd cooperatively a point z
⇤ 2 R n such that, for all z 2 R n ,
[Equation]
A special instance of decentralized stochastic MVIs is the decentralized stochastic SPP with local objectives
[Equation]
The relation to VI can be seen by considering the variable z = ⇥
[Equation]
and the gradient ﬁeld F m ( [Equation]
⇤ .
In the special case when f ( x, y ) is convex-concave, the corresponding operator
F ( z ) =
1 M
P
Mm =1 E ⇠ m F m ( z, ⇠ m ) is monotone.
However, in the context of GANs training, where x and y are the parameters of the generator and the discriminator, respectively, the local losses f m ( x, y ) are possibly non-convex-non-concave in x, y and one can not assume the monotonicity of F in general, see also [21].
In this paper, we develop a novel algorithm for solving problems (1) and (2).
Note that the gradient descent-ascent scheme for the problem (2) may diverge even in the simple convex-concave setting with M = 1 device [15].
Thus, unlike [58], we use extragradient updates [48, 37, 27] as a building block and combine them with a gossip-type communication protocol [98, 12] on arbitrary, possibly time-varying, network topologies.
One of the main challenges due to the communication constraints is a “network error” induced by the impossibility of all the devices to reach the exact consensus, i.e., to have exactly the same information about the current iterate of the algorithm.
Thus, each device stores a local variable, and only approximate consensus among the devices can be achieved by gossip steps [47].
Unlike other decentralized algorithms [84, 58], our method avoids multiple gossip steps per iteration, which leads to better practical performance and the possibility to work on time-varying networks.
Moreover, our method allows for multiple local updates between communication rounds to reduce the communication overhead.
This also makes our approach suitable for communication- and privacy-restricted FL or fully decentralized settings [101].
Our contributions.
1) Based on extragradient updates, we develop a novel algorithm for distributed stochastic MVIs (and, as a special case, for distributed stochastic SPPs) with heterogeneous data.
Our scheme supports a very general communication protocol that covers centralized settings as in
Federated Learning, fully decentralized settings, local steps in both the centralized/decentralized settings, and time-varying network topologies.
In particular, we are not aware of earlier works proposing or analyzing extragradient methods with local steps for the fully decentralized setting or decentralized algorithms for stochastic MVIs over time-varying networks.
2) Under the very general communication protocol and in the three settings of MVIs, i.e., with an operator that is strongly-monotone, monotone, or non-monotone under the Minty condition, we prove the convergence of our algorithm and give an explicit dependence of the rates on the problem parameters: characteristics of the network (e.g., mixing time), data heterogeneity, the variance of the data, number of devices, and other standard parameters.
These theoretical results translate to the corresponding three settings of SPPs (strongly-convex-strongly-concave, convex-concave, non-convex-non-concave under Minty condition).
All our theoretical results are valid in the important heterogeneous data regime and allow judging in a quantiﬁable way how different properties, e.g., data heterogeneity, the scale of the noise in the data, and network characteristics, inﬂuence the convergence rate of the algorithm.
Even for decentralized settings, our results are novel for time-varying graphs and three different settings of monotonicity.
See also Table 1 that gives more details on our contribution compared to the existing literature.
The main challenge of our analysis is to deal with the very general assumption about the communication protocol and cope with the errors caused by the stochastic nature and heterogeneity of the data and limited information exchange between the nodes of the communication network.
As a byproduct of independent interest, we analyze the stochastic extragradient method with biased oracle on unbounded domains, which was not done so far in the literature.
3) We verify our theoretical results in numerical experiments and demonstrate the practical effectiveness of the proposed scheme.
In particular, we train the DCGAN [79] architecture on the
CIFAR-10 [51] dataset.
1.1 Related Work
The research on MVIs dates back at least to 1962 [64] with the classical book [41] and the recent works [59, 56, 13, 21].
VIs arise in a broad variety of applications: image denoising [25, 14], game theory and optimal control [26], robust optimization [9], and non-smooth oprimization via smooth reformulations [74, 73].
In ML, MVIs and SPPs arise in GANs training [19, 15, 16], reinforcement learning [76, 36], and adversarial training [60].
Extragradient.
The extragradient method (EGM) was ﬁrst proposed in [48], generalized as the mirror-prox method for deterministic problems in [73] and for stochastic problems with bounded variance in [37].
Yet, if the stochastic noise is not uniformly bounded, the EGM may diverge, see [15, 66].
[-Beznosikov et al. 
-]
Reference
[-Hou et al. 
-]
[Table] [Table]
Decentralized algorithms for MVIs and SPPs are the most closely related to our work.
In Table 1, we summarize their features and make a comparison with our algorithm, showing that, e [Table] g., existing methods do not support arbitrary time-varying network typologies.
The methods that use multiple rounds of gossip averaging (sparse communication) per iteration [58, 11, 6] can give near-optimal theoretical rates, but are often unstable in practice.
Thus, it is preferable to have only one sparse communication per iteration [59, 69, 91, 80, 97].
The second column of the table refers to standard algorithms that are extended to distributed settings in the corresponding work.
In particular, the algorithm of [59] requires expensive proximal updates.
The closest work to ours is [11], where a decentralized EGM without local steps is analyzed in the (strongly-)monotone setting.
Unlike our more general algorithm with local steps, theirs require multiple gossip updates in each iteration which is not desired in practice.
For the FL, i.e., centralized, setting, [11] studies the EGM with local steps in the strongly-monotone setting, and [20, 32] study the descent-ascent method with local steps.
Yet, all three works do not consider arbitrary time-varying graphs as in our work.
Algorithm
In this section, we present and discuss the proposed algorithm (Algorithm 1) that is based on two main ideas: (i) the extragradient step, as in the classical methods for VIs [48, 73], and (ii) the gossip averaging [12, 71] widely used in decentralized optimization methods and in the literature on diffusion strategies in distributed learning [82, 83, 99, 2, 61].
Unlike these papers that propose algorithms for optimization problems by exploiting gradient descent, our algorithm is based on the extragradient method and is designed to solve VIs and SPPs.
Moreover, unlike the mentioned works, our method also allows for local steps in-between the communication rounds and for time-varying networks and has non-asymptotic theoretical convergence rate guarantees.
Each step of Algorithm 1 can be divided into two phases.
The local phase (lines 4–6) consists of a step of the stochastic extragradient method at each node using only local information.
As in the non-distributed case, the nodes ﬁrst make an extrapolation step “to look into the future” and then an update based on the operator value at the “future” point.
This is followed by the communication phase (gossip step) (line 7), during which the nodes share and average local iterates with their neighbors N km in the communication network graph corresponding to the iteration k .
The averaging process involves the weights w km,i which are the elements of the matrix W k called the mixing matrix:
Deﬁnition 2.1 (Mixing matrix) .
We call a matrix W 2 [0; 1]
M ⇥ M a mixing matrix if it satisﬁes the following conditions: 1) W is symmetric, 2) W is doubly stochastic ( W 1 = 1 , 1 T W = 1 T , wheredenotes the vector of all ones), 3) W is aligned with the network: w ij 6 =
if and only if i = j or the edge ( i, j ) is in the communication network graph.
Reasonable choices of mixing matrices are, for example, (i) W
k
= I M
L k  max ( L k ) , where L k is the
Laplacian matrix of the network graph at the step k
and I M is the identity matrix, or (ii) using some local rules in the graph, based on the degrees of the neighboring nodes [98].
Note that our setting has a great ﬂexibility since in-between the iterations the topology of the communication graph is allowed to change, and the matrix W k , that encodes the current structure of the network, changes accordingly.
This is encoded in line 2, where the matrix W k is generated by some rule W k which can have different nature.
Examples include deterministic choice of a sequence of matrices W k , sampling from a time-varying probability distribution on matrices.
Even local steps without communication can be encoded with a diagonal matrix W k .
Algorithm parameters: stepsize  > 0 , {W k } k  0 – rules or distributions for mixing matrix in
iteration k . initialize:
z 0 2 Z , 8 m : z 0 m = z1: for k = 0 , 1 , 2 , . . .
do 2:
Sample matrix W k from W k 3: for each node m do 4:
Generate independently ⇠ km ⇠ D k , ⇠ k +1 / 3 m ⇠ D k 5: z k +1 / 3 m =
z km   F m ( z km , ⇠ km ) 6: z k +2 / 3 m = z km   F m ( z k +1 / 3 m , ⇠ k +1 / 3 m ) 7: z k +1 m = P i
[Equation]
8: end for 9: end for
To ensure that it is possible to approach the consensus between the nodes, we need the following assumption on the mixing properties of the matrix sequence W k .
Assumption 2.2 (Expected Consensus Rate) .
We assume that there exist a constant p 2 (0 , 1] and an integer ⌧  1 such that, after K iterations, for all matrices Z 2 R d ⇥ M and all integers l 2 { 0 , . . . , K/ ⌧ } ,
[Equation]
where W l, ⌧ = W l ⌧ · . . .
· W ( l +1) ⌧
1 , we use the matrix notation Z =
[ z 1 , . . .
, z M ] , ¯ Z =
[¯ z, . .
, ¯ z ] with ¯ z = 1 M
P
Mm =1 z m , and the expectation E W is taken over distributions of W t and indices t 2 { l ⌧ , ..., ( l + 1) ⌧  1 } .
This assumption ensures that, after ⌧ gossip steps with such time-varying matrices, we improve the consensus between the nodes, i.e., how close each z m is to ¯ z , by the factor of 11  p .
Importantly, in this case, some matrices W k can be, for example, the identity matrix (which corresponds to performing only local steps in iteration k ).
Assumption 2.2 has been recently quite popular in the literature on distributed optimization methods [72, 44, 49].
Moreover, it is very general and covers many special cases of decentralized and centralized algorithms.
For example, if we ﬁx W k = W for some ﬁxed connected graph, we get a decentralized algorithm on this graph.
If, at the same time, we set the matrix W = 1 M 11 T , then it is easy to see that we get an analog of the centralized setting with the averaging over all nodes performed in each communication step.
If we take W k = W for some ﬁxed connected graph at every ⌧ -th step and in other steps use W k
= I M , we have a decentralized (and, in particular, centralized) algorithm with local steps [87, 30, 44] and communications after each ⌧ iterations.
Generic Assumption 2. covers also many other settings of time-varying decentralized topologies, e.g., random topologies, cliques, B -connected graphs [35, 70].
Below we show that, under an appropriate choice of the stepsize, our extragradient method provably converges under such a general assumption that covers centralized and decentralized settings, local steps in both centralized and decentralized settings, and changing topologies of the communication graph.
Even for decentralized settings, this is novel for time-varying graphs and three different settings of monotonicity which we consider.
Setting and Assumptions
In this section, we introduce necessary assumptions that are used to analyze the proposed algorithm.
Assumption 3.1 (Lipschitzness) .
For all m , the operator F m ( z ) is Lipschitz with constant L , i.e.,
[Equation]
This is a standard assumption that is used in the analysis of all the methods displayed in Table 1.
Assumption 3.2.
We consider three scenarios for the operator F , namely, when F is stronglymonotone, monotone and non-monotone, but with an additional assumption:
(SM)
Strong monotonicity.
There exists µ > 0 such that, for all z 1 , z 2 ,
[Equation]
(M) Monotonicity.
For all z 1 , z 2 , it holds that:
[Equation]
(NM) Non-monotonicity (Minty).
There exists z ⇤ such that, for all z ,
[Equation]
Assumptions (SM), (M) and (L) are standard and classical assumptions in the literature on VIs.
Assumption (NM) is sometimes called the Minty or Variational Stability condition and it has been widely used recently by the community as a structured variant of non-monotonicity [18, 34, 63, 59, 39, 33, 21], particularly, since it is appropriate in GANs training [57, 58, 22, 6].
The next assumption is standard for the stochastic setting.
Assumption 3.3 (Bounded noise) .
F m ( z, ⇠ ) is unbiased and has bounded variance, i.e., for all z ,
[Equation]
Our last assumption reﬂects the variability of the local operators compared to their mean and is usually called D -heterogeneity.
This assumption is widely used in the analysis of local-steps (and not only) algorithms for minimization problems [40, 96, 30, 85, 4, 1, 31, 17, 24].
Moreover, [20, 32] use this assumption for the analysis of centralized local-steps methods for SPPs.
The authors of [58] assume D = 0 for the decentralized training of GANs.
Even in this case algorithms’ analysis can be challenging.
Assumption 3.4 ( D -heterogeneity) .
The values of the local operator have bounded variablility, i.e., for all z ,
[Equation]
Main Results
In this section, we present the convergence rate results for the proposed method under different settings of Assumption 3.2.
To present the main result, we introduce notation ¯ z k := 1 M
[Equation]
¯ z k +1 / 3 :
= 1 M
P
[Equation] for the averaged among the devices iterates and
[-P ki =
-][Equation]
for the averaged among the devices and iterates sequence, a.k.a.
ergodic average.
Finally, we denote  = ⌧ p
[Equation] +  [Equation]  which plays the role of the consensus error, i.e., the error caused by the impossibility of reaching the exact consensus between the nodes.
Note that the data heterogeneity appears in the convergence rates only through the quantity  .
Theorem 4.1 (Main theorem) .
Let Assumptions 2.2, 3.1, 3.3, 3.4 hold and the sequences ¯ z k ,
b
z k be generated by Algorithm 1 that is run for K > 0 iterations.
Then,
• Strongly-monotone case: under Assumption 3.2(SM), with  = ˜
O
⇣
[Equation]
o⌘ it holds that E ⇥
[Equation]
• Monotone case: under Assumption 3.2(M), for any convex compact C s.t.
z 0 , z ⇤ 2 C and {+max z,z 
+}
[Equation]
k z  z 0 k  ⌦ C , with  = O ✓ min ⇢ 1 L
[Equation]
it holds that
[Equation]
Under the additional assumption that, for all k , k ¯ z k k  ⌦ , with
=
[Equation]
we have that
[Equation]
• Non-monotone case: under Assumption 3.2(NM) and if k z 0 k  ⌦ ,
k z
⇤ k  ⌦ , with  =
[Equation]
✓ min
[Equation]
Under the additional assumption that, for all k , k ¯ z k k  ⌦ , with  =
O
[Equation]
⇢ [Equation] , ⇣ ⌦ 2 KL  ⌘
[Equation]
, we have that
[Equation]
The proof of the theorem is given in the supplementary material, where one can also ﬁnd explicit dependence of the rates on the stepsize  before it is chosen optimally.
We underline that the standard analysis [37] does not apply for the following reasons.
Firstly, unlike [37], in our problem (1), the feasible set is not bounded, which is especially important for the analysis in the monotone and non-monotone settings.
Secondly, our algorithm has an additional communication step (line 7) between the computational nodes, which leads to the impossibility for all the nodes to have the same information about the global operator F ( z ) and about the current iterate z .
This, in order, leads to a biased oracle that, unlike existing works, has to be analyzed in the setting of an unbounded feasible set, which is quite challenging.
To analyze our variant of the extragradient method, we successfully handle this challenge.
Our key steps are to bound the bias (see, e.g., the last two terms in the r.h.s.
of Lemma C.8 that are caused by the network errors), prove the boundedness in expectation of the sequence of the iterates for monotone (see Section C.3.1 of the supplementary material) and non-monotone (see Section C.4.1 of the supplementary material) cases, which may be of independent interest and which we have not seen in the literature, even in the non-distributed setting with biased stochastic oracles.
Proving the boundedness is challenging due to the noise caused by the stochasticity and heterogeneity of the data and network effects due to the imperfect exchange of information.
Surprisingly, in the end, we still manage to analyze our algorithm under the very general [-Assumption2.
-]Assumption 2.2 and we are not aware of any results with similar generality of the settings: different network topologies (including time-varying), distributed architectures, different monotonicity assumptions.
The provided convergence rates have an explicit dependence on the problem parameters: the network that is characterized by the mixing time ⌧ and the mixing factor p , the data heterogeneity D (these three quantities appear in the convergence rates only through the quantity  ), the variance  2 of the noise in the data, the Lipschitz constant L , the strong monotonicity parameter µ , the number of nodes/devices M .
Thus, our rates allow judging how different properties, e.g., data heterogeneity, noise level, and network characteristics inﬂuence the convergence rates.
This, in particular, opens up an opportunity for a meta-optimization process if we can design the network and change M , ⌧ , p to achieve faster convergence.
We now discuss the convergence results obtained in the theorem, and also compare them with already existing algorithms (see Table 1) and their guarantees.
Firstly, all the estimates have a similar several-term structure.
The ﬁrst term corresponds to the deterministic setting and is similar to existing methods for smooth VIs in the non-distributed setting.
Only in the strongly-convex case, there is an additional factor ⌧ /p that increases the condition number L/µ of the problem.
The second (stochastic) term is also standard for the non-distributed setting and corresponds to the stochastic nature of the problem.
Note that, for a very general distributed setting, we have managed to obtain the corresponding terms similar to the non-distributed setting.
Moreover, we can see the beneﬁt of exploiting distributed computations: the leading stochastic term depends on  2 /M that decreases as the number M of the nodes increases.
The other terms correspond to the consensus error  and are due to the imperfect communications between the nodes, i.e., that all the nodes can’t have exactly the same information about the current iterate.
Importantly, in all the cases, this error does not make the overall convergence worse since the dependence on K is no worse in these terms than the dependence on K in the stochastic term.
In the experimental section, we illustrate that the network error is not an artifact of the analysis but is indeed present in practice.
Theorem 4.1 is formulated for a ﬁxed budget of iterations K and the corresponding stepsizes  that depend on K , which is pretty standard in the literature [37, 88, 10], where many algorithms ﬁx the stepsize depending on the budget of the iterations.
In Section D of the supplementary material, we present a simple restarting procedure that allows to extend the results of Theorem 4.1 to any-time convergence without a-priori ﬁxing K .
The idea is to set K t = 2 t for t = 0 , 1 , . . .
and restart the algorithm after each K t iterations.
We next make reﬁned comments for each particular setting of monotonicity.
• Strongly-monotone case: In the centralized setting with local updates, our rate is slightly better than in [11].
Unlike our algorithm, centralized algorithms with local steps for SPPs in [20, 32] are based on the gradient descent-ascent method that may diverge in the stochastic setting even for bilinear problems.
Moreover, their analysis implies a very small stepsize  ⇠ µpL 2 ⌧ (cf.
ours  ⇠ pL ⌧ ), which greatly slows down the convergence of the algorithm.
For the decentralized setting, [11] propose an optimal algorithm with the rate matching the lower bound which they also give.
Our rate is worse probably because of the generality of the Assumption 2.2. On the other hand, our algorithm is more practical since it avoids using multiple gossip steps at each iteration.
Also, our algorithm is more general, allowing us to work with time-varying topologies and local steps even in the decentralized setting. • Monotone case: The quantity sup z 2 C E
⇥⌦
F ( z ) , b z K  z ↵⇤ in the convergence rates estimates reﬂects the stochastic nature of the problem and is a counterpart of the standard restricted gap (or merit) function [75]: Gap C ( u ) :=
sup z 2 C [ h F ( z ) , u  z i ] .
When F is a monotone operator, if Gap C (ˆ u ) = 0 and C contains a neighborhood of ˆ u , then [75, 3] ˆ u is a solution to (1) and even more: it is a strong solution to the corresponding variational inequality, i.e., for all z , h F (ˆ u ) , ˆ u  z i  0 .
Thus, Gap C ( u ) is an appropriate measure of suboptimality in this setting and (7) guarantees that after a sufﬁcient number of iterations, we obtain an approximate solution in expectation.
Importantly, for (7), neither z nor ¯ z k are assumed to be bounded.
As in the previous works on non-distributed algorithms for MVIs [75, 3], we use Gap C ( u ) with an arbitrary compact set C that contains z 0 and z ⇤ (this can be a large set).
Further, (8) is a reﬁned version of the general result (7) under the additional assumption of the boundedness of the averaged iterates.
If the boundedness does not hold, we still have (7).
Moreover, (7) and (8) hold for the same method, and to run the algorithm, there is no need to know in advance whether the generated sequence is bounded or not.
Only [11, 80] consider MVIs with monotone operator in distributed setting.
Our algorithm is more general than theirs: our algorithm supports time-varying networks and local steps between communications.
The algorithm in [11] uses multiple gossip steps between the updates of the iterates.
On the one hand, this allows decreasing the consensus error  .
On the other hand, this leads to an additional factor in the number of communications compared to our estimates: the ﬁrst term in their bound is p  times larger than ours, where  > 1 is some condition number of the mixing matrix.
Moreover , multiple gossip steps may be impractical if the communication is performed through unstable channels or is expensive for some reason.
The paper [80] considers only deterministic setting.
• Non-monotone case: The same as in the previous case remark on the boundedness of ¯ z k , z ⇤ assumed to obtain (10) applies in this case.
Further, in this setting, the convergence is guaranteed up to some accuracy that is governed by the stochastic nature of the problem (the  2 -term) and by the distributed nature of the problem (the  -terms).
With this respect, the results are similar to the non-distributed stochastic extragradient method [7] and the distributed method [58] analyzed in the homogeneous case D = 0 .
To the best of our knowledge, convergence up to arbitrarily small accuracy can be guaranteed only for deterministic distributed methods [59], i.e., in a much simpler setting than ours.
Moreover, the methods of [59] are not the most robust since they require evaluating the proximal operator of a function and it is assumed that this can be done in a closed form, which is computationally expensive and may not hold in practice.
Note that, based on our result, it is possible to achieve convergence up to arbitrarily small accuracy if one considers the homogeneous case with D = 0 .
Indeed, choosing the right batch size, for example, proportionally to K ↵ with ↵ > 0 , one can replace  2 by  2 K ↵ in (9) and (10) and get convergence guarantees.
Experiments
In this section, we present two sets of experiments to validate the performance of Algorithm 1.
In
Section 5.1, we verify the obtained convergence guarantees on two examples: a strongly-monotone and a monotone bilinear problems, and in Section 5.2, we explore the non-monotone case with an application to GANs training.
Extended details of the experimental setup can be found in the supplementary material.
5.1 Verifying Theoretical Convergence Rate
First, we focus on the verifying whether the actual behaviour of Algorithm 1 is predicted by the theoretical convergence rate in Theorem 4.1.
Setup.
We consider a distributed bilinear SPP (2) with the objective functions f m ( x, y )
=
a 2 k x k 2 + bx > y  a 2 k y k 2 + c > m x, where x, y, c
m 2 R n , a, b 2 R and m 2 { 1 , . .
, M } .
This set of functions satisfy Assumptions 3.1, 3.2, 3.4 with constants µ = a, L 2 = a 2 + b 2 ,
D = max m k c m  ¯ c k .
In this section, we use a ring topology on M = 20 nodes with uniform averaging weights, and we set the dimension n = 5 , b = 1 , D ⇡ 3 , and keep ⌧ = 1 .
The value of the parameter p in this setting is approximately 0 .
288 [46, Table 1].
To satisfy Assumption 3.3, we generate stochastic gradients by adding to the exact gradients unbiased Gaussian noise with variance  2 .
Convergence Behaviour.
In Figure 1, we show the convergence of Algorithm 1 with a ﬁxed stepsize on the strongly-monotone ( a = 1 ) and monotone ( a = 0 ) instances.
In the strongly-monotone case, we see a linear convergence up to some level deﬁned by the heterogeneity parameter and the noise.
The convergence for the non-strongly-monotone problem is slower, but we also see a linear convergence up to some level (for bilinear problems this behavior is expected from the theoretical point of view [48]).
Note that the convergence to some limiting accuracy is expected since when a constant stepsize is used in stochastic optimization/stochastic variational inequalities with strong convexity/monotonicity, algorithms are usually guaranteed to converge only to a vicinity of the solution, see, e.g., Theorem 2 in [66].
This is also in accordance with Theorem 4.1 that, for a ﬁxed stepsize, guarantees the convergence to some non-zero limit accuracy and says that, to achieve the zero error, one needs to choose a decreasing stepsize.
We additionally validate in Section A.2 of the supplementary material that with a decreasing stepsize, the algorithm can converge to the zero error.
Dependence on the Heterogeneity parameter D .
In the second set of experiments, we aim to verify the dependence on the data heterogeneity parameter D .
Therefore, we consider the setting when  2 = 0 .
From our theory, equation (6), we predict that the most signiﬁcant term in the convergence rate when  2 = 0 scales as O [Equation]
(since the primary goal of this experiment is to study the dependence on p , D , K , we omit all the other ﬁxed parameters for simplicity).
We take b = 1 , a = 1 and conduct experiments with the number of iterations needed to achieve the error 1 M
P
Mm =
k z km  z ⇤ k 2 < " , for different " .
In all these experiments, the stepsize is tuned individually.
First, we verify the power of K in the bounds.
For this experiment, we keep D, p constant and vary the accuracy " .
As we can see from the leftmost subplot in Figure 2, the number of iterations scales as K / 1 p " , conﬁrming the predicted O  1 K dependency of the error on K .
Next, we measure the number of iterations sufﬁcient to reach the error " = 0 .
01 while varying D .
The middle plot shows that the number of iterations scales proportionally to D (showing D / K ).
Lastly, we depict the number of iterations to reach " = 0 .
01 while changing the graph parameter p and again observep / K .
Summarizing, these experiments verify the O
D 2 p 2 K term in the convergence rate.
5.2 Training GANs
Our algorithm allows combining in the distributed learning setting different communication graph topologies, as well as local steps.
Thus, our goal in this section is to illustrate this empirically with the experiments on GANs training. In Section A.1 of the supplementary material, we discuss to what extent our theoretical results hold for GANs training.
Data and model.
We consider the CIFAR-10 [51] dataset containing 60000 images, equally distributed over 10 classes. We increased the size of the dataset by 4 times using transformations and adding noise.
We simulate a distributed setup of 16 nodes on two GPUs and use Ray [67].
To emulate the heterogeneous setting, we partition the dataset into 16 subsets.
For each subset, we select a major class that forms 20% of the data, while the rest of the data split is ﬁlled uniformly by the other classes.
As a basic architecture we choose DCGAN [79], conditioned by class labels, similarly to [65] (the network architecture can be found in Section A.1).
We chose Adam [42] as the optimizer.
We make one local Adam step and then one gossip averaging step with time-varying matrices W k —similar to Algorithm 1.
Setting.
We compare the following three topologies (and the corresponding matrices W k ):
• Full.
Full graph at the end of each epoch, otherwise local steps.
This means that we makecommunication rounds (by communication round
we mean the exchange of information between a pair of devices) in an epoch. • Local.
Full graph at the end of each 5th epoch, otherwise local steps.
This means that we makecommunication rounds in an epoch (in average: 4 epochs without communications and 1 epoch with 120 rounds). • Clusters.
At the end of each epoch, clique clusters of size 4 are randomly formed (in totalcliques).
This means that we make 24 communication rounds in an epoch.
Note that the communication budget of the ﬁrst approach is 5 times larger.
We use the same learning rate equal to 0.002 for the generator and discriminator.
The rest of the parameters and features of the architecture can be found in the supplementary material.
Results.
The results of the experiment are presented in Figure 3 and Figure 6 (Section A.3).
In terms of the number of local epochs, all the methods converged quite close to each other and produced similar images.
In terms of communications, Local and Cluster topologies lead to much better results, and the Cluster topology is slightly better than the Local.  Conclusion
We propose a novel efﬁcient algorithm for solving decentralized stochastic MVIs and SPPs under a very general assumption on the network topology and communication constraints.
In particular, our method is the ﬁrst decentralized extragradient method with local steps for time-varying network topologies.
Moreover, for the proposed algorithm, we prove the convergence rate theorem in the SM, M and NM cases.
In the numerical experiments, we verify that the dependence of our rates on the data heterogeneity parameter D is tight in the SM case, and cannot be further improved in general.
By training
DCGAN on a decentralized topology, we demonstrate that our method is effective on practical DL tasks. As a future work it would be interesting to generalize such algorithms for inﬁnite-dimensional problems. Acknowledgments This research of A. Beznosikov has been supported by The Analytical Center for the Government of the Russian Federation (Agreement No.
70-2021-00143 dd.
01.11.2021, IGK 000000D730321P5Q0002).
The work by P. Dvurechensky in Section C.3 was funded by the Deutsche
Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy – The Berlin Mathematics Research Center MATH+ (EXC-2046/1, project ID: 390685689).
CIFAR and DCGAN
(b) Did you mention the license of the assets?
[N/A] , use open assets (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]