Original: Distributed Methods with Compressed Communication for Solving Variational Inequalities, with Theoretical Guarantees
Revised: Distributed Methods with Compressed Communication for Solving Variational Inequalities, with Theoretical Guarantees

Original: Abstract
Revised: Abstract

Original: Variational inequalities in general and saddle point problems in particular are increasingly relevant in machine learning applications, including adversarial learning, GANs, transport and robust optimization.
Revised: Variational inequalities in general and saddle point problems in particular are increasingly relevant in machine learning applications, including adversarial learning, GANs, transport and robust optimization.

Original: With increasing data and problem sizes necessary to train high performing models across various applications, we need to rely on parallel and distributed computing.
Revised: With increasing data and problem sizes necessary to train high performing models across various applications, we need to rely on parallel and distributed computing.

Original: However, in distributed training, communication among the compute nodes is a key bottleneck during training, and this problem is exacerbated for high dimensional and over-parameterized models.
Revised: However, in distributed training, communication among the compute nodes is a key bottleneck during training, and this problem is exacerbated for high dimensional and over-parameterized models.

Original: Due to these considerations, it is important to equip existing methods with strategies that would allow to reduce the volume of transmitted information during training while obtaining a model of comparable quality.
Revised: Due to these considerations, it is important to equip existing methods with strategies that would allow to reduce the volume of transmitted information during training while obtaining a model of comparable quality.

Original: In this paper, we present the ﬁrst theoretically grounded distributed methods for solving variational inequalities and saddle point problems using compressed communication:
Revised: In this paper, we present the ﬁrst theoretically grounded distributed methods for solving variational inequalities and saddle point problems using compressed communication:

Original: MASHA1 and MASHA2 .
Revised: MASHA1 and MASHA2 .

Original: Our theory and methods allow for the use of both unbiased (such as Rand k ; MASHA1 ) and contractive (such as Top k ; MASHA2 ) compressors.
Revised: Our theory and methods allow for the use of both unbiased (such as Rand k ; MASHA1 ) and contractive (such as Top k ; MASHA2 ) compressors.

Original: New algorithms support bidirectional compressions, and also can be modiﬁed for stochastic setting with batches and for federated learning with partial participation of clients.
Revised: New algorithms support bidirectional compressions, and also can be modiﬁed for stochastic setting with batches and for federated learning with partial participation of clients.

Original: We empirically validated our conclusions using two experimental setups: a standard bilinear min-max problem, and large-scale distributed adversarial training of transformers.
Revised: We empirically validated our conclusions using two experimental setups: a standard bilinear min-max problem, and large-scale distributed adversarial training of transformers.

Original:  Introduction 1.1 The expressive power of variational inequalities
Revised:  Introduction 1.1 The expressive power of variational inequalities

Original: Due to their abstract mathematical nature and the associated ﬂexibility they offer in modeling various practical problems of interests, variational inequalities (VI) have been an active area of research in applied mathematics for more than half a century [65, 31, 22].
Revised: Due to their abstract mathematical nature and the associated ﬂexibility they offer in modeling various practical problems of interests, variational inequalities (VI) have been an active area of research in applied mathematics for more than half a century [65, 31, 22].

Original: It is well known that VIs can be used to formulate and study optimization problems, saddle point problems (SPPs) , games and ﬁxed point problems, for example, in an elegant unifying mathematical framework [9].
Revised: It is well known that VIs can be used to formulate and study optimization problems, saddle point problems (SPPs) , games and ﬁxed point problems, for example, in an elegant unifying mathematical framework [9].

Original: Recently, a series of works by various authors [15, 26, 58, 13, 49] built a bridge between VIs/SPPs and GANs [28].
Revised: Recently, a series of works by various authors [15, 26, 58, 13, 49] built a bridge between VIs/SPPs and GANs [28].

Original: This allows to successfully transfer established insights and well-known techniques from the vast literature on VIs/SPPs, such as averaging and extrapolation, to the study of GANs.
Revised: This allows to successfully transfer established insights and well-known techniques from the vast literature on VIs/SPPs, such as averaging and extrapolation, to the study of GANs.

Original: Besides their usefulness in studying GANs and alternative adversarial learning models [57], VIs/SPPs have recently attracted considerable attention of the machine learning community due to their ability to model other situations where the minimization of a single loss function does not sufﬁce, such as auction theory [80], supervised learning with non-separable loss [39] or non-separable regularizer [7] and reinforcement learning [69, 66, 38].
Revised: Besides their usefulness in studying GANs and alternative adversarial learning models [57], VIs/SPPs have recently attracted considerable attention of the machine learning community due to their ability to model other situations where the minimization of a single loss function does not sufﬁce, such as auction theory [80], supervised learning with non-separable loss [39] or non-separable regularizer [7] and reinforcement learning [69, 66, 38].

Original: In summary, VIs have recently become a potent tool enabling new advances in practical machine learning situations reaching beyond supervised learning where optimization problems and techniques, which can be seen as special instances of VIs and methods for solving them, reign supreme.
Revised: In summary, VIs have recently become a potent tool enabling new advances in practical machine learning situations reaching beyond supervised learning where optimization problems and techniques, which can be seen as special instances of VIs and methods for solving them, reign supreme.

Original: 1.2 Training of supervised models via distributed optimization
Revised: 1.2 Training of supervised models via distributed optimization

Original: On the other hand, for classical and much better understood supervised machine learning /minimization problems, researchers and practitioners face other challenges, which, until recently, have been outside of VI’s research.
Revised: On the other hand, for classical and much better understood supervised machine learning /minimization problems, researchers and practitioners face other challenges, which, until recently, have been outside of VI’s research.

Original: Indeed, the training of modern supervised machine learning models in general, and deep neural networks in particular, is still extremely challenging.
Revised: Indeed, the training of modern supervised machine learning models in general, and deep neural networks in particular, is still extremely challenging.

Original: Due to their desire to improve the generalization of deployed models, machine learning engineers need to rely on training datasets of ever increasing sizes and on elaborate over-parametrized models [5].
Revised: Due to their desire to improve the generalization of deployed models, machine learning engineers need to rely on training datasets of ever increasing sizes and on elaborate over-parametrized models [5].

Original: Supporting workloads of such unprecedented magnitudes would be impossible without combining the latest advances in hardware acceleration, distributed systems and distributed algorithm design [83].
Revised: Supporting workloads of such unprecedented magnitudes would be impossible without combining the latest advances in hardware acceleration, distributed systems and distributed algorithm design [83].

Original: When training such modern supervised models in a distributed fashion, communication cost is often the bottleneck of the training system, and for this reason, a lot of effort was recently targeted at the design of communication efﬁcient distributed optimization methods [45, 76, 25, 29].
Revised: When training such modern supervised models in a distributed fashion, communication cost is often the bottleneck of the training system, and for this reason, a lot of effort was recently targeted at the design of communication efﬁcient distributed optimization methods [45, 76, 25, 29].

Original: A particularly successful technique for improving the communication efﬁciency of distributed ﬁrst order optimization methods is communication compression.
Revised: A particularly successful technique for improving the communication efﬁciency of distributed ﬁrst order optimization methods is communication compression.

Original: The idea behind this technique is rooted in the observation that in practical implementations it is often advantageous to communicate messages compressed via (often randomized) lossy compression techniques instead of communicating the full messages [75, 2].
Revised: The idea behind this technique is rooted in the observation that in practical implementations it is often advantageous to communicate messages compressed via (often randomized) lossy compression techniques instead of communicating the full messages [75, 2].

Original: If the number of parallel workers is large enough, the noise introduced by compression is reduced, and training with compressed communication will often lead to comparable test error while reducing the amount of communicated bits, which results in faster training, both in theory and practice [59, 29].
Revised: If the number of parallel workers is large enough, the noise introduced by compression is reduced, and training with compressed communication will often lead to comparable test error while reducing the amount of communicated bits, which results in faster training, both in theory and practice [59, 29].

Original: 1.3 Two classes of compression operators
Revised: 1.3 Two classes of compression operators

Original: The paper focuses on compression methods for distributed VIs and SPPs.
Revised: The paper focuses on compression methods for distributed VIs and SPPs.

Original: Let us give the main deﬁnitions.
Revised: Let us give the main deﬁnitions.

Original: We say that a (possibly) stochastic mapping Q : R d !
Revised: We say that a (possibly) stochastic mapping Q : R d !

Original: R d is an unbiased compression operator if there exists a constant q  1 such that
Revised: R d is an unbiased compression operator if there exists a constant q  1 such that

Original: [Equation]
Revised: [Equation]

Original: Further, we say that a stochastic mapping C : R d !
Revised: Further, we say that a stochastic mapping C : R d !

Original: R d is a contractive compression operator if there exists a constant   1 such that
Revised: R d is a contractive compression operator if there exists a constant   1 such that

Original: [Equation]
Revised: [Equation]

Original: If b is the number of bits needed to represent a single ﬂoat (e.g., b = 32 or b = 64 ), then the number of bits needed to represent a generic vector z 2 R d is k z k bits := bd .
Revised: If b is the number of bits needed to represent a single ﬂoat (e.g., b = 32 or b = 64 ), then the number of bits needed to represent a generic vector z 2 R d is k z k bits := bd .

Original: To describe how much a compression operator reduces its input vector on average, we deﬁne the notion of expected density, denoted via   1 := 1 bd E k Q
Revised: To describe how much a compression operator reduces its input vector on average, we deﬁne the notion of expected density, denoted via   1 := 1 bd E k Q

Original: ( z ) k bits , where k Q ( z ) k bits is the number of bits needed to represent the quantized vector Q ( z ) .
Revised: ( z ) k bits , where k Q ( z ) k bits is the number of bits needed to represent the quantized vector Q ( z ) .

Original: Note that   1 .
Revised: Note that   1 .

Original: For the Rand k operator [3, 10] we have q =  =
Revised: For the Rand k operator [3, 10] we have q =  =

Original: d/k .
Revised: d/k .

Original: 1.4 Towards communication-efﬁcient distributed methods for VIs and SPPs
Revised: 1.4 Towards communication-efﬁcient distributed methods for VIs and SPPs

Original: Classical VI/SPP algorithms such as the Extra Gradient method originally proposed by [46] and later studied by many authors [62, 41], including in a distributed environment [77, 52, 61, 73].
Revised: Classical VI/SPP algorithms such as the Extra Gradient method originally proposed by [46] and later studied by many authors [62, 41], including in a distributed environment [77, 52, 61, 73].

Original: Among them, a number of works stand out trying to solve the communication bottleneck challenge using various approaches such as local steps, data-similarity etc.[88, 34, 16, 11, 12].
Revised: Among them, a number of works stand out trying to solve the communication bottleneck challenge using various approaches such as local steps, data-similarity etc.[88, 34, 16, 11, 12].

Original: But despite the fact that the use of compression is one of the most popular communication-efﬁcient approaches for distributed minimization problems, no work has yet paid attention to the compression technique neither for distributed SPPs nor for VIs , with the exception of the work [88], which relies on rounding to the nearest integer multiple of a certain quantity.
Revised: But despite the fact that the use of compression is one of the most popular communication-efﬁcient approaches for distributed minimization problems, no work has yet paid attention to the compression technique neither for distributed SPPs nor for VIs , with the exception of the work [88], which relies on rounding to the nearest integer multiple of a certain quantity.

Original: This compression mechanism does not offer theoretical beneﬁts and does not even lead to convergence to the solution since the errors introduced through rounding persist and prevent the method from solving the problem.
Revised: This compression mechanism does not offer theoretical beneﬁts and does not even lead to convergence to the solution since the errors introduced through rounding persist and prevent the method from solving the problem.

Original:  Summary of Contributions
Revised:  Summary of Contributions

Original: In this paper, we investigate whether it is possible to design communication-efﬁcient algorithms for solving distributed VI/SPP by borrowing generic communication compression techniques (1) and (2) from the optimization literature [75, 2, 59, 29, 72] and embedding them into established, efﬁcient methods for solving VIs/SPPs [46, 62, 41, 1].
Revised: In this paper, we investigate whether it is possible to design communication-efﬁcient algorithms for solving distributed VI/SPP by borrowing generic communication compression techniques (1) and (2) from the optimization literature [75, 2, 59, 29, 72] and embedding them into established, efﬁcient methods for solving VIs/SPPs [46, 62, 41, 1].

Original: Whether or not this is possible is an open problem.
Revised: Whether or not this is possible is an open problem.

Original: In summary, we design the ﬁrst algorithms with compression for solving general distributed VI/SPP (see
Revised: In summary, we design the ﬁrst algorithms with compression for solving general distributed VI/SPP (see

Original: Section 3, Equation 3) in the deterministic (see (4) ), stochastic (see (44) ) and federated (see (54) ) regimes, supporting both unbiased ( MASHA1 = Algorithms 1, 5, 7) and contractive ( MASHA2 = Algorithms 2, 6, 8) compressors.
Revised: Section 3, Equation 3) in the deterministic (see (4) ), stochastic (see (44) ) and federated (see (54) ) regimes, supporting both unbiased ( MASHA1 = Algorithms 1, 5, 7) and contractive ( MASHA2 = Algorithms 2, 6, 8) compressors.

Original: Convergence of all our methods are analyzed in strongly-monotone (strongly convex - strongly concave), monotone (convex - concave) non-monotone/minty (non-convex-non-concave) cases.
Revised: Convergence of all our methods are analyzed in strongly-monotone (strongly convex - strongly concave), monotone (convex - concave) non-monotone/minty (non-convex-non-concave) cases.

Original: 2.1 Two types of compressors
Revised: 2.1 Two types of compressors

Original: We develop two approaches for distributed VIs/SPPs depending on whether we use unbiased (1) or contractive (2) compressors, since each type of compressor demands a different algorithmic design and a different analysis.
Revised: We develop two approaches for distributed VIs/SPPs depending on whether we use unbiased (1) or contractive (2) compressors, since each type of compressor demands a different algorithmic design and a different analysis.

Original: In particular, contractive compressors are notoriously hard to analyze even for optimization problems [44, 72].
Revised: In particular, contractive compressors are notoriously hard to analyze even for optimization problems [44, 72].

Original: Our method based on unbiased compressors is called MASHA (Algorithm 1), and our method based on contraction compressors is called MASHA2 (Algorithm 2).
Revised: Our method based on unbiased compressors is called MASHA (Algorithm 1), and our method based on contraction compressors is called MASHA2 (Algorithm 2).

Original: 2.2 Theoretical complexity results
Revised: 2.2 Theoretical complexity results

Original: We establish a number of theoretical complexity results for our methods, which we summarize in Table 1 (Appendix A).
Revised: We establish a number of theoretical complexity results for our methods, which we summarize in Table 1 (Appendix A).

Original: We consider the strongly monotone (strongly convex - strongly concave), monotone (convex - concave) regimes as well as the more general non-monotone/minty (non-convex-nonconcave) regime.
Revised: We consider the strongly monotone (strongly convex - strongly concave), monotone (convex - concave) regimes as well as the more general non-monotone/minty (non-convex-nonconcave) regime.

Original: In the strongly monotone case we obtain linear convergence results ( O (log 1 / ✏ ) ) in terms of the distance to solution, in the monotone we obtain fast sublinear convergence results ( O (1 / ✏ ) ) in terms of the gap function, and in the non-monotone case we have sublinear convergence results ( O (1 / ✏ 2 ) ) in terms of the Euclidean norm of the operator.
Revised: In the strongly monotone case we obtain linear convergence results ( O (log 1 / ✏ ) ) in terms of the distance to solution, in the monotone we obtain fast sublinear convergence results ( O (1 / ✏ ) ) in terms of the gap function, and in the non-monotone case we have sublinear convergence results ( O (1 / ✏ 2 ) ) in terms of the Euclidean norm of the operator.

Original: To get an estimate for the number of information transmitted, one need to multiply the estimates from Table 1 by 1 /  .
Revised: To get an estimate for the number of information transmitted, one need to multiply the estimates from Table 1 by 1 /  .

Original: Then we get that from the point of view of the transmitted information (and also time for communications),
Revised: Then we get that from the point of view of the transmitted information (and also time for communications),

Original: MASHA1 is better by a factor p 1 /  + 1 /M
Revised: MASHA1 is better by a factor p 1 /  + 1 /M

Original: ( M – number of workers) in comparison with the classical Extra Gradient.
Revised: ( M – number of workers) in comparison with the classical Extra Gradient.

Original: It means that we get an acceleration of min { p  ; p M } times.
Revised: It means that we get an acceleration of min { p  ; p M } times.

Original: For example,
Revised: For example,

Original: ADIANA from [48](the theoretical SOTA method with unbiased compressions for strongly convex minimization) has the same accelaration.
Revised: ADIANA from [48](the theoretical SOTA method with unbiased compressions for strongly convex minimization) has the same accelaration.

Original: The same situation is with MASHA2 .
Revised: The same situation is with MASHA2 .

Original: The method has the same compression dependent multiplier as ECLK from [71] (the theoretical SOTA with contractive compression for minimization).
Revised: The method has the same compression dependent multiplier as ECLK from [71] (the theoretical SOTA with contractive compression for minimization).

Original: Based on these facts, we hypothesize that MASHA1 and MASHA have unimprovable estimates (see Appendix B).
Revised: Based on these facts, we hypothesize that MASHA1 and MASHA have unimprovable estimates (see Appendix B).

Original: 2.3 Stochastic case and variance reduction
Revised: 2.3 Stochastic case and variance reduction

Original: MASHA1 and MASHA2 are designed to handle the deterministic setting.
Revised: MASHA1 and MASHA2 are designed to handle the deterministic setting.

Original: But often, in practice, the computation of the full operators/gradients is expensive, then we need to deal with stochastic realizations.
Revised: But often, in practice, the computation of the full operators/gradients is expensive, then we need to deal with stochastic realizations.

Original: In particular, a popular case is when each operator/gradient has a ﬁnite-sum structure on its own, e.g. , ﬁnite-sum of batches.
Revised: In particular, a popular case is when each operator/gradient has a ﬁnite-sum structure on its own, e.g. , ﬁnite-sum of batches.

Original: For this issue, we consider two modiﬁcations: VR-MASHA (Algorithm 5) and VR-MASHA2 (Algorithm 6).
Revised: For this issue, we consider two modiﬁcations: VR-MASHA (Algorithm 5) and VR-MASHA2 (Algorithm 6).

Original: Both are enhanced with bespoke variance-reduction techniques for better theoretical and practical performance.
Revised: Both are enhanced with bespoke variance-reduction techniques for better theoretical and practical performance.

Original: These results can be interesting in the non-distributed case.
Revised: These results can be interesting in the non-distributed case.

Original: As far as we know, we are the ﬁrst who consider variance reduction for nonmonotone VIs.
Revised: As far as we know, we are the ﬁrst who consider variance reduction for nonmonotone VIs.

Original: We found only one paper on non-convex-concave saddle point problems [87] under the PL condition.
Revised: We found only one paper on non-convex-concave saddle point problems [87] under the PL condition.

Original: See Appendix F for details.
Revised: See Appendix F for details.

Original: 2.4 Federated learning and partial participation
Revised: 2.4 Federated learning and partial participation

Original: Federated learning [45, 42] is an important and popular branch of distributed methods.
Revised: Federated learning [45, 42] is an important and popular branch of distributed methods.

Original: Therefore, a good bonus for the algorithm is that it can be easily adapted for it.
Revised: Therefore, a good bonus for the algorithm is that it can be easily adapted for it.

Original: In a federated setup where the computing devices are mobile phones, tablets, personal computers etc, the importance of the communication bottleneck is even higher.
Revised: In a federated setup where the computing devices are mobile phones, tablets, personal computers etc, the importance of the communication bottleneck is even higher.

Original: In such circumstances, devices can have weak and slow connections, or they can even disconnect for a while.
Revised: In such circumstances, devices can have weak and slow connections, or they can even disconnect for a while.

Original: At such moments, it is not necessary to interrupt the learning process, and only available devices can be used.
Revised: At such moments, it is not necessary to interrupt the learning process, and only available devices can be used.

Original: Therefore, we introduce two modiﬁcations: PP-MASHA1 (Algorithm 7) and PP-MASHA2 (Algorithm 8), that support the mode of partial participation of devices in the learning process.
Revised: Therefore, we introduce two modiﬁcations: PP-MASHA1 (Algorithm 7) and PP-MASHA2 (Algorithm 8), that support the mode of partial participation of devices in the learning process.

Original: For minimization problems, a combination of quantization and partial participation occurs in [33, 68, 29].
Revised: For minimization problems, a combination of quantization and partial participation occurs in [33, 68, 29].

Original: The results are contained in Appendix
Revised: The results are contained in Appendix

Original: G.
Revised: G.

Original: 2.5 Bidirectional compression
Revised: 2.5 Bidirectional compression

Original: Most methods, especially with contractive compressors, only use compression when transferring information from devices to the server.
Revised: Most methods, especially with contractive compressors, only use compression when transferring information from devices to the server.

Original: Meanwhile, quite often in practical situations, the transfer of information from the server to the device is also expensive [32, 81, 68].
Revised: Meanwhile, quite often in practical situations, the transfer of information from the server to the device is also expensive [32, 81, 68].

Original: In such situations it also makes sense to compress the information when sending it from the server to the agents.
Revised: In such situations it also makes sense to compress the information when sending it from the server to the agents.

Original: We can highlight some works on bidirectional unbiased [68] and contractive compressors [90, 81, 55, 23] for distributed minimization problems.
Revised: We can highlight some works on bidirectional unbiased [68] and contractive compressors [90, 81, 55, 23] for distributed minimization problems.

Original: But most of these methods have their small shortcomings in theoretical analysis such as deterministic setting only, homogeneity of local functions, etc.
Revised: But most of these methods have their small shortcomings in theoretical analysis such as deterministic setting only, homogeneity of local functions, etc.

Original: All our methods MASHA1 , MASHA2 and their modiﬁcations support bidirectional compression.
Revised: All our methods MASHA1 , MASHA2 and their modiﬁcations support bidirectional compression.

Original: See
Revised: See

Original: Appendix D and E for details.
Revised: Appendix D and E for details.

Original: 2.6 Experiments
Revised: 2.6 Experiments

Original: Toy experiments on bilinear problems show that methods with compression for minimization problems may not work (diverge) for SPPs.
Revised: Toy experiments on bilinear problems show that methods with compression for minimization problems may not work (diverge) for SPPs.

Original: Also we verify that MASHA1 and MASHA2 are much better than the classical Extra Gradient with added unbiased compression.
Revised: Also we verify that MASHA1 and MASHA2 are much better than the classical Extra Gradient with added unbiased compression.

Original: Experiments on adversarial training of large-scale transformer (ALBERT) show the practical importance of compression in distributed methods for large SPPs.
Revised: Experiments on adversarial training of large-scale transformer (ALBERT) show the practical importance of compression in distributed methods for large SPPs.

Original:  Problem Formulation and Assumptions 3.1 Problem formulation
Revised:  Problem Formulation and Assumptions 3.1 Problem formulation

Original: We study distributed variational inequality (VI) problem
Revised: We study distributed variational inequality (VI) problem

Original: Find z ⇤ 2 R d such that h F ( z ⇤ ) , z  z
Revised: Find z ⇤ 2 R d such that h F ( z ⇤ ) , z  z

Original: ⇤ i  0 , 8 z 2 R d , (
Revised: ⇤ i  0 , 8 z 2 R d , (

Original: [Equation]
Revised: [Equation]

Original: where F : R d !
Revised: where F : R d !

Original: R d is an operator with certain favorable properties (e.g., Lipschitzness and monotonicity).
Revised: R d is an operator with certain favorable properties (e.g., Lipschitzness and monotonicity).

Original: We assume that the training data describing F is distributed across M workers/nodes/clients
Revised: We assume that the training data describing F is distributed across M workers/nodes/clients

Original: [Equation]
Revised: [Equation]

Original: where F m : R d ! R d for all m 2 { 1 , 2 , . .
Revised: where F m : R d ! R d for all m 2 { 1 , 2 , . .

Original: , M } .
Revised: , M } .

Original: Next, we give main examples of VIs to show the breadth of this formalism.
Revised: Next, we give main examples of VIs to show the breadth of this formalism.

Original: Example 3.1 (Minimization) Consider the minimization problem:
Revised: Example 3.1 (Minimization) Consider the minimization problem:

Original: [Equation]
Revised: [Equation]

Original: Suppose that F ( z ) :
Revised: Suppose that F ( z ) :

Original: = r f ( z ) .
Revised: = r f ( z ) .

Original: Then, if f is convex, it can be proved that z ⇤ 2 R d is a solution for (3) if and only if z ⇤ 2 R d is a solution for (5) .
Revised: Then, if f is convex, it can be proved that z ⇤ 2 R d is a solution for (3) if and only if z ⇤ 2 R d is a solution for (5) .

Original: And if the function f is non-convex, then z ⇤ 2 R d is a solution for (3) if and only
Revised: And if the function f is non-convex, then z ⇤ 2 R d is a solution for (3) if and only

Original: if r f ( z ⇤ ) = 0 , i.e. z ⇤ is a stationary point.
Revised: if r f ( z ⇤ ) = 0 , i.e. z ⇤ is a stationary point.

Original: Example 3.2 (Saddle point problem) Consider the saddle point problem:
Revised: Example 3.2 (Saddle point problem) Consider the saddle point problem:

Original: [Equation]
Revised: [Equation]

Original: Suppose that F ( z ) :
Revised: Suppose that F ( z ) :

Original: = F ( x, y ) =
Revised: = F ( x, y ) =

Original: [ r x g ( x, y ) , r y g ( x, y )] and Z =
Revised: [ r x g ( x, y ) , r y g ( x, y )] and Z =

Original: R d x ⇥ R d y .
Revised: R d x ⇥ R d y .

Original: Then, if g is convex-concave, it can be proved that z ⇤ 2 Z is a solution for (3)
Revised: Then, if g is convex-concave, it can be proved that z ⇤ 2 Z is a solution for (3)

Original: if and only if z ⇤ 2 Z is a solution for (6) .
Revised: if and only if z ⇤ 2 Z is a solution for (6) .

Original: And if the function g is non-convex-non-concave, then z ⇤ 2 Z is a solution for (3) if and only if r x g ( x ⇤ , y ⇤ )
Revised: And if the function g is non-convex-non-concave, then z ⇤ 2 Z is a solution for (3) if and only if r x g ( x ⇤ , y ⇤ )

Original: = 

Revised: =

Original: and r y g ( x ⇤ , y ⇤ )
Revised: and r y g ( x ⇤ , y ⇤ )

Original: = 0 , i.e. z ⇤ is a stationary point.
Revised: = 0 , i.e. z ⇤ is a stationary point.

Original: If minimization problems are widely researched separately from variational inequalities.
Revised: If minimization problems are widely researched separately from variational inequalities.

Original: The study of saddle point problems often is associated with variational inequalities, therefore saddle point problems are strongly related to variational inequalities.
Revised: The study of saddle point problems often is associated with variational inequalities, therefore saddle point problems are strongly related to variational inequalities.

Original: Example 3.3 (Fixed point problem) Consider the ﬁxed point problem:
Revised: Example 3.3 (Fixed point problem) Consider the ﬁxed point problem:

Original: Find z ⇤ 2 R d such that T ( z ⇤ )
Revised: Find z ⇤ 2 R d such that T ( z ⇤ )

Original: =
Revised: =

Original: z ⇤ , (7) where T : R d !
Revised: z ⇤ , (7) where T : R d !

Original: R d is an operator.
Revised: R d is an operator.

Original: With F ( z ) =
Revised: With F ( z ) =

Original: z  T ( z ) , it can be proved that z ⇤ 2 R d is a solution for (3) if and only
Revised: z  T ( z ) , it can be proved that z ⇤ 2 R d is a solution for (3) if and only

Original: if F ( z ⇤ ) = 0 , i.e. z ⇤ 2 R d is a solution for (7) .
Revised: if F ( z ⇤ ) = 0 , i.e. z ⇤ 2 R d is a solution for (7) .

Original: 3.2 Assumptions
Revised: 3.2 Assumptions

Original: Next, we list two key assumptions - both are standard in the literature on VIs.
Revised: Next, we list two key assumptions - both are standard in the literature on VIs.

Original: Assumption 3.4 (Lipschitzness)
Revised: Assumption 3.4 (Lipschitzness)

Original: The operator F is L -Lipschitz continuous, i.e. for all z 1 , z
Revised: The operator F is L -Lipschitz continuous, i.e. for all z 1 , z

Original: 2 2 R d we have k F ( z 1 )  F ( z 2 )
Revised: 2 2 R d we have k F ( z 1 )  F ( z 2 )

Original: k  L k z 1  z 2 k .
Revised: k  L k z 1  z 2 k .

Original: Each operator F m is L m -Lipschitz continuous, i.e. for all z 1 , z 2 2 R d
Revised: Each operator F m is L m -Lipschitz continuous, i.e. for all z 1 , z 2 2 R d

Original: it holds k F m ( z 1 ) 
Revised: it holds k F m ( z 1 )

Original: F m ( z 2 )
Revised: F m ( z 2 )

Original: k
Revised: k

Original:  L m k z 1  z 2 k .
Revised:  L m k z 1  z 2 k .

Original: Let us deﬁne new constant ˜ L as follows ˜
Revised: Let us deﬁne new constant ˜ L as follows ˜

Original: L 2 = 1 M M
Revised: L 2 = 1 M M

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: For saddle point problems, these properties are equivalent to smoothness.
Revised: For saddle point problems, these properties are equivalent to smoothness.

Original: Assumption 3.5 (Monotonicity)
Revised: Assumption 3.5 (Monotonicity)

Original: We need three cases of monotonicity
Revised: We need three cases of monotonicity

Original: (SM)
Revised: (SM)

Original: Strong monotonicity.
Revised: Strong monotonicity.

Original: The operator F is µ -strongly monotone,
Revised: The operator F is µ -strongly monotone,

Original: i
Revised: i

Original: e. for all z 1 , z 2 2 R d we have
Revised: e. for all z 1 , z 2 2 R d we have

Original: h F ( z 1 )  F ( z 2 ) , z 1  z 2 i  µ k z 1  z 2 k 2 .
Revised: h F ( z 1 )  F ( z 2 ) , z 1  z 2 i  µ k z 1  z 2 k 2 .

Original: (M) Monotonicity.
Revised: (M) Monotonicity.

Original: The operator F is monotone,
Revised: The operator F is monotone,

Original: i
Revised: i

Original: e. for all z 1 , z 2 2 R d we have
Revised: e. for all z 1 , z 2 2 R d we have

Original: h F ( z 1 )  F ( z 2 ) , z 1  z 2 i  0 .
Revised: h F ( z 1 )  F ( z 2 ) , z 1  z 2 i  0 .

Original: (NM) Non-monotonicity.
Revised: (NM) Non-monotonicity.

Original: The operator F is non-monotone (minty), if and only if there exists z ⇤R d such that for all z 2 R d we have h F ( z ) , z  z
Revised: The operator F is non-monotone (minty), if and only if there exists z ⇤R d such that for all z 2 R d we have h F ( z ) , z  z

Original: ⇤ i  0 .
Revised: ⇤ i  0 .

Original: The last assumption is called the minty or variational stability condition.
Revised: The last assumption is called the minty or variational stability condition.

Original: It is not a general nonmonotonicity, but is already associated in the community with non-monotonicity [14, 37, 58, 53, 43, 36, 19], particularly with the setup, which is somewhat appropriate for GANS [51, 52, 21, 8].
Revised: It is not a general nonmonotonicity, but is already associated in the community with non-monotonicity [14, 37, 58, 53, 43, 36, 19], particularly with the setup, which is somewhat appropriate for GANS [51, 52, 21, 8].

Original:  MASHA
Revised:  MASHA

Original: In this Section we present new algorithms and their convergence.
Revised: In this Section we present new algorithms and their convergence.

Original: Section 4.1 is devoted to the algorithm ( MASHA1 ) with unbiased compression.
Revised: Section 4.1 is devoted to the algorithm ( MASHA1 ) with unbiased compression.

Original: Section 4.2 – to algorithm ( MASHA2 ) with contractive compression.
Revised: Section 4.2 – to algorithm ( MASHA2 ) with contractive compression.

Original: Appendix gives modiﬁcations for the stochastic case – Section F, and for the federated learning – Section G. Appendix B is devoted to the hypothesis about optimality of MASHA and MASHA2 .
Revised: Appendix gives modiﬁcations for the stochastic case – Section F, and for the federated learning – Section G. Appendix B is devoted to the hypothesis about optimality of MASHA and MASHA2 .

Original: 4.1 MASHA1 : Handling Unbiased Compressors
Revised: 4.1 MASHA1 : Handling Unbiased Compressors

Original: Before presenting our algorithm, let us discuss which approaches can be used to construct it.
Revised: Before presenting our algorithm, let us discuss which approaches can be used to construct it.

Original: As discussed in Sections 1 and 2, compression methods play an important role in distributed minimization problems.
Revised: As discussed in Sections 1 and 2, compression methods play an important role in distributed minimization problems.

Original: All these methods are modiﬁcations of the classical GD.
Revised: All these methods are modiﬁcations of the classical GD.

Original: For instance, the authors of [2] compress stochastic gradients.
Revised: For instance, the authors of [2] compress stochastic gradients.

Original: Therefore, it is a natural idea to use GD-type methods for VIs as well.
Revised: Therefore, it is a natural idea to use GD-type methods for VIs as well.

Original: But it is a well-known fact that GD-type methods can give bad convergence estimates (see
Revised: But it is a well-known fact that GD-type methods can give bad convergence estimates (see

Original: Section B.1 from [67]) or do not converge at all (see Section 7.2 and 8.2 from [27]) even on the simplest SPPs and VIs.
Revised: Section B.1 from [67]) or do not converge at all (see Section 7.2 and 8.2 from [27]) even on the simplest SPPs and VIs.

Original: From a practical point of view, this approach can also fail (see QSGD and
Revised: From a practical point of view, this approach can also fail (see QSGD and

Original: EF in Section 5.1).
Revised: EF in Section 5.1).

Original: In the non-distributed case, this problem has long been solved and the Extra
Revised: In the non-distributed case, this problem has long been solved and the Extra

Original: Gradient method [46, 62, 41] is used instead of GD:
Revised: Gradient method [46, 62, 41] is used instead of GD:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: This method is optimal for both VIs and SPPs and has an estimate of convergence
Revised: This method is optimal for both VIs and SPPs and has an estimate of convergence

Original: ˜ O
Revised: ˜ O

Original: ( L / µ ) in the strongly monotone case.
Revised: ( L / µ ) in the strongly monotone case.

Original: Therefore, the second idea for the compressed method is to add compression operators to the method (8), e.g. use Q k ( F ( z k )) and Q k +1 / 2 ( F ( z k +1 / 2 )) instead of F ( z k ) and
Revised: Therefore, the second idea for the compressed method is to add compression operators to the method (8), e.g. use Q k ( F ( z k )) and Q k +1 / 2 ( F ( z k +1 / 2 )) instead of F ( z k ) and

Original: F ( z k +1 / 2 ) .
Revised: F ( z k +1 / 2 ) .

Original: In Section H we analyse this method, but it gives an estimate ˜ O
Revised: In Section H we analyse this method, but it gives an estimate ˜ O

Original: [Equation]
Revised: [Equation]

Original: which is considerably worse in terms of L / µ than the original Extra Gradient method.
Revised: which is considerably worse in terms of L / µ than the original Extra Gradient method.

Original: The key problem is that in the analysis one has to deal with k Q k ( F ( z k +1 / 2 ))
Revised: The key problem is that in the analysis one has to deal with k Q k ( F ( z k +1 / 2 ))

Original:  Q k +1 / 2 ( F ( z k ))
Revised:  Q k +1 / 2 ( F ( z k ))

Original: k 2 .
Revised: k 2 .

Original: Without compression operators, such difference is easily evaluated using Assumption 3.4.
Revised: Without compression operators, such difference is easily evaluated using Assumption 3.4.

Original: But when the compression operators are different (in fact the same, but have different randomness) we cannot make a good estimate for this term.
Revised: But when the compression operators are different (in fact the same, but have different randomness) we cannot make a good estimate for this term.

Original: The idea arises to use the same randomness in both steps of the method (8), namely to substitute Q k ( F ( z k )) and Q k ( F ( z k +1 / 2 )) .
Revised: The idea arises to use the same randomness in both steps of the method (8), namely to substitute Q k ( F ( z k )) and Q k ( F ( z k +1 / 2 )) .

Original: But then z k +1 / 2 depends on the randomness Q k , and hence Q k ( F ( z k +1 / 2 )) is biased, which further complicates the analysis.
Revised: But then z k +1 / 2 depends on the randomness Q k , and hence Q k ( F ( z k +1 / 2 )) is biased, which further complicates the analysis.

Original: For exactly the same reasons, the various optimistic/single call modiﬁcations [70, 26, 35, 60] of the Extra Gradient method did not work for us either.
Revised: For exactly the same reasons, the various optimistic/single call modiﬁcations [70, 26, 35, 60] of the Extra Gradient method did not work for us either.

Original: We have also test the method (8) with compressions in practice (see CEG in Section 5.1), and it turns out to be worse than the method we will present below.
Revised: We have also test the method (8) with compressions in practice (see CEG in Section 5.1), and it turns out to be worse than the method we will present below.

Original: In the end, the use of variance reduction and negative momentum techniques [1] is key in creating our algorithm.
Revised: In the end, the use of variance reduction and negative momentum techniques [1] is key in creating our algorithm.

Original: These tricks are not in themselves relevant to distributed problems, but, in our case, they help in creating MASHA1 and MASHA2 .
Revised: These tricks are not in themselves relevant to distributed problems, but, in our case, they help in creating MASHA1 and MASHA2 .

Original: Algorithm 1 MASHA
Revised: Algorithm 1 MASHA

Original: Parameters: Stepsize  > 0 , parameter ⌧ 2 (0; 1) , number of iterations K .
Revised: Parameters: Stepsize  > 0 , parameter ⌧ 2 (0; 1) , number of iterations K .

Original: Initialization: Choose z
Revised: Initialization: Choose z

Original: =
Revised: =

Original: w 0 2 Z .
Revised: w 0 2 Z .

Original: Devices send F m ( w 0 ) to server and get F ( w 0 ) for k = 0 , 1 , 2 , . .
Revised: Devices send F m ( w 0 ) to server and get F ( w 0 ) for k = 0 , 1 , 2 , . .

Original: , K  1 do for each device m in parallel do z
Revised: , K  1 do for each device m in parallel do z

Original: [Equation]
Revised: [Equation]

Original: 
Revised:

Original: [Equation]
Revised: [Equation]

Original: w
Revised: w

Original: [Equation]
Revised: [Equation]

Original: Sends g km =
Revised: Sends g km =

Original: Q dev m ( F m ( z k +1 / 2 )  F m ( w k )) to server end for for server do
Revised: Q dev m ( F m ( z k +1 / 2 )  F m ( w k )) to server end for for server do

Original: Sends to devices g k = Q serv h 1 M
Revised: Sends to devices g k = Q serv h 1 M

Original: P
Revised: P

Original: Mm
Revised: Mm

Original: [Equation]
Revised: [Equation]

Original: g
Revised: g

Original: [Equation]
Revised: [Equation]

Original: i Sends to devices one bit b k : 1 with probability 1  ⌧ , 0 with with probability ⌧ end for for each device m in parallel do
Revised: i Sends to devices one bit b k : 1 with probability 1  ⌧ , 0 with with probability ⌧ end for for each device m in parallel do

Original: [Equation]
Revised: [Equation]

Original: If b k = 1 then w k +1 =
Revised: If b k = 1 then w k +1 =

Original: z k , sends F m ( w k +1 ) to server and gets F ( w k +1 )
Revised: z k , sends F m ( w k +1 ) to server and gets F ( w k +1 )

Original: else w k +

Revised: else w k +

Original: =
Revised: =

Original: w k end for end for
Revised: w k end for end for

Original: At the beginning of each MASHA1 iteration, all devices know the value of F ( w k ) , hence they can calculate the value of z k +1 / 2 locally without communications.
Revised: At the beginning of each MASHA1 iteration, all devices know the value of F ( w k ) , hence they can calculate the value of z k +1 / 2 locally without communications.

Original: Further, each device sends the com- pressed version of the difference F m ( z k +1 / 2 )
Revised: Further, each device sends the com- pressed version of the difference F m ( z k +1 / 2 )

Original:  F m ( w k ) to the server.
Revised:  F m ( w k ) to the server.

Original: The compression on these transfers is done by their local { Q dev m } operators.
Revised: The compression on these transfers is done by their local { Q dev m } operators.

Original: The server aggregates the information from devices, averages it, compresses by Q serv operator and makes a broadcast to all devices.
Revised: The server aggregates the information from devices, averages it, compresses by Q serv operator and makes a broadcast to all devices.

Original: As a result, an unbiased estimate of F ( z k +1 / 2 )  F ( w k ) appears at each node.
Revised: As a result, an unbiased estimate of F ( z k +1 / 2 )  F ( w k ) appears at each node.

Original: Also, the nodes receive one bit of information b k .
Revised: Also, the nodes receive one bit of information b k .

Original: This bit is generated randomly on the server and is equal to 1 with probability 1  ⌧ (where 1  ⌧ is small).
Revised: This bit is generated randomly on the server and is equal to 1 with probability 1  ⌧ (where 1  ⌧ is small).

Original: Note that b k can be generated locally, it is enough to use the same random generator and set the same seed on all devices.
Revised: Note that b k can be generated locally, it is enough to use the same random generator and set the same seed on all devices.

Original: Next, the devices locally make a ﬁnal update on z k +1 .
Revised: Next, the devices locally make a ﬁnal update on z k +1 .

Original: The ﬁnal step is an update of w k +1 : if b k = 1 , then w k +1 =
Revised: The ﬁnal step is an update of w k +1 : if b k = 1 , then w k +1 =

Original: z k or otherwise w k +

Revised: z k or otherwise w k +

Original: =
Revised: =

Original: w k .
Revised: w k .

Original: In the case when w k +

Revised: In the case when w k +

Original: =
Revised: =

Original: z k , we need to exchange the uncompressed values of F m ( w k +1 ) in order to ensure that at the beginning of the next iteration the value of F ( w k +1 ) is known to all agents.
Revised: z k , we need to exchange the uncompressed values of F m ( w k +1 ) in order to ensure that at the beginning of the next iteration the value of F ( w k +1 ) is known to all agents.

Original: We use a possibly difference compressor on each device and also on the server.
Revised: We use a possibly difference compressor on each device and also on the server.

Original: To distinguish between them, we denote the following notation: Q dev m , q dev m ,  dev m and Q serv , q serv ,  serv .
Revised: To distinguish between them, we denote the following notation: Q dev m , q dev m ,  dev m and Q serv , q serv ,  serv .

Original: Theorem 4.1 Let Assumption 3.4 and one case of Assumption 3.5 are satisﬁed.
Revised: Theorem 4.1 Let Assumption 3.4 and one case of Assumption 3.5 are satisﬁed.

Original: Then for some step  the following estimates on MASHA1 number of iterations to achieve " -solution holds
Revised: Then for some step  the following estimates on MASHA1 number of iterations to achieve " -solution holds

Original: • in strongly monotone case (in terms of E [ k z K  z ⇤ k 2 ] ⇠ " ): O ([ 11  ⌧ +
Revised: • in strongly monotone case (in terms of E [ k z K  z ⇤ k 2 ] ⇠ " ): O ([ 11  ⌧ +

Original: [Equation]
Revised: [Equation]

Original: µ p 1  ⌧ ] log 1 " ); • in monotone case (in terms of E max z 2 C [ h F ( u ) , ( 1 K
Revised: µ p 1  ⌧ ] log 1 " ); • in monotone case (in terms of E max z 2 C [ h F ( u ) , ( 1 K

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: • in non-
Revised: • in non-

Original: monotone
Revised: monotone

Original: [Equation]
Revised: [Equation]

Original: in terms of
Revised: in terms of

Original: [Equation]
Revised: [Equation]

Original: where C 2 q = q
Revised: where C 2 q = q

Original: [Equation]
Revised: [Equation]

Original: A full description of the algorithm, as well as a full statement of the theorem with proof, can be found in Appendix D.
Revised: A full description of the algorithm, as well as a full statement of the theorem with proof, can be found in Appendix D.

Original: The bounds in Theorem 4.1 are related to ⌧ .
Revised: The bounds in Theorem 4.1 are related to ⌧ .

Original: Let us ﬁnd an optimal way to choose it.
Revised: Let us ﬁnd an optimal way to choose it.

Original: Note that (in average) once per 1 / (1  ⌧ ) iterations (when b k = 1 ), we send uncompressed information.
Revised: Note that (in average) once per 1 / (1  ⌧ ) iterations (when b k = 1 ), we send uncompressed information.

Original: Based on this observation, we can ﬁnd the best option for ⌧ .
Revised: Based on this observation, we can ﬁnd the best option for ⌧ .

Original: Let us analyze the case of compressions only on the devices’ side ( q serv = 1 ).
Revised: Let us analyze the case of compressions only on the devices’ side ( q serv = 1 ).

Original: For simplicity, we put Q dev m =
Revised: For simplicity, we put Q dev m =

Original: Q with q dev m =
Revised: Q with q dev m =

Original: q and  dev m
Revised: q and  dev m

Original: =
Revised: =

Original:  , also
Revised:  , also

Original: L m =
Revised: L m =

Original: ˜ L = L .
Revised: ˜ L = L .

Original: Since compression is done only on devices, we assume that the server’s broadcast is cheap and we only care about devices.
Revised: Since compression is done only on devices, we assume that the server’s broadcast is cheap and we only care about devices.

Original: Then at each iteration the device sends O ( 1 /  + 1  ⌧ ) bits each
Revised: Then at each iteration the device sends O ( 1 /  + 1  ⌧ ) bits each

Original: time information compressed by  times and with probability 1  ⌧ we send the full package.
Revised: time information compressed by  times and with probability 1  ⌧ we send the full package.

Original: From where we immediately get the optimal choice for ⌧ :
Revised: From where we immediately get the optimal choice for ⌧ :

Original: Corollary 4.2 Let Assumption 3.4 and one case of Assumption 3.5 are satisﬁed.
Revised: Corollary 4.2 Let Assumption 3.4 and one case of Assumption 3.5 are satisﬁed.

Original: Then for some step  and 1  ⌧ = 1 /  the following estimates on MASHA1 number of iterations to achieve " -solution holds
Revised: Then for some step  and 1  ⌧ = 1 /  the following estimates on MASHA1 number of iterations to achieve " -solution holds

Original: • in strongly monotone case: O ([  + q q  M +  · Lµ ] log 1 " ); • in monotone case: O ( q q  M +  · L k z 

Revised: • in strongly monotone case: O ([  + q q  M +  · Lµ ] log 1 " ); • in monotone case: O ( q q  M +  · L k z

Original:  z
Revised:  z

Original: ⇤ k 2 " ); • in non-monotone case: O ([ q  M +  ] L 2 k z 

Revised: ⇤ k 2 " ); • in non-monotone case: O ([ q  M +  ] L 2 k z

Original:  z
Revised:  z

Original: ⇤ k 2 ") .
Revised: ⇤ k 2 ") .

Original: We can see that MASHA1 can outperform the uncompressed Extra Gradient method.
Revised: We can see that MASHA1 can outperform the uncompressed Extra Gradient method.

Original: Let us compare them in the strongly monotone case.
Revised: Let us compare them in the strongly monotone case.

Original: The communication complexity of the Extra Gradient method is ˜
Revised: The communication complexity of the Extra Gradient method is ˜

Original: O ( L / µ ) .
Revised: O ( L / µ ) .

Original: MASHA1 has communication complexity ˜
Revised: MASHA1 has communication complexity ˜

Original: O
Revised: O

Original: ( p q /  M + 1 /  · L / µ ) .
Revised: ( p q /  M + 1 /  · L / µ ) .

Original: For practical compressors [10],   q .
Revised: For practical compressors [10],   q .

Original: Then, one can note that the communication complexity of MASHA1 differs from the complexity of the uncompressed method by an additional factor ( p 1 / M + 1 /  ) .
Revised: Then, one can note that the communication complexity of MASHA1 differs from the complexity of the uncompressed method by an additional factor ( p 1 / M + 1 /  ) .

Original: It is easy to see that even for a small number of devices M and expected density  , this factor is less than 1 , hence MASHA1 outperforms the uncompressed method.
Revised: It is easy to see that even for a small number of devices M and expected density  , this factor is less than 1 , hence MASHA1 outperforms the uncompressed method.

Original: We think that this factor ( p 1 / M + 1 /  ) is theoretically unimprovable and optimal – see Section B for details.
Revised: We think that this factor ( p 1 / M + 1 /  ) is theoretically unimprovable and optimal – see Section B for details.

Original: One can also consider the case of bidirectional compression ( q serv 6 = 1 ).
Revised: One can also consider the case of bidirectional compression ( q serv 6 = 1 ).

Original: Table 1 (line 3) shows the result for q serv =
Revised: Table 1 (line 3) shows the result for q serv =

Original: q dev m = q ,  serv =
Revised: q dev m = q ,  serv =

Original:  dev m =
Revised:  dev m =

Original:  and 1  ⌧ = 1 /  .
Revised:  and 1  ⌧ = 1 /  .

Original: 4.2 MASHA2 : Handling Contractive Compressors
Revised: 4.2 MASHA2 : Handling Contractive Compressors

Original: The use of contractive compressions is a more complex issue.
Revised: The use of contractive compressions is a more complex issue.

Original: In particular, it is known that if one simply put a contractive cospressor instead of an unbiased one, the method may diverge even for quadratic problems [10].
Revised: In particular, it is known that if one simply put a contractive cospressor instead of an unbiased one, the method may diverge even for quadratic problems [10].

Original: To ﬁx this, an error compensation technique [78, 44, 79] is used.
Revised: To ﬁx this, an error compensation technique [78, 44, 79] is used.

Original: The point of this approach is to keep untransmitted information and add it to a new package at the next iteration.
Revised: The point of this approach is to keep untransmitted information and add it to a new package at the next iteration.

Original: This is the main difference between MASHA2 and MASHA1 .
Revised: This is the main difference between MASHA2 and MASHA1 .

Original: MASHA2 introduces additional sequences e k , e km for the server’s and devices’ error.
Revised: MASHA2 introduces additional sequences e k , e km for the server’s and devices’ error.

Original: To deﬁne contractive operators on devices and on the server, we introduce the following notation: C dev m ,  dev ,  dev and C serv m ,  serv ,  serv .
Revised: To deﬁne contractive operators on devices and on the server, we introduce the following notation: C dev m ,  dev ,  dev and C serv m ,  serv ,  serv .

Original: Algorithm
Revised: Algorithm

Original: Parameters: Stepsize  > 0 , parameter ⌧ , number of iterations K .
Revised: Parameters: Stepsize  > 0 , parameter ⌧ , number of iterations K .

Original: Initialization: Choose z
Revised: Initialization: Choose z

Original: =
Revised: =

Original: w 0 2 Z , e 0 m = 0 , e 0 = 0 .
Revised: w 0 2 Z , e 0 m = 0 , e 0 = 0 .

Original: Devices send F m ( w 0 ) to server and get F ( w 0 ) for k = 0 , 1 , 2 , . .
Revised: Devices send F m ( w 0 ) to server and get F ( w 0 ) for k = 0 , 1 , 2 , . .

Original: , K  1 do for each device m in parallel do z k +1 / 2 =
Revised: , K  1 do for each device m in parallel do z k +1 / 2 =

Original: ⌧ z k + (1  ⌧ ) w k   F ( w k )
Revised: ⌧ z k + (1  ⌧ ) w k   F ( w k )

Original: Sends g km =
Revised: Sends g km =

Original: C dev m (  F m ( z k +1 / 2 )   F m ( w k ) + e km ) to server e k +1 m =
Revised: C dev m (  F m ( z k +1 / 2 )   F m ( w k ) + e km ) to server e k +1 m =

Original: e km +  F m ( z k +1 / 2 )   F m ( w k )  g km end for for server do
Revised: e km +  F m ( z k +1 / 2 )   F m ( w k )  g km end for for server do

Original: Sends to devices g k = C serv h 1 M
Revised: Sends to devices g k = C serv h 1 M

Original: P
Revised: P

Original: Mm =1 g km + e k
Revised: Mm =1 g km + e k

Original: i e k
Revised: i e k

Original: +

Revised: +

Original: =
Revised: =

Original: e k + 1 M
Revised: e k + 1 M

Original: P
Revised: P

Original: Mm =1 g km  g k Sends to devices one bit b k : 1 with probability ⌧ , 0 with with probability ⌧ end for for each device m in parallel do z
Revised: Mm =1 g km  g k Sends to devices one bit b k : 1 with probability ⌧ , 0 with with probability ⌧ end for for each device m in parallel do z

Original: k +1 =
Revised: k +1 =

Original: z k +1 / 2   g k
Revised: z k +1 / 2   g k

Original: If b k = 1 then w k +1 =
Revised: If b k = 1 then w k +1 =

Original: z k , sends F m ( w k +1 ) to server and gets F ( w k +1 )
Revised: z k , sends F m ( w k +1 ) to server and gets F ( w k +1 )

Original: else w k +

Revised: else w k +

Original: =
Revised: =

Original: w k end for end for
Revised: w k end for end for

Original: In the case of MASHA1 , the key theoretical issue was the choice of a basic method (we discussed this at the beginning of Section 4.1).
Revised: In the case of MASHA1 , the key theoretical issue was the choice of a basic method (we discussed this at the beginning of Section 4.1).

Original: MASHA2 raises another problem for theoretical analysis, how to combine MASHA1 and the error feedback technique.
Revised: MASHA2 raises another problem for theoretical analysis, how to combine MASHA1 and the error feedback technique.

Original: The analysis of methods with error compensation for the minimization problem min x f ( x ) is entirely tied to the existence of the function f [79, 71, 72].
Revised: The analysis of methods with error compensation for the minimization problem min x f ( x ) is entirely tied to the existence of the function f [79, 71, 72].

Original: In particular, the differences ( f ( · )  f ( x ⇤ )) appear in the whole analysis and is key in the technical lemmas.
Revised: In particular, the differences ( f ( · )  f ( x ⇤ )) appear in the whole analysis and is key in the technical lemmas.

Original: As a result ( f ( · )  f ( x ⇤ )) is used as a convergence criterion even in the strongly convex case.
Revised: As a result ( f ( · )  f ( x ⇤ )) is used as a convergence criterion even in the strongly convex case.

Original: But for VIs there is no function f , only the operator F (the existence of g ( x, y ) in SPP setup does not save the situation).
Revised: But for VIs there is no function f , only the operator F (the existence of g ( x, y ) in SPP setup does not save the situation).

Original: This problem is solved in the proof of Theorem 4.3 by using an additional sequence k z k +1 / 2  w k k .
Revised: This problem is solved in the proof of Theorem 4.3 by using an additional sequence k z k +1 / 2  w k k .

Original: Theorem 4.3 Let Assumption 3.4 and one case of Assumption 3.5 are satisﬁed.
Revised: Theorem 4.3 Let Assumption 3.4 and one case of Assumption 3.5 are satisﬁed.

Original: Then for some step  the following estimates on MASHA2 number of iterations to achieve " -solution holds
Revised: Then for some step  the following estimates on MASHA2 number of iterations to achieve " -solution holds

Original: • in strongly monotone case (in terms of E [ k ˆ z K  z ⇤ k 2 ] ⇠ " ): O ([ 11  ⌧ +  dev  serv ˜
Revised: • in strongly monotone case (in terms of E [ k ˆ z K  z ⇤ k 2 ] ⇠ " ): O ([ 11  ⌧ +  dev  serv ˜

Original: [Equation]
Revised: [Equation]

Original: µ p 1  ⌧ ] log 1 " ); • in monotone case ( E max z 2 C [ h F ( u ) , ( 1 K
Revised: µ p 1  ⌧ ] log 1 " ); • in monotone case ( E max z 2 C [ h F ( u ) , ( 1 K

Original: [Equation]
Revised: [Equation]

Original: ); • in non-monotone case (in terms of E [ 1 K P
Revised: ); • in non-monotone case (in terms of E [ 1 K P

Original: [Equation]
Revised: [Equation]

Original: A full listing of the algorithm, as well as a full statement of the theorem with proof, can be found in
Revised: A full listing of the algorithm, as well as a full statement of the theorem with proof, can be found in

Original: Appendix E.
Revised: Appendix E.

Original: The same way as in Section 4.1 we can consider only devices’ or bidirectional compression.
Revised: The same way as in Section 4.1 we can consider only devices’ or bidirectional compression.

Original: In particular, in the line 2 of Table 1 we put results for  serv = 1 ,  dev =
Revised: In particular, in the line 2 of Table 1 we put results for  serv = 1 ,  dev =

Original:  , L m =
Revised:  , L m =

Original: 5.1 Bilinear Saddle Point Problem
Revised: 5.1 Bilinear Saddle Point Problem

Original: We start our experiments with a distributed bilinear problem, i.e. the problem ( ?? ) with
Revised: We start our experiments with a distributed bilinear problem, i.e. the problem (6) with

Original: [Equation]
Revised: [Equation]

Original: where A m 2 R d ⇥ d , a m , b m 2 R d .
Revised: where A m 2 R d ⇥ d , a m , b m 2 R d .

Original: This problem is  -strongly convex–strongly-concave and, moreover, all functions g m are k
Revised: This problem is  -strongly convex–strongly-concave and, moreover, all functions g m are k

Original: A m k 2 -smooth.
Revised: A m k 2 -smooth.

Original: Therefore, such a distributed problem is well suited for the primary comparison of our methods.
Revised: Therefore, such a distributed problem is well suited for the primary comparison of our methods.

Original: We take d = 100 and generate positive deﬁnite matrices
Revised: We take d = 100 and generate positive deﬁnite matrices

Original: A m and vectors a m , b m randomly,  is chosen as max m k
Revised: A m and vectors a m , b m randomly,  is chosen as max m k

Original: A m k 2 / 10 5 .
Revised: A m k 2 / 10 5 .

Original: The purpose of the experiment is to understand whether the MASHA1 and MASHA2 methods are superior to those in the literature.
Revised: The purpose of the experiment is to understand whether the MASHA1 and MASHA2 methods are superior to those in the literature.

Original: As a comparison, we take QGD [2] with Random 30%, classical
Revised: As a comparison, we take QGD [2] with Random 30%, classical

Original: Error Feedback [78] with Top 30% compression, as well as CEG (Section H) –
Revised: Error Feedback [78] with Top 30% compression, as well as CEG (Section H) –

Original: Compressed Extra
Revised: Compressed Extra

Original: Gradient, each step of which we use Random 30%.
Revised: Gradient, each step of which we use Random 30%.

Original: In MASHA1 (Algorithm 1) we also used Random 30%, in MASHA2 (Algorithm 2) –
Revised: In MASHA1 (Algorithm 1) we also used Random 30%, in MASHA2 (Algorithm 2) –

Original: Top 30%.
Revised: Top 30%.

Original: See Figure 1.
Revised: See Figure 1.

Original: The stepsizes of all methods are chosen for best convergence.
Revised: The stepsizes of all methods are chosen for best convergence.

Original: We see on Figurethat methods based on gradient descent (QSGD and EF) converge slowly.
Revised: We see on Figurethat methods based on gradient descent (QSGD and EF) converge slowly.

Original: This conﬁrms that one needs to use method speciﬁcally designed for saddle point problems (for example, the extragradient method), and not classical optimization methods.
Revised: This conﬁrms that one needs to use method speciﬁcally designed for saddle point problems (for example, the extragradient method), and not classical optimization methods.

Original: The much slower convergence of CEG shows the efﬁciency of our approach in which we compress the differences F m ( z k +1 / 2 )  F ( w k ) .
Revised: The much slower convergence of CEG shows the efﬁciency of our approach in which we compress the differences F m ( z k +1 / 2 )  F ( w k ) .

Original: MASHA2 wins MASHA1 .
Revised: MASHA2 wins MASHA1 .

Original: This shows that in practice a contractive compressor can perform better than an unbiased one with the same parameters.
Revised: This shows that in practice a contractive compressor can perform better than an unbiased one with the same parameters.

Original: 5.2 Adversarial Training of Transformers
Revised: 5.2 Adversarial Training of Transformers

Original: We now evaluate how compression performs for variational inequalities (and for saddle point problems, as a special case) in a more practically motivated scenario.
Revised: We now evaluate how compression performs for variational inequalities (and for saddle point problems, as a special case) in a more practically motivated scenario.

Original: Indeed, saddle point problems (special case of variational inequalities) have sample applications in machine learning, including adversarial training .
Revised: Indeed, saddle point problems (special case of variational inequalities) have sample applications in machine learning, including adversarial training .

Original: And our goal is to show that compression provides important improvements for such large-scale problems as well.
Revised: And our goal is to show that compression provides important improvements for such large-scale problems as well.

Original: We train a transformer-based masked language model [82, 18, 56] using a ﬂeet of 16 low-cost preemptible workers with T4 GPU and low-bandwidth interconnect.
Revised: We train a transformer-based masked language model [82, 18, 56] using a ﬂeet of 16 low-cost preemptible workers with T4 GPU and low-bandwidth interconnect.

Original: For this task, we use the compute-efﬁcient adversarial training regimen proposed for transformers by [91, 54].
Revised: For this task, we use the compute-efﬁcient adversarial training regimen proposed for transformers by [91, 54].

Original: Formally, the adversarial formulation of the problem is the min-max problem
Revised: Formally, the adversarial formulation of the problem is the min-max problem

Original: [Equation]
Revised: [Equation]

Original: where w are the weights of the model, { ( x n , y n ) }
Revised: where w are the weights of the model, { ( x n , y n ) }

Original: Nn =1 are pairs of the training data, ⇢ is the socalled adversarial noise which introduces a perturbation in the data, and  are the regularization parameters.
Revised: Nn =1 are pairs of the training data, ⇢ is the socalled adversarial noise which introduces a perturbation in the data, and  are the regularization parameters.

Original: To make our setup more realistic, we train ALBERT-large with layer sharing [47], which was recently shown to be much more communication-efﬁcient during training [74, 20].
Revised: To make our setup more realistic, we train ALBERT-large with layer sharing [47], which was recently shown to be much more communication-efﬁcient during training [74, 20].

Original: We train our model on a combination of Bookcorpus and Wikipedia datasets with the same optimizer (LAMB) and parameters as in the original paper [47], use the adversarial training conﬁguration of [91], and follow system design considerations for preemptible instances [74].
Revised: We train our model on a combination of Bookcorpus and Wikipedia datasets with the same optimizer (LAMB) and parameters as in the original paper [47], use the adversarial training conﬁguration of [91], and follow system design considerations for preemptible instances [74].

Original: In LAMB optimizer we change the original positive momentum to negative momentum, as in MASHA .
Revised: In LAMB optimizer we change the original positive momentum to negative momentum, as in MASHA .

Original: This means that we do not exactly use MASHA in these experiments, but a combination of MASHA and LAMB.
Revised: This means that we do not exactly use MASHA in these experiments, but a combination of MASHA and LAMB.

Original: In fact this approach is typical, e.g., in papers [15, 26, 58, 13, 49], the theoretical methods are combined with
Revised: In fact this approach is typical, e.g., in papers [15, 26, 58, 13, 49], the theoretical methods are combined with

Original: Adam.
Revised: Adam.

Original: In terms of communication, we consider 4 different setups for gradient compression: the “baseline” strategy with uncompressed gradients, full 8-bit quantization [17, 50], mixed 8-bit quantization, and
Revised: In terms of communication, we consider 4 different setups for gradient compression: the “baseline” strategy with uncompressed gradients, full 8-bit quantization [17, 50], mixed 8-bit quantization, and

Original: Power compression [84] with rank r =8 .
Revised: Power compression [84] with rank r =8 .

Original: For mixed 8-bit quantization and Power we only apply compression to gradient tensors with more than 2 16 elements, sending smaller ones uncompressed.
Revised: For mixed 8-bit quantization and Power we only apply compression to gradient tensors with more than 2 16 elements, sending smaller ones uncompressed.

Original: These small tensors represent layer biases and LayerNorm scales [6] that collectively amount to  1% of the total gradient, but can be more difﬁcult to compress than regular weight tensors.
Revised: These small tensors represent layer biases and LayerNorm scales [6] that collectively amount to  1% of the total gradient, but can be more difﬁcult to compress than regular weight tensors.

Original: Finally, since Power is a biased compression algorithm, we use error feedback [44, 72] with a modiﬁed formulation proposed by [84].
Revised: Finally, since Power is a biased compression algorithm, we use error feedback [44, 72] with a modiﬁed formulation proposed by [84].

Original: For all experimental setups, we report learning curves in terms of the model training objective, similarly to [24, 74].
Revised: For all experimental setups, we report learning curves in terms of the model training objective, similarly to [24, 74].

Original: To quantify the differences in training loss better, we also evaluate the downstream performance for each model on several popular tasks from [85] after each model was trained on approximately 80 billion tokens.
Revised: To quantify the differences in training loss better, we also evaluate the downstream performance for each model on several popular tasks from [85] after each model was trained on approximately 80 billion tokens.

Original: Finally, we measure the communication efﬁciency of each proposed strategy by measuring the average wall time per communication round when all 16 workers are active.
Revised: Finally, we measure the communication efﬁciency of each proposed strategy by measuring the average wall time per communication round when all 16 workers are active.

Original: [Table]
Revised: [Table]

Original: The learning curves in
Revised: The learning curves in

Original: Figure 2 (upper) follow a predictable pattern, with more extreme compression techniques demonstrating slower per-iteration convergence.
Revised: Figure 2 (upper) follow a predictable pattern, with more extreme compression techniques demonstrating slower per-iteration convergence.

Original: One curious exception to that is full 8-bit quantization, which was unable to achieve competitive training loss.
Revised: One curious exception to that is full 8-bit quantization, which was unable to achieve competitive training loss.

Original: The remaining three setups converge to similar loss values below 2.
Revised: The remaining three setups converge to similar loss values below 2.

Original: Both the baseline and mixed 8-bit compression show similar values in terms of downstream performance, with Power compression showing mild degradation.
Revised: Both the baseline and mixed 8-bit compression show similar values in terms of downstream performance, with Power compression showing mild degradation.

Original: But in terms of information transfer time, methods using compression (especially
Revised: But in terms of information transfer time, methods using compression (especially

Original: Power) are signiﬁcantly superior to the method without compression.
Revised: Power) are signiﬁcantly superior to the method without compression.

Original: This makes it possible to use such techniques to increase the training time without sacriﬁcing quality.
Revised: This makes it possible to use such techniques to increase the training time without sacriﬁcing quality.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper we present algorithms with unbiased and contractive compressions for solving distributed VIs and SPPs.
Revised: In this paper we present algorithms with unbiased and contractive compressions for solving distributed VIs and SPPs.

Original: Our algorithms are presented in deterministic, stochastic and federated versions.
Revised: Our algorithms are presented in deterministic, stochastic and federated versions.

Original: All basic algorithms and their modiﬁcations support bidirectional compression.
Revised: All basic algorithms and their modiﬁcations support bidirectional compression.

Original: Experiments conﬁrm the efﬁciency of both our algorithms and the use of compression for solving large-scale VIs in general.
Revised: Experiments conﬁrm the efﬁciency of both our algorithms and the use of compression for solving large-scale VIs in general.

Original: In future works it is important to address the issue of the necessity to forward uncompressed information in some iterations.
Revised: In future works it is important to address the issue of the necessity to forward uncompressed information in some iterations.

Original: Although full packages are rarely transmitted, this is a slight limitation of our approach.
Revised: Although full packages are rarely transmitted, this is a slight limitation of our approach.

Original: Lower bounds for compression methods are also an interesting area of research.
Revised: Lower bounds for compression methods are also an interesting area of research.

Original: At the moment there are neither such results for VIs and SPPs, nor for minimizations.
Revised: At the moment there are neither such results for VIs and SPPs, nor for minimizations.

Original: In Appendix
Revised: In Appendix

Original: B we only hypothesize the optimality of our methods and back it up with analogies, provable lower estimates could complete the story with compressed methods.
Revised: B we only hypothesize the optimality of our methods and back it up with analogies, provable lower estimates could complete the story with compressed methods.

Original: Acknowledgments
Revised: Acknowledgments

Original: This research of A. Beznosikov has been supported by The Analytical Center for the Government of the Russian Federation (Agreement No.
Revised: This research of A. Beznosikov has been supported by The Analytical Center for the Government of the Russian Federation (Agreement No.

Original: 70-2021-00143 dd.
Revised: 70-2021-00143 dd.

Original: 01.11.2021, IGK 000000D730321P5Q
Revised: 01.11.2021, IGK 000000D730321P5Q