Clipped Stochastic Methods for Variational Inequalities with Heavy-Tailed Noise
Abstract
Stochastic first-order methods such as Stochastic Extragradient ( SEG ) or Stochastic
Gradient Descent-Ascent ( SGDA ) for solving smooth minimax problems and, more generally, variational inequality problems (VIP) have been gaining a lot of attention in recent years due to the growing popularity of adversarial formulations in machine learning.
However, while high-probability convergence bounds are known to reflect the actual behavior of stochastic methods more accurately, most convergence results are provided in expectation.
Moreover, the only known highprobability complexity results have been derived under restrictive sub-Gaussian (light-tailed) noise and bounded domain assumption [Juditsky et al., 2011a].
In this work, we prove the first high-probability complexity results with logarithmic dependence on the confidence level for stochastic methods for solving monotone and structured non-monotone VIPs with non-sub-Gaussian (heavy-tailed) noise and unbounded domains.
In the monotone case, our results match the best-known ones in the light-tails case [Juditsky et al., 2011a], and are novel for structured non-monotone problems such as negative comonotone, quasi-strongly monotone, and/or star-cocoercive ones.
We achieve these results by studying SEG and SGDA with clipping.
In addition, we numerically validate that the gradient noise of many practical GAN formulations is heavy-tailed and show that clipping improves the performance of SEG / SGDA .
Introduction
Recently, game formulations have been receiving a lot of interest from the optimization and machine learning communities.
In such problems, different models/players competitively minimize their loss functions, e.g., see adversarial example games [Bose et al., 2020], hierarchical reinforcement learning [Wayne and Abbott, 2014, Vezhnevets et al., 2017], and generative adversarial networks (GANs)
[Goodfellow et al., 2014].
Very often, such problems are studied through the lens of solving a variational inequality problem (VIP)
[Harker and Pang, 1990, Ryu and Yin, 2021, Gidel et al.,
2019a].
In the unconstrained case, for an operator 3 F : R d → R d VIP can be written as follows: find x ∗ ∈
X ∗ where
X
∗
:= {
x ∗ ∈ R
d
such that F x
∗ )
= 0 } .
(VIP)
In machine learning applications, operator F usually has an expectation form, i.e., F ( x ) =
E ξ [ F ξ ( x )] , where x corresponds to the parameters of the model, ξ is a sample from some (possibly unknown) distribution D , and F ξ ( x ) is the operator corresponding to the sample ξ .
Such problems are typically solved via first-order stochastic methods such as Stochastic Extragradient ( SEG ), also known as Mirror-Prox algorithm
[Juditsky et al., 2011a], or Stochastic Gradient DescentAscent ( SGDA )
[Dem’yanov and Pevnyi, 1972, Nemirovski et al., 2009] due to their practical efficiency.
However, despite the significant attention to these methods and their modifications, their convergence is usually analyzed in expectation only, e.g., see [Gidel et al., 2019a, Hsieh et al., 2019, 2020, Mishchenko et al., 2020, Loizou et al., 2021].
In contrast, while high-probability analysis more accurately reflects the behavior of the stochastic methods [Gorbunov et al., 2020], a little is known about it in the context of solving VIP.
To the best of our knowledge, there is only one work addressing this question for monotone variational inequalities [Juditsky et al., 2011a].
However, [Juditsky et al., 2011a] derive their results under the assumption that the problem has “light-tailed” (sub-Gaussian) noise and bounded domain, which is restrictive even for simple classes of minimization problems [Zhang et al., 2020b].
This leads us to the following open question.
Q1:
Is it possible to achieve the same high-probability results as in [Juditsky et al., 2011a] without assuming that the noise is sub-Gaussian and the domain is bounded?
Next, in the context of GANs’ training, empirical investigation [Jelassi et al., 2022] and practical use [Gulrajani et al., 2017, Miyato et al., 2018, Tran et al., 2019, Brock et al., 2019, Sauer et al., 2022] indicate the practical superiority of Adam -based methods, e.g., alternating SGDA with stochastic estimators from Adam [Kingma and Ba, 2014], over classical methods like SEG or (alternating)
SGDA .
This interesting phenomenon has no rigorous theoretical explanation yet.
In contrast, there exists a partial understanding of why Adam -like methods perform well in different tasks such as training attention models [Zhang et al., 2020b].
In particular, Zhang et al.
[2020b] empirically observe that stochastic gradient noise is heavy-tailed in several NLP tasks and theoretically shows that vanilla SGD [Robbins and Monro, 1951] can diverge in such cases and its version with gradient clipping ( clippedSGD )
[Pascanu et al., 2013] converges.
Moreover, the state-of-the-art high-probability convergence results for heavy-tailed stochastic minimization problems are also obtained for the methods with gradient clipping [Nazin et al., 2019, Gorbunov et al., 2020, 2021, Cutkosky and Mehta, 2021].
Since
Adam can be seen as a version of adaptive clipped-SGD with momentum [Zhang et al., 2020b], these advances establish the connection between good performance of Adam , heavy-tailed gradient noise, and gradient clipping for minimization problems .
Motivated by these recent advances in understanding the superiority of Adam for minimization problems, we formulate the following research question
Q2:
In the training of popular GANs, does the gradient noise have heavy-tailed distribution and does gradient clipping improve the performance of the classical SEG / SGDA ?
In this paper, we give positive answers to Q1 and Q2 .
In particular, we derive high-probability results for clipped-SEG and clipped-SDGA for monotone and structured non-monotone VIPs with non-sub-Gaussian noise, validate that the gradient noise in several GANs formulations is indeed heavy-tailed, and show that gradient clipping does significantly improve the results of SEG / SGDA in these tasks.
That is, our work closes a noticeable gap in theory of stochastic methods for solving VIPs.
1.1 Technical Preliminaries
Before we summarize our main contributions, we introduce some notations and technical assumptions.
Notation.
Throughout the text ⟨ x, y ⟩ is the standard inner-product, ∥ x ∥ =
(cid:112) ⟨ x, x ⟩ denotes ℓ 2 -norm,
B r ( x ) = { u ∈ R d | ∥ u − x ∥ ≤ r } .
E [ X ] and E ξ [ X ] are full and conditional expectations w.r.t.
the randomness coming from ξ of random variable X .
P { E } denotes the probability of event E .
R 0 =
∥ x 0 − x ∗ ∥ denotes the distance between the starting point x 0 of a method and the solution x ∗ of VIP.
High-probability convergence for VIP.
For deterministic VIPs there exist several convergence metrics P ( x ) .
These metrics include restricted gap-function [Nesterov, 2007]
Gap R ( x ) := max y ∈ B R ( x ∗ ) ⟨ F ( y ) , x − y ⟩ , (averaged) squared norm of the operator ∥ F ( x ) ∥ 2 , and squared distance to the solution ∥ x − x ∗ ∥ 2 .
Depending on the assumptions on the problem, one or another criterion is preferable.
For example, for monotone and strongly monotone problems Gap R ( x ) and ∥ x − x ∗ ∥ 2 are valid metrics of convergence, while in the non-monotone case, one typically has to use ∥ F ( x ) ∥ 2 .
In the stochastic case, one needs either to upper bound E
[ P ( x )] or to derive a bound on P ( x ) that holds with some probability.
In this work, we focus on the second type of bounds.
That is, for a given confidence level β ∈ (0 , 1] , we aim at deriving bounds on P ( x ) that hold with probability at least 1 − β , where x is produced by clipped-SEG / clipped-SGDA .
However, to achieve this goal one has to introduce some assumptions on the stochastic noise such as bounded variance assumption, which is standard in the stochastic optimization literature [Ghadimi and Lan, 2012, 2013, Juditsky et al., 2011b, Nemirovski et al., 2009].
We rely on a weaker version of this assumption.
Assumption 1.1 (Bounded Variance) .
There exists a bounded set Q ⊆ R d and σ ≥ 0 such that
[Equation]
In contrast to most of the existing works assuming (1) on the whole space/domain, we need (1) to hold only on some bounded set Q .
More precisely, in the analysis, we rely on (1) only on some ball
B r ( x ∗ ) around the solution x ∗ and radius r ∼ R 0 .
Although we consider an unconstrained VIP, we manage to show that the iterates of clipped-SEG / clipped-SGDA stay inside this ball with high probability.
Therefore, for achieving our purposes, it is sufficient to make all the assumptions on the problem only in some ball around the solution.
We also notice that the majority of existing works providing high-probability analysis with logarithmic dependence 5 on 1 / β rely on the so-called light tails assumption: E [exp( ∥ F ξ ( x ) − F ( x ) ∥ 2 / σ 2 )] ≤ exp(1)
meaning that the noise has a sub-Gaussian distribution.
This assumption always implies (1) but not vice versa.
However, even for minimization problems, there are only few works that do not rely on the light tails assumption [Nazin et al., 2019, Davis et al., 2021, Gorbunov et al., 2020, 2021,
Cutkosky and Mehta, 2021].
In the context of solving VIPs, the existing high-probability guarantees in [Juditsky et al., 2011a] rely on the light tails assumption.
Optimization properties.
We also need to introduce several assumptions about the operator F .
We start with the standard Lipschitzness.
As we write above, all assumptions are introduced only on some bounded set Q , which will be specified later.
Assumption 1.2 (Lipschitzness
[Equation]
[Equation] Next, we need to introduce some assumptions 6 on the monotonicity of F , since approximating local first-order optimal solutions is intractable in the general non-monotone case [Daskalakis et al., 2021,
Diakonikolas et al., 2021].
We start with the standard monotonicity assumption and its relaxations.
Assumption 1.3 (Monotonicity) .
Operator F : R d → R d is monotone on Q ⊆ R d , i.e.,
[Equation]
Assumption 1.4 (Star-Negative Comonotonicity) .
Operator F : R d → R d is ρ -star-negatively comonotone on Q ⊆ R d for some ρ ∈ [0 , + ∞ ) , i.e., for any x ∗ we have
[Equation]
When ρ = 0 , the operator F is called star-monotone ( SM ) on Q .
(SNC) is also known as weak Minty condition [Diakonikolas et al., 2021], a relaxation of negative comonotonicity [Bauschke et al., 2021].
Another name of star-monotonicity (SM) is variational stability condition
[Hsieh et al., 2020].
The following assumption is a relaxation of strong monotonicity.
[Table]
[Table] [Table] [Table]
[Table]
Assumption 1.5 (Quasi-Strong Monotonicity) .
Operator F : R d → R d is µ -quasi strongly monotone on Q ⊆ R d for some µ ≥ 0 , i.e.,
[Equation]
where x ∗ is the unique solution of (VIP) .
(QSM)
Under this name, the above assumption is introduced by [Loizou et al., 2021], but it is also known as strong coherent Song et al.
[2020] and strong stability [Mertikopoulos and Zhou, 2019] conditions.
Moreover, unlike strong monotonicity that always implies monotonicity, (QSM) can hold for nonmonotone operators [Loizou et al., 2021, Appendix A.6].
However, in contrast to (Mon), (QSM) allows to achieve linear convergence 7 of deterministic methods for solving VIP.
Finally, we consider a relaxation of standard cocoercivity: ℓ ⟨ F ( x ) − F ( y ) , x − y ⟩ ≥
∥
F ( x ) − F ( y ) ∥ 2 .
Assumption 1.6 (Star-Cocoercivity) .
Operator F : R d → R d is ℓ -star-cocoercive on Q ⊆ R d for some ℓ > 0 , i.e.,
[Equation]
where x ∗ is the projection of x on X ∗ .
[Equation]
This assumption is introduced by [Loizou et al., 2021].
One can construct an operator F being star-cocoercive, but not cocoercive [Gorbunov et al., 2022b].
Moreover, although cocoercivity implies monotonicity and Lipschitzness, there exist operators satisfying (SC), but neither (Mon) nor (Lip)
[Loizou et al., 2021, §A.6].
We summarize the relations between the assumptions in Fig.
1.2 Our Contributions
⋄ New high-probability results for VIPs with heavy-tailed noise.
In our work, we circumvent the limitations of the existing high-probability analysis of stochastic methods for solving VIPs [Juditsky et al., 2011a] and derive the first high-probability results for the methods that solve monotone VIPs with heavy-tailed noise in the unconstrained case.
The key algorithmic ingredient helping us to achieve these results is a proper modification of SEG and SGDA based on the gradient clipping and leading to our clipped-SEG and clipped-SGDA .
Moreover, we derive several high-probability results for clipped-SEG / clipped-SGDA applied to solve structured non-monotone VIPs.
To the best of our knowledge, these results do not have analogs even under the light tails assumption.
We summarize the derived complexity results in Tbl. 1.
⋄ Tight analysis.
The derived complexities satisfy a desirable for high-probability results property: they have logarithmic dependence on 1 / β , where β is a confidence level.
Next, up to the logarithmic factors, our results match known lower bounds 8 in monotone and strongly monotone cases [Beznosikov et al., 2020].
Moreover, we recover and even improve the results from [Juditsky et al., 2011a], which are obtained in the light-tailed case, since our bounds do not depend on the diameter of the domain, see Tbl.
1 for the details.
⋄ Weak assumptions.
One of the key features of our theoretical results is that it relies on assumptions restricted to a ball around the solution .
We achieve this via showing that, with high probability, clipped-SEG / clipped-SGDA do not leave a ball (with a radius proportional to R 0 ) around the solution.
In contrast, the existing works on stochastic methods for solving VIPs usually make assumptions such as boundedness of the variance and Lipschitzness on the whole domain of the considered problem.
Since many practical tasks are naturally unconstrained, such assumptions become too unrealistic since, e.g., stochastic gradients and their variance in the training of neural networks with more than 2 layers grow polynomially fast when x goes to infinity.
However, for a large class of problems including the ones with polynomially growing operators, boundedness of the variance and Lipschitzness hold on any compact set .
That is, our analysis covers a broad class of problems.
⋄ Numerical experiments.
We empirically observe that heavy-tailed gradient noise arises in the training of several practical GANs formulations including StyleGAN2 and WGAN-GP.
Moreover, our experiments show that gradient clipping significantly improves the convergence of SEG / SGDA on such tasks.
These results shed a light on why Adam -based methods are good at training GANs.
Our codes are publicly available: https://github.com/busycalibrating/ clipped-stochastic-methods .
1.3 Closely Related Work
In this section we discuss the most closely related works.
Further discussion is deferred to § A.
High probability convergence.
To the best of our knowledge, the only work deriving highprobability convergence in the context of solving VIPs is [Juditsky et al., 2011a].
In particular,
Juditsky et al.
[2011a] consider monotone and Lipschitz VIP defined 9 on a convex compact set X with the diameter D :=
max x,y ∈X ∥
x − y ∥ , and assume that the noise in F ξ ( x ) is light-tailed.
In this setting, Juditsky et al.
[2011a] propose a stochastic version of the celebrated Extragradient method ( EG )
[Korpelevich, 1976] with non-Euclidean proximal setup – Mirror-Prox .
Moreover, Juditsky et al.
[2011a] derive that after K
= O (max { LD 2 ε , σ 2 D 2 ε 2 ln
2 1 β } ) stochastic oracle calls for some ε > 0 , β ∈ (0 , 1] , the averaged extrapolated iterate (cid:101) x K avg =
[Equation]
(see also clipped-SEG ) satisfies
Gap D ( (cid:101) x K avg ) ≤ ε with probability at least 1 − β .
Although this result has a desirable logarithmic dependence on 1 / β and optimal dependence on ε [Beznosikov et al., 2020], it is derived only for (i) light-tailed case and (ii) bounded domains.
Our results do not have such limitations.
Clipped Stochastic Extragradient
In this section, we consider a version of SEG with gradient clipping.
That is, we apply clipping operator clip ( y, λ ) :
= min { 1 , λ / ∥
y ∥ } y , which is defined for any y ∈ R d (when y =
we set clip (0 , λ ) := 0 ) and any clipping level λ > 0 , to the mini-batched stochastic estimators used both at the extrapolation and update steps of SEG .
This results in the following iterative algorithm:
[Equation]
where
[Equation]
clipped-
[Equation]
where { ξ i,k 1 } m 1 ,k i =1 , { ξ i,k 2 } m 2 ,k i =1 are independent samples from the distribution D .
In the considered algorithm, clipping bounds the effect of heavy-tailedness of the gradient noise, but also creates a bias
Deriving tight
In this case, the goal is to find x ∗ ∈
X such that inequality ⟨ F ( x ∗ ) , x − x ∗ ⟩ ≥ 0 holds for all x ∈ X .
that one has to properly control in the analysis.
Moreover, we allow different stepsizes γ 1 , γ 2 [Hsieh et al., 2020, Diakonikolas et al., 2021, Gorbunov et al., 2022a], different batchsizes m 1 ,k , m 2 ,k , and different clipping levels λ 1 ,k , λ 2 ,k .
In particular, taking γ 2 < γ 1 is crucial for our analysis to handle VIP satisfying star-negative comonotonicity (SNC) with ρ > 0 .
Our convergence results for clipped-SEG are summarized in the following theorem.
For simplicity, we omit here the numerical constants, which are explicitly given in § C.
Theorem 2.1.
Consider clipped-SEG run for K ≥ 0 iterations.
Let
R ≥ R 0 =
∥ x 0 − x ∗ ∥ .
Case 1.
Let Assump.
1.1, 1.2, 1.3 hold for Q = B 4 R ( x ∗ ) , where γ 1 = γ 2 = γ with 0 < γ =
O ( 1 / ( LA ) ) , λ 1 ,k
= λ 2 ,k ≡ λ =
Θ( R / ( γA ) ) , m 1 ,k
= m 2 ,k ≡ m = Ω(max { 1 , ( K +1) γ 2 σ 2 A / R 2 } ) , where A = ln 6( K +1) β , β ∈ (0 , 1] are such that A ≥ 1 .
Case 2.
Let Assump.
1.1, 1.2, 1.4 hold for Q = B 3 R ( x ∗ ) , where γ 2 + 2 ρ ≤ γ 1 ≤
O ( 1 / ( LA ) ) , λ 1 ,k ≡ λ 1 =
Θ( R / ( γ 1 A ) ) , λ 2 ,k ≡ λ 2 =
Θ( R / ( γ 2 A ) ) , m 1 ,k ≡ m 1 = Ω(max { 1 , max { γ 1 γ 2 ( K +1) , √ γ 31 γ
( K +1) } σ 2 A / R 2 } ) , m 2 ,k ≡ m 2 = Ω(max { 1 , ( K +1) γ 22 σ 2 A / R 2 } ) ,
where A = ln 8( K +1) β , β ∈ (0 , 1] are such that A ≥ 1 .
Case 3.
Let Assump.
1.1, 1.2, 1.5 hold for Q = B 3 R ( x ∗ ) , where γ 1 = γ 2 = γ with 0 < γ ≤ O ( 1 / ( LA ) ) , λ 1 ,k = λ 2 ,k = λ k
= Θ( exp( − γµ (1+ k / 2 )) R / ( γA ) ) , m 1 ,k
= m 2 ,k
= m k = Ω(max { 1 , ( K +1) γ 2 σ 2 A / (exp( − γµk )
R 2 ) } ) , where A = ln 6( K +1) β , β ∈ (0 , 1] are such that A ≥ 1 .
Then, to guarantee Gap R ( (cid:101)
x K avg ) ≤ ε in Case 1 with (cid:101) x
K avg = 1 K +1 (cid:80)
Kk =0 (cid:101) x k , 1 K +1 (cid:80)
Kk =
∥ F ( x k ) ∥ 2 ≤
Lε in Case 2 , ∥ x K − x ∗ ∥ 2 ≤ ε in Case 3 , with probability ≥ 1 − β clipped-SEG requires
Case 1 and 2 : (cid:101) O
[Equation]
max (cid:110)
[Equation]
[Equation] [Equation]
[Equation]
2 (cid:111)(cid:17) and
Case 3 : (cid:101) O (cid:16) max (cid:110)
[Equation]
µε (cid:111)(cid:17) (2) oracle calls.
The above guarantees hold in two different regimes: large step-sizes γ ≈ 1 / LA (requiring large batch-sizes), and small step-sizes, allowing small batch-sizes m =
O (1) .
Proof sketch in Case 1 .
Modifying the analysis of EG , we first derive that for all t ≥
we have
Gap R ( (cid:101) x t avg ) and ∥ x t − x ∗ ∥ 2 are not greater than max u ∈ B R ( x ∗ ) {∥ x 0 − u ∥ 2 + 2 γ (cid:80) t − 1 l =0 ⟨ x l − u − γF ( (cid:101) x l ) , θ l ⟩ + γ 2 (cid:80) t − 1 l =0 ( ∥ θ l ∥ 2 + 2 ∥ ω l ∥ 2 ) } if x l , (cid:101) x
l lie in B 4 R ( x ∗ ) for all l = 0 , 1 , . .
, t − 1 , where θ l = F ( (cid:101) x l ) − (cid:101) F ξ l 2 ( (cid:101) x l ) and ω l = F ( x l ) − (cid:101)
F ξ l 1 ( x l ) .
Next, using this recursion and the induction argument, we show that with high probability x t , (cid:101) x t ∈ B 4 R ( x ∗ ) for all t = 0 , 1 , . . .
, K + 1 .
This gives us an upper bound for Gap R ( (cid:101) x t avg ) .
After that, we upper bound max u ∈ B R ( x ∗ ) { 2 γ (cid:80) t − 1 l =0 ⟨ x l − u − γF ( (cid:101) x l ) , θ l ⟩ + γ 2 (cid:80) t − 1 l =0 ( ∥ θ l ∥ 2 + 2 ∥ ω l ∥ 2 ) } by 5 R 2 with high-probability.
We achieve this via the proper choice of the clipping level λ =
Θ( R / ( γA ) ) implying that ∥ F ( (cid:101) x l ) ∥ ≤ λ / 2 and ∥ F ( x l ) ∥ ≤ λ /with high probability for all l = 0 , 1 , . .
, t − 1 .
This clipping politics helps to properly bound the bias and the variance of the clipped estimators using Lem.
B.2.
After that, it remains to apply the
Bernstein inequality for the martingale differences (Lem. B.1).
See the detailed proof in § C.
In addition to the discussion given in the introduction (see § 1.2, 1.3 and Tbl. 1), we provide here several important details about the derived results.
First of all, we notice that up to the logarithmic factors depending on β our high-probability convergence results recover the state-of-the-art inexpectation ones for SEG in the monotone [Beznosikov et al., 2020], star-negative comonotone [Diakonikolas et al., 2021], and quasi-strongly monotone [Gorbunov et al., 2022a] cases.
Moreover, as we show in Corollaries C.1 and C.3, to achieve these results in monotone and quasi-strongly monotone cases, it is sufficient to choose constant batchsize m =
O (1) and small enough stepsize γ .
In contrast, when the operator is star-negatively comonotone, we do rely on the usage of large stepsize
γ 1 and large batchsize m
=
O ( K ) for the extrapolation step to obtain (2).
However, known in-expectation results from [Diakonikolas et al., 2021, Lee and Kim, 2021] also rely on large O ( K ) batchsizes in this case.
We leave the investigation of this limitation to the future work.
Clipped Stochastic Gradient Descent-Ascent
In this section, we extend the approach described above to the analysis of SGDA with clipping.
That is, we consider the following algorithm:
[Equation]
x k ) , where (cid:101) F
[Equation]
( x
[Equation]
clip
[Equation]
where { ξ i,k } mi =1 are independent samples from the distribution D .
In the above method, clipping plays a similar role as in clipped-SEG .
Our convergence results for clipped-SGDA are summarized below.
The general idea of the proof is similar to the one for clipped-SEG , see the details in
Appendix D, where we also explicitly give the constants that are omitted here for simplicity.
Theorem 3.1.
Consider clipped-SGDA run for K ≥ 0 iterations.
Let
R ≥ R 0 =
∥ x 0 − x ∗ ∥ .
[-Kk =
-]
Case 1.
Let Assump.
1.1, 1.3, 1.6 hold for Q = B 3 R ( x ∗ ) , where 0 <
γ =
[-Moscow Institute of Physics and Technology dated November 1, 
-]
[-= 
-]
O ( 1 / ( ℓA ) ) , λ k ≡ λ =
Θ( R / ( γA ) ) , m k ≡ m = Ω(max { 1 , ( K +1)
γ 2 σ 2 A / R 2 } ) , where β ∈ (0 , 1] and [-12 γ 1 = γ 
-]A =
[-, K + 
-]
[-For k = 
-]
[-is trivial since ∥ x 0 − u ∥ 2 ≤ 2 ∥ x 0 − x ∗ ∥ 2 + 
-]
ln 6( K +1) β ≥ 1 .
[-Applying the Bernstein’s inequality (Lemma B.1) with X l = 2 γ 2 (cid:16) ∥ θ ul ∥ 2 + 
-]
[-Upper bound for 2 γR (cid:13)(cid:13) (cid:13) (cid:80) T − 1 l =
-]
Case 2.
Let Assump.
[-, i.e., for k = K + 
-]
1.1, 1.6 hold for Q = B 2 R ( x ∗ ) , where 0 <
[-F ξ k 1 ( x k ) ∥ 2 ≥ 0 and, using (6) with α = 1 , we derive ∥ x k +1 − x ∗ ∥≤ ∥ x k − x ∗ ∥ 2 − γ 1 γ 2 2 (cid:0) 1 − 
-]
γ =
[-, K + 
-]
[-For k = 
-]
O ( 1 / ( ℓA ) ) , λ k ≡ λ =
[-R ≥ R 0 def = ∥ x 0 − x ∗ ∥ , and γ 1 = γ 2 = γ , 0 < γ ≤ 1 / 
-]
[-where in the last step we apply 0 < γ ≤ 1 / 
-]
[-, K + 
-]
Θ( R / ( γA ) ) , m k ≡ m = Ω(max { 1 , ( K +1)
[-For k = 
-]
γ 2 σ 2 A / R 2 } ) , where β ∈ (0 , 1] and A =
ln 4( K +1) β ≥ 1 .
Case 3.
Let Assump.
[-Kk =
-]
1.1, 1.5, 1.6 hold for Q = B 2 R ( x ∗ ) , where 0 <
[-, K + 
-]
[-For k = 
-]
γ =
O ( 1 / ( ℓA ) ) , λ k
=
Θ
( e − γµ (1+ k/ 2) R ( γA ) ) , m k = Ω(max { 1 , ( K +1) γ 2 σ 2 A e γµk R} ) , where β ∈ (0 , 1] and A =
ln 4( K +1) β ≥ 1 .
Then, to guarantee Gap R ( (cid:101)
[-, K + 
-]
x K avg ) ≤ ε in Case 1 with (cid:101) x
K avg = 1 K +1 (cid:80)
[-Kk =
-]
Kk =0 (cid:101) x k , 1 K +1 (cid:80)
{+Kk =
+}
∥ F ( x k ) ∥ 2 ≤ ℓε in Case 2 , ∥ x K − x ∗ ∥ 2 ≤ ε in Case 3 , with probability ≥ 1 − β , clipped-SGDA requires
Case 1 and 2 :
[-, K + 
-]
[-For k = 
-]
[Equation]
and
Case 3 :
[Equation]
oracle calls.
The above guarantees hold in two different regimes: large step-sizes γ ≈ 1 / ℓA (requiring large batch-sizes), and small step-sizes, allowing small batch-sizes m =
O (1) .
As for the clipped-SEG , up to the logarithmic factors depending on β , our high-probability convergence results from the theorem above recover the state-of-the-art in-expectation ones for SGDA under monotonicity and star-cocoercivity [Beznosikov et al., 2022], star-cocoercivity [Gorbunov et al., 2022a] 10 , quasi-strong monotonicity and star-cocoercivity [Loizou et al., 2021] assumptions.
Moreover, in all the cases , one can achieve the results from (3) using constant batchsize m =
O (1) and small enough stepsize γ (see Corollaries D.1, D.2, D.3 for the details).
Finally, to the best of our knowledge, we derive the first high-probability convergence results for SGDA -type methods.
Experiments
To validate our theoretical results, we conduct experiments on heavy-tailed min-max problems to demonstrate the importance of clipping when using non-adaptive methods such as SGDA or SEG .
We train a Wasserstein GAN with gradient penalty [Gulrajani et al., 2017] on CIFAR-10 [Krizhevsky et al., 2009] using SGDA , clipped-SGDA , and clipped-SEG , and show the evolution of the gradient noise histograms during training.
We demonstrate that gradient clipping also stabilizes the training of more sophisticated GANs when using SGD by training a StyleGAN2 model [Karras et al., 2020] on
FFHQ [Karras et al., 2019], downsampled to 128 × 128 .
Although the generated sample quality is not competitive with a model trained with Adam , we find that StyleGAN2 fails to generate anything meaningful when trained with regular SGDA whereas clipped methods learn meaningful features.
[-The total number of oracle calls equals (cid:80) Kk =
-]
We do not claim to outperform state-of-the-art adaptive methods (such as Adam ); our focus is to validate our theoretical results and to demonstrate that clipping improves SGDA in this context.
WGAN-GP.
In this section, we focus on the ResNet architecture proposed in Gulrajani et al.
[2017], and we adapt our code from a publicly available WGAN-GP implementation.
11 We first compute the gradient noise distribution and validate if it is heavy-tailed. Taking the fixed randomly initialized [Figure] [Figure]
[Figure]
D (
[Figure]
Hyperparameters weights, we iterate through 1000 steps (without parameter updates) to compute the noise norm for each minibatch (sample with replacement as in normal GAN training).
We show the distributions for the generator and discriminator in Fig.
2 and also compute p mR = F 1 . 5 ( X ) and p eR = F 3 ( X ) , where F λ ( X ) = P ( Q 3 + λ ( Q 3 − Q 1 ) <
X ) , X is the gradient noise distribution, and Q i is the i th quartile. This is a metric introduced by Jordanova and Petkova [2017] to measure how heavytailed a distribution is based on the distribution’s quantiles, where p mR and p eR quantify “mild” and “extreme” (right side) heavy tails respectively. A normal distribution should have p mR N ≈ 0 .
0035 and p eR N ≈ 1 . 2 × 10 − 6 . In all tests, we compute the ratios ρ mR = p mR /p mR N and ρ eR = p eR /p eR N and empirically find that we at least have mild heavy tails, and sometimes extremely heavy tails. We train the ResNet generator on CIFAR-10 with SGDA / SEG , and clipped-SGDA / SEG .
We use the default architectures and training parameters specified in Gulrajani et al.
[2017] ( λ GP = 10 , n dis = 5 , learning rate decayed linearly to 0 over 100k steps), with the exception of doubling the feature map of the generator. The clipped methods are implemented as standard gradient norm clipping, applied after computing the gradient penalty term for the critic. In addition to norm clipping, we also test coordinate-wise gradient clipping, which is more common in practice [Goodfellow et al., 2016b]. For all methods, we tune the learning rates and clipping threshold where applicable.
We do not use momentum following a standard practice for GAN training [Gidel et al., 2019b]. We find that clipped methods outperform regular SGDA and SEG . In addition to helping prevent exploding gradients, clipped methods achieve a better Fréchet inception distance (FID) score [Heusel et al., 2017]; the best FID obtained for clipped methods is 19 . 65 in comparison to 67 .
37 for regular SGDA , both trained for 100k steps. A summary of FIDs obtained during hyperparameter tuning is shown Fig.
2, where the best FID score obtained in the first 35k iterations is drawn for each hyperparameter configuration and optimization method. At a high level, we do a log-space sweep over [2e − 5 , 0 . 2] for the learning rates, [10 − 1 , 10] for the norm-clip parameter, and [10 − 3 , 10 − 1 ] for the coordinate clip parameter (with some exceptions) – please refer to § E for further details.
We also show the evolution of the gradient noise histograms during training for SGD and clipped-SGDA in
Fig. 4.
Note that for regular SGD , the noise distribution remains heavy tailed and does not appear to change much throughout training. In contrast, the noise histograms for clipped-SGDA (in particular the generator) seem to develop lighter tails during training.
StyleGAN2.
We extend our experiments to StyleGAN2, but limit our scope to clipped-SGDA with coordinate clipping as coordinate clipped-SGDA generally performs the best, and StyleGAN is expensive to train.
We train on FFHQ downsampled to 128 × 128 pixels, and use the recommended StyleGAN2 hyperparameter configuration for this resolution (batch size = 32 , γ = 0 . 1024 , map depth = 2 , channel multiplier = 16384 ), see further experimental details in § E. We obtain the gradient noise histograms at initialization and for the best trained clipped-SGDA from the hyperparameter sweep and once again observe heavy tails (especially in the discriminator, see Fig. 3).
We find that all regular SGDA runs fail to learn anything meaningful, with FID scores fluctuating G e n e r a t o r
D i s c
r
i m i n a t
o r
[Figure] [Figure]
[Table]
D
[Figure]
[Figure] [Figure]
[Table]
G e n e r a t o r
[Table]
D i s c
r
i m i n a t
o r
[Table] [Figure]
around 320 and only able to generate noise. In contrast, while there is a clear gap in quality when compared to what StyleGAN2 is capable of, a model trained with clipped-SGDA with appropriately set hyperparameters is able to produce images that distinctly resemble faces (see Fig. 3). Acknowledgments and Disclosure of Funding This work was partially supported by a grant for research centers in the field of artificial intelligence, provided by the Analytical Center for the Government of the Russian Federation in accordance with the subsidy agreement (agreement identifier 000000D730321P5Q0002) and the agreement with the
{+Moscow Institute of Physics and Technology dated November 1, 
 No. 70-2021-00138. The work by P. Dvurechensky was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy – The Berlin Mathematics Research+}
Center MATH+ (EXC-2046/1, project ID: 390685689)