Clipped Stochastic Methods for Variational Inequalities with Heavy-Tailed Noise
Abstract
Stochastic first-order methods such as Stochastic Extragradient ( SEG ) or Stochastic
Gradient Descent-Ascent ( SGDA ) for solving smooth minimax problems and, more generally, variational inequality problems (VIP) have been gaining a lot of attention in recent years due to the growing popularity of adversarial formulations in machine learning.
However, while high-probability convergence bounds are known to reflect the actual behavior of stochastic methods more accurately, most convergence results are provided in expectation.
Moreover, the only known highprobability complexity results have been derived under restrictive sub-Gaussian (light-tailed) noise and bounded domain assumption [Juditsky et al., 2011a].
In this work, we prove the first high-probability complexity results with logarithmic dependence on the confidence level for stochastic methods for solving monotone and structured non-monotone VIPs with non-sub-Gaussian (heavy-tailed) noise and unbounded domains.
In the monotone case, our results match the best-known ones in the light-tails case [Juditsky et al., 2011a], and are novel for structured non-monotone problems such as negative comonotone, quasi-strongly monotone, and/or star-cocoercive ones.
We achieve these results by studying SEG and SGDA with clipping.
In addition, we numerically validate that the gradient noise of many practical GAN formulations is heavy-tailed and show that clipping improves the performance of SEG / SGDA .
Introduction
Recently, game formulations have been receiving a lot of interest from the optimization and machine learning communities.
In such problems, different models/players competitively minimize their loss functions, e.g., see adversarial example games [Bose et al., 2020], hierarchical reinforcement learning [Wayne and Abbott, 2014, Vezhnevets et al., 2017], and generative adversarial networks (GANs)
[Goodfellow et al., 2014].
Very often, such problems are studied through the lens of solving a variational inequality problem (VIP)
[Harker and Pang, 1990, Ryu and Yin, 2021, Gidel et al.,
2019a].
In the unconstrained case, for an operator 3 F : R d → R d VIP can be written as follows: find x ∗ ∈
X ∗ where
X
∗
:= {
x ∗ ∈ R
d
such that F x
∗ )
= 0 } .
(VIP)
In machine learning applications, operator F usually has an expectation form, i.e., F ( x ) =
E ξ [ F ξ ( x )] , where x corresponds to the parameters of the model, ξ is a sample from some (possibly unknown) distribution D , and F ξ ( x ) is the operator corresponding to the sample ξ .
Such problems are typically solved via first-order stochastic methods such as Stochastic Extragradient ( SEG ), also known as Mirror-Prox algorithm
[Juditsky et al., 2011a], or Stochastic Gradient DescentAscent ( SGDA )
[Dem’yanov and Pevnyi, 1972, Nemirovski et al., 2009] due to their practical efficiency.
However, despite the significant attention to these methods and their modifications, their convergence is usually analyzed in expectation only, e.g., see [Gidel et al., 2019a, Hsieh et al., 2019, 2020, Mishchenko et al., 2020, Loizou et al., 2021].
In contrast, while high-probability analysis more accurately reflects the behavior of the stochastic methods [Gorbunov et al., 2020], a little is known about it in the context of solving VIP.
To the best of our knowledge, there is only one work addressing this question for monotone variational inequalities [Juditsky et al., 2011a].
However, [Juditsky et al., 2011a] derive their results under the assumption that the problem has “light-tailed” (sub-Gaussian) noise and bounded domain, which is restrictive even for simple classes of minimization problems [Zhang et al., 2020b].
This leads us to the following open question.
Q1:
Is it possible to achieve the same high-probability results as in [Juditsky et al., 2011a] without assuming that the noise is sub-Gaussian and the domain is bounded?
Next, in the context of GANs’ training, empirical investigation [Jelassi et al., 2022] and practical use [Gulrajani et al., 2017, Miyato et al., 2018, Tran et al., 2019, Brock et al., 2019, Sauer et al., 2022] indicate the practical superiority of Adam -based methods, e.g., alternating SGDA with stochastic estimators from Adam [Kingma and Ba, 2014], over classical methods like SEG or (alternating)
SGDA .
This interesting phenomenon has no rigorous theoretical explanation yet.
In contrast, there exists a partial understanding of why Adam -like methods perform well in different tasks such as training attention models [Zhang et al., 2020b].
In particular, Zhang et al.
[2020b] empirically observe that stochastic gradient noise is heavy-tailed in several NLP tasks and theoretically shows that vanilla SGD [Robbins and Monro, 1951] can diverge in such cases and its version with gradient clipping ( clippedSGD )
[Pascanu et al., 2013] converges.
Moreover, the state-of-the-art high-probability convergence results for heavy-tailed stochastic minimization problems are also obtained for the methods with gradient clipping [Nazin et al., 2019, Gorbunov et al., 2020, 2021, Cutkosky and Mehta, 2021].
Since
Adam can be seen as a version of adaptive clipped-SGD with momentum [Zhang et al., 2020b], these advances establish the connection between good performance of Adam , heavy-tailed gradient noise, and gradient clipping for minimization problems .
Motivated by these recent advances in understanding the superiority of Adam for minimization problems, we formulate the following research question
Q2:
In the training of popular GANs, does the gradient noise have heavy-tailed distribution and does gradient clipping improve the performance of the classical SEG / SGDA ?
In this paper, we give positive answers to Q1 and Q2 .
In particular, we derive high-probability results for clipped-SEG and clipped-SDGA for monotone and structured non-monotone VIPs with non-sub-Gaussian noise, validate that the gradient noise in several GANs formulations is indeed heavy-tailed, and show that gradient clipping does significantly improve the results of SEG / SGDA in these tasks.
That is, our work closes a noticeable gap in theory of stochastic methods for solving VIPs.
1.1 Technical Preliminaries
Before we summarize our main contributions, we introduce some notations and technical assumptions.
Notation.
Throughout the text ⟨ x, y ⟩ is the standard inner-product, ∥ x ∥ =
(cid:112) ⟨ x, x ⟩ denotes ℓ 2 -norm,
B r ( x ) = { u ∈ R d | ∥ u − x ∥ ≤ r } .
E [ X ] and E ξ [ X ] are full and conditional expectations w.r.t.
the randomness coming from ξ of random variable X .
P { E } denotes the probability of event E .
R 0 =
∥ x 0 − x ∗ ∥ denotes the distance between the starting point x 0 of a method and the solution x ∗ of VIP.
High-probability convergence for VIP.
For deterministic VIPs there exist several convergence metrics P ( x ) .
These metrics include restricted gap-function [Nesterov, 2007]
Gap R ( x ) := max y ∈ B R ( x ∗ ) ⟨ F ( y ) , x − y ⟩ , (averaged) squared norm of the operator ∥ F ( x ) ∥ 2 , and squared distance to the solution ∥ x − x ∗ ∥ 2 .
Depending on the assumptions on the problem, one or another criterion is preferable.
For example, for monotone and strongly monotone problems Gap R ( x ) and ∥ x − x ∗ ∥ 2 are valid metrics of convergence, while in the non-monotone case, one typically has to use ∥ F ( x ) ∥ 2 .
In the stochastic case, one needs either to upper bound E
[ P ( x )] or to derive a bound on P ( x ) that holds with some probability.
In this work, we focus on the second type of bounds.
That is, for a given confidence level β ∈ (0 , 1] , we aim at deriving bounds on P ( x ) that hold with probability at least 1 − β , where x is produced by clipped-SEG / clipped-SGDA .
However, to achieve this goal one has to introduce some assumptions on the stochastic noise such as bounded variance assumption, which is standard in the stochastic optimization literature [Ghadimi and Lan, 2012, 2013, Juditsky et al., 2011b, Nemirovski et al., 2009].
We rely on a weaker version of this assumption.
Assumption 1.1 (Bounded Variance) .
There exists a bounded set Q ⊆ R d and σ ≥ 0 such that
[Equation]
In contrast to most of the existing works assuming (1) on the whole space/domain, we need (1) to hold only on some bounded set Q .
More precisely, in the analysis, we rely on (1) only on some ball
B r ( x ∗ ) around the solution x ∗ and radius r ∼ R 0 .
Although we consider an unconstrained VIP, we manage to show that the iterates of clipped-SEG / clipped-SGDA stay inside this ball with high probability.
Therefore, for achieving our purposes, it is sufficient to make all the assumptions on the problem only in some ball around the solution.
We also notice that the majority of existing works providing high-probability analysis with logarithmic dependence 5 on 1 / β rely on the so-called light tails assumption: E [exp( ∥ F ξ ( x ) − F ( x ) ∥ 2 / σ 2 )] ≤ exp(1)
meaning that the noise has a sub-Gaussian distribution.
This assumption always implies (1) but not vice versa.
However, even for minimization problems, there are only few works that do not rely on the light tails assumption [Nazin et al., 2019, Davis et al., 2021, Gorbunov et al., 2020, 2021,
Cutkosky and Mehta, 2021].
In the context of solving VIPs, the existing high-probability guarantees in [Juditsky et al., 2011a] rely on the light tails assumption.
Optimization properties.
We also need to introduce several assumptions about the operator F .
We start with the standard Lipschitzness.
As we write above, all assumptions are introduced only on some bounded set Q , which will be specified later.
Assumption 1.2 (Lipschitzness
[Equation] [Equation]
Next, we need to introduce some assumptions 6 on the monotonicity of F , since approximating local first-order optimal solutions is intractable in the general non-monotone case [Daskalakis et al., 2021,
Diakonikolas et al., 2021].
We start with the standard monotonicity assumption and its relaxations.
Assumption 1.3 (Monotonicity) .
Operator F : R d → R d is monotone on Q ⊆ R d , i.e.,
[Equation]
Assumption 1.4 (Star-Negative Comonotonicity) .
Operator F : R d → R d is ρ -star-negatively comonotone on Q ⊆ R d for some ρ ∈ [0 , + ∞ ) , i.e., for any x ∗ we have
[Equation]
When ρ = 0 , the operator F is called star-monotone ( SM ) on Q .
(SNC) is also known as weak Minty condition [Diakonikolas et al., 2021], a relaxation of negative comonotonicity [Bauschke et al., 2021].
Another name of star-monotonicity (SM) is variational stability condition
[Hsieh et al., 2020].
The following assumption is a relaxation of strong monotonicity.
[Table]
[Table] [Table]
[Table] [Table]
Assumption 1.5 (Quasi-Strong Monotonicity) .
Operator F : R d → R d is µ -quasi strongly monotone on Q ⊆ R d for some µ ≥ 0 , i.e.,
[Equation]
where x ∗ is the unique solution of (VIP) .
(QSM)
Under this name, the above assumption is introduced by [Loizou et al., 2021], but it is also known as strong coherent Song et al.
[2020] and strong stability [Mertikopoulos and Zhou, 2019] conditions.
Moreover, unlike strong monotonicity that always implies monotonicity, (QSM) can hold for nonmonotone operators [Loizou et al., 2021, Appendix A.6].
However, in contrast to (Mon), (QSM) allows to achieve linear convergence 7 of deterministic methods for solving VIP.
Finally, we consider a relaxation of standard cocoercivity: ℓ ⟨ F ( x ) − F ( y ) , x − y ⟩ ≥
∥
F ( x ) − F ( y ) ∥ 2 .
Assumption 1.6 (Star-Cocoercivity) .
Operator F : R d → R d is ℓ -star-cocoercive on Q ⊆ R d for some ℓ > 0 , i.e.,
[Equation]
where x ∗ is the projection of x on X ∗ .
[Equation]
This assumption is introduced by [Loizou et al., 2021].
One can construct an operator F being star-cocoercive, but not cocoercive [Gorbunov et al., 2022b].
Moreover, although cocoercivity implies monotonicity and Lipschitzness, there exist operators satisfying (SC), but neither (Mon) nor (Lip)
[Loizou et al., 2021, §A.6].
We summarize the relations between the assumptions in Fig.
1.2 Our Contributions
⋄ New high-probability results for VIPs with heavy-tailed noise.
In our work, we circumvent the limitations of the existing high-probability analysis of stochastic methods for solving VIPs [Juditsky et al., 2011a] and derive the first high-probability results for the methods that solve monotone VIPs with heavy-tailed noise in the unconstrained case.
The key algorithmic ingredient helping us to achieve these results is a proper modification of SEG and SGDA based on the gradient clipping and leading to our clipped-SEG and clipped-SGDA .
Moreover, we derive several high-probability results for clipped-SEG / clipped-SGDA applied to solve structured non-monotone VIPs.
To the best of our knowledge, these results do not have analogs even under the light tails assumption.
We summarize the derived complexity results in Tbl. 1.
⋄ Tight analysis.
The derived complexities satisfy a desirable for high-probability results property: they have logarithmic dependence on 1 / β , where β is a confidence level.
Next, up to the logarithmic factors, our results match known lower bounds 8 in monotone and strongly monotone cases [Beznosikov et al., 2020].
Moreover, we recover and even improve the results from [Juditsky et al., 2011a], which are obtained in the light-tailed case, since our bounds do not depend on the diameter of the domain, see Tbl.
1 for the details.
⋄ Weak assumptions.
One of the key features of our theoretical results is that it relies on assumptions restricted to a ball around the solution .
We achieve this via showing that, with high probability, clipped-SEG / clipped-SGDA do not leave a ball (with a radius proportional to R 0 ) around the solution.
In contrast, the existing works on stochastic methods for solving VIPs usually make assumptions such as boundedness of the variance and Lipschitzness on the whole domain of the considered problem.
Since many practical tasks are naturally unconstrained, such assumptions become too unrealistic since, e.g., stochastic gradients and their variance in the training of neural networks with more than 2 layers grow polynomially fast when x goes to infinity.
However, for a large class of problems including the ones with polynomially growing operators, boundedness of the variance and Lipschitzness hold on any compact set .
That is, our analysis covers a broad class of problems.
⋄ Numerical experiments.
We empirically observe that heavy-tailed gradient noise arises in the training of several practical GANs formulations including StyleGAN2 and WGAN-GP.
Moreover, our experiments show that gradient clipping significantly improves the convergence of SEG / SGDA on such tasks.
These results shed a light on why Adam -based methods are good at training GANs.
Our codes are publicly available: https://github.com/busycalibrating/ clipped-stochastic-methods .
1.3 Closely Related Work
In this section we discuss the most closely related works.
Further discussion is deferred to § A.
High probability convergence.
To the best of our knowledge, the only work deriving highprobability convergence in the context of solving VIPs is [Juditsky et al., 2011a].
In particular,
Juditsky et al.
[2011a] consider monotone and Lipschitz VIP defined 9 on a convex compact set X with the diameter D :=
max x,y ∈X ∥
x − y ∥ , and assume that the noise in F ξ ( x ) is light-tailed.
In this setting, Juditsky et al.
[2011a] propose a stochastic version of the celebrated Extragradient method ( EG )
[Korpelevich, 1976] with non-Euclidean proximal setup – Mirror-Prox .
Moreover, Juditsky et al.
[2011a] derive that after K
= O (max { LD 2 ε , σ 2 D 2 ε 2 ln
2 1 β } ) stochastic oracle calls for some ε > 0 , β ∈ (0 , 1] , the averaged extrapolated iterate (cid:101) x K avg =
[Equation]
(see also clipped-SEG ) satisfies
Gap D ( (cid:101) x K avg ) ≤ ε with probability at least 1 − β .
Although this result has a desirable logarithmic dependence on 1 / β and optimal dependence on ε [Beznosikov et al., 2020], it is derived only for (i) light-tailed case and (ii) bounded domains.
Our results do not have such limitations.
Clipped Stochastic Extragradient
In this section, we consider a version of SEG with gradient clipping.
That is, we apply clipping operator clip ( y, λ ) :
= min { 1 , λ / ∥
y ∥ } y , which is defined for any y ∈ R d (when y =
we set clip (0 , λ ) := 0 ) and any clipping level λ > 0 , to the mini-batched stochastic estimators used both at the extrapolation and update steps of SEG .
This results in the following iterative algorithm:
[Equation]
where
[Equation]
clipped-
[Equation]
where { ξ i,k 1 } m 1 ,k i =1 , { ξ i,k 2 } m 2 ,k i =1 are independent samples from the distribution D .
In the considered algorithm, clipping bounds the effect of heavy-tailedness of the gradient noise, but also creates a bias Deriving tight
In this case, the goal is to find x ∗ ∈
X such that inequality ⟨ F ( x ∗ ) , x − x ∗ ⟩ ≥ 0 holds for all x ∈ X .
that one has to properly control in the analysis.
Moreover, we allow different stepsizes γ 1 , γ 2 [Hsieh et al., 2020, Diakonikolas et al., 2021, Gorbunov et al., 2022a], different batchsizes m 1 ,k , m 2 ,k , and different clipping levels λ 1 ,k , λ 2 ,k .
In particular, taking γ 2 < γ 1 is crucial for our analysis to handle VIP satisfying star-negative comonotonicity (SNC) with ρ > 0 .
Our convergence results for clipped-SEG are summarized in the following theorem.
For simplicity, we omit here the numerical constants, which are explicitly given in § C.
Theorem 2.1.
Consider clipped-SEG run for K ≥ 0 iterations.
Let
R ≥ R 0 =
∥ x 0 − x ∗ ∥ .
Case 1.
Let Assump.
1.1, 1.2, 1.3 hold for Q = B 4 R ( x ∗ ) , where γ 1 = γ 2 = γ with 0 < γ =
O ( 1 / ( LA ) ) , λ 1 ,k
= λ 2 ,k ≡ λ =
Θ( R / ( γA ) ) , m 1 ,k
= m 2 ,k ≡ m = Ω(max { 1 , ( K +1) γ 2 σ 2 A / R 2 } ) , where A = ln 6( K +1) β , β ∈ (0 , 1] are such that A ≥ 1 .
Case 2.
Let Assump.
1.1, 1.2, 1.4 hold for Q = B 3 R ( x ∗ ) , where γ 2 + 2 ρ ≤ γ 1 ≤
O ( 1 / ( LA ) ) , λ 1 ,k ≡ λ 1 =
Θ( R / ( γ 1 A ) ) , λ 2 ,k ≡ λ 2 =
Θ( R / ( γ 2 A ) ) , m 1 ,k ≡ m 1 = Ω(max { 1 , max { γ 1 γ 2 ( K +1) , √ γ 31 γ
( K +1) } σ 2 A / R 2 } ) , m 2 ,k ≡ m 2 = Ω(max { 1 , ( K +1) γ 22 σ 2 A / R 2 } ) ,
where A = ln 8( K +1) β , β ∈ (0 , 1] are such that A ≥ 1 .
Case 3.
Let Assump.
1.1, 1.2, 1.5 hold for Q = B 3 R ( x ∗ ) , where γ 1 = γ 2 = γ with 0 < γ ≤ O ( 1 / ( LA ) ) , λ 1 ,k = λ 2 ,k = λ k
= Θ( exp( − γµ (1+ k / 2 )) R / ( γA ) ) , m 1 ,k
= m 2 ,k
= m k = Ω(max { 1 , ( K +1) γ 2 σ 2 A / (exp( − γµk )
R 2 ) } ) , where A = ln 6( K +1) β , β ∈ (0 , 1] are such that A ≥ 1 .
Then, to guarantee Gap R ( (cid:101)
x K avg ) ≤ ε in Case 1 with (cid:101) x
K avg = 1 K +1 (cid:80)
Kk =0 (cid:101) x k , 1 K +1 (cid:80) Kk =
∥ F ( x k ) ∥ 2 ≤
Lε in Case 2 , ∥ x K − x ∗ ∥ 2 ≤ ε in Case 3 , with probability ≥ 1 − β clipped-SEG requires
Case 1 and 2 : (cid:101) O
[Equation]
max (cid:110) [Equation] [Equation] [Equation]
[Equation] 2 (cid:111)(cid:17) and Case 3 : (cid:101) O (cid:16) max (cid:110)
[Equation]
µε (cid:111)(cid:17) (2) oracle calls.
The above guarantees hold in two different regimes: large step-sizes γ ≈ 1 / LA (requiring large batch-sizes), and small step-sizes, allowing small batch-sizes m =
O (1) .
Proof sketch in Case 1 .
Modifying the analysis of EG , we first derive that for all t ≥
we have
Gap R ( (cid:101) x t avg ) and ∥ x t − x ∗ ∥ 2 are not greater than max u ∈ B R ( x ∗ ) {∥ x 0 − u ∥ 2 + 2 γ (cid:80) t − 1 l =0 ⟨ x l − u − γF ( (cid:101) x l ) , θ l ⟩ + γ 2 (cid:80) t − 1 l =0 ( ∥ θ l ∥ 2 + 2 ∥ ω l ∥ 2 ) } if x l , (cid:101) x
l lie in B 4 R ( x ∗ ) for all l = 0 , 1 , . .
, t − 1 , where θ l = F ( (cid:101) x l ) − (cid:101) F ξ l 2 ( (cid:101) x l ) and ω l = F ( x l ) − (cid:101)
F ξ l 1 ( x l ) .
Next, using this recursion and the induction argument, we show that with high probability x t , (cid:101) x t ∈ B 4 R ( x ∗ ) for all t = 0 , 1 , . . .
, K + 1 .
This gives us an upper bound for Gap R ( (cid:101) x t avg ) .
After that, we upper bound max u ∈ B R ( x ∗ ) { 2 γ (cid:80) t − 1 l =0 ⟨ x l − u − γF ( (cid:101) x l ) , θ l ⟩ + γ 2 (cid:80) t − 1 l =0 ( ∥ θ l ∥ 2 + 2 ∥ ω l ∥ 2 ) } by 5 R 2 with high-probability.
We achieve this via the proper choice of the clipping level λ =
Θ( R / ( γA ) ) implying that ∥ F ( (cid:101) x l ) ∥ ≤ λ / 2 and ∥ F ( x l ) ∥ ≤ λ /with high probability for all l = 0 , 1 , . .
, t − 1 .
This clipping politics helps to properly bound the bias and the variance of the clipped estimators using Lem.
B.2.
After that, it remains to apply the
Bernstein inequality for the martingale differences (Lem. B.1).
See the detailed proof in § C.
In addition to the discussion given in the introduction (see § 1.2, 1.3 and Tbl. 1), we provide here several important details about the derived results.
First of all, we notice that up to the logarithmic factors depending on β our high-probability convergence results recover the state-of-the-art inexpectation ones for SEG in the monotone [Beznosikov et al., 2020], star-negative comonotone [Diakonikolas et al., 2021], and quasi-strongly monotone [Gorbunov et al., 2022a] cases.
Moreover, as we show in Corollaries C.1 and C.3, to achieve these results in monotone and quasi-strongly monotone cases, it is sufficient to choose constant batchsize m =
O (1) and small enough stepsize γ .
In contrast, when the operator is star-negatively comonotone, we do rely on the usage of large stepsize
γ 1 and large batchsize m
=
O ( K ) for the extrapolation step to obtain (2).
However, known in-expectation results from [Diakonikolas et al., 2021, Lee and Kim, 2021] also rely on large O ( K ) batchsizes in this case.
We leave the investigation of this limitation to the future work.
Clipped Stochastic Gradient Descent-Ascent
In this section, we extend the approach described above to the analysis of SGDA with clipping.
That is, we consider the following algorithm:
[Equation]
x k ) , where (cid:101) F
[Equation]
( x
[Equation]
clip
[Equation]
where { ξ i,k } mi =1 are independent samples from the distribution D .
In the above method, clipping plays a similar role as in clipped-SEG .
Our convergence results for clipped-SGDA are summarized below.
The general idea of the proof is similar to the one for clipped-SEG , see the details in
Appendix D, where we also explicitly give the constants that are omitted here for simplicity.
Theorem 3.1.
Consider clipped-SGDA run for K ≥ 0 iterations.
Let
R ≥ R 0 =
∥ x 0 − x ∗ ∥ .
Case 1.
Let Assump.
1.1, 1.3, 1.6 hold for Q = B 3 R ( x ∗ ) , where 0 <
γ =
O ( 1 / ( ℓA ) ) , λ k ≡ λ =
Θ( R / ( γA ) ) , m k ≡ m = Ω(max { 1 , ( K +1)
γ 2 σ 2 A / R 2 } ) , where β ∈ (0 , 1] and A =
ln 6( K +1) β ≥ 1 .
Case 2.
Let Assump.
1.1, 1.6 hold for Q = B 2 R ( x ∗ ) , where 0 <
γ =
O ( 1 / ( ℓA ) ) , λ k ≡ λ =
Θ( R / ( γA ) ) , m k ≡ m = Ω(max { 1 , ( K +1)
γ 2 σ 2 A / R 2 } ) , where β ∈ (0 , 1] and A =
ln 4( K +1) β ≥ 1 .
Case 3.
Let Assump.
1.1, 1.5, 1.6 hold for Q = B 2 R ( x ∗ ) , where 0 <
γ =
O ( 1 / ( ℓA ) ) , λ k
=
Θ
( e − γµ (1+ k/ 2) R ( γA ) ) , m k = Ω(max { 1 , ( K +1) γ 2 σ 2 A e γµk R} ) , where β ∈ (0 , 1] and A =
ln 4( K +1) β ≥ 1 .
Then, to guarantee Gap R ( (cid:101)
x K avg ) ≤ ε in Case 1 with (cid:101) x
K avg = 1 K +1 (cid:80)
Kk =0 (cid:101) x k , 1 K +1 (cid:80)
Kk =
∥ F ( x k ) ∥ 2 ≤ ℓε in Case 2 , ∥ x K − x ∗ ∥ 2 ≤ ε in Case 3 , with probability ≥ 1 − β , clipped-SGDA requires
Case 1 and 2 :
[Equation]
and
Case 3 :
[Equation]
oracle calls.
The above guarantees hold in two different regimes: large step-sizes γ ≈ 1 / ℓA (requiring large batch-sizes), and small step-sizes, allowing small batch-sizes m =
O (1) .
As for the clipped-SEG , up to the logarithmic factors depending on β , our high-probability convergence results from the theorem above recover the state-of-the-art in-expectation ones for SGDA under monotonicity and star-cocoercivity [Beznosikov et al., 2022], star-cocoercivity [Gorbunov et al., 2022a] 10 , quasi-strong monotonicity and star-cocoercivity [Loizou et al., 2021] assumptions.
Moreover, in all the cases , one can achieve the results from (3) using constant batchsize m =
O (1) and small enough stepsize γ (see Corollaries D.1, D.2, D.3 for the details).
Finally, to the best of our knowledge, we derive the first high-probability convergence results for SGDA -type methods.
Experiments
To validate our theoretical results, we conduct experiments on heavy-tailed min-max problems to demonstrate the importance of clipping when using non-adaptive methods such as SGDA or SEG .
We train a Wasserstein GAN with gradient penalty [Gulrajani et al., 2017] on CIFAR-10 [Krizhevsky et al., 2009] using SGDA , clipped-SGDA , and clipped-SEG , and show the evolution of the gradient noise histograms during training.
We demonstrate that gradient clipping also stabilizes the training of more sophisticated GANs when using SGD by training a StyleGAN2 model [Karras et al., 2020] on
FFHQ [Karras et al., 2019], downsampled to 128 × 128 .
Although the generated sample quality is not competitive with a model trained with Adam , we find that StyleGAN2 fails to generate anything meaningful when trained with regular SGDA whereas clipped methods learn meaningful features.
We do not claim to outperform state-of-the-art adaptive methods (such as Adam ); our focus is to validate our theoretical results and to demonstrate that clipping improves SGDA in this context.
WGAN-GP.
In this section, we focus on the ResNet architecture proposed in Gulrajani et al.
[2017], and we adapt our code from a publicly available WGAN-GP implementation.
11 We first compute the gradient noise distribution and validate if it is heavy-tailed.
Taking the fixed randomly initialized
[Figure]
[Figure] [Figure]
D (
[Figure]
Hyperparameters weights, we iterate through 1000 steps (without parameter updates) to compute the noise norm for each minibatch (sample with replacement as in normal GAN training).
We show the distributions for the generator and discriminator in Fig.
2 and also compute p
mR = F 1 . 5 ( X ) and p eR =
F 3 ( X ) , where F λ ( X ) =
P ( Q 3 + λ ( Q 3 − Q 1 ) <
X ) , X is the gradient noise distribution, and Q i is the i th quartile.
This is a metric introduced by Jordanova and Petkova [2017] to measure how heavytailed a distribution is based on the distribution’s quantiles, where p mR and p eR quantify “mild” and “extreme” (right side) heavy tails respectively.
A normal distribution should have p mR N ≈ 0 .
0035 and p eR N ≈ 1 . 2 × 10 − 6 .
In all tests, we compute the ratios ρ mR = p mR /p mR N and ρ eR = p eR /p
eR N and empirically find that we at least have mild heavy tails, and sometimes extremely heavy tails.
We train the ResNet generator on CIFAR-10 with SGDA / SEG , and clipped-SGDA / SEG .
We use the default architectures and training parameters specified in Gulrajani et al.
[2017] ( λ GP = 10 , n dis = 5 , learning rate decayed linearly to 0 over 100k steps), with the exception of doubling the feature map of the generator.
The clipped methods are implemented as standard gradient norm clipping, applied after computing the gradient penalty term for the critic.
In addition to norm clipping, we also test coordinate-wise gradient clipping, which is more common in practice [Goodfellow et al., 2016b].
For all methods, we tune the learning rates and clipping threshold where applicable.
We do not use momentum following a standard practice for GAN training [Gidel et al., 2019b].
We find that clipped methods outperform regular SGDA and SEG .
In addition to helping prevent exploding gradients, clipped methods achieve a better Fréchet inception distance (FID) score [Heusel et al., 2017]; the best FID obtained for clipped methods is 19 .
65 in comparison to 67 .
37 for regular
SGDA , both trained for 100k steps.
A summary of FIDs obtained during hyperparameter tuning is shown Fig.
2, where the best FID score obtained in the first 35k iterations is drawn for each hyperparameter configuration and optimization method.
At a high level, we do a log-space sweep over [2e − 5 , 0 . 2] for the learning rates, [10 − 1 , 10] for the norm-clip parameter, and [10 − 3 , 10 − 1 ] for the coordinate clip parameter (with some exceptions) – please refer to § E for further details.
We also show the evolution of the gradient noise histograms during training for SGD and clipped-SGDA in
Fig. 4.
Note that for regular SGD , the noise distribution remains heavy tailed and does not appear to change much throughout training.
In contrast, the noise histograms for clipped-SGDA (in particular the generator) seem to develop lighter tails during training.
StyleGAN2.
We extend our experiments to StyleGAN2, but limit our scope to clipped-SGDA with coordinate clipping as coordinate clipped-SGDA generally performs the best, and StyleGAN is expensive to train.
We train on FFHQ downsampled to 128 × 128 pixels, and use the recommended StyleGAN2 hyperparameter configuration for this resolution (batch size = 32 , γ = 0 . 1024 , map depth = 2 , channel multiplier = 16384 ), see further experimental details in § E. We obtain the gradient noise histograms at initialization and for the best trained clipped-SGDA from the hyperparameter sweep and once again observe heavy tails (especially in the discriminator, see Fig. 3).
We find that all regular SGDA runs fail to learn anything meaningful, with FID scores fluctuating
G e n e r a t o r
D i s c
r
i m i n a t
o r
[Figure] [Figure]
[Table]
D
[Figure]
[Figure] [Figure]
[Table]
G e n e r a t o r
[Table]
D i s c
r
i m i n a t
o r
[Table]
[Figure]
around 320 and only able to generate noise.
In contrast, while there is a clear gap in quality when compared to what StyleGAN2 is capable of, a model trained with clipped-SGDA with appropriately set hyperparameters is able to produce images that distinctly resemble faces (see Fig. 3).
Acknowledgments and Disclosure of Funding
This work was partially supported by a grant for research centers in the field of artificial intelligence, provided by the Analytical Center for the Government of the Russian Federation in accordance with the subsidy agreement (agreement identifier 000000D730321P5Q0002) and the agreement with the
Moscow Institute of Physics and {+Technology dated November 1, 
+}
No. 70-2021-00138.
The work by P. Dvurechensky was funded by the Deutsche Forschungsgemeinschaft (DFG, German
Research Foundation) under Germany’s Excellence Strategy – The Berlin Mathematics Research
Center MATH+ (EXC-2046/1, project ID: 390685689)
E
Extra Experiments
E.1 WGAN-GP
A Further Related Work
Convergence in expectation.
Convergence in expectation of stochastic methods for solving VIPs is relatively well-studied in the literature.
In particular, versions of SEG are studied under bounded variance [Beznosikov et al., 2020, Hsieh et al., 2020], smoothness of stochastic realizations [Mishchenko et al., 2020], and more refined assumptions unifying previously used ones [Gorbunov et al., 2022a].
Recent advances on the in-expectation convergence of SGDA are obtained in [Loizou et al., 2021,
Beznosikov et al., 2022].
Gradient clipping.
In the context of solving minimization problems, gradient clipping [Pascanu et al., 2013] and normalization [Hazan et al., 2015] are known to have a number of favorable properties such as practical robustness to the rapid changes of the loss function [Goodfellow et al., 2016a], provable convergence for structured non-smooth problems with polynomial growth Zhang et al.
[2020a], Mai and Johansson [2021] and for the problems with heavy-tailed noise in convex [Nazin et al., 2019,
Gorbunov et al., 2020, 2021] and non-convex cases [Zhang et al., 2020b, Cutkosky and Mehta, 2021].
Our work makes a further step towards a better understanding of gradient clipping and is the first to study the theoretical convergence of clipped first-order stochastic methods for VIPs.
Structured non-monotonicity.
There is a noticeable growing interest of the community in studying the theoretical convergence guarantees of deterministic methods for solving VIP with non-monotone operators F ( x ) having a certain structure, e.g., negative comonotonicty [Diakonikolas et al., 2021,
Lee and Kim, 2021, Böhm, 2022], quasi-strong monotonicity [Song et al., 2020, Mertikopoulos and
Zhou, 2019] and/or star-cocoercivity [Loizou et al., 2021, Gorbunov et al., 2022b,a, Beznosikov et al., 2022].
In the context of stochastic VIPs, SEG (with different extrapolation and update stepsizes) is analyzed under negative comonotonicity by Diakonikolas et al.
[2021] and under quasi-strong monotonicity by Gorbunov et al.
[2022a], while SGDA is studied under quasi-strong monotonicity and/or star-cocoercivity by [Loizou et al., 2021, Beznosikov et al., 2022].
These results establish in-expectation convergence rates.
Our paper continues this line of works and provides the first high-probability analysis of stochastic methods for solving VIPs with structured non-monotonicity.
B Auxiliary Results
Useful inequalities.
For all a, b ∈ R d and α > 0 the following relations hold:
[Equation]
Bernstein inequality.
In our proofs, we rely on the following lemma known as Bernstein inequality for martingale differences [Bennett, 1962, Dzhaparidze and Van Zanten, 2001, Freedman et al., 1975].
Lemma B.1.
Let the sequence of random variables { X i } i ≥ 1 form a martingale difference sequence, i.e. E
[ X i |
X i − 1 , . . .
, X 1 ]
=
for all i ≥ 1 .
Assume that conditional variances σ 2 i def = E (cid:2)
X 2 i | X i − 1 , . . .
, X 1 (cid:3) exist and are bounded and assume also that there exists deterministic constant c > 0 such that |
X i | ≤ c almost surely for all i ≥ 1 .
Then for all b > 0 , G > 0 and n ≥
[Equation]
> b and
[Equation]
Bias and variance of clipped stochastic vector.
We also use the following properties of clipped stochastic estimators from [Gorbunov et al., 2020].
Lemma B.2 (Simplified version of Lemma F.5 from [Gorbunov et al., 2020]) .
Let X be a random vector in R d and (cid:101)
X
= clip ( X, λ ) .
Then,
[Equation]
Moreover, if for some σ ≥
E [ X ] =
[Equation]
d ,
[Equation] [Equation]
and x ≤ λ / 2 , then
[Equation]
Proof.
The proof of this lemma is identical to the original one, since Gorbunov et al.
[2020] rely only on (cid:101)
X
= clip ( X, λ ) to derive (8), and to prove (10)-(12) they use only (9), (cid:101) X =
clip
( X, λ ) and x ≤ λ /
C Clipped Stochastic Extragradient: Missing
Details C.1 Monotone Case
Lemma C.1.
Let Assumptions 1.1, 1.2, 1.3 hold for Q = B 4 R ( x ∗ ) , where R ≥ R 0 def = ∥ x 0 − x ∗ ∥ , and γ 1 = γ 2 = γ , 0 < γ ≤ 1 / √ 2 L .
If x k and (cid:101) x k lie in B 4 R ( x ∗ ) for all k = 0 , 1 , . .
, K for some
K ≥ 0 , then for all u ∈ B 4 R ( x ∗ ) the iterates produced by clipped-SEG satisfy
[Equation]
Proof.
Using the update rule of clipped-SEG , for all u ∈ B 4 R ( x ∗ ) we obtain
[Equation]
where in the last step we additionally use x k − (cid:101) x k = γ (cid:101) F ξ k 1 ( x k ) after the application of Lipschitzness of F .
Since γ ≤ 1 / √ 2 L , we have γ 2 (cid:0) 1 − 2 γ 2 L 2 (cid:1) ∥ (cid:101)
F ξ k 1 ( x k ) ∥
2 ≥ 0 , implying
[Equation]
Finally, we sum up the above inequality for k = 0 , 1 , . . .
, K and divide both sides of the result by 2 γ ( K + 1) :
[Equation]
This concludes the proof.
Theorem C.1.
Let Assumptions 1.1, 1.2, 1.3 hold for Q = B 4 R ( x ∗ ) , where R ≥ R 0 def = ∥ x 0 − x ∗ ∥ , and 12 γ 1 = γ {+
+} = γ ,
[Equation] [Equation]
[Equation]
for some K ≥ 0 and β ∈ (0 , 1] such that ln 6( K +1) β ≥ 1 .
Then, after K iterations the iterates produced by clipped-SEG with probability at least 1 − β satisfy
[Equation]
where (cid:101) x K avg is defined in (14) .
Proof.
We introduce new notation: R k = ∥
x k − x ∗ ∥ for all k ≥ 0 .
The proof is based on deriving via induction that R 2 k ≤ ˜ CR 2 for some numerical constant ˜ C > 0 .
In particular, for each k = 0 , . .
, K +
we define probability event E k as follows: inequalities
[Equation]
hold for t = 0 , 1 , . . .
, k simultaneously.
Our goal is to prove that P { E k } ≥ 1 − kβ / ( K +1) for all k = 0 , 1 , . .
, K + 1 .
We use the induction to show this statement.
For k =
the statement
, batch-sizes, and clipping levels.
However, our main goal is deriving results in terms of O ( · ) , where numerical constants are not taken into consideration.
Although it is possible to significantly improve the dependence on numerical factors, we do not do it for the sake of proofs’ simplicity.
is trivial since ∥ x 0 − u ∥ 2 ≤ 2 ∥ x 0 − x ∗ ∥ 2 +
∥ x ∗ − u ∥ 2 ≤ 4 R 2 ≤ 9 R 2 and ∥ γ (cid:80) k − 1 l =0 θ l ∥ =for any u ∈ B R ( x ∗ ) .
Next, assume that the statement holds for k = T − 1 ≤ K , i.e., we have P { E T − 1 } ≥ 1 − ( T − 1) β / ( K +1) .
We need to prove that P { E T } ≥ 1 − Tβ / ( K +1) .
First of all, we show that probability event E T − 1 implies R t ≤ 3 R for all t = 0 , 1 , . . .
, T .
For t = 0 we already proved it.
Next, assume that we have R t ≤ 3 R for all t = 0 , 1 , . . .
, t ′ , where t ′ < T .
Then, for all t = 0 , 1 , . . .
, t ′ we have
[Equation]
i.e., (cid:101) x t ∈ B 4 R ( x ∗ ) .
This means that the assumptions of Lemma C.1 hold and we have that probability event E T − 1 implies max
[Equation]
meaning that
[Equation]
≤
[Equation]
i.e., R t ′ +1 ≤ 3 R .
That is, we proved that probability event E T − 1 implies R t ≤ 3 R
and
[Equation] [Equation]
for all t = 0 , 1 , . . .
, T .
Moreover, in view of (23) E T − 1 also implies that ∥ (cid:101) x t − x ∗ ∥ ≤ 4 R for all t = 0 , 1 , . . .
, T .
Using this, we derive that E T − 1 implies
[Equation] [Equation]
for all t = 0 , 1 , . . .
, T .
Consider random vectors η t = (cid:26) x
[Equation]
∗ − γF ( (cid:101) x t ) , if ∥ x t − x ∗ − γF ( (cid:101) x t ) ∥
≤ 5 R
[Equation]
0 , otherwise, for all t = 0 , 1 , . .
, T .
We notice that η t is bounded with probability 1 :
[Equation]
for all t = 0 , 1 , . . .
, T .
Moreover, in view of (25), probability event E T − 1 implies η t = x
t − x ∗ − γF ( (cid:101) x t ) for all t = 0 , 1 , . . .
, T .
Therefore, E T − 1 implies
[Equation]
where A T is defined in (21).
To continue our derivation we introduce new notation:
[Equation]
for all l = 0 , . . .
, T − 1 .
By definition we have θ l = θ ul + θ bl , ω l = ω ul + ω
bl for all l = 0 , . . .
, T − 1 .
Using the introduced notation, we continue our derivation as follows:
E T − 1 implies
[Equation]
(cid:2)
∥ ω ul ∥
[Equation]
(cid:3)(cid:17)
[Equation] [Equation]
The rest of the proof is based on deriving good enough upper bounds for 2 γR
[Equation]
T − 1 l =0 θ l (cid:13)(cid:13)(cid:13) , ① , ② , ③ , ④ , ⑤ , i.e., we want to prove that 2 γR
[Equation]
+ ① + ② + ③ + ④ + ⑤ ≤ 5 R 2 with high probability.
Before we move on, we need to derive some useful inequalities for operating with θ ul , θ bl , ω ul , ω
bl .
First of all, Lemma B.2 implies that
[Equation] [Equation]
for all l = 0 , 1 , . . .
, T − 1 .
Next, since { ξ i,l 1 } mi =1 , { ξ i,l 2 } mi =1 are independently sampled from D , we have E
[Equation]
l 1 [ F
[Equation]
1 ( x l )]
= F ( x l ) , E
[Equation] [Equation]
F
[Equation]
and
[Equation]
for all l = 0 , 1 , . . .
, T − 1 .
Moreover, probability event E T − 1 implies
[Equation]
for all l = 0 , 1 , . . .
, T − 1 .
Therefore, in view of Lemma B.2, E T − 1 implies that
[Equation] [Equation]
for all l = 0 , 1 , . . .
, T − 1 .
Upper bound for ① .
Since E ξ l 2 [ θ ul ] = 0 , we have
[Equation]
Next, the summands in ① are bounded with probability 1 :
[Equation]
Moreover, these summands have bounded conditional variances σ
[Equation]
[Equation]
That is, sequence { 2 γ ⟨ η l , θ ul ⟩} l ≥ 0 is a bounded martingale difference sequence having bounded conditional variances { σ 2 l } l ≥ 0 .
Applying the Bernstein’s inequality (Lemma B.1) with X l = 2 γ ⟨ η l , θ ul ⟩ , c defined in (34), b = R 2 , G =
[Equation] [Equation]
6( K +
[Equation]
β , we get that
[Equation]
and
[Equation]
In other words
[Equation]
P { E ① } ≥ 1 − β 3( K +1)
[Equation]
where probability event E ① is defined as
[Equation]
either
[Equation]
Moreover
[Equation]
we notice here that probability event E T − 1 implies that
[Equation] [Equation]
[Equation] [Equation]
Next, the summands in ④ are bounded with probability 1 :
[Equation]
Moreover
[Equation]
these summands have bounded conditional variances
[Equation]
That is, sequence (cid:110)
[Equation]
2 (cid:16) ∥ θ ul ∥ 2 + 2 ∥ ω ul ∥ 2 − E
[Equation]
l 2 (cid:2) ∥ θ ul ∥ 2 (cid:3) − 2 E ξ l 1 (cid:2)
∥ ω ul ∥ 2 (cid:3)(cid:17)(cid:111) l ≥ 0 is a bounded martingale difference sequence having bounded conditional variances { (cid:101) σ 2 l } l ≥ 0 .
Applying the Bernstein’s inequality (Lemma B.1) with X l = 2 γ 2 (cid:16) ∥ θ ul ∥ 2 +
∥ ω ul ∥ 2 − E
[Equation]
2 (cid:2)
∥ θ ul ∥ 2 (cid:3) − 2 E
[Equation]
(cid:2)
∥ ω ul ∥ 2 (cid:3)(cid:17) , c defined in (42), b = 16 R 2 , G =
R216ln 6( K +1) β , we get that
P (cid:40) | ④ | > 1 6 R 2 and
[Equation]
In other words, P { E ④ } ≥ 1 − β 3( K +1) , where probability event E ④ is defined as
E ④ =
[Equation]
either
[Equation]
Moreover, we notice here that probability event E T − 1 implies that
[Equation]
Upper bound for ⑤ .
Probability event E T − 1 implies
[Equation]
Upper bound for 2 γR (cid:13)(cid:13) (cid:13) (cid:80) T − 1 l =
θ l (cid:13)(cid:13) (cid:13) .
To handle this term, we introduce new notation:
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
0 , otherwise for l = 1 , 2 , . .
, T − 1 .
By definition, we have ∥ ζ
[Equation]
∥ ≤ R.
Therefore, in view of (22), probability event E T − 1 implies
[Equation]
T − 1 (cid:88) l
= θ l (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
= 2 R (cid:118)(cid:117)(cid:117)(cid:116)
γ 2 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) T − 1 (cid:88) l = θ l (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
= 2 R (cid:118)(cid:117)(cid:117)(cid:116) γ 2 T − 1 (cid:88) l
= ∥
θ l ∥ 2 + 2 γ
T − 1 (cid:88) l
= (cid:42) γ l − 1 (cid:88) r
= θ r , θ l (cid:43) =
2 R (cid:118)(cid:117)(cid:117)(cid:116) γ 2 T − 1 (cid:88) l
= ∥
θ l ∥ 2 + 2 γ
T − 1 (cid:88) l = ⟨ ζ l , θ l ⟩ (27) ≤
[Equation]
R (cid:118)(cid:117)(cid:117)(cid:117) (cid:117)(cid:116) ③ + ④ + ⑤ + 2 γ
T − 1 (cid:88) l = ⟨ ζ l , θ ul ⟩ (cid:124) (cid:123)(cid:122) (cid:125) ⑥ + 2 γ
T − 1 (cid:88) l = ⟨ ζ l , θ bl (cid:124) (cid:123)(cid:122) (cid:125) ⑦ ⟩ .
Following similar steps as before, we bound ⑥ and ⑦ .
Upper bound for ⑥ .
Since E ξ l 2 [ θ ul ] = 0 , we have
[Equation]
Next, the summands in ④ are bounded with probability 1 :
[Equation]
Moreover
[Equation]
these summands have bounded conditional variances ˆ σ
[Equation]
[Equation]
That is, sequence { 2 γ ⟨ ζ l , θ ul ⟩} l ≥ 0 is a bounded martingale difference sequence having bounded conditional variances { ˆ σ 2 l } l ≥ 0 .
Applying Bernstein’s inequality (Lemma B.1) with X l = 2 γ ⟨ ζ l , θ ul ⟩ , c defined in (34), b = R 2 4 , G =
R96ln 6( K +1) β , we get that
P (cid:40) | ⑤ | >
[Equation]
R 2 and
[Equation]
In other words, P { E ⑥ } ≥ 1 −
[Equation]
, where probability event E ⑥ is defined as
[Equation]
either
[Equation] [Equation]
Moreover, we notice here that probability event E T − 1 implies that
[Equation]
ξ
[Equation]
Upper bound for ⑦ .
Probability event E T − 1 implies
[Equation]
Final derivation.
Putting all bounds together, we get that E T − 1 implies
[Equation]
Moreover, in view of (36)
[Equation] [Equation]
[Equation]
and our induction assumption
[Equation]
we have
[Equation]
where probability events E ① , E ④ , and E ⑥ are defined as
E ①
[Equation]
either
T − 1 (cid:88) l = σ 2 l >
R6 ln 6( K +1) β or | ① | ≤ R(cid:41) ,
E ④ = (cid:40) either
T − 1 (cid:88) l =0 (cid:101) σ 2 l >
R216 ln 6( K +1) β or | ④ | ≤ 1 6 R(cid:41) ,
E ⑥
=
(cid:40) either
T − 1 (cid:88) l
= ˆ σ 2 l >
R96 ln 6( K +1) β or | ⑥ | ≤ 1 4 R(cid:41) .
Putting all of these inequalities together, we obtain that probability event E T − 1 ∩ E ① ∩ E ④ ∩ E ⑥ implies
[Equation]
Moreover, union bound for the probability events implies
[Equation]
This is exactly what we wanted to prove (see the paragraph after inequalities (21), (22)).
Therefore, for all k = 0 , 1 , . . .
K + 1 we have P { E k } ≥ 1 − kβ / ( K +1) .
, i.e., for k = K +
we have that with probability at least 1 − β inequality
[Equation]
holds.
This concludes the proof.
Corollary C.1.
Let the assumptions of Theorem C.1 hold.
Then, the following statements hold.
Large stepsize/large batch
The choice of stepsize and batchsize
[Equation]
satisfies conditions (17) and (19) .
With such choice of γ, m , and the choice of λ as in (18) , the iterates produced by clipped-SEG after K iterations with probability at least 1 − β satisfy
[Equation]
In particular, to guarantee Gap R ( (cid:101)
x K avg ) ≤ ε with probability at least 1 − β for some ε >clipped-SEG requires,
[Equation]
Small stepsize/small batch.
The choice of stepsize and batchsize
[Equation]
satisfies conditions (17) and (19) .
With such choice of γ, m , and the choice of λ as in (18) , the iterates produced by clipped-SEG after K iterations with probability at least 1 − β satisfy Gap [Equation] K
[Equation]
max
[Equation]
In particular, to guarantee Gap R ( (cid:101)
x K avg ) ≤ ε with probability at least 1 − β for some ε > 0 , clipped-SEG requires
[Equation]
Proof.
Large stepsize/large batch.
First of all, we verify that the choice of γ and m from (57) satisfies conditions (17) and (19): (17) trivially holds and (19) holds since
[Equation]
Therefore, applying Theorem C.1, we derive that with probability at least 1 − β
Gap
[Equation] [Equation]
To guarantee Gap R ( (cid:101) x K avg ) ≤ ε , we choose K in such a way that the right-hand side of the above inequality is smaller than ε that gives
[Equation]
The total number of oracle calls equals
[Equation]
Small stepsize/small batch.
First of all, we verify that the choice of γ and m from (57) satisfies conditions (17) and (19):
[Equation]
Therefore, applying Theorem C.1, we derive that with probability at least 1 − β
Gap R
[Equation]

[Equation]
To guarantee Gap R ( (cid:101) x K avg ) ≤ ε , we choose K in such a way that the right-hand side of the above inequality is smaller than ε that gives
[Equation]
O
[Equation]
The total number of oracle calls equals 2 m ( K + 1) =
2( K + 1) .
C.2 Star-Negative Comonotone Case
Lemma C.2.
Let Assumptions 1.2, 1.4 hold for Q = B 3 R ( x ∗ )
=
{ x ∈ R d | ∥ x − x ∗ ∥ ≤ 3 R } , where R ≥ R 0 def = ∥ x 0 − x ∗ ∥ , and γ 2 + 2 ρ < γ 1 ≤ 1 / (2 L ) .
If x k and (cid:101) x k lie in B 3 R 0 ( x ∗ ) for all k = 0 , 1 , . .
, K for some K ≥ 0 , then the iterates produced by clipped-SEG satisfy
[Equation]
where θ k , ω k are defined in (15) , (16) .
Proof.
Using the update rule of clipped-SEG , we obtain ∥ x k +
[Equation]
− x
[Equation]
∥
[Equation]
where in the last step we additionally use x k − (cid:101) x k = γ 1 (cid:101) F ξ k 1 ( x k ) after the application of Lipschitzness of F and we use our assumption on γ 1 , γ 2 , ρ : γ 2 + 2 ρ ≤ γ 1 .
Since γ 1 ≤ 1 / (2 L ) , we have γ 1 γ 2 (cid:0) 1 − 2 γ 21 L 2 (cid:1) ∥
(cid:101)
F ξ k 1 ( x k ) ∥ 2 ≥ 0 and, using (6) with α = 1 , we derive ∥ x k +1 − x ∗ ∥≤ ∥ x k − x ∗ ∥ 2 − γ 1 γ 2 2 (cid:0) 1 −
γ 21 L 2 (cid:1)
∥ F ( x k ) ∥+2 γ 2 ⟨ [Equation] k − x ∗ − γ 2 F ( [Equation]
, θ
[Equation]
⟩ + γ 22 ∥ θ [Equation] ∥ [Equation]
+
[Equation]
γ
[Equation] γ(cid:18) [Equation] 2 − γ 21 L(cid:19)
∥ ω k ∥ 2 .
Rearranging the terms and using 32 − γ 21 L 2 ≤ 32 , 1 − 2 γ 21 L 2 ≥ 1 / 2 , we derive γ 1 γ
[Equation]
Finally, we sum up the above inequality for k = 0 , 1 , . . .
, K and divide both sides of the result by ( K + 1) :
[Equation]
This finishes the proof
[Equation]
Theorem C.2.
Let Assumptions 1.1, 1.2, 1.4 hold for Q = B 3 R ( x ∗ ) , where R ≥ R 0 def = ∥ x 0 − x ∗ ∥ , and
[Equation]
for some K ≥ 0 and β ∈ (0 , 1] such that ln 6( K +1) β ≥ 1 .
Then, after K iterations the iterates produced by clipped-SEG with probability at least 1 − β satisfy
[Equation]
Proof.
As in the proof of Theorem C.1, we use the following notation: R k = ∥
x k − x ∗ ∥ 2 , k ≥ 0 .
We will derive (69) by induction.
In particular, for each k = 0 , . .
, K +
we define probability event E k as follows: inequalities
[Equation]
hold for t = 0 , 1 , . . .
, k simultaneously.
Our goal is to prove that P { E k } ≥ 1 − kβ / ( K +1) for all k = 0 , 1 , . .
, K + 1 .
We use the induction to show this statement.
For k =
the statement is trivial since R 20 ≤ 4 R 2 by definition.
Next, assume that the statement holds for k = T − 1 ≤ K , i.e., we have P { E T − 1 } ≥ 1 − ( T − 1) β / ( K +1) .
We need to prove that P { E T } ≥ 1 − Tβ / ( K +1) .
First of all, since R 2 t ≤ 4 R 2 , we have x t ∈ B 2 R ( x ∗ ) .
Operator F is L -Lipschitz on B 3 R ( x ∗ ) .
Therefore, probability event E T − 1 implies
[Equation]
and
[Equation]
for all t = 0 , 1 , . . .
, T − 1 .
Next, we show that probability event E T − 1 implies ∥ (cid:101) x
t − x ∗ ∥ ≤ 3 R and derive useful inequalities related to θ t for all t = 0 , 1 , . .
, T − 1 .
Indeed, due to Lipschitzness of F probability event E T −implies
[Equation]
and ∥
[Equation]
for all t = 0 , 1 , . . .
, T − 1 .
That is, E T − 1 implies that x t , (cid:101) x t ∈ B 3 R ( x ∗ ) for all t = 0 , 1 , . .
, T − 1 .
Applying Lemma C.2, we get that probability event E T − 1 implies
[Equation]
To estimate the sums in the right-hand side, we introduce new vectors:
[Equation]
otherwise , (76) for t = 0 , 1 , . . .
, T − 1 .
First of all, we point out that vectors ζ t and η t are bounded with probability 1 , i.e., with probability
[Equation]
[Equation]
for all t = 0 , 1 , . . .
, T − 1 .
Next, we notice that E T − 1 implies
[Equation]
for t = 0 , 1 , . . .
, T − 1 , i.e., probability event E T − 1 implies η t = x
t − x ∗ − γ 2 F ( (cid:101) x t ) for all t = 0 , 1 , . .
, T − 1 .
Therefore, E T − 1 implies
[Equation]
As in the monotone case, to continue the derivation, we introduce vectors θ ul , θ bl , ω ul , ω bl defined as
[Equation]
for all l = 0 , . . .
, T − 1 .
By definition we have θ l = θ ul + θ bl , ω l = ω ul + ω
bl for all l = 0 , . . .
, T − 1 .
Using the introduced notation, we continue our derivation as follows:
E T − 1 implies
[Equation]
The rest of the proof is based on deriving good enough upper bounds for ① , ② , ③ , ④ , ⑤ , ⑥ , ⑦ , ⑧ , i.e., we want to prove that ① + ② + ③ + ④ + ⑤ + ⑥ + ⑦ + ⑧ ≤ 8 R 2 with high probability.
Before we move on, we need to derive some useful inequalities for operating with θ ul , θ bl , ω ul , ω
bl .
First of all, Lemma B.2 implies that
[Equation]
for all l = 0 , 1 , . . .
, T − 1 .
Next, since { ξ i,l 1 } m 1 i =1 , { ξ i,l 2 } m 2 i =1 are independently sampled from D , we have E ξ l 1 [ F ξ l 1 ( x l )]
= F ( x l ) , E ξ l 2 [
[Equation]
ξ l 2 ( (cid:101) x l )]
= F ( (cid:101) x l ) , and
[Equation]
for all l = 0 , 1 , . . .
, T − 1 .
Moreover, as we already derived, probability event E T − 1 implies that ∥ F ( x l ) ∥ ≤ λ l / 2 and ∥ F ( (cid:101) x l ) ∥ ≤ λ l / 2 for all l = 0 , 1 , . .
, T − 1 (see (71) and (74)).
Therefore, in view of Lemma B.2, E T − 1 implies that
[Equation]
for all l = 0 , 1 , . . .
, T − 1 .
Upper bound for ① .
Since E ξ l 2 [ θ ul ] = 0 , we have
[Equation]
Next, the summands in ① are bounded with probability 1 :
[Equation] [Equation]
Moreover, these summands have bounded conditional variances σ 2 l def = E ξ
[Equation]
[Equation]
That is, sequence { 2 γ 2 ⟨ η l , θ ul ⟩} l ≥ 0 is a bounded martingale difference sequence having bounded conditional variances { σ 2 l } l ≥ 0 .
Applying Bernstein’s inequality (Lemma B.1) with X l = 2 γ 2 ⟨ η l , θ ul ⟩ , c defined in (85), b = R 2 , G =
R6ln 6( K +1) β , we get that
P (cid:40) | ① | > R 2 and
[Equation]
In other words, P { E ① } ≥ 1 − β 3( K +1) , where probability event E ① is defined as
[Equation]
either
[Equation]
or |
[Equation]
Moreover, we notice here that probability event E T − 1 implies that
[Equation]
Upper bound for ② .
Probability event E T − 1 implies
[Equation]
Upper bound for ③ .
Probability event E T − 1 implies
[Equation]
Upper bound for ④ .
We have
[Equation]
Next, the summands in ④ are bounded with probability 1 :
[Equation]
Moreover, these summands have bounded conditional variances
[Equation]
That is, sequence {∥ θ ul ∥ 2 − E ξ l 2 [ ∥ θ ul ∥ 2 ] } l ≥ 0 is a bounded martingale difference sequence having bounded conditional variances { (cid:101) σ 2 l } l ≥ 0 .
Applying Bernstein’s inequality (Lemma B.1) with X l = ∥ θ ul ∥ 2 − E ξ l 2 [ ∥ θ ul ∥ 2 ] , c defined in (91), b = R 2 , G =
[Equation] [Equation]
6( K +1) β , we get that
P (cid:40) | ④ | > R 2 and
[Equation]
In other words, P { E ④ } ≥ 1 − β 3( K +1) , where probability event E ④ is defined as
[Equation]
either
[Equation] [Equation]
Moreover, we notice here that probability event E T − 1 implies that
[Equation]
Upper bound for ⑤ .
Probability event E T − 1 implies ⑤
[Equation]
Upper bound for ⑥ .
Probability event E
[Equation]
− 1 implies
[Equation]
Upper bound for ⑦ .
We have
[Equation]
Next, the summands in ⑦ are bounded with probability 1 :
[Equation]
Moreover, these summands have bounded conditional variances
[Equation]
That is, sequence {∥ ω ul ∥ 2 − E ξ l 1 [ ∥ ω ul ∥ 2 ] } l ≥ 0 is a bounded martingale difference sequence having bounded conditional variances { (cid:98) σ 2 l } l ≥ 0 .
Applying Bernstein’s inequality (Lemma B.1) with X l =
∥ ω ul ∥ 2 − E ξ l 1 [ ∥ ω u l ∥ 2 ] , c defined in (97),
b = R 2 , G =
[Equation]
6ln 6( K +1) β , we get that
P
[Equation]
| ⑦ | > R 2 and
[Equation]
In other words, P { E ⑦ } ≥ 1 − β 3( K +1) , where probability event E ⑦ is defined as
[Equation]
either
[Equation]
Moreover, we notice here that probability event E T − 1 implies that
[Equation]
Upper bound for ⑧ .
Probability event E T
[Equation]
1 implies
[Equation]
Final derivation.
Putting all bounds together, we get that E T − 1 implies
[Equation]
Moreover, in view of (87), (93), (99), and our induction assumption, we have
[Equation]
where probability events E ① , E ④ , and E ⑦ are defined as
[Equation]
either
[Equation]
or
[Equation]
E
[Equation]
either
[Equation]
Putting all of these inequalities together
[Equation]
we obtain that probability event E T − 1 ∩ E ① ∩ E ④ ∩ E ⑦ implies
[Equation]
Moreover, union bound for the probability events implies
[Equation]
This is exactly what we wanted to prove (see the paragraph after inequality (70)).
In particular, E K + implies
[Equation]
This finishes the proof.
Corollary C.2.
Let the assumptions of Theorem C.2 hold and
[Equation]
Then, the choice of step-sizes and batch-sizes
[Equation]
satisfies conditions (65) , (67) , (68) .
With such choice of γ, m 1 , m 2 , and the choice of λ 1 , λ 2 as in (66) , the iterates produced by clipped-SEG after K iterations with probability at least 1 − β satisfy
[Equation]
In particular, to guarantee
[Equation]
probability
[Equation] least [Equation] β for some ε >clipped-SEG requires,
[Equation]
oracle
[Equation]
Proof.
First of all, we verify that the choice of γ 1 , γ 2 and m 1 , m 2 from (104) satisfies conditions (65), (67), (68).
Inequality (65) holds since
[Equation]
(104) =
γand (67), (68) are satisfied since
[Equation]
Therefore
[Equation]
applying Theorem C
[Equation] [Equation]
we derive that with probability at least 1 − β K + K
[Equation]
k
[Equation]
To guarantee 1 K +
[Equation]
Kk
[Equation]
∥ F
[Equation]
x k ) ∥ 2 ≤ ε , we choose K in such a way that the right-hand side of the above inequality is smaller than ε that gives
[Equation]
The total number of oracle calls equals
[Equation]
C.3 Quasi-Strongly Monotone Case
Lemma C.3.
Let Assumptions 1.2, 1.5 hold for Q = B 3 R ( x ∗ )
= { x ∈ R d | ∥ x − x ∗ ∥ ≤ 3 R } , where
R ≥ R 0 def = ∥ x 0 − x ∗ ∥ , and γ 1 = γ 2 = γ , 0 < γ ≤ 1 /
( L +2 µ ) .
If x k and (cid:101) x k def = x
k − γF ( x k ) lie in B 3 R ( x ∗ ) for all k = 0 , 1 , . .
, K for some K ≥ 0 , then the iterates produced by clipped-SEG satisfy
[Equation]
where θ k , ω k are defined in (15) , (16) .
Proof.
Using the update rule of clipped-SEG , we obtain
[Equation]
Since F is µ -quasi strongly monotone
[Equation]
we have
[Equation]
Moreover, − 2 γ ⟨ x k − (cid:101) x k , F ( (cid:101) x k ) ⟩ can be rewritten as
[Equation]
Putting all together
[Equation]
we get
[Equation]
where in the last step we apply 0 < γ ≤ 1 /
( L +2 µ ) .
Unrolling the recurrence, we obtain (108).
Theorem C.3.
Let Assumptions 1.1, 1.2, 1.5, hold for Q = B 3 R ( x ∗ )
= { x ∈ R d | ∥ x − x ∗ ∥ ≤ 3 R } , where R ≥ R 0 def = ∥ x 0 − x ∗ ∥ , and γ 1 = γ 2 = γ ,
[Equation]
for some K ≥ 0 and β ∈ (0 , 1] such that ln 6( K +1) β ≥ 1 .
Then, after K iterations the iterates produced by clipped-SEG with probability at least 1 − β satisfy
[Equation] [Equation]
Proof.
As in the proof of Theorem C.1, we use the following notation: R k = ∥
x k − x ∗ ∥ 2 , k ≥ 0 .
We will derive (112) by induction.
In particular, for each k = 0 , . .
, K +
we define probability event E k as follows: inequalities
[Equation]
hold for t = 0 , 1 , . . .
, k simultaneously.
Our goal is to prove that P { E k } ≥ 1 − kβ / ( K +1) for all k = 0 , 1 , . .
, K + 1 .
We use the induction to show this statement.
For k =
the statement is trivial since R 20 ≤ 2 R 2 by definition.
Next, assume that the statement holds for k = T − 1 ≤ K , i.e., we have P { E T − 1 } ≥ 1 − ( T − 1) β / ( K +1) .
We need to prove that P { E T } ≥ 1 − Tβ / ( K +1) .
First of all, since R 2 t ≤ 2 exp( − γµt )
R 2 ≤ 9 R 2 , we have x t ∈ B 3 R ( x ∗ ) .
Operator F is L -Lipschitz on
B 3 R ( x ∗ ) .
Therefore, probability event E T − 1 implies
[Equation]
and ∥
[Equation] [Equation]
for all t = 0 , 1 , . . .
, T − 1 .
Next, we show that probability event E T − 1 implies ∥ (cid:101) x
t − x ∗ ∥ ≤ 3 R and derive useful inequalities related to θ t for all t = 0 , 1 , . .
, T − 1 .
Indeed, due to Lipschitzness of F probability event E T −implies
[Equation]
) and
[Equation]
λ
[Equation]
) for all t = 0 , 1 , . .
, T − 1 .
That is, E T − 1 implies that x t , (cid:101) x t ∈ B 3 R ( x ∗ ) for all t = 0 , 1 , . .
, T − 1 .
Applying Lemma C.3 and (1 − γµ ) T ≤ exp( − γµT ) , we get that probability event E T − 1 implies
[Equation]
To estimate the sums in the right-hand side, we introduce new vectors:
[Equation]
0 , otherwise , (119) for t = 0 , 1 , . . .
, T − 1 .
First of all, we point out that vectors ζ t and η t are bounded with probability 1 , i.e., with probability
[Equation]
[Equation]
for all t = 0 , 1 , . . .
, T − 1 .
Next, we notice that E T − 1 implies ∥ F ( x t )
∥ ≤ √ 2 L exp( − γµt / 2 )
R
(due to (114)) and
[Equation]
for t = 0 , 1 , . . .
, T − 1 , i.e., probability event E T − 1 implies ζ t = F ( x t ) and η t = x t − x ∗ − γF ( (cid:101) x t ) for all t = 0 , 1 , . .
, T − 1 .
Therefore, E T − 1 implies
R
[Equation]
As in the monotone case, to continue the derivation, we introduce vectors θ ul , θ bl , ω ul , ω bl defined as
[Equation]
for all l = 0 , . . .
, T − 1 .
By definition we have θ l = θ ul + θ bl , ω l = ω ul + ω
bl for all l = 0 , . . .
, T − 1 .
Using the introduced notation, we continue our derivation as follows:
E T − 1 implies
[Equation]
The rest of the proof is based on deriving good enough upper bounds for ① , ② , ③ , ④ , ⑤ , ⑥ , ⑦ , i.e., we want to prove that ① + ② + ③ + ④ + ⑤ + ⑥ + ⑦ ≤ exp( − γµT )
R 2 with high probability.
Before we move on, we need to derive some useful inequalities for operating with θ ul , θ bl , ω ul , ω
bl .
First of all, Lemma B.2 implies that
[Equation]
124) for all l = 0 , 1 , . . .
, T − 1 .
Next, since { ξ i,l 1 } m l i =1 , { ξ i,l 2 } m l
i =1 are independently sampled from D , we have E ξ l 1 [ F ξ l 1 ( x l )]
= F ( x l ) , E ξ l 2 [ F ξ l 2 ( (cid:101) x l )]
= F ( (cid:101) x l ) , and
E
[Equation] [Equation]
for all l = 0 , 1 , . . .
, T − 1 .
Moreover, as we already derived, probability event E T − 1 implies that ∥ F ( x l ) ∥ ≤ λ l / 2 and ∥ F ( (cid:101) x l ) ∥ ≤ λ l / 2 for all l = 0 , 1 , . .
, T − 1 (see (114) and (117)).
Therefore, in view of Lemma B.2, E T − 1 implies that
[Equation]
Upper bound for ① .
Since E ξ l 1 [ ω ul ] = 0 , we have
[Equation]
Next, the summands in ① are bounded with probability 1 :
[Equation]
Moreover, these summands have bounded conditional variances
[Equation]
That is, sequence {− 4 γ 3 µ (1 − γµ ) T − 1 − l ⟨ ζ l , ω ul ⟩} l ≥ 0 is a bounded martingale difference sequence having bounded conditional variances { σ 2 l } l ≥ 0 .
Applying Bernstein’s inequality (Lemma B.1) with
X l =
[Equation] [Equation]
, c defined in (128)
[Equation]
we get that
P
[Equation]
|
[Equation]
In other words
[Equation]
P { E ① } ≥ 1 − β 3( K +1) , where probability event E ① is defined as
[Equation]
either
[Equation]
Moreover
[Equation]
we notice here that probability event E T − 1 implies that
[Equation]
Upper bound for ② .
Probability event E T − 1 implies ②
[Equation]
Upper bound for ③ .
Since E ξ l 2 [ θ ul ] = 0 , we have
[Equation]
Next, the summands in ③ are bounded with probability 1 :
[Equation]
Moreover, these summands have bounded conditional variances
[Equation] [Equation]
That is, sequence { 2 γ (1 − γµ ) T − 1 − l ⟨ η l , θ ul ⟩} l ≥ 0 is a bounded martingale difference sequence having bounded conditional variances { (cid:101) σ 2 l } l ≥ 0 .
Applying Bernstein’s inequality (Lemma B.1) with
X l = 2 γ (1 − γµ ) T − 1 − l ⟨ η l , θ ul ⟩ , c defined in (133), b = 17 exp
[Equation]
−
[Equation]
R
[Equation]
exp
[Equation]
K +
[Equation]
, we get that
P (cid:40) | ③ | > 1 7 exp( − γµT ) R 2 and
[Equation]
In other words, P { E ③ } ≥ 1 − β 3( K +1) , where probability event E ③ is defined as
E ③ =
[Equation]
either
[Equation]
or | ③ | ≤ 1 7 exp( − γµT
[Equation]
(cid:41) .
Moreover, we notice here that probability event E T − 1 implies that
[Equation] [Equation]
Upper bound for ④ .
Probability event E T − 1 implies ④
[Equation]
Upper bound for ⑤ .
Probability event E T − 1 implies
[Equation]
Upper bound for ⑥ .
First of all, we have
[Equation]
Next, the summands in ⑥ are bounded with probability 1 :
[Equation]
Moreover, these summands have bounded conditional variances
[Equation]
E
[Equation] [Equation]
[Table]
[Equation] [Equation]
That is, sequence
[Equation]
is a bounded martingale difference sequence having bounded conditional variances { (cid:98) σ 2 l } l ≥ 0 .
Applying Bernstein’s inequality (Lemma B.1) with X l
[Equation] [Equation]
−
[Equation]
, c defined in (139), b
[Equation]
, we get that
P
[Equation]
and
[Equation]
In other words
[Equation]
P { E ⑥ } ≥ 1 − β 3( K +1)
[Equation]
where probability event E ⑥ is defined as
E
[Equation]
either
[Equation]
Moreover
[Equation]
we notice here that probability event E T − 1 implies that
T
[Equation]
[Equation] [Equation]
Upper for
Probability event E T − implies
[Equation]
Final derivation.
Putting all bounds together, we get that E T − 1 implies
[Equation]
Moreover, in view of (130), (135), (141), and our induction assumption, we have
[Equation]
where probability events E ① , E ③ , and E ⑥ are defined as
[Equation]
Putting all of these inequalities together
[Equation]
we obtain that probability event E T − 1 ∩ E ① ∩ E ③ ∩ E ⑥ implies
[Equation]
Moreover
[Equation]
union bound for the probability events implies
[Equation]
This is exactly what we wanted to prove (see the paragraph after inequality (113)).
In particular, with probability at least 1 − β satisfy we have
[Equation]
which finishes the proof.
Corollary C.3.
Let the assumptions of Theorem C.3 hold.
Then, the following statements hold.
Large stepsize/large batch.
The choice of stepsize and batchsize
[Equation]
satisfies conditions (109) and (111) .
With such choice of γ, m k , and the choice of λ k as in (110) [Equation] the iterates produced by clipped-SEG after K iterations with probability at least 1 − β satisfy
[Equation]
In particular, to guarantee ∥ x K +1 − x ∗ ∥ 2 ≤ ε with probability at least 1 − β for some ε > 0 clipped-SEG requires [Equation] iterations
[Equation]
oracle
[Equation]
Small stepsize/small batch.
The choice of stepsize and batchsize
[Equation]
, m k ≡(149) satisfies conditions (109) and (111) , where B K = max
[Equation]
O 
[Equation]
max 
[Equation]
    .
With such choice of γ, m k , and the choice of λ k as in (110) , the iterates produced by clipped-SEG after K iterations with probability at least 1 − β satisfy
[Equation]
In particular, to guarantee ∥ x K +1 − x ∗ ∥ 2 ≤ ε with probability at least 1 − β for some ε > 0 clipped-SEG requires
[Equation]
(151) iterations/oracle calls, where
[Equation]
 
[Equation]
Proof.
Large stepsize/large batch.
First of all, it is easy to see that the choice of γ and m k from (145) satisfies conditions (109) and (111).
Therefore, applying Theorem C.3, we derive that with probability at least 1 − β
[Equation]
To guarantee ∥ x K +1 − x ∗ ∥ 2 ≤ ε , we choose K in such a way that the right-hand side of the above inequality is smaller than ε that gives
[Equation]
The total number of oracle calls equals
[Equation]
Small stepsize/small batch.
First of all, we verify that the choice of γ and m k from (149) satisfies conditions (109) and (111): (109) trivially holds and (111) holds since for all k = 0 , . .
K
[Equation]
Therefore
[Equation]
applying Theorem C.
[Equation]
we derive that with probability at least 1 − β ∥ x K +1 − x
[Equation]
∥
[Equation]
To guarantee ∥ x K +1 − x ∗ ∥ 2 ≤ ε
[Equation]
we choose K in such a way that the right-hand side of the above inequality is smaller than ε that gives K of the order
[Equation]
where
[Equation]
  .
The total number of oracle calls equals (cid:80)
Kk =
2 m k = 2( K + 1) .
D Clipped
Stochastic Gradient Descent-Ascent: Missing Proofs and Details D.1 Monotone Star-Cocoercive Case
Lemma D.1.
Let Assumption 1.3 hold for Q = B 2 R ( x ∗ ) , where R ≥ R 0 def = ∥ x 0 − x ∗ ∥ and 0 < γ ≤ 2 / ℓ .
If x k lies in B 2 R ( x ∗ ) for all k = 0 , 1 , . .
, K for some K ≥ 0 , then for all u ∈ B 3 R ( x ∗ ) the iterates produced by clipped-SGDA satisfy
[Equation]
[Equation]
Proof.
Using the update rule of clipped-SGDA [Equation] we obtain
[Equation]
Rearranging the terms, we derive
[Equation]
Finally
[Equation]
we sum up the above inequality for k = 0 , 1 , . . .
, K and divide both sides of the result by ( K + 1) :
[Equation]
This finishes the proof.
We also derive the following lemma, which we use in the analysis of the star-cocoercive case as well.
Lemma D.2.
Let Assumption 1.6 hold for Q = B 2 R ( x ∗ ) , where R ≥ R 0 def = ∥ x 0 − x ∗ ∥ and 0 < γ ≤ 2 / ℓ .
If x k lies in B 2 R ( x ∗ ) for all k = 0 , 1 , . .
, K for some K ≥ 0 , then the iterates produced by clipped-SGDA satisfy
[Equation]
where ω k is defined in (154) .
Proof.
Using the update rule of clipped-SGDA , we obtain
[Equation]
Since 0 < γ ≤ 2 / ℓ , we have γ ( 2 / ℓ − γ )
∥
F ( x k ) ∥ 2 ≥ 0 and, rearranging the terms, we derive
[Equation]
Finally
[Equation]
we sum up the above inequality for k = 0 , 1 , . . .
, K and divide both sides of the result by ( K + 1) :
[Equation]
This finishes the proof.
Theorem D.1.
Let Assumptions 1.1, 1.3, 1.6, hold for Q = B 2 R ( x ∗ ) , where R ≥ R 0 def = ∥ x 0 − x ∗ ∥ , and
[Equation]
for some K ≥ 0 and β ∈ (0 , 1] such that ln 6( K +1) β ≥ 1 .
Then, after K iterations the iterates produced by clipped-SGDA with probability at least 1 − β satisfy
[Equation]
Proof.
We introduce new notation: R k = ∥
x k − x ∗ ∥ for all k ≥ 0 .
The proof is based on the induction.
In particular, for each k = 0 , . .
, K +
we define the probability event E k as follows: inequalities
[Equation]
hold for t = 0 , 1 , . . .
, k simultaneously.
Our goal is to prove that P { E k } ≥ 1 − kβ / ( K +1) for all k = 0 , 1 , . .
, K + 1 .
We use the induction to show this statement.
For k =
the statement is trivial since
R 20 ≤ 2 R 2 by definition and (cid:80) − 1 l =0 ω l = 0 .
Next, assume that the statement holds for k = T ≤ K , i.e., we have P { E T } ≥ 1 − Tβ / ( K +1) .
We need to prove that P { E T +1 } ≥ 1 − ( T +1) β /
( K +1) .
Let us notice that probability event E T implies x t ∈ B 2 R ( x ∗ ) for all t = 0 , 1 , . .
, T .
This means that the assumptions of Lemma D.2 hold and we have that probability event E T implies ( γ < 1 / ℓ ) γ ℓ ( T +
[Equation]
(cid:88)
[Equation]
T ++ 2 γ
T +
T (cid:88) t = ⟨ x t − x ∗ − [Equation] ( x t ) , ω t ⟩ + γ
T +
T (cid:88) t = ∥ ω t ∥(161) and ∥ F ( x t ) ∥ (SC) ≤ ℓ
[Equation]
[Equation] [Equation]
(157) ≤ λ (162) for all t = 0 , 1 , . .
, T .
From (161) we have
R
[Equation]
≤ R 20 +
[Equation]
γ
T (cid:88) t = ⟨ x t − x ∗ − γF ( x t ) , ω t ⟩ + γ
T (cid:88) t = ∥
ω t ∥ 2 .
Next, we notice that
[Equation]
(163) for all t = 0 , 1 , . . .
, T .
Consider random vectors
[Equation]
x t ) , if ∥ x t − x ∗ − γF ( x t ) ∥ ≤ 3 R, 0 , otherwise, for all t = 0 , 1 , . . .
, T .
We notice that η t is bounded with probability 1 :
[Equation]
(164) for all t = 0 , 1 , . . .
, T .
Moreover, in view of (163), probability event E T implies η t = x
t − x ∗ − γF ( x t ) for all t = 0 , 1 , . . .
, T .
Therefore, E T implies
[Equation]
(cid:88) t [Equation] [Equation]
T (cid:88) t = ∥
[Equation] ∥ [Equation]
To continue our derivation we introduce new notation:
[Equation]
By definition we have ω t = ω ut + ω bt for all t = 0 , . .
, T .
Using the introduced notation, we continue our derivation as follows: E T implies
[Equation]
We emphasize that the above inequality does not rely on monotonicity of F .
As we notice above, E T implies x t ∈ B 2 R ( x ∗ ) for all t = 0 , 1 , . . .
, T .
This means that the assumptions of Lemma D.1 hold and we have that probability event E T implies
[Equation]
We notice that E T implies η t = x t − x ∗ − γF ( x t ) for all t = 0 , 1 , . . .
, T as well as (161) and γ < 1 / ℓ .
Therefore, probability event E T implies
[Equation]
where ① , ② , ③ , ④ , ⑤ are defined in (166).
The rest of the proof is based on deriving good enough upper bounds for ① , ② , ③ , ④ , ⑤ , i.e., we want to prove that ① + ② + ③ + ④ + ⑤ ≤ R 2 and 2 γR (cid:13)(cid:13)(cid:13)(cid:80)
Tt =0 ω t (cid:13)(cid:13)(cid:13) ≤ 2 R 2 with high probability.
Before we move on, we need to derive some useful inequalities for operating with ω ut , ω bt .
First of all,
Lemma B.2 implies that
[Equation]
for all t = 0 , 1 , . . .
, T .
Next, since { ξ i,t } mi =1 are independently sampled from D , we have E ξ t
[ F ξ t ( x t )]
= F ( x t ) , and
[Equation]
for all l = 0 , 1 , . . .
, T .
Therefore, in view of Lemma B.2, E T implies that
[Equation]
for all l = 0 , 1 , . . .
, T .
Upper bound for ① .
Since E ξ t [ ω ut ] = 0 , we have
[Equation]
Next, the summands in ① are bounded with probability 1 :
[Equation]
Moreover, these summands have bounded conditional variances σ 2 t def = E ξ t (cid:2) 4 γ 2 ⟨ η t
[Equation]
ω
[Equation] [Equation]
[Equation]
That is, sequence { 2 γ ⟨ η t , ω ut ⟩} t ≥ 0 is a bounded martingale difference sequence having bounded conditional variances { σ 2 t } t ≥ 0 .
Applying Bernstein’s inequality (Lemma B.1) with X t = 2 γ ⟨ η t , ω ut ⟩ , c defined in (172), b = R 2 5 , G =
R150ln 6( K +1) β , we get that
P (cid:40) | ① | > R 2 5 and
T (cid:88)
[Equation]
In other words, P { E ① } ≥ 1 − β 3( K +1) , where probability event E ① is defined as
[Equation]
either
[Equation] [Equation]
Moreover, we notice here that probability event E T implies that
T (cid:88)
[Equation]
Upper bound for ② .
Probability event E T implies ②
[Equation] [Equation]
Upper bound
[Equation]
Upper bound for ④ .
We have
[Equation]
Next, the summands in ④ are bounded with probability 1 :
[Equation]
Moreover, these summands have bounded conditional variances
[Equation]
That is, sequence {∥ ω ut ∥ 2 − E ξ t
∥ ω ut ∥ 2 ] } t ≥ 0 is a bounded martingale difference sequence having bounded conditional variances { (cid:101) σ 2 t } t ≥ 0 .
Applying Bernstein’s inequality (Lemma B.1) with X t = ∥ ω ut ∥ 2 − E ξ t [ ∥ ω ut ∥ 2 ] , c defined in (178), b = R 2 5 , G =
R150ln 6( K +1) β , we get that
P (cid:40) | ④ | > R 2 5 and
T (cid:88)
[Equation]
In other words, P { E ④ } ≥ 1 − β 3( K +1) , where probability event E ④ is defined as
[Equation]
either
[Equation]
Moreover, we notice here that probability event E T implies that
[Equation]
Upper bound for ⑤ .
Probability event E T implies ⑤
[Equation]
[Equation]
Upper bound for γ (cid:13) (cid:13) (cid:13)(cid:80) Tt =0 ω t (cid:13) (cid:13) (cid:13) .
To handle this term, we introduce new notation:
[Equation] [Equation]
otherwise for l = 1 , 2 , . .
, T − 1 .
By definition, we have
[Equation]
Therefore, in view of (160), probability event E T implies
[Equation]
Following similar steps as before, we bound ⑥ and ⑦ .
Upper bound for ⑥ .
Since E ξ t [ ω ut ] = 0 , we have
[Equation]
Next, the summands in ⑥ are bounded with probability 1 :
[Equation]
Moreover, these summands have bounded conditional variances (cid:98) σ 2 t def = E ξ t
[Equation]
[Equation]
ut [Equation] [Equation] That is, sequence { 2 γ ⟨ ζ t , ω ut ⟩} t ≥ 0 is a bounded martingale difference sequence having bounded conditional variances { (cid:98) σ 2 t } t ≥ 0 .
Applying Bernstein’s inequality (Lemma B.1) with X t = 2 γ ⟨ ζ t , ω ut ⟩ , c defined in (172), b = R 2 5 , G =
R150ln 6( K +1) β , we get that
P (cid:40) | ⑥ | > R 2 5 and
[Equation]
In other words, P { E ⑥ } ≥ 1 − β 3( K +1) , where probability event E ⑥ is defined as
E ⑥
[Equation]
either
[Equation] [Equation]
Moreover, we notice here that probability event E T implies that
[Equation]
Upper
E implies
[Equation]
Final derivation.
Putting all bounds together, we get that E T implies
[Equation]
Moreover, in view of (174), (180), (189), and our induction assumption, we have
[Equation]
where probability events E ① , E ④ , and E ⑥ are defined as
E ① = (cid:40) either
T (cid:88)
[Equation]
or
[Equation]
E
[Equation]
=
[Equation]
either
[Equation]
or
[Equation]
④
[Equation]
(cid:41)
[Equation]
E
[Equation]
=
[Equation]
either
[Equation]
or |
[Equation]
|
[Equation]
(cid:41) .
Putting all of these inequalities together, we obtain that probability event E T ∩ E ① ∩ E ④ ∩ E ⑥ implies
[Equation]
Moreover, union bound for the probability events implies
[Equation]
This is exactly what we wanted to prove (see the paragraph after inequality (160)).
In particular, E K implies
Gap R
[Equation]
which finishes the proof.
Corollary D.1.
Let the assumptions of Theorem D.1 hold.
Then, the following statements hold.
Large stepsize/large batch.
The choice of stepsize and batchsize
[Equation]
satisfies conditions (156) and (158) .
With such choice of γ, m , and the choice of λ as in (157) , the iterates produced by clipped-SGDA after
K
iterations with
probability at least 1 − β
satisfy
[Equation]
In particular, to guarantee Gap ( x K avg ) ≤ ε with probability at least 1 − β for some ε >clipped-SGDA requires,
[Equation]
Small stepsize/small batch.
The choice of stepsize and batchsize
[Equation]
satisfies conditions (156) and (158) .
With such choice of γ, m , and the choice of λ as in (157) , the iterates produced by clipped-SGDA after K iterations with probability at least 1 − β satisfy
[Equation]
In particular, to guarantee Gap ( x K avg ) ≤ ε with probability at least 1 − β for some ε > 0 , clipped-SGDA requires [Equation] iterations
[Equation]
oracle
[Equation]
[Equation]
Proof.
Large stepsize/large batch.
First of all, we verify that the choice of γ and m from (190) satisfies conditions (156) and (158): (156) trivially holds and (158) holds since
[Equation]
Therefore, applying Theorem D.1, we derive that with probability at least 1 − β Gap
[Equation]
K [Equation] To guarantee Gap ( x K avg ) ≤ ε , we choose K in such a way that the right-hand side of the above inequality is smaller than ε that gives
[Equation]
The total number of oracle calls equals
[Equation]
ℓRε , σ 2 Rε(cid:27) ln (cid:18) ℓRεβ (cid:19)(cid:19) .
Small stepsize/small batch.
First of all, we verify that the choice of γ and m from (194) satisfies conditions (156) and (158):
[Equation]
Therefore, applying Theorem D.1, we derive that with probability at least 1 − β
[Equation]
To guarantee Gap ( x K avg ) ≤ ε , we choose K in such a way that the right-hand side of the above inequality is smaller than ε that gives
[Equation]
The total number of oracle calls equals K + 1 .
D.2 Star-Cocoercive Case
Theorem D.2.
Let Assumptions 1.1, 1.6, hold for Q = B 2 R ( x ∗ ) , where R ≥ R 0 def = ∥ x 0 − x ∗ ∥ , and
[Equation]
for some K ≥ 0 and β ∈ (0 , 1] such that ln 4( K +1) β ≥ 1 .
Then, after K iterations the iterates produced by clipped-SGDA with probability at least 1 − β satisfy
[Equation]
Proof.
We introduce new notation: R k = ∥
x k − x ∗ ∥ for all k ≥ 0 .
The proof is based on deriving via induction that R 2 k ≤ CR 2 for some numerical constant C > 0 .
In particular, for each k = 0 , . .
, K +
we define probability event E k as follows: inequalities
[Equation]
hold for t = 0 , 1 , . . .
, k simultaneously.
Our goal is to prove that P { E k } ≥ 1 − kβ / ( K +1) for all k = 0 , 1 , . .
, K + 1 .
We notice that inequalities (161) and (166) are derived without assuming monotonicity of F .
Therefore, following exactly the same step as in the proof of Theorem D.1 (up to the replacement of ln 6( K +1) β by ln 4( K +1) β ), we get that
[Equation]
Moreover, in view of (174)
[Equation]
(180), and our induction assumption, we have
[Equation]
where probability events E ① , and E ④ are defined as
E ①
[Equation]
either
T
[Equation]
either
[Equation]
Putting all of these inequalities together
[Equation]
we obtain that probability event E T − 1 ∩ E ① ∩ E ④ implies
[Equation]
Moreover, union bound for the probability events implies
[Equation]
This is exactly what we wanted to prove (see the paragraph after inequality (201)).
In particular, E K implies
[Equation]
This finishes the proof.
Corollary D.2.
Let the assumptions of Theorem D.2 hold.
Then, the following statements hold.
Large stepsize/large batch.
The choice of stepsize and batchsize
[Equation]
satisfies conditions (197) and (199) .
With such choice of γ, m , and the choice of λ as in (198) , the iterates produced by clipped-SGDA after K iterations with probability at least 1 − β satisfy
[Equation]
In particular, to guarantee
[Equation]
x
[Equation]
∥
[Equation]
≤ ε with probability at least 1 − β for some ε > 0 clipped-SGDA requires,
[Equation]
[Equation]
Small stepsize/small batch.
The choice of stepsize and batchsize
[Equation]
satisfies conditions (197) and (199) .
With such choice of γ, m , and the choice of λ as in (198) , the iterates produced by clipped-SGDA after K iterations with probability at least 1 − β satisfy
[Equation]
In particular, to guarantee
[Equation]
probability at least 1 − β for some ε > 0 , clipped-SGDA requires
[Equation]
iterations/oracle calls
[Equation]
Proof.
Large stepsize/large batch.
First of all, we verify that the choice of γ and m from (203) satisfies conditions (197) and (199): (197) trivially holds and (199) holds since
[Equation]
Therefore, applying Theorem D.2, we derive that with probability at least 1 − β K
[Equation]
[Equation]
K
[Equation]
To guarantee 1 K +
[Equation]
{+Kk =
 ∥ F (+} x k ) ∥ 2 ≤ ε , we choose K in such a way that the right-hand side of the above inequality is smaller than ε that gives
[Equation]
The total number of oracle calls equals
[Equation]
Small stepsize/small batch.
First of all, we verify that the choice of γ and m from (207) satisfies conditions (197) and (199): [Equation] R
[Equation]
Therefore, applying Theorem D.2, we derive that with probability at least 1 − β
K +
K (cid:88) k
= ∥
F ( x k ) ∥
[Equation]
To guarantee 1 K +
[Equation]
∥ F ( x k ) ∥ 2 ≤ ε , we choose K in such a way that the right-hand side of the above inequality is smaller than ε that gives
K =
O
[Equation]
ln
[Equation]
(cid:19)(cid:27)(cid:19) .
The total number of oracle calls equals K + 1 .
D.3 Quasi-Strongly Monotone Star-Cocoercive Case
Lemma D.3.
Let Assumptions 1.5, 1.6 hold for Q = B 2 R ( x ∗ ) , where R ≥ R 0 def = ∥ x 0 − x ∗ ∥ , and 0 < γ ≤ 1 / ℓ .
If x k lies in B 2 R ( x ∗ ) for all k = 0 , 1 , . .
, K for some K ≥ 0 , then the iterates produced by clipped-SGDA satisfy
[Equation]
where ω k is defined in (154) .
Proof.
Using the update rule of clipped-SGDA , we obtain ∥
[Equation]
Unrolling the recurrence, we obtain (210)
[Equation]
Theorem D.3.
Let Assumptions 1.1, 1.5, 1.6 hold for Q = B 2 R ( x ∗ )
= { x ∈ R d | ∥ x − x ∗ ∥ ≤ 2 R } , where R ≥ R 0 def = ∥ x 0 − x ∗ ∥ , and
[Equation]
for some K ≥ 0 and β ∈ (0 , 1] such that ln 4( K +1) β ≥ 1 .
Then, after K iterations the iterates produced by clipped-SGDA with probability at least 1 − β satisfy
[Equation]
Proof.
As in the proof of Theorem D.1, we use the following notation: R k = ∥
x k − x ∗ ∥ 2 , k ≥ 0 .
We will derive (214) by induction.
In particular, for each k = 0 , . .
, K +
we define probability event E k as follows: inequalities
[Equation]
(215) hold for t = 0 , 1 , . .
k simultaneously.
Our goal is to prove that P { E k } ≥ 1 − kβ / ( K +1) for all k = 0 , 1 , . .
, K + 1 .
We use the induction to show this statement.
For k =
the statement is trivial since R 20 ≤ 2 R 2 by definition.
Next, assume that the statement holds for k = T ≤ K , i.e., we have P { E T } ≥ 1 − Tβ / ( K +1) .
We need to prove that P { E T +1 } ≥ 1 − ( T +1) β /
( K +1) .
First of all, since
R 2 t ≤ 2 exp( − γµt ) R 2 ≤ 2 R 2 , we have x t ∈ B 2 R ( x ∗ ) .
Operator F is ℓ -star-cocoercive on B 2 R ( x ∗ ) .
Therefore, probability event E T implies
[Equation]
[Equation]
and
∥ ω t ∥
[Equation]
x
[Equation]
for all t = 0 , 1 , . . .
, T .
Applying Lemma D.3 and (1 − γµ ) T ≤ exp( − γµT ) , we get that probability event E T implies
[Equation]
To estimate the sums in the right-hand side, we introduce new vectors:
[Equation]
otherwise ,
[Equation]
) for t = 0 , 1 , . .
, T .
First of all, we point out that vector η t is bounded with probability 1 , i.e., with probability
[Equation]
[Equation]
for all t = 0 , 1 , . . .
, T .
Next, we notice that E T implies ∥ F ( x t )
∥ ≤ √ 2 ℓ exp( − γµt / 2 )
R
(due to (216)) for t = 0 , 1 , . . .
, T , i.e., probability event E T implies η t = x
t − x ∗ − γF ( x t ) for all t = 0 , 1 , . . .
, T .
Therefore, E T implies
[Equation]
As in the monotone case, to continue the derivation, we introduce vectors ω ut , ω bt defined as
[Equation]
for all t = 0 , . . .
, T .
By definition we have ω t = ω ut + ω bt for all t = 0 , . .
, T .
Using the introduced notation, we continue our derivation as follows: E T implies
[Equation]
The rest of the proof is based on deriving good enough upper bounds for ① , ② , ③ , ④ , ⑤ , i.e., we want to prove that ① + ② + ③ + ④ + ⑤ ≤ exp( − γµT )
R 2 with high probability.
Before we move on, we need to derive some useful inequalities for operating with ω u t , ω b t .
First of all,
Lemma B.2 implies that
[Equation]
∥ ≤
[Equation] [Equation]
for all t = 0 , 1 , . . .
, T .
Next, since { ξ i,t } m t i =1 are independently sampled from D , we have E ξ t
[ F ξ t ( x t )]
= F ( x t ) , and
[Equation]
for all l = 0 , 1 , . . .
, T .
Moreover, as we already derived, probability event E T implies that ∥ F ( x t ) ∥ ≤ λ t / 2 for all t = 0 , 1 , . . .
, T (see (216)).
Therefore, in view of Lemma B.2, E T implies that
[Equation]
for all l = 0 , 1 , . . .
, T .
Upper bound for ① .
Since E ξ t [ ω ut ] = 0 , we have
[Equation]
Next, the summands in ① are bounded with probability 1 :
[Equation]
Moreover, these summands have bounded conditional variances
[Equation]
That is, sequence { 2 γ (1 − γµ ) T − t ⟨ η t , ω ut ⟩} t ≥ 0 is a bounded martingale difference sequence having bounded conditional variances { σ 2 t } t ≥ 0 .
Applying Bernstein’s inequality (Lemma B.1) with X t = 2 γ (1 − γµ ) T − t ⟨ η t , ω ut ⟩ , c defined in (226), b = 15 exp( − γµT )
R 2 , G = exp( −
[Equation]
) R
[Equation]
4( K +1) β , we get that
P
[Equation]
and
[Equation]
In other words, P { E ① } ≥ 1 − β 2( K +1) , where probability event E ① is defined as
E
[Equation]
either
T (cid:88)
[Equation]
Moreover, we notice here that probability event E T implies that
[Equation]
Upper bound for ② .
Probability event E T implies
[Equation]
Upper bound for ③ .
Probability event E T implies
[Equation]
Upper bound for ④ .
First of all, we have
[Equation]
Next, the summands in ④ are bounded with probability 1 :
[Equation]
16 γ
[Equation]
Moreover, these summands have bounded conditional variances
[Equation]
That is, sequence (cid:8) 2 γ 2 (1 − γµ ) T − t (cid:0) ∥
ω ut ∥ 2 − E ξ t (cid:2) ∥
ω ut ∥ 2 (cid:3)(cid:1)(cid:9) t ≥ 0 is a bounded martingale difference sequence having bounded conditional variances { (cid:101) σ 2 t } t ≥ 0 .
Applying Bernstein’s inequality (Lemma B.1) with X t = 2 γ 2 (1 − γµ ) T − t (cid:0) ∥
ω ut ∥ 2 − E
[Equation]
t (cid:2) ∥
ω ut ∥ 2 (cid:3)(cid:1) , c defined in (232), b = 15 exp( − γµT )
R 2 , G = exp( − 2 γµT )
R
[Equation]
( K
[Equation] [Equation]
, we get that
P (cid:40) | ④ | > 1 5 exp( − γµT ) R 2 and
T
[Equation]
In other words, P { E ④ } ≥ 1 − β 2( K +1) , where probability event E ④ is defined as
[Equation]
either
[Equation]
Moreover, we notice here that probability event E T implies that
[Equation]
Upper bound for ⑤ .
Probability event E T implies
[Equation]
Final derivation.
Putting all bounds together
[Equation]
we get that E T implies
[Equation]
Moreover, in view of (228), (234), and our induction assumption, we have
[Equation]
where probability events E ①
[Equation]
and E ④ are defined as
[Equation]
either
[Equation]
Putting all of these inequalities together
[Equation]
we obtain that probability event E T ∩ E ① ∩ E ④ implies
[Equation]
Moreover, union bound for the probability events implies
[Equation]
This is exactly what we wanted to prove (see the paragraph after inequality (215)).
In particular, with probability at least 1 − β we have
[Equation]
which finishes the proof
[Equation]
Corollary D.3.
Let the assumptions of Theorem D.3 hold.
Then, the following statements hold.
Large stepsize/large batch.
The choice of stepsize and batchsize
[Equation]
satisfies conditions (211) and (213) .
With such choice of γ, m k , and the choice of λ k as in (212) , the iterates produced by clipped-SGDA after K iterations with probability at least 1 − β satisfy
[Equation]
In particular, to guarantee ∥ x K +1 − x ∗ ∥ 2 ≤ ε with probability at least 1 − β for some ε > 0 clipped-SGDA requires
[Equation]
oracle
[Equation]
[Equation]
Small stepsize/small batch.
The choice of stepsize and batchsize
[Equation]
satisfies conditions (211) and (213) , where B K = max (cid:26) 2 ,
[Equation]
K + [Equation] [Equation] O 
[Equation]

[Equation]
    .
With such choice of γ, m k , and the choice of λ k as in (212) , the iterates produced by clipped-SGDA after K iterations with probability at least 1 − β satisfy
[Equation]
 [Equation] [Equation] In particular, to guarantee ∥ x K +1 − x ∗ ∥ 2 ≤ ε with probability at least 1 − β for some ε > 0 clipped-SGDA requires
[Equation]
(244) iterations/oracle calls, where
[Equation]   .
Proof.
Large stepsize/large batch.
First of all, it is easy to see that the choice of γ and m k from (238) satisfies conditions (211) and (213).
Therefore, applying Theorem D.3, we derive that with probability at least 1 − β
[Equation]
To guarantee ∥ x K +1 − x ∗ ∥ 2 ≤ ε , we choose K in such a way that the right-hand side of the above inequality is smaller than ε that gives
[Equation]
The total number of oracle calls equals
K (cid:88) k
= m k (
[Equation]
=
[Equation]
Small stepsize/small batch.
First of all, we verify that the choice of γ and m k from (242) satisfies conditions (211) and (213): (211) trivially holds and (213) holds since for all k = 0 , . . .
, K
[Equation]
Therefore, applying Theorem D.3, we derive that with probability at least 1 − β ∥
x K +1 −
[Equation]
To guarantee ∥ x K +1 − x ∗ ∥ 2 ≤ ε , we choose K in such a way that the right-hand side of the above inequality is smaller than ε that gives K of the order
O
[Equation]
where
B
[Equation]
The total number of oracle calls equals (cid:80) Kk =
m k
= ( K + 1) .
E Extra Experiments
In this section, we provide more details for the experiments done in § 4, as well as additional tables, figures, and image samples from some of our trained models.
E.1 WGAN-GP
In all cases, everything in the experimental setup other than learning rates and clip values remained constant.
We use the same ResNet architectures and training parameters specified in Gulrajani et al.
[2017]: the gradient penalty coefficient λ GP = 10 , n dis = 5 where n dis is the number of discriminator steps for every generator step, and a learning rate decayed linearly to 0 over 100k steps.
The only exception is we double the feature map of the generator from 128 to 256 dimensions.
For all stochastic extragradient ( SEG ) methods, we use the ExtraSGD implementation provided by Gidel et al.
[2019a].
We alternate between exploration and update steps and do not treat the exploration steps as “free” – this means we only have 50k parameter updates as opposed to 100k for all SGDA methods (we decay the learning rate twice as fast such that it still reaches 0 after 50k parameter updates).
All of the hyperparameter sweeps performed for SGDA , clipped-SGDA , clipped-SEG , clippedSGDA (coordinate), and clipped-SEG (coordinate), as well as the associated best FID score obtained within the first 35k training steps, can be found in Tables 2, 6, 5, 6, and 7 respectively.
Bold rows denote the hyperparameters that were trained for the full 100k steps and are henceforth referred to as the “best models” .
For each of the methods tested, additional samples for the best models trained can be found in Figures 7, 8, 9, 10, & 11.
We also plot the evolution of the gradient noise histograms in
Figures 12, 13, 14, 15, & 16.
We emphasize that our goal is not to get the best possible FID score (e.g. are often able to obtain marginally better FIDs by training for longer), but rather to compare the systematic differences in performance between the various unclipped and clipped methods.
Therefore, log-space hyperparameter sweeps are appropriate for our experiments and we do not tune further.
E.2 StyleGAN
We train on FFHQ downsampled to 128 × 128 pixels, and use the recommended StyleGAN hyperparameter configuration for this resolution: batch size = 32 , γ = 0 .
1024 , map depth = 2 , and channel multiplier = 16384 .
For both SGDA and clipped-SGDA , we sweep over a (roughly) log-scale of learning rates and clipping values; a summary of the hyperparmaters and best FID scores obtained Table 8 and Table 9 respectively.
Based on the results in Table 9, the best hyperparameters are lr= 0 .
35 and clip= 0 .
0025 which we then used to train our “best model” .
We trained for longer, and decayed the learning rate twice (by a factor of × 10 ) when the FID plateaued or worsened.
The best schedule we found was to scale the learning rate by × 0 .
1 after 6000 kimgs (thousands of real images shown to the discriminator), by another × 0 . 1 after 3600 kimgs, and then train until the FID begins increasing (for another 8000 kimgs) – for a total of 17600 kimgs.
We did not explore different scale factors or other schedules (such as cosine annealing).
Additional samples for this model can be found in Figure 18(a).
In general, we observe that every SGDA -trained model for the wide range of learning rates we tested failed to improve the FID, while models trained with clipped-SGDA (with appropriately set hyperparameters) are generally able to learn some meaningful features and improve the FID.
We show this behaviour in Figure 17 – the FID scores for SGDA -trained models fluctuate around 320 and only generate noise such as the samples shown in Figure 18(b), which is in contrast to models trained with clipped-SGDA .
Note that the range of the hyperparameter sweep is fairly narrow and favourable for clipped-SGDA , while being quite wide for SGDA .
The purpose for these parameter ranges is not to directly compare the parameter sweeps (which would unfairly favour clipped-SGDA ), but to show that in general SGDA fails, while clipped-SGDA is capable of learning.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Figure] [Figure]
[Figure]
G e n e r a t o r   9 ||g(x)−∇f G (x)|| 2 0.
[Table] [Table]
[Table] [Table]
D e n s i t y
[Table]
x
[Table]
||
[Table]
n
[Table]
t y
[Table]
[Figure] [Figure]
[Figure]
D
[Figure]
c r i m i n a t
o r
[Figure] [Figure]
[Table] [Table]
[Table]
[Figure]