ODE A NALYSIS OF S TOCHASTIC G RADIENT M ETHODS WITH O PTIMISM AND A NCHORING FOR M INIMAX P ROBLEMS AND GAN S
Anonymous authors
Paper under double-blind review
A BSTRACT
Despite remarkable empirical success, the training dynamics of generative adversarial networks (GAN), which involves solving a minimax game using stochastic gradients, is still poorly understood.
In this work, we analyze last-iterate convergence of simultaneous gradient descent (simGD) and its variants under the assumption of convex-concavity, guided by a continuous-time analysis with differential equations.
First, we show that simGD, as is, converges with stochastic sub-gradients under strict convexity in the primal variable.
Second, we generalize optimistic simGD to accommodate an optimism rate separate from the learning rate and show its convergence with full gradients.
Finally, we present anchored simGD, a new method and show convergence with stochastic subgradients
I NTRODUCTION
Training of generative adversarial networks (GAN) (Goodfellow et al., 2014), solving a minimax game using stochastic gradients, is known to be difﬁcult.
Despite the remarkable empirical success of GANs, further understanding the global training dynamics empirically and theoretically is considered a major open problem (Goodfellow, 2016; Radford et al., 2016;
Metz et al., 2017; Mescheder et al., 2018; Odena, 2019).
The local training dynamics of GANs is understood reasonably well.
Several works have analyzed convergence assuming the loss functions have linear gradients and assuming the training uses full (deterministic) gradients.
Although the linear gradient assumption is reasonable for local analysis (even though the loss functions may not be continuously differentiable due to ReLU activation functions) such results say very little about global convergence.
Although the full gradient assumption is reasonable when the learning rate is small, such results say very little about how the randomness affects the training.
This work investigates global convergence of simultaneous gradient descent (simGD) and its variants for zero-sum games with a convex-concave cost using using stochastic sub gradients.
We speciﬁcally study convergence of the last iterates as opposed to the averaged iterates.
Organization.
Section 2 presents convergence of simGD with stochastic subgradients under strict convexity in the primal variable.
The goal is to establish a minimal sufﬁcient condition of global convergence for simGD without modiﬁcations.
Section 3 presents a generalization of optimistic simGD (Daskalakis et al., 2018), which allows an optimism rate separate from the learning rate.
We prove the generalized optimistic simGD using full gradients converges, and experimentally demonstrate that the optimism rate must be tuned separately from the learning rate when using stochastic gradients.
However, it is unclear whether optimistic simGD is theoretically compatible with stochastic gradients.
Section 4 presents anchored simGD, a new method, and presents its convergence with stochastic subgradients.
Anchoring represents what we consider to be the strongest contribution of this work.
The presentation and analyses of Sections 2, 3, and 4 are guided by continuous-time ﬁrstorder ordinary differential equations (ODE).
In particular, we interpret optimism and anchoring as discretizations of certain regularized dynamics.
Section 5 experimentally demonstrates the beneﬁt of optimism and anchoring for training GANs in some setups.
Prior work.
There are several independent directions for improving the training of GANs such as designing better architectures, choosing good loss functions, or adding appropriate regularizers (Radford et al., 2016; Arjovsky et al., 2017;
Sønderby et al., 2017; Arjovsky & Bottou, 2017; Gulrajani et al., 2017; Wei et al., 2018; Roth et al., 2017; Mescheder et al., 2018; 2017; Miyato et al., 2018).
In this work, we accept these factors as a given and focus on how to train (optimize) the model effectively.
Optimism is a simple modiﬁcation to remedy the cycling behavior of simGD, which can occur even under the bilinear convex-concave setup (Daskalakis et al., 2018; Daskalakis & Panageas, 2018; 2019; Mertikopoulos et al., 2019; Gidel et al., 2019a; Liang & Stokes, 2019; Mokhtari et al., 2019; Peng et al., 2019).
These prior work assume the gradients are linear and use full gradients.
Although the recent name ‘optimism’ originates from its use in online optimization (Chiang et al., 2012; Rakhlin & Sridharan, 2013a;b; Syrgkanis et al., 2015), the idea dates back to Popov’s work in the 1980s (Popov, 1980) and has been studied independently in the mathematical programming community (Malitsky & Semenov, 2014; Malitsky, 2015; Malitsky & Tam, 2018; Malitsky, 2019;
Csetnek et al., 2019).
We note that there are other mechanisms similar to optimism and anchoring such as “prediction” (Yadav et al., 2018), “negative momentum” (Gidel et al., 2019b), and “extragradient” (Korpelevich, 1976; Tseng, 2000; Chavdarova et al., 2019).
In this work, we focus on optimism and anchoring.
Classical literature analyze convergence of the Polyak-averaged iterates (which assigns less weight to newer iterates) when solving convex-concave saddle point problems using stochastic subgradients (Bruck, 1977; Nemirovski & Yudin, 1978;
Nemirovski et al., 2009; Juditsky et al., 2011; Gidel et al., 2019a).
For GANs, however, last iterates or exponentially averaged iterates (Yazıcı et al., 2019) (which assigns more weight to newer iterates) are used in practice.
Therefore, the classical work with Polyak averaging do not fully explain the empirical success of GANs.
We point out that we are not the ﬁrst to utilize classical techniques for analyzing the training of
GANs.
In particular, the stochastic approximation technique (Heusel et al., 2017; Duchi & Ruan, 2018), control theoretic techniques (Heusel et al., 2017; Nagarajan & Kolter, 2017), ideas from variational inequalities and monotone operator theory (Gemp & Mahadevan, 2018; Gidel et al., 2019a), and continuous-time ODE analysis (Heusel et al., 2017; Csetnek et al., 2019) have been utilized for analyzing GANs.
S TOCHASTIC SIMULTANEOUS SUBGRADIENT DESCENT
Consider the cost function L : R m × R n → R and the minimax game min x max u L ( x, u ) .
We say ( x (cid:63) , u (cid:63) ) ∈
R m × R n is a solution to the minimax game or a saddle point of L if
[Equation]
We assume
L is convex-concave and has a saddle point .
[Equation]
A
[Equation]
By convex-concave, we mean L ( x, u ) is a convex function in x for ﬁxed u and a concave function in u for ﬁxed x . Deﬁne
[Equation]
, where ∂ x and ∂ u respectively denote the subdifferential with respect to x and u .
For simplicity, write z = ( x, u ) ∈
R m + n and G ( z ) =
G ( x, u ) .
Note that 0 ∈ G ( z ) if and only if z is a saddle point.
Since L is convex-concave, the operator G is monotone (Rockafellar, 1970):
[Equation]
Let g ( z ; ω ) be a stochastic subgradient oracle, i.e., E ω g
( z ; ω ) ∈ G ( z ) for all z ∈ R m + n , where ω is a random variable.
Consider Simultaneous Stochastic Sub-Gradient Descent
[Equation]
for k = 0 , 1 , . . .
, where z 0 ∈ R m + n is a starting point, α 0 , α 1 , . .
. are positive learning rates, and ω 0 , ω 1 , . .
are IID random variables.
(We read SSSGD as “triple-SGD”.)
In this section, we provide convergence of SSSGD when L ( x, u ) is strictly convex in x .
[Figure]
C ONTINUOUS - TIME ILLUSTRATION
To understand the asymptotic dynamics of the stochastic discrete-time system, we consider a corresponding deterministic continuous-time system.
For simplicity, assume G is single-valued and smooth.
Consider
[Equation]
with an initial value z (0) =
z 0 .
(We introduce g ( t ) for notational simplicity.)
Let z (cid:63) be a saddle point, i.e., G ( z (cid:63) )
= 0 .
Then z ( t ) does not move away from z (cid:63) :
[Equation]
where we used (1).
However, there is no mechanism forcing z ( t ) to converge to a solution.
Consider the two examples L 0 ( x, u ) =
xu and L ρ ( x, u )
=
( ρ/ 2)
x 2 + xu with
[Equation]
where x ∈ R and u ∈ R and ρ >
Note that L 0 is the canonical counter example that also arises as the Dirac-GAN (Mescheder et al., 2018).
See Figure 1.
The classical LaSalle–Krasnovskii invariance principle (Krasovskii, 1959; LaSalle, 1960) states (paraphrased) if z ∞ is a cluster point of z ( t ) , then the dynamics starting at z ∞ will have a constant distance to z (cid:63) .
On the left of Figure 1, we can see (cid:107) z ( t ) − z
(cid:63) (cid:107) 2 is constant as ddt 12 (cid:107)
z ( t ) − z (cid:63) (cid:107) 2 =for all t .
On the right of Figure 1, we can see that although ddt 12 (cid:107) z ( t ) − z (cid:63) (cid:107) 2 = 0 when z ( t ) =
(0 , u ) for u (cid:54) =
(the dotted line)
this 0 derivative is temporary as z ( t ) will soon move past the dotted line.
Therefore, z ( t ) can maintain a constant constant distance to z (cid:63) only if it starts at 0 , and 0 is the only cluster point of z ( t ) .
2. D ISCRETE - TIME CONVERGENCE ANALYSIS
Consider the further assumptions
[Equation]
where ω 1 and ω 2 are independent random variables and R 1 ≥ 0 and R 2 ≥ 0 .
These assumptions are standard in the sense that analogous assumptions are used in convex minimization
Theorem 1.
Assume (A0) , (A1) , and (A2) .
Furthermore, assume L ( x, u ) is strictly convex in x for all u .
Then SSSGD converges in the sense of z k a.s.
→ z (cid:63) where z (cid:63) is a saddle point of L .
We can alternatively assume L ( x, u ) is strictly concave in u for all x and obtain the same result.
The proof uses the stochastic approximation technique of (Duchi & Ruan, 2018).
We show that the discrete-time process converges (in an appropriate topology) to a continuous-time trajectory satisfying a differential inclusion and use the LaSalle–Krasnovskii invariance principle to argue that cluster points are solutions.
Related prior work.
Theorem 3.1 of (Mertikopoulos et al., 2019) considers the more general mirror descent setup and proves convergence under the assumption of “strict coherence”, which is analogous to the stronger assumption of strict convex-concavity in both x and u .
S IMULTANEOUS GD WITH OPTIMISM
Consider the setup where L is continuously differentiable and we access full (deterministic) gradients
[Equation]
Consider Optimistic Simultaneous Gradient Descent
[Equation]
for k ≥ 0 , where z 0 ∈ R m + n is a starting point, z − 1 =
z 0 , α > 0 is learning rate, and β > 0 is the optimism rate .
Optimism is a modiﬁcation to simGD that remedies the cycling behavior; for the bilinear example L 0 of (2), simGD (case β = 0 ) diverges while SimGD-O with appropriate β >converges.
In this section, we provide a continuous-time interpretation of SimGD-O as a regularized dynamics and provide convergence for the deterministic setup.
3. C ONTINUOUS - TIME ILLUSTRATION
Consider the continuous-time dynamics
[Equation]
The discretization ˙ z ( t ) ≈
z k +1 − z k and ˙ g ( t )
≈ G ( z k )
− G ( z k − 1 ) yields
SimGD-O.
We discuss how this system arises as a certain regularized dynamics and derive the convergence rate
[Equation]
g
[Equation]
O
[Equation]
Regularized gradient mapping.
The Moreau–Yosida (Moreau, 1965; Yosida, 1948) regularization of G with parameter β > 0 is
[Equation]
To clarify, I : R m + n → R m + n is the identity mapping and ( I + βG ) − 1 is the inverse (as a function) of I + βG , which is well-deﬁned by Minty’s theorem (Minty, 1962).
It is straightforward to verify that G β ( z ) =
if and only if G ( z ) = 0 , i.e., G β and G share the same equilibrium points.
For small β , we can think of G β as an approximation G that is better-behaved.
Speciﬁcally, G is merely monotone (satisﬁes (1)), but G β is furthermore β -cocoercive, i.e.,
[Equation]
Regularized dynamics.
Consider the regularized dynamics
[Equation]
Reparameterize the dynamics ˙ ζ ( t ) = − αG β ( ζ ( t )) with z ( t )
=
( I + βG ) − 1 ( ζ ( t )) and g ( t ) =
G ( z ( t )) to get ζ ( t ) =
z ( t ) + βg ( t ) and ˙
z ( t )
+ β ˙ g ( t )
= ˙
ζ ( t ) = − α β ( ζ ( t ) − z ( t )) =
− αg ( t ) .
This gives us ˙ z ( t ) =
− αg ( t ) − β ˙ g ( t ) .
Rate of convergence.
We now derive a rate of convergence.
Let z (cid:63) satisfy G ( z (cid:63) )
=
(and therefore G β ( z (cid:63) ) = 0 ).
Then
[Equation]
where we use cocoercivity, (3).
This translates to
[Equation]
The quantity (cid:107) g ( t ) (cid:107) 2 is nonincreasing since
[Equation]
where we use cocoercivity
[Equation]
Finally
[Equation]
integrating (4) on both sides gives us
[Equation]
Related prior work.
The use of the Moreau–Yoshida regularization for the continuous-time analysis was inspired by Attouch et al.
(Attouch et al., 2002; Attouch & Peypouquet, 2019) who ﬁrst used the Moreau–Yosida regularization in continuous-time dynamics and Csetnek et al.
(2019) who interpreted a forward-backward-forward-type method as a discretization of continuous-time dynamics with the Douglas–Rachford operator.
Daskalakis et al.
(2018) interprets optimism as augmenting “follow the regularized leader” with the (optimistic) prediction that the next gradient will be the same as the current gradient in online learning setup.
Peng et al.
(2019) interprets optimism as “centripetal acceleration” but does not provide a formal analysis with differential equations.
3. D ISCRETE - TIME CONVERGENECE ANALYSIS
The discrete-time method SimGD-O converges under the assumption
L is differentiable and ∇ L is R -Lipschitz continuous.
(A
[Equation]
Theorem 2.
Assume (A0) and (A3) .
If 0 < α < 2 β (1 − 2 βR ) , then SimGD-O converges in the sense of
[Equation]
Furthermore, z k → z (cid:63) , where z (cid:63) is a saddle point of L .
The proof can be considered a discretization of the continuous-time analysis.
We further discuss the similarities and differences between the continuous and discrete analyses in Section A.
Corollary 1.
In the setup of Theorem 2, the choice α = 1 / (8 R ) and β =
α yields
[Equation]
Related prior work.
Peng et al.
(2019) show convergence of simGD-O for α (cid:54) =
β and bilinear
L .
Malitsky & Tam (2018) and Csetnek et al.
(2019) show convergence of simGD-O for α = β and convex-concave L .
Theorem 2 establishes convergence for α (cid:54) =
β and convex-concave L and presents an explicit rate.
D IFFICULTY WITH STOCHASTIC GRADIENTS
Training in machine learning usually relies on stochastic gradients, rather than full gradients.
We can consider a stochastic variation of SimGD-O:
[Equation]
with learning rate α k and optimism rate β k .
Figure 2 presents experiments of SimGD-OS on a simple bilinear problem.
The choice β k
= α k where α k → 0 does not lead to convergence.
Discretizing ˙
z ( t ) =
− αg ( t ) − β ˙ g ( t ) with a diminishing step h k leads to the choice α k = αh k and β k = β , but this choice does not lead to convergence either.
Rather, it is necessary to tune α k and β k separately as in Theorem 2 to obtain convergence and dynamics appear to be sensitive to the choice of α k and β k .
In particular, both α k and β k must diminish and α k must diminish faster than β k .
One explanation of this difﬁculty is that the ﬁnite difference approximation α − 1 k ( g ( z k ; ω k ) − g
( z k − 1 ; ω k − 1 ))
≈ ˙ g ( t ) is unreliable when using stochastic gradients.
Whether the observed convergence holds generally in the nonlinear convex-concave setup and whether optimism is compatible with subgradients is unclear.
This motivates anchoring of the following section which is provably compatible with stochastic subgradients.
Related prior work.
Gidel et al.
(2019a) show averaged iterates of SimGD-OS converge if iterates are projected onto a compact set.
Mertikopoulos et al.
(2019) show almost sure convergence of
SimGD-OS under strict convex-concavity (and more generally under “strict coherence”).
However, such analyses do not provide a compelling reason to use optimism since SimGD without optimism already converges under these setups.
S IMULTANEOUS GD WITH ANCHORING
Consider setup of Section 3.
We propose Anchored Simultaneous Gradient Descent
[Equation]
for k ≥ 0 , where z 0 ∈ R m + n is a starting point, p ∈ (1 / 2 , 1) , and γ > 0 is the anchor rate .
In this section, we provide a continuous-time illustration of SimGD-A and provide convergence for both the deterministic and stochastic setups.
4. C ONTINUOUS - TIME ILLUSTRATION
Consider the continuous-time dynamics
[Equation]
for
[Equation]
Discretizing the continuous-time ODE with diminishing steps (1 − p ) /
( k + 1) p leads to SimGD-A.
Rate of convergence. First note
[Equation]
Using this, we have
[Equation]
Using γ ≥ 1 , we have
[Equation]
Multiplying by t 2 and integrating both sides gives us
[Equation]
Reorganizing, we get
[Equation]
Using γ ≥ 1 , the monotonicity inequality, and Young’s inequality, we get
[Equation]
and conclude
[Equation]
Interestingly, anchoring leads to a faster rate O (1 /t 2 ) compared to the rate O (1 /t ) of optimism in continuous time.
The discretized method, however, is not faster than O (1 /k ) .
We further discuss this difference in Section A.
Related prior work.
Anchoring was inspired by Halpern’s method (Halpern, 1967; Wittmann, 1992; Lieder, 2017) and James–Stein estimator (Stein, 1956; James & Stein, 1961); these methods pull/shrink the iterates/estimator towards a speciﬁed point z 0 .
4. D ISCRETE - TIME CONVERGENECE ANALYSIS
We now present convergence results with anchoring.
In Theorem 3, we use deterministic gradients, and in Theorem 4, we use stochastic sub gradients.
Theorem 3.
Assume (A0) and (A3) .
If p ∈ (1 / 2 , 1) and γ ≥ 2 , then SimGD-A converges in the sense of
[Equation]
The proof can be considered a discretization of the continuous-time analysis.
Consider the setup of Section 2.
We propose Anchored Simultaneous Stochastic SubGradient Descent
[Equation]
(The small ε > 0 is introduced for the proof of Theorem 4.
See Section A for further discussion.)
Theorem 4.
Assume (A0) and (A2) .
If p ∈ (1 / 2 , 1) , ε ∈ (0 , 1 / 2) , and γ > 0 , then SSSGD-A converges in the sense of z k L 2 → z (cid:63) , where z (cid:63) is a saddle point.
(To clarify, we do not assume L is differentiable.)
Main contribution.
To the best of our knowledge, Theorem 4 is the ﬁrst result establishing lastiterate convergence for convex-concave cost functions using stochastic subgradients without assuming strict convexity or analogous assumptions.
E XPERIMENTS
In this section, we experimentally demonstrate the effectiveness of optimism and anchoring for training GANs.
We train Wasserstein-GANs (Arjovsky et al., 2017) with gradient penalty (Gulrajani et al., 2017) on the MNIST and CIFAR-10 dataset and plot the Fr´echet Inception Distance (FID) (Heusel et al., 2017; Lucic et al., 2018).
The experiments were implemented in PyTorch (Paszke et al., 2017).
We combine Adam with optimism and anchoring (described precisely in Appendix G) and compare it against the baseline Adam optimizer (Kingma & Ba, 2015).
The generator and discriminator architectures and the hyperparameters are described in Appendix G. For optimistic and anchored Adam, we roughly tune the optimism and anchor rates and show the curve corresponding to the best parameter choice.
Figure 4 shows that the MNIST setup beneﬁts from anchoring but not from optimism, while the
CIFAR-10 setup beneﬁts from optimism but not from anchoring.
We leave comparing the effects of optimism and anchoring in practical GAN training (where the cost function is not convex-concave) as a topic of future work.
C ONCLUSION
In this work, we analyzed the convergence of SSSGD, Optimistic simGD and Anchored SSSGD.
Under the assumption that the cost L is convex-concave, Anchored SSSGD provably converges under the most general setup.
Through experiments, we showed that the practical GAN training beneﬁts from optimism and anchoring in some (but not all) setups.
Generalizing these results to accommodate projections and proximal operators, analogous to projected and proximal gradient methods, is an interesting direction of future work.
Weight clipping (Arjovsky et al., 2017) and spectral normalization (Miyato et al., 2018) are instances where projections are used in training GANs.
A F URTHER DISCUSSION ON THE CONVERGENCE RESULTS
Theorems 1, 2, 3, and 4 use related but different notions of convergence.
Theorems 1 and 4 are asymptotic (has no rate) while Theorems 2 and 3 are non-asymptotic (has a rate).
Theoremsand 3 respectively show almost sure and L 2 convergence of the iterates.
Theorems 2 and 3 show convergence of the squared gradient norm for the best and last iterates, respectively.
We did not make these choices.
The choices were dictated by what we can prove based on the analysis.
The discrete-time analysis of SimGD-O of Theorem 2 bounds the squared gradient norm of the best iterate , while the continuous-time analysis bounds the squared gradient norm of the “last iterate” (at terminal time).
The discrepancy comes from the fact that while we have monotonic decrease of (cid:107) g ( t ) (cid:107) in continuous-time, we have no analogous monotonicity condition on (cid:107) g k (cid:107) in discrete-time.
To the best of our knowledge, there is no result establishing a O (1 /k ) rate on the squared gradient norm of the last iterate for SimGD-O or the related “extragradient method” Korpelevich (1976).
Theorem 3 is the ﬁrst result showing a rate close to O (1 /k ) on the last literate.
For SimGD-O and Corollary 1, the parameter choices are almost optimal.
The optimal choices that minimize the bound of Theorem 2 are α = 0 .
124897 /R
and β = 1 .
α ; they provide a factor of 135 .
771 , a very small improvement over the factor 136 of Corollary 1.
For SimGD-A and Theorem 3, there is a discrepancy in the rate between the continuous time analysis O (1 /t 2 ) and the discrete time rate O (1 /k 2 − 2 p ) for p ∈ (1 / 2 , 1) , which is slightly slower than
O (1 /k ) .
In discretizing the continuous-time calculations to obtain a discrete proof, errors accumulate and prevent the rate from being better than O (1 /k ) .
This is not an artifact of the proof.
Simple tests on bilinear examples show divergence when p < 1 / 2 .
SSSGD-A and Theorem 4 involves the parameter ε .
While the proof requires ε > 0 , we believe this is an artifact of the proof.
In particular, we conjecture that Lemma 17 holds with o ( s/τ ) rather than
O ( s/τ ) , and, if so, it is possible to establish convergence with ε = 0 .
In Figure 2, it seems that that the choice ε = 0 and p = 2 / 3 is optimal for SSSGD-A. While we do not have a theoretical explanation for this, we point out that this is not surprising as p = 2 / 3 is known to be optimal in stochastic convex minimization (Moulines & Bach, 2011; Taylor & Bach, 2019).
Theorems 2, 3, and 4 extend to monotone operators (Ryu & Boyd, 2016; Bauschke & Combettes, 2017) without any modiﬁcation to their proofs.
In inﬁnite dimensional setups (which is of interest in the ﬁeld of monotone operators)
Theorem 4 establishes strong convergence, while many convergence results (including Theorems 2 and 3) establish weak convergence.
However, Theorem 1 does not extend to monotone operators, as the use of the LaSalle–Krasnovskii principle is particular to convex-concave saddle functions.
B N OTATION AND PRELIMINARIES
Write R + to denote the set of nonnegative real numbers and (cid:104)· , ·(cid:105) to denote inner product, i.e., (cid:104) u, v (cid:105) =
u T v for u, v ∈ R m + n .
We say A is a point-to-set mapping on R d if A maps points of R d to subsets of R d .
For notational simplicity, we write
[Equation]
Using this notation, we deﬁne monotonicity of A with
[Equation]
where the inequality requires every member of the set to be nonnegative.
We say a monotone operator A is maximal if there is no other monotone operator B such that the containment {
[Equation]
) } is proper.
If L : R m × R n → R is convex-concave, then the subdifferential operator
[Equation]
is maximal monotone
(Rockafellar, 1970).
By Bauschke & Combettes (2017) Proposition 20.36,
G ( z ) is closed-convex for any z ∈ R m + n .
By Bauschke & Combettes (2017) Proposition 20.38(iii), maximal monotone operators are upper semicontinuous in the sense that if G is maximal monotone, then g k ∈ G ( z k ) for k = 0 , 1 , . . .
and ( z k , g k ) → ( z ∞ , g ∞ ) imply g ∞ ∈ G ( z ∞ ) .
(In other words, the graph of G is closed.)
Deﬁne Zer( G ) = { z ∈ R d | 0 ∈ G ( z ) } , which is the set of saddle-points or equilibrium points.
When G is maximal monotone, Zer( G ) is a closed convex set.
Write
[Equation]
for the projection onto Zer( G ) .
Write C ( R + , R d ) for the space of R d -valued continuous functions on R + .
For f k : R + → R m + n , we say f k → f in C ( R + , R d ) if f k → f uniformly on bounded intervals, i.e., for all T < ∞ , we have
[Equation]
In other words, we consider the topology of uniform convergence on compact sets.
We rely on the following inequalities, which hold for any a, b ∈ R m + n any ε > 0 .
[Equation]
[Equation]
Both inequalities are called Young’s inequality.
(Note, (6) follows from (5) with ε = 1 .)
Lemma 1 (Theorem 5.3.33 of Dembo (2019)) .
Let {F k } k ∈ N + be an increasing sequence of σalgebras.
Let ( m k , F k ) be a martingale such that
[Equation]
for all k ≥ 0 and
[Equation]
then m k converges almost surely to a limit.
Lemma 2 (Robbins & Siegmund (1971)) .
Let {F k } k ∈ N + be an increasing sequence of σ -algebras.
Let { V k } k ∈ N + , { S k } k ∈ N + , { U k } k ∈ N + , and { β k } k ∈ N + be nonnegative F k -measurable random se-
quences satisfying
[Equation]
If ∞ (cid:88)
[Equation]
U
[Equation]
holds almost surely, then
V k
→ V ∞ ,
S k →almost surely, where V ∞ is a random limit.
Deﬁne
[Equation]
Note that 0 (cid:54) =
˜ G ( z (cid:63) ) is possible even if 0 ∈ G ( z (cid:63) ) when L is not continuously differentiable.
Lemma 3.
Under Assumptions (A0) and (A2) , we have
[Equation]
for some R 3 > 0 and R 4 > 0 .
Proof.
Let z (cid:63) be a saddle point, which exists by Assumption (A0).
Let ω and ω (cid:48) be independent and identically distributed.
Then
[Equation]
where we use the fact that g ( z (cid:63) ; ω (cid:48) ) − ˜ G ( z (cid:63) ) is a zero-mean random variable, Assumption (A2), and (6).
The stated result holds with R 23 = 2 R 21 and R 24 = 2 R 22 + 2 (cid:107) ˜
G ( z (cid:63) )
(cid:107) 2 .
C A NALYSIS OF T HEOREM
For convenience, we restate the update, assumptions, and the theorem:
[Equation]
L is convex-concave and has a saddle point
[Equation]
Theorem 1.
Assume (A0) , (A1) , and (A2) .
Furthermore, assume L ( x, u ) is strictly convex in x for all u .
Then SSSGD converges in the sense of z k a.s.
→ z (cid:63) where z (cid:63) is a saddle point of L .
Differential inclusion technique.
We use the differential inclusion technique of Duchi & Ruan (2018), also recently used in Davis et al.
The high-level summary of the technique is very simple and elegant: (i) show the discrete-time process converges to a continuous-time trajectory satisfying a differential inclusion, (ii) show any solution of the differential inclusion has a desirable property, and (iii) translate the conclusion in continuous-time to discrete-time.
However, the actual execution of this technique does require careful and technical considerations.
Proof outline.
For step (i), we adapt the LaSalle–Krasnovskii principle to show that a solution of the continuous-time differential inclusion converges to a saddle point.
(Lemma 5.)
Then we carry out step (ii) showing the time-shifted interpolated discrete time process converges to a solution of the differential inclusion.
(Lemma 6.)
Finally, step (iii), the “Continuous convergence to discrete convergence”, combines these two pieces to conclude that the discrete time process converges to a saddle point.
The contribution and novelty of our proof is in our steps (i) and (iii).
Preliminary deﬁnitions and results.
Consider the differential inclusion
[Equation]
with the initial condition z (0) =
z 0 .
We say z : [0 , ∞ )
→ R m + n satisﬁes (7) if there is a Lebesgue integrable ζ : [0 , ∞ ) → R m + n such that
[Equation]
Write z ( t ) =
φ t ( z 0 ) and call φ t : R m + n → R m + n the time evolution operator .
In other words, φ t maps the initial condition of the differential inclusion to the point at time t , which is well deﬁned by the following result.
Lemma 4 (Theorem 5.2.1 of Aubin & Cellina (1984)) .
If G is maximal monotone, the solution to (7) exists and is unique.
Furthermore, φ t : R m + n → R m + n is 1 -Lipschitz continuous for all t ≥ 0 .
C. P ROOF OF T HEOREM
Lemma 5 and its proof can be considered an adaptation of the LaSalle–Krasnovskii invariance principle (Krasovskii, 1959; LaSalle, 1960) to the setup of differential inclusions.
The standard result applies to differential equations.
Lemma 5 (LaSalle–Krasnovskii) .
Assume (A0) .
Assume L ( x, u ) is strictly convex in x for all u .
If z ( · ) satisﬁes (7) , then z ( t ) → z ∞ as t → ∞ and z ∞ ∈ Zer( G ) .
Proof.
Consider any z (cid:63) ∈ Zer( G ) , which exists by Assumption (A0).
Since z ( t ) is absolutely continuous, so is (cid:107)
z ( t ) − z
[Equation]
(cid:107) 2 , and we have
[Equation]
for almost all t > 0 , where ζ ( · ) is as deﬁned in (8) and the inequality follows from (1), monotonicity of G .
Therefore, (cid:107) z ( t ) − z (cid:63) (cid:107) 2 is a nonincreasing function of t , and
[Equation]
for some limit χ ≥ 0 .
Since z ( t ) is a bounded sequence, it has at least one cluster point.
Let t k → ∞ such that z ( t k ) → z ∞ , i.e., z ∞ is a cluster point of z ( · ) .
Then, (cid:107) z ∞ − z (cid:63) (cid:107) 2 = χ .
Since φ t ( · ) (with ﬁxed t ) is continuous by Lemma 4, we have
[Equation]
for all s ≥ 0 .
This means φ s ( z ∞ ) is also a cluster point of z ( · ) and
[Equation]
for all s ≥ 0 .
Therefore
[Equation]
for almost all s ≥ 0 .
Write z ∞ = ( x ∞ , u ∞ ) and let z (cid:63) =
( x (cid:63) , u (cid:63) )
∈ Zer( G ) .
Write ( φ xs ( z (cid:63) ) , φ us ( z (cid:63) ))
=
( φ s ( z (cid:63) )) .
If φ xs ( z (cid:63) ) (cid:54) = x (cid:63) [Equation] G
[Equation]
φ [Equation] ( z
[Equation]
)) , φ
[Equation]
−
[Equation]
>by strict convexity, and, in light of (9), we conclude φ xs ( z (cid:63) )
=
x (cid:63) for almost all s ≥ 0 .
Then for almost all s ≥ 0 , we have
[Equation]
where the ﬁrst inequality follows from concavity of L ( x, u ) in u and the second inequality follows from the fact that u (cid:63) is a maximizer when x (cid:63) is ﬁxed.
Therefore, we have equality throughout, and
L ( x (cid:63) , φ us ( z ∞ )) =
L ( x (cid:63) , u (cid:63) ) , i.e., φ us ( z ∞ ) also maximizes L ( x (cid:63) , · ) .
Remember that φ s ( z ∞ ) is a continuous function of s for all s ≥ 0 .
Therefore, that φ xs ( z ∞ ) =
x (cid:63) and that φ us ( z ∞ ) maximizes L ( x (cid:63) , · ) for almost all s ≥ 0 imply that the conditions hold for s = 0 .
In other words, x ∞ = x (cid:63) and u ∞ maximizes L ( x (cid:63) , · ) , and therefore z ∞ ∈ Zer G .
Finally, since z ∞ is a solution, (cid:107) z ( t ) − z ∞ (cid:107) converges to a limit as t → ∞ .
Since (cid:107) z ( t k ) − z ∞ (cid:107) → 0 , we conclude that (cid:107) z ( t ) − z ∞ (cid:107) → 0 as t → ∞ .
The following lemma is the crux of the differential inclusion technique.
It makes precise in what sense the discrete-time process converges to a solution of the continuous-time differential inclusion.
Lemma 6 (Theorem 3.7 of Duchi & Ruan (2018)) .
Consider the update
[Equation]
Deﬁne t k = (cid:80) ki
=
α i and z
[Equation]
Deﬁne the time-shifted process
[Equation]
Let the following conditions hold:
(i)
The iterates are bounded, i.e., sup k (cid:107) z k (cid:107) < ∞ and sup k (cid:107) ζ k (cid:107) < ∞ .
(ii)
The stepsizes α k satisfy Assumption (A1) .
(iii)
The weighted noise sequence converges: (cid:80) ∞ k =
α k ξ k = v for some v ∈ R d .
(iv) For any increasing sequence n k such that z n k → z ∞ , we have
[Equation]
Then for any sequence { τ k } ∞ k =1 ⊂ R + , the sequence of functions { z τ k interp
[Equation]
) } is relatively compact in C ( R + , R d ) .
If τ k → ∞ , all cluster points of { z τ k interp ( · ) } satisfy the differential inclusion (8) .
We verify the conditions of Lemma 6 and make the argument that the noisy discrete time process is close to the noiseless continuous time process and the two processes converge to the same limit.
Verifying conditions of Lemma 6.
Condition (i).
Let z (cid:63) ∈ Zer( G ) .
Write F k for the σ -ﬁeld generated by ω 0 , . .
ω k − 1 .
Write ˜
G ( z ) =
E g ( z ; ω ) ∈ G
( z ) .
Then
[Equation]
E
[Equation]
where we used Assumption (A2) and Lemma 3.
Since (cid:80) ∞ k =
α 2 k < ∞ by Assumption (A1), this inequality and Lemma 2 tells us
[Equation]
for some limit, which implies z k is a bounded sequence.
Since z k is bounded, so is ˜ G
( z k ) since
[Equation]
by Lemma 3.
Condition (ii).
This condition is assumed.
Condition (iii).
Deﬁne ξ k = g
( z k ; ω k ) − ˜ G ( z k ) and m
[Equation]
i = α i ξ i .
Then ( m k , F k ) is a martingale and
[Equation]
almost surely, where the ﬁrst inequality is the second moment upper bounding the variance, the second inequality is Lemma 3, and the third inequality is (6) and condition (i).
Finally, we have (iii) by Lemma 1.
Condition (iv).
As discussed in Section B, G is maximal monotone, which implies G is upper semicontinuous, i.e., ( z n k , g n k ) →
( z ∞ , g ∞ ) implies g ∞ ∈ G ( z ∞ ) , and G ( z ∞ ) is a closed convex set.
Therefore, dist( ζ n k , G ( z ∞ ))
→ 0 as otherwise we can ﬁnd a further subsequence such that converging to ζ ∞ such that dist( ζ ∞ , G ( z ∞ )) > 0 .
(Here we use the fact that ζ k is bounded due to condition (i)).
Since G ( z ∞ ) is a convex set,
[Equation]
In the main proof, we show that cluster points of z interp ( · ) are solutions.
We need the following lemma to conclude that these cluster points are also cluster points of the original discrete time process z k .
Lemma 7.
Under the conditions of Lemma 6, z interp ( · ) and z k share the same cluster points.
Proof.
If z ∞ is a cluster point of z k , then it is a cluster point of z interp ( · ) by deﬁnition.
Assume z ∞ is a cluster point of z interp ( · ) , i.e., assume there is a sequence τ j
→ ∞ such that z interp ( τ j ) → z ∞ .
Deﬁne k j → ∞ with t k j ≤ τ j < t k j +1 .
Then
[Equation]
where we use the assumption (i) which states that (cid:107) ζ k (cid:107) is bounded and assumption (iii) which states that α k ξ k → 0 .
We conclude z k j → z ∞ .
Continuous convergence to discrete convergence.
Let k j → ∞ be a subsequence such that z k j → z ∞ .
Let k (cid:48)
j → ∞ be a further subsequence such that
[Equation]
for all T ≥ 0 , which exists by Lemma 6.
(The time-shifted interpolated process converges to a solution of the differential inclusion.)
By Lemma 5,
[Equation]
where φ t z ∞ → φ ∞ z ∞ as t → ∞ and φ ∞ z ∞ is a saddle point.
(The solution to the differential inclusion converges to a solution.)
These facts together imply that for any ε > 0 , there exists k (cid:48)
j and τ j large enough that
[Equation]
and
[Equation]
Together, these imply
[Equation]
since z τ interp ( · )
=
z interp ( τ + · ) .
Therefore, φ ∞ z ∞ is a cluster point of z interp ( · ) , and, by Lemma 7, φ ∞ z ∞ is a cluster point of z k .
Since (cid:107) z k − φ ∞ z ∞ (cid:107) converges to a limit and converges to 0 on this further subsequence, we conclude (cid:107) z k − φ ∞ z ∞ (cid:107) → 0 almost surely.
D A NALYSIS OF T HEOREM
For convenience, we restate the update, assumptions, and the theorem:
[Equation]
convex-concave and has a saddle point
[Equation]
L is differentiable and ∇ L is R -Lipschitz continuous
[Equation]
Theorem 2.
Assume (A0) and (A3) .
If 0 < α < 2 β (1 − 2 βR ) , then SimGD-O converges in the sense of
[Equation]
Furthermore, z k → z (cid:63) , where z (cid:63) is a saddle point of L .
D. P ROOF OF T HEOREM
Throughout this section [Equation] write g k
= G ( z k ) for k ≥ − 1 .
Since we can deﬁne ˜ G = αG and ˜ β
= β/α and write the iteration as
[Equation]
we assume α = 1 without loss of generality.
Then
[Equation]
where the inequality follows from (1)
[Equation]
monotonicity of G
[Equation]
and
[Equation]
We can bound
[Equation]
where the ﬁrst inequality follows from (5), Young’s inequality, with ε
=
R and the second inequality follows from Assumption (A3), R -Lipschitz continuity of G .
Putting these together we get
[Equation]
Since β > 1 / 2 and R < (2 β − 1) / (4 β 2 ) is assumed for Theorem 2, we have
[Equation]
By summing (10), we have
[Equation]
where we use z
=
z − 1 .
Next,
[Equation]
where we use (6).
Using (11)
[Equation]
we get k
[Equation]
Therefore,
[Equation]
z
[Equation]
+1 − z
[Equation]
R 2 (cid:107) z k − z k − 1 (cid:107) 2 → 0 and (cid:107) g k (cid:107) 2 → 0 .
Moreover, we have min
[Equation]
By scaling G by α , we get the ﬁrst stated result.
By summing (10), we have
[Equation]
and using the triangle inequality we get
[Equation]
as k → ∞ .
(Remember g k → 0 .)
So z k is a bounded sequence, and let z ∞ be the limit of a convergent subsequence z n k .
Since G is a continuous mapping with g n k = G ( z n k ) , z n k → z ∞ , and g n k → 0 , we have G ( z ∞ ) = 0 .
Finally, we show that the entire sequence z k converges to z ∞ .
Reorganizing (10), we get
[Equation]
So (cid:107)
z
k +1 + βg
k − z (cid:63) (cid:107) 2 + 2 β 2 R (cid:107)
z k +1 − z k (cid:107) 2 is a nonincreasing sequence, and the following limit exists
[Equation]
Since z (cid:63) can be any equilibrium point, we let z (cid:63) =
z ∞ .
This proves (cid:107) z k − z ∞ (cid:107) 2 → 0 , i.e., z k → z ∞ .
E A NALYSIS OF T HEOREME. P RELIMINARY LEMMAS
We quickly state a few identities and inequalities we later use.
As the veriﬁcation of these results are elementary, we only provide a short summary of their proofs.
Lemma 8.
For p ∈ (0 , 1) and k ≥ 1 ,
[Equation]
The proof follows from a basic application of the inequality
[Equation]
for x ∈ [0 , 1] and p ∈ (0 , 1) .
Lemma 9.
For p ∈ (0 , 1) and k ≥ 1 ,
[Equation]
The proof follows from integrating the decreasing function p/x 1 − p from k to k + 1 .
Lemma 10.
For p ∈ (0 , 1) and k ≥ 1 ,
[Equation]
The proof follows from Lemma 8.
Lemma 11.
Given any V 0 , V 1 , . . .
∈ R , we have
[Equation]
The proof follows from basic calculations.
This result can be thought of as the discrete analog of
[Equation]
Lemma 12.
Let z 0 , z 1 , . . .
∈
R m + n be an arbitrary sequence.
Then for any k = 0 , 1 , . . .
[Equation]
The proof follows from basic calculations.
This result can be thought of as the discrete analog of
[Equation]
E. C ONVERGENT SEQUENCE LEMMAS
In the proofs of Theorems 3 and 4, we establish certain descent inequalities.
The following lemmas state that these inequalities imply boundedness or convergence.
Lemma 13.
Let { V k } k ∈ N + and { U k } k ∈ N + be nonnegative (deterministic) sequences satisfying
[Equation]
where C 1 > 0 , C 2 > 0 , f ( k ) =
o (1 /k 1 − ε ) with ε ∈ [0 , 1) , and ∞ (cid:88) k
=
U k < ∞ .
Then lim sup k →∞ V k ≤ C 22 /C 21 .
Proof.
For any δ ∈ (0 , C 1 ) , there is a large enough K ≥ 0 such that for all k ≥ K ,
[Equation]
Deﬁne
[Equation]
for k ≥ 0 .
Then
V k
[Equation]
1 ≤
[Equation]
Note that max {√ x − 1 , x − 1 } ≤ max { 0 , x − 1 } for all x ≥ 0 .
So
[Equation]
for large enough k .
Since
[Equation]
− ν } + U k for large enough k , we have
[Equation]
With a standard recursion argument (e.g. Lemma 3 of (Polyak, 1987))
we conclude max { 0 , V k − ν } → 0 .
Since this holds for any δ > 0 , we conclude lim sup k →∞ V k ≤ C 22 /C 21 .
Lemma 14.
Let ε ∈
Let { V k } k ∈ N + and { U k } k ∈ N
+ be nonnegative (deterministic) sequences satisfying
[Equation]
where C > 0 , f ( k ) =
o (1 /k 1 − ε ) , g ( k ) =
O (1 /k ) , and ∞ (cid:88) k
=
U k < ∞ .
Then V k → 0 .
Proof.
For any δ > 0 , there is a large enough K ≥ 0 such that
[Equation]
for all k ≥ K .
By Lemma 13, we conclude lim sup k →∞ V k ≤ δ 2 / ( C − δ ) 2 .
Since this holds for all δ > 0 , we conclude V k → 0 .
E. P ROOF OF T HEOREM
For convenience, we restate the update, assumptions, and the theorem:
[Equation]
L is convex-concave and has a saddle point
[Equation]
L is differentiable and ∇ L is R -Lipschitz continuous
[Equation]
A3)
Theorem 3.
Assume (A0) and (A3) .
If p ∈ (1 / 2 , 1) and γ ≥ 2 , then SimGD-A converges in the sense of
[Equation]
Proof outline.
Lemma 15 shows the iterates z k are bounded.
Lemma 16 shows that (cid:107)
z
k +1 − 2 z k
+ z k − 1 (cid:107) 2 , the analog of (cid:107) ¨ z (cid:107) 2 , is small.
The second-order derivative ¨ z does not arise in the continuoustime analysis of Section 4.1.
In the discrete-time setup, (cid:107) z
k +1 − 2 z k
+ z k − 1 (cid:107) 2 does arise, but we use Lemma 16 to show that its contribution is small.
The main proof follows by mimicking the continuous-time analysis by bounding the higher-order terms.
Throughout this section, write g k
= G ( z k ) for k ≥ − 1 .
Lemma 15.
For SimGD-A,
[Equation]
for all k ≥ 0 for some C > 0 .
(This result depends on assumption p > 1 / 2 .)
Proof.
[Equation]
where the ﬁrst inequality follows from (1), the monotonicity inequality, and (6) and the second inequality follows from Assumption A3.
We conclude the statement with Lemma 13.
Lemma 16.
For SimGD-A,
[Equation]
Proof.
[Table]
[Equation]
where the ﬁrst and second inequalities follow from (6) and the third inequality follows from Assumptions (A3) and Lemma 8.
Main proof.
In Section
[Equation]
1, we showed
[Equation]
in continuous time.
We mimic analogous calculations in the discrete-time setup:
[Equation]
[Table]
p (1 − p ) 2 k− γ (1 − p ) 2 k ( k + 1) + γp (1 − p ) 2( k + 1)(cid:19) (cid:107) z k − z k − 1 (cid:107)p 2 k 2 (cid:107) z k − z k − 1 (cid:107) 2 + p 2 (cid:107) z
k +1 − 2 z k
+ z k − 1 (cid:107)+2 (cid:107)
z
k +1 − 2 z k + z k − 1 (cid:107)− γ (1 − p ) (cid:18) k
( k + 1) − ( k + 1) p − k p k p ( k + 1) (cid:19) (cid:28) z k − z k − 1 , z 0 − 1 2( z k + z k 1 ) (cid:29) = − (cid:18) γ (1 − p ) k
+ p k − p (1 − p ) 2 k− γ (1 − p ) 2 k ( k + 1) + γp (1 − p ) 2( k + 1) 2 − p 2 k(cid:19) (cid:107)
z k − z k − 1 (cid:107)+ 1 + p 2 (cid:107)
z k +1 − 2 z k + z k − 1 (cid:107)− γ (1 − p )k
( k + 1) (cid:28) z k − z k − 1 , z 0 − 1 2( z k + z k 1 ) (cid:29) − γ (1 − p ) (cid:18)
k ( k + 1) − ( k + 1) p − k p k p ( k + 1) − 1 − p k
( k + 1) (cid:19) (cid:124) (cid:123)(cid:122) (cid:125)
=
C 1 ( k,p ) (cid:28) z
[Equation]
where the ﬁrst inequality follows from (1), the monotonicity inequality, the second inequality follows from Lemma 8 and (6), and the third inequality follows from Lemma 8 and (5), Young’s inequality, with ε = k .
By Lemma 10, | C 1 ( k, p ) | ≤ p (1 − p ) 2 k 3 .
Using (5), Young’s inequality, with ε = 1 /k and (6) we get − γ
[Equation]
− p
[Equation]
Putting these together we get
[Equation]
≤ −
[Equation]
With Lemma 15 and Lemma
[Equation]
we get
[Equation]
Note that there is a K ∈ N such that C 3 ( k, γ, p, R ) ≤ 0 for all k ≥ K (with γ , p , and R ﬁxed).
In Section 4.
[Equation]
we multiplied the established inequality by t 2 and integrating both sides to get
[Equation]
We mimic analogous calculations in the discrete-time setup.
Multiply both sides with k ( k + 1) and sum both sides from k = 1 to k = k , and apply Lemma 11 and Lemma 12 to get
[Equation]
where C 4 < ∞ since C 3 ( k, γ, p, R ) > 0 for only ﬁnitely many k .
Reorganizing we get
[Equation]
Reorganizing yet again we get
[Equation]
where we use the assumption that γ ≥ 2 .
Reorganizing again, we get
[Equation]
for k ≥ 1 , where the second inequality follows from (1), the monotonicity inequality
[Equation]
and the third inequality follows from (5)
[Equation]
Young’s inquality, with ε = γ/ ( k + 1) 1 − p .
Finally, we have
[Equation]
with C = 4 γ 2 + 8 C 4 / (1 − p ) 2 .
F A NALYSIS OF T HEOREM
For convenience
[Equation]
we restate the update, assumptions, and the theorem:
[Equation]
L is convex-concave and has a saddle point
[Equation]
Theorem 4.
Assume (A0) and (A2) .
If p ∈ (1 / 2 , 1) , ε ∈ (0 , 1 / 2) , and γ > 0 , then SSSGD-A converges in the sense of z k L 2 → z (cid:63) , where z (cid:63) is a saddle point.
To clarify, we do not assume L is differentiable for Theorem 4.
Proof outline.
The key insight is to deﬁne ζ k to be something like a “ﬁxed point” of the k -th iteration of SSSGD-A and then to show z k shrinks towards to ζ k in the following sense
[Equation]
Lemma 17 states that ζ k slowly (stably) converges to a solution.
Using the fact that z k shrinks towards ζ k and the fact that ζ k is a slowly moving target converging to a solution, we conclude z k converges to a solution.
Preliminary deﬁnition and result.
More precisely, we deﬁne ζ k to satisfy
[Equation]
(However, ζ k is not actually a ﬁxed point, since SSSGD-A has noise and since G is a multi-valued operator.)
We equivalently write
[Equation]
Lemma 17 (Proposition 23.31 and Theorem 23.44 of Bauschke & Combettes (2017)) .
Let G be a maximal monotone operator such that Zer( G ) (cid:54) = ∅ .
Then ( I + τG ) − 1 ( z 0 )
→ P Zer( G ) ( z 0 ) and (cid:107) ( I + ( τ + s ) G ) − 1 ( z 0 ) − ( I + τG ) − 1 ( z 0 ) (cid:107) ≤ O (cid:16) s τ (cid:17)
F. P ROOF OF T HEOREM
Main proof.
Since 1 − p − ε >
[Equation]
Lemma 17 gives us
[Equation]
Then we have
[Equation]
where the ﬁrst inequality follows from (1)
[Equation]
the monotonicity inequality, Cauchy-Schwartz inequality, and (6), Now we take the full expectation to get
[Equation]
E (cid:104) (cid:107)
z
k − ζ k (cid:107) 2 (cid:105) + O (1 /k ) E (cid:104) (cid:107)
z k − ζ k (cid:107) 2 (cid:105) 1 /+
O (cid:18) ( k + 1) 2 p (cid:19) ( (cid:107) z 0 − z (cid:63) (cid:107) 2 + 1) + O (cid:18) ( k + 1) 2(1 − ε ) (cid:19) (cid:107) z 0 − z (cid:63) (cid:107)
+ O (cid:18) ( k + 1)(cid:19) , where we used E [ (cid:107)
z k − ζ k (cid:107) ] 2 ≤ E [ (cid:107)
z k − ζ k (cid:107) 2 ] .
Applying Lemma 14, we get E (cid:104) (cid:107)
z k − ζ k (cid:107) 2 (cid:105) → 0 .
Since ζ k → P Zer( G ) ( z 0 ) , we conclude z k L 2 → P Zer( G ) ( z 0 ) .
G E XPERIMENT DETAILS
In this section, we prodvide further details of the experiments of Section 5.
Our Optimistic Adam is a variation of the Optimistic Adam of (Daskalakis et al., 2018), which uses β = 1 while we allow for a general optimism rate β > 0 .
For Anchored Adam, we do not diminish the strength of the anchor proportional to 1 /k 1 − ε since Adam does not diminish the learning rate.
Rather, we maintain a constant anchor strength γ but refresh the anchor point every T iterations.
The notation ∇ 2 in algorithm tables denote the element-wise square operation.
[Table]
[Table] [Table]
z
[Table]
[Equation]
Perform anchored gradient step
[Equation]
Return z K
[Table]
Optimizer = “Adam”, “Optimistic Adam”, or “Anchored Adam”
[Table]
[Table] [Table]
Optimizer = “Adam”, “Optimistic Adam”, or “Anchored Adam”
[Table]