AGE: Enhancing the Convergence on GANs using Alternating extra-gradient with Gradient Extrapolation
Abstract
Generative adversarial networks (GANs) are notably difﬁcult to train since the parameters can get stuck in a local optimum.
As a result, methods often suffer not only from degeneration of the convergence speed but also from limitations in the representational power of the trained network.
Existing optimization methods to stabilize convergence require multiple gradient computations per iteration.
We propose AGE, an alternating extra-gradient method with nonlinear gradient extrapolation, that overcomes these computational inefﬁciencies and exhibits better convergence properties.
It estimates the lookahead step using a nonlinear mixing of past gradient sequences.
Empirical results on CIFAR10, CelebA and several synthetic datasets demonstrate that the introduced approach signiﬁcantly improves convergence and yields better generative models.
Introduction
Generative adversarial networks (GANs) have become the standard approach to generate plausible new samples [14].
Typically, GANs are trained by a generator and a discriminator in an adversarial way.
Generators are trained to mimic real samples, whereas discriminators are trained to classify between real samples and fake samples drawn from the generators.
While very powerful, GANs are notoriously hard to train and often fail to converge.
One reason for non-convergence is cycling around the optimum or even slow outward spiraling [24, 30].
There have been attempts to improve stability and tackle this training issue by proposing new formulations of the GAN objective [3, 29, 22].
[33] uses historical averaging of both generator and discriminator parameters as a regularization term in the objective function to improve stability.
Wasserstein GANs (WGANs)
[3] remedy the mode collapse that appears in the standard GANs by using the Wasserstein distance.
Orthogonal to the above strategies to make the GAN-game well-deﬁned and stable, a few studies have focused on developing speciﬁc optimization methods for GAN training.
In particular, the extra-gradient method has drawn signiﬁcant attention [11, 25, 10, 13].
These algorithms treat GANs as a variational inequality problem and aim to ﬁnd the Nash equilibrium of a minimax problem from a game theory perspective.
One key idea behind the extra-gradient method is to update the parameter using a lookahead gradient at each iteration.
Although these attempts have provided better results, there is still scope to improve the convergence of GANs.
Of particular interest is to accelerate the convergence of extra-gradient as it provides better guarantees.
In addition, the extra-gradient method requires multiple gradient computations which are computationally expensive.
We propose AGE (Alternating extra-gradient with Gradient Extrapolation) , a novel alternating update scheme that can accelerate the convergence of several minimax optimization problems.
Tangential to the previous approaches, it uses an adaptive gradient extrapolation procedure to replace the extrapolation step in extra-gradient by an adaptive nonlinear mixing of previous gradient sequences.
This avoids multiple gradient computations while also mitigating the noise arising from gradient computations.
In addition, AGE uses an alternating update scheme to obtain better generative models.
We summarize our main contributions:
• A method to extrapolate the current gradient based on an efﬁcient and adaptive nonlinear mixing of previous gradient sequences.
• Convergence and stability analysis of AGE .
An asymptotic quadratic convergence rate is provided as an illustrative example and asymptotically optimal convergence rates for optimizing stochastic problems are proved in Sec 4.
• Empirical evaluation of the advantages of AGE on both toy settings and two real datasets.
For all tasks, using AGE leads to improved convergence and higher-quality generative models.
Outline: We present the background on GANs and extra-gradient, and show a nonlinear extrapolation technique in Sec 2.
Then we propose our method AGE in Sec 3 and give its convergence properties in Sec 4.
We review the related work in Sec A.4 and provide experimental results in Sec 5.
Background
In this section, we ﬁrst review the extra-gradient method.
We then brieﬂy describe the standard nonlinear extrapolation algorithm, which will be used to extrapolate gradient sequences.
2.1 Extra-gradient method
GANs consist of two models: a generator G θ , that aims to generate samples from a noise distribution q θ that best matches the true distribution p of the data, and a discriminator D φ whose purpose is to classify genuine samples against generated ones, thereby training the generator.
It can be seen as a saddle point problem and solved by ﬁnding the Nash equilibrium (stationary point)
However, saddle point problems are known to be hard to optimize.
Both simultaneous and alternating gradient methods fail to converge but instead exhibit a cycling behavior.
Solving GAN can be considered as a variational inequality problem [2, 13].
First introduced in [20], extra-gradient method (EG) performs a “prediction” step to obtain an extrapolated point ( θ t + 1 2 , φ t + 1 2 ) , and then applies the gradients at the extrapolated point to the current iterate ( θ t , φ t ) as follows:
[Equation]
where η denotes the step size.
In the concept of a two-player game, extrapolation corresponds to an opponent shaping step: each player anticipates its opponent’s next moves to update its strategy.
For bilinear games, a slightly more generalized version was proposed in [23] with linear convergence proved.
It guarantees convergence for any convex-concave function L and any closed convex sets [16].
It has been shown that EG converges to saddle points sublinearly for smooth convex-concave games [27].
2.2 Nonlinear Extrapolation Acceleration
Extrapolation techniques are designed to accelerating an arbitrary sequence of objects in a vector space, such as vectors, matrices or tensors.
These methods perform a linear combination of previous k iterates of a given sequence to obtain one which can converge faster to the solution.
One such technique for matrices is the Vector Epsilon Algorithm (VEA)
The algorithm uses the generalized matrix inverse to extend the scalar (cid:15) -algorithm to sequences of matrices.
Assuming that { x t } is a sequence of matrices, VEA produces an extrapolated matrix from every k consecutive iterates x t and can be implemented by an elegant recursive formula without performing the unstable determiannt computations.
Let (cid:15) n 2 k denote the output of VEA algorithm at iteration n .
It can be shown that (cid:15) n 2 k essentially represents the k-term Shanks transforms of the original sequence { x t } : (cid:15) ( n ) 2 k = e k ( x n ) , where e k ( x n ) =
(cid:80) kt =
γ t x n + t with γ t being the solution to the linear system k (cid:88) t = γ t ∆ x m + t = 0 , m = n, . .
. , n + k − 1 , k (cid:88) t
= γ t =and ∆ x t = x t +1 − x t .
When the sequence satisﬁes the k-term Shanks kernel exactly, VEA will yield a constant sequence equal to the limit value.
In general, the extrapolated sequence has been shown to converge faster in various nonlinear problems [7, 32].
Under the assumption that the size of the epsilon table k is equal to the degree of minimal polynomial with respect to initial point during the beginning of each iteration, it can be shown that extrapolated sequence can achieve quardratic convergence rate asymptotically [36].
Proposed Method 3.1 Gradient extrapolation via VEA
Inspired by recent advances in parameter averaging and variance reduction schemes [1, 17], we propose an adaptive gradient extrapolation procedure on top of the VEA framework.
Different from most parameter averaging schemes that use an inner-outer pullback fashion, we apply VEA on gradient sequences directly during the training procedure.
This framework also mitigates the noise arising from gradient computation over different batches.
Assuming that { x t } is a sequence of gradient matrices, VEA produces an extrapolated gradient from every k consecutive iterates x t in the following manner:
[Equation]
The ﬁnal output of those sequences deﬁned in (cid:15) -algorithm equivalently represents the k-term Shanks transforms of the original sequence { x t } as shown in (2.2) [6].
As soon as a new gradient x t becomes available, we can immediately compute (cid:15) ( t − 1), (cid:15) ( t − 2), . . .
based on (3).
The ﬁnal output will be used as the extra-gradient in the next iteration.
Since the procedure is highly parallel and is dominated by dense matrix/vector operations, this extrapolation can be effectively implemented on GPUs.
[34] showed that the complexity of VEA is O ( Nk 2 ) , in which N is the parameter size and k is the sequence length (set as 3 in all experiments).
The relationship between (cid:15) -algorithm and Shanks transformation and implementation details can be found in the Appendix.
The relation between the gradient extrapolation via VEA and the one-step linear extrapolation proposed in [13] is shown in the following theorem.
Theorem 3.1.
Assume x i ’s are the gradient sequence generated by the extra-gradient method.
Then we have e 0 ( x i − 1 ) =
x i .
This theorem shows that when k is set to be 0 , the gradient extrapolation via VEA is equivalent to alternating gradient descent with one-step linear extrapolation.
For general k , we can show that the gradient extrapolation via VEA leads to faster convergence than the linear extrapolation extra-gradient method in Sec 4.
1: Given θ, φ , maximum iteration number T and learning rate η 2: Set γ = 0 .
Build a empty list for θ and φ , A θ , A φ 3: Let V EA ( x t ) denote e k
( x n − k − 1 ) .
4: while t < T do 5:
Use VEA to extrapolate gradients of θ and φ at t with the length size k : 6: ∇
[Equation]
t ,
[Equation] [Equation]
) 7: ∇
[Equation] [Equation]
) 8: η
[Equation]
min
[Equation]
η,
[Equation]
9: η
[Equation]
= min { η,
[Equation] [Equation] [Equation]
[Equation]
[Equation]
Calculate gradient of θ t + 12 using θ t + 12 and φ t +13: θ
[Equation]
=
[Equation] [Equation]
Store ∇ θ L θ (cid:16) θ t
[Equation]
12 , φ t
[Equation]
12 (cid:17) 15: Calculate gradient of φ t + 12 using θ t +1 and φ t +16: φ
[Equation]
(cid:16)
[Equation]
[Equation] 12 (cid:17) 17: Store ∇ φ L φ (cid:16) θ t +
[Equation]
, φ t + 12 (cid:17) 18: end while 19: return θ and φ 3.2 AGE
AGE combines the gradient extrapolation technique proposed in the previous section with the extragradient method for solving the minimax problem. First, we replace the extrapolation step in extra-gradient with gradient extrapolation. After we obtain the extrapolated gradient using (2.2), we use it to update model parameters with the same learning rate as the baseline optimizer. Different from extra-gradient method where the parameters are updated simultaneously as shown in (2), we propose an alternating extra-gradient updating scheme.
That is, after the gradient extrapolation, we update the parameters in an alternating way since well-performing GAN generators are closer to a saddle-point instead of a local minimum, which suggests that the local Nash, the typical solution concept for simultaneous games, may not be the most appropriate one for GANs [5].
The AGE algorithm is summarized in Algorithm 1.
Lines 4-6 perform gradient extrapolation where gradients computed up to iteration t − 1 (Lines 12 and 15) are used by VEA to estimate the gradients at iteration t .
Note that when two consecutive gradient sequences are too close, VEA may encounter numerical stability issues.
To resolves these issues, clipping (Lines 7-8) is used as a safeguard [41].
An extrapolation step is performed on the estimated (clipped) gradients in Lines 9-10.
We then calculate the gradient of θ t + 12 in Line 11 and the gradient of φ t + 12 using the latest θ t in Line 14.
We repeat this procedure until convergence or the max iteration number is reached.
Note that the gradient extrapolation step in Lines 5-6 has a constant computational overhead by ﬁxing the length of the VEA sequence for each parameter.
The latest gradient information is discarded once VEA extrapolation is completed.
Theoretical analysis
In this section, we demonstrate the convergence properties of the proposed method ( AGE ) for solving several minmax optimization problems with detailed comparisons to the extra-gradient method [13].
4.1 Uni-dimensional game
We ﬁrst consider the following simple model problem:
[Equation]
This toy example has been extensively studied for illustrating the properties of the methods of interest [13, 8, 26].
We compare several optimization methods on solving this problem and plot the results in Fig 1.
The plots show that extra-gradient methods are more preferable for solving this kind of problem.
In particular, AGE shows a signiﬁcant improvement of convergence over all methods .
Proposition 4.1.
The squared norm N 2 t for one dimensional min-max problem solved by alternating extra-gradient has the convergence rate N 2 t
+
=
O (1 − η 2 ) N 2 t .
The above proposition shows the convergence rate of alternating extra-gradient without VEA.
Next, we analyze the convergence rate of AGE for solving (4).
Applying Algorithm 1 to solve (4) leads to the following updating rule:
Extrapolation:
[Equation]
Update: [Equation] Note that applying VEA on a separable minimax problem like (4) is equivalent to applying VEA only on one variable θ .
Notice that the Nash equilibrium of this problem is ( θ ∗ , φ ∗ ) =
Denote
N 2 t def = θ 2 t + φ 2 t as the squared two norm of the gradient sequence.
The convergence rate can be monitored by the magnitude of N 2 t and is analyzed in the next theorem.
Theorem 4.1.
When t is large enough, N 2 t computed based on the updating rule (5) decreases for any η < 1 as
[Equation]
In the next theorem, we show that the extrapolated sequence e k ( θ t ) has the asymptotic quadratic convergence rate.
Theorem 4.2.
When ( θ t , φ t ) is near the optimum ( θ ∗ , φ ∗ ) , e k ( θ t ) has the convergence rate
[Equation]
Finally we can show that AGE can achieve quadratic convergence in certain cases.
Proposition 4.2.
Suppose θ t , φ t and e k ( θ t ) have the same order.
Then quadratic convergence for gradient norm N t =
C t N 2 t − 1 can be achieved for some bounded constant C t . [13] analyzes the convergence rates of two extra-gradient methods for solving (4).
These results are are summarized in the next theorem to facilitate the theoretical comparison.
Theorem 4.3 (EG [13]) .
N 2 t computed based on the following two updating rules decreases geometrically for any 0 < η < 1 as
[Equation]
A comparison among the convergence rates of these methods further conﬁrms that the proposed method shown in Algorithm 1 can be faster than standard extra-gradient methods for solving (4) when t is large.
4.2 Bilinear problem and Stochastic optimization
Next, we study the convergence property of the proposed method on a more general unconstrained bilinear problem:
[Equation]
, which is arguably the simplest nontrivial problem.
For each problem, the the trajectory of our method and other algorithms in low dimensional toy problems
We can observe signiﬁcantly where A ∈ R l × m , b ∈ R l and c ∈ R m .
Assuming the existence of the optimal solution allows us to rewrite the problem as
[Equation]
where c := − θ ∗(cid:62) Aϕ ∗ is a constant that does not depend on θ and ϕ .
If we deﬁne ˆ N 2 t := dist ( θ t , Θ ∗ ) 2 +dist ( ϕ t , Φ ∗ ) 2 , where ( Θ ∗ , Φ ∗ ) is the solution, we can obtain the following estimate on the bound of ˆ N 2 t for the above bilinear unconstrained problem.
Theorem 4.4.
When t is large enough, for any 0 < η < 1 σ max ( A ) we have
[Equation]
Particularly, for
[Equation]
=
[Equation]
∀ t ≥ 0 , where σ max and σ min are the maximal and minimal singular values of A, respectively, and κ := σ max ( A ) 2 σ min ( A ) 2 is the condition number of A (cid:62) [Equation]
A similar convergence rate has been proved in Corollary 1 of [13] for extra-gradient methods as
[Equation]
Thus, AGE can converge faster on this unconstrained bilinear problem than extra-gradient methods when t is large.
Experiments
In this section, we investigate whether the theoretical guarantees of AGE carry over from simple to practical problems.
Particularly, our experiments have three main aims: (1) to test if AGE converges to local minimax and improves the convergence rate compared to extra-gradient methods, (2) to test the capability of AGE to address the notorious mode collapse problem in GAN training, (3) to test the effectiveness of generating images in GANs via AGE .
5.1 Motivating examples
First, we verify our claim on improved local convergence rates as suggested by Proposition 4.2.
We run AGE on the simple low-dimensional problems in (4).
It is a simple two-dimensional problems, yet traditional gradient methods (e.g., Sim GD and Alt GD) fail to converge to the Nash equilibrium.
The results are shown in Figure 1, which empirically conﬁrms the improved convergence rate of AGE . [Table]
5.2 Generative adversarial networks
One particularly promising application of minimax optimization algorithms is training GANs.
The goal in this experimental section is to show that our proposed method achieves faster convergence of
GANs and its capability of yielding better generative models across different tasks.
It is important to note that our method will not provide new state-of-the-art results with architectural improvements or a new GAN formulation.
We demonstrate our method on mixtures of Gaussians, the CIFAR dataset, and the CelebA dataset.
All experiments were run on a NVIDIA V100 GPU.
5.2.1 Mixture of Gaussians
To test our proposed method, we evaluate our method on a simple 2-D example from a mixture [-of
-]of 8 Gaussians with standard deviations equal to 1 × 10 − 2 and modes uniformly distributed around the unit circle.
For both generator and discriminator, we use fully connected neural networks withhidden layers and 64 hidden units in each layer.
Except for the output layer of discriminator that uses a sigmoid activation, we use tanh-activation for all other layers.
We use an 8-dimensional Gaussian prior and the original GAN objective function as in [14].
The details of the experiment are as follows.
We run alternating SGD, Extra-gradient (denoted
ExtraSGD ) [13], and AGE with SGD for 40000 steps since simultaneous SGD suffers from mode collapse [4].
The learning rate is set as 5 × 10 − 3 after an extensive grid search which is close to the maximal possible stepsize under which the methods rarely diverge.
Fig 2a shows the output after { 0 , 10 K, 20 K, 30 K, 40 K } iterations.
It can be seen that our method converges faster to the target distribution without oscillation (see step 10K) and offers a signiﬁcant improvement over baselines.
[Figure]
[Table]
Similar to [40], we measure the Wasserstein-1 distance 1 , a metric of the difference between two distributions [12], with 2048 samples from the original 8-dimensional Gaussian mixture and the estimated mixture distributions at every 1000 iterations.
The averaged Wasserstein-1 distance curves of each method are shown in Fig 2b.
We can observe that our method is closer to the true Gaussian mixture model (i.e., better results) and also converges faster than the baselines.
Since different learning rates can impact the convergence, we also run the 3 methods with a less aggressive learning rate 1 × 10 − 3 .
Table 1 reports the averaged best Wasserstein-1 distance achieved for each method over 5 runs.
The results demonstrate that our method outperforms SGD and ExtraSGD for both learning rates.
5.2.2 CIFAR-
In our second experiment, we apply our method to the CIFAR10 dataset [21] and use the ResNet architecture with the WGAN-GP objective as in [15, 13].
We compare with Adam [19] and ExtraAdam [13] which offers signiﬁcant improvements over OMD [25, 10].
Since the loss serves as an ambiguous metric for method comparison, we use the Inception score [33] and FID [18] as the performance measure for generative models.
The Inception score (IS) and
FID are computed on 50,000 samples every 5,000 iterations.
Experiments were run with 5 random seeds for 100,000 updates of the generator.
For all methods, we did an extensive search over the hyperparameters of Adam.
It turns out the best performing learning rate is η dis =
× 10 − 4 , η gen = 2 × 10 − 4 .
We set β 1 = 0 . 0 , β 2 = 0 .
9 for Adam and λ = 10 for gradient penalty.
Figure 3a and 3b depict the IS and FID obtained by each method.
We note that our method converges to a better FID (from 21.45 to 19.32 ) and IS (from 7.76 to 8.02 ).
Furthermore, we observe AGE-Adam exhibits smaller variance during training.
We postulate the gradient obtained by gradient extrapolation is less noisy.
[Figure]
To understand the impact of learning rates, we ran AGE-Adam and AGE-Adam with three different learning rates as reported in Table 2.
We also conduct experiments using a more common training technique, TTUR [18].
It uses different learning rates for discriminator and generator.
Experiments were run with 5 random seeds for 500,000 updates of the generator.
The results indicate that AGEAdam generally yields better IS than ExtraAdam regardless of the learning rate.
A comparison of wall-clock time of the two methods also support the claim that AGE improves the training as it achieves a lower wall-clock time (16 hours to 22 hours).
[Figure]
We also test our method’s effectiveness on the cropped CelebA data (64 × 64) which is a higherresolution image dataset compared to CIFAR10.
We use the ResNet architecture with the WGAN-GP objective as described previously.
Since the Inception score serves as an less informative metric for CelebA, we use FID [18] as the performance measure for generative models.
The FID is computed on 50,000 samples every 1, iterations.
The pre-calculated statistics is computed on the cropped images for the purpose of FID evaluation.
Experiments were run with 5 random seeds for 50,000 updates of the generator.
For all methods, we did an extensive search over the hyperparameters and use the same setting used for
CIFAR-10 as it still provides best performance.
Figure 3c depicts the FID obtained by each method and Figure 4b shows the generated images using AGE-Adam .
It can be observed that AGE-Adam still outperforms the baselines as our method achieves a lower FID (from 8.2 to 7.88).
Conclusion
We have proposed a new way to extrapolate gradient using AGE that enhances the convergence rate of adversarial training.
Because AGE is based on mixing past gradient information nonlinearly, it can avoid noise in the gradient estimate.
We have also proven the fast convergence rate of AGE on a motivating example and bilinear problems.
We empirically show better performance using AGE on both toy datasets and a real-world deep learning setting.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] See(b)
Did you describe the limitations of your work?
[Yes] See 3.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] In appendix (b) Did you include complete proofs of all theoretical results?
[Yes] In appendix 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
ﬁrst 6: while do 7: − Calculate z
=
z
+
F 9:
Y
=
Reset
12: end while 13: Set
14: if
F 16: end if 17: return E , T
A Appendix A.1 Detailed algorithm for Vector Epsilon Algorithm (VEA)
In Sec 3, we proposed a gradient extrapolation technique that adopts the vector epsilon algorithm to extrapolate gradient and achieves faster convergence in the context of extra-gradient method.
A general form of vector epsilon algorithm is shown in Eq 2.2.
As it can be quite effective if we implement it correctly, we present a brief summarization of efﬁcient implementation in Algorithm 1.
A.2 Relationship between Shanks Transformation and VEA
The relationship between VEA and Shanks transformation is illustrated in the following theorem:
Theorem A.1 ([39]) .
Let { x m } be an arbitrary sequence.
Then, for any two integers n and k, the
Shanks transformation produces an approximation to the limit or antilimit of this sequence that is given by
[Equation]
with the γ i being the solution to the linear system k (cid:88) i
= γ i ∆
[Equation]
[Equation]
=
[Equation]
γ
[Equation]
=and e k ( x n ) has the determinant representation
[Equation]
A.3 Theoretical analysis
In this section, we demonstrate the convergence properties of the proposed method ( AGE ) for solving several minmax optimization problems with detailed comparisons to the extra-gradient method [13].
A.3.1 Uni-dimensional game
We ﬁrst consider the following simple model problem:
[Equation]
Proposition A.1.
The squared norm N 2 t for one dimensional min-max problem solved by alternating extra-gradient has the convergence rate N 2 t +
= O (1 − η 2 ) N 2 t .
The above proposition shows the convergence rate of alternating extra-gradient without VEA.
Next, we analyze the convergence rate of AGE for solving (8).
Applying Algorithm 1 to solve (8) leads to the following updating rule:
[Equation]
Note that applying VEA on a separable minimax problem like (8) is equivalent to applying VEA only on one variable θ .
Notice that the Nash equilibrium of this problem is ( θ ∗ , φ ∗ ) =
Denote
N 2 t def = θ 2 t + φ 2 t as the squared two norm of the gradient sequence.
The convergence rate can be monitored by the magnitude of N 2 t and is analyzed in the next theorem.
Theorem A.2.
When t is large enough, N 2 t computed based on the updating rule (9) - (10) decreases for any η < 1 as
[Equation]
Proof Sketch.
When t is large enough, the iterations satisfy the following equation
[Equation]
[Equation]
We can show that, θ t ∼ O (1 − η 2 ) t and θ t − e k ( θ t − k − 1 ) ∼ o (cid:0) η 2( k +1)
( t − k − 1) (cid:1) ∼ o ( θ t ) .
Thus for t large enough, the convergence rate is at least N 2 t +
= O (1 − η 2 )
N 2 t .
In the next theorem, we show that the extrapolated sequence e k ( θ t ) has the asymptotic quadratic convergence rate.
Theorem A.3.
When ( θ t , φ t ) is near the optimum ( θ ∗ , φ ∗ ) , e k ( θ t ) has the convergence rate
[Equation]
Proof Sketch.
This proof is mainly based on Sec 8 of [36].
Firstly, we can show the error in the intial data will propagate linearly using epsilon algorithm.
We decompose the computed value into
[Equation]
where e k ( x n ) is the value extrapolated using { x i } ni = n − 2 k − 1 .
(cid:15) nk is the error in the computation of last entry in epsilon table deﬁned in (2.2).
Thus the errors in the initial data is given by (cid:107) (cid:15) n 0 (cid:107) =
(cid:107) x ∗ n − x n (cid:107) .
Deﬁne (cid:107) (cid:15) (cid:107) = max 0 ≤ n ≤ 2 k (cid:107) (cid:15) n 0 (cid:107) , where k is the degree of the minimal polynomial of iteration law with respect to x j − x ,
it implies the following recursion formula (cid:80) ki =
c i x i
=
( (cid:80) ki =
c i ) x, where x is the limit of the sequence.
Then by induction we can show that
[Equation]
for all k, n if the epsilon table exits.
In fact, this is true for (cid:15) n 2 k
which is the error for the extrapolated value we will use from the epsilon table.
According to the VEA algorithm, the computed entries of epsilon table satisfy
[Equation]
Extracting the exact values, we have
[Equation]
where
[Equation]
and
[Equation]
According to the induction hypothesis Eq.(14), we know α − 1 =
O ( (cid:107) (cid:15) nk (cid:107) )
=
O ( (cid:107) (cid:15) (cid:107) ) .
Thus we ﬁnally have (cid:107) (cid:15) nk +
(cid:107) =
O ( (cid:107) (cid:15) (cid:107) ) , which means the errors during the process of epsilon algorithm is controlled linearly by errors in the initial data.
Given the sequence generated through the nonlinear iteration deﬁned by x j +1 = F ( x j ) , where F is a vector-valued function deﬁned on an open and connected domain D with a Lipschitz continuous derivative.
If s is a ﬁxed point in D of F, then Taylor expansion yields F ( x ) − s = F (cid:48) ( s )( x − s )
+ O ( (cid:107) x − s (cid:107) 2 ) for all x ∈ D .
Then we can consider the following iteration x {+j +
+}
=
Ax j + b + r j , where A = F (cid:48) ( s ) and r j is the error in a linearly generated sequence x j +
=
Ax j .
Now we consider the extrapolation in Eq.(9), and for each cycle, we know (cid:107) e k
θ t ) (cid:107) ∼ C k (cid:107) e k
( θ t − 1 ) (cid:107) 2 , since ( θ ∗ , φ ∗ ) =
Finally we can show that AGE can achieve quadratic convergence in certain cases.
Proposition A.2.
Suppose θ t , φ t and e k ( θ t ) have the same order.
Then quadratic convergence for gradient norm N t =
C t N 2 t − 1 can be achieved for some bounded constant C t .
Proof Sketch.
θ t , φ t and e k ( θ t ) have the same order, θ t +1 ∼ O
( e k ( θ t )) .
According to the quadratic convergence of e k ( θ t ) , θ t +1 inherits the quadratic convergence directly.
Since safeguard constants are used to clip extrapolated gradient and gradients, C t is bounded.
Thus we can derive the desired result. [13] analyzes the convergence rates of two extra-gradient methods for solving (8).
These results are are summarized in the next theorem to facilitate the theoretical comparison.
Theorem A.4 (EG [13]) .
N 2 t computed based on the following two updating rules decreases geometrically for any 0 < η < 1 as
[Equation]
A comparison among the convergence rates of these methods further conﬁrms that the proposed method shown in Algorithm 1 can be faster than standard extra-gradient methods for solving (4) when t is large.
A.3.2 Bilinear problem and Stochastic optimization
Next, we study the convergence property of the proposed method on a more general unconstrained bilinear problem:
[Equation]
(19) where A ∈ R l × m , b ∈ R l and c ∈ R m .
We assume the optimal solution ( θ ∗ , ϕ ∗ ) exists which is equivalent to
[Equation]
Assuming the existence of the optimal solution allows us to rewrite the problem as
[Equation]
where c := − θ ∗(cid:62) Aϕ ∗ is a constant that does not depend on θ and ϕ .
If we deﬁne
[Equation]
where ( Θ ∗ , Φ ∗ ) is the solution, we can obtain the following estimate on the bound of ˆ N 2 t for the above bilinear unconstrained problem.
Theorem A.5.
When t is large enough, for any 0 < η < 1 σ max ( A ) we have
[Equation]
Particularly, for η =
[Equation]
where σ max and σ min are the maximal and minimal singular values of A, respectively, and κ :
= σ max ( A ) 2 σ min ( A ) 2 is the condition number of A (cid:62) A .
Proof Sketch.
For our updating rule, the iterates can be written in the following form:
[Equation]
where γ st , ρ st ∈ R .
Using the SVD of A A
= U (cid:62) Σ V , we can get the following update:
[Equation]
Since Σ is a diagonal matrix of singular values of A, this can be transformed into the following equivalent problem:
[Equation]
For the coordinates that are vanishing, they can be set to (cid:104) ˜ θ ∗ (cid:105) i and (cid:104)
˜
φ ∗ (cid:105) i , which yields
N 2 it <
(1 − ( σ i η ) 2 )
N 2 i ( t − 1) , where N 2 it =
( (cid:104) ˜
θ t (cid:105) i − (cid:104) ˜
θ ∗ (cid:105) i ) 2 + ([ ˜ ϕ t ] i − [ ˜ ϕ ∗ ] i ) 2 .
Since ˆ N 2 t := dist ( θ t , Θ ∗ ) 2 + dist ( ϕ t , Φ ∗ ) 2 = (cid:80) ri =
N 2 it and N 2 it <
(1 − ( σ i η ) 2 )
N 2 i ( t − 1) , we have ˆ N 2 t +1 < (cid:16)
[Equation]
A similar convergence rate has been proved in Corollary 1 of [13] for extra-gradient methods as
[Equation]
Thus, AGE can converge faster on this unconstrained bilinear problem than extra-gradient methods when t is large.
We now consider a stochastic version of Algorithm 1.
Deﬁne ω def = ( θ , ϕ ) , Ω def =
Θ × Φ , an exact gradient operator F ( ω ) and a stochastic estimate of it, F ( ω , ξ ) , where ξ is a mini-batch of points coming from θ and φ .
Similar to the variational inequality perspective in [13], we consider the following merit function:
[Equation]
] (cid:62) max ω
[Equation] ω (cid:48) ) , otherwise (20)
If the following four assumptions hold, we can prove an upper bound for the merit function (20) in the following theorem.
Note that assumptions 1 and 2 are standard for the convergence of SGD and assumptions 3 and 4 make a zero-sum game well-deﬁned.
Assumption 1.
Bounded variance by σ 2 : E
[Equation]
Assumption 2. Bounded expected squared norm by M 2 : E ξ (cid:2) (cid:107) F ( ω
[Equation]
ξ ) (cid:107) 2 (cid:3) ≤ M
[Equation]
∀ ω ∈ Ω
[Equation]
Assumption 3 .
F is monotone almost surely : (cid:104) F ( x ) − F ( y ) , x − y (cid:105) ≥ 0 for all x, y ∈ R d .
Assumption 4.
Ω is a compact convex set with
[Equation]
Theorem A.6.
Under Assump. 1, 2, 3 and 4, the SGD version of Algorithm 1 with averaging and a constant step-size gives
[Equation]
where
[Equation]
Proof Sketch.
We adopt the variational inequality perspective by taking the merit function introduced in [13] to measure the error of the iterates.
Using a result from [35] about VEA and assuming the gradient function is L-Lipschitz, we can obtain the following inequality:
[Equation]
If we deﬁne N t = (cid:107) ω t − ω (cid:107) 22 , M 1 ( ω t , ξ t ) = 3 (cid:107) F ( ω t ) − e k ( F ( ω t − k − 1 , ξ t − k − 1 ) (cid:107) 22 , we obtain the following inequality where L is the Lipschitz constant of F and C Ω is the diameter of Ω :
[Equation]
By Assump.
1, M 1 = M 2 = 3 σ 2 and E [ F ( ω (cid:48) t ) − F ( ω (cid:48) t , ξ t ) | ω (cid:48) t , ∆ 0 , . . . ,
∆ t − 1 ]
=
E [
[Equation]
[Equation]
[Equation]
=
[Equation]
A suitable choice of η then yields
[Equation]
Although the stochastic version of AGE has a similar theoretical convergence bound with the extragradient methods [13], we can show that AGE still outperforms extra-gradient methods empirically on both real and synthetic datasets in Sec 5.
A.4 Related Work
There is a rich literature on different strategies to improve GAN training, such as using different objectives [3, 29, 22], using inner-outer update fashion [9], adding an asymmetric preconditioner on top of gradient descent ascent (GDA) [37], or keeping an average over iterates outside of the training loop [28, 40].Particularly, [13] proposed to apply extragradient to overcome the cycling behaviour of GDA.
One issue with the existing extra-gradient method is that it requires computing the gradient at two different positions for every single update which is likely to be inefﬁcient for large scale problems.
Optimistic gradient descent (OGD), originally proposed in [31] and rediscovered in [10], is more efﬁcient by storing and re-using the extrapolated gradient for the extrapolation step.
Without projection, OGD is equivalent to extrapolation from past.
Nevertheless, its performance in a stochastic setting is worse than extra-gradient.
This is because the method only uses the most recent gradient information for extrapolation and may be subject to noisy updates as well.