Recursive Reasoning in Minimax Games: A Level k Gradient Play Method
Abstract
Despite the success of generative adversarial networks (GANs) in generating visually appealing images, they are notoriously challenging to train.
In order to stabilize the learning dynamics in minimax games, we propose a novel recursive reasoning algorithm:
Level k Gradient Play (Lv. k GP) algorithm.
In contrast to many existing algorithms, our algorithm does not require sophisticated heuristics or curvature information.
We show that as k increases, Lv.
k GP converges asymptotically towards an accurate estimation of players’ future strategy.
Moreover, we justify that Lv.
∞ GP naturally generalizes a line of provably convergent game dynamics which rely on predictive updates.
Furthermore, we provide its local convergence property in nonconvex-nonconcave zero-sum games and global convergence in bilinear and quadratic games.
By combining Lv.
k GP with Adam optimizer, our algorithm shows a clear advantage in terms of performance and computational overhead compared to other methods.
Using a single Nvidia RTX3090 GPU andtimes fewer parameters than BigGAN on CIFAR-10, we achieve an FID of 10. for unconditional image generation within 30 hours, allowing GAN training on common computational resources to reach state-of-the-art performance.
Introduction
In recent years, there has been a surge of powerful models that require simultaneous optimization of several objectives.
This increasing interest in multi-objective optimization arises in various domainssuch as generative adversarial networks [21, 27], adversarial attacks and robust optimization [37, 7], and multi-agent reinforcement learning [36, 55] - where several agents aim at minimizing their objectives simultaneously.
Games generalize this optimization framework by introducing different objectives for different learning agents, known as players.
The generative adversarial network is a widely-used method of this type, which has demonstrated state-of-the-art performance in a variety of applications, including image generation [28, 52], image super-resolution [32], and image-to-image translation [10].
Despite their success at generating visually appealing images, GANs are notoriously challenging to train [40, 39].
Naive application of the gradient-based algorithm in GANs often leads to poor image diversity (sometimes manifesting as "mode collapse")
[40], Poincare recurrence [39], and subtle dependency on hyperparameters [18].
An immense corpus of work is devoted to exploring and enhancing the stability of GANs, including techniques as diverse as the use of optimal transport distance [1, 22], critic gradient penalties [54], different neural network architectures [26, 6], feature matching [50], pre-trained feature space [51], and minibatch discrimination [50].
Nevertheless, architectural modifications (e.g., StyleGANs [27]) require extensive computational resources, and many theoretically appealing methods (Follow-the-ridge [56], CGD[53]) require Hessian inverse operations, which is infeasible for most GAN applications.
To stabilize the learning dynamics in GANs, many recent efforts rely on sophisticated heuristics that allow the agents to anticipate each other’s next move [16, 53, 23].
This anticipation is an example of a recursive reasoning procedure in cognitive science [12]. Similar to how humans think, recursive
[Figure]
More steps in the reasoning process leading to better anticipations and faster convergence.
reasoning represents the belief reasoning process where each agent considers the reasoning process of other agents, based on which it expects to make better decisions.
Importantly, it enables the use of opponents that reason about the learning agent, rather than assuming fixed opponents; the process can therefore be nested in a form as ’I believe that you believe that I believe...’.
Based on this intuition, we introduce a novel recursive reasoning algorithm that utilizes only gradient information to optimize
GANs.
Our contributions can be summarized as follows:
(i)
We propose a novel algorithm: level k gradient play (Lv. k GP), which is capable of reasoning about players’ future strategy.
In a game, agents at Lv.
k adjust their strategies in accordance with the strategies of Lv.
k − 1 agents.
We justify that, while typical GANs optimizers, such as Learning with Opponent Learning Awareness (LOLA) and Symplectic Gradient Adjustment (SGA), approximate Lv. 2 and Lv. 3 GP, our algorithm permits higher levels of strategic reasoning.
In addition, the proposed algorithm is amenable to neural network optimizers like Adam [29].
(ii) We show that, in smooth games, Lv.
k GP converges asymptotically towards an accurate prediction of agents’ next move.
Under mutual opponent shaping, two Lv.
∞ agents will naturally have a consistent view of one another if the Lv.
k GP converges as k increases.
Based on this idea, we provide a closed-form solution for Lv.
∞ GP: the Semi-Proximal Point Method (SPPM).
(iii)
We prove the local convergence property of Lv.
∞ GP in nonconvex - nonconcave zero-sum games and its global convergence in bilinear and quadratic games.
The theoretical analysis we present indicates that strong interactions between competing agents can increase the convergence rate of Lv.
k GP agents in a zero-sum game.
(iv) By combining Lv.
k GP with Adam optimizer, our algorithm shows a clear advantage in terms of performance and computational overhead compared to other methods.
Using a single 3090 GPU with 30 times fewer parameters and 16 times smaller mini-batches than BigGAN [6] on CIFAR-10, we achieve an FID score [24] of 10.17 for unconditional image generation within 30 hours, allowing GAN training on common computing resources to reach state-of-the-art performance
Related Works
In recent years, minimax problems have attracted considerable interest in machine learning in light of their connection with GANs.
Gradient descent ascent (GDA), a generalization of gradient descent for minimax games, is the principal approach for training GANs in applications.
GDA alternates between a gradient descent step for the min-player and a gradient ascent step for the max-player.
The convergence of GDA in games is far from as well understood as gradient descent in single-objective problems.
Despite the impressive image quality generated by GANs, GDA fails to converge even in bilinear zero-sum games.
Recent research on GDA has established a unified picture of its behavior in bilinear games in continuous and discrete-time [39, 46, 13, 41, 20].
First, [39] revealed that continuous-time GDA dynamics in zero-sum games result in Poincare recurrence, where agents return arbitrarily close to their initial state infinitely many times.
Second, [3, 58] examined the discrete-time GDA dynamics, showing that simultaneous update of two players results in divergence while the agents’ strategies remain bounded and cycle when agents take turns to update their strategies.
The majority of existing approaches to stabilizing GDA follow one of three lines of research.
The essence of the first method is that the discriminator is trained until convergence while the generator parameters are frozen.
As long as the generator changes slowly enough, the discriminator still converges in the presence of small generator perturbations.
The two-timescale update rule proposed by [21, 24, 42] aims to keep the discriminator’s optimality while updating the generator at an appropriate step size.
[25, 14] proved that this two-timescale GDA with finite timescale separation converges towards the strict local minimax/Stackelberg equilibrium in differentiable games.
[15, 56] explicitly find the local minimax equilibrium in games with secon-order optimization algorithms.
The second line of research overcomes the failure of GDA in games with predictive updates.
Extragradient method (EG) [30] and optimistic gradient descent (OGD) [13] use the predictability of the agents’ strategy to achieve better convergence property.
Their variants are developed to improve the training performance of GANs [8, 19, 43].
[45] provided a unified analysis of EG and OGD, showing that they approximate the classical proximal point methods.
Competitive gradient descent [53] models the agents’ next move by solving a regularized bilinear approximation of the underlying game.
Learning with opponent learning awareness (LOLA) and consistent opponent learning awareness (COLA) [16, 57] introduced opponent shaping to this problem by explicitly modeling the learning strategy of other agents in the game.
LOLA models opponents as naive learners rather than LOLA agents, while COLA utilizes neural networks to predict opponents’ next move.
Lookahead-minimax [9] stabilizes GAN training by ‘looking ahead’ at the sequence of future states generated by an inner optimizer.
In the game theory literature, recent work has proposed Clairvoyant Multiplicative Weights
Update (CMWU) for regret minimization in general games [47].
Although CMWU is proposed to solve finite normal form games, which are different from unconstrained continuous games that Lv.
k
GP aims to solve, both CMWU and Lv.
k GP share the same motivation of enabling the learning agent to update their strategy based on the opponent’s future strategy.
From this aspect, Lv.
k GP can be viewed as a specialized variant of CMWU that is specific to the problem of two-player zero-sum games, but adapted for unconstrained continuous kernel games.
Other methods directly modify the GDA algorithm with ad-hoc modifications of game dynamics and introduction of additional regularizers.
Consensus optimization (CO) [40] and gradient penalty [22, 54] improve convergence by directly minimizing the magnitude of players’ gradients.
Symplectic gradient adjustment (SGA) [33, 4, 17] improves convergence by disentangling convergent potential components from rotational Hamiltonian components of the vector field.
Preliminaries 3.1 Notation
In this paper, vectors are lower-case bold letters (e.g. θ ), matrices are upper-case bold letters (e.g. A ).
For a function f : R d → R , we denote its gradient by ∇ f .
For functions of two vector arguments f ( x , y ) :
R d 1 × R d 2 → R we use ∇ x f, ∇ y f to denote its partial gradients.
We use ∇ xx f, ∇ yy f, ∇ xy f to denote its Hessian.
A stationary point of f denotes the point where ∇ x f = ∇ y f = 0 .
We use ∥ v ∥ to denote the Euclidean norm of vector v .
We refer to the largest and smallest eigenvalues of a matrix A by λ max ( A ) and λ min ( A ) , respectively.
Moreover, we denote the spectral radius of matrix
A by ρ ( A )
= max {| λ 1 | , . . .
, | λ n |} , i.e., the eigenvalue with largest absolute value.
3.2 Problem Definition
In order to justify the effectiveness of recursive reasoning procedure, in this paper, we consider the problem of training Generative Adversarial Networks (GANs)[21].
The GANs training strategy defines a two-player game between a generative neural network G θ ( · ) and a discriminative neural network D ϕ ( · ) .
The generator takes as input random noise z sampled from a known distribution
P z , e.g., z ∼ P z , and outputs a sample G θ ( z ) .
A discriminator takes as input a sample x (either sampled from the true distribution P x or from the generator) and attempts to classify it as real or fake.
The goal of the generator is to fool the discriminator.
The optimization of GAN is formulated as a two-player differentiable game where the generator G θ with parameter θ , and the discriminator
D ϕ with parameters ϕ , aim at minimizing their own cost function f
( θ , ϕ ) and g ( θ , ϕ ) respectively, as follows:
[Equation]
where the two function f and g : R m × R n → R .
When f = − g the corresponding optimization problem is called a two-player zero-sum game and it becomes a minimax problem:
[Equation]
In this work, we assume the cost functions have Lipschitz continuous gradients with respect to all model parameters ( θ , ϕ ) :
Assumption 3.1.
The gradient ∇ θ f
( θ , ϕ ) , is L θθ − Lipschitz with respect to θ and L θϕ − Lipschitz with respect to ϕ and the gradient ∇ ϕ g ( θ , ϕ ) , is L ϕϕ − Lipschitz with respect to ϕ and
L ϕθ − Lipschitz with respect to θ , i.e.,
[Equation]
for all [Equation]
We define L := max { L θθ , L θϕ , L ϕθ , L ϕϕ } .
Level k Gradient Play
In this section, we propose a novel recursive reasoning algorithm, Level k Gradient Play (Lv. k GP), that allows the agents to discover self-interested strategies while taking into account other agents’ reasoning processes.
In Lv.
k GP, k steps of recursive reasoning is applied to obtain an anticipated future state ( θ ( k ) t , ϕ ( k ) t ) , and the current states ( θ t , ϕ t ) are then updated as follows:
Reasoning:
[Equation]
[Equation]
We define the current state ( θ t , ϕ t ) , to be the starting point ( θ (0) t , ϕ (0) t ) , of the reasoning process.
Learning agents that adopt Lv.
k GP strategy are then called Lv.
k agents.
Lv.
1 agents act naively in response to the current state using GDA dynamics and Lv.
2 agents act in response to Lv.
1 agents by assuming its opponent as a naive learner.
Therefore, Lv.
k GP allows for higher levels of strategic reasoning.
The inspiration comes from how humans collaborate: humans are great at anticipating how their actions will affect others, so they frequently find out how to collaborate with other people to reach a "win-win" solution.
The key to human collaboration is their ability to understand how other humans think which helps them develop strategies that benefit their collaborators.
One of our main theoretical results is the following theorem, which demonstrates that agents adopting Lv.
k GP can precisely predict other players’ next move and reach a consensus on their future strategies:
Theorem 4.1.
Suppose Assumption 3.1 holds.
Let us define ω t
=
[ θ t , ϕ t ] T ∈ R m + n , ω ( k ) t = [ θ ( k ) t , ϕ ( k ) t ]
T ∈ R m + n and ∆ max = 2 × max( ∥∇ θ f
( θ t , ϕ t ) ∥ , ∥∇ ϕ g
( θ t , ϕ t ) ∥ ) .
Assume ω ( k ) t lie in a complete subspace of R m + n .
Then for Lv.k GP we have:
[Equation]
Suppose the learning rate satisfies: η < (2 L ) − 1 , then the sequence { ω ( k ) t } ∞ k =0 is a Cauchy sequence.
That is, given ϵ > 0 , there exists N such that, if a > b > N then:
[Equation]
Moreover, the sequence { ω ( k ) t } ∞ k =0 converges to a limit ω ∗ t : lim k →∞ ω ( k ) t [Equation] ∗
[Equation]
In accordance with Theorem 4.1, if we define ω t +1 = ω ∗ t , then Lv.
∞ GP is equivalent to the following implicit algorithm where we call it Semi-Proximal Point Method:
[Equation]
4.1 Algorithms as an Approximation of SPPM
SPPM players arrive at a consensus by knowing precisely what their opponents’ future strategies will be.
Existing algorithms are not able to offer this kind of agreement.
For instance, consensus optimization[40] forces the learning agents to cooperate regardless of their own benefits.
Agents employ extra-gradient method[30], SGA[4], and LOLA[16] consider their opponents as naive learners, ignoring their strategic reasoning ability.
CGD[53] takes into account the reasoning process of learning agents; however, it leads to an inaccurate prediction of agents in games that have cost functions with non-zero higher order derivatives ( n ≥ 3 ) [57].
In this section, we consider a subset of provably convergent variants of GDA in the Minimax setting, showing that, for specific choice of hyperparameters, the mentioned algorithms either approximate SPPM or approximate the approximations of SPPM:
[Table]
[Table] ] θ
[Equation]
[Table]
LOLA [16]
[Equation]
Lv [Equation] 2 GP
[Equation]
O
[Equation]
LEAD † [23]
[Equation]
[Equation] In Table [Equation] we compare the orders of precision of different algorithms as an approximation of SPPM in Minimax games with infinitely differentiable objective functions.
In accordance with Equation (3) of Theorem 4.1, Lv.k GP is an O ( η k ) approximation of SPPM.
In order to analyze how well existing algorithms approximate SPPM, we consider the first-order Taylor approximation to SPPM:
[Equation]
Under-brace terms correspond to the first-order Taylor approximation of Lv. 2 GP.
For an appropriate choice of hyperparameters, SGA ( γ = η ) and LOLA ( δ = η ) are identical to the first-order Taylor approximation of Lv.2 GP, where each agent models their opponent as a naive learner.
Hence, we list them above the Lv.2 GP, which approximates SPPM up to O ( η 2 ) .
CGD exactly recovers the first-order Taylor approximation of SPPM.
1 In games with cost functions that have non-negative higher order derivatives ( n ≥ 3 ), the remaining term in SPPM’s first-order approximation is an error of magnitude O ( η 3 ) , which means that CGD’s accuracy is in the same range as that of Lv.
3 GP.
In bilinear and quadratic games where the objective function is at most twice differentiable, CGD is equivalent to SPPM.
The distinction between LEAD ( α = η ) and CGD can be understood by considering their update rules.
LEAD is an explicit method where opponents’ potential next strategies are anticipated based on their most recent move ( ϕ t − ϕ t − 1 ) .
On the contrary, CGD accounts for this anticipation in an implicit manner, ( ϕ t +1 − ϕ t ) , where the future states appear in current states’ update rules.
Therefore, the computation of CGD updates requires solving an function involving additional Hessian inverse operations.
A numerical justification is also provided in Table 2, showing that the approximation accuracy of Lv.
k GP improves as k increases.
Convergence Property
In Theorem 4.1, we have analytically proved that Lv.
k GP convergences asymptotically towards
SPPM, we will use this result to study the convergence property of Lv.
k GP in games based on our analysis of SPPM.
The local convergence of SPPM in a non convex - non concave game can be analyzed via the spectral radius of the game Jacobian around a stationary point:
Theorem 5.1.
Consider the (Minimax) problem under Assumption 3.1 and Lv.
k GP.
Let ( θ ∗ , ϕ ∗ ) be a stationary point.
Suppose θ t − θ ∗ not in kernel of ∇ ϕθ f ( θ ∗ , ϕ ∗ ) , ϕ t − ϕ ∗ not in kernel of ∇ θϕ f ( θ ∗ , ϕ ∗ ) and η < ( L ) − 1 .
There exists a neighborhood U of ( θ ∗ , ϕ ∗ ) such that if SPPM started at ( θ 0 , ϕ 0 ) ∈ U , the iterates { θ t , ϕ t } t ≥ 0 generated by SPPM satisfy:
[Equation]
where f ∗ =
f ( θ ∗ , ϕ ∗ ) .
Moreover,for any η satisfying:
[Equation]
SPPM converges asymptotically to ( θ ∗ , ϕ ∗ ) .
Remark 5.1.
Following the same condition as in 5.1, the iterates generated by Lv.
2 k GP satisfies:
[Equation]
where
[Equation]
for odd k , or (1 − ( η 2 λ min ( ∇ θϕ f ∗ ∇ ϕθ f ∗ )) k )1 − ( η 2 λ max ( ∇ θϕ f ∗ ∇ ϕθ f ∗ ))
k for even k ,
[Equation]
[Figure]
We assume that the difference between the trajectory { θ t , ϕ t } t ≥ 0 and the stationary point ( θ ∗ , ϕ ∗ ) is not in the kernel of ∇
ϕθ f ( θ ∗ , ϕ ∗ ) and ∇ θϕ f ( θ ∗ , ϕ ∗ ) for the sake of simplicity.
Detailed proofs without this assumption are provided in Appendix A.4.
Following the same setting as Theorem 4.1, we have η <
L − 1 , which ensures that η 2 λ max ( ∇ θϕ f ∗ ∇ ϕθ f ∗ ) < 1 .
Therefore, as k → ∞ , a → 1 and b → 0 in Remark 5.1, and as such Lv.
k GP has similar local convergence properties to SPPM.
Figureillustrates this property in a quadratic game.
Lv.
k GP may behave differently than SPPM at lower values of k , but as k increases, both max step size η max ,k and distance to equilibrium for a fixed number of iterations under η max , ∞ converges to that of SPPM.
Thus, we observe that Lv.
k
GP is empirically similar to SPPM at higher values of k .
We study the global convergence property of SPPM by analyzing its behavior in bilinear and quadratic games.
Consider the following bilinear game:
[Equation]
where M is a full rank matrix.
The following theorem summarizes SPPM’s convergence property:
Theorem 5.2.
Consider the Bilinear game and the SPPM method.
Further, we define r t =
∥
θ t − θ ∗ ∥ 2 + ∥ ϕ t − ϕ ∗ ∥ 2 .
Then, for any η > 0 , the iterates { θ t , ϕ t } t ≥ 0 generated by SPPM satisfy
[Equation]
It is worth noting that the convergence property of Lv.
k GP in bi-linear game has been studied in [2].
Furthermore, we study the convergence property of SPPM in the Quadratic game:
[Equation]
(Quadratic game) where A ∈ R n × n is symmetric and positive definite, B ∈ R n × n is symmetric and negative definite and the interaction term C ∈ R n × n is full rank.
SPPM in quadratic games converges with the following rate:
Theorem 5.3.
Consider the Quadratic game and the SPPM.
Then, for any η > 0 , the iterates { θ t , ϕ t } t ≥ 0 generated by SPPM satisfy
[Equation]
[Figure]
i n t [Figure] r a c t i
[Figure] c Theorem 5.1,5.2 and 5.3 indicate that stronger interaction ∇ θϕ f ( θ ∗ , ϕ ∗ ) improve the convergence rate towards the stationary points.
By contrast, in existing modifications of GDA, the step size is chosen in inversely proportional to the interaction term ∇ θϕ f ( θ ∗ , ϕ ∗ ) [40, 13, 34].
In Figure 3, we showcase the effect of interaction on different algorithms in the Quadratic game setup with dimension n = 5 and the interaction matrix is defined as C
= c
I .
Stronger interaction corresponds to higher values of c .
A key difference between SPPM and Lv.
k GP in the experiments is that, SPPM converges with any step size - and so arbitrarily fast - while it is not the case for Lv.
k GP.
In order to approximate
SPPM, one needs Lv.
k GP to be a contraction and so have η < L − 1 .
This result implies an additional bound on step sizes for Lv.
k GP.
As long as the constraint on η is satisfied, stronger interaction only improves convergence for higher order Lv.
k GP while all other algorithms quickly diverge as η and c increase.
Experimental Results
In this section, we discuss our implementation of Lv.
k GP algorithm for training GANs.
Its performance is evaluated on 8-Gaussians and two representative datasets CIFAR-10 and STL-10.
6.1 Level k Adam
We propose to combine Lv.
k GP with the Adam optimizer [29].
Preliminary experiments find that Lv.
k Adam to converge much faster than Lv.
k GP, see A.8 for experiment results.
A detailed pseudo-code for Level k Adam (Lv. k Adam) on GAN training with loss functions L G and L D is given in Algorithm 1.
For the Adam optimizer, there are several possible choices on how to update the moments.
This choice can lead to different algorithms in practice.
Unlike [19] where the moments are updated on the fly, in Algorithm 1, we keep the moments fixed in the reasoning steps and update it together with model parameters.
In Table 2, our experiment result suggests that the proposed Lv. k
Adam algorithm converges asymptotically as the number of reasoning steps k increases.
6.2 8-Gaussians
In our first experiment, we evaluate Lv.
k Adam on generating a mixture of 8-Gaussians with standard deviations equal to 0 . 05 and modes uniformly distributed around the unit cir-
Input: Stopping time T , reasoning steps k , learning rate η θ , η ϕ , decay rates for momentum estimates β 1 , β 2 , initial weight ( θ 0 , ϕ 0 ) , P x and P z real and noise-data distributions, losses L G ( θ , ϕ , x , z ) and L D ( θ , ϕ , x , z ) , ϵ = 1 e − 8 .
Parameters : Initial parameters: θ 0 , ϕ
Initialize first moments: m θ, 0 ←− 0 , m ϕ, 0 ←−
Initialize second moments: v θ, 0 ←− 0 , v ϕ, 0 ←−for t=0,...,T-1 do
Sample new mini-batch: x , z ∼ P x , P z , θ (0) t ←− θ t , ϕ (0) t ←− ϕ t , for n=1,...,k do
Compute stochastic gradient:
[Equation]
Update estimate of first moment:
[Equation]
Update estimate of second moment: v
[Equation]
Correct the bias for the moments: ˆ
[Equation]
) , ˆ v ( n ) ϕ,t
= v ( n ) ϕ,t (1 − β t 2 )
Perform Adam update: θ ( n ) t = θ t − η θ ˆ m ( n ) θ,t (cid:113) ˆ v ( n ) θ,t + ϵ ; ϕ ( n ) t = ϕ
t − η ϕ ˆ m ( n ) ϕ,t (cid:113) ˆ v ( n ) ϕ,t
+ ϵ θ t +1 ←− θ ( k ) t , ϕ t +1 ←− ϕ ( k ) t ; m θ,t ←− m ( k ) θ,t , m ϕ,t ←− m ( k ) ϕ,t ; v θ,t ←− v ( k ) θ,t , v ϕ,t ←− v ( k ) ϕ,t cle.
We use a two layer multi-layer perceptron with ReLU activations, latent dimension of 64 and batch size of 128 .
The generated distribution is presented in Figure 4.
[Figure]
[Table]
[Figure]
k
[Figure]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] In addition to presenting the mode coverage of the generated distribution after training, we also study the convergence of the reasoning steps of Lv.
k Adam and Lv.
k GP.
Let us define the difference between the states of Lv.
k
and
Lv.
k − 1 agents at time t as:
[Equation]
We measure the difference averaged over the first 100 iterations, 1100 (cid:80) 100 t =1 r kt , for Lv.
10 Adam ( η = 10 − 4 ) and
Lv. 10 GP ( η = 10 − 2 ).
The result presented in Table demonstrates that both Lv.
k GP and Lv.
k Adam are converging as k increases.
Moreover, the estimation precision of Lv.
k GP improves rapidly and [-convergesto 
-]converges to 0 within finite steps, making it an accurate estimation for SPPM.
[Table]
[Figure]
6.3 Image Generation Experiments
We evaluate the effectiveness of our Lv.
k
Adam algorithm on unconditional generation of CIFAR- [31].
We use the Inception score (the higher the better) [50] and the Fréchet Inception distance (the lower the better) [24] as performance metrics for image synthesis.
For architecture, we use the
SN-GAN architecture based on [44].
For baselines, we compare the performance of Lv.
6 Adam to that of other first-order and second-order optimization algorithms with the same SN-GAN architecture, and to state-of-the-art models trained with Adam.
For Lv.
k Adam, we use β 1 = 0 and β 2 = 0 .
9 for all experiments.
We use different learning rates for the generator ( η θ = 4 e − 5 ) and the discriminator ( η ϕ = 2 e − 4 ).
We train Lv.
k Adam with batch size 128 for 600 epochs.
For testing, we use an exponential moving average of the generator’s parameters with averaging factor β = 0 .
In table 3, we present the performance of our method and baselines.
The best FID score and Inception score, 10 . 17 ± 0 .
16 and 8 .
78 ± 0 .
06 , on SN-GAN are obtained with our Lv.
Adam.
We also outperform BigGAN and StyleGAN2 in terms of FID score.
Notably, our model has 5.1M parameters [Table] in total, and is trained with a small batch size of 128, whereas BigGAN uses 158.3M parameters and a batch size of 2048.
The effect of k and losses: We evaluate values of k = { 2 , 4 , 6 } on non-saturated loss [21] (nonzero-sum formulation) and hinge loss [35] (zero-sum formulation).
The result is presented in Table 4.
Remarkably, our experiments demonstrate that few steps of recursive reasoning can result in significant performance gain comparing to existing GAN optimizers.
The gradual improvements in the FID scores justify the idea that better estimation of opponents’ next move improves performance.
Moreover, we observe performance gains in both zero-sum and non-zero-sum formulations which supplement our theoretical convergence guarantees in zero-sum games.
Experiment results on STL-10:
To test whether the proposed Lv.
k
Adam optimizer works on higher resolution images, we evaluate its performance on the STL-10 dataset [11] with 3 × 48 ×resolutions.
In our experiments, Lv. 6 Adam obtained an averaged FID of 25 .
43 ± 0 .
18 which outperforms that of the Adam optimizer, 30 .
25 ± 0 .
26 , using the same SN-GAN architecture.
Memory and computation cost: Compared to SGD, Lv.
k Adam requires the same extra memory as the EG method (one additional set of parameters per player).
The relative cost of one iteration versus
SGD is a factor of k and the computational cost increases linearly as k increases, we illustrate this relationship in Figure 5 (right).
We provide an accelerated version of Lv.
k Adam in A.8 which reduces the computation cost by half for k > 2 .
In Figure 5 (top-left), we compare the FID scores obtained by Lv.
k Adam, Adam, and LEAD on CIFAR-10 over the same number of gradient computations.
LEAD, Lv4 Adam, and Lv6 Adam all outperform Adam in this experiment.
Lv
Adam outperforms
LEAD after 2 × 10 5 gradient computations.
Our method is also compared with different algorithms on the 8-Gaussian problem in terms of its computational cost.
On the same architecture with different widths, Figure 5 (bottom-left) illustrates the wall-clock time per computation for different algorithms.
We observe that the computational cost of Lv.
k Adam while being much lower than CGD, is similar to LEAD, SGA and CO which involve JVP operations.
Each run on CIFAR-10 dataset takes 30 ∼hours on a Nvidia RTX3090 GPU.
Each experiment on STL-10 takes 48 ∼ 60 hours on a Nvidia
RTX3090 GPU.
Conclusion and Future Work
This paper proposes a novel algorithm:
Level k gradient play, capable of reasoning about players’ future strategies.
We achieve an average FID score of 10.17 for unconditional image generation on
CIFAR-10 dataset, allowing GAN training on common computing resources to reach state-of-the-art performance.
Our results suggest that Lv.
k GP is a flexible add-on that can be easily attached to existing GAN optimizers (e.g., Adam) and provides noticeable gains in performance and stability.
In future work, we will examine the effectiveness of our approach on more complicated GAN designs, such as Progressive GANs [26] and StyleGANs [27], where optimization plays a more significant role.
Additionally, we intend to examine the convergence property of Lv.
k GP in games with more than two player in the future.
Broader Impact Our work introduces a novel optimizer that improves the performance of GANs and may reduce the amount of hyperparameter tuning required by practitioners of generative modeling.
Generative models have been used to create illegal content(a.k.a.
deepfakes [5]).
There is risk of negative social impact resulting from malicious use of the proposed methods.
Acknowledgement This work was supported by a funding from a NSERC Alliance grant and
Huawei Technologies Canada.
ZL would like to thank Tianshi Cao for insightful discussions on algorithm and experiment design.
We are grateful to Bolin Gao and Dian Gadjov for their support.
A Appendix A.1 Proof of Theorem 4.
Proof.
To begin with, let us consider Lv.
1 GP:
[Equation]
The differences between ( θ (1) t , ϕ (1) t ) and ( θ (0) t , ϕ (0) t ) are:
[Equation]
Recall our definition of ∆ max we have:
[Equation]
Then, with ω t = [ θ t , ϕ t ] T and ω ( k ) t =
[ θ ( k ) t , ϕ ( k ) t ] T , the differences between Lv. 2 agents and Lv.agents are:
[Equation]
Recall that L := max { L θθ , L θϕ , L ϕθ , L ϕϕ } , using Equation (8) we have:
[Equation]
Similarly, we can derive the differences between Lv. 3 and Lv.
2 agents:
[Equation]
Consequently
[Equation]
the difference between any two consecutive states k and k − 1 are upper bounded by:
[Equation]
Since ∥ ω ( k ) t − ω
[Equation]
k
[Equation]
− ϕ
[Equation]
∥ we have:
[Equation]
Suppose η < (2 L ) − 1 , such that the difference between any two consecutive states is a contraction, then we consider the difference, ∥ ω ( a ) t − ω ( b ) t ∥ , where a > b > 0 .
We can rewrite it as:
[Equation]
Since η < (2 L ) − 1 , we have that ηL < 1 and for any ϵ > 0 , we can solve for b such that η b L ( b − 1) ∆ max < ϵ .
Therefore the sequence { ω kt } ∞ k =0 is a Cauchy sequence.
Moreover, in a complete space, every Cauchy sequence has a limit: lim k →∞ ω ( k ) t
[Equation]
A.2 Proof of Theorem
Theorem A.1.
Consider the (Minimax) problem under Assumption 3.1 and Lv.
k GP.
Let ( θ ∗ , ϕ ∗ ) be a stationary point.
Suppose θ t − θ ∗ not in kernel of ∇ ϕθ f ( θ ∗ , ϕ ∗ ) , ϕ t − ϕ ∗ not in kernel of ∇ θϕ f ( θ ∗ , ϕ ∗ ) and η < ( L ) − 1 .
There exists a neighborhood U of ( θ ∗ , ϕ ∗ ) such that if SPPM started at ( θ 0 , ϕ 0 ) ∈ U , the iterates { θ t , ϕ t } t ≥ 0 generated by SPPM satisfy:
[Equation]
where f ∗ =
f ( θ ∗ , ϕ ∗ ) .
Moreover, for any η satisfying:
[Equation]
SPPM converges asymptotically to ( θ ∗ , ϕ ∗ ) .
Proof.
Consider the learning dynamics:
[Equation]
Let us define
[Equation]
It follows immediately by linearizing the system about the stationary point ( θ ∗ , ϕ ∗ )
that
[Equation]
Let us denote the Jacobian by
[Equation]
Then we can rewrite the dynamics around the stationary point as
[Equation]
Similarly, for the other player we have
[Equation]
Let us define the symmetric matrices Q θ =
( I + η 2 BB T ) − 1 , Q ϕ
= ( I + η 2 B T B ) − 1 and
P θ =
( I − η A ) , P ϕ
= ( I + η C ) .
Further we define r t =
∥
ˆ θ t +1 ∥ 2 + ∥ ˆ ϕ t +1 ∥ 2 .
Based on these definitions
[Equation]
and the expressions in (52) and (53) we have
[Equation]
To simplify the expression in (54) we use the following lemma:
Lemma A.1.
The matrices Q θ =
( I + η 2 BB T ) − 1 , Q ϕ = [Equation] I
[Equation]
η [Equation] T B ) − 1 satisfy the following properties:
[Equation]
Using this lemma [Equation] we can show that
[Equation]
where the intermediate equality holds as a T b = b T a .
Hence, the expression in (54) can be simplified as
[Equation]
We simplify equation (57) as follows.
Consider the term involving ˆ θ t .
We have
[Equation] [Equation]
where the last equality follows by replacing Q θ by its definition.
The same procedure follows for the term involving ϕ t which leads to the expression
[Equation]
Substitute ∥ Q θ P θ ˆ θ t ∥ 2 + η 2 ∥ Q ϕ B T P
θ ˆ θ t ∥ 2 and ∥ Q ϕ P ϕ
ˆ ϕ t ∥ 2 + η 2 ∥ Q θ BP ϕ ˆ ϕ t ∥ 2 in (57) with the expressions in (22) and (23), respectively, to obtain
[Equation]
Note that, we assume that the trajectory { ˆ θ t , ˆ ϕ t } t ≥ 0 is not in the kernel of BB T and B T B , thus
BB T ˆ θ t ̸ = 0 and B T B ˆ ϕ t ̸
= 0 .
Now using the expression in (60) and the fact that P θ = P T θ ,
P ϕ
= P T ϕ and BB T and B T B have the same set of non-zero eigenvalues, if we denote the minimum non-zero eigenvalues by λ min ( BB T ) and λ min ( B T B ) , we can write
[Equation]
Replacing ∥ θ t +1 − θ ∗ ∥ 2 + ∥ ϕ t +1 − ϕ ∗ ∥ 2 and ∥ θ t − θ ∗ ∥ 2 + ∥ ϕ t − ϕ ∗ ∥ 2 with r t +1 and r t we have:
[Equation]
Recall that A = ∇ θθ f ( θ ∗ , ϕ ∗ ) , B = ∇ θϕ f ( θ ∗ , ϕ ∗ ) and C = ∇ ϕϕ f ( θ ∗ , ϕ ∗ ) , therefore for any η satisfying that:
[Equation]
we have r t +1 < r t .
Since we linearize the system about the stationary point ( θ ∗ , ϕ ∗ ) , there exists a neighborhood U around the stationary point, such that, SPPM started at ( θ 0 , ϕ 0 )
∈
U converges asymptotically to ( θ ∗ , ϕ ∗ ) .
A.3 Proof of Remark 5.
Proof.
To prove the local convergence of Lv.
k GP in non-convex non-concave games, we first consider the update rule of Lv.
k GP:
Reasoning:
[Equation]
Similar to Section A.2 let us denote
[Equation]
and we define the difference between states and stationary points as
[Equation]
and ˆ θ t
[Equation]
θ t −
[Equation]
ˆ
[Equation]
ϕ
[Equation]
ϕ ∗
Linearizing the dynamical system induced by Lv.
k GP about the stationary point ( θ ∗ , ϕ ∗ )
we get:
[Equation]
Note, in Lv.
k GP, we define θ (0) t
= θ t and ϕ (0) t = ϕ t , thus for Lv.
1 GP, we have:
[Equation]
For Lv.
2 GP, we have:
[Equation]
Substituting ˆ θ (1) t and ˆ ϕ (1) t into the update rule above we get:
[Equation]
Similarly, for Lv. 3 and Lv. 4 GP we have:
[Equation]
and
[Equation]
Summarizing the equations above we have that for Lv.
2 k GP, its update can be written as:
[Equation]
Similar to Appendix A.2, let us define Q θ =
( I + η 2 BB T ) − 1 , Q ϕ
= ( I + η 2 B T B ) − 1 and
P θ =
( I − η A ) , P ϕ
= ( I + η C ) .
Further, we define R ( k ) θ
[Equation]
BB T )
[Equation] , R [Equation]
k )
[Equation]
= ( (cid:80) k − 1 i =0 ( − η 2 B T B ) k ) and E ( k )
θ
[Equation]
Since η < L − 1 , we have that:
[Equation]
and
[Equation]
Also, from Lemma A.1 and the definition of the error terms, it can be verified that
[Equation]
Then we can rewrite the update rule of Lv.
2 k GP:
[Equation]
Let us consider the following sum:
[Equation]
( Q θ + E ( k ) θ ) (cid:104) P [Equation] ˆ θ t [Equation]
BP
[Equation] ˆ ϕ t (cid:105)(cid:13)(cid:13)(cid:13) [Equation] + (cid:13)(cid:13)(cid:13) ( Q ϕ + E ( k ) ϕ ) (cid:104) η B T P
θ ˆ θ t + P ϕ
ˆ ϕ t (cid:105)(cid:13)(cid:13)(cid:13)(30)
The R.H.S. of Eq.(30) can be written as:
[Equation]
(cid:105)(cid:13)(cid:13)(cid:13) [Equation] =
ˆ θ Tt P T θ Q 2 θ P
θ ˆ θ t − 2 η ˆ θ Tt P T θ Q 2 θ BP ϕ ˆ ϕ t + η 2 ˆ ϕ Tt P T ϕ B T Q 2 θ BP ϕ ˆ ϕ t
+2ˆ θ Tt P T
θ Q θ E ( k ) θ
P
θ ˆ θ t − 4 η ˆ θ Tt P T θ Q θ E ( k ) θ BP ϕ ˆ ϕ t + 2 η 2 ˆ ϕ Tt P T ϕ B T Q θ E ( k ) θ BP ϕ ˆ ϕ t +ˆ θ T t P T θ [ E ( k ) θ ] 2 P θ ˆ θ t − 2 η ˆ θ T t P T θ [ E ( k ) θ ] 2 BP ϕ ˆ ϕ t + η 2 ˆ ϕ T t P T ϕ B T [ E ( k ) θ ] 2 BP ϕ ˆ ϕ t
+ ˆ ϕ
Tt P T ϕ Q 2 ϕ P ϕ
ˆ ϕ t + 2 η ˆ θ
Tt P T θ BQ 2 ϕ P ϕ
ˆ ϕ t + η 2 ˆ θ Tt P T θ BQ 2 ϕ B T P θ ˆ θ t +2 ˆ ϕ
Tt P T ϕ Q ϕ E ( k ) ϕ
P ϕ
ˆ ϕ t + 4 η ˆ θ Tt P T θ BQ ϕ E ( k ) ϕ
P ϕ
ˆ ϕ t + 2 η 2 ˆ θ Tt P T θ BQ ϕ E ( k ) ϕ
B T P θ ˆ θ t
+ ˆ ϕ
Tt P T ϕ
[ E ( k ) ϕ ] 2 P ϕ ˆ ϕ t + 2 η ˆ θ Tt P T θ B [ E ( k ) ϕ ] 2 P ϕ ˆ ϕ t + η 2 ˆ θ Tt P T θ B [ E ( k ) ϕ ] 2 B T P
θ ˆ θ t (31)
Now, before adding all terms in Eq.(31), note that all of the cross terms in Eq.(31) cancel out.
For instance, using Lemma A.1 and Eq.(28), Eq.(29) we can show that
[Equation]
By using similar arguments it can be shown that terms in Eq.(31) leads to:
[Equation]
Similar to Eq.(22) we have the following simplification:
[Equation]
Now we can further simplify Eq.(31) as:
[Equation]
Using Eq.(26) and Eq.(27) and definition of Q θ and Q ϕ we have:
[Equation]
Similarly
[Equation]
we have that
[Equation]
Thus we simplify the R.H.S. of Eq.(30) as
[Equation]
Let us consider the L.H.S. of Eq.(30)
[Equation]
Substituting Eq.(34) and Eq.(33) into L.H.S. and R.H.S. of Eq.(30) respectively we get:
[Equation]
Now we have the following equation:
[Equation]
Note that
[Equation]
Similarly
[Equation]
Summing everything together we have:
[Equation]
η 2 BB T ) k
ˆ θ t ∥+
(( I − ( − η 2 B T B ) k ) P ϕ ˆ ϕ t ) T ( I + η 2 B T B ) − 1 (( I − ( − η 2 B T B ) k ) P ϕ
ˆ ϕ t )
+( ˆ ϕ t ) T ( η 2 B T B ) k
( ˆ ϕ t ) − ∥ ( − η 2 B T B )
k ˆ ϕ t ∥
Note that, we assume that the trajectory { ˆ θ t , ˆ ϕ t } t ≥ 0 is not in the kernel of BB T and B T B , thus
BB T ˆ θ t ̸ = 0 and B T B ˆ ϕ t ̸
= 0 .
Now using the expression in (60) and the fact that P θ = P T θ ,
P ϕ
= P T ϕ and BB T and B T B have the same set of non-zero eigenvalues, if we denote the minimum non-zero eigenvalues by λ min ( BB T ) and λ min ( B T B ) , we can write
[Equation]
Let us define the distance as:
[Equation]
Then we have
[Equation]
where
[Equation]
and
[Equation]
A.3.1 Proof
of Lemma [-A.
-]A.
Let B = U Σ V T be the singular value decomposition of B .
Here U and V are orthonormal matrices and Σ is a rectangular diagonal matrix.
Then we have:
[Equation]
Here we used the fact that U and V are orthonormal matrices.
Now, we simplify the other side to get:
[Equation]
Now we consider the following equation:
[Equation]
which indicates that Σ ( η 2 Σ T Σ + I )
=
( η 2 ΣΣ T + I ) Σ .
Multiplying both sides of this equation by ( η 2 ΣΣ T + I ) − 1 and ( η 2 Σ T Σ + I ) − 1 we have:
[Equation]
Therefore, we have Q θ B = BQ ϕ .
Using a similar argument, we can also prove the equality in
Equation (20).
A.4 Theorem 5.1 without kernel assumption
Theorem A.2.
Consider the (Minimax) problem under Assumption 3.1 and Lv.
k GP.
Let ( θ ∗ , ϕ ∗ ) be a stationary point.
Suppose η < ( L ) − 1 .
There exists a neighborhood U of ( θ ∗ , ϕ ∗ ) such that if
SPPM started at ( θ 0 , ϕ 0 ) ∈ U , the iterates { θ t , ϕ t } t ≥ 0 generated by SPPM satisfy:
[Equation]
where f ∗ =
f ( θ ∗ , ϕ ∗ ) .
Moreover, for any η satisfying:
[Equation]
SPPM converges asymptotically to ( θ ∗ , ϕ ∗ ) .
Proof.
Following the same setting and procedure as in Appendix A.2, we have that
[Equation]
Now using the fact that P θ = P T θ , P ϕ
= P T ϕ , we can write
[Equation]
For any η that satisfying
[Equation] [Equation]
we have that
[Equation]
48) i.e., SPPM converges asymptotically towards ( θ ∗ , ϕ ∗ ) .
A.5 Proof of Theorem 5.
Proof.
In order to proof the convergence of SPPM in bilinear games, we first show that the SPPM update rule is equivalent to that of the following Proximal Point Method:
[Equation]
In the Bilinear game, the SPPM update is:
[Equation]
and the PPM update is:
[Equation]
Thus SPPM and PPM are equivalent in the Bilinear game.
The convergence result of PPM in bilinear games has been proved in Theorem 2 of [49]:
Theorem A.3.
Consider the Bilinear game and the PPM method.
Further, we define r t =
∥
θ t − θ ∗ ∥ 2 + ∥ ϕ t − ϕ ∗ ∥ 2 .
Then, for any η > 0 , the iterates { θ t , ϕ t } t ≥ 0 generated by SPPM satisfy
[Equation]
Therefore, SPPM and PPM have the same convergence property in bilinear games.
A.6 Proof of Theorem 5.
Proof.
Consider the learning dynamics:
[Equation]
In the Quadratic game [Equation] the SPPM update rule can be written as:
[Equation]
Then we can rewrite the learning dynamics:
[Equation]
Similarly
[Equation]
for the other player we have
[Equation]
Let us define the symmetric matrices Q θ =
( I + η 2 CC T ) − 1 , Q ϕ
= ( I + η 2 C T C ) − 1 and
P θ =
( I − η A ) , P ϕ
= ( I + η B ) .
Further we define r t =
∥
θ t +1 ∥ 2 + ∥ ϕ t +1 ∥ 2 .
Based on these definitions, and the expressions in (52) and (53) we have
[Equation]
To simplify the expression in (54) we use Lemma A.1 to obtain the following equations:
[Equation]
Using this lemma
[Equation]
we can show that
[Equation]
where the intermediate equality holds as a T C = C T a .
Hence, the expression in (54) can be simplified as
[Equation]
We simplify equation (57) as follows.
Consider the term involving θ t .
We have
[Equation]
where the last equality follows by replacing Q θ by its definition.
The same procedure follows for the term involving ϕ t which leads to the expression
[Equation]
Substitute ∥ Q θ P θ θ t ∥ 2 + η 2 ∥ Q ϕ C T P θ θ t ∥ 2 and ∥ Q ϕ P ϕ ϕ t ∥ 2 + η 2 ∥ Q θ CP ϕ ϕ t ∥ 2 in (57) with the expressions in (58) and (59), respectively, to obtain
[Equation]
Now using the expression in (60) and the fact that P θ = P T θ
[Equation]
P ϕ
= P T ϕ and λ min ( C T C ) =
λ min ( CC T ) [Equation] we can write
[Equation]
A.7 Competitive Gradient Descent as an Approximation of SPPM
In this section, we justify our results in Section 4.1 that Competitive Gradient Descent is a first order
Taylor approximation of SPPM.
Firstly, we consider the standard definition of CGD:
[Equation]
ϕθ f
( θ t , ϕ t ))
− 1 ( ∇ θ f ( θ t , ϕ t ) + η ∇ θϕ f ( θ t , ϕ t ) ∇ ϕ f
( θ t , ϕ t ))
ϕ t +1 =
ϕ t + η
( I + η 2 ∇ ϕθ f
( θ t , ϕ t ) ∇ θϕ f ( θ t , ϕ t ))
− 1 ( ∇ ϕ f
( θ t , ϕ t ) − η ∇ ϕθ f
( θ t , ϕ t ) ∇ θ f ( θ t , ϕ t ))
Rewriting the update rules we can get:
[Equation]
Similarly, we have:
[Equation]
Therefore [Equation] CGD is a first order approximation of SPPM.
Then we prove that the standard definition of
CGD is equivalent to the update rule in Table 1.
Consider the update rule in Table 1 and its footnote, we have:
[Equation]
Substituting ( ϕ t +1 − ϕ t ) into the first equation of (61) we get:
[Equation]
η ∇ θϕ f ( θ t , ϕ t ) ∇ ϕ f
( θ t , ϕ t )) .
Substituting ( θ t +1 − θ t ) into the second equation of (61) and applying similar arguments we get: ϕ
[Equation]
Thus the update rule in Table 1 is equivalent to the standard definition of CGD and it is equivalent to the first order Taylor approximation of SPPM.
A.8 Experiments on Bilinear and Quadratic Games
Bilinear Game Consider the following bilinear game:
[Equation]
the example presented in Figure 1 is an example of using different algorithms to solve the bilinear game above with coefficient a = 10 .
For sake of completeness, we also provide a grid of experiment results for different algorithms with different coefficients a and learning rates η , starting from the same point ( θ 0 , ϕ 0 ) =
( − 12 , 10) .
The result is presented in Figure 6.
The experiment demonstrates that, in a bilinear game, Lv. 2 GP is equivalent to the extra-gradient method, and higher level Lv.
k GP performs better with increased coefficient a and learning rate η as long as it remains a contraction (i.e., η < a − 1 ).
Quadratic Game For the quadratic game presented in Figure 3, we randomly initialize the matrices
A and B :
A =   1 . 8398 0 .
7664 2 .
[Equation]
  1 . 0821 1 . 2427 1 . 0093 1 . 3335 0 .1 .
9693 0 .  where A is symmetric and positive definite and B is symmetric and negative definite.
The interaction matrix is defined as:
[Equation]
 where c represents the strength of the interaction between the two players.
The starting point θ 0 and ϕ 0 are [0 . 1270 , 0 . 9667 , 0 . 2605 , 0 . 8972 , 0 . 3767]
T and [0 .
T respectively.
A.9 Experiments on 8-Gaussians
Dataset The target distribution is a mixture of 8-Gaussians with standard deviation equal to 0 .and
modes uniformly distributed around a unit circle. GDA 0. LEAD 0. EG
[Figure] [Figure] Lv [Figure]
GP 0. [Figure] [Figure] Lv
[Figure] 10 GP 0. SPPM 1010 - 1010 - 1010 - 1010 - 101010c oe ff i c i en t a
Experiment For our experiments, we used the PyTorch framework.
Furthermore, the batch size we used is 128 .
A.10 Experiments on CIFAR-10 and STL-
For our experiments, we used the PyTorch 2 framework.
For experiments on CIFAR-10 and STL10, we compute the FID and IS metrics using the provided implementations in Tensorflow 3 for consistency with related works.
Lv.
k GP vs Lv.
k Adam
In experiments, we compare the performance of Lv.
k GP and Lv.
k Adam on the task of CIFAR-10 image generation.
The experiment results is presented in Figure 8.
The experiments on Lv.
k GP and Lv.
k
Adam use the same initialization and hyperparameters.
According to our experiments, Lv.
k Adam converges much faster than Lv.
k GP for the same choice of k and learning rates.
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
{+[Figure] GDA 0.06 0.07 0.08 0.09 0.1 0.11 0.
 SGA (LOLA) 0.06 0.07 0.08 0.09 0.1 0.11 0.
 LEAD 0.06 0.07 0.08 0.09 0.1 0.11 0.
+}
EG
[Figure]
[Figure]
[Figure]
[Figure]
Lv
[Figure]
[Figure]
[Figure]
Lv.
[Figure]
SPPM (CGD) 1010 - 1010 - 1010 - 1010 - 101010i n t e r a c t i on c
[Figure]
Adam vs Lv.
k
Adam We also present a comparison between the performance of Adam and Lv.
k
Adam optimizers on the task of STL-10 image generation.
The experiment results is presented in
Figure 9.
Under the same choice of hyperparameters and identical model parameter initialization,
Lv.
k Adam consistently outperforms the Adam optimizer in terms of FID score.
Accelerated Lv.
k Adam
In this section, we propose an accelerated version of Lv.
k Adam.
The intuition is that we update the min player θ and the max player ϕ in an alternating order.
The corresponding Lv.
k GP algorithm can be writen as:
Reasoning:
[Equation]
Instead of responding to θ ( k − 1) t , in Alt-Lv.
k GP, the max player ϕ
( k ) t acts in response to the min player’s current action, θ ( k ) t .
A Lv.
k min player in Alt-Lv.
k GP is equivalent to a Lv.
2 k − 1 player in [Figure] the Lv.
k GP, and a Lv.
k max player in Alt-Lv.
k GP is equivalent to a Lv. 2 k player in the Lv. k GP, respectively.
Therefore, it is easy to verify that Alt-Lv.
k GP converges two times faster than Lv.
k GP and the corresponding Alt-Lv.
k
Adam algorithm is provided in Algorithm 2.
Architecture In this section, we describe the model we used to evaluate the performance of Lv.
k
Adam for generating CIFAR-10 4 and STL-10 datasets.
With ’conv’ we denote a convolutional layer and ’transposed conv’ a transposed convolution layer.
The models use Batch Normalization and
Spectral Normalization.
The model’s parameters are initialized with Xavier initialization.
Images generated on CIFAR-10 and STL-10 In this section, we present sample images generated by the best performing trained generators on CIFAR-10 and STL-10.
Input: Stopping time T , reasoning steps k , learning rate
, decay rates for momentum estimates β 1 , β 2 , initial weight ( θ 0 , ϕ 0 ) , P x and P z real and noise-data distributions, losses L G ( θ , ϕ , x , z ) and L D ( θ , ϕ , x , z ) , ϵ = 1 e − 8 .
Parameters : Initial parameters: θ 0 , ϕ
Initialize first moments: m θ, 0 ←− 0 , m ϕ, 0 ←−
Initialize second moments: v θ, 0 ←− 0 , v ϕ, 0 ←−for t=0,...,T-1 do
Sample new mini-batch: x , z ∼ P x , P z , θ (0) t ←− θ t , ϕ (0) t ←− ϕ t , for n=1,...,k do
Compute stochastic gradient: g ( n ) θ ,t = ∇ θ L G ( θ t , ϕ ( n − 1) t , x , z );
Update estimate of first moment: m ( n ) θ,t = β 1 m θ,t − 1 + (1 − β 1 ) g ( n ) θ,t ;
Update estimate of second moment: v ( n ) θ,t = β 2 v θ,t − 1 + (1 − β 2 )( g ( n ) θ,t ) 2 ;
Correct the bias for the moments: ˆ m ( n ) θ,t = m ( n ) θ,t (1 − β t 1 ) , ˆ v ( n ) θ,t = v ( n ) θ,t (1 − β t 2 ) ;
Perform Adam update: θ ( n ) t = θ t − η θ ˆ m ( n ) θ,t (cid:113) ˆ v ( n ) θ,t + ϵ ;
Compute stochastic gradient: g ( n ) ϕ ,t = ∇ ϕ L D ( θ ( n ) t , ϕ t , x , z );
Update estimate of first moment: m ( n ) ϕ,t
=
β 1 m ϕ,t − 1 + (1 − β 1 ) g ( n ) ϕ,t ;
Update estimate of second moment
[Table]
Correct the bias for the moments:
[Table]
Perform
[Table]
G-ResBlock
Shortcut:
Upsample( × 2 )
Residual:
Batch NormalizationReLU
Upsample( × 2 ) conv (ker: 3 × 3 , 256 → 256 ; stride: 1 ; pad: 1 )
Batch Normalization
ReLU conv (ker: 3 × 3 , 256 → 256 ; stride: 1 ; pad: 1 )
D-ResBlock ( l − th block)
Shortcut: [ AvgPool (ker: 2 × 2 ) ] , if l =conv (ker: 1 × 1 , 3 l =1 / 128 l ̸ =1 → 128 ; stride: 1 )
Spectral Normalization [ AvgPool (ker: 2 × 2 , stride: 2 ) ] , if l =
Residual: [ ReLU ] , if l ̸ =conv (ker: 3 × 3 , 3 l =1 / 128 l ̸ =1 → 128 ; stride: 1 ; pad: 1 )
Spectral Normalization
ReLU conv (ker: 1 × 1 , 128 → 128 ; stride: 1 )
Spectral Normalization
AvgPool (ker: 2 × 2 )
[Table] [Table]