Recursive Reasoning in Minimax Games: A Level k Gradient Play Method
Anonymous Author(s)
AffiliationAddress
Abstract
Despite the success of generative adversarial networks (GANs) in generating visually appealing images, they are notoriously challenging to train.
In order tostabilize the learning dynamics in minimax games, we propose a novel recursivereasoning algorithm:
Level k Gradient Play (Lv. k GP) algorithm.
Our algorithmdoes not require sophisticated heuristics or second-order information, as do existing algorithms based on predictive updates.
We show that as k increases, Lv.
k
GP converges asymptotically towards an accurate estimation of players’ futurestrategy.
Moreover, we justify that Lv.
∞ GP naturally generalizes a line of provably convergent game dynamics which rely on predictive updates.
Furthermore,we provide its local convergence property in nonconvex-nonconcave zero-sumgames and global convergence in bilinear and quadratic games.
By combining
Lv.
k GP with Adam optimizer, our algorithm shows a clear advantage in termsof performance and computational overhead compared to other methods.
Usinga single Nvidia RTX3090 GPU and 30 times fewer parameters than BigGAN on
CIFAR-10, we achieve an FID of 10.17 for unconditional image generation within24 hours, allowing GAN training on common computational resources to reachstate-of-the-art performance.
Introduction
In recent years, there has been a surge of powerful models that require simultaneous optimization ofseveral objectives.
This increasing interest in multi-objective optimization arises in various domainssuch as generative adversarial networks [21, 27], adversarial attacks and robust optimization [37, 7],and multi-agent reinforcement learning [36, 55] - where several agents aim at minimizing theirobjectives simultaneously.
Games generalize this optimization framework by introducing differentobjectives for different learning agents, known as players.
The generative adversarial network is awidely-used method of this type, which has demonstrated state-of-the-art performance in a variety ofapplications, including image generation [28, 52], image super-resolution [32], and image-to-imagetranslation [10].
Despite their success at generating visually appealing images, GANs are notoriouslychallenging to train [40, 39].
Naive application of the gradient-based algorithm in GANs often leadsto poor image diversity (sometimes manifesting as "mode collapse")
[40], Poincare recurrence [39],and subtle dependency on hyperparameters [18].
An immense corpus of work is devoted to exploringand enhancing the stability of GANs, including techniques as diverse as the use of optimal transportdistance [1, 22], critic gradient penalties [54], different neural network architectures [26, 6], featurematching [50], pre-trained feature space [51], and minibatch discrimination [50].
Nevertheless,architectural modifications (e.g., StyleGANs [27]) require extensive computational resources, andmany theoretically appealing methods (Follow-the-ridge [56], CGD[53]) require Hessian inverseoperations, which is infeasible for most GAN applications.
To stabilize the learning dynamics in GANs, many recent efforts rely on sophisticated heuristics thatallow the agents to anticipate each other’s next move [16, 53, 23].
This anticipation is an example of
[Figure]
More steps in the reasoning process leading to better anticipations and faster convergence.
a recursive reasoning procedure in cognitive science [12].
Similar to how humans think, recursivereasoning represents the belief reasoning process where each agent considers the reasoning processof other agents, based on which it expects to make better decisions.
Importantly, it enables the use ofopponents that reason about the learning agent, rather than assuming fixed opponents; the process cantherefore be nested in a form as ’I believe that you believe that I believe...’.
Based on this intuition,we introduce a novel recursive reasoning algorithm that utilizes only gradient information to optimize
GANs.
Our contributions can be summarized as follows:
(i)
We propose a novel algorithm: level k gradient play (Lv. k GP), which is capable of reasoningabout players’ future strategy.
In a game, agents at Lv.
k adjust their strategies in accordance with thestrategies of Lv.
k − 1 agents.
We justify that, while typical GANs optimizers, such as Learning withOpponent Learning Awareness (LOLA) and Symplectic Gradient Adjustment (SGA), approximateLv. 2 and Lv. 3 GP, our algorithm permits higher levels of strategic reasoning.
In addition, the proposedalgorithm is amenable to neural network optimizers like Adam [29].
(ii) We show that, in smooth games, Lv.
k GP converges asymptotically towards an accurate
predictionof agents’
next move.
Under mutual opponent shaping, two Lv.
∞ agents
will
naturally
have a
consistent view of one another if
the
Lv.
k
GP
converges
as k increases
Based on this idea, we
provide a closed-form solution for Lv.
∞
GP
: the Semi-Proximal Point Method (SPPM
(iii)
We prove the local convergence property of Lv.
∞ GP in nonconvex - nonconcave zero-sum
games and its global convergence in bilinear and quadratic games.
The theoretical analysis we present
indicates that strong interactions between competing agents can increase the convergence rate of Lv.
k
GP agents in a zero-sum game.
iv)
By combining Lv.
k GP with Adam optimizer, our algorithm shows a clear advantage in terms ofperformance and computational overhead compared to other methods.
Using a single 3090 GPU with30 times fewer parameters and 16 times smaller mini-batches than BigGAN [6] on CIFAR-10, weachieve an FID score [24] of 10.17 for unconditional image generation within 24 hours, allowing
GAN training on common computing resources to reach state-of-the-art performance.
Related Works
In recent years, minimax problems have attracted considerable interest in machine learning in light oftheir connection with GANs.
Gradient descent ascent (GDA), a generalization of gradient descentfor minimax games, is the principal approach for training GANs in applications.
GDA alternates between a gradient descent step for the min-player and a gradient ascent step for the max-player.
Theconvergence of GDA in games is far from as well understood as gradient descent in single-objectiveproblems.
Despite the impressive image quality generated by GANs, GDA fails to converge even inbilinear zero-sum games.
Recent research on GDA has established a unified picture of its behaviorin bilinear games in continuous and discrete-time [39, 46, 13, 41, 20].
First, [39] revealed thatcontinuous-time GDA dynamics in zero-sum games result in Poincare recurrence, where agentsreturn arbitrarily close to their initial state infinitely many times.
Second, [3, 58] examined thediscrete-time GDA dynamics, showing that simultaneous update of two players results in divergencewhile the agents’ strategies remain bounded and cycle when agents take turns to update their strategies.
The majority of existing approaches to stabilizing GDA follow one of three lines of research.
Theessence of the first method is that the discriminator is trained until convergence while the generatorparameters are frozen.
As long as the generator changes slowly enough, the discriminator stillconverges in the presence of small generator perturbations.
The two-timescale update rule proposedby [21, 24, 42] aims to keep the discriminator’s optimality while updating the generator at anappropriate step size.
[25, 14] proved that this two-timescale GDA with finite timescale separationconverges towards the strict local minimax/Stackelberg equilibrium in differentiable games.
[15, 56]explicitly find the local minimax equilibrium in games with secon-order optimization algorithms.
The second line of research overcomes the failure of GDA in games with predictive updates.
Extragradient method (EG) [30] and optimistic gradient descent (OGD) [13] use the predictability of theagents’ strategy to achieve better convergence property.
Their variants are developed to improve thetraining performance of GANs [8, 19, 43].
[45] provided a unified analysis of EG and OGD, showingthat they approximate the classical proximal point methods.
Competitive gradient descent [53]models the agents’ next move by solving a regularized bilinear approximation of the underlying game.
Learning with opponent learning awareness (LOLA) and consistent opponent learning awareness(COLA) [16, 57] introduced opponent shaping to this problem by explicitly modeling the learningstrategy of other agents in the game.
LOLA models opponents as naive learners rather than LOLAagents, while COLA utilizes neural networks to predict opponents’ next move.
Lookahead-minimax[9] stabilizes GAN training by ‘looking ahead’ at the sequence of future states generated by an inneroptimizer.
In the game theory literature, recent work has proposed Clairvoyant Multiplicative Weights
Update (CMWU) for regret minimization in general games [47].
Although CMWU is proposed tosolve finite normal form games, which are different from unconstrained continuous games that Lv.
k
GP aims to solve, both CMWU and Lv.
k GP share the same motivation of enabling the learning agentto update their strategy based on the opponent’s future strategy.
From this aspect, Lv.
k GP can beviewed as a specialized variant of CMWU that is specific to the problem of two-player zero-sumgames, but adapted for unconstrained continuous kernel games.
Other methods directly modify the GDA algorithm with ad-hoc modifications of game dynamicsand introduction of additional regularizers.
Consensus optimization (CO) [40] and gradient penalty[22, 54] improve convergence by directly minimizing the magnitude of players’ gradients.
Symplecticgradient adjustment (SGA) [33, 4, 17] improves convergence by disentangling convergent potentialcomponents from rotational Hamiltonian components of the vector field.
Preliminaries
3.1 Notation
In this paper, vectors are lower-case bold letters (e.g. θ ), matrices are upper-case bold letters(e.g.
A ).
For a function f : R d → R , we denote its gradient by ∇ f .
For functions of twovector arguments f ( x , y ) :
R d 1 × R d 2 → R we use ∇ x f, ∇ y f to denote its partial gradients.
Weuse ∇ xx f, ∇ yy f, ∇ xy f to denote its Hessian.
A stationary point of f denotes the point where∇ x f = ∇ y f = 0 .
We use ∥ v ∥ to denote the Euclidean norm of vector v .
We refer to the largest andsmallest eigenvalues of a matrix A by λ max ( A ) and λ min ( A ) , respectively.
Moreover, we denotethe spectral radius of matrix
A by ρ ( A )
= max {| λ 1 | , . . .
, | λ n |} , i.e., the eigenvalue with largestabsolute value.
3.2 Problem Definition
In order to justify the effectiveness of recursive reasoning procedure, in this paper, we consider theproblem of training Generative Adversarial Networks (GANs)[21].
The GANs training strategydefines a two-player game between a generative neural network G θ ( · ) and a discriminative neural network D ϕ ( · ) .
The generator takes as input random noise z sampled from a known distribution
P z , e.g., z ∼ P z , and outputs a sample G θ ( z ) .
A discriminator takes as input a sample x (eithersampled from the true distribution P x or from the generator) and attempts to classify it as real orfake.
The goal of the generator is to fool the discriminator.
The optimization of GAN is formulatedas a two-player differentiable game where the generator G θ with parameter θ , and the discriminator
D ϕ with parameters ϕ , aim at minimizing their own cost function f
( θ , ϕ ) and g ( θ , ϕ ) respectively,as follows:
[Equation]
where the two function f and g : R m × R n → R .
When f = − g the corresponding optimizationproblem is called a two-player zero-sum game and it becomes a minimax problem:
[Equation]
In this work, we assume the cost functions have Lipschitz continuous gradients with respect to allmodel parameters ( θ , ϕ ) :
Assumption 3.1.
The gradient ∇ θ f
( θ , ϕ ) , is L θθ − Lipschitz with respect to θ and L θϕ − Lipschitzwith respect to ϕ and the gradient ∇ ϕ g ( θ , ϕ ) , is L ϕϕ − Lipschitz with respect to ϕ and
L ϕθ − Lipschitz with respect to θ , i.e.,
[Equation]
∥ ϕ 1 − ϕ 2 ∥ for all θ .
We define L := max { L θθ , L θϕ , L ϕθ , L ϕϕ } .
Level k Gradient Play
In this section, we propose a novel recursive reasoning algorithm, Level k Gradient Play (Lv. k
GP),that allows the agents to discover self-interested strategies while taking into account other agents’reasoning processes.
In Lv.
k GP, k steps of recursive reasoning is applied to obtain an anticipatedfuture state ( θ ( k ) t , ϕ ( k ) t ) , and the current states ( θ t , ϕ t ) are then updated as follows:
Reasoning:
[Equation]
We define the current state ( θ t , ϕ t ) , to be the starting point ( θ (0) t , ϕ (0) t ) , of the reasoning process.
Learning agents that adopt Lv.
k GP strategy are then called Lv.
k agents.
Lv.
1 agents act naively inresponse to the current state using GDA dynamics and Lv.
2 agents act in response to Lv.
1 agents byassuming its opponent as a naive learner.
Therefore, Lv.
k GP allows for higher levels of strategicreasoning.
The inspiration comes from how humans collaborate: humans are great at anticipatinghow their actions will affect others, so they frequently find out how to collaborate with other peopleto reach a "win-win" solution.
The key to human collaboration is their ability to understand howother humans think which helps them develop strategies that benefit their collaborators.
One of ourmain theoretical results is the following theorem, which demonstrates that agents adopting Lv.
k GPcan precisely predict other players’ next move and reach a consensus on their future strategies:
Theorem 4.1.
Suppose Assumption 3.1 holds.
Let us define ω t
=
[ θ t , ϕ t ] T ∈ R m + n , ω ( k ) t =[ θ ( k ) t , ϕ ( k ) t ]
T ∈ R m + n and ∆ max = 2 × max( ∥∇ θ f
( θ t , ϕ t ) ∥ , ∥∇ ϕ g
( θ t , ϕ t ) ∥ ) .
Assume ω ( k ) t lie ina complete subspace of R m + n .
Then for Lv.k GP we have:
[Equation]
Suppose the learning rate satisfies: η < (2 L ) − 1 , then the sequence { ω ( k ) t } ∞ k =0 is a Cauchy sequence.
That is, given ϵ > 0 , there exists N such that, if a > b > N then:
[Equation]
Moreover, the sequence { ω ( k ) t } ∞ k =0 converges to a limit ω ∗ t : lim k →∞ ω ( k ) t = ω ∗ t .
In accordance with Theorem 4.1, if we define ω t +1 = ω ∗ t , then Lv.
∞ GP is equivalent to thefollowing implicit algorithm where we call it Semi-Proximal Point Method:
[Equation]
4.1 Algorithms as an Approximation of SPPM
SPPM players arrive at a consensus by knowing precisely what their opponents’ future strategieswill be.
Existing algorithms are not able to offer this kind of agreement.
For instance, consensusoptimization[40] forces the learning agents to cooperate regardless of their own benefits.
Agentsemploy extra-gradient method[30], SGA[4], and LOLA[16] consider their opponents as naive learners,ignoring their strategic reasoning ability.
CGD[53] takes into account the reasoning process oflearning agents; however, it leads to an inaccurate prediction of agents in games that have costfunctions with non-zero higher order derivatives ( n ≥ 3 ) [57].
In this section, we consider asubset of provably convergent variants of GDA in the Minimax setting, showing that, for specificchoice of hyperparameters, the mentioned algorithms either approximate SPPM or approximate theapproximations of SPPM:
[Table]
SGA [4]
[Equation]
LOLA [16]
[Equation]
LEAD
[Equation]
[Equation]
In Table 1, we compare the orders of precision of different algorithms as an approximation of SPPMin Minimax games with infinitely differentiable objective functions.
In accordance with Equation (3)of Theorem 4.1, Lv.k GP is an O ( η k ) approximation of SPPM.
In order to analyze how well existingalgorithms approximate SPPM, we consider the first-order Taylor approximation to SPPM:
[Equation]
[Table]
GP
[Equation]
Under-brace terms correspond to the first-order Taylor approximation of Lv. 2 GP.
For an appropriatechoice of hyperparameters, SGA ( γ = η ) and LOLA ( δ = η ) are identical to the first-order Taylorapproximation of Lv.2 GP, where each agent models their opponent as a naive learner.
Hence, welist them above the Lv.2 GP, which approximates SPPM up to O ( η 2 ) .
CGD exactly recovers thefirst-order Taylor approximation of SPPM.
1 In games with cost functions that have non-negativehigher order derivatives ( n ≥ 3 ), the remaining term in SPPM’s first-order approximation is an errorof magnitude O ( η 3 ) , which means that CGD’s accuracy is in the same range as that of Lv.
3 GP.
In bilinear and quadratic games where the objective function is at most twice differentiable, CGD is equivalent to SPPM.
The distinction between LEAD ( α = η ) and CGD can be understood byconsidering their update rules.
LEAD is an explicit method where opponents’ potential next strategiesare anticipated based on their most recent move ( ϕ t − ϕ t − 1 ) .
On the contrary, CGD accounts forthis anticipation in an implicit manner, ( ϕ t +1 − ϕ t ) , where the future states appear in current states’update rules.
Therefore, the computation of CGD updates requires solving an function involvingadditional Hessian inverse operations.
A numerical justification is also provided in Table 2, showingthat the approximation accuracy of Lv.
k GP improves as k increases.
Convergence Property
In Theorem 4.1, we have analytically proved that Lv.
k GP convergences asymptotically towards
SPPM, we will use this result to study the convergence property of Lv. k GP in games based on ouranalysis of SPPM.
The local convergence of SPPM in a non convex - non concave game can beanalyzed via the spectral radius of the game Jacobian around a stationary point:
Theorem 5.1.
Consider the (Minimax) problem under Assumption 3.1 and Lv.
k GP.
Let ( θ ∗ , ϕ ∗ )be a stationary point.
Suppose θ t − θ ∗ not in kernel of ∇ ϕθ f ( θ ∗ , ϕ ∗ ) , ϕ t − ϕ ∗ not in kernel of∇ θϕ f ( θ ∗ , ϕ ∗ ) and η < ( L ) − 1 .
There exists a neighborhood U of ( θ ∗ , ϕ ∗ )
such that if SPPM startedat ( θ 0 , ϕ 0 ) ∈ U , the iterates { θ t , ϕ t } t ≥ 0 generated by SPPM satisfy:
[Equation]
where f ∗ =
f ( θ ∗ , ϕ ∗ ) .
Moreover,for any η satisfying:
[Equation]
SPPM converges asymptotically to ( θ ∗ , ϕ ∗ ) .
Remark 5.1.
Following the same condition as in 5.1, the iterates generated by Lv.
2 k GP satisfies: [Equation] wherea =  
[Equation]
odd k ,
[Equation]
even k and b =
[Equation]
[Figure]
[Equation]
[Figure]
We assume that the difference between the trajectory{ θ t , ϕ t } t ≥ 0 and the stationary point ( θ ∗ , ϕ ∗ ) is not inthe kernel of ∇
ϕθ f ( θ ∗ , ϕ ∗ ) and ∇ θϕ f ( θ ∗ , ϕ ∗ ) for thesake of simplicity.
Detailed proofs without this assumption are provided in Appendix.
Following the same setting as Theorem 4.1, we have η <
L − 1 , which ensuresthat η 2 λ max ( ∇ θϕ f ∗ ∇ ϕθ f ∗ ) < 1 .
Therefore, as k → ∞ ,a → 1 and b → 0 in Remark 5.1, and as such Lv.
k GP hassimilar local convergence properties to SPPM. Figure 2illustrates this property in a quadratic game. Lv. k GP maybehave differently than SPPM at lower values of k , butas k increases, both max step size η max ,k and distance toequilibrium for a fixed number of iterations under η max , ∞converges to that of SPPM.
Thus, we observe that Lv.
k
GP is empirically similar to SPPM at higher values of k .
We study the global convergence property of SPPM byanalyzing its behavior in bilinear and quadratic games.
Consider the following bilinear game:
[Equation]
where M is a full rank matrix.
The following theorem summarizes SPPM’s convergence property:
Theorem 5.2.
Consider the Bilinear game and the SPPM method.
Further, we define r t =
∥
θ t −θ ∗ ∥ 2 + ∥ ϕ t − ϕ ∗ ∥ 2 .
Then, for any η > 0 , the iterates { θ t , ϕ t } t ≥ 0 generated by SPPM satisfy
[Equation]
It is worth noting that the convergence property of Lv.
k GP in bi-linear game has been studied in [2].
Furthermore, we study the convergence property of SPPM in the Quadratic game:
[Equation]
where A ∈ R n × n is symmetric and positive definite, B ∈ R n × n is symmetric and negative definiteand the interaction term C ∈ R n × n is full rank.
SPPM in quadratic games converges with thefollowing rate:
Theorem 5.3.
Consider the Quadratic game and the SPPM. Then, for any η > 0 , the iterates{ θ t , ϕ t } t ≥ 0 generated by SPPM satisfy [Equation] [Figure]
SPPM (CGD) 1010i n t e r a c t i on c
Theorem 5.1,5.2 and 5.3 indicate that stronger interaction ∇ θϕ f ( θ ∗ , ϕ ∗ ) improve the convergencerate towards the stationary points.
By contrast, in existing modifications of GDA, the step size ischosen in inversely proportional to the interaction term ∇ θϕ f ( θ ∗ , ϕ ∗ ) [40, 13, 34].
In Figure 3, weshowcase the effect of interaction on different algorithms in the Quadratic game setup with dimensionn = 5 and the interaction matrix is defined as C
= c
I .
Stronger interaction corresponds to highervalues of c .
A key difference between SPPM and Lv.
k GP in the experiments is that, SPPM convergeswith any step size - and so arbitrarily fast - while it is not the case for Lv.
k GP.
In order to approximate
SPPM, one needs Lv.
k GP to be a contraction and so have η < L − 1 .
This result implies an additionalbound on step sizes for Lv.
k GP.
As long as the constraint on η is satisfied, stronger interaction onlyimproves convergence for higher order Lv.
k GP while all other algorithms quickly diverge as η and cincrease.
Experimental Results
In this section, we discuss our implementation of Lv.
k GP algorithm for training GANs.
Its performance is evaluated on 8-Gaussians and two representative datasets CIFAR-10 and STL-10.
6.1 Level k Adam
We propose to combine Lv.
k GP with the Adam optimizer [29].
Preliminary experiments findthat Lv.
k Adam to converge much faster than Lv.
k GP, see A.8 for experiment results.
A detailedpseudo-code for Level k Adam (Lv. k Adam) on GAN training with loss functions L G and L D isgiven in Algorithm 1.
For the Adam optimizer, there are several possible choices on how to updatethe moments.
This choice can lead to different algorithms in practice.
Unlike [19] where the momentsare updated on the fly, in Algorithm 1, we keep the moments fixed in the reasoning steps and updateit together with model parameters.
In Table 2, our experiment result suggests that the proposed Lv. k
Adam algorithm converges asymptotically as the number of reasoning steps k increases.
Input: Stopping time T , reasoning steps k , learning rate η θ , η ϕ , decay rates for momentum estimates β 1 , β 2 , initial weight ( θ 0 , ϕ 0 ) , P x and P z real and noise-data distributions, losses L G ( θ , ϕ , x , z ) and L D ( θ , ϕ , x , z ) , ϵ = 1 e − 8 .
Parameters : Initial parameters: θ 0 , ϕ
Initialize first moments: m θ, 0 ←− 0 , m ϕ, 0 ←−
Initialize second moments: v θ, 0 ←− 0 , v ϕ, 0 ←−for t=0,...,T-1 do
Sample new mini-batch: x , z ∼ P x , P z , θ (0) t ←− θ t , ϕ (0) t ←− ϕ t , for n=1,...,k do
Compute stochastic gradient:
[Equation]
Update estimate of first moment:
[Equation]
Update estimate of second moment: v
[Equation]
Correct the bias for the moments: ˆ
[Equation]
) , ˆ v ( n ) ϕ,t
= v ( n ) ϕ,t (1 − β t 2 )
Perform Adam update: θ ( n ) t = θ t − η θ ˆ m ( n ) θ,t (cid:113) ˆ v ( n ) θ,t + ϵ ; ϕ ( n ) t = ϕ
t − η ϕ ˆ m ( n ) ϕ,t (cid:113) ˆ v ( n ) ϕ,t
+ ϵ θ t +1 ←− θ ( k ) t , ϕ t +1 ←− ϕ ( k ) t ; m θ,t ←− m ( k ) θ,t , m ϕ,t ←− m ( k ) ϕ,t ; v θ,t ←− v ( k ) θ,t , v ϕ,t ←− v ( k ) ϕ,t
6.2 8-Gaussians
In our first experiment, we evaluate Lv.
k Adam on generating a mixture of 8-Gaussianswith standard deviations equal to 0 . 05 and modes uniformly distributed around the unit circle.
We use a two layer multi-layer perceptron with ReLU activations, latent dimensionof 64 and batch size of 128 .
The generated distribution is presented in Figure 4.
[Table]
In addition to presenting the mode coverage of the generated distribution after training, we also study the convergence of the reasoning steps of Lv.
k Adam and Lv.
k GP.
Let us define the difference between the states of Lv.
k
and
Lv.
k − 1 agents at time t as:
[Equation]
We measure the difference averaged over the first 100 iterations, 1100 (cid:80) 100 t =1 r kt , for Lv.
10 Adam ( η = 10 − 4 ) and
Lv. 10 GP ( η = 10 − 2 ).
The result presented in Table2demonstrates that both Lv.
k GP and Lv.
k Adam are con-
[Figure]
A v e r age t i m
[Figure]
pe r i t
[Figure]
r a t i
[Figure]
verging as k increases.
Moreover, the estimation precision of Lv.
{+k GP improves rapidly and convergesto 
+}
within finite steps, making it an accurate estimation for SPPM.
[Table]
6.3 Image Generation Experiments
We evaluate the effectiveness of our Lv.
k
Adam algorithm on unconditional generation of CIFAR-10[31].
We use the Inception score (the higher the better) [50] and the Fréchet Inception distance(the lower the better)
[24] as performance metrics for image synthesis.
For architecture, we use the
SN-GAN architecture based on [44].
For baselines, we compare the performance of Lv.
6 Adam to thatof other first-order and second-order optimization algorithms with the same SN-GAN architecture,and to state-of-the-art models trained with Adam.
For Lv.
k Adam, we use β 1 = 0 and β 2 = 0 .
9 forall experiments.
We use different learning rates for the generator ( η θ = 4 e − 5 ) and the discriminator( η ϕ = 2 e − 4 ).
We train Lv.
k Adam with batch size 128 for 600 epochs.
For testing, we use anexponential moving average of the generator’s parameters with averaging factor β = 0 .
[Table]
In table 3, we present the performance of our method and baselines.
The best FID score and Inceptionscore, 10 . 17 ± 0 .
16 and 8 .
78 ± 0 .
{+06 , on SN-GAN are obtained with our Lv. 
+}
Adam.
We alsooutperform BigGAN and StyleGAN2 in terms of FID score.
Notably, our model has 5.1M parametersin total, and is trained with a small batch size of 128, whereas BigGAN uses 158.3M parameters anda batch size of 2048.
The effect of k and losses: We evaluate values of k = { 2 , 4 , 6 } on non-saturated loss [21] (nonzero-sum formulation) and hinge loss [35] (zero-sum formulation).
The result is presented in Table4.
Remarkably, our experiments demonstrate that few steps of recursive reasoning can result insignificant performance gain comparing to existing GAN optimizers.
The gradual improvements inthe FID scores justify the idea that better estimation of opponents’ next move improves performance.
Moreover, we observe performance gains in both zero-sum and non-zero-sum formulations whichsupplement our theoretical convergence guarantees in zero-sum games.
Experiment results on STL-10:
To test whether the proposed Lv.
k
Adam optimizer works onhigher resolution images, we evaluate its performance on the STL-10 dataset [11] with 3 × 48 × 48resolutions.
In our experiments, Lv. 6 Adam obtained an averaged FID of 25 .
43 ± 0 .
18 whichoutperforms that of the Adam optimizer, 30 .
25 ± 0 .
26 , using the same SN-GAN architecture.
Memory and computation cost: Compared to SGD, Lv.
k Adam requires the same extra memory asthe EG method (one additional set of parameters per player).
The relative cost of one iteration versus
SGD is a factor of k and the computational cost increases linearly as k increases, we illustrate thisrelationship in Figure 5 (right).
We provide an accelerated version of Lv.
k Adam in A.8 which reducesthe computation cost by half for k > 2 .
In Figure 5 (top-left), we compare the FID scores obtainedby Lv.
k Adam, Adam, and LEAD on CIFAR-10 over the same number of gradient computations.
LEAD, Lv4 Adam, and Lv6 Adam all outperform Adam in this experiment.
{+Lv
+}
Adam outperforms
LEAD after 2 × 10 5 gradient computations.
Our method is also compared with different algorithmson the 8-Gaussian problem in terms of its computational cost.
On the same architecture with differentwidths, Figure 5 (bottom-left) illustrates the wall-clock time per computation for different algorithms.
We observe that the computational cost of Lv.
k Adam while being much lower than CGD, is similarto LEAD, SGA and CO which involve JVP operations.
Each run on CIFAR-10 dataset takes 30 ∼ 33hours on a Nvidia RTX3090 GPU.
Each experiment on STL-10 takes 48 ∼ 60 hours on a Nvidia
RTX3090 GPU.
Conclusion and Future Work
This paper proposes a novel algorithm:
Level k gradient play, capable of reasoning about players’future strategies.
We achieve an average FID score of 10.17 for unconditional image generation on
CIFAR-10 dataset, allowing GAN training on common computing resources to reach state-of-the-artperformance.
Our results suggest that Lv.
k GP is a flexible add-on that can be easily attached toexisting GAN optimizers (e.g., Adam) and provides noticeable gains in performance and stability.
Infuture work, we will examine the effectiveness of our approach on more complicated GAN designs,such as Progressive GANs [26] and StyleGANs [27], where optimization plays a more significantrole.
Additionally, we intend to examine the convergence property of Lv.
k GP in games with morethan two player in the future.
Broader Impact Our work introduces a novel optimizer that improves the performance of GANsand may reduce the amount of hyperparameter tuning required by practitioners of generative modeling.
Generative models have been used to create illegal content(a.k.a.
deepfakes [5]).
There is risk ofnegative social impact resulting from malicious use of the proposed methods.
Checklist
For all authors...
[Table]
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes]
[Table]
(b) Did you describe the limitations of your work?
[Yes] See Section 3 and 4(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] SeeSection 7(d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 3and
[Table]
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix 1-43.
If you ran experiments...
[Table]
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
[Table]
mental results (either in the supplemental material or as a URL)?
[Yes] We will provide
[Table]
the code needed to reproduce our main experimental results.
[Table]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Appendix 5-
[Table]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Section 6, all of our experiment results are averaged
[Table]
over multiple runs with different initializations.
[Table]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section 64.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cite CIFAR-10and STL-10 in section 6, and describe libraries that we use in appendix 5-7.
[Table]
(b) Did you mention the license of the assets?
[Yes] In appendix 7.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
We did not publish any new asset.
(d) Did you discuss whether and how consent was obtained from people whose data
you’re using/curating?
[Yes] We did not curate any new datasets.
Existing dataset arediscussed in appendix
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes] In appendix 7.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Appendix
A.1 Proof of Theorem 4.1
Proof.
To begin with, let us consider Lv.
1 GP:
[Equation]
The differences between ( θ (1) t , ϕ (1) t ) and ( θ (0) t , ϕ (0) t ) are:
[Equation]
Recall our definition of ∆ max we have:
[Equation]
Then, with ω t = [ θ t , ϕ t ] T and ω ( k ) t =
[ θ ( k ) t , ϕ ( k ) t ] T , the differences between Lv. 2 agents and Lv.
1agents are:
[Equation]
Recall that L := max { L θθ , L θϕ , L ϕθ , L ϕϕ } , using Equation (9) we have:
[Equation]
Similarly, we can derive the differences between Lv. 3 and Lv.
2 agents:
[Equation]
[Equation]
Consequently, the difference between any two consecutive states k and k − 1 are upper bounded by:
[Equation]
Since ∥ ω ( k ) t − ω ( k − 1) t ∥ ≤ ∥ θ
[Equation]
k
[Equation]
[Equation]
∥
[Equation]
∥ ϕ
( k
[Equation]
t − ϕ ( k −
[Equation]
t ∥ we have:
[Equation]
Suppose η < (2 L ) − 1 , such that the difference between any two consecutive states is a contraction,then we consider the difference, ∥ ω ( a ) t − ω ( b ) t ∥ , where a > b > 0 .
We can rewrite it as:
[Equation]
Since η < (2 L ) − 1 , we have that ηL < 1 and for any ϵ > 0 , we can solve for b such thatη b L ( b − 1) ∆ max < ϵ .
Therefore the sequence { ω kt } ∞ k =0 is a Cauchy sequence.
Moreover, in acomplete space, every Cauchy sequence has a limit: lim k →∞ ω ( k ) t =
ω ∗ t
A.2 Proof of Theorem
Theorem A.1.
Consider the (Minimax) problem under Assumption 3.1 and Lv.
k GP.
Let ( θ ∗ , ϕ ∗ )be a stationary point.
Suppose θ t − θ ∗ not in kernel of ∇ ϕθ f ( θ ∗ , ϕ ∗ ) , ϕ t − ϕ ∗ not in kernel of∇ θϕ f ( θ ∗ , ϕ ∗ ) and η < ( L ) − 1 .
There exists a neighborhood U of ( θ ∗ , ϕ ∗ )
such that if SPPM startedat ( θ 0 , ϕ 0 ) ∈ U , the iterates { θ t , ϕ t } t ≥ 0 generated by SPPM satisfy:
[Equation]
where f ∗ =
f ( θ ∗ , ϕ ∗ ) .
Moreover, for any η satisfying:
[Equation]
SPPM converges asymptotically to ( θ ∗ , ϕ ∗ ) . Proof. Consider the learning dynamics:
[Equation] Let us define [Equation]
It follows immediately by linearizing the system about the stationary point ( θ ∗ , ϕ ∗ ) that(cid:20)
ˆ θ t + ˆ ϕ t +
[Equation]
≃
[Equation]
ˆ
[Equation]
(cid:21) Let us denote the Jacobian by [Equation] Then we can rewrite the dynamics around the stationary point as
[Equation]
Similarly, for the other player we have
[Equation]
Let us define the symmetric matrices Q θ =
( I + η 2 BB T ) − 1 , Q ϕ
= ( I + η 2 B T B ) − 1 and
P θ =
( I − η A ) , P ϕ = ( I + η C ) .
Further we define r t =
∥
ˆ θ t +1 ∥ 2 + ∥ ˆ ϕ t +1 ∥ 2 .
Based on thesedefinitions
[Equation]
and the expressions in (53) and (54) we have
[Equation] To simplify the expression in (55) we use the following lemma: Lemma A.1. The matrices Q θ =
( I + η 2 BB T ) − 1 , Q ϕ = [Equation] I + η 2 B T B ) − 1 satisfy the followingproperties: [Equation]
Using this lemma, we can show that [Equation] where the intermediate equality holds as a T b = b T a . Hence, the expression in (55) can be simplifiedas
[Equation]
We simplify equation (58) as follows.
Consider the term involving ˆ θ t .
We have
[Equation] [Equation]
where the last equality follows by replacing Q θ by its definition.
The same procedure follows for theterm involving ϕ t which leads to the expression
[Equation]
Substitute ∥ Q θ P θ ˆ θ t ∥ 2 + η 2 ∥ Q ϕ B T P
θ ˆ θ t ∥ 2 and ∥ Q ϕ P ϕ
ˆ ϕ t ∥ 2 + η 2 ∥ Q θ BP ϕ ˆ ϕ t ∥ 2 in (58) with theexpressions in (23) and (24), respectively, to obtain
[Equation]
Note that, we assume that the trajectory { ˆ θ t , ˆ ϕ t } t ≥ 0 is not in the kernel of BB T and B T B , thus BB T ˆ θ t ̸ = 0 and B T B ˆ ϕ t ̸ = 0 . Now using the expression in (61) and the fact that P θ = P T θ ,
P ϕ
= P T ϕ and BB T and B T B have the same set of non-zero eigenvalues, if we denote the minimumnon-zero eigenvalues by λ min ( BB T ) and λ min ( B T B ) , we can write
[Equation]
Replacing ∥ θ t +1 − θ ∗ ∥ 2 + ∥ ϕ t +1 − ϕ ∗ ∥ 2 and ∥ θ t − θ ∗ ∥ 2 + ∥ ϕ t − ϕ ∗ ∥ 2 with r t +1 and r t we have:
[Equation]
Recall that A = ∇ θθ f ( θ ∗ , ϕ ∗ ) , B = ∇ θϕ f ( θ ∗ , ϕ ∗ ) and C = ∇ ϕϕ f ( θ ∗ , ϕ ∗ ) , therefore for any ηsatisfying that:
[Equation]
we have r t +1 < r t . Since we linearize the system about the stationary point ( θ ∗ , ϕ ∗ ) , there existsa neighborhood U around the stationary point, such that, SPPM started at ( θ 0 , ϕ 0 ) ∈ U convergesasymptotically to ( θ ∗ , ϕ ∗ ) .
A.3 Proof of Remark 5.
Proof.
To prove the local convergence of Lv. k GP in non-convex non-concave games, we firstconsider the update rule of Lv. k GP: Reasoning:
[Equation]
Update:
[Equation]
Similar to Section A.2 let us denote [Equation] and we define the difference between states and stationary points as [Equation]
Linearizing the dynamical system induced by Lv. k GP about the stationary point ( θ ∗ , ϕ ∗ ) we get:
[Equation]
Note, in Lv. k GP, we define θ (0) t = θ t and ϕ (0) t = ϕ t , thus for Lv. 1 GP, we have:
[Equation]
= [Equation]
For Lv.
2 GP, we have:
[Equation]
Substituting ˆ θ (1) t and ˆ ϕ (1) t into the update rule above we get: [Equation]
Similarly, for Lv. 3 and Lv. 4 GP we have: [Equation] and
[Equation] Summarizing the equations above we have that for Lv. 2 k GP, its update can be written as: [Equation]
Similar to Appendix A.2, let us define Q θ =
( I + η 2 BB T ) − 1 , Q ϕ = ( I + η 2 B T B ) − 1 and [Equation]
P θ = ( I − η A ) , P ϕ = ( I + η C ) . Further, we define R ( k ) θ
[Equation]
1 i =0 ( − η 2 BB T ) k ) , R ( k ) ϕ =( (cid:80) k − 1 i =0 ( − η 2 B T B ) k ) and E ( k ) θ = R ( k ) θ − Q θ , E ( k ) ϕ
= R ( k ) ϕ − Q ϕ .
Since η < L − 1 , we have that:
[Equation] and
E
[Equation]
k
[Equation]
θ
[Equation]
Also, from Lemma A.1 and the definition of the error terms, it can be verified that
[Equation]
Then we can rewrite the update rule of Lv. 2 k GP:
[Equation]
Let us consider the following sum:
[Equation]
t − ( − η 2 B T B ) k ˆ ϕ t ∥= (cid:13)(cid:13)(cid:13) ( Q θ + E ( k ) θ ) (cid:104) P θ ˆ θ t − η BP ϕ ˆ ϕ t (cid:105)(cid:13)(cid:13)(cid:13) 2 + (cid:13)(cid:13)(cid:13) ( Q ϕ + E ( k ) ϕ ) (cid:104) η B T P θ ˆ θ t + P ϕ ˆ ϕ t (cid:105)(cid:13)(cid:13)(cid:13)(31)
The R.H.S. of Eq.(31) can be written as:(cid:13)(cid:13)(cid:13) ( Q θ + E ( k ) θ ) (cid:104) P θ ˆ θ t − η BP ϕ ˆ ϕ t (cid:105)(cid:13)(cid:13)(cid:13) 2 + (cid:13)(cid:13)(cid:13) ( Q ϕ + E ( k ) ϕ ) (cid:104) η B T P θ ˆ θ t + P ϕ
ˆ ϕ t (cid:105)(cid:13)(cid:13)(cid:13)=ˆ θ Tt P T θ Q 2 θ P θ ˆ θ t − 2 η ˆ θ Tt P T θ Q 2 θ BP ϕ ˆ ϕ t + η 2 ˆ ϕ Tt P T ϕ B T Q 2 θ BP ϕ ˆ ϕ t +2ˆ θ Tt P T
θ Q θ E ( k ) θ P
θ ˆ θ t − 4 η ˆ θ Tt P T θ Q θ E ( k ) θ BP ϕ ˆ ϕ t + 2 η 2 ˆ ϕ Tt P T ϕ B T Q θ E ( k ) θ BP ϕ ˆ ϕ t +ˆ θ T t P T θ [ E ( k ) θ ] 2 P θ ˆ θ t − 2 η ˆ θ T t P T θ [ E ( k ) θ ] 2 BP ϕ ˆ ϕ t + η 2 ˆ ϕ T t P T ϕ B T [ E ( k ) θ ] 2 BP ϕ ˆ ϕ t + ˆ ϕ Tt P T ϕ Q 2 ϕ P ϕ ˆ ϕ t + 2 η ˆ θ
Tt P T θ BQ 2 ϕ P ϕ
ˆ ϕ t + η 2 ˆ θ Tt P T θ BQ 2 ϕ B T P θ ˆ θ t +2 ˆ ϕ
Tt P T ϕ Q ϕ E ( k ) ϕ
P ϕ
ˆ ϕ t + 4 η ˆ θ Tt P T θ BQ ϕ E ( k ) ϕ
P ϕ
ˆ ϕ t + 2 η 2 ˆ θ Tt P T θ BQ ϕ E ( k ) ϕ
B T P θ ˆ θ t
+ ˆ ϕ
Tt P T ϕ
[ E ( k ) ϕ ] 2 P ϕ ˆ ϕ t + 2 η ˆ θ Tt P T θ B [ E ( k ) ϕ ] 2 P ϕ ˆ ϕ t + η 2 ˆ θ Tt P T θ B [ E ( k ) ϕ ] 2 B T P θ ˆ θ t (32) Now, before adding all terms in Eq.(32), note that all of the cross terms in Eq.(32) cancel out.
For instance, using Lemma A.1 and Eq.(29), Eq.(30) we can show that
[Equation]
By using similar arguments it can be shown that terms in Eq.(32) leads to:
[Equation]
Similar to Eq.(23) we have the following simplification:
[Equation]
Now we can further simplify Eq.(32) as:
[Equation]
Using Eq.(27) and Eq.(28) and definition of Q θ and Q ϕ we have:
[Equation]
Similarly
[Equation]
we have that
[Equation]
Thus we simplify the R.H.S. of Eq.(31) as(cid:13)(cid:13)(cid:13)
( Q θ + E ( k ) θ ) (cid:104) P
[Equation]
ˆ θ t
[Equation]
η BP
[Equation]
ˆ ϕ t (cid:105)(cid:13)(cid:13)(cid:13) 2 + (cid:13)(cid:13)(cid:13) ( Q ϕ + E ( k ) ϕ ) (cid:104) η B T P
θ ˆ θ t + P ϕ
ˆ ϕ t (cid:105)(cid:13)(cid:13)(cid:13)
[Equation] [Equation] Let us consider the L.H.S. of Eq.(31) [Equation]
Substituting Eq.(35) and Eq.(34) into L.H.S. and R.H.S. of Eq.(31) respectively we get:
[Equation]
Now we have the following equation:
[Equation]
Note that
[Equation] Similarly [Equation] Summing everything together we have:
[Equation]
) k
(ˆ θ t ) − ∥ ( − η 2 BB T ) k
ˆ θ t ∥+
(( I − ( − η 2 B T B ) k ) P ϕ ˆ ϕ t ) T ( I + η 2 B T B ) − 1 (( I − ( − η 2 B T B ) k ) P ϕ ˆ ϕ t ) +( ˆ ϕ t ) T ( η 2 B T B ) k ( ˆ ϕ t ) − ∥ ( − η 2 B T B )
k ˆ ϕ t ∥
Note that, we assume that the trajectory { ˆ θ t , ˆ ϕ t } t ≥ 0 is not in the kernel of BB T and B T B , thus
BB T ˆ θ t ̸ = 0 and B T B ˆ ϕ t ̸
= 0 .
Now using the expression in (61) and the fact that P θ = P T θ ,
P ϕ
= P T ϕ and BB T and B T B have the same set of non-zero eigenvalues, if we denote the minimum non-zero eigenvalues by λ min ( BB T ) and λ min ( B T B ) , we can write
[Equation]
Let us define the distance as: [Equation] [Equation] Then we have∥
r
[Equation]
t
[Equation] where [Equation] even
[Equation]
and {+[Equation] A.3.  Proof of Lemma A.
+}
Let B = U Σ V T be the singular value decomposition of B .
Here U and V are orthonormal matricesand Σ is a rectangular diagonal matrix.
Then we have:
[Equation]
Here we used the fact that U and V are orthonormal matrices.
Now, we simplify the other side toget:
[Equation]
Now we consider the following equation:
[Equation]
which indicates that Σ ( η 2 Σ T Σ + I )
=
( η 2 ΣΣ T + I ) Σ .
Multiplying both sides of this equation by( η 2 ΣΣ T + I ) − 1 and ( η 2 Σ T Σ + I ) − 1 we have:
[Equation]
Therefore, we have Q θ B = BQ ϕ .
Using a similar argument, we can also prove the equality in
Equation (21).
A.4 Theorem 5.1 without kernel assumption
Theorem A.2.
Consider the (Minimax) problem under Assumption 3.1 and Lv.
k GP.
Let ( θ ∗ , ϕ ∗ )be a stationary point. Suppose η < ( L ) − 1 . There exists a neighborhood U of ( θ ∗ , ϕ ∗ ) such that if SPPM started at ( θ 0 , ϕ 0 ) ∈ U , the iterates { θ t , ϕ t } t ≥ 0 generated by SPPM satisfy:
[Equation] where f ∗ = f ( θ ∗ , ϕ ∗ ) . Moreover, for any η satisfying:
[Equation] SPPM converges asymptotically to ( θ ∗ , ϕ ∗ ) . Proof. Following the same setting and procedure as in Appendix A.2, we have that
[Equation] Now using the fact that P θ = P T θ , P ϕ = P T ϕ , we can write [Equation]
For any η that satisfying [Equation] [Equation] we have that
[Equation] [Equation]
i.e., SPPM converges asymptotically towards ( θ ∗ , ϕ ∗ ) .
A.5 Proof of Theorem 5.2
Proof.
In order to proof the convergence of SPPM in bilinear games, we first show that the SPPMupdate rule is equivalent to that of the following Proximal Point Method:
[Equation]
In the Bilinear game, the SPPM update is:
[Equation]
and the PPM update is:
[Equation]
Thus SPPM and PPM are equivalent in the Bilinear game.
The convergence result of PPM in bilineargames has been proved in Theorem 2 of [49]:
Theorem A.3.
Consider the Bilinear game and the PPM method.
Further, we define r t = ∥ θ t −θ ∗ ∥ 2 + ∥ ϕ t − ϕ ∗ ∥ 2 . Then, for any η > 0 , the iterates { θ t , ϕ t } t ≥ 0 generated by SPPM satisfy
[Equation]
Therefore, SPPM and PPM have the same convergence property in bilinear games.
Proof. Consider the learning dynamics: [Equation]
In the Quadratic game, the SPPM update rule can be written as:
[Equation]
Then we can rewrite the learning dynamics:
[Equation]
[Equation]
Similarly, for the other player we have
[Equation]
[Equation]
Let us define the symmetric matrices Q θ =
( I + η 2 CC T ) − 1 , Q ϕ
= ( I + η 2 C T C ) − 1 and
P θ = ( I − η A ) , P ϕ = ( I + η B ) . Further we define r t =
∥
θ t +1 ∥ 2 + ∥ ϕ t +1 ∥ 2 .
Based on thesedefinitions, and the expressions in (53) and (54) we have
[Equation]
To simplify the expression in (55) we use Lemma A.1 to obtain the following equations:
[Equation]
[Equation]
[Equation]
Using this lemma, we can show that [Equation] where the intermediate equality holds as a T C = C T a . Hence, the expression in (55) can besimplified as
[Equation]
We simplify equation (58) as follows.
Consider the term involving θ t . We have [Equation] where the last equality follows by replacing Q θ by its definition.
The same procedure follows for theterm involving ϕ t which leads to the expression
[Equation] Substitute ∥ Q θ P θ θ t ∥ 2 + η 2 ∥ Q ϕ C [Equation] P θ θ t ∥ 2 and ∥ Q ϕ P ϕ ϕ t ∥ 2 + η 2 ∥ Q θ CP ϕ ϕ t ∥ 2 in (58) with theexpressions in (59) and (60)
[Equation]
respectively
[Equation]
to obtain
[Equation]
Now using the expression in (61) and the fact that P θ = P T θ , P ϕ
= P T ϕ and λ min ( C T C ) =
λ min ( CC T ) , we can write
[Equation]
In this section, we justify our results in Section 4.1 that Competitive Gradient Descent is a first order Taylor approximation of SPPM. Firstly, we consider the standard definition of CGD:
[Equation]
Rewriting the update rules we can get:
[Equation]
Similarly, we have:
[Equation]
Therefore, CGD is a first order approximation of SPPM.
Then we prove that the standard definition of
CGD is equivalent to the update rule in Table 1.
Consider the update rule in Table 1 and its footnote,we have:
[Equation]
Substituting ( ϕ t +1 − ϕ t ) into the first equation of (62) we get:
[Equation]
ϕ t ) ∇ ϕ f
( θ t , ϕ t )) .
Substituting ( θ t +1 − θ t ) into the second equation of (62) and applying similar arguments we get:
[Equation]
Thus the update rule in Table 1 is equivalent to the standard definition of CGD and it is equivalent tothe first order Taylor approximation of SPPM. A.8 Experiments on Bilinear and Games Bilinear Game Consider the following bilinear game:
[Figure]
(63) the example presented in Figure 1 is an example of using different algorithms to solve the bilineargame above with coefficient a = 10 .
For sake of completeness, we also provide a grid of experimentresults for different algorithms with different coefficients a and learning rates η , starting from thesame point ( θ 0 , ϕ 0 ) =
( − 12 , 10) .
The result is presented in Figure 6.
The experiment demonstrates
[Figure]
c oe ff i c i en t a
688 that, in a bilinear game, Lv. 2 GP is equivalent to the extra-gradient method, and higher level Lv.
k GPperforms better with increased coefficient a and learning rate η
as long as it remains a contraction(i.e.
, η < a − 1 ).
Quadratic Game For the quadratic game presented in Figure 3, we randomly initialize the matrices
A and B :
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
7664 2 . 
[Equation]
  1 . 0821 1 . 2427 1 . 0093 1 . 3335 0 .1 .
9693 0 .  where A is symmetric and positive definite and B is symmetric and negative definite. The interactionmatrix is defined as: [Equation]  where c represents the strength of the interaction between the two players.
The starting point θ 0 andϕ 0 are [0 . 1270 , 0 . 9667 , 0 . 2605 , 0 . 8972 , 0 . 3767]
T and [0 .
T respectively.
[Figure]
i n t e r a c t i on c
A.9 Experiments on 8-Gaussians
Dataset The target distribution is a mixture of 8-Gaussians with standard deviation equal to 0 .
[-k GP improves rapidly and convergesto 
-]05and modes uniformly distributed around a unit circle.
Architecture The architecture for the generator and the discriminator, each consists of four hiddenlayers followed by ReLU activation function. The weight initialization uses PyTorch’s defaultinitialization scheme.
The architecture is presented in Table 8:
Experiment For our experiments, we used the PyTorch framework. Furthermore, the batch size weused is 128 . Generator
Discriminator
[Table] A.10 Experiments on CIFAR-10 and STL-10 For our experiments, we used the PyTorch 2 framework. For experiments on CIFAR-10 and STL10, we compute the FID and IS metrics using the provided implementations in Tensorflow 3 forconsistency with related works.
[Figure]
Lv.
k GP vs Lv.
k Adam
In experiments, we compare the performance of Lv.
k GP and Lv.
k Adamon the task of CIFAR-10 image generation. The experiment results is presented in Figure 8.
Theexperiments on Lv.
k GP and Lv.
k
Adam use the same initialization and hyperparameters.
Accordingto our experiments, Lv. k Adam converges much faster than Lv.[-
 Adam.-]
k GP for the same choice of k andlearning rates.
Adam vs Lv.
k
Adam We also present a comparison between the performance of Adam and Lv.
k
Adam optimizers on the task of STL-10 image generation. The experiment results is presented in Figure 9. Under the same choice of hyperparameters and identical model parameter initialization,
Lv.
k Adam consistently outperforms the Adam optimizer in terms of FID score.
Accelerated Lv.
k Adam
In this section, we propose an accelerated version of Lv.
k Adam.
Theintuition is that we update the min player θ and the max player ϕ in an alternating order. Thecorresponding Lv.
k GP algorithm can be writen as:
Reasoning: [Equation] pytorch [Equation]
[-Lv
-].org/
[Figure] Adam
Lv.6 Adam
Instead of responding to θ (
, in Alt-Lv. k GP, the max player ϕ ( k ) t acts in response to the minplayer’s current action, θ ( k ) t . A Lv.
k min player in Alt-Lv. k GP is equivalent to a Lv. 2 k − 1 player inthe Lv.
k GP, and a Lv.
k max player in Alt-Lv.
k GP is equivalent to a Lv. 2 k player in the Lv k GP,respectively.
Therefore, it is easy to verify that Alt-Lv.
k GP converges two times faster than Lv. k GPand the corresponding Alt-Lv. k
Adam algorithm is provided in Algorithm 2. Architecture In this section, we describe the model we used to evaluate the performance of Lv.
k
Adam for generating CIFAR-10 4 and STL-10 datasets.
With ’conv’ we denote a convolutional layerand ’transposed conv’ a transposed convolution layer.
The models use Batch Normalization and
Spectral Normalization.
The model’s parameters are initialized with Xavier initialization.
Images generated on CIFAR-10 and STL-10 In this section, we present sample images generatedby the best performing trained generators on CIFAR-10 and STL-10.
Input: Stopping time T , reasoning steps k , learning rate
, decay rates for momentum estimates β 1 , β 2 , initial weight ( θ 0 , ϕ 0 ) , P x and P z real and noise-data distributions, losses L G ( θ , ϕ , x , z ) and L D ( θ , ϕ , x , z ) , ϵ = 1 e − 8 .
Parameters : Initial parameters: θ 0 , ϕ
Initialize first moments: m θ, 0 ←− 0 , m ϕ, 0 ←−
Initialize second moments: v θ, 0 ←− 0 , v ϕ, 0 ←−for t=0,...,T-1 do
Sample new mini-batch: x , z ∼ P x , P z , θ (0) t ←− θ t , ϕ (0) t ←− ϕ t , for n=1,...,k do Compute stochastic gradient: g ( n ) θ ,t = ∇ θ L G ( θ t , ϕ ( n − 1) t , x , z );
Update estimate of first moment: m ( n ) θ,t = β 1 m θ,t − 1 + (1 − β 1 ) g ( n ) θ,t ; Update estimate of second moment: v ( n ) θ,t = β 2 v θ,t − 1 + (1 − β 2 )( g ( n ) θ,t ) 2 ; Correct the bias for the moments: ˆ m ( n ) θ,t = m ( n ) θ,t (1 − β t 1 ) , ˆ v ( n ) θ,t = v ( n ) θ,t (1 − β t 2 ) ; Perform Adam update: θ ( n ) t = θ t − η θ ˆ m ( n ) θ,t (cid:113) ˆ v ( n ) θ,t + ϵ ;
Compute stochastic gradient: g ( n ) ϕ ,t = ∇ ϕ L D ( θ ( n ) t , ϕ t , x , z );
Update estimate of first moment: m ( n ) ϕ,t = β 1 m ϕ,t − 1 + (1 − β 1 ) g ( n ) ϕ,t ; Update estimate of second moment:
[Table]
Correct the bias for the moments:
[Table]
Perform
[Table]
G-ResBlock
Shortcut: Upsample( × 2 ) Residual:
Batch NormalizationReLU Upsample( × 2 ) conv (ker: 3 × 3 , 256 → 256 ; stride: 1 ; pad: 1 )
Batch Normalization
ReLU conv (ker: 3 × 3 , 256 → 256 ; stride: 1 ; pad: 1 )
D-ResBlock ( l − th block)
Shortcut: [ AvgPool (ker: 2 × 2 ) ] , if l =conv (ker: 1 × 1 , 3 l =1 / 128 l ̸ =1 → 128 ; stride: 1 ) Spectral Normalization [ AvgPool (ker: 2 × 2 , stride: 2 ) ] , if l = Residual: [ ReLU ] , if l ̸ =conv (ker: 3 × 3 , 3 l =1 / 128 l ̸ =1 → 128 ; stride: 1 ; pad: 1 )
Spectral Normalization ReLU conv (ker: 1 × 1 , 128 → 128 ; stride: 1 )
Spectral Normalization AvgPool
[Equation]
[Table]
[Table]