Recursive Reasoning in Minimax Games: A Level k Gradient Play Method
Anonymous Author(s)
AffiliationAddress
Abstract
Despite the success of generative adversarial networks (GANs) in generating visually appealing images, they are notoriously challenging to train.
In order tostabilize the learning dynamics in minimax games, we propose a novel recursivereasoning algorithm:
Level k Gradient Play (Lv. k GP) algorithm.
Our algorithmdoes not require sophisticated heuristics or second-order information, as do existing algorithms based on predictive updates.
We show that as k increases, Lv.
k
GP converges asymptotically towards an accurate estimation of players’ futurestrategy.
Moreover, we justify that Lv.
∞ GP naturally generalizes a line of provably convergent game dynamics which rely on predictive updates.
Furthermore,we provide its local convergence property in nonconvex-nonconcave zero-sumgames and global convergence in bilinear and quadratic games.
By combining
Lv.
k GP with Adam optimizer, our algorithm shows a clear advantage in termsof performance and computational overhead compared to other methods.
Usinga single Nvidia RTX3090 GPU and 30 times fewer parameters than BigGAN on
CIFAR-10, we achieve an FID of 10.17 for unconditional image generation within24 hours, allowing GAN training on common computational resources to reachstate-of-the-art performance.
Introduction
In recent years, there has been a surge of powerful models that require simultaneous optimization ofseveral objectives.
This increasing interest in multi-objective optimization arises in various domainssuch as generative adversarial networks [21, 27], adversarial attacks and robust optimization [37, 7],and multi-agent reinforcement learning [36, 54] - where several agents aim at minimizing theirobjectives simultaneously.
Games generalize this optimization framework by introducing differentobjectives for different learning agents, known as players.
The generative adversarial network is awidely-used method of this type, which has demonstrated state-of-the-art performance in a variety ofapplications, including image generation [28, 51], image super-resolution [32], and image-to-imagetranslation [10].
Despite their success at generating visually appealing images, GANs are notoriouslychallenging to train [40, 39].
Naive application of the gradient-based algorithm in GANs often leadsto poor image diversity (sometimes manifesting as "mode collapse")
[40], Poincare recurrence [39],and subtle dependency on hyperparameters [18].
An immense corpus of work is devoted to exploringand enhancing the stability of GANs, including techniques as diverse as the use of optimal transportdistance [1, 22], critic gradient penalties [53], different neural network architectures [26, 6], featurematching [49], pre-trained feature space [50], and minibatch discrimination [49].
Nevertheless,architectural modifications (e.g., StyleGANs [27]) require extensive computational resources, andsome of the second-order game dynamics (Follow-the-ridge [55], CGD[52]) require Hessian inverseoperations, making well-performing models unavailable for standard computational budgets [51, 6].
To stabilize the learning dynamics in GANs, many recent efforts rely on sophisticated heuristics thatallow the agents to anticipate each other’s next move [16, 52, 23].
This anticipation is an example of
[Figure]
Level k Gradient Play - - -     - - -     - - -
More steps in the reasoning process leading to better anticipations and faster convergence.
a recursive reasoning procedure in cognitive science [12].
Similar to how humans think, recursivereasoning represents the belief reasoning process where each agent considers the reasoning processof other agents, based on which it expects to make better decisions.
Importantly, it enables the use ofopponents that reason about the learning agent, rather than assuming fixed opponents; the process cantherefore be nested in a form as ’I believe that you believe that I believe...’.
Based on this intuition,we introduce a novel recursive reasoning algorithm that utilizes only gradient information to optimize
GANs.
Our contributions can be summarized as follows:
(i)
We propose a novel algorithm: level k gradient play (Lv. k GP), which is capable of reasoningabout players’ future strategy.
In a game, agents at Lv.
k adjust their strategies in accordance with thestrategies of Lv.
k − 1 agents.
We justify that, while typical GANs optimizers, such as Learning withOpponent Learning Awareness (LOLA) and Symplectic Gradient Adjustment (SGA), approximateLv. 2 and Lv. 3 GP, our algorithm permits higher levels of strategic reasoning.
In addition, the proposedalgorithm is amenable to neural network optimizers like Adam [29].
(ii) We show that, in smooth games, Lv.
k GP converges asymptotically towards an accurate
predictionof agents’
next move.
Under mutual opponent shaping, two Lv.
∞ agents will naturally have a
consistent view of one another if the Lv.
k
GP converges
as
k increases
Based on this idea, we
provide a closed-form solution for Lv.
∞
GP
: the Semi-Proximal Point Method (SPPM).
(iii)
We prove the local convergence property of Lv.
∞ GP in nonconvex - nonconcave zero-sum
games and its global convergence in bilinear and quadratic games.
The theoretical analysis we
present
indicates that strong interactions between competing agents can increase the convergence rate of Lv.
k
GP
agents in a zero-sum game.
(iv) By
combining Lv.
k GP with Adam optimizer, our algorithm shows a clear advantage in terms ofperformance and computational overhead compared to other methods.
Using a single 3090 GPU with30 times fewer parameters and 16 times smaller mini-batches than BigGAN [6] on CIFAR-10, weachieve an FID score [24] of 10.17 for unconditional image generation within 24 hours, allowing
GAN training on common computing resources to reach state-of-the-art performance.
Related Works
In recent years, minimax problems have attracted considerable interest in machine learning in light oftheir connection with GANs.
Gradient descent ascent (GDA), a generalization of gradient descentfor minimax games, is the principal approach for training GANs in applications.
GDA alternatesbetween a gradient descent step for the min-player and a gradient ascent step for the max-player.
Theconvergence of GDA in games is far from as well understood as gradient descent in single-objectiveproblems.
Despite the impressive image quality generated by GANs, GDA fails to converge even in bilinear zero-sum games.
Recent research on GDA has established a unified picture of its behaviorin bilinear games in continuous and discrete-time [39, 46, 13, 41, 20].
First, [39] revealed thatcontinuous-time GDA dynamics in zero-sum games result in Poincare recurrence, where agentsreturn arbitrarily close to their initial state infinitely many times.
Second, [3, 57] examined thediscrete-time GDA dynamics, showing that simultaneous update of two players results in divergencewhile the agents’ strategies remain bounded and cycle when agents take turns to update their strategies.
The majority of existing approaches to stabilizing GDA follow one of three lines of research.
Theessence of the first method is that the discriminator is trained until convergence while the generatorparameters are frozen.
As long as the generator changes slowly enough, the discriminator stillconverges in the presence of small generator perturbations.
The two-timescale update rule proposedby [21, 24, 42] aims to keep the discriminator’s optimality while updating the generator at anappropriate step size.
[25, 14] proved that this two-timescale GDA with finite timescale separationconverges towards the strict local minimax/Stackelberg equilibrium in differentiable games.
[15, 55]explicitly find the local minimax equilibrium in games with secon-order optimization algorithms.
The second line of research overcomes the failure of GDA in games with predictive updates.
Extragradient method (EG) [30] and optimistic gradient descent (OGD) [13] use the predictability of theagents’ strategy to achieve better convergence property.
Their variants are developed to improve thetraining performance of GANs [8, 19, 43].
[45] provided a unified analysis of EG and OGD, showingthat they approximate the classical proximal point methods.
Competitive gradient descent [52]models the agents’ next move by solving a regularized bilinear approximation of the underlying game.
Learning with opponent learning awareness (LOLA) and consistent opponent learning awareness(COLA) [16, 56] introduced opponent shaping to this problem by explicitly modeling the learningstrategy of other agents in the game.
LOLA models opponents as naive learners rather than LOLAagents, while COLA utilizes neural networks to predict opponents’ next move.
Lookahead-minimax[9] stabilizes GAN training by ‘looking ahead’ at the sequence of future states generated by an inneroptimizer.
In the game theory literature, recent work has proposed Clairvoyant Multiplicative Weights Update (CMWU) for regret minimization in general games [47].
Although CMWU is proposed tosolve finite normal form games, which are different from unconstrained continuous games that Lv. k GP aims to solve, both CMWU and Lv. k GP share the same motivation of enabling the learning agentto update their strategy based on the opponent’s future strategy.
From this aspect, Lv. k GP can beviewed as a specialized variant of CMWU that is specific to the problem of two-player zero-sumgames, but adapted for unconstrained continuous kernel games.
Other methods directly modify the GDA algorithm with ad-hoc modifications of game dynamicsand introduction of additional regularizers.
Consensus optimization (CO) [40] and gradient penalty[22, 53] improve convergence by directly minimizing the magnitude of players’ gradients.
Symplecticgradient adjustment (SGA) [33, 4, 17] improves convergence by disentangling convergent potentialcomponents from rotational Hamiltonian components of the vector field.
Preliminaries
3.1 Notation
In this paper, vectors are lower-case bold letters (e.g. θ ), matrices are upper-case bold letters(e.g.
A ).
For a function f : R d → R , we denote its gradient by ∇ f .
For functions of twovector arguments f ( x , y ) :
R d 1 × R d 2 → R we use ∇ x f, ∇ y f to denote its partial gradients.
Weuse ∇ xx f, ∇ yy f, ∇ xy f to denote its Hessian.
A stationary point of f denotes the point where∇ x f = ∇ y f = 0 .
We use ∥ v ∥ to denote the Euclidean norm of vector v .
We refer to the largest andsmallest eigenvalues of a matrix A by λ max ( A ) and λ min ( A ) , respectively.
Moreover, we denotethe spectral radius of matrix
A by ρ ( A )
= max {| λ 1 | , . . .
, | λ n |} , i.e., the eigenvalue with largestabsolute value.
3.2 Problem Definition
In order to justify the effectiveness of recursive reasoning procedure, in this paper, we consider theproblem of training Generative Adversarial Networks (GANs)[21].
The GANs training strategydefines a two-player game between a generative neural network G θ ( · ) and a discriminative neuralnetwork D ϕ ( · ) .
The generator takes as input random noise z sampled from a known distribution
P z , e.g., z ∼ P z , and outputs a sample G θ ( z ) .
A discriminator takes as input a sample x (eithersampled from the true distribution P x or from the generator) and attempts to classify it as real or fake.
The goal of the generator is to fool the discriminator.
The optimization of GAN is formulatedas a two-player differentiable game where the generator G θ with parameter θ , and the discriminator
D ϕ with parameters ϕ , aim at minimizing their own cost function f
( θ , ϕ ) and g ( θ , ϕ ) respectively,as follows:
[Equation]
where the two function f and g : R m × R n → R .
When f = − g the corresponding optimizationproblem is called a two-player zero-sum game and it becomes a minimax problem:
[Equation]
In this work, we assume the cost functions have Lipschitz continuous gradients with respect to allmodel parameters ( θ , ϕ ) :
Assumption 3.1.
The gradient ∇ θ f
( θ , ϕ ) , is L θθ − Lipschitz with respect to θ and L θϕ − Lipschitzwith respect to ϕ and the gradient ∇ ϕ g ( θ , ϕ ) , is L ϕϕ − Lipschitz with respect to ϕ and
L ϕθ − Lipschitz with respect to θ , i.e.,
[Equation]
for
[Equation]
∥∇
[Equation]
ϕ ) [Equation] ϕ [Equation]
∥
[Equation]
L ϕθ ∥ θ
[Equation]
− θ
[Equation]
∥ for all ϕ
[Equation]
∥∇
[Equation]
( θ , ϕ 1 ) − ∇ ϕ
[Equation]
( θ , ϕ 2 )
∥
≤ L ϕϕ ∥ ϕ 1 − ϕ 2 ∥ for all θ .
We define L := max { L θθ , L θϕ , L ϕθ , L ϕϕ } .
Level k Gradient Play
In this section, we propose a novel recursive reasoning algorithm, Level k Gradient Play (Lv. k
GP),that allows the agents to discover self-interested strategies while taking into account other agents’reasoning processes.
In Lv.
k GP, k steps of recursive reasoning is applied to obtain an anticipatedfuture state ( θ ( k ) t , ϕ ( k ) t ) , and the current states ( θ t , ϕ t ) are then updated as follows:
Reasoning:
[Equation]
Update:
[Equation]
We define the current state ( θ t , ϕ t ) , to be the starting point ( θ (0) t , ϕ (0) t ) , of the reasoning process.
Learning agents that adopt Lv.
k GP strategy are then called Lv.
k agents.
Lv.
1 agents act naively inresponse to the current state using GDA dynamics and Lv.
2 agents act in response to Lv.
1 agents byassuming its opponent as a naive learner.
Therefore, Lv.
k GP allows for higher levels of strategicreasoning.
The inspiration comes from how humans collaborate: humans are great at anticipatinghow their actions will affect others, so they frequently find out how to collaborate with other peopleto reach a "win-win" solution.
The key to human collaboration is their ability to understand howother humans think which helps them develop strategies that benefit their collaborators.
One of ourmain theoretical results is the following theorem, which demonstrates that agents adopting Lv.
k GPcan precisely predict other players’ next move and reach a consensus on their future strategies:
Theorem 4.1.
Suppose Assumption 3.1 holds.
Let us define ω t
=
[ θ t , ϕ t ] T ∈ R m + n , ω ( k ) t =[ θ ( k ) t , ϕ ( k ) t ]
T ∈ R m + n and ∆ max = 2 × max( ∥∇ θ f
( θ t , ϕ t ) ∥ , ∥∇ ϕ g
( θ t , ϕ t ) ∥ ) .
Assume ω ( k ) t lie ina complete subspace of R m + n .
Then for Lv.k GP we have:
[Equation]
Suppose the learning rate satisfies: η < (2 L ) − 1 , then the sequence { ω ( k ) t } ∞ k =0 is a Cauchy sequence.
That is, given ϵ > 0 , there exists N such that, if a > b > N then:
[Equation]
Moreover, the sequence { ω ( k ) t } ∞ k =0 converges to a limit ω ∗ t : lim k →∞ ω ( k ) t = ω ∗ t .
In accordance with Theorem 4.1, if we define ω t +1 = ω ∗ t , then Lv.
∞ GP is equivalent to thefollowing implicit algorithm where we call it Semi-Proximal Point Method:
[Equation]
4.1 Algorithms as an Approximation of SPPM
SPPM players arrive at a consensus by knowing precisely what their opponents’ future strategieswill be.
Existing algorithms are not able to offer this kind of agreement.
For instance, consensusoptimization[40] forces the learning agents to cooperate regardless of their own benefits.
Agentsemploy extra-gradient method[30], SGA[4], and LOLA[16] consider their opponents as naive learners,ignoring their strategic reasoning ability.
CGD[52] takes into account the reasoning process oflearning agents; however, it leads to an inaccurate prediction of agents in games that have costfunctions with non-zero higher order derivatives ( n ≥ 3 ) [56].
In this section, we consider asubset of provably convergent variants of GDA in the Minimax setting, showing that, for specificchoice of hyperparameters, the mentioned algorithms either approximate SPPM or approximate theapproximations of SPPM:
[Table]
SGA [4] θ
[Equation]
t
[Equation]
ηγ
[Equation]
θ
[Equation]
ϕ t ) ——
LOLA [16]
[Equation]
Lv.2 GP
[Equation]
O
[Equation]
LEAD † [23]
[Equation]
In Table 1, we compare the orders of precision of different algorithms as an approximation of SPPMin Minimax games with infinitely differentiable objective functions.
In accordance with Equation (3)of Theorem 4.1, Lv.k GP is an O ( η k ) approximation of SPPM.
In order to analyze how well existingalgorithms approximate SPPM, we consider the first-order Taylor approximation to SPPM:
[Equation]
Under-brace terms correspond to the first-order Taylor approximation of Lv. 2 GP.
For an appropriatechoice of hyperparameters, SGA ( γ = η ) and LOLA ( δ = η ) are identical to the first-order Taylorapproximation of Lv.2 GP, where each agent models their opponent as a naive learner.
Hence, welist them above the Lv.2 GP, which approximates SPPM up to O ( η 2 ) .
CGD exactly recovers thefirst-order Taylor approximation of SPPM.
1 In games with cost functions that have non-negativehigher order derivatives ( n ≥ 3 ), the remaining term in SPPM’s first-order approximation is an errorof magnitude O ( η 3 ) , which means that CGD’s accuracy is in the same range as that of Lv.
3 GP.
In bilinear and quadratic games where the objective function is at most twice differentiable, CGDis equivalent to SPPM.
The distinction between LEAD ( α = η ) and CGD can be understood byconsidering their update rules.
LEAD is an explicit method where opponents’ potential next strategiesare anticipated based on their most recent move ( ϕ t − ϕ t − 1 ) .
On the contrary, CGD accounts forthis anticipation in an implicit manner, ( ϕ t +1 − ϕ t ) , where the future states appear in current states’update rules.
Therefore, the computation of CGD updates requires solving an function involvingadditional Hessian inverse operations.
A numerical justification is also provided in Table 2, showingthat the approximation accuracy of Lv.
k GP improves as k increases.
Convergence Property
In Theorem 4.1, we have analytically proved that Lv.
k GP convergences asymptotically towards
SPPM, we will use this result to study the convergence property of Lv.
k GP in games based on ouranalysis of SPPM.
The local convergence of SPPM in a non convex - non concave game can beanalyzed via the spectral radius of the game Jacobian around a stationary point:
Theorem 5.1.
Consider the (Minimax) problem under Assumption 3.1 and Lv.
k GP.
Let ( θ ∗ , ϕ ∗ )be a stationary point.
Suppose θ t − θ ∗ not in kernel of ∇ ϕθ f ( θ ∗ , ϕ ∗ ) , ϕ t − ϕ ∗ not in kernel of∇ θϕ f ( θ ∗ , ϕ ∗ ) and η < ( L ) − 1 .
There exists a neighborhood U of ( θ ∗ , ϕ ∗ )
such that if SPPM startedat ( θ 0 , ϕ 0 ) ∈ U , the iterates { θ t , ϕ t } t ≥ 0 generated by SPPM satisfy:
[Equation]
where f ∗ =
f ( θ ∗ , ϕ ∗ ) .
Moreover,for any η satisfying:
[Equation]
SPPM converges asymptotically to ( θ ∗ , ϕ ∗ ) .
Remark 5.1.
Following the same condition as in 5.1, the iterates generated by Lv.
2 k GP satisfies:
[Equation]
wherea =
 
[Equation]
even k and b =
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
We assume that the difference between the trajectory{ θ t , ϕ t } t ≥ 0 and the stationary point ( θ ∗ , ϕ ∗ ) is not inthe kernel of ∇
ϕθ f ( θ ∗ , ϕ ∗ ) and ∇ θϕ f ( θ ∗ , ϕ ∗ ) for thesake of simplicity.
Detailed proofs without this assumption are provided in Appendix.
Following the same settingas Theorem 4.1, we have η <
L − 1 , which ensures thatη 2 λ max ( ∇ θϕ f ∗ ∇ ϕθ f ∗ ) k < 1 .
Therefore, as k → ∞ ,a → 1 and b → 0 in Remark 5.1, and as such Lv.
k GP hassimilar local convergence properties to SPPM.
Figure 1illustrates this property in a quadratic game.
Lv.
k GP maybehave differently than SPPM at lower values of k , butas k increases, both max step size η max ,k and distance toequilibrium for a fixed number of iterations under η max , ∞converges to that of SPPM.
Thus, we observe that Lv.
k
GP is empirically similar to SPPM at higher values of k .
We study the global convergence property of SPPM byanalyzing its behavior in bilinear and quadratic games.
Consider the following bilinear game:
[Equation]
where M is a full rank matrix.
The following theorem summarizes SPPM’s convergence property:
Theorem 5.2.
Consider the Bilinear game and the SPPM method.
Further, we define r t =
∥
θ t −θ ∗ ∥ 2 + ∥ ϕ t − ϕ ∗ ∥ 2 .
Then, for any η > 0 , the iterates { θ t , ϕ t } t ≥ 0 generated by SPPM satisfy
[Equation]
It is worth noting that the convergence property of Lv.
k GP in bi-linear game has been studied in [2].
Furthermore, we study the convergence property of SPPM in the Quadratic game:
[Equation]
(Quadratic game) where A ∈ R n × n is symmetric and positive definite, B ∈ R n × n is symmetric and negative definiteand the interaction term C ∈ R n × n is full rank.
SPPM in quadratic games converges with thefollowing rate:
Theorem 5.3.
Consider the Quadratic game and the SPPM.
Then, for any η > 0 , the iterates{ θ t , ϕ t } t ≥ 0 generated by SPPM satisfy
[Equation]
[Figure]
i n t e r a c t i on c
Note that the update is equivalent to SPPM in quadratic games.
Theorem 5.1,5.2 and 5.3 indicate that stronger interaction ∇ θϕ f ( θ ∗ , ϕ ∗ ) improve the convergencerate towards the stationary points.
By contrast, in existing modifications of GDA, the step size ischosen in inversely proportional to the interaction term ∇ θϕ f ( θ ∗ , ϕ ∗ ) [40, 13, 34].
In Figure 3, weshowcase the effect of interaction on different algorithms in the Quadratic game setup with dimensionn = 5 and the interaction matrix is defined as C
= c
I .
Stronger interaction corresponds to highervalues of c .
A key difference between SPPM and Lv.
k GP in the experiments is that, SPPM convergeswith any step size - and so arbitrarily fast - while it is not the case for Lv.
k GP.
In order to approximate
SPPM, one needs Lv.
k GP to be a contraction and so have η < L − 1 .
This result implies an additionalbound on step sizes for Lv.
k GP.
As long as the constraint on η is satisfied, stronger interaction onlyimproves convergence for higher order Lv.
k GP while all other algorithms quickly diverge as η and cincrease.
Experimental Results
In this section, we discuss our implementation of Lv.
k GP algorithm for training GANs.
Its performance is evaluated on 8-Gaussians and two representative datasets CIFAR-10 and STL-10.
6.1 Level k Adam
We propose to combine Lv.
k GP with the Adam optimizer [29].
Preliminary experiments findthat Lv.
k Adam to converge much faster than Lv.
k GP, see A.8 for experiment results.
A detailedpseudo-code for Level k Adam (Lv. k Adam) on GAN training with loss functions L G and L D isgiven in Algorithm 1.
For the Adam optimizer, there are several possible choices on how to updatethe moments.
This choice can lead to different algorithms in practice.
Unlike [19] where the momentsare updated on the fly, in Algorithm 1, we keep the moments fixed in the reasoning steps and updateit together with model parameters.
In Table 2, our experiment result suggests that the proposed Lv. k
Adam algorithm converges
Input: Stopping time T , reasoning steps k , learning rate η θ , η ϕ , decay rates for momentum estimates β 1 , β 2 , initial weight ( θ 0 , ϕ 0 ) , P x and P z real and noise-data distributions, losses L G ( θ , ϕ , x , z ) and L D ( θ , ϕ , x , z ) , ϵ = 1 e − 8 .
Parameters : Initial parameters: θ 0 , ϕ
Initialize first moments: m θ, 0 ←− 0 , m ϕ, 0 ←−
Initialize second moments: v θ, 0 ←− 0 , v ϕ, 0 ←−for t=0,...,T-1 do
Sample new mini-batch: x , z ∼ P x , P z , θ (0) t ←− θ t , ϕ (0) t ←− ϕ t , for n=1,...,k do
Compute stochastic gradient:
[Equation]
Update estimate of first moment:
[Equation]
Update estimate of second moment: v
[Equation]
Correct the bias for the moments: ˆ
[Equation]
) , ˆ v ( n ) ϕ,t
= v ( n ) ϕ,t (1 − β t 2 )
Perform Adam update: θ ( n ) t = θ t − η θ ˆ m ( n ) θ,t (cid:113) ˆ v ( n ) θ,t + ϵ ; ϕ ( n ) t = ϕ
t − η ϕ ˆ m ( n ) ϕ,t (cid:113) ˆ v ( n ) ϕ,t
+ ϵ θ t +1 ←− θ ( k ) t , ϕ t +1 ←− ϕ ( k ) t ; m θ,t ←− m ( k ) θ,t , m ϕ,t ←− m ( k ) ϕ,t ; v θ,t ←− v ( k ) θ,t , v ϕ,t ←− v ( k ) ϕ,t
6.2 8-Gaussians
In our first experiment, we evaluate Lv.
k Adam on generating a mixture of 8-Gaussianswith standard deviations equal to 0 . 05 and modes uniformly distributed around the unit circle.
We use a two layer multi-layer perceptron with ReLU activations, latent dimensionof 64 and batch size of 128 .
The generated distribution is presented in Figure 4.
[Table]
In addition to presenting the mode coverage of the generated distribution after training, we also study the convergence of the reasoning steps of Lv.
k Adam and Lv.
k GP.
Let us define the difference between the states of Lv.
k
and
Lv.
k − 1 agents at time t as:
[Equation]
We measure the difference averaged over the first 100 iterations, 1100 (cid:80) 100 t =1 r kt , for Lv.
10 Adam ( η = 10 − 4 ) and
Lv. 10 GP ( η = 10 − 2 ).
The result presented in Table2demonstrates that both Lv.
k GP and Lv.
k Adam are con- [Figure] I [Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
A v e r age t i m e
[Figure]
r i t e r a t i on
[Figure]
A v
[Figure]
r age t i m e
[Figure]
r i t
[Figure]
r a t i
[Figure]
verging as k increases.
Moreover, the estimation precision of Lv.
k GP improves rapidly and convergesto
within finite steps, making it an accurate estimation for SPPM.
[Table]
6.3 Image Generation Experiments
We evaluate the effectiveness of our Lv.
k
Adam algorithm on unconditional generation of CIFAR-10[31].
We use the Inception score (the higher the better) [49] and the Fréchet Inception distance(the lower the better)
[24] as performance metrics for image synthesis.
For architecture, we use the
SN-GAN architecture based on [44].
For baselines, we compare the performance of Lv.
6 Adam to thatof other first-order and second-order optimization algorithms with the same SN-GAN architecture,and to state-of-the-art models trained with Adam.
For Lv.
k Adam, we use β 1 = 0 and β 2 = 0 .
9 forall experiments.
We use different learning rates for the generator ( η θ = 4 e − 5 ) and the discriminator( η ϕ = 2 e − 4 ).
We train Lv.
k Adam with batch size 128 for 600 epochs.
For testing, we use anexponential moving average of the generator’s parameters with averaging factor β = 0 .
[Table]
In table 3, we present the performance of our method and baselines.
The best FID score and Inceptionscore, 10 . 17 ± 0 .
16 and 8 .
78 ± 0 .
06 , on SN-GAN are obtained with our Lv.
Adam.
We alsooutperform BigGAN and StyleGAN2 in terms of FID score.
Notably, our model has 5.1M parametersin total, and is trained with a small batch size of 128, whereas BigGAN uses 158.3M parameters anda batch size of 2048.
The effect of k and losses: We evaluate values of k = { 2 , 4 , 6 } on non-saturated loss [21] (nonzero-sum formulation) and hinge loss [35] (zero-sum formulation).
The result is presented in Table4.
Remarkably, our experiments demonstrate that few steps of recursive reasoning can result insignificant performance gain comparing to existing GAN optimizers.
The gradual improvements inthe FID scores justify the idea that better estimation of opponents’ next move improves performance.
Moreover, we observe performance gains in both zero-sum and non-zero-sum formulations whichsupplement our theoretical convergence guarantees in zero-sum games.
Experiment results on STL-10:
To test whether the proposed Lv.
k
Adam optimizer works onhigher resolution images, we evaluate its performance on the STL-10 dataset [11] with 3 × 48 × 48resolutions.
In our experiments, Lv. 6 Adam obtained an averaged FID of 25 .
43 ± 0 .
18 whichoutperforms that of the Adam optimizer, 30 .
25 ± 0 .
26 , using the same SN-GAN architecture.
Memory and computation cost: Compared to SGD, Lv.
k Adam requires the same extra memory asthe EG method (one additional set of parameters per player).
The relative cost of one iteration versus
SGD is a factor of k .
We provide an accelerated version of Lv.
k Adam in A.8 which reduces thecomputation cost by half for k > 2 .
In Figure 5 (top-left), we compare the FID scores obtained by
Lv.
k GP, Adam, and LEAD on Cifar-10 over the same number of gradient computations.
LEAD, Lv
GP, and Lv6 GP all outperform Adam in this experiment.
Lv4 GP outperforms LEAD after 2 × 10 5gradient computations.
Our method is also compared with different algorithms on the 8-Gaussianproblem in terms of its computational cost.
On the same architecture with different widths, Figure 5(bottom-left) illustrates the wall-clock time per computation for different algorithms.
We observe thatthe computational cost of Lv.
k while being much lower than CGD, is similar to LEAD, SGA and
CO which involve JVP operations.
Each run on CIFAR-10 dataset takes 30 ∼ 33 hours on a Nvidia
RTX3090 GPU.
Each experiment on STL-10 takes 48 ∼ 60 hours on a Nvidia RTX3090 GPU.
Conclusion and Future Work
This paper proposes a novel algorithm:
Level k gradient play, capable of reasoning about players’future strategies.
We achieve an average FID score of 10.17 for unconditional image generation on
CIFAR-10 dataset, allowing GAN training on common computing resources to reach state-of-the-artperformance.
Our results suggest that Lv.
k GP is a flexible add-on that can be easily attached toexisting GAN optimizers (e.g., Adam) and provides noticeable gains in performance and stability.
Infuture work, we will examine the effectiveness of our approach on more complicated GAN designs,such as Progressive GANs [26] and StyleGANs [27], where optimization plays a more significantrole.
Additionally, we intend to examine the convergence property of Lv.
k GP in games with morethan two player in the future.
Broader Impact Our work introduces a novel optimizer that improves the performance of GANsand may reduce the amount of hyperparameter tuning required by practitioners of generative modeling.
Generative models have been used to create illegal content(a.k.a.
deepfakes [5]).
There is risk ofnegative social impact resulting from malicious use of the proposed methods.