We propose a variant of the standard min-max framework for GANs to learn a distribution, where the discriminator can update its strategy in a greedy manner until it reaches a first-order stationary point. We give an algorithm to train such a GAN and show that it provably converges from any initial point to an approximate local equilibrium for this framework . Our algorithm runs in time polynomial in the smoothness parameters of the loss function and independent of the dimension, and allows for loss functions that can be nonconvex and nonconcave in the parameters of the generator and discriminator. Empirically, GANs trained using our algorithm consistently learn a greater number of modes than gradient descent-ascent (GDA), optimistic mirror descent (OMD), andunrolled GANs when applied to a synthetic Gaussian mixture dataset. Moreover, they perform significantly better on CIFAR-10 than OMD and GDA when comparing the mean and standard deviation of the Inception Score respectively .
Motivated by the recent work of Mangoubi and Vishnoi (STOC 2021), we propose a variant of the min-max optimization framework where the max-player is constrained to update the maximization variable in a greedy manner until it reaches a *first-order* stationary point. We present an algorithm that provably converges to an approximate local equilibrium for our framework from any initialization and for nonconvex-nonconcave loss functions. Compared to the second-order algorithm of Mangoubi and Vishnoi, whose iteration bound is polynomial in the dimension, our algorithm is first-order and its iteration bound is independent of dimension. We empirically evaluate our algorithm on challenging nonconvex-nonconcave test-functions and loss functions that arise in GAN training. Our algorithm converges on these test functions and, when used to train GANs on synthetic and real-world datasets, trains stably and avoids mode collapse .
R
We
Motivated by the recent work of Mangoubi and Vishnoi (STOC 2021), we
D
standard
R
framework for GANs to learn a distribution, where the discriminator can update its strategy
optimization framework where the max-player is constrained to update the maximization variable
R
first-order
*first-order*
R
give an algorithm to train such a GAN and show that it provably converges from any initial point
present an algorithm that provably converges
R
this framework . Our algorithm runs in time
our framework from any initialization and for nonconvex-nonconcave loss functions. Compared to the second-order algorithm of Mangoubi and Vishnoi, whose iteration bound is
R
smoothness parameters of the loss function and independent of the dimension, and allows for
dimension, our algorithm is first-order and its iteration bound is independent of dimension. We empirically evaluate our algorithm on challenging nonconvex-nonconcave test-functions and
R
can be nonconvex and nonconcave in the parameters of the generator and discriminator. Empirically, GANs trained using our algorithm consistently learn a greater number of modes than gradient descent-ascent (GDA), optimistic mirror descent (OMD), andunrolled GANs when applied to a synthetic Gaussian mixture dataset. Moreover, they perform significantly better on CIFAR-10 than OMD and GDA when comparing the mean and standard deviation of the Inception Score respectively
arise in GAN training. Our algorithm converges on these test functions and, when used to train GANs on synthetic and real-world datasets, trains stably and avoids mode collapse