Despite their growing prominence, optimization in generative adversarial networks (GANs) is still a poorly-understood topic. In this paper, we analyze the "gradient descent" form of GAN optimization (i.e., the natural setting where we simultaneously take small gradient steps in both generator and discriminator parameters). We show that even though GAN optimization does not correspond to a convex-concave game, even for simple parameterizations, under proper conditions, equilibrium points of this optimization procedure are still locally asymptotically stable for the traditional GAN formulation. On the other hand, we show that the recentlyproposed Wasserstein GAN can have non-convergent limit cycles near equilibrium. Motivated by this stability analysis, we propose an additional regularization term for gradient descent GAN updates, which is able to guarantee local stability for both the WGAN and for the traditional GAN, and also shows practical promise in speeding up convergence and addressing mode collapse. We now present the Jacobian of the system at equilibrium in the presence of the gradient penalty. Recall that the penalty basically adds an extra −∇ θG ∇ θ D V (D θD , G θG ) 2 to the generator's update. Lemma F.1. For the dynamical system defined by the GAN objective in Equation 3 and the updates in Equation 5, the Jacobian at an equilibrium point (θ D , θ G ), under the Assumptions I and IV is: where J DD and J DG are terms in the Jacobian corresponding to the original updates, as described in Theorem 3.1. Proof. Note that the only change to the Jacobian would be in the rows corresponding to the generator parameters. Therefore, we will focus only on the additional terms in these rows.
Regarding the stability of GAN algorithms, REF proves that the GDA algorithm will be locally stable for the vanilla and regularized Wasserstein GAN problems.
Gradient descent GAN optimization is locally stable
ArXiv
ArXiv
Computer Science Mathematics