Precondition Layer and Its Use for GANs
Authors: Tiantian Fang, Alex Schwing, Ruoyu Sun
Keywords: GAN, Preconditioning, Condition Number
Abstract: One of the major challenges when training generative adversarial nets (GANs) is instability.  To address this instability spectral normalization (SN) is remarkably successful.  However, SN-GAN still suffers from training instabilities, especially when working with higher-dimensional data.  We find that those instabilities are accompanied by large condition numbers of the discriminator weight matrices. To improve training stability we study common linear-algebra practice and employ preconditioning.   Specifically,  we introduce a  preconditioning layer  (PC-layer)that performs a  low-degree polynomial preconditioning.   We use this  PC-layer in two ways:  1)  fixed preconditioning  (FPC)  adds a  fixed  PC-layer to all layers, and 2)  adaptive preconditioning  (APC)  adaptively controls the strength of preconditioning. Empirically, we show that FPC and APC stabilize the training of un-conditional GANs using classical architectures.  On LSUN256×256 data, APC improves FID scores by around 5 points over baselines.