Bridging the Gap Between Coulomb GAN and Gradient-regularized WGAN
@iisc.ac in css@iisc.ac.in
Abstract
Generative adversarial networks (GANs) are essentially a min-max game between the discriminator and a generator.
Coulomb GANs have a closely related formulation where the generator minimizes the potential difference between real (negative) and fake (positive) charge densities , wherein the discriminator approximates a low-dimensional Plummer kernel centered around the samples.
Motivated by links between electrostatic potential theory and the Poisson partial differential equation (PDE), we consider the underlying functional optimization in Coulomb GAN and show that the associated discriminator is the optimum of a first-order gradientregularized Wasserstein GAN (WGAN) cost.
Subsequently, we show that, within the regularized WGAN setting, the optimal discriminator is Green’s function to the Poisson PDE, which corresponds to the Coulomb potential.
As an alternative to training a discriminator in either WGAN or Coulomb GAN, we demonstrate, by means of synthetic data experiments, that the closed-form implementation of the optimal discriminator leads to a superior performance of the GAN generator.
Introduction
Generative adversarial networks (GANs) (Goodfellow et al., 2014) have gained immense popularity over the past decade due to their ability to generate realistic images, requiring low sampling times (Karras et al., 2019, 2020, 2021).
GANs are a min-max game, where a generator network G is trained to learn the underlying distribution p d of a target dataset with the aid of a discriminator network D .
The generator is essentially a transformation of a noise distribution p z , typically Gaussian, into the push-forward distribution p g
=
G # ( p z ) .
Wasserstein GANs (WGAN), introduced by Arjovsky et al.
(2017) consider a discriminator that approximates a distance (called integral probability metric or IPM) between the target and generator distributions, which in turn, induces a constraint class from which the discriminator function is drawn.
For example, WGANs consider Lipschitz-1 constraints, while WGAN-GP (Gulrajani et al., 2017) and Sobolev GANs (Mroueh et al., 2018) consider functions with bounds on the norm of the gradient.
Subsequent variants proposed by Gulrajani et al.
; Bellemare et al.
; Mescheder et al.
and Adler & Lunz (2018) also consider gradient-based regularizers to improve training stability.
However, the min-max nature of the optimization makes it difficult to derive convergence guarantees on the GAN training, either in the parametric sense (Li et al., 2017b), or in the functional sense (Unterthiner et al., 2018).
Although the WGAN loss is a popular choice in most applications (Kang et al., 2022), in an effort to derive GANs with guaranteed convergence to the Nash equilibrium of the game, Unterthiner et al.
(2018) consider a discriminator network that approximates the Plummer kernel evaluated at the real and generated sample points.
They showed that, given the sample approximation of the
Plummer-kernel discriminator, the GAN generator converges to the desired optimum, i.e. p ∗ g = p d .
However, the links between the Coulomb GAN and IPM based losses was not explored and the underlying optimization problem that the discriminator solves is unclear.
We therefore wish to ask the following question in this paper – How do the Plummer/Coulomb family of kernels link to classical IPM GAN optimization?
1.1 The Proposed Approach
We analyze the optimization problem in Coulomb GANs considering the Plummer kernel. In potential theory, the optimal Coulomb GAN discriminator kernel arises naturally as the consequence of gradient-regularization, which we show, in the present context, corresponds to optimizing the gradient-regularized Wasserstein GAN (WGAN).
We analyze the gradient-regularized WGAN cost in a variational setting, and show that the optimal WGAN discriminator solves a Poisson PDE, whose
Green’s function gives rise to the Coulomb GAN’s Plummer-kernel based potential function.
Unlike in Coulomb GANs, where a discriminator network is used to approximate the kernel function, we implement the optimum as is, and demonstrate on synthetic data, that the closed-form discriminator performs superior to both gradient-penalized WGANs, and the kernel-approximating Coulomb GANs.
The TensorFlow (Abadi et al., 2016)
based implementation of the proposed approach is available at https://github.com/DarthSid95/RBFCoulombGANs .
As a future scope of this work, one could solve Poisson PDEs in high-dimensional spaces through appropriate numerical solvers, similar to the approaches used in Diffusion models (Ho et al., 2020) and score-matching approaches (Song & Ermon, 2019).
The Coulomb GAN Discriminator
Unterthiner et al.
(2018) consider the task of generative learning, where the objective is to minimize the difference between the data and generator distributions, ρ ( x ) =
p g ( x ) − p d ( x ) , given data x ∈ R n .
Motivated by the connection to electrostatics, they consider the potential function associated with the difference, and a chosen kernel κ , given by
[Equation]
where κ ( x , y ) = [Equation] x − y ∥ 2 +
[Equation]
1) where d ≤ n − 2 , X denotes the support of the discriminator, typically the convex hull encompassing the supports of p d and p g , and ϵ is a small non-zero constant used to eliminate the singularity in κ .
For d = n − 2 , κ represents the Plummer kernel.
Intuitively, the potential in Eq.
(1) induces a field over X where the generator is forced to generate samples that are repelled from those samples deemed fake by the discriminator, while being attracted towards those classified as reals.
In practice, Φ is approximated by a sample average, computed over a mini-batch, and is given by
[Equation]
where d i and g j represent the N d and N g samples drawn from p d and p g , respectively.
The Coulomb
GAN discriminator is trained to approximate Φ in the least-squares sense, while the generator is trained to maximize D , the approximation of ˆΦ , for its generated samples, i.e.,
[Equation]
and
[Equation]
G
[Equation]
where p x ( x ) = 12 (cid:82) N ( x ; G ( z ) , ε I )
p z ( z ) d z + 12 (cid:82) N ( x ; y , ε I ) p d ( y )
d y denotes a distribution whose samples are drawn from a mix of p d and p g , and corrupted with white Gaussian noise having a small variance ε .
Unlike in the standard GAN and WGAN formulations, Unterthiner et al.
(2018) showed that the alternating minimization of L CD and L CG results in attaining the unique global optimum ( D ∗ , G ∗ ) , where p ∗ g
= G ∗ # ( p z ) = p d , with G ∗ # denoting the push-forward distribution through the optimal generator G ∗ .
However, the nonlinearities introduced by the choice of κ , and the slow learning of the discriminator caused by the least-squares cost, mandate the use of small learning rates for training the two networks.
Furthermore, the link between the Coulomb GAN discriminator cost L CD and other IPM based GAN optimization has not been fully explored.
In this paper, we show that Coulomb GANs can be linked to the WGANs through the solution to the Poisson PDE.
Gradient-regularized WGAN and the Plummer Kernel
Arjovsky et al.
(2017) presented the GAN learning problem as one of optimal transport , where the critic (or discriminator) minimizes the earth-mover’s distance or Wasserstein-1 distance between p d and p g .
Through the Kantorovich-Rubinstein duality, they defined the WGAN discriminator loss as L WD =
E x ∼ p g
[ D ( x )] − E x ∼ p
d [ D ( x )] and the generator loss as L WG =
−L WD , such that
D ( x ) is Lipschitz-1.
Gulrajani et al.
(2017) approximate the Lipschitz-1 constraint though a gradient penalty E x ∼ p int (cid:104) ( ∥∇ D ( x ) ∥ 2 − 1) 2 (cid:105) , where p int is an interpolated distribution between p d and p g .
Subsequent works (Kodali et al., 2017; Petzka et al., 2018; Terjék, 2020) replaced p int with alternatives that resulted in stabler training.
For example, Mescheder et al.
and Mroueh et al. (2018) consider Ω RD =
E x ∼ p r (cid:2) ∥∇ D ( x ) ∥ 22 (cid:3) for various choices of p r .
WGANs, PDEs and the Coulomb Kernel : Mroueh et al.
(2018) showed that the optimal solutions in Sobolev GANs solve a family of Fokker-Planck PDEs.
However, they do not invert the general
Fokker-Planck operator due to computational constraints.
Recall that, for x ∈ R n , the Plummer-like kernel, with ϵ = 0 represents the radially-symmetric Coulomb kernel, given by
[Equation]
While generative moment matching networks (GMMNs) (Li et al., 2015) and maximum mean discrepancy GANs (MMD-GANs) (Li et al., 2017a) consider Gaussian and inverse multi-quadric kernels in the context of GAN learning, the Coulomb kernels have been extensively studied in interpolation literature (Aronszajn et al., 1983; Iske, 2004; Fasshauer, 2007).
They are known to arise in mesh-free gradient-regularized least-squares interpolation, where the smoothness constraint is enforced over X .
Motivated by these connections, we consider the IPM GAN discriminator cost with a gradient-norm constraint, given by
[Equation]
The above penalty can be seen as a special case of Ω RD with p r set to the uniform measure over X .
The following theorem links the discriminator in gradient-regularized GANs to the Coulomb GAN.
Theorem 3.1.
The optimal discriminator that minimizes the L D solves the following Poisson PDE:
[Equation] [Equation]
where ∆ is the Laplacian operator.
The particular solution, D ∗ p ( x ) , is given by
[Equation]
[Equation] which is a multidimensional convolution with the Coulomb kernel, which in turn is the fundamental solution to the Laplace equation: ∆ ψ n ( x )
=
C n δ ( x ) , for some constant C n .
Proof.
The discriminator PDE can be derived through the Calculus of Variations applied to the cost in Equation (4).
PDEs of this type have been extensively researched and Evans (2010)’s book on
Partial Differential Equations is an excellent reference on this topic.
The details are provided in Appendix A.
Equation (5) indeed mimics results from electrostatics, and as observed in Coulomb GANs, the optimal discriminator is the potential function, with the generator and target distributions representing positive and negative charge densities, respectively.
Additionally, these results show that optimizing the discriminator in GANs is closely linked to solving PDEs.
While Theorem 3.1 provides the particular solution to the discriminator, the general solution also includes the homogeneous component, i.e., solutions to ∆ f ( x ) = 0 , that belong to the space P n 1 of first-degree polynomials P ( x ) = ⟨ a , x ⟩ + a 0 .
The general solution to the discriminator PDE is D ∗ ( x ) =
D ∗ p ( x ) + P ( x ) .
The exact choice of the polynomial P ( x ) ∈ P n 1 depends on the boundary conditions and will be discussed shortly.
The multidimensional convolution in a high-dimensional space makes Equation (6) impractical.
The following lemma presents an implementable form of the optimal discriminator.
[Figure]
[Figure]
Coulomb GAN
WGAN-GP
WGAN-LP
WGAN-R d
WGAN-R g
RBF-Coulomb GAN
[Figure]
Lemma 3.2.
The optimal discriminator D ∗ p ( x ) given in Eq.
(6) can be approximated through the following sample estimate:
[Equation]
where ψ n is as described in Eq.
Lemma 3.2 shows that the sample approximation of the gradient-regularized WGAN discriminator equals the potential function described in Equation 1 for ϵ = 0 .
As opposed to training the GAN discriminator on L D or L CD with gradient-descent, we directly implement ˆ D ∗ ( x ) =
ˆ D ∗ p ( x )+ ⟨ a , x ⟩ + a 0 .
ˆ D ∗ p ( x ) can be constructed using a radial basis function (RBF) network with pre-determined weights.
Preliminary experiments indicate that the choice of the polynomial does not affect the generator optimization.
Choosing a = 0 and a 0 = 0 results in the Coulomb GAN solution.
Experimental Validation
To validate using a RBF discriminator, we present preliminary experiments on learning 2-D unimodal and multimodal Gaussian data.
We compare the proposed approach, called RBF-Coulomb GAN , against the baseline Coulomb GAN, and gradient-regularized WGAN variants such as WGANGP (Gulrajani et al., 2017), WGAN-R d and WGAN-R g (Mescheder et al., 2018), and WGANLP (Petzka et al., 2018).
Network parameters and data specifications are described in Appendix C.
Performance comparisons using the Wasserstein-2 distance (cid:0) W 2 , 2 ( p d , p g ) (cid:1) are presented in Figures 1(a) and (b).
In both the scenarios, the proposed RBF-Coulomb GAN outperforms the baselines (lower W 2 , 2 scores).
RBF-Coulomb GAN is also less sensitive to learning-rate perturbations, compared to the trainable-discriminator counterparts.
These results showcase the superiority of solving the discriminator PDE over stochastic-gradient optimization.
Conclusions
We showed that the Coulomb GAN discriminator is, in fact, the optimum to a special case of gradientregularized WGAN.
More precisely, the solution comes up as the Green’s function to the Poisson
PDE, when approximated using RBFs.
Considering toy examples, we demonstrate the superior convergence of employing the closed-form RBF discriminator over the baseline methods.
To derive the generalization to Plummer-like kernels for d ≤ n − 2 , one could consider higher-order gradient penalties, which give rise to PDEs involving iterated Laplacian operators.
Exploring PDE solvers to implement the optimal discriminator in image learning scenario is a promising direction for research.
Results from denoising diffusion models (Ho et al., 2020; Xiao et al., 2022) and score-matching networks (Song & Ermon, 2019, 2020; Song et al., 2021) could be leveraged to learn the optimal
Coulomb GAN discriminator in a way that avoids issues with training such as learning-rate scheduling and mode collapse.
One could also derive PDEs and associated WGAN costs that correspond to generalizations of Plummer kernel dimension d .
The variational approach can also be used to derive
PDEs that govern other GAN discriminators that do not conform to IPM minimization, such as those with relativistic discriminators (Jolicoeur-Martineau, 2019).
Acknowledgements
This work is supported by the Microsoft Research Ph.D. Fellowship 2018, Qualcomm Innovation
Fellowship 2019, 2021 and and the Robert Bosch Centre for Cyber-Physical Systems Ph.D.
Fellowships (2020-2021; 2021-2022)
A Proof of Theorem 3.1: Gradient-regularized WGANs
First, we recall a few useful results from the Calculus of Variations .
Consider the functional optimization of a cost L defined as
[Equation]
where ∂
i f ( x ) =
∂f ∂x i denotes the partial derivative of f with respect to the i th entry of x .
The
Euler-Lagrange condition that the optimizer f ∗ ( x ) of the cost L must satisfy is given by
[Equation]
We can now derive the optimal discriminator in gradient-regularized WGAN.
Recall that
[Equation]
where ρ ( x ) =
p g ( x ) − p d ( x ) , and ∥∇ D ( x ) ∥ 22 = (cid:88) i ( ∂ i D ) 2 is the square of the ℓ 2 norm of the gradient vector.
Differentiating the integrand in L D gives ∂
[Equation]
and
[Equation]
Enforcing the Euler-Lagrange condition yields the PDE given in Equation (5):
[Equation]
The above Poisson PDE can be solved to obtain the optimal discriminator function.
As noted by Evans (2010)
; Unterthiner et al.
(2018), the family of Coulomb kernels ψ n ( x ) are Green’s functions to the Laplace operator, i.e., ∆ ψ n ( x ) =
C n δ ( x ) , where C n = Γ ( n 2 +1 ) n ( n − 2) π n 2 , where in turn Γ ( z ) = ∞ (cid:82) t z − 1 e − t d t denotes the Gamma function (Aronszajn et al., 1983).
Convolving both sides of the Poisson PDE with ψ n gives the optimal discriminator:
[Equation]
This completes the proof of Theorem 3.1.
B Proof of Lemma 3.2: Sample Estimate of the Optimal Discriminator
Consider the closed-form optimal discriminator given in Equation (6):
[Equation]
where λ d accounts for the effect of C n .
Without loss of generality, we assume that n > 2 .
From the definition of the convolution, we have
[Equation]
As in the case of Coulomb GAN (Unterthiner et al., 2018), the expectations can now be replaced with
N -sample estimates as follows:
[Equation]
where ˜ D ∗ p is essentially a radial basis function expansion.
A similar analysis could be carried out for n = 1 , 2 , and the corresponding discriminators are:
[Equation]
The generic form for ˜ D ∗ p is given by
[Equation]
where ψ n is as defined in Equation (3).
C Network Specifications and Training Parameters
Data : We consider unimodal learning on a 2-D Gaussian N ( 3 . 5 , 1 . 25 I ) , where 3 . 5 denotes a 2-D vector with both values equal to 3.5, and I is the identity matrix.
For the multimodal learning task, we consider an eight-component Gaussian mixture model (GMM) with component means selected from [0 , 1] 2 , each having standard deviation 0 .
Network : We consider the affine transformation of the noise z , given by x = M z + b to learn 2-D Gaussian data.
The discriminator for RBF-Coulomb GAN is an RBF network whose weights are computed out-of-the-loop, thereby obviating the need for network-based optimization of the discriminator.
On the GMM task, the generator consists of three fully-connected layers of 10,and 5 nodes, followed by an output layer of 2 nodes.
The activation used is Leaky ReLU.
The discriminator in the baselines consists of three fully-connected layers with Leaky ReLU activation and 20, 10, and 1 node(s).
The discriminator is identical for both the Gaussian and GMM learning tasks.
Training Parameters :
The networks are trained using the Adam optimizer (Kingma & Ba, 2015) with a learning rate of 0 .
02 for the generator and 0 .
075 for the discriminator.
The batch size is set to 500.
The models are evaluated using the Wasserstein-2 distance, estimated using the Python optimal transport (POT) library (Flamary et al., 2021).
Source Code : The Source code for RBF-Coulomb GANs is developed in TensorFlow 2.0 (Abadi et al., 2016), and is available at the following GitHub repository: https://github.com/DarthSid95/ RBFCoulombGANs .