R EFINING D EEP G ENERATIVE M ODELS VIA D ISCRIMINATOR G RADIENT F LOW
A BSTRACT
Deep generative modeling has seen impressive advances in recent years, to the point where it is now commonplace to see simulated samples (e.g., images) that closely resemble real-world data.
However, generation quality is generally inconsistent for any given model and can vary dramatically between samples.
We introduce Discriminator Gradient f low (DG f low), a new technique that improves generated samples via the gradient ﬂow of entropy-regularized f -divergences between the real and the generated data distributions.
The gradient ﬂow takes the form of a non-linear Fokker-Plank equation, which can be easily simulated by sampling from the equivalent McKean-Vlasov process.
By reﬁning inferior samples, our technique avoids wasteful sample rejection used by previous methods (DRS & MH-GAN).
Compared to existing works that focus on speciﬁc GAN variants, we show our reﬁnement approach can be applied to GANs with vector-valued critics and even other deep generative models such as VAEs and Normalizing
Flows.
Empirical results on multiple synthetic, image, and text datasets demonstrate that DG f low leads to signiﬁcant improvement in the quality of generated samples for a variety of generative models, outperforming the state-of-the-art Discriminator Optimal Transport (DOT) and
Discriminator Driven Latent Sampling (DDLS) methods.
I NTRODUCTION
Deep generative models (DGMs) have excelled at numerous tasks, from generating realistic images (Brock et al., 2019) to learning policies in reinforcement learning (Ho & Ermon, 2016).
Among the variety of proposed DGMs, Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) have received widespread popularity for their ability to generate high quality samples that resemble real data.
Unlike Variational Autoencoders (VAEs) (Kingma & Welling, 2014) and Normalizing Flows (Rezende & Mohamed, 2015; Kingma & Dhariwal, 2018), GANs are likelihood-free methods; training is formulated as a minimax optimization problem involving a generator and a discriminator.
The generator seeks to generate samples that are similar to the real data by minimizing a measure of discrepancy (between the generated samples and real samples) furnished by the discriminator.
The discriminator is trained to distinguish the generated samples from the real samples.
Once trained, the generator is used to simulate samples and the discriminator has traditionally been discarded.
However, recent work has shown that discarding the discriminator is wasteful — it actually contains useful information about the underlying data distribution.
This insight has led to sample improvement techniques that use this information to improve the quality of generated samples (Azadi et al., 2019; Turner et al., 2019; Che et al., 2020; Tanaka, 2019; Che et al., 2020).
Unfortunately, current methods either rely on wasteful rejection operations in the data space (Azadi et al., 2019; Turner et al., 2019), or require a sensitive diffusion term to ensure sample diversity (Che et al., 2020).
Prior work has also focused on improving GANs with scalar-valued discriminators, which excludes a large family of GANs with vector-valued critics, e.g., MMDGAN (Li et al., 2017; Bi´nkowski et al., 2018) and OCFGAN (Ansari et al., 2020), and likelihood-based generative models. [Table]
P [Table] [Figure] In this work, we propose Discriminator Gradient f low (DG f low) which formulates sample improvement as reﬁning inferior samples using the gradient ﬂow of fdivergences between the generator and the real data distributions (Fig.
DG f low avoids wasteful rejection operations and can be used in a deterministic setting without a diffusion term.
Existing state-of-the-art methods — speciﬁcally, Discriminator Optimal Transport (DOT) (Tanaka, 2019) and Discriminator Driven Latent
Sampling (DDLS) (Che et al., 2020) — can be viewed as special cases of DG f low.
Similar to DDLS, DG f low recovers the real data distribution when the gradient ﬂow is simulated exactly.
We further present a generalized framework that employs existing pre-trained discriminators to reﬁne samples from a variety of deep generative models: we demonstrate our method can be applied to GANs with vector-valued critics, and even likelihood-based models such as VAEs and Normalizing Flows.
Empirical results on synthetic datasets, and benchmark image (CIFAR10, STL10) and text (Billion Words) datasets demonstrate that our gradient ﬂow-based approach outperforms DOT and DDLS on multiple quantitative evaluation metrics.
In summary, this paper’s key contributions are:
• DG f low, a method to reﬁne deep generative models using the gradient ﬂow of f -divergences; • a framework that extends DG f low to GANs with vector-valued critics, VAEs, and Normalizing Flows; • experiments on a variety of generative models trained on synthetic, image (CIFAR10 & STL10), and text (Billion Words) datasets demonstrating that DG f low is effective in improving samples from generative models.
B ACKGROUND : G RADIENT F LOWS
The following gives a brief introduction to gradient ﬂows; we refer readers to the excellent overview by Santambrogio (2017) for a more thorough introduction.
Let ( X , (cid:107) · (cid:107) 2 ) be a Euclidean space and F : X → R be a smooth energy function.
The gradient ﬂow of F is the smooth curve { x t } t ∈ R + that follows the direction of steepest descent, i.e.,
[Equation]
The value of the energy F is minimized along this curve.
This idea of steepest descent curves can be characterized in arbitrary metric spaces via the minimizing movement scheme (Jordan et al., 1998).
Of particular interest is the metric space of probability measures that is endowed with the Wasserstein distance ( W p ); the Wasserstein distance is a metric and the W p topology satisﬁes weak convergence of probability measures (Villani, 2008, Theorem 6.9).
Gradient ﬂows in the 2-Wasserstein space ( P 2 (Ω) , W 2 ) — i.e., the space of probability measures with ﬁnite second moments and the 2-Wasserstein metric — have been studied extensively.
Let { ρ t } t ∈ R + be the gradient ﬂow of a functional
F in the 2-Wasserstein space, where ρ t is absolutely continuous with respect to the
Lebesgue measure.
The curve { ρ t } t ∈ R + satisﬁes the continuity equation (Ambrosio et al., 2008, Theorem 8.3.1),
[Equation]
The velocity ﬁeld v t in Eq.
(2) is given by
[Equation]
where δ F δρ denotes the ﬁrst variation of the functional F .
Since the seminal work of Jordan et al.
(1998) that showed that the Fokker-Plank equation is the gradient ﬂow of a particular functional in the Wasserstein space, gradient ﬂows in the Wasserstein metric have been a popular tool in the analysis of partial differential equations (PDEs).
For example, they have been applied to the study of the porous-medium equation (Otto, 2001), crowd modeling (Maury et al., 2010; 2011), and mean-ﬁeld games (Almulla et al., 2017).
More recently, gradient ﬂows of various distances used in deep generative modeling literature have been proposed, notably that of the sliced Wasserstein distance (Liutkus et al., 2019), the maximum mean discrepancy (Arbel et al., 2019), the Stein discrepancy (Liu, 2017), the and Sobolev discrepancy (Mroueh et al., 2019).
Gradient ﬂows have also been used for learning non-parametric and parametric implicit generative models (Liutkus et al., 2019; Gao et al., 2019; 2020).
As an example of the latter, Variational Gradient Flow (Gao et al., 2019) learns a mapping between latent vectors and samples evolved using the gradient ﬂow of f -divergences.
In this work, we present a method using gradient ﬂows of entropy-regularized f -divergences for reﬁning samples from deep generative models employing existing discriminators as density-ratio estimators.
G ENERATOR R EFINEMENT VIA D ISCRIMINATOR G RADIENT F LOWS
This section describes our main contribution: Discriminator Gradient f low (DG f low).
As an overview, we begin with the construction of the gradient ﬂow of entropy-regularized f -divergences and describe its application to sample reﬁnement.
We then discuss how to simulate the gradient ﬂow in the latent space of the generator — a procedure more suitable for high-dimensional datasets.
Finally, we present a simple technique that extends our method to generative models that have not yet been studied in the context of reﬁnement.
Due to space constraints, we focus on conveying the key concepts and relegate details (e.g., proofs) to the appendix.
The entropy-regularized f -divergence functional is deﬁned as
[Equation]
where the f -divergence term D f ( µ (cid:107) ρ ) ensures that the “distance” between the probability density ρ and the target density µ decreases along the gradient ﬂow.
The differential entropy term H ( ρ ) improves diversity and expressiveness when the gradient ﬂow is simulated for ﬁnite time-steps.
We now construct the gradient ﬂow of F fµ .
Lemma 3.1.
Deﬁne the functional F fµ : P 2 (Ω) → R as
[Equation]
where f is a twice-differentiable convex function with f (1)
= 0 .
The gradient ﬂow of the functional
F fµ ( ρ ) in the Wasserstein space ( P 2 (Ω) , W 2 ) is given by the following PDE,
[Equation]
[Equation]
) where ∇ x · and ∆ xx denote the divergence and the Laplace operators respectively.
The proof is given in Appendix A.1.
The PDE in Eq.
(6) is a type of Fokker-Plank equation (FPE).
FPEs have been studied extensively in the literature of stochastic processes and have a Stochastic
Differential Equation (SDE) counterpart (Risken, 1996).
In the case of Eq.
(6), the equivalent SDE is given by
[Equation]
where d w t denotes the standard Wiener process.
Eq.
(7) deﬁnes the evolution of a particle x t under the inﬂuence of drift and diffusion.
Speciﬁcally, it is a McKean-Vlasov process (Braun & Hepp, 1977) which is a type of non-linear stochastic process as the drift term at any time t depends on the distribution ρ t of the particle x t .
Eqs.
(6) and (7) are equivalent in the sense that the distribution of the particle x t in Eq.
(7) solves the PDE in Eq.
Consequently, samples from the density ρ t along the gradient ﬂow can be obtained by ﬁrst drawing samples x 0 ∼ ρ
and then simulating the
SDE in Eq.
The SDE can be approximately simulated via the stochastic Euler scheme (also known as the Euler-Maruyama method) (Beyn & Kruse, 2011) given by
[Equation]
where ξ τ n ∼ N ( 0 , I ) , the time interval [0 , T ] is partitioned into equal intervals of size η and τ 0 < τ 1 < · · · < τ N denote the discretized time-steps.
Eq.
(8) provides a non-parametric procedure to reﬁne samples from a generator g θ where we let µ be the density of real samples and ρ τ 0 the density of samples generated from g θ obtained by ﬁrst sampling from the prior latent distribution
z ∼ p
Z
( z ) and then feeding z into g θ .
We ﬁrst generate particles x 0 ∼ ρ τ 0 and then update the particles using Eq.
(8) for N time steps.
Given a binary classiﬁer (discriminator) D that has been trained to distinguish between samples from µ and ρ τ 0 , the density-ratio ρ τ 0 ( x ) /µ
( x ) can be estimated via the well-known density-ratio trick (Sugiyama et al., 2012),
[Equation]
(9) where D ( y = 1 | x ) denotes the conditional probability of the sample x being from µ and d ( x ) denotes the logit output of the classiﬁer D .
We term this procedure where samples are reﬁned via gradient ﬂow of f -divergences as Discriminator Gradient f low (DG f low).
3. R EFINEMENT IN THE LATENT SPACE
Eq.
(8) requires a running estimate of the density-ratio ρ τ n ( x ) /µ
( x ) , which can be approximated using the stale estimate
ρ τ n ( x ) /µ
( x ) ≈
ρ τ 0 ( x ) /µ
( x ) for η → 0 and small N , where the density ρ τ n will be close to ρ τ 0 .
However, our initial image experiments showed that reﬁning directly in high-dimensional data-spaces with the stale estimate is problematic; error is accumulated at each time-step leading to a visible degradation in the quality of data samples (e.g., appearance of artifacts in images).
To tackle this problem, we propose reﬁning the latent vectors before mapping them to samples in data-space using g θ .
We describe a procedure analogous to Eq.
(8) but in the latent space for generators
g θ that take a latent vector z ∈ Z as input and generate a sample x ∈ X .
We ﬁrst show in Lemma 3.2 that the density-ratio in the latent space between two distributions can be estimated via the density-ratio of corresponding distributions in the data space.
Lemma 3.2.
Let g : Z → X be a sufﬁciently well-behaved injective function where Z ⊆ R n and
X ⊂ R m with m > n .
Let p Z ( z ) , p ˆ Z (ˆ z ) be probability densities on Z and q X ( x ) , q ˆ X
(ˆ x ) be the densities of the pushforward measures g(cid:93)Z , g(cid:93)
ˆ
Z respectively.
Assume that p Z ( z ) and p ˆ Z (ˆ z ) have same support, and the Jacobian matrix J g has full column rank.
Then, the density-ratio p ˆ Z ( u ) /p
Z ( u ) at the point u ∈ Z is given by
[Equation]
Require: First derivative of f ( f (cid:48) ), generator ( g θ ), discriminator ( d φ ), number of update steps ( N ), stepsize ( η ), noise factor ( γ ).
[Equation]
The proof is in Appendix A.2.
We let p ˆ Z
(ˆ z ) be the density of the “correct” latent space distribution induced by a generator g θ , i.e., p ˆ Z (ˆ z ) is the density of a probability measure whose pushforward under g θ approximately equals the target data density µ .
The density-ratio of the prior latent distribution p Z ( z ) and p ˆ Z (ˆ z ) can now be computed by combining Lemma 3.2 with
Eq.
[Equation]
Although a generator g θ parameterized by a neural network may not satisfy the conditions of injectivity and full column rank Jacobian matrix J g θ , Eq.
(11) provides an approximation that works well in practice as shown by our experiments.
Combining Eq.
(11) with Eq.
(8) provides us with an update rule for reﬁning samples in the latent space
[Equation]
with u τ 0 ∼ p
Z
( z ) and the density-ratio p u τn /p
ˆ Z is approximated using the stale estimate p u τ 0 /p
ˆ Z = exp( − d ( g θ ( u ))) .
We summarize the complete algorithm in Algorithm 1.
3. R EFINEMENT FOR A LL
Thus far, prior work (Azadi et al., 2019; Turner et al., 2019; Tanaka, 2019; Che et al., 2020) has focused on improving samples for GANs with scalar-valued discriminators, which comprises the canonical GAN as well as recent variants, e.g., WGAN (Gulrajani et al., 2017), and SNGAN (Miyato et al., 2018).
Here, we propose a technique that extends our approach to reﬁne samples from a larger class of DGMs including GANs with vector-valued critics, VAEs, and Normalizing Flows.
Let p θ be the density of the samples generated by a generator g θ and µ be the density of the real data distribution.
We are interested in reﬁning samples from g θ ; however, a corresponding density-ratio estimator for p θ /µ is unavailable, as is the case with the aforementioned generative models.
Let D φ be a discriminator that has been trained on the same dataset but for a different generative model g φ (e.g., let g φ and D φ be the generator and discriminator of SNGAN respectively).
D φ can be used to compute the density ratio p φ /µ .
A straightforward technique would be to use the crude approximation p θ /µ
≈ p φ /µ , which could work provided p θ and p φ are not too far from each other.
Our experiments show that this simple approximation works to a limited extent (see appendix E).
To improve upon the crude approximation above, we propose to correct the density-ratio estimate.
Speciﬁcally, a discriminator D λ is initialized with the weights from D φ and is ﬁne-tuned on samples from g φ and g θ .
D φ and D λ are then used to approximate the density-ratio p θ /µ ,
[Equation]
where d φ and d λ are logits output from D φ and D λ , respectively.
We term the network D λ
the density ratio corrector , which experiments show produces higher quality samples than using p
θ /µ
≈ p φ /µ .
The estimate in Eq.
(13) is similar to telescoping density-ratio estimation (TRE), a technique proposed in very recent independent work (Rhodes et al., 2020).
In brief, Rhodes et al.
(2020) show that classiﬁer-based density ratio estimators perform poorly when distributions are “too far apart”; the classiﬁer can easily distinguish between the distributions, even with a poor estimate of the density ratio.
TRE expands the standard density ratio into a telescoping product of more difﬁcultto-distinguish intermediate density ratios.
Likewise, in Eq.
(13), we treat p φ as an intermediate distribution and estimate the ﬁnal density-ratio as a product of two density-ratios.
R ELATED W ORK
Azadi et al.
ﬁrst proposed the idea of improving samples from a GAN’s generator by discriminator rejection sampling (DRS), making use of the density-ratio provided by the discriminator to estimate the acceptance probability.
Metropolis-Hastings GAN (MH-GAN) (Turner et al., 2019) improved upon the costly rejection sampling procedure via the Metropolis-Hastings algorithm.
Unlike
DG f low, both of these methods reject inferior samples instead of reﬁning them.
Recent work has also sought to improve generative models via sample evolution in the training/generation process.
In energy-based generative models (Arbel et al., 2020; Deng et al., 2020), the energy functions can be viewed as a component that improves some base generator.
For example, the Generalized Energy-Based Model (GEBM) (Arbel et al., 2020) jointly trains an implicit generative model with an energy function and uses Langevin dynamics to sample from the combination of the two.
The Noise Conditional Score Network (NCSN) (Song & Ermon, 2019; 2020) — a score-based generative model — can be seen as a gradient ﬂow that reﬁnes a sample right from noise to data.
Latent Optimization GAN (LOGAN) (Wu et al., 2020) optimizes a latent vector via natural gradient descent as part of the GAN training process.
While related, these techniques re- sult in “self-contained” generative models; unlike DG f low, they are not general sample reﬁnement techniques that can be applied across generative models 1 .
Our method is closely related to recent state-of-the-art techniques, speciﬁcally Discriminator-Driven
Latent Sampling (DDLS) (Che et al., 2020) and Discriminator Optimal Transport (DOT) (Tanaka, 2019).
In fact, both these methods can be seen as special cases of DG f low.
DDLS treats a GAN as an energy-based model and uses Langevin dynamics to sample from the energy-based latent distribution p t ( z ) ∝ p Z ( z ) exp( d ( g θ ( z ))) induced by performing rejection sampling in the latent space.
This distribution is the same as p ˆ Z
(ˆ z ) , which can be seen by rearranging terms in Eq.
If we use the KL-divergence by setting f = r log r , DG f low is equivalent to
DDLS.
However, there are practical differences that make DG f low more appealing.
DDLS requires estimation of the score function ∇ z { log p
Z ( z ) +
d
( g θ ( z )) } to perform the update which becomes undeﬁned if z escapes the support of p
Z ( z ) , e.g., in the case of the uniform prior distribution commonly used in GANs; handling such cases would require techniques such as projected gradient descent.
This problem does not arise in the case of DG f low since it only uses the density-ratio that is implicitly deﬁned by the discriminator.
Moreover, DDLS uses Langevin dynamics which requires the sensitive diffusion term to ensure diversity and to prevent points from collapsing to the maximum-likelihood point.
In DG f low, the sample diversity is ensured by the density-ratio term and the diffusion term serves as an enhancement.
Note that DG f low performs well even without the diffusion term (i.e., with γ = 0 , see Tables 13 & 14 in the appendix).
This deterministic variant of DG f low is a practical alternative with one less hyperparameter to tune.
DOT reﬁnes samples by constructing an Optimal Transport (OT) map induced by the WGAN discriminator.
The OT map is realized by means of a deterministic optimization problem in the vicinity of the generated samples.
If we further analyze the case of DG f low with γ = 0 and solve the resulting ordinary differential equation (ODE) using the backward Euler method,
[Equation]
DOT emerges as a special case when we consider a single update step of Eq.
(14) using gradient descent and set f (cid:48) ( t )
= log( t ) 2 with λ = 12 .
This connection of DG f low to DOT, an optimal transport technique, is perhaps unsurprising given the relationship between gradient ﬂows and the dynamical Benamou-Brenier formulation of optimal transport (Santambrogio, 2017).
E XPERIMENTS
In this section, we present empirical results on various deep generative models trained on multiple synthetic and real world datasets.
Our primary goals were to determine if (a) DG f low is effective in improving the quality of samples from generative models, (b) the proposed extension to other generative models improves their sample quality, and (c) DG f low is generalizable to different types of data and metrics.
Note that we did not seek to achieve state-of-the-art results for the datasets studied but to demonstrate that DG f low is able to signiﬁcantly improve samples from the bare generators for different models.
We experimented with three f -divergences, namely the Kullback-Leibler (KL) divergence, the
Jensen-Shannon (JS) divergence, and the log D divergence (Gao et al., 2019).
The speciﬁc forms of the functions f and corresponding derivatives are tabulated in Table 7 (appendix).
We compare DG f low with two state-of-the-art competing methods: DOT and DDLS.
In this section we discuss the main results and relegate details to the appendix.
Our code is available online at https://github.com/clear-nus/DGflow .
5. 2D D ATASETS
We ﬁrst tested DG f low on two synthetic datasets, 25Gaussians and 2DSwissroll, to visually inspect the improvement in the quality of generated samples.
We generated 5000 samples from a trained
WGAN-GP generator and reﬁned them using DOT, DDLS, and DG
f low.
We performed reﬁnement in the latent space for DDLS and directly in the data-space for DOT and DG f low.
Fig.
2 shows techniques
This implies that f ( t ) =
t log t − t + 1 , which is a twice-differentiable convex function with f (0)
= 1 . [Figure] the samples generated from the WGAN-GP generator (blue) and the reﬁned samples using different techniques (red) against the real samples from the training dataset (brown).
Although the WGANGP generator learned the overall structure of the dataset, it also learned a number of spurious modes.
DOT is able to reﬁne the spurious samples but to a limited degree.
In contrast, DDLS and DG f low are able to correct almost all spurious samples and are able to recover the correct structure of the data.
Visualizations for DG f low with different f -divergences can be found in the appendix (Fig.
[Table]
We also compared the different methods quantitatively on two metrics: % high quality samples and kernel density estimate (KDE) score.
A sample is classiﬁed as a high quality sample if it lies within 4 standard deviations of its nearest Gaussian.
The KDE score is computed by ﬁrst estimating the KDE using generated samples and then computing the log-likelihood of the training samples under the KDE estimate.
We computed both the metrics 10 times using 5000 samples and report the mean in Table 1.
The quantitative metrics reinforce the qualitative analysis and show that DDLS and DG f low signiﬁcantly improve the samples from the generator, with DG f low performing slightly better than DDLS in terms of the KDE score.
5. I MAGE E XPERIMENTS
[Figure]
We conducted experiments on the
CIFAR10 and STL10 datasets to demonstrate the efﬁcacy of DG f low in the real-world setting.
We followed the setup of Tanaka (2019) for our image experiments.
We used the Fr´echet Inception Distance (FID) (Heusel et al., 2017) and Inception Score (IS) (Salimans et al., 2016) metrics to evaluate the quality of generated samples before and after reﬁnement.
A high value of IS and a low value of FID corresponds to high quality samples, respectively.
We ﬁrst applied DG f low to GANs with scalar-valued discriminators (e.g., WGAN-GP, SNGAN) trained on the CIFAR10 and the STL datasets.
Table 2 shows that DG f low signiﬁcantly improves the quality of [Table] [Table] the samples in terms of the FID score and outperforms DOT on multiple models.
The corresponding values of the Inception score can be found in the Appendix (Table 11), which shows DG f low outperforms DOT on all models.
In Table 3, we reproduce previously reported IS results for generative models and other sample improvement methods (DRS, MH-GAN, and DDLS) for completeness.
DG f low performs the best in terms of relative improvement from the base score and even outperforms the state-of-the-art BigGAN (Brock et al., 2019), a conditional generative model, without the need for additional labels.
Qualitatively, DG f low improves the vibrance of the samples and corrects deformations in the foreground object.
Fig.
3 shows the change in the quality of samples when using DG f low where the leftmost columns show the image generated form the base models and the successive columns show the reﬁned sample using DG f low over increments of 5 update steps.
We then evaluated the ability of DG f low to reﬁne samples from generative models without corresponding discriminators , namely MMDGAN, OCFGAN-GP, VAEs, and Normalizing Flows (Glow).
We used the SN-DCGAN (ns) as the surrogate discriminator D φ for these models and ﬁne-tuned density ratio correctors D λ for each model as described in section 3.2.
Table 4 shows the
FID scores achieved by these models without and with reﬁnement using DG f low.
We obtain a clear improvement in quality of samples when these generative models are combined with DG f low.
5. C HARACTER -L EVEL L ANGUAGE M ODELING
Finally, we conducted an experiment on the character-level language modeling task proposed by Gulrajani et al.
(2017) to show that DG f low works on different types of data.
We trained a character-level GAN language model on the Billion Words Dataset (Chelba et al., 2013), which was pre-processed into 32-character long strings.
We evaluated the generated samples using the JS-4 and
JS-6 scores which compute the Jensen-Shannon divergence between the 4-gram and 6-gram probabilities of the data generated by the model and the real data.
Table 5 (a) shows that DG f low leads to [Table] [Table]
[Table]
[Table] [Table]
would
In product that chance could rol I said thowe toot lind talker .
I said this stood line talked 10 Now their rarning injurer hows
Now their warning injurer shows Police report in B0sbu does off
Police report inturner will befe We gine jaid 121 , one bub like
We gave wall said left out like In years in 19mbisuch said he h
In years in 1900b such said he h an improvement in the JS-4 and JS-6 scores. Table 5 (b) shows example sentences where DG f low visibly improves the quality of generated text.
C ONCLUSION
In this paper, we proposed a technique to improve samples from deep generative models by reﬁning them using gradient ﬂow of f -divergences between the real and the generator data distributions.
We also presented a simple framework that extends the proposed technique to commonly used deep generative models: GANs, VAEs, and Normalizing Flows.
Experimental results indicate that gradient ﬂows provide an excellent alternative methodology to reﬁne generative models.
Moving forward, we are considering several technical enhancements to improve DG f low’s performance.
At present,
DG f low uses a stale estimate of the density-ratio, which could adversely affect sample evolution when the gradient ﬂow is simulated for larger number of steps; how we can efﬁciently update this estimate is an open question.
Another related question is when the evolution of the samples should be stopped; running chains for too long may modify characteristics of the original sample (e.g., orientation and color) which may be undesirable.
This issue does not just affect DG f low; a method for automatically stopping sample evolution could improve results across reﬁnement techniques.
A CKNOWLEDGEMENTS
This research is supported by the National Research Foundation Singapore under its AI Singapore
Programme (Award Number: AISG-RP-2019-011) to H. Soh.
Thank you to J. Scarlett for his comments regarding the proofs.
A P ROOFS A. L EMMA 3.
Proof.
Gradient ﬂows in the Wasserstein space are of the form of the continuity equation (see Ambrosio et al.
(2008), page 281), i.e,
[Equation]
The velocity ﬁeld v in Eq. (15) is given by
[Equation]
where δ F δρ ( ρ ) denotes the ﬁrst variation of the functional F . The ﬁrst variation is deﬁned as
[Equation]
where χ = ν − ρ for some ν ∈ P 2 (Ω) .
Let’s derive an expression for the the ﬁrst variation of F .
In the following, we drop the notation for dependence on x for clarity,
[Equation]
[Equation]
[Equation]
Substituting δ F δρ ( ρ ) in Eq.
(16) we get,
[Equation]
Substituting v in Eq.
(15) we get the gradient ﬂow,
[Equation]
where ∆ xx and ∇ x · denote the Laplace and the divergence operators respectively.
A. L EMMA 3.
Proof.
Let f be an integrable function on X .
If J g has full column rank and g is an injective function, then we have the following change-of-variables equation (Ben-Israel, 1999; Gemici et al., 2016), (cid:90) X [Equation] (cid:113) det [Equation]
z
[Equation]
d z [Equation] This implies that the inﬁnitesimal volumes d x and d z are related as d x = [Equation]
det J
[Equation]
g J g ( z ) d z and the densities p Z ( z ) and q X ( x ) are related as p Z ( z ) =
q X ( g ( z ))
(cid:113) det J (cid:62) g J g
( z ) .
Similarly, p ˆ Z (ˆ z ) =
q ˆ X ( g (ˆ z )) (cid:113) det J (cid:62) g J g
(ˆ z ) .
Finally, the density-ratio p ˆ Z ( u ) /p
Z ( u ) at the point u ∈ Z is given by
[Equation]
B A D ISCUSSION ON DG f LOW FOR WGAN
We apply DG f low to WGAN models by treating the output from their critics as the logit for the estimation of density-ratio.
However, it is well-known that WGAN critics are not density-ratio estimators as they are trained to maximize the 1-Wasserstein distance with an unconstrained output.
In this section, we provide theoretical justiﬁcation for the good performance of DG f low on
WGAN models.
We show that DG f low is related to the gradient ﬂow of the entropy-regularized 1-Wasserstein functional F W µ : P 2 (Ω) → R , [Equation] 1-Wasserstein distance
[Equation]
negative
[Equation]
[Equation]
where µ denotes the target density, (cid:107) d (cid:107) Lip denotes the Lipschitz constant of the function d .
Let d ∗ be the function that achieves the supremum in Eq.
This results in the functional,
[Equation]
Following a similar derivation as in Appendix A.1, the gradient ﬂow of F W µ ( ρ ) is given by the following PDE,
[Equation]
If d ∗ is approximated using the critic ( d φ ) of WGAN, we get the following gradient ﬂow,
[Equation]
[Equation]
which is same as the gradient ﬂow of entropy-regularized f -divergence with f = r log r (i.e., the KL divergence) when d φ is treated as a density-ratio estimator.
The gradient ﬂow of entropy-regularized f -divergence with f = r log r is simpliﬁed below,
[Equation] [Equation]
[Equation]
The equality of Eq.
(30) and Eq.
(33) implies that DG f low approximates the gradient ﬂow of the 1-Wasserstein distance when the critic of WGAN is used for density-ratio estimation.
C F URTHER D ISCUSSION ON R ELATED W ORK Energy-based & Score-based Generative Models
DG f low is related to recently proposed energy-based generative models (Arbel et al., 2020; Deng et al., 2020) — one can view the energy functions used in these methods as a component that improves some base model.
For example, the Generalized Energy-Based Model (GEBM) (Arbel et al., 2020) jointly trains an implicit generative model with an energy function and uses Langevin dynamics to sample from the combination of the two.
Similarly, in Deng et al.
(2020), a discriminator that estimates the energy function is combined with a language model to train an energy-based text-generation model.
Score-based generative modeling (SBGM) (Song & Ermon, 2019; 2020) is another active area of research closely-related to energy-based models.
Noise Conditional Score Network (NCSN) (Song & Ermon, 2019; 2020), a SBGM, trains a neural network to estimate the score function of a probability density at various noise levels.
Once trained, this score network is used to evolve samples from noise to the data distribution using Langevin dynamics.
NCSN can be viewed as a gradient ﬂow that reﬁnes a sample right from noise to data; however, unlike DG f low, NCSN is a complete generative models in itself and not a sample reﬁnement technique that can be applied to other generative models.
Other Related Work Monte Carlo techniques have been used for improving various components in generative models, e.g., Grover et al.
(2018) proposed Variational Rejection Sampling which performs rejection sampling in the latent space of VAEs to improve the variational posterior and Grover et al.
(2019) used likelihood-free importance sampling for bias correction in generative models .
Wu et al.
(2020) proposed Latent Optimization GAN (LOGAN) which optimizes the latent vector as part of the training process unlike DG f low that reﬁnes the latent vector post training.
D I MPLEMENTATION D ETAILS D. 2D D ATASETS
Datasets The 25 Gaussians dataset was constructed by generating 100000 samples from a mixture of 25 equally likely 2D isotropic Gaussians with means {− 4 , − 2 , 0 , 2 , 4 } × {− 4 , − 2 , 0 , 2 , 4 } ⊂ Rand standard deviation 0.05.
Once generated, the data-points were normalized by 2 √ 2 following
Tanaka (2019).
The 2DSwissroll dataset was constructed by ﬁrst generating 100000 samples of the 3D swissroll dataset using make swiss roll from scikit-learn with noise=0.25 and then only keeping dimensions { 0 , 2 } .
The generated samples were normalized by 7.5.
Base Models We trained a WGAN-GP model for both the datasets.
The generator was a fullyconnected network with ReLU non-linearities that mapped z ∼ N (0 , I 2 × 2 ) to x ∈ R 2 .
Similarly, the discriminator was a fully-connected network with ReLU non-linearities that mapped x ∈ Rto R .
We refer the reader to Gulrajani et al.
(2017) for the exact network structures.
The gradient penalty factor was set to 10.
The models were trained for 10K generator iterations with a batch size of 256 using the Adam optimizer with a learning rate of 10 − 4 , β 1 = 0 .
5 , and β 1 = 0 .
We updated the discriminator 5 times for each generator iteration.
Hyperparameters We ran DOT for 100 steps and performed gradient descent using the Adam optimizer with a learning rate of 0.01 and β = (0 ., 0 . 9) as suggested by Tanaka (2019).
DDLS was run for 50 iterations with a step-size of 0.01 and the Gaussian noise was scaled by a factor of 0.1 as suggested by Che et al.
For DG f low, we set the step-size η = 0 .
01 , the number of steps n = 100 , and the noise regularizer γ = 0 .
We used the output from the WGAN-GP discriminator directly as a logit for estimating the density ratio for DDLS and DG f low.
Metrics We compared the different methods quantitatively on two metrics: % high quality samples and kernel density estimate (KDE) score.
A sample is classiﬁed as a high quality sample if it lies within 4 standard deviations of its nearest Gaussian.
The KDE score is computed by ﬁrst estimating the KDE using generated samples and then computing the log-likelihood of the training samples under the KDE estimate.
KDE was performed using sklearn.neighbors.
KernelDensity with a
Gaussian kernel and a kernel bandwidth of 0.1.
The quantitative metrics were averaged over 10 runs with 5000 samples from each method.
D. I MAGE E XPERIMENTS
Datasets CIFAR
(Krizhevsky et al., 2009) is a dataset of 60K natural RGB images of size 32 × 32 from 10 classes.
STL10 is a dataset of 100K natural RGB images of size 96 × 96 from [Table]
classes. We resized the STL10 (Coates et al., 2011) dataset to 48 × 48 for SNGAN and WGANGP, and to 32 × 32 for MMDGAN, OCFGAN-GP, and VAE since the respective base models were trained on these sizes.
Base Models for CIFAR
We used the publicly available pre-trained models for WGAN-GP,
SN-DCGAN (hi), and SN-DCGAN (ns).
We refer the reader to Tanaka (2019) for exact details about these models.
For SN-ResNet-GAN and OCFGAN-GP we used the pre-trained models from
Miyato et al.
(2018) and Ansari et al.
(2020) respectively.
We used the respective discriminators of
SN-DCGAN (ns), SN-DCGAN (hi), and WGAN-GP for density-ratio estimation when reﬁning their generators.
For the SN-ResNet-GAN (hi) generator, we used SN-DCGAN (ns) discriminator as the non-saturating loss provides a better density-ratio estimation than a discriminator trained using the hinge loss.
We trained our own models for MMDGAN, VAE, and Glow.
We used the generator and discriminator architectures shown in Table 6 for MMDGAN with d = 32 .
VAE used the same architecture with d = 64 .
Our Glow model was trained using the code available at https: //github.com/y0ast/Glow-PyTorch with a batch size of 56 for 150 epochs.
The density ratio correctors, D λ (see section 3.2), were initialized with the weights from the SN-DCGAN (ns) released by Tanaka (2019).
D λ was then ﬁne-tuned on images from SN-DCGAN (ns)’s generator and the generator being improved (e.g., MMDGAN and OCFGAN-GP) using SGD with a learning rate of 10 − 4 and momentum of 0.9.
We ﬁne-tuned D λ for 10000 iterations with a batch size of 64.
Base Models for STL
We used the publicly available pre-trained models (Tanaka, 2019; Ansari et al., 2020) for WGAN-GP, SN-DCGAN (hi), SN-DCGAN (ns), and OCFGAN-GP.
We trained our own models for MMDGAN and VAE with the same architecture and training details as CIFAR10.
We ﬁne-tuned the density ratio correctors for STL10 for 5000 iterations with other details being the same as CIFAR10.
Hyperparameters We performed 25 updates of DG f low for CIFAR10 and STL10 with a step size of 0.1 for models that do not require density ratio corrections.
For STL10 models that require a density ratio correction, we performed 15 updates with a step size of 0.05.
The noise regularizer ( γ ), whenever used, was set to 0.01.
Metrics We used the Fr´echet Inception Distance (FID) (Heusel et al., 2017) and Inception Score (IS) (Salimans et al., 2016) metrics to evaluate the quality of generated samples before and after reﬁnement.
The IS denotes the conﬁdence in classiﬁcation of the generated samples using a pretrained
InceptionV3 network whereas the FID is the Fr´echet distance between multivariate Gaussians ﬁtted to the 2048 dimensional feature vectors extracted from the InceptionV3 network for real and gen- [Table] erated data.
Both the metrics were computed using 50K samples for all the models, except Glow where we used 10K samples.
Following Tanaka (2019), we used the entire training and test set (60K images) for CIFAR10 and the entire unlabeled set (100K images) for STL10 as the set of real images used to compute FID.
D. C HARACTER L EVEL L ANGUAGE M ODELING
Dataset We used the Billion Words dataset (Chelba et al., 2013) which was pre-processed into 32-character long strings.
Base Model Our generator was a 1D CNN which followed the architecture used by Gulrajani et al.
Hyperparameters We performed 50 updates of DG f low with a step size of 0.1 and noise factor γ = 0 .
Metrics The JS-4 and JS-6 scores were computed using the code provided by Gulrajani et al.
(2017) at https://github.com/igul222/improved wgan training .
We used 10000 samples from the models to compute the JS-4 score.
E A DDITIONAL R ESULTS
[Figure]
Fig.
4 shows the samples generated by WGAN-GP (leftmost, blue) and reﬁned samples generated using DG f low with different f -divergences (red).
Fig.
5 shows the deterministic component, −∇ x f (cid:48)
( ρ 0 /µ )( x 0 ) , of the velocity for different f -divergences on the 2D datasets.
Fig.(right) shows the latent space distribution recovered by DG f low when applied in the latent space for the 2D datasets.
This latent space is same as the one derived by Che et al.
(2020), i.e., p t ( z ) ∝ p Z ( z ) exp
( d ( g θ ( z ))) which is shown in Fig.
6 (left) for both datasets.
Table 11 shows the comparison of DG f low with DOT in terms of the inception score for the CIFAR10 and STL10 datasets.
DG f low outperforms DOT signiﬁcantly for all the base GAN models on both the datasets.
Table 12 compares different variants of DG f low applied to MMDGAN, [Figure] [Table] [Figure]
[Table]
OCFGAN-GP, VAE, and Glow generators in terms of the inception score.
DG f low leads to a signiﬁcant improvement in the quality of samples for all the models.
Tables 13 and 14 compare the deterministic variant of DG f low ( γ = 0 ) against DOT and DDLS.
These results show that the diffusion term only serves as an enhancement for DG f low, not a necessity, and it outperforms competing methods even without added noise.
Table 15 shows the results of DG f low on
MMDGAN, OCFGAN-GP, and VAE models when the SN-DCGAN (ns) discriminator is directly used as a density-ratio estimator without an additional density-ratio corrector.
Figures 7, 8, 9, andshow the samples generated by the base model (left) and the reﬁned samples DG f low CIFAR STL
[Figure]
Runtime
DG f low performs a backward pass through d φ ◦ g θ to compute the gradient of the density-ratio with respect to the latent vector.
This results in the same runtime complexity as that of DOT and DDLS.
Tableshows a comparison of the runtimes of DOT, DDLS, and, DG f low on the 25Gaussians dataset under same conditions.
As expected, these reﬁnement methods have similar runtimes in practice.
The wall-clock time required for DG f low (KL) to reﬁne 100 samples from different base models on the CIFAR10 and STL10 datasets is reported in tables 9 and 10.
[Table] [Table]
[Table] [Table]
[Table]
[Table] [Table]
[Figure] [Figure]
[Figure] [Figure]