Can Push-forward Generative Models Fit Multimodal Distributions?
Abstract
Many generative models synthesize data by transforming a standard Gaussian random variable using a deterministic neural network.
Among these models are the
Variational Autoencoders and the Generative Adversarial Networks.
In this work, we call them "push-forward" models and study their expressivity.
We formally demonstrate that the Lipschitz constant of these generative networks has to be large in order to fit multimodal distributions.
More precisely, we show that the total variation distance and the Kullback-Leibler divergence between the generated and the data distribution are bounded from below by a constant depending on the mode separation and the Lipschitz constant.
Since constraining the Lipschitz constants of neural networks is a common way to stabilize generative models, there is a provable trade-off between the ability of push-forward models to approximate multimodal distributions and the stability of their training.
We validate our findings on one-dimensional and image datasets and empirically show that the recently introduced diffusion models do not suffer of such limitation.
Introduction
Generative modeling has become over the last years one of the most popular research topics in machine learning and computer vision.
From a mathematical perspective, the goal of generative modeling can be seen as predicting new synthetic samples from an unknown probability distribution ν on R d given the information of m true samples x i drawn from ν (the data distribution).
A general approach to solve this problem is to define a parametric family of probability distributions ( ν θ ) θ ∈ Θ and solve the problem
[Equation]
where D is a similarity measure between probability distributions and δ x is the delta distribution at x .
Beside their direct application (Sandfort et al., 2019; Antoniou et al., 2018), generative models have been used in numerous applications in various machine learning subfields, such as solving inverse problems (Ravuri et al., 2021; Ledig et al., 2017) or machine translation (Isola et al., 2017;
Yang et al., 2018).
However, most generative modeling methods still lack theoretical understanding and it remains often unclear whether the method approaches correctly the probability distribution ν or only generates samples that appear to have been drawn from ν without fully recovering the underlying structure of the distribution.
In this work, we focus on the particular class of push-forward generative models .
Those models have in common that for any θ ∈ Θ , the parametric distribution ν θ approaching ν is of the form
[Equation]
where µ p = N(0 , Id p ) is the Gaussian standard distribution in dimension p , # is the push-forward operator 1 , and
g θ : R p
→ R d is a deterministic neural network of parameter θ .
This class includes two of the most popular generative models: the Variational Auto-Encoders (VAEs) 2 (Kingma and
Welling, 2014) and the Generative Adversarial Networks (GANs) (Goodfellow et al., 2014).
It also includes other models such as most of normalizing flows (Rezende and Mohamed, 2015).
Deep neural networks are most of the time Lipschitz mappings by design, since their activation functions are generally Lipschitz.
In the literature, constraining the Lipschitz constant of a neural network is widely used as a way to increase its robustness (Virmaux and Scaman, 2018; Fazlyab et al., 2019), in particular to adversarial attacks (Goodfellow et al., 2015).
Common approaches to bound Lipschitz constants of neural networks are spectral normalization (Miyato et al., 2018), adding a gradient penalization in the loss (Gulrajani et al., 2017; Mohajerin Esfahani and Kuhn, 2018), or
Jacobian regularization (Pennington et al., 2017).
These approaches have been widely used to stabilize the training of GANs, where Lipschitz constraints have been first imposed on discrimators (Arjovsky et al., 2017; Kodali et al., 2017; Fedus et al., 2018), while recent state-of-the-art architectures such as BigGAN (Brock et al., 2018), SAGAN (Zhang et al., 2019) or StyleGAN2 (Karras et al., 2020) also impose similar constraints on the generators through spectral normalization (Brock et al., 2018;
Zhang et al., 2019), or Jacobian regularization (Karras et al., 2020).
In contrast to GANs, the recent study of Kumar and Poole (2020) shows that the decoder Jacobian in VAEs is implicitly regularized, which limits its Lipschitz constant.
Recently, Dhariwal and Nichol (2021) trained an unconditional Score-based Generative Model (SGM) (Song and Ermon, 2019; Ho et al., 2020) on ImageNet (Russakovsky et al., 2015) and achieved state-of-the-art generation.
To the best of our knowledge, there is no push-forward generative model capable of reaching this kind of performance on such a complex dataset without explicitly adding any conditional label information in the model, see (Brock et al., 2018) for instance.
SGMs (also known as diffusion models ) proceed as follows: first, noise is progressively added to the data distribution until we reach a standard Gaussian distribution.
Then this forward dynamics is reversed leveraging recent advances in deep learning and tools from score-matching (Hyvärinen, 2005; Vincent, 2011).
We refer to Song et al.
for an introduction on SGMs.
In those models, the parametric distribution is also of the form ν θ = g
θ # µ p , where g θ is the whole reverse diffusion dynamic (which can be seen as a composition of deterministic Lipschitz mappings) and p =
d ( N + 1) with N being the total number of steps in the dynamic (More details can be found in Appendix S2).
However, those models are not push-forward generative models in the strict sense of the term, since the push-forward mapping is not a simple neural network anymore.
An important difference is that optimization is not directly performed on the push-forward mapping itself but on an auxiliary function (the score).
We therefore categorize them as indirect push-forward generative models in this work.
Contributions of the paper.
In this paper, we study the expressivity of direct and indirect pushforward generative models in relation to the Lipschitz constant of the push-forward mapping they learn.
More precisely, in Section 3, for a Lipschitz function g and a given multimodal probability distribution ν , we formally demonstrate that the Lipschitz constant of g must necessarily be large in order for g # µ p to approximate ν correctly, as it has been already intuitively observed in the literature (Lu et al., 2020;
Luise et al., 2020; Khayatkhoei et al., 2018).
As a direct consequence, we exhibit lower bounds on D ( g # µ p , ν ) , where D is the total variation distance or the Kullback-Leibler divergence, with an explicit dependence on the Lipschitz constant Lip( g ) of g , which highlights that there is a fundamental trade-off for (direct) push-forward generative models between expressivity and stability of training.
In Section 4, we illustrate these theoretical results on several experiments, showing the difficulties of GANs and VAEs to simulate multimodal distributions.
We compare these models with SGMs and show experimentally that SGMs seem to be able to generate correctly multimodal distributions while keeping the Lipschitz constant of the score network relatively small, suggesting that these models do not suffer of such previously mentioned limitations.
All the proofs are postponed to the appendix.
Related Works
Assessing the efficiency of push-forward models is a recurrent and important question in the literature.
Sajjadi et al.
(2018) and Kynkäänniemi et al.
(2019) propose Precison and Recall metrics to assess GANs, aiming to measure simultaneously the mode collapse and the proportion of off-manifold generated samples.
Using similar metrics, Tanielian et al.
(2020) prove an upper bound on the precision of vanilla GANs (the proportion of generated samples which could have been generated by the target distribution).
To overcome this limitation, they simply propose to reject samples associated with large values of the generator Jacobian.
The intuition behind this idea is that those samples lie in regions of the space where the discontinuous optimal generator would "jump" between modes and so are off-manifold.
In the context of normalizing flows, it has been shown that the invertibility constraint limits the expressivity of the model.
Indeed, Cornish et al.
(2020) show that distributions generated by invertible normalizing flows have a support which is necessarily homeomorphic to the support of the latent distribution.
As an outcome, the Lipschitz constant of the inverse flow has to approach infinity to correctly approximate distributions lying on disconnected manifolds (Cornish et al., 2020; Hagemann and Neumayer, 2021; Behrmann et al., 2021).
To improve the expressivity of normalizing flows, it has been proposed in Cornish et al.
(2020) and Wu et al. (2020) to inject stochasticity in the model. Another line of research focuses on the fact that the model has access to only the empirical distribution ν n = 1 n (cid:80) i δ x i and not to the true target distribution.
For instance Nagarajan et al.
(2018) study to what extend GANs only memorize the data.
Gulrajani et al.
(2018) highlight the fact that common GAN benchmarks prefer training set memorization to a model which imperfectly fits the true distribution but covers more of its support.
Related to this, Stéphanovitch et al.
(2022) study specifically the Wasserstein GAN case, where the latent distribution is uniform and construct an optimal generator which minimizes the Wasserstein distance of order 1 between the push-forward measure and the empirical distribution, thus deriving a lower bound on the 1 -Wasserstein distance.
In the same paper, and more related to our work, the authors study the asymptotic case of an infinite number of data and show that most of the time the minimal 1 -Wasserstein distance between the push-forward measure and the target distribution remains strictly positive.
Push-forward measure and Lipschitz mappings
In this section, we study the properties of the push-forward measure g # µ p when µ p = N(0 , Id p ) is the standard Gaussian distribution in dimension p and g is a Lipschitz mapping.
First, for any probability measure γ on R d and any Borel set A of R d , we define the γ -surface area of A by
[Equation]
where A ε = { x ∈ R d : there exists a ∈ A , ∥ x − a ∥ ≤ ε } is the ε -extension of A and ∂ A is the boundary of A .
The γ - surface area can be interpreted as the mass of γ on the hypersurface ∂ A .
Note that the support of γ and A can be sets of intrinsic dimension smaller than d , which is most of the time the case when working with real data which are likely to live on low dimensional manifolds (Pope et al., 2020).
The main theoretical result of this paper establishes some properties of push-forward measures depending on the regularity of the push-forward mapping.
Theorem 1.
Let g : R p
→ R d be a Lipschitz function with Lipschitz constant Lip( g ) .
Then for any
Borel set A ∈ B ( R d ) ,
[Equation]
(1) where φ ( x ) =
(2 π ) − 1 / 2 exp[ − x 2 / 2] and Φ( x ) = (cid:82) x
−∞ φ ( t )d t .
In addition, we have that for any r ≥
[Equation]
Sketch of proof.
The proof of this result consists in establishing lower-bounds on ( g # µ p ) + ( ∂ A ) and g # µ p ( A r ) which can be expressed as Gaussian integrals.
We conclude upon combining this result and the Gaussian isoperimetric inequality, see Sudakov and Tsirelson (1978)
µ + p ( ∂ A ) ≥ φ (Φ − 1 ( µ p ( A ))) .
Note that (2) implies (1) upon remarking that (2) is an equality for r = 0 , dividing by r and letting r → 0 .
Theorem 1 recovers the Gaussian inequality in the case where g is the identity mapping and extends it to all Lipschitz mappings.
As the Gaussian inequality, Theorem 1 is dimension free, in the sense that neither d , nor p , nor the intrinsic dimension of g ( R p ) play a role in the lower bounds.
In the following section, we are going to use Theorem 1 to (i) give a lower bound on the Lipschitz constant so that push-forward generative models exactly match the data distribution, (ii) give a lower bound on the total variation and the 0. Kullback-Leibler divergence between the push-forward and data distributions which depends on the Lipschitz constant of the model.
3.1 Lower bounding the Lipschitz constant of push-forward mappings
Equation (1) implies that the Lipschitz constant of g must necessarily be large for g # µ p to be multimodal.
It provides indeed a lower bound on the Lipschitz constant of the mappings g which push µ p into a given measure ν .
In the extreme case where the support of ν is composed of disconnected manifolds, we retrieve that there doesn’t exist any Lipschitz mapping which pushes µ p into ν since it can be found Borel sets A with null ν -surface area but such that the right-hand term of (1) is strictly positive (which occurs when 0 < ν ( A ) < 1 ).
In the intermediate case where the support of ν is connected but ν is multimodal, the less mass ν has between modes, the larger must be the Lipschitz constant of the mappings which push µ p into ν .
Indeed, if ν has little mass between its modes, one can find sets A with small ν -surface area and such that 0 <
ν
( A ) < 1 .
As a toy example, we get an explicit bound on the Lipschitz constant of the mappings which push µ p into a mixture of two isotropic Gaussians.
Corollary 2.
Let ν =
λ N( m 1 , σ 2 Id d ) + (1 − λ )N( m 2 , σ 2 Id d ) with m 1 , m 2 ∈ R d , σ > 0 and λ ∈ (0 , 1) .
Assume that there exists g : R p
→ R d Lipschitz such that g # µ p = ν .
Then
[Equation]
Sketch of proof.
The proof of this result consists into applying Inequality (1) of Theorem 1 on the half-space H such that ∂ H is the equidistant line of the means of the two Gaussians, and then to explicit the value of ν + ( ∂ H ) .
See Figure S1 for a visualization of ∂ H in the univariate case.
Note that assuming there exists g : R p → R d such that g # µ p =
ν implies p ≥ d since ν covers the whole ambient space and so g must be a surjective mapping.
This bound is maximal in the balanced case when λ = 1 / 2 since Φ − 1 ( λ ) =
in that case.
Otherwise, the more unbalanced the modes are, the smaller the bound is since the two terms in the exponential compensate each other more and more.
Extending this corollary to mixtures of more than two Gaussians with different covariance matrices is technically difficult but we could expect a similar exponential growth in the square distance between modes since it depends mainly on the order of magnitude of the local minima of the distribution density.
As a by product of Theorem 1, we also get the following result which shows that (in the one-dimensional case) the optimal transport map for the ℓ 2 cost minimizes the Lipschitz constant of the push-forward mapping.
Corollary 3.
Let ν be a probability measure on R with density w.r.t.
the Lesbesgue measure and such that supp ( ν ) =
R .
Assume that there exists g : R p
→ R Lipschitz such that ν =
g # µ p .
Let us denote T OT =
Φ − 1 ν ◦ Φ the Monge map between µ 1 and ν , where Φ ν is the cumulative distribution function of ν .
Then we have Lip( g ) ≥ Lip( T OT ) .
To the best of our knowledge, extending this proposition to the case where d > 1 remains an open problem.
We show now that Equation (2) of Theorem 1 allows to derive lower bounds on similarity measures between the push-forward measure and the target distribution.
3.2 Lower bounds on similarity measures between probability distributions
Equation (2) provides a bound on the minimal mass the push-forward measure g # µ p can have on a given set when g is fixed with Lipschitz constant Lip( g ) .
As a consequence, if ν is a distribution such that there exists sets on which ν has less mass than the minimal quantity that g # µ p can reach on those sets given the value of Lip( g ) , then g # µ p cannot be equal to ν , implying that most of similarity measures between g # µ p and ν will be automatically strictly positive.
In the following, we consider that g and ν are fixed and we derive lower bounds on the total variation distance and the
Kullback-Leibler divergence between g # µ p and ν .
We recall that the total variation distance between two probability measures on R d , ν 0 , ν 1 is given by
[Equation]
Similarly, we define the Kullback-Leibler divergence between two probability measures on R d , ν 0 , ν 1 , using the Donsker-Varadhan representation (Dupuis and Ellis, 2011, Lemma 1.4.3a):
[Equation]
where B ( R d , R ) denotes the set of all bounded mappings from R d to R .
In the following, we will denote for any A ∈ B ( R d ) and r > 0 ,
[Equation]
where α g ( A , r ) and β g ( A , r ) are the lower bounds of g # µ p ( A r ) and g # µ p ( A r \ A ) provided by
Theorem 1.
We start by proving lower bounds on the total variation distance.
Theorem 4.
Let ν be a probability measure on R d and let g : R p
→ R d be a Lipschitz function.
Then,
[Equation]
Sketch of Proof.
The proof of this result consists in bounding from below the total variation distance on one hand by | g # µ p
( A r \ A ) − ν
( A r \ A )
| and on the other hand by | g # µ p ( A r ) − ν ( A r ) | for a given A ∈ B ( R d ) and a given r > 0 , and then applying Theorem 1.
Observe that (3) always holds but the right-hand term may become negative if the Lipschitz constant of g is large enough.
The main idea behind this bound is to find a set A and a real r > 0 such that ν has a lot of mass on A but almost no mass on A r \ A .
For instance, if ν is a distribution on two disconnected manifolds M 1 and M 2 , an optimal choice for A would either be M 1 or M 2 and the optimal r would be the distance between the two manifolds.
Using Theorem 4, one can derive smaller but more explicit lower bounds only depending on ν and the Lipschitz constant of g .
As a first example, we derive an explicit lower bound in the case where ν is a bi-modal distribution on two disconnected manifolds.
Corollary 5.
Let ν be a measure on R d on two disconnected manifolds M 1 and M 2 such that ν ( M 1 ) =
λ
and ν ( M 2 ) =
1 − λ , with λ ∈ [1 / 2 , 1) , and let g : R p
→ R d be a Lipschitz function.
Then,
[Equation]
where d ( M 1 , M 2 ) =
inf {∥ m 1 − m 2 ∥ : m 1 ∈ M 1 , m 2 ∈ M 2 } .
As a second example, we also get an explicit lower bound in the case where ν is a mixture of two isotropic Gaussians.
For simplicity we stick to the balanced case.
Corollary 6.
Let ν =
(1 / 2)[N( m 1 , σ 2 Id d ) + N( m 2 , σ 2 Id d )] with m 1 , m 2 ∈ R d and σ ≥ 0 .
Let g : R p
→
R d be a Lipschitz function.
Then,
[Equation]
In both corollaries, the lower bound tends to 1 / 2 when the distance between the modes tends to infinity, meaning that g # µ p is far from well approaching ν .
Note that the lower bound exhibited in
Corollary 5 is always strictly positive regardless of the value of the Lipschitz constant of g .
One can also observe that this latter bound is maximal in the balanced case, when λ = 1 / 2 , since the standard normal distribution concentrates its mass around 0 .
Finally, we end this section by deriving a similar lower bound on the Kullback-Leibler divergence between g # µ p and ν .
We consider the
Kullback-Leibler divergence since this is a measure of similarity between measures which is bounded and is very sensitive to the mismatch of supports between the generated and the data distributions.
Theorem 7.
Let ν be a probability measure on R d and let g : R p
→ R d be a Lipschitz function.
Then,
[Equation]
Sketch of Proof.
The proof of this result consists in setting f = ζχ A r \
A with ζ > 0 and where χ A is the characteristic function of the set A and plugging it in the Donsker-Varadhan representation in order to get a lower bound depending on the probability g # µ p ( A r \ A ) for a given A , a given r and a given ζ .
Then we apply Theorem 1 and we derive the optimal value of ζ .
As above, this bound always holds but the right-hand term becomes negative if Lip( g ) is large enough.
As for Theorem 4, the main idea is to find a set A and a real r such that ν has a lot of mass on A , but ν has almost no mass on A r \ A .
Observe that if ν ( A r \ A ) tends to 0 , the left-hand term of the bound tends to infinity.
This is coherent with the behavior of the Kullback-Leibler divergence. Similarly to
Corollary 6, we also get an explicit lower bound in the case where ν is a mixture of two isotropic Gaussians. As for Corollary 6, we stick to the balanced case for simplicity. Corollary 8.
Let ν =
(1 / 2) (cid:2) N( m 1 , σ 2 Id d ) + N( m 2 , σ 2 Id d ) (cid:3) with m 1 , m 2 ∈ R d and σ ≥ 0 .
Let g : R p
→
R d be a Lipschitz function.
We denote
[Equation]
0 : x ∈ R d } (cid:1) , and we suppose without loss of generality, that λ ∈ (0 , 1 / 2] .
Then,
[Equation]
where
[Equation]
Observe that this time, Lip( g ) is no longer the only dependency in g since the bound also depends on the proportion of the modes of g # µ p .
However, it should be noted that when g # µ p approximates correctly ν , λ is automatically close to 1 / 2 and so Φ − 1 (1 − λ ) is small in that case.
To conclude, this section, we highlight the fact that, if our results are dimension free in theory, the dimension might be hidden in the distances between modes and the Lipschitz constant of g when working with real datasets.
Indeed, the order of magnitude of the Euclidean distance between two samples x i is likely to increases with the dimension d .
As an outcome, the orders of magnitude of the distance between modes and so the Lipschitz constant that g must reach for correct generation probably increase with d also.
Experiments
In what follows, we illustrate the pratical implications of our results by training GANs, VAEs and
SGMs on simple bi-modal distributions.
More precisely, we show on one hand that generating multimodal distributions with GANs and VAEs is difficult since for those models, good generation necessarily involves generative networks with large Lipschitz constants.
On the other hand, we show that SGMs seem to be able to generate multimodal distributions while keeping the Lipschitz constant of the score network relatively small and thus do not suffer of the same limitation.
First, we focus on the univariate case where we can easily assess the Lipschitz constants of the networks.
Then we illustrate our results in higher dimensions by training the three models on datasets derived from
MNIST (LeCun et al., 1998).
In all our experiments, we use the same architecture for the VAE decoder and the GAN generator in order to offer rigorous comparisons of the different models.
For score-based modeling, we use architectures with similar numbers of learnable parameters.
All details on the experiments and architecture of the networks can be found in Appendix S5.
4.1 Univariate case
First, we train a VAE and a GAN with one-dimensional latent spaces on 50000 independent samples drawn from a balanced mixture of two univariate Gaussians ν = (1 / 2)[N( − m, 1) + N( m, 1)] for different values of m > 0 .
We also train a SGM on the same samples.
Histograms of generated distributions.
Figure 2 shows histograms of generated distributions for m = 10 with the three different models.
VAE models seem to generate Gaussians modes but interpolate significantly between them, while GANs do not interpolate but fail to retrieve the structure of the target distribution and forget parts of their support, which is known as mode collapse and is a common pitfall of such models (Arjovsky and Bottou, 2017;
Metz et al., 2017).
On the same task, SGMs do not suffer from such shortcomings.
In the following section, we will link the interpolation/mode-collapsing properties of these models with their Lipschitz constants.
Lipschitz constant and mass between modes.
In Figure 3 (right), we observe that the GAN generator reaches much larger Lipschitz constants than the VAE decoder.
This explains the difference of behaviors between GAN and VAE observed in Figure 2, as the mapping learned by the VAE is not stiff enough to concentrate the push forward measure on the two modes.
One possible explanation for the interpolating behavior of the VAE is that the Euclidean norm of the Jacobian of the VAE decoder is implicitly regularized during training, as it has been demonstrated in Kumar and Poole (2020).
Both GAN and VAE saturate the constraint on g # µ p ([ − m/ 2 , m/ 2]) provided by Theorem 1, meaning that the generative networks minimize the amount of mass between modes as much as their Lipschitz constants allow it.
Finally, we can observe that the score network is able to keep a relatively small Lipschitz constant compared to the GAN, while managing to interpolate less than the latter.
A probable explanation for this follows from the fact that the score network is used multiple time during inference.
Hence, the Lipschitz constant of the push-forward mapping (the whole generation dynamic) is likely much larger than the Lipschitz constant of the neural network itself, and so the model is able to push-forward a Gaussian distribution into a multimodal distribution keeping a relatively small Lipschitz constant of the score network.
Finally, in Figure 3 (left), we observe that when m increases, the Lipschitz constant of the VAE decoder and the GAN generator becomes rapidly much smaller than the value of the lower bound provided by Corollary 2.
This means that for m large enough it is not possible to close the gap between the data distribution and the push-forward distribution.
We highlight that this observation does not apply to SGMs since in this setting the network is applied multiple times.
[Figure]
Stability of GAN and mode collapse.
Odena et al.
(2018) suggested that the magnitude of the norm of the generator jacobian may be causally related to instability and mode collapse.
This is why many state-of-the-art GANs apply spectral normalization (Miyato et al., 2018) on their generators.
In Figure 4 (left), we show that this technique cannot be used when training GANs on multimodal distributions: since spectral normalization constraints the Lipschitz constant of the generator to be smaller than 1 , the GAN is trained towards concentrating in one of the modes of the distribution over interpolating massively between them.
This has been referred to as mode dropping by Khayatkhoei et al.
To complete this analysis, we also train the GAN adding an additional gradient penalty term 10 /L 2 max z ∼ N(0 , Id p ) ( ∥∇ z g θ ( z ) ∥
22 − L ) 2 , in the generator loss function, similarly to WP-GAN (Gulrajani et al., 2017), where L is an hyperparameter corresponding to the targeted Lipschitz constant.
As expected, we can observe in Figure 4 (right), that when Lip ( g ) increases, the GAN begin to generate both modes but becomes also more and more prone to mode collapse.
This illustrates the fundamental trade-off between expressivity and robustness in push-forward generative models.
Influences of generator depth and time of training.
In Figure 5, we study the effect of increasing the number of layers of the generative network as well as increasing the training time on the value of the Lipschitz constant of the VAE decoder and the GAN generator.
In the VAE setting, the Lipschitz constant increases linearly with the depth of the decoder.
This is not the case in the GAN setting, where increasing the size of the model seems to dramatically affect its stability.
For both models, the Lipschitz constants of the generative network grow with the number of epochs.
Yet this growth seems to be logarithmic for the VAE and the GAN seems to becomes more unstable as the number of epochs increases.
[Figure]
Influence of generator architecture.
Finally, we study in Figure 6 the impact of the architecture of the generative network (i.e. the VAE decoder and the GAN generator) on its Lipschitz constant as well as on the training stability of the model by comparing three different architectures: first, we use a simple feed-forward network as precedently, then we add additive skip-connections of type "resnet" (He et al., 2016) to the previous backbone, and last we add concatenation skip-connections of type "densenet" (Huang et al., 2017) instead of additive skip-connections. For both models, it seems that more expressive decoder architectures do not help to reach larger values of Lipschitz constant. However, one can observe that in the GAN setting, even if the model remains certainly too unstable for correct distribution generation, adding additive skip-connections seems to stabilize the training a little since the colored bands are narrower than for the two other models. This suggests that some generator architectures may be better than others at learning mappings with large Lipschitz constants while staying stable.
[Figure]
4.2 Experiments on MNIST
We train a VAE, a GAN and a SGM on two datasets derived from MNIST (LeCun et al., 1998): first, two images of two different digits ( 3 and 7 ) are chosen and 10000 noisy versions of theses images are drawn with a noise amount of σ = 0 .
15 , forming a dataset of n =
independent samples drawn from a balanced mixture of two Gaussian distributions in dimension 784 = 28 × 28 .
Second, we train the models on the subset of all 3 and 7 of MNIST.
We emphasize that our goal is not reach state-of-the-art performance on this problem but rather to illustrate our theoretical results in a moderate dimensional setting.
Mixture of Gaussians.
For this experiment, we set the dimension of the latent space in the GAN and the VAE to 784 = 28 × 28 since it is the intrinsic dimension of the support of the data distribution.
In order to visualize the interpolation between modes, we project the data on the line passing through the mean of each Gaussian, i.e. the two original clean images, and we plot histograms of the onedimensional projections.
In order to understand which bins of data in the histograms correspond to which digit, we train a classifier and we assign a color in function of which digit the data have been classified as.
Results can be found in Figure 7 top.
Moreover, GAN and VAE both fail to generate noisy versions of the images.
As in the univariate case, the SGM is able to not interpolate between modes and seem to retrieve the Gaussian structure of the modes.
This suggests that while direct push-forward models fail at representing multimodal distributions, considering stacked models with noise input at each step (as in SGM) might help to close the gap between the generated and the data distributions.
However SGM does not manage to retrieve the right modes proportions.
This is a well-known shortcoming of score-based models which has been studied in (Wenliang and Kanagawa, 2020).
Subset of MNIST.
Finally we train the three different models on the subset of MNIST composed of all 3 and 7 (no Gaussian noise was added).
We choose a latent dimension of 20 for the VAE and the GAN.
Since the Euclidean distance is not a meaningful metric to compare the different digits of
MNIST, we use the deep Wasserstein embedding proposed by Courty et al.
(2018): an autoencoder is learned in a supervised fashion such that the Euclidean distance in the latent space approximates the
Wasserstein distance between pairs of images of MNIST.
In the learned Wasserstein space, we project data on the line passing through the Euclidean barycenters of all 3 and 7 and plot histograms of projections, using the same classifier as before.
Results can be found in Figure 7 (bottom).
Note that the distribution does not exhibit strong multimodality features contrary to the mixture of Gaussians settings, see Figure 7.
As before, the VAE interpolate between modes, the GAN manages to not interpolate but generate a narrower histogram, and the score-based model does not interpolate and seems to recover the structure of the distribution, but doesn’t retrieve the right modes proportions.
However, we emphasize that all these models seem to perform better than on the previous dataset.
A possible explanation of this is that the modes are less separated than in the Gaussian mixtures and therefore the model is easier to train.
Discussion
In this work, given a Lipschitz mapping g and a measure ν , we derived lower bounds on the total variation distance and the Kullback-Leibler divergence between the push-forward measure g # µ p and ν depending on the Lipschitz constant of the mapping g .
These bounds indicate how the mass between the modes of the push-forward measure depends on the Lipschitz constant of the push-forward mapping.
They highlight the trade-off between the ability of VAEs and GANs to fit multimodal distributions and the stability of their training.
A common assumption in the imaging literature, validated empirically by Pope et al.
(2020), is that distributions of natural images live on low dimensional manifolds.
Understanding whether these distributions are composed of separated modes or not remains, to the best of our knowledge, an open problem.
To that extent, the fact that unsupervised push-forward generative models perform well on datasets such as CelebA (Liu et al., 2015) could possibly be, in regard of our work, an indicator that the data distributions of those datasets are unimodal, or at least not composed of well separated modes.
Several techniques have been proposed in the literature to fit data distributions on disconnected manifolds.
Most of them consist in overparametrizing the model, either by using stacked generative networks (Khayatkhoei et al., 2018; Mehr et al., 2019) or by learning a more complex latent distribution than the standard Gaussian (Gurumurthy et al., 2017; Rezende and Mohamed, 2015; Kingma et al., 2016; Luise et al., 2020).
Other methods consist in rejecting a posteriori samples associated to large values of the Jacobian generator (Tanielian et al., 2020; Issenhuth et al., 2020).
In this work, we empirically showed that score-based models seemed to be able to fit separated manifolds without model overparametrization or additional posterior sample rejection scheme.
This suggests that the structure of the generation dynamic in these models is particularly adapted to (indirectly) learn mappings with large Lipschitz constants.
Their good performance on multimodal distributions might follow from the fact that these models do not optimize directly the push-forward mapping itself and/or that noise is injected at each step during the generation process.
Hence, a future perspective of work would be to study what are the structural aspects of diffusion models that play a significant role in their expressivity.
A possible limitation of this work is that the bounds derived on the Kullback-Leibler divergence and total variation distance are not tight (see Appendix S6), mainly because they take no account of the fact that when interpolating, g # µ p has automatically less mass than ν on the modes since a significant amount of its total mass is between them.
In future work, we plan to tighten the gap between our bounds and the true distance.
S Organization of the supplementary
The supplementary is organized as follows.
First, in Appendix S2, we give details on why the generated distribution in each aforementioned generative model is a push-forward of a standard Gaussian distribution. In Appendix S3, we give the full proofs of all the theoretical results of the paper.
In Appendix S4, we show a generalization of Corollary 5 in the case there is more than two disconnected manifolds.
In Appendix S5, we give details on the experiments.
Finally, in Appendix S6, we provide additional experimental results and additional visualizations of histograms of generated distributions for the univariate case, and generated data for the experiments on MNIST. S More details on push-forward generative models In this section, we specify why in each aforementioned model the generated distribution is of the form g θ # µ p with µ p = N(0 , Id p ) being the standard Gaussian distribution in dimension p and g θ : R p
→ R d being a deterministic mapping of parameter θ . S2.1 Direct push-forward models
In GANs, the generated distribution is trivially of the form g θ # µ p , where g θ is the generator and p is the dimension of the latent space.
This is also the case in most of normalizing flow models, where g θ is a composition of neural flows and p is automatically equal to d since the networks must be invertible.
In the Gaussian-VAE model described in Kingma and Welling (2014), the conditional probability p θ ( x | z ) , with z ∈ R p and x ∈ R d being respectively the latent and observable variables, is of the form p θ ( x | z ) =
N( f θ ( z ) , h 2 θ ( z ) Id d ) , where f θ ( z ) and g θ ) are the outputs of the decoder.
This is often simplified in practice to p θ ( x | z ) =
N( f θ ( z ) , c
2 Id d ) with c > 0 being an hyperparameter of the model.
Thus the generated distribution in Gaussian-VAE is of the form g # µ p , where g θ : R p + d → R d is the neural network defined as
[Equation]
Moreover, if relaxing the conditional probability to p θ ( x | z ) =
N( f θ ( z ) , h 2 θ ( z ) Id d ) or p θ ( x | z ) =
N( f θ ( z ) , c 2 Id d ) instead of δ f θ ( z ) is crucial for the training of VAEs, this is not particularly relevant during inference since the relaxed conditional distribution is simply a noisy version of the pushforward measure f θ # µ p .
For this reason, the inference in VAEs is often done by simply sampling from the single output f θ ( z ) of the decoder, with z ∼ µ p , and so the generated distribution is trivially of the form f θ # µ p in that case.
S2.2 Score-based generative models
In diffusion models, the generation process is an Euler-Maruyama discretization of the reverse-time denoising diffusion (Song et al., 2020)
[Equation]
where ( X t ) t ∈ [0 ,T ] is a random process on R d , ( B t ) t ∈ [0 ,T ] is a Brownian motion, and where f ( ., t ) :
R d → R d is a vector-valued function called the drift operator, g : R → R is a real-valued function called the diffusion coefficient and ∇ log p t ( X t ) is the score of the marginal law of X t , which is approximated by a neural network s θ ( X t , t ) .
This discretization yields for instance, for an appropriate choice of f and g , to the annealed Langevin dynamic (Song and Ermon, 2019):
[Equation]
where α k = εσ 2 k /σ 20 with ε > 0 being an hyperparameter of the model and ( σ k ) k ≥ 0 is such that it exists K (another hyperparameter of the model) such that ( σ Ki ) i ≥ 0 is a geometric progression, and for all k ≥ 0 , σ k = σ K ⌊ k/K ⌋ .
Denoting for k ≥ 1 , h kθ : R 2 d → R d the function defined as
[Equation]
it follows that when the data are generated with a Langevin dynamic of N iterations, the generated distribution is of the form g θ # µ p with p =
d ( N + 1) and where g θ : R d ( N +1) → R d is the function defined as
[Equation]
S Proofs of the theoretical results S3.1 Proof of Theorem
We start by recalling the Gaussian isoperimetric inequality Sudakov and Tsirelson (1978).
Lemma S9.
Let A ∈ B ( R p ) and µ p
= N(0 , Id p ) .
Then we have
[Equation]
where φ ( x ) = (2 π ) − 1 / 2 exp[ − x 2 / 2] and Φ( x ) = (cid:82) x
−∞ φ ( t )d t .
Equivalently, for all r ≥ [Equation]
In particular, using Lemma S9, one can show that among all sets of given Gaussian measure µ p , half-spaces have the minimal µ p -surface area.
We are now ready to turn to the proof of Theorem 1.
Proof of Theorem 1.
Let A ∈ B ( R d ) such that g # µ p ( A ) > 0 (note that if g # µ p ( A ) =
then the result is trivial).
First, we show that for any ε > 0 , g (( g − 1 ( A ))
ε/ Lip( g ) )
⊂
A ε .
Let x be in g (( g − 1 ( A ))
ε/ Lip( g ) ) .
There exists z 1 ∈ ( g − 1 ( A ))
ε/ Lip( g ) such that g ( z 1 ) = x .
There also exists z 2 ∈ g − 1 ( A ) such that
[Equation]
Hence, we have that
[Equation]
where a = g ( z 2 ) .
Since z 2 ∈ g − 1 ( A ) , a ∈ A , and therefore x ∈ A ε .
Using this result, the fact that g # µ p ( B ) = µ p ( g − 1 ( B )) and B ⊂ g − 1 ( g ( B )) for any B ∈ B ( R d ) , we have
[Equation]
Using Lemma S9, we have
[Equation]
Combining this result and (S1), we get that
[Equation]
In addition, using Lemma S9, we have for all r [-≥ 
-]≥µ p
[Equation]
g
[Equation]
A [Equation] r [Equation]
Lip( g ) + Φ [Equation] 1 ( µ p ( g − 1 ( A )))) . Using this result and that g (( g − 1 ( A )) r/ Lip( g ) )
⊂ A r , we have for any r ≥ [Equation] − 1 ( A ))
r/ Lip( g ) ) ≥ Φ( r/ Lip( g ) + Φ − 1 ( g # µ p ( A ))) .
S3.2 Proof of Corollary
We prove the corollary when ν = λ N( − m, σ 2 Id d ) + (1 − λ )N( m, σ 2 Id d ) since the problem can always be reduced to that case by translation and setting m = ( m 2 − m 1 ) / 2 .
Let H be defined by
H = { x ∈ R d | m T x ≥ 0 } .
Note that for any x ∈ ∂ H , ∥ x − m ∥ =
∥ x + m ∥ .
Since the problem is invariant by rotation, we can consider without any loss of generality that m = ( ∥ m ∥ , 0 , . .
In that case, we have ν =
ν 1 ⊗ N(0 , σ 2 Id d − 1 ) , where ν 1 = λ N( −∥ m
[Equation]
, σ 2 ) + [Equation] 1 − λ )N( ∥ m ∥ , σ 2 ) , and ⊗ is the tensor product between measures.
In this case, we have that H
= { x 1 ≥ 0 } × R d − 1 .
Therefore, we have
[Equation]
where p ν and p ν 1 are the respective densities of ν and ν [Equation] and h is the density of N(0 , σ 2 I d − 1 ) .
It follows that
[Equation]
Applying Theorem 1, we get that
[Equation]
Furthermore, one can derive that
[Equation]
Observing that λ − ν
( H ) is an increasing function of λ and λ − ν ( H )
=
if λ = 1 / 2 , we get that λ ≤ ν ( H ) if λ ≤ 1 / 2 and λ ≥ ν ( H ) if λ ≥ 1 / 2 .
Since φ ◦ Φ − 1 reaches its maximum in 1 / 2 , it follows that for any λ ∈ (0 , 1) we have
[Equation]
and thus
[Equation]
which concludes the proof.
[Figure]
S3.3 Proof of Corollary
Since ν admits a density p ν with respect to the Lesbegue measure, it follows that Φ ν is absolutely continuous and therefore differentiable almost everywhere w.r.t.
the Lebesgue measure using the
Lebesgue differentiation theorem.
Moreover, since supp( ν ) =
R , it follows that Φ ν : R → (0 , 1) is increasing and therefore is bijective, and so T OT =
Φ − 1 ν ◦ Φ is also differentiable almost everywhere w.r.t.
the Lebesgue measure and bijective, with inverse T − 1OT =
Φ − 1 ◦ Φ ν , using (Peyré and Cuturi, 2019, Remark 2.29).
Therefore, for any x ∈ R we have
[Equation]
Let y ∈ R .
Using Theorem 1 with A =
( −∞ , y ] we get that for any g : R p
→ R , Lipschitz such that g # µ p = ν ,
[Equation]
and so, since T OT is bijective
[Equation]
which concludes the proof.
S3.4 Proof of Theorem
Let A ∈ B ( R d ) and let r > 0 .
We have on one hand
[Equation]
Using Theorem 1, we get
[Equation]
and so
[Equation]
where α g ( A , r ) =
Φ (cid:0) r/ Lip( g ) + Φ − 1 ( g # µ p ( A )) (cid:1) .
On the other hand, we have [Equation] ) | ≤ | g # µ p ( A r ) − ν ( A r ) | + | ν ( A r ) |
≤ d TV ( g # µ p , ν )
+ ν ( A r \ A )
+ ν ( A ) .
Using Theorem 1, we get | g # µ p ( A r ) | ≥ Φ (cid:0) r/ Lip( g ) + Φ − 1 ( g # µ p ( A )) (cid:1) , and so d TV ( g # µ p , ν ) ≥ α g ( A , r ) − ν ( A ) − ν
( A r \ A ) , which concludes the proof.
S3.5 Proof of Corollary
To prove Corollary 5, we will need the following lemma:
Lemma S10.
Let A ∈ B ( R d ) and r > 0 .
We denote B =
( A r ) c .
Then
B r ⊂ ¯ A c , where ¯ A c denotes the closure of the complementary of A .
Proof.
Let x ∈ B r .
There exists b ∈ B such that ∥ x − b ∥ ≤ r .
Moreover, since B =
( A r ) c , it follows that for all a ∈ A , ∥ b − a ∥ > r .
Then r < ∥ b − x ∥ + ∥ x − a ∥ , and so, it follows that for all a ∈ A , ∥ x − a ∥ > 0 .
Thus x ∈ ¯ A c .
Now we are ready to turn to the proof of Corollary 5.
Proof of Corollary 5.
We set r = d ( M 1 , M 2 ) / 2 and A
=
( M 1 ) r .
Using Theorem 4, we have
[Equation]
First we suppose that g # µ p ( A ) ≥ ν ( A ) : since Φ is a non-decreasing function, it follows that
[Equation]
Moreover min { g # µ p
A ) , ν ( A ) } =
ν ( A ) =
λ = Φ(Φ − 1 ( λ )) and so it follows
[Equation]
since ν has no mass on A r \ A .
Now we suppose that g # µ p ( A ) ≤ ν ( A ) : we then set B =
A c .
Since g # µ p ( A ) ≤ ν ( A ) , we have g # µ p ( B ) ≥ ν ( B ) .
Applying Theorem 4, and the same reasoning as before we get
[Equation]
Using Lemma S10, we get that ν ( B r \ B ) ≤ ν ( ¯ A c \ ( A r ) c ) but ν ( ¯ A c \ ( A r ) c ) = 0 since ν has no mass on ¯ A c \ ( A r ) c except on its boundary and so its follows that
[Equation]
since Φ − 1 (1 − λ )
= − Φ − 1 ( λ ) .
Since λ ≥ 1 / 2 , it follows that Φ − 1 ( λ ) ≥ 0 and so
[Equation]
which concludes the proof.
S3.6 Proof of Corollary
As previously, we prove the corollary when ν = (1 / 2)[N( − m, σ 2 Id d ) + N( m, σ 2 Id d )] since the problem can always be reduced to that case by translation and setting m = ( m 2 − m 1 ) / 2 .
Since the problem is invariant by rotation, we can assume without any loss of generality that m = ( ∥ m ∥ , 0 , . . . , 0) .
Let H be the half-space of R d defined by H =
( −∞ , 0]
× R d − 1 and we set r = ∥ m ∥ / 2 σ .
First we suppose that g # µ p ( H ) ≥ ν ( H ) : using Theorem 4, we get that
[Equation]
ν ( H r \ H ) , with H r = ( −∞ , ∥ m ∥ / 2 σ ] × R d − 1 .
On one hand we have that ν
= ν 1 ⊗ N(0 , σ 2 Id d − 1 ) , where ν 1 = (1 / 2)[N( −∥ m ∥ , σ 2 ) + N( ∥ m ∥ , σ 2 )] and so ν ( H r \ H )
=
ν 1 ([0 , ∥ m ∥ / 2 σ ]) .
On the other hand we have that min { g # µ p
( H ) , ν ( H ) } =
ν ( H ) and g # µ p ( H ) ≥ 1 / 2 since g # µ p ( H ) ≥ ν
( H ) .
Hence it follows that
[Equation]
Now we suppose that g # µ p ( H ) ≤ ν ( H ) : we then set H
= (0 , + ∞ ]
× R d − 1 .
Since g # µ p ( H ) ≤ 1 / 2 , we get that g # µ p ( H 2 ) ≥ 1 / 2 and so g
# µ p ( H 2 ) ≥ ν
( H 2 ) .
Hence we retrieve the previous case and so it follows that
[Equation]
Since ν 1 ([ −∥ m ∥ / 2 σ, 0]) =
ν 1 ([0 , ∥ m ∥ / 2 σ ]) , we get in both cases d
[Equation]
g [Equation] [Equation] , ∥ m ∥ / 2 σ ]) .
Now we derive the value of ν 1 ([0 , ∥ m ∥ / 2 σ ]) : ν 1 ([
[Equation]
, ∥ m
[Equation]
[Equation]
]) = (1 /
[Equation]
) (cid:82) m/ 2 σ
[Equation]
/ 2 exp[ − (
[Equation]
[Equation]
] dx +(1 /
[Equation]
which concludes the proof
[Equation]
S3.7 Proof of Theorem
Let A ∈ B ( R d ) , r > 0 and ζ > 0 .
We set for any x ∈ R d f ( x ) =
ζχ A r \ A ( x ) , where χ A denotes the characteristic function of the set A .
Since f is bounded, it follows that
[Equation]
Using Theorem
[Equation]
we get
[Equation]
Thus we get
[Equation]
where the functional J is defined by
[Equation]
Differentiating J with respect to ζ [Equation] we get that
[Equation]
, where β g ( A , r ) =
Φ (cid:0) r/ Lip( g ) + Φ − 1 ( g # µ p ( A )) (cid:1) − g
# µ p
( A ) .
Applying the first order condition, we get that:
[Equation]
By re-injecting the value of ζ ∗ , we get
[Equation]
which concludes the proof.
S3.8 Proof of Corollary
As previously, we prove the corollary when ν = (1 / 2)[N( − m, σ 2 Id d ) + N( m, σ 2 Id d )] since the problem can always be reduced to that case by translation and setting m = ( m 2 − m 1 ) / 2 .
Since the problem is invariant by rotation, we can assume without any loss of generality that m = ( ∥ m ∥ , 0 , . . . , 0) .
Furthermore, observe that the half-space { ( m 2 − m 1 ) T ( x − ( m 1 + m 2 ) / 2) ≤ 0 : x ∈ R d } becomes ( −∞ , 0] × R d − 1 in that case, and that the condition λ ∈ (0 , 1 / 2] is indeed non-restrictive since the problem is invariant by rotation.
We set as before H =
( −∞ , 0]
× R d − 1 and r = ∥ m ∥ / 2 σ .
Applying Theorem 7, we get
[Equation]
# µ p || ν ) ≥ β g ( H [Equation] r ) log (cid:16) β g ( H [Equation]
r ) ν ( H r \ H ) (cid:17) + (
[Equation]
β g ( H [Equation] r )) log (cid:16) 1 − β g ( H ,r ) 1 − ν ( H r \ H ) (cid:17) .
On one hand, we get
[Equation]
noting λ
= g
# µ p ( H ) .
We replaced Φ − 1 ( λ ) by − Φ − 1 (1 − λ ) in order to emphasize that Φ − 1 ( λ ) ≤since λ ≤ 1 / 2 .
Observe that if we supposed λ ≥ 1 / 2 , we would have β g ( H c , r ) ≥ β g ( H , r ) and so the bound that we would have found by reasoning on H would have been sub-optimal.
On the other hand, observing as before that ν
= ν 1 ⊗ N(0 , σ 2 Id d − 1 ) , where ν 1 = (1 / 2)[N( −∥ m ∥ , σ 2 ) + N( ∥ m ∥ , σ 2 )] , we get that
[Equation]
which concludes the proof.
S Additional theoretical result
In this section we derive a generalization of Corollary 5 when ν is a distribution whose support is composed of more than two disconnected manifolds.
Corollary S11.
Let ν be a measure on R d on N disconnected manifolds ( M 1 , . . .
, M N ) , and let g : R p
→ R d be a Lipschitz function.
Then,
[Equation]
where for A , B ∈ B ( R d ) , d ( A , B ) = inf {∥ a − b ∥ : a ∈ A , b ∈ B } , and λ =
ν (cid:18) (cid:70) i ∈ I M i (cid:19)
if ν (cid:18) (cid:70) i ∈ I M i (cid:19) ≥ 1 / 2 and λ = 1 − ν (cid:18) (cid:70) i
[Equation]
I M i (cid:19) otherwise.
Proof.
Let I ⊂ (cid:74) 1 , N (cid:75) .
First, we suppose that ν (cid:18) (cid:70) i ∈ I M i (cid:19) ≥ 1 / 2 .
Since ν can be seen as a bi-modal distribution on the two disconnected sets (cid:70) i ∈ I M i and (cid:70) j ∈ (cid:74) 1 ,N (cid:75) \
I
M j , we can apply Corollary 5.
Thus we get
[Equation]
If ν (cid:18) (cid:70) i ∈ I M i (cid:19) ≤ 1 / 2 , we can still apply Corollary 5 by interchanging the roles of (cid:70) i ∈ I M i and (cid:70) j ∈
[Equation]
[Equation]
N
[Equation]
\ I
M j , thus we get also Inequality (S4) in that case, which concludes the proof.
S Experimental details
We detail our experiments in dimension 1 in Appendix S5.1.
In Appendix S5.2, we give details on our experiment on the synthetic mixture of two Gaussians derived from MNIST.
Finally, we detail the experiment on the subset of all 3 and 7 of MNIST in Appendix S5.3.
We trained our models using[-
-]
NVIDIA Titan Xp from the proprietary server of our institution with an estimated total training time of approximately 175 GPU hours. Code is available here 3 .
S5.1 Univariate case
In the univariate case we use a simple 3 -layer Multi Layer Perceptron (MLP) of shape (1 , 128 , 256 , 1) as decoder for the VAE and as generator for the GAN.
The network has a total of 33537 learnable parameters.
The score network uses also a a 3 -layer MLP block, this time of shape (1 , 96 , 196 , 1) , in which at each layer is injected the noise information transformed by a positional encoding (Vaswani et al., 2017) and then by another MLP block size (16 , 32 , 64) , see Figure S2.
The score network has a total of 34665 learnable parameters.
In all three models, we use LeakyReLU
(Maas et al., 2013) as non-linearity with a negative slope of 0 . 2 .
The three models are trained during 400 epochs with a batch size of 1000 using ADAM (Kingma and Ba, 2015) with a momentum of 0 .
9 and a learning rate of 10 − 4 .
In the following, we give more specific details for each model.
Variational autoencoder.
We use the vanilla VAE model as described in Kingma and Welling (2014).
In the following, we denote θ and ϕ the respective parameters of the decoder and the encoder.
The decoder f ϕ is composed of an MLP block of size (1 , 256 , 128) followed by two parallel fully connected layers of shape (128 , 1) which gives two outputs f 1 ϕ ( x ) and f 2 ϕ ( x ) .
Then the input z of the decoder g θ is obtained by the so-called reparametrization trick, which consists in sampling z ∼ q z | x ϕ , where q z | x ϕ = N( f 1 ϕ ( x ) , exp[ f 2 ϕ ( x )]) .
During training, the model minimizes the following loss function:
[Equation]
where p x | z θ = N( g θ ( z ) , c 2 Id d ) and ELBO is the Evidence Lower Bound (Blei et al., 2017), defined as follows:
[Equation]
The standard deviation c in p x | z θ is an hyperparameter of the model.
For our experiments, we observed that c = 0 .
1 gave good results.
Generative adversarial network.
As for the VAE, we use the vanilla GAN model as described in Goodfellow et al.
The discriminator is 4 -layer MLP of shape (1 , 512 , 256 , 128 , 1) with spectral normalization (Miyato et al., 2018) in order to reduce as much as possible mode collapse.
We train the model using the vanilla adversarial loss, that the discriminator d ϕ tries to maximize and that the generator g θ tries to minimize
[Equation]
We also tried with the hinge version of the adversarial loss, as proposed in Lim and Ye (2017) and
Tran et al.
and we obtained similar results.
Score-based generative modeling.
Our diffusion model is similar to the model introduced by
Song and Ermon (2019).
The neural network s θ learns to approximate, for a given x and a given σ , the score
∇ x p ν ( x, σ ) of the data distribution convoluted with a Gaussian distribution of standard deviation σ .
This is done by first defining a geometrical progression { σ i } Li =1 where L = 10 and where the ratio is chosen such that σ L ≈ 0 . 01 , and then minimizing the Fischer divergence (Vincent, 2011)
[Equation]
Then, in order to generate data, we use an annealed Langevin dynamic scheme as defined in Song and Ermon (2019).
In the Langevin dynamic, we set the step size to 2 × 10 − 5 and the number of step for each value of σ to
Influence of generator depth.
For this experiment, we increase the number of layers of the VAE decoder and the GAN generator from 2 to 6 .
At each new layer, we double the number of neurons at the previous layer.
For instance, the generative network with 2 layers is thus an MLP of shape (1 , 128 , 1) and the one with 6 layers is an MLP of shape (1 , 128 , 256 , 512 , 1024 , 2048 , 1) .
Specifically to the GAN model, we also increase the number of layers in the discriminator in order to keep the dynamic between this latter and the generator balanced.
As in the 3 -layers case, the discriminator is one layer deeper than the generator.
For instance, the discriminator associated to the generator with 2 layers is an MLP of shape (1 , 256 , 128 , 1) .
Influence of generator architecture.
For this experiment, we use a feed-forward MLP of shape (1 , 256 , 256 , 256 , 1) as backbone.
Then we add two additive pre-activation skip-connections of type "resnet" between the first and the second hidden layers and between the second and the third hidden layers.
Finally, we replace the two previous additive skip-connections of type "resnet" by concatenation pre-activation skip-connections of type "densenet".
S5.2 Synthetic mixture of Gaussians on MNIST
Models details.
We adapt our three models to MNIST, changing mainly the networks architectures and making small modifications that we describe in what follows.
We base the architecture of the GAN and the VAE on DCGAN (Radford et al., 2015), using the generator as decoder and the discriminator as encoder for our VAE.
This is done by doubling the last layer of the discriminator in order that the VAE encoder has two outputs as in the univariate case.
For the GAN model, we replaced the convolutional discriminator by a simple MLP of shape (784 , 512 , 256 , 128 , 1) because the dynamic between the generator and the discriminator seemed unbalanced otherwise.
We also update our GAN model using some features of SAGAN (Zhang et al., 2019): applying spectral normalization on the discriminator and using the unconditional hinge version of the adversarial loss (Lim and Ye, 2017; Tran et al., 2017):
[Equation]
Such loss function is equivalent to minimize the Kullback-Leibler divergence between the generated distribution and the data distribution.
The VAE decoder and the GAN generator havelearnable parameters.
For the score network architecture, we use the vanilla U-Net architecture (Ronneberger et al., 2015) in which we double the number of channels at each layer, we add group normalization (Wu and He, 2018) after each convolution and we replace the ReLU non-linearies by
SiLU (Elfwing et al., 2018).
As in the univariate case, we use positional encoding (Vaswani et al., 2017) followed by a MLP block of shape (1 , 16 , 32) to incorporate the noise information at each layer.
The score network has 1607392 learnable parameters.
For inference, we use the same Langevin dynamic scheme as above with the same hyperparameters as in the univariate case.
The three models are trained during 100 epochs with a batch size of 128 using ADAM with a momentum of 0 .
9 and a learning rate of 2 ×
10 − 4 .
Additional details.
The histograms of projection on the line passing through the mean of each
Gaussians are obtained using 20000 generated samples.
To assign a color to each bin of the histograms, we train a simple MLP of shape (784 , 1024 , 50 , 10) as classifier on MNIST.
The classifier is trained during 10 epochs using again ADAM with a momentum of 0 .
9 and a learning rate of 2 × 10 − 4 and reaches an accuracy of 0 .
98 on the test set.
S5.3 Subset of MNIST
Models details.
Since the dataset is more complex than before, we use bigger models.
For the score network, we use the architecture defined in Ho et al.
(2020), in which we set the number of channels to 64 instead to
and we remove the self attention layers (Wang et al., 2018) for computational resource purposes.
The score network has 6072065 learnable parameters.
Again, we use an annealed
Langevin dynamic scheme for inference with the same hyperparameters as before.
For the VAE and the GAN, we use the same architecture as before, using this time the convolutional discriminator of
DCGAN, and quadrupling the number of channels at each layer.
This is mainly done in order to scale the generator/decoder to the score network.
Hence the VAE decoder/GAN generator has learnable parameters.
We train all three models during 600 epochs with a batch size of 128 using
ADAM with a momentum of 0 .
9 and a learning rate of 2 ×
10 − 4 .
Additional details.
We use the deep Wasserstein embedding proposed by Courty et al.
(2018) in order to visualize histograms of projection in the Wasserstein space.
We use the exact same network architecture and the same training procedure that in Courty et al.
(2018): first, one million pairs of digits of MNIST are chosen randomly, in which 700000 are kept for the training set, 200000 for the test set, and 100000 for the validation set.
We normalize each image in order to consider it as a two-dimensional distribution and we compute the 1 -Wasserstein distance for each pair.
Then, we train an autoencoder in a supervised manner in a way that the images at output of the autoencoder are close to the images in input, and that the euclidean distance between two vectors in the latent space is close to the 1 -Wassertein distance between the two corresponding images of MNIST.
As in Courty et al.
, the latent Wasserstein space is of dimension 50 and the autoencoder is trained during 100 epochs with a batch size of 100 and with an early stopping criterion.
Again, we use ADAM with a momentum of 0 .
9 and a learning rate of 10 − 3 .
We use the same classifier as before to assign color to each bin of the histograms.
Finally, the histograms of projection on the line passing through the deep Wasserstein barycenters of all 3 and 7 are obtained using 20000 generated samples.
S Additional experimental results
In the following, we provide additional experimental results.
First, we compare estimates of the bounds of Theorem 4, Corollary 6, and Theorem 7 to estimates of the total variation distance and the Kullback-Leibler divergence in the univariate case.
Then we study the possible correlation between the size of the score network and the tendency of the score-based model to generate unbalanced modes.
Finally, we provide additional visualizations of histograms of generated distributions for the univariate case and generated samples for the experiments on MNIST.
[Equation]
[Figure]
In this experiment, we compare estimates of the bounds of Theorem 4, Corollary 6, and Theoremto estimates of the total variation distance and the Kullback-Leibler divergence.
We only provide results for the VAE since the bounds are not interesting for the GAN since they are consequences of interpolation between modes due to a small Lipschitz constant of the generative network.
Yet this latter in the GAN case achieves a large Lipschitz constant so does not interpolate significantly.
To estimate empirically the total variation distance and the Kullback-Leibler divergence, we used their respective analytical formula
[Equation]
where p g # µ p and p ν are the respective densities of g # µ p and ν .
In order to estimate the lower bounds of Theorem 4 and Theorem 7, we set A of the form ( −∞ , − r/ 2] and we perform a grid search on r .
In Figure S3, we can observe that the estimates of the bounds provided by Theorem 4 and Theoremare not tights.
This is possibly because we selected a sub-optimal A
but it most likely follows from the fact that the bounds don’t take into account that g # µ p has automatically less mass on the modes than ν since a significant amount of its total mass is between them.
One can also observe that the explicit lower bound of Corollary 6 is much smaller than the bound of Theorem 4.
This can be explained by the facts that ∥ m ∥ / 2 σ is probably a sub-optimal choice of r and that the bound of
Corollary 6 minimizes the interpolation between modes over all the mappings with Lipschitz constant
Lip ( g ) , regardless whether these mappings approximate well ν on its modes or not.
Since there is less interpolation if the modes are unbalanced (see Section 3.2), it is likely that the mappings g such that g # µ p is unbalanced are affecting the value of this bound in a bad way.
S6.2 Additional examples S6.2.1 Univariate histograms
We provide additional visualizations of histograms of generated data with the three models for various values of m .
[Figure]
stylebiblographie stylebiblographie
We can observe that the score-based model already generates unbalanced modes, but the phenomenon is globally less visible than in higher dimensions.
Secondly, we provide additional visualizations of histograms of generated data with GANs trained with an additional gradient penalty term in the generator loss for various values of L ≈ Lip( g ) .
S6.2.2 Visualization of generated data
Finally, we show randomly chosen generated samples with VAE, GAN and SGM on the synthetic mixture of Gaussian on MNIST and the subset of all 3 and 7 of MNIST.
[Figure]