Can Push-forward Generative Models Fit Multimodal Distributions?
Abstract
Many generative models synthesize data by transforming a standard Gaussianrandom variable using a deterministic neural network.
Among these models are the
Variational Autoencoders and the Generative Adversarial Networks.
In this work,we call them "push-forward" models and study their expressivity.
We formallydemonstrate that the Lipschitz constant of these generative networks has to belarge in order to fit multimodal distributions.
More precisely, we show that thetotal variation distance and the Kullback-Leibler divergence between the generatedand the data distribution are bounded from below by a constant depending onthe mode separation and the Lipschitz constant.
Since constraining the Lipschitzconstants of neural networks is a common way to stabilize generative models, thereis a provable trade-off between the ability of push-forward models to approximatemultimodal distributions and the stability of their training.
We validate our findingson one-dimensional and image datasets and empirically show that diffusion modelsdo not suffer of such limitation.
Introduction
Generative modeling has become over the last years one of the most popular research topics incomputer vision.
From a mathematical perspective, the goal of generative modeling can be seenas predicting new synthetic samples from an unknown probability distribution ν on R d given theinformation of m true samples x i drawn from ν (the data distribution).
A general approach to solvethis problem is to define a parametric family of probability distributions ( ν θ ) θ ∈ Θ and solve theproblem
[Equation]
where d is a similarity measure between probability distributions ( d can either be a divergence ora distance) and δ x is the delta distribution in x .
Beside their direct application (Sandfort et al.
,2019; Antoniou et al., 2018), generative models have been used in numerous applications in variousmachine learning subfields, such as solving inverse problems (Ravuri et al., 2021; Ledig et al.,2017) or machine translation (Isola et al., 2017;
Yang et al., 2018).
However, most generativemodeling methods still lack theoretical understanding and it remains often unclear whether themethod approaches correctly the probability distribution ν or only generates samples that appear tohave been drawn from ν without fully recovering the underlying structure of the distribution.
In thiswork, we focus on the particular class of push-forward generative models .
Those models have incommon that for any θ ∈ Θ , the parametric distribution ν θ approaching ν is of the form
[Equation]
where µ p = N(0 , Id p ) is the Gaussian standard distribution in dimension p , # is the push-forwardoperator 1 , and
g θ : R p
→ R d is a deterministic neural network of parameter θ .
This class includestwo of the most popular generative models: the Variational Auto-Encoders (VAEs) 2 (Kingma and
Welling, 2014) and the Generative Adversarial Networks (GANs) (Goodfellow et al., 2014).
Deep neural networks are most of the time Lipschitz mappings by design, since their activationfunctions are generally Lipschitz.
In the literature, constraining the Lipschitz constant of a neuralnetwork is widely used as a way to increase its robustness (Virmaux and Scaman, 2018; Fazlyabet al., 2019), in particular to adversarial attacks (Goodfellow et al., 2015).
Common approaches tobound Lipschitz constants of neural networks are spectral normalization (Miyato et al., 2018), addinga gradient penalization in the loss (Gulrajani et al., 2017; Mohajerin Esfahani and Kuhn, 2018), or
Jacobian regularization (Pennington et al., 2017).
These approaches have been widely used to stabilizethe training of GANs, where Lipschitz constraints have been first imposed on discrimators (Arjovskyet al., 2017; Kodali et al., 2017; Fedus et al., 2018), while recent state-of-the-art architectures suchas BigGAN (Brock et al., 2018), SAGAN (Zhang et al., 2019) or StyleGAN2 (Karras et al., 2020)also impose similar constraints on the generators through spectral normalization (Brock et al., 2018;
Zhang et al., 2019), or Jacobian regularization (Karras et al., 2020).
In contrast to GANs, the recentstudy of Kumar and Poole (2020) shows that the decoder Jacobian in VAEs is implicitely regularized,which limits naturally its Lipschitz constant.
Recently, Dhariwal and Nichol (2021) trained an unconditional Score-based Generative Model (SGM)(Song and Ermon, 2019; Ho et al., 2020) on ImageNet (Russakovsky et al., 2015) and achievedstate-of-the-art generation.
To the best of our knowledge, there is no push-forward generative modelscapable of reaching this kind of performance on such a complex dataset without explicitly adding anyconditional label information in the model, see (Brock et al., 2018) for instance.
SGMs (also knownas diffusion models ) proceed as follows: first, noise is progressively added to the data distributionuntil we reach a standard Gaussian distribution.
Then this forward dynamics is reversed leveragingrecent advances in deep learning and tools from score-matching (Hyvärinen, 2005; Vincent, 2011).
We refer to Song et al.
for an introduction on SGMs.
In those models, the parametric generateddistribution is also of the form ν θ = g
θ # µ p , where g θ is the whole reverse diffusion dynamic (whichcan be seen as a composition of deterministic Lipschitz mappings) and p =
d ( N + 1) with N beingthe total number of steps in the dynamic.
However, those models are not push-forward generativemodels in the strict sense of the term, since the push-forward mapping is not a simple neural networkanymore.
An important difference is that optimization is not directly performed on the push-forwardmapping itself but on an auxiliary function (the score).
We therefore categorize them as indirectpush-forward generative models in this work.
Contributions of the paper.
In this paper, we study the expressivity of direct and indirect pushforward generative models in relation to the Lipschitz constant of the push-forward mapping they learn.
More precisely, in Section 3, for a Lipschitz function g and a given multimodal probability distributionν , we formally demonstrate that the Lipschitz constant of g must necessarily be large in order thatg # µ p approximate ν correctly, as it has been already intuitively observed in the literature (Lu et al. ,2020; Luise et al., 2020; Khayatkhoei et al., 2018).
As a direct consequence, we exhibit lower boundson d
( g # µ p , ν ) , where d is the total variation distance or the Kullback-Leibler divergence, with anexplicit dependence on the Lipschitz constant Lip( g ) of g , which highlights there is a fundamentaltrade-off for (direct) push-forward generative models between expressivity and stability of training.
In Section 4, we illustrate these theoretical results on several experiments, showing the difficultiesof GANs and VAEs to simulate multimodal distributions. We also offer comparison with SGMsand show experimentally these latter models seem to be able to generate multimodal distributionswhile keeping the Lipschitz constant of the score network relatively small, which suggests thatmodels consisting of stacked networks with noise input at each step do not suffer of such previouslymentioned limitation.
All the proofs are postponed to the appendix.
Related Works
Assessing the efficiency of push-forward models is a recurrent and important question in the literature.
The authors of Sajjadi et al.
(2018) and Kynkäänniemi et al.
(2019) propose Precison and Recallmetrics to assess GANs, aiming to measure simultaneously the mode collapse and the proportion ofoff-manifold generated samples.
Using similar metrics, the authors of Tanielian et al.
(2020) showthat the precision (the proportion of generated samples which could have been generated by the targetdistribution) of standard GANs is always upper-bounded.
To overcome this limitation, they simplypropose to reject samples associated with large values of the generator Jacobian.
The intuition behindthis idea is that those samples lie in regions of the space where the discontinuous optimal generatorwould "jump" between modes and so are off-manifold.
Another line of research focuses on the fact that the model has access to only the empirical distributionν n = 1 n (cid:80) i δ x i and not to the true target distribution.
For instance Nagarajan et al.
(2018) studyto what extend GANs only memorize the data.
Gulrajani et al.
(2018) highlight the fact thatcommon GAN benchmarks prefer training set memorization to a model which imperfectly fits thetrue distribution but covers more of its support.
Related to this, Stéphanovitch et al.
(2022) studyspecifically the Wasserstein GAN case, where the latent distribution is uniform and construct anoptimal generator which minimizes the Wasserstein distance of order 1 between the push-forwardmeasure and the empirical distribution, thus deriving a lower bound on the 1 -Wasserstein distance.
Inthe same paper, and more related to our work, the authors study the asymptotic case of an infinitenumber of data and show that most of the time the minimal 1 -Wasserstein distance between thepush-forward measure and the target distribution remains strictly positive.
Push-forward measure and Lipschitz mappings
In this section, we study the properties of the push-forward measure g # µ p when µ p = N(0 , Id p )is the standard Gaussian distribution in dimension p and g is a Lipschitz mapping.
First, for anyprobability measure γ on R d and any Borel set A of R d , we define the γ -surface area of A by
[Equation]
where A ε = { x ∈ R d : there exists a ∈ A , ∥ x − a ∥ ≤ ε } is the ε -extension of A and ∂ A is theboundary of A .
The γ - surface area can be interpreted as a measure (not in the strict mathematicalsense of the term) of the mass of γ on the hypersurface ∂ A .
Note that the support of γ and A can besets of intrinsic dimension smaller than d , which is most of the time the case when working with realdata which are likely to live on low dimensional manifolds (Pope et al., 2020).
The main theoreticalresult of this paper establishes some properties of push-forward measures depending on the regularityof the push-forward mapping.
Theorem 1.
Let g : R p
→ R d be a Lipschitz function with Lipschitz constant Lip( g ) .
Then for any
Borel set A ∈ B ( R d ) ,
[Equation]
(1) where φ ( x ) =
(2 π ) − 1 / 2 exp[ − x 2 / 2] and Φ( x ) = (cid:82) x
−∞ φ ( t )d t .
In addition, we have that for anyr
[Equation] [Equation]
Sketch of proof.
The proof of this result consists in establishing lower-bounds on ( g # µ p ) + ( ∂ A )and g # µ p ( A r ) which can be expressed as Gaussian integrals.
We conclude upon combining thisresult and the Gaussian isoperimetric inequality, see Sudakov and Tsirelson (1978)
µ + p ( ∂ A ) ≥φ (Φ − 1 ( µ p ( A ))) .
Note that (2) implies (1) upon remarking that (2) is an equality for r = 0 and letting r → 0 .
Theorem 1 recovers the Gaussian inequality in the case where g is the identity mapping and extends itto all Lipschitz mappings.
As the Gaussian inequality, Theorem 1 is dimension free, in the sense thatneither d , nor p , nor the intrinsic dimension of g ( R p ) play a role in the lower bounds.
In the followingsection, we are going to use Theorem 1 to (i) give a lower bound on the Lipschitz constant so thatpush-forward generative models exactly match the data distribution, (ii) give a lower bound on thetotal variation and the Kullback-Leibler divergence between the push-forward and data distributionswhich depends on the Lipschitz constant of the model.
3.1 Lower bounding the Lipschitz constant of push-forward mappings
Equation (1) implies that the Lipschitz constant of g must necessarily be large for g # µ p to bemultimodal.
It provides indeed a lower bound on the Lipschitz constant of the mappings g which pushµ p into a given measure ν .
In the extreme case where the support of ν is composed of disconnectedmanifolds, we retrieve that there doesn’t exist any Lipschitz mapping which pushes µ p into ν since itcan be found Borel sets A with null ν -surface area but such that the right-hand term of (1) is strictlypositive (which occurs when 0 < ν ( A ) < 1 ).
In the intermediate case where the support of ν isconnected but ν is multimodal, the less mass ν has between modes, the larger must be the Lipschitzconstant of the mappings which push µ p into ν .
Indeed, if ν has little mass between its modes, onecan find sets A with arbitrarily small ν -surface area and such that 0 <
ν
( A ) < 1 .
As a toy example,we get an explicit bound on the Lipschitz constant of the mappings which push µ p into a mixture oftwo isotropic Gaussians.
Corollary 2.
Let ν =
λ N( m 1 , σ 2 Id d ) + (1 − λ )N( m 2 , σ 2 Id d ) with m 1 , m 2 ∈ R d , σ > 0 andλ ∈ (0 , 1) .
Assume that there exists g : R p
→ R d Lipschitz such that g # µ p = ν .
Then
[Equation]
Sketch of proof.
The proof of this result consists in applying Inequality (1) of Theorem 1 on thehalf-space H such that ∂ H is the equidistant line of the means of the two Gaussians, and then toexplicit the value of ν + ( ∂ H ) .
Note that assuming there exists g : R p → R d such that g # µ p =
ν implies p ≥ d since ν covers thewhole ambient space and so g must be a surjective mapping.
This bound is maximal in the balancedcase when λ
= 1 / 2 since Φ − 1 ( λ )
=
in that case.
Otherwise, the more unbalanced the modes are,the smaller the bound is since the two terms in the exponential compensate each other more and more.
Extending this corollary to mixtures of more than two Gaussians with different covariance matrices istechnically difficult but we could expect a similar exponential growth in the square distance betweenmodes since it depends mainly on the order of magnitude of the local minima of the distributiondensity.
As a by product of Theorem 1, we also get the following result which shows that (in theone-dimensional case) the optimal transport map for the ℓ 2 cost minimizes the Lipschitz constant ofthe push-forward mapping.
Corollary 3.
Let ν be a probability measure on R with density w.r.t.
the Lesbesgue measure andsuch that supp ( ν ) =
R .
Assume that there exists g : R p
→ R Lipschitz such that ν =
g # µ p .
Let usdenote T OT =
Φ − 1 ν ◦ Φ the Monge map between µ 1 et ν , where Φ ν is the cumulative distributionfunction of ν .
Then we have Lip( g ) ≥ Lip( T OT ) .
To the best of our knowledge, extending this proposition to the case where d > 1 remains an openproblem.
We show now that Equation (2) of Theorem 1 allows to derive lower bounds on similaritymeasures between the push-forward measure and the target distribution.
3.2 Lower bounds on similarity measures between probability distributions
Equation (2) provides a bound on the minimal mass the push-forward measure g # µ p can have ona given set when g is fixed with Lipschitz constant Lip( g ) .
As a consequence, if ν is a distributionsuch that there exists sets on which ν has less mass than the minimal quantity that g # µ p can reachon those sets given the value of Lip( g ) , then g # µ p cannot be equal to ν , implying that most ofsimilarity measures between g # µ p and ν will be automatically strictly positive.
In the following, weconsider that g and ν are fixed and we derive lower bounds on the total variation distance and the
Kullback-Leibler divergence between g # µ p and ν .
We recall that the total variation distance betweentwo probability measures on R d , ν 0 , ν 1 is given by
[Equation]
Similarly, we define the Kullback-Leibler divergence between two probability measures on R d , ν 0 , ν 1 ,using the Donsker-Varadhan representation (Dupuis and Ellis, 2011, Lemma 1.4.3a):
[Equation]
where B ( R d , R ) denotes the set of all bounded mappings from R d to R .
In the following, we willdenote for any A ∈ B ( R d ) and r > 0 ,
[Equation]
where α g ( A , r ) and β g ( A , r ) are the lower bounds of g # µ p ( A r ) and g # µ p ( A r \ A ) provided by
Theorem 1.
We start by proving lower bounds on the total variation distance.
Theorem 4.
Let ν be a probability measure on R d and let g : R p
→ R d be a Lipschitz function.
Then,
[Equation]
Sketch of Proof.
The proof of this result consists in bounding from below the total variation distanceon one hand by | g # µ p
( A r \ A ) − ν
( A r \ A )
| and on the other hand by | g # µ p ( A r ) − ν ( A r ) | for agiven A ∈ B
R d ) and a given r > 0 , and then applying Theorem 1.
Observe that (3) always holds but the right-hand term may become negative if the Lipschitz constantof g is large enough.
The main idea behind this bound is to find a set A and a real r > 0 such thatν has a lot of mass on A but almost no mass on A r \ A .
For instance, if ν is a distribution on twodisconnected manifolds M 1 and M 2 , an optimal choice for A would either be M 1 or M 2 and theoptimal r would be the distance between the two manifolds.
Using Theorem 4, one can derive smallerbut more explicit bounds only depending on ν and the Lipschitz constant of g .
As a first example, wederive an explicit lower bound in the case where ν is a bi-modal distribution on two disconnectedmanifolds.
Corollary 5.
Let ν be a measure on R d on two disconnected manifolds M 1 and M 2 such thatν ( M 1 )
=
λ
and ν ( M 2 ) =
1 − λ , with λ ∈ [1 / 2 , 1) , and let g : R p
→ R d be a Lipschitz function.
Then,
[Equation]
where d ( M 1 , M 2 ) =
inf {∥ m 1 − m 2 ∥ : m 1 ∈ M 1 , m 2 ∈ M 2 } .
As a second example, we also get an explicit lower bound in the case where ν is a mixture of twoisotropic Gaussians.
For simplicity we stick to the balanced case.
Corollary 6.
Let ν =
(1 / 2)[N( m 1 , σ 2 Id d ) + N( m 2 , σ 2 Id d )] with m 1 , m 2 ∈ R d and σ ≥ 0 .
Letg : R p
→
R d be a Lipschitz function.
Then,
[Equation]
In both corollaries, the lower bound tends to 1 / 2 when the distance between the modes tends toinfinity, meaning that g # µ p is far from well approaching ν .
Note that the lower bound exhibited in
Corollary 5 is always strictly positive regardless of the value of the Lipschitz constant of g .
Onecan also observe that this latter bound is maximal in the balanced case, when λ = 1 / 2 , since thestandard normal distribution concentrates its mass around 0 .
Finally, we end this section by derivinga similar lower bound on the Kullback-Leibler divergence between g # µ p and ν .
We consider the
Kullback-Leibler divergence since this is a measure of similarity between measures which is boundedand is very sensitive to the mismatch of supports between the generated and the data distributions.
Theorem 7.
Let ν be a probability measure on R d and let g : R p
→ R d be a Lipschitz function.
Then,
[Equation]
Sketch of Proof.
The proof of this result consists in setting f = ζχ A r \
A with ζ > 0 and where χ Ais the characteristic function of the set
A in order to get a lower bound of the Kullback-Leiblerdivergence depending on the probability g # µ p ( A r \ A ) for a given A , a given r and a given ζ .
Thenwe apply Theorem 1 and we derive the optimal value of ζ .
As above, this bound always holds but the right-hand term becomes negative if Lip( g ) is large enough.
As for Theorem 4, the main idea is to find a set A and a real r such that ν has a lot of mass on A , butν has almost no mass on A r \ A .
Observe that if ν ( A r \ A ) tends to 0 , the left-hand term of the boundtends to infinity.
This is coherent with the behavior of the Kullback-Leibler divergence.
To concludethis section, we highlight the fact that, if our results are dimension free in theory, the dimension islikely to be hidden in the distances between modes and the Lipschitz constant of g when workingwith real datasets.
Experiments
In what follows, we illustrate the pratical implications of our results by training GANs, VAEs and
SGMs on simple bi-modal distribution.
More precisely, we show on one hand that generatingmultimodal distributions with GANs and VAEs is difficult since for those models, good generationnecessarily involves generative networks with large Lipschitz constants.
On the other hand, we showthat SGMs seem to be able to generate multimodal distributions while keeping the Lipschitz constantof the score network relatively small and thus do not suffer of the same limitation.
First, we focuson the univariate case where we can easily assess the Lipschitz constants of the networks.
Thenwe illustrate our results in higher dimensions by training the three models on datasets derived from MNIST (LeCun et al., 1998).
In all our experiments, we use the same architecture for the VAEdecoder and the GAN generator in order to offer rigorous comparisons of the different models.
Forscore-based modeling, we use architectures with similar number of learnable parameters.
All detailson the experiments and architecture of the networks can be found in Appendix S4.
4.1 Univariate case
First, we train a VAE and a GAN with one-dimensional latent spaces on 50000 independent samplesdrawn from a balanced mixture of two univariate Gaussians ν = (1 / 2)[N( − m, 1) + N( m, 1)] fordifferent values of m > 0 .
We also train a SGM on the same samples.
Histograms of generated distributions.
Figure 1 shows histograms of generated distributionsfor m = 10 with the three different models.
VAE models seem to generate Gaussians modesbut interpolate significantly between them, while GANs do not interpolate but fail to retrieve thestructure of the target distribution and forgets parts of its support, which is known as mode collapseand is a common pitfall of such models (Arjovsky and Bottou, 2017;
Metz et al., 2017).
On thesame task, SGMs do not suffer from such shortcomings.
In the following section, we will link theinterpolation/mode-collapsing properties of these models with their Lipschitz constants.
Lipschitz constant and mass between modes.
In Figure 2 (right), we observe that the GANgenerator reaches much larger Lipschitz constants than the VAE decoder.
This explains the differenceof behaviors between GAN and VAE observed in Figure 1, as the learned mapping is not stiffenough to concentrate the push forward measure on the two modes in the case of VAE.
One possibleexplanation for the interpolating behavior of the VAE is to note that the Euclidean norm of the
Jacobian of the VAE decoder is implicitely regularized during training, as it has been demonstratedin Kumar and Poole (2020).
Both GAN and VAE saturate the constraint on g # µ p ([ − m/ 2 , m/ 2])provided by Theorem 1, meaning that the generative networks minimize the amount of mass betweenmodes as much as their Lipschitz constants allow it.
Finally, we can observe that the score network isable to keep a relatively small Lipschitz constant compared to the GAN, while managing to interpolate
[Figure]
less than the latter.
A probable explanation for this follows from the fact that the score network isused multiple time during inference.
Hence, the Lipschitz constant of the push-forward mapping(the whole generation dynamic) is likely to be much larger than the Lipschitz constant of the neuralnetwork itself, and so the model is able to push-forward a Gaussian distribution into multimodaldistributions keeping a relatively small Lipschitz constant of the score network.
Finally, in Figure 2(left), we observe that when m increases, the Lipschitz constant of the VAE decoder and the GANgenerator becomes rapidly much smaller than the value of the lower bound provided by Corollary 2.
This means that for m large enough it is not possible to close the gap between the data distributionand the push-forward distribution.
We highlight that our results do not cover SGMs since in this setting the network is applied multiple times.
[Equation]
Stability of GAN and mode collapse.
Odena et al.
(2018) suggested that the magnitude of thenorm of the generator jacobian may be causally related to instability and mode collapse.
This is whymany state-of-the-art GANs apply spectral normalization (Miyato et al., 2018) on their generators.
In Figure 3 (left), we show that this technique cannot be used when training GANs on multimodaldistributions: since spectral normalization constraints the Lipschitz constant of the generator to besmaller than 1 , the GAN is trained towards concentrating in one of the modes of the distribution overinterpolating massively between them.
This has been referred to as mode dropping by Khayatkhoeiet al.
To complete this analysis, we also train the GAN adding an additional gradient penaltyterm 10 /L 2 max z ∼ N(0 , Id p ) ( ∥∇ z g θ ( z ) ∥
22 − L ) 2 , in the generator loss function, similarly to WP-GAN(Gulrajani et al., 2017), where L is an hyperparameter corresponding to the targeted Lipschitzconstant.
As expected, we can observe in Figure 3 (right), that when Lip ( g ) increases, the GAN beginto generate both modes but becomes also more and more prone to mode collapse.
This illustrates thefundamental trade-off between expressivity and robustness in push-forward generative models.
Influences of model size and time of training.
Finally, we study the effect of the architecture andthe training time on the value of the Lipschitz constant of the VAE decoder and the GAN generator in
Figure 4.
In the VAE setting, the Lipschitz constant increases linearly with the depth of the decoder.
This is not the case in the GAN setting, where increasing the size of the model seems to dramaticallyaffect its stability.
For both models, the Lipschitz constants of the generative network grow with the number of epochs.
Yet this growth seems to be logarithmic for the VAE and the GAN seems tobecomes more unstable as the number of epochs increases. [Figure] i
[Figure]
4.2 Experiments on MNIST
We train a VAE, a GAN and a SGM on two datasets derived from MNIST (LeCun et al., 1998):first, two images of two different digits ( 3 and 7 ) are chosen and 10000 noisy versions of thesesimages are drawn with a noise amount of σ = 0 .
15 , forming a dataset of n =
independentsamples drawn from a balanced mixture of two Gaussian distributions in dimension 784 = 28 × 28 .
Second, we train the models on the subset of all 3 and 7 of MNIST.
We emphasize that our goal isnot reach state-of-the-art performance on this problem but rather to illustrate our theoretical results ina moderate dimensional setting.
Mixture of Gaussians.
For this experiment, we set the dimension of the latent space in the GANand the VAE to 784 = 28 × 28 since it is the intrinsic dimension of the support of the data distribution.
In order to visualize the interpolation between modes, we project the data on the line passing throughthe mean of each Gaussian, i.e. the two original clean images, and we plot histograms of the onedimensional projections.
In order to understand which bins of data in the histograms correspond towhich digit, we train a classifier and we assign a color in function of which digit the data have beenclassified as.
Results can be found in Figure 5 top.
Moreover, they both fail to generate noisy versionsof the images.
As in the univariate case, the SGM is able to not interpolate between modes and seem to retrieve the Gaussian structure of the modes.
This suggests that while direct push-forward modelsfail at representing multimodal distributions, considering stacked models with noise input at each step(as in SGM) might help to close the gap between the generated and the data distributions.
However SGM does not manage to retrieve the right modes proportions.
This is a well-known shortcoming ofscore-based models which has been studied in (Wenliang and Kanagawa, 2020).
We observed thatincreasing the size of the score network seems to be a way to reduce this issue (see Appendix S5).
Subset of MNIST.
Finally we train the three different models on the subset of MNIST composedof all 3 and 7 (no Gaussian noise was added).
We choose a latent dimension of 20 for the VAE andthe GAN.
Since the Euclidean distance is not a meaningful metric to compare the different digits of
MNIST, we use the deep Wasserstein embedding proposed by Courty et al.
(2018): an autoencoder islearned in a supervised fashion such that the Euclidean distance in the latent space approximates the
Wasserstein distance between pairs of images of MNIST.
In the learned Wasserstein space, we projectdata on the line passing through the Euclidean barycenters of all 3 and 7 and plot histograms ofprojections, using the same classifier as before.
Results can be found in Figure 5 (bottom).
Note thatthe distribution does not exhibit strong multimodality features contrary to the mixture of Gaussianssettings, see Figure 5.
As before, the VAE interpolate between modes, the GAN manages to notinterpolate but generate a narrower histogram, and the score-based model does not interpolate andseems to recover the structure of the distribution, but doesn’t retrieve the right modes proportions.
However, we emphasize that all these models seem to perform better than on the previous dataset.
Apossible explanation of this is that the modes are less separated than in the Gaussian mixtures andtherefore the model is easier to train.
Discussion
In this work, given a Lipschitz mapping g and a measure ν , we derived lower bounds on the totalvariation distance and the Kullback-Leibler divergence between the push-forward measure g # µ pand ν depending on the Lipschitz constant of the mapping g .
These bounds indicate how themass between the modes of the push-forward measure depends on the Lipschitz constant of thepush-forward mapping.
They highlight the trade-off between the ability of VAEs and GANs to fitmultimodal distributions and the stability of their training.
A common assumption in the imaging literature, validated empirically by Pope et al.
(2020), is thatdistributions of natural images live on low dimensional manifolds.
Understanding whether thesedistributions are composed of separated modes or not remains, to the best of our knowledge, an openproblem.
To that extent, the fact that unsupervised push-forward generative models perform wellon datasets such as CelebA (Liu et al., 2015) could possibly be, in regard of our work, an indicatorthat the data distributions of those datasets are unimodal, or at least not composed of well separatedmodes.
Several techniques have been proposed in the literature to fit data distributions on disconnectedmanifolds.
Most of them consist in overparametrizing the model, either by using stacked generativenetworks (Khayatkhoei et al., 2018; Mehr et al., 2019) or by learning a more complex latent distribution than the standard Gaussian (Gurumurthy et al., 2017; Rezende and Mohamed, 2015; Kingmaet al., 2016; Luise et al., 2020).
Other methods consist in rejecting a posteriori samples associated tolarge values of the Jacobian generator (Tanielian et al., 2020; Issenhuth et al., 2020).
In this work, weempirically showed that score-based models seemed to be able to fit separated manifolds withoutmodel overparametrization or additional posterior sample rejection scheme.
This suggests that thestructure of the generation dynamic in these models is particularly adapted to (indirectly) learnmappings with large Lipschitz constants. In particular, the fact that these models do not optimizedirectly the push-forward mapping itself might be a possible explanation of their good performanceson such datasets.
Hence, a future perspective of work would be to study if certain structures of thepush-forward map are better than others to learn mappings with large Lipschitz constants.
A possible limitation of this work is that the bounds derived on the Kullback-Leibler divergence andtotal variation distance are not tight (see Appendix S5), mainly because they take no account of thefact that when interpolating, g # µ p has automatically less mass than ν on the modes since a significantamount of its total mass is between them.
In future work, we plan to tighten the gap between ourbounds and the true distance.
S Organization of the supplementary
The supplementary is organized as follows.
In Appendix S2, we give the full proofs of all thetheoretical results of the paper.
In Appendix S3, we show two additional theoretical results: a corollaryof Theorem 7 in the case where ν is a mixture of two isotropic Gaussians and a generalization of
Corollary 5 in the case there is more than two disconnected manifolds.
In Appendix S4, we givedetails on the experiments.
Finally, in Appendix S5, we provide additional experimental resultsand additional visualizations of histograms of generated distributions for the univariate case, andgenerated data for the experiments on MNIST.
S Proofs of the theoretical resultsS2.1 Proof of Theorem 1
We start by recalling the Gaussian isoperimetric inequality Sudakov and Tsirelson (1978).
Lemma S8.
Let A ∈ B ( R p ) and µ p
= N(0 , Id p ) .
Then we have
[Equation]
where φ ( x ) = (2 π ) − 1 / 2 exp[ − x 2 / 2] and Φ( x ) = (cid:82) x
−∞ φ ( t )d t .
Equivalently, for all r ≥ 0µ p ( A r ) ≥ Φ( r + Φ − 1 ( µ p ( A ))) .
In particular, using Lemma S8, one can show that among all sets of given Gaussian measure µ p ,half-spaces have the minimal µ p -surface area.
We are now ready to turn to the proof of Theorem 1.
Proof of Theorem 1.
Let A ∈ B ( R d ) such that g # µ p ( A ) > 0 (note that if g # µ p ( A ) =
thenthe result is trivial).
First, we show that for any ε > 0 , g (( g − 1 ( A ))
ε/ Lip( g ) )
⊂
A ε .
Let x be ing (( g − 1 ( A ))
ε/ Lip( g ) ) .
There exists z 1 ∈ ( g − 1 ( A ))
ε/ Lip( g ) such that g ( z 1 ) = x .
There also existsz 2 ∈ g − 1 ( A ) such that
[Equation]
Hence, we have that
[Equation]
where a = g ( z 2 ) .
Since z 2 ∈ g − 1 ( A ) , a ∈ A , and therefore x ∈ A ε .
Using this result, the fact thatg # µ p ( B ) = µ p ( g − 1 ( B )) and B ⊂ g − 1 ( g ( B )) for any B ∈ B ( R d ) , we have
[Equation]
S
[Equation]
Using Lemma S8, we have
[Equation]
Combining this result and (S1), we get that
[Equation]
In addition, using Lemma S8, we have for all r ≥
[Equation]
A )))) . Using this result and that g (( g − 1 ( A )) r/ Lip( g ) )
⊂ A r ,
S2.2 Proof of Corollary
We prove the corollary when ν = λ N( − m, σ 2 Id d ) + (1 − λ )N( m, σ 2 Id d ) since the problem canalways be reduced to that case by translation and setting m = ( m 2 − m 1 ) / 2 .
Let H be defined by
H = { x ∈ R d | m T x ≥ 0 } .
Note that for any x ∈ ∂ H , ∥ x − m ∥ =
∥ x + m ∥ .
Since the problem isinvariant by rotation, we can consider without any loss of generality that m = ( ∥ m ∥ , 0 , . .
Inthat case, we have ν =
ν 1 ⊗ N(0 , σ 2 Id d − 1 ) , where ν
1 = λ N( −∥ m ∥ , σ 2 ) + (1 − λ )N( ∥ m ∥ , σ 2 ) ,and ⊗ is the tensor product between measures.
In this case, we have that H
= { x 1 ≥ 0 } × R d − 1 .
Therefore, we have
[Equation]
where p ν and p ν 1 are the respective densities of ν and ν 1 , and h is the density of N(0 , σ 2 I d − 1 ) .
Itfollows that
[Equation]
Applying Theorem 1, we get that
[Equation]
Furthermore, one can derive that
[Equation]
Observing that λ − ν
( H ) is an increasing function of λ and λ − ν ( H )
=
if λ = 1 / 2 , we get thatλ ≤ ν ( H ) if λ ≤ 1 / 2 and λ ≥ ν ( H ) if λ ≥ 1 / 2 .
Since φ ◦ Φ − 1 reaches its maximum in 1 / 2 , itfollows that for any λ ∈ (0 , 1) we have
[Equation]
and thus
[Equation]
which concludes the proof.
S2.3 Proof of Corollary
Since ν admits a density p ν with respect to the Lesbegue measure, it follows that Φ ν is absolutelycontinuous and therefore differentiable almost everywhere w.r.t.
the Lebesgue measure using the
Lebesgue differentiation theorem.
Moreover, since supp( ν ) =
R , it follows that Φ ν : R → (0 , 1) isincreasing and therefore is bijective, and
so T OT =
Φ − 1 ν ◦ Φ is also differentiable almost everywherew.r.t.
the Lebesgue measure and bijective, with inverse T − 1OT =
Φ − 1 ◦ Φ ν , using (Peyré and Cuturi,2019, Remark 2.29).
Therefore, for any x ∈ R we have
[Equation]
Let y ∈ R .
Using Theorem 1 with A =
( −∞ , y ] we get that for any g : R p
→ R , Lipschitz suchthat g # µ p
=
ν ,
[Equation]
and so, since T OT is bijective
[Equation]
which concludes the proof.
S
Proof
Let A ∈ B ( R d ) and let r > 0 .
We have on one hand| g # µ p ( A r \ A ) | ≤ | g
# µ p ( A r \ A ) − ν
( A r \ A ) | + | ν ( A r \ A ) |
≤ d TV ( g # µ p , ν )
+ ν ( A r \ A ) .
Using Theorem 1, we get| g # µ p ( A r \ A ) | =
g # µ p ( A r ) − g # µ p ( A ) ≥ Φ (cid:0) r/ Lip( g ) + Φ − 1 ( g # µ p ( A )) (cid:1) − g # µ p ( A ) , and so
[Equation]
[Equation]
where α g ( A , r ) =
Φ (cid:0) r/ Lip( g ) + Φ − 1 ( g # µ p ( A )) (cid:1) .
On the other hand, we have| g # µ p ( A r ) | ≤ | g # µ p ( A r ) − ν
( A r ) | + | ν ( A r ) |
≤ d TV ( g # µ p , ν )
+ ν ( A r \ A )
+ ν ( A ) .
Using Theorem 1, we get| g # µ p ( A r ) | ≥ Φ (cid:0) r/ Lip( g )
+ Φ − 1 ( g # µ p ( A )) (cid:1) , and sod
[Equation] , which concludes the proof.
S2.5 Proof of Corollary
To prove Corollary 5, we will need the following lemma:
Lemma S9.
Let A ∈ B ( R d ) and r > 0 .
We denote B =
( A r ) c .
Then
B r ⊂ ¯ A c , where ¯ A c denotes the closure of the complementary of A .
Proof.
Let x ∈ B r .
There exists b ∈ B such that ∥ x − b ∥ ≤ r .
Moreover, since B =
( A r ) c , it followsthat for all a ∈ A ,∥ b − a ∥ > r .
Thenr < ∥ b − x
∥ + ∥ x − a ∥ , and so, it follows that for all a ∈ A ,∥ x − a ∥ > 0 .
Thus x ∈ ¯ A c .
Now we are ready to turn to the proof of Corollary 5.
Proof of Corollary 5.
We set r = d ( M 1 , M 2 ) / 2 and A
=
( M 1 ) r .
Using Theorem 4, we have
[Equation]
First we suppose that g # µ p ( A ) ≥ ν ( A ) : since Φ is a non-decreasing function, it follows thatα g ( A , r ) =
Φ (cid:0) r/ Lip( g ) + Φ − 1 ( g # µ p ( A )) (cid:1) ≥ Φ (cid:0) r/ Lip( g ) + Φ − 1 ( ν ( A )) (cid:1) .
Moreover min { g # µ p
A ) , ν ( A ) } =
ν ( A ) =
λ = Φ(Φ − 1 ( λ )) and so it followsd TV ( g # µ p , ν ) ≥ Φ (cid:0) d
( M 1 , M 2 ) / (2Lip( g ))
+ Φ − 1 ( λ ) (cid:1) − Φ(Φ − 1 ( λ )) ≥ (cid:82) r/ Lip( g )
+
Φ − 1 ( λ ) Φ − 1 ( λ ) φ ( t ) dt , since ν has no mass on A r \ A .
Now we suppose that g # µ p ( A ) ≤ ν ( A ) : we then set B =
A c .
Sinceg # µ p ( A ) ≤ ν ( A ) , we have g # µ p ( B ) ≥ ν ( B ) .
Applying Theorem 4, and the same reasoning asbefore we get
[Equation] [Equation]
Using Lemma S9, we get that ν
( B r \ B ) ≤ ν
( ¯ A c \ ( A r ) c ) but ν ( ¯ A c \ ( A r ) c ) = 0 since ν has no masson ¯ A c \ ( A r ) c except on its boundary and so its follows that
[Equation]
since Φ − 1 (1 − λ )
= − Φ − 1 ( λ ) .
Since λ ≥ 1 / 2 , it follows that Φ − 1 ( λ ) ≥ 0 and so
[Equation]
which concludes the proof.
S2.6 Proof of Corollary
As previously, we prove the corollary when ν = (1 / 2)[N( − m, σ 2 Id d ) + N( m, σ 2 Id d )] sincethe problem can always be reduced to that case by translation and setting m =
( m 2 − m 1 ) / 2 .
Since the problem is invariant by rotation, we can assume without any loss of generality thatm =
( ∥ m ∥ , 0 , . .
Let H be the half-space of R d defined by H =
( −∞ , 0]
× R d − 1 and we setr = ∥ m ∥ / 2 σ .
First we suppose that g # µ p ( H ) ≥ ν ( H ) : using Theorem 4, we get that
[Equation] [Equation]
H ) , ν
[Equation]
H ) } − ν ( H r \ H ) , with H r = ( −∞ , ∥ m ∥ / 2 σ ] × R d − 1 .
On one hand we have that ν
= ν 1 ⊗ N(0 , σ 2 Id d − 1 ) , whereν 1 = (1 / 2)[N( −∥ m ∥ , σ 2 ) + N( ∥ m ∥ , σ 2 )] and so ν ( H r \ H )
=
ν 1 ([0 , ∥ m ∥ / 2 σ ]) .
On the other handwe have that min { g # µ p
( H ) , ν ( H ) } =
ν ( H ) and g # µ p ( H ) ≥ 1 / 2 since g # µ p ( H ) ≥ ν
( H ) .
Hence itfollows that
[Equation]
Now we suppose that g # µ p ( H ) ≤ ν ( H ) : we then set H
= (0 , + ∞ ]
× R d − 1 .
Since g # µ p [Equation] H ) ≤ 1 / 2 ,we get that g # µ p ( H 2 ) ≥ 1 / 2 and so g
# µ p ( H 2 ) ≥ ν
( H 2 ) .
Hence we retrieve the previous case andso it follows that
[Equation]
Since ν 1 ([ −∥ m ∥ / 2 σ, 0]) =
ν 1 ([0 , ∥ m ∥ / 2 σ ]) , we get in both cases
[Equation]
( r/ Lip( g )) − 1 / 2 − ν 1 ([0 , ∥ m ∥ / 2 σ ]) .
Now we derive the value of ν 1 ([0 , ∥ m ∥ / 2 σ ]) :
[Equation]
which concludes the proof
[Equation]
S2.7 Proof of Theorem
Let A ∈ B ( R d ) , r > 0 and ζ > 0 .
We set for any x ∈ R d f ( x ) =
ζχ A r \ A ( x ) , where χ
A denotes thecharacteristic function of the set A .
Since f is bounded, it follows that
[Equation]
Using Theorem
[Equation]
we get
[Equation]
A ) .
Thus we get
[Equation]
where the functional J is defined by
[Equation]
Differentiating J with respect to ζ , we get that
[Equation]
where β g ( A , r ) =
Φ (cid:0) r/ Lip( g ) + Φ − 1 ( g # µ p ( A )) (cid:1) − g
# µ p
( A ) .
Applying the first order condition,we get that:
[Equation]
By re-injecting the value of ζ ∗ , we get
[Equation]
which concludes the proof.
S Additional theoretical results
In this section we show two additional theoretical results.
First we start by a corollary of Theorem 7which exhibits an explicit lower bound of the Kullback-Leibler divergence in the case where ν is amixture of two isotropic Gaussians.
Second, we demonstrate a generalization of Corollary 5 when νis a distribution whose support is composed of more than two disconnected manifolds.
S3.1 A corollary of Theorem
We start by demonstrate the following corollary of Theorem 7, which exhibits an explicit lower boundof the Kullback-Leibler divergence in the case where ν is a mixture of two isotropic Gaussians.
Asfor Corollary 6, we stick to the balanced case for simplicity.
Corollary S10.
Let ν =
(1 / 2) (cid:2) N( m 1 , σ 2 Id d ) + N( m 2 , σ 2 Id d ) (cid:3) with m 1 , m 2 ∈ R d and σ ≥ 0 .
Let g : R p
→
R d be a Lipschitz function.
We denote λ =
g # µ p (cid:0) { ( m 2 − m 1 ) T x ≤ 0 : x ∈ R d } (cid:1)and we suppose without loss of generality, that λ ∈ (0 , 1 / 2] .
Then,
[Equation]
, where
[Equation]
with
[Equation]
Proof.
As previously, we prove the corollary when ν
=
(1 / 2)[N( − m, σ 2 Id d ) + N( m, σ 2 Id d )]since the problem can always be reduced to that case by translation and setting m = ( m 2 − m 1 ) / 2 .
Since the problem is invariant by rotation, we can assume without any loss of generality thatm =
( ∥ m ∥ , 0 , . .
Furthermore, observe that the half-space { ( m 2 − m 1 ) T x ≤ 0 : x ∈ R d }becomes ( −∞ , 0] × R d − 1 in that case, and that the condition λ ∈ (0 , 1 / 2] is indeed non-restrictivesince the problem is invariant by rotation.
We set as before H =
( −∞ , 0]
× R d − 1 and r = ∥ m ∥ / 2 σ .
Applying Theorem 7, we get
[Equation]
On one hand, we get
[Equation]
t
[Equation]
noting λ
= g
# µ p ( H ) .
We replaced Φ − 1 ( λ ) by − Φ − 1 (1 − λ ) in order to emphasize that Φ − 1 ( λ ) ≤ 0since λ ≤ 1 / 2 .
Observe that if we supposed λ ≥ 1 / 2 , we would have β g ( H c , r ) ≥ β g ( H , r ) and so thebound that we would have found by reasoning on H would have been sub-optimal.
On the other hand,observing as before that ν
= ν 1 ⊗ N(0 , σ 2 Id d − 1 ) , where ν 1 = (1 / 2)[N( −∥ m ∥ , σ 2 ) + N( ∥ m ∥ , σ 2 )] ,we get that
[Equation]
which concludes the proof.
S3.2 A generalization of Corollary
Second, we demonstrate the following corollary, which is a generalization of Corollary 5 when ν is adistribution on more than two disconnected manifolds.
Corollary S11.
Let ν be a measure on R d on N disconnected manifolds ( M 1 , . . .
, M N ) , and letg :
R p
→ R d be a Lipschitz function.
Then,
[Equation]
dt
[Equation]
where for A , B ∈ B ( R d ) , d ( A , B ) = inf {∥ a − b ∥ : a ∈ A , b ∈ B } , and λ =
ν (cid:18) (cid:70) i ∈ I M i (cid:19) ifν (cid:18) (cid:70)
i ∈
I M i (cid:19) ≥ 1 / 2 and λ = 1 − ν (cid:18) (cid:70) i ∈
I M i (cid:19)
otherwise.
Proof.
Let I ⊂ (cid:74) 1 , N (cid:75) .
First, we suppose that ν (cid:18) (cid:70) i ∈ I M i (cid:19) ≥ 1 / 2 .
Since ν can be seen as a bi-modaldistribution on the two disconnected sets (cid:70) i ∈ I M i and (cid:70) j ∈ (cid:74) 1 ,N (cid:75) \
I
M j , we can apply Corollary 5.
Thuswe get
[Equation]
d (
[Table]
i ,
[Table]
Φ
[Table]
λ ) φ ( t ) dt . If ν (cid:18) (cid:70) i ∈ I M i (cid:19) ≤ 1 / 2 , we can still apply Corollary 5 by interchanging the roles of (cid:70) i ∈
I M i and(cid:70) j ∈ (cid:74) 1 ,N (cid:75) \
I
M j , thus we get also Inequality (S3.2) in that case, which concludes the proof.
S Experimental details
We detail our experiments in dimension 1 in Appendix S4.1.
In Appendix S4.2, we give details onour experiment on the synthetic mixture of two Gaussians derived from MNIST.
Finally, we detail theexperiment on the subset of all 3 and 7 of MNIST in Appendix S4.3.
We trained our models using
NVIDIA Titan Xp from the proprietary server of our institution with an estimated total training timeof approximately 175 GPU hours.
S4.1 Univariate case
In the univariate case we use a simple 3 -layer Multi Layer Perceptron (MLP) of shape (1 , 128 , 256 , 1)as decoder for the VAE and as generator for the GAN.
The network has a total of 33537 learnableparameters.
The score network uses also a a 3 -layer MLP block, this time of shape (1 , 96 , 196 , 1) , inwhich at each layer is injected the noise information transformed by a positional encoding (Vaswaniet al., 2017) and then by another MLP block size (16 , 32 , 64) , see Figure S1.
The score network hasa total of 34665 learnable parameters.
In all three models, we use LeakyReLU
(Maas et al., 2013) asnon-linearity with a negative slope of 0 .
The three models are trained during 400 epochs with abatch size of 1000 using ADAM (Kingma and Ba, 2014) with a momentum of 0 .
9 and a learning rateof 10 − 4 .
In the following, we give more specific details for each model.
Variational autoencoder.
We use the vanilla VAE model as described in Kingma and Welling(2014).
In the following, we denote θ and ϕ the respective parameters of the decoder and the encoder.
The decoder f ϕ is composed of an MLP block of size (1 , 256 , 128) followed by two parallel fullyconnected layers of shape (128 , 1) which gives two outputs f 1 ϕ ( x ) and f 2 ϕ ( x ) .
Then the input zof the decoder g θ is obtained by the so-called reparametrization trick, which consists in samplingz ∼ q z | x ϕ , where q z | x ϕ = N( f 1 ϕ ( x ) , exp[ f 2 ϕ ( x )]) .
During training, the model minimizes the followingloss function:
[Equation]
where p x | z θ = N( g θ ( z ) , c Id d ) and ELBO is the Evidence Lower Bound (Blei et al., 2017), definedas follows:
[Equation]
The variance c in p x | z θ is an hyperparameter of the model.
For our experiments, we observed thatc = 0 .
1 gave good results.
Generative adversarial network.
As for the VAE, we use the vanilla GAN model as describedin Goodfellow et al.
The discriminator is 4 -layer MLP of shape (1 , 512 , 256 , 128 , 1) withspectral normalization (Miyato et al., 2018) in order to reduce as much as possible mode collapse.
We train the model using the vanilla adversarial loss, that the discriminator d ϕ tries to maximize andthat the generator g θ tries to minimize
[Equation]
We also tried with the hinge version of the adversarial loss, as proposed in Lim and Ye (2017) and
Tran et al.
and we obtained similar results.
Score-based generative modeling.
Our diffusion model is similar to the model introduced by
Song and Ermon (2019).
The neural network s θ learns to approximate, for a given x and a given σ ,the score
∇ x p ν ( x, σ ) of the data distribution convoluted with a Gaussian distribution of standarddeviation σ .
This is done by first defining a geometrical progression { σ i } Li =0 where L = 10 andwhere the ratio is chosen such that σ L ≈ 0 . 01 , and then minimizing the Fischer divergence (Vincent,2011)
[Equation]
Then, in order to generate data, we use an annealed Langevin dynamic scheme as defined in Songand Ermon (2019).
In the Langevin dynamic, we set the step size to 2 × 10 − 5 and the number of stepfor each value of σ to 100 as in Song and Ermon (2019).
Influences of model size.
For this experiment, we increase the number of layers of the VAE decoderand the GAN generator from 2 to 6 .
At each new layer, we double the number of neurons at theprevious layer.
For instance, the generative network with 2 layers is thus an MLP of shape (1 , 128 , 1)and the one with 6 layers is an MLP of shape (1 , 128 , 256 , 512 , 1024 , 2048 , 1) .
Specifically to the
GAN model, we also increase the number of layers in the discriminator in order to keep the dynamicbetween this latter and the generator balanced.
As in the 3 -layers case, the discriminator is one layerdeeper than the generator.
For instance, the discriminator associated to the generator with 2 layers isan MLP of shape (1 , 256 , 128 , 1) .
S4.2 Synthetic mixture of Gaussians on MNIST
Models details.
We adapt our three models to MNIST, changing mainly the networks architecturesand making small modifications that we describe in what follows.
We base the architecture ofthe GAN and the VAE on DCGAN (Radford et al., 2015), using the generator as decoder and thediscriminator as encoder for our VAE.
This is done by doubling the last layer of the discriminatorin order that the VAE encoder has two outputs as in the univariate case.
For the GAN model, wereplaced the convolutional discriminator by a simple MLP of shape (784 , 512 , 256 , 128 , 1) becausethe dynamic between the generator and the discriminator seemed unbalanced otherwise.
We alsoupdate our GAN model using some features of SAGAN (Zhang et al., 2019): applying spectralnormalization on the discriminator and using the unconditional hinge version of the adversarial loss(Lim and Ye, 2017; Tran et al., 2017):
[Equation]
Such loss function is equivalent to minimize the Kullback-Leibler divergence between the generateddistribution and the data distribution.
The VAE decoder and the GAN generator have 1713088learnable parameters.
For the score network architecture, we use the vanilla U-Net architecture(Ronneberger et al., 2015) in which we double the number of channels at each layer, we add groupnormalization (Wu and He, 2018) after each convolution
and we replace the ReLU non-linearies by
SiLU (Elfwing et al., 2018).
As in the univariate case, we use positional encoding (Vaswani et al.,2017) followed by a MLP block of shape (1 , 16 , 32) to incorporate the noise information at eachlayer.
The score network has 1607392 learnable parameters.
For inference, we use the same Langevindynamic scheme as above with the same hyperparameters as in the univariate case.
The three modelsare trained during 100 epochs with a batch size of 128 using ADAM with a momentum of 0 .
9 and alearning rate of 2 ×
10 − 4 .
Additional details.
The histograms of projection on the line passing through the mean of each
Gaussians are obtained using 20000 generated samples.
To assign a color to each bin of the histograms,we train a simple MLP of shape (784 , 1024 , 50 , 10) as classifier on MNIST.
The classifier is trainedduring 10 epochs using again ADAM with a momentum of 0 .
9 and a learning rate of 2 × 10 − 4 andreaches an accuracy of 0 .
98 on the test set.
S4.3 Subset of MNIST
Models details.
Since the dataset is more complex than before, we use bigger models.
For the scorenetwork, we use the architecture defined in Ho et al.
(2020), in which we set the number of channels to 64 instead to
and we remove the self attention layers (Wang et al., 2018) for computationalresource purposes.
The score network has 6072065 learnable parameters.
Again, we use an annealed
Langevin dynamic scheme for inference with the same hyperparameters as before.
For the VAE andthe GAN, we use the same architecture as before, using this time the convolutional discriminator of
DCGAN, and quadrupling the number of channels at each layer.
This is mainly done in order to scalethe generator/decoder to the score network.
Hence the VAE decoder/GAN generator has 7151104learnable parameters.
We train all three models during 600 epochs with a batch size of 128 using
ADAM with a momentum of 0 .
9 and a learning rate of 2 ×
10 − 4 .
Additional details.
We use the deep Wasserstein embedding proposed by Courty et al.
(2018) inorder to visualize histograms of projection in the Wasserstein space.
We use the exact same networkarchitecture and the same training procedure that in Courty et al.
(2018): first, one million pairs ofdigits of MNIST are chosen randomly, in which 700000 are kept for the training set, 200000 forthe test set, and 100000 for the validation set.
We normalize each image in order to consider it as atwo-dimensional distribution and we compute the 1 -Wasserstein distance for each pair.
Then, wetrain an autoencoder in a supervised manner in a way that the images at output of the autoencoder areclose to the images in input, and that the euclidean distance between two vectors in the latent space isclose to the 1 -Wassertein distance between the two corresponding images of MNIST.
As in Courtyet al.
, the latent Wasserstein space is of dimension 50 and the autoencoder is trained during100 epochs with a batch size of 100 and with an early stopping criterion.
Again, we use ADAM witha momentum of 0 .
9 and a learning rate of 10 − 3 .
We use the same classifier as before to assign colorto each bin of the histograms.
Finally, the histograms of projection on the line passing through thedeep Wasserstein barycenters of all 3 and 7 are obtained using 20000 generated samples.
S Additional experimental results
[Figure]
In the following, we provide additional experimental results.
First we study the influence of thearchitecture of the generative network by using more expressive neural network architectures inthe univariate case.
Then, we compare estimates of the bounds of Theorem 4, Corollary 6, and
Theorem 7 to estimates of the total variation distance and the Kullback-Leibler divergence in theunivariate case.
Then we study the possible correlation between the size of the score network and thetendency of the score-based model to generate unbalanced modes.
Finally, we provide additionalvisualizations of histograms of generated distributions for the univariate case and generated samplesfor the experiments on MNIST.
S5.1 Influence of the architecture of the generative network
[Figure]
In this experiment, we study the impact of the architecture of the generative network on its Lipschitzconstant and on the training stability of the model by comparing three different architectures: first, weuse a simple feed-forward MLP of shape (1 , 256 , 256 , 256 , 1) , then we add additive skip-connections of type "resnet" (He et al., 2016) to the previous backbone, and last we add concatenation skipconnections of type "densenet" (Huang et al., 2017) instead of additive "resnet" skip-connections.
The results can be found in Figure S2.
In the VAE setting, it seems that more expressive decoderarchitectures do not help to reach larger values of Lipschitz constant.
This supports the idea that the
Lipschitz constant is naturally limited by the VAE objective.
In the GAN setting, one can observethat adding skip-connections of type "resnet" seems to stabilize the training a little compared to thetwo other architectures since the colored bands are narrower.
However the model remains certainlytoo unstable for correct distribution generation.
This suggests that some generator architectures maybe better than others for staying stable while having a large Lipschitz constant.
[Figure]
S5.2 Bounds on TV distance and KL divergence in the
univariate case
[Figure]
In this experiment, we compare estimates of the bounds of Theorem 4, Corollary 6, and Theorem 7to estimates of the total variation distance and the Kullback-Leibler divergence.
We only provideresults for the VAE since the bounds are not interesting for the GAN since they are consequences ofinterpolation between modes due to a small Lipschitz constant of the generative network.
Yet thislatter in the GAN case achieves a large Lipschitz constant so does not interpolate significantly.
Toestimate empirically the total variation distance and the Kullback-Leibler divergence, we used theirrespective analytical formula
[Equation]
where p g # µ p and p ν are the respective densities of g # µ p and ν .
In order to estimate the lower boundsof Theorem 4 and Theorem 7, we set A of the form ( −∞ , − r/ 2] and we perform a grid search on r .
In Figure S3, we can observe that the estimates of the bounds provided by Theorem 4 and Theorem 7are not tights.
This is possibly because we selected a sub-optimal A
but it most likely follows fromthe fact that the bounds don’t take into account that g # µ p has automatically less mass on the modesthan ν since a significant amount of its total mass is between them.
One can also observe that theexplicit lower bound of Corollary 6 is much smaller than the bound of Theorem 4.
This can beexplained by the facts that ∥ m ∥ / 2 σ is probably a sub-optimal choice of r and that the bound of
Corollary 6 minimizes the interpolation between modes over all the mappings with Lipschitz constant
Lip ( g ) , regardless whether these mappings approximate well ν on its modes or not.
Since there isless interpolation if the modes are unbalanced (see Section 3.2), it is likely that the mappings g suchthat g # µ p is unbalanced are affecting the value of this bound in a bad way.
S5.3 Score-based modeling and modes proportions
We observed in our experiments that the size of the score network or/and its architecture seemedto be correlated with the tendency of the model to generate unbalanced modes.
More precisely,we trained several times the two U-Net architectures described above (i.e the modification of thebasic U-Net architecture (Ronneberger et al., 2015) and the architecture of (Ho et al., 2020)) on
[Table]
the synthetic Gaussian mixture on MNIST, increasing the number of channels for the former andadding Self Attention (SA) layers (Wang et al., 2018) for the latter.
In Table 1, "U-Net simple"corresponds to the modified basic U-Net architecture (Ronneberger et al., 2015) and "U-Net res"to the architecture of (Ho et al., 2020).
32 , 64 , and 128 correspond to the number of channels.
Themodels have respectively 545728 , 1607392 , 5291296 , 6072065 , and 6138369 learnable parameters.
We can see that a pattern seems to emerge: as the size increases, the proportion of the modes of thegenerated distributions seems less aleatory and the modes are globally less unbalanced.
Moreover,the add of self attention layers seems to reduce significantly the problem without increasing toomuch the number of parameters.
However, we only did three runs of this experiment for a matter ofcomputational resources.
This is probably not enough to conclude.
Additional examples
S5.4 Additional examplesS5.4.1 Univariate histograms
We provide additional visualizations of histograms of generated data with the three models for variousvalues of m .
[Equation]
stylebiblographie stylebiblographie
style
We can observe that the score-based model already generates unbalanced modes, but the phenomenonis globally less visible than in higher dimensions.
Secondly, we provide additional visualizationsof histograms of generated data with GANs trained with an additional gradient penalty term in thegenerator loss for various values of L ≈ Lip( g ) .
S5.4.2 Visualization of generated data
[Figure]
Finally, we show randomly chosen generated samples with VAE, GAN and SGM on the synthetic [Figure] mixture of Gaussian on MNIST and the subset of all 3 and 7 of MNIST.
[Figure]