G EN E VAL : A B ENCHMARK S UITE FOR E VALUATING G ENERATIVE M ODELS
Anonymous authors
Paper under double-blind review
A BSTRACT
Generative models are important for several practical applications, from low level image processing tasks, to model-based planning in robotics.
More generally, the study of generative models is motivated by the long-standing endeavor to model uncertainty and to discover structure by leveraging unlabeled data.
Unfortunately, the lack of an ultimate task of interest has hindered progress in the ﬁeld, as there is no established way to compare models and, often times, evaluation is based on mere visual inspection of samples drawn from such models.
In this work, we aim at addressing this problem by introducing a new benchmark evaluation suite, dubbed GenEval .
GenEval hosts a large array of distributions capturing many important properties of real datasets, yet in a controlled setting, such as lower intrinsic dimensionality, multi-modality, compositionality, independence and causal structure.
Any model can be easily plugged for evaluation, provided it can generate samples.
Our extensive evaluation suggests that different models have different strenghts, and that GenEval is a great tool to gain insights about how models and metrics work.
We offer GenEval to the community 1 and believe that this benchmark will facilitate comparison and development of new generative models.
I NTRODUCTION
Modeling uncertainty is a fundamental problem for machine learning.
In unsupervised settings, an ideal model should be able to describe all the possible events consistent with the provided context.
In supervised settings, there is often not a single correct output for a given input, and models need to be able to express the space of correct outputs and their relative likelihood.
One standard approach to this problem is to build a probability model of the output space, or more generally, an energy-based model that assigns a score to every possible output.
This kind of model is useful for comparing two possibilities, for example.
Recently there have been a number of proposed models that do not necessarily assign any score to possible outputs, but using a pseudo-random number generator, produce hallucinations that in some way resemble the true outputs.
This kind of model can be useful for examining possibilities.
In this work, we will consider the second kind of model, and take a ”generative model” to be any construction that has a method for outputting data points using a pseudo-random number generator.
In particular, we do not require ”generative models” to be able to compute a probability of a data point.
These kinds of models have become popular recently due to several works on Generative
Adversarial Networks (GANs) (Goodfellow et al., 2014), that have demonstrated the generation of realistic images (Radford et al., 2015; Karras et al., 2017).
However, this ﬁeld has struggled to solve its zeroth problem (even as it has made progress on its ﬁrst).
Before the design of “better” models , it is necessary to agree upon good metrics and methodology to evaluate the quality of samples generated by a model.
Part of the problem is that the downstream tasks of interest in the case of image generation have not been fully settled.
Nevertheless, there have been a few works that make rigorous attempts to quantify the capabilities of these models, restricted to GANs parameterized as convolutional nets in the setting of natural images (Lucic et al., 2017;
Huang et al., 2018), and concurrently Kurach et al.
In this work, we expand upon these with a suite of benchmark synthetic distributions that we use to evaluate not only GANs, but also other forms of generative models.
We deliberately avoid convolutional networks on images with the aim of decoupling the beneﬁts of various modeling paradigms from domain speciﬁc neural architectures.
If we believe that generative models should be studied as generic tools for dealing with uncertainty (as opposed to methods of generating images), this decoupling is necessary.
Additionally, practical, generic, and effective methods for measuring the distance between distributions in high dimensions with access only to samples do not currently exist.
On the other hand, in some cases where one of the distributions has some special structure, we can effectively measure distortion, see an example of this in Appendix sec.
A; using simple synthetic distributions allows us to be sure the distributions under consideration have the correct special structures.
Analysis of which methods are able to model which kinds of special structures (e.g. independence, causality, multi-modality) is critical for understanding these methods; with “real” data not fully under our control, such analysis can be difﬁcult.
A major contribution of this work is the GenEval toolbox to evaluate models.
Thanks to its general
API, new models can be easily plugged in and tested using a variety of metrics against popular baseline generative models.
Likewise, it’s very simple to add new distributions and evaluation metrics to the existing pool to highlight special properties of a model of interest.
R ELATED W ORK
Much recent literature on generative modeling of continuous distributions has focused on modeling natural images.
Restricting our attention to works aiming at generating samples, as opposed to scoring inputs, many recent models are based on Generative Adversarial Networks (Goodfellow et al., 2014; Arjovsky et al., 2017; Roth et al., 2017; Gulrajani et al., 2017; Denton et al., 2015; Karras et al., 2017), which are capable of generating very realistic high resolution images, a task that had challenged the research community for several decades (Geman and Geman, 1984).
For models trained on natural images, several metrics have been proposed, such as “inception score” (Salimans et al., 2016) and “Frechet inception” distance (Heusel et al., 2017).
Unfortunately, these metrics rely on yet another model for evaluation, and are known to only partially correlate with human judgment.
To further complicate matters, generation scores not only depend on the metric but also on the choice of architecture of the generator, typically a rather sophisticated convolutional neural network.
To summarize, comparison between different generative models is very difﬁcult, and it is often based on visual inspection of a handful of model samples (Denton et al., 2015; Arjovsky et al., 2017).
Several works have attempted to isolate the contribution of the estimation method by working in a controlled setting, e.g., by ﬁtting the model to a mixture of Gaussians in two or three dimensions (Arjovsky et al., 2017; Roth et al., 2017; Gulrajani et al., 2017), or in one dimension (Zaheer et al.).
This paper can be seen as a more comprehensive, thorough and extensive study building upon these earlier attempts.
While we share similar motivations, we do not restrict our focus to just GANs, but also include other popular models (Kingma and Welling, 2013; Dinh et al., 2016).
Recently, Lucic et al.
; Kurach et al.
(2018) have systematically compared different variants of GANs trained on natural images.
Unlike this work, these also focus on only GANs.
Morover, we restrict our study to purely synthetic distributions with known properties which we leverage at evaluation time (or, from another perspective, we broaden our study to distributions beyond those coming from the image domain).
The choice of metrics for generative models has been intense topic of debate.
Most works, overall those based on probabilistic modeling, report performance in terms of log-likelihood of samples generated by the data distribution according to the model.
As discussed in earlier work (Theis et al., 2016) and in sec.
5, this is not a good metric in general, as it cannot be estimated by all models, e.g., GANs, and it is very susceptible to the scaling of the input.
Interestingly, Lucic et al.
(2017) propose a precision/recall metric to evaluate the quality of samples, where precision measures the distance of each model sample to the data manifold and recall measures the distance of each data point to the closest model generation.
Unfortunately, such score disregards any weighting of the density, the fact that not all data points are equally likely and similarly, not all model samples have the same likelihood.
Recognizing this issue, we borrow the “two-sample test” criterion introduced by Lopez-Paz and Oquab (2016) and extended by Huang et al.
This two-sample statistic is based on a similar motivation but it operates on actual samples, using implicit Monte Carlo weighting to take into account the density, and obviating the need for an explicit optimization over a data-manifold.
D ISTRIBUTIONS
The benchmark focuses on distributions with support on R d for some d .
Their implementation in
GenEval requires two methods: one for computing the log-density at a point, and one for sampling.
GenEval implementation of distributions is modular, and allows combining distributions via products, mixtures, causal mappings, and isometric transforms.
Next, we brieﬂy describe the distributions we considered in this study.
Simple distributions Many of the distributions will be built from standard families of distributions, such as the multivariate Gaussian , and the Slab (or uniform) distribution with constant density on a rectanguloid in R d .
Such distributions can be embedded into a lower dimensional space R q with q < d , yielding what we call a “ﬂat” Guassian or Slab.
For instance, the density function of a ﬂat Slab is deﬁned as:
[Equation]
where χ R is the uniform distribution with support in R = { x : 0 ≤ x i ≤ r i , i = 1 , . .
. , q } , O is a d × q matrix with orthogonal columns, and η is a Gaussian density with small variance.
Mixtures Starting with K densities p 1 , ...., p k on R d , and a weighting vecor w on the simplex in R d , the density for a mixture is: p ( x ) =
(cid:80)
Ki
=
w i p i ( x ) .
When the bulk of the mass of p i is disjoint from the bulk of the mass of p j for all i (cid:54) =
j , the mixture model has a cluster structure.
We show results with these kinds of distributions in ﬁg. 1, 2 and in Appendix sec.
E.
Distributions with causal structure Given a function f : R q (cid:55)→ R d − q , q < d and a density p (cid:48) deﬁned in R q , we can can get a density with causal structure on R d by considering
[Equation]
where x ind are the independent ﬁrst q coordinates of x (drawn from a certain distribution), and x dep is the last d − q dependent coordinates of x , and η is a noise distribution (for example zero-mean
Gaussian with small variance).
Note that these kinds of distributions have locally lower dimensional structure than d , especially when d is large compared to q .
We show results with these kinds of distributions in ﬁg. 5.
Locally low dimensional distributions We consider three types of locally low dimensional distributions: causal distributions where f above is parameterized by a random fully-connected ReLU net or a random quadratic polynomial (to simulate non-linear manifolds), and an “image-like” distribution of shifted bumps.
For the random neural-networks, we use one-hidden-layer ReLU networks with default PyTorch (Paszke et al., 2017) initializations.
For the random quadratic polynomials, we take a sample for a two dimensional distribution and we project it into a d − 2 dimensional space with a matrix of normally distributed coefﬁcients, and then concatenate both vectors.
For the shifted bumps distribution, which is similar to the artiﬁcial triangle image distribution considered in (Lucic et al., 2017), we take a side-length s , a set of radii r 0 , ..., r l , and consider all shifts of an r i × r i square on a s × s background.
These form a set of s 2 ∗ l points in R s 2 ; we take a random orthogonal projection of this set to R d and place a ball of mass around each projected point.
Experiments with these distributions are shown in ﬁg. 3.
Distributions with independence structures We can increase the intrinsic dimension of any of these previous distributions by combining them as a product: p ( x ) =
Π Kj =1 p
i ( x ) , where p 1 , ..., p K are the basic components which have non-zero support on a disojint subset of coordinates.
Note that models are not provided with such information at training time.
We report results with product distributions in ﬁg. 4 and in Appendix.
G ENERATIVE MODELS
In this section we ﬁrst describe some classical baselines, meant to provide upper and lower-bound performances on each distribution, and then some neural-network based models.
Baselines:
The oracle baseline is the data distribution itself.
Given a metric assessing the discrepancy between samples drawn from the data and the model distributions (see sec. 5), oracle samples provide a lower bound on the error of an ideal model perfectly ﬁtting the data.
Ideal metrics would report an error close to 0 for oracle samples.
The next baseline we consider is the kernel density estimator (KDE) (Rosenblatt, 1956; Parzen, 1962) which places a Gaussian bump of a certain width around each training point.
KDE essentially memorizes the training set, and therefore, it is expected to provide great ﬁtting to the training data but poor generalization, overall in high dimensional spaces where the curse of dimensionality would require a vast amount of training samples to ﬁnely cover the data distribution.
Next, we have the multivariate Gaussian distribution ﬁtted by maximum likelihood (Pedregosa et al., 2011), meant to provide some sort of upper bound on the error, and the mixture of Gaussians with k components (MoG k ), which we ﬁt by using the expectation-maximization algorithm (Dempster et al., 1977).
We ﬁx the number of Gaussians in the mixture to 10 in all experiments.
Neural Models:
Among the neural models, we consider the Variational Auto-Encoder (VAE) (Kingma and Welling, 2013), the real non-volume preserving density estimator (RNVP)
(Dinh et al., 2016) and Generative Adversarial Network (GAN) (Goodfellow et al., 2014) with some of its variants, namely Wasserstein GAN (WGAN) (Arjovsky et al., 2017), WGAN with Gradient Penalty (WGAN-GP) (Gulrajani et al., 2017) and GAN with Noise Regularization (GAN-NR) (Roth et al., 2017).
See Appendixfor a brief review of these methods.
D ISTORTION S TATISTICS
Since all models must be capable of drawing samples, we measure ﬁtting error in terms of the distortion between two sets of points: samples drawn from the data distribution and from the model distribution.
We will discuss several measures of ﬁdelity to a distribution; and none of them are generically useful.
Measuring the distortion between real-valued distributions in high (or even not-so-high) dimensions continues to be an unsolved problem, especially if we add constraints on computational efﬁciency.
This is one of the reasons to use artiﬁcial data with special properties we control: although none of the metrics are generally useful, they can all be useful when the distributions have appropriate structure (see example in Appendix sec.
A).
Optimal Transport The Optimal Transport distance (OT) u ρ between sets S and T of points in a metric space with metric ρ , is deﬁned by:
[Equation]
We use the POT package 2 for computing the optimal transport, and assume ρ is Euclidean l 2 for the rest of this work, and drop it from the notation.
Sampling a ﬁxed number of points from two distributions P and Q and computing the OT distance between the samples does not give a distance on distributions (even if it were well deﬁned, it would not even give “distance” 0 from a distribution to itself), but it does correspond to an estimator of the OT distance between P and Q .
However, in high dimensions, with few samples, this estimator can have possibly counter-intuitive behaviors.
For example Arora et al. (2017), suppose we sample N points S and N points S (cid:48) from the uniform distribution over the unit sphere in R d , where d is large enough to see concentration-of-measure effects, and then N points T from δ 0 (the point mass at the origin).
If N is small compared to d , then the points in S and S (cid:48) are almost orthogonal, and so have distance roughly √ 2 between them.
On the other hand, the distance of any point in S to a point sampled from T is 1.
It is thus likely that u ( S, S (cid:48) ) >
u ( S, T ) even though S is sampled from the same distribution as S (cid:48) and T is not. On the other hand, OT has some pleasing properties.
It is continuous with respect to perturbations of the points in the sets. Furthermore, if two sets of points covers the same low-dimensional manifold in a high dimensional space, the OT distance between them will be relatively small. In the tables below, when using OT to measure the success of a generative model, we will always report the “oracle” OT score given by comparing two sets of samples from the true distribution. When the oracle scores are relatively high, the relative results should be taken cautiously.
Nearest Neighbor Two-Sample Statistic Two set of points, S ∼ p d and T ∼ p g , are close if the chance that a point from S has nearest neighbor belonging to T is 50% and vice versa.
This is gives the intuition behind the nearest neighbor two-sample statistic (2S) (Huang et al., 2018).
In this work, we use a version similar to the one in (Huang et al., 2018).
Assume | S | = | T | = N .
For each point x in S or T , deﬁne n ( x ) to be
[Equation]
, if min
[Equation]
min [Equation] x − y
[Equation]
| 0 , otherwise .
Then, we deﬁne the distance as the sum of the deviations from the optimal rate:
[Equation]
As N gets larger, it becomes easier to distinguish the distributions via this statistic; but different values of N may lead to different quality orderings. As the dimension of the distribution gets higher (for ﬁxed N ), it becomes harder to distinguish distributions via this statistic.
As with OT, we will always show the oracle value for a distribution when showing results of generative models.
Log-Likelihood In this work log-likelihood (LL) is estimated on the samples drawn from the model using the (known) data distribution - since we do not require models to necessarily be able to estimate data log-likelihood.
This gives a notion of how likely points generated by the model are, regardless of the overall ﬁt.
In other words, samples drawn from a model assigning all its mass to the mode of the data distribution will have even higher likelihood that samples drawn from the actual data distribution.
5. S PECIAL S TATISTICS
Our distributions have special structure that we can take advantage of, to better measure success in modeling that distribution.
Mode coverage (MC) Assuming a mixture distribution for the data and a uniform distribution over modes, we measure whether samples generated from the model have even coverage of the clusters.
We report “mode coverage” (MC) as the perpelxity of the mode assignments: MC = 2 H ( a ) , where
H is the entropy of the cluster assignment distribution a .
Causal Discovery Assuming a causal distribution (see sec. 3) and the ability of the model to perform conditional inference of a set of variables given the complementary set, we measure how close (in l 2 ) the recovered dependent coordinates are from what the ground truth values given a set of independent coordinates drawn from the true underlying distribution.
For all the neural models with a latent space, we estimate the missing variables by optimizing over the latent variable with an l 2 reconstruction loss over the observed coordinates, starting from a random point.
For KDE, we ﬁnd the nearest point from the training set in the constrained coordinates, and return the remaining coordinates of that training point.
For a mixture of Guassians, we take the nearest mean in the constrained coordinates among all components, and then return the maximum-likelihood estimate from that Gaussian of the remaining coordinates.
See ﬁg. 5 for results using this metric.
Independence Test Measuring the independence of high-dimensional real valued distributions is challenging.
In order to get some idea of the ability of the various models to detect independence, we will restrict our attention to distributions that are products of distributions that are easy to vectorquantize, and use categorical tests for independence on the quantized values.
That is, suppose we have a distribution p
= (cid:78) Ki =1 p i where p i is supported on the set of coordinates
c i .
Further suppose we have clustered p i into clusters C ij for j ∈ { 1 , ..., n i } , and that we have L samples X from a model.
Then, for each coordinate group c i , we make a table of size n i × ( n 1 n 2 ...n i − 1 n i +1 ...n K ) with the counts N s,t of the number of points in X that landed in each cell.
Here, s indexes rows of this table, corresponding to the n i centroids C
i 1 , ..., C in i for the points projected down to coordinate group c i , and t indexes columns of the table, corresponding to the product of all other clusters in all other coordinate groups.
Denote by M t = (cid:80) s N s,t /L
and M s = (cid:80) t N s,t /L ; we expect that
N s,t /L ∼ M s M t , so we take:
[Equation]
Independent OT and 2S Test
There is a partial remedy for dealing with the weaknesses of the distortion measures in high dimensions for distributions that we know have independence structure, see sec. 3.
Suppose that the distribution p we are interested in has support in R d .
Further suppose there are groups of coordinates I 0 , I 1 , ...
I L partitioning [1 , ..., d ] such that p
I j ⊥ p
I i for i (cid:54) =
j .
Here p I is the distribution given by projecting p onto the subspace spanned by the coordinates in I .
Then, for some distortion measure u , we can use the set distance given by:
[Equation]
where S I i is the projection of S onto I i (and likewise for T ).
In building such a distortion measure, we are using the knowledge of the true distribution.
However, because we are building the distributions in the benchmark, we are free to use this information in measuring distortion, even if the models should not get access to it during training.
Furthermore, note that this kind of distortion measure only can tell the difference between distributions that disagree on the marginals of the groups.
Even if it shows no distortion between two distributions, they may still be different.
On the other hand, because each group has smaller dimension, the distortion measure is less pre-disposed to show everything being different from everything else.
G EN E VAL
GenEval is written in Python and it consists of three main components: models, distributions, and metrics.
Models must deﬁne fit(X) and sample(N) methods, the former to train the model on a dataset of points X , the latter to draw N samples.
Models can optionally deﬁne a method to conditionally sample data given values for a set of ﬁxed coordinates.
Thanks to the general interface and modularity of GenEval, we simply incorporated the original implementations by the authors of the models whenever available.
As a result, GenEval does not enforce any speciﬁc machine learning framework.
For VAE(vae), GAN(gan, b), WGAN(wga, b), WGAN-GP(wga, a) and GAN-NR(gan, a) we used
PyTorch (Paszke et al., 2017), while for RNVP(nvp)
we used TensorFlow (Abadi et al., 2015).
For
Mixture of Gaussians and Kernel Density Estimation we employ the scikit-learn (Pedregosa et al., 2011) package.
Each distribution class must deﬁne the methods: sample(N) and logprob(X) .
Mixture distributions may also optionally deﬁne a method to estimate the most likely cluster assignment to each input data point, and product distributions can specify their components.
New distributions can easily be deﬁned via composition directly in conﬁguration ﬁles specifying which distribution to test on.
For instance, the library contains classes for a slab, a mixture of arbitrary distributions, and afﬁne transform.
Given these, one may deﬁne on-the-ﬂy a distribution over the surface of a rotated 3D box in some higher dimensional space.
Metrics methods take as input two sets of samples (from the data distribution and from the model) and output a scalar value.
Some metrics may optionally take as input a trained model and the data distribution to compute distribution dependent metrics like mode coverage.
Models and distributions are passed as input to GenEval via two Python script conﬁguration ﬁles, which specify all their eventual hyper-parameters.
Model hyper-parameters can also be speciﬁed via lists, which are used by GenEval to run a grid search over hyper-parameter values.
A user may then run GenEval with conﬁguration ﬁles that specify several distributions and several models.
GenEval then launches training and evaluation for the cartesian product of all possible combinations of model and distribution and report results on a table, as those shown in the Appendix.
E XPERIMENTS
We used GenEval to compare the models described in sec.
4 on several distributions.
For each distribution, GenEval ﬁrst runs a grid search over hyper-parameters, see for instance the conﬁguration ﬁles in Appendix sec.
D. To produce the results in this section for every method we ran a very extensive grid search over hyper-parameters, which reached about 20 , 000 conﬁgurations for some
GAN variants; and was in the thousands for all neural methods.
Afterwards, GenEval compares models across all metrics and compiles tables, such as those in Appendix E.
Unless otherwise stated, all experiments we discuss next have used 10,000 training samples, and unless otherwise speciﬁed in the ﬁgure, 1000 validation and test samples.
On the ﬁgures showing results with OT, OT was used as a cross-validation metric for the hyperparameter search; and 2S was used for ﬁgures showing results with 2S. For all other ﬁgures, cross validation was done using 2S.
Fig 1 shows two-sample results when the true distribution is a mixture of Gaussians with varying number of components in the mixture (left), variance (bottom) and intrinsic dimensionality (top versus bottom).
First, we observe that model performance is bounded by the oracle, as expected, with a nearly 0 distortion.
A mixture of Gaussian with 10 components achieves the lowest distortion when the number of components in the true distribution is less than 10 , but distortion degrades very rapidly as soon as the true number of components increases.
The KDE baseline instead has distortion which is constant with respect to number of components and variance 4 .
KDE often leads to competitive distortion compared to neural models, yet much higher than the oracle.
Second, in this experiment, GAN variants do not perform reliably better than the original GAN.
Third, we observe that none of the neural models perform well, despite the relative simplicity of the distribution.
Modeling distributions with more components, tighter variance and lower intrinsic dimensionality makes the learning problem considerably harder for these models.
The results are somewhat different if we cross-validate and evaluate according to OT, as shown in ﬁg. 2.
With this statistic, there is no performance gap between the oracle and KDE, highlighting the importance of the choice of metric when comparing methods.
Oracle performance according to OT degrades as a function of the number of components (left half of Figure 2), and especially as the intrinsic variance increases (bottom right of Figure 2), but not when the intrinsic dimensionality is lower (top right of Figure 2). Thus, as suggested by the example in 5, sampled OT can be an unreliable measure of success when the true distribution ﬁlls out space in high dimensions.
Next, we evaluate on various manifold distribution and product of manifold distributions, see ﬁg.and 4.
One interesting observation is that OT and 2S do not correlate very well (see discussion in sec. 5).
On these tasks, Real NVP performs well in terms of the nearest neighbor two-sample statistic, followed by MoG 10 , with other methods performing worse.
In terms of OT, KDE does best on the products of manifolds, and VAE does the best on the manifolds.
In fact, VAE does better than oracle there (see two bottom left plots in ﬁg. 3 and the note in 5), suggesting it is denoising the true distribution.
In the next experiment reported in ﬁg. 5, we analyze the ability of the models to discover causal structure on manifold distributions.
We do not report the results of Real NVP here because we cannot make a conditional estimate in the same way as the other neural models.
We observe that all models have hard time beating the MoG 10 baseline in this case.
Finally, we look at a product of independent Gaussian mixture distributions in ﬁg. 6.
None of the models are able to succesfully ﬁt this distribution, although WGAN-GP and Real NVP do the best.
KDE in particular fails to ﬁt this distribution, both in terms of two-sample and in terms of the χindependence test.
We can also see that all of the GAN variants improve on the mode coverage of the vanilla GAN.
D ISCUSSION
Perhaps to the delight of neural generative model skeptics, one sees that on almost all distributions and all metrics that we consider, one of KDE or mixture of 10 Gaussians are competitive with (and often superior to) the neural models.
These take a tiny fraction of the training cost, have essentially no hyper-parameter sweep, and have simple, well understood ﬁtting routines.
However, the neural models, especially GAN, are designed to be used where l 2 in the observation space makes little sense as a metric.
On most of our examples l 2 in the observation space is locally meaningful, and most of our metrics rely on l 2 to be (at least) locally meaningful.
Thus we consider success at our benchmarks neither necessary nor sufﬁcient for a model to be good at the perhaps more complex tasks for which neural models are designed.
We discourage neural model skeptics from dismissing neural models based on these results: worse results on simple tasks against unscaleable baselines better adapted to the simple tasks does not necessarily mean an approach should be abandoned.
On the other hand, neural generative model boosters should take heed of these results.
They show that on datasets that are not images, with networks that are not convolutional networks, neural models do not do well.
Moreover, speciﬁc to GANs, our results corroborate the results of Zaheer et al.;
Arora and Zhang (2017), and suggest that GANs do not necessarily learn to sample from the target distribution.
Furthermore, some protocols purported to improve GANs in the image setting do not reliably improve results across the metrics and datasets shown here (although cluster coverage does seem to be reliably improved).
This is natural, as practitioners have spent much effort tuning models for performance in the image domain, and of course neural models will do poorly without well adapted architectures 5 .
Nevertheless, most works trying to improve or understand GANs (empirically or theoretically) discuss the training protocol independently of the relationships between the inductive biases of the neural architectures and the properties of the distributions to be modeled.
Our results here suggest that these relationships cannot be ignored when studying GANs.
We also see that while no single neural model is all-around superior to the others, Real NVP does do better than the other models in many cases, especially in terms of the two-sample statistic.
However, it does not dominate, for example doing worse at products of manifolds in terms of OT, see ﬁg. 4, and worse on mixtures of Gaussians with large numbers of components, see ﬁg. 1 6 .
We can also see other quirks of the models.
For example, we can see that “denoising” effects of a
VAE can make it appear to be a good model on some datasets and metrics, but poor in others.
It is one of the better performing neural models in ﬁg. 5 when measuring causal error, showing it has learned the manifold and the functional dependence.
But at the same time, it is bad in terms of twosample (ﬁg. 3); perhaps because it is collapsing the two free components in the output.
More generally, these kinds of effects show that in general, it is important to consider multiple metrics.
The gold standard should be a success in a downstream task of interest, but in the absence of such a task, looking at a single metric to judge success can be misleading.
A B LOCK I NDEPENDENCE AND D ISTORTION M EASURE R ESOLUTION
We give a simple example showing how eq.
3 can increase the accuracy, or more precisely, the resolution of a distortion measure.
We take a product of 10 mixtures of Gaussians in R 12 , each with 3 components, with modes randomly distributed over the unit sphere in R 12 , as in ﬁg. 6.
The overall ambient dimension is 120 .
Measuring OT against a single best ﬁt Gaussian, which is clearly a poor model of the original distribution, without leveraging the groups structure (see again eq. 3), gives a distortion of 7 . 3 for the oracle samples versus 8 . 6 for the samples from the Gaussian.
If we do leverage information about the independence structure among the groups, we get a distortion of 5 . 9 for the oracle versus 19 .
3 for the
Gaussian.
Raising the number of mixtures in the product to 30 gives 17 .
4 vs 17 .
6 without groups and 17 .
5 vs 54 .
5 with.
It is therefore essential even in relatively low dimensions to leverage the special structure of the distributions in order to assess the real quality of the sameples generated by a model.
B O THER M ETRICS
In addition to the metrics introduced in sec.
5,we also considered the Hausdorff distance.
Hausdorff Distance
The Hausdorff distance (H) h ρ ∞ between sets S and T of points in a metric space with metric ρ , is deﬁned by:
[Equation]
As in the case of optimal transport, this deﬁnes a metric between sets; but when we sample from the distributions and compare the distance between the samples, we do not get a metric on the distributions.
We will denote by H ∞ ( P, Q ; N ) the statistic obtained by sampling N points from P ,
N points from Q , and computing the Hasudorff distance between the sampled points.
We will show results with an average version of this statistic: by H 2 ( P, Q ; N ) we take the average minimal square distance, instead of the max.
That is, deﬁne:
[Equation]
and set the statistic H 2 ( P, Q ; N ) to be h 2 ( S, T ) for N points S sampled from P and N points T sampled from Q .
Note that h 2 is not even a metric on sets, as it does not satisfy the triangle inequality.
The Hausdorff statistics should be taken as a measure of the difference of support between distributions.
That is, as the number of sampled points gets large, two distribution with the same support (but perhaps very different densities on that support) will have relatively small difference in the Hausdorff statistic.
C A NALYSES OF T WO -S AMPLE T EST
The Two-Sample Test depends on the number of points sampled from the distributions.
We analyze how number of test points and number of training points affects the metric.
We use random quadratic polynomial distribution and a distributions of shifted bumps as shown in ﬁg. 7 and as described in sec. 3.
For both distributions we observe a similar pattern.
As the number of points goes up the errors also goes up for all the models.
However, the degradation is much more prominent for KDE.
The same applied to the size of the training set: neural models may generalize from less amount of training data
[Figure]
D H YPER - PARAMETER S EARCH
As mentioned in the section 6, model conﬁgurations for each experiment are deﬁned in conﬁguration ﬁles.
Each ﬁle deﬁnes a MODELS variable with a list of dictionaries describing the model.
Each dictionary has three keys: name , model (refers to a model class), and args .
A user may want to re-use the same model in several cases to evaluate impact of different subsets of parameters.
E.g., all ﬂavors of the GAN models implemented in a single class and evaluated separately.
Hyper parameters for each case are deﬁned in args dictionary that maps parameter to one or more possible values.
All our models use multilayer perceptron as the base architecture.
The default activation function is ReLU, but in order to help the model to sample from a mixture of gaussians, we also considered q -ary multi maxout activation that is a max pooling over groups of q units.
The activation function is determined by maxout parameter.
Zero stands for ReLU, while positive values stand of q -ary multi maxout.
Parameter --nz deﬁnes the size of the latent space.
By default GAN models use
RMSProp optimizer unless --adam is speciﬁed.
RealNVP and VAE use Adam optimizer by default.
We followed the choice in the original implementations here.
(Arjovsky et al., 2017) noticed that for better results in training WGAN several discriminator updates should be done per generator updates.
Besides, discriminator should get 100 extra updates for ﬁrst 25 steps and every 500th step.
We captures these heuristics in --Diters and --boost discriminator ﬂags.
For RealNVP we use only channel based masking with 8 coupling layers (deﬁned by --chain length ).
Each layer is implemented by a multilayer perceptron with optional batch and weight normalization.
Below we show the complete conﬁguration ﬁle used for multimodal distributions.
[Table] [Table]
E M ULTIMODAL D ISTRIBUTIONS
In this section
[Table]
we report detailed results using various mixture of Gaussians.
“ad” means ambient dimension, “id” means intrinsic dimensions where the MoGs are embedded, “mt” speciﬁes the kind of mean allocation (on the circle “C” or at random “R”), “r” refers to the variance and “MoG” to the number of components in the mixture.
[Table] [Table]
[Table] [Table]