O N THE OPTIMAL PRECISION OF GAN S
Anonymous authors
Paper under double-blind review
A BSTRACT
Generative adversarial networks (GANs) are known to face model misspeciﬁcation when learning disconnected distributions.
Indeed, continuous mapping from a unimodal latent distribution to a disconnected one is impossible, so GANs necessarily generate samples outside of the support of the target distribution.
In this paper, we make the connection between the performance of GANs and their latent space conﬁguration.
In particular, we raise the following question: what is the latent space partition that minimizes the measure of out-of-manifold samples?
Building on a recent result of geometric measure theory, we prove a sufﬁcient condition for GANs to be optimal when the dimension of the latent space is larger than the number of modes.
In particular, we show the optimality of generators that structure their latent space as a ‘simplicial cluster’ - a Voronoi partition where centers are equally distant.
We derive both an upper and a lower bound on the optimal precision of GANs learning disconnected manifolds.
Interestingly, these two bounds have the same order of decrease: √ log m , m being the number of modes.
Finally, we perform several experiments to exhibit the geometry of the latent space and experimentally show that GANs have a geometry with similar properties to the theoretical one.
I NTRODUCTION
GANs (Goodfellow et al., 2014), a family of deep generative models, have shown great capacities to generate photorealistic images (Karras et al., 2019).
State-of-the-art models, like StyleGAN (Karras et al., 2019) or TransformerGAN (Jiang et al., 2021), show empirical beneﬁts from relying on overparametrized networks with high-dimensional latent spaces.
Besides, manipulating the latent representation of a GAN is also helpful for diverse tasks such as image editing (Shen et al., 2020;
Wu et al., 2021) or unsupervised learning of image segmentation (Abdal et al., 2021)
However, there is still a poor theoretical understanding of how GANs organize their latent space.
We argue that this is a crucial step in better apprehending the behavior of GANs.
To better understand GANs, the setting of disconnected distributions learning is enlightening.
Experimental and theoretical works (Khayatkhoei et al., 2018; Tanielian et al., 2020) have shown a fundamental limitation of GANs when dealing with such distributions.
Since the distribution modeled by GANs is connected, some areas of GANs’ support are necessarily mapped outside the true data distribution.
When covering modes of a disconnected distribution, GANs try to minimize the measure of the generated distribution lying outside the true modes ( e.g. the purple area on the right of Figure 1).
In other words, GANs need to minimize the measure of the borders between the modes in the latent space.
Considering a Gaussian latent space, minimizing this measure is closely linked to the ﬁeld of Gaussian isoperimetric inequalities (Ledoux, 1996).
This ﬁeld aims at deriving the partitions that decompose a Gaussian space with a minimal Gaussian-weighted perimeter.
We argue that the optimal partitions derived in Gaussian isoperimetric inequalities cast a light on the structure of the latent space of GANs.
Most notably, a recent result (Milman and Neeman, 2022) shows that, as long as the number of components m in the partition and the number of dimensions d of the Gaussian space are such that m ≤ d + 1, the optimal partition is a ‘simplicial cluster’:
a Voronoi diagram obtained from the cells of equidistant points (see left of Figure 1 for m = 3 and d = 3).
In this paper, we apply this result to the ﬁeld of GANs and show, both experimentally and theoretically, that GANs with ‘simplicial cluster’ latent space minimize out-of-distribution generated samples.
We draw the connection between GANs and Gaussian isoperimetric inequalities by using the precision metric (Sajjadi et al., 2018; Kynkäänniemi et al., 2019), which quantiﬁes the portion of generated points that support the target distribution.
We show that GANs with a latent space organized as a simplicial cluster reach optimal precision levels and derive both an upper and a lower bound on the precision of such GANs.
Experimentally, we show that the GANs with higher performances tend to organize their latent space as simplicial clusters.
To summarize, our contributions are the following:
• We are the ﬁrst to import the latest results from Gaussian isoperimetric inequalities by (Milman and Neeman, 2022) to the study and understanding of GANs.
We use it to show that the latent space structure has major implications on the precision of GANs.
• We derive a new theoretical analysis, stating both an upper bound and a lower bound on the precision of GANs.
We show that GANs with latent space organized as a simplicial cluster have an optimal precision whose lower bound decrease in the same order as the upper bound: √ log m , where m is the number of modes.
• Experimentally, we show that GANs tend to structure their latent space as ‘simplicial clusters’ on image datasets.
First, we explore two properties of the latent space: linear separability and convexity of classes.
Then, we play with the latent space dimension and highlight how it impacts the performance of GANs.
Finally, we show that overparametrization helps approaching the optimal structure and improving GANs performance
R ELATED W ORK 2. N OTATION
Data.
We assume that the target distribution µ (cid:63) is deﬁned on Euclidean space R D (potentially a high-dimensional space), equipped with the Euclidean norm (cid:107) · (cid:107) .
We denote S µ (cid:63) the support of this unknown distribution µ (cid:63) .
In practice, however, we only have access to a ﬁnite collection of i.i.d.
observations X 1 ,..., X m distributed according to µ (cid:63) .
Thus, for the remainder of the article, we let µ m be the empirical measure based on X 1 ,..., X m .
Generative model.
We consider G L the set of L -Lipschitz continuous functions from the latent space R d to the high-dimensional space R D .
Each generator aims at producing realistic samples.
The latent space distribution deﬁned on R d is supposed to be Gaussian and is noted γ .
Thus, each candidate distribution is the push forward between γ and a generator G and is noted as G (cid:93) γ .
This Lipschitzness assumption on G L is reasonable since Virmaux and Scaman (2018) has shown and presented an algorithm that upper-bounds the Lipschitz constant of any deep neural network.
In practice, one can enforce the Lipschitzness of generator functions by clipping the neural networks’ parameters (Arjovsky et al., 2017), penalizing the discriminative functions’ gradient (Gulrajani et al., 2017; Kodali et al., 2017; Wei et al., 2018;
Zhou et al., 2019), or penalizing the spectral norms (Miyato et al., 2018).
Note that some large-scale generators such as SAGAN (Zhang et al., 2019) and
BigGAN (Brock et al., 2019) also make use of spectral normalization for the generator.
2. GAN S AND DISCONNECTED DISTRIBUTIONS
A signiﬁcant ﬂaw of GANs (Goodfellow et al., 2014) is their difﬁculty in learning multi-modal distributions.
This phenomenon has been analyzed by Khayatkhoei et al.
and Tanielian et al.
The problem comes from this fundamental trade-off: GANs can either cover all modes and generate out-of-manifold samples or generate only good quality samples and neglect some modes (mode collapse).
Some methods have proposed ways to train disconnected generators (Khayatkhoei et al., 2018; Gurumurthy et al., 2017), but with little beneﬁts compared to single overparametrized generators with rejection mechanisms (Tanielian et al., 2020; Azadi et al., 2018).
Empirically, different works give intuition on the latent space structure of GANs.
Karras et al.
(2019) show that binary attributes are linearly separable in the Gaussian latent space and even better separated in an intermediate latent space.
Shen et al.
(2020) stress that face attributes are separated by hyperplanes, and edit images only by moving in the latent space orthogonally to these hyperplanes.
Arvanitidis et al.
and
Chen et al.
(2018a) view the latent space of generative models with a
Riemannian perspective and deﬁne a metric tensor using the generator’s Jacobian to ﬁnd the shortest paths on the data manifold.
However, these ﬁndings might not be sufﬁcient for a clear understanding of the required geometry of the latent space.
For instance, Karras et al.
(2019) use a very large latent space dimension ( R 512 ), while Sauer et al.
(2022) argue that the optimal latent space dimension is close to the intrinsic dimension of images ( R 64 for ImageNet).
Tanielian et al.
(2020) stress the relevance of this problem by showing that the precision of GANs can converge to 0 when the number of modes or the distance between them increases.
In this paper, we make a step towards a better understanding of the behavior of GANs and expose an optimal latent space conﬁguration when the number of modes m and the dimension of the latent space d are such that m ≤ d + 1.
2. E VALUATING GAN S
When learning disconnected manifolds, Sajjadi et al.
(2018) illustrated the need for new measures that simultaneously evaluate the quality (Precision), and the diversity (Recall) of the generated samples.
Kynkäänniemi et al.
(2019) highlighted an important drawback of this PR metric: it cannot correctly interpret situations when large numbers of samples are packed together.
They propose an Improved
PR metric based on the non-parametric estimation of manifolds to correct this.
Improved PR metric: Informally, for a generator G , precision ( α G ) quantiﬁes the proportion of generated samples that can be approximated with true samples, while recall ( β G ) measures the proportion of true samples that can be approximated with generated ones.
Applying this to GANs, using the target distribution µ (cid:63) and modeled distribution G (cid:93) γ , the Improved PR metric was shown, by Tanielian et al.
(2020, Theorem 1), to be asymptotically equivalent to:
[Equation] [Equation]
where S µ (cid:63) denotes the support of µ (cid:63) .
More recently, Naeem et al.
(2020) have shown that the
Improved PR metric (Kynkäänniemi et al., 2019) is not robust to outlier samples of both the target and the generated distribution.
To correct this and ﬁx the overestimation of the manifold around real outliers, Naeem et al.
(2020) propose the Density/Coverage metric.
Density/Coverage: Instead of counting how many fake samples belong to a real sample neighborhood, density counts how many real sample neighborhoods contain a generated sample.
On the other hand, coverage counts the number of real sample neighborhoods that contain at least one fake sample.
In the next section, we will use the notion of precision and recall deﬁned in (1).
Using this deﬁnition allows us to circumvent the non-parametric estimators involved in the existing metrics (Kynkäänniemi et al., 2019; Naeem et al., 2020).
D ETERMINING OPTIMAL PRECISION IN GAN S
We want to better understand the latent space of GANs and stress which GANs have the highest precision under speciﬁc constraints.
GANs are push-forward distributions of a unimodal (connected)
Gaussian distribution γ and a continuous function parameterized by G .
Consequently, the modeled generative distribution G (cid:93) γ will have connected support.
When learning a target distribution µ (cid:63) with disconnected manifolds, GANs necessarily map fake data points out of the true manifold.
This leads us to the following question: given that a generator samples data points in each of the distinct modes, what can be its maximum precision?
To begin with, let’s assume a target distribution µ (cid:63) composed of m disconnected modes.
Assumption 1 (Disconnected manifolds) .
The target distribution µ (cid:63) lays on m equally measured spheres M i , i ∈ [ 1 , m ] of radius r, each located at equal distance D (with D / 2 >> r).
The use of Assumption 1 is reasonable.
First, on many real-world datasets, data is correctly balanced in between the different modes.
The equal distance assumption can be justiﬁed from the concentration of distances in high-dimensional spaces: centers of modes will be approximately at equal distance (Beyer et al., 1999; Aggarwal et al., 2001).
It has also been shown that embeddings of deep neural networks trained for classiﬁcation tend to collapse around means that are equidistant to one another (Kothapalli et al., 2022).
This could thus pave the way for a new analysis where the chosen distance is no longer the euclidean distance in R D but a distance in the feature space of the generator or any pre-trained classiﬁer.
We also further discuss how this assumption could be relaxed.
For the rest of the paper, let us deﬁne the set of well-balanced generators that maps equally data points to the different modes of the data distribution:
Deﬁnition 1.
A generator G is well-balanced, if for all i ∈ [ 1 , m ] , G (cid:93)
γ ( M 1 )
= ...
= G (cid:93) γ ( M m ) .
Considering well-balanced generators is also fair since many empirical improvements such as
WGAN-GP (Gulrajani et al., 2017) or BigBiGAN (Donahue and Simonyan, 2019) have signiﬁcantly decreased the mode collapse.
GANs generate diverse output distribution on datasets such as CIFAR10,
CIFAR100, and ImageNet.
To validate the use of well-balanced generators, we ran a small experiment and evaluate the proportion of each class generated by GANs on MNIST and CIFAR10.
On MNIST, the minimal proportion of a class is 9 .
and the maximal 10 . 9, while it is respectively 8 . 3 and 11 .on CIFAR10 (in %).
The variance/mean ratio is equal to 0 .
03 for MNIST and 0 .
22 for CIFAR10.
3. P RECISION AND THE ASSOCIATED PARTITION
Now that the prerequisites for both the data and the model have been given, we propose to deﬁne our approach.
We create the connection between the set of generators from R d to R D and the set of partitions in the latent space.
In particular, for each given partition in R d , there exists a set of associated generators deﬁned as follows:
Deﬁnition 2.
For a given partition A
= { A 1 ,..., A m } on R d , we say that G is associated to A if: for all i ∈ [ 1 , m ] , for all z ∈
A
i , i = argmin
j ∈ [ 1 , m ]
d ( G ( z ) , M j ) , where d ( X , M j ) = min y ∈ M i (cid:107)
X − y (cid:107) .
It is clear that each given generator is associated with a unique partition in the latent space.
Moreover, the geometry of the partition partly explains its behavior and performance.
We are interested in maximizing the precision of generative models.
Any point in the intersection of two cells
A i ∩ A j , ( i , j ) ∈ [ 1 , m ] 2 is at equal distance of M i and M j and thus does not belong to any of these modes (since D / 2 >> r ).
Besides, due to the generator’s
Lipschitzness, there is a small neighborhood of the boundary such that any points in this neighborhood will be mapped out of the target manifold.
This region in the latent space thus reduces the precision.
For a given ε > 0, we now deﬁne the epsilon-boundary of the partition
A as follows:
Deﬁnition 3.
For a given partition A
= { A 1 ,..., A m } of R d and a given ε ∈ R (cid:63) + , we denote ∂ ε
A the epsilon-boundary of A , deﬁned as follows:
[Equation]
where A ε corresponds to the ε -extension of set A .
To better understand the link between the precision of a generator α G and its associated partition A , we state the following lemma:
Lemma 1.
Assume that Assumption 1 is satisﬁed and A be a partition in R d .
Then, any generator
G ∈ G L associated with A veriﬁes:
[Equation]
where ε
[Equation]
= D
[Equation]
Interestingly, this result holds independently of the partition A .
This result highlights that the geometry of the partition gives an upper-bound on the precision of the generator.
Consequently, to properly determine this bound on the precision levels of generative models, one might be interested in determining the measure of this epsilon-boundary ∂ ε A .
Furthermore, to exhibit generative models with optimal precision levels, one must look at partitions with the smallest epsilon-boundary measures γ ( ∂ ε A ) .
This is tightly connected to the theoretical ﬁeld of Gaussian isoperimetric inequalities.
3. O PTIMALITY IN GAN S
Isoperimetric inequalities link the measure of sets with their perimeters.
More speciﬁcally, isoperimetric inequalities highlight minimizers of the perimeter for a ﬁxed measure, e.g. the sphere in an euclidean space with a given Lebesgue measure.
Gaussian isoperimetric inequalities study a similar problem in Gaussian space.
Borell (1975) and Sudakov and Tsirel’son (1978) show that in a ﬁnite-dimensional Gaussian space, among all sets of a given measure, half-spaces have a minimum
Gaussian perimeter.
More formally, for any Borel set A in R d and a half-space H , if we have γ ( A ) (cid:62) γ ( H ) , then γ ( A ε ) (cid:62) γ ( H ε ) for any ε > 0, where A ε denotes the ε -extension of the set A .
The Gaussian multi-bubble conjecture was formulated when looking for a way to partition the
Gaussian space in m parts and with the least-weighted boundary.
It was recently proved by Milman and Neeman (2022) who showed that the best way to split a Gaussian space R d in m clusters of equal measure, with 2 (cid:54) m
(cid:54) d
+ 1, is by using ‘simplicial clusters’ obtained as the Voronoi cells of m equidistant points in R d .
Convex geometry theory tells us that each cell is a convex cone, whose borders are hyperplanes going through the origin of R d .
We note A (cid:63) any partition corresponding to this optimal conﬁguration, see Figure 1.
The aim of the present paper is to leverage this result to better understand the behaviour of GANs.
We argue that in the case where m (cid:54) d + 1, optimal models in levels of precision are closely linked to the optimal partitions A (cid:63) derived in the Gaussian Multi-Bubble conjecture (Milman and Neeman, 2022).
Besides, using results on the Gaussian boundary measure of those sets (Schechtman, 2012), we can also derive an upper-bound on the maximal precision of generative models, as follows:
Theorem 1 ( Upper-bounding the precision ) .
Assume that Assumption 1 is satisﬁed and m (cid:54) d + 1 .
For any δ > 0 , if L is large enough, then for any well-balanced generator G ∈ G L , we have:
[Equation]
In particular, there exists L with L (cid:62) D (cid:112) log ( m ) , such that for any well-balanced generator G ∈ G L :
[Equation]
where ε min =
D / L .
Theorem 1 links the precision of well-balanced generators with the optimal partition from Milman and Neeman (2022).
In particular, result in (3) gives an interesting insight when training GANs on a ﬁnite number of modes.
Tanielian et al.
(2020, Theorem 3) showed a similar result but for the asymptotic case when the number of modes increases:
[Equation]
For ε min (cid:112) log ( m ) =
o ( 1 ) both (3) and (4) have the same behaviour w.r.t.
to the number of modes.
Now, to further show the usefulness of A (cid:63) , we prove the following theorem:
Theorem 2 ( Lower-bounding the precision ) .
Assume that Assumption 1 is satisﬁed and m (cid:54) d + 1 .
For any δ > 0 , there exists C large enough and L (cid:62) D √ m (cid:112) π log ( Cm ) , and a well-balanced generator
G (cid:63) ∈ G L associated with A (cid:63) such that for any other well-balanced generator G ∈ G L , we have:
[Equation]
Moreover, if m ≤ d:
[Equation]
where ε
[Equation]
D
[Equation]
[Figure]
This theorem, which proof is delayed to Appendix, shows that the set of generators associated with A (cid:63) contains optimal generators w.r.t.
precision.
More importantly, it shows that when L is large enough, the bound in (3) may be tight, as it is almost reached by optimal generators deﬁned in
Theorem 2.
An example of such optimal generators for the 2D case is given in Figure 2.
This speciﬁc generator memorizes the dataset, since all samples are mapped to one of the center of the modes
M i , i ∈ [ 1 , m ] , except for those in ε -boundaries.
It is not clear however, whether those are the only generators with optimal precision.
We see that when the number of modes is less than the number of dimensions in the latent space, the only factor that impacts the precision is the number of modes.
What if modes are not equally distant?
This assumption is needed for the deﬁnition of a wellbalanced generator as the one proposed in Figure 2.
In R 2 for example, if there is no assumption at all on the location of the modes, there might not be any well-balanced generators associated with the optimal partition A (cid:63) .
As shown in Figure 3, the latent space conﬁguration obtained by the GANs on 3 aligned data points (right) is made of two parallel hyperplanes, much different from A (cid:63) deﬁned by
Milman and Neeman (2022) (left).
space using heatmaps of the norm of the gradient of the generator.
What if the dimension m > d + 1 ?
This setting is much more complex.
First, Assumption 1 is no longer valid, since one cannot ﬁnd m points equally distant from one another.
Second, the result from Milman and Neeman (2022) is no longer valid and the optimal partition of the Gaussian space in m equal cells is unknown.
In this generalized context, GANs could hint at the optimal partition geometry.
We show in Figure 4 examples when training a GAN from R 2 to R m with m equidistant target points in R m .
This could give some insights on how to divide the Gaussian space into m equitable areas with least Gaussian-weighted perimeter.
E XPERIMENTS : UNDERSTANDING THE LATENT SPACE OF GAN S
In the following experiments, we validate our theoretical analysis and derive insights for GANs trained on toy and image datasets.
We verify if the latent space geometry of GANs has similar properties than the minimizers of the Gaussian isoperimetric inequality (Milman and Neeman, 2022), and compute a series of experiments on the latent space of GANs to better understand its properties.
In all the following experiments, we train WGANs with gradient penalty (Arjovsky et al., 2017;
Gulrajani et al., 2017).
For mixture of Gaussians, generator and discriminator are MLP networks.
For MNIST (LeCun et al., 1998), both the generator and discriminator are standard convolutional architectures.
On CIFAR-10 and CIFAR-100 datasets (Krizhevsky et al., 2009), we use a Resnetbased (He et al., 2016) convolutional architecture with self-modulation in the generator (Chen et al., 2018b), and the transformer-based architecture from Jiang et al.
To evaluate the performance of GANs, we use both the precision (Kynkäänniemi et al., 2019), the FID (Heusel et al., 2017), and the density/coverage (Naeem et al., 2020).
As recommended by recent works (Naeem et al., 2020;
Kynkäänniemi et al., 2022), we use a dataset-speciﬁc classiﬁer to extract image features instead of an
ImageNet pre-trained classiﬁer, and thus refer to the FID as FD for Fréchet Distance.
Implementation details are given in Appendix and code is provided in Supplementary Material.
4. L INEAR SEPARABILITY AND CONVEXITY
Milman and Neeman (2022) show that the optimal conﬁguration in the latent space is obtained as the
Voronoi cells of m equidistant points in R d , if m ≤ d
+ 1.
This means that if GANs reach this optimal conﬁguration, each of the cells must be convex polytopes and thus verify the following properties:
1) each cell has ‘ﬂat’ sides, and are bounded exclusively by faces;
each cell is convex
In the following experiments, we study whether GANs’ latent spaces feature these two properties.
Are classes linearly separable in the latent space of GANs?
To verify this, we leverage a labeled dataset and investigate if a simple linear model ( e.g. , multinomial logistic regression) can map from latent space to label space.
If cells in the latent space are bounded by hyperplanes, the linear model is expected to be a good predictor of a generated sample’s label.
We consider a labeled dataset of samples with a ﬁxed number of classes.
G θ is a pre-trained generator, and C φ a pre-trained classiﬁer considered as an oracle.
Using G θ and C φ , we construct a dataset of latent vectors z ∈ R d and their associated labels y
=
C φ ( G θ ( z )) .
On CIFAR-10/100, similarly to
Razavi et al.
(2019), only data points with above a conﬁdence threshold are accepted.
This dataset is later split into 100k training points and 10k test points.
The mapping from latent vectors z to their labels y is learned by a multinomial logistic regression.
We report the test-set results in Tableunder the column LogReg Accuracy .
This accuracy reaches 90% on MNIST and 70% on CIFAR-10.
Interestingly, there is also a correlation between the linear separability of the latent space and the precision metric, which validates the optimality of the simplicial cluster partition.
Are classes convex in the latent space of GANs?
In this experiment, we draw two random latent vectors z 0 and z 1 that belong to the same class.
Then, we generate linear interpolations z ε = ε z
+ ( 1 − ε )
z 1 and verify if these new samples belong to the same class as z 0 and z 1 , i.e. whether C φ ( G θ ( z ε )) equals to C φ ( G θ ( z 0 )) .
We report the mean accuracy of this experiment in Table 1 under the column Convex Accuracy .
Again, the higher the precision, the ‘more convex’ each cell in the latent space seems to be.
For a qualitative evaluation, we show this phenomenon in Figure 5 and stress that linear interpolations conserve the image class.
[Table]
Visual inspection conﬁrms that latent linear interpolations between two the often conserve
Impact of the latent space dimension.
Now, we evaluate the impact of the latent space dimension on MNIST, CIFAR10 and CIFAR100 datasets.
To do so, we vary the number of latent dimensions d from 2 to 128 for each given dataset.
In Figure we exhibit two phases in the performance of GANs when changing the number of latent dimensions: ﬁrst, when d is below a dataset-speciﬁc threshold, the precision of the model falls when reducing the number of latent dimensions.
When d is above this threshold, the precision becomes constant, and increasing the number of latent dimensions does not bring any apparent improvement.
This threshold is a function of the complexity of the dataset and its number of modes.
We observe from Figure 6, that the more complex the dataset, the more it requires a large latent space for high precision levels.
This is coherent with the theoretical results, where the precision decreases w.r.t.
to the number of modes in (cid:112) log ( m ) when m (cid:54) d + 1.
Disentangling the manifold dimension.
As discussed in Roth et al.
(2017), two different problems can arise when training GANs: i) dimensional misspeciﬁcation where the true and modeled distributions do not have density functions w.r.t.
the same base measure, and ii) density misspeciﬁcation , where GANs try to ﬁt a disconnected manifold with a unimodal disitribution.
To isolate the density misspeciﬁcation studied in the current paper, we train a conditional GAN with a low-dimensional latent space R d ( e.g. R 5 in our setting), so that the dimension of the generated manifold is at most 5.
We later collect a dataset of synthetic generated samples Synthetic CIFAR-10 , and train unconditional
GANs by varying the dimension of the latent space.
Figure 6 shows both the Synthetic CIFARand the standard CIFAR-10 converge to the same limits for FD, Precision and Density, showing that with large latent space dimensions, the density misspeciﬁcation seems to be the main issue to cope with.
A synthetic experiment showing the importance of density misspeciﬁcation over dimensional misspeciﬁcation is available in appendix.
Impact of overparametrization.
Balaji et al.
(2020) already showed the importance of GANs’ overparametrization in both their convergence and performance.
Knowing that, we study whether overparametrization can help GANs obtain the optimal geometry of latent space.
In Table 2, we vary the width of ResNet generators, and highlight that overparametrized GANs better ﬁt the target distribution.
More importantly, we observe that overparametrization helps achieving better linear separability of their latent space, as shown by LogReg Accuracy .
[Table]
C ONCLUSION
This paper aims to make a step toward a better understanding of GANs learning disconnected distributions.
When the latent space dimension is large enough, we present an optimal latent space geometry of GANs: ‘simplicial clusters’, a Voronoi partition where each cell is a convex cone.
We further show experimentally that GANs with sufﬁcient latent capacity tend to respect this optimal geometry.
We believe that our analysis can foster exciting research on GANs, with both theoretical and practical impacts.
For example, understanding the optimal latent space’s geometry could help design semi-supervised or transfer algorithms from GANs.
Also, it could inspire new neural architectures with a bias for this ‘simplicial cluster’ partitioning of the latent space.
Finally, let us note that our results could potentially be extended to other types of generative models with Gaussian latent space and, thus, would allow a better understanding of their properties.
To adapt our analysis for variational auto-encoders or diffusion models, one would need to adapt our results to a stochastic generator.
This could be an exciting follow-up of our work.
Limitations.
We showed the existence of optimal generators and have shown experimentally that overparametrization plays a key role.
However, a limitation of our work is that we could not prove their uniqueness.
This is linked to partitions with the lowest ε -boundaries measures in the Gaussian space, which is a complex, unknown result.
A second limitation is that the derived optimal generators are not valid in the case m > d + 1, because the minimizers of Gaussian isoperimetric inequalities are not known in this case.
Potential negative societal impacts.
This work is mainly about understanding the behavior of deep generative models.
Thus, it may lead to practical improvements in this technology and increase its potential negative impacts, such as deepfakes (Fallis, 2020).
A I MPLEMENTATION DETAILS
[Table]
Training.
We use the Wasserstein loss with gradient-penalty on interpolations of fake and real data.
At each iteration, the discriminator is trained 2 steps and the generator 1 step with Adam optimizer.
The batch size is 256.
The learning rate of the discriminator is two times larger (Heusel et al., 2017), i.e. 5 × 10 − 5 for the generator and 1 × 10 − 4 for the discriminator.
GANs are trained for 80k steps on MNIST and for 100k steps on CIFAR datasets.
Architectures of generator and discriminator are described in Table 3 and Table 4.
Evaluation.
Features are extracted with a classiﬁer with simple architecture (convolutions, relu activation, no batch normalization).
The classiﬁer is trained on each dataset with cross-entropy loss.
Weights of the classiﬁers are given in the code.
For evaluation metrics, we follow the setting proposed by the authors.
For FID (Heusel et al., 2017), we use 50k real images and 50k fake images.
For precision, recall, density and coverage (Kynkäänniemi et al., 2019; Naeem et al., 2020), we use 10k real images and 10k fake images with nearest-k = 5.
Full results of the study on latent space dimension are presented in Table 5.
We also share the code for a better reproducibility.
GPUs.
For all datasets, the training of GANs was run on NVIDIA TESLA V100 GPUs (16 GB).
The training of a GAN for 100k steps on CIFAR takes around 30 hours.
[Table]
[Table]
B T ECHNICAL RESULTS B. P ROOF OF L EMMA
We want to show that generator G ∈
G A L is such that α G (cid:54) 1 − γ ( ∂ ε min A ) , where
[Equation]
Proof by contradiction .
Assume a generator G such that there exists z ∈ ∂ ε min A and i ∈ [ 1 , m ] such that G ( z ) ∈ M i .
Since G is associated with A , we have using Deﬁnition 2, that there exists z (cid:48) and j ∈ [ 1 , m ] , j (cid:54)
=
i such that (cid:107) z − z (cid:48) (cid:107) <
ε min / 2 and j = argmin k ∈ [ 1 , m ] (cid:107) G ( z (cid:48) )
− M k (cid:107) .
Thus, we have: (cid:107) G ( z ) − G ( z (cid:48) ) (cid:107) (cid:62) d ( G ( z (cid:48) ) , M i ) , (cid:62) d ( M i , M i ) / 2 , (cid:62) D min / 2 .
And, (cid:107) G ( z ) − G ( z (cid:48) ) (cid:107) (cid:107)
z − z (cid:48) (cid:107) >
D min / ε min , > L .
This contradicts G being in G A L .
B. P ROOF OF T HEOREM
Proving that: for m (cid:54) d + 1 , for any δ > 0 , if L is large enough, then, for any well-balanced
G ∈ G L , we have α G (cid:54) 1 − γ ( ∂ ε min A (cid:63) )
+ δ .
Let G be a well-balanced generator and A the partition associated with G .
Let us ﬁrst deﬁne the gaussian boundary measure P γ of a partition A of R d .
For partitions with smooth boundaries, it coincides with the ( d − 1 ) -dimensional gaussian measure of the boundary, deﬁned as follows:
[Equation]
Moreover, for sets with smooth boundaries, we have from Federer (1969, Theorem 3.2.29):
[Equation]
Let us denote A (cid:63) , the optimal partition deﬁned in Milman and Neeman (2022), based on simplicial clusters.
A (cid:63) is a standard partition where γ ( A (cid:63) 1 ) = ...
= γ ( A (cid:63) m ) for all i, and ∑ i γ ( A i )
= 1.
By the multi-bubble theorem (Milman and Neeman, 2022), simplicial clusters (such as A (cid:63) ) are the unique minimizers of the gaussian isoperimetric problem, thus:
[Equation]
where L A = lim
[Equation]
ε and L A (cid:63) =
lim ε → 0 γ ( ∂ ε A (cid:63) ) ε .
Then, for any δ > 0, there exists ε (cid:48) > 0 such that, for any ε < ε (cid:48) ,
[Equation]
− L A | < δ and L A (cid:63) (cid:54)
L A
Thus, for any δ > 0, there exists ε (cid:48) > 0 such that, for any ε < ε (cid:48) ,
[Equation]
Besides, we know that
[Equation]
Consequently, we have that:
[Equation] [Equation]
We conclude by choosing L big enough such that ε min is strictly smaller than δδ / 2 .
Second part of Theorem 1.
Let L , D be such that L (cid:62) D (cid:112) log ( m ) .
Let’s prove that for any wellbalanced generator G ∈ G L , we have:
[Equation]
Using the method from Schechtman (2012)
[Equation]
we have the measure of the border of cell i :
[Equation]
where t is such that
[Equation]
Thus:
[Equation]
m
[Equation]
Thus, we have
[Equation] [Equation]
Moreover, using ε min = DL and L (cid:62) D √ log m , so we get ε min √ log m (cid:54) 1: α G (cid:54) 1 − ε min (cid:112) log m e
[Equation]
[Equation]
B. P ROOF OF T HEOREM
For a given partition A , and a target distribution µ (cid:63) with m disconnected components M i , i ∈ [ 1 , m ] , we deﬁned X i , i ∈ [ 1 , m ] a set of sampled data points such that for all i ∈ [ 1 , m ] , we have X i ∈ M i .
Now, we deﬁne G (cid:63) ε with ε > 0, a generative model such that:
[Equation]
w i
[Equation]
where d ( z , A ) = min a ∈ A (cid:107) z − a (cid:107) , and S z = { j ∈ [ 1 , n ] such that z ∈
A ε j } denotes the set of cellextensions the point z belongs to.
We can see that G (cid:63) ε (cid:93) γ memorizes the dataset since every z close to the center of the cell
A i such that | S z | = 1 veriﬁes G (cid:63)
ε ( z ) =
X i .
An illustration is given in
Figure 2.
To be more precise, all samples are mapped to one of the center of the modes
X i , i ∈ [ 1 , m ] , except for those in ε -boundaries.
When z belongs to the intersection of two ε -boundaries, G ε ( z ) is a simple linear combination of 2 points.
It is only when | S z | (cid:62) 3 that more complex samples are generated.
A simple illustration of G (cid:63) ε for d = 2 and m = 3 is given in Figure 2.
Interestingly, one can also show that the image of G (cid:63) ε is equal to the convex hull of the diracs
X i , i ∈ [ 1 , m ] .
In particular, there exists a particularly interesting neighborhood ν of 0 where G (cid:63) ε ( ν ) is equal to the whole convex hull of the points
X i , i ∈ [ 1 , m ] .
Proof that G (cid:63) ε is well-balanced.
We recall that a generator is well-balanced if we have G (cid:93) γ ( M 1 )
= ...
= G (cid:93) γ ( M m ) .
By construction (8), we have that for any i ∈ [ 1 , m ]
[Equation]
So, for any z ∈ A i , we have that
[Equation]
Thus G (cid:63) ε is associated with the optimal partition A (cid:63) , .
Besides, for a given radius r of the different modes, since everything is symmetrical, we have that γ ( { z ∈ R d , (cid:107) G ( z ) − X 1 (cid:107) (cid:54) r } = ...
=
γ ( { z ∈ R d , (cid:107) G ( z ) − X m (cid:107) (cid:54) r } .
Thus, the generator is well-balanced.
Proof that G (cid:63) ε max is in G L .
To begin with, we have R d = ∂
ε A (cid:83) (cid:0) R d \ ∂ ε A (cid:1) .
[Equation]
(cid:91) i =
[Equation]
\ (cid:0) ( ∪ j (cid:54) =
i
[Equation]
j ) ε ∩
A i (cid:1) , and, we know that for each i ∈ [ 1 , m ] , G (cid:63) ε
( z ) is constant and thus L -Lipschitz.
Now,
[Equation]
A
[Equation]
Now, let S ∈ P
[ 1 , m ]) with card ( S )
=
k (cid:62) 2.
Let z , z (cid:48)
∈ S 2 .
Let α =
( α 1 ,..., α m ) and β = ( β 1 ,..., β m ) be two vectors, both in R m , such that for all i ∈ [ 1 , m ] :
[Equation]
and
[Equation]
i
[Equation]
z
[Equation]
We have that
[Equation]
where h is the function from R d → R m deﬁned as:
[Equation]
We can write h = f ◦ g with f the function deﬁned from R d → R m by
[Equation]
and g the function deﬁned on R m \{ 0 } by
[Equation]
We have that f is a √ m -Lipschitz functions (given that z (cid:55)→ d ( z , ( A ε m ) (cid:123) ) is 1-Lipschitz).
Besides, we know that outside the ball B ε ( 0 ) , the function g is ( 1 / ε ) -Lipschitz.
Using the convexity of function
z (cid:55)→ ∑ j ∈
A z d ( z , ( A ε j ))
(as a sum of convex functions), we can show that for all z ∈
A z , we have that f ( z ) (cid:54) ( m − 1 ) ε and f ( z ) is not B ε ( 0 ) .
Finally, the function h is √ m ε -Lipschitz.
Thus, we have that:
[Equation]
with D = (cid:107) X
i − X j (cid:107) , ( i , j ) ∈ [ 1 , m ] 2 , i (cid:54) =
j .
Consequently, by noting ε max = D √ mL , we have :
[Equation]
We can now conclude on the Lipschitzness of G (cid:63) on R d .
Proving that: for m (cid:54) d + 1 , for any δ > 0 , if L is large enough, then, for any well-balanced
G ∈ G L , we have α G (cid:63) ε max (cid:62) α G − δ .
From the proof of Theorem 1, we have that for any δ > 0, there exists ε min such that:
[Equation]
using (7) .
Now, by construction of G (cid:63) ε max , we have that
[Equation]
Consequently,
[Equation]
We conclude by choosing L big enough such that ε max is strictly smaller than
[Equation]
Proving the second part of Theorem 2.
The precision of G (cid:63) ε is thus such that:
[Equation]
However, since ∂ ε A ⊂ (cid:83) ni =
A ε i
[Equation]
we have that for any ε
[Equation]
Using results from Schechtman (2012, Proposition 1)
[Equation]
when m ≤ d
[Equation]
there exists C large enough
[Equation]
such that
[Equation]
Thus, we have α G (cid:63) ε max (cid:62) 1 − ε max (cid:112) π log ( Cm ) ,
To have α G (cid:63) ε max (cid:62) 0, we must have ε max (cid:54) 1 / (cid:112) π log ( Cm ) .
This is the case since we have ε max = D √ m / L and L (cid:62) D √ m (cid:112) π log ( Cm ) .
C A DDITIONAL RESULT
Impact of the number of modes.
To illustrate the results from Theorem 1 and Theorem 2, we propose to vary the number of modes of the data distribution.
On real-world data, the number of modes is set but usually unknown, and removing/adding classes as a proxy for modes usually does not give insightful results since some classes can be much more complex than others.
We thus use a synthetic setting, where we can easily control both the number of modes and their complexity.
Figure 7 stresses that as the number of modes increase, the precision decrease.
Interestingly, using large latent space dimension can relieve the problem, even if the latent space dimension is clearly below that of the target.
Recall the two problems that arise when training GANs: i) dimensional misspeciﬁcation where the true and modeled distributions do not have density functions w.r.t.
the same base measure, and ii) density misspeciﬁcation , where GANs try to ﬁt a disconnected manifold with a unimodal disitribution.
From the results we conclude that:
• With very low latent space dimensions, both problems i) and ii) have to be addressed and this leads to poor precision as the number of modes increases.
• With larger latent space dimensions, the problem i) is less of a burden even when there is a clear dimensional misspeciﬁcation and thus the GANs’ performance is more tied to problem ii).