AE-OT: A NEW G ENERATIVE M ODEL BASED ON EXTENDED SEMI - DISCRETE OPTIMAL TRANSPORT
Anonymous authors
Paper under double-blind review
A BSTRACT
Current generative models like generative adversarial networks (GANs) and variational autoencoders (VAEs) have attracted huge attention due to its capability to generate visual realistic images.
However, most of the existing models suffer from the mode collapse or mode mixture problems.
In this work, we give a theoretic explanation of the both problems by Figalli’s regularity theory of optimal transportation maps.
Basically, the generator compute the transportation maps between the white noise distributions and the data distributions, which are in general discontinuous.
However, deep neural networks (DNNs) can only represent continuous maps.
This intrinsic conﬂict induces mode collapse and mode mixture.
In order to tackle the both problems, we explicitly separate the manifold embedding and the optimal transportation; the ﬁrst part is carried out using an autoencoder (AE) to map the images onto the latent space; the second part is accomplished using a GPU-based convex optimization to ﬁnd the discontinuous transportation maps.
Composing the extended optimal transport (OT) map and the decoder, we can ﬁnally generate new images from the white noise.
This AE-OT model avoids representing discontinuous maps by DNNs, therefore effectively prevents mode collapse and mode mixture.
I NTRODUCTION
Generative adversarial networks (GANs) (Goodfellow et al.
and variational autoencoders (VAEs) (Kingma & Welling (2013)) emerge as the dominant approaches for unconditional image generation.
When trained on appropriate data sets, they are able to produce realistic and visual appealing samples.
GAN methods train an unconditional generator that regresses real images from random noise and a discriminator that measures the difference between generated samples and real images.
Despite GANs’ advantages, they have critical drawbacks.
1) Training of GANs are tricky and sensitive to hyperparameters.
2) GANs suffer from mode collapse, in which the generator only learns to generate few modes of data distribution while missing others, although samples from the missing modes occur throughout the training data (see e.g. Goodfellow (2016)).
While for the VAEs, the encoder is used to map the data distribution to a Gaussian latent distribution, which is then mapped back to the data distribution by the decoder.
While standard VAEs tend to capture all modes, they often generate ambiguous images on multimodal real data distributions.
These phenomena can be explained as follows:
Manifold Distribution Hypothesis In deep learning, the manifold distribution hypothesis is well accepted, which assumes the distribution of a speciﬁc class of natural data is concentrated on a low dimensional manifold embedded in the high dimensional data space Tenenbaum et al.
Therefore, GANs and VAEs implicitly aim to accomplish two major tasks: 1) manifold embedding: to ﬁnd the encoding/decoding maps between the data manifold embedded in the image space and the latent space; 2) probability distribution transport: to transport a given white noise distribution to the data distribution, either in the latent or in the image space.
Distribution Transformation
The generator of GAN model and the decoder of VAE model are trained to compute a transport map that transforms a known continuous distribution (e.g. Gaussian white noise) to a distribution that aligns well with the empirical real data distribution.
Namely, the transport map pushes forward the given noise distribution to a generated distribution to approximate the real data distribution, the similarity between the two distributions determines the generalization ability of the generator Ben-David et al. (2010). f 1 , ˆ ffˆ fffˆ f [Figure]
Discontinuity and Mode Collapse/Mixture
It is a common practice among GAN/VAE models that the generators/decoders are expressed by deep neural networks, which can only represent continuous mappings.
Unfortunately, as pointed out by works Nagarajan & Kolter (2017);
Khayatkhoei et al.
Xiao et al.
(2018), the transport maps may be discontinuous when there are multiple modes in the data distribution.
This intrinsic conﬂict can cause mode collapse or mode mixture.
The later means that the generated samples is a mixture of multiple modes and fall into the gap among the modes.
Currently, we discover a novel phenomena that even the real data distribution has a single mode , ambiguous data (e.g. a human face image with mismatched eye colors) can still present.
This can be explained by Brenier’s polar factorization theorem Brenier (1991b; 1987; 1991a) and Figalli’s regularity theorem Figalli (2010); Chen & Figalli (2017)
(Thm. 5 in Appendix B), which asserts that if the support of the target distribution is not convex, then there will be singularity sets on the support of the source distribution, such that the transport map is discontinuous on these sets.
This shows the intrinsic difﬁculties of conventional GANs/VAEs cannot be eliminated, as shown in Fig.
Conquering Mode Collapse/Mixture
However, according to Brenier (1987; 1991a) theorem, the optimal transport map can be represented as the gradient map of the Brenier potential.
At the regular points, the Brenier potential is differentiable, its gradient map (the transport map) is continuous; at the singularities, the Brenier potential is continuous but not differentialbe, and its gradient map is discontinuous.
Conventional GANs and VAEs model the gradient map directly and encounter the trouble of discontinuity.
In contrast, we propose to model the globally continuous Brenier potential to avoid mode collapse/mixture.
More speciﬁcally, our proposed AE-OT model separates the manifold embedding step and the probability distribution transformation step, the former is carried out by an autoencoder (AE), the latter is accomplished by a convex optimization framework (OT).
The OT step computes the Brenier potential explicitly and is able to locate the singularity set (the discontinuous points of the gradient map) based on Figalli’s theory.
Our experimental results demonstrate that the proposed method can not only cover all of the modes, but also avoid generating spurious samples (mode mixture).
Contributions (i) From theoretical aspect, this work gives a thorough explanation of mode collapse and mode mixture by the regularity theory of optimal transportation developed by Figalli (
Fields medalist) and the reasons why standard GANs/VAEs cannot solve this problem perfectly.
(ii) From practical aspect, this work proposes a novel model called AE-OT, which ﬁrst encodes the data manifold into the latent space, then compute the Brenier potential to represent the optimal transportation map in the latent space.
The Figalli’s singularity set can be located efﬁciently and avoided when generating new samples.
In this way, our model eliminates mode collapse and mode mixture successfully.
(iii)
The algorithm for ﬁnding the Brenier potential and the optimal transportation map can be accelerated with GPU based convex optimization algorithm.
The method converges to the unique global optimum with bounded error estimate.
(iv) Our experiment results demonstrate the efﬁciency and efﬁcacy of the proposed method.
R ELATED W ORK
Optimal Transport Optimal transport plays an important role in various engineering ﬁelds.
For more thorough reviews, we refer the readers to Peyré & Cuturi (2018) and Solomon (2018).
In
Gu et al.
(2016), the intrinsic connection between Brenier theory in OT and Alexandroff theory in convex geometry was established, and applied for deep learning in Lei et al.
(2017) by an convex optimization.
Figalli and the collaborators Figalli (2010); Chen & Figalli (2017) proposed that when the support of the data distribution is non-convex, the transport map will be discontinuous.
Generative models In machine learning, generative models have been becoming more important and popular recently.
A huge breakthrough for image generating comes from the scheme of Variational
Autoencoders (VAEs) (e.g. Kingma & Welling (2013)), where the decoders approximate real data distributions from a Gaussian distribution in a variational approach (e.g Kingma & Welling (2013) and Rezende et al.
Various recent works followed this scheme, including Adversarial
Autoencoders (AAEs) Makhzani et al.
and
Wasserstein Autoencoders (WAEs) Tolstikhin et al.
Although VAEs are relatively simple to train, images they generate look blurry.
Generative
Adversarial Networks (GANs) Goodfellow et al.
(2014) were proposed to solve this disadvantage.
While being a powerful tool in generating realistically looking samples, GANs can be hard to train and suffer from mode collapsing.
Various improvements have been proposed for better training of GANs, including changing the loss function (e.g. Wasserstein GAN Arjovsky et al.
, regularizing the discriminators to be Lipschitz (clipping Arjovsky et al.
(2017), gradient regularization Gulrajani et al.
(2017), Mescheder et al.
(2018) or spectral normalization Miyato et al.
Besides, various non-adversarial methods has also been proposed recently.
GLO Bojanowski et al.
(2017) employs an “encoder-less autoencoder" approach where a generative model is trained with a non-adversarial loss function.
IMLE Li & Malik (2018) proposed an ICP related generative model training approach.
Later GLANN Hoshen & Malik (2019) combines advantages of GLO and
GLANN, where an embedding from image space to latent space was ﬁrst found using GLO and then a transformation between an arbitrary distribution and latent code was computed using IMLE.
Mitigating Mode Collapsing Recently, Nagarajan & Kolter (2017);
Khayatkhoei et al.
Xiao et al.
(2018) also realize the training difﬁculties of GANs come from the approximation of discontinuous functions with continuous DNNs.
By the gradient-based regularization, GDGAN
Nagarajan & Kolter (2017) do relieve the mode collapse phenomenon of GANs, but mode mixture still exists.
Khayatkhoei et al.
(2018) proposes to use multiple GANs to overcome the mode collapse.
Xiao et al.
(2018) proposed to embed the images into a latent space according to Bourgain’s theorem, and train the generator by sampling a Gaussian mixture distribution in the latent space instead of a unimodal Gaussian.
The recently introduced normalized diversiﬁcation by Liu et al.
(2018) can also help overcome mode collapse successfully.
However, all of them cannot solve the mode mixture well.
All these works Nagarajan & Kolter (2017);
Khayatkhoei et al.
Xiao et al.
(2018) explain that if the target data distribution has multiple modes, the transport map is discontinuous, but DNNs can only represent continuous mappings, the intrinsic conﬂict causes mode collapse.
C OMPUTATIONAL A LGORITHMS
Overview of AE-OT Model Our AE-OT model is summarized in Fig.
2, it has two major components: i) (AE)
An autoencoder is trained to encode ( f θ ) the data manifold from the image space X to the latent space Z , and map the data distribution to the latent code distribution; then the decoder g ξ decodes the latent code back to the data manifold.
ii) (OT)
This module computes the optimal transportation map T from the noise distribution to the latent code distribution.
First, the Brenier potential is found by a convex optimization process according to Gu et al.
(2016), whose gradient is the semi-discrete optimal transport map, where the target is the discrete set of latent codes of training samples; then the transport map is piece-wise linearly extended to a global continuous map ˜ T , where the target becomes a polytope obtained by triangulating the above latent codes. Finally, the
[Figure]
singularity set in the source domain is located and avoided when generating new samples.
As a result, given a random noise x , we can get the generated image by g ξ ◦ ˜ T ( x ) .
Semi-Discrete OT Map Suppose the source measure µ (Gaussian or uniform distribution) is absolutely continuous deﬁned on a convex domain Ω ⊂ R d , the target domain is a discrete set,
Y = { y 1 , y 2 , · · · , y n } , y i ∈
R d , the target measure is a Dirac measure, ν = (cid:80) ni
=
ν i δ ( y − y i ) , i = 1 , 2 , . .
. , n , with the equal total mass as the source measure, µ (Ω)
=
(cid:80) ni
=
ν i .
Under a semidiscrete transport map T : Ω → Y , a cell decomposition is induced Ω
=
(cid:83) ni
=1 W i , such that every x in each cell W i is mapped to the target
y i , T : x ∈ W i (cid:55)→ y i .
The map T is measure preserving, denoted as T # µ = ν , if the µ -volume of each cell W i equals to the ν -measure of the image T ( W i )
=
y i , µ ( W i ) =
ν i .
The cost function is given by c : Ω × Y → R , where c ( x, y ) represent the cost for transporting a unit mass from x to y .
The total cost of T is given by (cid:82) Ω c ( x, T ( x )) dµ ( x )
= (cid:80) ni
=1 (cid:82) W i
c ( x, y i ) dµ ( x ) .
Semi-discrete optimal transport map is the measure-preserving map that minimizes the total cost, T ∗ := arg min T # µ = ν (cid:82) Ω c ( x, T ( x )) dµ ( x ) .
When the cost function is the L 2 distance c ( x, y )
= 1 / 2 (cid:107) x − y (cid:107) 2 , Brenier’s theorem claims that the semi-discrete OT map is given by the gradient map of a piece-wise (PL) convex function, the so-called Brenier potential u h : Ω → R , u h ( x ) :=
max ni =1 { π h,i ( x ) } , where π h,i ( x ) = (cid:104) x, y i (cid:105)
+ h
i is the hyperplane corresponding to y i ∈ Y .
As shown in Fig. 3(a), the projection of the graph of u h decomposes Ω into cells W i ( h ) , each cell W i ( h ) is the projection of the supporting plane π h,i ( x ) .
The height vector h is the unique optimizer of the following convex energy under the condition that (cid:80) i h i = 0 ,
E ( h ) =
(cid:90)
h n (cid:88) i =
w i ( η ) dη i − n (cid:88) i
= h
i ν i , (1) where w i ( η ) is the µ -volume of W i ( η ) .
The convex energy E ( h ) can be optimized simply by gradient descend method with ∇ E ( h ) =
( w i ( h ) − ν i ) T .
The key is to compute the µ -volume w i ( h ) of each cell
W i ( h ) , which can be estimated using conventional Monte Carlo method.
We draw N random samples from µ distribution, { x j } ∼ i.i.d.
µ , ∀ j ∈ J , the estimated µ -volume of each cell is ˆ w i ( h ) =
# { j ∈ J | x j ∈ W i
( h ) } /N .
Given x j , we can ﬁnd W i in which x j ∈ W i by i =
arg max i {(cid:104) x
j , y i (cid:105) + h i } , i = 1 , 2 , . . . , n .
When
N is large enough, ˆ w i ( h ) converges to w i ( h ) .
Then the gradient of the energy is approximated as ∇ E
≈
( ˆ w i ( h ) − ν i ) T .
Once the gradient is estimated, we can use Adam algorithm Kingma & Ba (2015) to minimize the energy.
Sampling of x is independent of each other and ﬁnding the cell that x is located only involves matrix multiplication and sorting.
Hence the Monte Carlo method has a natural parallel computation implementation on GPUs.
[Figure]
The approximation error is proportional to the inverse of the square root the amount of Monte Carlo samples.
Asymptotically the number of Monte Carlo samples increases exponentially with respect to the dimension d (see e.g. Weed & Bach (2017)).
This brings huge computational burdens.
To ﬁnd a good balance between precision and speed, we adaptively adjust the number of random samples.
In practice, we apply the following strategy: if the energy E ( h ) ceases decreasing for a number of consecutive steps, we double the amount of Monte Carlo samples.
The algorithmic details of semi-discrete OT map are summarized in Alg. 1. Algorithm
1: Input: Latent codes Y = { y i } i ∈I , empirical latent code dis- tribution ν = 1 |I| (cid:80) i ∈I δ yi ,numberofMonteCarlosamples
N , positive integer s .
2: Output: Optimal transport map T ( · ) .
3: Initialize h = ( h 1 ,h 2 ,...,h |I| ) ← (0 , 0 ,..., 0) . 4: repeat 5: Generate N uniformly distributed samples { x j } Nj =1 .
Calculate ∇ h
= ( ˆ w i ( h ) − ν i ) T . 7: ∇ h = ∇ h − mean ( ∇ h ) .
Update h by Adam algorithm with β 1 = 0 .
9 ,β 2 = 0 .
9: if E ( h ) has not decreased for s steps then 10:
N ← N × 2 .
11: end if 12: until Converge 13: OT map T ( · ) ← ∇ (max i (cid:104)· ,y i (cid:105)
+ h i ) .
1: Input: Optimal transport map T ( · ) , number of samples to gen- erate n , angle threshold ˆ θ . 2: Output: Generated latent code P .
Compute ˆ c i by Monte Carlo method.
4: repeat 5:
Sample x ∼ µ , Find the smallest d + 1 vertex around x as { d ( x, ˆ c i 0 ) ,d ( x, ˆ c i 1 ) ,...,d ( x, ˆ c id ) } . Compute dihedral angles θ ik between π i 0 and π ik . 7: Select θ ik with θ ik ≤ ˆ θ , result in ˆ i k = 0 , 1 ,...,d 1 .
8: if ∀ k,θ ik >
{+ˆ θ then Abandon x 9: else Generate latent code ˜ T ( x ) = (cid:80) d 1 k =
 λ k T (ˆ c ˆ ik ) with λ k = d − 1 ( x, ˆ c ˆ ik ) / (cid:80) d 1 j =
 d − 1 ( x, ˆ c ˆ ij ) . 10: end if 11: until Generate n new latent code+}
Piece-wise Linear Extension
The semi-discrete OT map ∇ u h : Ω → Y maps all x ∈ Ω to the latent codes of training samples { y i } ’s and won’t generate new samples.
Therefore, we extend the semi-discrete OT map T = ∇ u h to a piecewise linear (PL) mapping ˜ T as follows.
The projection of u h induces a cell decomposition of Ω , of which each cell is of µ -volume
ν i and is mapped to the corresponding y i .
By representing the cells by their µ -mass centers as c i := (cid:82) W i ( h ) xdµ ( x ) , we can get the point-wise map t : c i
(cid:55)→ y i .
The Poincaré of the cell decomposition induces a triangulation of the centers C
= { c i } : if W i ∩ W j (cid:54) = ∅ , then c
i is connected with c j to form an edge [ c i , c j ] .
Similarly, if W i 0 ∩ W i 1 · · · ∩ W i k (cid:54) = ∅ , then there is a k -dimensional simplex
[ c i 0 , c i 1 , . . .
, c i k ] .
All these simplices form a triangulation of C (a simplicial complex), denoted as T ( C ) (the green triangles in the left of Fig. 3(b)).
We can triangulate Y in the same way to obtain the triangulation
T ( Y ) (the green triangles in the right of Fig. 3(b)).
Once a random sample x is drawn from the distribution µ , we can ﬁnd the simplex σ in T ( C ) containing x .
Assume the simplex σ has d +vertices { c i 0 , c i 1 , . .
, c i d } , the bary-centric coordinates of x in σ is deﬁned as x = (cid:80) dk
=
λ k c
i k , and (cid:80) dk
=
λ k
= 1 with all λ k non-negative.
Then the generated latent code of x under the PL OT map is given by ˜ T ( x )
= (cid:80) dk
=
λ k
y i k
(In Fig. 3(b), the green dot x 1 is mapped to be ˜ T ( x 1) ).
Because all of the y i s are used to construct the simplicial complex T ( Y ) in the support of the target distribution, we can guarantee that no mode is lost.
In practice, the µ -mass center c i is approximated by the mean value of all the Monte-Carlo samples inside W i ( h ) ,
ˆ c i = (cid:80) x j ∈ W i x j / # { x j ∈ W i } , where x j ∼ µ .
The connectivity information T ( C ) is too complicated to construct and to store in high dimensional space, thus T ( C ) is not explicitly built.
Instead, we ﬁnd the simplex σ ∈ T ( C ) containing x as follows: given a random point x ∈ Ω , evaluate and sort its Euclidean distances to the centers d ( x, ˆ c i ) , i = 1 , 2 , . . .
, n in the ascending order.
Suppose the ﬁrst d + 1 items are { d ( x, ˆ c i 0 ) , d ( x, ˆ c i 1 ) , . . .
, d ( x, ˆ c i d ) } , then σ is formed by { ˆ c i k } .
The bary-centric coordinates ˆ λ i k are estimated as ˆ λ i k
= d − 1 ( x, ˆ c i k ) / (cid:80) dk
=
d − 1 ( x, ˆ c i k ) .
However, this may generate some spurious samples.
To overcome it, we need further to detect the singular set.
Singular Set Detection
According to Figalli’s theory Figalli (2010); Chen & Figalli (2017), if there are multiple modes or the support of the target distribution is concave, there will be singular sets Σ ⊂ Ω , where the Brenier potential is continuous but not differentiable, making its gradient map, i.e. the transport map, discontinuous.
As shown in Fig. 3(a), the source distribution is uniformly deﬁned on Ω , and the target empirical distribution has two modes.
There is one ridge (the red line) on the Brenier potential u h , whose projection is the singular set Σ (the red line in Ω ).
Ω \ Σ consists of two connected components, each of them is mapped onto a single mode.
Σ consists of codimension 1 facets of cells.
If
W i ( h ) ∩ W j ( h ) ⊂ Σ , then the dihedral angle between two supporting planes π h,i and π h,j of u h is prominently large.
Therefore, on the graph of Brenier potential, we pick the pairs of facets whose dihedral angles are larger than a given threshold, the projection of their intersection gives a co-dimension 1 cell in the singular set Σ .
During the generation process, if a random sample x is around Σ , it will be mapped by ˜ T to the gaps among the modes.
When generating new latent codes, we just abandon such samples.
And this help our method prevent the mode mixture phenomenon.
Given the extended OT map ˜ T ( x ) , some of the polyhedrons transverse the singular set (the red lines of Fig .3(b)), which means that different vertices of the polyhedron belongs to different mode.
If the sample x falls into such a polyhedron (the dotted red triangle), we just abandon it (as shown in Fig.
3(b)
, the red dot x 2 is just abandoned).
Speciﬁcally, given x , we can detect if it belongs to the singular set by checking the angles θ i k between π i 0 and
π i k , k
= 1 , 2 , . . .
, d as θ i k
= (cid:104)
y
i 0 , y i k
(cid:105) / (cid:107) y
i 0 (cid:107) · (cid:107) y
i k (cid:107) .
If all of the angles θ i k is larger than a threshold ˆ θ , we say x belongs to the singular set and just abandon it.
Or we just select a subset { π i k } with θ i k ≤ ˆ θ , denoted as { π ˆ i k , k = 0 , 1 , . . .
, d 1 } .
Then we can compute λ k
= d − 1 ( x, ˆ c ˆ i k ) / (cid:80) d 1 j =
d − 1 ( x, ˆ c ˆ i j ) and ˜ T ( x ) = (cid:80) d 1 k =
λ k T (ˆ c ˆ i k ) .
Intuitively, ˜ T ( · ) smooths the discrete function T ( · ) in regions where latent codes are dense and keep the discontinuity of T ( · ) where latent codes are very sparse.
In this way we avoid generating spurious latent code and thus improve the generation quality.
The algorithm to generate new code is shown in Alg.
2 and the effect of threshold ﬁltering is further investigated in Appendix C.1.
E XPERIMENTS
In order to validate that the proposed method can solve the mode collapse/mixture problems and generate controllable high quality images, several experiments are conducted.
The ﬁrst experiment focuses on toy sets, so that the complexity of the tasks can be manually controlled and the mode and quality of the generated samples can be accurately computed.
Lin et al.
(2018) did a large-scale comparison with previous methods that explicitly proposed to mitigate mode collapse and thus established a baseline for comparison.
For consistent evaluation, we set up our experiment on the same benchmark dataset as theirs, and make the comparison.
In the second experiment, we run the proposed method mainly on 4 public datasets, MNIST LeCun & Cortes (2010), MNIST-FANSION Han Xiao & Vollgraf (2017), CIFAR-10 Krizhevsky (2009) and
CelebA Zhang et al.
(2018), just like the authors of Hoshen & Malik (2019)
Sajjadi et al.
Lucic et al.
(2018) did in their papers.
Besides, the architecture of the decoder is the same as Lucic et al.
(2018), in which the authors did a large-scale study to evaluated the best performance ofdifferent generative models including various GAN models and VAE, and the encoder is set to be the mirror of decoder.
The training details, parameter setting and time consuming information are introduced in Section C.5 of the Appendix.
4. M ITIGATION OF MODE COLLAPSE AND MODE MIXTURE IN SYNTHETIC DATASET
Since synthetic dataset consists of explicit distributions and known modes, mode collapse and the quality of the generated sample can be accurately measured.
We choose the same synthetic datasets [Table] and metrics as in Lin et al.
Speciﬁcally, we use 2D-ring and 2D-grid for test sets and Number of modes , Percentage of high-quality samples , reverse Kullback-Leibler (KL) divergence as evaluation metrics.
Number of modes counts the amount of modes captured by samples produced a generative model.
Percentage of high-quality samples measures the proportion of samples that generated within three standard deviations of the nearest mode.
reverse KL divergence measures how well generated samples balance among all modes regarding the real distribution.
In Lin et al.
, the authors evaluated GAN, ALI, MD and PacGAN on synthetic sets with above three metrics.
Each networks are trained under the same generator architecture with a total of approximated 400K training parameters.
For AE-OT test, since the source space and the target space are both 2-dimensional, there is no need to train any autoencoder.
A two dimensional extended OT map is directly computed.
Our results are included in table 1, and benchmarks of previous methods are copied from Lin et al.
(2018) and Xiao et al.
Generally speaking, the samples generated by the proposed method can not only cover all of the modes, the quality of them is also better than others.
Also, we plot our results on 2D-ring dataset along with those of GAN, PacGAN and BourGAN in Fig. 10 in C.2 of Appendix.
Besides, we experiment on stack MNIST dataset and CelebA dataset to further illustrate the performance of the proposed method, and the results are shown in Section C.3 and C.4 in the Appendix.
4. Q UANTITATIVE COMPARISON WITH FID
FID is computed by: (1) extract the visual-meaningful features of both the generated and real images through the inception network, (2) ﬁt the features in both the generated and real feature spaces with
Gaussian distribution, and then (3) compute the distance between the two Gaussian distributions with the following formula FID = (cid:107) µ
[Equation] − [Equation] (cid:107) 22 + Tr (Σ r + Σ g − 2(Σ r Σ g ) 12 ) , where µ r , µ g mean the means of the real and generated features, Σ r , Σ g represent the variances of both distributions.
We report our results in Tab.
(2), in which the compared data comes from Lucic et al.
(2018)Hoshen & Malik (2019).
In general, the proposed model achieves better than or comparable scores to other state-of-the-art generative models.
Theoretically, the FID scores of our proposed generative models should be close to that of the pre-trained autoencoders, and this is also validated in our experiments.
The autoencoder architecture here we use cannot ﬁnd a good encoding for the CelebA dataset due to the limited capacity.
But the FID score of the generation model is still approach to the autoencoder.
In order to verify that with appropriate capacity of autoencoder, the proposed model works.
We use the generator of DCGAN Radford et al. as the decoder of the autoencoder, then the reported
FID score is 28.6.
Also, some of the generated images are displayed in Fig.
12 of the Appendix.
We also display the generating results for the four dataset in Fig.
It includes the original images, the results of autoencoders, the best generating results of Lucic et al.
(2018), including various GANs and VAE, the results of Hoshen & Malik (2019) and ours, column by column.
Because of the one-one correspondence between µ -mass centers and the images, we show the linear interpolation results (Fig. 13 of Appendix) between two given images in the dataset . [Table]
Precision and recall proposed in Sajjadi et al.
(2018) can compute the precision and the recall at the same time only given the same number of generated and reference images.
In Section C.6, we report the comparsion results with state-of-the-art methods.
C ONCLUSION
This work gives a theoretic explanation for mode collapse/mixture by Brenier’s theory and Figalli’s regularity theory of optimal transport maps.
When the target measure has concave support, the OT map is discontinuous on the signular sets.
But DNNs can only represent continuous functions, this conﬂict causes the both problems.
In order to solve this problem, the AE-OT model is proposed by separating manifold embedding and measure transformation.
The former step is computed using an autoencoder, the latter is carried out using the extended semi-discrete OT map based on GPUs.
The model is tested thoroughly and extensively by both synthetic and real data sets.
The experimental results validates the discontinuity of the OT maps and demonstrate the advantages comparing to the state-of-the-arts.
B F IGALLI ’ S T HEORY
In this section, we show the fact that even for the case of single mode , the transport map may still be discontinuous , which will cause the instability of the training process of GANs.
The arguments are mainly based on the regularity theory of transport maps developed by Figalli Chen & Figalli (2017);
Figalli (2010) and so on.
According to Brenier’s Theorem Brenier (1987; 1991a), any transport map can be decomposed into a measure preserving map and a solution to the Monge-Ampére equation, which is the optimal transport map under the L 2 cost function.
Therefore, the continuity of the transport map can be reduced to the regularity (smoothness) of the solution to the Monge-Ampére equation.
When the support of the target measure is convex and the density functions are smooth, Caffarelli showed the map is differentiable; otherwise if the target domain is not convex, Figalli showed the map is discontinuous, and gave precise description of the singularity set.
In this section, we brieﬂy introduce Figallis’ theory, and conduct an experiment using CelebA data set to show the existence of the singularity set, hence demonstrate the fact that the transport maps computed in GANs are discontinuous.
B. C ONVEX D OMAINS - C AFFARELLI T HEOREM
Let Ω and Λ are two bounded open sets in R n , and let f : R n → R and g : R n → R be two positive functions such that f = 0 in R n \ Ω , g = 0 in R n \ Λ , and (cid:90)
[Equation]
According to Brenier’s Theorem Brenier (1987; 1991a), there exists a globally Lipschitz convex function ϕ : R n → R such that ∇ ϕ # f = g and and ∇ ϕ ( x ) ∈ ¯Λ for L 2 -a.e.
x ∈ R n .
We say ϕ weakly solves the Monge-Ampére equation
[Equation]
together with the boundary condition ∇ ϕ ( R n ) ⊂ ¯Λ .
ϕ is called the Briener potential .
As shown by Caffarelli [9], if Λ is convex, then ϕ is strictly convex, and it solves the Monge-Ampére equation 6.
The regularity theory has been estabilished (see Caffarelli (1990a;b; 1991)), such as
if λ ≤ f, g ≤ 1 /λ for some λ > 0 , then ϕ ∈ C 1 ,α ( λ ) loc (Ω) .
if f ∈ C k,αloc (Ω) and g ∈ C k,αloc (Λ) , then ϕ ∈ C k +2 ,α loc (Ω) , ( k ≥ 0 , α ∈ (0 , 1)) .
B. N ON - CONVEX D OMAINS - F IGALLI T HEOREM
However, if Λ is not convex, the regularity of the Brenier potential can not be guaranteed.
For example one can ﬁnd an example, such that
Ω is convex, Λ is simply connected, but non-convex; 2.
the density functions f
and g are smooth, f ∈ C ∞ (Ω) and g ∈ C ∞ (Λ) ; 3.
the Brenier potential ϕ (cid:54)∈ C 1 (Ω) , the transport map ∇ ϕ is not continuous.
In this scenario, the transport map can not be learned using DNNs, and training process is unstable or the GAN model generates unrealistic samples.
Figalli’s construction Let ϕ : R n → R be a convex function.
Its subdifferential at a point x is deﬁned by
[Equation]
ϕ is differentiable at a point x if and only if ∂ϕ ( x ) is a singleton.
Figalli decomposes the set of non-differentiability points according to the dimension of the singular set:
[Equation]
For any k = 0 , . . . , n , the set Σ k ( ϕ ) is ( n − k ) -rectiﬁable.
The set of reachable subgradients at x as
[Equation]
It is known that the convex hull of ∇ ∗ ϕ ( x ) , coincides with ∂ϕ ( x ) .
Theorem 4 (Figalli) .
Assume that there exists λ > 0 such that λ ≤ f ≤ 1 /λ in Ω , λ ≤ g ≤ 1 /λ in Λ , and that ∂ Ω and ∂ Λ are continuous.
Then ϕ is strictly convex inside Ω .
Moreover there exist two open sets Ω (cid:48)
⊂ Ω and Λ (cid:48) ⊂ Λ , with L 2 (Ω \ Ω (cid:48) )
=
L 2 (Λ \ Λ (cid:48) )
= 0 , such that ϕ ∈ C 1 ,α (Ω (cid:48) ) , ∇ ϕ is a bi-Hölder homeomorphism between Ω (cid:48) and Λ (cid:48) , and ϕ is an Alexandrov solution of 6 inside Ω (cid:48) .
In particular, Caffarelli’s regularity theory for strictly convex Alexandrov solutions of the Monge-Ampére equations applies to ϕ inside Ω (cid:48) .
Figalli studies the singular set of ϕ in Ω , i.e. the set of points x ∈ Ω where ϕ is not differentiable, denoted as Sing .
Figalli shows the singularity set has the following characterization,
[Equation]
it can be decomposed into connected components Sing :
= ∪ i S i .
For planar case,
Theorem 5 (Figalli Singularity Set) .
The number of connected components of Sing is at most countable.
Moreover:
either S i coincides with an isolated point { xi } for some x i ∈ Ω , and in this case the boundary of ∂ ( xi ) is entirely contained inside ∂ Λ (so that ∂ϕ ( xi ) completely ﬁlls a hole in Λ ); 2. or S i can be written as a disjoint union as follows:
[Equation]
where γ ij : I ij → Sing are embedded Lipschitz curves parameterized by arc-lengh, I ij is an interval.
B. E LEMENTARY E XPERIMENTS
We have designed several numerical experiments to verify Figalli’s theorems in low dimensional cases.
As shown in Fig.
5, the source domain Ω is a rectangle, the target domain Λ is a dumb-cell shape, the density functions f
and g are constant 1 , namely, uniform distribution.
The optimal transport maps is obtained using our method, the Brenier potential ϕ is not differentiable, the singularity set is near the middle of the rectangle, Σ 1 ( ϕ ) =
γ 1 ∪ γ 2 as shown in the ﬁgure.
At γ 1 and γ 2 , ϕ is continuous but not differentiable.
Each point x ∈ Ω is mapped onto ∇ ϕ ( x ) ∈ Λ with the same color.
This shows even the target domain is simply connected, the concavity will induce the discontinuity of the transport map.
Fig.
6 shows another computational result, which demonstrates the singularity structure in Figalli’s theorem.
The source domain Ω is the unit disk, the target domain Λ is with complicated geometry.
The singularity set of the optimal transport map satisﬁes the description of Figalli’s theorem 4,
[Equation]
= 3 (cid:91) i = γ k , Σ 2 = 1 (cid:91) j
= x j .
[Figure] [Figure]
∂ϕ ( x 0 ) ﬁlls the hole on Λ .
For any interior point p ∈ γ 1 , ∂ϕ ( p ) is a line segment connecting two points on the boundary of Λ .
Fig.
7 shows the singularity set of an optimal transport map between volumetric domains.
Ω is the solid ball, Γ is the interior of Stanford bunny.
The probability distributions are the uniform distribution.
The Brenier potential ϕ is obtained by solving the Monge-Ampére equation.
The optimal transport map is visualized by a morphing sequence: f t :=
(1 − t )
id + t ∇ ϕ for 0 ≤ t ≤ 1 , the intermediate shape is given by f t (Λ) .
It is obvious that the boundary surface of the bunny is folded inside the ball, which form the singularity set of the optimal transport map ∇ ϕ .
B. E XPERIMENTS ON R EAL D ATA S ETS
The low dimensional experiments demonstrates Figalli’s theorems, which are general for any dimension.
In the following, we design and carry out an experiment for high dimensional real data,
CelebA.
[Figure]
As shown in Fig.
8, we use an Autoencoder to encode the CelebA data samples to the latent space, which can be treated as a probability distribution with density function g : Λ → R , whose support set is Λ .
We deﬁne the source domain Ω as the unit cube, the source density function f ≡ 1 , namely uniform distribution.
Then we compute the optimal transport map from (Ω , f ) to (Λ , g ) , the Briener potential is ϕ .
Then we random generate a sample x from (Ω , f ) , then map it to the data distribution (Λ , g ) , the image is ∇ ϕ ( x ) .
Finally, ∇ ϕ ( x ) is mapped back to the image space by the decoder map to obtain a generated facial image.
Fig.
8 left frame shows the examples of generated facial images.
We then search the evidence of the existence of the singularity set of the Brenier potential ϕ .
We randomly draw a line segment, γ , which produce morphing sequence between two facial images as shown in the right frame of Fig. 8.
The starting point γ (0) corresponds to the boy face with brown eyes on the left top corner, the ending point γ (1) represents the girl face with blue eyes on the right bottom corner.
For each t ∈ [0 , 1] , γ ( t ) is mapped by ∇ ϕ to the latent data distribution (Λ , g ) , then decoded to an facial image interpolating the boy and the girl facial images.
In the center of frame (b), for some speciﬁc t 0 ∈ (0 , 1) , the generated image by γ ( t 0 ) is with one blue eye and one brown eye.
In reality, such kind of persons are extremely rare, therefore, we can treat such kind of facial images as on the boundary of the support Λ of the data distribution g in the latent space.
This means t 0 is in the singularity set of ϕ , the subgradient ∂ϕ ( γ ( t 0 )) intersects ∂ Λ at multiple points.
The Brenier potential at γ ( t 0 ) is only continuous but not differentiable, ∇ ϕ at γ ( t 0 ) is discontinuous.
In summary, the generated unrealistic facial image shows the transport map is discontinuous, which veriﬁes our hypothesis: the concavity of the support of the real data distribution causes the discontinuity of the transport map, which can not be directly represented by DNNs and induces instability of the training process or generating unrealistic samples.
C A DDITIONAL E XPERIMENTS C. S INGLE P ARAMETER S ELECTIVE I NTERPOLATION
On synthetic datasets, effects of angle threshold ﬁltering can be visually inspected.
As illustrated in Fig.
9, number of mode is a monotonically increasing function with respect to angle threshold ˆ θ .
Quality of generated samples is effected directly by choosing different ˆ θ .
Generally, small ˆ θ encourages interpolation in between closely related real samples while too large ˆ θ will result in interpolation between samples from different modes, which might in turn lower generation quality.
On synthetic datasets, where modes are isotropic and different modes are clearly separable, an ideal ˆ θ that captures all modes while avoids generating low quality samples can be chosen within a relatively wide band.
For real world datasets of unknown modes, an ideal ˆ θ needs to be hand tuned as the separability of different modes depends largely on input data pattern and quality of the embedding map.
[Figure]
C. M ORE R ESULTS IN S YNTHETIC D ATASETS
For illustration purpose, we plot our results on 2D-ring dataset along with those of GAN, PacGAN and BourGAN in Fig.
It is obvious that our method not only covers all of the modes, also the generation of meaningless data is overcome.
[Figure]
BourGAN (e) Generated samples of AE-OT.
It is observed that samples generated by the GAN model fail to capture all modes.
Samples from PacGAN capture all modes, but with inter-mode spurious samples.
Though both
BourGAN and AE-OT capture the target multimodal distribution precisely, the generation quality of the proposed method is better than BourGAN.
C. S TACK MNIST E XPERIMENT
Experiments of varisou GAN models on stacked MNIST dataset are in consistent with Lin et al.
For AE-OT model, we use the architecture shown in table 3 and table 4, with the decoder architecture same as the consistent generator architectures in GANs, and encoder having mirrored architecture.
[Table]
[Table]
We test diversity of generated samples from our AE-OT method on stack MNIST dataset that consists of 128,000 samples in 1,000 modes with each sample stacking three handwritten digit images from
MNIST dataset LeCun et al.
Number of modes counts the amount of modes captured by samples produced a generative model.
The reverse KL divergence is computed by ﬁrst assign each samples to their nearest mode, and compute the KL divergence between histogram of sample count on each mode and the histogram of real data.
We choose angle threshold ˆ θ = 0 .
5 for AE-OT method.
Details of network architectures are listed in supplementary materials.
Experiments results are summarized in table 5, which show our method achieves best performance in terms of modes captured and reverse KL divergence on stacked MNIST dataset.
In WGAN, PacWGAN and AE-OT experiments, number of feature maps in each network layer is a quarter of those in other experiments.
[Table]
C. C ELEB A E XPERIMENT
we evaluate our method on CelebA dataset by measuring collision probability in a batch ofgenerated images of size 64-by-64.
If a pair of identical images appear, a collision is declared
[Table]
and thus higher collision probability means lower generation diversity.
The same metric has been used in Lin et al.
(2018) for evaluation of PacGAN.
To make a consistent comparison, we design our autoencoder network with encoder having the same architecture as in previous work and decoder having a mirrored architecture of encoder.
Angle threshold ˆ θ is chosen to be 0 .
7 for AE-OT test.
Results are listed in table 6, with corresponding images can be downloaded here.
Results have shown that our method achieves best result in terms of probability of collision.
Autoencoder network structures can be found at table 7 and 8.
[Table] [Table]
[Table]
P ARAMETER S ETTING AND T RAINING D ETAILS OF AE-OT ON
4 P UBLIC D ATASETS
During the training of the autoencoders for MNIST, FANSION, Cifar10 and CelebA, when the Lloss stops descending, which means that the network has found a good encoding function of the image space, we freeze the encoder part and continue to train the network from latent space to image space.
And for the autoencoders, we run 200 epochs in total, including 150 epochs before the freezing of decoder, and 50 epochs after.
The training loss before and after the freezing of decoder is shown in
Tab. 9.
Besides, the parameters involved in the OT computation is set as follows: we set the parameters of
Adam algorithm to be α = 1 . 0 , β 1 = 0 .
9 , β 2 = 0 .
5 for all the experiments.
The ˆ θ is different for different tasks.
Speciﬁcally, for the MNIST and the FANSION-MNIST dataset, ˆ θ = arccos 0 . 8 ; for
CIFAR10 and CelebA, ˆ θ = arccos 0 .
When the sum of measure difference (cid:80) N i =1 | ˆ w i ( h ) − ν i | is less than 0.05, the loops stop.
The time consumed by AE-OT is mainly composed of two parts: the training of autoencoder and the computation of semi-discrete OT.
The training details of the former is illustrated in Section C.5 of the Appendix and we report the latter in table 10 with the Intel Core i7-7820X CPU and NVIDIA
GTX1080Ti GPU.
Here the dimension of the latent code is 64 for all of the four datasets.
[Table] [Table]
C. Q UANTITATIVE COMPARISON WITH PRECISION AND RECALL
FID score is an effective method to test the difference between the generated distribution and real data, but it mainly focuses on precision, and cannot accurately capture how much portion of real data a generative model could cover.
The method proposed in Sajjadi et al.
(2018) can compute the precision and the recall at the same time only given the same number of generated and reference images.
Firstly, the images are encoded to the feature space by the inception netSalimans et al.
Secondly, the features corresponding to generated and real data were put together and clustered.
Thirdly, the histograms, marked as P ( ω ) and Q ( ω ) for the both kinds of images appeared in each cluster are computed.
Finally, the precision α ( λ ) and recall β ( λ ) can be deﬁned as follows:
[Equation]
With different λ , we can get different pairs of ( α ( λ ) , β ( λ )) .
After the above deﬁnition, we can used the concept of ( F 8 , F 1 / 8 ) deﬁned on Sajjadi et al.
(2018) to quantify the relative importance of precision and recall.
We add the results of Hoshen & Malik (2019) and ours into the original recall-precision point sets and display them in Fig.
11, with khaki dot and red dot.
Previously, the best ( F 8 , F 1 / 8 ) pairs were got by Hoshen & Malik (2019) in MNIST, FASHIONMNIST, and we get slightly better scores on the both datasets.
For the CIFAR-10 dataset, the precision of our model slightly underperform gan, but achieve better recall scores.
Our recall score is also better than GLANN Hoshen & Malik (2019).
The proposed model does not achieve better score than the highest one of GANs on CelebA dataset, due to imprecision of pre-trained autoencoder.
If we expand the capacity of the autoencoder network, our model can get better scores.
In particular, when adopting the GCGAN architecture Radford et al.
as the decoder and its mirror as encoder, our model attains the best ( F 8 , F 1 / 8 ) score at (0 . 807 , 0 . 906) .
We draw this point in Fig.
(11) as purple dot and the generated images is shown in Fig.
The core problem of AE-OT is that if the autoencoder fails to approximate the manifold in the image space well, then the proposed method may not give a realistic result, just as the results of CelebA.
But if the autoencoder approximates the original manifold well, our theory can guarantee to generate good results (proved by the FID and PRD scores of the expanded autoencoder and the images shown in Fig. 12).
C. L INEAR I NTERPOLATION IN THE LATENT SPACE
Given any two images in the dataset, we can ﬁnd the images between them by linear interpolation in the noise space because the one to one correspondence between µ mass centers in the noise space and the images in the dataset is provided by the proposed algorithm.
For other generation models, though the interpolation can be done successfully in the noise space, they cannot ﬁnd the correspondence from the noise space and the image space.
The results of the linear interpolation are shown in Fig.
F 8 (Recall) 0 .0 .0 .0 .0 .1 .
F 1 / 8 ( P r ec i s i o n )
A
B
C
D
GANVAE 0 .0 .0 .
F 8 (Recall) 0 .0 .0 .0 .0 .1 .
F 1 / 8 ( P r ec i s i o n )
A
B
C
D
GANVAE 0 .0 .0 .
F 8 (Recall) 0 .0 .0 .0 .0 .1 .
F 1 / 8 ( P r ec i s i o n )
A
B
C
D
GANVAE 0 .0 .0 .
F 8 (Recall) 0 .0 .0 .0 .0 .1 .
F 1 / 8 ( P r ec i s i o n )
A
B
C
D
GANVAE
[Figure]