P OTENTIAL F LOW G ENERATOR WITH LO PTIMAL T RANSPORT R EGULARITY FOR G ENERATIVE M ODELS
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a potential ﬂow generator with L 2 optimal transport regularity, which can be easily integrated into a wide range of generative models including different versions of GANs and normalizing ﬂow models.
With only a slight augmentation to the original generator loss functions, our generator not only tries to transport the input distribution to the target one, but also aims to ﬁnd the one with minimum Ltransport cost.
We show the effectiveness of our method in several 2D problems, and illustrate the concept of “proximity” due to the L 2 optimal transport regularity.
Subsequently, we demonstrate the effectiveness of the potential ﬂow generator in image translation tasks with unpaired training data from the MNIST dataset and the CelebA dataset with a comparison against vanilla WGAN-GP and CycleGAN.
I NTRODUCTION
Many of the generative models, for example, generative adversarial networks (GANs) (Goodfellow et al., 2014; Arjovsky et al., 2017; Salimans et al., 2018) and normalizing ﬂow models (Rezende &
Mohamed, 2015; Kingma & Dhariwal, 2018; Chen et al., 2018), aim to ﬁnd a generator that could map the input distribution to the target distribution.
In many cases, especially when the input distributions are purely noises, the speciﬁc maps between input and output are of little importance as long as the generated distributions match the target ones.
However, in other cases like imageto-image translations, where both input and target distributions are distributions of images, the generators are required to have additional regularity such that the input individuals are mapped to the “corresponding” outputs in some sense.
If paired input-output samples are provided, L p penalty could be hybridized into generators loss functions to encourage the output individuals to ﬁt the ground truth (Isola et al., 2017).
For the cases without paired data, a popular approach is to introduce another generator and encourage the two generators to be the inverse maps of each other, as in CycleGAN (Zhu et al., 2017),
DualGAN (Yi et al., 2017) and DiscoGAN
(Kim et al., 2017), etc.
However, such a pair of generators is not unique and lacks clear mathematical interpretation about its effectiveness.
In this paper we introduce a special generator, i.e., the potential ﬂow generator, with L 2 optimal transport regularity.
By applying such generator, not only are we trying to ﬁnd a map from the input distribution to the target one, but we also aim to ﬁnd the optimal transport map that minimizes the squared Euclidean transport distance.
In Figure 1 we provide a schematic comparison between generators with and without optimal transport regularity.
While both generators provide a scheme to map from the input distribution to the output distribution, the total squared transport distances in the
As we will show later, this “proximity” characteristic of L 2 optimal transport regularity could be utilized in image translation tasks.
Compared with other approaches like CycleGAN, the L 2 optimal transport regularity has a much clearer mathematical interpretation.
There have been other approaches to learn the optimal transport map in generative models.
For example, Seguy et al.
(2017) proposed to ﬁrst learn the regularized optimal transport plan and then the optimal transport map, based on the dual form of regularized optimal transport problem.
Also,
Yang & Uhler (2018) proposed to learn the unbalanced optimal transport plan in an adversarial way derived from a convex conjugate representation of divergences.
In the W2GAN model proposed by
Leygonie et al.
(2019), the discriminator’s objective is the 2-Wasserstein metric so that the generator is supposed to recover the L 2 optimal transport map.
All the above approaches need to introduce, and are limited to, speciﬁc loss functions to train the generators.
Our proposed potential ﬂow generator takes a different approach in that with only a slight augmentation to the original generator loss functions, our generator could be integrated into a wide range of generative models with various generator loss functions, including different versions of GANs and normalizing ﬂow models.
This simple modiﬁcation makes our method easy to adopt on various tasks considering the existing rich literature and the future developments of generative models.
In Section 2 we present a formal deﬁnition of optimal transport map and the motivation to apply Loptimal transport regularity to generators.
In Section 3 we give a detailed formulation of the potential ﬂow generator and the augmentation to the original loss functions.
Results are then provided in
Section 4.
We include the discussion and conclusions in Section 5.
G ENERATIVE M ODELS AND O PTIMAL T RANSPORT M AP
First, we introduce the concept of push forward , which will be used extensively in the paper.
Deﬁnition
Given two Polish space X and Y , B ( X ) and B ( Y ) the Borel σ -algebra on X and Y , and
P ( X ) , P ( Y ) the set of probability measures on B ( X ) and B ( Y ) .
Let f : X → Y be a Borel map, and µ ∈ P ( X ) .
We deﬁne f # µ ∈ P ( Y ) , the push forward of µ through f , by
[Equation]
With the concept of push forward, we can formulate the goal of GANs and normalizing ﬂow models as to train the generator G such that G # µ is equal to or at least close to ν in some sense, where µ and ν are the input and target distribution, respectively.
Usually, the loss functions for training the generators are metrics of closeness that vary for different models.
For example, in continuous normalizing ﬂows (Chen et al., 2018), such metric of closeness is D KL ( G # µ || ν ) or D KL ( ν || G # µ ) .
In Wasserstein GANs (WGANs) (Arjovsky et al., 2017), the metric of closeness is the Wassersteindistance between G # µ and ν , which is estimated in a variational form with the discriminator neural network.
As a result, the generator and discriminator neural networks are trained in an adversarial way:
[Equation]
where D is the discriminator neural network and the Lipschitz constraint could be imposed via the gradient penalty (Gulrajani et al., 2017), spectral normalization (Miyato et al., 2018), etc.
Now we introduce the concept of optimal transport map as follows:
Deﬁnition
Given a cost function c : X × Y → R , and µ ∈ P ( X ) , ν ∈ P ( Y ) , we let T be the set of all transport maps from µ to ν , i.e. T :=
{ f : f # µ = ν } .
Monge’s optimal transport problem is to minimize the cost functional C ( f ) among T , where
[Equation]
and the minimizer f ∗ ∈ T is called the optimal transport map.
In this paper, we are concerned mostly with the case where X =
Y =
R d with L 2 transport cost, i.e., the transport c ( x , y ) =
(cid:107) x − y (cid:107) 2 .
We assume that µ and ν are absolute continuous w.r.t.
Lebesgue measure, i.e. they have probability density functions.
In general, Monge’s problem could be ill-posed in that T could be empty set or there is no minimizer in T .
Also, the optimal transport map could be non-unique.
However, for the special case we consider, there exists a unique solution to Monge’s problem (Brenier, 1991; Gangbo & McCann, 1996).
Informally speaking, with L 2 transport cost the optimal transport map has the characteristic of “proximity”, i.e. the inputs tend to be mapped to nearby outputs.
In image translation tasks, such “proximity” characteristic would be helpful if we could properly embed the images into Euclidean space such that our preferred input-output pairs are close to each other.
A similar idea is also proposed in Yang & Uhler (2018) for unbalanced optimal transport.
Apart from image translations, the Loptimal transport problem is important in many other aspects.
For example, it is closely related to gradient ﬂow (Ambrosio et al., 2008), Fokker-Planck equations (Santambrogio, 2017), ﬂow in porous medium (Otto, 1997), etc.
P OTENTIAL F LOW G ENERATOR 3. P OTENTIAL F LOW F ORMULATION OF O PTIMAL T RANSPORT M AP
We assume that µ and ν have probability density ρ µ and ρ ν , respectively, and consider all smooth enough density ﬁelds ρ ( t, x ) and velocity ﬁelds v ( t, x ) , where t ∈ [0 , T ] , subject to the continuity equation as well as initial and ﬁnal conditions
[Equation]
The above equation states that such velocity ﬁeld will induce a transport map: we can construct an ordinary differential equation (ODE)
[Equation]
5) and the map from the initial point to the ﬁnal point gives the transport map from µ to ν .
As is proposed by Benamou & Brenier (2000), for the transport cost function c ( x , y ) = [Equation] − y
[Equation]
2 , the minimal transport cost is equal to the inﬁmum of
[Equation]
among all ( ρ, v ) satisfying equation (4).
The optimality condition is given by
[Equation]
In other words, the optimal velocity ﬁeld is actually induced from a ﬂow with time-dependent potential φ ( t, x ) .
The use of this formulation is well-known in optimal transport community (Trigila & Tabak, 2016; Peyr ´ e et al., 2019).
In this paper we integrate this formulation in the deep generative models. Instead of solving Monge’s problem and ﬁnd the exact L 2 optimal transport map, which is unrealistic due to the limited families of neural network functions as well as the errors arising from training the neural networks, our goal is to regularize the generators in a wide range of generative models, so that the generator maps could approximate the L 2 optimal transport map at least in low dimensional problems.
The maps would also be endowed with the characteristics of “proximity” so that we can apply them to engineering problems.
The potential φ ( t, x ) is the key function to estimate, since the velocity ﬁeld could be obtained by taking the gradient of the potential and consequently the transport map could be obtained from
Equation 5.
There are two strategies to use neural networks to represent φ .
One can take advantage of the fact that the time-dependent potential ﬁeld φ is actually uniquely determined by its initial condition from Equation 7, and use a neural network to represent the initial condition of φ , i.e. φ (0 , x ) , while approximating φ ( t, x ) via time discretization schemes.
Alternatively, one can use a neural network to represent φ ( t, x ) directly and later apply the PDE regularity for φ ( t, x ) in Equation 7.
We name the generators deﬁned in the above two approaches as discrete potential ﬂow generator and continuous potential ﬂow generator, respectively, and give a detailed formulation as follows.
3.2. D ISCRETE P OTENTIAL F LOW G ENERATOR
In the discrete potential ﬂow generator, we use the neural network ˜ φ 0 ( x ) : R d → R to represent the initial condition of φ ( t, x ) , i.e. φ (0 , x ) .
The potential ﬁeld φ ( t, x ) as well as the velocity ﬁeld v ( t, x ) could then be approximated by different time discretization schemes.
As an example, here we use the ﬁrst-order forward Eular scheme for the simplicity of implementation.
To be speciﬁc, suppose the time discretization step is ∆ t and the number of total steps is n with n ∆ t = T , then for i
= 0 , 1 ...n , φ ( i ∆ t, x ) could be represented by ˜
φ i ( x ) , where
[Equation]
[Equation]
[Equation] [Equation] [Equation] Consequently, the velocity ﬁeld v ( i ∆ t, x ) could be represented by ˜ v i ( x ) , where
[Equation]
Finally, we can build the transport map from Equation 5:
[Equation]
with G ( · ) =
˜
u n ( · ) be our transport map.
The discrete potential ﬂow generator has built-in optimal transport regularity since the optimal condition (Equation 7) is encoded in the time discretization (Equation 8).
However, such discretization also introduces nested gradients, which dramatically increases the computational cost when the number of total steps n is increased.
In our tests, we found that even n = 5 is almost intractable.
3.2. C ONTINUOUS P OTENTIAL F LOW G ENERATOR
In the continuous potential ﬂow generator, we use the neural network ˜ φ ( t, x ) : R × R d → R to represent φ ( t, x ) .
Consequently, the velocity ﬁeld v ( t, x ) can be represented by ˜ v ( t, x ) , where
[Equation]
With the velocity ﬁeld we could estimate the transport map by solving the ODE (Equation 5) using any numerical ODE solver.
As an example, we can use the ﬁrst-order forward Eular scheme, i.e.
[Equation]
with G ( · ) =
˜ u ( T, · ) be the transport map, where ∆ t is the time discretization step and n is the number of total steps with n ∆ t = T .
In the continuous potential ﬂow generator, increasing the number of total steps would not introduce high order differentiations, therefore we could have large n , for a better precision of the ODE solver.
Different from the discrete potential ﬂow generator, the optimal condition (Equation 7) is not encoded in the continuous potential ﬂow generator, therefore we need to penalize Equation 7 in the loss function, as we will discuss in the next subsection.
One may come up with another strategy of imposing the L 2 optimal transport regularity: to use a vanilla generator, which is a neural network directly mapping from inputs to outputs, and penalize the L 2 transport cost, i.e., the loss function is
[Equation]
(13) where L original is the original loss function for the generator, and α is the weight for the transport penalty.
We emphasize that such strategy is much inferior to penalizing Equation 7 in the continuous potential ﬂow generator.
When training the vanilla generator with L 2 transport penalty, no matter how we weight the L 2 transport cost penalty, we always have to make a trade off between “matching the generated distribution with the target one” and “reducing the transport cost” since there is always a conﬂict between them, and consequently G # µ will be biased towards µ .
On the other hand, there is no conﬂict between matching the distributions and penalizing Equation 7 in the continuous potential ﬂow generator.
As a consequence, the continuous potential ﬂow generator is robust with respect to different weights for the PDE penalty.
We will show this in Section 4.
3. T RAINING THE P OTENTIAL F LOW G ENERATOR
While the optimal condition (Equation 7) has been considered in the above two generators, the constraints of initial and ﬁnal conditions have so far been neglected.
However, the constraint of initial and ﬁnal conditions provides the principle to train the neural network: we need to tune the parameter in the neural network ˜ φ so that G # µ matches ν .
This could be done in the fashion of GANs or normalizing ﬂow models.
3.3. L OSS IN GAN M ODELS
For the discrete potential ﬂow generator, since the optimal transport regularity is already built in, the loss for training G is simply the GAN loss for the generator, i.e.
[Equation]
(14) where L GAN actually depends on the speciﬁc version of GANs.
For example, if we use WGAN-GP, then L GAN = − E z ∼ µ D ( G ( z )) , where D is the discriminator neural network.
For the continuous potential ﬂow generator, as mentioned above, we also need to make ˜ φ satisify
Equation 7 for the optimal transport regularity.
Inspired by the applications of neural networks in solving forward and backward problems of PDEs (Lagaris et al., 1998; Raissi et al., 2017a;b;
Sirignano & Spiliopoulos, 2018)
, we penalize the squared residual of the PDE on the so-called “residual” points.
In particular, the loss for continuous potential ﬂow generator would be
[Equation]
(15) where { ( t i , x i ) } Ni =1 are the residual points for estimating the residual of the PDE (Equation 7), and λ is the weight for the PDE penalty.
In this paper we set them as the points on “trajectories” of input samples, i.e.
[Equation]
where B is the set of batch samples from µ .
Note that the coordinates of the residual points involve ˜ u , but this should not be taken into consideration when calculating the gradient of the loss function with respect to the generator parameters.
We point out that the residual points should cover the whole spatial-temporal domain. Theoretically, only penalizing the squared residual of the PDE on “trajectories” could lead to failure in approximating the L 2 optimal transport map. However, in our numerical experiments, this ﬂawed sampling strategy still works. As an improvement, in each training iteration we can perturb the trajectory points with
Gaussian noise in space and uniform noise in time as residual points, so that in principle they are sampled from the whole spatial-temporal domain. 3.3. L OSS IN N ORMALIZING F LOW M ODELS
Our continuous potential ﬂow generator could be viewed as a further development of Neural ODE (Chen et al., 2018) applied to generative models, i.e. the continuous normalizing ﬂow.
The difference mainly lies in that we set the velocity as the gradient of a time-dependent potential function, and an augmented PDE loss is required for the generator.
While both density matching and maximum likelihood training could be applied, here we take the latter as an example: we assume that the density of µ and samples from ν are available, and we maximize E y ∼ ν [log p G # µ ( y )] , where p G # µ is the density of G # µ .
Then, the loss for the continuous potential ﬂow generator would be:
[Equation]
(17) where as in the GAN model, { ( t i , x i ) } Ni =1 are the residual points for estimating the residual of PDE (Equation 7), and λ is the weight for the PDE penalty.
E y ∼ ν
[log p G # µ ( y )] could be estimated via the approach introduced in Chen et al.
and we give a detailed description in Appendix B.
The discrete potential ﬂow cannot be trivially applied in normalizing ﬂow models since we found that the time step size is too large to calculate the density accurately.
R ESULTS 4. 2D P ROBLEMS
In this subsection, we apply the potential ﬂow generators to several two-dimensional problems.
We mainly study the following two problems where we know analytical solutions for the optimal transport maps.
In problem 1 we assume that both µ and ν are Gaussian distributions with µ =
N ([0; 0] , [0 . 25 , 0; 0 , 1]) and ν = N ([0; 0] , [1 , 0; 0 , 0 . 25]) .
In this case the optimal transport map is f (( x, y ))
= (2 x, 0 . 5 y ) .
In problem 2 we assume that µ and ν are concentrated on concentric rings.
In polar coordinates, suppose µ has ( r, θ ) uniformly distributed on [0 . 5 , 1] × [0 , 2 π ) , while ν has ( r, θ ) uniformly distributed on [2 , 2 . 5] × [0 , 2 π ) , where r and θ are radius and angular, respectively.
In this case the optimal transport map is f (( r, θ ))
=
( r + 1 . 5 , θ ) in polar coordinates.
We present the proofs in Appendix A. Samples from µ and ν as well as the optimal transport map in both problems are illustrated in Figure 2.
We prepared 40000 samples for the input distribution and target distribution in each problem as training data for Section 4.1.1 and 4.1.2, and 1000 samples for Section 4.1.3.
[Figure]
0 1 2 − −   2 Samples of μ and ν μ ν
[Figure]
[Figure]
[Figure] ν Vanilla [Figure]
Weight =
[Figure]
Vanilla
[Figure]
Weight
[Figure]
[Figure]
Weight
[Figure]
[Figure]
Weight
[Figure]
4.1. V ANILLA G ENERATOR V ERSUS P OTENTIAL F LOW G ENERATOR
For the above two problems we compare the following methods: (a) vanilla generator with Ltransport cost penalty, i.e., using the loss function in Equation 13 with GAN loss as L original , (b) discrete potential ﬂow generator, and (c) continuous potential ﬂow generator with PDE penalty.
For the vanilla generator and the continuous potential ﬂow generator, we test different weights for the penalty in order to compare the inﬂuence of penalty weights in both generators.
As for the GAN loss for generators we use the sliced Wasserstein distance 1 , due to its relatively low computational cost, robustness, and clear mathematical interpretation in low dimensional problems (Deshpande et al., 2018).
In Figure 2 we illustrate the maps of different generators.
A more systematic and quantitative comparison from three independent runs for each case is provided in Table 1, where the best results are marked as bold. The statistics come from 100,000 testing data.
[Table] As we already mentioned [Table] vanilla generators with L 2 transport penalty would make G # µ biased towards µ to reduce the transport cost from µ to G # µ .
This is clearly shown in both problems with the penalty weight α = 0 .
In fact, we observed more signiﬁcant biases with larger penalty weights.
For the cases with smaller penalty weights α = 0 .
001 , in some of the runs, while G # µ are close to ν , the maps of generators are far from the optimal ones, which shows that the L 2 transport penalty cannot provide sufﬁcient regularity if the penalty weight is too small.
These numerical results are consistent with our earlier discussion about the intrinsic limitation of the L 2 transport penalty.
On the other hand, the potential ﬂow generators give better matching between G # µ and ν , as well as smaller errors between the estimated transport maps and the analytical optimal transport maps.
Notably, in both problems the continuous potential ﬂow generators give good results with a wide range of PDE penalty weights ranging from 0 .
1 to 10 , which shows the superiority of PDE penalty in the continuous potential ﬂow generators compared with the transport penalty in vanilla generators.
We also report that while in the ﬁrst problem the discrete potential ﬂow generator achieves a comparable result with the continuous potential ﬂow generators, in the second problem we encountered “NAN” problems during training of the discrete potential ﬂow generator in some of the runs.
This indicates that the discrete potential ﬂow generator is not as robust as the continuous one, which could be attributed to the high order differentiations and small total time steps n in the discrete potential ﬂow generators. 4.1. C YCLE GAN V ERSUS P OTENTIAL F LOW G ENERATOR
We also apply CycleGAN on the above two problems, with different random seeds.
Here, we use feedforward networks as generators and discriminators, with WGAN-GP for the GAN loss function, and L 1 loss for cycle-consistency loss with weight 5. In Figure 2 we illustrate the maps of G (red arrows) and F (blue arrows), i.e. the two generators in CycleGAN. The red and blue arrows overlap with opposite directions, which indicates that G and F are approximately the inverse map of each other, as we expected from the cycle-consistency loss. However, the maps are totally different in the three runs with different random seeds, which agrees with our discussion in Section 1 that the generator pair in CycleGAN is not unique.
Moreover, the generator maps are less “regular” than the maps from the potential ﬂow generator. Speciﬁcally, we can hardly interpret the generator maps given by CycleGAN. 4.1. D ISCRETE R EGULARIZED O PTIMAL T RANSPORT S OLVER V ERSUS P OTENTIAL F LOW G ENERATOR Finally, we compare the continuous potential ﬂow generator in SWG and WGAN-GP with the discrete regularized optimal transport solver 2 introduced by Seguy et al.
(2017) on the above two problems.
The results of the output distributions, as well as the errors between estimated transport maps and the Large-Scale- All magnitude
Their
[Table]
analytical optimal transport maps are shown in Table 2, where the best results for different batch size setups are marked as bold. The statistics come from 100,000 testing data. We ﬁrst set the batch size as 50; in this case, the errors of maps are similar for all three methods, while the output distributions of the continuous potential ﬂow generator in WGAN-GP match the best with the target ones. As is well known, the gradients of the sample Wasserstein loss are biased (Bellemare et al., 2017), thus the output distributions are biased for GANs based on the Wasserstein loss.
This problem could be serious when the batch size is small. Therefore, we increased the batch size to 1000. In this case, we encounterred “NAN” problems when learning the barycentric mapping in the discrete regularized OT solver.
SWG and WGAN-GP with continuous potential ﬂow generators are stable
[Table]
and we can see an improvement in the output distributions and error of maps, as we expected.
[-−2 −1 0 
-]4.1. M ORE P ROBLEMS IN GAN M ODEL AND N ORMALIZING F LOW M ODEL
[Table]
−1 0 1 2 3 − −2 − 02μν −2 [-−
 [Figure]-]−1 2 − −    μν −   −   μν −3 −2 −1 0 1 2 3 − −2 − 02−2 −1 2 − −    −   −   (a) WGAN-GP −2 −1 0 1 2 − −    −3 −2 −1 0 1 2 3 − −2 − 02−2 −1 2 − −    −2 −1 0 1 2 − −    −3 −2 −1 0 1 2 3 − −2 − 02−2 −1 2 − −    0.00 0.
(b) Continuous normalizing ﬂow
Apart from the previous two problems, we also apply WGAN-GP and continuous normalizing ﬂow models with continuous potential ﬂow generators to several more complicated distributions. The results are illustrated in Figure 3.
We can see the match between G # µ and ν in each of the problems, as well as that the samples of µ tend to be mapped to nearby positions.
This shows the effectiveness of the continuous potential ﬂow generator in various generative models, as well as the characteristics of “proximity” in the potential ﬂow generator maps due to the L 2 optimal transport regularity.
4. I MAGE T RANSLATION T ASKS
In this section, we aim to show the capability of continuous potential ﬂow generator in dealing with high dimensional problems, and also to show its advantage in image translations tasks with unpaired training data. We use WGAN-GP for the GAN loss.
Before feeding the images into the generators, we embed them into a Euclidean space, where the L 2 distances between embedding vectors should quantify the similarities between images. In this paper we apply the principal component analysis (PCA) (Jolliffe, 2011), a simple but highly interpretable approach to conduct the image embedding. [Figure]
In each raw, the top images are the inputs, while the the
We ﬁrstly test the problem of translation between the MNIST images (LeCun et al., 2010).
We divide the MNIST training dataset into two clusters: (a) images of digits 0 to 4, and (b) images of digits 5 to 9.
We view the two clusters of images as samples of µ and ν , respectively, i.e., we want to ﬁnd the optimal transport map from images of digits 0 to 4 to images of digits 5 to 9.
The dimensionality of Euclidean space, i.e. the total components in PCA, is set as 100. In Figure 4 we randomly pick images from the test dataset and show the corresponding inputs and outputs (more images in Appendix D). As we can see, the potential ﬂow generator tends to translate images of digit 0 to digit 6, digit 1 to digit 7, digit 3 to digit 5 or 8, and digit 4 to digit 9. This is consistent with our previous discussion about the characteristics of “proximity” in that the input digits and output digits “look similar”, and the corresponding embedding vectors should be close in the L 2 distance.
[Equation] [Figure] [Equation] [Figure]
In this section we test the translation task between the CelebA images (Liu et al., 2015).
We randomly pick 60000 images from the CelebA training dataset and divide them into two clusters: (a) images with attribute “smiling” labeled as false, and (b) images with attribute “smiling” labeled as true.
The images are cropped so that only faces remain on the images.
We view the two clusters as samples of µ and ν , respectively.
The total number of components in PCA is set to 700.
Note that our goal is to generate images of smiling faces belonging to the same people as in input images.
The difﬁculty lies in that the training data are unpaired and actually we do not have the ground truth as a reference.
We compare our method against vanilla WGAN-GP, and CycleGAN with WGAN-GP for GAN loss.
In CycleGAN we use L 1 or L 2 loss as cycle-consistency loss, with weight = 10 as in Zhu et al. To make a fair comparison, for each generator and discriminator, we use a feedforward neural network (FNN) with the same size of hidden layers ( 5 × 256 ).
We also test the ofﬁcial CycleGANon
this problem, where we feed the cropped face images (with resizing but without random jitter cropping) instead of embedding vectors into the model. The results of images randomly picked from the test dataset are shown in Figure 5. For most of the images, our method could successfully translate the no-smiling faces to smiling faces belonging to the same people.
Some of the output images are blurred, since it is difﬁcult to learn the high order modes of PCA with FNN.
Vanilla WGAN-GP and CycleGAN with FNN totally failed, in that the input and output images come from different people. This comparison clearly showed the necessity of additional regularity for the generators in translation tasks with only unpaired data, and that GAN loss + cycle-consistency loss cannot provide sufﬁcient regularity. The ofﬁcial CycleGAN is decent in performance, generating images less blurred than our method, but failed to change the countenance for some images.
Note that the number of parameters in ofﬁcial CycleGAN is about 30 times more than that in our method, and the total training time is more than two times of our method on a single NVIDIA Tesla V100 GPU.
D ISCUSSION AND C ONCLUSIONS
In this paper we propose potential ﬂow generators with L 2 optimal transport regularity as plug-andplay generator modules that could be easiy integrated in a wide range of generative models.
In particular, we propose two versions: the discrete one and the continuous one.
For the discrete version, the L 2 optimal transport regularity is directly encoded in, while for the continuous version we only need a slight augmentation to the original generator loss functions to impose the L 2 optimal transport regularity.
We ﬁrstly show that the potential ﬂow generators are able to approximate the L 2 optimal transport maps in 2D problems.
The continuous potential ﬂow generator outperforms the discrete one in robustness.
The continuous potential ﬂow generator is also applied to WGAN-GP and continuous normalizing ﬂow models, where we illustrate the characteristic of “proximity” for the potential ﬂow generator due to the L 2 optimal transport regularity.
Consequently, we show the effectiveness of our method in image translation tasks using unpaired training data from the MNIST dataset and the CelebA dataset. We can see that our method signiﬁcantly outperforms the vanilla WGAN-GP and
CycleGAN using FNN with the same size of hidden layers. We think that the results of our method in translation tasks are impressive considering that we only use PCA, a linear embedding method, with only feedforward neural networks. Such a naive strategy actually leads to the blurred patterns in the output images, which is also the case (even more severe) for vanilla WGAN-GP and CycleGAN using the same strategy.
A possible improvement is to integrate the potential ﬂow generator with other embedding techniques like autoencoders with convolutional neural networks. Apart from image-to-image translations, it is also possible to apply the potential ﬂow generator to other translation tasks, if the translation objects could be properly embedded into Euclidean space.
Moreover, it was perceived that the training of ODE-based model is slow, but the training of our method could be accelerated by applying methods related to optimal transport, e.g. the Wasserstein natural gradient method (Tong Lin et al., 2018; Li & Mont´ufar, 2018).
We leave these possible improvements to future work.
A PROOF OF THE OPTIMAL MAPS IN S ECTION 4.
In problem 1, for Gaussian distributions µ and ν with mean m 1 and m 2 , as well as covariance matrices Σ 1 and Σ 2 , from Gelbrich (1990) we know that the minimum transport cost from µ to ν with cost function c ( x , y )
= (cid:107) x − y (cid:107) 2 is
[Equation]
18) which is known as the squared Wasserstein-2 distance between two Gaussian distributions.
In particular, the minimum transport cost is 0.5 in our problem.
For the map f (( x, y )) =
(2 x, 0 .
5 y )
f # µ is Gaussian since f is linear.
By checking the mean and covariance we have f # µ = ν .
Also, the transport map of f is
[Equation]
19) which is exactly the minimum transport cost, thus f is the optimal transport map.
We complete the proof of the optimal transport map in problem 1.
In problem 2, denote X =
[0 . 5 , 1] , Y =
[2 , 2 . 5] , O = [0 , 2 π ) , and m
1 =
U ( X ) , m
= U ( Y ) , m θ =
U ( O ) , where we use U ( A ) to represent the uniform probability measure on set A .
For f (( r, θ ))
=
( r + 1 . 5 , θ ) , µ =
U ( X ) × U
( O ) , ν =
U ( Y ) × U ( O ) , we have f # µ = ν .
For any transport map from µ to ν , denote as g (( r, θ ))
=
( g r ( r, θ ) , g θ ( r, θ )) in polar coordinates, we only need to show that the transport cost of g is no less than the cost of f .
Let h (( r, θ )) =
( g r ( r, θ ) , θ ) , then the transport cost of g is no less than the cost of h since the transport cost is reduced for any point ( r, θ ) .
Actually we could view g r ( r, θ ) as a multivalued function in r so that g r ( r, θ ) induces a transport plan H from m 1 to m 2 .
More formally, deﬁne measure H : B ( X × Y )
→ R by
[Equation]
where M ( · , · ) : X × B ( Y ) → R is deﬁned by
[Equation]
To see H is a transport plan from m 1 to m 2 , we need to check:
∀
A ∈ B ( X ) , H ( A × Y )
=
m 1 ( A ) .
This is true since
[Equation]
where we utilize that M
( x
Y ) =
∀
A ∈ B ( Y ) ,
H ( X × A )
= m 2 ( A ) .
Note that g # µ = ν , thus µ ( g − 1 ( A
×
O ))
=
ν
A × O )
= m
A ) .
Also
[Equation]
Therefore H ( X × A ) =
m 2 ( A ) .
We also claim that the L 2 transport cost of h equals to that of H .
The transport cost of h and H are
[Equation]
24) respectively.
By the deﬁnition of M , we have M ( x, · ) =
g r ( x, · ) # m θ for any x ∈ X , thus
[Equation]
[Equation]
for any x ∈ X .
Therefore C ( h ) =
C ( H ) .
Let F ( x ) = x + 1 . 5 be another transport plan from m 1 to m 2 , clearly the L 2 transport cost of f equals to that of F .
Note that the transport cost of H is no less than that of F , since the latter one is the optimal transport plan from m 1 to m 2 .
This complete the proof of claim that the transport cost of g is no less than that of f , and thus the proof of the optimal transport map in problem 2.
B D ETAILS OF L OSS F UNCTIONS IN C ONTINUOUS N ORMALIZING F LOWS
To estimate log p G # µ ( y ) , we have the ODE that connects the probability density at inputs and outputs of the generator: [Equation]
[Equation]
for all x in the support of µ , where the initial probability density p (˜ u (0 , x ))
= p µ ( x ) is the density of µ at input x , while the terminal probability density p (˜ u ( T, x ))
=
p G # µ ( G ( x )) is the density of
G # µ at output G ( x ) .
Also, we estimate x = G − 1 ( y ) by solving the ODE
[Equation]
with initial condition w (0) as y =
G ( x ) and w ( T ) as the corresponding x = G − 1 ( y ) .
For each y , we can use Equation 27 to estimate the corresponding x = G − 1 ( y ) , and consequently log( p µ ( x )) since we have the density of µ .
Then we apply Equation 26 to estimate log p G # µ ( y ) .
By sampling y ∼ ν , we can estimate E y ∼ ν
[log p G # µ ( y )] .
In practice, we also need to discretize
Equations 26 and 27 properly.
For example, we use the ﬁrst-order Euler scheme in our practice.
Note that when applying maximum likelihood training, the density of µ could be unnormalized, since multiplications with p µ would merely lead to a constant difference in the loss function.
C N ETWORKS AND H YPERPARAMETERS
Except in the regulized discrete optimal transport solver and ofﬁcial CycleGAN, all the neural networks are feedforward neural networks of 5 hidden layers, each of width 128 for 2D problems, or 256 in image translation tasks.
For potential ﬂow generator, the activation function is tanh for the smoothness of ˜ φ . The activation function in vanilla generator in Figure 2 is also tanh . All the other activation functions are leaky ReLU (Maas et al., 2013) with negative slope 0.2, except in the regulized discrete optimal transport solver and ofﬁcial CycleGAN.
The batch size is set as 1000 for all the cases except in Table 2 and in ofﬁcial CycleGAN.
We use 1000 random projection directions to estimate the sliced Wasserstein distances.
In WGAN-GP model the coefﬁcient for gradient penalty is 0 . 1 , and we do 5 discriminator updates per generator update.
In potential ﬂow generators, the time span T is 1 . 0 .
We set the number of total time steps n =in discrete potential ﬂow generators, while n = 100 in continuous potential ﬂow generators for 2D problems and n = 10 in image translation tasks.
The PDE penalty weight λ for continuous potential ﬂow generator is set as 1 . 0 by default, except those in the 2D problems where we compare different generators.
We use the Adam optimizer (Kingma & Ba, 2014) for all the problems.
In the 2D problems, the learning rate is set as l = 10 − 5 , β 1 = 0 .
5 , β 2 = 0 .
999 for sliced Wasserstein distance, while set as l = 10 − 5 , β 1 = 0 .
5 , β 2 = 0 .
9 in WGAN-GP. We train the generators for 100 , 000 iterations in
Figure 2 and Figure 3a, and 20 , 000 iterations in Table 2.
In the normalizing ﬂow model we set l = 10 − 4 , β 1 = 0 .
9 , β 2 = 0 .
999 , and train the generator for 10 , 000 iterations.
In image translation tasks we set l = 10 − 4 , β 1 = 0 .
5 , β 2 = 0 .
9 , and train the generator for 100 , 000 iterations for our method and CycleGANs with FNN, while 200 , 000 iterations for vanilla GAN.
D M ORE RESULTS ON THE MNSIT AND C ELEB A DATASET
[Figure] [Figure]