C
RAMER
-W OLD
A
UTO E NCODER
Anonymous authors
Paper under double-blind review
A BSTRACT
Assessing distance between the true and the sample distribution is a key component of many state of the art generative models, such as Wasserstein Autoencoder (WAE).
Inspired by prior work on Sliced-Wasserstein Autoencoders (SWAE) and
WAE-MMD (WAE using maximum mean discrepancy based distance function) we construct a new generative model – Cramer-Wold AutoEncoder (CWAE).
CWAE cost function is based upon a characteristic kernel we introduce, called CramerWold kernel, and has a a simple closed-form in the case of normal prior.
As a consequence, while simplifying the optimization procedure (no need of sampling necessary to evaluate the distance function in the training loop), CWAE performance matches quantitatively and qualitatively that of WAE-MMD and often improves upon SWAE.
I NTRODUCTION
One of the crucial aspects in construction of generative models is devising effective method for computing and minimizing distance between the true and the model distribution.
Originally in
Variational Autencoder (VAE) (Kingma & Welling, 2014) this computation was carried out using variational methods.
An important improvement was brought by the introduction of Wasserstein metric (Tolstikhin et al., 2017) and the construction of WAE-GAN and WAE-MMD models, which relax the need for variational methods.
WAE-GAN requires a separate optimization problem to be solved to approximate the used divergence measure, while in WAE-MMD the discriminator has the closed-form obtained from a characteristic kernel, i.e. one that is injective on distributions Muandet et al.
A recent contribution to this trend of simplifying the construction of generative models is Sliced-Wasserstein Autoencoder (SWAE, Kolouri et al.
(2018)), where a signiﬁcantly simpler
AutoEncoder based model based on Wasserstein distance is proposed.
sets obtained as the mean squared L 2 -distance of their smoothed projections on all onedimensional lines.
The main innovation of SWAE was the introduction of the sliced-Wasserstein distance – a fast to estimate metric for comparing two distributions, based on the mean
Wasserstein distance of one-dimensional projections.
However, even in SWAE there is no close analytic formula that would enable computing the distance of the sample from the standard normal distribution.
Consequently in SWAE two types of sampling are needed: (i) sampling from the prior distribution and (ii) sampling over one-dimensional projections.
Our main contribution is introduction of the CramerWold distance between distributions, which has a closed-form for the distance of a sample from standard multivariate normal distribution .
Its important feature is that it is given by a characteristic kernel which has a closed-form given by equation 7 for the product of radial Gaussians 1 .
We use it to construct an AutoEncoder based generative model, called Cramer-Wold AutoEncoder (CWAE), in which the cost function, for a normal prior distribution, has a closed analytic formula.
Thus
CWAE can be seen as a borderline model between SWAE and WAE-MMD, as we use a characteristic kernel to discriminate distributions, but its formula comes from the application of the sliced approach.
An introduction of sliced approach is given in the following section, the comparison of CWAE and
WAE-MMD is given in Appendix, Section
B. We benchmark CWAE against generative models using a non-parametrized distance functions: WAE-MMD (WAE variant from Tolstikhin et al.
based on the classical MMD distance) and SWAE.
Our results show that, while simplifying the optimization procedure by forgoing the need to sample data projections or from the prior distribution we obtain an
AutoEncoder based generative model which obtains quantitatively and qualitatively similar results to
WAE-MMD and SWAE models, see Section 5.
Let us now brieﬂy describe the outline of the paper.
In the following two sections we introduce and theoretically investigate the Cramer-Wold distance.
However, the reader interested mainly in the construction of our generative AutoEncoder model can proceed directly to Section 4.
Sectioncontains experiments.
Conclusions are given in Section 6.
C RAMER -W OLD DISTANCE : CONSTRUCTION
Motivated by the prevalent use of normal distribution as prior in modern generative models, we investigate whether it is possible to simplify optimization of such models.
As the ﬁrst step towards this, in this section we introduce Cramer-Wold distance, which has a simple analytical formula for computing normality of a high-dimensional sample.
On a high level our approach uses the traditional
L 2 distance of kernel-based density estimation, computed across multiple single-dimensional projections of the true data and the output distribution of the model.
We base our construction on the following two popular tricks of the trade:
Sliced based decomposition of a distribution:
Following the footsteps of Kolouri et al.
Deshpande et al.
(2018), the ﬁrst idea is to leverage the Cramer-Wold Theorem (Cramér & Wold, 1936) and Radon Transform (Deans, 1983) to reduce computing distance between two distributions to one dimensional calculations.
For v in the unit sphere S D ⊂ R D , the projection of the set X ⊂ R D onto the space spanned by v is given by v T X and the projection of N ( m, αI ) is N ( v T m, α ) .
Cramer-Wold theorem states that two multivariate distributions can be uniquely identiﬁed by their all one-dimensional projections.
For example, to obtain the key component of SWAE model, i.e. the sliced-Wasserstein distance between two samples X, Y ∈ R D , we compute the mean Wasserstein distance between all one-dimensional projections:
[Equation]
where S D denotes the unit sphere in R D and σ D is the normalized surface measure on S D .
This approach is effective since the one-dimensional Wasserstein distance between samples has the closed form, and therefore to estimate (1) one has to sample only over the projections.
Smoothing distributions: Using the sliced-based decomposition requires us to deﬁne distance between two sets of samples, in a single dimensional space.
To this end we will use a trick-of-trade applied commonly in statistics in order to compare samples or distributions which is to ﬁrst smoothen (sample) distribution with a Gaussian kernel.
For the sample R =
( r i )
i =1 ..n ⊂ R by its smoothing with Gaussian kernel N (0 , γ ) we understand
[Equation]
where by N ( m, S ) we denote the one-dimensional normal density with mean m and variance S .
This produces a distribution with regular density, and is commonly used in kernel density estimation.
If R comes from the normal distribution with standard deviation close to one, the asymptotically optimal choice of γ is given by the Silverman’s rule of thumb
γ = ( 43 n ) 2 / 5 , see Silverman (1986).
For continuous density f , its smoothing sm γ ( f ) is given by the convolution with N (0 , γ ) , and in the special case of Gaussians we have sm γ ( N ( m, S ))
=
N ( m, S + γ ) .
While in general kernel density estimations works well only in low-dimensional spaces, this ﬁts the bill for us, as we will only compute distances on single dimensional projections of the data.
Cramer-Wold distance.
We are now ready to introduce the Cramer-Wold distance .
In a nutshell, we propose to compute the squared distance between two samples by considering the mean squared
L 2 distance between their smoothed projections over all single dimensional subspaces.
By the squared
L 2 distance between functions f, g : R
→ R we refer to (cid:107)
f − g
(cid:107)
= (cid:82) | f
( x ) − g ( x ) | 2 dx .
A key feature of this distance is that it permits a closed-form in the case of normal distribution.
More precisely, the following algorithm fully deﬁnes the Cramer-Wold distance between two samples
X = ( x i ) i =1 ..n , Y = ( y j ) j
=1 ..k ⊂ R D (for illustration of Steps 1 and 2 see Figure 1):
given v in the unit sphere S (0 , 1) ⊂ R D consider the projections v T X =
( v T x i ) i =1 ..n and v T Y =
( v T y j ) j
=1 ..k , 2.
compute the squared L 2 distance of the densities sm γ ( v T X ) and sm γ ( v T X ) : (cid:107) sm γ ( v T X ) − sm γ ( v T Y ) (cid:107) 22 , 3.
to obtain squared Cramer-Wold distance average (integrate) the above formula over all possible v ∈ S D .
C RAMER -W OLD DISTANCE : THEORY
The key theoretical outcome of this paper is that the result of the computation of the Cramer-Wold distance from the previous section can be simpliﬁed to a closed form solution.
Consequently, to compute the distance of two samples there is no need of ﬁnding the optimal transport like in WAE or the necessity to sample over the projections like in SWAE.
For the case of simplicity we provide in this section the formulas for the distance between two samples and the distance of a sample from the standard normal density.
The general deﬁnition of Cramer-Wold metric is presented in Appendix,
Section A.
Theorem 3.1.
Let X =
( x i ) i =1 ..n , Y = ( y j ) j
=1 ..n ⊂ R D be given 2 .
We formally deﬁne the squared
Cramer-Wold distance by the formula
[Equation]
Then
[Equation]
where φ D ( s ) = 1 F 1 ( 12 ; D 2 ; − s ) and 1 F 1 is the Kummer’s conﬂuent hypergeometric function (see, e.g., Barnard et al.
Moreover, φ D ( s ) has the following asymptotic formula valid for D ≥ 20 :
[Equation]
To prove the Theorem 3.1 we will need the following crucial technical proposition.
Proposition 3.1.
Let z ∈ R D and γ > 0 be given.
Then
[Equation]
Proof.
By applying orthonormal change of coordinates without loss of generality we may assume that z = ( z 1 , 0 , . . . , 0) , and then v T z
=
z 1 v 1 for v =
( v 1 , . .
. , v D ) .
Consequently we get
[Equation]
Making use of the formula for slice integration of functions on spheres (Axler et al., 1992, Corollary A.6) we get:
[Equation]
where V K denotes the surface volume of a sphere S K ⊂ R K .
Applying the above equality for the function f ( v 1 , . .
. , v D )
=
N ( z 1 v 1 , γ )(0) and s =
z 21 / (2 γ ) = (cid:107) z (cid:107) 2 / (2 γ ) we consequently get that the LHS of (4) simpliﬁes to
[Equation]
which completes the proof since V
[Equation]
Proof of Theorem 3.1.
Directly from the deﬁnition of smoothing we obtain that
[Equation]
Now applying the one-dimensional formula the the L 2 -scalar product of two Gaussians: (cid:104)
[Equation]
[Equation]
and the equality (cid:107) f − g (cid:107) 22 = (cid:104) f, f (cid:105) 2 + (cid:104) g, g (cid:105) 2 − 2 (cid:104) f, g (cid:105) 2 (where (cid:104) f, g (cid:105)
= (cid:82) f
( x ) g ( x ) dx ), we simplify the squared- L 2 norm in the integral of RHS of (5) to
[Equation]
Applying directly Proposition 3.1 we obtain formula (2).
Proof of the formula for the asymptotics of the function φ D is provided in the Appendix.
Thus to estimate the distance of a given sample X to some prior distribution f , one can follow the common approach and take the distance between X and a sample from f .
As the main theoretical result of the paper we view the following theorem, which says that in the case of standard Gaussian multivariate prior, we can completely reduce the need for sampling (we omit the proof since it is similar to that of Theorem 3.1).
Theorem 3.2.
Let X
=
( x i ) i =1 ..n ⊂ R D be a given sample.
We formally deﬁne
[Equation]
Then
[Equation]
One can easily obtain the general formula for the distance between mixtures of radial distributions.
This follows from the fact that the Cramer-Wold distance is given by a scalar product (cid:104)· , ·(cid:105) cw which has a closed-form for the product of two radial Gaussians:
[Equation]
[Equation]
The above formula means that Cramer-Wold distance is deﬁned by Cramer-Wold kernel, for more details see Appendix, Section A.
C RAMER -W OLD A UTO E NCODER MODEL (CWAE)
This section is devoted to the construction of CWAE.
Since we base our construction on the AutoEncoder, to establish notation let us formalize it here.
AutoEncoder.
Let X =
( x i ) i =1 ..n ⊂ R N be a given data set.
The basic aim of AE is to transport the data to a typically, but not necessarily, less dimensional latent space Z = R D with reconstruction error as small as possible.
Thus, we search for an encoder E : R n → Z and decoder D : Z → R n functions, which minimize the reconstruction error on the data set X :
[Equation]
AutoEncoder based generative model.
CWAE, similarly to WAE, is a classical AutoEncoder model with modiﬁed cost function which forces the model to be generative, i.e. ensures that the data transported to the latent space comes from the (typically Gaussian) prior f .
This statement is formalized by the following important remark, see also (Tolstikhin et al., 2017).
Remark 4.1.
Let X be an N -dimensional random vector from which our data set was drawn, and let Y be a random vector with a density f on latent Z .
Suppose that we have constructed functions E : R N → Z and D : Z → R N (representing the encoder and the decoder) such that
D ( E x ) = x for x ∈ image( X ) , 2.
random vector E X has the distribution f .
Then by the point 1 we obtain that D ( E X ) =
X , and therefore
D Y has the same distribution as D ( E X ) =
X .
This means that to produce samples from X we can instead produce samples from Y and map them by the decoder D .
Since an estimator of the image of the random vector X is given by its sample X , we conclude that a generative model is correct if it has small reconstruction error and resembles the prior distribution in the latent.
Thus, to construct a generative AutoEncoder model (with Gaussian prior), we add to its cost function a measure of distance of a given sample from normal distribution.
CWAE cost function.
Once the crucial ingredient of CWAE is ready, we can describe its cost function.
To ensure that the data transported to the latent space Z are distributed according to the standard normal density, we add to the cost function logarithm 4 of the Cramer-Wold distance from standard multivariate normal density d 2cw ( X, N (0 , I )) :
[Equation]
Since the use of special functions involved in the formula for Cramer-Wold distance might be cumbersome, we apply in all experiments (except for the illustrative 2D case) the asymptotic form (12) of function φ D :
[Equation]
where γ n = ( 43 n ) 2 / 5 is chosen by the Silverman’s rule of thumb (Silverman, 1986).
Comparison with WAE and SWAE models.
Finally, let us brieﬂy recapitulate differences between the introduced CWAE, WAE variants of (Tolstikhin et al., 2017) and SWAE (Kolouri et al., 2018).
In contrast to WAE-MMD and SWAE, CWAE model does not require sampling from normal distribution (as in WAE-MMD) or over slices (as in SWAE) to evaluate its cost function, and in this sense uses a closed formula cost function.
In contrast to WAE-GAN, our objective does not require a separately trained neural network to approximate the optimal transport function, thus avoiding pitfalls of adversarial training.
In this paper we are interested in WAE-MMD and SWAE models, which do not use parameterized distance functions, e.g. trained adversarially like in WAE-GAN .
However, in future work we plan to introduce an adversarial version of CWAE and compare it with WAE-GAN
E XPERIMENTS
In this section we empirically validate the proposed CWAE model on standard benchmarks for generative models: CelebA, Cifar-10 and MNIST.
We will compare CWAE model with WAEMMD (Tolstikhin et al., 2017) and SWAE (Kolouri et al., 2018).
As we will see, our results match those of WAE-MMD, and in some cases improve upon SWAE, while using a simpler to optimize cost function (see the previous section for a more detailed discussion).
The rest of this section is structured as follows.
In Section 5.2 we report results on standard qualitative tests, as well as a visual investigations of the latent space.
In Section 5.3 we will turn our attention to quantitative tests using
Fréchet Inception Distance and other metrics.
5. E XPERIMENTATION SETUP
In the experiment we have used two basic architecture types.
Experiments on MNIST were performed using a feedforward network for both encoder and decoder, and a 20 neuron latent layer, all with
ReLU activations.
In case of CIFAR-10 and CelebA data sets we used convolution-deconvolution architectures.
Please refer to Appendix E for full details.
5. Q UALITATIVE TESTS
[Table]
The quality of a generative model is typically evaluated by examining samples or interpolations.
We present such a comparison between CWAE with WAE-MMD in Figure 2.
We follow the same procedure as in (Tolstikhin et al., 2017).
In particular, we use the same base neural architecture for both
CWAE and WAE-MMD.
We consider for each model (i) interpolation between two random examples from the test set (leftmost in Figure 2), (ii) reconstruction of a random example from the test set (middle column in Figure 2), and ﬁnally a sample reconstructed from a random point sampled from the prior distribution (right column in Figure 2).
The experiment shows that there are no perceptual differences between
CWAE and WAE-MMD generative distribution.
In the next experiment we qualitatively assess normality of the latent space.
This will allow us to ensure that CWAE does not compromise on the normality of its latent distribution, which recall is part of the cost function for all the models except AE.
We compare CWAE 5 with AE, VAE, WAE and SWAE on the MNIST data with using 2-dimensional latent space and a two dimensional Gaussian prior distribution.
Results are reported in
Figure 3.
As is readily visible, the latent distribution of CWAE is as close, or perhaps even closer, to the normal distribution than that of the other models.
Furthermore, the AutoEncoder presented in the second ﬁgure is noticeably different from a Gaussian distribution, which is to be expected because it does not optimize for normality in contrast to the other models.
To summarize, both in terms of perceptual quality and satisfying normality objective, CWAE matches
WAE-MMD.
The next section will provide more quantitative studies.
5. Q UANTITATIVE TESTS
In order to quantitatively compare CWAE with other models, in the ﬁrst experiment we follow the common methodology and use the Fréchet Inception Distance (FID) introduced by Heusel et al.
Further, we evaluate the sharpness of generated samples using the Laplace ﬁlter following Tolstikhin et al.
Results for CWAE and WAE are summarized in Tab. 1.
In agreement with the qualitative studies, we observe FID and sharpness scores of CWAE to be similar to WAE-MMD.
space we apply equality 1 F 1 (1 / 2 , 1 , − s )
=
e − s 2 I 0 (cid:0) s 2 (cid:1) jointly with the approximate formula (Abramowitz & Stegun, 1964, page 378) for the Bessel function of the ﬁrst kind
I 0 , for more details see Appendix C.
Test
Test reconstruction
Random sample
[Figure]
MM
[Figure]
C W
A E
Next, by comparing training time between CWAE and other models, we found that for batch-sizes up to 1024 , which covers the range of batch-sizes used typically for training autoencoders, CWAE is faster (in terms of time spent per batch) than other models.
More precisely, CWAE is approximately 2 × faster up to 256 batch-size.
Details are relegated to the Appendix D.
Finally, motivated by Remark 4.1 we propose a novel method for quantitative assessment of the models based on their comparison to standard normal distribution in the latent.
To achieve this we have decided to use one of the most popular statistical normality tests, i.e. Mardia tests (Henze, 2002).
Mardia’s normality tests are based on verifying whether the skewness b 1 ,D ( · ) and kurtosis b 2 ,D ( · ) of a sample X =
( x i ) i =1 ..n ⊂ R D : b 1 ,D ( X ) = 1 n 2 (cid:88)
j,k
( x Tj x k ) 3 , and b 2 ,D ( X )
= 1 n (cid:88) j (cid:107) x j (cid:107) 4 , are close to that of standard normal density.
The expected Mardia’s skewness and kurtosis for standard multivariate normal distribution is 0 and D ( D + 2) , respectively.
To enable easier comparison in experiments we consider also the value of the normalized Mardia’s kurtosis given by b 2 ,D ( X ) −
D ( D + 2) , which equals zero for the standard normal density.
[Table]
[Figure]
Optimal value of kurtosis, (i.e. for normal distribution) is given by a dash line.
Results are presented in Figure 4 and Table 2.
In Figure 4 we report for CelebA data set the value of FID score, Mardia’s skewness and kurtosis during learning process of AE, VAE, WAE, SWAE and CWAE (measured on the validation data set).
WAE, SWAE and CWAE models obtain the best reconstruction error, comparable to AE.
VAE model exhibits a sightly worse reconstruction error, but values of kurtosis and skewness indicating their output is closer to normal distribution.
As expected, the output of AE is far from normal distribution; its kurtosis and skewness grow during learning.
This arguably less standard evaluation, which we hope will ﬁnd adoption in the community, serves as yet another evidence that CWAE has strong generative capabilities which at least match performance of WAE-MMD.
Moreover we observe that VAE model’s output distribution is closest to the normal distribution, at the expense of the reconstruction error, which is reﬂected by the blurred reconstructions typically associated with VAE model.
Moreover, motivated by the above approach based on normality tests 6 we have veriﬁed how the
Cramer-Wold metric works as a Gaussian goodness of ﬁt, however, the results were not satisfactory.
The tests based on Cramer-Wold metric were, in general, in the middle of compared tests (Mardia,
Henze-Zirkler and Royston tests).
On the whole, WAE-MMD and CWAE achieve, practically speaking, the same level of performance in terms of FID score, sharpness, and our newly introduced normality test.
Additionally, CWAE fares better in many of these metrics than SWAE.
C ONCLUSIONS
In the paper we have presented a new autoencoder based generative model CWAE, which matches results of WAE-MMD, while using a cost function given by a simple closed analytic formula.
We hope this result will encourage future work in developing simpler to optimize analogs of strong neural models.
Crucial in the construction of CWAE is the use of the developed Cramer-Wold metric between samples and distributions, which can be effectively computed for Gaussian mixtures.
As a consequence we obtain a reliable measure of the divergence from normality.
Future work could explore use of the
Cramer-Wold distance in other settings, in particular in adversarial models
A PPENDICES A C RAMER -W OLD KERNEL
In this section we ﬁrst formally deﬁne the Cramer-Wold metric, and later show that it is given by a characteristic kernel which has closed-form for spherical Gaussians.
For more information on the kernels, and in general kernel embedding of distributions we refer the reader to (Muandet et al., 2017).
Let us ﬁrst introduce the general deﬁnition of the cw -metric.
To do so we generalize the notion of smoothing for arbitrary measures µ by the formula:
[Equation]
where ∗ denotes the convolution operator for two measures, and we identify the normal density
N (0 , γI ) with the measure it introduces.
It is well-known that the resulting measure has the density given by
[Equation]
Clearly
[Equation]
Moreover, by applying the characteristic function one obtains that if the smoothing of two measures coincide, then the measures also coincide:
[Equation]
We also need to deﬁne the transport of the density by the projection x
→ v T x , where v is chosen from the unit sphere S D .
The deﬁnition is formulated so that if X is a random vector with density f , then f v is the density of the random vector X v :
= v T X .
Then we have
[Equation]
where d D − 1 denotes the D − 1 -dimensional Lebesgue measure.
In general, if µ is a measure on R D , then µ v is the measure deﬁned on R by the formula
[Equation]
Since, if a random vector X has the density N ( a, γI ) , then the random variable X v has the density
N ( v T a, α ) , we directly conclude that
[Equation]
It is also worth noticing, that due to the fact that the projection of a Gaussian is a Gaussian, the smoothing and projection operators commute, i.e.:
[Equation]
Given ﬁxed γ > 0 , the two above notions allow us to formally deﬁne the cw -distance of two measures µ and ν by the formula
[Equation]
First observe that this implies that cw -distance is given by the kernel function
[Equation]
Let us now prove that the function d cw deﬁned by equation 10 is a metric (which, in the kernel function literature means that the kernel is characteristic).
Theorem A.1.
The function d cw is a metric.
Proof.
Since d cw comes from a scalar product, we only need to show that if a distance of two measures is zero, the measures coincide.
So let µ, ν be given measures such that d cw ( µ, ν ) = 0 .
This implies that
[Equation]
By equation 9 this implies that µ v = ν v .
Since this holds for all v ∈ S D , by the Cramer-Wold
Theorem we obtain that µ = ν .
Thus we can summarize the above by saying that the Cramer-Wold kernel is a characteristic kernel which has the closed-form the scalar product of two radial Gaussians given by equation 7:
[Equation]
Remark A.1.
Observe, that except for the Gaussian kernel it is the only kernel which has the closed form for the spherical Gaussians, which as we discuss in the next section is important, as the RBF (Gaussian) kernels cannot be successfully applied in AutoEncoder based generative models.
The reason is that the derivative of Gaussian decreases to fast, and therefore it does not enforce the proper learning of the model, see also the comments in (Tolstikhin et al., 2017, Section 4, WAE-GAN and WAE-MMD speciﬁcs).
B CWAE VS WAE-MMD
In this section we are going to compare CWAE model to WAE-MMD.
In particular we show that
CWAE can be seen as the intersection of the sliced-approach together with MMD-based models.
Since both WAE and CWAE use kernels to discriminate between sample and normal density, to compare the models we ﬁrst describe the WAE model.
WAE cost function for a given characteristic kernel k and sample X
=
( x i ) i =1 ..n ⊂ R D (in the
D -dimensional latent) is given by
[Equation]
where Y =
( y i ) i
=1 ..n is a sample from the standard normal density N (0 , I ) , and d 2 k ( X, Y ) denotes the kernel-based distance between the probability distributions representing X and Y , that is 1 n (cid:80) i
δ x i and 1 n (cid:80) i
δ y i , where δ z denotes the atom Dirac measure at z ∈ R D .
The inverse multiquadratic kernel k is chosen as default
[Equation]
where in experiments in (Tolstikhin et al., 2017) a value C = 2 Dσ 2 was used, where σ is the hyper-parameter denoting the size of the normal density.
Thus the model has hyper-parameters λ and σ , which were chosen to be λ = 10 , σ 2 = 1 in MNIST, λ = 100 , σ 2 = 2 in CelebA. Observe that the hyper-parameters do not depend on the sample size and that, in general, the WAE-MMD model hyper-parameters have to be chosen by hand.
Now let us describe the CWAE model.
CWAE cost function for a sample X =
( x i ) i =1 ..n ⊂ R D (in the D -dimensional latent) is given by
[Equation]
where distance between the sample and standard normal distribution is taken with respect to the
Cramer-Wold kernel with a regularizing hyperparameter γ given by the Silverman’s rule of thumb (the motivation for such a choice of hyper-parameters is explained in Section 2).
Thus, we have the following differences:
• Due to the properties of Cramer-Wold kernel, in the distance we are able to substitute the sample estimation of d 2 k ( X, N (0 , I ))
given in WAE-MMD by d 2cw ( X, Y ) by its exact formula.
• CWAE, as compared to WAE, has no hyper-parameters:
In our preliminary experiments we have observed that in many situations (like in the case of log-likelihood), taking the logarithm of the nonnegative factors of the cost function, which we aim to minimize to zero, improves the learning process.
Motivated by this, instead of taking the additional weighting hyper-parameter λ (as in WAEMMD), whose aim is to balance the MSE and divergence terms, we take the logarithm of the divergence.
Automatically (independently of dimension) balance those terms in the learning process.
The choice of regularization hyper-parameter is given by the Silverman’s rule of thumb, and depends on the sample size (contrary to WAE-MMD, where the hyper-parameters are chosen by hand, and in general do not depend on the sample size).
Summarizing, in CWAE model, contrary to WAE-MMD, we do not have to choose hyper-parameters.
Moreover, since we do not have the noise in the learning process given by the random choice of the sample Y from N (0 , I ) , the learning should be more stable.
As a consequence, see Figure 7, CWAE in generally learns faster then WAE-MMD, and has smaller standard deviation of the cost-function during the learning process.
C C OMPUTATION OF φ D
In this section we consider the estimation of values of the function
[Equation]
which is crucial in the formulation for the Cramer-Wold distance.
First we will provide its approximate asymptotic formula valid for dimensions D ≥ 20 , and then we shall consider the special case of
D = 2 (see Figure 5)
To do so, let us ﬁrst recall (Abramowitz & Stegun, 1964, Chapter 13) that the Kummer’s conﬂuent hypergeometric function 1 F 1 (denoted also by M ) has the following integral representation
[Equation]
valid for a, b > 0 such that b > a .
Since we consider that latent is at least of dimension D ≥ 2 , it follows that
[Equation]
By making a substitution u = x 2 , du = 2 xdx , we consequently get
[Equation]
Proposition C.1.
For large 7 D we have
[Equation]
Proof.
By (11) we have to estimate asymptotics of
[Equation]
Since for large D , for all x ∈ [ − 1 , 1] we have
[Equation]
we get φ D ( s ) ≈
[Equation]
To simplify the above we apply the formula (1) from (Tricomi & Erdélyi, 1951):
Γ( z + α )
[Equation]
with α, β ﬁxed so that α + β = 1 (so only the error term of order O ( | z | − 2 ) remains), and get Γ( D 2 )
Γ( D − 1 2 ) =
Γ(( D 2 − 34 ) + 34 ) Γ(( D 2 − 34 ) + 14 ) ≈ (cid:18) D 2 −  (cid:19)and
Γ( s + D 2 − 12 ) Γ( s + D 2 )
≈ (cid:18) s
+ D 2 − 3(cid:19)
−.
[Equation]
Summarizing,
[Equation]
In general one can obtain the iterative direct formulas for function φ D with the use of erf and modiﬁed
Bessel functions of the ﬁrst kind I
and I 1 , but for large D they are of little numerical value.
We consider here only the special case D = 2 since it is used in the paper for illustrative reasons in the latent for the MNIST data set.
Since we have the equality (Gradshteyn & Ryzhik, 2015, (8.406.3) and (9.215.3)):
[Equation]
to practically implement φ 2 we apply the approximation of I 0 from (Abramowitz & Stegun, 1964, page 378) given in the following remark.
Remark C.1.
Let s ≥ 0 be arbitrary and let t = s/ 7 . 5 .
Then φ 2 ( s ) ≈  
[Equation]
for s ∈ [0 , 7 . 5]
[Equation]
for s ≥ 7 . 5 .
D C OMPARISON OF LEARNING TIMES
Figure 6 gives comparison of mean learning time for different most frequently used batch-sizes.
Time spent on processing a batch is actually smaller for CWAE for a practical range of batch-sizes [32 , 512] .
For batch-sizes larger than 1024 , CWAE is slower due to its quadratic complexity with respect to the batch-size.
However, we note that batch-sizes larger even than 512 are relatively rarely used in practice for training autoencoders.
[Figure]
size
A v e r a g e b a t c h p r o c e ss i n g t
i m e
AE VAE WAE
SWAE CWAE
E A RCHITECTURE DETAILS
MNIST ( 28 × 28 images) an encoder-decoder feedforward architecture: encoder three feed-forward ReLU layers, 200 neurons each latent 20-dimensional, decoder three feed-forward ReLU layers, 200 neurons each.
CelebA (with images centered and cropped to 64 × 64 with 3 color layers) a convolution-deconvolution network: encoder four convolution layers with 4 × 4 ﬁlters and 2 × 2 strides (consecutively 32, 32, 64, and 64 output channels), all ReLU activations, two dense layers (1024 and 256 ReLU neurons) latent 64-dimensional, decoder ﬁrst two dense layers (256 and 1024 ReLU neurons), three transposed-convolution layers with 4 × 4 ﬁlters with 2 × 2 strides (consecutively 64, 32, 32 channels) with ReLU activation, transposed-convolution 4 × 4 with 2 × 2 stride, 3 channels, and sigmoid activation.
CIFAR-10 dataset ( 32 × images with three color layers): a convolution-deconvolution network encoder four convolution layers with 2 × 2 ﬁlters, the second one with 2 × 2 strides, other non-strided (3, 32, 32, and 32 channels) with ReLU activation, 128 ReLU neurons dense layer, latent with 64 neurons, decoder two dense ReLU layers with
128 and 8192 neurons, two transposed-convolution layers with 2 × 2 ﬁlters (32 and 32 channels) and ReLU activation, a transposed convolution layer with 3 × 3 ﬁlter and 2 × 2 strides (32 channels) and ReLU activation, a transposed convolution layer with 2 × 2 ﬁlter (3 channels) and sigmoid activation.
The last layer returns the reconstructed image.
The results for all above architectures are given in
Table 2.
All networks were trained with the Adam optimizer (Kingma & Ba, 2014).
The hyperparameters used were learning rate = 0 .
001 , β 1 = 0 .
9 , β 2 = 0 .
999 , (cid:15) = 1 e − 8 .
MNIST models were trained for 500 epochs, both CIFAR-10 and CelebA for 200.
Additionally, to have a direct comparison to WAE-MMD model on CelebA, an identical architecture was used as that in Tolstikhin et al.
(2017) utilized for the WAE-MMD model (WAE-GAN architecture is, naturally, different):
encoder four convolution layers with 5 × 5 ﬁlters, each layer followed by a batch normalization (consecutively 128, 256, 512, and 1024 channels) and ReLU activation, latent 64-dimensional, decoder dense 1024 neuron layer, three transposed-convolution layers with 5 × 5 ﬁlters, and each layer followed by a batch normalization with ReLU activation (consecutively 512, 256, and 128 channels), transposed-convolution layer with 5 × 5 ﬁlter and 3 channels, clipped output value.
The results for this architecture for CWAE compared to VAE and WAE-MMD models are given in
Table 1.
Similarly to Tolstikhin et al.
(2017), models were trained using Adam with for 55 epochs, with the same optimizer parameters.
F T RAINING RESULTS FOR THE C ELEB A DATASET
[Figure]
[Figure]
G T RAINING RESULTS FOR THE CIFAR10 DATASET
[Figure] [Figure]
H A DDITIONAL FIGURES
Test interpolation
Test reconstruction
Random sample
VA E
W
[Figure]
E
[Figure]
MM
D
S W
[Figure]