L ATENT V ARIABLES ON S PHERES FOR S AMPLING AND S PHERICAL I NFERENCE
Anonymous authors
Paper under double-blind review
A BSTRACT
Variational inference is a fundamental problem in Variational Auto-Encoder (VAE).
By virtue of high-dimensional geometry, we propose a very simple algorithm completely different from existing ones to solve the inference problem in VAE.
We analyze the unique characteristics of random variables on spheres in high dimensions and prove that the Wasserstein distance between two arbitrary datasets randomly drawn from a sphere are nearly identical when the dimension is sufﬁciently large.
Based on our theory, a novel algorithm for distribution-robust sampling is devised.
Moreover, we reform the latent space of VAE by constraining latent random variables on the sphere, thus freeing VAE from the approximate optimization pertaining to the variational posterior probability.
The new algorithm is named as Spherical Auto-Encoder (SAE), which is in essence the vanilla autoencoder with the spherical constraint on the latent space.
The associated inference is called the spherical inference, which is geometrically deterministic but is much more robust to various probabilistic priors than the variational inference in VAE for sampling.
The experiments on sampling and inference validate our theoretical analysis and the superiority of SAE.
I NTRODUCTION
Deep generative models, such as Variational Auto-Encoder (VAE) (Kingma & Welling, 2013; Rezende et al., 2014) and Generative Adversarial Network (GAN) (Goodfellow et al., 2014), play more and more important role in machine learning and computer vision.
However, the problem of variational inference in VAE is still challenging, especially for high-dimensional data like images.
To be formal, let X = { x 1 , . . .
, x n } denote the set of observable data points and Z = { z 1 , . . .
, z n } the set of desired latent vectors, where x i ∈ R
d x and z i ∈ R d z .
Let p g
( x | z ) denote the likelihood of generated sample conditioned on latent variable z and p ( z ) the prior, where g denotes the decoder .
The encoder f in VAE parameterizes the variational posterior q f ( z | x ) in light of the lower bound of the marginal log-likelihood
[Equation]
The ﬁrst term D KL [ q f ( z | x ) || p ( z )] constrains the encoded latent codes to the prior via the KLdivergence, and the second term E q [log p g ( x | z )] serves to guarantee the reconstruction accuracy of inputs.
For a Gaussian p g ( x | z ) of diagonal covariance matrix, log p g ( x | z ) reduces to the varianceweighted squared error (Doersch, 2016).
The lower-bound approximation of the log-likelihood provides a feasible solution for VAE.
But it also causes new problems.
For example, the generated sample g ( z ) deviates from the real distribution of X when sampling from the given prior due to that the learnt q f ( z | x ) is incapable of matching the prior distribution well.
Besides, the reconstruction g ( f ( x )) is not satisfactory either.
For imagery data, bluriness usually occurs.
In order to manipulate real images for GAN models, we usually need to formulate an encoder via the framework of VAE.
The variational inference also applies in this scenario.
The problems of VAE are the obstacles of putting the GAN encoder in the right way either.
There are other methods of learning
[Figure]
an encoder for GAN in the adversarial way such as (Dumoulin et al., 2017; Li et al., 2017; Ulyanov et al., 2017; Heljakka et al., 2018).
However, the structural precision of reconstruction is generally inferior to the VAE framework, because the high-level semantics of objects outweigh low-level structures for such methods (Donahue & Simonyan, 2019).
Besides, the concise architecture of VAE is more preferred in this scenario.
Therefore, learning precise latent variables z = f ( x ) is critical to applications of VAE and GAN.
Using a different theory in this paper, we propose a simple method to circumvent the problem.
Our contributions are summarized as follows.
We introduce the volume concentration of highdimensional spheres.
Based on the concentration property, we point out that projecting on a sphere for data that are distributed according to the spherical mass produces little difference from the viewpoint of the volume in high-dimensional spaces.
Thus, it is plausible to perform inference pertaining to VAE on the sphere.
We further analyze the probability distribution of distances between two arbitrary sets of random points on the sphere in high dimensions and illustrate the phenomenon of distance convergence.
Furthermore, we prove that the Wasserstein distance between two arbitrary datasets randomly drawn from a high-dimensional sphere are nearly identical, meaning that the data on the sphere are distribution-robust for generative models with respect to Wasserstein distance.
Based on our theoretical analysis, we propose a very simple algorithm for sampling generative models.
The same principle is also harnessed to reformulate VAE.
The spherical normalization is simply put on latent variables instead of variational inference while preserving randomness of latent variables by centerization.
In contrast to VAE and variational inference, we name such an autoencoder as Spherical Auto-Encoder (SAE) and the associated inference as spherical inference.
We perform extensive experiments to validate our theoretical analysis and claims with sampling and inference.
L ATENT VARIABLES ON SPHERE
For latent variables or data points sampled from some priors, the projection on the unit sphere can can be easily performed by
[Equation]
This spherical normalization for priors fed into the generator is employed in StyleGAN that is the phenomenal algorithm in GANs (Karras et al., 2018b).
To test the robustness of StyleGAN against the diverse distributions, we conduct two groups of experiments with the input z sphere-normalized and not sphere-normalized when training StyleGAN.
As shown in Figure 1, the diversity of generated
[Figure]
faces is good for two different distributions with normalized z , whereas the face modes become similar for the case of the uniform distribution that z is not normalized.
This experiment indicates that StyleGAN with sphere-normalized z is much more robust to the variation of variable modes from different distributions.
Inspired by this comparison, we interpret the beneﬁt of using random variables on spheres by virtue of high-dimensional geometry in this section.
Based on these theories, a novel algorithm is proposed for random sampling and spherical inference for GAN and VAE.
2. V OLUME CONCENTRATION
For high-dimensional spaces, there are many counter-intuitive phenomena that will not happen in low-dimensional spaces.
For a convenient analysis, we assume that the center of the sphere S d embedded in R d +1 is at the origin.
We ﬁrst present the concentration property of sphere volume in R d + .
One can ﬁnd the proof in (Blum et al., 2020).
Theorem 1.
Let V ( r ) and V ((1 − (cid:15) ) r ) denote the volumes of the two concentric spheres of radius r and (1 − (cid:15) ) r , respectively,
where 0 < (cid:15) < 1 .
Then
[Equation]
And if (cid:15) =
t/d , V ((1 − (cid:15) ) r ) /V
( r ) → e − t when d → ∞ , where t is a constant.
Theorem 1 says that the volume of the d -dimensional sphere of radius (1 − (cid:15) ) r rapidly goes to zero when d goes large, meaning that the interior of the high-dimensional sphere is empty.
In other words, nearly all the volume of the sphere is contained in the thin annulus of width (cid:15)r = rt/d .
The width becomes very thin when d grows.
For example, the annulus of width that is 0 .
9% of the radius contains 99% of all the volume for the sphere in R 512 .
To help understand this counter-intuitive geometric property, we make a schematic illustration in Figure 2.
The probabilistic operations can be very beneﬁcial from the volume concentration of spheres.
Suppose that we perform the probabilistic optimization pertaining to latent variables sampled according to the distribution of the sphere volume.
The probability mass is also empty due to the volume concentration.
Therefore, the error is controllable if we perform it on the sphere, provided that these latent variables lie in high-dimensional spaces.
For VAE, therefore, we can write a feasible approximation
[Equation]
where Int ( S d ) denotes the interior of S d , (cid:107) z (cid:107) ≤ r , and d is sufﬁciently large.
The spherical approximation for log p g ( x ) is an alternative scheme except the lower bound approximation presented in equation (1).
In fact, the distributions deﬁned on the sphere have been already exploited to reformulate VAE, such as the von Mises-Fisher distribution (Davidson et al., 2018; Xu & Durrett, 2018).
But the algorithms proposed in (Davidson et al., 2018; Xu & Durrett, 2018) still fall into the category using the variational inference like the vanilla VAE.
To eliminate this constraint, we need more geometric analysis presented in the following section.
2. D ISTANCE CONVERGENCE
To dig deeper, we examine the pairwise distance between two arbitrary points randomly sampled on S d .
The following important lemma was proved by (Lord, 1954; Lehnen & Wesenberg, 2002).
[Figure]
A v e r age d
i s t an c e and s t d
[Figure] [Figure]
Lemma 1.
Let ξ denote the Euclidean distance between two points randomly sampled on the sphere S d of radius r .
Then the probability distribution of ξ is
[Equation]
where the coefﬁcient c ( d ) is given by c ( d ) = √ π Γ (cid:0) d − 12 (cid:1) /
Γ (cid:0) d 2 (cid:1) .
And the mean distance ξ µ and the standard deviation ξ σ are
[Equation]
and
[Equation]
respectively, where Γ is the Gamma function.
Furthermore, ξ µ → √ 2 r (cid:0) [Equation] and ξ σ → r
[Equation]
when d goes large.
Lemma 1 tells that the pairwise distances between two arbitrary points randomly sampled on S d approach to be mutually identical and converge to the mean ξ µ = √ 2 r when d grows.
The associated standard deviation ξ σ → 0 .
This result is in some extent surprising compared to the intuition in low-dimensional spaces.
We display the average distance and its standard deviation in Figure 3, showing that the convergence process is fast.
Taking R 512 for example, we calculate that ξ µ = 1 .and ξ σ = 0 . 0313 .
The standard deviation is only 2 .
21% of the average distance, meaning that the distance discrepancy between arbitrary z i and z j is rather small.
This surprising phenomenon is also observed for neighborly polytopes when solving the sparse solution of underdetermined linear equation (Donoho, 2005) and for nearest neighbor search in high dimensions (Beyer et al., 1999).
With Lemma 1, we can study the property of two different random datasets on S d , which serves to distribution-free sampling and spherical inference in generative models.
To this end, we ﬁrst introduce the computational deﬁnition of Wasserstein distance.
Let Z = { z 1 , . . .
, z n } and Z (cid:48)
= { z (cid:48) 1 , . . .
, z (cid:48) n } be the datasets of random variables drawn from S d at random , respectively.
Then the 2 -Wasserstein distance is deﬁned as
[Equation]
where ω is the doubly stochastic matrix.
By Lemma 1, it is not hard to derive the following Theorem.
Theorem 2.
1 W 2 ( Z , Z (cid:48) )
→ √ 2 nr with zero standard deviation when d → ∞ .
Theorem 2 says that despite the diverse distributions, the 2 -Wasserstein distance between two arbitrary sets of random variables on the sphere converges to a constant when the dimension is sufﬁciently large.
For generative models, this unique characteristic for datasets randomly sampled from highdimensional spheres bring great convenience for distribution-robust sampling and spherical inference.
For example, if Z and Z (cid:48) obeys the different distributions, the functional role of Z (cid:48) nearly coincides with that of Z with respect to Wasserstein distance, provided that both Z and Z (cid:48) are randomly drawn from the high-dimensional sphere.
The speciﬁc distributions of Z and Z (cid:48) affect the result negligibly under such a condition.
We will present the speciﬁc application of Theorem 2 in the following section.
In fact, we can obtain the bounds of W 2 ( Z , Z (cid:48) ) using the proven proposition about the nearlyorthogonal property of two random points on high-dimensional spheres (Cai et al., 2013).
However,
Theorem 2 is sufﬁcient to solve the problem raised in this paper.
So, we bypass this analysis to simplify the theory for easy readability.
A LGORITHM FOR SAMPLING AND INFERENCE
We will detail the distribution-robust algorithms for sampling and spherical inference.
A new simpliﬁed form of VAE will be presented in this section as well.
3. S AMPLING
To acquire generative results from VAEs and GANs, we need to sample random vectors from a pre-deﬁned prior and then feed them into the decoder or the generator.
According to Theoremand Theorem 2, however, this prior limitation can be eliminated if we project these random vectors on the corresponding sphere.
To achieve this, we perform two-step manipulations on the sampled dataset from arbitrary prior distributions.
The procedure is detailed in Algorithm 1.
The centerization operation is motivated from the central limit theorem in probability, which transforms the distributionspeciﬁc Z to be nearly distribution-agnostic (on the sphere).
The spherization is to project these centerized vectors on the unit sphere.
We ﬁnd that in practice, this simple algorithm works well to reduce the bias caused by various distributions or data modes for VAEs and GANs.
1: Sample Z = { z 1 , . . . , z n } ∼ P ( z ) (cid:46) P ( z ) is an arbitrary distribution 2: Centerization by z ji ← z ji − 1 d (cid:80) j z ji for each z
i (cid:46)
z ji is the j -th entry of z i
According to Theorem 2, we may know that sampling is robust to random variables if they are randomly sampled from the high-dimensional sphere.
Theorem 1 guarantees that the error can be negligible even if they deviate from the sphere, as long as they are distributed near the spherical surface.
This tolerance to various modes of random variables allow us to devise a simple solution to replace the variational inference for VAE, i.e. the spherical inference we call.
To be speciﬁc, we only need to constrain the centerized latent variables on the sphere, as opposed to the conventional way of employing the KL-divergence D KL [ q f ( z | x ) || p ( z )] and its variants with diverse priors.
The sequential mappings of the autoencoder under our framework can be shown by
[Equation]
[Equation]
where ˆ z = 1 d (cid:80) j z ji and 1 is the all-one vector.
We can write the objective function for this type of autoencoder as
[Equation]
(11) where (cid:96) p denotes the p -norm.
The objective function of our algorithm is much simpler than that of
VAE and its variants based on the variational inference or various sophisticated regularizers on the latent space.
It is clear that we utilize the geometric characteristics of latent variables on the sphere rather than some additional losses to optimize the latent space.
Our algorithm is geometric and free from the probability optimization whose performance is usually limited with the approximation dilemma.
In fact, the framework of VAE in (10) reduces to a standard autoencoder with the spherical constraint.
There is no variational inference needed here.
To highlight this critical difference, we call our algorithm Spherical Auto-Encoder (SAE).
R ELATED WORK
Little attention has been paid on examining geometry of latent spaces in the ﬁeld of generative models.
So we ﬁnd few works directly related to ours.
Most relevant ones are the application of von Mises-Fisher (vMF) distribution as the probability prior (Davidson et al., 2018; Xu & Durrett, 2018).
The vMF distribution is deﬁned on the sphere.
The sampling and variational inference are both performed with latent variables drawn on the sphere.
However, the algorithms proposed in (Davidson et al., 2018; Xu & Durrett, 2018)
both rely on the variational inference as VAE does with inequality (1).
For our algorithm, the whole framework is deterministic and there is no approximation involved for inferring latent codes.
For sampling, our geometric analysis is directly inspired by ProGAN Karras et al.
(2018a)
and
StyleGAN (Karras et al., 2018b) that have already applied the spherical normalization for sampled inputs.
We study the related theory and extend the case to arbitrary distributions for both GANs and VAEs.
Another related method is to sample priors along the great circle when performing the interpolation in the latent space for GANs (White, 2016).
The empirical results show that such sampling yields more smooth interpolated generation.
This algorithm is perfectly compatible with our theory and algorithm.
Therefore, it can also be harnessed in our algorithm when performing interpolation as well.
Wasserstein Auto-Encoder (WAE) (Tolstikhin et al., 2018) is an alternative way of optimizing the model distribution and the prior distribution using Wasserstein distance. SAE is different from WAE because we do not really use Wasserstein distance for computation in the latent space. We just leverage Wasserstein distance to establish Theorem 2 for the theoretical analysis. Adversarial AutoEncoder (AAE) (Makhzani et al., 2015) is another interesting method of replacing the variational inference with adversarial learning in the latent space.
But both WAE and AAE need some priors to match, which are essentially different from SAE.
β -VAE (Higgins et al., 2017) improves the ﬂexibility of VAE by using a regularization coefﬁcient to modulate the capacity of latent information.
However, β -VAE is restricted by priors like VAE.
E XPERIMENT
We conduct the experiments to test our theory and algorithms in this section.
Three aspects pertaining to generative algorithms are taken into account, including sampling GANs, learning the variants of autoencoder, and sampling the decoders.
The FFHQ dataset (Karras et al., 2018b) is a more complex face dataset with large variations of faces captured in the wild.
We test VAE and our SAE algorithm with this benchmark dataset.
We use the image size of 128 × 128 , which is larger than the commonly chosen size in the related work and also more challenging than 64 × 64 or 32 × 32 for (variational) autoencoders to reconstruct.
5. S AMPLING GAN
Our ﬁrst experiment is to test the sampling effect using four distributions.
We employ StyleGAN trained with random variables sampled from the normal distribution.
The other three distributions are opted to test the generation with different priors for training, i.e. the uniform, Poisson, and Chi-squared distributions.
The shapes of these three distributions are signiﬁcantly distinctive from that of the normal distribution.
Thus, the generalization capability of the generative model can be effectively unveiled when fed with priors that are not involved during training.
We follow the experimental protocol in (Karras et al., 2018a;b) that StyleGAN is trained on the FFHQ face dataset and Fréchet inception distance (FID) (Borji, 2018) is used as the quality metrics of generative results. We take d z = 512 , which is set in StyleGAN.
This dimension is also used for both VAE and SAE for face data.
From Table 1, we can see that the generative results by the normal distribution is signiﬁcantly better than the others when tested with the original samples.
The uniform distribution is as good as the normal distribution when projected on the sphere.
This is because the values for each random vector are overall symmetrically distributed according to the origin.
They satisfy the condition in Theorem 2 after the spherical projection.
The accuracy of Poisson and Chi-squared distributions is considerably improved after centerization, even better than the vanilla uniform distribution. But the
[Table] accuracy difference between all the compared distributions is rather negligible after centerization and spherization, empirically verifying the theory presented in Theorem 2.
5. A UTOENCODERS
We compare the vanilla VAE with the normal distribution (Kingma & Welling, 2013) with our SAE algorithm for reconstruction and sampling tasks 2 .
From Figure 4, we can see that the face quality of SAE outperforms that of VAE.
The imagery details like semantic structures are preserved much better for SAE.
For example, the sunglasses in the sixth image is successfully recovered by SAE, whereas VAE distorts the face due to this occlusion.
It is worth emphasizing that the blurriness for images reconstructed by SAE is much less than that by VAE, implying that the spherical inference is superior to the variational inference in VAE.
The different accuracy measurements in Table 2 also indicate the consistently better performance of SAE.
To test the generative capability of the models, we also perform the experiment of sampling the decoders as done in section 5.1.
Prior samples are drawn from the normal, uniform, Poisson, and Chisquared distributions, respectively, and then fed into the decoders to generate faces.
Figure 5 illustrates the generated faces of signiﬁcantly different quality with respect to four types of samplings.
The style of the generated faces by SAE keeps consistent, meaning that SAE is rather robust to different probability priors.
This also empirically veriﬁes the correctness of Theorem 2 by solving the real problem.
As a comparison, the quality of the generated faces by VAE varies with probability priors.
In other words, VAE is sensitive to the outputs of the encoder with the variational inference, which is probably the underlying reason of the difﬁculty of training VAE with sophisticated architectures. We also present the experimental results on MNIST and CelebA in Appendix. [Table] [Figure]
C ONCLUSION
In this paper, we attempt to address the issue of the variational inference in VAE and the limitation of prior-sensitive sampling in GAN.
By analyzing the geometry of volume concentration and distance convergence on the high-dimensional sphere, we prove that the Wasserstein distance converges to be a constant for two datasets randomly sampled from the sphere when the dimension goes large.
Based on this unique characteristic, we propose a very simple algorithm for sampling and spherical inference.
The sampled data from priors are ﬁrst centerized and then projected onto the unit sphere before being fed into decoders (or generators).
Such random variables on the sphere are robust to the diverse prior distributions.
With our theory, the vanilla VAE can be reduce to a standard autoencoder with the spherical constraint on the latent space.
In other words, the conventional variational inference in VAE is replaced by the simple operations of centerization and spherization.
The new autoencoder is named as Spherical Auto-Encoder (SAE).
The experiments on the FFHQ face data validate the effectiveness of our new algorithm for sampling and spherical inference.
It is worth noting that the applications of our theory and the novel algorithm are not limited for VAEs and GANs.
Interested readers may explore the possibility in their scenarios.
A.
R ECONSTRUCTION ON FFHQ
[Figure]
A.
[Figure]
S AMPLING VAE AND SAE ON FFHQ
[Figure]
[Figure]
A. R ECONSTRUCTION ON C ELEB A
[Figure] [Figure] AMPLING [Figure]
C ELEB A
[Figure]
A. R ECONSTRUCTION ON MNIST
[Figure]
[Figure]
[Figure]
AMPLING S
[Figure]
VAE
[Figure]
VAE
[Figure] SAE ON MNIST
[Figure]
[Figure]
[Figure]