A G EOMETRIC P ERSPECTIVE ON V ARIATIONAL A UTOENCODERS
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we propose a geometrical interpretation of the Variational Autoencoder framework.
We show that VAEs naturally unveil a Riemannian structure of the learned latent space.
Moreover, we show that using these geometrical considerations can signiﬁcantly improve the generation from the vanilla VAE which can now compete with more advanced VAE models on four benchmark data sets.
In particular, we propose a new way to generate samples consisting in sampling from the uniform distribution deriving intrinsically from the Riemannian manifold learned by a VAE.
We also stress the proposed method’s robustness in the low data regime which is known as very challenging for deep generative models.
Finally, we validate the method on a complex neuroimaging data set combining both high dimensional data and low sample sizes.
I NTRODUCTION
Variational Autoencoders (Kingma & Welling, 2014; Rezende et al., 2014) are powerful generative models that map complex input data into a much lower dimensional space referred to as the latent space while driving the latent variables to follow a given prior distribution.
Their simplicity to use in practice has made them very attractive models to perform various tasks such as high-ﬁdelity image generation (Razavi et al., 2020), speech modeling (Blaauw & Bonada, 2016), clustering (Yang et al., 2019) or data augmentation (Chadebec et al., 2021).
Nonetheless, when taken in their simplest version, it was noted that these models produce most of the time blurry samples.
This undesired behavior may be due to several limitations of the VAE framework.
First, the training of a VAE aims at maximizing the Evidence Lower BOund (ELBO) which is only a lower bound on the true likelihood and so does not ensure that we are always actually improving the true objective (Burda et al., 2016;
Alemi et al., 2016; Higgins et al., 2017;
Cremer et al., 2018; Zhang et al., 2018).
Second, the prior distribution used as regularization and for sampling may be too simplistic (Dai & Wipf, 2018) leading to poor data generation and there exists no guarantee that the actual distribution of the latent code will match a given prior distribution inducing over-regularization (Connor et al., 2021).
Hence, trying to tackle those limitations through richer posterior distributions (Salimans et al., 2015; Rezende & Mohamed, 2015) or better priors (Tomczak & Welling, 2018) represents a major part of the proposed improvements over the past few years.
However, the tractability of the ELBO constrains the choice in distributions and so ﬁnding a trade-off between model expressiveness and tractability remains crucial.
In this paper, we take a rather different approach and propose to focus on the geometrical aspects a vanilla VAE is able to capture within its latent space.
In particular, we propose the following contributions:
• We show that VAEs unveil naturally a latent space having a geometrical structure which can be modelled as a Riemannian manifold through the learned covariance matrices in the posterior distributions.
• We propose a natural sampling scheme consisting in sampling from a uniform distribution deﬁned on the learned manifold and given by the metric.
We show that this procedure improves signiﬁcantly the generation process from a vanilla VAE and makes it able to perform as well as more advanced VAE models in terms of Frechet Inception Distance
(Heusel et al., 2017) and Precision and Recall distribution (Sajjadi et al., 2019) scores on four benchmark data sets.
• We also show that the propose method appears more robust to data set size changes and outperforms even more strongly peers when only smaller sample sizes are considered.
• We validate the method on complex neuroimaging data from the OASIS database (Marcus et al., 2007).
V ARIATIONAL A UTOENCODERS
Considering that we are given x ∈ X a set of data points deriving from an unknown distribution p ( x ) , the VAE framework aims at inferring such a distribution with a parametric model { p θ , θ ∈ Θ } using a maximum likelihood estimator.
The main assumption behind the VAE is to assume that the generation process also involves latent variables
z living in a lower dimensional space such that the generative model writes
[Equation]
where q prior is a prior distribution over the latent variables often taken as a standard Gaussian and p θ ( x | z ) is referred to as the decoder and is most of the time taken as a parametric distribution the parameters of which are estimated using neural networks.
Hence, the likelihood p θ writes:
[Equation]
As this integral is most of the time intractable so is p θ ( z | x ) , the posterior distribution.
Hence, Variational Inference (Jordan et al., 1999) is used and a simple parametrized variational distribution q φ ( z | x ) is introduced to approximate the posterior p θ ( z | x ) .
q φ ( z | x ) is referred to as the encoder and, in the vanilla VAE, q φ is chosen as a multivariate Gaussian whose parameters µ φ and Σ φ are again given by neural networks.
An unbiased estimate of the likelihood p θ ( x ) can then be derived using importance sampling with q φ ( z | x ) and the ELBO objective follows using Jensen’s inequality:
[Equation]
The Evidence Lower BOund (ELBO) is now tractable since both p θ ( x | z ) and q φ ( z | x ) are parametrized and so can be optimized with respect to the encoder and decoder parameters.
Remark
In practice, p θ ( x | z ) is often taken as a simple multivariate Gaussian N ( µ θ , σ I d ) and the decoder only outputs µ θ ( z ) .
Hence, the ELBO can also be seen as a two terms objective function (Ghosh et al., 2020).
The ﬁrst one is a reconstruction term reducing to a L2 loss between the decoded sample and the input data point while the second one is a regularizer corresponding to the
KL divergence between the posterior and the prior.
[Equation]
R ELATED W ORK
A natural way to improve the generation from VAEs consists in trying to use more complex priors (Hoffman & Johnson, 2016) than the standard Gaussian distribution used in the initial version such that they better match the true distribution of the latent codes.
For instance, a Mixture of Gaussian as prior distribution over the latent variable was proposed (Nalisnick et al., 2016; Dilokthanakul et al., 2017).
This idea was further extended with the introduction of the Variational Mixture of Posterior (VAMP) prior (Tomczak & Welling, 2018).
In the same vein, hierarchical latent variable models (Sønderby et al., 2016; Klushyn et al., 2019) or prior learning (Chen et al., 2016;
Pang et al., 2020;
Aneja et al., 2020) have recently emerged and aimed at ﬁnding the best suited prior distribution for a given data set.
Acceptance/rejection sampling method was also propose to try to improve the expressiveness of the prior distribution (Bauer & Mnih, 2019).
On the ground that the latent space must adapt to the data as well, geometry-aware latent space modelings as hypershpere (Davidson et al., 2018), torus (Falorsi et al., 2018) or Poincar´e disk (Mathieu et al., 2019) or discrete latent representations (Razavi et al., 2020) were proposed.
Other recent contributions proposed to see the latent space as a Riemannian manifold where the Riemannian metric is given by the Jacobian of the generator function (Arvanitidis et al., 2018;
Chen et al., 2018; Shao et al., 2018) or learned directly from the data throughout training thanks to geometry-aware normalizing ﬂows (Chadebec et al., 2020).
While these geometry-based methods show interesting property of the learned latent space they either require the computation of a time consuming model-dependent function, the Jacobian, or add further parameters to the model to learn the metric.
Arguing that VAEs are essentially autoencoder regularized with a Gaussian noise, Ghosh et al. (2020) proposed another very interesting interpretation of the VAE framework and showed that other types of regularization may be of interest as well.
Since the generation process from these autoencoders is no longer relying on the prior distribution, the authors proposed to use ex-post density estimation by ﬁtting simple distributions such a Gaussian mixture in the latent space.
While this paves the way for consideration of other ways to generate data, it mainly reduces the VAE framework to an autoencoder while we believe that it can also unveil geometrical aspects of the data that are of interest.
P ROPOSED M ETHOD
In this section, we argue that a VAE shows naturally a Riemannian structure of the latent space through the learned covariance matrices appearing in the posterior distributions.
We then propose a new natural generation scheme guided by this estimated geometry and consisting in sampling from the uniform distribution deriving intrinsically from the learned Riemannian manifold.
4. A W ORD ON R IEMANNIAN G EOMETRY
First, we brieﬂy recall some basic elements of Riemannian geometry needed in the rest of the paper.
A more detailed discussion on integration and probability densities on manifolds may be found in
Appendix A. A d-dimensional manifold M is a manifold which is locally homeomorphic to a ddimensional Euclidean space.
If the manifold M is further connected and differentiable it possesses a tangent space T z at any z ∈ M composed of the tangent vectors of the curves passing by z .
If M is equipped with a smooth inner product g = (cid:104)·|·(cid:105) z deﬁned on its tangent space T z for any z ∈ M then
M it is called a Riemannian manifold and g is the associated Riemannian metric.
Since g is an inner product, a local representation of g at any z ∈ M is given by the positive deﬁnite matrix
G ( z ) .
The notion of length of curves γ : R → M traveling in M can be deﬁned as follows
[Equation]
Curves minimizing such a length are called geodesics and a Riemannian distance between two points z 1 , z 2 ∈ M can then be introduced and equals
[Equation]
The manifold M is said to be geodesically complete if all geodesic curves can be extended to R .
In an Euclidean space, G reduces to the I d and the distance becomes the classic Euclidean one.
Remark
A simple extension of this Euclidean framework consists in assuming that the metric is given by a constant positive deﬁnite matrix Σ different from I d .
In such a case the induced
Riemannian distance is the well-known Mahalanobis distance which writes
[Equation]
4. T HE R IEMANNIAN G AUSSIAN D ISTRIBUTION
The notion of measure and so of probability distribution can be extended to geodesically complete
Riemannian manifolds as well (Pennec, 2006).
Given the Riemannian manifold M endowed with the Riemannian metric G and a chart z , an inﬁnitesimal volume element may be deﬁned on each tangent space T z of the manifold M as follows
[Equation]
with dz being the Lebesgue measure.
Hence, a deﬁnition of Riemannian Gaussian distribution on
M can be derived and consists in considering the Riemannian distance deﬁned in Eq.
4 instead of the Euclidean one as follows
[Equation]
where d M z is the volume element deﬁned by Eq.
Hence, the multivariate normal distribution is only a speciﬁc case of the Riemannian distribution with σ = 1 , deﬁned on the manifold M = R d endowed with the constant Riemannian metric G ( z ) =
Σ − 1 , ∀ z ∈ M .
4. G EOMETRICAL I NTERPRETATION OF THE VAE F RAMEWORK
Within the VAE framework, the variational distribution q φ ( z | x ) is voluntarily chosen as a simple multivariate Gaussian distribution deﬁned on R d with d being the latent space dimension.
Hence, as explained in the previous section, given an input data point x i , the posterior q φ ( z | x ) =
N ( µ ( x i ) , Σ ( x i )) can also be seen as a Riemannian Gaussian distribution where the Riemannian distance is simply the distance with respect to the metric tensor Σ − 1 ( x i ) .
Hence, the VAE framework can be seen as follows.
As with an autoencoder, the VAE provides a code µ ( x i ) which is a lower dimensional representation of an input data point x i .
However, it also gives a tensor Σ − 1 ( x i ) depending on x i which can be seen as the value of a Riemannian metric G at µ ( x i ) i.e.
[Equation]
This metric is crucial since it impacts the notion of distance in the latent space now seen as the Riemannian manifold M =
( R d , G ) and so changes the directions that are favored in the sampling from the posterior distribution q φ ( z | x ) .
Then, a sample z is drawn from a standard ( i.e. σ = 1 in Eq. 6)
Riemannian Gaussian distribution and fed to the decoder.
As ﬁrst approximation and since we only have access to a ﬁnite number of metric tensors Σ ( x i ) , the VAE model assumes that the metric is locally constant close to µ ( x i ) and so the Riemannian distance reduces to the Mahalanobis distance in the posterior distribution.
This simpliﬁed drastically the training process since now Riemannian distances have closed form and so are easily computable.
Interestingly, the VAE framework will impose through the ELBO expression given in Eq.
3, that z gives a sample x ∼ p θ ( x | z ) close to x i when decoded.
Since z has a probability density function imposing higher probability for samples having the smallest Riemannian distance to µ , the VAE imposes in a way that latent variables that are close in the latent space with respect to the metric Σ − 1 will also provide samples that are close in the data space X in terms of L2 distance as noticed in Remark.
Noteworthy is that the latter distance can be amended through the choice of the decoder p θ ( x | z ) .
This is a very interesting property since it allows the VAE to directly link the learned Riemannian distance in the latent space to the distance between two input data points.
On the other hand, the regularization term in Eq.ensures that the covariance matrices do not collapse to 0 d and constraints the latent codes to remain close to the origin to ease optimization.
Finally, at the end of training, we have a lower dimensional representation of the training data given by the means of the posteriors µ ( x i ) and a family of metric tensors ( G i =
Σ − 1 ( x i ) )
corresponding to the value of a Riemannian metric deﬁned locally on the latent space.
Inspired from Hauberg et al. (2012), we propose to build a smooth continuous Riemannian metric deﬁned on the entire latent space by performing the following interpolation:
[Equation]
where
[Equation]
where dist Σ − 1 ( x i ) ( z , µ i ) is the Riemannian distance between z and µ i with respect to the locally constant metric G ( µ ( x i ))
=
Σ − 1 ( x i ) .
Since the sum in Eq.
7 is made on the total number of training samples N , the number of reference metric tensors can be decreased for huge data sets by selecting only k < N metric tensors 1 and increasing ρ to reduce memory usage.
Then, we have:
Proposition 1 The Riemannian manifold M =
( R d , G ) is geodesically complete.
Prop. 1 (proved in Appendix B) allows now to refer to probability densities on M .
Rigorously, the metric deﬁned in Eq.
7 should have been used during the training process.
Nonetheless, this would have made the training longer and trickier since it would involve i) the computation of Riemannian distances that have no longer close form and so make the resolution of the optimization problem in
Eq. 4 needed and ii) the sampling from Eq. 6 which is not trival.
4. G EOMETRY -A WARE S AMPLING
Now that the VAE has learned and capture the intrinsic geometry of the data within the latent space seen as a Riemannian manifold, we propose to exploit this strong property to enhance the generation procedure.
A natural way to sample from such a latent space would consists in sampling from the uniform distribution intrinsically deﬁned on the learned manifold.
Similar to the Gaussian distribution presented in Sec.
4.2, the notion of uniform distribution can indeed be extended to Riemannian manifolds.
Given a bounded set A ⊂ M , the uniform distribution writes (Pennec, 2006)
[Equation]
This density is taken with respect to d M z , the Riemannian measure but using Eq.
5 and a coordinate system z allows to obtain a pdf now deﬁned with respect to the Lebesgue measure:
[Equation]
Since, the Riemannian metric has a closed form expression given by Eq. 7, sampling from such a distribution is now quite easy and may be performed using classic MCMC based sampling methods.
In this paper, we use the HMC sampler (Neal, 2005).
Now we are able to sample from the intrinsic uniform distribution which is a natural way of exploring the estimated manifold and the sampling is now guided by the geometry of the latent space.
4. I LLUSTRATION ON A T OY D ATA S ET
The usefulness of such sampling procedure may be easily appended in Figure 1 where a vanilla VAE was trained with a toy data set composed of binary images of circles and rings of different size and thickness (example inspired by Chadebec et al.
On the left is presented the learned latent space along with the means of the posteriors given by the colored dots.
The log of the metric volume element is given in gray scale.
In this example, we clearly see a geometrical structure appearing since the circles and rings seem to wrap around each other.
Obviously, sampling from the prior (taken as a
N (0 , I d ) ) to generate in such a case is far from being optimal since the sampling will be performed regardless of the underlying distribution of the latent variables and so will create irrelevant samples.
To further illustrate this, we propose to interpolate between points in the latent space using different cost functions in the middle and right plots.
Dashed lines represent afﬁne interpolations while the solid ones show interpolation aiming at minimizing the potential V ( z )
= ( (cid:112) det G ( z )) − 1 all along the curve i.e. solving the minimization problem
[Equation] [Equation]
Below are presented the decoded samples all along the interpolation curves.
Thanks to those interpolations we can see that i) the latent space seems to really have a speciﬁc geometrical structure since decoding all along the interpolation curves obtained by solving Eq.
8 leads to visually very
−−−   3 −−−
CirclesRings −    (a) −−−   3 −−−
OursAfﬁneCirclesRings      (b) −−−   3 −−−
OursAfﬁneCirclesRings
[Figure]
satisfying results, ii) certain locations of the latent space must be avoided since sampling there will produce irrelevant samples (see red frames and corresponding red dashes).
Using the proposed sampling scheme will allow to sample in the white areas and so ensure that the sampling remains close to the data i.e. where information is available and so does not produce irrelevant images when decoded.
E XPERIMENTS
In this section, we conduct a comparison with other VAE models using either other regularization schemes, more complex priors, ex-post density estimation or trying to take into account geometrical aspects including our method.
In the following and to ensure a fair comparison, all the models share the same auto-encoding neural network architectures which is described in Appendix D.
5. G ENERATION WITH B ENCHMARK D ATA S ETS
First, we compare the proposed sampling method to several variants of the VAE such as the Wasserstein Autoencoder (WAE) (Tolstikhin et al., 2018), Regularized Autoencoders (Ghosh et al., 2020) with either L2 decoder’s parameters regularization (RAE-L2), gradient penalty (RAE-GP), spectral normalization (RAE-SN) or simple L2 latent code regularization (RAE), a vamp-prior VAE (VAMP) (Tomczak & Welling, 2018), a geometry-aware VAE (RHVAE) (Chadebec et al., 2020) and a simple
Autoencoder (AE).
We elect these models since they use different ways to generate the data using either the prior distribution or ex-post density estimation.
For the latter, we use the same approach as in Ghosh et al.
and ﬁt a 10-component mixture of Gaussian in the latent space after training.
The models are trained on 4 well known databases: MNIST (LeCun, 1998), SVHN (Netzer et al., 2011), CIFAR 10 (Krizhevsky et al., 2009) and CELEBA (Liu et al., 2015) and we keep the model achieving the best validation loss.
See Appendix D for the comprehensive experimental setup.
Figure 2 shows a qualitative comparison between the resulting generated samples for MNIST and
CELEBA, the same plots are made available in Appendix C for SVHN and CIFAR 10.
Interestingly, using the non-prior based methods seems to produce visually speaking better samples (rows 6 to end).
Nonetheless, the resulting samples seem even sharper when the sampling takes into account geometrical aspect of the latent space as we propose (last row).
Additionally, even though the exact same model is used, we clearly see that using the proposed method represents a strong improvement
[Figure]
[Figure]
of the generation process from a vanilla VAE
when compared to the samples coming from a normal prior (second row)
This insists on the fact that even the simplest VAE model actually contains a lot of information in its latent space but the limited expressiveness of the prior impedes to access to it.
Hence, using more complex prior such as the vamp may be a tempting idea.
However, one must keep in mind that the ELBO objective in Eq. 2 must remain tractable and so using more expressive priors may be impossible.
These observations are even more supported by Table 1 where we report the Frechet Inception Distance (FID) (Heusel et al., 2017) and the precision and recall score (PRD) (Sajjadi et al., 2019) metrics against the test set to assess the sampling quality and diversity.
Again, ﬁtting a mixture of Gaussian (GMM) in the latent space appears to be a very interesting idea since it allows for a better expressiveness and latent space prospecting.
For instance, on MNIST the FID falls from 40.7 with the prior to 13.1 when using a 10-component GMM.
Nonetheless, with the proposed method we are able to make it even smaller (8.5) and PRD scores higher without changing the model and performing post processing.
This can also be observed on the 3 other data sets.
Impressively, in almost all cases, the proposed generation method can either compete or outperform peers both in
[Table]
RAE (L2) 9.
RAE (SN) 11.
RAE 11.
VAE - GMM 13.
VAE - Ours 8.
MNIST
[Figure]
CIFAR
[Figure]
terms of FID and PRD scores.
Finally, we check if the proposed method does not overﬁt the training data by showing the nearest neighbor in the train set to a generated image in Figure 3.
The outcome of such an experiment is that using post training latent space processing such as ex-post density estimation or adding some geometrical consideration to the model allows to strongly improve the sampling without adding more complexity.
5. I NVESTIGATING G ENERATION R OBUSTNESS IN L OW D ATA R EGIME
We perform a comparison using the same models and data sets as before but we decide to progressively decrease the size of the training set to see the robustness of the different sampling methods according to the number of samples.
This experiment is rarely performed in most generative models related papers even though it is well known that such a context may be very challenging for these models.
Nonetheless, it appears to us very important since in day-to-day applications collecting such large databases may reveal costly if not impossible (think of medicine for instance).
Hence, we consider MNIST, CIFAR10 and SVHN and use either the full data set size, 10k, 5k or 1k training samples.
For each experiment, the best retained model is again the one achieving the best ELBO on the validation set the size of which is set as 20% of the train size.
See Appendix D for further details about experiments setup.
Then, each trained model is used to generate 10k new samples and the
FID on the test set is reported on the curves presented in Figure 4.
Results obtained on SVHN are presented in Appendix E. Again, the proposed sampling method appears quite robust to the data set size since it outperforms the other models’ FID even when the number of training samples is smaller.
The second output of such a study is that as exposed in the paper, using more complex generation procedures such as ex-post density estimation seems to still enhance the generation capability of the
[Table]
model.
Nonetheless, for CIFAR, even though we see a huge gain using ex-post density estimation when the number of training samples is quite high ( ≥ 5k) [Table] this gain seems to worsen when the data set size is reduced.
5. G ENERATION WITH C OMPLEX D ATA
Finally
[Table]
we also propose to stress the proposed generation procedure in a day-to-day scenario where the limited data regime is more than common.
One of the main case study is medical imaging that may both combine high complexity data with low sample sizes.
To stress the model in such condition, we consider the publicly available OASIS database composed of 416 MRI of patients, 100 of whom were diagnosed with Alzheimer disease (AD).
We ﬁrst train the same models on these images and show the resulting sampling quality using each generation method in Appendix F. Since both FID and PRD scores are not relevant and reliable in such low data regime due to the lack of a large test set, we propose to assess quantitatively the generation quality with a data augmentation like task.
Hence, we split the initial data set into a train set (70%), a validation set (10%) and a test set (20%).
Each generative model is trained on each label of the train set and used to generate 2k new samples per class.
Then a simple CNN classiﬁer is trained for 50 epochs on i) the original train set and ii) the 4k generated samples from the generative models and tested on the test set.
For each run, the classiﬁer that is kept is the one achieving the best balanced accuracy (BA) on the validation set.
The same experiment is repeated on 20 independent runs for each case.
Then, we report in
Table 2, the mean BA across the 20 runs and F1 score for each label.
According to us, these metrics provide a good way to assess i) if the generative model can generate data that are not too far from the test set and add relevant information to the data useful for classiﬁcation and ii) allows to quantify the amount of overﬁtting.
The proposed method is the only one to be able to outperform the original data both in terms of balanced accuracy and F1 score for both labels meaning that generated samples are relevant to the classiﬁer.
This is also the sign of a good generalization power since the classiﬁer achieves classiﬁcation results that strongly outperform the one observed on the original data.
C ONCLUSION
In this paper, we provided a geometric understanding of the latent space learned by a VAE and showed that it can actually be seen as a Riemannian manifold.
Then, we proposed a new natural generation process consisting in sampling from the intrinsic uniform distribution deﬁned on this learned manifold.
It showed to be competitive with more advanced versions of the VAEs using either more complex priors, ex-post density estimation, geometry-aware normalizing ﬂows or other regularization schemes.
Interestingly, the proposed method revealed good robustness properties in complex settings such as high dimensional data or low sample sizes.
Future work would consist in trying to use this method to perform data augmentation in those challenging contexts and compare its reliability for such a task with state of the art augmentation methods.
A F URTHER E LEMENTS ON R IEMANNIAN G EOMETRY
In the ﬁeld of differential geometry, a Riemannian manifold M can be deﬁned as a connected and differentiable manifold endowed with a Riemannian metric g .
The metric g is a smooth inner product g : p → (cid:104)·|·(cid:105) p deﬁned on each tangent space T p M of the manifold with p ∈ M .
A chart (or coordinate system) ( U, x ) is a homeomorphism mapping an open set U of the manifold to an open set V of an Euclidean space.
The manifold M is further called a d − dimensional manifold if for each chart V ⊂ R d .
This means that there exists a neighborhood U of each point
p ∈ M such that U is homeomorphic to R d .
Given p ∈ U , a chart φ : ( x 1 , . . .
, x d ) induces a basis (cid:16) ∂∂x 1 , . .
, ∂∂x d (cid:17) p on the tangent space T p M .
Hence, the metric of a Riemannian manifold can be locally represented in the chart φ as a positive deﬁnite matrix G ( p ) =
( g i,j ) p, 0 ≤ i,j ≤ d = ( (cid:104)
∂∂x i | ∂∂x j (cid:105) p ) 0 ≤ i,j ≤ d for each point p of the manifold.
That is for v, w ∈ T p M and p ∈ M , the inner product writes (cid:104) u | w (cid:105) p =
u (cid:62) G ( p ) w .
Two ways of apprehending manifolds exist.
The ﬁrst one is the extrinsic view and it assumes that the manifold is embedded within a higher dimensional Euclidean space.
A simple example is the 2-dimensional sphere S 2 seen as a subspace of R 3 .
The second one which is adopted in this paper is the intrinsic view.
In the intrinsic view the manifold is studied using its underlying structure and so the length of a curve γ : R → M traveling in the manifold cannot be interpreted using the
Euclidean distance but requires to use the metric deﬁned onto the manifold itself.
Let z 1 , z 2 ∈ M be two points of the manifold, and γ be a curve traveling in M parametrized by t ∈ [0 , 1] such that γ (0) =
z 1 and γ (1) =
z 2 .
Then, the length of γ is given by
[Equation]
Curves γ that are minimizer(s) of this criteria are called geodesic curves.
A distance dist on the manifold M can then be derived and writes:
[Equation]
The manifold M is said to be geodesically complete if all geodesic curves can be extended to R .
For any p ∈ M , the exponential map at p maps a vector v of the tangent space T p M to a point of the manifold ˜ p ∈ M such that the geodesic starting at p with initial velocity v reached ˜ p at time 1.
[Equation]
where γ ( p,v ) (1) means γ (0)
=
p and ˙ γ (0) = v .
Saying that the manifold M is geodesically complete means that the exponential maps is deﬁned on the entire tangent space T p M for each element p ∈ M .
In Pennec (2006), the author discusses the statistical framework that can be developed on geodesically complete manifolds using an intrinsic point of view.
In particular, given a positively oriented
Riemannian manifold M and a chart φ = ( x 1 , · · · , x d ) , a volume form dVol g can be deﬁned as the d -form:
[Equation]
This represents an inﬁnitesimal volume element on each tangent space and so a measure on the manifold M
[Equation]
It follows that we are able to integrate functions f : M ⊃ U → R on a given chart ( U, φ ) .
[Equation]
This notion can then be extended to the whole manifold M using partition of unity.
In particular, such a property allows us to deﬁne probability distributions whose density is deﬁned with respect to the measure on the manifold.
We recall such deﬁnition from Pennec (2006) below
Deﬁnition 1 Let B ( M ) be the Borel σ -algebra of M .
The random point p has a probability density function ρ p if:
[Equation]
Finally, given a chart φ = ( z 1 , · · · , z n ) deﬁned on the whole manifold M and a random point p on
M , the point z = φ ( p ) is a random point whose density ρ (cid:48)
z may be written with respect to the
Lebesgue measure as such (Pennec, 2006):
[Equation]
B P ROOF OF P ROP .
We adapt the proof in Louis (2019) and Chadebec et al.
(2021) to our speciﬁc metric.
We will show that given the manifold M = R d and the Riemannian metric whose local representation is given by
Eq. 7, any geodesic curve γ :] a, b
[ → M is actually extensible to R .
Let us consider a geodesic curve γ such that γ cannot be extended to R .
There exists I =] a, b [ such that I is the maximum deﬁnition domain of γ .
We will show that with such an assumption we will end up with a contradiction.
We recall the shape of the Riemannian metric:
[Equation]
Since Σ i are positive deﬁnite matrices we have z (cid:62) Σ i z > 0 , ∀ z ∈ M − { 0 } .
We further have ω
i ( z ) > 0 , ∀ z ∈ M since the manifold is geodesically complete .
Let t 0 ∈ ] a, b [ we therefore have for any t
[Equation] [Equation]
where the last equality comes for the constant speed of geodesic curves (Carmo, 1992).
Hence we have:
[Equation]
This means that for any t ∈ ] a, b [ the geodesic curve γ remains within a compact set.
We show now that the curve can actually be extended.
Let us deﬁne the sequence t n −−−−→ n →∞ b .
Since the geodesic curves have a constant speed the set I = { ( t n , ˙ γ ( t n ) } n ∈ N is compact.
Moreover, using CauchyLipchitz theorem, we can ﬁnd ε > 0 such that for any n ∈ N , the geodesic γ can be extended to ] t n − ε, t n + ε [ .
Since, t n can be as close as b as desired we can assure that the curve deﬁnition domain can be extended to ] a, b + ε 2 [ .
C O THER G ENERATION
In this appendix, we gathered the resulting samplings from the different considered models for
SVHN, CIFAR 10 and OASIS data sets.
[Figure]
CIFAR
[Figure]
D E XPERIMENTAL S ET U P
The RAEs, VAEs and AEs are trained for 100 epochs for SVHN, MNIST 2 and Celeba and 200 on
CIFAR10.
Each time we use the ofﬁcial train and test split of the data.
For MNIST and SVHN, 10k samples out of the train set are reserved for validation and 40k for CIFAR10.
As to Celeba, we use the ofﬁcial validation set for validation.
The model that is kept at the end of training is the one achieving the best validation loss.
All the models are trained with a batch size of 100 and starting learning rate of 1 e − 3 (but CIFAR where the learning rate is set to 5 e − 4 ) with an Adam optimizer (Kingma & Ba, 2014).
We also use a scheduler decreasing the learning rate by half if the validation loss stops increasing for 5 epochs.
For the experiments on the sensitivity to the training set size, we keep the same set up.
For each data set we ensure that the validation set is 1 / 5 th the size of the train set but for CIFAR where we select the best model on the train set.
The neural networks architectures can be found in Table 3 and are inspired by Ghosh et al.
The metrics (FID and PRD scores) are computed with 10000 samples against the test set (for Celeba we selected only the 10000 ﬁrst samples of the ofﬁcial test set).
The factor ρ is set to ρ = max i min j (cid:54) =
i (cid:107) µ ( x i ) − µ ( x j ) (cid:107) 2 .
For models coming from peers, we use the parameters provided by the authors when available.
For the data augmentation task, the generative models are trained on each class for 1000 epochs with a batch size of 100 and a starting learning rate of 1 e − 4 .
Again a scheduler is used and the learning rate is cut by half if the loss does not improve for 20 epochs.
All the models have the autoencoding architecture described in Table 3.
As to the classiﬁer, it is trained with a batch size of 200 forepochs with a starting learning rate of 1 e − 4 and Adam optimizer.
A scheduler reducing the learning rate by half every 5 epoch if the validation loss does not improve is again used.
The best kept model is the on achieving the best balanced accuracy on the validation set.
Its neural network architecture may be found in Table 4.
MRIs are only pre-processed such that the maximum value of a voxel isand the minimum 0 for each data point.
[Table] [Table]
[Table]
[Table]
E D ATA S ET S IZE S ENSIBILITY ON SVHN
In Figure
[Figure]
we show the same plot for SVHN as in Sec. 5.2.
Again the proposed method appears to be part of the most robust generation procedures to data set changes.
[Figure]
F O THER C LASSIFICATION M ETRICS
In Table 5 are presented some further classiﬁcation results for each considered model while generated samples using each generation procedure are made available in Figure 8.
On OASIS database the proposed method appears to produce visually the sharpest samples.
This better generation performance is also supported by the classiﬁcation metrics provided in Table 2 and Table
[Table] [Table] [Figure]