Identifying latent distances with Finslerian geometry
Editors: Sophia Sanborn, Christian Shewmake, Simone Azeglio, Arianna Di Bernardo, Nina Miolane
Abstract
Riemannian geometry has been shown useful to explore the latent space of generative models.
Effectively, we can endow the latent space with the pullback metric obtained from the data space. Because most generative models are stochastic, this metric will be de facto stochastic, and, as a consequence, a deterministic approximation of the metric is required.
Here, we are defining a new metric as the expectation of the stochastic curve lengths induced by the pullback metric.
We show this metric is, in fact, a Finsler metric.
We compare it with a previously studied expected Riemannian metric, and we show that in high dimensions, the metrics converge to each other.
1. Introduction
In machine learning, one objective is to reduce the dimensions of the data using generative models.
When this data is assumed to lie near a manifold, we can compute lengths and distances in a low-dimensional latent space using the Riemannian geometry machinery.
Such a metric is obtained as the pullback of the immersion f : Z → X that maps the latent space Z ⊂ R q to the data space X ⊂ R D . In practice, this smooth map f can be the decoding part of a Variational Autoencoder (VAE) (Kingma and Ba, 2014) or a Gaussian Process Latent Variable Model (GP-LVM) (Lawrence, 2003).
One problem is that generative models are stochastic by default, and Riemannian geometry only involves deterministic objects.
To dodge this issue, a solution previously studied by
Tosi et al.
(2014) in the case of GP-LVMs and Arvanitidis et al.
(2018) for VAEs was to take the expectation of the Riemannian metric tensor.
It has been extensively studied by Hauberg (2018a) as well.
In this work , we focus on the quantity that matters to compare data points in the latent space: geodesics.
Instead of approximating the metric tensor, we directly approximate the stochastic length by its expectation.
This extends the work of Eklund and Hauberg (2019).
The findings are the following:
We prove that the expected curve length defines a norm and this norm is a Finsler metric.
When increasing the number of dimensions, the functionals (curve energy, curve length, volume measure) of the Riemannian and Finsler metrics converge to each other at rate O (cid:0) 1 D (cid:1) .
Combined, these findings provide groundings for the previously studied, but heuristically defined, expected Riemannian metric.
[Figure]
2. Finsler metric as the expected length on a random manifold
The metric pulled-back by a stochastic mapping is stochastic de facto.
To compute functionals in the latent space, this metric is approximated by its expectation:
Definition 1 (Tosi et al., 2014; Arvanitidis et al., 2018)
Let G be a stochastic Riemannian metric tensor.
We refer to E
[ G ] as the expected metric tensor , and g x : ( T x M , T x M ) →
R + : ( u, v ) → u ⊤ E [ G x ] v as the expected Riemannian metric , computed at a specific point x ∈ M .
This metric defines a Riemannian metric on R q .
Instead of considering the expected Riemannian metric, we aim to define a norm as the expectation of the random paths.
The norm of a vector v in T x M , induced by a random metric tensor G , is ∥ v ∥ G = √ v ⊤ Gv .
We define our new norm as: v → E (cid:104) √ v ⊤ Gv (cid:105) .
Proposition
The function: F x : T x M → R : v → E (cid:104)(cid:112) v ⊤ G x v (cid:105) defines a Finsler metric but is not induced by a Riemannian metric.
To compare the expected Riemannian metric with our Finsler metric, we will place ourselves in the specific case where f is a Gaussian process.
Gaussian Processes have the advantage of approximately describing the behaviour of deep bayesian neural networks (Matthews et al., 2018; Jacot et al., 2018).
The Finsler function can be rewritten in a closed form expression:
Proposition 3 Let f : R q → R D be a Gaussian process , then:
[Equation]
is the close-form expression of the previously defined Finsler metric at a point x ∈ M for a vector v ∈ T x M , with ω = ( v ⊤ Σ v ) − 1 ( v ⊤ E
[ J ] ⊤ E
[ J ] v ) , J the Jacobian of f : J ∼ (cid:81) Di =1 N ( µ i , Σ) and 1 F 1 the confluent hypergeometric function of the first kind.
3. Comparison of the Finsler and Riemannian metrics
In geometry, we need to define a metric to compute functionals.
In Riemannian geometry, the metric is conveniently obtained by constructing an inner product.
The inner product greatly simplifies subsequent computations but it is also restrictive.
Finsler geometry is an extension of Riemannian geometry by relaxing the assumption that the metric should be an inner product.
Instead the metric is defined as a norm, and so it is not necessarily bilinear or reversible.
Ipso facto, Finsler geometry is a generalisation of the Riemannian geometry when we disregard the bilinearity assumption.
This difference can be observed by studying indicatrices ( { v ∈ T x M , ∥ v ∥ = 1 } ), which is a way to see how a metric distorts the space.
In the indicatrix-field represented in Figure 2: we see that the Finsler indicatrices can have almost a rectangular shape while the Riemannian ones are ellipses.
In the area of low variance (near the data points), the Finsler and Riemannian indicatrices are alike.
This is explained because the metrics differ by a variance term.
If our mapping f was deterministic, both metrics would agree.
In general, we will see that in high dimensions, they converge to each other:
Proposition 4 The relative difference between the Finsler metric F [Equation] ⊤ G x v ] and the Riemannian metric g x : ( v, v ) → v ⊤ E [ G x ] v is:
[Equation]
This proposition is valid for any stochastic immersion.
In this proposition, we are comparing the norms induced by their respective metrics.
The
Finsler metric is a norm by definition, while the Riemannian metric is defined as an inner product, which defines a norm by taking its squareroot.
These metrics become equal in two cases: when the variance converges to zero, and when the number of dimensions increases.
The latter case is investigated below for a Gaussian process. [Figure]
Proposition {+
+} When f : R q → R D is a Gaussian process, the relative ratio becomes:
[Equation]
with ω = ( v ⊤ Σ v ) − 1 ( v ⊤ E
[ J ] ⊤ E [ J ] v ) and J the Jacobian of f : J ∼ (cid:81) Di =1 N ( µ i , Σ) . Under the assumption that ω is lower bounded and we are working on a compact set, we can find upper bounds for all the functionals derived from the Finsler and Riemannian metrics, in particular: the curve lengths ( L and L G ), the curve energy( E and E G ) and the volume measure ( V and V G ):
Proposition 6 In high number of dimensions, we have:
L G ( x ) [Equation]
L ( x
[Equation]
and, when D converges toward infinity: L G ∼ L , E G ∼ E and V G ∼ V .
4. Discussion
Generative models are often used to reduce data dimension in order to better understand the mechanisms behind the data generating process.
We consider the general setting where the mapping from latent variables to observations is driven by a smooth stochastic process, and the sample mappings span Riemannian manifolds.
The Riemannian geometry machinery has already been used in the past to explore the latent space.
The Finslerian view of the latent representation gives us a suitable general solution to explore a random manifold, but it does not immediately translate into a practical computational tool.
As Riemannian manifolds are better understood computationally than Finsler manifolds, we have raised the question: How good an approximation of the Finsler metric can be achieved by a Riemannian metric?
The answer turns out to be: quite good.
We have shown that as data dimension increases, the Finsler metric becomes increasingly Riemannian.
We have a justification for approximating the Finsler metric with a Riemannian metric such that computational tools become more easily available.
In practice we find that geodesics under the Finsler and the Riemannian metric are near identical except in regions of high uncertainty. Acknowledgments.
This work was funded in part by the Novo Nordisk Foundation through the Center for Basic Machine Learning Research in Life Science (NNF20OC0062606).
It also received funding from the European Research Council (ERC) under the European
Union’s Horizon 2020 research, innovation programme (757360). SH was supported in part by research grants (15334, 42062) from VILLUM FONDEN. The authors also acknowledge the support of the Pioneer Centre for AI, DNRF grant number P1. Appendix A. Notations and definitions
We define f : R q → R D a stochastic immersion (when specified, a Gaussian process), J its Jacobian, and G x = J ⊤ J the stochastic Riemannian metric tensor at a point x of the manifold.
Key functionals are also investigated: the curve length, the curve energy and the volume measure, denoted L G , E G and V G for the expected Riemannian metric and L , E and V for the Finsler metric.
The volume measure allows for integrating probabilities over the manifold, while the energy functionals (equivalently, the length functionals) are needed to compute geodesics.
F
Finsler metric: F [Equation] M , T [Equation]
M
[Equation]
R
M
Smooth differentiable manifold
T x M
Tangent space of the manifold M at a point x
D
Number of dimensions of the observational space q
Number dimensions of the latent space
J f
Jacobian of a stochastic function f : R q → R D Σ x
Covariance matrix of the Jacobian J x ∼ (cid:81) Di =1 N ( µ i , Σ x )
G x
Stochastic metric tensor defined as; G x = J ⊤ x J x
L , E , V
Length, energy and Busemann Hausdorff volume of the Finsler metric
L G , E G , V G
Length, energy and volume of the Riemannian metric with E
[ G ] the expected metric tensor.
immersion
An immersion f : M → R D is a smooth map such that its derivative ∂f : T x M → T f ( x ) R D is injective at each point x ∈ M .
indicatrix
An indicatrix is a befitted way to represent the distortion induced by the metric on a unit circle.
Rigorously speaking, for a given point on the manifold, an indicatrix of a metric is defined as the set of vectors in the tangent space such that norm obtained from the metric is equal to one: { v ∈ T x M ,
|| v || = 1 } .
If our metric is euclidean, we will only have a linear transformation between the latent and the observational spaces, and the indicatrix would still be a circle.
Because the Riemannian metric is quadratic, it will always generate an ellipse in the latent space.
The Finsler indicatrix, however, would have a convex, even asymetrical, shape.
curve length
We consider a curve γ ( t ) and its derivative ˙ γ ( t ) on a manifold M equipped with a norm ∥·∥ m endowed by a metric m – a Finsler metric or a Riemannian metric.
We define the length of the curve:
L ( γ ) =
(cid:82) ∥ ˙ γ ( t ) ∥ m dt .
geodesic
Locally length-minimising curves between two connecting points.
energy curve
We define the energy of the curve: E ( γ ) =
(cid:82) ∥ ˙ γ ( t ) ∥
2 m dt .
volume measure
The volume measure, or volume form, is a quantity that enables us to integrate over a manifold and accounts for the amount of distortion induced by the metric.
In Riemannian geometry, the volume measure is uniquely defined: V G = √ det G .
In Finsler geometry, the volume has different definition.
In this work, we are using the BusemannHausdorff definition: V = vol( B n (1)) /
vol( { v ∈ T x M| F x ( v ) < 1 } ).
Appendix B. Proofs
Lemma
The Hessian matrix 12 Hess ( F ( x, v ) 2 ) of the function F ( x, v ) =
E (cid:104)(cid:112) v ⊤ G x v (cid:105) is equal to:
[Equation]
Proof Let g : R q → R : v → √ v ⊤
Gv be a scalar-value stochastic function that takes as input a column vector v , and G a random symmetric matrix.
We would like to know the different derivatives of g with respect to v .
We name by default J g and H g , its Jacobian and
Hessian matrix.
Using the chain rule, we have: J g = ( v ⊤ Gv ) − 12 Gv and H g = ( v ⊤ Gv ) − 12 G − ( v ⊤ Gv ) − 32 ( Gvv ⊤ G ).
For the rest of the proof, we need to show that we can invert the derivatives with the expectation: ∂ E [ g ] ∂v i =
E [ ∂g∂v i ].
The sequence of functions ( ∂g∂v i ) being always upper bounded by Gv i and E
[ Gv i ] < ∞ , using the theorem of dominated convergence, we can invert the expectation with the derivatives.
Finding the Jacobian of the expectation of the function g is the same than finding the expectation of the Jacobian J g , and similarly for the Hessian
H g .
We can then write, for F : R q → R : E [ v → √ v ⊤ Gv ], then
[Equation]
We now consider the function h :
R q
→ R : v → E [ √ v ⊤ Gv ] 2 = F ( v ) 2 .
Using the chain rule again, and inverting expectation and derivatives, we have its Hessian
[Equation]
Finally, replacing J g and H g previously obtained in this expression, we conclude:
[Equation]
Proposition
The function: F x : T x M → R : v → E (cid:104)(cid:112) v ⊤ G x v (cid:105) defines a Finsler metric but is not induced by a Riemannian metric.
Proof First, we need to prove the function F x ( v ) =
E (cid:104)(cid:112) v ⊤ G x v (cid:105) is:
positive homogeneous : ∀ λ ∈ R + , F ( p, λv )
=
λF x ( v ) 2.
smooth : F x ( v ) is a C ∞ function on the slit tangent bundle T M \ { 0 } 1) Let λ ∈ R , then we have: F ( x, λv ) =
E (cid:104)(cid:112) λ 2 v ⊤ G x v (cid:105) =
| λ | (cid:104)(cid:112) v ⊤ G x v (cid:105) .
The multivariate function: R q \{ 0 } → R ∗ + : v → v ⊤ G x v is C ∞ and strictly positive,
since G x = J ⊤ f J f is positive definite.
The function R ∗ + → R ∗ + : x → √ x is also C ∞ .
Finally,
R ∗ + → R ∗ + : x → E [ x ] is by definition differentiable.
By composition, F x ( v ) is a C ∞ function on the slit tangent bundle T M \ { 0 } .
Second, we need to prove the function F x ( v ) =
E (cid:104)(cid:112) v ⊤ G x v (cid:105) satisfies strong convexity criterion :
Proving that F satisfies the strong convexity criterion is equivalent to show that
the
Hessian matrix H =
12 Hess( F x ( v ) 2 ) is strictly positive definite.
Thus, we need to prove that ∀ w ∈ R q \{ 0 } , w ⊤ Hw > 0.
According to Lemma 7, because the expectation is a positive function, it’s straightforward to see that ∀ w ∈ R q \{ 0 } , w ⊤ Hw ≥ 0.
The tricky part of this proof is to show that w ⊤ Hw > 0.
This can be obtained if one of the terms ( F · Hess( F ) or ∇ F ⊤ ∇ F ) is strictly positive.
First, let’s decompose H as the sum of matrices: H = F Hess( F ) + ∇ F ⊤ ∇ F (Lemma 7), with:
[Equation]
We will study two cases: when w ∈ span( v ), and when w / ∈ span( v ).
We will always assume that v ̸ = 0, and so by definition: F x ( v ) > 0.
Let w ∈ span( v ).
We will show that w ⊤ ∇ F ⊤ ∇ F w > 0.
We have w = αv, α ∈ R .
Because F is 1-homogeneous and using Euler theorem, we have: ∇ F x ( v ) v = F x ( v ).
Then ( αv ) ⊤ ∇ F ⊤ ∇ F ( αv ) =
α 2 F 2 , and α 2 F x ( v ) 2 > 0.
Let w / ∈ span( v ).
F being a scalar function, we have:
w ⊤ F Hess[ F ] w = F w ⊤ Hess[ F ] w .
We would like to show that: w ⊤ Hess[ F ] w > 0.
The strategy is the following: if we prove that the kernel of Hess[ F ] is equal to the span( v ), then w / ∈ span( v ) is equivalent to say that w / ∈ ker(Hess
[ F ]) and we can conclude that: w ⊤ Hess[ F ] w > 0.
Let’s prove span( v ) ∈ ker(Hess( F )).
We know that Hess( F ) v = 0, since F is 1-homogeneous, so we have span( v ) ∈ ker(Hess( F )).
To obtain the equality, we just need to prove that the dimension of the kernel is equal to 1.
Let z ∈ span( v ⊤ G ) ⊤ , which is ( Gv ) T z = 0.
We have dim(span( v ⊤ M ))
= 1, and thus: dim(span( v ⊤ G ) ⊤ ) =
q − 1.
Furthermore, z ⊤ Hess[ F ] z =
z ⊤ E (cid:104) M ( v ⊤ Mv ) − 12 (cid:105) z > 0, so we can deduce that dim(im(Hess[ F ]))
=
q − 1.
Using the Rank-Nullity theorem, we conclude that dim(ker(Hess( F )))
=
q − dim(im(Hess[ F ]))
= 1, which concludes the proof.
In conclusion, ∀ w ∈ R q \{ 0 } , w ⊤ 12 Hess( F x ( v ) 2 ) w > 0.
The function F satisfies the strong convexity criterion.
Proposition 9 Let f be a Gaussian process , then:
[Equation]
with ω = ( v ⊤ Σ x v ) − 1 ( v ⊤ E
[ J ] ⊤ E
[ J ] v ) and 1 F 1 the confluent hypergeometric function of the first kind.
Proof The objective of the proof is to show that, if the Jacobian J f follows a non-central normal distribution, then, ∀ v ∈ R q , the expectation E [ v ⊤ J ⊤ f J f v ] will follow a non-central
Nakagami distribution.
This is a particular case of the derivation of moments of non-central
Wishart distributions, previously shown and studied by Kent and Muirhead (1984); Hauberg (2018b).
By hypothesis, J f follows a non-central normal distribution: J f ∼ N ( E [ J ] , I D ⊗ Σ x ).
Then, G x = J ⊤ f J f follows a non-central Wishart distribution: G x ∼ W d ( D, Σ x , Σ − 1 x E
[ J ] ⊤ E
[ J ]).
According to (Kent and Muirhead, 1984, Theorem 10.3.5.), v ⊤ G x v will also follow a noncentral Wishart distribution: v ⊤ G x v ∼ W 1 ( D, v ⊤ Σ x v, ω ), with: ω = ( v ⊤ Σ x v ) − 1 ( v ⊤ E
[ J ] ⊤ E
[ J ] v ).
To compute E [ (cid:112) v ⊤ G x v ], we shall look at the derivation of moments.
(Kent and
Muirhead, 1984, Theorem 10.3.7.) states that: if X ∼ W q ( D, Σ , Ω ′ ), with q ≤ D , then
E [(det( X ))
[Equation] ] = [Equation]
− 12 Ω ′ ).
We directly apply the theorem to our case, knowing that v ⊤ G x v is a scalar term, so det( v ⊤ G x v ) = v ⊤ G x v , q = 1, and k = 1 2 :
[Equation]
Proposition 10 The relative difference between the Finsler metric: F x : v → E [ (cid:112) v ⊤ G x v ] and the Riemmanian metric g : ( v, v ) → v ⊤ E [ G x ] v is:
[Equation]
This proposition is valid for any stochastic immersion.
Proof We will directly use a sharpen version of Jensen’s inequality obtained by Liao and
Berg (2019):
Let X be a one-dimensional random variable with mean µ and P ( X ∈ ( a, b ))
= 1, where −∞ ≤ a ≤ b ≤ + ∞ .
Let ϕ a twice derivable function on ( a, b ).
We further define: h ( x, µ ) =
ϕ
( x ) − ϕ ( µ ) ( x − µ )− ϕ ′ ( µ ) x − µ .
Then:
[Equation]
In our case, we will chose ϕ :
z → √ z with z a one-dimensional random variable defined as z = v ⊤ G x v .
a = 0, b = + ∞ and µ = E
[ z ].
h ( z, µ ) =
( √ z −√ µ )( z − µ ) − 2 − (2( z − µ ) √ µ ) − 1 .
Because its first derivative ϕ ′ is convex, the function x → h ( x, µ ) is monotonically increasing.
Thus:
[Equation]
and
[Equation]
It finally gives:
[Equation]
Replacing F ( x, v ) =
E [ √ z ] and (cid:112) g ( v ) =
(cid:112) E [ z ] = √ µ concludes the proof.
Proposition 11 When f is a Gaussian process, the relative ratio becomes:
[Equation]
+ ω ) 2 . Proof (Kent and Muirhead, 1984, Theorem 10.3.7.) states that if z ∼ W 1 ( D, σ, ω ) then E [ z k ] =
σ k 2 k Γ( D 2 + k ) Γ( D 2 ) 1 F 1 ( − k, D 2 , − 12 Ω).
In particular, for k = 1 and k = 2, we have 1 F 1 ( − 1 , b, c ) = 1 − cb and 1 F 1 ( − 2 , b, c )
=
− 2 cb + c 2 b ( b +1) .
We also have Γ( D 2 +1)
Γ
( D 2 )
=
D 2 and Γ( D 2 +2)
Γ( D 2 ) =
D 2 (cid:0) D 2 + 1 (cid:1) , which leads to: E
[ z ] =
σ ( D + Ω) and E [ z 2 ]
= σ 2 (2 ω + 2( D + ω ) + ( D + ω ) 2 ) .
Finally, we conclude:
[Equation]
Knowing that z = v ⊤ G x v , and with the proposition: 0 ≤
[Equation]
, we have the desired result.
Lemma 12 Our finsler metric v → E [ (cid:112) v ⊤ G x v ] is defined with v ⊤ G x v ∼ W 1 ( D, v ⊤ Σ x v, ω ) , and ω = ( v ⊤ Σ x v ) − 1 ( v ⊤ E
[ J ] ⊤ E
[ J ] v ) .
We assume that E [ J ] is upper bounded.
Let M ∈ R + , .
We have:
[Equation]
Proof
By definition, Σ does not depend on D .
We assume that every element of the expected jacobian is upper bounded: E
[ J ] ij ≤ m , with m ∈ R ∗ + .
We call σ = v ⊤ Σ x v ∈ R ∗ + .
We have:
[Equation]
with M = σ − 1 m 2 ∥ v ∥ 2 ∈ R ∗ + , and M does not depend on D.
Proposition 13 In high number of dimensions, we have:
[Equation]
And, when D converges toward infinity: L G ∼ L , E G ∼ E and V G ∼ V .
Proof We assume that ω is lower bounded by m ∈ R + .
Using lemma 12, it is also upper bounded by M ∈ R +.
We can show that:
Thus:
[Equation]
with m 1 , m 2 , m 3 ∈ R + .
Using Taylor series expansion, when x ∼ 0, we have: 1 − (1 − x ) q = qx + o ( x 2 ).
Thus:
[Equation]
We also assume that our space is compact.
Using Heine Borel theorem, the metrics and their functional are bounded.
In high dimensions, all functionals converge to each others.
This concludes the proof.
Appendix C. Some experiments
We want to illustrate cases where those metrics could be useful in practice, for real world data.
For this we use three datasets (a synthetic dataset, composed of data representing a pinwheel projected onto a sphere), a font dataset Campbell and Kautz (2014).
We trained a GP-LVM model to learn a 2d-manifold.
From the optimised Gaussian process, we can access the Riemannian and Finsler metric, and minimise their respective curve energies to obtain geodesics.
As we can see, the Finsler and Riemannian geodesics coincide in all cases.
For all latent spaces (A.1. and B.1.
in Figure 3, The heatmap represents the Riemannian volume measure in logarithm scale.
The volume measure is low in area of high density and high in area of low density of data points.
[Figure]
A.1.
B.1.
A.2.
B.2.