Understanding Optimization Challenges when Encoding to Geometric Structures
Editors: Sophia Sanborn, Christian Shewmake, Simone Azeglio, Arianna Di Bernardo, Nina Miolane
Abstract
Geometric inductive biases such as spatial curvature, factorizability, or equivariance have been shown to enable learning of latent spaces which better reﬂect the structure of data and perform better on downstream tasks.
Training such models, however, can be a challenging task due to the topological constraints imposed by encoding to such structures.
In this paper, we theoretically and empirically characterize obstructions to training autoencoders with geometric latent spaces.
These include issues such as singularity (e g. self-intersection), incorrect degree or winding number, and non-isometric homeomorphic embedding.
We propose a method, isometric autoencoder, to improve the stability of training and convergence to an isometric mapping in geometric latent spaces.
We perform an empirical evaluation of this method over 2 domains, which demonstrates that our approach can better circumvent the identiﬁed optimization problems.
1. Introduction
Recent research has shown that geometric inductive biases can be helpful for representation learning (Higgins et al., 2022, 2018; Pfau et al., 2020).
Representations that are equivariant to transformations of the underlying generative factors, many of which can be elegantly described by geometric priors such as groups (rotation, translation, etc.), make it easier to reason about the similarities of diﬀerent instances in the dataset.
In this paper, we focus on geometric inductive biases in the context of unsupervised and weakly-supervised models for representation learning.
While the prevailing intuition is that an inductive bias that matches the underlying topology of the data will guide a model towards a useful representation, there are also indications that certain inductive biases can make a model more diﬃcult to train in practice (Park et al., 2022; Falorsi et al., 2018).
To understand why encoding to geometric structures can give rise to optimization challenges, we formalize diﬀerent topological defects that can occur in a randomly initialized encoder, such as discrepancies in the winding number or crossing number relative to those in a homeomorphic encoder.
We show that these topological defects will be preserved under the assumption of continuous optimization, which shows that escaping these local optima relies on the discrete jumps that are employed during optimization (Section 3).
Moreover, we show that, even in the absence of topological defects, a homeomorphic encoder may violate isometry, which is to say that equidistant points on the abstract manifold may not map to equidistant points in the latent space.
To alleviate these challenges to optimization, we propose an autoencoder objective (Section 4) that encourages equidistant points on the abstract manifold to map to equidistant points in latent space.
Experiments demonstrate that this objective can help to escape local optima during the early stages of training, resulting in more reliable convergence and a greater degree of isometry after training.
2. Problem Statement
Homeomorphic Autoencoders.
We focus on learning representations in domains where we can associate a known geometry with input data.
Concretely, we assume that our data lies on a low-dimensional manifold M that is embedded in a higher-dimensional input space X := R n via a mapping g X : M → X .
This is commonly known as the manifold hypothesis (Bengio et al., 2013).
We denote the image of the mapping by M x := g X ( M ) ⊆
X .
Then g X is a homeomorphism, or topological isomorphism, onto its image g : M ∼ −→ M x .
That is, g is continuous, bijective, and has a continuous inverse.
Our goal is to learn a mapping f φ : X → Z for some suitable latent space Z such that the encoder is itself a homeomorphism when restricted to the embedded manifold M x .
To do so, we follow the design proposed by Falorsi et al.
f φ : X h φ −→ Y π −→ Z .
Here Y :=
R d is an intermediate Euclidean space capable of embedding M .
The ﬁrst function h φ is an ordinary neural network.
The second function π is parameter-less mapping which can be interpreted as a projection to Z .
For example, For example, Y := R 2 and π is the Gram-Schmidt process in the case of group SO(2). As we assume we know the geometry of the data, we set Z to be M
To learn a homeomorphic encoder f φ | M x : M x → M in an unsupervised manner, we will introduce a decoder network f † θ : Z → X .
The composition f † θ ◦ f φ then deﬁnes an autoencoder that can be trained by minimizing the reconstruction loss L φ,θ ( x ) :
= d X ( x, ˆ x ) based on a distance metric d X , where ˆ x := f † θ ( f φ ( x )) .
Note that while minimizing the reconstruction loss ensures that f † θ ◦ f φ | M x approximates the identity, it need not be the case that the composition η φ = f φ ◦ g : M → M is also the identity.
Rather it is a self-bijection on M representing the diﬀerent potential choices of parameterizations of M .
3. Theory
In this paper, we uncover topological obstructions that are distinct from the homological obstruction identiﬁed by de Haan and Falorsi (2018).
The main insight that we put forward is that imposing a geometric structure on the latent space introduce additional obstructions during optimization , which we call optimization obstructions.
We now discuss several speciﬁc optimization obstructions.
We focus on the case where M is the Lie group SO(2) with underlying manifold S 1 , y ∈ R 2 , and z = π ( y ) =
y/ (cid:107) y
(cid:107) .
All the optimization obstructions we consider occur in this case and in the case of higher-dimensional manifolds M as well, but are simpler to describe for S 1 .
See Appendix B & E for more details and proofs.
3.1. Figure Eight Local Minima
To understand what obstructions might arise during optimization, we consider continuoustime training along a gradient ﬂow.
We denote the weights of the initialized encoder as φ (0) and the trained weights as φ (1).
We consider the idealized setting in which φ ( t ) is a continuous function of t .
That is, f φ ( t ) ( x ) is a homotopy from the initialized encoder f φ (0) ( x ) to the trained one f φ (1) ( x ).
Empirically, either at initialization or after some training, we often observe a “ﬁgure 8” pattern in Y (Figure 11, bottom left).
Once this local minimum is obtained, it is very diﬃcult to move out of it in practice using gradient descent.
We make this intuition precise by observing that continuous optimization preserves the ordering of points on the circle as demonstrated by Proposition 1 and Figure 13.
Proposition 1 Assume that f φ ( t ) undergoes continuous optimization. Assume that f φ ( t ) ◦ g is injective for all t . The cyclic ordering induced on k points by f φ (0) is equal to f φ (1) . Thus a ﬁgure 8 embedding, cannot be transformed to a homeomorphic embedding.
In other words, transition from a “ﬁgure 8” to a homeomorphic embedding is impossible without violating continuity during optimization. This suggests that when training with SGD, we will need to rely on the stochasticity of the gradient estimate and discrete jumps to escape this local optimum.
3.2. Degree Obstructions
A second obstruction is that continuous optimization preserves the winding number of a mapping.
This number is deﬁned as follows.
If the embedded image h φ ( M x ) does not contain the origin, then the mapping factors through R 2 \ { (0 , 0) } and is consequently continuous.
In this case η φ = π ◦ h φ ◦ g has a well-deﬁned degree, also known as winding number w ( η φ )
∈ Z .
In order to be a homeomorphic embedding the winding number must be w ( η φ ) ∈ {− 1 , 1 } .
Under random initialization, however, the initial network may have winding number equal to any integer.
Assuming continuous optimization then h φ ( t ) ( x ) is a continuous function of both x, t .
We assume that h φ ( t ) ( x ) (cid:54) =
(0 , 0) for any t, x and thus winding number is deﬁned for any time.
The following proposition thus holds.
Proposition 2 The winding number of the initialized model and ﬁnal model are equal w ( π ◦ h φ (0) ◦ g )
=
w ( π ◦ h φ (1) ◦ g ) .
In practice, neither the continuous optimization assumption nor the avoidance of the origin holds.
Rather h φ is updated by SGD in discrete jumps and h φ ◦ g may map to the origin.
Thus, empirically, we do see that if the weights and learning rate is high enough, the winding number may change.
For more discussion on this, see Appendix B. [Table]
4. Isometric Autoencoders
We argue that preserving relative distance is key for a representation to be useful in most downstream tasks.
For example, if we want to cluster our latent representations based on there orientation, we want the the distances in latent space to reﬂect the actual distance in the abstract manifold M .
For the purpose of representation learning, we do not necessary need the encodings to match the ground truth exactly as long as we are able to compare instances in a meaningful way.
We can formalize this concept in terms isometry .
A function η : X → Z is isometric if d X ( x 1 , x 2 ) =
d Z ( η ( x 1 ) , η ( x 2 )) for all x 1 , x 2 ∈ X .
We now deﬁne our notion of optimality based on this property.
Deﬁnition 3 Let g, f φ , η φ be deﬁned as above.
The encoder f φ is optimal if η φ is isometric.
While an isometry is desirable, it is generally almost impossible to guarantee and practically diﬃcult to verify for a function with neural network components.
Therefore, we need a notion of isometry that is both computable and achievable using standard gradient descent optimization methods.
Motivated by these practical limitations, we deﬁne a less restrictive property that can be veriﬁed in practice in Appendix A. Moreover, we propose an objective that encourages to learn an optimal encoder by making slightly stronger assumptions about the data. Concretely, we assume our dataset has the form D{{ x i, 1 . . . x i,K }} Ni =1 generated as follows.
We assume that M is a group manifold, for example SO(2). We then sample m 1 , m T ∈ M .
We iteratively apply the transformation m T to get a sequence m k = m T m k − 1 ∈ M . Then we pass m k to the ground truth generative network x k = g
( m k ).
Let z k = f ( x k ) be the encoding of x k and z ( k − 1 ,k ) = z k z − 1 k − 1 be the group element that sends z k − 1 to z k .
In each sequence, we know z ( k − 1 ,k ) should be identical for all k . We deﬁne our objective as [Equation]
5. Experiments
We perform a series of experiments to explore the diﬃculty of learning an optimal f φ .
Concretely, we investigate how often do we fall into one of the failure cases described in
Section 3.
We also examine whether training with the Isom-AE objective helps in terms of learning an optimal encoder.
The baselines are discussed in Appendix D. AE Isom-AE 0 1e A E 1e         1e
I s o m
A
E 1e Evaluation. We evaluate all models based on three criteria (1) homeomorphism, (2) isomtery, and (3) reconstructions. Similar to isometry, it is practically diﬃcult to determine if a learned mapping is homeomorphic.
Here, we verify homeomorphism by examining the encoder to check if it has (1) winding number 1 or -1, (2) crossing number 0, (3) yields a continuous path when interpolating in the data manifold from − π to π .
For evaluating isometry, we simply compute and report the term L p ( N ) described in Deﬁnition 4 for p =and N = 10000 rather than applying a binary threshold.
Images.
In this experiment, we train on images of an L-shaped tetromino, a teapot, and an airplane (Shilane et al., 2004).
The SO(2) manifold corresponding to each object is made by rotating the image of the object around the center.
We describe the full sequence generation process in Appendix H.
Results.
We report our ﬁndings in Table 1.
Similar to the 3D crown experiment, we observe that even though the types of obstructions vary across images, both AE and DAE in general fail to learn a homeomorphic encoder.
The Isom-AE objective improves performance noticeably across all metrics.
Contrary to the 3D-Crown experiment (Appendix D), we observe that y regularization is not very helpful.
We observed the main failure case for most of the non-homeomorphic encoders was due to discontinuity.
Isom-AE resolved this issue for the most part, which allows us to interpolate nicely in the Z -space (Figure 1).
What is very surprising is that in the case of airplane, we see that even supervised objective fails to overcome these optimizations obstructions, while an Isom-AE is able to achieve this nicely.
6. Conclusion
In this paper, we investigate obstructions to optimization encountered when learning encoders to topological spaces.
This work contains several limitations.
Firstly, our theoretical analysis is limited by the idealized assumptions necessary to analyze the method using topological tools which do not exactly match those encountered in practice.
Secondly, the metrics we deﬁne such isometry and harder to deﬁne and compute for higher dimensional manifolds.
Future work includes expanding our analysis and techniques to a boarder array of Lie groups and non-group manifolds.
Appendix A. Isometric Representations
Even when a learned representation is homeomorphic, it may not be isometric, in the sense that equidistant points on the abstract manifold M may not map to equidistant points in the latent space Z .
As an example, let us consider the case where M is the special orthogonal Lie group SO(2) deﬁned as
SO(
[Equation]
(cid:26)
[Equation]
(cid:20)
[Equation]
(cid:21) | y [Equation] (cid:107)
[Equation]
(cid:107)
[Equation]
(cid:27) , (2) where GL(2) is the general linear group, the group of invertible 2 × 2 matrices under matrix multiplication.
Deﬁne, for example, the embedding g : SO(2) → R 2 by z (cid:55)→ z (0 , 2) (cid:62) , embedding the manifold as a circle of radius 2.
We consider f φ mapping this embedding back to the group by ﬁrst encoding to Y ⊆ R 2 via h φ and then projecting to the circle S 1 , the manifold underlying the Lie group, via π : R 2 → SO(2) deﬁned y (cid:55)→ A ( y/ (cid:107) y (cid:107) ).
We can construct a mapping f φ = π ◦ h φ by ﬁrst mapping x ∈ R 2 to points on an ellipse,
[Equation] [Equation]
and then projecting them to SO(2) via π as described above.
Restricted to M x := g (SO(2)), f φ is an homeomorphism.
Moreover, as demonstrated in Figure 15 the resulting mapping η φ might not preserve the distances between instances on the manifold M when mapping to the latent space Z .
A.1. Approximate Isometry
While an isometry is desirable, it is generally almost impossible to guarantee and practically diﬃcult to verify for a function with neural network components.
Therefore, we need a notion of isometry that is both computable and achievable using standard gradient descent optimization methods.
Motivated by these practical limitations, we deﬁne a less restrictive property that can be veriﬁed in practice:
Deﬁnition 4 Let f : X → Y .
Let x i , x (cid:48)
i ∼ p
X
( x ) for 1 ≤ i ≤ N , y i = f
( x i ) , y (cid:48)
i =
f ( x (cid:48) i ) , and q i =
d X ( x i , x (cid:48) i ) /d
Y ( y i , y (cid:48) i ) .
Let 1 = (1 , . .
. , 1) ∈ R n .
Deﬁne
[Equation]
We say f is approximately-isometric with probability greater than 1 − α (for (cid:15), α, p, N ), if
[Equation]
Appendix B. Theory: Additional Details B.1. Proposition for the ﬁgure “8” shape
Let ( z 1 , z 2 , z 3 , z 4 ) denote the four end points of the two disjoint intervals of f φ (0) .
Since there is no starting point for the circle, an ordering of these points is only well deﬁned up to cyclic permutations in the group C 4 , that is ( z 1 , z 2 , z 3 , z 4 ) mod C 4 = ( z 2 , z 3 , z 4 , z 1 ) mod C 4 .
Proposition 1 now states that continuous optimization must preserve an ordering up to equivalence.
The proof is illustrated in Figure 13.
B.2. Winding number in practice
Empirically, we do see that if the weights and learning rate is high enough, the winding number may change.
However, if the initialization avoids origin, then due to the tendency of the magnitude of the unnormalized embeddings h φ ( x ) to grow during optimization, the winding number changing becomes more unlikely and thus a signiﬁcant obstruction to ﬁnding homeomorphic embeddings under optimization.
Winding number is also the primary optimization obstruction which makes it impractical to remove the hard projection π .
If instead we decode directly from y ∈ Y but push embeddings to the unit circle using the loss |(cid:107) y (cid:107) − 1 | , then it is far more likely we converge to discontinuous embeddings with the incorrect winding number (Appendix G).
B.3. Magnitude Growth in Y
Empirically, we observe the values of the embeddings in Y =
R 2 continually grow during training.
This phenomenon makes it more diﬃcult for the embedded data manifold h φ ( M x ) to cross the origin and for the winding number to change.
We give a theoretical explanation for this behavior.
Consider what would happen if the embedding y were updated directly based on the gradient of the loss ∇ y L with respect to y .
We assume the loss depends only on z = y/ (cid:107) y (cid:107) , and so has level sets which are unions of radial rays from the origin.
The gradient ∇ y L must then be tangent to a circle about the origin.
That is, for y =
( a, b ), the gradient ∇ y L = ( ± b, ∓ a ).
Under gradient ﬂow, the evolution of y in time y t would thus ﬂow along circles of ﬁxed radius and so (cid:107) y 0 (cid:107) =
(cid:107) y t (cid:107) .
Under gradient descent, however, due to the convexity of the ﬂow lines, which are circular, the embeddings y will tend to grow in magnitude.
For η ∈ R > 0 , we compute
[Equation]
In practice, however, we do not update y based on ∇ y L but rather based on the gradient with respect to model parameters φ .
Let F : Φ → Y be the map from model parameters φ to y given ﬁxed input data x .
Then the actual update to y is ˜ ∇ y L = dF T ◦ ∇ φ L where dF is the total derivative or Jacobian of the map F .
Since ∇ φ L = ( dF ) ∇ y L we have ˜ ∇ y L = dF T dF ∇ y L .
The angle between ˜ ∇ y L and ∇ y L is bounded by some θ a quantity depending on the eigenvalues of the operator dF .
Given that ∇ y L is tangential to the circle, assuming for simplicity ˜ ∇ y L has constant length L and uniform distribution [ − θ, θ ] in angle to ∇ y L , the norm of y still grows in expectation .
Proposition 5 Assume a circle of radius R .
Let v be a random vector at y on the circle of length L < R with angle to the tangent uniform in [ − θ, θ ] .
Then
[Equation]
Appendix C. Related Work Unsupervised or Weakly Supervised Learning of Geometric Representations
There has been a large amount of work concerned with learning representations from data with geometric structure in the unsupervised, weakly supervised, or semi-supervised setting. To this end, diﬀerent methodologies have been brought forward, including the use of hyperspherical prior distributions (Davidson et al., 2018a), specializations of the reparameterization trick (Falorsi et al., 2018; Rey et al., 2019), and making use of local connectivity information (Moor et al., 2020; Chen et al., 2021; Lee et al., 2021). Learning Disentangled Representations.
Topological group structure in data can also be used to deﬁne a notion of disentanglement based on the invariance properties of the representations under certain group transformations (Higgins et al., 2018).
This has given rise to a corpus of work that aims to learn disentangled group representation in various settings (Caselles-Dupr´e et al., 2019; Quessard et al., 2020; Tonnaer et al., 2022; Pfau et al., 2020; Zhu et al., 2021). Tonnaer et al. (2022) propose an objective which is similar our Isom-AE objective in Section 1 to learn Linear Symmetry-Based Disentangled (LSBD) representations.
However, this objective assumes knowledge of ground truth group elements while our objective only assumes that sequences are generated by repeated application of the same unknown group element. Our work diﬀers from these approaches in that we are not interested in disentanglement, but focus on fundamental topological obstructions when encoding to group with non-trivial topological structure
. Topological Obstructions in Learning
To develop a better understanding of the failure modes that we observe in Figure 14, we will formalize topological obstructions to training homeomorphic embeddings in Section 3.
Homological obstructions to auto encoding have been observed (de Haan and Falorsi, 2018; Batson et al., 2021; Falorsi et al., 2018; Rey et al., 2019) in previous work.
de Haan and Falorsi 2018, Theorem 1 give a formal condition.
For any latent space Z with non-trivial topology, it is possible to learn an encoder f φ that is continuous when restricted to M x ⊂ X , but this encoder must be discontinuous on the full space X .
For this reason, Falorsi et al. and others (Xu and Durrett, 2018; Davidson et al., 2018b; Meng et al., 2019) use the two-part encoder like ours, inserting discontinuous layers π when mapping to circles, spheres, SO( n ), or other manifolds.
This explicit discontinuity circumvents the homological obstruction without forcing the linear layers of the network to approximate discontinuities using large weights, which we and others ﬁnd leads to instability during training and inferior reconstructions (Section 5 and Falorsi et al.
Appendix D. Experimental Details Baselines.
Throughout our experiments, we compare against (1) a standard AE where the objective is only based on reconstruction loss, (2) a supervised AE where in addition to minimizing the reconstruction loss, the encoder is trained to predict the ground truth representations.
These two scenarios serve as extremes on the spectrum of guiding the model to the right representation.
As another baseline, we also compare against a denoising autoencoders (DAE) which aids to learn a smoother embedding space.
In addition to
Isom-AE, we also tried regularizing the y-space to be close S 1 (by penalizing ( (cid:107) y (cid:107) 2 − 1) 2 ) in order to mitigate the optimization problem discussed in Subsection B.3, which we refer to as “Isom-AE + reg- y ”.
We train all our models for 100 epochs with a batch-size of 100.
For optimization, we use the RAdam optimizer (Liu et al., 2019) with a learning rate of 5e-4.
For low-dimensional cases, we use a 4-layer MLP with 512 hidden units followed by a tanh activation for both the encoder and decoder architecture (Table 2).
For the high-dimensional image datasets, we use a 4-layer CNN with kernel, stride, and padding of size 4, 2 and 1 respectively followed by a leakyReLU activation (Table 3).
All models were initialized and trained with 15 1 diﬀerent random seeds.
Though we hypothesize that these topological defects may occur in a variety of topological structures, in this paper we restrict our experiments to the Lie group SO (2) with π deﬁned as in Section 2.
Encoder
[Table]
Decoder
[Table]
[Table]
[Table]
Appendix E. Proofs
We include the proofs for the propositions in the main text.
E.1. Figure Eight Local Minimum
Proposition 6 Assume that f φ ( t ) undergoes continuous optimization and is thus continuous in t .
Assume that π ◦ f φ ( t ) ◦ g is injective for all t .
The cyclic ordering induced on k points by f φ (0) is equal to that induced by f φ (1) .
Thus a ﬁgure 8 embedding, which corresponds to cyclic order ( z 1 , z 2 , z 4 , z 3 ) mod C 4 , cannot be transformed to a homeomorphic embedding, which has cyclic order ( z 1 , z 2 , z 3 , z 4 ) mod C 4 or ( z 4 , z 3 , z 2 , z 1 ) mod C 4 .
Proof Since we assume π ◦ f φ ( t ) ◦ g is injective for all t , the path z ( t ) =
( π ◦ f φ (0) ◦ g ( θ i )) 4 i =1 is inside the k -fold conﬁguration space on S 1 deﬁned Conf k
( S 1 )
=
{ ( z 1 , . . .
, z k ) ∈ ( S 1 ) k
: z i (cid:54) =
z j for i (cid:54) = j } .
In order to prove the claim, we will show that the path-connected components of Conf k ( S 1 ) correspond to cyclic orderings of ( z 1 , . . .
, z k ) and thus the start and end point of every path share a cyclic ordering.
Mapping ( z 1 , . . .
, z k ) (cid:55)→ ( z k , ( z − 1 k z 1 , . . .
, z − 1 k z k − 1 )) gives a homeomorphism Conf k
( S 1 ) ∼ =
SO(2) ×
Conf k − 1 ( S 1 \ { 1 } ) ∼ = SO(2) ×
Conf k − 1 ( R ).
Let ˜ z
i =
z − 1 k z i .
Consider D = { (˜ z 1 , . .
˜
z k − 1 ) : ˜
z 1 < . . .
<
˜
z k − 1 } ⊂ Conf k − 1 ( R ).
We can identify the connected components of Conf k − 1 ( R ).
The set D is a fundamental domain for the action of the symmetric group S k − 1 on Conf k − 1 ( R ).
Thus Conf k − 1 ( R ) = (cid:96) σ ∈ S k
σ ( D ) is a disjoint union.
Linear interpolation shows D is connected.
The sets D and σ ( D ) for σ ∈ S
k are not connected.
Consider a path from z = ( z 1 , . .
z k ) ∈
D to σ ( z ) ∈ σ ( D ).
The element σ must reverse the order of at least two elements z j < z i .
Thus the function f ( z ) =
z i − z
j must take the value 0 over the path by intermediate value theorem.
Hence the path cannot be in Conf k − 1 ( R ).
Thus the connected components of π 0 (Conf k − 1 ( R )) ∼ = S k − 1 .
Since SO(2) is connected, π 0 (Conf k ( S 1 )) ∼ = S k − 1 .
That is each connected component of Conf k
( S 1 ) is labeled by an element of S k − 1 describing the ordering of ˜
z 1 , . .
˜
z k − 1 in R .
Each ordering of (˜ z 1 , . . .
˜
z k − 1 ) in turn corresponds to a diﬀerent cyclic ordering of z 1 , . . .
, z k in S 1 , that is, a diﬀerent element of S k /C k .
Thus two k -point conﬁgurations are homotopic if and only if they have the same cyclic ordering.
E.2. Degree Obstruction
Proposition 7 The winding number of the initialized model and ﬁnal model are equal w ( π (cid:48) ◦ h φ (0) ◦ g )
= w ( π (cid:48) ◦ h φ (1) ◦ g ) .
Proof The winding number of a map is a continuous function t (cid:55)→ w ( π (cid:48) ◦ h φ ( t ) ◦ g ).
Since the output space Z is discreet, the winding number must be constant in t .
E.3. Magnitude Growth in Y
As noted in the main text, we have ˜ ∇ y L
= ( dF T dF ) ∇ y L .
We assume that dF is full rank , which is a reasonable assumption for an overparameterized neural network.
In that case
M = dF T dF is a positive deﬁnite symmetric matrix and can be orthogonally diagonalized
M = Q Λ Q T where Q is orthogonal and
[Equation]
and λ i > 0.
The maximum angle between x = ∇ y L and M x = ˜ ∇ y L can then computed in terms of the eigenvalues
λ i .
This maximum is computed for the case of an n × n symmetric positive deﬁnite matrix here 2 .
We include the proof for the 2 × 2 case we consider here for completeness.
Lemma 8 The maximum angle between x and Mx for x ∈ R 2 (cid:54) =0 is
[Equation]
Proof The angle is maximized at the minimum value of
[Equation]
It suﬃces to consider (cid:107) x (cid:107) = 1.
Substituting M = Q Λ Q T and y
=
Qx , we want to minimize
[Equation]
over all (cid:107) y (cid:107) = 1 since (cid:107) Qx (cid:107)
=
(cid:107) x (cid:107) = 1.
Letting a = y 21 and noting y
+ y 22 = 1, this is equal to minimizing
[Equation]
over 0 ≤ a ≤ 1.
Setting the derivative equal to 0 gives
[Equation]
and yields one critical value at a = λ 2 / ( λ 1 + λ 2 ) corresponding to value
[Equation]
λ
This is the global minimum since the boundary values a = 0 and a = 1 correspond to maxima with value 1.
Thus the angle between ∇ y L and ˜ ∇ y L is bounded by θ =
[Equation]
Given that ∇ y L is tangential to the circle, assuming for simplicity ˜ ∇ y L has constant length
L and uniform distribution [ − θ, θ ] in angle to ∇ y L , the norm of y grows in expectation .
Proposition 9 Assume a circle of radius R .
Let v be a random vector at y on the circle of length L with angle to the tangent uniform in [ − θ, θ ] .
Then
[Equation]
Proof Without loss of generality, y = (0 , R ) and v =
( L cos t, L sin t ) where | t | < θ .
Then we evaluate
[Equation]
by Pythagorean identity and the fact sin t is odd.
Appendix F. Continuity Metric
For measuring continuity, we adopt a similar method as Falorsi et al.
and evaluate continuity in terms of how the largest “jump” compare to others when walking a continuous path m i ∈ M for i = 1 · · · N pairwise close points.
We compute q i similar to Deﬁnition 4 by computing the relative distances
[Equation]
From the set { q i } i , we compute the continuity metric L cont as
[Equation]
In our experiments, we set α = 90.
There are two diﬀerences between how we evaluate continuity compared to (Falorsi et al., 2018).
First, we measure the continuity of η φ rather than f φ which we argue is more relevant in Section 4.
Second, in (Falorsi et al., 2018), the authors are mainly interested in verifying whether the encoder is discontinuous in topological sense (which they verify by examining the inequality M > γP α for some γ ).
We on the other hand report continuity on the spectrum by computing the γ that would make
[Table]
Appendix G. Decoding From Y
We consider the alternate strategy of removing the projection π and adding a loss so that y stays close to the desired manifold M in Y .
Winding number obstructions become far more prominent in this case.
In Figure 2, we show the latent space for diﬀerent random seeds in the teapot case when we train with such an objective.
[Table]
Appendix H. Additional Experiments H.1. 3D Crown
In our ﬁrst experiment [Table] we consider a simple dataset in R 3 with parameterization x
= (cos θ, sin θ, sin 4 θ ) for θ ∈ [ − π, π ].
We generate 60k sequences of data points with length
K =
In each sequence, a constant rotation of θ i is applied where θ i is sampled from
U( − π/ 5 , π/ 5).
For the objectives that do not rely on sequences, we simply ﬂatten data before training.
Results.
We report our ﬁndings in Table 4.
We can see that in the majority of cases, both AE and DAE fail to learn a homeomorphic encoder.
Interestingly, though the data is low-dimensional, it is still diﬃcult to model due to non-linearity of the data manifold embedding.
Concluding from the low crossing number (as well examining the latent spaces in Figure 10), the most common failure cases for learning a homeomorphic encoder are the ﬁgure eight obstruction as well as discontinuity.
Isom-AE mitigates these problems to an extent, but not perfectly.
We show an example of how isom-AE is able to unwind the latent the space to achieve a homeomorphic mapping in Figure 7 3 .
Isom-AE with y regularization, however, is able to avoid such obstructions completely, achieving a 100% homeomorphic success rate along with Sup-AE.
We can also observe from Table 4 that homeomorphic mappings are overall more stable during training.
Lastly, looking at the learning curves in
Figure 5, we observe that winding number is susceptible to changing during training.
e = e = e = e = e =  h φ Encode π
Project
Appendix I. Additional Figures
[Figure]
[Figure]
e =e =e =e =e =   1.
A E
I s o m
A
E
AE
Isom
[Figure]
AE *  1e
A E 1e          1e
I s o m
A
E 1e
[Figure]
[Figure]
AE
Isom-AE *  1e
A E 1e       2.
1e 2.
I s o m
A
E 1e
[Figure]
[Table]
The cyclic order of these 4 endpoints is preserved by homotopy.
Following the parameterization of the data manifold the cyclic order is ( z 1 , z 2 , z 4 , z 3 ), which is distinct from the cyclic order of a homeomorphic embedding, either ( z 1 , z 2 , z 3 , z 4 ) or ( z 4 , z 3 , z 2 , z 1 ).