Original: T OPOLOGICAL A UTOENCODERS
Revised: T OPOLOGICAL A UTOENCODERS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose a novel approach for preserving topological structures of the input space in latent representations of autoencoders.
Revised: We propose a novel approach for preserving topological structures of the input space in latent representations of autoencoders.

Original: Using persistent homology , a technique from topological data analysis, we calculate topological signatures of both the input and latent space to derive a topological loss term.
Revised: Using persistent homology , a technique from topological data analysis, we calculate topological signatures of both the input and latent space to derive a topological loss term.

Original: Under weak theoretical assumptions, we can construct this loss in a differentiable manner, such that the encoding learns to retain multi-scale connectivity information.
Revised: Under weak theoretical assumptions, we can construct this loss in a differentiable manner, such that the encoding learns to retain multi-scale connectivity information.

Original: We show that our approach is theoretically well-founded and that it exhibits favourable latent representations on a synthetic manifold as well as on real-world image data sets, while preserving low reconstruction errors.
Revised: We show that our approach is theoretically well-founded and that it exhibits favourable latent representations on a synthetic manifold as well as on real-world image data sets, while preserving low reconstruction errors.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: While topological features, in particular multi-scale features derived from persistent homology, have seen increasing usage in the machine learning community (Carrière et al., 2019; Guss &
Revised: While topological features, in particular multi-scale features derived from persistent homology, have seen increasing usage in the machine learning community (Carrière et al., 2019; Guss &

Original: Salakhutdinov, 2018; Hofer et al., 2017; 2019a;b; Ramamurthy et al., 2019; Reininghaus et al., 2015;
Revised: Salakhutdinov, 2018; Hofer et al., 2017; 2019a;b; Ramamurthy et al., 2019; Reininghaus et al., 2015;

Original: Rieck et al., 2019a;b), using topology directly as a constraint for current deep learning methods remains a challenge.
Revised: Rieck et al., 2019a;b), using topology directly as a constraint for current deep learning methods remains a challenge.

Original: This is due to the inherently discrete nature of these computations, making backpropagation through the computation of topological signatures immensely difﬁcult or only possible in certain special circumstances (Chen et al., 2019;
Revised: This is due to the inherently discrete nature of these computations, making backpropagation through the computation of topological signatures immensely difﬁcult or only possible in certain special circumstances (Chen et al., 2019;

Original: Hofer et al., 2019b; Poulenard et al., 2018).
Revised: Hofer et al., 2019b; Poulenard et al., 2018).

Original: In this work, we present a novel approach that permits obtaining gradients during the computation of topological signatures.
Revised: In this work, we present a novel approach that permits obtaining gradients during the computation of topological signatures.

Original: This permits employing topological constraints while training deep neural networks as well as building topology-preserving autoencoders based on the following contributions:
Revised: This permits employing topological constraints while training deep neural networks as well as building topology-preserving autoencoders based on the following contributions:

Original: We develop a novel topological loss term for autoencoders that helps harmonise the topology of the data space and the topology of the latent space.
Revised: We develop a novel topological loss term for autoencoders that helps harmonise the topology of the data space and the topology of the latent space.

Original: We prove that our approach is stable on the level of mini-batches, resulting in suitable approximations of the persistent homology of a data set.
Revised: We prove that our approach is stable on the level of mini-batches, resulting in suitable approximations of the persistent homology of a data set.

Original: We empirically demonstrate that our novel loss term aids in dimensionality reduction by preserving topological structures in data sets; in particular, the learned latent representations are useful in that the preservation
Revised: We empirically demonstrate that our novel loss term aids in dimensionality reduction by preserving topological structures in data sets; in particular, the learned latent representations are useful in that the preservation

Original: of topological structures
Revised: of topological structures

Original: can aid interpretability.
Revised: can aid interpretability.

Original:  B ACKGROUND : P ERSISTENT HOMOLOGY
Revised:  B ACKGROUND : P ERSISTENT HOMOLOGY

Original: Persistent homology (Edelsbrunner & Harer, 2008) is a method from the ﬁeld of computational topology, which develops tools for analysing topological features (connectivity-based features such as connected components) of data sets.
Revised: Persistent homology (Barannikov, 1994; Edelsbrunner & Harer, 2008) is a method from the ﬁeld of computational topology, which develops tools for analysing topological features (connectivitybased features such as connected components) of data sets.

Original: We ﬁrst need to introduce the underlying concept of simplicial homology.
Revised: We ﬁrst need to introduce the underlying concept of simplicial homology.

Original: For a simplicial complex K , i.e. a generalised graph with higher-order connectivity information such as cliques, simplicial homology employs matrix reduction algorithms to assign K a family of groups, the homology groups .
Revised: For a simplicial complex K , i.e. a generalised graph with higher-order connectivity information such as cliques, simplicial homology employs matrix reduction algorithms to assign K a family of groups, the homology groups .

Original: The d th homology group H d ( K ) of K contains d -dimensional topological features, such as connected components ( d = 0 ), cycles/tunnels ( d = 1 ), and voids ( d = 2 ).
Revised: The d th homology group H d ( K ) of K contains d -dimensional topological features, such as connected components ( d = 0 ), cycles/tunnels ( d = 1 ), and voids ( d = 2 ).

Original: Homology groups are typically summarised by their ranks, thereby obtaining a simple invariant “signature” of a manifold.
Revised: Homology groups are typically summarised by their ranks, thereby obtaining a simple invariant “signature” of a manifold.

Original: For example, a circle in R 2 has one feature with d = 1 (a cycle), and one feature with d = 0 (a connected component).
Revised: For example, a circle in R 2 has one feature with d = 1 (a cycle), and one feature with d = 0 (a connected component).

Original: In practice, the underlying manifold M is unknown and we are working with a point cloud X := { x 1 , . . .
Revised: In practice, the underlying manifold M is unknown and we are working with a point cloud X := { x 1 , . . .

Original: , x n } ⊆ R d and a metric dist: X × X
Revised: , x n } ⊆ R d and a metric dist: X × X

Original: → R such as the Euclidean distance.
Revised: → R such as the Euclidean distance.

Original: Persistent homology extends simplicial homology to this setting: instead of approximating M by means of a single simplicial complex, which wouldbe an unstable procedure
Revised: Persistent homology extends simplicial homology to this setting: instead of approximating M by means of a single simplicial complex, which would

Original: [Figure]
Revised: [Figure]

Original: As the distance threshold (cid:15) increases, the connectivity changes.
Revised: As the distance threshold (cid:15) increases, the connectivity changes.

Original: The creation and destruction of d -dimensional topological features is recorded in the d th persistence diagram D d .  due to the discrete nature of X , persistent homology tracks changes in the homology groups over multiple scales of the metric.
Revised: The creation and destruction of d -dimensional topological features is recorded in the d th persistence diagram D d . be an unstable procedure due to the discrete nature of X , persistent homology tracks changes in the homology groups over multiple scales of the metric.

Original: This is achieved by constructing a special simplicial complex, the Vietoris–Rips complex (Vietoris, 1927).
Revised: This is achieved by constructing a special simplicial complex, the Vietoris–Rips complex (Vietoris, 1927).

Original: For 0 ≤ (cid:15) < ∞ , the Vietoris–Rips complex of X at scale (cid:15) , denoted by R (cid:15) ( X ) , contains all simplices (i.e. subsets) of X whose elements { x 0 , x 1 , . . .
Revised: For 0 ≤ (cid:15) < ∞ , the Vietoris–Rips complex of X at scale (cid:15) , denoted by R (cid:15) ( X ) , contains all simplices (i.e. subsets) of X whose elements { x 0 , x 1 , . . .

Original: } satisfy dist( x i , x j ) ≤ (cid:15) for all i , j .
Revised: } satisfy dist( x i , x j ) ≤ (cid:15) for all i , j .

Original: Given a matrix A of pairwise distances of a point cloud X , we will use R (cid:15) ( A ) and R (cid:15) ( X ) interchangeably because constructing R (cid:15) only requires distances.
Revised: Given a matrix A of pairwise distances of a point cloud X , we will use R (cid:15) ( A ) and R (cid:15) ( X ) interchangeably because constructing R (cid:15) only requires distances.

Original: Vietoris–Rips complexes satisfy a nesting relation, i.e. R (cid:15) i ( X ) ⊆
Revised: Vietoris–Rips complexes satisfy a nesting relation, i.e. R (cid:15) i ( X ) ⊆

Original: R (cid:15) j
Revised: R (cid:15) j

Original: ( X ) for (cid:15) i ≤ (cid:15) j , making it possible to track changes in the homology groups as (cid:15) increases (Edelsbrunner et al., 2002).
Revised: ( X ) for (cid:15) i ≤ (cid:15) j , making it possible to track changes in the homology groups as (cid:15) increases (Edelsbrunner et al., 2002).

Original: Figure 1 illustrates this process.
Revised: Figure 1 illustrates this process.

Original: Since X contains a ﬁnite number of points, a maximum ˜ (cid:15) value exists for which the connectivity stabilises; thus, calculating R ˜ (cid:15) sufﬁces to obtain topological features at all scales.
Revised: Since X contains a ﬁnite number of points, a maximum ˜ (cid:15) value exists for which the connectivity stabilises; therefore, calculating R ˜ (cid:15) is sufﬁcient to obtain topological features at all scales.

Original: We write PH( R (cid:15) ( X )) for the persistent homology calculation of the Vietoris–Rips complex.
Revised: We write PH( R (cid:15) ( X )) for the persistent homology calculation of the Vietoris–Rips complex.

Original: It results in a tuple ( {D 1 , D 2 , . . .
Revised: It results in a tuple ( {D 1 , D 2 , . . .

Original: } , { π 1 , π 2 , . . . } ) of persistence diagrams ( 1 st component) and persistence pairings ( 2 nd component).
Revised: } , { π 1 , π 2 , . . . } ) of persistence diagrams ( 1 st component) and persistence pairings ( 2 nd component).

Original: The d -dimensional persistence diagram D d (Figure 1d) of R (cid:15) ( X ) contains coordinates of the form ( a, b ) , where a refers to a threshold (cid:15) at which a d -dimensional topological feature is created in the Vietoris–Rips complex, and b refers to a threshold (cid:15) (cid:48) at which it is destroyed.
Revised: The d -dimensional persistence diagram D d (Figure 1d) of R (cid:15) ( X ) contains coordinates of the form ( a, b ) , where a refers to a threshold (cid:15) at which a d -dimensional topological feature is created in the Vietoris–Rips complex, and b refers to a threshold (cid:15) (cid:48) at which it is destroyed.

Original: When d = 0 , for example, the threshold (cid:15) (cid:48) indicates at which distance two connected components in X are merged into one; this is related to spanning trees (Kurlin, 2015).
Revised: When d = 0 , for example, the threshold (cid:15) (cid:48) indicates at which distance two connected components in X are merged into one; this is related to spanning trees (Kurlin, 2015).

Original: The d -dimensional persistence pairing contains indices ( i, j ) corresponding to simplices s i , s j ∈ R (cid:15) ( X ) that create and destroy the topological feature identiﬁed by ( a, b ) ∈ D d , respectively.
Revised: The d -dimensional persistence pairing contains indices ( i, j ) corresponding to simplices s i , s j ∈ R (cid:15) ( X ) that create and destroy the topological feature identiﬁed by ( a, b ) ∈ D d , respectively.

Original: Persistence diagrams are known to be stable with respect to small perturbations in the data set (Cohen-Steiner et al., 2007).
Revised: Persistence diagrams are known to be stable with respect to small perturbations in the data set (Cohen-Steiner et al., 2007).

Original: Two diagrams D and D (cid:48) can be compared using the bottleneck distance d b ( D , D (cid:48) ) :
Revised: Two diagrams D and D (cid:48) can be compared using the bottleneck distance d b ( D , D (cid:48) ) :

Original: = inf η : D→D (cid:48) sup x ∈D (cid:107) x − η ( x ) (cid:107) ∞ , where η : D → D (cid:48) denotes a bijection between the points of the two diagrams, and (cid:107) · (cid:107) ∞ refers to the L ∞ norm.
Revised: = inf η : D→D (cid:48) sup x ∈D (cid:107) x − η ( x ) (cid:107) ∞ , where η : D → D (cid:48) denotes a bijection between the points of the two diagrams, and (cid:107) · (cid:107) ∞ refers to the L ∞ norm.

Original: We use D X to refer to the set of persistence diagrams of a point cloud X arising from PH( R (cid:15) ( X )) .
Revised: We use D X to refer to the set of persistence diagrams of a point cloud X arising from PH( R (cid:15) ( X )) .

Original:  P ROPOSED METHOD
Revised:  P ROPOSED METHOD

Original: We propose a generic framework for constraining autoencoders to preserve topological structures (measured via persistent homology) of the data space in their latent encodings.
Revised: We propose a generic framework for constraining autoencoders to preserve topological structures (measured via persistent homology) of the data space in their latent encodings.

Original: 3. V IETORIS –R IPS COMPLEX CALCULATION
Revised: 3. V IETORIS –R IPS COMPLEX CALCULATION

Original: Given a ﬁnite metric space S , we ﬁrst calculate the persistent homology of the Vietoris–Rips complex of its distance matrix A S .
Revised: Given a ﬁnite metric space S , we ﬁrst calculate the persistent homology of the Vietoris–Rips complex of its distance matrix A S .

Revised: Typically, the usual Euclidean distance is used to calculate A S , but both the persistent homology calculation and our method are not restricted to any particular distance; previous research (Wagner & Dłotko, 2014) shows that even similarity measures that do not satisfy the properties of a metric can be used successfully with PH( · ) .

Original: Let (cid:15) :
Revised: Subsequently, let (cid:15) :

Original: = max A S so that R (cid:15) (cid:0) A S (cid:1) is the corresponding Vietoris– Rips complex as described in Section 2.
Revised: = max A S so that R (cid:15) (cid:0) A S (cid:1) is the corresponding Vietoris–Rips complex as described in Section 2.

Original: Given a maximum dimension 1 of d ∈ N > 0 , we obtain a set of persistence diagrams D S , and a set of persistence pairings π S .
Revised: Given a maximum dimension 1 of d ∈ N > 0 , we obtain a set of persistence diagrams D S , and a set of persistence pairings π S .

Original: The d th persistence pairing π S d contains indices of simplices that are relevant for creating and destroying d -dimensional topological features.
Revised: The d th persistence pairing π S d contains indices of simplices that are relevant for creating and destroying d -dimensional topological features.

Original: We can consider each pairing to represent edge indices , namely the edges that are deemed to be “topologically relevant” by the computation of persistent homology (see below for more details).
Revised: We can consider each pairing to represent edge indices , namely the edges that are deemed to be “topologically relevant” by the computation of persistent homology (see below for more details).

Original: This works because the Vietoris–Rips complex is a clique complex , i.e. it is fully determined by its edges (Zomorodian, 2010).
Revised: This works because the Vietoris–Rips complex is a clique complex , i.e. it is fully determined by its edges (Zomorodian, 2010).

Original: For 0 -dimensional topological features, it is sufﬁcient to look at the edge indices , i.e. the indices of “destroyer” simplices, contained in π S 0 .
Revised: For 0 -dimensional topological features, it is sufﬁcient to look at the edge indices , i.e. the indices of “destroyer” simplices, contained in π S 0 .

Original: Each of these indices corresponds to an edge in the minimum spanning tree of the data set.
Revised: Each of these indices corresponds to an edge in the minimum spanning tree of the data set.

Original: This calculation is computationally efﬁcient, having a worst-case complexity of O (cid:0) m 2 · α (cid:0) m 2 (cid:1)(cid:1) , where m is the batch size and α ( · ) denotes the extremely slow-growing inverse Ackermann function (Cormen et al., 2009, Chapter 22).
Revised: This calculation is computationally efﬁcient, having a worst-case complexity of O (cid:0) m 2 · α (cid:0) m 2 (cid:1)(cid:1) , where m is the batch size and α ( · ) denotes the extremely slow-growing inverse Ackermann function (Cormen et al., 2009, Chapter 22).

Original: For 1 -dimensional features, where edges are paired with triangles, we obtain edge indices by selecting the edge with the maximum weight of the triangle.
Revised: For 1 -dimensional features, where edges are paired with triangles, we obtain edge indices by selecting the edge with the maximum weight of the triangle.

Original: While this procedure (and thus our method) generalises to higher dimensions, our current implementation supports no higher-dimensional features.
Revised: While this procedure (and thus our method) generalises to higher dimensions, our current implementation supports no higher-dimensional features.

Original: Since preliminary experiments showed that including 1 -dimensional topological features merely increases runtime, we will focus only on 0 -dimensional persistence diagrams in the subsequent experiments.
Revised: Since preliminary experiments showed that including 1 -dimensional topological features merely increases runtime, we will focus only on 0 -dimensional persistence diagrams in the subsequent experiments.

Original: We thus use (cid:0)
Revised: We thus use (cid:0)

Original: D S , π S (cid:1) to denote the 0-dimensional persistence diagram and pairing of S , respectively.
Revised: D S , π S (cid:1) to denote the 0-dimensional persistence diagram and pairing of S , respectively.

Original: 3. T OPOLOGICAL AUTOENCODER
Revised: 3. T OPOLOGICAL AUTOENCODER

Original: In the following, we consider a mini-batch X of size m from the data space X as a point cloud.
Revised: In the following, we consider a mini-batch X of size m from the data space X as a point cloud.

Original: Furthermore, we deﬁne an autoencoder as the composition of two functions h ◦ g , where g : X → Z represents the encoder and h : Z → X represents the decoder .
Revised: Furthermore, we deﬁne an autoencoder as the composition of two functions h ◦ g , where g : X → Z represents the encoder and h : Z → X represents the decoder .

Original: We denote latent codes with
Revised: We denote latent codes with

Original: Z := g ( X ) .
Revised: Z := g ( X ) .

Original: During a forward pass of the autoencoder, we compute the persistent homology of the mini-batch in both the data as well as the latent space, yielding two sets of tuples, i.e. (cid:0)
Revised: During a forward pass of the autoencoder, we compute the persistent homology of the mini-batch in both the data as well as the latent space, yielding two sets of tuples, i.e. (cid:0)

Original: D X , π X (cid:1) :
Revised: D X , π X (cid:1) :

Original: =
Revised: =

Original: PH( R (cid:15) ( X )) and (cid:0) D Z , π Z (cid:1) :
Revised: PH( R (cid:15) ( X )) and (cid:0) D Z , π Z (cid:1) :

Original: =
Revised: =

Original: PH( R (cid:15) ( Z )) .
Revised: PH( R (cid:15) ( Z )) .

Original: The values of the persistence diagram can be retrieved by subsetting the distance matrix with the indices provided by the persistence pairings; we write D X (cid:39)
Revised: The values of the persistence diagram can be retrieved by subsetting the distance matrix with the indices provided by the persistence pairings; we write D X (cid:39)

Original: A X (cid:2) π X (cid:3) to indicate that the diagram, which is a set, contains the same information as the distances we retrieve with the pairing.
Revised: A X (cid:2) π X (cid:3) to indicate that the diagram, which is a set, contains the same information as the distances we retrieve with the pairing.

Original: We treat A X (cid:2) π X (cid:3) as a vector in R | π X | .
Revised: We treat A X (cid:2) π X (cid:3) as a vector in R | π X | .

Original: Informally speaking, the persistent homology calculation can thus be seen as a selection of topologically relevant edges of the Vietoris–Rips complex, followed by the selection of corresponding entries in the distance matrix.
Revised: Informally speaking, the persistent homology calculation can thus be seen as a selection of topologically relevant edges of the Vietoris–Rips complex, followed by the selection of corresponding entries in the distance matrix.

Original: By comparing both diagrams, we can construct a topological regularisation term
Revised: By comparing both diagrams, we can construct a topological regularisation term

Original: L t :=
Revised: L t :=

Original: L t (cid:0) A X , A Z , π X , π Z (cid:1) , which we add to the reconstruction loss of an autoencoder, i.e.
Revised: L t (cid:0) A X , A Z , π X , π Z (cid:1) , which we add to the reconstruction loss of an autoencoder, i.e.

Original: [Equation]
Revised: [Equation]

Original: where λ ∈ R is a parameter to control the strength of the regularisation.
Revised: where λ ∈ R is a parameter to control the strength of the regularisation.

Original: Next, we discuss how to specify L t : in our case, the PH calculation represents a selection of topologically relevant distances from the distance matrix.
Revised: Next, we discuss how to specify L t : in our case, the PH calculation represents a selection of topologically relevant distances from the distance matrix.

Original: Each persistence diagram entry corresponds to a distance between two data points.
Revised: Each persistence diagram entry corresponds to a distance between two data points.

Original: We assume that the distances are unique so that each entry in the diagram has an inﬁnitesimal neighbourhood that only contains a single point.
Revised: We assume that the distances are unique so that each entry in the diagram has an inﬁnitesimal neighbourhood that only contains a single point.

Original: Given this ﬁxed pairing and a differentiable distance function, the persistence diagram entries are thus also differentiable with respect to the encoder parameter.
Revised: Given this ﬁxed pairing and a differentiable distance function, the persistence diagram entries are thus also differentiable with respect to the encoder parameter.

Original: Hence, the persistence pairing does not change upon a small perturbation of the underlying distances, thereby guaranteeing the existence of the derivative of our loss function.
Revised: Hence, the persistence pairing does not change upon a small perturbation of the underlying distances, thereby guaranteeing the existence of the derivative of our loss function.

Original: This, in turn, permits the calculation of gradients for backpropagation.
Revised: This, in turn, permits the calculation of gradients for backpropagation.

Original: A straightforward approach to impose the data space topology on the latent space would be to directly calculate a loss based on the selected distances in both spaces.
Revised: A straightforward approach to impose the data space topology on the latent space would be to directly calculate a loss based on the selected distances in both spaces.

Original: Such an approach will not result in informative gradients for the autoencoder, as it merely compares topological features without matching 2 the edges between R (cid:15) ( X ) and R (cid:15)
Revised: Such an approach will not result in informative gradients for the autoencoder, as it merely compares topological features without matching 2 the edges between R (cid:15) ( X ) and R (cid:15)

Original: ( Z ) .
Revised: ( Z ) .

Original: A cleaner approach would be to enforce similarity on the intersection of the selected edges in both complexes.
Revised: A cleaner approach would be to enforce similarity on the intersection of the selected edges in both complexes.

Original: However, this would initially include very few edges, preventing efﬁcient training and leading to very biased estimates of the topological alignments between the spaces 3 .
Revised: However, this would initially include very few edges, preventing efﬁcient training and leading to very biased estimates of the topological alignments between the spaces 3 .

Original: To overcome this, we account for the union of all selected edges in X and Z .
Revised: To overcome this, we account for the union of all selected edges in X and Z .

Original: Our topological loss term decomposes into two components, each handling the “directed” loss occurring as topological features in one of the two spaces remain ﬁxed.
Revised: Our topological loss term decomposes into two components, each handling the “directed” loss occurring as topological features in one of the two spaces remain ﬁxed.

Original: Hence, L t =
Revised: Hence, L t =

Original: L X→Z + L Z→X , where
Revised: L X→Z + L Z→X , where

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: We

Original: When initialising a random latent space Z , the persistence pairing in the latent space will select random edges, resulting in only 1 expected matched edge (independent of mini-batch size) between the two pairings.

Original: Thus, only one edge (referring to one pairwise distance between two latent codes) could be used to update the encoding of these two data points.

Original: respectively.
Revised: respectively.

Original: The key idea for both terms is to align and preserve topologically relevant distances from both spaces.
Revised: The key idea for both terms is to align and preserve topologically relevant distances from both spaces.

Original: By taking the union of all selected edges (and the corresponding distances), we obtain an informative loss term that is determined by at least | X |many distances.
Revised: By taking the union of all selected edges (and the corresponding distances), we obtain an informative loss term that is determined by at least | X | distances.

Original: This loss can be seen as a more generic version of the loss introduced by Hofer et al.
Revised: This loss can be seen as a more generic version of the loss introduced by Hofer et al.

Original: (2019b), whose formulation does not take the two directed components into account and uses a uniform distance value for all persistence tuples.
Revised: (2019b), whose formulation does not take the two directed components into account and optimises the destruction values of all persistence tuples with respect to a uniform parameter (see also Section 4 for a brief discussion).

Original: By contrast, our formulation aims to to align the distances between X and Z (which in turn will lead to an alignment of distances between X and Z ).
Revised: By contrast, our formulation aims to to align the distances between X and Z (which in turn will lead to an alignment of distances between X and Z ).

Original: If they are aligned perfectly, L X→Z =
Revised: If they are aligned perfectly, L X→Z =

Original: L Z→X = {+
+}
Revised: L Z→X = {+
+}

Original: because both pairings and their corresponding distances coincide.
Revised: because both pairings and their corresponding distances coincide.

Revised: The converse implication is not true: if L t = 0 , the persistence pairings (and their corresponding persistence diagrams) are not necessarily identical.

Original: Letting θ refer to the parameters of the encoder , we have
Revised: Letting θ refer to the parameters of the encoder , we have

Original: [Equation]
Revised: [Equation]

Original: where (cid:12)(cid:12)
Revised: where (cid:12)(cid:12)

Original: π X (cid:12)(cid:12) denotes the cardinality of a persistence pairing and A Z (cid:2) π X (cid:3)
Revised: π X (cid:12)(cid:12) denotes the cardinality of a persistence pairing and A Z (cid:2) π X (cid:3)

Original: i refers to the i th entry of the vector of paired distances.
Revised: i refers to the i th entry of the vector of paired distances.

Original: This derivation works analogously for L Z→X (with π X being replaced by π Z ).
Revised: This derivation works analogously for L Z→X (with π X being replaced by π Z ).

Original: Furthermore, any derivative of A X with respect to θ must vanish because the distances of the input samples do not depend on the encoding by deﬁnition.
Revised: Furthermore, any derivative of A X with respect to θ must vanish because the distances of the input samples do not depend on the encoding by deﬁnition.

Original: These equations presume inﬁnitesimal perturbations.
Revised: These equations presume inﬁnitesimal perturbations.

Original: In fact, the persistence diagrams change in a non-differentiable manner during the training phase but for a given update step, a diagram is robust to inﬁnitesimal changes of its entries (Cohen-Steiner et al., 2007).
Revised: In fact, the persistence diagrams change in a non-differentiable manner during the training phase but for a given update step, a diagram is robust to inﬁnitesimal changes of its entries (Cohen-Steiner et al., 2007).

Original: As a consequence, our topological loss is differentiable for each update step during training.
Revised: As a consequence, our topological loss is differentiable for each update step during training.

Original: We make our code publicly available 4 .
Revised: We make our code publicly available 4 .

Original: 3. S TABILITY
Revised: 3. S TABILITY

Original: With persistence diagrams being stable under small perturbations of the underlying space (CohenSteiner et al., 2007), we still have to analyse our topological approximation on the level of minibatches.
Revised: With persistence diagrams being stable under small perturbations of the underlying space (CohenSteiner et al., 2007), we still have to analyse our topological approximation on the level of minibatches.

Original: The following theorem guarantees that subsampled persistence diagrams are close to the persistence diagrams of the original point cloud.
Revised: The following theorem guarantees that subsampled persistence diagrams are close to the persistence diagrams of the original point cloud.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let X be a point cloud of cardinality n and X ( m ) be one subsample of X of cardinality m , i.e. X ( m ) ⊆
Revised: Let X be a point cloud of cardinality n and X ( m ) be one subsample of X of cardinality m , i.e. X ( m ) ⊆

Original: X , sampled without replacement.
Revised: X , sampled without replacement.

Original: We can bound the probability of the persistence diagrams of X ( m ) exceeding a threshold in terms of the bottleneck distance as
Revised: We can bound the probability of the persistence diagrams of X ( m ) exceeding a threshold in terms of the bottleneck distance as

Original: [Equation]
Revised: [Equation]

Original: where d H refers to the Hausdorff distance between the point cloud and its subsample.
Revised: where d H refers to the Hausdorff distance between the point cloud and its subsample.

Original: Proof.
Revised: Proof.

Original: See Section A.1 in the supplementary materials.
Revised: See Section A.1 in the supplementary materials.

Original: For m → n , we have lim m → n d H (cid:0) X, X ( m ) (cid:1) = 0 .
Revised: For m → n , we have lim m → n d H (cid:0) X, X ( m ) (cid:1) = 0 .

Original: Please refer to Section A.2 for an analysis of empirical convergence rates as well as a discussion of a worst-case bound.
Revised: Please refer to Section A.2 for an analysis of empirical convergence rates as well as a discussion of a worst-case bound.

Original: Given certain independence assumptions, the next theorem approximates the expected value of the Hausdorff distance.
Revised: Given certain independence assumptions, the next theorem approximates the expected value of the Hausdorff distance.

Original: The calculation of an exact representation is beyond the scope of this work, though.
Revised: The calculation of an exact representation is beyond the scope of this work, though.

Original: Theorem 2.
Revised: Theorem 2.

Original: Let A ∈ R n × m be the distance matrix between samples of X and X ( m ) , where the rows are sorted such that the ﬁrst m rows correspond to the columns of the m subsampled points with diagonal elements a ii = 0 .
Revised: Let A ∈ R n × m be the distance matrix between samples of X and X ( m ) , where the rows are sorted such that the ﬁrst m rows correspond to the columns of the m subsampled points with diagonal elements a ii = 0 .

Original: Assume that the entries a ij with i > m are random samples following a distance distribution F D with supp( f D ) ∈ R ≥ 0 .
Revised: Assume that the entries a ij with i > m are random samples following a distance distribution F D with supp( f D ) ∈ R ≥ 0 .

Original: The minimal distances δ i for rows with i > m follow a distribution F ∆ .
Revised: The minimal distances δ i for rows with i > m follow a distribution F ∆ .

Original: Letting Z := max 1 ≤ i ≤ n δ i with a corresponding distribution F Z , the expected Hausdorff distance between X and X ( m ) for m < n is bounded by:
Revised: Letting Z := max 1 ≤ i ≤ n δ i with a corresponding distribution F Z , the expected Hausdorff distance between X and X ( m ) for m < n is bounded by:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: See Section A.3 in the supplementary materials.
Revised: See Section A.3 in the supplementary materials.

Original: From Eq.
Revised: From Eq.

Original: 6, we obtain E [d H ( X, X m )]
Revised: 6, we obtain E [d H ( X, X m )]

Original: = 0 as m → n , so the expected value converges as the subsample size approaches the total sample size 5 .
Revised: = 0 as m → n , so the expected value converges as the subsample size approaches the total sample size 5 .

Original: We conclude that our subsampling approach results in point clouds that are suitable proxies for the large-scale topological structures of the point cloud X .
Revised: We conclude that our subsampling approach results in point clouds that are suitable proxies for the large-scale topological structures of the point cloud X .

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Computational topology and persistent homology (PH) have started gaining traction in several areas of machine learning research.
Revised: Computational topology and persistent homology (PH) have started gaining traction in several areas of machine learning research.

Original: PH is often used as as post hoc method for analysing topological characteristics of data sets.
Revised: PH is often used as as post hoc method for analysing topological characteristics of data sets.

Original: Thus, there are several methods that compare topological features of high-dimensional spaces with different embeddings to assess the ﬁdelity and quality of a speciﬁc embedding scheme (Khrulkov & Oseledets, 2018; Paul & Chalup, 2017; Rieck & Leitte, 2015; 2017; Yan et al., 2018).
Revised: Thus, there are several methods that compare topological features of high-dimensional spaces with different embeddings to assess the ﬁdelity and quality of a speciﬁc embedding scheme (Khrulkov & Oseledets, 2018; Paul & Chalup, 2017; Rieck & Leitte, 2015; 2017; Yan et al., 2018).

Original: PH can also be used to characterise the training of neural networks (Guss & Salakhutdinov, 2018; Rieck et al., 2019b), as well as their decision boundaries (Ramamurthy et al., 2019).
Revised: PH can also be used to characterise the training of neural networks (Guss & Salakhutdinov, 2018; Rieck et al., 2019b), as well as their decision boundaries (Ramamurthy et al., 2019).

Original: Our method differs from all these publications in that we are able to obtain gradient information to update a model while training.
Revised: Our method differs from all these publications in that we are able to obtain gradient information to update a model while training.

Original: Alternatively, topological features can be integrated into classiﬁers to improve classiﬁcation performance.
Revised: Alternatively, topological features can be integrated into classiﬁers to improve classiﬁcation performance.

Original: Hofer et al.
Revised: Hofer et al.

Original: (2017) propose a neural network layer that learns projections of persistence diagrams, which can subsequently be used as feature descriptors to classify structured data.
Revised: (2017) propose a neural network layer that learns projections of persistence diagrams, which can subsequently be used as feature descriptors to classify structured data.

Original: Moreover, several vectorisation strategies for persistence diagrams exist (Adams et al., 2017; Carrière et al., 2015), making it possible to use them in kernelbased classiﬁers.
Revised: Moreover, several vectorisation strategies for persistence diagrams exist (Adams et al., 2017; Carrière et al., 2015), making it possible to use them in kernelbased classiﬁers.

Original: These strategies have been subsumed (Carrière et al., 2019) in a novel architecture based on deep sets.
Revised: These strategies have been subsumed (Carrière et al., 2019) in a novel architecture based on deep sets.

Original: The commonality of these approaches is that they treat persistence diagrams as being ﬁxed; while they are capable of learning suitable parameters for classifying them, they cannot adjust input data to better approximate a certain topology.
Revised: The commonality of these approaches is that they treat persistence diagrams as being ﬁxed; while they are capable of learning suitable parameters for classifying them, they cannot adjust input data to better approximate a certain topology.

Original: Such topology-based adjustments have only recently become feasible.
Revised: Such topology-based adjustments have only recently become feasible.

Original: Poulenard et al.
Revised: Poulenard et al.

Original: (2018) demonstrated how to optimise real-valued functions based on their topology.
Revised: (2018) demonstrated how to optimise real-valued functions based on their topology.

Original: This constitutes the ﬁrst approach for aligning persistence diagrams by modifying input data; it requires the connectivity of the data to be known, and the optimised functions have to be node-based and scalar-valued.
Revised: This constitutes the ﬁrst approach for aligning persistence diagrams by modifying input data; it requires the connectivity of the data to be known, and the optimised functions have to be node-based and scalar-valued.

Original: By contrast, our method works directly on distances and sidesteps connectivity calculations via the Vietoris–Rips complex.
Revised: By contrast, our method works directly on distances and sidesteps connectivity calculations via the Vietoris–Rips complex.

Original: Chen et al.
Revised: Chen et al.

Original: (2019) use a similar optimisation technique to regularise the decision boundary of a classiﬁer.
Revised: (2019) use a similar optimisation technique to regularise the decision boundary of a classiﬁer.

Original: However, this requires discretising the space, which can be computationally expensive.
Revised: However, this requires discretising the space, which can be computationally expensive.

Original: A work by Hofer et al.
Revised: Hofer et al.

Original: (2019b) also presents a differentiable loss term.
Revised: (2019b), the closest work to ours, also presents a differentiable loss term.

Original: This is the closest work to ours; their formulation assumes a uniform distance value for all persistence tuples, though, and does not provide a “directed” loss term as we do.
Revised: They propose regularising the connectivity of the latent space by itself. Speciﬁcally, their formulation enforces a single scale on the latent space. The learned encoding is then applied to a classiﬁcation task. By contrast, our directed loss term aims to preserve the data space topology in the latent space for dimensionality reduction.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Our main task is to learn a latent space in an unsupervised manner such that topological features of the data space—measured using persistent homology approximations on every batch—are preserved as much as possible. 
Revised: Our main task is to learn a latent space in an unsupervised manner such that topological features of the data space—measured using persistent homology approximations on every batch—are preserved as much as possible. [Figure]

Original: 5. E XPERIMENTAL S ETUP
Revised: 5. E XPERIMENTAL S ETUP

Original: Data sets We use a S PHERES data set that consists of ten high-dimensional 100 -spheres living in 101 − dimensional space that are enclosed by one larger sphere that consists of the same number of points as the total of inner spheres (please refer to Section A.4 for more details).
Revised: Data sets We use a S PHERES data set that consists of ten high-dimensional 100 -spheres living in 101 − dimensional space that are enclosed by one larger sphere that consists of the same number of points as the total of inner spheres (please refer to Section A.4 for more details).

Original: We also use three image data sets (MNIST, F ASHION -MNIST, and CIFAR-10), which are particularly amenable to our topology-based analysis because real-world images are known to lie on or near low-dimensional manifolds (Lee et al., 2003; Peyré, 2009).
Revised: We also use three image data sets (MNIST, F ASHION -MNIST, and CIFAR-10), which are particularly amenable to our topology-based analysis because real-world images are known to lie on or near low-dimensional manifolds (Lee et al., 2003; Peyré, 2009).

Original: Baselines & Training We compare our approach with several dimensionality reduction techniques, including UMAP (McInnes et al., 2018), t-SNE (van der Maaten & Hinton, 2008), Isomap (Tenenbaum et al., 2000), PCA, as well as standard autoencoders (AE).
Revised: Baselines & Training We compare our approach with several dimensionality reduction techniques, including UMAP (McInnes et al., 2018), t-SNE (van der Maaten & Hinton, 2008), Isomap (Tenenbaum et al., 2000), PCA, as well as standard autoencoders (AE).

Original: We apply our proposed topological constraint to this standard autoencoder architecture (TopoAE).
Revised: We apply our proposed topological constraint to this standard autoencoder architecture (TopoAE).

Original: For comparability and interpretability,For m n , the [Figure] each method is restricted to two latent dimensions.
Revised: For comparability and interpretability, each method is restricted to two latent dimensions.

Original: We split each data set into training and testing (using the predeﬁned split if available; 90% versus 10% otherwise).
Revised: We split each data set into training and testing (using the predeﬁned split if available; 90% versus 10% otherwise).

Original: Additionally, we remove 15% of the training split as a validation data set for tuning the hyperparameters.
Revised: Additionally, we remove 15% of the training split as a validation data set for tuning the hyperparameters.

Original: We normalised our topological loss term by the batch size m in order to disentangle λ from it.
Revised: We normalised our topological loss term by the batch size m in order to disentangle λ from it.

Original: All autoencoders employ batch-norm and are optimized using ADAM (Kingma & Ba, 2014).
Revised: All autoencoders employ batch-norm and are optimized using ADAM (Kingma & Ba, 2014).

Original: Since t-SNE is not intended to be applied applying to previously unseen test samples, we evaluate this method only on the train split.
Revised: Since t-SNE is not intended to be applied applying to previously unseen test samples, we evaluate this method only on the train split.

Original: In addition, signiﬁcant computational scaling issues forced us to forgo running a hyperparameter search for Isomap on real-world data sets, so we only compare this algorithm on the synthetic data set.
Revised: In addition, signiﬁcant computational scaling issues forced us to forgo running a hyperparameter search for Isomap on real-world data sets, so we only compare this algorithm on the synthetic data set.

Original: Please refer to Section A.5 for more details on architectures and hyperparameters.
Revised: Please refer to Section A.5 for more details on architectures and hyperparameters.

Original: Evaluation We evaluate the quality of latent representations in terms of (1) low-dimensional visualisations, (2) dimensionality reduction quality metrics (evaluated between input data and latent codes), and (3) reconstruction errors (Data MSE; evaluated between input and reconstructed data), provided that invertible transformations are available 6 .
Revised: Evaluation We evaluate the quality of latent representations in terms of (1) low-dimensional visualisations, (2) dimensionality reduction quality metrics (evaluated between input data and latent codes), and (3) reconstruction errors (Data MSE; evaluated between input and reconstructed data), provided that invertible transformations are available 6 .

Original: For (2), we consider several measures (please refer to Section A.6 for more details).
Revised: For (2), we consider several measures (please refer to Section A.6 for more details).

Original: First, we calculate KL σ , the Kullback–Leibler divergence between the density estimates of the input and latent space, based on density estimates (Chazal et al., 2011; 2014b), where σ ∈ R > 0 denotes the length scale of the Gaussian kernel, which is varied to account for multiple data scales.
Revised: First, we calculate KL σ , the Kullback–Leibler divergence between the density estimates of the input and latent space, based on density estimates (Chazal et al., 2011; 2014b), where σ ∈ R > 0 denotes the length scale of the Gaussian kernel, which is varied to account for multiple data scales.

Original: We chose minimising KL 0 .
Revised: We chose minimising KL 0 .

Original: 1 as our hyperparameter search objective.
Revised: 1 as our hyperparameter search objective.

Original: Furthermore, we calculate common non-linear dimensionality reduction (NLDR) quality metrics, which use the pairwise distance matrices of the input and the latent space (as indicated by the “ (cid:96) ” in the abbreviations), namely the root mean square error ( (cid:96) -RMSE), which—despite its name—is not  related to the reconstruction error of the autoencoder but merely measures to what extent the two distributions of distances coincide, the mean relative rank error ( (cid:96) -MRRE), the continuity ( (cid:96) -Cont), and the trustworthiness ( (cid:96) -Trust) .
Revised: Furthermore, we calculate common non-linear dimensionality reduction (NLDR) quality metrics, which use the pairwise distance matrices of the input and the latent space (as indicated by the “ (cid:96) ” in the abbreviations), namely the root mean square error ( (cid:96) -RMSE), which—despite its name—is not [Figure] related to the reconstruction error of the autoencoder but merely measures to what extent the two distributions of distances coincide, the mean relative rank error ( (cid:96) -MRRE) , the continuity ( (cid:96) -Cont), and the trustworthiness ( (cid:96) -Trust) .

Original: The reported measures are computed on the testing splits (except for t-SNE where no transformation between splits is available, so we report the measures on a random subsample of the train split).
Revised: The reported measures are computed on the testing splits (except for t-SNE where no transformation between splits is available, so we report the measures on a random subsample of the train split).

Original: 5. R ESULTS
Revised: 5. R ESULTS

Original: Quantitative Results Table 1 reports the quantitative results.
Revised: Quantitative Results Table 1 reports the quantitative results.

Original: Overall, we observe that our method shines in preserving the data density over multiple length scales (as measured by KL ).
Revised: Overall, we observe that our method shines in preserving the data density over multiple length scales (as measured by KL ).

Original: Furthermore, we ﬁnd that TopoAE displays competitive continuity values ( (cid:96) -Cont) and reconstruction errors (Data
Revised: Furthermore, we ﬁnd that TopoAE displays competitive continuity values ( (cid:96) -Cont) and reconstruction errors (Data

Original: MSE).
Revised: MSE).

Original: The remaining classical measures favour the baselines (foremost the train (!) performance of t-SNE).
Revised: The remaining classical measures favour the baselines (foremost the train (!) performance of t-SNE).

Original: However, we will subsequently see that those classic measures fail at detecting the relevant[Figure] structural information, as exempliﬁed with known ground truth manifolds, such as the S PHERES data set.
Revised: However, we will subsequently see that those classic measures fail at detecting the relevant structural information, as exempliﬁed with known ground truth manifolds, such as the S PHERES data set.

Original: Visualisation of latent spaces For the S PHERES data set (Figure 2), we observe that only our method is capable of assessing the nesting relationship of the high-dimensional spheres correctly.
Revised: Visualisation of latent spaces For the S PHERES data set (Figure 2), we observe that only our method is capable of assessing the nesting relationship of the high-dimensional spheres correctly.

Original: By contrast, t-SNE “cuts open” the enclosing sphere, distributing most of its points around the remaining spheres.
Revised: By contrast, t-SNE “cuts open” the enclosing sphere, distributing most of its points around the remaining spheres.

Original: We see that the KL -divergence conﬁrms the visual assessment that only our proposed method preserves the relevant structure of this dataset.
Revised: We see that the KL -divergence conﬁrms the visual assessment that only our proposed method preserves the relevant structure of this dataset.

Original: Several classical evaluation measures, however, favour t-SNE, even though this method fails to capture the global structure and nesting relationship of the enclosing sphere manifold accounting for half of the dataset.
Revised: Several classical evaluation measures, however, favour t-SNE, even though this method fails to capture the global structure and nesting relationship of the enclosing sphere manifold accounting for half of the dataset.

Original: On F ASHION -MNIST
Revised: On F ASHION -MNIST

Original: (Figure 3a–e), we see that, as opposed to AE, which is purely driven by the reconstruction error, our method has the additional objective of preserving structure.
Revised: (Figure 3a–e), we see that, as opposed to AE, which is purely driven by the reconstruction error, our method has the additional objective of preserving structure.

Original: Here, this constraint helps the autoencoder to “organise” the latent space, resulting in a comparable pattern as in UMAP, which is also topologically motivated.
Revised: Here, this constraint helps the autoencoder to “organise” the latent space, resulting in a comparable pattern as in UMAP, which is also topologically motivated.

Original: Furthermore, we observe that t-SNE tends to fragment certain classes (dark orange, red) into multiple distinct subgroups.
Revised: Furthermore, we observe that t-SNE tends to fragment certain classes (dark orange, red) into multiple distinct subgroups.

Original: This is very likely not to reﬂect the underlying manifold structure, but constitutes an artefact frequently observed with this method.
Revised: This is very likely not to reﬂect the underlying manifold structure, but constitutes an artefact frequently observed with this method.

Original: For MNIST, the latent embeddings (Figure 3f–j) demonstrate that the non-linear competitors—mostly by pulling apart distinct classes—lose some  of the relationship information between clusters when comparing against our proposed method or
Revised: For MNIST, the latent embeddings (Figure 3f–j) demonstrate that the non-linear competitors—mostly by pulling apart distinct classes—lose some [Table] of the relationship information between clusters when comparing against our proposed method or

Original: PCA.
Revised: PCA.

Revised: Finally

Original: Finally, we observe that CIFAR-10 (Figure 3k–o), is challenging to embed in two latent dimensions in a purely unsupervised manner.
Revised: [Table] we observe that CIFAR-10 (Figure 3k–o) [Table] is challenging to embed in two latent dimensions in a purely unsupervised manner.

Original: Interestingly, our method (repeatedly) identiﬁed a linear substructure separating the latent space in two additional groups of classes.
Revised: Interestingly, our method (repeatedly) identiﬁed a linear substructure separating the latent space in two additional groups of classes [Table]

Original:  D ISCUSSION AND C ONCLUSIONS
Revised:  D ISCUSSION AND C ONCLUSIONS

Original: We presenteda topological autoencoders, a novel method for preserving topological information, measured in terms of persistent homology, of the input space when learning latent representations with deep neural networks.
Revised: We presented topological autoencoders, a novel method for preserving topological information, measured in terms of persistent homology, of the input space when learning latent representations with deep neural networks.

Original: Under weak theoretical assumptions, we showed how our persistent homology (PH) calculations can be combined with backpropagation; moreover, we proved that approximating PH on the level of mini-batches is theoretically justiﬁed.[Table]
Revised: Under weak theoretical assumptions, we showed how our persistent homology (PH) calculations can be combined with backpropagation; moreover, we proved that approximating PH on the level of mini-batches is theoretically justiﬁed.

Original: In our experiments [Table] we observed that our method is uniquely able to capture spatial relationships between nested high-dimensional spheres.
Revised: In our experiments, we observed that our method is uniquely able to capture spatial relationships between nested high-dimensional spheres.

Original: This is relevant [Table] as the ability to cope with several manifolds in the domain of manifold learning still remains a challenging task.
Revised: This is relevant, as the ability to cope with several manifolds in the domain of manifold learning still remains a challenging task.

Original: On real-world data sets, we observed that our topological loss leads to competitive performance in terms of numerous quality metrics (such as a density preservation metric), while not adversely affecting the reconstruction error.
Revised: On real-world data sets, we observed that our topological loss leads to competitive performance in terms of numerous quality metrics (such as a density preservation metric), while not adversely affecting the reconstruction error.

Original: In both synthetic and real-world data sets, we obtain interesting representations, as our method does not merely pull apart different classes, but tries to spatially arrange them meaningfully.
Revised: In both synthetic and real-world data sets, we obtain interesting representations, as our method does not merely pull apart different classes, but tries to spatially arrange them meaningfully.

Original: Thus, we do not observe mere distinct “clouds”, but rather entangled structures, which we consider to constitute a more meaningful representation of the underlying manifolds (an auxiliary analysis in
Revised: Thus, we do not observe mere distinct “clouds”, but rather entangled structures, which we consider to constitute a more meaningful representation of the underlying manifolds (an auxiliary analysis in

Original: Supplementary Section A.9 conﬁrms that our method inﬂuences topological features, measured using
Revised: Supplementary Section A.9 conﬁrms that our method inﬂuences topological features, measured using

Original: PH, in a beneﬁcial manner).
Revised: PH, in a beneﬁcial manner).

Original: Our topological loss formulation is highly generic; it only requires the existence of a distance matrix between individual samples (either globally, or on the level of batches).
Revised: Our topological loss formulation is highly generic; it only requires the existence of a distance matrix between individual samples (either globally, or on the level of batches).

Original: As a consequence, our topological loss term can be directly integrated into a variety of different architectures and is not limited to standard autoencoders.
Revised: As a consequence, our topological loss term can be directly integrated into a variety of different architectures and is not limited to standard autoencoders.

Original: For instance, we can also apply our constraint to variational setups (see Figure A.7 for a sketch).
Revised: For instance, we can also apply our constraint to variational setups (see Figure A.7 for a sketch).

Original: Employing our generic constraint to more involved architectures will be an exciting route for future work.
Revised: Employing our generic constraint to more involved architectures will be an exciting route for future work.

Original: One issue with the calculation is that, given the computational complexity of calculating
Revised: One issue with the calculation is that, given the computational complexity of calculating

Original: R (cid:15) ( · ), for higher-dimensional features we would scale progressively worse with increasing batch size.
Revised: R (cid:15) ( · ) for higher-dimensional features we would scale progressively worse with increasing batch size.

Original: However, in our low-dimensional setup, we observed that runtime tends to grow with decreasing batch-size, i.e. the mini-batch speed-up still dominates runtime (for more details concerning the effect of batch sizes, see Supplementary Section A.7).
Revised: However, in our low-dimensional setup we observed that runtime tends to grow with decreasing batch-size, i e. the mini-batch speed-up still dominates runtime (for more details concerning the effect of batch sizes, see Supplementary Section A.7).

Original: In future work, scaling to higher dimensions could be mitigated by approximating the calculation of persistent homology (Choudhary et al., 2018; Kerber &
Revised: In future work, scaling to higher dimensions could be mitigated by approximating the calculation of persistent homology (Choudhary et al., 2018; Kerber &

Original: Sharathkumar, 2013; Sheehy, 2013) or by exploiting recent advances in parallelising it (Bauer et al., 2014; Lewis & Morozov, 2015).
Revised: Sharathkumar, 2013; Sheehy, 2013) or by exploiting recent advances in parallelising it (Bauer et al., 2014; Lewis & Morozov, 2015).

Original: A A PPENDIX A. P ROOF OF T HEOREM
Revised: A A PPENDIX A. P ROOF OF T HEOREM

Original: Theorem 1.
Revised: Theorem 1.

Original: Let X be a point cloud of cardinality n and X ( m ) be one subsample of X of cardinality m , i.e. X ( m ) ⊆
Revised: Let X be a point cloud of cardinality n and X ( m ) be one subsample of X of cardinality m , i.e. X ( m ) ⊆

Original: X , sampled without replacement.
Revised: X , sampled without replacement.

Original: We can bound the probability of X ( m ) exceeding a threshold in terms of the bottleneck distance as
Revised: We can bound the probability of X ( m ) exceeding a threshold in terms of the bottleneck distance as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where d H refers to the Hausdorff distance between the point cloud and its subsample, i.e.
Revised: where d H refers to the Hausdorff distance between the point cloud and its subsample, i.e.

Original: [Equation]
Revised: [Equation]

Original: for a baseline distance dist( x, y ) such as the Euclidean distance.
Revised: for a baseline distance dist( x, y ) such as the Euclidean distance.

Original: Proof.
Revised: Proof.

Original: The stability of persistent homology calculations was proved by Chazal et al.
Revised: The stability of persistent homology calculations was proved by Chazal et al.

Original: (2014a) for ﬁnite metric spaces.
Revised: (2014a) for ﬁnite metric spaces.

Original: More precisely, given two metric spaces X and Y , we have
Revised: More precisely, given two metric spaces X and Y , we have

Original: [Equation]
Revised: [Equation]

Original: ) where d GH ( X, Y ) refers to the Gromov–Hausdorff distance (Burago et al., 2001, p. 254) of the two spaces.
Revised: ) where d GH ( X, Y ) refers to the Gromov–Hausdorff distance (Burago et al., 2001, p. 254) of the two spaces.

Original: It is deﬁned as the inﬁmum Hausdorff distance over all isometric embeddings of X and Y .
Revised: It is deﬁned as the inﬁmum Hausdorff distance over all isometric embeddings of X and Y .

Original: This distance can be employed for shape comparison (Chazal et al., 2009; Mémoli & Sapiro, 2004), but is hard to compute.
Revised: This distance can be employed for shape comparison (Chazal et al., 2009; Mémoli & Sapiro, 2004), but is hard to compute.

Original: In our case, with X = X and Y = X
Revised: In our case, with X = X and Y = X

Original: ( m ) , we consider both spaces to have the same metric (for Y , we take the canonical restriction of the metric from X to the subspace Y ).
Revised: ( m ) , we consider both spaces to have the same metric (for Y , we take the canonical restriction of the metric from X to the subspace Y ).

Original: By deﬁnition of the Gromov–Hausdorff distance, we thus have d GH ( X, Y ) ≤ d H ( X, Y ) , so Eq.leads
Revised: By deﬁnition of the Gromov–Hausdorff distance, we thus have d GH ( X, Y ) ≤ d H ( X, Y ) , so Eq.leads

Original: to
Revised: to

Original: [Equation]
Revised: [Equation]

Original: from which the original claim from Eq.
Revised: from which the original claim from Eq.

Original: 7 follows by taking probabilities on both sides.
Revised: 7 follows by taking probabilities on both sides.

Original: A. E MPIRICAL CONVERGENCE RATES OF d H (cid:0) X, X ( m ) (cid:1)
Revised: A. E MPIRICAL CONVERGENCE RATES OF d H (cid:0) X, X ( m ) (cid:1)

Original: Figure A.1 depicts the mean of the convergence rate (mean) of the Hausdorff distance for a subsample of size m of 100 points in a d -dimensional space, following a standard normal distribution.
Revised: Figure A.1 depicts the mean of the convergence rate (mean) of the Hausdorff distance for a subsample of size m of 100 points in a d -dimensional space, following a standard normal distribution.

Original: We can see that the convergence rate is roughly similar, but shown on different absolute levels that depend on the ambient dimension.
Revised: We can see that the convergence rate is roughly similar, but shown on different absolute levels that depend on the ambient dimension.

Original: While bounding the convergence rate of this expression is feasible (Chazal et al., 2015a;b), it requires more involved assumptions on the measures from which X and X ( m ) are sampled.
Revised: While bounding the convergence rate of this expression is feasible (Chazal et al., 2015a;b), it requires more involved assumptions on the measures from which X and X ( m ) are sampled.

Original: Additionally, we can give a simple bound using the diameter diam( X ) :
Revised: Additionally, we can give a simple bound using the diameter diam( X ) :

Original: = sup { dist( x, y )
Revised: = sup { dist( x, y )

Original: | x, y ∈ X } .
Revised: | x, y ∈ X } .

Original: We have d H (cid:0) X, X ( m ) (cid:1) ≤ diam( X ) because the supremum is guaranteed to be an upper bound for the Hausdorff distance.
Revised: We have d H (cid:0) X, X ( m ) (cid:1) ≤ diam( X ) because the supremum is guaranteed to be an upper bound for the Hausdorff distance.

Original: This worst-case bound does not account for the sample size (or mini-batch size) m , though (see Theorem 2 for an expression that takes m into account).
Revised: This worst-case bound does not account for the sample size (or mini-batch size) m , though (see Theorem 2 for an expression that takes m into account).

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: Prior to the proof we state two observations that arise from our special setting of dealing with ﬁnite point clouds.
Revised: Prior to the proof we state two observations that arise from our special setting of dealing with ﬁnite point clouds.

Original: Observation 1.
Revised: Observation 1.

Original: Since X ( m ) ⊆
Revised: Since X ( m ) ⊆

Original: X , we have sup x (cid:48)
Revised: X , we have sup x (cid:48)

Original: ∈ X ( m ) inf x ∈ X dist( x, x (cid:48) )
Revised: ∈ X ( m ) inf x ∈ X dist( x, x (cid:48) )

Original: = 0 .
Revised: = 0 .

Original: Hence, the Hausdorff distance simpliﬁes to:
Revised: Hence, the Hausdorff distance simpliﬁes to:

Original: [Equation]
Revised: [Equation]

Original: In other words, we only have to consider a “one-sided” expression of the distance because the distance from the subsample to the original point cloud is always zero.
Revised: In other words, we only have to consider a “one-sided” expression of the distance because the distance from the subsample to the original point cloud is always zero.

Original: Observation 2.
Revised: Observation 2.

Original: Since our point clouds of interest are ﬁnite sets, the suprema and inﬁma of the
Revised: Since our point clouds of interest are ﬁnite sets, the suprema and inﬁma of the

Original: Hausdorff distance coincide with the maxima and minima, which we will subsequently use for easier readability.
Revised: Hausdorff distance coincide with the maxima and minima, which we will subsequently use for easier readability.

Original: Hence, the computation of d H ( X, X ( m ) can be divided into three steps.
Revised: Hence, the computation of d H ( X, X ( m ) can be divided into three steps.

Original: Using the baseline distance dist( · , · ) , we compute a distance matrix A ∈ R n × m between all points in X and X ( m ) .
Revised: Using the baseline distance dist( · , · ) , we compute a distance matrix A ∈ R n × m between all points in X and X ( m ) .

Original: For each of the n points in X , we compute the minimal distance to the m samples of X ( m ) by extracting the minimal distance per row of A and gather all minimal distances in δ ∈ R n . 3.
Revised: For each of the n points in X , we compute the minimal distance to the m samples of X ( m ) by extracting the minimal distance per row of A and gather all minimal distances in δ ∈ R n . 3.

Original: Finally, we return the maximal entry of δ as d H (cid:0) X, X ( m ) (cid:1) .
Revised: Finally, we return the maximal entry of δ as d H (cid:0) X, X ( m ) (cid:1) .

Original: In the subsequent proof, we require an independence assumption of the samples.
Revised: In the subsequent proof, we require an independence assumption of the samples.

Original: Proof.
Revised: Proof.

Original: Using Observations 1 and 2 we obtain a simpliﬁed expression for the Hausdorff distance, i.e.
Revised: Using Observations 1 and 2 we obtain a simpliﬁed expression for the Hausdorff distance, i.e.

Original: [Equation]
Revised: [Equation]

Original: The minimal distances of the ﬁrst m rows of A are trivially 0 .
Revised: The minimal distances of the ﬁrst m rows of A are trivially 0 .

Original: Hence, the outer maximum is determined by the remaining n − m row minima { δ i | m < i ≤ n } with δ i = min 1 ≤ j ≤ m ( a ij ) .
Revised: Hence, the outer maximum is determined by the remaining n − m row minima { δ i | m < i ≤ n } with δ i = min 1 ≤ j ≤ m ( a ij ) .

Original: Those minima follow the distribution F ∆ ( y ) with
Revised: Those minima follow the distribution F ∆ ( y ) with

Original: [Equation]
Revised: [Equation]

Original: Next, we consider Z := max 1 ≤ i ≤ n δ i .
Revised: Next, we consider Z := max 1 ≤ i ≤ n δ i .

Original: To evaluate the density of Z , we ﬁrst need to derive its distribution F Z :
Revised: To evaluate the density of Z , we ﬁrst need to derive its distribution F Z :

Original: [Equation]
Revised: [Equation]

Original: Next, we approximate Z by Z (cid:48) by imposing i.i.d sampling of the minimal distances δ i from F ∆ .
Revised: Next, we approximate Z by Z (cid:48) by imposing i.i.d sampling of the minimal distances δ i from F ∆ .

Original: This is an approximation because in practice, the rows m + 1 to n are not stochastically independent because of the triangular inequality that holds for metrics.
Revised: This is an approximation because in practice, the rows m + 1 to n are not stochastically independent because of the triangular inequality that holds for metrics.

Original: However, assuming i.i.d.
Revised: However, assuming i.i.d.

Original: , we arrive at
Revised: , we arrive at

Original: [Equation]
Revised: [Equation]

Original: Since Z (cid:48) has positive support its expectation can then be evaluated as:
Revised: Since Z (cid:48) has positive support its expectation can then be evaluated as:

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: The independence assumption leading to Z (cid:48) results in overestimating the variance of the drawn minima δ i .
Revised: The independence assumption leading to Z (cid:48) results in overestimating the variance of the drawn minima δ i .

Original: Thus, the expected maximum of those minima, E [ Z (cid:48) ] , is overestimating the actual expectation of the maximum E [ Z ] , which is why Eq.
Revised: Thus, the expected maximum of those minima, E [ Z (cid:48) ] , is overestimating the actual expectation of the maximum E [ Z ] , which is why Eq.

Original: 17 to Eq.
Revised: 17 to Eq.

Original: 18 constitute an upper bound of
Revised: 18 constitute an upper bound of

Original: E [ Z ] , and equivalently, an upper bound of E (cid:2)
Revised: E [ Z ] , and equivalently, an upper bound of E (cid:2)

Original: d H ( X, X ( m ) ) (cid:3) .
Revised: d H ( X, X ( m ) ) (cid:3) .

Original: When increasing m , E [d H ( X, X m )] decreases monotonically since for a particular m , we draw n − m samples from the minimal distance distribution F ∆ , and their maximum determines the Hausdorff distance.
Revised: When increasing m , E [d H ( X, X m )] decreases monotonically since for a particular m , we draw n − m samples from the minimal distance distribution F ∆ , and their maximum determines the Hausdorff distance.

Original: In contrast, our preliminary upper bound on the left-hand side of Eq.
Revised: In contrast, our preliminary upper bound on the left-hand side of Eq.

Original: 18 forms a downwards-facing parabola due to the quadratic form in the exponent.
Revised: 18 forms a downwards-facing parabola due to the quadratic form in the exponent.

Original: This indicates that a tighter bound is achieved for m (cid:54) = n by using the minimal subsample size of m = 1 .
Revised: This indicates that a tighter bound is achieved for m (cid:54) = n by using the minimal subsample size of m = 1 .

Original: A. S YNTHETIC D ATASET
Revised: A. S YNTHETIC D ATASET

Original: S PHERES consists of eleven high-dimensional 100 -spheres living in 101 − dimensional space.
Revised: S PHERES consists of eleven high-dimensional 100 -spheres living in 101 − dimensional space.

Original: Ten spheres of radius r = 5 are each shifted in a random direction (by adding the same Gaussian noise vector per sphere).
Revised: Ten spheres of radius r = 5 are each shifted in a random direction (by adding the same Gaussian noise vector per sphere).

Original: To this end, we draw ten d -dimensional Gaussian vectors following N ( 0 , I ( 10 / √ d )) for d = 101 .
Revised: To this end, we draw ten d -dimensional Gaussian vectors following N ( 0 , I ( 10 / √ d )) for d = 101 .

Original: Crucially, to add interesting topological information to the data set, the ten spheres are enclosed by an additional larger sphere of radius 5 r .
Revised: Crucially, to add interesting topological information to the data set, the ten spheres are enclosed by an additional larger sphere of radius 5 r .

Original: The spheres were generated using the library scikit-tda .
Revised: The spheres were generated using the library scikit-tda .

Original: A. A RCHITECTURES AND H YPERPARAMETER T UNING
Revised: A. A RCHITECTURES AND H YPERPARAMETER T UNING

Original: Architectures for synthetic data set For the synthetically generated dataset we use a simple
Revised: Architectures for synthetic data set For the synthetically generated dataset we use a simple

Original: Multilayer perceptron architecture consisting of two hidden layer with 32 neurons each both encoder and decoder and a bottleneck of two neurons such that the sequence of hidden-layer neurons is 32 − 32 − 2 − 32 − 32 .
Revised: Multilayer perceptron architecture consisting of two hidden layer with 32 neurons each both encoder and decoder and a bottleneck of two neurons such that the sequence of hidden-layer neurons is 32 − 32 − 2 − 32 − 32 .

Original: ReLU non-linearities and batch normalization were applied between the layers excluding the output layer and the bottleneck layer.
Revised: ReLU non-linearities and batch normalization were applied between the layers excluding the output layer and the bottleneck layer.

Original: The networks were ﬁt using mean squared error loss.
Revised: The networks were ﬁt using mean squared error loss.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Architectures for real world data sets For the MNIST, F ASHION -MNIST, and CIFARdatasets, we use an architecture inspired by DeepAE (Hinton & Salakhutdinov, 2006).
Revised: Architectures for real world data sets For the MNIST, F ASHION -MNIST, and CIFARdatasets, we use an architecture inspired by DeepAE (Hinton & Salakhutdinov, 2006).

Original: This architecture is composed of 3 layers of hidden neurons of decreasing size ( 1000 − 500 − 250 ) for the encoder part, a bottleneck of two neurons, and a sequence of three layers of hidden neurons in decreasing size
Revised: This architecture is composed of 3 layers of hidden neurons of decreasing size ( 1000 − 500 − 250 ) for the encoder part, a bottleneck of two neurons, and a sequence of three layers of hidden neurons in decreasing size

Original: [Figure]
Revised: [Figure]

Original: ( 250 − 500 − 1000 ) for the decoder.
Revised: ( 250 − 500 − 1000 ) for the decoder.

Original: In contrast to the originally proposed architecture, we applied
Revised: In contrast to the originally proposed architecture, we applied

Original: ReLU non-linearities and batch normalization between the layers as we observed faster and more stable training.
Revised: ReLU non-linearities and batch normalization between the layers as we observed faster and more stable training.

Original: For the non-linearities of the ﬁnal layer, we applied the tanh non-linearity, such that the image of the activation matches the range of input images scaled between − 1 and 1 .
Revised: For the non-linearities of the ﬁnal layer, we applied the tanh non-linearity, such that the image of the activation matches the range of input images scaled between − 1 and 1 .

Original: Also here, we applied mean squared error loss.
Revised: Also here, we applied mean squared error loss.

Original: All neural network architectures were ﬁt using Adam and weight-decay of 10 − 5 .
Revised: All neural network architectures were ﬁt using Adam and weight-decay of 10 − 5 .

Original: Hyperparameter tuning For hyperparameter tuning we apply random sampling of hyperparameters using the scikit-optimize library (scikit-optimize contributers, 2018) with 20 calls per method on all datasets.
Revised: Hyperparameter tuning For hyperparameter tuning we apply random sampling of hyperparameters using the scikit-optimize library (scikit-optimize contributers, 2018) with 20 calls per method on all datasets.

Original: We select the best model parameters in terms of KL 0 .
Revised: We select the best model parameters in terms of KL 0 .

Original: 1 on the validation split and evaluate and report it on the test split.
Revised: 1 on the validation split and evaluate and report it on the test split.

Original: To estimate performance means and standard deviations, we repeated the evaluation on an independent test split 5 times by using the best parameters (as identiﬁed in the hyperparameter search on the validation split) and reﬁtting the models by resampling the train-validation split.
Revised: To estimate performance means and standard deviations, we repeated the evaluation on an independent test split 5 times by using the best parameters (as identiﬁed in the hyperparameter search on the validation split) and reﬁtting the models by resampling the train-validation split.

Original: Neural networks For the neural networks we sample the learning rate log-uniformly in the range 10 − 4 − 10 − 2 , the batch-size uniformly between 16 and 128 , and for the TopoAE method we sample the regularization strength log-uniformly in the range 10 − 1 − 3 .
Revised: Neural networks For the neural networks we sample the learning rate log-uniformly in the range 10 − 4 − 10 − 2 , the batch-size uniformly between 16 and 128 , and for the TopoAE method we sample the regularization strength log-uniformly in the range 10 − 1 − 3 .

Original: Each model was allowed to train for at most 100 epochs, whereas we applied early stopping with patience = 10 based on the validation loss.
Revised: Each model was allowed to train for at most 100 epochs, whereas we applied early stopping with patience = 10 based on the validation loss.

Original: Competitor methods For t-SNE, we sample the perplexity uniformly in the range 5 − 50 and the learning rate log-uniformly in the range 10 − 1000 .
Revised: Competitor methods For t-SNE, we sample the perplexity uniformly in the range 5 − 50 and the learning rate log-uniformly in the range 10 − 1000 .

Original: For Isomap and UMAP, the number of neighbors included in the computation was varied between 15 − 500 .
Revised: For Isomap and UMAP, the number of neighbors included in the computation was varied between 15 − 500 .

Original: For UMAP, we additionally vary the min_dist parameter uniformly between 0 and 1 .
Revised: For UMAP, we additionally vary the min_dist parameter uniformly between 0 and 1 .

Original: A. M EASURING THE QUALITY OF LATENT REPRESENTATIONS
Revised: A. M EASURING THE QUALITY OF LATENT REPRESENTATIONS

Original: Next to the reconstruction error (if available; please see the paper for a discussion on this), we use a variety of NLDR metrics to assess the quality of our method.
Revised: Next to the reconstruction error (if available; please see the paper for a discussion on this), we use a variety of NLDR metrics to assess the quality of our method.

Original: Our primary interest concerns the quality of the latent space because, among others, it can be used to visualise the data set.
Revised: Our primary interest concerns the quality of the latent space because, among others, it can be used to visualise the data set.

Original: We initially considered classical quality metrics from non-linear dimensionality reduction (NLDR) algorithms (see Bibal & Frénay (2019); Gracia et al.
Revised: We initially considered classical quality metrics from non-linear dimensionality reduction (NLDR) algorithms (see Bibal & Frénay (2019); Gracia et al.

Original: (2014); van der Maaten et al.
Revised: (2014); van der Maaten et al.

Original: (2009) for more in-depth descriptions), namely
Revised: (2009) for more in-depth descriptions), namely

Original: (1) the root mean square error ( (cid:96) -RMSE) between the distance matrix of the original space and the latent space (as mentioned in the main text, this is not related to the reconstruction error), (2) the mean relative rank error ( (cid:96) -MRRE), which measures the changes in ranks of distances in the original space and the latent space (Lee & Verleysen, 2009), (3) the trustworthiness ( (cid:96) -Trust) measure (Venna & Kaski, 2006), which checks to what extent the k nearest neighbours of a point are preserved when going from the original space to the latent space, and (4) the continuity ( (cid:96) -Cont) measure (Venna & Kaski, 2006), which is deﬁned analogously to (cid:96) -Trust, but checks to what extent neighbours are preserved when going from the latent space to the original space.
Revised: (1) the root mean square error ( (cid:96) -RMSE) between the distance matrix of the original space and the latent space (as mentioned in the main text, this is not related to the reconstruction error), (2) the mean relative rank error ( (cid:96) -MRRE), which measures the changes in ranks of distances in the original space and the latent space (Lee & Verleysen, 2009), (3) the trustworthiness ( (cid:96) -Trust) measure (Venna & Kaski, 2006), which checks to what extent the k nearest neighbours of a point are preserved when going from the original space to the latent space, and (4) the continuity ( (cid:96) -Cont) measure (Venna & Kaski, 2006), which is deﬁned analogously to (cid:96) -Trust, but checks to what extent neighbours are preserved when going from the latent space to the original space.

Original: All of these measures are deﬁned based on comparisons of the original space and the latent space; the reconstructed space is not used here.
Revised: All of these measures are deﬁned based on comparisons of the original space and the latent space; the reconstructed space is not used here.

Original: As an additional measure, we calculate the Kullback–Leibler divergence between density distributions of the input space and the latent space.
Revised: As an additional measure, we calculate the Kullback–Leibler divergence between density distributions of the input space and the latent space.

Original: Speciﬁcally, for a point cloud X with an associated distance dist , we ﬁrst use the distance to a measure density estimator (Chazal et al., 2011; 2014b), deﬁned as f σ X ( x ) :
Revised: Speciﬁcally, for a point cloud X with an associated distance dist , we ﬁrst use the distance to a measure density estimator (Chazal et al., 2011; 2014b), deﬁned as f σ X ( x ) :

Original: = (cid:80) y ∈X exp (cid:16) − σ − 1 dist( x, y ) 2 (cid:17) , where σ ∈ R > 0 represents a length scale parameter.
Revised: = (cid:80) y ∈X exp (cid:16) − σ − 1 dist( x, y ) 2 (cid:17) , where σ ∈ R > 0 represents a length scale parameter.

Revised: For dist , we use the Euclidean distance and normalise it between 0 and 1 .

Original: Given σ , we evaluate KL σ :=
Revised: Given σ , we evaluate KL σ :=

Original: KL (cid:16) f σX (cid:107) f σZ (cid:17) , which measures the similarity between the two density distributions.
Revised: KL (cid:16) f σX (cid:107) f σZ (cid:17) , which measures the similarity between the two density distributions.

Original: Ideally, we want the two distributions to be similar because this implies that density estimates in a low-dimensional representation are similar to the ones in the original space.
Revised: Ideally, we want the two distributions to be similar because this implies that density estimates in a low-dimensional representation are similar to the ones in the original space.

Original: A. A SSESSING THE BATCH SIZE
Revised: A. A SSESSING THE BATCH SIZE

Original: As we used ﬁxed architectures for the hyperparameter search, the batch-size remains the main determinant for the runtime of TopoAE.
Revised: As we used ﬁxed architectures for the hyperparameter search, the batch-size remains the main determinant for the runtime of TopoAE.

Original: In Figure A.6, we display trends (linear ﬁts) on how loss measures vary with batch size.
Revised: In Figure A.6, we display trends (linear ﬁts) on how loss measures vary with batch size.

Original: Addtionally, we draw runtime estimates.
Revised: Addtionally, we draw runtime estimates.

Original: As we applied early stopping, for better comparability, we approximated the epoch-wise runtime by dividing the execution time of a run by its number of completed epochs.
Revised: As we applied early stopping, for better comparability, we approximated the epoch-wise runtime by dividing the execution time of a run by its number of completed epochs.

Original: Interestingly, these plots suggest that the runtime grows with decreasing batch-size (even though the topological computation is more costly for larger batch-sizes!).
Revised: Interestingly, these plots suggest that the runtime grows with decreasing batch-size (even though the topological computation is more costly for larger batch-sizes!).

Original: In these experiments, sticking to 0 − dimensional topological features we conclude that the beneﬁt of using minibatches for neural network training still dominate the topological computations.
Revised: In these experiments, sticking to 0 − dimensional topological features we conclude that the beneﬁt of using mini-batches for neural network training still dominate the topological computations.

Original: The few steep peaks most likely represent outliers (the corresponding runs stopped after few epochs, which is why the effective runtime could be overestimated).
Revised: The few steep peaks most likely represent outliers (the corresponding runs stopped after few epochs, which is why the effective runtime could be overestimated).

Original: For the loss measures, we see that reconstruction loss tends to decrease with increasing batch-size, while our topological loss tends to increase with increasing batch-size (despite normalization).
Revised: For the loss measures, we see that reconstruction loss tends to decrease with increasing batch-size, while our topological loss tends to increase with increasing batch-size (despite normalization).

Original: The second observation might be due to larger batch-size enabling more complex data point arrangements and corresponding topologies.
Revised: The second observation might be due to larger batch-size enabling more complex data point arrangements and corresponding topologies.

Original: A. E XTENDING TO VARIATIONAL AUTOENCODERS
Revised: A. E XTENDING TO VARIATIONAL AUTOENCODERS

Original: In Figure A.7 we sketch a preliminary experiment, where we apply our topological constraint to variational autoencoders for the S PHERES dataset.
Revised: In Figure A.7 we sketch a preliminary experiment, where we apply our topological constraint to variational autoencoders for the S PHERES dataset.

Original: Also here, we observe that our constraint helps identifying the nesting structure of the enclosing sphere.
Revised: Also here, we observe that our constraint helps identifying the nesting structure of the enclosing sphere.

Original: A. T OPOLOGICAL DISTANCE CALCULATIONS
Revised: A. T OPOLOGICAL DISTANCE CALCULATIONS

Original: To assess the topological ﬁdelity of the resulting latent spaces, we calculate several topological distances between the test data set (full dimensionality) and the latent spaces obtained from each method (two dimensions).
Revised: To assess the topological ﬁdelity of the resulting latent spaces, we calculate several topological distances between the test data set (full dimensionality) and the latent spaces obtained from each method (two dimensions).

Original: More precisely, we calculate (i) the 1 st Wasserstein distance ( W 1 ), (ii) the 2 nd Wasserstein distance ( W 2 ), and (iii) the bottleneck distance ( W ∞ ) between the persistence diagrams obtained from the test data set of the S PHERES data and their resulting 2D latent representations.
Revised: More precisely, we calculate (i) the 1 st Wasserstein distance ( W 1 ), (ii) the 2 nd Wasserstein distance ( W 2 ), and (iii) the bottleneck distance ( W ∞ ) between the persistence diagrams obtained from the test data set of the S PHERES data and their resulting 2D latent representations.

Original: Even though our loss function is not optimising this distance, we observe in Table A.1 that
Revised: Even though our loss function is not optimising this distance, we observe in Table A.1 that

Original: [Figure]
Revised: [Figure]

Original: estimates.
Revised: estimates.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: the topological distance of our method (“TopoAE”) is always the lowest among all the methods.
Revised: the topological distance of our method (“TopoAE”) is always the lowest among all the methods.

Original: In particular, it is always smaller than the topological distance of the latent space of the autoencoder architecture; this is true for all distance measures, even though W ∞ , for example, is known to be susceptible to outliers.
Revised: In particular, it is always smaller than the topological distance of the latent space of the autoencoder architecture; this is true for all distance measures, even though W ∞ , for example, is known to be susceptible to outliers.

Original: Said experiment serves as a simple “sanity check” as it demonstrates that the changes induced by our method are beneﬁcial in that they reduce the topological distance of the latent space to the original data set.
Revised: Said experiment serves as a simple “sanity check” as it demonstrates that the changes induced by our method are beneﬁcial in that they reduce the topological distance of the latent space to the original data set.

Original: For a proper comparison of topological features between the two sets of spaces, a more involved approach would be required, though.
Revised: For a proper comparison of topological features between the two sets of spaces, a more involved approach would be required, though.

Original: [Table]
Revised: [Figure]

Revised: A. A LTERNATIVE LOSS FORMULATIONS

Revised: Our choice of loss function was motivated by the observation that only aligning the persistence diagrams between mini-batches of X and Z can lead to degenerate or ‘meaningless’ latent spaces.

Revised: As a simple example (see Figure A.8 for a visualisation), imagine three non-collinear points in the input space and the triangle they are forming.

Revised: Now assume that the latent space consists of the same triangle (in terms of its side lengths) but with permuted labels.

Revised: A loss term of the form

Revised: [Equation]

Revised: (19) only measures the distance between persistence diagrams (which would be zero in this situation) and would not be able to penalise such a conﬁguration.

Original: [Table] 
Revised: [Table] [Table]

Original: nd
Revised: nd

Original: [Table]
Revised: [Table]

Original: on
Revised: on

Original: [Table]
Revised: [Table]

Original: r
Revised: r

Original: [Table]
Revised: [Table]

Original: nd
Revised: nd

Original: [Table] [Table]
Revised: [Table] [Table]

Original: F A S H
Revised: F A S H

Original: I
Revised: I

Original: ON - M N
Revised: ON - M N

Original: I S T
Revised: I S T

Original: P C A
Revised: P C A

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: N
Revised: N

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: M
Revised: M

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: 0 .± 0 . 0 .± 0 .
Revised: 0 .± 0 . 0 .± 0 .

Original: ± 3 .
Revised: ± 3 .

Original: 0 .± 0 .
Revised: 0 .± 0 .

Original: T opo
Revised: T opo

Original: A
Revised: A

Original: E 0 .
Revised: E 0 .

Original: ± 0 . 0 .± 0 .
Revised: ± 0 . 0 .± 0 .

Original: 0 .± 0 . 0 .± 0 .
Revised: 0 .± 0 . 0 .± 0 .

Original: 0 .± 0 . 0 .± 0 .
Revised: 0 .± 0 . 0 .± 0 .

Original: 0 .± 0 . 0 .± 0 .
Revised: 0 .± 0 . 0 .± 0 .

Original: ± 0 .
Revised: ± 0 .

Original: 0 .± 0 .
Revised: 0 .± 0 .

Original: M N I S T
Revised: M N I S T

Original: P C A 0 .± 0 .
Revised: P C A 0 .± 0 .

Original: 0 .± 0 .
Revised: 0 .± 0 .

Original: 0 .± 0 . 0 .± 0 .
Revised: 0 .± 0 . 0 .± 0 .

Original: 0 .± 0 . 0 .± 0 .
Revised: 0 .± 0 . 0 .± 0 .

Original: 0 .± 0 . 0 .± 0 .
Revised: 0 .± 0 . 0 .± 0 .

Original: ± 0 .
Revised: ± 0 .

Original: 0 .± 0 .
Revised: 0 .± 0 .

Original: T S N E 0 .± 0 .
Revised: T S N E 0 .± 0 .

Original: 0 .± 0 .
Revised: 0 .± 0 .

Original: 0 .± 0 . 0 .± 0 .
Revised: 0 .± 0 . 0 .± 0 .

Original: 0 .± 0 . 0 .± 0 .
Revised: 0 .± 0 . 0 .± 0 .

Original: 0 .± 0 . 0 .± 0 .
Revised: 0 .± 0 . 0 .± 0 .

Original: ± 0 .
Revised: ± 0 .

Original: U M A P 0 .± 0 .
Revised: U M A P 0 .± 0 .

Original: 0 .± 0 .
Revised: 0 .± 0 .

Original: 0 .± 0 . 0 .± 0 .
Revised: 0 .± 0 . 0 .± 0 .

Original: 0 .± 0 . 0 .± 0 .
Revised: 0 .± 0 . 0 .± 0 .

Original: 0 .± 0 . 0 .± 0 .
Revised: 0 .± 0 . 0 .± 0 .

Original: ± 0 .
Revised: ± 0 .

Original: A E 0 .± 0 . 0 .± 0 .
Revised: A E 0 .± 0 . 0 .± 0 .

Original: 0 .± 0 .0 .± 0 . 0 .± 0 . 0 .± 0 .
Revised: 0 .± 0 .

Original: 0 .± 0 . 0 .± 0 .
Revised: 0 .± 0 . 0 .± 0 .

Original: ± 0 . 0 .± 0 .
Revised: 0 .± 0 . 0 .± 0 . 0 .± 0 .

Revised: ± 0 .

Revised: 0 .± 0 .

Original: T opo
Revised: T opo

Original: A
Revised: A

Original: E 0 .
Revised: E 0 .

Original: ± 0 . 0 .± 0 .
Revised: ± 0 . 0 .± 0 .

Original: 0 .± 0 . 0 .± 0 .
Revised: 0 .± 0 . 0 .± 0 .

Original: 0 .± 0 . 0 .± 0 .
Revised: 0 .± 0 . 0 .± 0 .

Original: 0 .± 0 . 0 .± 0 .
Revised: 0 .± 0 . 0 .± 0 .

Original: ± 0 .
Revised: ± 0 .

Original: 0 .± 0 .
Revised: 0 .± 0 .

Original: C
Revised: C

Original: I F
Revised: I F

Original: A R
Revised: A R

Original: P C A 0 .± 0 .
Revised: P C A 0 .± 0 .

Original: 0 .± 0 .
Revised: 0 .± 0 .

Original: 0 .± 0 . 0 .± 0 .
Revised: 0 .± 0 . 0 .± 0 .

Original: 0 .± 0 . 0 .± 0 .
Revised: 0 .± 0 . 0 .± 0 .

Original: 0 .± 0 . 0 .± 0 .
Revised: 0 .± 0 . 0 .± 0 .

Original: ± 0 .
Revised: ± 0 .

Original: 0 .± 0 .
Revised: 0 .± 0 .

Original: T S N E 0 .± 0 .
Revised: T S N E 0 .± 0 .

Original: 0 .± 0 .
Revised: 0 .± 0 .

Original: 0 .± 0 . 0 .± 0 .
Revised: 0 .± 0 . 0 .± 0 .

Original: 0 .± 0 . 0 .± 0 .
Revised: 0 .± 0 . 0 .± 0 .

Original: 0 .± 0 . 0 .± 0 .
Revised: 0 .± 0 . 0 .± 0 .

Original: ± 0 .
Revised: ± 0 .

Original: U M A P 0 .± 0 .
Revised: U M A P 0 .± 0 .

Original: 0 .± 0 .
Revised: 0 .± 0 .

Original: 0 .± 0 . 0 .± 0 .
Revised: 0 .± 0 . 0 .± 0 .

Original: 0 .± 0 . 0 .± 0 .
Revised: 0 .± 0 . 0 .± 0 .

Original: 0 .± 0 . 0 .± 0 .
Revised: 0 .± 0 . 0 .± 0 .

Original: ± 0 .
Revised: ± 0 .

Original: A E 0 .± 0 . 0 .± 0 .
Revised: A E 0 .± 0 . 0 .± 0 .

Original: 0 .± 0 . 0 .± 0 .
Revised: 0 .± 0 . 0 .± 0 .

Original: 0 .± 0 . 0 .± 0 .
Revised: 0 .± 0 . 0 .± 0 .

Original: 0 .± 0 . 0 .± 0 .
Revised: 0 .± 0 . 0 .± 0 .

Original: ± 0 .
Revised: ± 0 .

Original: 0 .± 0 .
Revised: 0 .± 0 .

Original: T opo
Revised: T opo

Original: A
Revised: A

Original: E 0 .
Revised: E 0 .

Original: ± 0 . 0 .± 0 .
Revised: ± 0 . 0 .± 0 .

Original: 0 .± 0 . 0 .± 0 .
Revised: 0 .± 0 . 0 .± 0 .

Original: 0 .± 0 . 0 .± 0 .
Revised: 0 .± 0 . 0 .± 0 .

Original: 0 .± 0 . 0 .± 0 .
Revised: 0 .± 0 . 0 .± 0 .

Original: [Table] [Table]
Revised: [Table] [Table]