L EARNING T OPOLOGY -P RESERVING D ATA R EPRESENTATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a method for learning topology-preserving data representations (dimensionality reduction).
The method aims to provide topological similarity between the data manifold and its latent representation via enforcing the similarity in topological features (clusters, loops, 2D voids, etc.) and their localization.
The core of the method is the minimization of the Representation Topology Divergence (RTD) between original high-dimensional data and low-dimensional representation in latent space.
RTD minimization provides closeness in topological features with strong theoretical guarantees.
We develop a scheme for RTD differentiation and apply it as a loss term for the autoencoder.
The proposed method “RTD-AE” better preserves the global structure and topology of the data manifold than state-of-theart competitors as measured by linear correlation, triplet distance ranking accuracy, and Wasserstein distance between persistence barcodes.
I NTRODUCTION
Dimensionality reduction is a useful tool for data visualization, preprocessing, and exploratory data analysis.
Clearly, immersion of high-dimensional data into 2D or 3D space is impossible without distortions which vary for popular methods.
Dimensionality reduction methods can be broadly classified into global and local methods.
Classical global methods (PCA, MDS) tend to preserve the global structure of a manifold.
However, in many practical applications, produced visualizations are non-informative since they don’t capture complex non-linear structures.
Local methods (UMAP (McInnes et al., 2018), PaCMAP (Wang et al., 2021), t-SNE (Van der Maaten &
Hinton, 2008), Laplacian Eigenmaps (Belkin & Niyogi, 2001), ISOMAP (Tenenbaum et al., 2000)) focus on preserving neighborhood data and local structure with the cost of sacrificing the global structure.
The most popular methods like t-SNE and UMAP are a good choice for inferring cluster structures but often fail to describe correctly the data manifold’s topology.
t-SNE and UMAP have hyperparameters influencing representations neighborhood size taken into account.
Different values of hyperparameters lead to significantly different visualizations and neither of them is the “canonical” one that correctly represents high-dimensional data.
We take a different perspective on dimensionality reduction.
We propose the approach based on
Topological Data Analysis (TDA) .
Topological Data Analysis (Barannikov, 1994; Zomorodian, 2001;
Chazal & Michel, 2017) is a field devoted to the numerical description of multi-scale topological properties of data distributions by analyzing point clouds sampled from them.
TDA methods naturally capture properties of data manifolds on multiple distance scales and are arguably a good trade-off between local and global approaches.
The state-of-the-art TDA approach of this kind is TopoAE (Moor et al., 2020).
However, it has several weaknesses: 1) the loss term is not continuous 2) the nullity of the loss term is only necessary but not a sufficient condition for the coincidence of topology, as measured by persistence barcodes, see more details in Appendix J.
In our paper, we suggest using the Representation Topology Divergence (RTD) (Barannikov et al., 2022) to produce topology-aware dimensionality reduction.
RTD measures the topological discrepancy between two point clouds with one-to-one correspondence between clouds and enjoys nice theoretical properties (Section 3.2).
The major obstacle to incorporate RTD into deep learning is its differentiation.
There exist approaches to the differentiation of barcodes, generic barcodes-based
[Figure]
functions with respect to deformations of filtration (Carri´ere et al., 2021) and TDA differentiation in special cases (Hofer et al., 2019; Poulenard et al., 2018).
In this paper, we make the following contributions:
We develop an approach for RTD differentiation.
Topological metrics are difficult to differentiate; the differentiability of RTD and its implementation on GPU is a valuable step forward in the TDA context which opens novel possibilities in topological optimizations; 2.
We propose a new method for topology-aware dimensionality reduction: an autoencoder enhanced with the differentiable RTD loss: “RTD-AE”.
Minimization of RTD loss between real and latent spaces forces closeness in topological features and their localization with strong theoretical guarantees; 3.
By doing computational experiments, we show that the proposed RTD-AE outperforms state-ofthe-art methods of dimensionality reduction and the vanilla autoencoder in terms of preserving the global structure and topology of a data manifold; we measure it by the linear correlation, the triplet distance ranking accuracy, Wasserstein distance between persistence barcodes and RTD.
In some cases, the proposed RTD-AE produces more faithful and visually appealing low-dimensional embeddings than state-of-the-art algorithms.
R ELATED WORK
Various dimensionality reduction methods have been proposed to obtain 2D/3D visualization of high-dimensional data (Tenenbaum et al., 2000; Belkin & Niyogi, 2001; Van der Maaten & Hinton, 2008; McInnes et al., 2018).
Natural science researchers often use dimensionality reduction methods for exploratory data analysis or even to focus further experiments (Becht et al., 2019;
Kobak &
Berens, 2019; Karlov et al., 2019; Andronov et al., 2021; Szubert et al., 2019).
The main problem with these methods is inevitable distortions (Chari et al., 2021; Batson et al., 2021;
Wang et al., 2021) and incoherent results for different hyperparameters.
These distortions can largely affect global representation structure such as inter-cluster relationships and pairwise distances.
As the interpretation of these quantities in some domain such as physics or biology can lead to incorrect conclusions, it is of high importance to preserve them as much as possible.
Visualizations from
UMAP and t-SNE are often sporadic and can’t be considered as “canonical” representation of a high-dimensional data.
An often overlooked issue is the initialization which significantly contributes to the performance of dimensionality reduction methods (Kobak & Linderman, 2021;
Wang et al., 2021).
Damrich & Hamprecht (2021) revealed that the UMAP’s true loss function is different from the purported from its theory because of negative sampling.
There is a number of works that try to tackle the distortion problem and preserve as much inter-data relationships as possible.
Authors of
PHATE (Moon et al., 2019) and ivis (Szubert et al., 2019) claim that their methods are able to capture local as well as global features, but provide no theoretical guarantees for this.
(Wagner et al., 2021) propose DIPOLE, an approach to dimensionality reduction combining techniques of metric geometry and distributed persistent homology.
From a broader view, deep representation learning is also dedicated to obtaining low-dimensional representation of data.
Autoencoder (Hinton & Salakhutdinov, 2006) and Variational Autoencoder (Kingma & Welling, 2013) are mostly used to learn representations of objects useful for solving downstream tasks or data generation.
They are not designed for data visualization and fail to preserve simultaneously local and global structure on 2D/3D spaces.
Though, their parametric nature makes them scalable and applicable to large datasets, which is why they are used in methods such as parametric UMAP (Sainburg et al., 2021) and ivis (Szubert et al., 2019) and ours.
Moor et al.
(2020) proposed TopoAE, including an additional loss for the autoencoder to preserve topological structures of the input space in latent representations.
The topological similarity is achieved by retaining similarity in the multi-scale connectivity information.
Our approach has a stronger theoretical foundation and outperforms TopoAE in computational experiments.
An approach for differentiation of persistent homology-based functions was proposed by Carri´ere et al.
Leygonie et al.
(2021) systematizes different approaches to regularisation of persistence diagrams function and defines notions of differentiability for maps to and from the space of persistence barcodes.
Luo et al.
(2021) proposed a topology-preserving dimensionality reduction method based on graph autoencoder.
Kim et al.
(2020) proposed a differentiable topological layer for general deep learning models based on persistence landscapes.
P RELIMINARIES 3. T OPOLOGICAL DATA ANALYSIS , PERSISTENT HOMOLOGY
Topology is often considered to describe the “shape of data”, that is, multi-scale properties of the datasets.
Topological information was generally recognized to be important for various data analysis problems.
In the perspective of the commonly assumed manifold hypothesis (Goodfellow et al., 2016), the datasets are concentrated on the low-dimensional manifold lying in the high-dimensional ambient space.
The standard direction is to study topological features of the underlying manifold.
The common approach is to cover the manifold via simplices.
Given the threshold α , we take sets of the points from the dataset X which are pairwise closer than α .
The family of such sets is called the Vietoris-Rips simplicial complex.
For further convenience, we introduce the fully-connected weighted graph G which vertices are the points from X and weights associated with edges are distances between the points.
Then, the Vietoris-Rips simplicial complex is defined as:
[Equation]
( G ) | m i,j ≤ α } , where m i,j is the distance between points, Vert ( G ) = { 1 , . .
. , | X |} is the vertices set of the graph G .
For each VR α ( G ) , we define the vector space C k , which consists of formal linear combinations of all k -dimensional simplices from VR α ( G ) with modulo 2 arithmetic.
The boundary operator ∂ k : C k
→ C k − 1 maps every simplex to the sum of its facets.
One can show that ∂ k ◦ ∂ k − 1 = 0 and the chain complex can be created:
[Equation]
The quotient vector space H k = ker ( ∂ k ) /im ( ∂ k +1 ) is called the k -th homology group, elements of
H k are called homology classes.
The dimension β k = dim ( H k ) is called the k -th Betti number and it approximates the number of basic topological features of the manifold represented by the point cloud X .
The immediate problem here is the selection of appropriate α which is not known beforehand.
The standard solution is to analyze all α > 0 .
Obviously, if α 1 ≤ α 2 ≤ . . .
≤ α m , then VR α 1 ( G ) ⊆
VR α 2 ( G ) ⊆ . . .
⊆
VR α m ( G ) ; the nested sequence is called the filtration.
Moreover, for each homological class one can track precisely the moments of “birth” and “death”, that is, α 1 and αwhen the particular homological class appears for the first time and vanishes.
So, α 2 − α 1 describes the “lifespan” or persistence of the homology class.
The sequence of the corresponding intervals ( α 1 , α 2 ) for the set of basic homology classes from H k is called the persistence barcode ; the whole theory is dubbed the persistent homology (Chazal & Michel, 2017; Barannikov, 1994; Zomorodian, 2001).
3. R EPRESENTATION T OPOLOGY D IVERGENCE (RTD)
The classic persistent homology is dedicated to the analysis of a single point cloud X .
Recently,
Representation Topology Divergence (RTD) (Barannikov et al., 2022) was proposed to measure the dissimilarity in the multi-scale topology between two point clouds X, ˜ X of equal size N with a one-toone correspondence between clouds.
Let G w , G ˜ w be graphs with weights on edges equal to pairwise distances of X, ˜ X .
To provide the comparison, the auxiliary graph ˆ G w, ˜ w with doubled set of vertices and edge weights matrix m ( w, ˜ w ) , see details in Appendix B, is created.
The persistence barcode of the graph ˆ G w, ˜ w is called the R-Cross-Barcode and it tracks the differences in the multi-scale topology of the two point clouds by comparing their α -neighborhood graphs for all α .
Here we give a simple example of an R-Cross-Barcode.
Suppose we have two point clouds A and B , of seven points each, with distances between points as shown in the top row of Figure 2.
Consider the R-Cross-Barcode 1 (A, B), it consists of 4 intervals (the bottom row of the figure).
The 4 intervals describe the topological discrepancies between connected components of α -neighborhood graphs of
A and B .
An interval is opened, i.e. a topological discrepancy appears, at threshold
α = ˜ w Buv when in the union of α -neighborhood graph of A and B , two vertex sets C 1 and C 2 disjoint at smaller thresholds, are joined into one connected component by the edge ( uv ) from B .
This interval is closed at threshold α = w Au ′ v ′ when the two vertex sets C 1 and C 2 are joined into one connected component in the α -neighborhood graph of A .
For example, a discrepancy appears at the threshold α = 0 .
53 when the vertex sets { 4 } and { 3 , 6 , 7 } are joined into one connected component in the union of neighborhood graphs of A and B by the edge (4 , 7) .
We identify the “death” of this R-Cross-Barcode feature at α = 0 .
57 , when these two sets are joined into one connected component in the neighborhood graph of cloud A (via the edge (4 , 7) in Figure 2 becoming grey).
By definition, RTD k ( X, ˜ X ) is the sum of intervals’ lengths in the R-Cross-Barcode k ( X, ˜ X ) and measures its closeness to an empty set.
Proposition 1 (Barannikov et al.
If RTD k ( X, ˜ X ) =
RTD k
( ˜ X, X )
=
for all k ≥ 1 , then the barcodes of the weighted graphs G w and G ˜ w are the same in any degree.
Moreover, in this case the topological features are located in the same places: the inclusions VR α ( G w ) ⊆
VR α ( G min( w, ˜ w ) ) , VR α ( G ˜ w ) ⊆
VR α ( G min( w, ˜ w ) ) induce homology isomorphisms for any threshold α .
The Proposition 1 is a strong basis for topology comparison and optimization.
Given a fixed data representation X , how to find ˜ X lying in a different space, and having a topology similar to X , in particular, similar persistence barcodes?
Proposition 1 states that it is sufficient to minimize (cid:80) i ≥ 1 (cid:16)
RTD i ( X, ˜ X ) + RTD
i ( ˜ X, X ) (cid:17) .
In most of our experiments we minimized RTD 1 ( X, ˜ X ) +
RTD 1 ( ˜ X, X ) .
RTD 1 can be calculated faster than RTD 2+ , also RTD 2+ are often close to zero.
To simplify notation, we denote RTD ( X, ˜ X ) :
= 1 / 2 ( RTD 1 ( X, ˜ X ) + RTD 1 ( ˜ X, X )) .
Comparison with TopoAE loss .
TopoAE (Moor et al., 2020) is the state-of-the-art algorithm for topology-preserving dimensionality reduction.
The TopoAE topological loss is based on comparison of minimum spanning trees in X and ˜ X spaces.
However, it has several weak spots.
First, when the TopoAE loss is zero there is no guarantee that persistence barcodes of X and ˜ X coincide.
Second, the TopoAE loss can be discontinuous in rather standard situations, see Appendix J.
At the same time, RTD loss is continuous, and its nullity guarantees the coincidence of persistence barcodes of X and ˜ X .
The continuity of the RTD loss follows from the stability of the R-Cross-Barcode k ( X, ˜ X ) which is proven in Appendix K.
M ETHOD 4. D IFFERENTIATION OF RTD
We propose to use RTD as a loss in neural networks.
Here we describe our approach to RTD differentiation.
Denote by Σ k the set of all k − simplices in the Vietoris-Rips complex of the graph ˆ G w, ˜ w , and by T k the set of all intervals in the
R-Cross-Barcode k ( X, ˜ X ) .
Fix (an arbitrary) strict order on T k .
There exists a function f k : ∪ ( b i ,d i )
∈T k { b i , d i } → Σ k that maps b i (or d i ) to a simplex σ whose appearance leads to “birth” (or “death”) of the corresponding homological class.
Let
[Equation]
denote the function of m ij equal to the filtration value at which the simplex σ joins the filtration.
Since ∂ RTD
[Equation]
= 1 , we obtain the following equation for the subgradient
[Equation]
Here, for any σ no more than one term has non-zero indicator.
Then
[Equation]
The only thing that is left is to obtain subgradients of RTD ( X, ˜ X ) by points from X and ˜ X .
Consider (an arbitrary) element m i,j of matrix m .
There are 4 possible scenarios:
i, j ≤ N , in other words m i,j is from the upper-left quadrant of m .
Its length is constant and thus ∀ l : ∂m i,j ∂X l = ∂m i,j ∂ ˜
X l = 0 .
i ≤ N < j , in other words m i,j is from the upper-right quadrant of m .
Its length is computed as Euclidean distance and thus ∂m i,j ∂X i =
( X i − X j − N ) (similar for X N − j ).
j ≤ N <
i , similar to the previous case.
N <
i, j , in other words m i,j is from the bottom-right quadrant of m .
Here we have subgradients like
[Equation]
Similar for X j − N , ˜ X
i − N and
˜ X j − N .
Subgradients ∂ RTD ( X, ˜ X ) ∂X i and ∂ RTD ( X, ˜ X )
∂
˜ X i can be derived from the beforementioned using the chain rule and the formula of full (sub)gradient.
Now we are able to minimize RTD ( X, ˜ X ) by methods of (sub)gradient optimization.
We discuss some possible tricks for improving RTD differentiation in
Appendix I.
4. RTD A UTOENCODER
Given the data X
= { x i } ni
=1 , x i ∈ R d , in high-dimensional space, our goal is to find the representation in low-dimensional space Z
= { z i } , z i ∈ R p .
For the visualization purposes, p = 2 , 3 .
Our idea is to find a representation Z which preserves persistence barcodes , that is, multi-scale topological properties of the point clouds, as much as possible.
The straightforward approach is to solve min Z RTD ( X, Z ) , where the optimization is performed over n vectors z i ∈ R p , in the flavor similar to UMAP and t-SNE.
This approach is workable albeit very time-consuming and could be applied only to small datasets, see Appendix F. A practical solution is to learn representations via the encoder network E ( w, x ) :
X → Z , see Figure 3.
Algorithm .
Initially, we train the autoencoder for E 1 epochs with the reconstruction loss 12 || X − X rec
|| 2 only.
Then, we train for E 2 epochs with the loss 12 || X − X
rec
||
+ RTD ( X, Z ) .
Both losses are calculated on mini-batches.
The two-step procedure speedups training since calculating
RTD ( X, Z ) for the untrained network takes much time.
E XPERIMENTS
In computational experiments, we perform dimensionality reduction to high-dimensional and 2D/3D space for ease of visualization.
We compare original data with latent representations by (1) linear correlation of pairwise distances, (2) Wasserstein distance (W.D.) between H 0 persistence barcodes (Chazal & Michel, 2017), (3) triplet distance ranking accuracy (Wang et al., 2021)
(4) RTD.
All of the quality measures are tailored to evaluate how the manifold’s global structure and topology are
[Table] [Table]
preserved.
We note that RTD, as a quality measure
[Table]
provides a more precise comparison of topology than the W
[Table]
D. between H 0 persistence barcodes.
First, RTD takes into account the localization of topological features, while W.D. does not.
Second, W.D. is invariant to permutations of points
[Table]
but we are interested in comparison between original data and latent representation where natural one-to-one correspondence holds.
We compare the proposed RTD-AE with t-SNE (Van der Maaten & Hinton, 2008), UMAP (McInnes et al., 2018), TopoAE (Moor et al., 2020), vanilla autoencoder (AE), PHATE (Moon et al., 2019),
Ivis (Szubert & Drozdov, 2019), PacMAP (Wang et al., 2021).
The complete description of all the used datasets can be found in Appendix L. See hyperparameters in Appendix H.
5. S YNTHETIC DATASETS
We start with the synthetic dataset “Spheres”: eleven 100D spheres in the 101D space, any two of those do not intersect and one of the spheres contains all other inside.
For the visualization, we perform dimensionality reduction to 3D space.
Figure 4 shows the results: RTD-AE is the best one preserving the nestedness for the “Spheres” dataset.
Also, RTD-AE outperforms other methods by quality measures, see Table 1.
We were unable to run MDS on “Spheres” dataset because it was too large for that method.
See more results in Appendix M.
5. R EAL WORLD DATASETS
We performed experiments with a number of real-world datasets: MNIST (LeCun et al., 1998),
F-MNIST (Xiao et al., 2017), COIL-20 (Nene et al., 1996), scRNA mice (Yuan et al., 2017), scRNA melanoma (Tirosh et al., 2016) with latent dimension of 16 and 2, see Tables 2, 5.
The choice of scRNA datasets was motivated by the increased importance of dimensionality reduction methods in natural sciences, as was previously mentioned.
RTD-AE is consistently better than competitors; moreover, the gap in metrics for the latent dimension 16 is larger than such for the latent dimension 2 (see Appendix D).
For the latent dimension 2, RTD-AE is the first or the second one among the methods by the quality measures (see Table 5, Figure 7 in Appendix D).
We conclude that the proposed RTD-AE does a good job in preserving global structure of data manifolds.
For the “Mammoth” (Coenen & Pearce, 2019b) dataset (Figure 1) we did dimensionality reduction 3D → 2D. Besides good quality measures, RTD-AE produced an appealing 2D visualization: both large-scale (shape) and low-scale (chest bones, toes, tusks) features are preserved.
5. A NALYSIS OF DISTORTIONS
Next, to study distortions produced by various dimensionality reduction methods we learn transformation from 2D to 2D space, see Figure 5.
Here, we observe that RTD-AE in general recovers the global structure for all of the datasets.
RTD-AE typically does not suffer from the squeezing (or bottleneck) issue, unlike AE, which is noticeable in “Random”, “3 Clusters” and “Circle”.
Whereas t-SNE and UMAP struggle to preserve cluster densities and intercluster distances, RTD-AE manages to do that in every case.
It does not cluster random points together, like t-SNE.
Finally, the overall shape of representations produced by RTD-AE is consistent, it does not tear apart close points, which is something UMAP does in some cases, as shown in the “Circle” dataset.
The metrics, presented in the Table 6 in Appendix E, also confirm the statements above.
RTD-AE has typically higher pairwise distances linear correlation and triplet accuracy, which accounts for good multi-scale properties, while having a lower Wasserstein distance between persistence barcodes.
5. L IMITATIONS AND COMPUTATIONAL COMPLEXITY
The main source of complexity is RTD computation.
For the batch size b , object dimensionality d and latent dimensionality k , the complexity is O ( b 2 ( d + k )) operations since all the pairwise distances should be calculated.
The R-Cross-Barcode computation is at worst cubic in the number of simplices involved.
However, the computation is often quite fast for batch sizes ≤ 256 since the boundary matrix is typically sparse for real datasets.
The selection of simplices whose addition leads to “birth” or “death” of the corresponding homological class doesn’t take extra time.
For RTD calculation and differentiation, we used GPU-optimized software.
As calculation relies heavily on the batch size, the training time of RTD-AE ranges from 1.5x the time of the basic autoencoder at batch size 8 to 4-6x the time in case of batch 512.
For COIL-20, the it took ∼ 10 minutes to train a basic AE and ∼minutes for RTD-AE.
Overall, the computation of a R-Cross-Barcode takes a similar time as in the previous step even on datasets of big dimensionality.
5. D ISCUSSION
Experimental results show that RTD-AE better preserves the data manifold global structure than its competitors.
The most interesting comparison is with TopoAE, the state-of-the-art, which uses an alternative topology-preserving loss.
The measures of interest for topology comparison are the
Wasserstein distances between persistence barcodes.
Tables 2, 6, 5 show that RTD-AE is better than
TopoAE.
RTD minimization has a stronger theoretical foundation than the loss from TopoAE (see
Section 3.2).
C ONCLUSIONS
In this paper, we have proposed an approach for topology-preserving representation learning (dimensionality reduction).
The topological similarity between data points in original and latent spaces is achieved by minimizing the Representation Topology Divergence (RTD) between original data and latent representations.
Our approach is theoretically sound: RTD=0 means that persistence barcodes of any degree coincide and the topological features are located in the same places.
We proposed how to make RTD differentiable and implemented it as an additional loss to the autoencoder, constructing
RTD-autoencoder (RTD-AE).
Computational experiments show that the proposed RTD-AE better preserves the global structure of the data manifold (as measured by linear correlation, triplet distance ranking accuracy, Wasserstein distance between persistence barcodes) than popular methods t-SNE and UMAP.
Also, we achieve higher topological similarity than the alternative TopoAE method.
Of course, the application of RTD loss is not limited to autoencoders and we expect more deep learning applications involving one-to-one correspondence between points.
The main limitation is that calculation of persistence barcodes and RTD, in particular, is computationally demanding.
We see here another opportunity for further research.
R EPRODUCIBILITY S TATEMENT
To provide reproducibility, we release the source code of the proposed RTD-AE, see Supplementary, for hyperparameters see Appendix H. For other methods, we used either official implementations or implementations from scikit-learn with default hyperparameters.
We used public datasets (see
Section 5, Appendix L).
We generated several synthetic datasets, the code for generation is available in Supplementary.
A S IMPLICIAL COMPLEXES AND FILTRATIONS
Here we briefly recall basic topological objects mentioned in our paper.
Suppose we have a full graph
X = { x 0 , x 1 , . . .
x n } a set of points in some metric space ( R, d ) .
Definition A.1 Any set σ ⊆
X is (combinatorial) simplex .
Its vertices are all points that belong to K .
Its dimensionality is number equal to | σ | − 1 .
Its faces are all proper subsets of σ .
Definition A.2 Simplicial complex C is a set of simplices such that for every simplex σ ∈ C it contains all faces of σ and for every two simplices σ 1 , σ 2 ∈ C their intersection is face of both of them.
Simplicial complexes can be seen as higher-dimensional generalization of graphs.
There are many ways to build a simplical complex from a set of points, but only two important for this work:
Vietoris-Rips and Chech complexes.
Definition A.3 Given a threshold α the Vietoris-Rips (simplicial) complex at threshold α (denotes as
VR α (X) ) is defined as set set of all simplices σ such that ∀ x i , x j ∈ σ holds d ( x i , x j ) ≤ α .
Definition A.4 Given a threshold α the ˇCech (simplicial) complex at threshold α (denotes as
Cech α (X) ) is defined as set set of all simplices σ such that all closed balls of radius α and with centers in vertices of σ have a non-empty intersection.
Alhough ˇCech complexes are rarely used in applications of Topological Data Analysis, they are important due to the fact that their fundamental topological properties are equal to those of the manifold ‘behind’ X (so-called Nerve theorem , see (Chazal & Michel, 2017) for proper explanation).
The Vietoris-Rips complexes ‘approximate’ ˇ Cech complexes :
[Equation]
Note that the definition of the Vietoris-Rips complex doesn’t require (even indirectly) function d ( . ) to be metric - it should only be symmetric and non-negative.
And so we can define the Vietoris-Rips complex of a weighted graph G =
( V, E ) .
To do so we modify Definition A.3 by replacing X with
V and taking d ( v i , v j ) as the weight of the edge between v i and v j for i ̸
= j and d ( v i , v i ) = 0 , ∀ i .
In the scope of this work we consider only Vietoris-Rips complexes of graphs.
Definition A.5 A filtration of a simplicial complex C is a nested family of subcomplexes ( C t ) t ∈ T ,
where T ⊆ R , such that for any t 1 , t 2 ∈ T , if t 1 ≤ t 2 then C t 1 ⊆
C t 2 , and C =
(cid:83) t ∈ T C t .
The set T may be either finite or infinite.
Vietoris-Rips filtration can ‘reflect’ topology of data set at every scale.
Usually data sets are finite so there is finite number of thresholds that give different Vietoris-Rips complexes and thus finite filtration is enough for it.
B F ORMAL DEFINITION OF RTD
The classic persistent homology is dedicated to the analysis of a single point cloud X .
Recently, Representation Topology Divergence (RTD) (Barannikov et al., 2022) was proposed to measure the dissimilarity in the multi-scale topology between two point clouds
X, ˜ X of equal size N with a one-to-one correspondence between clouds.
Let VR α ( G w ) , VR α ( G ˜ w ) be two Vietoris-Rips simplicial complexes, where w, ˜ w - are the distance matrices of X, ˜ X .
The idea behind RTD is to compare VR α ( G w ) with VR α
( G min ( w, ˜ w )) , where G min ( w, ˜ w ) is the graph having weights min ( w, ˜ w ) on its edges.
By definition, VR α ( G w ) ⊆
VR α ( G min ( w, ˜ w ) ) , VR α ( G ˜ w ) ⊆
VR α ( G min ( w, ˜ w ) ) .
To compare VR α ( G w ) with VR α
( G min ( w, ˜ w )) , the auxiliary graph is constructed with doubled set of vertices ˆ G w, ˜ w (Figure 6) and weights
[Table]
on edges given in the simplest case by:
[Equation]
where w + is the w matrix with lower-triangular part replaced by + ∞ , see ((Barannikov et al., 2022), section 2.2) for the general form of the matrix m .
The persistence barcode of the weighted graph
VR ( ˆ G w, ˜ w ) is called the R-Cross-Barcode (for Representations’ Cross-Barcode ).
Note that for every two nodes in the graph ˆ G w, ˜ w there exists a path with edges having zero weights.
Thus, the H 0 barcode in the R-Cross-Barcode is always empty.
Intuitively, the k − th barcode of VR α
( ˆ G w, ˜ w ) records the k -dimensional topological features that are born in VR
α
( G min( w, ˜ w ))
but are not yet born near the same place in VR α
( G w ) , and the ( k − 1) − dimensional topological features that are dead in VR α
( G min( w, ˜ w ) ) but are not yet dead at the same place in VR α ( G w ) .
The R-Cross-Barcode k ( X, ˜ X ) records the differences in the multi-scale topology of the two point clouds.
The topological features with longer lifespans indicate in general the essential features.
Basic properties of R-Cross-Barcode k ( X, ˜ X ) (Barannikov et al.
• if X = ˜
X , then for all k R-Cross-Barcode k ( X, ˜ X ) = ∅ ; • if all distances within ˜ X are zero i
e. all objects are represented by the same point in ˜ X , then for all k ≥ 0 : R-Cross-Barcode k +1 ( X, ˜ X ) =
Barcode k ( X ) the standard barcode of the point cloud X ; • for any value of threshold α , the following sequence of natural linear maps of homology groups
[Equation]
is exact, i
e. for any j the kernel of the map r j is the image of the map r j +1 .
C L EARNING REPRESENTATIONS IN HIGHER DIMENSIONS
The following table shows results of the experiment with latent dimensions 16, 32, 64 and 128 for the F-MNIST dataset.
RTD-AE and RTD-AE-MM (uses RTD modification, see Appendix G) are consistently better than the competitors.
D R EAL WORLD DATASETS , 2D LATENT SPACE
Table 5 and Figure 7 present the results.
[Table]
E S YNTHETIC DATASETS , 2D LATENT SPACE
Table 6 shows the results.
F RTD MINIMIZATION WITHOUT THE AUTOENCODER
Given the set X
= { x i } ni
=1 of n objects in high-dimensional space x i ∈ R d , our goal is to find their representations in low-dimensional space Z = { z i } , z i ∈ R k .
It is possible to solve
[Equation]
directly w.r.t n vectors z i ∈ R k , in the flavor similar to UMAP and t-SNE.
Figures 8, 9 show the results of two experiments with 3D → 2D dimensionality reduction.
We conclude that dimensionality reduction via RTD optimization better preserves data topology: meridians are kept connected (Figure 8) and the nestedness is retained (Figure 9).
The optimization took ∼ 1 hour.
For the experiment with nested spheres, the RTD optimization was warmstarted with the MDS solution.
[Table]
[Figure]
G A LTERNATIVE RTD VARIANT
RTD relies on the auxiliary graph with doubled set of vertices ˆ G w, ˜ w and weights on edges: m = (cid:18)( w + ) ⊺ w + min( w, ˜ w ) (cid:19) .
[Figure]
[Table]
An alternative variant of RTD is possible with the following matrix of weights:
[Equation]
Both of them share similar properties and guarantee that RTD ( X, Z )
=
when all the pairwise distances in point clouds X and Z are the same.
Also in both cases if RTD k ( X, Z ) =
RTD k
( Z, X ) = 0 for k ≥ 1 then the persistence diagrams of X and Z coincide.
The minimization of the sum of both variants of RTD leads to richer gradient information.
We used this loss denoted in the experiment with the “Mammoth” dataset and in the F-MNIST projected to dimensions 16-128 experiment where it was denoted RTD-AE-MM.
H H YPERPARAMETERS
In the experiments with projecting to 3D-space we trained model for 100 epochs using Adam optimizer.
We initially trained autoencoder for 10 epochs with only the reconstruction loss and learning rate 1e-4, then continued with RTD.
Epochs 11-30 were trained with learning rate 1e-2,
50 with and
4 was used.
was 80.
For 2D and high-dimensional projections, we used fully-connected autoencoders with hyperparameters specified in the Table 7.
The autoencoder was initially trained only with reconstruction loss for some number of epochs, and then the RTD loss kicked in.
The learning rate stayed the same for an entire duration of training.
For experiments we used NVIDIA TITAN RTX.
[Table]
I RTD OPTIMIZATION SPEED - UPS
For all computations of RTD-barcodes in this work we used modified version of Ripser++ software (Zhang et al., 2020).
Modification that we made was intended at decreasing computational time via exploration of the structure of graph ˆ G w, ˜ w (see Section 3.2).
The idea behind it lies in reducing size of filtered complex by excluding from it simplices that do not affect the persistence homology.
Here we consider only simplices of dimension at least 1.
All simplices spawned by any from first half of the vertices of graph ˆ G w, ˜ w (those are vertices corresponding to upper-left quadrant of graph‘s adjacency matrix m ) are exactly those excluded simplices.
All of them have diameters equal to zero and if any of them spawn a topological feature, it is immediately killed by another one of such simplices.
As before, let N be number of vertices in point clouds.
Then ˆ G w, ˜ w has 2 N vertices and our modification eliminates (cid:0) Nd (cid:1) out of (cid:0) 2 Nd (cid:1) simplices of dimension d − 1 .
In particular, from boundary matrix used for computation persistence pairs of dimension 1 this eliminates around 1 / 8 of rows and 1 / 4 of columns (around 1 / 3 cells in total).
On average it takes ≈ 45% less time comparing to standard Ripser++ to compute persistence intervals of dimension 1.
Next we will describe some techniques that can improve convergence when RTD is to be minimized without an autoencoder (F).
Usually we perform (sub)gradient descent to minimize RTD ( X, ˜ X ) between “movable” cloud X and given constant ˜ X .
Gradient smoothing.
Subgradients computed at each step of this procedure associate each homological class with at most 4 points from X , while topological structures often include much more.
Moreover, adjustments w.r.t.
them may be inconsistent for nearby points.
To overcome this we “smooth” gradients by passing to each point averaged gradients of all its neighbours.
Let ∇ ( k ) i be the gradient value for X i at step k and U ( X ( k ) i ) be some neighbourhood of
X ( k ) i .
Now the formula of each step of the gradient descent is
X
[Equation]
k )
i
[Equation]

Here β ∈ [0; 1] is some parameter.
Minimum bypassing.
Suppose we want to shorten an edge m i + N,j + N from bottom-right quadrant of matrix m (i.e. ∂ RTD ( X, ˜ X ) ∂m i + N,j + N < 0 ).
It may occur that w i,j > ˜ w i,j , so
[Equation]
and gradient descent will stuck here (since ˜ X is constant).
Thus there may appear a certain threshold below which RTD ( X, ˜ X ) can’t be minimized using our procedure.
But it can be further minimized if we move points X i and X j close enough to each other so w i,j < ˜ w i,j .
To do it, if ∂ RTD ( X, ˜ X ) ∂m i + N,j + N < 0 , we compute ∂m i,j ∂X i without indicator, i.e. as 2( X i − X j ) .
This will assure w i,j is decreasing and at certain point will became lower than ˜ w i,j .
If ∂ RTD ( X, ˜ X ) ∂m i + N,j + N ≥
we don’t change anything, because discussing effect appears only if we minimize minimum of a function and a constant.
J C OMPARISON WITH T OPO AE LOSS
The following simple example on Figure 10 shows that the TopoAE loss can be discontinuous in a rather standard situation.
The TopoAE loss (Moor et al., 2020) is constructed by calculating first the two minimal spanning trees Γ , ˜ Γ for each of the graphs G w , G ˜ w , whose weights are the distances within two point clouds X and ˜ X .
Then the TopoAE loss is the sum of two terms L TopoAE =
l + ˜ l .
One term is the sum over the set of edges of Γ : l = 12 (cid:80) ij ∈ Edges (Γ) ( w ij − ˜ w ij ) 2 , and the other is the analogous sum over the edges of ˜Γ : ˜ l = 12 (cid:80) ij ∈ Edges (˜Γ) ( w ij − ˜ w ij ) 2 .
Under a small perturbation of points, the minimal spanning tree Γ may change, e.g. with a change of pair of the closest points from two clusters.
But then the corresponding weights ˜ w change in general discontinuosly.
The point cloud ˜ X on Figure 10 consists of two clusters { 1 , 2 , 3 } and { 4 } .
The point cloud X consists of two clusters { 1 , 2 } and { 3 , 4 } .
We set the distances within each cluster to be of order 10 − 1 and the distance between the clusters equal to 10 3 ± 10 − 1 .
When the point 3 moves in X slightly as indicated, then the minimal spanning tree Γ , coloured by yellow, changes and the term ( w 14 − ˜ w 14 ) 2 ∼ 10 −in l is replaced by ( w 23 − ˜ w 23 ) 2 ∼ 10 6 .
Proposition 2. The RTD loss is continuous.
The RTD k ( X, ˜ X ) depends continuously on ( X, ˜ X ) .
The proof follows from the stability of the barcode of the filtered complex VR α
( ˆ G w, ˜ w ) with respect to the bottleneck distance under perturbation of the edge weights see Appendix K.
K S TABILITY OF R-Cross-Barcode AND RTD
Proposition 3. For any perturbations X ′ of a point cloud X and ˜ X ′ of a point cloud ˜ X , d B ( R-Cross-Barcode k ( X, ˜ X ) , R-Cross-Barcode k ( X ′ , ˜ X ′ ))
[Equation]
2) where d B denotes the bottleneck distance.
Proof.
By construction, the R-Cross-Barcode k ( X, ˜ X ) is the k − th persistence barcode of the weighted graph ˆ G w, ˜ w with the weights w ij =
∥ X i − X j ∥ and min( w ij , ˜ w ij ) , where ˜ w ij = ∥
˜ X i − ˜ X j ∥ .
If max i ∥ X ′ i − X i ∥ =
ε , then | w ′ ij − w ij | ≤ 2 ε for w ′ ij =
∥ X ′ i − X ′ j ∥ .
Similarly, | ˜ w ′ ij − ˜ w ij | ≤ 2˜ ε , where ˜ ε = max j ∥
˜ X ′ j − ˜
X j ∥ .
It follows that | min( w ′ ij , ˜ w ′ ij ) − min( w ij , ˜ w ij ) | ≤ 2 max( ε, ˜ ε ) .
Hence the filtration of each simplex in V R α ( ˆ G w, ˜ w ) changes at most by 2 max( ε, ˜ ε ) under the perturbation.
Next, it follows from e.g. the description of metamorphoses of canonical forms that the birth or the death of each segment in the k − th barcode of ˆ G w, ˜ w changes under the perturbation at most by 2 max( ε, ˜ ε ) .
L D ATASETS
The exact size, nature and dimension of the datasets are presented in Table 8.
The errors for the synthetic data is not reported as it is zero due to the small size of datasets.
L. S YNTHETIC DATA
The “Random” dataset consists of 500 points randomly distributed on a 2-dimensional unit square.
The choice for this dataset was inspired by Coenen & Pearce (2019a) and the ability of UMAP to find clusters in noise.
The “Circle” dataset is represented by 100 points randomly distributed on a 2D circle.
This dataset has a simple non-trivial topology.
The “2 Clusters” dataset consists of 200 points, half of which goes to a dense Gaussian cluster, and the other half goes to sparse Gaussian cluster with the same mean.
It is used to test the methods abilities to preserve cluster density.
The “3 Clusters” dataset consists of 3 Gaussian clusters each having 100 points.
Two clusters are located much closer to each other than the remaining one.
We propose it to test the preservation of the global structure, i.e. the distances between clusters. [Table]
L. R EAL - WORLD DATASETS
Both MNIST and F-MNIST are typical datasets, consisting of 60000 28 × 28 pixel pictures ofdifferent numbers and 10 types of clothes accordingly.
COIL-20 is a dataset of pictures ofobjects taken from 72 different angles spanning 360 degrees.
scRNA mice dataset has 1402 single nuclei extracted from hippocampal anatomical sub-regions (DG, CA1, CA2, and CA3), and scRNA melanoma dataset monitors expression of 4645 cells isolated from 19 metastatic melanoma patients (Szubert et al., 2019).
Datasets licences:
• Mammoth (Coenen & Pearce, 2019b), CC Zero License.
Mammuthus primigenius (blumbach), The Smithsonian Institute, https://3d.si.edu/object/3d/mammuthus-primigeniusblumbach:341c96cd-f967-4540-8ed1-d3fc56d31f • MNIST (LeCun et al., 1998), MIT License.
• Fashion-MNIST
(Xiao et al., 2017), MIT License.
• COIL-
(Nene et al., 1996).
• scRNA mice (Yuan et al., 2017).
• scRNA melanoma (Tirosh et al., 2016).
M M ORE DETAILS ON EXPERIMENTS WITH “S PHERES ” AND “T ORUS ”
We performed experiments on dimensionality reduction to 3D space to evaluate preservation of 3-dimensional structures in data by our method.
Experiments setup was similar to one described in
Section 4.
For this task we used 2 synthetic datasets.
The “Spheres” dataset consists of 17,250 points randomly distributed on surface of eleven 100-spheres in 101-dimensional space.
Any two of those do not intersect and one of the spheres contains all other inside.
Similar to “Circle” dataset (Section 4.1) UMAP splits bigger sphere (light grey) into 10 parts and wraps each small sphere into one of them.
PacMAP performs similar
but it also splits a part of bigger sphere into separate sphere.
PCA and Ivis preserve shape of inner shpheres only and turn all structure ‘inside out’.
Both t-SNE and regular AE projects all points onto one sphere without clear separation between clouds.
Additional of a topological loss — in both TopoAE and our RTD-AE allows to preserve global structure of inlaid clusters, however TopoAE flattens inner clusters into disks while RTD-AE makes them into (hollow) spheres.
[Table] The “Torus” dataset consists of 5,000 points randomly distributed on surface of a 2-torus ( T 2 ) immersed into 100-dimensional space.
Due to such nature of this dataset, PCA and MDS methods perform on it very well.
RTD-AE takes the third place but its quality is not much worse.
For “Spheres” dataset we also performed experiments on dimensionality reduction to 2D space.
Overall results are quite similar to those obtained for 3D case.
Behavior of baselines doesn’t essentially change from before.
Only interesting change is that RTD-AE now projects bigger sphere to a ring not circle and puts circles of lesser spheres into its hollow center.
Metrics presented at
Table show PCA and MDS are clearly winning in terms of Linear Correlation of distances with
RTD-AE slightly behind those two, but later is clearly winning when it comes to other two metrics –
Wasserstein distance and triplet accuracy.
All of the representations were generated with default parameters of baseline methods.
Results are presented at Figures 4 (“Spheres” to 3D space) and 11 (“Spheres” to 2D space and “Torus” to 3D).
[Table]
N A BLATION STUDY
In this section we investigate the effect of adding RTD loss on the performance of the model.
We add a hyperparameter λ responsible for the scale of the RTD loss variable:
L rec ( X, ˜ X ) + λ RTD ( X, Z ) .
We run our experiments on two datasets: COIL-20 and Circle.
The hyperparameter value ranged from 10 − 6 to 10 3 .
For each value of λ we run the procedure 8 times to get the confidence levels of our metrics.
The results are depicted at Figure 12.
O M ORE DIMENSIONALITY REDUCTION
METHODS
ON “M
AMMOTH ” DATASET
See Figure 13.
P R-C ROSS -B ARCODES
See Figure 14.
Q R ECONSTRUCTION LOSS
See Table 11 for results.
R H YPERPARAMETERS SEARCH FOR S PHERES DATASET ( INTO 2D)
For TopoAE we performed hyperparametrs search in full accordance with the original paper Moor et al.
(2020) and selected best combination according to KL 0 .
1 -divergence.
[Figure]
Results are presented at Table 12.
For Wasserstein Distance and Triplet Accuracy difference between means is lesser than standard derivations and due to this, we performed one-tailed Student’s t-test to [Figure]
The distinguishing topological feature between X and ˜ X is the cycle in ˜ X which is born at α =and dies at α = √ 2 .
The R-Cross-Barcode 1 ( ˜ X, X ) depicts this difference.
verify their relation. According to its results, we can reject the null hypothesis that the mean W.D. Hfor TopoAE is lower than the mean W.D. H 0 for RTD-AE at a significance level of 0.05.
Same result confirming the better performance of RTD-AE was obtained for the triplet accuracy.
We compare two point clouds X, ˜ X from Figure 16. For these point clouds, RTD ( X, ˜ X ) = 0 . 207 , while the topological part of the TopoAE loss equals 0 . The distinguishing topological feature between
X and ˜ X is the cycle in ˜ X which is born at α = 1 and dies at α = √ 2 . R-Cross-Barcode( ˜ X , X) depicts this difference. [Table] [Table]