Original: M ANIFOLD - ALIGNED N EIGHBOR E MBEDDING
Revised: M ANIFOLD - ALIGNED N EIGHBOR E MBEDDING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In this paper, we introduce a neighbor embedding framework for manifold alignment.
Revised: In this paper we introduce a neighbor embedding framework for manifold alignment.

Original: We demonstrate the efficacy of the framework using a manifold-aligned version of the uniform manifold approximation and projection algorithm.
Revised: We demonstrate the efficacy of the framework using a manifold-aligned version of the uniform manifold approximation and projection algorithm.

Original: We show that our algorithm can learn an aligned manifold that is visually competitive to when the true manifold is known.
Revised: We show that our algorithm can learn an aligned manifold that is visually competitive to embedding of the whole dataset

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Unsupervised clustering algorithms, like t-distributed stochastic neighbor embedding  (Maaten & Hinton, 2008) and uniform manifold approximation and projection  (McInnes et al., 2018), map points in high-dimensional data space to a more visualizable low-dimensional space.
Revised: Unsupervised clustering algorithms, like t-distributed stochastic neighbor embedding (tSNE) (Maaten & Hinton, 2008) and uniform manifold approximation and projection (UMAP) (McInnes et al., 2018), map points in high-dimensional data space to a more visualizable low-dimensional space.

Original: They are growing in popularity, as the learning is unsupervised and the resulting clusters are often interpretable.
Revised: They are growing in popularity, as the learning is unsupervised and the resulting clusters are often interpretable.

Original: However, the methods break down when points are taken from disparate datasets, as there is no link for pairwise comparison as the neighborhood graph networks are disconnected.
Revised: However, the methods break down when points are taken from disparate datasets, as there is no link for pairwise comparison as the neighborhood graph networks are disconnected.

Original: The problem of mapping correlated points in unrelated ways, known as manifold alignment (Ma & Fu, 2012), occurs whenever post-analysis data is added, e.g., for longitudinal studies, data compilations, and multi-modal measurements.
Revised: The problem of mapping correlated points in unrelated ways, known as manifold alignment (Ma & Fu, 2012), occurs whenever post-analysis data is added, e.g., for longitudinal studies, data compilations, and multi-modal measurements.

Original: It also arises with privacy and proprietary concerns, when source data cannot be transferred off-site.
Revised: It also arises with privacy and proprietary concerns, when source data cannot be transferred off-site.

Original: In each case, silos of datasets are created that cannot be part of a central representation learning scheme.
Revised: In each case, silos of datasets are created that cannot be part of a central representation learning scheme.

Original: Previous attempts at manifold alignment have used a variety of methods, including semi-supervised learning (Ham et al., 2005), spectral techniques (Wang & Mahadevan, 2009), and Procrustes analysis (Wang & Mahadevan, 2008).
Revised: Previous attempts at manifold alignment have used a variety of methods, including semi-supervised learning (Ham et al., 2005), spectral techniques (Wang & Mahadevan, 2009), and Procrustes analysis (Wang & Mahadevan, 2008).

Original: While the former uses known labels or bases for correlation, the latter is topological and integrates well with UMAP.
Revised: While the former uses known labels or bases for correlation, the latter is topological and integrates well with UMAP.

Original: Nevertheless, Procrustes transformations are not sufficient to align shared information among the datasets.
Revised: Nevertheless, Procrustes transformations are not sufficient to align shared information among the datasets.

Revised: This is because, Procrustes transformation is composed of various linear transforamtions which work on all the samples as an envelope of the data instead of individual samples.

Original: Here, we overcome this issue by iteratively optimizing the neighbor analysis for each dataset and jointly embedding the shared points.
Revised: Here, we overcome this issue by iteratively optimizing the neighbor analysis for each dataset and jointly embedding the shared points.

Original: We mathematically characterize the new approach and experimentally validate it on several widely used datasets.
Revised: We mathematically characterize the new approach and experimentally validate it on several widely used datasets.

Original:  M ANIFOLD - ALIGNED NEIGHBOR EMBEDDING (MANE) FRAMEWORK
Revised:  M ANIFOLD - ALIGNED NEIGHBOR EMBEDDING (MANE) FRAMEWORK

Original: Assume the individual n -dimensional local datasets D ( m )
Revised: Assume the individual n -dimensional local datasets D ( m )

Original: =
Revised: =

Original: { z ( m ) i } , where z i ∈ R n and k = 1 , 2 , 3 , . .
Revised: { z ( m ) i } , where z i ∈ R n and m = 1 , 2 , 3 , . .

Original: , M , cannot interact with each other.
Revised: , M , cannot interact with each other.

Original: The index i = 1 , 2 , 3 , . .
Revised: The index i = 1 , 2 , 3 , . .

Original: , N m indexes each data point in the dataset.
Revised: , N m indexes each data point in the dataset.

Original: These datasets are local as in the datasets my be located in different physical systems that cannot interact with each other or hidden from each other.
Revised: These datasets are local as in the datasets my be located in different physical systems that cannot interact with each other or hidden from each other.

Original: We construct a seeding dataset D (0) =
Revised: We construct a seeding dataset D (0) =

Original: { z i } , where i = 1 , 2 , 3 , . . .
Revised: { z i } , where i = 1 , 2 , 3 , . . .

Original: , N 0 , in order to construct extended datasets
Revised: , N 0 , in order to construct extended datasets

Original: D ( m 
Revised: D ( m ) =

Original: [Equation]

Original: = [Equation] ( m [Equation]
Revised: D (0) ∪ D ( m ) =

Original: = { x ( m ) i } , where i = 1 , 2 , 3 , . . .
Revised: { x ( m ) i } , where i = 1 , 2 , 3 , . . .

Original: , N 0 , N 0 + 1 , N 0 + 2 , . . .
Revised: , N 0 , N 0 + 1 , N 0 + 2 , . . .

Original: , N 0 + N m .
Revised: , N 0 + N m .

Original: Alternatively, D (0) is the shared data among all the datasets D ( m ) .
Revised: Alternatively, D (0) is the shared data among all the datasets D ( m ) .

Original: For notational simplicity and without loss of generality, we assume that x ( m )=
Revised: For notational simplicity and without loss of generality, we assume that x ( m )=

Original: [Equation] = z 
Revised: z 0 , x ( m )=

Original: [Equation] , x [Equation]
Revised: z 1 , . .

Original:  = z N ∈ D (0) and x ( m ) N 0 +1 , x ( m ) N 0 +2 , . . .
Revised: , x ( m ) N 0 = z N ∈ D (0) and x ( m ) N 0 +1 , x ( m ) N 0 +2 , . . .

Original: , x ( m ) N 0 + N m ∈ D ( m ) .
Revised: , x ( m ) N 0 + N m ∈ D ( m ) .

Original: Now we can define a high-dimensional weighted graph p ( m ) for each of the datasets using pair wise metric d  ( · , · ) .
Revised: Now we can define a high-dimensional weighted graph p ( m ) for each of the datasets using pairwise metric d H ( · , · ) .

Original: The entries of the adjacency matrix are given by
Revised: The entries of the adjacency matrix are given by

Original: [Equation]
Revised: [Equation]

Original: where f H ( · ) is a function that describes the weighted relation between points x i , x j ∈ D ( m ) subject to the distance metric.
Revised: where f H ( · ) is a function that describes the weighted relation between points x i , x j ∈ D ( m ) subject to the distance metric.

Revised: In t-SNE the function f H ( · ) is a Gaussian, whereas in UMAP it’s k-nearest neighbor based affinity function.

Original: It is to be noted that, this construction of graph p ( m ) can be computed in the same location where dataset D ( m ) is located.
Revised: It is to be noted that, this construction of graph p ( m ) can be computed in the same location where dataset D ( m ) is located.

Original: In what follows, the embedding algorithm or the global part of the algorithm effectively requires this adjacency matrix p ( m ) in order to compute the low-dimensional embedding, and any access to individual data points is not necessary.
Revised: In what follows, the embedding algorithm or the global part of the algorithm effectively requires this adjacency matrix p ( m ) in order to compute the low-dimensional embedding, and any access to individual data points is not necessary.

Original: We initialize the low-dimensional ( d -dimensional) embedding | D ( m ) | = { y ( m ) i } for each dataset, where y ( m ) i ∈
Revised: We initialize the low-dimensional ( d -dimensional) embedding | D ( m ) | = { y ( m ) i } for each dataset, where y ( m ) i ∈

Original: R
Revised: R

Original: d is the corresponding low-dimensional mapping of the high-dimensional point x ( m ) i ∈ D ( m ) .
Revised: d is the corresponding low-dimensional mapping of the high-dimensional point x ( m ) i ∈ D ( m ) .

Original: Now we can define a low-dimensional weighted graph for each of the datasets using the following pairwise relation
Revised: Now we can define a low-dimensional weighted graph for each of the datasets using the following pairwise relation

Original: [Equation]
Revised: [Equation]

Original: where f L ( · ) is a function of the weighted relation in the low dimension.
Revised: where f L ( · ) is a function of the weighted relation in the low dimension and d L ( · ) is some pairwise metric.

Original: Typically, d ≪ n .
Revised: Typically, d ≪ n .

Revised: In t-SNE f L ( · ) is the normalized Student’s t-distribution function with one degree of freedom.

Revised: UMAP skips the normalization step and uses a modified function similar to

Revised: Student’s t-distribution function of t-SNE’s but has tunable parameters.

Original: Finally the relation between the high-dimensional graphs and their joint low-dimensional embedding is established by optimizing the following problem
Revised: Finally the relation between the high-dimensional graphs and their joint low-dimensional embedding is established by optimizing the following problem

Original: [Equation]
Revised: [Equation]

Original: where, l ( · , · ) is the loss function.
Revised: where, l ( · , · ) is the loss function.

Original: The constraint in Eq.
Revised: The constraint in Eq.

Original: 3 indicates that in MANE, the seeding dataset captures the shared manifold and works as anchor points, around which other data points are aligned in the low-dimensional embedding. 
Revised: 3 indicates that in MANE, the seeding dataset captures the shared manifold and works as anchor points, around which other data points are aligned in the low-dimensional embedding. To the best of our knolwedge this is the first time the constraint in Eq. 3 is being used for manifold alignment.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we briefly describe the implementation of the framework in an algorithm and the evaluation metrics to validate our results.
Revised: In this section, we briefly describe the implementation of the framework in an algorithm and the evaluation metrics to validate our results.

Original: In the main text, we describe our results for FashionMNIST (Xiao et al., 2017).
Revised: In the main text, we describe our results for FashionMNIST (Xiao et al., 2017).

Original: Results for additional datasets are described in the Appendix C.
Revised: Results for additional datasets are described in the Appendix C.

Original: This framework can be implemented using the principles of most modern dimensionality reduction methods.
Revised: This framework can be implemented using the principles of most modern dimensionality reduction methods.

Original: Our implementation of the framework is based on the UMAP algorithm.
Revised: Our implementation of the framework is based on the UMAP algorithm.

Original: We follow the
Revised: We follow the

Original: UMAP principles to build the high-dimensional graphs p ( m ) i,j , low-dimensional graphs q ( m ) i,j for each dataset D ( m ) .
Revised: UMAP principles to build the high-dimensional graphs p ( m ) i,j , low-dimensional graphs q ( m ) i,j for each dataset D ( m ) .

Original: Then we define the loss function l ( p ( m ) i,j , q ( m ) i,j ) to be the cross-entropy loss function.
Revised: Then we define the loss function l ( p ( m ) i,j , q ( m ) i,j ) to be the cross-entropy loss function.

Original: We optimize the embeddings of each dataset jointly using the negative sampling Mikolov et al.
Revised: We optimize the embeddings of each dataset jointly using the negative sampling Mikolov et al.

Original: (2013) approach.
Revised: (2013) approach.

Original: We sample one point from one of the datasets and optimize the loss function employing the constraint in Eq.
Revised: We sample one point from one of the datasets and optimize the loss function employing the constraint in Eq.

Original: More details of the implementation are provided in Apendix A.
Revised: More details of the implementation are provided in Appendix A.

Original: We compare our results to two other UMAP based methods: 1) individual UMAP: when the datasets are embedded individually and 2) aligned UMAP: a software package of UMAP that aligns the learned manifolds using procrsutes analysis then optimizes each of the embeddgins with a regularizer constraint on the shared points.
Revised: We compare our results to two other UMAP based methods: 1) individual UMAP: when the datasets are embedded individually and 2) aligned UMAP: a software package of UMAP that aligns the learned manifolds using Procrustes analysis, then optimizes each of the embeddings with a regularizer constraint on the shared points.

Original: For each comparison, the number of nearest neighbors, minimum distance parameter, and negative sampling rate has been set to 30, 0.1, and 1.0, respectively, for all embeddings.
Revised: For each comparison, the number of nearest neighbors, minimum distance parameter, and negative sampling rate have been set to 30, 0.1, and 1.0, respectively, for all embeddings.

Original: Aligned UMAP optimizes each dataset for 200 epochs individually and then  optimizes the aligned embeddings for further 200 epochs.
Revised: Aligned UMAP optimizes each dataset for 200 epochs individually and then [Figure] [Figure] optimizes the aligned embeddings for further 200 epochs.

Original: For UMAP implementation, the precomputed distance matrix has been used.
Revised: For UMAP implementation, the precomputed distance matrix has been used.

Original: Our MANE embeddings were obtained using 200 epochs and no pre-embeddings were required.
Revised: Our MANE embeddings were obtained using 200 epochs and no pre-embeddings were required.

Original: We initialized the embeddings using the PCA axis of the shared data.
Revised: We initialized the embeddings using the axes obtained from the principal component analysis (PCA) of the shared data.

Original: To compare the embeddings numerically, we use Procrustes distance ( d p ) and trustworthiness ( T ) Venna & Kaski (2001) metrics.
Revised: To compare the embeddings numerically, we use Procrustes distance ( d p ) and trustworthiness ( T ) (Venna & Kaski, 2001) metrics.

Original: Procsustes distance is the minimum euclidean distance between to datasets under translation, scaling, rotation, and reflection.
Revised: Procrustes distance is the minimum euclidean distance between two datasets under translation, scaling, rotation, and reflection.

Original: Trustworthiness is a measure that gives a sense of how much of the nearest neighbors in the high dimension are preserved in the low dimension after embedding. 
Revised: Trustworthiness is a measure that gives a sense of how much of the nearest neighbors in the high dimension are preserved in the low dimension after embedding. More details of these metrics are provided in Appendix B.

Revised: The PCA initializatin scheme can be used as a linear baseline, which has been further illustrated in Appendix D.

Original: The Fashion-MNIST dataset contains 70,000 gray-scale images of fashion items.
Revised: The Fashion-MNIST dataset contains 70,000 gray-scale images of fashion items.

Original: The training data contains 60,000 images, which have been used in the experiments.
Revised: The training data contains 60,000 images, which have been used in the experiments.

Original: First, we show results by randomly splitting the Fashion-MNIST dataset into three datasets D ( m ) , where m
Revised: First, we show results by randomly splitting the Fashion-MNIST dataset into three datasets D ( m ) , where m

Original: = 1 , 2 , 3 , where
Revised: = 1 , 2 , 3 , where

Original: [Figure] [Figure] D (0) contains 10,000 samples and the other splits contains 25,000 samples each.
Revised: D (0) contains 10,000 samples and the other splits contains 25,000 samples each.

Original: Then we construct
Revised: Then we construct

Original:  D (1) = D (0) ∪D (1) and D (1) = D (0) ∪D (2) .
Revised: [Figure] D (1) = D (0) ∪ D (1) and D (1) = D (0) ∪ D (2) .

Original: The embeddings obtained using different schemes are shown in Figure 1.
Revised: The embeddings obtained using different schemes are shown in Figure 1.

Original: Since Individual UMAP and aligned UMAP provide separate embeddings for the shared points, we compute the trustworthiness metric of each dataset ( T (1)) and T (2)) ).
Revised: Since Individual UMAP and aligned UMAP provide separate embeddings for the shared points, we compute the trustworthiness metric of each dataset ( T (1) and T (2) ).

Original: On the other hand, our method embeds both datasets into the same metric space and thus we can compute trustworthiness as if the whole Fashion-MNIST dataset has been embedded.
Revised: On the other hand, our method embeds both datasets into the same metric space and thus we can compute trustworthiness as if the whole Fashion-MNIST dataset has been embedded.

Original: Thus in addition to reporting T (1)) and T (2)) , we also report trustworthiness of union of the two embeddings ( T ).
Revised: Thus in addition to reporting T (1) and T (2) , we also report trustworthiness of union of the two embeddings ( T ).

Original: We can observe from the figure that, all embeddings capture the general manifold of the Fashion-MNIST data as shown in the UMAP of the Fashion-MIST dataset.
Revised: We can observe from the figure that, all embeddings capture the general manifold of the Fashion-MNIST data as shown in the UMAP of the Fashion-MIST dataset.

Original: However, in individual UMAPs the embeddings | D (1) | and | D (2) | are not aligned to each other.
Revised: However, in individual UMAPs the embeddings | D (1) | and | D (2) | are not aligned to each other.

Revised: This can be realized by observing the large cluster on the left involving the labels sneaker, ankle boot and sandal (colored teal, violet and greenish-yellow, respectively) which is oriented differently in the two embeddings indicating misalignment.

Revised: Moreover, the other cluster on the lower right involving the labels t-shirt, pullover and dress (colored maroon, orange, and light green, respectively) is more compact in | D (2) | than it is in | D (1) | .

Original: On the other hand, both aligned UMAP and MANE produce embeddings that are aligned to each other.
Revised: On the other hand, both aligned UMAP and MANE produce embeddings that are aligned to each other as the discrepencies described above are absent for these two.

Original: The trustworthiness of MANE is closer to the trustworthiness metric when the whole dataset has been embedded using UMAP.
Revised: The trustworthiness of MANE is closer to the trustworthiness metric when the whole dataset has been embedded using UMAP.

Revised: The trustworthiness metric T (1) and T (2 ) are similar for all three metric, which indicates all of them have similar performance for embedding D (1) and D (2) whereas, the added benefit of aligned UMAP and MANE is the manifold alignment.

Revised: While aligned UMAP uses regularization constraint for alignment (along with some costly pre-processing), MANE uses a hard constraint described in Eq 3.

Original: This shows that jointly aligning and embedding using a hard constraint of Eq.
Revised: This shows that jointly aligning and embedding using a hard constraint of Eq.

Original: 3 is a viable way to obtain aligned embedding.
Revised: 3 is a viable way to obtain aligned embedding.

Original: In Figure 2, we examine how the shared points have been embedded.
Revised: In Figure 2, we examine how the shared points have been embedded.

Original: We can see that the shared points are not in agreement for individual UMAP and aligned UMAP, but they are in agreement for MANE.
Revised: We can see that the shared points are not in agreement for individual UMAP and aligned UMAP, but they are in agreement for MANE.

Original: Now we use the same scheme of the previous experiment but vary the value of N 0 , the number of shared points, to see how much  shared information influences the alignment.
Revised: Now we use the same scheme of the previous experiment but vary the value of N 0 , the number of shared points, to see how much the shared information influences the alignment.

Original: We kept the total[Figure] number of unique points to 60,000.
Revised: We kept the total number of unique points to 60,000.

Original: The resultant embeddings are shown in Figure 3.
Revised: The resultant embeddings are shown in Figure 3.

Original: It can be observed that  for N 0> = 5000 the embeddings are in good alignment.
Revised: It can be observed that except for N 0 = 1000 and N 0 = 100 , all other embeddings are in good alignment.

Original: However, for N 0 =and N 0 = 100 , the alignment is somewhat lost.
Revised: For N 0 = 1000 and N 0 = 100 , the alignment is somewhat lost.

Original: For these embeddings, the large structures are in similar places having similar orientation, however, finer structures are not aligned (i.e., the  structure  of sandals located in the leftportion of each embedding).
Revised: For these embeddings, the large structures are in similar places having similar orientation, however, finer structures are not aligned. For example, for N 0 = 1000 and N 0 = 100 , the larger cluster in the lower right is compact for | D (2) | compared to that of | D (1) | . For N 0 = 100 , the spike like structure in the cluster of sandal (colored greenish-yellow in the left of the figure) is different for | D (1) | and | D (2) | .

Original: This is due to the fact that the shared points work as anchors around which the manifold of both datasets is arranged.
Revised: This is due to the fact that the shared points work as anchors around which the manifold of both datasets is arranged.

Original: However, with merely 1000 or 100 points, the shared points cannot sample enough of the manifold.
Revised: However, with merely 1000 or 100 points, the shared points cannot sample enough of the manifold.

Original: Finally, we look into a case where more than two datasets are involved.
Revised: Finally, we look into a case where more than two datasets are involved.

Original: We split the Fashion-MNIST data into 5 datasets of equal size (14400 data points each).
Revised: We split the Fashion-MNIST data into 5 datasets of equal size (14400 data points each).

Original: The number of shared points is 3000.
Revised: The number of shared points is 3000.

Original: The resultant aligned embeddings are shown in Figure 4.
Revised: The resultant aligned embeddings are shown in Figure 4.

Original: The figures show that all 5 datasets are aligned.
Revised: The figures show that all 5 datasets are aligned.

Original:  C ONCLUSIONS AND F UTURE W ORK
Revised:  C ONCLUSIONS AND F UTURE W ORK

Original: In this paper, we introduce  a neighbor embedding approach for aligning the manifold of multiple datasets that use the same underlying manifold.
Revised: In this paper, we introduce MANE, a neighbor embedding approach for aligning the manifold of multiple datasets that use the same underlying manifold.

Original: We modeled the problem in terms of shared data points among multiple datasets.
Revised: We modeled the problem in terms of shared data points among multiple datasets.

Original: We implemented our approach using UMAP principles.
Revised: We implemented our approach using UMAP principles.

Original: The efficacy of our method in aligning datasets is demonstrated by several experiments.
Revised: The efficacy of our method in aligning datasets is demonstrated by several experiments.

Original: We showed the shared points can be in perfect alignment yet produce embeddings that are comparable to the embedding of the combined dataset.
Revised: We showed the shared points can be in perfect alignment yet produce embeddings that are comparable to the embedding of the combined dataset.

Original: We compared our method with aligned UMAP in which the shared data are not in perfect alignment.
Revised: We compared our method with aligned UMAP in which the shared data are not in perfect alignment.

Original: Overall,  the  methodis robust and can be used in practice where shared data is known or a seeding dataset is available.
Revised: Overall, MANE produce emeddings that are comparable to mebedding of combined dataset, and provides perfect alignment for the shared points. This method can be used in practice where shared data is known or a seeding dataset is available.

Original: In the future, a more interesting demonstration would be in a case where two datasets are of different modalities (say x-ray image and its corresponding report or two different medical image modalities).
Revised: In the future, a more interesting demonstration would be in a case where two datasets are of different modalities (say x-ray image and its corresponding report or two different medical image modalities).

Original: The dataset D (0) contains pairs of x-ray image and reports, whereas, D (1) and D (2) are datasets of x-rays, and reports, respectively.
Revised: The dataset D (0) contains pairs of x-ray image and reports, whereas, D (1) and D (2) are datasets of x-rays, and reports, respectively.

Original: Thus, using manifold-aligned neighbor embedding one can jointly obtain the embedding of x-rays and reports, which will be of great importance in medical AI.
Revised: Thus, using manifold-aligned neighbor embedding one can jointly obtain the embedding of x-rays and reports, which will be of great importance in medical AI.

Original: A I MPLEMENTATION
Revised: A CKNOWLEDGEMENT

Revised: This material is based upon work supported by the Air Force Office of Scientific Research (AFOSR) under Grant FA9550-18-1-0219 and by the Defense Advanced Research Projects Agency (DARPA) under Agreement No. HR00112090123.

Revised: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of DARPA, AFOSR, or the U.S. Government.

Revised: C ODE AND D ATA A VAILABILITY

Revised: The codebase of the paper can be obtained from https://github.com/tariqul-islam/ mane_paper .

Revised: All the data used in this paper are publicly availble.

Revised: Fashion-MNIST is available at https://github.com/zalandoresearch/ fashion-mnist .

Revised: MNIST is available at http://yann.lecun.com/exdb/mnist/ .

Revised: Swiss roll data can be generated from https://scikit-learn.org/stable/modules/ generated/sklearn.datasets.make_swiss_roll.html .

Revised: Single cell transcriptomes data is available at https://github.com/biolab/ tsne-embedding .

Revised: A I MPLEMENTATION

Original: In this paper, we show an implementation of the general framework described in Section 2 using
Revised: In this paper, we show an implementation of the general framework described in Section 2 using

Original: UMAP principles.
Revised: UMAP principles.

Original: We define the high-dimensional weighted relation for each dataset D ( m ) by
Revised: We define the high-dimensional weighted relation for each dataset D ( m ) by

Original: [Equation]
Revised: [Equation]

Original: ifx (
Revised: if

Original: [Equation]
Revised: [Equation]

Original: [Equation] , k ) otherwise (4) where KNN ( x ( m ) i , k ) is the set of k -nearest neighbors in dataset D ( m ) of the point x ( m ) i , ρ ( m ) i =
Revised: k ) otherwise (4) where KNN ( x ( m ) i , k ) is the set of k -nearest neighbors in dataset D ( m ) of the point x ( m ) i , ρ ( m ) i =

Original: min x ( m ) j ∈ KNN ( x ( m ) i ,k ) d ( x ( m ) i , x ( m ) j ) and σ ( m )
Revised: min x ( m ) j ∈ KNN ( x ( m ) i ,k ) d ( x ( m ) i , x ( m ) j ) and σ ( m )

Original: i is a scaling parameter set such that (cid:80) j p ( m ) i | j = log 2 ( k ) .
Revised: i is a scaling parameter set such that (cid:80) j p ( m ) i | j = log 2 ( k ) .

Original: The parameter ρ ( m ) i ensures that the point x ( m ) i has strong connectivity ( p ( m ) i | j = 1 ) to at least one of the nearest neighbors and the scaling parameter σ ( m ) i ensures the uniform manifold approximation.
Revised: The parameter ρ ( m ) i ensures that the point x ( m ) i has strong connectivity ( p ( m ) i | j = 1 ) to at least one of the nearest neighbors and the scaling parameter σ ( m ) i ensures the uniform manifold approximation.

Original: The adjacency matrix p ( m ) of the dataset D ( m ) in the high-dimension is then obtained by (the probabilistic t-conorm),
Revised: The adjacency matrix p ( m ) of the dataset D ( m ) in the high-dimension is then obtained by (the probabilistic t-conorm),

Original: [Equation]
Revised: [Equation]

Original: We define the low-dimensional weighted relation by a differentiable function
Revised: We define the low-dimensional weighted relation by a differentiable function

Original: [Equation]
Revised: [Equation]

Original: (6) where the parameters a and b determine how crowded the low-dimensional points become after embedding.
Revised: (6) where the parameters a and b determine how crowded the low-dimensional points become after embedding.

Original: These two parameters are chosen by fitting q ij toΨ
Revised: These two parameters are chosen by fitting q ij to

Original: [Equation]
Revised: [Equation]

Original: where d m is a user-defined parameter that regulates the minimum distance between two lowdimensional points.
Revised: where d m is a user-defined parameter that regulates the minimum distance between two lowdimensional points.

Original: Finally, we define the loss function
Revised: Finally, we define the loss function

Original: [Equation]
Revised: [Equation]

Original: The first term in the loss function provides the attractive force to bring similar points closer together, and the second term provides the repulsive force that enables far apart points in the high dimension to stay far apart in the low dimension.
Revised: The first term in the loss function provides the attractive force to bring similar points closer together, and the second term provides the repulsive force that enables far apart points in the high dimension to stay far apart in the low dimension.

Original: To optimize this loss function, we employed the negative sampling (Mikolov et al., 2013) approach that was used in the original UMAP implementation.
Revised: To optimize this loss function, we employed the negative sampling (Mikolov et al., 2013) approach that was used in the original UMAP implementation.

Original: In each optimization step, we sample one positive edge from one of the graphs and apply the attractive force to it.
Revised: In each optimization step, we sample one positive edge from one of the graphs and apply the attractive force to it.

Original: Then we sample negative edges and apply the repulsive forces to these edges.
Revised: Then we sample negative edges and apply the repulsive forces to these edges.

Revised: For a positive edge which is part of the seeding dataset, we apply the constraint in Eq.

Revised: 3 so that the embedding of the shared point is same across all datasets.

Revised: In the code, we ensure this by having a shared memory for these shared embedded points.

Original: B E VALUATION M ETRICS B. T RUSTWORTHINESS
Revised: B E VALUATION M ETRICS B. T RUSTWORTHINESS

Original: The trustworthiness metric measures how much of the local structure has been preserved after dimensionality reduction by
Revised: The trustworthiness metric measures how much of the local structure has been preserved after dimensionality reduction by

Original: [Equation]
Revised: [Equation]

Original: where KNN ( y i , k ) is the k -NN graph in the embedding space and r ( i, j ) is the rank of x j in the high-dimensional k -NN graph.
Revised: where KNN ( y i , k ) is the k -NN graph in the embedding space and r ( i, j ) is the rank of x j in the high-dimensional k -NN graph.

Original: Usually, the value of k is set to 5, which considers only 5 nearest neighbors in the low-dimensional embedding.
Revised: Usually, the value of k is set to 5, which considers only 5 nearest neighbors in the low-dimensional embedding.

Original: B. P ROCRUSTES D ISTANCE
Revised: B. P ROCRUSTES D ISTANCE

Original: Procrustes distance is the minimum distance between two sets of points under scaling, translation, reflection, and rotation.
Revised: Procrustes distance is the minimum distance between two sets of points under scaling, translation, reflection, and rotation.

Original: Functionally, for two datasets { x i } and { y i } we want to scale, translate and rotate one of the datasets in such a way that the two datasets are in maximum alignment.
Revised: Functionally, for two datasets { x i } and { y i } we want to scale, translate and rotate one of the datasets in such a way that the two datasets are in maximum alignment.

Original: Assuming { y ′ i } is a transformation of { y i } that achieves the desired goal.
Revised: Assuming { y ′ i } is a transformation of { y i } that achieves the desired goal.

Original: Then Procrustes distance is given by
Revised: Then Procrustes distance is given by

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: C A DDITIONAL R ESULTS
Revised: C A DDITIONAL R ESULTS

Original: In this section, we describe the results of some additional datasets.
Revised: In this section, we describe the results of some additional datasets.

Original: C. S WISS R OLL
Revised: C. S WISS R OLL

Original: The swiss roll (Roweis & Saul, 2000) is a two-dimensional manifold in a 3D space.
Revised: The Swiss roll (Roweis & Saul, 2000) is a two-dimensional manifold in a 3D space.

Original: Figure 5 shows 60,000 samples from the manifold.
Revised: Figure 5 shows 60,000 samples from the manifold.

Original: The data are colored according to their position in the manifold to understand their apparent location in the original space.
Revised: The data are colored according to their position in the manifold to understand their apparent location in the original space.

Original: The experiments performed are similar to that of Fashion-MNIST data that was performed in the main text.
Revised: The experiments performed are similar to that of Fashion-MNIST data that was performed in the main text.

Original: Figures 6-9 describe the results.
Revised: Figures 6-9 describe the results.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: C. MNIST D ATASET
Revised: C. MNIST D ATASET

Original: MNIST (LeCun et al., 1998) is a classic dataset used in many machine learning research.
Revised: MNIST (LeCun et al., 1998) is a classic dataset used in many machine learning research.

Original: The dataset has the same structure as the Fashion-MNIST dataset.
Revised: The dataset has the same structure as the Fashion-MNIST dataset.

Original: There are 60,000 training data and 10, test data consisting of 10 categorical classes.
Revised: There are 60,000 training data and 10, test data consisting of 10 categorical classes.

Original: Our results here involve only the training data.
Revised: Our results here involve only the training data.

Original: The results are compiled in Figures
Revised: The results are compiled in Figures

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: C. S INGLE - CELL TRANSCRIPTOMES
Revised: C. S INGLE - CELL TRANSCRIPTOMES

Original: This dataset was compiled by Macosko et al.
Revised: This dataset was compiled by Macosko et al.

Original: This includes 44,808 samples of single-cell transcriptomes obtained from the mouse retina.
Revised: This includes 44,808 samples of single-cell transcriptomes obtained from the mouse retina.

Original: Results are compiled in Figures 14-17.
Revised: Results are compiled in Figures 14-17.

Original: In figurewe split the data into two sets with 1000 shared points 
Revised: In figurewe split the data into two sets with 10,000 shared points [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Revised: D C OMPARISON TO PRINCIPAL COMPONENT ANALYSIS

Original: [Figure]
Revised: In this section, we describe a linear baseline using PCA. In this scheme, we obtain the PCA axes by performing PCA on D (0) . Using the computed axis then we project each of the datasets, D ( i ) and obtain the low dimensional embeddings. This scheme also ensures that the Procrustes distance isfor the shared points.

Original: [Figure]
Revised: [Figure]

Revised: F a s h i on - M N I S T

Revised: S w i ss

Revised: R o ll

Revised: M N I S T

Revised: S i ng l e

Revised: C e ll T r an sc r i p t

Revised: o m e s

Revised: [Figure]

Revised: In the experiments we use the same datasets we used in the precious sections.

Revised: The number of shared points N 0 is set to 10,000 and rest of the data is split into two datasets to obtain D (1) and D (2) .

Revised: The results are shown in Figure 18.

Revised: We can observe that, while the data are aligned to each other, the trustworthiness scores are significanly lower for the linear baseline except for Swiss roll.

Revised: Swiss roll data is a 2-dimensional manifold in 3-dimension which is easier to project in 2-dimension using

Revised: PCA axes compared to the rest of the data.