Original: T RI M AP : L ARGE - SCALE D IMENSIONALITY R EDUCTION U SING T RIPLETS
Revised: T RI M AP : L ARGE - SCALE D IMENSIONALITY R EDUCTION U SING T RIPLETS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We introduce “TriMap”; a dimensionality reduction technique based on triplet constraints that preserves the global accuracy of the data better than the other commonly used methods such as t-SNE, LargeVis, and UMAP.
Revised: We introduce “TriMap”; a dimensionality reduction technique based on triplet constraints that preserves the global accuracy of the data better than the other commonly used methods such as t-SNE, LargeVis, and UMAP.

Original: To quantify the global accuracy, we introduce a score which roughly reﬂects the relative placement of the clusters rather than the individual points.
Revised: To quantify the global accuracy, we introduce a score which roughly reﬂects the relative placement of the clusters rather than the individual points.

Original: We empirically show the excellent performance of TriMap on a large variety of datasets in terms of the quality of the embedding as well as the runtime.
Revised: We empirically show the excellent performance of TriMap on a large variety of datasets in terms of the quality of the embedding as well as the runtime.

Original: On our performance benchmarks, TriMap easily scales to millions of points without depleting the memory and clearly outperforms t-SNE, LargeVis, and UMAP in terms of runtime.
Revised: On our performance benchmarks, TriMap easily scales to millions of points without depleting the memory and clearly outperforms t-SNE, LargeVis, and UMAP in terms of runtime.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Data visualization based on dimensionality reduction (DR) is a core problem in data analysis and machine learning.
Revised: Data visualization based on dimensionality reduction (DR) is a core problem in data analysis and machine learning.

Original: The aim of DR is to provide a low-dimensional representation (typically in 2D or 3D) of a given high-dimensional dataset that preserves the overall structure of the data as much as possible.
Revised: The aim of DR is to provide a low-dimensional representation (typically in 2D or 3D) of a given high-dimensional dataset that preserves the overall structure of the data as much as possible.

Original: The earlier approaches for DR involve linear methods such as PCA (Pearson, 1901).
Revised: The earlier approaches for DR involve linear methods such as PCA (Pearson, 1901).

Original: PCA aims to maintain the second-order statistics of the data by projecting the points into the low dimensional space that preserves the maximum amount of variance among all such projections.
Revised: PCA aims to maintain the second-order statistics of the data by projecting the points into the low dimensional space that preserves the maximum amount of variance among all such projections.

Original: As a result, PCA has been shown to be effective in preserving the global structure of the data (Silva & Tenenbaum, 2003).
Revised: As a result, PCA has been shown to be effective in preserving the global structure of the data (Silva & Tenenbaum, 2003).

Original: The global structure includes the overall shape of the dataset, placement of the clusters, and existence of potential outliers.
Revised: The global structure includes the overall shape of the dataset, placement of the clusters, and existence of potential outliers.

Original: Unlike PCA, much of the focus of the more recent non-linear methods including t-SNE (Maaten & Hinton, 2008), LargeVis (Tang et al., 2016), and UMAP (McInnes et al., 2018) has been on preserving the local neighborhood structure of each individual point.
Revised: Unlike PCA, much of the focus of the more recent non-linear methods including t-SNE (Maaten & Hinton, 2008), LargeVis (Tang et al., 2016), and UMAP (McInnes et al., 2018) has been on preserving the local neighborhood structure of each individual point.

Original: Similarly, the common performance measures of DR such as trustworthinesscontinuity (Venna & Kaski, 2005), precision-recall (i.e. AUC) (Venna et al., 2010), and nearestneighbor accuracy have also been developed by retaining the same focus on reﬂecting the local accuracy of the embedding.
Revised: Similarly, the common performance measures of DR such as trustworthinesscontinuity (Venna & Kaski, 2005), precision-recall (i.e. AUC) (Venna et al., 2010), and nearestneighbor accuracy have also been developed by retaining the same focus on reﬂecting the local accuracy of the embedding.

Original: Thus, there has been a lack of attention on developing methods that focus on preserving the global structure of the data and likewise, practical performance measures to assess the global accuracy.
Revised: Thus, there has been a lack of attention on developing methods that focus on preserving the global structure of the data and likewise, practical performance measures to assess the global accuracy.

Original: We ﬁrst introduce the global score , a quantitative measure which reﬂects the closeness of a given embedding to the PCA embedding (which is optimal by means of preserving the data variance).
Revised: We ﬁrst introduce the global score , a quantitative measure which reﬂects the closeness of a given embedding to the PCA embedding (which is optimal by means of preserving the data variance).

Original: The purpose of this score is to measure the accuracy of an embedding in reﬂecting the overall placement of the clusters of points relative to their original representation in high-dimension.
Revised: The purpose of this score is to measure the accuracy of an embedding in reﬂecting the overall placement of the clusters of points relative to their original representation in high-dimension.

Original: By design, PCA yields the highest global score among all the DR methods and high values of global score indicates the efﬁcacy of a DR method in reﬂecting the global structure.
Revised: By design, PCA yields the highest global score among all the DR methods and high values of global score indicates the efﬁcacy of a DR method in reﬂecting the global structure.

Original: Next, we introduce TriMap , a DR method which focuses on preserving the global structure of the data in the embedding.
Revised: Next, we introduce TriMap , a DR method which focuses on preserving the global structure of the data in the embedding.

Original: Pairwise (dis)similarities between points (used by the previous DR methods) seem to be insufﬁcient in capturing the global structure.
Revised: Pairwise (dis)similarities between points (used by the previous DR methods) seem to be insufﬁcient in capturing the global structure.

Original: Instead, TriMap incorporates a higher order of structure to construct the embedding by means of triplets : 
Revised: Instead, TriMap incorporates a higher order of structure to construct the embedding by means of triplets : ( i, j, k ) ⇔ point i is closer to point j than point k . a r X i v : .

Original: [Equation]
Revised: 00204v [ c s . L

Original: The key idea behind TriMap stems from semi-supervised metric learning (Amid et al., 2016): Given an initial low-dimensional representation for the data points, the triplet information from the highdimensional representation of the points is used to enhance the quality of the embedding.
Revised: G ]

Original: Similarly,
Revised: O c t

Original: UMAP (0.16, 0.13)
Revised: Original (AUC

Original: TriMap (0.15, 0.80 ) PCA (0.
Revised: GS) t-SNE

Original: [Figure]
Revised: [Figure]

Original: Onthe other hand, TriMap successfully unveils the underlying structure in theoriginal dataset.
Revised: On successfully unveils the underlying structure in the dataset.

Original: Note that GS is the only DR performance measure that can reﬂect this property.
Revised: that GS is the only DR performance measure that can reﬂect this property.

Revised: The key idea behind TriMap stems from semi-supervised metric learning (Amid et al., 2016):

Revised: Given an initial low-dimensional representation for the data points, the triplet information from the highdimensional representation of the points is used to enhance the quality of the embedding.

Revised: Similarly,

Original: TriMap is initialized with the low dimensional PCA embedding, and this embedding is then modiﬁed using a set of carefully selected triplets from the high-dimensional representation.
Revised: TriMap is initialized with the low dimensional PCA embedding, and this embedding is then modiﬁed using a set of carefully selected triplets from the high-dimensional representation.

Original: With an extensive set of experiments, we show that TriMap produces excellent results on a variety of real-world as well as synthetic datasets.
Revised: With an extensive set of experiments, we show that TriMap produces excellent results on a variety of real-world as well as synthetic datasets.

Original: We show that in many cases TriMap outperforms all the competitor non-linear methods by means of global score and provides comparable local accuracy.
Revised: We show that in many cases TriMap outperforms all the competitor non-linear methods by means of global score and provides comparable local accuracy.

Original: While being signiﬁcantly faster than t-SNE, TriMap provides comparable runtime to UMAP and
Revised: While being signiﬁcantly faster than t-SNE, TriMap provides comparable runtime to UMAP and

Original: LargeVis while scaling drastically better to larger datasets.
Revised: LargeVis while scaling drastically better to larger datasets.

Original: On the Character Font Images dataset of ⇠ 1 . 7 M points, TriMap calculates the embedding in ⇠ 1 . 3 hours while LargeVis takes more thanhours and UMAP exceeds the 12 hours time limit.
Revised: On the Character Font Images dataset of ∼ 1 . 7 M points, TriMap calculates the embedding in ∼ 1 . 3 hours while LargeVis takes more thanhours and UMAP exceeds the 12 hours time limit.

Original: Our contributions can be summarized as follows:
Revised: Our contributions can be summarized as follows:

Original: • We introduce a global score to quantify the quality of a low-dimensional embedding in reﬂecting the global structure of the high-dimensional data such as placement of the clusters rather than the local neighborhood of individual points.
Revised: • We introduce a global score to quantify the quality of a low-dimensional embedding in reﬂecting the global structure of the high-dimensional data such as placement of the clusters rather than the local neighborhood of individual points.

Original: • We introduce TriMap, a fast dimensionality reduction method which provides embeddings of the data that are globally more accurate than other non-linear DR methods such as t-SNE, LargeVis, and UMAP.
Revised: • We introduce TriMap, a fast dimensionality reduction method which provides embeddings of the data that are globally more accurate than other non-linear DR methods such as t-SNE, LargeVis, and UMAP.

Original: • We provide an efﬁcient implementation 1 of TriMap that can easily scale to millions of points on commodity hardware and outperforms the competing methods in terms of runtime.
Revised: • We provide an efﬁcient implementation 1 of TriMap that can easily scale to millions of points on commodity hardware and outperforms the competing methods in terms of runtime.

Original: We also perform many large-scale experiments on various datasets to show the efﬁcacy of TriMap in terms of DR performance measures and runtime.
Revised: We also perform many large-scale experiments on various datasets to show the efﬁcacy of TriMap in terms of DR performance measures and runtime.

Original:  A M EASURE OF G LOBAL A CCURACY
Revised:  A M EASURE OF G LOBAL A CCURACY

Original: Consider the S -curve dataset 2 which consists of 5000 points in 3-D uniformly sampled from an
Revised: Consider the S -curve dataset 2 which consists of 5000 points in 3-D uniformly sampled from an

Original: S -shaped manifold (Figure 1.(a)).
Revised: S -shaped manifold (Figure 1.(a)).

Original: This dataset serves as a paradigmatic problem for evaluating the performance of DR methods.
Revised: This dataset serves as a paradigmatic problem for evaluating the performance of DR methods.

Original: In Figure 1, we show the results of 2-D embeddings of the S -curve dataset using t-SNE, UMAP, TriMap, and PCA.
Revised: In Figure 1, we show the results of 2-D embeddings of the S -curve dataset using t-SNE, UMAP, TriMap, and PCA.

Original: The top of each graph is labeled by the scoring pair (AUC, GS) where GS stands for global score (introduced below).
Revised: The top of each graph is labeled by the scoring pair (AUC, GS) where GS stands for global score (introduced below).

Original: Note that both t-SNE and
Revised: Note that both t-SNE and

Original: UMAP provide higher values of the AUC score and locally preserve the continuity of the manifold.
Revised: UMAP provide higher values of the AUC score and locally preserve the continuity of the manifold.

Original: However, they both fail to recover the global structure of the S -curve, which is naturally reﬂected in the PCA embedding.
Revised: However, they both fail to recover the global structure of the S -curve, which is naturally reﬂected in the PCA embedding.

Original: On the other hand, our TriMap method (formally deﬁned later) successfully recovers the structure of the S -curve by “unveiling” the curved shape of the manifold at both ends.
Revised: On the other hand, our TriMap method (formally deﬁned later) successfully recovers the structure of the S -curve by “unveiling” the curved shape of the manifold at both ends.

Original: Overall, the 2-D TriMap embedding resembles the original 3-D representation as much as possible.
Revised: Overall, the 2-D TriMap embedding resembles the original 3-D representation as much as possible.

Original: Note also that GS is the only measure that can reﬂect the global accuracy of the embedding
Revised: Note also that GS is the only measure that can reﬂect the global accuracy of the embedding.

Original: The previous example indicates that the local measures of DR performance (such as AUC) cannot reﬂect the global accuracy of a low-dimensional embedding.
Revised: The previous example indicates that the local measures of DR performance (such as AUC) cannot reﬂect the global accuracy of a low-dimensional embedding.

Original: In fact, the low-granular structure of the data can only be estimated by considering the global statistics of the dataset, as regarded by the 
Revised: In fact, the low-granular structure of the data can only be estimated by considering the global statistics of the dataset, as regarded by the [Figure] [Figure] [Figure]

Original: PCA method.
Revised: PCA method.

Original: PCA is a linear DR method that projects the high-dimensional data onto the top- d orthogonal directions having the highest variance.
Revised: PCA is a linear DR method that projects the high-dimensional data onto the top- d orthogonal directions having the highest variance.

Original: In order to calculate the mapping, PCA only considers the aggregate statistics of the dataset rather than the local information of each individual[Figure] data point.
Revised: In order to calculate the mapping, PCA only considers the aggregate statistics of the dataset rather than the local information of each individual data point.

Original: As a result, PCA is extremely well suited at retaining the global structure of the data, i.e. the overall shape of the dataset, placement of the clusters, and existence of potential outliers.
Revised: As a result, PCA is extremely well suited at retaining the global structure of the data, i.e. the overall shape of the dataset, placement of the clusters, and existence of potential outliers.

Original: However, by focusing on the global structure, PCA loses much of the local information such as the neighborhood structure of each data point.
Revised: However, by focusing on the global structure, PCA loses much of the local information such as the neighborhood structure of each data point.

Original: Given a low-dimensional mapping produced by PCA, it is possible to calculate an optimal inverse mapping into the original high-dimensional space by means of minimizing the squared error.
Revised: Given a low-dimensional mapping produced by PCA, it is possible to calculate an optimal inverse mapping into the original high-dimensional space by means of minimizing the squared error.

Original: The optimal inverse map also corresponds to a linear mapping 3 .
Revised: The optimal inverse map also corresponds to a linear mapping 3 .

Original: In order to quantify the global accuracy of a DR result, we focus on the accuracy of the embedding in reﬂecting the global structure of the data similar to PCA.
Revised: In order to quantify the global accuracy of a DR result, we focus on the accuracy of the embedding in reﬂecting the global structure of the data similar to PCA.

Original: That is, we consider the minimum reconstruction error of the original dataset by means of a linear inverse map.
Revised: That is, we consider the minimum reconstruction error of the original dataset by means of a linear inverse map.

Revised: Given n data points { x i ∈

Original: Given n data points { x i 2 R m } ni =1 , let X 2 R m ⇥ n denote the high-dimensional data matrix where the i -th column corresponds to x i .
Revised: R m } ni =1 , let X ∈ R m × n denote the high-dimensional data matrix where the i -th column corresponds to x i .

Original: Similarly, let Y 2 R d ⇥ n denote the matrix of the low-dimensional embedding of the points { y i [-
-]
Revised: Similarly, let Y ∈ R d × n denote the matrix of the low-dimensional embedding of the points { y i [-
-]∈ R

Original: R d } ni
Revised: d } ni

Original: =1 .
Revised: =1 .

Original: Without loss of generality, we assume both X and Y are centered.
Revised: Without loss of generality, we assume both X and Y are centered.

Original: We deﬁne the Minimum Reconstruction Error (MRE) from the embedding as
Revised: We deﬁne the Minimum Reconstruction Error (MRE) from the embedding as

Original: [Equation]
Revised: [Equation]

Original: where k · k F denotes the Frobenius norm 4 .
Revised: where (cid:107) · (cid:107) F denotes the Frobenius norm 4 .

Original: Note that PCA has the lowest possible MRE among all the DR methods.
Revised: Note that PCA has the lowest possible MRE among all the DR methods.

Original: Thus, in order to obtain a normalized measure of global accuracy of a given embedding Y for a data X , we deﬁne the global score (GS) as
Revised: Thus, in order to obtain a normalized measure of global accuracy of a given embedding Y for a data X , we deﬁne the global score (GS) as

Original: [Equation]
Revised: [Equation]

Original: where E PCA :
Revised: where E PCA :

Original: = E ( Y PCA | X ) denotes the MRE achieved by the PCA embedding Y PCA on the same dataset X .
Revised: = E ( Y PCA | X ) denotes the MRE achieved by the PCA embedding Y PCA on the same dataset X .

Original: Note that GS ( Y PCA | X ) =
Revised: Note that GS ( Y PCA | X ) =

Original: and we claim that larger values of GS indicate a higher capacity of a DR method to reﬂect the global structure of the data, as shown in the experiments.
Revised: and we claim that larger values of GS indicate a higher capacity of a DR method to reﬂect the global structure of the data, as shown in the experiments.

Original: In the remainder of the paper, we use GS as the global measure of performance.
Revised: In the remainder of the paper, we use GS as the global measure of performance.

Original: Due to the high computational complexity for calculating the trustworthiness-continuity and AUC scores for large data sets, we use nearest-neighbors accuracy as the local measure of performance henceforth.
Revised: Due to the high computational complexity for calculating the trustworthiness-continuity and AUC scores for large data sets, we use nearest-neighbors accuracy as the local measure of performance henceforth.

Original:  T HE T RI M AP M ETHOD
Revised:  T HE T RI M AP M ETHOD

Original: We now formally introduce the TriMap method.
Revised: We now formally introduce the TriMap method.

Original: Recall that a triplet consists of three points ( i, j, k ) where point i is closer to point j than point k .
Revised: Recall that a triplet consists of three points ( i, j, k ) where point i is closer to point j than point k .

Original: TriMap chooses a subset T =
Revised: TriMap chooses a subset T =

Original: { ( i, j, k ) } of tripletsand assigns a weight ! ijk  0 for each triplet: a higher value of ! ijk implies that the pair ( i, k ) is located much farther than the pair ( i, j ) . We deﬁne the loss of the triplet ( i, j, k ) as
Revised: { ( i, j, k ) } of triplets

Original: [Equation]
Revised: [Equation]

Original: ijk
Revised: [Table] This also handles possible rotation and scaling of the embedding.

Original: [Equation]
Revised: [Figure]

Original: , where
Revised: and assigns a weight ω ijk ≥ 0 for each triplet: a higher value of ω ijk implies that the pair ( i, k ) is located much farther than the pair ( i, j ) .

Original: [Equation] [Equation]
Revised: We deﬁne the loss of the triplet ( i, j, k ) as (cid:96) ijk := ω ijk s ( y i , y k ) s

Original: [Figure]
Revised: ( y i , y j ) + s ( y i , y k ) , where s ( y i , y j ) = (cid:0) 1 + | y

Original: [Figure] [Figure] [Figure] is a similarity function between y i and y j .
Revised: i − y j (cid:107) 2 (cid:1) − 1 , is a similarity function between y i and y j .

Original: The choice of s is motivated by the good performance of
Revised: The choice of s is motivated by the good performance of

Original: Student t-distribution for similarities in low-dimension in the t-SNE method.
Revised: Student t-distribution for similarities in low-dimension in the t-SNE method.

Original: Note that the loss of the triplet ( i, j, k ) approaches zero as k y i  y j k decreases and k y i
Revised: Note that the loss of the triplet ( i, j, k ) approaches zero as (cid:107) y i − y j (cid:107) decreases and (cid:107)

Original:  y k k increases.The value
Revised: y i − y k (cid:107) increases.

Original: NIL corresponds to no transformation.
Revised: corresponds to transformation

Original: We ﬁrst develop the weighing scheme for the triplets.
Revised: We ﬁrst develop the weighing scheme for the triplets.

Original: To reﬂect the relative similarities in high-dimension, we deﬁne the unnormalized weight of the triplet ( i, j, k ) as
Revised: To reﬂect the relative similarities in high-dimension, we deﬁne the unnormalized weight of the triplet ( i, j, k ) as

Original: [Equation]
Revised: [Equation]

Original: in which, d ij is any distance measure between x i and x j in highdimension.
Revised: in which, d ij is any distance measure between x i and x j in highdimension.

Original: For Euclidean distances, we use the scaling introduced in (Zelnik-Manor & Perona, 2005),
Revised: For Euclidean distances, we use the scaling introduced in (Zelnik-Manor & Perona, 2005),

Original: [Equation]
Revised: [Equation]

Original: where  ij =  i  j and  i is set to the average Euclidean distance between x i and the set of nearest-neighbors of x i from 4 -th to 6 -th neighbors.
Revised: where σ ij = σ i σ j and σ i is set to the average Euclidean distance between x i and the set of nearest-neighbors of x i from 4 -th to 6 -th neighbors.

Original: This choice of  ij adaptively adjusts the scaling based on the density of the data.
Revised: This choice of σ ij adaptively adjusts the scaling based on the density of the data.

Original: While the choice of weights ˜ ! ijk works well in practice, we adjust the weights further by applying a non-linear transformation that emphasizes the smaller weights.
Revised: While the choice of weights ˜ ω ijk works well in practice, we adjust the weights further by applying a non-linear transformation that emphasizes the smaller weights.

Original: Expanding the values of small weights has the effect of placing the nearest-neighbors closer to the point and pushing the remaining points farther away, thus improving the local accuracy (as shown in Figure 2 and discussed later).
Revised: Expanding the values of small weights has the effect of placing the nearest-neighbors closer to the point and pushing the remaining points farther away, thus improving the local accuracy (as shown in Figure 2 and discussed later).

Original: The ﬁnal value of the weight ! ijk is obtained by applying the  -scaled log-transformation (see Figure 4),
Revised: The ﬁnal value of the weight ω ijk is obtained by applying the γ -scaled log-transformation (see Figure 4),

Original: [Equation]
Revised: [Equation]

Original: in which W = max ( i 0 ,j 0 ,k 0 ) 2 T ˜ !
Revised: in which W = max [Equation] [Equation] ω

Revised: [Equation]

Original: [-i 
-]
Revised: [-i 
-]j

Original: j 0 k 0 ,  > 0 is a scaling factor, and  is a small constant.
Revised: [Equation] k (cid:48) , γ > 0 is a scaling factor, and δ is a small constant.

Original: We use  = 500 and  = 10  4 in all our experiments.
Revised: We use γ = 500 and δ = 10 − 4 in all our experiments.

Original: To construct the embedding, we consider a small subset of all possible triplets ( i, j, k ) for which, the closer point j belongs to the set of nearest-neighbors of the point i and the farther point k is among the points that are more distant from i than j , chosen uniformly at random.
Revised: To construct the embedding, we consider a small subset of all possible triplets ( i, j, k ) for which, the closer point j belongs to the set of nearest-neighbors of the point i and the farther point k is among the points that are more distant from i than j , chosen uniformly at random.

Original: For each point we consider its m = 10 nearest neighbors and sample m 0 = 5 triplets per nearest-neighbor.
Revised: For each point we consider its m = 10 nearest neighbors and sample m (cid:48) = 5 triplets per nearest-neighbor.

Original: This yields m ⇥ m [-
-]
Revised: This yields m × m [-
-](cid:48)

Original: = 50 nearest-neighbor triplets per point.
Revised: = 50 nearest-neighbor triplets per point.

Original: In addition, we also add r = 5 random triplets ( i, j, k ) per each point i where j and k are sampled uniformly at random and their order is possibly switched based on their nearness to i .
Revised: In addition, we also add r = 5 random triplets ( i, j, k ) per each point i where j and k are sampled uniformly at random and their order is possibly switched based on their nearness to i .

Original: This yields m ⇥ m [-
-]
Revised: This yields m × m [-
-](cid:48)

Original: + r = 55 triplets per point in total.
Revised: + r = 55 triplets per point in total.

Original: Thus, the overall complexity of the optimization step is linear in number of points n .
Revised: Thus, the overall complexity of the optimization step is linear in number of points n .

Original: The computational complexity is dominated by the nearest-neighbor search, which is shared among allthe recent methods such as t-SNE, LargeVis, and UMAP. We use ANNOY for the approximate nearest-neighbor search 5 which is based on random projection trees. While a random initialization for the embedding also works well in practice, we initialize the embedding to the PCA solution Y PCA (scaled by a small constant value for better convergence). The PCA
Revised: The computational complexity is dominated by the nearest-neighbor search, which is shared among all

Original: [Table]

Original: [Table]
Revised: [Table]

Original: Runtime of the methods in hh:mm:ss format on single machine with 2.6 GHz Intel Core i5 CPU and 16 GB of memory.
Revised: hh:mm:ss format on single machine with 2.6 GHz Intel Core i5 CPU and 16 GB of memory.

Original: We limit the runtime of each method to 12 hours.
Revised: We limit the runtime of each method to 12 hours.

Original: Also, UMAP runs out of memory on datasets larger than ⇠ 4M points.
Revised: Also, UMAP runs out of memory on datasets larger than ∼ 4M points.

Revised: the recent methods such as t-SNE, LargeVis, and UMAP.

Revised: We use ANNOY for the approximate nearest-neighbor search 5 which is based on random projection trees.

Revised: While a random initialization for the embedding also works well in practice, we initialize the embedding to the PCA solution Y PCA (scaled by a small constant value for better convergence).

Original:  initialization for TriMap allows faster convergence while preserving much of the global structure discovered by PCA.
Revised: The PCA initialization for TriMap allows faster convergence while preserving much of the global structure discovered by PCA.

Original: Note that the other DR methods such as t-SNE are extremely sensitive to the initialization and do not converge well with any initial solution other than small random initialization around the origin.
Revised: Note that the other DR methods such as t-SNE are extremely sensitive to the initialization and do not converge well with any initial solution other than small random initialization around the origin.

Original: We deﬁne the ﬁnal loss as the sum of the losses of the sampled triplets in T
Revised: We deﬁne the ﬁnal loss as the sum of the losses of the sampled triplets in T

Original: [Equation]
Revised: [Equation]

Original: The loss is minimized using the full-batch gradient descent with momentum using the delta-bar-delta method.
Revised: The loss is minimized using the full-batch gradient descent with momentum using the delta-bar-delta method.

Original: In all our experiments, we perform 400 iterations with the value of momentum parameter equal to 0 .
Revised: In all our experiments, we perform 400 iterations with the value of momentum parameter equal to 0 .

Original: 5 during the ﬁrst 250 iterations and 0 .
Revised: 5 during the ﬁrst 250 iterations and 0 .

Original: 8 afterwards.
Revised: 8 afterwards.

Original: Finally, note that there exists connections between TriMap and a number of triplet (aka ordinal) embedding methods such as t-STE (Van Der Maaten & Weinberger, 2012).
Revised: Finally, note that there exists connections between TriMap and a number of triplet (aka ordinal) embedding methods such as t-STE (Van Der Maaten & Weinberger, 2012).

Original: The triplet embedding methods have been developed for a different setting where the goal is to ﬁnd an embedding based on a given pre-speciﬁed set of triplets obtained from human evaluators (or some form of implicit feedback).
Revised: The triplet embedding methods have been developed for a different setting where the goal is to ﬁnd an embedding based on a given pre-speciﬁed set of triplets obtained from human evaluators (or some form of implicit feedback).

Original: For instance, t-STE maximizes the sum of log of the satisfaction probabilities of the triplets to calculate the embedding.
Revised: For instance, t-STE maximizes the sum of log of the satisfaction probabilities of the triplets to calculate the embedding.

Original: It is worth mentioning that TriMap is a DR method that is designed to sample the informative triplets from the high-dimensional representation of a set of points and assign weights to these triplets to reﬂect the relative similarities of these points.
Revised: It is worth mentioning that TriMap is a DR method that is designed to sample the informative triplets from the high-dimensional representation of a set of points and assign weights to these triplets to reﬂect the relative similarities of these points.

Original: Although
Revised: Although

Original: TriMap can also be used for the triplet embedding task, we only focus on the DR results 6 .
Revised: TriMap can also be used for the triplet embedding task, we only focus on the DR results 6 .

Original: 3. E FFECT OF D IFFERENT P ARAMETERS
Revised: 3. E FFECT OF D IFFERENT P ARAMETERS

Original: We brieﬂy discuss the effect of different parameters, namely the total number of triplets |T | and the  -scaled log-transformation, on the quality of the embedding.
Revised: We brieﬂy discuss the effect of different parameters, namely the total number of triplets |T | and the γ -scaled log-transformation, on the quality of the embedding.

Original: TriMap is particularly robust to the number of sampled triplet for constructing the embedding.
Revised: TriMap is particularly robust to the number of sampled triplet for constructing the embedding.

Original: This can be explained by the high amount of redundancy in the triplets (the triplets ( i, j, k ) and ( i, j, k 0 ) convey the same information if k and k 0 are nearest neighbors and also mapped nearby).
Revised: This can be explained by the high amount of redundancy in the triplets (the triplets ( i, j, k ) and ( i, j, k (cid:48) ) convey the same information if k and k (cid:48) are nearest neighbors and also mapped nearby).

Original: In Figure 3 we consider various values for m , m 0 , and r for the MNIST dataset while ﬁxing the remaining parameters.
Revised: In Figure 3 we consider various values for m , m (cid:48) , and r for the MNIST dataset while ﬁxing the remaining parameters.

Original: In fact, using large number of triplets can sometimes introduce an overhead and require larger number of iterations to converge.
Revised: In fact, using large number of triplets can sometimes introduce an overhead and require larger number of iterations to converge.

Original: A more important parameter is  which controls the trade-off between the local and global accuracy.
Revised: A more important parameter is γ which controls the trade-off between the local and global accuracy.

Original: Larger values of  increases the relative importance of triplets with smaller weights.
Revised: Larger values of γ increases the relative importance of triplets with smaller weights.

Original: This causes the method to focus on the nearest-neighbor points rather than the points that are far away, thus improving the local accuracy.
Revised: This causes the method to focus on the nearest-neighbor points rather than the points that are far away, thus improving the local accuracy.

Original: On the other hand, improving the local accuracy can impair the global accuracy.
Revised: On the other hand, improving the local accuracy can impair the global accuracy.

Original: In Figure 2, we plot the  -scaled log-transformation for various  values and illustrate the results on MNIST without the log-transformation as well as the results with different  values.
Revised: In Figure 2, we plot the γ -scaled log-transformation for various γ values and illustrate the results on MNIST without the log-transformation as well as the results with different γ values.

Original: For larger values of  the clusters tend to become more compressed and as a result, the nearest-neighbor accuracy is improved.
Revised: For larger values of γ the clusters tend to become more compressed and as a result, the nearest-neighbor accuracy is improved.

Original: On the other hand, the global score starts to decrease for larger  values.
Revised: On the other hand, the global score starts to decrease for larger γ values.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we apply TriMap on a set of real-world as well as synthetic datasets and compare the results to t-SNE, LargeVis, UMAP, and PCA methods.
Revised: In this section, we apply TriMap on a set of real-world as well as synthetic datasets and compare the results to t-SNE, LargeVis, UMAP, and PCA methods.

Original: The datasets used in our experiments are listed in Table 1 and a short description is given in the appendix.
Revised: The datasets used in our experiments are listed in Table 1 and a short description is given in the appendix.

Original: All experiments are conducted on a single machine with 2.6 GHz Intel Core i5 CPU and 16 GB of memory.
Revised: All experiments are conducted on a single machine with 2.6 GHz Intel Core i5 CPU and 16 GB of memory.

Original: We limit the runtime of each algorithm to 12 hours.
Revised: We limit the runtime of each algorithm to 12 hours.

Original: For implementations, we use the default sklearn implementation for t-SNE and the ofﬁcial implementations of LargeVis and UMAP provided by the authors 7,8 .
Revised: For implementations, we use the default sklearn implementation for t-SNE and the ofﬁcial implementations of LargeVis and UMAP provided by the authors 7,8 .

Original: Due to lack of space, we provide the comparison to the LargeVis results as well additional TriMap results on the larger datasets in the appendix.
Revised: Due to lack of space, we provide the comparison to the LargeVis results as well additional TriMap results on the larger datasets in the appendix.

Original: In order to have a fair comparison, we use the default parameter values for all methods, including ours ( m = 10 , m 0 = 5 , r = 5 ,  = 500 , and 400 iterations).
Revised: In order to have a fair comparison, we use the default parameter values for all methods, including ours ( m = 10 , m (cid:48) = 5 , r = 5 , γ = 500 , and 400 iterations).

Original: Also to reduce the overhead induced by the dimensionality of the data in the nearest-neighbor search step, we reduce the number of dimensions of the dataset to 100 if necessary, using the PCA method.
Revised: Also to reduce the overhead induced by the dimensionality of the data in the nearest-neighbor search step, we reduce the number of dimensions of the dataset to 100 if necessary, using the PCA method.

Original: To evaluate the local performance, we show the nearest-neighbor accuracy of each result.
Revised: To evaluate the local performance, we show the nearest-neighbor accuracy of each result.

Original: We also show the GS as a measure of global performance.
Revised: We also show the GS as a measure of global performance.

Original: The performance measures are shown on top of each ﬁgure as a pair (NN, GS) .
Revised: The performance measures are shown on top of each ﬁgure as a pair (NN, GS) .

Original: 4. R UNTIME
Revised: 4. R UNTIME

Original: The runtime of the methods are provided in Table 1 in the hh:mm:ss format.
Revised: The runtime of the methods are provided in Table 1 in the hh:mm:ss format.

Original: We limit the runtime of each method to 12 hours.
Revised: We limit the runtime of each method to 12 hours.

Original: As can be seen from the results, TriMap provides excellent runtime and outperforms all the other methods in most cases.
Revised: As can be seen from the results, TriMap provides excellent runtime and outperforms all the other methods in most cases.

Original: Also, TriMap easily scales to millions of points while the other methods exceed the time limit or run out of memory.
Revised: Also, TriMap easily scales to millions of points while the other methods exceed the time limit or run out of memory.

Original: For instance, UMAP causes an out of memory error for datasets larger than ⇠ 4M points.
Revised: For instance

Original: 4. V ISUALIZATIONS
Revised: V ISUALIZATIONS

Original: The visualizations of the datasets using TriMap as well as the other competing methods are shown in Figure 5 and 6.
Revised: The visualizations of the datasets using TriMap as well as the other competing methods are shown in Figure 5 and 6.

Original: For some results, we provide a zoomed in snippet over the main ﬁgure to provide a more detailed illustration.
Revised: For some results, we provide a zoomed in snippet over the main ﬁgure to provide a more detailed illustration.

Original: Overall, TriMap preserves the underlying global structure of the data better than the other competing methods.
Revised: Overall, TriMap preserves the underlying global structure of the data better than the other competing methods.

Original: This is reﬂected by the larger GS values for TriMap as well as visually comparing the embeddings to the PCA result.
Revised: This is reﬂected by the larger GS values for TriMap as well as visually comparing the embeddings to the PCA result.

Original: For example, TriMap recovers the continuous structure of the TV news dataset and separates the remaining outliers in the data which are also identiﬁed by the PCA method.
Revised: For example, TriMap recovers the continuous structure of the TV news dataset and separates the remaining outliers in the data which are also identiﬁed by the PCA method.

Original: This can be veriﬁed by comparing the placement of an example outlier point, marked with a red ⇥ , by the different methods: TriMap shows this point among other outliers whereas t-SNE and UMAP fail to uncover this information.
Revised: This can be veriﬁed by comparing the placement of an example outlier point, marked with a red × , by the different methods: TriMap shows this point among other outliers whereas t-SNE and UMAP fail to uncover this information.

Original: Also, the global score of TriMap on this dataset is much higher than the other methods.
Revised: Also, the global score of TriMap on this dataset is much higher than the other methods.

Original: Further discussion is given in the appendix.
Revised: Further discussion is given in the appendix.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: TriMap is a fast and efﬁcient method that can be easily applied to large datasets.
Revised: TriMap is a fast and efﬁcient method that can be easily applied to large datasets.

Original: While TriMap is extremely effective for uncovering the global structure of the data, other methods such as t-SNE can provide additional insight about the local neighborhood of individual points.
Revised: While TriMap is extremely effective for uncovering the global structure of the data, other methods such as t-SNE can provide additional insight about the local neighborhood of individual points.

Original: As a future research direction, we consider using pairwise constraints along with triplet constraints to improve the local accuracy.
Revised: As a future research direction, we consider using pairwise constraints along with triplet constraints to improve the local accuracy.

Original: The current implementation of TriMap utilizes a single core.
Revised: The current implementation of TriMap utilizes a single core.

Original: Parallel implementation of the method that can exploit multiple cores is another future direction.
Revised: Parallel implementation of the method that can exploit multiple cores is another future direction.

Original: Furthermore, the global accuracy is measured in terms of the global score which is based on the assumption that linear projection obtained by PCA is globally optimal.
Revised: Furthermore, the global accuracy is measured in terms of the global score which is based on the assumption that linear projection obtained by PCA is globally optimal.

Original: While our global score can provide insight about the global accuracy of the embedding in many cases, it appears to be ineffective when the data is highly non-linear or contains a large amount of outliers.
Revised: While our global score can provide insight about the global accuracy of the embedding in many cases, it appears to be ineffective when the data is highly non-linear or contains a large amount of outliers.

Original: Developing non-linear and more robust global performance measures could signiﬁcantly improve the assessment of the DR results and provide guidelines for developing more accurate DR techniques.
Revised: Developing non-linear and more robust global performance measures could signiﬁcantly improve the assessment of the DR results and provide guidelines for developing more accurate DR techniques.

Original: A D ATASETS
Revised: A D ATASETS

Original: The datasets used in the experiments are listed below.
Revised: The datasets used in the experiments are listed below.

Original: All datasets are publicly available online and a download link is provided.
Revised: All datasets are publicly available online and a download link is provided.

Original: • COIL-20 9 ( 1440 ): gray-scale images of 20 objects in uniformly sampled orientations (degrees of rotation, 72 images per object).
Revised: • COIL-20 9 ( 1440 ): gray-scale images of 20 objects in uniformly sampled orientations (degrees of rotation, 72 images per object).

Original: Each image is pre-processed by having the background removed and cropped into size 128 ⇥ 128 .
Revised: Each image is pre-processed by having the background removed and cropped into size 128 × 128 .

Original: • USPS 10 (11K): images of handwritten digits (0–9) of size 16 ⇥ 16 .
Revised: • USPS 10 (11K): images of handwritten digits (0–9) of size 16 × 16 .

Original: • Epileptic Seizure 11 (11.5K): EEG signal recordings of brain activity for seizure recognition.
Revised: • Epileptic Seizure 11 (11.5K): EEG signal recordings of brain activity for seizure recognition.

Original: It contains 178 -dimensional vectors belonging to 5 categories.
Revised: It contains 178 -dimensional vectors belonging to 5 categories.

Original: • 20 Newsgroup 11 (18K): newsgroup posts categorized into 20 topics.
Revised: • 20 Newsgroup 11 (18K): newsgroup posts categorized into 20 topics.

Original: We use a TF-IDF representation of the words in each document as the features.
Revised: We use a TF-IDF representation of the words in each document as the features.

Original: • Tabula Muris 12 ( ⇠ 54K): single cell transcriptome data from the mouse from 20 organs.
Revised: • Tabula Muris 12 ( ∼ 54K): single cell transcriptome data from the mouse from 20 organs.

Original: • MNIST 13 (70K): images of handwritten digits (0–9) of size 28 ⇥ 28 .
Revised: • MNIST 13 (70K): images of handwritten digits (0–9) of size 28 × 28 .

Original: • Fashion MNIST 14 (70K): gray-scale images of clothing items such as t-shirt, pullover, bag, etc. of size 28 ⇥ 28 .
Revised: • Fashion MNIST 14 (70K): gray-scale images of clothing items such as t-shirt, pullover, bag, etc. of size 28 × 28 .

Original: • TV News 11 ( ⇠ 129K): audio-visual features from TV news broadcast categorized into commercial and non-commercial.
Revised: • TV News 11 ( ∼ 129K): audio-visual features from TV news broadcast categorized into commercial and non-commercial.

Original: • 360K+ Lyrics 15 ( ⇠ 362K): lyrics of songs from 12 different genres.
Revised: • 360K+ Lyrics 15 ( ∼ 362K): lyrics of songs from 12 different genres.

Original: We group similar genres together (metal-rock, R&B-pop, etc.) to form 7 groups.
Revised: We group similar genres together (metal-rock, R&B-pop, etc.) to form 7 groups.

Original: We use the TF-IDF representation of the words in the song as the features.
Revised: We use the TF-IDF representation of the words in the song as the features.

Original: • Covertype 11 ( ⇠ 581K): cartographic features for forest cover type prediction.
Revised: • Covertype 11 ( ∼ 581K): cartographic features for forest cover type prediction.

Original: • RCV1 16 (800K): Reuters Corpus Volume I archive of categorized newswire stories.
Revised: • RCV1 16 (800K): Reuters Corpus Volume I archive of categorized newswire stories.

Original: • Character Font Images 11 ( ⇠ 1.7M): images of character from scanned and computer generated fonts.
Revised: • Character Font Images 11 ( ∼ 1.7M): images of character from scanned and computer generated fonts.

Original: • KDDCup99 11 ( ⇠ 4.9M): computer network intrusion detection.
Revised: • KDDCup99 11 ( ∼ 4.9M): computer network intrusion detection.

Original: • HIGGS 11 (11M): Higgs bosons recognition from a background process.
Revised: • HIGGS 11 (11M): Higgs bosons recognition from a background process.

Original: B M ORE V ISUALIZATIONS
Revised: B M ORE V ISUALIZATIONS

Original: We compare the results of TriMap to LargeVis in Figure 7 and 8.
Revised: We compare the results of TriMap to LargeVis in Figure 7 and 8.

Original: We also provide more visualizations obtained using TriMap in Figure 9.
Revised: We also provide more visualizations obtained using TriMap in Figure 9.

Original: C D ISCUSSION
Revised: C D ISCUSSION

Original: We brieﬂy discuss the results of TriMap and draw a comparison to the other methods.
Revised: We brieﬂy discuss the results of TriMap and draw a comparison to the other methods.

Original: TriMap generally provides better global accuracy compared to the competing methods.
Revised: TriMap generally provides better global accuracy compared to the competing methods.

Original: It also successfully maintains the continuity of the underlying manifold.
Revised: It also successfully maintains the continuity of the underlying manifold.

Original: This can be seen from the COILresult where certain clusters are located farther away from the remaining clusters.
Revised: This can be seen from the COILresult where certain clusters are located farther away from the remaining clusters.

Original: However, the underlying structure for the main cluster resembles the one provided by the other methods.
Revised: However, the underlying structure for the main cluster resembles the one provided by the other methods.

Original: TriMap also preserves the continuous structure in the Fashion MNIST and the TV News datasets.
Revised: TriMap also preserves the continuous structure in the Fashion MNIST and the TV News datasets.

Original: TriMap is also efﬁcient in uncovering the possible outliers in the data.
Revised: TriMap is also efﬁcient in uncovering the possible outliers in the data.

Original: For instance, PCA reveals a large number of outliers in the Tabula Muris and the 360+K Lyrics datasets.
Revised: For instance, PCA reveals a large number of outliers in the Tabula Muris and the 360+K Lyrics datasets.

Original: These outliers are located far away from the main clusters in the TriMap results.
Revised: These outliers are located far away from the main clusters in the TriMap results.

Original: However, the same points are located very close to the remaining points in the t-SNE results.
Revised: However, the same points are located very close to the remaining points in the t-SNE results.

Original: Additionally, both t-SNE and LargeVis tend to form spurious clusters by splitting the underlying connected manifold.
Revised: Additionally, both t-SNE and LargeVis tend to form spurious clusters by splitting the underlying connected manifold.

Original: This can be seen from the TV News results and the result of LargeVis on the
Revised: This can be seen from the TV News results and the result of LargeVis on the

Original: Covertype dataset.
Revised: Covertype dataset.

Original: Finally, notice that in some cases GS fails to reﬂect the global accuracy of the embeddings.
Revised: Finally, notice that in some cases GS fails to reﬂect the global accuracy of the embeddings.

Original: This can be seen from the low GS values for all methods on the Covertype dataset.
Revised: This can be seen from the low GS values for all methods on the Covertype dataset.

Original: GS may become uninformative when there exists a high degree of non-linearity in the data that cannot be reﬂected using PCA.
Revised: GS may become uninformative when there exists a high degree of non-linearity in the data that cannot be reﬂected using PCA.

Original: GS also cannot reﬂect the accuracy of the embedding in uncovering single outliers.
Revised: GS also cannot reﬂect the accuracy of the embedding in uncovering single outliers.

Original: Developing more accurate global measures for these scenarios is a future research direction.
Revised: Developing more accurate global measures for these scenarios is a future research direction.