Original: P ARAMETRIC UMAP: LEARNING EMBEDDINGS WITH DEEP NEURAL NETWORKS FOR REPRESENTATION AND SEMI - SUPERVISED LEARNING
Revised: P ARAMETRIC UMAP: LEARNING EMBEDDINGS WITH DEEP NEURAL NETWORKS FOR REPRESENTATION AND SEMI - SUPERVISED LEARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose Parametric UMAP, a parametric variation of the UMAP (Uniform
Revised: We propose Parametric UMAP, a parametric variation of the UMAP (Uniform

Original: Manifold Approximation and Projection) algorithm.
Revised: Manifold Approximation and Projection) algorithm.

Original: UMAP is a non-parametric graph-based dimensionality reduction algorithm using applied Riemannian geometry and algebraic topology to ﬁnd low-dimensional embeddings of structured data.
Revised: UMAP is a non-parametric graph-based dimensionality reduction algorithm using applied Riemannian geometry and algebraic topology to ﬁnd low-dimensional embeddings of structured data.

Original: The UMAP algorithm consists of two steps: (1) Compute a graphical representation of a dataset (fuzzy simplicial complex) and (2) Through stochastic gradient descent, optimize a low-dimensional embedding of the graph.
Revised: The UMAP algorithm consists of two steps: (1) Compute a graphical representation of a dataset (fuzzy simplicial complex) and (2) Through stochastic gradient descent, optimize a low-dimensional embedding of the graph.

Original: Here, we replace the second step of UMAP with a deep neural network that learns a parametric relationship between data and embedding.
Revised: Here, we replace the second step of UMAP with a deep neural network that learns a parametric relationship between data and embedding.

Original: We demonstrate that our method performs similarly to its non-parametric counterpart while conferring the beneﬁt of a learned parametric mapping (e g. fast online embeddings for new data).
Revised: We demonstrate that our method performs similarly to its non-parametric counterpart while conferring the beneﬁt of a learned parametric mapping (e g. fast online embeddings for new data).

Original: We then show that UMAP loss can be extended to arbitrary deep learning applications, for example constraining the latent distribution of autoencoders, and improving classiﬁer accuracy for semi-supervised learning by capturing structure in unlabeled data.
Revised: We then show that UMAP loss can be extended to arbitrary deep learning applications, for example constraining the latent distribution of autoencoders, and improving classiﬁer accuracy for semi-supervised learning by capturing structure in unlabeled data.

Original:  U NDERSTANDING UMAP AS A PARAMETRIC AND NON - PARAMETRIC DIMENSIONALITY REDUCTION ALGORITHM
Revised:  U NDERSTANDING UMAP AS A PARAMETRIC AND NON - PARAMETRIC DIMENSIONALITY REDUCTION ALGORITHM

Original: Current non-linear dimensionality reduction algorithms can be divided broadly into non-parametric algorithms which rely on the efﬁcient computation of probabilistic relationships from neighborhood graphs to extract structure in large datasets (e.g. UMAP (McInnes et al., 2018), t-SNE (van der
Revised: Current non-linear dimensionality reduction algorithms can be divided broadly into non-parametric algorithms which rely on the efﬁcient computation of probabilistic relationships from neighborhood graphs to extract structure in large datasets (e.g. UMAP (McInnes et al., 2018), t-SNE (van der

Original: Maaten & Hinton, 2008), LargeVis (Tang et al., 2016)), and parametric algorithms, which, driven by advances in deep-learning, optimize an objective function related to capturing structure in a dataset over neural network weights (e.g. Hinton & Salakhutdinov 2006; Ding et al.
Revised: Maaten & Hinton, 2008), LargeVis (Tang et al., 2016)), and parametric algorithms, which, driven by advances in deep-learning, optimize an objective function related to capturing structure in a dataset over neural network weights (e.g. Hinton & Salakhutdinov 2006; Ding et al.

Original: ; Ding &
Revised: ; Ding &

Original: Regev 2019; Szubert et al. 2019; Kingma & Welling 2013).
Revised: Regev 2019; Szubert et al. 2019; Kingma & Welling 2013).

Original: The goal of this paper is to wed those two classes of methods: learning a structured graphical representation of the data and using a deep neural network to embed that graph.
Revised: The goal of this paper is to wed those two classes of methods: learning a structured graphical representation of the data and using a deep neural network to embed that graph.

Original: Over the past decade several varients of the t-SNE algorithm have proposed parameterized forms of t-SNE (Van
Revised: Over the past decade several varients of the t-SNE algorithm have proposed parameterized forms of t-SNE (Van

Original: Der Maaten, 2009; Gisbrecht et al., 2015; Bunte et al., 2012; Gisbrecht et al., 2012).
Revised: Der Maaten, 2009; Gisbrecht et al., 2015; Bunte et al., 2012; Gisbrecht et al., 2012).

Original: In particular,
Revised: In particular,

Original: Parametric t-SNE (Van Der Maaten, 2009) performs exactly that wedding; training a deep neural network to minimize loss over a t-SNE graph.
Revised: Parametric t-SNE (Van Der Maaten, 2009) performs exactly that wedding; training a deep neural network to minimize loss over a t-SNE graph.

Original: However, the t-SNE loss function itself is not well suited to be optimized over deep neural networks using contemporary training schemes.
Revised: However, the t-SNE loss function itself is not well suited to be optimized over deep neural networks using contemporary training schemes.

Original: In particular, t-SNE’s optimization requires normalization over the entire dataset at each step of optimization, making batch-based optimization and on-line learning of large datasets difﬁcult.
Revised: In particular, t-SNE’s optimization requires normalization over the entire dataset at each step of optimization, making batch-based optimization and on-line learning of large datasets difﬁcult.

Original: In contrast, UMAP is optimized using negative sampling (Mikolov et al., 2013) and requires no normalization step, making it more well-suited to deep learning applications.
Revised: In contrast, UMAP is optimized using negative sampling (Mikolov et al., 2013) and requires no normalization step, making it more well-suited to deep learning applications.

Original: In the following section we broadly outline the algorithm underlying UMAP to explain why our proposed algorithm, Parametric UMAP, is particularly well suited to deep learning applications.
Revised: In the following section we broadly outline the algorithm underlying UMAP to explain why our proposed algorithm, Parametric UMAP, is particularly well suited to deep learning applications.

Original: We contextualize our discussion of UMAP in t-SNE, to outline the advantages that UMAP confers over t-SNE in the domain of parametric neural-network based embedding.
Revised: We contextualize our discussion of UMAP in t-SNE, to outline the advantages that UMAP confers over t-SNE in the domain of parametric neural-network based embedding.

Original: We then perform experiments comparing our algorithm, Parametric UMAP, to parametric and non-parametric algorithms.
Revised: We then perform experiments comparing our algorithm, Parametric UMAP, to parametric and non-parametric algorithms.

Original: Finally, we show a novel extension of Parametric UMAP to semisupervised learning.
Revised: Finally, we show a novel extension of Parametric UMAP to semisupervised learning.

Original: 1. G RAPH CONSTRUCTION AND EMBEDDING
Revised: 1. G RAPH CONSTRUCTION AND EMBEDDING

Original: UMAP and t-SNE have the same goal:
Revised: UMAP and t-SNE have the same goal:

Original: Given a D -dimensional data set X ∈ R D , produce a d dimensional embedding Z ∈ R d such that points that are close together in X (e.g. x i and x j ) are also close together in Z ( z i and z j ).
Revised: Given a D -dimensional data set X ∈ R D , produce a d dimensional embedding Z ∈ R d such that points that are close together in X (e.g. x i and x j ) are also close together in Z ( z i and z j ).

Original: Both algorithms are comprised of the same two broad steps, ﬁrst construct a graph of local relationships between datasets (Fig 1A), then optimize an embedding in low dimensional space which preserves the structure of the graph (Fig 1B).
Revised: Both algorithms are comprised of the same two broad steps, ﬁrst construct a graph of local relationships between datasets (Fig 1A), then optimize an embedding in low dimensional space which preserves the structure of the graph (Fig 1B).

Original: The parametric approach replaces the second step of this process with an optimization of the parameters of a deep neural network (Fig 1C).
Revised: The parametric approach replaces the second step of this process with an optimization of the parameters of a deep neural network (Fig 1C).

Original: 1.1. G RAPH C ONSTRUCTION Computing probabilities in X
Revised: 1.1. G RAPH C ONSTRUCTION Computing probabilities in X

Original: The ﬁrst step in both UMAP and t-SNE is to compute a distribution of probabilities P between pairs of points in X based upon the distances between points in data space.
Revised: The ﬁrst step in both UMAP and t-SNE is to compute a distribution of probabilities P between pairs of points in X based upon the distances between points in data space.

Original: Probabilities are initially computed as local, one-directional, probabilities between a point and its neighbors in data-space, then symmetrized to yield a ﬁnal probability representing the relationship between pairs of points.
Revised: Probabilities are initially computed as local, one-directional, probabilities between a point and its neighbors in data-space, then symmetrized to yield a ﬁnal probability representing the relationship between pairs of points.

Original: In t-SNE, these probabilities are treated as conditional probabilities of neighborhood ( p t-SNE i | j ) computed using a Gaussian distribution centered at x i .
Revised: In t-SNE, these probabilities are treated as conditional probabilities of neighborhood ( p t-SNE i | j ) computed using a Gaussian distribution centered at x i .

Original: [Equation]
Revised: [Equation]

Original: Where d ( x i , x j ) represents the distance between x i an x
Revised: Where d ( x i , x j ) represents the distance between x i an x

Original: i (e.g. Euclidean distance) and σ i is the standard deviation for the Gaussian distribution, set based upon the a perplexity parameter such that one standard deviation of the Gaussian kernel ﬁts a a set number of nearest-neighbors in X .
Revised: i (e.g. Euclidean distance) and σ i is the standard deviation for the Gaussian distribution, set based upon the a perplexity parameter such that one standard deviation of the Gaussian kernel ﬁts a a set number of nearest-neighbors in X .

Original: In UMAP, local, one-directional, probabilities ( P UMAP i | j ) are computed between a point and its neighbors to determine the probability with which an edge (or simplex exists), based upon an assumption that data is uniformly distributed across a manifold in a warped dataspace.
Revised: In UMAP, local, one-directional, probabilities ( P UMAP i | j ) are computed between a point and its neighbors to determine the probability with which an edge (or simplex exists), based upon an assumption that data is uniformly distributed across a manifold in a warped dataspace.

Original: Under this assumption, a local notion of distance is set by the distance to the k th nearest neighbor and the local probability is scaled by that local notion of distance.
Revised: Under this assumption, a local notion of distance is set by the distance to the k th nearest neighbor and the local probability is scaled by that local notion of distance.

Original: [Equation]
Revised: [Equation]

Original: Where ρ i is a local connectivity parameter set to the distance from x i to its nearest neighbor, and σ i is a local connectivity parameter set to match the local distance around x i upon its k nearest neighbors (where k is a hyperparameter).
Revised: Where ρ i is a local connectivity parameter set to the distance from x i to its nearest neighbor, and σ i is a local connectivity parameter set to match the local distance around x i upon its k nearest neighbors (where k is a hyperparameter).

Original: After computing the one-directional edge probabilities for each datapoint, UMAP computes a global probability as the probability of either of the two local, one-directional, probabilities occurring p ij = (cid:0) p
Revised: After computing the one-directional edge probabilities for each datapoint, UMAP computes a global probability as the probability of either of the two local, one-directional, probabilities occurring p ij = (cid:0) p

Original: j | i
Revised: j | i

Original: + p i | j (cid:1) − p
Revised: + p i | j (cid:1) − p

Original: j | i p i | j .
Revised: j | i p i | j .

Original: In contrast, t-SNE symmetrizes the conditional probabilities as p ij = p
Revised: In contrast, t-SNE symmetrizes the conditional probabilities as p ij = p

Original: j | i
Revised: j | i

Original: + p
Revised: + p

Original: i | j 2 N .
Revised: i | j 2 N .

Original: G RAPH E MBEDDING
Revised: G RAPH E MBEDDING

Original: After constructing a distribution of probabilistically weighted edges between points in X , UMAP and t-SNE initialize an embedding in Z corresponding to each data point, where a probability distribution ( Q ) is computed between points as was done with the distribution ( P ) in the input space.
Revised: After constructing a distribution of probabilistically weighted edges between points in X , UMAP and t-SNE initialize an embedding in Z corresponding to each data point, where a probability distribution ( Q ) is computed between points as was done with the distribution ( P ) in the input space.

Original: The objective of the UMAP and t-SNE is then to optimize that embedding to minimize the difference between P and Q .
Revised: The objective of the UMAP and t-SNE is then to optimize that embedding to minimize the difference between P and Q .

Original: Computing probabilities in Z
Revised: Computing probabilities in Z

Original: In embedding space, the pairwise probabilities are computed directly without ﬁrst computing local, one-directional probabilities.
Revised: In embedding space, the pairwise probabilities are computed directly without ﬁrst computing local, one-directional probabilities.

Original: In the t-SNE embedding space, the pairwise probability between two points q t-SNE i | j is computed in a similar manner to p t-SNE i | j , but where the Gaussian distribution is replaced with the fatter-tailed
Revised: In the t-SNE embedding space, the pairwise probability between two points q t-SNE i | j is computed in a similar manner to p t-SNE i | j , but where the Gaussian distribution is replaced with the fatter-tailed

Original: Student’s t-distribution (with one degree of freedom), which is used to overcome the ’crowding problem’ (van der Maaten & Hinton, 2008) in translating volume differences in high-dimensional spaces to low-dimensional spaces:
Revised: Student’s t-distribution (with one degree of freedom), which is used to overcome the ’crowding problem’ (van der Maaten & Hinton, 2008) in translating volume differences in high-dimensional spaces to low-dimensional spaces:

Original: [Equation]
Revised: [Equation]

Original: UMAP’s computation of the pairwise probability q UMAP ij between points in the embedding space Z uses a different family of functions:
Revised: UMAP’s computation of the pairwise probability q UMAP ij between points in the embedding space Z uses a different family of functions:

Original: [Equation]
Revised: [Equation]

Original: Where a and b are hyperparameters set based upon a desired minimum distance between points in embedding space.
Revised: Where a and b are hyperparameters set based upon a desired minimum distance between points in embedding space.

Original: Notably, the UMAP probability distribution in embedding space is not normalized, while the t-SNE distribution is normalized across the entire distribution of probabilities, meaning that the entire distribution of probabilities needs to be calculated before each optimization step of t-SNE.
Revised: Notably, the UMAP probability distribution in embedding space is not normalized, while the t-SNE distribution is normalized across the entire distribution of probabilities, meaning that the entire distribution of probabilities needs to be calculated before each optimization step of t-SNE.

Original: Cost function Finally the distribution of embeddings in Z is optimized to minimize the difference betwee Q and P .
Revised: Cost function Finally the distribution of embeddings in Z is optimized to minimize the difference betwee Q and P .

Original: In t-SNE, a Kullback-Leibler divergence between the two probability distributions is used, and gradient descent in t-SNE is computed over the embeddings C t-SNE = (cid:80) i (cid:54) =
Revised: In t-SNE, a Kullback-Leibler divergence between the two probability distributions is used, and gradient descent in t-SNE is computed over the embeddings C t-SNE = (cid:80) i (cid:54) =

Original: j p ij log p ij − p ij log q ij .
Revised: j p ij log p ij − p ij log q ij .

Original: In UMAP, the cost function is cross-entropy, also optimized using gradient descent C UMAP =
Revised: In UMAP, the cost function is cross-entropy, also optimized using gradient descent C UMAP =

Original: (cid:80) i (cid:54) =
Revised: (cid:80) i (cid:54) =

Original: j p ij log (cid:16) p
Revised: j p ij log (cid:16) p

Original: [Equation]
Revised: [Equation]

Original: (cid:17) + (1 − p ij ) log (cid:16)
Revised: (cid:17) + (1 − p ij ) log (cid:16)

Original: [Equation]
Revised: [Equation]

Original: − q ij (cid:17) .
Revised: − q ij (cid:17) .

Original: Attraction and repulsion Minimizing the cost function over every possible pair of points in the dataset would be computationally expensive.
Revised: Attraction and repulsion Minimizing the cost function over every possible pair of points in the dataset would be computationally expensive.

Original: UMAP and more recent varients of t-SNE both use shortcuts to bypass much of that computation.
Revised: UMAP and more recent varients of t-SNE both use shortcuts to bypass much of that computation.

Original: In UMAP, those shortcuts are directly advantageous to batch-wise training in a neural network.
Revised: In UMAP, those shortcuts are directly advantageous to batch-wise training in a neural network.

Original: The primary intuition behind these shortcuts is that the cost function of both t-SNE and UMAP can both be broken out into a mixture of attractive forces between locally connected embeddings and repulsive forces between non-locally connected embeddings.
Revised: The primary intuition behind these shortcuts is that the cost function of both t-SNE and UMAP can both be broken out into a mixture of attractive forces between locally connected embeddings and repulsive forces between non-locally connected embeddings.

Original: Attractive forces Both UMAP and t-SNE utilize a similar strategy in minimizing the computational cost over attractive forces: they rely on an approximate nearest neighbors graph 1 .
Revised: Attractive forces Both UMAP and t-SNE utilize a similar strategy in minimizing the computational cost over attractive forces: they rely on an approximate nearest neighbors graph 1 .

Original: The intuition for this approach is that elements that are further apart in data space have very small edge probabilities, which can be treated as effectively zero.
Revised: The intuition for this approach is that elements that are further apart in data space have very small edge probabilities, which can be treated as effectively zero.

Original: Thus, edge probabilities and attractive forces only need to be computed over the nearest neighbors, non-nearest neighbors can be treated as having an edge-probability of zero.
Revised: Thus, edge probabilities and attractive forces only need to be computed over the nearest neighbors, non-nearest neighbors can be treated as having an edge-probability of zero.

Original: Because nearest-neighbor graphs are themselves computationally expensive, approximate nearest neighbors (e.g. Dong et al.
Revised: Because nearest-neighbor graphs are themselves computationally expensive, approximate nearest neighbors (e.g. Dong et al.

Original: 2011) produce effectively similar results.
Revised: 2011) produce effectively similar results.

Original: Repulsive forces Because most data points are not locally connected, we do not need to waste computation on most pairs of embeddings.
Revised: Repulsive forces Because most data points are not locally connected, we do not need to waste computation on most pairs of embeddings.

Original: UMAP takes a shortcut motivated by the language model
Revised: UMAP takes a shortcut motivated by the language model

Original: word2vec
Revised: word2vec

Original: (Mikolov et al., 2013) and performs negative sampling over embeddings.
Revised: (Mikolov et al., 2013) and performs negative sampling over embeddings.

Original: Each training step iterates over positive, locallyconnected, edges and randomly samples edges from the remainder of the dataset treating their edge probabilities as zero to compute cross-entropy.
Revised: Each training step iterates over positive, locallyconnected, edges and randomly samples edges from the remainder of the dataset treating their edge probabilities as zero to compute cross-entropy.

Original: Because most data points are not locally connected and have a very low edge probability, these negative samples are, on average, correct, allowing
Revised: Because most data points are not locally connected and have a very low edge probability, these negative samples are, on average, correct, allowing

Original: UMAP to sample only sparsely over edges in the dataset.
Revised: UMAP to sample only sparsely over edges in the dataset.

Original: In t-SNE, repulsion is derived from the normalization of Q .
Revised: In t-SNE, repulsion is derived from the normalization of Q .

Original: A few methods for minimizing the amount of computation needed for repulsion have been developed.
Revised: A few methods for minimizing the amount of computation needed for repulsion have been developed.

Original: The ﬁrst, is the Barnes-Hut tree algorithm Van Der Maaten (2014), which bins the embedding space into cells and where repulsive forces can be computed over cells rather than individual data points within those cells.
Revised: The ﬁrst, is the Barnes-Hut tree algorithm Van Der Maaten (2014), which bins the embedding space into cells and where repulsive forces can be computed over cells rather than individual data points within those cells.

Original: Similarly, the more recent interpolation-based t-SNE (FIt-SNE; Linderman et al. 2017; 2019) divides the embedding space up into a grid and computes repulsive forces over the grid, rather than the full set of embeddings.
Revised: Similarly, the more recent interpolation-based t-SNE (FIt-SNE; Linderman et al. 2017; 2019) divides the embedding space up into a grid and computes repulsive forces over the grid, rather than the full set of embeddings.

Original: 1. P ARAMETRIC UMAP
Revised: 1. P ARAMETRIC UMAP

Original: To summarize, both t-SNE and UMAP rely on the construction of a graph, and a subsequent embedding that preserves the structure of that graph.
Revised: To summarize, both t-SNE and UMAP rely on the construction of a graph, and a subsequent embedding that preserves the structure of that graph.

Original: UMAP learns an embedding by minimizing cross entropy sampled over positively weighted edges (attraction), and using negative sampling randomly over the dataset (repulsion), allowing minimization to occur over sampled batches of the dataset.
Revised: UMAP learns an embedding by minimizing cross entropy sampled over positively weighted edges (attraction), and using negative sampling randomly over the dataset (repulsion), allowing minimization to occur over sampled batches of the dataset.

Original: t-SNE, meanwhile, minimizes a KL divergence loss function normalized over the entire set of embeddings in the dataset using different approximation techniques to compute attractive and repulsive forces.
Revised: t-SNE, meanwhile, minimizes a KL divergence loss function normalized over the entire set of embeddings in the dataset using different approximation techniques to compute attractive and repulsive forces.

Original: Because t-SNE optimization requires normalization over Q , gradient descent can only be performed after computing edge probabilities over the entire dataset.
Revised: Because t-SNE optimization requires normalization over Q , gradient descent can only be performed after computing edge probabilities over the entire dataset.

Original: Projecting an entire dataset into a neural network between each gradient descent step would be too computationally expensive to optimize however.
Revised: Projecting an entire dataset into a neural network between each gradient descent step would be too computationally expensive to optimize however.

Original: The trick that Parametric t-SNE proposes to this problem is to split the dataset up into large batches (e.g. 5000 datapoints in the original paper) that are independently normalized over and used constantly throughout training, rather than being randomized.
Revised: The trick that Parametric t-SNE proposes to this problem is to split the dataset up into large batches (e.g. 5000 datapoints in the original paper) that are independently normalized over and used constantly throughout training, rather than being randomized.

Original: Conversely, UMAP can be trained on batch sizes as small as a single edge, making it suitable for minibatch training needed for memory-expensive neural networks trained on large datasets as well as on-line learning.
Revised: Conversely, UMAP can be trained on batch sizes as small as a single edge, making it suitable for minibatch training needed for memory-expensive neural networks trained on large datasets as well as on-line learning.

Original: Given these design features, the UMAP algorithm is better suited to deep neural networks, and is more extendable to typical neural network training regimes.
Revised: Given these design features, the UMAP algorithm is better suited to deep neural networks, and is more extendable to typical neural network training regimes.

Original: Parametric UMAP can be deﬁned simply by applying the UMAP cost function to a deep neural network over mini batches using negative sampling.
Revised: Parametric UMAP can be deﬁned simply by applying the UMAP cost function to a deep neural network over mini batches using negative sampling.

Original: In our implementation, we keep all hyperparameters the same as the original
Revised: In our implementation, we keep all hyperparameters the same as the original

Original: UMAP implementation, and optimize over the UMAP loss function using the Adam (Kingma & Ba, 2014) optimizer 2 .
Revised: UMAP implementation, and optimize over the UMAP loss function using the Adam (Kingma & Ba, 2014) optimizer 2 .

Original:  UMAP LOSS AS A REGULARIZATION
Revised:  UMAP LOSS AS A REGULARIZATION

Original: In machine learning, regularization refers to a modiﬁcation of the learning algorithm to improve generalization to new data.
Revised: In machine learning, regularization refers to a modiﬁcation of the learning algorithm to improve generalization to new data.

Original: Here, we consider both regularizing neural networks with UMAP loss, as well as using additional loss functions to regularize UMAP.
Revised: Here, we consider both regularizing neural networks with UMAP loss, as well as using additional loss functions to regularize UMAP.

Original: While non-parametric UMAP optimizes UMAP loss directly over embeddings (Fig 2A), our proposed algorithm, Parametric UMAP, applies the same cost function over an encoder network (Fig 2B).
Revised: While non-parametric UMAP optimizes UMAP loss directly over embeddings (Fig 2A), our proposed algorithm, Parametric UMAP, applies the same cost function over an encoder network (Fig 2B).

Original: Beyond the parameters being optimized, the optimization steps are equivalent.
Revised: Beyond the parameters being optimized, the optimization steps are equivalent.

Original: Given this simple extension toward optimizing over a neural network, UMAP can further be extended to arbitrary neural network architectures and loss functions.
Revised: Given this simple extension toward optimizing over a neural network, UMAP can further be extended to arbitrary neural network architectures and loss functions.

Original: Speciﬁcally, we use UMAP loss over unlabeled data as a regularization term for a classiﬁer network trained on labeled data (semi-supervised learning), as well as an autoencoder loss as an additional regularization for UMAP latent projections.
Revised: Speciﬁcally, we use UMAP loss over unlabeled data as a regularization term for a classiﬁer network trained on labeled data (semi-supervised learning), as well as an autoencoder loss as an additional regularization for UMAP latent projections.

Original: In the experiments section, we quantitatively explore these networks.
Revised: In the experiments section, we quantitatively explore these networks.

Original: 2. A UTOENCODING WITH UMAP
Revised: 2. A UTOENCODING WITH UMAP

Original: AEs are by themselves a powerful dimensionality reduction algorithm (Hinton & Salakhutdinov, 2006).
Revised: AEs are by themselves a powerful dimensionality reduction algorithm (Hinton & Salakhutdinov, 2006).

Original: Thus, combining them with UMAP may yield additional beneﬁts in capturing latent structure.
Revised: Thus, combining them with UMAP may yield additional beneﬁts in capturing latent structure.

Original: We used an autoencoder as an additional regularization to Parametric UMAP (Fig 2C).
Revised: We used an autoencoder as an additional regularization to Parametric UMAP (Fig 2C).

Original: A UMAP/AE hybrid is simply the combination of the UMAP loss and a reconstruction loss and apply both over the network.
Revised: A UMAP/AE hybrid is simply the combination of the UMAP loss and a reconstruction loss and apply both over the network.

Original: VAEs have similarly been used in conjunction with Parametric t-SNE for capturing structure in animal behavioral data (Graving & Couzin, 2020) and combining t-SNE, which similarly emphasizes local structure, with AEs aids in capturing more global structure over the dataset (van der
Revised: VAEs have similarly been used in conjunction with Parametric t-SNE for capturing structure in animal behavioral data (Graving & Couzin, 2020) and combining t-SNE, which similarly emphasizes local structure, with AEs aids in capturing more global structure over the dataset (van der

Original: Maaten & Hinton, 2008; Graving & Couzin, 2020).
Revised: Maaten & Hinton, 2008; Graving & Couzin, 2020).

Original: 2. S EMISUPERVISED LEARNING
Revised: 2. S EMISUPERVISED LEARNING

Original: UMAP can also be used to regularize supervised classiﬁer networks, training the network on a combination of labeled data with the classiﬁer loss and unlabeled data with the UMAP loss (Fig 2D).
Revised: UMAP can also be used to regularize supervised classiﬁer networks, training the network on a combination of labeled data with the classiﬁer loss and unlabeled data with the UMAP loss (Fig 2D).

Original: Semi-supervised learning refers to the use of unlabeled data to jointly learn the structure of a dataset while labeled data is used to optimize the supervised objective function, such as classifying images.
Revised: Semi-supervised learning refers to the use of unlabeled data to jointly learn the structure of a dataset while labeled data is used to optimize the supervised objective function, such as classifying images.

Original: Here, we explore how UMAP can be jointly trained as an objective function in a deep neural network alongside a classiﬁer.
Revised: Here, we explore how UMAP can be jointly trained as an objective function in a deep neural network alongside a classiﬁer.

Original: In the example in Fig 4, we show an intuitive example of semisupervised learning using UMAP over the Moons dataset (Pedregosa et al., 2011).
Revised: In the example in Fig 4, we show an intuitive example of semisupervised learning using UMAP over the Moons dataset (Pedregosa et al., 2011).

Original: By training a Y-shaped network (Fig 2D) both on the classiﬁer loss over labeled datapoints (Fig 4A, red and blue) and the UMAP loss over unlabeled datapoints (Fig 4A, grey) jointly, the shared latent space between between the UMAP and classiﬁer network pulls apart the two moons (Fig 4B), resulting a a decision boundary that divides cleanly between the two distributions in dataspace.
Revised: By training a Y-shaped network (Fig 2D) both on the classiﬁer loss over labeled datapoints (Fig 4A, red and blue) and the UMAP loss over unlabeled datapoints (Fig 4A, grey) jointly, the shared latent space between between the UMAP and classiﬁer network pulls apart the two moons (Fig 4B), resulting a a decision boundary that divides cleanly between the two distributions in dataspace.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Experiments were performed comparing our novel algorithms, Parametric UMAP and a UMAP/AE hybrid, to nonparametric UMAP, nonparametric t-SNE (FIt-SNE)
Revised: Experiments were performed comparing our novel algorithms, Parametric UMAP and a UMAP/AE hybrid, to nonparametric UMAP, nonparametric t-SNE (FIt-SNE)

Original: (Poli ˇ car et al., 2019), Parametric t-SNE, an AE, a VAE, and a PCA projections.
Revised: (Poli ˇ car et al., 2019), Parametric t-SNE, an AE, a VAE, and a PCA projections.

Original: We additionally compare a second non-parametric
Revised: We additionally compare a second non-parametric

Original: UMAP implementation that has the same underlying code as Parametric UMAP, but where optimization is performed over embeddings directly, rather than neural network weights.
Revised: UMAP implementation that has the same underlying code as Parametric UMAP, but where optimization is performed over embeddings directly, rather than neural network weights.

Original: This comparison is made to provide a bridge between the UMAP-learn implementation and parametric UMAP, to control for variation exogenous to differences parametric versus non-parametric embedding.
Revised: This comparison is made to provide a bridge between the UMAP-learn implementation and parametric UMAP, to control for variation exogenous to differences parametric versus non-parametric embedding.

Original: Parametric t-SNE, Parametric UMAP, the AE, VAE, and the UMAP/AE hybrid use the same neural network architectures and optimizers within each dataset (described in A.1 and A.2).
Revised: Parametric t-SNE, Parametric UMAP, the AE, VAE, and the UMAP/AE hybrid use the same neural network architectures and optimizers within each dataset (described in A.1 and A.2).

Original: 3. E MBEDDINGS
Revised: 3. E MBEDDINGS

Original: To quantitatively measure the quality of embeddings we compared embedding algorithms on several metrics across datasets.
Revised: To quantitatively measure the quality of embeddings we compared embedding algorithms on several metrics across datasets.

Original: We compared each method/dataset on 2D and 64D projections (where possible).
Revised: We compared each method/dataset on 2D and 64D projections (where possible).

Original: Each metric is explained in detail in A.3.
Revised: Each metric is explained in detail in A.3.

Original: The 2D projection of each dataset/method is shown in Fig 3.
Revised: The 2D projection of each dataset/method is shown in Fig 3.

Original: The results are given in Figs 5-9 and Tables 2-6, and summarized below.
Revised: The results are given in Figs 5-9 and Tables 2-6, and summarized below.

Original: Trustworthiness Trustworthiness (Eq. 5, Venna & Kaski 2006) is a measure of how much of the local structure of a dataset is preserved in a set of embeddings.
Revised: Trustworthiness Trustworthiness (Eq. 5, Venna & Kaski 2006) is a measure of how much of the local structure of a dataset is preserved in a set of embeddings.

Original: In 2D, we observe each of the
Revised: In 2D, we observe each of the

Original: UMAP algorithms performs similarly in trustworthiness, with t-SNE being slightly more trustworthy in each dataset (Fig 5).
Revised: UMAP algorithms performs similarly in trustworthiness, with t-SNE being slightly more trustworthy in each dataset (Fig 5).

Original: At 64D, PCA, AE, VAE, and Parametric t-SNE are most trustworthy in comparison to each UMAP implementation, possibly reﬂecting the more approximate repulsion (negative sampling) used by UMAP.
Revised: At 64D, PCA, AE, VAE, and Parametric t-SNE are most trustworthy in comparison to each UMAP implementation, possibly reﬂecting the more approximate repulsion (negative sampling) used by UMAP.

Original: KNN-Classiﬁer A KNN-classiﬁer is used as a baseline to measure supervised classiﬁcation accuracy based upon local relationships in embeddings.
Revised: KNN-Classiﬁer A KNN-classiﬁer is used as a baseline to measure supervised classiﬁcation accuracy based upon local relationships in embeddings.

Original: We ﬁnd KNN-classiﬁer performance largely reﬂects trustworthiness (Figs 6, 7).
Revised: We ﬁnd KNN-classiﬁer performance largely reﬂects trustworthiness (Figs 6, 7).

Original: In 2D, we observe a broadly similar performance between UMAP and t-SNE varients, each of which is substantially better than the PCA, AE, or VAE projections.
Revised: In 2D, we observe a broadly similar performance between UMAP and t-SNE varients, each of which is substantially better than the PCA, AE, or VAE projections.

Original: At 64 dimensions UMAP projections are similar but in some datasets (FMNIST, CIFAR10) slightly underperform PCA, AE, VAE, and Parametric t-SNE .
Revised: At 64 dimensions UMAP projections are similar but in some datasets (FMNIST, CIFAR10) slightly underperform PCA, AE, VAE, and Parametric t-SNE .

Original: Silhouette score Silhouette score measures how clustered a set of embeddings are given ground truth labels.
Revised: Silhouette score Silhouette score measures how clustered a set of embeddings are given ground truth labels.

Original: In 2D, across datasets, we tend to see a better silhouette score for UMAP and Parametric
Revised: In 2D, across datasets, we tend to see a better silhouette score for UMAP and Parametric

Original: UMAP projections than t-SNE and Parametric t-SNE, which are in turn more clustered than PCA in all cases but CIFAR10, which shows little difference from PCA (Fig 8).
Revised: UMAP projections than t-SNE and Parametric t-SNE, which are in turn more clustered than PCA in all cases but CIFAR10, which shows little difference from PCA (Fig 8).

Original: The clustering of each dataset can also be observed in Fig 3, where t-SNE and Parametric t-SNE are more spread out within cluster than UMAP.
Revised: The clustering of each dataset can also be observed in Fig 3, where t-SNE and Parametric t-SNE are more spread out within cluster than UMAP.

Original: In 64D projections, we ﬁnd the silhouette score of Parametric t-SNE is near or below that of PCA, which are lower than UMAP-based methods.
Revised: In 64D projections, we ﬁnd the silhouette score of Parametric t-SNE is near or below that of PCA, which are lower than UMAP-based methods.

Original: We note, however, that the poor performance of Parametric t-SNE may reﬂect setting the degrees-of-freedom ( α ) at d − 1 which is only one of three parameterization schemes that Van Der Maaten (2009) suggests.
Revised: We note, however, that the poor performance of Parametric t-SNE may reﬂect setting the degrees-of-freedom ( α ) at d − 1 which is only one of three parameterization schemes that Van Der Maaten (2009) suggests.

Original: A learned degrees-of-freedom parameter might improve performance for parametric t-SNE at higher dimensions.
Revised: A learned degrees-of-freedom parameter might improve performance for parametric t-SNE at higher dimensions.

Original: Clustering To compare clustering directly across embeddings, we performed k -Means clustering over each latent projection and compared each embedding’s clustering on the basis of the normalized mutual information (NMI) between clustering schemes.
Revised: Clustering To compare clustering directly across embeddings, we performed k -Means clustering over each latent projection and compared each embedding’s clustering on the basis of the normalized mutual information (NMI) between clustering schemes.

Original: In both the 2D and 64D projections, we ﬁnd that NMI corresponds closely to the silhouette score.
Revised: In both the 2D and 64D projections, we ﬁnd that NMI corresponds closely to the silhouette score.

Original: UMAP and t-SNE show comperable clustering in 2D, both well above PCA in most datasets.
Revised: UMAP and t-SNE show comperable clustering in 2D, both well above PCA in most datasets.

Original: At 64D, each UMAP approach shows superior performance over t-SNE (Fig 9).
Revised: At 64D, each UMAP approach shows superior performance over t-SNE (Fig 9).

Original: 3. S PEED
Revised: 3. S PEED

Original: Training speed Optimization in non-parametric UMAP is not inﬂuenced by the dimensionality of the original dataset; the dimensionality of the dataset only comes into play in computing the nearest-neighbors graph.
Revised: Training speed Optimization in non-parametric UMAP is not inﬂuenced by the dimensionality of the original dataset; the dimensionality of the dataset only comes into play in computing the nearest-neighbors graph.

Original: In contrast, training speeds for Parametric UMAP are variable based upon the dimensionality of data and the architecture of the neural network used.
Revised: In contrast, training speeds for Parametric UMAP are variable based upon the dimensionality of data and the architecture of the neural network used.

Original: In Fig 10, we show the cross-entropy loss over time for Parametric and Non-parametric UMAP, for the MNIST, Fashion MNIST, and Retina datasets.
Revised: In Fig 10, we show the cross-entropy loss over time for Parametric and Non-parametric UMAP, for the MNIST, Fashion MNIST, and Retina datasets.

Original: Across each dataset we ﬁnd that non-parametric UMAP reaches a lower loss more quickly than Parametric UMAP, but that Parametric UMAP reaches a similar cross-entropy within an order of magnitude.
Revised: Across each dataset we ﬁnd that non-parametric UMAP reaches a lower loss more quickly than Parametric UMAP, but that Parametric UMAP reaches a similar cross-entropy within an order of magnitude.

Original: Thus, Parametric UMAP can train more slowly than non-parametric UMAP, but training times remain within a similar range making Paramatric UMAP reasonable alternative to non-parametric UMAP in terms of training time.
Revised: Thus, Parametric UMAP can train more slowly than non-parametric UMAP, but training times remain within a similar range making Paramatric UMAP reasonable alternative to non-parametric UMAP in terms of training time.

Original: Embedding and reconstruction speed
Revised: Embedding and reconstruction speed

Original: A parametric mapping allows embeddings to be inferred directly from data, resulting in a quicker embedding than non-parametric methods.
Revised: A parametric mapping allows embeddings to be inferred directly from data, resulting in a quicker embedding than non-parametric methods.

Original: The speed of embedding is especially important in signal-processing paradigms where near real-time embedding speeds are necessary.
Revised: The speed of embedding is especially important in signal-processing paradigms where near real-time embedding speeds are necessary.

Original: For example in brain-machine interfacing, bioacoustics, and computational ethology, fast embedding methods like PCA or deep neural networks are necessary for real-time analysis and manipulations and deep neural networks are increasingly being used (e.g. Pandarinath et al. 2018; Brown & De Bivort 2018;
Revised: For example in brain-machine interfacing, bioacoustics, and computational ethology, fast embedding methods like PCA or deep neural networks are necessary for real-time analysis and manipulations and deep neural networks are increasingly being used (e.g. Pandarinath et al. 2018; Brown & De Bivort 2018;

Original: Sainburg et al. 2019).
Revised: Sainburg et al. 2019).

Original: Here, we compare the embedding speed of a held-out test sample for each dataset, as well as the speed of reconstruction of the same held out test samples.
Revised: Here, we compare the embedding speed of a held-out test sample for each dataset, as well as the speed of reconstruction of the same held out test samples.

Original: Broadly, we observe similar embedding times for the non-parametric t-SNE and UMAP methods, which are several orders of magnitude slower than the parametric methods (Fig 11).
Revised: Broadly, we observe similar embedding times for the non-parametric t-SNE and UMAP methods, which are several orders of magnitude slower than the parametric methods (Fig 11).

Original: Because the same neural networks are used across the different parametric UMAP and t-SNE methods, we show only Parametric UMAP in Fig 12, which is only slightly slower than PCA, making it a viable candidate for fast embedding in embedding paradigms where PCA is currently used.
Revised: Because the same neural networks are used across the different parametric UMAP and t-SNE methods, we show only Parametric UMAP in Fig 12, which is only slightly slower than PCA, making it a viable candidate for fast embedding in embedding paradigms where PCA is currently used.

Original: Similar to the embedding speeds, we compared parametric and non-parametric UMAP reconstruction speeds (Fig 12).
Revised: Similar to the embedding speeds, we compared parametric and non-parametric UMAP reconstruction speeds (Fig 12).

Original: We ﬁnd that the reconstructions of Parametric UMAP are orders of magnitude faster than non-parametric UMAP, and slightly slower, but within the same order of magnitude, as
Revised: We ﬁnd that the reconstructions of Parametric UMAP are orders of magnitude faster than non-parametric UMAP, and slightly slower, but within the same order of magnitude, as

Original: PCA.
Revised: PCA.

Original: 3. A UTOENCODING
Revised: 3. A UTOENCODING

Original: The ability to reconstruct data from embeddings can both aid in understanding the structure of nonlinear embeddings, as well as allow for manipulation and synthesis of data based on the learned features of the dataset.
Revised: The ability to reconstruct data from embeddings can both aid in understanding the structure of nonlinear embeddings, as well as allow for manipulation and synthesis of data based on the learned features of the dataset.

Original: We compared the reconstruction accuracy across each method which had inverse-transform capabilities (i.e. Z → X ), as well as the reconstruction speed across the neural network-based implementations to non-parametric implementations and PCA.
Revised: We compared the reconstruction accuracy across each method which had inverse-transform capabilities (i.e. Z → X ), as well as the reconstruction speed across the neural network-based implementations to non-parametric implementations and PCA.

Original: In addition, we performed latent space algebra on Parametric UMAP embeddings both with and without an autoencoder regularization, and found that reconstructed data can be linearly manipulated in complex feature space.
Revised: In addition, we performed latent space algebra on Parametric UMAP embeddings both with and without an autoencoder regularization, and found that reconstructed data can be linearly manipulated in complex feature space.

Original: Reconstruction accuracy We measured reconstruction accuracy as Mean Squared Error (MSE) across each dataset.
Revised: Reconstruction accuracy We measured reconstruction accuracy as Mean Squared Error (MSE) across each dataset.

Original: in two dimensions, we ﬁnd that Parametric UMAP typically reconstructs better than non-parametric UMAP, which in turn performs better than PCA.
Revised: in two dimensions, we ﬁnd that Parametric UMAP typically reconstructs better than non-parametric UMAP, which in turn performs better than PCA.

Original: In addition, the autoencoder regularization slightly improves reconstruction performance.
Revised: In addition, the autoencoder regularization slightly improves reconstruction performance.

Original: At 64 dimensions, the AE regularized
Revised: At 64 dimensions, the AE regularized

Original: Parametric UMAP is generally comparable to the AE and VAE and performs better than Parametric
Revised: Parametric UMAP is generally comparable to the AE and VAE and performs better than Parametric

Original: UMAP without autoencoder regularization.
Revised: UMAP without autoencoder regularization.

Original: The non-parametric UMAP reconstruction algorithm is not compared at 64 dimensions because it relies on an estimation of Delaunay triangulation, which does not scale well with higher dimensions.
Revised: The non-parametric UMAP reconstruction algorithm is not compared at 64 dimensions because it relies on an estimation of Delaunay triangulation, which does not scale well with higher dimensions.

Original: Latent features Previous work shows that parametric embedding algorithms such as AEs (e.g.
Revised: Latent features Previous work shows that parametric embedding algorithms such as AEs (e.g.

Original: Variational Autoencoders) linearize complex data features in latent-space, for example the presence of a pair of sunglasses in pictures of faces (e.g. Radford et al.
Revised: Variational Autoencoders) linearize complex data features in latent-space, for example the presence of a pair of sunglasses in pictures of faces (e.g. Radford et al.

Original: 2015; White 2016; Sainburg et al.
Revised: 2015; White 2016; Sainburg et al.

Original: Here, we performed latent-space algebra and reconstructed manipulations on Parametric
Revised: Here, we performed latent-space algebra and reconstructed manipulations on Parametric

Original: UMAP latent-space to explore whether UMAP does the same.
Revised: UMAP latent-space to explore whether UMAP does the same.

Original: To do so, we use the CelebAMask-HQ dataset, which contains annotations for 40 different facial features over a highly-structured dataset of human faces.
Revised: To do so, we use the CelebAMask-HQ dataset, which contains annotations for 40 different facial features over a highly-structured dataset of human faces.

Original: We projected the dataset of faces into a CNN autoencoder architecture based upon the architecture deﬁned in Huang et al.
Revised: We projected the dataset of faces into a CNN autoencoder architecture based upon the architecture deﬁned in Huang et al.

Original: We trained the network ﬁrst using UMAP loss alone (Parametric UMAP; Fig 14 right), and second using the joint UMAP and AE loss (Fig 14 center).
Revised: We trained the network ﬁrst using UMAP loss alone (Parametric UMAP; Fig 14 right), and second using the joint UMAP and AE loss (Fig 14 center).

Original: Wh then ﬁt an OLS regression to predict the latent projections of entire dataset using the 40 annotated features (e.g. hair color, presence of beard, smiling, etc).
Revised: Wh then ﬁt an OLS regression to predict the latent projections of entire dataset using the 40 annotated features (e.g. hair color, presence of beard, smiling, etc).

Original: The vectors corresponding to each feature learned by the linear model were then treated as feature vectors in latent space, and added and subtracted from projected images, then passed through the decoder to observe the resulting image (as in Sainburg et al. 2018).
Revised: The vectors corresponding to each feature learned by the linear model were then treated as feature vectors in latent space, and added and subtracted from projected images, then passed through the decoder to observe the resulting image (as in Sainburg et al. 2018).

Original: We ﬁnd that complex latent features are linearized in latent space, both when the network is trained with UMAP loss alone as well as when the network is trained with AE loss.
Revised: We ﬁnd that complex latent features are linearized in latent space, both when the network is trained with UMAP loss alone as well as when the network is trained with AE loss.

Original: For example, in the last set of images in Figure 10B, a smile can be added or removed from the projected image by adding or subtracting its corresponding latent vector.
Revised: For example, in the last set of images in Figure 10B, a smile can be added or removed from the projected image by adding or subtracting its corresponding latent vector.

Original: 3. S EMI - SUPERVISED LEARNING
Revised: 3. S EMI - SUPERVISED LEARNING

Original: Real-word datasets are often comprised of a small number of labeled data, and a large number of unlabeled data.
Revised: Real-word datasets are often comprised of a small number of labeled data, and a large number of unlabeled data.

Original: Semisupervised learning (SSL) aims to use the unlabeled data to learn the structure of the dataset, aiding a supervised learning algorithm in making decisions about the data.
Revised: Semisupervised learning (SSL) aims to use the unlabeled data to learn the structure of the dataset, aiding a supervised learning algorithm in making decisions about the data.

Original: Current
Revised: Current

Original: SOTA approaches in many areas of supervised learning such as computer vision rely on deep neural networks.
Revised: SOTA approaches in many areas of supervised learning such as computer vision rely on deep neural networks.

Original: Likewise, semisupervised learning approaches modify supervised networks with stucturelearning loss using unlabeled data.
Revised: Likewise, semisupervised learning approaches modify supervised networks with stucturelearning loss using unlabeled data.

Original: Parametric UMAP, being a neural network that learns structure from unlabeled data, is well suited to semi-supervised applications.
Revised: Parametric UMAP, being a neural network that learns structure from unlabeled data, is well suited to semi-supervised applications.

Original: Here, we determine the efﬁcacy of UMAP for semisupervised learning by jointly training a Y-shaped neural network (Fig 2D) on classiﬁcation and UMAP compared to classiﬁcation alone on datasets with varying numbers of labeled data.
Revised: Here, we determine the efﬁcacy of UMAP for semisupervised learning by jointly training a Y-shaped neural network (Fig 2D) on classiﬁcation and UMAP compared to classiﬁcation alone on datasets with varying numbers of labeled data.

Original: For comparisons, we used three datasets, ranging from highly-structured (MNIST) to moderately structured (FMNIST) to unstructured (CIFAR10) in UMAP using a na¨ıve distance metric in data space (e.g. Euclidean distance over images).
Revised: For comparisons, we used three datasets, ranging from highly-structured (MNIST) to moderately structured (FMNIST) to unstructured (CIFAR10) in UMAP using a na¨ıve distance metric in data space (e.g. Euclidean distance over images).

Original: For each dataset, we used a deep convolutional neural networks (CNN) which performs with relatively high accuracy for CNN classiﬁcation on the fully supervised networks (see Table 8) based upon the CNN13 architecture commonly used in SSL (Oliver et al., 2018).
Revised: For each dataset, we used a deep convolutional neural networks (CNN) which performs with relatively high accuracy for CNN classiﬁcation on the fully supervised networks (see Table 8) based upon the CNN13 architecture commonly used in SSL (Oliver et al., 2018).

Original: Na¨ıve UMAP embedding For datasets where structure is learned in UMAP (e.g. MNIST, FMNIST) we expect that regularizing a classiﬁer network with UMAP loss will aid the network in labeling data by learning the structure of the dataset from unlabeled data.
Revised: Na¨ıve UMAP embedding For datasets where structure is learned in UMAP (e.g. MNIST, FMNIST) we expect that regularizing a classiﬁer network with UMAP loss will aid the network in labeling data by learning the structure of the dataset from unlabeled data.

Original: To test this, we compared a baseline classiﬁer to a network jointly trained on classiﬁer loss and UMAP loss.
Revised: To test this, we compared a baseline classiﬁer to a network jointly trained on classiﬁer loss and UMAP loss.

Original: We ﬁrst trained the baseline classiﬁer to asymptotic performance on the validation dataset, then using the pretrainedweights from the baseline classiﬁer, trained a Y-shaped network (Fig 2D) jointly on UMAP over
Revised: We ﬁrst trained the baseline classiﬁer to asymptotic performance on the validation dataset, then using the pretrainedweights from the baseline classiﬁer, trained a Y-shaped network (Fig 2D) jointly on UMAP over

Original: Euclidean distances and a classiﬁer loss over the dataset.
Revised: Euclidean distances and a classiﬁer loss over the dataset.

Original: We ﬁnd that for MNIST and FMNIST, where categorically-relevant structure is found in latent projections of the datasets, classiﬁcations are improved in the semi-supervised network over the supervised network alone, especially with smaller numbers of training examples (Fig 15).
Revised: We ﬁnd that for MNIST and FMNIST, where categorically-relevant structure is found in latent projections of the datasets, classiﬁcations are improved in the semi-supervised network over the supervised network alone, especially with smaller numbers of training examples (Fig 15).

Original: In contrast for CIFAR10, the additional UMAP loss impairs performance in the classiﬁer.
Revised: In contrast for CIFAR10, the additional UMAP loss impairs performance in the classiﬁer.

Original: Consistency regularization and learned invariance using data augmentation
Revised: Consistency regularization and learned invariance using data augmentation

Original: Several current
Revised: Several current

Original: SOTA SSL approaches employ a technique called consistency regularization (Sajjadi et al., 2016); training a classiﬁer to produce the same predictions with unlabeled data which have been augmented and data that have not been augmented (Sohn et al., 2020; Berthelot et al., 2019).
Revised: SOTA SSL approaches employ a technique called consistency regularization (Sajjadi et al., 2016); training a classiﬁer to produce the same predictions with unlabeled data which have been augmented and data that have not been augmented (Sohn et al., 2020; Berthelot et al., 2019).

Original: In a similar vein, we train the network to preserve the structure of the UMAP graph when data have been augmented.
Revised: In a similar vein, we train the network to preserve the structure of the UMAP graph when data have been augmented.

Original: We computed a UMAP graph over un-augmented data, and using augmented data, trained the network jointly using classiﬁer and UMAP loss, teaching the network to learn to optimize the same
Revised: We computed a UMAP graph over un-augmented data, and using augmented data, trained the network jointly using classiﬁer and UMAP loss, teaching the network to learn to optimize the same

Original: UMAP graph, invarient to augmentations in the data.
Revised: UMAP graph, invarient to augmentations in the data.

Original: We observe a further improvement in network accuracy for MNIST and FMNIST over the baseline, and the augmented baseline (Fig 16).
Revised: We observe a further improvement in network accuracy for MNIST and FMNIST over the baseline, and the augmented baseline (Fig 16).

Original: For the
Revised: For the

Original: CIFAR10 dataset, the addition of the UMAP loss, even over augmented data, reduces classiﬁcation accuracy.
Revised: CIFAR10 dataset, the addition of the UMAP loss, even over augmented data, reduces classiﬁcation accuracy.

Original: Learning a metric UMAP embeddings using a supervised network It is unsurprising that
Revised: Learning a metric UMAP embeddings using a supervised network It is unsurprising that

Original: UMAP confers no improvement for the CIFAR10 dataset, as UMAP computed over the pixelwise Euclidean distance between images in the CIFAR10 dataset does not capture very much categorically-relevant structure in the dataset.
Revised: UMAP confers no improvement for the CIFAR10 dataset, as UMAP computed over the pixelwise Euclidean distance between images in the CIFAR10 dataset does not capture very much categorically-relevant structure in the dataset.

Original: Because no common distance metric over CIFAR images is likely to capture such structure, we consider using supervision to learn a categoricallyrelevant distance metric for UMAP.
Revised: Because no common distance metric over CIFAR images is likely to capture such structure, we consider using supervision to learn a categoricallyrelevant distance metric for UMAP.

Original: We do so by training on a UMAP graph computed using distance over latent activations in the classiﬁer network (as in, e.g. Carter et al. 2019), where categorical structure can be seen in UMAP projection (Fig 17).
Revised: We do so by training on a UMAP graph computed using distance over latent activations in the classiﬁer network (as in, e.g. Carter et al. 2019), where categorical structure can be seen in UMAP projection (Fig 17).

Original: The intuition being that training the network with unlabeled data to capture distributional-structure within the network’s learned categorically-relevant space will aid in labeling new data.
Revised: The intuition being that training the network with unlabeled data to capture distributional-structure within the network’s learned categorically-relevant space will aid in labeling new data.

Original: We ﬁnd that in all three datasets, without augmentation, the addition of the learned UMAP loss confers little no improvement in classiﬁcation accuracy over the data (Fig 18).
Revised: We ﬁnd that in all three datasets, without augmentation, the addition of the learned UMAP loss confers little no improvement in classiﬁcation accuracy over the data (Fig 18).

Original: When we look at non-parametric projections of the graph over latent space activations, we see that the learned graph largely conforms to the network’s already-present categorical decision making (e.g. Fig 17 predictions vs. ground truth).
Revised: When we look at non-parametric projections of the graph over latent space activations, we see that the learned graph largely conforms to the network’s already-present categorical decision making (e.g. Fig 17 predictions vs. ground truth).

Original: In contrast, with augmentation, the addition of the UMAP loss improves performance in each dataset, including CIFAR10.
Revised: In contrast, with augmentation, the addition of the UMAP loss improves performance in each dataset, including CIFAR10.

Original: This contrast in improvement demonstrates that training the network to learn a distribution in a categorically-relevant space that is already intrinsic to the network does not confer any additional information that the network can use in classiﬁcation.
Revised: This contrast in improvement demonstrates that training the network to learn a distribution in a categorically-relevant space that is already intrinsic to the network does not confer any additional information that the network can use in classiﬁcation.

Original: Training the network to be invariant toward augmentations in the data, however, does aid in regularizing the classiﬁer, more in-line with directly training the network on consistency in classiﬁcations (Sajjadi et al., 2016).
Revised: Training the network to be invariant toward augmentations in the data, however, does aid in regularizing the classiﬁer, more in-line with directly training the network on consistency in classiﬁcations (Sajjadi et al., 2016).

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: In this paper we propose a novel parametric extension to UMAP using arbitrary deep neural networks.
Revised: In this paper we propose a novel parametric extension to UMAP using arbitrary deep neural networks.

Original: The non-parametric UMAP embedding algorithm is particularly well suited to deep learning applications, through batch-wise training.
Revised: The non-parametric UMAP embedding algorithm is particularly well suited to deep learning applications, through batch-wise training.

Original: We showed that the parametric embeddings perform similarly to non-parametric embeddings, with the added beneﬁt of a learned mapping between data space and embedding space.
Revised: We showed that the parametric embeddings perform similarly to non-parametric embeddings, with the added beneﬁt of a learned mapping between data space and embedding space.

Original: This parametric embedding makes UMAP feasible in ﬁelds where where dimensionality reduction of continuously generated signals plays an important role in real-time analysis and experimental control.
Revised: This parametric embedding makes UMAP feasible in ﬁelds where where dimensionality reduction of continuously generated signals plays an important role in real-time analysis and experimental control.

Original: We also show that parametric UMAP projections linearize complex features in latent space, similar to other latent neural networks such as AEs and GANs.
Revised: We also show that parametric UMAP projections linearize complex features in latent space, similar to other latent neural networks such as AEs and GANs.

Original: Finally, we showed that the UMAP loss can be used as a regularization for deep neural networks, such as classiﬁers.
Revised: Finally, we showed that the UMAP loss can be used as a regularization for deep neural networks, such as classiﬁers.

Original: This regularization serves SSL by jointly learning local distributional structure alongside training on supervised loss.
Revised: This regularization serves SSL by jointly learning local distributional structure alongside training on supervised loss.

Original: We showed that this regularization works well on data where categorical structure is captured well by UMAP projections, but poorly with less structured data.
Revised: We showed that this regularization works well on data where categorical structure is captured well by UMAP projections, but poorly with less structured data.

Original: Deﬁning a metric for UMAP that captures categorically relevant structure in UMAP is paramount to learning a useful data manifold.
Revised: Deﬁning a metric for UMAP that captures categorically relevant structure in UMAP is paramount to learning a useful data manifold.

Original: Future work can explore improving SSL with UMAP by using supervision to learn a categorically-relevant metric in coordination with manifold-learning to capture distributional structure under that metric.
Revised: Future work can explore improving SSL with UMAP by using supervision to learn a categorically-relevant metric in coordination with manifold-learning to capture distributional structure under that metric.

Original: A A PPENDIX A. D ATASETS
Revised: A A PPENDIX A. D ATASETS

Original: We performed experiments over several different datasets varying in complexity.
Revised: We performed experiments over several different datasets varying in complexity.

Original: The Cassin’s vireo song dataset (Hedley, 2016b;a; Sainburg et al., 2019) consists of spectrograms of 20 of the most frequently sung elements elements of Cassin’s vireo song (zero padded to 32 frequency by 31 time bins) produced by several individuals recorded in the Sierra Nevada mountains of California.
Revised: The Cassin’s vireo song dataset (Hedley, 2016b;a; Sainburg et al., 2019) consists of spectrograms of 20 of the most frequently sung elements elements of Cassin’s vireo song (zero padded to 32 frequency by 31 time bins) produced by several individuals recorded in the Sierra Nevada mountains of California.

Original: Despite being recorded in the wild, the Cassin’s vireo song is relatively low noise and vocal elements are highly stereotyped.
Revised: Despite being recorded in the wild, the Cassin’s vireo song is relatively low noise and vocal elements are highly stereotyped.

Original: MNIST is a benchmark handwritten digits dataset in 28x28 pixels (LeCun et al., 2010).
Revised: MNIST is a benchmark handwritten digits dataset in 28x28 pixels (LeCun et al., 2010).

Original: Fashion MNIST (FMNIST) is a dataset of fashion items in the same format as Fashion MNIST designed to be a more difﬁcult classiﬁcation problem than MNIST (Xiao et al., 2017).
Revised: Fashion MNIST (FMNIST) is a dataset of fashion items in the same format as Fashion MNIST designed to be a more difﬁcult classiﬁcation problem than MNIST (Xiao et al., 2017).

Original: CIFAR10 is a natural image dataset (32x32x3 pixels) with 10 classes (Krizhevsky, 2009).
Revised: CIFAR10 is a natural image dataset (32x32x3 pixels) with 10 classes (Krizhevsky, 2009).

Original: CIFAR classes are much less structured than the other datasets used.
Revised: CIFAR classes are much less structured than the other datasets used.

Original: For example, unlike FMNIST, subjects of images are not uniformly centered in the image and can have different background conditions that make neighborhood in pixel-space less likely between members of the same class.
Revised: For example, unlike FMNIST, subjects of images are not uniformly centered in the image and can have different background conditions that make neighborhood in pixel-space less likely between members of the same class.

Original: The single-cell retina transcriptome dataset consists of PCA projections (50D) of single-cell RNA transcriptome data from mouse retina (Macosko et al., 2015; Poli ˇ car et al., 2019).
Revised: The single-cell retina transcriptome dataset consists of PCA projections (50D) of single-cell RNA transcriptome data from mouse retina (Macosko et al., 2015; Poli ˇ car et al., 2019).

Original: The CelebAMask-HQ dataset consists of cropped and aligned photographs of celebrity faces with 40 facial feature annotations (Lee et al., 2020; Liu et al., 2015) and label masks corresponding to face-landmarks.
Revised: The CelebAMask-HQ dataset consists of cropped and aligned photographs of celebrity faces with 40 facial feature annotations (Lee et al., 2020; Liu et al., 2015) and label masks corresponding to face-landmarks.

Original: We removed the background of each image to make the task of learning a structured embedding simpler for the neural network.
Revised: We removed the background of each image to make the task of learning a structured embedding simpler for the neural network.

Original: A further description of each dataset is given in Table 1 and a 2D projection of each dataset using each embedding algorithm is given in Figure 3.
Revised: A further description of each dataset is given in Table 1 and a 2D projection of each dataset using each embedding algorithm is given in Figure 3.

Original: [Table]
Revised: [Table]

Original: A. E MBEDDING ALGORITHMS
Revised: A. E MBEDDING ALGORITHMS

Original: Neural network architectures for the Parametric networks differ between datasets.
Revised: Neural network architectures for the Parametric networks differ between datasets.

Original: MNIST, FMNIST, and CIFAR10 use convolutional neural networks.
Revised: MNIST, FMNIST, and CIFAR10 use convolutional neural networks.

Original: The Cassin’s vireo dataset uses an LSTM encoder (and decoder for the autoencoder).
Revised: The Cassin’s vireo dataset uses an LSTM encoder (and decoder for the autoencoder).

Original: The Retina dataset uses a 3-layer MLP with 100-neurons per layer.
Revised: The Retina dataset uses a 3-layer MLP with 100-neurons per layer.

Original: Parametric t-SNE and UMAP used the same neural network architectures and optimizer.
Revised: Parametric t-SNE and UMAP used the same neural network architectures and optimizer.

Original: For
Revised: For

Original: UMAP, the distance metric used for Cassin’s vireo song is a dynamic-time warping (DTW) metric.
Revised: UMAP, the distance metric used for Cassin’s vireo song is a dynamic-time warping (DTW) metric.

Original: The UMAP Autoencoder (UMAP AE) uses the same architecture of the Parametric UMAP and tSNE implementations, combined with a corresponding decoder network.
Revised: The UMAP Autoencoder (UMAP AE) uses the same architecture of the Parametric UMAP and tSNE implementations, combined with a corresponding decoder network.

Original: The encoder network and decoder network are jointly trained on a reconstruction and UMAP loss function.
Revised: The encoder network and decoder network are jointly trained on a reconstruction and UMAP loss function.

Original: We additionally trained a decoder for the Parametric UMAP network, in which the encoder is trained only on the
Revised: We additionally trained a decoder for the Parametric UMAP network, in which the encoder is trained only on the

Original: UMAP loss, and is not jointly trained on a reconstruction loss.
Revised: UMAP loss, and is not jointly trained on a reconstruction loss.

Original: For both t-SNE and Parametric t-SNE perplexity was left at its default value of 30 across datasets.
Revised: For both t-SNE and Parametric t-SNE perplexity was left at its default value of 30 across datasets.

Original: We also left the degrees of freedom for parametric t-SNE at α = d − 1 , where d is the number of dimensions in the latent projection.
Revised: We also left the degrees of freedom for parametric t-SNE at α = d − 1 , where d is the number of dimensions in the latent projection.

Original: A. E MBEDDING METRICS A.3. T RUSTWORTHINESS
Revised: A. E MBEDDING METRICS A.3. T RUSTWORTHINESS

Original: Trustworthiness (Venna & Kaski, 2006) is a measure of how much of the local structure of a dataset is preserved in a set of embeddings.
Revised: Trustworthiness (Venna & Kaski, 2006) is a measure of how much of the local structure of a dataset is preserved in a set of embeddings.

Original: Trustworthiness is quantiﬁed by comparing each datapoint’s nearest neighbors in the original space, to its nearest neighbors in the embedding space:
Revised: Trustworthiness is quantiﬁed by comparing each datapoint’s nearest neighbors in the original space, to its nearest neighbors in the embedding space:

Original: [Equation]
Revised: [Equation]

Original: where k is the number of nearest neighbors trustworthiness is being computed over, and for each sample i , N ki are the nearest k neighbors in the original space, and each sample j is i
Revised: where k is the number of nearest neighbors trustworthiness is being computed over, and for each sample i , N ki are the nearest k neighbors in the original space, and each sample j is i

Original: ’s r
Revised: ’s r

Original: ( i, j ) th
Revised: ( i, j ) th

Original: nearest neighbor.
Revised: nearest neighbor.

Original: We compared the Trustworthness using Scikit-learn’s default k value of 5.
Revised: We compared the Trustworthness using Scikit-learn’s default k value of 5.

Original: Trustworthiness is scaled between 0 and 1, with 1 being more trustworthy.
Revised: Trustworthiness is scaled between 0 and 1, with 1 being more trustworthy.

Original: A.3. KNN C LASSIFIER
Revised: A.3. KNN C LASSIFIER

Original: Related to trustworthiness, a k-Nearest Neighbor’s (KNN) classiﬁer is a supervised algorithm that classiﬁes each unlabeled point based on its k-nearest labeled datapoints.
Revised: Related to trustworthiness, a k-Nearest Neighbor’s (KNN) classiﬁer is a supervised algorithm that classiﬁes each unlabeled point based on its k-nearest labeled datapoints.

Original: We applied a KNN classiﬁer with k=1 (Fig 6) and k=5 (Fig 7) to each dataset.
Revised: We applied a KNN classiﬁer with k=1 (Fig 6) and k=5 (Fig 7) to each dataset.

Original: A.3. S ILHOUETTE SCORE
Revised: A.3. S ILHOUETTE SCORE

Original: As opposed to Trustworthiness, which measures the preservation of local structure in a projection, the silhouette score (Rousseeuw, 1987) measures how ’clustered’ a set of embeddings are, given ground truth labels.
Revised: As opposed to Trustworthiness, which measures the preservation of local structure in a projection, the silhouette score (Rousseeuw, 1987) measures how ’clustered’ a set of embeddings are, given ground truth labels.

Original: Silhouette score is computed as the mean silhouette coefﬁcient across embeddings, where the silhouette coefﬁcient is the distance between each embedding and all other embeddings in the same class, minus the distance to the nearest point in a separate class.
Revised: Silhouette score is computed as the mean silhouette coefﬁcient across embeddings, where the silhouette coefﬁcient is the distance between each embedding and all other embeddings in the same class, minus the distance to the nearest point in a separate class.

Original: Silhouette score is scaled between -1 and 1, with 1 being more clustered.
Revised: Silhouette score is scaled between -1 and 1, with 1 being more clustered.

Original: A.3. C LUSTERING
Revised: A.3. C LUSTERING

Original: To compare clustering directly across embeddings, we performed k -Means clustering over each latent projection and compared each embeddings clustering on the basis of the normalized mutual information (NMI) between clustering schemes.
Revised: To compare clustering directly across embeddings, we performed k -Means clustering over each latent projection and compared each embeddings clustering on the basis of the normalized mutual information (NMI) between clustering schemes.

Original: For each latent projection, k -Means was used to cluster the latent projection with k (the number of clusters) varied between 12 − 1 12 times the true number of categories in the dataset.
Revised: For each latent projection, k -Means was used to cluster the latent projection with k (the number of clusters) varied between 12 − 1 12 times the true number of categories in the dataset.

Original: The clustering was repeated ﬁve times per k .
Revised: The clustering was repeated ﬁve times per k .

Original: The best clustering was then picked on the basis of the silhouette score between the k clusters and the projections (i.e. without reference to ground truth).
Revised: The best clustering was then picked on the basis of the silhouette score between the k clusters and the projections (i.e. without reference to ground truth).

Original: B F IGURES
Revised: A. R ELATED W ORK

Revised: Beyond Parametric t-SNE and Parametric UMAP, a number of recent parametric dimensionality reduction algorithms utilizing structure-preserving constraints exist which were not compared here. Moor et al. , (topological autoencoders; 2019) and Hoffer et al.

Original: [Figure]
Revised: (Connectivity-Optimized Representation Learning; 2019) apply an additional topological structure-preserving loss using persistent homology over mini-batches to the latent space of an autoencoder. Ding et al. , (scvis; 2018) and Graving and Couzin (VAE-SNE; 2020) describe VAE-derived dimensionality reduction algorithms based upon the ELBO objective.

Original: [Figure]
Revised: Duque et al (geometry-regularized autoencoders; 2020) regularize an autoencoder with the PHATE (Potential of Heat-diffusion for Afﬁnity-based Trajectory Embedding) embedding algorithm (Moon et al., 2017).

Original: [Figure]
Revised: Szubert et al (ivis; 2019) and Robinson (Differential Embedding Networks; 2020) both make use of siamese neural network architectures with structurepreserving loss functions to learn embeddings. B F IGURES bison examples use a t-SNE parplexity of 500 and 150 nearest neighbors in UMAP to more

Original: [Figure]
Revised: [Figure]

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: B. R ESULTS TABLES
Revised: B. R ESULTS TABLES

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: method from Fig 8.
Revised: method from Fig 8.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: Work supported by [witheld for anonymity]
Revised: Work supported by [witheld for anonymity]