Original: F ROM t -SNE TO UMAP WITH CONTRASTIVE LEARNING
Revised: F ROM t -SNE TO UMAP WITH CONTRASTIVE LEARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Neighbor embedding methods t -SNE and UMAP are the de facto standard for visualizing high-dimensional datasets.
Revised: Neighbor embedding methods t -SNE and UMAP are the de facto standard for visualizing high-dimensional datasets.

Original: Motivated from entirely different viewpoints, their loss functions appear to be unrelated.
Revised: Motivated from entirely different viewpoints, their loss functions appear to be unrelated.

Original: In practice, they yield strongly differing embeddings and can suggest conflicting interpretations of the same data.
Revised: In practice, they yield strongly differing embeddings and can suggest conflicting interpretations of the same data.

Original: The fundamental reasons for this and, more generally, the exact relationship between t -SNE and UMAP have remained unclear.
Revised: The fundamental reasons for this and, more generally, the exact relationship between t -SNE and UMAP have remained unclear.

Original: In this work, we uncover their conceptual connection via a new insight into contrastive learning methods.
Revised: In this work, we uncover their conceptual connection via a new insight into contrastive learning methods.

Original: Noisecontrastive estimation can be used to optimize t -SNE, while UMAP relies on negative sampling, another contrastive method.
Revised: Noisecontrastive estimation can be used to optimize t -SNE, while UMAP relies on negative sampling, another contrastive method.

Original: We find the precise relationship between these two contrastive methods, and provide a mathematical characterization of the distortion introduced by negative sampling.
Revised: We find the precise relationship between these two contrastive methods, and provide a mathematical characterization of the distortion introduced by negative sampling.

Original: Visually, this distortion results in UMAP generating more compact embeddings with tighter clusters compared to t -SNE.
Revised: Visually, this distortion results in UMAP generating more compact embeddings with tighter clusters compared to t -SNE.

Original: We exploit this new conceptual connection to propose and implement a generalization of negative sampling, allowing us to interpolate between (and even extrapolate beyond) t -SNE and UMAP and their respective embeddings.
Revised: We exploit this new conceptual connection to propose and implement a generalization of negative sampling, allowing us to interpolate between (and even extrapolate beyond) t -SNE and UMAP and their respective embeddings.

Original: Moving along this spectrum of embeddings leads to a trade-off between discrete/local and continuous/global structures, mitigating the risk of over-interpreting ostensible features of any single embedding.
Revised: Moving along this spectrum of embeddings leads to a trade-off between discrete/local and continuous/global structures, mitigating the risk of over-interpreting ostensible features of any single embedding.

Original: We provide a PyTorch implementation.
Revised: We provide a PyTorch implementation.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Low-dimensional visualization of high-dimensional data is a ubiquitous step in exploratory data analysis, and the toolbox of visualization methods has been rapidly growing in the last years (McInnes et al., 2018;
Revised: Low-dimensional visualization of high-dimensional data is a ubiquitous step in exploratory data analysis, and the toolbox of visualization methods has been rapidly growing in the last years (McInnes et al., 2018;

Original: Amid & Warmuth, 2019; Szubert et al., 2019; Wang et al., 2021).
Revised: Amid & Warmuth, 2019; Szubert et al., 2019; Wang et al., 2021).

Original: Since all of these methods necessarily distort the true data layout (Chari et al., 2021), it is beneficial to have various tools at one’s disposal.
Revised: Since all of these methods necessarily distort the true data layout (Chari et al., 2021), it is beneficial to have various tools at one’s disposal.

Original: But only equipped with a theoretic understanding of the aims of and relationships between different methods, can practitioners make informed decisions about which visualization to use for which purpose and how to interpret the results.
Revised: But only equipped with a theoretic understanding of the aims of and relationships between different methods, can practitioners make informed decisions about which visualization to use for which purpose and how to interpret the results.

Original: The state of the art for non-parametric, non-linear dimensionality reduction relies on the neighbor embedding framework (Hinton & Roweis, 2002).
Revised: The state of the art for non-parametric, non-linear dimensionality reduction relies on the neighbor embedding framework (Hinton & Roweis, 2002).

Original: Its two most popular examples are are t -SNE (van der Maaten & Hinton, 2008; van der Maaten, 2014) and UMAP (McInnes et al., 2018).
Revised: Its two most popular examples are are t -SNE (van der Maaten & Hinton, 2008; van der Maaten, 2014) and UMAP (McInnes et al., 2018).

Original: Both can produce insightful, but qualitatively distinct embeddings.
Revised: Both can produce insightful, but qualitatively distinct embeddings.

Original: However, why their embeddings are different and what exactly is the conceptual relation between their loss functions, has remained elusive.
Revised: However, why their embeddings are different and what exactly is the conceptual relation between their loss functions, has remained elusive.

Original: Here, we answer this question and thus explain the mathematical underpinnings of the relationship between t -SNE and UMAP.
Revised: Here, we answer this question and thus explain the mathematical underpinnings of the relationship between t -SNE and UMAP.

Original: Our conceptual insight naturally suggests a spectrum of embedding methods complementary to that of B¨ohm et al.
Revised: Our conceptual insight naturally suggests a spectrum of embedding methods complementary to that of B¨ohm et al.

Original: (2022), along which the focus of the visualization shifts from global to local structure (Fig. 1).
Revised: (2022), along which the focus of the visualization shifts from global to local structure (Fig. 1).

Original: On this spectrum, UMAP and t -SNE are simply two instances and inspecting various embeddings helps to guard against over-interpretation of apparent structure.
Revised: On this spectrum, UMAP and t -SNE are simply two instances and inspecting various embeddings helps to guard against over-interpretation of apparent structure.

Original: As a practical corollary, our analysis identifies and remedies an instability in UMAP.
Revised: As a practical corollary, our analysis identifies and remedies an instability in UMAP.

Original: We provide the new connection between t -SNE and UMAP via a deeper understanding of contrastive learning methods.
Revised: We provide the new connection between t -SNE and UMAP via a deeper understanding of contrastive learning methods.

Original: Noise-contrastive estimation (NCE) (Gutmann & Hyv¨arinen, 2010; 2012) can be used to optimize t -SNE (Artemenkov & Panov, 2020), while UMAP relies on another contrastive method, negative sampling (NEG) (Mikolov et al., 2013).
Revised: Noise-contrastive estimation (NCE) (Gutmann & Hyv¨arinen, 2010; 2012) can be used to optimize t -SNE (Artemenkov & Panov, 2020), while UMAP relies on another contrastive method, negative sampling (NEG) (Mikolov et al., 2013).

Original: We investigate the discrepancy between
Revised: We investigate the discrepancy between

Original: NCE and NEG, show that NEG introduces a distortion, and this distortion explains how UMAP and
Revised: NCE and NEG, show that NEG introduces a distortion, and this distortion explains how UMAP and

Original: [Figure]
Revised: [Figure]

Original: t -SNE embeddings differ.
Revised: t -SNE embeddings differ.

Original: Finally, we discuss the relationship between neighbor embeddings and self-supervised learning (Wu et al., 2018;
Revised: Finally, we discuss the relationship between neighbor embeddings and self-supervised learning (Wu et al., 2018;

Original: He et al., 2020; Chen et al., 2020; Le-Khac et al., 2020).
Revised: He et al., 2020; Chen et al., 2020; Le-Khac et al., 2020).

Original: In summary our contributions are
Revised: In summary our contributions are

Original: A new connection between the contrastive methods NCE and NEG (Sec. 4).
Revised: A new connection between the contrastive methods NCE and NEG (Sec. 4).

Original: The exact relation of t -SNE and UMAP and a remedy for an instability in UMAP (Sec. 6).
Revised: The exact relation of t -SNE and UMAP and a remedy for an instability in UMAP (Sec. 6).

Original: A spectrum of ‘contrastive’ neighbor embeddings encompassing UMAP and t -SNE (Sec. 5).
Revised: A spectrum of ‘contrastive’ neighbor embeddings encompassing UMAP and t -SNE (Sec. 5).

Original: A connection between neighbor embeddings and self-supervised learning (Sec. 7).
Revised: A connection between neighbor embeddings and self-supervised learning (Sec. 7).

Original: A unified PyTorch framework for contrastive (non-)parametric neighbor embedding methods.
Revised: A unified PyTorch framework for contrastive (non-)parametric neighbor embedding methods.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: One of the most popular methods for data visualization is t -SNE (van der Maaten & Hinton, 2008; van der Maaten, 2014).
Revised: One of the most popular methods for data visualization is t -SNE (van der Maaten & Hinton, 2008; van der Maaten, 2014).

Original: Recently developed NCVis (Artemenkov & Panov, 2020) employs noisecontrastive estimation (Gutmann & Hyv¨arinen, 2010; 2012) to approximate t -SNE in a samplingbased way.
Revised: Recently developed NCVis (Artemenkov & Panov, 2020) employs noisecontrastive estimation (Gutmann & Hyv¨arinen, 2010; 2012) to approximate t -SNE in a samplingbased way.

Original: UMAP (McInnes et al., 2018) has matched t -SNE’s popularity at least in computational biology (Becht et al., 2019), and also uses sampling-based optimization, namely negative sampling (Mikolov et al., 2013), also employed by LargeVis (Tang et al., 2016).
Revised: UMAP (McInnes et al., 2018) has matched t -SNE’s popularity at least in computational biology (Becht et al., 2019), and also uses sampling-based optimization, namely negative sampling (Mikolov et al., 2013), also employed by LargeVis (Tang et al., 2016).

Original: Other recent samplingbased visualization methods include TriMap (Amid & Warmuth, 2019), which uses a triplet-based loss, and PacMap (Wang et al., 2021).
Revised: Other recent samplingbased visualization methods include TriMap (Amid & Warmuth, 2019), which uses a triplet-based loss, and PacMap (Wang et al., 2021).

Original: Given their success, t -SNE and UMAP have been scrutinized to find out which aspects are essential to their performance.
Revised: Given their success, t -SNE and UMAP have been scrutinized to find out which aspects are essential to their performance.

Original: Initialization was found to be strongly influencing the global structure in for both methods (Kobak & Linderman, 2021).
Revised: Initialization was found to be strongly influencing the global structure in for both methods (Kobak & Linderman, 2021).

Original: The exact choice of the low-dimensional similarity kernel (B¨ohm et al., 2022) or weights of the k -nearest-neighbor graph (Damrich & Hamprecht, 2021) were shown to be largely inconsequential.
Revised: The exact choice of the low-dimensional similarity kernel (B¨ohm et al., 2022) or weights of the k -nearest-neighbor graph (Damrich & Hamprecht, 2021) were shown to be largely inconsequential.

Original: Both algorithms have similar relevant hyper-parameters such as, e.g., the heavy-tailedness of the similarity kernel (Yang et al., 2009;
Revised: Both algorithms have similar relevant hyper-parameters such as, e.g., the heavy-tailedness of the similarity kernel (Yang et al., 2009;

Original: Kobak et al., 2019).
Revised: Kobak et al., 2019).

Original: While not obvious from the original presentations, the central difference between t -SNE and UMAP can therefore only be in their loss functions, which have been studied by Damrich & Hamprecht
Revised: While not obvious from the original presentations, the central difference between t -SNE and UMAP can therefore only be in their loss functions, which have been studied by Damrich & Hamprecht

Original: (2021); B¨ohm et al.
Revised: (2021); B¨ohm et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: (2021), but never conceptually connected.
Revised: (2021), but never conceptually connected.

Original: We achieve this by deepening the link between negative sampling (NEG) and noise-contrastive estimation (NCE).
Revised: We achieve this by deepening the link between negative sampling (NEG) and noise-contrastive estimation (NCE).

Original: NEG was introduced as an ad hoc replacement for NCE in the context of learning word embeddings (Mikolov et al., 2013).
Revised: NEG was introduced as an ad hoc replacement for NCE in the context of learning word embeddings (Mikolov et al., 2013).

Original: The relationship between NEG and NCE has been discussed before (Dyer, 2014; Levy & Goldberg, 2014; Ruder, 2016; Ma & Collins, 2018; Le-Khac et al., 2020), but here we go further and provide the precise meaning of NEG: We show that, unlike NCE, NEG learns a model proportional but not equal to the true data distribution.
Revised: The relationship between NEG and NCE has been discussed before (Dyer, 2014; Levy & Goldberg, 2014; Ruder, 2016; Ma & Collins, 2018; Le-Khac et al., 2020), but here we go further and provide the precise meaning of NEG: We show that, unlike NCE, NEG learns a model proportional but not equal to the true data distribution.

Original: Both t -SNE and UMAP have parametric versions (van der Maaten, 2009; Sainburg et al., 2021) with very different logic and implementations.
Revised: Both t -SNE and UMAP have parametric versions (van der Maaten, 2009; Sainburg et al., 2021) with very different logic and implementations.

Original: Here we present a unified PyTorch framework for nonparametric and parametric contrastive neighbor embeddings.
Revised: Here we present a unified PyTorch framework for nonparametric and parametric contrastive neighbor embeddings.

Original: As special cases, it includes UMAP and t -SNE approximations with NCE (like NCVis) and the InfoNCE loss (Jozefowicz et al., 2016; van den Oord et al., 2018), which has not yet been applied to neighbor embeddings.
Revised: As special cases, it includes UMAP and t -SNE approximations with NCE (like NCVis) and the InfoNCE loss (Jozefowicz et al., 2016; van den Oord et al., 2018), which has not yet been applied to neighbor embeddings.

Original: Our InfoNC- t -SNE elucidates the relationship between SimCLR (Chen et al., 2020) and neighbor embeddings.
Revised: Our InfoNC- t -SNE elucidates the relationship between SimCLR (Chen et al., 2020) and neighbor embeddings.

Original: The concurrent work of Hu et al.
Revised: The concurrent work of Hu et al.

Original: (2022) makes a qualitatively similar argument, but does not talk about negative samples.
Revised: (2022) makes a qualitatively similar argument, but does not talk about negative samples.

Original: Balestriero & LeCun (2022) show how SimCLR recovers
Revised: Balestriero & LeCun (2022) show how SimCLR recovers

Original: Isomap (Tenenbaum et al., 2000), while we connect SimCLR to more modern neighbor embeddings.
Revised: Isomap (Tenenbaum et al., 2000), while we connect SimCLR to more modern neighbor embeddings.

Original:  B ACKGROUND 3. N OISE - CONTRASTIVE ESTIMATION (NCE)
Revised:  B ACKGROUND 3. N OISE - CONTRASTIVE ESTIMATION (NCE)

Original: The goal of parametric density estimation is to fit a parametric model q θ to iid samples s 1 , . . .
Revised: The goal of parametric density estimation is to fit a parametric model q θ to iid samples s 1 , . . .

Original: , s N from an unknown data distribution p over a space X .
Revised: , s N from an unknown data distribution p over a space X .

Original: For maximum likelihood estimation (MLE) the parameters θ are chosen to maximize the log-likelihood of the observed samples
Revised: For maximum likelihood estimation (MLE) the parameters θ are chosen to maximize the log-likelihood of the observed samples

Original: [Equation]
Revised: [Equation]

Original: This approach crucially requires q θ to be a normalized model.
Revised: This approach crucially requires q θ to be a normalized model.

Original: It is otherwise trivial to increase the likelihood arbitrarily by scaling q θ .
Revised: It is otherwise trivial to increase the likelihood arbitrarily by scaling q θ .

Original: To circumvent the expensive computation of the partition function Z ( θ ) = (cid:80) x
Revised: To circumvent the expensive computation of the partition function Z ( θ ) = (cid:80) x

Original: ∈ X q θ ( x ) , Gutmann & Hyv ¨ arinen (2010; 2012) introduced NCE.
Revised: ∈ X q θ ( x ) , Gutmann & Hyv ¨ arinen (2010; 2012) introduced NCE.

Original: It turns the unsupervised problem of density estimation into a supervised problem in which the data samples need to be identified from a set containing the N data samples and m times as many noise samples t 1 , . . .
Revised: It turns the unsupervised problem of density estimation into a supervised problem in which the data samples need to be identified from a set containing the N data samples and m times as many noise samples t 1 , . . .

Original: t mN drawn from a noise distribution ξ , e.g., the uniform distribution over X .
Revised: t mN drawn from a noise distribution ξ , e.g., the uniform distribution over X .

Original: Briefly,
Revised: Briefly,

Original: NCE fits θ by minimizing the binary cross-entropy between the true class assignment and posterior probabilities P ( data | x ) =
Revised: NCE fits θ by minimizing the binary cross-entropy between the true class assignment and posterior probabilities P ( data | x ) =

Original: q θ ( x ) / (cid:0) q θ ( x )+ mξ ( x ) (cid:1) and P ( noise | x ) = 1 − P ( data | x ) (Supp.
Revised: q θ ( x ) / (cid:0) q θ ( x )+ mξ ( x ) (cid:1) and P ( noise | x ) = 1 − P ( data | x ) (Supp.

Original: A.1):
Revised: A.1):

Original: [Equation]
Revised: [Equation]

Original: The key advantage of NCE is that the model does not need to be explicitly normalized by the partition function, but nevertheless learns to equal the data distribution p and hence be normalized:
Revised: The key advantage of NCE is that the model does not need to be explicitly normalized by the partition function, but nevertheless learns to equal the data distribution p and hence be normalized:

Original: Theorem 1 (Gutmann & Hyv¨arinen (2010; 2012)) .
Revised: Theorem 1 (Gutmann & Hyv¨arinen (2010; 2012)) .

Original: Suppose there exists some θ ∗ such that q θ ∗ = p .
Revised: Suppose there exists some θ ∗ such that q θ ∗ = p .

Original: Then θ ∗ is a minimum of
Revised: Then θ ∗ is a minimum of

Original: [Equation]
Revised: [Equation]

Original: and any other minimum ˜ θ also satisfies q ˜ θ = p .
Revised: and any other minimum ˜ θ also satisfies q ˜ θ = p .

Original: In addition, if ξ ( x ) is non-zero wherever p ( x ) is non-zero, then these are the only extrema of L NCE ( θ ) .
Revised: In addition, if ξ ( x ) is non-zero wherever p ( x ) is non-zero, then these are the only extrema of L NCE ( θ ) .

Original: In NCE, the model typically includes an optimizable normalization parameter Z which we emphasize by writing q θ,Z = q θ /Z .
Revised: In NCE, the model typically includes an optimizable normalization parameter Z which we emphasize by writing q θ,Z = q θ /Z .

Original: But importantly, Thm. 1 applies to any model q θ that is able to match the data distribution p , even if it does not contain a learnable normalization parameter.
Revised: But importantly, Thm. 1 applies to any model q θ that is able to match the data distribution p , even if it does not contain a learnable normalization parameter.

Original: In the setting of learning language models, Jozefowicz et al.
Revised: In the setting of learning language models, Jozefowicz et al.

Original: (2016) proposed a different version of
Revised: (2016) proposed a different version of

Original: NCE, called InfoNCE.
Revised: NCE, called InfoNCE.

Original: Instead of classifying samples as data or noise, the aim here is to predict the position of a data sample in an ( m + 1) -tuple
Revised: Instead of classifying samples as data or noise, the aim here is to predict the position of a data sample in an ( m + 1) -tuple

Original: containing m noise samples and one data sample (Supp.
Revised: containing m noise samples and one data sample (Supp.

Original: A.2).
Revised: A.2).

Original: For the uniform noise distribution ξ , this yields the expected loss function
Revised: For the uniform noise distribution ξ , this yields the expected loss function

Original: [Equation]
Revised: [Equation]

Original: Ma & Collins (2018) showed that an analogue of Thm. 1 applies to InfoNCE.
Revised: Ma & Collins (2018) showed that an analogue of Thm. 1 applies to InfoNCE.

Original: 3. N EIGHBOR EMBEDDINGS
Revised: 3. N EIGHBOR EMBEDDINGS

Original: Neighbor embeddings (NE) (Hinton & Roweis, 2002) are a group of dimensionality reduction methods, including UMAP, NCVis, and t -SNE, that aim to find a low-dimensional embedding e 1 , . .
Revised: Neighbor embeddings (NE) (Hinton & Roweis, 2002) are a group of dimensionality reduction methods, including UMAP, NCVis, and t -SNE, that aim to find a low-dimensional embedding e 1 , . .

Original: , e n ∈ R d of high-dimensional input points x 1 , . . .
Revised: , e n ∈ R d of high-dimensional input points x 1 , . . .

Original: , x n ∈ R D , with D ≫ d and usually d = 2 for visualization.
Revised: , x n ∈ R D , with D ≫ d and usually d = 2 for visualization.

Original: NE methods define a notion of similarity over pairs of input points which encodes the neighborhood structure and informs the low-dimensional embedding.
Revised: NE methods define a notion of similarity over pairs of input points which encodes the neighborhood structure and informs the low-dimensional embedding.

Original: The exact high-dimensional similarity distribution differs between the NE algorithms, but recent work (B¨ohm et al., 2022; Damrich & Hamprecht, 2021) showed that t -SNE and UMAP results stay practically the same when using the binary symmetric k -nearest-neighbor graph (s k NN) instead of t -SNE’s Gaussian or UMAP’s Laplacian similarities.
Revised: The exact high-dimensional similarity distribution differs between the NE algorithms, but recent work (B¨ohm et al., 2022; Damrich & Hamprecht, 2021) showed that t -SNE and UMAP results stay practically the same when using the binary symmetric k -nearest-neighbor graph (s k NN) instead of t -SNE’s Gaussian or UMAP’s Laplacian similarities.

Original: An edge ij is in s k NN if x i is among the k nearest neighbors of x j or vice versa.
Revised: An edge ij is in s k NN if x i is among the k nearest neighbors of x j or vice versa.

Original: The high-dimensional similarity function is then given by p ( ij )
Revised: The high-dimensional similarity function is then given by p ( ij )

Original: = 1 ( ij ∈ s k NN ) / | s k NN | , where | s k NN | denotes the number of edges in the s k NN graph andis the indicator function.
Revised: = 1 ( ij ∈ s k NN ) / | s k NN | , where | s k NN | denotes the number of edges in the s k NN graph andis the indicator function.

Original: NCVis uses the same similarities.
Revised: NCVis uses the same similarities.

Original: There are further differences in the choice of low-dimensional similarity between t -SNE and UMAP, but B ¨ ohm et al.
Revised: There are further differences in the choice of low-dimensional similarity between t -SNE and UMAP, but B ¨ ohm et al.

Original: (2022) showed that they are negligible.
Revised: (2022) showed that they are negligible.

Original: Therefore, here we use the Cauchy kernel ϕ ( d ij )
Revised: Therefore, here we use the Cauchy kernel ϕ ( d ij )

Original: = 1 / ( d 2 ij + 1) for all NE methods to transform distances d ij = ∥
Revised: = 1 / ( d 2 ij + 1) for all NE methods to transform distances d ij = ∥

Original: e i − e j ∥ in the embedding space into low-dimensional similarities.
Revised: e i − e j ∥ in the embedding space into low-dimensional similarities.

Original: We abuse notation slightly by also writing ϕ ( ij ) =
Revised: We abuse notation slightly by also writing ϕ ( ij ) =

Original: ϕ
Revised: ϕ

Original: ( d ij ) .
Revised: ( d ij ) .

Original: All NE methods in this work can be cast in the framework of parametric density estimation.
Revised: All NE methods in this work can be cast in the framework of parametric density estimation.

Original: Here, p is the data distribution to be approximated with a model q θ , meaning that the space X on which both p and q θ live is the set of all pairs ij with 1 ≤ i < j ≤ n .
Revised: Here, p is the data distribution to be approximated with a model q θ , meaning that the space X on which both p and q θ live is the set of all pairs ij with 1 ≤ i < j ≤ n .

Original: The embedding positions e 1 , . .
Revised: The embedding positions e 1 , . .

Original: , e n become the learnable parameters θ of the model q θ .
Revised: , e n become the learnable parameters θ of the model q θ .

Original: For t -SNE, NCVis, and UMAP, q θ is proportional to ϕ ( ∥ e i − e j ∥ ) , but the proportionality and the loss functions are different.
Revised: For t -SNE, NCVis, and UMAP, q θ is proportional to ϕ ( ∥ e i − e j ∥ ) , but the proportionality and the loss functions are different.

Original: t -SNE uses MLE and therefore requires a normalized model q θ ( ij ) =
Revised: t -SNE uses MLE and therefore requires a normalized model q θ ( ij ) =

Original: ϕ ( ij ) /Z
Revised: ϕ ( ij ) /Z

Original: ( θ ) , where
Revised: ( θ ) , where

Original: Z ( θ ) = (cid:80) k ̸ = l ϕ ( kl ) is the partition function.
Revised: Z ( θ ) = (cid:80) k ̸ = l ϕ ( kl ) is the partition function.

Original: The loss function
Revised: The loss function

Original: [Equation]
Revised: [Equation]

Original: (5) is the expected negative log-likelihood of the embedding positions θ , making t -SNE an instance of
Revised: (5) is the expected negative log-likelihood of the embedding positions θ , making t -SNE an instance of

Original: MLE.
Revised: MLE.

Original: Usually t -SNE’s loss function is introduced as the Kullback-Leibler divergence between p and q θ , which is equivalent as the entropy of p does not depend on θ .
Revised: Usually t -SNE’s loss function is introduced as the Kullback-Leibler divergence between p and q θ , which is equivalent as the entropy of p does not depend on θ .

Original: NCVis uses NCE and optimizes the expected loss function
Revised: NCVis uses NCE and optimizes the expected loss function

Original: [Equation]
Revised: [Equation]

Original: where q θ,Z ( ij ) =
Revised: where q θ,Z ( ij ) =

Original: ϕ ( ij ) /Z with learnable Z and ξ is approximately uniform (see Supp.
Revised: ϕ ( ij ) /Z with learnable Z and ξ is approximately uniform (see Supp.

Original: D).
Revised: D).

Original: According to Thm. 1, NCVis has the same optimum as t -SNE and can hence be seen as a samplingbased approximation of t -SNE.
Revised: According to Thm. 1, NCVis has the same optimum as t -SNE and can hence be seen as a samplingbased approximation of t -SNE.

Original: Indeed, we found that Z in NCVis and the partition function Z ( θ ) in t -SNE converge approximately to the same value (Fig. S9).
Revised: Indeed, we found that Z in NCVis and the partition function Z ( θ ) in t -SNE converge approximately to the same value (Fig. S9).

Original: UMAP ’s expected loss function is derived in (Damrich & Hamprecht, 2021):
Revised: UMAP ’s expected loss function is derived in (Damrich & Hamprecht, 2021):

Original: [Equation]
Revised: [Equation]

Original: with q θ ( ij ) = ϕ ( ij ) and ξ is approximately uniform, see Supp.
Revised: with q θ ( ij ) = ϕ ( ij ) and ξ is approximately uniform, see Supp.

Original: D. This is the effective loss function actually implemented in the UMAP algorithm, but note that it has only about m/n of the repulsion compared to the loss stated in the original UMAP paper (McInnes et al., 2018), as shown in (B¨ohm et al., 2022;
Revised: D. This is the effective loss function actually implemented in the UMAP algorithm, but note that it has only about m/n of the repulsion compared to the loss stated in the original UMAP paper (McInnes et al., 2018), as shown in (B¨ohm et al., 2022;

Original: Damrich & Hamprecht, 2021)
Revised: Damrich & Hamprecht, 2021)

Original: (Supp. C).
Revised: (Supp. C).

Original: In practice, the expectations in UMAP’s and NCVis’ loss functions are evaluated via sampling, like in Eq.
Revised: In practice, the expectations in UMAP’s and NCVis’ loss functions are evaluated via sampling, like in Eq.

Original: This leads to a fast, O ( n ) , stochastic gradient descent optimization scheme.
Revised: This leads to a fast, O ( n ) , stochastic gradient descent optimization scheme.

Original: Both loss functions are composed of an attractive term pulling similar data points (edges of the s k NN graph) closer together, and a repulsive term pushing random pairs of points further apart.
Revised: Both loss functions are composed of an attractive term pulling similar data points (edges of the s k NN graph) closer together, and a repulsive term pushing random pairs of points further apart.

Original: Similarly, t -SNE’s loss yields attraction along the graph edges while repulsion arises through the normalization term.
Revised: Similarly, t -SNE’s loss yields attraction along the graph edges while repulsion arises through the normalization term.

Original:  F ROM NOISE - CONTRASTIVE ESTIMATION TO NEGATIVE SAMPLING
Revised:  F ROM NOISE - CONTRASTIVE ESTIMATION TO NEGATIVE SAMPLING

Original: In this section we work out the precise relationship between NCE and NEG, going beyond prior work (Dyer, 2014; Goldberg & Levy, 2014; Levy & Goldberg, 2014; Ruder, 2016).
Revised: In this section we work out the precise relationship between NCE and NEG, going beyond prior work (Dyer, 2014; Goldberg & Levy, 2014; Levy & Goldberg, 2014; Ruder, 2016).

Original: NEG differs from NCE by its loss function and by the lack of the learnable normalization parameter Z .
Revised: NEG differs from NCE by its loss function and by the lack of the learnable normalization parameter Z .

Original: In our setting, NEG’s loss function amounts to
Revised: In our setting, NEG’s loss function amounts to

Original: [Equation]
Revised: [Equation]

Original: In order to relate it to NCE’s loss function, our key insight is to generalize the latter by allowing to learn a model that is not equal but proportional to the true data distribution.
Revised: In order to relate it to NCE’s loss function, our key insight is to generalize the latter by allowing to learn a model that is not equal but proportional to the true data distribution.

Original: Corollary 2.
Revised: Corollary 2.

Original: Let ¯ Z, m ∈ R + .
Revised: Let ¯ Z, m ∈ R + .

Original: Assume that ξ ( x ) is non-zero wherever p ( x ) is non-zero and that there exist θ ∗ such that q θ ∗ =
Revised: Assume that ξ ( x ) is non-zero wherever p ( x ) is non-zero and that there exist θ ∗ such that q θ ∗ =

Original: ¯ Zp .
Revised: ¯ Zp .

Original: Then the generalized NCE loss function
Revised: Then the generalized NCE loss function

Original: [Equation]
Revised: [Equation]

Original: has its only extrema where q θ = ¯ Zp .
Revised: has its only extrema where q θ = ¯ Zp .

Original: Proof.
Revised: Proof.

Original: The result follows from Thm.
Revised: The result follows from Thm.

Original: 1 applied to the model distribution ˜
Revised: 1 applied to the model distribution ˜

Original: q
Revised: q

Original: θ :=
Revised: θ :=

Original: q θ / ¯ Z .
Revised: q θ / ¯ Z .

Original: Dyer (2014) and Ruder (2016) pointed out that for a uniform noise distribution ξ ( x ) = 1 / | X | and as many noise samples as the size of X ( m = | X | ), the loss functions of NCE and NEG coincide, since mξ ( x ) = 1 .
Revised: Dyer (2014) and Ruder (2016) pointed out that for a uniform noise distribution ξ ( x ) = 1 / | X | and as many noise samples as the size of X ( m = | X | ), the loss functions of NCE and NEG coincide, since mξ ( x ) = 1 .

Original: However, the main point of NCE and NEG is to use far fewer noise samples in order to attain a speed-up over MLE.
Revised: However, the main point of NCE and NEG is to use far fewer noise samples in order to attain a speed-up over MLE.

Original: Our Cor. 

Revised: Our Cor.

Original: for the first time explains NEG’s behavior in this more realistic setting ( m ≪ | X | ).
Revised: for the first time explains NEG’s behavior in this more realistic setting ( m ≪ | X | ).

Original: If the noise distribution is uniform, the generalized NCE loss function with ¯ Z = | X | /m equals the NEG loss function since ( | X | /m ) mξ ( x ) = 1 .
Revised: If the noise distribution is uniform, the generalized NCE loss function with ¯ Z = | X | /m equals the NEG loss function since ( | X | /m ) mξ ( x ) = 1 .

Original: By Cor. 2, any minimum θ ∗ of the NEG loss function yields q θ ∗ =
Revised: By Cor. 2, any minimum θ ∗ of the NEG loss function yields q θ ∗ =

Original: ( | X | /m ) p , assuming that there are parameters that make this equation hold.
Revised: ( | X | /m ) p , assuming that there are parameters that make this equation hold.

Original: In other words, NEG aims to find a model q θ that is proportional to the data distribution with the proportionality factor | X | /m which is typically huge.
Revised: In other words, NEG aims to find a model q θ that is proportional to the data distribution with the proportionality factor | X | /m which is typically huge.

Original: This is different from NCE, which aims to learn a model equal to the data distribution.
Revised: This is different from NCE, which aims to learn a model equal to the data distribution.

Original: Choosing m ≪
Revised: Choosing m ≪

Original: | X | does not only offer a computational speed-up but is necessary when optimizing
Revised: | X | does not only offer a computational speed-up but is necessary when optimizing

Original: NEG for a neural network with SGD, like we do in Sec.7.
Revised: NEG for a neural network with SGD, like we do in Sec.7.

Original: Only one single mini-batch is passed through the neural network during each iteration and is thus available for computing the loss.
Revised: Only one single mini-batch is passed through the neural network during each iteration and is thus available for computing the loss.

Original: Hence, all noise samples must come from the current mini-batch and their number m is upper-bounded by the mini-batch size b .
Revised: Hence, all noise samples must come from the current mini-batch and their number m is upper-bounded by the mini-batch size b .

Original: Mini-batches are typically much smaller than | X | .
Revised: Mini-batches are typically much smaller than | X | .

Original: Thus, this common training procedure necessitates m ≪ | X | highlighting the relevance of Cor. 2.
Revised: Thus, this common training procedure necessitates m ≪ | X | highlighting the relevance of Cor. 2.

Original: While NCE uses a model q θ /Z with learnable Z , we can interpret NEG as using a model q θ / ¯ Z with fixed and very large normalization constant ¯ Z = | X | /m .
Revised: While NCE uses a model q θ /Z with learnable Z , we can interpret NEG as using a model q θ / ¯ Z with fixed and very large normalization constant ¯ Z = | X | /m .

Original: As a result, q θ in NEG needs to attain much larger values to match the large ¯ Z .
Revised: As a result, q θ in NEG needs to attain much larger values to match the large ¯ Z .

Original: This can be illustrated in the setting of neighbor embeddings.
Revised: This can be illustrated in the setting of neighbor embeddings.

Original: Applying NEG to the neighbor embedding framework yields an algorithm that we call ‘Neg- t -SNE’.
Revised: Applying NEG to the neighbor embedding framework yields an algorithm that we call ‘Neg- t -SNE’.

Original: Recall that in this setting, | X | = (cid:0) n 2 (cid:1) is the number of pairs of points.
Revised: Recall that in this setting, | X | = (cid:0) n 2 (cid:1) is the number of pairs of points.

Original: B¨ohm et al.
Revised: B¨ohm et al.

Original: (2022) found empirically that t -SNE’s partition function
Revised: (2022) found empirically that t -SNE’s partition function

Original: Z ( θ ) is typically between 50 n and 100 n , while in Neg- t -SNE, ¯ Z
Revised: Z ( θ ) is typically between 50 n and 100 n , while in Neg- t -SNE, ¯ Z

Original: = O ( n 2 ) is much larger for modern big datasets.
Revised: = O ( n 2 ) is much larger for modern big datasets.

Original: To attain the larger values of ϕ ( ij ) required by NEG, points that are connected in the s k NN graph have to move much closer together in the embedding than in t -SNE.
Revised: To attain the larger values of ϕ ( ij ) required by NEG, points that are connected in the s k NN graph have to move much closer together in the embedding than in t -SNE.

Original: Indeed, using our PyTorch implementation of Neg- t -SNE on the MNIST dataset, we confirmed that Neg- t
Revised: Indeed, using our PyTorch implementation of Neg- t -SNE on the MNIST dataset, we confirmed that Neg- t

Original: -SNE
Revised: -SNE

Original: Fig. 1d) produced more compact clusters than t -SNE (Fig. 1f).
Revised: Fig. 1d) produced more compact clusters than t -SNE (Fig. 1f).

Original: See Supp.
Revised: See Supp.

Original: K and Alg. S1 for implementation details.
Revised: K and Alg. S1 for implementation details.

Original: We emphasize that the conclusions of this section only hold because NEG does not contain a learnable normalization parameter Z .
Revised: We emphasize that the conclusions of this section only hold because NEG does not contain a learnable normalization parameter Z .

Original: If it did, then such a learnable Z would be able to absorb the term ¯
Revised: If it did, then such a learnable Z would be able to absorb the term ¯

Original: Z in Eq.
Revised: Z in Eq.

Original: (9) while leaving the parameters θ unchanged.
Revised: (9) while leaving the parameters θ unchanged.

Original: [Figure]
Revised: [Figure]

Original: reference implementation annealing,
Revised: reference implementation annealing,

Original:  N EGATIVE SAMPLING SPECTRUM
Revised:  N EGATIVE SAMPLING SPECTRUM

Original: Varying the fixed normalization constant ¯ Z in Eq.
Revised: Varying the fixed normalization constant ¯ Z in Eq.

Original: (9) has important practical effects that lead to a whole spectrum of embeddings in the NE setting.
Revised: (9) has important practical effects that lead to a whole spectrum of embeddings in the NE setting.

Original: The original NEG loss function Eq.
Revised: The original NEG loss function Eq.

Original: (8) corresponds to Eq.
Revised: (8) corresponds to Eq.

Original: (9) with ¯ Z = | X | /m .
Revised: (9) with ¯ Z = | X | /m .

Original: We still refer to the more general case of using an arbitrary ¯ Z in Eq.
Revised: We still refer to the more general case of using an arbitrary ¯ Z in Eq.

Original: (9) as ‘negative sampling’, and ‘Neg- t -SNE’ in the context of neighbor embeddings.
Revised: (9) as ‘negative sampling’, and ‘Neg- t -SNE’ in the context of neighbor embeddings.

Original: Figs.
Revised: Figs.

Original: 1a–e show a spectrum of Neg- t -SNE visualizations of the MNIST dataset for varying ¯ Z .
Revised: 1a–e show a spectrum of Neg- t -SNE visualizations of the MNIST dataset for varying ¯ Z .

Original: Per
Revised: Per

Original: Cor. 2, higher values of ¯ Z induce higher values for q θ , meaning that points move closer together.
Revised: Cor. 2, higher values of ¯ Z induce higher values for q θ , meaning that points move closer together.

Original: Indeed, the scale of the embedding decreases for higher ¯ Z as evident from the scale bars in the bottom-right corner of each plot.
Revised: Indeed, the scale of the embedding decreases for higher ¯ Z as evident from the scale bars in the bottom-right corner of each plot.

Original: Moreover, clusters become increasingly compact and then even start to merge.
Revised: Moreover, clusters become increasingly compact and then even start to merge.

Original: For lower values of ¯ Z the embedding scale is larger, clusters are more spread out, and eventually lose almost any separation and start to overlap for very small ¯ Z .
Revised: For lower values of ¯ Z the embedding scale is larger, clusters are more spread out, and eventually lose almost any separation and start to overlap for very small ¯ Z .

Original: Cor.
Revised: Cor.

Original: 2 implies that the partition function (cid:80) x q θ ( x ) should grow with ¯ Z , and indeed this is what we observed (Fig. 1i).
Revised: 2 implies that the partition function (cid:80) x q θ ( x ) should grow with ¯ Z , and indeed this is what we observed (Fig. 1i).

Original: The match between between the sum of Cauchy kernels (cid:80) ij ϕ ( ij ) and ¯ Z was not perfect, but that was expected:
Revised: The match between between the sum of Cauchy kernels (cid:80) ij ϕ ( ij ) and ¯ Z was not perfect, but that was expected:

Original: The Cauchy kernel is bounded by 1 from above, so values ¯
Revised: The Cauchy kernel is bounded by 1 from above, so values ¯

Original: Z > (cid:0) n 2 (cid:1) are not matchable.
Revised: Z > (cid:0) n 2 (cid:1) are not matchable.

Original: Similarly, very small values of ¯ Z are difficult to match because of the heavy tail of the Cauchy kernel.
Revised: Similarly, very small values of ¯ Z are difficult to match because of the heavy tail of the Cauchy kernel.

Original: See also Supp.
Revised: See also Supp.

Original: I for a toy example where the match is perfect.
Revised: I for a toy example where the match is perfect.

Original: By adjusting the ¯ Z value, one can obtain Neg- t -SNE embeddings very similar to NCVis and t -SNE.
Revised: By adjusting the ¯ Z value, one can obtain Neg- t -SNE embeddings very similar to NCVis and t -SNE.

Original: If the NCVis loss function Eq.
Revised: If the NCVis loss function Eq.

Original: (6) has its minimum at some θ ∗ and Z NCVis , then the Neg- t -SNE loss function Eq.
Revised: (6) has its minimum at some θ ∗ and Z NCVis , then the Neg- t -SNE loss function Eq.

Original: (9) with ¯ Z = Z NCVis is minimal at the same θ ∗ .
Revised: (9) with ¯ Z = Z NCVis is minimal at the same θ ∗ .

Original: We confirmed this experimentally:
Revised: We confirmed this experimentally:

Original: Setting ¯ Z = Z NCVis , yields a Neg- t -SNE embedding (Fig. 1c) closely resembling that of NCVis (Fig. 1g).
Revised: Setting ¯ Z = Z NCVis , yields a Neg- t -SNE embedding (Fig. 1c) closely resembling that of NCVis (Fig. 1g).

Original: Similarly, setting ¯ Z to the partition function Z ( θ t -SNE ) , obtained by running t -SNE, yields a Neg- t -SNE embedding closely resembling that of t -SNE (compare Figs.
Revised: Similarly, setting ¯ Z to the partition function Z ( θ t -SNE ) , obtained by running t -SNE, yields a Neg- t -SNE embedding closely resembling that of t -SNE (compare Figs.

Original: 1b and 1f).
Revised: 1b and 1f).

Original: Our quantitative analysis in Supp.
Revised: Our quantitative analysis in Supp.

Original: H supports this finding.
Revised: H supports this finding.

Original: The Neg- t -SNE spectrum is strongly related to the attraction-repulsion spectrum from
Revised: The Neg- t -SNE spectrum is strongly related to the attraction-repulsion spectrum from

Original: B¨ohm et al.
Revised: B¨ohm et al.

Original: They introduced a prefactor (‘exaggeration’) to the repulsive term in the t -SNE’s loss, which exaggerates the attraction over the repulsion, and obtained embeddings similar to our spectrum when varying this parameter.
Revised: They introduced a prefactor (‘exaggeration’) to the repulsive term in the t -SNE’s loss, which exaggerates the attraction over the repulsion, and obtained embeddings similar to our spectrum when varying this parameter.

Original: We can explain this as follows.
Revised: We can explain this as follows.

Original: The repulsive term in the
Revised: The repulsive term in the

Original: NCE loss Eq.
Revised: NCE loss Eq.

Original: (3) has a prefactor m and our spectrum arises from the loss Eq.
Revised: (3) has a prefactor m and our spectrum arises from the loss Eq.

Original: (9) by varying ¯ Z in the term ¯ Zm .
Revised: (9) by varying ¯ Z in the term ¯ Zm .

Original: Equivalently, our spectrum can be obtained by varying the m value (number of noise samples per one s k NN edge) while holding ¯ Z fixed (Fig. S7).
Revised: Equivalently, our spectrum can be obtained by varying the m value (number of noise samples per one s k NN edge) while holding ¯ Z fixed (Fig. S7).

Original: In other words, our spectrum arises from varying the repulsion strength in the contrastive loss setting, while B¨ohm et al.
Revised: In other words, our spectrum arises from varying the repulsion strength in the contrastive loss setting, while B¨ohm et al.

Original: (2022) obtained the analogous spectrum by varying the repulsion strength in the t -SNE setting.
Revised: (2022) obtained the analogous spectrum by varying the repulsion strength in the t -SNE setting.

Original:  UMAP’ S CONCEPTUAL RELATION TO t -SNE
Revised:  UMAP’ S CONCEPTUAL RELATION TO t -SNE

Original: Our comparison of NEG and NCE in Sec.
Revised: Our comparison of NEG and NCE in Sec.

Original: 4 allows us for the first time to conceptually relate UMAP and t -SNE.
Revised: 4 allows us for the first time to conceptually relate UMAP and t -SNE.

Original: Originally McInnes et al.
Revised: Originally McInnes et al.

Original: (2018) motivated the UMAP algorithm by a specific choice of weights on the s k NN graph and the binary cross-entropy loss.
Revised: (2018) motivated the UMAP algorithm by a specific choice of weights on the s k NN graph and the binary cross-entropy loss.

Original: Following LargeVis (Tang et al., 2016), they implemented a sampling-based scheme which they referred to as ‘negative sampling’, although UMAP’s loss function Eq.
Revised: Following LargeVis (Tang et al., 2016), they implemented a sampling-based scheme which they referred to as ‘negative sampling’, although UMAP’s loss function Eq.

Original: (7) does not look like the NEG loss Eq.
Revised: (7) does not look like the NEG loss Eq.

Original: Thus, it has been
Revised: Thus, it has been

Original: [Figure]
Revised: [Figure]

Original: unclear if UMAP actually uses Mikolov et al.
Revised: unclear if UMAP actually uses Mikolov et al.

Original: (2013)’s
Revised: (2013)’s

Original: NEG.
Revised: NEG.

Original: Here, we settle this question by casting
Revised: Here, we settle this question by casting

Original: UMAP as a proper instance of NEG:
Revised: UMAP as a proper instance of NEG:

Original: Lemma 3.
Revised: Lemma 3.

Original: UMAP’s loss function (Eq. 7) is NEG (Eq.
Revised: UMAP’s loss function (Eq. 7) is NEG (Eq.

Original: 8) with the parametric model ˜
Revised: 8) with the parametric model ˜

Original: q θ ( ij ) = 1 /d 2 ij .
Revised: q θ ( ij ) = 1 /d 2 ij .

Original: Proof.
Revised: Proof.

Original: We compute q θ ( ij )
Revised: We compute q θ ( ij )

Original: = 1 / (1 + d 2 ij ) = (1 /d 2 ij ) / (1 /d 2 ij + 1) =
Revised: = 1 / (1 + d 2 ij ) = (1 /d 2 ij ) / (1 /d 2 ij + 1) =

Original: ˜
Revised: ˜

Original: q θ ( ij ) /
Revised: q θ ( ij ) /

Original: (˜ q θ ( ij ) + 1) .
Revised: (˜ q θ ( ij ) + 1) .

Original: Lem.
Revised: Lem.

Original: 3 tells us that UMAP uses NEG but not with a parametric model given by the Cauchy kernel.
Revised: 3 tells us that UMAP uses NEG but not with a parametric model given by the Cauchy kernel.

Original: Instead it uses a model ˜ q θ , which equals the inverse squared distance between embedding points.
Revised: Instead it uses a model ˜ q θ , which equals the inverse squared distance between embedding points.

Original: For large embedding distances d ij both models behave alike, but for nearby points they strongly differ: The inverse-square kernel 1 /d 2 ij diverges when d ij → 0 , unlike the Cauchy kernel 1 / (1+ d 2 ij ) .
Revised: For large embedding distances d ij both models behave alike, but for nearby points they strongly differ: The inverse-square kernel 1 /d 2 ij diverges when d ij → 0 , unlike the Cauchy kernel 1 / (1+ d 2 ij ) .

Original: Despite this qualitative difference, we found empirically that UMAP embeddings look very similar to Neg- t -SNE embeddings at ¯ Z = | X | /m , see Figs.
Revised: Despite this qualitative difference, we found empirically that UMAP embeddings look very similar to Neg- t -SNE embeddings at ¯ Z = | X | /m , see Figs.

Original: 1d and 1h for the MNIST example.
Revised: 1d and 1h for the MNIST example.

Original: To explain this observation, it is instructive to compare the loss terms between Neg- t -SNE and UMAP:
Revised: To explain this observation, it is instructive to compare the loss terms between Neg- t -SNE and UMAP:

Original: The attractive term amounts to − log(1 /
Revised: The attractive term amounts to − log(1 /

Original: ( d 2 ij + 1))
Revised: ( d 2 ij + 1))

Original: =
Revised: =

Original: log(1 + d 2 ij ) for UMAP and − log[1 / ( d 2 ij + 1) /
Revised: log(1 + d 2 ij ) for UMAP and − log[1 / ( d 2 ij + 1) /

Original: (1 / ( d 2 ij + 1) + 1)]
Revised: (1 / ( d 2 ij + 1) + 1)]

Original: = log(2 + d 2 ij ) for Neg- t -SNE, while the repulsive term equals log((

Revised: = log(2 + d 2 ij ) for Neg- t -SNE, while the repulsive term equals log((

Original: + d 2 ij ) /d 2 ij ) and log((2 + d 2 ij ) /
Revised: + d 2 ij ) /d 2 ij ) and log((2 + d 2 ij ) /

Original: (1 + d 2 ij )) , respectively.
Revised: (1 + d 2 ij )) , respectively.

Original: While the attractive terms are very similar, the repulsive term for UMAP diverges at zero but that of Neg- t -SNE does not (Fig. S6).
Revised: While the attractive terms are very similar, the repulsive term for UMAP diverges at zero but that of Neg- t -SNE does not (Fig. S6).

Original: This divergence introduces numerical instability into the optimization process of UMAP.
Revised: This divergence introduces numerical instability into the optimization process of UMAP.

Original: We found that UMAP strongly depends on annealing its learning rate down to zero (Figs.
Revised: We found that UMAP strongly depends on annealing its learning rate down to zero (Figs.

Original: 2a, b).
Revised: 2a, b).

Original: Without it, clusters appear fuzzy as noise pairs can experience very strong repulsion and get catapulted out of their cluster (Fig. 2a).
Revised: Without it, clusters appear fuzzy as noise pairs can experience very strong repulsion and get catapulted out of their cluster (Fig. 2a).

Original: While Neg- t -SNE also benefits from this annealing scheme (Fig. 2d), it produces a very similar embedding even without any annealing (Fig. 2c).
Revised: While Neg- t -SNE also benefits from this annealing scheme (Fig. 2d), it produces a very similar embedding even without any annealing (Fig. 2c).

Original: Thus, UMAP’s effective choice of the 1 /d 2 ij kernel makes it less numerically stable and more dependent on optimization tricks, compared to Neg- t -SNE.
Revised: Thus, UMAP’s effective choice of the 1 /d 2 ij kernel makes it less numerically stable and more dependent on optimization tricks, compared to Neg- t -SNE.

Original: 2 See Supp.
Revised: 2 See Supp.

Original: E for more details.
Revised: E for more details.

Original: Our conclusion is that at its heart, UMAP is NEG applied to the t -SNE framework.
Revised: Our conclusion is that at its heart, UMAP is NEG applied to the t -SNE framework.

Original: UMAP’s sampling-based optimization is much more than a mere optimization trick; it enables us to connect it theoretically to t -SNE.
Revised: UMAP’s sampling-based optimization is much more than a mere optimization trick; it enables us to connect it theoretically to t -SNE.

Original: When UMAP’s loss function is seen as an instance of NEG, UMAP does not use the Cauchy kernel but rather the inverse-square kernel.
Revised: When UMAP’s loss function is seen as an instance of NEG, UMAP does not use the Cauchy kernel but rather the inverse-square kernel.

Original: However, this does not make a strong difference due to the learning rate decay.
Revised: However, this does not make a strong difference due to the learning rate decay.

Original: As discussed in Sec. 4, the fixed normalization constant ¯ Z in Neg- t -SNE/UMAP is much larger than the learnt Z in NCVis or the partition function in t -SNE.
Revised: As discussed in Sec. 4, the fixed normalization constant ¯ Z in Neg- t -SNE/UMAP is much larger than the learnt Z in NCVis or the partition function in t -SNE.

Original: This explains why UMAP pulls embedding points closer together than both NCVis and t -SNE and is the reason for the typically more compact clusters in UMAP plots (B¨ohm et al., 2022).
Revised: This explains why UMAP pulls embedding points closer together than both NCVis and t -SNE and is the reason for the typically more compact clusters in UMAP plots (B¨ohm et al., 2022).

Original:  C ONTRASTIVE NE AND CONTRASTIVE SELF - SUPERVISED LEARNING
Revised:  C ONTRASTIVE NE AND CONTRASTIVE SELF - SUPERVISED LEARNING

Original: Contrastive self-supervised representation learning (van den Oord et al., 2018; Tian et al., 2020;
Revised: Contrastive self-supervised representation learning (van den Oord et al., 2018; Tian et al., 2020;

Original: He et al., 2020; Chen et al., 2020; Caron et al., 2020) and ‘contrastive neighbor embeddings’ (a term we suggest for NCVis, Neg- t -SNE, UMAP, etc.) are conceptually very similar.
Revised: He et al., 2020; Chen et al., 2020; Caron et al., 2020) and ‘contrastive neighbor embeddings’ (a term we suggest for NCVis, Neg- t -SNE, UMAP, etc.) are conceptually very similar.

Original: The key difference is that the latter use a fixed k NN graph to find pairs of similar objects, while the former rely on data augmentations or context to generate pairs of similar objects on the fly.
Revised: The key difference is that the latter use a fixed k NN graph to find pairs of similar objects, while the former rely on data augmentations or context to generate pairs of similar objects on the fly.

Original: Other differences include the representation dimension ( ∼ 128 vs. 2 ), the use of a neural network for parametric mapping, the flavor of contrastive loss (InfoNCE vs. NCE / NEG) or the number of noise samples m .
Revised: Other differences include the representation dimension ( ∼ 128 vs. 2 ), the use of a neural network for parametric mapping, the flavor of contrastive loss (InfoNCE vs. NCE / NEG) or the number of noise samples m .

Original: However, these other differences are not crucial, as we establish here with our unified PyTorch framework.
Revised: However, these other differences are not crucial, as we establish here with our unified PyTorch framework.

Original: As an example, we demonstrate that t -SNE can also be optimized using the InfoNCE loss, resulting in ‘InfoNC- t -SNE’ (Supp.
Revised: As an example, we demonstrate that t -SNE can also be optimized using the InfoNCE loss, resulting in ‘InfoNC- t -SNE’ (Supp.

Original: K and Alg. S1).
Revised: K and Alg. S1).

Original: The result of InfoNC- t -SNE on MNIST (Fig. 3c) was similar to the result of NCVis (Fig. 3b).
Revised: The result of InfoNC- t -SNE on MNIST (Fig. 3c) was similar to the result of NCVis (Fig. 3b).

Original: For the default number of noise samples, m = 5 , both algorithms produced embeddings that were visibly different from t -SNE proper (Fig. 1f).
Revised: For the default number of noise samples, m = 5 , both algorithms produced embeddings that were visibly different from t -SNE proper (Fig. 1f).

Original: Recent work in self-supervised learning (Chen et al., 2020) reports improved performance for more noise samples, in agreement with the theory of Gutmann & Hyv¨arinen (2012); Ma & Collins (2018).
Revised: Recent work in self-supervised learning (Chen et al., 2020) reports improved performance for more noise samples, in agreement with the theory of Gutmann & Hyv¨arinen (2012); Ma & Collins (2018).

Original: We observed that both NCVis and InfoNC- t -SNE visualizations using m = 500 approximated t -SNE much better (Figs. S17k and S18k).
Revised: We observed that both NCVis and InfoNC- t -SNE visualizations using m = 500 approximated t -SNE much better (Figs. S17k and S18k).

Original: Like t -SNE, but unlike the m = 5 setting, m =required early exaggeration to prevent cluster fragmentation, see Figs.
Revised: Like t -SNE, but unlike the m = 5 setting, m =required early exaggeration to prevent cluster fragmentation, see Figs.

Original: S17 and S18.
Revised: S17 and S18.

Original: We discuss
Revised: We discuss

Original: InfoNC- t -SNE’s relation to TriMap in Supp.
Revised: InfoNC- t -SNE’s relation to TriMap in Supp.

Original: F.
Revised: F.

Original: Next, we used our PyTorch framework to obtain parametric versions of all contrastive NE algorithms discussed here (NCVis, InfoNC- t -SNE, Neg- t -SNE, UMAP).
Revised: Next, we used our PyTorch framework to obtain parametric versions of all contrastive NE algorithms discussed here (NCVis, InfoNC- t -SNE, Neg- t -SNE, UMAP).

Original: We used a fully connected neural network with three hidden layers as a parametric R D → R d mapping and optimized its parameters using Adam (Kingma & Ba, 2015)
Revised: We used a fully connected neural network with three hidden layers as a parametric R D → R d mapping and optimized its parameters using Adam (Kingma & Ba, 2015)

Original: (Supp.
Revised: (Supp.

Original: K).
Revised: K).

Original: We used batch size b = 1024 and sampled all m negative samples from within the batch; the data set was shuffled each epoch before batching.
Revised: We used batch size b = 1024 and sampled all m negative samples from within the batch; the data set was shuffled each epoch before batching.

Original: Using all four loss functions, we were able to get parametric embeddings of MNIST that were qualitatively similar to their non-parametric versions (Fig. 3).
Revised: Using all four loss functions, we were able to get parametric embeddings of MNIST that were qualitatively similar to their non-parametric versions (Fig. 3).

Original: The parametric versions of NCE and
Revised: The parametric versions of NCE and

Original: InfoNCE produced much larger embeddings than their non-parametric counterparts, however the final loss values were very similar (Figs.
Revised: InfoNCE produced much larger embeddings than their non-parametric counterparts, however the final loss values were very similar (Figs.

Original: S8a, b).
Revised: S8a, b).

Original: For NCE, the larger scale of the parametric embedding was compensated by a smaller learned normalization parameter Z , so that both parametric and non-parametric versions were approximately normalized (Fig.
Revised: For NCE, the larger scale of the parametric embedding was compensated by a smaller learned normalization parameter Z , so that both parametric and non-parametric versions were approximately normalized (Fig.

Original: S8c).
Revised: S8c).

Original: Our parametric UMAP implementation is very similar to that of Sainburg et al.
Revised: Our parametric UMAP implementation is very similar to that of Sainburg et al.

Original: But our parametric, approximate t -SNE implementations strongly differ from the parametric t -SNE of van der Maaten (2009), which constructed separate k NN graphs within each batch and optimized the vanilla t -SNE loss function.
Revised: But our parametric, approximate t -SNE implementations strongly differ from the parametric t -SNE of van der Maaten (2009), which constructed separate k NN graphs within each batch and optimized the vanilla t -SNE loss function.

Original: Neighbor embeddings methods achieve impressive results with only few noise samples, while existent common practice in the self-supervised learning literature (Bachman et al., 2019;
Revised: Neighbor embeddings methods achieve impressive results with only few noise samples, while existent common practice in the self-supervised learning literature (Bachman et al., 2019;

Original: Chen et al., 2020; He et al., 2020) recommends very large m .
Revised: Chen et al., 2020; He et al., 2020) recommends very large m .

Original: As a proof-of-concept, we demonstrate that using only m = 16 non-curated (Wu et al., 2017; Roth et al., 2020) noise samples can suffice for imagebased self-supervised learning.
Revised: As a proof-of-concept, we demonstrate that using only m = 16 non-curated (Wu et al., 2017; Roth et al., 2020) noise samples can suffice for imagebased self-supervised learning.

Original: We used a SimCLR setup (Chen et al., 2020) to train representations of the CIFAR-10 dataset (Krizhevsky, 2009) using a ResNet18 (He et al., 2016) backbone, a fixed batch size of b = 1024 and varying number of noise samples m .
Revised: We used a SimCLR setup (Chen et al., 2020) to train representations of the CIFAR-10 dataset (Krizhevsky, 2009) using a ResNet18 (He et al., 2016) backbone, a fixed batch size of b = 1024 and varying number of noise samples m .

Original: Other implementation details follow Chen et al.
Revised: Other implementation details follow Chen et al.

Original: (2020)’s
Revised: (2020)’s

Original: CIFAR-10 setup (Supp. K).
Revised: CIFAR-10 setup (Supp. K).

Original: The classification accuracy, measured on the ResNet output, saturates already at m = 16 noise samples (Tab. 1).
Revised: The classification accuracy, measured on the ResNet output, saturates already at m = 16 noise samples (Tab. 1).

Original: Note that, different from the original SimCLR setup, we decoupled the batch size from the number of noise samples.
Revised: Note that, different from the original SimCLR setup, we decoupled the batch size from the number of noise samples.

Original: Recent work found conflicting evidence when varying m for a fixed batch size (Mitrovic et al., 2020;
Revised: Recent work found conflicting evidence when varying m for a fixed batch size (Mitrovic et al., 2020;

Original: Nozawa & Sato, 2021; Ash et al., 2022).
Revised: Nozawa & Sato, 2021; Ash et al., 2022).

Original: Future research is needed to perform more systematic benchmarks into the importance of m .
Revised: Future research is needed to perform more systematic benchmarks into the importance of m .

Original: Here, we present these results only as a proof-of-principle that a similarly low m as in neighbor embeddings (default m = 5 ) can work in a SimCLR setting.
Revised: Here, we present these results only as a proof-of-principle that a similarly low m as in neighbor embeddings (default m = 5 ) can work in a SimCLR setting.

Original:  D ISCUSSION AND C ONCLUSION
Revised:  D ISCUSSION AND C ONCLUSION

Original: In this work, we studied the relationship between two popular unsupervised learning methods, noisecontrastive estimation (NCE) and negative sampling (NEG).
Revised: In this work, we studied the relationship between two popular unsupervised learning methods, noisecontrastive estimation (NCE) and negative sampling (NEG).

Original: We focused on their application to neighbor embeddings (NE) because this is an active and important application area, but also because
Revised: We focused on their application to neighbor embeddings (NE) because this is an active and important application area, but also because

Original: NEs allow to directly visualize the NCE/NEG outcome and to form intuitive understanding of how different algorithm choices affect the result.
Revised: NEs allow to directly visualize the NCE/NEG outcome and to form intuitive understanding of how different algorithm choices affect the result.

Original: Our study makes three conceptual advances.
Revised: Our study makes three conceptual advances.

Original: [Table]
Revised: [Table]

Original: First, we showed that NEG replaces NCE’s learnable normalization parameter Z by a large constant ¯
Revised: First, we showed that NEG replaces NCE’s learnable normalization parameter Z by a large constant ¯

Original: Z , forcing NEG to learn a scaled data distribution.
Revised: Z , forcing NEG to learn a scaled data distribution.

Original: While not explored here, this implies that NEG can be used to learn probabilistic models and is not only applicable for embedding learning as previously believed (Dyer, 2014; Ruder, 2016; Le-Khac et al., 2020).
Revised: While not explored here, this implies that NEG can be used to learn probabilistic models and is not only applicable for embedding learning as previously believed (Dyer, 2014; Ruder, 2016; Le-Khac et al., 2020).

Original: In the NE setting, NEG led to the method Neg- t -SNE, that differs from NCVis (Artemenkov & Panov, 2020) (which could be nicknamed ‘NC- t -SNE’) by a simple switch from the learnable to a fixed normalization constant.
Revised: In the NE setting, NEG led to the method Neg- t -SNE, that differs from NCVis (Artemenkov & Panov, 2020) (which could be nicknamed ‘NC- t -SNE’) by a simple switch from the learnable to a fixed normalization constant.

Original: We argued that this can be a useful hyperparameter because it moves the embedding along the attraction-repulsion spectrum similar to (B ¨ ohm et al., 2022), and hence can either emphasize more discrete structure for higher repulsion (Fig.
Revised: We argued that this can be a useful hyperparameter because it moves the embedding along the attraction-repulsion spectrum similar to (B ¨ ohm et al., 2022), and hence can either emphasize more discrete structure for higher repulsion (Fig.

Original: S15) or more continuous data patterns when attraction is greater (see Figs.
Revised: S15) or more continuous data patterns when attraction is greater (see Figs.

Original: S12 – S13 for developmental single cell datasets).
Revised: S12 – S13 for developmental single cell datasets).

Original: Our quantitative evaluation in
Revised: Our quantitative evaluation in

Original: Supp.
Revised: Supp.

Original: H corroborates this interpretation.
Revised: H corroborates this interpretation.

Original: We believe that inspection of several embeddings along the spectrum can guard against over-interpreting any single embedding.
Revised: We believe that inspection of several embeddings along the spectrum can guard against over-interpreting any single embedding.

Original: Exploration of the spectrum does not require specialized knowledge.
Revised: Exploration of the spectrum does not require specialized knowledge.

Original: For UMAP, ¯ Z UMAP =
Revised: For UMAP, ¯ Z UMAP =

Original: (cid:0) n 2 (cid:1) /m ; for t -SNE, B ¨ ohm et al.
Revised: (cid:0) n 2 (cid:1) /m ; for t -SNE, B ¨ ohm et al.

Original: (2022) found that the partition function Z t -SNE typically lies in [50 n, 100 n ] .
Revised: (2022) found that the partition function Z t -SNE typically lies in [50 n, 100 n ] .

Original: Our PyTorch package allows the user to move along the spectrum with a slider parameter s such that s = 0 , 1 corresponds to ¯ Z
Revised: Our PyTorch package allows the user to move along the spectrum with a slider parameter s such that s = 0 , 1 corresponds to ¯ Z

Original: =
Revised: =

Original: Z t -SNE , ¯ Z UMAP , respectively, without any need for specifying ¯ Z directly.
Revised: Z t -SNE , ¯ Z UMAP , respectively, without any need for specifying ¯ Z directly.

Original: A caveat is that Thm. 1 and Cor. 2 both assume that the model is rich enough to perfectly fit the data distribution.
Revised: A caveat is that Thm. 1 and Cor. 2 both assume that the model is rich enough to perfectly fit the data distribution.

Original: This is a strong and unrealistic assumption.
Revised: This is a strong and unrealistic assumption.

Original: In the context of NEs, the data distribution is zero for most pairs of points, which is impossible to match using the Cauchy kernel.
Revised: In the context of NEs, the data distribution is zero for most pairs of points, which is impossible to match using the Cauchy kernel.

Original: Nevertheless, the consistency of MLE and thus t -SNE also depends on this assumption.
Revised: Nevertheless, the consistency of MLE and thus t -SNE also depends on this assumption.

Original: Even without it, the gradients of MLE, NCE, and NEG are similar (Supp.
Revised: Even without it, the gradients of MLE, NCE, and NEG are similar (Supp.

Original: B).
Revised: B).

Original: Finally, we validated our Cor. 2 in a toy setting in which its assumptions do hold in Supp.
Revised: Finally, we validated our Cor. 2 in a toy setting in which its assumptions do hold in Supp.

Original: I.
Revised: I.

Original: Second, we demonstrated that UMAP, which uses NEG, differs from Neg- t -SNE only in UMAP’s implicit use of a less numerically stable similarity function.
Revised: Second, we demonstrated that UMAP, which uses NEG, differs from Neg- t -SNE only in UMAP’s implicit use of a less numerically stable similarity function.

Original: This isolates the key aspect of UMAP’s success: Instead of UMAP’s appraised (Oskolkov, 2022; Coenen & Pearce, 2022) high-dimensional similarities, the refined Cauchy kernel, or its stated cross-entropy loss function, it is the application of NEG that lets UMAP perform well and makes clusters in UMAP plots more compact and connections between them more continuous than in t -SNE, in agreement with B ¨ ohm et al.
Revised: This isolates the key aspect of UMAP’s success: Instead of UMAP’s appraised (Oskolkov, 2022; Coenen & Pearce, 2022) high-dimensional similarities, the refined Cauchy kernel, or its stated cross-entropy loss function, it is the application of NEG that lets UMAP perform well and makes clusters in UMAP plots more compact and connections between them more continuous than in t -SNE, in agreement with B ¨ ohm et al.

Original: To the best of our knowledge, this is the first time UMAP’s and t -SNE’s loss functions, motivated very differently, have been conceptually connected.
Revised: To the best of our knowledge, this is the first time UMAP’s and t -SNE’s loss functions, motivated very differently, have been conceptually connected.

Original: Our use of the standard Cauchy kernel and the binary s k NN graph are only a mild limitations (B¨ohm et al., 2022; Damrich & Hamprecht, 2021).
Revised: Our use of the standard Cauchy kernel and the binary s k NN graph are only a mild limitations (B¨ohm et al., 2022; Damrich & Hamprecht, 2021).

Original: Third, we argued that contrastive NEs are closely related to contrastive self-supervised learning (SSL) methods such as SimCLR (Chen et al., 2020) which can be seen as parametric InfoNC- t -SNE for learning representations in S 128 based on the unobservable similarity graph implicitly constructed via data augmentations.
Revised: Third, we argued that contrastive NEs are closely related to contrastive self-supervised learning (SSL) methods such as SimCLR (Chen et al., 2020) which can be seen as parametric InfoNC- t -SNE for learning representations in S 128 based on the unobservable similarity graph implicitly constructed via data augmentations.

Original: We feel that this connection has been underappreciated, with the literature on NEs and on self-supervised contrastive learning staying mostly disconnected.
Revised: We feel that this connection has been underappreciated, with the literature on NEs and on self-supervised contrastive learning staying mostly disconnected.

Original: An exception is the concurrent work of Hu et al.
Revised: An exception is the concurrent work of Hu et al.

Original: They propose to use t -SNE’s Cauchy kernel for
Revised: They propose to use t -SNE’s Cauchy kernel for

Original: SSL.
Revised: SSL.

Original: Instead, we bridge the gap between NE and SSL, with our InfoNC- t -SNE, as well as with parametric versions of all considered NE methods, useful for adding out-of-sample data to an existing visualization.
Revised: Instead, we bridge the gap between NE and SSL, with our InfoNC- t -SNE, as well as with parametric versions of all considered NE methods, useful for adding out-of-sample data to an existing visualization.

Original: Moreover, we demonstrated that the feasibility of few noise samples in NE translates to the SimCLR setup.
Revised: Moreover, we demonstrated that the feasibility of few noise samples in NE translates to the SimCLR setup.

Original: We developed a concise PyTorch framework optimizing all of the above, which we hope will facilitate future dialogue between the two research communities.
Revised: We developed a concise PyTorch framework optimizing all of the above, which we hope will facilitate future dialogue between the two research communities.

Original: Overall, our insights into NEs should enable practitioners to explore their datasets more reliably.
Revised: Overall, our insights into NEs should enable practitioners to explore their datasets more reliably.

Original: Our conceptual relation between UMAP and t -SNE is therefore of practical impact – as is the fact that both are just instances on a spectrum highlighting more continuous or more discrete structures.
Revised: Our conceptual relation between UMAP and t -SNE is therefore of practical impact – as is the fact that both are just instances on a spectrum highlighting more continuous or more discrete structures.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: Our work analyses connections between popular contrastive learning methods and in particular visualization methods.
Revised: Our work analyses connections between popular contrastive learning methods and in particular visualization methods.

Original: Visualization methods are fundamental in exploratory data analysis, e.g., in computational biology.
Revised: Visualization methods are fundamental in exploratory data analysis, e.g., in computational biology.

Original: Given the basic nature of visualization it can of course potentially be used for malicious goals, but we expect mostly positive societal effects such as more reliable exploration of biological data.
Revised: Given the basic nature of visualization it can of course potentially be used for malicious goals, but we expect mostly positive societal effects such as more reliable exploration of biological data.

Original: Similarly, contrastive self-supervised learning has the potential to overcome the annotation bottleneck that machine learning faces.
Revised: Similarly, contrastive self-supervised learning has the potential to overcome the annotation bottleneck that machine learning faces.

Original: This might free countless hours of labelling for more productive use but could also enable institutions with sufficient resources to learn from larger and larger datasets, potentially concentrating power.
Revised: This might free countless hours of labelling for more productive use but could also enable institutions with sufficient resources to learn from larger and larger datasets, potentially concentrating power.

Original: Our PyTorch implementation of contrastive neighbor embeddings is slower than existing UMAP and
Revised: Our PyTorch implementation of contrastive neighbor embeddings is slower than existing UMAP and

Original: NCVis implementations and hence more resource intensive.
Revised: NCVis implementations and hence more resource intensive.

Original: We therefore recommend it primarily as a research tool into contrastive learning and contrastive neighbor embeddings, and suggest more efficient implementations for large scale applications.
Revised: We therefore recommend it primarily as a research tool into contrastive learning and contrastive neighbor embeddings, and suggest more efficient implementations for large scale applications.

Original: R EPRODUCIBILITY
Revised: R EPRODUCIBILITY

Original: All of our results are reproducible.
Revised: All of our results are reproducible.

Original: We included proofs for the theoretical statement Cor. 2 and those in Supp.
Revised: We included proofs for the theoretical statement Cor. 2 and those in Supp.

Original: D. The datasets used in the experiments including preprocessing steps are described in Supp.
Revised: D. The datasets used in the experiments including preprocessing steps are described in Supp.

Original: J. We discuss the most important implementation details in the main text, e.g., in Sec. 7.
Revised: J. We discuss the most important implementation details in the main text, e.g., in Sec. 7.

Original: An extensive discussion of our implementation is included in Supp.
Revised: An extensive discussion of our implementation is included in Supp.

Original: K and in Alg. S1.
Revised: K and in Alg. S1.

Original: The code and instructions for how to reproduce the experiments are submitted as supplementary material.
Revised: The code and instructions for how to reproduce the experiments are submitted as supplementary material.

Original: S UPPLEMENTARY TEXT A P ROBABILISTIC FRAMEWORKS OF NCE AND I NFO NCE
Revised: S UPPLEMENTARY TEXT A P ROBABILISTIC FRAMEWORKS OF NCE AND I NFO NCE

Original: In this section we recap the probabilistic frameworks of NCE (Gutmann & Hyv¨arinen, 2010; 2012) and InfoNCE (Jozefowicz et al., 2016; van den Oord et al., 2018).
Revised: In this section we recap the probabilistic frameworks of NCE (Gutmann & Hyv¨arinen, 2010; 2012) and InfoNCE (Jozefowicz et al., 2016; van den Oord et al., 2018).

Original: A. NCE
Revised: A. NCE

Original: In NCE the unsupervised problem of parametric density estimation is turned into a supervised problem in which the data samples need to be identified in a set S containing N data samples s 1 , . . .
Revised: In NCE the unsupervised problem of parametric density estimation is turned into a supervised problem in which the data samples need to be identified in a set S containing N data samples s 1 , . . .

Original: , s N and m times as many noise samples t 1 , . .
Revised: , s N and m times as many noise samples t 1 , . .

Original: , t mN drawn from a noise distribution ξ , which can be (but does not have to be) the uniform distribution.
Revised: , t mN drawn from a noise distribution ξ , which can be (but does not have to be) the uniform distribution.

Original: In other words, we are interested in the posterior probability P
Revised: In other words, we are interested in the posterior probability P

Original: ( y | x ) of element x ∈ S coming from the data ( y = data) rather than from the noise distribution ( y = noise).
Revised: ( y | x ) of element x ∈ S coming from the data ( y = data) rather than from the noise distribution ( y = noise).

Original: The probability of sampling x from noise, P ( x | noise ) , is just the noise distribution ξ , and similarly P ( x | data ) is the data distribution p .
Revised: The probability of sampling x from noise, P ( x | noise ) , is just the noise distribution ξ , and similarly P ( x | data ) is the data distribution p .

Original: Since the latter is unknown, it is replaced by the model q θ .
Revised: Since the latter is unknown, it is replaced by the model q θ .

Original: Since S contains m times as many noise samples as data samples, the prior class probabilities are P ( data )
Revised: Since S contains m times as many noise samples as data samples, the prior class probabilities are P ( data )

Original: = 1 / ( m + 1) and P ( noise )
Revised: = 1 / ( m + 1) and P ( noise )

Original: =
Revised: =

Original: m/ ( m + 1) .
Revised: m/ ( m + 1) .

Original: Thus, the unconditional probability of an element of S is P ( x ) =
Revised: Thus, the unconditional probability of an element of S is P ( x ) =

Original: ( q θ ( x ) + mξ ( x )) /
Revised: ( q θ ( x ) + mξ ( x )) /

Original: ( m + 1) .
Revised: ( m + 1) .

Original: The posterior probability for classifying some given element x of S as data rather than noise is thus
Revised: The posterior probability for classifying some given element x of S as data rather than noise is thus

Original: [Equation]
Revised: [Equation]

Original: NCE optimizes the parameters θ by maximizing the log-likelihood of the posterior class distributions, or, equivalently, by minimizing the negative log-likelihoods.
Revised: NCE optimizes the parameters θ by maximizing the log-likelihood of the posterior class distributions, or, equivalently, by minimizing the negative log-likelihoods.

Original: This is the same as a sum over binary cross-entropy losses (Eq.
Revised: This is the same as a sum over binary cross-entropy losses (Eq.

Original: (2) in the main text):
Revised: (2) in the main text):

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: In expectation, we have the loss function (Eq.
Revised: In expectation, we have the loss function (Eq.

Original: (4) in the main text)
Revised: (4) in the main text)

Original: [Equation]
Revised: [Equation]

Original: Since q θ ( x ) / (cid:0) q θ ( x ) + mξ ( x ) (cid:1) = 1 / (cid:104) 1 + (cid:16)
Revised: Since q θ ( x ) / (cid:0) q θ ( x ) + mξ ( x ) (cid:1) = 1 / (cid:104) 1 + (cid:16)

Original: q θ ( x ) / (cid:0) mξ ( x ) (cid:1)(cid:17) − 1 (cid:105) , NCE’s loss function can also be seen as binary logistic regression loss function with log (cid:16) q θ ( x ) mξ ( x ) (cid:17) as the input to the logistic function:
Revised: q θ ( x ) / (cid:0) mξ ( x ) (cid:1)(cid:17) − 1 (cid:105) , NCE’s loss function can also be seen as binary logistic regression loss function with log (cid:16) q θ ( x ) mξ ( x ) (cid:17) as the input to the logistic function:

Original: [Equation]
Revised: [Equation]

Original: NCE
Revised: NCE

Original: [Equation]
Revised: [Equation]

Original: (13) where σ ( x ) = 1 / (cid:0) 1 + exp( − x ) (cid:1) is the logistic function.
Revised: (13) where σ ( x ) = 1 / (cid:0) 1 + exp( − x ) (cid:1) is the logistic function.

Original: A. I NFO NCE
Revised: A. I NFO NCE

Original: Again, we are casting the unsupervised problem of density estimation as a supervised classification problem.
Revised: Again, we are casting the unsupervised problem of density estimation as a supervised classification problem.

Original: We consider a tuple of m + 1 samples T = ( x 0 , . . .
Revised: We consider a tuple of m + 1 samples T = ( x 0 , . . .

Original: , x m ) one of which comes from the data and the rest from the noise distribution.
Revised: , x m ) one of which comes from the data and the rest from the noise distribution.

Original: Instead of classifying each sample independently as noise or data (as in Sec.
Revised: Instead of classifying each sample independently as noise or data (as in Sec.

Original: A.1), here we are interested in identifying the position of the single data sample.
Revised: A.1), here we are interested in identifying the position of the single data sample.

Original: This allows us to see the problem as a multi-class classification problem with m +1 classes.
Revised: This allows us to see the problem as a multi-class classification problem with m +1 classes.

Original: Let Y be the random variable that holds the index of the data sample.
Revised: Let Y be the random variable that holds the index of the data sample.

Original: A priori, we have P ( Y = k )
Revised: A priori, we have P ( Y = k )

Original: = 1 / ( m + 1) for all k = 0 , . .
Revised: = 1 / ( m + 1) for all k = 0 , . .

Original: , m .
Revised: , m .

Original: Moreover, conditioned on sample k coming from the data distribution, all other samples must come from the noise distribution, i.e., we have P ( x i | Y = k ) =
Revised: Moreover, conditioned on sample k coming from the data distribution, all other samples must come from the noise distribution, i.e., we have P ( x i | Y = k ) =

Original: ξ ( x i ) for i ̸
Revised: ξ ( x i ) for i ̸

Original: = k .
Revised: = k .

Original: As the data distribution is unknown, we model it with P ( x k | Y = k ) =
Revised: As the data distribution is unknown, we model it with P ( x k | Y = k ) =

Original: q θ ( x k ) as in Sec.
Revised: q θ ( x k ) as in Sec.

Original: A.1.
Revised: A.1.

Original: This yields the likelihood of tuple T given the data index Y =
Revised: This yields the likelihood of tuple T given the data index Y =

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: Marginalizing over Y , we obtain
Revised: Marginalizing over Y , we obtain

Original: [Equation]
Revised: [Equation]

Original: Finally, we can compute the posterior via Bayes’ rule as
Revised: Finally, we can compute the posterior via Bayes’ rule as

Original: [Equation]
Revised: [Equation]

Original: where the last equality only holds for the uniform noise distribution.
Revised: where the last equality only holds for the uniform noise distribution.

Original: The InfoNCE loss is the crossentropy loss with respect to the true position of the data sample, i.e., in expectation and for uniform ξ it reads:
Revised: The InfoNCE loss is the crossentropy loss with respect to the true position of the data sample, i.e., in expectation and for uniform ξ it reads:

Original: [Equation]
Revised: [Equation]

Original: Similar to how the NCE loss can be seen as the binary logistic regression loss function (Sec. A.1), the InfoNCE loss can be viewed as the multinomial logistic regression loss function with the terms log (cid:16) q
Revised: Similar to how the NCE loss can be seen as the binary logistic regression loss function (Sec. A.1), the InfoNCE loss can be viewed as the multinomial logistic regression loss function with the terms log (cid:16) q

Original: [Equation]
Revised: [Equation]

Original: ξ ( x i ) (cid:17) entering the softmax function.
Revised: ξ ( x i ) (cid:17) entering the softmax function.

Original: B G RADIENTS OF MLE, NCE, AND NEG
Revised: B G RADIENTS OF MLE, NCE, AND NEG

Original: Here, we compute the gradients of MLE, NCE, and NEG, elaborating the discussion in (Artemenkov & Panov, 2020).
Revised: Here, we compute the gradients of MLE, NCE, and NEG, elaborating the discussion in (Artemenkov & Panov, 2020).

Original: Following the discussion in Secs. 3.
Revised: Following the discussion in Secs. 3.

Original: and 4, we consider a normalized model q θ ( x ) =
Revised: and 4, we consider a normalized model q θ ( x ) =

Original: ϕ θ ( x ) /Z
Revised: ϕ θ ( x ) /Z

Original: ( θ ) with partition function
Revised: ( θ ) with partition function

Original: Z ( θ ) = (cid:80) x ′ ϕ
Revised: Z ( θ ) = (cid:80) x ′ ϕ

Original: θ ( x ′ ) for MLE, a model q θ,Z ( x ) =
Revised: θ ( x ′ ) for MLE, a model q θ,Z ( x ) =

Original: ϕ θ ( x ) /Z with learnable normalization parameter Z for NCE, and a model q θ, ¯ Z ( x ) =
Revised: ϕ θ ( x ) /Z with learnable normalization parameter Z for NCE, and a model q θ, ¯ Z ( x ) =

Original: ϕ θ ( x ) / ¯ Z with fixed normalization constant ¯
Revised: ϕ θ ( x ) / ¯ Z with fixed normalization constant ¯

Original: Z = | X | /m and a uniform noise distribution ξ ( x ) = 1 / | X | for NEG.
Revised: Z = | X | /m and a uniform noise distribution ξ ( x ) = 1 / | X | for NEG.

Original: Thus, the losses are
Revised: Thus, the losses are

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: For MLE, we find
Revised: For MLE, we find

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: with
Revised: with

Original: [Equation]
Revised: [Equation]

Original: We used our templates at the third equality sign with f ( z ) =
Revised: We used our templates at the third equality sign with f ( z ) =

Original: q θ,Z ( x ) and c = mξ ( x ) .
Revised: q θ,Z ( x ) and c = mξ ( x ) .

Original: The derivation of the gradients for the NEG loss is entirely analogous to that for NCE with q θ,Z ( x ) replaced by q θ, ¯ Z ( x ) .
Revised: The derivation of the gradients for the NEG loss is entirely analogous to that for NCE with q θ,Z ( x ) replaced by q θ, ¯ Z ( x ) .

Original: Together, the gradients of the three losses are
Revised: Together, the gradients of the three losses are

Original: [Equation]
Revised: [Equation]

Original: The fractions involving mξ ( x ) in the gradients of L NCE and L NEG tend to one as m → ∞ highlighting that a higher number of noise samples improves NCE’s approximation of MLE (Artemenkov &
Revised: The fractions involving mξ ( x ) in the gradients of L NCE and L NEG tend to one as m → ∞ highlighting that a higher number of noise samples improves NCE’s approximation of MLE (Artemenkov &

Original: Panov, 2020).
Revised: Panov, 2020).

Original: All three gradients are the same up to this factor and the different choice of normalization.
Revised: All three gradients are the same up to this factor and the different choice of normalization.

Original: In all of them, the models q θ , q θ,Z , or q θ, ¯ Z try to match the data distribution p according to the first factor in each gradient.
Revised: In all of them, the models q θ , q θ,Z , or q θ, ¯ Z try to match the data distribution p according to the first factor in each gradient.

Original: This similarity of the gradients holds independent of the assumptions of
Revised: This similarity of the gradients holds independent of the assumptions of

Original: Thm. 1 and Cor. 2.
Revised: Thm. 1 and Cor. 2.

Original: We optimize all losses with stochastic gradient descent and can therefore expect our analysis to hold even if the assumptions are violated.
Revised: We optimize all losses with stochastic gradient descent and can therefore expect our analysis to hold even if the assumptions are violated.

Original: C UMAP’ S LOSS FUNCTION
Revised: C UMAP’ S LOSS FUNCTION

Original: In the original UMAP paper, McInnes et al.
Revised: In the original UMAP paper, McInnes et al.

Original: (2018) define weights µ ( ij ) ∈ [0 , 1] on the s k NN graph and state that these shall be reproduced by the low-dimensional similarities ϕ ( ij ) by means of a sum of binary cross-entropy loss functions, one for each edge ij
Revised: (2018) define weights µ ( ij ) ∈ [0 , 1] on the s k NN graph and state that these shall be reproduced by the low-dimensional similarities ϕ ( ij ) by means of a sum of binary cross-entropy loss functions, one for each edge ij

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Indeed, this loss has its minimum at µ ( ij ) =
Revised: Indeed, this loss has its minimum at µ ( ij ) =

Original: ϕ ( ij ) for all ij .
Revised: ϕ ( ij ) for all ij .

Original: However, it is of course not possible to achieve zero loss for any real-world data using the Cauchy kernel in two dimensions.
Revised: However, it is of course not possible to achieve zero loss for any real-world data using the Cauchy kernel in two dimensions.

Original: Experiments show that using this loss function in practice leads to an excess of repulsion and consequently to very poor embeddings (B¨ohm et al., 2022).
Revised: Experiments show that using this loss function in practice leads to an excess of repulsion and consequently to very poor embeddings (B¨ohm et al., 2022).

Original: The actual UMAP implementation has much less repulsion due to the sampling of repulsive edges, see below.
Revised: The actual UMAP implementation has much less repulsion due to the sampling of repulsive edges, see below.

Original: As the weights µ ( ij ) are only supported on the sparse s k NN graph, most of the 1 − µ ( ij ) terms are equal to one.
Revised: As the weights µ ( ij ) are only supported on the sparse s k NN graph, most of the 1 − µ ( ij ) terms are equal to one.

Original: To simplify the loss function, the UMAP paper replaces all 1 − µ ( ij ) terms by 1 , leading to the loss function
Revised: To simplify the loss function, the UMAP paper replaces all 1 − µ ( ij ) terms by 1 , leading to the loss function

Original: [Equation]
Revised: [Equation]

Original: In the implementation, UMAP samples the repulsive edges, which drastically changes the loss (B ¨ ohm et al., 2022) to the effective loss (Damrich & Hamprecht, 2021)
Revised: In the implementation, UMAP samples the repulsive edges, which drastically changes the loss (B ¨ ohm et al., 2022) to the effective loss (Damrich & Hamprecht, 2021)

Original: [Equation]
Revised: [Equation]

Original: where d i = (cid:80) j µ ( ij ) denotes the degree of node i and the number of negative samples m is a hyperparameter.
Revised: where d i = (cid:80) j µ ( ij ) denotes the degree of node i and the number of negative samples m is a hyperparameter.

Original: By default, m = 5 .
Revised: By default, m = 5 .

Original: Since d i ≈ log( k ) , the effective loss only has about m log( k ) /n of the repulsion in the originally stated loss function.
Revised: Since d i ≈ log( k ) , the effective loss only has about m log( k ) /n of the repulsion in the originally stated loss function.

Original: As a result, the µ ( ij ) are not reproduced in the embedding space by this loss function.
Revised: As a result, the µ ( ij ) are not reproduced in the embedding space by this loss function.

Original: We rewrite this effective loss function further to fit into our framework.
Revised: We rewrite this effective loss function further to fit into our framework.

Original: The attractive prefactors µ ( ij ) sum to (cid:80) ij µ ( ij ) , while the repulsive prefactors add up to m times this factor.
Revised: The attractive prefactors µ ( ij ) sum to (cid:80) ij µ ( ij ) , while the repulsive prefactors add up to m times this factor.

Original: Dividing the entire loss function by this term does not change its properties.
Revised: Dividing the entire loss function by this term does not change its properties.

Original: But then, we can write the prefactors as probability distributions p ( ij ) = µ ( ij ) / (cid:80) ij µ ( ij ) and ξ ( ij ) = (cid:0) p ( i ) + p ( j ) (cid:1)
Revised: But then, we can write the prefactors as probability distributions p ( ij ) = µ ( ij ) / (cid:80) ij µ ( ij ) and ξ ( ij ) = (cid:0) p ( i ) + p ( j ) (cid:1)

Original: /n using p ( i )
Revised: /n using p ( i )

Original: = (cid:80) j p ( ij ) / 2 .
Revised: = (cid:80) j p ( ij ) / 2 .

Original: With this, we can write the effective loss function as
Revised: With this, we can write the effective loss function as

Original: [Equation]
Revised: [Equation]

Original: or in the expectation form as
Revised: or in the expectation form as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: like we do in Eq.
Revised: like we do in Eq.

Original: D N OISE DISTRIBUTIONS
Revised: D N OISE DISTRIBUTIONS

Original: Here we discuss the various noise distributions used by UMAP, NCVis, and our framework.
Revised: Here we discuss the various noise distributions used by UMAP, NCVis, and our framework.

Original: The main claim is that all these noise distributions are sufficiently close to uniform, even though their exact shape depends on the implementation details.
Revised: The main claim is that all these noise distributions are sufficiently close to uniform, even though their exact shape depends on the implementation details.

Original: Since our actual implementation, as well as the reference implementations of UMAP and NCVis, considers edges ij and ji separately, we will do so from now on.
Revised: Since our actual implementation, as well as the reference implementations of UMAP and NCVis, considers edges ij and ji separately, we will do so from now on.

Original: Hence, there is now a total of
Revised: Hence, there is now a total of

Original: E := 

Revised: E :=

Original: | s k NN | edges.
Revised: | s k NN | edges.

Original: We always assume that p ( ij ) = p ( ji ) and adding up the probabilities for both directions yields one: (cid:80) ni,j =1 p ( ij )
Revised: We always assume that p ( ij ) = p ( ji ) and adding up the probabilities for both directions yields one: (cid:80) ni,j =1 p ( ij )

Original: = 1 .
Revised: = 1 .

Original: For a given data distribution over pairs of points ij , we define p ( i ) =
Revised: For a given data distribution over pairs of points ij , we define p ( i ) =

Original: (cid:80) nj =1 p ( ij ) so that (cid:80) i p
Revised: (cid:80) nj =1 p ( ij ) so that (cid:80) i p

Original: ( i )
Revised: ( i )

Original: = 1 .
Revised: = 1 .

Original: As discussed in Supp.
Revised: As discussed in Supp.

Original: B of (Damrich &
Revised: B of (Damrich &

Original: Hamprecht, 2021), the p ( i ) values are approximately constant when p ( ij ) is uniform on the s k NN graph or proportional to the UMAP similarities.
Revised: Hamprecht, 2021), the p ( i ) values are approximately constant when p ( ij ) is uniform on the s k NN graph or proportional to the UMAP similarities.

Original: UMAP’s noise distribution is derived in (Damrich & Hamprecht, 2021) and reads in our notation (Supp. C)
Revised: UMAP’s noise distribution is derived in (Damrich & Hamprecht, 2021) and reads in our notation (Supp. C)

Original: [Equation]
Revised: [Equation]

Original: Note that UMAP uses a weighted version of the s k NN graph.
Revised: Note that UMAP uses a weighted version of the s k NN graph.

Original: Still, ξ is close to uniform, see Supp.
Revised: Still, ξ is close to uniform, see Supp.

Original: B of (Damrich & Hamprecht, 2021).
Revised: B of (Damrich & Hamprecht, 2021).

Original: The noise distribution of NCVis is also close to being uniform and equals (Artemenkov & Panov, 2020):
Revised: The noise distribution of NCVis is also close to being uniform and equals (Artemenkov & Panov, 2020):

Original: [Equation]
Revised: [Equation]

Original: This is a slightly different noise distribution than in UMAP and in particular it is asymmetric.
Revised: This is a slightly different noise distribution than in UMAP and in particular it is asymmetric.

Original: However, we argue that in practice it is equivalent to the same noise distribution as in UMAP.
Revised: However, we argue that in practice it is equivalent to the same noise distribution as in UMAP.

Original: The noise distribution is used in two ways in NCVis: for sampling noise samples and in the posterior class probabilities
Revised: The noise distribution is used in two ways in NCVis: for sampling noise samples and in the posterior class probabilities

Original: [Equation]
Revised: [Equation]

Original: Both in the reference NCVis implementation and in ours, for the second role, the noise distribution is explicitly approximated by the uniform one and we use the posterior probabilities
Revised: Both in the reference NCVis implementation and in ours, for the second role, the noise distribution is explicitly approximated by the uniform one and we use the posterior probabilities

Original: [Equation]
Revised: [Equation]

Original: Together with the symmetry of the Cauchy kernel this implies that the repulsion on the embedding vectors e i and e j from noise sample ij and ji is the same.
Revised: Together with the symmetry of the Cauchy kernel this implies that the repulsion on the embedding vectors e i and e j from noise sample ij and ji is the same.

Original: As a result, the expectation
Revised: As a result, the expectation

Original: [Equation]
Revised: [Equation]

Original: is the same for ξ ( ij )
Revised: is the same for ξ ( ij )

Original: =
Revised: =

Original: p ( i ) /n
Revised: p ( i ) /n

Original: and for UMAP’s noise distribution
Revised: and for UMAP’s noise distribution

Original: [Equation]
Revised: [Equation]

Original: In our framework, the noise distribution is influenced by the batched training procedure (Alg. S1) because the negative samples can come only from the current training batch.
Revised: In our framework, the noise distribution is influenced by the batched training procedure (Alg. S1) because the negative samples can come only from the current training batch.

Original: In every epoch, we first shuffle the set of directed edges of the s k NN graph and then chunk it into batches.
Revised: In every epoch, we first shuffle the set of directed edges of the s k NN graph and then chunk it into batches.

Original: To emphasize, the batches consist of directed edges and not of the original data points.
Revised: To emphasize, the batches consist of directed edges and not of the original data points.

Original: For each edge in a batch, we take its head and sample m indices from the heads and tails of all edges in the batch (excluding the already selected head) and use them as tails to form negative sample pairs.
Revised: For each edge in a batch, we take its head and sample m indices from the heads and tails of all edges in the batch (excluding the already selected head) and use them as tails to form negative sample pairs.

Original: To obtain a negative sample pair ij , the batch must contain some directed edge ik , providing the head of the negative sample, and some pair lj or jl , providing the tail.
Revised: To obtain a negative sample pair ij , the batch must contain some directed edge ik , providing the head of the negative sample, and some pair lj or jl , providing the tail.

Original: We want to derive the expected number of times that a directed edge ij is considered as a negative sample in a batch.
Revised: We want to derive the expected number of times that a directed edge ij is considered as a negative sample in a batch.

Original: For simplicity, let us assume that the number of batches divides the number of directed edges E .
Revised: For simplicity, let us assume that the number of batches divides the number of directed edges E .

Original: As the set of s k NN edges is shuffled every epoch, the expected number of pairs ij as negative samples is the same for all batches.
Revised: As the set of s k NN edges is shuffled every epoch, the expected number of pairs ij as negative samples is the same for all batches.

Original: Let us consider a batch B of size b .
Revised: Let us consider a batch B of size b .

Original: We denote by Y rs the random variable that holds the number of times edge rs appears in B .
Revised: We denote by Y rs the random variable that holds the number of times edge rs appears in B .

Original: We also introduce random variables Y ¬ rs
Revised: We also introduce random variables Y ¬ rs

Original: = (cid:80) t ̸ = r Y ts and
Revised: = (cid:80) t ̸ = r Y ts and

Original: Y r ¬ s = (cid:80) t ̸ = s Y rt .
Revised: Y r ¬ s = (cid:80) t ̸ = s Y rt .

Original: Let p ( r ¬ s ) =
Revised: Let p ( r ¬ s ) =

Original: p ( ¬ sr ) :
Revised: p ( ¬ sr ) :

Original: = p ( r ) − p ( rs ) .
Revised: = p ( r ) − p ( rs ) .

Original: For each occurrence of an i as head of an edge in B , we sample m tails to create negative samples uniformly from all head and tails in B with replacement, but we prevent sampling the identical head i as negative sample tail.
Revised: For each occurrence of an i as head of an edge in B , we sample m tails to create negative samples uniformly from all head and tails in B with replacement, but we prevent sampling the identical head i as negative sample tail.

Original: If, however, the same node i is part of the other edges in the batch, then it may be sampled and would create a futile negative sample ii .
Revised: If, however, the same node i is part of the other edges in the batch, then it may be sampled and would create a futile negative sample ii .

Original: There are m chances for creating a negative sample edge ij for every head i and any occurrence of j in the batch.
Revised: There are m chances for creating a negative sample edge ij for every head i and any occurrence of j in the batch.

Original: The number of heads i in the batch is Y ij + Y
Revised: The number of heads i in the batch is Y ij + Y

Original: i
Revised: i

Original: ¬ j
Revised: ¬ j

Original: and the number of occurrences of j is
Revised: and the number of occurrences of j is

Original: Y ij + Y ji + Y ¬ ij + Y j ¬ i .
Revised: Y ij + Y ji + Y ¬ ij + Y j ¬ i .

Original: Since we sample the tail of a negative sample pair uniformly with replacement, any of the occurrences of j has probability 1 / (2 b − 1) to be selected.
Revised: Since we sample the tail of a negative sample pair uniformly with replacement, any of the occurrences of j has probability 1 / (2 b − 1) to be selected.

Original: Hence, the expected number N ij of times that the ordered pair ij with i ̸
Revised: Hence, the expected number N ij of times that the ordered pair ij with i ̸

Original: =
Revised: =

Original: j is considered as a negative sample in batch B is
Revised: j is considered as a negative sample in batch B is

Original: [Equation]
Revised: [Equation]

Original: Since a head i may not choose itself to form a negative sample, the expected number of times that ii appears as negative sample in the batch is
Revised: Since a head i may not choose itself to form a negative sample, the expected number of times that ii appears as negative sample in the batch is

Original: [Equation]
Revised: [Equation]

Original: Since the batches are sampled without replacement, the random variables Y rs are distributed according to a multivariate hypergeometric distribution, meaning that
Revised: Since the batches are sampled without replacement, the random variables Y rs are distributed according to a multivariate hypergeometric distribution, meaning that

Original: [Equation]
Revised: [Equation]

Original: We use these expressions and analogous ones together with the symmetries p ( rs ) = p ( sr ) to compute (leaving out intermediate algebra steps) the expectation of N ij over the shuffles:
Revised: We use these expressions and analogous ones together with the symmetries p ( rs ) = p ( sr ) to compute (leaving out intermediate algebra steps) the expectation of N ij over the shuffles:

Original: [Equation]
Revised: [Equation]

Original: Since we sample m negative samples for each positive sample and since each batch contains b positive samples, we need to divide E ( N ij ) by mb to obtain ξ ( ij ) :
Revised: Since we sample m negative samples for each positive sample and since each batch contains b positive samples, we need to divide E ( N ij ) by mb to obtain ξ ( ij ) :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Similarly,
Revised: Similarly,

Original: [Equation]
Revised: [Equation]

Original: and hence the noise distribution value for the pair ii is
Revised: and hence the noise distribution value for the pair ii is

Original: [Equation]
Revised: [Equation]

Original: We see that the noise distribution depends on the batch size b .
Revised: We see that the noise distribution depends on the batch size b .

Original: This is not surprising: For example, if the batch size is equal to one, the ordered pair ij can only be sampled as a negative sample in the single batch that consists of that pair.
Revised: This is not surprising: For example, if the batch size is equal to one, the ordered pair ij can only be sampled as a negative sample in the single batch that consists of that pair.

Original: Indeed, for b = 1 our formula yields
Revised: Indeed, for b = 1 our formula yields

Original: [Equation]
Revised: [Equation]

Original: meaning that the data and the noise distributions coincide.
Revised: meaning that the data and the noise distributions coincide.

Original: Conversely, if b = E and there is only one batch, we obtain
Revised: Conversely, if b = E and there is only one batch, we obtain

Original: [Equation]
Revised: [Equation]

Original: and the noise distribution is close to uniform.
Revised: and the noise distribution is close to uniform.

Original: For batch sizes between 1 and E the noise distribution is in between these two extremes.
Revised: For batch sizes between 1 and E the noise distribution is in between these two extremes.

Original: For MNIST, E ≈ 1 . 5 · 10 6 , and in our experiments we used b = 1024 .
Revised: For MNIST, E ≈ 1 . 5 · 10 6 , and in our experiments we used b = 1024 .

Original: This means that the prefactor of the share of the data distribution is about 0 .
Revised: This means that the prefactor of the share of the data distribution is about 0 .

Original: 0005 while that of the near-uniform distribution p ( i ) p ( j ) is about 0 .
Revised: 0005 while that of the near-uniform distribution p ( i ) p ( j ) is about 0 .

Original: 9995 , so the resulting noise distribution is close to uniform.
Revised: 9995 , so the resulting noise distribution is close to uniform.

Original: Note that Thm. 1 and Cor.
Revised: Note that Thm. 1 and Cor.

Original: 2 only require the noise distribution to be nonzero where the data distribution is nonzero, which is the case for any batch size.
Revised: 2 only require the noise distribution to be nonzero where the data distribution is nonzero, which is the case for any batch size.

Original: [Figure]
Revised: [Figure]

Original: E O PTIMIZATION TRICKS IN UMAP’ S REFERENCE IMPLEMENTATION
Revised: E O PTIMIZATION TRICKS IN UMAP’ S REFERENCE IMPLEMENTATION

Original: UMAP’s repulsive term
Revised: UMAP’s repulsive term

Original: [Equation]
Revised: [Equation]

Original: can lead to numerical problems if the two points of the negative sample pair are very close.
Revised: can lead to numerical problems if the two points of the negative sample pair are very close.

Original: In addition to the learning rate decay, discussed in Sec. 6, UMAP’s implementation uses further tricks to prevent unstable or even crashing training.
Revised: In addition to the learning rate decay, discussed in Sec. 6, UMAP’s implementation uses further tricks to prevent unstable or even crashing training.

Original: In non-parametric UMAP’s reference implementation, the gradient on embedding position e i exerted by a single sampled repulsive pair ij is actually
Revised: In non-parametric UMAP’s reference implementation, the gradient on embedding position e i exerted by a single sampled repulsive pair ij is actually

Original: [Equation]
Revised: [Equation]

Original: (65) with ζ = 0 .
Revised: (65) with ζ = 0 .

Original: 001 instead of ζ = 0 .
Revised: 001 instead of ζ = 0 .

Original: This corresponds to the full loss function
Revised: This corresponds to the full loss function

Original: [Equation]
Revised: [Equation]

Original: However, we found that ζ does not have much influence on the appearance of a UMAP embedding.
Revised: However, we found that ζ does not have much influence on the appearance of a UMAP embedding.

Original: Fig.
Revised: Fig.

Original: S1 shows MNIST embeddings obtained using the original UMAP implementation modified to use different values of ζ .
Revised: S1 shows MNIST embeddings obtained using the original UMAP implementation modified to use different values of ζ .

Original: Neither a much smaller positive value such as ζ = 10 − 10 nor setting ζ = 0 substantially changed the appearance of the embedding (even though some runs with ζ =did crash).
Revised: Neither a much smaller positive value such as ζ = 10 − 10 nor setting ζ = 0 substantially changed the appearance of the embedding (even though some runs with ζ =did crash).

Original: The learning rate annealing played a much bigger role in how the embedding looked like (Fig. S1, bottom row).
Revised: The learning rate annealing played a much bigger role in how the embedding looked like (Fig. S1, bottom row).

Original: The reference implementation of parametric UMAP uses automatic differentiation instead of implementing the gradients manually.
Revised: The reference implementation of parametric UMAP uses automatic differentiation instead of implementing the gradients manually.

Original: To avoid terms such as log(0) in the repulsive loss, it clips the argument of the logarithm from below at the value ε = 10 − 4 , effectively using the loss function
Revised: To avoid terms such as log(0) in the repulsive loss, it clips the argument of the logarithm from below at the value ε = 10 − 4 , effectively using the loss function

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: We employ a similar clipping in our code whenever we apply the logarithm function.
Revised: We employ a similar clipping in our code whenever we apply the logarithm function.

Original: Again, we found that the exact value of ε is not important for our UMAP reimplementation, while using the learning rate annealing is (Fig. S2, top two rows).
Revised: Again, we found that the exact value of ε is not important for our UMAP reimplementation, while using the learning rate annealing is (Fig. S2, top two rows).

Original: In the extreme case of setting ε = 

Revised: In the extreme case of setting ε =

Original: our UMAP runs crashed.
Revised: our UMAP runs crashed.

Original: We believe that the reason is that we allow negative sample pairs to be of the form ii , which would not send any gradient, but would lead to a zero argument to the logarithm.
Revised: We believe that the reason is that we allow negative sample pairs to be of the form ii , which would not send any gradient, but would lead to a zero argument to the logarithm.

Original: The reference implementation of UMAP excludes such negative sample pairs ii .
Revised: The reference implementation of UMAP excludes such negative sample pairs ii .

Original: Our Neg- t -SNE approach does not have any of these problems, as the repulsive term is
Revised: Our Neg- t -SNE approach does not have any of these problems, as the repulsive term is

Original: [Equation]
Revised: [Equation]

Original: and does not diverge for d ij → 0 .
Revised: and does not diverge for d ij → 0 .

Original: For this reason, Neg- t -SNE is not very sensitive to the value at which we clip arguments to the logarithm and works even with ε = 0 , both with and without learning rate annealing (Fig. S2, bottom two rows).
Revised: For this reason, Neg- t -SNE is not very sensitive to the value at which we clip arguments to the logarithm and works even with ε = 0 , both with and without learning rate annealing (Fig. S2, bottom two rows).

Original: The attractive terms in the loss functions do not pose numerical problems in practice due to the heavy tail of the Cauchy kernel.
Revised: The attractive terms in the loss functions do not pose numerical problems in practice due to the heavy tail of the Cauchy kernel.

Original: That said, in order to keep different experiments and losses comparable, we used clipping in both the attractive and the repulsive loss terms with ε = 10 − 10 for all neighbor embedding plots computed with our framework unless otherwise stated.
Revised: That said, in order to keep different experiments and losses comparable, we used clipping in both the attractive and the repulsive loss terms with ε = 10 − 10 for all neighbor embedding plots computed with our framework unless otherwise stated.

Original: A recent pull request (#856) to the parametric part of UMAP’s reference implementation proposed another way to ameliorate the numerical instabilities.
Revised: A recent pull request (#856) to the parametric part of UMAP’s reference implementation proposed another way to ameliorate the numerical instabilities.

Original: The clipping of the arguments to the logarithm was replaced with a sigmoid of the logarithm of the Cauchy kernel, so that the attractive and repulsive terms become
Revised: The clipping of the arguments to the logarithm was replaced with a sigmoid of the logarithm of the Cauchy kernel, so that the attractive and repulsive terms become

Original: [Equation]
Revised: [Equation]

Original: where σ ( x ) = 1 / (cid:0) 1 + exp( − x ) (cid:1) is the sigmoid function.
Revised: where σ ( x ) = 1 / (cid:0) 1 + exp( − x ) (cid:1) is the sigmoid function.

Original: This change can seem drastic as, e.g., for ϕ ( ij )
Revised: This change can seem drastic as, e.g., for ϕ ( ij )

Original: = 1 we have max (cid:8) ε, ϕ ( ij ) (cid:9) = 1 , but σ (cid:16) log (cid:0) ϕ ( ij ) (cid:1)(cid:17) =
Revised: = 1 we have max (cid:8) ε, ϕ ( ij ) (cid:9) = 1 , but σ (cid:16) log (cid:0) ϕ ( ij ) (cid:1)(cid:17) =

Original: But unravelling the definitions shows that this turns the loss function precisely into our Neg- t -SNE loss function since
Revised: But unravelling the definitions shows that this turns the loss function precisely into our Neg- t -SNE loss function since

Original: [Equation]
Revised: [Equation]

Original: So, in order to overcome the numerical problems incurred by UMAP’s implicit choice of 1 /d 2 ij as similarity kernel, the pull request suggested a fix that turns out to be equivalent to negative sampling using the Cauchy kernel.
Revised: So, in order to overcome the numerical problems incurred by UMAP’s implicit choice of 1 /d 2 ij as similarity kernel, the pull request suggested a fix that turns out to be equivalent to negative sampling using the Cauchy kernel.

Original: We encourage this change to UMAP as it makes its loss function equivalent to our Neg- t -SNE and thus also conceptually more related to t -SNE.
Revised: We encourage this change to UMAP as it makes its loss function equivalent to our Neg- t -SNE and thus also conceptually more related to t -SNE.

Original: We also suggest to implement it in the non-parametric case.
Revised: We also suggest to implement it in the non-parametric case.

Original: F T RI M AP AND I NFO NC- t -SNE
Revised: F T RI M AP AND I NFO NC- t -SNE

Original: A neighbor embedding method TriMap (Amid & Warmuth, 2019) also employs a contrastive technique to compute the layout.
Revised: A neighbor embedding method TriMap (Amid & Warmuth, 2019) also employs a contrastive technique to compute the layout.

Original: TriMap considers triplets ( i, j, k ) , where i and j are data points that are more similar than i and k .
Revised: TriMap considers triplets ( i, j, k ) , where i and j are data points that are more similar than i and k .

Original: It moves the embeddings e i and e j closer together and pushes e i and e k further apart.
Revised: It moves the embeddings e i and e j closer together and pushes e i and e k further apart.

Original: While a full investigation of TriMap is beyond the scope of our work, here we argue that it bears some similarity to InfoNC- t -SNE with m = 1 .
Revised: While a full investigation of TriMap is beyond the scope of our work, here we argue that it bears some similarity to InfoNC- t -SNE with m = 1 .

Original: First, we describe TriMap in more detail.
Revised: First, we describe TriMap in more detail.

Original: Most of the triplets ( i, j, k ) consist of nearest neighbors i and j and a random point k .
Revised: Most of the triplets ( i, j, k ) consist of nearest neighbors i and j and a random point k .

Original: By default, 12 nearest neighbors j are chosen for each i , using a modified Euclidean distance.
Revised: By default, 12 nearest neighbors j are chosen for each i , using a modified Euclidean distance.

Original: For each such pair ( i, j ) additional distinct points k (by default 4 ) are chosen randomly from the set of all points excluding i and all the chosen j .
Revised: For each such pair ( i, j ) additional distinct points k (by default 4 ) are chosen randomly from the set of all points excluding i and all the chosen j .

Original: This leads totriplets ( i, j, k ) for each point i .
Revised: This leads totriplets ( i, j, k ) for each point i .

Original: Additionally, several (by default 3 ) triplets are formed for each i with random j and k only ensuring that the similarity between i and j is larger than between i and k .
Revised: Additionally, several (by default 3 ) triplets are formed for each i with random j and k only ensuring that the similarity between i and j is larger than between i and k .

Original: Thus, for each point i there are in total 51 triplets ( i, j, k ) leading to a grand total of 51 n triplets.
Revised: Thus, for each point i there are in total 51 triplets ( i, j, k ) leading to a grand total of 51 n triplets.

Original: These triplets are formed ahead of the layout optimization phase and kept fixed throughout.
Revised: These triplets are formed ahead of the layout optimization phase and kept fixed throughout.

Original: TriMap also assigns a weight w ijk to each triplet.
Revised: TriMap also assigns a weight w ijk to each triplet.

Original: This weight is larger for triplets in which i and j are much more similar to each other than i and k .
Revised: This weight is larger for triplets in which i and j are much more similar to each other than i and k .

Original: In our notation, TriMap’s loss function can be written as
Revised: In our notation, TriMap’s loss function can be written as

Original: [Equation]
Revised: [Equation]

Original: This loss is very similar to the loss of InfoNC- t -SNE with m = 1 :
Revised: This loss is very similar to the loss of InfoNC- t -SNE with m = 1 :

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: The differences are: the absence of log ; the presence of weights; the presence of random triplets; nearest neighbors being based not exactly on Euclidean metric; and fixed triplets during optimization.
Revised: The differences are: the absence of log ; the presence of weights; the presence of random triplets; nearest neighbors being based not exactly on Euclidean metric; and fixed triplets during optimization.

Original: In the following we modify the reference TriMap implementation and ablate these differences in order to see which ones are more or less important (Figure S3).
Revised: In the following we modify the reference TriMap implementation and ablate these differences in order to see which ones are more or less important (Figure S3).

Original: We measure the quality of all embeddings using two metrics (Tab. S1): as a global metric we use the Spearman correlation between high- and low-dimensional pairwise distances between 5000 randomly chosen points (Kobak & Berens, 2019) and as a local metric we use k NN recall with k = 

Revised: We measure the quality of all embeddings using two metrics (Tab. S1): as a global metric we use the Spearman correlation between high- and low-dimensional pairwise distances between 5000 randomly chosen points (Kobak & Berens, 2019) and as a local metric we use k NN recall with k =

Original: (Kobak & Berens, 2019).
Revised: (Kobak & Berens, 2019).

Original: First, we modified TriMap to use the 15 nearest neighbors in Euclidean metric.
Revised: First, we modified TriMap to use the 15 nearest neighbors in Euclidean metric.

Original: This did not change the visual appearance of the MNIST embedding (Fig.
Revised: This did not change the visual appearance of the MNIST embedding (Fig.

Original: S3a, b).
Revised: S3a, b).

Original: Next, we additionally omitted the random triplets.
Revised: Next, we additionally omitted the random triplets.

Original: The resulting embedding became visually closer to the InfoNC- t -SNE embedding (Fig.
Revised: The resulting embedding became visually closer to the InfoNC- t -SNE embedding (Fig.

Original: S3c).
Revised: S3c).

Original: Finally, additionally setting all the weights w ijk to 1 had only a small visual effect (Fig.
Revised: Finally, additionally setting all the weights w ijk to 1 had only a small visual effect (Fig.

Original: S3d).
Revised: S3d).

Original: Together, these simplifications noticeably improved the global metric compared to the default TriMap, while leaving the local metric almost unchanged (Tab. S1).
Revised: Together, these simplifications noticeably improved the global metric compared to the default TriMap, while leaving the local metric almost unchanged (Tab. S1).

Original: Similar to the experiments of Wang et al.
Revised: Similar to the experiments of Wang et al.

Original: , our ablations made TriMap look very similar to
Revised: , our ablations made TriMap look very similar to

Original: InfoNC- t -SNE with m = 1 (Fig.
Revised: InfoNC- t -SNE with m = 1 (Fig.

Original: S3e), suggesting that the remaining differences between them do not play a major role.
Revised: S3e), suggesting that the remaining differences between them do not play a major role.

Original: This refers to the details of the optimization (e.g. fixed triplets vs. randomly sampled triplets in each epoch, full-batch vs. stochastic gradient descent, momentum and adaptive learning rates, etc.), as well as to the presence/absence of the logarithm in the loss function.
Revised: This refers to the details of the optimization (e.g. fixed triplets vs. randomly sampled triplets in each epoch, full-batch vs. stochastic gradient descent, momentum and adaptive learning rates, etc.), as well as to the presence/absence of the logarithm in the loss function.

Original: Since
Revised: Since

Original: [Equation]
Revised: [Equation]

Original: the gradients of TriMap and InfoNC- t -SNE point in the same direction, but are differently scaled.
Revised: the gradients of TriMap and InfoNC- t -SNE point in the same direction, but are differently scaled.

Original: InfoNC- t -SNE prioritizes triplets with high (cid:0) ϕ
Revised: InfoNC- t -SNE prioritizes triplets with high (cid:0) ϕ

Original: ( ij )
Revised: ( ij )

Original: + ϕ ( ik ) (cid:1) /ϕ
Revised: + ϕ ( ik ) (cid:1) /ϕ

Original: ( ij ) , i.e. triplets in which the embedding similarity does not respect the triplet well.
Revised: ( ij ) , i.e. triplets in which the embedding similarity does not respect the triplet well.

Original: Note that this is conceptually different from
Revised: Note that this is conceptually different from

Original: TriMap’s weights w ijk which are based on the high-dimensional similarities.
Revised: TriMap’s weights w ijk which are based on the high-dimensional similarities.

Original: However, at least on
Revised: However, at least on

Original: MNIST, this difference in gradients did not have a major effect (Figs.
Revised: MNIST, this difference in gradients did not have a major effect (Figs.

Original: S3d, e).
Revised: S3d, e).

Original: Note that our default for InfoNC- t -SNE is m = 5 instead of m = 1 .
Revised: Note that our default for InfoNC- t -SNE is m = 5 instead of m = 1 .

Original: On MNIST, this setting yields a better local score and a similar global score compared to m = 1 , and much better scores than the default TriMap (Tab. S1).
Revised: On MNIST, this setting yields a better local score and a similar global score compared to m = 1 , and much better scores than the default TriMap (Tab. S1).

Original: The work of Ma & Collins (2018) theoretically underpins our observation in Fig.
Revised: The work of Ma & Collins (2018) theoretically underpins our observation in Fig.

Original: S18 that the higher m gets, the better InfoNC- t -SNE approximates t -SNE.
Revised: S18 that the higher m gets, the better InfoNC- t -SNE approximates t -SNE.

Original: G R ELATION TO OTHER VISUALIZATION METHODS G. t -SNE APPROXIMATIONS
Revised: G R ELATION TO OTHER VISUALIZATION METHODS G. t -SNE APPROXIMATIONS

Original: A vanilla implementation of t -SNE has to compute the partition function Z ( θ ) and hence scales quadratically with the sample size n .
Revised: A vanilla implementation of t -SNE has to compute the partition function Z ( θ ) and hence scales quadratically with the sample size n .

Original: Therefore, nearly all modern t -SNE packages implement approximations.
Revised: Therefore, nearly all modern t -SNE packages implement approximations.

Original: Common are Barnes-Hut- t -SNE (BH- t -SNE)
Revised: Common are Barnes-Hut- t -SNE (BH- t -SNE)

Original: (Yang et al., 2013; van der Maaten, 2014) and FFT-accelerated interpolation-based t -SNE (FI t -SNE) (Linderman et al., 2019).
Revised: (Yang et al., 2013; van der Maaten, 2014) and FFT-accelerated interpolation-based t -SNE (FI t -SNE) (Linderman et al., 2019).

Original: Both methods approximate the repulsive interaction between all pairs of points and the computation of the partition function, achieving complexities
Revised: Both methods approximate the repulsive interaction between all pairs of points and the computation of the partition function, achieving complexities

Original: O ( kn log( n )2 d ) and O ( kn 2 d ) , respectively.
Revised: O ( kn log( n )2 d ) and O ( kn 2 d ) , respectively.

Original: In fact, all our t -SNE plots are done with the FI t -SNE implementation of openTSNE (Poliˇcar et al., 2019).
Revised: In fact, all our t -SNE plots are done with the FI t -SNE implementation of openTSNE (Poliˇcar et al., 2019).

Original: The focus of our work is not primarily on accelerating t -SNE, but on understanding its relation to contrastive neighbor embeddings.
Revised: The focus of our work is not primarily on accelerating t -SNE, but on understanding its relation to contrastive neighbor embeddings.

Original: Different from BH- t -SNE and FI t -SNE, these do not approximately compute the partition function, but either learn it (NCVis), use a fixed normalization parameter (Neg- t -SNE and UMAP), or do not require it at all (InfoNC- t -SNE).
Revised: Different from the t -SNE approximations above, these do not approximately compute the partition function, but either learn it (NCVis), use a fixed normalization parameter (Neg- t -SNE and UMAP), or do not require it at all (InfoNC- t -SNE).

Original: In addition, instead of considering the repulsive interactions between all pairs of points, they only sample nm repulsive interaction per epoch.
Revised: They only sample nm repulsive interactions per epoch.

Original: This cuts their complexity down to O ( kmnd ) .
Revised: This cuts their complexity down to O ( kmnd ) .

Original: In particular, they scale linearly with the embedding dimension d and can therefore be used for more general representation learning with d ≫ 2 , unlike BH- t -SNE and FI t -SNE.
Revised: In particular, they scale linearly with the embedding dimension d and can therefore be used for more general representation learning with d ≫ 2 , unlike BH- t -SNE and FI t -SNE.

Revised: G. O THER CONTRASTIVE t -SNE APPROXIMATIONS

Revised: In addition to NCVis and InfoNC- t -SNE, there are two other, recent sampling-based strategies to approximate t -SNE, GDR- t -SNE (Draganov et al., 2022) and SCE (Yang et al., 2021).

Revised: Both sample m repulsive interactions per each attractive interaction.

Original: G. L ARGE V IS
Revised: In order to scale these repulsive interactions properly, an estimate of the partition function is computed. In GDR- t -SNE, the sum of the Cauchy kernels of the mkn repulsive interactions over each epoch are added up to obtain an estimate of Z t -SNE ( θ ) · mkn/ (cid:0) n ( n − 1) (cid:1) . Since both the number of repulsive interactions and the estimate of the partition function are decreased by a factor of mk/ ( n − 1) compared to the vanilla t -SNE, the overall repulsion strength is correct.

Revised: In contrast, SCE (Yang et al., 2021) initializes its estimate of the partition function with the maximum value n ( n − 1) and then updates it using a moving average after each sampled repulsive pair.

Revised: Both Draganov et al.

Revised: (2022) and Yang et al.

Revised: (2021) use m = 1 .

Revised: These approaches can be seen as a different flavour of NCE (in addition to standard NCE and to InfoNCE): instead of fitting Z like NCE does, they estimate

Revised: Z using the sampled noise points.

Revised: G. L ARGE V IS

Original: LargeVis (Tang et al., 2016) was, to the best of our knowledge, the first contrastive neighbor embedding method.
Revised: LargeVis (Tang et al., 2016) was, to the best of our knowledge, the first contrastive neighbor embedding method.

Original: It uses NEG.
Revised: It uses NEG.

Original: Since it was quickly overshadowed by very similar UMAP, we focused the exposition in the main paper on UMAP.
Revised: Since it was quickly overshadowed by very similar UMAP, we focused the exposition in the main paper on UMAP.

Original: Damrich & Hamprecht (2021) computed LargeVis’ effective loss function in closed form.
Revised: Damrich & Hamprecht (2021) computed LargeVis’ effective loss function in closed form.

Original: In our notation it reads
Revised: In our notation it reads

Original: [Equation]
Revised: [Equation]

Original: The are some implementation differences between LargeVis and UMAP, but in terms of the loss function, the main difference is the γ factor in front of the repulsive term, which defaults to 7 .
Revised: The are some implementation differences between LargeVis and UMAP, but in terms of the loss function, the main difference is the γ factor in front of the repulsive term, which defaults to 7 .

Original: Therefore, our analysis of UMAP carries over to LargeVis: Its loss is essentially an instance of
Revised: Therefore, our analysis of UMAP carries over to LargeVis: Its loss is essentially an instance of

Original: NEG with inverse square kernel.
Revised: NEG with inverse square kernel.

Original: The additional factor γ = 7 moves it along the attraction-repulsion spectrum towards more repulsion, that is, towards t -SNE.
Revised: The additional factor γ = 7 moves it along the attraction-repulsion spectrum towards more repulsion, that is, towards t -SNE.

Original: G. P A CMAP
Revised: G. P A CMAP

Original: Pairwise Controlled Manifold Approximation Projection (PaCMAP) (Wang et al., 2021) is a recent visualization method also based on sampling m repulsive forces per one attractive force (with m =by default).
Revised: Pairwise Controlled Manifold Approximation Projection (PaCMAP) (Wang et al., 2021) is a recent visualization method also based on sampling m repulsive forces per one attractive force (with m =by default).

Original: It employs a large number of additional design choices, such as weak attraction between ‘mid-near’ points in addition to attraction between nearest neighbors (which are based on a modified
Revised: It employs a large number of additional design choices, such as weak attraction between ‘mid-near’ points in addition to attraction between nearest neighbors (which are based on a modified

Original: Euclidean distance, as in TriMap), several optimization regimes with dynamically changing loss weights, etc.
Revised: Euclidean distance, as in TriMap), several optimization regimes with dynamically changing loss weights, etc.

Original: Nevertheless, the core parts of its loss can be related to (generalized)
Revised: Nevertheless, the core parts of its loss can be related to (generalized)

Original: Neg-
Revised: Neg-

Original: t -SNE.
Revised: t -SNE.

Original: The attractive loss in PaCMAP is
Revised: The attractive loss in PaCMAP is

Original: [Equation]
Revised: [Equation]

Original: with c = 10 for nearest neighbors (and c = 10 000 for mid-near points but that part of the loss is switched off during the final stage of the optimization).
Revised: with c = 10 for nearest neighbors (and c = 10 000 for mid-near points but that part of the loss is switched off during the final stage of the optimization).

Original: The attractive loss for Neg- t -SNE with uniform noise distribution and ¯ Z = c
Revised: The attractive loss for Neg- t -SNE with uniform noise distribution and ¯ Z = c

Original: | X | /m is
Revised: | X | /m is

Original: [Equation]
Revised: [Equation]

Original: Thus, the differences are the logarithm and some rescaling of the distances.
Revised: Thus, the differences are the logarithm and some rescaling of the distances.

Original: Note that Neg- t -SNE with c = 10 typically produces very similar result compared to the default c = 1 (cf.
Revised: Note that Neg- t -SNE with c = 10 typically produces very similar result compared to the default c = 1 (cf.

Original: Fig 1).
Revised: Fig 1).

Original: The repulsive loss in PaCMAP is
Revised: The repulsive loss in PaCMAP is

Original: [Equation]
Revised: [Equation]

Original: Again it is similar to the repulsive loss of Neg- t -SNE, but now for ¯ Z = | X | /m , the default negative sampling value also used by UMAP:
Revised: Again it is similar to the repulsive loss of Neg- t -SNE, but now for ¯ Z = | X | /m , the default negative sampling value also used by UMAP:

Original: [Equation]
Revised: [Equation]

Original: Here, the difference is just in the logarithm, similar to the connection between TriMap and
Revised: Here, the difference is just in the logarithm, similar to the connection between TriMap and

Original: InfoNC- t -SNE in Supp.
Revised: InfoNC- t -SNE in Supp.

Original: F.
Revised: F.

Original: Empirically, PaCMAP embeddings of high-dimensional data like MNIST look similar to UMAP embeddings, and hence to Neg- t -SNE embeddings, in agreement with our analysis.
Revised: Empirically, PaCMAP embeddings of high-dimensional data like MNIST look similar to UMAP embeddings, and hence to Neg- t -SNE embeddings, in agreement with our analysis.

Original: H Q UANTITATIVE E VALUATION
Revised: H Q UANTITATIVE E VALUATION

Original: This section provides a quantitative evaluation of the Neg- t -SNE spectrum.
Revised: This section provides a quantitative evaluation of the Neg- t -SNE spectrum.

Original: For each embedding on the spectrum, we compute two metrics.
Revised: For each embedding on the spectrum, we compute two metrics.

Original: The first metric, k NN recall with k = 

Revised: The first metric, k NN recall with k =

Original: (Kobak &
Revised: (Kobak &

Original: Berens, 2019), measures the fraction of nearest neighbors in the embedding that are also nearest neighbors in the reference configuration.
Revised: Berens, 2019), measures the fraction of nearest neighbors in the embedding that are also nearest neighbors in the reference configuration.

Original: It is therefore a measure of local quality.
Revised: It is therefore a measure of local quality.

Original: The second metric is the Spearman correlation between the pairwise distances in the embedding and in the reference configuration (Kobak & Berens, 2019).
Revised: The second metric is the Spearman correlation between the pairwise distances in the embedding and in the reference configuration (Kobak & Berens, 2019).

Original: To speed up the computation, we consider all pairwise distances between a sample of 5000 points.
Revised: To speed up the computation, we consider all pairwise distances between a sample of 5000 points.

Original: Since most random pairs of points are not close to each other, this is a measure of global quality.
Revised: Since most random pairs of points are not close to each other, this is a measure of global quality.

Original: As reference configuration, we use three different layouts for each dataset.
Revised: As reference configuration, we use three different layouts for each dataset.

Original: We compare against the original high-dimensional data to measure how faithful Neg- t -SNE embeddings are.
Revised: We compare against the original high-dimensional data to measure how faithful Neg- t -SNE embeddings are.

Original: Additionally, we compare to the corresponding t -SNE and UMAP embeddings to investigate for which ¯ Z value
Revised: Additionally, we compare to the corresponding t -SNE and UMAP embeddings to investigate for which ¯ Z value

Original: Neg- t -SNE matches t -SNE and UMAP the best.
Revised: Neg- t -SNE matches t -SNE and UMAP the best.

Original: We found that compared to the high-dimensional data, the k NN recall followed an inverse U-shape across the spectrum (Fig.
Revised: We found that compared to the high-dimensional data, the k NN recall followed an inverse U-shape across the spectrum (Fig.

Original: S4a).
Revised: S4a).

Original: k NN recall was low for very large and for very small values of ¯ Z .
Revised: k NN recall was low for very large and for very small values of ¯ Z .

Original: It peaked when ¯ Z was close to t -SNE’s partition function Z t -SNE and NCVis’ learned normalization parameter Z NCVis .
Revised: It peaked when ¯ Z was close to t -SNE’s partition function Z t -SNE and NCVis’ learned normalization parameter Z NCVis .

Original: At UMAP’s normalization constant ¯ Z UMAP the k NN recall was usually lower.
Revised: At UMAP’s normalization constant ¯ Z UMAP the k NN recall was usually lower.

Original: This confirms our observation that the local quality of the embedding improves when decreasing ¯ Z from ¯ Z UMAP to Z t -SNE in agreement with B ¨ ohm et al.
Revised: This confirms our observation that the local quality of the embedding improves when decreasing ¯ Z from ¯ Z UMAP to Z t -SNE in agreement with B¨ohm et al.

Original: The k NN recall for the Kuzushijidataset at ¯ Z =
Revised: The k NN recall for the Kuzushijidataset at ¯ Z =

Original: Z t -SNE was very low, compared to other datasets.
Revised: Z t -SNE was very low, compared to other datasets.

Original: We suspect that this was due to incomplete convergence (Fig.
Revised: We suspect that this was due to incomplete convergence (Fig.

Original: S16, Tab. S4).
Revised: S16, Tab. S4).

Original: Conversely, the Spearman correlation mostly increased with ¯ Z (Fig. S4b), which aligns with our finding that higher attraction improves the global layout of the embedding.
Revised: Conversely, the Spearman correlation mostly increased with ¯ Z (Fig. S4b), which aligns with our finding that higher attraction improves the global layout of the embedding.

Revised: We also computed the k NN recall and the Spearman correlation for the proper t -SNE and UMAP embeddings (not depicted in Figs.

Revised: S4a, b).

Revised: The k NN recall was higher for t -SNE embeddings than for Neg- t -SNE at ¯ Z

Revised: = Z t -SNE (see, e.g., Fig.

Revised: S17l), likely because proper t -SNE considers the repulsive interaction between all points.

Original:  When comparing Neg- t -SNE embeddings to the proper t -SNE embedding, the best fit in terms of k NN recall (Fig.
Revised: The metrics for the UMAP embeddings and the Spearman When comparing Neg- t -SNE embeddings to the proper t -SNE embedding, the best fit in terms of k NN recall (Fig.

Original: S4c)
Revised: S4c)

Original: and in terms of Spearman correlation (Fig. S4d) was usually achieved when ¯ Z was close to Z t -SNE , in accordance with our theory.
Revised: and in terms of Spearman correlation (Fig. S4d) was usually achieved when ¯ Z was close to Z t -SNE , in accordance with our theory.

Original: Again, the Kuzushiji-49 dataset was an outlier, likely due to convergence issues.
Revised: Again, the Kuzushiji-49 dataset was an outlier, likely due to convergence issues.

Original: Finally, when comparing Neg- t -SNE embeddings to the UMAP embedding, the highest Spearman correlation was achieved by ¯ Z ≈ ¯ Z UMAP (Fig.
Revised: Finally, when comparing Neg- t -SNE embeddings to the UMAP embedding, the highest Spearman correlation was achieved by ¯ Z ≈ ¯ Z UMAP (Fig.

Original: S4f), in agreement with our theoretical predictions.
Revised: S4f), in agreement with our theoretical predictions.

Original: The highest k NN recall was typically achieved at a slightly lower ¯ Z (Fig. S4e).
Revised: The highest k NN recall was typically achieved at a slightly lower ¯ Z (Fig. S4e).

Original: Overall, these experiments provide empirical support for our interpretation of the Neg- t -SNE spectrum as implementing a trade-off between global and local structure preservation, and confirm the predicted locations of t -SNE- and UMAP-like embeddings on the spectrum.
Revised: Overall, these experiments provide empirical support for our interpretation of the Neg- t -SNE spectrum as implementing a trade-off between global and local structure preservation, and confirm the predicted locations of t -SNE- and UMAP-like embeddings on the spectrum.

Revised: We computed the k NN recall and the Spearman correlation also for the embeddings in Figs.

Revised: S and S18, where we varied the number of noise samples m and the initialization method for NCVis and InfoNC- t -SNE.

Revised: As expected, we found that higher m improves the k NN recall, but t -SNE proper achieved higher k NN recall still.

Revised: The Spearman correlation decreased with m when we initialized randomly and did not change much for other initialization methods.

Revised: Random initialization hurt the Spearman correlation significantly.

Original: I T OY DATASET
Revised: I T OY DATASET

Original: Cor.
Revised: Cor.

Original: 2 predicts that the partition function of a Neg- t -SNE embedding should equal the ¯ Z value.
Revised: 2 predicts that the partition function of a Neg- t -SNE embedding should equal the ¯ Z value.

Original: In our real-world examples (panels i in Figs.
Revised: In our real-world examples (panels i in Figs.

Original: 1, S10, S11, S12, etc.)
Revised: 1, S10, S11, S12, etc.)

Original: we observed a monotone relationship between them, but not a perfect agreement.
Revised: we observed a monotone relationship between them, but not a perfect agreement.

Original: As discussed in Sec. 5, this is due to the shape of the Cauchy kernel and the fact that the data distribution is zero for many pairs of points.
Revised: As discussed in Sec. 5, this is due to the shape of the Cauchy kernel and the fact that the data distribution is zero for many pairs of points.

Original: Here, we consider a toy example for which we observe a perfect match between Neg- t -SNE’s partition function and the ¯ Z value, confirming our Cor. 2.[Figure]
Revised: Here, we consider a toy example for which we observe a perfect match between Neg- t -SNE’s partition function and the ¯ Z value, confirming our Cor. 2.

Original: As we operate with the binary s k NN graph, the only possible high-dimensional similarities are zero and one.
Revised: As we operate with the binary s k NN graph, the only possible high-dimensional similarities are zero and one.

Original: Due to the heavy tail of the Cauchy kernel, we would like our toy example to have no pairs of points for which the data distribution is zero.
Revised: Due to the heavy tail of the Cauchy kernel, we would like our toy example to have no pairs of points for which the data distribution is zero.

Original: Thus, all pairs of points need to equal high-dimensional similarity.
Revised: Thus, all pairs of points need to equal high-dimensional similarity.

Original: In two dimensions, only up to three points can be placed equidistantly from each other.
Revised: In two dimensions, only up to three points can be placed equidistantly from each other.

Original: Therefore, we consider the simple case of placing three points according to the
Revised: Therefore, we consider the simple case of placing three points according to the

Original: Neg- t -SNE loss function with p ≡ 16 (there are six directed edges) for various values of ¯ Z .
Revised: Neg- t -SNE loss function with p ≡ 16 (there are six directed edges) for various values of ¯ Z .

Original: We keep the Cauchy kernel as similarity function, which has maximum 1 / (1 + 0 2 ) = 1 .
Revised: We keep the Cauchy kernel as similarity function, which has maximum 1 / (1 + 0 2 ) = 1 .

Original: Thus, it is not possible to match values above ¯ Z = 6 , and the three points end up having the same position in the embedding.
Revised: Thus, it is not possible to match values above ¯ Z = 6 , and the three points end up having the same position in the embedding.

Original: But for smaller values of ¯ Z , the resulting partition function is indeed exactly equal to ¯
Revised: But for smaller values of ¯ Z , the resulting partition function is indeed exactly equal to ¯

Original: Z (Fig. S5).
Revised: Z (Fig. S5).

Original: For this experiment we decreased the batch size to 6 and the learning rate to 0 .
Revised: For this experiment we decreased the batch size to 6 and the learning rate to 0 .

Original: 01 , but kept all other hyperparameters at their default values.
Revised: 01 , but kept all other hyperparameters at their default values.

Original: J D ATASETS
Revised: J D ATASETS

Original: We used the well-known MNIST (LeCun et al., 1998) dataset for most of our experiments.
Revised: We used the well-known MNIST (LeCun et al., 1998) dataset for most of our experiments.

Original: We downloaded it via the torchvision API from http://yann.lecun.com/exdb/mnist/ .
Revised: We downloaded it via the torchvision API from http://yann.lecun.com/exdb/mnist/ .

Original: This website does not give a license.
Revised: This website does not give a license.

Original: But https://keras.io/api/datasets/mnist/ and http://www.pymvpa.org/datadb/mnist.html name Yann LeCun and Corinna Cortes as copyright holders and claim MNIST to be licensed under CC BY-SA 3.0, which permits use and adaptation.
Revised: But https://keras.io/api/datasets/mnist/ and http://www.pymvpa.org/datadb/mnist.html name Yann LeCun and Corinna Cortes as copyright holders and claim MNIST to be licensed under CC BY-SA 3.0, which permits use and adaptation.

Original: The MNIST dataset consists of 70 000 grayscale images, 28 × 28 pixels each, that show handwritten digits.
Revised: The MNIST dataset consists of 70 000 grayscale images, 28 × 28 pixels each, that show handwritten digits.

Original: The Kuzushiji-49 dataset (Tarin et al., 2018) was downloaded from https://github.com/ rois-codh/kmnist where it is licensed under CC-BY-4.0.
Revised: The Kuzushiji-49 dataset (Tarin et al., 2018) was downloaded from https://github.com/ rois-codh/kmnist where it is licensed under CC-BY-4.0.

Original: It contains 270 912 grayscale images, 28 × 28 pixels each, that show 49 different cursive Japanese characters. 
Revised: It contains 270 912 grayscale images, 28 × 28 pixels each, that show 49 different cursive Japanese characters. [Figure]

Original: The
Revised: The

Original: SimCLR experiments were performed on the
Revised: SimCLR experiments were performed on the

Original: CIFAR-
Revised: CIFAR-

Original: (Krizhevsky, 2009) dataset, another standard machine learning resource.
Revised: (Krizhevsky, 2009) dataset, another standard machine learning resource.

Original: We downloaded it via the sklearn.datasets.fetch openml API from https://www.openml.org/search? type=data&sort=runs&id=40927&status=active .
Revised: We downloaded it via the sklearn.datasets.fetch openml API from https://www.openml.org/search? type=data&sort=runs&id=40927&status=active .

Original: Unfortunately, we were not able to find a license for this dataset.
Revised: Unfortunately, we were not able to find a license for this dataset.

Original: CIFAR-10 consists of 60 000 images, 32 × 32 RGB pixels each, depicting objects from five animal and five vehicle classes.
Revised: CIFAR-10 consists of 60 000 images, 32 × 32 RGB pixels each, depicting objects from five animal and five vehicle classes.

Original: The transcriptomic dataset of Kanton et al.
Revised: The transcriptomic dataset of Kanton et al.

Original: (2019) was downloaded from https: //www.ebi.ac.uk/arrayexpress/experiments/E-MTAB-7552/ , which permits free use.
Revised: (2019) was downloaded from https: //www.ebi.ac.uk/arrayexpress/experiments/E-MTAB-7552/ , which permits free use.

Original: We only used the 20 272 cells in the human brain organoid cell line ‘409b2’.
Revised: We only used the 20 272 cells in the human brain organoid cell line ‘409b2’.

Original: The transcriptomic dataset (Wagner et al., 2018) was downloaded in its scanpy version from https://kleintools.hms.harvard.edu/paper_websites/ wagner_zebrafish_timecourse2018/mainpage.html .
Revised: The transcriptomic dataset (Wagner et al., 2018) was downloaded in its scanpy version from https://kleintools.hms.harvard.edu/paper_websites/ wagner_zebrafish_timecourse2018/mainpage.html .

Original: The full dataset at https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE is free to download and reproduce.
Revised: The full dataset at https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE is free to download and reproduce.

Original: The dataset contains gene expressions for 63 530 cells from a developing zebrafish embryo.
Revised: The dataset contains gene expressions for 63 530 cells from a developing zebrafish embryo.

Original: The downloaded UMIs of both datasets were preprocessed as in (B¨ohm et al., 2022; Kobak & Berens, 2019).
Revised: The downloaded UMIs of both datasets were preprocessed as in (B ¨ ohm et al., 2022; Kobak & Berens, 2019).

Original: After selecting the 1000 most variable genes, we normalized the library sizes to the median library size in the dataset, log-transformed the normalized values with log 2 ( x + 1) , and finally reduced the dimensionality to 50 via PCA.
Revised: After selecting the 1000 most variable genes, we normalized the library sizes to the median library size in the dataset, log-transformed the normalized values with log 2 ( x + 1) , and finally reduced the dimensionality to 50 via PCA.

Original: The trascriptomic dataset of the C. elegans flatworm (Packer et al., 2019;
Revised: The trascriptomic dataset of the C. elegans flatworm (Packer et al., 2019;

Original: Narayan et al., 2021) was obtained from http://cb.csail.mit.edu/cb/densvis/datasets/ with consent of the authors who license it under CC BY-NC 2.0.
Revised: Narayan et al., 2021) was obtained from http://cb.csail.mit.edu/cb/densvis/datasets/ with consent of the authors who license it under CC BY-NC 2.0.

Original: It is already preprocessed to 100 principal components.
Revised: It is already preprocessed to 100 principal components.

Original: K I MPLEMENTATION
Revised: K I MPLEMENTATION

Original: All contrastive embeddings were computed with our PyTorch (Paszke et al., 2019) implementation of Neg- t -SNE, NCVis, UMAP, and InfoNC- t -SNE.
Revised: All contrastive embeddings were computed with our PyTorch (Paszke et al., 2019) implementation of Neg- t -SNE, NCVis, UMAP, and InfoNC- t -SNE.

Original: Exceptions are Fig.
Revised: Exceptions are Fig.

Original: 1, and analogous Figs.
Revised: 1, and analogous Figs.

Original: S10 –
Revised: S10 –

Original: S15.
Revised: S15.

Original: There for panel h we used the reference implementation of NCVis (Artemenkov & Panov, 2020) (with a fixed number of noise samples m , and not the default schedule), and for panel g we used UMAP 0.5.
Revised: There for panel h we used the reference implementation of NCVis (Artemenkov & Panov, 2020) (with a fixed number of noise samples m , and not the default schedule), and for panel g we used UMAP 0.5.

Original: The t -SNE plots were created with the openTSNE (Poliˇcar et al., 2019) (version 0.6.1) package.
Revised: The t -SNE plots were created with the openTSNE (Poli ˇ car et al., 2019) (version 0.6.1) package.

Original: Similarly, we used the reference UMAP implementation in Fig.
Revised: Similarly, we used the reference UMAP implementation in Fig.

Original: S1 and openTSNE in Figs.
Revised: S1 and openTSNE in Figs.

Original: S17 and S18.
Revised: S17 and S18.

Original: The TriMap plots in Supp.
Revised: The TriMap plots in Supp.

Original: F were computed with version 1.1.4 of the TriMap package by Amid & Warmuth (2019).
Revised: F were computed with version 1.1.4 of the TriMap package by Amid & Warmuth (2019).

Original: We extended these implementations of NCVis, UMAP, and t -SNE to make them accept custom embedding initializations and unweighted s k NN graphs and to log various quantities of interest.
Revised: We extended these implementations of NCVis, UMAP, and t -SNE to make them accept custom embedding initializations and unweighted s k NN graphs and to log various quantities of interest.

Original: We always used the standard Cauchy kernel for better comparability.
Revised: We always used the standard Cauchy kernel for better comparability.

Original: All PCAs were computed with sklearn (Pedregosa et al., 2011).
Revised: All PCAs were computed with sklearn (Pedregosa et al., 2011).

Original: We used PyKeOps (Charlier et al., 2021) to compute the exact s k NN graph and to handle the quadratic complexity of computing the partition functions on a GPU.
Revised: We used PyKeOps (Charlier et al., 2021) to compute the exact s k NN graph and to handle the quadratic complexity of computing the partition functions on a GPU.

Original: The same PCA initialization and s k NN graph with k = 15 were used for all embeddings.
Revised: The same PCA initialization and s k NN graph with k = 15 were used for all embeddings.

Original: The s k NN graph for MNIST and Kuzushiji-49 was computed on a 50dimensional PCA of the dataset.
Revised: The s k NN graph for MNIST and Kuzushiji-49 was computed on a 50dimensional PCA of the dataset.

Original: When computing logarithms during the optimization of neighbor embeddings, we clip the arguments to the range [10 − 10 , 1] , save for Fig.
Revised: When computing logarithms during the optimization of neighbor embeddings, we clip the arguments to the range [10 − 10 , 1] , save for Fig.

Original: S2, where we ablated this lower bound.
Revised: S2, where we varied this lower bound.

Original: The lower bound is smaller than in the reference implementation of parametric UMAP, where it is set to 10 − 4 .
Revised: The lower bound is smaller than in the reference implementation of parametric UMAP, where it is set to 10 − 4 .

Original: Our defaults were a batch size of 1024 , linear learning rate annealing from 1 (non-parametric) or 0 .
Revised: Our defaults were a batch size of 1024 , linear learning rate annealing from 1 (non-parametric) or 0 .

Original: 001 (parametric) to 0 (save for Figs. 2, S1, and S2), 750 epochs (save for Figs.
Revised: 001 (parametric) to 0 (save for Figs. 2, S1, and S2), 750 epochs (save for Figs.

Original: S9 and S16) and m = 5 noise samples (save for Figs.
Revised: S9 and S16) and m = 5 noise samples (save for Figs.

Original: S7, S9, S17, and S18).
Revised: S7, S9, S17, and S18).

Original: We initialized all embeddings with a scaled version of PCA (save for in Figs.
Revised: We initialized all embeddings with a scaled version of PCA (save for in Figs.

Original: S17, and S18).
Revised: S17, and S18).

Original: For t -SNE embeddings we rescaled the initialization so that the first dimension has a standard deviation of 0 . 

Revised: For t -SNE embeddings we rescaled the initialization so that the first dimension has a standard deviation of 0 .

Original: (as is default in openTSNE), for all other embeddings to a standard deviation of 1 .
Revised: (as is default in openTSNE), for all other embeddings to a standard deviation of 1 .

Original: We employed some version of ‘early exaggeration’ (van der Maaten & Hinton, 2008) for the firstepochs in most non-parametric plots.
Revised: We employed some version of ‘early exaggeration’ (van der Maaten & Hinton, 2008) for the firstepochs in most non-parametric plots.

Original: For t -SNE it is the default early exaggeration of openTSNE.
Revised: For t -SNE it is the default early exaggeration of openTSNE.

Original: When varying ¯ Z in non-parametric Neg- t -SNE, early exaggeration meant using ¯ Z = | X | /m for the first 250 epochs (save for Fig.
Revised: When varying ¯ Z in non-parametric Neg- t -SNE, early exaggeration meant using ¯ Z = | X | /m for the first 250 epochs (save for Fig.

Original: S10).
Revised: S10).

Original: When varying the number of noise samples in Figs.
Revised: When varying the number of noise samples in Figs.

Original: S7, S and S18, we still used m = 5 for the first 250 epochs.
Revised: S7, S and S18, we still used m = 5 for the first 250 epochs.

Original: In Figs. 2, 3, S1, and S2 as well as in all reference NCVis or UMAP plots, we did not use early exaggeration as neither small ¯ Z nor high m made it necessary.
Revised: In Figs. 2, 3, S1, and S2 as well as in all reference NCVis or UMAP plots, we did not use early exaggeration as neither small ¯ Z nor high m made it necessary.

Original: When we used some form of early exaggeration and learning rate annealing, the annealing to zero took place over the first 250 epochs, was then reset, and annealed again to zero for the remaining, typically 500 , epochs.
Revised: When we used some form of early exaggeration and learning rate annealing, the annealing to zero took place over the first 250 epochs, was then reset, and annealed again to zero for the remaining, typically 500 , epochs.

Original: Non-parametric runs were optimized with SGD without momentum and parametric runs with the
Revised: Non-parametric runs were optimized with SGD without momentum and parametric runs with the

Original: Adam optimizer (Kingma & Ba, 2015).
Revised: Adam optimizer (Kingma & Ba, 2015).

Original: Parametric runs used the same feed-forward neural net architecture as the reference parametric UMAP implementation.
Revised: Parametric runs used the same feed-forward neural net architecture as the reference parametric UMAP implementation.

Original: That is, four layers with dimensions input dimension − 100 − 100 − 100 − 2 with ReLU activations in all but the last one.
Revised: That is, four layers with dimensions input dimension − 100 − 100 − 100 − 2 with ReLU activations in all but the last one.

Original: We used the vectorized, 786 -dimensional version of MNIST as input to the parametric neighbor embedding methods (and not the 50-dimensional PCA; but the s k NN graph was computed in the PCA space for consistency with non-parametric embeddings).
Revised: We used the vectorized, 786 -dimensional version of MNIST as input to the parametric neighbor embedding methods (and not the 50-dimensional PCA; but the s k NN graph was computed in the PCA space for consistency with non-parametric embeddings).

Original: Like the reference NCVis implementation, we used the fractions q θ,Z ( x ) /
Revised: Like the reference NCVis implementation, we used the fractions q θ,Z ( x ) /

Original: ( q θ,Z ( x ) + m ) instead of q θ,Z ( x ) / (cid:0) q θ,Z ( x ) + mξ ( x ) (cid:1) .
Revised: ( q θ,Z ( x ) + m ) instead of q θ,Z ( x ) / (cid:0) q θ,Z ( x ) + mξ ( x ) (cid:1) .

Original: This is a mild approximation as the noise distribution is close to uniform.
Revised: This is a mild approximation as the noise distribution is close to uniform.

Original: But it means that the model learns a scaled data distribution (cf.
Revised: But it means that the model learns a scaled data distribution (cf.

Original: Cor. 2), so that we need to multiply the learned normalization parameter Z by n ( n − 1) when comparing to t -SNE or checking normalization of the NCVis model.
Revised: Cor. 2), so that we need to multiply the learned normalization parameter Z by n ( n − 1) when comparing to t -SNE or checking normalization of the NCVis model.

Original: Similarly, we also approximate the true noise distribution by the uniform distribution for the fractions q θ ( x ) /
Revised: Similarly, we also approximate the true noise distribution by the uniform distribution for the fractions q θ ( x ) /

Original: ( q θ ( x ) + ¯ Zm/ | X | ) — instead of q θ ( x ) / (cid:0) q θ ( x )
Revised: ( q θ ( x ) + ¯ Zm/ | X | ) — instead of q θ ( x ) / (cid:0) q θ ( x )

Original: + ¯ Zmξ ( x ) (cid:1)
Revised: + ¯ Zmξ ( x ) (cid:1)

Original: — in our Neg- t -SNE implementation.
Revised: — in our Neg- t -SNE implementation.

Original: We mentioned in Sec. 5 and showed in Fig.
Revised: We mentioned in Sec. 5 and showed in Fig.

Original: S7 that one can move along the attraction-repulsion spectrum also by changing the number of noise samples m , instead of the fixed normalization constant ¯ Z .
Revised: S7 that one can move along the attraction-repulsion spectrum also by changing the number of noise samples m , instead of the fixed normalization constant ¯ Z .

Original: In UMAP’s reference implementation, there is a scalar prefactor γ for the repulsive forces.
Revised: In UMAP’s reference implementation, there is a scalar prefactor γ for the repulsive forces.

Original: [Table] NCVis (our implementation) for [Table] Theoretically, adjusting γ should also move along the attraction-repulsion spectrum, but setting it higher than 1 led to convergence problems in (B¨ohm et al., 2022), Fig.
Revised: Theoretically, adjusting γ should also move along the attraction-repulsion spectrum, but setting it higher than 1 led to convergence problems in (B¨ohm et al., 2022), Fig.

Original: A11.
Revised: A11.

Original: When varying our ¯ Z , we did not have such issues.
Revised: When varying our ¯ Z , we did not have such issues.

Original: For panels i in Figs. 1, S10 – S15 the Neg- t -SNE spectra were computed for ¯ Z equal to Z ( θ t -SNE ) ,
Revised: For panels i in Figs. 1, S10 – S15 the Neg- t -SNE spectra were computed for ¯ Z equal to Z ( θ t -SNE ) ,

Original: Z NCVis , and n ( n − 1) m · x , where x ∈ { 5 · 10 − 5 , 1 · 10 − 4 , 2 · 10 − 4 , 5 · 10 − 4 , . .
Revised: Z NCVis , and n ( n − 1) m · x , where x ∈ { 5 · 10 − 5 , 1 · 10 − 4 , 2 · 10 − 4 , 5 · 10 − 4 , . .

Original: For the SimCLR experiments, we trained the model for 1000 epochs, of which we used 5 epochs for warmup.
Revised: For the SimCLR experiments, we trained the model for 1000 epochs, of which we used 5 epochs for warmup.

Original: The learning rate during warmup was linearly interpolated from 0 to the initial learning rate.
Revised: The learning rate during warmup was linearly interpolated from 0 to the initial learning rate.

Original: After the warmup epochs, we annealed the learning rate with a cosine schedule (without restarts) to 0 (Loshchilov & Hutter, 2017).
Revised: After the warmup epochs, we annealed the learning rate with a cosine schedule (without restarts) to 0 (Loshchilov & Hutter, 2017).

Original: We optimized the model parameters with SGD and momentum 0 .
Revised: We optimized the model parameters with SGD and momentum 0 .

Original: We used the same data augmentations as in (Chen et al., 2020).
Revised: We used the same data augmentations as in (Chen et al., 2020).

Original: We used a
Revised: We used a

Original: ResNet18 (He et al., 2016) as the backbone and a projection head consisting of two linear layers ( 512 − 1024 − 128 ) with a ReLU activation in-between.
Revised: ResNet18 (He et al., 2016) as the backbone and a projection head consisting of two linear layers ( 512 − 1024 − 128 ) with a ReLU activation in-between.

Original: The loss was applied to the L 2 normalized output of the projection head, but like Chen et al.
Revised: The loss was applied to the L 2 normalized output of the projection head, but like Chen et al.

Original: we used the output of the ResNet as the representation for the linear evaluation.
Revised: we used the output of the ResNet as the representation for the linear evaluation.

Original: As similarity function we used the exponential of the normalized scalar product (cosine similarity) and always kept the temperature at 0 .
Revised: As similarity function we used the exponential of the normalized scalar product (cosine similarity) and always kept the temperature at 0 .

Original: 5 , as suggested in
Revised: 5 , as suggested in

Original: Chen et al.
Revised: Chen et al.

Original: We clipped the arguments to logarithms to be non-negative.
Revised: We clipped the arguments to logarithms to be non-negative.

Original: When considering the entire batch as negative samples, we omit both head and tail of the considered positive sample, while we only omit the head when sampling a smaller number of negative samples.
Revised: When considering the entire batch as negative samples, we omit both head and tail of the considered positive sample, while we only omit the head when sampling a smaller number of negative samples.

Original: The ResNet was trained on the combined CIFAR-10 train and test sets.
Revised: The ResNet was trained on the combined CIFAR-10 train and test sets.

Original: When training the classifier, we froze the ResNet and only used the train set.
Revised: When training the classifier, we froze the ResNet and only used the train set.

Original: The reported metrics are computed on the test set.
Revised: The reported metrics are computed on the test set.

Original: We chose sklearn’s KNearestNeighbors classifier with cosine metric and k = 15 neighbors.
Revised: We chose sklearn’s KNearestNeighbors classifier with cosine metric and k = 15 neighbors.

Original: The linear classifier was a simple linear layer that maps from 512 dimensions to the 10 classes and was trained via a cross-entropy loss.
Revised: The linear classifier was a simple linear layer that maps from 512 dimensions to the 10 classes and was trained via a cross-entropy loss.

Original: The linear classifier was trained for 100 epochs and an initial learning rate of 30 that was annealed with a cosine schedule to 0 . 
Revised: The linear classifier was trained for 100 epochs and an initial learning rate of 30 that was annealed with a cosine schedule to 0 . We trained the linear classifier [Table] [Table]

Original: We trained the linear classifier using data augmentations as in (Chen et al., 2020), that is, we randomly flipped as well as randomly resized the images.
Revised: using data augmentations as in (Chen et al. [Table] [Table] that is

Revised: [Table]

Revised: we randomly flipped as well as randomly resized the images.

Original: Our code is included in the supplementary material.
Revised: Our code is included in the supplementary material.

Original: K. S TABILITY
Revised: K. S TABILITY

Original: Whenever we reported a metric or show a graph, we ran the experiments for 3 different random seeds, and reported the mean ± the standard deviation.
Revised: Whenever we reported a metric or show a graph, we ran the experiments for 3 different random seeds, and reported the mean ± the standard deviation.

Original: When the standard deviation was very small, we omitted it from the main text and report it in Tab. S3.
Revised: When the standard deviation was very small, we omitted it from the main text and report it in Tab. S3.

Original: Save for the usually approximate s k NN graph computation, t -SNE does not depend on a random seed.
Revised: Save for the usually approximate s k NN graph computation, t -SNE does not depend on a random seed.

Original: As we computed the s k NN exactly with PyKeOps (Charlier et al., 2021), t -SNE is deterministic in our framework.
Revised: As we computed the s k NN exactly with PyKeOps (Charlier et al., 2021), t -SNE is deterministic in our framework.

Original: Panels i in Figs.
Revised: Panels i in Figs.

Original: 1, S10 – S15 show the standard deviation as shaded area.
Revised: 1, S10 – S15 show the standard deviation as shaded area.

Original: Again, the standard deviations are very small and barely visible.
Revised: Again, the standard deviations are very small and barely visible.

Original: The ratio of standard deviation to mean was never larger than 0 .
Revised: The ratio of standard deviation to mean was never larger than 0 .

Original: 006 in panels i of Figs.
Revised: 006 in panels i of Figs.

Original: 1, S10 – S15.
Revised: 1, S10 – S15.

Original: Similarly, the standard deviation in Figs
Revised: Similarly, the standard deviation in Figs.

Revised: S and S9, shown as shaded area, is mostly smaller than the line width.

Original: K. C OMPUTE
Revised: K. C OMPUTE

Original: We ran our neighbor embedding experiments on a machine with 56 Intel(R)
Revised: We ran our neighbor embedding experiments on a machine with 56 Intel(R)

Original: Xeon(R) Gold
Revised: Xeon(R) Gold

Original: CPU @ 2 .
Revised: CPU @ 2 .

Original: 60 GHz, 502 GB RAM and 10 NVIDIA TITAN
Revised: 60 GHz, 502 GB RAM and 10 NVIDIA TITAN

Original: Xp GPUs.
Revised: Xp GPUs.

Original: The SimCLR experiments were conducted on a Slurm cluster node with 8 cores of an Intel(R) Xeon(R) Gold 5220 CPU @ [Table]
Revised: The SimCLR experiments were conducted on a Slurm cluster node with 8 cores of an Intel(R) Xeon(R) Gold 5220 CPU @ 2 .

Original: 2 . 20 GHz and a Nvidia V 100 GPU with a RAM limit of 54 .
Revised: 20 GHz and a Nvidia V 100 GPU with a RAM limit of 54 .

Original: 3 GB.
Revised: 3 GB.

Original: Each experiment used one GPU at most.
Revised: Each experiment used one GPU at most.

Original: Our total compute is dominated by the neighbor embedding runs for Figs.
Revised: Our total compute is dominated by the neighbor embedding runs for Figs.

Original: 1, S10, S12 – S15 and S9b and by the SimCLR experiments.
Revised: 1, S10, S12 – S15 and S9b and by the SimCLR experiments.

Original: Tab. S2 lists the number of runs and the average run time.
Revised: Tab. S2 lists the number of runs and the average run time.

Original: We thus estimate the total compute time to be about 640 hours.
Revised: We thus estimate the total compute time to be about 640 hours.

Original: Our implementation uses pure PyTorch and relies on GPUs.
Revised: Our implementation uses pure PyTorch and relies on GPUs.

Original: But for the non-parametric experiments, much of the computation consists of sampling negative neighbors and computationally light updates to the embedding positions.
Revised: But for the non-parametric experiments, much of the computation consists of sampling negative neighbors and computationally light updates to the embedding positions.

Original: Therefore, the CPU-based numba or C++ reference implementations of
Revised: Therefore, the CPU-based numba or C++ reference implementations of

Original: UMAP and NCVis are much faster.
Revised: UMAP and NCVis are much faster.

Original: However, our implementation is arguably easier to inspect and adapt by the machine learning community and seamlessly integrates non-parametric and parametric settings as well as all four contrastive loss functions.
Revised: However, our implementation is arguably easier to inspect and adapt by the machine learning community and seamlessly integrates non-parametric and parametric settings as well as all four contrastive loss functions.

Original: It is thus more suited as a research tool rather than for large throughput application.
Revised: It is thus more suited as a research tool rather than for large throughput application.

Original: As the change from NCVis to Neg- t -SNE is as simple as fixing the learnable normalization parameter to a constant, we have also adapted the original NCVis code to compute Neg- t -SNE.
Revised: As the change from NCVis to Neg- t -SNE is as simple as fixing the learnable normalization parameter to a constant, we have also adapted the original NCVis code to compute Neg- t -SNE.

Original: We have not used this for any of the experiments of the paper, but include it as part of the supplementary as this implementation would retain the speed of the original NCVis implementation.
Revised: We have not used this for any of the experiments of the paper, but include it as part of the supplementary as this implementation would retain the speed of the original NCVis implementation.

Original: input : list of directed s k NN graph edges E
Revised: input : list of directed s k NN graph edges E

Original: = [ i 1 j 1 , . .
Revised: = [ i 1 j 1 , . .

Original: , i | E | j | E | ] parameters θ // embeddings (non-param.)/ network weights (param.)
Revised: , i | E | j | E | ] parameters θ // embeddings (non-param.)/ network weights (param.)

Original: number of epochs T learning rate η number of noise samples m
Revised: number of epochs T learning rate η number of noise samples m

Original: Cauchy kernel q // of embeddings (non-param.)/ network output (param.)
Revised: Cauchy kernel q // of embeddings (non-param.)/ network output (param.)

Original: batch size b loss mode mode normalization constant ¯ Z // default | E | /m , required for mode =
Revised: batch size b loss mode mode normalization constant ¯ Z // default | E | /m , required for mode =

Original: Neg- t -SNE output: final embeddings e 1 , . . .
Revised: Neg- t -SNE output: final embeddings e 1 , . . .

Original: , e n 1 if mode =
Revised: , e n 1 if mode =

Original: NCVis then
Revised: NCVis then

Original: Z =3 for t = 0 to T do //
Revised: Z =3 for t = 0 to T do //

Original: Learning rate annealing η t = η · (1 − tT )
Revised: Learning rate annealing η t = η · (1 − tT )

Original: α = while α < | E | do
Revised: α = while α < | E | do

Original: L = for β = 1 , . . .
Revised: L = for β = 1 , . . .

Original: , b do //
Revised: , b do //

Original: Sample noise edge tails but omit head of considered edge
Revised: Sample noise edge tails but omit head of considered edge

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Uniform
Revised: Uniform

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Aggregate loss based on mode if mode =
Revised: Aggregate loss based on mode if mode =

Original: Neg- t -SNE then
Revised: Neg- t -SNE then

Original: [Table] [Table]
Revised: [Table] [Table]

Original: else if mode
Revised: else if mode

Original: [Table]
Revised: [Table]

Original: then
Revised: then

Original: [Table]
Revised: [Table]

Original: − (cid:80) mµ
Revised: − (cid:80) mµ

Original: =1 log (cid:16) 1 − q θ
Revised: =1 log (cid:16) 1 − q θ

Original: ( i α + β j − µ ) /Z
Revised: ( i α + β j − µ ) /Z

Original: q θ ( i α + β j − µ ) /Z
Revised: q θ ( i α + β j − µ ) /Z

Original: + m (cid:17)
Revised: + m (cid:17)

Original: L = L − log (cid:18) q θ
Revised: L = L − log (cid:18) q θ

Original: ( i α + β j α + β ) q θ
Revised: ( i α + β j α + β ) q θ

Original: ( i α + β j α + β )+ (cid:80) mµ =1 q θ
Revised: ( i α + β j α + β )+ (cid:80) mµ =1 q θ

Original: ( i α + β j − µ ) (cid:19) else
Revised: ( i α + β j − µ ) (cid:19) else

Original: L = L − log (cid:16) q θ
Revised: L = L − log (cid:16) q θ

Original: ( i α + β j α + β ) (cid:17) − (cid:80) mµ
Revised: ( i α + β j α + β ) (cid:17) − (cid:80) mµ

Original: =1 log (cid:16) 1 − q θ
Revised: =1 log (cid:16) 1 − q θ

Original: ( i α + β j − µ ) (cid:17) // Update parameters with SGD( param.) or Adam (param.)
Revised: ( i α + β j − µ ) (cid:17) // Update parameters with SGD param.) or Adam (param.)

Original: θ =
Revised: θ =

Original: θ − η t · ∇ θ L
Revised: θ − η t · ∇ θ L

Original: Z = Z − η t ∇ Z L α = α + b
Revised: Z = Z − η t ∇ Z L α = α + b

Original: Shuffle E 23 return θ
Revised: Shuffle E 23 return θ

Original: L A DDITIONAL FIGURES
Revised: L A DDITIONAL FIGURES

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original:  for the first  converged to but did not exactly equal
Revised: Both which meant noise for the first The learnt converged to but did not exactly equal

Original: NCVis’ partition function (cid:80) ij q θ (  , but was of the  order 
Revised: NCVis’ partition function (cid:80) ij q θ ( ij ) , but was of the same order of magnitude.

Original: Again  was better 
Revised: Again the was better more

Original: Since we 
Revised: Since we the after the early

Original: , there were  and 
Revised: , there were function and non

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: The spectrum produces embeddings very similar to those of (f) t -SNE and (
Revised: The spectrum produces embeddings very similar to those of (f) t -SNE and (

Original: ) NCVis when ¯ Z equals the partition function of t -SNE or the learned normalization parameter of NCVis.
Revised: ) NCVis when ¯ Z equals the partition function of t -SNE or the learned normalization parameter of NCVis.

Original: The UMAP embedding in (h) closely resembles the Neg- t -SNE embedding at
Revised: The UMAP embedding in (h) closely resembles the Neg- t -SNE embedding at

Original: Z = | X
Revised: Z = | X

Original: (cid:0)
Revised: (cid:0)

Original: (cid:1)

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]