Paper: MAP IT to Visualize Representations (ICLR-2024)

Reviewer comment (Experiments):
The authors only show results on tiny datasets and do not seem to have a working scalable implementation that would allow embedding even a modest-sized full MNIST, let alone datasets with millions of points.

Author response:
**Scalable implementation**: In **Appendix C, page 25** implementation details are discussed. The current code is not very efficient, and I can of course only agree that it would have been better to have a large-scale implementation. Hopefully, there is room to have the fundamentally different ideas that MAP IT represents and the current demonstrations published even if the implementation can still be improved.

Reviewer comment (Evaluation):
There is a complete lack of quantitative evaluations.

Author response:
**Lack og quantitative evaluations**: I agree that this should have been included from the start, thanks. I included both results for knn recall (input space vs. target space) and for class-wise knn recall in the target space. You can find this on **Appendix C, page 22** in the revised manuscript.

Reviewer comment (Writing):
In section 3.1, Z_p and Z_q are never formally defined. While it is understandable that Z_q is the sum over all N^2 Cauchy kernels, it is unclear what Z_p is in t-SNE. Is it simply 2n?

Author response:
**Section 3.1, $Z_q$ and $Z_p$ not formally defined**: In order to motivate the theory in Section 3 on CS as a projective divergence, I take as a starting point $q_{ij} = \left[ 1+||\vec z_i-\vec z_j||^2 \right]^{-1} / Z_q$ where $Z_q = \sum_{n, m} \left[ 1+||\vec z_n-\vec z_m||^2 \right]^{-1}$ is an explicit normalization factor.  Furthermore, $p_{ij} = \exp\left(-\kappa_i ||\vec x_i-\vec x_j||^2 \right)/ Z_p$, with normalization factor $Z_p = \sum_{n, m} \exp(-\kappa_i ||\vec x_n-\vec x_m||^2)$. This is now made explicit on **page 2**.

Reviewer comment (Theory):
The authors emphasize using unnormalized affinities (and replacing KL with CS) and also using marginal probabilities instead of pairwise probabilities. It is unclear if these two aspects are related. Could one use unnormalized affinities and CS loss with pairwise affinities? Could one use marginal probabilities in the KL loss? Are these independent suggestions that MAP IT happens to implement together, or are these two aspects related and derived from each other?

Author response:
**Conceptual question 1**: Thanks for raising these conceptual questions. The use of a projective divergence is what makes direct normalization of probabilities unnecessary, either for a cost function only based on pairwise affinities or for a cost function based on marginal affinities. Actually, in Proposition 11 I do show that MAP IT as a special case reduces to the CS divergence used with only pairwise affinities. However, your question regarding potentially having marginal probabilities for the KL loss is intriguing. In that case, the direct normalization of probabilities would be needed since the KL is not projective. I added a paragraph in **Appendix C, page 26** where I indicate that in future work, Definition 4 may be extended to a *map KL divergence*.

Reviewer comment (Theory):
The authors emphasize using unnormalized affinities, but their CS loss function performs normalization within the loss function (Equation 5). This normalization leads to the \(N^2\) repulsion term similar to t-SNE (Equation 8). This suggests that the authors may have simply moved the normalization from the affinities to the loss function, resulting in little to no change compared to t-SNE.

Author response:
**Conceptual question 2**: Thanks for pointing out the N^2 term. As mentioned to reviewer vQ36, the use of the CS divergence to derive MAP IT does render explicit normalization of probabilities unnecessary. But as you point out, a quadratic complexity still arises in the (entropy) weighting  and in future work the impact of these weighting factors and how they relate to each other should be further studied. I toned down the claim about MAP IT being inherently scalable in Abstract, etc, for that reason. I did perform a preliminary analysis where I computed these quantities by only using nearest neighbors of points in the computation, where the nearest neighbors are defined with respect to the input space. This is explained on **page 25** and the result is shown in **Figure 14**. This relates to the discussion in Section 3.4 **MAP IT via local neighborhoods**.

Reviewer comment (Reproducibility):
At the beginning of section 4, it is stated that MAP IT uses perplexity 15, but in the caption of Figure 3 and later in the text (especially Appendix C), the authors mention \(k=10\) as if MAP IT uses kNN graph with \(k=10\). The relationship between perplexity 15 and \(k=10\) is unclear and confusing.

Author response:
**Perplexity**: The perplexity is related to the Gaussian bandwith $\kappa_i$ parameter for computing $p_{ij}$. It is not related per se to the use of $k=10$ nearest neighbors for the computation of the attractive and repulsive forces, as discussed in Figure 3 and in Appendix C. However, I included in **Appendix C, page 22** a more detailed discussion of the definition of perplexity and the $\kappa_i$.

Reviewer comment (Experiments):
The authors only have an implementation that allows embedding \(n=2000\) data points. While acceptable decades ago, in 2023 this is a major limitation. In Figure 8, the authors show that they can reduce runtime 100-fold and still obtain nearly the same result, but only on the \(n=2000\) subset of MNIST. It is unclear why the authors did not run this approximation on the full MNIST dataset.

Author response:
**Scalable implementation**: In **Appendix C, page 25** implementation details are discussed. The current code is not very efficient, and I can of course only agree that it would have been better to have a large-scale implementation. Hopefully, there is room to have the fundamentally different ideas that MAP IT represents and the current demonstrations published even if the implementation can still be improved.

Reviewer comment (Evaluation):
There is no quantitative evaluation. For example, for MNIST one could do kNN classification in the 2D space and compare MAP IT with t-SNE/UMAP, or compute various NN preservation metrics. This would be far more interesting on the full MNIST dataset, not just the \(n=2000\) subset.

Author response:
**Lack og quantitative evaluations**: I agree that this should have been included from the start, thanks. I included both results for knn recall (input space vs. target space) and for class-wise knn recall in the target space. You can find this on **Appendix C, page 22** in the revised manuscript.

Reviewer comment (Novelty):
On page 3, in "the role of normalization", see Damrich et al. 2023 (https://openreview.net/forum?id=B8a1FcY0vi) for parametric t-SNE. This paper should be cited in various places, including Appendix A.

Author response:
**MINOR issues** Thanks for these good suggestions! They have all been implemented/fixed. For instance, including a motivation in Section 3.3, discussing Damrich (2023), discussing t-SNE vs Laplacian eigenmaps in **Appendix A**.

Reviewer comment (Writing):
On page 3, "marginal probabilities" appear in Definition 2 but have not yet been properly defined.

Author response:
**Section 3.1, $Z_q$ and $Z_p$ not formally defined**: In order to motivate the theory in Section 3 on CS as a projective divergence, I take as a starting point $q_{ij} = \left[ 1+||\vec z_i-\vec z_j||^2 \right]^{-1} / Z_q$ where $Z_q = \sum_{n, m} \left[ 1+||\vec z_n-\vec z_m||^2 \right]^{-1}$ is an explicit normalization factor.  Furthermore, $p_{ij} = \exp\left(-\kappa_i ||\vec x_i-\vec x_j||^2 \right)/ Z_p$, with normalization factor $Z_p = \sum_{n, m} \exp(-\kappa_i ||\vec x_n-\vec x_m||^2)$. This is now made explicit on **page 2**.

Reviewer comment (Writing):
In section 3.2, second line, one of the \(p_j =\) should be \(q_j =\).

Author response:
**MINOR issues** Thanks for these good suggestions! They have all been implemented/fixed. For instance, including a motivation in Section 3.3, discussing Damrich (2023), discussing t-SNE vs Laplacian eigenmaps in **Appendix A**.

Reviewer comment (Novelty):
On page 14, the relationship between t-SNE and Laplacian Eigenmaps was discussed in https://jmlr.org/papers/v23/21-0055.html and https://epubs.siam.org/doi/10.1137/18M1216134, and should be cited here. The claim that this relationship has "not to have been discussed much in the literature" is not exactly correct.

Author response:
**MINOR issues** Thanks for these good suggestions! They have all been implemented/fixed. For instance, including a motivation in Section 3.3, discussing Damrich (2023), discussing t-SNE vs Laplacian eigenmaps in **Appendix A**.

Reviewer comment (Theory):
The method does not seem scalable, contrary to the authors' claim. The CS divergence includes a summation over all p_j and q_j, hence you have quadratic complexity when optimizing.

Author response:
**Quadratic complexity**: The use of the CS divergence to derive MAP IT does render explicit normalization of probabilities unnecessary. But as you point out, a quadratic complexity still arises in the (entropy) weighting of the attractive and repulsive forces and in future work the impact of these weighting factors and how they relate to each other should be further studied. I toned down the claim about MAP IT being inherently scalable in Abstract, etc, for that reason. I did perform a preliminary analysis where I computed these quantities by only using nearest neighbors of points in the computation, where the nearest neighbors are defined with respect to the input space. This is explained on **page 25** and the result is shown in **Figure 14**. This relates to the discussion in Section 3.4 **MAP IT via local neighborhoods**. This is however a question which merits further work and analysis.

Reviewer comment (Experiments):
There are no large-scale visualizations. While the authors claim that their approach could be scaled, I am sceptical of it precisely because of the definition of the CS divergence.

Author response:
**Quadratic complexity**: The use of the CS divergence to derive MAP IT does render explicit normalization of probabilities unnecessary. But as you point out, a quadratic complexity still arises in the (entropy) weighting of the attractive and repulsive forces and in future work the impact of these weighting factors and how they relate to each other should be further studied. I toned down the claim about MAP IT being inherently scalable in Abstract, etc, for that reason. I did perform a preliminary analysis where I computed these quantities by only using nearest neighbors of points in the computation, where the nearest neighbors are defined with respect to the input space. This is explained on **page 25** and the result is shown in **Figure 14**. This relates to the discussion in Section 3.4 **MAP IT via local neighborhoods**. This is however a question which merits further work and analysis.

Reviewer comment (Novelty):
Some citations could be added, e.g., on p. 3 (The role of normalization) you could cite NCVis (Artemenkov & Panov, 2021, https://arxiv.org/abs/2001.11411), an approach that estimates the normalization constant via NCE (Gutmann & Hyvarinen, 2012). The role of the normalization constant is also discussed in Böhm et al. (2022, JMLR) as well as Damrich et al. (2023, ICLR).

Author response:
**Suggested papers**: I have included the suggested papers both in **Section 2, page 3**, in **Appendix A, page 13** and other places too.

Reviewer comment (Experiments):
Talking about the scalability of the method is fine, but there is no large-scale visualization that demonstrates that the method scales beyond what other methods can easily visualize today.

Author response:
**Quadratic complexity**: The use of the CS divergence to derive MAP IT does render explicit normalization of probabilities unnecessary. But as you point out, a quadratic complexity still arises in the (entropy) weighting of the attractive and repulsive forces and in future work the impact of these weighting factors and how they relate to each other should be further studied. I toned down the claim about MAP IT being inherently scalable in Abstract, etc, for that reason. I did perform a preliminary analysis where I computed these quantities by only using nearest neighbors of points in the computation, where the nearest neighbors are defined with respect to the input space. This is explained on **page 25** and the result is shown in **Figure 14**. This relates to the discussion in Section 3.4 **MAP IT via local neighborhoods**. This is however a question which merits further work and analysis.

Reviewer comment (Evaluation):
There are no timings reported for any of the methods.

Author response:
**Timings**: On **page 25** it is explained that the current implementation is based on a matrix operation $(\boldsymbol{D}- \boldsymbol{M}) \boldsymbol{X}$  where $\boldsymbol{M}$ encodes attractive and repulsive forces according to and where $\boldsymbol{D}$ is the diagonal degree matrix of $\boldsymbol{M}$. The matrix $\boldsymbol{X}$ stores all data points as rows. I am working on a more efficient knn graph implementation, although the main idea and concept is conveyed. This is why timing is not explicitly reported.

Reviewer comment (Experiments):
Why did you not choose to use commonly used optimization parameters for t-SNE? In Belkina et al. 2019 they highlight what approaches work well and they considerably improve the visualization quality. This would also improve the comparison that you draw from Figure 1.

Author response:
**Optimization parameters for SNE, Belkina et al. (2019)**: Thanks for pointing me to this paper. I appreciate it a lot. Since the benchmark datasets studied in the paper are quite typical of the type of datasets that state-of-the-art implementations of t-SNE, UMAP, PacMap, etc, have been developed for an optimized for with their hyperparameters, I opted for those. I have however on **page 25** included a paragraph where I discuss Belkina et al. (2019). I agree that it may be that e.g. t-SNE could perhaps benefit from some more parameter tuning. At the same time, I tried to keep MAP IT very clean, not even using early exaggeration to not end up not optimizing a true divergence, and to stay close to the basic t-SNE type of approach. MAP IT could probably be improved by optimizing hyperparameters more.

Reviewer comment (Experiments):
Why did you choose the delta-bar-delta method for optimization?

Author response:
**Delta-bar-delta**: I chose the delta-bar-delta since it was common in the t-SNE literature. I observed slightly faster growing of structure in the MAP IT embedding when using this approach.

Reviewer comment (Writing):
Typo in the sixth paragraph: "normaliztion".

Author response:
**Typos**: I thank you for pointing out the typos. I have fixed these and I hope that I have avoided adding new ones as I have worked on the revision.

Reviewer comment (Writing):
Typo after equation 18: "neigborhood".

Author response:
**Typos**: I thank you for pointing out the typos. I have fixed these and I hope that I have avoided adding new ones as I have worked on the revision.

Reviewer comment (Writing):
Typo after figure 3: "neightbors".

Author response:
**Typos**: I thank you for pointing out the typos. I have fixed these and I hope that I have avoided adding new ones as I have worked on the revision.

Reviewer comment (Writing):
The equation $D(P||Q) = D(P˜||Q˜), \forall Z_p,Z_q\neq0$ could be better understood than the mention of "$Z_p$ and $Z_q$ being normalizing constants", which implies $Z_p$ and $Z_q$ need to be some specific values.

Author response:
**Normalization constants**: I realized when reading your comment, and also when reading comments by reviewer L3aD, that I should have been more precise. I have now added definitions for the normalizing constants $Z_q$ and $Z_p$ in relation to **Eq. (1), page 2**, in **Definition 1**, just before **Eq. (7)** as well as several places in **Appendix C**.

Reviewer comment (Evaluation):
It does not provide quantitative results. It's common in this area to report 1NN results over various datasets in the projected space. While this is a somewhat arbitrary metric, it seems to be the best there is so far, and I think it would be prudent to include it.

Author response:
**Quantitative results**: Thank you very much for proposing to add quantitative results. I agree that this should have been included from the start. I included both results for knn recall (input space vs. target space) and for class-wise knn recall in the target space. You can find this on **page 22** in the revised manuscript. The quantitative results for MNIST shed light on MAP IT showing that the method is probably somewhat better at capturing cluster structure compared t-SNE and UMAP, maybe at the expense of capturing more continuous manifold-like structures. This makes intuitive sense when looking at Figure 1.

Reviewer comment (Novelty):
The paper claims two conceptual novelties, the consideration of neighborhoods and the lack of normalization. Neither of these seem to me as new as the paper claims. Normalization is also not required in UMAP (as the paper mentions). Considering local neighborhoods is done explicitly in the less recent LLE, and implicitly in spectral embedding and laplacian eigenmaps. I was under the impression that t-SNE also has a similar "dual" interpretation in terms of kernel density, but I might be mistaken.

Author response:
**Normalization**: You are right that normalization is not required by e.g. UMAP and I tried to be a bit more specific to say that the MAP IT theory shows when normalization is not needed when a statistical divergence is the base for the dimensionality reduction. For instance, in Appendix C, concluding remark, it now reads *The role of normalization for divergences to be used as dimensionality reduction methods follows directly from the MAP IT theory.*  

**Local neighborhoods**: Preservation of local neighborhoods is also a key part of e.g Laplacian eigenmaps and t-SNE itself, but via pairwise similarities only. For instance, on **page 21** I included the sentence *this shows that MAP IT is able to capture information in wider local neighborhoods compared to CS t-SNE which only captures local information via pairwise affinities, a property it shares with its KL counterpart t-SNE and methods such as Laplacian eigenmaps \citep{Belkin2003}.*

Reviewer comment (Theory):
Can you explain why the kernel smoothing view doesn't apply to t-SNE?

Author response:
**t-SNE and kernel density**: Actually, in Bunte, Haase and Villmann (2012) "Stochastic neighbor embedding (SNE) for dimension reduction and visualization using arbitrary divergences" a form of t-SNE kernel density variant was discussed, but this it quite different from MAP IT.

However, I have been pondering your question "Can you explain why kernel smoothing doesn't apply to t-SNE?" I think that similar to the new quantity I propose, which I called the map CS divergence (Definition 4) a map KL divergence can also be possible. This would probably yield a related KL-based interpretation, but it would be in terms of the marginal probabilities and not the pairwise similarities which standard t-SNE optimizes. I added a paragraph about this on **page 26**, as a potential interesting avenue for future research.