Paper: Exploring Pointwise Similarity of Representations (ICLR-2024)

Reviewer comment (Experiments):
if the assumption is that the neighbours of a single point in two representation spaces matter in the construction of useful similarity scores, then I think an easy and effective approach would be Jaccard distance, and its variants that take distances into consideration. I wonder how the proposed approach compares to Jaccard distance.

Author response:
Our primary goal in this work is to highlight the importance of analyzing representation similarity at the granularity of individual data points. To achieve this goal, we designed *one* instantiation of a pointwise measure, PNKA, that can provide similarities for individual points across representations. However, we note here that other pointwise measures can also be employed.

In **Appendix C.4**, we empirically show that PNKA is correlated with the overlap of $k$ nearest neighbors, i.e., if the PNKA score of point $i$ is higher than that of another point $j$, then there is a *higher chance* that $i$’s nearest neighbors overlap more across the representation space of two models than those of $j$. We observed, however, that the correlation depends on the choice of $k$, and varies across architectures/datasets. Similar results, showing the relationship between PNKA and the Jaccard similarity coefficient, are presented in **Appendix C.5**, and are also shown to depend on the choice of $k$.

Given the observed variability in the results due to the choice of $k$, we ran an alternative analysis of unstable points using the Jaccard coefficient and appended the results in **Appendix D.1.1** of the updated version. We ran the experiment for four different $k$ values (k=250, 500, 1000 and 2000), different architectures (ResNet-18, VGG-16), and datasets (CIFAR-10, CIFAR-100).

From the results, we can infer that *the relationship between unstable points and the misclassification rate is highly affected by the choice of $k$ in the Jaccard coefficient*, and the *decision of which $k$ to choose from is not trivial*. Moreover, the optimal $k$ for one architecture and dataset does not generalize to other architectures and datasets. We also show in **Appendix C.2.1** that the choice of reference points, and number of landmarks, does not heavily influence the results for PNKA. Thus, Jaccard coefficient is not able to provide the same insights as PNKA.

Reviewer comment (Experiments):
it seems natural that models tend to disagree on misclassified data points, so if that is the case, we would then only need to look at the misclassified points as the unstable points rather than using the proposed similarity score to determine the unstable ones?

Author response:
While it's true that models tend to exhibit differences in their predictions for dissimilarly represented points, it's essential to note that the analysis of misclassification is a task-dependent analysis that requires the annotation of ground-truth labels. PNKA, on the other hand, as a representation similarity measure, offers a valuable approach to understanding model behavior without relying on ground-truth labels, by analyzing the internal representations themselves. Thus, by applying PNKA, one can already identify unstable points without a task in mind, and without access to any annotation. The stability of a point, as defined in the paper, is an inherent property of both the input data and the models under analysis, and is *not* task-dependent. For example, determining whether a point is misclassified relies on the exact classification task, e.g. 100-class classification or a coarser 20-super-class classification for CIFAR-100 dataset. In each case, the set of misclassified points may vary, mainly due to the higher level of granularity in the latter task. Thus, using misclassification rates to define unstable points could result in shifting sets of unstable points, even when the data points and models remain unchanged. We, however, argue that a point's instability remains consistent, irrespective of the specific application.

Reviewer comment (Evaluation):
since the comparison is now conditioned on a single data point along with its reference points, when two models are presented to us, how do we determine which model to use? The submission mentioned transfer learning as a use case, but it seems relatively non-trivial to me in terms of how we use the score in selecting the better pre-trained model to transfer from.

Author response:
In Section 5, we show that PNKA can be applied for model selection by illustrating how a model designer can compare different model representations to check which one may be more desirable for a specific downstream task. Specifically, we analyzed how interventions to a model modify the representations of individual points. We applied this analysis in the context of fairness and showed that some debiasing approaches for word embedding do not modify the targeted group of words as expected. *Thus, in Section 5, the application is in choosing the model that impacts the representations the way it is desirable for a specific downstream application.*

Beyond model selection, our aim in identifying unstable points is to emphasize the importance of subjecting these points to further analysis and caution. In Section 4, we showed that there exist points whose representations differ only due to stochasticity present in the models, and that these points are not only more likely to originate from out-of-distribution sources but are also prone to higher misclassification rates. *Thus, in Section 4, the application is in choosing which predictions to trust (i.e. the stable points), rather than choosing which models to use.*

We want to emphasize that such insights are not possible to achieve using global-level similarity measures.

Reviewer comment (Writing):
The method explanation lacks some clarity. Multiple readings were necessary due to references to the use of yet-to-be-introduced concepts such as "neighborhood" and "reference points". I think there's also some mismatch between the method presentation and the general take that the neighborhoods of each point are important to their representation since the reference points are never restricted or searched in the neighbors.

Author response:
## Clarity on PNKA explanation
We revised the exposition of PNKA to address the clarity issues in the updated version of the paper, in **Section 3**, where PNKA is defined. In summary, **reference points** are used in the computation of PNKA to estimate a point's relative position with respect to other points, within each representation. **Overlap of the neighbors** is another measure by itself, used solely as an intuition behind PNKA's scores. To resolve the ambiguity in the text, we start defining PNKA without referring to these concepts. In the paragraph "Formally defining PNKA", we introduce the idea of comparing the *relative* position of a point with (all) the other points across representations to communicate the formulation of PNKA. Then, in the paragraph "Computing PNKA with stable reference points", we introduce the idea of using a subset of reference points to compute a point's relative position, and we discuss the desirable properties of reference points when analyzing the instability of points. Finally, in the paragraph "Properties", we introduce the intuition behind PNKA by showing the correspondence between PNKA scores and the overlap of neighbors across representations. By making these changes in the text we hope to have made these concepts clearer.

Reviewer comment (Experiments):
Despite the valuable insights from the experiments, their current setting might not be general enough, limiting broader application and significance. For example, the relationship between PNKA and model disagreement on specific data points is limited to only a cross-training setting, without considering other possible variations such as architectural ones. This is something I would expect in a more theoretical work. The reported results are convincing, but, as commendably acknowledged by the authors themselves in the discussion section, the variation in architecture/tasks is not enough to validate the robustness of the proposed claims.

Author response:
## Expanding evaluation to other settings 
Our goal in Section 4 is to identify unstable points across two models. We decided to *focus on models that differ by initialization since the representations of these points change by just a stochastic factor*. We agree with the reviewer that this experiment can be extended to other training choices. In fact, in our submission, **Appendix F** includes results on models that differ by architecture, as suggested by the reviewer. In all cases, *we make a similar observation*: points with lower representation similarity are more prone to be misclassified and are more likely to be out-of-distribution (OOD).

Reviewer comment (Novelty):
There is a noticeable overlap in methodology with cited prior work, particularly Moschella et al. Although the paper frames its method as a kernel method application, it bears a strong resemblance to the direct application of cosine similarity between relative encodings, a technique already explored in the mentioned work. This overlap reduces the novelty of the method but doesn't impact its interesting applications.

Author response:
## Addressing differences in methodology from [1]
We make an initial attempt to clarify this distinction in **Section 1.1.** of the updated version of the paper, and discuss it here as well. Both the method proposed in [1] and ours build on top of the observation made by previous representation similarity measure (RSM) work, e.g. CKA, that models that differ only due to stochasticity tend to have similar representations, and that they position the representations of the same points similarly. However, [1] and our measure differ in how they leverage these observations:
 1. In our paper, we **focus on those inputs for which the assumption made in [1] does *not* hold** and argue why they are important: [1] makes an assumption, described in Section 3 of [1], that the angles between elements of the latent space are kept the same for *all* elements, which is exactly what we demonstrate in our first result (section 4) as not being the case. Throughout the paper, we expose the importance of studying low-similarity points (i.e., unstable points whose relative position changes).
 2. The **goal of the papers is substantially different**: In [1], the authors propose using relative representations as a *method* for accomplishing zero-shot model stitching. In our work, we show the importance of studying representations at a pointwise level to obtain a deeper and more fine-grained understanding of DNNs representations. PNKA is just one instantiation of a pointwise representation similarity *measure* that is related to the global CKA measure, however, other measures can also be applied.
 3. Finally, as also pointed out by the reviewer, **the contributions of both papers are different**: all the experimental setups and findings differ. In [1], the experiments aim to show that relative representations work for zero-shot model stitching. In our work, the experiments aim to show *how important it is to look at representations at the level of individual points*, and we highlight different and important use cases of PNKA. For instance, in Section 5, we specifically focus on comparing how representations of individual points are altered by debiasing approaches – this experiment is meaningless in the context of model stitching.

We hope we have suitably addressed all of the reviewer’s concerns and we would happily go into more details if there are any remaining questions.

References:

[1] Moschella, Luca, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà. “Relative Representations Enable Zero-Shot Latent Space Communication.” ICLR, 2023

Reviewer comment (Writing):
As described in the weaknesses section, I would ask the authors to please clarify the "neighborhood" concept and its relationship with the reference points.

Author response:
## Clarity on PNKA explanation
We revised the exposition of PNKA to address the clarity issues in the updated version of the paper, in **Section 3**, where PNKA is defined. In summary, **reference points** are used in the computation of PNKA to estimate a point's relative position with respect to other points, within each representation. **Overlap of the neighbors** is another measure by itself, used solely as an intuition behind PNKA's scores. To resolve the ambiguity in the text, we start defining PNKA without referring to these concepts. In the paragraph "Formally defining PNKA", we introduce the idea of comparing the *relative* position of a point with (all) the other points across representations to communicate the formulation of PNKA. Then, in the paragraph "Computing PNKA with stable reference points", we introduce the idea of using a subset of reference points to compute a point's relative position, and we discuss the desirable properties of reference points when analyzing the instability of points. Finally, in the paragraph "Properties", we introduce the intuition behind PNKA by showing the correspondence between PNKA scores and the overlap of neighbors across representations. By making these changes in the text we hope to have made these concepts clearer.

Reviewer comment (Experiments):
In the current manuscript form, I'm recommending a weak reject. However, I'm willing to increase the score if either the lack of variety in experiments or the relationship with previous work is addressed since the former would improve the experimental contributions while the latter the theoretical ones.

Author response:
## Expanding evaluation to other settings 
Our goal in Section 4 is to identify unstable points across two models. We decided to *focus on models that differ by initialization since the representations of these points change by just a stochastic factor*. We agree with the reviewer that this experiment can be extended to other training choices. In fact, in our submission, **Appendix F** includes results on models that differ by architecture, as suggested by the reviewer. In all cases, *we make a similar observation*: points with lower representation similarity are more prone to be misclassified and are more likely to be out-of-distribution (OOD).

## Addressing differences in methodology from [1]
We make an initial attempt to clarify this distinction in **Section 1.1.** of the updated version of the paper, and discuss it here as well. Both the method proposed in [1] and ours build on top of the observation made by previous representation similarity measure (RSM) work, e.g. CKA, that models that differ only due to stochasticity tend to have similar representations, and that they position the representations of the same points similarly. However, [1] and our measure differ in how they leverage these observations:
 1. In our paper, we **focus on those inputs for which the assumption made in [1] does *not* hold** and argue why they are important: [1] makes an assumption, described in Section 3 of [1], that the angles between elements of the latent space are kept the same for *all* elements, which is exactly what we demonstrate in our first result (section 4) as not being the case. Throughout the paper, we expose the importance of studying low-similarity points (i.e., unstable points whose relative position changes).
 2. The **goal of the papers is substantially different**: In [1], the authors propose using relative representations as a *method* for accomplishing zero-shot model stitching. In our work, we show the importance of studying representations at a pointwise level to obtain a deeper and more fine-grained understanding of DNNs representations. PNKA is just one instantiation of a pointwise representation similarity *measure* that is related to the global CKA measure, however, other measures can also be applied.
 3. Finally, as also pointed out by the reviewer, **the contributions of both papers are different**: all the experimental setups and findings differ. In [1], the experiments aim to show that relative representations work for zero-shot model stitching. In our work, the experiments aim to show *how important it is to look at representations at the level of individual points*, and we highlight different and important use cases of PNKA. For instance, in Section 5, we specifically focus on comparing how representations of individual points are altered by debiasing approaches – this experiment is meaningless in the context of model stitching.

We hope we have suitably addressed all of the reviewer’s concerns and we would happily go into more details if there are any remaining questions.

References:

[1] Moschella, Luca, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà. “Relative Representations Enable Zero-Shot Latent Space Communication.” ICLR, 2023