From: ICLR.cc/2022/Workshop/GTRL/Paper46/Authors
title: Response to the above comment
comment: Thank you for the comment and for taking an interest in our work. We will address the above-mentioned points in the order in which they were written.

It is fair to point out that the current version of this work only focuses on linear CKA and in future iterations we definitely will consider CKA with a nonlinear kernel as well. Indeed some of our ongoing follow up work suggests CKA with RBF kernels have some similar robustness issues. However we would like to point out that our focus on linear CKA over nonlinear kernel CKA, and even over other representation similarity measures such as the Procrustes method, is not “odd” nor trivial. We focus on linear CKA because it is becoming overwhelmingly used in the literature to make measurements about neural network representations similarity. Oftentimes it is the only method used to support claims. Therefore it is especially important to better understand its limitations, even more so than for other methods that are less (or even not at all) used in practice.

Secondly, in the context of ANN representation comparison, “similarity” is a very broad term and to our knowledge there isn’t a single definition that has been identified by the scientific community as being superior to the others. As such, we argue that having similar local structure, the same separability hyperplanes and margins or the same accuracy on a given task are all relevant ways in which two sets of representations can be similar but which are not necessarily captured by linear CKA. Indeed, each of the presented transformations provides a counter-example, i.e. a notion of similarity that is NOT captured by CKA since we can perturb the CKA values with these transformations while keeping an important notion of similarity unchanged. As for the comment about insensitivity to changes to the margin specifically, we believe this might be a misunderstanding. The perturbations considered do not only preserve the separability hyperplanes but also the margin, which are, as mentioned above, highly important characteristics in the context of machine learning and which linear CKA seems to disregard. 

Finally, we believe that it is impossible to say with certainty that the considered scenarios don’t happen in practice and we respectfully disagree with the claim that they are contrived.  For example, CKA’s sensitivity to outliers wasn’t noticed in practice until CKA was applied to high-capacity models. In fact, only the “block structure” in the representations of such models was found at first, it was only in subsequent work that the phenomenon was explained as being caused by a couple of dominant examples. Opposite to this, we view our work as being “upstream” or proactive. We are trying to better understand the transformations to which CKA is or isn’t sensitive and which notions of similarity are or aren’t captured by CKA so that in future work we can better understand CKA results and the limitations of the claims made based on these results. Furthermore, the notion of “normality” in the training of deep learning models is not well defined as tasks can be solved using a multitude of optimizers, regularizations, loss functions, etc. and each combination of these factors can potentially yield different representations. An extensive study of how all of these factors affect representations has not been completed and would be almost impossible to run. Thus, we see no reason to dismiss the representation perturbations considered here. From a theoretical standpoint, we note that all the transformations considered are justified and relevant in the context of deep learning. Perhaps in future iterations of this work we can dedicate more space to these justifications and go into more detail.

From: ICLR.cc/2022/Workshop/GTRL/Paper46/Authors
title: Response to our reviewers
comment: We thank the reviewers for their thoughtful comments and suggestions. Their constructive criticism has been taken into consideration and has helped us improve the current version of this work and will also guide us to improve future iterations.

From: ~Simon_Kornblith1
title: Response
comment: I fear becoming the curmudgeonly scientist who clings to their old ideas in the face of evidence against them, but I don't find the arguments against CKA presented here as convincing as the authors evidently do.

First, though the title and abstract imply otherwise, this paper is solely about linear CKA. The scenarios in Section 3.2 and 3.3 are all scenarios where linear CKA finds low similarity even though neighbor relationships are preserved. These are cases where nonlinear kernels such as the RBF kernel would probably work better because they are able to emphasize local structure over global structure, and other linear methods that rely on variance explained, such as the Procrustes methods proposed by Ding et al. (2021) and Williams et al. (2021), will probably behave similarly to linear CKA. In our 2019 paper, we used linear CKA because we didn't find that RBF CKA produced qualitative differences *with real neural networks*, but RBF CKA would likely make a difference to many of the synthetic scenarios investigated here.

This raises a second point: the scenarios are somewhat contrived and the rationale for asking for invariance to these specific transformations is not established in the paper. Section 3.2 suggests that the authors want a similarity index that is invariant to changes that preserve local structure as well as invertible linear transformations, but gradient descent training is sensitive to those things (though perhaps less sensitive?), and invariance to invertible linear transformations is problematic because of the curse of dimensionality as we noted in our 2019 paper. Section 3.3 suggests that a similarity index should be insensitive to changes in the margin as long as the same hyperplane can still separate points, but the margin is an important property that affects the accuracy of various forms of linear classifiers trained on the representation. The outlier problem described in Section 3.3 is a legitimate issue in [very high-capacity networks](https://arxiv.org/abs/2202.07184), where placing different weight on these outliers can yield different measurements of representational similarity, but it's also trivial to normalize activations (i.e., use a cosine kernel) to reduce the influence of outlier data points. The remaining scenarios don't seem to happen between networks trained from different random initializations or else linear CKA would not show similarity across these networks.

Finally, interpreting the observation that it is possible to enforce a specific structure on the CKA heatmap with only a small drop in accuracy requires an independent way of measuring whether the representations are actually similar or different. The conclusions drawn in Section 3.4 appear to rest on the assumption that models that get similar accuracy have similar representations, but I am not convinced that this assumption is valid. We've shown that [loss functions that produce smaller accuracy differences than are shown here can lead to very different representations](https://openreview.net/forum?id=8twKpG5s8Qh), as measured in several different ways, including by the accuracy obtained when the representations are transferred from the source task to other tasks. The experiments in Section 3.4 need further analysis to show that changes to the CKA heatmaps induced by $\mathcal{L}_\mathrm{map}$ are truly superficial.

I agree with this paper's sentiment that representational similarity methods shouldn't be blindly applied and I am also quite certain that linear CKA has shortcomings, but I don't think there's particularly strong evidence provided here to substantiate the claim that CKA is inadequate as a measure of similarity _between deep neural network representations_. Such a claim seems like it ought to be based on the existence of neural network representations arrived at through normal training (but perhaps with different hyperparameters) where it can be established by some means that representations are similar but CKA determines they are different, as [Maheswaranathan et al. (2019)](https://proceedings.neurips.cc/paper/2019/hash/5f5d472067f77b5c88f69f1bcfda1e08-Abstract.html) have done for dynamics of RNNs. I understand that this is a workshop paper, but I also find the absence of any evaluation of different representational similarity methods beyond linear CKA somewhat odd. If there is a full-length version of this paper in preparation, I hope it will be more constructive.

From: ICLR.cc/2022/Workshop/GTRL/Program_Chairs
title: Paper Decision
decision: Accept (Poster)
comment: We decided to accept this submission as a poster presentation; while one of the reviewers mentioned potential issues wrt. scope, we share no such reservations—the analysis of internal representations is perfectly on-topic. We encourage the authors to use the feedback to further improve their paper before the start of the workshop.