Paper:
Insights on representational similarity in neural networks with canonical correlation
Comparing different neural network representations and determining how representations evolve over time remain challenging open questions in our understanding of the function of neural networks. Comparing representations in neural networks is fundamentally difficult as the structure of representations varies greatly, even across groups of networks trained on identical tasks, and over the course of training. Here, we develop projection weighted CCA (Canonical Correlation Analysis) as a tool for understanding neural networks, building off of SVCCA, a recently proposed method [22]. We first improve the core method, showing how to differentiate between signal and noise, and then apply this technique to compare across a group of CNNs, demonstrating that networks which generalize converge to more similar representations than networks which memorize, that wider networks converge to more similar solutions than narrow networks, and that trained networks with identical topology but different learning rates converge to distinct clusters with diverse representations. We also investigate the representational dynamics of RNNs, across both training and sequential timesteps, finding that RNNs converge in a bottom-up pattern over the course of training and that the hidden state is highly variable over the course of a sequence, even when accounting for linear transforms. Together, these results provide new insights into the function of CNNs and RNNs, and demonstrate the utility of using CCA to understand representations.

Review:
Edit after rebuttal: I thank the authors for addressing my concerns and provide further details, and I updated my overall score to vote that this paper should be accepted.  Summary: =======  The authors present an analysis of representations learned by neural networks using Canonical Correlation Analysis, building and expanding on the SVCCA paper[1]. I think the paper offered some interesting insights, that are likely of interest to the wider community, thus I am generally in favor of accepting this submission. But I am unsure about the clarity of the paper. Should these be addressed in the rebuttal, I am happy to update my score.  The authors first introduce "projection weighted CCA", their work-horse for the analysis. In my opinion, this is the weakest point of the paper. The authors often refer to [1] as the paper they build on, whose authors used SVD on the representation vectors to get rid of noise directions, and then used CCA to align representations of different networks. The authors here omitted the pruning-noise-via-SVD step, and instead weight the projection-directions to reduce noise. It is unclear to me why this change in methodology was needed -- if noise-directions within each layer's activation are reduced via SVD, then maybe it would not be necessary to down-weight uninformative CCA directions. I suggest the author expand on how their method of analysis differs from [1], and why they needed these changes.  My 2nd issue with the paper is that it gives very little details on experiments. It is impossible to reproduce the experiments in this paper, since there are too many details omitted. The authors do not specify the arcitectures of the networks they used, which would be crucial to better put the results into context. Furthermore, the experiments that lead to figure 2 are only rudimentarily described. I strongly suggest the authors add more details to make experiments reproducible (in the appendix, if space is an issue). I would also encourage the authors to provide the source code of their experiments to improve reproducability (though of course this is not a must).   Minor comments: ============== - Section 3.1 speculates that "solutions found by memorizing networks were as diverse as those found across entirely different dataset labelings".  It would be interesting (and presumably fairly easy for the authors) to test this instead of just speculating about it.  - I'm not convinced about Figure 4b: On the one hand, larger networks are more similar (fig 4a), on the other hand, larger networks achieve better accuracy. Isn't figure 4b simply a result of these two facts? I.e., there is a latent variable "network size" that influences both performance and CCA distance. One would need to eliminate the dependance on this latent variable to see how good the connection between CCA distance & performance is. It would be more convincing if more networks of the same size (but different test accuracy) would be plotted, instead of varying the size.  - The authors often use terms which are formally introduced only much later. E.g. the term "CCA vectors" is defined at line 113, but is first used from line 87 onwards.  - Equation (**) should be offset from the main text so it's easier to see (the authors later refer back to this equation, and it's hard to find )  - In the 3rd paragraph of Section 3.1, the fact that "the same set of random labels was used for all networks" is repeated in the next sentence as "the randomization of CIFAR 10 labels was consistent for all networks". Why is this mentioned twice right after each other. Do these mean different things?     References: ==========  [1]  Raghu et al, "SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability", NIPS 2017