Original: N EURAL N ETWORKS AS P ATHS THROUGH THE S PACE OF R EPRESENTATIONS
Revised: N EURAL N ETWORKS AS P ATHS THROUGH THE S PACE OF R EPRESENTATIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep neural networks implement a sequence of layer-by-layer operations that are each relatively easy to understand, but the resulting overall computation is generally difficult to understand.
Revised: Deep neural networks implement a sequence of layer-by-layer operations that are each relatively easy to understand, but the resulting overall computation is generally difficult to understand.

Original: We consider a simple hypothesis for interpreting the layer-by-layer construction of useful representations: perhaps the role of each layer is to reformat information to reduce the “distance” to the desired outputs.
Revised: We consider a simple hypothesis for interpreting the layer-by-layer construction of useful representations: perhaps the role of each layer is to reformat information to reduce the “distance” to the desired outputs.

Original: With this framework, the layer-wise computation implemented by a deep neural network can be viewed as a path through a high-dimensional representation space.
Revised: With this framework, the layer-wise computation implemented by a deep neural network can be viewed as a path through a high-dimensional representation space.

Original: We formalize this intuitive idea of a “path” by leveraging recent advances in metric representational similarity.
Revised: We formalize this intuitive idea of a “path” by leveraging recent advances in metric representational similarity.

Original: We extend existing representational distance methods by computing geodesics, angles, and projections of representations, going beyond mere layer distances.
Revised: We extend existing representational distance methods by computing geodesics, angles, and projections of representations, going beyond mere layer distances.

Original: We then demonstrate these tools by visualizing and comparing the paths taken by ResNet and VGG architectures on CIFAR-10.
Revised: We then demonstrate these tools by visualizing and comparing the paths taken by ResNet and VGG architectures on CIFAR-10.

Original: We conclude by sketching additional ways that this kind of representational geometry can be used to understand and interpret network training, and to describe novel kinds of similarities between different models.
Revised: We conclude by sketching additional ways that this kind of representational geometry can be used to understand and interpret network training, and to describe novel kinds of similarities between different models.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: A core design principle of modern neural networks is that they process information serially, progressively transforming inputs until the information is in a format that is immediately usable for some task (Rumelhart et al., 1988; LeCun et al., 2015).
Revised: A core design principle of modern neural networks is that they process information serially, progressively transforming inputs until the information is in a format that is immediately usable for some task (Rumelhart et al., 1988; LeCun et al., 2015).

Original: This idea of composing sets of simple units to construct more complicated functions is central to both artificial neural networks and how neuroscientists conceptualize various functions in the brain (Kriegeskorte, 2015; Richards et al., 2019;
Revised: This idea of composing sets of simple units to construct more complicated functions is central to both artificial neural networks and how neuroscientists conceptualize various functions in the brain (Kriegeskorte, 2015; Richards et al., 2019;

Original: Barrett et al., 2019).
Revised: Barrett et al., 2019).

Original: Our work is motivated by a spatial analogy for information-processing: we imagine that outputs are “far” from inputs if the mapping between them is complex, or “close” if it is simple.
Revised: Our work is motivated by a spatial analogy for information-processing: we imagine that outputs are “far” from inputs if the mapping between them is complex, or “close” if it is simple.

Original: In this spatial analogy, any one layer of a neural network contributes a single step and the composition of many steps transports representations along a path towards the desired target representation.
Revised: In this spatial analogy, any one layer of a neural network contributes a single step and the composition of many steps transports representations along a path towards the desired target representation.

Original: Formalizing this intuition requires a method to quantify if any two representations are “close” (similar) or “far” (dissimilar) (Kriegeskorte, 2009; Kornblith et al., 2019).
Revised: Formalizing this intuition requires a method to quantify if any two representations are “close” (similar) or “far” (dissimilar) (Kriegeskorte, 2009; Kornblith et al., 2019).

Original: In order to use this kind of spatial or geometric analogy for neural representations, we need some way to quantify the “distance” between representations.
Revised: In order to use this kind of spatial or geometric analogy for neural representations, we need some way to quantify the “distance” between representations.

Original: We build on recent work introducing metrics for quantifying representational dissimilarity (Williams et al., 2021; Shahbazi et al., 2021).
Revised: We build on recent work introducing metrics for quantifying representational dissimilarity (Williams et al., 2021; Shahbazi et al., 2021).

Original: Representational dissimilarity is quantified using a function d( X , Y ) :
Revised: Representational dissimilarity is quantified using a function d( X , Y ) :

Original: X × X → R + , where X  is the space of all m × n matrices forfixed m and for all n, that takes as input two matrices of neural activity in response to m inputs, and returns a nonnegative measure of the dissimilarity of X and Y .
Revised: X × X → R + that takes in two matrices of neural data and outputs a nonnegative value for their dissimilarity. Here, X = (cid:83) n =1 , 2 , 3 ,... R m × n is the space of all m × n matrices for all n .

Original: The matrices X and Y could be, for instance, the values of two hidden layers in a network with n x and n y units, respectively, in response to m inputs.
Revised: The matrices X and Y could be, for instance, the values of two hidden layers in a network with n x and n y units, respectively, in response to m inputs.

Original: What are desirable properties of such a representational dissimilarity function?
Revised: What are desirable properties of such a representational dissimilarity function?

Original: Previous work has argued that any sensible dissimilarity function should be nonnegative, so d( X , Y ) ≥ 0 , and should return zero between any equivalent representations, so d( X , Y )
Revised: Previous work has argued that any sensible dissimilarity function should be nonnegative, so d( X , Y ) ≥ 0 , and should return zero between any equivalent representations, so d( X , Y )

Original: = 

Revised: =

Original: ⇔ X ∼ Y , where X ∼ Y means that X and Y are in the same equivalence class.
Revised: ⇔ X ∼ Y , where X ∼ Y means that X and Y are in the same equivalence class.

Original: For example, we may say that  X and Y are equivalent representations if Y is a shifted copy of X , or if it is a non-degenerate scaling, rotation, or affine transformation of X (Kornblith et al., 2019; Williams et al., 2021; Shahbazi et al., 2021).
Revised: For example, we may wish to design the function d so that d( X , Y ) = if Y is a shifted copy of X , or if it is a non-degenerate scaling, rotation, or affine transformation of X (Kornblith et al., 2019; Williams et al., 2021; Shahbazi et al., 2021).

Original: A second desirable property is that d is symmetric , so d( X , Y ) =
Revised: A second desirable property is that d is symmetric , so d( X , Y ) =

Original: d( Y , X ) .
Revised: d( Y , X ) .

Original: A third is that d satisfies the triangle inequality , or d( X , Y ) ≤ d( X , Z ) + d( Z , Y ) .
Revised: A third is that d satisfies the triangle inequality , or d( X , Y ) ≤ d( X , Z ) + d( Z , Y ) .

Original: As argued by Williams et al.
Revised: As argued by Williams et al.

Original: (2021), a representational dissimilarity function that fails to satisfy the triangle inequality can lead to errant results when, for instance, clustering or embedding representations based on their pairwise dissimilarity.
Revised: (2021), a representational dissimilarity function that fails to satisfy the triangle inequality can lead to errant results when, for instance, clustering or embedding representations based on their pairwise dissimilarity.

Original: A dissimilarity function that satisfies all of the above properties – equivalence, symmetry, and the triangle inequality – qualifies as a metric 1 on X (Burago et al., 2001).
Revised: A dissimilarity function that satisfies all of the above properties – equivalence, symmetry, and the triangle inequality – qualifies as a metric 1 on X (Burago et al., 2001).

Original: Examples of metrics between neural representations were recently developed independently by Williams et al.
Revised: Examples of metrics between neural representations were recently developed independently by Williams et al.

Original: (2021) and Shahbazi et al.
Revised: (2021) and Shahbazi et al.

Revised: We are interested in using metrics between neural representations to explore how representations evolve spatially as they are transformed through the hidden layers of deep networks.

Original: Not all metrics are sufficient for the kind of spatial reasoningabout representations we are interested in – that is, not all metrics can be interpreted as distances .
Revised: Not all metrics are sufficient for the kind of spatial reasoning – that is, not all metrics can be interpreted as distances .

Original: For example, consider the trivial metric
Revised: For example, consider the trivial metric

Original: [Equation]
Revised: [Equation]

Original: This is a valid metric according to the equivalence, symmetry, and triangle inequality criteria, but it is useless as a tool for characterizing distances.
Revised: This is a valid metric according to the equivalence, symmetry, and triangle inequality criteria, but it is useless as a tool for characterizing distances.

Original: To be interpretable as a measure of distance, d( X , Y ) must satisfy an intuitive fourth condition called rectifiability : the distance between any two points must be realizable as the (infimum of the) sum of distances of segments along a path between them (Burago et al., 2001).
Revised: To be interpretable as a measure of distance, d( X , Y ) must satisfy an intuitive fourth condition called rectifiability : the distance between any two points must be realizable as the (infimum of the) sum of distances of segments along a path between them (Burago et al., 2001).

Original: While not all metrics are rectifiable (such as the trivial metric above), it is perhaps unsurprising that this condition is met by many sensible metrics, including those already developed by Williams et al.
Revised: While not all metrics are rectifiable (such as the trivial metric above), it is perhaps unsurprising that this condition is met by many sensible metrics, including those already developed by Williams et al.

Original: (2021) and Shahbazi et al.
Revised: (2021) and Shahbazi et al.

Original: In fact, all metrics considered in this paper are Riemannian metrics , which not only implies that they are rectifiable, but further requires that points live on a smooth manifold (Burago et al., 2001).
Revised: In fact, all metrics considered in this paper are Riemannian metrics , which not only implies that they are rectifiable, but further requires that points live on a smooth manifold (Burago et al., 2001).

Original: Each restriction on the type of metric comes with additional structure that we can use to inspect and visualize neural representations: rectifiability allows us to smoothly interpolate neural representations along a geodesic as well as compute projections and angles, and Riemannian structure allows us to meaningfully compare the direction of steps taken by different layers.
Revised: Each restriction on the type of metric comes with additional structure that we can use to inspect and visualize neural representations: rectifiability allows us to smoothly interpolate neural representations along a geodesic as well as compute projections and angles, and Riemannian structure allows us to meaningfully compare the direction of steps taken by different layers.

Original: The main contributions of this paper are
Revised: The main contributions of this paper are

Original: • We put forward the spatial “path” analogy for deep neural networks, and quantitatively evaluate it using recently developed methods for representational distance.
Revised: • We put forward the spatial “path” analogy for deep neural networks, and quantitatively evaluate it using recently developed methods for representational distance.

Original: • We develop a toolbox for analyzing geometric properties of sequences of neural network layers, and show how existing representational distance measures imply a rich set of geometric concepts beyond mere pairwise distance.
Revised: • We develop a toolbox for analyzing geometric properties of sequences of neural network layers, and show how existing representational distance measures imply a rich set of geometric concepts beyond mere pairwise distance.

Revised: • We create novel visualizations of how representations are transformed through the layers of deep networks.

Original: • We apply these techniques to compare paths taken by wide and deep residual networks, as well as four VGG architectures, all trained on CIFAR-10, extending the results of  Nguyen et al.
Revised: • We apply these techniques to compare paths taken by wide and deep residual networks, as well as four VGG architectures, all trained on CIFAR-10, finding differences in both the magnitude and direction of steps taken by layers in wide versus deep models Nguyen et al.

Original:  P RELIMINARIES 2. R ELATED W ORK
Revised:  P RELIMINARIES 2. R ELATED W ORK

Original: One motivation for thinking of neural networks as paths is that it provides a compelling analogy for the way that complex functions (deep networks) can be composed out of simple parts (layers).
Revised: One motivation for thinking of neural networks as paths is that it provides a compelling analogy for the way that complex functions (deep networks) can be composed out of simple parts (layers).

Original: Indeed, it is well known that both deeper (Poole et al., 2016; Raghu et al., 2017; Rolnick and Tegmark, 2017) and wider (Hornik et al., 1989) neural network architectures can express a larger class of functions than their shallower or narrower counterparts.
Revised: Indeed, it is well known that both deeper (Poole et al., 2016; Raghu et al., 2017; Rolnick and Tegmark, 2017) and wider (Hornik et al., 1989) neural network architectures can express a larger class of functions than their shallower or narrower counterparts.

Original: However, little is known about how implementing a particular complex function constrains the role of individual layers and intermediate representations in the intervening layers between input and output.
Revised: However, much less is known about how implementing a particular complex function constrains the role of individual layers and intermediate representations in the intervening layers between input and output.

Revised: Our work is in line with other recent efforts to characterize the features learned in hidden layers as smoothly varying between inputs and outputs (Chan et al., 2020; Yang et al., 2022; He and Su, 2022).

Original: In our path framework, relatively narrow layers take shorter steps, and chaining them together increases the total achievable length, while wide layers are capable of taking a single large step, and all such steps contribute by moving “closer” to the targets.
Revised: In our path framework, relatively narrow layers take shorter steps, and chaining them together increases the total achievable length, while wide layers are capable of taking a single large step, and all such steps contribute by moving “closer” to the targets.

Original: Our work is a first step, so to speak, towards formalizing this notion of composing simple functions in geometric terms.
Revised: Our work is a first step, so to speak, towards formalizing this notion of composing simple functions in geometric terms.

Original: B Angular CKA Metric C Shape Metric D

Original: Affine

Original: Invariant

Original: Riemannian Metric

Original: [Figure]

Original: H

Original: I

Original: J

Original: [Figure]

Original: j

Original: [Figure]

Original: i

Original: [Figure]

Original: E

Original: F

Original: G

Original: [Figure]

Original: i

Original: [Figure]

Original: t an

Original: [Figure]

Original: i) (ii) (iii) (iv) (v) (vi)

Original: [Figure]

Original: Conv

Original: SoftMax BatchNorm

Original: ReLU

Original: Project

Original: Skip

Original: Pool

Original: Linear

Original: Targets [Figure] onto the input-label geodesic. There is a rich literature applying geometric concepts like distance between representations to formalize notions of “similarity” in neuroscience and psychology (Edelman, 1998; J¨akel et al., 2008;
Revised: There is a rich literature applying geometric concepts like distance between representations to formalize notions of “similarity” in neuroscience and psychology (Edelman, 1998; J¨akel et al., 2008;

Original: Rodriguez and Granger, 2017; H´enaff et al., 2019; Kriegeskorte and Wei, 2021).
Revised: Rodriguez and Granger, 2017; H´enaff et al., 2019; Kriegeskorte and Wei, 2021).

Original: However, there is a crucial difference between measuring similarity or distance between points in a given space, and measuring distances between representational spaces themselves .
Revised: However, there is a crucial difference between measuring similarity or distance between points in a given space, and measuring distances between representational spaces themselves .

Original: The former includes questions like, “how far apart are the activation vectors for two inputs in a given layer?”
Revised: The former includes questions like, “how far apart are the activation vectors for two inputs in a given layer?”

Original: The latter, which is the subject of this paper, asks instead, “how far apart are two layers’ representations, considering all inputs?”
Revised: The latter, which is the subject of this paper, asks instead, “how far apart are two layers’ representations, considering all inputs?”

Original: Our work is most closely related to, and draws much inspiration from recent advances in representational similarity analysis (RSA).
Revised: Our work is most closely related to, and draws much inspiration from recent advances in representational similarity analysis (RSA).

Original: In particular, Kornblith et al.
Revised: In particular, Kornblith et al.

Original: (2019) showed that a kernel method for testing statistical dependence, known as CKA (Gretton et al., 2005; Cortes et al., 2012), is closely related to classic RSA Kriegeskorte (2009).
Revised: (2019) showed that a kernel method for testing statistical dependence, known as CKA (Gretton et al., 2005; Cortes et al., 2012), is closely related to classic RSA Kriegeskorte (2009).

Original: In follow-up work, Nguyen et al.
Revised: In follow-up work, Nguyen et al.

Original: (2021) used CKA to make layer-by-layer comparisons between wide and deep networks – which we elaborate on in section 3.2 below.
Revised: (2021) used CKA to make layer-by-layer comparisons between wide and deep networks – which we elaborate on in section 3.2 below.

Original: Separately, both Williams et al.
Revised: Independently, both Williams et al.

Original: (2021) and Shahbazi et al.
Revised: (2021) and Shahbazi et al.

Original: (2021) developed methods to computeRiemannian metrics between neural representations.
Revised: (2021) developed methods to compute metrics between neural representations.

Revised: Shahbazi et al.

Revised: (2021) proposed using the so-called Affine-Invariant Riemannian Metric on the space of symmetric positive-definite matrices (Pennec, 2006; 2019).

Revised: Williams et al.

Revised: (2021) derived a metric variation of CKA which we call Angular CKA , as well as a family of Generalized Shape Metrics .

Original: Weunify and extend their work by computing not just pairwise distance functions, but by also introducing a suite of tools for analyzing the geometry of representation-space induced by each of these distance functions.
Revised: We extend this prior work by computing not just pairwise distances, but by also introducing a suite of tools for analyzing the geometry of representation-space induced by each of these distance functions.

Original: Finally, unlike
Revised: Finally, whereas

Original: Williams et al.
Revised: Williams et al.

Original: (2021) and Shahbazi et al.
Revised: (2021) and Shahbazi et al.

Original: (2021), we compare representations within a single model to study the transformation of information from inputs to outputs through hidden layers.
Revised: (2021) compare representations across models , we compare representations within a single model to study the transformation of information from inputs to outputs through hidden layers.

Original: 2. D ISTANCE METRICS BETWEEN NEURAL REPRESENTATIONS
Revised: 2. D ISTANCE METRICS BETWEEN NEURAL REPRESENTATIONS

Original: In our notation, a deep neural network processes an input x 0 through a sequence of hidden layers, { x 1 , . . .
Revised: In our notation, a deep neural network processes an input x 0 through a sequence of hidden layers, { x 1 , . . .

Original: , x L − 1 } , and produces outputs x L , trained to match some target outputs (LeCun et al., 2015).
Revised: , x L − 1 } , and produces outputs x L , trained to match some target outputs (LeCun et al., 2015).

Original: In the common example of image classification, x 0 is the input image, targets are the class label, and x L a set of (log) class probabilities.
Revised: In the common example of image classification, x 0 is the input image, targets are the class label, and x L a set of (log) class probabilities.

Original: We will use subscripts to index inputs, so x li is the n l -dimensional response of the l th hidden layer to the i th input, x 0 i , with i ∈ { 1 , . . .
Revised: We will use subscripts to index inputs, so x li is the n l -dimensional response of the l th hidden layer to the i th input, x 0 i , with i ∈ { 1 , . . .

Original: , m } .
Revised: , m } .

Original: We use capital letters like X l , X k , or Y to refer to m × n matrices of neural representations in response to m inputs 2 , and we treat target labels as one-hot vectors (i.e. if Y is the target representation, then it is a m × 10 matrix for the CIFAR-10 dataset (Krizhevsky, 2009)).
Revised: We use capital letters like X l , X k , or Y to refer to m × n matrices of neural representations in response to m inputs 2 , and we treat target labels as one-hot vectors (i.e. if Y is the target representation, then it is a m × 10 matrix for the CIFAR-10 dataset (Krizhevsky, 2009)).

Original: We will adopt the convention that l < k if x l is a direct ancestor of x k through the layers of a network (layers will in general be partially but not strictly ordered).
Revised: We will adopt the convention that l < k if x l is a direct ancestor of x k through the layers of a network (layers will in general be partially but not strictly ordered).

Original: We are interested in functions d
Revised: We are interested in functions d

Original: ( X , Y ) for that quantify the “dissimilarity” between two neural representations; when d( X , Y ) satisfies the four conditions of a length metric – equivalence, symmetry, triangle inequality, and rectifiability – we will say that it measures their “representational distance.”
Revised: ( X , Y ) for that quantify the “dissimilarity” between two neural representations; when d( X , Y ) satisfies the four conditions of a length metric – equivalence, symmetry, triangle inequality, and rectifiability – we will say that it measures their “representational distance.”

Original: Note that X and Y may be different layers of the same model or layers from different models evaluated on the same inputs.
Revised: Note that X and Y may be different layers of the same model or layers from different models, as long as they are evaluated on the same inputs.

Original: We use d( X , Y ) to refer to the distance function between arbitrary representations X and Y , and d( X l , X k ) when the layer indices l and k are contextually important.
Revised: We use d( X , Y ) to refer to the distance function between arbitrary representations X and Y .

Original: Comparing representations is challenging because we often want to compare representations of different sizes, i.e. where n x ̸ = n y .

Original: Using X to denote the set of all m × n matrices (for all n ), all representational distance metrics we consider here solve this problem using a two-stage approach: first, X and Y are mapped to a common space M through an embedding function f : X → M , then distance is computed using a distance metric defined on M .
Revised: We can unify and organize existing representational distance metrics by recognizing that distance metrics use a two-stage approach to defining d( X , Y ) : first, X and Y are mapped to a common space M through an embedding function f : X → M , then distance is computed using a distance metric defined on M .

Original: More precisely,
Revised: More precisely,

Original: [Equation]
Revised: [Equation]

Original: where ˜ X = f( X ) is the result of mapping X from X to M .
Revised: where ˜ X = f( X ) is the result of mapping X from X to M .

Original: In all cases we consider here, M is a
Revised: In all cases we consider here, M is a

Original: Riemannian manifold with metric d M .
Revised: Riemannian manifold with metric d M .

Original: Equivalence relations on X can be built into this two-stage approach in either stage. For instance, d( X , Y ) can be made invariant to changes in the scale of X either by imposing a canonical scale in f , or by preserving scale in f but using a scale-invariant metric d M .
Revised: Equivalence relations on X can be built into this two-stage approach in either stage: d( X , Y ) can be made invariant to changes in the scale of X either by imposing a canonical scale in f , or by preserving scale in f but using a scale-invariant metric d M .

Original: We implemented three existing families of representational distance, each of which can be understood as different choices for f , M , and d M , and extended them to further compute various geometric quantities including geodesics, projections, tangent vectors, and angles.
Revised: We implemented three existing families of representational distance, each of which can be understood as different choices for f , M , and d M , and extended them to further compute various geometric quantities including geodesics, projections, tangent vectors, and angles.

Original: The representational distance metrics we investigate here are Angular CKA (with a linear kernel), Shape Metrics (with angular distance), and the Affine Invariant Riemannian Metric (with a squared exponential kernel and ridge regularization)
Revised: The representational distance metrics we investigate here are Angular CKA (with a linear kernel), Shape Metrics (with angular distance), and the Affine Invariant Riemannian Metric (with a squared exponential kernel and ridge regularization)

Original: (Williams et al., 2021; Shahbazi et al., 2021).
Revised: (Williams et al., 2021; Shahbazi et al., 2021).

Revised: We have found that results using Angular CKA are the most interpretable among the metrics we have tested, so our main results are presented using Angular CKA.

Original:  Table A.1 and Appendix A provide mathematical details on each. To get an initial impression of each metric’s behavior, Figure 1B-D shows a 2D visualization of the path taken by a Resnet-14 (architecture in Figure 1A).
Revised: Results using other metrics are shown in Figure C.1. In the following section, we will give mathematical details for Angular CKA; Table A.1 and Appendix A provide mathematical details on all metrics in one place, including a discussion of their invariances. 2. T HE GEOMETRY OF A NGULAR CKA

Original: Following the suggestion of Williams et al.
Revised: Angular CKA was introduced by Williams et al.

Original: (2021), we generated these low-dimensional path visualizations by first computing pairwise distances between all layers, as well as15 equally-spaced points along the input-target geodesic.
Revised: (eq (60) in their supplement). It is defined as the arccosine of CKA, which is itself derived from the Hilbert-Schmidt independence criterion (HSIC) (Gretton et al., 2005; Cortes et al., 2012; Kornblith et al., 2019).

Original: We then used multidimensional scaling to find approximate Euclidean coordinates for each layer and each point on the geodesic.
Revised: Because HSIC and CKA measure statistical dependence, distance measured by Angular CKA is small when the rows of X and Y are strongly statistically dependent, and large when they are independent.

Original: Figure 1B-D shows the top two principal components of these Euclidean spaces, and provides a quick visual impression of the main features of the path taken by the model according to each metric.
Revised: While Angular

Original: We emphasize that we use MDS only for these initial visualizations, and that all subsequent calculations are done using the native geometry of each metric.
Revised: CKA was originally introduced simply as a method for computing a metric between neural representations, here we exploit the fact that Angular CKA is the arc-length on a Hypersphere to compute additional geometric properties of the space.

Original: For Angular CKAin Figure 1B, the metric space M is  the space ofm × m centered and normalized  Gram matrices).
Revised: Angular CKA is equivalent to arc-length distance on the spherical manifold consisting of centered and normalized m × m Gram matrices.

Original: For the Shape Metric in Figure 1C, M is  the space of m × p matrices after selecting only the top p principal components of X .
Revised: Let G X denote the Gram matrix of X , i.e. G X ij is given by the inner-product between the i th and j th rows of X .

Original: For the Affine Invariant Riemannian metric in Figure 1D, M is the space of m × m Gram matrices, which we further regularize to ensure positive semi-definiteness as required by the metric (Pennec, 2006; 2019).
Revised: Optionally, this inner-product may be computed using a kernel; following previous work (Kornblith et al., 2019; Nguyen et al., 2021), results in this paper use Linear CKA, i.e. we use G X = XX ⊤ . Other kernels are in principle admissible and available in our python package at [link redacted].

Original: 2. F ROM DISTANCE METRICS TO GEOMETRY
Revised: The normalized and centered Gram matrix is given by

Original: Our main methodological contribution isrecognizing that any choice of representational distance metric implies rich geometric structure on the space of representations that goes beyond pairwise distances; representations are not merely separated by some distance, but also by some direction .
Revised: [Equation] where H = I m − 1 m 11 ⊤ is the m × m centering matrix , and ||·|| F is the Frobenius norm of a matrix. The Riemannian manifold M for Angular CKA consists of all centered and normalized symmetric positive definite matrices; it is a sphere because (cid:68) ˜ G X , ˜ G X (cid:69) F = 1 , where ⟨ A , B ⟩ F =

Original: We are releasing a standalone PyTorch (Paszke et al., 2019) library to compute all of these metrics and their associated geometric quantities, available at [redacted link during review process].
Revised: Tr ( A ⊤ B ) is the Frobenius inner-product.

Original: Interpolating along geodesics. The rectifiability property of distance metrics implies that we can instantiate points in M that interpolate ˜ X and ˜ Y along a shortest path between them.
Revised: Comparing to equation (1), we can see that the embedding function for Angular CKA is f( X ) = ˜ G X . Distance according to Angular CKA is

Original: That is, for any t ∈ [0 , 1] , we can construct a point ˜
Revised: [Equation]

Original: Z = geodesic ( ˜ X , ˜ Y , t ) such that d M ( ˜ X , ˜ Z ) + d M (˜ Z , ˜ Y ) = d M ( ˜  X , ˜  Y ) and d M ( ˜ X , ˜ Z )
Revised: (Figure 1B-D). Because Angular CKA is an arc length on a hypersphere, we can easily compute its geodesics : the shortest path connecting ˜ G X to ˜ G Y is

Original:  =
Revised: [Equation] where t ∈ [0 , 1] is the fraction of distance along the geodesic from X to Y , and Ω =

Original: t d M ( ˜  X , ˜  Y ) .
Revised: d M ( ˜ G X , ˜ G Y ) .

Original: (Shortest paths are unique in all cases we consider here, but in general there may be more than on ˜ Z that satisfies these expressions). This grants us the ability to interpret neural network layers not as discrete jumps from one representation to another, but as a piece-wise smooth curve from inputs to network outputs. In Figure 1C, we used this property to instantiate points along the shortest path connecting input images directly to output labels, and to compare this with the path taken by the network. Projection.
Revised: This is a direct application of the SLERP formula 3 for hyperspheres, applied to ˜ G -space.

Original: Points in Mcan be projected onto the geodesic spanningany other two  points.
Revised: For all metrics, we use numerical optimization methods to project a point in M onto the geodesic spanning two other points.

Original: This follows from the existence of geodesics: the projection of ˜ X onto the geodesic spanning ˜ Y and ˜ Z can be found by minimizing the distance from ˜ X to geodesic ( ˜ Y , ˜ Z , t ) with respect to t . 
Revised: The projection of ˜ X onto the geodesic spanning ˜ Y and ˜ Z can be found by minimizing the distance from ˜ X to geodesic ( ˜ Y , ˜ Z , t ) with respect to t . In practice, we solve this as an optimization problem with respect to t .

Original: In Figure 1H-J, we use this property to project each  layerof a Resnet-14 onto the geodesic connecting inputs  to targets.
Revised: Figure 1G demonstrates one way that this idea of projecting neural representations can be used: we project each hidden layer onto the geodesic connecting inputs (raw pixels) to targets (one-hot labels).

Revised: (Note that we keep only the representations labeled (i) through (vii) in Figure 1A so that the resulting path consists only of steps that perform comparable operations).

Original: Compared with measuring raw distance from inputs to hidden layers  or from hidden layers to outputs (Figure 1E-J), this measure of projected distance from inputs provides a useful 1-dimensional summary of the network’s progress through the layers.
Revised: This provides a more gradual picture of the “progress” being made by each layer towards the targets than mere distance between layers and inputs or targets (Figure 1H). For all metrics, we can compute angles between any three neural representations using the tangent space defined by the metric. The logarithmic map is a function that takes in two points in M and returns a tangent vector that “points” from one to the other (Burago et al., 2001). The logarithmic map for Angular CKA from ˜ G X to ˜ G Y is

Original: Angles.
Revised: [Equation]

Original: Any triplet of points in M defines an angle.
Revised: where W is the unit tangent vector at ˜ G X pointing towards ˜ G Y , given by [Equation]

Original: We use θ ( ˜ X , ˜ Y , ˜ Z ) to denote the angle formed by the ( ˜  X , ˜ Y ) and (˜ Z , ˜  Y )geodesics incident at ˜ Y . 
Revised: We then compute angles between triplets of points by computing the inner-product of their tangent vectors. In the case of Angular CKA in particular, let W XY denote the tangent vector pointing from X to Y , i.e. the result of log ˜ G X ( ˜ G Y ) . Then,

Original: We use this to quantify theinternal angle along the network’s path at layer l , θ ( ˜ X j , ˜ X k , ˜ X l ) – this measures the extent towhich the steps from layer j to k and from k to l lie along a straight  path according to a given metric.
Revised: [Equation] is the angle of the XYZ triangle at Y . Equations (3)–(5) are simply applications of the geometry of a hypersphere to the space of m × m centered and normalized Gram matrices. Figure 1I demonstrates two types of angles that are of interest: “internal angles,” which quantify how straight a path is, and “target angles,” which quantify to what extent each step points in the direction of the targets.

Original: A path through neural network layers may in principle be highly curved locally, but relatively straight when zooming out and considering a block consisting of multiple layers as a single step.
Revised: Interestingly, we find that nearly all internal angles are orthogonal, and only later layers begin to take steps in the direction of the targets (Figure 1J). 2. C OMPARISONS WITH OTHER METRICS The main results in this paper are presented using Angular CKA for ease of exposition.

Original: For this reason, we treat each residual block as a single path step.
Revised: Mathematical details for other metrics can be found in Table A.1 and Appendix A. Figure C.1 gives results like those in Figure 1 using the metrics proposed by Williams et al. (2021) (Angular Shape and Euclidean Shape) and Shahbazi et al. (Affine-Invariant Riemannian).

Original:  E XPERIMENTS 3. H OW PATHS CHANGE OVER TRAINING
Revised:  E XPERIMENTS 3. H OW PATHS CHANGE OVER TRAINING

Original: Whereas Figure 1 shows properties of the path taken by a well-trained Resnet-14 model, here we ask how properties of its path emerge over training.
Revised: Whereas Figure 1 visualizes properties of the path taken by a well-trained Resnet-14 model, here we ask how properties of its path emerge over training.

Original: For this, we trained a new model of the same architecture, and logged model parameters every 10 batches.
Revised: For this, we trained a new model of the same architecture, and logged model parameters every 10 batches.

Original: Since the vast majority of changes occur in the first epoch, Figure 2 shows how distances, projected distances, andinternal angles of the model’s As in Figure 1, the results for Angular CKA are the clearest and easiest to interpret.
Revised: Since the vast majority of changes occur in the first epoch, Figure 2 shows how distances, projected distances, and angles change for every 10 steps in the first epoch, plus a single additional point after 20 epochs to show where values converge after longer training.

Original: As expected, the network at initialization takes a path somewhere , but this path is originally not in the direction of the targets: the distance from inputs initially increases through the layers, while the distance to targets does not initially decrease (Figure 2, first row).
Revised: At initialization, the distance from inputs initially increases (Figure 2B), but the distance to targets is constant (Figure 2A). This implies that the network’s representations are changing in some direction that is orthogonal to making “progress” towards the targets.

Original: In comparison, the projected progress paints a clearer picture of the gradual process of moving closer to the targets as trainingprogresses (Figure 2, middle row): at initialization the path makes no progress, but by the time the first epoch completes (step 400), the net progress covers most of the distance to the targets. Results for the other metric are similar, but difficult to see because their distance is dominated by the last layers of the network.
Revised: Indeed, we find that plotting projected progress is a useful way to visualize training (Figure 2C). At initialization, projected progress is zero for all layers, and it quickly begins to ramp upwards after a few training steps.

Original: We hypothesized that training would cause the network’s path to become more straight , but surprisingly we found  the opposite to be true.
Revised: We hypothesized that training would cause the network’s path to become more straight , but surprisingly we found that this is not the case.

Original: In the bottom row of Figure 2, we show the internal angle between steps along the path (one step is one residual block). According to the Angular CKA metric, internal angles begin slightly above π/ 2 at initialization, and concentrate towards π/ 2 as training progresses.
Revised: In fact, internal angles begin slightly above π/ 2 at initialization, and concentrate towards π/ 2 as training progresses (Figure 2D). This means that the function implemented by each residual block applies an orthogonal operation (in representation space) compared to the blocks before or after it.

Original: This means that steps become more orthogonal through training. Results for the other metrics agree that steps are largely orthogonal, though they show no clear pattern over training.
Revised: To our knowledge, this kind of orthogonality between layers has not been previously observed. Figure 2E shows how “target angles” evolve over training. Consistent with past work, this visualization shows that the emergence of layers that point in the direction of the targets does not happen until later in training.

Original: 3. C OMPARING PATHS OF WIDE VERSUS DEEP MODELS
Revised: 3. C OMPARING PATHS OF WIDE VERSUS DEEP MODELS

Original: We replicate and extend the results of Nguyen et al.
Revised: Inspired by the results of Nguyen et al.

Original: (2021) by comparing paths taken by “wide” versus “deep” models.
Revised: we compare paths taken by “wide” versus “deep” models.

Original: In Figure3, we show the same quantities we computed in Figure2, but comparing different Resnet widths and depths.
Revised: In Figure we show the same quantities we computed in Figure but comparing different Resnet widths and depths.

Original: Our results confirm the main findings of Nguyen et al.
Revised: Our results confirm the main findings of Nguyen et al.

Original: (2021), namely that wide and deep models compute similar functions — in other words, widemodel paths and deep-model paths all follow roughly the same trajectory in representational space.
Revised: (2021), namely that wide and deep Resnets compute similar functions — in other words, wide-model paths and deep-model paths all follow roughly the same trajectory in representational space.

Revised: Note that the x-axis for these plots is scaled by the depth of each model; the fact that all Resnets are superimposed on each other means that all of them distribute the function of each layer relatively evenly over their depth.

Original: This is not universally true of all models – we also compare with VGG, and see salient differences between the VGG and Resnet architectures.
Revised: This is not universally true of all models – we also compare with VGG, and see salient differences between the VGG and Resnet architectures, where the progress made in VGG (without skip connections) is much more dependent on absolute rather than relative depth.

Original: To investigate how longer or shorter paths differ intheir movement towards the target, we further decompose each layer’s progress into the two orthogonal components that result when projecting each path segment onto the geodesic (Figure 4A).
Revised: We next asked to what extent each step in the path, i.e. each residual block, moves representations towards the targets ( progress ) or in orthogonal directions ( deviation ; Figure 4A).

Original: The components are “progress”: the distance travelled by the segment along the geodesic from the current position to the targets; and “deviation”: the distance by which  the segment moves away from this geodesic. In this the geodesic the shortest position  to the targets, and differs
Revised: Note that unlike “projected progress” above, which was measured relative to the input-targets geodesic, here we compare each step from X l to X l +1 to the geodesic connecting current position X l to the targets. A model with D blocks provides D-1 measures of both progress and deviation – one for each block.

Original: Figure 4B plots the mean and standard deviation ofthe progress and deviation made by each path segment of a variety of residual and VGG network architectures.
Revised: Figure 4B summarizes the net contribution of progress and deviation for all layers in models with varying architectures.

Original: We observe that deep networks have significantly less deviation and slightly less progress per step compared with shallow networks. Among residual networks of equal depth, wide networks have less deviation per segment than narrow networks. This may be due to the regularizing effect of increased network width. Although VGG networks follow similar trends, they form their own distinct cluster on the top right of the figure (yellow points).
Revised: Note the following salient effects: first, there is a strong overall upward trend – for every block that makes 0.1 radians of progress towards the targets (recall that Angular CKA is an arc-length), it incurs some deviation of about 0.3 radians in an orthogonal direction, and this trend is remarkably uniform for both Resnet and VGG architectures.

Original: VGG networks have overall higher ratio of progress to deviation than residual networks, which might be introduced by the shortcut connection and difference in the block design. In short, we find that in both convolutional based networks, even only in residual networks, different depth and  width brings networks a different progress and deviation in their representation path.
Revised: Second, deeper models are lower and to the left; deeper models parcel out both progress and deviation among their layers. Third, we can identify an effect of width at constant depth; at a constant depth, wider models incur less deviation than their narrower counterparts.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: In this work, we interpret neural networks spatially as paths from inputs to outputs through a space of intermediate representations.
Revised: In this work, we interpret neural networks spatially as paths from inputs to outputs through a space of intermediate representations.

Original: These paths have rich geometric structure, inherited from computing distances between representations, and we can use that structure to build intuitions about network structure, training, and comparisons between models.
Revised: These paths have rich geometric structure, inherited from computing distances between representations, and we can use that structure to build intuitions about network structure, training, and comparisons between models.

Original: We investigated three families of representational distance metrics — Angular CKA, Shape Metrics, and the Affine Invariant metric on symmetric positive definite matrices (Williams et al., 2021;
Revised: We investigated three families of representational distance metrics — Angular CKA, Shape Metrics, and the Affine Invariant metric on symmetric positive definite matrices (Williams et al., 2021;

Original: Shahbazi et al., 2021).
Revised: Shahbazi et al., 2021).

Original: Surprisingly, we found that trained networks take circuitous paths according to all of these metrics, and deviate far from the shortest paths from inputs to targets which are defined geometrically by each representational distance metric.
Revised: Surprisingly, we found that trained networks take circuitous paths according to all of these metrics, and deviate far from the shortest paths from inputs to targets which are defined geometrically by each representational distance metric.

Original: There are three potential explanations for this.
Revised: There are three potential explanations for this.

Original: First, networks may be taking short paths according to some metric other than those we investigated here, implying that our metrics may not reflect the distance between representations in terms of ease of computation.
Revised: First, networks may be taking short paths according to some metric other than those we investigated here, implying that our metrics may not reflect the distance between representations in terms of ease of computation.

Original: Second, neural networks may fail to take efficient paths.
Revised: Second, neural networks may fail to take efficient paths.

Original: The distance metrics we consider are all differentiable, and so an interesting question for future work is whether networks can be regularized to take shorter paths, and whether such regularization will improve or reduce their performance or generalization ability.
Revised: The distance metrics we consider are all differentiable, and so an interesting question for future work is whether networks can be regularized to take shorter paths, and whether such regularization will improve or reduce their performance or generalization ability.

Original: Third, it could be that networks take the shortest and most direct path which is possible under some architectural constraints, which may prevent the hidden layers of the network from moving directly along the geodesic.
Revised: Third, it could be that networks take the shortest and most direct path which is possible under some architectural constraints, which may prevent the hidden layers of the network from moving directly along the geodesic.

Original: This explanation must be at least partially true, since the dimensionality of the representation space M generally exceeds the number of parameters in each layer/block of the network.[Equation]
Revised: This explanation must be at least partially true, since the dimensionality of the representation space M generally exceeds the number of parameters in each layer/block of the network.

Original: We were also surprised to discover that according to all metrics we investigated, network paths tended to be jagged , and consist predominantly of 90 ◦ angle turns.
Revised: We were also surprised to discover that according to all metrics we investigated, network paths tended to be jagged , and consist predominantly of 90 ◦ angle turns.

Original: Although it is well known that random directions in highdimensional spaces such as the representation space M tend to be nearly always orthogonal, this does not explain why we found that path angles are straighter at initialization and become more orthogonal during training.
Revised: Although it is well known that random directions in high-dimensional spaces such as the representation space M tend to be nearly always orthogonal, this does not explain why we found that path angles are straighter at initialization and become more orthogonal during training.

Original: This is surprising because the training objective ostensibly should encourage all layers to point in the same direction towards the targets.
Revised: This is surprising because the training objective ostensibly should encourage all layers to point in the same direction towards the targets.

Revised: This means that steps become more orthogonal through training.

Revised: This is an especially surprising result in light of recent work by Chan et al.

Revised: (2020) that suggests that a sequence of residual blocks can be interpreted as a sequence of small gradient steps optimizing a rate reduction objective; in this interpretation, all layers ought to be moving in the same general direction.

Revised: However, this is not what we find in our models trained by backpropagation and where angles are quantified using

Revised: Angular CKA (nor using other metrics – see Figure C.1).

Original: This empirical finding  suggests that future theoretical work is needed to interpret the direction of steps taken in representation space in the context of a given representational distance metric, and to understand which directions are realizable by a given network architecture.
Revised: Ultimately, ours is an empirical finding which suggests that future theoretical work is needed to interpret the direction of steps taken in representation space in the context of a given representational distance metric, and to understand which directions are realizable by a given network architecture.

Original: As described in the introduction, we are motivated to develop a general spatial analogy for information-processing, where complex transformations of representations cover more “distance” than simple ones.
Revised: As described in the introduction, we are motivated to develop a general spatial analogy for information-processing, where complex transformations of representations cover more “distance” than simple ones.

Original: In this sense, a measure of representational distance ought to reflect the function complexity of transforming X into Y .
Revised: In this sense, a measure of representational distance ought to reflect the function complexity of transforming X into Y .

Original: We chose to extend and compare existing representational distance metrics in order to build directly on previous work, but the metrics we evaluated here may not be interpretable as measures of function complexity.
Revised: We chose to extend and compare existing representational distance metrics in order to build directly on previous work, but the metrics we evaluated here may not be interpretable as measures of function complexity.

Original: An exciting avenue for future work is thus to derive a measure of representational distance directly from a measure of function complexity.
Revised: An exciting avenue for future work is thus to derive a measure of representational distance directly from a measure of function complexity.

Original: Such a measure will likely violate the axioms of a distance metric.
Revised: Such a measure will likely violate the axioms of a distance metric.

Original: For example, the complexity of a function and its inverse are in general not equal, and so it may be desirable to have d( X , Y ) <
Revised: For example, the complexity of a function and its inverse are in general not equal, and so it may be desirable to have d( X , Y ) <

Original: d( Y , X ) if the transformation from X to Y can be implemented by a simpler function than the inverse.
Revised: d( Y , X ) if the transformation from X to Y can be implemented by a simpler function than the inverse.

Original: Notably, many of the geometric properties studied here (shortest paths, angles, projection, etc.) can be extended to the asymmetric case using the theory of Finsler rather than Riemannian manifolds (Burago et al., 2001).
Revised: Notably, many of the geometric properties studied here (shortest paths, angles, projection, etc.) can be extended to the asymmetric case using the theory of Finsler rather than Riemannian manifolds (Burago et al., 2001).

Original: The analogy of neural networks as paths in a representation space brings together ideas about representational similarity and the expressivity of deep networks, marrying these techniques with intuitive and mathematically rigorous geometric concepts.
Revised: The analogy of neural networks as paths in a representation space brings together ideas about representational similarity and the expressivity of deep networks, marrying these techniques with intuitive and mathematically rigorous geometric concepts.

Original: Our work takes a first step in exploring the possibilities of this new geometric framework, and we anticipate that it will spark new insights about model design, model training, and model comparison.
Revised: Our work takes a first step in exploring the possibilities of this new geometric framework, and we anticipate that it will spark new insights about model design, model training, and model comparison.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: We have made an effort to make all of our code shareable and reproducible.

Original: Our model specifications, training recipe, and data augmentation are all based on standard research practice: we use default training options for all models taken from the OpenLTH framework.
Revised: Our model specifications, training recipe, and data augmentation are all based on standard research practice: we use default training options for all models taken from the OpenLTH framework 4 .

Original: Our standalone PyTorch library for metric representational distance, along with example usage and unit tests to verify its performance, is available at [redacted github link], and our repository to train and analyze models specific to this paper is available at [redacted github link].R
Revised: Our standalone PyTorch library for metric representational distance is available at [redacted github link], and our repository to train and analyze models specific to this paper is available at [redacted github link].

Original: A. A NGULAR CKA
Revised: A. A NGULAR CKA

Original: Angular CKA was introduced by Williams et al.
Revised: Angular CKA was introduced by Williams et al.

Original: (eq (60) in their supplement).
Revised: (eq (60) in their supplement).

Original: It is defined as the arccosine of centered kernel alignment (CKA), which is itself derived from the Hilbert-Schmidt independence criterion (HSIC)
Revised: It is defined as the arccosine of centered kernel alignment (CKA), which is itself derived from the Hilbert-Schmidt independence criterion (HSIC)

Original: (Gretton et al., 2005; Cortes et al., 2012; Kornblith et al., 2019).
Revised: (Gretton et al., 2005; Cortes et al., 2012; Kornblith et al., 2019).

Original: Because HSIC and CKA measure statistical dependence, distance measured by Angular CKA is high when the rows of X and Y are statistically independent, and low when they are highly correlated.
Revised: Because HSIC and CKA measure statistical dependence, distance measured by Angular CKA is high when the rows of X and Y are statistically independent, and low when they are highly correlated.

Original: Angular CKA is equivalent to arc-length distance on the spherical manifold consisting of centered and normalized m × m Gram matrices.
Revised: Angular CKA is equivalent to arc-length distance on the spherical manifold consisting of centered and normalized m × m Gram matrices.

Original: Let G X denote the Gram matrix of X , i.e. G X ij is given by the inner-product between the i th and j th rows of X .
Revised: Let G X denote the Gram matrix of X , i.e. G X ij is given by the inner-product between the i th and j th rows of X .

Original: Optionally, this inner-product may be computed using a kernel; following previous work (Kornblith et al., 2019; Nguyen et al., 2021), results in this paper use Linear CKA, i.e. we use G X =
Revised: Optionally, this inner-product may be computed using a kernel; following previous work (Kornblith et al., 2019; Nguyen et al., 2021), results in this paper use Linear CKA, i.e. we use G X =

Original: XX ⊤ , but our python package supports the use of other kernels.
Revised: XX ⊤ , but our python package supports the use of other kernels.

Original: The normalized and centered Gram matrix is given by
Revised: The normalized and centered Gram matrix is given by

Original: [Equation]
Revised: [Equation]

Original: where H =
Revised: where H =

Original: I m − 1 m 11 ⊤ is the m × m centering matrix , and ||·|| F is the Frobenius norm of a matrix.
Revised: I m − 1 m 11 ⊤ is the m × m centering matrix , and ||·|| F is the Frobenius norm of a matrix.

Original: Note that both G and ˜ G are symmetric and positive definite matrices.
Revised: Note that both G and ˜ G are symmetric and positive definite matrices.

Original: The Riemannian manifold M for Angular CKA consists of all such centered, normalized, symmetric positive definite matrices; it is a sphere in sense that (cid:68)
Revised: The Riemannian manifold M for Angular CKA consists of all such centered, normalized, symmetric positive definite matrices; it is a sphere in sense that (cid:68)

Original: ˜
Revised: ˜

Original: G X , ˜ G X (cid:69) F = 1 , where ⟨ A , B ⟩ F =
Revised: G X , ˜ G X (cid:69) F = 1 , where ⟨ A , B ⟩ F =

Original: Tr ( A ⊤ B ) is the Frobenius innerproduct.
Revised: Tr ( A ⊤ B ) is the Frobenius innerproduct.

Original: The embedding function for Angular CKA is f( X ) =
Revised: The embedding function for Angular CKA is f( X ) =

Original: ˜ G X .
Revised: ˜ G X .

Original: Distance according to Angular CKA is equal to arc length on the sphere consisting of centered and normalized Gram matrices:
Revised: Distance according to Angular CKA is equal to arc length on the sphere consisting of centered and normalized Gram matrices:

Original: [Equation]
Revised: [Equation]

Original: Because Angular CKA is an arc length, its geodesics lie along great circles ˜ G -space.
Revised: Because Angular CKA is an arc length, its geodesics lie along great circles ˜ G -space.

Original: We therefore can compute points along the geodesic in closed-form using the SLERP formula:
Revised: We therefore can compute points along the geodesic in closed-form using the SLERP formula:

Original: [Equation]
Revised: [Equation]

Original: .2) where t ∈ [0 , 1] is the fraction of distance along the geodesic from X to Y , and Ω =
Revised: where t ∈ [0 , 1] is the fraction of distance along the geodesic from X to Y , and Ω =

Original: d M ( ˜ G X , ˜ G Y ) .
Revised: d M ( ˜ G X , ˜ G Y ) .

Original: The tangent space for Angular CKA is the space of all symmetric m × m matrices, and the innerproduct in the tangent space is simply the Frobenius inner-product.
Revised: The tangent space for Angular CKA is the space of all symmetric m × m matrices, and the innerproduct in the tangent space is simply the Frobenius inner-product.

Original: The logarithmic map computes tangent vectors from a base point that point towards another point.
Revised: The logarithmic map computes tangent vectors from a base point that point towards another point.

Original: In the case of Angular CKA, the logarithmic map from ˜ G X to ˜ G Y is a tangent vector (symmetric matrix) at ˜ G X given by
Revised: In the case of Angular CKA, the logarithmic map from ˜ G X to ˜ G Y is a tangent vector (symmetric matrix) at ˜ G X given by

Original: [Equation]
Revised: [Equation]

Original: where W is the unit tangent vector at ˜ G X pointing towards ˜ G Y , given by
Revised: where W is the unit tangent vector at ˜ G X pointing towards ˜ G Y , given by

Original: [Equation]
Revised: [Equation]

Original: The exponential map is the inverse of the logarithmic map – it is a function that “extrapolates” a tangent vector W from a point to give another point on the manifold.
Revised: The exponential map is the inverse of the logarithmic map – it is a function that “extrapolates” a tangent vector W from a point to give another point on the manifold.

Original: In the case of Angular CKA, the exponential map is given by
Revised: In the case of Angular CKA, the exponential map is given by

Original: [Equation]
Revised: [Equation]

Original: where sinc ( x ) = sin( x ) x .
Revised: where sinc ( x ) = sin( x ) x .

Original: For all metrics, we compute angles between triplets of points by computing the inner-product of their tangent vectors.
Revised: For all metrics, we compute angles between triplets of points by computing the inner-product of their tangent vectors.

Original: In the case of Angular CKA in particular, let W XY denote the tangent vector pointing from X to Y , i.e. the result of log ˜ G X
Revised: In the case of Angular CKA in particular, let W XY denote the tangent vector pointing from X to Y , i.e. the result of log ˜ G X

Original: ( ˜ G Y ) .
Revised: ( ˜ G Y ) .

Original: Then,
Revised: Then,

Original: [Equation]

Original: A.

Original: [Equation]
Revised: [Equation]

Original: A.1. I NVARIANCES OF A NGULAR CKA
Revised: A.1. I NVARIANCES OF A NGULAR CKA

Original: The invariances of Angular CKA depend on the kernel used to compute the Gram matrix.
Revised: The invariances of Angular CKA depend on the kernel used to compute the Gram matrix.

Original: In the simplest case of Linear CKA , the Gram matrix is simply G X = XX ⊤ .
Revised: In the simplest case of Linear CKA , the Gram matrix is simply G X = XX ⊤ .

Original: The resulting metric is
Revised: The resulting metric is

Original: • shift-invariant due to centering the Gram matrix.
Revised: • shift-invariant due to centering the Gram matrix.

Original: • scale-invariant due to normalizing the Gram matrix.
Revised: • scale-invariant due to normalizing the Gram matrix.

Original: • rotation-invariant since ( X R )( X R )
Revised: • rotation-invariant since ( X R )( X R )

Original: ⊤ =
Revised: ⊤ =

Original: X RR ⊤ X ⊤ = XX ⊤ for any orthonormal R .
Revised: X RR ⊤ X ⊤ = XX ⊤ for any orthonormal R .

Original: However, Angular CKA with is not invariant to arbitrary affine transformations – a feature it inherits from CKA and has been argued to be an important feature of CKA (Kornblith et al., 2019).
Revised: However, Angular CKA with is not invariant to arbitrary affine transformations – a feature it inherits from CKA and has been argued to be an important feature of CKA (Kornblith et al., 2019).

Original: Note that when using a nonlinear kernel to compute the Gram matrix, the resulting metric may lose these invariances.
Revised: Note that when using a nonlinear kernel to compute the Gram matrix, the resulting metric may lose these invariances.

Original: However, Angular CKA with a nonlinear kernel may still be shift-, scale-, and rotationinvariant if the kernel itself has those invariances.
Revised: However, Angular CKA with a nonlinear kernel may still be shift-, scale-, and rotationinvariant if the kernel itself has those invariances.

Original: For example the squared exponential kernel
Revised: For example the squared exponential kernel

Original: [Equation]
Revised: [Equation]

Original: is naturally shift- and rotation-invariant, and it can me made further scale-invariant by setting the length scale τ automatically based on the scale of the data.
Revised: is naturally shift- and rotation-invariant, and it can me made further scale-invariant by setting the length scale τ automatically based on the scale of the data.

Original: A. S HAPE M ETRICS
Revised: A. S HAPE M ETRICS

Original: Williams et al.
Revised: Williams et al.

Original: (2021) proposed using a generalization of Procrustes distance and Kendall’s shape space to measure metric distance between neural representations.
Revised: (2021) proposed using a generalization of Procrustes distance and Kendall’s shape space to measure metric distance between neural representations.

Original: Shape-space and Procrustes distance are a well-studied case of a Riemannian manifold between point clouds (Nava-Yazdani et al., 2020).
Revised: Shape-space and Procrustes distance are a well-studied case of a Riemannian manifold between point clouds (Nava-Yazdani et al., 2020).

Original: Williams et al.
Revised: Williams et al.

Original: (2021) consider two different shape metrics – one angular shape metric and one Euclidean shape metric.
Revised: (2021) consider two different shape metrics – one angular shape metric and one Euclidean shape metric.

Original: The key idea behind both of these metrics is as follows: m × n matrices of neural data are first transformed into a common m × p space, and interpreted as a point cloud consisting of p − dimensional points.
Revised: The key idea behind both of these metrics is as follows: m × n matrices of neural data are first transformed into a common m × p space, and interpreted as a point cloud consisting of p − dimensional points.

Original: Then, any two point clouds are scaled and rotated so that they maximally align with each other.
Revised: Then, any two point clouds are scaled and rotated so that they maximally align with each other.

Original: The final distance is then computed as some measure of discrepancy between these maximally-aligned point clouds.
Revised: The final distance is then computed as some measure of discrepancy between these maximally-aligned point clouds.

Original: The behavior of these shape metrics is tuned using two hyperparameters: the dimensionality p , and a partial whitening parameter α .
Revised: The behavior of these shape metrics is tuned using two hyperparameters: the dimensionality p , and a partial whitening parameter α .

Original: The role of the embedding function for shape metrics is to convert n − dimensional neural data into a canonical zero-mean p − dimensional space (i.e. M = R m × p is the space of all m × p matrices whose column means are all zero).
Revised: The role of the embedding function for shape metrics is to convert n − dimensional neural data into a canonical zero-mean p − dimensional space (i.e. M = R m × p is the space of all m × p matrices whose column means are all zero).

Original: Williams et al.
Revised: Williams et al.

Original: (2021) also include a partial whitening stage as part of the embedding.
Revised: (2021) also include a partial whitening stage as part of the embedding.

Original: This space of m × p zero-mean
Revised: This space of m × p zero-mean

Original: (and sometimes scaled) matrices is called the pre shape space (Nava-Yazdani et al., 2020).
Revised: (and sometimes scaled) matrices is called the pre shape space (Nava-Yazdani et al., 2020).

Original: In the case where n < p , this conversion from n to p dimensions is done by simply padding X with p − n columns of all zeros.
Revised: In the case where n < p , this conversion from n to p dimensions is done by simply padding X with p − n columns of all zeros.

Original: In the case where p < n , we reduce the dimensionality of X by keeping only the top p principal components.
Revised: In the case where p < n , we reduce the dimensionality of X by keeping only the top p principal components.

Original: Formally, let ¯
Revised: Formally, let ¯

Original: X = X −
Revised: X = X −

Original: [Equation]
Revised: [Equation]

Original: mi =1 X
Revised: mi =1 X

Original: i be the matrix of neural data with its mean subtracted, then
Revised: i be the matrix of neural data with its mean subtracted, then

Original: [Equation]
Revised: [Equation]

Original: where U : p stands for the first p principal components of ¯ X , as unit column vectors, and 0 is a m × ( p − n ) matrix of all zeros.
Revised: where U : p stands for the first p principal components of ¯ X , as unit column vectors, and 0 is a m × ( p − n ) matrix of all zeros.

Original: The partial whitening function begins by computing the eigendecomposition of its input, m − 1 ¯ X ⊤ ¯ X = V Σ V ⊤ (here, V is a p × p orthonormal matrix containing the top principal components of ¯ X , and Σ is a diagonal matrix of variances).
Revised: The partial whitening function begins by computing the eigendecomposition of its input, m − 1 ¯ X ⊤ ¯ X = V Σ V ⊤ (here, V is a p × p orthonormal matrix containing the top principal components of ¯ X , and Σ is a diagonal matrix of variances).

Original: Then, the partial whitening stage is
Revised: Then, the partial whitening stage is

Original: [Equation]
Revised: [Equation]

Original: Note that when α = 0 , this is equivalent to ZCA whitening, and when α = 1 it leaves ¯ X unchanged.
Revised: Note that when α = 0 , this is equivalent to ZCA whitening, and when α = 1 it leaves ¯ X unchanged.

Original: All shape metric results we report are with p = 100 and α = 0 .
Revised: All shape metric results we report are with p = 100 and α = 0 .

Original: We use α = 0 because this entails further invariances, making metrics more interpretable across disparate layer shapes and types (see section A.2.1 below for details on shape metric invariances).
Revised: We use α = 0 because this entails further invariances, making metrics more interpretable across disparate layer shapes and types (see section A.2.1 below for details on shape metric invariances).

Original: Both the angular and Euclidean shape metrics require aligning by rotating the embedded points by minimizing || ˜ X − ˜ Y R || F where R is a p × p orthonormal matrix.
Revised: Both the angular and Euclidean shape metrics require aligning by rotating the embedded points by minimizing || ˜ X − ˜ Y R || F where R is a p × p orthonormal matrix.

Original: This is known as the orthogonal
Revised: This is known as the orthogonal

Original: Procrustes problem, and its solution is given by
Revised: Procrustes problem, and its solution is given by

Original: [Equation]
Revised: [Equation]

Original: where U Σ V ⊤ =
Revised: where U Σ V ⊤ =

Original: ˜ X ⊤ ˜ Y is a singular value decomposition of ˜ X ⊤ ˜ Y .
Revised: ˜ X ⊤ ˜ Y is a singular value decomposition of ˜ X ⊤ ˜ Y .

Original: The generalized shape metrics introduced by Williams et al.
Revised: The generalized shape metrics introduced by Williams et al.

Original: (2021) include further restrictions on R , such as considering rotations across channel but not spatial dimensions of convolutional layers, but we omit these restrictions in our work.
Revised: (2021) include further restrictions on R , such as considering rotations across channel but not spatial dimensions of convolutional layers, but we omit these restrictions in our work.

Original: In the case of angular shape metrics, distance is defined as
Revised: In the case of angular shape metrics, distance is defined as

Original: [Equation]
Revised: [Equation]

Original: In the case of Euclidean shape metrics, distance is defined as
Revised: In the case of Euclidean shape metrics, distance is defined as

Original: [Equation]
Revised: [Equation]

Original: We compute geodesics in shape space after finding R to align ˜ Y to ˜ X .
Revised: We compute geodesics in shape space after finding R to align ˜ Y to ˜ X .

Original: Then, the geodesic from ˜ X to ˜ Y R in the angular case is given by the SLERP formula as in (A.2):
Revised: Then, the geodesic from ˜ X to ˜ Y R in the angular case is given by the SLERP formula as in (3):

Original: [Equation]
Revised: [Equation]

Original: where Ω = d M ( ˜ X , ˜ Y ) is the angular shape distance.
Revised: where Ω = d M ( ˜ X , ˜ Y ) is the angular shape distance.

Original: Note that this means that geodesic ( ˜ X , ˜ Y , 1) results in a point that is equivalent but not identical to ˜ Y .
Revised: Note that this means that geodesic ( ˜ X , ˜ Y , 1) results in a point that is equivalent but not identical to ˜ Y .

Original: Tangent vectors for Euclidean shape metrics can be any m × p matrix whose column-wise mean is zero.
Revised: Tangent vectors for Euclidean shape metrics can be any m × p matrix whose column-wise mean is zero.

Original: In the case of angular shape metrics, the tangent space is further restricted to the tangent space of the hypersphere of unit-Frobenius-norm matrices (i.e. a tangent vector W at ˜ X must satisfy (cid:68)
Revised: In the case of angular shape metrics, the tangent space is further restricted to the tangent space of the hypersphere of unit-Frobenius-norm matrices (i.e. a tangent vector W at ˜ X must satisfy (cid:68)

Original: ˜ X , W (cid:69)
Revised: ˜ X , W (cid:69)

Original: F = 0 in the angular case).
Revised: F = 0 in the angular case).

Original: The tangent space is further divided into so-called horizontal and vertical subspaces, where the vertical subspace captures changes to ˜ X that leave distance invariant, i.e. rotations that are removed by alignment by R , and the horizontal subspace captures changes that affect the metric (Nava-Yazdani et al., 2020).
Revised: The tangent space is further divided into so-called horizontal and vertical subspaces, where the vertical subspace captures changes to ˜ X that leave distance invariant, i.e. rotations that are removed by alignment by R , and the horizontal subspace captures changes that affect the metric (Nava-Yazdani et al., 2020).

Original: The vertical component of a tangent vector W at point ˜ X is given by vert ˜ X ( W ) =
Revised: The vertical component of a tangent vector W at point ˜ X is given by vert ˜ X ( W ) =

Original: ˜ X A , where A ∈ R p × p is the solution to the following Sylvester equation:
Revised: ˜ X A , where A ∈ R p × p is the solution to the following Sylvester equation:

Original: [Equation]
Revised: [Equation]

Original: Following the example of Miolane et al.
Revised: Following the example of Miolane et al.

Original: (2020), we use the solve sylvester function from
Revised: (2020), we use the solve sylvester function from

Original: Scipy to compute this (Virtanen et al., 2020).
Revised: Scipy to compute this (Virtanen et al., 2020).

Original: The horizontal component of a tangent vector is given by simply subtracting the vertical part of W :
Revised: The horizontal component of a tangent vector is given by simply subtracting the vertical part of W :

Original: [Equation]
Revised: [Equation]

Original: To compute the angle between any triplet of representations, we use the inner-product of tangent vectors, as in (A.5), but using only the horizontal part of each tangent vector.
Revised: To compute the angle between any triplet of representations, we use the inner-product of tangent vectors, as in (5), but using only the horizontal part of each tangent vector.

Original: As in Angular CKA, we compute horizontal tangent vectors from ˜ X to ˜ Y using the logarithmic map , which in the case of shape metrics is given by
Revised: As in Angular CKA, we compute horizontal tangent vectors from ˜ X to ˜ Y using the logarithmic map , which in the case of shape metrics is given by

Original: [Equation]
Revised: [Equation]

Original: in the Euclidean case, orhorizontal
Revised: in the Euclidean case, or

Original: [Equation]
Revised: [Equation]

Original: where W is the unit tangent vector at ˜ X pointing towards ˜ Y , given by
Revised: where W is the unit tangent vector at ˜ X pointing towards ˜ Y , given by

Original: [Equation]
Revised: [Equation]

Original: (Nava-Yazdani et al., 2020).
Revised: (Nava-Yazdani et al., 2020).

Original: As in (A.8) and (A.9), R is the rotation matrix that optimally aligns ˜ Y to ˜ X .
Revised: As in (A.5) and (A.6), R is the rotation matrix that optimally aligns ˜ Y to ˜ X .

Original: A.2. I NVARIANCES OF S HAPE M ETRICS
Revised: A.2. I NVARIANCES OF S HAPE M ETRICS

Original: The invariances of the shape metrics depend on a variety of hyperparameter settings.
Revised: The invariances of the shape metrics depend on a variety of hyperparameter settings.

Original: • All shape metrics are shift-invariant because the embedding function ˜ X = f( X ) subtracts the mean.
Revised: • All shape metrics are shift-invariant because the embedding function ˜ X = f( X ) subtracts the mean.

Original: • All shape metrics are rotation-invariant because of the Procrustes alignment procedure, and because rotation does not affect the principal component projection nor the zeropadding step of (A.7).
Revised: • All shape metrics are rotation-invariant because of the Procrustes alignment procedure, and because rotation does not affect the principal component projection nor the zeropadding step of (A.4).

Original: • The angular shape metric is scale-invariant because (A.8) divides by the norms of ˜ X and ˜ Y .
Revised: • The angular shape metric is scale-invariant because (A.5) divides by the norms of ˜ X and ˜ Y .

Original: • The Euclidean shape metric is not scale-invariant in general, but it is for the special case of α = 0 , since scale is removed by whitening.
Revised: • The Euclidean shape metric is not scale-invariant in general, but it is for the special case of α = 0 , since scale is removed by whitening.

Original: • Neither angular nor Euclidean shape metrics is affine-invariant in general, but both can become affine-invariant for the special case of α = 0 , since full-rank affine transforms are removed by whitening as long as n ≤ p .
Revised: • Neither angular nor Euclidean shape metrics is affine-invariant in general, but both can become affine-invariant for the special case of α = 0 , since full-rank affine transforms are removed by whitening as long as n ≤ p .

Original: However, in the n > p case, an affine transformation may amplify or suppress the principal components of the data, and as a result it can affect the embedding stage (A.7).
Revised: However, in the n > p case, an affine transformation may amplify or suppress the principal components of the data, and as a result it can affect the embedding stage (A.4).

Original: A. A FFINE I NVARIANT R IEMANNIAN M ETRIC
Revised: A. A FFINE I NVARIANT R IEMANNIAN M ETRIC

Original: The Affine Invariant Riemannian (AIR) metric is a metric between symmetric positive definite (SPD) matrices, originally derived for use in image processing (Pennec, 2006; 2019), and recently it was proposed to use it as a metric between neural representations by first converting neural data into a
Revised: The Affine Invariant Riemannian (AIR) metric is a metric between symmetric positive definite (SPD) matrices, originally derived for use in image processing (Pennec, 2006; 2019), and recently it was proposed to use it as a metric between neural representations by first converting neural data into a

Original: SPD matrix (Shahbazi et al., 2021).
Revised: SPD matrix (Shahbazi et al., 2021).

Original: The embedding function can therefore be any function that maps m × n matrices in X into M =
Revised: The embedding function can therefore be any function that maps m × n matrices in X into M =

Original: Sym + k for some k .
Revised: Sym + k for some k .

Original: Shahbazi et al.
Revised: Shahbazi et al.

Original: (2021) considered two possibilities for the embedding stage: either using the m × m Gram matrix f( X ) =
Revised: (2021) considered two possibilities for the embedding stage: either using the m × m Gram matrix f( X ) =

Original: G X , or using the n × n data covariance matrix f( X ) =
Revised: G X , or using the n × n data covariance matrix f( X ) =

Original: cov ( X ) =
Revised: cov ( X ) =

Original: 1 m − 1 X ⊤ H X .
Revised: 1 m − 1 X ⊤ H X .

Original: These correspond to complementary perspectives on the nature of neural representation, analogous to the difference between representational similarity analysis and pattern component analysis (Diedrichsen and Kriegeskorte, 2017).
Revised: These correspond to complementary perspectives on the nature of neural representation, analogous to the difference between representational similarity analysis and pattern component analysis (Diedrichsen and Kriegeskorte, 2017).

Original: The challenge when using the m × m Gram matrix approach is that, without further regularization, a m × m Gram matrix has rank n when n < m , which implies that it cannot be SPD (and the metric considers all rank-deficient matrices to be infinitely far away).
Revised: The challenge when using the m × m Gram matrix approach is that, without further regularization, a m × m Gram matrix has rank n when n < m , which implies that it cannot be SPD (and the metric considers all rank-deficient matrices to be infinitely far away).

Original: To address this, our toolbox implements the AIR metric between neural representations with additional regularization options.
Revised: To address this, our toolbox implements the AIR metric between neural representations with additional regularization options.

Original: In the Gram matrix case, we regularize in two ways: first, we compute the Gram matrix using a kernel that implicitly has an infinite feature space (so that m is much less than the number of features).
Revised: In the Gram matrix case, we regularize in two ways: first, we compute the Gram matrix using a kernel that implicitly has an infinite feature space (so that m is much less than the number of features).

Original: This alleviates the rank-deficiency problem in cases where n < m but rows are unique.
Revised: This alleviates the rank-deficiency problem in cases where n < m but rows are unique.

Original: However, when rows of X contain duplicates (notably, this is true for the target labels), G K is still rank-deficient.
Revised: However, when rows of X contain duplicates (notably, this is true for the target labels), G K is still rank-deficient.

Original: To address this, we include a second regularization stage where we add a small diagonal ridge with magnitude ϵ .
Revised: To address this, we include a second regularization stage where we add a small diagonal ridge with magnitude ϵ .

Original: The full embedding function in the Gram matrix case is given by
Revised: The full embedding function in the Gram matrix case is given by

Original: [Equation]
Revised: [Equation]

Original: where the ij th element of G X is given by k
Revised: where the ij th element of G X is given by k

Original: ( X i , X j ) .
Revised: ( X i , X j ) .

Original: For our results in the paper, we use ϵ = 0 .and
Revised: For our results in the paper, we use ϵ = 0 .and

Original: a squared exponential kernel for k as in (A.6), setting the length scale τ automatically to the median pairwise Euclidean distance between rows of X .
Revised: a squared exponential kernel for k as in (A.3), setting the length scale τ automatically to the median pairwise Euclidean distance between rows of X .

Original: The main challenge when using the covariance matrix approach is that it cannot be directly applied to compare layers with different numbers of neurons n .
Revised: The main challenge when using the covariance matrix approach is that it cannot be directly applied to compare layers with different numbers of neurons n .

Original: To address this, we first convert from m × n matrices of neural data into a common m × p size, using the same method as we use for the shape metrics as in (A.7), but without the whitening stage.
Revised: To address this, we first convert from m × n matrices of neural data into a common m × p size, using the same method as we use for the shape metrics as in (A.4), but without the whitening stage.

Original: We can then embed all layers into a common space of p × p covariance matrices.
Revised: We can then embed all layers into a common space of p × p covariance matrices.

Original: As in the Gram matrix case, we again run into rank-deficiency issues when n < m (e.g. for the one-hot embedding of targets for which n = 10 ), and so we again regularize by adding a diagonal ridge to the resulting covariance matrices.
Revised: As in the Gram matrix case, we again run into rank-deficiency issues when n < m (e.g. for the one-hot embedding of targets for which n = 10 ), and so we again regularize by adding a diagonal ridge to the resulting covariance matrices.

Original: The full embedding function in the covariance matrix case is given by
Revised: The full embedding function in the covariance matrix case is given by

Original: [Equation]
Revised: [Equation]

Original: (compare with (A.7)).
Revised: (compare with (A.4)).

Original: Let P = f( X ) and Q = f
Revised: Let P = f( X ) and Q = f

Original: ( Y ) be SPD matrices (we are using P and Q instead of ˜ X and ˜ Y to use a consistent notation with Pennec (2019)).
Revised: ( Y ) be SPD matrices (we are using P and Q instead of ˜ X and ˜ Y to use a consistent notation with Pennec (2019)).

Original: The AIR metric distance is defined as
Revised: The AIR metric distance is defined as

Original: [Equation]
Revised: [Equation]

Original: where d i is the i th eigenvalue of P − 12 QP − 12 (Pennec, 2006; 2019).
Revised: where d i is the i th eigenvalue of P − 12 QP − 12 (Pennec, 2006; 2019).

Original: Since P is SPD, its singular value decomposition can be written P = V Σ V ⊤ , where Σ is a diagonal matrix and V is orthonormal.
Revised: Since P is SPD, its singular value decomposition can be written P = V Σ V ⊤ , where Σ is a diagonal matrix and V is orthonormal.

Original: Following Pennec (2019), we use element-wise square root, exp, and log operations on the singular values to define the matrix square root, matrix exponential, and matrix logarithm:
Revised: Following Pennec (2019), we use element-wise square root, exp, and log operations on the singular values to define the matrix square root, matrix exponential, and matrix logarithm:

Original: [Equation]
Revised: [Equation]

Original: where the operations on the left hand side are matrix power, exponential, and log, whereas pow , exp , and log operations are performed element-wise on the diagonal of Σ .
Revised: where the operations on the left hand side are matrix power, exponential, and log, whereas pow , exp , and log operations are performed element-wise on the diagonal of Σ .

Original: P k is equivalent to pow ( P , k ) .
Revised: P k is equivalent to pow ( P , k ) .

Original: The geodesics from P to Q is given by
Revised: The geodesics from P to Q is given by

Original: [Equation]
Revised: [Equation]

Original: (combining equations (3.12) and (3.13) in Pennec (2019)).
Revised: (combining equations (3.12) and (3.13) in Pennec (2019)).

Original: Tangent vectors in this space are symmetric matrices, and the logarithmic map is given by
Revised: Tangent vectors in this space are symmetric matrices, and the logarithmic map is given by

Original: [Equation]
Revised: [Equation]

Original: (see equation (3.12) in Pennec (2019)).
Revised: (see equation (3.12) in Pennec (2019)).

Original: The exponential map of the tangent vector W at P is given by
Revised: The exponential map of the tangent vector W at P is given by

Original: [Equation]
Revised: [Equation]

Original: (see equation (3.13) in Pennec (2019)).
Revised: (see equation (3.13) in Pennec (2019)).

Original: One can easily verify that exp P (log P ( Q ))
Revised: One can easily verify that exp P (log P ( Q ))

Original: =
Revised: =

Original: Q .
Revised: Q .

Original: As before, we compute angles between triplets of representations X , Y , Z by computing the inner product of the log Y ( X ) and log Y ( Z ) tangent vectors, but unlike the previous metrics the definition of inner products for the AIR metric is not simply the Frobenius inner product.
Revised: As before, we compute angles between triplets of representations X , Y , Z by computing the inner product of the log Y ( X ) and log Y ( Z ) tangent vectors, but unlike the previous metrics the definition of inner products for the AIR metric is not simply the Frobenius inner product.

Original: For the AIR metric, the inner product of tangent vectors W and V at P is defined as
Revised: For the AIR metric, the inner product of tangent vectors W and V at P is defined as

Original: [Equation]
Revised: [Equation]

Original: A.3. I NVARIANCES OF A FFINE I NVARIANT R IEMANNIAN M ETRIC
Revised: A.3. I NVARIANCES OF A FFINE I NVARIANT R IEMANNIAN M ETRIC

Original: We will treat the Gram matrix (A.13) and the covariance matrix (A.14) cases separately.
Revised: We will treat the Gram matrix (A.10) and the covariance matrix (A.11) cases separately.

Original: In the Gram matrix case,
Revised: In the Gram matrix case,

Original: • The AIR metric is shift-invariant , scale-invariant , and/or rotation-invariant if and only if the kernel used to compute G X has the corresponding invariance.
Revised: • The AIR metric is shift-invariant , scale-invariant , and/or rotation-invariant if and only if the kernel used to compute G X has the corresponding invariance.

Original: Because we use a squared-exponential kernel with a length scale that adapts to the data scale, we have all three invariances.
Revised: Because we use a squared-exponential kernel with a length scale that adapts to the data scale, we have all three invariances.

Original: • The AIR metric is, despite its name, not affine-invariant in the sense we are interested in, since affine transformations of X will in general affect G X through the nonlinear kernel (e
Revised: • The AIR metric is, despite its name, not affine-invariant in the sense we are interested in, since affine transformations of X will in general affect G X through the nonlinear kernel (e

Original: g. the squared exponential kernel with an isotropic length scale is sensitive to nonisotropic scaling of X ).
Revised: g. the squared exponential kernel with an isotropic length scale is sensitive to nonisotropic scaling of X ).

Original: In the covariance matrix case,
Revised: In the covariance matrix case,

Original: • The AIR metric is shift-invariant because covariance subtracts the mean.
Revised: • The AIR metric is shift-invariant because covariance subtracts the mean.

Original: • The AIR metric is scale- and rotation-invariant due to the eponymous “affine-invariances” of the metric itself (Pennec, 2006; 2019).
Revised: • The AIR metric is scale- and rotation-invariant due to the eponymous “affine-invariances” of the metric itself (Pennec, 2006; 2019).

Original: • As in the case of shape metrics discussed above, the AIR metric may or may not be invariant
Revised: • As in the case of shape metrics discussed above, the AIR metric may or may not be invariant

Original: to arbitrary affine transformations of X due to the restriction to the top p principal components in the embedding stage.
Revised: to arbitrary affine transformations of X due to the restriction to the top p principal components in the embedding stage.

Original: Only in the case where n > p and A is a matrix such that X A changes the subspace of the top p principal components, then the resulting metric is not invariant to A
Revised: Only in the case where n > p and A is a matrix such that X A changes the subspace of the top p principal components, then the resulting metric is not invariant to A

Original: B M ODELS AND TRAINING DETAILS
Revised: B M ODELS AND TRAINING DETAILS

Original: We trained a collection of convolutional networks including both residual networks (He et al., 2016) and VGG (Simonyan and Zisserman, 2014) on CIFAR-10 (Krizhevsky, 2009) using PyTorch (Paszke et al., 2019), managing compute jobs using GNU Parallel (Tange, 2011).
Revised: We trained a collection of convolutional networks including both residual networks (He et al., 2016) and VGG (Simonyan and Zisserman, 2014) on CIFAR-10 (Krizhevsky, 2009) using PyTorch (Paszke et al., 2019), managing compute jobs using GNU Parallel (Tange, 2011).

Original: We used the
Revised: We used the

Original: OpenLTH framework 4 for training and checkpointing models, using the default hyperparameters for each model.
Revised: OpenLTH framework 6 for training and checkpointing models, using the default hyperparameters for each model.

Original: Following Nguyen et al.
Revised: Following Nguyen et al.

Original: (2021), we trained Residual networks of varying widths and depths.
Revised: (2021), we trained Residual networks of varying widths and depths.

Original: The “width” refers to the number of feature channels per convolutional layer, and took on values of { 16 , 32 , 64 , 128 , 160 } (corresponding to the base size of 16 multiplied by { 1 × , 2 × , 4 × , 8 × , 10 ×} ).
Revised: The “width” refers to the number of feature channels per convolutional layer, and took on values of { 16 , 32 , 64 , 128 , 160 } (corresponding to the base size of 16 multiplied by { 1 × , 2 × , 4 × , 8 × , 10 ×} ).

Original: The “depth” controls the number of residual blocks, according to the formula # blocks =
Revised: The “depth” controls the number of residual blocks, according to the formula # blocks =

Original: ( depth − 2) / 2 , since each block contains two convolutional layers, and there are two additional preprocessing/projection layers before/after the blocks.
Revised: ( depth − 2) / 2 , since each block contains two convolutional layers, and there are two additional preprocessing/projection layers before/after the blocks.

Original: We trained models of depths { 14 , 20 , 26 , 32 , 38 , 44 , 56 , 110 } .
Revised: We trained models of depths { 14 , 20 , 26 , 32 , 38 , 44 , 56 , 110 } .

Original: The VGG architecture supports “depths” of { 11 , 13 , 16 , 18 } , all at the same width.
Revised: The VGG architecture supports “depths” of { 11 , 13 , 16 , 18 } , all at the same width.

Original: The test accuracy of all models is shown in Table B.2.
Revised: The test accuracy of all models is shown in Table B.2.

Original: We analyzed the representational distances and geometry of a subset of these, focusing on depths 14 and 38 (for all widths), and widths 16 and 64 (for all depths).
Revised: We analyzed the representational distances and geometry of a subset of these, focusing on depths 14 and 38 (for all widths), and widths 16 and 64 (for all depths).

Original: All models were trained using the default training hyperparameters of OpenLTH.
Revised: All models were trained using the default training hyperparameters of OpenLTH 7 .

Original: Specifically, all models were trained by stochastic gradient descent for 160 epochs with a batch size of 128 (390. batches per epoch of 50k training items), an initial learning rate of 0 .
Revised: Specifically, all models were trained by stochastic gradient descent for 160 epochs with a batch size of 128 (390. batches per epoch of 50k training items), an initial learning rate of 0 .

Original: 1 reducing to 0 .
Revised: 1 reducing to 0 .

Original: 01 and 0 .after 80 and 120 epochs respectively, momentum of 0 .
Revised: 01 and 0 .after 80 and 120 epochs respectively, momentum of 0 .

Original: 9 , and weight decay of 0 . 0001 .
Revised: 9 , and weight decay of 0 . 0001 .

Original: During training, images were augmented by random horizontal flips and random ± 4 pixel left/right or up/down shifts (padding with zeros).
Revised: During training, images were augmented by random horizontal flips and random ± 4 pixel left/right or up/down shifts (padding with zeros).

Original: For the training analysis in Figure 2, we trained a second Resnet-14 with default width (16), and analyzed the representational distance and geometry once every 10 batches for the epoch, since the vast majority of the changes to the network performance and geometry occur in the first epoch.
Revised: For the training analysis in Figure 2, we trained a second Resnet-14 with default width (16), and analyzed the representational distance and geometry once every 10 batches for the epoch, since the vast majority of the changes to the network performance and geometry occur in the first epoch.

Original: [Table]
Revised: [Table]

Original: Resnet 14/ 90.
Revised: Resnet 14/ 90.

Original: Resnet 14/ 92.
Revised: Resnet 14/ 92.

Original: Resnet 14/ 93.
Revised: Resnet 14/ 93.

Original: Resnet 14/ 94. Resnet 14/ 94.
Revised: Resnet 14/ 94. Resnet 14/ 94.

Original: Resnet 20/ 91.
Revised: Resnet 20/ 91.

Original: Resnet 20/ 93.
Revised: Resnet 20/ 93.

Original: Resnet 20/ 94.
Revised: Resnet 20/ 94.

Original: Resnet 20/ 95. Resnet 20/ 95.
Revised: Resnet 20/ 95. Resnet 20/ 95.

Original: Resnet 26/ 91.
Revised: Resnet 26/ 91.

Original: Resnet 26/ 93.
Revised: Resnet 26/ 93.

Original: Resnet 26/ 94.
Revised: Resnet 26/ 94.

Original: Resnet 26/ 95. Resnet 26/ 95.
Revised: Resnet 26/ 95. Resnet 26/ 95.

Original: Resnet 32/ 93.
Revised: Resnet 32/ 93.

Original: Resnet 32/ 94. Resnet 32/ 94.
Revised: Resnet 32/ 94. Resnet 32/ 94.

Original: Resnet 32/ 95.
Revised: Resnet 32/ 95.

Original: Resnet 32/ 94.
Revised: Resnet 32/ 94.

Original: Resnet 38/ 92.
Revised: Resnet 38/ 92.

Original: Resnet 38/ 94.
Revised: Resnet 38/ 94.

Original: Resnet 38/ 95.
Revised: Resnet 38/ 95.

Original: Resnet 38/ 94. Resnet 38/ 94.
Revised: Resnet 38/ 94. Resnet 38/ 94.

Original: Resnet 44/ 93.
Revised: Resnet 44/ 93.

Original: Resnet 44/ 94.
Revised: Resnet 44/ 94.

Original: Resnet 44/ 95.
Revised: Resnet 44/ 95.

Original: Resnet 44/ 94. Resnet 44/ 94.
Revised: Resnet 44/ 94. Resnet 44/ 94.

Original: Resnet 56/ 93.
Revised: Resnet 56/ 93.

Original: Resnet 56/ 94. Resnet 56/ 94.
Revised: Resnet 56/ 94. Resnet 56/ 94.

Original: Resnet 56/ 95. Resnet 56/ 95.
Revised: Resnet 56/ 95. Resnet 56/ 95.

Original: Resnet 110/ 93.
Revised: Resnet 110/ 93.

Original: Resnet 110/ 94. Resnet 110/ 94.
Revised: Resnet 110/ 94. Resnet 110/ 94.

Original: Resnet 110/ 95. Resnet 110/ 95.
Revised: Resnet 110/ 95. Resnet 110/ 95.

Original: VGG91.
Revised: VGG91.

Original: VGG93. VGG93. 
Revised: VGG93. VGG93. VGG93.

Revised: Notably, all metrics agree with the finding that “internal angles” are close to orthogonal throughout the network and that “target angles” are close to orthogonal for early layers with the later layers being the only ones that appreciably point in the direction of the targets.

Original: VGG93.
Revised: Both the Angular Shape Metric and the Affine-Invariant Riemannian metric measure large distances in the last few layers – as the last convolutional layer is projected to logits and then passed through a softmax function. For both of these metrics, the first principal component describing the network path primarily describes just these last few layers (Figure C.1A,B,G,H). This suggests that, while these metrics give sensible results for distances between hidden layers, they may be ill-suited for measuring distances between internal representations and targets.

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]