On the Symmetries of Deep Learning Models and their Internal Representations
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Symmetry has been a fundamental tool in the exploration of a broad range ofcomplex systems.
In machine learning, symmetry has been explored in both models and data.
In this paper we seek to connect the symmetries arising from thearchitecture of a family of models with the symmetries of that family’s internalrepresentation of data.
We do this by calculating a set of fundamental symmetrygroups, which we call the intertwiner groups of the model.
Each of these arisesfrom a particular nonlinear layer of the model and different nonlinearities resultin different symmetry groups.
These groups change the weights of a model insuch a way that the underlying function that the model represents remains constantbut the internal representations of data inside the model may change.
We connectintertwiner groups to a model’s internal representations of data through a rangeof experiments that probe similarities between hidden states across models withthe same architecture.
Our work suggests that the symmetries of a network arepropagated into the symmetries in that network’s representation of data, providing us with a better understanding of how architecture affects the learning andprediction process.
Finally, we speculate that for ReLU networks, the intertwinergroups may provide a justiﬁcation for the common practice of concentrating modelinterpretability exploration on the activation basis in hidden layers rather thanarbitrary linear combinations thereof.
Introduction
Symmetry provides an important path to understanding across a range of disciplines.
This principleis well-established in mathematics and physics, where it has been a fundamental tool (e g., Noether’s
Theorem [Noe18]).
Symmetry has also been brought to bear on deep learning problems from anumber of directions.
There is, for example, a rich research thread that studies symmetries in datatypes that can be used to inform model architectures.
The most famous examples of this are standardconvolutional neural networks which encode the translation invariance of many types of semanticcontent in natural images into a network’s architecture.
In this paper, we focus on connectionsbetween two other types of symmetry associated with deep learning models: the symmetries in thelearnable parameters of the model and the symmetries across different models’ internal representationof the same data.
The ﬁrst of these directions of research starts with the observation that in modern neural networksthere exist models with different weights that behave identically on all possible input.
We show thatat least some of these equivalent models arise because of symmetries intrinsic to the nonlinearitiesof the network.
We call these groups of symmetries, each of which is attached to a particular typeof nonlinear layer σ of dimension n , the intertwiner groups G σ n of the model.
These intertwinergroups come with a natural action on network weights for which the realization map to function space of [JGH18] is invariant (proposition 3.4).
As such they provide a unifying framework in whichto discuss well-known weight space symmetries such as permutation of neurons [Bre+19] and scaleinvariance properties of ReLU and batch-norm [IS15; NH10].
Next, we tie our intertwiner groups to the symmetries between different model’s internal representations of the same data.
We do this through a range of experiments that we describe below; eachbuilds on a signiﬁcant recent advance in the ﬁeld.
Neural stitching with intertwiner groups: The work of [BNB21; Csi+21; LV15] demonstratedthat one can take two trained neural networks, say A and B, with the same architecture but trainedfrom different randomly initialized weights, and connect the early layers of network A to the laterlayers of network B with a simple “stitching” layer and achieve negligible loss in prediction accuracy.
This was taken as evidence of the similarity of strong model’s representations of data.
Though theoriginal experiments use a fully connected linear layer to stitch, we provide theoretical evidencein theorem 4.2 that much less is needed.
Indeed, we show that the intertwiner group (which hasfar fewer parameters in general) is the minimal viable stitching layer to preserve accuracy.
Weconduct experiments stitching networks at ReLU activation layers with the stitching layer restrictedto elements of the group G ReLU showing in ﬁg. 1 that one can stitch CNNs on CIFAR-10 [Kri09]with only elements of G ReLU incurring less than ≈ 10% accuracy penalty at most activation layers.
This is surprisingly close to the losses found when one allows for a much more expressive linearlayer to be used to stitch two networks together.
However, we see that there remains a signiﬁcantgap between the stitching accuracies obtained using G ReLU and fully connected linear layers; thisprovides independent conﬁrmation of earlier ﬁndings that neurons of independently trained networksare not simply permutations of each other [Li+15; Wan+18].
It is also consistent with observedphenomena such as distributed representations in hidden features [GBC16, §15.4] and perhaps alsopolysemantic neurons [Ola+20].
Representation dissimilarity measures for G ReLU :
In section 5 we present two statistical dissimilarity measures, G ReLU -Procrustes and G ReLU -CKA, for ReLU -activated hidden features in differentnetworks, say A and B .
Our measures are counterparts of orthogonal Procrustes distance (see e.g.[DDS21b]) and Centered Kernel {+Alignment (CKA) 
+}
[Kor+19] respectively, which are invariant toorthogonal transformations, and are maximized when the hidden features of networks A and B agreeup to orthogonal transformations. In contrast G ReLU -Procrustes and G ReLU -CKA are invariant to G ReLU transformations, and are maximized when hidden features agree up to G ReLU transformations.
We compare and contrast our measures with their orthogonal counterparts, as well as with stitchingexperiment results.
Figure 2 shows a comparison of G ReLU and orthogonal CKA measures.
Impact of activation functions on interpretability: An intriguing ﬁnding from [Bau+17, §3.2]was that individual neurons are more interpretable than random linear combinations of neurons.
Our results on intertwiner groups (theorem 3.3) predict that this is a particular feature of ReLU1 with linear or RBF kernel.
[Figure] [Figure]
networks. Indeed, ﬁg. 3 shows that in the absence of ReLU activations interpretability does notdecrease when one moves from individual neurons to linear combinations of neurons — section 6describes this experiment in further detail.
This result suggests that intertwiner groups provide atheoretical justiﬁcation for the explainable AI community’s focus on individual neuron activations,rather than linear combinations thereof [Erh+09; Na+19; Yos+15; ZF14; Zho+15], but that thisjustiﬁcation is only valid for layers with certain activation functions.
Taken together, our experiments provide evidence that a network’s symmetries (realized throughintertwiner groups) propagate down to symmetries of a model’s internal representation of data.
Sinceunderstanding how different models process the same data is a fundamental goal in ﬁelds such asoexplainable AI and the safety of deep learning systems, we hope that our results will provide anadditional lens under which to examine these problems.
Related Work
The research on symmetries of neural networks is extensive, hence we aim to provide a representativesample knowing it will be incomplete.
[Bre+19; FB17; GBC16; Yi+19] study the effect of weightspace symmetries on the loss landscape.
On the other hand, [BMC15; GBC16; Kun+21; Men+19]study the effect of weight space symmetries on training dynamics, while [GBC16; RK20] show thatweight space symmetries pose an obstruction to model identiﬁability.
Neural stitching was introduced as a means of comparing learned representations between networksin [LV15].
In [BNB21] it was shown to have intriguing connections with the “Anna Karenina ” (high performance models share similar internal representations of data) and “more is better” (stitchinglater layers of a weak model to early layers of a model trained with more data/parameters/epochs canimprove performance) phenomena.
[Csi+21] considered constrained stitching layers by restricting therank of the stitching matrix or by introducing an (cid:96) 1 sparsity penalty — our methods are distinct in thatwe explicitly optimize over the intertwiner group for ReLU nonlinearities (permutations and scalings).
Both [BNB21; Csi+21] compare their stitching results with statistical dissimilarity measures suchas CKA.
Our G ReLU -Procrustes measure is a close relative of the permutation Procrustes distanceintroduced in [Wil+21], and our G ReLU -CKA is a an instance of CKA [Kor+19] in which the kernelis taken to be max { x 1 · y 1 , . . .
, x d · y d } .
Approaches to deep learning interpretability sometimes assume that the activation basis is special[Erh+09; Na+19; Yos+15; ZF14; Zho+15].
Studying individual neurons rather than linear combinations of neurons signiﬁcantly reduces the complexity of low-level approaches to the understanding ofneural networks [Elh+21].
In tension with this, many different projections of hidden layer activationsappear to be semantically coherent [Sze+14].
However, [Bau+17] found evidence that the hiddenfeature vectors closer to the coordinate basis align more with human concepts than vectors sampleduniformly from the unit sphere.
The Symmetries of Nonlinearities
Let Mat n 1 ,n 0 ( R ) be the algebra of all n 0 × n 1 real matrices and GL n ( R ) be the group of all invertiblen × n matrices.
Let σ : R → R be a continuous function.
For any n ∈ N , we can build a nonlinearityσ n from R n to R n by applying σ coordinatewise, i.e., σ n ( x 1 , . . .
, x n )
= ( σ ( x 1 ) , . . .
, σ ( x n )) .
Fixsome k > 1 and for each 1 ≤ i < k let (cid:96) i : R n i − 1 → R n i be the composition of an afﬁnelayer and a nonlinear layer, so that (cid:96) i ( x ) :
= σ n i ( W i x + b i ) , and let (cid:96)
k
( x ) :
=
W k x + b k .
Here
W i ∈
Mat n i ,n i − 1 ( R ) and b i ∈ R n i are the weights and bias of layer i respectively.
We deﬁnef : R n 0 → R n k to be the neural network f = (cid:96) k ◦ · · · ◦ (cid:96) 1 .
For each 1 ≤ i < k − 1 we can thendecompose f as f = f >i ◦ f ≤ i wheref ≤ i = (cid:96) i ◦ · · · ◦ (cid:96)and f >i = (cid:96) k ◦ · · · ◦ (cid:96) i +1 .
We deﬁne
W :=
( W i , b i | i = 1 , . . .
, k ) and W :
= k (cid:89) i =
(Mat n i ,n i − 1 ( R ) ×
R n i ) where the former is the collection of all weights of f and the latter is the space of all possible weightsfor a given architecture.
When we want to emphasize the dependence of f on weights W , we writef ( − , W ) (and similarly f ≤ i ( − , W ) , f >i ( − , W ) ).
One of the topics this work will consider is vector space bases for f ’s hidden spaces R n i , for1 ≤ i ≤ k − 1 .
We will investigate the legitimacy of analyzing features f ≤
i ( D ) for dataset D ⊂ R n 0with respect to the activation basis for R n i which is simply the usual coordinate basis, e 1 , . .
, e n iwhere e j
=
[ δ j(cid:96) ]
n i (cid:96) =1 is naturally parameterized by individual neuron activations.
Note that R n i hasan inﬁnite number of other possible bases that could be chosen.
3.1 Intertwiner Groups
For any 0 ≤ i < k , elements of GL n i ( R ) can be applied to the hidden activation space R n i bothbefore and after the nonlinear layer σ n i .
We deﬁne
G σ ni :=
{ A ∈ GL n i ( R ) | there exists a B ∈ GL n i ( R ) such that σ n i ◦
A = B ◦ σ n i } .
Informally, we can understand G σ ni to be the set of all invertible linear transformations whose actionon R n i prior to the nonlinear layer σ n i has an equivalent invertible transformation after σ n i .
This isan instance of the common procedure of understanding a function by understanding those operatorsthat commute with it.
For any A ∈ GL n i ( R ) , we can write σ ( A ) for the n i × n i matrix formed byapplying σ to all entries in A .
Lemma 3.1.
Suppose σ ( I n ) is invertible and for each A ∈ GL n ( R ) deﬁne φ σ ( A )
=
σ ( A ) σ ( I n ) − 1 .
Then G σ n is a group, φ σ : G σ n → GL n ( R ) is a homomorphism and σ n ◦ A
= φ σ ( A ) ◦ σ n .
We defer all proofs to appendix E. We include concrete examples of σ for small n i there as well. Activation
G σ n φ σ ( A ) σ ( x ) = x (identity)
GL n ( R )
A σ ( x ) =
e x
[Equation] + [Equation] x Σ n
A σ ( x ) = ReLU( x )
Matrices PD , where D has positive entries
A σ ( x ) =
LeakyReLU
( x )
Same as ReLU as long as negative slope (cid:54) =
A σ ( x ) =
[Equation]
(RBF)
Matrices PD , where D has entries in {± 1 } abs
( A ) σ ( x ) = x
d (polynomial)
Matrices PD , where D has non-zero entries
A (cid:12) d
Deﬁnition 3.2.
When the hypotheses of lemma 3.1 are satisﬁed (namely, σ ( I n ) is invertible)
we call
G σ n the intertwiner group of the activation σ n .
We denote the image of the homomorphism φ σ asφ σ ( G σ n ) .
The intertwiner group G σ n and φ σ are concretely described for a range of activations in table 1 —the last two examples motivate the generality of deﬁnition 3.2.
Note also that in both of those cases
A (cid:55)→ φ σ ( A ) is not a homomorphism on all of GL n ( R ) , but is a homomorphism when restricted tothe appropriate subgroup G σ n .
While a substantial part of table 1 can be found scattered in prior work,our calculations in appendix E.2 deal with the different cases of table 1 in a uniform way, by whatamounts to an algorithm that compute G σ n and φ σ given in terms of elementary properties of any(reasonable) activation function σ . 2 As design of activation functions remains an active industry (forexample [Elh+22]), our techniques for computing G σ n could be useful in future studies of networksymmetries.
The following theorem shows that the activation basis is intimately related to the intertwiner group of
ReLU : G ReLU admits a natural group-theoretic characterization in terms of the rays spanned by theactivation basis, and dually the rays spanned by the activation basis can be recovered from G ReLU .
While both its statement and proof are elementary, our interest in this theorem lies in the question ofwhether it could potentially provide theoretical justiﬁcation for focusing model interpretation studieson individual activations.
We investigate this question further in section 6.
Theorem 3.3.
The group G ReLU n is precisely the stabilizer of the set of rays { R ≥
e i ⊂ R n | i =1 , . . . , n } .
Moreover if R ≥ 0 v 1 , . . .
, R ≥ 0 v N ⊆ R n is a ﬁnite set of rays stabilized by G ReLU , then foreach v i
=
[ v i 1 , . .
. , v in ] T , it must be that v ij = 0 for all but one j ∈ { 1 , . . . , n } .
Equivalently up tomultiplication by a positive scalar every v i is of the form ± e j for some j .
3.2 Weight space symmetries
The intertwiner group is also a natural way to describe the weight space symmetries of a neuralnetwork.
We denote by F ⊆ C ( R n 0 , R n k ) the space of continuous functions that can be describedby a network with the same architecture as f .
As described in [JGH18] there is a realization mapΦ :
W → F mapping weights W ∈ W to the associated function f ∈ F .
Φ arises because thereare generally multiple sets of weights that yield the same function.
We will show that Φ is invariantwith respect to an action of the intertwiner groups on W so that intertwiner groups form a set of“built-in” weight space symmetries of f .
This result, which encompasses phenomena includingpermutation symmetries of hidden neurons, is well known in many particular cases (e.g., [GBC16,§8.2.2], [Bre+19, §3], [FB17, §2], [Men+19, §3], [RK20, §3, A]).
From Proposition 3.4 we can alsoderive corollaries regarding symmetries of the loss landscape — these are included in appendix E.6.
Proposition 3.4.
Suppose A i ∈ G σ ni for 1 ≤ i ≤ k − 1 , and let
[Equation]
Then, as functions, for each m
[Equation]
In particular, f ( x, W (cid:48) ) =
f ( x, W ) for all x ∈ R n 0 .
Equivalently, we have Φ( W (cid:48) )
=
Φ( W ) ∈ F .
Remark 3.6 .
We will show in appendix E that the statement of this theorem must be modiﬁed if thearchitecture of f contains residual connections.
By placing suitable restrictions on the matrices
A i 3we can recover a form of eq.
(3.5) provided m occurs at the end of a residual block.
However, theredoesn’t seem to be a way to obtain such an identity when m occurs inside a residual block; we seeempirical evidence consistent with this point in ﬁgs.
2, 6 and 9 below.
Intertwining group symmetries and model stitching
In this section we provide evidence that some of the differences between distinct model’s internalrepresentations can be explained in terms of symmetries encoded by intertwiner groups.
We do thisusing the stitching framework from [BNB21; Csi+21], and begin by reviewing the concept of networkstitching.
Suppose f, ˜ f are two networks as in section 3, with weights W , ˜ W respectively.
For any 1 ≤ l ≤ k − 1we may form a stitched network S ( f, ˜ f, l, ϕ ) :
R n 0 → R n k deﬁned in the notation of section 3by S ( f, ˜ f, l, ϕ ) =
˜ f >l ◦ ϕ ◦ f ≤ l – here ϕ : R n l → R n l is a stitching layer .
In a typical stitchingexperiment one trains networks f and ˜ f from different initializations and freezes their weights,constrains ϕ to some simple function class S (e.g., afﬁne maps in [BNB21]), and trains S ( f, ˜ f, l, ϕ ) byoptimizing ϕ alone.
The ﬁnal validation accuracy Acc S ( f, ˜ f, l, ϕ ) of S ( f, ˜ f, l, ϕ ) is then considereda measure of similarity (or lack therof) of the internal representations of f and ˜ f in R n l — in thisframework the situation
[Equation]
corresponds to high similarity since the hidden representations of model f and ˜ f could be related bya transformation S .
Recall that even though the networks f ( W ) and f ( W (cid:48) ) may be equal as functions , their hiddenrepresentations need not be the same (an example of this is given in appendix C).
Our next result showsthat in the case where f and ˜ f do only differ up to an element of G σ nl , eq.
(4.1) is achievable evenwhen the stitching function class S is restricted down to elements of φ σ ( G σ nl ) (see deﬁnition 3.2).
Theorem 4.2.
Suppose ˜ W
=
( A 1 W 1 , A 1 b 1 , A 2 W 2 φ σ ( A − 1 1 ) , A 2 b 2 , . . .
, W k φ σ ( A − 1 k ) , b k ) where
A i ∈ G σ ni for all i .
Then eq.
(4.1) is achievable with equality if the stitching function class Scontaining ϕ contains φ σ ( G σ nl ) .
Motivated by theorem 4.2, we attempt to stitch various networks at ReLU activation layers using thegroup G ReLU described in Figure 1.
Every matrix A ∈ G ReLU can be written as PD , where P isa permutation matrix and D is diagonal with positive diagonal entries — hence optimization over
G ReLU requires optimizing over permutation matrices.
We use the well-known convex relaxationof permutation matrices to doubly stochastic matrices and describe our optimization procedure ingreater detail in D.2.
Figure 1 gives the difference between the average test error of Myrtle CNN [Pag18] networks 4 f and˜ f and the network S ( f, ˜ f, l, ϕ ) ,
[Equation]
where S ( f, ˜ f, l, ϕ ) was stitched together at layer l via a stitching transformations ϕ that was eitheroptimized over all afﬁne transformations, reduced rank afﬁne transformations as in [Csi+21] ortransformations restricted to G ReLU .
We consider only the ReLU activation layers, as these are theonly layers where the theory of section 3 applies, and we only act on the channel tensor dimension— in practice, this is accomplished by means of 1-by-1 convolution operations.
In particular, with
G ReLU we are only permuting and scaling channels .
Lower values indicate that the stitching layer wassufﬁcient to translate between the internal representation of f at layer l and the internal representationof ˜ f .
We ﬁnd that when we learn a stitching layer over arbitrary afﬁne transformations of channels, wecan nearly achieve the accuracy of the original models.
When we only optimize over G ReLU thereis an appreciable increase in test error difference. This is consistent with ﬁndings in [Csi+21; Li+15; Wan+18] discussed in section 2, and also consistent with observations that hidden featuresof neural networks exhibit distributed representations and polysemanticism [Ola+20].
Nonetheless,that S ( f, ˜ f, l, ϕ ) is able get within less than 10% of the accuracy of f and ˜ f in all but one layersuggests that elements of G ReLU can account for a substantial amount of the variation in the internalrepresentations of independently trained networks.
We include the reduced rank transformations asthe dimension of their parameter spaces is greater than that of G ReLU , and yet they incur signiﬁcantlyhigher stitching penalties.
If n l is the number of channels, we have dim G ReLU nl = n l whereas thedimension of rank r transformations is 2 n l · r − r 2 (hence greater than dim G ReLU nl even for r = 1 ).
Finally, in the speciﬁc case of the Myrtle CNNs the stitching penalties incurred when using any layerother than 1-by-1 convolution with a rank 1 matrix all follow similar trends: they increase up to thethird activation layer, then decrease at the ﬁnal activation layer.
Further stitching results on the ResNet20 architecture can be found in appendix D.3.
Dissimilarity measures for the intertwiner group of ReLU
Stitching penalties can be viewed as task oriented measures of hidden feature dissimilarity.
Froma different perspective, we can consider raw statistical measures of hidden feature dissimilarity.
In the design of measures of dissimilarity, a crucial choice is the group of transformations underwhich the dissimilarity measure is invariant.
For example, Centered Kernel Alignment ( CKA )[Kor+19] with the dot product kernel is invariant with respect to orthogonal transformations andisotropic scaling.
We ask for a statistical dissimilarity metric µ on datasets X, Y ∈ R N × d with theproperties that (0) 0 ≤ µ ( X, Y ) ≤ 1 , (i) ( G ReLU -Invariance )
If A, B ∈ G ReLU d and v, w ∈ R dthen µ ( XA + 1 v T , Y B + 1 w T ) = µ ( X, Y ) , and (ii) ( Alignment Property ) µ ( X, Y )
= [-
-]1 if ( ∗ )
Y = XA + 1 v T for some A ∈ G ReLU d , v ∈ R d .
To motivate this question, we note that givensuch a metric µ , one can detect if X and Y do not differ by an element of G ReLU by checking ifµ ( X, Y ) <
Our basic tool for ensuring (i) is the next lemma.
Lemma 5.1.
Suppose µ ( XA, Y B ) =
µ ( X, Y ) if A, B are either positive diagonal matrices orpermutation matrices.
Then, (i) holds.
In effect, this allows us to divide the columns of X and Y by their norms to achieve invariance tothe action of positive diagonal matrices and then apply dissimilarity measures for the permutationgroup such as those presented in [Wil+21].
Ensuring (ii) seems to require case-by-case analysis todetermine an appropriate normalization constant.
Deﬁnition 5.2 ( G ReLU -Procrustes) .
Let D X =
diag( | X [: ,i ] | ) and D Y = diag( | Y
[: ,i ] | ) .
Assumingthese are invertible, let ˜ X
=
XD − 1 X and ˜ Y
=
Y D − 1 Y .
Let δ be the permutation Procrustes distancebetween ˜ X, ˜ Y , deﬁned by δ :=
min P ∈ Σ d | ˜ X − ˜ Y P
| (as pointed out in [Wil+21] this can be computedvia the linear sum assignment problem).
Then the G ReLU -Procrustes measure isµ Procrustes ( X, Y ) : = 1 −
[Equation]
The factor of 2 √ d ensures this lies in [0 , 1] , and equals 1 if (and only if) the condition ∗ of (ii) holds.
[Table]
Conﬁdence intervals were obtained by evaluating similarities for 32 pairs of independently initialized models. For more details on how we sample models we refer
We apply G ReLU -Procrustes and orthogonal Procrustes similarities to 4 different hidden representations from Myrtle CNNs in table 2 and many more layers of ResNet20s in ﬁg. 9, all trained [Figure]
on CIFAR-10 [Kri09].
In keeping with the discussion of section 4, we only consider permutations or orthogonal transformations of channels (for details on how this is implemented we referto appendix D.6).
We see that distinct representations register less similarity in terms of G ReLUProcrustes than they do in terms of orthogonal Procrustes.
This makes sense as similarity up to G ReLU -transformation requires a greater degree of absolute similarity between representations thanis required of similarity up to orthogonal transformation (the latter being a higher-dimensional groupcontaining all of the permutations in G ReLU ).
Otherwise patterns in G ReLU -Procrustes similaritiylargely follow those of orthogonal Procrustes, with similarity between representations decreasing asone progresses through the network, only to increase again in the last layer.
This correlates with thestitching penalties of ﬁg. 1, which increase with depth only to decrease in the last layer.
Deﬁnition 5.3 ( G ReLU -CKA) .
Assume that X and Y are data matrices that have been centered bysubtracting means of rows: X ← X − 1 d 11 T X and Y ← Y − 1 d 11 T Y .
Let ˜ X
=
XD − 1 X and ˜ Y =
Y D − 1 Y .
Let ˜ x 1 , . . .
, ˜ x
N be the rows of ˜ X , and similarly for ˜ Y .
Form the matrices K, L ∈ R N × N ≥ 0deﬁned by K ij =
max(˜ x i (cid:12) ˜ x j ) and L ij =
max(˜
y i (cid:12) ˜
y j ) where (cid:12) is the Hadamard product.
Thenthe G ReLU -CKA for X and Y is deﬁned as:
[Equation]
where HSIC 1 is the unbiased form of the Hilbert-Schmidt independence criterion of [NRK21, eq. 3].
Symmetry of the max function ensures (i), the Cauchy-Schwarz inequality ensures µ CKA ( X, Y ) ∈[0 , 1] , and we claim that µ CKA ( X, Y )
= 1 if the condition ∗ of (ii) is met.
We do not claim‘if and only if’, however we point out the following in lemma E.44: if A is a matrix such thatmax( Ax 1 (cid:12) Ax 2 )
= max( x 1 (cid:12) x 2 ) for all x 1 , x 2 ∈ R d , then A is of the form PD where P is apermutation matrix and D is diagonal with diagonal entries in {± 1 } .
In fact, µ CKA is simply aninstance of CKA for a the “ max kernel.”
Lemma 5.5.
The function κ : R d × R d → R deﬁned by k ( x, y ) =
max( x (cid:12) y ) is a positivesemi-deﬁnite kernel.
As with CKA [Kor+19], this metric makes sense even if X, Y are datasets in R d , R d (cid:48) respectively withd (cid:54) =
d (cid:48) .
Results for a pair of Myrtle CNNs independently trained on CIFAR-10, as well as standardorthogonal CKA for comparison, are shown in ﬁg. 4.
Analogous results for ResNet20s are shown inﬁg.
We ﬁnd that G ReLU -CKA respects basic trends found in their orthogonal counterparts: modellayers at the same depth are more similar, early layers are highly similar, and the metric surfaces theblock structure of the ResNet in ﬁg. 2 (layers inside residual blocks are less similar than those atresidual connections).
One notable difference for G ReLU -CKA in ﬁgs. 2 and 4 is that the similaritydifference between early and later layers shown in (b) is less pronounced in (a), and in fact later layers are found to be less similar betweenruns
We found similar results for stitching in ﬁgs.
1 and 6.
Interpretability of the coordinate basis
In this section we explore the conﬂuence of model interpretability and intertwiner symmetries usingnetwork dissection from [Bau+17].
Network dissection measures alignment between the individualneurons of a hidden layer and single, pre-deﬁned concepts (see appendix F.1 for the methodology).
We use an experiment from [Bau+17] to compare the axis-aligned interpretability of hidden activationlayers with and without an activation function.
Bau et al. compares the interpretability of individualneurons, measured via network dissection, with that of random orthogonal rotations of neurons.
Welikewise rotate the hidden layer representations and then measure their interpretability.
Using themethodology from [Dia05], we deﬁne a random orthogonal transform Q drawn uniformly from
SO ( n ) by using Gram-Schmidt to orthonormalize the normally-distributed QR =
A ∈ R n 2 .
Like in[Bau+17], we also consider smaller rotations Q α ∈ SO ( n ) where 0 ≤ α ≤ 1 , where α is chosen toform a minimal geodesic rotating from I to Q .
[Bau+17] found that the number of interpretable unitsdecreased away from the activation basis as α increased for layer5 of an AlexNet.
We compare three models trained on ImageNet: a normal ResNet-50, a modiﬁed ResNet-50 where weremove the ReLU on the residual outputs (training details in appendix F.3), and a ConvNeXt [Liu+22]analog of the ResNet-50, which also does not have an activation function before the ﬁnal residualoutput.
We give results in ﬁg. 3, and provide sample unit detection outputs and full concept labelsfor the ﬁgures in appendix F.2.
As was shown in [Bau+17], interpretability decreases as we rotateaway from the axis for the normal ResNet-50 in appendix F.3.
On the other hand, with no activationfunction and therefore no canonical basis, neuron interpretability does not drop with rotation forthe modiﬁed ResNet-50 and the ConvNeXt.
Interestingly, while the number of interpretable unitsremains constant for the residual output of the modiﬁed ResNet-50, for the ConvNeXt model itactually increases .
We ﬁnd similar results, where the number of interpretable units increase withrotation, for the convolutional layer inside the residual block for the modiﬁed ResNet-50 in ﬁg. 18.
Limitations
Our theoretical analysis does not account for standard regularization techniques that are known to havesymmetry-breaking effects (for example weight decay reduces scaling symmetry).
More generally,we do not account for any implicit regularization of our training algorithms.
As illustrated in ﬁgs.
1and 6, stitching with intertwiner groups appears to have signiﬁcantly more architecture-dependentbehaviour than stitching with arbitrary afﬁne transformations (however, since different architectureshave different symmetries this is to be expected).
Our empirical tests of the dissimilarity measuresin section 5 are limited to what [Kor+19] terms “sanity tests”; in particular we did not perform thespeciﬁcity, sensitivity and quality tests of [DDS21a].
Conclusion
In this paper we describe groups of symmetries that arise from the nonlinear layers of a neuralnetwork.
We calculate these symmetry groups for a number of different types of nonlinearities, andexplore their fundamental properties and connection to weight space symmetries.
Next, we provideevidence that these symmetries induce symmetries in a network’s internal representation of the datathat it processes, showing that previous work on the internal representations of neural networks canbe naturally adapted to incorporate awareness of the intertwiner groups that we identify.
Finally,in the special case where the network in question has ReLU nonlinearities, we ﬁnd experimentalevidence that intertwiner groups justify the special place of the activation basis within interpretable
AI research.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 7.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
This
paper is largely focused on the mathematical aspects of deep learning so we do not
think there are any immediate negative societal impact to the methods described.
From
a broader perspective though, we see this work helping to create a more principled
groundwork for many interpretable AI techniques.
We explain why this could have
positive societal impacts in Section A.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] All proofs can be
found in appendix E.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[TODO]
We are in the process of making code publicly available.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See appendix D.
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See appendix D — while we did
not keep precise track of CPU/GPU hours, we do specify the hardware used.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] See Section G.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Societal Impact
Though deep learning models are in the process of being deployed for safety critical applications, westill have very little understanding of the structure and evolution of their internal representations.
Inthis paper we discuss one aspect of these representations.
We hope that by better illuminating theinner workings of these networks, we will be a small part of the larger effort to make deep learningmore understandable, reliable, and fair.
B Code availability
We are in the process of releasing code — once available it will be located athttps://github.com/TvfBaRoZCKqJ/FjEsAbqqjdeY.
C Examples
We ﬁrst give an example of two networks with distinct weights which are functionally equivalent.
Let f be a 2 layer network with ReLU activations and weight matrices
W 1 = (cid:20) 10(cid:21) and W 2 = (cid:20) 30(cid:21) (and biases = 0).
Let ˜ f be a network with the same architecture, but with weights
W 1 = (cid:20) 01(cid:21) and W 2 = (cid:20) 01(cid:21) .
Then one can verify that ˜ f
( x ) =
f
( x ) for all x ∈ R , but that the weights of f and ˜ f differ.
We also work through a small example of φ σ n where n = 2 .
Assume that σ is the ReLU nonlinearity.
Then,
A = (cid:20) 02(cid:21) belongs to G σ 2 , and we can compute directly that
ReLU ◦ (cid:20) 02(cid:21) (cid:20) xx(cid:21) =
(cid:20) ReLU( x 2 )
ReLU(2 x 1 ) (cid:21) =
(cid:20) ReLU( x 2 ) 2 ReLU( x 1 ) (cid:21) , where in the last equality we used the fact that ReLU( ax ) =
a ReLU( x ) when a is positive.
On theother hand,
[Equation]
D Experimental Details
In this section we provide additional experimental results, as well as implementation details for thepurposes of reproducibility.
All experiments were run on an Nvidia GPUs using PyTorch [Pas+19].
D.0.1 A “sanity test” for intertwiners
We include a simple experiment related to section 3.
In this experiment, we take an even simpler
Myrtle CNN (with no batch normalization) trained for 50 epochs, ﬁx a pre-activation layer l , andapply a transformation
A to the weights W l and biases b l to obtain AW l and Ab l (again, we only acton channels, hence in practice this is implemented by an auxiliary 1-by-1 convolution layer).
Weconsider 2 choices of A :
• A random element of G ReLU where we generate P via a random permutation and we
generate D via a diagonal matrix with random lognormal entries (normalized to ensure that
the expected value of D is the identity).
• A random orthogonal matrix, generated using the QR decomposition as in section
Next, we freeze layers up to and including l and ﬁnetune the later layers for another 50 epochs.
Based on the theory in section 3, when A ∈ G ReLU the network should be able to recover reasonableperformance even with the transformed features — for example, by updating W l +1 to W l +1 φ σ ( A ) − 1 .
On the other hand, with probability 1 there is no matrix B such that updating W l +1 to W l +1 Bcounteracts the effect of an orthogonal rotation A on all possible input.
We see that this is indeed thecase in 5: transforming by A ∈ G ReLU rather a random orthogonal matrix results in
D.1 Sampling pairs of independently initialized models
We began by training 100 independently initialized models for each of the following architectures:
(i) Myrtle CNN: a simple 5-layer feed-forward CNN with batch normalization.
(ii) ResNet20: a ResNet tailored to the CIFAR-10 dataset (numbers of channels are 16 , 32 , 64respectively in the 3 residual blocks).
(iii) ResNet18: an ImageNet-style ResNet adapted to the input size of CIFAR-10 — much widerthan the above (numbers of channels are 64 , 128 , 256 respectively in the 3 residual blocks).
More detailed architecture schematics are included in ﬁgs. 25a, 26a and 27a.
All models were trained for 50 epochs using the Adam optimizer with PyTorch’s default settings.
We use a batch size of 32, initial learning rate 0 .
001 and 4 evenly spaced learning rate drops withfactor 0 .
We augment data with translations of up to 2 pixels (padded as necessary with the mean
RGB value for CIFAR
Training this many CIFAR-10 models on a reasonable budget of time and computing resources wasgreatly aided by the excellent FFCV library [Lec+22].
In the later stitching and dissimilarity measure experiments, we sample pairs of models from these“zoos” uniformly with replacement (but of course making sure that the two models in the pair aredistinct).
Thus the cost of training hundreds of models is amortized across many runs of stitchingand dissimilarity measurement; this can be also viewed as bootstrap estimation of our experimentalquantities of interest using empirical samples from certain distributions of CIFAR-10 models.
D.2 Stitching Experiments
For stitching layers, we train for 20 epochs with batch size 32 and learning rate 0 .
001 (with no drops),however we use vanilla SGD with no momentum (we found the approximate second-order and/ormomentum aspects of Adam interacted in complicated ways with the PGD algorithm described inappendix D.2.1 below, even after following some helpful advice from the Internet 5 ).
Augmentation isdescribed in the previous paragraph.
We parameterize reduced rank 1-by-1 convolutions as a composition of 2 1-by-1 convolutions, within _ channels , out _ channels = in _ channels , rank and rank , in _ channels respectively.
Incontrast to [BNB21] we omit both batch norm and bias from stitching layers (to stick closely to thestatement of theorem 4.2).
D.2.1 Approximate Optimization over Permutation Matrices
By far the most complicated stitching layer is the one using G ReLU , which we describe here.
Recallthat G ReLU is equal to the n × n matrices of the form PD , where P ∈ Σ n is a permutation matrixand D is a diagonal matrix with positive entries We parameterize D simply as D = diag( λ i ) whereλ 1 , . . .
, λ n l ∈ R ≥ 0 — we preserve non-negativity during training by a projected gradient descentstep D ← ReLU( D ) .
During stitching layer training, we parameterize P as a doubly stochasticmatrix, that is, an element of the Birkhoff polytope
[Equation]
[Equation] and A 1 = 1 } — after each gradient descent step we project P back onto B by the operation P ← ReLU( P )followed by P ← sink( P ) , where “ sink ” denotes Sinkhorn iterations.
These consist of T iterationsof
A ← A diag
[Equation]
A
[Equation] − 1 followed by A ← diag( A 1 ) [Equation] 1 A (it is a theorem of Sinkhorn that this sequence converges to a doubly stochastic matrix of the form
DAE with D, E positive diagonal matrices [Sin64]).
We use T = 16 in all experiments (this choicedrew on the work of [Men+18]).
In addition, we add a regularization term − α | P | 2 to the stitchingobjective, where α > 0 is a hyperparameter (the motivation here is that permutation matrices areprecisely the elements of B with maximal (cid:96) 2 -norm).
Unless stated otherwise in our experimentsα = 0 .
We did experiment with choosing α by cross validation and found the particular choice ofα was not crucial; see appendix D.5 for further details.
At evaluation time, we threshold P to an actual permutation matrix via the Hungarian algorithm(speciﬁcally its implementation in scipy .
optimize.linear_sum_assignment [Vir+20]).
This amountsto
[Equation]
As stated above, we train for 20 epochs with batch size 32 and learning rate 0 .
001 (with no drops),using SGD with momentum 0.9.
However, we allow the permutation factor to get a “head start” bykeeping D ﬁxed at the identity I for the ﬁrst 10 epochs.
This is probably not essential, as shown inappendix D.5.
Finally, before evaluating the stitched model on the CIFAR-10 validation set, we perform a nogradient epoch on the training data with stitching layer P eval .
This is critical as it allows the batchnormalization running means and variances in later layers to adapt to the thresholded permutationmatrix P eval ; observe that if we omitted this step, during evaluation the “batch normalization layers”would not even be performing batch normalization per se, since their running statistics would becomputed from features produced by a layer P train no longer in use.
As an aside, we also experimented with the differeniable relaxation of permutation matrices SoftSort[PE20].
Our ﬁnal results were comparable, however this method took far longer ( > 10 × ) to optimizethan the Birkhoff polytope method.
It is perhaps of interest that we used SoftSort on permutationsfar larger than those of [PE20] (e.g., the 512 channels of late layers of our Myrtle CNN).
The nextsection (appendix D.2.2) contains some of our technical ﬁndings.
We wish to aknowledge a couple articles, [Fog+13] and [LW14], that provided us with usefulbackround on optimization over doubly stochastic matrices.
D.2.2 Stitching with SoftSort
We parameterized D simply as D = diag( e λ i ) where λ 1 , . . .
, λ n l ∈ R .
During stitching layertraining, we parameterized P using SoftSort [PE20], a continuous relaxation of permutation matricesgiven by the formula
P = SoftSort( s, τ ) :
= softmax (cid:0) − 1 τ (sort( s ) 1 T − 1 s T ) (cid:1) , where s ∈ R n l , sort( s ) denotes s sorted in descending order, and softmax is applied over rows.
The parameterτ > 0 controls softmax temperature, and we were only able to obtain reasonable results when tuningit according to τ ≈ 1 /n l .
At validation time, we threshold P to an actual permutation matrix byapplying arg max over rows as in [PE20].
D.3 Stitching and G ReLU -dissimilarity measures for ResNets
Here we include further results for ResNet20 and ResNet18 architectures.
Figure 6 and ﬁg. 7 includeresults for full 1-by-1 convolution, reduced randk 1-by-1 convolution and G ReLU 1-by-1 convolutionsstitching in the ResNet20 and ResNet18 architectures respectively.
Note that in general, layers insideresidual blocks incur higher penalties, consisent with remark 3.6.
This holds even in the full 1-by-1convolution case, a ﬁnding that to the best of our knowledge is new.
In the case of ResNet20 we also observe that the relative ranking of the different stitching constraintstends to change inside of residual blocks: whereas G ReLU stitching consistently outperforms rank 1(and sometimes rank 2) stitching outside residual blocks, it consistently underperforms all strategiesinside residual blocks.
Lastly, we remark that the ResNet20 is signiﬁcantly narrower than the Myrtle
CNN (channels are 16, 32, 64 vs. 64, 128, 256, see ﬁgs. 26a and 27a), and hence the low-ranktransformations account for a larger proportion of the available total rank (for example, in early layersof the ResNet20 rank 4 is 0 .
25 · fullrank whereas in the early layers of the Myrtle CNN rank 4 is0 .
0625 · fullrank ).
Heuristically, in the narrower network low-rank transformations may sufﬁce toalign for a larger fraction of the principal components of hidden features.
We also observe generally lower stitching penalties in the ResNet18 with the exception of thepenultimate inside-a-residual-block layer — we do not have a satisfactory explanation for randomchance performance at that layer.
We also remark that while the penalties in ﬁg. 6 are signiﬁcantlyhigher than those in ﬁg. 1, especially in later layers, we also saw signiﬁcant dissimilarity in ﬁg. 2 (a),especially in later layers.
We also modify the ResNet20 to use the LeakyReLU activation function and train models withdifferent negative slopes s .
Accuracy for two independently trained models at different LeakyReLUgiven in table 3.
We perform G ReLU stitching in ﬁg. 6.
Note that for a negative slope s = 1 , theactivation function is the identity.
We ﬁnd the results difﬁcult to interpret due to the signiﬁcantdecrease in CIFAR-10 accuracy for larger s .
With this being said, unlike for s << 1 , we note thatthe stitching penalties for s = 1 (and to a lesser extent, s = 0 . 9 ) are mostly constant throughout thelayers of the network.
This is most prominent for the ﬁnal two ResNet20 layers ( 72 and 75 ), wherethe stitching penalty for models with small LeakyReLU slopes is the lowest.
Figure 9 contains G ReLU and orthogonal Procrustes dissimilarities for the ResNet20.
The 2 measuresseem qualitatively quite similar in this case.
For the most part the same applies to the ResNet18 inﬁg.
10, with the exception of layer 70 (penultimate inside-a-residual-block layer), where we see high
G ReLU similarity , in conﬂict with both ﬁg. 7 and ﬁg. 11 below.
[Table]
We include G ReLU and orthogonal CKA dissimilarities for the wider ResNet18 in ﬁg. 11.
For themost part the qualitative remarks on ﬁg. 2 apply here as well — note also the extreme dissimilarity inlayer 70 (in both G ReLU and orthogonal cases) consistent with ﬁg. 7.
D.4 Stitching for a Vision Transformer
Here we include an additional stitching experiments with vision transformers from [Has+21] trainedon CIFAR-10.
Figure 12 include results for linear stitching and G ReLU stitching after each transformerencoder layer.
The large stitching penalties for G ReLU are expected due to the lack of activationfunctions after the linear (feedforward) layers for each encoder layer.
We train 10 Compact Convolutional Transformers with sinusoidal positional encodings and sixtransformer blocks.
The average model accuracy was 98% using the distributed training-from-scratchrecipe from [Has+21], which includes 6e − 2 weight decay, augmentations (namely mixup [Zha+18]
[Figure]
Accuracy of the independently initialized models with different LeakyReLU slopes is given in table 3.
Conﬁdence and CutMix [Yun+19]), label smoothing, and AdamW with a learning rate of 55e − 5 with cosinescheduling.
D.5 Choosing the negative- (cid:96) 2 regularization multiplier α with cross validation
Here we brieﬂy describe an experiment in which the multiplier α of appendix D.2.1 is chosen by crossvalidation.
Most of the details are as in appendix D.2.
However, we create a random 80-20 split of the
CIFAR10 training set into a smaller training and cross-validation set.
We then learn G ReLU stitchinglayers for each α ∈ { 10 k | k
= − 3 , − 2 , . .
, 1 } , as in appendix D.2.1, with the exception that weonly optimize over our training split for 5 epochs and do not give the permutations a head start.
Then,the α corresponding to highest accuracy on our cross validation set is selected, the correspondingmodel weights are loaded and we report accuracy on the regular CIFAR10 validation set.
In ﬁg. 13we obtain very similar results to those in ﬁg. 1.
Perhaps more interestingly, in ﬁg. 14 we see thatthere is substantial variance in the α selected by cross-validation, at all layers of our Myrtle CNNnetwork — for reference, α = 0 .
1 is used in the rest of this paper.
This suggests that the particularchoice of α is not essential to our method.
Results for ResNet architectures are qualitatively similar
[Figure]
D.6 Implementing dissimilarity measures
As mentioned in section 5, we aim to capture invariants to permuting and scaling channels, but notspatial coordinates.
This requires some care; practically speaking it means we cannot simply ﬂattenfeature vectors.
In all cases we compute our measures over the entire CIFAR-10 validation set.
In particular, we donot require batched computations as in [NRK21].
D.6.1 Procrustes
As in [DDS21b; Wil+21]
[Equation]
so it sufﬁces to consider minimizing the Frobenius norm-squared, and expanding as
[Equation]
we see that this is equivalent to maximizing tr( ˜ X T ˜ Y P ) .
In our case X, Y have shape ( N, C, H, W )where N is the size of the entire CIFAR-10 validation set and C, H, W are the channels, height, andwidth at the given hidden layer respectively.
We want P to be a C × C permutation matrix.
Hencefor ˜ X T ˜
Y we compute the tensor dot product( ˜ X T ˜ Y ) c [Equation] c
[Equation]
= (cid:88) n,h,w ˜
[Equation]
w
[Equation]
n
[Equation]
c (cid:48) ,h,w (D.1)
The same method is used for orthogonal
Procrustes, where instead ofscipy .optimize .linear_sum_assignment
we use the nuclear norm of eq.
(D.1) as in [DDS21a].
D.6.2 CKA
In this case for a set of hidden features X of shape ( N, C, H, W ) as above, we ﬁrst subtract the meanover all but the channel dimension:
[Equation]
and divide by the norms over all but the channel dimension:
[Equation]
n,c,
[Equation]
←
[Equation] Next, we compute a tensor dot product of X with itself over spatial dimensions , to obtain the shape( N, N, C ) tensor [Equation] n
[Equation]
,h,w and ﬁnally we apply max over the channel dimension to get
[Equation]
[Table]
Remark D.2 .
It could be interesting to refrain from applying a dot product over spatial dimensions,and thus measure not only similarity between hidden features of different images, but similaritybetween hidden features of different images at certain locations .
However, the memory requirementswould have been far beyond our computational limits.
D.7 Dissimilarity measures for network with constant channel width
A notable feature of our plots in ﬁgs. 2, 4 and 11 is that the G ReLU -CKA exhibits a much moresigniﬁcant decay with network depth than its orthogonal counterpart.
From a skeptical perspective,we thought this could have something to do with dimensionality.
All the networks we looked at up tothis point had the feature that their channel dimension grows exponentially with depth (as seen inthe last 3 ﬁgures of the appendix).
When we compute the kernels max(˜ x _ i (cid:12) ˜ x _ j ) , we encountermaxima of larger and larger sets of random variables as the channel dimension increases. If theproducts inside these maxima were independent normal random variables (we are not claiming thisis a reasonable heuristic), we’d expect the max to grow like Φ − 1 (1 − 1 n l ) where n l is the channeldimension. It seemed possible that something along these lines could cause G ReLU -CKA to drift asdepth (in our experiments correlated with channel dimension) increases. Note that the dot productkernel seems comparatively immune, since (with the same heuristics of normal distribution) theexpected value of (cid:104) ˜ x, ˜ y (cid:105) is 0 regardless of dimension.
Motivated by this train of thought, we evaluated all 4 dissimilarity measures of section 5 on a variantof our Myrtle CNN with constant channel dimension .
The architecture of this network is identical tothe one shown in ﬁg. 25a with the exception that all channel dimensions are 512.
In table 4 and ﬁg.
15we see that these constant width CNNs exhibit qualitatively very similar dissimilarity measures astheir non-constant width counterparts.
This suggests that the G ReLU -CKA decay with network depthis not an artifact of increasing channel dimension. We speculate that it’s possible that the decay of G ReLU -CKA is due to something like the superpositionhypothesis for hidden layer features [Elh+22; Ola+20]. Roughly, in overcomplete cases where themodel can use more features than basis directions in a hidden layer, it may be encoding m nearlyorthogonal features across n < m basis directions. If this encoding is not consistent across randomseeds, we expect G ReLU -CKA to be smaller.
Finally, polysemanticism may increase with depth. Ina simple thought experiment, if each basis direction in layer l has a features encoded, layer l + 1will have 2 a features per direction if it each neuron in l + 1 simply sums over two neurons in l . Again assuming the combinations of features occuring in this polysemanticism vary accross randomseeds, we would expect G ReLU -CKA to be smaller. Simply put, superposition and polysemanticismwould seem to preclude alignment of the hidden features of different networks with permutations andscaling alone.
E Proofs
E.1 A proof of lemma 3.1, plus some abstractions thereof
Proof of lemma 3.1.
Since by deﬁnition G σ n ⊆ GL n ( R ) , to prove G σ n is a subgroup it sufﬁcesto show that if A 1 , A 2 ∈ G σ n then A 1 A − 1∈ G σ n .
By hypotheses, there are matrices B 1 , B 2 ∈
GL n ( R ) so that
[Equation]
and
[Equation]
E
[Equation]
[Equation]
[Figure]
Applying A − 1 2 on the right hand side of eq.
(E.1) gives
[Equation]
On the other hand, applying A − 1 2 on the right hand side of eq.
(E.2) gives σ n = B 2 ◦ σ n ◦
( A − 1 2 ) andhence
[Equation]
Combining eqs.
(E.3) and (E.4) we obtain
[Equation]
and hence G σ n is a subgroup.
Next, we solve
[Equation]
for B in terms of A by evaluating both sides at e 1 , . . .
, e n ∈ R n (standard basis vectors).
Letting
A [: , j ] denote the j -th column of A we obtainσ n ( A [: , j ]) =
Bσ n ( e j ) , for j = 1 , . . . , n and stacking these columns to obtain the full n × n matrix yields
[Equation]
where I ∈ GL n ( R ) is the identity matrix and σ ( A ) denotes σ applied to the coordinates of A(similarly for σ ( I ) ).
As σ ( I ) is invertible by hypotheses, this implies B = σ ( A ) σ ( I ) − 1 =: φ σ ( A ) ,and ﬁnally substituting B i = φ σ ( A i ) for i
= 1 , 2 in eq.
(E.5) shows that
[Equation]
while at the same time
[Equation]
so that φ σ ( A 1 A − 1 2 ) ◦ σ n = φ σ ( A 1 ) φ σ ( A 2 ) − 1 ◦ σ n .
Using the invertibility of σ ( I ) one more time,we conclude
[Equation]
which implies φ σ is a homomorphism.
Remark E.6 (for the mathematically inclined reader) .
Here is a more abstract deﬁnition of G σ n thatmakes lemma 3.1 appear more natural: let X be a topological space with a continuous (left) actionof a topological group G .
There is a natural (right) action of G on C ( X, R ) by precomposition( ( f, g ) (cid:55)→ f ◦ g ).
For any subspace V ⊆ C ( X, R ) deﬁne
G V := { g ∈ G | V · g ⊆ V } (that is, the elements of G stabilize V as a subspace , but not necessarily pointwise — one can showthis is always a subgroup of G ).
Then, for every such subspace V , the group G V acts linearly on V , and if we have a basis f 1 , . . .
, f n ∈ V , we can obtain a matrix representation of G in GL n ( R ) .
Toobtain the special case in lemma 3.1, we take X = R n , G = GL n ( R ) with the usual action, and
V to be the subspace spanned by the functions
f i ( x 1 , . . .
, x n )
= σ ( x i ) .
The condition that σ ( I ) isinvertible is equivalent to the condition that the column space of the matrix ( f i ( e j ) is n -dimensional,which in turn implies V is n -dimensional.
We end this section with a lemma that allows for easy veriﬁcation that σ ( I ) is invertible.
We usedthis on all the activation functions considered in table 1.
Lemma E.7.
Let σ : R → R be any function and let I ∈ GL n ( R ) be the identity matrix.
Then σ ( I )is invertible provided
[Equation]
Proof.
Let N = 11 T − I .
Then
[Equation]
Note that the eigenvalues of 11 T are n (corresponding to eigenvector 1 ) and 0’s (corresponding tothe orthogonal complement of 1 ).
For any linear operator A ∈ Mat n ( R ) with eigenvector/eigenvaluepair ( v, λ ) , v is easily seen to be an eigenvector of A − I with eigenvalue λ − 1 .
Hence the eigenvaluesof N are n − 1 and − 1 , and it follows that the eigenvalues of σ ( I ) are
[Equation]
which are all non-zero if and only if eq.
(E.8) holds.
Remark E.9 .
In particular eq.
(E.8) holds when σ (1)
= 1 , σ (0) =
(which holds for example whenσ ( x ) = ReLU( x ) or σ ( x ) =
x d ).
In this situation, σ ( I ) =
I and φ σ ( A ) =
σ ( A ) (coordinatewiseapplication of σ ).
For example, if σ ≥ 0 is non-negative, then φ σ ( A )
=
σ ( A ) has non-negativeentries.
E.2 Calculating intertwiner groups (for table 1)
We begin with two lemmas:
the ﬁrst puts a “lower bound” on G σ n and the second is a “differentialform” of the deﬁnition of the intertwiner group from section 3.
Together, these two results effectivelyallow us to reduce calculation of intertwiner groups to the n = 1 case.
Lemma E.10 (cf.
[GBC16, §8.2.2], [Bre+19, §3]) .
G σ n always contains the permutation matricesΣ n , and φ σ restricts to the identity on Σ n .
Proof.
If A ∈ Σ n is a permutation matrix, so that Ae i =
e π ( i ) where π is a permutation of { 1 , . . . , n }then for any x ∈ R n we observe
[Equation]
which is exactly σ applied coordinatewise to (cid:80)
i x i e π ( i ) =
Ax .
Corollary E.11.
If A ∈ GL n ( R ) , P ∈ Σ n , and AP ∈ G σ n or PA ∈ G σ n , then A ∈ G σ n .
Proof.
If B = AP ∈ G σ n , then A = BP − 1 , where B ∈ G σ n by hypothesis and P ∈ G σ n bylemma E.10.
The result follows as G σ n is a group (lemma 3.1) and hence closed under multiplication.
The other case is similar.
Lemma E.12.
Suppose A, B ∈ GL n ( R ) and σ n ◦
A = B ◦ σ n .
Suppose x = ( x 1 , . . .
, x n ) T ∈ R nand assume σ is differentiable at x 1 , . . .
, x n as well as( Ax ) i = (cid:88)
j a ij x j , for i = 1 , . .
n.
Then,diag
[Equation]
) (here diag : R n → Mat n × n ( R ) takes a vector to a diagonal matrix).
Explicitly, for each i, j ∈{ 1 , . . . , n }
[Equation]
(E.13)
Proof.
By the chain rule [Rud76, Thm. 9.15], and since the differential of a matrix is itself,
[Equation]
Finally, by the deﬁnition of σ n
[Equation]
=
[Equation]
if i = j otherwise .
Theorem E.14.
Suppose σ is non-constant, non-linear, and differentiable on a dense open set withﬁnite complement.
Then,
(i)
Every A ∈ G σ n is of the form PD , where P ∈ Σ n and D is diagonal.
(ii) For a diagonal D = diag( λ 1 , . . . , λ n ) ∈ G σ n , we have λ i ∈
G σ 1 for i = 1 , . .
. , n and
[Equation]
where we make a slight abuse of notation: on the right hand side φ σ is the homomorphism
G σ 1 → GL ( R ) .
In particular, φ σ is determined by lemma E.10 and its behavior for n = 1 .
Proof.
For any A ∈ G ReLU we observe that the differentiability hypotheses of lemma E.12 holdsfor x ∈ U where U is a dense open set with measure-0 complement.
Indeed, if t 1 , . . . , t M ∈ R arethe points where σ fails to be differentiable, we can take U to be the complement of the hyperplanearangement given by
[Equation]
Fix a row i — the matrix A is invertible by hypotheses, and so there must be some j such that a ij (cid:54) = 0(otherwise the i -th row of A is 0).
For any x ∈ U , we have by lemma E.
[Equation]
15) and we claim that this cannot hold unless a ik = 0 for k (cid:54) =
j .
First, there is a ( x 1 , . . .
, x n ) ∈
U suchthat σ (cid:48) ( x j ) (cid:54) =
(otherwise σ would be constant).
Next, ﬁxing x j at a value with σ (cid:48) ( x j ) (cid:54) = 0 andrearranging eq.
(E.15) we have
[Equation]
By hypothesis σ is non-linear and so σ (cid:48) is non-constant — hence if there were some a ik (cid:54) =
0 fork (cid:54) =
j , the left hand side of eq.
(E.16) would be non-constant.
We have shown each row of A has at most one non-0 entry a ij and that a ij (cid:54) = 0 .
For A to be invertible,it must be that these non-0 entries land in distinct columns .
This is exactly the form described initem (i).
Next, we note that for any ij (without assuming a ij (cid:54) = 0 ) eqs.
(E.15) and (E.16) tell us [Equation]
, and hence if D = diag( λ 1 , . . . , λ n ) ∈ G σ n and σ n ◦ D
= E ◦ σ n (i.e. E = φ σ ( D ) ), it must be that
E = diag( µ 1 , . . . , µ n ) for some µ 1 , . . . , µ n ∈ R .
Now the equation σ n ◦ D =
E ◦ σ n is equivalenttoσ ( λ i x i )
=
β i σ ( x i ) for i = 1 , . . . , n which in turn is equivalent to λ i ∈ G σ 1 and β i =
φ σ ( λ i ) for i = 1 , . . . , n , proving item (ii).
In light of theorem E.14, to ﬁll in the table of table 1 it will sufﬁce to deal with the n = 1 cases,which we do below.
Calculation of G ReLU .
We remark that this is just the “positive homogeneous” property of ReLU ,which is quite well known (cf.
[GBC16, §8.2.2], [FB17, §2], [Kun+21, §3], [Men+19, §3], [RK20,§3, A], [Yi+19, §2-3]).
Using remark E.9 if a ∈ G σ {+
 [Equation]+}
If a < 0 then setting x = − 1 results in a = 0 , a contradiction.
So a > 0 and max { 0 , ax } =a max { 0 , x } , showing φ σ ( a )
= a .
Modiﬁcations for LeakyReLU .
By deﬁnition, for 0 < s (cid:28) 1 .
LeakyReLU
( x, a ) := (cid:26) sx for x <x for x
≥which we may simplify to LeakyReLU( x ) =
sx + (1 − s ) ReLU( x ) .
Suppose now that
LeakyReLU( ax ) =
b LeakyReLU( x ) , or using our simpliﬁcation sax + (1 − s ) ReLU( ax ) = b ( sx + (1 − s ) ReLU( x )) .
(E.17)
If a < 0 , we may choose x = − 1 to obtain− a = − sa − (1 − s ) a = − bs and x = 1 to obtainsa =
b showing that a = as 2 , which is impossible when 0 < s (cid:28) 1 .
So it must be a > 0 , and thenevaluating eq.
(E.17) at x = 1 gives a = b .
The sigmoid case: σ ( x ) = 1 / (1 + e x ) .
We will leverage of a useful fact about the sigmoid function: σ (cid:48)
( x ) is a smooth probability distribution function on R , with σ (cid:48)
( x ) > 0 for all x ∈ R .
[Equation]
If σ ( ax ) =
bσ ( x ) , differentiating with respect to x gives
[Equation]
Using eq.
(*) and the fact that to probability distribution functions are proportional if and only if theyare equal, we get σ (cid:48) ( ax ) = σ (cid:48) ( x ) .
Then integrating from −∞ to x tells us σ ( ax ) /a
= σ ( x )
; settingx =
we see 12 a = 12 , hence a = 1 .
To show φ σ = id , backtracking to eq.
(E.18) and setting x = 0 shows b = a .
The Gaussian RBF case: σ ( x ) = 1 √ 2 π e − x 2 2 .
We make use of several properties of this σ ( x ) :
i ) For any a > the function σ ( ax ) is a probability distribution function with mean 0 and
variance 1 a 2 ,and with σ ( ax ) > 0 for all x ∈ R and
(ii) σ is an even function ( σ ( − x ) =
σ ( x ) )
Now if σ ( ax ) =
bσ ( x ) , then the pdfs σ ( ax ) and σ ( x ) are proportional hence equal by item (i).
Therefore they have the same means and variances — since these are 0 , 1 a 2 and 0 , 1 respectively weconclude a = ± 1 .
Finally, we explain why φ σ ( A ) =
abs( A ) (entrywise absolute value).
Differentiating with respect tox gives
[Equation]
This implies b = 1 when a = 1 .
On the other hand differentiating item (ii) tells us − σ (cid:48)
( − x ) = σ (cid:48) ( x ) ,so when a = − {+
+}
[Equation]
and hence b = 1 .
The polynomial case: σ ( x ) = x d .
We remark that the description given in table 1 is implicit in[KTB19].
By theorem E.14 we only need to describe φ σ : G σ 1 → GL 1 ( R ) ;
for any a (cid:54) =
( ax ) d = a d x d
and this shows G σ 1 =
R \ { 0 } and φ σ ( a ) = a d . [Figure]
E.2.1 Gaussian error linear units ( GeLU s)
Introduced and ﬁrst studied in [HG16], these are deﬁned as GeLU( x ) = x Φ( x ) where Φ is thestandard normal cummulative distribution function:
[Equation]
By inspecting plots in ﬁg. 16a, we see that GeLU and ReLU are globally quite similar (they convergeas | x | → ∞ ) but that they differ when x within a few standard normal standard deviations of 0.
Onecan show that G GeLU n = Σ n : indeed, by theorem E.14 it sufﬁces to show that the only λ ∈ R \ { 0 }such that GeLU( λx ) =
φ ( λ ) GeLU( x ) for all x (where φ is some non-zero function of λ ) is λ = 1 .
Expanding, we see that
[Equation]
(E.20) and rearranging this becomes
[Equation]
(E.21) that is, the right hand side is constant as a function of x .
Then Φ( λx ) =
c Φ( x ) , and since Φ is positiveit must be c is too.
Moreover it must be λ > 0 , as otherwise Φ( λx ) is monotonically decreasingwhile Φ( x ) is increasing.
Finally, letting x → ∞ we see that c
= 1 , and from there we concludeλ = 1 by an argument similar to the use of item (i) in the Gaussian RBF case.
Despite the above calculation, it seems natural to ask how far G GeLU is from G ReLU , in other wordshow badly GeLU fails to be positive homogeneous.
One measure of this is obtained by letting
X ∼ N (0 , 1) be a standard normal variable and computing the root-mean-square error
[Equation]
as a function of λ > 0 , where the expectation is over X .
Here our choice of a standard normal X ismotivated by the same reasoning as discissed in [HG16], namely that activation inputs are roughlystandard normal, especially in the presence of batch normalization.
Evaluating eq.
(E.22) doesn’tseem particularly tractible analytically, but it does simplify to
[Equation]
In ﬁg. 16b we estimate ξ ( λ ) by sampling X and replacing the expectation with the correspondingaverage.
Evidently, as λ → ∞ the function ξ ( λ ) becomes linear: since Φ( x ) → 1 x ≥ 0 as λ → ∞(here 1 x ≥ 0 is the indicator of x > 0 , also known as the Heaviside or unit-step function), theasymptotic slope is (cid:112) E
[ | x ( 1 x ≥ 0 − Φ( x )) | 2 ] ≈ 0 .
E.3 Proof of theorem 3.3
Proof.
The explicit description of G (ReLU , n ) in table 1 is enough to show G (ReLU , n ) stabilizes{ R ≥
e i | i
= 1 , . .
. , n } .
Indeed, any A ∈ G (ReLU , n ) may be written as PD where D = diag( a i )for some a i > 0 and P is a permutation matrix associated to a permutation π .
It sufﬁces to show that
P and diag( a i ) each preserves { R ≥
e i | i
= 1 , . .
. , n } .
[-First,diag( a i ) { R ≥ 
 e i | i = 1 , . . . , n } = {-]First,
[-R ≥ 
-]
[-= 1 , . . . , n } = { R ≥ 
 e i | i = 1 , . .-][Equation]
since the a i > 0 so scaling by a i preserves the ray R ≥
e i .
Second,
P { R ≥[-
-]
[Equation] | i = 1 , . .
. , n }
[-R ≥ 
-][Equation] [Equation] Pe [Equation]
| [Equation] = 1 , . . . , n } =
[Equation] [Equation] = 1 , . . . , n } =[-{ R ≥ 
-]
[Equation]
| i = 1 , . .
. , n } (the last equality is due to the fact that we only consider the set of rays, not the ordered tuple of rays).
Conversely, if A ∈ GL ( n ) stabilizes { R ≥
e i | i
= 1 , . .
. , n } , then in particular for each j
Ae j ∈ R ≥
e i for some i and as A is invertible, setting j = π ( i ) yields a permutation of { 1 , . .
. , n } .
Moreover,since Ae j (cid:54) =
( A is invertible) there must be some a j > 0 so that Ae j = a j e i .
One cannow verify that A = PD where P is the permutation matrix associated to π and D = diag( a j )which matches the description of G (ReLU , n ) from table 1.
For the “moreover,” we prove the contrapositive, namely that if v = ( v 1 , . .
. , v n ) ∈ R n has at least2 non-0 coordinates v i , v j , then the G ReLU -orbit of the ray R ≥
v that v generates cannot be ﬁnite.
Indeed, suppose t ≥ 0 and let D =
diag(1 , . .
, 1 , t, 1 , . .
, 1) ( t in the i th position).
The n × 2matrix ( v | Dv ) has a 2 × 2 minor(cid:18) [Equation] v
[Equation]
(cid:19) with deteriminant (1 − t ) v i v j (cid:54) = [-
-]0 as long as t (cid:54) = 1 .
(E.24)
Thus v and Dv are linearly independent, and hence deﬁne distinct rays, for all t (cid:54) = 1 .
It follows thatany set of rays stabilized by G (ReLU , n ) that contains R ≥ 0 v is uncountable.
E.4 Proof of proposition 3.4
To give a rigorous proof we use induction on the depth k ; since this to some extent obscures the mainpoint, we brieﬂy outline an informal proof: consider a composition of 2 layers of the network f withweights W (cid:48) :
[Equation]
(E.25)
Using the deﬁning properties of G σ and φ σ , we can extract A i like
[Equation]
(E [Equation] [Equation] The resulting copy of φ σ ( A i ) on the right hand side of eq.
(E.26) is cancelled by the copy of φ σ ( A i ) − 1right-multiplying W
i +1 in eq.
(E.25), so that eq.
(E.25) reduces toσ ( A
{+i +1 W i +
 σ ( W i φ σ ( A i − 1 ) − 1 x + b i ) + A+}
{+i +
 b i +1 ) .+} In this way, in between any two layers (cid:96) i +1 ◦ (cid:96) i the A i in A
i W i φ σ ( A i − 1 ) − 1 and the φ σ ( A i ) − 1 in
A i +1 W
i +1 φ σ ( A i ) − 1 cancel .
However, the factors φ σ ( A m ) and φ σ ( A m ) − 1 appear on endpoints ofthe truncated networks f ≤ m , f >m
and so they are not cancelled.
To keep track of W, W (cid:48) while using the notation from section 3, we write(cid:96)
i ( x, W )
=
σ ( W i x + b i ) and (cid:96) i ( x, W (cid:48) )
=
σ ( W (cid:48)
i x + b (cid:48) i ) for i < k and so on.
Proof.
By induction on k , the depth of the network.
The case k = 1 is trivial, since there W (cid:48)
=
Wand there is nothing to prove.
For k > 1 we consider 2 cases:
Case m = 1 :
In this case let V =
( W 2 , b 2 , . . .
, W k , b k ) and V (cid:48) =(
A 2 W 2 , A 2 b 2 , A 3 W 3 φ σ ( A 2 ) − 1 , A 3 b 3 , . . .
, W k φ σ ( A − 1 k − 1 ) , b k ) , and letg ( x, V ) = (cid:96) k − 1 ( x, V ) ◦ · · · ◦ (cid:96) 1 ( x, V ) and g ( x, V (cid:48) ) =
[Table] where (cid:96) i ( x, V ) =
σ ( W i +1 x
+ b i +1 ) and similarly for V (cid:48) .
In other words, the weights V, V (cid:48) andfunction g represent the architecture obtained by removing the earliest layer of f .
Then, f ≤ 1 ( x, W ) =
σ ( W 1 x + b 1 ) and on the other hand
[Equation]
Using the identity σ ( A 1 z )
=
φ σ ( A 1 ) σ ( z ) for any z ∈ R n 1 , we obtainf ≤ 1 ( x, W (cid:48) ) = φ
[Equation]
( A
[Equation]
σ ( [Equation] [Equation] = φ σ ( A
[Equation]
) σ ( [Equation] 1 x [Equation]
This shows f ≤ 1 ( x, W (cid:48) )
=
φ σ ( A 1 ) ◦ f ≤ 1 ( x, W ) .
Next, f > 1 ( x, W ) =
g ( x, V ) but because V (cid:48) 1 =
A 2 W 2 whereas W (cid:48)
=
A 2 W 2 φ σ ( A 1 ) − {+
+}
[Equation]
= g ( x,
[Equation]
(cid:48)
[Equation]
−
[Equation]
By induction on k , we may assume g ( x, V ) =
g ( x, V (cid:48) ) and it follows that f > 1 ( x, W (cid:48) )
=f > 1 ( x, W ) ◦ φ σ ( A 1 ) −
Case m > 1 : Deﬁning V, V (cid:48) and g as above, we observe thatf ≤ i ( x, W ) =
g ≤ i − 1 ( σ ( W 1 x + b 1 ) , V ) and f ≤
[Equation]
( x,
[Equation]
A
[Equation]
[Equation]
[Equation]
(E.27)
By induction on k we may assume g ≤ i − 1 ( x, V (cid:48) )
=
φ σ ( A i ) g ≤ i − 1 ( x, V ) and sof [Equation]
i
[Equation] , W ) .
Finally, f >i ( x, W ) =
g >i − 1 ( x, V ) and f >i ( x, W (cid:48) )
=
g >i − 1 ( x, V (cid:48) ) and we may assume by induction on k that g >i − 1 ( x, V (cid:48) )
=
g >i − 1 ( x, V ) ◦ φ σ ( A i ) − 1 , hence f >i ( x, W (cid:48) )
=
f >i ( x, W ) ◦φ σ ( A i ) − 1 .
E.5 Proof of theorem 4.2
Proof.
Observe that by proposition 3.4, ˜ f >l = f >l ◦ φ σ ( A − 1 l ) .
Hence
[Equation]
If S contains φ σ ( G σ nl ) we may choose ϕ
= φ σ ( A l ) to achieve S ( f, ˜ f, l, ϕ )
=
f as functions.
Similarly, ˜ f ≤ l = φ σ ( A l ) ◦ f ≤ l so if S contains φ σ ( G σ nl ) we may choose ϕ = φ σ ( A − 1 l ) to achieve
S ( f, ˜ f, l, ϕ ) =
˜ f as functions.
In either case eq.
(4.1) holds.
E.6 Symmetries of the loss landscape
Given that the intertwiner group describes a large set of symmetries of a network, it is not surprisingthat it also provides a way of understanding the relationship between equivalent networks.
Proposition 3.4 has an interpretation in terms of the loss landscape of model architecture.
For any layer n iin f , the action of G σ ni on the weight space W , translates to the obvious group action on the losslandscape.
Corollary E.28.
For any 1 ≤ i ≤ k , the group G σ ni acts on W and for any test set D t ⊂ X × Y ,model loss on D t is invariant with respect to this action.
More precisely, if (cid:96) (Φ( W ) , D ) is the loss ofΦ( W ) on test set D t , then for any g ∈ G σ ni , (cid:96) (Φ( W ) , D ) = (cid:96) (Φ( gW ) , D ) .
E.7 Comparing capacities of stitching layers via discretization
Let Mat n × n ( R ) denote the space of n × n matrices.
For r = 1 , . . . , n let Mat rn × n ( R ) ⊆
Mat n × n ( R )denote the rank r matrices, and let G ReLU n be as described in table 1.
Suppose that each realdimension of Mat n × n ( R ) is replaced by a discrete grid N ( M, (cid:15) )
= {− M + i(cid:15) | i = 0 , . .
, (cid:98) 2 M (cid:15) (cid:99)− 1 }— here (cid:15) could represent the limits of numerical precision in a ﬂoating point number system, and Mcould represent the maximum numerical magnitude.
The size of each such grid is (cid:98) 2 M(cid:15) (cid:99) , and so thenumber of points in the resulting mesh grid N ( M, (cid:15) ) n 2 ⊂ Mat n × n ( R ) is (cid:98) 2 M(cid:15)
(cid:99) n 2 .
We now estimatethe number of points of G ReLU n and
G ReLU n is a disjoint union of n !
irreducible components, corresponding to the n !
possible permutations P in table 1.
Each of these components is n -dimensional, corresponding to the fact that thefactor D in table 1 is an arbitrary positive diagonal matrix.
Hence we obtain
[Equation]
On the other hand, any matrix A ∈ Mat rn × n ( R ) can be written as A = UV where U ∈
Mat n × r ( R )and V ∈ Mat r × n ( R ) .
These U and V are not unique: given any invertible r × r matrix W ∈ GL r ( R ) ,we have A = ( UW )( W − 1 V ) .
From this we obtain the approximation
[Equation]
It follows that
[Equation]
Ignoring the term (2 nr − r 2 ) log 2 , which is independent of M, (cid:15) , we get the approximationlog
[Equation]
Next, we make the coarse approximation
[Equation]
with this approximation the expression of eq.
(E.35) is approximated as
[Equation]
From this we conclude that as long as1.
r ≥ 1 (we actually already assumed this when deﬁning Mat rn × n ( R ) ) and2.
M(cid:15)
(cid:29) n , which roughly says that the number of grid points per dimension is greater thanthe number of rows (equivalently columns) in Mat n × n ( R ) ,
[Equation]
) using item [Equation] [Equation] < 0 using item 2.
[Equation]
E.41) The upshot is that our approximations and items 1 and 2 implylog | G ReLU n ∩ N ( M, (cid:15) ) n 2 | − log | Mat rn × n ( R ) ∩ N ( M, (cid:15) ) n 2 | < 0 , and hence (E.42) | G ReLU [Equation] E.
[Equation]
E.8 Calculations related to dissimilarity measures (for section 5)
Proof of lemma 5.5.
We must show that for any x 1 , . . .
, x r ∈ R d and any c 1 , . . .
, c r ∈ R that
[Equation]
We use the elementary “max ≥ mean” inequality:
[Equation]
where on the right hand side “ · ” denotes dot product.
This implies
[Equation]
where the last inequality follows from the fact that the dot product is a kernel function.
The last string of inequalities also shows (cid:80) i,j c
i c j max( x i (cid:12) x j ) > 0 when x 1 , . . .
, x r are linearlyindependent, but does not directly imply the converse.
It would be interesting to know conditions onx 1 , . . .
, x r that imply positive deﬁniteness of the matrix (max( x i (cid:12) x j )) .
Proof of lemma 5.1.
By table 1 any A ∈ G ReLU can be factored as A = PD with P a permutationmatrix and D a positive diagonal matrix, and we can obtain a similar factorization B = QE .
Thenµ ( XA, Y B ) =
µ ( XPD, Y QE ) =
µ ( XP, Y Q ) = µ ( X, Y ) where the second equality uses the hypothesis that µ is invariant to right multiplication by positivediagonal matrices, and the third uses the hypothesis that µ is invariant to right multiplication bypermutation matrices.
Lemma E.44.
Suppose A is a matrix such that max(
Ax 1 (cid:12) Ax 2 )
=
max( Ax 1 (cid:12) Ax 2 ) for allx 1 , x 2 ∈ R d .
Then, A is of the form PD where P is a permutation matrix and D is diagonal withdiagonal entries in {± 1 } .
Proof.
We only need the special case where x = y : observe that
[Equation]
This means that if max( Ax 1 (cid:12) Ax 2 ) =
max( Ax 1 (cid:12) Ax 2 ) , then A preserves the (cid:96) ∞ norm on R d ,hence in particular preserves the unit hypercube in R d , and it is known that symmetries of thehypercube have the form PD where P is a permutation matrix and D is diagonal with diagonalentries in {± 1 } (see for example [Ser77, §5.9]). E.9 Intertwiners and more general architecture features (justiﬁcation of remark 3.6)
Here we brieﬂy discuss how ubiquitous architecture features like batch normalization and residualconnections interact with intertwiner groups.
For simplicity in this section we only consider σ =
ReLU .
E.9.1 Batch normalization
A batch normalization layer that takes as input X ∈ R b · n (where b is the batch size and n is thedimension of the layer) and returns˜ X diag( ˜ X T ˜ X ) − 1 diag( γ ) + β where ˜ X
= X − 11 T X and where γ, β ∈ R n are the “gain” and “bias” parameters of the batch normalization layer, isinvariant under independent scaling of coordinates, that is transformations of the form X ← XDwhere D is an n × n positive diagonal matrix (see e.g. [BMC15]).
Hence a k -layer ReLU MLP as in section 3 enhanced with batch normalization ( pre -activation, as is standard) is invariant under theaction of the slightly larger group (cid:81) k − 1 l =1 ( R n l > 0 (cid:111)
G ReLU nl ) , where the action is given by {+
+}
[Equation]
E
[Equation]
[Equation]
Here,
• c l ∈ R n l > 0 and A l ∈ G ReLU nl , for all l = 1 , . . .
, k − 1• π : G ReLU nl → Σ n l is the homomorphism setting the positive entries to 1.
The key point is that we get another factor of R n l > 0 at each layer.
We also note that the space ofmatrices of the form π ( A l ) diag
c
l is, incidentally, exactly G ReLU nl , and that using eq.
(E.45) onecan generalize proposition 3.4 and theorem 4.2 to the case of networks with batch normalization.
E.9.2 Residual connections
We expand on remark 3.6 and explain what exactly transpires with residual connections below.
Suppose we have a k -layer MLP as in section 3 (again for simplicity with σ = ReLU ), together withresidual connections between a set of layers R = { r 1 , . . .
, r m } ⊆ { 2 , · · · , k − 1 } 10 : R n· · · R n ri −· · · R n ri · · · R n k −
R n L +1 σWσW ri −σW ri − 1+ id
W ri σ
[Equation]
k (E.46) (for legibility biases b l are suppressed).
In addition we assume that the depth of each residual blockis some ﬁxed, that is r i − r
i − 1 =
b = constant for all i .
First, we claim that a ( A 1 , . . .
A k − 1 ) ∈
(cid:81)
Ll
=1 G ReLU nl stabilizes the function f if (not claiming ifand only if)
A r i =
A r j for all r i , r j ∈ R .
To see this suppose g i ( x, W ) , i = 1 , . . .
, m are the depthb feedforward networks of the residual blocks, so that the f ≤ r i is given by
[Equation]
where
[Equation]
Assuming by induction on i that proposition 3.4 applies at the residual connections in R we note thatwith weights W (cid:48) eq.
(E.47) turns into
[Equation]
[Equation] Note that proposition 3.4 applies directly to the g i , so we may compute g i ( z, W (cid:48) ) =
A r i g i (
A − 1 r i − 1 z, W ) .
Hence
[Equation]
and we see that the only way f ≤ r i ( x, W (cid:48) ) =
Bf ≤ r i ( x, W ) for some matrix B is if A r i −
=
A r i ,proving our claim.
This also shows that if A r denotes the common value of the A r i for r i ∈ R ,we have f ≤ r i ( x, W (cid:48) )
=
A r f ≤ r i ( x, W ) for all i .
It is also true that f >r i ( x, W (cid:48) [Equation]
=
f >r i ( A − 1 r x, W ) :observe that
[Equation]
By descending induction on k , we may assume f r i +1 ( x, W (cid:48) )
=
f r i +1 ( A − 1 r x, W ) , and as aboveg i +1 ( z, W (cid:48) )
=
A r g i +
( A − 1 r z, W ) , so that with weights W (cid:48) eq.
(E.50) becomes
[Equation]
as claimed.
Finally, we describe how stitching fails inside a residual block.
Suppose we use weights W l forl ≤ r i + j where 0 < j < b )
(recall b = depth of our basic block) and weights W (cid:48) l for l > r
i + j .
The resulting stitched network is (attempting to use indentation to increase legibility)
[Equation]
By proposition 3.4 g >j ( z, W (cid:48) )
=
A r g >j
( A − 1 r i + j z, W ) , and we have shown f >r i +
( z, W (cid:48) )
=
f >r i +1 ( A − 1 r z, W ) .
Combining these facts eq.
(E.52) becomes
[Equation]
j (
A − 1 r i + j ϕg ≤ j ( f ≤ r i ( x, W ) , W ) ,
W ) ,
W ) .
(E.53)
Even after cancelling to remove the A − 1 r A r and in the ideal case where ϕ
=
A r i + j , we are left withan extra factor of A − 1 r left multiplying f ≤
r i ( x, W ) :
f r i +1 ( A − 1 r f ≤ r i ( x, W ) + g >j ( g ≤ j ( f ≤ r i ( x, W ) , W ) , W ) , W ) .
(E.54)
F Network dissection details
Here we include some supplementary results and experiments for examining coordinate basis interpretability with network dissection [Bau+17].
F.1 Network dissection methodology
The Broden concept dataset, compiled by Bau et al., contains pixel-level annotations for hierarchicalconcepts including colors, textures, objects, and scenes.
For every channel activation, networkdissection assesses the binary segmentation performance with every visual concept from Broden.
The method ﬁrst computes the channel activation for every Broden image.
The distribution ofthe activations for the channel is used to binarize the activation (where we threshold by the top0 .
5% of all activations for the channel) to deﬁne a segmentation mask for the channel activationwhich is interpolated to the size of the input image.
If the Intersection over Union (IoU) of theactivation segmentation mask and a concept mask is high enough (namely where IoU > 0 . 04 ),network dissection labels the activation an interpretable detector for the concept. [Figure] n t e r p r e t ab l e [Figure]
[Figure]
Change in basis
[Figure] F.2 Additional experiments
[Figure] ﬁg. 17 breaks down the categories of interpretable units for the models and rotations examined inﬁg.
The number of interpretable units tends to be dominated by the object and scene conceptdetectors for the ResNet-50 and the ConvNeXt models.
[Figure] We also perform an analogous network dissection experiment to section 6 within the residual blocksfor the normal and modiﬁed ResNet-50 in ﬁg. 18.
Like the ConvNeXt model in ﬁg.
we ﬁnd that,surprisingly, the percentage of interpretable units actually tends to increase away from the activationbasis.
Per-concept breakdowns produced by network dissection for three different rotation powers in theexperiment in ﬁg. 3 are given for the ResNet-50 in ﬁg. 19, the modiﬁed ResNet-50 (without a ReLUactivation function on the residual output) in ﬁg. 21, and the ConvNeXt in in ﬁg. 23.
We also includethe units with the highest concept intersection over union scores for the the three representativerotations for the ResNet-50 in ﬁg. 20, the modiﬁed ResNet-50 (without a ReLU activation functionon the residual output) in ﬁg. 22, and the ConvNeXt in in ﬁg. 
 [Figure] Rotation 0. Rotation
[Figure]
[Figure] Rotation [Figure] F.3 Model training details
[Figure] We train a ResNet-50 without ReLU (or any activation function) on the residual blocks in PyTorchusing [Lec+22] on ImageNet [Den+09].
We train with SGD with momentum for 88 epochs with acyclic learning rate rate of 1 . 7 , label smoothing of 0 .
1 , a batch size of 512, and weight decay of 10 − 4 .
The model achieves 76.1% top-1 accuracy.
We use pretrained weights for the ResNet-50 (unmodiﬁed)and ConvNeXt models from [MR10] and [Wig19] respectively. [Figure]
G Dataset Details
[Figure] CIFAR-10: CIFAR-10 is covered by the MIT License (MIT).
We use canonical train/test splits [Figure] (imported using torchvision). [Figure]
Broden: the code used to generate the dataset is covered by the MIT license.
[Figure]
ImageNet: ImageNet is covered by CC-BY 4.0.
We use canonical train/test splits.
[Figure]
) Myrtle CNN architecture, summary courtesy of torch- info.
[Figure]
(a) Our ResNet20 architecture, summary courtesy of torchinfo.
(b) Internals of the 1st BasicBlock (the sequential con- tains the residual connection).