On the Symmetries of Deep Learning Models and their Internal Representations
first.last@pnnl.gov
Abstract
Symmetry has been a fundamental tool in the exploration of a broad range of complex systems.
In machine learning, symmetry has been explored in both models and data.
In this paper we seek to connect the symmetries arising from the architecture of a family of models with the symmetries of that family’s internal representation of data.
We do this by calculating a set of fundamental symmetry groups, which we call the intertwiner groups of the model.
Each of these arises from a particular nonlinear layer of the model and different nonlinearities result in different symmetry groups.
These groups change the weights of a model in such a way that the underlying function that the model represents remains constant but the internal representations of data inside the model may change.
We connect intertwiner groups to a model’s internal representations of data through a range of experiments that probe similarities between hidden states across models with the same architecture.
Our work suggests that the symmetries of a network are propagated into the symmetries in that network’s representation of data, providing us with a better understanding of how architecture affects the learning and prediction process.
Finally, we speculate that for ReLU networks, the intertwiner groups may provide a justiﬁcation for the common practice of concentrating model interpretability exploration on the activation basis in hidden layers rather than arbitrary linear combinations thereof.
Introduction
Symmetry provides an important path to understanding across a range of disciplines.
This principle is well-established in mathematics and physics, where it has been a fundamental tool (e.g., Noether’s
Theorem [Noe18]).
Symmetry has also been brought to bear on deep learning problems from a number of directions.
There is, for example, a rich research thread that studies symmetries in data types that can be used to inform model architectures.
The most famous examples of this are standard convolutional neural networks which encode the translation invariance of many types of semantic content in natural images into a network’s architecture.
In this paper, we focus on connections between two other types of symmetry associated with deep learning models: the symmetries in the learnable parameters of the model and the symmetries across different models’ internal representation of the same data.
The ﬁrst of these directions of research starts with the observation that in modern neural networks there exist models with different weights that behave identically on all possible input.
We show in section 3 that at least some of these equivalent models arise because of symmetries intrinsic to the nonlinearities of the network.
We call these groups of symmetries, each of which is attached to a particular type of nonlinear layer σ of dimension n , the intertwiner groups G σ n of the model.
These intertwiner groups come with a natural action on network weights for which the realization map to function space of [JGH18] is invariant (proposition 3.4).
As such they provide a unifying framework in which to discuss well-known weight space symmetries such as permutation of neurons [Bre+19] and scale invariance properties of ReLU and batch-norm [IS15; NH10].
Next, we tie our intertwiner groups to the symmetries between different model’s internal representations of the same data.
We do this through a range of experiments that we describe below; each builds on a signiﬁcant recent advance in the ﬁeld.
Neural stitching with intertwiner groups: The work of [BNB21; Csi+21; LV15] demonstrated that one can take two trained neural networks, say A and B, with the same architecture but trained from different randomly initialized weights, and connect the early layers of network A to the later layers of network B with a simple “stitching” layer and achieve negligible loss in prediction accuracy.
This was taken as evidence of the similarity of strong model’s representations of data.
Though the original experiments use a fully connected linear layer to stitch, we provide theoretical evidence in theorem 4.2 that much less is needed.
Indeed, we show that the intertwiner group (which has far fewer parameters in general) is the minimal viable stitching layer to preserve accuracy.
We conduct experiments stitching networks at ReLU activation layers with the stitching layer restricted to elements of the group G ReLU showing in ﬁg. 1 that one can stitch CNNs on CIFAR-10 [Kri09] with only elements of G ReLU incurring less than ≈ 10% accuracy penalty at most activation layers.
This is surprisingly close to the losses found when one allows for a much more expressive linear layer to be used to stitch two networks together.
However, we see that there remains a signiﬁcant gap between the stitching accuracies obtained using G ReLU and fully connected linear layers; this provides independent conﬁrmation of earlier ﬁndings that neurons of networks trained with different random seeds (i.e. with independent initializations and different random batches) are not simply permutations of each other [Li+15; Wan+18].
It is also consistent with observed phenomena such as distributed representations in hidden features [GBC16, §15.4] and perhaps also polysemantic neurons [Ola+20].
Representation dissimilarity measures for G ReLU :
In section 5 we present two statistical dissimilarity measures, G ReLU -Procrustes and G ReLU -CKA, for ReLU -activated hidden features in different networks, say A and B .
Our measures are counterparts of orthogonal Procrustes distance (see [-e.g.[DDS21b]) and Centered Kernel Alignment (CKA) 
-]e.g.
[DDS21b]) and Centered Kernel Alignment (CKA) 1 [Kor+19] respectively, which are invariant to orthogonal transformations, and are maximized when the hidden features of networks A and B agree up to orthogonal transformations.
In contrast G ReLU -Procrustes and G ReLU -CKA are invariant to
G ReLU transformations, and are maximized when hidden features agree up to G ReLU transformations.
We compare and contrast our measures with their orthogonal counterparts, as well as with stitching experiment results.
Figure 2 shows a comparison of G ReLU and orthogonal CKA measures.
[Figure]
f r a c t
i on i n t e r p r e t ab l e un i t s
[Figure]
Impact of activation functions on interpretability: An intriguing ﬁnding from [Bau+17, §3.2] was that individual neurons are more interpretable than random linear combinations of neurons.
Our results on intertwiner groups (theorem 3.3) predict that this is a particular feature of ReLU networks.
Indeed, ﬁg. 3 shows that in the absence of ReLU activations interpretability does not decrease when one moves from individual neurons to linear combinations of neurons — sectiondescribes this experiment in further detail.
This result suggests that intertwiner groups provide a theoretical justiﬁcation for the explainable AI community’s focus on individual neuron activations, rather than linear combinations thereof [Erh+09; Na+19; Yos+15; ZF14; Zho+15], but that this justiﬁcation is only valid for layers with certain activation functions.
Taken together, our experiments provide evidence that a network’s symmetries (realized through intertwiner groups) propagate down to symmetries of a model’s internal representation of data.
Since understanding how different models process the same data is a fundamental goal in ﬁelds such as explainable AI and the safety of deep learning systems, we hope that our results will provide an additional lens under which to examine these problems.
Related work
The research on symmetries of neural networks is extensive, hence we aim to provide a representative sample knowing it will be incomplete.
[Bre+19; FB17; GBC16; Yi+19] study the effect of weight space symmetries on the loss landscape.
On the other hand, [BMC15; GBC16; Kun+21; Men+19] study the effect of weight space symmetries on training dynamics, while [GBC16; RK20] show that weight space symmetries pose an obstruction to model identiﬁability.
Neural stitching was introduced as a means of comparing learned representations between networks in [LV15].
In [BNB21] it was shown to have intriguing connections with the “Anna Karenina ” (high performance models share similar internal representations of data) and “more is better” (stitching later layers of a weak model to early layers of a model trained with more data/parameters/epochs can improve performance) phenomena.
[Csi+21] considered constrained stitching layers by restricting the rank of the stitching matrix or by introducing an (cid:96) 1 sparsity penalty. Our methods are distinct in that we explicitly optimize over the intertwiner group for ReLU nonlinearities (permutations and scalings).
Both [BNB21; Csi+21] compare their stitching results with statistical dissimilarity measures such as CKA.
Our G ReLU -Procrustes measure is a close relative of the permutation Procrustes distance introduced in [Wil+21], and our G ReLU -CKA is a an instance of CKA [Kor+19] in which the kernel is taken to be max { x 1 · y 1 , . . .
, x d · y d } .
[Li+15] developed algorithms for obtaining a permutation to align neurons, and [Wan+18] introduced neuron activation subspace matching and used it to study similarity of hidden feature representations.
[AHS22; Ent+22; Tat+20] all aligned neurons with permutations with the goal of obtaining low-loss paths between the weights of networks.
The objectives for neuron alignment used in these works include maximizing correlation ([AHS22; Li+15; Tat+20]), maximizing a “match” (as deﬁned in [Wan+18]), simulated annealing search algorithms ([Ent+22]), direct alignment of weights via a bilinear assignment problem and a “straight-through estimator” of back-propagated training loss ([AHS22]).
Each of these is distinct from our method, which explicitly seeks a permutation minimizing training loss and searches for one using standard convex relaxation methods for permutation optimization.
Approaches to deep learning interpretability sometimes assume that the activation basis is special [Erh+09; Na+19; Yos+15; ZF14; Zho+15].
Studying individual neurons rather than linear combinations of neurons signiﬁcantly reduces the complexity of low-level approaches to the understanding of neural networks [Elh+21].
In tension with this, many different projections of hidden layer activations appear to be semantically coherent [Sze+14].
However, [Bau+17] found evidence that the hidden feature vectors closer to the coordinate basis align more with human concepts than vectors sampled uniformly from the unit sphere.
The symmetries of nonlinearities
Let Mat n 1 ,n 0 ( R ) be the algebra of all n 0 × n 1 real matrices and GL n ( R ) be the group of all invertible n × n matrices.
Let σ : R → R be a continuous function.
For any n ∈ N , we can build a nonlinearity σ n from R n to R n by applying σ coordinatewise, i.e., σ n ( x 1 , . . .
, x n )
= ( σ ( x 1 ) , . . .
, σ ( x n )) .
Fix some k > 1 and for each 1 ≤ i < k let (cid:96) i : R n i − 1 → R n i be the composition of an afﬁne layer and a nonlinear layer, so that (cid:96) i ( x ) :
= σ n i ( W i x + b i ) , and let (cid:96)
k
( x ) :
=
W k x + b k .
Here
W i ∈
Mat n i ,n i − 1 ( R ) and b i ∈ R n i are the weights and bias of layer i respectively.
We deﬁne f : R n 0 → R n k to be the neural network f = (cid:96) k ◦ · · · ◦ (cid:96) 1 .
For each 1 ≤ i < k − 1 we can then decompose f as f = f >i ◦ f ≤ i where f ≤ i = (cid:96) i ◦ · · · ◦ (cid:96)and f >i = (cid:96) k ◦ · · · ◦ (cid:96) i +1 .
We deﬁne
W :=
( W i , b i | i = 1 , . . .
, k ) and W :
= k (cid:89) i =
(Mat n i ,n i − 1 ( R ) ×
R n i ) where the former is the collection of all weights of f and the latter is the space of all possible weights for a given architecture.
When we want to emphasize the dependence of f on weights W , we write f ( − , W ) (and similarly f ≤ i ( − , W ) , f >i ( − , W ) ).
One of the topics this work will consider is vector space bases for f ’s hidden spaces R n i , for 1 ≤ i ≤ k − 1 .
We will investigate the legitimacy of analyzing features f ≤
i ( D ) for dataset D ⊂ R nwith respect to the activation basis for R n i which is simply the usual coordinate basis, e 1 , . . .
, e n i where e j = [ δ j(cid:96) ] n i (cid:96) =1 is naturally parameterized by individual neuron activations.
Note that R n i has an inﬁnite number of other possible bases that could be chosen.
3.1 Intertwiner Groups
For any 0 ≤ i < k , elements of GL n i ( R ) can be applied to the hidden activation space R n i both before and after the nonlinear layer σ n i .
We deﬁne
G σ ni :=
{ A ∈ GL n i ( R ) | there exists a B ∈ GL n i ( R ) such that σ n i ◦
A = B ◦ σ n i } .
Activation
G σ n φ σ ( A ) [Equation] identity
[Equation] GL
[Equation] A σ
[Equation] A σ [Equation]
ReLU [Equation] x )
Matrices PD , where D has positive entries
A [Equation]
= LeakyReLU
[Equation]
x )
Same as ReLU as long as negative slope (cid:54) =
A
[Equation]
Matrices PD , where D has entries in {± 1 } abs
( A ) σ ( x ) = x
d (polynomial)
Matrices PD , where D has non-zero entries
A (cid:12) d
Informally, we can understand G σ ni to be the set of all invertible linear transformations whose action on R n i prior to the nonlinear layer σ n i has an equivalent invertible transformation after σ n i .
This is an instance of the common procedure of understanding a function by understanding those operators that commute with it.
For any A ∈ GL n i ( R ) , we can write σ ( A ) for the n i × n i matrix formed by applying σ to all entries in A .
Lemma 3.1. Suppose σ ( I n ) is invertible and for each A ∈ GL n ( R ) deﬁne φ σ ( A ) = σ ( A ) σ ( I n ) − 1 .
Then G σ n is a group, φ σ : G σ n → GL n ( R ) is a homomorphism and σ n ◦ A
= φ σ ( A ) ◦ σ n .
We defer all proofs to appendix E. We include concrete examples of σ for small n i there as well.
Deﬁnition 3.2.
When the hypotheses of lemma 3.1 are satisﬁed (namely, σ ( I n ) is invertible)
we call
G σ n the intertwiner group of the activation σ n .
We denote the image of the homomorphism φ σ as φ σ ( G σ n ) .
The intertwiner group G σ n and φ σ are concretely described for a range of activations in table 1 — the last two examples motivate the generality of deﬁnition 3.2.
Note also that in both of those cases
A (cid:55)→ φ σ ( A ) is not a homomorphism on all of GL n ( R ) , but is a homomorphism when restricted to the appropriate subgroup G σ n .
While a substantial part of table 1 can be found scattered in prior work, our calculations in appendix E.2 deal with the different cases of table 1 in a uniform way, by what amounts to an algorithm that compute G σ n and φ σ given in terms of elementary properties of any (reasonable) activation function σ . 2 As design of activation functions remains an active industry (for example [Elh+22]), our techniques for computing G σ n could be useful in future studies of network symmetries.
The following theorem shows that the activation basis is intimately related to the intertwiner group of
ReLU : G ReLU admits a natural group-theoretic characterization in terms of the rays spanned by the activation basis, and dually the rays spanned by the activation basis can be recovered from G ReLU .
While both its statement and proof are elementary, our interest in this theorem lies in the question of whether it could potentially provide theoretical justiﬁcation for focusing model interpretation studies on individual activations.
We investigate this question further in section 6.
Theorem 3.3.
The group G ReLU n is precisely the stabilizer of the set of rays { R ≥
e i ⊂ R n | i = 1 , . . . , n } .
Moreover if R ≥ 0 v 1 , . . .
, R ≥ 0 v N ⊆ R n is a ﬁnite set of rays stabilized by G ReLU , then for each v i
=
[ v i 1 , . . .
, v in ] T , it must be that v ij = 0 for all but one j ∈ { 1 , . . . , n } .
Equivalently up to multiplication by a positive scalar every v i is of the form ± e j for some j .
3.2 Weight space symmetries
The intertwiner group is also a natural way to describe the weight space symmetries of a neural network.
We denote by F ⊆ C ( R n 0 , R n k ) the space of continuous functions that can be described by a network with the same architecture as f .
As described in [JGH18] there is a realization map Φ :
W → F mapping weights W ∈ W to the associated function f ∈ F .
Φ arises because there are generally multiple sets of weights that yield the same function.
We will show that Φ is invariant with respect to an action of the intertwiner groups on W so that intertwiner groups form a set of “built-in” weight space symmetries of f .
This result, which encompasses phenomena including permutation symmetries of hidden neurons, is well known in many particular cases (e.g., [GBC16, §8.2.2], [Bre+19, §3], [FB17, §2], [Men+19, §3], [RK20, §3, A]).
From Proposition 3.4 we can also derive corollaries regarding symmetries of the loss landscape — these are included appendix
Proposition 3.4.
Suppose A i ∈ G σ ni for 1 ≤ i ≤ k − 1 , and let
[Equation]
Then, as functions, for each m
[Equation]
In particular, f ( x, W (cid:48) ) =
f ( x, W ) for all x ∈ R n 0 .
Equivalently, we have Φ( W (cid:48) )
=
Φ( W ) ∈ F .
Remark 3.6 .
We will show in appendix E that the statement of this theorem must be modiﬁed if the architecture of f contains residual connections.
By placing suitable restrictions on the matrices A iwe can recover a form of eq.
(3.5) provided m occurs at the end of a residual block.
However, there doesn’t seem to be a way to obtain such an identity when m occurs inside a residual block; we see empirical evidence consistent with this point in ﬁgs.
2, 6 and 9 below.
3.3 A “sanity test” for intertwiners
To test proposition 3.4 with a simple experiment, we begin with a Myrtle CNN [Pag18] networktrained for 50 epochs on the CIFAR-10 dataset, ﬁx a pre-activation layer l , and apply a transformation
A to the weights W l and biases b l to obtain AW l and Ab l (we only act on channels, hence in practice this is implemented by an auxiliary 1-by-1 convolution layer).
We consider 2 choices of A : (i) a random element of G ReLU , where P is a random permutation and the diagonal entries of D are sampled from a lognormal distribution, and (ii) a random orthogonal matrix, obtained as the “ Q ” in a
QR -decomposition of a random matrix X with independent standard normal entries. Next, we freeze layers up to and including l and ﬁnetune the later layers for another 50 epochs. We refer to the difference between the validation accuracy before and after applying the transformation A and ﬁnetuning as a rotation penalty .
Based on proposition 3.4, when A ∈ G ReLU the network should be able to recover reasonable performance even with the transformed features — for example, by updating W l +1 to W l +1 φ σ ( A ) − 1 .
On the other hand, with probability 1 there is no matrix B such that updating W l +1 to W l +1 B counteracts the effect of an orthogonal rotation A on all possible input.
We see that this is indeed the case in ﬁg. 4: transforming by A ∈ G ReLU rather a random orthogonal matrix results in signiﬁcantly smaller rotation penalties.
Intertwining group symmetries and model stitching
In this section we provide evidence that some of the differences between distinct model’s internal representations can be explained in terms of symmetries encoded by intertwiner groups.
We do this using the stitching framework from [BNB21; Csi+21], and begin by reviewing the concept of network stitching.
Suppose f, ˜ f are two networks as in section 3, with weights W , ˜ W respectively.
For any 1 ≤ l ≤ k −we may form a stitched network S ( f, ˜ f, l, ϕ ) :
R n 0 → R n k deﬁned in the notation of sectionby S ( f, ˜ f, l, ϕ ) =
˜ f >l ◦ ϕ ◦ f ≤ l – here ϕ : R n l → R n l is a stitching layer .
In a typical stitching experiment one trains networks f and ˜ f from different initializations and freezes their weights, constrains ϕ to some simple function class S (e.g., afﬁne maps in [BNB21]), and trains S ( f, ˜ f, l, ϕ ) by optimizing ϕ alone.
The ﬁnal validation accuracy Acc S ( f, ˜ f, l, ϕ ) of S ( f, ˜ f, l, ϕ ) is then considered a measure of similarity (or lack therof) of the internal representations of f and ˜ f in R n l — in this framework the situation
[Equation]
corresponds to high similarity since the hidden representations of model f and ˜ f could be related by a transformation S .
Recall that even though the networks f ( W ) and f ( W (cid:48) ) may be equal as functions , their hidden representations need not be the same (an example of this is given in appendix C).
Our next result shows that in the case where f and ˜ f do only differ up to an element of G σ nl , eq.
(4.1) is achievable even when the stitching function class S is restricted down to elements of φ σ ( G σ nl ) (see deﬁnition 3.2).
Theorem 4.2.
Suppose ˜ W
=
( A 1 W 1 , A 1 b 1 , A 2 W 2 φ σ ( A − 1 1 ) , A 2 b 2 , . . .
, W k φ σ ( A − 1 k ) , b k ) where
A i ∈ G σ ni for all i .
Then eq.
(4.1) is achievable with equality if the stitching function class S containing ϕ contains φ σ ( G σ nl ) .
Motivated by theorem 4.2, we attempt to stitch various networks at ReLU activation layers using the group G ReLU described in Figure 1.
Every matrix A ∈ G ReLU can be written as PD , where P is a permutation matrix and D is diagonal with positive diagonal entries — hence optimization over
G ReLU requires optimizing over permutation matrices.
We use the well-known convex relaxation of permutation matrices to doubly stochastic matrices and describe our optimization procedure in greater detail in D.2.
Figure 1 gives the difference between the average test error of Myrtle CNN networks f and ˜ f and the network S ( f, ˜ f, l, ϕ ) , which we call the stitching penalty :
[Equation]
In our experiments S ( f, ˜ f, l, ϕ ) was stitched together at layer l via a stitching transformation ϕ that was either optimized over all afﬁne transformations, reduced rank afﬁne transformations as in [Csi+21] or transformations restricted to G ReLU .
We consider only the ReLU activation layers, as these are the only layers where the theory of section 3 applies, and we only act on the channel tensor dimension — in practice, this is accomplished by means of 1-by-1 convolution operations.
In particular, with G ReLU we are only permuting and scaling channels .
Lower values indicate that the stitching layer was sufﬁcient to translate between the internal representation of f at layer l and the internal representation of ˜ f .
We ﬁnd that when we learn a stitching layer over arbitrary afﬁne transformations of channels, we can nearly achieve the accuracy of the original models.
When we only optimize over G ReLU there is an appreciable increase in test error difference.
This is consistent with ﬁndings in [Csi+21;
Li+15; Wan+18] discussed in section 2, and also consistent with observations that hidden features of neural networks exhibit distributed representations and polysemanticism [Ola+20].
Nonetheless, that S ( f, ˜ f, l, ϕ ) is able get within less than 10% of the accuracy of f and ˜ f
in all but one layer suggests that elements of G ReLU can account for a substantial amount of the variation in the internal representations of independently trained networks.
We include the reduced rank transformations as the dimension of their parameter spaces is greater than that of G ReLU , and yet they incur signiﬁcantly higher stitching penalties.
If n l is the number of channels, we have dim G ReLU nl = n l whereas the dimension of rank r transformations is 2 n l · r − r 2 (hence greater than dim G ReLU nl even for r = 1 ).
Finally, in the speciﬁc case of the Myrtle CNNs the stitching penalties incurred when using any layer other than 1-by-1 convolution with a rank 1 matrix all follow similar trends: they increase up to the third activation layer, then decrease at the ﬁnal activation layer.
Further stitching results on the ResNet20 architecture can be found in appendix D.3, including an experiment where we modify the architecture to have LeakyReLU activation functions, vary the negative slope of the LeakyReLU , and ﬁnd similar stitching penalties up to but not including a slope of 1.
This result is consistent with our calculations in table 1, where we ﬁnd that for any
LeakyReLU negative slope (cid:54) =
1 the intertwiner is the same as G ReLU (when the negative slope is 1 ,
LeakyReLU
( x ) = x
and so the intertwiner is all of GL n ).
Dissimilarity measures for the intertwiner group of ReLU
Stitching penalties can be viewed as task oriented measures of hidden feature dissimilarity.
From a different perspective, we can consider raw statistical measures of hidden feature dissimilarity.
In the design of measures of dissimilarity, a crucial choice is the group of transformations under which the dissimilarity measure is invariant.
For example, Centered Kernel Alignment ( CKA ) [Kor+19] with the dot product kernel is invariant with respect to orthogonal transformations and isotropic scaling.
We ask for a statistical dissimilarity metric µ on datasets X, Y ∈ R N × d with the properties that (0) 0 ≤ µ ( X, Y ) ≤ 1 , (i) ( G ReLU -Invariance )
If A, B ∈ G ReLU d and v, w ∈ R d then µ ( XA + 1 v T , Y B + 1 w T ) = µ ( X, Y ) , and (ii) ( Alignment Property ) µ ( X, Y )
= 1 if ( ∗ )
Y = XA + 1 v T for some A ∈ G ReLU d , v ∈ R d .
To motivate this question, we note that given such a metric µ , one can detect if X and Y do not differ by an element of G ReLU by checking if µ ( X, Y ) < 1 .
Our basic tool for ensuring (i) is the next lemma.
Lemma 5.1.
Suppose µ ( XA, Y B ) =
µ ( X, Y ) if A, B are either positive diagonal matrices or permutation matrices.
Then, (i) holds.
In effect, this allows us to divide the columns of X and Y by their norms to achieve invariance to the action of positive diagonal matrices and then apply dissimilarity measures for the permutation group such as those presented in [Wil+21].
Ensuring (ii) seems to require case-by-case analysis to determine an appropriate normalization constant.
Deﬁnition 5.2 ( G ReLU -Procrustes) .
Let D X =
diag( | X [: ,i ] | ) and D Y = diag( | Y
[: ,i ] | ) .
Assuming these are invertible, let ˜ X
=
XD − 1 X and ˜ Y
=
Y D − 1 Y .
Let δ be the permutation Procrustes distance between ˜ X, ˜ Y , deﬁned by δ :=
min
P ∈ Σ
d | ˜ X − ˜ Y P
| (as pointed out in [Wil+21] this can be computed via the linear sum assignment problem).
Then the G ReLU -Procrustes measure is
[Equation]
The factor of 2 √ d ensures this lies in [0 , 1] , and equals 1 if (and only if) the condition ∗ of (ii) holds.
[Table] with different random seeds. We apply G ReLU -Procrustes and orthogonal Procrustes similarities to 4 different hidden representations from Myrtle CNNs in table 2 and many more layers of ResNet20s in ﬁg. 9, all trained on CIFAR-10 [Kri09].
In keeping with the discussion of section 4, we only consider permutations or orthogonal transformations of channels (for details on how this is implemented we refer to appendix D.7).
We see that distinct representations register less similarity in terms of G ReLUProcrustes than they do in terms of orthogonal Procrustes.
This makes sense as similarity up to
G ReLU -transformation requires a greater degree of absolute similarity between representations than is required of similarity up to orthogonal transformation (the latter being a higher-dimensional group containing all of the permutations in G ReLU ).
Otherwise patterns in G ReLU -Procrustes similaritiy largely follow those of orthogonal Procrustes, with similarity between representations decreasing as one progresses through the network, only to increase again in the last layer.
This correlates with the stitching penalties of ﬁg. 1, which increase with depth only to decrease in the last layer.
Deﬁnition 5.3 ( G ReLU -CKA) .
Assume that X and Y are data matrices that have been centered by subtracting means of rows: X ← X − 1 d 11 T X and Y ← Y − 1 d 11 T Y .
Let ˜ X
=
XD − 1 X and ˜ Y =
Y D − 1 Y .
Let ˜ x 1 , . . .
, ˜ x
N be the rows of ˜ X , and similarly for ˜ Y .
Form the matrices K, L ∈ R N × N ≥deﬁned by K ij =
max(˜ x i (cid:12) ˜ x j ) and L ij =
max(˜
y i (cid:12) ˜
y j ) where (cid:12) is the Hadamard product.
Then the G ReLU -CKA for X and Y is deﬁned as:
[Equation]
where HSIC 1 is the unbiased form of the Hilbert-Schmidt independence criterion of [NRK21, eq. 3].
Symmetry of the max function ensures (i), the Cauchy-Schwarz inequality ensures µ CKA ( X, Y ) ∈ [0 , 1] , and we claim that µ CKA ( X, Y )
= 1 if the condition ∗ of (ii) is met.
We do not claim ‘if and only if’, however we point out the following in lemma E.44: if A is a matrix such that max( Ax 1 (cid:12) Ax 2 )
= max( x 1 (cid:12) x 2 ) for all x 1 , x 2 ∈ R d , then A is of the form PD where P is a permutation matrix and D is diagonal with diagonal entries in {± 1 } .
In fact, µ CKA is simply an instance of CKA for a the “ max kernel.”
Lemma 5.5.
The function κ : R d × R d → R deﬁned by k ( x, y ) =
max( x (cid:12) y ) is a positive semi-deﬁnite kernel.
As with CKA [Kor+19], this metric makes sense even if X, Y are datasets in R d , R d (cid:48) respectively with d (cid:54) =
d (cid:48) .
Results for a pair of Myrtle CNNs trained on CIFAR-10 with different random seeds, as well as standard orthogonal CKA for comparison, are shown in ﬁg. 5.
Analogous results for
ResNet20s are shown in ﬁg. 2.
We ﬁnd that G ReLU -CKA respects basic trends found in their orthogonal counterparts: model layers at the same depth are more similar, early layers are highly similar, and the metric surfaces the block structure of the ResNet in ﬁg. 2 (layers inside residual blocks are less similar than those at residual connections).
One notable difference for G ReLU -CKA in ﬁgs.
2 and 5 is that the similarity difference between early and later layers in the orthogonal CKA (discussed for ResNets in [Rag+21]) shown in (b) is less pronounced in (a), and in fact later layers are found to be less similar between runs.
We found similar results for stitching in ﬁgs.
1 and 6.
Interpretability of the coordinate basis
In this section we explore the conﬂuence of model interpretability and intertwiner symmetries using network dissection from [Bau neurons of a hidden layer and single, pre-deﬁned concepts (see appendix F.1 for the methodology).
We adapt an experiment from [Bau+17] to compare the axis-aligned interpretability of hidden activation layers with and without an activation function.
Bau et al. compares the interpretability of individual neurons, measured via network dissection, with that of random orthogonal rotations of neurons.
We likewise rotate the hidden layer representations and then measure their interpretability.
Using the methodology from [Dia05], we deﬁne a random orthogonal transform Q drawn uniformly from
SO ( n ) by using Gram-Schmidt to orthonormalize the normally-distributed QR =
A ∈ R n 2 .
Like in [Bau+17], we also consider smaller rotations Q α ∈ SO ( n ) where 0 ≤ α ≤ 1 , where α is chosen to form a minimal geodesic rotating from I to Q .
[Bau+17] found that the number of interpretable units decreased away from the activation basis as α increased for layer5 of an AlexNet.
We compare three models trained on ImageNet: a ResNet-50, a modiﬁed ResNet-50 where we remove the ReLU on the residual outputs (training details in appendix F.3), and a ConvNeXt [Liu+22] analog of the ResNet-50, which also does not have an activation function before the ﬁnal residual output.
We give results in ﬁg. 3, and provide sample unit detection outputs and full concept labels for the ﬁgures in appendix F.2.
As was shown in [Bau+17], interpretability decreases as we rotate away from the axis for the normal ResNet-50 in appendix F.3.
On the other hand, with no activation function, neuron interpretability does not drop with rotation for the modiﬁed ResNet-50 and the ConvNeXt.
We note that the models without residual activation functions also have far fewer concept covering units for a given basis.
Interestingly, while the number of interpretable units remains constant for the residual output of the modiﬁed ResNet-50, for the ConvNeXt model it actually increases .
We ﬁnd similar results, where the number of interpretable units increase with rotation, for the convolutional layer inside the residual block for the modiﬁed ResNet-50 in ﬁg. 19.
Limitations
Our theoretical analysis in section 3 does not account for standard regularization techniques that are known to have symmetry-breaking effects (for example weight decay reduces scaling symmetry).
More generally, we do not account for any implicit regularization of our training algorithms.
As illustrated in ﬁgs.
1 and 6, stitching with intertwiner groups appears to have signiﬁcantly more architecture-dependent behaviour than stitching with arbitrary afﬁne transformations (however, since different architectures have different symmetries this is to be expected).
Our empirical tests of the dissimilarity measures in section 5 are limited to what [Kor+19] terms “sanity tests”; in particular we did not perform the speciﬁcity, sensitivity and quality tests of [DDS21a].
Conclusion
In this paper we describe groups of symmetries that arise from the nonlinear layers of a neural network, calculate these symmetry groups for a number of different types of nonlinearities, and explore their fundamental properties and connection to weight space symmetries.
Next, we provide evidence that these symmetries induce symmetries in a network’s internal representation of the data that it processes, showing that previous work on the internal representations of neural networks can be naturally adapted to incorporate awareness of the intertwiner groups that we identify.
Finally, in the special case where the network in question has ReLU nonlinearities, we ﬁnd experimental evidence that intertwiner groups justify the special place of the activation basis within interpretable
AI research.
Acknowledgements
This research was supported by the Mathematics for Artiﬁcial Reasoning in Science (MARS) initiative at Paciﬁc Northwest National Laboratory.
It was conducted under the Laboratory Directed
Research and Development (LDRD) Program at at Paciﬁc Northwest National Laboratory (PNNL), a multiprogram National Laboratory operated by Battelle Memorial Institute for the U.S. Department of Energy under Contract DE-AC05-76RL01830.
The authors would also like to thank Nikhil Vyas for useful discussions related to this work.
A Societal Impact
Though deep learning models are in the process of being deployed for safety critical applications, we still have very little understanding of the structure and evolution of their internal representations.
In this paper we discuss one aspect of these representations.
We hope that by better illuminating the inner workings of these networks, we will be a small part of the larger effort to make deep learning more understandable, reliable, and fair.
B Code availability
We are in the process of releasing code — once available it will be located at https://github.com/davisrbr/modelsyms.
C Examples
We ﬁrst give an example of two networks with distinct weights which are functionally equivalent.
Let f be a 2 layer network with ReLU activations and weight matrices
[Equation] = [Equation] 0(cid:21) and W 2 = (cid:20) 30(cid:21) (and biases = 0).
Let ˜ f be a network with the same architecture, but with weights
W 1 = (cid:20) 01(cid:21) and W 2 = (cid:20) 01(cid:21) .
Then one can verify that ˜ f
( x ) =
f
( x ) for all x ∈ R , but that the weights of f and ˜ f differ.
We also work through a small example of φ σ n where n = 2 .
Assume that σ is the ReLU nonlinearity.
Then,
A = (cid:20) 02(cid:21) belongs to G σ 2 , and we can compute directly that
[Equation] where in the last equality we used the fact that ReLU( ax ) = a ReLU( x ) when a is positive. On the other hand,
[Equation] 02(cid:21) ◦ ReLU( (cid:20) xx(cid:21) )
= (cid:20) 02(cid:21) (cid:20) ReLU( x 1 )
ReLU( x 2 ) (cid:21) = (cid:20) ReLU( x 2 ) 2 ReLU( x 1 ) (cid:21) .
D Experimental Details
In this section we provide additional experimental results, as well as implementation details for the purposes of reproducibility.
All experiments were run on Nvidia GPUs using PyTorch [Pas+19].
D.1 Sampling pairs of models trained with different random seeds
We began by training 100 models with different random seeds (i.e. with independent initializations and different random batches) for each of the following architectures:
(i) Myrtle CNN: a simple 5-layer feed-forward CNN with batch normalization.(ii) ResNet20: a ResNet tailored to the CIFAR-10 dataset (numbers of channels are 16 , 32 ,respectively in the 3 residual blocks).
(iii) ResNet18: an ImageNet-style ResNet adapted to the input size of CIFAR-10 — much wider than the above (numbers of channels are
128 , 256 respectively in the 3 residual blocks
More detailed architecture schematics are included in ﬁgs. 26a, 27a and 28a.
All models were trained for 50 epochs using the Adam optimizer with PyTorch’s default settings.
We use a batch size of 32, initial learning rate 0 .
001 and 4 evenly spaced learning rate drops with factor 0 .
We augment data with translations of up to 2 pixels (padded as necessary with the mean RGB value for CIFAR-10) and left-right ﬂips, and we save the weights with best validation accuracy. In the rotation penalties experiment of ﬁg. 4 the ﬁne-tuning stage uses the same hyperparameters as the initial training phase (though of course only a subset of parameters recieve gradient updaates during ﬁne-tuning).
Training this many CIFAR-10 models on a reasonable budget of time and computing resources was greatly aided by the excellent FFCV library [Lec+22].
In the later stitching and dissimilarity measure experiments, we sample pairs of models from these “zoos” uniformly with replacement (but of course making sure that the two models in the pair are distinct).
Thus the cost of training hundreds of models is amortized across many runs of stitching and dissimilarity measurement; this can be also viewed as bootstrap estimation of our experimental quantities of interest using empirical samples from certain distributions of CIFAR-10 models.
D.2 Stitching Experiments
For stitching layers, we train for 20 epochs with batch size 32 and learning rate 0 .
001 (with no drops), however we use vanilla SGD with no momentum (we found the approximate second-order and/or momentum aspects of Adam interacted in complicated ways with the PGD algorithm described in appendix D.2.1 below, even after following some helpful advice from the Internet 7 ).
Augmentation is described in the previous paragraph.
We parameterize reduced rank 1-by-1 convolutions as a composition of 2 1-by-1 convolutions, with in _ channels , out _ channels = in _ channels , rank and rank , in _ channels respectively.
In contrast to [BNB21] we omit both batch norm and bias from stitching layers (to stick closely to the statement of theorem 4.2).
D.2.1 Approximate Optimization over Permutation Matrices
By far the most complicated stitching layer is the one using G ReLU , which we describe here.
Recall that G ReLU is equal to the n × n matrices of the form PD , where P ∈ Σ n is a permutation matrix and D is a diagonal matrix with positive entries We parameterize D simply as D = diag ( λ i ) where λ 1 , . . .
, λ n l ∈ R ≥ 0 — we preserve non-negativity during training by a projected gradient descent step D ← ReLU( D ) .
During stitching layer training, we parameterize P as a doubly stochastic matrix, that is, an element of the Birkhoff polytope
[Equation]
— after each gradient descent step we project P back onto B by the operation P ← ReLU( P ) followed by P ← sink( P ) , where “ sink ” denotes Sinkhorn iterations.
These consist of T iterations of
A ← A diag( 1 T A ) − 1 followed by A ← diag( A 1 ) − 1 A (it is a theorem of Sinkhorn that this sequence converges to a doubly stochastic matrix of the form
DAE with D, E positive diagonal matrices [Sin64]).
We use T = 16 in all experiments (this choice drew on the work of [Men+18]).
In addition, we add a regularization term − α | P | 2 to the stitching objective, where α > 0 is a hyperparameter (the motivation here is that permutation matrices are precisely the elements of B with maximal (cid:96) 2 -norm).
Unless stated otherwise in our experiments α = 0 .
We did experiment with choosing α by cross validation and found the particular choice of α was not crucial; see appendix D.5 for further details.
At evaluation time, we threshold P to an actual permutation matrix via the Hungarian algorithm (speciﬁcally its implementation in scipy .
optimize.linear_sum_assignment [Vir+20]).
This amounts to
[Equation]
As stated above, we train for 20 epochs with batch size 32 and learning rate 0 .
001 (with no drops), using SGD with momentum 0.9.
However, we allow the permutation factor to get a “head start” by keeping D ﬁxed at the identity I for the ﬁrst 10 epochs.
This is probably not essential, as shown in appendix D.5.
Finally, before evaluating the stitched model on the CIFAR-10 validation set, we perform a nogradient epoch on the training data with stitching layer P eval .
This is critical as it allows the batch normalization running means and variances in later layers to adapt to the thresholded permutation matrix P eval ; observe that if we omitted this step, during evaluation the “batch normalization layers” would not even be performing batch normalization per se, since their running statistics would be computed from features produced by a layer P train no longer in use.
As an aside, we also experimented with the differeniable relaxation of permutation matrices SoftSort [PE20].
Our ﬁnal results were comparable, however this method took far longer ( > 10 × ) to optimize than the Birkhoff polytope method.
It is perhaps of interest that we used SoftSort on permutations far larger than those of [PE20] (e.g., the 512 channels of late layers of our Myrtle CNN).
The next section (appendix D.2.2) contains some of our technical ﬁndings.
We wish to aknowledge a couple articles, [Fog+13] and [LW14], that provided us with useful backround on optimization over doubly stochastic matrices.
D.2.2 Stitching with SoftSort
We parameterized D simply as D = diag( e λ i ) where λ 1 , . . .
, λ n l ∈ R .
During stitching layer training, we parameterized P using SoftSort [PE20], a continuous relaxation of permutation matrices given by the formula
[Equation]
, where s ∈ R n l , sort( s ) denotes s sorted in descending order, and softmax is applied over rows.
The parameter τ > 0 controls softmax temperature, and we were only able to obtain reasonable results when tuning it according to τ ≈ 1 /n l .
At validation time, we threshold P to an actual permutation matrix by applying arg max over rows as in [PE20].
D.3 Stitching and G ReLU -dissimilarity measures for ResNets
Here we include further results for ResNet20 and ResNet18 architectures.
Figure 6 and ﬁg. 7 include results for full 1-by-1 convolution, reduced randk 1-by-1 convolution and G ReLU 1-by-1 convolutions stitching in the ResNet20 and ResNet18 architectures respectively.
Note that in general, layers inside residual blocks incur higher penalties, consisent with remark 3.6.
This holds even in the full 1-byconvolution case, a ﬁnding that to the best of our knowledge is new.
In the case of ResNet20 we also observe that the relative ranking of the different stitching constraints tends to change inside of residual blocks: whereas G ReLU stitching consistently outperforms rank(and sometimes rank 2) stitching outside residual blocks, it consistently underperforms all strategies inside residual blocks.
Lastly, we remark that the ResNet20 is signiﬁcantly narrower than the Myrtle
CNN (channels are 16, 32, 64 vs. 64, 128, 256, see ﬁgs. 27a and 28a), and hence the low-rank transformations account for a larger proportion of the available total rank (for example, in early layers of the ResNet20 rank 4 is 0 .
25 · fullrank whereas in the early layers of the Myrtle CNN rank 4 is 0 .
0625 · fullrank ).
Heuristically, in the narrower network low-rank transformations may sufﬁce to align for a larger fraction of the principal components of hidden features.
We also observe generally lower stitching penalties in the ResNet18 with the exception of the penultimate inside-a-residual-block layer — we do not have a satisfactory explanation for random chance performance at that layer.
We also remark that while the penalties in ﬁg. 6 are signiﬁcantly higher than those in ﬁg. 1, especially in later layers, we also saw signiﬁcant dissimilarity in ﬁg. 2 (a), especially in later layers.
We also modify the ResNet20 to use the LeakyReLU activation function and train models with different negative slopes s .
The accuracy for two models trained with different random seeds at different LeakyReLU is given in table 3.
We perform G ReLU stitching in ﬁg. 8.
Note that for a negative slope s = 1 , the activation function is the identity.
We ﬁnd the results difﬁcult to interpret due to the signiﬁcant decrease in CIFAR-10 accuracy for larger s .
With this being said, unlike for s << 1 , we note that the stitching penalties for s = 1 (and to a lesser extent, s = 0 . 9 ) are mostly constant throughout the layers of the network.
This is most prominent for the ﬁnal two ResNet layers ( 72 and 75 ), where the stitching penalty for models with small LeakyReLU slopes is the lowest.
[Table]
Figure 9 contains G ReLU and orthogonal Procrustes dissimilarities for the ResNet20.
The 2 measures seem qualitatively quite similar in this case.
For the most part the same applies to the ResNet18 in ﬁg. 10, with the exception of layer 70 (penultimate inside-a-residual-block layer), where we see high
G ReLU similarity , in conﬂict with both ﬁg. 7 and ﬁg. 11 below.
We include G ReLU and orthogonal CKA dissimilarities for the wider ResNet18 in ﬁg. 11.
For the most part the qualitative remarks on ﬁg. 2 apply here as well — note also the extreme dissimilarity in layer 70 (in both G ReLU and orthogonal cases) consistent with ﬁg. 7. [Figure]
D.4 Stitching for a Vision Transformer
Here we include an additional stitching experiments with vision transformers from [Has+21] trained on CIFAR-10.
Figure 12 include results for linear stitching and G ReLU stitching after each transformer encoder layer.
The large stitching penalties for G ReLU are expected due to the lack of activation functions after the linear (feedforward) layers for each encoder layer.
We train 10 Compact Convolutional Transformers with sinusoidal positional encodings and six transformer blocks.
The average model accuracy was 98% using the distributed training-from-scratch recipe from [Has+21], which includes 6e − 2 weight decay, augmentations (namely mixup [Zha+18] and CutMix [Yun+19]), label smoothing, and AdamW with a learning rate of 55e − 5 with cosine scheduling.
{+D.
+} Choosing the negative- (cid:96) 2 regularization multiplier α with cross validation
Here we brieﬂy describe an experiment in which the multiplier α of appendix D.2.1 is chosen by cross validation.
Most of the details are as in appendix D.2.
However, we create a random 80-20 split of the
CIFAR10 training set into a smaller training and cross-validation set.
We then learn G ReLU stitching layers for each α ∈ { 10 k | k
= − 3 , − 2 , . .
, 1 } , as in appendix D.2.1, with the exception that we only optimize over our training split for 5 epochs and do not give the permutations a head start.
Then, the α corresponding to highest accuracy on our cross validation set is selected, the corresponding model weights are loaded and we report accuracy on the regular CIFAR10 validation set.
In ﬁg.we obtain very similar results to those in ﬁg. 1.
Perhaps more interestingly, in ﬁg. 14 we see that [Figure] there is substantial variance in the α selected by cross-validation, at all layers of our Myrtle CNN network — for reference, α = 0 .
1 is used in the rest of this paper.
This suggests that the particular choice of α is not essential to our method.
Results for ResNet architectures are qualitatively similar and omitted for brevity.
D.6 Stitching with (cid:96) 1 -regularized (a.k.a. LASSO) fully-connected layers
In this section we present results of a small experiment stitching with full 1-by-1 convolutional layers with (cid:96) 1 penalty λ | W | 1 , where | W | 1 = (cid:80) ij | W ij | , as in [Csi+21].
We vary λ ∈ { 0 . 001 , 0 . 01 , 0 . 1 } and also tried λ = 1 but found the stitching optimization to be unstable due the magnitude of the (cid:96)penalty (possible this could have been counteracted by decreasing the learning rate).
We also record the sparsity of the stitching weights — if n l is the relevant channel dimension, and hence also the number of rows/columns in the square stitching matrix W , we measure this as
[Equation]
where τ is a threshold, in our experiments chosen to be 0 . 001 .
Note that the sparsity of a G ReLU is equal to n 2 l − n l n 2 l [Equation] n l .
Figure 15 illustrates the results of these experiments, and seems to show that G ReLU layers achieve low stitching penalties for their sparsity levels.
Also note that in the ﬁnal layer the scatter points corresponding to G ReLU and λ = 0 .
01 nearly overlap.
D.7 Implementing dissimilarity measures
As mentioned in section we aim to capture invariants to permuting and scaling channels, but not spatial coordinates.
This requires some care; practically speaking it means we cannot simply ﬂatten feature vectors.
In all cases we compute our measures over the entire CIFAR-10 validation set.
In particular, we do not require batched computations as in [NRK21].
D.7.1 Procrustes
As in [DDS21b; Wil+21]
[Equation]
so it sufﬁces to consider minimizing the Frobenius norm-squared, and expanding as
[Equation]
we see that this is equivalent to maximizing tr( ˜ X T ˜ Y P ) .
In our case X, Y have shape ( N, C, H, W ) where N is the size of the entire CIFAR-10 validation set and C, H, W are the channels, height, and width at the given hidden layer respectively.
We want P to be a C × C permutation matrix.
Hence for ˜ X T ˜
Y we compute the tensor dot product
[Equation]
The same method is used for orthogonal
Procrustes, where instead of scipy .optimize .linear_sum_assignment
we use the nuclear norm of eq.
(D.2)
as in [DDS21a].
D.7.2 CKA
In this case for a set of hidden features X of shape ( N, C, H, W ) as above, we ﬁrst subtract the mean over all but the channel dimension: [Equation]
[Equation]
w (cid:48) and divide by the norms over all but the channel dimension: [-
-][Equation] n [Equation]
c,h
[Equation]
w ←
[Equation]
(cid:113)(cid:80)
[Equation] Next, we compute a tensor dot product of X with itself over spatial dimensions , to obtain the shape ( N, N, C ) tensor J m,n,c [Equation]
= (cid:88) h,w
[Equation]
,c,h,w and ﬁnally we apply max over the channel dimension to get
[Equation]
Remark D.3 .
It could be interesting to refrain from applying a dot product over spatial dimensions, and thus measure not only similarity between hidden features of different images, but similarity between hidden features of different images at certain locations .
However, the memory requirements would have been far beyond our computational limits. [Table] [Figure]
D.8 Dissimilarity measures for network with constant channel width
A notable feature of our plots in ﬁgs. 2, 5 and 11 is that the G ReLU -CKA exhibits a much more signiﬁcant decay with network depth than its orthogonal counterpart.
From a skeptical perspective, we thought this could have something to do with dimensionality.
All the networks we looked at up to this point had the feature that their channel dimension grows exponentially with depth (as seen in the last 3 ﬁgures of the appendix).
When we compute the kernels max(˜ x _ i (cid:12) ˜ x _ j ) , we encounter maxima of larger and larger sets of random variables as the channel dimension increases.
If the products inside these maxima were independent normal random variables (we are not claiming this is a reasonable heuristic), we’d expect the max to grow like Φ − 1 (1 − 1 n l ) where n l is the channel dimension.
It seemed possible that something along these lines could cause G ReLU -CKA to drift as depth (in our experiments correlated with channel dimension) increases.
Note that the dot product kernel seems comparatively immune, since (with the same heuristics of normal distribution) the expected value of (cid:104) ˜ x, ˜ y (cid:105) is 0 regardless of dimension.
Motivated by this train of thought, we evaluated all 4 dissimilarity measures of section 5 on a variant of our Myrtle CNN with constant channel dimension .
The architecture of this network is identical to the one shown in ﬁg. 26a with the exception that all channel dimensions are 512.
In table 4 and ﬁg.we see that these constant width CNNs exhibit qualitatively very similar dissimilarity measures as their non-constant width counterparts.
This suggests that the G ReLU -CKA decay with network depth is not an artifact of increasing channel dimension.
We speculate that it’s possible that the decay of G ReLU -CKA is due to something like the superposition hypothesis for hidden layer features [Elh+22; Ola+20].
Roughly, in overcomplete cases where the model can use more features than basis directions in a hidden layer, it may be encoding m nearly orthogonal features across n < m basis directions.
If this encoding is not consistent across random seeds, we expect G ReLU -CKA to be smaller.
Finally, polysemanticism may increase with depth.
In a simple thought experiment, if each basis direction in layer l has a features encoded, layer l +will have 2 a features per direction if it each neuron in l + 1 simply sums over two neurons in l .
Again assuming the combinations of features occuring in this polysemanticism vary accross random seeds, we would expect G ReLU -CKA to be smaller.
Simply put, superposition and polysemanticism would seem to preclude alignment of the hidden features of different networks with permutations and scaling alone.
E Proofs E.1 A proof of lemma 3.1, plus some abstractions thereof
Proof of lemma 3.1.
Since by deﬁnition G σ n ⊆ GL n ( R ) , to prove G σ n is a subgroup it sufﬁces to show that if A 1 , A 2 ∈ G σ n then A 1 A − 1∈ G σ n .
By hypotheses, there are matrices B 1 , B 2 ∈
GL n ( R ) so that
[Equation]
and
[Equation]
Applying A − 1 2 on the right hand side of eq.
(E.1) gives
[Equation]
On the other hand, applying A − 1 2 on the right hand side of eq.
(E.2) gives σ n = B 2 ◦ σ n ◦
( A [Equation] 1 2 ) and hence
[Equation]
Combining eqs.
(E.3) and (E.4) we obtain
[Equation]
and hence G σ n is a subgroup.
Next, we solve
[Equation]
for B in terms of A by evaluating both sides at [Equation]
, e n ∈ R n (standard basis vectors).
Letting
A [: , j ] denote the j -th column of A we obtain [Equation]
j ) , for [Equation] [Equation] . , n and stacking these columns to obtain the full n × n matrix yields
[Equation]
where I ∈ GL n ( R ) is the identity matrix and σ ( A ) denotes σ applied to the coordinates of A (similarly for σ ( I ) ).
As σ ( I ) is invertible by hypotheses, this implies B = σ ( A ) σ ( I ) − 1 =: φ σ ( A ) , and ﬁnally substituting B i = φ σ ( A i ) for i
= 1 , 2 in eq.
(E.5) shows that
[Equation]
while at the same time
[Equation]
so that φ σ ( A 1 A − 1 2 ) ◦ σ n = φ σ ( A 1 ) φ σ ( A 2 ) − 1 ◦ σ n .
Using the invertibility of σ ( I ) one more time, we conclude
[Equation]
which implies φ σ is a homomorphism.
Remark E.6 (for the mathematically inclined reader) .
Here is a more abstract deﬁnition of G σ n that makes lemma 3.1 appear more natural: let X be a topological space with a continuous (left) action of a topological group G .
There is a natural (right) action of G on C ( X, R ) by precomposition ( ( f, g ) (cid:55)→ f ◦ g ).
For any subspace V ⊆ C ( X, R ) deﬁne
[Equation] (that is, the elements of G stabilize V as a subspace , but not necessarily pointwise — one can show this is always a subgroup of G ).
Then, for every such subspace V , the group G V acts linearly on V , and if we have a basis f 1 , . . .
, f n ∈ V , we can obtain a matrix representation of G in GL n ( R ) .
To obtain the special case in lemma 3.1, we take X = R n , G = GL n ( R ) with the usual action, and
V to be the subspace spanned by the functions
f i ( x 1 , . . .
, x n )
= σ ( x i ) .
The condition that σ ( I ) is invertible is equivalent to the condition that the column space of the matrix ( f i ( e j ) is n -dimensional, which in turn implies V is n -dimensional.
We end this section with a lemma that allows for easy veriﬁcation that σ ( I ) is invertible.
We used this on all the activation functions considered in table 1.
Lemma E.7.
Let σ : R → R be any function and let I ∈ GL n ( R ) be the identity matrix.
Then σ ( I ) is invertible provided
[Equation]
Proof.
Let N = 11 T − I .
Then
[Equation]
Note that the eigenvalues of 11 T are n (corresponding to eigenvector 1 ) and 0’s (corresponding to the orthogonal complement of 1 ).
For any linear operator A ∈ Mat n ( R ) with eigenvector/eigenvalue pair ( v, λ ) , v is easily seen to be an eigenvector of A − I with eigenvalue λ − 1 .
Hence the eigenvalues of N are n − 1 and − 1 , and it follows that the eigenvalues of σ ( I ) are
[Equation]
which are all non-zero if and only if eq.
(E.8) holds.
Remark E.9 .
In particular eq.
(E.8) holds when σ (1)
= 1 , σ (0) =
(which holds for example when σ ( x ) = ReLU( x ) or σ ( x ) = x d ).
In this situation, σ ( I ) =
I and φ σ ( A ) =
σ ( A ) (coordinatewise application of σ ).
For example, if σ ≥ 0 is non-negative, then φ σ ( A )
=
σ ( A ) has non-negative entries.
E.2 Calculating intertwiner groups (for table
We begin with two lemmas:
the ﬁrst puts a “lower bound” on G σ n and the second is a “differential form” of the deﬁnition of the intertwiner group from section 3.
Together, these two results effectively allow us to reduce calculation of intertwiner groups to the n = 1 case.
Lemma E.10 (cf.
[GBC16, §8.2.2], [Bre+19, §3]) .
G σ n always contains the permutation matrices Σ n , and φ σ restricts to the identity on Σ n .
Proof.
If A ∈ Σ n is a permutation matrix, so that Ae i =
e π ( i ) where π is a permutation of { 1 , . . . , n } then for any x ∈ R n we observe
[Equation]
which is exactly σ applied coordinatewise to (cid:80) i x
[Equation] e [Equation] [Equation]
= Ax .
Corollary E.11.
If A ∈ GL n ( R ) , P ∈ Σ n , and AP ∈ G σ n or PA ∈ G σ n , then A ∈ G σ n .
Proof.
If B = AP ∈ G σ n , then A = BP − 1 , where B ∈ G σ n by hypothesis and P ∈ G σ n by lemma E.10.
The result follows as G σ n is a group (lemma 3.1) and hence closed under multiplication.
The other case is similar.
Lemma E.12.
Suppose A, B ∈ GL n ( R ) and σ n ◦
A = B ◦ σ n .
Suppose x = ( x 1 , . . .
, x n ) T ∈ R n and assume σ is differentiable at x 1 , . . .
, x n as well as
[Equation]
for i = 1 , . . .
, n.
Then, diag( σ (cid:48)
(( Ax ) i )
| i = 1 , . . . , n ) A = B diag( σ (cid:48) ( x i ) | i = 1 , . . . , n ) (here diag : R n → Mat n × n ( R ) takes a vector to a diagonal matrix).
Explicitly, for each i, j ∈ { 1 , . . . , n }
[Equation]
Proof.
By the chain rule [Rud76, Thm. 9.15], and since the differential of a matrix is itself,
[Equation]
Finally, by the deﬁnition of σ n
[Equation]
Theorem E.14.
Suppose σ is non-constant, non-linear, and differentiable on a dense open set with ﬁnite complement.
Then,
(i)
Every A ∈ G σ n is of the form PD , where P ∈ Σ n and D is diagonal.
(ii) For a diagonal D = diag( λ 1 , . . . , λ n ) ∈ G σ n , we have λ i ∈
G σ 1 for i = 1 , . .
. , n and φ σ (diag( λ 1 , . . . , λ n ))
=
diag( φ σ ( λ 1 ) , . .
. , φ σ ( λ 1 )) where we make a slight abuse of notation: on the right hand side φ σ is the homomorphism G σ 1 → GL 1 ( R ) .
In particular, φ σ is determined by lemma E.10 and its behavior for n = 1 .
Proof.
For any A ∈ G ReLU we observe that the differentiability hypotheses of lemma E.12 holds for x ∈ U where U is a dense open set with measure-0 complement.
Indeed, if t 1 , . . . , t M ∈ R are the points where σ fails to be differentiable, we can take U to be the complement of the hyperplane arangement given by
[Equation]
Fix a row i — the matrix A is invertible by hypotheses, and so there must be some j such that a ij (cid:54) =(otherwise the i -th row of A is 0).
For any x ∈ U , we have by lemma [-E.
-]E.
[Equation]
and we claim that this cannot hold unless a ik = 0 for k (cid:54) =
j .
First, there is a ( x 1 , . . .
, x n ) ∈
U such that σ (cid:48) ( x j ) (cid:54) =
(otherwise σ would be constant).
Next, ﬁxing x j at a value with σ (cid:48) ( x j ) (cid:54) = 0 and rearranging eq.
(E.15) we have
[Equation]
By hypothesis σ is non-linear and so σ (cid:48) is non-constant — hence if there were some a ik (cid:54) = 0 for k (cid:54) =
j , the left hand side of eq.
(E.16) would be non-constant.
We have shown each row of A has at most one non-0 entry a ij and that a ij (cid:54) = 0 .
For A to be invertible, it must be that these non-0 entries land in distinct columns .
This is exactly the form described in item (i).
Next, we note that for any ij (without assuming a ij (cid:54) = 0 ) eqs.
(E.15) and (E.16) tell us a ij = 0 =
⇒ b ij = 0 , and hence if D = diag( λ 1 , . .
. , λ n ) ∈ G σ n and σ n ◦ D
= E ◦ σ n (i.e. E = φ σ ( D ) ), it must be that
E = diag( µ 1 , . . . , µ n ) for some µ 1 , . . . , µ n ∈ R .
Now the equation σ n ◦ D =
E ◦ σ n is equivalent to σ ( λ i x i )
=
β i σ ( x i ) for i = 1 , . . . , n which in turn is equivalent to λ i ∈ G σ 1 and β i =
φ σ ( λ i ) for i = 1 , . . . , n , proving item (ii).
In light of theorem E.14, to ﬁll in the table of table 1 it will sufﬁce to deal with the n = 1 cases, which we do below.
ReLU .
We remark that this is just the “positive homogeneous” property of ReLU , which is quite well known (cf.
[GBC16, §8.2.2], [FB17, §2], [Kun+21, §3], [Men+19, §3], [RK20, §3, A], [Yi+19, §2-3]).
Using remark E.9 if a ∈ G σ [Equation]
If a < 0 then setting x = − 1 results in a = 0 , a contradiction.
So a > 0 and max { 0 , ax } = a max { 0 , x } , showing φ σ ( a )
= a .
Modiﬁcations for LeakyReLU .
By deﬁnition, for 0 < s (cid:28) 1 .
LeakyReLU
( x, a ) := (cid:26) sx for x <x for x
≥which we may simplify to LeakyReLU( x ) =
sx + (1 − s ) ReLU( x ) .
Suppose now that
LeakyReLU( ax ) =
b LeakyReLU( x ) , or using our simpliﬁcation sax [Equation] sx
[Equation]
ReLU
[Equation] x )) . (E.17) If a < 0 , we may choose x = − 1 to obtain − a = − sa − (1 − s ) a = − bs and x = 1 to obtain sa =
b showing that a = as 2 , which is impossible when 0 < s (cid:28) 1 .
So it must be a > 0 , and then evaluating eq.
(E.17) at x = 1 gives a = b .
The sigmoid case: σ ( x ) = 1 / (1 + e x ) .
We will leverage of a useful fact about the sigmoid function: σ (cid:48)
( x ) is a smooth probability distribution function on R , with σ (cid:48)
( x ) > 0 for all x ∈ R .
[Equation]
If σ ( ax ) =
bσ ( x ) , differentiating with respect to x gives
[Equation]
Using eq.
(*) and the fact that to probability distribution functions are proportional if and only if they are equal, we get σ (cid:48) ( ax ) = σ (cid:48) ( x ) .
Then integrating from −∞ to x tells us σ ( ax ) /a
= σ ( x ) ; setting x =
we see 12 a = 12 , hence a = 1 .
To show φ σ = id , backtracking to eq.
(E.18) and setting x = 0 shows b = a .
The Gaussian RBF case: σ ( x ) = 1 √ 2 π e − x 2 2 .
We make use of several properties of this σ (
x ) : (i) For any a > 0 the function σ ( ax ) is a probability distribution function with mean 0 and variance 1 a 2 ,and with σ ( ax ) > 0 for all x ∈ R and (ii) σ is an even function ( σ ( − x )
= σ ( x ) ).
Now if σ ( ax ) =
bσ ( x ) , then the pdfs σ ( ax ) and σ ( x ) are proportional hence equal by item (i).
Therefore they have the same means and variances — since these are 0 , 1 a 2 and 0 , 1 respectively we conclude a = ± 1 .
Finally, we explain why φ σ ( A ) =
abs( A ) (entrywise absolute value).
Differentiating with respect to x gives
[Equation]
This implies b = 1 when a = 1 .
On the other hand differentiating item (ii) tells us − σ (cid:48)
( − x ) = σ (cid:48) ( x ) , so when a = [-− 
-]−bσ (cid:48)
[Equation]
and hence b = 1 .
The polynomial case: σ ( x ) = x d .
We remark that the description given in table 1 is implicit in [KTB19].
By theorem E.14 we only need to describe φ σ : G σ 1 → GL 1 ( R ) ; for any a (cid:54) [-= 
 (-]=( ax ) d = a d x d
and this shows G σ 1 =
R \ { 0 } and φ σ ( a ) = a d .
[Figure]
E.2.1 Gaussian error linear units ( GeLU s)
Introduced and ﬁrst studied in [HG16], these are deﬁned as GeLU( x ) = x Φ( x ) where Φ is the standard normal cummulative distribution function:
[Equation]
By inspecting plots in ﬁg. 17a, we see that GeLU and ReLU are globally quite similar (they converge as | x | → ∞ ) but that they differ when x within a few standard normal standard deviations of 0.
One can show that G GeLU n = Σ n : indeed, by theorem E.14 it sufﬁces to show that the only λ ∈ R \ { 0 } such that GeLU( λx ) =
φ ( λ ) GeLU( x ) for all x (where φ is some non-zero function of λ ) is λ = 1 .
Expanding, we see that
[Equation]
E.20)
and rearranging this becomes
[Equation]
21) that is, the right hand side is constant as a function of x .
Then Φ( λx ) =
c Φ( x ) , and since Φ is positive it must be c is too.
Moreover it must be λ > 0 , as otherwise Φ( λx ) is monotonically decreasing while Φ( x ) is increasing.
Finally, letting x → ∞ we see that c
= 1 , and from there we conclude λ = 1 by an argument similar to the use of item (i) in the Gaussian RBF case.
Despite the above calculation, it seems natural to ask how far G GeLU is from G ReLU , in other words how badly GeLU fails to be positive homogeneous.
One measure of this is obtained by letting
X ∼ N (0 , 1) be a standard normal variable and computing the root-mean-square error
[Equation]
as a function of λ > 0 , where the expectation is over X .
Here our choice of a standard normal X is motivated by the same reasoning as discissed in [HG16], namely that activation inputs are roughly standard normal, especially in the presence of batch normalization.
Evaluating eq.
(E.22) doesn’t seem particularly tractible analytically, but it does simplify to
[Equation]
In ﬁg. 17b we estimate ξ ( λ ) by sampling X and replacing the expectation with the corresponding average.
Evidently, as λ → ∞ the function ξ ( λ ) becomes linear: since Φ( x ) → 1 x ≥ 0 as λ → ∞ (here 1 x ≥ 0 is the indicator of x > 0 , also known as the Heaviside or unit-step function), the asymptotic slope is (cid:112) E
[ | x ( 1 x ≥ 0 − Φ( x )) | 2 ] ≈ 0 .
E.3 Proof of theorem 3.
Proof.
The explicit description of G (ReLU , n ) in table 1 is enough to show G (ReLU , n ) stabilizes { R ≥
e i | i
= 1 , . .
. , n } .
Indeed, any A ∈ G (ReLU , n ) may be written as PD where D = diag( a i ) for some a i > 0 and P is a permutation matrix associated to a permutation π .
It sufﬁces to show that
P and diag( a i ) each preserves { R ≥
e i | i
= 1 , . .
. , n } .
[-First,diag( a i ) { R ≥ 
 e i | i = 1 , . . . , n } = {-]First,
[-R ≥ 
-]
[-= 1 , . . . , n } = { R ≥ 
 e i | i = 1 , . .-][Equation]
since the a i > 0 so scaling by a i preserves the ray R ≥
e i .
Second,
[Equation]
(the last equality is due to the fact that we only consider the set of rays, not the ordered tuple of rays).
Conversely, if A ∈ GL ( n ) stabilizes { R ≥
e i | i
= 1 , . .
. , n } , then in particular for each j
Ae j ∈ R ≥
e i for some i and as A is invertible, setting j = π ( i ) yields a permutation of { 1 , . .
. , n } .
Moreover,since Ae j (cid:54) =
( A is invertible) there must be some a j > 0 so that Ae j = a j e i .
One can now verify that A = PD where P is the permutation matrix associated to π and D = diag( a j ) which matches the description of G (ReLU , n ) from table 1.
For the “moreover,” we prove the contrapositive, namely that if v = ( v 1 , . .
. , v n ) ∈ R n has at least 2 non-0 coordinates v i , v j , then the G ReLU -orbit of the ray R ≥
v that v generates cannot be ﬁnite.
Indeed, suppose t ≥ 0 and let D =
diag(1 , . .
, 1 , t, 1 , . .
, 1) ( t in the i th position).
The n ×matrix ( v | Dv ) has a 2 × 2 minor (cid:18) v i tv i v j v j (cid:19) with deteriminant (1 − t )[-v i v j (cid:54) = 
-]
v i v j (cid:54) = 0 as long as t (cid:54) = 1 .
(E.24)
Thus v and Dv are linearly independent, and hence deﬁne distinct rays, for all t (cid:54) = 1 .
It follows that any set of rays stabilized by G (ReLU , n ) that contains R ≥ 0 v is uncountable.
E.4 Proof of proposition 3.
To give a rigorous proof we use induction on the depth k ; since this to some extent obscures the main point, we brieﬂy outline an informal proof: consider a composition of 2 layers of the network f with weights W (cid:48) :
[Equation]
Using the deﬁning properties of G σ and φ σ , we can extract A i like
[Equation]
The resulting copy of φ σ ( A i ) on the right hand side of eq.
(E.26) is cancelled by the copy of φ σ ( A i ) −right-multiplying W
i +1 in eq.
(E.25), so that eq.
(E.25) reduces to {+σ ( A i +
+}
{+W i +
 σ ( W i φ σ ( A i − 1 ) − 1 x + b i ) + A i +
+}
b i +1 ) .
In this way, in between any two layers (cid:96) i +1 ◦ (cid:96) i the A i in A
i W i φ σ ( A i − 1 ) − 1 and the φ σ ( A i ) − 1 in
A i +1 W
i +1 φ σ ( A i ) − 1 cancel .
However, the factors φ σ ( A m ) and φ σ ( A m ) − 1 appear on endpoints of the truncated networks f ≤ m , f >m
and so they are not cancelled.
To keep track of W, W (cid:48) while using the notation from section 3, we write (cid:96) i ( x, W )
=
σ ( W i x + b i ) and (cid:96) i ( x, W (cid:48) )
=
σ ( W (cid:48)
i x + b (cid:48) i ) for i < k and so on.
Proof.
By induction on k , the depth of the network.
The case k = 1 is trivial, since there W (cid:48)
=
W and there is nothing to prove.
For k > 1 we consider 2 cases:
Case m = 1 :
In this case let V =
( W 2 , b 2 , . . .
, W k , b k ) and V (cid:48) =
( A 2 W 2 , A 2 b 2 , A 3 W 3 φ σ ( A 2 ) − 1 , A 3 b 3 , . . .
, W k φ σ ( A − 1 k − 1 ) , b k ) , and let g ( x, V ) = (cid:96) k − 1 ( x, V ) ◦ · · · ◦ (cid:96) 1 ( x, V ) and g ( x, V (cid:48) )
= (cid:96) k − 1 ( x, V (cid:48) ) ◦ · · · ◦ (cid:96) 1 ( x, V (cid:48) ) where (cid:96) i ( x, V ) =
σ ( W i +1 x
+ b i +1 ) and similarly for V (cid:48) .
In other words, the weights V, V (cid:48) and function g represent the architecture obtained by removing the earliest layer of f .
Then, f ≤ 1 ( x, W ) =
σ ( W 1 x + b 1 ) and on the other hand
[Equation]
Using the identity σ ( A 1 z )
=
φ σ ( A 1 ) σ ( z ) for any z ∈ R n 1 , we obtain
[Equation]
This shows f ≤ 1 ( x, W (cid:48) )
=
φ σ ( A 1 ) ◦ f ≤ 1 ( x, W ) .
Next, f > 1 ( x, W ) =
g ( x, V ) but because V (cid:48) 1 =
A 2 W 2 whereas W (cid:48)
=
A 2 W 2 φ σ ( A 1 ) −
[Equation]
By induction on k , we may assume g ( x, V ) =
g ( x, V (cid:48) ) and it follows that f > 1 ( x, W (cid:48) )
= f > 1 ( x, W ) ◦ φ σ ( A 1 ) −[-
-]
Case m > 1 : Deﬁning V, V (cid:48) and g as above, we observe that f ≤ i ( x, W ) =
g ≤ i − 1 ( σ ( W 1 x + b 1 ) , V ) and f ≤ i ( x, W
[Equation]
( W 1 x + b 1 ) , V (cid:48) ) .
(E.27)
By induction on k we may assume g ≤ i − 1 ( x, V (cid:48) )
=
φ σ ( A i ) g ≤ i − 1 ( x, V ) and so f ≤ i ( x, W (cid:48) )
=
φ σ ( A i ) g ≤ i − 1 ( σ ( W 1 x + b 1 ) , V ) =
φ σ ( A i ) f ≤ i ( x, W ) .
Finally, f >i ( x, W ) =
g >i − 1 ( x, V ) and f >i ( x, W (cid:48) )
=
g >i − 1 ( x, V (cid:48) ) and we may assume by induction on k that g >i − 1 ( x, V (cid:48) )
=
g >i − 1 ( x, V ) ◦ φ σ ( A i ) − 1 , hence f >i ( x, W (cid:48) )
= f >i ( x, W ) ◦ φ σ ( A i ) − 1 .
E.5 Proof of theorem 4.
Proof.
Observe that by proposition 3.4, ˜ f >l = f >l ◦ φ σ ( A − 1 l ) .
Hence
[Equation]
If S contains φ σ ( G σ nl ) we may choose ϕ
= φ σ ( A l ) to achieve S ( f, ˜ f, l, ϕ )
=
f as functions.
Similarly, ˜ f ≤ l = φ σ ( A l ) ◦ f ≤ l so if S contains φ σ ( G σ nl ) we may choose ϕ = φ σ ( A − 1 l ) to achieve
S ( f, ˜ f, l, ϕ ) =
˜ f as functions.
In either case eq.
(4.1) holds.
E.6 Symmetries of the loss landscape
Given that the intertwiner group describes a large set of symmetries of a network, it is not surprising that it also provides a way of understanding the relationship between equivalent networks.
Proposition 3.4 has an interpretation in terms of the loss landscape of model architecture.
For any layer n i in f , the action of G σ ni on the weight space W , translates to the obvious group action on the loss landscape.
Corollary E.28.
For any 1 ≤ i ≤ k , the group G σ ni acts on W and for any test set D t ⊂ X × Y , model loss on D t is invariant with respect to this action.
More precisely, if (cid:96) (Φ( W ) , D ) is the loss of Φ( W ) on test set D t , then for any g ∈ G σ ni , (cid:96) (Φ( W ) , D ) =
(cid:96) (Φ( gW ) , D ) .
E.7 Comparing capacities of stitching layers via discretization
Let Mat n × n ( R ) denote the space of n × n matrices.
For r = 1 , . . . , n let Mat rn × n ( R ) ⊆
Mat n × n ( R ) denote the rank r matrices, and let G ReLU n be as described in table 1.
Suppose that each real dimension of Mat n × n ( R ) is replaced by a discrete grid N ( M, (cid:15) )
= {− M + i(cid:15) | i = 0 , . .
, (cid:98) 2 M (cid:15) (cid:99)− 1 } — here (cid:15) could represent the limits of numerical precision in a ﬂoating point number system, and M could represent the maximum numerical magnitude.
The size of each such grid is (cid:98) 2 M(cid:15) (cid:99) , and so the number of points in the resulting mesh grid N ( M, (cid:15) ) n 2 ⊂ Mat n × n ( R ) is (cid:98) 2 M(cid:15)
(cid:99) n 2 .
We now estimate the number of points of G ReLU n and Mat rn × n ( R ) in such a mesh grid.
G ReLU n is a disjoint union of n !
irreducible components, corresponding to the n !
possible permutations P in table 1.
Each of these components is n -dimensional, corresponding to the fact that the factor D in table 1 is an arbitrary positive diagonal matrix.
Hence we obtain
[Equation]
On the other hand, any matrix A ∈ Mat rn × n ( R ) can be written as A = UV where U ∈
Mat n × r ( R ) and V ∈ Mat r × n ( R ) .
These U and V are not unique: given any invertible r × r matrix W ∈ GL r ( R ) , we have A = ( UW )( W − 1 V ) .
From this we obtain the approximation
[Equation]
It follows that
[Equation]
Ignoring the term (2 nr − r 2 ) log 2 , which is independent of M, (cid:15) , we get the approximation log( n !) + n log( M (cid:15) ) −
[Equation]
Next, we make the coarse approximation
[Equation]
with this approximation the expression of eq.
(E.35) is approximated as
[Equation]
From this we conclude that as long as 1.
r ≥ 1 (we actually already assumed this when deﬁning Mat rn × n ( R ) ) and 2.
M(cid:15)
(cid:29) n , which roughly says that the number of grid points per dimension is greater than the number of rows (equivalently columns) in Mat n × n ( R ) , n log
[Equation]
n
[Equation] [Equation]
M (cid:15) ) using item 1 (E.38)
[Equation]
(E
[Equation] [Equation]
n >(E.40) < 0 using item 2.
(E.41)
The upshot is that our approximations and items 1 and 2 imply log
[Equation]
ReLU
[Equation]
−
[Equation]
|
[Equation] rn × n ( R ) ∩ N [Equation] M, (cid:15) )
n 2 | < 0 , and hence (E.42) |
[Equation]
< [Equation] n ×
[Equation]
R [Equation] E. [Equation]
Proof of lemma 5.5.
We must show that for any x 1 , . . .
, x r ∈ R d and any c 1 , . . .
, c r ∈ R that
[Equation]
We use the elementary “max ≥ mean” inequality:
[Equation]
where on the right hand side “ · ” denotes dot product.
This implies
[Equation]
where the last inequality follows from the fact that the dot product is a kernel function.
The last string of inequalities also shows (cid:80) i,j c
i c j max( x i (cid:12) x j ) > 0 when x 1 , . . .
, x r are linearly independent, but does not directly imply the converse.
It would be interesting to know conditions on x 1 , . . .
, x r that imply positive deﬁniteness of the matrix (max( x i (cid:12) x j )) .
Proof of lemma 5.1.
By table 1 any A ∈ G ReLU can be factored as A = PD with P a permutation matrix and D a positive diagonal matrix, and we can obtain a similar factorization B = QE .
Then
[Equation]
where the second equality uses the hypothesis that µ is invariant to right multiplication by positive diagonal matrices, and the third uses the hypothesis that µ is invariant to right multiplication by permutation matrices.
Lemma E.44.
Suppose A is a matrix such that max(
Ax 1 (cid:12) Ax 2 )
=
max( Ax 1 (cid:12) Ax 2 ) for all x 1 , x 2 ∈ R d .
Then, A is of the form PD where P is a permutation matrix and D is diagonal with diagonal entries in {± 1 } .
Proof.
We only need the special case where x = y : observe that
[Equation]
This means that if max( Ax 1 (cid:12) Ax 2 ) =
max( Ax 1 (cid:12) Ax 2 ) , then A preserves the (cid:96) ∞ norm on R d , hence in particular preserves the unit hypercube in R d , and it is known that symmetries of the hypercube have the form PD where P is a permutation matrix and D is diagonal with diagonal entries in {± 1 } (see for example [Ser77, §5.9]).
E.9 Intertwiners and more general architecture features (justiﬁcation of remark 3.6)
Here we brieﬂy discuss how ubiquitous architecture features like batch normalization and residual connections interact with intertwiner groups.
For simplicity in this section we only consider σ =
ReLU .
E.9.1 Batch normalization
A batch normalization layer that takes as input X ∈ R b · n (where b is the batch size and n is the dimension of the layer) and returns [Equation] ˜
[Equation] and where γ, β ∈ R n are the “gain” and “bias” parameters of the batch normalization layer, is invariant under independent scaling of coordinates, that is transformations of the form X ← XD where D is an n × n positive diagonal matrix (see e.g. [BMC15]).
Hence a k -layer ReLU MLP as in section 3 enhanced with batch normalization ( pre -activation, as is standard) is invariant under the action of the slightly larger group (cid:81) k − 1 l =1 ( R n l > 0 (cid:111)
G ReLU nl ) , where the action is given by
[Equation] [Equation]
A k
[Equation] [Equation]
Here,
• c l ∈ R n l > 0 and A l ∈ G ReLU nl , for all l = 1 , . . .
, k −• π : G ReLU nl → Σ n l is the homomorphism setting the positive entries to 1.
The key point is that we get another factor of R n l > 0 at each layer.
We also note that the space of matrices of the form π ( A l ) diag c l is, incidentally, exactly G ReLU nl , and that using eq.
(E.45) one can generalize proposition 3.4 and theorem 4.2 to the case of networks with batch normalization.
E.9.2 Residual connections
We expand on remark 3.6 and explain what exactly transpires with residual connections below.
Suppose we have a k -layer MLP as in section 3 (again for simplicity with σ = ReLU ), together with residual connections between a set of layers R = { r 1 , . . .
, r m } ⊆ { 2 , · · · , k − 1 } 12 :
[Equation]
[Equation] n
[Equation] [Equation] [Equation] k
[Equation]
R n [Equation] σW
[Equation]
σW [Equation] σW [Equation]
[Equation]
id
W
[Equation]
σ
W [Equation] (E.46) (for legibility biases b l are suppressed).
In addition we assume that the depth of each residual block is some ﬁxed, that is r i − r
i − 1 =
b = constant for all i .
First, we claim that a ( A 1 , . . .
A k − 1 ) ∈
(cid:81)
Ll
=1 G ReLU nl stabilizes the function f if (not claiming if and only if) A r i =
A r j for all r i , r j ∈ R .
To see this suppose g i ( x, W ) , i = 1 , . . .
, m are the depth b feedforward networks of the residual blocks, so that the f ≤ r i is given by
[Equation]
Assuming by induction on i that proposition 3.4 applies at the residual connections in R we note that with weights W (cid:48) eq.
(E.47) turns into
[Equation]
[Equation]
Note that proposition 3.4 applies directly to the g i , so we may compute g i ( z, W (cid:48) ) =
A r i g i (
A − 1 r i − 1 z, W ) .
Hence
[Equation]
≤ [Equation] ) and we see that the only way f ≤ r i ( x, W (cid:48) ) =
Bf ≤ r i ( x, W ) for some matrix B is if A r i −
=
A r i , proving our claim.
This also shows that if A r denotes the common value of the A r i for r i ∈ R , we have f ≤ r i ( x, W (cid:48) )
=
A r f ≤ r i ( x, W ) for all i .
It is also true that f >r i ( x, W (cid:48) )
=
f >r i ( A − 1 r x, W ) : observe that
[Equation]
By descending induction on k , we may assume f r i +1 ( x, W (cid:48) )
=
f r i +1 ( A − 1 r x, W ) , and as above g i +1 ( z, W (cid:48) )
=
A r g i +
( A − 1 r z, W ) , so that with weights W (cid:48) eq.
(E.50) becomes
[Equation]
as claimed.
Finally, we describe how stitching fails inside a residual block.
Suppose we use weights W l for l ≤ r i + j where 0 < j < b )
(recall b = depth of our basic block) and weights W (cid:48) l for l > r
i + j .
The resulting stitched network is (attempting to use indentation to increase legibility)
[Equation]
By proposition 3.4 g >j ( z, W (cid:48) )
=
A r g >j
( A − 1 r i + j z, W ) , and we have shown f >r i +
( z, W (cid:48) )
=
f >r i +1 ( A − 1 r z, W ) .
Combining these facts eq.
(E.52) becomes
[Equation]
r i ( x, W ) , W ) , W ) , W ) .
(E.53)
Even after cancelling to remove the A − 1 r A r and in the ideal case where ϕ
=
A r i + j , we are left with an extra factor of A − 1 r left multiplying f ≤ r i ( x, W ) : f r i [Equation] [Equation]
[Equation] ( g
[Equation] [Equation] [Equation]
W [Equation] W ) [Equation]
W )
[Equation]
W )
[Equation]
(E.54)
F Network dissection details
Here we include some supplementary results and experiments for examining coordinate basis interpretability with network dissection [Bau+17].
F.1 Network dissection methodology
The Broden concept dataset, compiled by Bau et al., contains pixel-level annotations for hierarchical concepts including colors, textures, objects, and scenes.
For every channel activation, network dissection assesses the binary segmentation performance with every visual concept from Broden.
The method ﬁrst computes the channel activation for every Broden image.
The distribution of the activations for the channel is used to binarize the activation (where we threshold by the top 0 .
5% of all activations for the channel) to deﬁne a segmentation mask for the channel activation which is interpolated to the size of the input image.
If the Intersection over Union (IoU) of the activation segmentation mask and a concept mask is high enough (namely where IoU > 0 . 04 ), network dissection labels the activation an interpretable detector for the concept. % i n t e r p r e t ab l e [Figure] [Figure]
Change in basis
[Figure]
F.2 Additional experiments
ﬁg. 18 breaks down the categories of interpretable units for the models and rotations examined in ﬁg. 3.
The number of interpretable units tends to be dominated by the object and scene concept detectors for the ResNet-50 and the ConvNeXt models.
We also perform an analogous network dissection experiment to section 6 within the residual blocks for the normal and modiﬁed ResNet-50 in ﬁg. 19.
Like the ConvNeXt model in ﬁg.
we ﬁnd that, surprisingly, the percentage of interpretable units actually tends to increase away from the activation basis.
Per-concept breakdowns produced by network dissection for three different rotation powers in the experiment in ﬁg. 3 are given for the ResNet-50 in ﬁg. 20, the modiﬁed ResNet-50 (without a ReLU activation function on the residual output) in ﬁg. 22, and the ConvNeXt in in ﬁg. 24.
We also include the units with the highest concept intersection over union scores for the the three representative rotations for the ResNet-50 in ﬁg. 21, the modiﬁed ResNet-50 (without a ReLU activation function on the residual output) in ﬁg. 23, and the ConvNeXt in in ﬁg. [Figure]
Rotation 0. Rotation
[Figure]
[Figure] Rotation
[-% i n t e r p r e t ab l e un i t s ResNet-50, layer4.1.conv modified ResNet-50, layer4.1.conv activation function on the residual output) in ﬁg. 22, and the ConvNeXt in in ﬁg. 24. We also includethe units with the highest concept intersection over union scores for the the three representativerotations for the ResNet-50 in ﬁg. 21, the modiﬁed ResNet-50 (without a ReLU activation functionon the residual output) in ﬁg. 23, and the ConvNeXt in in ﬁg. 
-][Figure]
F.3 Model training details
We train a ResNet-50 without ReLU (or any activation function) on the residual blocks in PyTorch using [Lec+22] on ImageNet [Den+09].
We train with SGD with momentum for 88 epochs with a cyclic learning rate rate of 1 . 7 , label smoothing of 0 .
1 , a batch size of 512, and weight decay of 10 − 4 .
The model achieves 76.1% top-1 accuracy.
We use pretrained weights for the ResNet-50 (unmodiﬁed) and ConvNeXt models from [MR10] and [Wig19] respectively.
G Dataset Details
CIFAR-10: CIFAR-10 is covered by the MIT License (MIT).
We use canonical train/test splits (imported using torchvision).
Broden: the code used to generate the dataset is covered by the MIT license.
ImageNet: ImageNet is covered by CC-BY 4.0.
We use canonical train/test splits
(a) Myrtle CNN architecture, summary courtesy of torch- info.
(b) Myrtle CNN architecture without batch norm (only used in section 3.3), summary courtesy of torchinfo.
(a) Our ResNet20 architecture, summary courtesy of torchinfo.
(b) Internals of the 1st BasicBlock (the sequential con- tains the residual connection).