Original: A Classiﬁcation of G -Invariant Shallow Neural Networks
Revised: A Classiﬁcation of G -Invariant Shallow Neural Networks

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: When trying to ﬁt a deep neural network (DNN) to a G -invariant target functionwith respect to a group G , it only makes sense to constrain the DNN to be Ginvariant as well.
Revised: When trying to ﬁt a deep neural network (DNN) to a G -invariant target function with G a group, it only makes sense to constrain the DNN to be G -invariant as well.

Original: However, there can be many different ways to do this, thus raisingthe problem of “ G -invariant neural architecture design”:
Revised: However, there can be many different ways to do this, thus raising the problem of “ G -invariant neural architecture design”:

Original: What is the optimal Ginvariant architecture for a given problem?
Revised: What is the optimal Ginvariant architecture for a given problem?

Original: Before we can consider the optimizationproblem itself, we must understand the search space, the architectures in it, andhow they relate to one another.
Revised: Before we can consider the optimization problem itself, we must understand the search space, the architectures in it, and how they relate to one another.

Original: In this paper, we take a ﬁrst step towards this goal;we prove a theorem that gives a classiﬁcation of all G -invariant single-hidden-layeror “shallow” neural network ( G -SNN) architectures with ReLU activation for anyﬁnite orthogonal group G .
Revised: In this paper, we take a ﬁrst step towards this goal; we prove a theorem that gives a classiﬁcation of all G -invariant single-hidden-layer or “shallow” neural network ( G -SNN) architectures with ReLU activation for any ﬁnite orthogonal group G , and we prove a second theorem that characterizes the inclusion maps or “network morphisms” between the architectures that can be leveraged during neural architecture search (NAS).

Original: The proof is based on a correspondence of every G -SNNto a signed permutation representation of G acting on the hidden neurons. Theclassiﬁcation is equivalently given in terms of the ﬁrst cohomology classes of G ,thus admitting a topological interpretation. 
Revised: The proof is based on a correspondence of every G -SNN to a signed permutation representation of G acting on the hidden neurons; the classiﬁcation is equivalently given in terms of the ﬁrst cohomology classes of G , thus admitting a topological interpretation. The G -SNN architectures corresponding to nontrivial cohomology classes have, to our knowledge, never been explicitly identiﬁed in the literature previously.

Original: Based on a code implementation, weenumerate the G -SNN architectures for some example groups G and visualize theirstructure.
Revised: Using a code implementation, we enumerate the G -SNN architectures for some example groups G and visualize their structure.

Original: We draw the network morphisms between the enumerated architecturesthat can be leveraged during neural architecture search (NAS).

Original: Finally, we provethat architectures corresponding to inequivalent cohomology classesin a givencohomology ring coincide in function space only when their weight matrices arezero, and we discuss the implications of this in the context of NAS.
Revised: Finally, we prove that architectures corresponding to inequivalent cohomology classes coincide in function space only when their weight matrices are zero, and we discuss the implications of this for NAS.

Original:  Introduction
Revised:  Introduction

Original: When trying to ﬁt a deep neural network (DNN) to a target function that is known to be G -invariantwith respect to a group G , it is desirable to enforce G -invariance on the DNN as prior knowledge.
Revised: When trying to ﬁt a deep neural network (DNN) to a target function that is known to be G -invariant with respect to a group G , it is desirable to enforce G -invariance on the DNN as prior knowledge.

Original: This is a common scenario in many applications such as computer vision, where the class of an objectin an image may be independent of its orientation [Veeling et al., 2018] or point clouds that arepermutation-invariant [Qi et al., 2017].
Revised: This is a common scenario in many applications such as computer vision, where the class of an object in an image may be independent of its orientation [Veeling et al., 2018], or point clouds that are permutationinvariant [Qi et al., 2017].

Original: Numerous G -invariant DNN architectures have been proposedover the years, including group-equivariant convolutional neural networks (G-CNNs)
Revised: Numerous G -invariant DNN architectures have been proposed over the years, including G -equivariant convolutional neural networks ( G -CNNs)

Original: [Cohen and 
Revised: [Cohen and Welling, 2016],

Original: Welling, 2016], DNNs with high-order G -equivariant multilinear layers [Maron et al., 2019], and amore recent architecture built on a G -invariant sum-product layer [Kicki et al., 2020].
Revised: G -equivariant graph neural networks [Maron et al., 2019a], and a DNN stacked on a G -invariant sumproduct layer [Kicki et al., 2020].

Original: However, itis unclear which of these architectures a practitioner should choose for a given problem, and evenafter one is selected, additional design choices must be made; for G-CNNs alone, the practitionermust select a sequence of representations of G to determine the composition of layers, and it isunknown how best to do this.
Revised: However, it is unclear which of these architectures a practitioner should choose for a given problem, and even after one is selected, additional design choices must be made; for G -CNNs alone, the practitioner must select a sequence of representations of G to determine the composition of layers, and it is unknown how best to do this.

Original: Moreover, despite a complete classiﬁcation of G-CNNs [Kondor and Trivedi, 2018, Cohen et al., 2019], it is unknown if every G -invariant DNN is a G-CNN, and hencethe “optimal” G -invariant architecture may not even exist in the space of G-CNNs.
Revised: Moreover, despite a complete classiﬁcation of G -CNNs [Kondor and Trivedi, 2018, Cohen et al., 2019b], it is unknown if every G -invariant DNN is a G -CNN, and hence the “optimal” G -invariant architecture may not even exist in the space of G -CNNs.

Original: For some architectures, universality theorems have been proven guaranteeing the approximation ofany G -invariant function with arbitrarily small error [Maron et al., 2019, Ravanbakhsh, 2020, Kickiet al., 2020], and it is thus tempting to conclude that these universal architectures are sufﬁcient for all G -invariant problems.
Revised: For some architectures, universality theorems exist guaranteeing the approximation of any G -invariant function with arbitrarily small error [Maron et al., 2019b, Ravanbakhsh, 2020, Kicki et al., 2020], and it is thus tempting to conclude that these universal architectures are sufﬁcient for all G -invariant problems.

Original: However, it is well-known that universality [Cybenko, 1989] alone is not asufﬁcient condition for a good DNN model and that the function subspaces that a network traversesas it grows to the universality limit is just as important as the limit itself.
Revised: However, it is well-known that universality [Cybenko, 1989] alone is not a sufﬁcient condition for a good DNN model and that the function subspaces that a network traverses as it grows to the universality limit is just as important as the limit itself.

Original: This suggests that the wayin which a DNN is constrained to be G -invariant does matter, and different G -invariant architecturesmay be suitable for different problems.
Revised: This suggests that the way in which a DNN is constrained to be G -invariant does matter, and different G -invariant architectures may be suitable for different problems.

Original: This raises the fundamental question: For a given problem,what is the “best” way to constrain the parameters of a DNN such that it is G -invariant?
Revised: This raises the fundamental question: For a given problem, what is the “best” way to constrain the parameters of a DNN such that it is G -invariant?

Original: This paper takes a ﬁrst step towards answering the above question.
Revised: This paper takes a ﬁrst step towards answering the above question.

Original: Speciﬁcally, before we canconsider the optimization problem for the best G -invariant architecture, we must understand thesearch space:
Revised: Speciﬁcally, before we can consider the optimization problem for the best G -invariant architecture, we must understand the search space:

Original: What are all the possible ways to constrain the parameters of a DNN such that it is
Revised: What are all the possible ways to constrain the parameters of a DNN such that it is

Original: G -invariant, and how are these different G -invariant architectures related to one another?
Revised: G -invariant, and how are these different G -invariant architectures related to one another?

Original: The above is a special case of the broader and more fundamental problem of neural architecturedesign.
Revised: The above is a special case of the broader and more fundamental problem of neural architecture design.

Original: One of the most prominent approaches to this problem in the literature is neural architecturesearch (NAS), which at its core is trial-and-error [Elsken et al., 2019].
Revised: One of the most prominent approaches to this problem in the literature is neural architecture search (NAS), which at its core is trial-and-error [Elsken et al., 2019].

Original: While trial-and-error is—inprinciple—straightforward for determining, e.g., the optimal depth or hidden widths of a DNN, itis less clear for G -invariant architectures, where a practitioner does not even know all their options.
Revised: While trial-and-error is—in principle—straightforward for determining, e.g., the optimal depth or hidden widths of a DNN, it is less clear for G -invariant architectures, where a practitioner does not even know all their options.

Original: More generally, NAS presupposes knowledge about which architectures are in the search space,which ones are not, which ones are equivalent or special cases of others, and how best one shouldmove from one architecture to another.
Revised: More generally, NAS presupposes knowledge about which architectures are in the search space, which ones are not, which ones are equivalent or special cases of others, and how best one should move from one architecture to another.

Original: Thus, to apply even the simplest approach to G -invariantneural architecture design, we must ﬁrst be able to enumerate all G -invariant architectures.
Revised: Thus, to apply even the simplest approach to G -invariant neural architecture design, we must ﬁrst be able to enumerate all G -invariant architectures.

Original: Our main result is Thm. 5, which gives a classiﬁcation of all G -invariant single-hidden-layer or“shallow” neural network ( G -SNN) architectures with rectiﬁed linear unit (reLU) activation, for anyﬁnite orthogonal group G acting on the input space.
Revised: Our main result is Thm. 4, which gives a classiﬁcation of all G -invariant single-hidden-layer or “shallow” neural network ( G -SNN) architectures with rectiﬁed linear unit (reLU) activation, for any ﬁnite orthogonal group G acting on the input space.

Original: More precisely, every G -SNN architecture canbe decomposed into a sum of “irreducible” ones, and Thm.5 classiﬁes these.
Revised: More precisely, every G -SNN architecture can be decomposed into a sum of “irreducible” ones, and Thm. classiﬁes these.

Original: The classiﬁcationis based on a correspondence of each irreducible architecture to a representation of G in terms ofits action on the hidden neurons via so-called “signed permutations”, where the representation isrequired to satisfy an additional condition to eliminate degenerate (linear) architectures and redundantarchitectures equivalent to simpler ones.
Revised: The classiﬁcation is based on a correspondence of each irreducible architecture to a representation of G in terms of its action on the hidden neurons via so-called “signed permutations”, where the representation is required to satisfy an additional condition to eliminate degenerate (linear) architectures and redundant architectures equivalent to simpler ones.

Original: The classiﬁcation then boils down to the classiﬁcation ofthese representations.
Revised: The classiﬁcation then boils down to the classiﬁcation of these representations.

Original: These representations, and hence the corresponding architectures as well, areclassiﬁed in terms of the ﬁrst cohomology classes of G and thus admit a topological interpretation.
Revised: These representations, and hence the corresponding architectures as well, are classiﬁed in terms of the ﬁrst cohomology classes of G and thus admit a topological interpretation.

Original: We note that, while connections between neural networks and the group of signed permutations havebeen previously made in the literature [Ojha, 2000, Negrinho and Martins, 2014, Arjevani and Field,2020], to our knowledge, no such connection has yet been leveraged to begin a classiﬁcation programof G -invariant architectures. 
Revised: We note that, while connections between neural networks and the group of signed permutations have been previously made in the literature [Ojha, 2000, Negrinho and Martins, 2014, Arjevani and Field, 2020], to our knowledge, no such connection has yet been leveraged to begin a classiﬁcation program of G -invariant architectures. We also prove Thm. 5, which characterizes the “network morphisms” linking irreducible G -SNN architectures in architecture space.

Revised: In NAS, network morphisms furnish a topology on architecture space and describe how one should move from one architecture to another during the search [Wei et al., 2016].

Revised: Taken together, Thms. 4&5 give a complete description of G -SNN architecture space.

Original: This paper is perhaps most similar in spirit to the works of Kondor and Trivedi [2018] and Cohenet al.
Revised: This paper is perhaps most similar in spirit to the works of Kondor and Trivedi [2018] and Cohen et al.

Original: [2019] and draws on similar mathematical machinery; like them, this paper’s contribution isalso primarily theoretical.
Revised: [2019b] and draws on similar mathematical machinery; like them, this paper’s contribution is also primarily theoretical.

Original: Kondor and Trivedi [2018] prove that every G -invariant DNN is a g-cnnunder the assumption that every  layer is G -equivariant; that this is true without the assumption isonly conjectured.
Revised: Kondor and Trivedi [2018] prove that every G -invariant DNN is a G -CNN under the assumption that every afﬁne layer is G -equivariant; that this is true without the assumption is only conjectured.

Original: Cohen et al.
Revised: Cohen et al.

Original: [2019] generalize this to G-CNNs where hidden activations are vectorﬁelds and provide a classiﬁcation of all G-CNNs, but the conjecture of Kondor and Trivedi [2018]is left open.
Revised: [2019b] generalize this to G -CNNs where hidden activations are vector ﬁelds and provide a classiﬁcation of all G -CNNs, but the conjecture of Kondor and Trivedi [2018] is left open.

Original: In contrast to these works, in our paper, we do not assume the pre-activation afﬁnetransformation to be G -equivariant– only that the whole network is G -invariant.
Revised: In contrast to these works, in our paper, we do not assume the pre-activation afﬁne transformation to be G -equivariant– only that the whole network is G -invariant.

Original: Thus, a futureextension of Thm.
Revised: Thus, a future extension of Thm.

Original: 5 to deep architectures would either prove or refute the cited conjecture.
Revised: 4 to deep architectures would either prove or refute the cited conjecture, at least for ReLU networks.

Revised: Moreover, these works do not explicitly work out the group representations compatible with ReLU, and other works [Cohen and Welling, 2016, Cohen et al., 2019a] consider only unsigned permutations with ReLU.

Revised: In contrast, our classiﬁcation reveals G -SNN architectures (namely, those corresponding to proper signed permutation representations or nontrivial cohomology classes) that, to our knowledge, have never been explicitly identiﬁed in the literature previously.

Revised: We also note the work of Maron et al.

Revised: [2019a], who classify all G -equivariant linear layers for graph neural networks; however, they restrict their attention to unsigned permutation representations only, and there classiﬁcation is again not guaranteed to contain all G -invariant ReLU networks.

Original: The remainder of the paper is organized as follows:
Revised: The remainder of the paper is organized as follows:

Original: In Sec. 2, we give a classiﬁcation of the“signed permutation representations” of G and relate these representations to the cohomology classesof G .1 Then in Sec.
Revised: In Sec. 2, we give a classiﬁcation of the “signed permutation representations” of G and relate these representations to the cohomology classes of G .

Original:  3, we build towards the statement of our main classiﬁcation theorem of GSNN architectures.
Revised: Then in Sec. 3, we build towards and state our main classiﬁcation theorem of G -SNN architectures.

Original: While Sec. 2 makes little reference to G -SNNs, presenting it upfront helps tostreamline the exposition in Sec. 3, with much of the notation and terminology established.In Sec.
Revised: While Sec. 2 makes little reference to G -SNNs, presenting it upfront helps to streamline the exposition in Sec. 3, with much of the notation and terminology established.

Original: 4,we visualize the G -SNN architectures for some example groups G  and remark on the “networkmorphisms” between the architectures.
Revised: In Sec. 4, we visualize the G -SNN architectures for some example groups G , and in Sec. 5, we make a number of remarks including a theorem on the “network morphisms” between G -SNN architectures.

Original: Finally, in Sec. 5, we end with conclusions and next stepstowards the problem of G -invariant neural architecture design.
Revised: Finally, in Sec. 6, we end with conclusions and next steps towards the problem of G -invariant neural architecture design.

Original:  Signed permutation representations 
Revised:  Signed permutation representations 2.1 Preliminaries

Original: 2.1 Preliminaries

Original: Throughout this paper, let G be a ﬁnite group of m × m orthogonal matrices.
Revised: Throughout this paper, let G be a ﬁnite group of m × m orthogonal matrices.

Original: Let P ( n ) be the groupof all n × n permutation matrices and Z ( n ) the group of all n × n diagonal matrices with diagonalentries ± 1 .
Revised: Let P ( n ) be the group of all n × n permutation matrices and Z ( n ) the group of all n × n diagonal matrices with diagonal entries ± 1 .

Original: Let PZ( n ) =
Revised: Let PZ( n ) =

Original: P ( n ) (cid:110) Z ( n ) , which is the group of all signed permutations – i.e., thegroup of all permutations and reﬂections of the standard orthonormal basis { e 1 , . .
Revised: P ( n ) (cid:110) Z ( n ) , which is the group of all signed permutations – i.e., the group of all permutations and reﬂections of the standard orthonormal basis { e 1 , . .

Original: e n } .
Revised: e n } .

Original: This groupis also called the hyperoctahedral group in the literature [Baake, 1984].
Revised: This group is also called the hyperoctahedral group in the literature [Baake, 1984].

Original: A signed permutation representation (signed perm-rep) of degree n of G is a homomorphismρ : G (cid:55)→ PZ( n ) .
Revised: A signed permutation representation (signed perm-rep) of degree n of G is a homomorphism ρ : G (cid:55)→ PZ( n ) .

Original: Whenever we say ρ is a signed perm-rep, let it be understood that its degree is nunless we say otherwise.
Revised: Whenever we say ρ is a signed perm-rep, let it be understood that its degree is n unless we say otherwise.

Original: A signed perm-rep ρ is said to be irreducible if for every i, j = 1 , . . .
Revised: A signed perm-rep ρ is said to be irreducible if for every i, j = 1 , . . .

Original: , n ,there exists g ∈ G such that ρ ( g ) e i = ±
Revised: , n , there exists g ∈ G such that ρ ( g ) e i = ±

Original: e j .
Revised: e j .

Original: As we will see in Sec. 3.2, every G -SNN can be writtenas a sum of “irreducible” G -SNNs, and every irreducible G -SNN corresponds to an irreducible signedperm-rep.
Revised: As we will see in Sec. 3.2, every G -SNN can be written as a sum of “irreducible” G -SNNs, and every irreducible G -SNN corresponds to an irreducible signed perm-rep.

Original: It is therefore sufﬁcient for our purposes to classify all irreducible signed perm-reps of G ;moreover, this need only be done up to conjugacy as seen next.
Revised: It is therefore sufﬁcient for our purposes to classify all irreducible signed perm-reps of G ; moreover, this need only be done up to conjugacy as seen next.

Original: 2.2 Classiﬁcation up to conjugacy
Revised: 2.2 Classiﬁcation up to conjugacy

Original: Two signed perm-reps ρ, ρ (cid:48) are said to be conjugate if there exists A ∈ PZ( n ) such that ρ (cid:48) ( g ) =
Revised: Two signed perm-reps ρ, ρ (cid:48) are said to be conjugate if there exists A ∈ PZ( n ) such that ρ (cid:48) ( g ) =

Original: A − 1 ρ ( g ) A ∀ g ∈ G .
Revised: A − 1 ρ ( g ) A ∀ g ∈ G .

Original: We let ρ PZ denote the conjugacy class of the signed perm-rep ρ .
Revised: We let ρ PZ denote the conjugacy class of the signed perm-rep ρ .

Original: Note thatconjugation preserves the (ir)reducibility of a signed perm-rep (Prop. 7 in Supp.
Revised: Note that conjugation preserves the (ir)reducibility of a signed perm-rep (Prop. 7 in Supp.

Original: A.1), and it thusmakes sense to speak of the irreducibility of an entire conjugacy class ρ PZ .
Revised: A.1), and it thus makes sense to speak of the irreducibility of an entire conjugacy class ρ PZ .

Original: The signiﬁcance of theconjugacy relation is that conjugate signed perm-reps correspond to the same G -SNN (see Sec. 3.2);we are thus interested in the classiﬁcation of irreducible signed perm-reps only up to conjugacy.
Revised: The signiﬁcance of the conjugacy relation is that conjugate signed perm-reps correspond to the same G -SNN (see Sec. 3.2); we are thus interested in the classiﬁcation of irreducible signed perm-reps only up to conjugacy.

Original: Our ﬁrst theorem below gives the desired classiﬁcation of signed perm-reps. 2 For H, K ≤ G , let( H, K ) G denote the paired conjugacy class
Revised: Our ﬁrst theorem below gives the desired classiﬁcation of signed perm-reps. 2 For H, K ≤ G , let ( H, K ) G denote the paired conjugacy class

Original: [Equation]
Revised: [Equation]

Original: Deﬁne the following set of conjugacy classes of subgroup pairs:
Revised: Deﬁne the following set of conjugacy classes of subgroup pairs:

Original: [Equation]
Revised: [Equation]

Original: Theorem 1. Let ( H, K ) G ∈ C G ≤ 2 and ( g 1 , . . 
Revised: For every ( H, K ) G ∈ C G ≤ 2 , we deﬁne a signed perm-rep ρ HK as follows: Let ( g 1 , . . .

Original: . , g n )  a transversal of G/H with g 1 ∈ K .
Revised: , g n ) be a transversal of G/H with g 1 ∈ K .

Original: For eachi = 1 , . . . , n , deﬁne g − i
Revised: For each i = 1 , . . . , n , deﬁne g − i

Original: =
Revised: =

Original: g i h for some h ∈ H \ K if | H : K | = 2 and h
Revised: g i h for some h ∈ H \ K if | H : K | = 2 and h

Original: = 1 if | H : K | = 1 .
Revised: = 1 if | H : K | = 1 .

Original: Deﬁnethe signed perm-rep ρ HK such that ρ HK ( g ) e i =
Revised: Then deﬁne the signed perm-rep ρ HK such that ρ HK ( g ) e i =

Original: e j if gg i K
Revised: e j if gg i K

Original: =
Revised: =

Original: g j K for every i, j = ± 1 , . .
Revised: g j K for every i, j = ± 1 , . .

Original: . , ± n .
Revised: . , ± n .

Revised: Theorem 1.

Original: Then:
Revised: We have:

Original: (a) Every ρ HK is irreducible.
Revised: (a) Every ρ HK is irreducible.

Original: (b) For every irreducible signed perm-rep ρ (cid:48) , there exists a unique ( H, K ) G such that ρ (cid:48) is conjugate to ρ HK .
Revised: (b) For every irreducible signed perm-rep ρ (cid:48) , there exists a unique ( H, K ) G such that ρ (cid:48) is conjugate to ρ HK .

Original: Theorem 1 equivalently states that the set
Revised: Theorem 1 equivalently states that the set

Original: [Equation]
Revised: [Equation]

Original: is a partition on the set of all irreducible signed perm-reps into conjugacy classes.
Revised: is a partition on the set of all irreducible signed perm-reps into conjugacy classes.

Revised: We will say ρ HK has type | H : K | – i.e., type 1 if H = K and type 2 otherwise.

Revised: For the interested reader, we note that ρ HK is the rep of G induced from the rep φ : H (cid:55)→ {− 1 , 1 } where K = ker( φ ) .

Original: 2.3 Group cohomology
Revised: 2.3 Group cohomology

Original: Every irreducible signed perm-rep is either type 1 or type 2 , which differ in their properties; see Lemma 8 in Supp.
Revised: Group cohomology offers an alternative perspective on the classiﬁcation of signed perm-reps and is the basis for the directed graph visualizations in Sec.

Original: A.1 for details.
Revised: 4.1 and Supp.

Original: We originally introduced this dichotomy because the two typeswould often give rise to casewise results or proofs.
Revised: C.2.

Original: It turns out, however, thatthere is a deepermeaning as well, which will be important for interpreting the classiﬁcation of G -SNNs.
Revised: 3 Note, however, that a technical understanding of group cohomology is not required for most of this paper, and we give only a high-level overview here.

Original: For every ρ HK ∈ R ( G ) , let πH : G (cid:55)→ P ( n ) and ζ L HK : G (cid:55)→ Z ( n )be the unique functionssatisfying ρHK ( g ) = 
Revised: Let ρ be a signed perm-rep and π : G (cid:55)→ P ( n ) and ω : G (cid:55)→ Z ( n ) the unique functions satisfying ρ ( g ) = ω ( g )

Original: ζ L HK ( g ) πH ( g ) ∀ g ∈ G .
Revised: π ( g ) ∀ g ∈ G .

Original: 3 Note that π H is independent of K (Prop. 14 in 
Revised: The function ω is called a cocycle and describes the sign ﬂips associated to the action of G through ρ . It can be depicted using colored directed graphs as in Fig.

Original: Supp. A.3).
Revised: 1 (and Figs.

Original: The following proposition relates the structure of irreducible signed perm-reps of G toits cohomology. Proposition 2.
Revised: 4-6), where arcs of different colors represent the actions of diffrent generators of G and dashed arcs represent sign ﬂips. The cocycle ω thus encodes topological information about how a space “twists” as we move through it by the action of G .

Original: For every ρ PZ HK ∈ R PZ ( G ) , let M H be a G -module over the ﬁeld F 2 = { 0 , 1 } ,where G acts on M H by the perm rep π H . Deﬁne ˆ ζ L HK : G (cid:55)→ M H such that ˆ ζ L HK ( g ) = 12 [ I − diag( ζ L HK ( g ))] .
Revised: If ρ (cid:48) is another signed perm-rep conjugate to ρ by a diagonal matrix in Z ( n ) , then its corresponding cocycle ω (cid:48) is said to be cohomologous to ω , and the set of all cocycles cohomologous to ω is said to form a cohomology class .

Original: Then: (a) The ﬁrst cohomology group of G with coefﬁcients in M H is given by
Revised: the reason for this equivalence is that the number of sign ﬂips around a cycle is unique only up to an even number of sign ﬂips; thus, the solid vs. dashed arcs of the directed graphs in Fig.

Original: [Equation]
Revised: 1 are not unique.

Original: where [ ˆ ζ L HK ] is the set of all cocycles cohomologous to ζ L HK , and where the additionoperation satisﬁes [Equation] (b) The partition of the ﬁrst cohomology group into orbits under the action of the  G -module automorphism group aut( M H ) is given by
Revised: The set of all cohomology classes associated to a single π forms a cohomology group , and it turns out that the classiﬁcation of these cohomology groups and their elements gives another way to look at the classiﬁcation of the signed perm-reps of G ; see Prop.in Supp.

Original: [Equation]
Revised: A.3 for details.

Original: (c) ρ HK is type 1 if and only if ˆ ζ L HK is in the zero cohomology class. The type 1vs. type 2 dichotomy is thus rooted in whether a signed perm-rep “twists” over G/H . Proposition 2 also lets us interpret the notation ρ HK : The subgroup H determines the coefﬁcientmodule M H and hence the  cohomology ring, and the subgroup K determines the  cohomology classin H 1 ( G, M H ) .
Revised: Type 1 signed perm-reps are then the ones corresponding to the identity elements of these cohomology groups—i.e., the cohomology classes with no sign ﬂips—and typesigned perm-reps correspond to the elements describing nontrivial sign ﬂip patterns. Finally, as we are only interested in signed perm-reps up to conjugation, we regard two cohomology classes as equivalent if their corresponding signed perm-reps are related by conjugation with a permutation matrix in P ( n ) .

Original: Most importantly, Prop. 2 provides a topological interpretation of the organizationof the space of G -SNNs, as we will see later in this paper.
Revised: Concretely, this ensures that colored directed graphs isomorphic to those shown in Fig. 1 are in fact considered equivalent to them.

Original:  Classiﬁcation of G -SNNs 
Revised:  Classiﬁcation of G -SNNs 3.1 Canonical parameterization

Original: 3.1 Canonical parameterization

Original: A shallow neural network (SNN) is a function f : R m (cid:55)→
Revised: A shallow neural network (SNN) is a function f : R m (cid:55)→

Original: R of the form
Revised: R of the form

Original: [Equation]
Revised: [Equation]

Original: where W ∈ R n × m and a, b ∈ R n for some n  .
Revised: where W ∈ R n × m and a, b ∈ R n for some n , and d ∈ R .

Original: Here ReLU is the rectiﬁed linear unit activationfunction deﬁned as ReLU( x ) = max(0 , x ) elementwise.
Revised: Here ReLU is the rectiﬁed linear unit activation function deﬁned as ReLU( x ) = max(0 , x ) elementwise.

Original: The parameterization of an SNN givenin Eq.
Revised: The parameterization of an SNN given in Eq.

Original: 1 contains redundancies in the sense that different parameter conﬁgurations can deﬁne thesame function.
Revised: 1 contains redundancies in the sense that different parameter conﬁgurations can deﬁne the same function.

Original: For example, applying a permutation to the rows of a , b , and W generally results in adifferent parameter conﬁguration but always leaves f invariant.
Revised: For example, applying a permutation to the rows of a , b , and W generally results in a different parameter conﬁguration but always leaves f invariant.

Original: Also, by the identity
Revised: Also, by the identity

Original: [Equation]
Revised: [Equation]

Original: (where H is the Heaviside step function; see Prop. 15 in Supp.
Revised: (where H is the Heaviside step function; see Prop. 16 in Supp.

Original: B), the reﬂection of a row of theaugmented matrix [ W | b ] in Eq. 1 is equivalent to the addition of an afﬁne term, which can beinterpreted as  twoadditional hidden neurons.
Revised: B), the reﬂection of one or more rows of the augmented matrix [ W | b ] in Eq. 1 together with the addition of a linear term—which can be represented as the sum of two hidden neurons—leaves f invariant; e.g.,

Original:  Note that these permutation and reﬂection redundanciesform the group PZ( n ) .
Revised: [Equation] Note that these permutation and reﬂection redundancies form the group PZ( n ) .

Original: The lemma below will help us deﬁne a “canonical parameterization” in whichsuch redundancies are eliminated.
Revised: The lemma below will help us deﬁne a “canonical parameterization” in which such redundancies are eliminated.

Original: Lemma 3.
Revised: Lemma 2.

Original: Let Θ n be the set of all augmented matrices [ W | b ] ∈ R n × ( m +1) such that the rows of
Revised: Let Θ n be the set of all augmented matrices [ W | b ] ∈ R n × ( m +1) such that the rows of

Original: W have unit norms and no two rows of [ W | b ] are parallel.
Revised: W have unit norms and no two rows of [ W | b ] are parallel.

Original: Let Ω n ⊂ Θ n be a fundamental domain 5under the action of PZ( n ) .
Revised: Let Ω n ⊂ Θ n be a fundamental domainunder the action of PZ( n ) .

Original: Let f : R m (cid:55)→ R be an SNN of the form in Eq.
Revised: Let f : R m (cid:55)→ R be an SNN of the form in Eq.

Original: Then there exist uniquen ∗ ∈ N , [ W ∗ | b ∗ ] ∈ Ω n ∗ , a ∗ ∈ R n ∗ with nonzero elements, c ∗ ∈ R m , and d ∗ ∈ R such that:
Revised: Then there exist unique n ∗ ∈ N , [ W ∗ | b ∗ ] ∈ Ω n ∗ , a ∗ ∈ R n ∗ with nonzero elements, c ∗ ∈ R m , and d ∗ ∈ R such that:

Original: [Equation]
Revised: [Equation]

Revised: Here, since the rows of [ W ∗ | b ∗ ] are pairwise nonparallel, then no two hidden neurons can form a linear term; all such hidden neurons are collected in the unique term c (cid:62)∗ x .

Revised: As a result, n ∗ ≤ n as n ∗ is the smallest number of hidden neurons possible.

Original: We refer to the unique parameterization of the SNN f in terms of ( a ∗ , b ∗ , c ∗ , d ∗ , W ∗ ) as its canonicalparameterization , and the SNN is then said to be in canonical form .
Revised: We refer to the unique parameterization of the SNN f in terms of ( a ∗ , b ∗ , c ∗ , d ∗ , W ∗ ) as its canonical parameterization , and the SNN is then said to be in canonical form .

Original: We call b ∗ , W ∗ , and the rows of W ∗ the canonical bias , weight matrix , and weight vectors of the G -SNN respectively.
Revised: We call b ∗ , W ∗ , and the rows of W ∗ the canonical bias , weight matrix , and weight vectors of the G -SNN respectively.

Original: Note that thecanonical parameterization is a function of the choice of fundamental domain Ω n ∗ .
Revised: Note that the canonical parameterization is a function of the choice of fundamental domain Ω n ∗ .

Original: 3.2 G -SNNs and signed perm-reps
Revised: 3.2 G -SNNs and signed perm-reps

Original: For f to be a G -invariant SNN ( G -SNN), the action of g ∈ G on the domain of f must be equivalentto one of the redundancies in the parameterization of SNNs.
Revised: For f to be a G -invariant SNN ( G -SNN), the action of g ∈ G on the domain of f must be equivalent to one of the redundancies in the parameterization of SNNs.

Original: In the canonical parameterization,however, this means that the parameters of f and of f ◦ g must be identical.
Revised: In the canonical parameterization, however, this means that the parameters of f and of f ◦ g must be identical.

Original: This places constraintson the canonical parameters of a G -SNN, as made precise in the lemma below.
Revised: This places constraints on the canonical parameters of a G -SNN, as made precise in the lemma below.

Original: Lemma 4.
Revised: Lemma 3.

Original: Let f : R m (cid:55)→ R be an SNN expressed in canonical form with respect to a fundamentaldomain Ω n ∗ ⊂ Θ n ∗ .
Revised: Let f : R m (cid:55)→ R be an SNN expressed in canonical form with respect to a fundamental domain Ω n ∗ ⊂ Θ n ∗ .

Original: Then f is G -invariant if and only if there exists a unique signed perm-repρ : G (cid:55)→ PZ( n ∗ ) such that the canonical parameters of f satisfy the following equations for allg ∈ G :
Revised: Then f is G -invariant if and only if there exists a unique signed perm-rep ρ : G (cid:55)→ PZ( n ∗ ) such that the canonical parameters of f satisfy the following equations for all g ∈ G :

Original: [Equation]
Revised: [Equation]

Original: We see that the constraints on the canonical parameters of a G -SNN are not necessarily unique, as theydepend on a signed perm-rep ρ , whence the classiﬁcation program of G -SNNs in this paper.
Revised: We see that the constraints on the canonical parameters of a G -SNN are not necessarily unique, as they depend on a signed perm-rep ρ , whence the classiﬁcation program of G -SNNs in this paper.

Original: Lemma 4thus establishes the promised connection between G -SNNs and signed perm-reps.
Revised: Lemmathus establishes the promised connection between G -SNNs and signed perm-reps.

Original: To formalize thiscorrespondence, let SNN( G ) be the set of all G -SNNs and R PZ ( G ) the set of all conjugacy classesof signed perm-reps of G .
Revised: To formalize this correspondence, let SNN( G ) be the set of all G -SNNs and R PZ ( G ) the set of all conjugacy classes of signed perm-reps of G .

Original: Deﬁne the map F : SNN( G ) (cid:55)→ R PZ ( G ) such that if f ∈ SNN( G )and ρ is the corresponding signed perm-rep appearing in Lemma 4, then F ( f ) = ρ PZ .
Revised: Deﬁne the map F : SNN( G ) (cid:55)→ R PZ ( G ) such that if f ∈ SNN( G ) and ρ is the corresponding signed perm-rep appearing in Lemma 3, then F ( f ) = ρ PZ .

Original: Then Fis a well-deﬁned function in the sense that F ( f ) does not depend on the choice of fundamentaldomain Ω n ∗ .
Revised: Then F is a well-deﬁned function in the sense that F ( f ) does not depend on the choice of fundamental domain Ω n ∗ .

Original: Indeed, a change of fundamental domain Ω n ∗ → Ω (cid:48) n ∗ induces a transformation[ W ∗ | b ∗ ]
Revised: Indeed, a change of fundamental domain Ω n ∗ → Ω (cid:48) n ∗ induces a transformation [ W ∗ | b ∗ ]

Original: → A [ W ∗ | b ∗ ] for a unique A ∈ PZ( n ∗ ) .
Revised: → A [ W ∗ | b ∗ ] for a unique A ∈ PZ( n ∗ ) .

Original: By Eq. 3, this in turn induces the conjugationρ → Aρ ( · ) A − 1 , thereby leaving F ( f ) =
Revised: By Eq. 3, this in turn induces the conjugation ρ → Aρ ( · ) A − 1 , thereby leaving F ( f ) =

Original: ρ PZ invariant.
Revised: ρ PZ invariant.

Original: We now deﬁne a G -SNN architecture to be a subset S ⊆ SNN( G ) such that S
Revised: We now deﬁne a G -SNN architecture to be a subset S ⊆ SNN( G ) such that S

Original: = F − 1 ( ρ PZ ) for someρ PZ ∈ ran( F ) ; it consists of all G -SNNs that are constrained to respect the same representation of G .
Revised: = F − 1 ( ρ PZ ) for some ρ PZ ∈ ran( F ) ; it consists of all G -SNNs that are constrained to respect the same representation of G .

Original: In this language, the purpose of this paper is to classify all G -SNN architectures.
Revised: In this language, the purpose of this paper is to classify all G -SNN architectures.

Original: A G -SNN f is said to be irreducible if F ( f ) is a conjugacy class of irreducible signed perm-reps.
Revised: A G -SNN f is said to be irreducible if F ( f ) is a conjugacy class of irreducible signed perm-reps.

Original: Let f PZ denote the G -SNN architecture containing the G -SNN f , and observe that if f is irreducible,then so are all G -SNNs in f PZ ; in this case, f PZ is said to be an irreducible architecture .
Revised: Let f PZ denote the G -SNN architecture containing the G -SNN f , and observe that if f is irreducible, then so are all G -SNNs in f PZ ; in this case, f PZ is said to be an irreducible architecture .

Original: It can be shown that every G -SNN admits a decomposition into a sum of irreducible G -SNNs(Prop. 18 in Supp.
Revised: It can be shown that every G -SNN admits a decomposition into a sum of irreducible G -SNNs (Prop. 19 in Supp.

Original: B.2).
Revised: B.2).

Original: It follows that to classify all G -SNN architectures, it is enough to classify allirreducible G -SNN architectures.
Revised: It follows that to classify all G -SNN architectures, it is enough to classify all irreducible G -SNN architectures.

Original: This amounts to two tasks: (1) Classify all irreducible signed permrep conjugacy classes in ran( F ) , and (2) for every irreducible ρ PZ ∈ ran( F ) , give a parameterizationof all G -SNNs in the architecture F − 1 ( ρ PZ ) .
Revised: This amounts to two tasks: (1) Classify all irreducible signed permrep conjugacy classes in ran( F ) , and (2) for every irreducible ρ PZ ∈ ran( F ) , give a parameterization of all G -SNNs in the architecture F − 1 ( ρ PZ ) .

Original: 3.3 The classiﬁcation theorem
Revised: 3.3 The classiﬁcation theorem

Original: We now state our main theorem, but ﬁrst we introduce some notation.
Revised: We now state our main theorem, but ﬁrst we introduce some notation.

Original: If A is a linear operator(resp.
Revised: If A is a linear operator (resp.

Original: set of linear operators), then let P A be the orthogonal projection operator onto the vectorsubspace that is pointwise-invariant under the action of A (resp.
Revised: set of linear operators), then let P A be the orthogonal projection operator onto the vector subspace that is pointwise-invariant under the action of A (resp.

Original: all elements of A ).
Revised: all elements of A ).

Original: Note that if A isa ﬁnite orthogonal group, then [Serre, 1977, sec. 2.6]
Revised: Note that if A is a ﬁnite orthogonal group, then [Serre, 1977, sec. 2.6]

Original: [Equation]
Revised: [Equation]

Original: Let st G ( P A ) denote the stabilizer subgroupst G ( P A )
Revised: Let st G ( P A ) denote the stabilizer subgroup

Original: = { g ∈ G : gP A = P A } .
Revised: [Equation]

Original: Theorem 5.
Revised: Theorem 4.

Original: Let ρ HK be an irreducible signed perm-rep of G , and let { g 1 , . .
Revised: Let ρ HK be an irreducible signed perm-rep of G , and let { g 1 , . .

Original: . , g n } be a transversalof G/H such that ρ HK ( g i ) e 1{+
+} = e i .
Revised: . , g n } be a transversal of G/H such that ρ HK ( g i ) e {+
+} = e i .

Original: Let τ = | H : K | − 1 .
Revised: Let τ = | H : K | − 1 .

Original: Then:(a)
Revised: Then:

Original:  ρ PZ HK ∈ ran( F ) if and only if st G ( P K  τP H )
Revised: (a) ρ PZ HK ∈ ran( F ) if and only if st G ( P K − τP H )

Original: = K .
Revised: = K .

Original: (b) If ρ PZ HK ∈ ran( F ) , then f ∈ F − 1 ( ρ PZ HK )
Revised: (b) If ρ PZ HK ∈ ran( F ) , then f ∈ F − 1 ( ρ PZ HK )

Original: if and only if the canonical parameters  of f havethe following forms:
Revised: if and only if the canonical parameters 6 of f have the following forms

Original: [Equation]
Revised: [Equation]

Revised: The condition in Thm. 4 (a) helps to exclude architectures where Eq.

Original:  Combining this with the classiﬁcation of irreducible signed perm-reps up to conjugacy (Thm. 1), weobtain a complete classiﬁcation of the irreducible G -SNN architectures as an immediate corollary.
Revised: 7 yields redundant weight vectors. Combining this with the classiﬁcation of irreducible signed perm-reps up to conjugacy (Thm. 1), we obtain a complete classiﬁcation of the irreducible G -SNN architectures as an immediate corollary.

Original: Theorems 1&5 can be assembled into an algorithm that enumerates all irreducible G -SNN architectures for any given ﬁnite orthogonal group G .
Revised: Theorems 1&4 can be assembled into an algorithm that enumerates all irreducible G -SNN architectures for any given ﬁnite orthogonal group G .

Original: We implemented the enumeration algorithm usinga combination of GAP 6 and Python; our implementation currently supports, in principle, all ﬁnitepermutation groups G < P ( m ) .
Revised: We implemented the enumeration algorithm using a combination of GAP 7 and Python; our implementation currently supports, in principle, all ﬁnite permutation groups G < P ( m ) .

Original: 7 Using our code implementation, we enumerated all irreducible G -SNN architectures for one permutation representation of every group G , | G | ≤ 8 , up to isomorphism.
Revised: 8 Using our code implementation, we enumerated all irreducible G -SNN architectures for one permutation representation of every group G , | G | ≤ 8 , up to isomorphism.

Original: We report the number of architectures, broken down by type, for each group in Table 1 (see Supp.
Revised: We report the number of architectures, broken down by type, for each group in Table 1 (see Supp.

Original: C.1; a discussion is included there as well).
Revised: C.1; a discussion is included there as well).

Original: We end this section with a proposition stating thatarchitectures corresponding to distinct cohomologyclasses in a given cohomology ring are in a sense orthogonal, the implications of which we discuss in Sec. 4.
Revised: A key observation is that the number of type 2 architectures—which to our knowledge have never appeared in the literature previously—is signiﬁcant; e.g., for the dihedral permutation group G = D 4 on four elements, there are ﬁve typearchitectures compared to seven type 2 architectures.

Original: Proposition 6.
Revised: Script execution time for each group was under 2 seconds. Nevertheless, we remark that our code is not optimized for speed and scalability as our purpose was exploration and intuition. As future work, we will work out Thm.

Original: Let w 1 and w 2 be the ﬁrst rows of the canonical weight matrices of two irreducible
Revised: 4 for speciﬁc families of groups to derive more direct and efﬁcient implementations.

Original:  G -SNN architectures F − 1 ([ ρ PZ HK 1 ]) and F − 1 ([ ρ PZ HK 2 ]) where K 1 (cid:54) = K 2 . Then w (cid:62) 1 w 2= 0 .
Revised: We will also investigate how to build G -SNN architectures from smaller G 1 -SNN and G 2 -SNN architectures where G is a (semi)direct product of G 1 and G 2 .

Revised: Finally, we note that for the application of NAS, we will probably never enumerate all G -SNN architectures; instead, we will generate them on-the-ﬂy as we move through the search space.

Original: Examples
Revised: [Figure]

Original:   4.1 The cyclic permutation group
Revised:  Examples 4.1 The cyclic permutation group

Original: Consider the group G = C 6 of all cyclic permutations on the dimensions of the input space R 6 .[-
-]
Revised: Consider the group G = C 6 of all cyclic permutations on the dimensions of the input space R 6 .[-
-]

Original: There are six irreducible G -SNN architectures for G = C 6 (Fig. 1); “architecture i.j” refers to
Revised: There are six irreducible G -SNN architectures for G = C 6 (Fig. 1); “architecture i.j” refers to

Original: F − 1 ( ρ PZ H
Revised: F − 1 ( ρ PZ H

Original: i K j ) where H 0 , . . .
Revised: i K j ) where H 0 , . . .

Original: , H 3 are isomorphic to C 1 , C 2 , C 3 , and C 6 respectively and K j ≤ H isuch that | H i : K j | = j + 1 .
Revised: , H 3 are isomorphic to C 1 , C 2 , C 3 , and C 6 respectively and K j ≤ H i such that | H i : K j | = j + 1 .

Original: Architectures i.0 are thus exactly the type 1 ones, and two architecturesi.j and i.k for distinct j and k correspond to inequivalent cohomology classes in the same cohomologygroup.
Revised: Architectures i.0 are thus exactly the type 1 ones, and two architectures i.j and i.k for distinct j and k correspond to inequivalent cohomology classes in the same cohomology group.

Original: Note that the architectures with n hidden neurons correspond to H i ∼ = C 6 n .
Revised: Note that the architectures with n hidden neurons correspond to H i ∼ = C 6 n .

Original: The type 1 architectures i.0 correspond to ordinary  perm reps of G .
Revised: The type 1 architectures i.0 correspond to ordinary unsigned perm reps of G .

Original: These are the “obvious” architectures that practitioners probably could have intuited.
Revised: These are the “obvious” architectures that practitioners probably could have intuited.

Original: From Fig.
Revised: From Fig.

Original: 1, we see that the weight matricesof these architectures are constrained to have a circulant structure; cycling the input neurons is thusequivalent to cycling the hidden neurons, leaving the output invariant as all weights in the secondlayer (not depicted) are constrained to be equal.
Revised: 1, we see that the weight matrices of these architectures are constrained to have a circulant structure; cycling the input neurons is thus equivalent to cycling the hidden neurons, leaving the output invariant as all weights in the second layer (not depicted) are constrained to be equal.

Original: This circulant structure is also apparent in thecohomology class illustrations.
Revised: This circulant structure is also apparent in the cohomology class illustrations.

Original: Note that H i is the stabilizer subgroup of each row of the weightmatrices.

Original: Architectures i.1 are type 2 and are perhaps less obvious.
Revised: Architectures i.1 are type 2 and are perhaps less obvious.

Original: Cycling the input neurons is equivalent tocycling the hidden neurons only up to sign; if we cycle a weight vector around all the hidden neurons,then we do not return to the original weight vector but instead to its opposite.
Revised: Cycling the input neurons is equivalent to cycling the hidden neurons only up to sign; if we cycle a weight vector around all the hidden neurons, then we do not return to the original weight vector but instead to its opposite.

Original: If we think of the dashedarcs in the cohomology class illustrations as “half-twists” in a cylindrical band, then Architecturesi.1 correspond to a Möbius band, thereby distinguishing them topologically from architectures i.0.
Revised: If we think of the dashed arcs in the cohomology class illustrations as “half-twists” in a cylindrical band, then Architectures i.1 correspond to a Möbius band, thereby distinguishing them topologically from architectures i.0.

Original: Alternatively, in terms of graph colorings, if the nodes incident to a solid (resp. dashed) arc areconstrained to have the same (resp.
Revised: Alternatively, in terms of graph colorings, if the nodes incident to a solid (resp. dashed) arc are constrained to have the same (resp.

Original: different) color(s), then architectures i.1 are the only ones not2 -colorable.
Revised: different) color(s), then architectures i.1 are the only ones not 2 -colorable.

Original: [Figure] By Prop. 6, the ﬁrst row of the  weight matrix of architecture i.1 is constrained to be orthogonal tothat of architecture i.0; this is analogous to the constraint that every continuous global section of a
Revised: Observe that the top weight vector of architecture i.1 is constrained to be orthogonal to that of architecture i.0; this is made precise in Prop. 6 in Sec.

Original: Möbius band must vanish at some point.

Original: The upshot is that architectures i.0 and i.1 can coincide infunction space if and only if their weight matrices vanish– i.e., the architectures degenerate into linearfunctions.
Revised: The upshot is that architectures i.0 and i.1 can coincide in function space if and only if their weight matrices vanish– i.e., the architectures degenerate into linear functions.

Original: Since neural networks are trained with local optimization, then we think it is unlikelythat a G -SNN being ﬁt to a nonlinear dataset will degenerate to a linear function at any point in itstraining; assuming this is true, architectures i.0 and i.1 are effectively conﬁned from one another dueto their inequivalent topologies.
Revised: Since neural networks are trained with local optimization, then we think it is unlikely that a G -SNN being ﬁt to a nonlinear dataset will degenerate to a linear function at any point in its training; assuming this is true, architectures i.0 and i.1 are effectively conﬁned from one another due to their inequivalent topologies.

Original: We discuss this phenomenon in more detail in Sec.4.3.
Revised: We discuss this phenomenon in more detail in Sec.

Original: 4.2 The cyclic rotation group
Revised: 4.2 The cyclic rotation group

Original: Consider again the group G = C 6 , but this time a 2D orthogonal representation where each groupelement acts as a rotation by a multiple of 60 ◦ on the 2D plane.
Revised: Consider again the group G = C 6 , but this time a 2D orthogonal representation where each group element acts as a rotation by a multiple of 60 ◦ on the 2D plane.

Original: There are only two irreducible
Revised: There are only two irreducible

Original: G -SNN architectures– one of each type.
Revised: G -SNN architectures– one of each type.

Original: To visualize these architectures, we set w =
Revised: To visualize these architectures, we set w =

Original: [1 , 0] (cid:62) , a = 1 ,b = 0 .
Revised: [1 , 0] (cid:62) , a = 1 , b = 0 .

Original: 5 , c = 0 , and d ∗ = 0 in Thm.
Revised: 5 , c = 0 , and d ∗ = 0 in Thm.

Original: 5 (b).
Revised: 4 (b).

Original: Based on their contour plots (Fig. 2), we ﬁnd that the levelcurves of the type 1 (resp.
Revised: Based on their contour plots (Fig. 2), we ﬁnd that the level curves of the type 1 (resp.

Original: type 2) architecture are concentric regular dodecagons (resp. hexagons);both architectures are thus clearly invariant to 60 ◦ -rotations.
Revised: type 2) architecture are concentric regular dodecagons (resp. hexagons); both architectures are thus clearly invariant to 60 ◦ -rotations.

Original: In the type 2 architecture, the hexagonal level curves increase linearly with radial distance.
Revised: In the type 2 architecture, the hexagonal level curves increase linearly with radial distance.

Original: Since thebias is required to be zero (Eq. 9), a sharp minimum forms at the origin.
Revised: Since the bias is required to be zero (Eq. 9), a sharp minimum forms at the origin.

Original: The architecture has threeweight vectors and thus three hidden neurons, and it additionally has a linear term (whose gradient isshown in red in Fig. 2), which—when combined with weight vector 2 using Eq. 2—results in three weight vectors with C 3 symmetry.
Revised: The architecture has three weight vectors and thus three hidden neurons, and it additionally has a linear term (whose gradient is shown in red in Fig. 2), which—when combined with weight vector 2 using Eq. 2—results in three weight vectors with C 3 symmetry.

Original: Observe that if we cycle the three hidden neurons of the type2 architecture , so that each weight vector is rotated three times by 60 ◦ , then we obtain the threeweight vectors with reversed orientation; this is a manifestation of the nontrivial topology of the type2 architecture.
Revised: Observe that if we cycle the three hidden neurons of the type 2 architecture , so that each weight vector is rotated three times by 60 ◦ , then we obtain the three weight vectors with reversed orientation; this is a manifestation of the nontrivial topology of the type 2 architecture.

Original: The type 1 architecture has six weight vectors and thus six hidden neurons.
Revised: The type 1 architecture has six weight vectors and thus six hidden neurons.

Original: Observe that for eachweight vector, there is another that is its opposite.
Revised: Observe that for each weight vector, there is another that is its opposite.

Original: Thus, for [ W ∗ | b ∗ ] to have pairwise nonparallelrows (see Lemma 3), the bias b ∗ = b(cid:126) 1 must be nonzero, whence the dodecahedral region in theexample G -SNN (Fig. 2) where its value plateaus to zero.
Revised: Thus, for [ W ∗ | b ∗ ] to have pairwise nonparallel rows (see Lemma 2), the bias b ∗ = b(cid:126) 1 must be nonzero, whence the dodecahedral region in the example G -SNN (Fig. 2) where its value plateaus to zero.

Original: However, in the asymptotic limit b → 0 ,the type 1 architecture degenerates to the type 2 architecture but with twice the number of hiddenneurons.
Revised: However, in the asymptotic limit b → 0 , the type 1 architecture degenerates to the type 2 architecture but with twice the number of hidden neurons.

Original: Thus, even though the two architectures are topologically distinct, the type 1 architecturecan get arbitrarily close to the type 2 architecture in function space (see Supp.
Revised: Thus, even though the two architectures are topologically distinct, the type 1 architecture can get arbitrarily close to the type 2 architecture in function space (see Supp.

Original: C.3 for a richerexample—the dihedral rotation group D 6 —which has irreducible architectures that cannot easilyaccess one another).
Revised: C.3 for a richer example—the dihedral rotation group D 6 —which has irreducible architectures that cannot easily access one another).

Original: This has important consequences, which we discuss more in the next section.
Revised: This has important consequences, which we discuss more in the next section.

Original: 4.3  Remarks 
Revised:  Remarks 5.1 Numbers of hidden neurons

Original: Numbers of hidden

Original: neurons The type 1 and type 2 irreducible architectures for the G = C 6rotation group (Fig. 2) have six and three hidden neurons respectively.
Revised: The type 1 and type 2 irreducible architectures for the G = C 6 rotation group (Fig. 2) have six and three hidden neurons respectively.

Original: In addition, the linear term c ∗ xin the canonical form of a G -SNN—if not zero—can be interpreted as two additional hidden neurons.
Revised: In addition, the linear term c ∗ x in the canonical form of a G -SNN—if not zero—can be interpreted as two additional hidden neurons.

Original: It follows that a general G -SNN that is a sum of copies of the two irreducible architectures cannothave 3 k + 1 hidden neurons for any integer k .
Revised: It follows that a general G -SNN that is a sum of copies of the two irreducible architectures cannot have 3 k +1 hidden neurons for any integer k .

Original: Thus, if we ﬁt a traditional fully-connected SNN with3 k + 1 hidden neurons to a dataset invariant under 60 ◦ -rotations, then the ﬁt SNN can be a G -SNN ifand only if one or more of its hidden neurons are redundant– e.g., one hidden neuron is zeroed out, orfour hidden neurons sum to form a linear term, leaving (3 k + 1) − 4 = 3( k − 1) hidden neuronscorresponding to “proper” weight vectors.
Revised: Thus, if we ﬁt a traditional fully-connected SNN with 3 k + 1 hidden neurons to a dataset invariant under 60 ◦ -rotations, then the ﬁt SNN can be a G -SNN if and only if one or more of its hidden neurons are redundant– e.g., one hidden neuron is zeroed out, or four hidden neurons sum to form a linear term, leaving (3 k + 1) − 4 = 3( k − 1) hidden neurons corresponding to “proper” weight vectors.

Original: Although this is a rather simple example, it suggests thepossibility of more severe or complicated restrictions on numbers of hidden neurons for larger andricher groups G .
Revised: Although this is a rather simple example, it suggests the possibility of more severe or complicated restrictions on numbers of hidden neurons for larger and richer groups G .

Original: In these cases, the redundant hidden neurons could (1) make it more difﬁcult for the SNN to discover the symmetries in the dataset and (2) make the SNN more prone to overﬁtting, all atthe cost of additional computation.
Revised: In these cases, the redundant hidden neurons could perhaps make it more difﬁcult for the SNN to discover the symmetries in the dataset and thus weaken the model, all at the cost of additional computation.

Original: We thus conjecture that one factor that determines the optimalnumber of hidden neurons in traditional SNNs is whether the number admits a G -SNN architecture,or—going further—how many different G -SNN architectures the number admits.
Revised: We thus conjecture that one factor that determines the optimal number of hidden neurons in traditional SNNs is whether the number admits a G -SNN architecture, or—going further—how many different G -SNN architectures the number admits.

Original: Asymptotic inclusion Let f PZ i=
Revised: Let f PZ i

Original:  F − 1 ( ρ PZ i ) for i = 1 , 2 be two G -SNN architectures.
Revised: = F − 1 ( ρ PZ i ) for i = 1 , 2 be two G -SNN architectures.

Original: If forevery ε > 0 and f2 ∈ f PZ2, there exists  f 1 ∈ f PZ1 such that (cid:107) f 1 − f 2 (cid:107) < ε (for some choice offunctional norm), then we say f PZ1 is “asymptotically included” in f PZ2 and write f PZ1 (cid:44) → f PZ2 .
Revised: If for every f ∈ f PZ2 there exists a sequence { f n ∈ f PZ1 } ∞ n =1 that converges to f in the topology of uniform convergence on compact sets, 10 then we say f PZ2 is asymptotically included in f PZ1 and write f PZ2 (cid:44) → f PZ1 .

Original: In the cyclic rotation example in Sec. 4.2, as already discussed there, the type 2 architecture isasymptoticly included in the type 1 architecture.
Revised: In the cyclic rotation example in Sec. 4.2, as already discussed there, the type 2 architecture is asymptoticly included in the type 1 architecture.

Original: In the cyclic permutation example in Sec. 4.1, the asymptotic inclusions 9 furnish a 3 -partite topology on the space of irreducible architectures (Fig. 3).
Revised: In the cyclic permutation example in Sec. 4.1, the asymptotic inclusions 11 furnish a 3 -partite topology on the space of irreducible architectures (Fig. 3); here every directed path is an asymptotic inclusion, and the individual arcs could be called “irreducible asymptotic inclusions”.

Original: This topology provides the necessary structure to perform neural architecture search (NAS), wherethe inclusions serve as “network morphisms” [Wei et al., 2016].
Revised: This topology provides the necessary structure to perform neural architecture search (NAS), where the irreducible inclusions serve as the network morphisms [Wei et al., 2016].

Original: In NAS, network morphisms areused to map underﬁtting architectures to larger ones, after which training resumes; the upshot is thatthe larger architecture need not be re-initialized, thereby signiﬁcantly cutting computation time.
Revised: In NAS, network morphisms are used to map underﬁtting architectures to larger ones, after which training resumes; the upshot is that the larger architecture need not be re-initialized, thereby signiﬁcantly cutting computation time.

Original: Infuture work, we will run NAS on the space of irreducible G -SNN architectures to learn an optimal
Revised: In future work, we will run NAS on the space of irreducible G -SNN architectures to learn an optimal

Original: G -SNN in a greedy manner.
Revised: G -SNN in a greedy manner.

Original: Topological tunneling Recall in the cyclicpermutation example in Sec i.0 and i.1 correspond to distinct cohomology classes and coincide in function space only whentheir weight matrices are zero.
Revised: Although the above deﬁnition of asymptotic inclusion is functional-analytic, the following theorem gives a group-theoretic characterization that is more amenable to computation. Theorem 5. Let f PZ i

Revised: =

Revised: F − 1 ( ρ PZ H

Revised: i K i )

Revised: for i = 1 , 2 be two irreducible G -SNN architectures.

Revised: Then f PZ2 (cid:44) → f PZ1 iff there exists ( H, K ) ∈

Revised: ( H 1 , K 1 ) G such that H ≤ H 2 , K ≤ K 2 , and H ∩ K 2 = K .

Revised: The proof (see Supp.

Revised: D.1) relies on a non-canonical parameterization of G -SNNs; a lemma (Lemma 22) that invokes the Arzelà-Ascoli Theorem; and Thm.

Revised: (a).

Original: In the dihe- dral rotation group example (see Supp.
Revised: Theorem 5 can thus be used to generate network morphisms such as those in Fig. 3 algorithmicly. Observe that as a corollary, since subgroup lattices are connected and subgroup inclusion is transitive, then there are no “isolated” G -SNNs; every G -SNN architecture is connected to another by some network morphism.

Revised: 5.3 Topological tunneling

Original: C.3) generalization of  the cyclic rotation  example in
Revised: Recall the discussion in the ﬁnal paragraph of Sec. 4.1, where we said architectures i.0 and i.1 coincide in function space only when their weight matrices are zero. We see this phenomenon again in the dihedral rotation group example (see Supp. C.3).

Original: Sec. 4.2—we observe another instance of thispenomenon.
Revised: Both of these are instances of the following proposition, which states that architectures corresponding to distinct cohomology classes are in a sense orthogonal. Proposition 6. Let w 1 and w 2 be the ﬁrst rows of the canonical weight matrices of two irreducible G -SNN architectures F − 1 ( ρ PZ HK 1 ) and F − 1 ( ρ PZ HK 2 ) where K 1 (cid:54) =

Original:  We call this “topological conﬁnement”, and it implies that if, for example, wegenerate a nontrivial dataset using architecture1.1 for G = C 6 (Fig. 1) and if we initializea G -SNN with three hidden neurons and ﬁndthat–purely by chance–the initial parameters satisfy architecture 1.0 and not 1.1, then our network will fail to ﬁt to  the dataset; upon a reinitialization, if this time the initial parameterssatisfy architecture 1.1., the  network will succeed.
Revised: K 2 . Then w (cid:62) 1 w 2 = 0 . We call the resulting phenomenon “topological conﬁnement”, and it implies that if, for example, Alice generates a nontrivial dataset using architecture 1.1 for G = C 6 (Fig. 1) but Bob constructs architecture 1.0 to enforce G -invariance (as it is one of the more intuitive architectures), then Bob’s network will fail to ﬁt to Alice’s dataset, even though Bob’s network has the right “size”; this suggests that the way we enforce G -invariance in a network is important.

Original: Rather than relying on random initializations and chance, or resorting to a larger architecture such as 0.0, we propose to allow “topological tunneling”, where the cohomology class of anarchitecture is transformed by applying the appropriate orthogonal transformation to the top weightvector (see Prop. 6).
Revised: Rather than randomly selecting between architectures 1.0 and 1.1, or resorting to a larger architecture such as 0.0, we propose to allow “topological tunneling”, where the cohomology class of an architecture is transformed by applying the appropriate orthogonal transformation to the top weight vector. This allows us to transform one weight-sharing pattern into another in a way analogous to cutting and regluing a Möbius band to remove the twist.

Original: Topological tunneling thus introduces “shortcuts” between certain points inarchitecture space, hopefully facilitating NAS (Fig. 3).
Revised: Topological tunneling thus introduces “shortcuts” between certain points in architecture space, hopefully facilitating NAS (Fig. 3).

Original: We plan to test this in practice in future work.
Revised: We plan to test this in practice in future work.

Original:  Conclusion
Revised:  Conclusion

Original: We proved Thm. 5, which gives a classiﬁcation of all (irreducible) G -SNN architectures with ReLUactivation for any ﬁnite orthogonal group G acting on the input space.
Revised: We proved Thm. 4, which gives a classiﬁcation of all (irreducible) G -SNN architectures with ReLU activation for any ﬁnite orthogonal group G acting on the input space.

Original: The proof is based on acorrespondence of every G -SNN to a signed perm-rep of G acting on the hidden neurons.
Revised: The proof is based on a correspondence of every G -SNN to a signed perm-rep of G acting on the hidden neurons.

Original: Based onour code implementation, we enumerated the  irreducible G -SNN architecturesfor some examplegroups and made a number of remarks, primarily on how the architectures relate to one another infunction space (Sec. 4.3). Perhaps most interestingly, we found that architectures corresponding toinequivalent ﬁrst cohomology classes in a given cohomology ring of  G are practically “conﬁned”from one another in function space (Prop. 6).
Revised: We also proved Thm. 5, which characterizes the network morphisms between irreducible G -SNN architectures and thus—together with Thm. 4—completely describes G -SNN architecture space. A key implication of our theory is the existence of the type 2 G -SNN architectures, which to our knowledge have never been explicitly identiﬁed in the literature previously.

Original: Various next steps can be taken towards the ultimate goal of G -invariant neural architecture design.
Revised: Various next steps can be taken towards the ultimate goal of G -invariant neural architecture design.

Original: Onone hand, we could try to extend Thm.
Revised: On one hand, we could try to extend Thm.

Original: 5 to deep architectures, which would require us to understandthe redundancies of a deep network; on the other, we could  go ahead and investigate NAS on G -SNNsafter further developing the ideas of asymptotic inclusion and topological tunneling.
Revised: 4 to deep architectures, which would require us to understand the redundancies of a deep network. We could then investigate the behavior and utility of type 2 symmetry constraints in the context of real deep learning benchmark tasks. on the other hand, we could ﬁrst go ahead and investigate NAS on G -SNNs.

Original: Alternatively,we could strengthen our intuition about the condition in Thm.
Revised: For a scalable NAS implementation, we could work out Thm.

Original: 5 (a) used to eliminate degenerate (i.e.,linear) or redundant architectures; indeed, due to this condition, and as exempliﬁed by the dihedralrotation group (Supp.
Revised: 4 for speciﬁc families of groups to derive more efﬁcient implementations, and we could try to develop an “algebra” of G -SNNs where G is a (semi)direct product of smaller groups.

Original: C.3), the cohomology classes for which the condition is satisﬁed need not evenform a subgroup, and hence any patterns in  G -SNN architecture space are unclear.
Revised: Finally, we could consider what are “good” combinations of irreducible G -SNN architectures; e.g., which sequences of irreducible architectures converge in sum to universal G -invariant approximators fastest? Perhaps answers to these questions could aid in transforming G -invariant neural architecture design from an art to a science. Acknowledgments and Disclosure of Funding D.A. and J.O. were supported by DOE grant DE-SC0018175.

Original: Finally, we couldconsider what are “good” combinations of irreducible G -SNN architectures; e.g., which sequences ofirreducible architectures converge in sum to universal G -invariant approximators fastest?
Revised: D.A. was additionally supported by NSF award No. 2202990.

Original: Perhapsanswers to these questions could aid not just in G -invariant neural architecture design but could givehints about the design problem in general.
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope? [Yes] (b) Did you describe the limitations of your work? [Yes] See Sec. 1.

Revised: (c) Did you discuss any potential negative societal impacts of your work?

Revised: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Revised: [Yes] 2.

Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Revised: [Yes] See supplementary material.

Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Revised: [Yes] (b) Did you specify all the training details (e

Revised: g., data splits, hyperparameters, how they were chosen)?

Revised: [N/A] (c) Did you report error bars (e

Revised: g., with respect to the random seed after running experiments multiple times)?

Revised: [N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Revised: [N/A] 4.

Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Revised: [N/A] (b) Did you mention the license of the assets?

Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Revised: [N/A] (d) Did you discuss whether

Revised: and how consent was obtained from people whose data you’re using/curating?

Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Revised: [N/A] 5.

Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Revised: [N/A]