Group invariant machine learning by fundamental domain projections
Editor: Sophia Sanborn, Christian Shewmake, Simone Azeglio, Arianna Di Bernardo, Nina Miolane
Abstract
We approach the well-studied problem of supervised group invariant and equivariant machine learning from the point of view of geometric topology.
We propose a novel approach using a pre-processing step, which involves projecting the input data into a geometric space which parametrises the orbits of the symmetry group.
This new data can then be the input for an arbitrary machine learning model (neural network, random forest, support-vector machine etc).
We give an algorithm to compute the geometric projection, which is eﬃcient to implement, and we illustrate our approach on some example machine learning problems (including the well-studied problem of predicting Hodge numbers of CICY matrices), ﬁnding an improvement in accuracy versus others in the literature
1. Introduction
Many tasks in machine learning can be understood as approximating a function α : X → Y between a feature space and an output space.
Typically, these may be subsets of R n , but could be more complicated like Riemannian manifolds.
We consider the problem in the presence of symmetries —more precisely, suppose a group G that acts on X on the left, and α satisﬁes the invariance property
[Equation]
A simple example is recognising a single handwritten digit which may have been rotated by 90 ◦ , 180 ◦ , or 270 ◦ , so the problem is invariant under the action of Z 4 .
Machine learning models such as neural networks or random forests can approximate α but the resulting function β will not generally be G -invariant.
The key task is to deﬁne machine learning algorithms producing functions β : X → Y which are necessarily invariant.
1.1. Previous work
Machine learning models which are invariant (or equivariant) under the action of a group G have been extensively studied in the literature.
In (40), Yarotsky distinguishes two diﬀerent approaches to the problem: symmetrisation based and intrinsic approaches.
The ﬁrst involves averaging some non G -invariant model over the action of G to produce an (approximately)
G -invariant model; whereas intrinsic approaches involve designing the model to be G -invariant a priori by imposing conditions coming from the group action.
A standard symmetrisation based approach is data augmentation , which was used in early works such as (25), and is surveyed in (7).
It involves increasing the size of the training data
D train =
{ ( x, y ) | x ∈ X train ⊂ X, y =
α ( x ) ∈ Y } by applying sample elements G 0 ⊂ G to the inputs.
The new training data is then D augtrain := { ( g · x, y ) | ( x, y ) ∈ D train and g ∈ G 0 } .
A similar approach is to take a machine learning architecture β and apply it to several
G -translates of an input, before applying a pooling map to these diﬀerent outputs.
This yields a G -invariant map, and was studied in (2).
A more sophisticated version of this pooling technique was proposed in (1).
We now turn to examples of intrinsic approaches.
For neural networks, one can impose restrictions on the weights so that the resulting network is invariant under a group action on the input.
This was done using group equivariant hidden layers, for example, in (18; 41).
The same idea is also used in (29; 31; 32).
Methods to determine all equivariant linear layers were proposed in (10; 15).
Convolutional layers in neural networks are a standard tool to impose translational symmetry in image classiﬁcation tasks.
This idea has been generalised to group equivariant CNNs in (9) for actions by arbitrary discrete groups.
Another intrinsic approach is proposed in (40, Section 2) based on the theory of polynomial invariants of G .
All of these approaches are concerned with discrete symmetries.
The study of continuous symmetries was initiated in (24) and expanded in (35; 11); and the case of Euclidean transformations has received additional attention, for example in (16; 37).
An analogue of polynomial invariants that can handle some cases of continuous symmetries was proposed in (13).
1.2. Our contribution
Our approach to the problem is intrinsic, based on the fact that composing a G -invariant map with any other map, results in a G -invariant map.
More precisely, we suggest a G -invariant pre-processing step to be applied to the input data that can then be composed with any machine learning architecture.
The resulting composition is a G -invariant architecture.
We provide a general framework to deﬁne the G -invariant preprocessing step for general group actions and a concrete implementation for ﬁnite groups acting by coordinate permutations.
One way of getting a G -invariant self-map of the feature space is to map to a socalled fundamental domain F , which preserves the local geometry of the feature space.
The set F ⊂ X comes with a G -invariant map π : X → F onto its closure.
Let α be the restriction of α to F , then by G -invariance α
= α ◦ π .
Instead of ﬁtting a machine learning model β : X → Y to the training data D train , we train the model β : F → Y with
D π train := { ( π ( x ) , y ) | ( x, y ) ∈ D train } ⊂ F × Y which approximates α .
The resulting map β = β ◦ π : X → Y is G -invariant.
Figure 1 shows the diﬀerence between the pre-processing approaches of augmentation and our method.
This approach extends easily to G - equivariant machine learning, as explained at the end of Section 2.1. [Equation]
F [Equation] More precisely, G -invariant maps from X are parametrised by maps from the so-called quotient space X/G of X , see Appendix A.
The set F ⊂ X locally models X/G and, especially in the case X = R n and G acts by permuting coordinates, has the advantage of being extremely easy to compute.
Another advantage our approach has is that it can be applied directly to any supervised machine learning model.
In contrast, many existing methods, such as (18; 41; 29; 31; 32; 15; 10) only work for neural networks.
The computational cost of data augmentation and many equivariant machine learning approaches scales with the size of the symmetry group.
This is not the case for our approach, which we implement for groups of size 6 · 10 20 in Section 3.2.
In the case of small group sizes, our approach makes no notable diﬀerence.
That is for example the case in image recognition tasks with a rotational symmetry, which is one of our examples.
The rest of the paper is organised as follows.
Section 2 describes our approach in detail, and compares it with other approaches from the literature.
Section 3 discusses several applications of our approach, and compares the accuracies of machine learning architectures employing diﬀerent approaches to G -invariant machine learning.
Our main application is the learning of Hodge numbers from CICY matrices, ﬁrst studied in (20), and our method improves on the state of the art (14).
2. Mathematical approach
In principle, our approach works in a very general setting, however we will restrict ourselves to the case that the feature space X is a Riemannian manifold on which G acts discretely by isometries.
Recall an isometry is a map X
→ X which leaves the Riemannian metric unchanged.
Given x ∈ X , its orbit under the action of G is the set G · x = { g · x | g ∈ G } .
Recall an action is discrete if every orbit under G is a discrete subset of X .
The reason for these restrictions is that the group action preserves the geometry of X .
2.1. Fundamental domains
We want to approximate a G -invariant function
α satisfying Equation (1).
It follows that α takes the same value on every element of any G -orbit.
A set R ⊂ X is a set of orbit representatives if for all x ∈ X , R ∩ ( G · x ) (cid:54) = ∅ .
Approximating α on a set of orbit representatives essentially approximates it everywhere.
A nice choice for R which takes into account the geometry of the group action is given by a fundamental domain.
Deﬁnition 1 Let a group G act on X discretely by isometries.
A subset F ⊂ X is called a fundamental domain for G if (a) it is open and connected; (b) every G -orbit intersects F , the closure of F , in at least one point; and (c) whenever a G -orbit intersects F at a point in
F , then this is the unique point of intersection with F .
[Equation]
Given G acting on X we will ﬁnd a G -invariant map π : X → F , deﬁned as π ( x )
=
φ ( x ) · x , where φ : X → G is some suitable function.
We call such a map a projection onto the fundamental domain F .
We can now apply any machine leaning architecture to approximate the function α | F : F → Y trained on the data D π train yielding a function β .
This can then be used to compute the G -invariant approximation for α deﬁned on the whole of X by deﬁning β
= β ◦ π , see Figure 2.
If β is a neural network, then the universal approximation property is satisﬁed, namely β can approximate any continuous, G -invariant map α arbitrarily closely.
This follows from the standard universal approximation theorem and is proved in Appendix E.1.
For generic x ∈ X and any g ∈ G , φ has the property
[Equation]
from which we can show that β is indeed G -invariant:
[Equation]
Our method of producing a G -invariant architecture can easily be modiﬁed to the Gequivariant setting.
Let G act on X and Y , then a G -equivariant map α : X → Y satisﬁes α ( g · x )
= g · α ( x ) for all x ∈ X and g ∈ G .
Let π : X → F be the fundamental domain projection as above, and deﬁne φ : X → G be a function such that π ( x )
=
φ ( x ) · x .
Then we deﬁne the β model via
[Equation]
As above, for a generic x ∈ X , Equation (2) implies that β is G -equivariant.
Remark
The projection π is continuous on F , but may fail to be continuous on the boundary of F .
Here φ does not necessarily satisfy Equation (2) , and so the function β may not be strictly G -invariant/equivariant.
This only presents a problem if a signiﬁcant portion of D π train lies on the boundary.
An example is in Section 3.
Now we will describe two methods of ﬁnding a fundamental domain projection.
2.2. Dirichlet projections
The ﬁrst method of computing a projection map follows from a classical proof of the existence of a fundamental domain for a discrete group G acting by isometries on a metric space ( X, d ) (every Riemannian manifold is automatically a metric space).
The idea is to ﬁx some point x 0 which is only ﬁxed by elements of G which ﬁx the whole of X point-wise, and deﬁne F to be all points which are closer to x 0 than any other points in the orbit G · x 0 , see for example (36, §II.1.4).
Such an F is called an Dirichlet fundamental domain .
By its deﬁnition, we can rephrase the problem of ﬁnding a projection π : X → F as a minimisation problem for the metric on X : given x ∈ X ﬁnd g ∈ G which minimises d ( g · x, x 0 ) .
In practice, this can be approximated using a discrete gradient descent algorithm, see Appendix F.3.
In the special case that G acts on R n by orthogonal matrices, equivalently one can minimise the inner product (cid:104) g · x, x 0 (cid:105) instead of the distance.
2.3. Combinatorial projections
Finding a projection onto a Dirichlet fundamental domain may not always be the best option since the gradient descent algorithm must be applied to each input, and it cannot guarantee ﬁnding the global minimum, and hence can only be used to approximate π .
A preferable approach would be to have an explicit and easy to compute description of a projection π onto a fundamental domain.
We give an algorithm to compute such a projection for the rather general case that G < S n is a subgroup of the permutation group, which acts on X
=
R n by permuting coordinates.
More precisely, if x = ( x i ) i ∈ R n and s ∈ S n we say s acts on the left by s · ( x i )
i = (cid:0) x
s − 1 ( i ) (cid:1) i .
This induces an action of G on R n which we call a permutation action of G .
This action is discrete (since G is ﬁnite) and by isometries since it preserved the Euclidean metric.
Example
Let G = S n be the full permutation group acting on R n .
Given a point x = ( x 1 , . . .
, x n ) ∈ R n we can reorder the entries in any way by elements of S n , and a fundamental domain corresponds to a consistent choice of reordering.
One consistent choice would be to have the entries in increasing order, so F = { ( x 1 , . . .
, x n ) ∈ R n | x i < x i +1 for all i < n } .
The projection π : R n → F can be easily implemented using any sorting algorithm.
Another example of a simple group action is when G = Z n < S n acts by cyclically permuting the coordinates.
In this case we can make a consistent choice by ensuring that the ﬁrst entry is the smallest so F = { ( x 1 , . . .
, x n ) ∈ R n | x 1 < x i for all i > 1 } .
The algorithm to ﬁnd a fundamental domain projection in general is based on (12) in which the authors give an eﬃcient algorithm to ﬁnd a set of unique coset representatives for an arbitrary subgroup G (cid:54) S n , this is summarised in Appendix D.3.
A set of coset representatives can be turned into a set of orbit representatives for the permutation action of G on R n .
Appendix D details how we modify their algorithm so that this set of orbit representatives is in fact a fundamental domain, and so that it outputs an explicit projection map.
This map is easy to implement and eﬃcient to compute.
In fact, here we will deﬁne the ascending projection π ↑ , in Appendix B we discuss a few variations of this projection map and compute an example.
We have listed the outputs of the algorithm for several common examples of groups G (cid:54) S n in Appendix C.
Finding a base
Let N = { 1 , . .
. , n } , which we identify with the set of indices for the standard basis for R n so S n acting on R n corresponds to the right action of S n on N by i · s = s − 1 ( i ) .
The ﬁrst step of the algorithm is to ﬁnd a base for G (cid:54) S n .
Deﬁnition 4 A base for G (cid:54) S n is an ordered subset B = ( b 1 , . . .
, b k ) of N such that (cid:84) ki =1 Stab G ( b i ) = { 1 } , where Stab G ( b i ) =
{ g ∈ G | b i · g
= b i } is the stabiliser of b i in G .
Given a base let G 0 =
G and for 1 (cid:54) i (cid:54) k , deﬁne G i
=
Stab G i − 1 ( b i ) =
G i − 1 ∩ Stab G ( b i ) .
It follows that G k
= { 1 } .
One can always choose B = (1 , . . .
, n − 1) as a base, although an algorithm to compute an eﬃcient base is given in Appendix D.3.
Given a base B and the groups G i , we will also deﬁne ∆ i to be the orbit of b i under the action of G i − 1 .
Example 5 Let G be the subgroup in S 4 generated by the elements (1 2) and (3 4) , where we represent permutations using cycle notation: eg (1 2) swaps 1 and 2 and ﬁxes 3 and 4 .
Then
B = (1 , 3) is a base and we have stabilisers G
=
{ e, (1 2) , (3 4) , (1 2)(3 4) } , G 1 = { e, (3 4) } ,
G 2 = { e } , and orbits ∆ 1 = { 1 , 2 } , ∆ 2 = { 3 , 4 } .
Perturbing points in R n
We now need to deﬁne the map φ ↑ : X → G used in the deﬁnition of π ↑ .
The map φ ↑ will only be uniquely deﬁned on points x
= ( x i ) i ∈
R n all of whose entries are distinct.
We ﬁrst perturb x slightly to get a point with this property.
Choose a perturbation vector ε which has all distinct entries, for example ε = 12 n (1 , 2 , . . . , n ) .
Let d = min x i (cid:54) =
x j {| x i − x j |} (choose d = 1 if all entries of x are the same) and deﬁne x (cid:48)
=
x + dε , which is guaranteed to have all entries distinct.
The entries of x (cid:48) have the same relative order, ie if x (cid:48) i (cid:54) x (cid:48)
j then x i (cid:54) x j , and φ ↑ will depend only on this relative ordering of entries.
Then we deﬁne φ ↑ ( x ) =
φ ↑ ( x (cid:48) ) where φ ↑ ( x (cid:48) ) is deﬁned below.
The ascending projection map
We will deﬁne a sequence of permutations g i ∈ G for 1 (cid:54) i (cid:54) k as follows.
Assume g 1 , . .
, g i − 1 have already been found.
G i − 1 acts transitively on ∆ i , choose j ∈ ∆ i such that the j th entry of ( g i − 1 · · · g 1 ) · x (cid:48) is minimal among those entries indexed by ∆ i .
Choose g
i ∈
G i − 1 such that j · g i = g − 1 i ( j ) =
b i .
Now deﬁne φ ↑ ( x (cid:48) ) :
= g k · · · g 1 , note the choice of the g i ’s is not unique, but we will show in Appendix D.6 that φ ( x (cid:48) ) is uniquely deﬁned.
Appendix D is devoted to the proof of the following theorem which says that the map we have deﬁned is a projection onto a fundamental domain.
Theorem
Deﬁne π ↑ : R n → R n by π ↑ ( x ) =
φ ↑ ( x ) · x , and let F be the interior of its image.
Then F is a fundamental domain for G acting on R n .
Given a choice of base B and perturbation vector ε , the projection π ↑ is uniquely deﬁned.
Example 3 (Continued) For G = S n , then B = (1 , 2 , . . .
, n − 1) is the smallest base we can use, so G i =
Perm ( i + 1 , . . . , n ) and ∆ i
= { i, . . . , n } .
Fixing x (cid:48) ∈ X and following the algorithm above: g 1 ∈ G 0 =
S n is a permutation moving the smallest entry indexed by ∆ 1 = { 1 , . . . , n } to the entry indexed by b 1 = 1 .
Repeating this for each i up to n − 1 , g i moves the i th smallest entry of x (cid:48) to the i th position.
The result is that ( g n − 1 · · · g 1 ) · x (cid:48) has entries ordered from smallest to largest.
In a very similar way, applying the algorithm to G =
Z n using the base B = (1) yields the same projection onto a fundamental domain described at the start of Section 2.3.
In Appendix F.2 we discuss an algorithm to compute π ↑ for arbitrary permutation groups, and analyse the time and space complexity of this algorithm.
The worse case runtime of the algorithm depends only on the dimension n and the size of the base k < n , so in particular the algorithm remains computationally tractable for very large groups.
In general, the time complexity is O ( k 2 n 3 ) (Theorem 27).
For many groups this drops to O ( n 4 (log log n ) 2 ) (Theorem 33), and for the common groups discussed in Appendix C, it drops to O ( n 2 ) .
2.4. Comparing approaches to invariant and equivariant machine learning
We can compare the various approaches to invariant machine learning discussed in Section 1.
on a theoretical level; below we compare them experimentally.
Augmentation is a data pre-processing step and can be applied to any model.
However, the resulting model need not be G -invariant, and for large groups it is computationally impractical to augment by a representative subset of the group.
As for intrinsic approaches, group equivariant neural networks like (9; 15; 29; 41) are model-speciﬁc, and there are unavoidable limits on the universality while using low-order tensors (28).
Additionally, (9) requires the elements of G to be stored in memory, making it impractical for large groups.
The approach in (40) using polynomial invariants.
The approach in (40) is not practical outside of small group actions owing, in part, to the need to compute a basis of polynomial invariants.
This basis will in general be large, increasing the dimension of the feature space dramatically.
Fundamental domain projections are computationally easy to use, maintain the original dimension and geometry of the data, and are compatible with any machine learning model.
The resulting model is G -invariant almost everywhere (see Remark 2).
3. Examples and results
We show how G -invariant pre-processing can be applied to examples of classiﬁcation tasks in group theory, string theory, and image recognition.
In each case, the symmetry group acts diﬀerently on the input space.
We account for this by choosing appropriate projection maps from the previous section.
Experiments 3.1 and 3.3 are chosen as proof of concept, not to reach the state of the art, the main application of our approach is experiment 3.2.
Implementation details may be found in Appendix F.1.
3.1. Cayley tables
Multiplication tables of groups are called Cayley tables .
The following model problem was introduced in (21, Section 3.2.3): up to isomorphism, there are 5 groups with 8 elements.
Separate their Cayley tables into two classes and apply random permutations until 20tables in each class exist.
The problem is to assign a given table to one of two classes.
This map is invariant under the action of S 8 × S 8 acting on R 8 × 8 by row and column permutations.
Let π ↑ : R 8 × 8 → R 8 × 8 be the ascending projection map from Section 2.3, in particular as deﬁned in Appendices C.1 and C.6.
This has an explicit description as follows: given a choice of total order on the group elements, permute the columns so that the ﬁrst row is ordered smallest to biggest, and then permute the rows so that the ﬁrst column is ordered smallest to biggest.
Then, π ↑ is invariant under the action of S 8 × S 8 and can be eﬃciently computed for Cayley tables.
This pre-processing eﬀectively ‘undoes’ the permutations, which makes the machine learning problem trivial.
Consequently, we achieve nearly perfect accuracy using a linear support vector machine (SVM), see Table 1.
We compare our approach with the neural network from (21), with the Deep Sets architecture from (41), and with the S 8 × S 8 -invariant neural network from (18).
The Deep
Sets architecture is invariant under the action of the full S 8 · 8 = S 64 on R 8 × 8 .
As all Cayley tables are in the same orbit under this group action, the performance of this architecture can only be as good as random guessing.
Note that the general purpose architectures described in (29; 15) in this case are identical to (18).
Other architectures from the literature, such as (9), are diﬃcult to apply to this problem, since they require keeping a non-sparse map
S 8 × S 8 → R in memory.
This group has size 8! · 8!
≈ 1 . 6 · 10 9 .
3.2. CICY
Hodge numbers are crucial for understanding and distinguishing Calabi-Yau manifolds.
Algebraic algorithms to compute them explicitly exist, and in (17), a dataset of complex three-dimensional complete intersection Calabi-Yau manifolds (CICYs) and their Hodge [Table] numbers is given.
Hodge numbers are expensive to compute, and it is not feasible to compute all Hodge numbers explicitly on larger datasets.
For applications in string theory, it is important to have Calabi-Yau manifolds with large ﬁrst Hodge number.
In (20), it was suggested to use a neural network to approximately compute (among other tasks)
the ﬁrst Hodge number of a Calabi-Yau manifold, thereby quickly identifying the most promising candidates without resorting to the expensive exact calculation. For three-dimensional CICYs, the Hodge numbers have been computed, but it remains the most commonly used benchmark for comparison of machine learning models in the ﬁeld, so we apply our algorithm to this problem.
Here, CICYs are represented by matrices of size up to 12 × 15 , and the ﬁrst Hodge number is an integer.
The same problem was subsequently studied in (5; 6; 14), using more sophisticated machine learning models.
The problem is invariant under row and column permutations, ie an action of S 12 × S 15 on R 12 × 15 , but none of the machine learning models which have been implemented previously for the Hodge number classiﬁcation satisfy this invariance.
We compare two pre-processing maps: the map π Dir : R 12 × 15 → R 12 × 15 deﬁned in
Section 2.2, which we computed by performing discrete gradient descent; and π ↑ deﬁned in the same way as in Section 3.1.
We found that composing π Dir with existing neural networks slightly improves performance, but not signiﬁcantly.
We also considered an alternative training task in which input matrices ﬁrst had their rows and columns randomly permuted.
In this case, our model outperforms models from the literature by a large margin.
We also compare our model with the group invariant model from (18) in both training tasks, see Table 2.
Again, the approaches of (29) and (15) reduce to (18).
As for Cayley tables, the approach in (9) is impractical due to the large group size 12!
≈ 6 · 10 20 .
As our approach is intrinsic it is well suited for problems with a large symmetry group.
For all networks but the G -invariant multi layer perceptron (MLP) the accuracy decreases on the permuted dataset.
This suggests that the rows and columns of the CICY matrices are already systematically ordered in the original dataset.
The map π ↑ can be computed eﬃciently but need not be G -invariant on the boundary of the fundamental domain by
Remark 2.
This is a potential problem since the input data, which consists of integer-valued matrices, is discrete.
Indeed, a substantial proportion of the CICY matrices are very sparse and do lie on the boundary, which could be the reason why π ↑ performs relatively poorly on the permuted data set.
The projection map π Dir can only be approximated but is fully
G -invariant which is a crucial advantage on the permuted dataset.
[Table]
3.3. Classifying rotated handwritten digits
As an instructional example
[Table] we use the MNIST dataset of handwritten images from (26) [Table] on which Z 4 rotates the images by multiples of 90 ◦ .
We use the ascending averaging projection deﬁned in Appendix B, π ↑ av : R 28 × 28 → R 28 × 28 .
This map rotates each image so that its brightest quadrant is at the top-left.
We then compare performance of a linear classiﬁer, a shallow neural network [Table] and SimpNet (see (19)) which is among the top performers on the original MNIST task; ﬁrst on their own, then with data augmentation
[Table]
and ﬁnally with the projection map π ↑ av , but without data augmentation, see Table
[Table]
[Table]
For linear classiﬁers, data augmentation does not improve accuracy substantially due to their small number of parameters.
Unsurprisingly, pre-processing with π ↑ av improves performance because it is partially successful at rotating pictures into a canonical orientation.
For neural networks with more than one layer, data augmentation increases accuracy, because the model now has suﬃcient parameters to include the information from the additional training data.
Pre-processing using π ↑ av yields better accuracy than no pre-processing, but worse accuracy than full data augmentation.
If fewer training data points are added during the data augmentation step, the beneﬁt is comparable to applying the map π ↑ av .
This is one example of the fact that data augmentation may be the best pre-processing option if the symmetry group G has few elements and one can augment by the full group.
If | G | is very large, this is not possible, and pre-processing using a fundamental domain projection may be better than augmenting with a small, non-representative, subset of G .
4. Conclusion
The G -invariant pre-processing step proposed in this paper has a clear mathematical motivation.
It respects the geometry of the input space and we show in Appendix A it naturally ﬁts into a larger framework of G -equivariant machine learning.
There are also many practical advantages of our approach: it can be applied to any machine learning architecture, it preserves the dimension of the input space and in most cases it guarantees perfect G -invariance.
Furthermore, the computation cost is generally low even if | G | is very large.
For the image recognition task where | G | = 4 , many of these advantages are only relevant for networks with a small number of neurons.
For Cayley tables however, | G | = 8! · 8!
≈ 1 . 6 · 10 9 is very large and the G -invariance of our approach produces nearly perfect accuracy.
The symmetry group is even larger for CICY matrices even though both the action of the symmetry group as well as the classiﬁcation task itself are more complicated.
Our approach signiﬁcantly improves the most accurate architecture known so far for this task.
While our experiments all fall into the category of ﬁnite groups acting by permutations on R n , our approach works in the much broader setting of (possibly inﬁnite) groups acting discretely by isometries on manifolds.
Although not discussed in the paper, it also generalises to continuous Lie group actions, with the fundamental domain replaced by a geometrically nice set of orbit representatives. Acknowledgments
We would like to thank Yang-Hui He and Momchil Konstantinov for their comments on a draft of this paper.
We would also like to thank the anonymous reviewers for their helpful comments.
All three authors were supported by the Engineering and Physical Sciences Research Council [EP/L015234/1].
The EPSRC Centre for Doctoral Training in Geometry and Number Theory (The London School of Geometry and Number Theory), University College London. The second author was supported by the Simons Collaboration “Special Holonomy in Geometry, Analysis and Physics” and the ﬁrst author by the London Mathematical Society during part of the work. Appendix A. Unifying intrinsic approaches to equivariant machine learning
In Sections 2 and 3 we approximated G -invariant functions by considering them as functions on a fundamental domain.
In this appendix we show that approximating a G -invariant function is equivalent to approximating it on the quotient space .
We then pick up two approaches to G -invariant machine learning from the literature alongside our approach, and explain in which sense they can be viewed as machine learning on quotient spaces.
It will be convenient to treat the G -invariant problem as a special case of the G -equivariant problem, since an invariant function X
→ Y is an equivariant function where G acts trivially on Y .
We return to the setting that G acts by isometries on X and Y which are Riemannian manifolds.
To simplify the proof of Theorem 9 we make the stronger assumption that the action is properly discontinuous, rather than discrete, which means that for any compact subset K , the set { g ∈ G | g · K ∩ K (cid:54) = ∅} is ﬁnite.
A.1. Universality of quotient spaces
We begin by deﬁning the quotient space of a group action, and discuss its properties.
Deﬁnition 7 Let G be a group acting on X , then the quotient space X/G is the set of all
G -orbits of points in X , { G · x | x ∈ X } .
A quotient space is automatically equipped with a G -invariant map π X : X → X/G : x (cid:55)→ G · x .
If X is a subset of R n and the action is by isometries, X/G inherits a metric from the Riemannian metric on X , and π X is a local isometry.
For us the key is that quotient spaces are universal with respect to G -equivariant maps.
X
Y
X [Equation]
α π X π Y α
Universal property of quotient spaces
Given two spaces
X and Y and a group G acting on both, let π X : X → X/G and π Y :
Y → Y/G be the canonical projection maps.
Then for any G -equivariant map α : X → Y there is a unique map α : X/G → Y/G such that π Y ◦ α
= α ◦ π X (ie the diagram on the right commutes).
If the action of G on X and Y is suﬃciently ‘nice’, then the converse holds: equivariant maps X
→ Y are parametrised by certain maps X/G → Y/G .
Notice that for any Gequivariant function α , point stabilisers in X and Y have the property that for any x ∈ X ,
Stab G ( x ) ⊂ Stab G ( α ( x )) .
Deﬁnition
Let X and Y be simply connected, a continuous map α : X/G → Y/G is compatible with the G actions if for any x ∈ X , the stabiliser Stab G ( x ) is conjugate in G to a subgroup of Stab G ( y ) , where y is a lift under π Y of α ( π X ( x )) .
It follows from the observation above that if α is the map coming from the universal property, then α is automatically compatible with the G -action.
In the special case that G acts trivially on Y (ie the G -invariant case) then every continuous map α : X/G → Y/G
= Y is compatible.
Theorem Any compatible function α : X/G → Y/G lifts to a G -equivariant function α : X → Y .
Suppose α (cid:48) is another such lift, and assume there is some x 0 ∈ X such that α (cid:48) ( x 0 )
=
α ( x 0 ) where Stab G ( α ( x 0 )) ﬁxes Y point-wise.
Then α (cid:48)
( x ) = α ( x ) for all x ∈ X .
We omit the proof, which can be deduced using (8, Theorem 4.1.6).
This gives a converse to the universal property, and shows that up to an isometry of Y there is a oneto-one correspondence between G equivariant maps X → Y , and compatible maps of their quotients.
Now suppose α : X → Y is a G -equivariant function we want to approximate using a supervised machine learning algorithm.
Using an intrinsic approach, as discussed in
Section 1.1, means approximating α by a function β which is a priori G -equivariant.
By the theorem above, this is equivalent to approximating α by a compatible function β .
Below we will discuss how diﬀerent intrinsic approaches to the equivariant machine learning problem ﬁt into this framework.
A.2. Equivariant maps from polynomial invariants
We discuss brieﬂy the approach proposed by Yarotsky in (40, Section 2) based on the theory of polynomial invariants of a group G acting on X = R n .
A polynomial p ( x ) ∈ R [ x 1 , . . .
, x n ] is called a polynomial invariant if it satisﬁes Equation (1).
Similarly, if G also acts on R m , then q ( x ): R n → R m is a polynomial equivariant if q ( g · x ) =
g · q ( x ) for all g ∈ G and x ∈ R n .
The following, proved in (30) in the invariant case, was generalised to compact Lie groups in (38, Theorem 8.14.A).
The equivariant case is proved in (39, Section 4).
If a ﬁnite group G acts on R n and R m , then there are ﬁnite sets of invariants { p i ( x ) } ki =1 , and equivariants { q j ( x ) } lj =1 such that any polynomial equivariant q ( x ) can be written as q ( x )
= (cid:80) lj
=
1 q j ( x ) r j ( p 1 ( x ) , . . . ,
p k ( x )) for some r j ( x ) ∈ R [ x 1 , . . .
, x k ] .
In the invariant case we can take l = 1 , and q 1 ( x ) = 1 .
Yarotsky shows in (40, Proposition 2.4) that any continuous G -equivariant function α : R n → R m can be approximated on a compact set by a G -equivariant neural network of the form
[Equation]
for some a jh , b jhi , c jh ∈ R , where d ∈ N , and σ is a continuous non-polynomial activation function.
Notice that all the learnable parameters are contained in the inner two sums, which also constitute the neural network in the G -invariant case (cid:80) dh =
[Equation]
h σ (cid:16)(cid:80) ki
=1 b hi p
[Equation]
( x ) + c h (cid:17) , see (40, Proposition 2.3).
The relationship between this method and quotient spaces is shown by the following result (33).
Theorem
The map p ( x ) :
= ( p 1 ( x ) , . .
, p k ( x )) factors through R n /G , and induces a smooth embedding of R n /G into R k .
We can reinterpret the invariant version of Equation (3) as a fully-connected neural network β .
We then train this network on the data D p train = { ( [Equation] y ) |
[Equation] D [Equation] } which has been projected to the quotient space by p .
Thus β learns the map α directly.
A.3. Our approaches
Our approach of projecting onto a fundamental domain (whether by a combinatorial projection or a Dirichlet projection) ﬁts very naturally in this general framework.
Like Yarotsky’s approach, we want to try to approximate the function α rather than approximating α .
Instead of working directly with the quotient spaces, one can think of the map from the fundamental domain to the quotient space π X | F X : F X → X/G as a chart in the sense of diﬀerential geometry, and so F X locally parametrises X/G .
In the invariant case, we can approximate α = α ◦ π X | F X by approximating α .
In the equivariant case, we can also view π Y | F Y : F Y → Y/G as a chart, and because π Y | F Y is a bijection onto its image we can apply its inverse and approximate α = ( π Y | F Y ) − 1 ◦ α ◦ π X | F X by approximating α .
Note that π X | F X and π Y | F Y are not surjective in general, and there is no canonical way to extend their domain to make them so.
The ﬁx for this is to perturb points to lie in the preimage of F X as discussed in Section 2.3.
Remark 11 We do not work directly with the quotient spaces in our approach because the quotient space of a vector space is not itself a vector space.
It needs to be embedded ﬁrst in another vector space before being used as the input for a neural network, say.
Finding quotient space embeddings is an extremely diﬃcult problem.
The approach in (40) is to use polynomial invariants, which can be found using an algorithm.
The problem, in addition to being computationally infeasible in practice (the target dimension in the case of 28 × 28 pixel images as in Section 3.3 would be of order 10 8 ), is that they signiﬁcantly distort the training data leading to low accuracy.
To avoid this one must ﬁnd an isometric embedding which does not distort the data.
However, this is even more diﬃcult, and likewise signiﬁcantly increases the ambient dimension of the training data.
Further details on both the distortion of training data by polynomial invariants and quotient space projections may be found in (34).
A.4. Equivariant layers in neural networks
On the face of it, the various approaches to equivariant neural networks such as (29; 9; 15) bypass the compatible map α by approximation α directly, restricting the space of maps which can be used.
The central problem in these approaches is computing what the possible equivariant maps are, and (15) gives a very general approach to this problem.
The uniﬁed approach we discuss above provides another possible approach to this problem based on geometric methods as opposed to representation theory.
We sketch this in Appendix E.2.
Appendix B. Other combinatorial projection maps
There are three natural variations of the combinatorial projection map π ↑ we deﬁned in
Section 2.3 which may be more suited to speciﬁc applications.
We called that projection an ascending projection .
The variations are a descending projection π ↓ , and ascending and descending averaging projections π ↑ av and π ↓ av .
These projections each have their own version of Theorem 6 whose proof is essentially identical.
The descending projection is deﬁned via φ ↓ , which diﬀers from φ ↑ only when we deﬁne g i .
In this case G i − 1 acts transitively on ∆ i , and we choose j ∈ ∆ i such that the j th entry of ( g i − 1 · · · g 1 ) ·
ˆ x is maximal among those entries indexed by ∆ i .
Choose g
i ∈
G i − 1 such that j · g i = g − 1 i ( j ) =
b i .
If the input data for the machine learning algorithm consisted of vectors containing non-negative entries including many zeros, the descending projection in some sense prioritises the non-zero entries, so may yield diﬀerent results.
For the averaging projections, assume that G = H 1 × H 2 is a direct product of groups
H j (cid:54) S n j which acts the space of n 1 × n 2 matrices, R n 1 ⊗ R n 2 , by letting H 1 permute rows and H 2 permute columns.
In this case, identify N with the set of pairs
[Equation]
Deﬁne a transformation µ : R n 1 ⊗ R n 2 → R n 1 ⊗ R n 2 by µ : ( x lm ) lm (cid:55)→
[Equation]
Notice this is a G -equivariant linear map which replaces each entry of ( x lm ) lm by the sum of the averages of the entries in its row and column.
Now for any x ∈ R n 1 ⊗ R n 2 we deﬁne
[Equation]
and
[Equation]
These deﬁnitions generalise in the obvious way to the case G = (cid:81) rj
=1 H j acting on (cid:78) rj
=1 R n j component-wise, where H j (cid:54) S n j .
One might wish to use an averaging projection if, for example, one of the H j ’s is trivial, in which case a non-averaging projection ignores most of the entries, since they will not be in any of the orbits ∆ i .
This is the case in the application discussed in Section 3.3.
Example 12 Let G =
Z 3 × S 3 (cid:54) S 3 × S 3 act on R 3 ⊗ R 3 , thought of as the set of 3 ×matrices, by cyclically permuting the rows and freely permuting the columns.
In this case let
N = { ( l, m ) | 1 (cid:54) l (cid:54) 3 , 1 (cid:54) m (cid:54) 3 } and construct a base.
Let b 1 = (1 , 1) whose stabiliser is
G 1 = { 1 } × Sym ( { 2 , 3 } ) , and the orbit of b 1 under G
= G is ∆ 1 = N .
Now (2 , 1) and (3 , 1) are both ﬁxed by G 1 and so should not be the next element of the base.
Choose b 2 = (1 , 2) .
Then G 2 = { 1 } × { 1 } and the orbit of b 2 under G 1 is ∆ 2 = { (1 , 2) , (1 , 3) } .
Since G 2 ∼ = { 1 } we are done and B =
Let x (cid:48)
=
( x (cid:48) lm ) lm be a 3 × 3 matrix whose entries are distinct, we want to compute φ ↑ ( x (cid:48) ) .
Let ( p 1 , q 1 ) ∈ ∆ 1 = N be the pair such that x (cid:48)
p 1 q 1 = 1 is the minimal entry in x (cid:48) .
Then we can choose g 1 =
( s 1 , (1 q 1 ))
∈
Z 3 × S 3 where [Equation] [Equation] p
[Equation] [Equation]
) p
[Equation]
∈
[Equation]
Now let g 1 · x (cid:48) =
( x (cid:48)(cid:48) lm ) lm , and let (1 ,
q 2 ) ∈ ∆
= { (1 , 2) , (1 , 3) } minimise x (cid:48)(cid:48) 1 q 2 .
Deﬁne g 2 =
((1) , (2 q 2 ))
∈ G 1 and φ ↑ ( x (cid:48) ) =
g 2 g 1 .
Combinatorially we can describe the projection π ↑ as follows: transport the smallest entry of x (cid:48) to the top left corner by cyclically permuting rows and freely permuting columns, and then order columns 2 and 3 so that the entries in the ﬁrst row increase.
As an example, consider the matrix x and perturbation matrix ε x =  
[Equation]
[Equation]
  91 56   61 98  .
We can now apply π ↑ ( x ) =
φ ↑ ( x (cid:48) )
· x in the two step process described above: x
[Equation]
 [Equation] [Equation] [Equation]

[Equation]
    98 6156 91  g
[Equation]
(cid:55)−→     98 2756 57  = π ↑ ( x (cid:48) )
[Equation]
 [Equation] [Equation] [Equation]
 [Equation]  [Equation]
[- g 2 (cid:55)−→   0 
-][Equation] [Equation]  [Equation]
0 05 13 3
[Equation] x ) .
Similarly π ↓ ( x ) =  
[Equation]
 
[Equation]
We can also compute the averaging versions of these projections µ ( x ) =
1  23 1916 21 17  = ⇒ π ↑ av ( x ) =   0 01 53 3
 , and π ↓ av ( x ) =   5 34 03 5
 .
Appendix C. Examples of combinatorial projection maps
In this section we list combinatorial projection maps for several common examples of groups
G (cid:54) S n .
Notice that in each of the four examples of concrete groups below, implementation via a suitable sorting function circumvents the need to perturb inputs initially.
C.1. The symmetric group
If G = S n , let N = { 1 , . .
. , n } and we can choose the base B = (1 , 2 , . . .
, n − 1) .
The ascending projection π ↑ ( x ) permutes the entries so that they increase from left-to-right, and the descending projection π ↓ ( x ) permutes the entries so that they decrease.
C.2. The alternating group
If G =
A n < S n is the group of even permutations, we can choose B = (1 , 2 , . . . , n − 2) and the ascending (resp. descending) projection permutes the entries of x so that the ﬁrst n −entries increase (resp. decrease) from left-to-right, and the last two entries are greater than or equal to all the other entries.
If x contains repeated entries then the last to entries can also be ordered to be increasing (resp.
decreasing); otherwise their relative order depends on whether the permutation s x (cid:48) which maps i (cid:55)→ x (cid:48) i for 1 (cid:54) i (cid:54) n , is an even or odd permutation (see Section 2.3 for the deﬁnition of x (cid:48) i ).
C.3. The cyclic group
If G = Z n (cid:54) S n is the cyclic group generated by the permutation (1 2 · · · n ) , we can choose the base B = (1) .
The ascending (resp. descending) projection cyclically permutes the entries of x so that the ﬁrst entry is less (resp.
greater) than or equal to all other entries of x .
C.4. The dihedral group
If G = D n (cid:54) S n is the dihedral group generated by s 1 = (1 2 · · · n ) and s 2 = [-(
 [Equation]-](2 n )(3 ( n − 1))(4 ( n − 2)) · · · , we can choose base B = (1 , 2) .
The ascending (resp. descending) projection cyclically permutes the entries of x via s 1 so that the ﬁrst entry is less (resp.
greater) than or equal to all other entries of x , and then if the ﬁnal entry is less (resp. greater) than the second entry, it applies the permutation s 2 .
C.5. Products of groups acting on products of spaces
Suppose G = (cid:81) rj
=1 H j where H j (cid:54) S n j acts on (cid:76) rj
=1 R n j by each H j acting by permutations on the corresponding space R n j and trivially everywhere else.
Let
[Equation]
be a base for H j acting on R n j , then
[Equation]
is a base for G .
Let π j ↑ : R n j → R n j be the ascending projection corresponding to B j .
Then deﬁne π ↑ =
(cid:76) rj =
π j ↑ , to be the projection which equals π j ↑ when restricted to R n j .
Similarly π ↓ = (cid:76) rj =
π j ↓ .
C.6. Products of groups acting on tensors of spaces
Suppose G = (cid:81) rj
=1 H j where H j (cid:54) S n j acts on (cid:78) rj
=1 R n j by each H j acting by permutations on the j th component of (cid:78) rj =
1 R n j , and trivially on the other components.
For each 1 (cid:54) j (cid:54) r let B
j
= (cid:16) b (1) j , . . .
, b ( k j ) j (cid:17) ⊂ { 1 , . . . , n j } =
N j be a base for H j acting on R n j , and furthermore (for convenience) assume
that b (1) j = 1 .
Then choose B ⊂ (cid:81) rj
=1 N j
=: N to be
[Equation]
where a 1 in the j th position of an element of B should be thought of as b (1) j .
Suppose x = ( x l 1 ··· l r ) l 1 ··· l r ∈ (cid:78) rj
=1 R n j , and let x (cid:48) be as in Section 2.3.
Choose ( m 1 , . . .
, m r ) ∈ N to be the index in the G -orbit of 1 = (1 , . .
. , 1) with minimal entry in x (cid:48) .
For 1 (cid:54) j (cid:54) r deﬁne x (cid:48)
j :=
( x (cid:48) m 1 ··· l j ··· m r ) 1 (cid:54) l j (cid:54) n j ∈ R n j , which is the restriction of x (cid:48) to the R n j -vector containing the entry x (cid:48) m 1 ··· m j ··· m r .
Then deﬁne
[Equation]
where φ j ↑ : R n j → H j is the function deﬁned for H j acting on R n j , and similarly deﬁne φ ↓ ( x ) .
Then as before, π ↑ ( x ) :=
φ ↑ ( x ) · x and π ↓ ( x ) :
=
φ ↓ ( x ) · x .
Appendix D. Proof of Theorem
The idea of the proof is as follows.
In Appendix D.1 we shall outline an equivalence between subgroups of S n acting on R n by permuting coordinates, and them acting on S n by multiplication.
This will provide a dictionary between certain combinatorially deﬁned fundamental domains and sets of coset representatives satisfying simple algebraic properties,
Theorem 17.
We will then outline the work from (12) in Appendix D.3 which gives an algorithm to ﬁnd a set of coset representatives for an arbitrary subgroup of S n .
The main work is then to show this algorithm, with modiﬁcations, can produce a set of coset representatives with the desired algebraic properties so that it corresponds to a fundamental domain, culminating in Theorem 24.
Finally we will show in Theorem 25 that the algorithm outlined in Section 2.3 indeed produces a projection onto this fundamental domain.
D.1. Actions on R n and S n
Recall we have the group S n acting on R n on the left by s · ( x i ) i
= (cid:0) x
s − 1 ( i ) (cid:1) i .
We also have the normal action of S n on itself on the left by group multiplication: s acts on t by s · t = st for any s, t ∈ S n .
Here we shall show that in some sense these actions are equivalent.
This correspondence is known, at least to experts, so we will only outline the essential points.
Let x ∈ R n be a point, all of whose entries are distinct, and notice the set of such points is open and dense in R n .
Deﬁne a function which changes the i th entry x i of x to the integer |{ 1 (cid:54) j (cid:54) n | x j (cid:54) x i }| .
The result will be a list of the integers 1 , . .
. , n in the same relative order as the entries of x , and we denote the set of all such points C .
We can think of C as a discrete subset of R n , and the left action of S n on R n restricts to a left action on C .
Notice also that this map R n dist := { x ∈ R n | all entries are distinct } → C is continuous.
In other words the set of connected components of R n dist is in one-to-one correspondence with C , and indeed each component contains a point in C , its representative point .
We call these connected components chambers , and given c ∈ C we will write [ c ] ⊂ R n for the corresponding chamber.
The following is easy to check.
Lemma 13 Each chamber is a fundamental domain for the action of S n on R n .
The action of S n on R n preserves an ( n − 1) -simplex
in the orthogonal complement of the vector (1 , . . . , 1) .
In Figure 3 we show the 3-simplex preserved by S 4 , and use it to visualise the 24 =
| S 4 | chambers in this case.
On the other hand, we can view each element of C as a permutation in S n written in inline notation .
This means if c = ( c i )
i , as a permutation it sends i to c
i for each i ∈ { 1 , . . . , n } .
Thus S n is in one-to-one correspondence with C .
In fact, it is better in our situation to modify this correspondence by inverting elements of S n via the map ρ : S n → C : s (cid:55)→ ( s − 1 ( i ))
i .
The equivalence of the left action of S n on R n and the left action on itself comes in the following form.
Let s, t ∈ S n , and consider the action of s on ρ ( t ) :
[Equation]
Given any subgroup G (cid:54) S n , the map ρ deﬁnes an equivalence between G acting on R n , which restricts to an action of G on C , and G acting on S n by left multiplication.
We can use this equivalence to convert a set of right coset representatives for G in S n into a complete set of orbit representatives (deﬁned in Section 2.1) for G acting on R n .
Proposition
Let R be a set of right coset representatives for G (cid:54) S n , then the set
F = (cid:83) r ∈ R [ ρ ( r )] is a complete set of orbit representatives for G acting on R n , where [ ρ ( r )] is the closure of the chamber containing ρ ( r ) .
Proof Since R n dist is dense in R n and G acts by continuous maps which leave R n dist invariant as a set, it suﬃces to show that (cid:83) r ∈ R [ ρ ( r )] is a complete set of orbit representatives for G acting on R n dist .
In fact G simply permutes the components of R n dist so it suﬃces to show that (cid:83) r ∈ R ρ ( r ) is a complete set of orbit representatives for the induced action of G on C .
But now, ρ is a bijection which exhibits an equivalence between the action of G on C and the action of G on S n
so we just need to show that R is a complete set of orbit representatives for G acting on S n .
The orbits of this action are precisely the right cosets of G , which completes the proof.
D.2. Gallery connectedness and fundamental domains
Given a set of right coset representatives R for G (cid:54) S n , the interior of F as deﬁned in the proposition will generally not be a fundamental domain because it will not be connected.
We can reinterpret connectedness in terms of algebraic properties of R .
First some geometric deﬁnitions.
Deﬁnition 15 Let c, c (cid:48)
∈
C be distinct, we say the chambers [ c ] and [ c (cid:48) ] are adjacent if [ c ] ∩ [ c (cid:48) ] has codimension 1.
A gallery is a sequence of chambers [ c 1 ] , . . .
, [ c k ] such that consecutive chambers are adjacent.
A set of chambers is called gallery connected if any two distinct chambers in the set can be connected by a gallery which is completely contained in the set.
As a shorthand, we sometimes call a subset C (cid:48)
⊂
C gallery connected if the set { [ c ] | c ∈ C (cid:48) } is gallery connected.
It turns out that the decomposition of R n dist into chambers corresponds to the chamber system of S n acting on its Coxeter complex, about which we will not elaborate here, but the interested reader should consult (4).
The upshot of this viewpoint is two characterisations of adjacency of chambers.
Lemma 16 Let c (cid:54) =
c (cid:48) ∈ C and deﬁne s =
ρ − 1 ( c ) , s (cid:48)
=
ρ − 1 ( c (cid:48) ) .
Then the following are equivalent:
The chambers [ c ] and [ c (cid:48) ] are adjacent 2.
There is 1 (cid:54) j (cid:54) n − 1 such that s (cid:48) =
s ( j j + 1) where ( j j + 1) is a transposition in S n 3.
The vectors c and c (cid:48) diﬀer by swapping exactly two entries which are consecutive integers
Proof The equivalence of (1) and (2) is proved in, for example, Theorem I.5A of (4).
To see the equivalence of (2) and (3), notice that ρ ( s ( j j + 1)) =
(( s ( j j + 1)) − 1 ( i ))
i =:
( c (cid:48) i ) i .
For i (cid:54)∈ { j, j + 1 } , c (cid:48) i =
s ( i )
=
c i (where c :=
( c i ) i ), whereas c (cid:48)
j = c j +1 and c (cid:48)
j +1 =
c j .
The equivalence of (1) and 3 for the example of S 4 can be seen in (Figure 3).
We will use this characterisation to prove Theorem 23 which is key to showing that the image of π ↑ is connected.
We are now in a position to upgrade Theorem 14 so that it produces a fundamental domain for the action of G .
To state this, we deﬁne a right transversal of
G (cid:54) S n to be a minimal set of right coset representatives (ie a set containing exactly one element from every right coset).
Proposition 17 Let R ⊂ S n be a right transversal for G (cid:54) S n such that ρ ( R ) is gallery connected.
Then F , the interior of (cid:83) r ∈ R [ ρ ( r )] , is a fundamental domain for G acting on
R n .
Proof By the deﬁnition, if [ c ] and [ c (cid:48) ] are adjacent, then the interior of [ c ] ∪ [ c (cid:48) ] will be connected.
By induction on the length of galleries in { [ ρ ( r )] | r ∈ R } it follows that F is connected.
It is also open by deﬁnition.
By Theorem 14 we know that F is a complete set of orbit representatives for G .
Finally, suppose that some G -orbit meets F in at least two points, say x and x (cid:48) and g ∈ G is such that g · x = x (cid:48) .
Since the G -action permutes the chambers, there are two possibilities:
There are coset representatives r, r (
cid:48) ∈ R such that x ∈
[ ρ ( r )] and x (cid:48) ∈
[ ρ ( r (cid:48) )] .
There are coset representatives r 1 (cid:54) =
r 2 , r (cid:48) 1 (cid:54) =
r (cid:48) 2 ∈ R such that x ∈
[ ρ ( r 1 )] ∩ [ ρ ( r 2 )] and x (cid:48) ∈
[ ρ ( r (cid:48) 1 )]
∩ [ ρ ( r (cid:48) 2 )] .
In the ﬁrst case we must have that g · [ ρ ( r )]
=
[ ρ ( r (cid:48) )] , in which case it follows from
Theorem 13 and the fact that g (cid:54) = 1 , that r (cid:54) =
r (cid:48) .
But then by the equivalence of the action with the action on S n , we have that g · r = gr = r (cid:48) and r and r (cid:48) represent the same right coset of G .
This contradicts the assumption that R is minimal.
In the second case we can similarly argue that { r 1 , r 2 } (cid:54) =
{ r (cid:48) 1 , r (cid:48) 2 } but g · { r 1 , r 2 } =
{ r (cid:48) 1 , r (cid:48) 2 } , again contradicting the minimality of R .
In either case g cannot exist.
D.3. An algorithm to ﬁnd coset representatives
In this section we will summarise the main construction of (12) which gives an eﬃcient algorithm to compute a right transversal for an arbitrary subgroup G (cid:54) S n .
The ﬁrst step, as it is to ﬁnd a base B ⊂ N for G (cid:54) S n .
Set B 0 = () , the empty tuple.
We will assume that we have already constructed B i − 1 and computed G i − 1 .
If G i − 1 = { 1 } , B = B i − 1 is a base and we are done.
Otherwise, pick b i ∈ N with the largest orbit under G i − 1 and let B i be B i − 1 with b i appended.
Let B = ( b 1 , . . .
, b k ) be a base and recall we deﬁne G
= G and G i
= G i − 1 ∩ Stab G ( b i ) for 1 (cid:54) i (cid:54) k .
We also write ∆ i =
b i · G i − 1 for the orbit of b i under G i − 1 .
Recursively construct a partition Π i of N , starting with Π 0 = { N } .
Denote by Γ i the element of Π i −which contains b i .
One can check by induction that Γ i contains ∆ i as a subset.
Deﬁne Π
i by replacing Γ i in Π i − 1 by the non-empty subsets from the list: { b i } , ∆ i − { b i } , and Γ i − ∆ i .
Now let U i be a right transversal for the group Sym (∆ i )
×
Sym (Γ i − ∆ i ) in Sym (Γ i ) , where Sym (Ω) is the group of permutations of the set Ω (in the next section we will ﬁx a particular choice for U i ), and ﬁnally let
[Equation]
Then deﬁne R
= H k
U k
U k − 1 · · · U 1 , where for subsets A, B ⊂ S n , AB := { ab | a ∈ A, b ∈ B } .
Theorem 18 ((12) § 4 )
The set R is a right transversal for G (cid:54) S n .
D.4. Gallery connected sets of coset representatives
We will now show how the method described above can be used to construct a right transversal R for G such that ρ ( R ) is gallery connected.
This is done by choosing a suitable base B , possibly re-indexing the set N , and choosing appropriate right transversals
U i for
Sym (∆ i ) ×
Sym (Γ i − ∆ i ) in Sym (Γ i ) .
We will prove Theorem 6 assuming B and N have been chosen in this way, and then in Appendix D.6 show that the assumptions on B and N can be dropped.
We described how to ﬁnd a base for G by appending more and more elements of N to
B = () until G k = { 1 } in Appendix D.3.
The ﬁrst assumption we make is that each new b i is minimal in the orbit b i · G i − 1 with respect to the normal ordering on N .
We will call such a base orbit minimal .
We will use the following lemma to build gallery connected sets out of other gallery connected sets.
Lemma 19 Let A 1 , . . .
, A l be subsets of S n so that each contains the identity permutation (1) , and ρ ( A i ) is gallery connected for each i .
Then ρ ( A 1 A 2 · · · A l ) is gallery connected.
Proof Notice that A 1 A 2 contains (1)(1) =
Let a ∈
A 1 A 2 , and choose a 1 ∈
A 1 and a 2 ∈ A 2 such that a = a 1 a 2 .
Since ρ ( A 1 ) and ρ ( A 2 ) are gallery connected, there are galleries p a 1 ⊂ ρ
( A 1 ) and p a 2 ⊂ ρ ( A 2 ) which connect ρ ((1)) to ρ ( a 1 ) and ρ ((1)) to ρ ( a 2 ) respectively.
Then a 1 · p a 2 connects ρ ( a 1 ) to ρ ( a 1 a 2 ) in ρ ( a 1 A 2 ) , and the concatenation p a 1 ∗ ( a 1 · p a 2 ) is a gallery which connects ρ ((1)) to ρ ( a ) in ρ ( A 1 A 2 ) .
Call this gallery ˜ p a , its construction is illustrated in Figure 4.
Now given a, a (cid:48) in A 1 , A 2 the gallery ˜ p − 1 a ∪ ˜ p
a (cid:48) (where ˜ p − 1 a indicates ˜ p a traversed in reverse) connects ρ ( a ) to ρ ( a (cid:48) ) in ρ ( A 1 A 2 ) , so ρ ( A 1 A 2 ) is gallery connected—the claim follows by induction on l . [Equation] [Figure] [Equation]
a [Equation] · p a [Equation]
aa [Equation] [Figure] From the deﬁnition of R in the previous section, if we show that H k and each of the U
i ’s satisfy the hypotheses of this lemma, then it will follow that R is gallery connected.
We will ﬁrst consider H k .
Notice that in fact, if Π k = { N 1 , . . .
, N l } then
[Equation]
can be written as a product of sets, again as in the lemma.
Each Sym ( N i ) contains (1) , so we just need ρ ( Sym ( N i )) to be gallery connected for each i .
It follows immediately from
Theorem 16 that this is the case if and only if N i is a sequence of consecutive digits from N .
In general this will not be the case, however it can be readily achieved by re-indexing the set N .
In fact we can do this so that each part of each partition Π i is a set of consecutive digits, which will subsequently aid in showing that ρ ( U i ) is gallery connected.
Lemma 20 We can re-index N so that b i remains minimal in ∆ i and each part of Π
i is a set of consecutive digits for 1 (cid:54) i (cid:54) k .
Proof We do induction on i : note that in Π 0 = { N } the only part is a set of consecutive numbers.
Assume that each element of Π i − 1 is a set of consecutive digits, in particular Γ
i ∈ Π i − 1 is a set of consecutive digits.
Assume one of the three subsets { b i } , ∆ i − { b i } or Γ i − ∆ i is non-empty and does not consist of consecutive digits, then by the minimality of b i both ∆ i − { b i } and Γ i − ∆ i must be non-empty and not consist of consecutive digits.
Re-index the elements of Γ i so that overall the same set of digits is used, but now b i is the smallest, the next smallest digits lie in ∆ i − { b i } , and the remaining digits are in Γ i − ∆ i .
D.5. Choosing a right transversal U i
Finally we want to choose the right transversals
U i for Sym (∆ i )
×
Sym (Γ i − ∆ i ) in Sym (Γ i ) .
Write ∆ i = { d 1 , d 2 , . .
d m } , and Γ i − ∆ i = { d m +1 , d m +2 , . . .
d m + m (cid:48) } .
For some integer 0 (cid:54) l (cid:54) min { m, m (cid:48) } , choose d j 1 < d j 2 < · · · < d j l and d m + j (cid:48) 1 < d m + j (cid:48) 2 < · · · < d m + j (cid:48) l , and consider the product of transpositions
[Equation]
Deﬁne (cid:101) U i to be the set of all such products for any choice of l , and indices j k and j (cid:48) k .
Lemma 21 ((12) Lemma 2) (cid:101) U
i is a right transversal for Sym (∆ i ) ×
Sym (Γ i − ∆ i ) in
Sym (Γ i ) .
Let ˜ u ∈ (cid:101)
U
i have the form given in Equation (4).
If l = 0 , then ˜ u is the identity and thinking of it as an element of S n (cid:62)
Sym (Γ i ) , we get ρ ((1))
= (1 , . . . , n ) .
More generally ρ (˜ u ) will be the result of swapping each of the pairs d j k ↔ d m + j (cid:48)
k in this vector, for 1 (cid:54)
k (cid:54) l .
Let g ˜
u ∈ Sym (∆ i ) ×
Sym (Γ i − ∆ i ) be the permutation such that ρ ( g ˜ u · ˜ u ) has its ﬁrst m entries in increasing order, and
its last m (cid:48) entries in increasing order.
Deﬁne U
i = { g ˜ u · ˜ u | ˜ u ∈ (cid:101) U i } .
Lemma 22 U
i is a right transversal for Sym (∆ i ) ×
Sym (Γ i − ∆ i ) in Sym (Γ i ) .
Proof (cid:101) U i contains exactly one element from each right coset of Sym (∆ i ) ×
Sym (Γ i − ∆ i ) in Sym (Γ i ) .
For ˜ u ∈ (cid:101) U i , the element g ˜ u · ˜ u
= g ˜ u
˜ u lies in the same right coset as ˜ u since g ˜ u
∈
Sym (∆ i ) ×
Sym (Γ i − ∆ i ) .
Hence U i contains exactly one element from each right coset of Sym (∆ i ) ×
Sym (Γ i − ∆ i ) in Sym (Γ i ) .
We want to show that ρ ( U i ) is gallery connected, and for that we will use the re-indexing of N provided by Theorem 20.
Recall that b
i is the i th element of the base B ; it follows from our construction of U i that [Equation] indexed
[Equation]
indexed by
[Equation]
c j ∈ Γ i = { b i , b i + 1 , . . . , b i + m + m (cid:48) − 1 } for all 1 (cid:54) j (cid:54) m + m (cid:48) , c 1 < · · · < c m , and c m +1 < · · · < c m + m (cid:48) }
[Equation]
Since this is notationally rather cumbersome, we will abbreviate elements of ρ ( U i ) by
[Equation]
where the ﬁrst ‘half’ consists of entries indexed by ∆ i , and the second ‘half’ consists of entries indexed by Γ i − ∆ i .
Proposition 23 Let G (cid:54) S n , B be an orbit minimal base, and N indexed so that each part of Π i is a set of consecutive digits.
Then the set ρ ( U i ) is gallery connected.
Proof To help simplify notation, we will not distinguish between points in C and the chambers they represent.
We will show that ρ ( U i ) is gallery connected by explicitly constructing a gallery which joins an arbitrary chamber
[Equation]
to the chamber corresponding to the identity in U i ,
[Equation]
Theorem 16 gives the condition for consecutive chambers in this gallery to be adjacent: they must diﬀer by swapping two entries which are consecutive integers.
Furthermore we will ensure this gallery remains in ρ ( U i ) throughout.
This implies that after swapping the two entries, the two halves of c must remain properly ordered.
Taken together, this implies that:
The only swaps we can perform must switch the position of an entry in the left half with one in the right half, and these entries must be consecutive integers
Let c ∈ ρ ( U i ) be arbitrary, write ˆ c j = b
i + j − 1 for the j th entry of ˆ c , and deﬁne
[Equation]
 which measures the degree to which c and ˆ c diﬀer.
Claim
[Equation]
Let j (cid:54) m , then since the entries in the left half of c are ordered, distinct integers greater than or equal to b i ,
c j (cid:62) b
i + ( j − 1)
=
ˆ c j so each term in the ﬁrst sum is non-negative.
Similarly, for j > m the entries in the right half of c are ordered, distinct integers less than or equal to b i + m + m (cid:48) − 1 , so
c j (cid:54) b
i + m + m (cid:48) − 1 − ( m + m (cid:48)
− j )
=
b i + ( j − 1)
=
ˆ c j so each term in the second sum is non-positive.
(cid:3)
As a remark, it follows from this claim that δ equals the L 1 distance between c and ˆ c .
We shall perform a sequence of swaps as described above which have the eﬀect decreasing the value of δ ( c ) .
Since δ ( c ) = 0 implies that c
= ˆ c , the required gallery can be constructed by induction on δ ( c ) .
Assume c (cid:54) =
ˆ c , and let j be the minimal index such that c j (cid:54)
=
ˆ c j .
Since the two halves of c are ordered, c j is in the left half.
Claim
[Equation]
c [Equation] Indeed suppose it is in the left half, then by the ordering on c , j (cid:48) <
j , and by the minimality of j , c j (cid:48)
=
ˆ c j (cid:48)
=
b i + j (cid:48) − 1 .
But then ˆ c j (cid:54) = c j = c j (cid:48)
{++ 
 = b i + ( j (cid:48) + 1) − 
 = ˆ c j (cid:48)+}
+1 , so j (cid:54) =
j (cid:48) +since all entries of ˆ c are distinct.
But now c j (cid:48)
<
c j (cid:48)
+1 <
c j (by the ordering on c ), which contradicts the fact that these entries are distinct integers, and c j − c j (cid:48)
= 1 .
(cid:3)
Thus, c j and c j − 1 are entries in diﬀerent halves of c which are consecutive integers.
Let c (cid:48) be the result of swapping these two entries in c , then
[Equation]
so performing the swap strictly decreases δ .
By induction, there is a gallery in ρ ( U i ) joining c and ˆ c , and hence ρ ( U i ) is gallery connected.
It follows directly from this proposition, Theorem 18, and Theorem 17 that R as deﬁned in Appendix D.3 corresponds to a fundamental domain.
Corollary 24 Let G (cid:54) S n , B be an orbit minimal base, and N indexed so that each part of Π i is a set of consecutive digits.
Let R be the right transversal for G constructed above, then F , the interior of (cid:83) r ∈ R [ ρ ( r )] , is a fundamental domain for G acting on R n .
D.6. Finishing oﬀ the proof
To complete the proof of Theorem 6 we have two tasks: ﬁrst show that the map π ↑ as deﬁned in Section 2.3 has image in F =
(cid:83) r ∈ R [ ρ ( r )] , and so indeed projects onto a fundamental domain; and then remove the assumptions of orbit minimality and on how N is indexed.
Proposition 25 Let G (cid:54) S n , B be an orbit minimal base, and N indexed so that each part of Π i is a set of consecutive digits.
Then the image of π ↑ lies in (cid:83) r ∈ R [ ρ ( r )] .
Proof It suﬃces to show that the image of R n dist lies in F =
(cid:83) r ∈ R [ ρ ( r )] .
We claim that
[Equation]
It is clear that the deﬁnition of π ↑ implies that the right hand side of this is the image of π ↑ | C , so the proposition follows immediately from this claim.
Call the set on the right hand side C (cid:48) , ﬁrst we will show that ρ ( R ) ⊆
C (cid:48) .
By Equation (5) (note that the entries of ( c j ) j there are indexed diﬀerently there) we can see
[Equation]
all
[Equation]
Since U i ⊂
Sym (Γ i ) , which ﬁxes b i − 1 for i (cid:62) 2 , one can inductively check from the deﬁnition of ρ that ρ ( U k · · · U 1 ) ⊂ C (cid:48) .
Similarly, in the partition Π k , each b i appears as a singleton, so
H k also ﬁxes b i for 1 (cid:54) i (cid:54) k , hence ρ ( R ) ⊆
C (cid:48) .
To establish the claim we just need to show that | C (cid:48) | = | ρ ( R ) | ; since they are ﬁnite sets, this implies that they are equal.
On the one hand, since ρ is a bijection, and using Lagrange’s Theorem
[Equation]
On the other hand, each condition ‘ c b i (cid:54) c j for all j ∈ ∆ i ’ decreases the size of C by a factor of | ∆ i | , so
[Equation]
Since C is the bijective image of S n under ρ , | C | = | S n | .
By the Orbit-Stabiliser Theorem
[Equation]
Where the last equality follows from the deﬁnition G i
=
Stab G i − 1 ( b i ) .
Therefore
[Equation]
Hence | C (cid:48)
| = | ρ ( R ) | , which completes the proof.
Proof [Proof of Theorem 6] Let N = { 1 , . . . , n } , and choose B a base for G (cid:54) S n , and ε satisfying the conditions in Section 2.3.
Let s ∈ S n be a permutation of N such that
B s := B · s is an orbit minimal base, and each part of each partition Π si := Π
i · s is a set of consecutive digits.
That s exists is clear by ﬁrst permuting k times so that B is orbit minimal (note b
i (cid:54)∈ ∆ j for all j > i ) and then applying Theorem 20.
Write b si =
b
i · s so that
B s =
( b s 1 , . . .
, b sk ) .
Let G s =
s −
Gs be the conjugate of G by s in S n , then for any g ∈ G and m ∈ N ,
[Equation]
where g s = s − 1 gs .
In other words, permuting by s and then acting by G s is the same as acting by G and then permuting by s .
It follows that G si :=
s − 1 G i s =
G si − 1 ∩ Stab G s ( b si ) , and ∆ si :=
∆ i · s = b si · G si − 1 .
Finally deﬁne φ s ↑ and π s ↑ as in Section 2.3 with respect to B s and ε .
We claim that for x (cid:48) as deﬁned in Section 2.3, φ s ↑ ( x (cid:48) ) =
( φ ↑ ( x (cid:48) ))
s
= g sx (cid:48) .
Indeed by deﬁnition φ s ↑ ( x (cid:48) )
=
˜
g k · · · ˜ g 1 where ˜ g i ∈ G si such that ˜  ·
˜ g
i =
b si and ˜  ∈ ∆ si is chosen such that the ˜  th entry of (˜ g
i − 1 · · · ˜ g 1 ) · x (cid:48) is minimal among those entries indexed by ∆ si .
But now ∆ si = ∆ i · s means ˜  =
j · s (where j ∈ ∆ i is the index found in the deﬁnition of φ ↑ ).
Thus
[Equation]
so we can certainly choose ˜ g
i = g si .
Then as claimed
[Equation]
Expanding out φ s ( x (cid:48) )
=
s − 1 φ ( x (cid:48) )
s , we can now compute π s ↑ in terms of π ↑ and s :
[Equation]
Writing F for the interior of the image of π ↑ , and F s for the interior of the image of π s ↑ , this implies F s = s · F (because s − 1 · R n = R n ).
But Theorem 24 together with Theorem 25 says that F s is a fundamental domain for G s and π s ↑ is a projection onto F s ; so F = s − 1 · F s is a fundamental domain for G and π ↑ is a projection onto F .
To prove the ﬁnal claim of the theorem, that π ↑ : R n → R n is uniquely deﬁned by the choice of B and ε , we just need to show that a diﬀerent choice of the elements g 1 , . .
, g k given x ∈ R n does not change φ ↑ .
In fact φ ↑ is determined completely by what it does to the points x
(cid:48) ∈ R n dist , and φ ↑ ( x (cid:48) ) will lie inside the fundamental domain (not on its boundary).
By the deﬁnition of a fundamental domain, any diﬀerent choice g (cid:48) 1 , . .
, g (cid:48)
k must necessarily combine to give the same element g x (cid:48) (no non-trivial element of G acts trivially), and hence φ ↑ is uniquely determined.
Appendix E. Other mathematical results E.1. Universal approximation theorem
The universal approximation theorem is a fundamental result in the theory of machine learning that any continuous function α : X → R m on a compact subset X ⊂ R n can be arbitrarily well approximated by a neural network with one hidden layer.
To state the theorem precisely, one needs to specify that closeness between two continuous functions α, β : X → R m is measured by the L p norm (for 1 (cid:54) p ) which is given by (cid:107) f (cid:107) p
= (cid:18)(cid:90) X f ( x ) p dµ ( x ) (cid:19) 1 p , where µ is the Lebesgue measure on X ⊂ R n . Then the distance between α, β is (cid:107) α − β (cid:107) p .
Denote by L p ( µ, R m ) the set of functions f : X → R m such that (cid:107) f (cid:107) p < ∞ , and by M n,m ( σ ) the set of functions R n → R m implemented by a neural network with activation function σ : R → R , one hidden layer with arbitrarily many neurons and m output neurons.
The universal approximation theorem, as in Theorem 1 of (22), states that if σ is unbounded and non-constant, then M n,m ( σ ) is dense in L p ( µ, R m ) (for any compact subset X of R n ).
To obtain a G -invariant version of this theorem, assume that X is a G -invariant subset of R n and let L pG ( µ, R m ) be the set of G -invariant functions f on X such that (cid:107) f (cid:107) p < ∞ . Denote by M Gn,m ( π, σ ) the set of functions of the form α ◦ π where π : X → R n is a projection onto a fundamental domain, and α ∈ M l,m ( σ ) .
Theorem 26 ( G -invariant Universal Approximation Theorem) Let X ⊂ R n be a compact G -invariant subset, σ an unbounded and non-constant function and π : R n → R n a projection onto a fundamental domain. Then M Gn,m ( π, σ ) is dense in L pG ( µ, R m ) .
Proof We need to show that given ε > 0 , a G -invariant map α : X → R m ∈ L pG ( µ, R m ) , and a projection π : R n → R n , there is a neural network β ∈ M l,m ( σ ) such that (cid:107) α − β ◦ π (cid:107) p < ε .
Let α : π ( X ) → R m be the restriction of α to the intersection of the closure of the fundamental domain with X , ie π ( X ) . Since α ∈ L pG ( µ, R m ) , it follows immediately that α ∈ L pG ( µ, R m ) , where µ is the restriction of µ to π ( X ) .
By the universal approximation theorem (Theorem 1 in (22)) applied to π ( X ) , there is a network β : R n → R m such that
[Equation]
Furthermore, since α is G -invariant, α = α ◦ π , which implies that
[Equation]
as required.
E.2. Computing the space of linear equivariant maps
A G -equivariant neural network (see for example (29; 9; 15)) consists of a series of Gequivariant linear maps λ i separated by some non-linear activation function σ , yielding β
= λ k ◦ σ ◦ · · · ◦ σ ◦ λ 1 .
Restrictions are placed on the learnable parameters of each λ
i to ensure they are G -equivariant.
For example, if λ i : R n → R n is equivariant with respect to
S n acting on each copy of R n by permuting coordinates, then it was shown in Lemma 3 of (41), that it must have the form
[Equation]
where a, b ∈ R are learnable parameters, I is the identity matrix, and 1 = (1 , 1 , . . .
The main task is to describe the space of all G -equivariant linear maps λ : R n 1 → R n 2 which could map between layers in the neural network.
Here we sketch an approach which is combinatorial and involves putting a G -invariant simplicial complex structure on R n i and applying the compatibility criterion Theorem 9 to the cells in the simplicial structure induced on the quotient space.
To slightly simplify matters for the purpose of exposition, assume that G acts on R n 1 and
R n 2 discretely and irreducibly by orthogonal matrices on each space.
Because the action is orthogonal (ie it ﬁxes the origin and preserves the Euclidean metric), it leaves the unit sphere
S n i − 1 invariant in R n i .
The sphere is compact and the action is properly discontinuous, so it is possible to ﬁnd some G -invariant triangulation of S n i − 1 .
This means a decomposition of S n i − 1 into ( n i − 1) -dimensional simplices (edges, triangles, tetrahedra, etc) such that each g ∈ G acts on S n i − 1 by sending k -simplices to k -simplices for each 0 (cid:54)
k (cid:54) n i − 1 .
Moreover, possibly after subdividing once, any two points in the interior of some k -simplex have the same stabiliser in G , and if some simplex σ is a face of another simplex σ (cid:48) , then
Stab G ( σ (cid:48) )
(cid:54) Stab G ( σ ) .
Now, the projection map π i : S n i − 1 → S n i − 1 /G
=:
Q i induces a simplicial structure on the quotient, and we can label each simplex σ in the quotient by the set of stabilisers of all simplices σ ∈ S n i − 1 which are mapped to σ , π i ( σ ) = σ .
In fact the label of σ deﬁned in this way will be exactly the conjugacy class of Stab G ( σ ) in G for some (equivalently, any) σ such that π i ( σ ) = σ .
We can now try to construct a compatible map λ (cid:48) :
Q 1 → Q 2 by mapping simplices σ 1 ∈ Q 1 to simplices σ 2 ∈ Q 2 such that some element of the label σ 1 is a subgroup of some element of the label of σ 2 (note σ 1 and σ 2 need not have the same dimension) so that these maps glue together in a continuous way.
This reduces the problem of checking the compatibility criterion on every point in Q 1 to only checking it on a ﬁnite number of simplices, and checking that the maps glue together, which is also a ﬁnite simplicial problem.
Appendix F. Implementation F.1. Machine Learning Experiments
All neural networks were trained using Keras.
SVMs were ﬁtted using sci-kit learn.
In each case the performance was averaged over 10 runs, and sample standard deviations are given in the tables in Section 3.
The only exception to this is the training of SimpNet reported in
Table 3, where we only trained a single run.
Cayley tables
There are ﬁve isomorphism classes of groups with eight elements: C 8 ,
C 4 × C 2 , D 4 , Q 8 , C 2 × C 2 × C 2 .
We generated a dataset of 40000 Cayley tables: a sixth of the tables were permutations of the group C 8 , another sixth of the group C 4 × C 2 , another sixth of the group D 4 , a quarter of the group Q 8 , a quarter of the group C 2 × C 2 × C 2 .
The dataset was split into training and test dataset of equal size.
Two models were compared: a fully connected neural network with two hidden layers of size 100 and 10 with activation function ReLU using the Adam optimiser with learning rate 0 .
001 and cross-entropy loss as a loss function training for 200 epochs; and a linear SVM.
CICY
The neural network architectures and training parameters for the entries MLP,
MLP + pre-processing, and Inception in Table 2 were taken from the given references.
The only diﬀerence we found is that in the case π Dir + Inception and π ↑
+ Inception on the randomly permuted dataset, accuracy increased when not removing outliers from the training data, and we therefore decided to not remove outliers from the training data in this case, contrary to what was done in (14).
The group invariant neural network had six equivariant layers (with four trainable parameters each) with 100 channels and cross-channel interactions, followed by sum pooling, and two fully connected layers with 64 and 32 neurons.
No dropout was used.
We experimented with max pooling, dropout, diﬀerent numbers and sizes of layers and found the above parameters to work best.
We found that test accuracy varied strongly for large batch sizes and eventually trained with batch size one.
We randomly split the dataset into training and test sets of equal size.
Classifying rotated handwritten digits
Three neural networks were compared: ﬁrst, a fully connected neural network with no hidden layers.
Second, a fully connected neural network with two hidden layers of size 128 and 64 with activation function ReLU.
Third, a small variation of the SimpNet architecture from (19), namely a convolutional neural network with the following layers: 13 convolutional layers with 64 channels in the ﬁrst layer andchannels in all other layers, ﬁlter size 1 × 1 in layers 11 and 12 and ﬁlter size 3 × 3 in all other layers, max pooling layers with ﬁlter size 2 × 2 and strides (2 , 2) after the fourth and seventh layer and ﬁlter size 2 × 2 and strides (1 , 1) after the 9 th , 12 th , and 13 th convolutional layers.
All followed by one global max pooling layer, two hidden fully connected layers with 128 and 64 neurons respectively, and activation function ReLU after all hidden layers.
We used the Adam optimiser with learning rate 0 .
001 and cross-entropy loss as a loss function.
We trained for 100 epochs with early stopping if the training error does not decrease forepochs.
F.2. Computing combinatorial projections
In this section we will give algorithms to compute combinatorial projection maps for permutation group actions, and analyse the time and space complexity of these algorithms.
These work for any permutation group G , although they are not the ones we used in Section 3 which employed more eﬃcient ad hoc methods described in Appendix C.
The general algorithms here fall into two parts: ﬁrst are the algorithms which are applied as a one-oﬀ to compute data like a base and the orbits ∆ i , and which run in O ( k 2 n 3 ) time, and O ( n 2 log n ) space, where n is the dimension of the input space and k is the size of the base, see subsection F.2.3.
Second are the algorithms which actually implement the projection π ↑ , and which must therefore be run for each input datum.
They do this in O ( k 2 n 2 ) time and O ( n 2 log n ) space.
Since π ↑ merely permutes the entries of a datum, it does not change the space required to store the input data.
Throughout we will maintain the same notation as before, where we are working with a subgroup of S n which acts by permuting the coordinates of R n indexed by N = { 1 , . .
. , n } .
As initial data we will assume we have a subgroup G of S n given by a generating set of permutations.
Moreover, we will assume that these permutations are given in cycle or one-line notation, so that each can be stored in O ( n log n ) space, and multiplying two permutations together can be performed in O ( n ) time.
Similarly, given x ∈ R n and a permutation g , the point g · x can be computed in O ( n ) time.
F.2.1. Computing initial data
We will make use of the method of representing permutation groups introduced by Jerrum in (23) which we summarise.
First we will explain the notations.
Write N = { 1 , · · · , n } , we will work with directed graphs (ie graphs whose edges have an orientation) of the form ( N, E ) , which have N as their vertex set and E as their edge set.
These graphs will contain no edge loops, and at most one edge joining any pair of vertices, and so we can write lm to denote an edge which starts at l and ends at m .
A (directed) path in ( N, E ) is a sequence of vertices l 0 l 1 · · · l m such that l j l j +1 ∈ E for each j , which is said to have length m (cid:62) 0 .
A directed graph is called a branching if it contains no paths of length m (cid:62) 1 with the same start and end points and each vertex has at most one incoming edge.
If ( N, E ) is a directed graph, an edge labelling is a map σ : E → S n : b l b m (cid:55)→ σ lm which assigns to each edge a permutation of N .
This labelling extends to a labelling of paths by setting σ P = σ b l 0 b l 1 · · · σ l m − 1 l m ∈ S n , where P = l 0 l 1 · · · l m .
We can now give a Theorem-Deﬁnition of the representation of a permutation group, see Theorem 3.3 and Section 4 of (23).
Theorem 27 Let G (cid:54) S n be a permutation group given by a set of generators, then there is an algorithm which yields a small base B = ( b 1 , . . .
, b k ) for G (see subsection F.2.
for a quantitative discussion of what it means for a base to be small).
Extend B to a ﬁxed ordering ˆ B = ( b 1 , · · · , b k , b k +1 , . . . , b n ) on the set N .
As before let G 0 = G , and deﬁne G i
=
Stab G i − 1 ( b i ) for 1 (cid:54) i (cid:54) k .
Then there exists an edge labelled directed graph Υ( G ) =
( N, E, σ ) satisfying the following properties:
Υ is a branching 2.
For all b l b m ∈ E (a) l < m and l (cid:54) k (b) σ b l b m ∈ G l −(c) b l · σ b l b m =
b m 3.
The set U i := { σ P | P is a path in Υ starting at b i } is a right transversal for G
i in G i − 1 for each 1 (cid:54) i (cid:54) k .
Then Υ( G ) will be called a Jerrum representation of G .
Given a generating set for G , a
Jerrum representation can be computed alongside B in O ( k 2 n 3 ) time and O ( n 2 log n ) space.
This algorithm also computes the orbits ∆ i
=
b i · G i − 1 for 1 (cid:54) i (cid:54) k .
Remark
The algorithm presented in (23) in fact assumes that (1 , 2 , · · · , n ) has been chosen a priori to be the base, but it is straightforward to amend the algorithm to compute a more eﬃcient base using the greedy algorithm mentioned at the start of Appendix D.3, compare with (3).
F.2.2. Applying π ↑ to input data
Fix a permutation group G and let Υ
= Υ( G ) be a Jerrum representation for G .
First we will prove a useful characterisation of the orbit ∆ i +1 .
Lemma
The orbit ∆ i is the set of b l ∈ N such that there exists a path P in Υ which starts at b i and ends at b l .
Proof
Note that by induction on m , for any path P =
b l 0 · · · b l m , (2) in Theoremgeneralises to say
(a) l 0 < l m and l m − 1 (cid:54) k
(b) σ P ∈ G l 0 −(c) b l 0 · σ P =
b l m .
If P starts at b i , then σ P ∈ G i − 1 and b i · σ P = b l so b l ∈ ∆ i .
Conversely, if b l ∈ ∆ i , there is some g ∈ G i − 1 such that b
i · g = b l .
Note that the cosets of G i in G i − 1 are exactly the sets of the form { g ∈ G i − 1 |
b
i · g = b } for ﬁxed b ∈ ∆ i .
Indeed g, g (cid:48)
∈ G i − 1 are in the same coset iﬀ g (cid:48)
g − 1 ∈ G i iﬀ b i · ( g (cid:48)
g − 1 )
=
b i iﬀ b i · g (cid:48)
=
b i · g .
Since U i is a complete set of representatives, it contains an element from every coset, and hence an element which maps b i to b l .
Call this element u b l , then by deﬁnition there is some path P which starts at b i such that u b l = σ P , and by the observation above, the end point of P must be b l .
With this Lemma we can give an algorithm to perform the main task in computing π ↑ , computing φ ↑ as a product of permutations g i ∈
G i − 1 , see Section 2.3.
Proposition 30 Given x
(cid:48) ∈ R n , all of whose entries are distinct, and a Jerrum representation Υ
=
Υ
( G ) for G , there is an algorithm to compute φ ↑ ( x (cid:48) ) in O ( k 2 n 2 ) time and
O ( n 2 log n ) space.
Data: A point x (cid:48) ∈ R n , a Jerrum representation Υ = Υ
( G ) , and the orbits ∆ i .
Result: φ ↑ ( x (cid:48) ) .
1 for 1 (cid:54) i (cid:54)
k do //
Loop runs k times
Set j to be the index in ∆ i such that x (cid:48)
j (cid:54) x (cid:48) l for all l ∈ ∆ i ; //
This will be the current working vertex in Υ
Set g i =
e ; // This will accumulate edge labels from Υ while j (cid:54) =
b i do //
Loop runs at most | ∆ i | times
Set l to be the unique index in ∆ i such that lj is an edge of Υ
Set g i =
σ lj
g i Set j = l end
Set x (cid:48) =
g i · x (cid:48) 9 end 10 Set φ ↑ ( x (cid:48) )
=
g k · · · g
Algorithm 1: Computing φ ↑ .
Proof The algorithm we will use is Algorithm 1.
Recall the deﬁnition of φ ↑ ( x (cid:48) ) .
Assume g 1 , . .
, g i − 1 have already been found, G i − 1 acts transitively on ∆ i , choose j ∈ ∆ i such that the j th entry of ( g i − 1 · · · g 1 ) · x (cid:48) is minimal among those entries indexed by ∆ i .
Choose g
i ∈
G i − 1 such that j · g i = g − 1 i ( j ) =
b i .
Then deﬁne φ ↑ ( x (cid:48) ) :
= g k · · · g 1 .
The job of ﬁnding each g
i is made easy by Theorem 29, since we just need to ﬁnd a path P in Υ joining b i to j , which is guaranteed to exist.
Since Υ is branching, each vertex has at most one incoming edge, and hence starting from j and working backwards we are guaranteed to reach b i .
Making note of each edge label as we construct P , we can choose g i =
σ P .
This is achieved by the loop starting in Line 3.
We consider the time complexity of this algorithm.
Following (23), Υ can be represented by an n × n array, whose pq -entry is NULL if pq is not an edge of Υ , and σ pq otherwise.
Finding l in Line 4 requires searching the j th column of this array for the unique non- NULL entry, whose index row index will be l , and so takes O ( n ) steps.
As mentioned above Linealso takes O ( n ) steps, so the while loop at Line 3 takes O ( n | ∆ i | ) steps.
Finding j in Line 1 requires O ( | ∆ i | ) steps, searching through each entry of x (cid:48) indexed by ∆ i and comparing it with the current minimal entry found; while Line 7 takes O ( n ) steps.
Thus the dominant step in the main for loop is the while loop.
Overall then this for loop takes O ( kn (cid:80) ki =1 | ∆ i | ) steps, which is greater than the O ( kn ) steps to compute Line 9.
Noting that | ∆ i | (cid:54) n , this algorithm runs in O ( k 2 n 2 ) time.
As for space, Υ is an n 2 array, containing at most n − 1 non-
NULL entries (since the graph is has no cycles, its number of edges is bounded by n − 1 ), each of which takes O ( n log n ) space to store.
An eﬃcient encoding can then use O ( n 2 log n ) space.
The other signiﬁcant space cost is storing the set of g
i ’s, which takes O ( kn log n ) which is at most O ( n 2 log n ) .
Theorem 31 Given x ∈ R n , a perturbation vector ε , and a Jerrum representation Υ for G , there is an algorithm to compute π ↑ ( x ) in O ( k 2 n 2 ) time and O ( n 2 log n ) space.
Data: A point x ∈ R n , a perturbation vector ε , a Jerrum representation Υ = Υ
( G ) , and the orbits ∆ i .
Result: π ↑ ( x ) .
1 if x ∈ R 1 then
Set d =3 else
Set d = min {| x i − x j | | x i (cid:54) =
x j } 5 end 6 Set x (cid:48)
= x + dε 7 Compute φ ↑ ( Algorithm8 Set π ↑ ( x ) =
φ ↑ ( x (cid:48) ) · x
Proof Algorithm 2 follows exactly the procedure outlined in Section 2.3.
It is clear that
Line 6 dominates in terms of both time and space complexity.
F.2.3. Analysing time complexity
Naïvely one may assume that the size k of the base B is O ( n ) , indeed this is the case for
G = S n or A n for example, but in practice we can do a lot better.
Write b ( G ) for the size of the smallest base for G , then in (3) Blaha showed that the greedy algorithm used in
Theorem 27 to ﬁnd a base will not necessarily yield a minimal base, but nearly—in particular k will be O ( b ( G ) log log n ) .
When looking at permutation groups, it is natural to focus on the case of so-called primitive permutation groups, ie subgroups G (cid:54) S n which act transitively on N = { 1 , . .
. , n } such that there are no non-trivial G -invariant partitions.
This is because arbitrary permutation groups can be built up out of primitive ones.
In this setting Liebeck proved the following in (27).
Theorem 32 Let G be primitive, and not S n or A n , then there is some absolute constant c such that b ( G ) < c √ n .
It follows that in this case, the base found above will have size O ( √ n log log n ) .
Combining this with the observation in Appendix C that for S n and A n , π ↑ can be computed using a sorting algorithm, we get the following.
Theorem 33 Let G (cid:54) S n be primitive, then either
• G = S n or A n : no initial data needs to be computed and π ↑ can be computed in O ( n 2 ) time per datum (using worst case for quicksort); or • Initial data can be computed in O ( n 4 (log log n ) 2 ) time, and π ↑ can be computed in O ( n 3 (log log n ) 2 ) time per datum.
F.3. Computing Dirichlet projections
We focus on the case G acts discretely on R n by orthogonal matrices.
Choose a point x 0 ∈ R n which is only ﬁxed by elements of G which ﬁx the whole of R n point-wise.
The map φ from Section 2.2 maps x ∈ R n to the element in G which minimises the Euclidean distance d
( g · x, x 0 ) .
In this case, the inner product (cid:104)· , ·(cid:105) is invariant.
It is eﬃcient to compute (cid:104)· , ·(cid:105) , which varies inversely with the Euclidean distance d between points, so we can perform gradient descent to minimize
[Equation]
Here the second equality comes from applying the cosine rule and the geometric deﬁnition of the inner product.
Our main application of the discrete gradient descent algorithm is for CICY matrices when G = S 12 × S 15 .
Since | G | ≈ 6 × 10 20 one cannot minimise a function over G by simply evaluating it at all elements of G .
F.3.1. Approximations using gradient descent
It is natural to compute the minimiser of N on a group orbit using gradient descent.
The steps in the descent are restricted to the discrete G -orbit of the input point x , so we must deﬁne what a gradient is in this case.
A generating set T for G is a subset such that any element g ∈ G can be written as a ﬁnite product of elements in T , ie there are t 1 , . . .
, t l ∈ T such that g = t 1 · · · t l .
Two points x, x (cid:48) ∈ R n are adjacent with respect to T if there is t ∈ T such that x (cid:48)
=
t · x , so in particular, adjacent points are in the same G -orbit.
Deﬁnition
Given an action of a ﬁnite group G on R n , a generating set T of G , a function N : R n → R and x ∈ R n , the discrete gradient descent is an approximation for
[Equation]
and is deﬁned iteratively as follows.
Let x 0 = x .
Given x i , deﬁne
[Equation]
The output of the algorithm is x i when x i +
= x i .
Since we can get between any two points in G by a ﬁnite sequence of steps by generators this algorithm always terminates.
In general, there are many choices for a generating set T resulting in diﬀerent approximations for φ .
For G = S n a natural choice for a generating set is given by T = { (1 2) , (2 3) , . . .
, ( n − 1 n ) } .
In particular, one has in this case | T | = n − 1 (cid:28) n ! =
| S n | .
By taking the union of these generating sets for S n and S m one obtains a generating set of size n + m − 2 for S n × S m .
Choosing a larger generating set increases the computational cost of the algorithm but potentially also its accuracy.
For example, consider the set T (cid:48)
= { tt (cid:48) | t, t (cid:48) ∈ T ∪ { e }} .
This is a generating set for S n and again yields a generating set for S n × S m in a similar way.
When applied to the CICY dataset, we found that choosing T (cid:48) instead of T leads to a signiﬁcant increase in computation cost, but not so in accuracy.
Instead, we have used discrete gradient with diﬀerent seeds.
For a 12 × 15 CICY matrix x , The seeds are x km :
=
C 12 k xC 15 m , where C i are cyclic permutation matrices 1 ≤ k ≤ 12 and 1 ≤ m ≤ 15 .
To each x km , apply the discrete gradient descent algorithm above and pick the minimum of all seeds.
The result is still a permutation of x since C i are permutation matrices.
This increases the computation cost by a constant factor 11 × 14 + 1 = 155 but has led to a signiﬁcant accuracy boost.
We are unable to give a bound for the number of generators applied to an input until a local minimum is reached.
Experiments on the CICY dataset show that this number is very low compared to the size of the group.
On the original CICY dataset the average number of iterations is ≈ 17 . 4 , with standard deviation ≈ 15 .
6 and maximum 163 .
On an augmented dataset, which contains 10 permutations of each matrix, the average number of iterations is ≈ 22 . 5 , with standard deviation ≈ 16 .
9 and maximum 198 .
F.3.2. The existence of an exact algorithm
Computing a Dirichlet projection can be hard and we do not expect that in general there is a polynomial time algorithm to compute it.
If such an algorithm exists, this would imply a solution to the graph isomorphism problem in polynomial time, and it is currently unknown whether such an algorithm exists.
To see this, let n ∈ N and Γ 1 , Γ 2 be graphs with adjacency matrices
A 1 , A 2 ∈ { 0 , 1 } n × n .
The symmetric group S n acts on the graphs by relabelling the vertices, which can be viewed as an action on { 0 , 1 } n × n .
Let
[Equation]
and denote the corresponding Dirichlet projection by π : { 0 , 1 } n × n → { 0 , 1 } n × n .
Because of our choice of x 0 , for all A ∈ { 0 , 1 } n × n there exists a unique g ∈ S n that minimises (cid:104) g
( A ) , x 0 (cid:105) .
The element g can be characterised as the permutation that makes the matrix A as small as possible, when read as a binary number row by row.
The graphs Γ 1 and Γ 2 are isomorphic if and only if A 1 and A 2 are in the same orbit under the S n action, which is the case if and only if π ( A 1 )
=
π ( A 2 ) .
Therefore, if one is able to compute π ( A i ) for i ∈ { 1 , 2 } in polynomial time, one can decide if Γ 1 and Γ 2 are isomorphic in polynomial time.