L EARNING I RREDUCIBLE R EPRESENTATIONS OF N ONCOMMUTATIVE L IE G ROUPS
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent work has constructed neural networks that are equivariant to continuous symmetry groups such as 2D and 3D rotations.
This is accomplished using explicit group representations to derive the equivariant kernels and nonlinearities.
We present two contributions motivated by frontier applications of equivariance beyond rotations and translations.
First, we relax the requirement for explicit Lie group representations, presenting a novel algorithm that ﬁnds irreducible representations of noncommutative Lie groups given only the structure constants of the associated
Lie algebra.
Second, we demonstrate that Lorentz-equivariance is a useful prior for object-tracking tasks and construct the ﬁrst object-tracking model equivariant to the Poincar ´ e group
I NTRODUCTION
Many tasks in machine learning exactly or approximately obey a continuous symmetry such as 2D rotations.
An ML model is said to be equivariant to such a symmetry if the model respects it automatically (without training).
Equivariant models have been applied to tasks ranging from computer vision to molecular chemistry, leading to a generalization of equivariance techniques beyond 2D rotations to other symmetries such as 3D rotations.
This is enabled by known mathematical results about each new set of symmetries.
Speciﬁcally, explicit group representation matrices for each new symmetry group are required.
For many important symmetries, formulae are readily available to produce these representations.
For other symmetries we are not so lucky, and the representations may be difﬁcult to ﬁnd explicitly.
In the worst cases, the classiﬁcation of the group representations is an open problem in mathematics.
For example, in the important case of the homogeneous Galilean group , which we deﬁne in section 2, the classiﬁcation of the ﬁnite dimensional representations is a so-called “wild algebraic problem” for which we have only partial solutions (De Montigny et al., 2006; Niederle & Nikitin, 2006; Levy-Leblond, 1971).
To construct an equivariant network without prior knowledge of the group representations, novel approaches are needed.
In this work, we propose an algorithm LearnRep that ﬁnds the representation matrices with high precision.
We validate that LearnRep succeeds for the Poincar´e group , a set of symmetries governing phenomena from particle physics to object tracking.
We further validate
LearnRep on two additional sets of symmetries where formulae are known.
We apply the Poincar´e group representations obtained by LearnRep to construct SpacetimeNet , a Poincar´e-equivariant object-tracking model.
As far as we are aware, LearnRep is the ﬁrst automated solver which can ﬁnd explicit representation matrices for sets of symmetries which form noncompact, noncommutative Lie groups Further, SpacetimeNet is the ﬁrst object-tracking model with a rigorous guarantee of Poincar´e group equivariance.
1. G ROUP R EPRESENTATIONS AND E QUIVARIANT M ACHINE L EARNING
Group theory provides the mathematical framework for describing symmetries and building equivariant ML models.
Informally, a symmetry group G is a set of invertible transformations α, β ∈ G which can be composed together using a product operation αβ .
We are interested in continuous symmetries for which G is a Lie group .
In prior constructions of Lie group-equivariant models, group representations are required.
For a group G , an n − dimensional (real) group representation ρ : G → R n × n is a mapping from each element α ∈ G to an n × n -dimensional matrix ρ ( α ) , such that for any two elements α, β ∈ G , we have ρ ( α ) ρ ( β ) =
ρ ( αβ ) .
Two parallel techniques have been developed for implementing Lie group equivariant neural networks.
The ﬁrst approach was described in general by Cohen et al.
For the latter approach taken by
Thomas et al.
; Anderson et al.
; Bogatskiy et al.
, convolutions and nonlinearities are performed directly on the irreducible representations of the group, which we deﬁne in section 2.4.
A common thread in these works has been to utilize existing formulas derived for the matrix elements of these irreducible representations.
However, these formulas are only available for speciﬁc Lie groups where the representation theory is well-understood.
A more convenient approach for extending equivariance to novel Lie groups would utilize an automated computational technique to obtain the required representations.
The primary contribution of this work is such a technique.
1. C ONTRIBUTIONS
In this work, we automate the generation of explicit group representation matrices of Lie groups using an algorithm called LearnRep .
LearnRep poses an optimization problem deﬁned by the Lie algebra associated with a Lie group, whose solutions are the representations of the algebra.
A penalty term is used to prevent the formation of trivial representations .
Gradient descent of the resulting loss function produces nontrivial representations upon convergence.
We apply LearnRep to three noncommutative
Lie groups for which the ﬁnite-dimensional representations are well-understood, allowing us to verify that the representations produced are irreducible by computing their Clebsch-Gordan coefﬁcients and applying Schur’s Lemma .
One of the Lie groups where LearnRep performs well is the Lorentz group of special relativity.
Prior work has applied Lorentz-equivariant models to particle physics.
In this work we explain that the
Lorentz group along with the larger Poincar´e group also governs everyday object-tracking tasks.
We construct a Poincar´e-equivariant neural network architecture called SpacetimeNet and demonstrate that it can learn to solve a 3D object-tracking task subject to “motion equivariance,” where the inputs are a time series of points in space.
In summary, our contributions are:
• LearnRep , an algorithm which can ﬁnd irreducible representations of a noncompact and noncommutative Lie group.
• SpacetimeNet , a Poincar ´ e group-equivariant neural network applied to object-tracking tasks.
Our work contributes towards a general framework and toolset for building neural networks equivariant to novel Lie groups, and motivates further study of Lorentz equivariance for object tracking.
1. O RGANIZATION
We summarize all necessary background and terminology in section 2.
We describe the LearnRep algorithm in section 3.1 and SpacetimeNet in section 3.2.
We summarize related work in section 4.
We present our experimental results in section 5: our experiments in learning irreducible Lie group representations with LearnRep in section 5.1 and the performance of our Poincar´e-equivariant
SpacetimeNet model on a 3D object tracking task in section 5.2.
T ECHNICAL B ACKGROUND
We explain the most crucial concepts here and defer to Appendix A.1 for a derivation of the representation theory of the Lorentz group.
2. S YMMETRY G ROUPS SO( n ) AND SO( m, n )
A 3D rotation may be deﬁned as a matrix A : ∈ R 3 × 3 which satisﬁes the following properties, in which (cid:104) (cid:126)u,(cid:126)v (cid:105) =
(cid:80) 3 i =1 u i v i :
[Equation]
; these imply the set of 3D rotations forms a group under matrix multiplication and this group is denoted SO(3) .
This deﬁnition directly generalizes to the n − dimensional rotation group SO( n ) .
For n ≥ 3 , the group SO( n ) is noncommutative, meaning there are elements A, B ∈ SO( n ) such that
AB (cid:54) = BA .
Allowing for rotations and translations of n dimensional space gives the n − dimensional special Euclidean group SE( n ) .
SO( n ) is generalized by a family of groups denoted SO( m, n ) , with
SO( n ) =
SO( n, 0) .
For integers m, n ≥ 0 , we deﬁne (cid:104) (cid:126)u,(cid:126)v (cid:105) m,n = (cid:80) mi =1 u i v
i − (cid:80) m + n i
= m
+1 u
i v i .
The group SO( m, n ) is the set of matrices A ∈ R ( m + n ) × ( m + n ) satisfying (i-ii) below:
[Equation]
these imply that SO( m, n ) is also a group under matrix multiplication.
While the matrices in
SO( n ) can be seen to form a compact manifold for any n , the elements of SO( m, n ) form a noncompact manifold whenever n, m ≥ 1 .
For this reason SO( n ) and SO( m, n ) are called compact and noncompact Lie groups respectively.
The representations of compact Lie groups are fairly well understood, see Bump (2004);
Cartan (1930).
2. A CTION OF SO( m, n ) ON S PACETIME
We now explain the physical relevance of the groups SO( m, n ) by reviewing spacetime .
We refer to Feynman et al.
(2011) (ch. 15) for a pedagogical overview.
Two observers who are moving at different velocities will in general disagree on the coordinates { ( t i , (cid:126)u i ) }
⊂ R 4 of some events in spacetime.
Newton and Galileo proposed that they could reconcile their coordinates by applying a spatial rotation and translation (i.e., an element of SE(3) ), a temporal translation (synchronizing their clocks), and ﬁnally applying a transformation of the following form:
[Equation]
in which (cid:126)v is the relative velocity of the observers.
The transformation equation 1 is called a Galilean boost .
The set of all Galilean boosts along with 3D rotations forms the homogeneous Galilean group denoted HG (1 , 3) .
Einstein argued that equation 1 must be corrected by adding terms dependent on || (cid:126)v || 2 /c , in which c is the speed of light and || (cid:126)v || 2 is the (cid:96) 2 norm of (cid:126)v .
The resulting coordinate transformation is called a Lorentz boost , and an example of its effect is shown in ﬁgure 1.
The set of 3D rotations along with Lorentz boosts is exactly the group SO(3 , 1) .
In the case of 2 spatial dimensions, the group is SO(2 , 1) .
Including spacetime translations along with the Lorentz group
SO( n, 1) gives the larger Poincar ´
e group P n with n spatial dimensions.
The Poincar ´ e group P 3 is the group of coordinate transformations between different observers in special relativity.
Consider an object tracking task with input data consisting of a spacetime point cloud with n dimensions of space and 1 of time, and corresponding outputs consisting of object class along with location and velocity vectors.
A perfectly accurate object tracking model must respect the action of P n on the input.
That is, given the spacetime points in any observer’s coordinate system, the perfect model must give the correct outputs in that coordinate system .
Therefore the model should be
P n -equivariant.
For low velocities the symmetries of the homogeneous Galilean groups HG ( n, 1) provide a good approximation to SO( n, 1) symmetries, so Galilean-equivariance may be sufﬁcient for some tasks.
Unfortunately the representations of HG ( n, 1) are not entirely understood De Montigny et al.
; Niederle & Nikitin (2006); Levy-Leblond (1971).
2. L IE G ROUPS AND L IE A LGEBRAS
Here we give an intuitive summary of Lie groups and Lie algebras, deferring to Bump (2004) for a rigorous technical background.
A Lie group G gives rise to a Lie algebra A as its tangent space at the identity.
This is a vector space V along with a bilinear product called the Lie bracket : [ a, b ] which must behave like 1 the commutator for an associative ring R with multiplication operation × R :
[Equation]
The Lie algebra for SO(3) , denoted so (3) , has a basis { J 1 , J 2 , J 3 } satisfying
[Equation]
in which (cid:15) ijk ∈ {± 1 , 0 } is the totally antisymmetric Levi-Civita symbol.
2 Intuitively, the Lie bracket shows how group elements near the identity fail to commute.
For example, the matrices R x , R y , R z 1 Speciﬁcally, the Lie bracket must satisfy the Jacobi identity and [ a, a ] = 0 .
The symbol (cid:15) ijk simply expresses in equation 2 that [ J 1 , J 2 ]
=
J 3 , [ J 2 , J 3 ]
=
J 1 , [ J 3 , J 1 ]
=
J 2 .
[Figure]
for rotations about the x and y axes by a small angle θ satisfy R x R y − R y R x = R z
+ O ( θ 2 ); more generally the Lie bracket of equation 2 is satisﬁed to ﬁrst order in θ .
The Lia algebra so (3 , 1) of the Lorentz Group SO(3 , 1) also satisﬁes equation 2 for the generators J 1 , J 2 , J 3 of its subalgebra isomorphic to so (3) .
It has 3 additional generators denoted K 1 , K 2 , K 3 , which satisfy:
[Equation]
These K i correspond to the Lorentz boosts in the same way that the J i correspond to the rotations.
In general, if A is a t -dimensional Lie algebra with generators T 1 , ..., T t such that
[Equation]
we call the tensor A ijk the structure constants of A .
For connected matrix Lie groups such as
SO( m, n ) , the structure constants A ijk are easily obtained.
For example, one may apply the matrix logarithm to several elements of the group to obtain elements of the algebra, then ﬁnd a complete basis for the algebra and write the commutator of all basis elements in this basis.
2. G ROUP R EPRESENTATIONS AND THE T ENSOR P RODUCT
Let G be a Lie group and ρ : G → R n × n be a representation of G as deﬁned in section 1.1.
Then ρ deﬁnes a group action on R n : given a vector (cid:126)u ∈ R n and a group element α ∈ G , we can deﬁne
[Equation]
using the matrix product.
We then say that ρ is irreducible if it leaves no nontrivial subspace invariant – for every subspace V ⊂ R n with 0 < dim V < n , there exists α ∈ G,(cid:126)v ∈ V such that α (cid:63) ρ (cid:126)v / ∈ V .
Given two G -representations ρ 1 : G → R n 1 × n 1 , ρ 2 : G → R n 2 × n 2 , we deﬁne their tensor product as
[Equation]
in which ⊗ on the right hand side denotes the usual tensor product of matrices.
It is easy to check that ρ 1 ⊗ ρ 2 is also a representation of G using the fact that for matrices A 1 , A 2 ∈ R n 1 × n 1 and
B 1 , B 2 ∈ R n 2 × n 2 ,
[Equation]
⊗ ( B 1 B 2 ) .
For ρ 1 , ρ 2 as above we also deﬁne their direct sum as
[Equation]
For two groups H, G we say that H is isomorphic to G and write H ∼ =
G if there exists a bijection f : H → G such that f ( αβ ) =
f ( α ) f
( β ) .
For ρ 1 , ρ 2 as above, their images ρ i ( G ) form groups
and we say that ρ 1 and ρ 2 are isomorphic and write ρ 1 ∼ = ρ 2 if these groups are isomorphic, i.e. ρ 1 ( G ) ∼ = ρ 2 ( G ) .
Some familiar representations of SO(3) act on scalars ∈ R , vectors ∈ R 3 , and tensors (e.g., the Cauchy stress tensor) – these representations are all nonisomorphic.
For many Lie groups such as SO( n, 1) and SO( n ) , a property called complete reducibility guarantees that any representation is either irreducible, or isomorphic to a direct sum of irreducible representations.
For such groups it sufﬁces to identify the irreducible representations to understand all other representations and construct equivariant models.
2. C LEBSCH -G ORDAN C OEFFICIENTS AND T ENSOR -P RODUCT N ONLINEARITIES
Clebsch-Gordan Coefﬁcients: Let G be a completely reducible Lie group, and let ρ 1 , ρ 2 , ρ 3 be irreducible G -representations on the vector spaces R n 1 , R n 2 , R n 3 .
Consider the tensor product representation ρ 1 ⊗ ρ 2 .
Since G is completely reducible, there exists a set S of irreducible representations such that ρ 1 ⊗ ρ 2 ∼
= (cid:76) ρ ∈ S ρ .
Suppose that ρ 3 ∈ S .
Then there exists a matrix C ∈ R n 3 × ( n 1 n 2 ) which projects the space of the n 3 -dimensional group representation ρ 3 from the tensor product space R n 1 ⊗ R n 2 .
That is,
[Equation]
The matrices C satisfying equation 5 for various ρ 3 are called the Clebsch-Gordan coefﬁcients .
In equation 5 there are n 1 n 2 n 3 linear constraints on C, and therefore this is a well-posed homogeneous linear program (LP) for C .
The entries of C may be found numerically by sampling several distinct α ∈ G and concatenating the linear constraints (equation 5) to form the ﬁnal LP.
The solutions for C form a linear subspace of R n 3 × ( n 1 n 2 ) given by the nullspace of some matrix we denote C [ ρ 1 , ρ 2 , ρ 3 ] .
Tensor Product Nonlinearities: Tensor product nonlinearities, including norm nonlinearities, use the Clebsch-Gordan coefﬁcients deﬁned above to compute equivariant quadratic functions of multiple
G -representations within the G -equivariant model.
This was demonstrated for the case of SE(3) by
Thomas et al.
; Kondor et al.
(2018) and for SO(3 , 1) by Bogatskiy et al.
M ETHODS 3. L EARNING L IE G ROUP R EPRESENTATIONS
For a matrix M ∈ R n × n we denote its Frobenius and L 1 norms by
[Equation]
The approach of LearnRep is to ﬁrst learn a Lie algebra representation and then obtain its corresponding group representation through the matrix exponential.
Fix a t -dimensional Lie algebra A with structure constants A ijk as deﬁned in equation 4.
Fix a positive integer n as the dimension of the representation of A .
Then let the matrices T 1 , ..., T t ∈ R n × n be optimization variables, and deﬁne the following loss function on the T i :
[Equation]
This is the magnitude of violation of the structure constants of A , multiplied by a norm penalty term
N [ T i ] − 1 (this penalty is plotted separately in ﬁgure 2).
The purpose of the norm penalty is to avoid convergence to a solution where T i = 0 n × n for any i , which will act trivially when restricted to the nontrivial subgroup { e tT
i : t ∈ R } .
We pose the optimization problem:
[Equation]
The generators were initialized with entries from the standard normal distribution.
Gradient descent was performed in PyTorch with the Adam optimizer (Kingma & Ba, 2014) with initial learning rate 0 .
The learning rate decreased exponentially when loss plateaued.
The results are shown in ﬁgure 2.
3.1. V ERIFYING I RREDUCIBILITY OF L EARNED R EPRESENTATIONS
Suppose we have converged to T 1 , . . .
T t such that L [ T i ] = 0 .
Then the T 1 , ..., T t are a nonzero n -dimensional representation of the Lie algebra A .
The groups considered here are covered by the exponential map applied to their Lie algebras, so for each α ∈ G there exist b 1 , . . .
, b t ∈ R such that
[Equation]
where ρ is any n − dimensional representation of G and exp is the matrix exponential.
This ρ : G (cid:55)→ R n × n is then a representation of the Lie group .
Throughout this section, ρ denotes this representation.
In general ρ may leave some nontrivial subspace invariant.
In this case it is reducible and splits as the direct sum of lower-dimensional irreducible representations ρ i as explained in 2.4:
[Equation]
Recall that any representation may be obtained as such a direct sum of irreducible representations with dimensions n 1 , . . . , n (cid:96) satisfying n = (cid:80) (cid:96)i =1 n i .
If n is set to the minimum dimension of a nontrivial irreducible representation, the only permissible partitions of n have (cid:96) = 1 and (cid:96) = n – as the latter representation is trivial, equation 6 diverges, so LearnRep can only converge to an irreducible n dimensional representation.
It is important to verify that the learned ρ is indeed irreducible with (cid:96) = 1 .
To validate that ρ is irreducible, LearnRep computes its tensor product structure and compares with the expected structure.
Speciﬁcally, it computes the Clebsch-Gordan coefﬁcients for the direct-sum decomposition of the tensor product of the learned representation ρ with several other known representations ρ 1 , ..., ρ r .
section 2.5 deﬁnes these coefﬁcients and explains how they are computed from the nullspace of the matrix C
= C
[ ρ, ρ 1 , ρ 2 ] , in which ρ 2 appears in the decomposition of ρ ⊗ ρ 1 .
Let ρ 1 , ρ 2 denote two other known representations, and consider the Clebsch-Gordan coefﬁcients C such that Cρ ⊗ ρ 1 = ρ 3 C .
The dimension of the nullspace of C indicates the number of unique nonzero matrices C of Clebsch-Gordan coefﬁcients.
The singular values of C are denoted SV 1 ( C ) ≤ ... ≤ SV (cid:96) ( C ) .
The ratio
[Equation]
diverges only if the nullspace is one dimensional which therefore corresponds to a unique solution for
C .
The number of expected solutions is known (e.g., it may be computed using the same technique from the formulae for the irreducible representations).
Therefore if r ( C ) diverges for exactly the choices of ρ 1 , ρ 2 where the theory indicates that unique nonzero Clebsch-Gordan coefﬁcients exist, then this is consistent with our having learned an irreducible representation of the group G .
Clearly the tensor product with the trivial representation ρ 1 = 1 is ρ ⊗ 1 = ρ .
In this case, the permissible C correspond to G − linear maps R n → R n 2 .
By a result of Schur (1905) (Schur’s
Lemma), the only such (nonzero) maps are isomorphisms.
Therefore a divergent value of r ( C ) when ρ 1 = 1 indicates that ρ ∼ = ρ 2 .
This is shown in the top row of ﬁgure 3 and discussed further in section 5.1.
3.1. S TOPPING CONDITION
Similar to (Rao & Ruderman, 1999), LearnRep restarts gradient descent several times starting from random initialization points.
A restart is triggered if loss plateaus and the learning rate is smaller than the loss by a factor of at most 10 − 4 .
The tensor product structure is computed upon convergence to loss under 10 − 9 , a restart is triggered if the divergences of r ( C ) do not agree with the theoretical prediction, indicating a reducible representation.
3. S PACETIME N ET A RCHITECTURE
We obtain all Clebsch-Gordan coefﬁcients through the procedure explained in section 2.5.
We place them in a tensor: C g,qr,ls,mt .
This notation corresponds to taking the tensor product of an element of the l th group representation space indexed by s with an element of the m th group representation space indexed by t , and projecting it onto the q th group representation space indexed by r . The space This experiments
SO(3) n = 3 . of possible Clebsch-Gordan coefﬁcients can be multidimensional.
4 We use an index g to carry the dimension within the space of Clebsch-Gordan coefﬁcients.
The trainable weights in SpacetimeNet are complex-valued ﬁlter weights denoted f kqg and channelmixing weights denoted W kqcgd .
Each layer builds a collection of equivariant convolutional ﬁlters
F kxijqr from the geometry of the point cloud.
Let q (cid:48) denote the index of the group representation in which the points are embedded.
Let X xir denote the point coordinates, in which x indexes the batch dimension, i indexes the points, and r indexes the q (cid:48) group representation space.
Deﬁne the (globally) translation-invariant quantity ∆ X xijr := X xjr − X xir .
The equivariant ﬁlters at layer k are:
[Equation]
The forward pass consists of tensor product nonlinearities between equivariant ﬁlters and activations.
The input and activations for the k th layer of the network are deﬁned on a tensor V kximct , where x is the batch dimension, i indexes the points, m is the group representation index, c is the channel index, t indexes the group representation space.
Our mixing weights are then deﬁned for the k th layer as
W kqcgd with layer update rule:
[Equation]
A proof that SpacetimeNet is P n -equivariant is given in Appendix A.2.
R ELATED W ORK 4. L EARNING L IE G ROUP R EPRESENTATIONS
Several authors have investigated automated means of identifying Lie group representations.
(Rao &
Ruderman, 1999) used gradient descent with several starting points to ﬁnd the Lie group generators, given many examples of data which had been transformed by the group.
Applying the technique requires knowledge of how the group acts on a representation space.
Here we know the Lie algebra structure but we do not know how to compute its representations.
Tai et al.
(2019) gave a closed-form solution for the canonical coordinates for Lie groups.
But their formula only applies for Abelian oneparameter Lie groups, excluding SO(3) , SO(2 , 1) , and SO(3 , 1) .
Cohen & Welling (2014) devised a probabilistic model to learn representations of compact, commutative Lie groups from pairs of images related by group transformations.
In the present work we demonstrate a new approach to handle noncompact and noncommutative groups such as SO(3) , SO(2 , 1) , and SO(3 , 1) .
Computer algebra software such as the LiE package developed by Van Leeuwen et al.
(1992) automates some calculations related to completely reducible Lie groups.
Unfortunately this limits us when considering novel Lie groups where the representation theory is less well-understood.
4. E QUIVARIANT N EURAL N ETWORKS
Beginning with the success of (approximately) translation-equivariant CNNs introduced by LeCun et al.
(1989) for image recognition, a line of work has extended equivariance to additional continuous symmetry groups.
Most relevant are the architectures for groups SE(2)
(Worrall et al., 2017; Weiler & Cesa, 2019), SE(3) (Weiler et al., 2018;
Cohen et al., 2019; Kondor et al., 2018; Thomas et al., 2018;
Cohen et al., 2018; Kondor, 2018; Gao et al., 2020; Anderson et al., 2019; Fuchs et al., 2020;
Eismann et al., 2020), and the group of Galilean boosts (Zhu et al., 2019).
The work by Thomas et al.
; Kondor et al.
; Anderson et al.
; Bogatskiy et al.
(2020) used Clebsch-Gordan coefﬁcients in their equivariant neural networks.
Weiler et al.
(2018), generalized by Cohen et al.
(2019) showed all equivariant linear maps are convolutions whose kernels satisfy some linear constraints.
In our work we obtain Clebsch-Gordan coefﬁcients from similar linear constraints (equation 5) and use them to show that the learned representations are irreducible.
We also use them in SpacetimeNet.
Grifﬁths & Grifﬁths (2005) provide an introductory exposition of
Clebsch-Gordan coefﬁcients and Gurarie (1992) provides a more general exposition.
One of the ﬁrst constructions that addressed spatiotemporal symmetries was by Zhu et al.
They introduce motion-equivariant networks to handle linear optical ﬂow of an observer moving at a ﬁxed speed.
They use a canonical coordinate system in which optical ﬂow manifests as a translation, as described for general one dimensional Lie groups by Tai et al.
This allows them to use the translation equivariance of CNNs to produce Galilean boost-equivariance.
However, this gives up equivariance to translation in the original coordinate system.
To maintain approximate translation-equivariance, the authors apply a spatial transformer network (Jaderberg et al., 2015) to predict a landmark position in each example.
This is similar to the work of Esteves et al.
(2018), which achieved equivariance to 2D rotation and scale, and approximate equivariance to translation.
The ﬁrst mention of Poincar´e-equivariant networks appears to be a work by Cheng et al.
on the link between covariance in ML and physics.
Concurrently to our work, Bogatskiy et al.
(2020) constructed a Lorentz-equivariant model which operated on irreducible representations of the
Lorentz group, derived similarly to Appendix A.1.
This work also made use of the Clebsch-Gordan coefﬁcients, and the model was applied to experimental particle physics rather than object-tracking.
Another work by Finzi et al.
(2020) concurrent to our own proposed a framework for building models equivariant to arbitrary Lie groups.
This work also made use of the exponential and logarithm maps between Lie algebra and group.
It does not provide a technique for identifying the Lie algebra representations.
Our ideas complement this line of work by providing an algorithm (LearnRep) that solves for the representations numerically.
E XPERIMENTS 5. C ONVERGENCE OF L EARN R EP TO I RREDUCIBLE R EPRESENTATIONS
We apply LearnRep to SO(3) , SO(2 , 1) , and SO(3 , 1) to learn 3 , 3 , and 4 dimensional irreducible representations respectively.
The loss function converges arbitrarily close to 0 with the penalty term bounded above by a constant.
We exponentiate the resulting algebra representation matrices to obtain group representations and calculate the tensor product structure as described in section 3.1.
The details of this calculation are in Appendix A.4 and shown in ﬁgure 3.
The results indicate that the learned representations are irreducible representations of the associated Lie algebras to within numerical error of about 10 − 6 .
Schur’s Lemma in the special case of the tensor product with the trivial representation indicates the isomorphism class of each learned group representation.
5. P OINCAR ´ E -E QUIVARIANT O BJECT -T RACKING N ETWORKS
We created MNIST-Live, a benchmark dataset of spacetime point clouds sampled from digits from the
MNIST dataset moving uniformly through space.
Each sample consists of 64 points with uniformly random times t ∈ [ − 1 / 2 , 1 / 2] , and spatial coordinates sampled from a 2D probability density function proportional to the pixel intensity.
Using instances of the 0 and 9 classes, we train on examples with zero velocity and evaluate on examples with random velocity and orientation.
This dataset is analogous to data from an event camera (see (Orchard et al., 2015)) or LIDAR system.
We train 3 layer SO(2 , 1) and SO(3 , 1) -equivariant
SpacetimeNet models with 3 channels and batch size 16 on 4096 MNIST-Live examples and evaluate on a dev set of 124 examples.
We obtain dev accuracy of 80 ± 5% as shown in ﬁgure 4 of the Appendix.
5. C ONCLUSION
We envision many applications of Poincar´e-equivariant deep neural networks beyond the physics of particles and plasmas.
SpacetimeNet can identify and track simple objects as they move through 3D space.
This suggests that Lorentz-equivariance is a useful prior for object-tracking tasks.
With a treatment of bandlimiting and resampling as in Worrall et al.
; Weiler et al.
(2018), our work could be extended to build Poincar´e-equivariant networks for volumetric data.
More broadly, understanding the representations of noncompact and noncommutative Lie groups may enable the construction of networks equivariant to new sets of symmetries such as the Galilean group.
Since the representation theory of these groups is not entirely understood, automated techniques such as
LearnRep could play a beneﬁcial role.
A A PPENDIX A. A NALYTIC D ERIVATION OF L ORENTZ G ROUP R EPRESENTATIONS
To compare our learned group representations with those obtained through prior methods, we require analytical formulae for the Lie algebra representations for the algebras so (3) , so (3 , 1) , and so (2 , 1) .
The case of so (3) has a well-known solution (see Grifﬁths & Grifﬁths (2005)).
If complex matrices are permissible the library QuTiP Johansson et al.
(2013) has a function “jmat” that readily gives the representation matrices.
A formula to obtain real-valued representation matrices is given in Pinchon & Hoggan (2007) and a software implementation is available at Cohen et al.
The threedimensional Lie algebra so (2 , 1) =
span { K x , K y , J z } has structure constants given by equation 3.
In fact, these three generators K x , K y , J z may be rescaled so that they satisfy equation 2 instead.
This is due to the isomorphism so (3) ∼ = so (2 , 1) .
Speciﬁcally, leting { L x , L y , L z } denote a Lie algebra representation of so (3) , deﬁning
[Equation]
it may be easily checked that K x , K y , J z satisfy the applicable commutation relations from equation 3.
This reﬂects the physical intuition that time behaves like an imaginary dimension of space.
The ﬁnal Lie algebra for which we require explicit representation matrix formulas is so (3 , 1) .
Following Weinberg (1995), we deﬁne new generators
A i , B i as
[Equation]
we see that the so (3 , 1) commutators equation 2, equation 3 become
[Equation]
Therefore so (3 , 1) ∼ = so (3) ⊕ so (3)
[Equation]
and the irreducible algebra representations of so (3 , 1) may be obtained as the direct sum of two irreducible algebra representations of so (3) .
A. P ROOF THAT S PACETIME N ET IS P OINCAR ´ E -E QUIVARIANT
Consider an arbitrary Poincar ´ e group transformation α ∈ P n
[Equation]
and write α = βt in which β ∈
SO( n, 1) and t is a translation.
Suppose we apply this α to the inputs of equation 8 through the representations indexed by q : ρ q ( α )
st
[Equation]
in which s [Equation] t index the representation matrices.
Then since the translation t leaves ∆ X invariant
[Equation]
the resulting ﬁlters will be
[Equation]
∆ X xijr (cid:48) + (cid:88) g,s (cid:48) ,t (cid:48) (cid:32)(cid:88) s,t
C g,
[Equation]
,q (cid:48) s
[Equation]
(cid:48)
t
ρ q (cid:48) ( β ) ss (cid:48)
[Equation]
q (cid:48) ( β ) tt (cid:48) (cid:33) f kqg
[Equation]
X xijs (cid:48)
∆ X xijt (cid:48) = δ [Equation]
(cid:48) (cid:88) r (cid:48)
[Equation]
(cid:48) (
[Equation]
) rr (cid:48)
[Equation]
(cid:48)
+ (cid:88)
[Equation]
[Equation]
[Equation]
[Equation]
= (cid:88) r (cid:48)
[Equation]
q (cid:48) (
[Equation]
) rr (cid:48)  
[Equation]
(cid:48)
[Equation]
xijr (cid:48)
[Equation]
(cid:88)
[Equation]
(cid:48)
[Equation]
(cid:48) ,
[Equation]
(cid:48)
[Equation]
(cid:48)
[Equation]
f
[Equation]
xijs
[Equation]
xijt   = (cid:88) r (cid:48) ρ
[Equation]
(cid:48) ( β ) rr (cid:48)
F kxijqr (cid:48) , where we have used equation 5.
The network will be equivariant if each layer update is equivariant.
Recall the layer update rule of equation 9:
[Equation]
Suppose for the same transformation α
= βt above [Equation] that V k and ∆ X are transformed by α .
Then because the activations associated with each point are representations of SO( n, 1) , they are invariant to the global translation t of the point cloud and we have
[Equation]
where again we applied equation 5.
A. E QUIVARIANT C ONVOLUTIONS
Consider data on a point cloud consisting of a ﬁnite set of spacetime points { (cid:126)x i } ⊂ R 4 , a representation ρ 0 : SO(3 , 1)
→ R 4 × 4 of the Lorentz group deﬁning its action upon spacetime, and feature maps { (cid:126)u i } ⊂ R m , { (cid:126)v i } ⊂ R n associated with representations ρ u : SO(3 , 1) → R m × m and ρ v : SO(3 , 1) → R n × n .
A convolution of this feature map can be written as
[Equation]
in which κ ( (cid:126)x ) : R 4 → R n × m , a matrix-valued function of spacetime, is the ﬁlter kernel.
P 3 -equivariance dictates that for any α ∈ SO(3 , 1) ,
[Equation]
Therefore a single kernel matrix in R n × m may be learned for each coset of spacetime under the action of SO(3 , 1) .
The cosets are indexed by the invariant
[Equation] The kernel may then be obtained at an arbitrary point (cid:126)x ∈ R 4 from equation 12 by computing an α that relates it to the coset representative (cid:126)x 0 : (cid:126)x = ρ 0 ( α ) (cid:126)x 0 .
A natural choice of coset representatives for SO(3 , 1) acting upon R 4 is the set of points { ( t, 0 , 0 , 0) :
t ∈ R + } ∪ { (0 , x, 0 , 0) : x ∈ R + } ∪ { ( t, ct, 0 , 0) :
t ∈ R + } .
A. T ENSOR P RODUCT S TRUCTURE OF L EARNED SO(3) , SO(2 , 1) , SO(3 , 1) G ROUP R EPRESENTATIONS
We quantify the uniqueness of each set of Clebsch-Gordan coefﬁcients in terms of the diagnostic ratio r ( C ) deﬁned in equation 7.
Recall that the value of r becomes large only if there is a nondegenerate nullspace corresponding to a unique set of Clebsch-Gordan coefﬁcients.
For SO(3) and SO(2 , 1) , the irreducible group representations are labeled by an integer which is sometimes called the spin .
We label learned group representations with a primed ( i (cid:48) ) integer.
For the case of SO(3 , 1) the irreducible group representations are obtained from two irreducible group representations of so (3) as explained in section A.1 and we label these representations with both spins i.e. ( s 1 , s 2 ) . We
[Figure] again label the learned group representations of SO(3 , 1) with primed spins, i.e. ( s (cid:48) 1 , s (cid:48) 2 ) .
The tensor product structures of the representations is shown in ﬁgure 3.
We have produced a software library titled Lie Algebraic Networks (LAN) built on PyTorch, which derives all Clebsch-Gordan coefﬁcients and computes the forward pass of Lie group equivariant neural networks.
LAN also deals with Lie algebra representations, allowing for operations such as taking the tensor product of multiple group representations.
ﬁgure 5 demonstrates the LAN library.
Starting from several representations for a Lie algebra, LAN can automatically construct a neural network equivariant to the associated Lie group with the desired number of layers and channels.
We present our experimental results training SO(2 , 1) and SO(3 , 1) -equivariant object-tracking networks in section 5.2.
A. S UPPLEMENTARY FIGURES
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
V(1/2 0 ,1/2 0 ) (1/2 0 ,1/2 0 ) ( 0 , 0 ) ( 1 / 2 ' , 1 / 2 ' )
[Figure]
[Figure]
S V 2 / S V(1/2 0 ,1/2 0 ) (1/2,1/2) (
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
SO(3 , 1)
The diagnostic r (deﬁned in section 2.5) is plotted on the y -axis with a log scale for each subﬁgure.
The labelling of group representations is explained in section 5.1, recall that the primed integers indicate learned representations.
The ﬁrst row demonstrates by Schur’s Lemma that to within numerical error of about ∼ 10 − 6 the learned SO(3) group representation denoted 1 (cid:48) is isomorphic to the spin-1 irreducible group representation obtained from known formulae, i.e. 1 (cid:48) SO(3) ∼ = 1 SO(3) .
The ﬁrst row also indicates that 1 (cid:48)
SO(2 , 1) ∼ = 1 SO(2 , 1) , and (1 / 2 (cid:48) , 1 / 2 (cid:48) )
SO(3 , 1) ∼ =
(1 / 2 , 1 / 2) SO(3 , 1) .
The remaining
[Figure] [Figure]
from lan import LieAlgebraRepresentation, \
[Figure]
LieAlgebraRepresentationDirectSum, \
[Figure]
LieAlgebraTensorProductRepresentation, \
[Figure]
LieGroupEquivariantNeuralNetwork
[Figure]