C LIFFORD N EURAL L AYERS FOR PDE M ODELING
Anonymous authors
Paper under double-blind review
A BSTRACT
Partial differential equations (PDEs) see widespread use in sciences and engineering to describe simulation of physical processes as scalar and vector fields interacting and coevolving over time.
Due to the computationally expensive nature of their standard solution methods, neural PDE surrogates have become an active research topic to accelerate these simulations.
However, current methods do not explicitly take into account the relationship between different fields and their internal components, which are often correlated.
Viewing the time evolution of such correlated fields through the lens of multivector fields allows us to overcome these limitations.
Multivector fields consist of scalar, vector, as well as higherorder components, such as bivectors and trivectors.
Their algebraic properties, such as multiplication, addition and other arithmetic operations can be described by Clifford algebras.
To our knowledge, this paper presents the first usage of such multivector representations together with Clifford convolutions and Clifford
Fourier transforms in the context of deep learning.
The resulting Clifford neural layers are universally applicable and will find direct use in the areas of fluid dynamics, weather forecasting, and the modeling of physical systems in general.
We empirically evaluate the benefit of Clifford neural layers by replacing convolution and Fourier operations in common neural PDE surrogates by their Clifford counterparts on 2D Navier-Stokes and weather modeling tasks, as well as 3D Maxwell equations.
For similar parameter count, Clifford neural layers consistently improve generalization capabilities of the tested neural PDE surrogates.
I NTRODUCTION
Most scientific phenomena are described by the evolution and interaction of physical quantities over space and time.
The concept of fields is one widely used construct to continuously parameterize these quantities over chosen coordinates (McMullin, 2002).
Prominent examples include (i) fluid mechanics, which has applications in domains ranging from mechanical and civil engineering, to geophysics and meteorology, and (ii) electromagnetism, which provides mathematical models for electric, optical, or radio technologies.
The underlying equations of these examples are famously described in various forms of the Navier-Stokes equations (Temam, 2001) and Maxwell’s equations (Griffiths, 2005).
For the majority of these equations, solutions are analytically intractable, and obtaining accurate predictions necessitates falling back on numerical approximation schemes often with prohibitive computation costs.
Deep learning’s success at overcoming the curse of dimensionality in many fields has led to a surge of interest in scientific applications, especially at augmenting and replacing numerical solving schemes in fluid dynamics with neural networks (Thuerey et al., 2021;
Li et al., 2020; Kochkov et al., 2021;
Lu et al., 2021; Rasp & Thuerey, 2021; Keisler, 2022;
Weyn et al., 2020;
Sønderby et al., 2020; Wang et al., 2020a; Pathak et al., 2022).
Taking weather simulations as our motivating example to ground our discussion, two different kinds of fields emerge: scalar fields such as temperature or humidity, and vector fields such as wind velocity or pressure gradients.
Current deep learning based approaches treat different vector field components the same as scalar fields, and stack all scalar fields along the channel dimension, thereby omitting the geometric relations between different components, both within vector fields as well as between individual vector and scalar fields.
This practice leaves out important inductive bias information present in the input data.
For example, wind velocities in the x - and y - directions are strongly related, i.e. they form a vector field.
Additionally, the wind vector field and the scalar pressure field are related since the gradient of the pressure field causes air movement and subsequently influences
[Figure]
the wind components.
In this work, we therefore build neural PDE surrogates which not only model local and global relationships, as current methods do, but also the relation between different fields (e.g. wind and pressure field) and field components (e.g. x - and y - component of the wind velocities).
Figure 1 shows an example of a wind vector field as per the Earth’s shallow water model in two dimensions, and the related scalar pressure field.
Clifford algebras (Suter, 2003; Hestenes, 2003; 2012; Dorst et al., 2010; Renaud, 2020) are at the core intersection of geometry and algebra, introduced to simplify spatial and geometrical relations between many mathematical concepts.
For example, Clifford algebras naturally unify real numbers, vectors, complex numbers, quaternions, exterior algebras, and many more.
Most notably, in contrast to standard vector analysis where primitives are scalars and vectors, Clifford algebras have additional spatial primitives for representing plane and volume segments.
An expository example is the crossproduct of two vectors in 3 dimensions, which naturally translates to a plane segment spanned by these two vectors.
The cross product is often represented as a vector because it has 3 independent components, but has a sign flip under reflection that a true vector does not.
In Clifford algebras, different spatial primitives can be summarized into objects called multivectors , as illustrated in
Figure 2.
In this work, we replace operations over feature fields in deep learning architectures by their Clifford algebra counterparts, which operate on multivector feature fields.
For example, we will endow a convolutional kernel with multivector components, such that it can convolve over multivector feature maps.
Introducing Clifford neural layers yields two advantages: (i) scalar and vector components can be naturally grouped together into one multivector object, with operations on, and mappings between multivectors defined by Clifford algebras; and (ii) higher order geometric information such as bivectors and trivectors arise, which match aforementioned higher order objects.
For example, the curl of a vector field can be described via bivectors, and thus it is beneficial to have bivector filter components to match those.
[Figure]

To summarize our contributions: (i) We define Clifford convolution and Clifford Fourier transforms over multivector fields, building on core concepts of Clifford algebras.
(ii)
We implement Clifford layers for different algebras, proposing the first PyTorch based implementation of Clifford neural layers which we plan to open-source.
(iii)
We evaluate the effect of using Clifford based architectures on 2 -dimensional problems of fluid mechanics and weather modeling, as well as on the 3 -dimensional Maxwell’s equations.
We demonstrate that Clifford neural layers consistently improve generalization capabilities of the tested architectures.
B ACKGROUND : C LIFFORD ALGEBRAS
We introduce important mathematical concepts and discuss three prominent Clifford algebras,
Cl 2 , 0 ( R ) , Cl 0 , 2 ( R ) , Cl 3 , 0 ( R ) , which are directly related to the layers introduced in Section 3.
A more detailed introduction as well as connections to complex numbers and quaternions is given in
Appendix A.
Consider the vector space R n with standard Euclidean product ⟨ ., .
⟩ , where n = p + q , and p and q are non-negative integers.
A real Clifford algebra Cl p,q ( R ) is an associative algebra 1 generated by p + q orthonormal basis elements e 1 , . . .
, e p + q of the generating vector space R n , such that the following quadratic relations hold:
[Equation]
The pair ( p, q ) is called the signature and defines a Clifford algebra Cl p,q ( R ) , together with the basis elements that span the vector space G p + q of Cl p,q ( R ) .
Vector spaces of Clifford algebras have scalar elements and vector elements, but can also have elements consisting of multiple basis elements of the generating vector space R n , which can be interpreted as plane and volume segments.
Exemplary low-dimensional Clifford algebras are: (i) Cl 0 , 0 ( R ) which is a one-dimensional algebra that is spanned by the basis element { 1 } and is therefore isomorphic to R , the field of real numbers;
(ii) Cl 0 , 1 ( R ) which is a two-dimensional algebra with vector space G 1 spanned by { 1 , e 1 } where the basis vector e 1 squares to − 1 , and is therefore isomorphic to C , the field of complex numbers; (iii)
Cl 0 , 2 ( R ) which is a 4-dimensional algebra with vector space G 2 spanned by { 1 , e 1 , e 2 , e 1 e 2 } , where e 1 , e 2 , e 1 e 2 all square to − 1 and anti-commute.
Thus, Cl 0 , 2 ( R ) is isomorphic to the quaternions H .
Grade, dual, geometric product.
The grade of a Clifford algebra basis element is the dimension of the subspace it represents.
For example, the basis elements { 1 , e 1 , e 2 , e 1 e 2 } of the vector space
G 2 of the Clifford algebra Cl 2 , 0 ( R ) have the grades { 0 , 1 , 1 , 2 } .
Using the concept of grades, we can divide Clifford algebras into linear subspaces made up of elements of each grade.
The grade subspace of smallest dimension is M 0 , the subspace of all scalars (elements with 0 basis vectors of the generating vector space).
Elements of M 1 are called vectors, elements of M 2 are bivectors, and so on.
In general, a vector space G p + q of a Clifford algebra Cl p,q ( R ) can be written as the direct sum of all of these subspaces: G p + q = M 0 ⊕ M 1 ⊕ . . .
⊕ M p + q .
The elements of a Clifford algebra are called multivectors , containing elements of subspaces, i.e. scalars, vectors, bivectors, ..., k -vectors.
The basis element with the highest grade is called the pseudoscalar 2 , which in Rcorresponds to the bivector e 1 e 2 , and in R 3 to the trivector e 1 e 2 e 3 .
The dual a ∗ of a multivector a is defined as a ∗ = a i p + q , where i p + q represents the respective pseudoscalar of the Clifford algebra.
This definition allows us to relate different multivectors to each other, which is a useful property when defining Clifford Fourier transforms.
For example, for
Clifford algebras in R 2 the dual of the scalar is the bivector, and in R 3 , the dual of the scalar is the trivector.
The geometric product is a bilinear operation on multivectors.
For arbitrary multivectors a , b , c ∈
G p + q , and scalar λ , the geometric product has the following properties: (i) closure, i.e. ab ∈ G p + q ,
(ii) associativity
, i.e. ( ab ) c = a ( bc ) ; (iii) commutative scalar multiplication, i.e. λ a = a λ ; (iv) distributive over addition, i.e. a ( b + c ) =
ab + ac .
The geometric product is in general non-commutative, i.e. ab ̸ = ba .
Note that Equations 1, 2, and 3 describe the geometric product specifically between basis elements of the generating vector space.
Clifford algebras Cl 2 , 0 ( R ) and Cl 0 , 2 ( R ) .
The 4-dimensional vector spaces of these Clifford algebras have the basis vectors { 1 , e 1 , e 2 , e 1 e 2 } where e 1 , e 2 , e 1 e 2 square to +1 for Cl 2 , 0 ( R ) and to − 1 for Cl 0 , 2 ( R ) .
For Cl 2 , 0 ( R ) , the geometric product of two multivectors a = a
+
a 1 e 1 + a 2 e
+ a 12 e 1 e 2 and b =
b
+ b 1 e 1 + b 2 e
+ b 12 e 1 e 2 is given by:
[Equation]
which can be derived by collecting terms that multiply the same basis elements.
A detailed derivation can be found in Appendix A.
[Figure]
A vector x = ( x 1 , x 2 ) ∈ R 2 with standard Euclidean product ⟨ ., .
⟩ can be related to x 1 e 1 + x 2 e 2 ∈ R 2 ⊂ G 2 .
Clifford multiplication of two vectors x, y ∈ R 2 ⊂ G 2 yields the geometric product xy :
[Equation]
where ∧ is the exterior or wedge product.
The asymmetric quantity x ∧ y = − y ∧ x is associated with the bivector , which can be interpreted as an oriented plane segment as shown in Figure 3 for vectors x and y .
A unit bivector i 2 , spanned by the (orthonormal) basis vectors e
and e 2 is determined by the product:
[Equation]
which if squared yields i 22 = − 1 .
Thus, i 2 represents a geometric √ − 1 .
From Equation 6, it follows that e 2 =
e 1 i 2 = − i 2 e 1 and e 1 = i 2 e 2 = − e 2 i 2 .
Using the pseudoscalar i 2 , the dual of a scalar is a bivector and the dual of a vector is again a vector.
The dual pairs of the base vectors are 1 ↔ e 1 e 2 and e 1 ↔ e 2 .
These dual pairs allow us to write an arbitrary multivector a as a = a 0 + a 1 e 1 + a 2 e 2 + a 12 e 1 e
= 1 (cid:0) a
+ a 12 i 2 (cid:1) (cid:124) (cid:123)(cid:122) (cid:125) spinor part + e 1 (cid:0) a 1 + a 2 i 2 (cid:1) (cid:124) (cid:123)(cid:122) (cid:125) vector part , (7) which can be regarded as two complex-valued parts: the spinor 3 part, which commutes with the base element 1 , i.e. 1 i 2 =
i 2 1 , and the vector part, which anti-commutes with the respective base element e 1 , i.e. e 1 i 2 =
e 1 e 1 e 2 = − e 1 e 2 e
=
−
i 2 e 1 .
This decomposition will be the basis for
Clifford Fourier transforms which naturally act on dual pairs.
The Clifford algebra Cl 0 , 2 ( R ) is isomorphic to the quaternions H , which are an extension of complex numbers and are commonly written in the literature as a + b ˆ ı + c ˆ ȷ + d ˆ k .
Quaternions also form a 4-dimensional algebra spanned by { 1 , ˆ ı, ˆ ȷ, ˆ k } , where ˆ ı , ˆ ȷ , ˆ k all square to − 1 .
The algebra isomorphism to Cl 0 , 2 ( R ) is easy to verify since e 1 , e 2 , e 1 e 2 all square to − 1 and anti-commute.
The basis element 1 is often called the scalar part, and the basis elements ˆ ı , ˆ ȷ , ˆ k are called the vector part of a quaternion.
Quaternions have practical uses in applied mathematics, particularly for calculations involving three-dimensional rotations, which we will use to define the rotational Clifford convolution layer in Section 3.
Clifford algebra Cl 3 , 0 ( R ) .
The 8-dimensional vector space G 3 of the Clifford algebra Cl 3 , 0 ( R ) has the basis vectors { 1 , e 1 , e 2 , e 3 , e 1 e 2 , e 3 e 1 , e
2 e 3 , e 1 e 2 e 3 } , i.e. it consists of one scalar, three vectors { e 1 , e 2 , e 3 } , three bivectors { e 1 e 2 , e 3 e 1 , e 2 e 3 } 4 , and one trivector e 1 e 2 e 3 .
The trivector is the pseudoscalar i 3 of the algebra.
The geometric product of two multivectors is defined analogously to the geometric product of Cl 2 , 0 ( R ) , see Appendix A.
The dual pairs of Cl 3 , 0 ( R ) are: 1 ↔ e 1 e 2 e 3 =
i 3 , e 1 ↔ e 2 e 3 , e 2 ↔ e 3 e 1 , and e 3 ↔ e 1 e 2 .
An intriguing example of the duality of the multivectors of Cl 3 , 0 ( R ) emerges when writing the expression of the electromagnetic field F in terms of an electric vector field E and a magnetic vector field B ,
such that F = E + Bi 3 , where
E =
E x e
+ E y e
+ E z e 3 and B = B x e 1 + B y e
+ B z e 3 .
In this way the electromagnetic field F decomposes into electric vector and magnetic bivector parts via the pseudoscalar i 3 (Hestenes, 2003).
For example, for the base component B x e 1 of B it holds that B x e
i 3 =
B x e 1 e 1 e 2 e 3 =
B x e 2 ewhich is a bivector and the dual to the base component E x e 1 of E .
Consequently, the multivector representing F consists of three vectors (the electric field components) and three bivectors (the magnetic field components multiplied by i 3 ).
This viewpoint gives Clifford neural layers a natural advantage over their default counterparts as we will see in Section 4.
C LIFFORD NEURAL LAYERS
We introduce Clifford convolution and Clifford Fourier transform layers for 2-dimensional problems.
Appendix B contains extensions to 3 dimensions, and discussions about normalization and initialization procedures.
In Appendices B, D the relationship to existing literature is discussed.
Clifford CNN layers.
Regular convolutional neural network (CNN) layers take as input feature maps f : Z 2 → R c in and convolve 5 them with a set of c out filters { w i } c out i =1 with w i : Z 2 → R c in :
[Equation]
which can be interpreted as an inner product of input feature maps with the corresponding filters at every point y ∈
Z 2 .
By applying c out filters, the output feature maps can be interpreted as c out dimensional feature vectors at every point y ∈
Z 2 .
We now want to extend convolution layers such that the element-wise product of scalars f j ( y ) w i,j ( y − x ) is replaced by the geometric product of multivector inputs and multivector filters f j ( y ) w i,j ( y − x ) .
We therefore replace the feature maps f : Z 2 → R c in by multivector feature maps f : Z 2 → ( G 2 ) c in and convolve them with a set of c out multivector filters { w i } c out i =1 : Z 2 → ( G 2 )
[Equation] [Equation]
[Figure]
Geometricproduct
Multivectorinputfields
Multivectorkernels
Multivectoroutputfields
[Figure]
Note that each (geometric) product, indexed by i ∈ { 1 , ..., c out } and j ∈ { 1 , ..., c in } , now results in a new multivector rather than a scalar.
Hence, the output of a layer is a grid of c out multivectors.
We can implement a 2D Clifford
CNN layer using Equation 4 where { b 0 , b 1 , b 2 , b 12 } → { w i,j 0 , w i,j 1 , w i,j 2 , w i,j 12 } correspond to 4 different kernels representing one 2D multivector kernel, i.e. 4 different convolution layers, and { a 0 , a 1 , a 2 , a 12 } → { f j 0 , f j 1 , f j 2 , f j 12 } correspond to the scalar, vector and bivector parts of the input multivector field.
The channels of the different layers represent different stacks of scalars, vectors, and bivectors.
Analogously, we can implement a 3D Clifford CNN layer using Equationin Appendix B. A schematic sketch of a Clifford convolution layer is shown in Figure 4.
Rotational Clifford CNN layers.
Here we introduce an alternative parameterization to the Clifford CNN layer introduced in Equation 9 by using the isomorphism of the Clifford algebra Cl 0 , 2 ( R ) to quaternions.
We take advantage of the fact that a quaternion rotation can be realized by a matrix multiplication (Jia, 2008; Kuipers, 1999; Schwichtenberg, 2015).
Using the isomorphism, we can represent the feature maps f j and filters w i,j as quaternions: f j = f j 0 + f j 1 ˆ ı + f j 2 ˆ ȷ + f j 3 ˆ k and w i,j = w i,j 0 + w i,j 1 ˆ ı + w i,j 2 ˆ ȷ + w i,j 3 ˆ k 6 .
We can now devise an alternative parameterization of the product between the feature map f j and w i,j .
To be more precise, we introduce a composite operation that results in a scalar quantity and a quaternion rotation, where the latter acts on the vector part of the quaternion f j and only produces nonzero expansion coefficients for the vector part of the quaternion output.
A quaternion rotation w i,j f j ( w i,j ) − 1 acts on the vector part ( ˆ ı, ˆ ȷ, ˆ k ) of f j , and can be algebraically manipulated into a vector-matrix operation
R i,j f j , where R i,j : H → H is built up from the elements of w i,j (Kuipers, 1999).
In other words, one can transform the vector part ( ˆ ı , ˆ ȷ , ˆ k ) of f j ∈ H via a rotation matrix R i,j that is built from the scalar and vector part ( 1 , ˆ ı , ˆ ȷ , ˆ k ) of w i,j ∈ H .
Altogether, a rotational multivector filter { w i rot } c out i =1 : Z 2 → ( G 2 ) c in acts on the feature map f j through a rotational transformation R i,j ( w i,j rot , 0 , w i,j rot , 1 , w i,j rot , 2 , w i,j rot , 12 ) acting on vector and bivector parts of the multivector feature map f : Z 2 → ( G 2 ) c in , and an additional scalar response of the multivector filters:
[Equation]
where (cid:2)
f j ( y ) w i,j rot ( y − x )) (cid:3)
= f
j 0 w i,j rot , 0 − f j 1 w i,j rot , 1 − f j 2 w i,j rot , 2 − f j 12 w i,j rot , 12 , which is the scalar output of Equation 4.
A detailed description of the rotational multivector filters R i,j ( y − x ) is outlined in
Appendix B. While in principle the Clifford CNN layer in Equation 9 and the rotational Clifford
CNN layer in Equation 10 are equally flexible, our experiments in Section 4 show that rotational
Clifford CNN layers lead to better performance.
Clifford convolutions satisfy the property of equivariance under translation of the multivector inputs, as shown in theorem 1 in Appendix B. Analogous to Theorem 1, translation equivariance can also be shown for rotational Clifford CNN layers.
However, the current definition of Clifford convolutions is not equivariant under multivector rotations or reflections.
In Appendix B.4, we derive a general kernel constraint for multivector kernels, which sketches a path towards generalized Clifford convolutions which are equivariant w.r.t rotations or reflections.
Clifford Fourier layers.
In arbitrary dimension n , the discrete Fourier transform of an ndimensional complex signal f ( x ) =
f
( x 1 , . . .
, x n ) : R n → C at M 1 × . . .
× M n grid points is defined as:
[Equation]
where ( ξ 1 , . .
. , ξ n ) ∈
Z M 1 . . .
× . . .
Z M n .
In Fourier Neural Operators (FNO) (Li et al., 2020), discrete Fourier transforms on real-valued input fields and respective back-transforms – implemented as
Fast Fourier Transforms on real-valued inputs (RFFTs) 7 – are interleaved with a weight multiplication by a complex weight matrix of shape c in × c out for each mode, which results in a complex-valued weight tensor of the form W ∈ C c in × c out × ( ξ max 1 × ...
× ξ max n ) , where Fourier modes above cut-off frequencies ( ξ max 1 , . .
. , ξ max n ) are set to zero.
Additionally, a residual connection is usually implemented as convolution layer with kernel size 1.
In Figure 5a, a sketch of an FNO layer is shown.
The 2dimensional Clifford Fourier transform (Ebling & Scheuermann, 2005; Ebling, 2006; Hitzer, 2012) and for multivector valued functions f ( x ) : R 2 → G 2 and vectors x, ξ ∈ R 2 is defined as:
[Equation]
[Equation]
provided that the integral exists.
In contrast to standard Fourier transforms, f ( x ) and ˆ f ( ξ ) represent multivector fields in the spatial and the frequency domain
[Equation]
respectively.
Furthermore
[Equation]
i 2 = e 1 e 2 is used in the exponent.
Inserting the definition of multivector fields
[Equation]
we can rewrite Equation 12 as:
[Equation]
We obtain a Clifford Fourier transform by applying two standard Fourier transforms to the dual pairs f 0 = f 0 ( x ) + f 12 ( x ) i 2 and f 1 = f 1 ( x ) + f 2 ( x ) i 2 , which both can be treated as a complexvalued signals f 0 , f 1 : R 2 → C .
Consequently, f ( x ) can be understood as an element of C 2 .
R FF T
F ( f ( x ))
R FF T −f ( x ) f ∗ ( x ) f † ( x )
W w (a) FNO layer f ( x ) v ( x ) s ( x ) f ∗ ( x ) v ∗ ( x ) s ∗ ( x )
FFT F ( f ( x ))
FFT
FFT −
FFT −f † ( x )
W w (b) Clifford FNO layer
The real valued Fast Fourier transform (RFFT) over real valued scalar input fields f ( x ) is replaced by the complex Fast Fourier transform (FFT) over the complex valued dual parts v ( x ) and s ( x ) of multivector fields f ( x ) .
Pointwise multiplication in the Fourier space via complex weight tensor W is replaced by the geometric product in the Clifford Fourier space via multivector weight tensor W .
Additionally, the convolution path is replaced by Clifford convolutions with multivector kernels w .
The 2D Clifford Fourier transform is the linear combination of two classical Fourier transforms.
Discrete versions of Equation 13 are obtained analogously to Equation 11, see Appendix B. Similar to FNO, multivector weight tensors W ∈ ( G
) c in × c out × ( ξ max 1 × ξ max 2 ) are applied, where again Fourier modes above cut-off frequencies ( ξ max 1 , ξ max 2 ) are set to zero.
In doing so, we point-wise modify the
Clifford Fourier modes ˆ f ( ξ )
=
F{ f } ( ξ ) =
ˆ f 0 ( ξ )+ ˆ f 1 ( ξ ) e 1 + ˆ f 2 ( ξ ) e
+ ˆ f
( ξ ) e 12 via the geometric product.
The Clifford Fourier modes follow naturally when combining spinor and vector parts of
Equation 13.
Finally, the residual connection is replaced by a Clifford convolution with multivector kernel k .
A schematic sketch is shown in Figure 5b.
3D Clifford Fourier transforms follow a similar elegant construction, where we apply four separate Fourier transforms to f 0 ( x ) = f 0 ( x ) + f 123 ( x ) i 3 , f 1 ( x ) = f 1 ( x )+ f 23 ( x ) i 3 , f 2 ( x ) = f 2 ( x )+ f 31 ( x ) i 3 , and f 3 ( x ) = f 3 ( x )+ f 12 ( x ) i 3 , i.e. scalar/trivector and vector/bivector components are combined into complex fields and then subjected to a Fourier transform.
E XPERIMENTS
We assess Clifford neural layers for different architectures in three experimental settings: the incompressible Navier-Stokes equations, shallow water equations for weather modeling, and 3dimensional Maxwell’s equations.
All datasets share the common trait of containing multiple input and output fields.
More precisely, one scalar and one 2 -dimensional vector field in case of the
Navier-Stokes and the shallow water equations, and a 3 -dimensional (electric) vector field and its dual (magnetic) bivector field in case of the Maxwell’s equations.
We replace carefully designed
ResNet and FNO based architectures by their Clifford counterparts.
In doing so, we change convolutions and Fourier transforms to their respective Clifford operation, and substitute normalization techniques and activation functions with Clifford counterparts, keeping the number of parameters similar.
We evaluate different training set sizes, and report losses for scalar and vector fields.
Inputs to the neural PDE surrogates are respective fields at previous t timesteps, where t varies between different PDEs.
The one-step loss is the mean-squared error at the next timestep summed over fields.
The rollout loss is the mean-squared error after applying the neural PDE surrogate 5 times, summing over fields and time dimension.
More information on the datasets, implementation details of the tested architectures, loss functions, and more detailed results can be found in Appendix C.
Navier-Stokes in 2D.
The incompressible Navier-Stokes equations (Temam, 2001) are built upon momentum and mass conservation of fluids expressed for the velocity flow field.
In addition to the velocity field, we introduce a scalar field representing a scalar quantity, i.e. smoke, that is being transported via the velocity field.
The scalar field is advected by the vector field, i.e. as the vector field changes, the scalar field is transported along with it, whereas the scalar field influences the vector field only via an external force term.
We call this weak coupling between vector and scalar fields.
Results for one-step loss and rollout loss on the test set are shown in Figure 6.
For ResNet-like architectures, we observe that both CResNet and CResNet rot improve upon the ResNet baseline.
[Figure]
Additionally, we observe that rollout losses are also lower for the two Clifford based architectures, which we attribute to better and more stable models that do not overfit to one-step predictions so easily.
Lastly, while in principle CResNet and CResNet rot based architectures are equally flexible, CResNet rot ones in general perform better than CResNet ones.
For FNO and respective Clifford
Fourier based (CFNO) architectures, the loss is in general much lower than for
ResNet based architectures.
CFNO architectures improve upon FNO architectures for all dataset sizes, and for onestep as well as rollout losses.
Shallow water equations.
The shallow water equations (Vreugdenhil, 1994) describe a thin layer of fluid of constant density in hydrostatic balance, bounded from below by the bottom topography and from above by a free surface viacoupled PDEs.
For example, the deep water propagation of a tsunami can be described by the shallow water equations.
For simplified weather modeling, the shallow water equations express the velocity in x - direction, or zonal velocity v x , the velocity in the y - direction, or meridional velocity v y , the acceleration due to gravity g , and the vertical displacement of free surface, which subsequently is used to derive pressure fields.
The relation between vector and scalar components is relatively strong ( strong coupling between vector and scalar fields due to the 3-coupled PDEs.
We observe similar results than for the Navier-Stokes experiments.
However, performance differences between baseline and Clifford architectures are even more pronounced, which we attribute to the stronger coupling of the scalar and the vector fields.
[Figure]
Maxwell’s equations in matter in 3D.
In isotropic media, Maxwell’s equations (Griffiths, 2005) in matter propagate solutions of the displacement field
D , which is related to the electrical field via D = ϵ
ϵ r E , where ϵ 0 is the permittivity of free space and ϵ r is the permittivity of the medium, and the magnetization field H , which is related to the magnetic field B via H = µ 0 µ r B , where µ 0 is the permeability of free space and µ r is the permeability of the medium.
The electromagnetic field F has the intriguing property that the electric field E and the magnetic field B are dual pairs, thus F = E + Bi 3 , i.e. strong coupling between the electric field and its dual (bivector) magnetic field.
This duality also holds for D and H .
We randomly place 18 (6 in the x - y plane, 6 in the x - z plane, 6 in the y - z plane) different light sources outside a cube which emit light with different amplitudes and different phase shifts, causing the resulting D and H fields to interfere.
[Figure]
We test FNO based architectures and respective Clifford counterparts (CFNO).
Note that Maxwell’s equations are in general relatively simple due to the wave-like propagation of electric and magnetic fields.
It is however a good playground to demonstrate the inductive bias advantages of Clifford based architectures, which leverage the vectorbivector character of electric and magnetic field components.
Results for one-step loss and rollout loss on the test set are shown in Figure 8.
CFNO architectures improve upon FNO architectures, especially for low numbers of trajectories.
Results demonstrate the much stronger inductive bias of Clifford based 3-dimensional Fourier layers, and their general applicability to 3-dimensional problems, which are structurally even more interesting than 2-dimensional ones.
C ONCLUSION
We introduced Clifford neural layers, both for convolution and Fourier operations, that handle the various scalar (e.g. charge density), vector (e.g. electric field), bivector (magnetic field) and higher order fields as proper geometric objects organized as multivectors.
This geometric algebra perspective, allowed us to naturally generalize convolution and
Fourier transformations to their Clifford counterparts, and provides an elegant and natural rule to design new neural network layers.
The multivector viewpoint with our Clifford neural layers led to better representation of the relationship between different fields and their individual components, allowing us to significantly outperform equivalent standard architectures for neural PDE surrogates.
We found that the performance gap increased with less data availability across all settings, denoting better inductive bias of our approach.
Limitations.
One limitation is the current speed of Fast Fourier Transform (FFT) operations on machine learning accelerators like GPUs.
While an active area of research, current available versions of cuFFT 8 kernels wrapped in PyTorch (Paszke et al., 2019) are not yet as heavily optimized 9 , especially for the gradient pass.
Furthermore, FFTs of real-valued signals are Hermitian-symmetric, and outputs contain only positive frequencies below the Nyquist frequency.
Such real-valued FFTs are used for Fourier Neural Operator (FNO) layers.
Clifford Fourier layers on the other hand use complex-valued FFT operations where the backward pass is approximately twice as slow.
For similar parameter counts, inference times of FNO and CFNO networks are similar.
The speed of geometric convolution layers was already investigated by Grassucci et al.
We found that on the one hand Clifford convolutions are more parameter efficient since they share parameters among filters, on the other hand the net number of operations is larger for the same number of parameters resulting in increased training times by a factor of about 2 .
Finally, from a PDE point of view, the presented approaches to obtain PDE surrogates are limited since the neural networks have to be retrained for different equation parameters or e.g. different ∆ t .
Future work.
Besides PDE modeling, weather predictions, and fluid dynamics, we see clear applications of multivector representations and Clifford layers in neural implicit representations (Xie et al., 2022; Rella et al., 2022).
Extensions towards Clifford graph networks and Clifford attention based models, where the geometric product replaces the dot product will be useful to explore.
Similarly, the potential merit of equivariant Clifford layers would be interesting to explore in the future.
Custom multivector GPU kernels can overcome many of the speed issues as the compute density of
Clifford operations is much higher which is better for hardware accelerators (Hoffmann et al., 2020).
Use of a just-in-time compiled language with better array abstractions like Julia (Bezanson et al., 2017) can significantly simplify the interface, especially for higher-order Clifford algebras.
Finally, combining the idea of multivector modeling together with various physics-informed neural network approaches (Raissi et al., 2019; Lutter et al., 2018; Gupta et al., 2019; Cranmer et al., 2020; Zubov et al., 2021) is an attractive next step. R EPRODUCIBILITY AND ETHICAL STATEMENT Reproducibility statement.
We have included error bars, and ablation studies wherever we found it necessary and appropriate. We have described our architectures in Section 4 and provided further implementation details in Appendix Section C. We have further include pseudocode for the newly proposed layers in Section B.6. We will open-source the codebase for our new layers for others to use and develope further.
Ethical statement.
Neural PDE surrogates will play an important role in modeling many natural phenomena, and thus developing them further might enable us to achieve shortcuts or alternatives for computationally expensive simulations.
For example, if used as such, PDE surrogates will potentially help to advance different fields of research, especially in the natural sciences.
Examples related to this paper are fluid dynamics or weather modeling.
Therefore, PDE surrogates might potentially be directly or indirectly related to reducing the carbon footprint.
On the downside, relying on simulations always requires rigorous cross-checks and monitoring, especially when we “learn to simulate”
B Clifford neural layers
B.1 Clifford convolution layers . .
B.1.
Translation equivariance of Clifford convolutions . . . . . . . . . . . . . .
B.1.
Rotational Clifford CNN layers . .
B.1.
3D Clifford convolution layers . .
B.2 Clifford normalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.3 Clifford initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.4 Equivariance under rotations and reflections . . . . . . . . . . . . . . . . . . . . .
B.5 Clifford Fourier layers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.5.
2D Clifford Fourier transform . . . . . . . . . . . . . . . . . . . . . . . .
B.5.
2D Clifford convolution theorem . . . . . . . . . . . . . . . . . . . . . . .
B.5.
3D Clifford Fourier transform . . . . . . . . . . . . . . . . . . . . . . . .
B.5.
3D Clifford convolution theorem . . . . . . . . . . . . . . . . . . . . . . .
B.5.
Implementation of Clifford Fourier layers . .
B.6 Pseudocode . .
C Experiments
C .1 Loss function and metrics
[Table] [Table]
and
[Table] [Table]
[Table] [Table]
Maxwell’s
[Table]
D Related work
E Glossary
A M ATHEMATICAL BACKGROUND
This appendix supports Section 2 of the main paper.
We give a more detailed explanation of real
Clifford algebras and have a closer look at Cl 2 , 0 ( R ) , Cl 0 , 2 ( R ) , and Cl 3 , 0 ( R ) .
For a detailed introduction into Clifford algebras we recommend Suter (2003);
Hestenes (2003; 2012); Dorst et al.
; Renaud (2020)
A. C LIFFORD ALGEBRAS
Vector spaces and algebras over a field.
A vector space over a field F is a set V together with two binary operations that satisfy the axioms for vector addition and scalar multiplication.
The axioms of addition ensure that if two elements of V get added together, we end up with another element of V .
The elements of F are called scalars.
Examples of a field F are the real numbers R and the complex numbers C .
Although it is common practice to refer to the elements of a general vector space V as vectors, to avoid confusion we will reserve the usage of this term to the more specific case of elements of R n .
As we will see below, general vector spaces can consist of more complicated, higher-order objects than scalars, vectors or matrices.
An algebra over a field consists of a vector space V over a field F together with an additional bilinear law of composition of elements of the vector space, V × V → V , that is, if a and b are any two elements of V , then ab : V × V
→ V is an element of V , satisfying a pair of distribution laws: a ( λ 1 b + λ 2 c ) =
λ 1 ab + λ 2 ac
and ( λ 1 a + λ 2 b ) c
= λ 1 ac + λ 2 bc for λ 1 , λ 2 ∈ F and a, b, c ∈ V .
Note that general vector spaces don’t have bilinear operations defined on their elements.
Clifford algebras over R .
In this manuscript we will focus on Clifford algebras over R .
For a more general exposition on Clifford algebras over different fields the reader is referred to Lounesto (1986).
A real Clifford algebra is generated by the n -dimensional vector space R n through a set of relations that hold for the basis elements of the vector space R n .
Let us denote the basis elements of R n with e 1 , ..., e n , and without loss of generality choose these basis elements to be mutually orthonormal.
Taking two nonnegative integers p and q , such that p + q = n , then a real Clifford algebra Cl p,q ( R ) with the “signature” ( p, q ) , is generated through the following relations that define how the bilinear product of the algebra operates on the basis elements of R n :
[Equation]
Through these relations we can generate a basis for the vector space of the Clifford algebra, which we will denote with G .
Equations 14 and 15 show that the product between two vectors yields a scalar.
According to the aforementioned definition of an algebra over a field, a Clifford algebra with a vector space G is equipped with a bilinear product G × G (cid:55)→ G , that combines two elements from the vector space G and yields another element of the same space G .
Therefore, both scalars and vectors must be elements of the vector space G .
Equation 16 shows that besides scalar and vector elements, higher order elements consisting of a combination of two basis elements, such as e i e j and e j e i , are also part of the vector space G .
Finally, by combining Equations 14, 15, 16 we can create even higher order elements such as e i e j e k for i ̸
=
j ̸ = k , or e 1 e 2 ...
e p + q , which all must be part of the vector space G .
In order to determine what the basis elements are that span the vector space G of Cl p,q ( R ) , we note that elements e σ (1) e σ (2) ...
e σ ( k ) and e 1 e 2 ...
e k are related through a simple scalar multiplicative factor of plus or minus one, depending on the sign of the permutation σ .
Therefore, it suffices to consider the unordered combinations of basis elements of R n : the basis of the vector space G is given by { 1 , e 1 , e 2 , ..., e p + q , e 1 e 2 , ..., e p + q − 1 e p + q , ..., e 1 e 2 ...
e p + q } .
In summary, we have introduced two different vector spaces.
First, the vector space R n which generates the Clifford algebra, and second the vector space G , which is the vector space spanned by the basis elements of the Clifford algebra Cl p,q ( R ) .
Convention is to denote the vector space of a real Clifford algebra with a superscript n of the dimension of the generating vector space, yielding
G n for a generating vector space R n .
Note that the dimension of the vector space G n is 2 n = 2 p + q .
Exemplary low-dimensional Clifford algebras are: (i) Cl 0 , 0 ( R ) which is a one-dimensional algebra that is spanned by the vector { 1 } and is therefore isomorphic to R , the field of real numbers; (ii)
Cl 0 , 1 ( R ) which is a two-dimensional algebra with vector space G 1 spanned by { 1 , e 1 } where the basis vector e 1 squares to − 1 , and is therefore isomorphic to C , the field of complex numbers; (iii)
Cl 0 , 2 ( R ) which is a 4-dimensional algebra with vector space G 2 spanned by { 1 , e 1 , e 2 , e 1 e 2 } , where e 1 , e 2 , e 1 e 2 all square to − 1 and anti-commute.
Thus, Cl 0 , 2 ( R ) is isomorphic to the quaternions H .
Definition 1: Grade of Clifford algebra element
The grade of a Clifford algebra basis element is the dimension of the subspace it represents.
For example, the basis elements { 1 , e 1 , e 2 , e 1 e 2 } of the Clifford algebras Cl 0 , 2 ( R ) and Cl 2 , 0 ( R ) have the grades { 0 , 1 , 1 , 2 } .
Using the concept of grades, we can divide the vector spaces of Clifford algebras into linear subspaces made up of elements of each grade.
The grade subspace of smallest dimension is M 0 , the subspace of all scalars (elements with 0 basis vectors).
Elements of M 1 are called vectors, elements of M 2 are bivectors, and so on.
In general, the vector space G p + q of a
Clifford algebra Cl p,q can be written as the direct sum of all of these subspaces:
[Equation]
The elements of a Clifford algebra are called multivectors , containing elements of subspaces, i.e. scalars, vectors, bivectors, trivectors etc.
The basis element with the highest grade is called the pseudoscalar 10 , which in R 2 corresponds to the bivector e 1 e 2 , and in R 3 to the trivector e 1 e 2 e 3 .
The pseudoscalar is often denoted with the symbol i p + q .
From hereon, only multivectors will be denoted with boldface symbols.
Geometric product.
Using Equations 14, 15, 16, we have seen how basis elements of the vector space G p + q of the Clifford algebra are formed using basis elements of the generating vector space
V .
We now, look at how elements of G p + q are combined, i.e. how multivectors are bilinearly operated on.
The geometric product is the bilinear operation on multivectors in Clifford algebras.
For arbitrary multivectors a , b , c ∈ G p + q , and scalar λ the geometric product has the following properties:
[Equation]
The geometric product is in general non-commutative, i.e. ab ̸ = ba .
As we describe later, the geometric product is made up of two things: an inner product (that captures similarity) and exterior (wedge) product that captures difference.
Definition 2: Dual of a multivector
The dual a ∗ of a multivector a is defined as:
[Equation]
[Equation]
where i p + q represents the respective pseudoscalar of the Clifford algebra.
This definition allows us to relate different multivectors to each other, which is a useful property when defining Clifford Fourier transforms.
For example, for Clifford algebras in R 2 the dual of a scalar is a bivector, and for the Clifford algebra R 3 the dual of a scalar is a trivector.
A. E XAMPLES OF LOW - DIMENSIONAL C LIFFORD ALGEBRAS A.2. C LIFFORD ALGEBRA Cl 0 , 1 ( R )
The Clifford algebra Cl 0 , 1 ( R ) is a two-dimensional algebra with vector space G 1 spanned by { 1 , e 1 } , and where the basis vector e 1 squares to − 1 .
Cl 0 , 1 ( R ) is thus algebra-isomorphic to C , the field of complex numbers.
This becomes more obvious if we identify the basis element with the highest grade, i.e. e 1 , as the pseudoscalar i 1 which is the imaginary part of the complex numbers.
The geometric product between two multivectors a = a
+ a 1 e 1 and b =
b 0 + b 1 e 1 is therefore also isomorphic to the product of two complex numbers:
[Equation]
A.2. C LIFFORD ALGEBRA Cl 2 , 0 ( R )
The Clifford algebra Cl 2 , 0 ( R ) is a 4-dimensional algebra with vector space G 2 spanned by the basis vectors { 1 , e 1 , e 2 , e 1 e 2 } where e 1 , e 2 , e 1 e 2 all square to +1 .
The geometric product of two multivectors a = a 0 + a 1 e 1 + a 2 e 2 + a 12 e 1 e 2 and b =
b
+ b 1 e 1 + b 2 e
+ b 12 e 1 e 2 is defined via:
[Equation]
[Equation]
Using the relations e 1 e 1 = 1 , e 2 e 2 = 1 , and e i e j = − e j e i for i ̸
= j ∈ { e 1 , e 2 } , from which it follows that e 1 e 2 e 1 e 2 = − 1 , we obtain:
[Equation]
A vector x ∈ R 2 ⊂ G 2 is identified with x 1 e 1 + x 2 e 2 ∈ R 2 ⊂ G 2 .
Clifford multiplication of two vectors x, y ∈ R 2 ⊂ G 2 yields the geometric product xy :
[Equation]
[Equation]
Inner product Outer/Wedge product
The asymmetric quantity x ∧ y = − y ∧ x is associated with the now often mentioned bivector , which can be interpreted as an oriented plane segment.
Equation 26 can be rewritten to express the (symmetric) inner product and the (anti-symmetric) outer product in terms of the geometric product:
[Equation]
[Equation]
From the basis vectors of the vector space G 2 of the Clifford algebra Cl 2 , 0 ( R ) , i.e. { 1 , e 1 , e 2 , e 1 e 2 } , probably the most interesting is e 1 e 2 .
We therefore have a closer look the unit bivector i 2 =
e 1 ewhich is the plane spanned by e 1 and e 2 and determined by the geometric product:
[Equation]
where the inner product ⟨ e 1 , e 2 ⟩ is zero due to the orthogonality of the base vectors.
The bivector i 2 if squared yields i 22 = − 1 , and thus i 2 represents a true geometric √− 1 .
From Equation 29, it follows that
[Equation]
Using definition 2, the dual of a multivector a ∈ G 2 is defined via the bivector as i 2 a .
Thus, the dual of a scalar is a bivector and the dual of a vector is again a vector.
The dual pairs of the base vectors are 1 ↔ e 1 e 2 and e 1 ↔ e 2 .
These dual pairs allow us to write an arbitrary multivector a as
[Equation]
which can be regarded as two complex-valued parts: the spinor part, which commutes with i 2 and the vector part, which anti-commutes with i 2 .
A.
2. C LIFFORD ALGEBRA Cl 0 , 2 ( R )
The Clifford algebra Cl 0 , 2 ( R ) is a 4-dimensional algebra with vector space G 2 spanned by the basis vectors { 1 , e 1 , e 2 , e 1 e 2 } where e 1 , e 2 , e 1 e 2 all square to − 1 .
The Clifford algebra Cl 0 , 2 ( R ) is algebra-isomorphic to the quaternions H , which are commonly written in literature (Schwichtenberg, 2015) as a + b ˆ ı + c ˆ ȷ + d ˆ k , where the (imaginary) base elements ˆ ı , ˆ ȷ , and ˆ k fulfill the relations:
[Equation]
Quaternions also form a 4-dimensional algebra spanned by { 1 , ˆ ı, ˆ ȷ, ˆ k } , where ˆ ı , ˆ ȷ , ˆ k all square to − 1 .
The basis element 1 is often called the scalar part, and the basis elements ˆ ı , ˆ ȷ , ˆ k are called the vector part of a quaternion.
The geometric product of two multivectors a = a 0 + a 1 e 1 + a 2 e 2 + a 12 e 1 e 2 and b =
b
+ b 1 e 1 + b 2 e
+ b 12 e 1 e 2 is defined as:
[Equation]
[Equation]
Using the relations e 1 e 1 = − 1 , e 2 e 2 = − 1 , and e i e j = − e j e i for i ̸
= j ∈ { e 1 , e 2 } , from which it follows that e 1 e 2 e 1 e 2 = − 1 , we obtain:
[Equation]
A.2. C LIFFORD ALGEBRA Cl 3 , 0 ( R )
The Clifford algebra is a 8-dimensional algebra with vector space G 3 spanned by the basis vectors { 1 , e 1 , e 2 , e 3 , e 1 e 2 , e 1 e 3 , e 2 e 3 , e 1 e 2 e 3 } , i.e. one scalar, three vectors { e 1 , e 2 , e 3 } , three bivectors { e 1 e 2 , e
1 e 3 , e 2 e 3 } , and one trivector e 1 e 2 e 3 .
The trivector is the pseudoscalar i 3 of the algebra.
The geometric product of two multivectors is defined analogously to the geometric product of Cl 2 , 0 ( R ) , following the associative and bilinear multiplication of multivectors follows:
[Equation]
Using Definition 2, the dual pairs of Cl 3 , 0 are:
[Equation]
The geometric product for Cl 3 , 0 ( R ) is defined analogously to the geometric product of Cl 2 , 0 ( R ) via:
[Equation]
b 1 e 2 e 1 + a 2 b 2 e 2 e
+ a 2 b 3 e 2 e+ a 2 b 12 e 2 e 1 e 2 + a 2 b 13 e 1 e 3 e 2 + a 2 b 23 e 2 e 2 e 3 − a 2 b 123 e 2 e 2 e 1 e+ a 3 b 0 e 3 + a 3 b 1 e 3 e 1 + a 3 b 2 e 3 e 2 + a 3 b 3 e 3 e+ a 3 b 12 e 1 e 3 e 2 − a 3 b 13 e 1 e 3 e 3 − a 3 b 23 e 2 e 3 e 3 + a 3 b 123 e 1 e 2 e 3 e+ a 12 b 0 e 1 e 2 − a 12 b 1 e 2 e 1 e 1 + a 12 b 2 e 1 e 2 e 2 + a 12 b 3 e 1 e 2 e+ a 12 b 12 e 1 e 2 e 1 e 2 − a 12 b 13 e 1 e 1 e 2 e 3 + a 12 b 23 e 2 e 2 e 1 e+ a 12 b 123 e 1 e 2 e 1 e 2 e+ a 13 b 0 e 1 e 3 − a 13 b 1 e 3 e 1 e 1 + a 13 b 2 e 1 e 3 e 2 + a 13 b 3 e 1 e 3 e− a 13 b 12 e 1 e 1 e 3 e 2 + a 13 b 13 e 1 e 3 e 1 e 3 − a 13 b 23 e 1 e 2 e 3 e+ a 13 b 123 e 1 e 3 e 1 e 2 e+ a 23 b
e 2 e 3 + a 23 b 1 e 1 e 3 e 2 + a 23 b
e 2 e 3 e 2 + a 23 b 3 e 2 e 3 e+ a 23 b 12 e 2 e 3 e 1 e 2 − a 23 b 13 e 2 e 1 e 3 e 3 + a 23 b 23 e 2 e 3 e 2 e+ a 23 b 123 e 2 e 3 e 1 e 2 e+ a 123 b 0 e 1 e 2 e
+ a 123 b 1 e 1 e 2 e 3 e 1 − a 123 b 2 e 1 e 3 e 1 e 2 + a 123 b 3 e 1 e 2 e 3 e+ a 123 b 12 e 1 e
[Equation]
where minus signs appear to do reordering of basis elements.
Equation 41 simplifies to ab = a 0 b 0 + a 1 b 1 + a 2 b 2 + a 3 b 3 − a 12 b 12 − a 13 b 13 − a 23 b 23 − a 123 b 123 ‘ + ( a 0 b 1 + a 1 b 0 − a 2 b 12 − a 3 b 13 + a 12 b 2 + a 13 b 3 − a 23 b 123 − a 123 b 23 ) e+ ( a 0 b 2 + a 1 b 12 + a 2 b 0 − a 3 b 23 − a 12 b 1 + a 13 b 123 + a 23 b 3 + a 123 b 13 ) e+
( a 0 b 3 + a 1 b 13 + a 2 b 23 + a 3 b 0 − a 12 b 123 − a 13 b 1 − a 23 b 2 − a 123 b 12 ) e+ ( a 0 b 12 + a 1 b 2 − a 2 b 1 + a 3 b 123 + a 12 b 0 − a 13 b 23 + a 23 b 13 + a 123 b 3 ) e 1 e+ ( a 0 b 13 + a 1 b 3 − a 2 b 123 − a 3 b 1 + a 12 b 23 + a 13 b 0 − a 23 b 12 − a 123 b 2 ) e 1 e+ ( a 0 b 23 + a 1 b 123 + a 2 b 3 − a 3 b 2 − a 12 b 13 − a 13 b 12 + a 23 b 0 + a 123 b 1 ) e 2 e+ ( a 0 b 123 + a 1 b 23 − a 2 b 13 + a 3 b 12 + a 12 b 3 − a 13 b 2 + a 23 b 1 + a 123 b 0 )
e 1 e 2 e 3 .
A. T HE ELECTROMAGNETIC FIELD IN 3 DIMENSIONS
An intriguing example of the duality of multivectors is found when writing the expression of the electromagnetic field F in terms of an electric vector field E and a magnetic vector field B (Hestenes, 2003), such that
[Equation]
Both the electric field E and the magnetic field B are described by Maxwell’s equations (Griffiths, 2005).
The two fields are strongly coupled, e.g. temporal changes of electric fields induce magnetic fields and vice versa.
Probably the most illustrative co-occurence of electric and magnetic fields is when describing the propagation of light.
In standard vector algebra, E is a vector while
B is a pseudovector, i.e. the two kinds of fields are distinguished by a difference in sign under space inversion.
Equation 43 naturally decomposes the electromagnetic field into vector and bivector parts via the pseudoscalar i 3 .
For example, for the base component B x e 1 of B it holds that
B x e 1 i 3 =
B x e 1 e 1 e 2 e 3 =
B x e 2 e 3 , which is a bivector and the dual to the base component e 1 of E .
Geometric algebra reveals that a pseudovector is nothing else than a bivector represented by its dual, so the magnetic field B in Equation 43 is fully represented by the complete bivector Bi 3 , rather than B alone.
Consequently, the multivector representing F consists of three vectors (the electric field components) and three bivectors e 1 i 3 =
e 2 e 3 , e 2 i 3 =
e 3 e 1 , e 3 i 3 =
e 1 e 2 (the magnetic field components multiplied by i 3 ).
B C LIFFORD NEURAL LAYERS
This appendix supports Section 3 of the main paper.
Clifford convolutions are related to the work on complex networks by Trabelsi et al.
(2017), and closely related to work on quaternion neural networks (Zhu et al., 2018; Parcollet et al., 2018a;
Gaudet & Maida, 2018; Parcollet et al., 2018b; 2019; 2020; Nguyen et al., 2021).
Probably the most related work are (i) by Zang et al.
(2022) who build geometric algebra convolution networks to process spatial and temporal data, and (ii) Spellings (2021) who build rotation- and permutationequivariant graph network architectures based on geometric algebra products of node features.
Higher order information is built from available node inputs.
B. C LIFFORD CONVOLUTION LAYERS
We derive the implementation of translation equivariant Clifford convolution layers for multivectors in G 2 , i.e. multivectors of Clifford algebras generated by the 2-dimensional vector space R 2 .
Finally, we make the extension to Clifford algebras generated by the 3-dimensional vector space R 3 .
Regular CNN layers.
Regular convolutional neural network (CNN) layers take as input feature maps f : Z 2 → R c in and convolve 11 them with a set of c out filters { w i } c out i =1 : Z 2 → R c in :
[Equation]
[Equation] [Equation]
Equation 44 can be interpreted as inner product of the input feature maps with corresponding filters at every point y ∈
Z 2 .
By applying c out filters, the output feature maps can be interpreted as c out − dimensional features vectors at every point y ∈
Z 2 .
We now want to extend convolution layers such that the elementwise product of scalars f j ( y ) w i,j ( y − x ) are replaced by the geometric product of multivector inputs and multivector filters f j ( y ) w i,j ( y − x ) .
Clifford CNN layers.
We replace the feature maps f : Z 2 → R c in by multivector feature maps f : Z 2 → ( G 2 ) c in and convolve them with a set of c out multivector filters { w i } c out i =1 : Z 2 → ( G 2 ) c in :
[Equation]
B.1. T RANSLATION EQUIVARIANCE OF C LIFFORD CONVOLUTIONS
Theorem 1: Translation equivariance of Clifford convolutions
Let f : Z 2 → ( G 2 ) c in be a multivector feature map and let w : Z 2 → ( G 2 ) c in be a multivector kernel, then [[ L t f ] ⋆ w ] ( x ) =
[ L t [ f ⋆ w ]] ( x ) .
Proof.
[Equation]
x − t ))
ã e+ Å f 0 ( y )
w 2 ( y − ( x − t ))
+ f 1 ( y ) w 12 ( y − ( x − t ))
+ f 2 ( y ) w
( y − ( x − t ))
− f 12 ( y ) w 1 ( y − ( x − t ))
ã e+ Å f 0 ( y )
w 12 ( y − ( x − t ))
+ f 1 ( y ) w 2 ( y − ( x − t ))
− f 2 ( y ) w 1 ( y − ( x − t )) + f 12 ( y ) w
( y − ( x − t ))
ã e 1 e= [ L t [ f ⋆ w ]] ( x ) .
Implementation of Cl 2 , 0 ( R ) layers.
We can implement a 2D Clifford CNN layer using Equation 25 where { b 0 , b 1 , b 2 , b 12 } → { w i,j 0 , w i,j 1 , w i,j 2 , w i,j 12 } correspond to 4 different kernels representing one 2D multivector kernel, i.e. 4 different convolution layers, and { a 0 , a 1 , a 2 , a 12 } → { f j 0 , f j 1 , f j 2 , f j 12 } correspond to the scalar, vector and bivector parts of the input multivector field.
The channels of the different layers represent different stacks of scalars, vectors, and bivectors.
All kernels have the same number of input and output channels (number of input and output multivectors), and thus the channels mixing occurs for the different terms of Equations 25, 42 individually.
Lastly, usually not all parts of the multivectors are present in the input vector fields.
This can easily be accounted for by just omitting the respective parts of Equations 25, 42.
A similar reasoning applies to the output vector fields.
B.1. R OTATIONAL C LIFFORD CNN LAYERS
Here we introduce an alternative parameterization to the Clifford CNN layer introduced in Equation 9 by using the isomorphism of the Clifford algebra Cl 0 , 2 ( R ) to quaternions 12 .
We take advantage of the fact that a quaternion rotation can be realized by a matrix multiplication (Jia, 2008;
Kuipers, 1999; Schwichtenberg, 2015).
Using the isomorphism, we can represent the feature maps f j and filters w i,j as quaternions: f j = f j 0 + f j 1 ˆ ı + f j 2 ˆ ȷ + f j 3 ˆ k and w i,j = w i,j 0 + w i,j 1 ˆ ı + w i,j 2 ˆ ȷ + w i,j 3 ˆ k 13 .
Leveraging this quaternion representation, we can devise an alternative parameterization of the product between the feature map f j and w i,j .
To be more precise, we introduce a composite operation that results in a scalar quantity and a quaternion rotation, where the latter acts on the vector part of the quaternion f j and only produces nonzero expansion coefficients for the vector part of the quaternion output.
A quaternion rotation w i,j f j ( w i,j ) − 1 acts on the vector part ( ˆ ı, ˆ ȷ, ˆ k ) of f j , and can be algebraically manipulated into a vector-matrix operation
R i,j f j , where R i,j : H → H is built up from the elements of w i,j (Kuipers, 1999).
In other words, one can transform the vector part ( ˆ ı , ˆ ȷ , ˆ k ) of f j ∈ H via a rotation matrix R i,j that is built from the scalar and vector part ( 1 , ˆ ı , ˆ ȷ , ˆ k ) of w i,j ∈ H .
Altogether, a rotational multivector filter { w i rot } c out i =1 : Z 2 → ( G 2 ) c in acts on the feature map f j through a rotational transformation R i,j ( w i,j rot , 0 , w i,j rot , 1 , w i,j rot , 2 , w i,j rot , 12 ) acting on vector and bivector parts of the multivector feature map f : Z 2 → ( G 2 ) c in , and an additional scalar response of the multivector filters:
[Equation]
where (cid:2)
f j ( y ) w i,j rot ( y − x )) (cid:3)
= f
j 0 w i,j rot
[Equation]
f
[Equation]
w i,j rot , 1 − f j 2 w i,j rot , 2 − f j 12 w i,j rot , 12 , which is the scalar output of Equation 25.
The rotational matrix R i,j ( y − x ) in written out form reads:
[Equation]
[Table]
where ˆ w i,j rot ( y − x )
=
ˆ w i,j rot , 0 ( y − x ) + ˆ w i,j rot , 1 ( y − x ) e 1 + ˆ w i,j rot , 2 ( y − x ) e
+ ˆ w i,j rot , 12 ( y − x ) e 12 is the normalized filter with ∥
ˆ w i,j rot ∥ = 1 .
The dependency ( y − x ) is omitted inside the rotation matrix
R i,j for clarity.
B.1. 3D C LIFFORD CONVOLUTION LAYERS
Implementation of Cl 3 , 0 ( R ) layers.
Analogously to the 2-dimensional case, we can implement a 3D Clifford CNN layer using Equation 42, where { b 0 , b 1 , b 2 , b 12 , b 13 , b 23 , b 123 } correspond to 8 different kernels representing one 3D multivector kernel, i.e. 8 different convolution layers, and { a 0 , a 1 , a 2 , a 12 , a 13 , a 23 , a 123 } correspond to the scalar, vector, bivector, and trivector parts of the input multivector field.
B. C LIFFORD NORMALIZATION
Different normalization schemes have been proposed to stabilize and accelerate training deep neural networks (Ioffe & Szegedy, 2015; Ba et al., 2016; Wu & He, 2018; Ulyanov et al., 2017).
Their standard formulation applies only to real values.
Simply translating and scaling multivectors such that their mean is 0 and their variance is 1 is insufficient because it does not ensure equal variance across all components.
Batch normalization Trabelsi et al.
(2017) extended the batch normalization formulation to apply to complex values.
We build on the same principles to first propose an appropriate batch normalization scheme for multivectors, similar to the work of Gaudet & Maida (2018) for quaternions.
For 2D multivectors of the form a = a 0 +
[Equation]
e
[Equation]
+
[Equation]
+
[Equation]
12 e 1 e 2 , we can formulate the problem of batch normalization as that of whitening 4D vectors:
[Equation]
[Equation]
where the covariance matrix V is
[Equation]
[Equation]
The shift parameter β is a multivector with 4 learnable components and the scaling parameter γ is 4 × 4 positive semi-definite matrix.
The multivector batch normalization is defined as:
[Equation]
When the batch sizes are small, it can be more appropriate to use Group Normalization or Layer
Normalization.
These can be derived with appropriate application of Eq.
50 along appropriate tensor dimensions.
B. C LIFFORD INITIALIZATION
Parcollet et al.
(2018a)
; Gaudet & Maida (2018) introduced initialization schemes for quaternions which expands upon deep network initialization schemes proposed by Glorot & Bengio (2010);
He et al.
Similar to Clifford normalization, quaternion initialization schemes can be adapted to
Clifford layers in a straight forward way.
Effectively, tighter bounds are required for the uniform distribution form which Clifford weights are sampled.
However, despite intensive studies we did not observe any performance gains over default PyTorch initialization schemes 14 for 2-dimensional experiments.
Similar findings are reported in Hoffmann et al.
However, 3-dimensional implementations necessitate much smaller initialization values (factor 1 / 8 ).
B. E QUIVARIANCE UNDER ROTATIONS AND REFLECTIONS
Clifford convolutions satisfy the property of equivariance under translation of the multivector inputs, as shown in this
Appendix B. However, the current definition of Clifford convolutions is not equivariant under multivector rotations or reflections.
Here, we derive a general kernel constraint which allows us to build generalized Clifford convolutions which are equivariant w.r.t rotations or reflections of the multivectors.
That is, we like to prove equivariance of a Clifford layer under rotations and reflections (i.e. orthogonal transformations) if the multivector kernel multivector filters { w i } c out i =1 : Z 2 → ( G ) c in satisfies the constraint:
[Equation]
for 0 ≤ j < c in .
We first define an orthogonal transformation on a multivector by,
[Equation]
where u and f are multivectors which are multiplied using the geometric product.
The minus sign is picked by reflections but not by rotations, i.e. it depends on the parity of the transformation.
This construction is called a “versor” product.
The construction can be found in e.g. Suter (2003) for vectors and its extension to arbitrary multivectors.
The above construction makes it immediately clear that T ( fg ) =
( T f )( T g ) .
When we write Tx , we mean an orthogonal transformation of an
Euclidean vector (which can in principle also be defined using versors).
To show equivariance, we wish to prove for multivectors f : Z 2 → ( G ) c in and a set of c out multivector filters { w i } c out i =1 : Z 2 → ( G ) c in that:
[Equation] [Equation]
and
[Equation]
Equations 54, 55 yield:
[Equation]
That is: if the input multivector field transforms as a multivector, and the kernel satisfies the stated equivariance constraint, then the output multivector field also transforms properly as a multivector.
Note that T might act differently on the various components (scalars, vectors, pseudoscalars, pseudovectors) under rotations and/or reflections.
Now,
[Equation] [Equation]
where in the fourth line we transform variables y → y ′ , in the fifth line we use the invariance of the summation “measure” under T , in the sixth line we use the transformation property of f and equivariance for w i , in the seventh line we use the property of multivectors, and in the eighth line we use linearity of T .
B. C LIFFORD F OURIER LAYERS
We derive the implementation of Clifford Fourier layers for multivectors in G 2 and G 3 , i.e. multivectors of Clifford algebras generated by the 2-dimensional vector
Classical Fourier transform.
In arbitrary dimension n , the Fourier transform ˆ f ( ξ ) =
F{ f } ( ξ ) for a continuous n -dimensional complex-valued signal f ( x ) =
f
( x 1 , . . .
, x n ) : R n → C is defined as:
[Equation]
provided that the integral exists, where x and ξ are n -dimensional vectors and ⟨ x, ξ ⟩ is the contraction of x and ξ .
Usually, ⟨ x, ξ ⟩ is the inner product, and ξ is an element of the dual vector space R n⋆ .
The inversion theorem states the back-transform from the frequency domain into the spatial domain:
[Equation]
We can rewrite the Fourier transform of Equation 58 in coordinates:
[Equation]
Discrete/Fast Fourier transform.
The discrete counterpart of Equation 58 transforms an ndimensional complex signal f ( x ) =
f
( x 1 , . . .
, x n ) : R n → C at M 1 × . . .
× M n grid points into its complex Fourier modes via:
[Equation]
where ( ξ 1 , . .
. , ξ n ) ∈
Z M 1 . . .
× . . .
Z M n .
Fast Fourier transforms (FFTs) (Cooley & Tukey, 1965;
Van Loan, 1992) immensely accelerate the computation of the transformations of Equation 61 by factorizing the discrete Fourier transform matrix into a product of sparse (mostly zero) factors.
B.5. 2D C LIFFORD F OURIER TRANSFORM
Analogous to Equation 58, the Clifford Fourier transform (Ebling & Scheuermann, 2005; Hitzer, 2012) and the respective inverse transform for multivector valued functions f ( x ) : R 2 → G 2 and vectors x, ξ ∈ R 2 are defined as:
[Equation]
(63) provided that the integrals exist.
The differences to Equations 58 and 59 are that f ( x ) and ˆ f ( ξ ) represent multivector fields in the spatial and the frequency domain, respectively, and
that the pseudoscalar i 2 = e 1 e 2 is used in the exponent.
Inserting the definition of multivector fields, we can rewrite Equation 62 as:
[Equation]
We obtain a Clifford Fourier transform by applying two standard Fourier transforms for the dual pairs f 0 = f 0 ( x ) + f 12 ( x ) i 2 and f 1 = f 1 ( x ) + f 2 ( x ) i 2 , which both can be treated as a complexvalued signal f 0 , f 1 : R 2 → C .
Consequently, f ( x ) can be understood as an element of C 2 .
The 2D Clifford Fourier transform is the linear combination of two classical Fourier transforms.
The discretized versions of the spinor/vector part ( ˆ f s/v ) reads analogously to Equation 61:
[Equation] [Equation]
where again ( ξ 1 , ξ 2 ) ∈
Z M 1 × Z M n .
Similar to Fourier Neural Operators (FNOs) where weight tensors are applied pointwise in the Fourier space, we apply multivector weight tensors W ∈ ( G 2 ) c in × c out × ( ξ max 1 × ξ max 2 ) point-wise.
Fourier modes above cut-off frequencies ( ξ max 1 , ξ max 2 ) are set to zero.
In doing so, we modify the Clifford Fourier modes
[Equation]
via the geometric product.
The Clifford Fourier modes follow naturally when combining spinor and vector parts of Equation 64.
Analogously to FNOs, higher order modes are cut off.
Finally, the residual connections used in FNO layers is replaced by a multivector weight matrix realized as
Clifford convolution, ideally a Cl 2 , 0 ( R ) convolution layer.
A schematic sketch of a Clifford Fourier layer is shown in Figure 5b in the main paper.
B.5. 2D C LIFFORD CONVOLUTION THEOREM
In contrast to Ebling & Scheuermann (2005), we proof the 2D Clifford convolution theorem for multivector valued filters applied from the right, such that filter operations are consistent with Clifford convolution layers.
We first need to show that the Clifford kernel commutes with the spinor and anti-commutes with the vector part of multivectors.
We can write the product a e i 2 s for every scalar s ∈ R and multivector a ∈ G 2 as
[Equation]
[Equation]
For the basis of the spinor part, we obtain 1 i
=
i 2 1 , and for the basis of the vector part e 1 i 2 =
e 1 e 1 e 2 = − e 1 e 2 e 1 = − i 2 e 1 .
Thus, the Fourier kernel e − 2 πi 2 ⟨ x,ξ ⟩ commutes with the spinor part, and anti-commutes with the vector part of a .
We therefore have to proof the convolution theorem for the commuting spinor and the anti-commuting vector part of a .
Theorem 2: 2D Clifford convolution theorem.
Let the field f : R 2 → G 2 be multivector valued, the filter k s : R 2 → G 2 be spinor valued, and the filter k v : R 2 → G 2 be vector valued, and let F{ f } , F{ k s } , F{ k v } exist, then
F{ f ⋆ k s }
[Equation]
ξ )
=
F{ f } ( ξ ) · F † { k
[Equation]
where F † { k s } ( ξ ) =
F{ k s } ( − ξ ) and F † { k v } ( ξ ) =
F{ k v } ( − ξ ) .
Proof.
F
[Equation] [Equation]
ξ
[Equation]
(cid:124) (cid:123)(cid:122) (cid:125) F † { k v } ( ξ ) e 2 πi 2 ⟨ y,ξ ⟩ = e − 2 πi 2 ⟨ y,ξ ⟩ F{ k v } ( ξ ) , where − ξ → ξ (cid:35) dy =
[Equation]
πi
[Equation]
y,ξ
[Equation]
F{
[Equation]
v
[Equation]
[Equation]
B.5. 3D C LIFFORD F OURIER TRANSFORM
Analogous to Equation 58, the Clifford Fourier transform (Ebling & Scheuermann, 2005) and the respective inverse transform for multivector valued functions f : R 3 → G 3 and vectors x, ξ ∈ Rare defined as:
[Equation]
provided that the integrals exist.
A multivector valued function f : R 3 → G 3 ,
[Equation]
can be expressed via the pseudoscalar i 3 =
e 1 e 2 e 3 as:
[Equation]
We obtain a 3-dimensional Clifford Fourier transform by applying four standard Fourier transforms for the four dual pairs f 0 = f 0 ( x )+ f 123 ( x ) i 3 , f 1 = f 1 ( x )+ f 23 ( x ) i 3 , f 2 = f 2 ( x )+ f 31 ( x ) i 3 , and f 3 = f 3 ( x )+ f 12 ( x ) i 3 , which all can be treated as a complex-valued signal f 0 , f 1 , f 2 , f 3 : R 3 → C .
Consequently, f ( x ) can be understood as an element of C 4 .
The 3D Clifford Fourier transform is the linear combination of four classical Fourier transforms:
[Equation]
Analogous to the 2-dimensional Clifford Fourier transform, we apply multivector weight tensors W ∈ ( G 3 ) c in × c out × ( ξ max 1 × ξ max 2 × ξ max 3 ) point-wise.
Fourier modes above cut-off frequencies ( ξ max 1 , ξ max 2 , ξ max 3 ) are set to zero.
In doing so, we modify the Clifford Fourier modes ˆ f
[Equation]
via the geometric product.
The Clifford Fourier modes follow naturally when combining the four parts of Equation 74 .
Finally, the residual connections used in FNO layers is replaced by a multivector weight matrix realized as Clifford convolution, ideally a Cl 3 , 0 ( R ) convolution layer.
B.5. 3D C LIFFORD CONVOLUTION THEOREM
This theorem adapted from Ebling & Scheuermann (2005)).
First, again let’s check if the Clifford kernel commutes with the different parts of multivectors.
We can write the product a e i 3 s for every scalar s ∈ R and multivector a ∈ G 3 as
[Equation]
First, we check again if the different basis vectors of the Fourier transforms of Equation 74 commute with the pseudoscalar i 3 :
[Equation]
In contrast to the 2-dimensional Clifford Fourier transform, now all four parts of the multivector of
Equation 73 commute with i 3 .
Theorem 3: 3D Clifford convolution theorem.
Let the field f : R 3 → G 3 be multivector valued, the filter k a : R 3 → G 3 be multivector valued
[Equation]
and let F{ f } , F{ k a } exist
[Equation]
then
[Equation]
where F † { k a } ( ξ ) =
F{ k a } ( − ξ ) .
Proof.
[Equation]
B.5. I MPLEMENTATION OF C LIFFORD F OURIER LAYERS
We implement a 2D Clifford Fourier layer by applying two standard Fourier transforms on the dual pairs of Equation 13.
These dual pairs can be treated as complex valued inputs.
Similarly, we implement a 3D Clifford Fourier layer by applying four standard Fourier transforms on the dual pairs of e.g. Cl 3 , 0 (Equation 37 - Equation 40).
Since Clifford convolution theorems hold both for the vector and the spinor parts and for the four dual pairs for Cl 2 , 0 and Cl 3 , 0 , respectively, we multiply the modes in the Fourier space using the geometric product.
Finally, we apply an inverse
Fourier transformation and resemble the multivectors in the spatial domain.
B. P SEUDOCODE
Algorithm 1 sketches the implementation of a Clifford convolution, Algorithm 2 of a rotational
Clifford convolution, and Algorithm 3 of a Clifford Fourier layer.
 
W [0] W [1]
W [2] − W [3]
W [1] W [0] − W [3]
W [2]
W [2] W [3]
W [0] − W [1]
W [3] W [2] − W [1]
W [0] 
: return
1: function C LIFFORD K ERNEL 2D ROT ( W ) 2: sq 12 ← W [1] 2 + W [2]3: sq 13 ← W [1] 2 + W [3]4: sq 23 ← W [2] 2 + W [3]5: sumsq ← W [0] 2 + W [1] 2 + W [2] 2 + W [3] 2 + ϵ 6: rot 12 ← W [0] W [1] /sumsq 7: rot 13 ← W [0] W [2] /sumsq 8: rot 14 ← W [0] W [3] /sumsq 9: rot 23 ← W [1] W [2] /sumsq 10: rot 24 ← W [1] W [3] /sumsq 11: rot 34 ← W [2] W [3] /sumsq 12: kernel ←   
W [0] − W [1] − W [2] − W [3]
W [5]
W [4](1 .
− sq 23 )
W [4]( rot 23 − rot 14 ) W [4]( rot 24 + rot 13 )
W [5] W [4]( rot 23 + rot 14 )
W [4](1 .
− sq 13 )
W [4]( rot 34 − rot 12 )
W [5] W [4]( rot 24 − rot 13 ) W [4]( rot 34 + rot 12 )
W [4](1 .
− sq 12 ) 
  13: return kernel 14: function C LIFFORD C ONV ROT ( W , x ) 15: kernel ← C LIFFORD K ERNEL 2D ROT ( W ) 16: input ← VIEW AS REALVECTOR ( x ) 17: output ← C ONV (kernel, input) 18: return VIEW AS MULTIVECTOR (output)
1: function C LIFFORD S PECTRAL C ONV ( W , x , m 1 , m 2 ) 2: x f ← FFT2 ( x ) 3: x ∗ f ← ï x f [: m 1 , : m 2 ] x f [: m 1 , − m 2 :] x f [ − m 1 : , : m 2 ] x f [ − m 1 : , − m 2 :] ò 4: y ← x ∗ f @ W ▷ Matrix multiply 5: x ∗ ← IFFT2 ( y ) 6: return x ∗ 7: function C LIFFORD F OURIER L AYER ( W f , W c , x ) 8: y ← VIEW AS COMPLEXVECTOR ( x ) 9: y 1 ← C LIFFORD S PECTRAL C ONV ( W f , y ) 10: x 1 ← VIEW AS MULTIVECTOR ( y 1 ) 11: x 2 ← C LIFFORD C ONV ( W c , x ) 12: out ← ACTIVATION ( x 1 + x 2 ) 13: return out 14:
C E XPERIMENTS
This appendix supports Section 4 of the main paper.
C. L OSS FUNCTION AND METRICS
We report the summed MSE (SMSE) loss defined as:
[Equation]
where u is the target, ˆ u the model output, N fields comprises scalar fields as well as individual vector field components, and N y is the total number of spatial points.
Equation 79 is used for training with
N t = 1 , and further allows us to define four metrics:
• One-step loss where N t = 1 and N fields comprises all scalar and vector components.
• Vector loss where N t = 1 and N fields comprises only vector components.
• Scalar loss where N t = 1 and N fields comprises only the scalar field.
• Rollout loss where N t = 5 and N fields comprises all scalar and vector components.
For Maxwell’s equation, electric and magnetic loss are defined analogously to the vector and the scalar loss for Navier-Stokes and shallow water experiments.
C. M ODELS
We experiment with two architecture families: ResNet models (He et al., 2016) and Fourier Neural
Operators (FNOs) (Li et al., 2020).
All baseline models are fine-tuned for all individual experiments with respect to number of blocks, number of channels, number of modes (FNO), learning rates, normalization and initialization procedures, and activation functions.
The best models are reported, and for reported Clifford results each convolution layer is substituted with a Clifford convolution, each Fourier layer with a Clifford Fourier layer, each normalization with a Clifford normalization and each non-linearity with a Clifford non-linearity.
A Clifford non-linearity in this context is a the application of the corresponding default linearity to the different multivector components.
ResNet architectures.
For Navier-Stokes and shallow water experiments, we use ResNet architectures with 8 residual blocks, each consisting of two convolution layers with 3 × 3 kernels, shortcut connections, group normalization (Wu & He, 2018), and GeLU activation functions (Hendrycks &
Gimpel, 2016).
We further use two embedding and two output layers, i.e. the overall architectures could be classified as Res-20 networks.
In contrast to standard residual networks for image classification, we don’t use any down-projection techniques, e.g. convolution layers with strides larger than 1 or via pooling layers.
In contrast, the spatial resolution stays constant throughout the network.
We therefore also use the same number of hidden channels throughout the network, that is 128 channels per layer.
Overall this results in roughly 1.6 million parameters.
Increasing the number of residual blocks or the number of channels did not increase the performance significantly.
Clifford ResNet architectures.
For every ResNet-based experiment, we replaced the fine-tuned
ResNet architectures with two Clifford counterparts: each CNN layer is replaced with a (i) Clifford
CNN layer, and (ii) with a rotational Clifford CNN layer.
To keep the number of weights similar, instead of 128 channels the resulting architectures have 64 multivector channels, resulting again in roughly 1.6 million floating point parameters.
Additionally for both architectures, GeLU activation functions are replaced with Clifford GeLU activation functions, group normalization is replaced with Clifford group normalization.
Using Clifford initialization techniques did not improve results.
Fourier Neural Operator architectures.
For Navier-Stokes and shallow water experiments, we used 2-dimensional Fourier Neural Operators (FNOs) consisting of 8 FNO blocks, two embedding and two output layers.
Each FNO block comprised a convolution path with a 1 × 1 kernel and an
FFT path.
We used 16 Fourier modes (for x and y components) for point-wise weight multiplication, and overall use 128 hidden channels.
We used GeLU activation functions (Hendrycks & Gimpel, 2016).
Additional shortcut connections or normalization techniques, such as batchnorm or group, norm did not improve performance, neither did larger numbers of hidden channels, nor more FNO blocks.
Overall this resulted in roughly 140 million parameters for FNO based architectures.
For 3-dimensional Maxwell experiments, we used 3-dimensional Fourier Neural Operators (FNOs) consisting of 4 FNO blocks, two embedding and two output layers.
Each FNO block comprised a 3D convolution path with a 1 × 1 kernel and an FFT path.
We used 6 Fourier modes (for x , y , and z components) for point-wise weight multiplication, and overall used 96 hidden channels.
Interestingly, using more layers or more Fourier modes degraded performances.
Similar to the 2D experiments, we applied GeLU activation functions, and neither apply shortcut connections nor normalization techniques, such as batchnorm or groupnorms.
Overall this resulted in roughlymillion floating point parameters for FNO based architectures.
Clifford Fourier Neural Operator architectures.
For every FNO-based experiment, we replaced the fine-tuned FNO architectures with respective Clifford counterparts: each FNO layer is replaced by its Clifford counterpart.
To keep the number of weights similar, instead of 128 channels the resulting architectures have 48 multivector channels, resulting in roughly the same number of parameters.
Additionally, GeLU activation functions are replaced with Clifford GeLU activation functions.
Using Clifford initialization techniques did not improve results.
For 3-dimensional Maxwell experiments, we replaced each 3D Fourier transform layer with a 3D
Clifford Fourier layer and each 3D convolution with a respective Clifford convolution.
We also use 6 Fourier modes (for x , y , and z components) for point-wise weight multiplication, and overall used 32 hidden multivector channels, which results in roughly the same number of parameters ( millions).
In contrast to 2-dimensional implementations, Clifford initialization techniques proved important for 3-dimensional architectures.
Most notably, too large initial values of the weights of
Clifford convolution layers hindered gradient flows through the Clifford Fourier operations.
C. T RAINING AND MODEL
SELECTION .
We optimized models using the Adam optimizer (Kingma & Ba, 2014) with learning rates [10 − 4 , 2 · 10 − 4 , 5 · 10 − 4 ] for 50 epochs and minimized the summed mean squared error (SMSE) which is outlined in Equation 79.
We used cosine annealing as learning rate scheduler (Loshchilov & Hutter, 2016) with a linear warmup.
For baseline ResNet models, we optimized number of layers, number of channels, and normalization procedures.
We further tested different activation functions.
For baseline FNO models, we optimized number of layers, number of channels, and number of Fourier modes.
Larger numbers of layers or channels did not improve the performances for both ResNet and FNO models.
For the respective Clifford counterparts, we exchanged convolution and Fourier layers by Clifford convolution and Clifford Fourier layers.
We further used Clifford normalization schemes.
We decreased the number of layers to obtain similar numbers of parameters.
We could have optimized Clifford architectures slightly more by e.g. using different numbers of hidden layers than the baseline models did.
However, this would (i) slightly be against the argument of having “plug- and play” replace layers, and (ii) would have added quite some computational overhead.
Finally, we are quite confident that the used architectures are very close to the optimum for the current tasks.
Computational resources.
All FNO and CFNO experiments used 4 × 16 GB NVIDIA V100 machines for training.
All ResNet and Clifford ResNet experiments used 8 × 32 GB NVIDIA V machines.
Average training times varied between 3 h and 48 h , depending on task and number of trajectories.
Clifford runs on average took twice as long to train for equivalent architectures and epochs.
C. N AVIER -S TOKES IN 2D
The incompressible Navier-Stokes equations are built upon momentum and mass conservation of fluids.
Momentum conservation yields for the velocity flow field v
[Equation]
where v ·∇ v is the convection, µ ∇ 2 v the viscosity, ∇ p the internal pressure and f an external force.
Convection is the rate of change of a vector field along a vector field (in this case along itself), viscosity is the diffusion of a vector field, i.e. the net movement form higher valued regions to lower concentration regions, µ is the viscosity coefficient.
The incompressibility constrained yields mass conservation via
[Equation]
Additional to the velocity field v ( x ) , we introduce a scalar field s ( x ) representing a scalar quantity that is being transported through the velocity field.
For example, v might represent velocity of air inside a room, and s might represent concentration of smoke.
As the vector field changes, the scalar field is transported along it, i.e. the scalar field is advected by the vector field.
Similar to convection, advection is the transport of a scalar field along a vector field:
[Equation]
We implement the 2D Navier-Stokes equation using Φ Flow 15 (Holl et al., 2020).
Solutions are propagated where we solve for the pressure field and subtract its spatial gradients afterwards.
SemiLagrangian advection (convection) is used for v , and MacCormack advection for s .
Additionally, we express the external buoyancy force f in Equation 80 as force acting on the scalar field.
Solutions are obtained using Boussinesq approximation (Kleinstreuer, 1997), which ignores density differences except where they appear in terms multiplied by the acceleration due to gravity.
The essence of the Boussinesq approximation is that the difference in inertia is negligible but gravity is sufficiently strong to make the specific weight appreciably different between the two fluids.
Equation details.
We obtain data for the 2D Navier-Stokes equations on a grid with spatial resolution of 128 × 128 ( ∆ x = 0 .
25 , ∆ y = 0 . 25 ), and temporal resolution of ∆ t = 1 .
5 s .
The equation is solved on a closed domain with Dirichlet boundary conditions ( v = 0 ) for the velocity, and Neumann boundaries ∂s∂x = 0 for the scalar smoke field.
The viscosity parameter is set to ν = 0 . 01 , and a buoyancy factor of (0 , 0 . 5)
T is used.
The scalar field is initialized with random Gaussian noise fluctuations, and the velocity field is initialized to 0 .
We run the simulation for 21 s and sample every 1 . 5 s .
Trajectories contain scalar and vector fields at 14 different time points.
Results.
Results are summarized in Figures 10, 9, and detailed in Table 1.
Figure 11 displays examples of Navier-Stokes rollouts of scalar and vector fields obtained by Clifford Fourier surrogates, and contrasts them with ground truth trajectories.
For ResNet-like architectures, we observe that both CResNet and CResNet rot improve upon the ResNet baseline.
Additionally, we observe that rollout losses are also lower for the two Clifford based architectures, which we attribute to better and more stable models that do not overfit to one-step predictions so easily.
Lastly, while in principle CResNet and CResNet rot based architectures are equally flexible, CResNet rot ones in general perform better than CResNet ones.
For FNO and respective Clifford Fourier based (CFNO) architectures, the loss is in general much lower than for ResNet based architectures.
CFNO architectures improve upon FNO architectures for all dataset sizes, and for one-step as well as rollout losses.
[Figure]
[Figure] [Figure]
[Table]
P re d i c t i o n
T a r g e t
P re d i c t i o n
T a r g e t
[Figure]
P re d i c t i o n
T a r g e t
P re d i c t i o n
T a r g e t
[Figure]
C. S HALLOW WATER EQUATIONS .
The shallow water equations (Vreugdenhil, 1994) describe a thin layer of fluid of constant density in hydrostatic balance, bounded from below by the bottom topography and from above by a free surface.
For example, the deep water propagation of a tsunami can be described by the shallow water equations, and so can a simple weather model.
The shallow water equations read:
[Equation]
where v x is the velocity in the x -direction, or zonal velocity, v y is the velocity in the y -direction, or meridional velocity, g is the acceleration due to gravity, η ( x, y ) is the vertical displacement of free surface, and h ( x, y ) is the topography of the earth’s surface.
We modify the implementation in SpeedyWeather.jl 16 (Kl¨ower et al., 2022) to further randomize initial conditions to generate our dataset.
SpeedyWeather.jl combines the shallow water equations with spherical harmonics for the linear terms and Gaussian grid for the non-linear terms with the appropriate spectral transforms.
It internally uses a leapfrog time scheme with a Robert and William’s filter to dampen the computational modes and achieve 3rd oder accuracy.
SpeedyWeather.jl is based on the atmospheric general circulation model SPEEDY in Fortran (Molteni, 2003; Kucharski et al., 2013).
Equation details.
We obtain data for the 2D shallow water equations on a grid with spatial resolution of 192 × 96 ( ∆ x = 1 . 875 ◦ , ∆ y = 3 . 75 ◦ ), and temporal resolution of ∆ t = 6 h .
The equation is solved on a closed domain with periodic boundary conditions.
We rollout the simulation for 20 days and sample every 6 h .
Here 20 days is of course not the actual simulation time but rather the simulated time.
Trajectories contain scalar pressure and wind vector fields at 84 different time points.
Results.
Results are summarized in Figures 12, 13, 14, and detailed in Tables 2, 3.
Figuredisplays examples of shallow water equations rollouts of scalar pressure and vector wind fields obtained by Clifford Fourier surrogate models, and contrasts them with ground truth trajectories.
The predictions are fairly indistinguishable from ground truth trajectories.
We observe similar results than for the Navier-Stokes experiments.
However, performance differences between baseline and
Clifford architectures are even more pronounced, which we attribute to the stronger coupling of the scalar and the vector fields.
For ResNet-like architectures, CResNet and CResNet rot improve upon the ResNet baseline, rollout losses are much lower for the two Clifford based architectures, and CResNet rot based architectures in general perform better than CResNet based ones.
For Fourier based architectures, the loss is in general much lower than for ResNet based architectures (a training set size of 56 trajectories yields similar (C)FNO test set performance than a training set size oftrajectories for ResNet based architectures).
CFNO architectures improve upon FNO architectures for all dataset sizes, and for one-step as well as rollout losses, which is especially pronounced for low number of training trajectories.
[Figure]
[Figure] [Figure]
P re d i c t i o n
T a r g e t
P re d i c t i o n
T a r g e t
[Figure]
P re d i c t i o n
T a r g e t
P re d i c t i o n
T a r g e t
[Figure]
P re d i c t i o n
T a r g e t
P re d i c t i o n
T a r g e t
[Figure]
[Table] [Table]
C.
M
AXWELL ’ S
EQUATIONS IN MATTER
IN
3D.
Electromagnetic simulations play a critical role in understanding light–matter interaction and designing optical elements.
Neural networks have been already successful applied in inverse-designing photonic structures (Ma et al., 2021b;
Lim & Psaltis, 2022).
Maxwell’s equations in matter read:
[Equation]
In isotropic media, the displacement field D is related to the electrical field via D
= ϵ 0 ϵ r E , where ϵis the permittivity of free space and ϵ r is the permittivity of the media.
Similarly, the magnetization field H in isotropic media is related to the magnetic field B via H = µ 0 µ r B , where µ 0 is the permeability of free space and µ r is the permeability of the media.
Lastly, j is the electric current density and ρ the total electric charge density.
We propagate the solution of Maxwell’s equation in matter using a finite-difference time-domain method 17 , where the discretized Maxwell’s equations are solved in a leapfrog manner.
First, the electric field vector components in a volume of space are solved at a given instant in time.
Second, the magnetic field vector components in the same spatial volume are solved at the next instant in time.
Equation details.
We obtain data for the 3D Maxwell’s equations on a grid with spatial resolution of 32 × 32 × 32 ( ∆ x = ∆ y
=
∆ z = 5 · 10 − 7 m ), and temporal resolution of ∆ t = 50 s .
We randomly place 18 (6 in the x − y plane, 6 in the x − z plane, 6 in the y − z plane) different light sources outside a cube which emit light with different amplitude and different phase shifts, causing the resulting
D and H fields to interfere with each other.
The wavelength of the emitted light is 10 − 5 m , The equation is solved on a closed domain with periodic boundary conditions.
We run the simulation for 400 s and sample data every 50 s .
Trajectories contain displacement D and the magnetization field
H components.
Exemplary trajectories are shown in Figure 16.
D x :yz-plane
D y :yz-plane
D z :yz-plane − 2 − 11 2 · 10 −
H x :yz-plane
H y :yz-plane
H z :yz-plane − 6 − 4 − · 10 −
D x :yz-plane
D y :yz-plane
D z :yz-plane − 2 − 11 2 · 10 −
H x :yz-plane
H y :yz-plane
H z :yz-plane − 55 · 10 −
D x :yz-plane
D y :yz-plane
D z :yz-plane − 2 − · 10 −
H x :yz-plane
H y :yz-plane
H z :yz-plane − 55 · 10 −
Results.
Results are summarized in Figure 17 and detailed in Table 4.
[Figure]
[Figure]
[Table]
D R ELATED WORK
This appendix supports detailed discussions of how our work relates to complex and quaternion neural networks, to work on Clifford algebras and Clifford Fourier transforms in computer vision, to Fourier Neural Operators, equivariant neural networks and geometric deep learning approaches, to neural operator learning and neural PDE surrogates.
Clifford (geometric) algebra and Clifford Fourier transform.
(Real)
Clifford algebras (also known as geometric algebras), as an extension of elementary algebra to work with geometrical objects such as vectors are extensively discussed in Suter (2003);
Hestenes (2003);
Dorst et al.
Hestenes (2012); Renaud (2020).
Compared to other formalisms for manipulating geometric objects, Clifford algebras are tailored towards vector manipulation of objects of different dimensions.
Hypercomplex and quaternion Fourier transforms are extensively discussed in Ell (1992; 1993);
Ell & Sangwine (2006); Ell et al.
This work heavily builds on the concepts introduced in Ebling & Scheuermann (2003; 2005); Ebling (2006).
Comprehensive summaries of Clifford and quaternion
Fourier transforms can be found in Hitzer & Sangwine (2013);
Brackx et al.
; Hitzer (2021).
Clifford algebras and Clifford Fourier transforms are already deployed to solve PDEs numerically in Alfarraj & Wei (2022).
More precisely, the Clifford-Fourier transform is used to solve the mode decomposition process in PDE transforms.
Clifford neural networks.
Neural networks in the Clifford domain were proposed already inby Pearson & Bisset (1994), and later by Pearson (2003).
These works put the emphasis on the geometric perceptron (Melnyk et al., 2021), i.e. how to recast vanilla multilayer perceptrons (MLPs) as
Clifford MLPs.
Similarly, (Hoffmann et al., 2020) generalized from complex numbers and quaternions to a set of alternative algebras.
Besides Clifford MLPs, Clifford algebras have been used in recurrent neural networks (RNNs) (Kuroe, 2011), and have been used to formulate quantum neural networks (Trindade et al., 2022).
Their applicability to neural computing has been studied in general (Buchholz & Sommer, 2001; Buchholz, 2005), exploring global exponential stabilities of
Clifford MLPs with time-varying delays and impulsive effects.
Probably the most related wors are: (i
Zang et
al.
) who build geometric algebra convolution networks to process spatial and temporal data of 3D traffic data.
Multidimensional traffic parameters are encoded as multivectors which allows to model correlation between traffic data in both spatial and temporal domains.
(ii) Spellings (2021) who build rotation- and permutation-equivariant graph network architectures based on geometric algebra products of node features.
Higher order information is built from available node inputs.
In contrast to previous works, we are the first to introduce the multivector viewpoint of field components which allows us to effectively connect Clifford neural layers with the geometric structure of the input data.
We further connect neural Clifford convolutions on multivectors with various works on complex numbers and quaternions.
We are further the first to introduce neural Clifford Fourier transforms.
Complex and quaternion neural networks.
Trabelsi et al.
(2017) introduced the key components for complex-valued deep neural networks.
More precisely, they introduced convolutional (LeCun et al., 1998) feed-forward and convolutional LSTM (Shi et al., 2015; Hochreiter & Schmidhuber, 1997) networks, together with complex batch-normalization, and complex weight initialization strategies.
Quaternions are a natural extension of complex neural networks.
Already in classical computer vision, quaternions as hypercomplex convolution (Sangwine & Ell, 2000) and hypercomplex correlation (Moxey et al., 2003) techniques were introduced for color image processing.
Quaternion based deep learning architectures are a natural extension of complex neural networks.
In quaternion neural networks (Zhu et al., 2018;
Parcollet et al., 2018a; Gaudet & Maida, 2018;
Parcollet et al., 2018b; 2019; 2020; Nguyen et al., 2021), concepts such as complex convolution, complex batchnorm, and complex initialization are transfered from the complex numbers C , which are algebra-isomorph to Cl (0 , 1)( R ) to Cl (0 , 2)( R ) , which is algebra-isomorph to the quaternions H .
Although Hoffmann et al.
(2020) generalized these from complex numbers and quaternions to a set of alternative algebras, their tasks did not really leverage any multivector structure in data.
Fourier Neural Operators.
Fourier Neural Operators (FNOs) (Li et al., 2020) have had tremendous impact towards improving neural PDE solver surrogates.
Efficient implementations of FNO layers come as physics-informed neural networks (PINO) (Li et al., 2021b), as U-shaped network architectures (UNO) (Rahman et al., 2022b), as spectral surrogate for vision transformer architectures (Rao et al., 2021;
Guibas et al., 2021), as Markov neural operators (MNO) for chaotic systems (MNO) (Li et al., 2021a), and as generative adversarial neural operators (GANOs) (Rahman et al., 2022a).
Applications range from weather forecasting (Pathak et al., 2022), CO 2 -water multiphase problems (Wen et al., 2022), multiscale method for crystal plasticity (Liu et al., 2022), seismic wave propagation (Yang et al., 2021), photoaccustic wave propagation (Guan et al., 2021),
PDE-constrained control problems (Hwang et al., 2022), and for thermochemical curing of composites (Chen et al., 2021).
Recently, FNOs have been successfully applied to PDEs on general geometries (Li et al., 2022b).
Furthermore, universal approximation and error bounds have been studied for FNOs (Kovachki et al., 2021).
Neural PDE solvers/surrogates.
The intersection of PDE solving, deep learning, fluid dynamics, and weather forecasting has developed into a very active hub of research lately (Thuerey et al., 2021).
We roughly group recent approaches to learn neural PDE surrogates and neural PDE solvers into three categories:
(i) hybrid approaches , where neural networks augment numerical solvers or replace parts of numerical solvers; (ii) direct approaches , (a) where the mapping from an initial state to a solution is learned, i.e. the solution function of the underlying PDE is approximated; (b) where the mapping from an initial state to a final state of an underlying PDE, i.e. the solution operator is learned, ideally as mapping between function spaces to be able to generalize across
e.g. parameters.
Ad (i): Neural networks augment numerical solvers by learning data-driven discretizations for
PDEs (Bar-Sinai et al., 2019) or by controlling learned approximations inside the calculation of standard numerical solver used for computational fluid dynamics (Kochkov et al., 2021).
In Greenfeld et al.
(2019), a prolongation is learned which maps from discretized PDE solutions to multigrid solutions, Hsieh et al.
(2019) learn to modify the updates of an existing solver, Praditia et al.
(2021) adopt the numerical structure of Finite Volume Methods (FVMs), and Um et al.
(2020) learn a correction function of conventional PDE solvers to improve accuracy.
All these approaches are hybrid approaches (Garcia Satorras et al., 2019), where the computational graph of the solver is preserved and heuristically-chosen parameters are predicted with a neural network.
A different flavor of hybrid approaches can be assigned to the works of Sanchez-Gonzalez et al.
; Pfaff et al.
Mayr et al.
(2021) who predict accelerations of particles/meshes to numerical update the respective positions.
Finally, PauliNet (Hermann et al., 2020) and FermiNet (Pfau et al., 2020) approximate wave-functions of many-electron systems, and thus replace the hand-crafted ansatz which is conventionally used in variational quantum Monte Carlo methods.
Ad (ii.a): Sirignano & Spiliopoulos (2018); Han et al.
(2018) approximate the solution of high-dimensional Black-Scholes and Hamilton-Jacobi-Bellman equations, respectively.
Physicsinformed neural networks (PINNs) (Raissi et al., 2019) embed the underlying physics in the training process, and can be used to solve both forward (Jin et al., 2021) as well as backward (Raissi et al., 2020) dynamics.
Zubov et al.
(2021) allow automating many of these aspects under a single coherent interface.
Ad (ii.b): Guo et al.
(2016) learned a surrogate CNN-based model to approximate steady-state flow field predictions, similarly Bhatnagar et al.
(2019) trained a surrogate CNN-based model to predict solutions for unseen flow conditions and geometries, and Zhu & Zabaras (2018) used Baysian CNNs for surrogate PDE modeling and uncertainty quantification.
Fourier Neural Operators (FNOs) (Li et al., 2020) proposed the mapping from parameter space to solution spaces, and had tremendous impact towards improving neural PDE solver surrogates.
In parallel, Lu et al.
(2021) introduced
DeepONet, which learns mappings between function spaces, and was successfully applied to many parametric ODEs and PDEs.
Both, FNOs and DeepONets have been combined with PINNs and trained in a physics-informed style (Li et al., 2022b; Wang et al., 2021).
A comprehensive comparison of these two neural operator approaches is done by Lu et al.
Other directions include the modeling of PDE solution operators via latent space models, transformers, and graph neural networks (GNNs).
Wu et al.
(2022) present the modeling of the systems dynamics in a latent space with fixed dimension where the latent modeling is done via MLPs, and the encoding and decoding via CNNs, which can also be replaced by graph neural networks (GNNs).
Li et al.
(2022a) proposed a transformer based neural operator approach, which provides a flexible way to implicitly exploit the patterns within inputs.
Brandstetter et al.
(2022b) formulated a message passing neural network approach that representationally contains several conventional numerical PDE solving schemes.
Further GNN based approaches are L¨otzsch et al.
(2022) who learn the operator for boundary value problems on finite element method (FEM)
(Brenner et al., 2008) ground truth data, and Lienen & G¨unnemann (2022) who derive their GNN models from FEM in a principled way.
A practical use case for neural PDE surrogates is replacing expensive classical PDE solvers.
There is however a major chicken-and-egg problem here (Brandstetter et al., 2022a; Shi et al., 2022): obtaining high quality ground truth training data for neural PDE surrogates often requires using these expensive solvers.
Minimizing this data requirement is beginning to be approached in recent works.
Geneva & Zabaras (2020); Wandel et al.
(2020; 2022) achieve “data-free” training in various settings.
“Data-free” refers to the self-supervised training steps, which are done without ground truth data.
The current state-of-art generic approach is introduced in Shi et al.
(2022) as the mean squared residual (MSR) loss constructed by the discretized PDE itself.
However, for e.g. generating realistic initial conditions numerical solvers are still needed.
Pestourie et al.
(2021) identify how incorporating limited physical knowledge in the form of a low-fidelity “coarse” solver can allow training PDE surrogate models with an order of magnitude less data.
Another direction to improve data efficiency is by exploiting the Lie point symmetries of the underlying PDEs, either via data augmentation (Brandstetter et al., 2022a) or by building equivariant PDE surrogates (Wang et al., 2020b).
Our current work in a way also improves data efficiency by capturing the inductive bias appropriate for multivector fields.
Overall we believe hybrids of such approaches are going to be necessary for making neural PDE surrogates of practical use in many domains.
Neural PDE surrogates for fluid flow and weather forecasting applications are gaining momentum.
In weather forecasting, Pathak et al.
(2022) introduced FourCastNet as high-resolution weather modeling built on Adaptive Fourier Neural Operators (Guibas et al., 2021), Keisler (2022) successfully applied a graph neural network based approach to weather forecasting, Rasp & Thuerey (2021) achieved data-driven medium-range weather prediction with a ResNet which was pretrained on climate simulations, Weyn et al.
(2020) use CNNs on a cubed sphere for global weather prediction,
Weyn et al.
(2021) forecast weather sub-seasonally with a large ensemble of deep-learning weather prediction models, Arcomano et al.
(2020) build a reservoir computing-based, low-resolution, global prediction model, and MetNet (Sønderby et al., 2020) takes as input radar and satellite data to forecast probabilistic precipitation maps.
Finally, data assimilation is improved by deep learning techniques in Frerix et al.
(2021) and Maulik et al.
Similarly, in fluid dynamics, Ma et al.
(2021a) applied U-Nets (Ronneberger et al., 2015) to achieve physics-driven learning of steady
Navier-Stokes equations, Stachenfeld et al.
(2021) learned coarse models for turbulence simulations,
TF-Net (Wang et al., 2020a) introduced domain-specific variations of U-Nets along with trainable spectral filters in a coupled model of Reynolds-averaged Navier-Stokes and Large Eddy Simulation.
This exhaustive list of neural PDE solver surrogates shows that many of the architectures are based on convolutional or Fourier layers.
For these two, Clifford layers are applicable as a drop-in replacement in almost all cases.
For graph neural network and attention based architectures, we leave the implementation of respective Clifford counterparts to future work.
Geometric deep learning.
The core idea of geometric deep learning (Bronstein et al., 2017; 2021) is to exploit underlying low-dimensionality and structure of the physical world, in order to design deep learning models which can better learn in high dimensional spaces.
Incorporating underlying symmetries would be one way to achieve this.
If done correctly, it can drastically shrink the search space, which has proven to be quite successful in multiple scenarios.
The most obvious examples are CNNs (Fukushima & Miyake, 1982; LeCun et al., 1998), where the convolution operation commutes with the shift operator, and thus provides a way to equip layers and subsequently networks with translation equivariant operations.
Group convolution networks (Cohen & Welling, 2016a;
Kondor & Trivedi, 2018;
Cohen et al., 2019) generalize equivariant layers beyond translations, i.e. provide a concept of how to build general layers that are equivariant to a broader range of groups, such as rotation groups.
An appealing way of how to build such group equivariant layers is via so-called steerable basis functions (Hel-Or & Teo, 1998), which allow to write transformation by specific groups as a linear combination of a fixed, finite set of basis functions.
This concept leads to steerable group convolution approaches (Cohen & Welling, 2016b; Worrall et al., 2017).
Two concrete examples are: (i) circular harmonics, which are respective basis functions for building layers that are equivariant to the group SO( 2 ), the rotation group in 2 dimensions (Worrall et al., 2017;
Weiler & Cesa, 2019); (ii) spherical harmonics, which are respective basis functions for building layers that are equivariant to the group SO( 3 ), the rotation group in 3 dimensions (Weiler et al., 2018; Geiger & Smidt, 2022; Brandstetter et al., 2021).
The similarity to multivector fields becomes more obvious if we have a closer look at spherical harmonics, which are defined as homogeneous polynomials of degree l , where the l = 0 case corresponds to scalars, the l = 1 case to vectors, and l ≥ 2 to higher order objects.
Finally, Jenner & Weiler (2021) built steerable PDE operators such as curl or divergence as equivariant neural network components.
Grouped convolution.
In their seminal work, Krizhevsky et al.
(2012) introduced filter grouping, which allowed them to reduce the parameters in CNNs.
The respective grouped convolutions (not to be confused with group convolutions) divide the filter maps at channel dimension, as the channel dimension most of the time increases strongly for deeper layers, and thus dominates the parameter count.
Subsequent work showed that it is beneficial to additionally shuffle the channels for each filter group (Zhang et al., 2018), and to adaptively recalibrate channel-wise feature responses (Hu et al., 2018).
All these approaches can be seen in the wider spectrum of effective model scaling (Tan & Le, 2019; Sandler et al., 2018).
Clifford convolutions in contrast do not have groupings in the channel dimensions, but instead group together elements as multivectors.
In Clifford convolution, the Clifford kernel is therefore a constrained object where weight blocks appear multiple times (due to the nature of the geometric product).
Thus, Clifford convolutions are more parameter efficient than standard convolutions, and all tricks of effective model scaling could in principle be applied on top of Clifford convolutions.
Findings from Hoffmann et al.
(2020) with respect to higher compute density of alternative algebras are applicable to our work as well.
E G LOSSARY
This short appendix summmarizes notations used throughout the paper (Table 5)
[Table]
and contrasts the most fundamental concepts which arise when using Clifford algebras.
[Table]
Geometric, Exterior
[Table]
and Clifford algebras.
A geometric algebra is a Clifford algebra of the real numbers.
Since we are only using Cl 2 , 0 ( R ) , Cl 0 , 2 ( R )
[Table]
and Cl 3 , 0 ( R ) , we are effectively working with geometric algebras.
The exterior or Grassmann algebra is built up from the same concepts of scalars, vectors, bivectors, ..., k -vectors, but only exterior (wedge) products exist.
Therefore, the exterior algebra has a zero quadratic form (all base vectors square to zero).
Clifford algebras are a generalization thereof with nonzero quadratic forms.
Complex numbers, quaternions, hypercomplex numbers.
Hypercomplex numbers are elements of finite-dimensional algebras over the real numbers that are unital, i.e. contain a multiplicative identity element, but not necessarily associative or commutative.
Elements are generated for a basis { ˆ ı, ˆ ȷ, . . .
} such that ˆ ı 2 , ˆ ȷ 2 , . . .
∈ {− 1 , 0 , 1 } .
Complex numbers, quaternions, octonions are all hypercomplex numbers which can be characterized by different Clifford algebras.
The bivector, trivector (and higher objects) of the Clifford algebras directly translate into basis elements of the respective algebras.
For example, quaternions (which are of the form a + b ˆ ı + c ˆ ȷ + d ˆ k , where ˆ ı 2 = ˆ ȷ 2 = ˆ k 2 = − 1 ) are isomorphic to the Clifford algebra Cl 0 , 2 ( R ) where the basis element e 1 , e 2 , and e 1 e 2 directly translate to ˆ ı , ˆ ȷ , ˆ k .
Spinor.
Spinors arise naturally in discussions of the Lorentz group, the group to describe transformations in special relativity.
One could say that a spinor is the most basic sort of mathematical object that can be Lorentz-transformed.
In its essence, a spinor is a complex two-component vectorlike quantity in which rotations and Lorentz boosts (relativistic translations) are built into the overall formalism.
More generally, spinors are elements of complex vector spaces that can be associated with Euclidean vector spaces.
However, unlike vectors, spinors transform to their negative when the space is rotated by 360 ◦ .
In this work, the subalgebra Cl 0 (2 , 0)( R ) , spanned by even-graded basis elements of Cl 2 , 0 ( R ) , i.e. 1 and e 1 e 2 , determines the space of spinors via linear combinations ofand e 1 e 2 .
It is thus isomorphic to the field of complex numbers C .
Most notably, spinors of Cl 2 , 0 ( R ) commute with the Fourier kernel, whereas vectors do not.
For a detailed introduction to spinors we recommend Steane (2013), and the comprehensive physics book of Schwichtenberg (2015).
Pseudoscalar.
A pseudoscalar – unlike a scalar – changes sign when you invert the coordinate axis.
The easiest example of a pseudoscalar is the scalar triplet product of three arbitrary vectors of an Euclidean vector space x, y, z ∈ R n with inner product ⟨ ., .
The scalar triplet product becomes negative for any parity inversion, i.e. swapping any two of the three operands: x · ( y × z )
= − x · ( z × y ) = − y · ( x × z ) =
− z · ( y × x ) .
Scalar field, vector field.
A field is any (physical) quantity which takes on different values at different points in space (space-time).
A scalar field is map D
→ R , where D ⊆ R n .
A vector field is map D
→ R n , where D ⊆ R n .
For example, n = 2 results in a vector field in plane, and n = 3 in a vector field in space.
For an interesting history of the evolution of the concept of fields in physics we recommend Mirowski (1991);
McMullin (2002).
In Table 6, we list various important vector and scalar fields for comparison.
[Table]
along its streamline 1 ( v )
Vector
R 2 → R 2 , R 3 → R
Flow speed field
Length of flow velocity vector ( | v | )
Scalar
R 2 → R , R 3 → R
Wind velocity
field
Air flow velocity field ( v )
Vector
R 2 → R 2 , R 3 → R
Temperature field
Temperature at space point ( K )
Scalar
R 3 → R
Signed distance field (SDF)
Signed distance
Scalar
R 3 → R
Occupancy field
Occupancy
Scalar
R 3 → R