Equivariant Networks for Crystal Structures
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Supervised learning with deep models has tremendous potential for applications inmaterials science.
Recently, graph neural networks have been used in this context,drawing direct inspiration from models for molecules.
However, materials aretypically much more structured than molecules, which is a feature that these modelsdo not leverage.
In this work, we introduce a class of models that are equivariantwith respect to crystalline symmetry groups.
We do this by deﬁning a generalizationof the message passing operations that can be used with more general permutationgroups, or that can alternatively be seen as deﬁning an expressive convolutionoperation on the crystal graph.
Empirically, these models achieve competitiveresults with state-of-the-art on the Materials Project dataset.
Introduction
Deep learning has seen remarkable applications in computational chemistry, both on the side ofmolecular property prediction and molecule generation.
For small molecules, these methods areclose to a level of precision that would make them suitable for practical applications [48].
However,less attention has been put to the problem of designing neural network architectures for the largerassemblies of atoms that constitute materials.
This is a crucial problem, as deep learning couldadvantageously complement computationally demanding ab initio simulation methods like DFT[39, 28].
Random forests and kernel methods have already yielded promising results for concreteapplications in materials science, thanks to a large amount of materials data now available.
Examplesinclude determination of stable structures [76], photovoltaic cells design [72, 35] and discovery ofmagnetic and superconducting materials [42, 62, 17].
In this paper, we address the problem of designing equivariant layers for supervised learning onmaterials.
We focus on crystals, materials characterized by the ordered arrangement of their atoms inlattices.
Crystalline materials are largely present around us and essential in technological applications,as they include a large number of metals, ceramics, and salts, among others [3].
They can bedescribed as the regular repetition of a set of atoms, called the unit cell , in all directions of space.
These patterns are analogous to wallpaper patterns, for which the repeated structure is an image.
Crystals are characterized by a high degree of symmetry which is fundamental in understanding theirphysical properties.
Consequently, we suggest that equivariant deep learning provides an appropriateframework to design function approximators for crystals.
Graph Neural Networks (GNNs) have recently been proposed for supervised prediction tasks oncrystals [56, 74, 12].
We hypothesize here that these models may fall short in exploiting crystalsymmetry and that structural information is lost when mapping a crystal structure to a graph.
Inparticular, GNNs are equivariant to the permutation of atoms given as input to the model.
Thiscondition is overly restrictive and amounts to forgetting about the ordered nature of a crystal.
Wepropose to use models equivariant to a product of groups G Λ × S C , where G Λ acts at the level of the
Bravais lattice , the underlying periodic grid of the crystal, and S C on the unit cells.
We show that ourproposed equivariant architecture is more expressive than GNNs on this data structure
By using the crystal structure, our approach amounts to deﬁning a group-equivariant convolutionkernel on the crystal in a way that is completely analogous to convolutional neural networks (CNNs).
This convolution is deﬁned on a graph associated with a crystal structure.
Our contributions are thefollowing: 1) We derive different equivariant models based on the group-theoretical properties ofcrystals.
We show some of the limitations of GNNs for crystal data and propose an alternativedata structure to be used with our architecture.
We perform a rigorous analysis, cleaning, andprocessing of the Materials Project database [33] and share the resulting processed dataset to serveas a benchmark for materials applications.
We perform experimental tests of our models on the
Materials Project database and report results comparable to or better than baselines.
Related works
Equivariant neural networks It is well known that neural networks have to incorporate inductivebiases to be useful in practice [70, 7].
Using models that are invariant or equivariant to the symmetryof the data has proven to be a particularly important inductive bias to promote generalization [8].
Theﬁrst notable application of this idea was the CNN, for which each convolution layer is equivariant totranslation [44].
An alternative parameter-sharing view also appears in early works [59]; while morerecent equivariant networks have used both convolutional [13, 41, 15, 20], and parameter-sharing view[53, 25].
A few notable symmetries considered in recent years are rotation in image and volumetricdata [13, 71, 68], permutation symmetry in sets [77, 52] and graphs [40, 46], as well as Euclidean[63, 10], and rotational symmetry [14, 2, 22, 57, 24].
Deep learning for materials The increasing availability of large materials datasets from highthroughput calculations [19, 33, 37, 49, 31, 11], makes deep learning more and more relevantfor materials science.
Following the successes of GNNs on molecular data, similar models havebeen proposed for materials as alternatives to methods based on feature engineering.
Note that inwhat follows, we refer to GNNs in a general sense that includes message passing neural networks(MPNNs)
[27].Many variants of GNNs exist [36, 65, 75, 46], the underlying idea being for each nodeto aggregate features of neighboring nodes in a permutation invariant, or in the case of [40] in anequivariant way.
The CGCNN [74] and MEGNet models [12] rely on mapping crystal structures tographs and applying GNNs to obtain a prediction.
For the SchNet model, the correspondence withgraphs is less explicit [56], but still present.
Other approaches combine permutation equivariancewith E(3)-equivariance to design models for molecules and materials [5, 55].
Background on crystal symmetry
We ﬁrst start by introducing some principles of crystallography that will be used to derive our mainresults.
The exposition presented in the present work is inspired by [61].
A comprehensive treatmentcan also be found in [21], whereas basics of group theory are covered in [54] for example.
Lattices A crystal can be described as the periodic and inﬁnite repetition of a pattern in all directionsof space.
Crystals are conveniently described using lattices as their underlying structures.
An n -dimensional lattice Λ can be deﬁned as the set of integral combinations of the linearly independent lattice basis vectors a i ∈ R n :
[Equation]
The lattice is entirely speciﬁed by its basis vectors a i .
A lattice is associated with a group oftranslations T Λ for which the multiplication rule is addition.
This captures the translational symmetryof a crystal.
A lattice Λ also deﬁnes subsets of R n called unit cells .
These subsets have the propertyof tilling the space when translated by lattice vectors.
Of particular importance is the primitive cell Ufor the basis a i :
[Equation]
In a material, the unit cell comprises a set of atomic positions S
= { ( Z i , x i )
| x i ∈ U } , where theinteger Z i is the atomic number and x i the position of the atom.
S can contain an arbitrary number of atoms and must not possess a particular structure.
Together with the lattice Λ , the atomic positionsprovides a complete description of the crystal structure.
It is often useful to deﬁne the concept of a sublattice .
A sublattice Λ P of Λ is a lattice with basisvectors b i such that Λ P ⊂ Λ .
Correspondingly, the supercell C P is the unit cell associated withthe sublattice Λ P , for which C P ⊃ U .
The full lattice is generated by translations of the sublatticeby a set of centring vectors { 0 . . .
v s } .
More formally, the sublattice is associated with a normalsubgroup T Λ P of lattice translations, which speciﬁes a coset decomposition of the original lattice
T Λ = ( 0 + T Λ P ) ∪ · · · ∪ ( v s + T Λ P ) .
The centring vectors are coset representatives with respect tothat decomposition.
It is clear that the centering vectors are also the set of lattice points contained inthe supercell C P .
Example 3.1 (Graphene) In graphene, carbon atoms are arranged in a two-dimensional crystal with honeycomb structure.
The underlying lattice Λ has basis vectors a 1 = a 2 (cid:2) 3 √ 3 (cid:3) and a 2 =
[Equation]
(cid:2) 3 −√ 3 (cid:3) , where a is a lattice constant.
The set of atoms in the unit cell is S
=
(cid:8)(cid:0)
[Equation]
In Fig.1, blue dots represent carbon atoms.
Black and white dots identify points of lattice Λ with unit cell U .
Black points identify a sublattice Λ P deﬁned by basis vectors b 1 and b 2 and with supercell U P .
The centring vectors { 0 , a 1 } correspond to lattice points contained in the supercell.
Space groups Symmetry plays a major in their description of crystals.
This symmetry is oftendirectly visible through facets in naturally occurring crystals.
Mathematically, it is described by a [Figure] space group G , the set of isometries that maps a crystal structure to itself.
As isometries, spacegroups are subgroups of the Euclidean group E ( n ) .
A space group element can be described asa tuple ( W , t ) , where W is the linear part of the transformation and t a translation.
An elementmaps a vector x ∈ R n to Wx + t .
The multiplication of space group elements is thereforegiven by ( W 1 , t 1 ) ( W 2 , t 2 ) =
( W 1 W 2 , W 2 t 1 + t 2 ) .
The point group P of a space group is thegroup obtained from linear part operations in G , which will in general be rotations and reﬂections.
Considering only elements in G for which the linear part is identity ( I , t ) , we obtain the translationsubgroup of the space group T .
It is a normal subgroup, which allows deﬁning the factor group G/Tisomorphic to the point group P .
The crystallographic restriction theorem guarantees that only certain ﬁnite groups are valid pointgroups of space groups [18].
In particular, in 2 and 3 dimensions, only n -fold rotations withn ∈ { 2 , 3 , 4 , 6 } are allowed.
Two space groups G and G (cid:48) are said to belong to be of same type if theycan be related by a change of coordinate system.
There are 17 space group types in 2 dimensions and230 in 3 dimensions [61].
For a lattice Λ , we call the Bravais group of the lattice P Λ the set of linearisometries that map Λ to itself.
Bravais groups provide a way to classify lattices according to theirsymmetry.
We say that two lattices Λ and Λ (cid:48) belong to the same Bravais type if their Bravais groupsare the same matrix groups when written for suitable basis vectors of each lattice.
The 5 Bravaistypes in 2 dimensions and the 14 in 3 dimensions are enumerated in the Appendix.
The full symmetrygroup G Λ of lattice a Λ is given by the semidirect product of its translation group with its Bravaisgroup : G Λ = T Λ (cid:111) P Λ .
Consider the space group G a crystal structure with lattice Λ and unit cell U .
From the deﬁnition of acrystal structure, it is clear that the translation subgroup of G will be T Λ .
However, the point groupof G will, in general, not be P = P Λ .
This is because the unit cell may have less symmetry than theunderlying lattice, which will result in P ⊆ P Λ (see example 3.2).
This is the reason why the numberof space groups is much larger than the number of Bravais lattices.
Example 3.2 (Wallpaper pattern)
The wallpaper in Figure 2 is described by a square Bravais lattice Λ and unit cell U with 4-fold rotational symmetry.
The square lattice has a symmetry group
P 4 m , with D 4 Bravais group.
However, the unit cell reduces the symmetry of the overall pattern since it does not have reﬂection axes.
The symmetry group of the wallpaper is, therefore, P 4 , with point group C 4 .
Limitations of GNNs
In this section, we examine some of the limitations of GNNs on crystalline data, motivating our modelby addressing these limitations.
Typical models for crystalline data [74, 12] will build a graph fromatoms in the unit cell and assign a feature vector h i ∈ R n to each.
Edge features encoding distancemight also be used.
For convenience, this graph can be represented as a sparse tensor H ∈ R N 2 × n ,where N is the number of atoms considered, and in which node and edge features are encoded ondiagonal and off-diagonal entries, respectively.
A GNN is a function approximator f : R N 2 × n → Ythat produce a prediction, by successive application of layers φ : R N 2 × n (cid:96) → R N 2 × n (cid:96) +1 , where (cid:96) isthe layer count.
Expressivity GNNs are usually stack of layers that are permutation equivariant
[Equation]
where P ( g ) is the permutation matrix associated with the group element g .
While P ( g ) permutesnodes, P ( g ) ⊗ P ( g ) permutes the vectorized adjacency matrix.
In other words, our use of Kroneckerproduct is due to the equality vec ( PAP (cid:62) )
=
( P ⊗ P ) vec ( A ) for A ∈ R N × N .
This captures the fact that these models are designed to be insensitive to node ordering in H .
Thematerial is, in a sense, treated as if it was a molecule.
Much of the crystal structure is forgottenbecause it cannot be captured by ordering the nodes in a speciﬁc way and is only encoded in positionsor distances.
We argue that permutation equivariance is an overly strong requirement and was mainlyused for practical reasons.
When possible, it should be more beneﬁcial to use a model equivariant tothe actual symmetry of the data; in this case, the crystal space groups, which will, in general, be muchsmaller than the symmetric group.
Being equivariant to a smaller group results in less restrictiveparameter sharing and a more expressive model.
Note that this requirement is different from that of E (3) -equivariance characteristic of some architectures [63, 55, 5].
In these models, the E ( n ) group only acts on the position x i ∈ R 3 part of eachatom’s feature vector.
By contrast, the symmetry group of a crystal structure maps the crystal to itself,and its action is a permutation.
Revision It therefore offers an alternative to building more powerfularchitectures for these systems that does not suppose that coordinate information is available.
Inparticular, this approach is more suitable for abstract condensed matter systems, like spin and freefermion lattice models, in which coordinates are not relevant.
Moreover, space groups are subgroupsof the Euclidean group, therefore equivariance to space groups is less restrictive.
Therefore, in thiswork, we choose to concentrate on space group equivariance.
Note that equivariance (and not onlyinvariance) is important, as expressive invariant functions can be built by composing equivariantlayers with an output pooling layer. Equivariant functions can also be used to predict local propertieslike magnetization and charge distribution for example.
Invariance and symmetry breaking Even if the arrangement of atoms in a crystal is symmetric,this does not have to carry over to all local properties of the material.
Spontaneous symmetry breakingis common in materials and crucial to describing phenomena such as magnetism and superconductivity[43, 6].
Building a graph only at the unit cell level does not allow to capture local properties that differ across unit cells.
Using a supercell of multiple unit cells does not sufﬁce to solve this problemsince equivariant models have the property that identical inputs will be mapped to identical outputs[60, 78].
Input representation
Supercell Following the arguments of Section 4 we consider the set of atoms in a supercell insteadof only in the unit cell to increase the expressivity of the model.
Since this increases the computationalcomplexity of the method, we choose to keep the supercells small and deﬁne them with sublatticevectors b i = 2 a i .
The supercell is therefore 8 times larger than the unit cell, with centring vectors 165 (cid:110)(cid:80)i m i a i | m i ∈ { 0 , 1 } (cid:111) .
For each atom, we build a feature vector with a one-hot encoding of theatomic number and a one-hot encoding identiﬁcation of the unit cell it belongs to using the closestcentering vector.
We keep track of the index of each atom within the unit cell, and the index of theatoms’ unit cell h ( a,i ) =
[ onehot ( Z i ) , onehot ( a )] , where a ∈ { 1 , . .
, 8 } and i ∈ C , where C is thenumber of atoms in the unit cell.
The encoding of the unit cell allows to break the symmetry betweenatoms mapped into each other by lattice translations.
Graph We construct a graph from atoms in the supercell, encoding relative distances betweenatoms as edge features.
Inspired by [56], an edge feature vector is built from the distance betweenatoms d ij = (cid:107) x
i − x j (cid:107) as e ij =
exp (cid:16) − γ ( d ij − µµµ ) 2 (cid:17) .
The vector of Gaussian centers µµµ and γare hyperparameters.
Revision We use this approach to facilitate comparaison to previous works[56, 74], although Bessel encondings [38] could also be considered.
It can be seen as “soft" binningof interatomic distances.
Using this approach, we use only distance features in contrast to methodsthat use position vectors.
This has the beneﬁt of simplicity while still allowing a complete descriptionof the input structure [69, 4, 66].
Sparsity has proven to be a useful inductive bias in graph representation learning [26], and it is alsobeneﬁcial in reducing computational complexity.
However, atomic bonds are not unambiguouslydeﬁned in crystals [16, 1].
We choose to follow a similar approach to [32]: an edge is drawn betweenatoms if they share a Voronoi face and if the distance between atoms is smaller than the sum ofatomic Cordero radii plus a cutoff ∆ = 0 .
5 Å.
This approach has the advantage of being physicallysound and producing graphs that are relatively sparse.
We provide more details on the graph-buildingstrategy and compute metrics on the resulting graphs in Appendix A.1.
To preserve translational invariance for atoms at the boundary of the supercell, edges are initiallyalso drawn to atoms outside the supercell.
Then, if an edge points outside the supercell, its head ismapped to the corresponding representative node inside the supercell.
This is analogous to circularpadding in image processing.
Equivariant crystal networks
Product groups Consider a crystal structure with space group G and in which each atom has afeature vector h i .
Then the space group acts as a permutation of the input atoms
[Equation]
where π Sg and π Ug are the permutations associated with group element g on the supercells and withinthe unit cells respectively.
If a dataset contains only samples that share the same crystal structure, then a model equivariantwith respect to G can readily be used.
However, the case of a dataset with multiple different crystalstructures that would be used in a typical supervised learning setting is more challenging for tworeasons.
First, the samples may have different space groups, which would require different modelsthat would each be trained with a fraction of the data.
Second, the group action may be different evenfor the same space group.
This is because the unit cells may have different structures and numbers ofatoms. [Figure]
A solution to address the second issue is to consider equivariance to a direct product of groups
G Λ × S C , where the symmetry group of the lattice G Λ acts across unit cells and S C within unit cells:
[Equation]
Revision In this way, differences in unit cell structures are dealt with by the symmetric group, whereparameter-sharing can handle variable-sized inputs [77].
We still have to accommodate 14 differentgroup actions G Λ corresponding to the different Bravais lattices.
To avoid using a different model foreach lattice, we propose two groups to deal with all the lattices.
The ﬁrst option is to consider theleast symmetric Bravais lattice of primitive triclinic type and use its symmetry group P ¯1 =
T Λ (cid:111) C 2 .
This group is a subgroup of all the other lattice symmetry groups.
The second option we consideris to simply use the symmetric group S Λ that is the symmetric group across unit cells, which is anovergroup of all the lattice symmetry groups.
This leaves us with a hierarchy of groups, with S N , thesymmetric group over all atoms of a supercell, being the largest : [Equation]
[Equation]
In our experiments, we use the two groups in this hierarchy, which offer different levels of expressivity.
Equivariant message passing Having deﬁned the group action, we can now build the Equivariant Crystal Network (ECN).
We will seek to use the message passing framework, which has demonstratedgood performance on molecular data, and generalize it to obtain equivariance to other groups thanthe symmetric group. The update equations for message passing framework are
[Equation]
The idea is to deﬁne parameter-sharing patterns for functions φ e and φ h , such that there can bemultiple ones while still retaining equivariance.
Following [53], we ﬁrst deﬁne the parametersharing pattern of the set of input nodes N , with respect to group G as the colored bipartite graphΩ ≡ ( N , α, β ) , with the edge-color function α : N × N → { 1 , . . .
, C e } and node-color functionβ : N → { 1 , . . .
, C h } .
We also consider the action of the group G on edges Ω as g · ( i, j ) .
=(
π g ( i ) , π g ( j ))
∀ g ∈ G .
We deﬁne the orbit G · ( i, j ) of edge ( i, j ) as the set of edges in which itcan be moved to by the group action : G · ( i, j ) .
= { g · ( i, j ) | g ∈ G } .
A similar deﬁnition applies tothe orbit of a node, G · i .
= { π g ( i ) | g ∈ G } .
We then make the following claim:
Claim 6.1 The layer deﬁned by the C e functions φ α ( i,j ) e and the C n functions φ β ( i ) h
[Equation]
is G-equivariant if the parameter-sharing pattern Ω respects the equivariance condition:
[Equation]
The proof of this claim follows in Appendix A.2.
The computational process for producing the patternis to ﬁnd the orbit of G-action on the edges (nodes) and assign the same color to all edges (nodes)in the same orbit [53].
The computational cost of this orbit-ﬁnding process grows linearly with thenumber of edges (nodes) [30].
This layer generalizes both MPNNs and equivariant multilayer perceptrons, such as CNNs.
The
MPNN is recovered with G = S n and a standard CNN with circular convolution with G = T Λ ,φ α ( i,j ) e
[Equation]
We now consider a product group G × H , acting according to Eq.
From Claim 1 of [67], weknow that the equivariant linear map for this group will be the Kronecker product of equivariant mapsfor individual groups; see also [47].
The reformulation for parameter-sharing patterns is the following.
If parameters-sharing patterns Ω 1 and Ω 2 satisfy the equivariance condition for G and H respectively,then the parameter-sharing pattern Ω =
( N × M , α, β ) satisﬁes the equivariant condition if
[Equation]
and
[Equation] [Equation] [Equation]
This simply means that a new color is deﬁned in the product pattern for each possible combination ofcolors in the original patterns.
For our model, we choose functions φ α ( i,j ) e and φ β ( i ) h to be MLPs with one hidden layer.
In addition,we add a weighting factor to the edge aggregation 9, as this as been shown to be beneﬁcial by [74, 55]:
[Equation]
i
[Equation]
where e ij = φ a
[Equation]
(15) and φ a is simply a linear layer.
This change does not affect the equivariance of the model.
Example 6.
In our running example, the parameter-sharing pattern for the message passing is produced by the Kronecker product of the pattern for the P 2 group (see Appendix A.3 for a similar example with the P 6 m group) and the pattern for the symmetric group S 2 as shown in the ﬁgure below (Top left).
However, we only need to keep the colors for which there is a corresponding edge (Top right).
Message passing is used on the resulting edge/node colored graph where similar colored nodes and edges use the same functions in message passing.
Model implementation
Network architecture We aimed to keep the architecture of our ECN model as simple as possible.
The model receives crystal structure graphs as input, with one-hot encoded feature vectors foreach atom.
We use 128-dimensional embeddings and keep the same dimension for hidden layersthroughout the network.
The ECN consists of 6 layers of the equivariant message passing operation6.1.
This is followed by a mean-pooling operation over node embeddings of each graph and atwo-layer MLP that outputs the ﬁnal prediction.
Mean pooling is preferred to other options becausewe only predict intensive physical quantities.
These properties are "per-atom" and do not depend onthe choice of supercell.
Therefore, selecting a pooling operation that respects this invariance makessense.
If we were to predict extensive quantities, sum pooling should be the preferred option.
Input features Many choices to encode atomic features are possible and have been suggested in theliterature.
We perform experiments over these variants.
For each atom, we encode its type in a featurevector.
We consider two strategies, using only information available from the periodic table.
The ﬁrstone is to use only the atomic number.
The second one is to use the one-hot encoded group and periodnumbers for each atom.
The second strategy could beneﬁt from promoting better generalizationsince embeddings of atoms belonging to the same group or period will show a certain similarity.
Around 24% of the structures retained in the Materials Project dataset have been computed usingthe Hubbard-
U extension of DFT.
Since this information can signiﬁcantly inﬂuence the resultingproperties [64] it is added to the atomic feature vector as a binary feature.
Other details We implemented our model using Pytorch [51].
We use a sparse implementationof the equivariant message-passing based on the Pytorch Scatter package [23].
We use the AdamWoptimizer [45] with weight decay regularization.
The full hyperparameter setup is provided in Appendix A.5.
Experiments
Materials Project We perform experiments using the Materials Project dataset [33] 1 .
This standarddataset of materials informatics comprises more than 120K materials with a complete speciﬁcation oftheir crystal structure and some important physical properties obtained with high-throughput DFTcalculations. [Table]
The Materials Project datasethas not beeninitially builtto serve as amachine learningbenchmark; weperform somepreprocessing to make it suitable.
First, since multiple DFT calculations were sometimes performedon close initial conﬁgurations, some samples only show marginal structure differences and resultingproperties.
This is exempliﬁed by the compound Li
Mn 2 Co 5 O 16 , which appears 322 times inthe database.
Such duplicates can result in training-test leakage.
To prevent this, we considertwo structures redundant if they have the same unit cell chemical formula and the same spacegroup.
Amongst a set of duplicate structures, the one with the lowest formation energy is chosen.
Second, we ﬁlter out one-dimensional and two-dimensional materials from the dataset to only keepthree-dimensional materials.
Finally, we remove materials for which the unit cell contains morethan 50 atoms.
These are often associated with molecular and inorganic crystals with very differentproperties than the other materials.
Training, validation, and test splits are 80%, 10%, and 10% of thedataset.
We provide statistics on the processed dataset in Appendix A.4.
Following previous work, we predict a few relevant energetic properties: the formation energy E ,the Fermi energy E F , and the band gap E g for the subset of insulating materials.
We also predictthe binary insulator or conductor character material.
Finally, we also predict the magnetic momentper atom M .
This can be seen as a graph regression or classiﬁcation task.
For regression, training isperformed using the mean-squared error (MSE) loss function, but we report the mean-absolute error(MAE).
For classiﬁcation, we use the cross-entropy loss function.
Revision We compare the results obtained by our models to baselines [56, 74, 12].
Note thatbecause these papers used different training and test splits and preprocessing schemes (even betweenthemselves), our results cannot be directly compared.
To alleviate that, we trained our own versionsof two of the baselines using our splits and a similar training procedure.
We obtain slightly better orcomparable results to the baselines on all targets when evaluated on the same splits.
The S Λ versionoffers better performance than the P ¯1 model overall, showing that it is more beneﬁcial to lean on theside of having slightly more symmetry than necessary at the cost of some expressivity.
We provideadditional results for the model variants in Appendix A.6.
The beneﬁts of the increased expressivityon this task is in the not crucial, which we think can be explained in part by the relatively small sizeof the Materials Project dataset.
In a larger data regime, we expect that the beneﬁt of increasedexpressivity will outweight the cost in generalization capability.
{+Perov-
 Finally, we perform experiments using the Perov-5 dataset [9] as provided by [73]. Inthis dataset, all the materials share the same Perovskite crystal structure. The task considered isthe regression of the heat of formation computed through DFT.+}
Results are shown on Table 2.Theimprovement on this dataset is signiﬁcantly more important for the proposed model compared to thebaselines than on the Materials Project dataset.
We hypothesize that the fact that all the structures areshared in this dataset allows the model to specialize more efﬁciently leading to better generalization.
[Table]
Conclusion
We have shown how to leverage crystal symmetry to build moreexpressive and physically motivated neural networks for materialsdata.
This allows us to obtain a close equivalent of group equivariantconvolution on this data structure.
These models show excellentaccuracy in supervised property prediction, which supports the ideathat symmetry is a useful inductive bias.
Such models could be used for other tasks on materials suchas dynamics prediction Revision , if the dynamics approximately preserves the crystal structure. We also think that these models have signiﬁcant potential on more abstract condensed matter systemssuch as spin models and free-fermion models.
We have also deﬁned equivariant message passing, ageneralization of the MPNN framework that can potentially be used on any data structures for whicha group can capture the symmetry in sparse interactions between the basic elements.
One limitation of this approach and area of future improvement is on the computational efﬁciency of the equivariantmessage passing, which does not beneﬁt from optimized algorithms available for convolutions.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accuratelyreﬂect the paper’s contributions and scope?
[Yes] On both the theoretical and experimental side, the paper’s contributions are alignedwith the abstract and introduction.(b) Did you describe the limitations of your work?
[Yes] See Conclusion(c)
Did you discuss any potential negative societal impacts of yourwork?
[N/A] We do not think that this is relevant to this work(d)
Have you read the ethics review guidelines and ensured that yourpaper conforms to them?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?[Yes]
See Theorem 6.1 and its proof in the Appendix A.2(b)
Did you include complete proofs of all theoretical results?
[Yes] Theonly proof is included in Appendix A.23.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproducethe main experimental results (either in the supplemental material oras a URL)?
[Yes] The details needed to reproduce the experimentsare included in the supplementary material Appendix A.5.
A linkto the relevant code will also be accessible upon publication of thepaper.(b)
Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] The details needed to reproduce the experiments are included in the supplementary materialAppendix A.5.(c)
Did you report error bars (e.g., with respect to the random seed afterrunning experiments multiple times)?
[No] This is due to a lack ofcomputational resources.(d)
Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Yes] This is provided in the supplementary material Appendix A.
If you are using existing assets (e.g., code, data, models) or curat- ing/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]See Introduction and Model implementation sections(b)
Did you mention the license of the assets?
[Yes] This is mentionedin the supplementary material Appendix [-A.
 (c)-]A.4(c) Did you include any new assets either in the supplemental material oras a URL?
[No] The used dataset will be provided upon publicationof the paper.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
This is not relevant according to the license of the used assets.
(e) Did you discuss whether the data you are using/curating contains
personally identiﬁable information or offensive content?
[N/A] Thisis not relevant for this paper.5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants andscreenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to In-
stitutional Review Board (IRB) approvals, if applicable?
[N/A]
[Table] (c) Did you include the estimated hourly wage paid to participants andthe total amount spent on participant compensation?
[N/A]
[Table]
A.1 Graph-building strategies
Revision The graphs were built using the IsayevNN class from thepymatgen [50] package.
It implements the commonly used Voronoitessalation to deﬁne neighbors.
Two atoms are considered bonded ifthey share a face in the Voronoi tessalation of the supercell and theirdistance is less than the sum of the atomic Cordero radii (a measureof the atomic radius) plus a cutoff ∆ = 0 .
5 Å .
This value of the cutoff was increase compared to [32]to reduce the number of disconnected graphs.
We provide statistics for the graphs obtained by the method describedin Section 5.
A hard cutoff on atomic distances of 6 Å is also imposedon atomic distances.
A.2 Proof of Claim 6.
We prove that the function deﬁned by
[Equation]
[Equation]
[Equation]
18) with
[Equation]
[Equation]
[Equation]
is G -equivariant by proving the equivariance of every step and using the fact that function compositionpreserves equivariance.
First, we need to show equivariance of the message function on
Equation (16)
[Equation]
Using Equation (19), we have
[Equation]
[Equation]
The message aggregation step at Equation (17) is permutation equivariant
[Equation]
Finally, we need to show that the node function on Equation (18) isalso equivariant
[Equation]
Using Equation (20), we ﬁnd
[Equation]
The right-hand side is equal to h t +1 π g
( i ) by deﬁnition.
A.3 Parameter sharing patterns
Revision The parameter sharing patterns are computed using agroup-theoretical orbit ﬁnding algorithm that has linear complexityin the number of size of the generating sets of a group and in thenumber of edges in the coloured bipartite graphs [29, 58].
We show the parameter sharing patterns for the different Bravaislattice groups (Figure 7 and Figure 8).
For the 2-dimensional groups,we use a 2 × 2 supercell and for the 3-dimensional groups
a 2 × 2 × 2supercell.
Note that the numbering of the unit cells within thesupercell is chosen by convention and can vary for different lattices.
Notice that the parameter-sharing patterns for different groups can bethe same.
This is because, for different groups, the group action caninduce the same orbits on the bipartite graph.
In particular, for somegroups (Figure 7c), the pattern collapses to that of the symmetricgroup.
This can be undesirable since it reduces expressivity.
Thiscan be alleviated by using larger supercells or eliminated by considering the group acting on itselfinstead of on supercells, as done in [13].
[Figure]
[Figure]
[Figure]
A.4 Materials Project dataset
We hereafter report the number of samples in the Materials Projectdataset at different levels of the preprocessing scheme :
(a) Full dataset(b) No duplicates(c) No duplicates and unit cell size constraint(d) No duplicates, unit cell size constraint and 3D materials(e) No duplicates, unit cell size constraint, 3D materials and valid graphs(f) Insulators, no duplicates, unit cell size constraint, 3D materials and validgraphs
We also report the mean and standard deviation of each target forthe processed dataset
[Figure]
[Figure]
[Table]
[Figure]
[Table]
A.5 Hyperparameters
We use the same training setup for all the models.
The learning rateis initialized at 1 × 10 − 3 with a scheduler halving it after each 25epoch plateau on the validation loss.
Training is perfomed for 1000epochs or until the learning rate reaches 1 × 10 − 6 .
Revision Weperformed sweeps over learning rates for all the models to verifythat this is indeed a setup in which all models train well.
Revision For the ECN models, we optimized the number of layers,the embedding dimension, the weight decay parameter and dropout(without noticing signiﬁcant improvement).
We use 6 layers ofmessage passing operations.
The φ α ( i,j ) e and φ β ( i ) h functions are1-hidden layer MLPs.
For the φ α ( i,j ) e functions, the weights for thehidden layer and output layers are shared across all α ( i, j ) .
This was found to perform slightly better.
The S WISH activation function is used.
The feature dimension of node embeddings is 100 across allthe network.
For edge embeddings, the dimension is set at 20.
[Table]
For the CGCNN model, we used the same embedding sizes (theywere not speciﬁed in the original paper) and 2 convolution layers.
For MEGNet, we used the same architecture setup as in the original paper.
A.6 Supplementary results
[Table]
[Table]