Periodic Graph Transformers for Crystal Material Property Prediction
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We consider representation learning on periodic graphs encoding crystal materials.
Different from regular graphs, periodic graphs consist of a minimum unit cellrepeating itself on a regular lattice in 3D space.
How to effectively encode these periodic structures poses unique challenges not present in regular graph representationlearning.
In addition to being E(3) invariant, periodic graph representations needto be periodic invariant.
That is, the learned representations should be invariant toshifts of cell boundaries as they are artiﬁcially imposed.
Furthermore, the periodicrepeating patterns need to be captured explicitly as lattices of different sizes andorientations may correspond to different materials.
In this work, we propose atransformer architecture, known as Matformer, for periodic graph representationlearning.
Our Matformer is designed to be invariant to periodicity and can capturerepeating patterns explicitly.
In particular, Matformer encodes periodic patternsby efﬁcient use of geometric distances between the same atoms in neighboringcells.
Experimental results on multiple common benchmark datasets show thatour Matformer outperforms baseline methods consistently.
In addition, our resultsdemonstrate the importance of periodic invariance and explicit repeating patternencoding for crystal representation learning.
Introduction
Crystal material property prediction is important for the discovery of new materials with desirableproperties [35, 32–34, 45, 47, 38, 3, 31, 4].
Different from molecules and proteins [13, 46, 39, 7,44, 24, 41], which are commonly represented as graphs [42, 27, 10, 11, 23, 25], crystals consist ofa minimum unit cell repeating itself on a regular lattice in 3D space.
Thus, crystals are naturallyrepresented as periodic graphs.
A key challenge of crystal material property prediction lies in how toeffectively encode periodic structures that are not present in regular molecular graph representations [8,1, 18, 38, 21, 20, 29, 22, 36, 15].
Speciﬁcally, periodic graphs of crystals require unique periodicinvariance. Periodic invariance has two facets; those are, the learned representations should beinvariant to both the scaling up of the minimum repeatable unit cells and shifts of periodic boundaries.
Although the former has been considered in Xie et al. [47, 48], the later is rarely identiﬁed explicitlyand sometimes not considered by previous studies [47, 3, 43, 38, 4, 48, 31, 49]. Furthermore, therepeating patterns of periodic graphs should be captured explicitly. Given a ﬁxed minimum unit cellstructure, lattices of different sizes and orientations may correspond to different materials.
Withoutsuch periodic patterns, the inﬁnite structures of crystals may not be represented accurately. However,the explicit encoding of periodic patterns is not explored in previous studies [47, 3, 38, 4, 31].
In this work, we propose to tackle periodic graph representation learning by incorporating bothperiodic invariance and periodic pattern encoding.
We propose a periodic graph transformer, known as
Matformer, that is periodic invariant and can capture periodic repeating patterns explicitly for crystal [Figure] [Table] [Figure]
[Table]
+kTufA5fwHTqQqs/hHz8/wk553gRZ0rb9oeVm5mdm1/ILxaWlldW14rrGw0VxpLQOgl5KFseVpQzQeuaaU5bkaQ48DhtesOzcd68oVKxUFzrJKJugPuC+YxgbayrclLuFkt2xc6E/oLzDaXq7qf9uHO3VesW3zu9kMQBFZpwrFTbsSPtplhqRjgdFTqxohEmQ9ynbYMCB1S5adbqCO0bp4f8UJojNMrcny9SHCiVBJ6pDLAeqOlsbP6XtWPtn7opE1GsqSCTj/yYIx2i8dyoxyQlmicGMJHM9IrIAEtMtNlOwSzBmR75LzQOK85x5ejSKVXPbyFTHrZhDw7AgROowgXUoA4E+nAPT/BscevBerFeJ6U5a3LDJvyS9fYFjnGQsA==</latexit> y representation learning.
Matformer achieves periodic invariance through
construction methods encodes periodic patterns by efﬁcient use of geometric distancesbetween the same atoms in neighboring cells, thereby capturing the lattice size and orientationof a given crystal.
We conduct experiments on two commonly used material benchmark datasets,including the Materials Project [14] and Jarvis [6].
Results show that Matformer outperforms baselinemethods consistently on various tasks.
In addition, our results demonstrate the importance of bothperiodic invariance and periodic pattern encoding for crystal representation learning.
Background
Crystal property prediction.
Given a crystal represented as ( A , P , L ) , crystal property predictionaims to predict a target property value y , which is either real as y 2 R or categorical as y 2 N forregression or classiﬁcation task.
Speciﬁcally, a crystal is represented as a unit cell with periodicpatterns.
A unit cell is a minimum repeatable structure for the given crystal, and it can be describedby matrices A and P .
A = [ a 1 , a 2 , · · · , a n ] T 2 R n ⇥ d a is the atom feature matrix, where a i 2 R d ais the d a -dimensional feature vector for atom i in the unit cell.
P =
[ p 1 , p 2 , · · · , p n ] T 2 R n ⇥ 3 isthe position matrix, where p i 2 R 3 contains the Cartesian coordinates for atom i in 3D space.
Tofurther encode periodic patterns, an additional lattice matrix L = [ ` 1 , ` 2 , ` 3 ] T 2 R 3 ⇥ 3 is used todescribe how a unit cell repeats itself in three directions, including ` 1 , ` 2 , and ` 3 .
We show a 2Dcase of periodic patterns for easy illustration in Fig.
1 (a).
Note that crystals usually possess irregularshapes in practice.
Hence, ` 1 , ` 2 , and ` 3 are not always orthogonal in 3D space.
Formally, given acrystal representation ( A , P , L ) , the inﬁnite crystal structure can be represented as
[Equation]
Here, set ˆ P contains all possible positions for each atom i , and set ˆ A contains the correspondingatom feature vector for each atom i .
Multi-edge graph construction for crystals.
The multi-edge graph construction proposed by Xie and
Grossman [47] aims to capture atom interactions across cell boundaries, which are imposed artiﬁcially.
In a regular molecular graph, a node corresponds to a single atom.
In contrast, in a multi-edge graph,node i represents atom i and all its duplicates in the inﬁnite 3D space.
Apparently, node i contains theatom features vector a i and all positions in the set { ˆ p i | ˆ p
i = p
i + k 1 ` 1 + k 2 ` 2 + k 3 ` 3 , k 1 , k 2 , k 3 2 Z } ,which is a subset of the inﬁnite set ˆ P deﬁned in Eq.
Formally, the multi-edge graph constructionmethod builds edges between nodes as follows.
Given a preﬁxed radius r 2 R , if there existsany 3-tuple ( k 0 1 , k 0 2 , k 0 3 ) , where k 0 1 , k 0 2 , k 0 3 2 R , such that the Euclidean distance d 0 ji 2 R satisﬁes
[Figure]
[Table] [Figure] d 0 ji = ||
p j + k 0 1 {+` 
+} + k {+0 
 `+} 2 + k 0 3 ` 3  p
i || 2  r , an edge is built from j to i with the initial edge featured 0 ji .
An example of the edge construction is shown in Fig. 1 (b).
Intuitively, if there exist m positionsof node j within the radius of the center node i , this method builds m edges from node j to nodei .
By considering all possible positions of every node within a predeﬁned radius in 3D space, themulti-edge graph construction method can in essence capture atom interactions across cell boundaries.
This graph construction method has been widely used in many following up studies [37, 31, 3, 38, 4].  The proposed Matformer
Different from molecular graphs, crystal graphs consist of a minimum unit cell repeating itself on aregular lattice in 3D space.
When encoding such periodic structures, unique challenges lie in periodicinvariance and periodic pattern encoding.
In this section, we ﬁrst propose to formally deﬁne periodicinvariance and periodic pattern encoding. We then provide analyses to reveal the importance ofthese two considerations for the representation learning of periodic graphs encoding crystal materials. Finally, we present Matformer, a transformer architecture for periodic graph representation learning. It can be seen Matformer is invariant to periodicity and can capture repeating patterns explicitly.
3.1 Periodic invariance for crystals
Periodic invariance is proposed based on E(3) invariance for a unit cell.
Hence, we ﬁrst provide theformal deﬁnition of unit cell E(3) invariance as below.
Deﬁnition 1 (Unit Cell E(3) Invariance) .
A function f : ( A , P , L ) ! X is unit cell E(3) invariantsuch that for all Q 2 R 3 ⇥ 3 , | Q | = ± 1 and b 2 R 3 , we have f ( A , P , L ) =
f ( A , Q P + b, Q L ) ,where Q is rotation and reﬂection transformations, and b is translation transformations in 3D space.
Intuitively, unit cell E(3) invariance shows that the structure of a cell remains the same when eitherapplying rotations and reﬂections to position matrix P and lattice matrix L together, or applyingtranslations to P only.
In addition to unit cell E(3) invariance, periodic invariance is also shown necessary for generatingvalid crystal representations.
Speciﬁcally, when the lattice matrix L = [ ` 1 , ` 2 , ` 3 ] T 2 R 3 ⇥ 3 is ﬁxedfor a crystal, we can still obtain different position matrices P 2 R n ⇥ 3 and different unit cell structuresby shifting the period boundaries.
As shown in Fig.
2 (a) and (b), the formed unit cell structures aredifferent for the same crystal by shifting period boundaries.
To this end, we further introduce periodicinvariance, which shows that when the periodic boundaries are shifted or scaled up, the periodicinvariant representation should remain the same.
Formally, based on Sec. 2, we further deﬁne afunction  : ( ˆ A , ˆ P , L , p) ! ( A , P ) simulating how to form different unit cells from a given inﬁnitecrystal structure.
For an inﬁnite crystal structure represented as ( ˆ A , ˆ P ) ,  uses a corner point p andshape matrix L to form a unit cell represented as ( A , P ) .
In addition, we use ↵ 2 N 3+ to indicate the scaling up of a repeating unit cell formed by periodic boundaries.
Then the formal deﬁnition ofperiodic invariance is provided as below.
Deﬁnition 2 (Periodic Invariance) .
A unit cell E(3) invariant function f : ( A , P , L ) ! X is periodicinvariant if f ( A , P , L ) =
f (  ( ˆ A , ˆ P , ↵ L , p) , ↵ L ) holds for all p 2 R 3 and ↵ 2 N 3+ .
Signiﬁcance of periodic invariance.
Breaking periodic invariance will result in different crystalgraphs for the same crystal.
In Fig.
2 (a) and (b), we show a graph construction method thatbreaks periodic invariance.
This method is employed by a transformer-based model, known as
Graphormer [49], for learning from adsorbates and catalyst surface systems [2].
In particular,
Graphormer ﬁrst uses radius to include all the atoms of interest in nearby cells and then builds a fullyconnected graph.
A detailed illustration is shown in Fig.
5 in Appendix.
A.1.
Based on the formaldeﬁnition of periodic invariance, in this study, we aim to integrate such an important componentin our Matformer.
By doing this, our model is able to construct a distinct crystal graph for a givencrystal structure, resulting in a more informative and discriminative crystal learning scheme.
[Figure]
3.2 Periodic pattern encoding
[Figure]
As introduced in Sec. 2, L = [ ` 1 , ` 2 , ` 3 ] T [Figure] R 3 ⇥ 3 containing periodic patterns is another key
[Figure]
component to describe crystal structures.
Essentially, periodic patterns show how the minimumrepeatable structure ( A , P ) expands itself in inﬁnite 3D space.
Without such periodic patternencoding in L , crystal structures are treated asﬁnite structures similar to molecules.
As shownin Fig.
3, the widely used multi-edge graph construction method [47, 3, 38, 37, 4] only captureslocal interactions among atoms but ignores theimportant periodic patterns.
However, such periodic repeating patterns need to be capturedexplicitly, as lattices of different sizes and orientations may correspond to different materials.
Hence, To better represent the inﬁnite structuresof crystals, we argue that the periodic patterns
L 2 R 3 ⇥ 3 should be explicitly taken into consideration in crystal learning.
3.3 The proposed graph construction methods
In this section, we introduce the proposed graph construction methods for Matformer.
Our methodseffectively integrate the above two components for accurate crystal representation learning.
Invariant crystal graph construction .
We consider two crystal graph construction methods, including radius-based graph construction and fully connected graph construction.
Both of them satisfyperiodic invariance, and mathematical proofs can be found in Appendix.
A.2.
In Fig.
2, we show byexample that treating atoms as single nodes breaks periodic invariance.
Instead, in our methods, wefollow the fashion introduced in Sec. 2 to treat node i in a constructed graph as atom i and all itsrepeated duplicates in inﬁnite 3D space.
We use the multi-edge graph construction [47] introduced in Sec. 2 as an alternative crystal graphconstruction method.
Note that although the multi-edge graph construction satisﬁes periodic invariance as in Fig.
4 (a), several existing works [47, 31] do not follow the settings exactly thus breakingperiodic invariance.
Within the given radius shown in Fig.
4 (a), these studies form the neighborhoodof node i by selecting t nearest neighbors ranked by geometric distances.
If there exist severaldifferent atoms with the same distance to node i , there is no deterministic way to select from them, asshown in Fig.
6 in Appendix.
A.1.
As a result, periodic invariance cannot be guaranteed.
Given the fully-connected fashion employed in Graphormer has achieved impressive performance onmolecular learning, we further propose another graph construction method for Matformer, known asthe fully connected graph construction.
This method uses a different strategy to determine neighbors
[Figure]
[Figure] [Table] 1 + ` [Table]
|
[Table] [Figure] for each center node. Speciﬁcally, for node i and j , it builds edges for the entries corresponding tothe t smallest distances in { d ij | d ij = ||
p i  p j + k 0 1 {+` 
+} + k {+0 
 `+} 2 + k 0 3 ` 3 || 2 , k 0 1 , k 0 2 , k 0 3 2 Z } .
It can be 153 seen every pair i and j is connected in the constructed graph and there are t edges between them.
Overall, the used multi-edge graph construction and proposed fully-connected graph construction bothsatisfy the important periodic invariance.
Particularly, they possess great ﬂexibility to be used in futurestudies for crystal learning.
In this study, we employ both methods as part of our proposed Matformer,and in main experiments, the radius-based method is used due to better empirical performance.
Periodic pattern encoding with self-connecting edges .
In this study, we propose to encode theimportant L =
[ ` 1 , ` 2 , ` 3 ] T 2 R 3 ⇥ 3 into crystal graphs by adding self-connecting edges.
Asmentioned in Sec. 3.2, periodic patterns describe sizes and orientations of lattices of a crystalstructure, eventually determining the properties of this crystal.
A natural step for encoding suchrepeating periodic patterns is to consider the relative positions between an atom and its nearbyrepeated duplicates.
Formally, given an atom i with position p i and L = [ ` 1 , ` 2 , ` 3 ] T 2 R 3 ⇥ 3 , weneed to encode the atom’s three nearby duplicates with positions p i + ` 1 , p i + ` 2 , and p i + ` 3 .
Itis widely known that a direction vector ` i is determined by both its length
|| ` i || 2 and orientation. Essentially, || ` i || 2 indicates the geometric distance between atom i and its corresponding duplicate.
However, the computing of orientation information, such as angles, usually induces high complexity.
Hence, it is not practical to encode such orientation information into transformer architectures.
To thisend, we propose to use geometric distances solely to implicitly consider the orientation information.
Speciﬁcally, we use additional distances to determine angles between any two direction vectors in L .
For example, the angle between ` 1 and ` 2 can be easily computed by || ` 1 || 2 , || ` 2 || 2 , and an additional 172 distance || ` 1 + ` 2 || 2 .
Hence, based on these three distances, we can determine lengths of ` 1 and ` 2 , 173 and the relative orientation between them.
Extensively, we use six geometric distances,
including|| ` 1 || 2 , || ` 2 || 2 , || ` 3 || 2 , || ` 1 + ` 2 || 2 , || ` 2 + ` 3 || 2 , and || ` 1 + ` 3 || 2 in our study, as shown in Fig.
4 (b).
By doing this, the length of each direction vector and the angle between any two direction vectorscan all be determined.
As a result, the shape formed by lattice matrix L is then ﬁxed.
Overall, webuild the aforementioned six geometric distances as six self-connecting edges for node i .
By doingthis, our model is capable of encoding periodic patterns in L completely, resulting in a more accuratecrystal representation learning scheme.
Importantly, our approach also guarantees periodic invariance.
This is because the pairwise distances do not change when the periodic boundaries are shifted.
Overall, the graph construction for Matformer consists of two necessary stages, including invariantgraph construction and periodic pattern encoding.
For the ﬁrst stage, we rigorously prove that themulti-edge graph construction satisﬁes periodic invariance and show that several previous works [47,31] break periodic invariance using a different neighbor selection strategy.
Additionally, we proposea fully-connected crystal graph construction method satisfying periodic invariance, and the methodcould be used in future studies for crystal representation learning.
For the second stage, we naturallyencode periodic patterns into constructed graphs by adding self-connecting edges without breakingperiodic invariance.
A constructed graph in 2D case is shown in Fig.
4 (c).
3.4 Message passing scheme
Building on constructed graphs introduced in Sec.
3.3, we propose our message passing scheme for
Matformer.
Formally, we denote a constructed crystal graph as G =
( A , E ) .
Here, each a i 2 Ais the d a -dimensional feature vector for atom i , as introduced in Sec. 2.
Particularly, e hij 2 E isd e -dimensional feature vector for the h -th edge between nodes i and j .
We follow the regular attentionmechanism that computes query, key, and value [28, 26].
Our proposed message passing schemeis composed of three steps; those are, edge-wise attention coefﬁcients computing, edge-wise valuemessage computing, and node updating.
Formally, we let f ? i denote the input feature vector of node ifor the ` -th layer of Matformer.
The message passing scheme of the ` -th layer is described as below.
In the ﬁrst step, q hij , k hij and ↵ hij for the h -th edge between i and j are computed as
[Equation]
where LN Q , LN K , and LN E denote the linear transformations to compute query, key, and edgeembedding in ` -th layer, respectively.
e h 0 ij is the intermediate output for e hij .
We use  and | to denote
Hadamard Product and concatenation.
Particularly, we omit the softmax to enhance the model’scapability to distinguish nodes with different degrees, and to make the whole network more efﬁcient.
After obtaining the attention coefﬁcients ↵ hij , in the second step, we compute m hij that is the messageof edge e hij as
[Equation]
where LN V and LN update are the linear transformations to compute value and the updated message,and LNorm denotes the layer normalization operation.
In particular, we apply sigmoid and Layernormalization to ↵ hij to stabilize the training.
Finally, in the third step, we compute node i ’s feature vector f i .
Speciﬁcally, we ﬁrst obtain messagem i by aggregating information from node i ’s neighborhood over multiple edges, then achieve f i as
[Equation]
where  is the used activation function, and BN indicates batch normalization.
In addition, LN msgand LN fea are linear transformations to update the messages on edges and the old atom features.
Graphormer represents an effective transformer variant for molecular graph learning.
The differencesbetween Graphormer and the proposed Matformer lie in both graph construction and message passingscheme.
Firstly, Graphormer treats every atom as a single node.
Also, as there is only one geometricdistance between node i and j in 3D space, a single edge will be built between them.
Apparently, Graphormer breaks periodic invariance, as mentioned in Sec. 3.1.
However, our proposed Matformersatisﬁes periodic invariance by employing the crystal graph construction methods described in Sec. 3.3.
For the message passing, Graphormer uses the node-wise attention and encodes pairwise distancesas attention bias.
It cannot work properly on multi-edge graphs for crystals.
While Matformer isspeciﬁcally designed for multi-edge crystal graphs by performing edge-wise attention and encodinggeometric information into edge-wise messages, as described above.
The detailed architecture of our Matformer is shown in Fig.
7 in Appendix.
Related work
Crystal property prediction .
Several existing methods [43, 16, 17, 12] model crystals as chemicalformulas and employ sequence models to process such sequential formulas.
Other studies [47, 37,31, 3, 38, 4] consider 3D structures of crystals and formulate crystals as 3D graphs, then apply GNNsto learn from crystal graphs.
As crystals are essentially periodically repeated structures, the graphconstruction needs to consider periodic invariance and periodic pattern encoding.
There are limitedefforts to identify these two unique components.
As an early work, CGCNN [47] proposes to captureatom interactions across artiﬁcial cell boundaries by using multi-edge graphs described in Sec. 2.
The multi-edge graph satisﬁes periodic invariance as described in Sec. 3.1, but fails to consider theimportant periodic patterns, as described in Sec. 3.2.
The multi-edge graph construction method iswidely used in the following studies [37, 31, 3, 38, 4].
Based on the constructed crystal graphs, many GNN variants have been proposed for effective crystal representation learning [47, 37, 31, 3, 38, 4].
Recently, ALIGNN [4] achieves the state-of-the-art performance on two major material datasets.
It uses angle information in the message passing procedure to generate more informative anddiscriminative representations.
However, the use of angles introduces excessive time complexity.
Geometric GNNs and graph transformer .
Many efforts have been made to incorporate 3D geometric information in molecular learning.
Exemplary studies include SchNet [38], DimeNet [21, 20],
SphereNet [29], GemNet [22], etc.
However, these methods are designed for molecules without periodic patterns.
Recently, graph transformers [49, 9] using geometric information, e.g.,
Graphormer [49], have shown great potential on real-world graph data.
However, Graphormerconsiders neither periodic invariance no periodic pattern encoding.
Differences with our method.
To the best of our knowledge, periodic invariance and periodicpattern encoding described in Sec.
3 are rarely identiﬁed and explored in existing works for crystalproperty prediction.
CGCNN [47] breaks periodic invariance on some corner cases because it usestwelve nearest neighbors determined only by distances as described in Sec. 3.3.
In addition, previousmethods including CGCNN [47], SchNet [38], MEGNET [3], CYATT [37], GATGNN [31], and
ALIGNN [4], all fail to consider the important periodic pattern encoding as introduced in Sec. 3.2.
For
Graphormer [49], although it achieved remarkable success on the Open Catalyst Challenge [2], theemployed graph construction method breaks periodic invariance when applied to crystals.
Comparedwith Graphormer, Matformer is speciﬁcally designed for crystals considering both periodic invarianceand periodic patterns.
In addition, different from the node-wise attention used in Graphormer, our
Matformer employs edge-wise attention and aggregates information from edges as in Sec. 3.4.
Experimental studies
5.1 Experimental setup
We conduct experiments on two material benchmark datasets, including The Materials Project [14]and JARVIS [6].
Baseline methods include CFID [5], CGCNN [47], SchNet [38], MEGNET [3],
GATGNN [31], and ALIGNN [4].
Unless otherwise speciﬁed, for all the baseline methods, we [Table] [Table]
SchNet [38] 0.033 (0.035) 0.345 (-) 0.066 (-) 0.099 (-)
MEGNET [3] 0.030 (0.028) 0.307 (0.33) 0.060 (0.050) 0.099 (0.079)
GATGNN [31] 0.033 (0.048) 0.280 (0.31) 0.
ALIGNN [4] 0.
Matformer 0.
report the results taken from the referred papers or provided by original authors.
All Matformermodels are trained using the Adam optimizer [19] with weight decay [30] and one cycle learning ratescheduler [40].
We only slightly adjust learning rates from 0.001 and training epochs from 500 fordifferent tasks.
Detailed Matformer conﬁgurations for different tasks are provided in Appendix.
A.3.
5.2 The Materials Project dataset
We ﬁrst use The Materials Project-2018.6.1 dataset [3], which contains 69239 crystals, to evaluate Matformer.
We notice that previous works [47, 38, 3, 31, 4] compare with each other either usingdatasets of different sizes, or using datasets with the same size but splitting the datasets with differentrandom seeds.
To make the comparison clear and fair, we retrain all corresponding models usingexactly the same training, validation and test sets across all methods and report the results in Table.
To avoid confusion, we still put original results from referred papers in Table.
1 inside parentheses.
In particular, for tasks of formation energy and band gap, we directly follow ALIGNN [4] and usethe same training, validation, and test set, including 60000, 5000, and 4239 crystals, respectively.
Fortasks of Bulk Moduli and Shear Moduli, we follow GATGNN [31], the recent state-of-the-art methodfor these two tasks, and use the same training, validation, and test sets, including 4664, 393, and393 crystals. In Shear Moduli, one validation sample is removed because of the negative GPa value. We either directly use the publicly available codes from the authors, or re-implement models basedon their ofﬁcial codes and conﬁgurations to produce the results. Detailed conﬁgurations of theseretrained models are provided in Appendix.
A.4.
The used metric is test MAE following previousstudies [47, 38, 3, 31, 4].
It can be seen from Table.
1 that Matformer achieves the best performance on all tasks consistentlyby signiﬁcant margins.
Speciﬁcally, it reduces the formation energy by 4.5% of the second bestmodel, which is a signiﬁcant margin.
Furthermore, for Bulk Moduli and Shear Moduli tasks with only4664 training samples, Matformer achieves the best performances, indicating Matformer’s adaptiveability to tasks of small training scales. The overall excellent performances for different tasks on The Materials Project reveal the effectiveness of our proposed crystal graph representations in Matformer. 5.3 JARVIS dataset
JARVIS is a newly released database proposed by Choudhary et al. For JARVIS dataset, wefollow ALIGNN [4] and use the same training, validation, and test set. We evaluate our Matformeron ﬁve important crystal property tasks, including formation energy, bandgap(OPT), bandgap(MBJ),total energy, and Ehull. The training, validation, and test set contains 44578, 5572, and 5572 crystalsfor tasks of formation energy, total energy, and bandgap(OPT).
The numbers are 44296, 5537, 5537for Ehull, and 14537, 1817, 1817 for bandgap(MBJ). The used metric is test MAE. The results for allbaselines are taken from ALIGNN [4]. [Table]
The quantitative results for Jarvis are shownin Table.
Matformer outperforms the baseline methods signiﬁcantly on all of these ﬁvetasks.
Compared with ALIGNN, Matformer hasstronger discriminative ability due to explicitencoding of periodic patterns.
Speciﬁcally, Matformer reduces Jarvis Ehull by 0.012, which is15.8% of ALIGNN.
Furthermore, Matformerachieves the best performances for Bulk Moduli and Shear Moduli in the Mateirals Project with 4664training samples, and Bandgap(MBJ) in JARVIS with 14537 training samples, indicating its adaptiveability to tasks of various data scales.
Overall, the superior performances of our Matformer on varioustasks of two common material benchmarks show effectiveness of periodic pattern encoding andgreat potential of our Matformer message passing layer for crystal graphs.
In addition, comparedwith ALIGNN, the previous state-of-the-art method for crystal property prediction, our Matformeris more efﬁcient.
We evaluate the efﬁciency of Matformer by comparing with ALIGNN using JARVIS formation energy dataset.
The mean time of ten runs for training and inference time, usingcorresponding best model conﬁgurations of ALIGNN and Matformer, is reported.
And the total timeis computed by Time/epoch ⇥ corresponding training epoch number, which is 300 for ALIGNN and500 for Matformer.
In Table.
2, we show that Matformer is three times faster than ALIGNN in totaltraining time and near three times faster in inference time, for the whole test set.
[Table]
5.4 Ablation study
In this section, we demonstrate the importance of periodic invariance and explicit repeating patternencoding for crystal representation learning by ablation studies.
We also evaluate the building blocksparticularly designed for our Matformer.
Speciﬁcally, we conduct experiments on JARVIS formationenergy, and the test MAE is used as the quantitative evaluation metric.
[Table]
Periodic invariant graph construction .
We demonstrate the importance of periodic invariant graph construction by comparing radius multiedge graph, denoted as Radius, withthe graph construction method proposed by Graphormer, denoted as OCgraph.
Note that the constructed crystal graphs by OCgraph are super large,containing more than n 2 edges, wheren is the atom number in a cell.
We adjust the Matformer conﬁgurations totrain these large graphs on a single RTX A6000 GPU.
We use same model and training conﬁgurationsfor these two graph construction methods.
It can be seen from Table.
4 that when using OCgraph toour Matformer, the test MAE drops dramatically of 53% because of breaking periodic invariance,compared with radius-based multi-edge graphs used in Matformer.
Encoding of repeating patterns .
We denote periodic pattern encoding as PE.
In Table.
4, we showthat omitting the periodic pattern encoding results in a signiﬁcant drop of test MAE from 0.0325 to0.0337, revealing the importance of periodic patterns for crystal representation learning.
[Table]
Building blocks of Matformer .
Firstly, in Table.
4, we compare two periodic invariant graph construction methods described in Sec. 3.3.
We deonte fully connected graph witht smallest pairwise distances as T-fully, and we use t = 3 .
Results show that although both methods satisfy periodic invariance, Radius is better than T-fully.
In addition, we evaluate theoperation of sigmoid and normalization instead of softmax inour message passing, which is designed to capture the node degree information in multi-edge crystalgraphs, described in Sec. 3.4.
Compared with using softmax with scalar attention coefﬁcient, denotedas Softmax-scalar, or softmax with vector attention coefﬁcient, denoted as Softmax-vector, usingsigmoid and normalization leading to signiﬁcant performance improvement in terms of test MAE,justifying the effectiveness of this operation in Matformer message passing design, shown in Table.
[Table]
Conclusion
In this work, we ﬁrst propose to formally deﬁne periodic invariance and periodic pattern encodingfor periodic graph learning.
[-+ k 1 ℓ 
-]
We then propose Matformer for periodic graph representation learning,which is invariant to periodicity and can capture repeating patterns explicitly.
[-+ k 1 ℓ 
-]
Experimental results oncommon benchmark datasets show that our Matformer outperforms baseline methods consistently.
Inaddition, our results demonstrate the importance of periodic invariance and explicit periodic patternencoding for crystal representation learning.