Paper: Crystals with Transformers on Graphs, for predictions of crystal material properties (ICLR-2024)

Reviewer comment (Writing):
In section 3.1.3, if I understand correctly, $u'$ and $v'$ actually represent edges? If so, it's better to use $e$ notations to differ from the atom nodes. If this purely represents the angle between edges, pls make this definition more clear and draw connections with $t_{ijk}$ from Fig. 1.

Author response:
$\displaystyle u’ $ and $\displaystyle v’ $ represent edges, while $\displaystyle (u’, v’) $ represents $\displaystyle t_{ijk} $. I will use e notations as you advised in the revised edition.

Reviewer comment (Experiments):
In section 3.2, I understand the motivations of adding positional encoding. But I hope authors can give more insights on why multiple encodings need to be used? What if you only choose 1 or 2? Some ablation studies on this point would be helpful.

Author response:
I conducted an ablation study on the positional encoding, but I did not include this in my paper. PE 1 and 2 are the encoding for the structural information while PE 3 and 4 encode the connectivity, neither of them are neglectable. The ablation study proved this as well: the result in MAE has such order: with all PE < with PE 3 or 4 << with PE 1 or 2 < without PE.

Reviewer comment (Novelty):
There are many existing works on crystal materials property predictions. For example, [1, 2] uses GNN and contrastive learning to predict density of states, and [2] uses prototypical classifiers to analyze crystal structures. These works are worth discussing in the related work.

Author response:
There are many existing works on this topic, where I may have missed some important ones. I will read these papers and add them to the discussion in the revised edition.

Reviewer comment (Presentation):
To facilitate understanding, it would be better if you can use a running example (e.g., one simple crystal structure) like Fig 2 to explain how each module and inputs/outputs are hooked.

Author response:
That is a good idea to make the paper clearer and easier to understand. I will try to add this in the revised edition.

Reviewer comment (Experiments):
I understand you used kNN to build up edges between atoms. I wonder what if you choose k=8 or k=6. How much difference does it make? Also, is it for simplicity purpose that you set the same k for all atom nodes?

Author response:
I have also tried k=8 and varied k. Firstly, k=8 and k=6 are less reasonable chemically than k=12 because there are many cases in the crystals that atoms have 12 neighbors such as close-packed structures (see the right half of Figure 2, 12 atoms surround the orange atom in the center). This is also proved in the test result: with the same split in training set and validation set, the MAEs are 0.0242, 0.0261, 0.0346 for k=12, varied k, k=8, respectively. The varied k may lead to some loss of structural information of local environment and too much complexity.

Reviewer comment (Theory):
For GwT, it seems that only the neighboring edges are added through $e_{ij}$, why it can capture long-range interactions? Do you expect that when N grows larger, it keeps drawing information from more distant neighbors?

Author response:
The GwT is a graph-wise transformer, capturing long-range interactions by calculating the attention across all nodes in the graph, with structural and connectivity information added as the PE. The neighboring edges are also considered as a supplementary of the PE for connectivity.

Reviewer comment (Presentation):
I didn't see $t_{ijk}$ in Fig 4. Is it used in encoding?

Author response:
It is not used in encoding. We have 2 eTGC layers in a single eTGC block, taking $\displaystyle e $, $\displaystyle t $ and $\displaystyle x $, $\displaystyle e $ as input respectively. tijk is the edge feature in the line graphs, representing the angle between the bonds. We only demonstrated one eTGC layer in Figure 4 for simplicity, while there are 2 eTGC layers taking different input.

Reviewer comment (Presentation):
For edge-engaged transformer graph convolution, does it involve any convolution operations in Fig. 4 (a)?

Author response:
The message passing with transformation is mainly in the multi-head neighbor attention, which was expanded in Figure 4c.

Reviewer comment (Novelty):
It is not described why modeling the long-range interaction is the key to modeling the periodic molecules. To my understanding, the tackled GNN's limitation is a general problem in the graph field and does not need to be restricted in modeling the periodic structures.

Author response:
It is true that the tackled GNN’s problem is a general problem in the graph field. However, from another aspect, periodicity is not the only character of crystals. Chemical properties and activities of crystals are often strongly influenced by long-range interactions such as long-range symmetry and stacking patterns, sometimes even more than the ionic bonds in short-range [1, 2]. Unlike those dissociated small molecules where long-range interactions like coulomb forces and van der Waals forces can be calculated with equations, the long-range interaction in crystals spans a way larger range and much more complexity to model. From this point, crystallography system is a good demonstration of the tackled GNN’s problem.

Reviewer comment (Evaluation):
From the results of Figure 5 and 6, the effectiveness of modeling the long-range interaction by using the transformer seems to be insignificant, even though the long-range interaction is the main point of this paper. In other words, if the number of eTGC blocks becomes larger, the effect of GwT seems to be weakened and the performances are similar.

Author response:
As in the paragraph above Figure 6, the crystal cells are much smaller in dataset log_gvrh than in mp_e_form. This makes long-range interactions not able to be long enough. However, the GwT is essential in modelling large crystal structures.

Reviewer comment (Evaluation):
Even though the proposed method leverages the transformers which require more memory space, the performances are not significant compared to the baselines.

Author response:
Although the transformer-based structure requires a relatively large memory space, the baseline which take direct graph, line graph and dihedral graph as input requires more.

Reviewer comment (Writing):
It is not explained what the tasks are. The material properties may not be familiar to the AI researchers and noting the tasks as abbreviations without any explanation could confuse the readers.

Author response:
The target of the tasks will be explained in the appendix in the next edition. Thank you for this advice.

Reviewer comment (Experiments):
If many GwT blocks make the local information from the eTGC lost, have you ever applied the eTGC and GwT by crossovering them? (for example, $\text{GwT}(\text{eTGC}(\mathcal{G}, L(\mathcal{G})) \times N$)

Author response:
Following your suggestion, we tried this type of structure, however, found it not as competible as the original one. Performance of GwT(eTGC(G, L(G)) x 3) x 3 was not significantly different from GwT(eTGC(G, L(G)) x 3) x 1.

Reviewer comment (Theory):
Why do you use softmax attention in eTGC?

Author response:
The major incentive of using softmax attention is to normalize the total message passed to a single node.

Reviewer comment (Experiments):
Why do you use the simple transformer architecture for GwT not the graph transformer architectures?

Author response:
I use the traditional transformer to take non-ordered datasets. I use the simple transformer to calculate attention across the long-range interactions.

Reviewer comment (Experiments):
Why is it insufficient to use deeper GNN layers instead of using transformers?

Author response:
Similar with comment 1, we sometimes need to construct the graph from very large structures that contain up to hundreds of atoms instead of a small repetitive unit. Thus, the depth of GNN is never deep enough. In Figure 5 (left), the average of atoms in a crystal cell is 29, and the transformer is not likely able to be substituted by deeper GNN layers.

Reviewer comment (Theory):
The positional encoding is not E(3) invariant and thus is not suitable for modeling the properties of materials. Nodes of the crystal graph are represented with a 3D coordinate of the particular atom in the crystal grid. For a machine learning model to successfully predict the material properties, the feature representation needs to be equivariant or invariant towards E(3) transformations of these coordinates. The proposed positional encoding is not invariant or equivariant to these transformations.

Author response:
Dear reviewer, 

 

Thank you for your time on reviewing our manuscript. I appreciate your recognition of the importance of this topic. However, there are certain scientific flaws in your comment. You may be an expert in machine learning, but clearly you have LITTLE understanding in crystallography. 

 

E(3) invariance is crucial in modelling dissociated small molecules, however, this does NOT apply to crystal materials. Crystal cells are NOT applicable for the rotations due to the certain construction rules. a, b and c axis are NOT exchangeable. Also, restricted by the PBC nature, crystal cells can expand in the space ONLY in a certain direction, especially in hexagonal, trigonal, monoclinic and triclinic crystals.

Reviewer comment (Theory):
Cartesian coordinates: If I understand correctly, then these are extrinsic coordinates. Any function that directly depends on these extrinsic coordinates is not equivariant to E(3) transformations.

Author response:
3. Cartesian coordinates and fractional coordinates are extrinsic, Laplacian encoding and random walk encoding are intrinsic.

Reviewer comment (Novelty):
Multi Head Neighbor Attention appears to be just normal multi head attention on the line graph and should be stated as such and not be presented as a new method.

Author response:
1. Multi head neighbor attention is NOT just normal multi head attention. As I described in section 3.3.1, The k and v are summed with information from the central node, neighboring node and the edge. Nevertheless, the word neighbor appears mainly to discriminate attention calculated on neighbors and self attention which later applied in GwT.

Reviewer comment (Writing):
Introduction, 2nd paragraph, contains a typo: "enxtends" should be corrected.

Author response:
2. Thank you for making some contribution to the revision by locating my typo.

Reviewer comment (Writing):
Which of the positional encodings are intrinsic and which are extrinsic?

Author response:
3. Cartesian coordinates and fractional coordinates are extrinsic, Laplacian encoding and random walk encoding are intrinsic.