Paper: EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representations (ICLR-2024)

Reviewer comment (Evaluation):
Although the ablation studies show the improvement of proposed modules, the results on QM9 is similar compared to Equiformer.

Author response:
We disagree. Reiterating results from Table 5 in the paper, EquiformerV2 achieves better results than Equiformer on 9 out of 12 tasks on QM9, is similar on 1 task, and worse on 2. 

As for the much larger and diverse OC20 dataset, EquiformerV2 is considerably better (Table 6), by up to 25.8% on forces and 4.2% on energies. 

In Section 4.3, we do acknowledge that the performance gain from using higher degrees and the improved architecture are not as significant on QM9 as they are on OC20. This is consistent with prior work, which has shown that trends on small and large datasets are not always consistent [1]. Due to the small size of QM9, and limited diversity in atom types, counts and angular variation, higher degree networks are more prone to overfitting. In Section 4.3, we also outline potential ways of mitigating this overfitting by first pre-training on large datasets and then transferring to smaller datasets, as has recently been demonstrated by [2].

Reference:

[1] Gasteiger et al. GemNet-OC: Developing Graph Neural Networks for Large and Diverse Molecular Simulation Datasets. TMLR 2022.

[2] Shoghi et al. From Molecules to Materials: Pre-training Large Generalizable Models for Atomic Property Prediction. ArXiv 2023.

Reviewer comment (Experiments):
Although the authors did a fantastic job on the experiments, EquiformerV2 is an incremental improvement over existing methods of both eSCN and Equiformer w.r.t. theory. And the novelty lies in those three specific techniques and enhancements. To see if these techniques are generalizable, I would like to see the ablation study of attention re-normalization, separable $S^2$ activation, and separable layer normalization, respectively, on the QM9 dataset like what the authors did in Table (a) for OC20.

Author response:
We conduct ablation studies similar to Table 1(a) using the task of $\Delta \varepsilon$ on QM9 and compare with the Equiformer baseline [1]. The mean absolute error (MAE) results are as below.

| Index  | Attention re-normalization |   Activation  |     Normalization    | $L_{max}$ | MAE (meV) |
|:------:|:--------------------------:|:-------------:|:--------------------:|:-------:|:---------:|
| 0      | Equiformer baseline        |               |                      | 2       | 29.98     |
| 1      |                            |      Gate     |      Layer norm      |    4    |   30.46   |
| 2      |              ✔             |      Gate     |      Layer norm      |    4    |   29.51   |
| 3      |              ✔             |      $S^2$      |      Layer norm      |    4    |   30.23   |
| 4      |              ✔             | Separable $S^2$ |      Layer norm      |    4    |   29.31   |
| 5      |              ✔             | Separable $S^2$ | Separable layer norm |    4    |   29.03   |

- Index 1: Naively increasing $L_{max}$ from 2 to 4 and using eSCN convolutions degrade the performance. This is due to overfitting since the QM9 dataset is overall smaller, and each structure in QM9 has fewer atoms, less diverse atom types and much less angular variations than OC20 and OC22. 
- Index 2: Attention re-normalization improves the MAE result.
- Index 3: Although using $S^2$ activation is stable here (unlike OC20), it results in higher error than using gate activation (Index 2) and the Equiformer baseline (Index 0).
- Index 4: The proposed separable $S^2$ activation achieves lower error than gate activation and $S^2$ activation.
- Index 5: The proposed separable layer normalization further improves the result.

Comparing Index 0 and Index 5, the three proposed architectural improvements are necessary to achieve better results than the baseline when using higher degrees on QM9. Overall, these ablation results follow the same trends as OC20.

We have added this ablation study to Table 13 in the revision.

[1] Liao et al. Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs. ICLR 2023.

Reviewer comment (Writing):
Equation (4) in Appendix A.3: Commonly, the left side of an equation is used for assigning new notation. I recommend write $D^{(L)} = D^{(L)}(R_{ts})$ and $\tilde{x}_s^{(L)} = D^{(L)} x_s^{(L)}$ for a degree $L$ before Equation (4).

Author response:
Thanks! We have updated this.