Paper: Equivariant Graph Network Approximations of High-Degree Polynomials for Force Field Prediction (ICLR-2024)

Reviewer comment (Experiments):
I recommend adding some experiments to empirically show (1) whether PACE empirically approximates the equivariant high-degree polynomials; and (2) whether PACE can surpass existing works on approximating these polynomials.

Author response:
**W1.** Thanks for your comments. We will further consider this experiment in future work.

Reviewer comment (Experiments):
Some other highly-related popular benchmarks, such as AcAc, which is adopted in (Batatia et al. 2022a; Batatia et al.2022b) should also be included.

Author response:
**W2.** Thanks for your suggestion. During the discussion period, we conducted experiments on AcAc and added them to Appendix D1 in the revised paper. 

|      |   | BOTNet     | NeuqIP      | MACE       | Ours            |
|------|---|------------|-------------|------------|-----------------|
| 300K | E | 0.89 (0.0) | 0.81 (0.04) | 0.9 (0.03) | **0.81** (0.05) |
|      | F | 6.3 (0.0)  | 5.90 (0.38) | 5.1 (0.1)  | **4.8** (0.3)   |
| 600K | E | 6.2 (1.1)  | 6.04 (1.26) | 4.6 (0.3)  | **4.3** (0.3)   |
|      | F | 29.8 (1.0) | 27.8 (3.29) | 22.4 (0.9) | **21.2** (0.7)  |

Reviewer comment (Presentation):
I think Section 4.3 should be “Algorithm efficiency” instead of “Ablation study”.

Author response:
**W3.** Thank you for your advice. Since we performed more experiments this week, we modified Section 4.3 to "More Empirical Analyses", and now "Algorithm Efficiency" is under it. The text and table of "Algorithm Efficiency" are moved to Appendix D4 in the revised paper.

Reviewer comment (Experiments):
The empirical analysis could be enhanced to better support the motivation. As I understand, the primary contribution of PACE, as compared to MACE, is the introduction of self-interaction layers that map the atomic base $A_i$ to different subspaces for different body orders, thereby increasing expressive power of PACE. However, the paper lacks an ablation study directly analyzing the significance of this operation. Although MACE could be considered a natural ablative baseline, it's unclear whether the two models share the same hyper-parameters, such as those mentioned in Table 5. I recommend that the authors conduct a direct ablation experiment to clarify this point.

Author response:
**W1.** Thank you for suggesting the direct ablation experiment. During the discussion period, we conducted this ablation experiment on three molecules from rMD17, and the comparison is shown below. In this experiment, we removed additional self-interaction operations and used the same hyperparameters. The results confirm the effectiveness of our polynomial many-body interaction module. We have added this ablation experiment to Appendix D3 in the revised paper.

|         |   | PACE (-SI) | PACE |
|---------|---|------------|------|
| Ethanol | E | 0.4        | **0.3**  |
|         | F | 2.3        | **2.0**  |
| Toluene | E | **0.2**        | **0.2**  |
|         | F | 1.4        | **1.2**  |
| Uracil  | E | **0.3**        | **0.3**  |
|         | F | 2.3        | **2.0**  |

Reviewer comment (Writing):
Here is a sentence not easy to follow. In the 3rd paragraph in sec 3.1, ``While TFN has the ability to approximate all equivariant polynomial functions with an infinite number of layers, the approximation capacity of the equivariant message passing scheme is inherently limited.'' The authors should clarify the logic between these sentences.

Author response:
**Q1.** Thanks for pointing it out. In the revised paper, we have modified this sentence to the following expression.
“While TFN has the ability to approximate all equivariant polynomial functions with an infinite number of layers, the approximation capacity of a single TFN layer is limited, raising the need to develop modules improving the approximation capacity.”

Reviewer comment (Evaluation):
Considering that all baseline models except MACE possess the universal approximation ability for all equivariant polynomials, it's puzzling why they do not achieve performance comparable to PACE. Could the authors elaborate on this?

Author response:
**Q2.** Thanks for this question.  We wish to clarify that these architectures can achieve universal approximation to all equivariant polynomial functions up to the highest degree $D$, denoted as $\mathcal{P}^{D}$. As shown in Table $1$, a single PACE layer can achieve $D=v=3$ in the experiment setup involving two layers, whereas a single NequIP can only reach $D=1$, and Allegro, with its three layers, attains $D=3$.

Reviewer comment (Evaluation):
The improvement in the experiments is so marginal. Recent years, people [1] have already noticed that achieving a small energy and force error is only the first step towards a successful neural network force field. The advantage of a neural network force field should be judged on the prediction of macroscopic properties and sampling of conformations on a long-time large-scale molecular dynamic simulation. An improvement on the energy/force error does not necessarily lead to a better MD prediction, unless the improvement is very obvious. Based on the numbers shown in Table 2 and 3, as well as the limit of the datasets, I'm not convinced that the proposed model is a better model than the baselines. Also the proposed model is slower than MACE, making it more useless in applications.

Author response:
**W1.** During the discussion period, we performed MD simulations using our PACE model on three molecules, Aspirin, Ethanol, and 3BPA. To compare the MD trajectories generated by PACE with those generated by our baseline MACE, we compute Wright's Factor (WF) [1] as a metric and present the comparison in the table below. Here, WF quantitatively measures the difference between the RDFs of the ground truth and the generated MD trajectories. For each method, we take the average of 5 MD simulations for 1000 timesteps with 5 different initial structures. 

|            |  MACE |      PACE |
|------------|------:|----------:|
| Aspirin    |  7.15 |  **6.88** |
| Ethanol    | 19.16 | **16.86** |
| 3BPA 300k  |  2.14 |  **1.98** |
| 3BPA 600k  |  3.01 |  **3.00** |
| 3BPA 1200k |  2.68 |  **2.40** |

In addition, we also visualize the radial distribution functions (RDFs) of the ground truth MD trajectories and PACE-generated MD trajectories, which can be found in Figure 5 in the revised paper. These results of the MD simulations further affirm that our PACE model not only achieves state-of-the-art (SOTA) performance in force field prediction but also holds practical value in realistic simulations.

We have added MD experiments to Appendix D2 in the revised paper. We agree that low energy and force errors do not guarantee better MD simulations. However, there’s still a lack of standards to evaluate the complex MD simulations. Currently, energy and force errors are still the most common metrics to assess the ability of machine learning models to predict force fields [2].  We will consider developing machine learning models more specifically for realistic MD simulations in future works.