Paper: EGraFFBench: Evaluation of Equivariant Graph Neural Network Force Fields for Atomistic Simulations (ICLR-2024)

Reviewer comment (Reproducibility):
I have serious doubts about the experimental results in Table 1. The MD17 results of NequIP and Equiformer are much worse than reported in their original papers. Although BOTNet and MACE report rMD17 results instead of MD17 in their papers, it doesn’t make any sense that their performance on MD17 will be such bad. I don’t think authors run their code correctly.

Author response:
We thank the reviewer for the careful reading and critical comments. We appreciate the opportunity to address your concerns, which has indeed enhanced the quality of our work. Please find below our detailed response to the issues raised:

1. Rectification of Methodological Issues: We are grateful for your observation regarding the high MAE values. We carefully looked at the codes, log files, and the datasets, and identified an issue related to the MACE model for the MD17 dataset. Specifically, while training the models on MD17, the periodic boundary condition was kept as 'True' while keeping the box large enough to avoid any self interaction. However, the internal architectural setup of MACE (and BOTNet) requires the lattice parameter to be specifically mentioned when the PBC is set to "True", even for small molecules. To address this concern, we have now re-trained all the models on the MD17 dataset without PBC and updated all the Tables and results associated with it. We believe these revised results align more closely with the expected outcomes based on the original papers of the compared models.

2. Comparison with literature data: Further to compare our reported results with existing literature studies, we have included a new Table 19 in our revised manuscript. This table provides a direct comparison of our results with those reported in the existing literature. It can be observed that the values obtained in the present work are comparable with those available in the literature. We believe this addition will offer a clearer context and a more comprehensive understanding of our experimental outcomes.

However, it should be noted that the results cannot be directly compared with the literature in many cases as the literature does not clearly provide the exact training-dev-test dataset. To address this concern, we have included the exact training-dev-test datasets, codes, hyperparameters, config files, and log files in the repository associated with the present work. For example, DimeNET and PaiNN model in Fu et al. was trained on 9500 configurations. In our study, we have used a training set of 950 configurations across all the models for a fair comparison. The dataset size was limited due to the high computational cost in transformer models. 

Overall, with the release of all hyper-parameters, train-dev-test splits, etc., we believe we present the most transparent and reproducible evaluation of GNN force fields. Nonetheless, if there exist any outstanding concerns regarding codes or results, we request the reviewer to raise any of those.

Reviewer comment (Experiments):
Since the molecular dynamic simulations are based on frozen models that are trained to predict energy and force, the quality of Table 1 makes the MD simulation not convincing.

Author response:
We agree with the reviewer that higher energy and force error in training will make the MD simulation unrealistic. As mentioned in the previous comment, we have now retrained the models for the MD17 datasets, and performed MD simulation with the new trained models and corrected the results accordingly in Tables 1, 2 and 3. 

Moreover, as mentioned earlier, all the codes and datasets to reproduce to the present work are made available in the GitHub (https://anonymous.4open.science/r/MDBENCHGNN-BF68). The logfiles are also included for easy analysis. With these additional data, we hope the reviewer can evaluate the results and authenticity of the present work in a transparent fashion. 

Nevertheless, if there are any outstanding concerns, we request the reviewer to raise those.

Reviewer comment (Novelty):
Authors claim that they propose three new challenging tasks. However, the evaluation of model generalizability to higher temperatures (Sec 4.4.2) is not new. Actually, it’s proposed by LinearACE [1] and this task has also been studied by Allegro, BOTNet, and MACE. Although these works are focused on energy and force errors, the conclusion that “OOD is challenging” is not surprising to me.

Author response:
We agree with the reviewer's comment that generalizability to higher temperatures (Sec 4.4.2) is not new, it is proposed by LinearACE [1]. However, the evaluation metrics used to evaluate the performance at higher temperatures are indeed new. Nevertheless, to address the comment we change the following text in the main manuscript abstract.  
*'three new challenging tasks' to 'three challenging tasks'*

While the conclusion 'OOD is challenging' is not surprising, the specific question we have tried to raise from the task proposed on LIPS20 is that whether a system trained on a wide range of compositions in the liquid state can reasonably capture the dynamics of a crystalline system or other disordered systems. This is a task of high importance in materials discovery. Specifically, it is a common practice while training machine learning potentials to train the system on a liquid and in many cases, use the forcefield to discover new crystalline structures. The present work shows that caution should be exercised when using ML potentials for such task. 

We hope the reviewer appreciates the relevance of this conclusion.

Reviewer comment (Novelty):
Concluding insights in Sec 5 are not surprising to researchers studying graph neural networks for force fields. Technical novelty is very limited.

Author response:
We want to reiterate that the present work aims at benchmarking equivariant graph neural network forcefield and is the first work in this direction. Accordingly, the work is submitted to the benchmarking and dataset topic of ICLR. Indeed, we agree that some of the conclusions may not be surprising to researchers working on forcefield. However, the main novelty of the present work is to propose challenging tasks, rigorous metrics, and use them to evaluate equivariant GNN forcefields, on new datasets. Such a work can inform the future work in terms of the choice of potentials for specific tasks, and systems.

Reviewer comment (Writing):
In Figure 3, does PDF refer to Pair Distribution Functions?

Author response:
Indeed, PDF refers to the Pair Distribution Functions.

To address the comment, we have changed the caption of Figure 3 in the main manuscript, as below.

>Figure 3 :Pair distribution function(PDF) over the simulation trajectory. Reference PDF in red and generated PDF in blue represent ground truth and predicted PDFs. The values are computed as the average of five forward simulations for 1000 timesteps on each dataset with different initial conditions

Reviewer comment (Evaluation):
Errors in energy and force are commonly used as evaluation metrics for force field prediction. The EV and FV proposed by this work are similar to these two metrics. Considering RDF is a good idea, but there’re more metrics can be considered to evaluate molecular dynamics. For example, RMSD and temperature are used in [2]. Could you discuss why other metrics like RMSD and temperature are not selected as metrics in this benchmark?

Author response:
Thank you for the suggestions. We would like to clarify that the Energy Violation (EV) and Force violation (FV) proposed in this work are evaluated over MD simulation trajectories. Thus, these metrics show how the energy and force diverge from the ground truth, when evaluated on an MD simulation trajectory. In contrast, the MAE or RMSE are computed over the static structure. Thus, EV and FV provide insights into the stability of the MD simulations. We also draw attention to Figures 4 and 5 in the Appendix, where this is clearly elucidated.

We agree that in addition to the structure and dynamics metrics proposed in the work, RMSD could be an additional metric that captures the dynamics of the simulation. Accordingly, we have now added RMSD plots for all models in the appendix section A.9. 

We could not add temperature metric as some ground truth datasets do not contain velocity information. Accordingly, we did not include this metric. Nevertheless, we agree that this could also be an additional metric that captures the stability of the simulation.

**Appeal to the reviewer:** With the additional experiments, results, and explanations, we now hope the reviewer finds the manuscript suitable for publication. Accordingly, we request you to raise the score for the manuscript. Please do let us know if there are any further queries.

Reviewer comment (Experiments):
I believe this would be difficult to do given the amount of work it would take, but it would be really valuable to have more of an ablation style study of what particular architectural choices help / hurt in different metrics. Even with the present benchmarks, there are many architectural difference between these models that the takeaways are a bit binary -- e.g. this model is or is not enough for this task -- rather than -- e.g. this architectural choice seems to help with X. If you think that your results can support such guidance I think that would be extremely useful to the community.

Author response:
Indeed, we observe that models perform different across different atomic systems. This could be attributed to the differences in the nature of bonding, size of the molecules or atomic systems, and the nature of the training set in terms of how much scenario it covers. A detailed analysis of which features associated with each architecture benefits a given dataset is proposed as a future work. 

To address this comment, we have now included additional text in the Limitations and future work section of the main manuscript as follows.
> Another interesting aspect is the empirical evaluation of which particular architectural feature of a model helps in giving a superior performance for a given dataset or system (defined by the type of bonding, number of atoms, crystalline vs disordered, etc.). Such a detailed analysis can be a guide to designing improved architecture while also providing thumb rules toward the use of an appropriate architecture for a given system.

Reviewer comment (Evaluation):
The work neglects some useful metrics from the previous works, like stability, RDF, diffusivity, etc in [1]. It would be better to include the performance of latest models on these metrics.

Author response:
Thank you for raising this concern. Indeed, previous work referred to by the reviewer uses stability, RDF, and diffusivity. Among these metrics, while stability is an evaluation of dynamics, RDF is an evaluation of the structure. In the present work, we have proposed metrics that capture both structure and dynamics in a more systematic fashion. Specifically, to evaluate the structure, we have already added two RDF-based metrics, namely JS Divergence and Wright Factor. While the first metric captures how close the distributions of predicted and actual RDF is, the second metric represents the relative error between the predicted and actual RDF. Similarly, stability of the dynamics is captured through energy and force violation. Specifically, in these two metrics, we compare the energy and force on a molecular dynamics (MD) simulation performed with the machine learned potential with respect to the ground truth trajectory. Thus, the force and energy violation error clearly evaluate the divergence of force and energy of the simulated trajectory with respect to the ground truth. These metrics, in turn, give a clear indication of the stability of the simulation. 

The third metric proposed, namely, diffusion is not necessarily metric; it is rather a property of the system which is derived from the dynamics. Note that there are several such properties which can be obtained from the MD simulations. Nevertheless, in order to address the concern, we have now **included RMSD** (note that diffusion constant is the slope of MSD with respect to time) of the trajectories for all the datasets and models in Appendix A.9 (see Figs. 8 and 9).

Reviewer comment (Novelty):
Though new datasets and metrics are proposed, the major conclusion that low energy/force doesn't guarantee performing well in molecular simulations is not fresh, as pointed out in [1].

Author response:
We agree with the reviewer that the low energy or force doesn't inherently ensure superior performance in molecular simulations, which has also been highlighted previously [1]. However, we do not consider this to be the major contribution of the work. 

The main contributions of our work are in the novel tasks proposed to evaluate equivariant graph neural network forcefields as detailed below. 
1. **Complex tasks:** We introduce complex tasks that evaluate ML potentials for realistic applications. This involves training on diverse liquid systems with varying chemical compositions (LiPS20, see App. A.2, Table 11) and assessing performance on the dynamic simulations of unseen crystalline and disordered structures. To the best of our knowledge, this is the first time such a benchmarking task has been performed on equivariant graph neural network force fields (see Sec 4.4). Such tasks can be a critical one for evaluating any new forcefield for materials discovery.
2. **Equivariant graph neural network forcefield:** This is the first work that evaluates state-of-the-art equivariant graph neural network forcefields in a consistent fashion on the same dataset and on challenging tasks with rigorous structure and dynamics metrics. To further emphasize this, we show table 21 and table 10, the datasets on which previous works have evaluated the equivariant GNN forcefields. It can be observed that there is a clear lack of systematic evaluation of these forcefields.

We also thank the reviewer for noting the contribution of new datasets and metrics in our work. In addition, the key insights obtained from the work are included in the conclusion section in a point-by-point fashion.

Reviewer comment (Experiments):
Some benchmarking settings may not be convincing and not reflect the scenario in real applications. More discussions are included in the following Questions section.

Author response:
**Response:** In our assessment, we've thoroughly considered the questions raised regarding the benchmarking settings. Our approach aims to encompass a diverse range of tasks and metrics to gauge performance comprehensively. We have addressed the concerns raised by the reviewer in the Questions section. We're open to further discussions to refine our benchmarking methodology and ensure a more accurate representation aligned with real applications.

Reviewer comment (Experiments):
The work includes latest equivariant models. However, most models reported in previous works [1] are ignored. The authors may consider adding some more models to further validate the limitations of previous datasets and metrics.

Author response:
The primary objective of this work is to benchmark state-of-the-art **equivariant** models. Among those mentioned in [1], only NequIP, GemNET-dT and PaiNN are equivariant. We have already included NequIP in the present work. GemNET-dT lacks energy conservation and hence is not suited to be used as a forcefield. 

To address reviewer's comment, we have now included **two additional models, namely, PaiNN and DimeNet** in the work. See updated Tables 1, 2, and 3 and Section 3 in the main text.

Reviewer comment (Experiments):
In section 4.4.1, ML models are trained on a subset of 3 molecules in MD17 and evaluated on another molecule. Not surprisingly, none of the models perform well in this setting as the types and number of molecular data are limited. However, this may not reflect the real application. [2] unveils that allegro pre-trained on SPICE, a large dataset with molecules < 100 atoms, can conduct molecular simulations of large molecular systems with 1M atoms. So a more realistic setting for OOD generalization may be training on large datasets with a wide variety of small molecules and test how it performs on other molecular systems.

Author response:
We agree with the reviewer that the test on unseen molecule after training on a single molecule may not be a realistic scenario. Accordingly, we have now moved this example to the Appendix (see App. 12).

However, the scenario that the reviewer has proposed is exactly what we have performed on the LiPS20 dataset. Here, we train the model on several different compositions, and system sizes. Then, the model is tested on unseen crystalline and disordered compositions to check the models OOD generalization. See the details of the LiPS20 data in Appendix A.2. Further, the details of the train test split are also included.

Reviewer comment (Evaluation):
For the dynamics metrics, if two molecular systems start from the same initial structure but different initial velocities, they can diverge later. When that happens, EV & FV may fail to provide meaningful evaluations. Are there controls over the initial configuration when comparing ML-based simulations with ground truth?

Author response:
We agree with the reviewer that the EV and FV metrics provide meaningful evaluation only when initial conditions are kept same for ML-based and ground-truth simulation. We have ensured in our evaluations that the initial structure and velocities are the same. To address the comment, we include the following text in the main paper Section 4.3.2:

>To evaluate the ability of the trained models to simulate realistic structures and dynamics, we perform MD simulations using the trained models, which are compared with ground truth simulations, both employing the same initial configuration and velocities

Reviewer comment (Reproducibility):
How does EGraFFBench handle periodic boundary condition (PBC)?

Author response:
In all datasets, a parameter defining the periodic boundary condition across three directions is maintained. It is set as 'True' for bulk systems such as LiPS, LiPS20, and GeTe, and 'False' for molecular systems like 3BPA, acetylacetone, and MD17 molecules.

Reviewer comment (Presentation):
In Table 2, some models are not properly highlighted though superior performance is achieved.

Author response:
Thank you for raising this concern, we have updated Table 2 with corrected cell coloring.