Forces are not Enough: Benchmark and Critical Evaluation for Machine Learning Force Fields with Molecular Simulations
Anonymous Author(s)
AffiliationAddress email
Abstract
Molecular dynamics (MD) simulation techniques are widely used for variousnatural science applications.
Increasingly, machine learning (ML) force field (FF)models begin to replace ab-initio simulations by predicting forces directly fromatomic structures.
Despite significant progress in this area, such techniques areprimarily benchmarked by their force/energy prediction errors, even though thepractical use case would be to produce realistic MD trajectories.
We aim to fillthis gap by introducing a novel benchmark suite for ML MD simulation.
Wecurate representative MD systems, including water, organic molecules, peptide, andmaterials, and design evaluation metrics corresponding to the scientific objectivesof respective systems.
We benchmark a collection of state-of-the-art (SOTA) ML
FF models and illustrate, in particular, how the commonly benchmarked forceaccuracy is not well aligned with relevant simulation metrics.
We demonstratewhen and how selected SOTA methods fail, along with offering directions forfurther improvement.
Specifically, we identify stability as a key metric for MLmodels to improve.
Our benchmark suite comes with a comprehensive open sourcecodebase for training and simulation with ML FFs to facilitate further work.
Introduction
[Equation]
T
F o r c e M AE
RD F M AE
S t ab ili t y
Molecular Dynamics (MD) simulations provide atomistic insights into physical phenomena in materials and biologicalsystems.
Such simulations are typically based on force fields(FFs) that characterize the underlying potential energy surface(PES) of the system and then use Newtonian forces to simulatelong trajectories [14].
The PES itself is challenging to computeand would ideally be done through quantum chemistry whichis computationally expensive.
Traditionally, the alternativehas been parameterized force fields that are surrogate modelsbuilt from empirically chosen functional forms [21].
Recently,machine learning (ML) force fields [59] have shown promiseto accelerate MD simulations by orders of magnitude whilebeing quantum chemically accurate.
The evidence supporting the utility of ML FFs is often based on their accuracy inreconstituting forces across test cases [12].
The evaluations invariably do not involve simulations.
[Figure] However, we show that force accuracy alone does not suffice for effective simulation (Figure 1,
Force MAE and RDF MAE are the lower the better.
Stability is the higher the better.).
MD simulation not only describes microscopic details on how the system evolves, but also entailsmacroscopic observables that characterize system properties.
Calculating meaningful observablesoften requires long simulations to sample the underlying equilibrium distribution.
These observablesare designed to be predictive of material properties such as diffusivity in electrolyte materials [67],and reveal detailed physical mechanisms, such as the folding kinetics of protein dynamics [33, 36].
Although these observables are critical products of MD simulations, systematic evaluations havenot been sufficiently studied in existing ML FF literature.
To gain insight into the performance ofexisting models in a simulation setting, we propose a series of simulation-based benchmark protocols.
Compared to the popular multistep prediction task in the learned simulator community [43], MDobservables focus on distributional quantities.
The exact recovery of the trajectories given the initialconditions is not the ultimate goal.
Evaluating learned models through MD simulations requires careful design over the selection ofsystems, the simulation protocol, and the evaluation metrics : (1) A benchmark suite should coverdiverse and representative systems to reflect the various challenges in different MD applications.(2) Simulations can be computationally expensive.
An ideal benchmark needs to balance the costof evaluation and the complexity of the system so that meaningful metrics can be obtained withreasonable time and computation.
(3) Selected systems should be well studied in the simulationdomain, and chosen metrics should characterize the system’s important degrees of freedom orgeometric features.
Are current state-of-the-art (SOTA)
ML FFs capable of simulating a variety of MD systems?
Whatmight cause a model to fail in simulations?
In this paper, our aim is to answer these questions with anovel benchmark study.
The contributions of this paper include:
• We introduce a novel benchmark suite for ML MD simulation with simulation protocols and quantitative metrics.
We perform extensive experiments to benchmark a collection of SOTA
ML models.
We provide a complete codebase for training and simulating MD with ML FFs
to lower the barrier to entry and facilitate future work.
• We show that many existing models are inadequate when they are evaluated on simulation-
based benchmarks, even when they show accurate force prediction (as shown in Figure 1).
• By performing and analyzing MD simulations, we summarize common failure modes and
discuss the causes and potential solutions to motivate future research
Datasets and Metrics
We refer interested readers to Appendix A for a discussion on related work and Appendix B for aintroduction to the problem setting of learning to simulate MD.
We note that, popular benchmarkdatasets, such as MD17, focus on the force prediction task for gas-phase small molecules.
However,successes in these tasks are not sufficient evidence for (1) capturing complex interatomic interactionsthat are critical for condensed phase systems; and (2) recovery of critical simulation observables thatcannot be directly indicated by force prediction accuracy.
We wish to extend existing benchmarks toexpand coverage of the types of systems that manifest complex intermolecular interactions at multiple [Table] [Table] scales.
Furthermore, beyond force predictions
[Table] we conduct simulations and benchmark observablesthat reflect the actual simulation quality [Table] along with stability and computational efficiency.
We choosefour representative MD systems (summarized in Table 1): small molecules (MD17), liquid water(water), peptide (alanine dipeptide) [Table] and solid-state materials (LiPS).
To facilitate relatively easy andfast benchmarking, we limit the size of the systems to less than 200 atoms.
Detailed description foreach dataset is in Appendix D.
Quantifying simulation stability.
ML FFs can produce unstable dynamics, as the learned force fieldmay not extrapolate robustly to the undersampled configuration space.
As a result, the trajectoriescan enter nonphysical states that are not meaningful for observable calculations.
Therefore, weclosely monitor how much the simulated structure deviates from the physical configurations, withthe RDF for condensed phase systems and bond lengths for flexible molecules.
We say that asimulation becomes “unstable” when the deviation exceeds a threshold, which implies sampling ofhighly nonphysical structures.
We then use the time duration for which a model remains stable insimulations to measure its stability.
The ensemble statistics are only computed over the stable part ofthe simulated trajectories.
Details on the stability criterion are included in Appendix C.
Experiments
Benchmarked models.
We adopt the Open Catalyst Project implementation of SchNet [46],
[-FPS 109 .
-]
[-2 Salicylic Acid Force 14 .26 .
-]
[-FPS 94 .111 .
 ,-]DimeNet [17], ForceNet [25], GemNet-T/dT [16], and the official implementation of DeepPotSE [72] and NequIP [4].
A summary of all benchmarked models is in Table 2.
These models havebeen popular in previous benchmark studies for force/energy prediction.
They use different representations for atomistic interactions and respect different levels of euclidean symmetry.
We follow alloriginal hyperparameters introduced in the respective papers and only make minimal adjustmentswhen the training is unstable.
More details on the hyperparameters can be found in Appendix E.
Key observations.
We make two key observations as evidenced in the experimental results:1.
Despite being widely used, force prediction is not sufficient for evaluating ML FFs.
Itgenerally does not align with simulation stability and performance in estimating ensembleproperties.2.
While often neglected, stability can be a major bottleneck for ML FFs.
Lower force errorand more training data does not necessarily give rise to more stable simulations, suggestingstability as a fundamental consideration for comparison and model design.
We next go through experimental results of all four datasets in detail to demonstrate the key observations while making other observations. [Table] Stability [Table]
[Table]
Salicylic Acid Force
[Table] Stability [Table] [Table]
[Table]
[Table] [Table] MD17. As shown in Table 3, more recent models that lie on the right side of the table generallyachieve a lower force error, but may lack stability.
Figure 3 selects results from Table 3 to demonstratethe non-aligned trends of force prediction performance vs. simulation performance, which supportskey observation 1 .
GemNet-T/dT can attain a very low force error for all four molecules, but oftencollapse before the simulation finishes.
This observation constitutes key observation 2 .
We note thatalthough the stable portion of simulated trajectories produced by GemNet-T/dT can recover the h ( r )relatively accurately, stability will become a bigger issue when the statistics of interest require longsimulations, as demonstrated in other experiments.
On the other hand, despite having a relativelyhigh force error, DeepPot-SE performs very well on simulation-based metrics on all molecules exceptfor Aspirin (Figure 3).
With the highest molecular weight, Aspirin is indeed the hardest task in
MD17 in the sense that all models attain high force prediction errors on it.
We further observe that good stability does not imply accurate recovery of trajectory statistics.
Although ForceNet remains stable for Ethanol and Naphthalene , the extracted h ( r ) deviates a lotfrom the reference (Table 3), indicating that ForceNet does not learn the underlying PES correctly,possibly due to its lack of energy conservation and rotational equivariance.
Overall, NequIP is thebest-performing model on MD17.
It achieves the best performance in both force prediction andsimulation-based metrics for all molecules while requiring the highest computational cost.
Moredetailed results on MD17 including individual h ( r ) are included in Appendix E.
Water.
Under different challenges posed by a condensed phase system, key observation 1 and 2 arestill evident according to Table 4: GemNet-T/dT and DimeNet are the top-3 models in terms of force [Table] [Figure] R D F ( O , O )
M
AE 0. 7.5 10.0 12.5 Force MAE [meV/ Å ] 0. D i ff u s i v i t y M AE [9 m 2 / s ]
Model
DeepPot-SE
SchNet
DimeNet
ForceNet
GemNet-T
GemNet-dT
NequIP prediction, but all lack stability.
With insufficient simulation time due to early collapse, the RDFscomputed from their simulated trajectories are of high variance.
The water diffusivity coefficientrequires long (100 ps in our experiments) trajectories to estimate and thus cannot be extracted forunstable models.
Like MD17, DeepPot-SE does not achieve the best force prediction performancebut demonstrates decent stability and highly accurate recovery of simulation statistics.
Figure 4 further compares model performance with different training dataset sizes.
Key observation1 and 2 are clearly shown: Models located on the left of each scatter plot have very low forceerror but may have poor stability or high error in simulation statistics.
More specifically, althoughmore training data almost always improve force prediction performance, its effect on simulationperformance is not entirely clear.
On the one hand, GemNet-T/dT, DimeNet, and ForceNet are notstable even when under the highest training data budget.
On the other hand, we observe a clearimprovement of DeepPot-SE when more training data is used.
NequIP is again the best performingmodel, achieving very low force error, excellent stability, and accurate recovery of ensemble statistics,even under the lowest data budget of 1,000 training+validation structures.
However, when the trainingdataset is sufficiently large (90k), DeepPot-SE has equally good results as NequIP while being morethan 20 times faster – dataset size also influences the model of choice for a certain dataset.
Moreresults, including tables for water-1k/90k and a study on model size, are included in Appendix E.
Alanine dipeptide poses unique challenges in sampling different metastable states separated byhigh free energy barriers.
Table 5 shows all models have high force errors due to the random forcesintroduced by the lack of explicit account of water molecules.
Although the force errors are in thesame order of magnitude, all models except NequIP fail to simulate stably.
The FES reconstructiontask requires stable simulation for the entire 5 ns.
NequIP is the only model that manages to finishfive simulations out of six but produces inaccurate statistics.
All other models are not stable enoughto produce meaningful results.
We further analyze the results on this task in Section 4. [Table] [Table] [Table]
[Table]
LiPS.
Compared to flexible molecules and liquid water, this solid material system features slowerkinetics.
From Table 6 we observe that most models are capable of finishing 50-ns simulationsstably.
In this dataset, the performance on diffusivity estimation and force prediction align well.
We observe that both GemNet-T and GemNet-dT show excellent force prediction, stability, andrecovery of observables, while GemNet-dT is 2.6 times faster.
The better efficiency comes from thedirect prediction of atomic forces F instead of taking the derivative F = ∂E/∂ x , which also makes
GemNet-dT not energy-conserving – a potential issue we further discuss in Section 4.
Implications on model architecture.
More recent models utilizing SE(3)/E(3)-equivariant representations and operations such as GemNet and NequIP are more expressive and can capture interatomicinteractions more accurately.
This is reflected by their very low force error and accurate recovery ofensemble statistics when not bottlenecked by stability.
Moreover, NequIP shows that excellent accuracy and stability can be simultaneously achieved.
The stability may come from parity-equivarianceand the explicit architecture in manipulating higher-order geometric tensors.
We believe further investigations into the extrapolation behavior induced by different equivariant geometric representationsand operations [3] is a fruitful direction in designing more powerful ML FFs.
Failure Modes: Causes and Future Directions
A case study on alanine dipeptide simulation.
NequIP achieves decent performance on all our tasksbut fails on alanine dipeptide.
It is also the only model that can simulate stably for 5 ns.
Figure 5 (a)demonstrates how NequIP fails to reconstruct the FES: it does not manage to sample much of thetransition regions and the configuration space with ϕ ∈ [0 , 180 ◦ ] .
Figure 5 (b) demonstrates thereconstructed FES, which significantly deviates from the reference.
This failure can be partiallyexplained by Figure 5 (c), the training data distribution produced by the reference potential.
Therelatively high-energy (low-density) regions are exactly those that are not reachable by NequIP.
Eventhough our MD trajectory is well-equilibrated, the relative difference in populations of differentmeta-stable states creates data imbalance, making it more challenging for the model to learn PES forhigher-energy configurations where density is relatively low.
In our experiments, we observe thatsimulations starting from the low-density meta-stable state (e.g., black star marked in Figure 5 (c))tend to fail.
This implies that generalization across different regions in the conformational space isan important challenge for ML FFs.
To prevent ML FFs from sampling nonphysical regions, whichis a common precursor to failed simulation (Figure 6), one can deliberately include distorted andoff-equilibrium geometries in the training data to improve model robustness[51].
Alternatively, onecan resort to active learning [66, 61, 48] to acquire new data points based on model uncertainty. [Figure] t = 3702 fs t = 3704 fs t = 3706 fs t = 3622 fs t = 3716 fs t = 28330 fs t = 28370 fs t = 28372 fs t = 28373 fs t = 28376 fs
[Figure]
Energy conservation.
Models that directly predict forces may not conserve energy.
Figure 5 (d)demonstrates the evolution of model-predicted total energy for selected models on LiPS, in a microcanonical (NVE) ensemble.
The energy of an isolated system in the NVE ensemble is in principleconserved.
We observe that GemNet-T conserves energy, whereas GemNet-dT fails to conserve thepredicted total energy.
The existence of non-conservative forces breaks the time reversal symmetryand, therefore, may not be able to reach equilibrium for observable calculation.
However, in ourexperiment, GemNet-dT performs well on the LiPS dataset when coupled with a thermostat.
Previousworks [30] also found that energy conservation is not required for SOTA performance on OC20.
Theusability of non-conservative FFs in simulations requires further careful investigations.
Simulation instability is a major bottleneck for highly accurate models such as GemNet-T to fail onseveral simulation tasks.
Moreover, in our water experiments, we find a larger amount of trainingdata does not resolve this issue for GemNet-T/dT and DimeNet (Figure 4).
We further experimentwith smaller simulation time steps for GemNet-T on water (Figure 5 (e)), but stability still doesnot improve.
On the other hand, [51] demonstrates that the stability of GemNet improves withlarger training sets on QM7-x, which includes high-energy off-equilibrium geometries obtained fromnormal mode sampling.
We hypothesize that including these distorted geometries may improvethe model’s robustness against going into nonphysical configurations.
We also observe that [-t = 3702 fs t = 3704 fs t = 3706 fs t = 3622 fs t = 3716 fs t = 28330 fs t = 28370 fs t = 28372 fs t = 28373 fs t = 
 fs [Figure]-]thesimulation can collapse within a short time window after a long period of stable simulation, asvisualized in Figure 6.
In both cases, we observe that the nonphysical configurations first emergeat local regions (circled), which cascade to the entire system very quickly as extremely large forcesare being produced and subsequently integrated into the dynamics.
At the end of the visualization,the bonds in the alanine dipeptide system are broken.
Therefore, the local-descriptor-based NequIPmodel predicts very small forces.
For the water system, the particles are packed in a finite periodicbox.
The nonphysical configurations exhibit incorrect coordination structures and extremely largeforces.
Regarding stability, past works found adding noise paired with a denoising objective duringtraining helpful in improving out-of-distribution test performance on OC20 [19], and in stabilizinglearned simulations [43].
Another relevant line of work in coarse-grained MD simulation has studiedregularization with an empirical “prior energy” [63] and post-prediction refinement [15] to battlesimulation instability.
Conclusion and Outlook
We have introduced a diverse suite of MD simulation tasks and conducted a thorough comparison of
SOTA ML FFs to reveal novel insights into ML for MD simulation.
As shown in our experiments,benchmarking only force error is not sufficient, and simulation-based metrics should be used toreflect the practical utility of a model.
We demonstrate case studies on the failure of existing trainingschemes/models to better understand their limitations and emphasize the importance of simulationstability.
Our experiments also show that the performance of a model can be highly case-dependent.
For more challenging MD systems, more expressive atomistic representations may be required.
For example, recent work has explored non-local descriptors [27] aiming at capturing long-rangeinteractions in large molecules.
Strictly local equivariant representations [37] are studied for verylarge systems where computational scalability is critical.
New datasets [11] and benchmarks havebeen playing an important role in inspiring future work in the thriving field of ML MD simulation.
The possibility of ML in advancing MD simulation is not limited to ML force fields.
Enhancedsampling methods enable fast sampling of rare events and have been augmented with ML techniques [44, 52, 24].
Differentiable simulations [45, 64, 10, 26, 20] offer a principled way of learningthe force field by directly training the simulation process to reproduce experimental observables[64, 65, 53].
We hope our datasets and benchmarks will encourage future developments in all relatedaspects to push the frontier in ML for MD simulations.
A Related Work
ML force fields learn the potential energy surface (PES) from the data by applying expressiveregressors such as kernel methods [8] and neural networks on symmetry-preserving representations of atomic environments [5, 29, 50, 2, 60, 72, 71, 31, 54].
Recently, graph neural networkarchitectures [18, 46, 17] have gained popularity as they provide a systematic strategy for buildingmany-body correlation functions to capture highly complex PES. In particular, equivariant representations have been shown powerful in representing atomic environments [55, 40, 47, 4, 16, 35], leadingto significant improvements in benchmarks such as MD17 and OC22/20. Some works presentedsimulation-based results [58, 38, 4, 37] but do not compare different models with simulation-basedmetrics. Existing benchmarks for ML force fields [41, 8] mostly focus on force/energy prediction, withsmall molecules being the most typical systems.
The catalyst-focused OC20 [6] and OC22 [56]benchmarks proposed the Initial Structure to Relaxed Structure/Energy tasks, which aim to predictthe relaxed structure/energy through structural optimizations.
These tasks do not characterize systemproperties under a structural ensemble which requires simulations.
Several recent works [42] havealso studied the utility of certain ML FFs in MD simulations.
In particular, [51] uses GemNet [16] tosimulate small molecules in the QM7-x [23] dataset, with a focus on simulation stability.
[70] appliesthe DeepMD [71] architecture to simulate water and demonstrates its shortcoming in generalizationacross different phases.
However, systematic benchmarks for simulation-based metrics are lacking inthe existing literature, which obscures the challenges in applying ML FF for MD applications.
B Preliminaries
Training.
An ML FF aims to learn the potential energy surface ˆ E ( x ) ∈ R as a function of atomiccoordinates x ∈ R N × 3 ( N is the number of atoms), by fitting atom-wise forces ˆ F ( x ) and energiesfrom a training dataset: { x i , F i , E i } N data i =1 , where x i ∈ R N × 3 , F ∈ R N × 3 , E ∈ R .
For evaluation,the test force prediction accuracy is used as a proxy to quantify the quality of the learned PES.
Theforce field learning protocol has been well established [59].
MD simulation.
Simulating molecular behaviors requires integrating a Newtonian equation ofmotion with forces obtained by differentiating ˆ E ( x ) :
F ( x ) = − ∂ ˆ E ( x ) /∂
x .
To mimic desiredthermodynamic conditions, an appropriate thermostat and barostat are chosen to augment the equationof motion with extended variables.
The simulation produces a time series of positions: { x t ∈ 473 R N ×} Tt =0 , where t is the temporal order index, and T is the total simulation steps.
Observables.
From the time series observations of positions and velocities, observables O ( x t ) canbe computed to characterize the state of the system at different granularities.
Under the ergodichypothesis, the time averages of the simulation observables converge to distributional averages underthe Gibbs measure: ⟨ O ⟩ = 1 T lim T →∞ (cid:80)
Tt O ( x t )
=
(cid:82) dx p ( x ) O ( x ) ,
where p ( x ) ∝ exp( − ˆ E ( x ) k B T )with T as the bath temperature and k B as the Boltzmann constant.
Calculations of such observablesrequire the system to reach equilibrium.
Simulation observables connect simulations to experimentalmeasurements and are predictive of macroscopic properties of matter.
Common observables includeradial distribution functions (RDFs), virial stress tensor, mean-squared displacement (MSD), etc.
The metrics used in this work are all well-established observables in the respective types of systems.
Definitions of benchmarked observables are in Appendix C.
C Evaluation Metrics
Distribution of interatomic distances is a low-dimensional description of the 3D structure andhas been studied in previous work [71].
For a given configuration x , the distribution of interatomicdistances h ( r ) is computed with:
[Equation]
where r is the distance from a reference particle; N is the total number of particles; i, j indicates thepairs of atoms that contribute to the distance statistics; δ is the Dirac Delta function to extract valuedistributions.
To calculate the ensemble average, h ( r ) is calculated and averaged over frames fromequilibrated trajectories.
For learned simulations, we compute h ( r ) using only the stable part of thesimulation.
RDF.
As one of the most informative simulation observables, the radial distribution function (RDF)describes the structural/thermodynamic properties of the system and is also experimentally measurable.
By definition, the RDF describes how density varies as a function of distance from a particle(illustrated in Figure 7 (a)).
For a given configuration x , the RDF can be computed with the followingformula:
[Equation]
where r is the distance from a reference particle; N is the total number of particles; i, j indicates thepairs of atoms that contribute to the distance statistics; ρ is the density of the system; δ is the Dirac
Delta function to extract value distributions.
To calculate the ensemble average, RDF( r ) is calculatedand averaged over frames from equilibrated trajectories.
The final RDF MAE is then calculated byintegrating r:
[Equation]
(3) where ⟨·⟩ is the averaging operator, ⟨ RDF( r ) ⟩ is the reference equilibrium RDF, and ⟨ ˆRDF( r ) ⟩ isthe model-predicted RDF.
Diffusivity coefficient.
The diffusivity coeffcient D quantifies the time-correlation of the translationaldisplacement (illustrated in Figure 7 (b)), and can be computed from the mean square displacement:
[Equation]
where x i ( t ) is the coordinate of particle i at time t , and N ′ is the number of particles being trackedin the system.
For the water system we monitor the liquid diffusivity coefficient and track all 64oxygen atoms.
For the LiPS system, we monitor Li-ion Diffusivity and track all 27 Li-ions.
As thedefinition implies, D is a quantity that converges with longer simulation time.
Accurate recovery of
D requires sufficient long trajectories sampled from the Hamiltonian with ML FFs.
In this paper,we only compute diffusivity for stable trajectories of at least 100 ps for water and 40 ps for LiPS.
As we simulate multiple random runs for each system/model, we average the diffusivity coefficientextracted from each valid trajectory to obtain the final prediction.
Free energy surface.
Given the probability distributions over configurations p ( x ) and a chosengeometric coordinate ξ transformed from x .
Based on the marginalized density p ( ξ ) , the free energycan be calculated from the following:
[Equation]
In the specific case of alanine dipeptide, there are two main conformational DOF: dihedral angle ϕ of
C − N − C α − C and dihedral angle ψ of N − C α − C − N (illustrated in Figure 7 (c)).
Therefore,the FES w.r.t ϕ and ψ is the most physically informative.
We propose our quantitative metric
Diffusion t t+ Radial Distribution r ϕ ψ Dihedral Angles (a) (b) (c) MAE F ( ϕ ) ,F ( ψ ) based on the absolute error in reconstructing the FES along the ϕ and ψ coordinates.
We integrate the absolute difference between the reference free energy F and the model predicted ˆ Ffrom [ − π, π ) :
[Equation]
Stability criterion.
Abstractly speaking, stability is a notion of staying within physical (low-energy)configuration spaces.
Since all MD systems studied in this paper are in equilibrium, practicallywe keep track of stability by closely monitoring equilibrium statistics.
For systems with periodicboundary conditions, we monitor the RDF and say a simulation becomes “unstable” at time T when
[Equation]
(8) where ⟨·⟩ is the averaging operator, τ is a short time window, and ∆ is the stability threshold.
In thispaper we use τ = 1 ps , ∆ = 3 . 0 for water, and τ = 1 ps , ∆ = 1 . 0 for LiPS.
For water, we assertunstable if any of the three element-conditioned RDFs: RDF ( O,O ) , RDF ( H,H ) , RDF ( H,O ) exceedsthe threshold.
For flexible molecules, we keep track of stability through the bond lengths and say asimulation becomes “unstable” at time T when:
[Equation]
(9) where B is the set of all bonds, i, j are the two endpoint atoms of the bond, and b i,j is the equilibriumbond length.
We use ∆ = 0 .
5 Å for both MD17 molecules and alanine dipeptide.
We measure the stability of a learned model through the simulation time it remains stable.
For eachdataset, the threshold ∆ we adopt is rather relaxed that an “unstable” equilibrium statistic usuallyindicates the system is already in a highly nonphysical configuration.
FPS.
All frames per second (FPS) metrics are measured with an NVIDIA Tesla V100-PCIe GPU.
We present FPS as a reference for models’ computational efficiency but also note that code speed canbe affected by many factors, and likely has room for improvement.
D Dataset Details
The MD17 dataset 1 [8] and the LiPS dataset 2 [4] are adapted from previous works and are publiclyavailable.
We refer interested readers to the respective papers for more details on the data generation process.
The water dataset and alanine dipeptide dataset are generated by ourselves, and will be madepublicly available.
MD17 [8] contains AIMD calculations for eight small organic molecules and is widely used as aforce prediction benchmark for ML FFs.
We adopt four molecules from MD17 and benchmark thesimulation performance.
In addition to force error, we evaluate the stability and the distributionof interatomic distances h ( r ) .
For each molecule, we randomly sample 9,500 configurations fortraining and 500 for validation from the MD17 database.
We randomly sample 10,000 configurationsfrom the rest of the data for force error evaluation.
We perform five simulations of 300 ps for eachmodel/molecule by initializing from 5 randomly sampled testing configurations, with a time step of0.5 fs, at 500 K temperature, under a Nosé–Hoover thermostat. Water is arguably the most important molecular fluid in biological and chemical processes. Due toits complex thermodynamic and phase behavior, it poses great challenges for molecular simulations. In addition to force error, we evaluate simulation stability and recovery of both equilibrium statisticsand dynamical statistics, namely the element-conditioned RDFs and liquid diffusion coefficient.
Our dataset consists of 100,000 structures collected every 10 fs from a 1 ns trajectory sampled atequilibrium and a temperature of 300 K. We benchmarked all models with various training+validationdataset sizes (1k/10k/90k randomly sampled structures) and used the remaining 10,000 structuresfor testing.
We performed 5 simulations of 500 ps by initializing from 5 randomly sampled testingconfigurations, with a time step of 1 fs, at 300 K temperature, under a Nosé–Hoover thermostat.
Alanine dipeptide features multiple metastable states, making it a classic benchmark for the development of MD sampling methods and force field development [22, 28].
Its geometric flexibility iswell represented by the central dihedral (torsional) angles ϕ and ψ .
Our reference data are obtainedfrom simulations with explicit water molecules, with detailed protocols described in the next section.
For faster simulation, we learn an implicitly solvated FF following a protocol similar to [7].
Ourtask is more challenging in that it aims to learn the implicitly solvated atomistic FF rather than theimplicit solvation correction in [7].
To facilitate accelerated sampling, we apply metadynamics withϕ and ψ as the collective variables.
We evaluate force prediction, simulation stability, and free energysurface (FES) reconstruction F ( ϕ, ψ ) .
Our dataset consists of 50,000 structures dumped every 2 psfrom a 100 ns trajectory at a temperature of 300 K. We used 38,000 randomly sampled structuresfor training, 2,000 for validation, and the rest as a test set.
We performed 6 simulations of 5 ns byinitializing from six local minima on the FES (Figure 5) with a time step of 2 fs at 300 K, and undera Langevin thermostat to mimic random noise from solvation effects.
LiPS is a crystalline superionic lithium conductor relevant to battery development and a representativesystem for MD simulation usage in studying kinetic properties in materials.
We adopt this datasetfrom [4], and benchmark all models on their force error, stability, RDF recovery, and Li-ion diffusivitycoefficient.
The dataset has 25,000 structures in total, from which we use 19,000 randomly sampledstructures for training, 1,000 structures for validation, and the rest for computing force error.
Weconduct 5 simulations of 50 ps by initializing from 5 randomly sampled testing configurations, with atime step of 0.25 fs, at 520 K temperature, under a Nosé–Hoover thermostat.
D.1 Data Generation and Simulation Protocols
Water. Our water dataset is generated from molecular dynamics simulations of a simple classicalwater model, namely, the flexible version of the Extended Simple Point Charge water model (SPC/Efw) [68] at temperature T = 300 K and pressure P = 1 atm.
This classical model has been wellstudied in previous work [68, 69] and has shown reasonable predictions of the physical properties ofliquid water.
It provides a computationally inexpensive way to generate a large amount of trainingdata.
The experience and knowledge gained from the benchmark based on the simple model can bereadily extended to systems with higher accuracy, such as the ab-initio models.
Alanine dipeptide.
Our dataset is generated from the MD simulation of an alanine dipeptidemolecule solvated in explicit water (1164 water molecules) performed in GROMACS [1] using the AMBER-03 [39] force-field.
The NPT ensemble is applied in simulations, with hydrogen bond [Figure] length constraints using LINear Constraint Solver (LINCS) and a time step of 2 fs.
The temperatureand pressure of the system are controlled at T = 300 K and P = 1 bar using a stochastic velocityrescaling thermostat with damping frequency t v = 0 .
1 ps and Parrinello-Rahman barostat withcoupling frequency t p = 2 .
ps, respectively.
The Particle Mesh Ewald approach is used to computelong-range electrostatics with periodic boundary conditions applied to the x, y, and z directions.
Implicit solvation.
The explicit solvent of 1164 water molecules is not the subject of study but adds asignificant computational burden.
In this task, we attempt to learn an implicit solvent model (ISM) ofthe alanine dipeptide, in which the explicit solvent environment is incorporated in the learned FF.
The
ISM is commonly used in drug design [62] because it can speed up the computation by dramaticallydecreasing the number of particles required for simulation.
In general, the mean-field estimation in ISM ignores the effect of solvent, thermal fluctuations, and solvent friction [13].
Thus, molecularkinetics is not directly comparable to the explicit solvation simulation.
However, the equilibriumconfigurations can be explicitly compared, as conducted in [7].
Metadynamics simulation.
Simulating energy barrier jump usually requires a long samplingof the trajectory in MD simulations.
The conformational change of alanine dipeptide in waterinvolves such a process, making it difficult to extract the complete free energy surface, i.e., the
Ramachandran plot, in normal MD.
In order to examine the learned ML FFs within a reasonable timelimit, metadynamics [32] is employed to explore the learned FES of the solvated alanine dipeptide.
Metadynamics is a widely used technique in atomistic simulations to accelerate the sampling of rareevents and estimate the FES of a certain set of degrees of freedom.
It is based on iteratively “filling”the potential energy using a sum of Gaussians deposited on a set of suitable collective variables(CVs) along the trajectory.
At evaluation time, we perform metadynamics with dihedral angles ϕ andψ as CVs 3 , starting from the configurations located at one of the six energy minimums in the freeenergy surface indicated in Figure 5 (c).
The Gaussians with height h = 1 . 2 and sigma σ = 0 .
35are deposited every 1 ps centered on ψ and ϕ .
As shown in Figure 8, the estimated FES of both ϕand ψ do not significantly change after 5 ns.
In addition, the height of the bias gaussian potentialsmoothly converges to ∼ 0 in the time limit of 5 ns.
Therefore, a simulation time of 5 ns is sufficientfor the convergence of the metadynamics.
This metadynamics simulation of alanine dipeptide with AMBER force-fields is carried out using GROMACS [1] integrated with the PLUMED library [57, 9]of version 2.8.
E Experimental details
The Open Catalyst Project 4 codebase and the official codebases of DeepPot-SE 5 and NequIP 6 areall publicly available.
We build our MD simulation framework based on the Atomic {+Simulation
+}
In practice, the selection of suitable collective variables can be a case-by-case challenge [49].https://github.com/Open-Catalyst-Project/
https://github.com/deepmodeling/deepmd-kithttps://github.com/mir-
/nequip [Table] [Table] [Table]
Diffusivity
[Table]
[Table]
Environment (ASE) library 7 [34].
Our code will be made publicly available after the reviewingprocess.
Hyperparameters.
We adopt the original model hyperparameters in the respective papers and findthey can produce good force prediction results that match the trend and numbers for MD17 reported inprevious work.
As we introduce new datasets, we set training hyperparameters such as the batch sizeand summarize them in Table 7.
For water and LiPS, we use a batch size of 1 like in previous work [4]as each structure already contains a reasonable number of atoms and interactions.
Following previouswork, we use an initial learning rate of 0.001 for all experiments except for NequIP, which uses 0.005as the initial learning rate in the original paper.
For models that minimize a mixture of force loss andenergy loss, we set the force loss coefficient λ F to be 1000 and the energy loss coefficient λ E to be1 , if not specified in the original paper for a dataset.
A higher force loss coefficient is common inprevious work [71, 4] as simulations do not directly rely on the energy.
Notably, NequIP proposed several sets of hyperparameters for different datasets, including MD17,a water+ice dataset from [71], LiPS, etc.
We follow the MD17 hyperparameters of NequIP forour MD17 and alanine dipeptide datasets; the water+ice hyperparameters of NequIP for our waterdataset; and the LiPS hyperparameters of NequIP for the same LiPS dataset.
The only architecturaladjustment we made is because we observed training instability for ForceNet on water using theoriginal hyperparameters.
We resolve this issue by reducing the network width from 512 to 128 for ForceNet in our water experiments.
To facilitate fair benchmarking, we stop the training of a model if either of the following conditions ismet: (1) a maximum training time of 7 days is reached on an NVIDIA Tesla V100-PCIe GPU; (2) amaximum number of epochs specified in Table 7 is reached; (3) The learning rate drops below 10 − 6with a ReduceLROnPlateau scheduler with factor 0.8 and learning rate (LR) patience specifiedin Table 7.
We also report the longest time for an ML model to finish our benchmark simulationin Table 7.
All numbers are results of NequIP.
The high computational cost for evaluating MDsimulations has been a major consideration in designing our benchmark datasets and metrics.
Complete water results.
We present results on water-1k in Table 8 and results on water-90k in
Table 9.
Results on water-10k is presented in Table 4 in the main text.
All models generally achieve [Table] [Table] =32, r=
[Table]
[Table]
[Table]
Width=64, r=
[Table]
=64, r=
[Table]
[Table]
=128, r=
[Table]
lower force error when trained with more data, but stability and estimation of ensemble statisticsdon’t necessarily improve.
Notably, DeepPot-SE shows clear improvement with more training dataand becomes as good as NequIP on water-90k.
SchNet demonstrates significant improvement instability, but the estimation of ensemble statistics does not improve.
This may be due to the limitedaccuracy of SchNet coming from the limited expressiveness of the invariant atomic representation.
Influence of model size.
Table 10 shows an ablation study over the model size and radius cutoffof NequIP over water-1k.
We observe that all models are highly stable and attain equally goodperformance in simulation-based metrics.
Although a small radius cutoff of 4 leads to worseperformance in force prediction, it is more computationally efficient and preserves the trajectorystatistics.
These results show that there exists a trade-off between accuracy and efficiency whenchoosing the hyperparameters of an ML force field, and force error may not be the preferred criterionfor model selection.
Distribution of interatomic distances for MD17.
Figure 9 shows the h ( r ) curves for all modelsand molecules benchmarked.
We randomly selected one simulation out of the five simulationswe conducted for each model and molecule.
We observe that due to lack of stability, DeepPotSE produces noisy h ( r ) on Aspirin.
ForceNet does not manage to learn the correct interatomicinteractions and produces incorrect h ( r ) curves.
Most models are able to produce h ( r ) that matchwell with the reference, with SchNet being less accurate on Aspirin and Ethanol .
RDFs for water.
Selected RDF curves for water-1k/10k/90k are in Figure 10, Figure 11, and Figure 12.
Most noisy curves are due to insufficient sampling time, which results in a small numberof frames to be averaged in obtaining the RDF curves.
We observe that SchNet and ForceNet produceinaccurate curves that are not very noisy, showing that their failure is not entirely due to lack ofstability but because of inaccurate modeling of interactions caused by limited expressiveness andlower sample efficiency.
Further, we note that the reference curves have zero values below a certainthreshold, as any pair of atoms cannot get too close to each other.
However, DimeNet and GemNet-Texhibit abnormal high values for very small distances, indicating the simulations have gone intononphysical configurations and collapsed.
RDFs for LiPS.
As shown in Figure 13, DeepPot-SE does not manage to stay stable on LiPS.
ForceNet learns inaccurate interactions and produces inaccurate RDFs.
All other models can producehighly accurate RDF and can reproduce Li-ion diffusivity relatively accurately, as demonstrated in
Table 6.
[Figure]
RD F (r)
[ Å 1 ]
DeepPot-SE, O-O
Reference Prediction
DeepPot-SE, H-H
Reference Prediction
DeepPot-SE, H-O
Reference Prediction
RD F (r)
[ Å 1 ]
SchNet, O-O
Reference Prediction
SchNet, H-H
Reference Prediction      {+
+}
SchNet, H-O
Reference Prediction
RD F (r)
[ Å 1 ]
DimeNet, O-O
Reference Prediction
DimeNet, H-H
Reference Prediction
DimeNet, H-O
Reference Prediction
RD F (r)
[ Å 1 ]
ForceNet, O-O
Reference Prediction
ForceNet, H-H
Reference Prediction
ForceNet, H-O
Reference Prediction
RD F (r)
[ Å 1 ]
GemNet-T, O-O
Reference Prediction
[Figure] [Figure] RD F (r)
[ Å 1 ]
DeepPot-SE, O-O
{+Reference Prediction      
+}
DeepPot-SE, H-H
{+Reference Prediction      
+}
DeepPot-SE, H-O
{+Reference Prediction      
+}
RD F (r)
[ Å 1 ]
SchNet, O-O
Reference Prediction[-
-]
[Figure]
[Figure] [Figure]
Reference
[Figure] [Figure]
{+Reference Prediction      
+}
RD F (r)
[ Å 1 ]
DimeNet, O-O
{+Reference Prediction      
+}
[Figure]
[Figure]
[Table]
Prediction
[Figure]
DimeNet,
[Figure]
[Figure]
Reference Prediction      {+
+}
RD F (r)
[ Å 1 ]
ForceNet, O-O
{+Reference Prediction      
+}
ForceNet, H-H
Reference Prediction
[Figure]
ForceNet, H-
[Figure]
{+Reference Prediction      
+}
RD F (r)
[ Å 1 ]
GemNet-T, O-O
Reference
[Figure]
RD F (r)
[ Å 1 ]
DeepPot-SE, O-O
{+Reference Prediction      
+}
DeepPot-SE, H-H
{+Reference Prediction      
+}
DeepPot-SE, H-O
{+Reference Prediction      
+}
RD F (r)
[ Å 1 ]
{+SchNet, O-O Reference Prediction      
 SchNet, H-H Reference+} Prediction
SchNet, H-O
{+Reference Prediction      
+}
RD F (r)
[ Å 1 ]
DimeNet, O-O
{+Reference Prediction      
+}
{+DimeNet, H-H Reference Prediction      
 DimeNet, H-O Reference Prediction      
+}
RD F (r)
[ Å 1 ]
ForceNet, O-O
{+Reference Prediction      
+}
ForceNet, H-H
{+Reference Prediction      
+}
ForceNet, H-O
{+Reference Prediction      
+}
RD F (r)
[ Å 1 ]
GemNet-T, O-O
Reference Prediction
[Figure]
[Figure]
r [
[Figure]
[Figure]
Å
[Figure]
RD
[Figure]
[Figure]
Å
[Figure]
GemNet
[Figure]