Original: VIMPNN: P HYSICS INFORMED DNN TO ESTIMATE POTENTIAL ENERGIES OF UNSTABLE SYSTEMS
Revised: VIMPNN: P HYSICS INFORMED DNN TO ESTIMATE POTENTIAL ENERGIES OF UNSTABLE SYSTEMS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Simulation of molecular and crystalline systems enables insight into interesting chemical properties that beneﬁt processes ranging from drug discovery to material synthesis.
Revised: Simulation of molecular and crystalline systems enables insight into interesting chemical properties that beneﬁt processes ranging from drug discovery to material synthesis.

Original: However, these simulations can be computationally expensive and time consuming despite the approximations through Density Functional Theory (DFT).
Revised: However, these simulations can be computationally expensive and time consuming despite the approximations through Density Functional Theory (DFT).

Original: We propose the Valence Interaction Message Passing Neural Network (VIMPNN) to approximate DFT’s potential energy calculations.
Revised: We propose the Valence Interaction Message Passing Neural Network (VIMPNN) to approximate DFT’s potential energy calculations.

Original: VIMPNN integrates prior physics knowledge, namely the information exchange between atoms being driven and modulated by different interatomic bonds, and the relevance of speciﬁc physical properties to the problem of estimating more accurate energies.
Revised: VIMPNN integrates prior physics knowledge, namely the information exchange between atoms being driven and modulated by different interatomic bonds, and the relevance of speciﬁc physical properties to the problem of estimating more accurate energies.

Original: Furthermore, while many previous machine learning methods consider only stable systems, our proposed method is demonstrated on unstable systems at different interatomic distances.
Revised: Furthermore, while many previous machine learning methods consider only stable systems, our proposed method is demonstrated on unstable systems at different interatomic distances.

Original: VIMPNN predictions can be used to determine the stable locations for atoms, a necessary step for the future simulation of crystal growth.
Revised: VIMPNN predictions can be used to determine the stable locations for atoms, a necessary step for the future simulation of crystal growth.

Original: Our method is extensively evaluated on an augmented version of the QM9 dataset that includes unstable molecules, as well as 2 new datasets of periodic and growing crystals.
Revised: Our method is extensively evaluated on an augmented version of the QM9 dataset that includes unstable molecules, as well as 2 new datasets of periodic and growing crystals.

Original: VIMPNN has comparable accuracy with DFT, while allowing for 5 orders of magnitude in computational speed up.
Revised: VIMPNN has comparable accuracy with DFT, while allowing for 5 orders of magnitude in computational speed up.

Original: It also produces more accurate and informative potential energy curves than Message Passing Neural Network.
Revised: It also produces more accurate and informative potential energy curves than Message Passing Neural Network.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Chemical simulations have many useful industrial applications ranging from drug discovery to the production of materials for daily use (Sch¨utt et al., 2017).
Revised: Chemical simulations have many useful industrial applications ranging from drug discovery to the production of materials for daily use (Sch¨utt et al., 2017).

Original: Simulating crystal systems in particular provides useful properties such as surface absorption, chemical reactions, and surface magnetism (Bilek & Sk ´ ala, 1978).
Revised: Simulating crystal systems in particular provides useful properties such as surface absorption, chemical reactions, and surface magnetism (Bilek & Sk ´ ala, 1978).

Original: Quantum Mechanical (QM) simulations can be used for the calculation of potential energies based upon the interaction of atoms.
Revised: Quantum Mechanical (QM) simulations can be used for the calculation of potential energies based upon the interaction of atoms.

Original: By simulating a chemical system at different interatomic distances, it is possible to determine a stable conﬁguration where the atomic interaction is at an equilibrium (minimum potential energy).
Revised: By simulating a chemical system at different interatomic distances, it is possible to determine a stable conﬁguration where the atomic interaction is at an equilibrium (minimum potential energy).

Original: While QM simulations can be less time consuming than physical experimentation, they typically require large amounts of computing resources and do not scale well to larger system sizes (Jiang et al., 2003; Erba et al., 2017).
Revised: While QM simulations can be less time consuming than physical experimentation, they typically require large amounts of computing resources and do not scale well to larger system sizes (Jiang et al., 2003; Erba et al., 2017).

Original: To address this difﬁcultly in determining the potential energy, the Kohn-Sham Density Functional
Revised: To address this difﬁcultly in determining the potential energy, the Kohn-Sham Density Functional

Original: Theory (DFT) may be used to simplify the calculations by considering the electronic density in place of individual electrons.
Revised: Theory (DFT) may be used to simplify the calculations by considering the electronic density in place of individual electrons.

Original: DFT has proved useful in QM due to its good trade-off between speed of computation and chemical accuracy (Cohen et al., 2012).
Revised: DFT has proved useful in QM due to its good trade-off between speed of computation and chemical accuracy (Cohen et al., 2012).

Original: However, as DFT calculations are proportional to the number of interacting electrons (Lanyon et al., 2010), computation for large systems remains intractable.
Revised: However, as DFT calculations are proportional to the number of interacting electrons (Lanyon et al., 2010), computation for large systems remains intractable.

Original: As many interesting and realistic systems are formed from a large number of atoms, a computationally efﬁcient and accurate method for chemical property estimation that scales well to these large systems would prove signiﬁcantly useful in practical applications (Gomes et al., 2008).
Revised: As many interesting and realistic systems are formed from a large number of atoms, a computationally efﬁcient and accurate method for chemical property estimation that scales well to these large systems would prove signiﬁcantly useful in practical applications (Gomes et al., 2008).

Original: Moreover, several simulations are often necessary to identify stable conﬁgurations away from local minima.
Revised: Moreover, several simulations are often necessary to identify stable conﬁgurations away from local minima.

Original: Thus, fast approximation methods are desirable in such scenarios.
Revised: Thus, fast approximation methods are desirable in such scenarios.

Original: Machine learning (ML) was proposed to efﬁciently classify (stable) molecules and estimate chemical properties by reproducing the results of DFT, with recent advancements through a Message Passing
Revised: Machine learning (ML) was proposed to efﬁciently classify (stable) molecules and estimate chemical properties by reproducing the results of DFT, with recent advancements through a Message Passing

Original: Neural Network (MPNN) (Gilmer et al., 2017).
Revised: Neural Network (MPNN) (Gilmer et al., 2017).

Original: However, unstable molecules and crystals are often not considered, although this would allow discovering new stable conﬁgurations.
Revised: However, unstable molecules and crystals are often not considered, although this would allow discovering new stable conﬁgurations.

Original: In this article we demonstrate the applicability of ML for the energy prediction of unstable molecular and crystalline systems, where the atoms are at positions which are not at the DFT calculated minimum of potential energy, in the following referred to as either “unstable” or “out-of-equilibrium”.
Revised: In this article we demonstrate the applicability of ML for the energy prediction of unstable molecular and crystalline systems, where the atoms are at positions which are not at the DFT calculated minimum of potential energy, in the following referred to as either “unstable” or “out-of-equilibrium”.

Original: These energies are determined at the electronic ground-state at given positions of atoms for static systems.
Revised: These energies are determined at the electronic ground-state at given positions of atoms for static systems.

Original: We propose a new DNN framework, the VIMPNN (Valence Interaction Message Passing
Revised: We propose a new DNN framework, the VIMPNN (Valence Interaction Message Passing

Original: Neural Network), that considers the different interatomic interactions driven by different valences.
Revised: Neural Network), that considers the different interatomic interactions driven by different valences.

Original: It produces comparable accuracy to that of DFT while also improving the computation time by 5 orders of magnitude.
Revised: It produces comparable accuracy to that of DFT while also improving the computation time by 5 orders of magnitude.

Original: We demonstrate that our method also produces more accurate energy estimations than that of MPNN, while being 30% slower at 0.007 seconds per energy estimate, which is marginal compared to the gain from DFT and may be improved with some obvious code optimisation.
Revised: We demonstrate that our method also produces more accurate energy estimations than that of MPNN, while being 30% slower at 0.007 seconds per energy estimate, which is marginal compared to the gain from DFT and may be improved with some obvious code optimisation.

Original: ML in quantum chemistry requires the availability of large datasets of chemical structures and their (measured or computed) chemical properties.
Revised: ML in quantum chemistry requires the availability of large datasets of chemical structures and their (measured or computed) chemical properties.

Original: Online repositories such as Quantum Machine 1 store a collection of datasets resulting from various QM simulations.
Revised: Online repositories such as Quantum Machine 1 store a collection of datasets resulting from various QM simulations.

Original: QM9 (Blum & Reymond, 2009;
Revised: QM9 (Blum & Reymond, 2009;

Original: Montavon et al., 2013) is one such dataset containing 134K molecules using carbon, hydrogen, oxygen, nitrogen, and ﬂuorine atoms.
Revised: Montavon et al., 2013) is one such dataset containing 134K molecules using carbon, hydrogen, oxygen, nitrogen, and ﬂuorine atoms.

Original: However, as all molecules in this dataset are at their stable conﬁguration, it is not suitable for designing and testing accurate energy prediction methods for unstable systems.
Revised: However, as all molecules in this dataset are at their stable conﬁguration, it is not suitable for designing and testing accurate energy prediction methods for unstable systems.

Original: We address this shortcoming by augmenting the QM9 dataset with out-of-equilibrium conﬁgurations for 10K of its molecules at a [90%, 150%] range of interatomic distances.
Revised: We address this shortcoming by augmenting the QM9 dataset with out-of-equilibrium conﬁgurations for 10K of its molecules at a [90%, 150%] range of interatomic distances.

Original: In addition, we create two datasets composed of inﬁnite-size periodic crystals and ﬁnite-size growing crystals of aluminium (Al) and copper (Cu) atoms.
Revised: In addition, we create two datasets composed of inﬁnite-size periodic crystals and ﬁnite-size growing crystals of aluminium (Al) and copper (Cu) atoms.

Original: In summary, the contributions of this work are:
Revised: In summary, the contributions of this work are:

Original: • A new VIMPNN model that accounts for the physics of atomic bonds within a molecule or crystal to improve the estimation of chemical properties.
Revised: • A new VIMPNN model that accounts for the physics of atomic bonds within a molecule or crystal to improve the estimation of chemical properties.

Original: This model also introduces auxiliary chemical property estimations to help learn descriptors that are closer to the physics of the problem.
Revised: This model also introduces auxiliary chemical property estimations to help learn descriptors that are closer to the physics of the problem.

Original: • A new use of ML for potential energy estimation of unstable molecules and crystals.
Revised: • A new use of ML for potential energy estimation of unstable molecules and crystals.

Original: • The public release of new periodic crystal and growing crystal datasets and an augmented QM9 dataset.
Revised: • The public release of new periodic crystal and growing crystal datasets and an augmented QM9 dataset.

Original: The remainder of this article is organised as follows: Section 2 discusses previous works on molecular properties estimation by ML.
Revised: The remainder of this article is organised as follows: Section 2 discusses previous works on molecular properties estimation by ML.

Original: We describe the creation of 3 new datasets in Section 3.
Revised: We describe the creation of 3 new datasets in Section 3.

Original: In Sectionwe introduce our new physics informed DNN for accurate energy estimation, which we evaluate in
Revised: In Sectionwe introduce our new physics informed DNN for accurate energy estimation, which we evaluate in

Original: Section 5.
Revised: Section 5.

Original: We conclude in Section 6.
Revised: We conclude in Section 6.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Several ML methods have been used for estimating a variety of chemical properties from different representations of chemical systems.
Revised: Several ML methods have been used for estimating a variety of chemical properties from different representations of chemical systems.

Original: Li et al.
Revised: Li et al.

Original: (2015) constructed a covariance matrix from Euclidean distances of atoms within molecules, then use Bayesian regression to estimate interatomic forces.
Revised: (2015) constructed a covariance matrix from Euclidean distances of atoms within molecules, then use Bayesian regression to estimate interatomic forces.

Original: Wang et al.
Revised: Wang et al.

Original: (2013) used kernel ridge regression on the numbers of coarse ‘building blocks’ (such as
Revised: (2013) used kernel ridge regression on the numbers of coarse ‘building blocks’ (such as

Original: CH 2 ) in molecules to estimate many chemical properties such as electron afﬁnity and atomization energy.
Revised: CH 2 ) in molecules to estimate many chemical properties such as electron afﬁnity and atomization energy.

Original: Recent research by Shi et al.
Revised: Recent research by Shi et al.

Original: (2019) used strain information and a Fourier transform-based representation of crystal lattice for a feed-forward neural network (NN) to estimate an electronic bandgap structure in silicon.
Revised: (2019) used strain information and a Fourier transform-based representation of crystal lattice for a feed-forward neural network (NN) to estimate an electronic bandgap structure in silicon.

Original: The bandgap refers to the energy difference between the valence and conduction bands in insulators and semiconductors, and therefore the energy required to transition between stable excitation states.
Revised: The bandgap refers to the energy difference between the valence and conduction bands in insulators and semiconductors, and therefore the energy required to transition between stable excitation states.

Original: Silicon crystal in a known equilibrium state is strained in the range of -10% to 10% in each strain component (such as thermal properties) to search the bandgap space for another stable state, for example transforming silicon from a semiconductor to a metal.
Revised: Silicon crystal in a known equilibrium state is strained in the range of -10% to 10% in each strain component (such as thermal properties) to search the bandgap space for another stable state, for example transforming silicon from a semiconductor to a metal.

Original: Therefore, this work uses the NN to describe the bandgap as a function of the strain tensors.
Revised: Therefore, this work uses the NN to describe the bandgap as a function of the strain tensors.

Original: In contrast, our method infers the effect of spatial deformations on a system’s potential energy.
Revised: In contrast, our method infers the effect of spatial deformations on a system’s potential energy.

Original: Some approaches focused on the prediction of stable geometries for chemical systems.
Revised: Some approaches focused on the prediction of stable geometries for chemical systems.

Original: Timoshenko et al.
Revised: Timoshenko et al.

Original: (2018) used experimental spectra data as input for an NN to estimate the probability distribution of the stable bonding distance between pairs of atoms.
Revised: (2018) used experimental spectra data as input for an NN to estimate the probability distribution of the stable bonding distance between pairs of atoms.

Original: They considered different combinations of
Revised: They considered different combinations of

Original: Pd and Au atoms (i.e. an NN for Au-Au bonds, another for Pd-Au).
Revised: Pd and Au atoms (i.e. an NN for Au-Au bonds, another for Pd-Au).

Original: When considering crystals made of two different atom types, Takahashi & Takahashi (2019) used 8 quantities associated with each atom type (such as atomic radii, electronegativity, and number of atoms) to perform a random forest classiﬁcation on how the resulting stable crystal structure would form among a set ofdifferent possible structures.
Revised: When considering crystals made of two different atom types, Takahashi & Takahashi (2019) used 8 quantities associated with each atom type (such as atomic radii, electronegativity, and number of atoms) to perform a random forest classiﬁcation on how the resulting stable crystal structure would form among a set ofdifferent possible structures.

Original: The predicted lattice type requires further optimisation using DFT to ﬁnd its exact size/scaling.
Revised: The predicted lattice type requires further optimisation using DFT to ﬁnd its exact size/scaling.

Original: However, this method may also predict metastable conﬁgurations (i.e. local minima).
Revised: However, this method may also predict metastable conﬁgurations (i.e. local minima).

Original: Whereas the aforementioned methods attempted to directly obtain the stable distance between atoms, we estimate potential energies for various geometries with the aim to minimise them into stable conﬁgurations.
Revised: Whereas the aforementioned methods attempted to directly obtain the stable distance between atoms, we estimate potential energies for various geometries with the aim to minimise them into stable conﬁgurations.

Original: While this approach may require more training data and more complex modelling to handle arbitrary conﬁgurations, we argue that it has a stronger generalisation potential.
Revised: While this approach may require more training data and more complex modelling to handle arbitrary conﬁgurations, we argue that it has a stronger generalisation potential.

Original: It also has the added beneﬁt of allowing the (future) observation of progressive crystal growth.
Revised: It also has the added beneﬁt of allowing the (future) observation of progressive crystal growth.

Original: Previous works on estimating the potential energy of chemical systems include that of Rupp et al.
Revised: Previous works on estimating the potential energy of chemical systems include that of Rupp et al.

Original: (2012) who used a feed-forward NN on the Coulomb matrix (a pairwise matrix that describes the low-level electrostatic interaction between atoms) of molecules.
Revised: (2012) who used a feed-forward NN on the Coulomb matrix (a pairwise matrix that describes the low-level electrostatic interaction between atoms) of molecules.

Original: However, predictions on this matrix are sensitive to permutations of atoms which can result in different property values being estimated for a same crystal.
Revised: However, predictions on this matrix are sensitive to permutations of atoms which can result in different property values being estimated for a same crystal.

Original: Montavon et al.
Revised: Montavon et al.

Original: (2012) provided an invariant solution by training the NN on a set of randomly permuted matrices.
Revised: (2012) provided an invariant solution by training the NN on a set of randomly permuted matrices.

Original: To show the beneﬁt of their method, they train different ML models on the QM7 dataset and improved on the accuracy of previous approaches by a factor of 3.
Revised: To show the beneﬁt of their method, they train different ML models on the QM7 dataset and improved on the accuracy of previous approaches by a factor of 3.

Original: A recent advancement by Gilmer et al.
Revised: A recent advancement by Gilmer et al.

Original: (2017) provided a more versatile representation for molecules as a chemical graph where nodes describe atoms and edges encode the distances and bond type between them.
Revised: (2017) provided a more versatile representation for molecules as a chemical graph where nodes describe atoms and edges encode the distances and bond type between them.

Original: Their Message Passing Neural Network (MPNN) simulates the atomic interactions through the passing of messages between the nodes.
Revised: Their Message Passing Neural Network (MPNN) simulates the atomic interactions through the passing of messages between the nodes.

Original: So far, this work has only been evaluated on stable conﬁgurations of molecules from the QM9 dataset.
Revised: So far, this work has only been evaluated on stable conﬁgurations of molecules from the QM9 dataset.

Original: We further evaluate MPNN on unstable molecules and crystals.
Revised: We further evaluate MPNN on unstable molecules and crystals.

Original: Furthermore, we extend on their approach to allow for more accurate predictions by better accounting for the physics of atomic interactions.
Revised: Furthermore, we extend on their approach to allow for more accurate predictions by better accounting for the physics of atomic interactions.

Original:  D ATASETS
Revised:  D ATASETS

Original: Many previous ML approaches and associated datasets only consider the case of molecules and crystals at their stable conﬁguration, therefore learning the interaction of atoms only at an equilibrium state.
Revised: Many previous ML approaches and associated datasets only consider the case of molecules and crystals at their stable conﬁguration, therefore learning the interaction of atoms only at an equilibrium state.

Original: The estimation of potential energies for unstable systems needs to be supported by a dataset of unstable systems.
Revised: The estimation of potential energies for unstable systems needs to be supported by a dataset of unstable systems.

Original: We created 3 such datasets to train and benchmark VIMPNN.
Revised: We created 3 such datasets to train and benchmark VIMPNN.

Original: The ﬁrst is an augmented QM9 dataset (Section 3.1) to investigate the complexity of atomic interactions from a variety of atom types in small but diverse molecules.
Revised: The ﬁrst is an augmented QM9 dataset (Section 3.1) to investigate the complexity of atomic interactions from a variety of atom types in small but diverse molecules.

Original: In the second dataset, inﬁnite-size crystals (Section 3.2) allow learning regular bonding patterns that arise in periodic structures.
Revised: In the second dataset, inﬁnite-size crystals (Section 3.2) allow learning regular bonding patterns that arise in periodic structures.

Original: Thirdly, a dataset containing growing crystals of an increasing (ﬁnite) size and complexity enables experimenting with large scale interactions (Section 3.3).
Revised: Thirdly, a dataset containing growing crystals of an increasing (ﬁnite) size and complexity enables experimenting with large scale interactions (Section 3.3).

Original: 3. A UGMENTED QM9 D ATASET
Revised: 3. A UGMENTED QM9 D ATASET

Original: An augmented QM9 dataset is created by taking the ﬁrst 10,000 molecules of QM9 and modifying the interatomic distances at 10 regular intervals between 90 and 150% of the original stable conﬁguration.
Revised: An augmented QM9 dataset is created by taking the ﬁrst 10,000 molecules of QM9 and modifying the interatomic distances at 10 regular intervals between 90 and 150% of the original stable conﬁguration.

Original: This dataset therefore contains 100K different systems.
Revised: This dataset therefore contains 100K different systems.

Original: At each interval, the potential energy is calculated using DFT 2 to serve as the target energy for our supervised learning task.
Revised: At each interval, the potential energy is calculated using DFT 2 to serve as the target energy for our supervised learning task.

Original: 3. I NFINITE C RYSTAL D ATASET
Revised: 3. I NFINITE C RYSTAL D ATASET

Original: Learning to estimate the potential energy for an inﬁnitely sized crystal might beneﬁt from the regular pattern in the lattice structure, possibly reducing the complexity of modelling for the ML algorithm.
Revised: Learning to estimate the potential energy for an inﬁnitely sized crystal might beneﬁt from the regular pattern in the lattice structure, possibly reducing the complexity of modelling for the ML algorithm.

Original: A dataset of inﬁnite-sized crystals, although not necessarily relevant from a physical point of view, is therefore an interesting case study for the design and training of ML methods.
Revised: A dataset of inﬁnite-sized crystals, although not necessarily relevant from a physical point of view, is therefore an interesting case study for the design and training of ML methods.

Original: We use the simple
Revised: We use the simple

Original: Face-Centred Cubic (fcc)
Revised: Face-Centred Cubic (fcc)

Original: Bravais lattice illustrated in Fig.
Revised: Bravais lattice illustrated in Fig.

Original: 1 left for both Al and Cu atoms.
Revised: 1 left for both Al and Cu atoms.

Original: We iteratively compress/dilate the cell size (thus, the interatomic distances) and compute the potential energies using DFT at 20,000 uniform intervals from 90-150% of the stable conﬁguration.
Revised: We iteratively compress/dilate the cell size (thus, the interatomic distances) and compute the potential energies using DFT at 20,000 uniform intervals from 90-150% of the stable conﬁguration.

Original: The change of atomic distances are performed isometrically (i.e. the same change in all axes)
Revised: The change of atomic distances are performed isometrically (i.e. the same change in all axes)

Original: 3. C RYSTAL G ROWTH D ATASET
Revised: 3. C RYSTAL G ROWTH D ATASET

Original: In contrast to the regular lattice pattern created by an inﬁnite crystal, crystal growth allows for learning over more complex atomic interactions in non-regular systems.
Revised: In contrast to the regular lattice pattern created by an inﬁnite crystal, crystal growth allows for learning over more complex atomic interactions in non-regular systems.

Original: Starting from a basic fcc crystal seed of 14 atoms (Fig. 1 centre), new crystalline systems are generated by iteratively placing atoms at a random location on the surface of the growing crystal following its lattice pattern (Fig.right), with sizes ranging from 15 to 114 atoms.
Revised: Starting from a basic fcc crystal seed of 14 atoms (Fig. 1 centre), new crystalline systems are generated by iteratively placing atoms at a random location on the surface of the growing crystal following its lattice pattern (Fig.right), with sizes ranging from 15 to 114 atoms.

Original: Such a dataset enables evaluation of an ML method’s ability to learn how each atom contributes to the ﬁnal potential energy.
Revised: Such a dataset enables evaluation of an ML method’s ability to learn how each atom contributes to the ﬁnal potential energy.

Original: We generate two subsets of this dataset: one that includes compression/dilation of the interatomic distances at 10 regular intervals between 90 to 150% of the stable distance, referred to as the Unstable
Revised: We generate two subsets of this dataset: one that includes compression/dilation of the interatomic distances at 10 regular intervals between 90 to 150% of the stable distance, referred to as the Unstable

Original: Crystal Growth subset, and another that contains only stable conﬁgurations, referred to as the Stable
Revised: Crystal Growth subset, and another that contains only stable conﬁgurations, referred to as the Stable

Original: Crystal Growth subset.
Revised: Crystal Growth subset.

Original: The latter aims to evaluate an ML model’s capability to estimate energy values at varying system sizes (i.e. number of atoms) without the added complexity of learning energy as a function of distance.
Revised: The latter aims to evaluate an ML model’s capability to estimate energy values at varying system sizes (i.e. number of atoms) without the added complexity of learning energy as a function of distance.

Original: Both subsets contain Al and Cu crystalline systems.
Revised: Both subsets contain Al and Cu crystalline systems.

Original: For the Stable subset, we use 20 random seeds for each atom type, thus creating 40 varied crystal growths and 4,000 crystalline systems.
Revised: For the Stable subset, we use 20 random seeds for each atom type, thus creating 40 varied crystal growths and 4,000 crystalline systems.

Original: For the Unstable subset, we select 5 of these random seeds for each atom type to consider 1,000 basic conﬁgurations, each being compressed/dilated for a total of 10,000 crystalline systems.
Revised: For the Unstable subset, we select 5 of these random seeds for each atom type to consider 1,000 basic conﬁgurations, each being compressed/dilated for a total of 10,000 crystalline systems.

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: VIMPNN estimates the potential energies for molecular or crystaline systems through learning the energy as a function of the geometry of bonded atoms.
Revised: VIMPNN estimates the potential energies for molecular or crystaline systems through learning the energy as a function of the geometry of bonded atoms.

Original: By querying VIMPNN at various interatomic distances, we may ﬁnd a stable conﬁguration (Fig. 2).
Revised: By querying VIMPNN at various interatomic distances, we may ﬁnd a stable conﬁguration (Fig. 2).

Original: VIMPNN extends MPNN (Gilmer et al., 2017) that simulates atomic interactions through the passing of messages between pairs of atoms (Section 4.1).
Revised: VIMPNN extends MPNN (Gilmer et al., 2017) that simulates atomic interactions through the passing of messages between pairs of atoms (Section 4.1).

Original: However, VIMPNN further accounts for the physics of the system through an architecture that passes information according to the different forms of bonding between atoms (Section 4.2), and through reinforcing the physical relevance of the analysis and learnt features using auxiliary physical properties (Section 4.3).
Revised: However, VIMPNN further accounts for the physics of the system through an architecture that passes information according to the different forms of bonding between atoms (Section 4.2), and through reinforcing the physical relevance of the analysis and learnt features using auxiliary physical properties (Section 4.3).

Original: A representation of the VIMPNN model is depicted in Fig.
Revised: A representation of the VIMPNN model is depicted in Fig.

Original: 4. P REREQUISITE : MPNN
Revised: 4. P REREQUISITE : MPNN

Original: MPNN learns to predict potential energy from an undirected graph G , where atoms are nodes and edges e vw describe the relation between two nodes v and w (e.g. their distance).
Revised: MPNN learns to predict potential energy from an undirected graph G , where atoms are nodes and edges e vw describe the relation between two nodes v and w (e.g. their distance).

Original: MPNN iterates through three functions for a ﬁxed number of iterations (recommended between 3 and 8): 1) the Message function ( M t in Eq. 1, implemented with a perceptron) creates a message packet that symbolises the action of a neighbouring atom on node v .
Revised: MPNN applies three functions for a ﬁxed number of iterations (recommended between 3 and 8): 1) the Message function ( M t in Eq. 1, implemented with a perceptron) creates a message packet that symbolises the action of a neighbouring atom on node v .

Original: This action does not replicate the physics of atomic interactions happening in real chemical systems, but it enforces the NN model to process information in a physics-inspired way.
Revised: This action does not replicate the physics of atomic interactions happening in real chemical systems, but it enforces the NN model to process information in a physics-inspired way.

Original: A ﬁnal message vector m tv for node v is created in Eq.
Revised: A ﬁnal message vector m tv for node v is created in Eq.

Original: 1 by combining the actions of all neighbours.
Revised: 1 by combining the actions of all neighbours.

Original: An update function ( U t in Eq. 2) updates the hidden state h tv of node v using its previous hidden state h t − 1 v and message m tv .
Revised: An update function ( U t in Eq. 2) updates the hidden state h tv of node v using its previous hidden state h t − 1 v and message m tv .

Original: It accounts for the effect of atomic interactions on the (information) properties of the node’s atom.
Revised: It accounts for the effect of atomic interactions on the (information) properties of the node’s atom.

Original: It is implemented by a Gated Recurrent Unit (GRU).
Revised: It is implemented by a Gated Recurrent Unit (GRU).

Original: The Readout function ( R in Eq.
Revised: The Readout function ( R in Eq.

Original: 3, implemented with a perceptron) uses the set of hidden state of all nodes at all timesteps to estimate the system’s potential energy.
Revised: 3, implemented with a perceptron) uses the set of hidden state of all nodes at all timesteps to estimate the system’s potential energy.

Original: Thus, it exploits the properties of atoms and of their exchanges of information, to learn to estimate the energy through minimising a mean-squared error loss.
Revised: Thus, it exploits the properties of atoms and of their exchanges of information, to learn to estimate the energy through minimising a mean-squared error loss.

Original: [Equation]
Revised: [Equation]

Original: Full details on the implementation of these three functions can be found in the original paper (see
Revised: Full details on the implementation of these three functions can be found in the original paper (see

Original: Gilmer et al.
Revised: Gilmer et al.

Original: Next, we detail how VIMPNN adapts these functions to further incorporate physics principles in order to better handle out-of-equilibrium systems.
Revised: Next, we detail how VIMPNN adapts these functions to further incorporate physics principles in order to better handle out-of-equilibrium systems.

Original: 4. A CCOUNTING FOR B OND T YPES
Revised: 4. A CCOUNTING FOR B OND T YPES

Original: The type of bonds between different atoms are an important factor when considering their interaction and its contribution to the energy of the system.
Revised: The type of bonds between different atoms are an important factor when considering their interaction and its contribution to the energy of the system.

Original: Chemical bonds between atoms can be characterised by the number of valence electrons exchanged in the process of bonding.
Revised: Chemical bonds between atoms can be characterised by the number of valence electrons exchanged in the process of bonding.

Original: For example, ionic bonding requires one donor electron and one acceptor, while in situations where the bonding atoms have similar electronegativity, some electrons would be shared between the bonding atoms resulting in a covalent bond.
Revised: For example, ionic bonding requires one donor electron and one acceptor, while in situations where the bonding atoms have similar electronegativity, some electrons would be shared between the bonding atoms resulting in a covalent bond.

Original: Moreover, most atomic systems have minimum potential energy (stable bond) at an optimal atom separation (Feinberg & Ruedenberg, 1971).
Revised: Moreover, most atomic systems have minimum potential energy (stable bond) at an optimal atom separation (Feinberg & Ruedenberg, 1971).

Original: Therefore, the potential energy of a system may be better estimated by accounting for the contribution of different bond types.
Revised: Therefore, the potential energy of a system may be better estimated by accounting for the contribution of different bond types.

Original: Gilmer et al.
Revised: Gilmer et al.

Original: (2017) acknowledged this fact and introduced bond type information as a feature in the edges e vw of the undirected graph representing the chemical system.
Revised: (2017) acknowledged this fact and introduced bond type information as a feature in the edges e vw of the undirected graph representing the chemical system.

Original: This information was used together with interatomic distance by the message function M t in Eq.
Revised: This information was used together with interatomic distance by the message function M t in Eq.

Original: 1 to estimate the inﬂuence of neighbouring atoms on a node.
Revised: 1 to estimate the inﬂuence of neighbouring atoms on a node.

Original: Thus, messages exchanged by the NN nodes are impacted by the bond type.
Revised: Thus, messages exchanged by the NN nodes are impacted by the bond type.

Original: Gilmer et al.
Revised: Gilmer et al.

Original: (2017) demonstrated this approach results in better energy estimates over using distance alone.
Revised: (2017) demonstrated this approach results in better energy estimates over using distance alone.

Original: These encouraging results inspired us to take this physics integration principle further and to modify the design of information exchange within the NN to better follow the physics of atomic interactions.
Revised: These encouraging results inspired us to take this physics integration principle further and to modify the design of information exchange within the NN to better follow the physics of atomic interactions.

Original: We propose to introduce different specialised communication channels based on bond type , with a direct effect on the update of nodes (Eq. 2).
Revised: We propose to introduce different specialised communication channels based on bond type , with a direct effect on the update of nodes (Eq. 2).

Original: This is supported by separate messages m v | BT for each bond type, computed as in Eq.
Revised: This is supported by separate messages m v | BT for each bond type, computed as in Eq.

Original: 1 to combine all message packets of a same bond type.
Revised: 1 to combine all message packets of a same bond type.

Original: Bond type is quickly predetermined based on the atoms’ valency and electronegativity using the RDKit software 3 .
Revised: Bond type is quickly predetermined based on the atoms’ valency and electronegativity using the RDKit software 3 .

Original: In practice, the principle of having different messaging channels and node update based on bond type may be implemented in various ways.
Revised: In practice, the principle of having different messaging channels and node update based on bond type may be implemented in various ways.

Original: We experimented with: a) combining the messages m v | BT as a weighted sum, where weights are learnt by the NN to control the inﬂuence of each m v | BT :
Revised: We experimented with: a) combining the messages m v | BT as a weighted sum, where weights are learnt by the NN to control the inﬂuence of each m v | BT :

Original: [Equation]
Revised: [Equation]

Original: where λ BT is the weight for bond type BT , and t is omitted for readability.
Revised: where λ BT is the weight for bond type BT , and t is omitted for readability.

Original: We tried with λ BT being i) a vector of same size as the message (with an element-wise product in Eq. 4), and ii) a simple scalar.
Revised: We tried with λ BT being i) a vector of same size as the message (with an element-wise product in Eq. 4), and ii) a simple scalar.

Original: The ﬁnal combined message m v is provided to the GRU of Eq.
Revised: The ﬁnal combined message m v is provided to the GRU of Eq.

Original: 2 to compute a node update where different bond types contribute differently.
Revised: 2 to compute a node update where different bond types contribute differently.

Original: Another possibility is: b) concatenating the messages m v | BT before providing them to the GRU of Eq.
Revised: Another possibility is: b) concatenating the messages m v | BT before providing them to the GRU of Eq.

Original: We experimented with i) letting the
Revised: We experimented with i) letting the

Original: GRU handle the concatenated m v | BT freely, and with ii) imposing a similar handling of the different m v | BT through using the same GRU parameters, duplicated for each m v | BT with a simple (learnt) scaling to allow different weightings of the m v | BT messages in the node update.
Revised: GRU handle the concatenated m v | BT freely, and with ii) imposing a similar handling of the different m v | BT through using the same GRU parameters, duplicated for each m v | BT with a simple (learnt) scaling to allow different weightings of the m v | BT messages in the node update.

Original: Finally, we also experimented with: c) having one GRU for each bond type, handling its corresponding message m v | BT .
Revised: Finally, we also experimented with: c) having one GRU for each bond type, handling its corresponding message m v | BT .

Original: The updates of all GRUs were summed to compute the ﬁnal nodes’ update:
Revised: The updates of all GRUs were summed to compute the ﬁnal nodes’ update:

Original: [Equation]
Revised: [Equation]

Original: where U BT is the update function implemented by the GRU associated with bond type BT , and t being again omitted for readability.
Revised: where U BT is the update function implemented by the GRU associated with bond type BT , and t being again omitted for readability.

Original: As demonstrated in the Appendix (Table 3), we found that the best results were obtained in the case a.ii) with combining all messages m v | BT in a simple way while letting the NN weight the inﬂuence of each bond type specialised message m v | BT .
Revised: As demonstrated in the Appendix (Table 3), we found that the best results were obtained in the case a.ii) with combining all messages m v | BT in a simple way while letting the NN weight the inﬂuence of each bond type specialised message m v | BT .

Original: 4. E STIMATING A UXILIARY P HYSICAL P ROPERTIES
Revised: 4. E STIMATING A UXILIARY P HYSICAL P ROPERTIES

Original: We hypothesise that, by encouraging the VIMPNN’s hidden states to relate more to basic physical properties, we may obtain a more accurate energy estimator that generalises better to new systems.
Revised: We hypothesise that, by encouraging the VIMPNN’s hidden states to relate more to basic physical properties, we may obtain a more accurate energy estimator that generalises better to new systems.

Original: In addition, as the bond type is characterised by the valence property of atoms involved in the bonding process, a more physically relevant hidden state may better support the differentiation of messages and node updates per bond type (Section 4.2).
Revised: In addition, as the bond type is characterised by the valence property of atoms involved in the bonding process, a more physically relevant hidden state may better support the differentiation of messages and node updates per bond type (Section 4.2).

Original: In practice, we encourage this greater relationship with physical parameters through the estimation of auxiliary properties through parallel readout functions similar to Eq. 3.
Revised: In practice, we encourage this greater relationship with physical parameters through the estimation of auxiliary properties through parallel readout functions similar to Eq. 3.

Original: We present this strategy as a second and easy way of integrating knowledge (of task-related physical properties) into the model.
Revised: We present this strategy as a second and easy way of integrating knowledge (of task-related physical properties) into the model.

Original: We experimented with: 1) the number of atoms of each type present in the system, 2) the number of orbitals associated with each atom type – a property that is directly relevant to the determination of bond type –, and 3) a probability distribution for the scaling to the stable distance, estimated as a Gaussian function.
Revised: We experimented with: 1) the number of atoms of each type present in the system, 2) the number of orbitals associated with each atom type – a property that is directly relevant to the determination of bond type –, and 3) a probability distribution for the scaling to the stable distance, estimated as a Gaussian function.

Original: For each auxiliary estimation, a mean-squared error loss term (weighted by an α = 0 .
Revised: For each auxiliary estimation, a mean-squared error loss term (weighted by an α = 0 .

Original: 3 hyperparameter) is minimised during training.
Revised: 3 hyperparameter) is minimised during training.

Original:  R ESULTS AND D ISCUSSION
Revised:  R ESULTS AND D ISCUSSION

Original: In this section, the performance impact for each of the physics integration strategies introduced in
Revised: In this section, the performance impact for each of the physics integration strategies introduced in

Original: Section 4 are ﬁrst measured in turn on our augmented QM9 dataset (Section 5.1).
Revised: Section 4 are ﬁrst measured in turn on our augmented QM9 dataset (Section 5.1).

Original: The VIMPNN model, that includes all augmentations, is further evaluated on all datasets to demonstrate its versatility on different scenarios (Section 3), using MPNN as a baseline for comparison (Section 5.2).
Revised: The VIMPNN model, that includes all augmentations, is further evaluated on all datasets to demonstrate its versatility on different scenarios (Section 3), using MPNN as a baseline for comparison (Section 5.2).

Original: In particular, we test the generalisation ability of VIMPNN by training it on crystalline systems of up to 25 atoms to predict the energies of systems of up to 75 atoms (Section 5.3).
Revised: In particular, we test the generalisation ability of VIMPNN by training it on crystalline systems of up to 25 atoms to predict the energies of systems of up to 75 atoms (Section 5.3).

Original: We ﬁnally explore the physics relevance and interpretability of VIMPNN by visualising its internal states (Section 5.4).
Revised: We ﬁnally explore the physics relevance and interpretability of VIMPNN by visualising its internal states (Section 5.4).

Original: In all experiments, the dataset is split into training, testing, and validation sets in proportions appropriate for the complexity of the data.
Revised: In all experiments, the dataset is split into training, testing, and validation sets in proportions appropriate for the complexity of the data.

Original: The Augmented QM9 training set consists of all data augmentations, i.e. 10 compression/dilation operations, for 80% of the QM9 molecules present in the dataset, so 8,000 molecules, totalling 80,000 training samples.
Revised: The Augmented QM9 training set consists of all data augmentations, i.e. 10 compression/dilation operations, for 80% of the QM9 molecules present in the dataset, so 8,000 molecules, totalling 80,000 training samples.

Original: Following the same rules, the testing and validation sets each contain 10% of the dataset, so 10,000 samples.
Revised: Following the same rules, the testing and validation sets each contain 10% of the dataset, so 10,000 samples.

Original: As the Inﬁnite Crystal dataset has 2 different crystal types (Cu and Al crystals), 8 data points are uniformly sampled from their respective range of compression/dilation for training.
Revised: As the Inﬁnite Crystal dataset has 2 different crystal types (Cu and Al crystals), 8 data points are uniformly sampled from their respective range of compression/dilation for training.

Original: The rest of data is split evenly between validation and test.
Revised: The rest of data is split evenly between validation and test.

Original: For the Crystal Growth dataset, we use a 60/20/20 split with the same procedure.
Revised: For the Crystal Growth dataset, we use a 60/20/20 split with the same procedure.

Original: 5. P HYSICS I NTEGRATION P ERFORMANCES
Revised: 5. P HYSICS I NTEGRATION P ERFORMANCES

Original: We ﬁrst evaluate the effectiveness of the physics integration strategies listed in Section 4 by comparing them, in turn, against the baseline of no physics integration (i.e. e vw only contains distance and no bond type information) and MPNN (i.e. bond type information used in e vw ).
Revised: We ﬁrst evaluate the effectiveness of the physics integration strategies listed in Section 4 by comparing them, in turn, against the baseline of no physics integration (i.e. e vw only contains distance and no bond type information) and MPNN (i.e. bond type information used in e vw ).

Original: We report, in Table 1, average and standard deviation metrics for the mean-absolute error (MAE), the mean-squared error (MSE), and the relative error (RE = | ˆ y − y | | y | ) between the true y and predicted ˆ y energy values.
Revised: We report, in Table 1, average and standard deviation metrics for the mean-absolute error (MAE), the mean-squared error (MSE), and the relative error (RE = | ˆ y − y | | y | ) between the true y and predicted ˆ y energy values.

Original: All proposed methods have a positive effect on energy estimation, with the introduction of bond type information (rows 2-3) having a particularly positive impact.
Revised: All proposed methods have a positive effect on energy estimation, with the introduction of bond type information (rows 2-3) having a particularly positive impact.

Original: This conﬁrms Gilmer et al.
Revised: This conﬁrms Gilmer et al.

Original: (2017)’s
Revised: (2017)’s

Original: [Table] [Table]
Revised: [Table] [Table]

Original: observation that the bond type is a good feature for the estimation of energy.
Revised: observation that the bond type is a good feature for the estimation of energy.

Original: Furthermore, the use of bond type information in the design of the NN architecture (case a.ii in Section 4.2) has a stronger positive impact than using it as a feature in MPNN.
Revised: Furthermore, the use of bond type information in the design of the NN architecture (case a.ii in Section 4.2) has a stronger positive impact than using it as a feature in MPNN.

Original: This suggests that having specialised messaging channels based on bond type better captures the physics of atomic interactions than just including bond type information in generic messages.
Revised: This suggests that having specialised messaging channels based on bond type better captures the physics of atomic interactions than just including bond type information in generic messages.

Original: However, MPNN does improve signiﬁcantly on the baseline, therefore in future works it may be interesting to combine these two strategies.
Revised: However, MPNN does improve signiﬁcantly on the baseline, therefore in future works it may be interesting to combine these two strategies.

Original: The auxiliary estimation of physical properties also improve on the baseline model, but less so than the integration of bond type information.
Revised: The auxiliary estimation of physical properties also improve on the baseline model, but less so than the integration of bond type information.

Original: We interpret this as a result of the model having to implicitly discover and encode the useful physics representations required for accurate predictions.
Revised: We interpret this as a result of the model having to implicitly discover and encode the useful physics representations required for accurate predictions.

Original: Indeed, the auxiliary quantities used are related to the global properties of the molecule, and in the case of the distance scaling to a high level system optimisation.
Revised: Indeed, the auxiliary quantities used are related to the global properties of the molecule, and in the case of the distance scaling to a high level system optimisation.

Original: In future works, it may be interesting to investigate the effects of auxiliary quantities linked to the physics of individual atoms/nodes.
Revised: In future works, it may be interesting to investigate the effects of auxiliary quantities linked to the physics of individual atoms/nodes.

Original: The more modest effect of auxiliary properties may still be beneﬁcial in combination to the bond type strategy, as we demonstrate in the next experiment and the Appendix, thus we continue using them.
Revised: The more modest effect of auxiliary properties may still be beneﬁcial in combination to the bond type strategy, as we demonstrate in the next experiment and the Appendix, thus we continue using them.

Original: 5. E VALUATION OF VIMPNN ON OUT - OF - EQUILIBRIUM MOLECULAR AND CRYSTAL DATA
Revised: 5. E VALUATION OF VIMPNN ON OUT - OF - EQUILIBRIUM MOLECULAR AND CRYSTAL DATA

Original: In this experiment, we update the baseline model of Section 5.1 with our physics integration strategies, namely bond type-specialised communication channels and node updates (case a.ii of Section 4.2) and all 3 of the auxiliary estimations to create VIMPNN.
Revised: In this experiment, we update the baseline model of Section 5.1 with our physics integration strategies, namely bond type-specialised communication channels and node updates (case a.ii of Section 4.2) and all 3 of the auxiliary estimations to create VIMPNN.

Original: The energy estimation is assessed on the different scenarios provided by our 3 datasets.
Revised: The energy estimation is assessed on the different scenarios provided by our 3 datasets.

Original: On the Crystal Growth subsets, we consider system sizes ranging from 15 to 75 atoms, instead of the available 114, due to memory restrictions.
Revised: On the Crystal Growth subsets, we consider system sizes ranging from 15 to 75 atoms, instead of the available 114, due to memory restrictions.

Original: The performance of MPNN and VIMPNN are reported in Table 2 using the same metrics as previously, and the distance to stable conﬁguration (DSC) where the interatomic distance at the minimum of the predicted energies is compared to the known at-equilibrium distance by absolute difference.
Revised: The performance of MPNN and VIMPNN are reported in Table 2 using the same metrics as previously, and the distance to stable conﬁguration (DSC) where the interatomic distance at the minimum of the predicted energies is compared to the known at-equilibrium distance by absolute difference.

Original: Our ﬁrst observation, when comparing with Table 1, is that the auxiliary estimates, which were found to be beneﬁcial to the model in the previous experiment when used in isolation, also prove to be beneﬁcial when used in conjunction with the bond type-specialised communication channels.
Revised: Our ﬁrst observation, when comparing with Table 1, is that the auxiliary estimates, which were found to be beneﬁcial to the model in the previous experiment when used in isolation, also prove to be beneﬁcial when used in conjunction with the bond type-specialised communication channels.

Original: VIMPNN generally obtains better scores than MPNN on all datasets, meaning that energy estimates are similar or more accurate and that their minimum is closer to the true stable interatomic distance (Fig. 3 left).
Revised: VIMPNN generally obtains better scores than MPNN on all datasets, meaning that energy estimates are similar or more accurate and that their minimum is closer to the true stable interatomic distance (Fig. 3 left).

Original: The improvement over MPNN is particularly strong on the Crystal Growth dataset, hinting that VIMPNN is able to generalise better to new geometries.
Revised: The improvement over MPNN is particularly strong on the Crystal Growth dataset, hinting that VIMPNN is able to generalise better to new geometries.

Original: We also note that MPNN’s metrics, especially DSC, suffer from a high standard deviation.
Revised: We also note that MPNN’s metrics, especially DSC, suffer from a high standard deviation.

Original: Upon examination, we ﬁnd that there exists some more difﬁcult cases 4 which may explain these results.
Revised: Upon examination, we ﬁnd that there exists some more difﬁcult cases 4 which may explain these results.

Original: Indeed, when examining these cases more closely (Fig. 3 right), we ﬁnd that MPNN’s estimations are usually close to the true energy values but do not contain a minimum.
Revised: Indeed, when examining these cases more closely (Fig. 3 right), we ﬁnd that MPNN’s estimations are usually close to the true energy values but do not contain a minimum.

Original: On the other hand, while VIMPNN’s estimates are slightly worsened by an offset, it is still able to account for the physics of the system, such as the repulsive effect of atoms getting too close to each other, to predict an energy minimum at a reasonable interatomic distance.
Revised: On the other hand, while VIMPNN’s estimates are slightly worsened by an offset, it is still able to account for the physics of the system, such as the repulsive effect of atoms getting too close to each other, to predict an energy minimum at a reasonable interatomic distance.

Original: The energy offset may be addressed in future works, for example by a different loss term or new auxiliary estimates.
Revised: The energy offset may be addressed in future works, for example by a different loss term or new auxiliary estimates.

Original: The Inﬁnite Crystal dataset is the only case where
Revised: The Inﬁnite Crystal dataset is the only case where

Original: VIMPNN doesn’t improve on MPNN in term of the accuracy of energy estimation, both methods obtaining similar scores, but with VIMPNN still producing better DSC measures by almost an order of magnitude.
Revised: VIMPNN doesn’t improve on MPNN in term of the accuracy of energy estimation, both methods obtaining similar scores, but with VIMPNN still producing better DSC measures by almost an order of magnitude.

Original: As we only consider two mono-atomic inﬁnite crystals in our experiments, we may see further improvements by expanding this dataset to include a higher number of structures outside of the fcc lattice and a wider variety of atom types.
Revised: As we only consider two mono-atomic inﬁnite crystals in our experiments, we may see further improvements by expanding this dataset to include a higher number of structures outside of the fcc lattice and a wider variety of atom types.

Original: Furthermore, when trained on the Crystal Growth dataset, the results of MPNN on Cu crystals are noticeably worse than for Al (see Fig. 4).
Revised: Furthermore, when trained on the Crystal Growth dataset, the results of MPNN on Cu crystals are noticeably worse than for Al (see Fig. 4).

Original: We interpret this as MPNN not being able to learn how to handle both crystal types simultaneously.
Revised: We interpret this as MPNN not being able to learn how to handle both crystal types simultaneously.

Original: VIMPNN, however, maintains a similarly good accuracy in both cases.
Revised: VIMPNN, however, maintains a similarly good accuracy in both cases.

Original: We hypothesise that the auxiliary estimations of number of each type of atom may help
Revised: We hypothesise that the auxiliary estimations of number of each type of atom may help

Original: VIMPNN to better differentiate between these atom types.
Revised: VIMPNN to better differentiate between these atom types.

Original: We test this hypothesis by removing the atom-type estimation from VIMPNN.
Revised: We test this hypothesis by removing the atom-type estimation from VIMPNN.

Original: The resulting high errors for Cu crystals demonstrate that the auxiliary estimation has been an effective way to integrate knowledge into the model, in this case by directing the attention of the model to atom type, a very relevant physical property for this problem.
Revised: The resulting high errors for Cu crystals demonstrate that the auxiliary estimation has been an effective way to integrate knowledge into the model, in this case by directing the attention of the model to atom type, a very relevant physical property for this problem.

Original: Other combinations of auxiliary estimations have been tested in the Appendix (Table 4).
Revised: Other combinations of auxiliary estimations have been tested in the Appendix (Table 4).

Original: We further evaluate how VIMPNN scales to larger systems by comparing energy estimates on crystalline systems of various sizes (i.e. number of atoms) from the Stable Crystal Growth subset.
Revised: We further evaluate how VIMPNN scales to larger systems by comparing energy estimates on crystalline systems of various sizes (i.e. number of atoms) from the Stable Crystal Growth subset.

Original: The increasing energy errors for larger sizes in Fig.
Revised: The increasing energy errors for larger sizes in Fig.

Original: 6 left suggest that the model struggles with handling many simultaneous interactions at a higher range of interatomic separations.
Revised: 6 left suggest that the model struggles with handling many simultaneous interactions at a higher range of interatomic separations.

Original: When examining examples of such systems more closely, e.g. in Fig.
Revised: When examining examples of such systems more closely, e.g. in Fig.

Original: 5, we observe that VIMPNN ﬁnds an energy minimum at the correct stable interatomic distance for the small system (Fig.
Revised: 5, we observe that VIMPNN ﬁnds an energy minimum at the correct stable interatomic distance for the small system (Fig.

Original: 5 left, 15 atoms) even if the estimated energy has an offset, but in extreme cases (Fig.
Revised: 5 left, 15 atoms) even if the estimated energy has an offset, but in extreme cases (Fig.

Original: 5 right, 75 atoms)
Revised: 5 right, 75 atoms)

Original: the predicted energies
Revised: the predicted energies

Original: VIMPNN no longer present a correctly located minimum.
Revised: VIMPNN no longer present a correctly located minimum.

Original: These results indicate that further work is needed towards better accounting for atomic interactions within large scale structures.
Revised: These results indicate that further work is needed towards better accounting for atomic interactions within large scale structures.

Original: 5. G ENERALISATION TO LARGER SYSTEMS
Revised: 5. G ENERALISATION TO LARGER SYSTEMS

Original: In this section we investigate VIMPNN’s ability to learn basic principles of atomic interactions in small chemical systems that are transferable to arbitrary (especially larger) system sizes.
Revised: In this section we investigate VIMPNN’s ability to learn basic principles of atomic interactions in small chemical systems that are transferable to arbitrary (especially larger) system sizes.

Original: This ability would allow for quicker training times while generalising well irrespective of the graph size.
Revised: This ability would allow for quicker training times while generalising well irrespective of the graph size.

Original: To test whether VIMPNN is capable of such task, we train the model on a ﬁltered version of the Stable
Revised: To test whether VIMPNN is capable of such task, we train the model on a ﬁltered version of the Stable

Original: Crystal Growth subset that includes only systems of up to 25 atoms.
Revised: Crystal Growth subset that includes only systems of up to 25 atoms.

Original: The validation and testing are done on the full subset, thus the model estimates energies for systems of up to 75 atoms 5 .
Revised: The validation and testing are done on the full subset, thus the model estimates energies for systems of up to 75 atoms 5 .

Original: Fig.
Revised: Fig.

Original: 6 (right) shows that the MAE increases with the system size above 25 atoms, but this trend is only marginally stronger than when the model is trained on all sizes (Fig. 6 left, of the previous experiment).
Revised: 6 (right) shows that the MAE increases with the system size above 25 atoms, but this trend is only marginally stronger than when the model is trained on all sizes (Fig. 6 left, of the previous experiment).

Original: This indicates that VIMPNN is learning some basic principles about the atomic interactions that are transferable to larger, unfamiliar, systems.
Revised: This indicates that VIMPNN is learning some basic principles about the atomic interactions that are transferable to larger, unfamiliar, systems.

Original: However, for VIMPNN to make best use of this positive effect, future work will need to address the overall performance on large system sizes.
Revised: However, for VIMPNN to make best use of this positive effect, future work will need to address the overall performance on large system sizes.

Original: 5. I NTERPRETATION OF THE NODES ’ HIDDEN STATES
Revised: 5. I NTERPRETATION OF THE NODES ’ HIDDEN STATES

Original: When using VIMPNN in-place of DFT, we are placing trust in the model for computational speed with minimal impact in accuracy.
Revised: When using VIMPNN in-place of DFT, we are placing trust in the model for computational speed with minimal impact in accuracy.

Original: While DFT includes approximations to make computations more tractable, it is grounded in fundamental principles of atomic interaction.
Revised: While DFT includes approximations to make computations more tractable, it is grounded in fundamental principles of atomic interaction.

Original: VIMPNN however makes inferences about the system’s energy by searching for a generalised function that matches well the input (chemical graph) and output (energy) data.
Revised: VIMPNN however makes inferences about the system’s energy by searching for a generalised function that matches well the input (chemical graph) and output (energy) data.

Original: It is therefore helpful to investigate the learnt representations of atomic interactions within the NN, to understand how to interpret and trust the model’s estimates and its ability to generalise to new systems.
Revised: It is therefore helpful to investigate the learnt representations of atomic interactions within the NN, to understand how to interpret and trust the model’s estimates and its ability to generalise to new systems.

Original: To do so, we visualise the hidden states of the nodes during the readout phase (step 3 of Section 4.1) that conveniently reduces the nodes’ vector to scalar values before summing to produce the energy estimate.
Revised: To do so, we visualise the hidden states of the nodes during the readout phase (step 3 of Section 4.1) that conveniently reduces the nodes’ vector to scalar values before summing to produce the energy estimate.

Original: Thus, these reduced node values may be considered as the contributions of the nodes to the energy estimate, and these values, normalised across the nodes, are referred to as node’s contribution in the rest of this section.
Revised: Thus, these reduced node values may be considered as the contributions of the nodes to the energy estimate, and these values, normalised across the nodes, are referred to as node’s contribution in the rest of this section.

Original: After training VIMPNN on the Unstable Crystal Growth subset, we consider the node of an atom that is newly added to a (stable) crystal seed of 14 atoms.
Revised: After training VIMPNN on the Unstable Crystal Growth subset, we consider the node of an atom that is newly added to a (stable) crystal seed of 14 atoms.

Original: When compressing and dilating the distance of this (single) atom to the rest of the crystal, we examine its node contribution and that of the rest of the (static) atoms (Fig. 7) 6 .
Revised: When compressing and dilating the distance of this (single) atom to the rest of the crystal, we examine its node contribution and that of the rest of the (static) atoms (Fig. 7) 6 .

Original: At its stable location, the atom’s contribution is marginal, in line with the crystal being at its minimum energy driven by the regular structure of the crystal.
Revised: At its stable location, the atom’s contribution is marginal, in line with the crystal being at its minimum energy driven by the regular structure of the crystal.

Original: As the atom is moved closer or pulled away, its contribution increases simultaneously with the potential energy.
Revised: As the atom is moved closer or pulled away, its contribution increases simultaneously with the potential energy.

Original: At the same time, the relative contributions of the static atoms slightly decrease to give way to the perturbation of the moving atom.
Revised: At the same time, the relative contributions of the static atoms slightly decrease to give way to the perturbation of the moving atom.

Original: This strongly suggests that the model learnt to pay attention to the location of individual atoms, although it was trained on systems that are isometrically compressed/dilated.
Revised: This strongly suggests that the model learnt to pay attention to the location of individual atoms, although it was trained on systems that are isometrically compressed/dilated.

Original: This conﬁrms the previous experiment’s conclusion that VIMPNN learns transferable principles of atomic interaction.
Revised: This conﬁrms the previous experiment’s conclusion that VIMPNN learns transferable principles of atomic interaction.

Original: This visualisation method could inform further improvements in the model, such as related to the aforementioned issues of large range atomic interactions.
Revised: This visualisation method could inform further improvements in the model, such as related to the aforementioned issues of large range atomic interactions.

Original: It may also be helpful in the simulation of crystal growth where a lattice is not imposed, by pinpointing stable atomic positions that entail equal contributions of all atoms to the system’s energy.
Revised: It may also be helpful in the simulation of crystal growth where a lattice is not imposed, by pinpointing stable atomic positions that entail equal contributions of all atoms to the system’s energy.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have shown how integrating physics knowledge into data driven learning models may help improve accuracy and generalisation.
Revised: We have shown how integrating physics knowledge into data driven learning models may help improve accuracy and generalisation.

Original: We proposed two integration strategies, namely specialised information passing channels within a DNN to better account for bond type in the simulation of atomic interactions, and estimating auxiliary physical properties that further relate the learnt representations to the underlying physics of the studied phenomenon.
Revised: We proposed two integration strategies, namely specialised information passing channels within a DNN to better account for bond type in the simulation of atomic interactions, and estimating auxiliary physical properties that further relate the learnt representations to the underlying physics of the studied phenomenon.

Original: Our methods are incorporated into a Valence
Revised: Our methods are incorporated into a Valence

Original: Interaction Message Passing Neural Network (VIMPNN) that estimates the potential energy of both molecular and crystalline systems.
Revised: Interaction Message Passing Neural Network (VIMPNN) that estimates the potential energy of both molecular and crystalline systems.

Original: The energy estimate is shown to be ﬁne enough to identify stable spatial conﬁgurations that correspond to an energy minimum.
Revised: The energy estimate is shown to be ﬁne enough to identify stable spatial conﬁgurations that correspond to an energy minimum.

Original: Three datasets are introduced to benchmark the performance of DNN models: an augmented QM9 dataset to include new unstable conﬁgurations of molecules, inﬁnite-size periodic crystals, and a more complex stable and unstable
Revised: Three datasets are introduced to benchmark the performance of DNN models: an augmented QM9 dataset to include new unstable conﬁgurations of molecules, inﬁnite-size periodic crystals, and a more complex stable and unstable

Original: Crystal Growth dataset with crystalline systems at varying sizes and different spatial conﬁgurations.
Revised: Crystal Growth dataset with crystalline systems at varying sizes and different spatial conﬁgurations.

Original: We evaluate both VIMPNN and MPNN on these datasets, and ﬁnd that our model performs as well or better than MPNN.
Revised: We evaluate both VIMPNN and MPNN on these datasets, and ﬁnd that our model performs as well or better than MPNN.

Original: VIMPNN proves able to learn principles of atomic interactions that are transferable to new system sizes and spatial conﬁgurations.
Revised: VIMPNN proves able to learn principles of atomic interactions that are transferable to new system sizes and spatial conﬁgurations.

Original: However, more work is needed to attain a more robust generalisation to larger chemical systems.
Revised: However, more work is needed to attain a more robust generalisation to larger chemical systems.

Original: This may be achieved in future works by integrating more physics knowledge on long-range atomic interactions into the model.
Revised: This may be achieved in future works by integrating more physics knowledge on long-range atomic interactions into the model.

Original: A A PPENDIX A. F URTHER ASSESSMENT OF THE PROPOSED PHYSICS INTEGRATION STRATEGIES
Revised: A A PPENDIX A. F URTHER ASSESSMENT OF THE PROPOSED PHYSICS INTEGRATION STRATEGIES

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A. T HE EFFECT OF VARYING TIMESTEPS
Revised: A. T HE EFFECT OF VARYING TIMESTEPS

Original: In all experiments of this paper
Revised: In all experiments of this paper

Original: [Table]
Revised: [Table]

Original: three iterations of information passing between nodes were used as the basis for both VIMPNN and MPNN following the default recommendation from Gilmer et al.
Revised: three iterations of information passing between nodes were used as the basis for both VIMPNN and MPNN following the default recommendation from Gilmer et al.

Original: However, we also experimented with both 1 and 8 timesteps, with results reported in Table 5.
Revised: However, we also experimented with both 1 and 8 timesteps, with results reported in Table 5.

Original: We found empirically that 3 iterations works best for VIMPNN as well.
Revised: We found empirically that 3 iterations works best for VIMPNN as well.

Original: [Table]
Revised: [Table]

Original: A. I MPLEMENTATION DETAILS
Revised: A. I MPLEMENTATION DETAILS

Original: All models were trained using an Nvidia Tesla V-100 with PyTorch 1.2.0, Python 3.7.2, and CUDA 10.1.
Revised: All models were trained using an Nvidia Tesla V-100 with PyTorch 1.2.0, Python 3.7.2, and CUDA 10.1.

Original: Each model was trained for a maximum of 360 epochs, with 50 epochs of no validation loss improvement as the criteria for early stopping.
Revised: Each model was trained for a maximum of 360 epochs, with 50 epochs of no validation loss improvement as the criteria for early stopping.

Original: During training, the best model (determined by the validation loss) was saved to be used for testing.
Revised: During training, the best model (determined by the validation loss) was saved to be used for testing.

Original: The following hyperparameters were used for all experiments:
Revised: The following hyperparameters were used for all experiments:

Original: • Batch size: 20 for the Augmented QM9 and Inﬁnite Crystal datasets, and 5 for the Crystal Growth dataset due to memory constraints.
Revised: • Batch size: 20 for the Augmented QM9 and Inﬁnite Crystal datasets, and 5 for the Crystal Growth dataset due to memory constraints.

Original: • Learning rate: 1e- • Learning rate decay: 0.
Revised: • Learning rate: 1e- • Learning rate decay: 0.

Original: • Early stopping: 50 epochs • Auxiliary alpha: 0.
Revised: • Early stopping: 50 epochs • Auxiliary alpha: 0.

Original: • Size of bond type specialised message m v | BT :• Number of bond types:• Number of message passing iterations:
Revised: • Size of bond type specialised message m v | BT :• Number of bond types:• Number of message passing iterations:

Original: When training with the different implementations of the bond type integration, though the size of the bond type specialised message m v | BT remains consistent at the aforementioned 73 dimensions, in case b) the concatenation of bond type messages entails that the nodes’ hidden state size is 219.
Revised: When training with the different implementations of the bond type integration, though the size of the bond type specialised message m v | BT remains consistent at the aforementioned 73 dimensions, in case b) the concatenation of bond type messages entails that the nodes’ hidden state size is 219.

Original: For clarity, we detail the size of nodes’ hidden state here for each of the methods:
Revised: For clarity, we detail the size of nodes’ hidden state here for each of the methods:

Original: • a:• b:• c:
Revised: • a:• b:• c: