Gold-standard solutions to the Schrödinger equation using deep learning: How much physics do we need?
Abstract
Finding accurate solutions to the Schrödinger equation is the key unsolved challenge of computational chemistry.
Given its importance for the development of newchemical compounds, decades of research have been dedicated to this problem, butdue to the large dimensionality even the best available methods do not yet reachthe desired accuracy.
Recently the combination of deep learning with Monte Carlomethods has emerged as a promising way to obtain highly accurate energies andmoderate scaling of computational cost.
In this paper we significantly contributetowards this goal by introducing a novel deep-learning architecture that achieves40-70% lower energy error at 8x lower computational cost compared to previousapproaches.
Using our method we establish a new benchmark by calculating themost accurate variational ground state energies ever published for a number ofdifferent atoms and molecules.
We systematically break down and measure ourimprovements, focusing in particular on the effect of increasing physical priorknowledge.
We surprisingly find that increasing the prior knowledge given to thearchitecture can actually decrease accuracy.
Introduction
The challenge of the Schrödinger Equation Accurately predicting properties of molecules andmaterials is of utmost importance for many applications, including the development of new materialsor pharmaceuticals.
In principle, any property of any molecule can be calculated from its wavefunction, which is obtained by solving the Schrödinger equation.
In practice, computing accuratewavefunctions and corresponding energies is computationally extremely difficult for two reasons:
First, the wavefunction is a high-dimensional function, depending on all coordinates of all electrons,subjecting most methods to the curse of dimensionality.
Second, the required level of accuracy isextremely high.
While total energies of small molecules are typically hundreds of Hartrees, thechemically relevant energy differences are on the order of 1 milli-Hartree as depicted in Fig.
Decades of research have produced a plethora of methods, which all require a trade-off betweenaccuracy and computational cost: On one end of the spectrum are approximate methods such as
Hartree-Fock (HF) or Density Functional Theory (DFT), which was awarded the Nobel prize in1998.
These methods can treat thousands of particles but can often only crudely approximate chemical properties.
On the other end of the spectrum are "gold-standard" methods such as FCI (Full
Configuration Interaction) or CCSD(T) (Coupled Clusters Singles Doubles (Perturbative Triples))which yield energies that often closely agree with experiments, but can only treat up to 100 particles.
Despite all these efforts, even for small molecules there do currently not exist highly accurate energycalculations.
A 2020 benchmark of state-of-the-art methods for the benzene molecule found a spreadof 4 mHa across different methods [1] – as a matter of fact, our results show that the absolute energies
[Figure]
calculated in [1] are off by at least 600 mHa, due to the small basis set used in their calculations.
An important characteristic of a method is the type of approximation being made: Hartree-Fock or
FCI are "variational", meaning that their predicted energies at least upper-bound the ground-truthenergy.
Since a lower energy is always guaranteed to be a closer approximation of the true energy,this makes assessment of these methods straight-forward.
In contrast, CCSD(T) or DFT do not haveany guarantees on the accuracy of their results.
The approximations resulting from such methods,while working well for many common systems, often fail for chemically challenging situations suchas breaking of chemical bonds [2, 3].
Deep-learning-based variational Monte Carlo Combining deep learning and Monte Carlo methods has recently emerged as a promising new approach for solving the Schrödinger equation [4, 5].
These methods offer high accuracy, moderate scaling of computational cost with system size andobey the variational principle.
Within a few years deep-learning-based methods have managed tooutperform conventional high-accuracy methods for many different molecules, potentially defining anew gold-standard for high-accuracy solutions.
In the Born-Oppenheimer approximation a molecule,consisting of n nuc nuclei and n el electrons, is fully described by its Hamiltonian
[Equation]
Here R I , Z I , I ∈ { 1 , . . .
, n nuc } denote the coordinates and charges of the nuclei, r =( r 1 , . . . ,
r n ↑ , . . . ,
r n el ) ∈ R 3 × n el denotes the set of n el Cartesian electron coordinates differentiatedbetween n ↑ spin-up and n ↓ spin-down electrons.
We define the inter-particle vectors r ij = r
i − r jand ρ iJ =
r i − R J .
All properties of the molecule depend on the wavefunction ψ ( r ) , which mustfulfill the antisymmetry constraint: ψ ( P r )
= − ψ ( r ) for any permutation P of two electrons withthe same spin [6].
The wavefunction ψ can be found as the solution to the Schrödinger equation
Hψ = E 0 ψ with the ground-state energy and smallest eigenvalue E 0 .
By the Rayleigh-Ritz principle,the ground-state energy and the corresponding wavefunction can be found through minimization ofthe loss
[Equation]
for a trial wavefunction ψ θ , parameterized by parameters θ .
The trial function ψ θ is represented by aneural network and typically has the form
[Equation]
with Λ dki : R 3 × n el → R , Ω dk : R 3 → R , α i ∈ {↑ , ↓} , i ∈ { 1 , . . . , n el } , k ∈ { 1 , . . . , n el } .
Thedeterminant enforces antisymmetry, Ω dk are envelope functions enforcing the boundary conditionlim | r |→∞ ψ θ ( r ) = 0 , and Λ dki are neural networks.
The loss can be minimized by gradient basedmethods and the computation of the expectation value in eq.
1 over the high-dimensional space R 3 × n elis done using Monte Carlo integration by sampling electron coordinates r distributed accordingto ψ 2 θ using the Metropolis-Hastings [7] algorithm.
A thorough discussion of deep-learning-basedvariational Monte Carlo (DL-VMC) can be found in [8].
Related work Two major neural network architectures and their extensions have emerged throughout literature: PauliNet [8] and FermiNet [9].
PauliNet puts emphasis on maximizing physical priorknowledge, by focusing on the the envelope function.
They use the output of CASSCF (Complete
Active Space Self Consistent Field, a sophisticated conventional quantum-chemistry method) as Ωand use a relatively small ( ∼ 100k weights) neural network for Λ .
FermiNet on the other hand uses a simple exponential function as envelope Ω and uses a large ( ∼ 700k weights) neural network for Λ .
Both approaches have been applied with great success to many different systems and properties, suchas energies of individual molecules [4, 9, 8], ionization energies [9], potential energy surfaces [10,11], forces [10], excited states [12], model-systems for solids [13, 14] and actual solids [15].
Severalapproaches have been proposed to increase accuracy or decrease computational cost, most notablyarchitecture simplifications [16], alternative antisymmetric layers [17], effective core potentials[18] and Diffusion Monte Carlo (DMC) [19, 20].
FermiNet commonly reaches lower (i.e. moreaccurate) energies than PauliNet[9], but PauliNet has been observed to converge faster [11].
It hasbeen proposed [8] that combining the embedding of FermiNet and the physical prior knowledge of PauliNet could lead to a superior architecture.
Our contribution In this work we present the counter-intuitive observation that the oppositeapproach might be more fruitful.
By combining a PauliNet-like neural network embedding withthe envelopes of FermiNet and adding several improvements to the embedding, input features, andinitialization of parameter (Sec. 2), we obtain the currently best neural network architecture forthe numerical solution of the electronic Schrödinger equation.
Combining our new architecturewith VMC we establish a new benchmark by calculating the most accurate variational ground stateenergies ever published for a number of different atoms and molecules - both when comparing todeep-learning-based methods, as well as when comparing to classical methods (Sec. 3).
Acrosssystems we reduce energy errors by 40-70% and achieve these results with 4-8x fewer optimizationepochs compared to FermiNet.
In Sec.
4 we systematically break down which changes cause theseimprovements.
We hypothesize that including too much physical prior knowledge can actually hinderoptimization and thus deteriorate accuracy – we provide ample experimental evidence in Sec. 5.
Improved approach
Similar to FermiNet, our architecture expresses Λ dki as a linear combination of high-dimensionalelectron embeddings h Li , and the envelopes Ω dα i
k
as a sum of exponential functions
[Equation]
3) where W dα i k , π dα i kI , ω dα i kI are trainable parameters and we enforce ω dα i kI ≥ 0 .
We compute theseembeddings
h
Li by first transforming the inputs
R I , r i into feature vectors
[Equation]
where [ · ] denotes the concatenation operation and then applying L iterations of an embedding network(Fig. 2a).
The local difference vectors ˜ ρ iI are obtained by applying rotation matrices onto ρ iI asdescribed in Sec. 2.2.
[Figure]
2.1 Convolutional layers in embedding
Our embedding network uses four residual neural network streams (Fig. 2b): A primary oneelectron stream that embeds a single electron, and three auxiliary streams modelling the two-particleinteractions (electrons with same spins, electrons with different spin, and electron-ion).
[Equation]
Here l denotes the embedding iteration, A l denote fully connected neural networks, and g lij , k liIdenote electron-electron- and electron-nucleus-embeddings.
We use σ ij = ’same’ for same-spinpairs of electrons and σ ij = ’diff’ for pairs of electrons with different spin.
Similar to FermiNet, ineach iteration we assemble the input f li to the primary stream from the auxiliary streams (Fig. 2c):
[Equation]
Inspired by the success of SchNet [21] and the efficiency of the PauliNet embedding, we use thesum of element-wise multiplication ( ⊙ ), effectively forming a convolution, to aggregate the auxiliarytwo-particle streams:
[Equation]
Eq.
5 and 6 form the core of the architecture and are the key difference between FermiNet, PauliNetand our architecture.
The PauliNet architecture emphasizes two-particle interactions and essentiallyonly uses convolutions as input features: f li =
[ s l, el i , s l, nuc i ] .
In addition to not using the h i as inputfeatures, PauliNet also limits its effective depth by making the convolutional kernels B functionsof the electron-electron distances | r ij | instead of the high-dimensional embedded representationsg ij .
The FermiNet architecture on the other hand emphasizes the one-electron stream and only usessums over g ij as input features, essentially corresponding to B l
= Id, C ( · ) = 1 .
Furthermore
FermiNet does not contain an explicit stream for electron-nucleus interactions.
Since our architectureadequately models both the one-electron-embedding as well as the two-particle-interactions, weexpect our architecture to be more expressive than either predecessor, as demonstrated in Sec. 4.
2.2 Local, invariant input features
The first stage of any VMC wavefunction model is typically the computation of suitable input featuresfrom the raw electron coordinates r and nuclear coordinates { R I } .
While the subsequent embeddingstage could in principle take the raw coordinates, appropriate features allow to explicitly enforcesymmetries and improve the model’s transferability and accuracy.
Input features should have threeproperties: First, they should be sufficiently expressive to encode any physical wavefunction.
Second,the features should be invariant under geometric transformations.
Third, the features should primarilydepend on the local environment of a particle, i.e. Similar local geometries should generate similarlocal features, mostly independent of changes to the geometry far from the particle in question.
Published architectures have so far not been able to address all three points: PauliNet [9] usesonly distances as input features, making them invariant and local, but not sufficiently expressive, asdemonstrated by [11].
FermiNet [9] uses raw distances and differences, making the inputs expressiveand local, but not invariant under rotations.
PESNet [11] proposes a global coordinate system alongthe principle axes of a molecule, making the inputs invariant and sufficiently expressive, but not local.
We propose using local coordinate systems centered on every nucleus and evaluating the electronnuclei differences in these local coordinate systems.
Effectively this amounts to applying a rotationmatrix U J to the raw electron-nucleus differences: ˜ ρ iJ =
U J ρ iJ .
As opposed to [11] where Uis constant for all nuclei, in our approach U J can be different for every nucleus.
These electronnucleus differences ˜ ρ iJ are invariant under rotations, contain all the information contained in raw
Cartesian coordinates and depend primarily on the local environment of an atom.
To compute the3x3 rotation matrices U J , we first run a single Hartree-Fock calculation using a minimal basis set toobtain a density matrix D .
For each atom J we choose the 3x3 block of the density matrix whichcorresponds to the 3 p-orbitals of the atom J and compute its eigenvectors U J = eig ( D J ) . To make
Global coordinates (PESNet)
Local coordinates (this work) our coordinate system unique we sort the eigenvectors of D J by their corresponding eigenvalues.
Ifthe eigenvalues are degenerate, we pick the rotation of the eigenvectors that maximizes the overlapwith the coordinate-axes of the previous nucleus.
Fig.
3 shows the resulting coordinate systemsspanned by U J .
Note that the local coordinate system generally depicts more physically meaningfuldirections such as "along the chain".
We find that these local coordinates slightly increase the accuracyfor single geometries, but more importantly we expect the wavefunction to generalize better acrossdifferent molecules or geometries.
This should improve the accuracy of approaches that attempt tolearn wavefunctions for multiple geometries at once [10, 11].
2.3 Initialization of orbital envelope weights
[Figure]
When considering a single atom, the entries of the wavefunction determinant have essentially theform
[Equation]
In [9], the exponential envelope was purely motivated by the boundary condition that the orbitals mustdecay to zero, and initialization with ω k = 1 was proposed.
However, when comparing this ansatzto analytical solutions, an interesting parallel can be found: Analytical solutions to the Schrödingerequation for atoms with a single electron – the only systems that have analytical solutions – are of theform
[Equation]
where (cid:101) Λ k ( ρ iI ) is a product of a Laguerre polynomial and a spherical harmonic, and n k ∈ N +is known as the principal quantum number.
This suggests ω k ≈ Z / n k , which we also find whenanalyzing the weights of a fully trained wavefunction.
When initializing with ω k = Z / n k instead ofω k
= 1 , we observe faster convergence, lower final energies, and lower variance of the energies (Sec.4).
The effect is most notable for molecules containing nuclei with large Z , where Z / n k ≫ 1 .
2.4 Improved hyperparameters
Beyond the improved neural network architecture we fine-tuned the hyperparameters to reducethe number of optimization steps required for convergence.
Starting from the hyperparametersproposed by [9], we increased the norm constrain by 3x for the second-order optimizer KFAC [22,23], decreased learning rate by 0.5x, and decreased learning rate decay time by 0.4x.
We observe thatthese changes stabilize the optimization and enable usage of 50% fewer Monte Carlo walkers, whichresults in ∼ 2x faster optimization and reduced memory allocation.
A complete set of hyperparameterscan be found in appendix B.
Results of improved approach
We evaluated the accuracy of our approach by comparing our computed energies against the mostaccurate references available in the literature.
Fig.
4 compares our energies against variationalmethods – for which lower energies are guaranteed to be more accurate – as well as non-variationalhigh-accuracy methods.
We find that across many different systems (small and large atoms, moleculesat equilibrium geometry, molecules in transition states), our approach yields substantially lower –and thus more accurate – energies than previous variational results.
Across all tested systems, weoutperform almost all existing variational methods, both deep-learning-based methods as well as classical ones.
When comparing to high-accuracy FermiNet VMC calculations, we not only reachsubstantially lower energies, but also do so using 4-8x fewer training steps, with each step being 25%faster (cf. appendix C).
Comparing to a concurrently published Diffusion Monte Carlo approach,which used ∼ 10x more computational resources, we achieve similar or better accuracy for moleculeslike N 2 and cyclobutadiene and slightly lower accuracy for benzene.
Non-variational methods (e.g.
CCSD(T)) yield slightly lower energies than our calculations for some molecules, but since thosemethods do not provide upper bounds or uncertainty guarantees they do not provide a ground-truth.
For many applications not only absolute energies are important, but energy differences between
[Figure]
[Table]
[Figure]
K
Fe
Glycine
C 6 H
192 different molecules or geometries are of interest, for example to determine the energy required tobreak a chemical bond.
A particularly difficult challenge is the dissociation of the N 2 molecule, i.e.the energy of an N 2 molecule at different bond lengths (inset Fig. 5).
Even methods that are generallyregarded as highly accurate, such as CCSD(T), predict energies that deviate far from experimentaldata at bond-lengths from 2.5 - 4 bohr.
Fig.
5 depicts this deviation between experimental andcomputed energy for our method and the best available reference calculations.
We find that ourresults are closer to the experimental absolute energies than all previous work, and are similar toconcurrently published FermiNet-DMC results which require 5-10x more computational resources.
When comparing relative energies, our approach outperforms all other deep-learning-based methodsand CCSD(T), and is only beaten by the highly specialized r12-MR-ACPF method [26].
Similar toabsolute energies, we also find that our relative energies converge substantially faster than for otherdeep-learning-based methods, with relative energies being almost fully converged after 50k epochs.
Ablation study
To investigate which specific changes lead to the observed improvements in accuracy, we startfrom the improved FermiNet architecture proposed in [16] and incrementally add improvementsin the following order: First, we use dense n el × n el determinants as described in [17] instead ofblock-diagonal determinants.
This increases computational cost and parameter count (cf. appendix C) but has been found to better describe the wavefunctions nodal surface.
Second, we changehyperparameters as described in Sec. 2.4, which increases throughput by ∼ 2x.
Third, we augmentthe electron embedding using our new SchNet-like neural network architecture described in Sec. 2.1.
This leads to a moderate increase in parameter count and computational cost.
Fourth, we switch tolocal, invariant input features as described in Sec. 2.2 and remove the electron-electron differencevectors r ij as inputs.
Lastly we switch to initializing
ω dkI
= Z / n k as described in Sec. 2.3, resulting in [Figure] E E e x p e r
i m e n t / m H a curve
[Figure] [Figure] our proposed final method.
We note that the accuracy gains of these changes are not fully independentof each other and the relative attribution depends on the order in which they are applied: Earlierchanges will generally generate larger energy improvements compared to later changes.
At each stepwe compute total energies for three different molecules: ethene, N 2 at the challenging bond-lengthof 4.0 bohr, and the K-atom.
Fig.
6 depicts the energy of our implementation of FermiNet, and theenergy change caused by each subsequent improvement.
Each experiment was repeated two timeswith different RNG seeds (appendix D), the errorbars depict the spread in energy.
Overall we find thatall our changes combined yield a ∼ 3-20x improvement in the energy error.
For ethene, the dominant
[Figure]
223 contribution (3.7 mHa) comes from improved hyperparameters, which lead to the results being mostlyconverged after 50k epochs vs. the original settings which require 200k epochs for convergence.
Architectural improvements (embedding and input features) lower the energy error by additional 1.4mHa.
For N 2 it has already been observed that a single dense determinant can outperform modelswith multiple block-diagonal determinants [17].
We find the corresponding result that 32 densedeterminants substantially lower the energy relative to an ansatz with 32 block-diagonal determinants.
Comparing N 2 to ethene, we observe larger contributions from our architectural improvements and smaller contributions from improved hyperparameters.
For the K atom, the overall gains are largest,totalling 60mHa, with substantial accuracy gains from all improvements.
Since K has a much largernuclear charge (Z=19) than the constituents of ethene (Z=1,6) and N 2 (Z=7), also the physics-inspiredinitialization of the envelope parameters yields a substantial contribution.
Incorporating prior knowledge
To further understand the effect of incorporating prior knowledge into neural network architecturesfor physical problems as the electronic Schrödinger equation, we examined two distinct ways ofincreasing prior information in our model:
First, by including a built-in approximate physical model,analogous to PauliNet.
Second, by increasing the number of pre-training steps to more closely matcha reference wavefunction before starting the optimization.
Explicitly include CASSCF PauliNet maximizes the physical prior information by computing theenvelopes Ω with CASSCF, a sophisticated conventional method, and explicitly enforcing the Katocusp conditions [28].
Starting from our proposed architecture, we modified our approach step-by-stepuntil we arrived at a PauliNet-like architecture.
Fig.
7a shows the energies of an NH 3 moleculetrained for 50k epochs at each step.
[Figure]
transitioning from our to
Energy error for 3 architectures over number of training epochs.
c) Mean and standard deviation of the energy error as a function of pre-training steps.
Pre-training used CASSCF or HF as a reference, and the energy was evaluated after 20k or 50k subsequent variational optimization epochs.
First, we switch from dense determinants to block-diagonal determinants as used by the original
PauliNet, leading to small loss in accuracy.
Second, we exchange our embedding for the PauliNet-likeembedding using the hyperparameters proposed in [10], leading to a substantial loss in accuracy,presumably caused by a loss in expressiveness.
Next, we replace the simple exponential envelopes bythe more physically inspired CASSCF-envelopes, causing a large loss in accuracy.
We then removethe vector r i − R I as input feature (keeping only its absolute value) as done in the original PauliNetarchitecture [8].
This again deteriorates accuracy, presumably due to enforcing rotational invariancewhich is too restrictive of a symmetry class as pointed out by [11].
Lastly we replace the electronelectron distances | r ij | (which are not smooth at r ij = 0 and thus lead to cusps) by smooth, cusplessradial basis functions as input feature and add an explicit term to enforce the electron-electron cuspcondition.
Since the CASSCF-envelopes are designed to fulfill the electron-ion cusp condition, thischange leads to an enforced cusp condition, slightly improving the energy accuracy.
The largest lossin accuracy throughout these changes is caused by introducing the CASSCF-envelopes, suggestingthat they introduce a strong bias of the wavefunction that cannot be easily overcome during training.
Fig.
7b shows that architectures using exponential envelopes converge to lower absolute energiescompared to the CASSCF-based PauliNet and outperform PauliNet already after ∼ 5000 epochs.
Increase pre-training accuracy Before starting the unsupervised variational optimization of thewavefunction, we run a short supervised pre-training of the wavefunction to roughly match a givenreference wavefunction.
This is computationally inexpensive because it only requires evaluation ofψ θ and a back-propagation step to update the neural network weights θ but not the second derivativeof the Hamiltonian.
If the reference method yields a decent approximation to the true wavefunction,this pre-training significantly speeds-up optimization and avoids unstable parameter regimes [9].
Toincorporate more prior knowledge, one could either use a more sophisticated reference method (e.g.
CASSCF instead of HF) or increase the number of pre-training steps.
In Fig.
7c we pre-trainedthe wavefunction with a block diagonal determinant for the NH 3 molecule using a CASSCF and
Hartree-Fock reference.
We increased pre-training iteration steps and evaluated the energy aftersubsequent 20 k and 50 k variational optimization epochs, each run was repeated with five differentseeds.
Increasing the number of pre-training steps initially increases accuracy – since it provides abetter starting point for the subsequent variational optimization – but when increasing pre-trainingbeyond 20k steps, accuracy deteriorates for both methods.
Surprisingly, we observe a drop in accuracywhen using CASSCF as a reference method compared to the simpler Hartree-Fock method.
Thiseffect is even more pronounced when increasing the number of pre-training steps.
It suggests thatexcessive pre-training introduces a bias that is hard to overcome during variational optimization,similarly to a built-in reference method.
Discussion and Limitations
Discussion We find that our approach yields substantially more accurate absolute energies than allprevious work – both classical as well as deep-learning-based – and that we reach these accuracies4-8x faster than the next best method (FermiNet).
Especially for larger systems, such as 4th row atomsor the amino acid glycine, we outperform conventional "gold-standard" methods like MRCI-F12(Q)by ∼ 100 mHa.
This corroborates the fact that deep-learning-based methods are emerging as anew gold-standard in computational chemistry and showcases the immense potential of machinelearning-based methods in the natural sciences.
A concurrent work [19] was able to achieve similaraccuracies by applying Diffusion Monte Carlo (DMC) on top of a FermiNet VMC calculation,highlighting the potential of deep-learning Monte Carlo methods.
However, [19] required ∼ 10x morecomputational resources and their VMC results – already by themselves 5x more expensive thenour calculations – are consistently inferior to our results.
This showcases a promising route towardsfurther improvements by using our substantially cheaper and more accurate VMC results as a startingpoint for a DMC calculation.
Regarding the question of how much physics to include in the model, we find varying results.
Forexact physical constraints, such as symmetries or the cusp conditions, inclusion in the model generallyappears to be helpful.
However for prior knowledge from existing approximate solutions (such as
CASSCF) the situation is more subtle.
On the one hand, soft physical guidance such as shortsupervised pre-training or physics-inspired weight initialization accelerates optimization.
On theother hand, we show empirically that increasing physical prior knowledge, e.g. by incorporating
CASSCF or extensive supervised pre-training, does not necessarily increase accuracy, but can in factintroduce detrimental biases that are hard to overcome during wavefunction optimization.
Limitations and outlook Despite the proposed improvements and favorable scaling of the method,computation of energies for large molecules still takes days of GPU-time on current hardware.
While the same holds true for conventional high-accuracy approaches, substantial speed-ups arestill required to make DL-VMC more accessible for practitioners.
Additionally, when increasingthe nuclear charges, the wavefunction becomes increasingly localised, which leads to a reductionin average Monte Carlo stepsize and potentially correlated samples.
We circumvent this effect for4th row atoms by increasing the number of intermediate Monte Carlo steps, but further researchinto Monte Carlo sampling methods [29, 30] is required to fully address this issue.
Despite ourimprovements for the accuracy of energy differences between different molecules or geometries, DLVMC is still outperformed by other, computationally cheaper methods in some cases.
Initial researchinto the regularity of the wavefunction across different molecules [10, 11] provides a promisingroute to improvements.
We note in passing that thanks to the local coordinate input features, ourarchitecture fulfills the required rotational invariance required for these approaches.
Checklist
For all authors
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes] We claim 2x higher accuracy, which we demonstrate in
Fig. 4 and Fig.
We claim 8x speed-up which we demonstrate to be a combination ofneeding 4-6x fewer epochs (Sec. 4) and ∼ 40% faster epochs
Sec. 6 in main text anddetailed further in {+Tab. 
+}
in the appendix)
(b) Did you describe the limitations of your work?
[Yes] in Sec. 6 in the paragraph
Limitations
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Whilewe do not foresee any near-term negative impact from this basic research, we discuss
potential impacts in appendix E.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] Our paper conforms to the ethics review guidelines.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] Code,
configuration files and data are in the supplemental material.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] The choice of the most relevant training details is discussed in
Sec. 2.4, the full set of hyperparameters is listed in Tab. 2 in the appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] We depict error bars where possible (e.g. in Fig.
and show in Sec.
D that errors wrt.
the random seed are small.
(d) Did you include the total amount of compute and the type of resources used (e.g.,
type of GPUs, internal cluster, or cloud provider)?
[Yes] We state the used compute resources in C in the appendix.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cite all used
assets, in particular code and reference energies.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We include our source code in the supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Absolute energies
[Table]
Smallmolecules
N 2 , d = 2 .− 109 .
5416 b − 109 .
[Table]
−
[Table]
Largermolecules
[Table]
B Computational settings
Neural network
The main hyperparameters used in this work can be found in Tab. 2 with minorchanges for larger systems.
For the third and fourth row atoms we increased as in [16] the singleelectron stream to 512 .
For K and Fe we choose a higher damping for the second-order optimizer
KFAC with 4 × 10 − 3 to stabilize the optimization.
Additionally, we observed that with increasingnuclear charge Z the MCMC stepsize tends to decrease because of a high probability peak aroundthe core resulting in two distinct length scales for the core and valence electrons.
To circumventhigh correlation between samples we increased the number of decorrelation steps to 60 for Fe.
Weobserved for larger systems that 1000 pretraining steps is not sufficiently close to a Hartree-Focksolution and therefore increased for glycine, benzene, cyclobutadiene, K and Fe the number ofpretraining steps to 4000 .
For glycine and benzene the batch size and number of MCMC walkerswas decreased to 1700 allowing us to use a single NVIDIA A100 GPU and for glyzine, benzeneand cyclobutadiene no envelope initialization at the beginning of optimization was performed.
Allcomputations were computed on either a single NVIDIA A100 or A40 GPU.
As opposed to previouswork [9, 16] – which use an average over the last 20% of optimization steps to compute final energies– we use separate 10 k evaluation steps in which we do not update the neural network parameters tocompute all final energies, to ensure fully unbiased energies.
Reference calculation with MRCI We carried out reference calculations for several smallmolecules (see Table 3) and atoms (see Table 4) using the MOLPRO package [32, 33].
On theone hand, as single-reference method, we tested coupled cluster with single and double excitationsincluding perturbative triple excitations in its explicitly correlated F12 formulation (CCSD(T)-F12)[34], which requires a restricted Hartree-Fock (HF) calculation in the employed implementation.
[Table]
Markov Chain Monte Carlo
[Table]
On the other hand, as multi-reference methods, we used multi-reference configuration interaction inits explicitly correlated F12 formulation (MRCI-F12), which necessitates a complete active spaceself-consistent field (CASSCF) calculation.
We also carried out perturbative Davidson corrections onthe MRCI-F12 values (abbreviated as MRCI-F12(Q)).
Among all of these methods, HF, CASSCF (asused here in its state-specific variant) and MRCI-F12 are variational.
In contrast, CCSD(T)-F12 and
MRCI-F12(Q) contain perturbative contributions to the ground-state energy and, thus, do not obeythe variational principle.
The active spaces for the molecules and atoms is denoted here in the format multiplicity (number ofelectrons, number of orbitals) and was chosen as: H 2 O 1 (8,6), cyclobutadiene 1 (4,4), benzene 1 (6,6),glycine 1 (6,4), K 2 (1,6), Fe 5 (8,6).
For the molecules, the correlation-consistent quadruple-zetabasis set cc-pVQZ-F12 [35] was employed together with the recommnended parameter GEM_BETA= 1.5 a − 1 0 [36] (which lead to convergence problems for the benzene multi-reference calculation).
In order to judge the basis set convergence, also calculations with the smaller triple-zeta basis setcc-pVTZ-F12 was carried out together with GEM_BETA = 1.4 a − 1 0 .
These basis sets were notavailable for the atoms and the def2-QZVPP basis set together with the corresponding density-fittingdef2-QZVPP-JKfit basis were employed instead [37].
[Table]
cc
[Table]
pVQZ-F
[Table]
cc-
[Table] [Table]
Cyclobutadiene cc-pVQZ-F -153. [Table] [Table]
[Table]
[Table]
cc-pVTZ-F -153.
[Table] [Table]
Benzene cc-pVQZ-F -230. – – cc-pVTZ-F -230.
Glycine cc-pVQZ-F -282.
cc-pVTZ-F -282.
[Table]
C Computational cost
[Table] For a better comparison of the computational cost of our proposed architecture to FermiNet wereport in Tab. 5 the runtime per optimization epoch and the number of all trainable parameters forall settings tested in Sec. 4.
The majority of the runtime per epoch is due to the second derivativefor the kinetic energy of the Hamiltonian.
Comparing runtime for the K atom we are ∼ 40 % fasterper epoch then FermiNet with a block determinant structure and even ∼ 60 % faster when using afull determinant.
The option of full determinant comes not free of costs and increases substantiallythe number of total trainable parameters and a slightly more expensive optimization iteration for FermiNet.
This effects is compensated by a faster convergence as has been seen in Fig.
To enablea full like-for-like comparison we compare all runtimes vs. our own implementation of FermiNet,which achieves the same runtime as the original code.
Overall we used approximately 50k GPUh intotal for this work (10k on A100 GPUs, 40k on A40 GPUs).
[Table]
SchNet-like embedding
8 s 3 , 175 , Local input features 3 . 6 s 3 , 175 ,
Envelope initialization (This work
6 s 3 , 175 ,
D Run-to-run variation
Due to the stochastic nature of initialization and Monte Carlo sampling, our results show smallrun-to-run variations, depending on the initialization of the random number generator.
To estimatethe extent of the variation we repeated each experiment in the ablation study two times, and depictedmean and standard deviation of these two runs in Fig.
To further quantify this uncertainty, werepeated the optimization of an N 2 molecule 10 times.
We used two different settings (our proposedapproach and FermiNet using dense-determinants, before applying our improvements) and optimizedfor 50k epochs.
We find a standard deviation of 0.2 mHa for our approach and 0.8 mHa for the
FermiNet approach.
We attribute the lower spread of our approach to the fact that our results arealready converged after 50k epochs (with the remaining uncertainty primarily caused by Monte Carloevaluation uncertainty), while the unconverged FermiNet results depend more strongly on the randominitialization.
E Broader impact
Advancements in computational chemistry may prompt new discoveries in chemistry and biology,which could include positive outcomes such as the development of new drugs or materials.
Like everycomputational chemistry method, our work could hypothetically also be misused for the developmentof chemical weapons or other potential risks to humanity.
As of now, this seems highly unlike due tobasic nature of our research.