Original: Spherical Channels for Modeling Atomic Interactions
Revised: Spherical Channels for Modeling Atomic Interactions

Original: Abstract
Revised: Abstract

Original: Modeling the energy and forces of atomic systems is a fundamental problem in computational chemistry with the potential to help address many of the world’s most pressing problems, including those related to energy scarcity and climate change.
Revised: Modeling the energy and forces of atomic systems is a fundamental problem in computational chemistry with the potential to help address many of the world’s most pressing problems, including those related to energy scarcity and climate change.

Original: These calculations are traditionally performed using Density Functional Theory, which is computationally very expensive.
Revised: These calculations are traditionally performed using Density Functional Theory, which is computationally very expensive.

Original: Machine learning has the potential to dramatically improve the efficiency of these calculations from days or hours to seconds.
Revised: Machine learning has the potential to dramatically improve the efficiency of these calculations from days or hours to seconds.

Original: We propose the Spherical Channel Network (SCN) to model atomic energies and forces.
Revised: We propose the Spherical Channel Network (SCN) to model atomic energies and forces.

Original: The SCN is a graph neural network where nodes represent atoms and edges their neighboring atoms.
Revised: The SCN is a graph neural network where nodes represent atoms and edges their neighboring atoms.

Original: The atom embeddings are a set of spherical functions, called spherical channels, represented using spherical harmonics.
Revised: The atom embeddings are a set of spherical functions, called spherical channels, represented using spherical harmonics.

Original: We demonstrate, that by rotating the embeddings based on the 3D edge orientation, more information may be utilized while maintaining the rotational equivariance of the messages.
Revised: We demonstrate, that by rotating the embeddings based on the 3D edge orientation, more information may be utilized while maintaining the rotational equivariance of the messages.

Original: While equivariance is a desirable property, we find that by relaxing this constraint in both message passing and aggregation, improved accuracy may be achieved.
Revised: While equivariance is a desirable property, we find that by relaxing this constraint in both message passing and aggregation, improved accuracy may be achieved.

Original: We demonstrate state-of-the-art results on the large-scale Open Catalyst 2020 dataset in both energy and force prediction for numerous tasks and metrics.
Revised: We demonstrate state-of-the-art results on the large-scale Open Catalyst 2020 dataset in both energy and force prediction for numerous tasks and metrics.

Original:  Introduction
Revised:  Introduction

Original: Modeling the properties of atomic systems is a foundational challenge in computational chemistry and critical to advancing technologies across numerous application domains.
Revised: Modeling the properties of atomic systems is a foundational challenge in computational chemistry and critical to advancing technologies across numerous application domains.

Original: Notable applications include drug discovery [32, 39] and the design of new catalysts for renewable energy storage to help in addressing climate change [48, 33].
Revised: Notable applications include drug discovery [32, 39] and the design of new catalysts for renewable energy storage to help in addressing climate change [48, 33].

Original: For catalyst discovery, new materials are currently evaluated using Density Functional Theory (DFT) that can estimate atomic energies and forces but is computationally very expensive; taking hours or days to evaluate a single material.
Revised: For catalyst discovery, new materials are currently evaluated using Density Functional Theory (DFT) that can estimate atomic energies and forces but is computationally very expensive; taking hours or days to evaluate a single material.

Original: Machine Learning (ML) has the potential to approximate DFT and dramatically speed up these calculations, allowing for high throughput screening of new materials to help address some of the world’s most pressing challenges.
Revised: Machine Learning (ML) has the potential to approximate DFT and dramatically speed up these calculations, allowing for high throughput screening of new materials to help address some of the world’s most pressing challenges.

Original: Our goal is to approximate DFT calculations using ML.
Revised: Our goal is to approximate DFT calculations using ML.

Original: An ML model takes as input a set of atom positions and their atomic numbers.
Revised: An ML model takes as input a set of atom positions and their atomic numbers.

Original: As outputs, a model calculates the structure’s energy (or other properties) and the per-atom forces, i.e., the forces exerted on each atom by the other atoms.
Revised: As outputs, a model calculates the structure’s energy (or other properties) and the per-atom forces, i.e., the forces exerted on each atom by the other atoms.

Original: A common approach to this problem is to use Graph Neural Networks (GNNs) [19, 47] where each node represents an atom and the set of nearby atoms as edges [37, 17, 24, 35, 38, 45, 31, 13].
Revised: A common approach to this problem is to use Graph Neural Networks (GNNs) [19, 47] where each node represents an atom and the set of nearby atoms as edges [37, 17, 24, 35, 38, 45, 31, 13].

Original: A key challenge in network design is balancing the use of model constraints.
Revised: A key challenge in network design is balancing the use of model constraints.

Original: When modeling atomic systems, a common constraint is SO(3) rotation equivariance [43, 3, 1, 42, 34, 36], i.e ., if the atomic system is rotated, the energies should remain constant and the atomic forces should similarly rotate.
Revised: When modeling atomic systems, a common constraint is SO(3) rotation equivariance [43, 3, 1, 42, 34, 36], i.e ., if the atomic system is rotated, the energies should remain constant and the atomic forces should similarly rotate.

Original: While this provides strong priors on the model to help in generalization, especially for smaller datasets[7, 32], it can result in limiting the expressiveness of the network due to restrictions on
Revised: While this provides strong priors on the model to help in generalization, especially for smaller datasets[7, 32], it can result in limiting the expressiveness of the network due to restrictions on

Original: [Figure]
Revised: [Figure]

Original: non-linear transformations for equivariant models [3, 42, 34, 36], or limiting interactions to pairs [35, 38, 45], triplets [13] or quadruplets of atoms [28, 14, 15] for invariant models.
Revised: non-linear transformations for equivariant models [3, 42, 34, 36], or limiting interactions to pairs [35, 38, 45], triplets [13] or quadruplets of atoms [28, 14, 15] for invariant models.

Original: Alternatively, a non-equivariant model [23] can provide more freedom to the model, but lead to the model needing to learn approximate equivariance through methods such as data augmentation [40].
Revised: Alternatively, a non-equivariant model [23] can provide more freedom to the model, but lead to the model needing to learn approximate equivariance through methods such as data augmentation [40].

Original: To draw an analogy with image detection, the use of a CNN [27] provides translation equivariance and removes the need for the network to learn how to detect the same object at different locations.
Revised: To draw an analogy with image detection, the use of a CNN [27] provides translation equivariance and removes the need for the network to learn how to detect the same object at different locations.

Original: However, most
Revised: However, most

Original: CNNs are not equivariant to scale or rotation [44, 43], but are still effective in learning approximate equivariance through rotation and scale diversity in the training data.
Revised: CNNs are not equivariant to scale or rotation [44, 43], but are still effective in learning approximate equivariance through rotation and scale diversity in the training data.

Original: What is the analogous balance of constraints for modeling atomic systems?
Revised: What is the analogous balance of constraints for modeling atomic systems?

Original: In this paper, we propose a GNN [19, 47] that balances the use of model constraints to aid in generalization while providing the network with the flexibility to learn accurate representations.
Revised: In this paper, we propose a GNN [19, 47] that balances the use of model constraints to aid in generalization while providing the network with the flexibility to learn accurate representations.

Original: We introduce the Spherical Channel Network (SCN) that explicitly models relative orientations of all neighboring atoms; information that is critical to accurately predicting atomic properties.
Revised: We introduce the Spherical Channel Network (SCN) that explicitly models relative orientations of all neighboring atoms; information that is critical to accurately predicting atomic properties.

Original: Each node’s embedding is a set of functions defined on the surface of a sphere ( S 2 → R ) .
Revised: Each node’s embedding is a set of functions defined on the surface of a sphere ( S 2 → R ) .

Original: The functions are represented using spherical harmonics, similar to approaches that build strictly equivariant models [42, 1, 3, 10].
Revised: The functions are represented using spherical harmonics, similar to approaches that build strictly equivariant models [42, 1, 3, 10].

Original: During message passing, angular information between atoms is conveyed by rotating or steering [5] the embeddings based on each edge’s orientation [41, 28].
Revised: During message passing, angular information between atoms is conveyed by rotating or steering [5] the embeddings based on each edge’s orientation [41, 28].

Original: We identify an expanded set of spherical harmonic coefficients that are invariant to rotation, which can provide rich information while maintaining a message’s rotation equivariance.
Revised: We identify an expanded set of spherical harmonic coefficients that are invariant to rotation, which can provide rich information while maintaining a message’s rotation equivariance.

Original: In addition, we demonstrate that if the equivariance constraint is relaxed, improved performance can be achieved by using additional coefficients.
Revised: In addition, we demonstrate that if the equivariance constraint is relaxed, improved performance can be achieved by using additional coefficients.

Original: We further improve the expressivity of the network by performing a pointwise non-linear function, which is only approximately equivariant, on the embeddings during message aggregation.
Revised: We further improve the expressivity of the network by performing a pointwise non-linear function, which is only approximately equivariant, on the embeddings during message aggregation.

Original: We demonstrate our Spherical Channel Network on the large-scale Open Catalyst 2020 (OC20) dataset [6], which contains atomic structures useful for numerous applications important to addressing climate change.
Revised: We demonstrate our Spherical Channel Network on the large-scale Open Catalyst 2020 (OC20) dataset [6], which contains atomic structures useful for numerous applications important to addressing climate change.

Original: State-of-the-art results are achieved for atomic force and initial structure to relaxed energy prediction with improvements of 8% − 11% .
Revised: State-of-the-art results are achieved for atomic force and initial structure to relaxed energy prediction with improvements of 8% − 11% .

Original: In addition, we demonstrate our model is more sample efficient as compared to other state-of-the-art models.
Revised: In addition, we demonstrate our model is more sample efficient as compared to other state-of-the-art models.

Original:  Approach
Revised:  Approach

Original: Given an atomic structure with n atoms, our goal is to predict the structure’s energy E and the per-atom forces f i for each atom i ∈ n .
Revised: Given an atomic structure with n atoms, our goal is to predict the structure’s energy E and the per-atom forces f i for each atom i ∈ n .

Original: These values are estimated using a Graph Neural Network (GNN) [19, 47] where each node represents an atom and edges represent nearby atoms.
Revised: These values are estimated using a Graph Neural Network (GNN) [19, 47] where each node represents an atom and edges represent nearby atoms.

Original: As input, the network is given the distance d ij between atoms i and j , and each atom’s atomic number a i .
Revised: As input, the network is given the distance d ij between atoms i and j , and each atom’s atomic number a i .

Original: The neighbors N i for an atom i are determined using a fixed distance threshold, or by picking a fixed number of closest atoms.
Revised: The neighbors N i for an atom i are determined using a fixed distance threshold, or by picking a fixed number of closest atoms.

Original: [Figure]
Revised: [Figure]

Original: 2.1 Node Embeddings
Revised: 2.1 Node Embeddings

Original: The angular or relative orientation information between atoms is critical to accurately modeling atomic properties such as energies and forces [13, 14].
Revised: The angular or relative orientation information between atoms is critical to accurately modeling atomic properties such as energies and forces [13, 14].

Original: Inspired by this, our node embedding models the angular information from all neighboring atoms using spherical functions.
Revised: Inspired by this, our node embedding models the angular information from all neighboring atoms using spherical functions.

Original: Each node i ’s embedding s
Revised: Each node i ’s embedding s

Original: i is a set of C functions or channels represented on a sphere ( S 2 → R ) , whose argument is a 3D unit vector indicating the orientation.
Revised: i is a set of C functions or channels represented on a sphere ( S 2 → R ) , whose argument is a 3D unit vector indicating the orientation.

Original: That is, s ic (ˆ r ) is the value of channel c for node i for some orientation ˆ r ∈ R 3 .
Revised: That is, s ic (ˆ r ) is the value of channel c for node i for some orientation ˆ r ∈ R 3 .

Original: Since the spherical channels contain orientation information over the entire sphere, the network may reason about geometric information for all neighboring atoms and not just atom pairs, triplets, etc.
Revised: Since the spherical channels contain orientation information over the entire sphere, the network may reason about geometric information for all neighboring atoms and not just atom pairs, triplets, etc.

Original: Each spherical channel c may be represented using several different approaches, such as discrete 2D grids sampled over the sphere, or spherical harmonics.
Revised: Each spherical channel c may be represented using several different approaches, such as discrete 2D grids sampled over the sphere, or spherical harmonics.

Original: As we describe later, we use spherical harmonics due to their property of being SO(3)-equivariant (3D rotation equivariant).
Revised: As we describe later, we use spherical harmonics due to their property of being SO(3)-equivariant (3D rotation equivariant).

Original: Using real spherical harmonics, a function s c (ˆ r ) defined on the sphere is represented by a set of weighted spherical harmonics Y lm with s c (ˆ r ) = (cid:80) l,m x lmc Y lm (ˆ r ) where l and m are the degree and order of the basis functions with m ∈ [ − l, l ] and l ∈ L .
Revised: Using real spherical harmonics, a function s c (ˆ r ) defined on the sphere is represented by a set of weighted spherical harmonics Y lm with s c (ˆ r ) = (cid:80) l,m x lmc Y lm (ˆ r ) where l and m are the degree and order of the basis functions with m ∈ [ − l, l ] and l ∈ L .

Original: We refer to the spherical harmonic coefficients, x lmc , as the coefficients of the spherical channel.
Revised: We refer to the spherical harmonic coefficients, x lmc , as the coefficients of the spherical channel.

Original: For every degree l there exists 2 l +spherical harmonics (see Figure 1(a) for an illustration of functions for l ≤ 4 ), which results in a total of ( L + 1) 2 basis functions and coefficients up to degree L .
Revised: For every degree l there exists 2 l +spherical harmonics (see Figure 1(a) for an illustration of functions for l ≤ 4 ), which results in a total of ( L + 1) 2 basis functions and coefficients up to degree L .

Original: Therefore for a maximum degree of L with C channels, x has size ( L + 1) 2 × C .
Revised: Therefore for a maximum degree of L with C channels, x has size ( L + 1) 2 × C .

Original: The spherical channels are updated by the GNN through message passing for K layers to obtain the final node embeddings S ( K ) .
Revised: The spherical channels are updated by the GNN through message passing for K layers to obtain the final node embeddings S ( K ) .

Original: S (0) is initialized from an embedding based on the atom’s atomic number a i for l = 0 coefficients and the l ̸ = 0 coefficients are set to zero.
Revised: S (0) is initialized from an embedding based on the atom’s atomic number a i for l = 0 coefficients and the l ̸ = 0 coefficients are set to zero.

Original: The nodes’ embeddings are updated by first calculating a set of messages m ij for each edge, which are then aggregated at each node.
Revised: The nodes’ embeddings are updated by first calculating a set of messages m ij for each edge, which are then aggregated at each node.

Original: Finally, the energy and forces are estimated from S ( K ) .
Revised: Finally, the energy and forces are estimated from S ( K ) .

Original: We describe each of these steps in turn.
Revised: We describe each of these steps in turn.

Original: 2.2 Message Passing
Revised: 2.2 Message Passing

Original: Given a target node t and its neighbors s ∈ N t we want to update the embeddings x ( k ) t at iteration k ∈ K .
Revised: Given a target node t and its neighbors s ∈ N t we want to update the embeddings x ( k ) t at iteration k ∈ K .

Original: The embeddings x ( k ) t are a set of spherical harmonic coefficients indexed by their degree l , order m , and channel c .
Revised: The embeddings x ( k ) t are a set of spherical harmonic coefficients indexed by their degree l , order m , and channel c .

Original: An important and useful property of spherical harmonics is the function represented by the coefficients is steerable [11, 5], i.e ., it can be rotated in 3D space using a linear transformation of the coefficients.
Revised: An important and useful property of spherical harmonics is the function represented by the coefficients is steerable [11, 5], i.e ., it can be rotated in 3D space using a linear transformation of the coefficients.

Original: Specifically, for a 3D rotation matrix R there exists a matrix called a Wigner D-matrix D l of size (2 l + 1 × 2 l + 1) that rotates the coefficients of degree l by the rotation
Revised: Specifically, for a 3D rotation matrix R there exists a matrix called a Wigner D-matrix D l of size (2 l + 1 × 2 l + 1) that rotates the coefficients of degree l by the rotation

Original: R .
Revised: R .

Original: If x ilc are all coefficients across orders m for node i of degree l and channel c , then for any 3D rotation R there exists a D l that for all orientations ˆ r :
Revised: If x ilc are all coefficients across orders m for node i of degree l and channel c , then for any 3D rotation R there exists a D l that for all orientations ˆ r :

Original: [Equation]
Revised: [Equation]

Original: When calculating the message m st from atom s to atom t , we want to use the information contained in both x ( k ) s and x ( k ) t given the context of the edge’s orientation ˆ d st , ˆ d st = d st / | d st | .
Revised: When calculating the message m st from atom s to atom t , we want to use the information contained in both x ( k ) s and x ( k ) t given the context of the edge’s orientation ˆ d st , ˆ d st = d st / | d st | .

Original: We do this by rotating the embeddings by R st for which R st ˆ d st = [0 , 0 , 1] ⊤ , i.e ., the direction of the edge’s unit vector ˆ d st is aligned with the z-axis.
Revised: We do this by rotating the embeddings by R st for which R st ˆ d st = [0 , 0 , 1] ⊤ , i.e ., the direction of the edge’s unit vector ˆ d st is aligned with the z-axis.

Original: Thus the orientation of the atoms with respect to each other is implicit in the rotated embeddings.
Revised: Thus the orientation of the atoms with respect to each other is implicit in the rotated embeddings.

Original: This simplifies the task of the network since it only needs to learn the relationship between atoms that are aligned along the z-axis and not an arbitrary rotation.
Revised: This simplifies the task of the network since it only needs to learn the relationship between atoms that are aligned along the z-axis and not an arbitrary rotation.

Original: After calculating the messages using a neural network F e , they are rotated back to the global coordinate frame:
Revised: After calculating the messages using a neural network F e , they are rotated back to the global coordinate frame:

Original: [Equation]
Revised: [Equation]

Original: where the matrices D st and D − 1 st perform the rotation on the embeddings’ coefficients corresponding to the 3D rotation matrices R st and R − 1 st respectively.
Revised: where the matrices D st and D − 1 st perform the rotation on the embeddings’ coefficients corresponding to the 3D rotation matrices R st and R − 1 st respectively.

Original: To simplify notation, we assume the coefficients in x s across degree l and order m are flattened, and D st ∈ R ( L +1) 2 × R ( L +1) 2 is a block diagonal matrix constructed from the set of Wigner D-matrices across degrees, l ∈ L .
Revised: To simplify notation, we assume the coefficients in x s across degree l and order m are flattened, and D st ∈ R ( L +1) 2 × R ( L +1) 2 is a block diagonal matrix constructed from the set of Wigner D-matrices across degrees, l ∈ L .

Original: D − 1 st is the inverse of D st , with D − 1 st = D ⊤ st .
Revised: D − 1 st is the inverse of D st , with D − 1 st = D ⊤ st .

Original: In addition to the rotated embeddings, F e is also provided input edge information that is invariant to rotations; | d st | is the magnitude of the distance between atoms s and t , and a s and a t their respective atomic numbers.
Revised: In addition to the rotated embeddings, F e is also provided input edge information that is invariant to rotations; | d st | is the magnitude of the distance between atoms s and t , and a s and a t their respective atomic numbers.

Original: Note that the rotation matrix R st is not unique, since the roll rotation around the vector d st is not specified and is randomly sampled during training.
Revised: Note that the rotation matrix R st is not unique, since the roll rotation around the vector d st is not specified and is randomly sampled during training.

Original: The implications of this are discussed below in Section 2.2.1.
Revised: The implications of this are discussed below in Section 2.2.1.

Original: The message function F e is computed using a neural network as illustrated in Figure 2(a).
Revised: The message function F e is computed using a neural network as illustrated in Figure 2(a).

Original: The atomic numbers a s and a t are used to look up two independent embeddings of size E = 128 , and a set of 1D basis functions are used to represent | d st | using equally spaced Gaussians every 0.02 Å from 0 to the 8 Å with σ = 0 .
Revised: The atomic numbers a s and a t are used to look up two independent embeddings of size E = 128 , and a set of 1D basis functions are used to represent | d st | using equally spaced Gaussians every 0.02 Å from 0 to the 8 Å with σ = 0 .

Original: 04 followed by a linear layer with E outputs.
Revised: 04 followed by a linear layer with E outputs.

Original: Their values are added together and passed through a neural network to produce a vector of size H .
Revised: Their values are added together and passed through a neural network to produce a vector of size H .

Original: The rotated embeddings D st x ( k ) s and
Revised: The rotated embeddings D st x ( k ) s and

Original: D st x ( k ) t are concatenated and passed through a single layer neural network to produce another vector of size H .
Revised: D st x ( k ) t are concatenated and passed through a single layer neural network to produce another vector of size H .

Original: These two are multiplied together to combine the edge information with the information contained in the rotated embeddings.
Revised: These two are multiplied together to combine the edge information with the information contained in the rotated embeddings.

Original: Two more fully connected layers are performed with each followed by a SiLU non-linear activation function [22].
Revised: Two more fully connected layers are performed with each followed by a SiLU non-linear activation function [22].

Original: Finally, a single linear layer is used to expand the output to the original size of the embeddings.
Revised: Finally, a single linear layer is used to expand the output to the original size of the embeddings.

Original: 2.2.1 Message Equivariance
Revised: 2.2.1 Message Equivariance

Original: For the network to be equivariant to rotations, our message function (Equation (2)) needs to be equivariant.
Revised: For the network to be equivariant to rotations, our message function (Equation (2)) needs to be equivariant.

Original: In general, this is not the case since the matrix D st is not unique, i.e ., the roll rotation around the vector d
Revised: In general, this is not the case since the matrix D st is not unique, i.e ., the roll rotation around the vector d

Original: st
Revised: st

Original: is not specified, and in practice it is randomly chosen.
Revised: is not specified, and in practice it is randomly chosen.

Original: The roll rotation corresponds to a rotation about the z-axis after rotating the coefficients x ( k ) s and x ( k ) t by D st .
Revised: The roll rotation corresponds to a rotation about the z-axis after rotating the coefficients x ( k ) s and x ( k ) t by D st .

Original: Due to this, all of the rotated coefficients will vary based on the random roll rotation chosen, except the m = 0 coefficients that are symmetric about the z-axis; see Appendix D and the m = 0 bases highlighted in green in Figure 1(a).
Revised: Due to this, all of the rotated coefficients will vary based on the random roll rotation chosen, except the m = 0 coefficients that are symmetric about the z-axis; see Appendix D and the m = 0 bases highlighted in green in Figure 1(a).

Original: If rotation equivariance is desired for messages, the input and output coefficients to F e can be restricted to only those for which m = 0 .
Revised: If rotation equivariance is desired for messages, the input and output coefficients to F e can be restricted to only those for which m = 0 .

Original: Since the other inputs | d st | , a s and a t to F e are also invariant to rotations, F e is invariant to rotations if only m = 0 coefficients are used.
Revised: Since the other inputs | d st | , a s and a t to F e are also invariant to rotations, F e is invariant to rotations if only m = 0 coefficients are used.

Original: The resulting messages m st are equivariant to rotations once they are rotated back to the global coordinate frame using D − 1 st .
Revised: The resulting messages m st are equivariant to rotations once they are rotated back to the global coordinate frame using D − 1 st .

Original: While equivariance is a desirable property, as we demonstrate later, using m ∈ [ − 1 , 1] coefficients can help improve performance even though equivariance is not strictly enforced.
Revised: While equivariance is a desirable property, as we demonstrate later, using m ∈ [ − 1 , 1] coefficients can help improve performance even though equivariance is not strictly enforced.

Original: If we choose to use coefficients beyond just m = 0 for F e for message passing in Equation (2), two approaches may be taken.
Revised: If we choose to use coefficients beyond just m = 0 for F e for message passing in Equation (2), two approaches may be taken.

Original: The first is to simply add the coefficients to the inputs and outputs of
Revised: The first is to simply add the coefficients to the inputs and outputs of

Original: F e and assume the neural network will learn a roughly equivariant mapping through the random sampling of the roll rotation.
Revised: F e and assume the neural network will learn a roughly equivariant mapping through the random sampling of the roll rotation.

Original: As we demonstrate in Appendix A, the learned functions are indeed roughly equivariant.
Revised: As we demonstrate in Appendix A, the learned functions are indeed roughly equivariant.

Original: The second strategy takes a more direct approach to encouraging the network to learn equivariant mappings by taking advantage of the proprieties of the coefficients as they’re rotated by ϕ about the z-axis.
Revised: The second strategy takes a more direct approach to encouraging the network to learn equivariant mappings by taking advantage of the proprieties of the coefficients as they’re rotated by ϕ about the z-axis.

Original: If D ϕ rotates a set x of coefficients about the z-axis by ϕ , the m = coefficients are constant as a function of ϕ , while the m ∈ {− 1 , 1 } coefficients are sine and cosine functions of ϕ (see Appendix D):
Revised: If D ϕ rotates a set x of coefficients about the z-axis by ϕ , the m = coefficients are constant as a function of ϕ , while the m ∈ {− 1 , 1 } coefficients are sine and cosine functions of ϕ (see Appendix D):

Original: [Equation]
Revised: [Equation]

Original: for some set of vectors α , β and γ of size L .
Revised: for some set of vectors α , β and γ of size L .

Original: x (0) are the m = 0 coefficients and similarly for x ( − 1) and x (1) for the m = − 1 and m = 1 coefficients respectively.
Revised: x (0) are the m = 0 coefficients and similarly for x ( − 1) and x (1) for the m = − 1 and m = 1 coefficients respectively.

Original: Similar proprieties hold for m >and m < − 1 .
Revised: Similar proprieties hold for m >and m < − 1 .

Original: We take advantage of these properties to encourage the output of the message block towards equivariance by computing F e at multiple rotations ϕ about the z-axis:
Revised: We take advantage of these properties to encourage the output of the message block towards equivariance by computing F e at multiple rotations ϕ about the z-axis:

Original: [Equation]
Revised: [Equation]

Original: For m ∈ [ − 1 , 1] , we compute four samples or "taps" at ϕ ∈ { 0 , 12 π, π, 32 π } , and combine them based on m using:
Revised: For m ∈ [ − 1 , 1] , we compute four samples or "taps" at ϕ ∈ { 0 , 12 π, π, 32 π } , and combine them based on m using:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where F ϕe (0) , F ϕe ( − 1) and F ϕe (1) are the output coefficients for m = 0 , m = − 1 and m = 1 respectively.
Revised: where F ϕe (0) , F ϕe ( − 1) and F ϕe (1) are the output coefficients for m = 0 , m = − 1 and m = 1 respectively.

Original: The top line of Equation 7 is simply taking the average, since the m = 0 coefficients should be constant regardless of ϕ .
Revised: The top line of Equation 7 is simply taking the average, since the m = 0 coefficients should be constant regardless of ϕ .

Original: Similarly for m ∈ {− 1 , 1 } , the second and third lines average 4 values that should be equal if Equations 4 and 5 hold, e.g., α sin( ϕ + β ) = − α cos( ϕ + β + π/ 2) .
Revised: Similarly for m ∈ {− 1 , 1 } , the second and third lines average 4 values that should be equal if Equations 4 and 5 hold, e.g., α sin( ϕ + β ) = − α cos( ϕ + β + π/ 2) .

Original: If higher m are desired, similar calculations can be performed.
Revised: If higher m are desired, similar calculations can be performed.

Original: However, a larger number of taps will be needed, e.g ., m ∈ [ − 2 , 2] requires 8 taps separated by 14 π radians.
Revised: However, a larger number of taps will be needed, e.g ., m ∈ [ − 2 , 2] requires 8 taps separated by 14 π radians.

Original: The number of coefficients used during message passing when calculating F e may be varied based on the edge properties to reduce the memory required by the network.
Revised: The number of coefficients used during message passing when calculating F e may be varied based on the edge properties to reduce the memory required by the network.

Original: For instance, atoms that are far away from each other may not need the same resolution of the spherical functions as those close together, so a lower maximum degree L may be used.
Revised: For instance, atoms that are far away from each other may not need the same resolution of the spherical functions as those close together, so a lower maximum degree L may be used.

Original: By training several message passing networks with varying degrees for different ranges of edge distances, memory usage can be reduced while not impacting accuracy.
Revised: By training several message passing networks with varying degrees for different ranges of edge distances, memory usage can be reduced while not impacting accuracy.

Original: Similarly, the range of m can typically be truncated at [ − 1 , 1] or [ − 2 , 2] since their observed utility to the network is substantially reduced for values of m greater than 2 (or less than -2).
Revised: Similarly, the range of m can typically be truncated at [ − 1 , 1] or [ − 2 , 2] since their observed utility to the network is substantially reduced for values of m greater than 2 (or less than -2).

Original: See Figure 1(b) for examples of the spherical channels when the range of m is reduced.
Revised: See Figure 1(b) for examples of the spherical channels when the range of m is reduced.

Original: Note that while the number of coefficients used by F e may be reduced, the node embeddings maintain their original size based on the maximum L , so they may be able to aggregate information from all neighboring atoms at different distances and angles.
Revised: Note that while the number of coefficients used by F e may be reduced, the node embeddings maintain their original size based on the maximum L , so they may be able to aggregate information from all neighboring atoms at different distances and angles.

Original: 2.3 Message Aggregation
Revised: 2.3 Message Aggregation

Original: At this point in the network, the per edge messages are computed and rotated back to the global coordinate frame.
Revised: At this point in the network, the per edge messages are computed and rotated back to the global coordinate frame.

Original: Ideally, to allow for complex interactions between the messages, a non-linear function would be applied after summing all messages directed at a target node.
Revised: Ideally, to allow for complex interactions between the messages, a non-linear function would be applied after summing all messages directed at a target node.

Original: If we did this by naively passing the summed coefficients m ( k +1) t
Revised: If we did this by naively passing the summed coefficients m ( k +1) t

Original: [Equation]
Revised: [Equation]

Original: through a fully connected neural network, the network may have difficulties learning representations that are approximately rotation equivariant (a fundamental property of atom forces).
Revised: through a fully connected neural network, the network may have difficulties learning representations that are approximately rotation equivariant (a fundamental property of atom forces).

Original: Another option is to place constraints on the non-linearities performed to enforce equivariance [26, 5, 36, 42, 3, 2].
Revised: Another option is to place constraints on the non-linearities performed to enforce equivariance [26, 5, 36, 42, 3, 2].

Original: We propose applying an unconstrained non-linear pointwise function on the sphere, i.e., a function that is applied at every orientation without knowledge of the orientation from which the point was sampled [9].
Revised: We propose applying an unconstrained non-linear pointwise function on the sphere, i.e., a function that is applied at every orientation without knowledge of the orientation from which the point was sampled [9].

Original: For our function, we use a non-linear neural network F c ( R C → R C ) that combines information across channels.
Revised: For our function, we use a non-linear neural network F c ( R C → R C ) that combines information across channels.

Original: In practice, F c is applied at a discrete number of orientations, which and are converted from a spherical harmonic representation to a spherical grid representation using G .
Revised: In practice, F c is applied at a discrete number of orientations, which and are converted from a spherical harmonic representation to a spherical grid representation using G .

Original: The channels are concatenated and passed through a 3 layer 1 × 1 CNN.
Revised: The channels are concatenated and passed through a 3 layer 1 × 1 CNN.

Original: Each layer is followed by a SiLU activation.
Revised: Each layer is followed by a SiLU activation.

Original: Finally, the channels are converted back to a spherical harmonic representation using G − 1 and added to x ( k ) t to create x ( k +1) t .
Revised: Finally, the channels are converted back to a spherical harmonic representation using G − 1 and added to x ( k ) t to create x ( k +1) t .

Original: results in a transformation that we numerically demonstrate (see Appendix A) is approximately equivariant to rotations:
Revised: results in a transformation that we numerically demonstrate (see Appendix A) is approximately equivariant to rotations:

Original: [Equation]
Revised: [Equation]

Original: where G is a function that converts a spherical function represented by spherical harmonic coefficients, to one represented by point samples on a sphere.
Revised: where G is a function that converts a spherical function represented by spherical harmonic coefficients, to one represented by point samples on a sphere.

Original: For G we use a spherical coordinate system (polar and azimuthal) to generate a 2D discrete representation of the spherical functions, see Figure 3.
Revised: For G we use a spherical coordinate system (polar and azimuthal) to generate a 2D discrete representation of the spherical functions, see Figure 3.

Original: A 1 × 1 convolutional neural network F c is applied to each discrete sample containing C channels, and the result is then converted back to the spherical harmonic representation using G − 1 .
Revised: A 1 × 1 convolutional neural network F c is applied to each discrete sample containing C channels, and the result is then converted back to the spherical harmonic representation using G − 1 .

Original: This is analogous to having a 2D image represented in the frequency domain, converting it to the spatial domain using an inverse discrete Fourier transform, applying a transformation and converting back to the frequency domain.
Revised: This is analogous to having a 2D image represented in the frequency domain, converting it to the spatial domain using an inverse discrete Fourier transform, applying a transformation and converting back to the frequency domain.

Original: Since the same operation is applied to all orientations on the sphere, a transformation that is approximately equivariant (up to discrete sampling limitations) to rotations may be learned, see Appendix A for a more detailed discussion.
Revised: Since the same operation is applied to all orientations on the sphere, a transformation that is approximately equivariant (up to discrete sampling limitations) to rotations may be learned, see Appendix A for a more detailed discussion.

Original: A diagram of the network is shown in Figure 3.
Revised: A diagram of the network is shown in Figure 3.

Original: To provide additional information to the network, x ( k ) t is also converted to a spherical grid representation, concatenated with G ( m ( k +1) t ) and provided as input to the neural network to compute the final updates to the coefficients x ( k +1) t at iteration k +1 .
Revised: To provide additional information to the network, x ( k ) t is also converted to a spherical grid representation, concatenated with G ( m ( k +1) t ) and provided as input to the neural network to compute the final updates to the coefficients x ( k +1) t at iteration k +1 .

Original: The network can learn circular functions on the sphere if varying resolutions of x ( k ) are provided to the network, similar to Difference of Gaussian filters for 2D images [30].
Revised: The network can learn circular functions on the sphere if varying resolutions of x ( k ) are provided to the network, similar to Difference of Gaussian filters for 2D images [30].

Original: The resolution of x ( k ) can be reduced by lowering the degree L before transforming by G .
Revised: The resolution of x ( k ) can be reduced by lowering the degree L before transforming by G .

Original: In our experiments, if multiple resolutions or bands are used, we use resolutions of degree L and L − 1 .
Revised: In our experiments, if multiple resolutions or bands are used, we use resolutions of degree L and L − 1 .

Original: Three layers of 1 ×convolutions on 2 C channels (or 4 C if two bands are used) are performed with each followed by a
Revised: Three layers of 1 ×convolutions on 2 C channels (or 4 C if two bands are used) are performed with each followed by a

Original: SiLU non-linear activation function.
Revised: SiLU non-linear activation function.

Original: To avoid aliasing, the spherical grid is sampled at a resolution of 2 ∗ ( L + 1) .
Revised: To avoid aliasing, the spherical grid is sampled at a resolution of 2 ∗ ( L + 1) .

Original: See Figure 4 for several example illustrations of spherical channels for different atomic structures.
Revised: See Figure 4 for several example illustrations of spherical channels for different atomic structures.

Original: 2.4 Energy and Forces Estimation
Revised: 2.4 Energy and Forces Estimation

Original: We compute the energy E by first estimating a per-atom energy using a pointwise function on the sphere and taking its integral over all possible orientations.
Revised: We compute the energy E by first estimating a per-atom energy using a pointwise function on the sphere and taking its integral over all possible orientations.

Original: The per-atom energy estimates are summed to obtain the overall system’s energy:
Revised: The per-atom energy estimates are summed to obtain the overall system’s energy:

Original: [Equation]
Revised: [Equation]

Original: where F energy is a three layer fully connected neural network ( R C → R 1 ) with SiLU activation functions.
Revised: where F energy is a three layer fully connected neural network ( R C → R 1 ) with SiLU activation functions.

Original: Forces may be calculated using two approaches: First, by calculating the gradients of the energy with respect to the atom positions.
Revised: Forces may be calculated using two approaches: First, by calculating the gradients of the energy with respect to the atom positions.

Original: This approach enforces energy conservation, but due to the need to back-propagate the gradients is computationally much more expensive.
Revised: This approach enforces energy conservation, but due to the need to back-propagate the gradients is computationally much more expensive.

Original: The second approach computes the forces in a manner similar to energy prediction, which is computationally more efficient, but does not enforce energy conservation.
Revised: The second approach computes the forces in a manner similar to energy prediction, which is computationally more efficient, but does not enforce energy conservation.

Original: In this approach, the forces are calculated by estimating a force magnitude | f | =
Revised: In this approach, the forces are calculated by estimating a force magnitude | f | =

Original: F force ( s ( K ) i (ˆ r )) in every direction ˆ r over the sphere.
Revised: F force ( s ( K ) i (ˆ r )) in every direction ˆ r over the sphere.

Original: Integration is performed after multiplying the magnitude by the orientation ˆ r to obtain directional vectors:
Revised: Integration is performed after multiplying the magnitude by the orientation ˆ r to obtain directional vectors:

Original: [Equation]
Revised: [Equation]

Original: where F force is a three layer fully connected neural network ( R C → R 1 ).
Revised: where F force is a three layer fully connected neural network ( R C → R 1 ).

Original: In practice, a discrete approximation of the integral is performed for equations (9) and (10) using a set of 128 evenly distributed points on the sphere (see Appendix C.2).
Revised: In practice, a discrete approximation of the integral is performed for equations (9) and (10) using a set of 128 evenly distributed points on the sphere (see Appendix C.2).

Original: Note that the application of a pointwise function of an arbitrary neural network at a finite number of discrete orientations is not invariant (energies) or equivariant (forces) to rotations.
Revised: Note that the application of a pointwise function of an arbitrary neural network at a finite number of discrete orientations is not invariant (energies) or equivariant (forces) to rotations.

Original: However in practice, numerically this operation does closely approximate these properties (see Appendix A.2).
Revised: However in practice, numerically this operation does closely approximate these properties (see Appendix A.2).

Original:  Experiments
Revised:  Experiments

Original: We present results on the Open Catalyst 2020 (OC20) dataset [6] that is released under a Creative
Revised: We present results on the Open Catalyst 2020 (OC20) dataset [6] that is released under a Creative

Original: Commons Attribution 4.0 License.
Revised: Commons Attribution 4.0 License.

Original: OC20 contains over 130M training examples for the task of predicting atomic energies and forces for catalysts used in renewable energy storage and other important applications [48].
Revised: OC20 contains over 130M training examples for the task of predicting atomic energies and forces for catalysts used in renewable energy storage and other important applications [48].

Original: This dataset is a popular benchmark for the ML community.
Revised: This dataset is a popular benchmark for the ML community.

Original: We begin by comparing results across all tasks on the test set.
Revised: We begin by comparing results across all tasks on the test set.

Original: Next, we show numerous ablation studies comparing model variations on the smaller OC20 2M dataset.
Revised: Next, we show numerous ablation studies comparing model variations on the smaller OC20 2M dataset.

Original: Finally, since several papers only report results on the IS2RE task using approaches that directly predict relaxed energies, we also train a model specifically for this task and approach, and compare in Appendix B.
Revised: Finally, since several papers only report results on the IS2RE task using approaches that directly predict relaxed energies, we also train a model specifically for this task and approach, and compare in Appendix B.

Original: 3.1 Implementation Details
Revised: 3.1 Implementation Details

Original: The implementation of spherical harmonics and their transformations uses the code provided by
Revised: The implementation of spherical harmonics and their transformations uses the code provided by

Original: Euclidean neural networks (e3nn) [16].
Revised: Euclidean neural networks (e3nn) [16].

Original: For message passing, two resolutions of spherical harmonics are used based on a atom’s nearest neighbors, see Appendix C.1 for exact parameters.
Revised: For message passing, two resolutions of spherical harmonics are used based on a atom’s nearest neighbors, see Appendix C.1 for exact parameters.

Original: Unless otherwise stated C = 128 , K = 16 , H = 1024 , E = 128 , and for F e in message passing only orders m ∈ [ − 1 , 1] are used.
Revised: Unless otherwise stated C = 128 , K = 16 , H = 1024 , E = 128 , and for F e in message passing only orders m ∈ [ − 1 , 1] are used.

Original: All forces are estimated directly as an output of the network, unless stated that the energy conserving gradient-based approach was used.
Revised: All forces are estimated directly as an output of the network, unless stated that the energy conserving gradient-based approach was used.

Original: During training, the coefficients for the force and energy losses are 100 and 2 respectively.
Revised: During training, the coefficients for the force and energy losses are 100 and 2 respectively.

Original: Training is performed using the AdamW optimizer [29] with a learning rate of 0.0004.
Revised: Training is performed using the AdamW optimizer [29] with a learning rate of 0.0004.

Original: The effective batch size is increased using data parallelism and
Revised: The effective batch size is increased using data parallelism and

Original: PyTorch’s Automatic Mixed Precision (AMP).
Revised: PyTorch’s Automatic Mixed Precision (AMP).

Original: All model code will be open sourced with an MIT license in the Open Catalyst Github repo.
Revised: All model code will be open sourced with an MIT license in the Open Catalyst Github repo.

Original: Please see the supplementary for details on the PaiNN[36] baseline.
Revised: Please see the supplementary for details on the PaiNN[36] baseline.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: The validation results are averaged across the four OC20 Validation set splits.
Revised: The validation results are averaged across the four OC20 Validation set splits.

Original: All SCN models are trained on 16 GPUs for 12 epochs with the learning rate reduced by 0.3 at 5, 7, 9, and 11 epochs, except SCN with L = 8 and 16 layers that used 32 GPUs to obtain a larger batch size.
Revised: All SCN models are trained on 16 GPUs for 12 epochs with the learning rate reduced by 0.3 at 5, 7, 9, and 11 epochs, except SCN with L = 8 and 16 layers that used 32 GPUs to obtain a larger batch size.

Original: Batch sizes vary based on the number of instances that can be fit in 32GB RAM.
Revised: Batch sizes vary based on the number of instances that can be fit in 32GB RAM.

Original: 3.2 OC20 Tasks
Revised: 3.2 OC20 Tasks

Original: We compare against numerous models trained on the OC20 2M, All and MD datasets across the
Revised: We compare against numerous models trained on the OC20 2M, All and MD datasets across the

Original: Structure to Energy and Forces (S2EF), Initial Structure to Relaxed Structure (IS2RS) and
Revised: Structure to Energy and Forces (S2EF), Initial Structure to Relaxed Structure (IS2RS) and

Original: Initial
Revised: Initial

Original: Structure to Relaxed Energy (IS2RE) tasks [6].
Revised: Structure to Relaxed Energy (IS2RE) tasks [6].

Original: OC20 2M, OC20 All and OC20 MD have 2, 133, and 38 million training examples respectively.
Revised: OC20 2M, OC20 All and OC20 MD have 2, 133, and 38 million training examples respectively.

Original: The 12 and 16 layer SCN models significantly outperform state-of-the-art GemNet-OC [15] on S2EF force prediction ( ≈ 14% improvement) and IS2RE energy predictions ( ≈ 10% improvement) when trained on OC20 2M, Table 1.
Revised: The 12 and 16 layer SCN models significantly outperform state-of-the-art GemNet-OC [15] on S2EF force prediction ( ≈ 14% improvement) and IS2RE energy predictions ( ≈ 10% improvement) when trained on OC20 2M, Table 1.

Original: Comparable results are found for energy MAE on S2EF with GemNet-OC.
Revised: Comparable results are found for energy MAE on S2EF with GemNet-OC.

Original: We also report throughput efficiency.
Revised: We also report throughput efficiency.

Original: Note that while the SCN models process fewer samples per second, they are more sample efficient than GemNet (Figure 2(b)).
Revised: Note that while the SCN models process fewer samples per second, they are more sample efficient than GemNet (Figure 2(b)).

Original: Efficiency improvements have also been noted for fully equivariant models [3].
Revised: Efficiency improvements have also been noted for fully equivariant models [3].

Original: We compare our largest SCN models trained on the All and the All+MD datasets (133M + 38M examples) across all OC20 tasks in Table 2.
Revised: We compare our largest SCN models trained on the All and the All+MD datasets (133M + 38M examples) across all OC20 tasks in Table 2.

Original: For these experiments we use a deep 1-tap model, and a slightly shallower 4-tap model since the use of 4-taps requires more memory.
Revised: For these experiments we use a deep 1-tap model, and a slightly shallower 4-tap model since the use of 4-taps requires more memory.

Original: Our model achieves state-of-the-art results for force MAE ( ≈ 9% improvement) and force cosine for the S2EF tasks.
Revised: Our model achieves state-of-the-art results for force MAE ( ≈ 9% improvement) and force cosine for the S2EF tasks.

Original: The
Revised: The

Original: SCN L = 6 model performs similarly to the recently released GemNet-OC model [15] on energy
Revised: SCN L = 6 model performs similarly to the recently released GemNet-OC model [15] on energy

Original: MAE for S2EF.
Revised: MAE for S2EF.

Original: On the challenging relaxation based IS2RE task that requires models to predict both accurate forces and energies, SCN outperforms GemNet-OC by over 7% .
Revised: On the challenging relaxation based IS2RE task that requires models to predict both accurate forces and energies, SCN outperforms GemNet-OC by over 7% .

Original: Note that we only train a single model to predict both energy and forces, where GemNet-OC-L-F+E + MD is the combination of two models, one trained specifically for forces and the other for energy.
Revised: Note that we only train a single model to predict both energy and forces, where GemNet-OC-L-F+E + MD is the combination of two models, one trained specifically for forces and the other for energy.

Original: 3.3 Ablation studies
Revised: 3.3 Ablation studies

Original: We explore numerous model variations in Table 1 trained on the OC20 2M dataset and evaluated on the OC20 validation set.
Revised: We explore numerous model variations in Table 1 trained on the OC20 2M dataset and evaluated on the OC20 validation set.

Original: The first set of ablation experiments explores reducing the complexity of our model.
Revised: The first set of ablation experiments explores reducing the complexity of our model.

Original: If messages are not rotated before and after applying F m in Equation 2, i.e ., A st
Revised: If messages are not rotated before and after applying F m in Equation 2, i.e ., A st

Original: [Table]
Revised: [Table]

Original: is an identity matrix, the results are significantly worse for energies and even more so for forces since forces are highly dependent on angular information.
Revised: is an identity matrix, the results are significantly worse for energies and even more so for forces since forces are highly dependent on angular information.

Original: If we replace the non-linear aggregation (Equation 8) with a simple summation (No 1x1 conv) in a model with m ∈ [ − 1 , 1] , the results also degrade across all metrics.
Revised: If we replace the non-linear aggregation (Equation 8) with a simple summation (No 1x1 conv) in a model with m ∈ [ − 1 , 1] , the results also degrade across all metrics.

Original: Results improve for m ∈ [ − 1 , 1] as compared to only using
Revised: Results improve for m ∈ [ − 1 , 1] as compared to only using

Original: m =coefficients, which demonstrates that if the equivariant to rotation constraint is relaxed, improved results may be achieved.
Revised: m =coefficients, which demonstrates that if the equivariant to rotation constraint is relaxed, improved results may be achieved.

Original: For higher m , m ∈ [ − 2 , 2] or m ∈ [ − 4 , 4] diminishing returns are noticed with increased computational cost.
Revised: For higher m , m ∈ [ − 2 , 2] or m ∈ [ − 4 , 4] diminishing returns are noticed with increased computational cost.

Original: In some applications, such as molecular dynamics, energy conserving models are needed.
Revised: In some applications, such as molecular dynamics, energy conserving models are needed.

Original: We trained an energy conserving model that estimates forces based on the gradients of the energy with respect to the atom positions.
Revised: We trained an energy conserving model that estimates forces based on the gradients of the energy with respect to the atom positions.

Original: Since this approach is more expensive both in memory and computation, we used a smaller model with L = 4 , H = 512 and 12 layers (grad-forces).
Revised: Since this approach is more expensive both in memory and computation, we used a smaller model with L = 4 , H = 512 and 12 layers (grad-forces).

Original: While the accuracy of the model is good, the results are slightly worse and is 10 times slower than a comparable model that directly estimates forces (direct-forces).
Revised: While the accuracy of the model is good, the results are slightly worse and is 10 times slower than a comparable model that directly estimates forces (direct-forces).

Original: When scaling the network, increasing L has a more significant impact on force estimation, while depth improves energy estimation.
Revised: When scaling the network, increasing L has a more significant impact on force estimation, while depth improves energy estimation.

Original: This may be due to higher L resulting in greater angular resolution, which is important to force estimation.
Revised: This may be due to higher L resulting in greater angular resolution, which is important to force estimation.

Original: Greater depth, which allows information to travel further, may lead to better energy predictions since energy is function of the entire atomic structure.
Revised: Greater depth, which allows information to travel further, may lead to better energy predictions since energy is function of the entire atomic structure.

Original: Sampling multiple z-axis rotations when computing F e with 4-taps (Equation 7) produces significantly improved force predictions, while using two bands ( L and L − 1 ) when aggregating messages improves both energy and force prediction.
Revised: Sampling multiple z-axis rotations when computing F e with 4-taps (Equation 7) produces significantly improved force predictions, while using two bands ( L and L − 1 ) when aggregating messages improves both energy and force prediction.

Original: The use of 2 bands reduces model throughput a small amount, while sampling with 4-taps reduces throughput by over 2 × .
Revised: The use of 2 bands reduces model throughput a small amount, while sampling with 4-taps reduces throughput by over 2 × .

Original: However, sample efficiency is further increased with 4-taps, resulting in similar training times for the same accuracy, Figure 2(b).
Revised: However, sample efficiency is further increased with 4-taps, resulting in similar training times for the same accuracy, Figure 2(b).

Original:  Related work
Revised:  Related work

Original: Molecular and atomic property prediction has made significant recent progress.
Revised: Molecular and atomic property prediction has made significant recent progress.

Original: Models relying on hand-crafted representations like Behler-Parrinello[4] and sGDML[8] have recently been surpassed by learned feature representations using GNNs [3, 12, 14, 36].
Revised: Models relying on hand-crafted representations like Behler-Parrinello[4] and sGDML[8] have recently been surpassed by learned feature representations using GNNs [3, 12, 14, 36].

Original: Early GNN developments focused only on invariant representations.
Revised: Early GNN developments focused only on invariant representations.

Original: CGCNN[45] and SchNet[35] make energy and force estimations using only atomic species and pair-wise distance information.
Revised: CGCNN[45] and SchNet[35] make energy and force estimations using only atomic species and pair-wise distance information.

Original: DimeNet[12, 13], SphereNet[28], and
Revised: DimeNet[12, 13], SphereNet[28], and

Original: GemNet[14, 15] extend this to explicitly capture triplet and quadruplet angles, which are scalars.
Revised: GemNet[14, 15] extend this to explicitly capture triplet and quadruplet angles, which are scalars.

Original: Utilizing invariant representations offers flexibility in model architecture design since the node or edge features are inherently invariant and model constraints are not needed to maintain equivariance.
Revised: Utilizing invariant representations offers flexibility in model architecture design since the node or edge features are inherently invariant and model constraints are not needed to maintain equivariance.

Original: The enumeration of triplets and quadruplets of atoms to calculate functions of relative or dihedral angles requires careful model design to maintain efficiency.
Revised: The enumeration of triplets and quadruplets of atoms to calculate functions of relative or dihedral angles requires careful model design to maintain efficiency.

Original: More recently, equivariant models, capturing both scalar and equivariant features, have outperformed traditional GNNs on small molecular datasets including MD17[7] and QM9[32].
Revised: More recently, equivariant models, capturing both scalar and equivariant features, have outperformed traditional GNNs on small molecular datasets including MD17[7] and QM9[32].

Original: While not strictly equivariant, our work falls into this later class of GNNs for atomic property predictions.
Revised: While not strictly equivariant, our work falls into this later class of GNNs for atomic property predictions.

Original: Models that strictly enforce equivariance to the SO(3) (3D rotations) or E(3) group (3D rigid transformations) share similarities to our use of spherical channels [36, 1, 3, 9, 5].
Revised: Models that strictly enforce equivariance to the SO(3) (3D rotations) or E(3) group (3D rigid transformations) share similarities to our use of spherical channels [36, 1, 3, 9, 5].

Original: Both represent atoms by higher-order tensors and not just scalar information.
Revised: Both represent atoms by higher-order tensors and not just scalar information.

Original: For instance, Tensor Field Networks [42], Cormorant [1], NequIP [3], SEGNN [5] and BOTNet [2] use the same spherical harmonic representations as SCNs, while PaiNN [36] is an l = 1 variation.
Revised: For instance, Tensor Field Networks [42], Cormorant [1], NequIP [3], SEGNN [5] and BOTNet [2] use the same spherical harmonic representations as SCNs, while PaiNN [36] is an l = 1 variation.

Original: All models place constraints on the operations that can be performed to ensure equivarance, such as restricting non-linear operators to invariant scalar inputs, e.g., pair-wise atom distances.
Revised: All models place constraints on the operations that can be performed to ensure equivarance, such as restricting non-linear operators to invariant scalar inputs, e.g., pair-wise atom distances.

Original: In networks like TFN, geometric information of higher degree l are mixed into the l = 0 features via tensor products on which a gated non-linear function is applied.
Revised: In networks like TFN, geometric information of higher degree l are mixed into the l = 0 features via tensor products on which a gated non-linear function is applied.

Original: Similar to our approach, the recently proposed SEGNN uses the spherical harmonic representation to enable steerable features based on edge orientations.
Revised: Similar to our approach, the recently proposed SEGNN uses the spherical harmonic representation to enable steerable features based on edge orientations.

Original: SEGNN introduces steerable MLPs that enforce equivariance by restricting the learnable parameters to those that scale the Clebsch-Gordan tensor products followed by gated non-linearities [43].
Revised: SEGNN introduces steerable MLPs that enforce equivariance by restricting the learnable parameters to those that scale the Clebsch-Gordan tensor products followed by gated non-linearities [43].

Original: In our approach, we increase the expressivity of the model by identifying a set of m = 0 coefficients across degrees l that are invariant to rotations given an edge’s orientation to which unconstrained functions may be applied.
Revised: In our approach, we increase the expressivity of the model by identifying a set of m = 0 coefficients across degrees l that are invariant to rotations given an edge’s orientation to which unconstrained functions may be applied.

Original: We also demonstrate that improved performance can be achieved by relaxing the requirement for strict message equivariance by including a broader set of coefficients beyond m = 0 .
Revised: We also demonstrate that improved performance can be achieved by relaxing the requirement for strict message equivariance by including a broader set of coefficients beyond m = 0 .

Original: Furthermore, we project the spherical channels onto a grid and perform a pointwise 1 × 1 convolution followed by a non-linearity.
Revised: Furthermore, we project the spherical channels onto a grid and perform a pointwise 1 × 1 convolution followed by a non-linearity.

Original: Although not strictly equivariant, the pointwise operation allows for complex mixing of different degrees of spherical harmonics resulting in rich geometric descriptions.
Revised: Although not strictly equivariant, the pointwise operation allows for complex mixing of different degrees of spherical harmonics resulting in rich geometric descriptions.

Original: Perhaps as a result of the network’s increased expessivity, we see an increase in accuracy with higher degrees ( L = 4 , 6 , 8 ) unlike previous approaches that typically use L = 1 or L = 2 .
Revised: Perhaps as a result of the network’s increased expessivity, we see an increase in accuracy with higher degrees ( L = 4 , 6 , 8 ) unlike previous approaches that typically use L = 1 or L = 2 .

Original:  Discussion
Revised:  Discussion

Original: A limitation of SCN is that it is very computationally expensive if energy conservation is enforced through the estimation of forces using energy gradients with respect to atom positions.
Revised: A limitation of SCN is that it is very computationally expensive if energy conservation is enforced through the estimation of forces using energy gradients with respect to atom positions.

Original: This may limit its use in chemistry applications such as molecular dynamics.
Revised: This may limit its use in chemistry applications such as molecular dynamics.

Original: Non-energy conserving models are still practically useful for applications such as structure relaxations and transition state searches.
Revised: Non-energy conserving models are still practically useful for applications such as structure relaxations and transition state searches.

Original: The SCN model scales O ( n 2 ) with respect to L , which may limit using L larger than 8.
Revised: The SCN model scales O ( n 2 ) with respect to L , which may limit using L larger than 8.

Original: Using a larger set of m coefficients during message passing also faces challenges, since the network may find it more difficult to learn approximately equivariant mappings.
Revised: Using a larger set of m coefficients during message passing also faces challenges, since the network may find it more difficult to learn approximately equivariant mappings.

Original: In Table 1, we saw that increased depth improves results, especially to energy prediction.
Revised: In Table 1, we saw that increased depth improves results, especially to energy prediction.

Original: It remains an open question whether deeper networks will see further improvement.
Revised: It remains an open question whether deeper networks will see further improvement.

Original: Given SCN’s increased expressivity due to the relaxation of the equivariance constraint and larger model sizes, it is prone to overfitting on smaller datasets.
Revised: Given SCN’s increased expressivity due to the relaxation of the equivariance constraint and larger model sizes, it is prone to overfitting on smaller datasets.

Original: SCN’s results on datasets such as MD17[7] and QM9[32] are not state-of-the-art.
Revised: SCN’s results on datasets such as MD17[7] and QM9[32] are not state-of-the-art.

Original: However, in future work it would be interesting to explore pre-training on larger datasets, such as OC20, and see if improved results can be achieved on smaller datasets through finetuning [25].
Revised: However, in future work it would be interesting to explore pre-training on larger datasets, such as OC20, and see if improved results can be achieved on smaller datasets through finetuning [25].

Original: While this work is motivated by problems we face in addressing climate change, advances in chemistry may have numerous use cases.
Revised: While this work is motivated by problems we face in addressing climate change, advances in chemistry may have numerous use cases.

Original: Some of these use cases may have unintended consequences, such as the development of the Haber-Bosch process for ammonia fertilizer production that enabled the world to feed a growing population, but over fertilization has recently led to ocean “dead zones” and its production is very carbon intensive.
Revised: Some of these use cases may have unintended consequences, such as the development of the Haber-Bosch process for ammonia fertilizer production that enabled the world to feed a growing population, but over fertilization has recently led to ocean “dead zones” and its production is very carbon intensive.

Original: More alarmingly, the knowledge gained from fertilizer production was used to create explosives during wartime [20].
Revised: More alarmingly, the knowledge gained from fertilizer production was used to create explosives during wartime [20].

Original: We hope to steer research in this area towards beneficial applications by utilizing datasets such as OC20.
Revised: We hope to steer research in this area towards beneficial applications by utilizing datasets such as OC20.

Original: In conclusion, we propose a GNN that uses spherical harmonics to represent channel embeddings that contain explicit orientation information.
Revised: In conclusion, we propose a GNN that uses spherical harmonics to represent channel embeddings that contain explicit orientation information.

Original: While the network is encouraged to learn output mappings that are equivariant to rotations, best performance is achieved by relaxing the equivariant constraints and allowing for more expressive non-linear transformations.
Revised: While the network is encouraged to learn output mappings that are equivariant to rotations, best performance is achieved by relaxing the equivariant constraints and allowing for more expressive non-linear transformations.

Original: We demonstrate stateof-the-art results across numerous tasks on the large-scale OC20 dataset for modeling catalyst for applications addressing climate change.A Empirical measurement of equivariance to rotations
Revised: We demonstrate stateof-the-art results across numerous tasks on the large-scale OC20 dataset for modeling catalyst for applications addressing climate change.

Original: The SCN is not strictly equivariant to rotations, but depending on the design choices approximate equivarance may be achieved.

Original: We begin by empirically measuring the network’s invariance and equivariance to rotation for energy and forces respectively.

Original: We accomplish this by computing the absolute difference between pairs of results computed from an input and a randomly rotated version of the input:

Original: [Equation]

Original: where d is the 3D positional difference between pairs of atoms, a are their atomic numbers, and R is a random 3D rotation matrix.

Original: SCN energy and SCN force are the network’s predictions for energy and forces respectively.

Original: Mean Absolute Difference (MAD) results for various model choices are shown in Table 3 for models with 12 layers and L = 6 .

Original: Differences are averaged over a model’s outputs for a random 1,000 atomic structures.

Original: There are four sources that may lead the network to predict different values for rotated versions of the input: 1) the use of m ̸ = 0 coefficients during message passing, 2) non-linear message aggregation, Equation (3), 3) the energy and force output blocks, Equations (4,5), and 4) limits to numerical precision especially when using Automatic Mixed Precision (AMP).

Original: In Table 3, we observe that the MAD increases when m ∈ [ − 1 , 1] coefficients are used during message passing as compared to the m = 0 message passing that is equivariant.

Original: However, the MAD of the m ∈ [ − 1 , 1] model that uses 4 taps during message passing is nearly identical to m = 0 when not using AMP.

Original: This shows that the 4-tap model is close to producing equivariant results.

Original: Note MAD is non-zero even for m = 0 due to the other sources of errors.

Original: The 1 × 1 convolution results in greater rotational differences for forces, while m ∈ [ − 1 , 1] during message passing has a bigger impact on energies.

Original: If AMP is used, higher MADs are found due to the limits of numerical precision.

Original: A.1 Message aggregation

Original: Next, we discuss the degree to which our message aggregation Equation (3) is empirically equivariant.

Original: If Equation (3) was equivariant, the following would hold:

Original: [Equation]

Original: where D is a block diagonal matrix containing Wigner-D matrices that perform a rotation of the coefficients.

Original: That is, the same result should be achieved whether the input or output is rotated.

Original: Since the transformation G uses a discrete sampling of the sphere, Equation (3) is clearly not strictly equivariant.

Original: The neural network F c may introduce non-linearities that result in frequencies higher than those able to be represented by the maximum degree L used by the spherical harmonics.

Original: As a result, the sampling used by G may be below the Nyquist rate needed to avoid aliasing.

Original: This may be minimized but not eliminated by utilizing smooth activation functions such as SiLU.

Original: However, in practice we find that the use of Equation (3) with sampling resolution of 2( L + 1) or higher does in practice lead to results that are approximately equivariant.

Original: In Table 4, we show several empirical results with different depths and activation functions.

Original: If only a linear layer is used, the function is equivariant and there is no difference in computed values.

Original: For non-linear activations, such as ReLU and SiLU, differences of 4% − 7% are seen between the rotated versions.

Original: Percentages are the Mean

Original: [Table]

Original: [Table]

Original: Absolute Difference (MAD) divided by the mean absolute value of the outputs.

Original: MAD is computed using:

Original: [Equation]

Original: Minimal differences are seen between using SiLU or ReLU.

Original: However, higher percentage differences are observed when using two layers of non-linearities when compared to one.

Original: We use two-layers followed by a linear layer in the models for this paper.

Original: A.2 Energy and Force Output Blocks

Original: Our final set of experiments measure the empirical invariance and equivariance to rotations for the output energy and force blocks.

Original: The differences between the not rotated and rotated versions is computed using:

Original: [Equation]

Original: 16) where R is a random 3D rotation matrix and D its corresponding block diagonal matrix containing the Wigner-D matrices.

Original: Results are shown in Table 5.

Original: For both energy and forces the results are nearly equivariant, and demonstrate the outputs blocks have negligible negative impact on the overall network’s equivariance.

Original: This is likely due to the force and energy output blocks integrating over the entire sphere, which may result in any differences negating each other.

Original: [Table]

Original: [Table]

Original: B OC20 IS2RE Direct Results

Original: The task of Initial Structure to Relaxed Energy (IS2RE) may be accomplished using two approaches: 1) Use an S2EF model to relax the positions of the atoms (find local energy minima) and output the energy at the relaxed structure, or 2) directly predict the relaxed energy from the initial structure without performing the relaxation.

Original: Empirically, the first approach reported previously in the paper achieves higher accuracy (see Tables 1 and 2 in the main paper), while the second approach is more efficient both during training and inference.

Original: Although solving the same problem, the second approach of direct prediction is a fundamentally different problem than the first.

Original: The direct approach cannot take advantage of detailed position and angular information, since the initial structure only contains a rough placement of the atoms.

Original: Instead it must rely more on global knowledge of how the atoms are arranged.

Original: Thus, an approach that works well for relaxation based approaches to IS2RE might not work well for direct approaches.

Original: As shown in Table 6, the SCN model achieves state-of-the-art results for energy MAE across all test splits for models that do not use ancillary loses [46, 18].

Original: Energy within Threshold (EwT) is comparable to SEGNN [5].

Original: Improvements for SCN are most pronounced for the out-of-domain splits, demonstrating its increased ability to generalize to unseen data.

Original: C Implementation details

Original: In this section, we describe additional implementation details not in the main paper.

Original: C.1 Varying spherical channel resolution

Original: For atoms that are far away from each other, the messages may be computed using lower resolution spherical channels to reduce memory usage while maintaining similar accuracy.

Original: For all results in this paper, the 12 closest neighbors to each atom use the settings described in the paper.

Original: For neighbors ranked 13 to 40 by distance, L is reduced by 2 with a minimum value of 4, and H is reduced by a factor of 4.

Original: Messages are not calculated for neighboring atoms with ranked distances greater thanor a cutoff distance greater than 8 Å.

Original: C.2 Point sampling on the sphere for output blocks

Original: For both the energy and force output blocks the integrals in Equations 4 and 5 in the main paper are approximated using points samples on the sphere.

Original: Finding a set of evenly distributed points on a sphere for an arbitrary number of points remains an open problem.

Original: We use points that are approximately evenly distributed using spherical Fibonacci point sets [21].

Original: Since the density of points does vary slightly across the sphere, we weight each point using a Gaussian weighted counting of nearby points on the unit sphere with σ = 0 .

Original: C.3 PaiNN baseline

Original: The PaiNN baseline is our reimplementation of Schütt et al .

Original: [36] with the difference that forces are predicted directly from vectorial features via a gated equivariant block instead of gradients of the energy output.

Original: This breaks energy conservation but is essential for good performance on OC20.

Original: D Note on spherical harmonics properties

Original: In the paper it is stated that m =

Original: spherical harmonics are invariant to rotations about the z-axis.

Original: This is easily seen by looking at the equations for the real spherical harmonics.

Original: We show the Y lm equations for up to l = 2 below, parameterized by θ (polar) and ϕ (longitudinal rotation about the z-axis).

Original: [-Notice that all m = 
-]
Revised: [-Notice that all m = 
-]

Original: spherical harmonics are only a function of θ , and thus invariant to changes in ϕ .

Original: In Equations 4 and 5 we state that the m = − 1 and m = 1 values are sine and cosine functions of ϕ .

Original: If we integrate or fix θ , we see that the values for m = {− 1 , 1 } do indeed take the form of Equations 4 and 5.

Original: [Equation]

Original: E Overfitting on the training dataset

Original: Figure 5 shows the train and validation errors during training.

Original: Interestingly, we see very little overfitting on forces, but energy has significant overfitting.

Original: This may be due to the energies being a per-structure property, while the forces have more examples since they are a per-atom property.

Original: However, overfitting on the energies is still surprising given the large number of examples in the

Original: OC20 All + MD training dataset.

Original: This indicates that the SCN model can fit complex functions but could improve on its ability to better generalize through model improvements or data augmentation.

Original: Similar trends are found when trained on the OC20 2M dataset.

Original: F Impact of model size

Original: Model size has an impact on the accuracy of the SCN model.

Original: In Figure 6 we compare the accuracy and model size of SCN and GemNet-OC.

Original: For similar model sizes, we see SCN achieves better accuracy than GemNet-OC.

Original: This demonstrates that the improvement we see from SCN is more than just the use of larger model sizes.

Original: In Figure 7, we see the accuracy of the SCN model improves as larger models are used on the 2M dataset.

Original: However, the accuracy for even the smallest model still outperforms other approaches reported in Table 1 in the main paper.