Original: E FFICIENT L ONG -R ANGE C ONVOLUTIONS FOR P OINT C LOUDS
Revised: E FFICIENT L ONG -R ANGE C ONVOLUTIONS FOR P OINT C LOUDS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The efﬁcient treatment of long-range interactions for point clouds is a challenging problem in many scientiﬁc machine learning applications.
Revised: The efﬁcient treatment of long-range interactions for point clouds is a challenging problem in many scientiﬁc machine learning applications.

Original: To extract global information, one usually needs a large window size, a large number of layers, and/or a large number of channels.
Revised: To extract global information, one usually needs a large window size, a large number of layers, and/or a large number of channels.

Original: This can often signiﬁcantly increase the computational cost.
Revised: This can often signiﬁcantly increase the computational cost.

Original: In this work, we present a novel neural network layer that directly incorporates long-range information for a point cloud.
Revised: In this work, we present a novel neural network layer that directly incorporates long-range information for a point cloud.

Original: This layer, dubbed the long-range convolutional (LRC)-layer, leverages the convolutional theorem coupled with the non-uniform Fourier transform.
Revised: This layer, dubbed the long-range convolutional (LRC)-layer, leverages the convolutional theorem coupled with the non-uniform Fourier transform.

Original: In a nutshell, the LRC-layer molliﬁes the point cloud to an adequately sized regular grid, computes its Fourier transform, multiplies the result by a set of trainable Fourier multipliers, computes the inverse
Revised: In a nutshell, the LRC-layer molliﬁes the point cloud to an adequately sized regular grid, computes its Fourier transform, multiplies the result by a set of trainable Fourier multipliers, computes the inverse

Original: Fourier transform, and ﬁnally interpolates the result back to the point cloud.
Revised: Fourier transform, and ﬁnally interpolates the result back to the point cloud.

Original: The resulting global all-to-all convolution operation can be performed in nearly-linear time asymptotically with respect to the number of input points.
Revised: The resulting global all-to-all convolution operation can be performed in nearly-linear time asymptotically with respect to the number of input points.

Original: The LRC-layer is a particularly powerful tool when combined with local convolution as together they offer efﬁcient and seamless treatment of both short and long range interactions.
Revised: The LRC-layer is a particularly powerful tool when combined with local convolution as together they offer efﬁcient and seamless treatment of both short and long range interactions.

Original: We showcase this framework by introducing a neural network architecture that combines LRC-layers with short-range convolutional layers to accurately learn the energy and force associated with a N -body potential.
Revised: We showcase this framework by introducing a neural network architecture that combines LRC-layers with short-range convolutional layers to accurately learn the energy and force associated with a N -body potential.

Original: We also exploit the induced two-level decomposition and propose an efﬁcient strategy to train the combined architecture with a reduced number of samples.
Revised: We also exploit the induced two-level decomposition and propose an efﬁcient strategy to train the combined architecture with a reduced number of samples.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Point-cloud representations provide detailed information of objects and environments.
Revised: Point-cloud representations provide detailed information of objects and environments.

Original: The development of novel acquisition techniques, such as laser scanning, digital photogrammetry, light detection and ranging (LIDAR), 3D scanners, structure-from-motion (SFM), among others, has increased the interest of using point cloud representation in various applications such as digital preservation, surveying, autonomous driving (Chen et al., 2017), 3D gaming, robotics(Oh & Watanabe, 2002), and virtual reality (Park et al., 2008).
Revised: The development of novel acquisition techniques, such as laser scanning, digital photogrammetry, light detection and ranging (LIDAR), 3D scanners, structure-from-motion (SFM), among others, has increased the interest of using point cloud representation in various applications such as digital preservation, surveying, autonomous driving (Chen et al., 2017), 3D gaming, robotics (Oh & Watanabe, 2002), and virtual reality (Park et al., 2008).

Original: In return, this new interest has fueled the development of machine learning frameworks that use point clouds as input.
Revised: In return, this new interest has fueled the development of machine learning frameworks that use point clouds as input.

Original: Historically, early methods used a preprocessing stage that extracted meticulously hand-crafted features from the point cloud, which were subsequently fed to a neural network (Chen et al., 2003; Rusu et al., 2008; Rusu et al., 2009; Aubry et al., 2011), or they relied on voxelization of the geometry (Savva et al., 2016; Wu et al., 2015; Riegler et al., 2017; Maturana & Scherer, 2015).
Revised: Historically, early methods used a preprocessing stage that extracted meticulously hand-crafted features from the point cloud, which were subsequently fed to a neural network (Chen et al., 2003; Rusu et al., 2008; Rusu et al., 2009; Aubry et al., 2011), or they relied on voxelization of the geometry (Savva et al., 2016; Wu et al., 2015; Riegler et al., 2017; Maturana & Scherer, 2015).

Original: The pointNet architecture (Qi et al., 2017) was the ﬁrst to handle raw point cloud data directly and learn features on the ﬂy.
Revised: The PointNet architecture (Qi et al., 2017) was the ﬁrst to handle raw point cloud data directly and learn features on the ﬂy.

Original: This work has spawned several related approaches, aiming to attenuate drawbacks from the original methodology, such as pointNet++ (Qi et al., 2017), or to increase the accuracy and range of application (Wang et al., 2019; Zhai et al., 2020;
Revised: This work has spawned several related approaches, aiming to attenuate drawbacks from the original methodology, such as PointNet++ (Qi et al., 2017), or to increase the accuracy and range of application (Wang et al., 2019; Zhai et al., 2020;

Original: Li et al., 2018; Liu et al., 2019).
Revised: Li et al., 2018; Liu et al., 2019).

Original: Even though such methods have been quite successful for machine learning problems, they rely on an assumption of locality, which may produce large errors when the underlying task at hand exhibits long-range interactions (LRIs).
Revised: Even though such methods have been quite successful for machine learning problems, they rely on an assumption of locality, which may produce large errors when the underlying task at hand exhibits long-range interactions (LRIs).

Original: To capture such interactions using standard convolutional layers, one can use wider window sizes, deeper networks, and/or a large number of features, which may increase the computational cost signiﬁcantly.
Revised: To capture such interactions using standard convolutional layers, one can use wider window sizes, deeper networks, and/or a large number of features, which may increase the computational cost signiﬁcantly.

Original: Several approaches have been proposed to efﬁciently capture such interactions in tasks such as semantic segmentation, of which the ideas we brieﬂy summarize below.
Revised: Several approaches have been proposed to efﬁciently capture such interactions in tasks such as semantic segmentation, of which the ideas we brieﬂy summarize below.

Original: In the multi-scale type of approaches, features are progressively processed and merged.
Revised: In the multi-scale type of approaches, features are progressively processed and merged.

Original: Within this family, there exist several variants, where the underlying neural networks can be either recursive neural networks (Ye et al., 2018), convolutional layers (Xu et al., 2019; Xu et al., 2018) or autoencoders (Yang et al., 2018; Deng et al., 2018).
Revised: Within this family, there exist several variants, where the underlying neural networks can be either recursive neural networks (Ye et al., 2018), convolutional layers (Xu et al., 2019; Xu et al., 2018) or autoencoders (Yang et al., 2018; Deng et al., 2018).

Original: Some works have proposed skip connections, following an U-net (Ronneberger et al., 2015) type architecture (Zhou & Tuzel, 2018; Qi et al., 2017), while others have focused on using a tree structure for the clustering of the points (Klokov & Lempitsky, 2017; Zeng & Gevers, 2018; Gadelha et al., 2018), or using an reference permutohedral lattices to compute convolutions (Jampani et al., 2016) whose results are interpolated back to the point cloud (Su et al., 2018).
Revised: Some works have proposed skip connections, following an U-net (Ronneberger et al., 2015) type architecture (Zhou & Tuzel, 2018; Qi et al., 2017), while others have focused on using a tree structure for the clustering of the points (Klokov & Lempitsky, 2017; Zeng & Gevers, 2018; Gadelha et al., 2018), or using an reference permutohedral lattices to compute convolutions (Jampani et al., 2016) whose results are interpolated back to the point cloud (Su et al., 2018).

Revised: Although these methods have been shown to be successful in a range of applications, when the task at hand presents symmetries, such as rotation, translation, and permutation invariance, there is no systematic framework to embed those symmetries into the algorithmic pipelines.

Original: Another line of work, relies on interpreting the point cloud as a graph and use spectral convolutions (Bruna et al.;
Revised: Another line of work, relies on interpreting the point cloud as a graph and use spectral convolutions (Bruna et al.;

Original: Defferrard et al., 2016).
Revised: Defferrard et al., 2016), whose cost can scale super-linearly when dealing with LRIs.

Original: In applications of machine learning to scientiﬁc computing, several classical multilevel matrix factorizations have been rewritten in the context of machine learning (Kondor et al., 2014), which have been adapted to handle long-range interactions in the context of end-to-end maps using voxelized geometries in (Fan et al.; 2019; Khoo & Ying, 2019; Fan & Ying, 2019) resulting in architectures similar to U-nets (Ronneberger et al., 2015), which have been extended to point clouds in (Li et al., 2020). 
Revised: In applications of machine learning to scientiﬁc computing, several classical multilevel matrix factorizations have been rewritten in the context of machine learning (Kondor et al., 2014), which have been adapted to handle long-range interactions in the context of end-to-end maps using voxelized geometries in (Fan et al., 2019b;a; Khoo & Ying, 2019; Fan & Ying, 2019) resulting in architectures similar to U-nets (Ronneberger et al., 2015), which have been extended to point clouds in (Li et al., 2020). Due to underlying voxelization of the geometry, it may be difﬁcult for these networks to generalize when the resolution of the voxelization changes.

Original: The efﬁcient treatment of LRI for point clouds is also a prominent problem in many physical applications such as molecular modeling and molecular dynamics simulation.
Revised: The efﬁcient treatment of LRI for point clouds is also a prominent problem in many physical applications such as molecular modeling and molecular dynamics simulation.

Original: While long-range electrostatic interactions are omnipresent, it has been found that effectively short-ranged models can already describe the N -body potential and the associated force ﬁeld (Behler & Parrinello, 2007;
Revised: While long-range electrostatic interactions are omnipresent, it has been found that effectively short-ranged models can already describe the N -body potential and the associated force ﬁeld (Behler & Parrinello, 2007;

Original: Zhang et al., 2018a;b) for a wide range of physical systems.
Revised: Zhang et al., 2018a;b) for a wide range of physical systems.

Original: There have also been a number of recent works aiming at more general systems beyond this regime of effective short-range interactions, such as the work of Ceriotti and co-workers (Grisaﬁ & Ceriotti, 2019; Grisaﬁ et al.
Revised: There have also been a number of recent works aiming at more general systems beyond this regime of effective short-range interactions, such as the work of Ceriotti and co-workers (Grisaﬁ & Ceriotti, 2019; Grisaﬁ et al.

Original: ; Nigam et al., 2020;
Revised: ; Nigam et al., 2020;

Original: Rossi et al., 2020), as well as the works of (Yao et al., 2018;
Revised: Rossi et al., 2020), as well as the works of (Yao et al., 2018;

Original: Ko et al. ; Hirn et al., 2017; Rupp et al., 2012; Huo & Rupp; Deng et al., 2019; Bereau et al., 2018; Zhang et al., 2019).
Revised: Ko et al., 2009; Hirn et al., 2017; Rupp et al., 2012; Huo & Rupp; Deng et al., 2019; Bereau et al., 2018; Zhang et al., 2019).

Original: The general strategy is to build parameterized long-range interactions into the kernel methods or neural network models, so that the resulting model can characterize both short-range, as well as long-range electrostatic interactions.
Revised: The general strategy is to build parameterized long-range interactions into the kernel methods or neural network models, so that the resulting model can characterize both short-range, as well as long-range electrostatic interactions.

Original: In the neural network context, the computational cost of treating the LRI using these methods can grow superlinearly with the system size.
Revised: In the neural network context, the computational cost of treating the LRIs using these methods can grow superlinearly with the system size.

Original: The idea of this work is aligned with the approaches in the molecular modeling community, which constructs a neural network layer to directly describe the LRI.
Revised: The idea of this work is aligned with the approaches in the molecular modeling community, which constructs a neural network layer to directly describe the LRI.

Original: In particular, we present a new longrange convolutional (LRC)-layer, which performs a global convolutional operation in nearly-linear time with respect to number of units in the layer.
Revised: In particular, we present a new longrange convolutional (LRC)-layer, which performs a global convolutional operation in nearly-linear time with respect to number of units in the layer.

Original: By leveraging the non-uniform Fourier transform (NUFFT) (Dutt & Rokhlin, 1993; Greengard & Lee, 2004; Barnett et al., 2019) technique, the
Revised: By leveraging the non-uniform Fourier transform (NUFFT) (Dutt & Rokhlin, 1993; Greengard & Lee, 2004; Barnett et al., 2019) technique, the

Original: LRC-layer implements a convolution with a point-wise multiplication in the frequency domain with trainable weights known as Fourier multipliers .
Revised: LRC-layer implements a convolution with a point-wise multiplication in the frequency domain with trainable weights known as Fourier multipliers .

Original: The NUFFT is based on the regular fast Fourier transform (FFT) (Cooley & Tukey, 1965) with a fast gridding algorithms, to allow for fast convolution on unstructured data.
Revised: The NUFFT is based on the regular fast Fourier transform (FFT) (Cooley & Tukey, 1965) with a fast gridding algorithms, to allow for fast convolution on unstructured data.

Original: This new LRC-layer provides a new set of descriptors, which can be used in tandem with the descriptors provided by short-range convolutional layers to improve the performance of the neural network.
Revised: This new LRC-layer provides a new set of descriptors that can seamlessly satisfy relevant symmetries. For instance, when the kernel of the LRI is rotationally invariant, such symmetry can be directly built into the parameterization of the Fourier kernel. Such descriptors can be used in tandem with the descriptors provided by short-range convolutional layers to improve the performance of the neural network.

Original: Efﬁcient training of a neural network with the LRC-layer for capturing the information of LRIs is another challenging problem.
Revised: Efﬁcient training of a neural network with the LRC-layer for capturing the information of LRIs is another challenging problem.

Original: Short-range models can often be trained with data generated with a relatively small computational box (called the small-scale data), and they can be seamlessly deployed in large-scale systems without signiﬁcantly increasing the generalization error.
Revised: Short-range models can often be trained with data generated with a relatively small computational box (called the small-scale data), and they can be seamlessly deployed in large-scale systems without signiﬁcantly increasing the generalization error.

Original: On the other hand, long-range models need to be trained directly with data generated in a large computational box (called the large-scale data), and the generation process of such large-scale data can be very expensive.
Revised: On the other hand, long-range models need to be trained directly with data generated in a large computational box (called the large-scale data), and the generation process of such large-scale data can be very expensive.

Original: For instance, in molecular modeling, the training data is often generated with highly accurate quantum mechanical methods, of which the cost can scale steeply as O ( N α ) , where N is the system size and α ≥ 3 .
Revised: For instance, in molecular modeling, the training data is often generated with highly accurate quantum mechanical methods, of which the cost can scale steeply as O ( N α ) , where N is the system size and α ≥ 3 .

Original: Therefore it is desirable to minimize the number of samples with a large system size.
Revised: Therefore it is desirable to minimize the number of samples with a large system size.

Original: In many applications, the error of the effective short-range model is already modestly small.
Revised: In many applications, the error of the effective short-range model is already modestly small.

Original: This motivates us to propose a two-scale training strategy as follows.
Revised: This motivates us to propose a two-scale training strategy as follows.

Original: We ﬁrst generate many small-scale data (cheaply and possibly in parallel), and train the network without the LRC-layer.
Revised: We ﬁrst generate many small-scale data (cheaply and possibly in parallel), and train the network without the LRC-layer.

Original: Then we use a small number of large-scale data, and perform training with both the short- and long-range convolutional layers.
Revised: Then we use a small number of large-scale data, and perform training with both the short- and long-range convolutional layers.

Original: In order to demonstrate the effectiveness of the LRC-layer and the two-scale training procedure, we apply our method to evaluate the energy and force associated with a model N -body potential that exhibit tunable short- and long-range interactions in one, two and three dimensions.
Revised: In order to demonstrate the effectiveness of the LRC-layer and the two-scale training procedure, we apply our method to evaluate the energy and force associated with a model N -body potential that exhibit tunable short- and long-range interactions in one, two and three dimensions.

Revised: The input point cloud consists of the atomic positions, and the output data include the N -body potential, local potential, and the force (derivative of the N -body potential with respect to atomic positions).

Revised: In particular, the local potential and the force can be viewed as point clouds associated with the atomic positions.

Revised: The evaluation of the N -body potential is a foundational component in molecular modeling, and LRI plays an important role in the description of ionic systems, macroscopically polarized interfaces, electrode surfaces, and many other problems in nanosciences (French et al., 2010).

Original: Our result veriﬁes that the computational cost of the long-range layer can be reduced from O ( N 2 ) using a direct implementation, to O ( N ) (up to logarithmic factors) using NUFFT.
Revised: Our result veriﬁes that the computational cost of the long-range layer can be reduced from O ( N 2 ) using a direct implementation, to O ( N ) (up to logarithmic factors) using NUFFT.

Original: Furthermore, we demonstrate that the force, i.e. the derivatives of the potential with respect to all inputs can be evaluated with O ( N ) cost (up to logarithmic factors).
Revised: Furthermore, we demonstrate that the force, i.e. the derivatives of the potential with respect to all inputs can be evaluated with O ( N ) cost (up to logarithmic factors).

Original: In terms of sample efﬁciency, we ﬁnd that for the model problem under study here, the two-scale training strategy can effectively reduce the number of large-scale samples by over an order of magnitude to reach the target accuracy.
Revised: In terms of sample efﬁciency, we ﬁnd that for the model problem under study here, the two-scale training strategy can effectively reduce the number of large-scale samples by over an order of magnitude to reach the target accuracy.

Revised: This can be particularly valuable in the context of molecular modeling, where accurate data are often obtained from ﬁrst principle electronic structure calculations.

Revised: Such calculations are often very expensive for large scale systems, and the number of large-scale samples is thus limited.

Original:  L ONG - RANGE C ONVOLUTIONAL L AYER
Revised:  L ONG - RANGE C ONVOLUTIONAL L AYER

Original: Convolutional layers are perhaps the most important building-block in machine learning, due to their great success in image processing and computer vision.
Revised: Convolutional layers are perhaps the most important building-block in machine learning, due to their great success in image processing and computer vision.

Original: A convolutional layer convolves the input, usually an array, with a rectangular mask containing the trainable parameters.
Revised: A convolutional layer convolves the input, usually an array, with a rectangular mask containing the trainable parameters.

Original: When the mask can be kept small (for example while extracting localized features), the convolution layer is highly efﬁcient and effective.
Revised: When the mask can be kept small (for example while extracting localized features), the convolution layer is highly efﬁcient and effective.

Original: A different way for computing a convolution is to use the convolutional theorem as follows: (1) compute the Fourier transform of the input, (2) multiply with the Fourier transform of the mask, i.e.m the Fourier multiplier, and (3) inverse Fourier transform back.
Revised: A different way for computing a convolution is to use the convolutional theorem as follows: (1) compute the Fourier transform of the input, (2) multiply with the Fourier transform of the mask, i.e.m the Fourier multiplier, and (3) inverse Fourier transform back.

Original: In this case, the trainable parameters are the DOFs of the Fourier multipliers and the Fourier transforms are computed using the fast Fourier transform (FFT).
Revised: In this case, the trainable parameters are the DOFs of the Fourier multipliers and the Fourier transforms are computed using the fast Fourier transform (FFT).

Original: This alternative approach is particularly attractive for smooth kernels with large support (i.e., smooth long-range interactions) because the computational cost does not increase with the size of the mask.
Revised: This alternative approach is particularly attractive for smooth kernels with large support (i.e., smooth long-range interactions) because the computational cost does not increase with the size of the mask.

Original: To the best of our knowledge, this direction has not been explored for LRIs and below we detail now to apply this to point clouds.
Revised: To the best of our knowledge, this direction has not been explored for LRIs and below we detail now to apply this to point clouds.

Original: Given a point cloud { x i } Ni =1 ⊂ R d and scalar weights { f i } Ni =1 , we consider the problem of computing the quantity u i :=
Revised: Given a point cloud { x i } Ni =1 ⊂ R d and scalar weights { f i } Ni =1 , we consider the problem of computing the quantity u i :=

Original: (cid:80)Nj =
Revised: (cid:80)

Original: 1 φ θ ( x i − x j ) f j for each  i .
Revised: Nj =1 φ θ ( x i − x j ) f j at each x i .

Original: Here the function φ θ ( · ) is the kernel with a generic trainable parameter θ .
Revised: Here the function φ θ ( · ) is the kernel with a generic trainable parameter θ .

Original: At ﬁrst glance the cost of this operation scales as O ( N 2 ) : we need to evaluate u i for each point x i , which requires O ( N ) work per evaluation.
Revised: At ﬁrst glance the cost of this operation scales as O ( N 2 ) : we need to evaluate u i for each point x i , which requires O ( N ) work per evaluation.

Original: By introducing a generalized function f ( y ) =
Revised: By introducing a generalized function f ( y ) =

Original: (cid:80) i f i · δ ( y − x i ) and deﬁning a function u ( x ) =
Revised: (cid:80) i f i · δ ( y − x i ) and deﬁning a function u ( x ) =

Original: (cid:82) φ θ ( x − y )
Revised: (cid:82) φ θ ( x − y )

Original: f ( y ) dy , one notices that u i is the value of u ( x ) at x = x i .
Revised: f ( y ) dy , one notices that u i is the value of u ( x ) at x = x i .

Original: The advantage of this viewpoint is that one can now invoke the connection between convolution and Fourier transform
Revised: The advantage of this viewpoint is that one can now invoke the connection between convolution and Fourier transform

Original: [Equation]
Revised: [Equation]

Original: where ˆ φ θ ( k ) is a trainable Fourier multiplier.
Revised: where ˆ φ θ ( k ) is a trainable Fourier multiplier.

Original: This approach is suitable for point clouds since the trainable parameters are decoupled from the geometry of the point cloud.
Revised: This approach is suitable for point clouds since the trainable parameters are decoupled from the geometry of the point cloud.

Original: To make this approach practical, one needs to address two issues: (1) the non-uniform distribution of the point cloud and (2) how to represent the multiplier ˆ φ θ ( k ) .
Revised: To make this approach practical, one needs to address two issues: (1) the non-uniform distribution of the point cloud and (2) how to represent the multiplier ˆ φ θ ( k ) .

Original: Non-uniform distribution of the point cloud Equation 1 suggests that one can compute the convolution directly using the convolution theorem, which typically relies on the FFT to obtain a low-complexity algorithm.
Revised: Non-uniform distribution of the point cloud Equation 1 suggests that one can compute the convolution directly using the convolution theorem, which typically relies on the FFT to obtain a low-complexity algorithm.

Original: Unfortunately, { x i } Ni =1 do not form a regular grid, thus FFT can not be directly used.
Revised: Unfortunately, { x i } N i =1 do not form a regular grid, thus FFT can not be directly used.

Original: We overcome this difﬁculty by invoking the NUFFT 1 (Dutt & Rokhlin, 1993), which serves as the corner-stone of  the LRC-layer.
Revised: We overcome this difﬁculty by invoking the NUFFT 1 (Dutt & Rokhlin, 1993), which serves as the corner-stone of our instance of the LRC-layer 2 .

Original: Algorithm 1 Long-range convolutional layer
Revised: We point out, one could in practice use an fast summation algorithm, such as the fast multipole method (FMM) introduced by Greengard & Rokhlin (1987), to evaluate u i . This would results in the same complexity if

Original: Input: { x i } Ni =1 , { f i } Ni =
Revised: Input: { x i } Ni =1 , { f i } Ni =

Original: Output:  u i = (cid:80) Nj [-=
-]
Revised: Output: { x i } Ni =1 , { u i } Ni =1 , where u i = (cid:80) Nj [-=
-]= [Equation]

Original: f
Revised: f

Original: j φ θ ( x i − xj ) , for i
Revised: [Equation] [Equation] x

Original:  = 1 , ..., N .{+(cid:80) Nj =
+}
Revised: [Equation] 1: Deﬁne the generalized function: f ( x ) = {+(cid:80) Nj =
+}

Original: 1: Deﬁne x f j  ( x − x j ) : Mollify  f 
Revised: f j δ ( x − x j ) 2: Mollify the Dirac f x

Original: ( x ) = (cid:80) Nj =
Revised: (cid:80) Nj =

Original: f j g τ ( x − x j ) , g is deﬁned 
Revised: f j g τ ( x − x j ) , g is deﬁned in

Original: Appendix B.

Original: The LRC-layer is summarized in Alg.
Revised: The LRC-layer is summarized in Alg.

Original: 1, where τ is chosen following Dutt & Rokhlin (1993).
Revised: 1, where τ is chosen following Dutt & Rokhlin (1993).

Original: The inputs of this layer are the point cloud { x i } Ni =1 and the corresponding weights { f i } Ni =1 .The outputs
Revised: The inputs of this layer are the point cloud { x i } Ni =1 and the corresponding weights { f i } Ni =1 .

Original: Appendix B.2 for further details.

Original:  are u i ≡ u ( x i ) for i = 1 , ..., N .
Revised: The outputs are u i ≡ u ( x i ) for i = 1 , ..., N .

Original: The number of elements in the underlying grid N FFT = L d FFT is chosen such that the kernel is adequately sampled and the complexity remains low. 
Revised: The number of elements in the underlying grid N FFT = L d FFT is chosen such that the kernel is adequately sampled and the complexity remains low. As shown in Appendix C.5, one only needs a relatively small L FFT .

Revised: Even though the precise number is problemspeciﬁc, given that the goal is to approximate LRIs that are supposedly smooth, it can be captured with a relatively small number of Fourier modes.

Original: The LRC-layer is composed of three steps: (1) It computes the Fourier transform from the point cloud to a regular grid using the NUFFT algorithm (lines 2 − 5 in Alg. 1).
Revised: The LRC-layer is composed of three steps: (1) It computes the Fourier transform from the point cloud to a regular grid using the NUFFT algorithm (lines 2 − 5 in Alg. 1 and showcased in Fig. 2).

Original: (2) It multiplies the result by a set of trainable Fourier multipliers (line 6 in Alg. 1).
Revised: (2) It multiplies the result by a set of trainable Fourier multipliers (line 6 in Alg. 1).

Original: (3) It computes the inverse Fourier transform from the regular grid back to the point cloud (lines 7 − 9 in Alg. 1).
Revised: (3) It computes the inverse Fourier transform from the regular grid back to the point cloud (lines 7 − 9 in Alg. 1).

Original: Within the LRC-layer in Alg. 1, the only trainable component is the parameter θ of the Fourier multiplier ˆ φ θ ( k ) .
Revised: Within the LRC-layer in Alg. 1, the only trainable component is the parameter θ of the Fourier multiplier ˆ φ θ ( k ) .

Original: The remaining components, including the molliﬁer g τ ( · ) and the Cartesian grid size, are taken to be ﬁxed.
Revised: The remaining components, including the molliﬁer g τ ( · ) and the Cartesian grid size, are taken to be ﬁxed.

Original: One can in principle train them as well, but it comes with a much higher cost.
Revised: One can, in principle, train them as well, but it comes with a much higher cost.

Original: Among the steps of Alg. 1, the sampling operator, the rescaling operator, the interpolation operator, and the Fourier transforms, are all linear and non-trainable .
Revised: Among the steps of Alg. 1, the sampling operator, the rescaling operator, the interpolation operator, and the Fourier transforms, are all linear and non-trainable .

Original: Therefore, derivative computations of backpropagation just go through them directly.
Revised: Therefore, derivative computations of backpropagation just go through them directly.

Original: Alg. 1 is presented in terms of only one single channel or feature dimension, i.e., f j ∈ R and u i ∈ R .
Revised: Alg. 1 is presented in terms of only one single channel or feature dimension, i.e., f j ∈ R and u i ∈ R .

Original: However, it can be easily generalized to multiple channels, for example f j ∈ R d 1 and u
Revised: However, it can be easily generalized to multiple channels, for example f j ∈ R d 1 and u

Original: i ∈ R d 2 .
Revised: i ∈ R d 2 .

Original: In this case, the Fourier multiplier ˆ φ θ ( k ) at each point k is a d 2 × d 1 matrix, and all Fourier transforms are applied component-wise.
Revised: In this case, the Fourier multiplier ˆ φ θ ( k ) at each point k is a d 2 × d 1 matrix, and all Fourier transforms are applied component-wise.

Original: Representation of the Fourier multiplier
Revised: Representation of the Fourier multiplier

Original: A useful feature of the LRC-layer is that it is quite easy to impose symmetries on the Fourier multipliers.
Revised: A useful feature of the LRC-layer is that it is quite easy to impose symmetries on the Fourier multipliers.

Original: For example, if the convolution kernel φ θ ( · ) is constrained to have parity symmetry, rotational symmetry, smoothness or decay properties, these constraints can be imposed accordingly on the coefﬁcients of the Fourier multipliers ˆ φ θ ( k ) .
Revised: For example, if the convolution kernel φ θ ( · ) is constrained to have parity symmetry, rotational symmetry, smoothness or decay properties, these constraints can be imposed accordingly on the coefﬁcients of the Fourier multipliers ˆ φ θ ( k ) .

Original: When the size of the training data is limited, it is often necessary to reduce the number of trainable parameters in order to regularize the kernel.
Revised: When the size of the training data is limited, it is often necessary to reduce the number of trainable parameters in order to regularize the kernel.

Original: For example, we may parametrize the Fourier multiplier as a linear combination of several predetermined functions on the Fourier grid.
Revised: For example, we may parameterize the Fourier multiplier as a linear combination of several predetermined functions on the Fourier grid.

Original: This is the procedure used in molecular modeling (Grisaﬁ & Ceriotti, 2019; Yao et al., 2018;
Revised: This is the procedure used in molecular modeling (Grisaﬁ & Ceriotti, 2019; Yao et al., 2018;

Original: Ko et al.), and also in our numerical examples in equation 7.
Revised: Ko et al., 2009), and also in our numerical examples in equation 7.

Original: We also remark that the LRC-layer described here can be applied to point clouds a way similar to a standard convolution layer applied to images and multiple LRC-layers can be composed on top of each other. 
Revised: We also remark that the LRC-layer described here can be applied to point clouds a way similar to a standard convolution layer applied to images and multiple LRC-layers can be composed on top of each other. the kernel is ﬁxed.

Revised: However, in order for the kernel to be trainable, this would require a different algorithm for each

Revised: , including the computation of the derivatives, thus increasing the computational cost and rendering the implementation more

Original: L EARNING THE N - BODY POTENTIAL
Revised: [Table] J/XTU1Y9adcJqlByZaLwlQQE5P512TAFTIjJpZQpri9lbARVZQZm03BhuCtvrxOWldlr1K [Table] blRKtWoWRx7O4BwuwYMbqMEd1KEJDBCe4RXenAfnxXl3PpatOSebOYU/

Revised: [Table]

Revised: [Table]

Revised: =</latexit> {

Revised: [Equation]

Revised: <latexit sha1_

Revised: [Table]

Revised: ="

Revised: [Table]

Revised: +

Revised: [Table]

Revised: =">AAACHHicbVDLSgMxFM34rPU16tJNsAgVpMxoxW6EghtXUsE

Revised: [Table]

Revised: oFNLJs20oZkHyR2xDPMhbvwVNy4UceNC8G/

Revised: [Table]

Revised: [Table]

Revised: +Or2Qxj4LgAqiVNu2IugkRAKngqV5J1YsInRI+qytaUB8pjrJ+LgUH2qlh71Q6hcAHqu/JxLiKzXyXd3pExioWS8T//PaMXiVTsKDKAYW0MkiLxYYQpwlhXtcMgpipAmhkuu/YjogklDQeWYh2LMnz5PGSckul85uyoVqZRpHDu2jA1RENjpHVXSFaqiOKHpEz+gVvRlPxovxbnxMWheM6cwe+lPG1w9426Iw</latexit> f

Revised: [Equation]

Revised: N

Revised: [Equation]

Revised: <

Revised: [Table]

Revised: [Table]

Revised: </

Revised: [Table]

Revised: > f ⌧ ( x ` ) =

Revised: N

Revised: X j = g ⌧ ( x `  x j ) <latexit sha1_base64="xBA9/PTg3eHQ5avpSkVxfJfaLOc=">AAACJXicbVDLSgMxFM3UV62vUZdugkVoF5YZqdiFhYIbV1LBPqAzDpk006bNPEgy0jLMz7jxV9y4sIjgyl8xfSy09UDgcM653NzjRowKaRhfWmZtfWNzK7ud29nd2z/QD4+

Revised: [Table]

Revised: ">AAACIHicbVDLSgMxFM34rPVVdekmWIS6KTNF0Y1QFIrLCrYVOrVk0kwbmnmY3BFKmE9x46+4caGI7vRrTNtZaOuBwMk595Dc48WCK7DtL2thcWl5ZTW3ll/f2NzaLuzsNlWUSMoaNBKRvPWIYoKHrAEcBLuNJSOBJ1jLG16O/dYDk4pH4Q2MYtYJSD/kPqcEjNQtnNZKw6NzV91L0K4vCdVuzFPtAknSFLM7PbzTlRRP7rjWnRom0i0U7bI9AZ4nTkaKKEO9W/h0exFNAhYCFUSptmPH0NFEAqeCpXk3USwmdEj6rG1oSAKmOnqyYIoPjdLDfiTNCQFP1N8JTQKlRoFnJgMCAzXrjcX/vHYC/llH8zBOgIV0+pCfCAwRHreFe1wyCmJkCKGSm79iOiCmJjCd5k0JzuzK86RZKTvH5ZPr42L1Iqsjh/bRASohB52iKrpCddRAFD2iZ/SK3qwn68V6tz6mowtWltlDf2B9/wBJ3KOs

Revised: /latexit>

Revised: Finally in order to obtain F ( k ) , we rescale the signal to undo the spatial convolution.

Original: To demonstrate the effectiveness of the LRC-layer, we consider the problem of learning the energy and force associated with a model N -body potential in the context of molecular modelling.
Revised: To demonstrate the effectiveness of the LRC-layer, we consider the problem of learning the energy and force associated with a model N -body potential in the context of molecular modeling.

Original: As mentioned in Section 1, the potential evaluation often invokes expensive ab-initio calculations that one would like to bypass for efﬁciency reasons.
Revised: As mentioned in Section 1, the potential evaluation often invokes expensive ab-initio calculations that one would like to bypass for efﬁciency reasons.

Original: The setup of this learning problem is as follows.
Revised: The setup of this learning problem is as follows.

Original: First, we assume access to a black-box model potential, which consists of both short- and long-range interactions.
Revised: First, we assume access to a black-box model potential, which consists of both short- and long-range interactions.

Original: However, internal parameters of the potential are inaccessible to the training architecture and algorithm.
Revised: However, internal parameters of the potential are inaccessible to the training architecture and algorithm.

Original: A set of training samples are generated by the model, where each sample consists of a conﬁguration of the points { x i } along with the potential and force.
Revised: A set of training samples are generated by the model, where each sample consists of a conﬁguration of the points { x i } along with the potential and force.

Original: Second, we set up a deep neural network that includes (among other components) the LRC-layer for addressing the long-range interaction.
Revised: Second, we set up a deep neural network that includes (among other components) the LRC-layer for addressing the long-range interaction.

Original: This network is trained with stochastic gradient type of algorithms using the collected dataset and the trained network can be used for predicting the potential and forces for new point cloud conﬁgurations.
Revised: This network is trained with stochastic gradient type of algorithms using the collected dataset and the trained network can be used for predicting the potential and forces for new point cloud conﬁgurations.

Original: These two components are described in the following two subsections in detail.
Revised: These two components are described in the following two subsections in detail.

Original: 3. M ODEL PROBLEM AND D ATA G ENERATION
Revised: 3. M ODEL PROBLEM AND D ATA G ENERATION

Original: Model We suppose that Ω = [0 , L ] d , and we denote the point cloud by x = { x i } Ni =1 ⊂ Ω ⊂ R d , for d = 1 , 2 , or 3 .
Revised: Model We suppose that Ω = [0 , L ] d , and we denote the point cloud by x = { x i } Ni =1 ⊂ Ω ⊂ R d , for d = 1 , 2 , or 3 .

Original: We deﬁne the total energy, the local potential and the forces acting on particle j by
Revised: We deﬁne the total energy, the local potential and the forces acting on particle j by

Original: U =(cid:88) i [Equation] j ≤
Revised: U =

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: respectively, where the interaction kernel ψ ( r ) is a smooth function, besides a possible singularity at the origin and decreases as (cid:107) r (cid:107) → ∞ .
Revised: respectively, where the interaction kernel ψ ( r ) is a smooth function, besides a possible singularity at the origin and decreases as (cid:107) r (cid:107) → ∞ .

Original: Sampling We deﬁne a snapshot as one conﬁguration 2 of particles, x (cid:96) =
Revised: Sampling We deﬁne a snapshot as one conﬁguration 3 of particles, x (cid:96) =

Original: { x [ (cid:96) ] j } Nj =1 , together with the global energy U
Revised: { x [ (cid:96) ] j } Nj =1 , together with the global energy U

Original: [ (cid:96) ] and the forces F [ (cid:96) ] , where (cid:96) is the index representing the number in the training/testing set.
Revised: [ (cid:96) ] and the forces F [ (cid:96) ] , where (cid:96) is the index representing the number in the training/testing set.

Original: We sample the conﬁguration of particles x (cid:96) randomly, with the restriction that two particles can not be closer than a predetermined value δ min in order to avoid the singularity.
Revised: We sample the conﬁguration of particles x (cid:96) randomly, with the restriction that two particles can not be closer than a predetermined value δ min in order to avoid the singularity.

Original: After an admissible conﬁguration is computed we generate the energy and forces following Appendix A.
Revised: After an admissible conﬁguration is computed we generate the energy and forces following Appendix B.

Original: This process is repeated until obtaining N sample snapshots.
Revised: This process is repeated until obtaining N sample snapshots.

Original: 3. A RCHITECTURE
Revised: 3. A RCHITECTURE

Original: Our network architecture consists of separate descriptors for the short- interactions and long-range interactions, respectively.
Revised: Our network architecture consists of separate descriptors for the short- interactions and long-range interactions, respectively.

Original: To capture the short-range interaction, we compute a local convolution using for each point only its neighboring points within a ball of predetermined radius.
Revised: To capture the short-range interaction, we compute a local convolution using for each point only its neighboring points within a ball of predetermined radius.

Original: For the long-range interactions, we compute an all-to-all convolution using the LRC-layer introduced in
Revised: For the long-range interactions, we compute an all-to-all convolution using the LRC-layer introduced in

Original: Section 2, whose output is distributed to each particle and then fed to a sequence of subsequent layers.
Revised: Section 2, whose output is distributed to each particle and then fed to a sequence of subsequent layers.

Original: Short-range descriptor For a given particle x i , and an interaction radius R , we deﬁne I i , the interaction list of x i , as the indices j such that (cid:107) x
Revised: Short-range descriptor For a given particle x i , and an interaction radius R , we deﬁne I i , the interaction list of x i , as the indices j such that (cid:107) x

Original: i − x j (cid:107) <
Revised: i − x j (cid:107) <

Original: R , i.e., the indices of the particles that are inside a ball of radius R centered at x i .
Revised: R , i.e., the indices of the particles that are inside a ball of radius R centered at x i .

Original: Thus for each particle x i we build the generalized coordinates
Revised: Thus for each particle x i we build the generalized coordinates

Original: s i,j
Revised: s i,j

Original: = x i − x j , and the short-range descriptor
Revised: = x i − x j , and the short-range descriptor

Original: [Equation]
Revised: [Equation]

Original: where f θ : R d → R m sr is a function represented by a neural network speciﬁed in Appendix B.1, where m sr is the number of short-range features.
Revised: where f θ : R d → R m sr is a function represented by a neural network speciﬁed in Appendix C.1, where m sr is the number of short-range features.

Original: By construction f θ ( s ) is smooth and it satisﬁes f θ ( s ) = 0 for (cid:107) s (cid:107) >
Revised: By construction f θ ( s ) is smooth and it satisﬁes f θ ( s ) = 0 for (cid:107) s (cid:107) >

Original: R .
Revised: R .

Original: Long-range descriptor We feed the LRC-layer with the raw point cloud represented by { x i } Ni = with weights { f i } Ni =1 , which for simplicity can be assumed to be equal to one here, i.e., f i = 1 for i = 1 , ..., N .
Revised: Long-range descriptor We feed the LRC-layer with the raw point cloud represented by { x i } Ni = with weights { f i } Ni =1 , which for simplicity can be assumed to be equal to one here, i.e., f i = 1 for i = 1 , ..., N .

Original: The output of the layer is a two-dimensional tensor u k ( x i ) with i = 1 , . . .
Revised: The output of the layer is a two-dimensional tensor u k ( x i ) with i = 1 , . . .

Original: , N and k
Revised: , N and k

Original: = 1 , . .
Revised: = 1 , . .

Original: , K chnls .
Revised: , K chnls .

Original: Then for each x i , its corresponding slice given by the vector [ u 1 ( x i ) , u 2 ( x i ) , · · · , u K chnls ( x i )] , is fed to a function g θ : R K chnls → R m lr , which is represented by a neural network with non-linear activation functions.
Revised: Then for each x i , its corresponding slice given by the vector [ u 1 ( x i ) , u 2 ( x i ) , · · · , u K chnls ( x i )] , is fed to a function g θ : R K chnls → R m lr , which is represented by a neural network with non-linear activation functions.

Original: Here θ is a generic set of trainable parameters and m lr is the number of long-range features.
Revised: Here θ is a generic set of trainable parameters and m lr is the number of long-range features.

Original: The descriptor for particle x i , which depends on all the other particles thanks to the LRC-layer, is deﬁned by
Revised: The descriptor for particle x i , which depends on all the other particles thanks to the LRC-layer, is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: Short-range network When only the short-range interaction is present, the short-range descriptor for each particle is fed particle-wise to a ﬁtting network F sr : R m sr
Revised: Short-range network When only the short-range interaction is present, the short-range descriptor for each particle is fed particle-wise to a ﬁtting network F sr : R m sr

Original: → R .
Revised: → R .

Original: In this case F sr ( D i sr ) only depends on particle
Revised: In this case F sr ( D i sr ) only depends on particle

Original: x i and its neighbors.
Revised: x i and its neighbors.

Original: Finally, the contributions from each particle are accumulated so the short-range neural network (NN) energy and forces are given by[Equation] and
Revised: Finally, the contributions from each particle are accumulated so the short-range neural network (NN) energy and forces are given by

Original: [Equation]
Revised: [Equation]

Original: respectively (see Fig. 1(left)).
Revised: respectively (see Fig. 2(left)).

Original: The derivatives are computed using Tensorﬂow (Abadi et al., 2015) directly.
Revised: The derivatives are computed using Tensorﬂow (Abadi et al., 2015) directly.

Original: This network, as shown by (Zhang et al., 2018b) is rotation, translation and permutation invariant (Zaheer et al., 2017).
Revised: This network as shown by (Zhang et al., 2018b) is rotation, translation, and permutation invariant (Zaheer et al., 2017).

Revised: We point out that this architecture can be understood as a non-linear local convolution: for each particle i one applies the same function f θ to each of its neighbors.

Revised: The result is then pooled into the descriptor D

Revised: i sr , then processed locally by F sr (akin to a non-linear convolution with a ﬁlter of width one), and ﬁnally pooled globally into U NNsr .

Original: Full-range network When both the short-range and long-range interactions are present, the long range descriptor and the local descriptor are combined and fed particle-wise to a ﬁtting network
Revised: Full-range network When both the short-range and long-range interactions are present, the long range descriptor and the local descriptor are combined and fed particle-wise to a ﬁtting network

Original: F : R m sr
Revised: F : R m sr

Original: + m lr → R to produce the overall neural network (NN) energy and forces
Revised: + m lr → R to produce the overall neural network (NN) energy and forces

Original: [Equation]
Revised: [Equation]

Original:  and
Revised: , and

Original: [Equation]
Revised: [Equation]

Original: respectively (see Fig. 1(right)).
Revised: respectively (see Fig. 2(right)).

Original: Following Section 2, the long-range descriptor is translation invariant by design and can be easily made rotation invariant.
Revised: Following Section 2, the long-range descriptor is translation invariant by design and can be easily made rotation invariant.

Original: Furthermore, it is well known (Zaheer et al., 2017) that this construction is permutation invariant.
Revised: Furthermore, it is well known (Zaheer et al., 2017) that this construction is permutation invariant.

Original: Further details on the implementation of the network can be found in Appendix B.3.
Revised: Further details on the implementation of the network can be found in Appendix C.3.

Original: From the structures shown in Fig.
Revised: From the structures shown in Fig.

Original: 1 3 , it is clear that we can recover the ﬁrst architecture from the second, by zeroing some entries at the ﬁtting network, and removing the LRC-layer.
Revised: 2 4 , it is clear that we can recover the ﬁrst architecture from the second, by zeroing some entries at the ﬁtting network, and removing the LRC-layer.

Original: Finally, let us comment on the inference complexity of the proposed network where, for simplicity we assume that O ( K chnls )
Revised: Finally, let us comment on the inference complexity of the proposed network where, for simplicity we assume that O ( K chnls )

Original: =
Revised: =

Original: O ( m sr )
Revised: O ( m sr )

Original: =
Revised: =

Original: O ( m lr )
Revised: O ( m lr )

Original: =
Revised: =

Original: O (1) , and that the depth of the neural networks is O (1) .
Revised: O (1) , and that the depth of the neural networks is O (1) .

Original: The cost for computing U NNsr is O ( N ) , provided that each particle has a bounded number of neighbors.
Revised: The cost for computing U NNsr is O ( N ) , provided that each particle has a bounded number of neighbors.

Original: The complexity for computing the forces also scales linearly in N , albeit with higher constants.
Revised: The complexity for computing the forces also scales linearly in N , albeit with higher constants.

Original: The complexity of computing both U NN and associated forces 4 is O ( N + N FFT log N FFT ) .
Revised: The complexity of computing both U NN and associated forces 5 is O ( N + N FFT log N FFT ) .

Original:  N UMERICAL RESULTS
Revised:  N UMERICAL RESULTS

Original: The loss function is the mean squared error of the forces
Revised: The loss function is the mean squared error of the forces

Original: [Equation]
Revised: [Equation]

Original: where the i -index runs on the points of each snapshot, and (cid:96) runs on the test samples.
Revised: where the i -index runs on the points of each snapshot, and (cid:96) runs on the test samples.

Original: We also generate 100 snapshots of data to test the performance of network.
Revised: We also generate 100 snapshots of data to test the performance of network.

Original: This particular loss could lead to shift the potential energy by up to a global constant, which can be subsequently ﬁxed by including the error of the energy in the loss (Zhang et al., 2018b).
Revised: This particular loss could lead to shift the potential energy by up to a global constant, which can be subsequently ﬁxed by including the error of the energy in the loss (Zhang et al., 2018b).

Original: For the testing stage of we use the relative (cid:96)error of the forces as metric, which is deﬁned as (cid:15) rel 
Revised: For the testing stage of we use the relative (cid:96)error of the forces as metric, which is deﬁned as (cid:15) rel :=

Original: [Equation]
Revised: [Equation]

Original: The standard training parameters are listed in appendix B.4.
Revised: The standard training parameters are listed in Appendix C.4.

Revised: The experiments shown in the sequel are designed to provide a fair comparison with state-of-the-art methods for localized interactions.

Original:  The kernels ψ used in the experiment typically exhibit two interaction lengths: ψ ( · ) ≡ α 1 ψ µ 1 ( · ) + α 2 ψ µ 2 ( · ) , where each of ψ µ 1 and ψ µ 2 is either a simple exponential kernel or screened-Coulomb kernel (also known as the Yukawa kernel).
Revised: They showcase that, by adding a single LRC-layer, one can outperform these methods signiﬁcantly. The kernels ψ used in the experiment typically exhibit two interaction lengths: ψ ( · ) ≡ α 1 ψ µ 1 ( · ) + α 2 ψ µ 2 ( · ) , where each of ψ µ 1 and ψ µ 2 is either a simple exponential kernel or screened-Coulomb kernel (also known as the Yukawa kernel).

Original: For each of ψ µ 1 and ψ µ 2 , the superscripts denote the reciprocal of the interaction length, i.e., length scale ∼ µ − 1 1 or ∼ µ − 1 2 .
Revised: For each of ψ µ 1 and ψ µ 2 , the superscripts denote the reciprocal of the interaction length, i.e., length scale ∼ µ − 1 1 or ∼ µ − 1 2 .

Original: Without loss of generality, µ 1 > µ 2 , so that µ 1 corresponds to the short-range scale and µ 2 the long-range scale.
Revised: Without loss of generality, µ 1 > µ 2 , so that µ 1 corresponds to the short-range scale and µ 2 the long-range scale.

Original: We also assume that 0 ≤ α 2 ≤ α 1 and α 1 + α 2 = 1 , so that the effect of the long-range interaction can be smaller in magnitude compared to that of the short-range interaction.
Revised: We also assume that 0 ≤ α 2 ≤ α 1 and α 1 + α 2 = 1 , so that the effect of the long-range interaction can be smaller in magnitude compared to that of the short-range interaction.

Original: In the special case of α 2 = 0 , the kernel exhibits only a single scale ∼ µ − 1 1 .
Revised: In the special case of α 2 = 0 , the kernel exhibits only a single scale ∼ µ − 1 1 .

Original: The precise deﬁnition of the kernel depends on the spatial dimension and boundary conditions, which are explained in Appendix A.
Revised: The precise deﬁnition of the kernel depends on the spatial dimension and boundary conditions, which are explained in Appendix B.

Original: For a ﬁxed set of kernel parameters ( µ 1 , µ 2 , α 1 , α 2 ), we consider two types of data: large- and small-scale data, generated in the domains Ω lr and Ω sr respectively (details to be deﬁned in each experiment).
Revised: For a ﬁxed set of kernel parameters ( µ 1 , µ 2 , α 1 , α 2 ), we consider two types of data: large- and small-scale data, generated in the domains Ω lr and Ω sr respectively (details to be deﬁned in each experiment).

Original: The Fourier multiplier within the LRC-layer is parametrized as
Revised: The Fourier multiplier within the LRC-layer is parameterized as

Original: [Equation]
Revised: [Equation]

Original: where β and λ are trainable parameters.
Revised: where β and λ are trainable parameters.

Original: This is a simple parameterization, and a more complex model can be used as well with minimal changes to the procedure.
Revised: This is a simple parameterization, and a more complex model can be used as well with minimal changes to the procedure.

Original: For all experiments shown below, two kernel channels are used and as a result there are only four trainable parameters in the LRC-layer.
Revised: For all experiments shown below, two kernel channels are used and as a result there are only four trainable parameters in the LRC-layer.

Original: The numerical results aim to show namely two properties: i) the LRC-layer is able to efﬁciently capture LRIs, and ii) the two-scale training strategy can reduce the amount of large-scale data signiﬁcantly.
Revised: The numerical results aim to show namely two properties: i) the LRC-layer is able to efﬁciently capture LRIs, and ii) the two-scale training strategy can reduce the amount of large-scale data signiﬁcantly.

Original: To demonstrate the ﬁrst property, we gradually increase the interaction length of the kernel.
Revised: To demonstrate the ﬁrst property, we gradually increase the interaction length of the kernel.

Original: The accuracy of the short-range network with a ﬁxed interaction radius is supposed to decrease rapidly, while using the LRC-layer improves the accuracy signiﬁcantly.
Revised: The accuracy of the short-range network with a ﬁxed interaction radius is supposed to decrease rapidly, while using the LRC-layer improves the accuracy signiﬁcantly.

Original: To show the second property, we generate data with two interaction lengths and train the full-range network using the one- and  two-scale strategies.
Revised: To show the second property, we generate data with two interaction lengths and train the full-range network using the one- and [Table] two-scale strategies.

Original: Finally, we also aim to demonstrate that the LRC-layer is competitive against a direct convolution in which the all-to-all computation is performed explicitly.[Table]
Revised: Finally, we also aim to demonstrate that the LRC-layer is competitive against a direct convolution in which the all-to-all computation is performed explicitly.

Original: 1D In the ﬁrst set of experiments, the domain Ω = [0 , 5] , N = 20 and N sample 
Revised: 1D In the ﬁrst set of experiments, the domain Ω = [0 , 5] , N = 20 and N sample = 1000 , where

Original: = 1000 .
Revised: N sample is the number of snapshots and N is the total number of points in each snapshot.

Original: For the kernel, we set α 2 and vary µ 1 to generate datasets at different interaction lengths.
Revised: For the kernel, we set α 2 and vary µ 1 to generate datasets at different interaction lengths.

Original: For each dataset we train both short-range and full-range networks using the one-scale data.
Revised: For each dataset we train both short-range and full-range networks using the one-scale data.

Original: The results are summarized in Table 1, where we can observe that as the characteristic interaction length increases, the accuracy of the short-range network decreases while using the full-range network can restore the accuracy.
Revised: The results are summarized in Table 1, where we can observe that as the characteristic interaction length increases, the accuracy of the short-range network decreases while using the full-range network can restore the accuracy.

Revised: This experiment shows that local networks are often highly accurate when the interactions are localized, but the accuracy quickly deteriorates as the interaction length increases (i.e. as µ 1 decreases).

Original: For the second set of experiments we used two sets of kernel parameters: one heavily biased towards a localized interaction length, and another in which both interaction lengths are equally weighted.
Revised: For the second set of experiments we used two sets of kernel parameters: one heavily biased towards a localized interaction length, and another in which both interaction lengths are equally weighted.

Original: For each set of kernel parameters, we generate 10 , 000 small-scale snapshots using Ω sr =
Revised: For each set of kernel parameters, we generate 10 , 000 small-scale snapshots using Ω sr =

Original: [0 , 5] and
Revised: [0 , 5] and

Original: N = 20 , and a large number of large-scale snapshots using Ω lr = [0 , 50] and N = 200 particles.
Revised: N = 20 , and a large number of large-scale snapshots using Ω lr = [0 , 50] and N = 200 particles.

Original: The interaction radius R = 1 . 5 , δ min = 0 .
Revised: The interaction radius R = 1 . 5 , δ min = 0 .

Original: 05 , and N FFT is 501 .
Revised: 05 , and N FFT is 501 .

Original: We train the network with the oneand two-scale training strategies described in the prequel.
Revised: We train the network with the oneand two-scale training strategies described in the prequel.

Original: Fig.
Revised: Fig.

Original: 2 (left) depicts the advantage of using the two-scale training strategy: we obtain roughly the same accuracy at a fraction of the number of large-scale training samples.
Revised: 3 (left) depicts the advantage of using the two-scale training strategy: we obtain roughly the same accuracy at a fraction of the number of large-scale training samples.

Revised: We observe that when the number of large-scale training samples is sufﬁciently large, the resulting test accuracy is independent of the training strategy.

Revised: We also observe that the training dynamics is stable with respect to different random seeds.

Original: We compare the LRC-layer with a direct all-to-all computation.
Revised: We compare the LRC-layer with a direct all-to-all computation.

Original: We benchmark the wall time of both layers, with increasingly number of particles.
Revised: We benchmark the wall time of both layers, with increasingly number of particles.

Original: To account for implementation effects we normalize the wall times in Fig.
Revised: To account for implementation effects we normalize the wall times in Fig.

Original: 2 (right) and the results corroborate the complexity claims made in Section 2.
Revised: 3 (right) and the results corroborate the complexity claims made in Section 2.

Original: 2D We perform the same experiments as in the one-dimensional case.
Revised: 2D We perform the same experiments as in the one-dimensional case.

Original: We ﬁx Ω = [0 , 15] 2 , N =and N sample = 10000 .
Revised: We ﬁx Ω = [0 , 15] 2 , N =and N sample = 10000 .

Original: The results are summarized in Table 2, which shows that as µ decreases, the full-range network outperforms the short-range one.[Table]
Revised: The results are summarized in Table 2, which shows that as µ decreases, the full-range network outperforms the short-range one.

Original: For the second set of experiments, R = 1 . 5 , δ min = 0 .
Revised: For the second set of experiments, R = 1 . 5 , δ min = 0 .

Original: 05 , and N FFT is 31 2 .
Revised: 05 , and N FFT is 31 2 .

Original: For the small-scale data, Ω sr =
Revised: For the small-scale data, Ω sr =

Original: [0 , 3] 2 , N = 18 , and N sample = 10 , 000 .
Revised: [0 , 3] 2 , N = 18 , and N sample = 10 , 000 .

Original: For the large-scale data, Ω lr = [0 , 15] 2 ,
Revised: For the large-scale data, Ω lr = [0 , 15] 2 ,

Original: N = 450 .
Revised: N = 450 .

Original: Similarly to the 1 D case, we train the networks with both strategies using different amounts of large-scale data.
Revised: Similarly to the 1 D case, we train the networks with both strategies using different amounts of large-scale data.

Original: The results summarized in Fig.
Revised: The results summarized in Fig.

Original: 3 show that the two-scale strategy efﬁciently captures the long-range interactions with only a small number of the long-range data.
Revised: 4 show that the two-scale strategy efﬁciently captures the long-range interactions with only a small number of the long-range training samples.

Original: [Table]
Revised: [Table]

Original: 3D We choose Ω =
Revised: Analogously to the 1 D case, we can observe that for sufﬁciently large-scale training samples the resulting test accuracy is identical regardless of the training strategy used. Also similar to Fig. 3 (left), we ﬁnd that the lowest achievable test error is larger in Fig. 4 (right, with a larger α 2 ) than that in

Revised: Fig.

Original:  [0 , 3] 3 with27 cells 2 points each.
Revised: 4 (left, with a smaller α 2 ). Nonetheless, we observe that the test error of the two-scale training strategy becomes less sensitive with respect to the number of training samples when α 2 becomes larger, i.e. the LRI becomes more prominent. 3D The domain Ω is [0 , 3] 3 with 2 points in each of the 27 unit cells.

Original: The  interaction radius R = 1 . 0 , δ min = 0 . 1 , and N sample = 1000 .
Revised: The other parameters are the interaction radius R = 1 . 0 , δ min = 0 . 1 , and N sample = 1000 .

Original: We select a Fourier domain and N FFT = 25 3 .
Revised: The Fourier domain used is of size N FFT = 25 3 .

Original:  Table 3 shows full-range network can restore the accuracy when the characteristic interactions length increases.
Revised: The results in Table 3 demonstrate that full-range network is capable of maintaining good accuracy for a wide range of characteristic interactions lengths.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have presented an efﬁcient long-range convolutional (LRC) layer, which leverages the nonuniform fast Fourier transform (NUFFT) to reduce the cost from quadratic to nearly-linear with respect to the number of degrees of freedom.
Revised: We have presented an efﬁcient long-range convolutional (LRC) layer, which leverages the nonuniform fast Fourier transform (NUFFT) to reduce the cost from quadratic to nearly-linear with respect to the number of degrees of freedom.

Original: We have also introduced a two-scale training strategy to effectively reduce the number of large-scale samples.
Revised: We have also introduced a two-scale training strategy to effectively reduce the number of large-scale samples.

Original: This can be particularly important when the generation of these large-scale samples dominates the computational cost.
Revised: This can be particularly important when the generation of these large-scale samples dominates the computational cost.

Original: While this paper demonstrates the effectiveness of the LRC-layer for computing the energy and force associated with a model N -body potential, we expect  that the LRC-layer can be a useful tool for a wide range of machine learning (regression / classiﬁcation) tasks as well.
Revised: While this paper demonstrates the effectiveness of the LRC-layer for computing the energy and force associated with a model N -body potential, we expect the LRC-layer to become a useful component in designing neural networks for modeling real chemical and materials systems, where the LRI cannot be accurately captured using short ranged models. We also expect that the LRC-layer can be a useful tool for a wide range of machine learning (such as regression and classiﬁcation) tasks. [Table] A N OTATION

Revised: A table of notations is summarized in Table 4.

Original: A D ATA G ENERATION
Revised: B D ATA G ENERATION

Original: We provide further details about the data generation process and how the parameter µ dictates the characteristic interaction length.
Revised: We provide further details about the data generation process and how the parameter µ dictates the characteristic interaction length.

Original: Exponential kernel: Suppose Ω be the torus [0 , L ] d and that x = { x i } Ni =1 ⊂ Ω ⊂ R d for d = 1 , 2 , or 3 .
Revised: Exponential kernel: Suppose Ω be the torus [0 , L ] d and that x = { x i } Ni =1 ⊂ Ω ⊂ R d for d = 1 , 2 , or 3 .

Original: The exponential kernel is deﬁned as
Revised: The exponential kernel is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where (cid:107) · (cid:107) is the Euclidean norm over the torus.
Revised: where (cid:107) · (cid:107) is the Euclidean norm over the torus.

Original: Following Section 3.1 we deﬁne the total energy and the potential as
Revised: Following Section 3.1 we deﬁne the total energy and the potential as

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: respectively.
Revised: respectively.

Original: The forces are given by
Revised: The forces are given by

Original: [Equation]
Revised: [Equation]

Original: Screened-Coulomb kernel:
Revised: Screened-Coulomb kernel:

Original: In 3 D, the screened-Coulomb potential with free space boundary condition is given by
Revised: In 3 D, the screened-Coulomb potential with free space boundary condition is given by

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: Total number of Fourier modes

Revised: Over the torus [

Original: Over the torus [0 , L ] d , the kernel ψ µ ( x − y ) is the Green’s function G µ ( x, y ) deﬁned via
Revised: [Table] L ] d [Table] the kernel ψ µ ( x − y ) is the Green’s function G µ ( x, y ) deﬁned via

Original: [Equation]
Revised: [Equation]

Original: (12) with the periodic boundary condition.
Revised: (12) with the periodic boundary condition.

Original: In order to compute the screened-Coulomb potential numerically, a spectral method is used: in particular,
Revised: In order to compute the screened-Coulomb potential numerically [Table] a spectral method is used: in particular,

Original: [Equation]
Revised: [Equation]

Original:  where F − 1 stands for the inverse Fourier transform and χ (cid:15) ( k ) is a smoothing factor, usually Gaussian, to numerically avoid the Gibbs phenomenon.
Revised: (13) where F − 1 stands for the inverse Fourier transform and χ (cid:15) ( k ) is a smoothing factor, usually Gaussian, to numerically avoid the Gibbs phenomenon.

Original: Similar to the exponential case, the parameter µ controls the localization of the potential.
Revised: Similar to the exponential case, the parameter µ controls the localization of the potential.

Original: In addition, the derivatives are taken numerically in the Fourier domain.
Revised: In addition, the derivatives are taken numerically in the Fourier domain.

Original: Visualization: To visualize the relation between µ and the characteristic interaction length in 1 D, consider a given particle, e.g., x 100 and compute the force contribution from the other particles.
Revised: Visualization: To visualize the relation between µ and the characteristic interaction length in 1 D, consider a given particle, e.g., x 100 and compute the force contribution from the other particles.

Original: Fig.shows that force contribution is extremely small outside a small interaction region for µ = 5 . 

Revised: Fig.shows that force contribution is extremely small outside a small interaction region for µ = 5 .

Original: while the interaction region for µ = 0 .
Revised: while the interaction region for µ = 0 .

Original: 5 is much larger.
Revised: 5 is much larger.

Original: B D ETAILS OF ARCHITECTURE AND TRAINING B. S HORT - RANGE D ESCRIPTOR
Revised: C D ETAILS OF ARCHITECTURE AND TRAINING C. S HORT - RANGE D ESCRIPTOR

Original: Here we specify the structure of D i introduced in Section 3.2.
Revised: Here we specify the structure of D i introduced in Section 3.2.

Original: For a given particle x i , and an interaction radius R , deﬁne the interaction list
Revised: For a given particle x i , and an interaction radius R , deﬁne the interaction list

Original: I i of x i as the set of indices j such that (cid:107) x
Revised: I i of x i as the set of indices j such that (cid:107) x

Original: i − x j (cid:107) <
Revised: i − x j (cid:107) <

Original: R , where (cid:107) · (cid:107) stands for the distance over the torus [0 , L ] d .
Revised: R , where (cid:107) · (cid:107) stands for the distance over the torus [0 , L ] d .

Original: To simplify the discussion, we assume that there exists a maximal number of neighbors N maxNeigh for each x i .
Revised: To simplify the discussion, we assume that there exists a maximal number of neighbors N maxNeigh for each x i .

Original: We stack the neighbors in a tensor whose dimensions are constant across different particles.
Revised: We stack the neighbors in a tensor whose dimensions are constant across different particles.

Original: This value is chosen to be sufﬁciently large to cover the number of elements in the interaction list.
Revised: This value is chosen to be sufﬁciently large to cover the number of elements in the interaction list.

Original: If the cardinality of I i is less than
Revised: If the cardinality of I i is less than

Original: N maxNeigh , we pad the tensor with dummy values.
Revised: N maxNeigh , we pad the tensor with dummy values.

Original: In the 1 D case the generalized coordinates are deﬁned as
Revised: In the 1 D case the generalized coordinates are deﬁned as

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: for j ∈ I i .
Revised: for j ∈ I i .

Original: We introduce two fully-connected neural networks f θ 1 , f θ 2 : R + → R m sr / 2 , where each consists of ﬁve layers with the number of units doubling at each layer and ranging from 2 to 32 .
Revised: We introduce two fully-connected neural networks f θ 1 , f θ 2 : R + → R m sr / 2 , where each consists of ﬁve layers with the number of units doubling at each layer and ranging from 2 to 32 .

Original: The activation function after each layer is tanh and the initialization follows Glorot normal distribution.
Revised: The activation function after each layer is tanh and the initialization follows Glorot normal distribution.

Original: For particle x i the short-range descriptor is deﬁned as the concatenation of
Revised: For particle x i the short-range descriptor is deﬁned as the concatenation of

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: D

Original: [Equation]
Revised: [Equation]

Original: where ˆ r i,j , ˆ s i,j are the normalized copies of r i,j and s i,j with mean zero and standard deviation equals to one.
Revised: where ˆ r i,j , ˆ s i,j are the normalized copies of r i,j and s i,j with mean zero and standard deviation equals to one.

Original: The mean and standard deviation are estimated by using a small number of snapshots.
Revised: The mean and standard deviation are estimated by using a small number of snapshots.

Original: We multiply the network’s output f θ by ˆ r
Revised: We multiply the network’s output f θ by ˆ r

Original: i,j (which is zero if j is a dummy particle).
Revised: i,j (which is zero if j is a dummy particle).

Original: This procedure enforces a zero output for particles not in the interaction list.
Revised: This procedure enforces a zero output for particles not in the interaction list.

Original: The construction satisﬁes the design requirement mentioned in Section 3.2.
Revised: The construction satisﬁes the design requirement mentioned in Section 3.2.

Original: In the short-range network, one concatenates the two descriptor above and feeds them particle-wise to the short-range ﬁtting network.
Revised: In the short-range network, one concatenates the two descriptor above and feeds them particle-wise to the short-range ﬁtting network.

Original: The ﬁtting network F sr : R m sr → R is a residual neural network (ResNet) with six layers, each with 32 units.
Revised: The ﬁtting network F sr : R m sr → R is a residual neural network (ResNet) with six layers, each with 32 units.

Original: The activation function and initialization strategy are the same as the ones for the short-range descriptors.
Revised: The activation function and initialization strategy are the same as the ones for the short-range descriptors.

Original: Fig.
Revised: Fig.

Original: 5 shows the detailed architecture of the short-range network.
Revised: 6 shows the detailed architecture of the short-range network.

Original: [Equation]
Revised: [Equation]

Original: In 2D and 3D, there is a slight difference of generalized coordinates: we compute
Revised: In 2D and 3D, there is a slight difference of generalized coordinates: we compute

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: where s i,j is a vector now.
Revised: where s i,j is a vector now.

Original: The local descriptors are deﬁned in the following forms:
Revised: The local descriptors are deﬁned in the following forms:

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: B. NUFFT
Revised: C. NUFFT

Original: In this section we provide further details for the NUFFT implementation.
Revised: In this section we provide further details for the NUFFT implementation.

Original: Suppose that the input of the NUFFT is given by { x i } Ni =1 ⊂ R d , where each point has a given associated weight f i .
Revised: Suppose that the input of the NUFFT is given by { x i } N i =1 ⊂ R d , where each point has a given associated weight f i .

Original: The ﬁrst step is to construct the weighted train of Dirac deltas as
Revised: The ﬁrst step is to construct the weighted train of Dirac deltas as

Original: [Equation]
Revised: [Equation]

Original: We point out that in some of the experiments f j simply equals to 1 .
Revised: We point out that in some of the experiments f j simply equals to 1 .

Original: One then deﬁnes a periodic
Revised: One then deﬁnes a periodic

Original: Gaussian convolution kernel
Revised: Gaussian convolution kernel

Original: [Equation]
Revised: [Equation]

Original: where L is the length of the interval and τ determines the size of molliﬁcation.
Revised: where L is the length of the interval and τ determines the size of molliﬁcation.

Original: In practice a good choice is τ = 12(
Revised: In practice a good choice is τ = 12(

Revised: L 2 πL FFT ) 2 (Dutt & Rokhlin, 1993)

Original: [Equation]
Revised: [Equation]

Original: 2 πL FFT ) 2 (Dutt & Rokhlin, 1993), where L FFT is the number of points in each dimension and N FFT = L d FFT .
Revised: where L FFT is the number of points in each dimension and N FFT = L d FFT .

Original: We deﬁne
Revised: We deﬁne

Original: [Equation] 
Revised: [Equation] [Equation]

Original: With the Fourier transform deﬁned as
Revised: With the Fourier transform deﬁned as

Original: [Equation]
Revised: [Equation]

Original: for k ∈ Z d , we compute its discrete counterpart
Revised: for k ∈ Z d [Equation] we compute its discrete counterpart

Original: [Equation]
Revised: [Equation]

Original:  This operation can be done in O ( N FFT log( N FFT )) steps, independently of the number of inputs.
Revised: [Equation] This operation can be done in O ( N FFT log( N FFT )) steps, independently of the number of inputs.

Original: Once this is computed, one can compute the Fourier transform of f at each frequency point by
Revised: Once this is computed, one can compute the Fourier transform of f at each frequency point by

Original: [Equation]
Revised: [Equation]

Original: [Equation] Once the Fourier transform of the Dirac delta train is ready, we multiply it by the Fourier multiplier ˆ φ ( k ) , which is the Fourier transform of φ :[Equation]
Revised: Once the Fourier transform of the Dirac delta train is ready, we multiply it by the Fourier multiplier ˆ φ ( k ) , which is the Fourier transform of φ :

Original: [Equation]
Revised: [Equation]

Original: In the next sage, one needs to compute the inverse transform, and evaluate into the target points { x i } .
Revised: In the next sage, one needs to compute the inverse transform, and evaluate into the target points { x i } .

Original: First we deconvolve the signal
Revised: First we deconvolve the signal

Original: [Equation]
Revised: [Equation]

Original: and compute the inverse Fourier transform
Revised: and compute the inverse Fourier transform

Original: [Equation]
Revised: [Equation]

Original: Next, we interpolate to the point cloud
Revised: Next, we interpolate to the point cloud

Original: [Equation]
Revised: [Equation]

Original: Even though in the current implementation all the parameters of the NUFFT are ﬁxed, they can in principle be trained along with the rest of the networks.
Revised: Even though in the current implementation all the parameters of the NUFFT are ﬁxed, they can in principle be trained along with the rest of the networks.

Original: This training, if done naively increases signiﬁcantly the computational cost.
Revised: This training, if done naively increases signiﬁcantly the computational cost.

Original: How to perform this operation efﬁciently is a direction of future research.
Revised: How to perform this operation efﬁciently is a direction of future research.

Original: Derivatives For the computation of the forces in equation 5 one needs to compute the derivatives of the total energy U NN with respect to the inputs, in nearly-linear time.
Revised: Derivatives For the computation of the forces in equation 5 one needs to compute the derivatives of the total energy U NN with respect to the inputs, in nearly-linear time.

Original: The main obstacle is how to compute the derivatives of the LRC-layer with respect to the point-cloud efﬁciently.
Revised: The main obstacle is how to compute the derivatives of the LRC-layer with respect to the point-cloud efﬁciently.

Original: To simplify the notation, we only discuss the case that d = 1 , but the argument can be seamlessly extended to the case when d > 1 .
Revised: To simplify the notation, we only discuss the case that d = 1 , but the argument can be seamlessly extended to the case when d > 1 .

Original: Recall that u i =
Revised: Recall that u i =

Original: [Equation]
Revised: [Equation]

Original: Nj =1 φ θ [Equation] x i −x j
Revised: i −

Original: [Equation]
Revised: [Equation]

Original: f j , then the Jacobian of the vector u with respect to the inputs is given by
Revised: f j , then the Jacobian of the vector u with respect to the inputs is given by

Original: [Equation]
Revised: [Equation]

Original: As it will be explained in the sequel, for the computation of the forces in equation 5 one needs to compute the application of the Jacobian of u to a vector.
Revised: As it will be explained in the sequel [Equation] for the computation of the forces in equation 5 one needs to compute the application of the Jacobian of u to a vector.

Original: For a ﬁxed vector v ∈ R N , the product ( ∇ u ) · v can be written component-wise as
Revised: For a ﬁxed vector v ∈ R N , the product ( ∇ u ) · v can be written component-wise as

Original: [Equation]
Revised: [Equation]

Original: where we have added ± v i f i φ (cid:48) (0) in the last equation and then distributed it within both sums.
Revised: where we have added ± v i f i φ (cid:48) (0) in the last equation and then distributed it within both sums.

Original: Let us deﬁne the following two long-range convolutions
Revised: Let us deﬁne the following two long-range convolutions

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: each of which can be performed in O ( N + N FFT log N FFT ) steps using the NUFFT algorithm combined with the convolution theorem.
Revised: each of which can be performed in O ( N + N FFT log N FFT ) steps using the NUFFT algorithm combined with the convolution theorem.

Original: In this case the derivative of φ can be computed numerically in the Fourier domain to a very high accuracy.
Revised: In this case the derivative of φ can be computed numerically in the Fourier domain to a very high accuracy.

Original: Now one can leverage the expression above to rewrite ( ∇ u ) · v as
Revised: Now one can leverage the expression above to rewrite ( ∇ u ) · v as

Original: [Equation]
Revised: [Equation]

Original: which can then be computed in nearly-linear time.
Revised: which can then be computed in nearly-linear time.

Original: The same is also true for v · ( ∇ u ) 
Revised: The same is also true for v · ( ∇ u ) .

Original: [Equation]

Original: B. L ONG - RANGE D ESCRIPTOR
Revised: C. L ONG - RANGE D ESCRIPTOR

Original: As mentioned before, the output of the LRC-layer is given by { u ( x i ) } Ni =1 .
Revised: As mentioned before, the output of the LRC-layer is given by { u ( x i ) } Ni =1 .

Original: For each particle we feed the output u ( x i ) to the long-range descriptor network h θ : R → R m lr , whose structure is the same as the local descriptor f θ mentioned in appendix B.1 except that the activation function is taken to be
Revised: For each particle we feed the output u ( x i ) to the long-range descriptor network h θ : R → R m lr , whose structure is the same as the local descriptor f θ mentioned in appendix C.1 except that the activation function is taken to be

Original: ReLU.
Revised: ReLU.

Original: The long-range descriptor, deﬁned as
Revised: The long-range descriptor, deﬁned as

Original: [Equation]
Revised: [Equation]

Original: [Equation] for the particle x i is concatenated with the corresponding short-range descriptor (which it is itself the concatenation of two short-range descriptors) and fed together to the total ﬁtting network F : R m sr + m lr → R .
Revised: ) for the particle x i is concatenated with the corresponding short-range descriptor (which it is itself the concatenation of two short-range descriptors) and fed together to the total ﬁtting network F : R m sr + m lr → R .

Original: The results are then added together to obtain the total energy
Revised: The results are then added together to obtain the total energy

Original: [Equation]
Revised: [Equation]

Original: It is clear that the energy can be evaluated in nearly-linear complexity.
Revised: It is clear that the energy can be evaluated in nearly-linear complexity.

Original: In what follows we show that the force computation is also of nearly-linear.
Revised: In what follows we show that the force computation is also of nearly-linear.

Original: For simplicity we focus on the one-dimensional network and assume that K chnls = 1 , O ( m sr )
Revised: For simplicity we focus on the one-dimensional network and assume that K chnls = 1 , O ( m sr )

Original: =
Revised: =

Original: O ( m lr )
Revised: O ( m lr )

Original: =
Revised: =

Original: O (1) and that the depth of the neural networks is O (1) .
Revised: O (1) and that the depth of the neural networks is O (1) .

Original: As deﬁned in the prequel the forces are given by
Revised: As deﬁned in the prequel the forces are given by

Original: F NN = [Equation] x U NN , Which can be written component wise as
Revised: F NN = −∇ x U NN , Which can be written component wise as

Original: [Equation]
Revised: [Equation]

Original: or in a more compact fashion as
Revised: or in a more compact fashion as

Original: [Equation]
Revised: [Equation]

Original: Here v sr , and v lr are vectors deﬁned component-wise as ( v sr ) i =
Revised: Here v sr , and v lr are vectors deﬁned component-wise as ( v sr ) i =

Original: ∂ 1 F ( D i sr , D i lr ) , and ( v lr ) i =
Revised: ∂ 1 F ( D i sr , D i lr ) , and ( v lr ) i =

Original: ∂ 2 F ( D i sr , D i lr ) g (cid:48)
Revised: ∂ 2 F ( D i sr , D i lr ) g (cid:48)

Original: θ ( u i ) .
Revised: θ ( u i ) .

Original: In addition ( D sr ) i,j = ∂ x j D i sr and ∇ u is deﬁned above.
Revised: In addition ( D sr ) i,j = ∂ x j D i sr and ∇ u is deﬁned above.

Original: The ﬁrst term in the right-hand side is easy to compute, given that D sr is sparse: the i, j entry is non-zero only if the particle x
Revised: The ﬁrst term in the right-hand side is easy to compute, given that D sr is sparse: the i, j entry is non-zero only if the particle x

Original: i is in the interaction list of x j .
Revised: i is in the interaction list of x j .

Original: Given that the cardinality of the interaction list is bounded, D sr has O ( N ) non-zero entries in which each entry requires O (1) work, thus the ﬁrst term in the right-hand side of equation 37 can be computed in O ( N ) .
Revised: Given that the cardinality of the interaction list is bounded, D sr has O ( N ) non-zero entries in which each entry requires O (1) work, thus the ﬁrst term in the right-hand side of equation 37 can be computed in O ( N ) .

Original: At ﬁrst glance the complexity of second term seems to be much higher.
Revised: At ﬁrst glance the complexity of second term seems to be much higher.

Original: However, as discussed above, by using equation 33, we can apply the matrix (or its transpose) to a vector in O ( N + N FFT log N FFT ) time and the computation of vector v lr requires O (1) work per entry, thus resulting in a complexity of O ( N + N FFT log N FFT ) for computing the second term in equation 37.
Revised: However, as discussed above, by using equation 33, we can apply the matrix (or its transpose) to a vector in O ( N + N FFT log N FFT ) time and the computation of vector v lr requires O (1) work per entry, thus resulting in a complexity of O ( N + N FFT log N FFT ) for computing the second term in equation 37.

Original: Finally, adding both contributions together results in an overall O ( N + N FFT log N FFT ) complexity for the forces.
Revised: Finally, adding both contributions together results in an overall O ( N + N FFT log N FFT ) complexity for the forces.

Original: To summarize, both the computation of the energy and the forces can be performed in O ( N ) time.
Revised: To summarize, both the computation of the energy and the forces can be performed in O ( N ) time.

Original: B. T RAINING
Revised: C. T RAINING

Original: We use the Adam optimizer (Kingma & Ba, 2015) along with an exponential scheduler.
Revised: We use the Adam optimizer (Kingma & Ba, 2015) along with an exponential scheduler.

Original: The learning rate with the initial learning rate taken to be 0 .
Revised: The learning rate with the initial learning rate taken to be 0 .

Original: 001 and, for every 10 epochs, it decreases by a factor of 0 .
Revised: 001 and, for every 10 epochs, it decreases by a factor of 0 .

Original: In order to balance the computational time and the accuracy, a multi-stage training is adopted, where at each stage we modify the batch-size and the number of epochs.
Revised: In order to balance the computational time and the accuracy, a multi-stage training is adopted, where at each stage we modify the batch-size and the number of epochs.

Original: In particular, four stages are used: we start using a batch size of 8 snapshots and train the network 200 epochs and then at each stage we double both the size of the batch size and the number of epochs.
Revised: In particular, four stages are used: we start using a batch size of 8 snapshots and train the network 200 epochs and then at each stage we double both the size of the batch size and the number of epochs.

Original: In the two-scale training strategy, the same training parameters deﬁned above are used for each stage. 
Revised: In the two-scale training strategy, the same training parameters deﬁned above are used for each stage. C. D EPENDENCY ON N FFT

Revised: We measure the impact of N FFT on the approximation error, using a couple of examples in the oneand two-dimensional settings.

Revised: For the one-dimensional case, we test a screened-Coulomb type potential with parameters µ 1 = 5 . 0 , µ 2 = 0 .

Revised: 5 , α 1 = 0 .

Revised: 5 , α 2 = 0 .

Revised: 5 , and N sample = 1000 .

Revised: The domain Ω is [0 , 50] and N = 200 .

Revised: We run the one-scale training procedure with varying N FFT (the number of Fourier multipliers), starting from N FFT = 63 and doubling them until N FFT = 501 .

Revised: Table 5 shows that the errors are relatively insensitive to the value of N FFT .

Revised: The accuracy achieved by the architecture without the LRC-layer (denoted as None in Tables 5) is added in order to demonstrate that the architecture is indeed capturing the LRIs.

Revised: [Table]

Revised: For the two-dimensional case, a screened-Coulomb type potential is tested with µ 1 = 10 . 0 , µ 2 = 1 . 0 , α 1 = 0 .

Revised: 9 , α 2 = 0 .

Revised: Here Ω = [0 , 5] 2 , N = 50 and N sample = 1000 .

Revised: Starting with N FFT = 21 2 , we steadily increase its value and repeat the same training procedure.

Revised: The results are summarized in

Revised: Table 6 where one observes the same trend as in the one-dimensional case.

Revised: [Table]

Revised: In addition, we recall that the Fourier multipliers are parametrized following

Revised: [Equation]

Revised: [Equation]

Revised: where β and λ are two trainable parameters with λ providing a measure of the decay in space.

Revised: Therefore, N FFT only determines the number of Fourier modes and not the parameters of the ansatz.

Revised: As long as the Fourier kernel is properly sampled, the method is able to compute the correct characteristic interaction length.

Revised: One can observe this phenomenon in the experiment above, in which we extract the terminal value after training of the parameters λ 1 and λ 2 that correspond to the two channels in the LRC-layer, as summarized in Table 7.

Revised: We observe that the value of λ 2 is very close to that of µ 2 , which is responsible for the LRIs even for small values of N FFT .

Revised: [Table]