Original: L EARNING TO D ISCRETIZE : S OLVING 1D S CALAR C ONSERVATION L AWS VIA D EEP R EINFORCEMENT
Revised: L EARNING TO D ISCRETIZE : S OLVING 1D S CALAR C ONSERVATION L AWS VIA D EEP R EINFORCEMENT

Original: L EARNING
Revised: L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Conservation laws are considered to be fundamental laws of nature.
Revised: Conservation laws are considered to be fundamental laws of nature.

Original: It has broad application in many ﬁelds including physics, chemistry, biology, geology, and engineering.
Revised: It has broad application in many ﬁelds including physics, chemistry, biology, geology, and engineering.

Original: Solving the differential equations associated with conservation laws is a major branch in computational mathematics.
Revised: Solving the differential equations associated with conservation laws is a major branch in computational mathematics.

Original: Recent success of machine learning, especially deep learning, in areas such as computer vision and natural language processing, has attracted a lot of attention from the community of computational mathematics and inspired many intriguing works in combining machine learning with traditional methods.
Revised: Recent success of machine learning, especially deep learning, in areas such as computer vision and natural language processing, has attracted a lot of attention from the community of computational mathematics and inspired many intriguing works in combining machine learning with traditional methods.

Original: In this paper, we are the ﬁrst to explore the possibility and beneﬁt of solving nonlinear conservation laws using deep reinforcement learning.
Revised: In this paper, we are the ﬁrst to view numerical PDE solvers as a MDP and to use (deep) RL to learn new solvers.

Original: As a proof of concept, we focus on 1-dimensional scalar conservation laws.
Revised: As a proof of concept, we focus on 1-dimensional scalar conservation laws.

Original: We deploy the machinery of deep reinforcement learning to train a policy network that can decide on how the numerical solutions should be approximated in a sequential and spatial-temporal adaptive manner.
Revised: We deploy the machinery of deep reinforcement learning to train a policy network that can decide on how the numerical solutions should be approximated in a sequential and spatial-temporal adaptive manner.

Original: We will show that the problem of solving conservation laws can be naturally viewed as a sequential decision making process and the numerical schemes learned in such a way can easily enforce long-term accuracy.
Revised: We will show that the problem of solving conservation laws can be naturally viewed as a sequential decision making process and the numerical schemes learned in such a way can easily enforce long-term accuracy.

Original: Furthermore, the learned policy network is carefully designed to determine a good local discrete approximation based on the current state of the solution, which essentially makes the proposed method a meta-learning approach.
Revised: Furthermore, the learned policy network is carefully designed to determine a good local discrete approximation based on the current state of the solution, which essentially makes the proposed method a meta-learning approach.

Original: In other words, the proposed method is capable of learning how to discretize for a given situation mimicking human experts.
Revised: In other words, the proposed method is capable of learning how to discretize for a given situation mimicking human experts.

Original: Finally, we will provide details on how the policy network is trained, how well it performs compared with some state-of-the-art numerical solvers such as WENO schemes, and how well it generalizes.
Revised: Finally, we will provide details on how the policy network is trained, how well it performs compared with some state-of-the-art numerical solvers such as WENO schemes, and how well it generalizes.

Original: Our code is released anomynously at https://github.com/ qwerlanksdf/L2D .
Revised: Our code is released anomynously at https://github.com/qwerlanksdf/L2D .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Conservation laws are considered to be one of the fundamental laws of nature, and has broad applications in multiple ﬁelds such as physics, chemistry, biology, geology, and engineering.
Revised: Conservation laws are considered to be one of the fundamental laws of nature, and has broad applications in multiple ﬁelds such as physics, chemistry, biology, geology, and engineering.

Original: For example, Burger’s equation, a very classic partial differential equation (PDE) in conservation laws, has important applications in ﬂuid mechanics, nonlinear acoustics, gas dynamics, and trafﬁc ﬂow.
Revised: For example, Burger’s equation, a very classic partial differential equation (PDE) in conservation laws, has important applications in ﬂuid mechanics, nonlinear acoustics, gas dynamics, and trafﬁc ﬂow.

Original: Solving the differential equations associated with conservation laws has been a major branch of computational mathematics (LeVeque, 1992; 2002), and a lot of effective methods have been proposed, from classic methods such as the upwind scheme, the Lax-Friedrichs scheme, to the advanced ones such as the ENO/WENO schemes (Liu et al., 1994; Shu, 1998), the ﬂux-limiter methods (Jerez Galiano & Uh Zapata, 2010), and etc.
Revised: Solving the differential equations associated with conservation laws has been a major branch of computational mathematics (LeVeque, 1992; 2002), and a lot of effective methods have been proposed, from classic methods such as the upwind scheme, the Lax-Friedrichs scheme, to the advanced ones such as the ENO/WENO schemes (Liu et al., 1994; Shu, 1998), the ﬂux-limiter methods (Jerez Galiano & Uh Zapata, 2010), and etc.

Original: In the past few decades, these traditional methods have been proven successful in solving conservation laws.
Revised: In the past few decades, these traditional methods have been proven successful in solving conservation laws.

Original: Nonetheless, the design of some of the high-end methods heavily relies on expert knowledge and the coding of these methods can be a laborious process.
Revised: Nonetheless, the design of some of the high-end methods heavily relies on expert knowledge and the coding of these methods can be a laborious process.

Original: To ease the usage and potentially improve these traditional algorithms, machine learning, especially deep learning, has been recently incorporated into this ﬁeld.
Revised: To ease the usage and potentially improve these traditional algorithms, machine learning, especially deep learning, has been recently incorporated into this ﬁeld.

Original: For example, the
Revised: For example, the

Original: ENO scheme requires lots of ‘if/else’ logical judgments when used to solve complicated system of equations or high-dimensional equations.
Revised: ENO scheme requires lots of ‘if/else’ logical judgments when used to solve complicated system of equations or high-dimensional equations.

Original: This very much resembles the old-fashioned expert systems.
Revised: This very much resembles the old-fashioned expert systems.

Original: The recent trend in artiﬁcial intelligence (AI) is to replace the expert systems by the so-called ‘connectionism’, e.g., deep neural networks, which leads to the recent bloom of AI.
Revised: The recent trend in artiﬁcial intelligence (AI) is to replace the expert systems by the so-called ‘connectionism’, e.g., deep neural networks, which leads to the recent bloom of AI.

Original: Therefore, it is natural and potentially beneﬁcial to introduce deep learning in traditional numerical solvers of conservation laws.
Revised: Therefore, it is natural and potentially beneﬁcial to introduce deep learning in traditional numerical solvers of conservation laws.

Original: 1. R ELATED WORKS
Revised: 1. R ELATED WORKS

Original: In the last few years, neural networks (NNs) have been applied to solving ODEs/PDEs or the associated inverse problems.
Revised: In the last few years, neural networks (NNs) have been applied to solving ODEs/PDEs or the associated inverse problems.

Original: These works can be roughly classiﬁed into three categories according to the way that the NN is used.
Revised: These works can be roughly classiﬁed into three categories according to the way that the NN is used.

Original: The ﬁrst type of works propose to harness the representation power of NNs, and are irrelevant to the numerical discretization based methods.
Revised: The ﬁrst type of works propose to harness the representation power of NNs, and are irrelevant to the numerical discretization based methods.

Original: For example, Raissi et al.
Revised: For example, Raissi et al.

Original: (2017a;b); Yohai Bar-Sinai (2018) treated the NNs as new ansatz to approximate solutions of PDEs.
Revised: (2017a;b); Yohai Bar-Sinai (2018) treated the NNs as new ansatz to approximate solutions of PDEs.

Revised: It was later generalized by Wei et al.

Revised: (2019) to allow randomness in the solution which is trained using policy gradient.

Original: More recent works along this line include (Magiera et al., 2019; Michoski et al., 2019; Both et al., 2019).
Revised: More recent works along this line include (Magiera et al., 2019; Michoski et al., 2019; Both et al., 2019).

Original: Besides, several works have focused on using NNs to establish direct mappings between the parameters of the PDEs (e.g. the coefﬁcient ﬁeld or the ground state energy) and their associated solutions (Khoo et al., 2017; Khoo & Ying, 2018; Li et al., 2019; Fan et al., 2018b).
Revised: Besides, several works have focused on using NNs to establish direct mappings between the parameters of the PDEs (e.g. the coefﬁcient ﬁeld or the ground state energy) and their associated solutions (Khoo et al., 2017; Khoo & Ying, 2018; Li et al., 2019; Fan et al., 2018b).

Original: Furthermore, Han et al.
Revised: Furthermore, Han et al.

Original: ; Beck et al.
Revised: ; Beck et al.

Original: (2017) proposed a method to solve very high-dimensional PDEs by converting the PDE to a stochastic control problem and use NNs to approximate the gradient of the solution.
Revised: (2017) proposed a method to solve very high-dimensional PDEs by converting the PDE to a stochastic control problem and use NNs to approximate the gradient of the solution.

Original: The second type of works focus on the connection between deep neural networks (DNNs) and dynamic systems (Weinan, 2017;
Revised: The second type of works focus on the connection between deep neural networks (DNNs) and dynamic systems (Weinan, 2017;

Original: Chang et al., 2017; Lu et al., 2018; Long et al., 2018b; Chen et al., 2018).
Revised: Chang et al., 2017; Lu et al., 2018; Long et al., 2018b; Chen et al., 2018).

Original: These works observed that there are connections between DNNs and dynamic systems (e.g. differential equations or unrolled optimization algorithms) so that we can combine deep learning with traditional tools from applied and computational mathematics to handle challenging tasks in inverse problems (Long et al., 2018b;a; Qin et al.,
Revised: These works observed that there are connections between DNNs and dynamic systems (e.g. differential equations or unrolled optimization algorithms) so that we can combine deep learning with traditional tools from applied and computational mathematics to handle challenging tasks in inverse problems (Long et al., 2018b;a; Qin et al.,

Original: 2018).The main focus of these works, however, is to solve inverse problems, instead of learning numerical discretizations of differential equations.
Revised: 2018).The main focus of these works, however, is to solve inverse problems, instead of learning numerical discretizations of differential equations.

Original: Nonetheless, these methods are closely related to numerical differential equations since learning a proper discretization is often an important auxiliary task for these methods to accurately recover the form of the differential equations.
Revised: Nonetheless, these methods are closely related to numerical differential equations since learning a proper discretization is often an important auxiliary task for these methods to accurately recover the form of the differential equations.

Original: The third type of works, which target at using NNs to learn new numerical schemes, are closely related to our work.
Revised: The third type of works, which target at using NNs to learn new numerical schemes, are closely related to our work.

Original: However, we note that these works mainly fall in the setting of supervised learning (SL).
Revised: However, we note that these works mainly fall in the setting of supervised learning (SL).

Original: For example, Discacciati et al.
Revised: For example, Discacciati et al.

Original: (2019) proposed to integrate NNs into high-order numerical solvers to predict artiﬁcial viscosity; Ray & Hesthaven (2018) trained a multilayer perceptron to replace traditional indicators for identifying troubled-cells in high-resolution schemes for conservation laws.
Revised: (2019) proposed to integrate NNs into high-order numerical solvers to predict artiﬁcial viscosity; Ray & Hesthaven (2018) trained a multilayer perceptron to replace traditional indicators for identifying troubled-cells in high-resolution schemes for conservation laws.

Original: These works greatly advanced the development in machine learning based design of numerical schemes for conservation laws.
Revised: These works greatly advanced the development in machine learning based design of numerical schemes for conservation laws.

Original: Note that in Discacciati et al.
Revised: Note that in Discacciati et al.

Original: (2019), the authors only utilized the one-step error to train the artiﬁcial viscosity networks without taking into account the longterm accuracy of the learned numerical scheme.
Revised: (2019), the authors only utilized the one-step error to train the artiﬁcial viscosity networks without taking into account the longterm accuracy of the learned numerical scheme.

Original: Ray & Hesthaven (2018)
Revised: Ray & Hesthaven (2018)

Original: ﬁrst constructed several functions with known regularities and then used them to train a neural network to predict the location of discontinuity, which was later used to choose a proper slope limiter.
Revised: ﬁrst constructed several functions with known regularities and then used them to train a neural network to predict the location of discontinuity, which was later used to choose a proper slope limiter.

Original: Therefore, the training of the NNs is separated from the numerical scheme.
Revised: Therefore, the training of the NNs is separated from the numerical scheme.

Original: Then, a natural question is whether we can learn discretization of differential equations in an end-to-end fashion and the learned discrete scheme also takes long-term accuracy into account.
Revised: Then, a natural question is whether we can learn discretization of differential equations in an end-to-end fashion and the learned discrete scheme also takes long-term accuracy into account.

Original: This motivates us to employ reinforcement learning to learn good solvers for conservation laws.
Revised: This motivates us to employ reinforcement learning to learn good solvers for conservation laws.

Original: 1. O UR A PPROACH
Revised: 1. O UR A PPROACH

Original: The main objective of this paper is to design new numerical schemes in an autonomous way.
Revised: The main objective of this paper is to design new numerical schemes in an autonomous way.

Original: We propose to use reinforcement learning (RL) to aid the process of solving the conservation laws.
Revised: We propose to use reinforcement learning (RL) to aid the process of solving the conservation laws.

Original: To our best knowledge, we are the ﬁrst to propose using (deep) RL to solve evolution PDEs .
Revised: To our best knowledge, we are the ﬁrst to regard numerical PDE solvers as a MDP and to use (deep) RL to learn new solvers .

Original: We carefully design the proposed RL-based method so that the learned policy can generate high accuracy numerical schemes and can well generalize in varied situations.
Revised: We carefully design the proposed RL-based method so that the learned policy can generate high accuracy numerical schemes and can well generalize in varied situations.

Original: Details will be given in section 3.
Revised: Details will be given in section 3.

Original: Here, we ﬁrst provide a brief discussion on the beneﬁts of using RL to solve conservation laws (the arguments apply to general evolution PDEs as well):
Revised: Here, we ﬁrst provide a brief discussion on the beneﬁts of using RL to solve conservation laws (the arguments apply to general evolution PDEs as well):

Original: • Most of the numerical solvers of conservation law can be interpreted naturally as a sequential decision making process (e.g., the approximated grid values at the current time instance deﬁnitely affects all the future approximations).
Revised: • Most of the numerical solvers of conservation law can be interpreted naturally as a sequential decision making process (e.g., the approximated grid values at the current time instance deﬁnitely affects all the future approximations).

Original: Thus, it can be easily formulated as a Markov Decision Process (MDP) and solved by RL.
Revised: Thus, it can be easily formulated as a Markov Decision Process (MDP) and solved by RL.

Original: • In almost all the RL algorithms, the policy π (which is the AI agent who decides on how the solution should be approximated locally) is optimized with regards to the values Q π ( s 0 , a 0 ) =
Revised: • In almost all the RL algorithms, the policy π (which is the AI agent who decides on how the solution should be approximated locally) is optimized with regards to the values Q π ( s 0 , a 0 ) =

Original: r ( s 0 , a 0 ) + (cid:80) ∞ t =1 γ t r ( s t , a t ) , which by deﬁnition considers the long-term accumulated reward (or, error of the learned numerical scheme), thus could naturally guarantee the long-term accuracy of the learned schemes, instead of greedily deciding the local approximation which is the case for most numerical PDEs solvers.
Revised: r ( s 0 , a 0 ) + (cid:80) ∞ t =1 γ t r ( s t , a t ) , which by deﬁnition considers the long-term accumulated reward (or, error of the learned numerical scheme), thus could naturally guarantee the long-term accuracy of the learned schemes, instead of greedily deciding the local approximation which is the case for most numerical PDEs solvers.

Original: Furthermore, it can gracefully handle the cases when the action space is discrete, which is in fact one of the major strength of RL.
Revised: Furthermore, it can gracefully handle the cases when the action space is discrete, which is in fact one of the major strength of RL.

Original: • By optimizing towards long-term accuracy and effective exploration, we believe that RL has a good potential in improving traditional numerical schemes, especially in parts where no clear design principles exist.
Revised: • By optimizing towards long-term accuracy and effective exploration, we believe that RL has a good potential in improving traditional numerical schemes, especially in parts where no clear design principles exist.

Original: For example, although the WENO-5 scheme achieves optimal order of accuracy at smooth regions of the solution (Shu, 1998), the best way of choosing templates near singularities remains unknown.
Revised: For example, although the WENO-5 scheme achieves optimal order of accuracy at smooth regions of the solution (Shu, 1998), the best way of choosing templates near singularities remains unknown.

Original: Our belief that RL could shed lights on such parts is later veriﬁed in the experiments: the trained RL policy demonstrated new behaviours and is able to select better templates than WENO and hence approximate the solution better than WENO near singularities.
Revised: Our belief that RL could shed lights on such parts is later veriﬁed in the experiments: the trained RL policy demonstrated new behaviours and is able to select better templates than WENO and hence approximate the solution better than WENO near singularities.

Original: • Non-smooth norms such as the inﬁnity norm of the error is often used to evaluate the performance of the learned numerical schemes.
Revised: • Non-smooth norms such as the inﬁnity norm of the error is often used to evaluate the performance of the learned numerical schemes.

Original: As the norm of the error serves as the loss function for the learning algorithms, computing the gradient of the inﬁnity norm can be problematic for supervised learning, while RL does not have such problem since it does not explicitly take gradients of the loss function (i.e. the reward function for RL).
Revised: As the norm of the error serves as the loss function for the learning algorithms, computing the gradient of the inﬁnity norm can be problematic for supervised learning, while RL does not have such problem since it does not explicitly take gradients of the loss function (i.e. the reward function for RL).

Original: • Learning the policy π within the RL framework makes the algorithm meta-learning-like (Schmidhuber, 1987; Bengio et al., 1992; Andrychowicz et al., 2016; Li & Malik, 2016; Finn et al., 2017).
Revised: • Learning the policy π within the RL framework makes the algorithm meta-learning-like (Schmidhuber, 1987; Bengio et al., 1992; Andrychowicz et al., 2016; Li & Malik, 2016; Finn et al., 2017).

Original: The learned policy π can decide on which local numerical approximation to use by judging from the current state of the solution (e.g. local smoothness, oscillatory patterns, dissipation, etc).
Revised: The learned policy π can decide on which local numerical approximation to use by judging from the current state of the solution (e.g. local smoothness, oscillatory patterns, dissipation, etc).

Original: This is vastly different from regular (non-meta-) learning where the algorithms directly make inference on the numerical schemes without the aid of an additional network such as π .
Revised: This is vastly different from regular (non-meta-) learning where the algorithms directly make inference on the numerical schemes without the aid of an additional network such as π .

Original: As subtle the difference as it may seem, meta-learning-like methods have been proven effective in various applications such as in image restoration (Jin et al., 2017;
Revised: As subtle the difference as it may seem, meta-learning-like methods have been proven effective in various applications such as in image restoration (Jin et al., 2017;

Original: Fan et al., 2018a; Zhang et al., 2019).
Revised: Fan et al., 2018a; Zhang et al., 2019).

Original: See (Vanschoren, 2018) for a comprehensive survey on meta-learning.
Revised: See (Vanschoren, 2018) for a comprehensive survey on meta-learning.

Original: • Another purpose of this paper is to raise an awareness of the connection between MDP and numerical PDE solvers, and the general idea of how to use RL to improve PDE solvers or even ﬁnding brand new ones.
Revised: • Another purpose of this paper is to raise an awareness of the connection between MDP and numerical PDE solvers, and the general idea of how to use RL to improve PDE solvers or even ﬁnding brand new ones.

Original: Furthermore, in computational mathematics, a lot of numerical algorithms are sequential, and the computation at each step is expert-designed and usually greedy, e.g., the conjugate gradient method, the fast sweeping method (Zhao, 2005), matching pursuit (Mallat & Zhang, 1993), etc.
Revised: Furthermore, in computational mathematics, a lot of numerical algorithms are sequential, and the computation at each step is expert-designed and usually greedy, e.g., the conjugate gradient method, the fast sweeping method (Zhao, 2005), matching pursuit (Mallat & Zhang, 1993), etc.

Original: We hope our work could motivate more researches in combining RL and computational mathematics, and stimulate more exploration on using RL as a tool to tackle the bottleneck problems in computational mathematics.
Revised: We hope our work could motivate more researches in combining RL and computational mathematics, and stimulate more exploration on using RL as a tool to tackle the bottleneck problems in computational mathematics.

Original: Our paper is organized as follows.
Revised: Our paper is organized as follows.

Original: In section 2 we brieﬂy review 1-dimensional conservation laws and the WENO schemes.
Revised: In section 2 we brieﬂy review 1-dimensional conservation laws and the WENO schemes.

Original: In section 3, we discuss how to formulate the process of numerically solving conservation laws into a Markov Decision Process.
Revised: In section 3, we discuss how to formulate the process of numerically solving conservation laws into a Markov Decision Process.

Original: Then we present details on how to train a policy network to mimic human expert in choosing discrete schemes in a spatial-temporary adaptive manner by learning upon WENO
Revised: Then, we present details on how to train a policy network to mimic human expert in choosing discrete schemes in a spatial-temporary adaptive manner by learning upon WENO.

Original: In section  we conduct numerical experiments on 1-D conservation laws to demonstrate the performance of our trained policy network.Our experimental results show that the trained
Revised: In section 4, we conduct numerical experiments on 1-D conservation laws to demonstrate the performance of our trained policy network.

Original:  policy network indeed learned to adaptively choose good discrete schemes that offer better results than the state-of-the-art WENO scheme which is 5th 
Revised: Our experimental results show that the trained policy network indeed learned to adaptively choose good discrete schemes that offer better results than the state-of-the-art WENO scheme which is 5th order accurate in space and 4th order accurate in time.

Original: order accurate in space and

Original: 4th

Original: order accurate in

Original: time.

Original: This serves as an evidence that the proposed RL framework has the potential to design high-performance numerical schemes for conservation laws in a data-driven fashion.
Revised: This serves as an evidence that the proposed RL framework has the potential to design high-performance numerical schemes for conservation laws in a data-driven fashion.

Original: Furthermore, the learned policy network generalizes well to other situations such as different initial conditions, mesh sizes, temporal discrete schemes, etc
Revised: Furthermore, the learned policy network generalizes well to other situations such as different initial conditions, mesh sizes, temporal discrete schemes, etc.

Original: The paper ends with a conclusion in section 5, where possible future research directions are also discussed.
Revised: The paper ends with a conclusion in section 5, where possible future research directions are also discussed.

Original:  P RELIMINARIES 2. N OTATIONS
Revised:  P RELIMINARIES 2. N OTATIONS

Original: In this paper, we consider solving the following 1-D conservation laws:
Revised: In this paper, we consider solving the following 1-D conservation laws:

Original: [Equation]
Revised: [Equation]

Original: For example, f = u 2 2 is the famous Burger’s Equation.
Revised: For example, f = u 2 2 is the famous Burger’s Equation.

Original: We discretize the ( x, t ) -plane by choosing a mesh with spatial size ∆ x and temporal step size ∆ t , and deﬁne the discrete mesh points ( x j , t n ) by x j = a + j ∆ x, t n = n ∆ t with j = 0 , 1 , ..., J = b − a ∆ x , n = 0 , 1 , ..., N = T ∆ t.
Revised: We discretize the ( x, t ) -plane by choosing a mesh with spatial size ∆ x and temporal step size ∆ t , and deﬁne the discrete mesh points ( x j , t n ) by x j = a + j ∆ x, t n = n ∆ t with j = 0 , 1 , ..., J = b − a ∆ x , n = 0 , 1 , ..., N = T ∆ t.

Original: We denote x j + 12 = x j + ∆ x/ 2 = a + ( j
Revised: We denote x j + 12 = x j + ∆ x/ 2 = a + ( j

Original: + 12 )∆ x.
Revised: + 12 )∆ x.

Original: The ﬁnite difference methods will produce approximations U nj to the solution u ( x j , t n ) on the given discrete mesh points.
Revised: The ﬁnite difference methods will produce approximations U nj to the solution u ( x j , t n ) on the given discrete mesh points.

Original: We denote pointwise values of the true solution to be u nj = u ( x j , t n ) , and the true point-wise ﬂux values to be f nj = f ( u ( x j , t n )) .
Revised: We denote pointwise values of the true solution to be u nj = u ( x j , t n ) , and the true point-wise ﬂux values to be f nj = f ( u ( x j , t n )) .

Original: 2. WENO – W EIGHTED E SSENTIALLY N ON -O SCILLATORY S CHEMES
Revised: 2. WENO – W EIGHTED E SSENTIALLY N ON -O SCILLATORY S CHEMES

Original: WENO (
Revised: WENO (

Original: Weighted Essentially Non-Oscillatory)
Revised: Weighted Essentially Non-Oscillatory)

Original: (Liu et al., 1994) is a family of high order accurate ﬁnite difference schemes for solving hyperbolic conservation laws, and has been successful for many practical problems.
Revised: (Liu et al., 1994) is a family of high order accurate ﬁnite difference schemes for solving hyperbolic conservation laws, and has been successful for many practical problems.

Original: The key idea of WENO is a nonlinear adaptive procedure that automatically chooses the smoothest local stencil to reconstruct the numerical ﬂux.
Revised: The key idea of WENO is a nonlinear adaptive procedure that automatically chooses the smoothest local stencil to reconstruct the numerical ﬂux.

Original: Generally, a ﬁnite difference method solves Eq.1 by using a conservative approximation to the spatial derivative of the ﬂux:du
Revised: Generally, a ﬁnite difference method solves Eq.1 by using a conservative approximation to the spatial derivative of the ﬂux:

Original: [Equation]
Revised: [Equation]

Original: (2) where u j ( t ) is the numerical approximation to the point value u ( x j , t ) and ˆ f j + 12 is the numerical ﬂux generated by a numerical ﬂux policy
Revised: where u j ( t ) is the numerical approximation to the point value u ( x j , t ) and ˆ f j + 12 is the numerical ﬂux generated by a numerical ﬂux policy

Original: [Equation]
Revised: [Equation]

Original: which is manually designed.
Revised: which is manually designed.

Original: Note that the term “numerical ﬂux policy" is a new terminology that we introduce in this paper, which is exactly the policy we shall learn using RL.
Revised: Note that the term “numerical ﬂux policy" is a new terminology that we introduce in this paper, which is exactly the policy we shall learn using RL.

Original: In WENO, π f works as follows.
Revised: In WENO, π f works as follows.

Original: Using the physical ﬂux values { f j − 2 , f j − 1 , f j } , we could obtain a 3 th order accurate polynomial interpolation ˆ f − 2 j + 12 , where the indices { j − 2 , j − 1 , j } is called a ‘stencil’.
Revised: Using the physical ﬂux values { f j − 2 , f j − 1 , f j } , we could obtain a 3 th order accurate polynomial interpolation ˆ f − 2 j + 12 , where the indices { j − 2 , j − 1 , j } is called a ‘stencil’.

Original: We could also use the stencil { j − 1 , j, j +1 } , { j, j +1 , j +2 } or { j +1 , j +2 , j +3 } to obtain another three interpolants ˆ f − 1 j + 12 , ˆ f 0 j + 

Revised: We could also use the stencil { j − 1 , j, j +1 } , { j, j +1 , j +2 } or { j +1 , j +2 , j +3 } to obtain another three interpolants ˆ f − 1 j + 12 , ˆ f 0 j +

Original: and ˆ f 1 j + 12 .
Revised: and ˆ f 1 j + 12 .

Original: The key idea of WENO is to average (with properly designed weights) all these interpolants to obtain the ﬁnal reconstruction: ˆ {+f j + 
 = (cid:80) [Equation]+}
Revised: The key idea of WENO is to average (with properly designed weights) all these interpolants to obtain the ﬁnal reconstruction: ˆ {+f j + 
 = (cid:80) [Equation]+}

Revised: r

Original: [Equation]
Revised: [Equation]

Revised: 2 w r ˆ f rj +

Revised: [Equation]

Revised: 2 , (cid:80) 1 r = − 2 w r = 1 .

Original: The weight w i depends on the smoothness of the stencil.
Revised: The weight w i depends on the smoothness of the stencil.

Original: A general principal is: the smoother is the stencil, the more accurate is the interpolant and hence the larger is the weight.
Revised: A general principal is: the smoother is the stencil, the more accurate is the interpolant and hence the larger is the weight.

Original: To ensure convergence, we need the numerical scheme to be consistent and stable (LeVeque, 1992).
Revised: To ensure convergence, we need the numerical scheme to be consistent and stable (LeVeque, 1992).

Original: It is known that WENO schemes as described above are consistent.
Revised: It is known that WENO schemes as described above are consistent.

Original: For stability, upwinding is required in constructing the ﬂux.
Revised: For stability, upwinding is required in constructing the ﬂux.

Original: The most easy way is to use the sign of the Roe speed ¯ a j + 

Revised: The most easy way is to use the sign of the Roe speed ¯ a j +

Original: =
Revised: =

Original: ( f j + 12 − f j − 12 ) / ( u j + 12 − u j − 12 ) to determine the upwind direction: if ¯ a j + 12 ≥ 0 , we only average among the three interpolants ˆ f − 2 j + 12 , ˆ f − 1 j + 1 2{+
+} and ˆ f 0 j + 1 2 ; if ¯ a j + 1 2 < 0 , we use ˆ f − 1 j + 1 2 , ˆ f 0 j +1 

Revised: ( f j + 12 − f j − 12 ) / ( u j + 12 − u j − 12 ) to determine the upwind direction: if ¯ a j + 12 ≥ 0 , we only average among the three interpolants ˆ f − 2 j + 12 , ˆ f − 1 j + {+
+} and ˆ f 0 j + 12 ; if ¯ a j + 12 < 0 , we use ˆ f − 1 j + 12 , ˆ f 0 j +

Original: and ˆ f 1 j + 1 2 .
Revised: and ˆ f 1 j + 12 .

Original: Some further thoughts.
Revised: Some further thoughts.

Original: WENO achieves optimal order of accuracy (up to 5) at the smooth region of the solutions (Shu, 1998), while lower order of accuracy at singularities.
Revised: WENO achieves optimal order of accuracy (up to 5) at the smooth region of the solutions (Shu, 1998), while lower order of accuracy at singularities.

Original: The key of the WENO method lies in how to compute the weight vector ( w 1 , w 2 , w 3 , w 4 ) , which primarily depends on the smoothness of the solution at local stencils.
Revised: The key of the WENO method lies in how to compute the weight vector ( w 1 , w 2 , w 3 , w 4 ) , which primarily depends on the smoothness of the solution at local stencils.

Original: In WENO, such smoothness is characterized by handcrafted formula, and was proven to be successful in many practical problems when coupled with high-order temporal discretization.
Revised: In WENO, such smoothness is characterized by handcrafted formula, and was proven to be successful in many practical problems when coupled with high-order temporal discretization.

Original: However, it remains unknown whether there are better ways to combine the stencils so that optimal order of accuracy in smooth regions can be reserved while, at the same time, higher accuracy can be achieved near singularities.
Revised: However, it remains unknown whether there are better ways to combine the stencils so that optimal order of accuracy in smooth regions can be reserved while, at the same time, higher accuracy can be achieved near singularities.

Original: Furthermore, estimating the upwind directions is another key component of WENO, which can get quite complicated in high-dimensional situations and requires lots of logical judgments (i.e. “if/else").
Revised: Furthermore, estimating the upwind directions is another key component of WENO, which can get quite complicated in high-dimensional situations and requires lots of logical judgments (i.e. “if/else").

Original: Can we ease the (some time painful) coding and improve the estimation at the aid of machine learning?
Revised: Can we ease the (some time painful) coding and improve the estimation at the aid of machine learning?

Original:  M ETHODS
Revised:  M ETHODS

Original: In this section we present how to employ reinforcement learning to solve the conservation laws given by Eq.1.
Revised: In this section we present how to employ reinforcement learning to solve the conservation laws given by Eq.1.

Original: To better illustrate our idea, we ﬁrst show in general how to formulate the process of numerically solving a conservation law into an MDP.
Revised: To better illustrate our idea, we ﬁrst show in general how to formulate the process of numerically solving a conservation law into an MDP.

Original: We then discuss how to incorporate a policy network with the WENO scheme.
Revised: We then discuss how to incorporate a policy network with the WENO scheme.

Original: Our policy network targets at the following two key aspects of
Revised: Our policy network targets at the following two key aspects of

Original: WENO: (1) Can we learn to choose better weights to combine the constructed ﬂuxes?
Revised: WENO: (1) Can we learn to choose better weights to combine the constructed ﬂuxes?

Original: (2) Can we learn to automatically judge the upwind direction, without complicated logical judgments?
Revised: (2) Can we learn to automatically judge the upwind direction, without complicated logical judgments?

Original: 3. MDP F ORMULATION
Revised: 3. MDP F ORMULATION

Revised: ﬂux

Original: [-1 Input : initial values u 00 , u 01 , ..., u 0 J , ﬂux f ( u ) , ∆ x , ∆ t , evolve time N , left shift r and right shift s . 2 Output : { U nj | j = 0 , ..., J, n = 1 , ..., N } 3 U 
 j = u 0 j , j = 0 , ..., J 4 for n = 1 to N do for j = 0 to J do-]
Revised: [-1 Input : initial values u 00 , u 01 , ..., u 0 J , ﬂux f ( u ) , ∆ x , ∆ t , evolve time N , left shift r and right shift s . 2 Output : { U nj | j = 0 , ..., J, n = 1 , ..., N } 3 U 
 j = u 0 j , j = 0 , ..., J 4 for n = 1 to N do for j = 0 to J do-]−

Original: Compute the numerical ﬂux
Revised: [Equation] [Equation] [Equation]

Original: ˆ f nj − 12 =
Revised: nj g [Table]

Original: [-π f ( U n − 1 j − r − 1 , U n − 1 j − r , ..., U n − 1 j + s − 1 ) and ˆ f nj + 
-]
Revised: [-π f ( U n − 1 j − r − 1 , U n − 1 j − r , ..., U n − 1 j + s − 1 ) and ˆ f nj + 
-][Table]

Original: =
Revised: =

Original: [-π f ( U n − 1 j − r , U n − 1 j − r +
 ..., U n − 1 j + s ) , e.g., using the WENO scheme-]
Revised: [-π f ( U n − 1 j − r , U n − 1 j − r +
 ..., U n − 1 j + s ) , e.g., using the WENO scheme-][Table]

Original: Compute du j ( t ) dt = − 1∆ x
Revised: − 1 j + ∆ t du j ( t ) dt [Table]

Original: ( ˆ f nj + 12 − ˆ f nj − 12 )
Revised: [Table] [Table] [Table]

Original: Compute U nj = π t ( U n − 1 j , du j ( t ) dt ) , e.g., using the Euler scheme U nj = U n − 1 j + ∆ t du j ( t ) dt 9 Return U nj | j = 0 , ..., J, n = 1 , ..., N }
Revised: [Table]

Original: As shown in Algorithm 1, the procedure of numerically solving a conservation law is naturally a sequential decision making problem.
Revised: As shown in Algorithm 1, the procedure of numerically solving a conservation law is naturally a sequential decision making problem.

Original: The key of the procedure is the numerical ﬂux policy π f and the temporal scheme π t as shown in line 6 and 8 in Algorithm 1.
Revised: The key of the procedure is the numerical ﬂux policy π f and the temporal scheme π t as shown in line 6 and 8 in Algorithm 1.

Original: Both policies could be learned using
Revised: Both policies could be learned using

Original: RL.
Revised: RL.

Original: However, in this paper, we mainly focus on using RL to learn the numerical ﬂux policy π f , while leaving the temporal scheme π t with traditional numerical schemes such as the Euler scheme or the
Revised: However, in this paper, we mainly focus on using RL to learn the numerical ﬂux policy π f , while leaving the temporal scheme π t with traditional numerical schemes such as the Euler scheme or the

Original: Runge– Kutta methods.
Revised: Runge–Kutta methods.

Original: A quick review of RL is given in the appendix.
Revised: A quick review of RL is given in the appendix.

Original: Now, we show how to formulate the above procedure as an MDP and the construction of the state S , action A , reward r and transition dynamics P .
Revised: Now, we show how to formulate the above procedure as an MDP and the construction of the state S , action A , reward r and transition dynamics P .

Original: Algorithm 2 shows in general how RL is incorporated into the procedure.
Revised: Algorithm 2 shows in general how RL is incorporated into the procedure.

Original: In Algorithm 2, we use a single RL agent.
Revised: In Algorithm 2, we use a single RL agent.

Original: Speciﬁcally, when computing U nj :
Revised: Speciﬁcally, when computing U nj :

Original: • The state for the RL agent is s nj =
Revised: • The state for the RL agent is s nj =

Original: g s ( U n − 1 j − r − 1 , ..., U n − 1 j + s ) , where g s is the state function.
Revised: g s ( U n − 1 j − r − 1 , ..., U n − 1 j + s ) , where g s is the state function.

Original: • In general, the action of the agent is used to determine how the numerical ﬂuxes
Revised: • In general, the action of the agent is used to determine how the numerical ﬂuxes

Original: ˆ f nj + 

Revised: ˆ f nj +

Original: and ˆ f nj − 12 is computed.
Revised: and ˆ f nj − 12 is computed.

Original: In the next subsection, we detail how we incorporate a nj to be the linear weights of the ﬂuxes computed using different stencils in the WENO scheme.
Revised: In the next subsection, we detail how we incorporate a nj to be the linear weights of the ﬂuxes computed using different stencils in the WENO scheme.

Original: • The reward should encourage the agent to generate a scheme that minimizes the error between its approximated value and the true value.
Revised: • The reward should encourage the agent to generate a scheme that minimizes the error between its approximated value and the true value.

Original: Therefore, we deﬁne the reward function as r nj = g r
Revised: Therefore, we deﬁne the reward function as r nj = g r

Original: ( U nj − r − 1 − u nj − r − 1 , · · · , U nj + s − u nj + s ) , e.g., a simplest choice is g r = −|| · || 2 .
Revised: ( U nj − r − 1 − u nj − r − 1 , · · · , U nj + s − u nj + s ) , e.g., a simplest choice is g r = −|| · || 2 .

Original: • The transition dynamics P is fully deterministic, and depends on the choice of the temporal scheme at line 10 in Algorithm 2.
Revised: • The transition dynamics P is fully deterministic, and depends on the choice of the temporal scheme at line 10 in Algorithm 2.

Original: Note that the next state can only be constructed when we have obtained all the point values in the next time step, i.e., s n +1 j = g s ( U nj − r − 1 , ..., U nj + s ) does not only depends on action a nj , but also on actions
Revised: Note that the next state can only be constructed when we have obtained all the point values in the next time step, i.e., s n +1 j = g s ( U nj − r − 1 , ..., U nj + s ) does not only depends on action a nj , but also on actions

Original: a nj − r − 1 , ..., a nj + s (action a nj can only determine the value U nj ).
Revised: a nj − r − 1 , ..., a nj + s (action a nj can only determine the value U nj ).

Original: This subtlety can be resolved by viewing the process under the framework of multi-agent RL, in which at each mesh point j we use a distinct agent A RLj , and the next state s n + j = g s ( U nj − r − 1 , ..., U nj + s ) depends on these agents’ joint action a nj = ( a nj − r − 1 , ..., a nj + s ) .
Revised: This subtlety can be resolved by viewing the process under the framework of multi-agent RL, in which at each mesh point j we use a distinct agent A RLj , and the next state s n +1 j = g s ( U n j − r − 1 , ..., U n j + s ) depends on these agents’ joint action a n j = ( a n j − r − 1 , ..., a n j + s ) .

Original: However, it is impractical to train J different agents as J is usually very large, therefore we enforce the agents at different mesh point j to share the same weight, which reduces to case of using just a single agent.
Revised: However, it is impractical to train J different agents as J is usually very large, therefore we enforce the agents at different mesh point j to share the same weight, which reduces to case of using just a single agent.

Original: The single agent can be viewed as a counterpart of a human designer who decides on the choice of a local scheme based on the current state in traditional numerical methods. 
Revised: The single agent can be viewed as a counterpart of a human designer who decides on the choice of a local scheme based on the current state in traditional numerical methods. j = − 1∆ x ( ˆ f nj + 12 − ˆ f nj −

Revised: ∆ trained RL policy π RL .

Original: 3. RL E MPOWERED WENO
Revised: 3. RL E MPOWERED WENO

Original: We now present how to transfer the actions of the RL policy to the weights of WENO ﬂuxes.
Revised: We now present how to transfer the actions of the RL policy to the weights of WENO ﬂuxes.

Original: Instead of directly using π RL to generate the numerical ﬂux, we use it to produce the weights of numerical ﬂuxes computed using different stencils in WENO.
Revised: Instead of directly using π RL to generate the numerical ﬂux, we use it to produce the weights of numerical ﬂuxes computed using different stencils in WENO.

Original: Since the weights are part of the conﬁgurations of the WENO scheme, our design of action essentially makes the RL policy a meta-learner, and enables more stable learning and better generalization power than directly generating the ﬂuxes.
Revised: Since the weights are part of the conﬁgurations of the WENO scheme, our design of action essentially makes the RL policy a meta-learner, and enables more stable learning and better generalization power than directly generating the ﬂuxes.

Original: Speciﬁcally, at point x j (here we drop the time superscript n for simplicity), to compute the numerical ﬂux ˆ f j − 12 and ˆ f j + 12 , we ﬁrst construct four ﬂuxes { ˆ f ij − 12 } 1 i = − 2 and { ˆ f ij + 12 } 1 i = − 2 using four different stencils just as in WENO, and then use the RL policy π RL to generate the weights of these ﬂuxes:[Equation] [Equation] [Equation]
Revised: Speciﬁcally, at point x j (here we drop the time superscript n for simplicity), to compute the numerical ﬂux ˆ f j − 12 and ˆ f j + 12 , we ﬁrst construct four ﬂuxes { ˆ f ij − 12 } 1 i = − 2 and { ˆ f ij + 12 } 1 i = − 2 using four different stencils just as in WENO, and then use the RL policy π RL to generate the weights of these ﬂuxes:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The numerical ﬂux is then constructed by averaging these ﬂuxes: ˆ f j − 12 = (cid:80) 1 i = − 2 w ij − 12 ˆ f ij − 12 , and ˆ f j + 12 = (cid:80) 

Revised: The numerical ﬂux is then constructed by averaging these ﬂuxes: ˆ f j − 12 = (cid:80) 1 i = − 2 w ij − 12 ˆ f i j − 12 , and ˆ f j + 12 = (cid:80)

Original: i = − 2 w ij + 12 ˆ f ij + 12 .
Revised: i = − 2 w ij + 12 ˆ f ij + 12 .

Original: Note that the determination of upwind direction is automatically embedded in the RL policy since it generates four weights at once.
Revised: Note that the determination of upwind direction is automatically embedded in the RL policy since it generates four weights at once.

Original: For instance, when the roe speed ¯ a j + 12 ≥ 0 , we expect the 4 th weight w 1 j + 

Revised: For instance, when the roe speed ¯ a j + 12 ≥ 0 , we expect the 4 th weight w 1 j +

Original: ≈ 0 and when ¯ a j + 12 < 0 , we expect w − 2 j +
Revised: ≈ 0 and when ¯ a j + 12 < 0 , we expect w − 2 j +

Original: ≈ 0 .
Revised: ≈ 0 .

Original: Note that the upwind direction can be very complicated in a system of equations or in the high-dimensional situations, and using the policy network to automatically embed such a process could save lots of efforts in algorithm design and implementation.
Revised: Note that the upwind direction can be very complicated in a system of equations or in the high-dimensional situations, and using the policy network to automatically embed such a process could save lots of efforts in algorithm design and implementation.

Original: Our numerical experiments show that π RL can indeed automatically determine upwind directions for 1D scalar cases.
Revised: Our numerical experiments show that π RL can indeed automatically determine upwind directions for 1D scalar cases.

Original: Although this does not mean that it works for systems and/or in high-dimensions, it shows the potential of the proposed framework and value for further studies.  
Revised: Although this does not mean that it works for systems and/or in high-dimensions, it shows the potential of the proposed framework and value for further studies.  E XPERIMENTS

Revised: In this section, we describe training and testing of the proposed RL conservation law solver and compare it with WENO.

Revised: More comparisons and discussions can be found in the appendix.

Original: E XPERIMENTS 4. S ETUP
Revised: 4. S ETUP

Original: In this subsection, we explain the general training setup.
Revised: In this subsection, we explain the general training setup.

Original: We train the RL policy network on the
Revised: We train the RL policy network on the

Original: Burger’s equation, whose ﬂux is computed as f ( u ) = 12 u 2 .
Revised: Burger’s equation, whose ﬂux is computed as f ( u ) = 12 u 2 .

Original: In all the experiments, we set the left-shift r = 2 and the right shift s = 3 .
Revised: In all the experiments, we set the left-shift r = 2 and the right shift s = 3 .

Original: The state function g s ( s j ) =
Revised: The state function g s ( s j ) =

Original: g s ( U j − r − 1 , ..., U j + s ) will generate two vectors: s l
Revised: g s ( U j − r − 1 , ..., U j + s ) will generate two vectors: s l

Original: = ( f j − r − 1 , ..., f j + s − 1 , ¯ a j − 12 ) , and s r =
Revised: = ( f j − r − 1 , ..., f j + s − 1 , ¯ a j − 12 ) , and s r =

Original: ( f j − r , ..., f j + s , ¯ a j + 12 ) for computing ˆ f j − 12 and ˆ f j + 1 2 respectively.
Revised: ( f j − r , ..., f j + s , ¯ a j + 12 ) for computing ˆ f j − 12 and ˆ f j + 12 respectively.

Original: s l and s r will be passed into the same policy neural network π RL θ to produce the desired actions, as described in section 3.2.
Revised: s l and s r will be passed into the same policy neural network π RLθ to produce the desired actions, as described in section 3.2.

Original: The reward function g r simply computes the inﬁnity norm, i.e., g r ( U j − r − 1 − u j − r − 1 , ..., U j + s − u j + s )
Revised: The reward function g r simply computes the inﬁnity norm, i.e., g r ( U j − r − 1 − u j − r − 1 , ..., U j + s − u j + s )

Original: =
Revised: =

Original: −|| ( U j − r − 1 − u j − r − 1 , ..., U j + s − u j + s ) || ∞ .
Revised: −|| ( U j − r − 1 − u j − r − 1 , ..., U j + s − u j + s ) || ∞ .

Original: The policy network π RLθ is a feed-forward Multi-layer Perceptron with 6 hidden layers, each has 64 neurons and use Relu (Goodfellow et al., 2016) as the activation function.
Revised: The policy network π RLθ is a feed-forward Multi-layer Perceptron with 6 hidden layers, each has 64 neurons and use Relu (Goodfellow et al., 2016) as the activation function.

Original: We use the Deep
Revised: We use the Deep

Original: Deterministic Policy Gradient Algorithm (Lillicrap et al., 2015) to train the RL policy.
Revised: Deterministic Policy Gradient Algorithm (Lillicrap et al., 2015) to train the RL policy.

Original: To guarantee the generalization power of the trained RL agent, we randomly sampled 20 initial conditions in the form u 0 ( x ) = a + b · func ( cπx ) , where | a | + | b | ≤ 3 . 5 , func ∈ { sin, cos } and c ∈ { 2 , 4 , 6 } .
Revised: To guarantee the generalization power of the trained RL agent, we randomly sampled 20 initial conditions in the form u 0 ( x ) = a + b · func ( cπx ) , where | a | + | b | ≤ 3 . 5 , func ∈ { sin, cos } and c ∈ { 2 , 4 , 6 } .

Original: The goal of generating such kind of initial conditions is to ensure they have similar degree of smoothness and thus similar level of difﬁculty in learning.
Revised: The goal of generating such kind of initial conditions is to ensure they have similar degree of smoothness and thus similar level of difﬁculty in learning.

Original: The computation domain is − 1 ≤ x ≤ 1 and 0 ≤ t ≤ 0 . 8 with ∆ x = 0 .
Revised: The computation domain is − 1 ≤ x ≤ 1 and 0 ≤ t ≤ 0 . 8 with ∆ x = 0 .

Original: 02 , ∆ t = 0 .
Revised: 02 , ∆ t = 0 .

Original: 004 , and evolve steps N = 200 (which ensures the appearance of shocks).
Revised: 004 , and evolve steps N = 200 (which ensures the appearance of shocks).

Original: When training the RL agent, we use the Euler scheme for temporal discretization.
Revised: When training the RL agent, we use the Euler scheme for temporal discretization.

Original: The true solution needed for reward computing is generated using WENO on the same computation domain with ∆ x = 0 .
Revised: The true solution needed for reward computing is generated using WENO on the same computation domain with ∆ x = 0 .

Original: 001 , ∆ t = 0 . 

Revised: 001 , ∆ t = 0 .

Original: and the 4th order Runge-Kutta (RK4).
Revised: and the 4th order Runge-Kutta (RK4).

Original: In the following, we denote the policy network that generates the weights of the WENO ﬂuxes (as described in section 3.2) as RL-WENO.
Revised: In the following, we denote the policy network that generates the weights of the WENO ﬂuxes (as described in section 3.2) as RL-WENO.

Original: We randomly generated another different 10 initial conditions in the same form as training for testing.
Revised: We randomly generated another different 10 initial conditions in the same form as training for testing.

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: R ESULTS
Revised: R ESULTS

Original: We compare the performance of RL-WENO and WENO.
Revised: We compare the performance of RL-WENO and WENO.

Original: We also test whether the trained RL policy can generalize to different temporal discretization schemes, mesh sizes and ﬂux functions that are not included in training.
Revised: We also test whether the trained RL policy can generalize to different temporal discretization schemes, mesh sizes and ﬂux functions that are not included in training.

Original: Table 1 and Table 2 present the comparison results, where the number shows the relative error (computed as || U − u
Revised: Table 1 and Table 2 present the comparison results [Table] where the number shows the relative error (computed as || U − u

Original: || 

Revised: ||

Original: || u ||with the 2-norm taking over all x ) between theapproximated solution U and the true solution u , averaged over 250 evolving steps ( T = 1 . 0 ) and 10 random initial values. Numbers in the bracket shows the standard deviation over the 10 initial conditions. Several entries in the table are marked as ‘-’ because the corresponding CFL number ( dt/dx ) is not small enough to guarantee convergence.
Revised: || u ||with the 2-norm taking over all x ) between the to guarantee convergence.

Original: Recall that training of the RL-WENO was conducted with Euler time discretization, (∆ x, ∆ t ) =
Revised: Recall that training of the RL-WENO was conducted with Euler time discretization, (∆ x, ∆ t ) =

Original: , T = 0 .
Revised: , T = 0 .

Original: 8 and f ( u ) = 12 u 2 .
Revised: 8 and f ( u ) = 12 u 2 .

Original: Our experimental results show that, compared with the high order accurate WENO (5th order accurate in space and 4th order accurate in time), the linear weights learned by RL not only achieves smaller errors, but also generalizes well to: 1) longer evolving time
Revised: Our experimental results show that, compared with the high order accurate WENO (5th order accurate in space and 4th order accurate in time), the linear weights learned by RL not only achieves smaller errors, but also generalizes well to: 1) longer evolving time

Original: ( T = 0 . 8 for training and T = 1 . 0 for testing); 2) new time discretization schemes (trained on Euler, tested on RK4); 3) new mesh sizes (see Table 1 and Table 2 for results of varied ∆ x and ∆ t ); and 4) a new ﬂux function (trained on f ( u ) = 12 u 2 shown in Table 1, tested on 116 u 4 Table 2).
Revised: ( T = 0 . 8 for training and T = 1 . 0 for testing); 2) new time discretization schemes (trained on Euler, tested on RK4); 3) new mesh sizes (see Table 1 and Table 2 for results of varied ∆ x and ∆ t ); and 4) a new ﬂux function (trained on f ( u ) = 12 u 2 shown in Table 1, tested on 116 u 4 Table 2).

Original: Figure 1 shows some examples of the solutions.
Revised: Figure 1 shows some examples of the solutions.

Original: As one can see, the solutions generated by RL-WENO not only achieve the same accuracy as WENO at smooth regions, but also have clear advantage over
Revised: As one can see, the solutions generated by RL-WENO not only achieve the same accuracy as WENO at smooth regions, but also have clear advantage over

Original: WENO near singularities which is particularly challenging for numerical PDE solvers and important in applications.
Revised: WENO near singularities which is particularly challenging for numerical PDE solvers and important in applications.

Original: Figure 2 shows that the learned numerical ﬂux policy can indeed correctly determine upwind directions and generate local numerical schemes in an adaptive fashion.
Revised: Figure 2 shows that the learned numerical ﬂux policy can indeed correctly determine upwind directions and generate local numerical schemes in an adaptive fashion.

Original: More interestingly,
Revised: More interestingly,

Original: Figure 2 further shows that comparing to WENO, RL-WENO seems to be able to select stencils in a different way from it, and eventually leads to a more accurate solution.
Revised: Figure 2 further shows that comparing to WENO, RL-WENO seems to be able to select stencils in a different way from it, and eventually leads to a more accurate solution.

Original: This shows that the proposed RL framework has the potential to surpass human experts in designing numerical schemes for conservation laws.
Revised: This shows that the proposed RL framework has the potential to surpass human experts in designing numerical schemes for conservation laws.

Original: [Equation]
Revised: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper we proposed a general framework to learn how to solve 1-dimensional conservation laws via deep reinforcement learning.
Revised: In this paper, we proposed a general framework to learn how to solve 1-dimensional conservation laws via deep reinforcement learning.

Original: We ﬁrst discussed how the procedure of numerically solving conservation laws can be naturally cast in the form of Markov Decision Process.
Revised: We ﬁrst discussed how the procedure of numerically solving conservation laws can be naturally cast in the form of Markov Decision Process.

Original: We then elaborated how to relate notions in numerical schemes of PDEs with those of reinforcement learning.
Revised: We then elaborated how to relate notions in numerical schemes of PDEs with those of reinforcement learning.

Original: In particular, we introduced a numerical ﬂux policy which was able to decide on how numerical ﬂux should be designed locally based on the current state of the solution.
Revised: In particular, we introduced a numerical ﬂux policy which was able to decide on how numerical ﬂux should be designed locally based on the current state of the solution.

Original: We carefully design the action of our RL policy to make it a meta-learner.
Revised: We carefully design the action of our RL policy to make it a meta-learner.

Original: Our numerical experiments showed that the proposed RL based solver was able to outperform high order WENO and was well generalized in various cases.
Revised: Our numerical experiments showed that the proposed RL based solver was able to outperform high order WENO and was well generalized in various cases.

Original: As part of the future works, we would like to consider using the numerical ﬂux policy to inference more complicated numerical ﬂuxes with guaranteed consistency and stability.
Revised: As part of the future works, we would like to consider using the numerical ﬂux policy to inference more complicated numerical ﬂuxes with guaranteed consistency and stability.

Original: Furthermore, we can use the proposed framework to learn a policy that can generate adaptive grids and the associated numerical schemes.
Revised: Furthermore, we can use the proposed framework to learn a policy that can generate adaptive grids and the associated numerical schemes.

Original: Lastly, we would like consider system of conservation laws in 2nd and 3rd dimensional space.
Revised: Lastly, we would like consider system of conservation laws in 2nd and 3rd dimensional space.

Original: A R EVIEW OF R EINFORCEMENT L EARNING A. R EINFORCEMENT L EARNING
Revised: A C OMPLEMENTARY E XPERIMENTS A. C OMPARISON WITH SUPERVISED LEARNING (SL) BASED METHODS

Revised: We ﬁrst note that most of the neural network based numerical PDE solvers cited in the introduction requires retraining when the initialization, terminal time, or the form of the PDE is changed; while the proposed RL solver is much less restricted as shown in our numerical experiments. This makes proper comparisons between existing NN-based solvers and our proposed solver very difﬁcult. Therefore, to demonstrate the advantage of our proposed RL PDE solver, we would like to propose a new SL method that does not require retraining when the test setting (e.g. initialization, ﬂux function, etc.) is different from the training.

Revised: However, as far as we are concerned, it is challenging to design such SL methods without formulating the problem into an MDP.

Revised: One may think that we can use WENO to generate the weights for the stencil at a particular grid point on a dense grid, and use the weights of WENO generated from the dense grid as the label to train a neural network in the coarse grid.

Revised: But such setting has a fatal ﬂaw in that the stencils computed in the dense grids are very different from those in the coarse grids, especially near singularities.

Revised: Therefore, good weights on dense grids might perform very poorly on coarse grids.

Revised: In other words, simple imitation of WENO on dense grids is not a good idea.

Revised: One might also argue that instead of learning the weights of the stencils, we could instead generate the discrete operators, such as the spatial discretization of ∂u j ∂x , or the temporal discretization of ∂u j ∂t , the numerical ﬂuxes f j + 12 ( u ) , f j − 12 ( u ) , etc., on a dense grid, and then use them as labels to train a neural network in the supervised fashion on a coarse grid.

Revised: However, the major problem with such design is that there is no guarantee that the learned discrete operators obey the conservation property of the equations, and thus they may also generalize very poorly.

Revised: After formulating the problem into a MDP, there is indeed one way that we can use back-propagation (BP) instead of RL algorithms to optimize the policy network.

Revised: Because all the computations on using the stencils to calculate the next-step approximations are differentiable, we can indeed use SL to train the weights.

Revised: One possible way is to minimize the error (e.g. 2 norm) between the approximated and the true values, where the true value is pre-computed using a more accurate discretization on a ﬁne mesh.

Revised: The framework to train the SL network is described in Algorithm 3.

Revised: Note that the framework to train the SL network is essentially the same as that of the proposed RL-WENO (Algorithm 2).

Revised: The only difference is that we train the SL network using BP and the RL network using DDPG.

Revised: Algorithm 3: Using BP instead of RL algorithm to train the policy do do do

Revised: Compute the weights ( w n, − 2 j − 12 , w n, − 1 j − 12 , w n, 0 j − 12 , w n, 1 j − 12 , w n, − 2 j + 12 , w n, − 1 j + 12 , w n, 0 j + 12 , w n, 1 j + 12 ) =

Revised: π θ ( s nj )

Revised: Compute the ﬂuxes ˆ f nj − 12 =

Original: {+(cid:80) 1 i = − 2 w n,i j − 12 ˆ f n,i j − 12 , ˆ f j + 12 = (cid:80) 
+}
Revised: {+(cid:80) 1 i = − 2 w n,i j − 12 ˆ f n,i j − 12 , ˆ f j + 12 = (cid:80) 
+}

Revised: i = − 2 w ij + 12 ˆ f n,i j + 12 , where ˆ f n,i j ± 12 are the ﬂuxes computed by WENO

Revised: Compute du j ( t ) dt

Revised: [Equation]

Revised: ˆ f nj

Revised: [Equation]

Revised: Compute U nj =

Revised: π t ( U n − 1 j , du j ( t ) dt ) , e.g., the Euler scheme U nj =

Revised: U n − 1 j + ∆ t du j ( t ) dt

Revised: Compute the loss for θ :

Revised: [Equation]

Revised: +

Revised: [Equation]

Revised: u

Revised: [Equation]

Revised: [Equation]

Revised: | 22 .

Revised: Perform a gradient descent on θ w.r.t L nj ( θ )

Revised: Construct the next states s n +1 j = g s ( u nj − r − 1 , ..., u nj + s ) for j = 0 , ..., J 15 Return the BP optimized policy π θ .

Revised: However, we argue that the main drawback of using SL (BP) to optimize the stencils in such a way is that it cannot enforce long-term accuracy and thus cannot outperform the proposed RL-WENO.

Revised: To support such claims, we have added experiments using SL to train the weights of the stencils, and the results are shown in table 3 and 4.

Revised: The SL policy is trained till it achieves very low loss (i.e., converges) in the training setting.

Revised: However, as shown in the table, the SL-trained policy does not perform well overall.

Revised: To improve longer time stability, one may argue that we could design the loss of SL to be the accumulated loss over multiple prediction steps, but in practice as the dynamics of our problem (computations for obtaining multiple step approximations) is highly non-linear, thus the gradient ﬂow through multiple steps can be highly numerically unstable, making it difﬁcult to obtain a decent result.

Revised: [Table]

Revised: [Table]

Revised: A. RL- WENO ’ S PERFORMANCE ON SMOOTH AND SINGULAR REGIONS

Revised: As mentioned in section 2.2, WENO itself already achieves an optimal order of accuracy in the smooth regions.

Revised: Since RL-WENO can further improve upon WENO, it must have obtained higher accuracy especially near singularities.

Revised: Here we provide additional demonstrations on how RL-WENO performs in the smooth/singular regions.

Revised: We run RL-WENO and WENO on a set of initial conditions, and record the approximation errors at every locations and then separate the errors in the smooth and singular regions for every time step.

Revised: We then compute the distribution of the errors on the entire spatial-temporal grids with multiple initial conditions.

Revised: The results are shown in ﬁgure 3.

Revised: In ﬁgure 3, the x -axis is the logarithmic (base 10) value of the error and the y-axis is the number of grid points whose error is less than the corresponding value on the x -axis, i e., the accumulated distribution of the errors.

Revised: The results show that RL-WENO indeed performs better than WENO near singularities

Revised: A. I NFERENCE T IME OF RL-WENO AND WENO

Revised: In this subsection we report the inference time of RL-WENO and WENO.

Revised: Although the computation complexity of the trained RL policy (a MLP) is higher than that of WENO, we could parallel and accelerate the computations using GPU.

Revised: Our test is conducted in the following way: for each grid size ∆ x , we ﬁx the initial condition as u 0 ( x ) = 1 + cos (6 πx ) , the evolving time T = 0 .

Revised: 8 and the ﬂux function f = u 2 .

Revised: We then use

Revised: RL-WENO and WENO to solve the problem 20 times, and report the average running time.

Revised: For completeness, we also report the relative error of RL-WENO and WENO in each of these grid sizes in table 6.

Revised: Note that the relative error is computed on average of several initial functions, and our

Revised: RL-WENO policy is only trained on grid (∆ x, ∆ t ) =

Revised: [Figure]

Revised: RL-WENO we test it on both and on

Revised: [Table]

Revised: [Table]

Revised: From the table we can tell that as ∆ x decreases, i.e., as the grid becomes denser, all methods, except for the RL-WENO (GPU), requires signiﬁcant more time to ﬁnish the computation.

Revised: The reason that the time cost of the GPU-version of RL-WENO does not grow is that on GPU, we can compute all approximations in the next step (i.e., to compute ( U t + , U t + , ..., U t +1 J ) given ( U t 0 , U t 1 , ..., U tJ ) , which dominates the computation cost of the algorithm) together in parallel.

Revised: Thus, the increase of grids does not affect much of the computation time.

Revised: Therefore, for coarse grid, well-optimized

Revised: WENO indeed has clear speed advantage over RL-WENO (even on GPU), but on a much denser grid,

Revised: RL-WENO (GPU) can be faster than well-optimized WENO by leveraging the paralleling nature of the algorithm.

Revised: B R EVIEW OF R EINFORCEMENT L EARNING B. R EINFORCEMENT L EARNING

Original: Reinforcement Learning (RL) is a general framework for solving sequential decision making problems.
Revised: Reinforcement Learning (RL) is a general framework for solving sequential decision making problems.

Original: Recently, combined with deep neural networks, RL has achieved great success in various tasks such as playing video games from raw screen inputs (Mnih et al., 2015), playing Go (Silver et al., 2016), and robotics control (Schulman et al., 2017).
Revised: Recently, combined with deep neural networks, RL has achieved great success in various tasks such as playing video games from raw screen inputs (Mnih et al., 2015), playing Go (Silver et al., 2016), and robotics control (Schulman et al., 2017).

Original: The sequential decision making problem RL tackles is usually formulated as a Markov Decision Process (MDP), which comprises ﬁve elements: the state space S , the action space A , the reward r : S ×
Revised: The sequential decision making problem RL tackles is usually formulated as a Markov Decision Process (MDP), which comprises ﬁve elements: the state space S , the action space A , the reward r : S ×

Original: A → R , the transition probability of the environment
Revised: A → R , the transition probability of the environment

Original: P : S ×
Revised: P : S ×

Original: A × S → [0 , 1] , and the discounting factor γ .
Revised: A × S → [0 , 1] , and the discounting factor γ .

Original: The interactions between an RL agent and the environment forms a trajectory τ =
Revised: The interactions between an RL agent and the environment forms a trajectory τ =

Original: ( s 0 , a 0 , r 0 , ..., s T , a T , r T , ... ) .
Revised: ( s 0 , a 0 , r 0 , ..., s T , a T , r T , ... ) .

Original: The return of τ is the discounted sum of all its future rewards:
Revised: The return of τ is the discounted sum of all its future rewards:

Original: [Equation]
Revised: [Equation]

Original: Similarly, the return of a state-action pair ( s t , a t ) is:
Revised: Similarly, the return of a state-action pair ( s t , a t ) is:

Original: [Equation]
Revised: [Equation]

Original: A policy π in RL is a probability distribution on the action A given a state S :
Revised: A policy π in RL is a probability distribution on the action A given a state S :

Original: π : S × A → [0 , 1] .
Revised: π : S × A → [0 , 1] .

Original: We say a trajectory τ is generated under policy π if all the actions along the trajectory is chosen following π , i.e., τ ∼ π means a t ∼ π ( ·| s t ) and s t +1 ∼ P ( ·| s t , a t ) .
Revised: We say a trajectory τ is generated under policy π if all the actions along the trajectory is chosen following π , i.e., τ ∼ π means a t ∼ π ( ·| s t ) and s t +1 ∼ P ( ·| s t , a t ) .

Original: Given a policy π , the value of a state s is deﬁned as the expected return of all the trajectories when the agent starts at s and then follows π :
Revised: Given a policy π , the value of a state s is deﬁned as the expected return of all the trajectories when the agent starts at s and then follows π :

Original: [Equation]
Revised: [Equation]

Original: Similarly, the value of a state-action pair is deﬁned as the expected return of all trajectories when the agent starts at s , takes action a , and then follows π :
Revised: Similarly, the value of a state-action pair is deﬁned as the expected return of all trajectories when the agent starts at s , takes action a , and then follows π :

Original: [Equation]
Revised: [Equation]

Original: As aforementioned in introduction, in most RL algorithms the policy π is optimized with regards to the values Q π ( s, a ) , thus naturally guarantees the long-term accumulated rewards (in our setting, the long-term accuracy of the learned schemes).
Revised: As aforementioned in introduction, in most RL algorithms the policy π is optimized with regards to the values Q π ( s, a ) , thus naturally guarantees the long-term accumulated rewards (in our setting, the long-term accuracy of the learned schemes).

Original: Bellman Equation, one of the most important equations in RL, connects the value of a state and the value of its successor state:
Revised: Bellman Equation, one of the most important equations in RL, connects the value of a state and the value of its successor state:

Original: [Equation]
Revised: [Equation]

Original: The goal of RL is to ﬁnd a policy π to maximize the expected discounted sum of rewards starting from the initial state s 0 , J ( π [Equation] =
Revised: The goal of RL is to ﬁnd a policy π to maximize the expected discounted sum of rewards starting from the initial state s 0 , J ( π ) =

Original: E [Equation] ∼ 
Revised: E s 0 ∼ ρ

Original: [Equation] [Equation] s 0 )] , where ρ is the initial state distribution.
Revised: [ V π ( s 0 )] , where ρ is the initial state distribution.

Original: If we parameterize π using θ , then we can optimize it using the famous policy gradient theorem:
Revised: If we parameterize π using θ , then we can optimize it using the famous policy gradient theorem:

Original: [Equation]
Revised: [Equation]

Original: where ρ π θ is the state distribution deduced by the policy π θ .
Revised: where ρ π θ is the state distribution deduced by the policy π θ .

Original: In this paper we focus on the case where the action space
Revised: In this paper we focus on the case where the action space

Original: A is continuous, and a lot of mature algorithms has been proposed for such a case, e.g., the Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al., 2015), the Trust Region Policy
Revised: A is continuous, and a lot of mature algorithms has been proposed for such a case, e.g., the Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al., 2015), the Trust Region Policy

Original: Optimization algorithm (Schulman et al., 2015), and etc.
Revised: Optimization algorithm (Schulman et al., 2015), and etc.