Original: Deep Reinforcement Learning for Online Control of Stochastic Partial Differential Equations
Revised: Deep Reinforcement Learning for Online Control of Stochastic Partial Differential Equations

Original: Abstract
Revised: Abstract

Original: In many areas, such as the physical sciences, life sciences, and ﬁnance, control approaches are used to achieve a desired goal in complex dynamical systems governed by differential equations.
Revised: In many areas, such as the physical sciences, life sciences, and ﬁnance, control approaches are used to achieve a desired goal in complex dynamical systems governed by differential equations.

Original: In this work we formulate the problem of controlling stochastic partial differential equations (SPDE) as a reinforcement learning problem.
Revised: In this work we formulate the problem of controlling stochastic partial differential equations (SPDE) as a reinforcement learning problem.

Original: We present a learning-based, distributed control approach for online control of a system of SPDEs with high dimensional state-action space using deep deterministic policy gradient method.
Revised: We present a learning-based, distributed control approach for online control of a system of SPDEs with high dimensional state-action space using deep deterministic policy gradient method.

Original: We tested the performance of our method on the problem of controlling the stochastic Burgers’ equation, describing a turbulent ﬂuid ﬂow in an inﬁnitely large domain.
Revised: We tested the performance of our method on the problem of controlling the stochastic Burgers’ equation, describing a turbulent ﬂuid ﬂow in an inﬁnitely large domain.

Original:  Introduction
Revised:  Introduction

Original: Partial differential equations (PDE) are commonly used to explain complex dynamical systems such as ﬂuid dynamics, electromagnetism, etc.
Revised: Partial differential equations (PDE) are commonly used to explain complex dynamical systems such as ﬂuid dynamics, electromagnetism, etc.

Original: While PDEs successfully describe a wide range of dynamical systems, random perturbations require further modelling in the governing equations of some dynamical systems, e g., turbulent ﬂows.
Revised: While PDEs successfully describe a wide range of dynamical systems, random perturbations require further modelling in the governing equations of some dynamical systems, e g., turbulent ﬂows.

Original: Stochastic partial differential equations (SPDEs) are used to model the inherent stochasticities in dynamical systems (Walsh [1986]).
Revised: Stochastic partial differential equations (SPDEs) are used to model the inherent stochasticities in dynamical systems (Walsh [1986]).

Original: Drawing by recent advances in design and control of the dynamical systems in engineering applications, PDE control has attracted a lot of attention (Krsti ´ c [2008], Ahuja et al.
Revised: Drawing by recent advances in design and control of the dynamical systems in engineering applications, PDE control has attracted a lot of attention (Krsti ´ c [2008], Ahuja et al.

Original: [2011], Brunton and Noack [2015], Burns and Hu [2013]); whereas control of SPDEs remains scarce.
Revised: [2011], Brunton and Noack [2015], Burns and Hu [2013]); whereas control of SPDEs remains scarce.

Original: PDE control methods are incorporated successfully in many engineering applications; however, the majority of them require far too much knowledge of the system, including a complete model of the PDE and the design of a customized controller, both of which are computationally expensive and not robust to environmental changes.
Revised: PDE control methods are incorporated successfully in many engineering applications; however, the majority of them require far too much knowledge of the system, including a complete model of the PDE and the design of a customized controller, both of which are computationally expensive and not robust to environmental changes.

Original: Reinforcement learning (RL) was recently used for PDE control with the application of heating, ventilating, air conditioning (HVAC) control design in a room as an example(Farahmand et al.
Revised: Reinforcement learning (RL) was recently used for PDE control with the application of heating, ventilating, air conditioning (HVAC) control design in a room as an example(Farahmand et al.

Original: [2016, 2017], Pan et al.
Revised: [2016, 2017], Pan et al.

Original: In these works, the heat transfer equation with a pre-determined ﬂow ﬁeld velocity was formulated as an MDP, and RL algorithms used to control the state of the ventilator.
Revised: In these works, the heat transfer equation with a pre-determined ﬂow ﬁeld velocity was formulated as an MDP, and RL algorithms used to control the state of the ventilator.

Original: Unlike PDE control, there have been few studies on controlling SPDEs.
Revised: Unlike PDE control, there have been few studies on controlling SPDEs.

Original: Due to the nonlinearity and randomness of the system, such a problem is extremely challenging for traditional control methods to address (Rosseel and Wells [2012], Øksendal [2005]).
Revised: Due to the nonlinearity and randomness of the system, such a problem is extremely challenging for traditional control methods to address (Rosseel and Wells [2012], Øksendal [2005]).

Original: In this work, we present an RL framework for online control of stochastic dynamical systems.
Revised: In this work, we present an RL framework for online control of stochastic dynamical systems.

Original: Inspired by the promising performance of recent
Revised: Inspired by the promising performance of recent

Original: RL algorithms in PDE control, we formulate the problem of controlling a SPDE as an MDP with an inﬁnite dimensional state-action space, and use Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al.
Revised: RL algorithms in PDE control, we formulate the problem of controlling a SPDE as an MDP with an inﬁnite dimensional state-action space, and use Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al.

Original: [2016]) to train our agent.
Revised: [2016]) to train our agent.

Original: We evaluate our framework to control the stochastic
Revised: We evaluate our framework to control the stochastic

Original: Burgers’ equation, by damping the shock wave induced by inherent perturbations.
Revised: Burgers’ equation, by damping the shock wave induced by inherent perturbations.

Original:  Methodology
Revised:  Methodology

Original: We consider a (controlled) stochastic dynamical system governed by
Revised: We consider a (controlled) stochastic dynamical system governed by

Original: [Equation]
Revised: [Equation]

Original: where u ( x , t ) denotes the state variable of the dynamical system which depends on spatial free variables x
Revised: where u ( x , t ) denotes the state variable of the dynamical system which depends on spatial free variables x

Original: = ( x 1 , · · · , x n ) , within an n -dimensional domain Ω , x 1: n ∈ Ω ⊂ R n , and time t ∈ R + , and its derivatives with respect to the free variables.
Revised: = ( x 1 , · · · , x n ) , within an n -dimensional domain Ω , x 1: n ∈ Ω ⊂ R n , and time t ∈ R + , and its derivatives with respect to the free variables.

Original: F ( . ) is a nonlinear general function of the derivatives parameterized by p .
Revised: F ( . ) is a nonlinear general function of the derivatives parameterized by p .

Original: In the above equation ξ ( x , t ) is a multidimensional noise that is white in time and correlated in space, and f ( x , t ) is a control input forcing function, which will be further discussed later.
Revised: In the above equation ξ ( x , t ) is a multidimensional noise that is white in time and correlated in space, and f ( x , t ) is a control input forcing function, which will be further discussed later.

Original: The above SPDE is solved, either numerically or analytically, on domain Ω bounded to the boundary ∂ Ω = Γ , with boundary condition u Γ .
Revised: The above SPDE is solved, either numerically or analytically, on domain Ω bounded to the boundary ∂ Ω = Γ , with boundary condition u Γ .

Original: An example of the control problem here can be deﬁned as setting the state variable to a desirable value u = u ∗ , within an arbitrary controlled sub-domain of Ω c bounded to the boundary ∂ Ω c
Revised: An example of the control problem here can be deﬁned as setting the state variable to a desirable value u = u ∗ , within an arbitrary controlled sub-domain of Ω c bounded to the boundary ∂ Ω c

Original: = Γ c , at all time t .
Revised: = Γ c , at all time t .

Original: A realistic example of this sort of control problem is an HVAC unit system where we attempt to set the temperature to a comfortable value, in a speciﬁc part of a building, in the presence of an air ﬂow.
Revised: A realistic example of this sort of control problem is an HVAC unit system where we attempt to set the temperature to a comfortable value, in a speciﬁc part of a building, in the presence of an air ﬂow.

Original: The desired control problem can be formally deﬁned as the following optimal control problem:
Revised: The desired control problem can be formally deﬁned as the following optimal control problem:

Original: [Equation]
Revised: [Equation]

Original: To solve this (optimal) control problem, we propose using an RL approach.
Revised: To solve this (optimal) control problem, we propose using an RL approach.

Original: To this end, the continuous space-time SPDE is discretized in both space and time by a numerical solver, and the discretized SPDE is formulated as a discrete space-time, continuous state-action space, discounted MDP ( S , A , P , R , γ ) (Sutton and Barto [2018], Szepesvári [2010]), where state space S and action space
Revised: To this end, the continuous space-time SPDE is discretized in both space and time by a numerical solver, and the discretized SPDE is formulated as a discrete space-time, continuous state-action space, discounted MDP ( S , A , P , R , γ ) (Sutton and Barto [2018], Szepesvári [2010]), where state space S and action space

Original: A are vector functions depending on the function space of SPDE F ( Z ) , deﬁned over the domain Z of SPDE.
Revised: A are vector functions depending on the function space of SPDE F ( Z ) , deﬁned over the domain Z of SPDE.

Original: The current value of the target variable, within Ω c at each discrete time t , is the current state s t =
Revised: The current value of the target variable, within Ω c at each discrete time t , is the current state s t =

Original: u
Revised: u

Original: ( x , t ) .
Revised: ( x , t ) .

Original: Assuming that the system is equipped with a number of actuators, a localized control input function is the continuous action a t = f
Revised: Assuming that the system is equipped with a number of actuators, a localized control input function is the continuous action a t = f

Original: ( x , t ) , within the controlled domain Ω c .
Revised: ( x , t ) , within the controlled domain Ω c .

Original: A scalar reward function is deﬁned to penalize the RL agent when the target variable deviates from the desired value, given by:
Revised: A scalar reward function is deﬁned to penalize the RL agent when the target variable deviates from the desired value, given by:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: (3) which is negative of the cost function, regularized by the cost of choosing an action (which is small or zero for cheap controls and large for expensive controls).
Revised: which is negative of the cost function, regularized by the cost of choosing an action (which is small or zero for cheap controls and large for expensive controls)

Original: A new state given the chosen action and the previous state is sampled as s t +1 ∼ P ( · | s t , a t ) , where the transition probability kernel P depends on the SPDE.
Revised: A new state given the chosen action and the previous state is sampled as s t +1 ∼ P ( · | s t , a t ) , where the transition probability kernel P depends on the SPDE.

Original:  Experiments
Revised:  Experiments

Original: The potential beneﬁts of SPDE control is known to be signiﬁcant in controlling turbulent ﬂows that occur in various engineering applications (Naseri and Malek [2014]).
Revised: The potential beneﬁts of SPDE control is known to be signiﬁcant in controlling turbulent ﬂows that occur in various engineering applications (Naseri and Malek [2014]).

Original: Attempts to control turbulent ﬂows in engineering applications have focused on the manipulation of coherent structures, such as a shock wave.
Revised: Attempts to control turbulent ﬂows in engineering applications have focused on the manipulation of coherent structures, such as a shock wave.

Original: A shock wave is a region within the ﬂuid ﬂow where physical conditions undergo an abrupt change because of high ﬂow variable gradients, and this damages structures as it propagates.
Revised: A shock wave is a region within the ﬂuid ﬂow where physical conditions undergo an abrupt change because of high ﬂow variable gradients, and this damages structures as it propagates.

Original: 1D Stochastic Burgers’ equation (SBE) is a simple model that best describes this phenomenon:
Revised: 1D Stochastic Burgers’ equation (SBE) is a simple model that best describes this phenomenon:

Original: [Equation]
Revised: [Equation]

Original: where η t ( x ) is a space-time white noise with the intensity of (cid:15) , and it is realized as the generalized derivative of the Brownian sheet, i.e. η t ( x ) =
Revised: where η t ( x ) is a space-time white noise with the intensity of (cid:15) , and it is realized as the generalized derivative of the Brownian sheet, i.e. η t ( x ) =

Original: ∂
Revised: ∂

Original: t ∂ x W t ( x ) , where W t ( x ) is a Gaussian process.
Revised: t ∂ x W t ( x ) , where W t ( x ) is a Gaussian process.

Original: Due to the presence of nonlinear convective terms, solution of the 1D Burgers’ equation is prone to exhibit a chaotic behaviour.
Revised: Due to the presence of nonlinear convective terms, solution of the 1D Burgers’ equation is prone to exhibit a chaotic behaviour.

Original: Here, we test our proposed RL framework to control the ﬂuid ﬂow governed by the SBE to damp the developed shock wave and stabilize the system in an online manner, and compare it against previous non-RL approaches (Choi et al.
Revised: Here, we test our proposed RL framework to control the ﬂuid ﬂow governed by the SBE to damp the developed shock wave and stabilize the system in an online manner, and compare it against previous non-RL approaches (Choi et al.

Original: [1993], Munteanu [2019]).
Revised: [1993], Munteanu [2019]).

Original: A distributed control method is used by introducing an external forcing term, f ( x, t ) , as a control input of the system.
Revised: A distributed control method is used by introducing an external forcing term, f ( x, t ) , as a control input of the system.

Original: The control problem here is formalized to ﬁnd the proper f ( x, t ) applied to the ﬂuid ﬂow in the solution domain to squash the shock wave at each time.
Revised: The control problem here is formalized to ﬁnd the proper f ( x, t ) applied to the ﬂuid ﬂow in the solution domain to squash the shock wave at each time.

Original: Squashing in our terminology means inf f E [ (cid:82) Ω c ( u ( x, t )
Revised: Squashing in our terminology means inf f E [ (cid:82) Ω c ( u ( x, t )

Original: − ¯ u ) 2 d x ] in each time step, where ¯ u = E [ 12 π (
Revised: − ¯ u ) 2 d x ] in each time step, where ¯ u = E [ 12 π (

Original: cid:82) Ω c u ( x, t )d x ] .
Revised: cid:82) Ω c u ( x, t )d x ] .

Original: The described control problem above can be expressed as the following inverse problem:
Revised: The described control problem above can be expressed as the following inverse problem:

Original: [Equation]
Revised: [Equation]

Original: Periodic Boundary condition
Revised: Periodic Boundary condition

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: 3.1 Training Procedure
Revised: 3.1 Training Procedure

Original: SPDE solver: A uniform computational grid of n x = 151 points is used for spatial discretization.
Revised: SPDE solver: A uniform computational grid of n x = 151 points is used for spatial discretization.

Original: A Crank-Nicolson method (Crank et al.
Revised: A Crank-Nicolson method (Crank et al.

Original: [1947]) in time and second-order centred differences in space are used to discretize the equation, an explicit iterative method is used to solve the discretized nonlinear equation.
Revised: [1947]) in time and second-order centred differences in space are used to discretize the equation, an explicit iterative method is used to solve the discretized nonlinear equation.

Original: Instantaneous velocity ﬁeld for evolution of a shock wave is shown in Figure 2.
Revised: Instantaneous velocity ﬁeld for evolution of a shock wave is shown in Figure 2.

Original: It is observed that the shock wave energy dissipates over time by the structural diffusivity of system.
Revised: It is observed that the shock wave energy dissipates over time by the structural diffusivity of system.

Original: RL:
Revised: RL:

Original: Given the continuous state-action nature of the SPDE, we use a deep deterministic policy gradient (DDPG) approach to train the RL agent.
Revised: Given the continuous state-action nature of the SPDE, we use a deep deterministic policy gradient (DDPG) approach to train the RL agent.

Original: We implemented an actor/critic DDPG framework according to Lillicrap et al.
Revised: We implemented an actor/critic DDPG framework according to Lillicrap et al.

Original: The DDPG states and actions are u ( . ; t ) and f ( . ; t ) , respectively.
Revised: The DDPG states and actions are u ( . ; t ) and f ( . ; t ) , respectively.

Original: To facilitate this algorithm implementation in the real world, the action space is limited to a set of 4-interval piecewise constant functions.
Revised: To facilitate this algorithm implementation in the real world, the action space is limited to a set of 4-interval piecewise constant functions.

Original: Similar fully connected 2-layer networks are used as the actor and critic networks architecture, that consist of hidden layers of ReLU units, and a linear output layer.
Revised: Similar fully connected 2-layer networks are used as the actor and critic networks architecture, that consist of hidden layers of ReLU units, and a linear output layer.

Original: Network parameters are initialized using Xavier initialization (Glorot and Bengio [2010]), and Adam optimizer(Kingma and Ba [2015]) is used for optimization.
Revised: Network parameters are initialized using Xavier initialization (Glorot and Bengio [2010]), and Adam optimizer(Kingma and Ba [2015]) is used for optimization.

Original: An experience replay with the buffer size of 1000000 is used.
Revised: An experience replay with the buffer size of 1000000 is used.

Original: Details on the hyperparameters used for training is provided in
Revised: Details on the hyperparameters used for training is provided in

Original: Appendix A.
Revised: Appendix A.

Original: 3.2 Results
Revised: 3.2 Results

Original: Here we show some preliminary results on controlling the SBE, and compare them against benchmarks.
Revised: Here we show some preliminary results on controlling the SBE, and compare them against benchmarks.

Original: Figure 3(a) and Figure 3(b) show the free evolution of the shock wave (black line), evolution under online control of the RL agent (red dotted-line), and the input control function at t = 0 . 8 and averaged over N = 20 runs with 90% conﬁdence interval, respectively.
Revised: Figure 3(a) and Figure 3(b) show the free evolution of the shock wave (black line), evolution under online control of the RL agent (red dotted-line), and the input control function at t = 0 . 8 and averaged over N = 20 runs with 90% conﬁdence interval, respectively.

Original: It can be seen that the control input function damps the shock wave and reduces the gradients throughout the domain.
Revised: It can be seen that the control input function damps the shock wave and reduces the gradients throughout the domain.

Original: We compare the average return for N eps = 100 episodes obtained by the RL against uncontrolled system, and the suboptimaloptimal control approach used by Choi et al.
Revised: We compare the average return for N eps = 100 episodes obtained by the RL against uncontrolled system, and the suboptimal control approach used by Choi et al.

Original: [1993], given the same form of piecewise constant functions as the action space in Figure 3(c).
Revised: [1993], given the same form of piecewise constant functions as the action space in Figure 3(c).

Original: Finally, the average return for three gradually more expressive action functions are compared in Figure 3(d).
Revised: Finally, the average return for three gradually more expressive action functions are compared in Figure 3(d).

Original:  Conclusion
Revised:  Conclusion

Original: In this work we presented a learning-based approach, based on reinforcement learning, to control behaviour of a SPDE that describes a noisy dynamical system.
Revised: In this work we presented a learning-based approach, based on reinforcement learning, to control behaviour of a SPDE that describes a noisy dynamical system.

Original: Preliminary results for online control of the 1D SBE showed promising results in damping the shock wave and reducing sharp gradients.
Revised: Preliminary results for online control of the 1D SBE showed promising results in damping the shock wave and reducing sharp gradients.

Original: Although boundary control approaches are more practical in ﬂuid mechanics we adopted a distributed control approach for the case of 1D SBE because it is deﬁned in an inﬁnitely large domain, represented by periodic boundary conditions on a 2 π length cell.
Revised: Although boundary control approaches are more practical in ﬂuid mechanics we adopted a distributed control approach for the case of 1D SBE because it is deﬁned in an inﬁnitely large domain, represented by periodic boundary conditions on a 2 π length cell.

Original: Our proposed RL control approach is easily extendable to a boundary control strategy to control bounded systems.
Revised: Our proposed RL control approach is easily extendable to a boundary control strategy to control bounded systems.

Original: We will extend this work to more realistic higher dimensional SPDEs in our future work.
Revised: We will extend this work to more realistic higher dimensional SPDEs in our future work.

Original:  Acknowledgment
Revised:  Acknowledgment

Original: AMF acknowledges the funding from the Canada CIFAR AI Chairs program.
Revised: AMF acknowledges the funding from the Canada CIFAR AI Chairs program.

Original: A Hyperparameters
Revised: A Hyperparameters

Original: [Table]
Revised: [Table]