Original: L ONG - TERM P LANNING , S HORT - TERM A DJUSTMENTS
Revised: L ONG - TERM P LANNING , S HORT - TERM A DJUSTMENTS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep reinforcement learning (RL) algorithms can learn complex policies to optimize agent operation over time.
Revised: Deep reinforcement learning (RL) algorithms can learn complex policies to optimize agent operation over time.

Original: RL algorithms have shown promising results in solving complicated problems in recent years.
Revised: RL algorithms have shown promising results in solving complicated problems in recent years.

Original: However, their application on real-world physical systems remains limited.
Revised: However, their application on real-world physical systems remains limited.

Original: Despite the advancements in RL algorithms, the industries often prefer traditional control strategies.
Revised: Despite the advancements in RL algorithms, the industries often prefer traditional control strategies.

Original: Traditional methods are simple, computationally efﬁcient and easy to adjust.
Revised: Traditional methods are simple, computationally efﬁcient and easy to adjust.

Original: In this paper, we propose a new Q-learning algorithm for continuous action space, which can bridge the control and RL algorithms and bring us the best of both worlds.
Revised: In this paper, we propose a new Q-learning algorithm for continuous action space, which can bridge the control and RL algorithms and bring us the best of both worlds.

Original: Our method can learn complex policies to achieve long-term goals and at the same time it can be easily adjusted to address short-term requirements without retraining.
Revised: Our method can learn complex policies to achieve long-term goals and at the same time it can be easily adjusted to address short-term requirements without retraining.

Original: We achieve this by modeling both short-term and long-term prediction models.
Revised: We achieve this by modeling both short-term and long-term prediction models.

Original: The short-term prediction model represents the estimation of the system dynamic while the long-term prediction model represents the Q-value.
Revised: The short-term prediction model represents the estimation of the system dynamic while the long-term prediction model represents the Q-value.

Original: The case studies demonstrate that our proposed method can achieve short-term and long-term goals without complex reward functions.
Revised: The case studies demonstrate that our proposed method can achieve short-term and long-term goals without complex reward functions.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Optimal control methodologies use system dynamic equations to design actions that minimize desired cost functions.
Revised: Optimal control methodologies use system dynamic equations to design actions that minimize desired cost functions.

Original: A cost function can be designed to track a trajectory, reach a goal, or avoid obstacles.
Revised: A cost function can be designed to track a trajectory, reach a goal, or avoid obstacles.

Original: It is also possible to design a cost function to achieve a combination of goals.
Revised: It is also possible to design a cost function to achieve a combination of goals.

Original: Model
Revised: Model

Original: Predictive Control (MPC) is a common optimal control technique and has been applied to many industrial applications such as pressure control and temperature control in chemical processes (7).
Revised: Predictive Control (MPC) is a common optimal control technique and has been applied to many industrial applications such as pressure control and temperature control in chemical processes (Garcia et al., 1989).

Original: The traditional control solutions are not adequate to address the challenges raised with the evolution of industrial systems.
Revised: The traditional control solutions are not adequate to address the challenges raised with the evolution of industrial systems.

Original: Recently, deep reinforcement learning (RL) has shown promising results in solving complex problems.
Revised: Recently, deep reinforcement learning (RL) has shown promising results in solving complex problems.

Original: For example, it has generated superhuman performance in chess and shogi (15).
Revised: For example, it has generated superhuman performance in chess and shogi (Silver et al., 2017).

Original: The following advantages make deep RL a strong candidate to overcome traditional control limitations.
Revised: The following advantages make deep RL a strong candidate to overcome traditional control limitations.

Original: First, deep RL has an advantage in solving complex problems, especially when the consequences of an action are not immediately obvious.
Revised: First, deep RL has an advantage in solving complex problems, especially when the consequences of an action are not immediately obvious.

Original: Moreover, it can learn an optimal solution without requiring detailed knowledge of the systems or their engineering designs.
Revised: Moreover, it can learn an optimal solution without requiring detailed knowledge of the systems or their engineering designs.

Original: Finally, deep RL is not limited to time-series sensors and can use new sensors such as vision for a better control.
Revised: Finally, deep RL is not limited to time-series sensors and can use new sensors such as vision for a better control.

Original: [Figure]
Revised: [Figure]

Original: However, deep RL has not been applied to address industrial problems in a meaningful way.
Revised: However, deep RL has not been applied to address industrial problems in a meaningful way.

Original: There are several key issues that limit the application of deep RL to real-world problems.
Revised: There are several key issues that limit the application of deep RL to real-world problems.

Original: Deep RL algorithms typically require many samples during training (sample complexity).
Revised: Deep RL al- gorithms typically require many samples during training (sample complexity).

Original: Sample complexity leads to high computational costs.
Revised: Sample complexity leads to high computational costs.

Original: A high computational cost can be justiﬁed for industries as a onetime charge.
Revised: A high computational cost can be justiﬁed for industries as a onetime charge.

Original: However, oftentimes small changes in the system goal, such as changing the desired temperature in a chemical reactor, or a new constraint such as a maximum allowable temperature in the reactor, require retraining the model.
Revised: However, oftentimes small changes in the system goal, such as changing the desired temperature in a chemical reactor, or a new constraint such as a maximum allowable temperature in the reactor, require retraining the model.

Original: Moreover, industrial systems often have several short-term and long-term objectives.
Revised: Moreover, industrial systems often have several short-term and long-term objectives.

Original: For example, consider the crane system shown in Figure 1.
Revised: For example, consider the crane system shown in Figure 1.

Original: The long-term goal is to convey the payload to the target location as soon as possible.
Revised: The long-term goal is to convey the payload to the target location as soon as possible.

Original: However, when the payload gets close to the target, it must have minimum sway for the safety of the operators.
Revised: However, when the payload gets close to the target, it must have minimum sway for the safety of the operators.

Original: Designing a reward function that can capture these short-term and long-term goals concurrently can be challenging or even infeasible.
Revised: Designing a reward function that can capture these short-term and long-term goals concurrently can be challenging or even infeasible.

Original: A class of short-term objectives related to safe exploration during RL training have been studied recently.
Revised: A class of short-term objectives related to safe exploration during RL training have been studied recently.

Original: Gu et al
Revised: Gu et al.

Original: (8) presented an application of deep RL for robotic manipulation control.
Revised: (2017) presented an application of deep RL for robotic manipulation control.

Original: To ensure safe exploration, they set maximum limits for the joint positions, and joint velocities.
Revised: To ensure safe exploration, they set maximum limits for the joint positions, and joint velocities.

Original: Moreover, they set a sphere boundary for the end-effector position and when the boundaries were about to be violated, they used correction velocity to force the end-effector position back to the center of the sphere.
Revised: Moreover, they set a sphere boundary for the end-effector position and when the boundaries were about to be violated, they used correction velocity to force the end-effector position back to the center of the sphere.

Original: Dalal et al (5) formulated the safe exploration as an optimization problem.
Revised: Dalal et al. (2018) formulated the safe exploration as an optimization problem.

Original: They proposed to add a safety layer that modiﬁes the action at each time step.
Revised: They proposed to add a safety layer that modiﬁes the action at each time step.

Original: Toward this end, they learn the constraint function using a linear model and use this model to ﬁnd the minimal change to the action such that the safety constraints are met at each time step.
Revised: Toward this end, they learn the constraint function using a linear model and use this model to ﬁnd the minimal change to the action such that the safety constraints are met at each time step.

Original: To the best of our knowledge, there is no study addressing short-term objectives during application.
Revised: To the best of our knowledge, there is no study addressing short-term objectives during application.

Original: In this paper, we present a Locally Linear Q-Learning (LLQL) algorithm for continuous action space.
Revised: In this paper, we present a Locally Linear Q-Learning (LLQL) algorithm for continuous action space.

Original: The LLQL includes a short-term prediction model, a long-term prediction model, and a controller.
Revised: The LLQL includes a short-term prediction model, a long-term prediction model, and a controller.

Revised: The short-term prediction model represents a locally linear model of the dynamic system, while the long-term prediction model represents the value function + a locally linear advantage function.

Original: The controller uses the short-term prediction model and the long-term prediction model to generate actions that achieve short-term and long-term goals simultaneously.
Revised: The controller uses the short-term prediction model and the long-term prediction model to generate actions that achieve short-term and long-term goals simultaneously.

Original: It adopts a policy that maximizes Q-value while achieving short-term goals.
Revised: It adopts a policy that maximizes Q-value while achieving short-term goals.

Original: The LLQL algorithm has the following advantages:
Revised: The LLQL algorithm has the following advantages:

Original: • It does not require designing sensitive reward functions for achieving short-term and longterm goals concurrently.
Revised: • It does not require designing sensitive reward functions for achieving short-term and longterm goals concurrently.

Original: • It shows better performance in achieving short-term and long-term goals compared to the traditional reward modiﬁcation methods.
Revised: • It shows better performance in achieving short-term and long-term goals compared to the traditional reward modiﬁcation methods.

Original: • It is possible to modify the short-term goals without time-consuming retraining.
Revised: • It is possible to modify the short-term goals without time-consuming retraining.

Original: The remainder of this paper is organized as follows.
Revised: The remainder of this paper is organized as follows.

Original: Section 2 represents the background in dynamic systems and RL.
Revised: Section 2 represents the background in dynamic systems and RL.

Original: Section 3 represents the LLQL algorithm.
Revised: Section 3 represents the LLQL algorithm.

Original: Section 4 presents our methodology to achieve short-term and long-term goals using LLQL.
Revised: Section 4 presents our methodology to achieve short-term and long-term goals using LLQL.

Original: Section 5 presents our experimental results.
Revised: Section 5 presents our experimental results.

Original: Section 6discusses the related work. Section 7 presents the conclusions of the paper. 
Revised: Section 6 presents the conclusions of the paper. Section A discusses the related work.

Original: Sectionpresents additional experiments for those interested.
Revised: Section B presents additional experiments for those interested.

Original:  B ACKGROUND AND D EFINITIONS
Revised:  B ACKGROUND AND D EFINITIONS

Original: In this section, we review the backgrounds in dynamic systems and reinforcement learning.
Revised: In this section, we review the backgrounds in dynamic systems and reinforcement learning.

Original: 2. D YNAMIC S YSTEMS
Revised: 2. D YNAMIC S YSTEMS

Original: A continuous-time dynamic system can be represented as:
Revised: A continuous-time dynamic system can be represented as:

Original: [Equation]
Revised: [Equation]

Original: where given the system parameters, p , f maps the state variables, x 2 X , and actions, u 2 U , to the state derivative, dxdt at time t .
Revised: where given the system parameters, p , f maps the state variables, x 2 X , and actions, u 2 U , to the state derivative, dxdt at time t .

Original: In state space control, the goal is to design a control policy, ⇡ control ( u ( t )
Revised: In state space control, the goal is to design a control policy, ⇡ control ( u ( t )

Original: | x ( t )) , that generates proper actions so as the state variables follow the given desired trajectory, x d ( t ) .
Revised: | x ( t )) , that generates proper actions so as the state variables follow the given desired trajectory, x d ( t ) .

Original: It is challenging to design a control policy for a nonlinear complex system represented in equation (1).
Revised: It is challenging to design a control policy for a nonlinear complex system represented in equation (1).

Original: The control problem becomes much easier to address when this system is linear with respect to the input (4).
Revised: The control problem becomes much easier to address when this system is linear with respect to the input (Chen et al., 2003).

Original: We can present these systems as:
Revised: We can present these systems as:

Original: [Equation]
Revised: [Equation]

Original: Since measurements are typically sampled in discrete times, we derive a discrete time version of linear system (2).
Revised: Since measurements are typically sampled in discrete times, we derive a discrete time version of linear system (2).

Original: Using a ﬁrst-order approximation:
Revised: Using a ﬁrst-order approximation:

Original: [Equation]
Revised: [Equation]

Original: where t k represents time at sample point k .
Revised: where t k represents time at sample point k .

Original: In this paper, we assume the sampling rate is constant;  = t k +1  t k .
Revised: In this paper, we assume the sampling rate is constant;  = t k +1  t k .

Original: Using (2) and (3), we have:
Revised: Using (2) and (3), we have:

Original: [Equation]
Revised: [Equation]

Original: For brevity, we present t k by k , f
Revised: For brevity, we present t k by k , f

Original: ( x ( t k )) by f ( x k ) , and g ( x ( t k )) by g ( x k ) in the remainder of the paper.
Revised: ( x ( t k )) by f ( x k ) , and g ( x ( t k )) by g ( x k ) in the remainder of the paper.

Original: Therefore, we can represent a dynamic system as:
Revised: Therefore, we can represent a dynamic system as:

Original: [Equation]
Revised: [Equation]

Original: 2. R EINFORCEMENT L EARNING
Revised: 2. R EINFORCEMENT L EARNING

Original: The goal of RL is to learn a policy, ⇡ RL ( u k | x k ) , that generates a set of actions, u 2 U , that maximize the expected sum of rewards in the environment, E n .
Revised: The goal of RL is to learn a policy, ⇡ RL ( u k | x k ) , that generates a set of actions, u 2 U , that maximize the expected sum of rewards in the environment, E n .

Original: Consider:
Revised: Consider:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where  < 1 is the discount factor, r is the reward function and T represents the end time and can be set to T = 1 .
Revised: where  < 1 is the discount factor, r is the reward function and T represents the end time and can be set to T = 1 .

Original: The goal is to learn ⇡ RL for environment, E n , such that:
Revised: The goal is to learn ⇡ RL for environment, E n , such that:

Original: [Equation]
Revised: [Equation]

Original:  Unlike control algorithms, model-free reinforcement learning algorithms assume the system dynamic is unknown.
Revised: [Equation] Unlike control algorithms, model-free reinforcement learning algorithms assume the system dynamic is unknown.

Original: Q-function, Q ⇡ ( x k , u k ) is deﬁned as the expected return at state x k when we take action u k and adopt policy ⇡ afterward:
Revised: Q-function, Q ⇡ ( x k , u k ) is deﬁned as the expected return at state x k when we take action u k and adopt policy ⇡ afterward:

Original: [Equation]
Revised: [Equation]

Original: Q-learning algorithms (16) are among the most common model-free RL methods for discrete action space problems.
Revised: Q-learning algorithms (Watkins & Dayan, 1992) are among the most common model-free RL methods for discrete action space problems.

Original: These algorithms use the Bellman recursive equation to model Q-function:
Revised: These algorithms use the Bellman recursive equation to model Q-function:

Original: [Equation]
Revised: [Equation]

Original: where µ represents a greedy deterministic policy that selects the action which maximizes Q-value at each step:
Revised: where µ represents a greedy deterministic policy that selects the action which maximizes Q-value at each step:

Original: [Equation]
Revised: [Equation]

Original: Q-learning algorithms learn the parameters of the function approximator, ✓ Q , by minimizing the
Revised: Q-learning algorithms learn the parameters of the function approximator, ✓ Q , by minimizing the

Original: Bellman error:
Revised: Bellman error:

Original: [Equation]
Revised: [Equation]

Original: where y k is the ﬁxed target Q-function, and  represents the exploration policy.
Revised: where y k is the ﬁxed target Q-function, and  represents the exploration policy.

Original: For continuous action domain problems, it is not trivial to solve equation (10) at each time step.
Revised: For continuous action domain problems, it is not trivial to solve equation (10) at each time step.

Original: Finding an action to maximize Q which can be a complex nonlinear function is computationally expensive or even infeasible.
Revised: Finding an action to maximize Q which can be a complex nonlinear function is computationally expensive or even infeasible.

Original: To address this problem, Lillicrap et al (12) proposed the Deep Deterministic Policy Gradient (DDPG) algorithm, which learns two networks simultaneously.
Revised: To address this problem, Lillicrap et al. (2015) proposed the Deep Deterministic Policy Gradient (DDPG) algorithm, which learns two networks simultaneously.

Original: The critic network learns Q-function by minimizing the Bellman error, and the actor network learns parameters of the policy to maximize the estimated value of Q-function.
Revised: The critic network learns Q-function by minimizing the Bellman error, and the actor network learns parameters of the policy to maximize the estimated value of Q-function.

Original: Gu et al
Revised: Gu et al.

Original: (9) proposedNormalized
Revised: (2016) proposed

Original:  Advantage Function (NAF) Q-learning which formulates the Q-function as the sum of the value function, V ( x ) , and the advantage function, A ( x, u ) .
Revised: Normalized Advantage Function (NAF) Q-learning which formulates the Q-function as the sum of the value function, V ( x ) , and the advantage function, A ( x, u ) .

Original: [Equation]
Revised: [Equation]

Original: 12) where
Revised: 12) where

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: P ( x | ✓ P ) =
Revised: P ( x | ✓ P ) =

Original: L [Equation] x | ✓ P ) L ( x | ✓ P ) T , where L ( x | ✓ P ) is a lower-triangular matrix.
Revised: L ( x | ✓ P ) L ( x | ✓ P ) T , where L ( x | ✓ P ) is a lower-triangular matrix.

Original: The value function is not a function of action, u .
Revised: The value function is not a function of action, u .

Original: Therefore, the action which maximizes advantage function, A , maximizes the Q function.
Revised: Therefore, the action which maximizes advantage function, A , maximizes the Q function.

Original: P ( x | ✓ P ) is a positive-deﬁnite matrix, and therefore, the action that maximizes the advantage function and the Q -function is given by µ ( x | ✓ u ) .
Revised: P ( x | ✓ P ) is a positive-deﬁnite matrix, and therefore, the action that maximizes the advantage function and the Q -function is given by µ ( x | ✓ u ) .

Original:  L OCALLY L INEAR Q-L EARNING
Revised:  L OCALLY L INEAR Q-L EARNING

Original: [Figure]
Revised: [Figure]

Original: In this section, we propose the LLQL algorithm, which like (12) and (9) can handle continuous action space.
Revised: In this section, we propose the LLQL algorithm, which like (Lillicrap et al., 2015) and (Gu et al., 2016) can handle continuous action space.

Original: Our approach learns short-term and long-term prediction models.
Revised: Our approach learns short-term and long-term prediction models.

Original: Using the long-term and short-term models, a controller generates actions that guide the system toward its short-term and long-term goals.
Revised: Using the long-term and short-term models, a controller generates actions that guide the system toward its short-term and long-term goals.

Original: Figure 2 shows our proposed structure to learn the parameters of the short-term and long-term prediction models.
Revised: Figure 2 shows our proposed structure to learn the parameters of the short-term and long-term prediction models.

Original: Short-term prediction: consider the nonlinear system presented in equation (5).
Revised: Short-term prediction: consider the nonlinear system presented in equation (5).

Original: In this work, we use deep neural networks to estimate system functions, f ( x k ) , and g ( x k ) at each operating point.
Revised: In this work, we use deep neural networks to estimate system functions, f ( x k ) , and g ( x k ) at each operating point.

Original: Substituting the network estimations for these functions in equation (5), we can predict the next state as:
Revised: Substituting the network estimations for these functions in equation (5), we can predict the next state as:

Original: [Equation]
Revised: [Equation]

Original: (14) where ˆ x k +1 represents our estimation of the next step, and ✓ f and ✓ g are the network parameters.
Revised: ) where ˆ x k +1 represents our estimation of the next step, and ✓ f and ✓ g are the network parameters.

Original:  is a constant hyper parameter.
Revised:  is a constant hyper parameter.

Original: In dynamic systems, the difference between two consecutive states, x k +1  x k , is typically very small.
Revised: In dynamic systems, the difference between two consecutive states, x k +1  x k , is typically very small.

Original: Considering a small  leads to reasonable f and g values and, therefore, improves learning time and accuracy.
Revised: Considering a small  leads to reasonable f and g values and, therefore, improves learning time and accuracy.

Original: We call this dynamic system model short-term prediction model .
Revised: We call this dynamic system model short-term prediction model .

Original: The controller uses this model to generate actions, which lead the system toward its short-term goals.
Revised: The controller uses this model to generate actions, which lead the system toward its short-term goals.

Original: Note that previous work have used the system short-term dynamic model for generating additional samples in imagination rollout (for example, see (9), and (14)).
Revised: Note that previous work have used the system short-term dynamic model for generating additional samples in imagination rollout (for example, see (Gu et al., 2016), and (Racani`ere et al., 2017)).

Original: In this paper, we show that this model can also be used to design actions to achieve short-term goals.
Revised: In this paper, we show that this model can also be used to design actions to achieve short-term goals.

Original: To learn the parameters of our short-term prediction model, ✓ f and ✓ g , we minimize the short-term loss function, L 1 , as it is presented in Algorithm 1.
Revised: To learn the parameters of our short-term prediction model, ✓ f and ✓ g , we minimize the short-term loss function, L 1 , as it is presented in Algorithm 1.

Original: Long-term prediction: Q-function represents the maximum cumulative reward that can be achieved from current state, x k , taking an action u k .
Revised: Long-term prediction: Q-function represents the maximum cumulative reward that can be achieved from current state, x k , taking an action u k .

Original: Therefore, by learning Q-function, we learn the longterm prediction model for the system.
Revised: Therefore, by learning Q-function, we learn the longterm prediction model for the system.

Original: Like NAF (9) (see equation (12)), we present Q-function as a sum of value function and advantage function.
Revised: Like NAF (Gu et al., 2016) (see equation (12)), we present Q-function as a sum of value function and advantage function.

Original: However, we present the advantage function, A ( x, u | ✓ A ) using a locally linear function of x k and u k as:
Revised: However, we present the advantage function, A ( x, u | ✓ A ) using a locally linear function of x k and u k as:

Original: [Equation]
Revised: [Equation]

Revised: where h ( x k | ✓ h ) and d ( x k | ✓ d ) networks model the locally linear advantage function.

Original: Note the NAF advantage function is a special case of the LLQL advantage function when d ( x k | ✓ d ) =
Revised: Note the NAF advantage function is a special case of the LLQL advantage function when d ( x k | ✓ d ) =

Original: I , where I represents the identity matrix.
Revised: I , where I represents the identity matrix.

Original: To maximize Q-function, we have to design u k which minimizes h ( x k | ✓ h )
Revised: To maximize Q-function, we have to design u k which minimizes h ( x k | ✓ h )

Original: + d ( x k | ✓ d ) u k .
Revised: + d ( x k | ✓ d ) u k .

Original: For simplicity, we present h ( x k | ✓ h ) , and d ( x k | ✓ d ) with h k and d k respectively in the remainder of the paper.
Revised: For simplicity, we present h ( x k | ✓ h ) , and d ( x k | ✓ d ) with h k and d k respectively in the remainder of the paper.

Original: To maximize Q-function and achieve the long-term goal, we can use simple pseudo-inverse matrix multiplication and derive a solution with the least squares error as:
Revised: To maximize Q-function and achieve the long-term goal, we can use simple pseudo-inverse matrix multiplication and derive a solution with the least squares error as:

Original: [Equation]
Revised: [Equation]

Original: When || d k || = 0 , it means the network predicts that our action has no impact on the advantage function.
Revised: When || d k || = 0 , it means the network predicts that our action has no impact on the advantage function.

Original: Therefore, we choose a random action.
Revised: Therefore, we choose a random action.

Original: Random exploration is an important part of any deep RL algorithm.
Revised: Random exploration is an important part of any deep RL algorithm.

Original: Therefore, in addition to this unlikely case, we add noise, N k , to the action, u k , during the training.
Revised: Therefore, in addition to this unlikely case, we add noise, N k , to the action, u k , during the training.

Original: We reduce the amount of noise injected to the action as the algorithm converges.[Table] [Table] [Table]
Revised: We reduce the amount of noise injected to the action as the algorithm converges.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: do 14:

Original: Randomly

Original: [Table]

Original: [Table]

Original: 1 N s

Original: P

Original: N s i

Original: =

Original: [-1 || x i +
-]
Revised: [-1 || x i +
-]

Original:  x i   ( f (

Original: [Table]

Original: i | ✓ f )

Original: +

Original: [Table]

Original: [Table]

Original: [Table]

Original: : I do 17:

Original: Randomly

Original: [Table]

Original: Set y

Original: [Table]

Original: r i +  Q 0 ( x i +1 | ✓ Q 0 ) .

Original: [Table]

Original: [Table]

Original: [Table]

Original: P

Original: [-N l i =
-]
Revised: [-N l i =
-]

Original: ||

Original: y i

Original:  Q ( x i , u i | ✓ Q )

Original: || .

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: Q

Original: [Table] [Table] [Equation] In the application, the controller solves u k with additional constraints to achieve the desired shortterm trajectories.
Revised: In the application, the controller solves u k with additional constraints to achieve the desired shortterm trajectories.

Original: We will discuss our short-term adjustment algorithms in the next section.
Revised: We will discuss our short-term adjustment algorithms in the next section.

Original: To learn
Revised: To learn

Original: Q-function, in addition to the state estimation error, we minimize the long-term loss function, L 2 , as it is presented in Algorithm 1.
Revised: Q-function, in addition to the state estimation error, we minimize the long-term loss function, L 2 , as it is presented in Algorithm 1.

Original: Note that having the short-term model, it is straightforward to add imagination rollout to our algorithm to increase sample efﬁciency.
Revised: Note that having the short-term model, it is straightforward to add imagination rollout to our algorithm to increase sample efﬁciency.

Original: However, improving sample efﬁciency in RL is not the focus of this work.
Revised: However, improving sample efﬁciency in RL is not the focus of this work.

Original:  C ONTROL S TRATEGY
Revised:  C ONTROL S TRATEGY

Original: By separating action design from prediction models, LLQL gives us the freedom to design different control strategies for achieving short-term and long-term goals.
Revised: By separating action design from prediction models, LLQL gives us the freedom to design different control strategies for achieving short-term and long-term goals.

Original: Moreover, the linear structure of short-term and long-term models simpliﬁes the control design.
Revised: Moreover, the linear structure of short-term and long-term models simpliﬁes the control design.

Original: Consider the case where LLQL has learned a perfect long-term model for an environment using Algorithm 1.
Revised: Consider the case where LLQL has learned a perfect long-term model for an environment using Algorithm 1.

Original: In this case, the optimum solution to achieve the long-term goal is given by equation (16).
Revised: In this case, the optimum solution to achieve the long-term goal is given by equation (16).

Original: When we have one or more shortterm goals as well, we can formulate the control design as an optimization problem to satisfy both short-term and long-term goals as much as possible.
Revised: When we have one or more shortterm goals as well, we can formulate the control design as an optimization problem to satisfy both short-term and long-term goals as much as possible.

Original: In this paper, we consider two types of short-term goals: 1) desired trajectory, and 2) constraint.
Revised: In this paper, we consider two types of short-term goals: 1) desired trajectory, and 2) constraint.

Original: In the ﬁrst scenario, the agent has a short-term desired trajectory.
Revised: In the ﬁrst scenario, the agent has a short-term desired trajectory.

Original: For example, a car may be required to travel with speciﬁc speed during certain periods.
Revised: For example, a car may be required to travel with speciﬁc speed during certain periods.

Original: In the second scenario, the agent has some limitation for a speciﬁc period of time.
Revised: In the second scenario, the agent has some limitation for a speciﬁc period of time.

Original: For example, a car is required to keep its speed below certain thresholds at some periods during the trip.
Revised: For example, a car is required to keep its speed below certain thresholds at some periods during the trip.

Original: To address the ﬁrst problem, we add an additional term to the cost function for the short-term goal and solve for the action.
Revised: To address the ﬁrst problem, we add an additional term to the cost function for the short-term goal and solve for the action.

Original: We deal with the second problem as a constraint optimization.
Revised: We deal with the second problem as a constraint optimization.

Original: 4. S HORT - TERM TRAJECTORY
Revised: 4. S HORT - TERM TRAJECTORY

Original: Let x d represent our desired short-term trajectory.
Revised: Let x d represent our desired short-term trajectory.

Original: We develop a control strategy to track x d while pursuing the long-term goals.
Revised: We develop a control strategy to track x d while pursuing the long-term goals.

Original: Using system dynamic functions f k and g k , we can deﬁne our control optimization problem as:
Revised: Using system dynamic functions f k and g k , we can deﬁne our control optimization problem as:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where x d ( k +1) represents the desired trajectory at time k + 1 .
Revised: where x d ( k +1) represents the desired trajectory at time k + 1 .

Original:  1 and  2 are positive coefﬁcients and can be adjusted to give higher weights to the short-term or long-term goals.
Revised:  1 and  2 are positive coefﬁcients and can be adjusted to give higher weights to the short-term or long-term goals.

Original: Note that in this work, we assume the short-term goals are temporary and when their time expires the system goes to the long-term optimum policy given by (16).
Revised: Note that in this work, we assume the short-term goals are temporary and when their time expires the system goes to the long-term optimum policy given by (16).

Original: For example, we may require a car to have a speciﬁc speed at some speciﬁc locations.
Revised: For example, we may require a car to have a speciﬁc speed at some speciﬁc locations.

Original: We can apply a similar pseudo-inverse matrix multiplication, and derive a solution with the least squares error for (17) as:[Equation]
Revised: We can apply a similar pseudo-inverse matrix multiplication, and derive a solution with the least squares error for (17) as:

Original: [Equation]
Revised: [Equation]

Original:  S HORT - TERM CONSTRAINT
Revised: 4. S HORT - TERM CONSTRAINT

Original: The LLQL algorithm provides a framework to design the actions considering different constraints.
Revised: The LLQL algorithm provides a framework to design the actions considering different constraints.

Original: For safe operation, the agent may have to avoid speciﬁc states for a period of time (for example, high speed or locations close to an obstacle).
Revised: For safe operation, the agent may have to avoid speciﬁc states for a period of time (for example, high speed or locations close to an obstacle).

Original: For simplicity, we assume at each moment we only have maximum one constraint on one state variable, x i .
Revised: For simplicity, we assume at each moment we only have maximum one constraint on one state variable, x i .

Original: This is a reasonable assumption, because in physical systems the agent is close to one of the boundaries at any moment in time.
Revised: This is a reasonable assumption, because in physical systems the agent is close to one of the boundaries at any moment in time.

Original: When this is not the case, we can deﬁne new constraints as a combination of constraints.
Revised: When this is not the case, we can deﬁne new constraints as a combination of constraints.

Original: Consider c ik as the constraint on the state variable, x i , at time k .
Revised: Consider c ik as the constraint on the state variable, x i , at time k .

Original: We can deﬁne the constraint optimization problem for
Revised: We can deﬁne the constraint optimization problem for

Original: LLQL as:
Revised: LLQL as:

Original: [Equation]
Revised: [Equation]

Original: such that:
Revised: such that:

Original: [Equation]
Revised: [Equation]

Original: 19) 12 is a coefﬁcient added to simplify the mathematical operation.
Revised: 12 is a coefﬁcient added to simplify the mathematical operation.

Original: Using our estimation of the next step, x ik +

Revised: Using our estimation of the next step, x i k +

Original: = x ik +  ( f ik + g ik u k ) , we can derive the optimum action which satisﬁes the constraint as:
Revised: = x i k +  ( f i k + g i k u k ) , we can derive the optimum action which satisﬁes the constraint as:

Original: [Equation]
Revised: [Equation]

Original: where ↵ 1 =  g
Revised: where ↵ 1 =  g

Original: ik d Tk ( d k d Tk )  1 , ↵ 2 =  g ik ( d Tk d k )  1 d Tk , and  ⇤ =
Revised: ik d Tk ( d k d Tk )  1 , ↵ 2 =  g ik ( d Tk d k )  1 d Tk , and  ⇤ =

Original: x ik +  f ik  c k +1  ↵ 2 h k ↵ 1 ↵.
Revised: x ik +  f ik  c k +1  ↵ 2 h k ↵ 1 ↵.

Original: The derivation details for short-term constraints are presented in Section 10.
Revised: The derivation details for short-term constraints are presented in Section C.

Revised: Using DDPG model, the car reached the top in 110 steps.

Revised: The car never reached the top when we used the

Revised: MPC model.

Original:   E XPERIMENTAL R ESULTS
Revised: The mean absolute error for LLQL po- sition estimation is 0 . 00078 , the mean absolute error for LLQL velocity estimation is 0 . 000084 , the mean absolute error for MPC position estimation is 0 .and the mean absolute error for MPC velocity estima- tion is 0 . 00014 .  E XPERIMENTAL R ESULTS

Original: In this section, we demonstrate the performance of LLQL using Mountain Car with Continuous
Revised: In this section, we demonstrate the performance of LLQL using Mountain Car with Continuous

Original: Action (MountainCarContinuous) from OpenAI Gym 1 .
Revised: Action (MountainCarContinuous) from OpenAI Gym 1 .

Original: In Section 8, we apply LLQL to control the crane system shown in Figure 1.
Revised: In Section B, we apply LLQL to control the crane system shown in Figure 1.

Original: 5. M OUNTAIN C AR C ONTINUOUS
Revised: 5. M OUNTAIN C AR C ONTINUOUS

Original: The MountainCarContinuous has two state variables: 1) car’s position  1 .
Revised: The MountainCarContinuous has two state variables: 1) car’s position  1 .

Original: 2  x k  0 .
Revised: 2  x k  0 .

Original: 5 and 2) car’s velocity  0 .
Revised: 5 and 2) car’s velocity  0 .

Original: 07  v k  0 . 07 .
Revised: 07  v k  0 . 07 .

Original: u k is the continuous action at time k .
Revised: u k is the continuous action at time k .

Original: A negative action pushes the car to the left and a positive action pushes the car to the right.
Revised: A negative action pushes the car to the left and a positive action pushes the car to the right.

Original: The experience stops aftersteps or when the car reaches the goal on top of the mountain, x k = 0 .
Revised: The experience stops after 1 ,steps or when the car reaches the goal on top of the mountain, x k = 0 .

Original: 5 , whichever occurs ﬁrst.
Revised: 5 , whichever occurs ﬁrst.

Original: In the beginning of each episode, the car is randomly positioned at  0 .
Revised: In the beginning of each episode, the car is randomly positioned at  0 .

Original: 6  x 0   0 .
Revised: 6  x 0   0 .

Original: The reward for each episode is 100 for reaching the goal on top of the hill minus the squared sum of actions from start to the goal.
Revised: The reward for each episode is 100 for reaching the goal on top of the mountain minus the squared sum of actions from start to the goal.

Revised: Figure 3a shows the cumulative rewards during the training for the LLQL, the DDPG, and a model-based reinforcement learning based on MPC presented by (Nagabandi et al., 2018).

Revised: The details of the MPC based solution is presented in Section E.

Original: Figure 3a shows the  cumulative rewards during the training.
Revised: For each approach we performed the training 20 times, and selected the top 5 models with the maximum cumulative rewards to calculate mean and standard deviation of each episode in Figure 3a. The MPC based solution uses the learned short-term predictive model (system dynamic model) to generate a sequence of actions that maximize the reward over a ﬁnite horizon. Figure 3b shows that the short-term predictive model estimates future states with high precision. However, optimizing for a ﬁnite horizon is a disadvantage for the model-based solution in achieving long-term goals.

Original: We performed the training several times, and the LLQL algorithm converged in less than 20 episodes  in all cases.
Revised: Increasing the horizon may improve the long-term performance, but it also increases the computational costs in the application phase. In our experiments, the car never reached the top of the mountain using the model-based method. Figure 3b presents the ﬁrst 110 steps of a sample experiment. Unlike the MPC based solution, the LLQL and the DDPG algorithms converged in less than 40 episodes (see Figure 3a) and reach the top of the mountain in all experiments (see Figure 3b).

Original: Note that it is possible to achieve faster convergence  by applying imagination rollout.
Revised: Note that it is possible to improve the LLQL convergence time by applying imagination rollout.

Original: In fact, our shortterm prediction model can be used to generate imaginary scenarios.
Revised: In fact, our short-term prediction model can be used to generate imaginary scenarios.

Original: However, sample efﬁciency is beyond the focus of this work.
Revised: However, sample efﬁciency is beyond the focus of this work.

Revised: Figure 3 shows by applying the long-term predictive model, the LLQL algorithm outperforms the model-based reinforcement algorithm in achieving the long-term goals.

Revised: In the next two subsections, we will show the LLQL algorithm can outperform the

Revised: DDPG algorithm in achieving the short-term goals by using the short-term predictive model.

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] [Figure]

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] [Figure]

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] [Figure]

Revised: [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: 5. S HORT - TERM TRAJECTORY
Revised: 5. S HORT - TERM TRAJECTORY

Original: Figure 3b shows that the policy presented in equation (16) can lead the car  the top of the mountain.
Revised: Figure 3b shows that the policy presented in equation (16) can lead the car to the top of the mountain using the LLQL algorithm.

Original: We can see that the car’s velocity is above 0 .
Revised: We can see that the car’s velocity is above 0 .

Original: 025 (the red line) when it reaches the top.
Revised: 025 (the red line) when it reaches the top.

Original: Now consider the case where we want the car to reach the top of the hill with our desired velocity, v d = 0 . 
Revised: Now consider the case where we want the car to reach the top of the mountain with our desired velocity, v d = 0 . 025 .

Original: Using equation (18), we can design a control strategy to reach this goal without requiring model retraining.
Revised: Using equation (18), we can design a control strategy to reach this goal without requiring retraining the LLQL model.

Original: We apply the following hybrid control strategy to reach the top of the mountain with our desired speed.
Revised: We apply the following hybrid control strategy to reach the top of the mountain with our desired speed.

Original: [Equation]
Revised: [Equation]

Original: Figure 4a shows  the car can reach the  mountain with our desired velocity.
Revised: Figure 4a shows that the car can reach the top of the mountain with our desired velocity.

Original: When we did not impose our desired speed to the system, the car reached the top of the mountain in 83 steps (see Figure 3b).
Revised: When we did not impose our desired speed to the system, the car reached the top of the mountain in 78 steps (see Figure 3b).

Original: Demanding a lower speed, slowed down the car and increased the number of steps to 88 (see Figure 4a).
Revised: Demanding a lower speed, slowed down the car and increased the number of steps to 82 (see Figure 4a).

Original: Figure 5b shows the actions with and without the short-term trajectory.
Revised: Figure 4b shows the actions with and without the short-term trajectory.

Original: We can see that the action temporarily becomes negative to reduce the velocity to the desired level and then goes back to positive to push the car to the top of the mountain.
Revised: We can see that the action temporarily becomes negative to reduce the velocity to the desired level and then goes back to positive to push the car to the top of the mountain.

Original: To solve this problem in the traditional way, we had to modify the reward function to achieve both short-term and long-term goals.
Revised: To solve this problem in the traditional way, we had to modify the reward function to achieve both short-term and long-term goals.

Original: For comparison, we perform the following experience.
Revised: For comparison, we perform the following experience.

Original: We apply
Revised: We apply

Original: DDPG networks  with the modiﬁed reward functions shown in Table 1 to solve the MountainCarContinuous with the short-term and long-term goal problem.To achieve the short-term trajectory, we [Figure]
Revised: DDPG networks and MPC based networks with the modiﬁed reward functions shown in Table 1 to solve the MountainCarContinuous with the short-term and long-term goal problem.

Original:  have considered different functions of the absolute error between the car’s velocity and the desired velocity [Table] | v d  v k | in the ﬁnal stages as an additional penalty.
Revised: To achieve the short-term trajectory, we have considered different functions of the absolute error between the car’s velocity and the desired velocity, | v d  v k | in the ﬁnal stages as an additional penalty.

Original: We have tried various architectures and hyper parameters to design the DDPG network.
Revised: We have tried various architectures and hyper parameters to design the DDPG network.

Original: The ﬁnal parameters are presented in Section 9. We train a model for each modiﬁed reward function up to 300 episodes and save the model with maximum cumulative reward for our experiment.
Revised: The ﬁnal parameters are presented in Section D. We train a model for each modiﬁed reward function up to 300 episodes and save the model with maximum cumulative reward for our experiment.

Original: We run each model 10 times and measure the average number of steps  to reach the top of the mountain [Table] and the average error between the car’s ﬁnal speed and the desired speed, v d = 0 .
Revised: We run each model 10 times and measure the average number of steps it takes to reach the top of the mountain, and the average error between the car’s ﬁnal speed and the desired speed, v d = 0 .

Original: Table 1 shows thateven though all the models with  modiﬁed  functions can achieve the long-term goal in all the experiences, reaching the top of the mountain in 10 out of 10 experiments, they cannot achieve the short-term goal, v d = 0 . On the other hand,
Revised: Table 1 shows that the MPC based solution cannot guide the car to the top. The DDPG with all the modiﬁed reward functions can achieve the long-term goal in all the experiences, reaching the top of the mountain in 10 out ofexperiments. However, the DDPG based solutions do not perform very well with regard to the shortterm goal.

Original:  LLQL does not require additional training[Table] or reward modiﬁcation, achieves the long-term goal and has the least velocity error, 0 .[Table]
Revised: On the other hand, LLQL does not require additional training or reward modiﬁcation, achieves the long-term goal and has the least velocity error, 0 .

Original: 5. S HORT - TERM CONSTRAINT
Revised: 5. S HORT - TERM CONSTRAINT

Original: Now consider the case where it is unsafe to drive the car above a speciﬁc speed, for example, we plan to keep the speed under v k  0 . 035 .
Revised: Now consider the case where it is unsafe to drive the car above a speciﬁc speed, for example, we plan to keep the speed under v k  0 . 035 .

Original: We can use the following hybrid control strategy to achieve the long-term goal while keeping the speed safe:
Revised: We can use the following hybrid control strategy to achieve the [Table] [Table] leads

Revised: sharp

Revised: ment

Original:  [Equation]
Revised: the gets close to long-term goal while keeping the speed safe: [Equation]

Revised: use equation (20) , otherwise

Original:  We selected the boundary slightly less than the hazardous area (0.033 instead of 0.035) to be safe.
Revised: [Equation] We selected the boundary slightly less than the hazardous threshold (0.033 instead of 0.035) to be safe.

Original: Figure 5a shows that  the car reaches its goal while staying outside of hazardous areas.
Revised: Figure 5a shows that with the LLQL policy the car reaches its goal while staying outside of hazardous areas.

Revised: The MPC based solution keeps the car outside of hazardous areas but cannot deliver the long-term goal (reaching the top of the mountain).

Revised: The DDPG + modiﬁed reward reaches the top but fails to deliver the short-term goal (keeping the car out of hazardous areas).

Original: Like the previous section, we apply DDPG network + modiﬁed reward function  to compare LLQL with the traditional  reward engineering approach and select the model with maximum cumulative rewards during 300 episodes of training.
Revised: Like the previous section, we apply DDPG network + modiﬁed reward function and MPC + modiﬁed reward function to compare LLQL with the traditional model-free and model-based reward engineering approaches. We select the model with maximum cumulative rewards during 300 episodes of training.

Original: Table 2 shows  unlike LLQL, the modiﬁed rewards fail to keep the car below the allowed speed 100% of the time.
Revised: Table 2 shows that unlike LLQL, the modiﬁed rewards fail to keep the car below the allowed speed while reaching the top of the mountain.

Revised: The model-based reinforcement learning algorithm baseline presented in Section E uses the same short-term network as the LLQL network.

Revised: Table 1 and Tableshow that even though the short-term part of our solution is useful in achieving short-term goals, it is not enough to solve the entire problem and achieve the long-term goal.

Original: [Table]
Revised: [Table]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: 5. E FFECT OF SHORT - TERM GOALS ON LONG - TERM PERFORMANCE
Revised: 5. E FFECT OF SHORT - TERM GOALS ON LONG - TERM PERFORMANCE

Original: Using equations (18) or (20) for deriving a set of actions is equivalent of solving a sub-optimum solution for the long-term goal in order to satisfy the short-term desired trajectories or constraints.
Revised: Using equations (18) or (20) for deriving a set of actions is equivalent to solving a sub-optimum solution for the long-term goal in order to satisfy the short-term desired trajectories or constraints.

Original: When the short-term goals are far from the global optimum solution, the long-term performance degrades.
Revised: When the short-term goals are far from the global optimum solution, the long-term performance degrades.

Original: Figure 6a shows  lower desired velocities lead to longer traveling time for the MountainCar.
Revised: Figure 6a shows that lower desired velocities lead to longer traveling time for the MountainCar.

Original: Similarly, Figure 6b shows  further limiting the maximum velocity degrades the long-term performance.  
Revised: Similarly, Figure 6b shows that further limiting the maximum velocity degrades the long-term performance.  C ONCLUSIONS

Revised: In this work, we presented LLQL as a new model-based RL algorithm with the capability of achieving both short-term and long-term goals without requiring complex reward functions.

Revised: By presenting the advantage function with a locally linear model and separating designing actions from the learning process, our method is capable of adopting control strategies to achieve different short-term goals without retraining the model.

Original: R ELATED W ORK
Revised: This can be very signiﬁcant for industrial applications where the RL algorithms have not been used due to the necessity of different short-term adjustments. The LLQL algorithm deviates from the optimal policy temporarily to address local short-term goals (trajectories or constraints). The agent would return to the optimum policy if the deviation is small enough that the agent is still in the environment explored during the training.

Revised: In the future work, we will investigate conditions where short-term goals are feasible and develop a more analytical approach to set the meta parameters for the controller to guarantee short-term and long-term goals.

Revised: Moreover, we will model uncertainties in short-term prediction model and apply robust control theory to design robust control solutions.

Revised: A R ELATED W ORK

Original: Our work can be categorized as a new model-based RL approach.
Revised: Our work can be categorized as a new model-based RL approach.

Original: Model-based RL algorithms use the environment model which represents the state transition function to plan ahead and select actions that lead to higher rewards.
Revised: Model-based RL algorithms use the environment model which represents the state transition function to plan ahead and select actions that lead to higher rewards.

Original: Several model-based algorithms assume the environment model is known.
Revised: Several model-based algorithms assume the environment model is known.

Original: Among them, AlphaZero (15) is one of the most famous.
Revised: Among them, AlphaZero (Silver et al., 2017) is one of the most famous.

Original: AlphaZero uses the game’s rules (Chess, Shogi and Go) as the environment model to generate a series of self-play simulated games.
Revised: AlphaZero uses the game’s rules (Chess, Shogi and Go) as the environment model to generate a series of selfplay simulated games.

Original: During the simulations, the actions for both players are selected using a Monte-Carlo Tree Search (MCTS) algorithm.
Revised: During the simulations, the actions for both players are selected using a Monte-Carlo Tree Search (MCTS) algorithm.

Original: The MCTS performs as a planning algorithm by generating candidate actions which are superior to the current policy.
Revised: The MCTS performs as a planning algorithm by generating candidate actions which are superior to the current policy.

Original: The neural network parameters are updated at the end of each game to minimize the game prediction error (loss, draw or win) and maximize the similarity of policy vector to the planning algorithm.
Revised: The neural network parameters are updated at the end of each game to minimize the game prediction error (loss, draw or win) and maximize the similarity of policy vector to the planning algorithm.

Original: AlphaZero is limited to the discrete action space problems.
Revised: AlphaZero is limited to the discrete action space problems.

Original: The environment model is typically unknown in real-world applications.
Revised: The environment model is typically unknown in real-world applications.

Original: Therefore, many modelbased RL algorithms learn the state transition model from the data.
Revised: Therefore, many model-based RL algorithms learn the state transition model from the data.

Original: NAF (9) learns a linear model for state transition at each operating point and uses this model to generate additional samples through imagination rollout.
Revised: NAF (Gu et al., 2016) learns a linear model for state transition at each operating point and uses this model to generate additional samples through imagination rollout.

Original: World Models (10) uses a Variational Auto Encoder (VAE) to map a state variable, x 2 X to a lower dimensional variable z in a latent space Z .
Revised: World Models (Ha & Schmidhuber, 2018) uses a Variational Auto Encoder (VAE) to map a state variable, x 2 X to a lower dimensional variable z in a latent space Z .

Original: It then uses a recurrent neural network (RNN) to learn the state transition model in the latent space.
Revised: It then uses a recurrent neural network (RNN) to learn the state transition model in the latent space.

Original: Finally, it applies a simple linear controller to z and the hidden state in the RNN, h , to control the system.
Revised: Finally, it applies a simple linear controller to z and the hidden state in the RNN, h , to control the system.

Original: Imagination-Augmented Agents ( I2As ) (14) introduces two paths: 1) model-free path and 2) imagination path.
Revised: Imagination-Augmented Agents ( I2As ) (Racani`ere et al., 2017) introduces two paths: 1) model-free path and 2) imagination path.

Original: The imagination path learns a transition model and uses this model to generate imagination rollouts.
Revised: The imagination path learns a transition model and uses this model to generate imagination rollouts.

Original: These rollouts are aggregated with the samples in the model-free path.
Revised: These rollouts are aggregated with the samples in the model-free path.

Original: To generate actions in the imagination path, I2As uses the model-free path policy.
Revised: To generate actions in the imagination path, I2As uses the model-free path policy.

Original: Therefore, the rollouts in the imagination path improve as the I2As policy improves.
Revised: Therefore, the rollouts in the imagination path improve as the I2As policy improves.

Original: Using the imagination rollouts, I2As converge faster than a model-free network with the same number of parameters.
Revised: Using the imagination rollouts, I2As converge faster than a model-free network with the same number of parameters.

Original: Nagabandi et al (13) showed that a two-step control policy based on 1) learning the dynamic model and 2) applying MPC to the learned model is signiﬁcantly more sample efﬁcient than model-free RL.
Revised: Nagabandi et al. (2018) showed that a two-step control policy based on 1) learning the dynamic model and 2) applying MPC to the learned model is signiﬁcantly more sample efﬁcient than model-free RL.

Original: However, this approach cannot achieve high rewards.
Revised: However, this approach cannot achieve high rewards.

Original: To achieve higher rewards and preserve sample efﬁciency, they proposed a hybrid model-based and model-free ( MBMF ) algorithm which runs the model-based approach to achieve the initial result in a sample efﬁcient way, it then trains a model-free policy to mimic the learned model-based controller, and uses the resulting imitation policy as the initialization for the ﬁnal model-free RL algorithm.
Revised: To achieve higher rewards and preserve sample efﬁciency, they proposed a hybrid model-based and model-free ( MBMF ) algorithm which runs the model-based approach to achieve the initial result in a sample efﬁcient way, it then trains a modelfree policy to mimic the learned model-based controller, and uses the resulting imitation policy as the initialization for the ﬁnal model-free RL algorithm.

Original: Feinberg et al (6) proposed Model-based Value Expansion ( MVE ) algorithm, which limits the uncertainty in the model by only allowing imagination up to a ﬁxed number of steps, H. MVE uses the learned system dynamic model to generate simulation data up to H steps into the future, and applies these sample points to estimate Q-function.
Revised: Feinberg et al. (2018) proposed Model-based Value Expansion ( MVE ) algorithm, which limits the uncertainty in the model by only allowing imagination up to a ﬁxed number of steps, H. MVE uses the learned system dynamic model to generate simulation data up to H steps into the future, and applies these sample points to estimate Q-function.

Original: Instead of saving simulated samples in an imagination buffer, MVE retrains the dynamic model and generates a new imagination rollout at each step. 
Revised: Instead of saving simulated samples in an imagination buffer, MVE retrains the dynamic model and generates a new imagination rollout at each step. Buckman et al.

Original: (3) expanded MVE algorithm by proposing Stochastic Ensemble Value Expansion ( STEVE ), to generate a solution more robust to model uncertainty.
Revised: (2018) expanded MVE algorithm by proposing Stochastic Ensemble Value Expansion ( STEVE ), to generate a solution more robust to model uncertainty.

Original: Dalal et al (5) proposed safe exploration by modeling constraints using a linear model and applied Lagrangian optimization to modify the action in order to guarantee safety.
Revised: Dalal et al. (2018) proposed safe exploration by modeling constraints using a linear model and applied Lagrangian optimization to modify the action in order to guarantee safety.

Original: In this work, we also used Lagrangian optimization for short-term constraints.
Revised: In this work, we also used Lagrangian optimization for short-term constraints.

Original: However, our approach is different in two ways: 1) our method does not modify the RL action to achieve the goals.
Revised: However, our approach is different in two ways: 1) our method does not modify the RL action to achieve the goals.

Original: Instead, it derives an action by considering both long-term goals and short-term constraints.
Revised: Instead, it derives an action by considering both long-term goals and short-term constraints.

Original: This is possible because our algorithm uses a locally linear model to represent the advantage function.
Revised: This is possible because our algorithm uses a locally linear model to represent the advantage function.

Original: 2) Unlike safe exploration, the focus of this paper is in handling new constraints in the application phase without retraining the model.
Revised: 2) Unlike safe exploration, the focus of this paper is in handling new constraints in the application phase without retraining the model.

Original: C ONCLUSIONS

Original: In this work, we presented LLQL as a new model-based RL algorithm with the capability of achieving both short-term and long-term goals without requiring complex reward functions. By presenting the advantage function with a locally linear model and separating designing actions from the learning process, our method is capable of adopting control strategies to achieve different short-term goals without retraining the model. This can be very signiﬁcant for industrial applications where the RL algorithms have not been used due to the necessity of different short-term adjustments.
Revised: [Figure]

Original: In the future work, we will investigate conditions where short-term goals are feasible and develop a more analytical approach to set the meta parameters for the controller in order to guarantee short-term and long-term goals.

Original: Moreover, we will model uncertainties in short-term prediction model and apply robust control theory to design robust control solutions.

Original:  C RANE C ONTROL S YSTEM
Revised: B C RANE C ONTROL S YSTEM

Original: Gantry cranes are widely used in industrial production lines and construction projects for transferring heavy and hazardous materials.
Revised: Gantry cranes are widely used in industrial production lines and construction projects for transferring heavy and hazardous materials.

Original: The objective of the crane system is to convey the payload from the start position to the destination position as soon as possible while keeping the payload sway at the destination minimum.
Revised: The objective of the crane system is to convey the payload from the start position to the destination position as soon as possible while keeping the payload sway at the destination minimum.

Original: Higher traveling speed improves the efﬁciency and reduces costs.
Revised: Higher traveling speed improves the efﬁciency and reduces costs.

Original: However, excessive movements at the destination wastes time and energy and can lead to accidents.
Revised: However, excessive movements at the destination wastes time and energy and can lead to accidents.

Original: To move the payload as fast as possible and stop the sway at the destination, skillful operators are required.
Revised: To move the payload as fast as possible and stop the sway at the destination, skillful operators are required.

Original: Labor shortage in industries, and risk of human error, have motivated us to develop an automated solution for crane control.
Revised: Labor shortage in industries, and risk of human error, have motivated us to develop an automated solution for crane control.

Original: The crane dynamic system is highly nonlinear.
Revised: The crane dynamic system is highly nonlinear.

Original: Traditional nonlinear control techniques such as sliding control (1) and adaptive control (2) have been applied to these systems.
Revised: Traditional nonlinear control techniques such as sliding control (Bartolini et al., 2002) and adaptive control (Boustany & d’Andrea Novel, 1992) have been applied to these systems.

Original: These methods require detailed mathematical model of the system and its environment, which can be complicated and expensive to derive.
Revised: These methods require detailed mathematical model of the system and its environment, which can be complicated and expensive to derive.

Original: When a simulator is available for a crane system, RL algorithms can provide a compelling alternative to traditional control methodologies.
Revised: When a simulator is available for a crane system, RL algorithms can provide a compelling alternative to traditional control methodologies.

Original: This is the case in many industries, where for intellectual property concerns the companies are willing to provide simulators to the costumers but refuse to reveal mathematical models of their products.
Revised: This is the case in many industries, where for intellectual property concerns the companies are willing to provide simulators to the costumers but refuse to reveal mathematical models of their products.

Original: Our crane simulator provides us six state variables: 1) trolley location, x trolley 2) trolley velocity, v trolley , 3) payload angle,  payload , 4) payload angular velocity, !
Revised: Our crane simulator provides us six state variables: 1) trolley location, x trolley 2) trolley velocity, v trolley , 3) payload angle,  payload , 4) payload angular velocity, !

Original: payload , 5) payload horizontal location, x payload , and 6) payload vertical location, y payload .
Revised: payload , 5) payload horizontal location, x payload , and 6) payload vertical location, y payload .

Original: The only action is the force applied to[Figure] the trolley, u trolley .
Revised: The only action is the force applied to the trolley, u trolley .

Original: The overall goal is to reach the ﬁnal destination x pd and y pd in the shortest time possible.
Revised: The overall goal is to reach the ﬁnal destination x pd and y pd in the shortest time possible.

Original: We choose the following reward function to learn a policy to do so. 
Revised: We choose the following reward function to learn a policy to do so. [Equation]

Original: [Equation]
Revised: [Equation]

Original: where ✏ is a small constant.
Revised: where ✏ is a small constant.

Original: Figure 7b shows  our learned policy pushes the trolley with maximum force, u trolley is eqaual 1 for the entire episode, till the payload reaches the goal x pd = 6 , y pd = 10 .
Revised: Figure 7b shows that our learned policy pushes the trolley with maximum force, u trolley is eqaual 1 for the entire episode, till the payload reaches the goal x pd = 6 , y pd = 10 .

Original: In additional to the long-term goal, our short-term goal is to minimize the object’s sway when it reaches to the ﬁnal destination.
Revised: In additional to the long-term goal, our short-term goal is to minimize the object’s sway when it reaches to the ﬁnal destination.

Original: Instead of designing complicated reward functions to achieve minimum travel time and minimum sway, we consider !
Revised: Instead of designing complicated reward functions to achieve minimum travel time and minimum sway, we consider !

Original: payload = 0 at the ﬁnal destination as a short-term desired trajectory.
Revised: payload = 0 at the ﬁnal destination as a short-term desired trajectory.

Original: We consider the following hybrid strategy to reach the ﬁnal destination with close to zero sway.
Revised: We consider the following hybrid strategy to reach the ﬁnal destination with close to zero sway.

Original: [Equation]
Revised: [Equation]

Original: Figure 8 shows  our strategy can reach the destination with close to zero swing.
Revised: Figure 8 shows that our strategy can reach the destination with close to zero swing.

Original: N ETWORKS P ARAMETERS
Revised: C D ERIVATION D ETAILS FOR S HORT - TERM CONSTRAINTS

Revised: Consider equation (19). Substituting our estimation of the next step from equation (14) in (19), we have [Equation]

Revised: such that:

Revised: [Equation]

Revised: Using Lagrangian method at each time step k , we have

Revised: [Equation]

Revised: Taking the gradient of L with respect to u k , we can write the Karush-Kuhn-Tucker (KKT) (Kuhn &

Revised: Tucker, 2014) conditions for optimal solution of equation (25), { u ⇤ k ,  ⇤ } as:

Revised: [Equation]

Revised: [Equation]

Revised: With this assumption, we can show

Revised: [Equation]

Revised: [Equation]

Revised: Note that when there is no constraint:  ⇤ = 0 , we have u ⇤ k

Revised: =  ( d Tk d k )  1 d Tk h k .

Revised: This is exactly the input we computed in equation (16).

Revised: When  ⇤ 6 = 0 , we have

Revised: [Equation]

Revised: We deﬁne ↵ 1 =

Revised: g

Revised: ik d Tk ( d k d Tk )  1 , and ↵ 2 =  g ik ( d Tk d k )  1 d Tk .

Revised: ↵ 1 and ↵ 2 are scalar.

Revised: Substituting u ⇤ k from equation (28) in equation (29) we have

Revised: [Equation]

Original:  We used the following network structures and parameters in the experimental studies.
Revised: Therefore, [Equation] D N ETWORKS P ARAMETERS We used the following network structures and parameters in the experimental studies.

Original: LLQL for MountainCarContinuous: h , d , V , f and g networks each has two hidden layers withneurons in each layer.
Revised: LLQL for MountainCarContinuous: h , d , V , f and g networks each has two hidden layers withneurons in each layer.

Original: All the activation functions are Rectiﬁed Linear Units (ReLUs).
Revised: All the activation functions are Rectiﬁed Linear Units (ReLUs).

Original: Each episode is maximum 1000 steps.
Revised: Each episode is maximum 1000 steps.

Original: The number of iterations for short-term and long-term prediction model:
Revised: The number of iterations for short-term and long-term prediction model:

Original: I s =
Revised: I s =

Original: I l 
Revised: I l = 5 . The learning rate for the long-term prediction model is 0 . 001 . The batch size for this model is 10. The discount rate  = 0 . 999 .

Revised: The target model update rate, ⌧ = 0 .

Revised: The learning rate for the short-term prediction model is 0 .

Original:  LLQL for the crane system: h , d , V , f and g networks each has two hidden layers with 200 neurons in each layer.
Revised: 001 for the ﬁrst 20000 steps and then reduces to 0 . 0001 .  = 0 . The batch size for this model is 100. LLQL for the crane system: h , d , V , f and g networks each has two hidden layers with 200 neurons in each layer.

Original: All the activation functions are ReLUs.
Revised: All the activation functions are ReLUs.

Original: Each episode includes maximum 1000 actions.
Revised: Each episode includes maximum 1000 actions.

Original: The number of iterations for short-term and long-term prediction model: I s =
Revised: The number of iterations for short-term and long-term prediction model: I s =

Original: I l = 5 .
Revised: I l = 5 .

Original: The learning rate for the long-term prediction model is 0 . 001 .
Revised: The learning rate for the long-term prediction model is 0 . 001 .

Original: The batch size for this model is 10.
Revised: The batch size for this model is 10.

Original: The discount rate  = 0 . 999 .
Revised: The discount rate  = 0 . 999 .

Original: The target model update rate, ⌧ = 0 .
Revised: The target model update rate, ⌧ = 0 .

Original: The learning rate for the short-term prediction model is 0 .
Revised: The learning rate for the short-term prediction model is 0 .

Original: 01 for the ﬁrst 20000 steps
Revised: 01 for the ﬁrst 20000 steps

Original: and then reduces to 0 .
Revised: and then reduces to 0 .

Original:  = 0 .
Revised:  = 0 .

Original: The batch size for this model is 200.
Revised: The batch size for this model is 200.

Original: DDPG networks with modiﬁed reward functions: The Q-network, and the deterministic policy network each has two hidden layers with 200 neurons.
Revised: DDPG networks with modiﬁed reward functions: The Q-network, and the deterministic policy network each has two hidden layers with 200 neurons.

Original: All the activation functions are ReLUs.
Revised: All the activation functions are ReLUs.

Original: The learning rate for the Q-network is 0 . 00001 , and the learning rate for the deterministic policy network is 0 . 000001 .
Revised: The learning rate for the Q-network is 0 . 00001 , and the learning rate for the deterministic policy network is 0 . 000001 .

Original: The discount rate  = 0 . 99 .
Revised: The discount rate  = 0 . 99 .

Original: The batch size is 8.
Revised: The batch size is 8.

Original: The target model update rate for both networks is 0 . 1 .
Revised: The target model update rate for both networks is 0 . 1 .

Original: In all the networks we shift and scale the state variables to zero mean and unit standard deviation for a better learning.
Revised: In all the networks we shift and scale the state variables to zero mean and unit standard deviation for a better learning.

Original: For exploration, we add an additive normal noise to the action:
Revised: For exploration, we add an additive normal noise to the action:

Original: [Equation]
Revised: [Equation]

Original: 25) where u ( k ) ⇤ represents the optimum action generated by LLQL or DDPG, and N =
Revised: [Equation] where u ( k ) ⇤ represents the optimum action generated by LLQL or DDPG, and N =

Original: U (  1 , 1) is a continuous uniform random variable.
Revised: U (  1 , 1) is a continuous uniform random variable.

Original: We consider ↵ N = 0 .
Revised: We consider ↵ N = 0 .

Original: in the beginning, and ↵ N = .
Revised: in the beginning, and ↵ N = .

Original: 99 ⇥ ↵ N after each episode with positive cumulative rewards.
Revised: 99 ⇥ ↵ N after each episode with positive cumulative rewards.

Original: D ERIVATION D ETAILS FOR S HORT - TERM CONSTRAINTS
Revised: E M ODEL - BASED R EINFORCEMENT L EARNING

Original: Consider equation (19).
Revised: Nagabandi et al.

Original: Substituting our estimation of  the next step from equation (14) in (19), we have [Equation] such that:
Revised: (2018) proposed a model-based RL based on MPC. Their approach uses the system dynamic model + reward function to generate a sequence of actions that maximize the cumulative reward.

Original: [Equation]
Revised: To represent the effect of short-term and long-term predictive models in our algorithm, we present a modiﬁed version of (Nagabandi et al., 2018)

Original: Using Lagrangian method at each time step k , we have [Equation]
Revised: algorithm using our short-term predictive model, f and g in equation (14).

Original: [Equation] Taking the gradient of L with respect to u k , we can write the Karush-Kuhn-Tucker (KKT) (11) conditions for optimal solution of equation (26), { u⇤ k ,  ⇤ } as: [Equation]
Revised: The model based RL uses the learned dynamic model, f and g networks, and generates a sequence of actions U Hk = ( u k , ..., u k + H  1 ) to maximize reward over a ﬁnite horizon, H .

Original: [Equation]
Revised: [Equation]

Original: With this assumption, we can show
Revised: such that: ˆ x k = x k and

Original: [Equation]
Revised: [Equation]

Original: Note that when there is no constraint:  ⇤ = 0 , we have u ⇤ k
Revised: Solving equation (33) for the exact solution is computationally expensive. Therefore, Nagabandi et al.

Original: =  ( d Tk d k )  1 d Tk h k .
Revised: (2018) applied a simple random sampling method and select the candidate action sequence with the highest expected cumulative reward.

Original: This is exactly the input we computed in equation (16).
Revised: In this work, we apply the same method to solve equation (33).

Original: When  ⇤ 6 = 0 , we have [Equation] We deﬁne ↵ 1 = 
Revised: We set H = 25 and select 10000 samples at each step.

Original: g ik d Tk ( d k d Tk )  1 , and ↵ 2 =  g ik ( d Tk d k )  1 d Tk .
Revised: Following the MPC closedloop control framework, the algorithm only executes the ﬁrst action in the sequence and calculate a new sequence in the next step.

Original: ↵ 1 and ↵ 2 are scalar. Substituting u ⇤ k from equation (29) in equation (30) we have
Revised: Algorithm 2 presents the model-based RL.

Original: [Equation]
Revised: [Equation]

Original: Therefore,

Original: [Equation]
Revised: [Table]