Continuous Deep Q-Learning in Optimal Control Problems: Normalized Advantage Functions Analysis
Anonymous Author(s)
AffiliationAddress email
Abstract
One of the most effective continuous deep reinforcement learning algorithms isnormalized advantage functions (NAF).
The main idea of NAF consists in theapproximation of the Q-function by functions quadratic with respect to the actionvariable.
This idea allows to apply the algorithm to continuous reinforcementlearning problems, but on the other hand, it brings up the question of classesof problems in which this approximation is acceptable.
The presented paperdescribes one such class.
We consider reinforcement learning problems obtainedby the time-discretization of certain optimal control problems.
Based on the ideaof NAF, we present a new family of quadratic functions and prove its suitableapproximation properties.
Taking these properties into account, we provide severalways to improve NAF.
The experimental results confirm the efficiency of ourimprovements.
Introduction
The standard reinforcement learning (RL) [Sutton and Barto, 2018] setup consists of an agentinteracting with an environment.
At each step of the interaction, the agent determines an action basedon its policy and its current state, gets a reward, and makes a transition to the next state.
The aim ofthe agent is to learn the policy that maximizes the sum of rewards.
One of the most widespread approach for solving RL problems is Q-learning [Watkins and Dayan,1992], which consist in seeking the optimal action-value function (Q-function) as a solution of the
Bellman optimality equation.
After the learning, the agent can act optimally by the found Q-function.
Initially, Q-learning approach was applied for solving RL problems with finite state and action spaces.
In this case, Q-function can be represented by a finite table.
For the case of a large or continuous statespace, Q-learning has later been extended to Deep Q-learning algorithm
[Mnih et al., 2015] whichallows to seek the approximations of Q-function in the class of neural networks by means of thestochastic gradient descent.
Deep Q-learning and its modifications have shown efficiency for a rangeof challenging tasks [Wang et al., 2016, van Hasselt et al., 2016, Schaul et al., 2016, Hessel et al.
However, this algorithm can not be directly applied for solving RL problems with continuousaction spaces.
The reason is that Deep Q-learning involves maximizing of approximate Q-function bythe action variable on each step of the learning, which is a complex procedure for continuous actionspaces.
Among various approaches to overcome this problem [Lillicrap et al., 2016, Haarnoja et al.
,2017, Kalashnikov et al., 2018, Lim et al., 2019, Ryu et al., 2020, Lutter et al., 2021], we focus on theidea of the normalized advantage functions (NAF) algorithm
[Gu et al., 2016], which consists in theapproximation of Q-function by the function quadratic with respect to the action variable.
It allows tocalculate the maximum quite fast and precisely and solve some challenging control problems [Guet al., 2017, Dong et al., 2018, Ikemoto and Ushio, 2021], but on the other hand, it brings up the question of classes of RL problems in which this approximation is acceptable.
The presented paperdescribes one of the possible answers to this question.
Note that the class of LQR problems [Bradtke et al., 1994] has Q-functions quadratic with respectto the action variable.
However, this class, being quite special, is not suitable for the descriptionof non-linear controlled processes.
In this paper, we study a wider (in some sense) class of RLproblems.
We consider RL problems which are obtained by time-discretization of finite-horizonoptimal control problems [Bardi and Dolcetta, 1997] in dynamical system is described by affine incontrol ordinary differential equations and the cost functional is quadratic in control.
Note that a lotof RL problems with continuous action spaces arise from control problems for mechanical or roboticsystems [Lillicrap et al., 2016, Gu et al., 2016, Haarnoja et al., 2017, Gu et al., 2017, Kalashnikovet al., 2018] which are, in fact, described by affine in control ordinary differential equations.
Besides,often, the quality index in a control problem can be determined as quadratic in control.
For instance, ifthe quality index in an initial problem does not depend on control, then one can add a small quadraticterm such that, on the one hand, the quality index becomes quadratic, and on the other hand, the senseof the problem does not changed much.
Thus, the class of control problems under considerationseems quite general.
For this class, based on the idea of NAF, we present a new family of quadratic functions and prove that,first, this family is sufficiently rich to approximately solve the Bellman optimality equation (Theorem1), and second, any sufficiently accurate solution, from this family, of the Bellman optimality equationallows to approximately obtain the optimal policy in the corresponding optimal control problem(Theorem 2).
Moreover, we prove that it is impossible to get the same results for the original familyof functions from Gu et al.
[2016] (Theorem 3).
Next, we provide several ways to utilize the obtainedtheoretical knowledge in order to improve NAF.
The experimental results confirm the efficiency ofour improvements.
Note that, in contrast to LQR problems, we can not say that Q-function is quadratic in our class ofproblems.
Moreover, it is not true in general case.
Nonetheless, according to presented statements(Theorem 1,2), Q-function can be approximated by quadratic functions with sufficient order ofaccuracy to obtain near-optimal policy.
Background
The standard reinforcement learning (RL) setup consists of an agent interacting with an environment [Sutton and Barto, 2018].
This interaction is described by a Markov Decision Process (MDP), which is a tuple ( S , U , P , R , ρ 0 , γ ) , where S is a state space, U is an action space, P ( s ′ | s, u ) is a transition distribution, R ( s, u ) is a reward function, ρ 0 ( s ) is an initial state distribution, and γ ∈ [0 , 1] is a discount factor.
The aim of the agent is to learn optimal policy µ ∗ ( s ) which maximizes the value
J
[Equation]
In the general statement of reinforcement learning problems, a policy of the agent can be stochastic,however, within this paper, we assume that the policy is deterministic.
One of the most effective approach for solving RL problems is Q-learning [Watkins and Dayan, 1992], which consist in seeking the optimal action-value function
[Equation]
as a solution of the Bellman optimality equation
[Equation]
In other words, the agent solves the following minimization problem:
[Equation]
If the agent knows the function Q ∗ ( s, a ) , it can act optimally by the greedy policy
[Equation]
Initially, the Q-learning approach was applied for solving RL problems with finite state and action spaces.
In this case, the Q-function is represented by a finite table and problem (1) is finitedimensional.
For the case of a large or continuous state space, Q-learning has later been extended to
Deep Q-learning algorithm
[Mnih et al., 2015] which allows to seek approximations of Q-functionin the class of neural networks Q ( x, u | θ Q ) , where θ Q is the parameter vector of the neural network.
During the learning, the experiences ( s i , u i , r i , s i +1 ) are stored in the buffer D and simultaneouslythe parameter vector θ Q is updated by means of the stochastic gradient descent minimizing the lossfunction
[Equation]
where U ( D ) is the uniform distribution on D .
Deep Q-learning and its modifications are effective fora range of challenging tasks [Wang et al., 2016, van Hasselt et al., 2016, Schaul et al., 2016, Hesselet al., 2018].
However, note that this algorithm can not be directly applied for solving RL problemswith continuous action spaces.
The reason is that Deep Q-learning involves the maximizing in (2) oneach step of the learning, which is a complex procedure for continuous U .
Among various approachesto overcome this problem [Lillicrap et al., 2016, Haarnoja et al., 2017, Kalashnikov et al., 2018, Limet al., 2019, Ryu et al., 2020, Lutter et al., 2021], we focus on an idea of the normalized advantagefunctions (NAF) algorithm
[Gu et al., 2016].
This idea consists in the approximation of Q-functionby the following quadratic with respect to u functions:
[Equation]
) where V ( s | θ V ) , µ ( s | θ µ ) , and P ( s | θ P ) are neural networks with parameters θ V , θ µ , and θ P , respectively; P ( s | θ P ) is a positive-definite square matrix for each s and θ P ; θ
A = { θ µ , θ P } andθ Q = { θ A , θ V } .
Under the condition
[Equation]
it allows to get the maximum and argmaximum values directly by values of V ( s | θ V ) and µ ( s | θ µ ) :
[Equation]
but on the other hand, it brings up the question of classes of RL problems in which quadraticapproximations is acceptable.
Below, we describes one such class.
Problem statement
In this section, we consider a certain class of optimal control problems and show that discreteapproximations of these problems can be formalized as RL problems.
Consider the following optimal control problem: it is required to maximize the functional
[Equation]
6) over all u ( · ) , where x ( · ) is the solution [Filippov, 1988, §1] of the differential equationd
[Equation]
(7) under the initial condition
[Equation]
Here t is the time variable, T > 0 is the terminal instant of time, x
( t ) ∈
R n is the current statevector, u ( t ) ∈
U is the current control action vector forming the measurable function u ( · ) , U ⊂ R mis the nonempty compact set, z ∈ R n is the fixed initial state vector, f ( t, x ) ∈ R n , g ( t, x ) ∈ R n × m , q ( t, x ) ∈ R , r ( t, x ) ∈ R m × m , ( t, x ) ∈ [0 , T ] × R n are continuous with respect to t and continuouslydifferentiable with respect to x functions, r ( t, x ) is the positive-definite matrix for each ( t, x ) ∈[0 , T ] × R n , and σ ( x ) ∈ R , x ∈ R n is the continuously differentiable function.
We assume that thereexists a constant c fg > 0 such that
[Equation]
Note that, under these conditions, for each function u ( · ) , there exists a unique solution x ( · ) ofequation (7) under the initial condition (8) [Filippov, 1988, §1].
Define the value function in optimal control problem (6), (7) by
[Equation]
where, for each u ( · ) , x ( · ) is the solution of equation (7) on the interval [ t ∗ , T ] under the initialcondition x ( t ∗ )
= x ∗ .
Define the sets
[Equation]
Let k ∈ N , ∆ t k = T/k , and t i = i ∆ t k , i ∈ 0 , k .
Consider the corresponding discrete optimalcontrol problem: it is required to maximize the function
[Equation]
over all u i ∈ U , i ∈ 0 , k − 1 , where ( x 0 , x 1 , . . .
, x k ) is defined by
[Equation]
Let us show that problem (12), (13) can be formalized as the RL problem.
First, we define the stateand actions spaces, the initial state distribution, and the discount factor as follows:
[Equation]
Here s T is some fictional terminal state, δ is Dirac delta distribution.
Next, for every i ∈ 0 , k − 1 ,x ∈ S ( t i ) , and u ∈ U , we define the transition distribution and the reward function by
P ( s ′ | s =
( t i , x ) , u ) =
δ ( s ′ =
( t i +1 , x ′ )) ,
[Equation]
where x ′ = x
+ ( f ( t i , x ) + g
( t i , x ) u )∆ t k .
Taking into account (9) and (11), one can prove theinclusion ( t i +1 , x ′ ) ∈ S .
Hence, the transition distribution P is well-defined.
For i =
k , we set
P ( s ′ | s =
( t k , x ) ,
[Equation]
=
δ ( s ′ =
s T ) ,
[Equation]
U.
In order to make dynamical processes (13) formally infinite, we put
[Equation]
Thus, we define MDP which describes the RL problem corresponding to problem (12), (13).
Next,we show that such RL problems is suitable for using quadratic approximations of the Q-function.
Quadratic approximations of the Q-function
Denote by Q the family of functions Q such that
[Equation]
18) where
A ( t, x, u ) = − ( u − ˜ µ ( t, x ))
T P ( t, x )( u − ˜ µ ( t, x ))
[Equation]
Here V ( t, x ) , ˜ µ ( t, x ) , and P ( t, x ) are continuous functions; P ( t, x ) is a positive-definite squarematrix for each ( t, x ) ∈ [0 , T )
× R n .
Note that, functions Q from the family Q satisfy the equalities
[Equation]
as well as (see (5))
quadratic functions from family (3).
However, these function families are different.
The difference is that we do not assume the inclusion ˜ µ ( t, x ) ∈ U as opposed to assumption (4).
The theorems below establish a connection between the optimal control problem (6)–(8) and minimization problems (1) for MDP (14)–(17).
Theorem 1.
Let the value function V ∗ ( t, x ) be continuously differentiable.
Then, for every ε > 0 ,there exists k ∗ > 0 such that, for every k ≥ k ∗ , the function Q ∈ Q defined by (18) and (19) where
V ( t, x ) =
[Equation]
∗ ( t [Equation] x
[Equation]
satisfies the inequality
[Equation]
[Equation] 22) where we assume Q ( t k , x ′ , u ′ )
=
σ ( x ′ ) .
Theorem 2.
Let the value function V ∗ ( t, x ) be continuously differentiable.
Let ε > 0 and k ∗ > 0 bedefined according to Theorem 1.
Take k ≥ k ∗ and suppose that a function Q ∈ Q satisfies inequality(22).
Then the following estimate holds:
[Equation]
where the function J k ( u 0 , u 1 , . . .
u k − 1 ) is defined by (12) with u i = µ ( t i , x i ) , i ∈ 0 , k − 1 and thefunction µ ( t, x ) is defined by Q according to (18) and (19).
Thus, Theorem 1 shows that the function family Q is sufficiently rich to contain approximate solutionsof problem (1) with a predetermined accuracy and Theorem 2 establishes that all such approximatesolutions contained in Q allow us to get the policy, which approximately provides the optimal resultin optimal control problem (6)–(8).
Now, let us consider the original family of functions from Gu et al.
Denote by Q NAF thefamily of functions Q such that
[Equation]
where V ( t, x ) , µ ( t, x ) , and P ( t, x ) are continuous functions; P ( t, x ) is a positive-definite squarematrix for each ( t, x ) ∈ [0 , T ) × R n ; µ ( t, x ) ∈ U for each ( t, x ) ∈ [0 , T )
× R n .
The theorem belowestablishes that if we take the family Q NAF instant of Q , then Theorem 1 can not be proved even inthe simplest cases of optimal control problem (6)–(8).
Theorem 3.
Let n = m = 1 , T = 1 , U =
[ − 1 , 1] , f ( t, x ) =
q ( t, x ) = 0 , g ( t, x ) =
r ( t, x ) = 1 , σ ( x ) = − x 2 , and z = 2 .
Then, for every k ≥ 4 and Q ∈ Q NAF , there exist i ∈ 0 , k − 1 , x ∈ S ( t i ) , and u ∈ U such that
[Equation]
where we assume Q ( t k , x ′ , u ′ )
= − ( x ′ ) 2 .
Proofs of the theorems are given in the Appendix A.
Experiments
We consider four examples of optimal control problems (6)–(8) described in Table 1, where
[Equation]
[Table]
A target problem
[Table]
Van der Pol oscillator is a famous model of a non-conservative oscillator with non-linear damping.
The aim of the control is to stabilize the oscillator at the terminal time.
Pendulum is a traditional problem for testing control algorithms.
The aim of the control is thestabilization of the pendulum in the top position at the terminal time.
Dubins car is a quite famous model which describes a motion of the point particle moving at aconstant speed on the plane.
The problem is to find a control providing the closeness of the motionwith a target point at the terminal time.
A target problem is an optimal control problem presented in Munos [2006].
The dynamic systemdescribes a hand holding a spring to which is attached a mass.
It is required to control the hand suchthat the mass achieve the target point at the terminal time.
5.1 Bounded NAF
First, we modify NAF algorithm, proposed in Gu et al.
[2016], based on the function family Q .
Note that, the considered examples have U =
[ α, β ] m .
Denote clip( ν ) =
max { α, min { β, ν }} , ν ∈ R m , α, β ∈ R m .
Then, according to (18) and (19), we can use the following approximation of the Q-function, within NAF algorithm:
[Equation]
where V ( t, x | θ V ) , ˜ µ ( t, x | θ ˜ µ ) , and P ( t, x | θ P ) are neural networks with parameters θ V , θ ˜ µ , and θ P ,respectively; P ( t, x | θ P ) is a positive-definite square matrix for each ( t, x ) and θ P ; θ
A = { θ ˜ µ , θ P }and θ Q = { θ A , θ V } .
To be short, we call this algorithm Bounded NAF (BNAF), because it is essentialfor our modification of NAF that the set U is bounded.
5.2 Reward-based BNAF
If we know the function r ( t, x ) , then, according to Theorem 1, we can use the function r ( t, x )∆ t kinstead of the neural network P ( t, x | θ P ) to reduce the number of learning parameters.
This variantof BNAF is called Reward-based BNAF (RB-BNAF).
5.3 Gradient-based BNAF
If we also know the function g ( t, x ) , then, according to the Theorem 1, we can use the function
[Equation]
instead of the neural network ˜ µ ( t, x | θ ˜ µ ) . It also reduces the number of learning parameters.
Thisvariant of BNAF is called Gradient-based BNAF (GB-BNAF).
5.4 Experimental results
We use the same learning parameters of every our tasks.
We apply neural networks with two layersof 256 and 128 rectified linear units (ReLU) and learn their used ADAM with the learning ratelr = 1 e − 3 .
We use batch size n bs = 128 and smoothing parameter τ = 1 e − 2 .
Also we take∆ t = 0 .
All calculations were performed on a personal computer in a standard way.
We compare NAF, BNAF, RB-BNAF, GB-BNAF algorithms, and DDPG algorithm presented in[Lillicrap et al., 2016].
Figure 1 shows learning curves of the algorithms for the considered examples.
One can note that RB-BNAF algorithm has a good performance in all examples.
GB-BNAF is slightlyinferior to RB-BNAF everywhere except A target problem.
DDPG and BNAF have excellent resultsonly one example out of four.
NAF algorithm shows not the worst performance only in Pendulumexample.
Thus, taking the presented experiments into account, we can give the following general recommendations.
For solving optimal control problems (6)–(8), it is rational to use GB-BNAF and especially
RB-BNAF algorithms along with NAF and DDPG algorithms.
There is a high probability that theywill show better results and more stable learning.
Related works
Many different ways to apply reinforcement learning approaches for solving optimal control problemsare investigated [Baird, 1994, Doya, 1995, 2000, Munos, 2006, Tallec et al., 2019, Kim and Yang,2020, Lutter et al., 2021].
Among them, a time-discretization is perhaps the most obvious and widelyused tool [Lillicrap et al., 2016, Gu et al., 2016, Haarnoja et al., 2017, Gu et al., 2017, Kalashnikovet al., 2018].
From the theoretical point of view, it is known [Bardi and Dolcetta, 1997, p.388] thatsolutions of time-discrete optimal control problems converge to the solution of the initial problemas the discretization step tends to zero.
In the present paper, we also use the time-discretization andstudy approximating solutions of Bellman equations and the corresponding greedy politics dependingon the discretization step (see Theorems 1 and 2).
Other studies of dependencies on the discretizationstep of reinforcement learning methods can be found in Munos [2006], Tallec et al.
We focus on the idea from Gu et al.
[2016] to expand Q-learning algorithm to optimal controland reinforcement learning problems with continuous actions.
Other approaches for solving suchproblems are investigated in Lillicrap et al.
[2016], Haarnoja et al.
[2017], Kalashnikov et al.
Lim et al.
[2019], Ryu et al.
[2020], Lutter et al.
The paper Lutter et al.
[2021] seems theclosest to the presented paper.
In Lutter et al.
[2021], the similar optimal control problem and feedbackcontrol policy are considered, but the proposed algorithms require knowledge of the right-hand sideof differential equations in contrast to algorithms suggested in presented paper.
Let us also note the family of functions (18), (19) is included, in some sense, to families of Q-functionsconsidered in Wang et al.
[2016], Tallec et al.
It seems to be expected because more generalclasses of problems are considered in these papers.
Nevertheless, the proposed algorithms and resultsare very different from presented in this paper.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] See Abstract
(b) Did you describe the limitations of your work?
[Yes] See Introduction
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See SectionProblem statement
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] See Supple-
mentary material
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section Experiments
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See Section Experiments
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section Experiments
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]