Continuous Deep Q-Learning in Optimal Control Problems: Normalized Advantage Functions Analysis
Anonymous Author(s)
AffiliationAddress email
Abstract
One of the most effective continuous deep reinforcement learning algorithms isnormalized advantage functions (NAF).
The main idea of NAF consists in theapproximation of the Q-function by functions quadratic with respect to the actionvariable.
This idea allows to apply the algorithm to continuous reinforcementlearning problems, but on the other hand, it brings up the question of classesof problems in which this approximation is acceptable.
The presented paperdescribes one such class.
We consider reinforcement learning problems obtainedby the time-discretization of certain optimal control problems.
Based on the ideaof NAF, we present a new family of quadratic functions and prove its suitableapproximation properties.
Taking these properties into account, we provide severalways to improve NAF.
The experimental results confirm the efficiency of ourimprovements.
Introduction
The standard reinforcement learning (RL) [Sutton and Barto, 2018] setup consists of an agentinteracting with an environment.
At each step of the interaction, the agent determines an action basedon its policy and its current state, gets a reward, and makes a transition to the next state.
The aim ofthe agent is to learn the policy that maximizes the sum of rewards.
One of the most widespread approach for solving RL problems is Q-learning [Watkins and Dayan,1992], which consist in seeking the optimal action-value function (Q-function) as a solution of the
Bellman optimality equation.
After the learning, the agent can act optimally by the found Q-function.
Initially, Q-learning approach was applied for solving RL problems with finite state and action spaces.
In this case, Q-function can be represented by a finite table.
For the case of a large or continuous statespace, Q-learning has later been extended to Deep Q-learning algorithm
[Mnih et al., 2015] whichallows to seek the approximations of Q-function in the class of neural networks by means of thestochastic gradient descent.
Deep Q-learning and its modifications have shown efficiency for a rangeof challenging tasks [Wang et al., 2016, van Hasselt et al., 2016, Schaul et al., 2016, Hessel et al.
However, this algorithm can not be directly applied for solving RL problems with continuousaction spaces.
The reason is that Deep Q-learning involves maximizing of approximate Q-function bythe action variable on each step of the learning, which is a complex procedure for continuous actionspaces.
Among various approaches to overcome this problem [Lillicrap et al., 2016, Haarnoja et al.
,2017, Wu et al., 2017, Kalashnikov et al., 2018, Nachum et al., 2018, Fujimoto et al., 2018, Limet al., 2019, Ryu et al., 2020, Lutter et al., 2021], we focus on the idea of the normalized advantagefunctions (NAF) algorithm
[Gu et al., 2016], which consists in the approximation of Q-function bythe function quadratic with respect to the action variable.
It allows to calculate the maximum quitefast and precisely and solve some challenging control problems [Gu et al., 2017, Dong et al., 2018, Ikemoto Ushio, 2021], but on the other hand, it brings up the question of classes of RL problems in which this approximation is acceptable.
The presented paper describes one of the possible answersto this question.
Note that the class of LQR problems [Bradtke et al., 1994] has Q-functions quadratic with respectto the action variable.
However, this class, being quite special, is not suitable for the descriptionof non-linear controlled processes.
In this paper, we study a wider (in some sense) class of RLproblems.
We consider RL problems which are obtained by time-discretization of finite-horizonoptimal control problems [Bardi and Dolcetta, 1997] in dynamical system is described by affine incontrol ordinary differential equations and the cost functional is quadratic in control.
Note that a lotof RL problems with continuous action spaces arise from control problems for mechanical or roboticsystems [Lillicrap et al., 2016, Gu et al., 2016, Haarnoja et al., 2017, Gu et al., 2017, Kalashnikovet al., 2018] which are, in fact, described by affine in control ordinary differential equations.
Besides,often, the cost functional in a control problem can be determined as quadratic in control.
For instance,if the quality index in an initial problem does not depend on control, then one can add a smallquadratic term such that, on the one hand, the quality index becomes quadratic, and on the otherhand, the sense of the problem does not changed much.
Thus, the class of control problems underconsideration seems quite general.
For this class, based on the idea of NAF, we present a new family of quadratic functions and prove that,first, this family is sufficiently rich to approximately solve the Bellman optimality equation (Theorem1), and second, any sufficiently accurate solution, from this family, of the Bellman optimality equationallows to approximately obtain the optimal policy in the corresponding optimal control problem(Theorem 2).
Moreover, we prove that it is impossible to get the same results for the original familyof functions from Gu et al.
[2016] (Theorem 3).
Next, we provide several ways to utilize the obtainedtheoretical knowledge in order to improve NAF.
The experimental results confirm the efficiency ofour improvements.
Note that, in contrast to LQR problems, we can not say that Q-function is quadratic in our class ofproblems.
Moreover, it is not true in general case.
Nonetheless, according to presented statements(Theorem 1,2), Q-function can be approximated by quadratic functions with sufficient order ofaccuracy to obtain near-optimal policy.
Background
The standard reinforcement learning (RL) setup consists of an agent interacting with an environment [Sutton and Barto, 2018].
This interaction is described by a Markov Decision Process (MDP), which is a tuple ( S , U , P , R , ρ 0 , γ ) , where S is a state space, U is an action space, P ( s ′ | s, u ) is a transition distribution, R ( s, u ) is a reward function, ρ 0 ( s ) is an initial state distribution, and γ ∈ [0 , 1] is a discount factor.
The aim of the agent is to learn optimal policy µ ∗ ( s ) which maximizes the value
J ( µ [Equation] =
[Equation]
In the general statement of reinforcement learning problems, a policy of the agent can be stochastic,however, within this paper, we assume that the policy is deterministic.
One of the most effective approach for solving RL problems is Q-learning [Watkins and Dayan, 1992], which consist in seeking the optimal action-value function
[Equation]
as a solution of the Bellman optimality equation
[Equation]
In other words [Equation] the agent solves the following minimization problem: [Equation]
[Equation]
If the agent knows the function Q ∗ ( s, a ) , it can act optimally by the greedy policy
[Equation]
Initially, the Q-learning approach was applied for solving RL problems with finite state and action spaces.
In this case, the Q-function is represented by a finite table and problem (1) is finitedimensional.
For the case of a large or continuous state space, Q-learning has later been extended to
Deep Q-learning algorithm
[Mnih et al., 2015] which allows to seek approximations of Q-functionin the class of neural networks Q ( x, u | θ Q ) , where θ Q is the parameter vector of the neural network.
During the learning, the experiences ( s i , u i , r i , s i +1 ) are stored in the buffer D and simultaneouslythe parameter vector θ Q is updated by means of the stochastic gradient descent minimizing the lossfunction
[Equation]
where U ( D ) is the uniform distribution on D .
Deep Q-learning and its modifications are effective fora range of challenging tasks [Wang et al., 2016, van Hasselt et al., 2016, Schaul et al., 2016, Hesselet al., 2018].
However, note that this algorithm can not be directly applied for solving RL problemswith continuous action spaces.
The reason is that Deep Q-learning involves the maximizing in (2) oneach step of the learning, which is a complex procedure for continuous U .
Among various approachesto overcome this problem [Lillicrap et al., 2016, Haarnoja et al., 2017, Kalashnikov et al., 2018, Limet al., 2019, Ryu et al., 2020, Lutter et al., 2021], we focus on an idea of the normalized advantagefunctions (NAF) algorithm
[Gu et al., 2016].
This idea consists in the approximation of Q-functionby the following quadratic with respect to u functions:
[Equation]
) where V ( s | θ V ) , µ ( s | θ µ ) , and P ( s | θ P ) are neural networks with parameters θ V , θ µ , and θ P , respectively; P ( s | θ P ) is a positive-definite square matrix for each s and θ P ; θ
A = { θ µ , θ P } andθ Q = { θ A , θ V } .
Under the condition
[Equation]
it allows to get the maximum and argmaximum values directly by values of V ( s | θ V ) and µ ( s | θ µ ) :
[Equation]
but on the other hand, it brings up the question of classes of RL problems in which quadraticapproximations is acceptable.
Below, we describes one such class.
Problem statement
In this section, we consider a certain class of optimal control problems and show that discreteapproximations of these problems can be formalized as RL problems.
Consider the following optimal control problem: it is required to maximize the functional
[Equation]
[Equation] over all u ( · ) , where x ( · ) is the solution [Filippov, 1988, §1] of the differential equationd
[Equation]
(7) under the initial condition
[Equation]
Here t is the time variable, T > 0 is the terminal instant of time, x
( t ) ∈
R n is the current statevector, u ( t ) ∈
U is the current control action vector forming the measurable function u ( · ) , U ⊂ R mis the nonempty compact set, z ∈ R n is the fixed initial state vector, f ( t, x ) ∈ R n , g ( t, x ) ∈ R n × m , q ( t, x ) ∈ R , r ( t, x ) ∈ R m × m , ( t, x ) ∈ [0 , T ] × R n are continuous with respect to t and continuouslydifferentiable with respect to x functions, r ( t, x ) is the positive-definite matrix for each ( t, x ) ∈[0 , T ] × R n , and σ ( x ) ∈ R , x ∈ R n is the continuously differentiable function.
We assume that thereexists a constant c fg > 0 such that
[Equation]
Note that, under these conditions, for each function u ( · ) , there exists a unique solution x ( · ) ofequation (7) under the initial condition (8) [Filippov, 1988, §1].
Define the value function in optimal control problem (6), (7) by
[Equation]
where, for each u ( · ) , x ( · ) is the solution of equation (7) on the interval [ t ∗ , T ] under the initialcondition x ( t ∗ )
= x ∗ .
Define the sets
[Equation]
Let k ∈ N , ∆ t k = T/k , and t i = i ∆ t k , i ∈ 0 , k .
Consider the corresponding discrete optimalcontrol problem: it is required to maximize the function
[Equation]
over all u i ∈ U , i ∈ 0 , k − 1 , where ( x 0 , x 1 , . . .
, x k ) is defined by
[Equation]
Let us show that problem (12), (13) can be formalized as the RL problem.
First, we define the stateand actions spaces, the initial state distribution, and the discount factor as follows:
[Equation]
Here s T is some fictional terminal state, δ is Dirac delta distribution.
Next, for every i ∈ 0 , k − 1 ,x ∈ S ( t i ) , and u ∈ U , we define the transition distribution and the reward function by
P ( s ′ | s =
( t i , x ) , u ) =
δ ( s ′ =
( t i +1 , x ′ )) ,
[Equation]
where x ′ = x
+ ( f ( t i , x ) + g
( t i , x ) u )∆ t k .
Taking into account (9) and (11), one can prove theinclusion ( t i +1 , x ′ ) ∈ S .
Hence, the transition distribution P is well-defined.
For i =
k , we set
P ( s ′ | s =
[Equation]
k , [Equation]
[Equation]
=
[Equation] =
[Equation]
T
[Equation] [Equation] In order to make dynamical processes (13) formally infinite, we put
[Equation]
Thus, we define MDP which describes the RL problem corresponding to problem (12), (13).
Next,we show that such RL problems is suitable for using quadratic approximations of the Q-function.
Quadratic approximations of the Q-function
Denote by Q the family of functions Q such that
[Equation]
18) where
A ( t, x, u ) = − ( u − ˜ µ ( t, x ))
T P ( t, x )( u − ˜ µ ( t, x ))
[Equation]
Here V ( t, x ) , ˜ µ ( t, x ) , and P ( t, x ) are continuous functions; P ( t, x ) is a positive-definite squarematrix for each ( t, x ) ∈ [0 , T )
× R n .
Note that, functions Q from the family Q satisfy the equalities
[Equation]
as well as (see (5))
quadratic functions from family (3).
However, these function families are different.
The difference is that we do not assume the inclusion ˜ µ ( t, x ) ∈ U as opposed to assumption (4).
Inother words, the vertex of a quadratic form (3) belongs to the set U , while this is not required in thefunction family Q .
Thus, Q is a wider family of quadratic functions.
The theorems below establish a connection between the optimal control problem (6)–(8) and minimization problems (1) for MDP (14)–(17).
Theorem 1.
Let the value function V ∗ ( t, x ) be continuously differentiable.
Then, for every ε > 0 ,there exists k ∗ > 0 such that, for every k ≥ k ∗ , the function Q ∈ Q defined by (18) and (19) where
V ( t, x ) =
V ∗ ( t, x ) ,
[Equation]
satisfies the inequality
[Equation]
where we assume Q ( t k , x ′ , u ′ )
=
σ ( x ′ ) .
Theorem 2.
Let the value function V ∗ ( t, x ) be continuously differentiable.
Let ε > 0 and k ∗ > 0 bedefined according to Theorem 1.
Take k ≥ k ∗ and suppose that a function Q ∈ Q satisfies inequality(22).
Then the following estimate holds:
[Equation]
where the function J k ( u 0 , u 1 , . . .
u k − 1 ) is defined by (12) with u i = µ ( t i , x i ) , i ∈ 0 , k − 1 and thefunction µ ( t, x ) is defined by Q according to (18) and (19).
Thus, Theorem 1 shows that the function family Q is sufficiently rich to contain approximate solutionsof problem (1) with a predetermined accuracy and Theorem 2 establishes that all such approximatesolutions contained in Q allow us to get the policy, which approximately provides the optimal resultin optimal control problem (6)–(8).
Now, let us consider the original family of functions from Gu et al.
Denote by Q NAF thefamily of functions Q such that
[Equation]
where V ( t, x ) , µ ( t, x ) , and P ( t, x ) are continuous functions; P ( t, x ) is a positive-definite squarematrix for each ( t, x ) ∈ [0 , T ) × R n ; µ ( t, x ) ∈ U for each ( t, x ) ∈ [0 , T )
× R n .
The theorem belowestablishes that if we take the family Q NAF instant of Q , then Theorem 1 can not be proved even inthe simplest cases of optimal control problem (6)–(8).
Theorem 3.
Let n = m = 1 , T = 1 , U =
[ − 1 , 1] , f ( t, x ) =
q ( t, x ) = 0 , g ( t, x ) =
r ( t, x ) = 1 , σ ( x ) = − x 2 , and z = 2 .
Then, for every k ≥ 4 and Q ∈ Q NAF , there exist i ∈ 0 , k − 1 , x ∈ S ( t i ) , and u ∈ U such that
[Equation]
where we assume Q ( t k , x ′ , u ′ )
= − ( x ′ ) 2 .
Proofs of the theorems are given in the Appendix A.
Note that, in the particular case U =
R m , the family Q coincides with family Q NAF .
However,within this paper, U is required to be bounded, which is significantly for the presented theorems. Indeed, unboundedness U implies unboundedness the set S (see (11)) which make it impossible toobtain inequality (22) uniformly on this set. Thus, these theorems are not applicable to any unbounded U and, in particular, U = R m .
[Table]  Experiments We consider four examples of optimal control problems (6)–(8) described in Table [Table]
where σ 1 ( x ) = − x 21 − x 22 ,
[Equation]
| [Equation] [Equation]
Van der
[Table]
is a famous model of a non-conservative oscillator with non-linear damping.
The aim of the control is to stabilize the oscillator at the terminal time.
Pendulum is a traditional problem for testing control algorithms.
The aim of the control is thestabilization of the pendulum in the top position at the terminal time.
Dubins car is a quite famous model which describes a motion of the point particle moving at aconstant speed on the plane.
The problem is to find a control providing the closeness of the motionwith a target point at the terminal time.
A target problem is an optimal control problem presented in Munos [2006].
The dynamic systemdescribes a hand holding a spring to which is attached a mass.
It is required to control the hand suchthat the mass achieve the target point at the terminal time.
5.1 Bounded NAF
First, we modify NAF algorithm, proposed in Gu et al.
[2016], based on the function family Q .
Note that, the considered examples have U =
[ α, β ] m .
Denote clip( ν ) =
max { α, min { β, ν }} ,ν ∈ R m , α, β ∈ R m .
Then, according to (18) and (19), we can use the following approximation ofthe Q-function, within NAF algorithm:
[Equation]
where V ( t, x | θ V ) , ˜ µ ( t, x | θ ˜ µ ) , and P ( t, x | θ P ) are neural networks with parameters θ V , θ ˜ µ , and θ P ,respectively; P ( t, x | θ P ) is a positive-definite square matrix for each ( t, x ) and θ P ; θ
A = { θ ˜ µ , θ P }and θ Q = { θ A , θ V } .
To be short, we call this algorithm Bounded NAF (BNAF), because it is essentialfor our modification of NAF that the set U is bounded.
5.2 Reward-based BNAF
In practice in many cases, the function r ( t, x ) is known. For example, if a researcher who formalizesa control problem has the right to determine an appropriate cost functional. Then, based on thesecond equality in (21), the function r ( t, x )∆ t k can be used instead of the neural network P ( t, x | θ P )to reduce the number of learning parameters.
This variant of BNAF is called Reward-based BNAF(RB-BNAF).
5.3 Gradient-based BNAF
One can also imagine the situation in which, taking into account mechanical laws and experimentalmeasurements, the function g ( t, x ) also turns out to be known for the researcher. Then, according tothe third equality in (21), the function
[Equation]
can be used instead of the neural network ˜ µ ( t, x | θ ˜ µ ) , which also reduces the number of learningparameters.
This variant of BNAF is called Gradient-based BNAF (GB-BNAF).
5.4 Experimental results
We use the same learning parameters of every our tasks.
We apply neural networks with two layersof 256 and 128 rectified linear units (ReLU) and learn their used ADAM with the learning ratelr = 1 e − 3 .
We use batch size n bs = 128 and smoothing parameter τ = 1 e − 2 .
Also we take∆ t = 0 .
All calculations were performed on a personal computer in a standard way.
We compare NAF, BNAF, RB-BNAF, GB-BNAF algorithms, and DDPG algorithm presented in[Lillicrap et al., 2016].
We utilize the hyperbolic tangent on the output layer of the neural network corresponding to the policy µ ( s | θ µ ) in NAF and DDPG algorithms in order to provide inclusion(4).
Figure 1 shows learning curves of the algorithms for the considered examples.
One can notethat RB-BNAF algorithm has a good performance in all examples.
GB-BNAF is slightly inferior to RB-BNAF everywhere except A target problem.
DDPG and BNAF have excellent results only oneexample out of four.
NAF algorithm shows not the worst performance only in Pendulum example.
Thus, taking the presented experiments into account, we can give the following general recommendations.
For solving optimal control problems (6)–(8), it is rational to use GB-BNAF and especially
RB-BNAF algorithms along with NAF and DDPG algorithms.
There is a high probability that theywill show better results and more stable learning.
Let us also emphasize that the presented algorithms can be successfully applied to problems withoutthe dependence of the cost functional (6) on u ( t ) .
In this case, the quadratic term with small r ( t, x ( t ))can be added in the initial problem.
Then, on the one hand, smallness of this term does not result inpoor performance of the algorithms (see pendulum and a target problem examples) and on the otherhand the obtained results are close to the solution of the initial problem.
Related works
Many different ways to apply reinforcement learning approaches for solving optimal control problemsare investigated [Baird, 1994, Doya, 1995, 2000, Munos, 2006, Tallec et al., 2019, Kim and Yang,2020, Lutter et al., 2021].
Among them, a time-discretization is perhaps the most obvious and widelyused tool [Lillicrap et al., 2016, Gu et al., 2016, Haarnoja et al., 2017, Gu et al., 2017, Kalashnikovet al., 2018].
From the theoretical point of view, it is known [Bardi and Dolcetta, 1997, p.388] thatsolutions of time-discrete optimal control problems converge to the solution of the initial problemas the discretization step tends to zero.
In the present paper, we also use the time-discretization andstudy approximating solutions of Bellman equations and the corresponding greedy policies dependingon the discretization step (see Theorems 1 and 2).
Other studies of dependencies on the discretizationstep of reinforcement learning methods can be found in Munos [2006], Tallec et al.
We focus on the idea from Gu et al.
[2016] to expand Q-learning algorithm to optimal controland reinforcement learning problems with continuous actions.
Other approaches for solving suchproblems are investigated in Lillicrap et al.
[2016], Haarnoja et al.
[2017], Kalashnikov et al.
Lim et al.
[2019], Ryu et al.
[2020], Lutter et al.
The paper Lutter et al.
[2021] seems theclosest to the presented paper.
In Lutter et al.
[2021], the similar optimal control problem and feedbackcontrol policy are considered, but the proposed algorithms require knowledge of the right-hand sideof differential equations in contrast to algorithms suggested in presented paper.
Let us also note the family of functions (18), (19) is included, in some sense, to families of Q-functionsconsidered in Wang et al.
[2016], Tallec et al.
It seems to be expected because more generalclasses of problems are considered in these papers.
Nevertheless, the proposed algorithms and resultsare very different from presented in this paper.  Conclusion
We consider reinforcement learning problems which are obtained by time-discretization of finitehorizon optimal control problems described by affine in control ordinary differential equations andquadratic in control cost functionals.
Based on the idea of NAF algorithm Gu et al. [2016], wepresent a new family of quadratic functions and prove that, first, this family is sufficiently rich toapproximately solve the Bellman optimality equation (Theorem 1), and second, any sufficientlyaccurate solution, from this family, of the Bellman optimality equation allows to approximately obtainthe optimal policy (Theorem 2). Moreover, we establish that it is impossible to get the same resultsfor the function family from Gu et al. [2016] (Theorem 3).
Next, we utilize the obtained theoreticalresults to propose new algorithms improving NAF
The experimental results confirm their efficiency
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] See Abstract
(b) Did you describe the limitations of your work?
[Yes] See Introduction
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See SectionProblem statement
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] See Supple-
mentary material
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Section Experiments
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See Section Experiments
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section Experiments
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]