Continuous Deep Q-Learning in Optimal Control Problems: Normalized Advantage Functions Analysis
Abstract
One of the most effective continuous deep reinforcement learning algorithms is normalized advantage functions (NAF).
The main idea of NAF consists in the approximation of the Q-function by functions quadratic with respect to the action variable.
This idea allows to apply the algorithm to continuous reinforcement learning problems, but on the other hand, it brings up the question of classes of problems in which this approximation is acceptable.
The presented paper describes one such class.
We consider reinforcement learning problems obtained by the time-discretization of certain optimal control problems.
Based on the idea of NAF, we present a new family of quadratic functions and prove its suitable approximation properties.
Taking these properties into account, we provide several ways to improve NAF.
The experimental results confirm the efficiency of our improvements.
Introduction
The standard reinforcement learning (RL) [Sutton and Barto, 2018] setup consists of an agent interacting with an environment.
At each step of the interaction, the agent determines an action based on its policy and its current state, gets a reward, and makes a transition to the next state.
The aim of the agent is to learn the policy that maximizes the sum of rewards.
One of the most widespread approach for solving RL problems is Q-learning [Watkins and Dayan, 1992], which consist in seeking the optimal action-value function (Q-function) as a solution of the
Bellman optimality equation.
After the learning, the agent can act optimally by the found Q-function.
Initially, Q-learning approach was applied for solving RL problems with finite state and action spaces.
In this case, Q-function can be represented by a finite table.
For the case of a large or continuous state space, Q-learning has later been extended to Deep Q-learning algorithm
[Mnih et al., 2015] which allows to seek the approximations of Q-function in the class of neural networks by means of the stochastic gradient descent.
Deep Q-learning and its modifications have shown efficiency for a range of challenging tasks [Wang et al., 2016, van Hasselt et al., 2016, Schaul et al., 2016, Hessel et al., 2018].
However, this algorithm can not be directly applied for solving RL problems with continuous action spaces.
The reason is that Deep Q-learning involves maximizing of approximate Q-function by the action variable on each step of the learning, which is a complex procedure for continuous action spaces.
Among various approaches to overcome this problem [Lillicrap et al., 2016, Haarnoja et al., 2017, Wu et al., 2017, Kalashnikov et al., 2018, Nachum et al., 2018, Fujimoto et al., 2018, Lim et al., 2019, Ryu et al., 2020, Lutter et al., 2021], we focus on the idea of the normalized advantage functions (NAF) algorithm
[Gu et al., 2016], which consists in the approximation of Q-function by the function quadratic with respect to the action variable.
It allows to calculate the maximum quite fast and precisely and solve some challenging control problems [Gu et al., 2017, Dong et al., 2018,
Ikemoto and Ushio, 2021], but on the other hand, it brings up the question of classes of RL problems in which this approximation is acceptable.
The presented paper describes one of the possible answers to this question.
Note that the class of LQR problems [Bradtke et al., 1994] has Q-functions quadratic with respect to the action variable.
However, this class, being quite special, is not suitable for the description of non-linear controlled processes.
In this paper, we study a wider (in some sense) class of RL problems.
We consider RL problems which are obtained by time-discretization of finite-horizon optimal control problems [Bardi and Dolcetta, 1997] in dynamical system is described by affine in control ordinary differential equations and the cost functional is quadratic in control.
Note that a lot of RL problems with continuous action spaces arise from control problems for mechanical or robotic systems [Lillicrap et al., 2016, Gu et al., 2016, Haarnoja et al., 2017, Gu et al., 2017, Kalashnikov et al., 2018] which are, in fact, described by affine in control ordinary differential equations.
Besides, often, the cost functional in a control problem can be determined as quadratic in control.
For instance, if the quality index in an initial problem does not depend on control, then one can add a small quadratic term such that, on the one hand, the quality index becomes quadratic, and on the other hand, the sense of the problem does not changed much.
Thus, the class of control problems under consideration seems quite general.
For this class, based on the idea of NAF, we present a new family of quadratic functions and prove that, first, this family is sufficiently rich to approximately solve the Bellman optimality equation (Theorem 1), and second, any sufficiently accurate solution, from this family, of the Bellman optimality equation allows to approximately obtain the optimal policy in the corresponding optimal control problem (Theorem 2).
Moreover, we prove that it is impossible to get the same results for the original family of functions from Gu et al.
[2016] (Theorem 3).
Next, we provide several ways to utilize the obtained theoretical knowledge in order to improve NAF.
The experimental results confirm the efficiency of our improvements.
Note that, in contrast to LQR problems, we can not say that Q-function is quadratic in our class of problems.
Moreover, it is not true in general case.
Nonetheless, according to presented statements (Theorem 1,2), Q-function can be approximated by quadratic functions with sufficient order of accuracy to obtain near-optimal policy.
Background
The standard reinforcement learning (RL) setup consists of an agent interacting with an environment [Sutton and Barto, 2018].
This interaction is described by a Markov Decision Process (MDP), which is a tuple ( S , U , P , R , ρ 0 , γ ) , where S is a state space, U is an action space, P ( s ′ | s, u ) is a transition distribution, R ( s, u ) is a reward function, ρ 0 ( s ) is an initial state distribution, and γ ∈ [0 , 1] is a discount factor.
The aim of the agent is to learn optimal policy µ ∗ ( s ) which maximizes the value
J ( µ ) =
E (cid:20) ∞ (cid:88)
[Equation]
In the general statement of reinforcement learning problems, a policy of the agent can be stochastic, however, within this paper, we assume that the policy is deterministic.
One of the most effective approach for solving RL problems is Q-learning [Watkins and Dayan, 1992], which consist in seeking the optimal action-value function
[Equation]
as a solution of the Bellman optimality equation
[Equation]
In other words
[Equation]
the agent solves the following minimization problem:
[Equation]
If the agent knows the function Q ∗ ( s, a ) , it can act optimally by the greedy policy
[Equation]
Initially, the Q-learning approach was applied for solving RL problems with finite state and action spaces.
In this case, the Q-function is represented by a finite table and problem (1) is finitedimensional.
For the case of a large or continuous state space, Q-learning has later been extended to
Deep Q-learning algorithm
[Mnih et al., 2015] which allows to seek approximations of Q-function in the class of neural networks Q ( x, u | θ Q ) , where θ Q is the parameter vector of the neural network.
During the learning, the experiences ( s i , u i , r i , s i +1 ) are stored in the buffer D and simultaneously the parameter vector θ Q is updated by means of the stochastic gradient descent minimizing the loss function
[Equation]
where U ( D ) is the uniform distribution on D .
Deep Q-learning and its modifications are effective for a range of challenging tasks [Wang et al., 2016, van Hasselt et al., 2016, Schaul et al., 2016, Hessel et al., 2018].
However, note that this algorithm can not be directly applied for solving RL problems with continuous action spaces.
The reason is that Deep Q-learning involves the maximizing in (2) on each step of the learning, which is a complex procedure for continuous U .
Among various approaches to overcome this problem [Lillicrap et al., 2016, Haarnoja et al., 2017, Kalashnikov et al., 2018, Lim et al., 2019, Ryu et al., 2020, Lutter et al., 2021], we focus on an idea of the normalized advantage functions (NAF) algorithm
[Gu et al., 2016].
This idea consists in the approximation of Q-function by the following quadratic with respect to u functions:
[Equation]
where V ( s | θ V ) , µ ( s | θ µ ) , and P ( s | θ P ) are neural networks with parameters θ V , θ µ , and θ P , respectively; P ( s | θ P ) is a positive-definite square matrix for each s and θ P ; θ
A = { θ µ , θ P } and θ Q = { θ A , θ V } .
Under the condition
[Equation]
it allows to get the maximum and argmaximum values directly by values of V ( s | θ V ) and µ ( s | θ µ [Equation]
[Equation]
but on the other hand, it brings up the question of classes of RL problems in which quadratic approximations is acceptable.
Below, we describes one such class.
Problem statement
In this section, we consider a certain class of optimal control problems and show that discrete approximations of these problems can be formalized as RL problems.
Consider the following optimal control problem: it is required to maximize the functional
[Equation] [Equation]
over all u ( · ) , where x ( · ) is the solution [Filippov, 1988, §1] of the differential equation [Equation] ( t ) , t
[Equation]
(7) under the initial condition
[Equation]
Here t is the time variable, T > 0 is the terminal instant of time, x
( t ) ∈
R n is the current state vector, u ( t ) ∈
U is the current control action vector forming the measurable function u ( · ) , U ⊂ R m is the nonempty compact set, z ∈ R n is the fixed initial state vector, f ( t, x ) ∈ R n , g ( t, x ) ∈ R n × m , q ( t, x ) ∈ R , r ( t, x ) ∈ R m × m , ( t, x ) ∈ [0 , T ] × R n are continuous with respect to t and continuously differentiable with respect to x functions, r ( t, x ) is the positive-definite matrix for each ( t, x ) ∈ [0 , T ] × R n , and σ ( x ) ∈ R , x ∈ R n is the continuously differentiable function.
We assume that there exists a constant c fg > 0 such that
[Equation]
Note that, under these conditions, for each function u ( · ) , there exists a unique solution x ( · ) of equation (7) under the initial condition (8) [Filippov, 1988, §1].
Define the value function in optimal control problem (6), (7) by
[Equation]
where, for each u ( · ) , x ( · ) is the solution of equation (7) on the interval [ t ∗ , T ] under the initial condition x ( t ∗ )
= x ∗ .
Define the sets
[Equation]
Let k ∈ N , ∆ t k = T/k , and t i = i ∆ t k , i ∈ 0 , k .
Consider the corresponding discrete optimal control problem: it is required to maximize the function
[Equation]
over all u i ∈ U , i ∈ 0 , k − 1 , where ( x 0 , x 1 , . . .
, x k ) is defined by
[Equation]
Let us show that problem (12), (13) can be formalized as the RL problem.
First, we define the state and actions spaces, the initial state distribution, and the discount factor as follows:
[Equation]
Here s T is some fictional terminal state, δ is Dirac delta distribution.
Next, for every i ∈ 0 , k − 1 , x ∈ S ( t i ) , and u ∈ U , we define the transition distribution and the reward function by
P ( [Equation] |
[Equation] i , x [Equation] , u
[Equation] = δ
[Equation]
where x ′ = x
+ ( f ( t i , x ) + g
( t i , x ) u )∆ t k .
Taking into account (9) and (11), one can prove the inclusion ( t i +1 , x ′ ) ∈ S .
Hence, the transition distribution P is well-defined.
For i =
k , we set
P ( s ′ | s =
( t k
[Equation]
In order to make dynamical processes (13) formally infinite
[Equation]
we put
[Equation]
Thus
[Equation]
we define MDP which describes the RL problem corresponding to problem (12), (13).
Next, we show that such RL problems is suitable for using quadratic approximations of the Q-function
[Equation]
Quadratic approximations of the Q-function
Denote by Q the family of functions Q such that
[Equation]
where
[Equation]
Here V ( t, x ) , ˜ µ ( t, x ) , and P ( t, x ) are continuous functions; P ( t, x ) is a positive-definite square matrix for each ( t, x ) ∈ [0 , T )
× R n .
Note that, functions Q from the family Q satisfy the equalities
[Equation]
as well as (see (5))
quadratic functions from family (3).
However, these function families are different.
The difference is that we do not assume the inclusion ˜ µ ( t, x ) ∈ U as opposed to assumption (4).
In other words, the vertex of a quadratic form (3) belongs to the set U , while this is not required in the function family Q .
Thus, Q is a wider family of quadratic functions.
The theorems below establish a connection between the optimal control problem (6)–(8) and minimization problems (1) for MDP (14)–(17).
Theorem 1.
Let the value function V ∗ ( t, x ) be continuously differentiable.
Then, for every ε > 0 , there exists k ∗ > 0 such that, for every k ≥ k ∗ , the function Q ∈ Q defined by (18) and (19) where
V ( t, x ) =
V ∗ ( t, x ) ,
[Equation]
[Equation]
satisfies the inequality
[Equation]
where we assume Q ( t k , x ′ , u ′ )
=
σ ( x ′ ) .
Theorem 2.
Let the value function V ∗ ( t, x ) be continuously differentiable.
Let ε > 0 and k ∗ > 0 be defined according to Theorem 1.
Take k ≥ k ∗ and suppose that a function Q ∈ Q satisfies inequality (22).
Then the following estimate holds:
[Equation]
where the function J k ( u 0 , u 1 , . . .
u k − 1 ) is defined by (12) with u i = µ ( t i , x i ) , i ∈ 0 , k − 1 and the function µ ( t, x ) is defined by Q according to (18) and (19).
Thus, Theorem 1 shows that the function family Q is sufficiently rich to contain approximate solutions of problem (1) with a predetermined accuracy and Theorem 2 establishes that all such approximate solutions contained in Q allow us to get the policy, which approximately provides the optimal result in optimal control problem (6)–(8).
Now, let us consider the original family of functions from Gu et al.
Denote by Q NAF the family of functions Q such that
[Equation]
where V ( t, x ) , µ ( t, x ) , and P ( t, x ) are continuous functions; P ( t, x ) is a positive-definite square matrix for each ( t, x ) ∈ [0 , T ) × R n ; µ ( t, x ) ∈ U for each ( t, x ) ∈ [0 , T )
× R n .
The theorem below establishes that if we take the family Q NAF instant of Q , then Theorem 1 can not be proved even in the simplest cases of optimal control problem (6)–(8).
Theorem 3.
Let n = m = 1 , T = 1 , U =
[ − 1 , 1] , f ( t, x ) =
q ( t, x ) = 0 , g ( t, x ) =
r ( t, x ) = 1 , σ ( x ) = − x 2 , and z = 2 .
Then, for every k ≥ 4 and Q ∈ Q NAF , there exist i ∈ 0 , k − 1 , x ∈ S ( t i ) , and u ∈ U such that
[Equation]
where we assume Q ( t k , x ′ , u ′ )
= − ( x ′ ) 2 .
Proofs of the theorems are given in the Appendix A.
Note that, in the particular case U =
R m , the family Q coincides with family Q NAF .
However, within this paper, U is required to be bounded, which is significantly for the presented theorems.
Indeed, unboundedness U implies unboundedness the set S (see (11)) which make it impossible to obtain inequality (22) uniformly on this set.
Thus, these theorems are not applicable to any unbounded
U and, in particular, U = R m .
[Table]
Experiments
We consider four examples of optimal control problems (6)–(8) described in Table
[Table]
where σ 1 ( x ) =
[Equation]
x 21 − x 22 ,
[Equation] [Equation]
Van der
[Table]
is a famous model of a non-conservative oscillator with non-linear damping.
The aim of the control is to stabilize the oscillator at the terminal time.
Pendulum is a traditional problem for testing control algorithms.
The aim of the control is the stabilization of the pendulum in the top position at the terminal time.
Dubins car is a quite famous model which describes a motion of the point particle moving at a constant speed on the plane.
The problem is to find a control providing the closeness of the motion with a target point at the terminal time.
A target problem is an optimal control problem presented in Munos [2006].
The dynamic system describes a hand holding a spring to which is attached a mass.
It is required to control the hand such that the mass achieve the target point at the terminal time.
5.1 Bounded NAF
First, we modify NAF algorithm, proposed in Gu et al.
[2016], based on the function family Q .
Note that, the considered examples have U =
[ α, β ] m .
Denote clip( ν ) =
max { α, min { β, ν }} , ν ∈ R m , α, β ∈ R m .
Then, according to (18) and (19), we can use the following approximation of the Q-function, within NAF algorithm:
[Equation]
where V ( t, x | θ V ) , ˜ µ ( t, x | θ ˜ µ ) , and P ( t, x | θ P ) are neural networks with parameters θ V , θ ˜ µ , and θ P , respectively; P ( t, x | θ P ) is a positive-definite square matrix for each ( t, x ) and θ P ; θ
A = { θ ˜ µ , θ P } and θ Q = { θ A , θ V } .
To be short, we call this algorithm Bounded NAF (BNAF), because it is essential for our modification of NAF that the set U is bounded.
5.2 Reward-based BNAF
In practice in many cases, the function r ( t, x ) is known.
For example, if a researcher who formalizes a control problem has the right to determine an appropriate cost functional.
Then, based on the second equality in (21), the function r ( t, x )∆ t k can be used instead of the neural network P ( t, x | θ P ) to reduce the number of learning parameters.
This variant of BNAF is called Reward-based BNAF (RB-BNAF).
5.3 Gradient-based BNAF
One can also imagine the situation in which, taking into account mechanical laws and experimental measurements, the function g ( t, x ) also turns out to be known for the researcher.
Then, according to the third equality in (21), the function
[Equation]
can be used instead of the neural network ˜ µ ( t, x | θ ˜ µ ) , which also reduces the number of learning parameters.
This variant of BNAF is called Gradient-based BNAF (GB-BNAF).
5.4 Experimental results
We use the same learning parameters of every our tasks.
We apply neural networks with two layers of 256 and 128 rectified linear units (ReLU) and learn their used ADAM with the learning rate lr = 1 e − 3 .
We use batch size n bs = 128 and smoothing parameter τ = 1 e − 2 .
Also we take ∆ t = 0 .
All calculations were performed on a personal computer in a standard way.
We compare NAF, BNAF, RB-BNAF, GB-BNAF algorithms, and DDPG algorithm presented in [Lillicrap et al., 2016].
We utilize the hyperbolic tangent on the output layer of the neural network corresponding to the policy µ ( s | θ µ ) in NAF and DDPG algorithms in order to provide inclusion (4).
Figure 1 shows learning curves of the algorithms for the considered examples.
One can note that RB-BNAF algorithm has a good performance in all examples.
GB-BNAF is slightly inferior to
RB-BNAF everywhere except A target problem.
DDPG and BNAF have excellent results only one example out of four.
NAF algorithm shows not the worst performance only in Pendulum example.
Thus, taking the presented experiments into account, we can give the following general recommendations.
For solving optimal control problems (6)–(8), it is rational to use GB-BNAF and especially
RB-BNAF algorithms along with NAF and DDPG algorithms.
There is a high probability that they will show better results and more stable learning.
Let us also emphasize that the presented algorithms can be successfully applied to problems without the dependence of the cost functional (6) on u ( t ) .
In this case, the quadratic term with small r ( t, x ( t )) can be added in the initial problem.
Then, on the one hand, smallness of this term does not result in poor performance of the algorithms (see pendulum and a target problem examples) and on the other hand the obtained results are close to the solution of the initial problem.
Related works
Many different ways to apply reinforcement learning approaches for solving optimal control problems are investigated [Baird, 1994, Doya, 1995, 2000, Munos, 2006, Tallec et al., 2019, Kim and Yang, 2020, Lutter et al., 2021].
Among them, a time-discretization is perhaps the most obvious and widely used tool [Lillicrap et al., 2016, Gu et al., 2016, Haarnoja et al., 2017, Gu et al., 2017, Kalashnikov et al., 2018].
From the theoretical point of view, it is known [Bardi and Dolcetta, 1997, p.388] that solutions of time-discrete optimal control problems converge to the solution of the initial problem as the discretization step tends to zero.
In the present paper, we also use the time-discretization and study approximating solutions of Bellman equations and the corresponding greedy policies depending on the discretization step (see Theorems 1 and 2).
Other studies of dependencies on the discretization step of reinforcement learning methods can be found in Munos [2006], Tallec et al.
We focus on the idea from Gu et al.
[2016] to expand Q-learning algorithm to optimal control and reinforcement learning problems with continuous actions.
Other approaches for solving such problems are investigated in Lillicrap et al.
[2016], Haarnoja et al.
[2017], Kalashnikov et al.
Lim et al.
[2019], Ryu et al.
[2020], Lutter et al.
The paper Lutter et al.
[2021] seems the closest to the presented paper.
In Lutter et al.
[2021], the similar optimal control problem and feedback control policy are considered, but the proposed algorithms require knowledge of the right-hand side of differential equations in contrast to algorithms suggested in presented paper.
Let us also note the family of functions (18), (19) is included, in some sense, to families of Q-functions considered in Wang et al.
[2016], Tallec et al.
It seems to be expected because more general classes of problems are considered in these papers.
Nevertheless, the proposed algorithms and results are very different from presented in this paper.
Conclusion
We consider reinforcement learning problems which are obtained by time-discretization of finitehorizon optimal control problems described by affine in control ordinary differential equations and quadratic in control cost functionals.
Based on the idea of NAF algorithm Gu et al.
[2016], we present a new family of quadratic functions and prove that, first, this family is sufficiently rich to approximately solve the Bellman optimality equation (Theorem 1), and second, any sufficiently accurate solution, from this family, of the Bellman optimality equation allows to approximately obtain the optimal policy (Theorem 2).
Moreover, we establish that it is impossible to get the same results for the function family from Gu et al.
[2016] (Theorem 3).
Next, we utilize the obtained theoretical results to propose new algorithms improving NAF.
The experimental results confirm their efficiency.
Acknowledgments and Disclosure of Funding
The work was performed as part of research conducted in the Ural Mathematical Center with the financial support of the Ministry of Science and Higher Education of the Russian Federation (Agreement number 075-02-2022-874).
A Appendix
Denote
[Equation]
Lemma 1.
Let the value function V ∗ ( t, x ) be continuously differentiable.
Then, the following
Hamilton-Jacobi-Bellman equation holds:
[Equation] The proof can be found, for example, in J. Yong. Differential Games: A Concise Introduction.
World Scientific, University of Central Florida, 238 USA, 2015. doi: 10.1142/9121.
Proof of Theorem 1.
Note that, due to (19), (21), and (25), we have [Equation] for any ( t, x, u ) ∈ [0 , T ] × R n × U .
Due to (9) and (11), we have [Equation] [Equation]
Let ε > 0 . Since V ∗ ( t, x ) is continuously differentiable, there exists δ > 0 such that
[Equation]
α [Equation] 28) for any ( t, x ) , ( t ′ , x ′ ) ∈ S satisfying | t − t ′
| + ∥ x − x ′ ∥ ≤ δ .
Put k ∗ = (1 + α fg ) T/δ . Let k ≥ k ∗ , i ∈ 0 , k − 1 , x ∈ S ( t i ) , u ∈ U , and x ′ = x
+ ( f ( t i , x ) + g ( t i , x ) u )∆ t k . Due to (9) and (11), we have ( t i +1 , x ′ ) ∈ S . Then, from (27) and (28), we derive
[Equation] Next, according to (18), (20), (25), (26), (29), and Lemma 1, we obtain
[Equation] The theorem is proved.
Proof of Theorem 2. Let ε > 0 . Define k ∗ according to Theorem 1. Let k ≥ k ∗ .
Let Q ∈ Q satisfy (22). Let x i , i ∈ 0 , k be defined by (13), where u i ∈ µ ( t i , x i ) . Then, due to (12), (20), and (22), we have [Equation]
Let us define Q ∗ ∈ Q according to (21). Let x ∗ i , i ∈ 0 , k be defined by (13), where u ∗ i ∈ argmax u ∈ U Q ∗ ( t i , x i , u ) .
Then, due to the Theorem 1 and (20), we have [Equation] [Equation] Note that, according to definition (10) of the value function V ∗ , we have V ∗ ( t k , x ) = σ ( x ) , x ∈ R n .
Then, from (20), (22), and (31), we derive
[Equation]
[Equation]
Due to this inequality and inequality (30), we obtain J k ( u 0 , . . .
, u k − 1 ) ≥ V ∗ (0 , z ) − 3 Tε .
Taking into account (6) and (10) we have sup u ( · ) J ( u ( · ))
= V ∗ (0 , z ) .
Thus, the theorem is proved.
Proof of Theorem 3. Let k ≥ 4 and Q ∈ Q NAF .
Let us take i = k − 1 and x = 2 . For the sake of brevity, denote V = V ( t k − 1 , 2) , µ = µ ( t k − 1 , 2) , and P = P ( t k − 1 , 2) . Then, in order to prove the theorem, we have to show that [Equation]
Arguing by contradiction assume that
[Equation]
Then, for u = − 1 , u = 1 , and u = 0 , we have
[Equation]
Adding up inequalities (32), (33), and twice inequity (34), we derive
[Equation]
Adding up twice inequalities (32), (34), and inequity (35), we obtain
[Equation]
From this estimate, taking into account (35) and the inequality k ≥ 4 , we conclude
[Equation]
This inequality contradicts the inclusion µ ∈ U =
[ − 1 , 1] , which should be valid for Q ∈ Q NAF .