R EINFORCEMENT L EARNING FOR C ONTROL WITH P ROBABILISTIC S TABILITY G UARANTEE
Anonymous authors
Paper under double-blind review
A BSTRACT
Reinforcement learning is promising to control dynamical systems for which the traditional control methods are hardly applicable.
However, in control theory, the stability of a closed-loop system can be hardly guaranteed using the policy/controller learned solely from samples.
In this paper, we will combine
Lyapunov’s method in control theory and stochastic analysis to analyze the mean square stability of MDP in a model-free manner.
Furthermore, the ﬁnite sample bounds on the probability of stability are derived as a function of the number M and length T of the sampled trajectories.
And we show that there is a lower bound on T and the probability is much more demanding for M than
T. Based on the theoretical results, a REINFORCE-like algorithm is proposed to learn the controller and the
Lyapunov function simultaneously.
I NTRODUCTION
Reinforcement learning (RL) has achieved superior performance on some complicated control tasks (Kumar et al., 2016;
Xie et al., 2019; Hwangbo et al., 2019) for which the traditional control engineering methods can be hardly applicable ( ˚Astr ¨ om and Wittenmark, 1973; Morari and Zaﬁriou, 1989; Slotine et al., 1991).
The dynamical system to be controlled is often highly stochastic and nonlinear which is typically modeled by Markov decision process (MDP), i.e.,
[Equation]
where s ∈ S ⊂ R n denotes the state, a ∈ A ⊂ R m denotes the action and P ( s t +1 | s t , a t ) is the transition probability function.
An optimal controller can be learned from samples through “trial and error” by memorizing what has been experienced (Kaelbling et al., 1996; Bertsekas, 2019).
However, there is a major caveat that prevents the real-world application of learning methods for control engineering applications.
Without using a mathematical model, the current sample-based
RL methods cannot guarantee the stability of the closed-loop system, which is the most important property of any control system as in control theory.
The most useful and general approach for studying the stability of a dynamical system is Lyapunov’s method Lyapunov (1892), which is dominant in control engineering Jiang and Jiang (2012); Lewis et al.
; Boukas and Liu (2000).
In Lyapunov’s method, a suitable “energy-like” Lyapunov function L ( s ) is selected and its derivative along the system trajectories is ensured to be negative semi-deﬁnite, i.e., L ( s t +1 )
− L (
s t ) < 0 for all time instants and states, so that the state goes in the direction of decreasing the value of Lyapunov
function
and eventually
converges
to the origin or a sub-level set of the Lyapunov function.
In the traditional control engineering methods, a mathematical model must be given, i.e., the transition probability function in (1) is known.
Thus the stability can be analyzed without the need to assess all possible trajectories.
However, in learning methods, as the dynamic model is unknown, the “energy decreasing” condition has to be veriﬁed by trying out all possible consecutive data pairs in the state space, i.e., to verify inﬁnite inequalities L ( s t +1 ) − L ( s t ) < 0 .
Obviously, the “inﬁnity” requirement makes it impractical to directly exploit Lyapunov’s method in a model-free framework.
In this paper, we show that the mean square stability of the system can be analyzed based on a ﬁnite number of samples without knowing the model of the system.
The contributions of this paper are summarized as follows:
Instead of verifying an inﬁnite number of inequalities over the state space, it is possible to analyze the stability through a sampling-based method where only one inequality is needed.
Instead of using inﬁnite sample pairs { s t +1 , s t } , a ﬁnite-sample stability theorem is proposed to provide a probabilistic stability guarantee for the system, and the probability is an increasing function of the number M and length T of sampled trajectories and converging to 1 as M and T grow.
As an independent interest, we also derive the policy gradient theorem for learning stabilizing policy with sample pairs and the corresponding algorithm.
We further reveal that the classic
REINFORCE algorithm
(Williams, 1992) is a special case of the proposed algorithm for the stabilization problem.
We also conclude two takeaways for the paper:
• Samples of a ﬁnite number M and length T of trajectories can be used for stability analysis with a certain probability.
The probability is monotonically converging to 1 when M and T grow.
• There is a lower bound on T and the probability is much more demanding for M than T .
• The REINFORCE like algorithm can learn the controller and Lyapunov function simultaneously.
The paper is organized as follows:
In Section 2, related works are introduced.
In Section 3, the deﬁnition of mean-square stability (MSS) and the problem statement is given.
In Section 4, the sample-based MSS theorem is proposed.
In Section 5, we propose the probabilistic stability guarantee when only a ﬁnite number of samples are accessible and the probabilistic bound in a relation to the number and length of sampled trajectories is derived.
In Section 6, based on the stability theorems, the policy gradient is derived and a model-free RL algorithm is given.
Finally, a simulated Cartpole stabilization task is considered to demonstrate the effectiveness of the proposed method.
R ELATED W ORKS
Lyapunov’s Method
As a basic tool in control theory, the construction/learning of the Lyapunov function is not trivial and many works are devoted to this problem (Noroozi et al., 2008; Prokhorov, 1994; Serpen, 2005; Prokhorov and Feldkamp, 1999).
In Perkins and Barto (2002), the RL agent controls the switch between designed controllers using Lyapunov domain knowledge so that any policy is safe and reliable.
Petridis and Petridis (2006) proposes a straightforward approach to construct the Lyapunov functions for nonlinear systems using neural networks.
Richards et al.
(2018) proposes a learning-based approach for constructing Lyapunov neural networks with the maximized region of attraction.
However, these approaches require the model of the system dynamics explicitly.
Stability analysis in a model-free manner has not been addressed.
In Berkenkamp et al.
(2017), local stability is analyzed by validating the “energy decreasing” condition on discretized points in the subset of state space with the help of a learned model, meaning that only a ﬁnite number of inequalities need to be checked.
This approach is further extended by using a Noise Contrastive
Prior Bayesian RNN in Gallieri et al.
Nevertheless, the discretization technique may become infeasible as the dimension and space of interest increases, limiting its application to rather simple and low-dimensional systems.
Reinforcement Learning In model-free reinforcement learning (RL), stability is rarely addressed due to the formidable challenge of analyzing and designing the closed-loop system dynamics by solely using samples Bus¸oniu
et al.
(2018), and the associated stability theory in model-free RL remains as an open problem Bus¸oniu
et al.
Gorges (2017).
Recently, Lyapunov analysis is used in model-free RL to solve control problems with safety constraints Chow et al.
In Chow et al.
(2018), a Lyapunov-based approach for solving constrained Markov decision processes is proposed with a novel way of constructing the Lyapunov function through linear programming.
In Chow et al.
(2019), the above results were further generalized to continuous control tasks.
It should be noted that even though Lyapunov-based methods were adopted in these results, neither of them addressed the stability of the system.
In Postoyan et al.
(2017), an initial result is proposed for the stability analysis of deterministic nonlinear systems with optimal controller for inﬁnite-horizon discounted cost, based on the assumption that discount is sufﬁciently close to 1 .
However, in practice, it is rather difﬁcult to guarantee the optimality of the learned policy unless certain assumptions on the system dynamics are made Murray et al.
; Abu-Khalaf and Lewis (2005); Jiang and Jiang (2015).
Furthermore, the exploitation of multi-layer neural networks as function approximations
Mnih et al.
; Lillicrap et al.
(2015) only adds to the impracticality of this requirement.
Given certain information on the model, Adaptive dynamic programming (ADP) can guarantee convergence to the optimal solution, and thus stability is naturally ensured Balakrishnan et al.
For nonlinear systems with input-afﬁne structure, model-free ADP algorithms can guarantee the stability of the closed-loop system Murray et al.
; Abu-Khalaf and Lewis (2005); Shih et al.
; Jiang and Jiang (2015); Deptula et al.
This paper steps beyond the scope of controlafﬁne systems and are devoted to learning a controller with a stability guarantee for the general stochastic nonlinear system.
To the best of the author’s knowledge, the ﬁnite sample-based approach for the stability analysis of stochastic nonlinear systems considered in this paper is still missing.
For the model-based approaches, promising results on stability analysis are reported but generally based on certain model assumptions.
Model predictive control (MPC) has long been studying the issue of optimal control of various dynamical systems without violating state and action constraints, and Lyapunov stability is naturally guaranteed Mayne and Michalska (1990); Michalska and Mayne (1993); Mayne et al.
Favorable as it may seem, the nice properties above are built upon the accurate and concise modeling of the dynamics, which narrows its scope to certain ﬁelds.
In Ostafew et al.
(2014), a learning-based nonlinear MPC algorithm is proposed to learn the disturbance model online and improve the tracking performance of ﬁeld robots, but ﬁrst, a priori model is required.
Aswani et al.
(2013) proposed a new learning-based MPC scheme that can provide deterministic guarantees on robustness while performance is improved by identifying a richer model.
However, it is limited to the case that a linear model with known uncertainty bound is available.
Other results concerning learning-based MPC are referred to Aswani et al.
; Bouffard et al.
Di Cairano et al.
In Bobiti (2017); Bobiti and Lazar (2018), a sampling-based approach for stability analysis and domain of attraction estimation is proposed for deterministic nonlinear systems.
The reliability of the estimation is addressed with a probabilistic bound on the number of samples, however, based on the assumption that all the samples are independently distributed.
This infers that given multiple state trajectories, only the ﬁrst-step data are applicable for the stability analysis, which is overwhelmingly inefﬁcient and will be ﬁxed in this paper.
The aforementioned approach is further incorporated with a learned model in Gallieri et al.
for probabilistic stability analysis.
It should be noted that this paper is to address the stability analysis and control of stochastic systems, while the results above are focused on the deterministic nonlinear systems.
P ROBLEM S TATEMENT
Before establishing any stability theorem, the deﬁnition of stability needs to be properly given.
In this paper, we will focus on the mean square stability (MSS) which is commonly known in control theory.
The deﬁnition of MSS is given as follows.
Deﬁnition 1 (Shaikhet, 1997)
The stochastic system is said to be mean square stable (MSS) if there exists a positive constant b such that lim t →∞ E s t (cid:107) s t (cid:107) 22 = 0 holds for any initial condition s 0 ∈ { s 0 |(cid:107) s 0 (cid:107) 22 ≤ b } .
If b is arbitrarily large then the stochastic system is globally mean square stable (GMSS).
MSS basically says that, on average, the state of a system, starting from an initial position in the state space, tends towards the equilibrium as time goes to inﬁnity.
It should be noted that the stability conditions of Markov chains have been reported in (Shaikhet, 1997; Meyn and Tweedie, 2012), however, of which the validation requires verifying inﬁnite inequalities on the state space if S is continuous.
Unfortunately, the ﬁnite sample-based approach for stability analysis where only one inequality needs to be checked is still missing.
For the sample-based approach, the key challenge is the theoretical gap in “ﬁnity” guarantees, i.e., (1) from inﬁnite L ( s t +1 ) − L ( s t ) to only a single inequality related to sample expectation; (2) from inﬁnite samples expectation to ﬁnite samples expectation.
Thus in this paper, two sets of theoretical questions need to be answered.
Q1.
What does a sample-based Lyapunov theorem look like and what are the assumptions and conditions needed to use a single E inﬁnite samples ( L ( s t +1 ) − L ( s t )) instead of the inﬁnite
L ( s t +1 ) − L ( s t ) ?
Q2.
What will be the number of samples needed to guarantee stability for a given probability, if
E inﬁnite samples is changed to E ﬁnite samples ?
What is the analytical form of the probability as a function of the number M and length T of the sampled trajectories?
Before proceeding, some notations are to be deﬁned.
We introduce c ( s ) (cid:44) min( (cid:107) s (cid:107) 22 , c ) , c > 0 to denote the clipped norm of state.
The closed-loop transition probability is denoted as P π ( s (cid:48) | s ) (cid:44) (cid:82)
A π ( a | s ) P ( s (cid:48) | s, a )d a .
We also introduce the closed-loop state distribution at a certain instant t as
P ( s | ρ, π, t ) , which could be deﬁned iteratively: P ( s (cid:48) | ρ, π, t + 1) =
(cid:82) S P π ( s (cid:48) | s ) P ( s | ρ, π, t )
d s, ∀ t ∈
Z [1 , ∞ ) and P ( s | ρ, π, 1) =
ρ ( s ) , where ρ ( s ) is the starting state distribution.
S AMPLE -B ASED L YAPUNOV S TABILITY G UARANTEE
In this section, we will answer Q1 in Section 3 and present the key results on sample-based stability analysis.
We will show that only a single inequality E inﬁnite samples ( L ( s t +1 ) − L ( s t )) ≤ 0 is enough for the veriﬁcation of stability.
First, we make the following assumption which is commonly exploited by many RL literature (Sutton et al., 2009; Korda and La, 2015; Bhandari et al., 2018; Zou et al., 2019).
Assumption 1 The Markov chain induced by policy π is ergodic.
It follows that there exists a unique stationary distribution q π ( s ) =
lim t →∞ P ( s | ρ, π, t ) .
The veriﬁcation of ergodicity is in general an open question in practice.
There are many systems proved to be ergodic in physics, statistic mechanics, economics, e.g. gambling games Peters (2019), the Anosov ﬂow Anosov (2010), and dynamical billiards Park (2014), etc.
The study of ergodicity of various systems and its veriﬁcation composed a major branch of mathematics.
If the transition probability is known for all states, the validation is possible but requires a large source of computation power to enumerate through the state space.
As a matter of fact, the existence of the stationary state distribution is generally assumed to hold in the RL literature Melo et al.
; Levin and Peres (2017); Bhandari et al.
; Zou et al.
In this paper, we focus on analyzing the stability of such systems with a probabilistic bound, as well as developing an algorithm to ﬁnd stabilizing controllers.
In Deﬁnition 1, stability is deﬁned in relation to the set of starting states, which is also called the region of attraction (ROA).
If the MSS system starts within the ROA, its trajectory will be surely attracted to the equilibrium.
To build a sample-based stability guarantee, we need to ensure that the states in ROA are accessible for the stability analysis.
Thus the following assumption is made to ensure that every state in ROA has a chance to be sampled.
Assumption 2 There exists a positive constant b such that ρ ( s ) > 0 , ∀ s ∈ { s | c ( s ) ≤ b } .
Based on the above assumptions, we can exploit Lyapunov’s method to prove the sample-based stability theorem.
In Lyapunov’s method, a positive deﬁnite function called Lyapunov function is needed.
The selection of the Lyapunov function is not trivial and largely determines the result of stability analysis.
In this paper, we construct the Lyapunov function using the following parameterization,
[Equation]
where f φ ( s ) is a fully-connected neural network (NN) with ReLU activation function.
φ denotes the parameters of the network and σ is a small positive constant.
Theorem 1 The stochastic system (1) is mean square stable if there exists a function L : S → R + and positive constants
α 1 , α 2 and α 3 , such that
[Equation]
where µ
[Equation]
is the inﬁnite sampling distribution (ISD).
Proof: The proof can be found in Section A in the Appendix.
The general idea of the proof will be summarized in the following.
First, we prove that ISD µ π exists if q π exists.
Then we exploit the
Abelian theorem and Egorov theorem to prove that L ( s t ) converges to zero at the inﬁnite instant.
Finally, (3) establishes the relation between L ( s t ) and c ( s t ) and concludes the proof.
Remark 1 For the Lyapunov function in (2), the value of α 2 can be approximately estimated.
In practice, we are typically concerned with the stability in a ﬁnite space S where the (cid:107) s (cid:107) 22 ≤ c and c ( s ) = (cid:107) s (cid:107) 22 .
Thus max s | f φ ( s ) − f φ (0) | 2 / (cid:107) s (cid:107)
+ σ is a valid choice for α 2 .
Considering that the neural network f φ with Relu activation is Lipschitz continuous, it follows that α 2 =
L f + σ where the Lipschitz constant L f can be efﬁciently estimated by using approaches in the literature (Scaman and Virmaux, 2018;
Fazlyab et al., 2019; Zou et al., 2020).
It can be found that in Theorem 1, the inﬁnite number of energy decreasing conditions are replaced by only a single sample-based inequality (4).
However, the validation of stability through a sample-based approach comes with a cost: it theoretically requires a tremendous, if not inﬁnite, number of samples to thoroughly estimate the state distributions at instants from 0 to inﬁnity, which is impractical.
Theorem 1 is valid for both model-free and model-based approach since the sample-based energy decreasing condition is aimed at canceling the requirement of point-wise energy decreasing condition.
Nevertheless, in the model-free setting, the estimation of transition probability in (4) only adds to the complexity of sampling.
In the next section, we will show that a ﬁnite number of samples should be informative enough to guarantee stability with a certain probability.
More speciﬁcally, the probabilistic stability bound will be given by closing the gap between inﬁnite and ﬁnite-sample guarantees.
F INITE S AMPLE P ROBABILISTIC S TABILITY B OUND
In this section, we will answer Q2 in Section 3 and present the ﬁnite sample-based stability theorem.
To estimate the µ π in Theorem 1, an inﬁnite number of trajectories of inﬁnite time steps are needed, whereas in practice only M trajectories of T time steps are accessible.
Thus in this section, we will ﬁrst introduce the ﬁnite-time sampling distribution (FSD) µ
Tπ (cid:44) 1 T (cid:80)
Tt =1 P ( s | ρ, π, t ) , as an intermediate to study the effect of the ﬁnite sample-based estimation.
Apparently, lim T →∞ µ Tπ = µ π .
The general idea of exploiting µ Tπ is: we ﬁrst derive the deviation of E µ Tπ ∆ L ( s ) from E µ π ∆ L ( s ) with respect to T , where
[Equation]
then we study the effect of estimating E µ Tπ ∆ L ( s ) with sample average and derive the probabilistic bound.
Finally, the above effects are uniﬁed to propose the ﬁnite sample-based stability guarantee.
Now, we ﬁrst close the ﬁrst gap in terms of deviation between E µ Tπ ∆ L ( s ) and E µ π ∆ L ( s ) .
To quantitatively analyze this effect, we introduce the following assumption,
Assumption 3 There exist a constant γ ∈ (0 , 1) such that for any π
[Equation]
where (cid:107) P ( s | π, ρ, t ) − q π ( s ) (cid:107) 1 denotes the L 1 -distance between probability measures P and Q .
Remark 2 It should be noted that the assumption above is not strict and should be generally easy to satisfy for ergodic MDPs.
Because q π denotes the stationary state distribution, it naturally follows that T (cid:80)
[Equation]
P
[Equation]
s | π, ρ, t )
− q π
[Equation]
s )
[Equation]
≤
[Equation]
T
[Equation]
( T ) ≤ 2 T , where γ ( T ) ∈ [0 , 1] without any further assumption.
The assumption proposed here merely replaces this time-varying γ ( T ) with a constant.
As a matter of fact, uniform ergodic for irreducible and aperiodic Markov chains (Levin and Peres, 2017; Bhandari et al., 2018;
Zou et al., 2019) is a special case of the above assumption, where the state distribution is required to converge to q π exponentially at the rate of γ t .
Nevertheless,
Assumption 3 allows us to give the quantitative bound for the deviation between E µ π ∆ L ( s ) and E µ Tπ ∆ L ( s ) with respect to T .
Based on Assumption 3, we introduce the following Lemma.
Lemma 1 Let T denotes the length of trajectories (also known as episodes or sequences).
If there exist positive constants α 1 , α 2 such that (3) hold, then
[Equation]
Proof: The proof can be found in Section B in the Appendix.
As shown in (6), the deviation of ﬁnite-time estimation of ∆ L ( s ) from the inﬁnite time estimation decreases as T grows and converges to zero if T is inﬁnity
In the following, we will derive the probabilistic bound on estimating E µ Tπ ∆ L ( s ) with M trajectories of T steps.
It is worth mentioning that since M trajectories are independent from each other, each trajectory as a whole is applicable for the estimation of ∆ L ( s ) under µ Tπ .
This will be demonstrated in the following lemma, where increasing M is desirable for the reduction of estimation deviation, while T doesn’t effect the probabilistic bound.
Lemma 2 Let M denote the number of trajectories and T denote the length of trajectories.
If there exist positive constants α 1 , α 2 such that (3) hold, then ∀ β ≥ 0 ,
[Equation]
where s t,m denotes the sampled state in the m -th trajectory at time t .
Proof: The proof can be found in Section C in the Appendix.
A noteworthy fact is that L ( · ) in (7) is bounded since (3) hold and c is a positive semi-deﬁnite variable clipped by c .
Thus it is straightforward to apply Hoeffding’s inequality to derive the probabilistic bound.
Now, the ﬁnite sample estimation of ∆ L ( s ) and E µ π ∆ L ( s ) are connected with E µ Tπ ∆ L ( s ) respectively by Lemma 1 and 2, we will unify them to derive the desired probabilistic stability guarantee.
Theorem 2 If there exists a function L : S → R + and positive constants α 1 , α 2 and α 3 , such that (3) hold, and for a number of M trajectories with T time steps there exists a positive constant (cid:15) such that
[Equation]
then the stochastic system can be guaranteed to be mean square stable with probability at least
[Equation]
where b 1 = 2( α 3 + α 2 ) c and b 2 = (2 α 2 + α 3 ) c .
If the desired conﬁdence of stability guarantee is at least δ , the associated overall sample complexity is at least O (log( 11 − δ )) .To
achieve a conﬁdence δ , M and T have to satisfy M ( (cid:15) − T γ − 1 b 1 ) 2 ≥ 2 c 2 (2 α
+ α 3 ) 2 log( 11 − δ ) .
Proof: The proof can be found in Section D in the Appendix.
The idea is that we estimate ∆ L ( s ) with ﬁnite samples in (9) and strengthen this ﬁnite sample-based condition with a constant (cid:15) , such that a small deviation in estimation will not cause misjudgment of stability.
In practice, (cid:15) is a hyperparameter to be tuned according to the number of samples available.
Remark
In (Kearns and Singh, 2002; Strehl et al., 2006; Jin et al., 2018), the ﬁnite-sample analysis and asymptotic convergence of various classical RL algorithms have been extensively studied.
However, to the best of our knowledge, Theorem 2 is the ﬁrst ﬁnite-sample analysis for sample-based stability analysis, providing a probabilistic stability guarantee that is related to the number of samples.
The probabilistic bound (10) is a monotone increasing function of T and M , and approaches to 1 as T and M tend to inﬁnity.
Intuitively, the trajectories with inadequate length cannot reﬂect the evolution of the system dynamics and thus are not applicable in the stability analysis.
Thus the requirement that the length of the trajectories T be greater than a minimum value (8) is reasonable.
Nevertheless, it is possible to derive tighter bounds in the future by applying other inequalities, such as Bernstein inequality.
The sharpness of the derived bound will be illustrated combined with a Cartpole example in Section 7.
S AMPLE -B ASED C ONTROL WITH S TABILITY G UARANTEE
Based on the theoretical results in the previous sections, one can judge whether the system is stable given several ﬁnite-length trajectories by estimating (9).
The theoretical results in the stability theorems using Lyapunov’s method do not, however, give a prescription for determining the Lyapunov function and controller.
To translate the theorem into practical algorithms, the high-level plan is to parameterize L ( s ) with (2) and the controller π ( a | s ) with an arbitrary NN π θ ( a | s ) .
Then φ and θ will be updated separately and iteratively using stochastic gradient descent algorithms until system (1) is stabilized such that (9) is satisﬁed.
We use τ to denote a trajectory ( τ = { s 1 , a 1 , s 2 ...s T } ), and τ ∼ π is the shorthand for indicating that the distribution over trajectories depends on π :
P ( τ
[Equation]
=
[Equation]
6. P OLICY G RADIENT
In this subsection, we will focus on how to learn the controller in an iterative manner, repeatedly estimating the policy gradient of the target function with samples and updating θ through stochastic gradient descent.
∆ L ( s ) is temporarily assumed to be given, i.e., φ are ﬁxed.
In Section 6.2, we will show how the Lyapunov function is selected and learned after θ is updated.
Since the left-hand side of (9) is the unbiased estimate of ∆ L ( s ) on µ
Tπ , the problem can be formulated by
[Equation]
A straightforward way of solving the constrained optimization problem above would be the ﬁrst-order method Bertsekas (2014) (Chapter 4), also known as gradient descent.
At each update step, the gradient of (11) with respect to θ is estimated with samples, and θ updates a small step in the opposite direction of the estimated gradient vector.
The gradient of (11) with respect to θ is derived in the following theorem.
Theorem 3 The gradient of Lyapunov condition (11) is given by the following,
[Equation]
where C t 1 : t 2 = (cid:80) t 2 t = t 1 c ( s t ) denotes the sum of cost c over a time interval and C t +1:
t =
The proof of Theorem 3 can be found in Section E in the Appendix.
Surprisingly, we found that the policy gradient derived in Theorem 3 is very similar to that used in the vanilla policy gradient method, i.e., REINFORCE Sutton and Barto (2018), in the classic RL paradigm.
In RL, the objective is to minimize a certain objective function J θ =
E
[Equation]
∼
[Equation]
and the policy gradient is given as follows:
[Equation]
Essentially, despite the scale of 1 T , (12) and (13) are equivalent if one chooses c ( s ) to be the Lyapunov function and sets α 3 = 1 .
This implies that given system (1), REINFORCE actually updates the policy towards a solution that can stabilize the system, although it is now aware of under what conditions the solution is guaranteed to be stabilizing.
In particular, we can view REINFORCE as a special case of our result, since we prove that many other choices of α 3 and Lyapunov functions are admissible to ﬁnd a stabilizing solution.
The default setting of c ( s ) as L ( s ) and α 3 = 1 in
REINFORCE may not satisfy (9), while we reveled that many other feasible combinations of L and α 3 potentially exist.
In light of this connection with REINFORCE, it is natural to propose a similar learning procedure based on Theorem 3, which we name as Lyapunov-REINFORCE (termed as ‘L-REINFORCE’).
L-REINFORCE updates the policy with the policy gradient proposed in (12).
Instead of minimizing the objective function (6.1), L-REINFORCE aims to learn a stochastic policy
π ( a | s ) such that the conditions in Theorem 2 are satisﬁed.
Furthermore, to reduce the variance in the estimation of (12) and speed up learning, it is desirable to introduce a baseline function b ( s ) in (12) and the estimation is still unbiased Sutton and Barto (2018):
[Equation]
6. L YAPUNOV F UNCTION
The Lyapunov function is parameterized using a DNN f φ in (2).
In fact, any real function f is admissible in (2) to construct Lyapunov function, thus many ways of updating φ are applicable in our framework, e.g. Prokhorov (1994); Petridis and Petridis (2006);
Richards et al.
In this paper, we intuitively choose the value function to be the update target for f to examine the effectiveness of proposed results as exploited in Berkenkamp et al.
; Chow et al.
and leave other possible choices for future work.
To wrap up, the L-REINFORCE algorithm is summarized in Algorithm 1.
Algorithm 1 L-REINFORCE repeatfor 1, 2, ..., M do
Collect transition pairs following π θ for T steps end for θ ← θ − α ∇ θ E µ Tπθ ∆ L ( s )
Update φ of Lyapunov function/value network to approximate the designed target until There exists α 3 such that E µ Tπθ ∆ L ( s ) < − (cid:15)
E XPERIMENT 7. A C ARTPOLE E AXAMPLE
To demonstrate the effectiveness of the proposed method, we consider the stabilization task of a simulated Cartpole Brockman et al.
The goal is to stabilize the pole vertically at the position x = 0 .
We adopt REINFORCE as the baseline method for comparison.
In addition, soft actor-critic (SAC) Haarnoja et al.
(2018), a state-of-the-art off-policy RL algorithm, is also included.
L-REINFORCE and REINFORCE select the action among {− 10 , 0 , 10 } .
SAC selects the control force in the continuous space with the same minimal and maximal value, thus better performance can be potentially achieved.
The detailed experiment setup and hyperparameters are presented in
Appendix F.
[Figure]
It is important to note that the stability of a system can not be judged from the cumulative cost (or return) because stability is an inherent property of the system dynamics and a stable system may not be optimal in terms of the return.
Thus in Figure 1, we show the transient system behavior under the learned policies of L-REINFORCE, REINFORCE, and SAC.
The agents are initialized at different positions in the space and their subsequent behaviors are observed.
As shown in Figure 1, starting from different initial states, L-REINFORCE can efﬁciently stabilize the system to the origin.
On the contrary, the cartpole diverges under the control of REINFORCE (Figure 1. b).
SAC can also keep the pole vertically, but the cart can not be stabilized to the position where x = 0 .
In some cases, the
SAC agent even slowly drifts away (see the left side of Figure 1. c).
To have a more clear view, these trajectories are further illustrated in the time domain in Appendix F.
To let the readers have an intuitive sense of the probabilistic stability bound, the bound for Cartpole under the control of the L-REINFORCE agent is shown in Figure 2.
As shown in Figure 2, the probability of stability increases sharply as the minimum T requirement (8) is satisﬁed.
Increasing
M and T are both helpful for raising the conﬁdence of stability guarantee.
The probabilistic stability guarantee measures the reliability of learned policies.
By tuning the hyperparameters such as M , T , and α 3 , one can achieve the conﬁdence of stability guarantee according to the real needs.
[Figure] [Figure]
E XAMPLES
In this part, we will illustrate the effectiveness of the proposed framework on some high dimensional control problems, where the system dynamics are highly nonlinear and even corrupted by various noises, thus making them more stochastic and challenging.
Three examples are included: a highdimensional continuous control problem of 3D robots, HalfCheetah, the molecular synthetic biological gene regulatory networks (GRN) corrupted by the additive and multiplicative uniform noises.
In the living cells of biological systems, gene expression is very noisy and there are strong evidence on the genetic basis on these noises in literature of genetic regulatory networks Swain et al.
Bar-Even et al.
Details of the experimental setup are referred to the Appendix.
To achieve high performance on these continuous control tasks, we further incorporate the maximum entropy method (Shi et al., 2019; Haarnoja et al., 2018; Zhao et al., 2019) in the proposed framework.
By introducing the entropy regularization, the policy is encouraged to explore more and less easy to early convergence to suboptimal solutions.
To have a fair comparison, only SAC is included as the baseline in these examples, given to its superior performance on continuous control tasksHaarnoja et al.
(2018), while REINFORCE is excluded due to its poor performance.
In Figure 3, 4 and 5, the state trajectories of the systems are shown in time-domain.
It is observed that even though the systems are highly nonlinear and stochastic due to the noises, L-REINFORCE is still able to stabilize the tracking error to zero in the mean.
In comparison, although SAC succeeded in stabilization in some of the trials, see Figure 4 and 5, but its success appears to be very random and can be hardly guaranteed.
C ONCLUSION
In this paper
[Figure]
we proposed a sampling-based approach for stability analysis of nonlinear stochastic systems modeled by the Markov decision process in a model-free manner.
Instead of verifying energy decreasing point-wisely on the state space, we proposed a stability theorem where only one sampling-based inequality is to be checked.
Furthermore, we showed that with a ﬁnite number of trajectories of ﬁnite length, it is possible to guarantee stability with a certain probability and the probabilistic bound is derived.
Finally, we proposed a model-free learning algorithm to learn the controller with a stability guarantee and revealed its connection to REINFORCE.
REINFORCE is not the state-of-the-art RL algorithm for complicated continuous tasks.
In the future, an important direction is to extend the theoretical analysis to more efﬁcient algorithms than REINFORCE.
[Figure]
[Figure] [Figure]
[Figure]
Appendix A P ROOF OF T HEOREM
The existence of the sampling distribution µ π ( s ) is guaranteed by the existence of q π ( s ) .
Since the sequence { P ( s | ρ, π, t ) , t ∈ Z + } converges to q π ( s ) as t approaches ∞ , then by the Abelian theorem, the sequence { 1 T (cid:80) Tt =1 P ( s | ρ, π, t ) , T ∈ Z + } also converges and µ π ( s ) =
q π ( s ) .
Step
We assume that P ( s | ρ, π, t ) converges uniformly to q π ( s ) in S and the state space S is bounded which means that
[Equation]
and
[Equation]
Then by (3) and (4) we get
[Equation]
By the deﬁnition of ISD, we know that the sequence ( 1 N N (cid:80)
[Equation]
P
[Equation]
s | ρ
[Equation]
π
[Equation]
t ))
L ( s ) converges pointwise to the function q π ( s ) L ( s ) and ( 1 N N (cid:80) t
=0 P ( s | ρ, π, t ))
E P π ( s (cid:48) | s ) L ( s (cid:48) )
converges point-wise to q π ( s ) E P π ( s (cid:48) | s ) L ( s (cid:48) ) .
According to the Lebesgue’s Dominated convergence theorem, if a sequence of function f n ( s ) converges point-wise to a function f and is dominated by some integrable function g in the sense that
[Equation]
Then
[Equation]
Then we get
[Equation] [Equation]
So we have
[Equation]
In general, we assume that lim t →∞ E P ( s | ρ,π,t ) L ( s ) exists, then by Abelian theorem we have
[Equation]
Step
If we only have point-wise limit q π ( s ) =
lim t →∞ P ( s | ρ, π, t ) exists.
For arbitrary positive real number R > 0 , we consider domain B R (0) , by Egorov theorem, ∀ (cid:15) > 0 , ∃ A (cid:15),R ⊂ B R (0) such that lebesgue measure of B R (0)
\ A (cid:15),R less than (cid:15) and P ( s | ρ, π, t ) converges to q π uniformly on A (cid:15),R .
[Equation]
We consider the second part
[Equation]
Because on S ∩ A (cid:15)
[Equation]
R
[Equation]
P ( s | ρ
[Equation]
π, t )
→ q π ( s ) uniformly
[Equation]
we can get the following equation as the same way in Equation A.
[Equation]
As (cid:15) → 0 and R → + ∞ , we have
[Equation]
So we get the A.8.
Suppose that there exists a starting state s 1 ∈ { s 1 | c ( s 1 ) ≤ b } and a positive constant d such that lim t →∞ E P ( s | s 1 ,π,t ) c ( s ) =
d or lim t →∞ E P ( s | s 1 ,π,t )
c ( s ) = ∞ .
Consider that ρ ( s 1 ) > 0 for all starting states in { s 1 | c ( s 1 ) ≤ b } (Assumption 2), then lim t →∞ E s ∼ P ( ·| π,ρ,t ) c ( s ) > 0 , which is contradictory with A.8.
Thus ∀ s 1 ∈ { s 1 | c ( s 1 ) ≤ b } , lim t →∞ E P ( s | s 1 ,π,t ) c
( s )
= 0 .
Thus the system (1) is mean square stable by Deﬁnition 1.
B P ROOF OF L EMMA
First of all, using Assumption 3, we can quantify the variation of E µ π ∆ L ( s ) − E µ Tπ ∆ L ( s ) with respect to T .
[Equation]
Thus it follows that
[Equation]
where (cid:107) µ π ( s ) − P ( s | ρ, π, t ) (cid:107) 1 is L 1 -distance between µ π and P ( s | ρ, π, t ) , which is bounded by 2 .
(cid:107) ∆ L ( s ) (cid:107) ∞ is the upper bound of ∆ L , according to (3),
[Equation]
Thus we have
[Equation]
At last, one obtains the bound in (6),
[Equation]
where i is obtained through the sum of proportional series.
C P ROOF OF L EMMA
The high-level plan is to apply Hoeffding’s inequality to establish the probabilistic bound.
First, the
Hoeffding’s inequality holds under the condition that random variables follow the same distribution and are independent of each other.
[Equation]
Then the probabilistic bound δ of interest becomes
[Equation]
which contains a number of M random sequences of variables
L ( s ot +1 ,m ) ∈ [0 , α 2 c ] and α 3 c ( s t,m ) −
L ( s t,m ) ∈ [ − α 2 c, α 3 c ] .
Thus by Hoeffding’s inequality, the following inequality hold ∀ β ≥ 0 , δ ≤ exp (cid:18) − 2 M 2 β
M (2 α 2 + α 3 ) 2 c(cid:19) = exp (cid:18) − 2 Mβ(
α 2 + α 3 ) 2 c(cid:19)
(C
[Equation] [Equation]
D P ROOF OF T HEOREM
We will use ∆ L M,T ( s ) to represent the sample average on the left hand side of (9) for improved readability.
Essentially, ∆ L M,T ( s ) is the unbiased estimation of E µ Tπ ∆ L ( s ) , of which the probabilistic bound is derived in Lemma 2.
However, in Lemma 1 we show that E µ Tπ ∆ L ( s ) is not an unbiased estimation of E µ π ∆ L ( s ) since T is ﬁnite.
Now based on these lemmas, we need to investigate the effect of estimating E µ π ∆ L ( s ) in (4) with ∆ L M,T ( s ) .
Let ω denote the bound in (6).
According to (9), L M,T ( s ) + (cid:15) is semi-negative deﬁnite, then
[Equation]
Then we apply the probabilistic bound derived in Lemma 2, by substituting β in (7) by
[Equation]
So
[Equation]
The desired bound for guaranteeing mean square stability in probability can be obtained in (10)
[Equation]
E P ROOF OF T HEOREM
In this section
[Equation]
we will show that
[Equation]
To proceed
[Equation]
we introduce P ( s t | ρ, π θ ) .
=
P ( s | ρ, π θ , t ) to make the derivation more clear.
First, the following equalities hold,
[Equation]
Take the derivative of the above equation with respect to θ , one has
[Equation]
For any t ∈ [
[Equation]
+ ∞ ) and function f ( s t ) ,
[Equation]
Iterate the above relation to t = 1 , one has
[Equation]
Substitute the above equation into (E.3),
[Equation]
For the second term in (E.6), we move the ﬁrst sum into the expectation and exchange its order with the second sum, then
[Equation]
which concludes the proof.
F E XPERIMENT DETAILS F. E XPERIMENT SETUPS a. Cartpole
The initial horizontal position of cart x is uniformly distributed on [ − 1 , 1]( m ) and other states are uniformly distributed on [ − 0 .
The episode ends if the angle θ exceeds the thresholds ± 0 .
35 rads or the total time step exceeds 250 .
The control input is the horizontal force F ∈ {− 10 , 0 , 10 } ( N ) applied to the cart.
b. HalfCheetah
The goal is to control a 17-dimensional 2-legged robot simulated in the MuJoCo simulator.
The control task belongs to the reference tracking problem, i.e., to enable the robot to run at the speed of 1m/s in the X-axis direction.
The cost is determined by the Euclidean difference between current speed and target speed.
The control input is the torque implemented at each joint.
c. GRN with additive noises
The GRN is a synthetic three-gene regulatory network where the dynamics of mRNAs and proteins follow an oscillatory behavior (Elowitz and Leibler, 2000).
A discrete-time mathematical description of the GRN with additive noises, which includes both transcription and translation dynamics, is given by the following set of discrete-time equations:
[Equation]
(cid:20)
[Equation]
a
[Equation]
26 ( t )
[Equation]
u
[Equation]
( t ) (cid:21)
[Equation]
1 ( t ) , x
[Equation]
( t
[Equation] [Equation]
x
[Equation]
( t ) + dt · (cid:20)
[Equation]
[Equation] [Equation]
u
[Equation]
( t ) (cid:21) + ξ 2 ( t ) , x 3 ( t + 1) = x 3 ( t ) + dt · (cid:20) − γ 3 x 3 ( t )
+ a
K 3 + x 25 ( t ) + u 3 ( t ) (cid:21) + ξ 3 ( t ) , x 4 ( t + 1) =
x 4 ( t ) + dt · [ − c 1 x 4 ( t ) + β 1 x 1 ( t )]
+ ξ 4 ( t ) , x 5 ( t + 1) =
x 5 ( t ) + dt · [ − c 2 x 5 ( k ) + β 2 x 2 ( t )]
+ ξ 5 ( t ) , x 6 ( t + 1) =
x 6 ( t ) + dt · [ − c 3 x 6 ( t ) + β 3 x 3 ( t )]
+ ξ 6 ( t ) .
(F.1)
Here, x 1 , x 2 , x 3 (resp.
x 4 , x 5 , x 6 ) denote the concentrations of the mRNA transcripts (resp.
proteins) of genes 1, 2, and 3, respectively.
ξ i , ∀ i are i.i.d.
uniform noise ranging from [ − δ, δ ] , i.e., ξ i ∼
U ( − δ, δ ) , δ = 0 .
a 1 , a 2 , a 3 denote the maximum promoter strength for their corresponding gene, γ 1 , γ 2 , γ 3 denote the mRNA degradation rates, c 1 , c 2 , c
3 denote the protein degradation rates, β 1 , β 2 , β 3 denote the protein production rates, and K 1 , K 2 , K 3 are the dissociation constants.
The set of equations in Eq.(F.1) corresponds to a topology where gene 1 is repressed by gene 2 , gene 2 is repressed by gene 3 , and gene 3 is repressed by gene 1 .
dt is the discretization time step.
The task is to control the concentration of Protein 1 ( x 4 ) to follow a desired reference signal (a sine wave r = 8 + 7 sin(2 πt/ 200) ).
In practice, only the protein concentrations are observed and given as readouts,for instance via ﬂuorescent markers (e.g., green ﬂuorescent protein, GFP or red ﬂuorescent protein, mCherry).
The control scheme u i will be implemented by light control signals which can induce the expression of genes through the activation of their photo-sensitive promoters.
To simplify the system dynamics and as it is usually done for the repressilator model (Elowitz and Leibler, 2000), we consider the corresponding parameters of the mRNA and protein dynamics for different genes to be equal.
More background on mathematical modeling and control of synthetic biology gene regulatory networks can be referred to Strelkowa and Barahona (2010);
Sootla et al.
In this example, the parameters are as follows:
[Equation]
We uniformly initialized between 0 to 5, i.e., x i (0) ∼ U (0 , 5) , which is the range we train the policy in Section 7.
d. GRN with multiplicative noises
In this example, the basic system dynamic is the same with the GRN with additive noises, except that the magnitude of the noise is dependent on the state, as shown in the following equations.
[Equation]
F
[Equation]
All the parameters are the same with the GRN with additive noises.
ξ i ( t ) is still a uniform noise, ranging from [ − δ, δ ] , δ = 5 .
F. I MPLEMENTATION DETAILS
In L-REINFORCE, there are two networks: the policy network µ ( s ) and the Lyapunov network f ( s ) .
For the policy network, we use a fully-connected MLP with one hidden layer of 32 units activated by
Relu, outputting the logits for softmax.
[Equation]
where µ i ( s ) denotes the i th dimension of µ ( s ) .
f ( s ) is a fully-connected MLP with two hidden layers activated by Relu.
f is trained to approximate the value function, V ( s t ) =
(cid:80)
[Equation]
−
[Equation]
] , where b is a designed positive constant.
We found that this constant is helpful to encourage the agent to sustain for longer.
The parameter of f is updated through soft replacement, i.e. φ k +1 ← (1 − τ ) φ k
+ τ ( φ k − α ∇ φ δ 2 ) , where δ is the temporal difference.
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
In Figure S7 and S8, we show each dimension of the state respectively in the time domain.
As shown in the ﬁgures, under the control of L-REINFORCE, the average of the state converge to the origin
[Figure]
θ     500 0.
[Figure]
Angular velocity ˙ θ           (
[Figure]
position x     500 0.6 0.
[Figure]
with low variance.
In comparison, though SAC can also stabilize part of the states (angular and horizontal velocity), the other states are left drifting away slowly on average.
[Table] [Table]
[Table] [Table]