SYNC: S AFETY -A WARE N EURAL C ONTROL FOR S TABILIZING S TOCHASTIC D ELAY -D IFFERENTIAL E QUATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Stabilization of the systems described by stochastic delay-differential equations is a challenging task in control community.
Here, to achieve this task, we leverage neural networks to learn control policies using the information of the controlled systems in some prescribed regions.
The two learned control policies, the neural deterministic controller (NDC) and the neural stochastic controller (NSC), work effectively because the learning procedures use, respectively, the well-known
LaSalle-Type theorem and the newly-established theorem for guaranteeing the stochastic stability in SDDEs.
We theoretically investigate the performance of the proposed NDC and NSC in terms of convergence time and energy cost.
More practically and significantly, we improve our learned control policies through considering the situation where the controlled trajectories can only evolve in some specific safety set.
Such successful stabilization based on neural networks restricted in safety set is attributed to our further developed theory for safety verification of SDDEs using the stochastic control barrier function, and we name it as SYNC ( S afet Y -aware N eural C ontrol).
The efficacy of all the articulated control policies, including the SYNC, is demonstrated systematically by using representative control problems.
I NTRODUCTION
[Figure]
∗ d ( ) = ( ( ), ( − ), )d + + + ( ( ), ( − ), )d
[Figure]
(Mao, 1996; Lin & He, 2005; Sun & Cao, 2007;
Guo et al., 2016) have been leveraged to characterize the complex dynamical behavior emergent in real-world systems with dependence on the current state, the past state, and the noise.
Efficiently controlling these systems is a longstanding crucial problem, with the consequent emphasis being placed on the design of control policies and analysis of stability in SDDEs.
Traditional control methods in stochastic settings have been fully developed in the convex optimization frameworks using the control Lyapunov stability theory, e.g. the quadratic programming (QP)
(Fan et al., 2020; Sarkar et al., 2020).
These methods cannot provide the analytical form of feedback controllers and own a high computational cost, requiring solving QP problems at each iteration step.
To overcome these difficulties, utilizing neural networks (NNs) to automatically design controllers becomes one of the mainstream approaches in recent years (Zhang et al., 2022; Chang et al., 2019).
However, existing machine-learning-based methods either focus on controlling systems without time-delay or aim at learning the control Lyapunov function instead of the control policy (Khansari-Zadeh & Billard, 2014), promoting us to design neural controllers for general nonlinear SDDEs.
The safety verification of controlled systems plays an important role in many branches of cybernetics and industry.
For example, with the safety verification, one can reduce a significant economic burden and loss of life (Ames et al., 2016; Wang et al., 2016).
In particular, the dominant framework for safety control in stochastic settings is the use of stochastic control barrier function (SCBF)
(Clark, 2019; 2021; Santoyo et al., 2021).
The core idea of designing a candidate SCBF is that its value tends to explode as the system’s state leaves the safe region, implying a safety guarantee as long as one could design a controller such that the SCBF is always finite within the controlled time duration.
Unfortunately, the existing theories of SCBF either require a lot of inequality constraints or are limited in handling systems without time-delay.
In this paper, we utilize neural networks (NNs) to learn control policies for SDDEs based on the corresponding stability theories.
Additionally, we develop a simplified SCBF theory for SDDEs and then use it to construct the neural controller with safety guarantee, named SYNC.
All these control policies are intuitively depicted in Figure 1.
The major contributions of this paper are listed as follows
• designing a novel and practical framework of neural deterministic control based on the existing LaSalle-Type stability theory, • proposing a simplified stability theorem and designing the second novel neural stochastic control framework that can benefit from noise according to this theorem, • providing theoretical estimation for proposed neural controller in terms of convergence time and energy cost, • cultivating a simplified SCBF theory for SDDEs and exploiting it in our neural framework to obtain a safety guarantee, and • demonstrating the efficacy of the proposed neural control methods through numerical comparisons with the typical existing control methods on several representative physical systems.
P RELIMINARIES
To begin with, we consider the SDDE of the following general form:
[Equation]
where x ( t ) = ξ ( t ) ∈ C F 0 ([ − τ, 0]; R d ) is the initial function, the drift term F : R d × R d × R + → R d and the diffusion term G : R d × R d × R + → R d × r are Borel-measurable functions, and B t is a standard r -dimensional
( r -D)
Brownian motion defined on probability space (Ω , F , {F t } t ≥ 0 , P ) with a filtration {F t } t ≥ 0 satisfying the regular conditions.
Without loss of generality, we assume
F ( 0 , 0 , t ) = 0 , G ( 0 , 0 , t ) = 0 to guarantee that x ( t ) ≡ 0 , t ≥ 0 is an equilibrium of Eq.
For simplicity, the following notations and assumptions are used throughout the paper.
Assumption 2.1 Eq.
(1) has a unique solution x ( t, ξ ) on t ≥ 0 for any ξ ∈ C F 0 ([ − τ, 0]
[Equation]
R d ) and for every integer n ≥ 1 , there is a number K n >
such that
[Equation]
for any ( x , y , t ) ∈ R d × R d × R + with ∥ x ∥ ∨ ∥ y ∥ ≤ n , where ∥ · ∥ denotes the L 2 -norm and ∥ · ∥ F denotes the Frobenius norm, i.e. ∥ G ( x , y , t
[Equation]
Definition 2.1 (Derivative Operator) Define the differential operator L associated with Eq.
(1) by
[Equation]
According to the above definition of the derivative operator, an operation of L on the function
V ∈ C 2 , 1 ( R d × R + ; R ) yields:
[Equation]
Here, V t , ∇ V and H V represent, respectively, the time derivative, the gradient and the Hessian matrix of V .
Notably, the following LaSalle-Type stability theorem will be used in the establishment of our part of the results.
Theorem 2.2 (Mao, 2002) Suppose that Assumptions 2.1 holds.
Assumes there are functions V ∈
C 2 , 1 ( R d × R + ; R + ) , γ ∈ L 1 ( R + ; R + ) , and w 1 , w 2 ∈ C ( R d ; R + ) such that [Equation] x , [Equation]
[Equation]
x
[Equation]
and
lim ∥
x
[Equation]
inf [Equation] ∞ . Then, Ker( w 1 − w 2 ) ̸ = ∅ and lim t →∞ dist( x ( t, ξ ) , Ker( w 1 − w 2 ))
=
a.s. , where
Ker( w 1 − w 2 ) ≜ { x : w 1 ( x ) − w 2 ( x ) = 0 } , dist( x, K ) ≜ inf y ∈ K
∥ x − y ∥ for a set K ⊆ R d , and a.s. stands for the abbreviation of almost surely.
Problem Statement We assume that the zero solution of the following SDDE:
[Equation]
is unstable, i.e. lim t →∞ x ( t ; ξ ) ̸ = 0 on some set of positive measures.
We aim to stabilize the zero solution with control based on neural networks (NNs).
In other words, our goal is to leverage the
NNs to design an appropriate controller u
= ( u f , u g ) with u f ( 0 , 0 , t ) =
u g ( 0 , 0 , t ) =
such that the controlled system
[Equation]
is stabilized to the zero solution.
We call u f : R d × R d × R + → R d as deterministic control and u g : R d × R d × R + → R d × r as stochastic control since they are integrated with d t and d B t , respectively.
The major difficulty of this problem comes from the non-Markovian property of SDDEs, hence we cannot apply MDP (Markov decision process)-based methods, such as reinforcement learning, to control SDDEs.
Existing works prefer to learning deterministic control and regard the noise as a negative part that may destroy the natural dynamics of f .
In what follows, we not only show that the deterministic control can achieve stabilization, but also that it can be done by elaborately-designed stochastic control, yielding two frameworks, the neural deterministic control (Section 3) and the neural stochastic control (Section 4).
We make all our code and data available at https://anonymous.4open.science/r/SYNC-35E8 .
N EURAL D ETERMINISTIC C ONTROL
In this section, we propose a framework of neural deterministic control (NDC) based on Theorem 2.2 to stabilize system (3) and provide experimental comparisons with traditional control methods.
In addition, we analytically and experimentally investigate that the framework cannot be directly generalized to the stochastic control version.
Such a problem will be addressed in the next section.
3. M ETHOD : L EARNING C ONTROL AND A UXILIARY F UNCTIONS
The core idea of our method is that once we construct the auxiliary functions V, γ, w 1 , w 2 and the neural controller u , resulting in the controlled system (4) satisfying all the conditions assumed in Theorem 2.2, then the solution x ( t ; ξ ) converges to the Ker( w 1 − w 2 ) .
In particular, if we set Ker( w 1 − w 2 ) = { 0 } , the unstable zero solution of the control-free system (3) can be stabilized.
To this end, we first provide the reasonable constructions of NNs to learn these candidate functions.
Thus, we design the explicit form of loss function in the learning step.
Auxiliary Function
We employ a multi-layer feedforward neural network, denoted by NN ( · ; θ ) , to design all the functions.
To be concrete, θ 1 is the parameter vector of the positive function V ( x , t ; θ 1 ) , and the L 2 term ∥ x ∥ 2 is added to guarantee lim
∥ x ∥→∞ inf 0 ≤ t<
∞ V ( x , t [Equation] θ [Equation]
= ∞ , that is
[Equation]
In our framework, it requires V ∈ C 2 , 1 ( R d × R + ) .
We therefore choose C 2 activation function of an NN, such as the hyperbolic tangent function, Tanh ( · ) .
In order to design an integrable positive function γ ( t ) with an NN, we use an activation function with at most linear growth such as ReLU and multiply an exponential decay factor to the output of the NN, that is
[Equation]
For simplicity, we design w ( x , θ w ) =
NN ( x ; θ w ) 2 as a positive function.
Additionally, we set
[Equation]
Deterministic Control Function We first consider the deterministic control, i.e. u = ( u f , 0 ) .
To guarantee the same zero solution of the control-free system (3) and the controlled system (4), the NDC u f : R d × R d × R + → R d should satisfy u f ( 0 , 0 , t ) = 0 .
One feasible way to meet such a condition is to set u f ( x , y , t ) =
NN ( x , y , t ; θ f ) − NN ( 0 , 0 , t ; θ f ) or u f ( x , y , t ) =
diag( x ) NN ( x , y , t ; θ f ) .
Here, diag( x ) is a diagonal matrix with its i -th diagonal element being x i .
Loss Function Once the learned functions V, γ, w 1 , w 2 and u with the coefficient functions, f u ≜ f + u and g , in the controlled system (4), meet all the conditions assumed in Theorem 2.2, the stability of zero solution is naturally assured.
To achieve this, we demand a suitable loss function to evaluate the likelihood that those conditions are satisfied.
It can be seen from our construction that the only condition needed to be satisfied is L V ( x , y , t ) ≤ γ ( t ) − w 1 ( x ) + w 2 ( y ) .
Hence, we define LaSalle’s loss function for the controlled system (4) as follows.
Definition 3.1 ( LaSalle’s Loss) Consider the above parameterized candidate functions V, γ, w 1 , wand a controller u f for the controlled system (4).
Then, LaSalle’s loss is defined as
[Equation]
where the state variable x obeys the distribution µ .
In practice, we consider the following empirical loss function through the Monte Carlo sampling
[Equation]
where { x i , y i , t i } Ni =1 are sampled from the distribution µ = µ (Ω) and Ω is some closed domain in R d
× R d × R + .
In summary, the developed NDC framework is shown in Algorithm 1 in Appendix A.3.1.
Remark 3.1 The proposed NDC framework can be easily applied to the autonomous SDDE d x ( t ) =
f ( x , x ( t − τ ))
d t + g ( x , x ( t − τ ))
d B t .
In particular, one can simply consider the autonomous auxiliary function V and the control function, and set γ ( t ) = 0 .
For sample distribution µ (Ω) , here we select the uniform distribution on a large enough closed region Ω as used in (Han et al., 2016;
Chang et al., 2019), and we provide further analysis about the influence of µ in Appendix A.2.1.
3. N UMERICAL AND A NALYTICAL I NVESTIGATIONS
Comparison with Existing Methods Recent works on controlling time-delayed systems mainly focus on elaborately designing the analytical form of control to satisfy the conditions in the LaSalleType Theorem 2.2 (Lin & He, 2005; Xu et al., 2014), or simultaneously designing control and the
Lyapunov function to satisfy the conditions based on the Lyapunov theory (Yu & Cao, 2007).
It should be noted that all these methods require a delicate design of functions for specific dynamics, and thus are limited in practical application for controlling general time-delayed systems.
However, our neural method leverages neural networks to automatically learn the control policies, and can be easily applied in any kinds of time-delayed systems having stochastic settings.
In Figure 3, we numerically compare the NDC and a baseline, the linear control (LC) proposed in (Lin & He, 2005), on a noised driving-response Chua’s circuit.
Here, Chua’s circuit is a third-order autonomous dynamical system with only one nonlinear element, producing typical chaotic dynamics (Matsumoto, 1984).
In the experiment, we show that the NDC can find the neural control for the response system y
=
( y 1 , y 2 , y 3 ) with the autonomous and even the nonautonomous time-delay noise while the nonautonomous time-delay noise is beyond the concern in (Lin & He, 2005).
The configuration of the experiment is described in Appendix A.3.4.
[Figure]
Failure in Finding Stochastic Control As we can see that the NDC performs well, a natural idea is to utilize the noise part to achieve the stabilization of the SDDE ( 3).
To explore this idea, we adopt the same
NN of u f , design u g = NN ( x , y , t ; θ g ) , and train its parameters θ g with LaSalle’s loss ( 8).
However, in Figure 2, we show that the loss cannot converge to zero in controlling a simple 1 -D toy system via the stochastic controller u g : d x ( t ) = [ x ( t ) + x ( t − τ )]
d t +
[Figure]
[ x ( t − τ ) + u g ( x ( t ) , x ( t − τ ); θ g )]
d B t .
Actually, this phenomena can be analytically explained.
Notice that θ g arises in loss function as a quadratic term l ( θ g ) =
Tr[ u ⊤ g H V u g ] according to Eq.
(2), the sign of this term depends on the convexity of V , i.e. the maximum eigenvalue’s sign of H V .
Nevertheless, the positive function V with lim ∥
x ∥→∞ V ( x , t )
= ∞ implies l ( θ g ) ≥
at most time.
Hence, the ideal case l ( θ g ) = 0 in the training procedure is equivalent to u g = 0 , which means that we are unable to learn a stochastic controller under LaSalle’s loss (8) satisfying the sufficient conditions assumed in Theorem 2.2.
Drawbacks of L 2 regularization in V
Adding L 2 regularization to objective functions is a classical operation to avoid over-fitting
(Ying, 2019) and guarantee the positive definiteness (Gallieri et al., 2019).
However, the explicit form ε
∥ x ∥ 2 may fail in learning an effective neural control as this function cannot be the candidate V function in some cases (Zhang et al., 2022).
The following example illustrates this point.
Example 3.2 Consider a 2 -D SDDE as follows:
[Equation]
In Appendix A.1.3, the solution of this system is validated to satisfy lim t →∞ x ( t ; ξ ) =
a.s. with any initial data ξ ∈ C F 0 ([ − 1 , 0]; R 2 ) ; however, k ∥ x ∥ 2 for any k ∈ R + cannot be a useful auxiliary
V function to identify the sufficient conditions in Theorem 2.2.
N EURAL S TOCHASTIC C ONTROL
To address the illustrated problems above, we design a new concise framework for fast learning a stochastic controller, called the neural stochastic control (NSC).
To this end, we first provide the following theoretical result on stabilization of general stochastic functional differential equations (SFDEs) with the proof provided in Appendix A.1.4.
Theorem 4.1 Consider the SFDE d x ( t ) =
F ( x t , t )d t + G ( x t , t )d B ( t ) , with F, G being locally Lipschitzian functions, F ( 0 , t ) = 0 , and G ( 0 , t ) = 0 .
For every M > 0 , assume that min ∥ x t (0) ∥
=
M ∥ x t (0)
⊤ G ( x t , t ) ∥ > 0 .
If there exists a number α ∈ (0 , 1) such that
[Equation]
for x t ∈ C ([ − τ, 0] , R d ) , where x t ( s ) =
x ( t + s ) for s ∈ [ − τ, 0] .
Then, the solution of the SFDE satisfies lim t →∞ x ( t ; ξ ) =
a.s. for any ξ ∈ C F 0 ([ − τ, 0]; R d ) .
Remark 4.2 The SFDE in Theorem 4.1 is formulated in a very general type, including the SDDE d x ( t ) =
F ( x ( t ) , x ( t − τ 1 ) , · · · , x ( t − τ q ) , t )d t + G ( x ( t ) , x ( t − τ 1 ) , · · · , x ( t − τ q ) , t )d B t with τ 1 < τ 2 < · · · < τ q ∈ [0 , τ ] .
This indicates that our framework can be generalized to stabilize the
SDDEs with multiple delays and even more general SFDEs as well.
In light of Theorem 4.1, we establish a more general framework for learning a neural controller of system (4) with the form u
= ( u f , u g ) designed in the same NN architecture as the one used in the NDC framework.
We focus on stochastic control with u f = 0 and provide more control combinations in Appendix A.3.3, whereas the loss function is differently designed as follows.
Definition 4.1 ( Asymptotic Loss)
Utilizing the notations in Definition 3.1 and g u
= g + u g , the loss function for the controlled system (4) with the controller u is defined as:
[Equation]
where θ =
( θ f , θ g ) .
Akin to Definition 3.1, we use the empirical loss function for training.
Here, α is an adjustable parameter, which is related to the convergence rate and the control energy.
We further discuss the design of the asymptotic loss in Appendix A.2.2 and numerically investigate the role of α in Appendix A.4.1.
We summarize the framework in Algorithm 3 in Appendix A.3.1.
And we further compare the computational complexity in Appendix A.3.2.
4. E XPERIMENTS OF THE C OMBINATION M ETHODS
[Table]
We compare our neural control methods on a noiseperturbed kinematic bicycle model for car-like vehicles (Rajamani, 2011) in terms of the convergence time and the energy cost, which are two important indexes to measure the quality of a controller (Yan et al., 2012; Li et al., 2017; Sun et al., 2017).
To quantify the energy cost in the control process, we first denote the stopping time by τ ϵ ≜ inf { t > 0 :
∥ x ( t ) ∥
= ϵ } and then denote the energy cost by E ϵ ≜ E (cid:2)(cid:82) τ ϵ 0 (cid:0) ∥
u f ∥ 2 + ∥
u g ∥ 2 (cid:1) d t (cid:3) .
We approximate this expectation value by the empirical value as 1 N (cid:80) Ni =1 (cid:82) τ iϵ 0 (cid:0) ∥ u if ∥ 2 + ∥ u ig ∥ 2 (cid:1) d t through the Monte Carlo sampling.
We show the results in Figure 4 and Table 1.
Table 1 includes the training time (Tt), empirical energy cost E 0 .
001 , nearest distance (Nd) between the bicycle and target position, empirical expectation E [ τ 0 .
001 ] of different methods.
We provide more experimental details in Appendix A.3.5.
We can see that the ranking of the comprehensive performance is that NSC is greater than NDC than QP.
This means that we can really benefit from introducing noise as control.
An explanation for this phenomena is that when we regard energy cost as objective function to be minimized, randomness can lead this functional to the shortest path quicker than the deterministic control, just like stochastic gradient descent outperforms the full-batch gradient descent.
We show the NSC can enlarge the region of attraction of 100 -D gene regulatory networks in Appendix A.4.2.
Uncontrollable Fluctuation
Although the neural stochastic method we propose outperforms the control methods including deterministic control, there exists an obvious disadvantage that the method can cause uncontrollable fluctuation due to the stochasticity.
However, we always want to bound this perturbation in practical application owing to physical and engineering restrictions in real world.
We tackle this safety guarantee problem for our methods in Section 6.
[Figure]
T HEORETICAL R ESULTS FOR NDC AND NSC
We have mentioned the stopping time and the energy cost in section 4.1 and numerically compare the proposed neural controllers with these indexes.
These two indexes are the classic factors to measure the performance of the controller (Sun et al., 2017).
In this section, we provide theoretical estimation for the upper bound of these indexes in controlled dynamics with NDC/NSC.
To begin with, we choose suitable activation function in our NN control such that the neural control function u ( x ) is
Lipschitz continuous with Lipschitz constant k u (Fazlyab et al., 2019; Aziznejad et al., 2020).
Then we have the following two theorems and we provide the proofs in Appendix A.1.7,A.1.8.
Theorem 5.1 ( Estimation for NDC) Consider the SDDE with NDC controller as
[Equation]
where ∥ f ( x , y ) − f (¯ x , ¯ y ) ∥∨∥ u f ( x , y ) − u f
(¯ x , ¯ y ) ∥ ≤ L ( ∥ x − ¯ x ∥ + ∥ y − ¯ y ∥ ) .
Assume the controlled system satisfies the similar conditions in Theorem 2.2 and Remark 3.1 with Ker ( w 1 − w 2 )
= 0 .
Denote stopping time as η ε = inf { t > 0 : ∥ x ( t ) ∥ = ε } , the corresponding energy cost in the control process with ϵ <
∥ x 0 ∥ as E ( η ε , T ) =
E [ (cid:82)
η ε ∧ T∥ u [Equation] x ( s ) , x ( s − τ ))
∥ 2 d s ] and under the same notations in Theorem 2.2, we have
[Equation]
where C 0 =
∥ x 0 ∥ 2 + (2 L 2 + L + k u ) (cid:82) 0 − τ ξ ( s ) 2 d s and ξ ∈ C [ − τ, 0] is the initial data.
Theorem 5.2 ( Estimation for NSC) Consider the SDDE with NSC controller as
[Equation]
where f, g are the same as those in Theorem 6.2.
Assume the controlled system satisfies the similar conditions in Theorem 4.1, under the same notations in Theorem 4.1, if the term in (9) further satisfies
[Equation]
− δ ε < 0 with G
= g + u g
[Equation]
we have
[Equation]
where C 1 = ∥ x 0 ∥ 2 + (4 L 2 + L + 2 k 2 u ) (cid:82) 0 − τ ξ ( s ) 2 d s and ξ ∈ C [ − τ, 0] is the initial data.
We can design the neural network structure according to these theoretical results.
Here we only analyse T ε because the energy cost E ( η ε , T ε ) explicitly depends on T ε .
First, the upper bound for E [ η ε ] of NDC implies that the convergence time decrease as the slope of w = w 1 − w 2 near the origin grows due to the fact that w ′ ( 0 ) ≈ w ( ε ) /ε , and the same effect holds for V .
Hence, we can construct w 1 , 2 and V as neural networks with steeper slope at the origin to accelerate the control process, and thus reduce the upper bound of energy cost, for NDC.
Secondly, the time upper bound of NSC is directly relates to the hyperparameter α in the training period, so we can choose α ∗ =
arg min α ( ∥ x α 0 − ε α ∥ ) /α to obtain the optimal NSC controller with the least upper bound of convergence time and energy cost.
We numerically investigate the influence of α in Appendix A.4.1.
S AFETY G UARANTEE FOR SDDE S
In this section, we provide safety guarantee for our NDC and NSC, that is, the SYNC.
We first extend the recent results on stochastic control barrier functions (Clark, 2019) to the SDDE, deriving constraints on the stochastic control.
This guarantees that the process x ( t ; ξ ) satisfies the safety constraint, i.e., x ( t ; ξ ) ∈ int( C ) for all t with the initial value ξ (0) ∈ int( C ) .
The set C is defined by a locally Lipschitzian function h : R d → R as C
= { x : h ( x ) ≥ 0 } .
We summarize the extension results in the following proposition, and include the proof of this proposition in Appendix A.1.5.
Proposition 6.1 Let the function B : R d → R be locally Lipschitz and twice-differentiable on int ( C ) .
If there exists class-K functions α 1 ( x ) , α 2 ( x ) and α 3 ( x ) such that,
[Equation]
where class-K denotes the set of all functions: R + → R + , which are continuous, strictly increasing and vanishing at zero.
Then B ( x ) is called a stochastic control barrier function (SCBF).
Suppose there exists a SCBF for the SDDE (1).
Then for all t, P (cid:0) x ( t ) ∈ int ( C ) (cid:1) = 1 , provided that x (0) ∈ int( C ) .
Baseline With Proposition 6.1 and Theorem 2.2, the traditional deterministic control methods based on the Quadratic Program (QP) in (Fan et al., 2020; Sarkar et al., 2020) can be easily applied to the
SDDE.
We use this QP method as baseline and the specific algorithm is shown in Appendix A.3.1.
We further use the classic MPC method as a baseline control.
Proposed SYNC A natural idea is to integrate Proposition 6.1 into our proposed neural framework, but there are three inequality constraints in this proposition, making the NN constructions complicated and hard to train.
To simplify the construction, we propose the following new Theorem for safety guarantee, which is a significantly novel contribution to the existing barrier function theory.
Theorem 6.2 For stochastic functional differential equation d x ( t ) =
F ( x t , t )d t + G ( x t , t )d B ( t ) , with F, G satisfying locally lipschitz condition and locally linear growth condition, x t ( s ) =
x ( t + s ) for s ∈ [ − τ, 0] , if there exist a number α ∈ (0 , 1) and a class-K function λ ( x ) such that 2 BLB− (1 − α )
∥∇B ⊤ g ∥ 2 ≤ 0 for x ∈ R d , where B ( x ) = 1 λ
( h ( x )) .
Then, the solution of this equation satisfies
P ( x ( t ; ξ ) ∈ int( C ))
= 1 for any ξ ∈ C F 0 ([ − τ, 0]; R d ) with ξ (0) ∈ int( C ) .
We provide the proof of this theorem in Appendix A.1.6.
Now we only need to construct a neural candidate class-K fucntion λ and add the condition in Theorem 6.2 to loss function in our aforementioned framework.
Here we utilize the input convex neural network (Amos et al., 2017a) to insure λ ( x ; θ λ )
=
NN( x ; θ λ ) − NN(0; θ λ ) is a class-K function, and we define the loss function as follows.
Definition 6.1 ( Barrier Loss) Provided with the fixed function h ( x ) and the barrier function B ( x ) =
λ ( h ( x )) , we define the loss function for the controlled system (4) as:
[Equation]
(11) and, still, we use the empirical loss through the Monte Carlo sampling in our training.
A significant difference is that here we pick up the uniform distribution as µ = µ (int( C )) .
6. E XPERIMENTAL C OMPARISON
We investigate the ability of the proposed safe control method to limit the fluctuation in the control process on the task of control noise-perturbed inverted pendulum, which is a standard nonlinear control problem for testing different control methods (Anderson, 1989; Huang & Huang, 2000).
Mathematically, the pendulum has two state variables: the angular position θ , that is the angle deviating from the vertical position x = 0 , and angular velocity ˙ θ .
The external effects on the pendulum can be characterized by a time-delay noise.
Now, we apply the safe method to steer the system to the inverted position x = 0 without rotating a full circle, i.e. | θ | ≤ 2 π .
The results are shown in Figure 5 and the experimental details are provided in Appendix A.3.6.
We observed that the safe control method significantly outperms the baseline and the stochastic control method in terms of stabilization and safety guarantee.
Finally, notice that the Barrier Loss defined in Defintion 6.1 computes the Hessian matrix for the barrier function B ( x ) .
This indicates that we should trade off between the computational cost and the safety for specific tasks.
R ELATED W ORKS
Stability Theory of SDDEs
The early endeavors to develop the stability theory for SDDEs were attributed to (Mao, 1999; 2002) inspired by LaSalle’s theory (LaSalle, 1968).
The subsequent
[Figure]
0.2 0.4 t
Baseline
QP MPC -0.
0.4 tNSC -0.
0.2 0.4 tNSC+Safe
[Figure]
developments have been systematically and fruitfully achieved in the last twenty years in the control community Appleby (2003); Song et al.
; Liu et al.
; Zhu (2018); Peng et al.
These works reveal the positive effect of multiplicative noise to the stochastic dynamics with delays.
These, therefore, motivate us to develop only neural stochastic control to stabilize dynamical systems in this work.
Stabilization Controller Traditional control methods focus on transforming control criteria, such as the control Lyapunov functions (CLFs), into the QP (Fan et al., 2020; Sarkar et al., 2020) or the semi-definite planning (SDP) problems (Henrion & Garulli, 2005; Jarvis-Wloszek et al., 2003; Parrilo, 2000) to find optimal control iteratively.
These methods have high computational complexity since they cannot give the closed form of the control.
Hence, machine-learning based control methods have been introduced to improve the generalization and efficiency of the original convex optimal problems (Khansari-Zadeh & Billard, 2014; Ravanbakhsh & Sankaranarayanan, 2019; Gurriet et al., 2018).
From the Model Predictive Control based learning (Wagener et al., 2019; Williams et al., 2018), to the neural Lyapunov control (Chang et al., 2019), and even to the recently neural stochastic control (Zhang et al., 2022), all these approaches consider dynamics without time-delay.
In comparison, our work focuses much on learning control policies for more general dynamics with both time-delay and stochastic settings.
Theory and Application of Control Barrier Function The barrier function method has been extensively researched in the problem of safety verification of controlled dynamics in both deterministic setting (Prajna & Jadbabaie, 2004; Jankovic, 2018) and stochastic setting (Prajna et al., 2004;
Clark, 2019; 2021).
However, the existing works always require barrier function to satisfies many inequality constraints and do not consider nonlinear systems with time-delay in either drift term or diffusion term.
In comparison, we propose a new provable simplified theorem to achieve safety guarantee for SFDE(SDDE) with only one inequality constraint.
Existing works for constructing barrier functions in application typically based on quadratic programming (Ames et al., 2014; 2016;
Khojasteh et al., 2020; Fan et al., 2020).
Machine learning methods have also be introduced in safe control fields in (Robey et al., 2020;
Dean et al., 2020; Taylor et al., 2020), but these works mainly focus on learning model uncertainty or barrier function instead of control policy.
To the best of our knowledge, our neural safe methods is the first result that simultaneously learn safe control policy and barrier functions from data.
D ISCUSSION
We have proposed SYNC, a novel control family including two frameworks to learn control policies for nonlinear systems with time-delay in stochastic setting with safe guarantee.
To this end, We prove two new theorems on stability and safety criteria for SFDEs that are more concise than the existing theories.
The frameworks significantly simplify the process of control design, and we theoretically and numerically investigate the propose methods in terms of energy cost and convergence time with comparison of the existing methods.
Furthermore, our frameworks can easily generalize to find control policies for SFDEs with the theorems we proved.
Finally, we provide some limitations of the
SYNC in Appendix A.2, which need in-depth investigation in future works.