SYNC: S AFETY -A WARE N EURAL C ONTROL FOR S TABILIZING S TOCHASTIC D ELAY -D IFFERENTIAL E QUATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Stabilization of the systems described by stochastic delay-differential equations is a challenging task in control community.
Here, to achieve this task, we leverage neural networks to learn control policies using the information of the controlled systems in some prescribed regions.
The two learned control policies, the neural deterministic controller (NDC) and the neural stochastic controller (NSC), work effectively because the learning procedures use, respectively, the well-known
LaSalle-Type theorem and the newly-established theorem for guaranteeing the stochastic stability in SDDEs.
We theoretically investigate the performance of the proposed NDC and NSC in terms of convergence time and energy cost.
More practically and significantly, we improve our learned control policies through considering the situation where the controlled trajectories can only evolve in some specific safety set.
Such successful stabilization based on neural networks restricted in safety set is attributed to our further developed theory for safety and stability guarantee of SDDEs using the stochastic control barrier function and spatial discretization, and we name it as SYNC ( S afet Y -aware N eural C ontrol).
The efficacy of all the articulated control policies, including the SYNC, is demonstrated systematically by using representative control problems.
I NTRODUCTION
[Figure]
[Equation]
[Figure]
Stochastic delay-differential equations (SDDEs) (Mao, 1996; Lin & He, 2005; Sun & Cao, 2007;
Guo et al., 2016) have been leveraged to characterize the complex dynamical behavior emergent in real-world systems with dependence on the current state, the past state, and the noise.
Efficiently controlling these systems is a longstanding crucial problem, with the consequent emphasis being placed on the design of control policies and analysis of stability in SDDEs.
Traditional control methods in stochastic settings have been fully developed in the convex optimization frameworks using the control Lyapunov stability theory, e.g. the quadratic programming (QP)
(Fan et al., 2020; Sarkar et al., 2020).
These methods cannot provide the analytical form of feedback controllers and own a high computational cost, requiring solving QP problems at each iteration step.
To overcome these difficulties, utilizing neural networks (NNs) to automatically design controllers becomes one of the mainstream approaches in recent years (Zhang et al., 2022; Chang et al., 2019).
However, existing machine-learning-based methods either focus on controlling systems without time-delay or aim at learning the control Lyapunov function instead of the control policy (Khansari-Zadeh & Billard, 2014), promoting us to design neural controllers for general nonlinear SDDEs.
The safety verification of controlled systems plays an important role in many branches of cybernetics and industry.
For example, with the safety verification, one can reduce a significant economic burden and loss of life (Ames et al., 2016; Wang et al., 2016).
In particular, the dominant framework for safety control in stochastic settings is the use of stochastic control barrier function (SCBF)
(Clark, 2019; 2021; Santoyo et al., 2021).
The core idea of designing a candidate SCBF is that its value tends to explode as the system’s state leaves the safe region, implying a safety guarantee as long as one could design a controller such that the SCBF is always finite within the controlled time duration.
Unfortunately, the existing theories of SCBF either require a lot of inequality constraints or are limited in handling systems without time-delay.
In this paper, we utilize neural networks (NNs) to learn control policies for SDDEs based on the corresponding stability theories.
Additionally, we develop a simplified SCBF theory for SDDEs and then use it to construct the neural controller with safety guarantee, named SYNC.
All these control policies are intuitively depicted in Figure 1.
The major contributions of this paper are listed as follows
• designing a novel and practical framework of neural deterministic control based on the existing LaSalle-Type stability theory, • proposing a simplified stability theorem and designing the second novel neural stochastic control framework that can benefit from noise according to this theorem, • cultivating a SCBF theory for SDDEs and providing analytical results of the safety guarantee and stability guarantee in neural network settings, • providing theoretical estimation for proposed neural controller in terms of convergence time and energy cost based on the safety and stability guarantee, and • demonstrating the efficacy of the proposed neural control methods through numerical comparisons with the typical existing control methods on several representative physical systems.
P RELIMINARIES
To begin with, we consider the SDDE of the following general form:
[Equation]
where x ( t ) = ξ ( t ) ∈ C F 0 ([ − τ, 0]; R d ) is the initial function, the drift term F : R d × R d × R + → R d and the diffusion term G : R d × R d × R + → R d × r are Borel-measurable functions, and B t is a standard r -dimensional
( r -D)
Brownian motion defined on probability space (Ω , F , {F t } t ≥ 0 , P ) with a filtration {F t } t ≥ 0 satisfying the regular conditions.
Without loss of generality, we assume
F ( 0 , 0 , t ) = 0 , G ( 0 , 0 , t ) = 0 to guarantee that x ( t ) ≡ 0 , t ≥ 0 is an equilibrium of Eq.
For simplicity, the following notations and assumptions are used throughout the paper.
Assumption 2.1 Eq.
(1) has a unique solution x ( t, ξ ) on t ≥ 0 for any ξ ∈ C F 0 ([ − τ, 0]; R d ) and for every integer n ≥ 1 , there is a number K n > 0 such that
[Equation]
for any ( x , y , t ) ∈ R d × R d × R + with ∥ x ∥ ∨ ∥ y ∥ ≤ n , [-there is a number K n > 
 such that-]where ∥ · ∥ denotes the L 2 -norm and ∥ · ∥ F denotes the Frobenius norm, i.e. ∥ G ( x , y , t ) ∥
[Equation]
x [Equation] y
[Equation]
Definition 2.1 (Derivative Operator) Define the differential operator L associated with Eq.
(1) by
[Equation]
According to the above definition of the derivative operator, an operation of L on the function
V ∈ C 2 , 1 ( R d × R + ; R ) yields:
[Equation]
Here, V t , ∇ V and H V represent, respectively, the time derivative, the gradient and the Hessian matrix of V .
Notably, the following LaSalle-Type stability theorem will be used in the establishment of our part of the results.
Theorem 2.2 (Mao, 2002) Suppose that Assumptions 2.1 holds.
Assumes there are functions V ∈
C 2 , 1 ( X × R + ; R + ) , γ ∈ L 1 ( R + ; R + ) , and w 1 , w 2 ∈ C ( X ; R + ) such that L V ( x , y , t ) ≤ γ ( t ) − w 1 ( x )+ w 2 ( y ) , w 1 ( x ) ≥ w 2 ( x ) , and lim ∥ x ∥→∞ inf 0 ≤ t ≤∞ V ( x , t ) = ∞ , here, X ⊂ R d is the state space.
Then, Ker( w 1 − w 2 ) ̸ = ∅ and lim t →∞ dist( x ( t, ξ ) , Ker( w 1 − w 2 ))
=
a.s. , where
Ker( w 1 − w 2 ) ≜ { x : w 1 ( x ) − w 2 ( x ) = 0 } , dist( x, K ) ≜ inf y ∈ K
∥ x − y ∥ for a set K ⊆ R d , and a.s. stands for the abbreviation of almost surely.
Problem Statement We assume that the zero solution of the following SDDE:
[Equation]
is unstable, i.e. lim t →∞ x ( t [Equation] ξ ) ̸ = 0 on some set of positive measures.
We aim to stabilize the zero solution with control based on neural networks (NNs).
In other words, our goal is to leverage the
NNs to design an appropriate controller u
= ( u f , u g ) with u f ( 0 , 0 , t ) =
u g ( 0 , 0 , t ) =
such that the controlled system
[Equation]
is stabilized to the zero solution.
We call u f : R d × R d × R + → R d as deterministic control and u g : R d × R d × R + → R d × r as stochastic control since they are integrated with d t and d B t , respectively.
The major difficulty of this problem comes from the non-Markovian property of SDDEs, hence we cannot apply MDP (Markov decision process)-based methods, such as reinforcement learning, to control SDDEs.
Existing works prefer to learning deterministic control and regard the noise as a negative part that may destroy the natural dynamics of f .
In what follows, we not only show that the deterministic control can achieve stabilization, but also that it can be done by elaborately-designed stochastic control, yielding two frameworks, the neural deterministic control (Section 3) and the neural stochastic control (Section 4).
We make all our code and data available at https://anonymous.4open.science/r/SYNC-35E8 .
N EURAL D ETERMINISTIC C ONTROL
In this section, we propose a framework of neural deterministic control (NDC) based on Theorem 2.2 to stabilize system (3) and provide experimental comparisons with traditional control methods.
In addition, we analytically and experimentally investigate that the framework cannot be directly generalized to the stochastic control version.
Such a problem will be addressed in the next section.
3. M ETHOD : L EARNING C ONTROL AND A UXILIARY F UNCTIONS
The core idea of our method is that once we construct the auxiliary functions V, γ, w 1 , w 2 and the neural controller u , resulting in the controlled system (4) satisfying all the conditions assumed in Theorem 2.2, then the solution x ( t ; ξ ) converges to the Ker( w 1 − w 2 ) .
In particular, if we set
Ker( w 1 − w 2 ) = { 0 } , the unstable zero solution of the control-free system (3) can be stabilized.
To this end, we first provide the reasonable constructions of NNs to learn these candidate functions.
Thus, we design the explicit form of loss function in the learning step.
Auxiliary Function
We employ a multi-layer feedforward neural network, denoted by NN ( · ; θ ) , to design all the functions.
To be concrete, θ 1 is the parameter vector of the positive function V ( x , t ; θ 1 ) , and the L 2 term ∥ x ∥ 2 is added to guarantee lim
∥ x ∥→∞ inf 0 ≤ t<
∞ V ( x , t ; θ
[Equation]
= ∞ , that is
[Equation]
In our framework, it requires V ∈ C 2 , 1 ( R d × R + ) .
We therefore choose C 2 activation function of an NN, such as the hyperbolic tangent function, Tanh ( · ) .
We further discuss the influence of the Lterm in Appendix A.1.3.
In order to design an integrable positive function γ ( t ) with an NN, we use an activation function with at most linear growth such as ReLU and multiply an exponential decay factor to the output of the NN, that is
[Equation]
For simplicity, we design w ( x , θ w ) =
NN ( x ; θ w ) 2 as a positive function.
Additionally, we set w 2 = w, w 1 = w + p ( x ) , p ≥ 0 , Ker( p ) = { 0 } .
[Equation]
Deterministic Control Function We first consider the deterministic control, i.e. u = ( u f , 0 ) .
To guarantee the same zero solution of the control-free system (3) and the controlled system (4), the NDC u f : R d × R d × R + → R d should satisfy u f ( 0 , 0 , t ) = 0 .
One feasible way to meet such a condition is to set u f ( x , y , t ) =
NN ( x , y , t ; θ f ) − NN ( 0 , 0 , t ; θ f ) or u f ( x , y , t ) =
diag( x ) NN ( x , y , t ; θ f ) .
Here, diag( x ) is a diagonal matrix with its i -th diagonal element being x i .
Loss Function Once the learned functions V, γ, w 1 , w 2 and u with the coefficient functions, f u ≜ f + u and g , in the controlled system (4), meet all the conditions assumed in Theorem 2.2, the stability of zero solution is naturally assured.
To achieve this, we demand a suitable loss function to evaluate the likelihood that those conditions are satisfied.
It can be seen from our construction that the only condition needed to be satisfied is L V ( x , y , t ) ≤ γ ( t ) − w 1 ( x ) + w 2 ( y ) .
Hence, we define
LaSalle’s loss function for the controlled system (4) as follows.
Definition 3.1 ( LaSalle’s Loss) Consider the above parameterized candidate functions V, γ, w 1 , wand a controller u f for the controlled system (4).
Then, LaSalle’s loss is defined as
[Equation]
where { x i , y i , t i } Ni =1 are sampled from some distribution µ on R d × R d × R + .
In summary, the developed NDC framework is shown in Algorithm 1 in Appendix A.3.1.
Remark 3.1 The proposed NDC framework can be easily applied to the autonomous SDDE d x ( t ) =
f ( x , x ( t − τ ))
d t + g ( x , x ( t − τ ))
d B t .
In particular, one can simply consider the autonomous auxiliary function V and the control function, and set γ ( t ) = 0 .
For sample distribution µ (Ω) , here we select the uniform distribution on a large enough closed region Ω as used in (Han et al., 2016;
Chang et al., 2019), and we provide further analysis about the influence of µ in Appendix A.2.1.
3. N UMERICAL AND A NALYTICAL I NVESTIGATIONS
Comparison with Existing Methods Recent works on controlling time-delayed systems mainly focus on elaborately designing the analytical form of control to satisfy the conditions in the LaSalleType Theorem 2.2 (Lin & He, 2005; Xu et al., 2014), or simultaneously designing control and the
Lyapunov function to satisfy the conditions based on the Lyapunov theory (Yu & Cao, 2007).
It should be noted that all these methods require a delicate design of functions for specific dynamics, and thus are limited in practical application for controlling general time-delayed systems.
However, our neural method leverages neural networks to automatically learn the control policies, and can be easily applied in any kinds of time-delayed systems having stochastic settings.
In Figure 3, we numerically compare the NDC and a baseline, the linear control (LC) proposed in (Lin & He, 2005), on a noised driving-response Chua’s circuit.
Here, Chua’s circuit is a third-order autonomous dynamical system with only one nonlinear element, producing typical chaotic dynamics (Matsumoto, 1984).
In the experiment, we show that the NDC can find the neural control for the response system y
=
( y 1 , y 2 , y 3 ) with the autonomous and even the nonautonomous time-delay noise while the nonautonomous time-delay noise is beyond the concern in (Lin & He, 2005).
The configuration of the experiment is described in Appendix A.3.4.
[Figure]
Failure in Finding Stochastic Control As we can see that the NDC performs well, a natural idea is to utilize the noise part to achieve the stabilization of the
SDDE ( 3).
To explore this idea, we adopt the same
NN of u f , design u g = NN ( x , y , t ; θ g ) , and train its parameters θ g with LaSalle’s loss ( 8).
However, in
Figure 2, we show that the loss cannot converge to zero in controlling a simple 1 -D toy system via the stochastic controller u g : d x ( t ) =
[ x ( t )
+ x ( t − τ )]
d t + [ x ( t − τ ) + u g ( x ( t ) , x ( t − τ ); θ g )]
d B t .
Actually, this phenomena can be analytically explained.
Notice that θ g arises in loss function as a quadratic term l ( θ g ) =
Tr[ u ⊤ g H V u g ] according to
Eq.
(2), the sign of this term depends on the convexity of V , i.e. the maximum eigenvalue’s sign of
H V .
Nevertheless, the positive function V with lim ∥
x ∥→∞ V ( x , t )
= ∞ implies l ( θ g ) ≥
at most time.
Hence, when we minimize l ( θ g ) ≥ 0 in the training procedure, the ideal case l ( θ g ) = 0 is
[Figure]
[-In Appendix A.1.3, the solution of this system is validated to satisfy lim t →∞ x ( t ; ξ ) = 
-]
equivalent to u g = 0 , which means that we are unable to learn a stochastic controller under LaSalle’s loss (8) satisfying the sufficient conditions assumed in Theorem 2.2.
N EURAL S TOCHASTIC C ONTROL
To address the illustrated problems above, we design a new concise framework for fast learning a stochastic controller, called the neural stochastic control (NSC).
To this end, we first provide the following theoretical result on stabilization of general stochastic functional differential equations (SFDEs) with the proof provided in Appendix A.1.4.The major idea here is to construct the stability condition that makes the diffusion term contribute a negative number, contrary to the Theorem 2.2.
Theorem 4.1 Consider the SFDE d x ( t ) =
F ( x t , t )d t + G ( x t , t )d B ( t ) , with F, G being locally Lipschitzian functions, F ( 0 , t ) = 0 , and G ( 0 , t ) = 0 .
For every M > 0 , assume that min ∥ x t (0) ∥
=
M ∥ x t (0)
⊤ G ( x t , t ) ∥ > 0 .
If there exists a number α ∈ (0 , 1) such that ∥
[Equation]
[Equation]
x [Equation] [Equation] 9) for x t ∈ C ([ − τ, 0] , X ) , where x t ( s ) =
x ( t + s ) for s ∈ [ − τ, 0] and X is the state space.
Then, the solution of the SFDE satisfies lim t →∞ x ( t ; ξ ) =
a.s. for any ξ ∈ C F 0 ([ − τ, 0]; R d ) .
Remark 4.2 The SFDE in Theorem 4.1 is formulated in a very general type, including the SDDE d x ( t ) =
F ( x ( t ) , x ( t − τ 1 ) , · · · , x ( t − τ q ) , t )d t + G ( x ( t ) , x ( t − τ 1 ) , · · · , x ( t − τ q ) , t )d B t with τ 1 < τ 2 < · · · < τ q ∈ [0 , τ ] .
This indicates that our framework can be generalized to stabilize the
SDDEs with multiple delays and even more general SFDEs as well.
In light of Theorem 4.1, we establish a more general framework for learning a neural controller of system (4) with the form u
= ( u f , u g ) designed in the same NN architecture as the one used in the NDC framework.
We focus on stochastic control with u f = 0 and provide more control combinations in Appendix A.3.3, whereas the loss function is differently designed as follows.
Definition 4.1 ( Asymptotic Loss)
Utilizing the notations in Definition 3.1 and g u
= g + u g , the loss function for the controlled system (4) with the controller u is defined as:
[Equation]
where θ =
( θ f , θ g ) .
Akin to Definition 3.1, we use the empirical loss function for training.
Here, α is an adjustable parameter, which is related to the convergence rate and the control energy.
We further discuss the design of the asymptotic loss in Appendix A.2.2 and numerically investigate the role of α in Appendix A.4.1.
We summarize the framework in Algorithm 3 in Appendix A.3.1.
And we further compare the computational complexity in Appendix A.3.2.
E XPERIMENTS OF THE C OMBINATION M ETHODS
We compare our neural control methods on a noise-perturbed kinematic bicycle model for car-like vehicles (Rajamani, 2011) in terms of the convergence time and the energy cost, which are two important indexes to measure the quality of a controller (Yan et al., 2012; Li et al., 2017; Sun et al., 2017). [Table]
To quantify the energy cost in the control process, we first denote the stopping time by τ ϵ ≜ inf { t > 0 :
∥ x ( t ) ∥
= ϵ } and then denote the energy cost by E ϵ ≜ E (cid:2)(cid:82) τ ϵ 0 (cid:0) ∥
u f ∥ 2 + ∥
u g ∥ 2 (cid:1) d t (cid:3) .
We approximate this expectation value by the empirical value as [Equation] through the Monte Carlo sampling.
We show the results in Figure 4 and Table 1.
Table 1 includes the training time (Tt), empirical energy cost E 0 .
001 , nearest distance (Nd) between the bicycle and target position, empirical expectation E [ τ 0 .
001 ] of different methods.
We provide more experimental details in Appendix A.3.5.
We can see that the ranking of the comprehensive performance is that NSC is greater than NDC than QP.
This means that we can really benefit from introducing noise as control.
An explanation for this phenomena is that when we regard energy cost as objective function to be minimized, randomness can lead this functional to the shortest path quicker than the deterministic control, just like stochastic gradient descent outperforms the full-batch gradient descent.
We show the NSC can enlarge the region of attraction of 100 -D gene regulatory networks in Appendix A.4.2.
Uncontrollable Fluctuation
Although the neural stochastic method we propose outperforms the control methods including deterministic control, there exists an obvious disadvantage that the method can cause uncontrollable fluctuation due to the stochasticity.
However, we always want to bound this perturbation in practical application owing to physical and engineering restrictions in real world.
We tackle this safety guarantee problem for our methods in Section 5. [Figure]
S AFETY G UARANTEE FOR SDDE S
[Figure]
We check the safety condition on discretiza- tion points with mesh r .
In this section, we study the safety and stability guarantees for the SYNC framework.
Based on the stochastic control barrier functions, we derive the theoretical result of the safety guarantee problem for SDDEs, which guarantees that the process x ( t ; ξ ) satisfies the safety constraint, i.e., x ( t ; ξ ) ∈ int( C ) for all t with the initial value ξ (0) ∈ int( C ) , where C is a compact set and defined by a local
Lipschitz function h : R d → R as C =
{ x : h ( x ) ≥ 0 } , and h is called a stochastic control barrier function (SCBF).
Inspired by (Lechner et al., 2022), we prove that the safety and stability conditions for neural network form functions can be guaranteed through a stricter condition on finite samples.
We include proofs of all the theoretical results in Appendix A.1.
Definition 5.1 A continuous function α : ( − b, + ∞ ) → ( −∞ , + ∞ ) is said to belong to the extended class-K function for some b > 0 if it is strictly increasing and α (0) = 0 .
Baseline We extend the recent results on stochastic control barrier functions in SDEs (Clark, 2019) to the SDDEs and summarize the results in Proposition 5.1.
With this proposition and Theorem 2.2, the traditional deterministic control methods based on the Quadratic Program (QP) in (Fan et al.,
2020; Sarkar et al., 2020) can be easily applied to the SDDEs.
We use this QP method as the baseline and the specific algorithm is shown in Appendix A.3.1.
We further use the classic MPC method as the baseline.
Proposition 5.1 Let the function B : R d → R be locally Lipschitz and twice-differentiable on int( C ) .
If there exists extended class-K functions α 1 ( x ) , α 2 ( x ) and α 3 ( x ) such that, 1 α 1 ( h ( x )) ≤ B ( x ) ≤α 2 ( h ( x )) , and LB ( x , y , t ) ≤ α 3 ( h ( x )) for the SDDE (1).
Then for all t, P (cid:0) x ( t ) ∈ int( C ) (cid:1) = 1 , provided that x (0) ∈ int( C ) .
A natural idea is to integrate Proposition 5.1 into our proposed neural framework, but the main flaw of this proposition is that B ( x ) is unbounded on C and lacks Lipschitz continuity, which makes it impossible to guarantee the expected conditions through verification on finite samples.
To conquer the difficulty, we propose the following theorem for safety guarantee, which is a significantly novel contribution to the existing barrier function theory.
Theorem 5.2 For stochastic functional differential equation d x ( t ) =
F ( x ( t ) , x ( t − τ ))
d t +
G ( x ( t ) , x ( t − τ ))
d B ( t ) , with F, G satisfying locally lipschitz condition and locally linear growth condition, if there exists an extended class-K function λ ( x ) such that L h ≥ − λ ◦ h for x ∈ D , where ◦ represents the function composition, D is compact and C ⊂ D .
Then, the solution of this equation satisfies P ( x ( t ; ξ ) ∈ int( C ))
= 1 for any ξ ∈ C F 0 ([ − τ, 0]; R d ) with ξ (0) ∈ int( C ) .
Discretization and Safety Guarantee.
Based on the Theorem 5.2, we can construct a neural candidate class-K function λ and combine it with the NDC and NSC to learn a safe controller, where the candidate λ needs to satisfy the expected condition in Theorem 5.2.
However, the main difficulty is to guarantee the condition for every point x ∈ D , since we can basically guarantee this condition on finite training data ˜ D with ˜ D being a discretization of D .
However, the following theorem shows that it is sufficient to check a slightly stricter condition on states in ˜ D to establish the safety guarantee on the whole D .
Theorem 5.3 Let M = M ( F, G, h, λ, D ) be the maximum of the Lipschitz constants of L h and λ ◦ h on D .
r denotes the mesh size of ˜ D , implying that for each x ∈ D , there exists ˜ x ∈ ˜ D such that ∥ x − ˜ x
∥ 2 < r .
Suppose there exists some non-negative constant δ ≤ Mr such that
[Equation]
Then λ satisfies the safety condition, i.e. the condition in Theorem 5.2.
Remark 5.4 Here the non-negative δ is regarded as the tolerance error in training stage, that is, we can stop the training until the safety loss is smaller than Mr .
Construct Neural Networks with Bounded Lipschitz Constant.
We can define the loss function for safety as the left hand side in Eq.
However, the Lipschitz constant M depends on the
Lipschitz constants of the neural network functions λ and u ,which may cause the loss function very complex and difficult to train.
To simplify the loss function, we construct the neural networks with bounded Lipschitz constants for λ and u .
Specifically, we add the spectral normalization for neural control function to constrain its Lipschitz constant lower than 1 (Miyato et al., 2018; Yoshida &
Miyato, 2017).
We apply the monotonic neural networks to construct the candidate extended class-K function as λ θ λ ( x ) =
(cid:82) x 0 q θ λ ( s )d s , where the output of the neural network q θ λ ( · ) is definitely positive (Wehenkel & Louppe, 2019).
To constrain the Lipschitz constant of λ θ λ , we modify the integral formula as λ θ λ ( x ) =
(cid:82) x 0 min( q θ λ ( s ) , M λ )
d s , where M λ is a predefined hyperparameter.
Then the Lipschitz constant of λ θ λ is smaller than M λ .
Then we can calculate the Lipschitz constant
M from the considered functions and M λ .
Other Lipschitz regularization methods can be applied in our framework (Gouk et al., 2021;
Liu et al., 2022) as well.
We define the loss function for safety guarantee of the controlled system (4) as follows,
[Equation]
and we can break the training process once L ˜ D ,M λ
( θ , θ λ ) is less than Mr .
[Figure]
c c
[Figure]
From Safety Guarantee to Stability Guarantee.
Akin to the safety guarantee, we provide the stability guarantee for candidate neural control functions satisfying the condition in Theorems 2.
and 4.1.
The main difficulty is that both two theorems expect their conditions holds for each point x ∈ X ⊂ R d , but we can neither obtain a finite discretization nor a bounded Lipschitz constant on unbounded X .
Ingeniously, this difficulty can be conquered with the help of safety guarantee since the safety condition restrict X ⊂ D and D is compact.
We can obtain the similar theoretical results for stability guarantee of NDC and NSC as that in Theorem 5.3 and we summarized them in
Appendix A.1.8.
We investigate the ability of the proposed safe control method to limit the fluctuation in the control process on the task of control noise-perturbed inverted pendulum with time-delay, which is a standard nonlinear control problem for testing different control methods (Anderson, 1989; Huang & Huang, 2000).
We apply the safe method to steer the system to the upright position without rotating a semi-circle, i.e. | θ | ≤ π .
The results are shown in Figure 6 and the experimental details are provided in Appendix A.3.6.
It is observed that the safe control method significantly outperms the baseline and the stochastic control method in terms of stabilization and safety guarantee.
T HEORETICAL R ESULTS FOR NDC AND NSC
We have mentioned the stopping time and the energy cost in section 4.1 and numerically compare the proposed neural controllers with these indexes.
These two indexes are the classic factors to measure the performance of the controller (Sun et al., 2017).
From the construction in Section 5, we can bound the Lipschitz constant k u of control function. Based on the safety and stability guarantee, the neural controller can satisfy the conditions in the Theorem 2.2 and Theorem 4.1.
Then we have the following two theorems and we provide the proofs in Appendix A.1.9,A.1.10.
Theorem 6.1 ( Estimation for NDC) Consider the SDDE with NDC controller as [Equation] [Equation] )d
[Equation]
, x ( [Equation] ) = x 0 ∈ R d , where ∥ f ( x , y ) − f (¯ x , ¯ y ) ∥∨∥ u f ( x , y ) − u f
(¯ x , ¯ y ) ∥ ≤ L ( ∥ x − ¯ x ∥ + ∥ y − ¯ y ∥ ) .
Assume the controlled system satisfies the similar conditions in Theorem 2.2 and Remark 3.1 with Ker ( w 1 − w 2 )
= 0 .
Denote stopping time as η ε = inf { t > 0 : ∥ x ( t ) ∥ = ε } , the corresponding energy cost in the control process with ϵ <
∥ x 0 ∥ as E ( η ε , T ) =
E [ (cid:82)
η ε ∧ T∥ u ( x ( s ) , x ( s − τ ))
∥ 2 d s ] and under the same notations in Theorem 2.2, we have
[Equation]
{+C2( L 2 + L + k u ) (cid:2) exp (cid:0) 4( L 2 + L + k u ) T ε (cid:1) − 1 (cid:3) + (cid:90) 
 − τ k 2 u ξ 2 ( s )d s.+} where C 0 =
∥ x 0 ∥ 2 + (2 L 2 + L + k u ) (cid:82) 0 − τ ξ ( s ) 2 d s and ξ ∈ C [ − τ, 0] is the initial data.
Theorem 6.2 ( Estimation for NSC) Consider the SDDE with NSC controller as
[Equation]
where f, g are the same as those in Theorem 5.2.
Assume the controlled system satisfies the similar conditions in Theorem 4.1, under the same notations in Theorem 4.1, if the term in (9) further satisfies max
[Equation]
x t [Equation]
[Equation] ∥ x t [Equation] 0) ∥
[Equation]
4 ( ∥ x t (0) ∥ 2 (2 ⟨ x t (0) , f ( x t ) ⟩ + ∥ G ( x t ) ∥ 2F ) − (2 − α ) ∥ x t (0) ⊤ G ( x t ) ∥ 2 ) = − δ ε < 0 with G = g + u g , we have  
[Equation]
where C 1 = ∥ x 0 ∥ 2 + (4 L 2 + L + 2 k 2 u ) (cid:82) 0 − τ ξ ( s ) 2 d s and ξ ∈ C [ − τ, 0] is the initial data.
We can design the neural network structure according to these theoretical results.
Here we only analyse T ε because the energy cost E ( η ε , T ε ) explicitly depends on T ε .
First, the upper bound for E [ η ε ] of NDC implies that the convergence time decrease as the slope of w = w 1 − w 2 near the origin grows due to the fact that w ′ ( 0 ) ≈ w ( ε ) /ε , and the same effect holds for V .
Hence, we can construct w 1 , 2 and V as neural networks with steeper slope at the origin to accelerate the control process, and thus reduce the upper bound of energy cost, for NDC.
Secondly, the time upper bound of NSC is directly relates to the hyperparameter α in the training period, so we can choose α ∗ =
arg min α ( ∥ x α 0 − ε α ∥ ) /α to obtain the optimal NSC controller with the least upper bound of convergence time and energy cost.
We numerically investigate the influence of α in Appendix A.4.1.
R ELATED W ORKS
Stability Theory of SDDEs.
The early endeavors to develop the stability theory for SDDEs were attributed to (Mao, 1999; 2002) inspired by LaSalle’s theory (LaSalle, 1968).
The subsequent developments have been systematically and fruitfully achieved in the last twenty years in the control community Appleby (2003); Song et al.
; Liu et al.
; Zhu (2018); Peng et al.
These works reveal the positive effect of multiplicative noise to the stochastic dynamics with delays, and motivate us to develop only neural stochastic control to stabilize dynamical systems . Finding Stabilization Controller.
Traditional control methods focus on transforming control criteria, such as the control Lyapunov functions (CLFs), into the QP (Fan et al., 2020; Sarkar et al., 2020) or the semi-definite planning (SDP) problems (Henrion & Garulli, 2005; Jarvis-Wloszek et al., 2003; Parrilo, 2000) to find optimal control iteratively.
These methods have high computational complexity since they cannot give the closed form of the control.
Hence, machine-learning based control methods have been introduced to improve the generalization and efficiency of the original convex optimal problems (Khansari-Zadeh & Billard, 2014; Ravanbakhsh & Sankaranarayanan, 2019; Gurriet et al., 2018).
However, all the existing learning methods consider dynamics without time-delay (Wagener et al., 2019; Williams et al., 2018; Chang et al., 2019; Zhang et al., 2022).
In comparison, our work focuses much on learning control policies for more general dynamics with both time-delay and stochastic settings.
The barrier function method has been extensively researched in the problem of safety verification of controlled dynamics (Prajna & Jadbabaie, 2004; Jankovic, 2018;
Prajna et al., 2004; Clark, 2019; 2021).
Existing works for constructing barrier functions in application typically based on quadratic programming (Ames et al., 2014; 2016;
Khojasteh et al., 2020; Fan et al., 2020).
Machine learning methods have also be introduced in safe control fields in (Robey et al., 2020;
Dean et al., 2020; Taylor et al., 2020).
D ISCUSSION
We have proposed SYNC, a novel control family including two frameworks to learn control policies for nonlinear systems with time-delay in stochastic setting with safe guarantee.
To this end, We prove two new theorems on stability and safety criteria for SDDEs that are more concise than the existing theories.
We provide analytical results to show that we can obtain safety and stability guarantee for the neural controllers.
The frameworks significantly simplify the process of control design, and we theoretically and numerically investigate the propose methods in terms of energy cost and convergence time with comparison of the existing methods.
Furthermore, our frameworks can easily generalize to find control policies for SFDEs with the theorems we proved.
Finally, we provide some limitations of the SYNC in Appendix A.2, which need in-depth investigation in future works.