Neural Stochastic Control
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Control problems are always challenging since they arise from the real-world systems where stochasticity and randomness are of ubiquitous presence.
This naturally and urgently calls for developing efﬁcient neural control policies for stabilizing not only the deterministic equations but the stochastic systems as well.
Here, in order to meet this paramount call, we propose two types of controllers,viz., the exponential stabilizer (ES) based on the stochastic Lyapunov theory andthe asymptotic stabilizer (AS) based on the stochastic asymptotic stability theory.
The ES can render the controlled systems exponentially convergent but it requiresa long computational time; conversely, the AS makes the training much faster butit can only assure the asymptotic (not the exponential) attractiveness of the controltargets.
These two stochastic controllers thus are complementary in applications.
We also investigate rigorously the linear control in both convergence time and energy cost and numerically compare it with the proposed controllers in these terms.
More signiﬁcantly, we use several representative physical systems to illustrate theusefulness of the proposed controllers in stabilization of dynamical systems
Introduction
[Figure]
Asympyotic Stabilizer
ICNN or Quadratic form
FNN
Slow
Fast
High
Low
C o m p u t a t i o n a l c o s t
L o n g t
i m e
Controlledorbit
Orbit
FNN =
( , ( ), ( ), , ) =
( , ( ), ( ), ) e t a r e c n e g r e v n o C ( ) =
( ) + ( ) ( )
= ( ) + ( ( ) + ( ))
[Figure]
Original [Figure]
Controlled system 1 x 2 x 3 x 1 d x − d x . . .
∗ ∗ ∗ with fully connected feedforward NN (FNN).
In the ﬁeld of controlling dynamical systems,one of the major missions is to ﬁnd efﬁcientcontrol policies for stabilizing ordinary differential equations (ODEs) to targeted equilibriums.
The policies for stabilizing linear or polynomial dynamical systems have been fully developed using the standard Lyapunov stability theory, e.g., the linear quadratic regulator(LQR) Khalil (2002) and the sum-of-squares(SOS) polynomials through the semi-deﬁniteplanning (SDP)
Parrilo (2000).
As for stabilizing more general and nonlinear dynamicalsystems, linearization technique around the targeted states is often utilized and thus the existing control policies are effective in the vicinityof the targeted states (Sastry & Isidori, 1989)but likely lose efﬁcacy in the region far awayfrom those states.
Moreover, in real applications, the explicit forms of the controlled nonlinear systems are often partially or completely unknown, so it is very difﬁcult to directly design controllersonly using the Lyapunov stability theory.
To overcome these difﬁculties, designing the controllers via training neural networks (NNs) become one of the mainstream approaches in the community ofcybernetics (Polycarpou, 1996).
Recent outstanding developments using NNs include enlarging thesafe region (Richards et al., 2018), learning the stable dynamics (Takeishi & Kawahara, 2021), andconstructing the Lyapunov function and the control function simultaneously (Chang et al., 2019).
In Kolter & Manek (2019) a projected NN has been constructed to directely learn a stable dynamical system and ﬁts the observed time series data well, but it did not focus on learning a controlpolicy to stabilize the original dynamics.
All these existing developments are formulated only fordeterministic systems but inapplicable directly to the dynamical systems described by stochasticdifferential equations (SDEs), requiring us to include the stochasticity appropriately into the use ofneural controls to different types of dynamical systems.
The stability theory for stochastic systems has been systematically developed in the past severaldecades.
Representative contributions in the literature include the Lyapunov-like stability theoryfor SDEs Mao (2007), stabilization of unstable states in ODEs only using noise perturbations Mao(1994b), and the stability induced by randomly switching structures Guo et al.
Generally, forany SDEs governed by d x = f ( x )d t + g ( x )d B t , control policies as u
= ( u f , u g ) are introduced,which transforms the original equations into the controlled system d x = [ f ( x )+ u f ( x )]
d t +[ g ( x )+u g ( x )]
d B t .
Appropriate forms of control policies are able to steer the controlled system to theequilibriums that are unstable in the original SDEs.
Traditional control methods focus on designingdeterministic control u f and regard noise as negative part.
Innovatively, we treat noise as a beneﬁcialpart and design stochastic control u g to achieve the stabilization.
In this article, we articulate two frameworks of neural stochastic control which can complementeach other in terms of convergence rate and the computational time of training NNs.
Additionally,we analytically investigate the convergence time and the energy cost for the classic linear controland numerically compare them to the proposed neural stochastic control.
The major contributionsof this article are multi-folded, including:
• designing two frameworks of neural stochastic control, viz., the ES and the AS, and presenting their advantages in the stochastic control,• computing the convergence time and the energy cost of particular stochastic neural control,• demonstrating the efﬁcacy of the proposed stochastic neural control in important controlproblems arising from representative physical systems, and we make our code available at
https://anonymous. 4open
1.1 Related Works
Lyapunov Method in Machine Learning
The recent work Chang et al.
(2019) proposed an
NN framework of learning the Lyapunov function and the linear control function simultaneouslyfor stabilizing ODEs.
In comparison, we select several speciﬁc types of NNs which have typicalproperties of the Lyapunov function.
For instance, we use the input convex neural network (ICNN)
Amos et al.
(2017), constructing a positively deﬁnite convex function as a neural Lyapunov function(Kolter & Manek, 2019; Takeishi & Kawahara, 2021),
and we construct the NN in a quadratic form(Richards et al., 2018; Gallieri et al., 2019) for linear or sublinear systems where the SDP method isoften used to ﬁnd the SOS-type Lyapunov function (Henrion & Garulli, 2005; Jarvis-Wloszek et al.
,2003; Parrilo, 2000).
Stochastic Stability Theory of SDEs Stochastic stability theory for SDEs have been systematically and fruitfully achieved in the past several decades (Kushner, 1967; Arnold, 2007; Mao, 1991,1994a).
The positive effects of stochasticity have also been cultivated in control ﬁelds (Mao et al.
Deng et al., 2008; Caraballo et al., Appleby et al., Mao et al., 2007).
These, therefore, motivate us to develop only neural stochastic control to stabilize different sorts of dynamicalsystems in this article.
More stochastic stability theory for different kinds of systems are included in
Preliminaries
To begin with, we consider the SDE which is written in a general form as:
[Equation]
where F : R d → R d is the drift function, and G : R d → R d × r is the diffusion function with R d × r ,a space of d × r matrices with real entries, and B t ∈ R r , a r -dimensional
( r -D) Brownian motion.
Without loss of generality, we set F ( 0 ) = 0 and G ( 0 ) =
so that x 0 = 0 is a zero solution Eq.
Notations.
Denote by ∥·∥ the L 2 -norm for any given vector in R d .
Denote by |·| the absolute valueof a scalar number or the modulus length of a complex number number.
For A = ( a ij ) , a matrix ofdimension d × r , denote by ∥ A ∥ 2F =
∑ di =1 ∑ rj =1 a 2 ij the Frobenius norm.
Assumption 2.1 (Locally Lipschitzian Continuity)
For every integer n ≥ 1 , there is a number K n >0 such that
[Equation]
for any x , y ∈ R d with ∥ x ∥ ∨ ∥ y ∥ ≤ n .
Deﬁnition 2.1 (Derivative Operator) Deﬁne the differential operator L associated with Eq.
(1) by
[Equation]
Deﬁnition 2.2 (Exponential Stability)
The zero solution of Eq.
(1) is said to be almost surely exponentially stable, if lim sup t →∞ 1 t log ∥ x ( t ; x 0 )
∥ < 0 a.s. for all x 0 ∈ R d .
Here and throughout,a.s. stands for the abbreviation of almost surely.
Then, the following Lyapunov stability theorem will be used in the establishment of our main results.
Theorem 2.2 Mao (2007) Suppose that Assumptions 2.1 holds.
Suppose further that there exist afunction V ∈ C 2 ( R d ; R + ) with V ( 0 ) = 0 , constants p > 0 , c 1 > 0 , c 2 ∈ R and c 3 ≥ 0 , such that(i) c 1 ∥ x ∥ p ≤ V ( x ) , (ii) L V ( x ) ≤ c 2 V ( x ) , and (iii) |∇ V ⊤ ( x ) G ( x ) | 2 ≥ c 3 V 2 ( x ) for all x ̸ = 0and t ≥ 0 .
Then,
[Equation]
In particular, if c 3 − 2 c 2 > 0 , the zero solution of Eq.
(1) is exponentially stable almost surely.
The following asymptotic theorem also will be used in the establishment of our main results.
Theorem 2.3 Appleby et al.
Suppose that Assumptions 2.1 holds.
Suppose furthermin ∥ x ∥ =
M ∥ x ⊤ G ( x ) ∥ > 0 for any M > 0 and there exists a number α ∈ (0 , 1) such that
[Equation]
Then, the unique and global solution of Eq.
(1) satisﬁes lim t →∞ x ( t, x 0 ) =
a.s. , and we call thisproperty as asymptotic attractiveness.
Designing Stable Stochastic Controller
Here, we assume that the zero solution of the following SDE:
[Equation]
is unstable.
Note that, for any nontrivial targeted equilibrium x ∗ , a direct transformation y = x
− x ∗can make the zero solution as the equilibrium of the transformed system.
Thus, our mission is tostabilize the zero solution only.
As such, we are to use the NNs to design the control u : R d → R d × rwith u ( 0 )
=
and apply it to Eq.
(4) as
[Equation]
Since u is integrated with d B t in the controlled system (5), we regard it as a stochastic controller.
Inwhat follows, two frameworks of neural stochastic control, the exponential stabilizer (ES) and theasymptotic stabilizer (AS), are articulated, respectively, in Sections 3.1 and 3.2.
All these controlpolicies are intuitively depicted in Figure 1.
3.1 Exponential Stabilizer
Once we ﬁnd the Lyapunov function V and the neural controller u , making the controlled system (5)meet all the conditions assumed in Theorem 2.2, the equilibrium 0 can be exponentially stabilized.
To this end, we ﬁrst provide two different types of functions for constructing V , which actuallycould be complementary in applications.
Then, we design the explicit forms of control function andloss function.
ICNN V Function.
We use the ICNN (Amos et al., 2017) to represent the candidate Lyapunovfunction V .
This guarantees V as a convex function with respect to the input x .
In order to furthermake V as a true Lyapunov function, we use the following form:
[Equation]
as introduced in Manek & Kolter (2020).
Here, W i , b i are real-valued weights, U i are positiveweights, σ i are convex, monotonically non-decreasing activation functions in the i -th layer, ε isa small positive constant, and F is a continuously differentiable and invertible function.
In ourframework, we require V ∈ C 2 ( R d ; R + ) according to Deﬁnition 2.1; however, each activationfunction σ i ≡ σ in Manek & Kolter (2020) is C 1 only.
Thus, we modify the original function as:
[Figure]
[Equation]
otherwise
[Equation]
which not only approximates the typical ReLU activation but also becomes continuously differentiable to the second order (see Figure 2).
Quadratic V Function.
For any x ∈ R d , let V θ ∈ R
d be a multilayered feedforward NN of theinput x and with tanh( · ) as the activation functions, where θ is the parameter vector.
To meet thecondition used in Deﬁnition 2.1, we cannot use the ReLU , a non-smooth function, as the activationfunction.
Hence, we use the candidate Lyapunov function as:
[Equation]
which was introduced in Gallieri et al.
Here, ε is a small positive constant.
Control Function.
We introduce a multi-layer feedforward NN (FNN), denoted by NN ( x ) ∈ R r ,to design the controller u .
Since we require u ( 0 ) = 0 , we set u ( x ) ≜ NN ( x ) − NN ( 0 ) or u ( x ) ≜diag( x ) NN ( x ) with r = d .
Here, diag( x ) is a diagonal matrix with its i -th diagonal element as x i . [Figure]
Remark 3.1 As reported in Changet al.
(2019), a single-layer NN without the bias constants in its arguments,which degenerates as linear control,could sufﬁciently take effect in the stabilization of many deterministic systems.
However, this is NOT always the casefor achieving the stabilization of highlynonlinear systems or even SDEs.
The following proposition with Figure 3 provides an example,where neither the classic linear controller nor the stochastic linear controller can stabilize the unstable equilibrium in a particular SDE.
The proof of this proposition is included in Appendix A.1.
Proposition 3.2 Consider the following 1 -D SDE:
[Equation]
[Equation] for r = 0 to m − 1 do Compute ∇ V i ) , HV ( x i ) , i = 1 , · · · , n Compute ES loss: L ( θ r , u from
[Equation]
▷ Update parameters if L ( θ r +1 , u r +1 ) ≤ δ then break with a zero solution x ∗ = 0 .
Then, for u ( x ) = kx with any k and x 0 ̸ = 0 , x ∗ = 0 is neither exponentially stable nor of globally asymptotic attractiveness almost surely.
For u ( x ) = 2 x 2 , x ∗ = 0 isof globally asymptotic attractiveness.
For u ( x ) ≡ 0 , the deterministic system cannot be stabilizedby any classic linear controller.
Loss Function.
When the learning procedure updates the parameters in the NNs such that theconstructed V and u with the coefﬁcient functions, f and g u ≜ g + u , in the controlled system (5)meet all the conditions assumed in Theorem 2.2, the exponential stability of the controlled system isassured.
Thus, we demand a suitable loss function to evaluate the likelihood that those conditions aresatisﬁed.
First, from Theorem 2.2, it follows that V ( x ) ≥ ε ∥ x ∥ 2 for all x ∈ R d .
Thus, Conditions(ii)-(iii) together with c 3 − 2 c 2 > 0 in Theorem {+2.
+} equivalently becomeinf x ̸ = ( ∇ V ( x ) ⊤ g u ( x )) V ( x )
[Equation]
These conditions further imply that
[Equation]
With these reduced conditions, we design the ES loss function for the controlled system (5) asfollows.
Deﬁnition 3.1 ( ES loss) Consider a candidate Lyapunov function V and a controller u for the controlled system (5).
Then, the ES loss is deﬁned as
[Equation]
where the state variable x obeys the distribution µ .
In practice, we consider the following empiricalloss function:
[Equation]
where { x i } Ni =1 are sampled from the distribution µ = µ (Ω) and Ω is some closed domain in R d .
For convenience, we summarize the developed framework in Algorithm 1.
Here, b is a hyperparameter that can be adjusted as required by solving a speciﬁc problem.
Remark 3.
In Section 5, we show numerically that the conditions reduced in (11) are sufﬁcientlyeffective for designing the ES loss.
Actually, it is not necessary to design the loss function using theconditions in (10) .
Now, for controlling any nonlinear ODEs or SDEs, we design the ES according to Algorithm 1.
Assuch, using the ES framework can not only stabilize those unstable equilibriums (constant states) Input: Data { x i } ni =1 sampled from µ (Ω) , parameter α ∈ (0 , 1) used in (13), and all other parameters, m , γ , δ , f , and g , deﬁned in the same manner as those in Algorithm 1. Output: Controller u ( x i ) . for r = 0 to m − 1 do
Compute loss function: L ( u r ) from (13) u r +1 = u r − γ · ∇ u L ( u r ) ▷ Update parameters if L ( u r +1 ) ≤ δ then break of the given systems, but also can stabilize those unstable oscillators, e.g., the limit cycle. Thisis because the solution corresponding to the oscillator can be regarded as a zero solution of thecontrolled system after appropriate transformations are implemented.
Another point needs attention.
During the construction of V in (6), ε ∥ x ∥ 2 , the L 2 -regularization, isused to guarantee the positive deﬁniteness of V .
However, often in the application of the Lyapunovstability theory, the form of ∥ x ∥ 2 is not always a suitable candidate for the Lyapunov function.
Itmay restrict the generalizability of using our framework, so it needs necessary adjustments.
Thefollowing example illustrates this point.
Example 3.4 Consider a 2 -D SDE as follows:
[Equation]
In Appendix A.2, the zero solution of this system is validated to be exponentially stable almost surely;however, k ∥ x ∥ 2 for any k ∈ R cannot be a useful auxiliary function to identify the exact stability ofthe zero solution.
To be candid, using the current framework takes a longer time for training and constructing the neural
Lyapunov function.
In the next subsection, we thus establish an alternative control framework thatcan reduce the training time.
3.2 Asymptotic Stabilizer
Here, in light of Theorem 2.3, we are to establish the second framework, the AS, for stabilizingthe unstable equilibrium of system (5).
This framework only makes the equilibrium asymptoticallyattractive almost surely.
Its control function is designed in the same way as the one used in the ESframework, whereas the loss function is differently designed.
Deﬁnition 3.2 ( AS loss) Utilization of the notations used in Deﬁnition 3.1, the loss function for thecontrolled system (5) with the controller u is deﬁned as:
[Equation]
Akin to Deﬁnition 3.1, we set the empirical loss function as:
[Equation]
Here, α is an adjustable parameter, which is related to the convergence time and the energy cost usingthe controller u .
We show in Appendix A.6 the inﬂuence of selecting different α .
For convenience,we summarize the AS framework in Algorithm 2.
Convergence Time and Energy Cost
The convergence time and the energy cost are the crucial factors to measure the quality of a controller(Yan et al., 2012;
Li et al., 2017; Sun et al., 2017).
In this section, we provide a comparative study between the traditional stochastic linear control and the AS, the above-articulated neural stochasticcontrol.
To this end, we ﬁrst present a theorem on the estimations of the convergence time and the energycost for the stochastic linear control on a general SDE.
Theorem 4.1 Consider the SDE with a stochastic linear controller as:
[Equation]
where ⟨ x , f ( x ) ⟩ ≤ L ∥ x ∥ 2 and u ( x ) =
k x with | k | > √ 2 L .
Then, for ϵ < ∥ x 0 ∥ , we have
[Equation]
where, for a sufﬁciently small ϵ > 0 , we denote the stopping time by τ ϵ ≜ inf { t > 0 :
∥ x ( t ) ∥
= ϵ }and denote the energy cost by
[Equation]
The proof of this theorem is provided in Appendix A.3.
Now, we numerically compare the performances of the linear controller u ( x ) =
kx and the ASon the convergence time and the energy cost of the controls applied to system (14) with speciﬁcconﬁgurations (see Figure 4).
We numerically ﬁnd that u ( x ) = kx can efﬁciently stabilize theequilibrium for k > k c = 5 . 6 .
Without loss of generality, we ﬁx k = 6 . 0 , and compare thecorresponding performances.
As clearly shown in Figure 4, the AS outperforms u ( x ) = kx fromboth perspectives, the convergence speed and the energy cost.
In the simulations, the energy cost 227 E ( τ ϵ , T ϵ ) deﬁned above is computed in a ﬁnite-time duration as E ( τ ϵ , T ∧ T ϵ ) , where T < ∞ isselected to be appropriately large.
We leave more results of the comparison study in Appendix A.4.5.
[Figure]
Experiments
In this section, we demonstrate the efﬁcacy of the above-articulated frameworks of stochastic neuralcontrol, the ES and the AS, on several representative physical systems.
We also compare thesetwo frameworks, highlighting their advantages and weaknesses.
The detailed conﬁgurations forthese experiments are included in Appendix A.4.
Additional illustrative experiments are included in
Appendix A.5.
5.1 Harmonic Linear Oscillator
0 1 . 0 2 . 0 3 . 0 −  x 1 x 1 .2 .3 .
Time −  x
ES+ICNNES+QuadAS 2 .
8 3 . 0 − 0 .
0 .x 1 .2 .3 .
Time −  x ES+ICNNES+QuadAS 2 . 8 3 . 0 − 0 . 0 .x [Figure]
First, consider the harmonic linear oscillator¨ y + 2 β ˙
y + w 2 y = 0 , where w is the natural frequency and β > 0 is the damping coefﬁcient representing the strength of the external force on the vibrator (Dekker, 1981).
Although this system is exponentially stable, thesystem with stochastic perturbations ¨ y + (2 β +ξ 2 ) ˙
y
+( w 2 + ξ 1 ) y
= 0 becomes unstable even if 245 E [ ξ 1 ( t )]
=
E [ ξ 2 ( t )] = {+
+}
(Arnold et al., 1983).
Now, we apply the nonlinear ES(+ICNN), thelinear ES(+Quadratic), and the nonlinear AS,respectively, to stabilizing this unstable dynamics (A.4.1) with w 2 = 1 , β = 0 .
5 , ζ 1 = − 3 ,and ζ 2 = 2 . 15 , the results are shown in Figure 5.
Indeed, we ﬁnd that the two nonlinear stochasticneural controls are more robust than the linear control, and that the ES(+ICNN), rather than the AS,makes the controlled system more stable. Tt Ni Di
Ct ES(+ICNN) 276 . 385 s 1e0 . 459 ES(+Quadratic) 78 .
071 s 107 0 .3 . 683 AS 4 . 839 s 0 .2 . In Table 1 we compute the training time (Tt)used for the loss function converging to 0 , thenumber of iterations (Ni), the distance (Di) between the trajectory and the targeted equilibrium at time T = 4 , and the convergence time(Ct) when the distance between the trajectoryand the targeted equilibrium is less than 0 . 05 .
The results are obtained through averaging the corresponding quantities produced by 20 randomlysampled trajectories and the detailed training conﬁgurations are shown in Appendix A.4.1. Inverted Pendulum
and we compare the energy cost between linear control u ( x ) =
kx
and our Asymptotic Stabilizer learning control in Figure 4.
We put more details and experiments in the appendix
[Figure]
with ME = 1375 , which is significantly lower than linear control. 5. Experiments In this section we consider some problems with actual physical background that are not all direct equation stability problems, but can be transformed into equivalent forms of finding the stable control of the corresponding equations, and some recent articles have partially solved these problems. So it can be used as a benchmark to compare with our method with respect to convergence speed and computation time.
We refer to Appendix C for implementation details and to Appendix D for additional experiments. 5.1. Harmonic Linear Oscillator First, we consider harmonic linear oscillator ¨ y + 2 β ˙ y
+ w 2 y = 0 , where w is the natural frequency of oscillator, β > 0 is the damping coefficient representing the strength of external force on the vibrator. Although the original system is exponentially stable, the perturbation system with random damping and random restoring force [Equation] becomes unstable even if E ξ 1 ( t ) =
{+E ξ [Equation] ) = 
 (Arnold et al., 1983).+}
Now we use our algorithms to stabilize this unstable random dynamics for w 2 = 1 ,β = 0 . 5 ,k 1 = − 3 ,k 2 = 2 .
and we plot the results in Figure [Figure]
5.2. Inverted Pendulum
The inverted pendulum is a classic nonlinear control problem governed by the differential equation [Equation] [Equation] where m is the pendulum mass, g is the gravitational accel- eration, l is the pole length, and β is the friction coefficient.
This system has two
[Figure]
˙ θ
[Figure]
π
[Figure]
− π
[Figure]
[Figure]
θ
[Figure]
We hope that the inverted pendulum can remain upside down, but this position corresponds to the unstable equilibrium point x = 0 of the system.
We illustrate this phenomenon in Figure 6.
We use the Asymptotic Stabilizer in
Algorithm 2 to stabilize the system (28) and show the results in Figure 6. [Figure] Controlling the inverted pendulum is one of theclassic nonlinear control problems (Anderson,1989; Huang & Huang, 2000), which is governed by ml 2 ¨ θ = mgl sin θ − β ˙ θ.
Here, mis the mass of the pendulum, g is the gravitational acceleration, l is the pole length, andβ is the friction coefﬁcient.
Mathematically,the pendulum can be written as a system withtwo state variables: θ , the angle deviating fromthe vertical position x = 0 , and ˙ θ , the angular velocity.
Denote the 2 -D state variable byx =
( θ, ˙ θ ) .
Then, we apply the AS articulatedin Algorithm 2 to steer the system to the equilibrium x = 0 (sustaining the pendulum inverted).
The schematic diagram of the inverted pendulum system and successful control results are shownin Figure 6.
The sampled trajectories are initiated from different angles and angular velocities [i.e.,( θ (0) , ˙ θ (0))
=
( − 5 , 5) , ( − 1 , 3) , (3 , − 4) ].
5.3 Stuart-Landau Equations
Finally, we show that our frameworks are beneﬁcial to realizing the control and the synchronizationof complex networks.
To this end, we consider the single StuartLandau oscillator which is governedby the following complex-valued ODE:
[Equation]
This equation is a paradigmatic model undergoing the so-called Andronov bifurcation (Kuznetsov,2013): Stability of the equilibrium changes and the limit cycle emerges as the parameter passessome critical value.
In what follows, we consider two cases based on system (15).
[Figure]
Case 1 We set β = − 25 , γ = 1 , andµ = 1 , so that system (15) has a stable equilibrium ρ = 0 and an unstable limit cycle ρ = 5 .
Here, Z = x + i y
= ρe i θ .
Now, the ASsteers the dynamics to the unstable limit cycle, as successfully shown in Figure 7.
Thetrajectories (the left column) and the phase orbits (the right column) of system (15), initiatedfrom 30 randomly-selected initial states, without control (the upper panels) and with controlof the AS (the lower panels). The initial valuesinside (resp., outside) the limit cycle ρ = 5 areindicated by the blue (resp., purple) pentagrams. [Figure] W i t h C o n t r o l x   3 Time −  y − 1 x −  y
Case 2 Next, we consider the synchronization problem of the coupled Stuart-Landauequations.
Successful deterministic methodshave been systematically developed for realizing synchronization, including the adaptivecontrol with time delay Selivanov et al.
(2012)and the open-loop temporal network controller Zhang & Strogatz (2021).
These methods majorly depend on the technique of linearizationin the vicinity of the synchronization manifold.
Here, we show how to apply our frameworkto achieving the synchronization in the coupledsystem.
We set the corresponding Laplace matrix L =
( L jk ) n × n as ∑ nk =1 L jk = 0 , which guarantees the synchronization manifold is aninvariant manifold of the coupled system (Pecora & Carroll, 1998).
Speciﬁcally, we select asn = 20 , σ = 0 .
01 , c 1 = − 1 . 8 , c 2 = 4 , and L jk =
δ jk − 1 n , where δ jk is the Kronecker function.
Then, we apply the AS to this system and realize the stabilization of the synchronous manifold, asshown in Figure 8.
In addition to the above controlled systems, we include the other illustrative examples, the controlledreservoir computing and the controlled Lorenz system, in Appendix A.5.
Conclusion and Future Works
In this article, we have proposed two frameworks of neural stochastic control for stabilizing differenttypes of dynamical systems, including the SDEs.
We have shown that the neural stochastic controloutperforms the classic stochastic linear control in both the convergence time and the energy cost fortypical systems.
More importantly, using several representative physical systems, we have demonstrated the advantages of our frameworks and showed part of their weaknesses possibly emergent inreal applications.
Also, we present some limitations of the proposed frameworks in Appendix A.7.
Moreover, we suggest several directions for further investigations: (i) acceleration of the trainingprocess of the ES, (ii) the basin stability of the neural stochastic control (Menck et al., 2013), (iii)the trade-off between the deterministic controller u g using the NNs and the stochastic controller u fusing the NNs, (iv) the safe learning in Robotic control with small disturbances (Berkenkamp et al.,2017), and (v) the design of the purely data-driven stochastic neural control.
Checklist
For all authors
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See the Section
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] See the supplemen-
tary material.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] See the
supplementary material.
(b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
[Yes] See the supplementary material.
(c) Did you report error bars (e g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See the Section
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See the supplementary material.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
The repos we used do not mention
the license.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data
you’re using/curating?
[No]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁ-
able information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]