N ON -P ARAMETRIC N EURO -A DAPTIVE C ONTROL
Anonymous authors
Paper under double-blind review
A BSTRACT
We develop a learning-based algorithm for the control of autonomous systems governed by unknown, nonlinear dynamics to satisfy user-speciﬁed tasks expressed via time-varying reference trajectories.
Most existing algorithms either assume certain parametric forms for the unknown dynamic terms or resort to unnecessarily large control inputs in order to provide theoretical guarantees.
The proposed algorithm addresses these drawbacks by integrating neural-network-based learning with adaptive control.
More speciﬁcally, the algorithm learns a controller, represented as a neural network, using training data that correspond to a collection of system parameters and tasks.
These parameters and tasks are derived by varying the nominal parameters and the reference trajectories, respectively.
It then incorporates this neural network into an online closed-form adaptive control policy in such a way that the resulting behavior satisﬁes the user-deﬁned task.
The proposed algorithm does not use any a priori information on the unknown dynamic terms or any approximation schemes.
We provide formal theoretical guarantees on the satisfaction of the task.
Numerical experiments on a robotic manipulator and a unicycle robot demonstrate that the proposed algorithm guarantees the satisfaction of 50 user-deﬁned tasks, and outperforms control policies that do not employ online adaptation or the neuralnetwork controller.
Finally, we show that the proposed algorithm achieves greater performance than standard reinforcement-learning algorithms in the pendulum benchmarking environment.
I NTRODUCTION
Learning and control of autonomous systems with uncertain dynamics is a critical and challenging topic that has been widely studied during the last decades.
One can identify plenty of motivating reasons, ranging from uncertain geometrical or dynamical parameters and unknown exogenous disturbances to abrupt faults that signiﬁcantly modify the dynamics.
There has been, therefore, an increasing need for developing control algorithms that do not rely on the underlying system dynamics.
At the same time, such algorithms can be easily implemented on different, heterogeneous systems, since one does not need to be occupied with the tedious computation of the dynamic terms.
A promising step towards the control of systems with uncertain dynamics is the use of data obtained a priori from system runs.
However, engineering systems often undergo purposeful modiﬁcations (e.g., substitution of a motor or link in a robotic arm or exposure to new working environments) or suffer gradual faults (e.g., mechanical degradation), which might change the systems’ dynamics or operating conditions.
Therefore, one cannot rely on the aforementioned data to provably guarantee the successful control of the system.
On the other hand, the exact incorporation of these changes in the dynamic model, and consequently, the design of new model-based algorithms, can be a challenging and often impossible procedure.
Hence, the goal in such cases is to exploit the data obtained a priori and construct intelligent online policies that achieve a user-deﬁned task while adapting to the aforementioned changes.
There has been a large variety of works that tackle the problem of control of autonomous systems with uncertain dynamics, exhibiting, however, certain limitations.
The existing algorithms are based on adaptive and learning-based approaches or the so-called funnel control (Krstic et al.
Kanellakopoulos et al.
; Xu & Ioannou (2003); Ge & Wang (2004); Wen et al.
; Chen et al.
; Slotine & Li (1987); Slotine & Coetsee (1986); Chen et al.
; Xian et al.
Vamvoudakis & Lewis (2010); Berger et al.
; Bechlioulis & Rovithakis (2014); Joshi et al.
; Capotondi et al.
(2020); Bertsekas & Tsitsiklis (1996); Sutton & Barto (2018)).
Nevertheless, adaptive control methodologies are restricted to system dynamics that can be linearly parameterized with respect to certain unknown parameters (e.g., masses, moments of inertia), assuming the system structure perfectly known; funnel controllers employ reciprocal terms that drive the control input to inﬁnity when the tracking error approaches a pre-speciﬁed funnel function, creating thus unnecessarily large control inputs that might damage the system actuators.
Data-based learning approaches either consider some system characteristic known (e.g., a nominal model, Lipschitz constants, or global bounds), or use neural networks to learn a tracking controller or the system dynamics; the correctness of such methods, however, relies on strong assumptions on the parametric approximation by the neural network and knowledge of the underlying radial basis functions.
Finally, standard reinforcementlearning techniques (Bertsekas & Tsitsiklis (1996); Sutton & Barto (2018)) usually assume certain state and/or time discretizations of the system and rely on exhaustive search of the state space, which might lead to undesirable transient properties (e.g., collision with obstacles while learning).
1. C ONTRIBUTIONS AND S IGNIFICANCE
This paper addresses the control of systems with continuous, unknown nonlinear dynamics subject to tasks expressed via time-varying reference trajectories.
Our main contribution lies in the development of a learning-based control algorithm that guarantees the accomplishment of a given task using only mild assumptions on the system dynamics.
The algorithm draws a novel connection between adaptive control and learning with neural network representations, and consists of the following steps.
Firstly, it trains a neural network that aims to learn a controller that accomplishes a given task from data obtained off-line.
Secondly, we develop an online adaptive feedback control policy that uses the trained network to guarantee convergence to the given reference trajectory and hence satisfaction of the task.
Essentially, our approach builds on a combination of off-line trained controllers and on-line adaptations, which was recently shown to signiﬁcantly enhance performance with respect to single use of the off-line part (Bertsekas (2021)).
The major signiﬁcance of our contribution is twofold.
Firstly, we guarantee the theoretical correctness of the proposed algorithm by considering only mild conditions on the neural network, removing the long-standing assumptions on parametric approximations and boundedness of the estimation error.
Secondly, we demonstrate via the experimental results the generality of the algorithm with respect to different tasks and system parameters.
That is, the training data that we generate for the training of the neural network in the ﬁrst step correspond to tasks that are different from the given one to be executed 1 .
Additionally, we employ systems with different dynamic parameters to generate these data.
We evaluate the proposed algorithm in numerous scenarios comprising different variations of the given task and different system dynamic parameters, which do not necessarily match the training data.
We show that the algorithm, owing to its adaptation properties, is able to guarantee the satisfaction of the respective tasks in all the aforementioned scenarios by using the same neural network.
1. R ELATED WORK
A large variety of previous works considers neuro-adaptive control with stability guarantees, focusing on the optimal control problem (Vamvoudakis & Lewis (2010);
Yang et al.
; Cheng et al.
Fan et al.
; Kiumarsi et al.
(2017); Zhao & Gan (2020); Vrabie & Lewis (2009); Sun & Vamvoudakis (2020);
Kamalapurkar et al.
; Huang et al.
(2018b)
; Mo et al.
; Joshi et al.
Nevertheless, the related works draw motivation from the neural network density property (see, e.g., (Cybenko (1989))) 2 and assume sufﬁciently small approximation errors and linear parameterizations of the unknown terms of the form W ( x ) θ (dynamics, optimal controllers, or value functions), with W and θ known and unknown, respectively.
Similarly, more traditional adaptive control methodologies that handle uncertain nonlinear systems assume either linear parameterizations of the unknown dynamic terms (Krstic et al.
; Hong et al.
; Chen (2019); Huang et al.
(2018a);
Kanellakopoulos et al.
; Xu & Ioannou (2003); Ge & Wang (2004); Wen et al.
Chen et al.
; Slotine & Li (1987); Slotine & Coetsee (1986); Chen et al.
(2008)), use known upper-bound functions (Slotine & Coetsee (1986)), or provide local stability results dictated by the dynamic bounds (Xian et al.
; Chen et al.
This paper relaxes the aforementioned assumptions and proposes a non-parametric neuro-adaptive controller, whose stability guarantees rely on a mild boundedness condition of the closed-loop system state that is driven by the learned controller.
The proposed approach exhibits similarities with (Liu et al.
(1994)), which employs off-line-trained neural networks with online feedback control, but fails to provide convergence guarantees.
Similarly, Zeng et al.
(2020) develops an algorithm that combines a learning module and a physics-based controller for robot throwing of arbitrary objects, assuming however access to a physics simulator and not providing theoretical guarantees.
Other learning-based related works include modeling with Gaussian processes (Capotondi et al.
(2020); Leahy et al.
; Jain et al.
(2018); Berkenkamp & Schoellig (2015)), or use neural networks (Ma et al.
; Shah et al.
(2018); Yan & Julius (2021);
Liu et al.
; Hahn et al.
Cai et al.
; Wang et al.
(2020); Camacho & McIlraith (2019); Hahn et al.
Riegel et al.
Hu et al.
Ivanov et al.
to accomplish reachability, veriﬁcation or temporal logic speciﬁcations.
Nevertheless, the aforementioned works either use partial information on the underlying system dynamics, or do not consider them at all.
In addition, works based on
Gaussian processes usually propagate the dynamic uncertainties, possibly leading to conservative results.
Similarly, data-driven model-predictive control techniques (Nubert et al.
Maddalena et al.
(2020)) use data to over-approximate additive disturbances or are restricted to linear systems.
Control of unknown nonlinear continuous-time systems has been also tackled in the literature by using funnel control, without necessarily using off-line data or dynamic approximations (Berger et al.
(2018); Bechlioulis & Rovithakis (2014); Lindemann et al.
(2017); Verginis & Dimarogonas (2018);
Verginis et al.
Nevertheless, funnel controllers usually depend on reciprocal time-varying barrier functions that drive the control input to inﬁnity when the error approaches a pre-speciﬁed funnel, creating thus unnecessarily large control inputs that might damage the system actuators.
P ROBLEM F ORMULATION
Consider a continuous-time dynamical system governed by the 2 nd-order continuous-time dynamics
[Equation]
where ¯ x :=
[ x (cid:62) , ˙ x (cid:62) ] (cid:62) ∈
R 2 n , n ∈ N , is the system state, assumed available for measurement,and u : R 2 n × [0 , ∞ )
→ R n is the time-varying feedback-control input.
The terms f ( · ) and g ( · ) are nonlinear vector ﬁelds that are locally Lipschitz in ¯ x over R 2 n for each ﬁxed t ≥ 0 , and uniformly bounded in t over [0 , ∞ ) for each ﬁxed ¯ x ∈ R 2 n .
The dynamics (1) comprise a large class of nonlinear dynamical systems (Zhong & Leonard (2020); Yu et al.
; Doya (1997))
that capture contemporary engineering problems in mechanical, electromechanical and power electronics applications, such as rigid/ﬂexible robots, induction motors and DC-to-DC converters, to name a few.
The continuity in time and state provides a direct link to the actual underlying system, and we further do not require any time or state discretizations.
We consider that f ( · ) and g ( · ) are completely unknown; we do not assume any knowledge of the structure, Lipschitz constants, or bounds, and we do not use any scheme to approximate them.
Note also that we do not assume global Lipschitz continuity or global boundedness of f ( · , t ) and g ( · , t ) or the solution ¯ x ( t ) of (1).
Nevertheless, we do assume that g (¯ x, t ) is positive deﬁnite:
Assumption 1.
The matrix g (¯ x, t ) is positive deﬁnite, for all
(¯ x, t ) ∈ R 2 n × [0 , ∞ ) .
Such assumption is a sufﬁciently controllability condition for (1); intuitively, it states that the multiplier of u (the input matrix) does not change the direction imposed to the system by the underlying control algorithm.
Systems not covered by (1) or Assumption 1 consist of underactuated or non-holonomic systems, such as unicycle robots or underactuated aerial vehicles.
Nevertheless, we provide an extension of our results for a non-holonomic unicycle vehicle in Appendix B. Moreover, the 2 nd-order model (1) can be easily extended to account for higher-order integrator systems (Slotine et al.
Consider now a time-constrained task expressed as a time-varying reference trajectory p
d : R ≥
→ R n .
The objective of this paper is to construct a time-varying feedback-control algorithm u (¯ x, t ) such that the state of the closed-loop system (1) asymptotically tracks p d , i.e., lim t →∞ ( x ( t ) − p d ( t ))
= 0 .
M AIN R ESULTS
This section describes the proposed algorithm, which consists of two steps.
Firstly, it learns a controller, represented as a neural network, using training data that correspond to a collection of different tasks and system parameters.
Secondly, we design an adaptive, time-varying feedback controller that uses the neural-network approximation and guarantees tracking of the reference trajectory, consequently achieving satisfaction of the task.
3. N EURAL - NETWORK LEARNING
As discussed in Section 1, we are inspired by cases where systems undergo changes that modify their dynamics and hence the underlying controllers no longer guarantee the satisfaction of a speciﬁc task.
In such cases, instead of carrying out the challenging and tedious procedure of identiﬁcation of the new dynamic models and design of new model-based controllers, we aim to exploit data from off-line system trajectories and develop an online policy that is able to adapt to the aforementioned changes and achieve the task expressed via p d .
Consequently, we assume the existence of ofﬂine data from a ﬁnite set of T system trajectories that satisfy a collection of tasks, corresponding to bounded reference trajectories, including p d , and possibly produced by systems with different dynamic parameters.
The data from each trajectory i ∈ { 1 , . .
, T } comprise a ﬁnite set of triplets { ¯ x s ( t ) , t, u s ( t ) } t ∈T i , where
T i is a ﬁnite set of time instants, ¯ x s ( t ) ∈ R 2 n are system states, and
u s ( t ) ∈ R n are the respective control inputs, compliant with the dynamics (1).
We use the data to train a neural network in order to approximate the respective controller u (¯ x, t ) .
More speciﬁcally, we use the pairs (¯ x s ( t ) , t ) t ∈T i as input to a neural network, and u s ( t ) t ∈T i as the respective output targets, for all trajectories i ∈ { 1 , . . .
, T } .
For given ¯ x ∈ R 2 n , t ∈ R ≥ 0 , we denote by u nn (¯ x, t ) the output of the neural network.
Note that the controller u (¯ x, t ) , which the neural network aims to approximate, is not associated to the speciﬁc task expressed via p d and mentioned in Section 2, but a collection of several tasks.
Therefore, we do not expect the neural network to learn how to track p d , but rather to be able to adapt to the entire collection of tasks.
This is an important attribute of the proposed scheme, since it can generalize over tasks.
The motivation for training the neural network with different tasks and dynamic parameters is the following.
Since the tasks correspond to bounded trajectories, the respective stabilizing controllers compensate successfully the dynamics in (1).
Therefore, as will be clariﬁed in the next section, the neural network aims to approximate an “average” controller the retains this property, i.e., the boundedness of the dynamics of (1).
By using such approximation, the online feedback-control policy - illustrated in the next section - is able to guarantee tracking of p d without using any explicit information on the dynamics.
3. F EEDBACK CONTROL DESIGN
As mentioned in Section 3.1, we do not expect the neural-network controller to accomplish tracking of p d , since the system (1) is trained on potentially different tasks and different system parameters, and (2) the neural network provides only an approximation of a stabilizing controller; potential deviations in certain regions of the state space might lead to instability.
Moreover, the neural-network controller has no error feedback with respect to the open-loop trajectory p d ; such feedback is substantial in the stability of control systems with dynamic uncertainties.
Therefore, this section is devoted to the design of a feedback-control policy to track the trajectory p d ( t ) by using the output of the trained neural network (see Fig. 1).
The goal is to drive the error e := x − p d to zero.
As mentioned in
Section 3.1, the motivation for training the neural network with several different tasks and dynamic parameters is the learning of a controller that is able to retain the property of the training controllers to compensate the system dynamics (1).
This is ofﬁcially stated in the following assumption regarding the closed-loop system trajectory that is driven by the neural network’s output.
Assumption 2.
The output u nn (¯ x, t ) of the trained neural network satisﬁes
[Equation]
for positive constants d , B , for all ¯ x ∈ R 2 n , t ≥ 0 .
Intuitively, Assumption 2 states that the neural-network controller u nn (¯ x, t ) is able to maintain the boundedness of the system state by the constants d , B , which are considered to be unknown .
The assumption is motivated by the property of neural networks to approximate a continuous function arbitrarily well in a compact domain for a large enough number of neurons and layers (Cybenko (1989)) 3 .
As mentioned before, since the collection of tasks, which the neural network is trained with, correspond to bounded trajectories, the system states are expected to remain bounded.
Since f (¯ x, t ) , and g (¯ x, t ) are continuous in ¯ x and bounded in t , they are also expected to be bounded as per (2).
Contrary to the related works (e.g., (Vamvoudakis & Lewis (2010); Yang et al.
; Cheng et al.
; Fan et al.
(2018))), however, we do not adopt approximation schemes for the system dynamics and we do not impose restrictions on the size of d , B .
Moreover, Assumption 2 does not imply that the neural network controller u nn (¯ x, t ) guarantees tracking of the open-loop trajectory p d .
Instead, it is merely a growth condition.
Additionally, note that inequality 2 does not depend speciﬁcally on any of the tasks that the neural network is trained with.
We exploit this property in the control design and achieve task generalization; that is, the open-loop trajectory p d to be tracked (corresponding to the task ϕ ) can be any of the tasks that the neural network is trained with.
We now deﬁne the feedback-control policy.
Consider the adaptation variables ˆ (cid:96) 1 , ˆ (cid:96) 2 , corresponding to upper bounds of d , B in (2), with ˆ (cid:96) 1 (0) > 0 , ˆ (cid:96) 2 (0) > 0 .
We design ﬁrst a reference signal for ˙ x as
[Equation]
e, (3) that would stabilize the subsystem (cid:107) e (cid:107) 2 , where k 1 is a positive control gain constant.
Following the back-stepping methodology (Krstic et al.
, we deﬁne next the respective error e v := ˙
x − v d and design the neural-network-based adaptive control law as
[Equation]
4b
[Equation]
where k 2 , k (cid:96) 1 , k (cid:96) 2 are positive constants, and ˆ e v = e v (cid:107)
e v (cid:107) if e v (cid:54) = 0 , and ˆ e v =
if e v = 0 .
The control design is inspired by adaptive control methodologies (Krstic et al.
(1995)), where the time-varying gains ˆ (cid:96) 1 ( t ) , ˆ (cid:96) 2 ( t ) , adapt to the unknown dynamics and counteract the effect of d and B in (2) in order to ensure closed-loop stability.
Note that the policy (3), (4) does not use any information on the system dynamics f ( · ) , g ( · ) or the constants B , d .
The tracking of p d is guaranteed by the following theorem.
Theorem 1.
Let a system evolve according to (1) and let an open-loop trajectory p
d : R ≥
→ Rencoding a user-deﬁned task.
Under Assumption 2, the control algorithm (4) guarantees lim t →∞ ( e ( t ) , e v ( t ))
= 0 , as well as the boundedness of all closed-loop signals.
We provide a sketch of the proof here and give the details in Appendix A.
Proof (Sketch).
In view of (2), one can ﬁnd positive constants d 1 , d 2 , D such that
[Equation]
for all ¯ x ∈ R 2 n , t ≥ 0 , where g is the minimum eigenvalue of g (¯ x, t ) , which is positive owing to the positive deﬁnitiveness of g ( · ) .
The constants d 1 , d 2 , and D deﬁne upper bounds (cid:96) 1 and (cid:96) 2 , which we aim to approximate by the adaptation variables ˆ (cid:96) 1 and ˆ (cid:96) 2 , respectively.
We next deﬁne the continuously differentiable function
[Equation]
which represents an energy-like function of the system, and which we wish to drive to zero
One can prove, by analyzing the derivative ˙
V and using the control policy (4), that V is non-decreasing along the solutions of the closed-loop system and that lim t →∞ ˙ V ( t ) = 0 , which implies that lim t →∞ e ( t ) =
lim t →∞ e v ( t ) = 0 .
Note that, contrary to works in the related literature (e.g., (Verginis & Dimarogonas (2020); Bechlioulis & Rovithakis (2014))), we do not impose reciprocal terms in the control input that grow unbounded in order to guarantee closed-loop stability.
The resulting controller is essentially a simple linear feedback on ( e ( t ) , e v ( t )) with time-varying adaptive control gains, accompanied by the neural network output that ensures the growth condition (2).
The positive gains k 1 , k 2 , k (cid:96) 1 , k (cid:96) 2 do not affect the stability results of Theorem 1, but might affect the evolution of the closed-loop system; e.g., larger gains lead to faster convergence but possibly larger control inputs.
The proposed control algorithm does not require any of the long-standing assumptions on the system dynamics (1), such as linear parameterization, growth conditions, or boundedness by known functions (e.g., Krstic et al.
; Hong et al.
; Chen (2019);
Huang et al.
(2018a);
Kanellakopoulos et al.
; Xu & Ioannou (2003); Ge & Wang (2004); Wen et al.
; Chen et al.
; Slotine & Li (1987); Slotine & Coetsee (1986); Chen et al.
; Slotine & Coetsee (1986); Chen et al.
; Xian et al.
Additionally, we do not assume the boundedness of the solution of (1) or of the dynamic terms f ( · , t ) , g ( · , t ) ; instead, the control algorithm guarantees via Theorem 1 the boundedness of the system state as well as the asymptotic tracking of p d ( t ) .
The only boundedness condition that we require is (2) in Assumption 2, which can be accomplished by neural-network component in view of the universal approximation property Cybenko (1989).
N UMERICAL E XPERIMENTS
This section is devoted to a series of numerical experiments.
More details can be found in Appendix B. We ﬁrst test the proposed algorithm on a 6 -dof UR5 robotic manipulator with dynamics
[Equation]
) where x, ˙ x ∈ R 6 are the vectors of robot joint angles and angular velocities, respectively; B ( x ) ∈ R 6 ×is the positive deﬁnite inertia matrix, C (¯ x ) ∈ R 6 × 6 is the Coriolis matrix, g ( x ) ∈ R 6 is the gravity vector, and d (¯ x, t ) ∈
R 6 is a vector of friction terms and exogenous time-varying disturbances.
∈ { 1 , . .
The workspace consists of four points of interest T 1 , T 2 ,
T 3 , T 4 (end-effector position and Euler-angle orientation), as depicted in Fig. 2, which correspond to the joint-angle vectors c 1 , c 2 , c 3 , c 4 .
More information is provided in
Appendix A. We consider a nominal task expressed via the spatio-temporal constraint (cid:86) i ∈{ 1 ,..., 4 } G [0 , ∞ )
F I i ( (cid:107) x
1 − c i (cid:107) ≤ 0 . 1) , where G and F are the always and eventually operators respectively.
The task consists of visits of x 1 to c i ∈ R 6 (within the radius 0 . 1 ) inﬁnitely often within the time intervals dictated by I i , for i ∈ { 1 , . . .
We create 150 problem instances by varying the positions of c i , the time intervals
I i , the dynamic parameters of the robot (masses and moments of inertia of the robot’s links and actuators), the friction and disturbance term d ( · ) , the initial position and velocity of the robot, and the sequence of visits to the points
c i , as dictated by φ , i.e., one instance
[Figure]
might correspond to the visit sequence (( x (0) , 0)
→ ( c 1 , t 1 1 ) → ( c 2 , t 1 2 ) → ( c 3 , t 1 3 ) → ( c 4 , t 1 4 ) , and another to (( x (0) , 0)
→ ( c 3 , t 1 3 ) → ( c 1 , t 1 1 )
→ ( c 4 , t 1 4 ) → ( c 2 , t 1 2 ) .
We separate the aforementioned 150 problem instances into 100 training instances and 50 test instances.
We generate trajectories using the 100 training instances from system runs that satisfy different variations of one cycle of the task (i.e., one visit to each point).
Each trajectory consists of 500 points and is generated using a nominal model-based controller.
We use these trajectories to train a neural network and we test the control policy (4) in the 50 test instances.
We also compare our algorithm with the non-adaptive controller u c (¯ x, t ) =
u nn ( x, t ) − k 1 e − k 2 ˙ e , as well as with a modiﬁed version u d (¯ x, t ) of (4) that does not employ the neural network (i.e., the term u nn (¯ x, t ) ).
The comparison results are depicted in Fig.
3, which depicts the mean and standard deviation of the signal (cid:107) e ( t ) (cid:107) + (cid:107) ˙
e ( t ) (cid:107) for the 50 instances and 20 seconds.
It is clear from the ﬁgure that the proposed algorithm performs better than the non-adaptive and no-neural-network policies both in terms of convergence speed and steady-state error.
It is worth noting that the non-adaptive policy results on average in unstable closed-loop system.
We next test the proposed algorithm, following a similar procedure, on a unicycle robot (the respective dynamics and control algorithm can be found in Appendix B).
Fig.
5 depicts the mean and standard deviation of the errors e d ( t ) , e β ( t ) for 50 test instances.
We note that the performance of the noneural-network control policy is much more similar to the proposed one than in the UR5 case.
This can be attributed to (1) the lack of gravitational terms in the unicycle dynamics, which often lead to instability, and (2) the chosen control gains of the no-neural-network policy, which are sufﬁciently large to counteract the effect of the dynamic uncertainties.
Finally, we compare the performance of the proposed control policy with the reported data of (Wang et al.
on the benchmarking enivronment of the pendulum , where a single-link mechanical structure aims to reach the upright position.
Following (Wang et al.
, the reward and costs are r pend ( t ) = − cos q ( t ) − 0 .
1 sin q ( t ) − 0 .
1 ˙ q ( t ) − 0 .
001 u ( t )and
J pend =
(cid:80)
Ht
=1 γ t r pend ( t ) , respectively.
[Figure]
Similar to the previous cases and in contrast to
Wang et al.
(2019), we generate 150 instances by varying the system parameters (pendulum length and mass), the external disturbances, and the initial conditions.
We generate 100 trajectories, consisting of 500 points each, for the 100 training instances, by employing a nominal controller, and we use them train a neural network.
Moreover, in order to guarantee the feasibility of the proposed algorithm (4) we consider larger control-action bounds than in (Wang et al.
The original bounds render these systems under-actuated, which is not included in the considered class of systems (1) and consist part of our future work.
It should be noted that such larger bounds affect negatively the acquired rewards.
We test the control policy (4) on the 50 test instances, for 5000 steps each, corresponding to 10 seconds.
We set H = 200 and γ =
(Wang et al.
Fig.
4 depicts the mean and standard deviation of the time-varying reward and cost functions, illustrating successful regulation to the upright position.
After 5000 steps, we obtain a mean reward and cost of 0 . 99 and 200 , respectively, showing better performance than the reported cost of 180 in (Wang et al.
Moreover, the proposed control algorithm achieves this performance without resorting to exhaustive exploration of the state-space, which is the case in the reinforcement-learning algorithms used in (Wang et al.
This is a very important property in practical engineering systems, where safety is of paramount signiﬁcance and certain areas of the state space must be avoided.
D ISCUSSION AND L IMITATIONS
As shown in the experimental results, the control algorithm is able to asymptotically track reference trajectories which were not considered when generating the training data.
Similarly, the trajectories used in the training data were generated using systems with different dynamical parameters, and not speciﬁcally the ones used in the tests.
The aforementioned attributes signify the ability of the proposed algorithm to generalize to different tasks and systems with different parameters.
Nevertheless, the proposed control policy is currently limited by systems satisfying Assumption 1 and is not able to take into account under-actuated systems (e.g., the acrobot or cart-pole system); such systems consist part of our future work.
Moreover, the proposed algorithm is dependent on the performance of the learning component to satisfy Assumption 2.
Lack of satisfactory training might cause the neural-network output to add extra terms in the dynamics that render the closed-loop system unstable.
A less conservative and more robust approach consists of using the off-line data to train multiple neural networks, each for a certain region of the state space; such an approach constitutes part of our future work.
Finally, the discontinuities of (4), (12) might be problematic and create chattering when implemented in real actuators.
A continuous approximation that has shown to yield satisfying performance is the boundary-layer technique (Slotine et al.
C ONCLUSION AND F UTURE W ORK
We develop a novel control algorithm for the control of robotic systems with unknown nonlinear dynamics subject to tasks expressed as reference trajectories.
The algorithm integrates neural networkbased learning and adaptive control.
We provide formal guarantees and perform extensive numerical [Figure] experiments.
Future directions will focus on relaxing the considered assumptions and extending the proposed methodology to underactuated systems.
R EPRODUCIBILITY STATEMENT
We provide the proof of Theorem 1 in Appendix A. Additionally, we elaborate on the required assumptions (Assumptions 1, 2, 3) throughout the text - see pages 3, and 7.
Finally, we provide implementation details, instructions, and the respective code, required to reproduce the results, in
Appendix C and the supplementary material
A A PPENDIX
We provide here the proof of Theorem 1.
We ﬁrst give some preliminary notation and background on systems with discontinuous dynamics.
N OTATION
Given a function f : R n → R k , its Filippov regularization is deﬁned as (Paden & Sastry (1987))
[Equation] [Equation]
) where (cid:84) µ ( ¯ N )=0 is the intersection over all sets ¯ N of Lebesgue measure zero, co ( E ) is the closure of the convex hull co ( E ) of the set E , and B ( x, δ ) is the ball with radius δ centered at x .
N ONSMOOTH A NALYSIS
Consider the following differential equation with a discontinuous right-hand side:
[Equation]
where f : D × [ t 0 , ∞ ) → R n , D ⊂ R n , is Lebesgue measurable and locally essentially bounded.
Deﬁnition 1 (Def. 1 of (Fischer et al.
A function x : [ t 0 , t 1 ) → R n , with t 1 > t 0 , is called a Filippov solution of (7) on [ t 0 , t 1 ) if x ( t ) is absolutely continuous and if, for almost all t ∈ [ t 0 , t 1 ) , it satisﬁes ˙ x ∈ K [ f ]( x, t ) , where K [ f ]( x, t ) is the Filippov regularization of f ( x, t ) .
Lemma 1 (Lemma 1 of (Fischer et al.
Let x ( t ) be a Filippov solution of (7) and V :
D × [ t 0 , t 1 ) → R be a locally Lipschitz, regular function 4 .
Then V ( x ( t ) , t ) is absolutely continuous, ˙ V ( x ( t ) , t ) =
∂∂t V ( x ( t ) , t ) exists almost everywhere (a.e.), i.e., for almost all t ∈ [ t 0 , t 1 ) , and ˙ V ( x ( t ) , t ) a.e ∈ ˙ (cid:101) V
( x ( t ) , t ) , where
[Equation]
and ∂V ( x, t ) is Clarke’s generalized gradient at ( x, t )
(Fischer et al.
Corollary 1 (Corollary 2 of (Fischer et al.
For the system given in (7), let D ⊂ R n be an open and connected set containing x = 0 and suppose that f is Lebesgue measurable and x (cid:55)→ f ( x, t ) is essentially locally bounded, uniformly in t .
Let V : D × [ t 0 , t 1 ) → R be locally
Lipschitz and regular such that W 1 ( x ) ≤ V ( x, t ) ≤ W 2 ( x ) , ∀ t ∈ [
[Equation]
, t
[Equation]
, x ∈ D , and
[Equation]
where W 1 and W 2 are continuous positive deﬁnite functions and W is a continuous positive semideﬁnite on D .
Choose r > 0 and c > 0 such that ¯ B (0 , r ) ⊂ D and c < min (cid:107) x (cid:107) =
r W 1 ( x ) .
Then for all Filippov solutions x : [ t 0 , t 1 ) → R n of (7), with x ( t 0 ) ∈ D := { x ∈ ¯ B (0 , r ) :
W 2 ( x ) ≤ c } , it holds that t 1 = ∞ , x ( t ) ∈ D , ∀ t ∈
[ t 0 , ∞ ) , and lim t →∞ W ( x ( t )) = 0 .
Proof of Theorem 1.
We re-write ﬁrst the condition of Assumption 2.
Note that p d ( t ) and its derivatives are bounded functions of time.
Moreover, since e = x − p d , ˙ e = ˙ x − ˙ p d , it holds that
[Equation]
¯ x (cid:107) ≤ (cid:107) e (cid:107) + (cid:107) ˙
e (cid:107) + (cid:107) p d (cid:107) + (cid:107) ˙
p d (cid:107) and ˙ v d = ¨ p d − k 1 ˙ e , as well as e v = ˙
x − v d = ˙
e + k 1 e implying ˙
e = e v − k 1 e .
Therefore, in view of (2), one can ﬁnd positive constants d 1 , d 2 , D such that
[Equation]
for all ¯ x ∈ R 2 n , t ≥ 0 , where g is the minimum eigenvalue of g (¯ x, t ) , which is positive owing to the positive deﬁnitiveness of g ( · ) .
Inequality (8) will be used later in the proof.
Let now a constant α such that d 1 α 2 < k 1 .
As will be clariﬁed later, the adaptation variables ˆ (cid:96) 1 , ˆ (cid:96) 2 aim to approximate the constants d 1 2 α + d 2 and D , respectively.
Therefore, let (cid:96) 1 :=
d 1 2 α + d 2 , (cid:96) 2 := D , and the respective error terms (cid:101) (cid:96) 1 :
= ˆ (cid:96) 1 − (cid:96) 1 , (cid:101) (cid:96) 2 :
= ˆ (cid:96) 2 − (cid:96) 2 , as well as the overall state (cid:101) x :=
[ e (cid:62) , e (cid:62) v , (cid:101) (cid:96) 1 , (cid:101) (cid:96) 2 ] (cid:62) ∈ R 2 n +2 .
Since the control policy is discontinuous, we use the notion of
Filippov solutions.
The Filippov regularization of u is K [ u ] =
u nn ( x, t ) − k 2 e v − ˆ (cid:96) 1 e v − ˆ (cid:96) 2 ˆ E v , where ˆ E v :=
e v (cid:107) e v (cid:107) if e v (cid:54) =
and ˆ E v ∈ ( − 1 , 1) otherwise.
Note that, in any case, it holds that e (cid:62) v ˆ E v = (cid:107) e v (cid:107) .
Let now the continuously differentiable function
[Equation]
which satisﬁes W 1 ( (cid:101) x ) ≤ V ( (cid:101) x ) ≤ W 2 ( (cid:101) x ) for W 1 ( (cid:101)
[Equation] [Equation]
max { 12 , 12 g ,
[Equation]
According to Lemma 1, ˙ V ( (cid:101) x ) a.e. ∈ ˙
(cid:101) V ( (cid:101) x ) , with ˙ (cid:101) V :
= (cid:84) ξ ∈ ∂V ( (cid:101) x ) K [˙ (cid:101) x ] .
Since V ( (cid:101) x ) is continuously differentiable, its generalized gradient reduces to the standard gradient and thus it holds that ˙ (cid:101) V (
[Equation]
x ) = ∇ V
[Equation]
K [
[Equation]
x ] , where ∇ V =
[ e
[Equation]
e
[Equation]
] (cid:62) .
By differentiating
V and using (3), one obtains
[Equation]
and by using ˙ x = e v + v d and substituting the control policy (3), (4), and inequality (8),
[Equation]
e
[Table]
ge + f
[Table]
, t
[Table]
g
[Table] [Table]
u
[Table]
t
[Table]
e
[Table]
(cid:96)
[Table]
(cid:96)
[Table]
(cid:1) + (cid:101) (cid:96) 1 (cid:107) e v (cid:107) 2 + (cid:101) (cid:96) 2 (cid:107)
e v (cid:107)
[Table]
+
[Table] [Table]
e v (cid:107) 2 + (cid:101) (cid:96) 2 (cid:107) e v (cid:107)
Note from (4) and the fact that ˆ (cid:96) 1 (0) > 0 , ˆ (cid:96) 2 (0) > 0 that ˆ (cid:96) 1 ( t ) > 0 and ˆ (cid:96) 2 ( t )) > 0 for all t ≥ 0 .
Moreover, recall that g (¯ x, t ) is positive deﬁnite for all ¯ x ∈ R 2 n , t ≥ 0 , with g being its minimum (and positive) eigenvalue.
Therefore, we conclude that (cid:102) W s becomes (cid:102) W s ≤ − k
[Table]
e
[Table]
e
[Table]
e
[Table]
2 + D
[Table]
e v
[Table]
− ( k 2 + ˆ
[Table] [Table]
e v (cid:107) 2 − ˆ (cid:96) 2 (cid:107) e v (cid:107) + (cid:101) (cid:96) 1 (cid:107) e v (cid:107) 2 + (cid:101) (cid:96) 2 (cid:107) e v (cid:107)
By incorporating the term α in the term d 1 (cid:107) e v (cid:107)(cid:107) e (cid:107) , we obtain d 1 (cid:107) e v (cid:107)(cid:107) e (cid:107) =
d 1 α (cid:107) e v (cid:107) α (cid:107) e (cid:107) and by using the property ab ≤ 12 a
+ 12 b 2 for any constants a , b , we obtain d 1 α (cid:107) e v | α (cid:107) e (cid:107) ≤ d 1 α 2 (cid:107) e (cid:107) 2 + d 1 2 α (cid:107) e v (cid:107) 2 .
Therefore, (cid:102) W s becomes
[Equation]
=
[Equation]
[Table]
Therefore, it holds that ζ ≤ − Q ( (cid:101) x ) , for all ζ ∈ ˙ (cid:101) V , with Q being a continuous and positive semi-deﬁnite function in R 2 n +2 , since k 1 − d 1 α> 0 .
Choose now any ﬁnite r > 0 and let c < min (cid:107) (cid:101) x (cid:107) =
r W 1 ( (cid:101) x ) .
Hence, all the conditions of Corollary 1 are satisﬁed and hence, all Filippov solutions starting from (cid:101) x (0) ∈
Ω f := { (cid:101) x ∈ B (0 , r ) : (cid:102) W 2 ( (cid:101) x ) ≤ c } are bounded and remain in Ω f , satisfying lim t →∞ Q ( (cid:101) x
( t ))
= 0 .
Note that r , and hence c , can be arbitrarily large allowing and ﬁnite initial condition (cid:101) x (0) .
Moreover, the boundedness of (cid:101) x implies the boundedness of e , ˙ e , e v , (cid:101) (cid:96) 1 , and (cid:101) (cid:96) 2 , and hence of ˆ (cid:96) 1 ( t ) and ˆ (cid:96) 2 ( t ) , for all t ∈ R ≥ 0 .
In view of (5), we ﬁnally conclude the boundendess of u ( · ) , ˙ˆ (cid:96) 1 , and ˙ˆ (cid:96) 2 , for all t ∈ R ≥ 0 , leading to the conclusion of the proof.
B A PPENDIX Extension to non-holonomic unicycle dynamics
As mentioned in Section 1, the dynamics 1 do not represent all kinds of systems, with one particular example being when non-holonomic constraints are present.
In such cases, the control law design (4) and Theorem 1 no longer hold.
In this section, we extend the control policy to account for unicycle vehicles subject to ﬁrst-order non-holonomic constraints.
More speciﬁcally, we consider the dynamics
[Equation]
where x = [ p 1 , p 2 , φ ] (cid:62) ∈ R 3 are the unicycle’s position and orientation, ( v, ω ) are its linear and angular velocity (see Fig. 6), θ :=
[ θ R , θ L ] (cid:62) ∈ R 2 are its wheel’s angular positions, and u = [ u R , u L ] (cid:62) ∈ R 2 are the wheel’s torques, representing the control input.
The unicycle vehicle is subject to the non-holonomic constraint ˙ p 1 sin φ − ˙ p 2 cos φ = 0 , which implies that the vehicle cannot move laterally.
Additionally, M ∈ R 2 × 2 is the vehicle’s inertia matrix, which is symmetric and positive deﬁnite, and f θ ( · ) is a function representing friction and external disturbances.
The velocities satisfy the relations v = r 2 ( ˙ θ R + ˙ θ L ) , ω = r 2 R ( ˙ θ R − ˙ θ L )) , where r and R are the wheels’ radius and axle length, respectively.
The terms r , R , M , and f θ ( · ) are considered to be completely unknown .
As before, the goal is for the vehicle’s position p :=
[ p 1 , p 2 ] (cid:62) to track a desired trajectory p d
=
[ p d , 1 , p d , 2 ] (cid:62) ∈ R 2 .
Towards that end, we deﬁne the error variables e 1 :=
p 1 − p d , 1 , e 2 :=
p 2 − p d , 2 , e d := (cid:107) p − p d (cid:107) , as well as the angle β measured from the the longitudinal axis of the vehicle, i.e., the unicycle’s direction vector [cos φ, sin φ ] , to the error vector − [ e 1 , e 2 ] (see Fig.
The angle β can be derived by using the cross product between the aforementioned vectors, i.e., e d sin( β ) =
[cos φ, sin φ ] × [ − e 1 , − e 2 ] =
e 1 sin φ − e 2 cos φ .
The purpose of the control design, illustrated next, is to drive e d and β to zero.
By differentiating the latter and using (9) as well as the relations e
[Equation]
[Equation] [Equation]
( φ
[Equation]
β ) (see Fig. 6), we derive
[Equation]
10b
[Equation]
In view of (10), we set reference signals for the vehicle’s velocity as
[Equation]
11b
[Equation]
where k d , k β are positive gains, aiming to create exponentially stable subsystems via the terms k d e d and k β β .
We deﬁne next the respective velocity errors e v : =
v − v d , e ω : = ω − ω d and design the adaptive and neural-network-based control input as u
[Equation]
where ˆ (cid:96) v , ˆ (cid:96) ω , ˆ (cid:96) i are adaptation variables (similar to (4)), with ˆ (cid:96) v (0) > 0 , ˆ (cid:96) ω (0) >
0 and k v , k ω , k i ,
are positive gains, i ∈ { 1 , 2 } ; ˆ e a , with a ∈ { v, ω } , is deﬁned as ˆ e a = e a | e a | if e a (cid:54) =
and ˆ e a =otherwise.
We now re-state Assumption 2 to apply for the unicycle analysis as follows.
Assumption 3.
The output u nn (¯ x, t ) of the trained neural network satisﬁes (cid:107) u nn (¯ x, t ) + f θ (¯ x, t ) (cid:107) ≤ d (cid:107) ¯ x (cid:107) + B , for positive, unknown constants d , B .
Similar to assumption 2, assumption 3 is merely a growth-boundedness condition by the unknown constants d and B .
The stability of the proposed scheme is provided in the next corollary.
Corollary 2.
Let the unicycle system (9) and let an open-loop trajectory p
d : R ≥
→ R 2 .
Assume that β ( t ) ∈ ( − ¯ β, ¯ β ) , | ˙ p d , 1 sin φ − ˙ p d,2 cos
φ | < e d α 1 , sin β < e d α 2 for positive constants ¯ β < π 2 , α 1 , α 2 and all t ≥ 0 .
Under Assumption 3, the control policy (12) guarantees lim t →∞ ( e d ( t ) , β ( t ) , e v ( t ) , e ω ( t ))
= 0 , and the boundedness of all closed-loop signals.
Proof of Corollary 2.
The derivatives of e d , β in (10) can be written as
[Equation]
where
[Equation]
It can be veriﬁed that the reference velocity signals, designed in (11), can be written as
[Equation]
and therefore, by using the relations v = e v + v d and ω = e ω + ω d , ˙ e d and ˙ β can be written as
[Equation]
The control design follows the back-stepping methodology (Krstic et al.
Let, therefore, the function V 1 :=
12 ( e 2 d + β 2 ) , which, after time differentiation and use of (13), yields
[Equation]
We will cancel the two last terms of ˙ V 1 using the control input (12).
First, we note that the inertia matrix of the system M , appearing in the unicycle dynamics (9), has the form (Ivanjko et al.
M = (cid:20) M
M 2 M
M(cid:21) with M 1 :=
mr 2 4 + ( I C + md 2 ) r 2 4 R+ I 0 ,
M 2 :=
mr 2 4 − ( I C + md 2 ) r 2 4 R, and where I C is the moment of inertia of the vehicle with respect to point
C (see Fig. 6), I 0 is the moment of inertia of the the wheels, and d is the distance of the between point C and the vehicle’s center of mass ( p 1 , p 2 ) .
Therefore, the second part of the unicycle dynamics (9) can be written as
[Equation]
with f θ,R and f θ,L denoting the elements of f θ .
By summing and subtracting the aforementioned equations, we obtain
[Equation]
From the deﬁnition of e v and e ω , it holds that ˙ e v = ˙
v − ˙ v d , ˙ e ω
= ˙
ω − ˙ ω d , which, by using the relations v = r 2 ( ˙ θ R + ˙ θ L ) , ω = r 2 R ( ˙ θ R − ˙ θ L ) and (15), becomes
[Equation]
Deﬁne now (cid:96) v := 2 M 1 + M 2 r , (cid:96) ω := 2 R M 1 − M 2 r .
The adaptation terms ˆ (cid:96) v , and ˆ (cid:96) ω , used in the control mechanism (12), aim to approximate (cid:96) v and (cid:96) ω , respectively.
Note that ˆ (cid:96) v and ˆ (cid:96) ω are positive, and we deﬁne the errors (cid:101) (cid:96) v :
= ˆ (cid:96) v − (cid:96) v , (cid:101) (cid:96) ω :
= ˆ (cid:96) ω − (cid:96) ω .
We re-write next the condition of Assumption 3.
It holds that (cid:107) p (cid:107) =
(cid:112) p 21 + p 22 ≤ 2 e d + | p d , 1 | + | p d , 2 | .
Moreover, in view of (11) as well as the fact that β ∈ ( − ¯ β, ¯ β ) ⊂ ( − π 2 , π 2 ) , where ¯ β is the bound of β , stated in Corollary 2, it holds that | v | ≤ | e v | + | v d | ≤ | e v | + 1 cos(¯ β ) ( | ˙ p d , 1 | + | ˙ p d , 2 | + k d e d ) , | ω | ≤ | e ω | + | ω d | ≤ | e ω | + 1 cos ¯ β ( α 1 + k d )+ k β | β | , where we also use the fact that | ˙ p d , 1 sin φ − ˙ p d , 2 cos φ | e d ≤ α 1 .
By also recalling that φ moves on the unit circle, we conclude that Assumption 3 becomes
[Equation]
with d d := d (2 + k d cos ¯ β ) , d β := dk β , and D := d (2¯ p d + 2 π + 1 cos ¯ β ( α 1 + 2¯ v d + 1)) , where ¯ p d and ¯ v d are the upper bounds of p d ,i , v d ,i , respectively, i ∈ { 1 , 2 } .
Let now positive constants γ d and γ β such that k d > 2 d d γ d and k β > 2 d β γ β , and deﬁne
[Equation]
[Equation]
which is the constant to be approximated by ˆ (cid:96) 1 ; its derivation will be clariﬁed later.
Let also (cid:96) 2 := 2 D , which we aim to approximate with the adaptation variable
ˆ
(cid:96) 2 .
We deﬁne hence the respective errors (cid:101) (cid:96) 1 :
= ˆ (cid:96) 1 − (cid:96) 1 and (cid:101) (cid:96) 2 :
= ˆ (cid:96) 2 − (cid:96) 2 and deﬁne the overall vector (cid:101) x :=
[ e d , β, e v , e ω , (cid:101) (cid:96) v , (cid:101) (cid:96) ω , (cid:101) (cid:96) 1 , (cid:101) (cid:96) 2 ] (cid:62) ∈ R 8 .
Since the control mechanism is discontinuous, we use again the notion of Filippov solutions.
The
Filippov regularization K [ u ] of u differs from u only in the terms ˆ e v
and ˆ e ω , which are replaced by ˆ E v , deﬁned as ˆ E v :=
e v | e v | if e v (cid:54) = 0 , and ˆ E v ∈ ( − 1 , 1) otherwise, and ˆ E ω , deﬁned as ˆ E ω :=
e ω | e ω | if e ω (cid:54) = 0 , and ˆ E ω ∈ ( − 1 , 1) , respectively.
Consider now the continuously differentiable and positive deﬁnite function
[Equation]
(cid:101) (cid:96) 2 i which satisﬁes W 1 ( (cid:101) x ) ≤ V ( (cid:101) x ) ≤ W 2 ( (cid:101) x ) for W 1 ( (cid:101) x ) :
= min { 12 , (cid:96) v 2 , (cid:96) ω 2 , 12 k v , 12 k ω , 12 k 1 , 12 k 2 }(cid:107) (cid:101) x (cid:107) 2 ,
W 2 ( (cid:101) x ) :
= max { 1 2 , (cid:96) v 2 , (cid:96) ω 2 , 1 2 k v , 1 2 k ω , 1 2 k 1 , 1 2 k 2 }(cid:107) (cid:101) x (cid:107) 2 .
According to Lemma 1, ˙ V ( (cid:101) x ) a.e. ∈ ˙
(cid:101) V ( (cid:101) x ) , with ˙ (cid:101) V :
= (cid:84) ξ ∈ ∂V ( (cid:101) x ) K [˙ (cid:101) x ] .
Since V ( (cid:101) x ) is continuously differentiable, its generalized gradient reduces to the standard gradient and thus it holds that ˙ (cid:101) V ( (cid:101) x )
= ∇ V (cid:62) K [ (cid:101) x ] , where ∇ V =
[ e d , β, (cid:96) v e v , (cid:96) ω e ω , 1 k v (cid:101) (cid:96) v , 1 k ω (cid:101) (cid:96) ω , 1 k 1 (cid:101) (cid:96) 1 , 1 k 2 (cid:101) (cid:96) 2 ] (cid:62) .
Therefore, by differentiating V and employing (14) and (15), we obtain
[Equation]
By substituting the control and adaptation policy (12), (cid:102) W becomes
[Equation]
where u R, nn and u L, nn are the elements of u nn , i.e., u nn = [ u R, nn , u L, nn ] (cid:62) .
It is easy to conclude that | u nn ,R + u nn ,L + f θ,R + f θ,L | ≤ | u nn ,R + f θ,R | + | u nn ,L + f θ,L | ≤ 2 (cid:107) u nn + f θ (cid:107) and | u nn ,R − u nn ,L + f
[Equation]
−
[Equation]
| ≤ | u nn ,R +
[Equation]
| + | u nn ,L + f θ,L |
[Equation]
2 (cid:107) u nn + f θ (cid:107) and hence, in view of (17),
[Equation]
Therefore, (cid:102) W becomes
[Equation]
The term ¯ D ( | e v | + | e ω | ) yields
[Equation]
By setting 2 d d e d | e v | = 2 d d γ d e d | e v | γ d , 2 d d e d | e ω | = 2 d d γ d e d | e ω | γ d , and 2 d β | β
||
e v | = 2 d β γ β | β | | e v | γ β , 2 d β | β || e ω | = 2 d β γ β | β | | e ω | γ β and completing the squares, one obtains
[Equation]
Hence, (cid:102) W becomes
[Equation]
1 ( e 2 v + e 2 ω ) − ˆ (cid:96) 2 ( | e v | + | e ω | )
+ (cid:96) 1 ( e 2 v + e 2 ω )
+ (cid:96) 2 ( | e v | +
[Equation]
ω | ) + (cid:101) (cid:96) 1 ( e 2 v + e 2 ω ) + (cid:101)
[Equation]
[Equation]
Therefore, it holds that ζ ≤ − Q ( (cid:101) x ) , for all ζ ∈ ˙ (cid:101) V , with Q being a continuous and positive semideﬁnite function in R 8 , since k d − 2 d d γ d > 0 , k β − 2 d β γ β > 0 .
Choose now any ﬁnite r > 0 and let c < min (cid:107) (cid:101) x (cid:107) =
r W 1 ( (cid:101) x ) .
Hence, all the conditions of Corollary 1 are satisﬁed and hence, all Filippov solutions starting from (cid:101) x (0) ∈
Ω f := { (cid:101) x ∈ B (0 , r ) : (cid:102) W 2 ( (cid:101) x ) ≤ c } are bounded and remain in Ω f , satisfying lim t →∞ Q ( (cid:101) x
( t ))
= 0 .
Note that r , and hence c , can be arbitrarily large allowing and ﬁnite initial condition (cid:101) x (0) .
Moreover, the boundedness of (cid:101) x implies the boundedness of e , ˙ e , e v , (cid:101) (cid:96) 1 , and (cid:101) (cid:96) 2 , and hence of ˆ (cid:96) 1 ( t ) and ˆ (cid:96) 2 ( t ) , for all t ∈ R ≥ 0 .
Finally, in view of the conditions | sin φ ˙ p d
The assumptions | ˙ p d , 1 sin φ − ˙ p d,2 cos φ | < e d α 1 , sin β < e d α 2 are imposed to avoid the singularity of e d = 0 ; note that β and ω d are not deﬁned in that case.
Intuitively, they imply that e d will not be driven to zero faster than β or ˙ p d , 1 sin φ − ˙ p d,2 cos φ ; the latter becomes zero when the vehicle’s velocity vector v aligns with the desired one ˙ p d .
In the experiments, we tune the control gains according to k β ∼ 10 k d in order to satisfy these assumptions.
C A PPENDIX
We provide here more information on the experimental results of Section 4.
All the results were obtained on MATLAB environment using the ODE simulator.
We performed the training of the neural networks in with the pytorch library in Python environment.
The neural networks consist offully connected layers of 512 neurons; each layer is followed by a batch normalization module and a
ReLU activation function.
For the training we use the adam optimizer and the mean-square-error loss function.
In all cases we choose a batch size of 256, and we train until a desirable average (per batch) loss of the order of 10 − 4 is achieved.
All the values of the data used for the training were normalized in [0 , 1] .
Regarding the robotic-arm task, the points of interest are set as
[Equation]
and T=
[Equation]
and the corresponding joint-angle vectors as
[Equation]
07 , − 1 .
45 , 2 . 3 , 1 . 37 , − 1 .
33] (cid:62) , c= [1 . 28 , 0 .
03 , 0 . 1 , − 1 .
22] (cid:62) , c= [ − 0 .
08 , 0 . 85 , − 0 .
58 , 2 . 09 , − 2 .
36] (cid:62) , c 4 =
[ − 0 .
7 , − 0 .
76 , − 1 . 05 , − 0 .
05 , − 3 .
08 , 2 . 37] (cid:62) (radians).
We set a nominal value for the time intervals as I i =
[0 , 20] (seconds), and we create 150 problem instances by varying the following attributes: ﬁrstly, we add uniformly random offsets in [ − 0 .
3] (radians) to the elements of all c i , and in [ − 2 , 2] (seconds) to the right end-points of the intervals
I i ; secondly, we add random offsets to the dynamic parameters of the robot (masses and moments of inertia of the robot’s links and actuators) and we set a different friction and disturbance term d ( · ) , leading to a different dynamic model in (5); thirdly, we set different sequences of visits to the points
c
i , i ∈ { 1 , . . .
, 4 } , as dictated by φ , i.e., one trajectory might correspond to the visit sequence (( x (0) , 0) → ( c 1 , t 1 1 ) →
( c 2 , t 1 2 ) → ( c 3 , t 1 3 )
→ ( c 4 , t 1 4 ) , and another to (( x (0) , 0) → ( c 3 , t 1 3 )
→ ( c 1 , t 1 1 ) → ( c 4 , t 1 4 ) → ( c 2 , t 1 2 ) .
Finally, we add uniformly random offsets in [ − 0 .
5] to the initial position of the robot (from the ﬁrst point of the sequence),
and we set its initial velocity randomly in the interval [0 , 1] 6 .
Regarding the dynamics (5), we use the methodology described in (Siciliano et al.
to derive the B , C , and g terms.
We set nominal link masses and moments of inertia as m =
7] (kg) and I =
(kgm 2 ), respectively, and we add random offsets in ( − m 2 , m 2 ) , ( − I 2 , I 2 ) in the created instances.
Regarding the function d () used in (5); we set d (¯ x, t ) =
d t ( t ) + d f (¯ x ) , where
[Equation]
A t = diag { A t i } i ∈{ 1 ,..., 6 } ∈ R 6 × 6 , A t i is a random term in (0 , 2 m i ) , η i is a random term in (0 , 1) , ϕ
i is a random term in (0 , 2) , B t ∈ R 6 × 36 is a random matrix taking values in (0 , 2 m i ) , and ⊗ denotes the Kronecker product.
We chose the control gains of the control policy (4) as k 1 = 1 , k 2 = 10 , and k (cid:96) 1 = k (cid:96) 2 = 10 .
Further experimental results are depicted in Figs.
7-11; Fig.
7a depicts the mean and standard deviation of (cid:107) e v ( t ) (cid:107) of the proposed control policy for the 50 test instances, whereas Figs.
10a depicts the control input that results from the control policy (4) as well as the the neural-network output.
Note that the control input converges to the neural-network output, i.e., lim t →∞ ( u ( t ) − u nn ( t ))
= 0 , which can be also veriﬁed by (4) and the fact that lim t →∞ e v ( t )
= 0 .
Fig.
11a depicts the mean and standard deviation of the adaptation signals ˆ (cid:96) 1 ( t ) , ˆ (cid:96) 2 ( t ) for the 50 test instances.
Finally, Fig.shows timestamps of one of the test trajectories, illustrating the visit of the robot end-effector to the points of interest at the pre-speciﬁed time stamps.
[Figure] [Figure]
[Figure]
Regarding the unicycle experiments, the dynamic terms in (9) have the form
[Equation]
with M 1 :
= mr 2 4 + ( I
[Equation]
+ md 2 ) r 2 4 R
[Equation]
, and where I C is the moment of inertia of the vehicle with respect to point C (see Fig. 6), I 0 is the moment of inertia of the the wheels, and d is the distance of the between point C and the vehicle’s center of mass ( p 1 , p 2 ) .
The term d (¯ x, t ) is chosen as in the robotic-manipulator case.
The points to visit are chosen here as c 1 =
[0 , 0] (cid:62) , c 2 = [0 , 2] (cid:62) , c 3 =
[2 , 0] (cid:62) , c 4 =
[2 , 2] (cid:62) and I i is chosen as I i =
[0 , 20] , i ∈ { 1 , . .
We derive 150 problem instances by varying the following attributes: we vary c
i with random offsets in [ − 0 . 3 , 0 . 3] and the right end-points of I i with random offsets in [ − 2 , 2] ; we add random offsets to the dynamic parameters (elaborated subsequently) and the function d (¯ x, t ) , and we set different sequences of visits to the points
c
i ; we further vary the unicycle’s initial position from the ﬁrst c
i with random offsets in [ − 0 . 3 , 0 . 3] , and its initial orientation with random offsets in [ − 0 .
25] (rad) from θ (0) =
arctan( e 2 (0) /e
; we further set random values in [ − 0 .
(rad/s) to the initial wheel velocities.
We set nominal values for the dynamic and geometric parameters, appearing in the inertia matrix, as m = 28 (kg), I C = 0 .
1 (kgm 2 ), I 0 = 0 .
01 (kgm 2 ), r = 0 .
01 (m), d = 0 .
01 (m), and we added random offsets in ( − m 2 , m 2 ) , ( − I A 2 , I A 2 ) , ( − I 0 2 , I 0 2 ) , ( − r 2 , r 2 ) , ( − d 2 , d 2 ) , respectively, for the problem instances.
We chose the control gains of (12) as k d = 0 .
25 , k β = 1 , k v = k ω
=
k (cid:96) 1 = k (cid:96) 2 = 10 , k v = k ω
= 1 .
The non-adaptive control policy we compared our algorithm with was selected as
[Equation]
where v d , ω d are chosen as (11) and M S , M D are static estimates of 2 M 1 + M 2 r , 2 R M 1 − M 2 r , respectively.
More speciﬁcally, we set a deviation of 25% in the parameters appearing in these terms to form
M S and M D .
Further experimental results are depicted in Figs.
7-11; Fig.
7b depicts the mean and standard deviation of the velocity errors e v ( t ) , e β ( t ) for the 50 test instances, showing convergence to
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
) , ˆ (cid:96) 1 ( t ) , ˆ (cid:96) 2 ( t ) for the proposed control policy and the unicycle environment.
zero, whereas Figs. 10b depicts the control input that results from the control policy (4) as well as the the neural-network output.
Similarly to the robotic-manipulator case, the control input converges to the neural-network output.
Finally, Fig.
11b depicts the mean and standard deviation of the adaptation signals ˆ (cid:96) v ( t ) , ˆ (cid:96) β ( t ) , ˆ (cid:96) 1 ( t ) , ˆ (cid:96) 2 ( t ) for the 50 test instances.
Finally, regarding the pendulum environment, we consider the dynamics
[Equation]
where L is the pendulum’s link length, g is the gravitational constant, and d ( t ) is a term of exogenous disturbances.
We created 150 instances by varying the mass and link length of the pendulum, the external disturbances, and setting its initial position and velocity randomly in [ − 1 , 1] (rad) and [ − 1 , 1] (rad/s), respectively.
We selected unitary value for its nominal link length, and we added random offsets in ( − 0 .
5 , 0 . 5) in each instance.
We set the external disturbances as d = A sin( ηt + φ ) , with
A , η , and φ taking random values in [0 , 0 . 2] , [0 , 1] , [0 , 2] , respectively.
In contrast to the robotic manipulator case, we assume feedback of sin q , cos q , ˙ q , and set the error e in (3) as 1 − cos( q − π ) .
Finally, we chose the control gains as k 1 = k 2 = 1 , and k (cid:96) 1 = k (cid:96) 2 = 10 .