QuaDUE-CCM: Interpretable Distributional Reinforcement Learning using Uncertain Contraction Metrics for Precise Quadrotor Trajectory Tracking
Abstract: Accuracy and stability are common requirements for quadrotor trajectory tracking systems.
Designing an accurate and stable tracking controller remains challenging, particularly in unknown and dynamic environments with complex aerodynamic disturbances.
We propose a Quantile-approximation-based
Distributional-reinforced Uncertainty Estimator (QuaDUE) to accurately identify the effects of aerodynamic disturbances, i.e., the uncertainties between the true and estimated Control Contraction Metrics (CCMs).
Taking inspiration from contraction theory and integrating the QuaDUE for uncertainties, our novel CCMbased trajectory tracking framework tracks any feasible reference trajectory precisely whilst guaranteeing exponential convergence.
More importantly, the convergence and training acceleration of the distributional RL are guaranteed and analyzed, respectively, from theoretical perspectives.
We also demonstrate our system under unknown and diverse aerodynamic forces.
Under large aerodynamic forces ( > 2 m/s 2 ), compared with the classic data-driven approach, our QuaDUECCM achieves at least a 56.6% improvement in tracking error.
Compared with
QuaDRED-MPC, a distributional RL-based approach, QuaDUE-CCM achieves at least a 3 times improvement in contraction rate.
Introduction
Designing a precise and safe trajectory tracking controller for autonomous Unmanned Aerial Vehicles (UAVs), such as quadrotors, is a critical yet extremely different problem, particularly in unknown and dynamic environments with unpredictable aerodynamic forces.
To achieve reliable trajectory tracking for agile quadrotor ﬂights, two key challenges are: 1) solving for highly variable uncertainties caused by these aerodynamic forces; and 2) ensuring the tracking controller performs precisely and reliably under these dynamic uncertainties.
Previous research shows that aerodynamic effects deriving from drag forces and moment variations caused by the rotors and the fuselage [1] are the primary source of uncertainty, where these aerodynamic effects appear prominently at ﬂight speeds greater than 5 ms − 1 in wind tunnel experiments [2].
The causes of these effects are extremely complex: combinations of individual propellers, airframe [3], rotor–rotor and airframe–rotor turbulent interactions [4], and other turbulent propagation [5].
These aerodynamic effects are therefore chaotic and difﬁcult to model directly.
To address these concerns, two areas of research have emerged.
The ﬁrst uses a distributional Reinforcement Learning (RL) [6] method to interact with the complex and changeable uncertainties.
The second extends the contraction theory [7] to a control-afﬁne system, called Control Contraction Metric (CCM).
This contraction certiﬁcate guarantees that the system will exponentially converge any feasible reference trajectory [ applications such as quadrotor tracking we are concerned with the whole tracking process rather than just stabilizing to a ﬁxed point.
One common problem with certiﬁcate control approaches, like CCM-based control, is that they heavily rely on the accurate knowledge of the system model.
Thus, when the model is uncertain, robust or adaptive versions must be considered
We propose Qua ntile-approximation-based D istributional U ncertainty E stimatior for C ontrol
C ontraction M etric (QuaDUE-CCM), as a precise, reliable and systematic quadrotor tracking framwork for use with high variance aerodynamic effects.
Our contributions can be summarized:
1) QuaDUE , a distributional-RL-based uncertainty estimator with quantile approximation that can sufﬁciently estimate variable uncertainties of contraction metrics.
In the cases tested, we show that QuaDUE outperforms traditional RL, such as Deep Deterministic Policy Gradient (DDPG) [9], and prior Distributional RL approaches, such as C51 [6]; 2) QuaDUE-CCM , a quadrotor trajectory tracking framework that integrates a distributional-RLbased uncertainty estimator into a CCM-based nonlinear optimal control problem; 3) Theoretical understanding and mathematical proofs ,
i
e., the convergence guarantee and acceleration analysis are provided for the interpretability of the distributional RL.
Related Work
Quadrotor Uncertainty Modelling: Precise dynamics modeling in quadrotor autonomous navigation is challenging, since a great variety of aerodynamic effects, such as unknown drag coefﬁcients and wind gusts, can be generated by agile ﬂight in high speeds and accelerations.
Data-driven approaches, such as Gaussian Processes (GP) [1, 10] and neural networks [11] combined with Model
Predictive Control (MPC), have been shown accurate modelling of aerodynamic effects.
Due to the nonparametric nature of GP, the GP-based approach hardly scale to the large datasets of complex environments [12].
While neural network-based approaches learn the nonlinear dynamical effects more accurately [13], achieving adaptability and robustness from these learning-based approaches are still ongoing challenges.
One important reason is that these training datasets are collected from simulation and/or real-world historical records, from which it is difﬁcult to fully describe the complexity in their environments [14].
Reinforcement Learning for uncertainty:
Compared with the existing GP-based and neuralnetwork-based approaches, RL, an adaptive and interactive learning approach, is introduced to model highly dynamic uncertainties in recent work [15].
Distributional RL constructs the entire distributions of the action-value function instead of the traditional expectation, where, to some extent, it addresses the key challenge of traditional RL, i.e., biasing the actions with high variance values in policy optimization [9].
Since some of these values will be overestimated by random chance [16], such actions should be avoided in risk-sensitive or safety-critical applications such as autonomous quadrotor navigation.
More importantly, recent work [17] shows that the distributional
RL has better interpretability, where we can understand or even accelerate the entire RL training process.
This is an encouraging trend, especially for safety-critical applications such as quadrotor autonomous navigation, towards closing the gap between theory and practice in distributional RL.
Control Certiﬁcate Techniques:
Control certiﬁcate techniques guarantee the synthesis of control policies and certiﬁcates [18], where the controller can optimize control policies whilst ensuring the satisfaction of the certiﬁcate properties.
A Control Lyapunov Function (CLF)-based controller [19] ensures that the system state is Lyapunov stable, and a Control Barrier Function (CBF)-based controller [20] ensures that the system state is maintained in deﬁned safety sets given by the barrier function.
Contraction is a property of the closed-loop system [7].
[21] reformulates the CCM certiﬁcate as a Linear Matrix Inequality problem and solves using Sum-of-Squares (SoS).
However, the SoS-based approach cannot extend to general robot systems since an assumption is that the system dynamics need to be polynomial equations or can be approximated as polynomial equations.
Then [22, 23, 24] propose a contraction-metric-based control framework, which extends neural networks to certiﬁcate learning for contraction metrics.
Moreover, based on the framework proposed by Tsukamoto et al., [8, 25, 26, 27, 28] are used to address higher dimensional control problems.
Preliminaries and notations
Nominal Quadrotor Dynamic Model and its Control-afﬁne Form: the quadrotor is assumed as a six Degrees of Freedom (DoF) rigid body of mass m , i.e., three linear motions and three angular motions [1].
Different from [29, 30], the aerodynamic effect (disturbance) e f is integrated into the quadrotor dynamic model as follows [10]:
[Equation]
where P WB , V WB and q WB are the position, linear velocity and orientation expressed in the world frame, and ω B is the angular velocity expressed in the body frame (more detailed deﬁnitions please see [10]).
Then we reformulate Equation 1 in its control-afﬁne form: ˙ x
= f ( x , e fk ) + g ( x ) u + w ,
Where f : R n (cid:55)→ R n and g : R n (cid:55)→ R n × m are assumed by the standard as Lipschitz continuous [31, 8].
x = [ P WB , V WB , q WB , ω B ] T ∈ X , u = T i ∈ U and w ∈ W are the state, input and additive uncertainty of the dynamic model, where T i is the thrust of the i -th ( i ∈ [0 , 3] ) motor, X ⊆ R n , U ⊆
R n u and W ⊆ R n w are compact sets as the state, input and uncertainty space, respectively.
c is the collective thrust c
= [0 , 0 , (cid:80) T i ] T .
e fk is the aerodynamic effect estimated by VID-Fusion [32] in the k -th timestamp.
Given an input : R ≥
(cid:55)→ U and an initial state x 0 ∈ X , our goad is to design a quadrotor tracking controller u such that the state trajectory x can track any reference state trajectory x ref (satisﬁed the quadrotor dynamic limits) under a bounded uncertainty w : R ≥ [-
-]0 (cid:55)→ W .
Control Contraction Certiﬁcates: most existing methods on quadrotor trajectory tracking have so far been demonstrated in stabilization, i.e., a ﬁxed-point-tracking controller [31, 33, 14].
These ﬁxed-point-tracking controllers are relatively efﬁcient in a low-speed and low-variance trajectory.
However, to track a high-speed and high-variance trajectory in agile ﬂight, we need a broader speciﬁcation [34] rather than Lyapunov guarantees that simply stabilize in a ﬁxed point.
Contraction theory [7] gives an analysis on the convergence evolution between the pairs of close trajectories in nonlinear systems, i.e., incremental stability.
When extended to a control-afﬁne system, the change rate is deﬁned as δ ˙ x
= A ( x , u ) δ x + B ( x ) δ u , where A ( x , u ) :
=
∂ f ( x ) ∂ x + (cid:80) mi =1 u
i ∂b i
∂
x , B ( x ) =
g ( x ) , b i is the i -th vector of B , and u i is the i -th element of u .
Just as a CLF is an extension from the idea of Lyapunov function V ( x ) : X (cid:55)→ R , the contraction theory has been extended to a CCM M ( x ) :
X (cid:55)→ R n × n , i.e., a distance measurement between neighboring trajectories (a contraction metric) that shrinks exponentially, for tracking controller design [35, 36, 37].
As the matrix-valued function M ( x ) denotes a symmetric and continuously differentiable metric, δ x T Mδ x deﬁnes a Riemannian manifold [38, 7].
If there exists M {+(cid:31) 
+} (uniformly positive deﬁnite) and exponential convergence of δ x to 0 , the Riemannian inﬁnitesimal length will converge to 0 , then the system is contraction.
We have the following theorem:
Theorem 1 (Contraction Metric [7, 34] ) :
The Lie derivative ∂ v ( · ) of a matrix-valued function M ( x m ) along a vector v ∈ R n is ∂ v M := (cid:80)
ni =1 ∂M∂x mi ˙ v i . In a control-afﬁne system, if
The simulated speed is set as 0 . 4 . there exists: ˙
M + sym( M ( A + BK )) + 2 λM ≺ 0 , then the inequality (cid:107) x ( t ) − x ref ( t ) (cid:107) ≤
Re − λt (cid:107) x (0) − x ref (0) (cid:107) with ∀ t ≥ 0 , R ≥ 1 and λ > 0 holds, where
A =
A ( x , u ) and B = B ( x ) are deﬁned above, and ˙ M = ∂
f ( x )+ g ( x ) u M = (cid:80) ni
[-=
-]=1 ∂M∂x mi ˙ x i , sym( M ) =
M + M T , and x ref ( t ) is a sequential reference trajectory.
Therefore, we say the system is contraction.
(Proof see Supplementary part A)
Overview of the Control Framework: the overall framework is shown in Fig.
The aim of this work is to design a quadrotor controller, which combines wind estimation - i.e., VID-Fusion [32], for tracking the reference trajectory x ref ( t ) of the nominal model (Equation 1) under highly variable aerodynamic disturbances.
We propose a CCM-based MPC, where the parameter θ corresponds to weights of a neural network that estimates the uncertainty ∆ 1 and ∆ 2 in the CCM dynamics.
Combining with the neural network, a Distributional-RL-based Estimator, i.e., QuaDUE, is used to learn the uncertainties adaptively whilst satisfying the constraints of the contraction metric.
Distributional Reinforcement Learning for CCM
Quantile-approximation-based Distributional-reinforced Uncertainty Estimator: in policy evaluation setting, given a 5-tuple Markov Decision Process [39]: MDP := (cid:104) S, A, P, R, γ (cid:105) (where
S , A , P , R and γ ∈ [0 , 1] are the state spaces, action spaces, transition probability, immediate reward function and discount rate), a distributional Bellman equation deﬁnes the state-action distribution Z and the Bellman operator T π as [6, 40]: T π Z ( s , a ) D :=
R ( s , a ) + γZ ( s (cid:48) , a (cid:48) ) , where s ∈ S and a ∈ A are the state and action vector.
π is a deterministic policy under the policy evaluation.
In policy control setting, based on the quantile approximation [40], the distributional Bellman optimality operator T is deﬁned as: T Z ( s , a ) D :=
R ( s , a ) + γZ ( s (cid:48) , arg max a (cid:48)
E p ,R [ Z ( s (cid:48) , a (cid:48) )]) where
Z θ ( s , a ) :=
[Equation]
{+N N (cid:80) i =
 δ q i ( s+} , a ) ∈ Z Q is a quantile distribution mapping one state-action pair ( s, a ) to a uniform probability distribution supported on q i .
Z Q is the space of quantile distribution within N supporting quantiles.
δ z denotes a Dirac with z ∈ R .
Then the maximal form of the Wasserstein metric is proved to be a contraction in [6] as:
[Equation]
where W p , p ∈ [1 , ∞ ] denotes the p -Wasserstein distance.
Π W 1 is a quantile approximation under the minimal 1-Wasserstein distance W 1 . The maximal form of the p -Wasserstein metric is deﬁned as − d p := sup W p ( Z 1 , Z 2 ) .
Equation 2 illustrates that, for a ﬁxed policy π , the Bellman operator T π over value distribution Z 1 and Z 2 is a contraction in the maximal form of the Wasserstein metric, which is an important support for the convergence guarantee below, i.e., Proposition 2 (Policy Evaluation) .
Distributional-RL-based Estimator for CCM Uncertainty: denoting ˙
M + sym( M ( A + BK )) in Theorem 1 by C m ( x , x ref , u ref , θ nominal − metric ) , where θ nominal − metric is the parameter of the neural network in the contraction metric learning [8] for the nominal function Equation 1 with e f = 0 , then the truth C m under uncertainty becomes: C m =
(cid:80) ni
=
∂M∂x
i ˙ x
i + sym( M ( A + BK ))
+ ∆ M 1 ( x ) + ∆ M 2 ( x ) u
= C m + ∆ M 1 ( x )
+ ∆ M 2 ( x ) u .
We employ an agent of QuaDUE, the detailed algorithm for which is given in Supplementary Algorithm 1, to estimate the uncertain terms in (cid:98) C m : ∆ M 1 ( x ) and ∆ M 2 ( x ) .
Therefore, an estimation (cid:98) C m is constructed by: (cid:98) C m =
C m + µ Mθ rl , 1 ( x ) + µ Mθ rl , 2 ( x ) u , where [ µ Mθ rl , 1 ( x ) , µ Mθ rl , 2 ( x )] - i.e., a symmetric matrix - is the output (action) of QuaDUE.
θ rl is again the neural network parameters.
Then our goal of the Distributional-RL-based Estimator - i.e., QuaDUE - is clear: learn a policy µ M 1 , µ M 2 such that the estimation (cid:98) C m as close as to the true value C m .
Notice here that we use the same neural network θ rl for µ Mθ rl , 1 ( x ) and µ Mθ rl , 2 ( x ) .
In Section 5 below, we will demonstrate our reasoning and give proof.
A framework of QuaDUE is illustrated in Fig.1.
The QuaDUE learns a policy which combines with the CCM uncertainties ∆ M 1 ( x ) , ∆ M 2 ( x ) and other dynamic constraints.
Then we design the reward function as R t +1 ( s, a, θ )
=
R contraction ( s, a, θ ) + R track ( s, a ) , where θ ∈ Θ represents the neural network parameters.
Thus, the reward function is deﬁned in detail as:
[Equation]
where m , m are hyper-parameters, H 1 and H 2 are positive deﬁnite matrices, and [ A ] ND is for penalizing positive deﬁniteness
where [ A ] ND =
iff.
A ≺ 0 , and [ A ] ND ≥ 0 iff.
A (cid:23) 0 .
These QuaDUE outputs, i.e., µ Mθ rl , 1 ( x ) and µ Mθ rl , 2 ( x ) , are then fed into the CCM constraints derived from the nominal model.
Speciﬁcally, the estimated (cid:98) C m is used as the best guess of C m for MPC to satisfy the true constraints.
The MPC formulation combining with CCM constraints are shown in
Fig.
1, where the learned uncertainties are used to construct the contraction constraints of the MPC.
Then the MPC is speciﬁed to a quadratic optimization formulation and implemented using CasADi [41] and ACADOS [42].
Certiﬁed Learning on a Contraction Metric for the Nominal Model: as shown in Fig.
1, a certiﬁed learning controller proposed in [34] is constructed to learn a Contraction Metric for the nominal model (Equation 1).
An expert control policy is employed by solving a nonlinear MPC problem [10]. The contraction metric M and control policy µ is represented as two neural networks [8], where the output of the metric network A and the metric is achieved M = sym ( A ) . The parameters of the metric and policy networks are optimized simultaneously in the training process, where the loss function is: L =
L metric + L policy = (1 /N train )( (cid:80) i [ mI − M ] ND ( x i ) + (cid:80) i
[ M + mI ] ND ( x i ) + (cid:80) i
[ (cid:98) C m + 2 λM ]
ND ( x i , µ i ) + (1 /N train ) (cid:80) i (cid:107) µ i − µ expert,i (cid:107) .  Properties of QuaDUE: Convergence and Acceleration analysis In this section, the properties of the proposed Distributional-RL-based Estimator - i.e., QuaDUEare analyzed, including convergence guarantees and training acceleration analysis.
In particular, we empirically verify a more stable and accelerated convergence process than traditional RL algorithms (shown in Section 6), and demonstrate a theoretical understanding of the proposed QuaDUE.
Convergence Analysis of QuaDUE:
Equation 2 shows that the Bellman operator T π is a pcontraction under the p -Wasserstein metric − d p [6].
The Wasserstein distance W p between a distribution Z and its Bellman update T π Z can be minimized iteratively in Temporal Difference learning.
Therefore, we present the convergence analysis of QuaDUE for Policy Evaluation and Policy
Improvement , respectively.
Proposition 2 (Policy Evaluation [40, 14]) :
Given a deterministic policy π , a quantile approximator Π W 1 and Z k +
( s , a ) =
Π W 1 T π Z k
( s , a ) , the sequence Z k ( s , a ) converges to a unique ﬁxed point ∼
Z π under the maximal form of ∞ -Wasserstein metric − d ∞ .
See Supplementary part A for Proof.
Proposition 3 (Policy Improvement [14]) :
Denoting an old policy by π old and a new policy by π new , there exists E [ Z ( s, a )] π new ( s, a ) ≥ E
[ Z ( s, a )]
π old ( s, a ) , ∀ s ∈ S and ∀ a ∈ A .
See Supplementary part A for Proof.
Based on Proposition 2 and Proposition 3 , we are ready to present Theorem 4 to demonstrate the convergence of QuaDUE.
Theorem 4 (Convergence) :
Denoting the policy of the i -th policy improvement by π i , there exists π i → π ∗ , i → ∞ , and E [ Z k ( s, a )] π ∗ ( s,
a ) ≥ E
[ Z k ( s, a )]
π i ( s, a ) , ∀ s ∈ S and ∀ a ∈ A .
See
Supplementary part A for Proof.
Training Acceleration of QuaDUE: training acceleration is investigated for our QuaDUE in order to guarantee a more stable and predictable RL training process.
For implementation execution, we use Kullback-Leibler (KL) divergence [43] instead of Wasserstein Metric.
KL divergence is proven and implemented ’usable’ in [6].
Thus we use KL divergence to analyze the acceleration in RL training process, which is closer to the real implementation.
Similar to [14], we denote J θ ( s, a )
=
D KL ( p s,a , q s,aθ ) as the histogram distributional loss , where p s,a ( x ) and q s,aθ are the true and approximated density function of Z ( s, a ) .
We denote
E J ( θ ) =
E ( s,a ) ∼ ρ π
[ J θ ( s, a )] is the expectation of J θ , where ρ π is the generated distribution under the deterministic policy π .
Since the unbiased gradient estimation of KL divergence, there exits
E ( s,a ) ∼ ρ π
(cid:107)∇ J θ ( p µs,a , q s,aθ ) − ∇ E J ( θ ) (cid:107) 2 ] = κσ 2 , where we represent the approximation error between p µs,a and q s,aθ .
A good degree of the approximation leads to a small κ , which then effects the training acceleration of the distribution RL [17], as shown in Theorem 5 .
Next we deﬁne a stationary-point for the derivative of E J ( θ )
[17]: if there exits (cid:107)∇ E J ( θ T ) (cid:107) ≤ τ ( τ ∈ (0 , 1) ), the updated parameters θ T after T steps is a ﬁrst-order τ -stationary point.
Denote F(s) as the feature on each state s and let the support of Z ( s, a ) have k partitions.
Assuming (cid:107) F(s) (cid:107) <
l , we immediately present Theorem 5 for the training acceleration.
Theorem 5 (Training Acceleration)
In the training process of the distribution RL (i.e., QuaDUE):
1) Let sampling steps T = 4 kl 2 E J
( θ 0 )
/τ 2 , if there exists κ ≤ 0 .
25 τ 2 /σ 2 , the J θ optimized by stochastic gradient descend converges to a τ -stationary point. Let sampling steps T
= kl 2 E J ( θ 0 ) /
( κτ 2 ) , if there exists κ > 0 .
25 τ 2 /σ 2 , the J θ optimized by stochastic gradient descend will not converge to a τ -stationary point.
[-∥ 
-]
(Proof see Supplementary part
[-(cid:80) T − 1 t =
-]
A
[-(cid:80) T − 1 t =
-]
Theorem 5 demonstrates two cases of training acceleration in our QuaDUE.
In (1) of Theorem 5 , the convergence of QuaDUE is accelerated where there is a low approximation error κ between p s,a ( x ) and q s,aθ .
In (2) of Theorem 5 , a relatively larger κ > 0 .
25 τ 2 /σ 2 will give no guarantee for acceleration convergence.
But if (1 /T ) (cid:80) T − 1 t [-=
-]=0 E [ (cid:107)∇ E J ( θ t ) (cid:107) 2 ] ≤ 4 κτ 2 is satisﬁed, the distributional RL still converges to the stationary optimization point.
This understanding from theoretical analysis is consistent with our empirical experiment results in Section 6: when we add small external forces (disturbances) in the speciﬁc scenarios Fig.
4, the performance of the two distributional RL - i.e., ‘QuaDUE-CCM’ and ‘QuaDRED-MPC’ - is less than ‘DDPG-CCM’.
The cases in [40] also verify this observation from simple Atari games.
Evaluation of Performance
To evaluate the performance of the proposed QuaDUE-CCM, we compare against the convergence performance of DDPG [9], C51-DDPG [6] in training process, the tracking performance of GP-MPC [1, 10], DDPG-CCM and QuaDRED-MPC [14] under highly variable aerodynamic disturbances (uncertainties).
The parameters of our QuaDUE-CCM are summarized in Tableaccording to the benchmark works (see [1, 14, 44]).
The parameters ω c, 1 , ω c, 2 and ω c, 3 in Equation 3 are {+set as 1 e − 4 , 1 e − 4 and 1 e − 4 [8]. The matrices H 1 and H 2 in Equation 3 are chosen as H 1 = diag { 2 . 5 e − 2 , 2 . 5 e − 2 , 2 . 5 e − 2 , 1 e − 3 , 1 e − 3 , 1 e − 3 , 2 . 5 e − 3 , 2 . 5 e − 3 , 2 . 5 e − 3 , 2 . 5 e − 3 , 1 e − 5 , 1 e − 5 , 1 e − 5 } and H 2 = diag { 1 . 25 e − 4 , 1 . 25 e − 4 , 1 . 
+}
e − 4 , 1 . 25 e − 4 } , respectively [14]. Comparative Performance of QuaDUE Training: we have two main training process: ﬁrstly, we learn a Contraction Metric for the nominal model (shown in Fig. 1), where both metric and policy networks are 2 hidden layers of 64 neurons and N train = 1 e 4 .
Then we operate a training process by generating external forces in RotorS [45], where the programmable external forces are in the horizontal plane with range [-3,3] ( m/s 2 ).
The training process has 1000 iterations where the quadrotor state is recorded at 16 Hz .
The convergence curves of the training are illustrated in Fig.
3, external forces are generated by Kino-JSS [10].
where we show the comparative training performance of DDPG, C51-DDPG and our QuaDUE.
Our performance illustrates that the two distributional RL approaches, C51-DDPG and QuaDUE, outperform the traditional DDPG RL approach, whilst our proposed QuaDUE achieves the largest cumulative return.
More importantly, our QuaDUE maintains the highest convergence speed, which again veriﬁes the theoretical guarantees demonstrated in Section 5.
Comparative Performance of QuaDUE-CCM in Variable Aerodynamic Disturbance: we evaluate our proposed QuaDUE-CCM with variable aerodynamic forces generated in ZONE 1 and ZONE 2 of the speciﬁc scenario, as described in Fig.
We compare the tracking position errors and derivatives of tracking error with different and opposite heading aerodynamic forces with [0.0, 1.5, 0.0] m/s 2 and [0.0, -1.0, 0.0] m/s 2 in ZONE 1 and ZONE 2, respectively.
As shown in Fig.
5, the three benchmarks are: 1) GP-MPC, a state-of-the-art trajectory tracking method [1, 10]; 2) DDPG-CCM, an adaptive approach extending from RL-CBF-CLF-QP [31] to solve the CCM uncertainties along with a nonlinear MPC; 3) QuaDRED-MPC, a novel learningbased MPC for uncertainties combining with a distributional RL agent and a nonlinear MPC [14]. [Table] D
[Table]
In Fig.
5, the two distributional-RLestimator-based frameworks (i.e.,
QuaDRED-MPC andQuaDUE-CCM) reacttothesudden aerodynamic effects more sufﬁciently than the traditional RL-based estimator (i.e., DDPG).
Among the two distributionalbased methods, the QuaDRED-MPC’s trackingperformance is slightly better (accumulatederror 6 .
73% less) than our proposed QuaDUE-CCM.
This is mainly because the QuaDRED-MPC assesses the environment uncertainties and then directly feeds the uncertainty estimation into the nominal model while the QuaDUE-CCM is an indirect method required to satisfy the contraction metrics, which makes the tracking actions more conservative.
However, based on the contraction theory, our proposed QuaDUE-CCM has 20 .
7% bigger derivative of tracking error than QuaDRED-MPC and the contraction rate is Ce − λt with C = 3 .
59 and λ = 1 . 046 .
This again means that, under variable uncertainties, our QuaDUE-CCM converges to a reference trajectory in a higher contraction rate (exponential contraction) both with theoretical (contraction theory) and empirical guarantees.
[Table]
larger and more complex external forces, i.e., 0.0] ( m/s 2 ), are generated in ZONE 1 and ZONE 2, respectively, as shown in Fig.
In Table 2, we compare the success rate, operation time, accumulated tracking error and contraction rate with the corresponding variable external forces in ZONE 1 and ZONE 2.
Our results show that RL-based methods are not always better, especially with relatively small external forces.
For example, the success rate of RL-based methods (i.e., ‘DDPG + CCM’, ‘QuaDRED-MPC’ and ‘QuaDUE-CCM’) is lower than ‘GP-MPC’.
We also ﬁnd that the two distributional-RL-based methods have close tracking errors, where in some cases ‘QuaDRED-MPC’ is sightly better than ‘QuaDUE-CCM’.
However, compared to ‘QuaDRED-MPC’, our ‘QuaDUE-CCM’ achieves 302%, 312% and 335% improvements in contraction rate.
Conclusion
In this paper, we propose a precise and reliable trajectory tracking framework, QuaDUE-CCM, for quadrotors in dynamic and unknown environments with highly variable aerodynamic effects.
QuaDUE-CCM combines a distributional-RL-based uncertainty estimator and CCMs to address large and variable uncertainties on quadrotor tracking.
A Quantile-approximation-based
Distributional-reinforced Uncertainty Estimator, QuaDUE, is proposed to learn the uncertainties of contraction metrics adaptively, where the convergence and acceleration in the RL training process are analyzed from theoretical perspectives.
Based on the contraction theory, CCMs are used to guarantee exponential convergence to any feasible reference trajectory, which signiﬁcantly improve the reliability of our tracking systems.
We empirically demonstrate that our proposed approach can track agile trajectories with at least 36.2% improvement under large aerodynamic effects, where experimental data can favourably verify our theoretical results.
Limitation
As mentioned in Sections 5 & 6, in comparison with GP-MPC, our QuaDUE-CCM performs weakly under small disturbances.
The potential reasons include QuaDUE-CCM’s indirectly variable uncertainties modelling and the relatively conservative outputs caused by its theoretical guarantees.
Our ongoing work concerns incorporating direct uncertainty estimation, i.e., adding uncertainties into the controller directly, and certiﬁcate-based uncertainty estimation to improve the performance under relatively small uncertainties.
Another limitation is the computational complexity, where we only use 16 Hz for the CCM uncertainty estimation, whereas GP-MPC uses 50 Hz.
To improve this, QuaDUE-CCM may be deployed on dedicated hardware, for example, via FPGA implementation.
Ongoing work concerns evaluating performance in real-world ﬂight tests, during which our system will face a greater variety of aerodynamic disturbances. Acknowledgments
This work was partially supported by UKRI [grant number NE/T011467/1].