E NFORCING ROBUST CONTROL GUARANTEES WITHIN NEURAL NETWORK POLICIES
Anonymous authors
Paper under double-blind review
A BSTRACT
When designing controllers for safety-critical systems, practitioners often face a challenging tradeoff between robustness and performance.
While robust control methods provide rigorous guarantees on system stability under certain worst-case disturbances, they often result in simple controllers that perform poorly in the average (non-worst) case.
In contrast, nonlinear control methods trained using deep learning have achieved state-of-the-art performance on many control tasks, but often lack robustness guarantees.
In this paper, we propose a technique that combines the strengths of these two approaches: a generic nonlinear control policy class, parameterized by neural networks, that nonetheless enforces the same provable robustness criteria as robust control.
Speciﬁcally, we show that by integrating custom convex-optimization-based projection layers into a nonlinear policy, we can construct a provably robust neural network policy class that outperforms robust control methods in the average (non-adversarial) setting.
We demonstrate the power of this approach on several domains, improving in performance over existing robust control methods and in stability over (non-robust) RL methods.
I NTRODUCTION
The ﬁeld of robust control, dating back many decades, has been able to provide rigorous guarantees on when controllers will succeed or fail in controlling a system of interest.
In particular, if the uncertainties in the underlying dynamics can be bounded in speciﬁc ways, these techniques can produce controllers that are provably robust even under worst-case conditions.
However, as the resulting policies tend to be simple (i.e., often linear), this can limit their performance in typical (rather than worst-case) scenarios.
In contrast, recent high-proﬁle advances in deep reinforcement learning have yielded state-of-the-art performance on many control tasks, due to their ability to capture complex, nonlinear policies.
However, due to a lack of robustness guarantees, these techniques have still found limited application in safety-critical domains where an incorrect action (either during training or at runtime) can substantially impact the controlled system.
In this paper, we propose a method that combines the guarantees of robust control with the ﬂexibility of deep reinforcement learning (RL).
Speciﬁcally, we consider the setting of controlling nonlinear, time-varying systems with unknown dynamics, but where (as is common in robust control) these dynamics can be bounded in ways amenable to obtaining provable performance guarantees.
Our work builds upon speciﬁcations provided by traditional robust control methods in these settings, constructing a new class of nonlinear policies that are parameterized by neural networks, but that are nonetheless provably robust .
In particular, we project the outputs of a nominal (deep neural networkbased) controller onto the space of stabilizing actions characterized by robust control speciﬁcations.
The resulting nonlinear control policies are trainable using standard approaches in deep RL, and are guaranteed to be stable under the same worst-case conditions as the original robust controller.
We describe our proposed deep nonlinear control policy class and derive efﬁcient, differentiable projections for this class under various models of system uncertainty common in robust control.
We demonstrate our approach on several different domains, including synthetic linear differential inclusion (LDI) settings, the cart-pole task, a quadrotor domain, and a microgrid domain.
Although these domains are simple by modern RL standards, we show that purely RL-based methods often produce unstable policies in the presence of system disturbances, either during or after training.
In contrast, we show that our method remains stable even when worst-case disturbances are present, while improving upon the performance of traditional robust control methods.
R ELATED WORK
We employ techniques from robust control, (deep) RL, and differentiable optimization to learn provably robust nonlinear controllers.
We discuss these areas of work in connection to our approach.
Robust control.
Robust control is concerned with the design of feedback controllers for dynamical systems with modeling uncertainties and/or external disturbances (Zhou and Doyle, 1998; Ba¸sar and Bernhard, 2008), in particular aiming to design controllers with guaranteed performance under worst-case conditions.
Many classes of robust control problems in both the time and frequency domains can be formulated using linear matrix inequalities (LMIs) (Boyd et al., 1994; Kothare et al., 1996); for reasonably-sized problems, these LMIs can then be solved using off-the-shelf numerical solvers based on interior-point method or ﬁrst-order (gradient-based) methods.
However, providing these stability guarantees often requires the use of simple (often linear) controllers, which greatly limits their average-case performance.
Our work seeks to improve performance via nonlinear controllers that nonetheless retain the same stability guarantees.
Reinforcement learning (RL).
In contrast to robust control, RL (and speciﬁcally, deep RL) is not restricted to simple controllers or problems where the system dynamics are known a priori.
Instead, deep RL seeks to learn an optimal control policy, represented by a neural network, by directly interacting with an unknown environment.
These methods have shown impressive results in a variety of complex control tasks (Mnih et al., 2015; Akkaya et al., 2019); see Bu¸soniu et al.
(2018) for a survey.
However, due to its lack of safety guarantees, deep RL has been predominantly applied to simulated environments or highly-controlled real-world problems, where potential system failures are either not costly or not possible.
Efforts to address the lack of safety and stability in RL fall into two main categories.
The ﬁrst tries to combine control-theoretic ideas, predominantly robust control, with the nonlinear control policy beneﬁts of RL (Morimoto and Doya, 2005; Abu-Khalaf et al., 2006;
Feng et al., 2009; Liu et al., 2013; Wu and Luo, 2013; Luo et al., 2014; Friedrich and Buss, 2017; Pinto et al., 2017; Jin and
Lavaei, 2018; Zhang et al., 2019; Chang et al., 2019; Han et al., 2019).
For example, RL has been used to address stochastic stability in H ∞ control synthesis settings by jointly learning Lyapunov functions and policies in these settings (Han et al., 2019). In deterministic stability settings (the setting we consider here), as an example, RL has been used to address H ∞ control synthesis for continuous-time systems via min-max differential games, in which the controller and disturbance are the “minimizer” and “maximizer” (Morimoto and Doya, 2005).
The other main category of methods addressing this challenge is safe RL.
These methods are able to learn control policies while maintaining some notion of safety either during (Berkenkamp et al., 2017) or after (Taleghan and Dietterich, 2018) learning.
Typically, these methods attempt to restrict the RL algorithm to a safe region of the state space by making strong assumptions about the smoothness of underlying dynamics, such as the fact that the dynamics can be modeled as a
Gaussian process (GP) (Turchetta et al., 2016;
Akametalu et al., 2014) or are Lipschitz continuous (Berkenkamp et al., 2017; Wachi et al., 2018).
This framework is in theory more general than our approach, which requires employing stringent uncertainty bounds (e.g. state-control norm bounds) from robust control.
However, there are two key beneﬁts to our approach.
First, norm bounds or polytopic uncertainty can accommodate sharp discontinuities in the continuous-time dynamics.
Secondly, convex projections onto these sets (as used in our method) scale polynomially with the state-action size, whereas GPs in particular scale exponentially (and are therefore difﬁcult to extend to high-dimensional problems).
Differentiable optimization layers.
There has been a great deal of recent work in creating differentiable optimization-based layers for use in neural networks: for instance, layers for quadratic programming (Amos and Kolter, 2017), SAT solving (Wang et al., 2019), submodular optimization (Djolonga and Krause, 2017; Tschiatschek et al., 2018), cone programs (Agrawal et al., 2019), and other classes of optimization problems (Gould et al., 2019).
These layers can be used to construct neural networks with useful inductive bias for particular domains (just as, e.g., convolutional layers yield inductive bias for images) or to enforce that neural networks obey hard constraints dictated by the settings in which they are used.
We create fast, custom differentiable optimization layers for the latter purpose, namely, to project neural network outputs into a set of certiﬁably stabilizing actions.
B ACKGROUND ON LQR AND ROBUST CONTROL SPECIFICATIONS
In this paper, our aim is to control nonlinear (continuous-time) dynamical systems of the form
[Equation]
where x ( t ) ∈
R s denotes the state at time t ; u ( t ) ∈ R a is the control input; w ( t ) ∈ R
d captures both external (possibly stochastic) disturbances and any modeling discrepancies; ˙ x ( t ) denotes the time derivative of the state x at time t ; and
A ( t ) ∈ R s × s , B ( t ) ∈
R s × a , G ( t ) ∈
R s × d .
This class of models is referred to as linear differential inclusions (LDIs); however, we note that despite the name, this class does indeed characterize nonlinear systems, as, e.g., w ( t ) can depend arbitrarily on x ( t ) and u ( t ) (though we omit this dependence above for brevity).
Within this class of models, it is often possible to construct robust control speciﬁcations certifying system stability.
Given these speciﬁcations, our proposal is to learn nonlinear (deep neural network-based) policies that provably satisfy these speciﬁcations while optimizing some objective of interest.
We start by giving background on the robust control speciﬁcations and objectives considered in this work.
3. R OBUST CONTROL SPECIFICATIONS
In the continuous-time, inﬁnite-horizon settings we consider here, the goal of robust control is often to construct a time-invariant control policy u ( t ) =
π ( x ( t )) , alongside some certiﬁcation that guarantees that the controlled system will be stable (i.e., that trajectories of the system will converge to an equilibrium state, usually x = 0 by convention; see Haddad and Chellaboina (2011) a more formal deﬁnition).
For many classes of systems, 1 this certiﬁcation is typically in the form of a positive deﬁnite Lyapunov function V : R s → R , with V (0)
=
and V ( x ) > {+
+} for all x (cid:54) = 0 , such that the function is decreasing along trajectories, e.g.,
[Equation]
for some design parameter α > 0 .
(This particular condition implies exponential stability with a rate of convergence α . 2 )
For certain classes of bounded dynamical systems, time-invariant linear control policies u ( t ) =
Kx ( t ) , and quadratic Lyapunov functions V ( x )
=
x T Px , it is often possible to construct such guarantees using semideﬁnite programming.
For instance, consider the class of norm-bounded LDIs (NLDIs)
[Equation]
where A ∈ R s × s , B ∈ R s × a , G ∈ R s × d , C ∈ R k × s , and D ∈ R k × a are time-invariant (and known), and the disturbance w ( t ) is arbitrary (and unknown) but obeys the norm bounds above.
For these systems, it is possible to specify a set of stabilizing policies via a set of linear matrix inequalities (LMIs, Boyd et al.
[Equation]
where S ∈ R s × s and Y ∈ R a × s .
For matrices S and Y satisfying (4), K = Y S − 1 and P = S −are then a stabilizing linear controller gain and Lyapunov matrix, respectively.
While the LMI above is speciﬁc to NLDI systems, this general paradigm of constructing stability speciﬁcations using
LMIs applies to many settings commonly considered in robust control (e g., settings with normbounded disturbances or polytopic uncertainty, or H ∞ control settings).
More details about these types of formulations are given in, e.g., Boyd et al.
; in addition, we provide the relevant LMI constraints for the settings we consider in this work in Appendix A.
A slightly more complex formulation involves an additional term in the norm bound, i e.
Cx ( t )+ Du ( t )+ Hw ( t ) , which creates a quadratic inequality in w .
The mechanics of obtaining robustness speciﬁcations in this setting are largely the same as presented here, though with some additional terms in the equations.
As such, as is often done, we assume that H = 0 for simplicity.
3. LQR CONTROL OBJECTIVES
In addition to designing for stability, it is often desirable to optimize some objective characterizing controller performance.
In our setting, we consider the well-known inﬁnite-horizon “linearquadratic regulator” (LQR) performance output, deﬁned as
[Equation]
for some Q ∈ S s × s (cid:23) 0 and R ∈ S a × a (cid:31) 0 .
In the special case of time-invariant linear control policies described above, minimizing the LQR cost subject to the stability constraints can be cast as an SDP and solved using off-the-shelf numerical solvers (a fact that we exploit in our work).
For example, for the NLDI systems described above, we can solve [Equation] [Equation] holds
[Equation]
to obtain an optimal linear time-invariant controller.
(For other systems characterized by LMIs, we would simply replace the constraint above with the relevant LMI for that system.)
E NFORCING ROBUST CONTROL GUARANTEES WITHIN NEURAL NETWORKS
We now present the main methodological contribution of our paper: a class of nonlinear control policies, potentially parameterized by deep neural networks, that is guaranteed to obey the same stability conditions enforced by the robustness speciﬁcations described above.
The key insight of our approach is that, while it is difﬁcult in general to write conditions that globally characterize the stability of a nonlinear controller, given known robustness speciﬁcations, we can create a sufﬁcient condition for stability by simply enforcing that our policy satisﬁes these speciﬁcations at all t .
For instance, given a known Lyapunov function, we can enforce exponential stability by ensuring that our policy sufﬁciently decreases this function (e.g., satisﬁes Equation (2)) at any given x ( t ) .
In the following sections, we present our nonlinear policy class, as well as our general framework for learning provably robust policies using this policy class. We then derive the instantiation of this framework for various settings of interest.
In particular, this involves constructing custom differentiable projections that can be used to adjust the output of a nominal neural network to satisfy desired robustness criteria.
For simplicity of notation, we will often suppress the t -dependence of x , u , and w , but we note that these are continuous-time quantities as before.
4. A PROVABLY ROBUST NONLINEAR POLICY CLASS
Given a dynamical system of the form (1) and a quadratic Lyapunov function V ( x ) = x T Px , let
[Equation]
denote a set of actions that, for a ﬁxed state x ∈ R s , are guaranteed to satisfy the exponential stability condition (2) (even under worst-case realizations of the disturbance w ).
We note that this set is non-empty if P satisﬁes the relevant system-speciﬁc LMI constraints characterizing robust linear time-invariant controllers (e.g., system (4) for NLDIs), as there is then some K corresponding to P such that Kx ∈ C ( x ) for all states x .
Using this set of actions, we then construct a robust nonlinear policy class that projects the output of some neural network onto this set.
More formally, consider an arbitrary nonlinear (neural networkbased) policy class ˆ π θ : R s → R a parameterized by θ , and let P ( · ) denote the projection operator for some set ( · ) .
We then deﬁne our robust policy class as π θ : R s → R a , where
[Equation]
We note that this policy class is differentiable, as the projections can be implemented in a differentiable manner (e.g., using convex optimization layers (Agrawal et al., 2019), though we construct efﬁcient custom solvers for our purposes).
Importantly, as all policies in this class satisfy condition (2) for all states x and at all time t , these policies are certiﬁably robust under the same conditions as the original (linear) controller for which V ( x ) was constructed.
Given this policy class and some performance objective (cid:96) (e.g., LQR cost), our goal is to then ﬁnd parameters θ such that the corresponding policy optimizes this objective – i.e., to solve
[Equation]
1: input performance objective (cid:96) //
e.g., LQR cost 2: input stability requirement // e.g., ˙ V ( x ) ≤ − αV
( x ) 3: input optimizer
A // e.g., a planning or RL algorithm constraints // e.g., by optimizing (6) Since π θ is differentiable, we can solve this problem via a variety of approaches, e.g., a modelbased planning algorithm if the true dynamics are known, or virtually any (deep) RL algorithm if the dynamics are unknown.
This general procedure for constructing stabilizing controllers is summarized in Algorithm 1.
While seemingly simple, this formulation presents a powerful paradigm: by simply transforming the output of a neural network, we can employ an expressive policy class to optimize an objective of interest while ensuring the resultant policy will stabilize the system during both training and testing.
We instantiate our framework by constructing relevant sets C ( x ) and their associated (differentiable) projections P C ( x ) for three settings of interest: NDLIs, polytopic linear differential inclusions (PLDIs), and H ∞ control settings.
As an example, we describe this procedure below for NLDIs, and refer readers to Appendix B for corresponding formulations for the additional settings we consider.
4. E XAMPLE : NLDI S
In order to apply our framework to the NLDI setting (3), we ﬁrst compute a quadratic Lyapunov function V ( x ) = x T Px by solving the optimization problem (6) for the given system via semideﬁnite programming.
We then use the resultant Lyapunov function to compute the system-speciﬁc set
C ( x ) , and then create a fast, custom differentiable solver to project onto this set.
4.2. C OMPUTING SETS OF STABILIZING ACTIONS
Given P , we compute C NLDI ( x ) as the set of actions u ∈ R a that, for each state x ∈ R s , satisfy the stability condition (2) at that state under even a worst-case realization of the dynamics (i.e., in this case, even under a worst-case disturbance w ).
The form of the resultant set is given below.
Theorem 1.
Consider the NLDI system (3) , some stability parameter α > 0 , and a Lyapunov function V ( x ) = x T Px with P satisfying Equation (4) .
Assuming P exists, deﬁne
[Equation] NLDI
[Equation]
for all states x ∈ R s .
For all x , C NLDI ( x ) is a non-empty set of actions that satisfy the exponential stability condition (2) .
Further, C NLDI ( x ) is a convex set in u .
Proof.
We seek to ﬁnd a set of actions such that the condition (2) is satisﬁed along all possible trajectories of (3).
A set of actions satisfying this condition at a given x is given by
[Equation]
Let S :
= { w : (cid:107) w (cid:107) 2 ≤ (cid:107)
Cx + Du (cid:107) 2 } .
We can then rewrite the left side of the above inequality as
[Equation]
by the deﬁnition of the NLDI dynamics and the closed-form minimization of a linear term over an (cid:96) 2 ball.
Rearranging yields an inequality of the desired form.
We note that by deﬁnition of the speciﬁcations (4), there is some K corresponding to P such that the policy u
= Kx satisﬁes the exponential stability condition (2); thus,
Kx ∈ C NLDI , and C NLDI is non-empty.
Further, as the above inequality represents a second-order cone constraint in u , this set is convex in u .
We also consider the special case where D = 0 , i.e., where the norm bound on w does not depend on the control action.
This form of NLDI arises in many common settings (e.g., where w characterizes linearization error in a nonlinear system but the dynamics depend only linearly on the action), and is one for which we can compute the relevant projection in closed form (described shortly).
Corollary 1.1.
Consider the NLDI system (3) with D = 0 , some stability parameter α > 0 , and
Lyapunov function V ( x ) =
x T Px with P satisfying Equation (4) .
Assuming P exists, deﬁne
[Equation]
for all states x ∈ R s .
For all x , C NLDI-0 ( x ) is a non-empty set of actions that satisfy the exponential stability condition (2) .
Further, C NLDI-0 ( x ) is a convex set in u .
Proof.
The result follows by setting D = 0 in Theorem 1, and rearranging terms.
As the above inequality represents a linear constraint in u , this set is convex in u .
4.2. D ERIVING EFFICIENT , DIFFERENTIABLE PROJECTIONS
For the general NLDI setting (3), we note that the relevant projection P C NLDI ( x ) represents a projection onto a second-order cone constraint.
As this projection does not necessarily have a closed form, we implement this projection via a custom differentiable optimization solver.
This solver employs an accelerated projected dual gradient method for the forward pass, and employs implicit differentiation through the ﬁxed point equations of this solution method to compute relevant gradients for the backward pass. Derivations and additional details are provided in Appendix C.
In the case where D = 0 , we note that the projection operation P C NLDI-0 ( x ) does have a closed form, and can in fact be implemented via a single ReLU operation.
Speciﬁcally, deﬁning η T := 2 x T PB and ζ := − x T (2 PA + αP ) x − 2 (cid:107) G T Px (cid:107) 2 (cid:107) Cx (cid:107) 2 , we see that
[Equation]
E XPERIMENTS
Having instantiated our general framework, we demonstrate the power of our approach on a variety of simulated control domains. 5 In particular, we evaluate performance on the following metrics:
• Average-case performance: How well does the method optimize the performance objective (i
e., LQR cost) under average (non-worst case) dynamics?
• Worst-case stability: Does the method remain stable even when subjected to adversarial (worst-case) dynamics?
In all cases, we show that our method is able to improve performance over traditional stable controllers under average conditions, while still guaranteeing stability under worst-case conditions.
5. D ESCRIPTION OF DYNAMICS SETTINGS
We evaluate our approach on ﬁve NLDI settings: two synthetic NLDI domains, the cart-pole task, a quadrotor domain, and a microgrid domain.
(Additional experiments for PLDI and H ∞ control settings are given in Appendix I.)
For each setting, we choose a time discretization based on the speed at which the system evolves, and run each episode for 200 steps over this discretization.
In all cases except for the microgrid setting, we use a randomly generated LQR objective where the matrices Q 1 / 2 and R 1 / 2 are drawn i.i.d.
from a standard normal distribution.
L o ss
N L D I ( D = 0 ) Non-robust Methods
Robust Methods
[Figure]
L o ss
N L
{+D I ( D 0 ) 
+}
L o ss
{+C a r t p o l e 
+}
L o ss
Q u a d r o t o r
{+Training epochs 
+}
L o ss
M i c r
o g r i d
Training epochs
[Figure] Synthetic NLDI settings. We generate NLDIs of the form (3) with s = 5 , a = 3 , and d = k = 2 by generating matrices A, B, G, C and D i.i.d.
from normal distributions, and producing the disturbance w ( t ) using a randomly-initialized neural network (with its output scaled to satisfy the norm-bound on the disturbance).
We investigate settings both where D (cid:54) = 0 and where D = 0 .
In both cases, episodes are run for 2 seconds at a discretization of 0.01 seconds.
Cart-pole.
In the cart-pole task, our goal is to balance an inverted pendulum resting on top of a cart by exerting horizontal forces on the cart.
For our experiments, we linearize this system as an NLDI, with D (cid:54) =
(see Appendix D), and add a small additional randomized disturbance satisfying the
NLDI bounds.
Episodes are run for 10 seconds at a discretization of 0.05 seconds.
Planar quadrotor.
In this setting, our goal is to stabilize a quadcopter in the two-dimensional plane by controlling the amount of force provided by the quadcopter’s right and left thrusters.
We linearize this system as an NLDI, with D =
(see Appendix E), and add a small disturbance as in the cart-pole setting.
Episodes are run for 4 seconds at a discretization of 0.02 seconds.
Microgrid.
In this ﬁnal setting, we aim to stabilize a microgrid by controlling a storage device and a solar inverter.
We augment the system given in Lam et al.
(2016) with LQR matrices and NLDI bounds (see Appendix F).
Episodes are run for 2 seconds at a discretization of 0.01 seconds.
5. E XPERIMENTAL SETUP
We demonstrate our approach by constructing a robust policy class (8) for each of these settings, and optimizing this policy class in different ways.
Speciﬁcally, we construct a nominal nonlinear control policy class as ˆ π θ ( x ) =
Kx + ˜
π
θ ( x ) , where K is obtained via robust LQR optimization (6), and where ˜ π θ ( x ) is a feedforward neural network.
To construct the projections P C , we employ the value of P obtained when solving for K .
For the purposes of demonstration, we then optimize our robust policy class π θ ( x ) =
P C (ˆ π θ ( x )) using two different methods: • Robust MBP (ours) : A model-based planner that assumes the true dynamics are known. • Robust PPO (ours) : An RL approach based on PPO (Schulman et al., 2017) that does not assume known dynamics (beyond the bounds used to construct the robust policy class).
Robust MBP is optimized using gradient descent for 1,000 updates, where each update samplesroll-outs.
Robust PPO is trained for 50,000 updates, where each update samples 8 roll-outs; we choose the model that performs best on a hold-out set of initial conditions during training.
We note that while we use PPO for our demonstration, our approach is agnostic to the particular method of training, and can be deployed with many different (deep) RL paradigms.
We compare our robust neural network-based method against the following baselines: • Robust LQR: Robust (linear) LQR controller obtained via Equation (6). • Robust MPC: A model-predictive control algorithm (Kothare et al., 1996) based on statedependent linear matrix inequalities.
(As the relevant LMIs are not always guaranteed to solve, our implementation temporarily reverts to the Robust LQR policy when that occurs.) • RARL:
The robust adversarial reinforcement learning algorithm (Pinto et al., 2017), which trains an RL agent in the presence of an adversary. (We note that unlike the other robust control methods considered here, this method is not provably robust.)
• LQR:
A standard non-robust (linear) LQR controller. • MBP and PPO: The non-robust neural network policy class ˆ π θ ( x ) optimized via a modelbased planner and the PPO algorithm, respectively.
In order to evaluate performance, we train all methods on the dynamical settings described in the previous section, and evaluate them on two different variations of the dynamics:
• Original dynamics:
The dynamical settings described above (“average case”).
• Adversarial dynamics: Modiﬁed dynamics with an adversarial test-time disturbance w ( t ) generated to maximize loss (“worst case”). We generate this disturbance separately for each method described above (see Appendix G for more details). We randomly generate initialization states for all experiments.
For the synthetic NLDI and microgrid settings, these are generated from a standard normal distribution.
For both cart-pole and quadrotor, because our NLDI bounds model linearization error, we must generate initial points within a region where this linearization holds.
In particular, the linearization bounds only hold for a speciﬁed L ∞ ball, B NLDI , around the equilibrium; we use a simple heuristic to construct this ball, and jointly ﬁnd a smaller L ∞ ball, B init , such that there exists a level set L of the robust LQR Lyapunov function with
B init ⊆ L ⊆ B NLDI (details in Appendix H). This guarantees that Rohbust LQR (and by extension our methods), when initialized starting from any point inside B init , will never leave B NLDI – i.e., that our NLDI bounds will always hold throughout the trajectories produced by these methods. 5. R ESULTS
Table 1 shows the performance of the above methods.
We report the integral of the quadratic loss over the prescribed time horizon on a test set of states, or indicate cases where the relevant method became unstable (i.e., the loss became orders of magnitude larger than for other approaches). (Sample trajectories for these methods are also provided in Appendix H.)
These results illustrate the basic advantage of our approach.
In particular, both our Robust MBP and Robust PPO methods show improved “average-case” performance over the other provably robust methods (namely, Robust LQR and Robust MPC). As expected, however, the non-robust
LQR, MBP, and PPO methods often perform better within the original nominal dynamics, as they [Table] Robust MPC in the average case, while remaining stable under adversarial dynamics, whereas the non-robust methods and RARL either go unstable or receive much larger losses.
are optimizing for expected performance, but do not need to consider robustness under worst-case scenarios.
However, when we apply allowable adversarial perturbations (that still respect our disturbance bounds), the non-robust LQR, MBP, and PPO approaches diverge or perform very poorly.
Similarly, the RARL agent performs well under the original dynamics, but diverges under adversarial perturbations in the generic NLDI settings.
In contrast, both of our provably robust approaches (as well as robust LQR) remain stable under even “worst-case” adversarial dynamics .
(We note that the baseline Robust MPC method goes unstable in one instance, though this is due to numerical instability issues, rather than issues with theoretical guarantees.)
Figure 1 additionally shows the performance of all neural network-based methods on the test set over training epochs.
While the robust and non-robust MBP and PPO approaches both converge quickly to their ﬁnal performance levels, both non-robust versions become unstable under the adversarial dynamics very early in the process.
The RARL method also frequently destabilizes during training.
Our Robust MBP and PPO policies, on the other hand, remain stable throughout the entire optimization process , i.e., do not destabilize during either training or testing.
Overall, these results show that our method is able to learn policies that are more expressive than traditional robust methods, while guaranteeing these policies will be stable under the same conditions as robust LQR.
C ONCLUSION
In this paper, we have presented a class of nonlinear control policies that combines the expressiveness of neural networks with the provable stability guarantees of traditional robust control.
This policy class entails projecting the output of a neural network onto a set of stabilizing actions, parameterized via robustness speciﬁcations from the robust control literature, and can be optimized using a model-based planning algorithm if the dynamics are known, or virtually any RL algorithm if the dynamics are unknown.
We instantiate our general framework for dynamical systems characterized by several classes of linear differential inclusions that capture many common robust control settings.
In particular, this entails deriving efﬁcient, differentiable projections for each setting, via implicit differentiation techniques.
We show over a variety of simulated domains that our method improves upon traditional robust LQR techniques while, unlike non-robust LQR and neural network methods, remaining stable even against worst-case allowable perturbations of the underlying dynamics.
We believe that our approach highlights the possible connections between traditional control methods and (deep) RL methods.
Speciﬁcally, by enforcing more structure in the classes of networks we consider, it is possible to produce networks provably satisfying many of the constraints that have typically been thought of as outside the realm of RL.
We hope that this work paves the way for future approaches that can combine more structured uncertainty or robustness guarantees with RL, in order to improve control performance in settings traditionally dominated by classical robust control.
A D ETAILS ON ROBUST CONTROL SPECIFICATIONS
As described in Section 3.1, for many dynamical systems of the form (1), it is possible to specify a set of linear, time-invariant policies guaranteeing inﬁnite-horizon exponential stability via a set of LMIs.
Here, we derive the LMI (4) provided in the main text for the NLDI system (3), and additionally describe relevant LMI systems for systems characterized by polytopic linear differential inclusions (PLDIs) and for H ∞ control settings.
A. E XPONENTIAL STABILITY IN NLDI S
Consider the general NLDI system (3).
We seek to design a time-invariant control policy u ( t ) =
Kx ( t ) and a quadratic Lyapunov function V ( x ) = x T Px with P (cid:31) 0 for this system that satisfy the exponential stability criterion ˙
V ( x ) ≤ − αV ( x ) , ∀ t .
We derive an LMI characterizing such a controller and Lyapunov function, closely following and expanding upon the derivation provided in Boyd et al.
Speciﬁcally, consider the NLDI system (3), reproduced below:
[Equation]
The time derivative of this Lyapunov function along the trajectories of the closed-loop system is
[Equation]
The exponential stability condition ˙ V ( x )
[Equation]
is thus implied by inequality
[Equation]
Additionally, the norm bound on w can be equivalently expressed as
[Equation]
Using the S-procedure
[Equation]
it follows that for some λ ≥ 0 , the following matrix inequality is a sufﬁcient condition for exponential stability:
[Equation]
Using Schur Complements, this matrix inequality is equivalent to
[Equation]
Left- and right-multiplying both sides by P − 1 , and making the change of variables S
= P − 1 ,
Y = KS , and µ = 1 /λ, we obtain
[Equation]
Using Schur Complements again on this inequality
[Equation]
we obtain our ﬁnal system of linear matrix inequalities as
[Equation]
where then K = Y S − 1 and P = S − 1 .
Note that the ﬁrst matrix inequality is homogeneous; we can therefore assume µ = 1 (and therefore, λ = 1 ), without loss of generality.
A. E XPONENTIAL STABILITY IN PLDI S
Consider the setting of polytopic linear differential inclusions (PLDIs), where the dynamics are of the form
[Equation]
Here, A ( t ) ∈ R s × s and B ( t )
∈
R s × a can vary arbitrarily over time, as long as they lie in the convex hull (denoted Conv ) of the set of points above, where A i ∈ R s × s , B i ∈ R s × a for i = 1 , . . .
, L .
We seek to design a time-invariant control policy u ( t ) =
Kx ( t ) and quadratic Lyapunov function
V ( x ) =
x T Px with P (cid:31) 0 for this system that satisfy the exponential stability criterion ˙
V ( x ) ≤ − αV ( x ) , ∀ t .
Such a controller and Lyapunov function exist if there exist S ∈ R s × s (cid:31) 0 and
Y ∈ R a × s such that
[Equation]
where then K = Y S − 1 and P = S − 1 .
The derivation of this LMI follows similarly to that for exponential stability in NLDIs, and is well-described in Boyd et al.
A. H ∞ CONTROL
Consider the following H ∞ control setting with LTI dynamics
[Equation]
where A , B , and G are time-invariant as for the NLDI case, and where we deﬁne L 2 as the set of time-dependent signals with ﬁnite L 2 norm.
In cases with such as these with larger or more unstructured disturbances, it may not be possible to guarantee asymptotic convergence to an equilibrium.
In these cases, our goal is to construct a robust controller with bounds on the extent to which disturbances affect some performance output (e.g., LQR cost), as characterized by the L 2 gain of the disturbance-to-output map.
Speciﬁcally, we consider the stability requirement that this L 2 gain be bounded by some parameter γ > 0 when disturbances are present, and that the system be exponentially stable in the disturbance-free case.
This requirement can be characterized via the condition that for all t and some σ ≥ 0 ,
[Equation]
We note that when E ( x ( t ) , ˙ x ( t ) , u ( t )) ≤ 0 for all t , both of our stability criteria are met.
To see this, note that integrating both sides of (A.12) from 0 to ∞ and ignoring the non-negative terms on the left hand side after integration yields
[Equation]
This is precisely the desired bound on the L 2 gain of the disturbance-to-output map (see Khalil and
Grizzle (2002)).
We also note that in the disturbance-free case, substituting w = 0 into (A.12) yields
[Equation]
where the last inequality follows from the non-negativity of the LQR cost; this is precisely our condition for exponential stability.
We now seek to design a time-invariant control policy u ( t ) =
Kx ( t ) and quadratic Lyapunov function V ( x ) = x T Px with P (cid:31) 0 that satisﬁes the above condition.
In particular, we can write
E
[Equation]
x ( t ) ,
[Equation]
A
[Equation]
) x ( t )
[Equation]
( t ) ,
[Equation]
t
[Equation]
(cid:20) x
[Equation]
t
[Equation]
w ( t ) (cid:21)
[Equation]
t
[Equation]
w ( t ) (cid:21)
[Equation]
(A.15) where
[Equation]
Therefore, we seek to ﬁnd a P ∈ R s × s (cid:31) 0 and K ∈ R s × a that satisfy M 1 (cid:22) 0 , for some design parameters α > 0 and σ > 0 .
Using Schur complements, the matrix inequality M 1 (cid:22) 0 is equivalent to
[Equation]
As in Appendix A.1, we left- and right-multiply both sides by P − 1 , and make the change of variables
S =
P − 1 , Y = KS , and µ = 1 /σ to obtain
[Equation]
Using Schur Complements again, we obtain the LMI
[Equation] [Equation]
where then K = Y S − 1 ,
P = S − 1 , and σ = 1 /µ .
B D ERIVATION OF SETS OF STABILIZING POLICIES AND ASSOCIATED PROJECTIONS
We describe the construction of the set of actions C ( x ) , deﬁned in Equation (7), for PLDI systems (A.9) and H ∞ control settings (A.11).
(The relevant formulations for the NLDI system (3) are described in the main text.)
B. E XPONENTIAL STABILITY IN PLDI S
For the general PLDI system (A.9), relevant sets of exponentially stabilizing actions C PLDI are given by the following theorem.
Theorem B.1.
Consider the PLDI system (A.9) , some stability parameter α > 0 , and a Lyapunov function V ( x ) = x T Px with P satisfying (A.10) .
Assuming P exists, deﬁne
[Equation]
PLDI ( x
[Equation]
  for all states x ∈ R s .
For all x , C PLDI ( x ) is a non-empty set of actions that satisfy the exponential stability condition (2) .
Further, C PLDI ( x ) is a convex set in u .
Proof.
We seek to ﬁnd a set of actions such that the condition (2) is satisﬁed along all possible trajectories of (A.9), i.e., for any allowable instantiation of ( A ( t ) , B ( t )) .
A set of actions satisfying this condition at a given x is given by
[Equation]
Expanding the left side of the inequality above, we see that for some coefﬁcients γ i ∈ R ≥ 0 , i = 1 , . . .
, L satisfying (cid:80)
[Equation]
=
[Equation]
i
[Equation]
t ) =
[Equation]
by deﬁnition of the PLDI dynamics and of the convex hull.
Thus, if we can ensure
[Equation]
then we can ensure that exponential stability holds.
Rearranging this condition and writing it in matrix form yields an inequality of the desired form.
We note that by deﬁnition of the speciﬁcations (A.10), there is some K corresponding to P such that the policy u
= Kx satisﬁes all of the above inequalities; thus, Kx ∈ C PLDI ( x ) , and C PLDI ( x ) is non-empty.
Further, as the above inequality represents a linear constraint in u , this set is convex in u .
We note that the relevant projection P C PLDI ( x ) represents a projection onto an intersection of halfspaces, and can thus be implemented via differentiable quadratic programming (Amos and Kolter, 2017).
B. H ∞ CONTROL
For the H ∞ control system (A.11), relevant sets of actions satisfying the condition (A.12) are given by the following theorem.
Theorem B.2.
Consider the system (A.11) , some stability parameter α > 0 , and a Lyapunov function V ( x ) = x T Px with P satisfying Equation (A.18) .
Assuming P exists, deﬁne
[Equation]
for all states x ∈ R s .
For all x , C H ∞ ( x ) is a non-empty convex set of actions that guarantee condition (A.12) , i.e., that the L 2 gain of the disturbance-to-output map is bounded by γ and that the system is exponentially stable in the disturbance-free case.
Further, C H ∞ ( x ) is convex in u .
Proof.
We seek to ﬁnd a set of actions such that the condition E ( x, ˙ x, u ) ≤ 0 is satisﬁed along all possible trajectories of (A.11), where E is deﬁned as in (A.12).
A set of actions satisfying this condition at a given x is given by
[Equation]
To begin, we note that
[Equation]
We then maximize E over w :
[Equation]
Therefore,
[Equation]
Expanding and rearranging terms, this becomes
[Equation]
We note that by deﬁnition of the speciﬁcations (A.18), there is some K corresponding to P such that the policy u
=
Kx satisiﬁes the conditions above (see (A.17))
; thus, Kx ∈ C H ∞ , and C H ∞ is non-empty.
We note further that C H ∞ is an ellipsoid in the control action space, and is thus convex in u .
We rewrite the set C H ∞ ( x ) such that the projection P C H ∞ ( x ) can be viewed as a second-order cone projection, in order to leverage our fast custom solver (Appendix C).
In particular, deﬁning ˜ P = σR , ˜ q
= B T Px , and ˜ r = x T (cid:0) PA + A T P + αP + σQ
[Equation]
T P/
[Equation]
2 ) (cid:1) x , we can rewrite the ellipsoid above as
[Equation]
We note that as ˜ P (cid:31) 0 and ˜ r − ˜ q (cid:62) ˜ P − 1 ˜ q < 0 , this ellipsoid is non-empty (see, e.g., section B.1 in
Boyd and Vandenberghe (2004)).
We can then rewrite the ellipsoid as
[Equation]
where ˜ A = (cid:113)
[Equation]
q .
The constraint (cid:107) ˜
Au + ˜ b (cid:107) 2 ≤ 1 is then a second-order cone constraint in u .
C A FAST , DIFFERENTIABLE SOLVER FOR SECOND - ORDER CONE PROJECTION
In order to construct the robust policy class described in Section 4 for the general NLDI system (3) and the H ∞ setting (A.11), we must project a nominal (neural network-based) policy onto the second-order cone constraints described in Theorem 1 and Appendix B.2, respectively.
As this projection operation does not necessarily have a closed form, we implement it via a custom differentiable optimization solver.
More generally, consider a set of the form
[Equation]
(C.1) for some A ∈ R m × n , b ∈ R m , c ∈ R n , and d ∈ R .
Given some input y ∈ R n , we seek to compute the second-order cone projection P C ( y ) by solving the problem minimize x ∈ R n 2 (cid:107)
[Equation]
(cid:107)subject to (cid:107)
Ax + b (cid:107) 2 ≤ c T x + d.
(C.2)
Let F denote the (cid:96) 2 norm cone, i.e., F := { ( w, t ) | (cid:107) w (cid:107) 2 ≤ t } .
Introducing the auxiliary variable z ∈ R m +1 , we can then rewrite the above optimization problem equivalently as minimize x ∈ R n , z ∈ R m +  2 (cid:107) x − y (cid:107)
+ 1 F ( z ) subject to z = (cid:20)
Ax + b c T x + d (cid:21) =:
Gx + h, (C.3) where for brevity we deﬁne G =
(cid:20) Ac T (cid:21) and h = (cid:20) bd (cid:21) , and where 1 F denotes the indicator function for membership in the set F .
We describe our fast solution technique for computing this projection, as well as our method for obtaining gradients through the solution.
[Equation]
OMPUTING THE PROJECTION
We construct a fast solver for problem (C.3) using an accelerated projected dual gradient method.
Speciﬁcally, deﬁne µ = R m +1 as the dual variable on the equality constraint in Equation (C.3).
The
Lagrangian for this problem can then be written as
[Equation]
and the dual problem is given by max µ min x,z L ( x, z, µ ) .
To form the dual problem, we minimize the Lagrangian with respect to x and z as
[Equation]
We note that the ﬁrst term on the right side is minimized at x (cid:63) ( µ ) =
y + G T µ .
Thus, we see that
[Equation]
For the second term, denote µ = (˜ µ, s ) and z =
(˜ z, t ) .
We can then rewrite this term as
[Equation]
For a ﬁxed t ≥ 0 , the above objective is minimized at ˜ z
= − t ˜ µ/ (cid:107) ˜ µ (cid:107) 2 .
(The problem is infeasible for t < 0 .)
Substituting this minimizer into (C.7) and minimizing the result over t ≥ 0 yields
[Equation]
where the last identity follows from deﬁnition of the second-order cone F .
Hence the negative dual problem becomes
[Equation]
We now solve this problem via Nesterov’s accelerated projected dual gradient method (Nesterov, 2013).
For notational brevity, deﬁne f ( µ ) :
= 12 µ T GG T µ + µ T ( Gy + h ) .
Then, starting from arbitrary µ ( − 1) , µ (0) ∈
R m +1 we perform the iterative updates
[Equation]
where L f = λ max ( GG T ) is the Lipschitz constant of f , and P F is the projection operator onto F (which has a closed form solution; see Bauschke (1996)).
Letting m f = λ min ( GG T ) denote the strong convexity constant of f , the momentum parameter is then scheduled as (Nesterov, 2013)
[Equation]
After computing the optimal dual variable µ (cid:63) , i.e., the ﬁxed point of (C.10), the optimal primal variable can be recovered via the equation x (cid:63) =
y + G T µ (cid:63) (as can be observed from the ﬁrst-order conditions of the Lagrangian (C.4)).
C. O BTAINING GRADIENTS
In order to incorporate the above projection into our neural network, we need to compute the gradients of all problem variables (i.e., G , h , and y ) through the solution x (cid:63) .
In particular, we note that x (cid:63) has a direct dependence on both G and y , and an indirect dependence on all of G , h , and y through µ (cid:63) .
To compute the relevant gradients through µ (cid:63) , we apply the implicit function theorem to the ﬁxed point of the update equations (C.10).
Speciﬁcally, as these updates imply that µ (cid:63) =
ν (cid:63) , their ﬁxed point can be written as
[Equation]
Deﬁne
[Equation]
(cid:63)
[Equation]
∇
[Equation]
and note that ∇ f ( µ (cid:63) ) =
GG T µ (cid:63) + Gy + h .
The differential of the above ﬁxed-point equation is then given by
[Equation]
Rearranging terms to separate the differentials of problem outputs from problem variables, we see that
[Equation]
where I is the identity matrix of appropriate size.
As described in e.g. Amos and Kolter (2017), we can then use these equations to form the Jacobian of µ (cid:63) with respect to any of the problem variables by setting the differential of the relevant problem variable to I and of all other problem variables to 0 ; solving the resulting equation for d µ (cid:63) then yields the value of the desired Jacobian.
However, as these Jacobians can be large depending on problem size, we rarely want to form them explicitly.
Instead, given some backward pass vector ∂(cid:96) ∂µ (cid:63) ∈ R 1 × ( m +1) with respect to the optimal dual variable, we want to directly compute the gradient of the loss with respect to the problem variables: e.g., for y , we want to directly form the result of the product ∂(cid:96)∂µ (cid:63) ∂µ (cid:63) ∂y ∈ R 1 × n .
We do this via a similar method as presented in Amos and Kolter (2017), and refer the reader there for a more in-depth explanation of the method described below.
Deﬁne J :=
I − M + 1 L f MGG T to represent the coefﬁcient of d µ (cid:63) on the left side of Equation (C.14).
Given ∂(cid:96)∂µ (cid:63) , we then compute the intermediate term
[Equation]
We can then form the relevant gradient terms directly as
[Equation]
In these computations, we note that as our solver returns x (cid:63) , the backward pass vector we are given is actually ∂(cid:96)∂x
[Equation]
∈ R 1 × n ; thus, we compute ∂(cid:96)∂µ
[Equation]
= ∂(cid:96)∂x (cid:63) G T for use in Equation (C.15).
Accounting additionally for the direct dependence of some of the problem variables on x (cid:63) (recalling that x (cid:63)
=
y + G T u (cid:63) ), the desired gradients are then given by (cid:18) ∂(cid:96) ∂G
(cid:19) T = (cid:18)
∂(cid:96) ∂x (cid:63) ∂x (cid:63) ∂G + ∂(cid:96)
∂x (cid:63) ∂x (cid:63) ∂u (cid:63) ∂u (cid:63) ∂G (cid:19)
[Equation]
(cid:18) ∂(cid:96)
∂h (cid:19) T =   (cid:8)(cid:8)(cid:8)(cid:8)(cid:8)(cid:42)
∂(cid:96) ∂x (cid:63) ∂x (cid:63) ∂h + ∂(cid:96) ∂x (cid:63) ∂x (cid:63) ∂u (cid:63) ∂u (cid:63) ∂h 

T =
L f M d
[Equation]
(cid:18) ∂(cid:96)
∂y (cid:19) T = (cid:18) ∂(cid:96) ∂x (cid:63) ∂x (cid:63) ∂y + ∂(cid:96) ∂x (cid:63) ∂x (cid:63) ∂u (cid:63) ∂u (cid:63) ∂y (cid:19) T = (cid:18) ∂(cid:96)
[Equation]
D W RITING THE CART - POLE PROBLEM AS AN NLDI
In the cart-pole task, our goal is to balance an inverted pendulum resting on top of a cart by exerting horizontal forces on the cart.
Speciﬁcally, the state of this system is deﬁned as x = [ p x , ˙ p x , ϕ, ˙ ϕ ] T , where p x is the cart position and ϕ is the angular displacement of the pendulum from its vertical position; we seek to stabilize the system at x = (cid:126) 0 by exerting horizontal forces u ∈ R on the cart.
For a pendulum of length (cid:96) and mass m p , and for a cart of mass m c , the dynamics of the system are (as described in Tedrake (2009)):
[Equation]
where g = 9 . 81 m / s 2 is the acceleration due to gravity.
We rewrite this system as an NLDI by deﬁning ˙ x = f ( x, u ) and then linearizing the system about its equilibrium point as
[Equation]
where J f is the Jacobian of the dynamics, w = f ( x, u ) − J f (0 , 0)
[ x u ] T is the linearization error, and I n is the n × n identity matrix.
We bound this linearization error by numerically obtaining the matrices C and D , assuming that x and u are within a neighborhood of the origin.
We describe this process in more detail below.
As a note, while we employ an NLDI here to characterize the linearization error, it is also possible to characterize this error via polytopic uncertainty (see Appendix J); we choose to use an NLDI here as it yields a much smaller problem description than a
PLDI in this case.
D. D ERIVING J f (0 , 0)
For ˙ x = f ( x, u ) , we see that
[Equation]
(D.3) where ∂
[Equation]
We thus see that
[Equation]
D. O BTAINING C AND D
We then seek to construct matrices C and D that bound the linearization error w between the true dynamics ˙
x and our ﬁrst-order linear approximation J f (0 , 0)
[Equation]
xu
[Equation]
To do so, we bound the error of this approximation entry-wise: that is, for each entry i = 1 , . . .
, s , we want to ﬁnd F i such that for all x in some region x ≤ x ≤ ¯ x
[Equation]
and all u in some region u ≤ u ≤ ¯ u ,
[Equation]
D
[Equation] [Equation]
Then, given the matrix
[Equation] [Equation]
we can then obtain C
= M 1: s and D
= M s : s + m (where the subscripts
We solve separately for each F i to minimize the difference between the right and left sides of Equation (D.5) (while enforcing that the right side is larger than the left side) over a discrete grid of points within x ≤ x ≤ ¯ x and u ≤ u ≤ ¯ u .
By assuming that F i is symmetric, we are able to cast this as a linear program in the upper triangular entries of F i .
To obtain the matrices C and D used for the cart-pole experiments in the main paper, we let ¯ x = [1 . 5 2 0 .
5] T , ¯ u = 10 , x = − ¯ x , and u = − ¯ u .
As each entry-wise difference in
Equation (D.5) contained exactly three variables (i.e., a total of three entries from x and u ), we solved each entry-wise linear program over a mesh grid of 50 points per variable.
E W RITING QUADROTOR AS AN NLDI
In the planar quadrotor setting, our goal is to stabilize a quadcopter in the two-dimensional plane by controlling the amount of force provided by the quadcopter’s right and left thrusters.
Speciﬁcally, the state of this system is deﬁned as x = [ p x p
[Equation]
˙ ϕ ] T , where ( p x , p z ) is the position of the quadcopter in the vertical plane and ϕ is its roll (i.e., angle from the horizontal position); we seek to stabilize the system at x = (cid:126) 0 by controlling the amount of force u
=
[ u r , u l ] T from right and left thrusters.
We assume that our action u is additional to a baseline force of [ mg/ 2 mg/ 2] T provided by the thrusters by default to prevent the quadcopter from falling.
For a quadrotor with mass m , moment-arm (cid:96) for the thrusters, and moment of inertia J about the roll axis, the dynamics of this system are then given by (as modiﬁed from Singh et al.
[Equation]
x cos
[Equation]
−
[Equation]
sin ϕ
[Equation]
p
[Equation]
sin
[Equation]
˙ ϕ
[Equation]
p
[Equation]
−
[Equation]
p
[Equation]

 +          1 /m 1 /m (cid:96)/J − (cid:96)/J 
 u, (E.1) where g = 9 .
81 m / s 2 .
We linearize this system via a similar method as for the cart-pole setting, i.e., as in Equation (D.2).
We describe this process in more detail below.
We note that since the dependence of the dynamics on u is linear, we have that D = 0 for our resultant NLDI.
As for cart-pole, while we employ an NLDI here to characterize the linearization error, it is also possible to characterize this error via polytopic uncertainty (see Appendix J); we choose to use an NLDI here as it yields a much smaller problem description than a PLDI in this case.
E. D ERIVING J f (0 , 0)
For ˙ x = f ( x, u ) , we see that
J f ( x, u ) =     0 0 − ˙
p x sin ϕ − ˙ p z cos ϕ cos ϕ − sin ϕ  0˙ p x cos ϕ − ˙
p z sin ϕ sin ϕ cos ϕ  0     0− g cos ϕ ˙ ϕ ˙
p z 0g sin ϕ − ˙ ϕ
− ˙ p x 0         , (E.2) and thus
J f (0 , 0) =
  0 1 0 00 0 1 00 0 0 10 0 − g 0 0 00 0 0 00 0 {+
+} 0
[Equation]
(E.3)
E. O BTAINING C AND D
We obtain the matrices C and D via a similar method as described in Appendix D, though in practice we only consider the linearization error with respect to x (i.e., since the dynamics are linear with respect to u , we have D = 0 ).
We let ¯ x = [1 1 0 .
6 1 . 3] and x = − ¯ x .
As for cart-pole, each entry wise difference in the equivalent of Equation (D.5) contained exactly three variables (i.e., a total of three entries from x and u ), and each entry-wise linear program was solved over a mesh grid of 50 points per variable.
F D ETAILS ON THE MICROGRID SETTING
For our experiments, we build upon the microgrid setting given in Lam et al.
In this system, the state x ∈ R 3 captures voltage deviations, frequency deviations, and the amount of power generated by a diesel generator connected to the grid; the action u ∈ R 2 describes the current associated with a storage device and a solar PV inverter; and the disturbance w ∈ R describes the difference between the amount of power demanded and the amount of power produced by solar panels on the grid.
The authors also deﬁne a performance index y ∈ R 2 which captures voltage and frequency deviations (i.e., two of the entries of the state x ).
To construct an NLDI of the form (3) for this system, we directly use the A , B , and G matrices given in Lam et al.
We generate C i.i.d.
from a normal distribution and let D = 0 , to represent the fact that the disturbance w and the entries of the state x are correlated, but that w is likely not correlated with the actions u .
Finally, we let Q and R be diagonal matrices with 1 in the entries corresponding to quantities represented in the performance index y , and with 0.1 in the rest of the diagonal entries, to emphasize that the variables in y are the most important in describing the performance of the system.
G G ENERATING AN ADVERSARIAL DISTURBANCE
In the NLDI settings explored in our experiments, we seek to construct an “adversarial” disturbance w ( t ) that obeys the relevant norm bounds (cid:107) w ( t ) (cid:107) 2 ≤
(cid:107) Cx ( t )
+ Du ( t ) (cid:107) 2 while maximizing the loss.
To do this, we use a model predictive control method where the actions taken are w ( t ) .
Speciﬁcally, for each policy π , we model w ( t ) as a neural network speciﬁc to that policy.
Every 10 steps of a roll-out, we optimize w ( t ) through gradient descent to maximize the loss over a horizon of 40 steps, subject to the constraint (cid:107) w ( t ) (cid:107) 2 ≤ (cid:107) Cx ( t ) + Du ( t ) (cid:107) 2 .
H A DDITIONAL EXPERIMENTAL DETAILS
Initial states.
To pick initial states in our experiments, for the synthetic settings, we sample each attribute of the state i.i.d.
from a standard Gaussian distribution.
For cart-pole and planar quadrotor, we sample uniformly from bounds chosen such that the non-robust LQR algorithm (under the original dynamics) did not go unstable.
For cart-pole, these bounds were chosen to be p x ∈ [ − 1 , 1] , ϕ ∈ [ − 0 .
1] , ˙ p x
= ˙ ϕ
= 0 .
For planar quadrotor, these bounds were p x , p z ∈ [ − 1 , 1] , ϕ ∈ [ − 0 .
05] , ˙
p x = ˙
p z
= ˙ ϕ = 0 .
Constructing NLDI bounds.
Given these initial states, for the cart-pole and quadrotor settings, we needed to construct our NLDI disturbance bounds such that they would hold over the entire trajectory of the robust policy; if not, the robustness speciﬁcation (A.8) would not hold, and our agent might in fact increase the Lyapunov function.
To ensure this approximately, we used a simple heuristic: we ran the (non-robust)
LQR agent for a full episode with 50 different starting conditions, and constructed an L ∞ ball around all states reached in any of these trajectories.
We then used these L ∞ balls on the states to construct the matrices C and D for our disturbance bounds, using the procedure described in Appendices D and E.
Computing infrastructure and runtime.
All experiments were run on an XPS 13 laptop with an
Intel i7 processor.
The planar quadrotor and synthetic NLDI experiment with D = 0 took aboutday to run (since the projections were simple half-space projections), while all the other synthetic domains and cart-pole took about 3 days to run.
The majority of the run-time was in computing the adversarial disturbances for test-time evaluations.
Hyperparameter selection.
For our experiments, we did not perform large parameter searches.
The learning rate we chose for our model-based planner, (both robust and non-robust) remained constant for the different domains; we tried learning rates of 1 × 10 − 3 , 1 × 10 − 4 , 1 × 10 − 5 and found 1 × 10 − 3 worked best for the non-robust version and 1 × 10 − 4 worked best for the robust version.
For our PPO hyperparameters, we simply used those used in the original PPO paper.
One parameter we had to tune for each environment was the time step.
In particular, we had to pick a time step high enough that we could run episodes for a reasonable total length of time (within which the non-robust agents would go unstable), but low enough to reasonably approximate a continuoustime setting (since, for our robustness guarantees, we assume the agent’s actions evolve in continuous time).
Our search space was small, however, consisting of 0 .
02 , 0 . 01 , and 0 .
005 seconds.
10 t 2.
x x (a) LQR     10 t 2.
x x
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
x x [Figure] [Figure] [Figure]
x x
[Figure]
Trajectory plots. Figure H.1 shows sample trajectories of different methods in the cart-pole domain under adversarial dynamics. The non-robust LQR and model-based planning approaches both diverge and the non-robust PPO doesn’t diverge, but doesn’t clearly converge after 10 seconds. The robust methods, on the other hand, all clearly converge after 10 seconds.
In addition to the NLDI settings explored in the main text, we test the performance of our method on PLDI and H ∞ control settings. As for the experiments in the main text, we choose a time discretization based on the speed at which the system evolves, and run each episode for 200 steps [Figure]
Non-robust Methods
Robust Methods
[Figure]
Training epochs [Figure] Training epochs Setting:
[Figure]
RARL
Robust MBP *
Robust PPO *
Original
Adversarial
[Figure]
PPO
RobustLQR
RobustMPC
RARL
RobustMBP ∗
RobustPPO ∗
[Figure] PLDI
O 96. A ——— unstable ——— 43. unstable 21.
[Figure]
O[-
-]
N/A  125 A
N/A In both cases, we use a randomly generated LQR objective where the matrices Q 1 / 2 and R 1 / 2 are drawn i.i.d.
from a standard normal distribution.
Synthetic PLDI setting.
We generate PLDI instances (A.9) with s = 5 , a = 3 , and L = 3 .
Speciﬁcally, we generate convex hull matrices ( A 1 , B 1 ) , . . .
, ( A 3 , B 3 ) i.i.d.
from normal distributions, and generate ( A ( t ) , B ( t )) by using a randomly-initialized neural network with softmax output to weight the convex hull matrices.
Episodes were run for 2 seconds at a discretization of 0.01 seconds.
Synthetic H ∞ setting.
We generate H ∞ control instances (A.11) with s = 5 , a = 3 , and d = 2 by generating matrices A, B and G i.i.d.
from normal distributions.
The disturbance w ( t ) was produced using a randomly-initialized neural network, with its output scaled to satisfy the L 2 bound on the disturbance.
Speciﬁcally, we scaled the output of the neural network to satisfy an attenuating normbound on the disturbance; at time t , the norm-bound was given by 20 × f (2 × t/T ) , where T is the time horizon and f is the standard normal PDF function.
Episodes were run for T = 2 seconds at a discretization of 0.01 seconds.
Results are given in Figure I.1 and Table I.1.
J N OTES ON LINEARIZATION VIA PLDI S AND NLDI S
While we linearize the cart-pole and quadrotor dynamics via NLDIs in our experiments, we note that these dynamics can also be characterized via PLDIs.
More generally, in this section, we show how we can use the framework of PLDIs to model linearization errors arising in the analysis of nonlinear systems.
Consider the nonlinear dynamical system ˙ x =
[Equation]
x, u ) with f (0 ,
[Equation]
) = 0 .
(J.1) for x ∈ R s and u ∈ R a .
Deﬁne ξ =
( x, u ) .
We would like to represent the above system as a PLDI in the region R :=
{ ξ | ξ ≤ ξ ≤ ¯ ξ } including the origin.
The mean value theorem states that for each component of f , we can write
[Equation]
(J.2) for some z = tξ , where t ∈ [0 , 1] .
Now, let p =
s + a .
Deﬁning the Jacobian of f as
[Equation]
(J.3) and recalling that f (0) = 0 , we can rewrite (J.2) as
[Equation]
(J.4)
Now, suppose we can ﬁnd component-wise bounds on the matrix J f ( z ) over R , i.e,
M ≤ J f ( z ) ≤ ¯ M for all z ∈ R .
(J.5)
We can then write
J f ( z ) =
(cid:88)
[Equation]
≤ p
[Equation]
with m ij ( t ) ∈ [ m ij , ¯ m ij ]
[Equation]
(J.6) where E ij =
e
i e Tj and e i is the i -th unit vector in R p .
We now seek to bound the Jacobian using polytopic bounds.
To do this, note that we can write
J
[Equation]
where A κ ’s are the vertices of the polytope in (J.6), i.e.,
[Equation]
Together, Equations (J.2), (J.4), (J.7), and (J.8) characterize the original nonlinear dynamics as a
PLDI.
We note that this PLDI description is potentially very large; in particular, the size of V is exponential in the square of the number of non-constant entries in the Jacobian J f ( z ) , which could be as large as 2 p 2 =
( s + a ) 2 .
This problem size may therefore become intractable for larger control settings.
We note, however, that we can in fact express this PLDI more concisely as an NLDI.
More precisely, we would like to ﬁnd matrices A, B, C parameterizing the form of NLDI below, which is equivalent to that presented in Equation (3) (see Chapter 4 of Boyd et al.
[Equation]
It can shown that the solution to the SDP minimize Tr( V + W ) subject to W (cid:31)
[Equation]
yields the matrices A , B , and C with V =
C T C and W = BB T , which can be used to construct
NLDI (J.9).
While the NLDI here is more concise than the PLDI, the trade-off is that the NLDI norm bounds obtained via this method may be rather loose.
As such, for our settings, we obtain
NLDI bounds numerically (see Appendices D and E), as these are tighter than NLDI speciﬁcations obtained via the above method (though they are potentially slightly inexact).
An alternative approach would be to examine how to tighten the conversion from PLDIs to NLDIs, which has been explored in other work (e.g. Kuiava et al.