Safe control under input limits with neural control barrier functions
Abstract: We propose new methods to synthesize control barrier function (CBF)- based safe controllers that avoid input saturation, which can cause safety violations.
In particular, our method is created for high-dimensional, general nonlinear systems, for which such tools are scarce.
We leverage techniques from machine learning, like neural networks and deep learning, to simplify this challenging problem in nonlinear control design.
The method consists of a learner-critic architecture, in which the critic gives counterexamples of input saturation and the learner optimizes a neural CBF to eliminate those counterexamples.
We provide empirical results on a 10D state, 4D input quadcopter-pendulum system.
Our learned
CBF avoids input saturation and maintains safety over nearly 100% of trials.
Introduction
In theory, control barrier functions are an appealing tool for safe control.
However, it is difﬁcult to make the derived controllers respect input limits, which reduces their usage in practice.
CBFs target the set invariance class of safety problems, in which safety is deﬁned as keeping a system’s state to a prescribed region.
A large part of their appeal is that they offer mathematical guarantees of safety.
Such assurances are essential for safety-critical robotics applications, like collision-free drone ﬂight [1, 2], manipulators that work safely around humans [3], and stable bipedal walking [4].
However, these safety guarantees break down when input saturation occurs, since that means the system cannot exert the force required for an evasive maneuver.
The system then becomes endangered (or dangerous), with the possibility of expensive equipment failure or people getting harmed.
It is therefore imperative that we account for input limits when designing CBFs.
CBFs are energy functions which map states to an energy value, with safe states having lower energy.
In energy function methods , an energy function is found and then a controller is crafted that dissipates the energy.
The core problem of these methods is constructing the energy function.
A valid energy function has to meet complex constraints that depend on the input limits, system dynamics, and safety speciﬁcation.
So far, this problem of designing CBFs around input limits has been studied to a limited degree, mostly for small or simple systems.
To our knowledge, nothing has been proposed which handles the nonlinear and high-dimensional systems that are more realistic in robotics.
Currently, many state-of-the-art approaches rely on hand-designing CBFs.
This works well for certain simple systems, like the kinematic bicycle system [5, 6, 7, 8, 9, 10, 11, 12]. Some of these hand-design methods are more systematic, deriving non-saturating CBF for special classes of systems, like polynomial systems [13, 14].
Yet another variation of hand-design is to hand-select a parametric function for the CBF and optimize the parameters to avoid saturation [15, 16, 10]. The problem of designing a non-saturating CBF is also equivalent to computing a control invariant set , a well-known problem in the controls community [17]. This area has a long history and has been studied under different viewpoints and names, including viability kernel computation [18] and inﬁnite-time reachable set computation [19]. For a condensed survey, see [20].
The most generic framework for computing control invariant sets is HJ Reachability [21].
Although it can handle nonlinear systems and gives formal guarantees against saturation, it cannot typically scale past systems of 6 or 7 dimensions in the state.
This paper takes a different approach from HJ Reachability and abandons formal guarantees for better scalability.
For the most part, existing approaches for synthesizing non-saturating CBF apply to narrow classes of systems and can involve considerable human effort.
In contrast, we envision a framework for automating CBF synthesis that has a wide range of applicability .
To achieve this, we borrow ideas from machine learning (ML).
We take inspiration from the related ﬁeld of Lyapunov function (LF) synthesis, which has incorporated ML with success.
LFs certify stabilization , rather than safety.
However, ﬁnding a non-saturating CBF is essentially ﬁnding a function that satisﬁes a constraint on a set of inputs, which is the same problem as in LF synthesis.
Recent works in LF synthesis represent the LF as a NN and then train it to satisfy the function constraints [22, 23, 24, 25].
We borrow this paradigm for the unique problem of input saturation of CBFs. For additional related work, please see Appendix Sec. 6.1. There are several advantages to framing the problem as NN training.
Firstly, it allows us to handle synthesis for nonlinear systems. Good non-saturating CBFs for nonlinear systems tend to be nonlinear functions, and NN are a richly expressive class of nonlinear functions.
Secondly, it allows us to handle synthesis for systems with large state dimensions ( ≥ 10 D ). Neural networks can be trained quickly for inputs (here, the system state) of this size.
We synthesize CBFs that respect input limits by posing this as a problem of training a neural function to satisfy limit-related constraints.
Our contributions are as follows:
A novel way to frame the synthesis of non-saturating CBF. The design of a training framework, including a neural CBF design, loss function and training algorithm design. Experimental validation on a 10D state, 4D input nonlinear system.
The rest of this paper is laid out as follows: Sec. 2 explains how CBFs work and carefully deﬁne the input saturation problem.
Then, Sec. 3 details our approach, including the design of the neural
CBF, training losses, and training algorithm.
Finally, Sec. 4 describes how we test our method on a challenging quadcopter-pendulum system against several baselines.
Preliminaries
In this section, we provide a review of CBFs, mathematically deﬁne a non-saturating CBF, and explain the premise of CBF synthesis.
First, some notation: for a function c : R n → R , let C (cid:44) { c } ≤ 0 be its zero-sublevel set, ∂ C
= { c } =
the boundary of this set, and Int ( C ) =
{ c } < 0 the interior.
Now, we assume the following is given: (1) a control-afﬁne system ˙ x = f ( x ) + g ( x ) u , where x ∈ D ⊂ R n , u ∈ U ⊆
R m and f : R n → R n , g : R n → R n × m are locally Lipschitz continuous on R n , (2) input set U , a bounded convex polytope, and (3) a safety speciﬁcation ρ : R n → R , which implicitly deﬁnes the allowable set as A (cid:44) { ρ } ≤ 0 .
Further, assume ρ ( x ) is continuous and smooth.
Given ρ ( x ) , we can deﬁne r ∈ Z + as the relative degree from ρ ( x ) to u (i.e. the ﬁrst derivative of ρ ( x ) where u appears). We now walk through the process of producing a safe controller via CBF methods.
In safe control, the goal is to keep some subset of the allowable set forward invariant , which means keeping any trajectory starting within the subset inside of it for all time.
We call this subset the safe set and it will be deﬁned by a function, the control barrier function, which we design.
The CBF will also be used to deﬁne the safe controller that ensures forward invariance. In the absence of input limits, we would just form a CBF as a known function of the safety speciﬁcation ρ ( x ) :
[Equation]
where c i < 0 .
See [5, 26] for an explanation.
The safe set S ⊆ A deﬁned by this limit-blind CBF is elaborated in Appendix Sec. 6.2.
In the presence of input limits, we will have to modify this design, but we will come back to this.
Next, to deﬁne a safe controller using a CBF is straightforward.
A safe controller simply needs to repel the system back into the safe set whenever it reaches the set boundary.
The following theorem formalizes this idea:
Theorem 1 (Taken from [5]) .
Given a CBF φ and safe set S , any feedback controller k ( x ) : R n → R m satisfying
[Equation]
renders the system forward invariant over S .
Note that you can also require a CBF to satisfy a stricter inequality ˙ φ ( x ) ≤ − α ( φ ( x )) for all x ∈ D , where α ( · ) is a class- κ function.
We elaborate on this alternative in Appendix Sec. 6.2.
The theorem above requires the controller to repel the system (decrease φ ) from the boundary ( x ∈ ∂ S ).
We are allowed to use any nominal controller, k nom , as long as we modify its inputs to satisfy Eqn. 1 at the boundary.
Thus, a CBF-based safe controller simply ﬁlters (modiﬁes) a nominal controller online by applying the QP below at every step of control execution:
[Equation]
The issue with using a limit-blind CBF for this controller is that it can cause controller saturation.
Speciﬁcally, saturation occurs when no u ∈ U exists that satisﬁes Eqn. 1 (constraint 2), causing the loss of safety guarantees.
1 What we need is to synthesize a non-saturating CBF , which is mathematically deﬁned as:
Deﬁnition 1 ( Non-saturating CBF ) .
A function φ : R n → R is a non-saturating CBF over a set S if for all x ∈ ∂ S :
[Equation]
Intuitively, Def. 1 just requires that there exist a feasible control input to decrease φ (push the system to the interior of S ) at every state on the boundary, ∂ S .
We approach this problem by modifying the limit-blind CBF:
[Equation]
where ρ ∗ ( x ) : R n → R such that { ρ ∗ ( x ) } ≤ 0 ⊆ { ρ ( x ) } ≤ 0 .
This modiﬁcation acts to shrink the associated safe set from S to S ∗ . With a proper choice of function ρ ∗ ( x ) , we can exclude irrecoverable states (states where no feeasible input preserves safety) from the safe set boundary. In the rest of the paper, we focus on learning the function ρ ∗ ( x ) to produce a non-saturating CBF.
Problem scope: to review, our proposed method applies to nonlinear, control-afﬁne systems and safety problems that can be described by a smooth safety speciﬁcation function ρ ( x ) . We also assume the system is deterministic and fully known. The method also applies to systems of high relative degree, since we based the modiﬁed CBF off of a higher-order CBF.
Learning Non-Saturating Control Barrier Functions
In this section, we present our neural CBF design (Sec. 3.1) and then discuss the training framework which optimizes it with respect to control limits (Sec. 3.2, 3.3).
Learning is formulated as a min-max optimization problem of the following form, where θ denotes the parameters of φ ∗ :
[Equation]
We call this loss L ( θ, x ) the saturation risk .
This min-max problem is solved using a learner-critic algorithm (Alg. 1), where the critic and learner repeatedly ﬁnd where the worst saturation occurs and then update the CBF to reduce saturation there.
We also propose some strategies for training stably, boosting critic efﬁciency, and for increasing the volume of the safe set.
For visualization, we plot the critic’s counterexamples for a toy cartpole problem and intuitively justify their correctness (Fig. 1).
[Figure]
3.1 Neural CBF Design
Building upon the previous work discussed in Sec. 2, to design a non-saturating CBF, we only need to consider the design of the function ρ ∗ ( x ) from the modiﬁed CBF.
Let nn : R n → R be a multilayer perceptron with tanh activations.
Then, we deﬁne
[Equation]
where x e is a state, identiﬁed by the user, that should belong to any reasonable learned safe set.
Speciﬁcally, x e should belong to the allowable set A and should satisfy ρ ( i ) ( x e ) ≤ 0 for i ∈ [0 , r − 1] (this constrains any possible higher-order components in x e ; for example, velocities should be directed away from unsafe zones). For safety problems that limit the system’s distance from an equilibrium, the equilibrium itself should lie in any reasonable safe set. For anti-collision problems, x e can be a point far from the ego robot.
For additional recommendations, see the Appendix. We have designed ρ ∗ to obey three constraints: Constraint 1 : ρ ∗ is smooth.
This is required to preserve the smoothness of φ ∗ , which allows us to make existence and uniqueness arguments for the closedloop system.
Our ρ ∗ satisﬁes this because nn has smooth tanh activations and also ρ is assumed smooth.
Constraint 2 : The 0-sublevel set of ρ ∗ is contained within the 0-sublevel set of ρ : { ρ ∗ } ≤ 0 ⊆ { ρ } ≤ 0 .
The allowable set can be shrunk but not enlarged; otherwise, dangerous states may be incorporated.
Our design meets this criterion because ρ ∗ ≥ ρ .
Constraint 3 : S ∗ is nonempty, where S ∗ is deﬁned by ρ ∗ .
With our design, x e ∈ S ∗ by the deﬁnition of S ∗ (see Appendix) and our assumptions on x e .
3.2 Training Framework
In the following sections, we present a loss function that encourages satisfaction of Eqn. 1 and the algorithm for solving the min-max problem on this loss.
First, we deﬁne θ as the parameters of φ ∗ , which include the weights of nn and the c i coefﬁcients.
Our loss function, called saturation risk , is deﬁned as:
[Equation]
It is a measure of the best-case saturation at a given state x .
When L ( θ, x ) ≤ 0 , then no saturation occurs at x ; when L ( θ, x ) > 0 , it measures how severe the saturation is.
Thus, our min-max problem (Eqn. 4) is to minimize the worst best-case saturation over the boundary.
When the worst best-case is negative, i.e.
[Equation]
1: function L EARN ( X ce , θ ) 2:
Set learning rate α l 3: θ ← θ − α l · ∇ θ (cid:2)(cid:80) x ∈ X ce softmax
( L ( θ, x ))
+ R ( θ ) (cid:3) (cid:46) From Eqn.
6,4: return θ 5: end function 6: function C OMPUTE CE( θ ) 7: Set learning rate α c , number of gradient steps N 8: X ← uniformly sample a set on the boundary (cid:46) See Alg.9: for i in [0 , . . . , N ] do 10:
G ← ∇ X L ( θ, X ) (cid:46) Batch gradient 11:
P ← project G along boundary 12:
X ← X + α c · P (cid:46) Batch update 13:
X ← project X to boundary (cid:46) See Alg.14: end for 15:
X ce ← worst saturating states in X 16: return X ce 17: end function 18: function M AIN ( ) 19:
Input: dynamical system ˙ x , safety speciﬁcation ρ 20:
Randomly initialize neural CBF parameters θ 21:
Repeat: 22:
X ce ← C OMPUTE CE ( θ ) (cid:46) X ce is a set of counterexamples 23: θ ← L EARN ( X ce , θ ) 24:
Until convergence 25: return θ 26: end function then we have successfully found a non-saturating CBF.
To solve the min-max problem on L ( θ, x ) (Eqn. 4), we propose a learner-critic algorithm (Alg. 1).
Essentially, the algorithm alternates between the critic computing counterexamples (maximization with respect to x ) and the learner updating the CBF (minimization with respect to θ ).
The critic uses projected gradient descent to produce an approximate maximizer, ˆ x ∗ , and then the learner uses gradient descent to minimize the saturation loss at ˆ x ∗ .
Since both learner and critic perform gradient descent on L ( θ, x ) , it is useful to re-express it as an analytic function, rather than a continuous minimization.
To ﬁnd the analytic expression, observe that this L ( θ, x ) is a minimization over u where the objective is afﬁne (from Eqn. 1) and the constraint set is a convex polyhedron U , by assumption.
This means the minimizing u ∗ is one of the vertices v ∈ V ( U ) of the constraint set.
Thus, L ( θ, x ) can be computed as a discrete minimization, which is analytic:
[Equation]
(analytic risk) 3.3 Practical Training Methods Batch optimization: in practice, training works better if both learner and critic use a batch of counterexamples, rather than just a single one.
This means the critic optimizes a set of differently initialized counterexamples at once and the learner takes a weighted loss over a subset of the best counterexamples:
[Equation]
This has the advantage of stabilizing convergence without adding much overhead. It stabilizes convergence by preventing (1) deadlock and (2) inaccurate gradients throughout training.
Deadlock is when improvement at one counterexample causes saturation at another; averaging the learner’s loss on a group of counterexamples avoids this by encouraging progress on many counterexamples at once.
Inaccurate gradients refers to the fact that the learner should be using the gradient
[Figure]
at the optimal counterexample x ∗ ( ∇ θ L ( θ, x ∗ ) ) , but since the critic is suboptimal, it uses a different gradient, ∇ θ L ( θ, ˆ x ∗ ) .
Clearly, this could derail training.
However, we ﬁnd that with batching, the critic produces better counterexamples, giving us a closer estimate of the true gradient ( ∇ θ L ( θ, ˆ x ∗ )
≈ ∇ θ L ( θ, x ∗ ) ).
Plus, with averaging, the learner averages out the effect of inaccurate gradients.
We show in the Appendix that the algorithm requires large batches of counterexamples to perform well (Table 2).
All in all, these techniques are a cheap and effective way to avoid the kind of training instability found in other counterexample-based methods, like adversarial training for image classiﬁers [27, 28, 29].
in this section, we introduce a regularization term that we add to the training objective to help enlarge the safe set. One measure of quality for safe sets is size.
Since CBFs will allow a system to move freely inside a safe set, a larger safe set provides greater freedom towards accomplishing control objectives. Thus, a larger safe set enables better task performance. To this end, we add a regularization term to the training objective to encourage a large safe set.
For context, recall that our CBF φ ∗ θ deﬁnes a safe set S ∗ . Speciﬁcally, there is a function ss ∗ of φ ∗ θ that implicitly deﬁnes S ∗ as its 0-sublevel set: S ∗ = { x | ss ∗ ( x ) ≤ 0 } ) ; ss ∗ ( x ) is deﬁned in the Appendix. To clearly indicate its dependence on the CBF and its learned parameters θ , we write it as ss ∗ θ here.
Next, we evaluate the regularization term at some sampled states X reg . We have:
[Equation]
The idea behind the sigmoid is to encourage states near the boundary ( ss( φ ∗ θ ( x )) ≈ 0 ) to move (further) inside the safe set ( ss( φ ∗ θ ( x )) << 0 ).
Sigmoid gives a gradient which encourages values near zero to become (more) negative.
We demonstrate in the Appendix that including this term can increase the volume by several factors.
Experiments
In this section, we train a neural CBF on a challenging nonlinear, high-dimensional, and inputlimited robotic system.
We pose two experimental questions:
Q1.
How well do we achieve our training objective?
Q2.
Does the CBF-based safe controller ensure FI?
Our system is a pendulum on top of a quadcopter (Fig. 2).
This is a coupled system, with the dynamics of both components found by the Euler-Lagrangian method [31].
The states are quadcopter position and roll-pitch-yaw orientation ( x, y, z and γ, α, β ) and roll-pitch pendulum orientation ( φ p , θ p ), as well as the ﬁrst derivatives of these states.
The inputs are thrust and torque ( F, τ γ , τ β , τ α ), which are limited to a bounded convex polytope set.
See the Appendix for the system dynamics.
The safety [Table] speciﬁcation is to prevent the pendulum from tipping and the quadcopter from rolling:
[Equation]
where δ p = arccos(cos( φ p ) cos ( θ p )) is the pendulum’s angle from the vertical.
Since the quadcopter position does not impact safety and the position dynamics can be decoupled, we exclude position and consider the resulting 10D state, 4D input system.
Finally, in the design of ρ ∗ ( x ) , we let x e = (cid:126) 0 , which is the system’s equilibrium.
Next, we propose metrics to answer each of our experimental questions:
Q1 metrics.
% of non-saturating states on ∂ S ∗ : we measure how well we satisfy Eqn. 3 by uniformly sampling states on ∂ S ∗ and calculating what percentage of them are non-saturating.
We also use these samples to compute the mean and variance of the severity of saturation , E [ L ( θ, x )] and σ [ L ( θ, x )] .
To approximate the severity of the worst saturation , L ( θ, x ∗ ) , we apply our critic and allow it to use more samples and computation time than during CBF training.
Q2 metrics.
Q2 considers the in-the-loop control performance of our learned CBF.
We measure % of simulated rollouts that are FI by initializing rollouts randomly inside S ∗ and simulating their trajectories under the safe controller (CBF-QP) until just after they reach the boundary.
Then, we record whether the system exited or remained inside S ∗ after arriving at the boundary.
The value of this metric depends on the choice of nominal controller, k nom .
We try k 0 ( x ) = 0 , which yields an unactuated system, and k lqr and k lqr − agg , which are linear quadratic regulator (LQR) stabilizing controllers, with k lqr − agg tuned to be more aggressive.
We also choose two well-known alternatives as our baselines: Hand-designed CBF: for CBFs, this is a typical alternative. We hand-pick a parametric CBF and then optimize the parameters for non-saturation.
Safe MPC: MPC is often used for safe planning and control and it can take input limits into account.
The safety speciﬁcation ( ρ ≤ 0 ) becomes a nonlinear state constraint and we also have to set the terminal constraint to be a smaller, known invariant set for the MPC solution to be forward invariant. For details on any of these baselines, see the Appendix Sec 6.4. Note that safe MPC deﬁnes its safe set implicitly . This means the boundary of the safe set is not deﬁned by a function.
Hence, it would be too time-consuming to sample states on the boundary and compute the metrics for Q1. For safe MPC, we only report the results for Q2. Code.
Our code can be found at https://github.com/sliu2019/input_limit_cbf
Training details.
The learning framework and metrics were implemented using Python and PyTorch [32].
Training took about 2 hours on a single NVIDIA GeForce RTX 2080 Ti GPU.
Discussion.
As we can see in Table 1, for our learned CBF, almost 100% of the boundary states are non-saturating and almost 100% of the rollouts are FI across the nominal policies.
This shows that our neural CBF and learning framework are an effective combination and capable of handling systems of high complexity.
We are not able to attain 100% non-saturating states and FI rollouts, which is either due to suboptimality of training or limitations of the function class, as NNs are only universal approximators when their size is taken to the inﬁnite.
For our learned CBF, the severity of saturation at the saturating states is not negligible, but it is still low, and the worst saturation is large, but rarely encountered.
The hand-designed CBF does poorly across all metrics (only 80% of the boundary states are non-saturating and around 80% of rollouts are FI).
Safe MPC is equally good as the learned CBF at ensuring safety (almost 100% rollout safety) 2 . However, its signiﬁcant MPC disadvantage is that its safe sets are just a fraction of the size of our own ( 8 . 4% respectively). Size is an important measure of the quality of a safe set; these small sizes imply that this baseline can only ensure safety from relatively few states.
The root of the issue is that safe MPC constructs a safe set by effectively expanding a smaller, “seed” invariant set provided by the user. The size of the safe set therefore depends greatly on the size of the seed invariant set. As we have already established, it can be quite difﬁcult for users to design invariant sets (equivalently, non-saturating CBFs) of any reasonable size. Visualizing slices of the safe set can provide deeper insight into the results of training (Fig. 2).
Recall that a safe set contains only states that are recoverable from danger, given our input limits.
We observe that our CBF has diamond-shaped safe sets in slices B and C, which makes sense because small angles can still be recoverable at larger speeds.
In slice B, safe MPC’s set largely captures states where the signs of γ, ˙ γ differ.
This makes sense too, since these are states where the angular velocity acts to return the angle to 0 . Safe MPC’s set in slice B is also larger than ours. Since it is a non-saturating safe set, just like ours, we have to conclude that our algorithm could have found a larger non-saturating safe set. The reason for this suboptimality is that our volume regularization strategy is imperfect.
It encourages expansion only at scattered points on the boundary, which means that some areas of the boundary may be unaffected. However, our regularization strategy seems to work well overall, since the volume of our safe set in 10D is much larger than that of safe MPC. In slices C and D, we get a glimpse of why that is. In both of these slices, safe MPC’s set is tiny.
It very tightly constraints the pendulum’s angular velocity.
While it makes sense for safe MPC to be more conservative towards the pendulum than the quadcopter (the pendulum is not directly actuated, while the quadcopter is), it is unnecessarily conservative.
Next, we analyze the shapes of the sets in slice D. Our safe set indicates that there should be a maximum safe angular speed for quadcopter and pendulum.
This makes sense, since higher angular velocities are certainly harder to pull back from.
We also observe a larger range for the quadcopter’s pitch velocity than the pendulum’s, which makes sense because again, the quadcopter is directly actuated and the pendulum is not.
On the other hand, the hand-designed CBF does not restrict ˙ θ or ˙ β in slice D. Due to the functional form of the hand-designed CBF (see Appendix Section 6.4), ˙ θ and ˙ β are only restricted when ˙ θθ > 0 or ˙ ββ >
(that is, when angular velocity is strictly acting to destabilize).
We’ve assumed θ, β = 0 in slice D.
This safety criterion is clearly too lenient, since large angular velocities that are currently swinging the system to the origin can cause overshooting and toppling shortly after.
Overall, we can see that the non-neural CBF does not have the right function form.
In general, it can be hard to guess what the right form might be for a system of this complexity.
However, our algorithmic approach, combined with the NN functional representation, removes the need for guessing.
Limitations and future work: in the future, we intend to loosen our assumptions (see last paragraph of Sec. 2), particularly the assumption of a known and deterministic model.
We would also like to test this method on more high-dimensional, nonlinear systems.
One limitation of our work is that we had to perform a change of variables on the states input to the neural CBF before it would train successfully (see Appendix for details).
While the dynamics on the new states were still nonlinear, this indicates that a simple feed-forward network might not be the best neural function class.
Conclusion
In summary, we proposed a framework for facilitating CBF synthesis under input limits.
Thanks to our neural CBF representation and our effective and efﬁcient learning framework, our method scales to higher-dimensional nonlinear systems.
We learned a virtually non-saturating CBF on one such system, the quadcopter-pendulum.
We hope that this safe control tool will makes CBFs more accessible and of practical value to roboticists. Acknowledgments
This material is supported by the United States Air Force and DARPA under Contract No. FA875018-C-0092 and the National Science Foundation under Grant No. 2144489.
We thank Tianhao Wei,
Weiye Zhao, Yiwei Lyu and Qin Lin for useful conversations.
We are also very grateful to Kate
Shih, Qin Lin, Harrison Zheng, Raffaele Romagnoli, Tianhao Wei, Soumith Udatha, Tamas Molnar, and Jason Choi for reviewing drafts of this work.