Safety Guarantees for Neural Network Dynamic Systems via Stochastic Barrier Functions
Equal Contribution
Abstract
Neural Networks (NNs) have been successfully employed to represent the state evolution of complex dynamical systems.
Such models, referred to as NN dynamic models (NNDMs), use iterative noisy predictions of NN to estimate a distribution of system trajectories over time.
Despite their accuracy, safety analysis of NNDMs is known to be a challenging problem and remains largely unexplored.
To address this issue, in this paper, we introduce a method of providing safety guarantees for
NNDMs.
Our approach is based on stochastic barrier functions, whose relation with safety are analogous to that of Lyapunov functions with stability.
We ﬁrst show a method of synthesizing stochastic barrier functions for NNDMs via a convex optimization problem, which in turn provides a lower bound on the system’s safety probability.
A key step in our method is the employment of the recent convex approximation results for NNs to ﬁnd piece-wise linear bounds, which allow the formulation of the barrier function synthesis problem as a sum-of-squares optimization program.
If the obtained safety probability is above the desired threshold, the system is certiﬁed.
Otherwise, we introduce a method of generating controls for the system that robustly minimize the unsafety probability in a minimally-invasive manner.
We exploit the convexity property of the barrier function to formulate the optimal control synthesis problem as a linear program.
Experimental results illustrate the efﬁcacy of the method.
Namely, they show that the method can scale to multi-dimensional NNDMs with multiple layers and hundreds of neurons per layer, and that the controller can signiﬁcantly improve the safety probability.
Introduction
Safety is a major concern for autonomous dynamical systems, especially in safety-critical applications.
Examples include UAVs, autonomous cars, and surgical robots.
To this end, various methods are developed to provide safety guarantees for such systems [1–3].
Most of these methods, however, assume (simple) analytical models, which are often unavailable or too expensive to obtain due to complexity (nonlinearities) in real-world systems [4].
This has given rise to recent efforts to harvest the representational power of neural networks (NNs) to predict how the state of a system evolves over time, referred to as NN dynamic models (NNDMs) [5, 6].
Despite their efﬁcient and accurate representation of realistic dynamics, it has been difﬁcult to make formal claims about the behavior of
NNDMs over time.
This poses a major challenge in safety-critical domains: how to provide safety guarantees for NNDMs?
In this paper, we propose a framework to provide safety certiﬁcates for NNDMs.
The approach is based on stochastic barrier functions [7, 8], whose role for safety is analogous to that of Lyapunov functions for stability, and which can guarantee forward invariance of a safe set.
Speciﬁcally, we show a method of synthesizing barrier functions for NNDMs via an efﬁcient convex optimization method, which in turn provides a lower bound on the safety probability of the system trajectories.
The beneﬁt of our approach is that it does not require unfolding of the NNDM to compute this probability; instead, we base our reasoning on non-negative super-martingale inequalities to bound the probability of leaving the safe set.
The key to this is to utilize the recent convex approximation results for NNs to obtain piece-wise linear bounds, and a formulation of the optimization problem that enables the use of efﬁcient solvers.
If the obtained safety probability is above a desired threshold, the system can be certiﬁed.
For the cases that such safety certiﬁcates cannot be provided, we introduce a method of generating controls for the system, which robustly maximizes the safety probability in a minimally-invasive manner.
We exploit the convexity property of the barrier function to formulate the control synthesis problem as a linear program.
Experimental results on various NNDMs trained using state-of-the-art RL techniques [9–11] illustrate the efﬁcacy of the method.
Namely, they show that the method can scale to multi-dimensional models with various layers and hundreds of neurons per layer, and that the controller can signiﬁcantly improve the safety probability.
In summary, this paper makes the following main contributions: (i) a method of computing a lower bound for the safety probability of NNDMs via stochastic barrier functions and efﬁcient nonlinear optimization, thereby providing safety guarantees, (ii) a correct-by-construction control synthesis framework for NNDMs to ensure safety while being minimally-invasive , i.e., intervening only when needed, (iii) demonstration of the efﬁcacy and scalability of the framework in benchmarks on various
NNDMs with multiple architectures, i.e., multiple hidden layers and hundreds of neurons per layer.
Related Work
In recent years, NNs have been successfully employed to model complex dynamical systems [5, 6].
Such models use iterative noisy predictions of an NN to estimate the distribution of the system over time.
They provide means of predicting state evolution of the systems that have no analytical form, e.g., soft robotics [12].
In addition, because of their ability to quickly unfold trajectories, NNDMs have been used to enhance controller training in Reinforcement Learning (RL) frameworks [9, 11].
In these works, speciﬁcally, a NNDM of the system is trained in closed-loop with an NN controller, which can be concatenated in a single NN representing the dynamics of the closed-loop system.
Despite their popularity and increased usage, little work has gone to safety analysis of NNDMs.
This paper aims to ﬁll this gap.
Barrier functions provide a formal methodology to prove safety of dynamical systems [13, 14].
The state-of-the-art to ﬁnd stochastic barrier functions is arguably to rely on Sum-of-Squares (SOS) optimization [8].
Barrier certiﬁcates obtained via SOS optimization have been studied for deterministic, uncontrolled nonlinear systems [15], and control-afﬁne stochastic systems [16, 8], as well as for hybrid models [13, 14].
In addition, the control synthesis formulation in this paper relaxes the limitations of the existing SOS methods, which generally leads to a non-convex optimization problem [8], by reducing it to a simple linear program.
Many recent studies have focused on the veriﬁcation of NNs.
However, most of them only aim to provide certiﬁcation guarantees against adversarial examples [17], with approaches ranging from
SMT [18], and convex relaxations [19] to linear programming [20].
Such methods, because of their local nature, cannot be directly employed to perform safety analysis on the entire state space and over the trajectories as required for NNDMs.
Another line of work focuses on inﬁnite time horizon safety of Bayesian neural networks [21], or the veriﬁcation of neural ODEs with stochastic guarantees [22].
They consider neural network controllers with known dynamics [23].
Other methods have been employed for trajectory-level properties of NNs based on solution of the Bellman equations [24], formal (ﬁnite) abstractions [25], and mixed integer programming [26], which do not support noisy dynamics.
In contrast, our approach supports additive noise and only requires to check static properties on the NN by showing that there exists a function whose composition with the NN forms a stochastic barrier function.
Problem Formulation
In this work, we are interested in providing safety guarantees for dynamical systems that are described by NNs.
Speciﬁcally, we consider the following discrete-time stochastic process whose time evolution is given by iterative predictions of an NN
[Equation]
where f w is a trained, fully-connected, feed-forward NN with general continuous activation functions, and w represents the maximum likelihood weights.
Term v k is a random variable modelling an additive noise of stationary Gaussian distribution with zero mean and covariance matrix R ∈ R n × n , i.e., the probability density function of v k is N (¯ x | 0 , R ) .
Remark 1.
System (1) can be viewed as a closed-loop system, where f w is the composition of a
NNDM (that represents the open-loop system) with an NN controller.
NNDMs such as the one in
System (1) are increasingly employed in robotics for both model representation and NN controller training.
These models are often obtained by state-of-the-art model-based RL techniques (e.g., [9–11]) or perturbations (step response) of the actual system (e.g., [12]).
The evolution of System (1) can be characterized by the predictive posterior distribution p x
(¯ x | x ) , which describes the probability density of x k +1 when x k = x ∈ R n .
This distribution is induced by noise v k and hence is Gaussian with mean f w ( x ) and covariance R , i.e., p x (¯ x | x ) =
N (¯ x | f w ( x ) , R ) .
For a subset of states X ⊆ R n and (initial) state x ∈ R n , we call T ( X | x ) =
(cid:82)
X
p x (¯ x | x ) d ¯ x the stochastic kernel of System (1).
From the deﬁnition of T it follows that, for a given initial condition x 0 = x 0 ∈ R n , x k is a Markov process with a well-deﬁned probability measure Pr uniquely generated by the stochastic kernel T [27, Proposition 7.45].
For
X 0 , X k ⊆ R n , this probability measure is inductively deﬁned as
[Equation]
where indicator function 1 X 0 ( x 0 )
=
if x 0 ∈ X 0 , otherwise 0.
The deﬁnition of Pr allows one to make probabilistic statements over the trajectories of System (1).
In this work, we are particularly interested in two main problems: safety certiﬁcation and safe controller synthesis for System (1).
Problem 1 (Safety Certiﬁcate) .
Let X s ⊂ R n and X 0 ⊆
X s be basic closed semi-algebraic sets representing respectively the safe set and the set of initial states.
Denote the probability that the system initialized in X 0 remains in the safe set in the next N ∈ N ≥ 0 steps by
[Equation]
Then, given threshold δ s ∈ [0 , 1] , provide a safety certiﬁcate for System (1) by proving that
P s ( X s , X 0 , N ) ≥ δ s .
Problem 1 seeks to compute the probability that x k remains within a safe set, e.g., it avoids obstacles or any undesirable states.
Computation of this probability is particularly challenging because System (1) is stochastic and NN ( f w ) is generally a non-convex function.
The assumption that X s and X 0 are basic closed semi-algebraic sets is not limiting, as these sets are deﬁned as an intersection of a ﬁnite number of polynomial inequalities and, for instance, include convex polytopes and spectrahedra [28].
In the case that a safety certiﬁcate cannot be generated, we are interested in synthesizing a controller that by-design guarantees safety.
To this end, we extend System (1) with an afﬁne input
[Equation]
where g : R c
→ R n is given, u k ∈ U ⊂ R c is a control action, and U is bounded.
Our goal is to synthesize a feedback controller π : R n → U such that u k
= π ( x k ) guarantees safety of System (3).
Problem 2 (Safe Control Synthesis) .
Let X s ⊂ R n and X 0 ⊆
X s be basic closed semi-algebraic sets representing respectively the safe set and the set of initial states, and N ∈ N ≥
a time horizon.
Then, given threshold δ s , synthesize a feedback controller π such that P s ( X s , X 0 , N, π ) ≥ δ s , where
P s ( X s , X 0 , N, π ) is the safety probability obtained for control-afﬁne System (3) .
Remark 2.
There may exist many possible controllers that are solutions to Problem 2.
Since
System (1) represents a closed-loop system with a given controller, one may want to speciﬁcally synthesize a safety controller that is “minimally-invasive,” i.e., it minimally intervenes to guarantee safety.
In Section 5, we elaborate on this notion and introduce methods of generating such controllers.
Overall Approach Our approaches to Problems 1 and 2 are based on (stochastic) barrier functions.
Speciﬁcally, we aim to synthesize barrier functions that can serve as a safety certiﬁcate by providing a lower bound for P s .
To achieve this, we rely on recent results for local convex relaxation of an
NN to ﬁnd piece-wise linear over- and under-approximations of f w .
This allows us to formulate the problem of ﬁnding a stochastic barrier function for System (1) as an SOS optimization problem.
Our (minimally-invasive) control synthesis method (Problem 2) focuses on injecting controls to
System (3) only in the regions of the state space that potentially contribute to unsafety of the system.
To this end, we exploit two main properties of the barrier function: the required (super-) martingale property and the convexity property of SOS polynomials.
The former allows us to design a method of identifying the “unsafe” regions, and the latter enables us to formulate a Linear Programming (LP) optimization problem to ﬁnd controllers that robustly minimize the probability of unsafety.
Remark 3.
For ease of presentation, g in (3) is taken to be a constant, however, we note that the proposed approach can also handle g being a continuous function of x , or even a NN.
For the latter, we introduce an interval bound propagation approach in Section 4.1 that allows the computation of upper and lower bounds on g ( x ) , which in turn can be incorporated in the LP optimization formulation in Section 5 for control synthesis (Problem 2) in a straightforward manner.
Preliminaries
In this section, we provide a brief background on stochastic barrier functions and non-negative polynomials, which are central components of our framework.
3. Stochastic Barrier Functions
Consider stochastic process x k +1 =
F ( x k , v k ) , where x k ∈ X ⊆ R n , v k ∈ V ⊆
R m is a welldeﬁned stochastic process, and F : X
× V → X is a locally Lipschitz continuous function.
Then, given initial set X 0 ⊂ X , safe set X s ⊆ X , and unsafe set X u ⊂ X , a twice differentiable function
B : X → R is called a stochastic barrier function or simply barrier function if the following conditions hold for α ≥ 1 and β, η ∈ [0 , 1] :
[Equation]
If such a B exists, then for any N ∈ N ≥ 0 , it follows, assuming α = 1 without loss of generality, that
[Equation]
Intuitively, Conditions (4a)-(4d) allow one to use non-negative super-martingale inequalities to bound the probability that the system reaches the unsafe region.
This results in the probability bound in (5) (see [29, Chapter 3, Theorem 3]).
A major beneﬁt of using barrier functions for safety analysis is that while Conditions (4b)-(4d) are static properties, they allow one to make probabilistic statements on the time evolution of the system without the need to evolve the system over time.
3. Non-negative Polynomials
Synthesis of barrier functions can be formulated as a nonlinear optimization problem.
To solve it efﬁciently via convex programming, we formulate the problem using SOS polynomials, which is a sufﬁcient condition for polynomial non-negativity [30].
This formulation allows the use of efﬁcient semideﬁnite programming solvers, which have polynomial worst-case complexity [31].
Deﬁnition 1 (SOS Polynomial) .
A multivariate polynomial λ ( x ) is a Sum-Of-Squares (SOS) for x ∈ R n if there exists some polynomials
λ i , i = 1 , . . .
, r , for some r ∈ N , such that λ ( x ) =
(cid:80) ri
=
λ 2 i ( x ) .
If λ ( x ) is an SOS, then λ ( x ) ≥ 0 for all x ∈ R n .
The set of all SOS polynomials is denoted by Λ .
Now, consider a basic closed semi-algebraic set
X = { x ∈ R n | h i ( x ) ≥ 0 ∀ i ∈ { 1 , . . .
, l }} , which is deﬁned as the intersection of l polynomial ( h i ( x ) ) inequalities.
Then, Proposition 1 allows one to enforce non-negativity on these semi-algebraic sets.
Proposition 1 (Putinar’s Certiﬁcate, [32]) .
Let X ⊂ R n be a compact basic semi-algebraic set deﬁned as X = { x ∈ R n | h
i ( x ) ≥ 0 ∀ i ∈ { 1 , . . .
, l }} , where h i ( x ) is a polynomial.
Then, polynomial γ ( x ) is non-negative on X
if, for some λ 0 ( x ) , λ i ( x ) ∈ Λ , γ ( x ) = λ 0 ( x ) + (cid:80) li
=
λ i ( x ) h i ( x ) .
Corollary 1.
Let γ ( x ) be a non-negative polynomial on X = { x ∈ R n |
h i ( x ) ≥∀ i ∈ { 1 , . . .
, l }} , and h ( x ) be an l -dimensional vector of polynomials with h i ( x ) as its i -th dimension for all i ∈ { 1 , . . .
, l } .
Further, let L be an l -dimensional vector of SOS polynomials, i.e., the i -th element of L is λ i ( x ) ∈ Λ .
Then, it holds that γ ( x ) − L T h ( x ) ∈ Λ .
Barrier Functions for Neural Network Dynamical Models
In this section, we introduce a formulation for synthesizing a barrier function that guarantees safety of System (1).
Our approach relies on the local convex relaxations of f w , which we use to formulate an SOS optimization problem that generates a valid barrier function for System (1).
4. Local Convex Relaxation of NNDMs
Our approach uses a convex relaxation of f w .
Speciﬁcally, we utilize the recent relaxation results for NNs [19] to compute piece-wise linear functions that under- and over-approximate f w .
The advantage of this method is that it is computationally efﬁcient and produces tight bounds for small neighborhoods.
To this end, we ﬁrst partition
X s to a ﬁnite set of regions Q = { q 1 , ...., q | Q | } , where q i ⊆
X s for all i ∈ { 1 , . . .
, | Q |} is a basic (closed) semi-algebraic set.
Such a decomposition can be obtained using, e.g., a grid or half-spaces.
Then, for every q ∈ Q , we compute linear bounds f q
[Equation]
such that ∀
[Equation] [Equation]
The above linear functions can be produced in several ways.
One method is to forward propagate the bounds in the input domain of f w ( x ) using Interval Bound Propagation (IBP) based approaches [33, 34].
These approaches are efﬁcient but generally lead to loose approximations.
Another method is Linear Relaxation based Perturbation Analysis (LiRPA) algorithms, which perform a symbolic back propagation followed by a forward evaluation of the bounds [35, 36, 19].
Such LiRPA-based algorithms produce tighter bounds but can be slow.
In our implementation, we use the off-the-shelf toolbox [36] to compute f q ( x ) and f q ( x ) in (6) for its trade-off between speed and accuracy.
4. Barrier Function (Certiﬁcate) Synthesis
Recall that a valid barrier function B ( x ) must satisfy Conditions (4b)-(4d).
With the above local linear bounds for f w , Condition (4d) can be redeﬁned over the partitioned regions in Q .
That is,
Condition (4d) can be replaced with the following constraints: for each q ∈ Q ,
[Equation]
In this formulation, f wq ( x ) is a (free) variable that is bounded by two linear functions, avoiding the non-convexity of f wq ( x ) .
Consequently, an efﬁcient evaluation of the constraint becomes feasible.
Remark 4.
Note that there are in fact three constraints in (7) (one super-martingale condition on B and two bounds on f wq ( x ) ).
If they are treated separately, it leads to a conservative barrier function because each constraint needs to hold for all x ∈ q separately.
A less conservative approach is to pose all constraints for the same x at the same time.
In Theorem 1, we show how this can be achieved.
As commonly practiced in the literature, e.g., [15, 8, 37], we restrict our search for a B ( x ) to the class of polynomial functions.
Then, the degree of the polynomial becomes a design parameter.
In the following lemma, we show that the degree of the polynomial must be at least 2 for the polynomial to be a valid candidate barrier function for System (1).
That is, there does not exist a linear function satisfying Conditions (4b)-(4d) for η < 1 , which implies that the lower bound in (5) is trivially 0 .
Lemma 1.
For k ∈ R n and c ∈ R , consider linear function B ( x ) =
k T x + c .
Then, B ( x ) satisﬁes
Conditions (4b) - (4d) iff k = 0 and c = 1 .
Proof.
There are two cases to consider: (i) k = 0 :
in this case B ( x ) = c , which satisﬁes Conditions (4b)-(4d) iff c
=
η = 1 .
(ii) k
(cid:54) = 0 : in this case, due to the linearity of B ( x ) , there always exists x (cid:48)
∈ R n such that B ( x (cid:48) )
< 0 , violating Condition (4a).
Having established that the polynomial degree must be at least 2, we further restrict B ( x ) to SOS polynomials.
Theorem 1 shows a formulation of the barrier function synthesis problem as an SOS program, which can be solved in polynomial time.
In particular, the theorem shows how to formulate
Conditions (4b)-(4c) and (7) as SOS constraints.
Theorem 1 (NNDM Barrier Certiﬁcate) .
Consider SOS polynomial function B ( x ) , and safe set
X s = { x ∈ R n | h s ( x ) ≥ 0 } , initial set X
= { x ∈ R n | h 0 ( x ) ≥ 0 } , unsafe set X u
= R n \ X s = { x ∈ R n | h u ( x ) ≥ 0 } , and partition region q = { x ∈ R n | h q ( x ) ≥ 0 } for all q ∈ Q .
Let
L s ( x ) , L 0 ( x ) and L u ( x ) be vectors of SOS polynomials with the same dimensions as h s , h 0 , and h u , respectively.
Likewise, let L q,x ( x ) and L q,y
( x ) be vectors of SOS polynomials with the same dimension as h q .
Then, a stochastic barrier certiﬁcate B ( x ) for System (1) with time horizon
N ∈ N ≥ 0 can be obtained by solving the following SOS optimization problem for η, β ∈ [0 , 1] :
[Equation]
subject to:
[Equation]
which guarantees safety probability P s ( X s , X 0 , N ) ≥ 1 − ( η + βN ) .
Proof.
It sufﬁces to show that if B satisﬁes Constraints (8a)-(8d), then Conditions (4a)-(4d) are satisﬁed.
As Constraint (8a) guarantees that B is a SOS polynomial, hence non-negative by deﬁnition,
Condition (4a) holds.
By Corollary 1 it holds that if Constraints (8b)-(8c) hold, then B is respectively smaller that η in X 0 and greater than 1 in X u .
What is left to show is that Constraint (8d) guarantees the satisfaction of Condition (4d).
This can be done as follows.
For every region q ∈ Q , the expectation term in Condition (4d) can be expressed as E [ B ( y + v ) | x ] , where y is bounded by under-approximation
f q ( x ) and over-approximation
f q ( x ) of f w ( x ) in System (1) for all x ∈ q .
Then by Corollary 1, Constraint (8d) is obtained for each q .
Note that B ( x ) is a polynomial, so if
E [ B ( y + v ) | x ] is also a polynomial, the condition is a valid SOS constraint.
Given that y + v is linear, E [ B ( y + v ) | x ] is a sum of monomials in terms of y and expectation moments
E
[ v d ] , where d ≥ 0 .
Since, random variable v has a normal distribution, E [ v d ] is constant.
Therefore,
E [ B ( y + v ) | x ] is a polynomial only in terms of (components of) y .
Remark 5.
We note that Condition (8d) uses linear equations in (6) to bound f w ( x ) for all x ∈ q .
Removing the dependence on x , by replacing the equations for f q ( x ) and f q ( x ) with their extreme values in region q , relaxes the optimization problem and results in a speed-up.
It is however an additional over-approximation, and hence the obtained safety probability bound is strictly smaller than the one generated by using the actual linear equations for f q ( x ) and f q ( x ) .
This is the classical “efﬁciency versus accuracy” trade-off.
In our implementation, we compare the two approaches.
Minimally-invasive Barrier Controller Synthesis
Here, we focus on Problem 2.
The assumption is that a barrier function B ( x ) is synthesized for
System (1) via Theorem 1, which has a safety probability that is lower than threshold δ s .
Hence, our goal is to design a feedback controller for System (3) that guarantees safety.
Ideally, this controller is minimal in the interruptions it causes to the system evolution.
Below, we ﬁrst provide a deﬁnition for a minimally-invasive controller, and then show how it can be synthesized.
Recall that a feedback controller π : R n → U is a function that assigns a control value to each state.
Let X π ⊆ X
s be the set of states, to which π assigns non-zero control values, i.e., π ( x ) (cid:54)
=
for all x ∈ X π .
Further, we denote by V ( X ) the volume of set X ⊂ R n .
Then, we say a feedback controller π is minimally-invasive if it minimizes V ( X π ) .
Moreover, we say π is (cid:15) - minimally-invasive if
V ( X π ) − min π (cid:48)
V ( X π (cid:48) ) ≤ (cid:15) .
In this work, we seek (cid:15) - minimally-invasive controller with an arbitrary small (cid:15) .
To this end, we make the observation that if the discretization of X s discussed in Section 4. is uniform, V ( X π ) for a given π is directly proportional to the number of discrete regions in Q to which π applies a non-zero control.
We denote by Q π the set of such regions.
Then, a feedback controller π that minimizes | Q π | is (cid:15) - minimally-invasive , and as the discretization becomes ﬁner, (cid:15) monotonically approaches zero.
Therefore, given barrier function B ( x ) , our ﬁrst goal is to identify the minimal number of regions in Q that require a non-zero controller to guarantee safety.
Recall that, given B ( x ) , we require the safety probability to be 1 − ( η + βN ) ≥ δ s .
While η is related to the value of B ( x ) in the initial set, β is a compensation needed to turn a sub-martingale inequality to a super-martingale inequality (Condition (8d) or (4d)).
We can compute this compensation for each region q ∈ Q by evaluating Condition (8d) or (4d).
Let β q denote the computed compensation term for region q .
Then, region q requires a non-zero controller if β q > (1 − δ s − η ) /N .
We now turn our attention to designing a controller that reduces β q .
In our approach, we tap directly into the convex nature of the SOS polynomial B ( x ) (SOS-convex).
Lemma 2 shows that a controller that drives System (3) to a point closest to the minimum point of B ( x ) , also robustly minimizes β q .
Lemma 2.
Given B ( x ) , let x ∗ =
arg min x B ( x ) , x (cid:48) =
f w ( x ) + gu + v as in System (3) , and β q ≥
E [ B ( x (cid:48) )
| x ] − B ( x ) for all x in region q ⊆ X s .
Then, the controller that minimizes (cid:107) ˜ x
(cid:48) − x ∗ (cid:107) 1 , where ˜ x (cid:48)
=
f w ( x ) + gu , also robustly minimizes β q , i.e., minimizes an upper bound of β q .
Proof.
Given that B is a polynomial and x (cid:48) is afﬁne in both u and v , we can write E [ B ( x (cid:48) )
| x ] =
B ( E [ x (cid:48)
| x ]) + ξ , where ξ , called Jensen’s gap [38], is a non-negative polynomial of x , u , and moments E
[ v d ] , where 2 ≤ d ≤ m is an even integer.
This polynomial can always be upper bounded by a constant ξ ≥ 0 for bounded x and u .
Then, with ˜ x (cid:48)
=
E [ x (cid:48) ] , the min β q is upper bounded by
[Equation]
The second inequality holds because B is a non-negative function resulting from a SOS optimization, and is thus SOS-convex [39].
Since B (˜ x (cid:48) ) monotonically decreases as ˜ x (cid:48)
→ x ∗ , max x ∈ q B (˜ x (cid:48) | x ) is minimized by arg min u (cid:107) f w ( x ) + gu − x ∗ (cid:107) 1 ∀ x ∈ q .
We note that, in the proof of Lemma 2, the moments E [ v d ] appear in every monomial of polynomial ξ (Jensen’s gap).
Hence, for v with variance less than 1, Jensen’s gap is very small and vanishes as the variance becomes smaller, making the controller in the lemma optimal.
Furthermore, even though Lemma 2 uses L 1 norm for (cid:107) ˜ x (cid:48) − x ∗ (cid:107) 1 , the statement also holds for L 2 and L ∞ norms.
We speciﬁcally use L 1 norm to be able to compute the controller via an LP as stated in Theorem 2.
Theorem 2 (Controller Synthesis) .
Consider convex barrier function B ( x ) and its minimum argument x ∗ as speciﬁed in Lemma 2, and linear bounds f q ( x ) and f q ( x ) for f w ( x ) in region q ⊆ R n as in (6) .
A controller that robustly minimizes β q for System (3) is the solution to the following LP optimization problem, where θ = ( θ 1 , θ 2 , . .
, θ n ) ∈ R n ≥ 0 , and the vector inequality relations are element-wise: min
θ i ,z [Equation] ,z
[Equation]
,u n
[Equation]
to: .
[Equation]
Proof.
By Lemma 2 it holds that to robustly minimizes β q , it sufﬁces to minimize (cid:107) ˜ x
(cid:48) − x ∗ (cid:107) 1 .
This norm is the sum of n absolute values, each of which can be set to θ i with linear Constraints (10a) and (10b).
Hence, the minimization of (cid:107) ˜ x
(cid:48) − x ∗ (cid:107) 1 is equivalent to minimization of (cid:80) ni
=
θ i subject to (10a) and (10b).
The bounds on the dynamics of System (3) can be expressed as linear
Conditions (10d) and (10e), each of which has to hold for all z ∈ q , i.e., z is a free variable.
To turn the optimization into an LP, z is expressed as the difference of two non-negative decision variables z (cid:48) and z (cid:48)(cid:48) in Constraint (10c).
Remark 6.
Note that the controller obtained via the LP is a vector of scalars for each region q .
This leads to a feedback controller π over the regions q ∈ Q .
It is possible to express u as SOS polynomials, which would be a feedback controller over x .
That could potentially provide a tighter bound for β q at the cost of extra computation since SOS optimization is more expensive than LP.
5. Control Synthesis Algorithm
Our framework for computing a barrier certiﬁcate as well as minimally-invasive controllers for
NNDMs is outlined in Algorithm 1.
First, the safe set X s is uniformly partitioned into a ﬁnite set of regions | Q | , and bounds of f w are computed as piece-wise linear functions per Section 4.1.
Then, for a given polynomial degree m ≥ 2 , a barrier certiﬁcate along with its corresponding safety probability bound are computed in accordance with Theorem 1.
If this probability is greater than or equal to δ s , the NNDM is certiﬁed, and the algorithm terminates.
Otherwise, it iteratively searches for (cid:15) -minially-invasive controllers.
In each iteration, upper bound η is set on η (initially with its largest possible value 1 − δ s and then reduced by ∆ η > 0 ), and a new barrier function is computed with this bound, where the objective function is minimization of β to ensure (cid:15) -minimally-invasive controllers.
Then, for each q ∈ Q , β q is checked against the threshold.
If exceeding, a controller is computed for q in accordance with Theorem 2.
This process repeats until a satisfactory P s is obtained or ¯ η < 0 .
We highlight two important properties of this algorithm: it clearly terminates in ﬁnite time, and if
P s ≥ δ s , the synthesized controller is correct-by-construction.
Case Studies
We demonstrate the efﬁcacy of our framework on several case studies.
We ﬁrst show that our veriﬁcation method is able to produce non-trivial safety guarantees for NNDMs trained using imitation learning techniques [40] from data gathered by rolling out an RL agent trained with state-of-the-art RL techniques [41].
We then show that our control approach signiﬁcantly increases the safety probability.
Models We trained the Pendulum ( 2 D ), Cartpole ( 4 D ), and Acrobot ( 6 D ) agents from the OpenAI gym environment, as well as 4 D and 5 D Husky robot models.
For the OpenAI models, we collected input-output states of the system under an expert controller, either as a look-up table or an NN controller.
We picked the best controller available for the agents from the OpenAI Leaderboard [41].
The Husky models were trained to move from one point to another while staying within a lane.
We initially trained a NNDM along with an NN controller on the trajectory data generated from PyBullet (physics simulator [42]) as outlined in [9].
We then used these NNs to generate input-output data and trained closed-loop NNDMs as speciﬁed above.
In total, we trained 10 fully-connected multi-layer perceptron NNDM architectures with up to 5 hidden layers and 512 neurons per layer, each with
ReLU activation function.
The dimensionality and architecture information is provided in Table 1.
The state space of the Pendulum consists of the pole angle θ and angular velocity ˙ θ , with the safe set deﬁned as [ − π/ 15 , π/ 15] × [ − 1 , 1] , and the initial set as θ 0 ∈ [ − π/ 36 , π/ 36] .
For the Cartpole, the state space consists of the cart position x , velocity ˙ x , pole angle θ , and angular velocity ˙ θ .
The safe sets for cart position are deﬁned as x ∈ [ − 1 , 1] and θ ∈ [ − π/ 15 , π/ 15] , with the initial set θ 0 ∈ [ − π/ 36 , π/ 36] .
For the 4D Husky, the states are position x and y , orientation θ , and linear velocity v .
The task of the controller is to keep the robot within a lane, and hence the safe set is y ∈ [ − 1 , 1] , and the initial set is deﬁned as any position ( x, y ) that is within a radius of 0.1 from the
[Table]
origin.
For the 5 D Husky, the additional state over the 4 D model is the angular velocity ω ; the task, the safe set, and the initial set remain the same.
Finally, for the Acrobot, the state space consists of the cosines and sines of θ
1 and θ 2 , and angular velocities ˙ θ 1 and ˙ θ 2 .
Here θ 1 and θ 2 are the pole angle of the ﬁrst link and the angle of the second link relative to the ﬁrst link, respectively.
The task is for the tip of the second link to reach a height of at least y = 1 , with a safety constraint to never reach beyond y = 1 . 2 .
Thus, the safe set is deﬁned as sin( θ 1 ) ∈
[ − 0 .
6 , 0 . 6] and sin( θ 2 ) ∈ [ − 0 .
6 , 0 . 6] , and the initial set is any point within a radius of 0.1 around the origin in the ﬁrst 4 dimensions.
Implementation and Experimental Setup:
We implemented our algorithms in Python and Julia (code available in [43]).
For collecting data and training, we used the TensorFlow framework [44].
To compute the linear approximations of the NNDMs, we utilized α -CROWN [36].
For the optimization, we used Julia’s SumOfSquares.jl package [45, 46].
The optimizations are all performed single-threaded on a computer with 3.9 GHz 8-Core CPU and 64 GB of memory.
Benchmarks
To the best of our knowledge, there is no other work on safety veriﬁcation of
NNDMs, against which we can compare.
Hence as a baseline, we compare our approach based on linear under and over approximation of the NNDM against interval bounding approaches cf.
Remark 5.
The interval bounding approach results in a relaxed optimization problem at the cost of more conservative safety probabilities.
The results are shown in Table 1.
The computation times indicate the time to synthesize the barrier and the controller using Theorems 1 and 2, respectively.
The barrier degree is 4 for all the case studies.
Below, we provide a brief discussion on the results.
For more in-depth discussions and details on the discretization and hyperparameters, see Appendix A.
Veriﬁcation: As expected, the safety probability always increases with | Q | (ﬁner partitions), since the linear and interval bounds for f w become more accurate.
This behavior is encountered in all the systems and architectures, regardless of the number of hidden layers or neurons per layer.
However, it is also observed that this trend of ﬁner discretizations and increased safety probability consistently comes at the cost of an increased computation time.
This is due to the fact that the number of partitions directly dictate the number of constraints in the optimization problems.
Note that the probability of safety using interval bounds is strictly worse than linear approximations for all the experiments.
Using either linear or interval bounds, we are able to compute non-trivial probability of safety for all the case-studies and observe the discretization vs accuracy trend.
Take for instance, the 1-layer Cartpole model, where an increase in | Q | from 960 to 3840 causes an increase in P s from
[Figure]
375 to 0 .
602 , at the expense of an increased computation time of more than 12 times.
This behavior is also true for the 4D Husky model, where increasing | Q | from 1800 to 4800 for the 2-layer model, increases P s from 0 .
011 to 0 .
384 at the price of a more than 5-fold increase in the computation time.
Control Synthesis: When the certiﬁcation probability is below δ s = 0 .
95 , the control strategy is utilized to increase the probability of safety.
Using interval bounds, controllers had to be synthesized for all case-studies, which illustrates the looseness of the bounds.
In fact, in several case studies, the controller was not able to meet the desired threshold.
For the linear bounds, observe that for all except two NNDMs, whose P s < δ s , the controller synthesis algorithm was able to produce
P s ≥ δ s , showing the effectiveness of our control method.
The ﬁrst exception is for the 4 D Husky model with 2-hidden layers and | Q | = 1800 , where the controller increases P s from 0.011 to 0.
(73-fold increase) yet cannot meet the 0.95 threshold.
Similarly, for the 1-layer 5D Husky model with | Q | = 432 , the controller increases P s from 0.00 to 0.851, but the 0.95 threshold cannot reached.
Nonetheless, for all the linear bound cases, the controller reduces β to 10 − 6 .
Figure 1 illustrates the minimally-invasive aspect of the controllers.
It shows the β q values for the Pendulum NNDM with 2 layers before and after the application of the controllers with three different sizes of | Q | .
For | Q | = 120 , all the regions require controllers, but for | Q | = 240 and 480, respectively, only 69.2% and 23.75% of the regions require controllers; hence, the minimally-invasive controller focuses speciﬁcally on those regions and does not affect those with already-small β q .
Conclusion
In this work, we introduced a methodology for providing safety guarantees for NNDMs using stochastic barrier functions.
We showed that the problem of ﬁnding a stochastic barrier function for a NNDM can be relaxed to an SOS optimization problem.
Furthermore, we derived a novel framework for synthesis of an afﬁne controller with the goal of maximizing the safety probability in a minimally-invasive manner.
Experiments showcase that, along with a minimally-invasive controller, we can guarantee safety for various standard reinforcement learning problems.
A bottleneck to our approach is the discretization step, which is required to obtain piece-wise linear under- and over-approximations of the neural network, and can be expensive for higher dimensional spaces.
Future directions should focus on mitigating this curse of dimensionality.
Acknowledgments and Disclosure of Funding
This work was supported in part by the NSF grant 2039062 and NASA COLDTech Program under grant #80NSSC21K1031.