Safe Neurosymbolic Learning with Differentiable Symbolic Execution
Abstract
We study the problem of learning worst-case-safe parameters for programs that use neural networks as well as symbolic, human-written code.
Such neurosymbolic programs arise in many safety-critical domains.
However, because they need not be continuous, let alone differentiable, they cannot be learned using existing gradient-based approaches to safe learning.
Our method, Differentiable Symbolic
Execution (D SE ), learns such programs by sampling code paths using symbolic execution, constructing gradients of a worst-case “safety loss” along these paths, and then backpropagating these gradients through program operations using a generalization of the R EINFORCE estimator.
We evaluate the method on realworld benchmarks.
Our experiments show that D SE signiﬁcantly outperforms the state-of-the-art D IFF A I method on these tasks.
Introduction
Safety on worst-case inputs has recently emerged as a key challenge in deep learning research.
Formal veriﬁcation of neural networks [2] is an established response to this challenge.
In particular, an exciting body of recent work [22, 20, 9, 27] has sought to incorporate formal veriﬁcation into the training of neural networks.
D IFF A
I , among the most prominent of such approaches, uses a neural network veriﬁer to construct a differentiable, worst-case safety loss for the learner.
This loss is used to regularize a standard data-driven loss, biasing the learner towards parameters that are both performant and safe.
A weakness of these methods is that they only consider functional properties (such as adversarial robustness) of isolated neural networks.
By contrast, in real-world applications, neural networks are often embedded within human-written symbolic code and correctness requirements apply to the entire neurosymbolic composition.
For example, consider a car directed by a neural controller [25].
Safety properties for the car are functions of its trajectories, and these trajectories depend not just on the controller but also the symbolic equations that deﬁne the environment.
While recent work [8] has studied the veriﬁcation of such neurosymbolic programs, there is no prior work on integrating veriﬁcation and learning for such systems.
In this paper, we present the ﬁrst steps towards such an integration.
The fundamental technical challenge here is that while a neural network is differentiable over its parameters, the code surrounding may be non-differentiable or even discontinuous.
This puts our problem beyond the scope of existing methods for integrated learning and veriﬁcation.
We overcome this difﬁculty using a new method, Differentiable Symbolic Execution (D SE ), for estimating gradients of worst-case safety losses of nondifferentiable neurosymbolic programs.
D SE is based on a generalization of the classic R EINFORCE estimator, which backpropagates gradients through non-differentiable operations by approximating integrals with sampling.
In our problem, the integral is the aggregate of the safety losses along symbolic control ﬂow paths in the program.
To apply R EINFORCE -like ideas here, we need to represent and sample paths in a symbolic way.
We do so through an adaptation of the classic method of symbolic execution .
We evaluate D SE through several case studies in the embedded control and navigation domains.
Our baselines include an extended version of D IFF
A I , the current state of the art, and an ablation that does not use an explicit safety loss.
Our experiments show that D SE signiﬁcantly outperforms the baselines in ﬁnding safe and high-performance model parameters.
Problem Formulation
Programs.
We deﬁne programs with embedded neural networks as symbolic transition systems (STS) [21].
Formally, a program F θ is a tuple ( Loc , X, l 0 , Init , Safe , Trans θ ) .
Here, Loc is a ﬁnite set of (control) locations , X = { x 1 , . . .
, x m } is a set of real-valued variables, and l 0 ∈ Loc is the initial location .
Init , a boolean formula over X , is an initial condition for the program.
Safe is a map from locations to constraints over X ; intuitively, Safe ( l ) is a safety requirement asserted at location l .
Finally, Trans θ is a transition relation consisting of transitions ( l, G, U θ , l (cid:48) )
such that: (i) l is the source location and l (cid:48) is the destination location ; (ii) the guard G is a constraint over X ; and (iii) the update U θ is a vector (cid:104) U 1 ,θ , . .
, U m,θ (cid:105) , where each U i,θ is a real-valued expression over X constructed using standard symbolic operators and neural networks with parameters θ .
We assume that each U i,θ is differentiable in θ .
Also, we assume that the programs are deterministic .
That is, if G and G (cid:48) are guards for two distinct transitions from the same source state, then G ∧ G (cid:48) is unsatisﬁable.
Safety Semantics.
Let a state of F θ be a pair s
=
( l, v ) , where l is a location and v ∈ R m is an assignment of values to the variables (i.e., v ( i ) is the value of x i ).
Such a state is said to be at location l .
A state ( l 0 , v ) , where v satisﬁes Init , is an initial state .
A state ( l, v ) is safe if v satisﬁes Safe ( l ) .
Let v ∈ R m be an assignment to the variables.
For a real-valued expression E over X , let E ( v ) be the value of E when x i is substituted by v ( i ) .
For an update U = (cid:104) U 1 , . . .
, U n (cid:105) , we deﬁne U ( v ) as the assignment (cid:104) U 1 ( v ) , . . .
, U n ( v ) (cid:105) .
A length- n trajectory of F θ is a sequence τ =
(cid:104) s 0 , . .
, s n (cid:105) , with s i =
( l i , v i ) , such that: (i) s 0 is an initial state; and (ii) for each i , there is a transition ( l i , G, U, l i +1 )
such that v i satisﬁes G and v
i +
=
U ( v i ) .
We denote this trajectory by τ ( s 0 ) .
Let us assume a real-valued loss Unsafe ( s ) that quantiﬁes the unsafeness of each state s .
We require
Unsafe ( s ) =
if s is safe and Unsafe ( s ) > 0 otherwise.
We lift this measure to trajectories τ by letting Unsafe ( τ ) =
(cid:80) s appears in τ Unsafe ( s ) .
The safety loss C ( θ ) for F θ is now deﬁned as:
[Equation]
Thus, C ( θ ) =
if and only if all program trajectories are safe.
Problem Statement.
Our learning problem formalizes a setting in which we have training data for neural networks inside a program F θ .
While training the networks with respect to this data, we must ensure that the overall program satisﬁes its safety requirements.
To ensure that the parameters of the different neural networks in F θ are not incorrectly entangled, we assume that only one of these networks, NN θ , has trainable parameters.
We expect as input a training set of i.i.d.
samples from an unknown distribution over the inputs and outputs of NN θ , and a differentiable data loss Q ( θ ) that quantiﬁes the network’s ﬁdelity to this training set.
Our learning goal is to solve the following constrained optimization problem:
[Equation]
Approach
Learning Framework.
Our learning approach is based on two ideas.
First, we directly apply a recently-developed equivalence between constrained and regularized learning [1, 18] to reduce
Equation (2) to a series of unconstrained optimization problems.
(For more details, see Appendix A.1.)
Second, we use the novel technique of Differentiable Symbolic Execution (D SE ) to solve these unconstrained problems.
A key feature of our high-level algorithm is that it repeatedly solves the optimization problem
[Equation]
for ﬁxed values of λ .
This problem is challenging because while Q ( θ ) is differentiable in θ , C ( θ ) depends on the entirety of F θ and may not even be continuous.
As we demonstrate in Section 4, this makes it difﬁcult to apply state-of-the-art gradient-based approaches to worst-case safe learning.
D SE , our main contribution, addresses this challenge by estimating gradients ∇ θ C # ( θ ) of a differentiable approximation C # ( θ ) of C ( θ ) .
Background on Symbolic Execution.
D SE is a reﬁnement of symbolic execution [5], a classic technique for systematic formal analysis of programs.
A symbolic executor systematically searches the set of symbolic trajectories of programs, which we now deﬁne.
Consider a program
F θ =
( Loc , X
=
x 1 , . .
, x m } , l 0 , Init , Safe , Trans θ ) as in Section 2.
First, we ﬁx a syntactically restricted class V θ of boolean constraints over the variables X and parameters θ .
V θ is required to be closed under conjunction.
Next, we deﬁne an abstraction function α : 2 R m → V θ that maps sets S of assignments to X to overapproximations drawn from V θ .
We use α to construct an overapproximate update U # θ : V θ → V θ for each update U θ in F θ .
For all
V ∈ V
θ , we have U # θ ( V )
=
α ( { U θ ( v ) : v satisﬁes V } ) .
A symbolic state of F θ is now deﬁned as a pair σ θ =
( l, V θ ) , where l is a location and V ∈ V θ .
Intuitively, σ θ represents the set of states of F θ that are at location l and satisfy the property V θ .
Let us call a transition t =
( l, G, U, l (cid:48) ) enabled at a symbolic state ( l, V ) if ( G ∧ V ) is satisﬁable.
A symbolic trajectory of F θ is a sequence τ # θ = (cid:104) σ 0 , . . .
, σ n (cid:105) with the following properties: (i).
σ 0 = ( l 0 , V 0 ) , with Init ⊇ V 0 ; (ii).
Let σ i =
( l i , V i ) .
Then there exists a transition t =
( l i , G i , U i , l i +1 )
such that: (i) t is enabled at σ i , and (ii) V
i +
=
U # ( G i ∧ V i ) .
In this case, we write σ i +
=
t ( σ i ) .
Intuitively, τ # θ represents an overapproximation of the set of concrete trajectories of F θ that pass through the “control path” (cid:104) l 0 , . .
, l n (cid:105) .
The above symbolic trajectory is safe if for all i , V i ⇒ Safe ( l i ) .
Intuitively, in this case, all concrete trajectories that the symbolic trajectory represents follow the program’s safety requirements.
Probabilistic Symbolic Execution.
A key difﬁculty with using symbolic execution to estimate the safety loss C ( θ ) is that our programs need not be differentiable, or even continuous.
On the one hand, a discontinuous conditional statement can assign very different values to the variable in different branches.
On the other hand, a slight change in θ can sometimes cause the guard G of the conditional to go from being True on some values of the variables used in conditional statement x , to being
False on all values of x .
If the symbolic trajectory τ # in which G is True serves as an edge case to judge whether a program is worst-case-safe, learning safe parameters would be difﬁcult, as there would be no gradient guiding the optimizer back to the case in which G is True .
D SE overcomes these difﬁculties using a probabilistic approach to symbolic execution.
Speciﬁcally, at a symbolic state σ i =
( l i , V i ) , the symbolic executor in D SE samples its next action following a probability distribution p θ ( t | σ i ) , where t ranges over program transitions or a special action Stop .
For a boolean expression V θ over the program variables and parameters, let Vol ( V θ ) denote the volume of the assignments to X that satisfy V θ .
Then we deﬁne: (i).
If there is a unique program transition t that is enabled at σ i , then p θ ( t | σ i ) = 1 ; (ii).
If there is no transition t that is enabled at σ i , then p θ ( Stop | σ i ) = 1 ; and (iii).
Otherwise, let t 1 , . . .
, t k be the transitions that are enabled at σ i , with t i =
( l i , G i , U i , l i +1 ) .
Then p θ ( t j | σ i ) =
Vol ( G j ∧ V i )
Vol ( V i ) .
D SE uses the distribution p θ ( t | σ i ) to sample symbolic trajectories.
Let τ # θ = (cid:104) σ 0 , . . .
, σ n (cid:105) , with σ i +
=
t i ( σ i ) .
The probability of sampling τ # is p θ ( τ # θ ) = (cid:81)
i p θ ( t i | σ i − 1 ) .
We note that p θ is differentiable in θ .
Approximate Safety Loss.
A key decision in D SE is to approximate the overall worst-case loss
C ( θ ) by the expectation of a differentiable safety loss computed per sampled symbolic trajectory.
Recall the safety loss
Unsafe ( s ) for individual states.
For σ = ( l, V ) , we deﬁne Unsafe θ ( σ ) as a differentiable approximation of the worst-case loss max s satisﬁes V Unsafe ( s ) .
We lift this loss to symbolic trajectories τ # θ = (cid:104) σ 0 , . . .
, σ n (cid:105) by deﬁning Unsafe θ ( τ # ) =
(cid:80) i
Unsafe θ ( σ i ) .
We observe that Unsafe θ is differentiable in θ .
Our approximation to the safety loss is now given by:
[Equation]
Gradient Estimation.
Our ultimate goal is to compute the gradient ∇ θ C # ( θ ) of the approximate safety loss.
At ﬁrst sight, the classic R EINFORCE estimator, or its lower-variance reﬁnements, seems suitable for this task, given that they can differentiate an integral over sampled “paths”.
However, a subtlety in our setting is that the losses for individual paths (symbolic trajectories) is a function of θ .
This calls for a generalization of traditional R EINFORCE -like estimators. [Figure]
More precisely, we adapt the derivation of R EINFORCE to our setting as follows:
[Equation]
The above derivation can be further reﬁned to incorporate the variance reduction techniques for
R EINFORCE that are commonly employed in generative modeling and reinforcement learning.
The use of such techniques is orthogonal to our main contribution, and we ignore it in this paper.
Evaluation
System Setup.
Our framework is built on top of PyTorch [24].
We use the Adam Optimizer [17] for all the experiments with default parameters and a weight decay of 0 . 000001 .
We ran all the experiments using a single-thread implementation on a Linux system with Intel Xeon Gold2.30GHz CPUs and GeForce RTX 2080 Ti GPUs.
Baselines.
We use two types of baselines: (i).
Ablation , which does not use safety constraint explicitly, but train each neural network modules in the programs for each task with its own dataset. (ii).
D IFF A
I + , an extended version of the original D IFF A I method [22].
D IFF A
I + extends D IFF A I by adding the meet and join operations following the abstract interpretation technique [10].
Benchmarks.
We give 3 case studies as benchmarks (See A.5 for detailed programs.
): (i).
Thermostat models the controlling dynamics of a thermostat, where two neural networks modeling the controller are the target to learn; (ii).
Racetrack is a variant of the navigation benchmark [6, 8].
Two vehicles are trained by path planners separately and the racetrack system is expected to learn two safe controllers so that vehicles do not crash into walls or with each other; and (iii).
Aircraft-Collision (AC) aims to learn safe parameters for an airplane that performs maneuvers to avoid a collision with a second plane.
Evaluation of Safety and Data Loss. Once training is over, we expect the learned program to fare well in terms of safety and data loss values. We evaluate the test data loss by running the learned program on 10000 initial states that were not seen during training. We evaluate the safety loss using an abstract interpreter [10] that splits the initial condition into 10000 intervals, then constructs symbolic trajectories from these programs.
This analysis is sound (unlike the approximate loss employed during D SE training) , meaning that the program is safe if the safety loss evaluates to 0. Our safety metric is the provably safe portion , which is the fraction of these 10000 trajectories that are safe.
Results.
From Figure 1a, 1c, 1e, we exhibit that D SE can learn programs with 0.99 provably safe portion even with 200 data points and the results keep when increasing the number of data used.
Meanwhile, both D IFF A I + and ablation can not provide safe programs for AC and racetrack.
For thermostat, ablation and D IFF A I + can achieve 0.8 provably safe portion only when using 5000 and 10000 data points to train.
When increasing the number of data used, ablation and D IFF
A I + can reach 0.95 provably safe portion with 10000 data points for the thermostat case. However, the variance of results are large as the minimum provably safe portion can reach 0.74 and 0.0 for ablation and
D IFF A I +.
In the thermostat case, D IFF A I +’s performance mainly comes from the guidance from the data loss rather than the safety loss.
For the other two cases, larger data size can not help the ablation and D IFF A I + to give much safer programs.
For AC, the program learnt from ablation is very close to the safe area but the ablation is still not accurate enough to give a safe program in every step.
In the racetrack, increasing the data size to train vehicles’ controllers can only satisfy the safety of “not crashing into walls”.
The property, “distance between vehicles” which depends on the interaction between neural network modules, can not be learnt to be safe when increasing data size as this property is not represented by isolated neural networks.
Meanwhile, our test data loss is sometimes larger yet comparable with the ablation and
D IFF A I + from Figure 1b, 1f.
For AC speciﬁcally, the safety constraint can help the learner overcome some local optimal yet unsafe areas to get a safer result with more accurate behaviors.  Related Work
Veriﬁcation of Neural and Neurosymbolic Models.
There are many recent papers on the veriﬁcation of worst-case properties of neural networks [4, 13, 15, 11, 31].
There are also several recent papers [14, 30, 29, 8] on the veriﬁcation of compositions of neural networks and symbolic systems (for example, plant models). To our knowledge, the present effort is the ﬁrst to integrate a method of this sort — propagation of worst-case intervals through neurosymbolic program — with the gradient-based training of neurosymbolic programs. Veriﬁed Deep Learning. There is a growing literature on methods that incorporate worst-case objectives (safety, robustness, or stability) into the training loops of neural networks [32, 22, 20, 9, 27].
Most prior work on this topic focuses on the training of single neural networks.
There are a few domain-speciﬁc efforts that consider the environment of the neural networks being trained.
For example, [26] uses spectral normalization to constrain the neural network module of a neurosymbolic controller and ensure that it respects certain stability properties.
Unlike D SE , the training process in these methods treat the neural network in an isolated way and does not consider the interactions between the neural network modules and surrounding human-written code [28, 25, 23, 16].
Veriﬁed Parameter Synthesis for Symbolic Code.
There is a large body of work on parameter synthesis for traditional symbolic code [3, 7] with respect to worst-case correctness constraints.
However, because these methods do not use contemporary gradient-based learning, scaling them to programs with neural modules is impractical.
Conclusion
We presented D SE , the ﬁrst approach to worst-case-safe parameter learning for neurosymbolic programs that embed neural networks inside potentially discontinuous symbol code.
Our key innovation is that we design and implement a new way to bridge symbolic execution and stochastic gradient estimator to represent and learn the loss of symbolic properties.
We exhibit that D SE outperforms a state-of-the-art approach to worst-case-safe deep learning.
Our current implementation of D SE uses an interval representation of symbolic states.
Future work should explore more precise representations such as zonotopes.
One challenge in D SE is that learning here can get harder as the symbolic state representation gets more precise.
In particular, if we increase the number of components of the initial symbolic state beyond a point, each component would only lead to a unique symbolic trajectory, and there would be no gradient signal to adjust the relative weights of the different symbolic trajectories.
Future work should seek to identify good, possible adaptive, tradeoffs between precision of symbolic states and ease of learning.
Acknowledgments: Work on this paper was supported by the United States Air Force and DARPA under Contract No. FA8750-20-C-0002, by ONR under Award No. N00014-20-1-2115, and by NSF under grant #1901376.
We use an equivalence between constrained and regularized learning that [1, 18], among others, have recently developed in other learning settings, we reduce our problem to a series of unconstrained optimization tasks.
We convexify the program set { [[ F θ ]] : θ ∈ R k } by considering stochastic mixtures [18] and represent the convexiﬁed set as a probabilistic function [[ F ˆ θ ]]( v ) .
Following D SE , we rewrite Equation 2 in terms of these mixtures:
[Equation]
We convert Equation 5 to a Lagrangian function
[Equation] [Equation]
, where λ ∈ R + is a Lagrange multiplier .
Following the equilibrium computation technique by [12],
Equation 5 can be rewritten as
[Equation]
Solutions to this problem can be interpreted as equilibria of a game between a λ -player and a ˆ θ -player in which the ˆ θ -player minimizes L (ˆ θ, λ ) given the current λ , and the λ -player maximizes L (ˆ θ, λ ) given the current ˆ θ .
Our overall algorithm is shown in Algorithm 1.
In this pseudocode, ν is a predeﬁned positive real.
Uniform ( θ 1 , . .
. , θ t ) is the mixture that selects
a θ i out of { θ 1 , . .
, θ t } uniformly at random.
Best θ ( λ ) refers to the ˆ θ player’s best response for a given value of λ (it can be shown that this best response is a single parameter value, rather than a mixture of parameters).
Computing this best response amounts to solving the unconstrained optimization problem min θ L ( θ, λ ) , i.e.,
[Equation]
Best λ (ˆ θ ) is the λ -player’s best response to a particular parameter mixture ˆ θ .
We deﬁne this function as
[Equation]
where S is the upper bound on λ .
Intuitively, when C # (ˆ θ ) is non-positive, the current parameter mixture is safe.
As L is a linear function of λ , the minimum value of L is achieved in this case when λ is zero.
For other cases, the minimum L is reached by setting λ to the maximum value S .
Finally, λ - Update ( θ t ) computes, in constant time, a new value of λ based on the most recent best response by ˆ θ player.
We do not describe this function in detail; please see [1] for more details.
Following prior work, we can show that Algorithm 1 converges to a value (ˆ θ, ˆ λ ) that is within additive distance ν from a saddle point for Equation 8.
To solve our original problem (Equation 2), we take
1 Example(x): // x ∈ [−5, 5] y :
=
NN θ (x)if y ≤ 1.0:z := x + 10.0else :
z := x − 5.0assert (z <= 1)
Loc = { (cid:96) 2 ,(cid:96) 3 ,(cid:96) 7 } ,X
=
{ x,y,z } ,l 0 = { (cid:96) 2 }
Init = ( − 5 ≤ x ≤ 5)
[Figure] ( l 7 )
[Figure] ( z ≤ 1) , Safe ( l 2 ) =
True , Safe ( l 3 ) =
True
Trans
[Figure] = { ( (cid:96) [Figure] , True , (cid:104)
[Figure] NN [Figure] [Figure]
(cid:105) ,(cid:96) 3 ) , ( (cid:96) [Figure] , ( y [Figure]
1 . 0) , (cid:104) z := x + 10 . 0 (cid:105) ,(cid:96) 7 ) , ( (cid:96) 3 , ( y > 1 . 0) , (cid:104)
z := [Figure] − 5 .
(cid:105) ,(cid:96) 7 ) the returned ˆ θ
and then return the real-valued parameter θ i to which this mixture assigns the highest probability.
It is easy to see that this value is an approximate solution to Equation 2.
A.2 An Example Program Following Our Approach
Programs in higher-level languages can be translated to the STS notation in a standard way.
For example, Figure 2 (left) shows a simple high-level program.
The STS for this program appears in
Figure 2 (right).
Remarkably, while the program is simple, the state-of-the-art D IFF A I approach to veriﬁed learning fails to learn safe parameters for it.
Assume that
NN θ ∈ [ − 2 , 1] when x > 0 , and NN θ ∈ (1 , 2] when x ≤ 0 .
This program has two symbolic trajectories from x ∈ [ − 5 , 5] .
We represent the state set over x, y, z :
• τ #1 = (cid:104) ( (cid:96) 2 , x ∈ [ − 5 , 5]) , ( (cid:96) 3 , x ∈ [ − 5 , 5] ∧ y ∈ [ − 2 , 2]) , ( (cid:96) 7 , x ∈ [ − 5 , 5] ∧ y ∈ [ − 2 , 1] ∧ z ∈ [5 , 15]) (cid:105) • τ #2 = (cid:104) ( (cid:96) 2 , x ∈ [ − 5 , 5]) , ( (cid:96) 3 , x ∈ [ − 5 , 5] ∧ y ∈ [ − 2 , 2]) , ( (cid:96) 7 , x ∈ [ − 5 , 5] ∧ y ∈ (1 , 2] ∧ z ∈ [ − 10 , 0]) (cid:105) .
We note that only τ #2 is safe.
We have p ( τ #1 ) =
p ( t 2 | σ 2 ) , as the other transitions in τ #1 have probability 1.
We compute p ( t 2 | σ 2 ) =
Vol ([ − 2 , 1])
Vol ([ − 2 , 2]) = 0 .
Thus, p ( τ #1 ) = 0 .
75 and similarly, p ( τ #2 ) = 0 .
A.3 Abstract Update for Neural Network Modules
We consider the box domain in the implementation.
For a program with p variables, each component in the domain represents a p -dimensional box.
Each component of the domain is a pair b = (cid:104)
b c , b e (cid:105) , where b c ∈ R p is the center of the box and b e ∈ R p ≥ 0 represents the non-negative deviations.
The interval concretization of the i -th dimension variable of b is given by
[Equation]
Now we give the abstract update for the box domain following [22].
Add.
For a concrete function f that replaces the i -th element in the input vector x ∈ R p by the sum of the j -th and k -th element:
[Equation]
The abstraction function of f is given by:
[Equation]
where M ∈ R p × p can replace the i -th element of x by the sum of the j -th and k -th element by M · b c .
Multiplication.
For a concrete function f that multiplies the i -th element in the input vector x ∈ R p by a constant w :
[Equation]
The abstraction function of f is given by:
[Equation]
where M w · b c multiplies the i -th element of b c by w and M | w | · b e multiplies the i -th element of b e with | w | .
Matrix Multiplication.
For a concrete function f that multiplies the input x ∈ R p by a ﬁxed matrix M ∈ R p [Equation] × p :
[Equation]
The abstraction function of f is given by:
[Equation]
where M is an element-wise absolute value operation.
Convolutions follow the same approach, as they are also linear operations.
ReLU.
For a concrete element-wise ReLU operation over x ∈ R p :
[Equation]
the abstraction function of ReLU is given by:
[Equation]
where b c + b e and b c − b e denotes the element-wise sum and element-wise subtraction between b c and b e .
Sigmoid.
As Sigmoid and ReLU are both monotonic functions, the abstraction functions follow the same approach.
For a concrete element-wise Sigmoid operation over x ∈ R p :
[Equation]
the abstraction function of Sigmoid is given by:
[Equation]
[Equation]
where b c + b e and b c − b e denotes the element-wise sum and element-wise subtraction between b c and b e .
All the above abstract updates can be easily differentiable and parallelized on the GPU.
A.4 Instantiation of the Unsafe Function
In general, the Unsafe ( s ) function over individual states can be any differentiable distance function between a point and a set which satisﬁes the property that Unsafe ( s )
=
if s is in the safe set, A , and Unsafe ( s ) > 0 if s is not in A .
We give the following instantiation as the unsafeness score over individual states:
[Equation]
where D IST denotes the euclidean distance between two points.
Similarly, the Unsafe ( σ ) function over symbolic states can be any differentiable distance function between two sets which satisﬁes the property that Unsafe ( σ ) =
if V is in A , and Unsafe ( σ ) >
if
V is not in A .
We give the following instantiation as a differentiable approximation of the worst-case loss max s satisﬁes V Unsafe ( s ) in our implementation:
[Equation]
A.5 Benchmarks for Case Studies
The detailed programs describing Aircraft-Collision, Thermostat and Racetrack are in Figure 3,and 5.
The π θ of Aircraft Collision and π cool θ , π heat θ in Thermostat are a 3 layer feed forward net with 64 nodes in each and a ReLU after each layer except the last one.
A sigmoid layer serves as the last layer.
Both the π agent1 θ and π agent2 θ in Racetrack are a 3 layer feed forward net with 64 nodes in each and a ReLU after each layer.
1 aircraftcollision(x): x1, y1 := x, -15.
x2, y2 := 0.0, 0.
steps := i = while i < steps: p0, p1, p2, p3, step :=
π θ (x1, y1, x2, y2, step, stage) stage := argmax(p0, p1, p2, p3) if stage ==
CRUISE:
x1, y1 := MOVE_CRUISE(x1, y1) else if stage ==
LEFT: x1, y1 :=
MOVE_LEFT(x1, y1) else if stage ==
STRAIGHT: x1, y1 := MOVE_STRAIGHT(x1, y1) else : x1, y1 := MOVE_RIGHT(x1, y1) x2, y2 :=
MOVE_2(x2, y2) i :
= i + assert (!
IS_CRASH(x1, y1, x2, y2)) return
1 thermostat(x): steps = isOn = 0.
i = while i < steps: if isOn ≤ 0.5: isOn :=
π cool θ (x) x := COOLING(x) else : isOn, heat :
=
π heat θ (x) x := WARMING(x, heat)
i :=
i + assert (!EXTREME_TEMPERATURE(x)) return
A.
6 Experimental Details and Additional Results A.6.1 System Setup.
Our framework is built on top of PyTorch [24]. We use the Adam Optimizer [17, 19] for all the experiments with default parameters and a weight decay of 0 . 000001 . For training and testing of the data loss, we use a batch size of 512. For the safety loss representation, we consider the input set as one component if not speciﬁed.
We ran all the experiments using a single-thread implementation on a Linux system with Intel Xeon Gold 5218 2.30GHz CPUs and GeForce RTX 2080 Ti GPUs.
All the experiments are tested via 10000 data for the test data loss and splitting the input set tocomponents evenly to measure safety. We use provably safe portion , which is the portion of the safe symbolic trajectories among the symbolic trajectories from the 10000 components, to denote the safety.
A.6.2 Data Generation
For each program to be synthesized, we have a benchmark program consisting of predeﬁned functions (in the form of classic programs without neural networks).The program-form module is replaced by neural networks with unknown parameters in the training and evaluation steps.
1 racetrack(x): x1, y1, x2, y2 := x, 0.0, x, 0.
steps := while i < steps: p10, p11, p12 := π agent1 θ (x1, y1) p20, p21, p22 :
= π agent2 θ (x2, y2) action1 :
= argmax(p10, p11, p12) action2 :
= argmax(p20, p21, p22) x1, y1 := MOVE(x1, y1, action1) x2, y2 :=
MOVE(x2, y2, action2)
i :
= i + assert (!
CRASH_WALL(x1, y1) && !
CRASH_WALL(x2, y2) && !CRASH(x1, y1, x2, y2)) return For each benchmark, we uniformly sample the points from a continuous input set and execute the predeﬁned benchmark program to get the input-output example dataset for each neural network modules. Speciﬁcally, as we generate the data when treating the neurosymbolic program as a whole and each program is safe, a neurosymbolic program with neural network modules 100% ﬁtting the datasets satisﬁes the safety constraint when the safety property can be measured. That said, any concrete trajectories produced by the benchmark program, which starts from the points in the input set are safe .
Interestingly, there are also benchmark programs where the safety can not be measured.
Take racetrack as an example, we consider a racetrack system safe if two vehicles do not crash into the maps and each other. The two vehicles are trained separately as we only require each of the vehicle to follow the path planner in one map and there is no speciﬁc distance requirement of them when mimicing the path planner’s behaviors. Therefore, there is no safety property representation over the distance between vehicles during data generation. The benchmark program satisﬁes other properties which exist, e.g. each vehicle not crashing into the wall.
A.6.3 Training Performance
Figure 6 illustrates the training performance of safety loss when we use 200 data points to train for
D IFF A
I + and D SE .
We can see that D SE can converge
while D IFF A I + is quite unstable for these three cases and can not converge to a small safety loss.
[Figure]
Figure 7 displays the trajectories from the learnt programs of Ablation, D IFF A I + and D SE .
The larger portion of the symbolic trajectories is provably safe from D SE than the ones from baselines which is indicated by the less overlapping between the green trajectories and the gray area.
Meanwhile, we give the concrete trajectories to remove the inﬂuence from the potential over-approximation of the symbolic trajectories.
More concrete trajectories of thermostat, aircraft-collision, and the distance property of racetrack from D SE fall into the safe area compared with D IFF A
I + and Ablation.
[Figure]
A.6.5 Target Problems
For cases studies, the three cases are all neurosymbolic programs with neural modules to make decisions.
Meanwhile, the interaction between neural networks is important for these benchmarks.
In thermostat and aircraft-collision, the neural modules’ outputs decide the actions that the neural module to take in the next steps.
In racetrack, the distance between two neural modules regulates each step of the vehicles’ controllers.
In summary, D SE performs much better in the cases where neural modules are used in the conditional statement and the interaction between neural modules is important for the safety property.