S AFE N EUROSYMBOLIC L EARNING WITH D IFFERENTIABLE S YMBOLIC E XECUTION
Anonymous authors
Paper under double-blind review
A BSTRACT
We study the problem of learning worst-case-safe parameters for programs that use neural networks as well as symbolic, human-written code.
Such neurosymbolic programs arise in many safety-critical domains.
However, because they need not be continuous, let alone differentiable, they cannot be learned using existing gradient-based approaches to safe learning.
Our method, Differentiable Symbolic
Execution (D SE ), learns such programs by sampling code paths using symbolic execution, constructing gradients of a worst-case “safety loss” along these paths, and then backpropagating these gradients through program operations using a generalization of the R EINFORCE estimator.
We evaluate the method on a mix of synthetic tasks and real-world benchmarks.
Our experiments show that D SE signiﬁcantly outperforms the state-of-the-art D IFF A I method on these tasks
I NTRODUCTION
Safety on worst-case inputs has recently emerged as a key challenge in deep learning research.
Formal veriﬁcation of neural networks (Albarghouthi, 2021) is an established response to this challenge.
In particular, an exciting body of recent work (Mirman et al., 2018; Madry et al., 2017; Cohen et al., 2019; Singh et al., 2018) has sought to incorporate formal veriﬁcation into the training of neural networks.
D IFF A
I , among the most prominent of such approaches, uses a neural network veriﬁer to construct a differentiable, worst-case safety loss for the learner.
This loss is used to regularize a standard data-driven loss, biasing the learner towards parameters that are both performant and safe.
A weakness of these methods is that they only consider functional properties (such as adversarial robustness) of isolated neural networks.
By contrast, in real-world applications, neural networks are often embedded within human-written symbolic code, and correctness requirements apply to the entire neurosymbolic composition.
For example, consider a car directed by a neural controller (Qin &
Badgwell, 2000).
Safety properties for the car are functions of its trajectories, and these trajectories depend not just on the controller but also the symbolic equations that deﬁne the environment.
While recent work (Christakis et al., 2021) has studied the veriﬁcation of such neurosymbolic programs, there is no prior work on integrating veriﬁcation and learning for such systems.
In this paper, we present the ﬁrst steps towards such an integration.
The fundamental technical challenge here is that while a neural network is differentiable over its parameters, the code surrounding may be non-differentiable or even discontinuous.
This puts our problem beyond the scope of existing methods for integrated learning and veriﬁcation.
We overcome this difﬁculty using a new method, Differentiable Symbolic Execution (D SE ), for estimating gradients of worst-case safety losses of nondifferentiable neurosymbolic programs.
D SE is based on a generalization of the classic R EINFORCE estimator, which backpropagates gradients through non-differentiable operations by approximating integrals with sampling.
In our problem, the integral is the aggregate of the safety losses along symbolic control ﬂow paths in the program.
To apply R EINFORCE -like ideas here, we need to represent and sample paths in a symbolic way.
We do so through an adaptation of the classic method of symbolic execution .
We evaluate D SE through several case studies in the embedded control and navigation domains.
Our baselines include an extended version of D IFF
A I , the current state of the art, and an ablation that does not use an explicit safety loss.
Our experiments show that D SE signiﬁcantly outperforms the
1 Example(x): // x ∈ [−5, 5] y :
=
NN θ (x)if y ≤ 1.0:z := x + 10.0else :
z := x − 5.0assert (z <= 1)
[Figure]
baselines in ﬁnding safe and high-performance model parameters.
We also do a series of synthetic experiments that highlight the practical difﬁculty that
D IFF A
I -style approaches face in our setting.
To summarize, our main contributions are:
• We present the ﬁrst approach to worst-case-safe parameter learning for neural networks embedded within nondifferentiable, symbolic programs.
• As part of our learning algorithm, we give a new way to bring together symbolic execution and stochastic gradient estimators that might have applications outside our immediate task.
• We present experimental results that indicate the advantages of D SE over the state-of-the-art in veriﬁed learning.
P ROBLEM F ORMULATION
Programs.
We deﬁne programs with embedded neural networks as symbolic transition systems (STS)
(Manna & Pnueli, 2012).
Formally, a program F θ is a tuple ( Loc , X, l 0 , Init , Safe , Trans θ ) .
Here, Loc is a ﬁnite set of (control) locations , X = { x 1 , . . .
, x m } is a set of real-valued variables, and l 0 ∈ Loc is the initial location .
Init , a boolean formula over X , is an initial condition for the program.
Safe is a map from locations to constraints over X ; intuitively, Safe ( l ) is a safety requirement asserted at location l .
Finally, Trans θ is a transition relation consisting of transitions ( l, G, U θ , l (cid:48) )
such that: (i) l is the source location and l (cid:48) is the destination location ; (ii) the guard
G is a constraint over X ; and (iii) the update U θ is a vector (cid:104) U 1 ,θ , . .
U m,θ (cid:105) , where each U i,θ is a real-valued expression over X constructed using standard symbolic operators and neural networks with parameters θ .
Intuitively, U i,θ represents the update to the i -th variable.
We assume that each
U i,θ is differentiable in θ .
Also, we assume that the programs are deterministic .
That is, if G and G (cid:48) are guards for two distinct transitions from the same source state, then G ∧ G (cid:48) is unsatisﬁable.
Programs in higher-level languages can be translated to the STS notation in a standard way.
For example, Figure 1 (left) shows a simple high-level program.
The STS for this program appears in
Figure 1 (right).
Remarkably, while the program is simple, the state-of-the-art D IFF A I approach to veriﬁed learning fails to learn safe parameters for it.
Safety Semantics.
In classical formal methods, a program is considered to be safe if all of its executions satisfy a logical constraint.
However, in learning settings, it helps to know not only whether a program is unsafe, but also the extent to which it is unsafe.
Consequently, we deﬁne the safety semantics of programs in a quantitative way, in terms of a (worst-case) safety loss that quantiﬁes the extent program’s safeness.
Formally, let a state of F θ be a pair s
=
( l, v ) , where l is a location and v ∈ R m is an assignment of values to the variables (i.e., v ( i ) is the value of x i ).
Such a state is said to be at location l .
For boolean constraints B and assignments v to variables in B , let us write B ( v ) if v satisﬁes B .
A state ( l 0 , v ) , where Init ( v ) , is an initial state .
A state ( l, v ) is safe if ( Safe ( l ))( v ) .
Let v ∈ R m be an assignment to the variables.
For a real-valued expression E over X , let E ( v ) be the value of E when x i is substituted by v ( i ) .
For an update U = (cid:104) U 1 , . . .
, U n (cid:105) , we deﬁne U ( v ) as the assignment (cid:104) U 1 ( v ) , . . .
, U n ( v ) (cid:105) .
A length- n trajectory of F θ is a sequence τ =
(cid:104) s 0 , . .
, s n (cid:105) , with s i =
( l i , v i ) , such that: (i) s 0 is an initial state; and (ii) for each i , there is a transition ( l i , G, U, l i +1 ) such that G ( v i ) and v i +
=
U ( v i ) .
Let us ﬁx a size bound N for trajectories.
A trajectory τ = (cid:104) s 0 , . .
, s n (cid:105) is maximal if it has length
N , or if there is no trajectory τ (cid:48)
= (cid:104) s 0 , . .
, s n , s n +1 (cid:105) with length ≤ N .
Because our programs are deterministic, there is a unique maximal trajectory from each s 0 .
We denote this trajectory by τ ( s 0 ) .
Let us assume a real-valued loss Unsafe ( s ) that quantiﬁes the unsafeness of each state s .
We require
Unsafe ( s ) =
if s is safe and Unsafe ( s ) > 0 otherwise.
We lift this measure to trajectories τ by letting Unsafe ( τ )
=
(cid:80) s appears in τ Unsafe ( s ) .
The safety loss C ( θ ) for F θ is now deﬁned as:
[Equation]
Thus, C ( θ ) =
if and only if all program trajectories are safe.
Problem Statement.
Our learning problem formalizes a setting in which we have training data for neural networks inside a program F θ .
While training the networks with respect to this data, we must ensure that the overall program satisﬁes its safety requirements.
To ensure that the parameters of the different neural networks in F θ are not incorrectly entangled, we assume that only one of these networks, NN θ , has trainable parameters.
1 We expect as input a training set of i.i.d.
samples from an unknown distribution over the inputs and outputs of NN θ , and a differentiable data loss Q ( θ ) that quantiﬁes the network’s ﬁdelity to this training set.
Our learning goal is to solve the following constrained optimization problem:
[Equation]
L EARNING A LGORITHM
Our learning approach is based on two ideas.
First, we directly apply a recently-developed equivalence between constrained and regularized learning (Agarwal et al., 2018; Le et al., 2019) to reduce
Equation (2) to a series of unconstrained optimization problems.
Second, we use the novel technique of Differentiable Symbolic Execution (D SE ) to solve these unconstrained problems.
At the highest level, we convexify the class of programs F θ by allowing stochastic combination of programs.
That is, we now allow programs of the form F ˆ θ = (cid:80)
Tt =
1 α t F θ t , where (cid:80) i
α t = 1 .
To execute the program F θ from a given initial state, one samples a speciﬁc program F θ t from the distribution ( α 1 , . .
. , α t )
and then executes it from that state.
Equation (2) can now be written into the problem max λ ∈ R + min ˆ θ Q (ˆ θ ) + λC (ˆ θ ) , which in turn can be solved using a classic algorithm (Freund & Schapire, 1999) for computing equilibria in a twoplayer game.
We omit further details about this algorithm because it follows Le et al.
(For more details, see Appendix A.1).
A key feature of our high-level algorithm is that it repeatedly solves the optimization problem
[Equation]
for ﬁxed values of λ .
This problem is challenging because while Q ( θ ) is differentiable in θ , C ( θ ) depends on the entirety of F θ and may not even be continuous.
As we demonstrate in Section 5, this makes it difﬁcult to apply state-of-the-art gradient-based approaches to worst-case safe learning.
D SE , our main contribution, addresses this challenge by estimating gradients ∇ θ C # ( θ ) of a differentiable approximation
C # ( θ ) of C ( θ ) .
We present the details of this method in the next section.
D IFFERENTIABLE S YMBOLIC E XECUTION
Background on Symbolic Execution.
D SE is a reﬁnement of symbolic execution (Baldoni et al., 2018), a classic technique for systematic formal analysis of programs.
Symbolic execution has been
1 trainable neural modules.
In these tasks, each neural module comes with its own training data; thus, in effect, we have k instances of the learning problem.
Our implementation learns the modules simultaneously by interleaving their gradient updates.
However, the gradient of Q for each module is only inﬂuenced by its own data.
recently used to analyze the safety of neural networks (Gopinath et al., 2018).
However, we are not aware of any prior attempt to incorporate symbolic execution into neural network training.
A symbolic executor systematically searches the set of symbolic trajectories of programs, which we now deﬁne.
Consider a program F θ =
( Loc , X
=
x 1 , . .
, x m } , l 0 , Init , Safe , Trans θ ) as in Section 2.
First, we ﬁx a syntactically restricted class V θ of boolean constraints over the variables X and parameters θ .
V θ is required to be closed under conjunction.
For example, in our implementation,
V θ consists of formulas that specify closed or open intervals, with bounds given by expressions over parameters, in which each variable lies.
A symbolic state of F θ is a pair σ θ =
( l, V θ ) , where l is a location and V θ
∈ V θ .
Intuitively, σ θ represents the set of states of F θ that are at location l and satisfy the property V θ .
In particular, we designate a ﬁnite set of initial symbolic states ( l 0 , V 10 ) , . . .
, ( l p , V p 0 ) .
It is required that Init ⇒ (cid:87) i V i 0 .
Intuitively, these initial symbolic states “cover” all the possible initial states of the program F θ .
We also construct an overapproximate update U # θ : V θ → V θ for each update U θ in F θ .
For all
V ∈ V
θ , we have Post ⇒ U
# θ ( V ) , where Post ( v ) is the formula ∃ v (cid:48) :
V ( v (cid:48) )
∧ ( U ( v (cid:48) )
= v ) .
Intuitively, U # θ ( V ) is an overapproximate representation of the set of states reached by applying U θ to a state that satisﬁes V .
Let us call a transition t =
( l, G, U, l (cid:48) ) enabled at a symbolic state ( l, V ) if ( G ∧ V ) is satisﬁable.
A symbolic trajectory of F θ is a sequence τ # θ = (cid:104) σ 0 , . . .
, σ n (cid:105) with the following properties:
• σ 0 is an initial symbolic state.
• Let σ i =
( l i , V i ) .
Then there exists a transition t =
( l i , G i , U i , l i +1 )
such that: (i) t is enabled at σ i , and (ii) V
i +
=
U # ( G i ∧ V i ) .
In this case, we write σ i +
=
t ( σ i ) .
Intuitively, τ # θ represents an overapproximation of the set of concrete trajectories of F θ that pass through the “control path” (cid:104) l 0 , . .
, l n (cid:105) .
The above symbolic trajectory is safe if for all i , V i ⇒ Safe ( l i ) .
Intuitively, in this case, all concrete trajectories that the symbolic trajectory represents follow the program’s safety requirements.
Example(Cont.) Consider the example program in Figure 1.
Let us assume a single initial symbolic state where x ∈ [ − 5 , 5] and NN θ ∈ [ − 2 , 2] when x ∈ [ − 5 , 5] .
The program has two symbolic trajectories from this state:
• τ #1 = (cid:104) ( (cid:96) 2 , x ∈ [ − 5 , 5]) , ( (cid:96) 3 , x ∈ [ − 5 , 5] ∧ y ∈ [ − 2 , 2]) , ( (cid:96) 7 , x ∈ [ − 5 , 5] ∧ y ∈ [ − 2 , 1] ∧ z ∈ [5 , 15]) (cid:105) • τ #2 = (cid:104) ( (cid:96) 2 , x ∈ [ − 5 , 5]) , ( (cid:96) 3 , x ∈ [ − 5 , 5] ∧ y ∈ [ − 2 , 2]) , ( (cid:96) 7 , x ∈ [ − 5 , 5] ∧ y ∈ (1 , 2] ∧ z ∈ [ − 10 , 0]) (cid:105) .
We note that only τ #2 is safe.
Probabilistic Symbolic Execution.
A key difﬁculty with using symbolic execution to estimate the safety loss C ( θ ) is that our programs need not be differentiable, or even continuous.
To understand the issues here, consider our running example once again.
Here, the parameter θ is used to calculate the variable y , which is used in a discontinuous conditional statement that assigns very different values to the variable z in its two branches.
Additionally, a slight change in θ can sometimes cause the guard G of the conditional to go from being True on some values of y , to being False on all values of y .
In the second case, the updated θ would have one, rather than two, symbolic trajectories.
If the symbolic trajectory τ # in which G is True serves as an edge case to judge whether a program is worst-case-safe, learning safe parameters would be difﬁcult, as there would be no gradient guiding the optimizer back to the case in which G is True .
D SE overcomes these difﬁculties using a probabilistic approach to symbolic execution.
Speciﬁcally, at a symbolic state σ i =
( l i , V i ) , the symbolic executor in D SE samples its next action following a probability distribution p θ ( t | σ i ) , where t ranges over program transitions or a special action Stop .
For a boolean expression V θ over the program variables and parameters, let Vol ( V θ ) denote the volume of the assignments to X that satisfy V θ .
Then we deﬁne:
• If there is a unique program transition t that is enabled at σ i , then p θ ( t | σ i ) = 1 .
• If there is no transition t that is enabled at σ i , then p θ ( Stop | σ i )
= 1 .
• Otherwise, let t 1 , . .
t k be the transitions that are enabled at σ i , with t i =
( l i , G i , U i , l i +1 ) .
Then p θ ( t j | σ i ) =
Vol ( G j ∧ V i )
Vol ( V i ) .
We also deﬁne a volume-weighted probability distribution p ( σ 0 ) over the (ﬁnite set of) initial symbolic states σ i 0 .
Speciﬁcally, for each σ i
=
( l 0 , V i 0 ) , we have
[Equation]
D SE uses the distributions p θ ( t | σ i ) and p ( σ 0 ) to sample symbolic trajectories.
Let τ # θ = (cid:104) σ 0 , . . .
, σ n (cid:105) , with σ i +
=
t i ( σ i ) .
The probability of sampling τ # is
[Equation]
We note that p θ is differentiable in θ .
Approximate Safety Loss.
A key decision in D SE is to approximate the overall worst-case loss
C ( θ ) by the expectation of a differentiable safety loss computed per sampled symbolic trajectory.
Recall the safety loss
Unsafe ( s ) for individual states.
For σ = ( l, V ) , we deﬁne Unsafe θ ( σ ) as a differentiable approximation of the worst-case loss max s satisﬁes V Unsafe ( s ) .
We lift this loss to symbolic trajectories τ # θ = (cid:104) σ 0 , . . .
, σ n (cid:105) by deﬁning Unsafe θ ( τ # ) =
(cid:80) i
Unsafe θ ( σ i ) .
We observe that Unsafe θ is differentiable in θ .
Our approximation to the safety loss is now given by:
[Equation]
Intuitively, C # ( θ ) guides the learner towards either making the unsafe trajectories safer or lowering the probability of the program falling into unsafe trajectories.
If C # ( θ ) equals zero, the program is provably safe.
That said, in practice, we must estimate the expectation in Equation (6) using sampling.
As this sampling process may miss trajectories, a low empirical estimate of C # ( θ ) need not guarantee worst-case safety.
As an extreme example, suppose a program has one unsafe symbolic trajectory, but the volume portion of certain states in this trajectory is near-zero.
This trajectory is likely to be missed by the sampling process, leading to an artiﬁcially low value of C # ( θ ) .
Example(Cont.) Consider the trajectories τ #1 and τ #2 in our running example.
We have p ( τ #1 ) =
p ( t 2 | σ 2 ) , as the other transitions in τ # 1 have probability 1.
We compute p ( t 2 | σ 2 ) =
Vol ([ − 2 , 1])
Vol ([ − 2 , 2])
= 0 .
Thus, p ( τ #1 ) = 0 .
75 and similarly, p ( τ #2 ) = 0 .
Gradient Estimation.
Our ultimate goal is to compute the gradient ∇ θ C # ( θ ) of the approximate safety loss.
At ﬁrst sight, the classic R EINFORCE estimator, or its lower-variance reﬁnements, seems suitable for this task, given that they can differentiate an integral over sampled “paths”.
However, a subtlety in our setting is that the losses for individual paths (symbolic trajectories) is a function of θ .
This calls for a generalization of traditional R EINFORCE -like estimators.
More precisely, we adapt the derivation of R EINFORCE to our setting as follows:
[Equation]
The above derivation can be further reﬁned to incorporate the variance reduction techniques for
R EINFORCE that are commonly employed in generative modeling and reinforcement learning.
The use of such techniques is orthogonal to our main contribution, and we ignore it in this paper.
E VALUATION
Our experimental evaluation seeks to to answer the following research questions:
( RQ1): How does D SE compare with the state of the art in its ability to learn safe and performant parameters?
( RQ2): How does data size inﬂuence D SE and baselines?
( RQ3): How well does D SE scale as the program being learned gets more complex?
5. E XPERIMENTAL S ETUP
System Setup.
Our framework is built on top of PyTorch (Paszke et al., 2019).
We use the Adam
Optimizer (Kingma & Ba, 2014) for all the experiments with default parameters and a weight decay of 0 . 000001 .
We ran all the experiments using a single-thread implementation on a Linux system with Intel Xeon Gold 5218 2.30GHz CPUs and GeForce RTX 2080 Ti GPUs.
Provably Safe Portion
Template
Model
D IFF AI+
D SE
Pattern
NNSmall 0.
NNMed 0.
NNBig 0.
Pattern
NNSmall 0.
NNMed 0.
NNBig 0.
Pattern
NNSmall 1.
NNMed 1.
NNBig 1.
Pattern
NNSmall 0.
NNMed 0.
NNBig 0.
Pattern
NNSmall 0.
NNMed 0.
NNBig 0.
We use two types of baselines: (i)
Ablation , which ignores the safety constraint and learns only from data; (ii) D IFF AI+, an extended version of the original D IFF A I method (Mirman et al., 2018).
D IFF A
I does not directly ﬁt our setting of neurosymbolic programs as it does not handle general conditional statements.
D IFF AI+ extends
D IFF A I by adding the meet and join operations following the abstract interpretation framework (Cousot & Cousot, 1977).
Also, we allow D IFF AI+ to split the input set into several (100 if not speciﬁed) initial symbolic states to increase the precision of symbolic execution of each trajectory.
(The original
D IFF A I does not need to perform such splits because, being focused on adversarial robustness, it only propagates small regions around each input point through the model.)
However, the parts of D IFF
A
I that propagate losses and their gradients through neural networks remain the same in D IFF AI+.
We use the box (interval) domain for both D SE and
D IFF AI+
in all the experiments.
D SE samples 50 symbolic trajectories for each starting component.
Benchmarks.
Our evaluation uses 5 synthetic microbenchmarks and 3 case studies.
The microbenchmarks (Appendix A.5) are small programs consisting of basic neural network modules, arithmetic function assignment and conditional branches and are designed to highlight the relationship between D IFF AI+ and D SE .
The case studies, described below, involve more complicated programs (see Appendix A.4 for more details):
• In Thermostat , we want to learn safe parameters for two neural networks that control the temperature of a room.
• Racetrack is a navigation benchmark (Barto et al., 1995; Christakis et al., 2021).
Two vehicles are trained by path planners separately.
The racetrack system is expected to learn two safe controllers so that vehicles do not crash into walls or into each other.
[Figure]
• In Aircraft-Collision (AC), the goal is to learn safe parameters for an airplane that performs maneuvers to avoid a collision with a second plane.
For each benchmark, we ﬁrst write a ground-truth program that does not include neural networks and provably satisﬁes a safety requirement.
We identify certain modules of these programs as candidates for learning.
Then we replace the modules with neural networks with trainable parameters.
For all benchmarks with the exception of Racetrack, we execute the ground-truth programs on uniformly sampled inputs to collect input-output datasets for each of these modules.
As for Racetrack, recall that this benchmark involves two simulated vehicles with neural controllers; learned parameters for these controllers are safe if the vehicles do not crash into each other or a wall.
To model real-world navigation, we generate the training data for the controllers from a ground-truth trajectory constructed using a path planner.
This path planner only tries to avoid the walls in the map and does not have a representation of distances from other vehicles.
Thus, in this problem, even a very large data set does not contain all the information needed for learning safe parameters.
Evaluation of Safety and Data Loss.
Once training is over, we expect the learned program to fare well in terms of safety and data loss values.
We evaluate the test data loss by running the learned program on 10000 initial states that were not seen during training.
We evaluate the safety loss using an abstract interpreter (Cousot & Cousot, 1977) that splits the initial condition intointervals, then constructs symbolic trajectories from these programs.
This analysis is sound (unlike the approximate loss employed during D SE training) , meaning that the program is safe if the safety loss evaluates to 0.
Our safety metric is the provably safe portion , which is the fraction of these 10000 trajectories that are safe.
5. R ESULTS
RQ1: Overall Quality of Learned Parameters.
Figure 2 exhibits the provably safe portion of
D IFF AI+ and D SE on our microbenchmarks.
Here, the ﬁrst two patterns only use neural modules in the condition part and the last three benchmarks use the neural modules both in the conditional statement and the assignment statement which is directly related to the ﬁnal safety representation.
We see that D IFF AI+ cannot learn safe highly non-differentiable modules (pattern1, pattern2) while
D SE can.
For the cases where optimization across branches is not required to give safe programs,
D IFF AI+ can handle them (pattern3).
For the patterns with neural assignments, D IFF AI+ ﬁnds it hard to learn to jump from one conditional branch to another to increase safety (pattern4, pattern5).
[Figure]
We show the training-time and test-time safety and data losses for our larger benchmarks in Figure 3.
From Figure 3a, 3b, 3c, we exhibit that D SE can learn programs with 0.99 provably safe portion even with 200 data points and the results keep when the number of training data points is increased.
Meanwhile, both D IFF AI+ and Ablation fail to provide safe programs for AC and Racetrack.
For
Thermostat, Ablation and D IFF AI+ can achieve a provably safe portion of 0.8 only when usingand 10000 data points to train.
Our test data loss is sometimes larger yet comparable with the Ablation and D IFF AI+ from Figure 3d, 3f.
For AC speciﬁcally, the safety constraint can help the learner overcome some local optimal yet unsafe areas to get a safer result with more accurate behaviors.
Overall, we ﬁnd that D SE outperforms D IFF AI+
when neural modules are used in the conditional statement and the interaction between neural modules is important for the safety property.
This is the case in our larger benchmarks.
For example, in Thermostat and AC, the neural modules’ outputs decide the actions that the neural module to take in the next steps.
In Racetrack, the distance between two neural modules regulates each step of the vehicles’ controllers.
We display representative symbolic trajectories from the programs learned by Ablation, D IFF AI+ and D SE in Figure 4.
The larger portion of the symbolic trajectories is provably safe from D SE than the ones from baselines, which is indicated by the less overlapping between the green trajectories and the gray area.
Because the symbolic trajectories are overapproximate, we also depict some representative concrete trajectories for each approach.
We note that more concrete trajectories of
Thermostat, AC, and the distance property of Racetrack from D SE fall into the safe area compared with D IFF AI+ and Ablation.
RQ2: Impact of Data Size.
Figure 3 compares the different methods’ performance on the test metrics as the number of training data points is changed.
The Ablation and D IFF AI+ can reach 0.
provably safe portion with 10000 data points for Thermostat.
However, the variance of results is large, as the minimum provably safe portion across training can reach 0.74 and 0.0 for Ablation and
D IFF AI+.
In the Thermostat benchmark, D IFF AI+’s performance mainly comes from the guidance from the data loss rather than the safety loss.
For the other two cases, a larger training data set cannot help the Ablation and D IFF AI+ to give much safer programs.
For AC, as is in Figure 4e, the program learnt from Ablation is very close to the safe area but the Ablation is still not accurate enough to give a safe program in the third step.
In Racetrack, increasing the data size to train the vehicles’ controllers can only satisfy the requirement of “not crashing into walls”.
The networks cannot learn to satisfy the property of not crashing into other vehicles as this information is not available in the training data.
RQ3: Scalability.
As our solution for the safety loss does not rely on the number of data points, the solution is scalable in terms of both data size and neural network size.
In Figure 2, we use the three types of neural networks: (i) NNSmall (with about 33000 parameters); (ii) NNMed (with over 0.5 million parameters); (iii) NNBig (with over 2 million parameters), the training time per epoch is 0.09 sec, 0.10sec, 0.11sec for NNSmall, NNMed, NNBig separately.
For the synthetic benchmarks, the program can converge within about 200 epochs which takes 18 sec, 20sec, and 22 sec.
Our Thermostat, AC and Racetrack benchmarks consist of neural networks with 4933, 8836 and 4547 parameters respectively, and loops with 20, 15 and 20 iterations respectively.
There are 2, 1 and 2 neural modules (respectively) in each benchmark.
The training times for D IFF AI+ and D SE are comparable on these benchmarks.
Speciﬁcally, for Thermostat, AC, and Racetrack, the total training time of D SE is less than 1 hour, less than 90 minutes, and around 13 hours.
The corresponding numbers are over 90 minutes, more than 2 hours, and more than 17 hours for D IFF AI+.
R ELATED W ORK
Veriﬁcation of Neural and Neurosymbolic Models.
There are many recent papers on the veriﬁcation of worst-case properties of neural networks (Anderson et al., 2019; Gehr et al., 2018; Katz et al., 2017; Elboher et al., 2020; Wang et al., 2018).
There are also several recent papers (Ivanov et al., 2019; Tran et al., 2020; Sun et al., 2019; Christakis et al., 2021) on the veriﬁcation of compositions of neural networks and symbolic systems (for example, plant models).
To our knowledge, the present effort is the ﬁrst to integrate a method of this sort — propagation of worst-case intervals through neurosymbolic program — with the gradient-based training of neurosymbolic programs.
Veriﬁed Deep Learning.
There is a growing literature on methods that incorporate worst-case objectives (safety, robustness, or stability) into the training loops of neural networks (Zhang et al., 2019; Mirman et al., 2018; Madry et al., 2017; Cohen et al., 2019; Singh et al., 2018).
Most prior work on this topic focuses on the training of single neural networks.
There are a few domainspeciﬁc efforts that consider the environment of the neural networks being trained.
For example, (Shi et al., 2019) uses spectral normalization to constrain the neural network module of a neurosymbolic controller and ensure that it respects certain stability properties.
Unlike D SE , the training process in these methods treat the neural network in an isolated way and does not consider the interactions between the neural network modules and surrounding human-written code (Stone et al., 2016; Qin & Badgwell, 2000; Nassi et al., 2020; Katz et al., 2021).
Veriﬁed Parameter Synthesis for Symbolic Code.
There is a large body of work on parameter synthesis for traditional symbolic code (Alur et al., 2013; Chaudhuri et al., 2014) with respect to worst-case correctness constraints.
However, because these methods do not use contemporary gradient-based learning, scaling them to programs with neural modules is impractical.
C ONCLUSION
We presented D SE , the ﬁrst approach to worst-case-safe parameter learning for neurosymbolic programs that embed neural networks inside potentially discontinuous symbol code.
Our key innovation is that we design and implement a new way to bridge symbolic execution and stochastic gradient estimator to represent and learn the loss of symbolic properties.
We exhibit that D SE outperforms a state-of-the-art approach to worst-case-safe deep learning.
Our current implementation of D SE uses an interval representation of symbolic states.
Future work should explore more precise representations such as zonotopes.
Incorporating modern variance reduction techniques in our gradient estimator is another natural next step.
Finally, one challenge in D SE is that learning here can get harder as the symbolic state representation gets more precise.
In particular, if we increase the number of initial symbolic states beyond a point, each state would only lead to a unique symbolic trajectory, and there would be no gradient signal to adjust the relative weights of the different symbolic trajectories.
Future work should seek to identify good, possible adaptive, tradeoffs between precision of symbolic states and ease of learning.
A A PPENDIX A. L EARNING F RAMEWORK
We use an equivalence between constrained and regularized learning that (Agarwal et al., 2018;
Le et al., 2019), among others, have recently developed in other learning settings, we reduce our problem to a series of unconstrained optimization tasks.
We convexify the program set { F θ : θ ∈ R k } by considering stochastic mixtures (Le et al., 2019) and represent the convexiﬁed set as a probabilistic function F ˆ θ .
Following D SE , we rewrite Equationin terms of these mixtures:
[Equation]
We convert Equation 7 to a Lagrangian function
[Equation] [Equation]
, where λ ∈ R + is a Lagrange multiplier .
Following the equilibrium computation technique by
Freund & Schapire (1996), Equation 7 can be rewritten as
[Equation]
Solutions to this problem can be interpreted as equilibria of a game between a λ -player and a ˆ θplayer in which the ˆ θ -player minimizes L (ˆ θ, λ ) given the current λ , and the λ -player maximizes
L (ˆ θ, λ ) given the current ˆ θ .
Our overall algorithm is shown in Algorithm 1.
In this pseudocode, ν is a predeﬁned positive real.
Uniform ( θ 1 , . .
. , θ t ) is the mixture that selects
a θ i out of { θ 1 , . .
, θ t } uniformly at random.
Best θ ( λ ) refers to the ˆ θ player’s best response for a given value of λ (it can be shown that this best response is a single parameter value, rather than a mixture of parameters).
Computing this best response amounts to solving the unconstrained optimization problem min θ L ( θ, λ ) , i.e.,
[Equation]
Best λ (ˆ θ ) is the λ -player’s best response to a particular parameter mixture ˆ θ .
We deﬁne this function as
[Equation]
where S is the upper bound on λ .
Intuitively, when C # (ˆ θ ) is non-positive, the current parameter mixture is safe.
As L is a linear function of λ , the minimum value of L is achieved in this case when λ is zero.
For other cases, the minimum L is reached by setting λ to the maximum value S .
Finally, λ - Update ( θ t ) computes, in constant time, a new value of λ based on the most recent best response by ˆ θ player.
We do not describe this function in detail; please see (Agarwal et al., 2018) for more details.
Following prior work, we can show that Algorithm 1 converges to a value (ˆ θ, ˆ λ ) that is within additive distance ν from a saddle point for Equation 10.
To solve our original problem (Equation 2), we take the returned ˆ θ
and then return the real-valued parameter θ i to which this mixture assigns the highest probability.
It is easy to see that this value is an approximate solution to Equation 2.
A. A BSTRACT U PDATE FOR N EURAL N ETWORK M ODULES
We consider the box domain in the implementation.
For a program with m variables, each component in the domain represents a m -dimensional box.
Each component of the domain is a pair b = (cid:104)
b c , b e (cid:105) , where b c ∈ R m is the center of the box and
b e ∈ R m ≥ 0 represents the non-negative deviations.
The interval concretization of the i -th dimension variable of b is given by [
[Equation]
Now we give the abstract update for the box domain following Mirman et al.
Add.
For a concrete function f that replaces the i -th element in the input vector x ∈ R m by the sum of the j -th and k -th element:
[Equation]
The abstraction function of f is given by:
[Equation]
where M ∈ R m × m can replace the i -th element of x by the sum of the j -th and k -th element by
M · b c .
Multiplication.
For a concrete function f that multiplies the i -th element in the input vector x ∈ R m by a constant w :
[Equation]
The abstraction function of f is given by:
[Equation]
where M w · b c multiplies the i -th element of b c by w and M | w | · b e multiplies the i -th element of b e with | w | .
Matrix Multiplication.
For a concrete function f that multiplies the input x ∈ R m by a ﬁxed matrix M ∈ R m (cid:48) × m :
[Equation]
The abstraction function of f is given by:
[Equation]
where M is an element-wise absolute value operation.
Convolutions follow the same approach, as they are also linear operations.
ReLU.
For a concrete element-wise ReLU operation over x ∈ R m :
[Equation]
the abstraction function of ReLU is given by:
[Equation]
where b c + b e and b c − b e denotes the element-wise sum and element-wise subtraction between b c and b e .
Sigmoid.
As Sigmoid and ReLU are both monotonic functions, the abstraction functions follow the same approach.
For a concrete element-wise Sigmoid operation over x ∈ R m :
[Equation]
the abstraction function of Sigmoid is given by:
[Equation]
where b c + b e and b c − b e denotes the element-wise sum and element-wise subtraction between b c and b e .
All the above abstract updates can be easily differentiable and parallelized on the GPU.
A. I NSTANTIATION OF THE Unsafe F UNCTION
In general, the Unsafe ( s ) function over individual states can be any differentiable distance function between a point and a set which satisﬁes the property that Unsafe ( s )
=
if s is in the safe set, A , and Unsafe ( s ) > 0 if s is not in A .
We give the following instantiation as the unsafeness score over individual states:
[Equation]
where D IST denotes the euclidean distance between two points.
Similarly, the Unsafe ( σ ) function over symbolic states can be any differentiable distance function between two sets which satisﬁes the property that Unsafe ( σ ) =
if V is in A , and Unsafe ( σ ) >if V is not in A .
We give the following instantiation as a differentiable approximation of the worstcase loss max s satisﬁes V Unsafe ( s ) in our implementation:
[Equation]
A. B ENCHMARKS FOR C ASE S TUDIES
The detailed programs describing Aircraft Collision, Thermostat and Racetrack are in Figure 5,and 7.
The π θ of Aircraft Collision and π cool θ , π heat θ in Thermostat are a 3 layer feed forward net with 64 nodes in each and a ReLU after each layer except the last one.
A sigmoid layer serves as the last layer.
Both the π agent1 θ and π agent2 θ in Racetrack are a 3 layer feed forward net with 64 nodes in each and a ReLU after each layer.
1 aircraftcollision(x): x1, y1 := x, -15.
x2, y2 := 0.0, 0.
N :=
i
assert
return
A. S YNTHETIC M ICROBENCHMARKS
Figure 8 exhibits the 5 synthetic microbenchmarks in our evaluation.
Speciﬁcally, the neural network modules are used as conditions in pattern 1 and pattern 2.
The neural network modules serve as both conditions and the assignment variable in pattern 3, pattern 4 and pattern 5.
To highlight the impact from the safety loss, we only train with safety loss for synthetic microbenchmarks.
1 pattern1(x): # x ∈ [-5, 5] y :
= π θ (x) if y ≤ 1.0: z := else : z: = assert (z ≤ 1) return z (a) Pattern 1 pattern2(x): # x ∈ [-5, 5] y :
= π θ (x) if y ≤ 1.0: z := x + else : z: = xassert (z ≤ 0) return z (b) Pattern 1 pattern3(x): # x ∈ [-5, 5] y :
= π θ (x) if y ≤ 1.0: z := 10 - y else : z: = assert (z ≤ 1) return z (c) Pattern 1 pattern4(x): # x ∈ [-5, 5] y :
= π θ (x) if y ≤ -1.0: z := else : z: = 2 + y * y assert (z ≤ 1) return z (d) Pattern 1 pattern5(x): # x ∈ [-1, 1] y :
= π θ (x) if y ≤ 1.0: z :=
y else : z: = -  assert (z ≤ 0 &
[Figure] [Figure]
Figure 9 and Figure 10 illustrates the training performance of safety loss and data loss when we vary data points (2%, 10%, 20%, 50%, 100% of the training dataset) used by D IFF A
I + and D SE .
We can see that the safety loss of D SE can converge while the variance of the safety loss of D IFF
A
I + is quite large for these three cases and can not converge to a small safety loss.
While we found that starting from random initialization can give us safe programs in Thermostat
[Figure]
we achieved much quicker convergence when initializing Thermostat programs trained purely with data and then continue to train with safety loss.
We show the results and training performance for
Thermostat with data-loss initialized program.
The other two benchmarks do not beneﬁt a lot from the data loss initialized program.
We keep the training of them with random initialization
[Figure]
A.6. S AFETY L OSS
[Figure]
[Figure]
A.6. D ETAILS OF T RAINING AND T EST D ATA
We also attach the detailed training data loss ( Q ), training safety loss ( C # ), test data loss and provably safe portion for Thermostat(Figure 11), AC(Figure 12) and Racetrack(Figure 13).
Speciﬁcally, training safety loss measures the expectation of the quantitative safety of the symbolic trajectories and provably safe portion measures the portion of the symbolic trajectories that is provably safe.
Intuitively, in training, if the number of unsafe symbolic trajectories is larger and the unsafe symbolic trajectories are further away from safe areas, C # is larger.
In test, if the number of provably safe symbolic trajectories is larger, the provably safe portion is larger.
Data Size
Approach
Q
C #
Test Data Loss
Provably Safe Portion
D SE 0.
D IFF AI+ 0.
D SE 0.
D IFF AI+ 0.
D SE 0.
D IFF AI+ 0.
D SE 0.
D IFF AI+ 0.
D SE 0.
D IFF AI+ 0.
Data Size
Approach
Q
C #
Test Data Loss
Provably Safe Portion
D SE 0.
D IFF AI+ 0.
D SE 0.
D IFF AI+ 0.
D SE 0.
D IFF AI+ 0. D SE 0.
D IFF AI+ 0.
D SE 0.
D IFF AI+ 0.
Data Size
Approach
Q
C #
Test Data Loss
Provably Safe Portion
D SE 0.
D IFF AI+ 0.
D SE 0.
D IFF AI+ 0.
D SE 0.
D IFF AI+ 0.
D SE 0.
D IFF AI+ 0.
D SE 0.
D IFF AI+ 0.
A. D ATA G ENERATION
In this section we provide more details about our data generation process.
We give our ground-truth programs and the description of each benchmark:
• Thermostat: Figure 6 exhibits the program with initial temperature x ∈ [60 . 0 , 64 . 0] in training, where a 20-length loop is involved.
The COOLING and WARMING are two differentiable functions updating the temperature, x .
The π cool θ and π heat θ are two linear feed-forward neural networks.
We set the safe temperature to the area [55 . 0 , 83 . 0] .
EXTREME_TEMPERATURE measures that whether x is not within the safe temperature scope.
The ground-truth program replaces π cool θ and π heat θ with two different functional mechanisms(including branches and assignment), which decides the value of isOn and heat .
To mimic the real-world signal sensing process of thermostat, we carefully add noise to the output of these functional mechanisms and restricts the noise does not inﬂuence the safety of the ground-truth program.
• AC:
Figure 5 illustrates the program in training with initial input x-axis of aircraft1 x ∈ [12 . 0 , 16 . 0] , where a 15-length loop is involved.
The MOVE_* functions update the position of aircraft in a differentiable way.
The π θ is one linear feed-forward neural networks.
IS_CRASH measures the distances between two aircraft and the safe distance area is set to be larger than 40.0.
In the ground-truth program, the π θ is replaced by one functional mechanism mimicing the real-world aircraft planner.
If the stage is CRUISE , the groundtruth planner detects the distance between aircraft and decides the next step’s stage by assignment values 1 or 0 to p0, p1, p2, p3 .
The step is set to 0.
When the stage is in LEFT or RIGHT , stage keeps and the step increases if the number of steps in this stage is within a threshold, and the stage changes to STRAIGHT or CRUISE and the step is set to 0 otherwise.
Speciﬁcally, we convert the argmax to a nested If-Then-Else(ITE) block to select the index of the p with the maximum value.
Racetrack: Figure
illustrates the program in training with the input x-axis of each agent x ∈ [5 . 0 , 6 . 0]
(The two agents start from the same location.)
This program has a 20-length loop with a dynamic length of 223 lines (The unfold ITE of argmax consists of 3 lines code.)
In training, the π agent1 θ and π agent2 θ are two linear feed-forward neural network.
The MOVE is differentiable functions that update agents’ positions.
CRASH _ WALL measures whether the agents’ position collides with the wall.
CRASH measures whether the two agents crash into each other.
In ground-truth program, π agent1 θ and π agent2 θ are replaced by functional path planners guiding the agent’s direction.
The path planner only tries to avoid wall-collision in the map.
That is to say, in the trajectories data generated, an arbitrary pair of trajectories of agent1 and agent2 is not guaranteed to be nocollision with each other.
To model real-world navigation, we also add noise in the next step selection for one position to generate independent trajectories of each agent.
In the ground-truth program, the noise is added by uniformly selecting from the safe next step area.
When generating the trajectory dataset, we uniformly sample the input from the input space and run the ground-truth program to get the trajectory.
Speciﬁcally, the trajectories in the dataset is represented by a sequence of neural network input-output pairs and the index of the neural networks.
Take Thermostat as an example, we get a sequence of ([ x 0 ] , [ isOn 0 ] , cool ) , . . .
, ([ x k ] , [ isOn k , h k ] , heat ) , . . .
for one starting point.
The k represents the data collected in the k -th step of the program.
A. T RAINING P ROCEDURE
Our framework is built on top of PyTorch (Paszke et al., 2019).
We use the Adam Optimizer (Kingma & Ba, 2014) for all the experiments with a learning rate of 0 .
001 and a weight decay of 0 . 000001 .
We ran all the experiments using a single-thread implementation on a Linux system with Intel Xeon
Gold 5218 2.30GHz CPUs and GeForce RTX 2080 Ti GPUs.
We represent the data loss( Q ) following imitation learning techniques.
The trajectory dataset is converted to several input-output example pair sets.
Each input-output pair set represents the inputoutput pairs for each neural network.
Q is calculated by the average of the Mean Squared Error of the each neural network over the corresponding input-output pair set.
The C # of D SE follows the algorithm given in the Section 4 with one symbolic state covering the input space.
In D IFF AI+, we use sound join operation, which is used in classic abstract interpretation
Cousot & Cousot (1977), whenever encountering branches.
Sound join means that the symbolic state after the branch block is the join of the symbolic states computed from each branch.
The concretization of the symbolic state after sound join covers the union of the concretization of the symbolic states from each branch.
Intuitively, starting from one input symbolic state, D IFF AI+ gets one symbolic trajectory which covers all the potential trajectories starting from the concrete input satisfying the input symbolic state.
For the safety loss of D IFF AI+, we ﬁrst split the input space into 100 subregions evenly to give more accuracy for D IFF AI+.
After that, we can get 100 symbolic trajectories.
We compute the average of the safety loss of all the 100 symbolic trajectories as the training safety loss C # of D IFF AI+.
We set the convergence criterion as a combination of epoch number and loss.
In each training iteration, the ﬁnal loss is represented as Q ( θ )+ C # ( θ ) .
We set the maximum epoch for Thermostat,
AC and Racetrack as 1500, 1200 and 6000.
We set the early stop if the ﬁnal loss has less than a 1%
A. A DDITIONAL P ATTERN A NALYSIS
Figure 2 exhibits safety performance of learnt patterns.
The details of the patterns are in Figure 8.
We highlight the patterns that cover the scope differences between D IFF AI+ and D SE , including parameterized branch conditions(Pattern 1-4) and bad joins(Pattern 5).
In addition, we describe the characteristics in each pattern that inﬂuence D SE ’s performance, including deep local minimum in branches and limitations on sampling.
In implementation, we randomly initialize the neural network’s parameter (D IFF AI+ and D SE start from the initialization).
We observe that all the neural networks are initialized to come with an output y ≤ 1 when x ∈ [ − 5 , 5] , which gives 0 .
0 provably unsafe portion for Pattern 1, 2 and 5 in initialization.
We give a detailed description of each patterns’ characteristic (with results analysis), where all the patterns’ safety loss is over the only variable z in assertion:
Pattern1: The output, y , of the neural network module is only used as the parameters to calculate the branch condition satisfaction.
In each branch, z is only updated with constants.
When the gradient back-propagation executes in D IFF AI+, the gradients over θ is constantly zero since the branch splitting computation is not involved in safety loss.
While for D SE , we consider the y by using the volume-based probability in safety loss calculation.
Therefore, D SE is able to guide the program to fall into the second branch thoroughly by reducing the probability of y falling into the ﬁrst branch.
• Pattern2: Similar to Pattern 1, the output of the neural network is used in branch condition.
Different from Pattern 1, the input of the neural network, x , participates the assignment for z .
D IFF AI+ has the same “0 gradient" issue as in Pattern 1.
D SE sometimes reaches an area during training where only a super small portion of y falls into the ﬁrst branch and the sampling operation does not pick the portion because the probability is super small.
In this way, D SE still steadily converges with a C # = 0 . 0 .
However, the program is safer but not worst-case safe because C # = 0 .
represents all the trajectories sampled by D SE during training is safe.
• Pattern3:
Pattern 3 uses y in the branch condition as well.
For the two branches, z in the ﬁrst branch is directly assigned by a function over y and the second branch is naturally safe.
Therefore, the goal for learning safe programs is to optimize z := 10 − y to the safe area.
Both D IFF AI+ and D SE success on this pattern only optimizing over one branch is enough to learn a safe program.
Pattern4:
Pattern 4 is a typical example of local minimum.
As the branch condition is y ≤ − 1 . 0 , the neural network may be initialized to fall into both branches.
The ﬁrst branch is naturally safe and the second branch comes with a local minimum z = 2 .
D IFF AI+ is not able to learn the safe programs guided by the safe loss and D SE can learn.
D SE not achieving 1 . 0 provably safe portion as there is a part of training gets stuck in the local minimum with the second branch.
There is an exception for D IFF AI+: it gets 0.58 for NNBig.
The 0.58 can not be viewed as a result beneﬁting from the safety loss calculation as it speciﬁcally beneﬁts from the random initialization.
Since this is not a complicated task, DiffAI+ beneﬁts from weights initialization when using NNBig.
Speciﬁcally, some random initializations of NNBig may pick neural network parameters that make all the y fall into the ﬁrst branch(safe).
This leads to a not bad result of DiffAI+ for Pattern4 with NNBig.
If there are other random initializations that can beneﬁt the approach, DiffAI+ and D SE should beneﬁt in the same way.
D SE does not speciﬁcally beneﬁt from the random initialization for Pattern 4 since DiffAI+ fails on NNSmall and NNMed while D SE does not and D SE gives a safer program on NNBig. •
Pattern5:
The ﬁrst branch in this pattern is a function over the output of the neural network and the second branch is unsafe.
There are bad joins for D IFF AI+.
For here, one symbolic representation of y falling into both branches always joins with the unsafe results ( z := − 10 ).
There is no gradient from z := − 10 . 0 can direct the neural network to the safe area.
D SE can work on this pattern as there is a volume-based probability used in learning to guide the y to fall into the ﬁrst branch.
A. A DDITIONAL D ETAILS OF THE C ORE A LGORITHM
The key point of “sampling based on volume” is the volume-based probability, which is measured by the volume portion satisfying one condition.
Therefore, the volume portion must be differentiable (w.r.t θ ).
In addition, the volume is not required to be non-zero.
We give a detailed description as below: - We represent the V θ by the box domain (The detailed deﬁnition of V θ is in Appendix A.2).
- When the branch condition splits the box domain into two polyhedra (direct volume calculation for polyhedra is -P hard [1]), we add one new dimension (with a new variable) to allow that the splitting is over this new dimension.
Formally, starting from one state s = ( l, V ) , a condition if f ( x 1 , . . .
, x k ) <
=
M : . . .
{+, D SE transforms the program ﬁrst by converting the condition to x k +
+}
= f
( x 1 , . . .
, x k ); if x k +1 <
=
M : . . . .
Then, the volume portion is measured by the intersection portion of the concrete interval representation of x k +1 .
Speciﬁcally, if the additional variable’s concrete interval length is 0, it indicates that the variable represents one point.
It falls into one branch (Say branch 1) completely.
Then the probability to select branch 1 is 1.0, and the probability to select other branches is 0.0.
A. A DDITIONAL S CALABILITY E VALUATION
For most of the additional experiments for scalability, we take Thermostat.
We take AC as the base benchmark for ‘different neural network architectures’ as we add convolutional layers.
There is one variable as the input for the NN in Thermostat, and using Conv1d with kernel size=1 is the same as fully connected layers.
In the original Thermostat, there is a 20 -length loop with a dynamic program length of 20 ∗ 6 = 120 .
In each iteration, there are 2 branches.
Branches stacking on each other would increase the number of branches exponentially.
Thus, in the original Thermostat, there is 2 20 paths to go in total.
We evaluate the scalability over Thermostat by increase the number of branch in each iteration, doubling the loop length, and use a super reﬁned input size.
A.11. D IFFERENT N UMBER OF B RANCHES Data Size Approach
Q
C #
Test Data Loss
Provably Safe Portion
Ablation 0.
D IFF AI+ 0.
D SE 0.
Ablation 0.
D IFF AI+ 0.
D SE 0.
With Thermostat, we increase the number of branches in the iteration from 2 to 3, by allowing the isOn has three branches to go.
We convert the branches from {cool, heat} to {cool, lowHeat, highHeat}.
That said, the entire number of branches in the program increases from 2 20 to 3 20 .
We show that D SE still performs well.
And DiffAI+ can not give very good results even if ablation gives a reasonable number.
A.11. D IFFERENT D YNAMIC L ENGTH
With the same structure as the original thermostat, we set the loop length of 40, which doubles the dynamic length and the number of branches increases from 2 20 to 2 40 accordingly.
[Figure]
Q
C #
Test Data Loss
Provably Safe Portion
Ablation 0.
D IFF AI+ 0.
D SE 0.
Ablation 0.
D IFF AI+ 0.
D SE 0.
We show that with 200 data, D SE gives a result better than baselines but deﬁnitely less-than-perfect.
In the training where D SE did not give safe programs in the cases where the symbolic state (split by too many branches) is too reﬁned.
The symbolic state gets stuck in an area where it’s not safe but it falls into one trajectory fully.
In this way, the volume-based probability is always 1.0 for the later transitions and the gradient over θ is 0.0.
The highly unsafe stuck state gives very high state safety loss, which is exhibited by the large training safety loss for D SE .
When D SE is not stuck, it can learn safe programs.
That’s why the average provable safe portion is still larger than baselines.
We also show that, with more data, D SE can give a very good safe program for this challenging task while baselines can not.
Speciﬁcally, D IFF AI+ (splitting 100) easily gets stuck when guided by the highly non-differentiable representation of the safety loss even if ablation gives a reasonable result.
[Figure]
Q
C #
Test Data Loss
Provably Safe Portion
Ablation 0.
D IFF AI+ 0.
D SE 0.
Ablation
[Figure]
[Figure]
[Figure]
IFF AI+
[Figure]
[Figure]
[Figure]
[Figure]
With the Thermostat, this task starts from a super reﬁned input size: [60.0, 60.1].
The input in the original Thermostat is [60.0, 64.0].
We can see here D SE does not give good results with 200 data and gives the same result with the Ablation when using the full dataset.
This benchmark exhibits the trade-off between reﬁnement and the ease of learning of D SE .
When the input size is small enough that it only fully falls into one trajectory with an updated θ during learning.
In this way, the volume-base probability is also 1.0 for this trajectory and there is not gradient guiding the state to jump from this trajectory to another.
Thus, when this case occurs in the training, D SE gets stuck.
A.11. D IFFERENT N EURAL N ETWORK A RCHITECTURE
With AC, we use a NN with Conv1d(1, 1, 2)-ReLU()-Conv1d(1, 1, 2)-ReLU()-Linear(4, 32)-
ReLU()-Linear(32, 6)-Sigmoid(), where Conv1d(X, Y, Z) means an input channel of X, an output channel of Y and a kernel size of Z and Linear(X, Y) means an input channel of X and an output channel of Y.
The above table exhibits that D SE can work for neural networks with convolutional layers.
For
AC speciﬁcally, all the training data comes from safe trajectories.
There is a lack of generality for
Ablation to get good test data loss.
When adding additional safety constraint(e.g.
In D SE ), it helps to overcome some local minimum areas and gives a better test data loss even if the training data loss is worse than Ablation.
Data Size
Approach
Q
C #
Test Data Loss
Provably Safe Portion
Ablation 0.
D IFF AI+ 0.
D SE 0.
Ablation 0.
D IFF AI+ 0.
D SE 0.
[Figure]
In the Cartpole experiment, we use the trajectories data from Imitation Package.
Starting from
{+S 
+}
= [ − 0 .
] 4 , we follow Bastani et al.
(2018) to train a cart of which the pole keeps upright ( θ ∈ [ − 12 ∗ 2 ∗ math.pi/ 360 , 12 ∗ 2 ∗ math.pi/ 360] , the deﬁnition of upright is the standard one from OpenAI Gym).
The state space is 4 and the action space is 2.
We use a 3 layer fully connected neural network (each layer followed by a ReLU and the last layer followed by a Sigmoid) to train.
We follow the two points in Bastani et al.
(2018) below to setup the experiment:
• We approximate the system using a ﬁnite time horizon T max = 10; • we use a linear approximation f(s, a) (cid:39) As + Ba.
We did the veriﬁcation by splitting the input space into 20 4 boxes.
And we extract the percentage of safe concrete trajectories from 100 uniformly sampled starting points.
The results for restricting pole angle are as shown in Figure 18.
Approach
Q
C #
Test Data Loss
Provably Safe Portion
Percentage of Safe Concrete Trajectories
Ablation 0.
D IFF AI+ 0.
D SE 0.
As indicated by the above ﬁgure, pure imitation learning(Ablation) can already learn a pole keeping upright.
As the training data is almost all ‘safe’, D SE ’s generability(test data loss) beneﬁts from the additional safety constraint.
To show D SE ’s ability to train a safe program without too much help from the data loss, we add another experiment following the same setting above except the constraint.
We require the cart’s position to be within a range.
That said, the cart keeps in the middle of the ﬁgure.
We set the constraint of x in the trajectory to be [-0.1, 0.1].
Approach
Q
C #
Test Data Loss
Provably Safe Portion
Percentage of Safe Concrete Trajectories
Ablation 0.
D IFF AI+ 0.
D SE 0.
From the Figure 19, we can see that D SE is better than baselines in terms of safety.
We also added the concrete trajectories measurement to clarify the safety measurement with provably safe portions.
A.11. T RAIN D SE FROM H IGHLY U NSAFE D ATA
In summary, D SE can scale to programs with reasonable length, branches and different neural network architectures.
One challenge in D SE is that learning becomes harder when symbolic state rep-
Benchmark
Data Size
Approach
Q
C #
Test Data Loss
Provably Safe Portion
Thermostat(45% Unsafety)
Ablation 0.
D IFF AI+ 0.
D SE 0.
Ablation 0.
D IFF AI+ 0.
D SE 0.
AC(49% Unsafety)
Ablation 0.
D IFF AI+ 0.
D SE 0.
Ablation 0.
D IFF AI+ 0.
D SE 0.
resentation becomes more reﬁned(including super reﬁned input space, safety constraint, and branch splitting).
We leave this open for future works to seek to identify better tradeoffs between precision of symbolic states and ease of learning.