S CALABLE N EURAL L EARNING FOR V ERIFIABLE C ONSISTENCY WITH T EMPORAL S PECIFICATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Formal veriﬁcation of machine learning models has attracted attention recently, and signiﬁcant progress has been made on proving simple properties like robustness to misclassiﬁcation under perturbations of the input features.
In this context, it has also been observed that folding the veriﬁcation procedure into training makes it easier to train veriﬁably robust models.
In this paper, we extend the applicability of veriﬁed training by extending it to (1) recurrent neural network architectures and (2) complex speciﬁcations that go beyond simple adversarial robustness, particularly speciﬁcations that capture temporal properties like requiring that a robot periodically visits a charging station or that a language model always produces sentences of bounded length.
Experiments show that while models trained using standard training often violate desired speciﬁcations, our veriﬁed training method produces models that both perform well (in terms of test error or reward) and can be shown to be provably consistent with speciﬁcations.
I NTRODUCTION
While deep neural networks (DNNs) have shown immense progress on diverse tasks (Sutskever et al., 2014; Mnih et al., 2015; Silver et al., 2016), they are often deployed without formal guarantees of their correctness and functionality.
Their performance is typically evaluated using test data, or sometimes with adversarial evaluation (Carlini & Wagner, 2017; Uesato et al., 2018; Ebrahimi et al., 2018; Wang et al., 2019).
However, such evaluation does not provide formal guarantees regarding the absence of rare but possibly catastrophic failures (Administration; Board; Ross & Swetlitz, 2018).
Researchers have therefore started investigating formal veriﬁcation techniques for DNNs.
Most of the focus in this direction has been restricted to feedforward networks and robustness to adversarial perturbations (Tjeng et al., 2017; Raghunathan et al., 2018b;
Ko et al., 2019).
However, many practically relevant systems involve DNNs that lead to sequential outputs (e.g., an RNN that generates captions for images, or the states of an RL agent).
These sequential outputs can be interpreted as real-valued, discrete-time signals.
For such signals, it is of interest to provide guarantees with respect to temporal speciﬁcations (e.g., absence of repetitions in a generated sequence, or that a generated sequence halts appropriately).
Temporal logic provides a compact and intuitive formalism for capturing such properties that deal with temporal abstractions.
Here, we focus on Signal Temporal Logic (STL) (Donzé & Maler, 2010) as the speciﬁcation language and exploit its quantitative semantics to integrate a veriﬁcation procedure into training to provide guarantees with regard to temporal speciﬁcations.
Our approach builds on recent work (Mirman et al., 2018; Gowal et al., 2018), which is based on propagating differentiable numerical bounds through DNNs, to include speciﬁcations that go beyond adversarial robustness.
Additionally, we propose extensions to Mirman et al.
; Gowal et al.
(2018) that allow us to train auto-regressive
GRUs/RNNs to certiﬁably satisfy temporal speciﬁcations.
We focus on the problem of veriﬁed training for consistency rather than post-facto veriﬁcation.
To summarize, our contributions are as:
• We present extensions to Mirman et al.
; Gowal et al.
(2018) that allow us to extend veriﬁed training to novel architectures and speciﬁcations, including complex temporal speciﬁcations.
To handle the auto-regressive decoder often used in RNN-based systems, we leverage differentiable approximations of the non-differentiable operations.
• We empirically demonstrate the applicability of our approach to ensure veriﬁable consistency with temporal speciﬁcations while maintaining the ability of neural networks to achieve high accuracy on the underlying tasks across domains.
For supervised learning, veriﬁed training on the train-data enables us to provide similar veriﬁcation guarantees for unseen test-data.
• We show that veriﬁed training results in robust DNNs whose speciﬁcation conformance is signiﬁcantly easier to guarantee than those trained adversarially or with data augmentation.
R ELATED W ORK
Here, we discuss the most closely related approaches covering AI safety and DNN veriﬁcation.
2. N EURAL N ETWORK V ERIFICATION
There has been considerable recent progress on developing techniques for neural network veriﬁcation, starting with the pioneering work of Katz et al.
(2017), where a satisﬁability-modulo-theories (SMT) solver was developed to verify simple properties for piecewise linear deep neural networks.
Subsequently, several mature solvers that rely on combinatorial search (Tjeng et al., 2017; Dutta et al., 2017; Bunel et al., 2018) have helped scale these techniques to larger networks.
More recently, veriﬁcation of neural networks using incomplete over-approximations – using dual based approaches (Dvijotham et al., 2018), and propagating bounds through the neural network (Gehr et al., 2018;
Wang et al., 2018c; Weng et al., 2018a; Singh et al., 2019) – has emerged as a more scalable alternative for neural network veriﬁcation.
Raghunathan et al.
(2018a)
; Wong et al.
(2018) showed that folding the veriﬁcation procedure into the training loop (called veriﬁed training) enables us to obtain stronger guarantees.
Building on this line of work, Gowal et al.
(2018) showed that training with simple interval bounds, with carefully chosen heuristics, is an effective approach towards training for veriﬁability.
However, the focus of the above mentioned works on veriﬁed training is limited to adversarial robustness properties, and feedforward networks with monotonic activation functions.
In this work, we build on Mirman et al.
; Gowal et al.
(2018) to consider richer speciﬁcations that capture desired temporal behavior, and novel architectures with non-differentiable components.
Table 1 compares different methods developed for veriﬁed training.
[Table]
In independent and concurrent work
[Table]
Jia et al.
(2019) develop an approach for verifying LSTMs,
CNNs and networks with attention-mechanism.
They use a similar approach as developed in our paper to compute bounds through the softmax function, word-substitutions
[Table]
and also extend boundpropagation to handle the gating mechanism.
Their main focus is robustness to misclassiﬁcation.
In contrast, we consider complex temporal speciﬁcations
[Table]
and auto-regressive architectures.
2. S ATISFACTION OF TEMPORAL PROPERTIES
Temporal Speciﬁcations
While training networks to satisfy temporal logic speciﬁcations has been considered before, it has largely been from the perspective of encouraging RL agents to do so through a modiﬁed reward (Icarte et al., 2018b; Aksaray et al., 2016; Hasanbeig et al., 2018; Icarte et al., 2018a;
Sadigh et al., 2014; Wen et al., 2017; Li et al., 2017a).
The temporal logic speciﬁcation is used to express the task to be performed by the agent, rather than as a veriﬁable property of the system.
Ghosh et al.
(2018) encourage speciﬁcation conformance during training by regularizing with a loss arising from the desired speciﬁcation.
However, the speciﬁcation is enforced on speciﬁc inputs and does not guarantee that the property holds across continuous regions of the input space (e.g., all inputs in the neighborhood of a given image).
In contrast, we train DNNs to veriﬁably satisfy rich temporal speciﬁcations over large/continuous sets of inputs.
Further, we note that there is work on falsifying STL speciﬁcations using stochastic optimization(Annpureddy et al., 2011; Donzé, 2010), however our focus here is on veriﬁed training.
Safe RL Safe exploration methods (Garcıa & Fernández, 2015) consider explorations that do not visit unsafe states.
Temporal logics, in general, permit richer speciﬁcations of desired temporal behaviors than avoiding unsafe states.
Junges et al.
(2016) synthesize a scheduler that limits the agent explorations to safe regions of the environment speciﬁed using probabilistic computation tree logic (PCTL).
An alternative mechanism, called shields, monitors actions of an agent and restricts it to a safe subset ensuring conformance to speciﬁcations in linear temporal logic (Alshiekh et al., 2018) or PCTL (Jansen et al., 2018).
Instead of using external mechanisms to restrict the agent choices, we incorporate temporal logic objectives into training and achieve veriﬁed training of our agents.
Furthermore, our work is not restricted to training veriﬁable RL agents.
We demonstrate the generality of our approach on image captioning and language generation tasks involving RNNs.
Veriﬁcation using Interpretable Policies PIRL (Verma et al., 2018) and VIPER (Bastani et al., 2018) extract interpretable policies that are also amenable to veriﬁcation.
These approaches ﬁrst learn
DNN agents and then use imitation learning to extract interpretable/veriﬁable policies.
Wang et al.
(2018a) analyze RNNs by distillation to a Deterministic Finite Automaton (DFA), and demonstrate successful distillation on recognizing Tommika Grammar.
However, this distillation is often difﬁcult, and remains an open challenge for vision/language processing tasks.
Further, there are no guarantees about the original DNN.
In contrast, we focus on guarantees for the DNN itself.
F ORMULATING T EMPORAL C ONSTRAINTS ON ML M ODELS WITH S IGNAL T EMPORAL L OGIC
We consider the problem of veriﬁed training of a model with respect to a desired property.
In what follows, we describe how signal temporal logic provides a formalism to describe properties of interest.
3. P RELIMINARIES
We use Signal Temporal Logic (STL) (Donzé & Maler, 2010), an extension of Linear Temporal Logic (LTL) (Pnueli, 1977) that can reason about real-valued signals.
Syntax and Qualitative Semantics STL has the following syntax:
[Equation]
where true is the Boolean constant for truth, and ¬ and ∧ are Boolean negation and conjunction operators.
The symbol q is a quantiﬁer-free non-linear real-valued arithmetic function over the vector-valued state denoted by s ; the formula q ( s ) ≥ 0 is called an atom.
The formula ϕ 1 U I ϕ 2 is the until temporal operator, meaning ϕ 1 holds until ϕ 2 holds in the interval
I .
We deﬁne ♦ I ϕ
(meaning ϕ eventually holds in the interval I ) as true U
I ϕ
and (cid:3) I ϕ (meaning ϕ always holds in I ) as ¬ ♦ I ¬ ϕ .
In this work, we interpret STL formulae over a trace σ which is a discrete-time, vector-valued signal; σ t denotes the value of the signal at time t .
We write ( σ, t ) | = ϕ to indicate that ϕ holds for σ at time t .
An atom q ( s ) ≥ 0 holds at time t if q ( σ t ) ≥ 0 .
Trivially, ( σ, t ) | =
true always holds.
An until formula ϕ 1 U I ϕ 2 , with I =
[ a, b ] , holds at a time instance t if ( σ, t (cid:48) )
| = ϕ 2 for some time t (cid:48) ∈
[ t + a, t + b ] and ( σ, t (cid:48)(cid:48) )
| =
ϕ 1 for all t (cid:48)(cid:48) ∈
[ t, t (cid:48) ] .
In the rest of the paper, we use t + I to denote [ t + a, t + b ] for I =
[ a, b ] .
A formula ϕ is said to be satisﬁed over a trace σ if ( σ, 0) | = ϕ .
In this work, we restrict I to be bounded-intervals of time.
We refer the reader to Donzé & Maler (2010) for a detailed introduction to the semantics of STL speciﬁcations.
While bounded-time STL properties can be unrolled through time into logical properties using the
Boolean conjunction and disjunction operators (Raman et al., 2015), STL provides a succinct and intuitive notation for expressing desired temporal properties.
In contrast with prior work on veriﬁed training that only considers adversarial robustness (a linear constraint on the logits), we consider general speciﬁcations that assert temporal properties over input-output behaviors of neural networks.
Section 3.2 lists several examples of relevant properties that can be expressed in bounded-time STL.
3. STL S PECIFICATIONS FOR L EARNING T ASKS
To illustrate our approach, we consider three temporal properties that we want our networks to satisfy.
3.2. B OUNDING C APTION L ENGTH FOR I MAGE -C APTIONING
Multi-MNIST images consist of non-overlapping MNIST digits on a canvas of ﬁxed size (Figure 1).
The number of digits in each image varies between 1 and 3.
The task is to label the sequence of digits in the image, followed by an end of sequence token.
Prior work on this task (Wang et al., 2019) has shown image-to-sequence models to be vulnerable to generating sequences longer than the true number of digits in the image, under small adversarially chosen perturbations.
Here, we consider the task of training a DNN that does not output sequences longer than the desired length, while achieving similar nominal task performance.
Let y :
= f
( x ) be the sequence of logits output by the RNN model when given input image x .
For an image x , the termination speciﬁcation is formalized as follows:
[Equation]
where ϕ x ( y ) := ♦ [0 ,t ∗ x ] (cid:86) i (cid:54) =
e ( y [ t ] e − y
[ t ] i ) ≥ 0 , t ∗ x is the true number of digits in the image x , e is the label corresponding to the end of sequence token, (cid:15) > 0 is the perturbation bound.
Informally, this speciﬁcation enforces that the end of sequence token is output no later than after the true number of digits have been output by the RNN, for all inputs within (cid:15) distance from a true-image.
3.2. V ERIFYING THAT A R OBOT N EVER R UNS O UT OF C HARGE
To demonstrate our approach in the RL setting, we consider a task with a vacuum cleaning robot.
We summarize this task here (See Appendix F for more details).
The agent (robot) operates in a continuous domain with its location in ( x, y ) ∈ [0 , 25] 2 (Figure 2, Appendix).
The room is divided into discrete cells, and the agent gets a reward for visiting any “dirty” cell which has not been visited in the previous T dirt time-steps.
The agent must visit one of the recharge cells every T recharge time-steps, or the episode is terminated with no further reward.
The policy maps observations (of the agent location and a map of the room) to continuous velocity controls.
We use f θ to denote the result of applying the policy, parameterized by θ , followed by the environment update.
For this agent, we want to verify the speciﬁcation: ∀ z ∈ S (cid:15) .
( f θ ( z ) , 0) | = (cid:3) [0 ,T ] ♦ [0 ,T recharge ] ϕ recharge , where ϕ recharge corresponds to the agent being in one of the recharge cells.
This speciﬁcation ensures that, for a set of feasible starting positions S (cid:15) , for every time-step t in [0 , T ] , the agent recharges itself at least once within T recharge time-steps.
See Appendix F.1 for a detailed description of S (cid:15) .
3.2. V ERIFYING G ENERATED O UTPUTS FROM A L ANGUAGE M ODEL
A common failure mode for language models is their tendency to fall into degenerate loops, often repeating a stop-word (Wang et al., 2019).
To illustrate the applicability of STL speciﬁcations in this setting, we show how to formalize the property that a GRU language model does not repeat words consecutively.
We call this speciﬁcation bigram non-repetition.
More concretely, the desired speciﬁcation is that the output sequence does not contain bigram repetition amongst the 100 most frequent tokens in the training corpus vocabulary.
We want to verify this property over a large set of possible conditioning inputs for the generative model.
Concretely, we deﬁne an input set S of roughly 25 million preﬁxes generated from a syntactic template (See Appendix G.1 for details).
These preﬁxes are input to the LM, and then we evaluate the speciﬁcation on the model output.
Now, consider a preﬁx x and the sequence of logits y output by the recurrent GRU network f (i.e. y =
f
( x ) ), with y ( t ) k referring to the logit corresponding to the k th most-frequent token in the vocabulary at time t .
A compact formal speciﬁcation ϕ bigram ruling out bigram repetition is:
[Equation]
where T sample denotes the length of the generated sample, in our case 10.
The RNN f is required to satisfy the speciﬁcation ∀ x ∈ S. ( f ( x ) , 0)
| =
ϕ bigram .
V ERIFIABLE DNN T RAINING FOR STL S PECIFICATIONS
We consider the problem of learning a trace-valued function f θ to veriﬁably satisfy a speciﬁcation of the form ∀ x ∈ S. ( f θ ( x ) , 0) | =
ϕ , where input x ranges over set S , and f θ ( x ) is the trace generated by f θ when evaluated on x , θ represents the trainable parameters, and ϕ is an STL speciﬁcation.
We drop θ for brevity, and simply denote f θ ( x ) as f ( x ) .
Formally, our problem statement is:
Given a set of inputs S , train the parameters θ of f θ so that ∀ x ∈ S. ( f θ ( x ) , 0) | = ϕ , where ϕ is a bounded-time STL speciﬁcation.
4. O PTIMIZATION F ORMULATION OF STL V ERIFICATION
For an STL speciﬁcation ϕ , its quantitative semantics can be used to construct a function ρ ( ϕ, f ( x ) , t ) whose scalar valued output is such that ρ ( ϕ, f ( x ) , t ) ≥
⇐⇒ ( f ( x ) , t ) | =
ϕ (Donzé & Maler, 2010).
In terms of the quantitative semantics, the veriﬁcation problem is equivalent to showing that ∀ x ∈ S. ρ ( ϕ, f ( x ) , 0) ≥ 0 .
This veriﬁcation task can be written as the optimization problem of ﬁnding the sequence of inputs x such that the sequence of outputs f ( x ) result in the strongest violation of the speciﬁcation with regard to the quantitative semantics:
[Equation]
If the solution to equation 4 is negative, then there exists an input leading to the violation of ϕ .
4. B OUND P ROPAGATION
The optimization problem in equation 4 itself is often intractable; even in the case when the speciﬁcation is limited to robustness against perturbations in a classiﬁcation task, it is NP-hard (Katz et al., 2017).
There are tractable approaches to bounding the problem in equation 4 (Raghunathan et al., 2018a; Dvijotham et al., 2018), but the bounds are often too loose to provide meaningful guarantees.
To obtain a tighter bound tractably, interval bound propagation – which by itself provides loose bounds, but is efﬁcient to compute – can be leveraged for veriﬁed training to give meaningful bounds on robustness under l ∞ perturbations (Mirman et al., 2018; Gowal et al., 2018).
Our general approach for doing bound propagation on the function f is to use standard interval arithmetic.
While this is straightforward when f is a feedforward DNN (Gowal et al., 2018), here we extend bound propagation to a richer set of (temporal) speciﬁcations and architectures.
First, we highlight the novel aspects of bound propagation required for (a) auto-regressive RNNs/GRUs, (b) STL speciﬁcations.
Bound Propagation through GRUs Computing bounds across GRU cells involves propagating bounds through a multiplication operation (as a part of gating mechanisms), which can be handled by a straightforward application of interval arithmetic (Hickey et al., 2001)
(see Appendix H).
Bound Propagation through auto-regressive RNNs For language modeling and image captioning, we use GRU decoders with greedy decoding.
Greedy-decoding involves a composition of the one-hot and the argmax operations.
Both of these operations are non-differentiable.
To overcome this and compute differentiable bounds (during training), we approximate this composition with a softmax operator (with a low temperature T ).
In the limit, as T → 0 , the softmax operator converges to the composition one-hot ( argmax ( · ))
For propagating bounds through the softmax operator, we leverage that the bounds are monotonic in each of the individual inputs.
Formally, given a lower ( p ) and upper ( p ) bound on the input p to a softmax layer (i.e., p ≤ p ≤ p ), the lower w and upper bound( w ) on the output can be computed as:
[Equation]
where p i is the i th coordinate of p and p ∈ R N .
During evaluation, the one-hot ( argmax ( . )) function is used as is.
Given bounds on each coordinate of p (i.e., p ≤ p ≤ p ) and s = one-hot ( argmax ( p )) , bounds on coordinate s i can be computed as:
[Equation]
We discuss bound propagation for discrete inputs in Appendix C
Bound Propagation through the speciﬁcation First, we extend the quantiative semantics for STL speciﬁcations (Donzé & Maler, 2010) to allow us to reason over sets of inputs.
For a STL speciﬁcation ϕ in negation normal form (NNF) (See Appendix A for details on the quantitative semantics and conversion to NNF), we ﬁrst deﬁne a lower bound for the quantitative semantics of ϕ over the set
S , which we denote by ω S,f ( ϕ, 0) .
We deﬁne this bound assuming we have lower bounds on all the atoms occurring in ϕ .
Speciﬁcally, let Ω S,f ( q, t ) be a lower bound on q ( f ( x ) t ) over all inputs x ∈ S ; in other words, at each time t we have ∀ x ∈ S. Ω S,f ( q, t ) ≤ q ( f ( x ) t ) .
Now, we deﬁne the lower bound on a speciﬁcation ϕ
inductively as:
• ω S,f ( true , t ) =
+ ∞ , ω S,f
( ¬ true , t ) =
−∞ , ω S,f ( q ( s ) ≥ 0 , t ) =
Ω S,f ( q, t ) • ω S,f ( ϕ 1 ∧ ϕ 2 , t ) = min( ω S,f ( ϕ 1 , t ) , ω S,f ( ϕ 2 , t ))
• ω S,f ( ϕ 1 ∨ ϕ 2 , t ) = max( ω S,f ( ϕ 1 , t ) , ω S,f ( ϕ 2 , t ))
• ω S,f ( ϕ 1 U ϕ 2 , t ) =
max t (cid:48) ∈ t
+ I min (cid:16) ω S,f ( ϕ 2 , t (cid:48) ) , min t (cid:48)(cid:48) ∈
[ t,t (cid:48) ] ω S,f ( ϕ 1 , t (cid:48)(cid:48) )
(cid:17) .
Lemma 1.
For any time t , given lower bounds Ω S,f ( q, t ) on all the atoms q ( s ) ≥ 0 in ϕ , we have:
[Equation]
Corollary 1.
If ω S,f ( ϕ, t ) ≥ 0 , then ∀ x ∈ S. ( f ( x ) , 0) | = ϕ .
See Appendix B for proof of Lemma 1.
In order to compute the lower bounds Ω S,f ( q, t ) required for
Lemma 1, given bounds on the input x , we can ﬁrst compute bounds on the outputs f ( x ) t at each time t .
For the atoms q ( s ) ≥ 0 appearing in ϕ , given bounds on the input s we can compute bounds on q ( s ) .
These bounds can then be propagated through the speciﬁcation inductively.
4. V ERIFIED T RAINING FOR STL S PECIFICATIONS
In this section, we describe how to train a network to satisfy an STL speciﬁcation ϕ .
The quantitative semantics ρ ( ϕ, σ, 0) gives a degree to which σ satisﬁes ϕ .
First, we compute lower bounds on the values of the atoms in ϕ at each instance of time.
Then, by application of Lemma 1, we can compute the lower bound ω S,f ( ϕ, 0) satisfying ∀ x ∈ S. ω S,f ( ϕ, 0)
≤ ρ ( ϕ, f ( x ) , 0) .
Subsequently we optimize the lower bound ω S,f ( ϕ, 0) to be non-negative, thereby guaranteeing that the speciﬁcation of interest holds: ∀ x ∈ S. ρ ( ϕ, f ( x ) , 0) ≥ 0 .
Let L obj be the loss objective corresponding to the base task, for example, the cross-entropy loss for classiﬁcation tasks.
Training thus requires balancing two objectives: minimizing loss on the base task by optimizing L obj ( f θ ) , and ensuring the positivity of ω S,f θ .
We can use gradient descent to directly optimize the joint loss: L obj ( f θ ) − λ min { ω S,f θ ( ϕ, 0) , τ , where λ is a scalar hyper-parameter, τ is a positive scalar threshold ( τ ∈ R + ).
The clipping avoids having to carefully balance the two losses.
The quantitative semantics of an STL speciﬁcation ϕ is a non-smooth function of the weights of the neural network, and is difﬁcult to optimize directly with gradient descent.
We ﬁnd in practice that curriculum training, similar to Gowal et al. (2018), works best for optimizing the speciﬁcation loss, starting with enforcing the speciﬁcation over a subset S (cid:48)
⊂ S , and gradually covering the entire S .
Empirically, the curriculum approach means that the task performance ( L obj ) does not degrade much.
E XPERIMENTAL R ESULTS 5. S EQUENTIAL C APTIONING OF M ULTI -MNIST I MAGES
For this task, we perform veriﬁed training to enforce the termination speciﬁcation ϕ x (equation 4) on the training data as discussed in Section 4.3.
Post training, for unseen test-set images, we evaluate the quantitative speciﬁcation loss ω S x,(cid:15) ,f ( ϕ x , 0) .
For an image x from the test-set, if ω S x,(cid:15) ,f ( ϕ x , 0) is positive, it is guaranteed that there is no input within an l ∞ radius of (cid:15) around the current image that can cause the RNN to generate a longer sequence than the number of true digits in the image.
[Table]
RNN with ReLU activations from (Wang et al., 2019) to be veriﬁable with (cid:15) = 0 .
3 , and compare its veriﬁability with
MILP based veriﬁcation reported in Wang et al.
(2019) at different perturbation radii.
The nominal accuracy for the model trained to be veriﬁable is 93 .
9% and model trained in a standard manner is 96 .
For larger perturbations, the MILP solver times out.
‘–’ indicates that we were unable to certify robustness for any of the points in the test-set, for the given perturbation within the time-out window of 30 minutes.
[Table]
In Tables 2 and 3, veriﬁed termination accuracy refers to the fraction of unseen data for which we can verify the absence of counter-examples to the termination property (equation 4).
Nominal accuracy refers to the percentage of correctly predicted tokens – including the end of sequence token.
Table 2 compares veriﬁed training with nominal and adversarial training.
Veriﬁed training outperforms both adversarial and nominal training on both adversarial and veriﬁed termination accuracy metrics.
The pixel values are scaled to be in the range [0 , 1] .
At perturbations of size (cid:15) = 0 .
5 , the images can be turned gray; however, the DNN remains robust to such large perturbations by predicting that the image has no more than a single digit at large perturbations, while maintaining nominal accuracy on clean data.
This in contrast with robustness against misclassiﬁcation, where it is not possible to be robust at large perturbations because the speciﬁcations for images from different classes conﬂict.
Adversarial accuracy is evaluated with the iterative attack from Wang et al.
(2019) (10000 steps).
Run-time Considerations As another baseline, we compare with veriﬁed termination accuracies from Wang et al.
(2019)(Table 3).
In Wang et al.
(2019), the greedy-decoding and the speciﬁcation are turned into a MILP-query solved with the SCIP solver (Gleixner et al., 2018).
Further, we use
ReLU RNNs here because GRUs are not amenable to MILP solvers.
Veriﬁed training allows us to certify speciﬁcation conformance for much larger perturbations ( ≈ 2 orders of magnitude larger).
5. A N RL M OBILE -R OBOT A GENT
We consider the recharging speciﬁcation ϕ recharge over a time-horizon of T = 10 , for an agent starting within a l ∞ distance of (cid:15) from the center of the any of the cells (See Appendix F.1 for details on feasible initial states).
To regularize the DNN to be veriﬁable with regard to ϕ recharge , the speciﬁcation loss is obtained by rolling out the current policy through time, and propagating bounds through the rolled out policy and the dynamics.
This assumes a deterministic dynamics model.
We compare our veriﬁably trained agent to both a vanilla RL agent, and an agent trained with reward shaping as in Li et al.
(2017b).
All agents achieve a similar reward, and we do not ﬁnd speciﬁcation violations for roll-outs from 10 6 random (feasible) initial states.
To compare veriﬁability, we discretize a region within a distance of (cid:15) to each cell-center into 10 2 l ∞ balls, and verify with bound-propagation that the agent satisﬁes ϕ recharge for each sub-region.
Agents trained with veriﬁed training are signiﬁcantly more veriﬁable than agents trained otherwise, with little degradation in performance (Table 4), which is consistent with prior work in classiﬁcation (Wong & Kolter, 2018)
[Table]
[Table]
L ANGUAGE G ENERATION
[Table]
Our language model consists of a 2-layer GRU with 64 hidden nodes per layer, trained on the tiny Shakespeare corpus using a word embedding dimension of 32, and vocabulary truncated to the 2500 most frequent training words.
We evaluate the model’s ability to satisfy ϕ bigram .
We compare both a nominal model trained using log-likelihood, a model that randomly samples preﬁxes from the input space and penalizes violations to the speciﬁcation, and veriﬁed training that covers the full input space (Details in Appendix
G).
We report test set perplexity and count of violations observed over the 25M preﬁxes (Table 5).
We ﬁnd that while standard training achieves the best perplexity results, it also produces numerous speciﬁcation failures.
Sampling preﬁxes and regularizing them to avoid bigram repetition using ρ ( ϕ bigram , f ( x ) , 0) eliminates failures, but the overall evaluation cost of the exhaustive search is large.
Veriﬁable training with bound propagation, by contrast, comes with a constant computational cost of ≈ 2 forward passes.
This is because matrix multiplications form a signiﬁcant majority of the computational cost during a forward pass, and propagating bounds through a layer of the form y =
σ ( Wx + b ) , where σ is a monotonic activation function (e.g. ReLU, sigmoid, tanh), can be performed such that it only costs twice as much as a normal forward pass (Gowal et al., 2018).
Run-time Considerations Veriﬁcation with propagating bounds can be performed in under 0.
seconds (including propagating bounds through the spec), while exhaustive search over 25M preﬁxes for speciﬁcation violations takes over 50 minutes .
Further, as possible word substitutions increase, the cost for exhaustive search grows exponentially while that for bound propagation stays constant.
C ONCLUSION
Temporal properties are commonly desired from DNNs in settings where the outputs have a sequential nature.
We extend veriﬁed training to tasks that require temporal properties to be satisﬁed, and to architectures such as auto-regressive RNNs whose outputs have a sequential nature.
Our experiments suggest that veriﬁed training leads to DNNs that are more veriﬁable, and often with fewer failures.
Future work includes extending veriﬁcation/veriﬁed training to unbounded temporal properties.
Another important direction is to develop better bound propagation techniques that can be leveraged for veriﬁed training.
In the RL setting, an important direction is data-driven veriﬁcation in the absence of a known model of the environment.
Appendices A STL Q UANTITATIVE S EMANTICS
In addition to the qualitative semantics discussed in the main text, STL formulae have quantitative semantics (Donzé et al., 2013; Donzé & Maler, 2010) deﬁned inductively by the function ρ below.
For a given trace σ , with σ t indicating the value of the signal at time t , the quantitative semantics is given by:
• ρ ( true , σ, t ) =
+ ∞ • ρ ( q ( s ) ≥ 0 , σ, t ) =
q ( σ t ) • ρ ( ¬ ϕ, σ, t )
=
− ρ ( ϕ, σ, t ) • ρ ( ϕ 1 ∧ ϕ 2 , σ, t ) =
min ( ρ ( ϕ 1 , σ, t ) , ρ ( ϕ 2 , σ, t )) • ρ ( ϕ 1 U I ϕ 2 , σ, t )
= max t (cid:48)
∈ t
+ I min
[Equation]
One can obtain the qualitative semantics from the sign of the quantitative semantics.
Speciﬁcally, ( σ, t ) | =
ϕ ⇐⇒ ρ ( ϕ, σ, t ) ≥ 0 .
We can convert the formulae to their equivalent negation normal form by following the standard procedure until negations are only associated with atoms and Boolean constants.
In particular, we interpret ρ ( ¬ true , σ, t ) =
−∞ and use the disjunction operator deﬁned as ρ ( ϕ 1 ∨ ϕ 2 , σ, t ) =
max( ρ ( ϕ 1 , σ, t ) , ρ ( ϕ 2 , σ, t )) .
The normal form for ¬ ( ϕ 1 U I ϕ 2 ) is obtained by pushing the negation into the subformulae, and swapping min with max .
Finally, we turn ¬ ( q ( s ) ≥ 0) into − q ( s ) >which we approximate by − q ( s ) − δ ≥ 0 for some small δ > 0 .
B P ROOF OF L EMMA
Proof.
We proceed by induction on ϕ .
The base cases and the conjunction case are straightforward, and the atom case follows by assumption.
The disjunction case requires us to show: ω S,f ( ϕ 1 ∨ ϕ 2 , t ) ≤ min x ∈ S
(max( ρ ( ϕ 1 , f ( x ) , t ) , ρ ( ϕ 2 , f ( x ) , t ))) .
Applying the max-min inequality, the right hand side is at least max(min x ∈ S ( ρ ( ϕ 1 , f ( x ) , t )) , min x ∈ S ( ρ ( ϕ 2 , f ( x ) , t )) .
Then using the inductive hypotheses, we know this is at least max( ω S,f ( ϕ 1 , t ) , ω S,f ( ϕ 2 , t )) , and the case follows.
The case for the U operator has a similar proof based on the max-min inequality.
C B OUNDS FOR D ISCRETE I NPUTS
Tasks with discrete inputs, such as language generation tasks, encode a preﬁx sentence as conditioning before decoding a follow-up sequence of words.
Consider preﬁxes of the form x = x 0 , x 1 , . . .
such that x i ∈ S i , where S i is a ﬁnite set of tokens that can appear at position i in the input sequence.
We can propagate perturbations in the preﬁx by ﬁrst projecting the tokens S i through the embedding layer
E , and then considering the maximum and the minimum value along each embedding dimension to bound the output from E .
Formally,
[Equation]
Jia et al.
; Huang et al.
(2019) also consider bound propagation for word substitutions.
[Table]
D D ETAILED COMPARISON WITH RELATED WORK ON VERIFICATION / VERIFIED TRAINING E
MMNIST
Figure 1 depicts a sample image from the MMNIST dataset.
For this task, for the GRU decoder, we train a 2 layer-convolution network with 32 ﬁlters in each layer, and the GRU decoder has 2 cells, each with a latent space of 64 dimensions.
Further, to help training with the speciﬁcation stabilize, we normalize the weights in the linear-layers of the GRU with the l 1 norm (similar to the Salimans & Kingma (2016), where the weights are normalized with the l 2 -norm.
For the curriculum training, we ﬁrst train on the original task for 5000 steps to get the full clean accuracy, and then train with the speciﬁcation loss for another 300000 steps to regularize the model to satisfy the speciﬁcation.
During this phase, we gradually ramp up the perturbation radius (cid:15) to the desired value, and also simultaneously increase the coefﬁcient λ .
The normalizing prevents the bounds from getting large during the wamring-up phase of training.
For adversarial training, we use a 7-step PGD attack from Madry et al.
and we found that a similar curriculum helps stabilize adversarial training.
Adversarial training with the largest perturbation radius as in Madry et al.
(2017) degraded the performance on the nominal task signﬁciantly, while curriculum based adversarial training degrades performance to a much lesser extent
For the test-train data split on this task, we use the same as in Wang et al.
[Equation]
F.
S (cid:15) corresponds to the states ( x, y ) within a l ∞ distance of (cid:15) from the centre of each of the cells.
We formally deﬁne this set below.
For a cell i (for Figure 3, i ∈ { 1 , 2 . . . 25 } ) with center x c i , y c i , the (cid:15) -ball
S i,(cid:15) corresponds to the set of positions ( x a , y a ) for the agent such that (cid:107) ( x a − x c i , y a − y c i ) (cid:107) ∞ ≤ (cid:15) .
Formally,
S i,(cid:15) :
= { ( x a , y a ) : (cid:107) ( x a − x c i , y a − y c i ) (cid:107) ∞ ≤ (cid:15) } .
We can then deﬁne S = ∪ i S i,(cid:15) as the set of feasible initial states of the agent for which we wish to verify the property.
Table 4 reports the fraction of cells i for which we are able to verify that the agent recharges on starting from S i,(cid:15) .
F. T ASK AND TRAINING DETAILS
The agent’s observation at each time-step t contains i) its own coordinates ( x t , y t ) , ii) for each cell, the time remaining until that cell is dirty, and iii) the (ﬁxed) locations of the recharge cells.
The agent learns a policy from these observations to a continuous control action ( a x,t , a y,t ) ∈ R 2 .
The continuous part of the agent’s state is updated as:
[Equation]
The policy is represented in the parameters θ , and the result of applying the policy then the environment update is our function f θ .
Here, we consider verifying an agents trained with Deep-Q learning in an environment with
T recharge = 3 and T dirt = 4 , i.e, every cell accumulates dust four time-steps after it was cleaned, and the robot needs to recharge itself every 4 time-steps.
Additionally, the initial cell where the robot starts can have between 0 − 0 .
1% uncertainty in the amount of dirt, the charge that can be acquired from the recharge station, and the initial battery (as another element of uncertainty in the initial position).
If the agent leaves the domain, it is clipped back into the problem domain.
The agent’s each havediscrete actions and are trained with a combination of vanilla Deep-Q learning, Deep-Q learning + reward-shaping, and Deep-Q learning + veriﬁable training.
The actions corresponds to velocities in the 4 cardinal directions, i.e., { (0 , 5) , (5 , 0) , ( − 5 , 0) , (0 , − 5) } .
For the agent trained to be veriﬁable, in addition to loss from Deep-Q learning, the agent is also trained to be veriﬁable with respect to the temporal speciﬁcation presented in Section 3.2.2.
The veriﬁcation losses are optimized using the Adam optimizer Kingma & Ba (2015) with learning rate 10 − 3 .
The weight of the veriﬁcation loss anneals linearly between 0 and 1.5 during the ﬁrst 70K steps.
The model is trained to be veriﬁable starting at the center of the grids, and eventually covering the region around the centers, with (cid:15) = 0 . 015 during the same 70k steps.
We ﬁnd that this regularizes the model to be veriﬁable in regions outside the region in which the model was trained to be veriﬁable.
G S HAKESPEARE
For the text generation task, we train a “seq2seq” model of Wu et al.
(2016) with a GRU core.
We use the standard test-train split for this task.
The reconstruction and veriﬁcation losses are optimized using the Adam optimizer (Kingma & Ba, 2015) with an initial learning rate of 10 − 1 .
The learning is decayed every 1K steps by a factor 10, till it reaches 0.001.
The weight of the veriﬁcation loss which is computed from the speciﬁcation in Section 3.2.3 varies linearly between 20 and 1 during 100000 steps.
Finally, the veriﬁcation loss is clipped between below 10.
The gradients during optimization are clipped at 0 .
1 to prevent exploding gradients from the differentiable approximations of one-hot(softmax(.))
G. P REFIX T EMPLATE
The
Language
Model preﬁxes are generated using the following syntax: <pronoun>, <person>, <action-verb>, <connector>, <person>, <pronoun>, <action-verb> , where: <pronoun> = { ’my’, ’your’, ’his’, ’her’, ’our’, ’their’ } <person> = { ’sister’, ’brother’, ’father’, ’mother’, ’son’, ’daughter’, ’king’, ’queen’, ’knight’, ’noble’,’lord’,’duke’, ’duchess’, ’cousin’, ’palace’, ’widow’,’nurse’, ’marshal’, ’archbishop’, ’mayor’, ’maid’ } <action-verb> =
[’changed’,’despised’, ’loved’, ’married’, ’accused’, ’anointed’, ’danced’, ’rejoiced’, ’killed’, ’came’, ’left’, ’prayed’, ’stood’, ’read’, ’consorted’, ’denied’, ’condemned’, ’ruled’, ’proved’, ’parted’ ’resolved’, ’committed’, ’raised’, ’urged’, ’painted’, ’provoked’, ’lived’, ’charged’, ’yielded’, ’accursed’, ’assured’], } <connector> = { ’but’, ’while’, ’yet’, ’and’, ’because’] }
The space of combinations holds 25779600 possibilities to condition the language model generation upon.
An example preﬁx is :‘ Our lord yielded and their king left ’.
H B OUND P ROPAGATION T HROUGH f
We describe how to perform bound propagation for a general recurrent neural network.
The neural network takes as input x and produces a sequence of outputs y τ for τ = 0 , ...,
K so the overall output is ( y 0 , y 1 , . . .
, y K ) .
We assume that we are given bounds on the input x l 0 ≤ x ≤ u 0 .
Our goal is to obtain bounds on y τ given bounds on x for each τ .
Each output is produced conditioned on the preceding outputs: y τ depends on y 0 , . . .
y τ − 1 .
We proceed recursively, assuming that we have already computed bounds on y 0 , . . .
y τ − 1 .
We stack the set of inputs to the computation as ( x, y 0 , . . .
, y τ − 1 ) ∈
[ l τ ;0 , u τ ;0 ] .
We study the computation graph mapping these inputs to the output y τ .
At each node in this computation graph, we perform a computation of the form
[Equation]
where h, ˜ h are element-wise nonlinear operations (sigmoid, tanh, relu etc.)
and z τ ; − i denotes the elements of computational graph that are ancestors of the node i .
The second term represents multiplicative interactions (gating interactions) common in recurrent networks like LSTMs and GRUs.
Suppose we have already computed lower and upper bounds l τ ; − i , u τ ; − i on the preceding elements.
Then, we have
[Equation]
τ ; i ) (cid:12) ( u τ ; i ) , ˜ w i (cid:12) ˜ h
( u τ ; i ) (cid:12) ( u τ ; i ) (cid:19)
z τ,i ≤ max ( w i , 0)
T h ( u τ ; − i )
+ min ( w i , 0)
T h ( l τ ; − i )
+ 1 T max (cid:18)
˜
w i (cid:12) ˜ h
( l τ ; i ) (cid:12) ( l τ ; i ) , ˜ w
i (cid:12) ˜ h ( u τ ; i ) (cid:12) ( l τ ; i ) , ˜ w i (cid:12) ˜ h ( l τ ; i ) (cid:12) ( u τ ; i ) , ˜ w i (cid:12) ˜ h
( u τ ; i ) (cid:12) ( u τ ; i ) (cid:19)
Setting l τ,i to the lower bound above and u τ,i to the upper bound, we have computed bounds on z τ,i .
Thus, we can recursively compute bounds until we obtain bounds l τ ≤ y τ ≤ u τ , which can then be used to compute bounds on y τ +1 .
Proceeding recursively, we obtain lower and upper bounds on ( y 0 , y 1 , . . .
, y K ) .
I S HAKESPEARE T EXT G ENERATIVE M ODEL S AMPLES
Method
Preﬁx
Generated Sequence
Veriﬁed Training
Very well;
and <eol> if we are like the king; and so the <eol> which of some other part
Would you <eol> could would be such other of the whole but is <eol> the a piece of
If I must not, <eol> as to be told general of the people, till he <eol> be the king to
Soft!
who comes <eol> with the city but the whole prince and he <eol> are not them to the
Nominal Training
Very well; and let <eol> 3 man <eol> will will not the new <eol> which the man of his
Would you <eol> shalt not the ﬁrst to his face that you were not <eol> to the prince
If I must not, <eol> thing the matter but the man that he have <eol> not a piece of the
Soft!
who comes <eol> these thing for the man that is the king <eol> <eol> is no matter of