V ERIFICATION OF N ON - LINEAR S PECIFICATIONS FOR N EURAL N ETWORKS
Chongli Qin , Krishnamurthy (Dj) Dvijotham *
Brendan O’Donoghue Rudy Bunel
Robert Stanforth Sven Gowal Jonathan Uesato Grzegorz
Swirszcz
Pushmeet Kohli
Prior work on neural network veriﬁcation has focused on speciﬁcations that are linear functions of the output of the network, e.g., invariance of the classiﬁer output under adversarial perturbations of the input.
In this paper, we extend veriﬁcation algorithms to be able to certify richer properties of neural networks.
To do this we introduce the class of convex-relaxable speciﬁcations , which constitute nonlinear speciﬁcations that can be veriﬁed using a convex relaxation.
We show that a number of important properties of interest can be modeled within this class, including conservation of energy in a learned dynamics model of a physical system; semantic consistency of a classiﬁer’s output labels under adversarial perturbations and bounding errors in a system that predicts the summation of handwritten digits.
Our experimental evaluation shows that our method is able to effectively verify these speciﬁcations.
Moreover, our evaluation exposes the failure modes in models which cannot be veriﬁed to satisfy these speciﬁcations.
Thus, emphasizing the importance of training models not just to ﬁt training data but also to be consistent with speciﬁcations.
I NTRODUCTION
Deep learning has been shown to be effective for problems in a wide variety of different ﬁelds from computer vision to machine translation (see Goodfellow et al.
; Sutskever et al.
Krizhevsky et al.
However, due to the black-box nature of deep neural networks, they are susceptible to undesirable behaviors that are difﬁcult to diagnose.
An instance of this was demonstrated by Szegedy et al.
(2013), who showed that neural networks for image classiﬁcation produced incorrect results on inputs which were modiﬁed by small, but carefully chosen, perturbations (known as adversarial perturbations ).
This has motivated a ﬂurry of research activity on designing both stronger attack and defense methods for neural networks (Goodfellow et al., 2014; Carlini & Wagner, 2017b; Papernot et al., 2016; Kurakin et al., 2016; Athalye et al., 2018; Moosavi-Dezfooli et al., 2016; Madry et al., 2017).
However, it has been observed by Carlini & Wagner (2017b) and Uesato et al.
(2018) that measuring robustness of networks accurately requires careful design of strong attacks, which is a difﬁcult task in general; failure to do so can result in dangerous underestimation of network vulnerabilities.
This has driven the need for formal veriﬁcation : a provable guarantee that neural networks are consistent with a speciﬁcation for all possible inputs to the network.
Remarkable progress has been made on the veriﬁcation of neural networks (Tjeng & Tedrake, 2017; Cheng et al., 2017; Huang et al., 2017; Ehlers, 2017; Katz et al., 2017; Bunel et al., 2017; Weng et al., 2018; Gehr et al., 2018;
Wong & Kolter, 2018; Dvijotham et al., 2018b).
However, veriﬁcation has remained mainly limited to the class of invariance speciﬁcations , which require that the output of the network is invariant to a class of transformations, e.g., norm-bounded perturbations of the input.
There are many speciﬁcations of interest beyond invariance speciﬁcations.
Predictions of ML models are mostly used not in a standalone manner but as part of a larger pipeline.
For example, in an
ML based vision system supporting a self-driving car, the predictions of the ML model are fed into the controller that drives the car.
In this context, what matters is not the immediate prediction errors of the model, but rather the consequences of incorrect predictions on the ﬁnal driving behavior produced.
More concretely, it may be acceptable if the vision system mistakes a dog for a cat, but not if it mistakes a tree for another car.
More generally, speciﬁcations that depend on the semantics of the labels are required to capture the true constraint that needs to be imposed.
Our Contributions.
Invariance speciﬁcations are typically stated as linear constraints on the outputs produced by the network.
For example, the property that the difference between the logit for the true label and any incorrect label is larger than zero.
In order to support more complex speciﬁcations like the ones that capture label semantics, it is necessary to have veriﬁcation algorithms that support nonlinear functions of the inputs and outputs.
In this paper, we extend neural network veriﬁcation algorithms to handle a broader class of convex-relaxable speciﬁcations .
We demonstrate that, for this broader class of speciﬁcations veriﬁcation can be done by solving a convex optimization problem.
The three example speciﬁcations that we study in this paper are: a) Physics speciﬁcations: Learned models of physical systems should be consistent with the laws of physics (e.g., conservation of energy), b) Downstream task speciﬁcations:
A system that uses image classiﬁers on handwritten digits to predict the sum of these digits should make only small errors on the ﬁnal sum under adversarial perturbations of the input and c) Semantic speciﬁcations:
The expected semantic change between the label predicted by a neural network and that of the true label should be small under adversarial perturbations of the input.
Finally, we show the effectiveness of our approach in verifying these novel speciﬁcations and their application as a diagnostic tool for investigating failure modes of ML models.
Related work.
Veriﬁcation of neural networks has received signiﬁcant attention in both the formal veriﬁcation and machine learning communities recently.
The approaches developed can be broadly classiﬁed into complete and incomplete veriﬁcation.
Complete veriﬁcation approaches are guaranteed to either ﬁnd a proof that the speciﬁcation is true or ﬁnd a counterexample proving that the speciﬁcation is untrue.
However, they may require exhaustive search in the worst case, and have not been successfully scaled to networks with more than a few thousand parameters to-date.
Furthermore, these exhaustive approaches have only been applied to networks with piecewise linear activation functions.
Examples of complete veriﬁcation approaches include those based on Satisﬁability Modulo Theory (SMT) solvers (Huang et al., 2017; Ehlers, 2017; Katz et al., 2017) and those based on mixed-integer programming (Bunel et al., 2017; Tjeng & Tedrake, 2017; Cheng et al., 2017).
Incomplete veriﬁcation algorithms, on the other hand, may not ﬁnd a proof even if the speciﬁcation is true.
However, if they do a ﬁnd a proof, the speciﬁcation is guaranteed to be true.
These algorithms are signiﬁcantly more scalable thus can be applied to networks with several hundred thousand parameters.
Moreover, they have been extended to apply to arbitrary feedforward neural networks and most commonly used activation functions (sigmoid, tanh, relu, maxpool, resnets, convolutions, etc.).
Incomplete veriﬁcation algorithms include those based on propagating bounds on activation functions (Weng et al., 2018; Mirman et al., 2018a) and convex optimization (Dvijotham et al., 2018b).
By making veriﬁcation both scalable to larger models and more efﬁcient – these approaches have enabled veriﬁcation to be folded into training so that networks can be trained to be provably consistent with speciﬁcations like adversarial robustness (Mirman et al., 2018b; Wong et al., 2018; Raghunathan et al., 2018).
Organization of the Paper
In Section 2 we give examples of different non-linear speciﬁcations which neural networks should satisfy, in Section 3 we formalize our approach and use it to show how the non-linear speciﬁcations in Section 2 can be veriﬁed.
Finally, in Section 4 we demonstrate experimental results.
S PECIFICATIONS BEYOND ROBUSTNESS
In this section, we present several examples of veriﬁcation of neural networks beyond the usual case of robustness under adversarial perturbations of the input – all of which are non-linear speciﬁcations.
There has been research done on relaxations of non-linear activation functions including ReLU (Wong & Kolter, 2018; Dvijotham et al., 2018a;
Weng et al., 2018; Mirman et al., 2018a), sigmoid, tanh and other arbitrary activation functions (Dvijotham et al., 2018b; Huang et al., 2017).
However, there is a distinction between relaxing non-linear activations vs non-linear speciﬁcations.
The former assumes that the relaxations are done on each neuron independently, whereas in non-linear speciﬁcations the relaxations will involve interactions across neurons within the neural network such as the softmax function.
Prior veriﬁcation algorithms that focus on adversarial robustness are unable to handle such nonlinearities, motivating the need for the new algorithms developed in this paper.
Speciﬁcations derived from label semantics: Supervised learning problems often have semantic structure in the labels.
For example, the CIFAR-100 dataset, a standard benchmark for evaluating image classiﬁcation, has a hierarchical organization of labels, e.g., the categories ‘hamster’ and ‘mouse’ belong to the super-category ‘small mammals’.
In many real world applications, misclassiﬁcation within the same super-category is more acceptable than across super-categories.
We formalize this idea by using a distance function on labels d ( i, j ) – e.g., deﬁned via the above mentioned hierarchy – and require the probability of a classiﬁer making a ‘large distance’ mistake to be small.
Formally, if we consider an input example x with true label i , the expected semantic distance can be deﬁned as E [ d ( i, j )]
= (cid:80)
j d
i, j ) P ( j | x ) , where P ( j | x ) is the probability that the classiﬁer assigns label j to input x , and we assume that d ( i, i ) = 0 .
We can write down the speciﬁcation that the expected distance is not greater than a threshold (cid:15) as
[Equation]
Here, we consider neural network classiﬁers where P ( j | x ) is the softmax over the logits produced by the network.
We note that this is a non-separable nonlinear function of the logits.
Speciﬁcations derived from physics: An interesting example of veriﬁcation appears when employing ML models to simulate physical systems (see Stewart & Ermon (2017); Ehrhardt et al.
(2017); Bar-Sinai et al.
In this context, it may be important to verify that the learned dynamic models are consistent with the laws of physics.
For example, consider a simple pendulum with damping, which is a dissipative physical system that loses energy.
The state of the system is given by ( w, h, ω ) where w and h represent the horizontal and vertical coordinates of the pendulum respectively, and ω refers to the angular velocity.
For a pendulum of length l and mass m the energy of the system is given by
[Equation]
here g is the gravitational acceleration constant.
Suppose we train a neural network, denoted f , to model the next-state of the system given the current state: ( w (cid:48) , h (cid:48) , ω (cid:48) )
=
f ( w, h, ω ) , where ( x (cid:48) , h (cid:48) , ω (cid:48) ) represents the state at the next time step (assuming discretized time).
For this dissipative physical system the inequality
[Equation]
should be true for all input states { w, h, ω } .
We note, this requires our veriﬁcation algorithm to handle quadratic speciﬁcations.
Speciﬁcations derived from downstream tasks: Machine learning models are commonly used as a component in a pipeline where its prediction is needed for a downstream task.
Consider an accounting system that uses a computer vision model to read handwritten transactions and subsequently adds up the predicted amounts to obtain the total money inﬂow/outﬂow.
In this system, the ultimate quantity of interest is the summation of digits - even if the intermediate predictions are incorrect, the system cares about the errors accumulated in the ﬁnal sum.
Formally, given a set of handwritten transactions { x n } Nn =1 and corresponding true transaction values { i n } Nn =1 , the expected error is
[Equation]
where J is the set of possible output labels.
We would like to verify that the expected error accumulated for the inﬂow should be less than some (cid:15) >
[Equation]
In contrast to the standard invariance speciﬁcation, this speciﬁcation cares about the error in the sum of predictions from using the ML model N times.
F ORMULATION OF NONLINEAR SPECIFICATIONS
We now formalize the problem of verifying nonlinear speciﬁcations of neural networks.
Consider a network mapping f : X → Y , with speciﬁcation F : X × Y → R which is dependent on both the network inputs x and outputs y = f
( x ) .
Given S in ⊆ X , the set of inputs of interest, we say that the speciﬁcation F is satisﬁed if
[Equation]
For example, the case of perturbations of input x nom in an l ∞ -norm ball uses
[Equation]
where δ is the perturbation radius.
For this kind of speciﬁcation, one way to solve the veriﬁcation problem is by falsiﬁcation , i.e., searching for an x that violates (6) using gradient based methods (assuming differentiability).
More concretely, we can attempt to solve the optimization problem
[Equation]
using, for example, a projected gradient method of the form
[Equation]
where η i is the learning rate at time-step i and Proj denotes the Euclidean projection
[Equation]
If this iteration ﬁnds a point x ∈ S in such that F ( x, f ( x )) > 0 , then the speciﬁcation is not satisﬁed by the network f .
This is a well-known technique for ﬁnding adversarial examples (Carlini &
Wagner, 2017a), but we extend the use here to nonlinear speciﬁcations.
Maximizing the objective
Eq.
(6) for an arbitrary non-linear speciﬁcation is hard.
Since the objective is non-convex, gradient based methods may fail to ﬁnd the global optimum of F ( x, f ( x )) over x ∈ S in .
Consequently, the speciﬁcation can be perceived as satisﬁed even if there exists x ∈ S in that can falsify the speciﬁcation, the repercussions of which might be severe.
This calls for algorithms that can ﬁnd provable guarantees that there is no x ∈ S in such that F ( x, f ( x )) >
0 - we refer to such algorithms as veriﬁcation algorithms .
We note that, even when F is linear, solving the veriﬁcation problem for an arbitrary neural network f has been shown to be NP-hard (Weng et al., 2018) and hence algorithms that exactly decide whether (6) holds are likely to be computationally intractable.
Instead, we settle for incomplete veriﬁcation : i.e., scalable veriﬁcation algorithms which can guarantee that the speciﬁcation is truly satisﬁed.
Incomplete algorithms may fail to ﬁnd a proof that the speciﬁcation is satisﬁed even if the speciﬁcation is indeed satisﬁed; however, in return we gain computational tractability.
In order to have tractable veriﬁcation algorithms for nonlinear speciﬁcations, we deﬁne the class of convex-relaxable speciﬁcations - which we outline in detail in Section 3.2.
We show that this class of speciﬁcations contains complex, nonlinear, veriﬁcation problems and can be solved via a convex optimization problem of size proportional to the size of the underlying neural network.
In later sections, we show how the speciﬁcations derived in the prior sections can be modeled as instances of this class.
3. N EURAL NETWORK
Here let x ∈ R n , y ∈ R m denote either a single input output pair, i.e., y = f
( x ) or multiple input output pairs (for multiple inputs we apply the mapping to each input separately).
For classiﬁcation problems, the outputs of the network are assumed to be the logits produced by the model (prior to the application of a softmax).
For clarity of presentation, we will assume that the network is a feedforward network formed by K layers of the form
[Equation]
where x 0 = x , x K = y , and g k is the activation function applied onto the k th layer.
While our framework can handle wider classes of feedforward networks, such as ResNets (see Wong et al.
, we assume the structure above as this can capture commonly used convolutional and fully connected layers.
Assume that the inputs to the network lie in a bounded range S in = { x : l 0 ≤ x ≤ u 0 } .
This is true in most applications of neural networks, especially after rescaling.
We further assume that given that the input to the network is bounded by [ l 0 , u 0 ] we can bound the intermediate layers of the network, where the k th hidden layer is bounded by [ l k , u k ] .
Thus, we can bound the outputs of our network
[Equation]
Several techniques have been proposed to ﬁnd the tightest bounds for the intermediate layers such as Wong & Kolter (2018); Bunel et al.
(2017); Ehlers (2017); Dvijotham et al.
(2018b).
We refer to these techniques as bound propagation techniques.
3. C ONVEX - RELAXABLE SPECIFICATIONS
Most prior work on neural network veriﬁcation has assumed speciﬁcations of the form:
[Equation]
where c, d are ﬁxed parameters that deﬁne the speciﬁcation and S in is some set.
Here, we introduce the class of convex-relaxable speciﬁcations.
Assumption 1.
We assume that S in ⊆ R n , S out ⊆
R m are compact sets and that we have access to an efﬁciently computable 1 procedure that takes in F, S in , S out and produces a compact convex set
C ( F, S in , S out ) such that
[Equation]
When the above assumption holds we shall say that the speciﬁcation
[Equation]
[Equation]
is convex-relaxable.
Assumption 1 means we can ﬁnd an efﬁciently computable convex relaxation of the speciﬁcation which allows us to formulate the veriﬁcation as a convex optimization problem.
3. C ONVEX OPTIMIZATION FOR VERIFICATION
We can propagate the bounds of our network layer by layer given that the inputs are bounded by [ l 0 , u 0 ] thus we can construct the following optimization problem maximize z
[Equation]
13) subject to ( x 0 , x K , z
[Equation]
∈ C ( F
[Equation]
in
[Equation]
out
[Equation]
over the variables x, x 1 , · · · , x k .
Here, Relax( g k ) represents the relaxed form of the activation function g k .
For example, in Bunel et al.
(2017); Ehlers (2017) the following convex relaxation of the ReLU( x ) = max( x, 0) function for input x ∈ [ l, u ] was proposed:
[Equation]
For simplicity, we use g k = ReLU and the above relaxation throughout, for other activation functions see Dvijotham et al.
(2018b).
Lemma 3.1.
Given a neural network f , input set S in , and a convex-relaxable speciﬁcation F , we have that F ( x, y ) ≤ 0 for all x ∈ S in , y = f
( x ) if the optimal value of (13) is smaller than 0 and further (13) it is a convex optimization problem.
Proof.
This follows from Assumption 1, for details we refer to Appendix A.
3. M ODELING SPECIFICATIONS IN A CONVEX - RELAXABLE MANNER
Speciﬁcations involving the Softmax.
The speciﬁcations on label semantics from (1) can be modeled as linear functions of the softmax layer of the neural network: P ( j | x ) =
e y j / (cid:80) k
e y k [Equation] y = f ( x ) .
The semantic distance constraint (1) can then be written as
[Equation]
where we use d j = d ( i, j ) for brevity, and d i = 0 .
Given initial constraints on the input x (of the form x ∈ S in = { x : l 0 ≤ x ≤ u 0 } ), we use bound propagation techniques to obtain bounds on
S out =
{ y : l K ≤ y ≤ u K } (as described in Section (Bunel et al., 2017; Dvijotham et al., 2018b)).
Thus the speciﬁcation can be rephrased as
[Equation]
The set C ( F, S in , S out ) can be constructed by noting that the exponential function is bounded above by the linear interpolation between ( l K , exp( l K )) and ( u K , exp( u K )) .
This is given by
[Equation]
We propose the following convex relaxation of the exp function:
[Equation]
Given the above relaxation it can be shown (refer to Appendix B), the set below satisﬁes (11):
[Equation]
A similar procedure can be extended to the downstream speciﬁcations stated in (5).
Speciﬁcations involving quadratic functions.
The energy speciﬁcation (3) involves a quadratic constraint on both inputs and outputs of the network.
Here we show how general quadratic constraints are convex relaxable.
Consider an arbitrary quadratic speciﬁcation on the input and output of the network:
[Equation]
where Q ∈ R (1+ n + m ) × (1+ n + m ) is a ﬁxed matrix.
This can be rewritten as F ( x, y ) =
Tr ( QX ) where X = αα T , Tr ( X ) =
(cid:80) ni X ii , and α T = (cid:2) 1 x T y T (cid:3) .
Note that X ij = α i α j , and we have bounds on x and y which implies bounds on α .
We denote these bounds as l i ≤ α i ≤ u i with l 0 = u 0 = 1 .
The set { ( x, y, z ) : F ( x, y )
=
z, x ∈ S in , y ∈ S out } can be rewritten as
[Equation]
Further we note that along the diagonal, we have the functional form X ii = α 2 i .
This quadratic form is bounded above by the linear interpolation between ( l, l 2 ) and ( u, u 2 ) which is:
[Equation]
We can relax the constraint X = αα T as follows (for more details see Appendix C):
[Equation]
where the notation A (cid:23) B is used to indicate that A − B is a symmetric positive semideﬁnite matrix.
Given the above relaxation we can show (refer to Appendix C) that the set below satisﬁes (11):
[Equation]
E XPERIMENTS
We have proposed a novel set of speciﬁcations to be veriﬁed as well as new veriﬁcation algorithms that can verify whether these speciﬁcations are satisﬁed by neural networks.
In order to validate our contributions experimentally, we perform two sets of experiments: the ﬁrst set of experiments tests the ability of the convex relaxation approach we develop here to verify nonlinear speciﬁcations and the second set of experiments tests that the speciﬁcations we verify indeed provide useful information on the ability of the ML model to solve the task of interest.
Tightness of veriﬁcation: We proposed incomplete veriﬁcation algorithms that can produce conservative results, i.e., even if the model satisﬁes the speciﬁcation our approach may not be able to prove it.
In this set of experiments, we quantify this conservatism on each of the speciﬁcations from
Section 2.
In order to do this, we compare the results from the veriﬁcation algorithm with those from the gradient based falsiﬁcation algorithm (8) and other custom baselines for each speciﬁcation.
We show that our veriﬁcation algorithm can produce tight veriﬁcation results, i.e., the fraction of instances on which the veriﬁcation algorithm fails to ﬁnd a proof of a correctness but the falsiﬁcation algorithm cannot ﬁnd a counter-example is small.
In other words, empirically our strategy is able to verify most instances for which the property to be veriﬁed actually holds.
Value of veriﬁcation:
In this set of experiments, we demonstrate the effectiveness of using veriﬁcation as a tool to detect failure modes in neural networks when the speciﬁcations studied are not satisﬁed.
For each of the speciﬁcations described in Section 2, we compare two models A and
B which satisfy our speciﬁcation to varying degrees (see below and Appendix D for the details).
We show that models which violate our convex-relaxed speciﬁcations more can exhibit interesting failure modes which would otherwise be hard to detect.
4. E VALUATION METRICS
Ideally, we would like to verify that the speciﬁcation is true for all “reasonable” inputs to the model.
However, deﬁning this set is difﬁcult in most applications - for example, what is a reasonable input to a model recognizing handwritten digits?
Due to the difﬁculty of deﬁning this set, we settle for the following weaker veriﬁcation: the speciﬁcation is true over S in ( x nom , δ ) =
{ x : (cid:107) x − x nom (cid:107) ∞ ≤ δ } , where x nom is a point in the test set.
Since the test set consists of valid samples from the data generating distribution, we assume that inputs close to the test set will also constitute reasonable inputs to the model.
We then deﬁne the following metrics which are used in our experiments to gauge the tightness and effectiveness our veriﬁcation algorithm:
Veriﬁcation bound: This is the fraction of test examples x nom for which the speciﬁcation is provably satisﬁed over the set S in ( x nom , δ ) using our veriﬁcation algorithm.
Adversarial bound: This is the fraction of test examples x nom for which the falsiﬁcation algorithm based on (8) was not able to ﬁnd a counter-example in the set S in ( x nom , δ ) .
The real quantity of interest is the fraction of test examples for which the speciﬁcation is satisﬁeddenoted as β .
We would like the veriﬁcation bound to be close to β .
However, since β is difﬁcult to compute (due to the intractability of exact veriﬁcation), we need a measurable proxy.
To come up with such a proxy, we note that the veriﬁcation bound is always lower than the adversarial bound, as the attack algorithm would not be able to ﬁnd a counter-example for any test example that is veriﬁable.
Formally, the following holds:
[Equation]
Thus | Veriﬁcation bound − β | ≤ Adversarial bound − Veriﬁcation bound.
Hence, we use the difference between the Adversarial bound and the Veriﬁcation bound to gauge the tightness of our veriﬁcation bound.
4. E XPERIMENTAL SETUP
For each of the speciﬁcations we trained two networks (referred to as model A and B in the following) that satisfy our speciﬁcation to varying degrees.
In each case model A has a higher veriﬁcation bound than model B (i.e, it is consistent with the speciﬁcation over a larger fraction of the test set) – while both networks perform similarly wrt.
predictive accuracy on the test set.
The experiments in
Section 4.3 only use model A while experiments in Section 4.4 uses both model A and model B.
In brief (see the Appendix D for additional details):
MNIST and CIFAR-10: For these datasets both models was trained to maximize the log likelihood of true label predictions while being robust to adversarial examples via the method described in (Wong & Kolter, 2018).
The training for model
A places a heavier loss than model B when robustness measures are violated.
Mujoco: To test the energy speciﬁcation, we used the Mujoco physics engine (Todorov et al., 2012) to create a simulation of a simple pendulum with damping friction.
We generate simulated trajectories using Mujoco and use these to train a one-step model of the pendulum dynamics.
Model B was trained to simply minimize (cid:96) 1 error of the predictions while model A was trained with an additional penalty promoting conservation of energy.
4. T IGHTNESS OF V ERIFICATION
[Figure]
P e r c e n t a g e
V e r
i f i e d i n T e s t S e t [Figure] n t [Figure]
g
[Figure]
V
[Figure]
r i f i [Figure] Semantic speciﬁcations on CIFAR-10. We study the semantic distance speciﬁcation (1) in the context of the CIFAR-10 dataset.
We deﬁne the distances d ( i, j ) between labels as their distance according to Wordnet (Miller, 1995)
(the full distance matrix used is shown in Appendix F).
We require that the expected semantic distance from the true label under an adversarial perturbation of the input of radius δ is smaller than (cid:15)
= .
The threshold 0 .
23 was chosen so that subcategories of man-made transport are semantically similar and subcategories of animals are semantically similar.
The results are shown in Fig.
1 (left).
For all δ ≤ 6 / 255 , the gap between the adversarial and veriﬁcation bound is smaller than .
Errors in a system predicting sums of digits.
We study a speciﬁcation of the type (5) with N =and (cid:15) = 1 .
The results are shown in Figure 1 (middle).
For δ ≤ 2 / 255 , the gap between the adversarial and veriﬁcation bound is less than 1 .
At δ = 6 / 255 , the gap increases to about 13% .
We suspect this is because of the product of softmax probabilities, that degrades the tightness of the relaxation.
Tighter relaxations that deal with products of probabilities are in interesting direction for future work - signomial relaxations (Chandrasekaran & Shah, 2016) may be useful in this context.
Conservation of energy in a simple pendulum.
We study energy conservation speciﬁcation in a damped simple pendulum, as described in (3) and learn a dynamics model x t +1 = f ( x t ) .
Since the state x t is only three dimensional (horizontal position, vertical position and angular velocity), it is possible to do exhaustive veriﬁcation (by discretizing the state space with a very ﬁne grid) in this example, allowing us to create a “golden baseline”.
Note that this procedure is computationally expensive and the brute force search is over 239000 points.
The results are shown in Fig.
1 (right).
At a perturbation radius up until 0 .
02 , the exhaustive veriﬁcation can prove that the speciﬁcation is not violated for 73.98% of the test data points while using our convex relaxation approach achieved a lower bound of 70.00%.
As we increase the size of perturbation up to 0.06, the gap between the two increases to 22.05%.
However, the convex relaxation runs an order of magnitude faster given the same computational resources - this gap in computation time would increase even further as the state dimension grows.
4. V ALUE OF VERIFICATION
Consider the following scenario:
We are given a pair of ML models for a task - model A and model
B. We do not necessarily know how they have been trained but would like to evaluate them before deploying them.
Standard evaluation in terms of predictive performance on a hold-out test set cannot distinguish between the models.
We consider using veriﬁcation of a speciﬁcation relevant to the task as an additional way to evaluate these models.
If model A is more veriﬁable (the veriﬁcation bound is higher) than the other, we hope to show that model
A is better at solving the task than model
B. The following experiments demonstrate this for each of the speciﬁcations studied in this paper, demonstrating the value of veriﬁcation as a diagnostic tool for tasks with complex speciﬁcations.
[Figure]
Energy speciﬁcations and cascading errors in model dynamics: Models A and B were both trained to learn dynamics of a simple pendulum (details in Appendix D).
At δ = .
06 2 , the veriﬁcation bound (for the energy speciﬁcation (3)) for model
A is 64.16% and for model B is 34.53%.
When a model of a dynamical systems makes predictions that violate physical laws, its predictions can have signiﬁcant errors over long horizons.
To evaluate long-term predictive accuracy of the two models, we rollout of a long trajectory (200 steps) under models A, B and compare these to the ground truth (a simulation using the actual physics engine Mujoco).
The results are shown in
Fig.
For model
A the rolled out trajectory matches the ground truth much more closely.
Further, the trajectory from model A eventually stabilizes at the vertical position (a stable equilibrium for the pendulum) while that from model B keeps oscillating. airplane ship truck [Figure]
airplane ship truck
Model B:
Expected semantic distance.
Here, we verify models A and B with respect to the semantic speciﬁcation (15) and a veriﬁcation bound of 93.5% for model A and 84.3% for model
B. Thus, we expect that model B will be more susceptible to produce labels from semantically dissimilar categories than model
A under small perturbations of the input.
To show this, we visualize the decision boundaries of the two models projected onto a 2-D surface created by interpolating between three images which are in semantically similar categories (man-made transportation).
Fig.
3 shows the decision boundaries.
Indeed, we can observe that the model with a higher veriﬁcation bound displays higher semantic consistency in the decision boundaries.
C ONCLUSIONS
We have developed veriﬁcation algorithms for a new class of convex-relaxable speciﬁcations , that can model many speciﬁcations of interest in prediction tasks (energy conservation, semantic consistency, downstream task errors).
We have shown experimentally that our veriﬁcation algorithms can verify these speciﬁcations with high precision while still being tractable (requiring solution of a convex optimization problem linear in the network size).
We have shown that verifying these speciﬁcations can indeed provide valuable diagnostic information regarding the ultimate behavior of models in downstream prediction tasks.
While further work is needed to scale these algorithms to real applications, we have made signiﬁcant initial steps in this work.
Further, inspired by (Wong &
Kolter, 2018; Wong et al., 2018; Raghunathan et al., 2018), we believe that folding these veriﬁcation algorithms can help signiﬁcantly in training models that are consistent with various speciﬁcations of interest.
We hope that these advances will enable the development of general purpose speciﬁcationdriven AI, where we have general purpose systems that can take speciﬁcations stated in a high level language along with training data and produce ML models that achieve high predictive accuracy while also being consistent with application-driven speciﬁcations.
We believe that such an advance will signiﬁcantly aid and accelerate deployment of AI in applications with safety and security constraints. A CKNOWLEDGEMENTS
We thank Jost Tobias Springenberg and Jan Leike for careful proof-reading of this paper.
A P ROOF FOR L EMMA 3.
This can be done by proof of contradiction.
Suppose that the optimal value for C ( F, S in , S out ) is less than zero, but there exists a point in the set ( x F , y F , z F ) ∈ T ( F, S in , S out ) such that F ( x F , y F ) > 0 , then by deﬁnition ( x F , y F , z F ) ∈ C ( F, S in , S out ) which gives a contradiction.
Further we note that the objective in (13) is linear and the constraints imposed form a convex set thus it is a convex optimization problem.
B S OFTMAX R ELAXATION
We outline in detail the convex relaxations we used for the softmax function to obtain the set
C smax ( F, S in , S out ) .
Here we note that the denominator of a softmax cannot be zero as it is an exponential function over the real line, thus the following is true:
[Equation]
⇒
[Equation]
F
[Equation]
S in
[Equation]
S [Equation] [Equation] For the relaxation of the exponential function on auxiliary variable α , we note that the convex constraint is the linear interpolation between ( l K,i , exp( l K,i )) and ( u K,i , exp( u K,i )) shown in Fig. 4.
This linear interpolation is given by the following:
[Equation]
The bounds exp( y i ) ≤ α ≤ G U ( y i ) deﬁnes a convex set on α , further we note that
[Equation]
Thus, the following holds
[Equation]
For implementation of the lower bound exp( y i ) ≤ α i , we transform this bound into a set of linear tangential constraints (shown in Fig. 4), such that it can be solved with an LP.
The tangent at point η ∈ [ l, u ] is given by
[Equation]
This allows us to relax the exponential function by the following set of linear constraints:
[Equation]
C Q UADRATIC R ELAXATION
Here we outline in detail the convex relaxation of the quadratic function to obtain the set
C quad ( F, S in , S out ) .
The quadratic function is of the form
[Equation]
For simplicity of notation we will use α = (1 , x, y ) , we denote its ith element as x i .
For this particular case, the speciﬁcation can be written as F ( x, y ) =
(cid:80) ij Q ij α i α j .
Now we derive our relaxation for the quadratic function, Relax(Quad)( α, l, u ) .
With the constraints we place on α : l i ≤ α i ≤ u i , we can use the McCormick envelopes
[Equation]
to obtain constraints on the quadratic function.
Given (22) we can relax X =
αα T with the following set of constraints:
[Equation]
We can enforce extra constraints down the diagonal of the matrix X , as the diagonal is of the form
X ii =
α 2 i .
Since this is a convex function on the auxiliary variable α i , we can again use the convex constraint which is the linear interpolation between points ( l, l 2 ) and ( u, u 2 ) , given by:
[Equation]
Thus the following denotes a valid convex set on the diagonal of X :
[Equation]
Further we note that X ij is a symmetric semi-positive deﬁnite matrix, we can impose the following constraint (see Luo et al.
[Equation]
Explicitly we enforce the constraint:
[Equation]
A thing to note is that constraints
[Equation]
becomes the same constraint when X − αα T (cid:23) 0 is enforced, since X is a symmetric matrix thus one was dropped in (20).
We note that αα T ∈ Relax(Quad)( X, l, u ) , therefore the following holds:
[Equation]
D T RAINING D ETAILS
This paper is primarily focused on new techniques for veriﬁcation rather than model training.
However, generally speaking, training neural networks using standard methods does not produce veriﬁable models.
In fact, it has been observed that if one trains networks using standard cross entropy loss or even using adversarial training, networks that seem robust empirically are not easily veriﬁed due to the incomplete nature of veriﬁcation algorithms (Wong & Kolter, 2018).
In Wong & Kolter (2018); Wong et al.
; Gowal et al.
(2018), the importance of training networks with a special loss function that promotes veriﬁability was shown when attempting to obtain veriﬁably robust networks (against (cid:96) ∞ adversarial perturbations).
Similar observations have been made in (Gehr et al., 2018; Raghunathan et al., 2018).
The speciﬁcations we study in this paper (1,3, 5) build upon the standard adversarial robustness speciﬁcation.
Hence, we use the training method from Wong et al.
(2018) (which has achieved state of the art veriﬁable robustness against (cid:96)
∞ adversarial perturbations on MNIST and CIFAR-10) to train networks for our CIFAR-10 and MNIST experiments.
Cifar 10: Model A
We use a network that is veriﬁably robust to adversarial pertubations of size / 255 (where 255 is the range of pixel values on
of
the
test
examples
, with respect to the standard speciﬁcation that the output of the network should remain invariant to adversarial perturbations of
the input.
The network consists of 4 convolutional layers and 3 linear layers
in total paramters Cifar 10: Model B We use a network that is veriﬁably robust to adversarial pertubations of size 2 / 255 (where 255 is the range of pixel values) on 39 .
25% of the test examples, with respect to the standard speciﬁcation that the output of the network should remain invariant to adversarial perturbations of the input.
The architecture used is the same as Model A above.
Mujoco We trained a network using data collected from the Mujoco simulator (Todorov et al., 2012), in particular, we used the pendulum swingup environment in the DM control suite (Tassa et al., 2018).
The pendulum is of length 0.5m and hangs 0.6m above ground.
When the perturbation radius is 0.01, since the pendulum is 0.5m in length, the equivalent perturbation in space is about 0.005 m in the x and y direction.
The perturbation of the angular velocity is ω ± 0 .
1 radians per second.
The pendulum environment was used to generate 90000 (state, next state) pairs, 27000 was set aside as test set.
For training the timestep between the state and next state pairs is chosen to be 0.1 seconds (although the simulation is done at a higher time resolution to avoid numerical integration errors).
The pendulum models consists of two linear layers in total 120 parameters and takes (cos( θ ) , sin( θ ) , ω/ 10) as input.
Here θ is the angle of the pendulum and ω is the angular velocity.
The data is generated such that the initial angular velocity lies between (-10, 10), by scaling this with a factor of 10 ( ω/ 10 ) we make sure (cos( θ ) , sin( θ ) , ω/ 10) lies in a box where each side is bounded by [-1, 1].
Pendulum: Model A
We train with an (cid:96) 1 loss and energy loss on the next state prediction, the exact loss we impose this model is (we denote ( w T , h T , sω T ) as ground truth state): l ( f
[Equation]
T ) (cid:107) (cid:124) (cid:123)(cid:122) (cid:125) (cid:96) 1 loss +
[Equation]
(cid:124) (cid:123)(cid:122) (cid:125) energy difference loss
[Equation]
ReLU( E ( f ( w, h, sω ))
− E ( w, h, sω ))
(cid:124) (cid:123)(cid:122) (cid:125) increase in energy loss ,
[Equation]
25) where s = 0 .
1 is a scaling parameter we use on angular velocity.
E is given in (2).
The loss we calculate on the test set is of the (cid:96) 1 loss only and we obtain 0 . 072 .
Pendulum: Model B We train with an (cid:96) 1 loss on the next state prediction
[Equation]
[Table]
Digit Sum: Model A
We use a network, consisting of two linear layers in total 15880 parameters.
This network is veriﬁably robust to adversarial pertubations of size 25 / 255 (where 255 is the range of pixel values) on 68 .
16% of the test examples with respect to the standard speciﬁcation.
The standard speciﬁcation being that the output of the network should remain invariant to adversarial perturbations of the input.
Falsiﬁcation: The falsiﬁcation method (8) is susceptible to local optima.
In order to make the falsiﬁcation method stronger, we use 20 random restarts to improve the chances of ﬁnding better local optima.
E S CALING AND I MPLEMENTATION :
The scaling of our algorithm is dependent on the relaxations used.
If the relaxations are a set of linear constraints, the scaling is linear with respect to the input and output dimensions of the network, if they are quadratic constraints they will scale quadratically.
For the Semantic Speciﬁcation and Downstream Speciﬁcation, the relaxations consisted of only linear constraints thus can be solved with a linear program (LP).
For these tasks, we used GLOP as the LP solver.
This solver takes on approximately 3-10 seconds per data point on our desktop machine (with 1 GPU and 8G of memory) for the largest network size we handled which consists of 4 convolutional layers and 3 linear layers and in total 860000 parameters.
For the conservation of energy, we used SDP constraints - this relaxation scales quadratically with respect to the input and output dimension of the network.
To solve for these set of constraints we used CVXPY, which is much slower than GLOP, and tested on a smaller network which has two linear layers withparameters.
F D ISTANCES BETWEEN CIFAR-10 LABELS IN W ORD N ET
The synsets chosen from WordNet are: airplane.n.01, car.n.01, bird.n.01, cat.n.01, deer.n.01, dog.n.01, frog.n.01, horse.n.01, ship.n.01, truck.n.01.
The path similarity, here we denote as x , in WordNet is between [0, 1/3], to make this into a distance measure between labels which are not similar we chose d = 1 / 3 − x . automobile bird cat deer dog frog horse ship truck airplane 0.
[Table]
G
CIFAR-10: S EMANTIC D ECISION B OUNDARIES truck automobile ship
Model A: truck automobile ship
Model B:
H C OMPARISON OF T IGHTNESS
For veriﬁcation purposes, it is essential to ﬁnd a convex set which is sufﬁciently tight such that the upper bound found on F ( x, y ) is a close representative of max x ∈ S in F ( x, f ( x )) .
We use the difference between the veriﬁcation bound and adversarial bound as a measure of how tight the veriﬁcation algorithm is.
More concretely, the difference between the adversarial bound and veriﬁcation bound should decrease as the veriﬁcation scheme becomes tighter.
Here, we compare the tightness of two different convex-relaxations for the physics speciﬁcation on the pendulum model (Model B).
The SDP convex relaxation (the one we have shown in the main paper) uses the following relaxation of the quadratic function:
[Equation]
To enforce the constraint X − αα T (cid:23) 0 means that the complexity of veriﬁcation increases quadratically with respect to the dimension of α .
Thus, we can trade complexity for a looser convexrelaxation of the quadratic function given by following:
[Equation]
Comparing both relaxations can give us a gauge of how much enforcing the SDP constraint X − αα T (cid:23) 0 gives us in practise.
The results are shown in Fig.
What we see is that enforcing the
SDP constraints at a small perturbation radius (i.e. δ ≤ 0 . 02 ), gives us a veriﬁcation bound which allows us to verify only 0 .
9% more test points.
As the perturbation radius increases SDP constraint allowed us to verify 4 .
8 % more test data points than without.
In this case, these results suggest that the perturbation radius δ , which is less or equal to 0.06, is sufﬁciently small such that enforcing higher order constraints makes for only slightly tighter convex-relaxations.
In general it is often the case that we can enforce the cheapest (computationally) relaxations if our input set S in is sufﬁciently small.
[Figure]
[Figure]
[Figure]
|
[Figure]
| C |
H. D OWNSTREAM S PECIFICATION : T IGHTNESS AS WE INCREASE DIGITS TO SUM
For the digit sum problem, we also investigated into the tightness of the veriﬁcation bound with respect to the adversarial bound as we increase the number of digits, N , which we sum over.
We used the same network (Model A) for all results.
The results are shown in Fig.
The nominal percentage (namely when the input set S in consists only of the nominal test point) which satisﬁes the speciﬁcation decreases from to 72.6% to 53.3% as N increases from 2 to 4.
At the same time the difference (we denote as ∆ ) between the veriﬁcation bound and adversarial bound increases as N increases (shown in top left plot).
We note that at small perturbation radius where δ = 2 / 255 , ∆ = 4 .
2% , 14 . 1% , 26 . 1% for N = 2 , 3 , 4 respectively.
The increase in ∆ is linear with respect to increase in N when δ is sufﬁciently small.
At a larger perturbation radius, as the percentage of test data points veriﬁable tends towards zero, the rate of increase in ∆ as N increases slows down.
Concretely, at δ = 6 / 255 , ∆ = 19 .
8% for N = 2 , 3 , 4 .
Here, we note that the increase in ∆ going from N = 3 to 4 is only 2.1%.
H. T IGHTNESS WITH A T OY E XAMPLE
To give intuition regarding the tightness of a convex-relaxed set, we use a toy example to illustrate.
Here we consider the true set, T ( F, S in , S out ) where S in = { x : − b x ≤ x ≤ b x } , S out =
{ y : − b y ≤ y ≤ b y } and F ( x, y ) =
x 2 − y 2 ≤ 4 (throughout this example we keep b y = 9 ).
We can ﬁnd its corresponding convex-relaxed set using Relax(Quad) which results in the following:
[Equation]
 
[Equation]
The sets are depicted in Fig.
8, where we see that C is simply a box around the true set T (green).
With this toy example we can also ask the question what the difference in volume between the two sets is, as this can give us an idea of the tightness of the convex relaxation.
The difference is shown on the right plot of Fig.
We can see that as we increase b x , while b y = 9 , the fractional difference in volume increases linearly up to when b x is the same size as b y = 9 . [Figure]
δ [Figure]
P [Figure] r c e n t a g e
V e r
i f i e d i n T e s t S e t
MNIST: Entropy Specification
[Figure]
I E NTROPY S PECIFICATION
Another speciﬁcation which we considered was the entropy speciﬁcation.
Lets consider the scenario, where we would never want our network to be 100% conﬁdent about a certain image.
That is to say we would like this network to always maintain a level of entropy E .
Thus for this our speciﬁcation is the following:
[Equation]
We note that this speciﬁcation can be rewritten as the following:
[Equation]
Here the relaxations we use is constructed upon Relax(exp) and Relax(Quad) .
Concretely, our relaxation is given by:
[Equation]
   here n is the number of labels or equivalently the dimensions of the output y .
We show the results in Fig.
The model we have used is a MNIST classiﬁer with two linear layers and 15880 parameters.
At a perturbation radius of δ = 2 / 255 , the difference between the veriﬁcation and adversarial bound is 19 .
9% , as we increase this to 6 pixel perturbations, the difference between the two bounds becomes close to 44% .